blobxfer Command-Line Usage
blobxfer
operates using a command followed by options. Each
command will be detailed along with all options available.
Quick Navigation
Commands
download
Downloads remote Azure paths, which may contain many resources, to the local machine. This command requires at the minimum, the following options if invoked without a YAML configuration file:
--mode
specifies the source Azure Storage mode. This defaults toauto
which will target Azure Blob storage (any blob type). To source from Azure File storage, set this option tofile
.--storage-account
for the source remote Azure path--remote-path
for the source remote Azure path. This must have, at the minimum, a container or file share name.--local-path
Additionally, an authentication option for the storage account is required. Please see the Authentication sub-section below under Options.
upload
Uploads local paths to a remote Azure path or set of remote Azure paths. The local path may contain many resources on the local machine. This command requires at the minimum, the following options if invoked without a YAML configuration file:
--mode
specifies the destination Azure Storage mode. This defaults toauto
which will target Azure Blob storage (any blob type). To upload to Azure File storage, set this option tofile
.--local-path
--storage-account
for the destination remote Azure path--remote-path
for the destination remote Azure path. This must have, at the minimum, a container or file share name.
Additionally, an authentication option for the storage account is required. Please see the Authentication sub-section below under Options.
If piping from stdin
, --local-path
should be set to -
as per
convention.
synccopy
Synchronously copies remote Azure paths to other remote Azure paths. This command requires at the minimum, the following options if invoked without a YAML configuration file:
--mode
specifies the source Azure Storage mode. This defaults toauto
which will target Azure Blob storage (any blob type). To upload to Azure File storage, set this option tofile
.--storage-account
for the source remote Azure path--remote-path
for the source remote Azure path. This must have, at the minimum, a container or file share name.--sync-copy-dest-mode
for the destination mode. Ifauto
is specified for this option, the destination mode will be set the same as the source--mode
. These options need not be the same which allows for object-transform copy operations.--sync-copy-dest-remote-path
for the destination remote Azure path. This must have, at the minimum, a container or file share name.--sync-copy-dest-storage-account
for the destination remote Azure path
Additionally, an authentication option for both storage accounts is required.
Please see the Authentication
and Connection
sub-section below under the
next section.
Options
General
--config
specifies the YAML configuration file to use. This can be optionally provided through an environment variableBLOBXFER_CONFIG_FILE
.--chunk-size-bytes
is the chunk size in bytes. For downloads, this is the maximum length of data to transfer per request. For uploads, this corresponds to one of block size for append and block blobs, page size for page blobs, or file chunk for files. Only block blobs can have a block size of up to 100MiB, all others have a maximum of 4MiB.--enable-azure-storage-logger
enables the Azure Storage logger.--file-attributes
or--no-file-attributes
controls if POSIX file attributes (mode and ownership) should be stored or restored. Note that to restore uid/gid,blobxfer
must be run as root or under sudo.--file-md5
or--no-file-md5
controls if the file MD5 should be computed.--local-path
is the local resource path. Set to-
if piping fromstdin
.--log-file
specifies the log file to write to. This must be specified for a progress bar to be output to console.--mode
is the operating mode. The default isauto
but may be set toappend
,block
,file
, orpage
. If specified with theupload
command, then all files will be uploaded as the specifiedmode
type. Ifmode
isauto
while uploading, then.vhd
and.vhdx
files are uploaded automatically as page blobs while all other files are uploaded as block blobs. If specified withdownload
, then only remote entities with thatmode
type are downloaded. Note thatfile
should be specified for themode
if interacting with Azure File shares.--overwrite
or--no-overwrite
controls clobber semantics at the destination.--progress-bar
or--no-progress-bar
controls if a progress bar is output to the console.--log-file
must be specified for a progress bar to be output.-q
or--quiet
enables quiet mode--recursive
or--no-recursive
controls if the source path should be recursively uploaded or downloaded.--remote-path
is the remote Azure path. This path must contain the Blob container or File share at the begining, e.g.,mycontainer/vdir
--resume-file
specifies the resume database to write to or read from. Resume files should be specific for a session.--show-config
will show the configuration for the execution. Use caution with this option as it will output secrets.-h
or--help
can be passed at every command level to receive context sensitive help.-v
or--verbose
increases logging verbosity
Authentication
blobxfer
supports both Storage Account access keys and Shared Access
Signature (SAS) tokens. One type must be supplied with all commands in
order to successfully authenticate against Azure Storage. These options are:
--sas
is a shared access signature (SAS) token. This can can be optionally provided through an environment variableBLOBXFER_SAS
instead.--storage-account-key
is the storage account access key. This can be optionally provided through an environment variableBLOBXFER_STORAGE_ACCOUNT_KEY
instead.--sync-copy-dest-sas
is a shared access signature (SAS) token for the destination Azure Storage account for thesynccopy
command. This can be optionally provided through an environment variableBLOBXFER_SYNC_COPY_DEST_SAS
instead.--sync-copy-dest-storage-account-key
specifies the destination Azure Storage account key for thesynccopy
command. This can be optionally provided through an environment variableBLOBXFER_SYNC_COPY_DEST_STORAGE_ACCOUNT_KEY
instead.
Concurrency
Please see the performance considerations document for more information regarding concurrency options.
--crypto-processes
is the number of decryption offload processes to spawn.0
will in-line the decryption routine with the main thread.--disk-threads
is the number of threads to create for disk I/O.--md5-processes
is the number of MD5 offload processes to spawn for comparing files withskip_on
md5_match
.--transfer-threads
is the number of threads to create for transferring to/from Azure Storage.
Connection
--endpoint
is the Azure Storage endpoint to connect to; the default is Azure Public regions, orcore.windows.net
. Note that this is the base endpoint name and not a full URL (as blobxfer deals with both Azure Blob Storage and Files). You can use one of the following if you are not connecting to an Azure Public region:- Azure China Cloud:
core.chinacloudapi.cn
- Azure Germany Cloud:
core.cloudapi.de
- Azure US Government Cloud:
core.usgovcloudapi.net
- Azure China Cloud:
--max-retries
is the maximum number of retries for a request. Negative values result in unlimited retries. The default is 1000.--proxy-host
specifies the IP:port for an HTTP Proxy if required. This can be optionally provided through an environment variableBLOBXFER_PROXY_HOST
instead.--proxy-username
specifies a username for an HTTP Proxy if required. This can be optionally provided through an environment variableBLOBXFER_PROXY_USERNAME
instead.--proxy-password
specifies the password for the username for an HTTP Proxy if required. This can be optionally provided through an environment variableBLOBXFER_PROXY_PASSWORD
instead.--storage-account
specifies the storage account to use. This can be optionally provided through an environment variableBLOBXFER_STORAGE_ACCOUNT
instead.--sync-copy-dest-storage-account
specifies the destination remote Azure storage account for thesynccopy
command. This can be optionally provided through an environment variableBLOBXFER_SYNC_COPY_DEST_STORAGE_ACCOUNT
instead.--sync-copy-dest-remote-path
specifies the destination remote Azure path under the synchronous copy destination storage account.--timeout
is the timeout value, in seconds, applied to both connect and read operations. To apply separate values for these operations individually, use YAML configuration instead.
Encryption
--rsa-private-key
is the RSA private key in PEM format to use. This can be provided for uploads but must be specified to decrypt encrypted remote entities for downloads. This can be optionally provided through an environment variableBLOBXFER_RSA_PRIVATE_KEY
.--rsa-private-key-passphrase
is the RSA private key passphrase. This can be optionally provided through an environment variableBLOBXFER_RSA_PRIVATE_KEY_PASSPHRASE
.--rsa-public-key
is the RSA public key in PEM format to use. This can only be provided for uploads. This can be optionally provided through an environment variableBLOBXFER_RSA_PUBLIC_KEY
.
Filtering
--exclude
is an exclude pattern to use; this can be specified multiple times. Exclude patterns are applied after include patterns. If both an exclude and an include pattern match a target, the target is excluded.--include
is an include pattern to use; this can be specified multiple times
Skip On
--skip-on-filesize-match
will skip the transfer action if the filesizes match between source and destination. This should not be specified for encrypted files.--skip-on-lmt-ge
will skip the transfer action:- On upload if the last modified time of the remote file is greater than or equal to the local file.
- On download if the last modified time of the local file is greater than or equal to the remote file.
--skip-on-md5-match
will skip the transfer action if the MD5 hash match between source and destination. This can be transparently used through encrypted files that have been uploaded withblobxfer
.
Vectored IO
Please see the Vectored IO document for more information
regarding Vectored IO operations in blobxfer
.
--distribution-mode
is the Vectored IO distribution modedisabled
which is default (no Vectored IO)replica
which will replicate source files to target destinations on upload. Note that replicating across multiple destinations will require a YAML configuration file.stripe
which will stripe source files to target destinations on upload. Note that striping across multiple destinations will require a YAML configuration file.
--stripe-chunk-size-bytes
is the stripe chunk width for stripe-based Vectored IO operations
Other
--access-tier
sets the access tier of the object (block blobs only) for upload and synccopy operations. If this is not specified, which is the default, then no access tier is actively set for the object which then inherits from whatever the default access tier is for the account. Note that the access tier is applied to all files transferred for the blobxfer invocation. This can only be set for Blob Storage and General Purpose V2 storage accounts. Valid values for this option arehot
,cool
, orarchive
.--delete
deletes extraneous files (including blob snapshots if the parent is deleted) at the remote destination path on uploads and at the local resource on downloads. This actions occur after the transfer has taken place, similarly to rsync's delete after option. Note that this interacts with other filters such as--include
and--exclude
.--one-shot-bytes
controls the number of bytes to "one shot" a block Blob upload. The maximum value that can be specified is 256MiB. This may be useful when using account-level SAS keys and enforcing non-overwrite behavior.--rename
renames a single file to the target destination or source path. This can only be used when transferring a single source file to a destination and can be used with any command. This is automatically enabled when usingstdin
as a source.--stdin-as-page-blob-size
allows a page blob size to be set if known beforehand when usingstdin
as a source and the destination is a page blob. This value will automatically be page blob boundary aligned.--strip-components N
will strip the leadingN
components from the local file path on upload or remote file path on download. The default is0
.
Example Invocations
download
Examples
Download an Entire Encrypted Blob Container to Current Working Directory
blobxfer download --storage-account mystorageaccount --sas "mysastoken" --remote-path mycontainer --local-path . --rsa-private-key ~/myprivatekey.pem
Download an Entire File Share to Designated Path and Skip On Filesize Matches
blobxfer download --mode file --storage-account mystorageaccount --storage-account-key "myaccesskey" --remote-path myfileshare --local-path /my/path --skip-on-filesize-match
Download only Page Blobs in Blob Container Virtual Directory Non-recursively and Cleanup Local Path to Match Remote Path
blobxfer download --mode page --storage-account mystorageaccount --storage-account-key "myaccesskey" --remote-path mycontainer --local-path /my/pageblobs --no-recursive --delete
Resume Incomplete Downloads Matching an Include Pattern and Log to File and Restore POSIX File Attributes
blobxfer download --storage-account mystorageaccount --storage-account-key "myaccesskey" --remote-path mycontainer --local-path . --include '*.bin' --resume-file myresumefile.db --log-file blobxfer.log --file-attributes
Download a Blob Snapshot
blobxfer download --storage-account mystorageaccount --sas "mysastoken" --remote-path "mycontainer/file.bin?snapshot=2017-04-20T02:12:49.0311708Z" --local-path .
Download an Entire File Share Snapshot
blobxfer download --mode file --storage-account mystorageaccount --sas "mysastoken" --remote-path "myshare?snapshot=2017-04-20T02:12:49.0311708Z" --local-path .
Download using a YAML Configuration File
blobxfer download --config myconfig.yaml
upload
Examples
Upload Current Working Directory as Encrypted Block Blobs Non-recursively
blobxfer upload --storage-account mystorageaccount --sas "mysastoken" --remote-path mycontainer --local-path . --no-recursive --rsa-public-key ~/mypubkey.pem
Upload Specific Path Recursively to a File Share, Store File MD5 and POSIX File Attributes to a File Share and Exclude Some Files
blobxfer upload --mode file --storage-account mystorageaccount --sas "mysastoken" --remote-path myfileshare --local-path . --file-md5 --file-attributes --exclude '*.bak'
Upload Single File with Resume and Striped Vectored IO into 512MiB Chunks
blobxfer upload --storage-account mystorageaccount --sas "mysastoken" --remote-path mycontainer --local-path /some/huge/file --resume-file hugefileresume.db --distribution-mode stripe --stripe-chunk-size-bytes 536870912
Upload Specific Path but Skip On Any MD5 Matches, Store File MD5 and Cleanup Remote Path to Match Local Path
blobxfer upload --storage-account mystorageaccount --sas "mysastoken" --remote-path mycontainer --local-path /my/path --file-md5 --skip-on-md5-match --delete
Upload From Piped stdin
curl -fSsL https://some.uri | blobxfer upload --storage-account mystorageaccount --sas "mysastoken" --remote-path mycontainer --local-path -
Upload using a YAML Configuration File
blobxfer upload --config myconfig.yaml
synccopy
Examples
Synchronously Copy an Entire Path Recursively to Another Storage Account
blobxfer synccopy --storage-account mystorageaccount --sas "mysastoken" --remote-path mysourcecontainer --sync-copy-dest-storage-account mydestaccount --sync-copy-dest-storage-account-key "mydestkey" --sync-copy-dest-remote-path mydestcontainer
Synchronously Copy using a YAML Configuration File
blobxfer synccopy --config myconfig.yaml
General Notes
blobxfer
does not take any leases on blobs or containers. It is up to the user to ensure that blobs are not modified while download/uploads are being performed.- No validation is performed regarding container and file naming and length restrictions.
blobxfer
will attempt to download from blob storage as-is. If the source filename is incompatible with the destination operating system, then failure may result.- When using SAS, the SAS key must have container- or share-level permissions if performing recursive directory upload or container/file share download.
- If uploading via service-level SAS keys, the container or file share must
already be created in Azure storage prior to upload. Account-level SAS keys
with the signed resource type of
c
(i.e., container-level permission) is required for to allow conatiner or file share creation. - When uploading files as page blobs, the content is page boundary
byte-aligned. The MD5 for the blob is computed using the final aligned data
if the source is not page boundary byte-aligned. This enables these page
blobs or files to be skipped during subsequent download or upload with the
appropriate
skip_on
option, respectively. - Globbing of wildcards must be disabled by your shell (or properly quoted)
during invoking
blobxfer
such that include and exclude patterns can be read verbatim without the shell expanding the wildcards.