blobxfer Performance Considerations
Please read the following carefully regarding considerations that should
be applied with regard to performance and
please review the
Azure Storage Scalability and Performance Targets
for an overview of general performance targets that apply to Azure Blobs,
File shares and Storage Account types (GRS, LRS, ZRS, etc).
blobxferoffers four concurrency knobs. Each one should be tuned for maximum performance according to your system and network characteristics.
- Disk threads: concurrency in reading (uploads) and writing (downloads) to disk is controlled by the number of disk threads.
- Transfer threads: concurrency in the number of threads transferring from/to Azure Storage is controlled by the number of transfer threads.
- MD5 processes: computing MD5 for potential omission from transfer due
md5_matchbeing specified are offloaded to the specified number of processors.
- Crypto processes: decrypting encrypted blobs and files can be offloaded to the specified number of processors. Due to the inherent non-parallelizable encryption algorithm used, this is ignored for encryption (uploads).
- The thread concurrency options (disk and transfer) can be set to a non-positive number to be automatically set as a multiple of the number of cores available on the machine.
- For uploads, there should be a sufficient number of disk threads to ensure that all transfer threads have work to do. For downloads, there should be sufficient number of disk threads to write data to disk so transfer threads are not artificially blocked.
Chunk sizing refers to the
chunk_size_bytes option and the meaning of which
varies upon the context of uploading or downloading. To ensure that
high-throughput block blob
upload speeds are enabled (only for block blobs), your chunk size should be
greater than 4MiB. With default chunk sizes, this behavior should be enabled
blobxfer uses two timeout values, a connect timeout and a read timeout.
The read timeout should be set to something that is reasonably large
enough to transmit each chunk given your bandwidth limitations.
For uploads, chunk sizes correspond to the maximum amount of data to transfer with a single request. The Azure Storage service imposes maximums depending upon the type of entity that is being written. For block blobs, the maximum is 100MiB (although you may "one-shot" up to 256MiB). For page blobs, the maximum is 4MiB. For append blobs, the maximum is 4MiB. For Azure Files, the maximum is 4MiB.
For block blobs, setting the chunk size to something greater than 4MiB will
not only allow you larger file sizes (recall that the maximum number of
blocks for a block blob is 50000, thus at 100MiB blocks, you can create a
4.768TiB block blob object) but will allow you to amortize larger portions of
data transfer over each request/response overhead. Additionally, this will
activate HTBB (please see chunk size above) for high throughput transfers.
blobxfer can automatically select the proper block size given your file,
but will not automatically tune the chunk size as that depends upon your
system, network characteristics, and source storage scalability targets.
For downloads, chunk sizes correspond to the maximum amount of data to
request from the server for each request. It is important to keep a balance
between the chunk size and the number of in-flight operations afforded by
transfer_threads concurrency control.
blobxfer does not automatically
tune this (but can automatically set it to a value that should work for
most situations) due to varying system and network conditions.
Additionally, disk write performance is typically lower than disk read
performance so you need to ensure that the number of
disk_threads is not
set to a very large number to prevent thrashing and highly random write
For sync copy sources which are block blobs, the block size will determine the chunk size. Thus, block blobs with block sizes which fall below the HTBB chunk size cut off will not be eligible for HTBB throughput speeds.
Azure File Share Performance
File share performance can be "slow" or become a bottleneck, especially for file shares containing thousands of files as multiple REST calls must be performed for each file. Currently, a single file share has a limit of up to 60 MB/s and 1000 8KB IOPS. Please refer to the Azure Storage Scalability and Performance Targets for performance targets and limits regarding Azure Storage File shares. If scalable high performance is required, consider using Blob storage instead.
MD5 hashing will impose some performance penalties to check if the file should be uploaded or downloaded. For instance, if uploading and the local file is determined to be different than its remote counterpart, then the time spent performing the MD5 comparison is effectively "lost."
Client-side encryption will naturally impose a performance penalty on
blobxfer both for uploads (encrypting) and downloads (decrypting) depending
upon the processor speed and number of cores available. Additionally, for
uploads, encryption is not parallelizable within an object and is in-lined
with the main process.
Resume Files (Databases)
Enabling resume support may slightly impact performance as a key-value shelve for bookkeeping is kept on disk and is updated frequently.
As of requests 2.6.0 and Python versions < 2.7.9 (i.e., interpreter found on
default Ubuntu 14.04 installations, 16.04 is not affected), if certain
packages are installed, as those found in
requests[security] then the
underlying urllib3 package will utilize the
ndg-httpsclient package which
pyOpenSSL. This will ensure the peers are fully validated. However,
this incurs a rather larger performance penalty. If you understand the
potential security risks for disabling this behavior due to high performance
requirements, you can either remove
ndg-httpsclient or use
blobxfer in a
virtualenv environment without the
ndg-httpsclient package. Python
versions >= 2.7.9 are not affected by this issue.
requests uses) may use
may result in exceptions being thrown that are not normalized by
This may result in exceptions that should be retried, but are not. It is
recommended to upgrade your Python where
pyOpenSSL is not required for
fully validating peers and such that
blobxfer can operate without
pyOpenSSL in a secure fashion. You can also run
blobxfer via Docker
or in a virtualenv environment without