Azure Storage limitations

azure

Are there any limitations to the amount of files that can be stored in Azure Storage?

I am concerned about storing all uploads in a single directory and wondering if they should be sharded somehow into a collection of directories.

I've searched the docs but can not find anything about this.

Best Answer

Windows Azure Storage (and I'm assuming you're referring to blobs) does not have a limit to the number of objects you can store. It's only limited by the 200TB-per-storage-account cap. And you can have multiple storage accounts in your subscription (each storage account maps to a namespace such as mystorage.blob.core.windows.net).

Same goes for Table storage: no limit to the number of tables or the number of entities per table. Just the 200TB restriction.

Now, as far as directories go: Blob storage is organized by namespace.blob.core.windows.net/containername/blobname.ext. These aren't really directories. If you want true filesystem directories, you'll need to set up a disk in a blob (basically a vhd formatted as ntfs / ext3 / ext4 etc) and mounted to your OS disk. A disk is limited to 1TB (the max. size of a page blob). Once you do that, you have a complete file system to write to. Just a warning though: Only one VM can attach to a drive at a given time, so this isn't ideal if you're trying to have a file-share set up (direct blob storage is much better for that, or you'll need to set up an smb server for yourself).

One more thing about Azure storage: You don't have to worry about sharding; this is a massive-scale durable data storage system. Each blob is in its own logical partition, and the storage service organizes / reorganizes storage as necessary. This is something you don't need to worry about. The only thing I guide against is storing all objects in a single container, since enumerating blobs within a container could take a while if you have 10's of thousands of objects there. I typically don't enumerate containers, since I usually store blob uri's in another database as metadata (maybe in a SQL store, or MongoDB document store).

Related Solutions

Azure – Setting write permissions for temporary storage provided by Azure (Linux-based VM)

ln -sTf /mnt/pgsql_tmp $PGDATA/base/pgsql_tmp  # did you get that correct?
chown -R postgres              /mnt/pgsql_tmp
chmod o+x    /mnt          # note: no -R,  this turns 750 to 751 for /mnt only

The effect:

 su - postgres
 echo 1 > /mnt/file   # fails
 ls   /mnt            # fails
 cat /mnt/anything    # fails
 echo 1 >  /mnt/pgsql_tmp/file   # works
 ls /mnt/pgsql_tmp               # works
 cat /mnt/pgsql_tmp/file         # works

The chmod o+x means to give [o]thers the +e[x]ecute bit, which gives them a permission to traverse across /mnt into any subdirectory. Other permissions are not necessary for /mnt only for /mnt/pgsql_tmp

Azure – Move files between Azure and S3/Glacier

Your question is unclear, please edit your question to add more details. Do you want to mirror files between Azure storage and AWS storage without any kind of VM?

AWS Data Pipeline is a service you should look at. You might have to write a little bit of code, but it would run on AWS managed infrastructure. I think this will work.

Someone probably makes a service that will do it for you, if you pay them - a quick search found Cloudsfer which may or may not do what you want.

Otherwise, you use the public API for the service you're interested in. S3 API, Glacier. You need to run code for this.

There will also be plenty of different libraries or methods to access S3 made by others. There's also the AWS Storage Gateway, which does require a VM to be run, but makes accessing S3 easy.

Best Answer

Related Solutions

Azure – Setting write permissions for temporary storage provided by Azure (Linux-based VM)

Azure – Move files between Azure and S3/Glacier

Related Topic