Google Storage – Low upload speed with gsutil

google-cloud-platformupload

We are currently evaluating using Google Storage Nearline for backups. The servers in question are located within a Hetzner data center, on average 80-100Mbit/s upstream and 300-500Mbit/s downstream. We're using Googles gsutil for accessing the storage and its rsync command to synchronize our backup directories. While downloads are adequately fast, uploading files takes way too long: Maximum upload speed is about 8-11Mbit/s. This is only happening with gsutil and not really usable for a backup storage location.
I know large files should be uploaded using parallel composite uploads, however Google recommends to not enable it for Nearline.

Now, my question is, are these low upload speeds expected? Is there a way to optimize it? As it is, we generate more data than we can push to Google Storage, and will have to look for other solutions.

Best Answer

If you have a large number of objects to synchronize you might want to use the gsutil -m option, to perform parallel (multi-threaded/multi-processing) synchronization: gsutil -m rsync -d -r data gs://mybucket/data

“BE CAREFUL WHEN USING -d OPTION!”.

Related Solutions

Centos – Low upload speed (CentOS 6)

You don't provide much context around the dmesg timing, but if this is off of a fresh boot that immediately started exhibiting the problem, then your issue is that the switch port is flapping, which could be caused by just about anything (your driver, your hardware, the cable, switch's hardware, or the switch's configuration). Given that you said this is being hosted by a "company", the support division of any hosting company worth even a fraction of their salt should take this issue at least as far as eliminating everything other than your driver from consideration as the culprit (e.g. by plugging in and testing with an alternate piece of hardware). A managed hosting company (like BlackMesh or Rackspace) would take care of the whole issue from end-to-end.

Gsutil config for google-cloud-storage

gsutil will use the ~/.boto file. By default, gsutil will look for the configuration file in /etc/boto.cfg and ~/.boto.

If these files do not exist, it will use the legacy_credentials instead.

You can change all of this behavior by modifying the the BOTO_PATH and BOTO_CONFIG environment parameters.

Here is the documentation on how to use those environment variables.

BOTO_PATH: gsutil will load each configuration file found in the path in order. You may want to use this environment variable if you want to share a boto config amongst users. An example would be something like this :

BOTO_PATH=/etc/group/.boto.cfg:/home/user/.boto

BOTO_CONFIG: Use this variable to set the preference order. An example would be something like this :

BOTO_CONFIG=~/testa/.boto:~/testb/.boto .

This would put the preference on testa's boto file. If that is not there then gsutil would load the testb boto.

Best Answer

Related Solutions

Centos – Low upload speed (CentOS 6)

Gsutil config for google-cloud-storage

Related Topic