Google Cloud Storage – How to Enable Parallel Downloads

google-cloud-storage

Does anyone have experience with Google Cloud Storage? Is it made for a potential 10K of parallel requests to download 10-70mb files? Or would that require a CDN?

Best Answer

https://cloud.google.com/storage/quotas

There is no limit to reads of an object. Buckets initially support roughly 5000 reads per second and then scale as needed.

You may have trouble if you frequently go from zero to 10k parallel requests (assuming they're really parallel, and not, say, "in the same couple of minutes") rapidly:

As a bucket approaches its IO capacity limit, Cloud Storage typically takes on the order of minutes to detect and accordingly redistribute the load across more servers. Consequently, if the request rate on your bucket increases faster than Cloud Storage can perform this redistribution, you may run into temporary limits, specifically higher latency and error rates. Ramping up the request rate gradually for your buckets, as described below, avoids such latency and errors.