How to run gsutil rsync without auth

google-cloud-platformgoogle-cloud-storagegsutil

aws s3 sync provides a parameter named --no-sign-request, so user can sync a publicly-visible bucket without any auth in advance.

gsutil help anon says that:

gsutil users can access publicly readable data without obtaining
credentials. For example, the gs://uspto-pair bucket contains a number
of publicly readable objects, so any user can run the following command
without first obtaining credentials:

gsutil ls gs://uspto-pair/applications/0800401*

Users can similarly download objects they find via the above gsutil ls
command.

However, running this command without login makes gsutil report errors:

➜  mirror-docker git:(v2) gsutil ls 'gs://uspto-pair/applications/0800401*'
Your credentials are invalid. Please run
$ gcloud auth login

Currently, we are setting up a mirror for common buckets on gcloudstorage, and want to find a way to perform sync without auth in advance. Is it possible?

Best Answer

Prior using the gsutil tool, you need to authenticate yourself as per this: https://cloud.google.com/storage/docs/quickstart-gsutil#before-you-begin which is one of the requirements. Even if you have a publicly accessible bucket and/or object you cannot unfortunately perform a sync without authentication in advance.

Related Solutions

Slow synchronisation stage on gsutil rsync

I have discovered that changing

output_chunk.writelines(unicode(''.join(current_chunk)))

output_chunk.write(unicode(''.join(current_chunk)))

in /gsutil/gslib/commands/rsync.py makes a big difference. Thanks to Mike from the GS Team for his help - this simple change has been rolled out on github already:

https://github.com/GoogleCloudPlatform/gsutil/commit/a6dcc7aa7706bf9deea3b1d243ecf048a06a64f2

Using JSON keys with google cloud gsutil

The short version is to run the following command and follow instructions:

gsutil config -e

The gsutil tool has built-in help which can be consulted for all kinds of options and modes of operation. When running gsutil help creds, one of the help options recommended when running gsutil alone, we can read the section on "OAuth2 Service Account" to see the instructions for using a service account's json key file:

OAuth2 Service Account:

This is the preferred type of credential to use when authenticating on
behalf of a service or application (as opposed to a user). For example, if
you will run gsutil out of a nightly cron job to upload/download data,
using a service account allows the cron job not to depend on credentials of
an individual employee at your company. This is the type of credential that
will be configured when you run "gsutil config -e".

It is important to note that a service account is considered an Editor by
default for the purposes of API access, rather than an Owner. In particular,
the fact that Editors have OWNER access in the default object and
bucket ACLs, but the canned ACL options remove OWNER access from
Editors, can lead to unexpected results. The solution to this problem is to
ensure the service account is an Owner in the Permissions tab for your
project. To find the email address of your service account, visit the
`Google Developers Console <https://cloud.google.com/console#/project>`_,
click on the project you're using, click "APIs & auth", and click
"Credentials".

To create a service account, visit the Google Developers Console and then:

   - Click "APIs & auth" in the left sidebar.

   - Click "Credentials".

   - Click "Create New Client ID".

   - Select "Service Account" as your application type.

   - Save the JSON private key or the .p12 private key and password
     provided.

For further information about account roles, see:
  https://developers.google.com/console/help/#DifferentRoles

For more details about OAuth2 service accounts, see:
  https://developers.google.com/accounts/docs/OAuth2ServiceAccount

Best Answer

Related Solutions

Slow synchronisation stage on gsutil rsync

Using JSON keys with google cloud gsutil

Related Topic