Attempting to manually test creating an exact backup for a GCS bucket via Data Transfer Service in the GCP browser UI and getting error…
Couldn't read source object ACLs. Source bucket must not have storage.uniformBucketLevelAccess enabled and the service account must have storage.objects.getIAMPolicy on the source object.
Yet, my source bucket does not have uniform Access Control and I do have storage.objects.getIAMPolicy permissions on the project. I'm not very experienced with GCP, so IDK what else could be going wrong here.
Does anyone have any thoughts on what could be going wrong here or any debugging things to try?
My DTS job configs look like this:
My source bucket configs look like…
My destination bucket configs look like…
And my user IAM permissions look like…
(IDK why there are no "excess permissions" added for my Storage Object Admin role (I think has something do do with this), but storage.objects.getIamPolicy was indeed part of the permissions diff list when I set that role.)
Tried looking at the docs here (https://cloud.google.com/storage-transfer/docs/source-cloud-storage#user_permissions), but IDK that these docs are accurate / up to date.
If you're using the Google Cloud console to create your transfer, and have the permissions listed in User permissions.
Looking at the User Permissions section…
The Storage Legacy Bucket Owner role (roles/storage.legacyBucketOwner) or the Storage Admin role (roles/storage.admin) provide the required permissions.
My personal user already has the Storage Admin role (the Storage Legacy Bucket role appear to not exist when I attempt to edit my user permissions) — there is a Storage Transfer Admin role, but the policy change simulator shows this would not add any new permissions.
Following the instructions re. granting the required permissions to the service agent here to ID the DTS service agent principal email (https://cloud.google.com/storage-transfer/docs/source-cloud-storage#find), I followed along and found that the service agent principal was already added to the DTS source bucket with Storage Legacy Bucket Reader and Storage Object Viewer roles. So, unfortunately, it appears there was no issue with the automatic granting yet the DTS job does not work.
(And there is nothing in the Logs Explorer re. that bucket or GCS other than the logged actions of me creating it — ie. as if it's not even registering the failed DTS job runs I manually triggered).
Does anyone have any thoughts on what could be going wrong here or any debugging things to try?
Best Answer
Got it working.
Google has extensive usage of Service Agents: https://cloud.google.com/iam/docs/service-agents
I was informed that something may have gone wrong with the automatic grant creation: https://cloud.google.com/storage-transfer/docs/source-cloud-storage#auto-granting_permissions_in_the
This is how I configured everything to get the DTS job working:
Source bucket Permissions configs for the STS agent (whoes full email/name I ID'ed by following the instructions here):
(The instructions on exactly which perms to grant the STS agent for a bucket are not clear, but I chose the ones that made sense for a bucket that is being considered the "source" bucket and that seemed to work).
Destination bucket Permissions configs for the STS service agent:
(The instructions on exactly which perms to grant the STS service agent for a bucket are not clear, but I chose the ones that made sense for a bucket that is being considered the "destination" bucket and that seemed to work). EDIT*: that first permission for the destination bucket should actually be
Storage Legacy Bucket Writer, but it doesn't really matter in this basic solution because of theStorage Object Adminpermission that I have (which I explain later).Note that the
storage.objects.getIAMPolicypermission is granted by theStorage Object Adminrole; I also could have created a custom role that just included this specific permission, rather than give the STS agent that broader pre-existing role, but IDK that it is a big deal (though, for the sake of minimizing unneeded access, perhaps I should still do this (as having the admin role actually makes the other roles redundant, as confirmed by the Policy Test change simulation)).This role is required, even though not specified in the instructions here because I guess it assumes you are using the Uniform ACL policy, but the source bucket I'm testing with (like the one I actually want to do the DTS from) uses Fine-Grained ACLs, this is why I think the
storage.objects.getIAMPolicyperm is required to run the STS without error.Note that this also means that when you are doing the DTS to the destination bucket, you have to also have the ACL settings be set to Fine-Grained for that as well.