GCE project-wide SSH keys not propogating

google-cloud-platformgoogle-compute-enginessh-keysubuntu-14.04

I have noticed in my GCE cluster that when I add an SSH key to the project metadata that it is then automatically added to /home/[user]/.ssh/authorized_keys. This is the behaviour I expect (and desire for the task I am doing at the moment). However, one of my machines in the cluster is not having it's authorized_keys file updated at all, whether I remove or add keys to the project metadata. I have even tried removing the authorized_keys file completely to see if it will be recreated (it isn't).

All the servers are up-to-date Ubuntu 14.04, and as far as I can tell it's only this one instance that has the issue. It is the 'oldest' instance in the cluster, so when reading https://cloud.google.com/compute/docs/instances/adding-removing-ssh-keys#addkey I wondered if it was subject to the following statement:

If your instance does not support the newer metadata values, add or remove the older instance-only sshKeys value that applies your key only to a specific instance and also blocks all project-wide keys when the value is set.

But I have removed instance only sshKeys and the problem persists. Any ideas before I destroy & recreate the image to make sure I am able to use the 'new' metadata?

Best Answer

First, make sure that your VM instance can communicate with metadata server and can read the project-wide sshKeys value. To verify this, connect to your VM and run the following command:

curl http://metadata.google.internal/computeMetadata/v1/project/attributes/sshKeys -H "Metadata-Flavor: Google"

The output of the command should be SSH keys that you've added to the metadata of your project. If the command can not communicate with metadata, that means your internal firewall blocks the traffic between your VM and metadata server.

If the communication is successful and you can read the value of sshKeys, then you should verify that account manager daemon is running. In Ubuntu 14.04, the daemon script is located at:

/usr/share/google/google_daemon/manage_accounts.py

Use the following command to verify its running status:

sudo ps aux | grep manage_accounts.py

Related Solutions

Ssh – Why does Google recommend removing SSH keys from GCE for security

The critical detail is that the page you've referenced is about creating a new Compute Engine machine image. Specifically, when you create a new virtual machine image, you want to ensure it does NOT include any host keys. That way, when the image is cloned and reconstituted into an actual VM, the sshd startup script will recognize that there are no host keys, and automatically generate new ones. This is desirable because having multiple machines using the same host key is a very bad idea.

So, in the general case, please do not go deleting your host keys, but if you are creating a new image, it's an important step in order to ensure there's a one-to-one relationship between host keys and machines.

Redhat – cloud-init does not insert instance level ssh keys in gce

Huh, can't get any GCE answer it seems. Figured it out though. First of all current Red Hat Enterprise Linux (v7.2) cloud-init version does not support instance ssh keys (it handles only project level keys). cloud-init trunk though does support them already so hopefully downstream will pick up soon. In the meantime I used the following user-data to emulate this (again YAML representation): metadata:

 items:
 - key: sshKeys
   value: root:ssh-rsa AAAAB... non@nan 
 - key: user-data
   value:|
     #cloud-config
     disable_root: false
     preserve_hostname: true
     runcmd:
     - "curl 'http://metadata.google.internal/computeMetadata/v1/instance/attributes/sshKeys' -H 'Metadata-Flavor: Google' | sed -r -e 's/(^|,)[^\\S]*:/\\1/g' -e 's/,/\\n/g' >> /root/.ssh/authorized_keys"

Note that google documentation talks only about startup-script metadata key. To my reading even upstream cloud-init does not care about that metadata key. It is looking for the user-data key as shown above.

Hope this helps.

Best Answer

Related Solutions

Ssh – Why does Google recommend removing SSH keys from GCE for security

Redhat – cloud-init does not insert instance level ssh keys in gce

Related Topic