Centos – Can’t SSH into Google Cloud Compute instance using OS Login after `dnf upgrade`

centosgoogle-cloud-platformgoogle-compute-enginesshssh-keys

I have a Google Cloud Platform Compute instance running CentOS 7. It has been working just fine, where I can use it with the web-based SSH interface without any problem. Recently, I switched to "OS Login" and set it up so I can use RSA keys to authenticate for SSH (using my computer's terminal program, rather than using a browser terminal window). This, too, worked well. (For some reason GCP made a second user account on the CentOS system for me to use, but it doesn't really matter to me.)
The other day, I ran sudo dnf upgrade to update the system. The update finished, and I ran sudo shutdown -r now to reboot.

Running uname -a now gives me Linux centos7 3.10.0-1062.12.1.el7.x86_64 #1 SMP Tue Feb 4 23:02:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux.

All of a sudden, I can't SSH into the system using OS Login. (Today I turned OS Login off, and the old method does work.) Here's what I was running into:
– The SSH key I was using didn't work ("Permission denied"). I checked to make sure it was the right one and I know that it didn't expire.
– I couldn't use the GCP browser-based SSH mechanism like I did before. (I'm not sure, but that might be because I turned on OS Login.) If I try to use that, I get a message pop-up that says (one after the other):
"Connecting … Transferring SSH keys to the VM."
"Connecting… Establishing connection to SSH server."
"Could not connect, retrying (1/3)…"
"Timed out connecting to the SSH server."
and
"The VM serial console output may provide details to aid in troubleshooting connection problems. See our help document for other possible causes of this issue." (The "(1/3)" part changes to "(2/3)" and then "(3/3)" after a few seconds.)

The help document link doesn't fix the issue. (https://cloud.google.com/compute/docs/ssh-in-browser#couldnotconnecterror)
– I'm not running a custom OS image.
– I know the disk isn't filled up.
– I haven't been doing anything funny with SSH key file permissions.

The instance did seem to respond to the web interface's start/stop controls, so I restarted it, which didn't change anything.

Looking back at the pop-up message, I checked the serial console output. You can see a copy on pastebin: https://pastebin.com/p8yjGazG
The only thing I can see that might be a problem is a message about not being able to release an IPv6 lease, but then I'm not an expert.

Did the dnf upgrade break something? Am I just missing something obvious? (Is a Google employee playing a prank?)
If anyone could offer some insight into this problem, that would be great!

Best Answer

This is a normal behavior of GCP. You can not have both OS-login enabled and metadata-based SSH key at the same time. please review provided document.[1]

[1] https://cloud.google.com/compute/docs/instances/managing-instance-access

Related Solutions

Ssh – Unable to connect Google Compute Engine instance via SSH in browser

It looks like you've added AllowUsers in /etc/ssh/sshd_config configuration file.

To resolve this issue, you'll need to attach the boot disk of your VM instance to a healthy instance as the second disk. Mount it, edit the configuration file and fix the issue.

Here are the steps you can take to resolve the issue:

First of all, take a snapshot of your instance’s disk, in case if a loss or corruption happens you can recover your disk.
In the Developers Console, click on your instance. Uncheck Delete boot disk when instance is deleted and then delete the instance. The boot disk will remain under “Disks”, and now you can attach the disk to another instance. You can also do this step using gcloud command:
```
$ gcloud compute instances delete NAME --keep-disks all
```
Now attach the disk to a healthy instance as an additional disk. You can do this through the Developers Console or using the gcloud command:
```
$ gcloud compute instances attach-disk EXAMPLE-INSTANCE --disk DISK --zone ZONE
```
SSH into your healthy instance.
Determine where the secondary disk lives:
```
$ ls -l /dev/disk/by-id/google-*
```

Mount the disk:

$ sudo mkdir /mnt/tmp
$ sudo mount /dev/disk/by-id/google-persistent-disk-1-part1 /mnt/tmp

Where google-persistent-disk-1 is the name of the disk

Edit sshd_config configuration file and remove AllowUsers line and save it.
```
$ sudo nano /mnt/tmp/etc/ssh/sshd_config
```
Now unmout the disk:
```
$ sudo umount /mnt/tmp
```
Detach it from the VM instance. This can be done through the Developers Console or using the command below:
```
$ gcloud compute instances detach-disk EXAMPLE-INSTANCE --disk DISK
```
Now create a new instance using your fixed boot disk.

Google Compute – SSH Port 22 Blocked by UFW

By Default, all ports are blocked other than port 22 to allow you to ssh to the VM instance. You also have the option to open port 80 and/or 443. You can see which ports are open by default by checking the Firewall rules within the Cloud Console. I am mentioning this to inform you that you do not need to install ufw in the future.

You will have to provide a startup script to the VM instance to enable SSH. You can just create a simple bash script with either command listed below.

$ufw allow ssh

ufw allow 22

You do not have to use the sudo prefix since the startup script runs as root already.

Best Answer

Related Solutions

Ssh – Unable to connect Google Compute Engine instance via SSH in browser

Google Compute – SSH Port 22 Blocked by UFW

Related Topic