Google Compute Engine – Can’t Connect via SSH? VM Loses Network Access?

google-compute-engine

EDIT: This was an out-of-control application process, not GCE. Here's the issue, and answered below:

I just had some kind of outage with my CE VM on a trial account, but I don't see any outage reported on the Google Compute Outage list.

I'm not sure how long it lasted since I'm not sure when it started. From the behavior it matches something that seemed to happen a few weeks ago (losing the ability to log in with SSH over the Compute Engine dashboard until the VM was rebooted).

My test VM disconnected my SSH connection in the last day or so, and when I noticed today I was unable to reconnect. I then tried to connect with SSH using "SSH" connect on the Compute Engine VM list, and that failed. The only thing I could do was get a prompt on the serial console… but I didn't have a password-enabled account at all, I was relying on SSH (now fixed). I had to stop the VM and restart it… then I could connect using the "SSH" connect option on the VM list, although I could NOT connect from outside. I connected to the serial console and saw some network error messages trying to connect to various snaps. I tried to SSH to a remote server from my SSH window into the VM, and initially could not. After a minute or so that worked, and suddenly remote connections worked again.

EDIT: I got a response from my support request from Google. They're saying I experienced a Live Migration event. That doesn't sound right. This was at least 10 minutes of disrupted networking. I could connect to the serial console, and it seemed responsive. It was only after rebooting and the failure of the google management snaps to initialize that it appeared to suddenly start working. Maybe a failure of communication in boot triggered the migration event? I don't know.

EDIT: I removed my worrying about GCE's stability since the infrastructure had nothing to do with the problem.

Best Answer

There may be a number of reasons for this to happen. I would recommend checking the SSH troubleshooting document for more information about how to troubleshoot this issue.

This issue could also occur if the Linux guest environment did not initiate properly after the live migration. The guest environments includes a set of scripts and processes that run contents from a metadata server and creates the proper environment for a virtual machine to run. It might be possible that the SSH keys were not set properly during the guest environment setup.

You may also set the 'automaticRestart' field to 'true' as mentioned in this document. This will automatically restart your instance if it crashes due to a hardware issue or after a live migration. This will ensure that the SSH keys were set up correctly. Feel free to read the live migration documentation if you need further information about live migration in Google Cloud Platform.

Related Solutions

Ssh – Unable to connect Google Compute Engine instance via SSH in browser

It looks like you've added AllowUsers in /etc/ssh/sshd_config configuration file.

To resolve this issue, you'll need to attach the boot disk of your VM instance to a healthy instance as the second disk. Mount it, edit the configuration file and fix the issue.

Here are the steps you can take to resolve the issue:

First of all, take a snapshot of your instance’s disk, in case if a loss or corruption happens you can recover your disk.
In the Developers Console, click on your instance. Uncheck Delete boot disk when instance is deleted and then delete the instance. The boot disk will remain under “Disks”, and now you can attach the disk to another instance. You can also do this step using gcloud command:
```
$ gcloud compute instances delete NAME --keep-disks all
```
Now attach the disk to a healthy instance as an additional disk. You can do this through the Developers Console or using the gcloud command:
```
$ gcloud compute instances attach-disk EXAMPLE-INSTANCE --disk DISK --zone ZONE
```
SSH into your healthy instance.
Determine where the secondary disk lives:
```
$ ls -l /dev/disk/by-id/google-*
```

Mount the disk:

$ sudo mkdir /mnt/tmp
$ sudo mount /dev/disk/by-id/google-persistent-disk-1-part1 /mnt/tmp

Where google-persistent-disk-1 is the name of the disk

Edit sshd_config configuration file and remove AllowUsers line and save it.
```
$ sudo nano /mnt/tmp/etc/ssh/sshd_config
```
Now unmout the disk:
```
$ sudo umount /mnt/tmp
```
Detach it from the VM instance. This can be done through the Developers Console or using the command below:
```
$ gcloud compute instances detach-disk EXAMPLE-INSTANCE --disk DISK
```
Now create a new instance using your fixed boot disk.

Ssh – Unable to SSH Google Cloud Engine instance through gcloud & Putty from Windows 10

OP has resolved this issue by copying his SSH keys to authorized_keys file. This issue can be caused if Google accounts daemon is not running on your VM. This daemon is responsible for syncing ssh keys and user account information between the metadata and VM. For more information on accounts daemon you can visit this link.

Best Answer

Related Solutions

Ssh – Unable to connect Google Compute Engine instance via SSH in browser

Ssh – Unable to SSH Google Cloud Engine instance through gcloud & Putty from Windows 10

Related Topic