SSH – Handling Remote Host Identification Changed Alert on Hetzner Cloud

SecuritysshUbuntu

I have small cloud server on Hetzner that I power on daily (using Hetzner API) from my home server at 3am and then I login there via SSH, do some work then I shut it down (it's all automatic process)

Everything was fine for months, I didn't touch neither my home server or cloud server yet today I received an email with warning

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is

It was very suspicious since there was no changes to my servers so I logged in and checked /var/log/auth.log :

Dec 10 03:02:19 htznr useradd[1007]: new group: name=ubuntu, GID=1001
Dec 10 03:02:19 htznr useradd[1007]: new user: name=ubuntu, UID=1001, GID=1001, home=/home/ubuntu, shell=/bin/bash
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to group 'adm'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to group 'dialout'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to group 'cdrom'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to group 'floppy'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to group 'sudo'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to group 'audio'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to group 'dip'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to group 'video'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to group 'plugdev'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to group 'lxd'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to group 'netdev'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to shadow group 'adm'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to shadow group 'dialout'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to shadow group 'cdrom'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to shadow group 'floppy'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to shadow group 'sudo'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to shadow group 'audio'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to shadow group 'dip'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to shadow group 'video'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to shadow group 'plugdev'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to shadow group 'lxd'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to shadow group 'netdev'
Dec 10 03:02:19 htznr passwd[1014]: password for 'ubuntu' changed by 'root'
Dec 10 03:02:19 htznr systemd-logind[1057]: New seat seat0.
Dec 10 03:02:19 htznr systemd-logind[1057]: Watching system buttons on /dev/input/event0 (Power Button)
Dec 10 03:02:19 htznr systemd-logind[1057]: Watching system buttons on /dev/input/event1 (AT Translated Set 2 keyboard)
Dec 10 03:02:19 htznr sshd[1094]: Server listening on 0.0.0.0 port 222.
Dec 10 03:02:19 htznr sshd[1094]: Server listening on :: port 222.

03:02:19 is when machine was powered on. As you can see user 'ubuntu' was created with password.

I also realized all keys in /etc/ssh/ has been changed:

-rw-------  1 root root  672 Dec 10 03:02 ssh_host_dsa_key
-rw-r--r--  1 root root  598 Dec 10 03:02 ssh_host_dsa_key.pub
-rw-------  1 root root  227 Dec 10 03:02 ssh_host_ecdsa_key
-rw-r--r--  1 root root  170 Dec 10 03:02 ssh_host_ecdsa_key.pub
-rw-------  1 root root  399 Dec 10 03:02 ssh_host_ed25519_key
-rw-r--r--  1 root root   90 Dec 10 03:02 ssh_host_ed25519_key.pub
-rw-------  1 root root 1.7K Dec 10 03:02 ssh_host_rsa_key
-rw-r--r--  1 root root  390 Dec 10 03:02 ssh_host_rsa_key.pub

What can be the reason? I login to server using Hetzner's IP address.

I'm worried because I know many bots try to login with common usernames like 'centos', 'ubuntu', 'fedora' etc.

I use Ubuntu and have unattended upgrades enabled.

This is new entry in /etc/shadow

ubuntu:!:18971:0:99999:7:::

Best Answer

As comments suggests, the answer is cloud-init package. It's responsible for re-initialization of images of virtual machines, and one of the steps for initialization is re-generation of host's ssh keys.

It's done once 'per instance' based on reported change in instance ID (what is 'instance ID' is highly dependent on cloud provider, or even on ID on "NoCloud" provider for baremetal).

Related Topic