VMware ESXi – Protecting Root Account from Lockout

firewallvmware-esxi

I have a VMWare ESXi instance, running version 6.0.0. Our staff was locked out of the ESXi thick client (the "vSphere Client" Windows application) for a substantial amount of time today. We were presented with an "incorrect username or password" error message when attempting to log in. After some research we determined that we were being locked out of our own ESXi host due to v6.0's root lockout feature which locks down the account for a set amount of time (default: 2 minutes) after 3 consecutive failed password attempts. It seems that the attacker continued for some hours until finally relenting. At that point we were able to log in ourselves using the root account.

We are a bit confused about why this could happen. The server is hosted in a fairly large and reputable data centre and is a true dedicated instance. However, said facility wants to charge rather excessive rates to put this VM server behind a hardware firewall. So we have been relying on the built-in firewall of ESXi.

On the Configuration -> Security Profile -> Firewall section, we have the following services (which are defined by default) to be IP restricted, to only allow our office IP:

SSH Server
vSphere Web Access
vSphere Web Client
vsanvp
vMotion

Despite this, it seems the attacker is still able to at least get through and somehow trigger a 'wrong password' error because the server's ESXi event log shows numerous lines like this:

Remote access for ESXi local user account 'root' has been locked for 120 seconds after 563 failed login attempts.

This is despite the fact that only our office IP is authorized, and we know that nobody here is initiating this.

What are we doing wrong?

Best Answer

a) You shouldn't be using the .net/Windows client, it goes away entirely with 6.5 which is imminent and VMware have been strongly urging users to move away from it for literally years now.

b) I'm unclear, are you all logging into the host directly, i.e. without a vCenter and if so are you logging in as root?

c) It appears you've not put the host into strict lockdown mode - I'd disable SSH too, as a service and in the firewall.

Related Solutions

Permission denied for root to ESXi host

Are your VMs set to startup on Power on? If yes, you should be able to reboot the physical server to bring the virtual hosts back up.

This would be found on the configuration tab for each ESXi host in the Vsphere console under "Software> Virtual Machine Startup/Shutdown". You'd then click on properties in the top right corner and select "allow virtual machines to start and stop automatically with the system".

Security – how to secure an vmware esxi host on a root server

so c33s, i've got a answer for you after researching this topic myself :-)

i'll state 4 steps for ESXi 4 (should work on 5 as well). on ESXi 5 VMware included a packet filter (you should find many on this topic on google).

let's get started: first you need SSH access to your ESXi. here are the 4 steps to improve the situation on a root server a bit:

1) remove ESXi welcome screen: a request to https://your-esxi/ shows a page telling you how to get started with ESXi. Nobody needs to now this exept you. So read it and after that rename the file:

mv /usr/lib/vmware/hostd/docroot/index.html /usr/lib/vmware/hostd/docroot/index.html.bak

2) allow only auth_key login on ssh generate an SSH auth_key for your admin machine (i call it "YOUR-SSH-RSA" further in text). check your setup by running this code on your ESXi

mkdir /.ssh
chmod 0600 -R /.ssh
echo "YOUR-SSH-RSA" >> /.ssh/authorized_keys

Check if logging in works. If so, you can put these lines into rc.local using

vi /etc/rc.local

This is needed because ESXi forgets about this after a reboot. To disable password login do the following:

vi /etc/inetd.conf

Add parameter -s to following lines:

ssh stream tcp nowait root /sbin/dropbearmulti dropbear ++min=0,swap,group=shell -i -K60
ssh stream tcp6 nowait root /sbin/dropbearmulti dropbear ++min=0,swap,group=shell -i -K60

Should look like this afterwards:

ssh stream tcp nowait root /sbin/dropbearmulti dropbear ++min=0,swap,group=shell -s -i -K60
ssh stream tcp6 nowait root /sbin/dropbearmulti dropbear ++min=0,swap,group=shell -s -i -K60

reboot (or atleast restarting inetd) is needed to take effect.

3) Change SSH default port

vi /etc/services

search for

ssh             22/tcp                         # SSH Remote Login Protocol
ssh             22/udp                         # SSH Remote Login Protocol

change the port 22 to what ever you like. look out for conflicts with other ports!

reboot (or atleast restarting inetd) is needed to take effect.

4) Change routing Because ESXi 4 has no packet filter we need to change the routing so it doesn't know how to talk to everyone. this is a bit dangerous, because wrong routing could lead to an ESXi-Management only reachable from local console! You need a static IP or known Network from which you want to administer your ESXi. We add a route to this and delete the default route afterwards.

esxcfg-route -a x.x.x.x/sub y.y.y.y

where "x.x.x.x" is your network or ip. "sub" the subnet mask. and "y.y.y.y" your gateway.

for example we have a ESXi with default gateway 12.34.56.78 and want to add only a single ip 98.76.54.21 the command is

esxcfg-route -a 98.76.54.21/32 12.34.56.78

check if your route is correctly set:

esxcfg-route -l

if so, delete your default route

esxcfg-route -d default y.y.y.y

When all was done correct, you should still reach your ESXi. If not you have to log in localy and change everything back.

Best Answer

Related Solutions

Permission denied for root to ESXi host

Security – how to secure an vmware esxi host on a root server

Related Topic