Don't use a password. Generate a passphrase-less SSH key and push it to your VM.
If you already have an SSH key, you can skip this step…
Just hit Enter for the key and both passphrases:
$ ssh-keygen -t rsa -b 2048
Generating public/private rsa key pair.
Enter file in which to save the key (/home/username/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/username/.ssh/id_rsa.
Your public key has been saved in /home/username/.ssh/id_rsa.pub.
Copy your keys to the target server:
$ ssh-copy-id id@server
id@server's password:
Now try logging into the machine, with ssh 'id@server', and check-in:
.ssh/authorized_keys
Note: If you don't have .ssh dir and authorized_keys file, you need to create it first
to make sure we haven’t added extra keys that you weren’t expecting.
Finally, check to log in…
$ ssh id@server
id@server:~$
You may also want to look into using ssh-agent if you want to try keeping your keys protected with a passphrase.
SSH private keys should never leave the machine they were generated on, unless to be moved into a more secure platform, such as an HSM. Likewise, they should never be accessible to anyone other than the person who generated them and will be using them, e.g. an administrator's workstation, or better, an HSM/smartcard. If you're going to be accessing a server from multiple machines, generate a unique key on each machine and deploy the public keys accordingly, or put a public key from a smart card or other strong authenticator into the authorized keys.
For starters, I always recommend protecting private keys with a passphrase. This will mitigate the chances of unauthorized key usage if it falls into the wrong hands. A higher security measure is to embed the private key into a smartcard, and make use of the cryptographic functions on it via a middleware. Redhat makes it easy to do all this.
How do you protect your private ssh keys? I read about ssh-agent but I
am not sure how to use it or if it can be used in this situation.
ssh-agent is used in a terminal session, to cache the unencrypted version of the private key, making it unnecessary for you to enter the key's passphrase each time it is used. You can also use an external authenticator by which you make use of an authentication challenge/response, but never have access to the cleartext. Please take a peek at http://www.gooze.eu/howto/using-openssh-with-smartcards/using-ssh-authentication-agent-ssh-add-with-smartcards in order to review how you can use ssh with a secure authenticator.
If a sysadmin leaves and he copies all the private ssh keys, then he
has access to all the servers. How do you deal with this situation?
This is a very strong argument as to why private keys should never be shared, and certainly why multiple users' keys should never be stored together. Plain and simple, there is no reason to do so. No one else has any business getting your private key. That said, even if you were in a situation where someone got their hands on a bunch of active private keys, if they were properly passphrase-protected by a strong passphrase, it would largely be a non-issue.
Again, by credentialing your users with a strong authenticator having cryptographic functions, you can avoid this situation altogether. When a machine needs to perform cryptographic functions, rather than store a private key on the filesystem, you can store it on an HSM to prevent exfiltration (theft) of the private keys.
SSH keys private keys can be a component of multi-factor authentication system, such as a smart card, or another type of hardware security module (USB form factor, PCIe card, fortezza, networked HSM, etc) This is pretty standard now for security conscious businesses, governments and the military operate simply because one compromise can lead to massive exposure; passwords are too weak.
A smart card, depending on the security of it, can be either (A) unvalidated, a (B) FIPS 140-2 level 2 device, all the way up to a (C) controlled cryptographic device usable for Top Secret SCI information and Suite B algorithms. In the case of (B), a government authorized lab has independently validated the cryptography, and (C), the NSA has evaluated the device in question. B is considered equivalent to best commercial grade security, and C is considered military / intelligence community grade. Prices go up for the robust stuff!
Best Answer
It is not possible to "use" a private key without being able to read - and thus copy - that private key. So the idea that your admins can login to the jumpbox, then go onwards using a private key that they cannot directly access, is not going to fly.
But you can arrange things so that the private keys can only be used from the jumpbox (see
man sshd,from=in AUTHORIZED_KEYS FILE FORMAT). Then, denying a given admin access to the jumpbox leaves him or her with keys that are of no further use.Does that satisfy your security constraints?
Note that you write as if all remote admins were accessing a shared account on the jumpbox. If it were my system, I wouldn't do that; I'd have each remote admin have an individual account on the jumpbox, all such accounts having access to - or copies of - the relevant captive onward keypair. That makes revoking an individual admin's access to the jumpbox much easier and more reliable.