The passphrase that can be set on the private key is unrelated to the SSH server or the connection to it. Setting a passphrase to the private key is merely a security measure the key owner may take in order to prevent access to his remote shell by a third party in case the private key gets stolen.
Unfortunately, you cannot force users to secure their private keys with passphrases. Sometimes, unprotected private keys are required in order to automate access to the remote SSH server. One good habit I highly recommend for such cases is to advise the users to hash the known_hosts file (stored at ~/.ssh/known_hosts), which keeps information about the remote hosts the user connects to, using the following command:
ssh-keygen -H -f ~/.ssh/known_hosts
This way, even if a third party gained access to an unprotected private key, it would be extremely difficult to find out for which remote hosts this key is valid. Of course, clearing the shell history is mandatory for this technique to be of any value.
Also, another thing you should always bear in mind, is not allow root to login remotely by adding the following in your SSH server's configuration (sshd_config):
PermitRootLogin no
On the other hand, if you want to prevent users from using keys to authenticate, but instead use passwords, you should add the following to your sshd_config:
PasswordAuthentication yes
PubkeyAuthentication no
Don't use a password. Generate a passphrase-less SSH key and push it to your VM.
If you already have an SSH key, you can skip this step…
Just hit Enter for the key and both passphrases:
$ ssh-keygen -t rsa -b 2048
Generating public/private rsa key pair.
Enter file in which to save the key (/home/username/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/username/.ssh/id_rsa.
Your public key has been saved in /home/username/.ssh/id_rsa.pub.
Copy your keys to the target server:
$ ssh-copy-id id@server
id@server's password:
Now try logging into the machine, with ssh 'id@server', and check-in:
.ssh/authorized_keys
Note: If you don't have .ssh dir and authorized_keys file, you need to create it first
to make sure we haven’t added extra keys that you weren’t expecting.
Finally, check to log in…
$ ssh id@server
id@server:~$
You may also want to look into using ssh-agent if you want to try keeping your keys protected with a passphrase.
Best Answer
The only possible benefit is that you know the "attacking" IP is a "bad guy" or compromised machine, and probably don't want to talk to them anyway. It's likely they'll try other protocols. If you have none open, nothing to worry about.
It might reduce bandwidth slightly. It would definitely reduce the spam in your logs (I change my SSH port to 2222 for this reason; but don't recommend that tactic unless you have a small group of admins accessing the box).
It's technically possible that they could guess a SSH Key, but wholly unrealistic to think it will ever happen. I would recommend changing your SSH Keys every few years (to ensure you're using "current" technology, and to verify documentation surrounding the system).