Yes, this is one of the problems with passwordless SSH keys. If you store a private key on server A that allows you to connect to server B, gaining access to server A is effectively gaining access to server B. (Not the reverse is not true - gaining access to server B wouldn't result in an immediate compromise of server A, assuming you didn't also have SSH keys set up to allow passwordless logins in that direction.

There are a few things you can do to mitigate this:

• If the process doesn't need to be fully automated, add a password to your SSH keys. (This probably won't work for you, since you note it's for a backup)
• For passwordless logins under your own account to multiple machines, I recommend creating a passworded key for each machine you physically type on, and use an SSH agent to store the key in-memory while you use it. Agent forwarding should allow you to "hop" from host to host without creating keys on each remote host.
• For automated, passwordless SSH keys, I recommend restricting the commands the key can run. In your authorized_keys file, prefix your each key with:
  command="<allowed command line here>",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty

8-9 years ago I worked in an shared user environment with hundreds of local users, and SSH key-based logins were disabled because there was no way to enforce password policies on the keys. So long as you're controlling the scenario fully, nowadays SSH keys are definitely better than just using passwords.

For Git Bash

If you are running msysgit (I am assuming you are) and are looking to run Git Bash (I recommend it over TortoiseGit, but I lean to the CLI more than GUI now), you need to figure out what your home directory is for Git Bash by starting it then type pwd (On Windows 7, it will be something like C:\Users\phsr I think). While you're in Git Bash, you should mkdir .ssh.

After you have the home directory, and a .ssh folder under that, you want to open PuTTYgen and open the key (.ppk file) you have previously created. Once your key is open, you want to select Conversions -> Export OpenSSH key and save it to HOME\.ssh\id_rsa. After you have the key at that location, Git Bash will recognize the key and use it.

Note: Comments indicate that this doesn't work in all cases. You may need to copy the OpenSSH key to Program Files\Git\.ssh\id_rsa (or Program Files (x86)\Git\.ssh\id_rsa).

For TortoiseGit

When using TortoiseGit, you need to set the SSH key via pacey's directions. You need to do that for every repository you are using TortoiseGit with.