For Git Bash

If you are running msysgit (I am assuming you are) and are looking to run Git Bash (I recommend it over TortoiseGit, but I lean to the CLI more than GUI now), you need to figure out what your home directory is for Git Bash by starting it then type pwd (On Windows 7, it will be something like C:\Users\phsr I think). While you're in Git Bash, you should mkdir .ssh.

After you have the home directory, and a .ssh folder under that, you want to open PuTTYgen and open the key (.ppk file) you have previously created. Once your key is open, you want to select Conversions -> Export OpenSSH key and save it to HOME\.ssh\id_rsa. After you have the key at that location, Git Bash will recognize the key and use it.

Note: Comments indicate that this doesn't work in all cases. You may need to copy the OpenSSH key to Program Files\Git\.ssh\id_rsa (or Program Files (x86)\Git\.ssh\id_rsa).

For TortoiseGit

When using TortoiseGit, you need to set the SSH key via pacey's directions. You need to do that for every repository you are using TortoiseGit with.

It turns out that much of the configuration data for RDSH is stored in the Win32_TSGeneralSetting class in WMI in the root\cimv2\TerminalServices namespace. The configured certificate for a given connection is referenced by the Thumbprint value of that certificate on a property called SSLCertificateSHA1Hash.

UPDATE: Here's a generalized Powershell solution that grabs and sets the thumbprint of the first SSL cert in the computer's personal store. If your system has multiple certs, you should add a -Filter option to the gci command to make sure you reference the correct cert. I've left my original answer intact below this for reference.

# get a reference to the config instance
$tsgs = gwmi -class "Win32_TSGeneralSetting" -Namespace root\cimv2\terminalservices -Filter "TerminalName='RDP-tcp'"

# grab the thumbprint of the first SSL cert in the computer store
$thumb = (gci -path cert:/LocalMachine/My | select -first 1).Thumbprint

# set the new thumbprint value
swmi -path $tsgs.__path -argument @{SSLCertificateSHA1Hash="$thumb"}

In order to get the thumbprint value

1. Open the properties dialog for your certificate and select the Details tab
2. Scroll down to the Thumbprint field and copy the space delimited hex string into something like Notepad
3. Remove all the spaces from the string. You'll also want to watch out for and remove a non-ascii character that sometimes gets copied just before the first character in the string. It's not visible in Notepad.
4. This is the value you need to set in WMI. It should look something like this: 1ea1fd5b25b8c327be2c4e4852263efdb4d16af4.

Now that you have the thumbprint value, here's a one-liner you can use to set the value using wmic:

wmic /namespace:\\root\cimv2\TerminalServices PATH Win32_TSGeneralSetting Set SSLCertificateSHA1Hash="THUMBPRINT"

Or if PowerShell is your thing, you can use this instead:

$path = (Get-WmiObject -class "Win32_TSGeneralSetting" -Namespace root\cimv2\terminalservices -Filter "TerminalName='RDP-tcp'").__path
Set-WmiInstance -Path $path -argument @{SSLCertificateSHA1Hash="THUMBPRINT"}

Note: the certificate must be in the 'Personal' Certificate Store for the Computer account.