Windows – Server address requested and the certificate subject do no match

Well, first of all don't use port 443, that's the default SSL port. If you use a different port for RDP (your explanation doesn't ring true with me - I've never seen an ISP that filters port 3389, but I suppose anything's possible), then you should at least use a port that's not also in use by a common protocol or service.

Second of all, TMG lets you configure a Remote Access VPN. This is what you probably should be doing instead of, or in addition to setting up a Remote Desktop Gateway.

Third, sounds like you'd benefit from hiring an experienced Windows SA to set this up for you, even if only on a contract or consultant basis.

It turns out that much of the configuration data for RDSH is stored in the Win32_TSGeneralSetting class in WMI in the root\cimv2\TerminalServices namespace. The configured certificate for a given connection is referenced by the Thumbprint value of that certificate on a property called SSLCertificateSHA1Hash.

UPDATE: Here's a generalized Powershell solution that grabs and sets the thumbprint of the first SSL cert in the computer's personal store. If your system has multiple certs, you should add a -Filter option to the gci command to make sure you reference the correct cert. I've left my original answer intact below this for reference.

# get a reference to the config instance
$tsgs = gwmi -class "Win32_TSGeneralSetting" -Namespace root\cimv2\terminalservices -Filter "TerminalName='RDP-tcp'"

# grab the thumbprint of the first SSL cert in the computer store
$thumb = (gci -path cert:/LocalMachine/My | select -first 1).Thumbprint

# set the new thumbprint value
swmi -path $tsgs.__path -argument @{SSLCertificateSHA1Hash="$thumb"}

In order to get the thumbprint value

1. Open the properties dialog for your certificate and select the Details tab
2. Scroll down to the Thumbprint field and copy the space delimited hex string into something like Notepad
3. Remove all the spaces from the string. You'll also want to watch out for and remove a non-ascii character that sometimes gets copied just before the first character in the string. It's not visible in Notepad.
4. This is the value you need to set in WMI. It should look something like this: 1ea1fd5b25b8c327be2c4e4852263efdb4d16af4.

Now that you have the thumbprint value, here's a one-liner you can use to set the value using wmic:

wmic /namespace:\\root\cimv2\TerminalServices PATH Win32_TSGeneralSetting Set SSLCertificateSHA1Hash="THUMBPRINT"

Or if PowerShell is your thing, you can use this instead:

$path = (Get-WmiObject -class "Win32_TSGeneralSetting" -Namespace root\cimv2\terminalservices -Filter "TerminalName='RDP-tcp'").__path
Set-WmiInstance -Path $path -argument @{SSLCertificateSHA1Hash="THUMBPRINT"}

Note: the certificate must be in the 'Personal' Certificate Store for the Computer account.