FTPS connection stalled on TLS initialization

You're correct that FileZilla or WinSCP are needed. Basically those FTP settings say that anyone trying to use the normal method over port 21 should be denied, and it doesn't attempt to reestablish a different type of connection. Thus IE and the command line options won't work.

A good test is to install FileZilla locally on the FTP server, or on another server that doesn't have a firewall in-between. Get it working there to prove that it works, and then start looking at your firewall policy to allow it through. Most likely you're running again a firewall rule that blocks the dynamic ports necessary for FTPS.

In FileZilla, use "explicit FTP over TLS" and test using active and passive mode. Active and passive require different firewall ports.

TMG doesn't support FTPS (that's FTP + SSL, rather than SFTP, or SSH File Transfer, which generally works fine)

FTPS is Really Hard to do with an FTP filter, because the FTP NAT filter watches for information the client behind NAT sends to the FTP server, and encryption causes that not to be visible. And that annoys NAT inspectors.

If Total Commander supports using HTTP proxies, you can work around this fairly easily by configuring it to use an HTTPS connection instead (via ye olde CONNECT-to-get-a-plain-TCP-channel hack).

For that to work, you'll also need to configure TunnelPortRanges to allow the destination port as if it were SSL. http://technet.microsoft.com/en-us/library/cc302450.aspx

Alternative alternative!

If (long list here)

• you're using PASV only (so multiple outbound connections)
• your initial connection is on port 21
• you aren't unhappy with the client using an "all protocols" rule

then you could set up

• a protocol definition for TCP outbound, destination port 21, not bound to the FTP filter ("NONFTP FTPS")
• a rule allowing NONFTP FTPS from the client IP to the server IP
  • (I'm assuming it needs to be locked down to only one client and server; otherwise "anywhere" is fine if it's fine with you)
• a rule immediately following that, DENYING FTP (the real FTP) with the same source/dest
• a rule following that one that allows all outbound traffic between source and dest IP (or at least, all TCP connections on expected ports)

And this set of rules should be ordered before any other rules allowing the client full access to all protocols and ports.

It's important the DENY FTP rule is

a) immediately after the Special FTP rule that uses the protocol with the application filter unbound, and

b) before any other rules that might allow the client to use Good Old FTP to the target IP (otherwise TMG prefers the protocol definition with the application filter, which is what we don't want here - we want TMG to treat it like straight TCP)

I think this would cover all the FTPS scenarios. Same principle as this.