My understanding of FTP over SSL (ftps) is that it doesn't work well with firewalls and NAT. In an ordinary FTP session, the information about data connections is read, and for NAT modified, by the firewall in order for the firewall to dynamically open the needed ports. If that information is secured by SSL, the firewall can't read it or change it.
Using SFTP, or scp, makes the network administrator's job a lot easier - everything happens on the server's port 22, and the transaction follows the normal client/server model.
One thing not mentioned is whether or not your firewall is performing NAT and whether or not it is static NAT or dynamic NAT. If your client machine has a static address or is being statically NATed, you may not need to make any firewall changes, assuming you allow all outbound traffic and the server operates only in Passive mode (PASV).
To know exactly what ports you will need to open, you will need to either:
a) talk to the vendor to get specifics about how their system has been configured.
b) Use a protocol analyzer, such as tcpdump or wireshark, to look at the traffic, both from outside your firewall and inside your firewall
You need to find out which port is the Control Connection. You list 3, which seems odd to me. Assuming the server only works in PASV (passive) mode, you need to figure out how the server is configured to allocated DATA ports. Have they locked down the DATA channel to a single inbound port? Have they locked down the DATA channel to a small range or ports?
With these answers, you can start configuring your firewall.
Best Answer
Which version of OS X Server? Lion?
Are you sure it's "Server" & not just an OS X client you're using as a server?
If it is indeed OS X Server, make sure you're using Server Admin to manage the Firewall, not System Preferences -> Security & Privacy. The latter controls the OS X Application Firewall, which it sounds like you were using. The former controls the lower level BSD PF firewall, which is much better & actually acclaimed by most as the best firewall out there. You can also use pfctl from the command line to manage the PF firewall if you want.
Edit
Sorry, some of that info above is wrong. Apparently Server Admin still only manages ipfw, which is an older BSD firewall. Still good, but not as good as PF.