TLS just enables encryption on the smtp session and doesn't directly affect whether or not Postfix will be allowed to relay a message.
The relaying denied message occurs because the smtpd_recipient_restrictions rules was not matched. One of those conditions must be fulfilled to allow the message to go through:
smtpd_recipient_restrictions =
permit_sasl_authenticated
check_recipient_access hash:/etc/postfix/filtered_domains
permit_mynetworks
reject_unauth_destination
To explain those rules:
permit_sasl_authenticated
permits authenticated senders through SASL. This will be necessary to authenticate users outside of your network which are normally blocked.
check_recipient_access
This will cause postfix to look in /etc/postfix/filtered_domains for rules based on the recipient address. (Judging by the file name on the file name, it is probably just blocking specific domains... Check to see if gmail.com is listed in there?)
permit_mynetworks
This will permit hosts by IP address that match IP ranges specified in $mynetworks. In the main.cf you posted, $mynetworks was set to 127.0.0.1, so it will only relay emails generated by the server itself.
Based on that configuration, your mail client will need to use SMTP Authentication before being allowed to relay messages. I'm not sure what database SASL is using. That is specified in /usr/lib/sasl2/smtpd.conf Presumably it also uses the same database as your virtual mailboxes, so you should be able enable SMTP authentication in your mail client and be all set.
I am always reluctant to put services on ports that they don't usually live-- not because of a software deficiency, but because of people deficiencies. A sysadmin who inherits this setup has to be pretty good to track down the architecture of "where emails go when they bounce", or else your documentation needs to be pretty clear (and easy to find).
So, to answer your first question-- my suggestion is to have a separate server that handles bounces. This makes things nicely documented via dns, instead of buried in a config file for postfix.
If you choose to ignore that advice, utilizing the transport maps of postfix will allow you to do so. For example, adding this to main.cf:
transport_maps = regexp:/etc/postfix/transport
and using something like this in your transport file:
/bounce.*/ smtp:bounces.mydomain.com:8025
(don't forget to 'postmap /etc/postfix/transport' and 'postfix reload')
Best Answer
No SMTP Server is required on your Windows Web Server.
Your Web App on the Windows Server should use the 'smtp' protocol to send email directly to your Redhat mail server. The smtp functionality should be part of the framework/libraries used by your web application, and may already be implemented as such (and just waiting on you to specify the IP Address for the mail server.
Your Postfix server (on your Redhat Server) should handle all transactions from there onwards (i.e. ISP and the rest of the world, including email back into the office.)
This obviously implies that there is a route for the Web Server to talk to the Mail Server (whether directly or through a firewall/gateway.)
Restricting mail 'clients' in Postfix
If your Web Server is using a Public IP address, or in a DMZ, then you will need to add this server IP to Postfix mynetworks configuration in main.cf.
If your Web Server is using a Private IP address within your firewalled/NAT'd LAN then it may already be factored for in your standard configuration (just confirm the server IP and the above mynetworks option in main.cf