This could be caused from 1 of 3 potential issues. Either A) Google mail is catching the email as spam, B) something on the local machine is hotwiring the emails or C) the SMTP server isn't setup correctly to allow relaying.
In the case of A), it's worth doing in any case. In your TXT record for your DNS zone, make sure to add all of the IP addresses used for outgoing email from your server. Maybe it's 82.80.200.20 and already there if your web server only has 1 IP address, but best to confirm. If you are sending from something not listed as an approved SPF sending server then your messages will be flagged as potential spam. Also, check for any unusual patterns in your emails that gmail may not like (blank subjects, spamming keywords, etc).
For B), confirm that your IIS SMTP settings don't define your domain name as an account. The IIS install should be very straight forward. Basically just set 127.0.0.1 as an allowed relay, everything else should be default for an outgoing smtp server.
For C), it's basically what I mentioned in the previous point, to make sure that you have the relay set to 127.0.0.1, or you're authenticating when sending.
Another test is to send to a non-gmail account and see if it arrives. That will confirm that the smtp server is working correctly.
Additionally, you can do a nslookup test to confirm that there isn't another DNS server hotwiring DNS. You can test it with the following from the command line:
nslookup
set type=mx
yourdomain.com
Please see my question here. It's for Exchange, but what you mainly want to make sure of is your MX record, SPF record and reverse DNS to be identified as legit mail. Most of it still fits, as those are steps any mail server needs to take.
Is this SMTP server going to be used for internal or external sending?
Best Answer
You could read up a bit on SMTP security on Windows Server 2003 with the following Microsoft KB Article 324285 (sorry, you'll have to find that one on your own; Server Fault only allows new users to post one URL per post). That article shows that you can grant or deny access to the SMTP virtual server based on IP addresses. This could be yet another alternative way of restricting mail flow to only those few places that you want to allow.