This could be caused from 1 of 3 potential issues. Either A) Google mail is catching the email as spam, B) something on the local machine is hotwiring the emails or C) the SMTP server isn't setup correctly to allow relaying.
In the case of A), it's worth doing in any case. In your TXT record for your DNS zone, make sure to add all of the IP addresses used for outgoing email from your server. Maybe it's 82.80.200.20 and already there if your web server only has 1 IP address, but best to confirm. If you are sending from something not listed as an approved SPF sending server then your messages will be flagged as potential spam. Also, check for any unusual patterns in your emails that gmail may not like (blank subjects, spamming keywords, etc).
For B), confirm that your IIS SMTP settings don't define your domain name as an account. The IIS install should be very straight forward. Basically just set 127.0.0.1 as an allowed relay, everything else should be default for an outgoing smtp server.
For C), it's basically what I mentioned in the previous point, to make sure that you have the relay set to 127.0.0.1, or you're authenticating when sending.
Another test is to send to a non-gmail account and see if it arrives. That will confirm that the smtp server is working correctly.
Additionally, you can do a nslookup test to confirm that there isn't another DNS server hotwiring DNS. You can test it with the following from the command line:
nslookup
set type=mx
yourdomain.com
Best Answer
You can setup your own SMTP server to send email messages from, while using Google Apps in parallel. However, these messages will not be in your "Sent Mail" mail unless you moved using IMAP and replies to these messages will go to the mail servers in your MX records (Google by default).
You can set it up to send using a different sub domain, but that is not a requirement. You can use the same domain. Just call the self hosted SMTP server something different like smtp2.yourdomain.com.
If you are using SPF records [1] for your domain, then make sure to add the SMTP server IP to the list of allowed IP addresses.
[1] http://www.google.com/support/a/bin/answer.py?hl=en&answer=33786