I've recently hit a similar issue, albeit a slightly different. I wanted to route only TCP port 25 (SMTP) over an OpenVPN tap0 interface, while routing all other traffic (even for the same host) over the default interface.
To do so, I had to mark packets and set up rules for handling it. First, add a rule that make the kernel route packets marked with 2 through table 3 (explained later):
ip rule add fwmark 2 table 3
You could have added a symbolic name to /etc/iproute2/rt_tables, but I did not bother to do so. The number 2 and 3 are randomly chosen. In fact, these can be the same but for clarity I did not do it in this example (although I do it in my own setup).
Add a route for redirecting traffic over a different interface, assuming the gateway being 10.0.0.1:
ip route add default via 10.0.0.1 table 3
Very important! Flush your routing cache, otherwise you will not get a response back and sit with your hands in your hair for some hours:
ip route flush cache
Now, set a firewall rule for marking designated packets:
iptables -t mangle -A OUTPUT -p tcp --dport 465 -j MARK --set-mark 2
The above rule applies only if the packets come from the local machine. See http://inai.de/images/nf-packet-flow.png. Adjust it to your requirements. For instance, if you only want to route outgoing HTTP traffic over the tap0 interface, change 465 to 80.
To prevent the packets sent over tap0 getting your LAN address as source IP, use the following rule to change it to your interface address (assuming 10.0.0.2 as IP address for interface tap0):
iptables -t nat -A POSTROUTING -o tap0 -j SNAT --to-source 10.0.0.2
Finally, relax the reverse path source validation. Some suggest you to set it to 0, but 2 seems a better choice according to https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt. If you skip this, you will receive packets (this can be confirmed using tcpdump -i tap0 -n), but packets do not get accepted. The command to change the setting so packets get accepted:
sysctl -w net.ipv4.conf.tap0.rp_filter=2
Best Answer
The iptables -m owner trick can only track packets against a user that are sent out bound (by definition). It can't be used to track packets received for that user.
I admit, off the top of my head, I don't see any nice way of doing this. At a stab in the dark, it's going to involve applying a patch at the kernel level to, for example, only allow specific users to bind to particular port ranges on the network stack (similar to the idea that only root can bind to network sockets on port 1024 and lower). Then you could apply iptables traffic logging on these port ranges and know for sure that any traffic is for, and only for, the corresponding user who is allowed to bind to those ports. Downside is, this is going to cause havoc for user applications who are not aware of these limitations, when they then decide to try and bind to a port and the kernel says no.
It may also be possible to do this with SE Linux, but I suspect has the potential to become a sys-admin maintenance nightmare: http://www.linuxquestions.org/questions/linux-server-73/how-can-i-restrict-ports-for-users-to-bind-to-667153/