I currently have a set of four virtual hosts on a private network, with a fifth host running pfSense to act as the gateway.
When communicating with hosts outside of the private network, all traffic is routed via the pfSense gateway (10.0.0.1). However, when two hosts on the same subnet communicate, they do not do so via the gateway. I have gathered that these hosts realise they are on the same subnet and switch and opt to communicate via MAC address w/ARP rather than using the IP gateway I have set up.
I would like these hosts to communicate via the gateway so that I can monitor the traffic between the hosts on the private network. I have attempted to set static routes, for example:
ip route add 10.0.0.4 via 10.0.0.1 dev eth0
The first traceroute subsequent to this will correctly route via the gateway:
root@file-server ~ # traceroute 10.0.0.4
traceroute to 10.0.0.4 (10.0.0.4), 30 hops max, 60 byte packets
1 10.0.0.1 (10.0.0.1) 0.241ms 0.287ms 0.274ms
2 10.0.0.4 (10.0.0.4) 0.559ms 0.567ms 0.574ms
If I then perform the traceroute again however, the route stops using the gateway.
root@file-server ~ # traceroute 10.0.0.4
1 10.0.0.4 (10.0.0.4) 0.345ms 0.376ms 0.362ms
My guess would be that perhaps my host has populated its ARP table with the MAC address of the traceroute target after the initial traceroute, and opts to use that for communication in future.
If that is the case, I am not sure how to counter it. I believed that my static route would take precedent over any other decisions taken by the system. I would like to research further, but I am not actually sure where to look to solve this issue as I am not sure why my static route is not adhered to.
Can someone please provide insight as to why my static route is only adhered to with my initial traceroute, and how I would go about forcing a static route to the gateway for same-subnet communications?
**Edit: ** I should add that this set-up is for experimentation, so I am not restricted to solely things that would be a good idea in the real world.
Best Answer
You would have to capture packets to be sure, but I suspect what is going on is an ICMP Redirect.
If a Router receives a packet for which a better next-hop exists, it can send an ICMP Redirect to the original sender to inform them to use the better next-hop when trying to speak to that same destination.
The initial sender will update its routing table temporarily with the new information. I believe for 10 minutes, but this is probably different with each OS/vendor/implementation.
There is generally a way to disable ICMP redirects at the Router. And there is more than likely a way to disable accepting ICMP redirects on the Host. If not specifically, then indirectly by blocking it with an ACL or iptables entry.
You can read more about ICMP Redirects here: http://www.cymru.com/gillsr/documents/icmp-redirects-are-bad.htm
The are now-a-days considered a security vulnerability, but when they were created they were based on good intentions. As the Internet got bigger, and less trust-able, however, their use has been significantly reduced -- and discouraged, for that matter.