Linux – Circumstances Ignoring Route Table for MAC Address Selection

While debugging a problem with the networking configuration for a set of virtual machines running under KVM, I discovered a circumstance where the kernel in the guest VM has decided to stamp the outgoing Ethernet frame with a destination address that is in conflict with the address it would choose if it was respecting with the kernel IP route table.

So, in that example, I expected the outbound frame to be delivered to de:ad:be:3b:24:48 which corresponds to the host that owns the IP address 10.11.11.2 and that owns a route to 10.8.0.0/24.

What actually happened is that the kernel decided to stamp the frame with a destination of 00:10:db:ff:70:01 which sent the frame in the direction of 10.11.11.1 which doesn't know how to route to 10.8.0.0/24 and as a result the packet was dropped.

This decision was in contravention of the local routing table that clearly specified that the route to 10.8.0.0/24 was via 10.11.11.2. See the original problem report for details.

[The incorrect destination address was visible by running tcpdump in the guest VM that was sending the frame in the incorrect direction.].

In fact, by kludging the local arp table to make 10.11.11.1's apparent MAC address to be identical to 10.11.11.2's actual MAC address, I was able to get the frame flowing in the correct direction.

So my question is: what mechanism, in either the guest VM or the KVM host, could cause the guest kernel to ignore the local route table and send the packet in a frame towards the (incorrect) host at 10.11.11.1, even though 10.11.11.1 wasn't listed as a gateway for the destination network (10.8.0.0/24)?

Note: iptables was disabled in the guest at the time. I don't know if ebtables is enabled in the KVM host, but even if it was, how would this cause the kernel in the guest VM to want send the packet in the direction of 10.11.11.1?

One behaviour I did notice is that if I purge the ARP tables of the affected host and send a ping request to the affected host from the 10.8.0.0/24 network, it receives the request, then sends an arp broadcast for 10.11.11.1 immediately prior to sending the ping response in the direction of 10.11.11.1 instead of 10.11.11.2 and hence 10.8.0.0/24. What is causing it to try 10.11.11.1 which is not specified as a gateway?