Linux – How to set up internal routing between virtual interfaces and real ethernet interface (Linux)

interfaceiptableslinuxroutingvirtualization

I have a Linux box with one real (as opposes to virtual, a.k.a. aliased) Ethernet interface I can use (eth0 is used for other purposes – can't use it, nor can I add more NICs).
Say it's eth1

I need to control some objects/entities via SNMP, so I set up a virtual Ethernet interface for each object, with its appropriate MAC address.
I do this by (example for vif1):

ip -family inet link add link eth1 name vif1 address <the MAC addr> type macvlan
ip link set vif1 up multicast on
ip route del default dev vif1 table main /* enable the pings/TFTP going out! */
ip route add default via 192.168.1.1 table main proto static metric /* restore orig */

eth1,vif1,vif2,… all get an IP addresses from a single (remote) DHCP server.
All these IP addresses are, of course, on the same IP subnet, say 10.11.1.0/24

Problem: ping from Linux box to the DHCP server (say 10.11.1.1) machine works.
ping from the DHCP server machine to eth1 IP or any vif#X IP works, BUT (the issue, I suppose…)
only eth1 responds to the ICMP packets (verified by ifconfig counters and by wireshark sniffing)
This problem causes the inability to connect to the SNMP agents associated with vif interfaces' IP addresses.

I'm guessing that I need to set up internal routing so that IP packets reach their destination vif#X. I've tried adding an ip rule, with a new ip routing table, but probably didn't set it (the new table) correctly…
Any one can tell me how (and preferably also why) to do this?

The Linux box runs Ubuntu9.04 and the DHCP-server runs Windows XP SP3

Best Answer

Solved it, finally: It's an ARP related issue.

  1. The DHCP server assigns an IP address to the virtual interface MAC address and sets this pair in the server's local ARP table
  2. The Linux box ties the new IP address to the virtual interface that requested it.
  3. PINGs are working on both ways:
  4. When pinging from Linux to server, it goes out through the real interface (which is on the same IP subnet)
  5. When pinging from server to Linux, again the real interface responds so it seems as if everything is OK...

BUT

When the server sends IP packets (in my case, SNMP messages) it uses the virtual interface's MAC address. When it reaches to the Linux box, the kernel just discards these frames, since it doesn't know how to forward them; Running Wireshark displays these messages since usually the interface is put in promiscuous mode

In order to make the SNMP messages reach the SNMP agent which is bind to the virtual interface, the IP packet must have the real interface's MAC address (I think that only then the kernel does VLAN routing, based on the IP address...)

The way to achieve this, is to send a gratuitous ARP request from the Linux' box real interface to the server, stating that the newly assigned IP address (to one of the virtual interfaces...) is "owned" by the real interface's MAC address. This updates the server's ARP table correctly.

BTW, this also explains why waiting some time before starting the SNMP traffic works: the server's ARP table entry is aged out, so the server sends an ARP-request which is responded correctly by the real interface

Related Topic