Linux – IP address reuse on macvlan devices

linuxlxc

I'm trying to create easy to use and possibly simple testing environment for some product and got some strange behaviour of macvlan's.

What I'm trying to achieve: make a toolset for one-line start/stop of lxc containers(via docker) bound to external ip(I have enough of it on host machine).

So, I'm doing something like this:

docker run -d -name=container_name container_image
pipework eth1 container_name ip/prefix_len@gateway

and pipework here does this:

GUEST_IFNAME=ph$NSPID$eth1

ip link add link eth1 dev $GUEST_IFNAME type macvlan mode bridge
ip link set eth1 up
ip link set $GUEST_IFNAME netns $NSPID

ip netns exec $NSPID ip link set $GUEST_IFNAME name eth1
ip netns exec $NSPID ip addr add $IPADDR dev eth1
ip netns exec $NSPID ip route delete default 
ip netns exec $NSPID ip link set eth1 up
ip netns exec $NSPID ip route replace default via $GATEWAY

ip netns exec $NSPID arping -c 1 -A -I eth1 $IPADDR

And it works for first time per IP. But for second time and later packets for containers IP isn't getting into container, while all configuration seem fine.

So it looks like this:

External machine

➤ ping 212.76.131.212
....silence....

Host machine

root@ubuntu:~# ip link show eth1
2: eth1:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:15:17:c9:e1:c9 brd ff:ff:ff:ff:ff:ff

root@ubuntu:~# ip addr show eth1
2: eth1:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:15:17:c9:e1:c9 brd ff:ff:ff:ff:ff:ff

root@ubuntu:~# tcpdump -v -i eth1 icmp
tcpdump: WARNING: eth1: no IPv4 address assigned
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
00:00:46.542042 IP (tos 0x0, ttl 60, id 9623, offset 0, flags [DF], proto ICMP (1), length 84)
    5.134.221.98 > 212.76.131.212: ICMP echo request, id 6718, seq 2345, length 64
00:00:47.549969 IP (tos 0x0, ttl 60, id 9624, offset 0, flags [DF], proto ICMP (1), length 84)
    5.134.221.98 > 212.76.131.212: ICMP echo request, id 6718, seq 2346, length 64
00:00:48.558143 IP (tos 0x0, ttl 60, id 9625, offset 0, flags [DF], proto ICMP (1), length 84)
    5.134.221.98 > 212.76.131.212: ICMP echo request, id 6718, seq 2347, length 64
00:00:49.566319 IP (tos 0x0, ttl 60, id 9626, offset 0, flags [DF], proto ICMP (1), length 84)
    5.134.221.98 > 212.76.131.212: ICMP echo request, id 6718, seq 2348, length 64
00:00:50.573999 IP (tos 0x0, ttl 60, id 9627, offset 0, flags [DF], proto ICMP (1), length 84)
    5.134.221.98 > 212.76.131.212: ICMP echo request, id 6718, seq 2349, length 64
^C
5 packets captured
5 packets received by filter
0 packets dropped by kernel
1 packet dropped by interface

Host machine, netns of container

root@ubuntu:~# ip netns exec 32053 ip link show eth1
48: eth1@if2:  mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether b2:12:f7:cc:a1:9d brd ff:ff:ff:ff:ff:ff
root@ubuntu:~# ip netns exec 32053 ip addr show eth1
48: eth1@if2:  mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether b2:12:f7:cc:a1:9d brd ff:ff:ff:ff:ff:ff
    inet 212.76.131.212/29 scope global eth1
    inet6 fe80::b012:f7ff:fecc:a19d/64 scope link 
       valid_lft forever preferred_lft forever
root@ubuntu:~# ip netns exec 32053 tcpdump -v -i eth1 icmp
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
....silence....
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel

So, can anyone say, what can it be? Can this be caused by not a bug in macvlan implementation? Is there any tools I can use to debug that configuration?

Best Answer

This was an ARP cache issue.

ARP cache on gateway was holding entry for IP with MAC of already dead macvlan and was not updating by just arping.

Solved by adding

ip netns exec $NSPID ping -c 1 -I eth1 $GATEWAY

to the end of pipework script. This ping forces gateway to update ARP cache entry/