CentOS7 KVM – Troubleshooting Bridge, VLAN, and Bonding Issues

I'm running into an issue where my KVM guests have no network traffic in- or outgoing. After a lot of hours debugging I'm pretty sure I'm missing something essential.

What I have a server with 2 10 Gbit interfaces, bonded in failover. Relevant configs and outputs:

Primary interface:

# Autogenerated by /usr/sbin/iface-genconf on 2020-01-09 13:08:26
# backup file moved into /etc/sysconfig/network-scripts/.ens1f0-1578575306

DEVICE=ens1f0
ONBOOT=yes
TYPE=Ethernet
MASTER=bond0
SLAVE='yes'
BOOTPROTO=none
MTU=9000
NM_CONTROLLED=no

Secondary interface:

# Autogenerated by /usr/sbin/iface-genconf on 2020-01-09 13:08:26
# backup file moved into /etc/sysconfig/network-scripts/.ens1f1-1578575306

DEVICE=ens1f1
ONBOOT=yes
TYPE=Ethernet
MASTER=bond0
SLAVE='yes'
BOOTPROTO=none
MTU=9000
NM_CONTROLLED=no

Bond status:

Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: ens1f0
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0
ARP Polling Interval (ms): 500
ARP IP target/s (n.n.n.n form): 10.29.55.102, 10.29.55.103, 10.29.55.104

Slave Interface: dummy0
MII Status: down
Speed: Unknown
Duplex: Unknown
Link Failure Count: 1
Permanent HW addr: 32:15:f6:14:3b:18
Slave queue ID: 0

Slave Interface: ens1f0
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 0c:c4:7a:bd:3d:cc
Slave queue ID: 0

Slave Interface: ens1f1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 0c:c4:7a:bd:3d:cd
Slave queue ID: 0

On top of this bond are running 2 VLAN interfaces, one for management traffic, VLAN 302 and one for public traffic, VLAN 101.

Config VLAN 302:

DEVICE=bond0.302
ONBOOT=yes
TYPE=Ethernet
BOOTPROTO=none
VLAN=yes
USERCTL=no
BRIDGE=manbr0

Config VLAN 101:

DEVICE=bond0.101
ONBOOT=yes
TYPE=Ethernet
BOOTPROTO=none
VLAN=yes
USERCTL=no
BRIDGE=cloudbr0

As you can see each VLAN interface is connected to a bridge. manbr0 for management, cloudbr0 for public traffic.

Config manbr0:

DEVICE=manbr0
BOOTPROTO=none
ONBOOT=yes
TYPE=Bridge
USERCTL=no
NAME=manbr0
IPADDR=10.29.55.106
PREFIX=24
GATEWAY=10.29.55.1

Config cloudbr0:

DEVICE=cloudbr0
BOOTPROTO=none
ONBOOT=yes
TYPE=Bridge
USERCTL=no
DELAY=0
MTU=1500```

brctl show output:
```[root@hv2 ~]# brctl show
bridge name     bridge id               STP enabled     interfaces
manbr0             8000.da7b71f50c30       no              bond0.302
cloudbr0                8000.da7b71f50c30       no              bond0.101

So far so good. Management network is working like a charm. If I configure a public IP on the cloudbr0 all works as expected. Configured interface:

cloudbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
        inet 185.x.x.222  netmask 255.255.255.0  broadcast 185.x.x.255
        inet6 fe80::d87b:71ff:fef5:c30  prefixlen 64  scopeid 0x20<link>
        ether da:7b:71:f5:0c:30  txqueuelen 1000  (Ethernet)
        RX packets 37448  bytes 1751235 (1.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8  bytes 656 (656.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Ping from remote host:

PING 185.X.X.222 (185.X.X.222) 56(84) bytes of data.

--- 185.X.X.222 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9004ms
rtt min/avg/max/mdev = 0.119/0.211/0.793/0.195 ms

So far all looks fine. I'm now deploying a VM with that same IP, after removing the IP from the cloudbr0 of course:

cloudbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
        inet6 fe80::d87b:71ff:fef5:c30  prefixlen 64  scopeid 0x20<link>
        ether da:7b:71:f5:0c:30  txqueuelen 1000  (Ethernet)
        RX packets 80714  bytes 3802772 (3.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 22  bytes 1916 (1.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

So now I have this VM running at the same IP I was testing with before. But it's not reachable. When I start a ping from within the VM to a remote box and a run a tcpdump on the remote server, I see the the requests coming in:

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bond0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:53:24.074319 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:53:25.075022 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:53:25.465094 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 12035, seq 0, length 64
09:53:25.465146 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 12035, seq 1, length 64
09:53:26.074392 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 12035, seq 2, length 64
09:53:27.074432 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 12035, seq 3, length 64
09:53:28.074464 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 12035, seq 4, length 64
09:53:29.074502 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 12035, seq 5, length 64
09:53:30.074540 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 12035, seq 6, length 64
09:53:31.074610 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 12035, seq 7, length 64
09:53:32.074651 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 12035, seq 8, length 64
09:53:33.074688 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 12035, seq 9, length 64
09:53:34.074730 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:53:35.074841 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:53:36.078219 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:53:38.074966 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:53:39.078221 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:53:40.081571 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:53:42.075139 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:53:43.078240 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:53:44.081570 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:53:46.075282 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:53:47.078253 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:53:48.081594 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:53:50.075420 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:53:51.078295 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:53:52.081614 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:53:54.075748 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:53:55.078285 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:53:55.465183 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 12035, seq 30, length 64
09:53:55.465223 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 12035, seq 31, length 64
09:53:56.075960 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 12035, seq 32, length 64
09:53:57.076013 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 12035, seq 33, length 64
09:53:58.076086 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 12035, seq 34, length 64
09:53:59.076122 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 12035, seq 35, length 64
09:54:00.076168 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 12035, seq 36, length 64
09:54:01.076178 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 12035, seq 37, length 64
09:54:02.076215 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 12035, seq 38, length 64
09:54:03.076286 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 12035, seq 39, length 64
09:54:04.076339 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:54:05.078322 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:54:06.081631 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:54:08.076504 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:54:09.078345 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:54:10.081670 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:54:12.076686 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:54:13.078363 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:54:14.081681 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:54:16.076836 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:54:17.078402 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:54:18.081705 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:54:20.077154 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:54:21.078355 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:54:22.081734 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:54:24.077303 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46
09:54:25.078430 ARP, Request who-has 185.X.X.1 tell 185.X.X.222, length 46

This snippet above is a pattern, notice the seq in the ICMP requsts. So it's able to send out 10 ICMP requests, then 20 are failing. 10 requests are send out, 20 are failing. And so on and so on.

When tcpdumping on the cloudbr0 bridge interface I see the same pattern, 10 replies, missing 20, 10 replies, missing 20, again, notice the seq:

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on cloudbr0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:57:55.464992 IP 185.X.X.25 > 185.X.X.222: ICMP echo reply, id 12035, seq 270, length 64
09:57:55.465005 IP 185.X.X.25 > 185.X.X.222: ICMP echo reply, id 12035, seq 271, length 64
09:57:56.086069 IP 185.X.X.25 > 185.X.X.222: ICMP echo reply, id 12035, seq 272, length 64
09:57:57.086122 IP 185.X.X.25 > 185.X.X.222: ICMP echo reply, id 12035, seq 273, length 64
09:57:58.086181 IP 185.X.X.25 > 185.X.X.222: ICMP echo reply, id 12035, seq 274, length 64
09:57:59.086215 IP 185.X.X.25 > 185.X.X.222: ICMP echo reply, id 12035, seq 275, length 64
09:58:00.086272 IP 185.X.X.25 > 185.X.X.222: ICMP echo reply, id 12035, seq 276, length 64
09:58:01.086238 IP 185.X.X.25 > 185.X.X.222: ICMP echo reply, id 12035, seq 277, length 64
09:58:02.086288 IP 185.X.X.25 > 185.X.X.222: ICMP echo reply, id 12035, seq 278, length 64
09:58:03.086362 IP 185.X.X.25 > 185.X.X.222: ICMP echo reply, id 12035, seq 279, length 64
09:58:25.465128 IP 185.X.X.25 > 185.X.X.222: ICMP echo reply, id 12035, seq 300, length 64
09:58:25.465138 IP 185.X.X.25 > 185.X.X.222: ICMP echo reply, id 12035, seq 301, length 64
09:58:26.087363 IP 185.X.X.25 > 185.X.X.222: ICMP echo reply, id 12035, seq 302, length 64
09:58:27.087421 IP 185.X.X.25 > 185.X.X.222: ICMP echo reply, id 12035, seq 303, length 64
09:58:28.087463 IP 185.X.X.25 > 185.X.X.222: ICMP echo reply, id 12035, seq 304, length 64
09:58:29.087485 IP 185.X.X.25 > 185.X.X.222: ICMP echo reply, id 12035, seq 305, length 64
09:58:30.087799 IP 185.X.X.25 > 185.X.X.222: ICMP echo reply, id 12035, seq 306, length 64
09:58:31.087577 IP 185.X.X.25 > 185.X.X.222: ICMP echo reply, id 12035, seq 307, length 64
09:58:32.087616 IP 185.X.X.25 > 185.X.X.222: ICMP echo reply, id 12035, seq 308, length 64
09:58:33.087610 IP 185.X.X.25 > 185.X.X.222: ICMP echo reply, id 12035, seq 309, length 64

When I run tcpdump on the virtual interface I see the same pattern, 10 requests can be send out, 20 can't. Rinse and repeat. Furthermore I never see a reply coming back onto the virtual interface:

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on one-79-0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:12:25.494959 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 14339, seq 12, length 64
11:12:25.494970 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 14339, seq 13, length 64
11:12:25.600742 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 14339, seq 14, length 64
11:12:26.600781 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 14339, seq 15, length 64
11:12:27.600832 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 14339, seq 16, length 64
11:12:28.600875 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 14339, seq 17, length 64
11:12:29.600914 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 14339, seq 18, length 64
11:12:30.600950 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 14339, seq 19, length 64
11:12:31.600986 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 14339, seq 20, length 64
11:12:32.601023 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 14339, seq 21, length 64
11:12:55.494978 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 14339, seq 42, length 64
11:12:55.494987 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 14339, seq 43, length 64
11:12:55.601972 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 14339, seq 44, length 64
11:12:56.602010 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 14339, seq 45, length 64
11:12:57.602051 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 14339, seq 46, length 64
11:12:58.602091 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 14339, seq 47, length 64
11:12:59.602128 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 14339, seq 48, length 64
11:13:00.602163 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 14339, seq 49, length 64
11:13:01.602200 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 14339, seq 50, length 64
11:13:02.602241 IP 185.X.X.222 > 185.X.X.25: ICMP echo request, id 14339, seq 51, length 64

My virtual interface "one-74-0" is connected to the bridge:

[root@hv2 ~]# brctl show
bridge name     bridge id               STP enabled     interfaces
manbr0             8000.da7b71f50c30       no              bond0.302
cloudbr0                8000.da7b71f50c30       no              bond0.101
                                                        one-74-0

What's interesting is that there are quite a few dropped packets on that virtual interface, on the KVM host:

one-74-0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
        inet6 fe80::fc00:b9ff:fe64:83de  prefixlen 64  scopeid 0x20<link>
        ether fe:00:b9:64:83:de  txqueuelen 1000  (Ethernet)
        RX packets 441  bytes 27054 (26.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1196384  bytes 288740710 (275.3 MiB)
        TX errors 0  dropped 23902 overruns 0  carrier 0  collisions 0

iptables output:

[root@hv2 ~]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

ebtables output:

[root@hv2 ~]# ebtables -L
Bridge table: filter

Bridge chain: INPUT, entries: 0, policy: ACCEPT

Bridge chain: FORWARD, entries: 0, policy: ACCEPT

Bridge chain: OUTPUT, entries: 0, policy: ACCEPT

ip_forward:

[root@hv2 ~]# cat /proc/sys/net/ipv4/ip_forward
1

The qemu process:

[root@hv2 ~]# ps -ef | grep qemu
oneadmin  9198     1  6 10:06 ?        00:02:27 /usr/libexec/qemu-kvm -name guest=one-76,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-4-one-76/master-key.aes -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid e1660604-09ce-452a-a8d9-5cfe8d7a88a9 -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=26,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive file=/var/lib/one//datastores/103/76/disk.0,format=raw,if=none,id=drive-ide0-0-0,readonly=on,cache=none,discard=unmap,aio=native -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,write-cache=on -drive file=/var/lib/one//datastores/103/76/disk.1,format=raw,if=none,id=drive-ide0-0-1,readonly=on -device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -netdev tap,fd=29,id=hostnet0,vhost=on,vhostfd=30 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=02:00:b9:64:83:de,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,fd=31,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -vnc 0.0.0.0:76 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on

virsh XML of guest:

<domain type='kvm'>
  <name>one-76</name>
  <uuid>e1660604-09ce-452a-a8d9-5cfe8d7a88a9</uuid>
  <title>sysrescuecd-76</title>
  <metadata>
    <one:vm xmlns:one="http://opennebula.org/xmlns/libvirt/1.0">
      <one:system_datastore><![CDATA[/var/lib/one//datastores/103/76]]></one:system_datastore>
      <one:name><![CDATA[sysrescuecd-76]]></one:name>
      <one:uname><![CDATA[oneadmin]]></one:uname>
      <one:uid>0</one:uid>
      <one:gname><![CDATA[oneadmin]]></one:gname>
      <one:gid>0</one:gid>
      <one:opennebula_version>5.10.1</one:opennebula_version>
      <one:stime>1578992757</one:stime>
      <one:deployment_time>1578992777</one:deployment_time>
    </one:vm>
  </metadata>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <cputune>
    <shares>1024</shares>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.6.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>
      <source file='/var/lib/one//datastores/103/76/disk.0'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/one//datastores/103/76/disk.1'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <alias name='ide0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='piix3-uhci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='02:00:b9:64:83:de'/>
      <source bridge='cloudbr0'/>
      <target dev='one-76-0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-4-one-76/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <graphics type='vnc' port='5976' autoport='no' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+9869:+9869</label>
    <imagelabel>+9869:+9869</imagelabel>
  </seclabel>
</domain>

A couple of more interesting things:

• tcpdumping within the VM while trying to ping outside the VM renders no packets whatsoever – nothing is coming in from the IP I'm pinging towards. The replies do end up on the bond, not in the VM.
• tcpdumping within the VM and not limiting the src IP to the host I'm pinging does show other broadcast traffic within this VLAN.
• A very interesting one. When I run manually "ifconfig cloudbr0 down; ifconfig cloudbr0 up" sometimes, like 1 in 100 times, 1 ping aimed towards the VM's IP comes through.

I have the feeling something is wrong between the bridge and the VM NIC. As said – when I configure the same IP directly on the bridge, all is fine. So on the hardware/network side of things all seems to be working.
Any idea what my next best step is in debugging this one?