I have strange virtual (docker bridges) networking condition
I have two dockers connected to the same bridge via docker-compose. One docker is "probe" and one is "injector". Injector uses tcpreplay to replay capture and "probe" should receive it via tcpdump. Needless to say the capture replayed does not have any relation to IPs or macs of the NICS attached to the bridge. pinging is working fine between the hosts.
Now there's a third NIC exposed to the host machine automatically by docker.
+--->NIC1 ["injector" docker / uses tcpreplay to inject ]
bridge +--->NIC2 ["probe" docker / uses tcpdump to listen]
|
+--- NIC3 host [used for testing sometimes as injector and sometimes as listener]
Now what actually happens is that when tcpreplay is run from HOST (injects capture via NIC3) everything is working fine, and tcpdump on "probe" shows the replayed traffic. However when tcpreplay is used on injector and injects the capture via NIC1 only first two packets of the capture can be seen on "probe" and then all traffic on "probe" stops (also injecting from host will stop working). if tcpdump is run on NIC3 it's receiving all captured traffic from injector normally.
- ifconfig on "probe" doesn't show any dropped packets
- iptables on host does not increase counters of dropped packets (hopefully i a am doing it correctly "sudo iptables -L -v -n | grep -i drop")
- tcpdump automatically enables promiscuous mode on probe
Does anyone have explanation for this asymmetric behavior? Any idea how to debug it?
Injector and host – AlmaLinux:8, probe -Centos:7
tcpreplay version 4.4.1
Best Answer
The reason that bridge was not passing the frames is because of mac learning mechanism. Let's say we want to inject traffic from MAC1->MAC2 on NIC1 at the injector container, the bridge filter will pass (flood to the probe) the traffic from MAC1 --> MAC2 and learn that MAC1 is located on the NIC1, but then it will refuse to flood everything coming from MAC2-->MAC1 on the same port because it thinks MAC1 is located on the same port the traffic is coming from. Again the traffic that injected was recorded somewhere else and has no any connection to the actual bridge topology.
Disabling the mac learning mechanism on the bridge as explained here resolves the issue and all frames injected from injector will be flooded to probe.
The reason for assymetric behaviour when the traffic is injected from the host is not clear. But I see that bridge is not really activating its learning mechanism when traffic is injected from NIC3, e.g. the macs from injected traffic do not appear on the learned mac list from the beginning for some reason.