Docker – Route Docker container traffic through another container

I'm trying to route tcp traffic of container B (10.10.1.2, custom binary using mysql and running on port 4242) through container A (10.10.1.3, haproxy, setup in transparent proxy mode) to the outside world, they share the same user defined docker network 10.10.1.0/24 in bridge mode

Container B doesn't expose ports, only container A.

On container B:

# ip route show
default via 10.10.1.1 dev eth0
10.10.1.0/24 dev eth0 proto kernel scope link src 10.10.1.2
# ip route replace default via 10.10.1.3
# ip route show
default via 10.10.1.3 dev eth0
10.10.1.0/24 dev eth0 proto kernel scope link src 10.10.1.2

On container A (haproxy):

sysctl -w net.ipv4.ip_forward=1
sysctl -w net.ipv4.ip_nonlocal_bind=1
iptables -t mangle -N DIVERT
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 1
iptables -t mangle -A DIVERT -j ACCEPT
ip rule add fwmark 1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100

net.ipv4.ip_forward is set to 1 everywhere, Docker run with default options.

But when I change the route; because it must answer back to container A; container B can't reach anything, a simple ping google.com timeout.

I'm not a specialist but according to the tcpdump commands I have ran on the 3 parts, there is an exchange between them. Maybe some paquets are not passing through ?

I'm I missing nothing? it should be pretty simple but I can't get I working I don't understand why.

Thank you

EDIT:

tcpdump on haproxy container when trying to ping/curl google.com

15:12:46.576882 IP (tos 0x0, ttl 64, id 24297, offset 0, flags [DF], proto UDP (17), length 56)
    myproject_containerb_1.myproject_mynetwork.52246 > cdns.ovh.net.53: [bad udp cksum 0x025f -> 0x68e7!] 46224+ A? google.com. (28)
15:12:46.576945 IP (tos 0x0, ttl 63, id 24297, offset 0, flags [DF], proto UDP (17), length 56)
    myproject_containerb_1.myproject_mynetwork.52246 > cdns.ovh.net.53: [bad udp cksum 0x025f -> 0x68e7!] 46224+ A? google.com. (28)
15:12:46.578608 IP (tos 0x0, ttl 64, id 1536, offset 0, flags [DF], proto UDP (17), length 72)
    28768a341dc6.35740 > cdns.ovh.net.53: [bad udp cksum 0x0270 -> 0xa31b!] 56294+ PTR? 99.33.186.213.in-addr.arpa. (44)
15:12:46.578846 IP (tos 0x0, ttl 64, id 24298, offset 0, flags [DF], proto UDP (17), length 56)
    myproject_containerb_1.myproject_mynetwork.54291 > cdns.ovh.net.53: [bad udp cksum 0x025f -> 0xa70e!] 28241+ AAAA? google.com. (28)
15:12:46.578854 IP (tos 0x0, ttl 63, id 24298, offset 0, flags [DF], proto UDP (17), length 56)
    myproject_containerb_1.myproject_mynetwork.54291 > cdns.ovh.net.53: [bad udp cksum 0x025f -> 0xa70e!] 28241+ AAAA? google.com. (28)
15:12:46.580544 IP (tos 0x0, ttl 56, id 23868, offset 0, flags [none], proto UDP (17), length 98)
    cdns.ovh.net.53 > 28768a341dc6.35740: [udp sum ok] 56294 q: PTR? 99.33.186.213.in-addr.arpa. 1/0/0 99.33.186.213.in-addr.arpa. PTR cdns.ovh.net. (70)
15:12:48.559050 IP (tos 0x0, ttl 64, id 19298, offset 0, flags [DF], proto TCP (6), length 60)
15:12:48.797205 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 28768a341dc6 tell myproject_containerb_1.myproject_mynetwork, length 28
15:12:48.797226 ARP, Ethernet (len 6), IPv4 (len 4), Reply 28768a341dc6 is-at 02:42:0a:0a:01:03 (oui Unknown), length 28
15:12:51.580010 IP (tos 0x0, ttl 64, id 24376, offset 0, flags [DF], proto UDP (17), length 56)
    myproject_containerb_1.myproject_mynetwork.35939 > cdns.ovh.net.53: [bad udp cksum 0x025f -> 0xeebe!] 28241+ AAAA? google.com. (28)
15:12:51.580178 IP (tos 0x0, ttl 63, id 24376, offset 0, flags [DF], proto UDP (17), length 56)
    myproject_containerb_1.myproject_mynetwork.35939 > cdns.ovh.net.53: [bad udp cksum 0x025f -> 0xeebe!] 28241+ AAAA? google.com. (28)
15:12:51.580799 IP (tos 0x0, ttl 64, id 24377, offset 0, flags [DF], proto UDP (17), length 56)
    myproject_containerb_1.myproject_mynetwork.51485 > cdns.ovh.net.53: [bad udp cksum 0x025f -> 0x6be0!] 46224+ A? google.com. (28)
15:12:51.580820 IP (tos 0x0, ttl 63, id 24377, offset 0, flags [DF], proto UDP (17), length 56)
    myproject_containerb_1.myproject_mynetwork.51485 > cdns.ovh.net.53: [bad udp cksum 0x025f -> 0x6be0!] 46224+ A? google.com. (28)
15:12:51.613200 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.10.1.1 tell 28768a341dc6, length 28
15:12:51.613286 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 28768a341dc6 tell 10.10.1.1, length 28
15:12:51.613300 ARP, Ethernet (len 6), IPv4 (len 4), Reply 28768a341dc6 is-at 02:42:0a:0a:01:03 (oui Unknown), length 28
15:12:51.613319 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.10.1.1 is-at 02:42:38:6d:07:86 (oui Unknown), length 28
15:12:51.613928 IP (tos 0x0, ttl 64, id 2532, offset 0, flags [DF], proto UDP (17), length 68)
    28768a341dc6.58522 > cdns.ovh.net.53: [bad udp cksum 0x026c -> 0x5a50!] 17457+ PTR? 1.1.10.10.in-addr.arpa. (40)
15:12:51.615887 IP (tos 0x0, ttl 56, id 844, offset 0, flags [none], proto UDP (17), length 127)
    cdns.ovh.net.53 > 28768a341dc6.58522: [udp sum ok] 17457 NXDomain* q: PTR? 1.1.10.10.in-addr.arpa. 0/1/0 ns: 10.in-addr.arpa. SOA localhost. nobody.invalid. 1 3600 1200 604800 10800 (99)
15:12:56.584820 IP (tos 0x0, ttl 64, id 24781, offset 0, flags [DF], proto UDP (17), length 71)
    myproject_containerb_1.myproject_mynetwork.60514 > cdns.ovh.net.53: [bad udp cksum 0x026e -> 0xdfaf!] 13663+ A? google.com.openstacklocal. (43)
15:12:56.584847 IP (tos 0x0, ttl 63, id 24781, offset 0, flags [DF], proto UDP (17), length 71)
    myproject_containerb_1.myproject_mynetwork.60514 > cdns.ovh.net.53: [bad udp cksum 0x026e -> 0xdfaf!] 13663+ A? google.com.openstacklocal. (43)
15:12:56.587511 IP (tos 0x0, ttl 64, id 24782, offset 0, flags [DF], proto UDP (17), length 71)
    myproject_containerb_1.myproject_mynetwork.36461 > cdns.ovh.net.53: [bad udp cksum 0x026e -> 0x2d86!] 10878+ AAAA? google.com.openstacklocal. (43)
15:12:56.587522 IP (tos 0x0, ttl 63, id 24782, offset 0, flags [DF], proto UDP (17), length 71)
    myproject_containerb_1.myproject_mynetwork.36461 > cdns.ovh.net.53: [bad udp cksum 0x026e -> 0x2d86!] 10878+ AAAA? google.com.openstacklocal. (43)
15:12:58.560867 IP (tos 0x0, ttl 64, id 12123, offset 0, flags [DF], proto TCP (6), length 60)
15:12:59.549284 IP (tos 0xc0, ttl 64, id 40262, offset 0, flags [none], proto ICMP (1), length 94)
    28768a341dc6 > myproject_containerb_1.myproject_mynetwork: ICMP redirect ns3012985.ip-149-202-74.eu to host 10.10.1.1, length 74
    IP (tos 0x0, ttl 63, id 61603, offset 0, flags [DF], proto TCP (6), length 66)
15:12:59.551076 IP (tos 0x0, ttl 64, id 3564, offset 0, flags [DF], proto UDP (17), length 73)
    28768a341dc6.34089 > cdns.ovh.net.53: [bad udp cksum 0x0271 -> 0xe0da!] 3332+ PTR? 155.74.202.149.in-addr.arpa. (45)
15:12:59.637667 IP (tos 0x0, ttl 56, id 1829, offset 0, flags [none], proto UDP (17), length 113)
    cdns.ovh.net.53 > 28768a341dc6.34089: [udp sum ok] 3332 q: PTR? 155.74.202.149.in-addr.arpa. 1/0/0 155.74.202.149.in-addr.arpa. PTR ns3012985.ip-149-202-74.eu. (85)
15:13:01.591313 IP (tos 0x0, ttl 64, id 24864, offset 0, flags [DF], proto UDP (17), length 71)
    myproject_containerb_1.myproject_mynetwork.51194 > cdns.ovh.net.53: [bad udp cksum 0x026e -> 0x0418!] 13663+ A? google.com.openstacklocal. (43)
15:13:01.591352 IP (tos 0x0, ttl 63, id 24864, offset 0, flags [DF], proto UDP (17), length 71)
    myproject_containerb_1.myproject_mynetwork.51194 > cdns.ovh.net.53: [bad udp cksum 0x026e -> 0x0418!] 13663+ A? google.com.openstacklocal. (43)
15:13:01.592629 IP (tos 0x0, ttl 64, id 24865, offset 0, flags [DF], proto UDP (17), length 71)
    myproject_containerb_1.myproject_mynetwork.42581 > cdns.ovh.net.53: [bad udp cksum 0x026e -> 0x159e!] 10878+ AAAA? google.com.openstacklocal. (43)
15:13:01.592638 IP (tos 0x0, ttl 63, id 24865, offset 0, flags [DF], proto UDP (17), length 71)
    myproject_containerb_1.myproject_mynetwork.42581 > cdns.ovh.net.53: [bad udp cksum 0x026e -> 0x159e!] 10878+ AAAA? google.com.openstacklocal. (43)
15:13:03.555973 IP (tos 0x0, ttl 64, id 21583, offset 0, flags [DF], proto TCP (6), length 60)
    28768a341dc6.34260 > myproject_containerb_1.myproject_mynetwork.8081: Flags [S], cksum 0x1647 (incorrect -> 0xeed8), seq 2021736358, win 29200, options [mss 1460,sackOK,TS val 126381033 ecr 0,nop,wscale 7], length 0

EDIT 2:

Another logs from haproxy container when pinging 8.8.8.8

17:07:20.714651 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 1, length 64
17:07:20.714683 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 1, length 64
17:07:20.715559 IP 6eb4896e0f04.53502 > cdns.ovh.net.53: 49544+ PTR? 8.8.8.8.in-addr.arpa. (38)
17:07:20.717433 IP cdns.ovh.net.53 > 6eb4896e0f04.53502: 49544 1/0/0 PTR google-public-dns-a.google.com. (82)
17:07:20.718841 IP 6eb4896e0f04.46346 > cdns.ovh.net.53: 61016+ PTR? 99.33.186.213.in-addr.arpa. (44)
17:07:20.720686 IP cdns.ovh.net.53 > 6eb4896e0f04.46346: 61016 1/0/0 PTR cdns.ovh.net. (70)
17:07:21.725413 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 2, length 64
17:07:21.725573 IP 6eb4896e0f04 > myproject_containerb_1.myproject_mynetwork: ICMP redirect google-public-dns-a.google.com to host 10.10.1.1, length 92
17:07:21.725579 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 2, length 64
17:07:21.727624 IP 6eb4896e0f04.45201 > cdns.ovh.net.53: 21323+ PTR? 1.1.10.10.in-addr.arpa. (40)
17:07:21.729570 IP cdns.ovh.net.53 > 6eb4896e0f04.45201: 21323 NXDomain* 0/1/0 (99)
17:07:22.726499 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 3, length 64
17:07:22.726527 IP 6eb4896e0f04 > myproject_containerb_1.myproject_mynetwork: ICMP redirect google-public-dns-a.google.com to host 10.10.1.1, length 92
17:07:22.726530 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 3, length 64
17:07:23.741248 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 4, length 64
17:07:23.741277 IP 6eb4896e0f04 > myproject_containerb_1.myproject_mynetwork: ICMP redirect google-public-dns-a.google.com to host 10.10.1.1, length 92
17:07:23.741281 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 4, length 64
17:07:24.765267 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 5, length 64
17:07:24.765314 IP 6eb4896e0f04 > myproject_containerb_1.myproject_mynetwork: ICMP redirect google-public-dns-a.google.com to host 10.10.1.1, length 92
17:07:24.765319 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 5, length 64
17:07:25.725183 ARP, Request who-has 6eb4896e0f04 tell 10.10.1.1, length 28
17:07:25.725213 ARP, Reply 6eb4896e0f04 is-at 02:42:0a:0a:01:03 (oui Unknown), length 28
17:07:25.766514 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 6, length 64
17:07:25.766543 IP 6eb4896e0f04 > myproject_containerb_1.myproject_mynetwork: ICMP redirect google-public-dns-a.google.com to host 10.10.1.1, length 92
17:07:25.766546 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 6, length 64
17:07:26.767908 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 7, length 64
17:07:26.767930 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 7, length 64
17:07:27.773201 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 8, length 64
17:07:27.773226 IP 6eb4896e0f04 > myproject_containerb_1.myproject_mynetwork: ICMP redirect google-public-dns-a.google.com to host 10.10.1.1, length 92
17:07:27.773230 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 8, length 64
17:07:28.774537 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 9, length 64
17:07:28.774555 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 9, length 64
17:07:29.789239 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 10, length 64
17:07:29.789260 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 10, length 64

EDIT 3:

# Generated by iptables-save v1.6.0 on Mon Jan 28 21:38:44 2019
*mangle
:PREROUTING ACCEPT [1095680:979414720]
:INPUT ACCEPT [1087344:977384769]
:FORWARD ACCEPT [8231:2022517]
:OUTPUT ACCEPT [733683:74635430]
:POSTROUTING ACCEPT [741914:76657947]
COMMIT
# Completed on Mon Jan 28 21:38:44 2019
# Generated by iptables-save v1.6.0 on Mon Jan 28 21:38:44 2019
*nat
:PREROUTING ACCEPT [2457:111645]
:INPUT ACCEPT [1538:56511]
:OUTPUT ACCEPT [15:1088]
:POSTROUTING ACCEPT [917:55188]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 10.10.1.0/24 ! -o br-a52f541ced84 -j MASQUERADE
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 10.10.1.3/32 -d 10.10.1.3/32 -p tcp -m tcp --dport 4242 -j MASQUERADE
-A DOCKER -i br-a52f541ced84 -j RETURN
-A DOCKER -i docker0 -j RETURN
-A DOCKER ! -i br-a52f541ced84 -p tcp -m tcp --dport 8081 -j DNAT --to-destination 10.10.1.3:4242
COMMIT
# Completed on Mon Jan 28 21:38:44 2019
# Generated by iptables-save v1.6.0 on Mon Jan 28 21:38:44 2019
*filter
:INPUT ACCEPT [11:784]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [15:2336]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o br-a52f541ced84 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-a52f541ced84 -j DOCKER
-A FORWARD -i br-a52f541ced84 ! -o br-a52f541ced84 -j ACCEPT
-A FORWARD -i br-a52f541ced84 -o br-a52f541ced84 -j ACCEPT
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 10.10.1.3/32 ! -i br-a52f541ced84 -o br-a52f541ced84 -p tcp -m tcp --dport 4242 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i br-a52f541ced84 ! -o br-a52f541ced84 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o br-a52f541ced84 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Mon Jan 28 21:38:44 2019