On AWS, I have run a consul container and a overlay network on a Docker host.
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
77b97d803d8c busybox "sh" 5 days ago Up 5 days busybox
206af8b467c2 daocloud.io/library/httpd "httpd-foreground" 5 days ago Up 5 days 0.0.0.0:32768->80/tcp web1
b40d24fd8f91 progrium/consul "/bin/start -serve..." 5 days ago Up 5 days 53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp consul
docker network inspect myoverlay
[
{
"Name": "myoverlay",
"Id": "a1752c608259d44a5bc0ad8b47e8b340d0252d026bf113cea681e8a92633e844",
"Created": "2017-09-05T06:37:13.271539119Z",
"Scope": "global",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "172.16.16.0/24",
"IPRange": "172.16.16.128/25",
"AuxiliaryAddresses": {
"ip2": "172.16.16.2"
}
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"206af8b467c2020b51017524984b322f1544e6b8594b672d4fa6bcb7b263b564": {
"Name": "web1",
"EndpointID": "d44ecac9eb248a90dd895d31170aca65333f248616a0998e22b81e4a1c7414b2",
"MacAddress": "02:42:ac:10:10:81",
"IPv4Address": "172.16.16.129/24",
"IPv6Address": ""
},
"77b97d803d8c0c3c3ceb0bcb6ca4f82971f9a68273ec32266cadd950350a4a7e": {
"Name": "busybox",
"EndpointID": "1a00a326b49b448c7b280faca58bb7ca67cea5d11c513ce64928b0a76fb673f3",
"MacAddress": "02:42:ac:10:10:82",
"IPv4Address": "172.16.16.130/24",
"IPv6Address": ""
},
"ep-f774da45dbf1655150fe5dd0a76b29c02873dd79181f26d1c39f12fd2523897d": {
"Name": "web2",
"EndpointID": "f774da45dbf1655150fe5dd0a76b29c02873dd79181f26d1c39f12fd2523897d",
"MacAddress": "02:42:ac:10:10:83",
"IPv4Address": "172.16.16.131/24",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {}
}
]
As you can see, there are three containers running on the Docker host that is running the Consul container.
And the containers within one Docker host can ping each other with the overlay IP.
docker exec 77b97d803d8c ping 172.16.16.129 PING 172.16.16.129 (172.16.16.129): 56 data bytes 64 bytes from 172.16.16.129: seq=0 ttl=64 time=0.098 ms 64 bytes from 172.16.16.129: seq=1 ttl=64 time=0.096 ms 64 bytes from 172.16.16.129: seq=2 ttl=64 time=0.096 ms 64 bytes from 172.16.16.129: seq=3 ttl=64 time=0.098 ms ^C
The following output is from another Docker host that has joined in the overlay network.
docker network inspect myoverlay
[
{
"Name": "myoverlay",
"Id": "a1752c608259d44a5bc0ad8b47e8b340d0252d026bf113cea681e8a92633e844",
"Created": "2017-09-05T06:37:13.271539119Z",
"Scope": "global",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "172.16.16.0/24",
"IPRange": "172.16.16.128/25",
"AuxiliaryAddresses": {
"ip2": "172.16.16.2"
}
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"9caaf8d7fe01104f4609053d94ca231fc4c91d5862b90244768c2c08f99cb65e": {
"Name": "web2",
"EndpointID": "f774da45dbf1655150fe5dd0a76b29c02873dd79181f26d1c39f12fd2523897d",
"MacAddress": "02:42:ac:10:10:83",
"IPv4Address": "172.16.16.131/24",
"IPv6Address": ""
},
"ep-1a00a326b49b448c7b280faca58bb7ca67cea5d11c513ce64928b0a76fb673f3": {
"Name": "busybox",
"EndpointID": "1a00a326b49b448c7b280faca58bb7ca67cea5d11c513ce64928b0a76fb673f3",
"MacAddress": "02:42:ac:10:10:82",
"IPv4Address": "172.16.16.130/24",
"IPv6Address": ""
},
"ep-d44ecac9eb248a90dd895d31170aca65333f248616a0998e22b81e4a1c7414b2": {
"Name": "web1",
"EndpointID": "d44ecac9eb248a90dd895d31170aca65333f248616a0998e22b81e4a1c7414b2",
"MacAddress": "02:42:ac:10:10:81",
"IPv4Address": "172.16.16.129/24",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {}
}
]
The docker containers from one Docker host somehow can not ping the ones in another Docker host.
docker exec 77b97d803d8c ping 172.16.16.131
The security group of the Docker hosts have allowed all traffic from each other.
I tried to telnet 7946 port of Serf of one Docker host, but it only returns no route to host.
Is there anything wrong with my configuration? Or any lack of configuration for the overlay? Thanks in advance for any help.
Best Answer
The behavior you're seeing suggests that there is a firewall somewhere between the docker hosts, maybe a firewall on the host itself, or perhaps on the network. You need the following open:
The iptables commands for this (if you have iptables defaulting to drop):
I'd recommend using tcpdump on each of the nodes to monitor for whether this traffic is being blocked somewhere.
The other possibility is that you're encountering a defect with Docker, like issue 32195. However, that would only affect some requests occasionally, not every request.