Docker ports do not seem to be reachable from outside server

dockerlinux-networkingnetworkingslessles12

I am facing a weird situation with Docker in a SUSE Linux Enterprise Server 12 SP1.

I am connected to the server using SSH. First, I tried to run a simple nginx server to test:

docker run -d -p 8081:80 nginx:alpine --name nginxtest

The container starts successfully. Then, running curl http://localhost:8081 works!

<!-- [...] -->
<title>Welcome to nginx!</title>
<!-- [...] -->

However, when I try to access it from a browser from my computer, at http://10.etc.etc.etc:8081, the request times out.

Surprisingly, though, if I forget about docker for a moment, and use a simple HTTP server directly from the server, such as python3 -m http.server, I get:

Serving HTTP on 0.0.0.0 port 8000 ...

And when I access it from the browser in my computer, at http://10.etc.etc.etc:8000 it works!!

So this must be something about how docker exposes its ports, but it's weird because curl http://localhost:8081 works…

How can I further troubleshoot and then fix this?

Note: Things were working perfectly until yesterday, when I ran systemctl restart wicked (I was trying to investigate another unrelated issue). I also tried restarting the server but it didn't help.


Here is some more output that may be relevant…

  • ifconfig docker0
docker0   Link encap:Ethernet  HWaddr 02:49:68:4D:40:9B
          inet addr:172.17.0.1  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::42:8e6e:ee3f:6918/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:353842611 errors:0 dropped:0 overruns:0 frame:0
          TX packets:450340200 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:509803756314 (486186.7 Mb)  TX bytes:497391388709 (474349.3 Mb)
  • brctl show docker0
bridge name     bridge id               STP enabled     interfaces
docker0         8000.0249143a607f       no              veth0e4b1f4
                                                        veth1f2fcc2
                                                        veth389f6fe
                                                        [[others]]
  • docker ps | grep nginx is 0.0.0.0:8081->80/tcp

  • cat /etc/sysctl.conf

# net.ipv6.conf.all.disable_ipv6 = 1
# net.ipv6.conf.all.disable_ipv6 = 1
net.ipv4.ip_forward = 0
net.ipv6.conf.all.forwarding = 0
  • (netstat -ltunp | head -2) && (netstat -ltunp | grep docker)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 :::9000                 :::*                    LISTEN      28216/docker-proxy
tcp        0      0 :::5000                 :::*                    LISTEN      28165/docker-proxy
tcp        0      0 :::8081                 :::*                    LISTEN      30341/docker-proxy
  • iptables -n --list
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
DOCKER-ISOLATION  all  --  0.0.0.0/0            0.0.0.0/0
DOCKER     all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain DOCKER (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  0.0.0.0/0            172.17.0.11          tcp dpt:5000
ACCEPT     tcp  --  0.0.0.0/0            172.17.0.13          tcp dpt:9000
ACCEPT     tcp  --  0.0.0.0/0            172.17.0.2           tcp dpt:80

Chain DOCKER-ISOLATION (1 references)
target     prot opt source               destination
RETURN     all  --  0.0.0.0/0            0.0.0.0/0
  • iptables -nt nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DOCKER     all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
DOCKER     all  --  0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  all  --  172.17.0.0/16        0.0.0.0/0
MASQUERADE  tcp  --  172.17.0.11          172.17.0.11          tcp dpt:5000
MASQUERADE  tcp  --  172.17.0.13          172.17.0.13          tcp dpt:9000
MASQUERADE  tcp  --  172.17.0.2           172.17.0.2           tcp dpt:80

Chain DOCKER (2 references)
target     prot opt source               destination
RETURN     all  --  0.0.0.0/0            0.0.0.0/0
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:5000 to:172.17.0.11:5000
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:9000 to:172.17.0.13:9000
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:8081 to:172.17.0.2:80
  • docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
20c691cc38e6        bridge              bridge              local
cd95c7d14c38        host                host                local
eb6d8228f366        none                null                local
  • docker network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "20c691cc38e65be9bf0a377fd8560d49430f523608094b68145e8769e24b1764",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Containers": {
            "26887c686d3c8612e460b923692f371bef881065a525496dff9af993ed4b7949": {
                "Name": "sleepy_fermi",
                "EndpointID": "3e79d8c15423b2e145a900e796d316159d2dc51dcc10ced2099b77d1111b03e7",
                "MacAddress": "02:49:68:4D:40:9B",
                "IPv4Address": "172.17.0.6/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]
  • iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 172.17.0.11/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 5000 -j ACCEPT
-A DOCKER -d 172.17.0.13/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 9000 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER-ISOLATION -j RETURN
  • docker version
Client:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   78d1802
 Built:        Thu Mar  2 12:26:00 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   78d1802
 Built:        Thu Mar  2 12:26:00 2017
 OS/Arch:      linux/amd64

Best Answer

Here, our Docker container is reachable through docker0 (network: 172.17.x.x)

From your browser, you're trying to connect to some 10.x.x.x address.

Using a Browser on the Docker host

Assuming you just want to connect from your browser to a locally-hosted container, then the easier would just be to reach your container address (172.17.x.x).

In some cases, you could even use ssh -X connecting to your Docker host then start your web browser, without exposing them to your LAN.

Using HTTP Proxy

Assuming you want to expose http/https service to clients on your LAN, then you may use some reverse proxy (nginx, traefik, apache, ...).

dnf install httpd
setsebool -P httpd_can_network_connect on
cat <<EOF >/etc/httpd/conf.d/welcome.conf
<VirtualHost *:80>
    ServerName my-application.example.com
    LogLevel debug
    ErrorLog logs/tunnel_error.log
    CustomLog logs/tunnel_access.log combined
    <Location />
        Require all granted
    </Location>
    ProxyPass / http://<container-address>:8080/
    ProxyPassReverse / http://<container-address>:8080/
</VirtualHost>
EOF
systemctl enable httpd
systemctl start httpd

Configuring Firewall

Assuming you want to expose tcp/udp services to clients on your LAN, you may setup your firewall accordingly:

sysctl -w net.ipv4.conf.all.forwarding=1
echo net.ipv4.conf.all.forwarding=1 >>/etc/sysctl.conf
iptables -A FORWARD -i docker0 -o eth0 -j ACCEPT
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
iptables -t nat -A PREROUTING -p tcp -d <my-eth0-address> --dport 8080 -j DNAT --to <my-container-address>:8080

Note those iptables rules would not persist, if you reboot your system, or somehow reset your firewall. Not much familiar with Wicked.