Iptables – EC2 instance’s Docker ports not not accessible after changing instance type

amazon ec2amazon-web-servicesdockeriptablessecurity-groups

When changing the EC2 instance type I ran into a problem. The machine had 3 Docker containers that needed to be restarted and after the restart their ports became unreachable.

What could be the issue and how should I go about obtaining other necessary debug information?

  • No changes were made to the Security Groups configuration within AWS, all the required ports are still enabled.

  • I am still able to SSH into the EC2 instance, but the ports used by
    Docker (80, 8181) are not accessible (connection timeout).

  • Within a web browser it doesn't matter if I'm trying to access a port that is being used or
    not, the browser behavior is always the same (loading indicator stops
    at the beginning, followed by timeout, nothing gets logged in e.g. Apache's access.log or error.log
    ).

  • Within a web browser neither addressing the instance by its Public DNS (IPv4), IPv4 Public
    IP, or its original domain name works.

  • Restarting the instance or changing its type again doesn't help

I am able to ping/telnet/wget the ports used by the Docker containers from
within the instance:

$ docker exec f227cf8d9481 wget 127.0.0.1:8181
converted 'http://127.0.0.1:8181' (ANSI_X3.4-1968) -> 'http://127.0.0.1:8181' (UTF-8)
--2018-01-15 23:49:10--  http://127.0.0.1:8181/
Connecting to 127.0.0.1:8181... connected.
HTTP request sent, awaiting response... 401 Unauthorized

But not from the outside (the IP address still gets resolved):

$ wget <aws-ip>.<aws-zone>.<instance>.amazonaws.com:8181
--2018-01-16 00:53:32--  http://<aws-ip>.<aws-zone>.<instance>.amazonaws.com:8181/
Resolving <aws-ip>.<aws-zone>.<instance>.amazonaws.com... xxx.xxx.xxx.xxx
Connecting to <aws-ip>.<aws-zone>.<instance>.amazonaws.com|xxx.xxx.xxx.xxx|:8181... failed: Operation timed out.
Retrying.

The docker containers are running and mapping between Docker ports is
done correctly:

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                                                                    NAMES
f227cf8d9481        cloud9              "forever /cloud9/s..."   3 seconds ago       Up 2 seconds        0.0.0.0:8080-8081->8080-8081/tcp, 80/tcp, 0.0.0.0:8181->8181/tcp, 0.0.0.0:81->3000/tcp   my-cloud9
fa0d2bbce863        wordpress           "docker-entrypoint..."   59 minutes ago      Up 59 minutes       0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp                                                 goofy_torvalds
6ada961a5ea0        mysql               "docker-entrypoint..."   About an hour ago   Up About an hour    0.0.0.0:3306->3306/tcp  

Iptables setting seems to have the Docker ports enabled:

$ sudo iptables --table nat --list
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  anywhere            !loopback/8           ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  <aws-ip>.<aws-zone>.<instance>.internal/16  anywhere            
MASQUERADE  tcp  --  <aws-ip>.<aws-zone>.<instance>.internal  <aws-ip>.<aws-zone>.<instance>.internal  tcp dpt:mysql
MASQUERADE  tcp  --  <aws-ip>.<aws-zone>.<instance>.internal  <aws-ip>.<aws-zone>.<instance>.internal  tcp dpt:https
MASQUERADE  tcp  --  <aws-ip>.<aws-zone>.<instance>.internal  <aws-ip>.<aws-zone>.<instance>.internal  tcp dpt:http
MASQUERADE  tcp  --  <aws-ip>.<aws-zone>.<instance>.internal  <aws-ip>.<aws-zone>.<instance>.internal  tcp dpt:8181
MASQUERADE  tcp  --  <aws-ip>.<aws-zone>.<instance>.internal  <aws-ip>.<aws-zone>.<instance>.internal  tcp dpt:tproxy
MASQUERADE  tcp  --  <aws-ip>.<aws-zone>.<instance>.internal  <aws-ip>.<aws-zone>.<instance>.internal  tcp dpt:webcache
MASQUERADE  tcp  --  <aws-ip>.<aws-zone>.internal  <aws-ip>.<aws-zone>.<instance>.internal  tcp dpt:hbci

Chain DOCKER (2 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            
DNAT       tcp  --  anywhere             anywhere             tcp dpt:mysql to:<docker-ip>:3306
DNAT       tcp  --  anywhere             anywhere             tcp dpt:https to:<docker-ip>:443
DNAT       tcp  --  anywhere             anywhere             tcp dpt:http to:<docker-ip>:80
DNAT       tcp  --  anywhere             anywhere             tcp dpt:8181 to:<docker-ip>:8181
DNAT       tcp  --  anywhere             anywhere             tcp dpt:tproxy to:<docker-ip>:8081
DNAT       tcp  --  anywhere             anywhere             tcp dpt:webcache to:<docker-ip>:8080
DNAT       tcp  --  anywhere             anywhere             tcp dpt:81 to:<docker-ip>:3000

No there doesn't seem to be any noticeable network activity (around 2KB every 5 mins) using the Monitoring tool of the EC2 instance. Except for the some spikes from the times I used SSH to log in:

enter image description here

Best Answer

The issue was with the DNS and SSL certificate configuration, because the instance was configured to use HTTPS only..

After the change of the instance type, the new instance was automatically assigned a new URL, which had to be updated with both the DNS provider and the CA.