I have two Docker containers on a Debian 8.3 node. One is the official postgres image, one is a basic phoenix / elixir app. I'm linking both with a docker stack file. But phoenix is not able to connect to postgres, unless I publish the port. This lets me think that something with the internal docker network is wrong and since the node is a fresh Debian install it might be the iptables. It also kind of excludes that the password or hostname is wrong.
What am I doing wrong and what is the correct way to set up the iptables rules to allow two containers to communicate?
Error message in phoenix app:
app-1 | 2016-03-15T16:15:18.019402549Z ** (Mix) The database for App.Repo couldn't be created, reason given: psql: could not connect to server: Connection refused
app-1 | 2016-03-15T16:15:18.019456447Z Is the server running on host "postgres" (10.7.0.1) and accepting
app-1 | 2016-03-15T16:15:18.019468609Z TCP/IP connections on port 5432?
Output of logfile in postgres container
postgres-1 | 2016-03-15T16:46:32.457844697Z LOG: MultiXact member wraparound protections are now enabled
postgres-1 | 2016-03-15T16:46:32.464806051Z LOG: database system is ready to accept connections
postgres-1 | 2016-03-15T16:46:32.465087076Z LOG: autovacuum launcher started
My Docker Stack File
app:
image: myrepo/app
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgrespassword
PORT: 4000
links:
- postgres
ports:
- 80:4000
postgres:
image: postgres:9.5
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgrespassword
volumes:
- /var/dbdata:/var/lib/postgresql/data
Database config in Phoenix App (prod.secret.exs)
config :data_bucket, DataBucket.Repo,
adapter: Ecto.Adapters.Postgres,
username: System.get_env("POSTGRES_USER"),
password: System.get_env("POSTGRES_PASSWORD"),
database: "app_prod",
hostname: "postgres",
pool_size: 20
Result of $ sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere anywhere
REJECT all -- anywhere loopback/8 reject-with icmp-port-unreachable
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT tcp -- anywhere anywhere tcp dpt:http
ACCEPT tcp -- anywhere anywhere tcp dpt:https
ACCEPT tcp -- anywhere anywhere tcp dpt:2375
ACCEPT tcp -- anywhere anywhere tcp dpt:6783
ACCEPT udp -- anywhere anywhere udp dpt:6783
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ssh
ACCEPT icmp -- anywhere anywhere icmp echo-request
LOG all -- anywhere anywhere limit: avg 5/min burst 5 LOG level debug prefix "iptables denied: "
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable
DROP tcp -- anywhere 172.17.0.1 tcp dpt:6783
DROP udp -- anywhere 172.17.0.1 udp dpt:6783
DROP udp -- anywhere 172.17.0.1 udp dpt:6784
ACCEPT udp -- anywhere anywhere udp dpt:domain
ACCEPT tcp -- anywhere anywhere tcp dpt:domain
Chain FORWARD (policy ACCEPT)
target prot opt source destination
DROP all -- anywhere anywhere
DROP all -- anywhere anywhere
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable
ACCEPT all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere anywhere
Chain DOCKER (1 references)
target prot opt source destination
Result of $ sudo docker version
Client:
Version: 1.9.1-cs2
API version: 1.21
Go version: go1.4.3
Git commit: 4ade326
Built: Mon Nov 30 21:56:07 UTC 2015
OS/Arch: linux/amd64
Server:
Version: 1.9.1-cs2
API version: 1.21
Go version: go1.4.3
Git commit: 4ade326
Built: Mon Nov 30 21:56:07 UTC 2015
OS/Arch: linux/amd64
Result of sudo docker ps
:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f113435b781b myrepo/app:latest "elixir --erl '-smp d" 35 seconds ago Up 34 seconds 0.0.0.0:80->4000/tcp app-1.App.15ffa2c2
6e2879fd9f2c postgres:9.5 "/docker-entrypoint.s" 37 minutes ago Up 5 minutes 5432/tcp postgres-1.App.cbe400ac
For the last output: The Phoenix App of course only starts if I don't run migrations. Normally this is at the end of my Dockerfile:
CMD ["elixir", "--erl", "-smp disable", "/usr/local/bin/mix", "do", "compile", ",", "ecto.create", ",", "ecto.migrate", ",", "phoenix.server"]
Which I changed to the following to get this output
CMD ["elixir", "--erl", "-smp disable", "/usr/local/bin/mix", "do", "compile", ",", "phoenix.server"]
Best Answer
I found the solution: The problem were the iptables settings. I had a rule in there to drop all forwarding unless otherwise defined:
Getting rid of this rule was the solution.
Edit:
As @Zoredache pointed out just deleting the rule is obviously a bad idea and I'm an idiot. The correct way is to put it at the very end of the rule chain, after the rules created by docker. So if you have the same problem, this works for me:
Get rid of the rule
Add it again to move it to the end of the set
Make sure they are in the right order. So the reject all else rule should be at the very end.
On Debian this is how you can make sure the rules still apply when restarting the server:
Once you are happy, save the new rules to the master iptables file:
To make sure the iptables rules are started on a reboot we'll create a new file:
Add these lines to it:
The file needs to be executable so change the permissions: