Python – Celery workers unable to connect to redis on docker instances

celerydjangodockerdocker-composepython

I have a dockerized setup running a Django app within which I use Celery tasks. Celery uses Redis as the broker.

Versioning:

Docker version 17.09.0-ce, build afdb6d4
docker-compose version 1.15.0, build e12f3b9
Django==1.9.6
django-celery-beat==1.0.1
celery==4.1.0
celery[redis]
redis==2.10.5

Problem:

My celery workers appear to be unable to connect to the redis container located at localhost:6379. I am able to telnet into the redis server on the specified port. I am able to verify redis-server is running on the container.

When I manually connect to the Celery docker instance and attempt to create a worker using the command celery -A backend worker -l info I get the notice:

[2017-11-13 18:07:50,937: ERROR/MainProcess] consumer: Cannot connect to redis://localhost:6379/0: Error 99 connecting to localhost:6379. Cannot assign requested address.. Trying again in 4.00 seconds...

Notes:

I am able to telnet in to the redis container on port 6379. On the redis container, redis-server is running.

Is there anything else that I'm missing? I've gone pretty far down the rabbit hole, but feel like I'm missing something really simple.

DOCKER CONFIG FILES:

docker-compose.common.yml here
docker-compose.dev.yml here

Best Answer

When you use docker-compose, you aren't going to be using localhost for inter-container communication, you would be using the compose-assigned hostname of the container. In this case, the hostname of your redis container is redis. The top level elements under services: are your default host names.

So for celery to connect to redis, you should try redis://redis:6379/0. Since the protocol and the service name are the same, I'll elaborate a little more: if you named your redis service "butter-pecan-redis" in your docker-compose, you would instead use redis://butter-pecan-redis:6379/0.

Also, docker-compose.dev.yml doesn't appear to have celery and redis on a common network, which might cause them not to be able to see each other. I believe they need to share at least one network in common to be able to resolve their respective host names.

Networking in docker-compose has an example in the first handful of paragraphs, with a docker-compose.yml to look at.

TLDR

Use --network="host" in your docker run command, then 127.0.0.1 in your docker container will point to your docker host.

Note: This mode only works on Docker for Linux, per the documentation.

Note on docker container networking modes

Docker offers different networking modes when running containers. Depending on the mode you choose you would connect to your MySQL database running on the docker host differently.

docker run --network="bridge" (default)

Docker creates a bridge named docker0 by default. Both the docker host and the docker containers have an IP address on that bridge.

on the Docker host, type sudo ip addr show docker0 you will have an output looking like:

[vagrant@docker:~] $ sudo ip addr show docker0
4: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 56:84:7a:fe:97:99 brd ff:ff:ff:ff:ff:ff
    inet 172.17.42.1/16 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::5484:7aff:fefe:9799/64 scope link
       valid_lft forever preferred_lft forever

So here my docker host has the IP address 172.17.42.1 on the docker0 network interface.

Now start a new container and get a shell on it: docker run --rm -it ubuntu:trusty bash and within the container type ip addr show eth0 to discover how its main network interface is set up:

root@e77f6a1b3740:/# ip addr show eth0
863: eth0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 66:32:13:f0:f1:e3 brd ff:ff:ff:ff:ff:ff
    inet 172.17.1.192/16 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::6432:13ff:fef0:f1e3/64 scope link
       valid_lft forever preferred_lft forever

Here my container has the IP address 172.17.1.192. Now look at the routing table:

root@e77f6a1b3740:/# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         172.17.42.1     0.0.0.0         UG    0      0        0 eth0
172.17.0.0      *               255.255.0.0     U     0      0        0 eth0

So the IP Address of the docker host 172.17.42.1 is set as the default route and is accessible from your container.

root@e77f6a1b3740:/# ping 172.17.42.1
PING 172.17.42.1 (172.17.42.1) 56(84) bytes of data.
64 bytes from 172.17.42.1: icmp_seq=1 ttl=64 time=0.070 ms
64 bytes from 172.17.42.1: icmp_seq=2 ttl=64 time=0.201 ms
64 bytes from 172.17.42.1: icmp_seq=3 ttl=64 time=0.116 ms

docker run --network="host"

Alternatively you can run a docker container with network settings set to host. Such a container will share the network stack with the docker host and from the container point of view, localhost (or 127.0.0.1) will refer to the docker host.

Be aware that any port opened in your docker container would be opened on the docker host. And this without requiring the -p or -P docker run option.

IP config on my docker host:

[vagrant@docker:~] $ ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:98:dc:aa brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe98:dcaa/64 scope link
       valid_lft forever preferred_lft forever

and from a docker container in host mode:

[vagrant@docker:~] $ docker run --rm -it --network=host ubuntu:trusty ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:98:dc:aa brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe98:dcaa/64 scope link
       valid_lft forever preferred_lft forever

As you can see both the docker host and docker container share the exact same network interface and as such have the same IP address.

Connecting to MySQL from containers

bridge mode

To access MySQL running on the docker host from containers in bridge mode, you need to make sure the MySQL service is listening for connections on the 172.17.42.1 IP address.

To do so, make sure you have either bind-address = 172.17.42.1 or bind-address = 0.0.0.0 in your MySQL config file (my.cnf).

If you need to set an environment variable with the IP address of the gateway, you can run the following code in a container :

export DOCKER_HOST_IP=$(route -n | awk '/UG[ \t]/{print $2}')

then in your application, use the DOCKER_HOST_IP environment variable to open the connection to MySQL.

Note: if you use bind-address = 0.0.0.0 your MySQL server will listen for connections on all network interfaces. That means your MySQL server could be reached from the Internet ; make sure to setup firewall rules accordingly.

Note 2: if you use bind-address = 172.17.42.1 your MySQL server won't listen for connections made to 127.0.0.1. Processes running on the docker host that would want to connect to MySQL would have to use the 172.17.42.1 IP address.

host mode

To access MySQL running on the docker host from containers in host mode, you can keep bind-address = 127.0.0.1 in your MySQL configuration and all you need to do is to connect to 127.0.0.1 from your containers:

[vagrant@docker:~] $ docker run --rm -it --network=host mysql mysql -h 127.0.0.1 -uroot -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 36
Server version: 5.5.41-0ubuntu0.14.04.1 (Ubuntu)

Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

note: Do use mysql -h 127.0.0.1 and not mysql -h localhost; otherwise the MySQL client would try to connect using a unix socket.

Telling ‘make’ to ignore dependencies when the top target has been created

Use "intermediate files" feature of GNU Make:

Intermediate files are remade using their rules just like all other files. But intermediate files are treated differently in two ways.

The first difference is what happens if the intermediate file does not exist. If an ordinary file b does not exist, and make considers a target that depends on b, it invariably creates b and then updates the target from b. But if b is an intermediate file, then make can leave well enough alone. It won't bother updating b, or the ultimate target, unless some prerequisite of b is newer than that target or there is some other reason to update that target.

The second difference is that if make does create b in order to update something else, it deletes b later on after it is no longer needed. Therefore, an intermediate file which did not exist before make also does not exist after make. make reports the deletion to you by printing a rm -f command showing which file it is deleting.

Ordinarily, a file cannot be intermediate if it is mentioned in the makefile as a target or prerequisite. However, you can explicitly mark a file as intermediate by listing it as a prerequisite of the special target .INTERMEDIATE. This takes effect even if the file is mentioned explicitly in some other way.

You can prevent automatic deletion of an intermediate file by marking it as a secondary file. To do this, list it as a prerequisite of the special target .SECONDARY. When a file is secondary, make will not create the file merely because it does not already exist, but make does not automatically delete the file. Marking a file as secondary also marks it as intermediate.

So, adding the following line to the Makefile should be enough:

.INTERMEDIATE : hugefileB hugefileC

Invoking make for the first time:

$ make
touch hugefileB
touch hugefileC
cat hugefileB hugefileC > digestA
rm hugefileB hugefileC

And the next time:

$ make
make: `digestA' is up to date.