How to Bind to UDP Port 53 in GCE (Container-Optimized OS)

dockergoogle-cloud-platformgoogle-compute-enginelinux-networkingtcpip

Is there any way to free up UDP port 53 on my GCE instance so that I can bind to it?

I'm running a container on GCE that needs to listen on UDP port 53, as it's listening for DNS connections. However, when I try to run my container and bind to UDP port 53, I get the following error message:

docker: Error response from daemon: driver failed programming external
connectivity on endpoint
():
Error starting userland proxy: listen udp 0.0.0.0:53: bind: address
already in use.

Checking for open ports shows that 127.0.0.53 is connected on UDP port 53:

$ netstat -tuln | grep 53
tcp        0      0 0.0.0.0:5355            0.0.0.0:*               LISTEN     
tcp6       0      0 :::5355                 :::*                    LISTEN     
udp        0      0 127.0.0.53:53           0.0.0.0:*                          
udp        0      0 0.0.0.0:5355            0.0.0.0:*                          
udp6       0      0 :::5355                 :::*

Reading through the documentation, the only reference to something like this was zonal DNS, which can be disabled by setting the instance or project metadata VmDnsSetting=GlobalOnly (article here). I did this, and the port binding is still there.

Best Answer

Solution

It appears that the culprit is the systemd-resolved service.

You can stop the service with the command $ sudo systemctl stop systemd-resolved.

More details

In order to do this on GCE using Google's container-optimized OS ("COS"), I needed to deselect the option to boot from a container image, and then manually go in and enter the relevant commands to get the environment up and running as follows.

The following 2 commands are needed to pull from a private GCR registry:

$ docker-credential-gcr configure-docker

$ docker pull gcr.io/<project>/<container>

Stop the systemd-resolved service:

$ sudo systemctl stop systemd-resolved

Start the docker container (note the --network host flag, which is what COS does by default when you select the option to boot from a container image in the instance configuration):

$ docker run -d --network host --restart=unless-stopped <container>

Open the relevant ports (needed when not explicitly booting from a container image when configuring the instance):

$ sudo iptables -w -A INPUT -p tcp --dport 22 -j ACCEPT

$ sudo iptables -w -A INPUT -p udp --dport 53 -j ACCEPT

A note on automation

Note that you can't put the first command ($ docker-credential-gcr configure-docker) in the startup script, as it fails due to the /root/.docker/ directory being read-only; startup scripts are run as root once networking is available.

I still haven't figured this one out yet, but I will update this answer once I have, to make it a little less manual for anyone else reading this.

Update on automation

It turns out that you can create a user in a cloud-init script, then use $ su - <user> in your startup script before running the $ docker-credential-gcr configure-docker command as that user to save the credentials (referenced in this support forum post). However, this is way too hacky for me, so I've opted to just run the commands manually once the GCE instance is spun up to get Docker running properly.

tl;dr: How to unbind port 53 on startup

However, for anyone who may be looking to just unbind port 53 on startup, you can do so with the following startup script:

#! /bin/bash

systemctl stop systemd-resolved

Instructions for adding a startup script to your VM on GCE can be found here.

Related Topic