Nginx Docker GitLab – Fix GitLab Pages Inaccessible from Outside Docker Container

dockergitlabnginx

I have a GitLab instance running in docker. GitLab works fine however I can't access GitLab Pages.

The setup

Overview

           +------------+
Request+-->+ Cloudflare |
           +-----+------+
                 |
                 v       +---------+
               Nginx+--->+Docker   |
                         |  +------+
                         |  |GitLab|
                         +---------+

The problem is that Nginx can't pass the request to the GitLab Pages server (note that GitLab it self works).
Nginx error log entry

[error] 14932#14932: *30505 connect() failed (111: Connection refused) while connecting to upstream, [...]

Docker

image: gitlab/gitlab-ce
version: 13.7.1 (latest)
ip: 172.17.0.7 (dynamic)
published ports:
    172.17.0.1:8080 -> 80
    172.17.0.1:8090 -> 8090

Nginx

Server entry for Pages

server {
        listen 80 default_server;
        listen 443 default_server;
        server_name _;
        
        location / {
                proxy_pass      http://172.17.0.1:8090;
        }
}

GitLab

grep -v '^#|^$' gitlab.rb1

nginx['listen_port'] = 80
nginx['listen_https'] = false
pages_external_url "http://pages.example.com/"
gitlab_pages['enable'] = true
gitlab_pages['external_http'] = []
gitlab_pages['listen_proxy'] = "localhost:8090"
gitlab_pages['inplace_chroot'] = true
gitlab_pages['metrics_address'] = ":9235"
pages_nginx['enable'] = true
pages_nginx['listen_https'] = false
pages_nginx['redirect_http_to_https'] = false

Also tried with the absolute minimum config, only defining pages_external_url and gitlab_pages['enable'].

Tracing down the problem

  1. Request to pages.example.com failed with 502 (Bad Gateway) from CF
  2. Checked Nginx log, found the above mentioned log entry
  3. Made multiple requests from the host to the container
    1. # curl 172.17.0.1:8090 -> curl: (7) Failed to connect to 172.17.0.1 port 8090: Connection refused
    2. # curl 172.17.0.7:8090 -> curl: (7) Failed to connect to 172.17.0.7 port 8090: Connection refused
  4. Made a request from the container
    # curl localhost:8090 -> 404 error page

From this I assume that something is blocking incomming traffic for 8090 (GitLab Pages), but requests to 80 (GitLab) are completed successfully. I spent a couple of days googling this issue but I couldn't find anything.

1 Truncated; SMTP, LDAP and omniauth settings removed

Best Answer

Just found the cause of my problem after many hours of searching the wrong thing (but more on that later).

TL;DR

I had to change the gitlab_pages['listen_proxy'] value to listen on every interface, so it looks like this:

gitlab_pages['listen_proxy'] = "0.0.0.0:8090"

Detailed

I was searching this issue with keywords related to gitlab pages and just now I tought maybe I need to think outside the box, what if it is a more general problem and not even related to GitLab Pages. After the first google search i found a nice article about this.

I got the "connection refused" message because nothing was listening on port 8090, because localhost refers to the loopback address, i.e. 127.0.0.1, but exposed ports are forwarded for the container's IP, in my case to a dynamic IP which as writing the question was 172.17.0.7. So the solution is to listen on every interface which is possible using 0.0.0.0 as the IP addres.

Two figures which visualize the problem and the solution

(Figures from the mentioned article, but slightly modified to match the question better and made it text only to improve readability when using force-darkmode)
When using localhost:8090:

   Request      +--------------------------------------+
      +         | Default network namespace            |
      |         |                                      |
      |         |               +-------+              |
      v         |        +----->+ Nginx +------+       |
+-----+------+  |        |      +-------+      |       |
| Cloudflare |  |        |                     |       |
+-----+------+  |        |                     v       |
      |         +--------+-----------+-+-------+-------+
      |         | Physical interface | | Docker bridge |
      +-------->+ 10.0.0.1           | | 172.17.0.1    |
                +--------------------+ +-------+-------+
                                               |
                                               |
                                               v
                                       +-------+-------+
                                       | Docker bridge |
                                       | 172.17.0.7    |
                 +-------+-----------+-+---------------+
                 |       | Loopback  |                 |
                 |       | 127.0.0.1 |                 |
                 |       +-----+-----+                 |
                 |             |                       |
                 |             |      +--------------+ |
                 |             +----->+ Pages server | |
                 |                    +--------------+ |
                 |GitLab's Network namespace           |
                 +-------------------------------------+

Listening on every interface, using 0.0.0.0:8090:

   Request      +--------------------------------------+
      +         | Default network namespace            |
      |         |                                      |
      |         |               +-------+              |
      v         |        +----->+ Nginx +------+       |
+-----+------+  |        |      +-------+      |       |
| Cloudflare |  |        |                     |       |
+-----+------+  |        |                     v       |
      |         +--------+-----------+-+-------+-------+
      |         | Physical interface | | Docker bridge |
      +-------->+ 10.0.0.1           | | 172.17.0.1    |
                +--------------------+ +-------+-------+
                                               |
                                               |
                                               v
                                       +-------+-------+
                                       | Docker bridge |
                                       | 172.17.0.7    |
                 +-------+-----------+-+-------+-------+
                 |       | Loopback  |         |       |
                 |       | 127.0.0.1 |         |       |
                 |       +-----+-----+         |       |
                 |             |               v       |
                 |             |      +--------+-----+ |
                 |             +----->+ Pages server | |
                 |                    +--------------+ |
                 |GitLab's Network namespace           |
                 +-------------------------------------+
Related Topic