I have a weird issue with nginx proxy+cache setup proxying for two identical upstreams. I have two identical upstream servers setup in an active/backup configuration. The cache is working fine for one of the servers, but as soon as I switch the active/backup config and reload nginx, I do not get any cache hits, and the error log fills up with these:

2017/06/15 10:14:21 [warn] 19048#19048: *2562 an upstream response is buffered to a temporary file /mnt/cache/temp/0000000504 while reading upstream, client: <CLIENT IP>, server: , request: "GET /secure/assets/file.ts?id=456 HTTP/1.1", upstream: "http://<UPSTREAM IP>:80/assets/file.ts", host: "server-host"

Now, I cannot find out why nginx has to buffer the responses for one of the upstreams and not the other, as the upstream servers have the exact same files and are sitting in the same network.

Now, the relevant parts of the config:

http {

    tcp_nodelay on;

    proxy_cache_path /mnt/cache/cache keys_zone=one:20m inactive=24h loader_threshold=300 loader_files=200 max_size=180g;
    proxy_cache_key $uri;

    upstream master-cdn {
        server cdn1 backup max_fails=10 fail_timeout=60s;
        server cdn0 max_fails=10 fail_timeout=60s;
        keepalive 10;
    }

    proxy_next_upstream error timeout http_500 http_502 http_503 http_504 http_403 http_404;

        server {

                listen 80;
                if_modified_since off;
                add_header          Cache-Control no-cache;
                add_header          Pragma "no-cache";
                root                /var/www/html;

                location /assets {
                        sendfile on;
                        tcp_nopush on;
                        internal;
                        proxy_pass http://master-cdn;
                        proxy_cache             one;
                        proxy_cache_min_uses    1;
                        proxy_cache_lock        on;
                        proxy_cache_valid       200 24h;
                        proxy_temp_path /mnt/cache/temp;
                }
}

Note: the clients are hitting the /secure url, which gets sent to an application server, which will then use the X-Accel-Redirect header to tell nginx to serve the static file from the internal location.

The whole setup works just fine, but only with the cdn0 upstream server. As soon as I move the backup statement to the cdn0 server and reload nginx, I get the upstream buffering and cache misses.

Does anyone have any ideas on how to debug this further?

Thanks in advance.