Nginx – Some Nginx Reverse Proxy Configs Stop Working Once a Day

nginxreverse-proxy

I have an nginx reverse-proxy which proxies requests from an outer amazon ELB to internal ELBs.

I have 6 backend instances that handles the requests. The site-enabled configs looks like this, but there are different port numbers and proxy_pass. Everything else is identical:

server {
    listen 3000;
    location / {
            proxy_pass http://internal-prod732r8-PrivateE-1GJ070M0745TT-348518554.eu-west-1.elb.amazonaws.com:3000;
            include /etc/nginx/proxy.conf;
    }

}

Once about every 24h one of the configurations stops working. All other proxies works fine. If i restart nginx all configurations works again. There is nothing in error.log, nothing weird in access log, syslog or dmesg.

Is this something known? Have i done something wrong with my proxy configs? Are there any other logs i can look in?

Best Answer

The answer to this question is that ELBs sometimes change ip adresses and nginx does name resolving during start.

To fix this there is always a DNS server in your VPC at 0.2. So if the local ip CIDR is 10.0.0.0/16 the DNS server is at 10.0.0.2.

Add this to the nginx config.

resolver 10.0.0.2 valid=10s;

The proxy_pass also needs to be defined as a variable otherwise nginx will only resolve it once. So based on the configuration above this is the correct config:

server {
    listen 3000;
    location / {
            resolver 10.0.0.2 valid=10s;
            set $backend "http://internal-prod732r8-PrivateE-1GJ070M0745TT-348518554.eu-west-1.elb.amazonaws.com:3000"
            proxy_pass $backend;
            include /etc/nginx/proxy.conf;
    }
}

Related Solutions

Nginx – How to Set Up as a Caching Reverse Proxy

I don't think that there is a way to explicitly invalidate cached items, but here is an example of how to do the rest. Update: As mentioned by Piotr in another answer, there is a cache purge module that you can use. You can also force a refresh of a cached item using nginx's proxy_cache_bypass - see Cherian's answer for more information.

In this configuration, items that aren't cached will be retrieved from example.net and stored. The cached versions will be served up to future clients until they are no longer valid (60 minutes).

Your Cache-Control and Expires HTTP headers will be honored, so if you want to explicitly set an expiration date, you can do that by setting the correct headers in whatever you are proxying to.

There are lots of parameters that you can tune - see the nginx Proxy module documentation for more information about all of this including details on the meaning of the different settings/parameters: http://nginx.org/r/proxy_cache_path

http {
  proxy_cache_path  /var/www/cache levels=1:2 keys_zone=my-cache:8m max_size=1000m inactive=600m;
  proxy_temp_path /var/www/cache/tmp; 


  server {
    location / {
      proxy_pass http://example.net;
      proxy_cache my-cache;
      proxy_cache_valid  200 302  60m;
      proxy_cache_valid  404      1m;
    }
  }
}

NGINX Reverse Proxy – URL Rewrite

Any redirect to localhost doesn't make sense from a remote system (e.g. client's Web browser). So the rewrite flags permanent (301) or redirect (302) are not usable in your case.

Please try following setup using a transparent rewrite rule:

location  /foo {
  rewrite /foo/(.*) /$1  break;
  proxy_pass         http://localhost:3200;
  proxy_redirect     off;
  proxy_set_header   Host $host;
}

Use curl -i to test your rewrites. A very subtle change to the rule can cause nginx to perform a redirect.

Best Answer

Related Solutions

Nginx – How to Set Up as a Caching Reverse Proxy

NGINX Reverse Proxy – URL Rewrite

Related Topic