How to debug/log 502 Bad Gateway from reverse proxy

apache-2.4debuggingloggingreverse-proxy

We're running a web service and hitting 502 Bad Gateway randomly. The 502 is reported by reverse proxy (Apache 2.4.x) while talking to backend server (Apache 2.4.x). The problem randomly re-appears around once per 100,000 served requests so it's really hard to reproduce. I see that Apache allows configuring ProxyBadHeader directive where possible options are IsError|Ignore|StartBody. I have problems finding the exact cause of the error. I know that if I set ProxyBadHeader to StartBody the partial output would end up to the browser. I really would not want to emit the data to unknown end user so I need something better. Is it possible to log all the partially received data from the bad gateway?

The problem seems similar to Apache bug 37770 but it should be fixed in the version we're running.

Best Answer

In my experience, you are correct to point to Apache bug 37770, which in the Apache's reverse proxy functionality. If you examine the patches cited there from 2008 until 2016, you'll see yourself that these are really workarounds. If the root cause disappeared in httpd-2.4, then this is not apparent neither from the conversation there or from the observed httpd behavior.

Either

SetEnv proxy-initial-not-pooled 1     # prevent a small percentage of 502 errors, comes with a performance penalty

or accept that a little percentage of requests get 502'd. Or migrate to haproxy.

I guess that if you did gather the backend's conversation, you would see that backend doesn't see the HTTP GET. This was result of my little investigation and I concluded that the connection gets somehow lost inside reverse proxy code (perhaps where backend connection is taken from pool or returned there).

Related Solutions

NGINX Reverse Proxy – URL Rewrite

Any redirect to localhost doesn't make sense from a remote system (e.g. client's Web browser). So the rewrite flags permanent (301) or redirect (302) are not usable in your case.

Please try following setup using a transparent rewrite rule:

location  /foo {
  rewrite /foo/(.*) /$1  break;
  proxy_pass         http://localhost:3200;
  proxy_redirect     off;
  proxy_set_header   Host $host;
}

Use curl -i to test your rewrites. A very subtle change to the rule can cause nginx to perform a redirect.

Nginx – 502 bad gateway when trying to log out through nginx reverse proxy

My guess would be you need to adjust fastcgi_buffers and/or fastcgi_buffer_size and possibly proxy_buffers, proxy_buffer_size, proxy_busy_buffers_size, proxy_read_timeout.

Best Answer

Related Solutions

NGINX Reverse Proxy – URL Rewrite

Nginx – 502 bad gateway when trying to log out through nginx reverse proxy

Related Topic