HAProxy breaks HTTP request

apache-2.4http

I'm not really sure what's going on here. I made a little internet of things thing a while back which builds HTTP requests that look like:

POST /update.py HTTP/1.1
Host: iot.example.com
Connection: close
Content-Type: application/x-www-form-urlencoded
Content-Length: [...]

iv=[...]&msg=base64(encrypted_msg)

And a corresponding server side python script which extracted information from the POST data. I recently changed my firewall's reverse proxy from squid to HAProxy and suddenly Apache is returning HTTP 400 statuses, seeminly before the IOT device even gets a chance to send the POST data. Here's a wireshark TCP stream of one of these interactions:

POST /update.py HTTP/1.1
Host: iot.example.com
Connection: close
Content-Type: application/x-www-form-urlencoded
Content-Length: 100
X-Forwarded-Proto: http
X-Forwarded-For: xx.xx.xx.xx

HTTP/1.1 400 Bad Request
Date: Sat, 17 Jun 2017 22:29:45 GMT
Server: Apache/2.4.25 (FreeBSD) mod_wsgi/4.5.15 Python/2.7
Content-Length: 226
Connection: close
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
</p>
</body></html>
iv=En98cormFMA7NO5e-4qh2Q==&msg=ek_JDRJPqPUvNlztUVH6FTtfVVdHgODWMimBcZklos2XntlMOM1RweBjsp5z-zY=
  • Note: Everything between the HTTP/1.1 400 Bad Request and </html> are server replies, everything else is what the server is receiving. The X-Forwarded* headers were set by HAProxy. As far as I know Squid didn't use these

The really weird thing here is that Apache is responding even before it gets the POST data. The server responds fine to GET requests, and I'm running some other Apache servers behind the same firewall which all seem to work. The only real difference I can think of is that these HTTP requests are pretty much hard-coded to the microcontroller they're coming from, and maybe they aren't quite right to begin with. Any suggestions would be really appreciated. Thanks in advance!

Best Answer

It turns out the problem was the line endings in my HTTP requests. I was sending a mixture of LF and CRLF. RFC2616 requires that HTTP/1.1 only use CRLF, but also states that servers should implement a "tolerance provision". However, Apache >= 2.4.25 defaults to HttpProtocolOptions Strict which returns a HTTP/1.1 400 status to all malformed headers.

FreeBSD has packaged Apache 2.4.25 since December, but I assume Squid was regularizing headers for me because it didn't break then. I guess HAProxy is more laissez-fare, so the change from Squid to HAProxy exposed the server to my old firmware bug.

The right fix would be to re-flash the microcontroller with patched firmware, but that's a hassle. A quick and dirty workaround is to tell Apache to relax. Adding HttpProtocolOptions Unsafe to httpd.conf is a quick "fix" and at least gets the data logging back up and running for now.

Related Topic