Nginx – How to prevent HAProxy from dropping HTTP headers with underscores

haproxyhttp-headersnginxPROXY

We have an API backend server that requires certain HTTP headers which contain underscores. I know that this is not best practice and headers should use hyphens but I cannot change this.

We were using nginx as a proxy server with the option underscores_in_headers on. That causes nginx to not drop these headers.

Now we want to switch to HAProxy and as nginx before the configuration change it drops headers with underscores. Is there a way to prevent HAProxy from dropping headers with underscores?

Best Answer

Have you actually tested that HAProxy drops headers with underscores?

From the source code it seems like it doesn't. In this answer I'll try to explain:

  • why underscores in headers are allowed in HTTP
  • why Nginx drops them by default and
  • why I believe HAProxy wouldn't do so.

HTTP/1.1 & Nginx

According to the HTTP/1.1 specification in RFC 7230 3.2.6, there's nothing wrong with underscores (_) in header fields; it's only uncommon.

Field Value Components

Most HTTP header field values are defined using common syntax components (token, quoted-string, and comment) separated by whitespace or specific delimiting characters. Delimiters are chosen from the set of US-ASCII visual characters not allowed in a token (DQUOTE and (),/:;<=>?@[\]{}).

token         = 1*tchar

tchar         = "!" / "#" / "$" / "%" / "&" / "'" / "*"
              / "+" / "-" / "." / "^" / "_" / "`" / "|" / "~"
              / DIGIT / ALPHA
              ; any VCHAR, except delimiters

Nginx Pitfalls and Common Mistakes: Missing (disappearing) HTTP Headers explains why they are still silently dropped by default:

If you do not explicitly set underscores_in_headers on;, NGINX will silently drop HTTP headers with underscores (which are perfectly valid according to the HTTP standard). This is done in order to prevent ambiguities when mapping headers to CGI variables as both dashes and underscores are mapped to underscores during that process.

HAProxy

Unfortunately, HAProxy does not have such a setting. If you search underscore through the HAProxy Configuration Manual, it's only mentioned in the contexts of environment variables, node names (like in DNS names), proxy names and ACL names. Not mentioned in HTTP request chapter.

If HAProxy does drop the headers with underscores, there's nothing you can do from the configuration and you would have to either stick to the Nginx or modify HAProxy's source code.

However, I tried to find where HAProxy drops there headers and couldn't find such a thing from proto_http.c's functions void capture_headers() and void http_msg_analyzer().

Also, proto_http.c lists all ASCII characters by type, and (),/:;<=>?@[\]{} are listed as HTTP_FLG_SEP while underscore is listed as a normal token:

495 /* It is about twice as fast on recent architectures to lookup a byte in a
496  * table than to perform a boolean AND or OR between two tests. Refer to
497  * RFC2616/RFC5234/RFC7230 for those chars. A token is any ASCII char that is
498  * neither a separator nor a CTL char. An http ver_token is any ASCII which can
499  * be found in an HTTP version, which includes 'H', 'T', 'P', '/', '.' and any
500  * digit. Note: please do not overwrite values in assignment since gcc-2.95
501  * will not handle them correctly. It's worth noting that chars 128..255 are
502  * nothing, not even control chars.
503  */
504 const unsigned char http_char_classes[256] = {
505     [  0] = HTTP_FLG_CTL,

545     ['('] = HTTP_FLG_SEP,
546     [')'] = HTTP_FLG_SEP,
547     ['*'] = HTTP_FLG_TOK,
548     ['+'] = HTTP_FLG_TOK,
549     [','] = HTTP_FLG_SEP,
550     ['-'] = HTTP_FLG_TOK,
551     ['.'] = HTTP_FLG_TOK | HTTP_FLG_VER,

570     ['A'] = HTTP_FLG_TOK,
571     ['B'] = HTTP_FLG_TOK,
572     ['C'] = HTTP_FLG_TOK,

600     ['_'] = HTTP_FLG_TOK,

Here, _ is just a normal HTTP_FLG_TOK just like A, B and C; it shouldn't cause anything special.

Related Topic