A client (developed by a third party) is sending a request with headers like foo.meta-digest
(note the "dot" in the header name).
My nginx reverse proxy is removing these headers from the request even if I added ignore_invalid_headers off;
(see http://nginx.org/r/ignore_invalid_headers) to the server { ... }
block for this vhost:
upstream backend {
keepalive 128;
keepalive_requests 1000;
keepalive_timeout 120s;
server 127.0.0.1:32000 max_fails=0 fail_timeout=10;
}
server {
listen 443 ssl http2;
server_name foo.com;
root /var/www/empty;
ssl_certificate /etc/nginx/certs/foo.com.crt;
ssl_certificate_key /etc/nginx/certs/foo.com.key;
ignore_invalid_headers off;
location / {
proxy_pass http://backend;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Connection "";
}
}
I cannot reproduce this problem with any other client, except for this specific one and I don't have access to the source code of this client, it's proprietary software.
If I send the very same request using curl
or any other web client, all the headers, including the ones with a "dot" in the name, are passed to the backend as they should.
I tried even running a mitm attack against the nginx host and intercepted the network traffic from the client. The client is definitely sending the foo.meta-digest
header as expected.
Somehow nginx is filtering the header, but only for this specific client.
What else can I do to debug this problem?
Best Answer
Just had a similar problem, and I only fixed it by adding the
ignore_invalid_headers off
also to the default serverThe client in my case was doing two weird (and bad-practice) things:
This is used by the client to give (in cleartext) the domain name it is connecting to, so that the server is able to use the right certificate for the right vhost.
Without this, the client always gets the certificate for the default vhost.
...which the client had pinned, so that it was accepted even though it was invalid for the domain.
Note that this only affects the TLS certs you will get from the server! In theory (see point 2 later) nginx will still use the
Host:
header to determine which is the correct vhost, and which conf to use.Note that unless you are pinning the certificate in the client application, and unless you are sure about the certificate, this is a very unsafe thing to do. You would be opening yourself to MITM attacks.
Also, SNI was developed like 12 years ago, and it is necessary for virtual hosting.
Host:
Header to know which vhost configuration to apply.As you can guess since I told you to add the
ignore_invalid_headers
in the default server, this does not work on a special case:RFC7230, section 5.4 says that the
Host
header SHOULD be the first thing you send.In my case (and proably yours) it wasn't.
The headers before the
Host
Headers are thus parsed as in the default server configuration, then nginx finds theHost
and switches to the correct vhost.Weird, almost non-standard, definitely bad practice, but not technically wrong.
Hope it helps