Nginx – Direct Python server access with SSH port forwarding fails, but succeeds with intermediate nginx

nginxport-forwardingsshssh-tunnelweb-server

I'm using SSH port forwarding – as marvelously illustrated and elaborated here – to access my server.

SSH port forwarding

I've spent more time that I'd like to admit, troubleshooting why access to my Python web server 404s. Full length question for completion here.

My issue is, that if I use port forwarding to directly forward my requests to the port, where the Python server is running (e.g. 8282), all requests 404.

The Python server, for simplicity's sake (serving an index.html, containing the string hello):

python -m http.server 8282

The SSH port forwarding from the host, where I want to gain access:

ssh -L 8181:<farawayhost-ip>:8282 <sshuser>@<remotehost-ip>

For the subsequent requests on the host with curl, I've defined the environment variable http_proxy:

export http_proxy=127.0.0.1:8181

Apparently the requests are not correct, as they're prepended by the IP address. For comparison here's the Python http.server log, first accessing it locally from the server, then from the external host, using SSH port forwarding.

Serving HTTP on 0.0.0.0 port 8282 ...
<server-public-ip> - - [<timestamp>] "GET /index.html HTTP/1.1" 200 -
<proxy-ip> - - [<timestamp>] code 404, message File not found
<proxy-ip> - - [<timestamp>] "GET http://<server-public-ip>:8282/index.html HTTP/1.1" 404 - 

I was able to gain access to the Python web service from the external host, by placing an Nginx proxy between the Python web server and the external host.

Nginx is installed on the same server as the Python web service is and simply listens on port 80 and proxies to localhost:8282.

Nginx configuration:

server {
    listen      80 default_server;

    location / {
                proxy_pass http://127.0.0.1:8282;
    }

On the external host, the SSH port forwarding is configured to port 80, instead of 8282.

ssh -L 8181:<farawayhost-ip>:80 <sshuser>@<remotehost-ip>
export http_proxy=127.0.0.1:8181 # proxy for curl

Access to the web service now succeeds.

Nginx log:

<remotehost-ip> - - [<timestamp>] "GET http://<server-ip>/ HTTP/1.1" 200 6 "-" "<user-agent>"

Python http.server log:

127.0.0.1 - - [<timestamp>] "GET / HTTP/1.0" 200 -

To my question:

My initial problem is solved, but I want to understand why.

How do these requests differ, why does the direct access to the Python server with SSH port forwarding fail and the access via the Nginx proxy succeeds?

Best Answer

Your problem is that SSH tunnel is not a HTTP proxy (while nginx is) so setting http_proxy to point to SSH tunnel end point causes curl to send HTTP requests in format suitable for HTTP proxy while SSH tunnel sends these requests directly to your python app without any special processing as HTTP proxy would do. With SSH tunnel you need to remove http_proxy variable and access your app directly with curl http://localhost:8181/.