NGINX Proxy pass to WebSocket and PHP not working with SSL

I'm trying to route requests in Nginx in the following way:

1. Requests made to / should go to a PHP script (proxy.php, which itself is a proxy)
2. Requests made to /websocket should be proxied to http://localhost:4000/websocket
3. All other requests should be proxied to http://localhost:4000/

I could get 2. and 3. to work using the following config:

server {
    listen 443 ssl;

    server_name proxy.domain.com;

    ssl_certificate /etc/nginx/ssl/proxy.domain.com/468446/server.crt;
    ssl_certificate_key /etc/nginx/ssl/proxy.domain.com/468446/server.key;

    location = /websocket/ {
        proxy_pass http://127.0.0.1:4000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }

    location / {
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   Host      $http_host;
        proxy_pass         http://127.0.0.1:4000;
    }

}

Then I tried to figure out a way to add 1. and came up with the following:

server {
    listen 443 ssl;

    server_name proxy.domain.com;

    root /var/www/dowmain;

    ssl_certificate /etc/nginx/ssl/proxy.domain.com/468446/server.crt;
    ssl_certificate_key /etc/nginx/ssl/proxy.domain.com/468446/server.key;

    location = /websocket/ {
        proxy_pass http://127.0.0.1:4000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }

    location / {
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   Host      $http_host;
        proxy_pass         http://127.0.0.1:4000;
    }

    location = / {
        include fastcgi_params;
        fastcgi_param   SCRIPT_FILENAME  $document_root/proxy.php;
        fastcgi_pass unix:/var/run/php/php7.2-fpm.sock;
    }

}

However, with this configuration, 2. does not work anymore. Requests made to /websocket are processed by the PHP script. Seems like the /websocket location block is not working anymore.

Interestingly, if I switch the config to http://, everything works fine.

Any idea what I'm doing wrong?

UPDATE

I think I can rule out the location /websocket { ... } as the source of the issue, because if I replace the PHP config stuff in the location = / { ... } block with the rules from the location / { ... } block, it works fine (but is not what I need). So I suspect PHP, that messes it all up.

UPDATE II

It even works on my local machine with certificates from mkcert, with the exact same config.

So the only difference is the Let's Encrypt certificate vs the local one. Even the Nginx and PHP Versions are the basically the same (PHP 7.2.7, Nginx 1.15.1 on my local machine vs. PHP 7.2.13, Nginx 1.15.6)