Nginx – Single front NGINX to proxy pass SSL to sites with their own certificates

Current setup:

1. Front NGINX proxy passes 80 and 443 to local VM
2. Local VM's NGINX redirects everything to HTTPS (if request is HTTP/80)
3. Local VM's NGINX listens to 443 and proxy passes to the upstream backend

Front NGINX:

server {
        listen 80;
        server_name domain1.com;
        location / {
                proxy_set_header X-Real-IP  $remote_addr;
                proxy_set_header X-Forwarded-For $remote_addr;
                proxy_set_header Host $host;
                proxy_pass http://192.168.0.24;
         }
}

Domain1.com's NGINX:

upstream unicorn_domain1.com {
  server unix:/tmp/unicorn.domain1.com.sock fail_timeout=0;
}

server {
    listen 80;
    server_name domain1.com;

    ### REDIRECT EVERYTHING TO HTTPS
    return 301 https://domain1.com$request_uri;
}

server {
    listen 443 ssl spdy;
    listen [::]:443 ssl spdy;
    server_name domain1.com localhost;
    root /var/www/domain1.com/production/current/public;
    spdy_headers_comp 1;

    ### ALL THE CERT_ CONF HERE

    try_files $uri/index.html $uri @unicorn;
    location @unicorn {
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Host $http_host;
      proxy_set_header X-Forwarded-Proto https;
      proxy_redirect off;
      proxy_pass http://unicorn_domain1.com;
    }

    ### Omitted    
}

Problem:
Domain1 is a local network VM. This is currently the VM that has port 443 open to it. That means that only one site at the moment can listen on 443. That won't work when adding a second site. So we need the "front NGINX" to listen on port 443, and just proxy pass to the VMs, that use their own certificates.

What I've tried:
A seemingly normal proxy pass config like this:

server {
    listen 80;
    listen 443 ssl; # Ensure your certificate is for nginx.domain.com;
    server_name domain1.com;

    location / {
        proxy_pass https://192.168.0.24;
    }
}

But this got us an Connection refused error (Chrome message). What am I missing? There shouldn't be any changes to Domain1.com's NGINX imo.