Nginx reverse stream proxy with multiple ports to the same server

I'm trying to use nginx as a reverse proxy to two different servers. The servers require the use of client-side certificates for authentication, which means nginx is configured as a stream proxy leveraging the map $ssl_preread_server_name for SNI inspection to send to the correct server.

This works great for the pair of servers it's hosting now. Both listen on 443 but provide completely different services, but the redirection via SNI is working great.

The trouble is that one of the servers also uses port 9997 for communication (TLS) and we need to add more of these into the mix. Currently we're just hard-coding the traffic in nginx to the one server that uses 9997. This wont work as we move forward and have additional servers hosting content on 9997

How can I configure nginx to stream both 443 and 9997 to the box that needs those communications, while also continuing to send 443 to the other server when needed?

It needs to be dynamic so that the traffic is sent to the RIGHT server.

Here's the config that works now (some info redacted):

#user  nobody;
worker_processes  1;

error_log   /var/log/nginx/error.log;
#pid        logs/nginx.pid;


events {
    worker_connections  1024;
}

stream {

    map $ssl_preread_server_name $upstream {
        server1.domain.com server1;
        server2.domain.com server2;
    }

    server {
        listen 443;
        proxy_pass $upstream;

        ssl_preread on;
    }

    server {
        listen 9997;
        proxy_pass 1.2.3.4:9997;
    }


    upstream server1 {
        server 1.2.3.4:443;
    }

    upstream server2 {
        server 1.2.3.5:443;
    }

}