IIS 8 – Default SSL Site Breaks SNI

The problem is that SSL negotiation is done before the host headers are sent, so you can only have one SSL cert per IP address. This is why you are getting the invalid certificate on b.mysite.com

there are two ways to handle this:

1. Get a wild card cert for *.mydomain.com
2. Assign a second IP to the server and run b.mysite.com off this second IP with the proper cert.

For the most up-to-date information on Apache and SNI, including additional HTTP-Specific RFCs, please refer to the Apache Wiki

FYsI: "Multiple (different) SSL certificates on one IP" is brought to you by the magic of TLS Upgrading. It works with newer Apache servers (2.2.x) and reasonably recent browsers (don't know versions off the top of my head).

RFC 2817 (upgrading to TLS within HTTP/1.1) has the gory details, but basically it works for a lot of people (if not the majority).
You can reproduce the old funky behavior with openssl's s_client command (or any "old enough" browser) though.

Edit to add: apparently curl can show you what's happening here better than openssl:

SSLv3

mikeg@flexo% curl -v -v -v -3 https://www.yummyskin.com
* About to connect() to www.yummyskin.com port 443 (#0)
*   Trying 69.164.214.79... connected
* Connected to www.yummyskin.com (69.164.214.79) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: /usr/local/share/certs/ca-root-nss.crt
  CApath: none
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using DHE-RSA-AES256-SHA
* Server certificate:
*    subject: serialNumber=wq8O9mhOSp9fY9JcmaJUrFNWWrANURzJ; C=CA; 
              O=staging.bossystem.org; OU=GT07932874;
              OU=See www.rapidssl.com/resources/cps (c)10;
              OU=Domain Control Validated - RapidSSL(R);
              CN=staging.bossystem.org
*    start date: 2010-02-03 18:53:53 GMT
*    expire date: 2011-02-06 13:21:08 GMT
* SSL: certificate subject name 'staging.bossystem.org'
       does not match target host name 'www.yummyskin.com'
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
curl: (51) SSL: certificate subject name 'staging.bossystem.org'
does not match target host name 'www.yummyskin.com'

TLSv1

mikeg@flexo% curl -v -v -v -1 https://www.yummyskin.com
* About to connect() to www.yummyskin.com port 443 (#0)
*   Trying 69.164.214.79... connected
* Connected to www.yummyskin.com (69.164.214.79) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: /usr/local/share/certs/ca-root-nss.crt
  CApath: none
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using DHE-RSA-AES256-SHA
* Server certificate:
*    subject: C=CA; O=www.yummyskin.com; OU=GT13670640;
              OU=See www.rapidssl.com/resources/cps (c)09;
              OU=Domain Control Validated - RapidSSL(R);
              CN=www.yummyskin.com
*    start date: 2009-04-24 15:48:15 GMT
*    expire date: 2010-04-25 15:48:15 GMT
*    common name: www.yummyskin.com (matched)
*    issuer: C=US; O=Equifax Secure Inc.; CN=Equifax Secure Global eBusiness CA-1
*    SSL certificate verify ok.