Web-server – HTTP Hostname to IP Address Routing

This sounds weird. 40 VMs for 40 sites?

IIS can host multiple web sites per machine. Can you not have, say, 3 servers, and use IIS host headers to host 15 sites a piece? Or does each web application instance have to be that dedicated?

Regardless you will need to do some kind of reverse proxy if you want to share a single IP across 40 boxes. You will need to set up each host name in the reverse proxy so it knows to which box to send that host header...

It's random and unpredictable, and if you care, you are doing something horribly wrong.

Because the server uses a naive implementation of HTTP, it only inspects the first packet of an HTTP request, and misses the arguments sent in the second packet. This causes the request to fail.

I can't figure out what you're trying to say there. If it's actually looking at packets, it's an implementation of TCP, not HTTP (since the packets are packets according to the TCP specification). A sensible HTTP implementation would never be looking at packets at all.

If you really mean the HTTP server looks at the packets, then it has to reconstruct the TCP data stream and then follow the HTTP protocol. The end of an HTTP request is marked with two <CRLF>s in the TCP byte stream. Any other method of detecting the end of an HTTP request is broken.

Since the first packet does not contain an HTTP end-of-request sequence, it clearly cannot contain the entire request.

An HTTP server must follow the HTTP specification, or it's not an HTTP server. The HTTP specification only specifies what bytes go in the TCP data stream. It says nothing about segmentation, which is good because in practice applications would have no way to control that. It specifies how the end of a request is identified, and code that doesn't comply with that specification is not an HTTP server. It implements an HTTP-like protocol that may or may not be compatible with HTTP.