Hmnn, how many requests per second (split up on static and dynamic) are we talking about, and on which hardware? Your post doesn't provide specifics, but it sounds a bit strange that IIS should be overwhelmed and still only use 40% CPU.
I would definitely not use VMWare for this. Virtualization is a great technique, and has many good uses, but this is not one. All virtualization has a performance overhead over running on 'bare metal', and this would make you worse off in this scenario.
Go dig into logfiles, and look into Performance Monitor metrics for CPU, network I/O, disk I/O and RAM usage, and see if this points you in any specific direction.
Are there any devices in front of the IIS that could be overloaded? Cheap SOHO firewalls, any HTTP proxies, any "content inspection" firewalls or IDS systems?
The aspx code that you're running, have you looked at it? Does it have any stupid design decisions that could pause the entire computer, such as doing massive disk thrashing or exhausting thread pools?
If the server really is at its limits, and there is no quick win, then the following could be considered:
A cheap solution could be to set up 2 DNS hostnames, i.e. www.companyname.com and static.companyname.com and then set up 2 sites in IIS, one optimized for static file serving, one for dynamic content (aspx). For static files I would generally enable persistent HTTP connections, for dynamic content I would disable it. Optimizations might get you some room to breathe performance-wise, but it may or may not be enough. AFAIK there is no way to guarantee a specific resource allocation between sites in IIS 6.
The optimal way to go with regards to separating static content is to move all static file serving completely off the server, either by dedicating a new server to the task, or even better by using a CDN.
Pricing for low-end CDNs has come down dramatically, with Amazon S3 + Cloudfront, CacheFly and others having a low entry price. Of course adding a CDN means adding another single point of failure to your overall architecture, but CDNs are mostly good at keeping a high uptime, due to their distributed nature. Adding a CDN will potentially make your site feel faster for overseas users.
Best Answer
I agree with MattB that it's likely not because of the logon locally setting.
As a side, if you set the anonymous user to use the app pool identity, you only need to maintain 1 user. That's my preferred setting. Note that you should map 1 site to 1 app pool, or all sites in the app pool need to trust each other. If that's not the case then keep it with the defaults.