IIS 7.0 503 errors with generic handler (.ashx) implementing IHttpAsyncHandler

I'm running into some performance issues using a generic handler that implements IHttpAsyncHandler. At its simplest, the handler receives a GET request, and 20 seconds later ends the response after writing '< timeout / >' to the response.

When hammering the .ashx with 10000-20000 simultaneous requests, it fails with 503 server unavailable after precisely 5000 requests. When switching to synchronous mode, and ending the request immediately, the problem goes away.

I've tinkered with a number of settings, yet the only thing I've managed to acheive is lower the request threshold at which this error occurs.

Here's a rundown of the settings I've toyed with:

machine.config:

<configuration>
    ...
    <system.web>
        ...
        <processModel enable="true" requestQueueLimit="10000"/>
        ...

web.config:

<configuration>
    ...
    <system.web>
        ...
        <httpRuntime enable="true" appRequestQueueLimit="10000"/>
        ...

IIS Manager > ApplicationPools > Advanced Settings

Queue Length : 65535

Although I can't be sure, it seems like these settings work good and fine if the requests are synchronous, but when async, I can't seem to get beyond exactly 5000 requests before the server starts telling me to go away. If I set things lower (can't remember exactly which setting that would be from the above, but I've tried them all), then the 503 count goes up accordingly, but I can never stop it happening beyond 5000 when under serious load.

It seems that there a a number of settings scattered in a myriad of places that might affect this, but the 5000 seems fairly set in stone. I see here that appRequestQueueLimit cannot exceed 5000, but can find no further info about this, and wonder if this is misinformation.

Is there any kind of "flood-control" setting in IIS that might be limiting a single host to no more than 5000 requests? How can I get IIS to handle more that 5000 concurrent asynchronous requests?

Edit2: Are there any counters or other indicators of which limit might be being exceeded, and how would I investigate further?

Edit: Here's the loadgenerator code:

using System;
using System.Net;
using System.Threading;

namespace HammerTime
{
    class Program
    {
        private static int counter = 0;
        static void Main(string[] args)
        {
            var limit = 5000;
            ServicePointManager.DefaultConnectionLimit=limit;
            for (int i = 0; i < limit;++i )
            {
                StartWebRequest(i.ToString());

            }
            Console.ReadLine();
        }

        private static void StartWebRequest(string channelId)
        {
            string uri = "http://spender2008/test/Test.ashx?channel="+channelId;
            HttpWebRequest request = (HttpWebRequest) WebRequest.Create(uri);
            request.BeginGetResponse(responseHandler, request);
        }

        private static void responseHandler(IAsyncResult ar)
        {
            try
            {
                HttpWebRequest state = (HttpWebRequest)ar.AsyncState;
                HttpWebResponse response = (HttpWebResponse)state.EndGetResponse(ar);

            }
            catch(Exception e)
            {
                Console.WriteLine(e.Message);
            }
            finally
            {
                Console.WriteLine(Interlocked.Increment(ref counter));
            }

        }
    }
}