C# – Web Api + HttpClient: An asynchronous module or handler completed while an asynchronous operation was still pending

asp.netasp.net-web-apicdotnet-httpclient

I'm writing an application that proxies some HTTP requests using the ASP.NET Web API and I am struggling to identify the source of an intermittent error.
It seems like a race condition… but I'm not entirely sure.

Before I go into detail here is the general communication flow of the application:

  • Client makes a HTTP request to Proxy 1.
  • Proxy 1 relays the contents of the HTTP request to Proxy 2
  • Proxy 2 relays the contents of the HTTP request to the Target Web Application
  • Target Web App responds to the HTTP request and the response is streamed (chunked transfer) to Proxy 2
  • Proxy 2 returns the response to Proxy 1 which in turn responds to the original calling Client.

The Proxy applications are written in ASP.NET Web API RTM using .NET 4.5.
The code to perform the relay looks like so: 

//Controller entry point.
public HttpResponseMessage Post()
{
    using (var client = new HttpClient())
    {
        var request = BuildRelayHttpRequest(this.Request);

        //HttpCompletionOption.ResponseHeadersRead - so that I can start streaming the response as soon
        //As it begins to filter in.
        var relayResult = client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead).Result;

        var returnMessage = BuildResponse(relayResult);
        return returnMessage;
    }
}

private static HttpRequestMessage BuildRelayHttpRequest(HttpRequestMessage incomingRequest)
{
    var requestUri = BuildRequestUri();
    var relayRequest = new HttpRequestMessage(incomingRequest.Method, requestUri);
    if (incomingRequest.Method != HttpMethod.Get && incomingRequest.Content != null)
    {
       relayRequest.Content = incomingRequest.Content;
    }

    //Copies all safe HTTP headers (mainly content) to the relay request
    CopyHeaders(relayRequest, incomingRequest);
    return relayRequest;
}

private static HttpRequestMessage BuildResponse(HttpResponseMessage responseMessage)
{
    var returnMessage = Request.CreateResponse(responseMessage.StatusCode);
    returnMessage.ReasonPhrase = responseMessage.ReasonPhrase;
    returnMessage.Content = CopyContentStream(responseMessage);

    //Copies all safe HTTP headers (mainly content) to the response
    CopyHeaders(returnMessage, responseMessage);
}

private static PushStreamContent CopyContentStream(HttpResponseMessage sourceContent)
{
    var content = new PushStreamContent(async (stream, context, transport) =>
            await sourceContent.Content.ReadAsStreamAsync()
                            .ContinueWith(t1 => t1.Result.CopyToAsync(stream)
                                .ContinueWith(t2 => stream.Dispose())));
    return content;
}

The error that occurs intermittently is:

An asynchronous module or handler completed while an asynchronous operation was still pending.

This error usually occurs on the first few requests to the proxy applications after which the error is not seen again.

Visual Studio never catches the Exception when thrown.
But the error can be caught in the Global.asax Application_Error event.
Unfortunately the Exception has no Stack Trace.

The proxy applications are hosted in Azure Web Roles.

Any help identifying the culprit would be appreciated.

Best Answer

Your problem is a subtle one: the async lambda you're passing to PushStreamContent is being interpreted as an async void (because the PushStreamContent constructor only takes Actions as parameters). So there's a race condition between your module/handler completing and the completion of that async void lambda.

PostStreamContent detects the stream closing and treats that as the end of its Task (completing the module/handler), so you just need to be sure there's no async void methods that could still run after the stream is closed. async Task methods are OK, so this should fix it:

private static PushStreamContent CopyContentStream(HttpResponseMessage sourceContent)
{
  Func<Stream, Task> copyStreamAsync = async stream =>
  {
    using (stream)
    using (var sourceStream = await sourceContent.Content.ReadAsStreamAsync())
    {
      await sourceStream.CopyToAsync(stream);
    }
  };
  var content = new PushStreamContent(stream => { var _ = copyStreamAsync(stream); });
  return content;
}

If you want your proxies to scale a bit better, I also recommend getting rid of all the Result calls:

//Controller entry point.
public async Task<HttpResponseMessage> PostAsync()
{
  using (var client = new HttpClient())
  {
    var request = BuildRelayHttpRequest(this.Request);

    //HttpCompletionOption.ResponseHeadersRead - so that I can start streaming the response as soon
    //As it begins to filter in.
    var relayResult = await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);

    var returnMessage = BuildResponse(relayResult);
    return returnMessage;
  }
}

Your former code would block one thread for each request (until the headers are received); by using async all the way up to your controller level, you won't block a thread during that time.

Related Topic