C# – HttpClient throwing “An error occurred while sending the request.”

asp.net-web-apiasp.net-web-api2chttpclientrest

I have three layer application architecture.

My Client –> My service A (REST hosted in IIS) –> Other Team's service X (REST).

Service A is ASP.Net 4.6.1 framework, not ASP.Net Core.

Client is communicating to A with HttpClient and A is communicating to X with HttpClient.

Client is firing almost 2500 calls to my service to A and to X.

Out of 2500 calls service A randomly (may be 10 calls) fails with below exception. Its not reproducible. 

System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> 
System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a 
receive. ---> System.IO.IOException: Unable to read data from the transport connection: An     
established connection was aborted by the software in your host machine. ---> 
System.Net.Sockets.SocketException: An established connection was aborted by the software in your 
host machine
at System.Net.Sockets.Socket.BeginReceive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags 
socketFlags, AsyncCallback callback, Object state)
at System.Net.Sockets.NetworkStream.BeginRead(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback 
callback, Object state)
 --- End of inner exception stack trace ---
at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult)
at System.Net.TlsStream.EndRead(IAsyncResult asyncResult)
at System.Net.Connection.ReadCallback(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
--- End of inner exception stack trace --

Here is my service A call. A in IIS calls below code block and its called by each request. X is taking user credentials and returning data based on user, so we are not sharing HttpClient between calls.

var user = (System.Security.Principal.WindowsIdentity)HttpContext.Current.User.Identity;
            System.Security.Principal.WindowsIdentity.RunImpersonated(user.AccessToken, () =>
      {
        static HttpClient Client = new HttpClient();
        static string CallX(string[] args)
        {
            HttpClientHandler handler = new HttpClientHandler
            {
                UseDefaultCredentials = true
            };    

            Client = new HttpClient(handler)
            {
                BaseAddress = new Uri("http://XserviceUrl/api/")
            };
            Client.Timeout = TimeSpan.FromSeconds(600);    
            var result = Client.PostAsync("Fake X controller"
                , new StringContent(JsonConvert.SerializeObject(args)
                , Encoding.UTF8, "application/json")).Result;    

            result.EnsureSuccessStatusCode();

            var json = result.Content.ReadAsStringAsync().Result;
            return DosomethingWithResult(json);    
        }
    });

Things I tried:

Some SO post suggested might be timeout issue. So I added 600 seconds in Client and in Service A. I also changed IIS request timeout from default 2 minutes to 10 (600 seconds).

Best Answer

After some digging I resolved the issue. When A was sending the request to X, A was setting the Connection: keep-alive and the X was responding with Connection: Close property in header.

So after some calls A was exhaust with opened tcp connection and randomly it was throwing the error.

(Fiddler helped me to figure this out)

So all we had to do was setting ConnectionClose property of HttpClient

_client.DefaultRequestHeaders.ConnectionClose = true;
Related Topic