C# – HttpWebRequest.Create returns (500) Internal Server Error

asp.netasp.net-mvcchttpwebrequest

It works fine to browse this page : http://www.litteraturmagazinet.se/arga-bibliotekstanten/boklogg/favorit-i-repris-9560835 in a regular browser(for example Chrome).

But when I use the following code to fetch the website I get Internal Server Error (500)?

This is the code I use (and it works great on all other webpages I have tried) :

HttpWebRequest request;
WebResponse webresponse;
request = (HttpWebRequest)HttpWebRequest.Create(url);
webresponse = (HttpWebResponse)request.GetResponse();

The exception is thrown in GetResponse.

I have found for example this : HttpWebRequest.GetResponse() returns error 500 Internal Server Error, but I do not understand it? Why does not request.GetResponse work with this specific webpage? And How do I know what to put in the header(It would be great if it dident hade to be updated later on to diffrent versions)?

Best Answer

I tried your url with wfetch with no headers and i get a 500 as well.

you have to set a proper user-agent in the headers of your request.

HttpWebRequest myHttpWebRequest=(HttpWebRequest)WebRequest.Create("http://www.contoso.com");
myHttpWebRequest.UserAgent=".NET Framework Test Client";

Related Solutions

C# HttpWebRequest times out after two server 500 errors

I suspect this is the problem:

this.webResponse = (HttpWebResponse)httpRequest.GetResponse();
Stream responseStream = webResponse.GetResponseStream();
StreamReader responseReader = new StreamReader(responseStream, Encoding.UTF8);
String responseString = responseReader.ReadToEnd();

You're not disposing of any of these disposable objects. That means the connection to the web server will be maintained until finalization, so the connection pool (of two connections per server) won't allow any other requests.

I suggest that:

You move the GetResponse call out of the using statement for the request stream
You remove the explicit calls to Flush and Close for the request stream (disposing of it is good enough)
You make webResponse and webRequest local variables instead of instance variables unless you really need them later, in which case you should dispose of them in your Dispose method.
You use using statements for the WebResponse, Stream, and StreamReader. (The last isn't strictly necessary, as disposing of the stream is good enough.)
You make GetPages not implement IDisposable unless you really need it to.

C# – How to get error information when HttpWebRequest.GetResponse() fails

Is this possible using HttpWebRequest and HttpWebResponse?

You could have your web server simply catch and write the exception text into the body of the response, then set status code to 500. Now the client would throw an exception when it encounters a 500 error but you could read the response stream and fetch the message of the exception.

So you could catch a WebException which is what will be thrown if a non 200 status code is returned from the server and read its body:

catch (WebException ex)
{
    using (var stream = ex.Response.GetResponseStream())
    using (var reader = new StreamReader(stream))
    {
        Console.WriteLine(reader.ReadToEnd());
    }
}
catch (Exception ex)
{
    // Something more serious happened
    // like for example you don't have network access
    // we cannot talk about a server exception here as
    // the server probably was never reached
}

Best Answer

Related Solutions

C# HttpWebRequest times out after two server 500 errors

C# – How to get error information when HttpWebRequest.GetResponse() fails

Related Topic