You may use the signal package if you are running on UNIX:
In [1]: import signal
# Register an handler for the timeout
In [2]: def handler(signum, frame):
...: print("Forever is over!")
...: raise Exception("end of time")
...:
# This function *may* run for an indetermined time...
In [3]: def loop_forever():
...: import time
...: while 1:
...: print("sec")
...: time.sleep(1)
...:
...:
# Register the signal function handler
In [4]: signal.signal(signal.SIGALRM, handler)
Out[4]: 0
# Define a timeout for your function
In [5]: signal.alarm(10)
Out[5]: 0
In [6]: try:
...: loop_forever()
...: except Exception, exc:
...: print(exc)
....:
sec
sec
sec
sec
sec
sec
sec
sec
Forever is over!
end of time
# Cancel the timer if the function returned before timeout
# (ok, mine won't but yours maybe will :)
In [7]: signal.alarm(0)
Out[7]: 0
10 seconds after the call signal.alarm(10)
, the handler is called. This raises an exception that you can intercept from the regular Python code.
This module doesn't play well with threads (but then, who does?)
Note that since we raise an exception when timeout happens, it may end up caught and ignored inside the function, for example of one such function:
def loop_forever():
while 1:
print('sec')
try:
time.sleep(10)
except:
continue
Took the code above and fixed because it throws Internal Server Error 500. There are some problems with \r\n badly positioned and spaces etc. Applied the refactoring with memory stream, writing directly to the request stream. Here is the result:
public static void HttpUploadFile(string url, string file, string paramName, string contentType, NameValueCollection nvc) {
log.Debug(string.Format("Uploading {0} to {1}", file, url));
string boundary = "---------------------------" + DateTime.Now.Ticks.ToString("x");
byte[] boundarybytes = System.Text.Encoding.ASCII.GetBytes("\r\n--" + boundary + "\r\n");
HttpWebRequest wr = (HttpWebRequest)WebRequest.Create(url);
wr.ContentType = "multipart/form-data; boundary=" + boundary;
wr.Method = "POST";
wr.KeepAlive = true;
wr.Credentials = System.Net.CredentialCache.DefaultCredentials;
Stream rs = wr.GetRequestStream();
string formdataTemplate = "Content-Disposition: form-data; name=\"{0}\"\r\n\r\n{1}";
foreach (string key in nvc.Keys)
{
rs.Write(boundarybytes, 0, boundarybytes.Length);
string formitem = string.Format(formdataTemplate, key, nvc[key]);
byte[] formitembytes = System.Text.Encoding.UTF8.GetBytes(formitem);
rs.Write(formitembytes, 0, formitembytes.Length);
}
rs.Write(boundarybytes, 0, boundarybytes.Length);
string headerTemplate = "Content-Disposition: form-data; name=\"{0}\"; filename=\"{1}\"\r\nContent-Type: {2}\r\n\r\n";
string header = string.Format(headerTemplate, paramName, file, contentType);
byte[] headerbytes = System.Text.Encoding.UTF8.GetBytes(header);
rs.Write(headerbytes, 0, headerbytes.Length);
FileStream fileStream = new FileStream(file, FileMode.Open, FileAccess.Read);
byte[] buffer = new byte[4096];
int bytesRead = 0;
while ((bytesRead = fileStream.Read(buffer, 0, buffer.Length)) != 0) {
rs.Write(buffer, 0, bytesRead);
}
fileStream.Close();
byte[] trailer = System.Text.Encoding.ASCII.GetBytes("\r\n--" + boundary + "--\r\n");
rs.Write(trailer, 0, trailer.Length);
rs.Close();
WebResponse wresp = null;
try {
wresp = wr.GetResponse();
Stream stream2 = wresp.GetResponseStream();
StreamReader reader2 = new StreamReader(stream2);
log.Debug(string.Format("File uploaded, server response is: {0}", reader2.ReadToEnd()));
} catch(Exception ex) {
log.Error("Error uploading file", ex);
if(wresp != null) {
wresp.Close();
wresp = null;
}
} finally {
wr = null;
}
}
and sample usage:
NameValueCollection nvc = new NameValueCollection();
nvc.Add("id", "TTR");
nvc.Add("btn-submit-photo", "Upload");
HttpUploadFile("http://your.server.com/upload",
@"C:\test\test.jpg", "file", "image/jpeg", nvc);
It could be extended to handle multiple files or just call it multiple times for each file. However it suits your needs.
Best Answer
I believe that the problem is that the
WebRequest
measures the time only after the request is actually made. If you submit multiple requests to the same address then theServicePointManager
will throttle your requests and only actually submit as many concurrent connections as the value of the correspondingServicePoint.ConnectionLimit
which by default gets the value fromServicePointManager.DefaultConnectionLimit
. Application CLR host sets this to 2, ASP host to 10. So if you have a multithreaded application that submits multiple requests to the same host only two are actually placed on the wire, the rest are queued up.I have not researched this to a conclusive evidence whether this is what really happens, but on a similar project I had things were horrible until I removed the
ServicePoint
limitation.Another factor to consider is the DNS lookup time. Again, is my belief not backed by hard evidence, but I think the
WebRequest
does not count the DNS lookup time against the request timeout. DNS lookup time can show up as very big time factor on some deployments.And yes, you must code your app around the
WebRequest.BeginGetRequestStream
(forPOST
s with content) andWebRequest.BeginGetResponse
(forGET
s andPOSTS
s). Synchronous calls will not scale (I won't enter into details why, but that I do have hard evidence for). Anyway, theServicePoint
issue is orthogonal to this: the queueing behavior happens with async calls too.