R – Async urlfetch on App Engine

asynchronousgoogle-app-enginegoogle-cloud-datastoreurlfetch

My app needs to do many datastore operations on each request. I'd like to run them in parallel to get better response times.

For datastore updates I'm doing batch puts so they all happen asynchronously which saves many milliseconds. App Engine allows up to 500 entities to be updated in parallel.

But I haven't found a built-in function that allows datastore fetches of different kinds to execute in parallel.

Since App Engine does allow urlfetch calls to run asynchronously, I created a getter URL for each kind which returns the query results as JSON-formatted text. Now my app can do async urlfetch calls to these URLs which could parallelize the datastore fetches.

This technique works well with small numbers of parallel requests, but App Engine throws errors when attempting to run more than 5 or 10 of these urlfetch calls at the same time.

I'm only testing now, so each urlfetch is the identical query; since they work fine in small volumes but start failing with more than a handful of simultaneous requests, I'm thinking it must have something to do with the async urlfetch calls.

My questions are:

Is there a limit to the number of urlfetch.create_rpc() calls that can run asynchronously?
The synchronous urlfecth.fetch() function has a 'deadline' parameter that will allow the function to wait up to 10 seconds for a response before failing. Is there any way to tell urlfetch.create_rpc() how long to wait for a response?
What do the errors shown below mean?
Is there a better server-side technique to run datastore fetches of different kinds in parallel?

File "/base/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 501, in get_result
return self.__get_result_hook(self)
File "/base/python_lib/versions/1/google/appengine/api/urlfetch.py", line 331, in _get_fetch_result
raise DownloadError(str(err))
InterruptedError: ('The Wait() request was interrupted by an exception from another callback:', DownloadError('ApplicationError: 5 ',))

Best Answer

Since App Engine allows async urlfetch calls but does not allow async datastore gets, I was trying to use urlfetch RPCs to retrieve from the datastore in parallel.

The lack of async datastore gets is an acknowledged issue:

http://code.google.com/p/googleappengine/issues/detail?id=1889

And there's now a third-party tool that allows async queries:

http://code.google.com/p/asynctools/

"asynctools is a library allowing you to execute Google App Engine API calls in parallel. API calls can be mixed together and queued up and then all are kicked off in parallel."

This is exactly what I was looking for.

Related Solutions

R – Why is urlfetch throwing Download Errors when calling some Google services

Without having a look at your code I will be guessing but since the URL that you are following is a going to redirect are you allowing redirects in your call? Note the follow_redirects=True

e.g. urlfetch.fetch(url, payload=None, method=GET, headers={}, allow_truncated=False, follow_redirects=True, deadline=None)

http://code.google.com/appengine/docs/python/urlfetch/fetchfunction.html

C# – How and when to use ‘async’ and ‘await’

When using async and await the compiler generates a state machine in the background.

Here's an example on which I hope I can explain some of the high-level details that are going on:

public async Task MyMethodAsync()
{
    Task<int> longRunningTask = LongRunningOperationAsync();
    // independent work which doesn't need the result of LongRunningOperationAsync can be done here

    //and now we call await on the task 
    int result = await longRunningTask;
    //use the result 
    Console.WriteLine(result);
}

public async Task<int> LongRunningOperationAsync() // assume we return an int from this long running operation 
{
    await Task.Delay(1000); // 1 second delay
    return 1;
}

OK, so what happens here:

Task<int> longRunningTask = LongRunningOperationAsync(); starts executing LongRunningOperation
Independent work is done on let's assume the Main Thread (Thread ID = 1) then await longRunningTask is reached.

Now, if the longRunningTask hasn't finished and it is still running, MyMethodAsync() will return to its calling method, thus the main thread doesn't get blocked. When the longRunningTask is done then a thread from the ThreadPool (can be any thread) will return to MyMethodAsync() in its previous context and continue execution (in this case printing the result to the console).

A second case would be that the longRunningTask has already finished its execution and the result is available. When reaching the await longRunningTask we already have the result so the code will continue executing on the very same thread. (in this case printing result to console). Of course this is not the case for the above example, where there's a Task.Delay(1000) involved.

Best Answer

Related Solutions

R – Why is urlfetch throwing Download Errors when calling some Google services

C# – How and when to use ‘async’ and ‘await’

Related Topic