is there a good pattern for how to send multiple calls to a web service but without taxing it and ensuring the data is sent back? I don't know enough to correctly describe the problem to even start googling it properly – current google results compare streaming vs. non-streaming wcf answers?
Scenario: I am working on an app (I'm a jr. dev.) that needs to gather information from several sources about a 'customer' and the domains they own
Technical:
for one of the sources, I need to send a string array of domains to a web service and this web service returns an entry for each domain name, but this list of domain names will be thousands long – I would like to attempt to divide this list into bite-size chunks (1k domain names each) and then… queue them up to send to that web service, but ensure the web service doesn't skip one
PseudoRequirements:
Consumer of web page does not care how long it takes, but would like a list of results up front that does not require pagination to navigate.
Current Theory:
Should I take my massive 30k list, break it into 1k chunks, stuff each 1k-sized chunk into a 'request' object, assemble those 'request chunks' into a 'request chunk list' and iterate over that list (sequentially / blocking, so I don't strangle the WS) and for each 'request chunk' get back a 'response chunk' assemble those into a list, and then pass that list back into the front end for viewing? is this a viable method? is there a better way to queue items? Does anyone know off-hand any useful articles for this sort of queuing? are there any 'gotchas' or additional items to consider before I attempt my first pass?
Additional Edits:
-I do not have full control to the receiving service, I can not view it's code and the developers that manage it are… less than responsive to email. I do not currently know the stress testing limits of the web service. I emailed the owners of that component but have yet to receive a response – I was going to work up my design while I waited on them.
Best Answer
I think your strategy is OK. Other things to consider: Before sending any messages, know how many you plan to send. If you have n messages to send, you might not want to display the full results set until you get all of your responses.
There are other things you could do such as adding correlation IDs or sequence numbers to make tracking all the messages easier, but that would assume you have control over the receiving service, and you don't say if you do.
Other questions: have you actually tested this web service to see what it can reasonably handle in a single request? You might find that it can handle more than you think. Or less than you think. Do you know for certain that a single request with 1000 domain names is optimal?