Approach to multithreaded loading of object data
I have a project where i load a number of objects from a web service. For each of these objects i need to call multiple webservices to "enrich" the object. Like e.g. a book, where i need to look up author, year, comments, recommendations etc.
One of the focus areas in this project is performance, so im a little worried about this, and the best approach for loading/enriching all these objects.
Any suggestions for this architecture is welcome.
Right now im planning on looping the books (the first list) and then async query the webservices to get the information i need for "enriching" each book. Is this a good approach? Or is there a better approach?
If you are using .NET 4 then the Task Parallel Library may be the way forward. I've used it to make multiple simultaneous calls to a web service in a similar scenario and it has worked quite well.
These two blog posts may be helpful:
- Tasks that create more work
- Using semaphores to restrict access to resources (I found that under high load I was killing the service at the other end so I needed to be able to throttle access to it)
I hope that helps.