I'm planning on pulling data from a webservice and push the data on to a
buffer/queue for each response. This should happen in a thread, while
another thread parses the data (heavy) and writes them to disk. Time is
of the essence, so the goal is to run the webservice requests parallel
to processing data in the queue. Do you have any tips on how to "drive"
this? Can I put pop/push event handlers on the queue, event handlers on
webservice responses or event handlers on complete disk operations? All
of the above? Will I need polling to check if the disk operation is done
or other types of waiting mechanisms? I don't like polling... What about
thread safety and the queue? Some sort of locking technique is required?
Any tips, links to examples etc. will be much appreciated.
An "event handler" is just a delegate, and you can indeed have a queue
that contains delegates. But it doesn't sound like your queued data is
event handlers. It sounds more like you have data that you've downloaded,
and that data is what would be on the queue. If you download the data to
memory, then the object containing the data would be in the queue. If you
download to disk before processing, then the queue might just have a
FileInfo object, or even just a string with the filename.
The queue sounds like a standard producer/consumer sort of scenario. So
that'd be a reasonable starting point for that part of the design.
Events are a fine way to communicate between threads, if you have a need.
In this case, I mean either waitable events (derived from WaitHandle
class), or raise-able events (i.e. those declared with the C# "event"
keyword). You can have a consumer thread blocking on an EventWaitHandle;
the producer would set the event whenever it adds something to the queue,
the consumer would clear the event handle any time it was released from
waiting on the event (or just use AutoResetEvent).
Additionally, you can use the C# event to allow some other code know when
the consumer part of the design is done with a specific operation. The
consumer would, upon completing whatever work you want to be alerted
about, raise that event. Then somewhere else in your code you'd subscribe
to that event, so that each time some unit of work was done, you'd find
out about it.
Don't use polling. All of the things can be done without it. As far as
thread safety goes, yes...you'll need to protect any shared data
structures. The Monitor class, or the C# "lock()" statement (which uses
Monitor internally) would likely be fine.
Jon Skeet wrote a lengthy discussion about threaded programming in general
that you may find very useful. You can find it here:
http://www.yoda.arachsys.com/csharp/threads/
Pete
