I'm building a windows service which will monitor the file system for new
files, process the information in any new file that appears, and put that
information into a database. My worry is that many files may appear (not
necessarily at once, but close enough) and will fire the file system watcher
events, opening up many threads. Is there a way I can limit the threads
that open, yet still keep the new file information in some sort of queue to
process when a thread is free? Is there a better approach so I wouldn't be
drowing system resources and constantly opening and closing db connections?
Others may disagree, but this is how I'd approach this problem.
I would have the main service thread and one worker thread. The main thread
blocks on a timer and the worker thread blocks on file watching. The worker
thread's only job is to set the timer's interval to a minimum value when it
sees file activity of interest. The main thread processes some (or all, but
I like some better) files in the directory when the timer wakes it up. As
long as it does some work, it leaves the timer interval at minimum value.
During periods of idleness, it raises the interval toward a ceiling. For
example, the minimum could be 10 milliseconds, the maximum 10 seconds, and
the raising operation could be to multiply by 10, so the values the interval
takes on are 10, 100, 1000, and 10000 milliseconds. At transition from 1000
to 10000, the main thread can close the db connection.
This design is resource friendly on several fronts. It also overcomes a
problem with file watcher technology wherein some file watcher events can be
lost if file activity happens fast enough. All the file watcher thread needs
to do is see one file activity to wake up the main thread, and the main
thread will eventually process everything it finds in the directory. In
other words, the file watcher is used only as a trigger. All that matters is
that it sees something, and specifically what files it sees is of no
consequence.
You will need to make access to the timer interval thread-safe. I guess I
would use synclock, and I would have the worker thread read the interval and
then update it only if it was unequal to the minimum value. I would create
the timer on the main thread, so the worker probably needs to do thread
marshalling to update the interval.
This design does single thread db access. If you really need many threads
performing concurrent db accesses, then this design is not for you. From
your original post, it sounds like the db updates can happen in any order (ie
roughly as files show up in the directory) and that db parallelism is not an
issue.
Other responders may suggest your original design with thread pooling. I
don't like it, never have, never will, and I'm kind of obstinate about it.
But in theory, it could work for you. It is just not my cup of tea.
