.net 2.0 : looking for a "best practice" for multi threading jobs

[Steve B.]

Hi,

I'm building an application that follow this scenario.

1. Download a file on a server with http
2. Analyse this file and extract other files in one of the section of the
file (like dependent files)
3. Foreach found files, repeat the whole process.

The goal is to download the resource and all its dependencies (not know
until the first file is downloaded). Each file can have multiple
dependencies and one file can be a dependence of several other files.

The process of downloading rely on the HttpWebResponse class (I cannot use
WebClient, because I use some http headers) and is working well.
The process of analysing consists of parsing the file with a custom business
object and is also working well.

Everything is working corectly in a "console" application, but I'd like to
build onto the process a user friendly form that display progress bar,
warning messages, etc.
I also need to reduce the whole process so I want to have several threads
working in parralel... in a "pool of object" manner.

The question is : how can I quickly and correctly design my code to have my
winform dynamic, and the pool of working threads?

My first reflexion was to store the whole data in a DataSet (easy
databinding) which can be flat (only on DataTable required).
I also created an abstract class "BaseAction" that is inherited by two
classes : "DownloadAction" and "AnalyseAction".
My process object defines a Queue<Action> where I add all required actions.
But I'm confused for the next steps... how can I have multiples trheads that
can "Dequeue" actions? moreover, I don't know how to manage the multiples
threeads since sometimes I'll have only one item in te queue, and this item
can add new items...


Thanks in advance for any tips.
Steve

 

[Michael Nemtsev]

Hello Steve B.,

S> I'm building an application that follow this scenario.
S> 1. Download a file on a server with http
S> 2. Analyse this file and extract other files in one of the section of
S> the
S> file (like dependent files)
S> 3. Foreach found files, repeat the whole process.

S> The question is : how can I quickly and correctly design my code to
S> have my winform dynamic, and the pool of working threads?


S> My first reflexion was to store the whole data in a DataSet (easy
S> databinding) which can be flat (only on DataTable required).

so, you keep serialized objects?! why not to just save files on your drive?

S> I also created an abstract class "BaseAction" that is inherited by
S> two classes : "DownloadAction" and "AnalyseAction".
S> My process object defines a Queue<Action> where I add all required
S> actions.
S> But I'm confused for the next steps... how can I have multiples
S> trheads that can "Dequeue" actions? moreover, I don't know how to manage
the
S> multiples threeads since sometimes I'll have only one item in te queue,
and
S> this item can add new items...

You can use ThreadPool class (standard .net) or Produce/consummer pattern

http://www.yoda.arachsys.com/csharp/threads/threadpool.shtml
http://groups.google.ru/group/micro...read/thread/a33378c87b856678/abc251c95cbac487

The idea is that you can dequeue your files and each tread will analyze your
files dequeuing them


---
WBR, Michael Nemtsev [C# MVP].
My blog: http://spaces.live.com/laflour
Team blog: http://devkids.blogspot.com/

"The greatest danger for most of us is not that our aim is too high and we
miss it, but that it is too low and we reach it" (c) Michelangelo

 

[Steve B.]

Thanks for your answer.

When I talked about whole data in a DataSet, I mean whole CONTROL data... ie
downloaded files, location of file ont the disk or current progression of an
action.

The link you provided is very interesting. I'll take a deep look but I'm
quite sure by now that the Monitor.Pulse will solve my problem .

Thanks very much !

Steve

 

[Steve B.]

Hi again,

I'm getting troubles in implementing the producer/consumer pattern for
multiple parralel threads...

Here where I am (a project class)... the code is simplified for better
read...

In some conditions, all threads are "blocked" in the GetNectAction() method
waiting in the Monitor.WaitOne method...

Do you have any idea what is wrong ?
Thanks,
Steve

public class Project
{
#region Properties
private int m_numberOfThreads;
private object m_numberOfThreadsLockObject = new object();
public int NumberOfThreads
{
get
{
lock (m_numberOfThreadsLockObject)
{
return m_numberOfThreads;
}
}
}
private int m_runningThreads = 0;
private object m_runningThreadsLockObject = new object();
public int RunningThreads
{
get
{
lock (m_runningThreadsLockObject)
{
return m_runningThreads;
}
}
}
private Exporter m_exporter;
public Exporter Exporter
{
get { return m_exporter; }
}
#region Cancellable worker
private readonly object m_stoppingLockObject = new object();
private bool m_stopping;
public bool Stopping
{
get
{
lock (m_stoppingLockObject)
{
return m_stopping;
}
}
}
private bool m_stopped;
public bool Stopped
{
get
{
lock (m_stoppingLockObject)
{
return m_stopped;
}
}
}
private void SetStopped()
{
Debug.WriteLine("Fin des threads par " + Thread.CurrentThread.Name);
lock (m_numberOfThreadsLockObject)
{
this.m_stopped = true;
}
}
#endregion
private Queue<Action> m_actions;
private object m_actionsLockObject = new object();
protected Queue<Action> Actions
{
get
{
lock (m_actions)
{
return m_actions;
}
}
}
#endregion
public Project(
int numberOfThreads
)
{
this.m_numberOfThreads = numberOfThreads;
this.m_actions = new Queue<Action>();
}
public void Execute()
{
for (int i = 0; i < m_numberOfThreads; i++)
{
lock (m_runningThreadsLockObject)
{
ThreadStart ts = new ThreadStart(WorkerMethod);
Thread t = new Thread(ts);
t.Name = "Thread N°" + i.ToString();
t.Start();
m_runningThreads++;
}
}
AddAction(new DownloadAction(
new Uri("http://myserver/app/rootfile")
));
}
}
public void AddAction(Action a)
{
Debug.WriteLine("AddAction dans thread " + Thread.CurrentThread.Name);
lock (m_actionsLockObject)
{
Actions.Enqueue(a);
Debug.WriteLine("Monitor.Pulse " + Thread.CurrentThread.Name);
Monitor.Pulse(m_actionsLockObject);
}
}
private Action GetNextAction()
{
lock (m_actionsLockObject)
{
Monitor.Wait(this.m_actionsLockObject);
return Actions.Dequeue();
}
}
private void WorkerMethod()
{
Debug.WriteLine("WorkerMethod dans thread " + Thread.CurrentThread.Name);
Action a;
while (!Stopping && (a = GetNextAction()) != null)
{
a.Execute();
}
lock (m_runningThreadsLockObject)
{
if (--m_runningThreads == 0)
{
SetStopped();
}
}
Debug.WriteLine("Fin du thread " + Thread.CurrentThread.Name);
}
public void Stop()
{
lock (m_actionsLockObject)
{
Monitor.PulseAll(m_actionsLockObject);
}
Debug.WriteLine("Stop dans thread " + Thread.CurrentThread.Name);
lock (m_stoppingLockObject)
{
this.m_stopping = true;
}
}
}

Hello Steve B.,

S> I'm building an application that follow this scenario.
S> 1. Download a file on a server with http
S> 2. Analyse this file and extract other files in one of the section of
S> the
S> file (like dependent files)
S> 3. Foreach found files, repeat the whole process.

S> The question is : how can I quickly and correctly design my code to
S> have my winform dynamic, and the pool of working threads?


S> My first reflexion was to store the whole data in a DataSet (easy
S> databinding) which can be flat (only on DataTable required).

so, you keep serialized objects?! why not to just save files on your
drive?

S> I also created an abstract class "BaseAction" that is inherited by
S> two classes : "DownloadAction" and "AnalyseAction".
S> My process object defines a Queue<Action> where I add all required
S> actions.
S> But I'm confused for the next steps... how can I have multiples
S> trheads that can "Dequeue" actions? moreover, I don't know how to
manage the
S> multiples threeads since sometimes I'll have only one item in te queue,
and
S> this item can add new items...

You can use ThreadPool class (standard .net) or Produce/consummer pattern

http://www.yoda.arachsys.com/csharp/threads/threadpool.shtml
http://groups.google.ru/group/micro...read/thread/a33378c87b856678/abc251c95cbac487

The idea is that you can dequeue your files and each tread will analyze
your files dequeuing them


---
WBR, Michael Nemtsev [C# MVP]. My blog: http://spaces.live.com/laflour
Team blog: http://devkids.blogspot.com/

"The greatest danger for most of us is not that our aim is too high and we
miss it, but that it is too low and we reach it" (c) Michelangelo

 

[Jon Skeet [C# MVP]]

I'm getting troubles in implementing the producer/consumer pattern for
multiple parralel threads...

Here where I am (a project class)... the code is simplified for better
read...

In some conditions, all threads are "blocked" in the GetNectAction() method
waiting in the Monitor.WaitOne method...

Do you have any idea what is wrong ?

Well, a few points:

1) You don't usually need one lock per variable. I tend to have one
lock for a whole instance of a class unless I have good reason to have
more granularity.

2) Point 1 opens you up for bugs where you lock on the wrong thing: for
example, SetStopped locks on m_numberOfThreadsLockObject to set
m_stopped, but the Stopped property locks on m_stoppingLockObject to
fetch it.

3) In Stop, you pulse all the waiting threads before setting the
stopping flag. Any thread which is still working when the pulse occurs
but requests a next action before m_stopping is set to true will wait
forever.

That's just what I could spot in a very quick scan. It would be much
better if I could run the code.

Could you post a short but complete program which demonstrates the
problem?

See http://www.pobox.com/~skeet/csharp/complete.html for details of
what I mean by that.