possible to use Threadpool

  • Thread starter Thread starter Daniel
  • Start date Start date
D

Daniel

Hi all

I have written a app that gets emails from a DB and checks if it's a
correct email (connects to the remote mailserver and checks the
response).
I would like to improve the performance with using multiple threads.
can I use a threadpool for this? and how do I have to do this.
For starting i have all emailaddresses in a Array.

thanks for your help.

daniel
 
Hello Daniel,

D> I have written a app that gets emails from a DB and checks if it's a
D> correct email (connects to the remote mailserver and checks the
D> response).
D> I would like to improve the performance with using multiple threads.
D> can I use a threadpool for this?

You could, but it can be not the very good solution for the 1.1 fw if u have
a big number of long runned tasks, because the number of TP's threads there
are limited by 25 thread, and in case if you have numbers of long-runned
operations u got the blocked threads which wait till one of 25 threads complete
its work.

In 2.0 u can set the desired number of threads, but take into account that
25 is the general optimal number of threads, and overcoming have negative
effect due to context switching

D> and how do I have to do this.
D> For starting i have all emailaddresses in a Array.

The samples of using ThreadPool class are into MSDN

---
WBR,
Michael Nemtsev :: blog: http://spaces.live.com/laflour

"At times one remains faithful to a cause only because its opponents do not
cease to be insipid." (c) Friedrich Nietzsche
 
Michael said:
Hello Daniel,

D> I have written a app that gets emails from a DB and checks if it's a
D> correct email (connects to the remote mailserver and checks the
D> response).
D> I would like to improve the performance with using multiple threads.
D> can I use a threadpool for this?

You could, but it can be not the very good solution for the 1.1 fw if u have
a big number of long runned tasks, because the number of TP's threads there
are limited by 25 thread, and in case if you have numbers of long-runned
operations u got the blocked threads which wait till one of 25 threads complete
its work.

In 2.0 u can set the desired number of threads, but take into account that
25 is the general optimal number of threads, and overcoming have negative
effect due to context switching

D> and how do I have to do this.
D> For starting i have all emailaddresses in a Array.

The samples of using ThreadPool class are into MSDN

---
WBR,
Michael Nemtsev :: blog: http://spaces.live.com/laflour

"At times one remains faithful to a cause only because its opponents do not
cease to be insipid." (c) Friedrich Nietzsche


Thanks for the answer. I use the 2.0 FW. So it should work fine.
As far as I could read in the docs, the Pool is using max. 25 threads
at the time. Is there a way to show in a label or so which thread is
working on with emailaddress? So I could have a view of the process.
I have to write back the not existing emailaddresses to the database.
when i do it from the thread, it will open the connection write it back
and close the connection. right? but this could produce a lot of work
for the db. is there a way to open the conecction ones and then write
back all addresses and then close the connection? or is this not so
important?
 
Hello Daniel,

D> Thanks for the answer. I use the 2.0 FW. So it should work fine.
D> As far as I could read in the docs, the Pool is using max. 25 threads
D> at the time.

Yep, but it's per process per processor, and your can change it.

D> Is there a way to show in a label or so which thread is
D> working on with emailaddress?

see ThreadPool.GetAvailableThreads method

So I could have a view of the process.
D> I have to write back the not existing emailaddresses to the database.
D> when i do it from the thread, it will open the connection write it
D> back
D> and close the connection. right? but this could produce a lot of work
D> for the db. is there a way to open the conecction ones and then write
D> back all addresses and then close the connection? or is this not so
D> important?

In MSSQL server your DB connections are pooled by defauilt (in others
DB they are not), so don't mind it

---
WBR,
Michael Nemtsev :: blog: http://spaces.live.com/laflour

"At times one remains faithful to a cause only because its opponents do not
cease to be insipid." (c) Friedrich Nietzsche
 
Daniel said:
I have written a app that gets emails from a DB and checks if it's a
correct email (connects to the remote mailserver and checks the
response).
I would like to improve the performance with using multiple threads.
can I use a threadpool for this? and how do I have to do this.
For starting i have all emailaddresses in a Array.

I would suggest not using the thread pool for this - you could easily
swamp it, which would make other things which try to use it less
responsive.

I would suggest you use a producer/consumer queue, and then you can
"manually" create as many threads as you want to be consumers of the
queue. This would be a sort of threadpool in itself, *just* for this
purpose. You'd have a lot more control over things that way.

See http://www.pobox.com/~skeet/csharp/threads/deadlocks.shtml (half
way down) for some sample producer/consumer code.
 
Jon said:
I would suggest not using the thread pool for this - you could easily
swamp it, which would make other things which try to use it less
responsive.

I would suggest you use a producer/consumer queue, and then you can
"manually" create as many threads as you want to be consumers of the
queue. This would be a sort of threadpool in itself, *just* for this
purpose. You'd have a lot more control over things that way.

See http://www.pobox.com/~skeet/csharp/threads/deadlocks.shtml (half
way down) for some sample producer/consumer code.

Thanks for the answer.

I was now just playing around with this. Not sure if I understad it
right.
Here What I have:

using System;
using System.Collections;
using System.Threading;

public class Test
{
static CheckEmail queue;
public static int rows = 100;

static void Main()
{
queue = new CheckEmail();
new Thread(new ThreadStart(EmailChecker)).Start();
new Thread(new ThreadStart(EmailChecker)).Start();
new Thread(new ThreadStart(EmailChecker)).Start();
new Thread(new ThreadStart(EmailChecker)).Start();
new Thread(new ThreadStart(EmailChecker)).Start();

Random rng = new Random(0);
for (int i = 0; i < rows; i++)
{
Console.WriteLine("Producing {0}", i);
queue.Produce("Daniel " + i.ToString());
}

while (queue.Count > 0)
{
Console.WriteLine("There are {0} emails Waiting",
queue.Count);
Thread.Sleep(2000);
}
//Here I would like to go ahead when all EMails are processed.
}

static void EmailChecker()
{
// Make sure we get a different random seed from the
// first thread
Random rng = new Random(2);
// We happen to know we've only got 10
// items to receive
for (int i = 0; i < rows; i++)
{
object o = queue.EmailCheck();
//Process Email
Console.WriteLine("\t\t\t\tConsuming {0}", o);
Thread.Sleep(rng.Next(1000));
}
}
}

public class CheckEmail
{
readonly object listLock = new object();
Queue queue = new Queue();
private string Recipient;

public int Count
{
get { return queue.Count; }
}

public void Produce(object o)
{
lock (listLock)
{
queue.Enqueue(o);
Recipient = o.ToString();

// We always need to pulse, even if the queue wasn't
// empty before. Otherwise, if we add several items
// in quick succession, we may only pulse once, waking
// a single thread up, even if there are multiple threads
// waiting for items.
Monitor.Pulse(listLock);
}
}

public object EmailCheck()
{
lock (listLock)
{
// If the queue is empty, wait for an item to be added
// Note that this is a while loop, as we may be pulsed
// but not wake up before another thread has come in and
// consumed the newly added object. In that case, we'll
// have to wait for another pulse.
while (queue.Count == 0)
{
// This releases listLock, only reacquiring it
// after being woken up by a call to Pulse
Monitor.Wait(listLock);
}
return queue.Dequeue();
}
}
}

So I produce the queue with all emailaddresses (200'000) and then walk
trought with 5 Threads (in this exemple).
Then I wait till the que is empty. How do I know if the last Thread has
finished the work?

So I have to Write the emailaddresses in the Function EmailChecker()
back to the database right?
I also have to write the Logfile in the function EmailChecker().

How can I close the Queue to exit correctly?

I want to run this only if somone requests it by pressing the button.

Thanks for your help

Daniel
 
So I produce the queue with all emailaddresses (200'000) and then walk
trought with 5 Threads (in this exemple).
Then I wait till the que is empty. How do I know if the last Thread has
finished the work?

One way would be to have a special value (eg null) which means "all
work is complete". At the end of adding all the "real" values to the
queue, add the same number of nulls as there are threads. Then change
the threading code to exit the loop when they see a null. Finally, from
the main loop, call Join on each of the worker threads in turn.
So I have to Write the emailaddresses in the Function EmailChecker()
back to the database right?

Presumably - it's not entirely obvious from your original description
what you need to write back.
I also have to write the Logfile in the function EmailChecker().

I'd suggest having a single thread which writes log entries, to make
sure you don't end up trying to write one log entry in the middle of
another one. This is sort of the reverse of the producer/consumer queue
you've got - there you'd have several producers and one consumer.
Alternatively, you could just synchronise writing out the log entries.
How can I close the Queue to exit correctly?

You could have a separate flag that is visible to all threads which
indicates that they should quit "early" even if there's more work to
do.
I want to run this only if somone requests it by pressing the button.

That's just a matter of responding to an appropriate button click
event.
 
Jon said:
One way would be to have a special value (eg null) which means "all
work is complete". At the end of adding all the "real" values to the
queue, add the same number of nulls as there are threads. Then change
the threading code to exit the loop when they see a null. Finally, from
the main loop, call Join on each of the worker threads in turn.


Presumably - it's not entirely obvious from your original description
what you need to write back.


I'd suggest having a single thread which writes log entries, to make
sure you don't end up trying to write one log entry in the middle of
another one. This is sort of the reverse of the producer/consumer queue
you've got - there you'd have several producers and one consumer.
Alternatively, you could just synchronise writing out the log entries.


You could have a separate flag that is visible to all threads which
indicates that they should quit "early" even if there's more work to
do.


That's just a matter of responding to an appropriate button click
event.

Hi Jon

Sorry it's me again. I don't really understand what you mean.

So I did a class called CheckedEmail:

public class CheckedEmail
{
private bool bolBounce;
private string strEmail;
private string strLog;
public CheckedEmail(bool Spam, string Email, string Log)
{
bolBounce = Bounce;
strEmail = Email;
strLog = Log;
}

public bool Bounce
{
set { bolBounce = value; }
get { return bolBounce; }
}

public string Email
{
set { strEmail = value; }
get { return strEmail; }
}

public string Log
{
set { strLog = value; }
get { return strLog; }
}
}



Then I fill the Queue

for (int i = 0; i < 10; i++)
{
objEmail = new CheckedEmail(false, "Daniel " + i, "");
Console.WriteLine("Producing {0}", objEmail.Email);
queue.Produce(objEmail);
}



So in the EmailChecker() i have:

for (int i = 0; i < 10; i++)
{
CheckedEmail o = (CheckedEmail) queue.EmailCheck();
Console.WriteLine("\t\t\t\tConsuming {0}", o.Email);

//Check Emailaddress
objEmail.Bounce = true;
objEmail.Log = "Connect\r\nHelo OK+\r\nsender not allowed";
Thread.Sleep(rng.Next(1000));

/*
if (objEmail.Bounce)
{
Add objEmail.Email to the DB
Add objEmail.Log to a Logfile
}
else
//Do nothing
*/
}

How can I put the objEmail in a queue to update the DB and write the
Log when objEmail.Bounce is true?

Also with the finish Flag I don't understand what you mean. In witch
loop do I have to put what? After filling the Queue with emailaddresses
I added 5 object's with objEmail.Bounce=true. So I can check if this
value is true. If so I can quit the Thread. But I don't know how and
where.

Thanks for your Help and your patience

Daniel
 
Jon said:
One way would be to have a special value (eg null) which means "all
work is complete". At the end of adding all the "real" values to the
queue, add the same number of nulls as there are threads. Then change
the threading code to exit the loop when they see a null. Finally, from
the main loop, call Join on each of the worker threads in turn.


Presumably - it's not entirely obvious from your original description
what you need to write back.


I'd suggest having a single thread which writes log entries, to make
sure you don't end up trying to write one log entry in the middle of
another one. This is sort of the reverse of the producer/consumer queue
you've got - there you'd have several producers and one consumer.
Alternatively, you could just synchronise writing out the log entries.


You could have a separate flag that is visible to all threads which
indicates that they should quit "early" even if there's more work to
do.


That's just a matter of responding to an appropriate button click
event.



Hi Jon

Sorry it's me again. I don't really understand what you mean.


So I did a class called CheckedEmail:


public class CheckedEmail
{
private bool bolBounce;
private string strEmail;
private string strLog;
public CheckedEmail(bool Spam, string Email, string Log)
{
bolBounce = Bounce;
strEmail = Email;
strLog = Log;
}


public bool Bounce
{
set { bolBounce = value; }
get { return bolBounce; }
}


public string Email
{
set { strEmail = value; }
get { return strEmail; }
}


public string Log
{
set { strLog = value; }
get { return strLog; }
}



}


Then I fill the Queue

for (int i = 0; i < 10; i++)
{
objEmail = new CheckedEmail(false, "Daniel " + i, "");
Console.WriteLine("Producing {0}", objEmail.Email);
queue.Produce(objEmail);
}


So in the EmailChecker() i have:


for (int i = 0; i < 10; i++)
{
CheckedEmail o = (CheckedEmail) queue.EmailCheck();
Console.WriteLine("\t\t\t\tConsuming {0}", o.Email);


//Check Emailaddress
o.Bounce = true;
o.Log = "Connect\r\nHelo OK+\r\nsender not allowed";
Thread.Sleep(rng.Next(1000));


/*
if (o.Bounce)
{
Add objEmail.Email to the DB
Add objEmail.Log to a Logfile
}
else
//Do nothing
*/
}


How can I put the objEmail in a queue to update the DB and write the
Log when objEmail.Bounce is true?


Also with the finish Flag I don't understand what you mean. In witch
loop do I have to put what? After filling the Queue with emailaddresses

I added 5 object's with objEmail.Bounce=true. So I can check if this
value is true. If so I can quit the Thread. But I don't know how and
where.


Thanks for your Help and your patience


Daniel
 
Daniel said:
So in the EmailChecker() i have:

for (int i = 0; i < 10; i++)
{
CheckedEmail o = (CheckedEmail) queue.EmailCheck();

It's not very clear why this is called EmailCheck rather than just
Consume - the queue can be a reusable class which can handle any class,
not just CheckedEmail objects.
How can I put the objEmail in a queue to update the DB and write the
Log when objEmail.Bounce is true?

Well, you can have another queue, like the first one, which all the
worker threads add to. You'd have:

Main thread (produces email addresses to check)

/ | \
Thread 1 Thread 2 Thread 3 etc (checking mails)
\ | /

Logging/DB thread (logging and writing to the database)
Also with the finish Flag I don't understand what you mean. In witch
loop do I have to put what? After filling the Queue with emailaddresses

I added 5 object's with objEmail.Bounce=true. So I can check if this
value is true. If so I can quit the Thread. But I don't know how and
where.

I don't think that's the right way of signalling things - it's better
to have a different type of object, or null, or something else to make
it clear that it means you've finished. Either way though, just after
you've consumed the item, you check whether it means "end of queue" and
just return if so.
 
Jon said:
It's not very clear why this is called EmailCheck rather than just
Consume - the queue can be a reusable class which can handle any class,
not just CheckedEmail objects.


Well, you can have another queue, like the first one, which all the
worker threads add to. You'd have:

Main thread (produces email addresses to check)

/ | \
Thread 1 Thread 2 Thread 3 etc (checking mails)
\ | /

Logging/DB thread (logging and writing to the database)


I don't think that's the right way of signalling things - it's better
to have a different type of object, or null, or something else to make
it clear that it means you've finished. Either way though, just after
you've consumed the item, you check whether it means "end of queue" and
just return if so.

Thanks. The null Object works fine.

So I have to create a second queue with:

static ProducerConsumer logqueue;

or do I have to write a second ProducerCustomer class?

In the ConsumerJob() I have to add the following:

while (queue.Count >= 0)
{
object o = queue.Consume();
if (o == null)
break;
objEmail = (CheckedEmail)o;

//Check the Emails

if (objEmail.Bounce)
logque.Produce(objEmail);
....

How can I start a Thread that looks onely in logque and processes if
there is a entry?
I start the Threads with:
new Thread(new ThreadStart(ConsumerJob)).Start();
There i can't choose the queue. Also in the ConsumerJob() I can't
choose the queue.

Thanks
 
Daniel said:
Thanks. The null Object works fine.

So I have to create a second queue with:

static ProducerConsumer logqueue;

or do I have to write a second ProducerCustomer class?

No, you can do it with one class.
In the ConsumerJob() I have to add the following:

while (queue.Count >= 0)
{
object o = queue.Consume();
if (o == null)
break;
objEmail = (CheckedEmail)o;

//Check the Emails

if (objEmail.Bounce)
logque.Produce(objEmail);
....

You shouldn't have

while (queue.Count >= 0)

It should be

while (true)

because you don't want to quit the threads just because you may consume
the jobs before they've all been produced.
How can I start a Thread that looks onely in logque and processes if
there is a entry?

In a similar way - you call queue.Consume(), which will wait until
there is a job to do.
I start the Threads with:
new Thread(new ThreadStart(ConsumerJob)).Start();
There i can't choose the queue. Also in the ConsumerJob() I can't
choose the queue.

It sounds like you want to look at:
http://www.pobox.com/~skeet/csharp/threads/parameters.shtml
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Back
Top