After recursively creating threads, how do I tell when they are done?

D

David Beoulve

In my windows C# app, I create a thread to separate the main code block
from the GUI, which recursively sorts through all the directories
starting from a given point, and every time it finds a "*.html" file,
it starts a new thread to go to work on that file. Since each
individual thread doesn't have to touch another's file, this works
fine.

What I need to know is when all of these threads are finished.
Theoretically a lot of threads could be cued, and I'm having trouble
figuring out how to store a list of references to them so I can check
their ".IsAlive" state.

Can anyone give me any pointers?
 
J

Jon Skeet [C# MVP]

David Beoulve said:
In my windows C# app, I create a thread to separate the main code block
from the GUI, which recursively sorts through all the directories
starting from a given point, and every time it finds a "*.html" file,
it starts a new thread to go to work on that file. Since each
individual thread doesn't have to touch another's file, this works
fine.

Hmm... it sounds like a bit of a waste though. Why not just create a
list of html files to look through? Chances are adding extra threads
here isn't going to help you - it'll just add complexity. (I'm not
suggesting you do the work in the UI thread - just that you only create
a few threads, rather than one per file).
What I need to know is when all of these threads are finished.
Theoretically a lot of threads could be cued, and I'm having trouble
figuring out how to store a list of references to them so I can check
their ".IsAlive" state.

Can anyone give me any pointers?

Well, you can add Thread references to a list just like you'd add any
other reference. You just need access to a common list. Rather than
using the IsAlive property, however, I'd call Join on each thread in
turn to wait until it's finished. (You could do this in a single extra
thread which then "told" the UI thread that everything had finished, if
you want.) If you only have a couple of threads, however, this may not
be necessary - you may well find that separating the app into maybe the
UI thread, the "searching" thread and the "processing" thread would
naturally make life simpler.
 
M

Michael Nemtsev

Hello David,

DB> In my windows C# app, I create a thread to separate the main code
DB> block from the GUI, which recursively sorts through all the
DB> directories starting from a given point, and every time it finds a
DB> "*.html" file, it starts a new thread to go to work on that file.
DB> Since each individual thread doesn't have to touch another's file,
DB> this works fine.

Did u take into account that thread, by default, takes 1mb of virtual memory.
You can end up with RAM limit very shorty in this desing.

---
WBR,
Michael Nemtsev :: blog: http://spaces.msn.com/laflour

"At times one remains faithful to a cause only because its opponents do not
cease to be insipid." (c) Friedrich Nietzsche
 
W

William Stacey [MVP]

I agree. Unless you have 2x+ cpus, the chances are high this approach will
be slower then just using one thread as all your threads will just be
competing for time and HD I/O (jumping the reads like ping-pong) with each
other. Many times, more threads does not mean faster or better.

--
William Stacey [MVP]

| Hmm... it sounds like a bit of a waste though. Why not just create a
| list of html files to look through? Chances are adding extra threads
| here isn't going to help you - it'll just add complexity. (I'm not
| suggesting you do the work in the UI thread - just that you only create
| a few threads, rather than one per file).
 
W

William Stacey [MVP]

Or just a List<Thread> if he still wants to go with multiple threads.

--
William Stacey [MVP]

| Try putting them into a Hashtable.
| Peter
|
| --
| Co-founder, Eggheadcafe.com developer portal:
| http://www.eggheadcafe.com
| UnBlog:
| http://petesbloggerama.blogspot.com
|
|
|
|
| "David Beoulve" wrote:
|
| > In my windows C# app, I create a thread to separate the main code block
| > from the GUI, which recursively sorts through all the directories
| > starting from a given point, and every time it finds a "*.html" file,
| > it starts a new thread to go to work on that file. Since each
| > individual thread doesn't have to touch another's file, this works
| > fine.
| >
| > What I need to know is when all of these threads are finished.
| > Theoretically a lot of threads could be cued, and I'm having trouble
| > figuring out how to store a list of references to them so I can check
| > their ".IsAlive" state.
| >
| > Can anyone give me any pointers?
| >
| >
 
D

David Beoulve

Coding this was, partly, an exercise in learning threading in the first
place. The last time I had a college programming course -- I was
learning linear C++.

It's good to know that threads take up so much virtual memory -- I had
no idea as nothing bothers to mention that, anywhere.

Efficiency isn't the order of the day when "Learning" is, so I'll press
on and code things differently in applications which need the former.

To "continue the experiment," such as it is, I just need to figure out
how to, as I create threads, store a reference to them and have a
"watcher" thread think about this, as someone here suggested (it was a
course of action I've already coded, but it has nothing to do yet).

How would I use a Hashtable? Or a List<Thread>? I know about Join() but
I need a way to dynamically store all the threads created, as they are
created, and periodically go through the list of threads and see if
anyone is alive to find out when the application is "done."

My thanks in advance for your time and expertise, guys.
 
D

David Beoulve

Coding this was, partly, an exercise in learning threading in the first
place. The last time I had a college programming course -- I was
learning linear C++.

It's good to know that threads take up so much virtual memory -- I had
no idea as nothing bothers to mention that, anywhere.

Efficiency isn't the order of the day when "Learning" is, so I'll press
on and code things differently in applications which need the former.

To "continue the experiment," such as it is, I just need to figure out
how to, as I create threads, store a reference to them and have a
"watcher" thread think about this, as someone here suggested (it was a
course of action I've already coded, but it has nothing to do yet).

How would I use a Hashtable? Or a List<Thread>? I know about Join() but
I need a way to dynamically store all the threads created, as they are
created, and periodically go through the list of threads and see if
anyone is alive to find out when the application is "done."

My thanks in advance for your time and expertise, guys.
 
J

Jon Skeet [C# MVP]

How would I use a Hashtable? Or a List<Thread>? I know about Join() but
I need a way to dynamically store all the threads created, as they are
created, and periodically go through the list of threads and see if
anyone is alive to find out when the application is "done."

I'm not sure I understand what the issue is. You create Thread objects
in order to start them, presumably - where are you having trouble in
adding those references to a list?

Could you post a short but complete program which demonstrates the
problem?

See http://www.pobox.com/~skeet/csharp/complete.html for details of
what I mean by that.
 
D

David Beoulve

Sure thing, Jon. I love your explanation page -- but it's much too
wordy. I skimmed it and found that my assumptions before clicking your
link were right -- you wanted a compilable program that didn't have
extraneous logic.

Here it goes: (this compiles)
-------------------------------------------------------------------------------------------------------------------------
"Form1.cs"
-------------------------------------------------------------------------------------------------------------------------
using System;
using System.IO;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Text;
using System.Windows.Forms;
using System.Threading;

namespace Navigation_Creator_2
{
public delegate void doUpdateUIthreadCounter();
public partial class Form1 : Form
{
private Thread[] oThreads = new Thread[5000];
private int iThreadIndex = 0;
public Form1()
{
InitializeComponent();
}
private void Form1_Load(object sender, EventArgs e)
{
}

private void buttonRunProgram_Click(object sender, EventArgs e)
{

// Create a new thread to separate it from the UI
Thread tMain = new Thread(new
ThreadStart(doFoldersThread));
tMain.Name = "Main Thread";
tMain.IsBackground = true;
tMain.Start();

// add it to our watch list of threads.
addThreadToThreadWatch(tMain, tMain.Name);

Thread tWatcher = new Thread(new
ThreadStart(awakenTheWatcher));
tWatcher.Name = "The Watcher";
tWatcher.IsBackground = true;
tWatcher.Start();

/* disable the RUN button - we don't want threads trying
* to access the same files */
buttonRunProgram.Enabled = false;
}
private void addThreadToThreadWatch(Thread t, string
strThreadName)
{
lock (oThreads)
{
if (t.IsAlive)
{
// add this thread to our stack list.
oThreads[iThreadIndex] = t;
// tmpFileWriter.WriteLine(strThreadName + " added
to watch. " + t.Name + "/" + t.ToString());
}
}
}
void doUpdateUIthreadCounter()
{
if (InvokeRequired)
{
BeginInvoke(new
doUpdateUIthreadCounter(invokeUpdateUIthreadCounter) /*, new object[] {
strThreadName }*/);
}
else
{
lock (oThreads)
{
textBoxThreadsRunning.Text += "[" +
iThreadIndex.ToString() + "] ";
}
}
}
private void invokeUpdateUIthreadCounter()
{
lock (oThreads)
{
textBoxThreadsRunning.Text += "[" +
iThreadIndex.ToString() + "] ";
}
}
private void awakenTheWatcher()
{
bool bStayAwake = false;

/* As long as the watcher is awake, he will watch the
threads.
* once all the threads are dead, he too, will pass. */
do
{
bStayAwake = false;

Thread.Sleep(500);
lock (oThreads)
{
iThreadIndex = 0;
int iCounter = 0;
while (oThreads[iCounter] != null)
{
if (oThreads[iCounter].IsAlive)
{
iThreadIndex++;
bStayAwake = true;
}
iCounter++;
}
}
// update the THREAD counter
doUpdateUIthreadCounter();
} while (bStayAwake);
}
private void doFoldersThread()
{
// where to start
string strCurrentDir = "C:\\Inetpub\\wwwroot\\";

// get all the directories in this area...
// string[] strDirectories =
Directory.GetDirectories(strCurrentDir);

// get all the .html files in this area
// string[] strFiles = Directory.GetFiles(strCurrentDir,
"*.html");

// Set up a log file... we'll list the directories and
files we find here.
TextWriter tmpFileWriter = new
StreamWriter("C:\\Inetpub\\wwwroot\\content\\logfile.tmp");

if (tmpFileWriter != null)
{
// First, we search the files in the ROOT area...
DirSearchFiles(strCurrentDir, tmpFileWriter, false);

// Then we recursively go through the directories...
DirSearch(strCurrentDir, tmpFileWriter);
}
else
{
// it didn't work

tmpFileWriter.WriteLine(DateTime.Now.TimeOfDay.ToString() +
"\tERROR creating LOG file.");
}

// Close the log file.
tmpFileWriter.Close();
}
private void DirSearch(
string sDir,
TextWriter tmpFileWriter)
{
// open the log file to write to...

try
{
/* This boolean will tell us whether or not, during our
look through
* every file in a given directory, if we have already
listed that
* directory as having .html files in it. */
bool bDoneAlready = false;

/* Go through each directory at the current level... */
foreach (string d in Directory.GetDirectories(sDir))
{
bDoneAlready = false; // reset the boolean

/* Go through all of the files at the current
level. If one
* is an .html file, then we must list this
directory
* (and add it to the stack). */
DirSearchFiles(d, tmpFileWriter, bDoneAlready);

/* Recursively do this same routine for each
directory! */
DirSearch(d, tmpFileWriter);
}
}
catch (System.Exception error)
{
// Update UI:
// Let the user know what went wrong.

tmpFileWriter.WriteLine(DateTime.Now.TimeOfDay.ToString() +
error.ToString());
}
}
private void DirSearchFiles(
string d,
TextWriter tmpFileWriter,
bool bDoneAlready)
{
foreach (string f in Directory.GetFiles(d, "*.html"))
{
if (File.Exists(f))
{ // we have a directory that has valid .html files!
if (!bDoneAlready)
{
bDoneAlready = true;
tmpFileWriter.WriteLine(d);
}
else
{
tmpFileWriter.WriteLine("\t" + f);
}

// ADD TO DIRECTORY STACK
ThreadDoFile objContainer = new ThreadDoFile(d, f);
Thread objThread = new Thread(new
ThreadStart(objContainer.ThreadFunction));
objThread.Name = "File Thread " + f;

objThread.Start();

addThreadToThreadWatch(objThread, objThread.Name);
}
}
}
}
class ThreadDoFile
{
// directory for this file...
string strDirectory;

// what it says... but includes the path.
string strFileName;

public ThreadDoFile(
string strDirectoryPassed,
string strFileNamePassed)
{
strDirectory = strDirectoryPassed;
strFileName = strFileNamePassed;
//strPageName = "ERROR: No Page Name Instantiated";
}

public void ThreadFunction()
{
// strFileName is a variable of this class.
string strCurrentFile = strFileName;

// open the temporary file to write to...
TextWriter logFileWriter =
doOpenFileForWriting(strCurrentFile + ".tmp");

/* try to take the current file and read until we find
<CREATENAV>
* and then we can insert new navigation information here
*/

logFileWriter.WriteLine(DateTime.Now.TimeOfDay.ToString() + "\tStarting
ThreadDoFile() " + strFileName);
// doStuff();

logFileWriter.WriteLine(DateTime.Now.TimeOfDay.ToString() + "\tEnding
ThreadDoFile() " + strFileName);
logFileWriter.Close();
}
private TextWriter doOpenFileForWriting(string strFileName)
{
try
{
// create a writer and open the file
TextWriter tmpFileWriter = new
StreamWriter(strFileName);
return tmpFileWriter;
}
catch (Exception error)
{
// Let the user know what went wrong.
return null;
}
}
}
}
-------------------------------------------------------------------------------------------------------------------------
"Form1.Designer.cs"
-------------------------------------------------------------------------------------------------------------------------
namespace Navigation_Creator_2
{
partial class Form1
{
/// <summary>
/// Required designer variable.
/// </summary>
private System.ComponentModel.IContainer components = null;

/// <summary>
/// Clean up any resources being used.
/// </summary>
/// <param name="disposing">true if managed resources should be
disposed; otherwise, false.</param>
protected override void Dispose(bool disposing)
{
if (disposing && (components != null))
{
components.Dispose();
}
base.Dispose(disposing);
}

#region Windows Form Designer generated code

/// <summary>
/// Required method for Designer support - do not modify
/// the contents of this method with the code editor.
/// </summary>
private void InitializeComponent()
{
this.buttonRunProgram = new System.Windows.Forms.Button();
this.textBoxThreadsRunning = new
System.Windows.Forms.TextBox();
this.SuspendLayout();
//
// buttonRunProgram
//
this.buttonRunProgram.Font = new
System.Drawing.Font("Verdana", 9.75F, System.Drawing.FontStyle.Regular,
System.Drawing.GraphicsUnit.Point, ((byte)(0)));
this.buttonRunProgram.Location = new
System.Drawing.Point(13, 13);
this.buttonRunProgram.Name = "buttonRunProgram";
this.buttonRunProgram.Size = new System.Drawing.Size(75,
29);
this.buttonRunProgram.TabIndex = 0;
this.buttonRunProgram.Text = "Start";
this.buttonRunProgram.UseVisualStyleBackColor = true;
this.buttonRunProgram.Click += new
System.EventHandler(this.buttonRunProgram_Click);
//
// textBoxThreadsRunning
//
this.textBoxThreadsRunning.BackColor =
System.Drawing.SystemColors.InactiveCaptionText;
this.textBoxThreadsRunning.Font = new
System.Drawing.Font("Verdana", 9F, System.Drawing.FontStyle.Regular,
System.Drawing.GraphicsUnit.Point, ((byte)(0)));
this.textBoxThreadsRunning.ForeColor =
System.Drawing.Color.OliveDrab;
this.textBoxThreadsRunning.Location = new
System.Drawing.Point(13, 49);
this.textBoxThreadsRunning.Multiline = true;
this.textBoxThreadsRunning.Name = "textBoxThreadsRunning";
this.textBoxThreadsRunning.ScrollBars =
System.Windows.Forms.ScrollBars.Vertical;
this.textBoxThreadsRunning.Size = new
System.Drawing.Size(757, 405);
this.textBoxThreadsRunning.TabIndex = 1;
//
// Form1
//
this.AutoScaleDimensions = new System.Drawing.SizeF(6F,
13F);
this.AutoScaleMode =
System.Windows.Forms.AutoScaleMode.Font;
this.ClientSize = new System.Drawing.Size(782, 466);
this.Controls.Add(this.textBoxThreadsRunning);
this.Controls.Add(this.buttonRunProgram);
this.Name = "Form1";
this.Text = "Form1";
this.Load += new System.EventHandler(this.Form1_Load);
this.ResumeLayout(false);
this.PerformLayout();
}
#endregion
private System.Windows.Forms.Button buttonRunProgram;
private System.Windows.Forms.TextBox textBoxThreadsRunning;
}
}
 
D

David Beoulve

Hmm, I should have READ your entire page. It has a lot of useful clues
in there. But I'm off-topic.
 
D

David Beoulve

Declare a public variable:
static readonly object stopLock = new object();
And then lock it anytime you're adding a thread, and this works fine.
lock (stopLock)
{
// ADD TO DIRECTORY STACK
ThreadDoFile objContainer = new ThreadDoFile(d,
f);
oThreads[iThreadIndex] = new Thread(new
ThreadStart(objContainer.ThreadFunction));
oThreads[iThreadIndex].Name = "File Thread[" +
iThreadIndex.ToString() + "]" + f;
oThreads[iThreadIndex].Start();
iThreadIndex++;
}
 
J

Jon Skeet [C# MVP]

David Beoulve said:
Declare a public variable:
static readonly object stopLock = new object();
And then lock it anytime you're adding a thread, and this works fine.
lock (stopLock)
{
// ADD TO DIRECTORY STACK
ThreadDoFile objContainer = new ThreadDoFile(d,
f);
oThreads[iThreadIndex] = new Thread(new
ThreadStart(objContainer.ThreadFunction));
oThreads[iThreadIndex].Name = "File Thread[" +
iThreadIndex.ToString() + "]" + f;
oThreads[iThreadIndex].Start();
iThreadIndex++;
}

Does that mean you're sorted now? (I would use an ArrayList rather than
a straight array, by the way, but there we go.)

You were right about the "complete" bit (although what you gave didn't
compile for me - I didn't look into it due to this post) but the
"short" bit seemed to have gone by the wayside a bit :)

If you have any suggestions for improving the page, by the way, I'm all
ears. One thing I've been thinking of is an alternative where you
supply a failing unit test instead of a complete program. Not much help
to those who don't do unit testing, but great otherwise.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Top