Threading question

[John Rogers]

I have some code that does a recursive copy of whatever folder I give to
it to copy the structure where I want it. I have it running in a single
thread
to make it a bit more efficient. So far it works good, but I am trying to
add
more directories so I can do a bulk copy.

Every article that I have read on threads have always showed two different
functions to run your two threads etc. My idea is to make a little utility
that
does the same thing as a download manager, but instead it will copy files to
different folders drives etc.

This is where I am lost on the threading issues.

I want to add every dir to a listview, lets say I have some items in a
listview.

c:\backup
c:\source
c:\drivers
c:\utils

I want to copy these items to d:\BACKUPS

Now in the listview (this is my dilemma) I want to start a filecopy of every
checked item in a different thread and copy them to the destination dir.
I know how to go through the listview to see what is checked and what is
not,
but the threading part is whats getting me lost.

Also, do I need a different routine to copy the files for each thread? I
ask this
because I do not think that I can use the same file copy routine for all of
the threads
running at the same time.

CopyFilesandDirs() etc.

I hope I explained this with enough detail, let me number a few things.

1) Go through the listview and find each checked item.
2) Copy each checked item using a different thread to the destination dir.
3) Only run about 3-4 threads at once, no more than that.


Appreciate any help guys.

John

 

[Marra]

If each part of your code is taking quite a while to run then threads
are good.

I tried clearing a large buffer with 4 threads and the threads were
slower !
I can only guess there is quite a bit of overhead code using threads.

 

[Sergey Zyuzin]

I have some code that does a recursive copy of whatever folder I give to
it to copy the structure where I want it. šI have it running in a single
thread
to make it a bit more efficient. šSo far it works good, but I am trying to
add
more directories so I can do a bulk copy.

Every article that I have read on threads have always showed two different
functions to run your two threads etc. šMy idea is to make a little utility
that
does the same thing as a download manager, but instead it will copy files to
different folders drives etc.

This is where I am lost on the threading issues.

I want to add every dir to a listview, lets say I have some items in a
listview.

c:\backup
c:\source
c:\drivers
c:\utils

I want to copy these items to d:\BACKUPS

Now in the listview (this is my dilemma) I want to start a filecopy of every
checked item in a different thread and copy them to the destination dir.
I know how to go through the listview to see what is checked and what is
not,
but the threading part is whats getting me lost.

Also, do I need a different routine to copy the files for each thread? šI
ask this
because I do not think that I can use the same file copy routine for all of
the threads
running at the same time.

CopyFilesandDirs() etc.

I hope I explained this with enough detail, let me number a few things.

1) Go through the listview and find each checked item.
2) Copy each checked item using a different thread to the destination dir.
3) Only run about 3-4 threads at once, no more than that.

Appreciate any help guys.

John

Hi, John

I think multithreaded implementation in this case won't benefit much,
because bottleneck in this case is hard drive but not CPU, unless you
are copying from/to many physical drives.
I suspect in some cases it can be even slower.

Nonetheless, to implement multithreaded copy I would create a Queue
where I put
all items to be copied. Then I would create 3-4 threads, which will
run the same routine.
This routine will take an item from the Queue in loop and perform
corresponding copy operation.
If there's no items left in the Queue the routine (thread) will
finish. You will need to lock the queue
when you unqueue items from it and probably you will also want to wait
until all threads stop.

Thanks,
Sergey

 

[John Rogers]

Appreciate it Sergey,

My knowledge on threads is pretty limited. I guess I will try to figure out
how to have my
thread go through one item at a time in the listview and then do the
copying.

Thanks again.


John-



I think multithreaded implementation in this case won't benefit much,
because bottleneck in this case is hard drive but not CPU, unless you
are copying from/to many physical drives.
I suspect in some cases it can be even slower.

Nonetheless, to implement multithreaded copy I would create a Queue
where I put
all items to be copied. Then I would create 3-4 threads, which will
run the same routine.
This routine will take an item from the Queue in loop and perform
corresponding copy operation.

 

[Anand]

John

You dont need to use threads for what you are doing.

Delegates is another way to do this.

I am pasting a sample code that I put together.

Hope it helps.

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using System.IO;
using System.Threading;

namespace BackupDirectories
{
public partial class Form1 : Form
{
delegate void backuphandler(String item);
Queue<String> foldersQueue;
static Int32 numthreads;
backuphandler backuphandlerdelegate;
public Form1()
{
InitializeComponent();
backuphandlerdelegate = new backuphandler(backup);
label1.Text = String.Empty;
foldersQueue = new Queue<String>();
numthreads = 0;
}

private void button1_Click(object sender, EventArgs e)
{
this.button1.Enabled = false;
this.listView1.Clear();
// go through the parent for each folder
foreach(String dir in Directory.GetDirectories(@"C:\Temp"))
{
listView1.Items.Add(dir);
listView1.Refresh();
// queue the folders
foldersQueue.Enqueue(dir);
}

// have a progress bar
progressBar1.Maximum = listView1.Items.Count;
progressBar1.Value = 0;

// dequeue the entry and do BeginInvoke which will create worker
thread that will run asynchronously
if (listView1.Items.Count > 0)
{
lock (this.foldersQueue)
{
while (foldersQueue.Count > 0)
{
// I never seem to hit this mainly because the
worker thread completes real quick
if (numthreads > 4) continue;
String folder = foldersQueue.Dequeue();
label2.Text = folder;
label2.Refresh();
BeginInvoke(backuphandlerdelegate, new object[] {
folder });
}
}
}
}

// i dont have a recursive call here. but you can call your
recursive method inside backup method here
private void backup(String folder)
{
++numthreads;
this.listView1.Refresh();
Directory.CreateDirectory(@"C:\temp\backup");
int totalitems = listView1.Items.Count;
int cnt;
lock (this.foldersQueue)
{
cnt = totalitems - foldersQueue.Count;
}
String newpath = Path.Combine(@"C:\temp\backup",
Path.GetFileName(folder));

// show the current folder that is being processed
richTextBox1.AppendText(newpath);
richTextBox1.AppendText(Environment.NewLine);
richTextBox1.Refresh();

// have a text that shows what item we are on against total items
label1.Text = "Loading. Please wait... " + cnt.ToString() + "/"
+ totalitems;
label1.Refresh();

// update the progress bar for visual feedback
if (cnt < progressBar1.Maximum)
progressBar1.Value = cnt;
else
{
progressBar1.Value = progressBar1.Maximum;
label1.Text = "Complete loading";
label1.Refresh();
this.button1.Enabled = true;
}

// I dont have a recursive call, I just create the folder that
is given under the new path c:\temp\backup
Directory.CreateDirectory(newpath);
--numthreads;
}

// when I click on the Form UI, I would like it to be refreshed
private void Form1_Click(object sender, EventArgs e)
{
this.Refresh();
}
}
}

 

[John Rogers]

Thanks a lot Anand,

Now I know how to do this with more than one thread.


John-