Help with a long running process?

[Kyote]

I'm trying to make, what I thought, would be a simple application.
This application will help me organize files I have a lot of, on my
computer(ebooks, pictures, etc..). I'm currently dealing with over
25,000 files in a couple hundred directories.

I'm currently trying to collect the list using this

Try
For Each foundfile As String In
My.Computer.FileSystem.GetFiles(mySource,
FileIO.SearchOption.SearchAllSubDirectories, "*.*")
Application.DoEvents()
listOfFiles.Add(foundfile)
Next
Catch ex As System.UnauthorizedAccessException
MessageBox.Show(ex.Message & vbNewLine & "Please choose another
source folder")
End Try

But with only the 25,000 files I currently have, I get the clr warning
message. I put the Application.DoEvents() in hoping to solve the
problem but it doesn't. So I stepped through the code and it is
hanging on the for each line.

I've been googling around and I think I can probably solve my problem
by using a backgroundworker. But after reading some on the use of it
I'm simply lost. Heck, I'm not even sure I need to use the
backgroundworker to solve the problem. But from what I've read it
should be able to solve the problem I'm having.

If there's simple way to do what I need could someone tell me about
it?

But also, could someone please give a full, short example, for a
novice to vb.net, to using a backgroundworker to list all files using
for each basing the example on my code above? Something I can plug
into a new app and step through to gain some understanding of the
backgroundworkers usage? From what I've read it seems like it could be
very very useful to know how to use it.

 

[RobinS]

I'm not really sure what your problem is, you never say, other than "I get
the clr warning message". That's not specific enough to provide you with
some help.

This has an example of using the background worker.

http://msdn2.microsoft.com/en-us/library/system.componentmodel.backgroundworker.aspx

Robin S.

 

[Branco Medeiros]

I'm trying to make, what I thought, would be a simple application.

Join the club... =)))

This application will help me organize files I have a lot of, on my
computer(ebooks, pictures, etc..). I'm currently dealing with over
25,000 files in a couple hundred directories.

I'm currently trying to collect the list using this

Try
For Each foundfile As String In
My.Computer.FileSystem.GetFiles(mySource,
FileIO.SearchOption.SearchAllSubDirectories, "*.*")
Application.DoEvents()
listOfFiles.Add(foundfile)
Next
Catch ex As System.UnauthorizedAccessException
MessageBox.Show(ex.Message & vbNewLine & "Please choose another
source folder")
End Try

But with only the 25,000 files I currently have, I get the clr warning
message. I put the Application.DoEvents() in hoping to solve the
problem but it doesn't. So I stepped through the code and it is
hanging on the for each line.

<snip>

Before the loop enters its initial cicle, it will first have to grab
all the files from all sub folders off the base path you specified.
You can't put a DoEvents into this, cause it all will happen in the
GetFiles() instruction. Also, there's not much sense to put this in a
loop unless you want to do something with each file.

One possible way to execute what you want is just to assign the array
of file names directly to listOfFiles (which I assume is a List(Of
String)) or return it as an array.

Nevertheless, if you want to have some control over the loop, you may
consider loading the files folder by folder:

<aircode>
Sub LoadFiles(ByVal Folder As String, _
ByVal List As List(Of String))
Dim Items() As String = _
System.IO.Directory.GetFiles(Folder)
ShowProgress(Folder, Items)
Application.DoEvents() '<-- Ops...
List.AddRange(Items)
For Each SubFolder As String _
In System.IO.Directory.GetDirectories(Folder)
Try
LoadFiles(SubFolder, List)
Catch Ex As Exception
End Try
Next
End Sub
</aircode>

As you can see, it's a recursive method that calls itself for each sub
folder. The 'Ops...' remark near DoEvents is just to remind you that
there are better ways to keep the UI alive during a long operation
(threads or the background worker, for instance). Anyway, this may
become your starting point...

HTH.

Regards,

Branco.

 

[Kyote]

I'm not really sure what your problem is, you never say, other than "I get
the clr warning message". That's not specific enough to provide you with
some help.

Sorry Robin. I thought thatwas enough to show the problem, along with the code snippet, I was
having with the long running process timing out, and giving me the
message\warning. All I can say is I was extremely tired and apparently
wasn't thinking too clearly at the time. But the other reply seems to
be what I think I need to solve the problem. But thank you for trying.

This has an example of using the background worker.

http://msdn2.microsoft.com/en-us/library/system.componentmodel.backgroundworker.aspx

Robin S.

Also, thank you very much for this link. I'm sure it will help me
understand what I was wanting to understand about the
backgroundworker. I have an older version of the msdn installed on
here and, I guess, being tired, I simply didn't think to check the
online msdn for the info. The good thing, I went to bed shortly after,
which I should have done much sooner. Thank you very much for the
info.

 

[Kyote]

Thank you very very much Branco. This does indeed look like what I
need to solve the problem.

<snip>

Before the loop enters its initial cicle, it will first have to grab
all the files from all sub folders off the base path you specified.

This is what I was having the trouble with. I had hoped there was
something I didn't know about that would allow me to do a DoEvents in
there. As I'm constantly finding situations where there are more
elegant and much simpler ways to accomplish some of the things I'm
writing code for.

You can't put a DoEvents into this, cause it all will happen in the
GetFiles() instruction. Also, there's not much sense to put this in a
loop unless you want to do something with each file.

Thats exactly what I was doing. I was parsing the filenames, for each
file, into smaller segments. Then adding all bits dealing with that
filename into a class to store the parsed info. I was then adding each
class to a List.

One possible way to execute what you want is just to assign the array
of file names directly to listOfFiles (which I assume is a List(Of
String)) or return it as an array.

Nevertheless, if you want to have some control over the loop, you may
consider loading the files folder by folder:

<aircode>
Sub LoadFiles(ByVal Folder As String, _
ByVal List As List(Of String))
Dim Items() As String = _
System.IO.Directory.GetFiles(Folder)
ShowProgress(Folder, Items)
Application.DoEvents() '<-- Ops...
List.AddRange(Items)
For Each SubFolder As String _
In System.IO.Directory.GetDirectories(Folder)
Try
LoadFiles(SubFolder, List)
Catch Ex As Exception
End Try
Next
End Sub
</aircode>

As you can see, it's a recursive method that calls itself for each sub
folder. The 'Ops...' remark near DoEvents is just to remind you that
there are better ways to keep the UI alive during a long operation
(threads or the background worker, for instance). Anyway, this may
become your starting point...

I had initially started with something fairly similar to this. Then I
found a supposedly simpler way to do this and changed it around. But
that was back when I was only dealing with about 5,000-10,000
filenames. I completely forgot about it. Thank you for pointing it out
to me. This should solve the problem.

Join the club... =)))

LMAO, I'm happy to see it's not just me. Speaking of which, how do
you, more effectively, deal with program idea/concept improvements?

I often think of a simple app I'd like to make to simplify something I
want to do. Then while I'm writing away at it I get the 'Brillant'
idea for an improvement, or added feature. Something that can
'supposedly' make the app so much better than my original idea would
have been. But after a couple of those I sometimes get pretty confused
and frustrated.

Is there some kind of best practice or something for managing new
idea's/features? I guess I can stubbornly stick to my original app
specifications. Then once it's finished maybe then go back and revise
them. But is there a better way?

 

[Branco Medeiros]

Kyote wrote:

I often think of a simple app I'd like to make to simplify something I
want to do. Then while I'm writing away at it I get the 'Brillant'
idea for an improvement, or added feature. Something that can
'supposedly' make the app so much better than my original idea would
have been. But after a couple of those I sometimes get pretty confused
and frustrated.

The pitfall of adding new (and usually unrelated) features before the
main goal has been achieved is a sure recipe to stall a project. I've
been tricked into this many times, seeing projects that were to last a
few months actually never see the light of day. An extreme example
was a team project here that never passed the glorified login screen,
with everybody getting lost in the multitude of options and features
the app was suppose to provide.

But this is specially tricky when you happen to be the developer,
manager *and* designer of the application, as often is the case here
at my current job.

Is there some kind of best practice or something for managing new
idea's/features? I guess I can stubbornly stick to my original app
specifications. Then once it's finished maybe then go back and revise
them. But is there a better way?

The KISS principle is valuable, to some extent (Keep It Simple,
Stupid). But you'll probably benefit more from formal approaches, such
as eXtreme Programming: test (yes, test before develop!), develop,
run. Rinse, refactor and repeat, adding more features.

I'm sure there are many more knowledgeable people here that can
provide advice on this...

Regards,

Branco.

 

[RobinS]

It's not a problem. Apparently Branco was able to interpolate and figure
out what your warning message was. He's brilliant that way.

Good luck with the background worker.

Now go get some sleep. ;-)

Robin S.
-------------------------------------