Garbage Collection/Release RAM and Handles

  • Thread starter Thread starter Guest
  • Start date Start date
G

Guest

I have an application that collects statistics on usage of storage on our
servers - number of directories, number of files, size of files, allocated
by various categories. I originally developed the app using the VBA folder
objects. It worked quite well except for being excruciatingly slow (we have
several terabytes of data to be analyzed). In the hope of speeding it up, I
recoded it using the Windows API file management functions instead of the VBA
folder objects. On some smaller subsets, I got correct results and
significantly improved performance (roughly 3 times as fast), but on the
largest set the application crashed. After some study I isolated the problem
to being a basic RAM cram situation. I finally created my own folder object
as a VBA class module, in the naive hope that the VBA documentation would be
correct - when the object goes out of scope the resources are released.
Didn't help. I have also added the Windows functions to close files and
handles, but that doesn't help either. In fact the resources don't get
released even when the form is closed; the resources are not released until
Access itself is closed.

Any ideas on how to cope with this situation would be greatly appreciated.
 
Ummm... You can't always trust VBA to release things just because they
go out of scope, and I suspect that this may be especially true when
you're working - as I guess you are - with handles to lots of API
structures. So make sure that you explicitly close everything you open.
For instance, do you have a FindClose() for every FindFirstFile()?

Faced with this task myself, I'd have thought in terms of writing Perl
code to execute something like this
ls -s -R -1 > filelist.txt
or
DIR /S > filelist.txt
for each server, and then parse the resulting files and assemble the
stats. But chacun à son gout.
 
John:
Yes, I have noticed that you can't trust VBA for much of anything. Yes, I
do issue a FindClose as part of a cleanup routine when I am done with my
object. The FindClose returns a success code, but there is no change either
in open handles or in memory usage, despite its claim of success. (Perhaps
it should run for office.) It would never have occurred to me to try Perl,
since I don't know it, but something like that might offer a rather
convoluted work around. I do, however, have over 30 million files in more
than 600k directories to traverse, with all data being stored within an
Access table for analysis. Quindi mi piacce farlo così.

Thank you for your input.
Peter
 
If the FindClose is returning a success code, I can't see how you can
blame Access/VBA for anything. You call a windows api. Windows
holds on to the findfirst handle even after you use a Windows api to
tell Windows to release the handle.

Not that this is particularly surprising:

"The NTFS file system delays updates to the last access time for
a file by up to one hour after the last access. "

How do you think NTFS knows to update the "last access time"
an hour after you have released a file? Of course it keeps a handle.

BTW, here is some one using Perl who seems to have had
a similar problem:
http://www.opensubscriber.com/message/[email protected]/464580.html

(david)
 
Why do that as a class object? Just write a single procedure that uses
the relevant win32 API calls to navigate recursively through all the
folders & files on the specified drive (or whatever). I've done this
for other purposes, and it works fine - though I don't have anywhere
near "terabytes" of data. And it should work waaaaaaaaaaaaaay faster
than 3 times the equivalent VBA code!

What is a "basic RAM cram" situation?

HTH,
TC
 
The original version was a single proceduure. Worked very well for
relatively small sets of data (e.g., 30K folders and 3 - 4 million files
total within those folders). I have one back breaker set, however, with more
than 600K directories. The app choked when processing this one. The class
object was simply a desperate attempt to contrive a work around to the
problem of the resources not being released. Since there seems to be no
native garbage collection I was attempting to fashion some myself.
Interestingly, when you use the VBA folder objects for this purpose,
resources do get released, so evidently they do some garbage collection.

The "Ram Cram" situation is simply the build-up of open file handles and
allocated RAM. It appears that Windows allocates a 2K block of RAM for each
handle you open, and even though I am explicitly closing (or attempting to
close) the handles when I am done with them, the RAM doesn't get released.
At a certain point, Access crashes. The problem may be purely an Access
issue, since at the time of crash there still appears to be RAM available. I
haven't yet pinned down a specific point where the crash happens (takes
several hours of run time to get there), so I can't tell if it may be related
to Access's memory addressing or something like that.

Thanks for your input. I appreciate your taking the time. Happy New Year!
Peter
 
<speculation mode="idle">
Have you tried some strategically placed calls to DoEvents?
</speculation>
 
Back
Top