Explorer Eating Memory When Viewing VERY Large Folders

[BIGGERPLUM]

Hi,

I am working with a network moving very large amounts of data round and
am finding that windows explorer jsut goes nuts on memory. If you try
viewing the top level of a large folder structure then it jsut hangs
for ages and makes the machine unusable and from task manager you can
see explorer is jsut using large amounts of memory (eg . 475MB on a 512
machine).

Now even after you stop browsing that folder structure periodically
explorer will just shoot up again and lock the machine, im wondering
whether it is trying to make an index for this massive folder
structure.

The folder structure trying to be viewed is 250,000 folders with alot
of smaller files.

My question is there a fix that allows windows to handle this amount of
data better? (without having to use unix)

Adrian

PS: This problem occurs on XP pro machines with 2G of memoery, Windows
Server 2003 with 4G of memory and a Cut down version of 2003 for a
files server with 512M of memory

Any ideas or comments gratefully recieved

 

[Betelgeuse]

You might try this registry tweak for one thing:
Windows Explorer - File Access and Datestamps.
Whenever a directory on an NTFS drive is accessed a time stamp is added.
Improve the spped of file access and turn this facility off.

a.. Locate the following key in the registry:
'HKEY_LOCAL_MACHINE\System\CurrentControlet\Control\Fileystem'
b.. Create a new DWORD value 'NtfsDisableLastAccesUpdate'
c.. Set the value to

 

[BIGGERPLUM]

Thanks , that is a really good idea , that could definatly speed things
up with teh amount of files

 

[Gerry Cornell]

You could look at your problem from the opposite direction! Why do I
need to have so many large files?

Linking files unnecessarily can also be problematic. I encountered this
problem one time using Excel. You can use a macro to update
information thereby avoiding permanent links which load into memory.

--

Hope this helps.

Gerry
~~~~
FCA
Stourport, England

Enquire, plan and execute
~~~~~~~~~~~~~~~~~~~

 

[Alec S.]

You could look at your problem from the opposite direction! Why do I
need to have so many large files?

Linking files unnecessarily can also be problematic. I encountered this
problem one time using Excel. You can use a macro to update
information thereby avoiding permanent links which load into memory.

They didn't say anything about large files, just a large number of files and folders. As to your question, that's just how some
programs are. I've seen lots of big programs (Macromedia apps, Maya, etc.) that come with tens, even hundreds of thousands of tiny
html files in hundreds or thousands of folders. Usually they comprise the extensive documentation scattered amongst thousands of
files (which eat up slack space like crazy!) instead of one large consolidated file (like hlp, chm, pdf, doc, etc.)

 

[Gerry Cornell]

Alec

Files sitting in a folder will not "eat memory" -they will only eat
memory if they are loaded into memory at the same time! If files
are linked, as can be with Excel, then small files when combined
can be a drain on limited available memory!

If a user is viewing files in a folder and does not close a file
after seeing the contents then this will also waste available
memory.

My comments are directed to the original question

"My question is there a fix that allows windows to handle this amount of
data better? (without having to use unix)"

The original question was asking about using existing memory
more efficiently whereas I am suggesting the user makes the
task less demanding of memory.

--

Regards.

Gerry
~~~~
FCA
Stourport, England

Enquire, plan and execute
~~~~~~~~~~~~~~~~~~~

 

[Alec S.]

Files sitting in a folder will not "eat memory" -they will only eat
memory if they are loaded into memory at the same time! If files
are linked, as can be with Excel, then small files when combined
can be a drain on limited available memory!

I meant on the disk. When you've got thousands of small files, the slack space—the unused part of the last cluster of each file—can
add up to quite a bit of wasted space, especially with the larger cluster sizes that are used these days (8KB clusters are the
minimum for partitions>50GB). Of course I'm talking FAT here, but if I remember correctly, NTFS has waste as well.

If a user is viewing files in a folder and does not close a file
after seeing the contents then this will also waste available memory.

Explorer is pretty bad in the way it handles files. It leaves open handles, allocated memory, etc. That's why the memory usage of
explorer.exe goes up and up over a session, and files (particularly graphics files) become locked and cannot be deleted.

My comments are directed to the original question

"My question is there a fix that allows windows to handle this amount of
data better? (without having to use unix)"

The original question was asking about using existing memory
more efficiently whereas I am suggesting the user makes the
task less demanding of memory.

I was referring to your question "You could look at your problem from the opposite direction! Why do I
need to have so many large files?" It may not be up to the user at all; it is not necessarily the design that the OP has created,
but rather the way that a program they use ships, and they are getting peeved by the usage caused by the thousands of files that the
program has.

 

[Ken Blake, MVP]

I meant on the disk. When you've got thousands of small files, the
slack space-the unused part of the last cluster of each file-can add
up to quite a bit of wasted space,

Note that, despite what many people think, the size of the file is almost
completely irrelevant in this regard. Unless the file is an exact multiple
of the cluster size, on the average, the last cluster wastes half a cluster.

especially with the larger cluster
sizes that are used these days (8KB clusters are the minimum for
partitions>50GB).

FAT32 clusters larger than 32GB have, by default, 32KB clusters

Of course I'm talking FAT here, but if I remember
correctly, NTFS has waste as well.

Yes, that's correct.

Back in the days of small expensive hard drives, the waste due to slack was
an important consideration. But in these days of very large inexpensive hard
drive, the actual value of that waste, in dollars and cents, doesn't amount
to a hill of beans, and can be safely ignored by most people.

 

[Alec S.]

Note that, despite what many people think, the size of the file is almost
completely irrelevant in this regard. Unless the file is an exact multiple
of the cluster size, on the average, the last cluster wastes half a cluster.

Bingo. Of course that's a universal, global, theoretical really, average. In practice it could be anything since no two systems
are alike.

FAT32 clusters larger than 32GB have, by default, 32KB clusters

I'm not sure about the default, it depends on the partitioning software (some use one, some use another), but there are definite
minimums depending on the size of the partiton. In fact the size of the FAT itself could (in the past) become and issue. A few
years ago I had to make a partition of 49.99GB (or something like that) to avoid moving up one whole cluster size for just a few
hundred extra meg in the partition. Moving up a size would have been quite determinetal since it was for games which tend to have
thousands of small files. I actually tested it and there was more than ½GB of slack with the larger cluster size; that's a whole
extra game! :| (I don't remember what the record I've seen is.)

Yes, that's correct.

Back in the days of small expensive hard drives, the waste due to slack was
an important consideration. But in these days of very large inexpensive hard
drive, the actual value of that waste, in dollars and cents, doesn't amount
to a hill of beans, and can be safely ignored by most people.

It's worth thinking about (considering what will be on it) for a new drive, but for existing ones, the time it takes to change it,
and the risk of data loss usually make it less than desirable. Although, if you are running low on space, those extra couple of
gigabytes being wasted amongst the partitions can look awfully good.

 

[Gerry Cornell]

Alec

Please see inline:


Snip

I meant on the disk. When you've got thousands of small files, the slack
space—the unused part of the last cluster of each file—can
add up to quite a bit of wasted space, especially with the larger cluster
sizes that are used these days (8KB clusters are the
minimum for partitions>50GB). Of course I'm talking FAT here, but if I
remember correctly, NTFS has waste as well.

The comments by Alec Nicholl present a balanced view:

Considerations of economy and performance

In a virtual memory system like Windows XP, the ideal size of disk clusters
matches the internal “page size” used by the Intel processors — 4 kilobytes.
An NTFS partition of almost any size you are likely to meet will use this,
but it is only used in FAT32 up to an 8 GB partition. Above that, the
cluster size for FAT increases, and the wastefulness of the “cluster
overhang” grows. (For a table of the varying default cluster sizes used by
FAT16, FAT32, and Win XP’s version of NTFS, for partitions of varying sizes,
click here.)

On the other hand NTFS takes much more space for holding descriptive
information on every file in that file’s own block in the Master File Table
(MFT). This can use quite a large proportion of the disk, though this is
offset by a possibility that the data of a very small file may be stored
entirely in its MFT block. Because NTFS holds significant amounts of these
structures in memory, it places larger demands on memory than does FAT.

Searching directories in NTFS uses a more efficient stucture for its access
to files, so searching a FAT partition is a slower process in big
directories. Scanning the FAT for the pieces of a fragmented file is also
slower. On the other hand, NTFS carries the overhead of maintaining the
“journalized” recovery.

http://aumha.org/win5/a/ntfs.htm

Adrian does not say how his disk is formatted or the size of the disk /
partition. He also does not say whether the folder is on his computer or on
another computer in the network!

Explorer is pretty bad in the way it handles files. It leaves open
handles, allocated memory, etc. That's why the memory usage of
explorer.exe goes up and up over a session, and files (particularly
graphics files) become locked and cannot be deleted.

Not my forte. Do open handles, allocated memory, etc impact significantly on
system performance when the system does not resort to using virtual save for
specific tasks requiring virtual memory?

I was referring to your question "You could look at your problem from the
opposite direction! Why do I
need to have so many large files?" It may not be up to the user at all;
it is not necessarily the design that the OP has created,
but rather the way that a program they use ships, and they are getting
peeved by the usage caused by the thousands of files that the
program has.

True. Again Adrian does not mention what programme created the files and
unfortunately neither of us thought to ask.


--

Regards.

Gerry
~~~~
FCA
Stourport, England

Enquire, plan and execute
~~~~~~~~~~~~~~~~~~~