Slow file opening with thousands of files in folder

[k1w1]

Scenario is as follows: A small peer network of 5 computers, 4 XP Pro, 1
Win2000. Recently installed new computers, previously 4 Win98's and the 2000
computer.

There is an application installed on the main computer that acts as a
server. This program allows the scanning of documents into a folder that are
then able to be opened through the default program ie Windows Picture and
Fax Viewer on the XP computers.

This folder contains over 10 500 files. On the XP computers it takes ~30
seconds to open a file. The Win2000 and 98 computers (when they were there)
open the files virtually instantly.

The memory usage on XP goes haywire, jumping to up around 1GB of page file
usage when browsing to the folder.

Presumably XP is trying to do some caching on this directory. (Thumbnails
not enabled).
Is there a way to disable this to speed up the viewing of the files over the
network?

Thanks.

 

[Guest]

Why do you have all those file stored in ONE directory. I, personally, think
that it would be better in every aspect if those files were spread out
through multiple directories.

-Chris

 

[k1w1]

Because that's what the program does with them. I agree, it's not ideal, but
that is what we have to work with.

 

[Galen]

In k1w1 had this to say:

My reply is at the bottom of your sent message:

Scenario is as follows: A small peer network of 5 computers, 4 XP
Pro, 1 Win2000. Recently installed new computers, previously 4
Win98's and the 2000 computer.

There is an application installed on the main computer that acts as a
server. This program allows the scanning of documents into a folder
that are then able to be opened through the default program ie
Windows Picture and Fax Viewer on the XP computers.

This folder contains over 10 500 files. On the XP computers it takes
~30 seconds to open a file. The Win2000 and 98 computers (when they
were there) open the files virtually instantly.

The memory usage on XP goes haywire, jumping to up around 1GB of
page file usage when browsing to the folder.

Presumably XP is trying to do some caching on this directory.
(Thumbnails not enabled).
Is there a way to disable this to speed up the viewing of the files
over the network?

Thanks.

XP (and 2k should too actually) will try to process the information with
them and show thumbnails and cache them. If they're opened often and the
likes then (and I'm sure you can disable thumbnail caching entirely if you
wanted) caching the files would be good in *most* instances. Instead - and
this is going to sound tweaked - I'd consider the following?

First... Are the XP boxed all set to be the master browser? If so (check a
search engine) then consider alternatives - more so if there's a DHCP server
running and a router today generally handles DCHP just fine so you
*probably* don't need that. There's a guy named Chuck in here who can likely
give you more information on that.

The second is a bit more difficult and going to sound truly insane is to run
explorer in it's own separate instance of memory. Often times the desktop
will be using it's own memory as well as the taskbar and those are a part of
the explore process. Sometimes, oddly, making explorer run in two instances
(where one is the desktop) is a solution for stuff similar to this though
I've not seen it specifically in your case.

Explorer as a Separate Process:
http://kgiii.info/windows/XP/tips/sep_explorer.html

--
Galen - MS MVP - Windows (Shell/User & IE)
http://dts-l.org/
http://kgiii.info/

"I am glad of all details, whether they seem to you to be relevant or
not." - Sherlock Holmes

 

[Chuck]

In k1w1 had this to say:

XP (and 2k should too actually) will try to process the information with
them and show thumbnails and cache them. If they're opened often and the
likes then (and I'm sure you can disable thumbnail caching entirely if you
wanted) caching the files would be good in *most* instances. Instead - and
this is going to sound tweaked - I'd consider the following?

First... Are the XP boxed all set to be the master browser? If so (check a
search engine) then consider alternatives - more so if there's a DHCP server
running and a router today generally handles DCHP just fine so you
*probably* don't need that. There's a guy named Chuck in here who can likely
give you more information on that.

The second is a bit more difficult and going to sound truly insane is to run
explorer in it's own separate instance of memory. Often times the desktop
will be using it's own memory as well as the taskbar and those are a part of
the explore process. Sometimes, oddly, making explorer run in two instances
(where one is the desktop) is a solution for stuff similar to this though
I've not seen it specifically in your case.

Explorer as a Separate Process:
http://kgiii.info/windows/XP/tips/sep_explorer.html

That's not a typical browser issue.

I've seen that in here before though. I'm thinking this article references it,
if not, it still contains good advice:
http://support.microsoft.com/kb/816375
"... you can minimize network traffic by implementing a well-structured folder
organization on your file servers ..."

I've been using the DesktopProcess registry tweak, as well as the
NoRemoteRecursiveEvents, for a while. IIRC, I started from the OPs scenario,
but I was nowhere near 10K files in one folder. I think that's the first change
that I would make - structure the folders!!!

 

[Guest]

In short - NO! (You should try it over a dial-up connection, Ughhh! ;>)
You could use remote desktop sharing to avoid opening the folder across the
LAN,
or move the files into sub-folders as previously suggested.

 

[Guest]

What exactly does this program do? I know you mentioned it, but could you
clarify for me?

-Chris

 

[Galen]

In Chuck had this to say:

My reply is at the bottom of your sent message:

That's not a typical browser issue.

I've seen that in here before though. I'm thinking this article
references it, if not, it still contains good advice:
http://support.microsoft.com/kb/816375
"... you can minimize network traffic by implementing a
well-structured folder organization on your file servers ..."

I've been using the DesktopProcess registry tweak, as well as the
NoRemoteRecursiveEvents, for a while. IIRC, I started from the OPs
scenario, but I was nowhere near 10K files in one folder. I think
that's the first change that I would make - structure the folders!!!

I just loaded up a bunch of old images (about 2.3 GB worth of family photos)
at 11,000+ files and opened it via the network... Why, why oh why would
someone author an application without folder structuring to ensure the data
is at least categorized and able to be in multiple segmented folders
(ideally with a good naming structure) is beyond me.

--
Galen - MS MVP - Windows (Shell/User & IE)
http://dts-l.org/
http://kgiii.info/

"I am glad of all details, whether they seem to you to be relevant or
not." - Sherlock Holmes

 

[Chuck]

In Chuck had this to say:

I just loaded up a bunch of old images (about 2.3 GB worth of family photos)
at 11,000+ files and opened it via the network... Why, why oh why would
someone author an application without folder structuring to ensure the data
is at least categorized and able to be in multiple segmented folders
(ideally with a good naming structure) is beyond me.

Maybe the OP will enlighten us.

 

[k1w1]

Exact Dental http://www.soeidental.com/Default.aspx?PageID=5

It's a Dental Practise management program. The issue with the files are
scanned documents, x-rays etc that corresponde to patients. The program
keeps all of these files in a single "data" directory From within the
program you can check out the patitent's records, and call up these
documents.

 

[k1w1]

How did you go opening the files? Was there a delay?

I've posted a response back to Christoper Isherwood giving a few more
details about the program in question. I have no argument what so ever that
having so many files in one folder isn't a smart move. That's what I have to
work with, and that's what was working fine until the computers were
upgraded. The files are all linked to patients and moving them would
obiously break those linkages.

Speaking with the software helpdesk (in New Zealand) gave no great insight.
This isn't exactly the largest dental practise in the world, the population
here is under 10 000 people, so I'd have thought it would have been more of
an issue for them elsewhere.

The network setup, I forgot to mention is statically assigned IP addresses.
There's no router, and for what it's worth, no internet connection.

I'll have a look into your suggestions during the week ahead.

Thanks

 

[q_q_anonymous]

Scenario is as follows: A small peer network of 5 computers, 4 XP Pro, 1
Win2000. Recently installed new computers, previously 4 Win98's and the 2000
computer.

There is an application installed on the main computer that acts as a
server. This program allows the scanning of documents into a folder that are
then able to be opened through the default program ie Windows Picture and
Fax Viewer on the XP computers.

This folder contains over 10 500 files. On the XP computers it takes ~30
seconds to open a file. The Win2000 and 98 computers (when they were there)
open the files virtually instantly.

The memory usage on XP goes haywire, jumping to up around 1GB of page file
usage when browsing to the folder.

Presumably XP is trying to do some caching on this directory. (Thumbnails
not enabled).
Is there a way to disable this to speed up the viewing of the files over the
network?

Thanks.

dunno really. Is it the "My Documents" folder or a subfolder within it?

That is a special kind of directory , i've heard that filling it with
too many files can be bad.

I would check if this problem occurs with Any directory.
How well would it deal with a directory with opening one enormous text
file
how quickly does the directory list all the files?

whether the answers are yes or no to those questions, I am not sure
what to conclude, only a better understanding of the problem.

maybe a more advanced task manger can deal with tellign you exactly
what process or DLL is going crazy with the processor usage.

you haven't mentioned what OS is running on the main computer. 2K or
XP.
If it's 2k, maybe install XP

 

[Galen]

In k1w1 had this to say:

My reply is at the bottom of your sent message:

How did you go opening the files? Was there a delay?

I've posted a response back to Christoper Isherwood giving a few more
details about the program in question. I have no argument what so
ever that having so many files in one folder isn't a smart move.
That's what I have to work with, and that's what was working fine
until the computers were upgraded. The files are all linked to
patients and moving them would obiously break those linkages.

Speaking with the software helpdesk (in New Zealand) gave no great
insight. This isn't exactly the largest dental practise in the world,
the population here is under 10 000 people, so I'd have thought it
would have been more of an issue for them elsewhere.

The network setup, I forgot to mention is statically assigned IP
addresses. There's no router, and for what it's worth, no internet
connection.
I'll have a look into your suggestions during the week ahead.

Thanks

<snip>

I went ahead and trimmed some of this to make sure that I'm saving a few
bytes of transfer for people. Those who followed it know what we're on about
anyhow.

I'd really consider the explorer as a separate process tweak on BOTH systems
involved. Chuck's listed another one that I have never tried.

Anyhow, I loaded up the images - some of them were quite large - and then
meandered off to browse them via another PC. To ensure that I would be in a
worst-case situation I disabled the wireless and used the old wired CAT5e
cable. It was cut at 100' so that should be about the limit of what most
people would be using for a wired connection.

Opening the directory that contained the folder was where it began to stall.
Explorer appeared to freeze for about 30 seconds. The network is a simple
10/100 and the PCs involved are all newer than a year. Unfortunately, I'm a
goober, I had it set so that it tried to show me thumbnails once inside the
folder... That was, shall we say, one of the most insane processes ever and
CPU usage spiked and Explorer nearly croaked at that point but lots of RAM
saved the day and in about a minute the files began to appear. (There are a
LOT of images in there...) After they processed to thumbnail images I tried
scrolling down and found that only the first few screen widths had been
processed and again had to wait.

The explorer as a separate process might at least prevent it from crashing
and it's my opinion that if this keeps up you will likely find the system
crashing and the desktop suddenly will disappear. Running the desktop as a
separate process will, at least, prevent that.

The mass folder storage is simply unacceptable. My ideas aren't even going
to help with that as that's a grossly arrogant (negligent too) way of
programming and the author should consider revising their application. You
might consider cracking open the folder itself and having a look at the
details view. Inside there you may find that there are some files that can
(at least) be moved to another directory manually. You may need to train the
end-user on finding them or consider an alternative but that's about the
only real solution that I can give at this time. There may be (and again I'd
refer to that Chuck guy - believe it or not I have his messages
automatically marked in a brown color because he's far more knowledgeable
than I will ever be with networking issues I'm sure) a cache tweak that you
can us? Mapping the folder as a drive with automatically logging on might
even make some sense if that's a viable option. Really though I'd look for
older (sorry but dead too) client information that can be moved. Note that I
don't say deleted because at least in this country there are regulations
that require data remain in storage for varied lengths of time depending on
the industry. Moving duplicated data or data less likely to be used to
another folder (add a note to remember to include that folder in backup
cycles) is a bit of a process but one that will have the greatest impact
other than changing software applications, writing in-house, or requesting
updates from the author.

--
Galen - MS MVP - Windows (Shell/User & IE)
http://dts-l.org/
http://kgiii.info/

"I am glad of all details, whether they seem to you to be relevant or
not." - Sherlock Holmes

 

[k1w1]

In k1w1 had this to say:

My reply is at the bottom of your sent message:

<snip>

I went ahead and trimmed some of this to make sure that I'm saving a few
bytes of transfer for people. Those who followed it know what we're on
about anyhow.

I'd really consider the explorer as a separate process tweak on BOTH
systems involved. Chuck's listed another one that I have never tried.

Anyhow, I loaded up the images - some of them were quite large - and then
meandered off to browse them via another PC. To ensure that I would be in
a worst-case situation I disabled the wireless and used the old wired
CAT5e cable. It was cut at 100' so that should be about the limit of what
most people would be using for a wired connection.

Opening the directory that contained the folder was where it began to
stall. Explorer appeared to freeze for about 30 seconds. The network is a
simple 10/100 and the PCs involved are all newer than a year.
Unfortunately, I'm a goober, I had it set so that it tried to show me
thumbnails once inside the folder... That was, shall we say, one of the
most insane processes ever and CPU usage spiked and Explorer nearly
croaked at that point but lots of RAM saved the day and in about a minute
the files began to appear. (There are a LOT of images in there...) After
they processed to thumbnail images I tried scrolling down and found that
only the first few screen widths had been processed and again had to wait.

The explorer as a separate process might at least prevent it from crashing
and it's my opinion that if this keeps up you will likely find the system
crashing and the desktop suddenly will disappear. Running the desktop as a
separate process will, at least, prevent that.

The mass folder storage is simply unacceptable. My ideas aren't even going
to help with that as that's a grossly arrogant (negligent too) way of
programming and the author should consider revising their application. You
might consider cracking open the folder itself and having a look at the
details view. Inside there you may find that there are some files that can
(at least) be moved to another directory manually. You may need to train
the end-user on finding them or consider an alternative but that's about
the only real solution that I can give at this time. There may be (and
again I'd refer to that Chuck guy - believe it or not I have his messages
automatically marked in a brown color because he's far more knowledgeable
than I will ever be with networking issues I'm sure) a cache tweak that
you can us? Mapping the folder as a drive with automatically logging on
might even make some sense if that's a viable option. Really though I'd
look for older (sorry but dead too) client information that can be moved.
Note that I don't say deleted because at least in this country there are
regulations that require data remain in storage for varied lengths of time
depending on the industry. Moving duplicated data or data less likely to
be used to another folder (add a note to remember to include that folder
in backup cycles) is a bit of a process but one that will have the
greatest impact other than changing software applications, writing
in-house, or requesting updates from the author.

--
Galen - MS MVP - Windows (Shell/User & IE)
http://dts-l.org/
http://kgiii.info/

"I am glad of all details, whether they seem to you to be relevant or
not." - Sherlock Holmes

Thanks for that Galen, that seems to confirm everything.
The links Chuck has provided, especially the one that lead onto registry
settings for disabling file caching looks promising. I'll go and try that
during their lunch break today. Failing that, I'll go and get back onto the
support desk. I can't believe this isn't an issue elsewhere.

 

[k1w1]

I tried the infocache level registry change
http://support.microsoft.com/kb/816375 this afternoon. It made no
difference.

I spoke again to the help desk, and didn't get very far. They informed me
that they have around 3 sites with the same issue. The person I spoke to
informed me that he knew of a site with over 20 000 files in a single
directory, and all was working fine. He did admit that they weren't running
windows XP at that place.

I installed Irfan view as an alternative to the MS Picture/Fax viewer. This
opened the files instantly! But you couldn't close them for ~30 secs At
the suggestion of the help desk I'm going to try ACDSee tomorrow.

 

[Guest]

I can't understand why the computers would go bonkers on the paging and
memory usage, but consider MS KB article Q177266. It affects large directory
reads on many computers. Nearly every peer-to-peer network I've used with
NT4/2000/XP has this issue. Unfortunately, the SizeReqBuf trick has yet to
solve it for me, but it may work for you.

Dave