NEWBIE: XP handling 10s of 1000s of data files vs 98SE?

[Cogwheel]

Hi,

Considering moving to XP (now using Win98SE - PIII 500, 512RAM, 40 GB
HD).

I am working with digital images and vector mapping data, giving me
10s of 1000s of data files (close to 100,000)- images around 1Mb and
vector map files around 1kb ea.

It seems that Win98SE just can't handle large numbers of files - it
just grinds to a halt. If I point explorer to a directory with 10,000
files it stalls while it "thinks about it". It doesn't crash, but a
simple action takes minutes. Defrags take hours.

How is XP at handling large numbers of files?

Any advice appreciated.

Ken

 

[Lord Gazwad of Grantham]

Cogwheel wrote:
| Hi,
|
| Considering moving to XP (now using Win98SE - PIII 500, 512RAM, 40 GB
| HD).
|
| I am working with digital images and vector mapping data, giving me
| 10s of 1000s of data files (close to 100,000)- images around 1Mb and
| vector map files around 1kb ea.
|
| It seems that Win98SE just can't handle large numbers of files - it
| just grinds to a halt. If I point explorer to a directory with 10,000
| files it stalls while it "thinks about it". It doesn't crash, but a
| simple action takes minutes. Defrags take hours.
|
| How is XP at handling large numbers of files?
|
| Any advice appreciated.
|
| Ken

If you value your data you should buy a new PC, install XP (preferably pro)
and use NTFS, you would also be well advised to configure the machine so you
have a smallish partition just to run windows and a separate partition for
your data. You may also wish to consider a backup plan.

Keep the old PC intact until you are happy with the new one.

You should find that although there may be a slight lag opening a folder
with a large number of files it will almost go unnoticed, especially in
comparison to your old system.

--
Gazwad

Freelance scientist and people tester.
Guardian: alt.os.windows-xp
Moderator: alt.warez.uk
www.gazwad.com

 

[*Vanguard*]

"Cogwheel" said in news:[email protected]:

Hi,

Considering moving to XP (now using Win98SE - PIII 500, 512RAM, 40 GB
HD).

I am working with digital images and vector mapping data, giving me
10s of 1000s of data files (close to 100,000)- images around 1Mb and
vector map files around 1kb ea.

It seems that Win98SE just can't handle large numbers of files - it
just grinds to a halt. If I point explorer to a directory with 10,000
files it stalls while it "thinks about it". It doesn't crash, but a
simple action takes minutes. Defrags take hours.

How is XP at handling large numbers of files?

Any advice appreciated.

Ken

I don't have problems opening folders under Windows 2000/XP that have
over 15,000 files. However, do NOT use thumbnail view. This results in
updating the thumbs.db file and that can take a long time. If you must
use thumbnail view, use the option to cache the database with preloaded
images if the contents of the folder are rather static. If the contents
change a lot then the cache is worthless and you'll end up spending as
much time waiting for the database to get updated as if it were cleared
and had to start caching all over again.

 

[Rob Schneider]

What you are seeing is how Explorer works with files, not necessarily
the operating system.

My hunch is that that many files will probably tax Explorer in XP also.
In order to display the files names (and information about the files),
Explorer has to read each and every file. It will take the time it
takes to do that. More files, more time. I suspect (but do not know)
there is caching going on, and i suspect it's got some level of
sophistication to make it as fast as reasonably possible, but you are
talking about a lot of files.

Since this is a key issue for you, perhaps a way to check this out for
your own satisfaction prior to moving to XP is to run some experiments
on an XP machine. Create a 100,000 files (with a batch file or
something) and test performance. See if it meets your expectations.
Compare performance with what you have.

For kicks, I created 100,000 14k files on my little low-spec laptop
(Toshiba 1135) running XP Pro with NTFS. Perhaps this comparision will
be of interest to you. It took about 35 minutes to create the files (a
simple copy of a base 14k file into 100,000 new files with other
programs running while I was using Mozilla to read today's paper
online), and about 190 seconds for Explorer to diplay all of them in
when I clicked on the folder the first time. Doing a refresh in
Explorer took the same (about) 190 seconds. Doing a "dir" in a command
prompt box took about 175 seconds to gather data and display all files.

Your machine is not particularly speedy compared to contemporary
machines. I suspect the key variable which will affect perfomance will
be to get a high-performance disk.

Also, consider using folders to segregate the files to speed up the
"information gathering" phase of Explorer Works (or any file managment
program that does it a folder at a time). That's why folders were
invented. You may also find that there are more sophisticated file
management programs available for handling file systems that are this large.

If you do decide to upgrade to XP, ensure to check out the specs of your
machine by running the test programs available on Microsoft's XP web
site to test suitability of the machine for XP.

Hope this is useful to you. Let us know.

rms

 

[SQLit]

Hi,

Considering moving to XP (now using Win98SE - PIII 500, 512RAM, 40 GB
HD).

I am working with digital images and vector mapping data, giving me
10s of 1000s of data files (close to 100,000)- images around 1Mb and
vector map files around 1kb ea.

It seems that Win98SE just can't handle large numbers of files - it
just grinds to a halt. If I point explorer to a directory with 10,000
files it stalls while it "thinks about it". It doesn't crash, but a
simple action takes minutes. Defrags take hours.

How is XP at handling large numbers of files?

Any advice appreciated.

Ken

Have you considered a clean install of SE and then removing all of the icons
on the desktop. Removing everything that you can from the start up group.
Reboot..
How much free hard disk do you have? When you get close to 90% full windoze
gets pissy. I do not think XP is going to give you much of a boost. When
was the last time you defragged? I use disk keeper, I think it is much
better. Consider using a different view instead of the thumb nail if that
is what your using. Last suggestion start creating new folders and organize
the files into 4-25k folders see if that helps. Are you connected to a LAN
when your doing this? I shut my network connect off when burning movies. It
helps.

 

[Cogwheel]

Rob

I've put your name forward to the Queen (here in UK) for service above
and beyond the call of duty.....


VERY interesting. I neglected to say in my original post, that the
100,000 files do no reside in one directory, but when I run disk
doctor (Norton scandisk) which is often, it sees all images together
and DD stops when it hits images and grinds aways for minutes. The
smaller vector map files do end-up in one directory, due to software
restrictions. Here is where I run into a bit of what you did,
explorer grinding away. Except my system falls over 2/3s of the time.

I haven't had trouble with thumbnails, per se, because they are
divided into numerous directories. I am just now trying-out the
latest ACDSee (6.03) which has a much improved database system for
thumbnails which now handles up to about 70,000 images. ACDSee,
however, are selling what is really a beta version of the software as
the finished product, so it is quite unstable.

Your test gives me a lot to think about in planning a new system.

THANX a lot,

Ken

 

[Cogwheel]

Have you considered a clean install of SE and then removing all of the icons
on the desktop. Removing everything that you can from the start up group.
Reboot..
How much free hard disk do you have? When you get close to 90% full windoze
gets pissy. I do not think XP is going to give you much of a boost. When
was the last time you defragged? I use disk keeper, I think it is much
better. Consider using a different view instead of the thumb nail if that
is what your using. Last suggestion start creating new folders and organize
the files into 4-25k folders see if that helps. Are you connected to a LAN
when your doing this? I shut my network connect off when burning movies. It
helps.

What a great group. Thanks everyone. I have a lot to think about in
structuring a new system.

I neglected to mention in my first post that all of my files do not
reside in one directory. The map vector files - in the 1000s - end-up
in one directory, due to software limitations. This causes explorer
problems.

Based on your comments, I now think that XP and a large faster HD will
solve my problems.

I do use thumbnails, but my images are divided into numerous
directories. I am using the new ACDSee 6.03 for viewing thumbnails.
It has a very much improved image database system which can handle
over 70,000 thumbnails. The problem is that they have released BETA
software as a full release; it is very unstable.

Thanks again,

Ken

 

[Cogwheel]

Rob,

This may be a repeat. I was in the middle of sending you a reply when
I was interrupted and I am not sure if it was sent.

I wanted to say thanx......... Your test is VERY helpful. I now
have a clear idea of how to setup a new system.

I think that XP will work for me, especially with a larger, faster HD
and a bit better organized system of directories.

Thanks Rob,

Ken

 

[Alex Nichol]

I am working with digital images and vector mapping data, giving me
10s of 1000s of data files (close to 100,000)- images around 1Mb and
vector map files around 1kb ea.

It seems that Win98SE just can't handle large numbers of files - it
just grinds to a halt. If I point explorer to a directory with 10,000
files it stalls while it "thinks about it". It doesn't crash, but a
simple action takes minutes. Defrags take hours.

How is XP at handling large numbers of files?

It is an aspect where moving to NTFS will probably help a lot. FAT 32
is limited by having no real order in directories, so that any access to
a file has to be a linear scan in the directory. that being said,
organising your material in subfolders so as *not* to have so many items
in a single directory would probably help more.