too many small files, cause the machines very slow response

  • Thread starter Thread starter Guest
  • Start date Start date
G

Guest

I am using a XP machine to archive all the document in the company. The
solutions work very well intiially.

When the machine collect a large number of documents, the machine response
time is very horrorable, expectaly in Explorer.

Is ther any way to improve the performance by tunning the XP?
What is the alternatives to setup a machines that will contains alot of
small documents?
 
Kam said:
I am using a XP machine to archive all the document in the company. The
solutions work very well intiially.

When the machine collect a large number of documents, the machine response
time is very horrorable, expectaly in Explorer.

Is ther any way to improve the performance by tunning the XP?
What is the alternatives to setup a machines that will contains alot of
small documents?
Click to expand...

Limit the number of files per folder to less than 5,000, by
creating a number of suitably named subfolders.
 
Quite impossible. As over 5000 invoices per day.
Is there any way to tune the machine to improve the performance?\


If I put it in window 2003 server, do u think the responsible time will be
good.

or

Linux?
 
Kam said:
Quite impossible. As over 5000 invoices per day.
Is there any way to tune the machine to improve the performance?\
If I put it in window 2003 server, do u think the responsible time will be
good.
or

Linux?
Click to expand...

With "over 5000 invoices per day", you obviously need to implement a
database solution.
Windows XP is not a database solution.
Steve
 
Until you have implemented the database solution
recommended by Kam, create one folder for every
day. It may be cumbersome but it will make things
manageable. I can let you have a little batch file to
split up the existing folder.
 
With "over 5000 invoices per day", you obviously need to implement a
database solution.
Windows XP is not a database solution.
Steve
Click to expand...

hear, hear.
And posting this question on this newsgroup is not enough either.
Sounds you should get in touch with some IT consultant, to choose an
administrative software solution.
Your problems may very well be a lot bigger than you think.
Get professional help from someone you know.
 
Osiris said:
hear, hear.
And posting this question on this newsgroup is not enough either.
Sounds you should get in touch with some IT consultant, to choose an
administrative software solution.
Your problems may very well be a lot bigger than you think.
Get professional help from someone you know.
Click to expand...

I have a related question to MS experts on this list. There are
thousands of files much smaller than 4 kb in Windows XP. Many are even
zero byte. Each one of them will occupy 4096 bytes on the computer.
Does this not take away a little from the relative efficiency of NTFS
as compared to FAT32 and FAT? E.g., boot.ini on my computer is 211
bytes, and occupies 4096 bytes. In FAT32 and FAT a similar file would
take up much less space. I might add that I have a lot of pdf files on
my computer, downloaded from the Net with Getright or Opera 9. I like
to keep a log of the original URL in a text file, which is naturally
quite small. I have been zipping the pairs of pdf and txt files as a
single zip file, believing that it will save hard disk space. Am I
right?
 
I have a related question to MS experts on this list. There are
thousands of files much smaller than 4 kb in Windows XP. Many are even
zero byte. Each one of them will occupy 4096 bytes on the computer.
Does this not take away a little from the relative efficiency of NTFS
as compared to FAT32 and FAT? E.g., boot.ini on my computer is 211
bytes, and occupies 4096 bytes. In FAT32 and FAT a similar file would
take up much less space. I might add that I have a lot of pdf files on
my computer, downloaded from the Net with Getright or Opera 9. I like
to keep a log of the original URL in a text file, which is naturally
quite small. I have been zipping the pairs of pdf and txt files as a
single zip file, believing that it will save hard disk space. Am I
right?
Click to expand...

sure, zipping a lot of small files will save disk space.
As always, there is a down with the up: You cannot access the file
quickly.
Making the minimum of 4096 smaller will have an adverse effect on the
size of the FAT. It is more costly to search a large FAT for a file
entry, than with a smaller FAT.
So 4096 makes the disk less efficient, space-wise, but faster
access-wise.

store the URL in the file's SUMMARY ?
 
Until you have implemented the database solution
recommended by Kam, create one folder for every
day. It may be cumbersome but it will make things
manageable. I can let you have a little batch file to
split up the existing folder.
Click to expand...

or zip em ? but then... than may slow down the processing of the
invoices.
 
witan said:
I have a related question to MS experts on this list. There are
thousands of files much smaller than 4 kb in Windows XP. Many are even
zero byte. Each one of them will occupy 4096 bytes on the computer.
Does this not take away a little from the relative efficiency of NTFS
as compared to FAT32 and FAT?
Click to expand...

In NTFS, small files are entirely stored in the MFT. Does that mean
that a file smaller then the cluster size can live in there with no
slack space? I don't know.
I like to keep a log of the original URL in a text file, which is naturally
quite small. I have been zipping the pairs of pdf and txt files as a
single zip file, believing that it will save hard disk space. Am I
right?
Click to expand...

Other than the "small files in the MFT" question above, yes. Actually,
this would be a really nifty application for NTFS's alternate data
streams. Again, I don't know just what the slack space implications
would be.

NTFS is not openly documented like the FAT systems. There are lots of
articles about many facets of NTFS, but (AFAIK) no publicly available
definitive spec. The closest I've seen is this:
http://www.linux-ntfs.org/content/view/103/42/

It's an open-source effort to reverse-engineer NTFS so that Linux can
be enabled to use NTFS partitions.
 
witan said:
Does this not take away a little from the relative efficiency of NTFS
as compared to FAT32 and FAT? E.g., boot.ini on my computer is 211
bytes, and occupies 4096 bytes. In FAT32 and FAT a similar file would
take up much less space.
Click to expand...

Uh, if I remember correctly, with my FAT32 drive on Win98 a one-byte
file used 64k of disk space. You might get below 4k on a really small
FAT32 disk, but in my experience with modern disks (i.e. 100GB+) NTFS
is far more efficient... moving the files to an NTFS disk after
installing XP on that machine saved me a couple of gigabytes.

Mark
 
Uh, if I remember correctly, with my FAT32 drive on Win98 a one-byte
file used 64k of disk space.
Click to expand...


How much space it used depended on how big the partition was, but in
Windows 98, the maximum cluster size was 32KB, not 64KB.

You might get below 4k on a really small
FAT32 disk,
Click to expand...


Yes, *really* small. You don't get cluster sizes below 4K until you get
below 256MB partition sizes. Almost nobody has such a small partition these
days.

but in my experience with modern disks (i.e. 100GB+) NTFS
is far more efficient...
Click to expand...


Ture, but my view is that NTFS is a much bettter and more reliable file
system, and that's the reason it should be preferred. In these days of very
inexpensive hard drives, efficiency considerations (although they are
present) should be ignored and decisions should never be made on that basis.

moving the files to an NTFS disk after
installing XP on that machine saved me a couple of gigabytes.
Click to expand...


The reason I say to ignore efficiency considerations is that they don't
amount to a hill of beans. You talk about 100GB+ hard drives. A 120GB hard
drive can be bought for under $70US. Two GB may sound like a lot, but at
that price the couple of gigabytes you saved are worth only $1.17.
 
Uh, if I remember correctly, with my FAT32 drive on Win98 a one-byte
file used 64k of disk space.
Click to expand...

If your clusters were 64KB, yes that's exactly what would happen.
You might get below 4k on a really small
FAT32 disk, but in my experience with modern disks (i.e. 100GB+) NTFS
is far more efficient... moving the files to an NTFS disk after
installing XP on that machine saved me a couple of gigabytes.
Click to expand...

Cluster sizes are more flexible in FAT32 than FAT16, but for larger
partitions you're going to need larger clusters. You're absolutely
right, NTFS makes better use of your disk space.
 
Size is not a problem (the OP said they were small) but the
sheer number.
Click to expand...

zipping would reduce number drastically (as in putting 1 days invoces
in one .zip)...
But anyway, a DB system is the way to go. there should be hardly any
human interference in his system... no retyping...
 
Back
Top