FAT32 much faster than NTFS!??

[Guest]

I have a 100 GB (FAT32) drive full with images organized
in one main directory with a couple of subdirectories,
which in turn have more subdirectories.

I copied the whole drive content (directory tree) to an
identical 100 GB drive but formated in NTFS.

The first thing I noticed was that while it only took 24
sec to find out the directory size (right click directory
and properties) on the FAT32 drive, it took several
minutes on the NTFS drive (I cancelled before done).

I next opened an image directory (using ACDSee to display
the files in the directory) on the FAT32 drive, which took
8.7 sec. In contrast, opening the identical directory on
the NTFS drive took 50.6 sec!

I disabeled the indexing on the NTFS drive, but it did not
help (not sure even if it is relevant).

What is going on here? Should I convert all my drives to
FAT32?

p.s. There are ~20,000 files in each subdirectory. Maybe
FAT32 is faster for large file collections?

 

[BT]

p.s. There are ~20,000 files in each subdirectory. Maybe

FAT32 is faster for large file collections?

It does seem to be that way for you. You've benchmarked the two FS's on your system with your individual needs and have a
determination which file system to use. I don't think a generic inference can be made though. But then again, you may have
discovered a replicatable issue that could be of significance.

How you copied the contents may have something to do with diminished performance. did you defrag the FAT drive prior so that there
was contigency? Have you cheked fragmentation of the NTFS drive? Which drive is the paging file on?

Come to think of it, for you to answer your own question, suggest reformatting the NTFS. The open some files with an app, change
the files and save them to each drive see how they benchmark? There are very strong arguments for using each file system. I do use
a FAT and an NTFS drive for different needs.

 

[Dmitry Korolyov [MVP]]

Were the FAT and NTFS disks attached to the same channel of the same controller when you did the benchmark? There might be hardware issues in the first place behind it all, if you used different controllers and channels - disk access modes (PIO/DMA) may be set differently for different controller channels, your antivirus (if any) might have been configured to monitor drive D but not monitor drive E, etc. Also, partition fragmentation plays a big role in access time.

--
Dmitry Korolyov [[email protected]]
MVP: Windows Server - Active Directory


I have a 100 GB (FAT32) drive full with images organized
in one main directory with a couple of subdirectories,
which in turn have more subdirectories.

I copied the whole drive content (directory tree) to an
identical 100 GB drive but formated in NTFS.

The first thing I noticed was that while it only took 24
sec to find out the directory size (right click directory
and properties) on the FAT32 drive, it took several
minutes on the NTFS drive (I cancelled before done).

I next opened an image directory (using ACDSee to display
the files in the directory) on the FAT32 drive, which took
8.7 sec. In contrast, opening the identical directory on
the NTFS drive took 50.6 sec!

I disabeled the indexing on the NTFS drive, but it did not
help (not sure even if it is relevant).

What is going on here? Should I convert all my drives to
FAT32?

p.s. There are ~20,000 files in each subdirectory. Maybe
FAT32 is faster for large file collections?

 

[Dmitry Korolyov [MVP]]

Interesting. What is the average file size in your case?
Try to reformat the NTFS drive explicitly defining 16k cluster size (the default used by FAT afair; for NTFS it's 4k, but varies depending on overall partition size), and compare the results with the initial test.

--
Dmitry Korolyov [[email protected]]
MVP: Windows Server - Active Directory


Thank you BT and Dmitry for you comments.
I have now done more experiments. Note that the NTFS
partition is defragmented, and I am using identical (WD)
drives. I have tried connecting the drives as external
firewire drives (on a 700 MHz P3, W2K machine) and on the
Raid controller on my Abit motherboard (2 GHz P4, XP
machine; 80 pin cables). The results are qualitatively
very similar on both machines and OSs.

I am reporting the results on the XP machine here. In all
cases ACDSee was used to open a directory and list all the
18,864 image files (2.1 GB; detail view). The times listed
are measured with a stopwatch from the time I dbl-clicked
on the directory until the file listing displayed.

Start with:
FAT32 drive on controller ch1
NTFS drive on controller ch2

Open dir on NTFS drive => 59.1 sec
Open dir on FAT32 drive => 3.5 sec
(in this order)
**Reboot
Open dir on FAT32 drive => 3.3 sec
Open dir on NTFS drive => 59.2 sec
** Power down
** Swap Cables at the drive (FAT32 on ch2; NTFS on ch1)
Open dir on FAT32 drive => 3.5 sec
Open dir on NTFS drive => 58.8 sec

**Disable 8.3 DOS names (fsutil behavior set disable8dot3)
**Reboot and verify change
Open dir on NTFS drive => 58.7 sec
Open dir on FAT32 drive => 3.3 sec

On *another* pair of drives (also copied FAT32 files to
NTFS) I had very similar results, so it is not a problem
with this particular drive (which is brand new by the
way). Furthermore, when I reformatted the NTFS drive to
FAT32 the performance of the drive increased and became
identical to the original FAT32 drive.

Thus, I am convinced that FAT32 is able to display a
directory listing of many thousands of files much faster
(**almost 20 times faster**) than NTFS. Note that the
results are very similar when displaying the files using
Windows Explorer [as when using ACDSee].

I have no idea what other performance differences there
may be between FAT32 and NTFS. I doubt that file transfer
rates etc would be different though.

p.s. I am not running Antivirus (I have a clean XP
install). The page file and OS is residing on my 3rd hard
drive.

 

[Rolf]

Thank you BT and Dmitry for you comments.
I have now done more experiments. Note that the NTFS
partition is defragmented, and I am using identical (WD)
drives. I have tried connecting the drives as external
firewire drives (on a 700 MHz P3, W2K machine) and on the
Raid controller on my Abit motherboard (2 GHz P4, XP
machine; 80 pin cables). The results are qualitatively
very similar on both machines and OSs.

I am reporting the results on the XP machine here. In all
cases ACDSee was used to open a directory and list all the
18,864 image files (2.1 GB; detail view). The times listed
are measured with a stopwatch from the time I dbl-clicked
on the directory until the file listing displayed.

Start with:
FAT32 drive on controller ch1
NTFS drive on controller ch2

Open dir on NTFS drive => 59.1 sec
Open dir on FAT32 drive => 3.5 sec
(in this order)
**Reboot
Open dir on FAT32 drive => 3.3 sec
Open dir on NTFS drive => 59.2 sec
** Power down
** Swap Cables at the drive (FAT32 on ch2; NTFS on ch1)
Open dir on FAT32 drive => 3.5 sec
Open dir on NTFS drive => 58.8 sec

**Disable 8.3 DOS names (fsutil behavior set disable8dot3)
**Reboot and verify change
Open dir on NTFS drive => 58.7 sec
Open dir on FAT32 drive => 3.3 sec

On *another* pair of drives (also copied FAT32 files to
NTFS) I had very similar results, so it is not a problem
with this particular drive (which is brand new by the
way). Furthermore, when I reformatted the NTFS drive to
FAT32 the performance of the drive increased and became
identical to the original FAT32 drive.

Thus, I am convinced that FAT32 is able to display a
directory listing of many thousands of files much faster
(**almost 20 times faster**) than NTFS. Note that the
results are very similar when displaying the files using
Windows Explorer [as when using ACDSee].

I have no idea what other performance differences there
may be between FAT32 and NTFS. I doubt that file transfer
rates etc would be different though.

p.s. I am not running Antivirus (I have a clean XP
install). The page file and OS is residing on my 3rd hard
drive.

 

[Ndi]

Have you also tried to bench how Explorer behaves displaying the files? Or
another tool?

This might have something to do how ACDSee accesses files. I'd try to
"dir" while redirecting to a file (listing in console is very time
consuming). You might find that the dir is very, very fast. In that case
maybe ACDSee is trying to optimize somthing, failing. The speed is bench'd
in FF/FN iterations and additional processing is not exactly accurate.

Secondly, try to bench subsequent "dir" commands, when cache is at work.

I did a dir > temp.txt on my hive (2x60G - mounted) and it took just 76
secs to iterate all files and dump the text file. File is 12 MB in size and
lists:

Total Files Listed:
198452 File(s) 66,846,815,432 bytes
26619 Dir(s) 2,997,366,784 bytes free

That's 200.000 files in 25.000 folders and 80 seconds. It's quite fast.

True, however, I average 10 files per folder. My largest folders don't
exceed 5.000 files. Here, caching makes folder display qute fast after first
attempt. First attempt takes about 3-4 seconds.

 

[Rolf]

Andrei,

Yes, I also tried Windows Explorer (as described in the message above). The
timings are almost identical. Furthermore, as I wrote originally, just
checking directory properties is painstakingly slow on the NTFS formated
drive.

If I try to to do a file list again (e.g. on the NFTS) it is very fast
because of the cashing.

The first time I run Dir E:\directory1\subdirectory > E.txt (i.e. after
reboot - no cashing)
It took 1.5 sec on the FAT32 drive and 20.4 sec on the NTFS drive...

Still a huge difference in speed between FAT32 and NTFS....

(Next I will try to reformat the NTFS in 32 K clusters instead of 4 K..)

 

[Rolf]

Hi again Dmitry,
The average file size (per defrag report) is 157 KB.
My FAT32 drive uses 32 KB clusters, so I decided to try a NTFS partion using
32 KB cluste size as well (instead of the previous default 4 KB).

Then performed the same test as above....
Indeed, while it took 59 sec to open the directory using 4KB clusters, it
only took 33 sec to open it using 32KB clusters (again drive was not
fragmented). Surprised that it even helped!

Still, this is a far cry from the 3 sec it takes on a 32 KB cluster size
FAT32, but it is going in the right direction.

Not sure that I can accept the 10 times slower NTFS though....

I suspect that the problem lies in the huge amount of files in a directory.
I wonder if changing the settings for the Master File Table (MFT) could
help?

Where would I turn to get things like this answered....?

Rolf

Interesting. What is the average file size in your case?
Try to reformat the NTFS drive explicitly defining 16k cluster size (the
default used by FAT afair; for NTFS it's 4k, but varies depending on overall
partition size), and compare the results with the initial test.

--
Dmitry Korolyov [[email protected]]
MVP: Windows Server - Active Directory


Thank you BT and Dmitry for you comments.
I have now done more experiments. Note that the NTFS
partition is defragmented, and I am using identical (WD)
drives. I have tried connecting the drives as external
firewire drives (on a 700 MHz P3, W2K machine) and on the
Raid controller on my Abit motherboard (2 GHz P4, XP
machine; 80 pin cables). The results are qualitatively
very similar on both machines and OSs.

I am reporting the results on the XP machine here. In all
cases ACDSee was used to open a directory and list all the
18,864 image files (2.1 GB; detail view). The times listed
are measured with a stopwatch from the time I dbl-clicked
on the directory until the file listing displayed.

Start with:
FAT32 drive on controller ch1
NTFS drive on controller ch2

Open dir on NTFS drive => 59.1 sec
Open dir on FAT32 drive => 3.5 sec
(in this order)
**Reboot
Open dir on FAT32 drive => 3.3 sec
Open dir on NTFS drive => 59.2 sec
** Power down
** Swap Cables at the drive (FAT32 on ch2; NTFS on ch1)
Open dir on FAT32 drive => 3.5 sec
Open dir on NTFS drive => 58.8 sec

**Disable 8.3 DOS names (fsutil behavior set disable8dot3)
**Reboot and verify change
Open dir on NTFS drive => 58.7 sec
Open dir on FAT32 drive => 3.3 sec

On *another* pair of drives (also copied FAT32 files to
NTFS) I had very similar results, so it is not a problem
with this particular drive (which is brand new by the
way). Furthermore, when I reformatted the NTFS drive to
FAT32 the performance of the drive increased and became
identical to the original FAT32 drive.

Thus, I am convinced that FAT32 is able to display a
directory listing of many thousands of files much faster
(**almost 20 times faster**) than NTFS. Note that the
results are very similar when displaying the files using
Windows Explorer [as when using ACDSee].

I have no idea what other performance differences there
may be between FAT32 and NTFS. I doubt that file transfer
rates etc would be different though.

p.s. I am not running Antivirus (I have a clean XP
install). The page file and OS is residing on my 3rd hard
drive.

 

[Tom]

Did you keep the directory structure the same?

You didn't combine all 20,000 files into one directory
did you?

I know you were dealing with NTFS and FAT32 but...
Directory processing under the old FAT system was a killer.

Let me give you and example....

I once had a half dozen zip files where each zip file
contained a few hundred icons. {small hundred byte files}
I didn't like dealing with zip files so I thought I would
unzip and put them all on a single 1.4 MB floppy.

How long would it take to fill a single directory
floppy with icon images? {remembering the root directory
had a limitation to the number of entries}
Heck it only takes a couple of minutes to format the whole
floppy.

IT TOOK SEVERAL HOURS!!!!

I started this around 10pm and wasn't finished at 2am!

Remember a sector {or cluster} will only hold a few
directory entries. Directory processing under the old FAT
was sequential. After the first few hundred files are
laid down, you are reading dozens of sectors {maybe
hundreds or thousands depending on the directory size}
just to find the next free entry.

What I learned from this is it is wise to keep directories
to a managable size...

 

[Rolf]

Hi Tom,

Yes the directory structure was the same on the FAT32 and the NTFS dives
(they were exact copies of each other). I have several directories (maybe
~200) and each one has about between 5,000-20,000 files in them. No problem
for FAT32, big problem for NTFS :-(

I will continue to collect large numbers of files in the directories. If I
have to use FAT32 for this, I will. Just surprised that NTFS is so poor in
dealing with it. Is there maybe a fix/workaround? Modify the MFT properties
maybe?

Rolf

 

[Cindy]

I've been having a problem with NTFS drives being slow as
well. This is noticably worse after completing a disk
cleanup and defrag.

Does anyone else notice this? It's driving my users (and
me) crazy.

-----Original Message-----
Thank you BT and Dmitry for you comments.
I have now done more experiments. Note that the NTFS
partition is defragmented, and I am using identical (WD)
drives. I have tried connecting the drives as external
firewire drives (on a 700 MHz P3, W2K machine) and on the
Raid controller on my Abit motherboard (2 GHz P4, XP
machine; 80 pin cables). The results are qualitatively
very similar on both machines and OSs.

I am reporting the results on the XP machine here. In all
cases ACDSee was used to open a directory and list all the
18,864 image files (2.1 GB; detail view). The times listed
are measured with a stopwatch from the time I dbl-clicked
on the directory until the file listing displayed.

Start with:
FAT32 drive on controller ch1
NTFS drive on controller ch2

Open dir on NTFS drive => 59.1 sec
Open dir on FAT32 drive => 3.5 sec
(in this order)
**Reboot
Open dir on FAT32 drive => 3.3 sec
Open dir on NTFS drive => 59.2 sec
** Power down
** Swap Cables at the drive (FAT32 on ch2; NTFS on ch1)
Open dir on FAT32 drive => 3.5 sec
Open dir on NTFS drive => 58.8 sec

**Disable 8.3 DOS names (fsutil behavior set disable8dot3)
**Reboot and verify change
Open dir on NTFS drive => 58.7 sec
Open dir on FAT32 drive => 3.3 sec

On *another* pair of drives (also copied FAT32 files to
NTFS) I had very similar results, so it is not a problem
with this particular drive (which is brand new by the
way). Furthermore, when I reformatted the NTFS drive to
FAT32 the performance of the drive increased and became
identical to the original FAT32 drive.

Thus, I am convinced that FAT32 is able to display a
directory listing of many thousands of files much faster
(**almost 20 times faster**) than NTFS. Note that the
results are very similar when displaying the files using
Windows Explorer [as when using ACDSee].

I have no idea what other performance differences there
may be between FAT32 and NTFS. I doubt that file transfer
rates etc would be different though.

p.s. I am not running Antivirus (I have a clean XP
install). The page file and OS is residing on my 3rd hard
drive.

-----Original Message-----
I have a 100 GB (FAT32) drive full with images organized
in one main directory with a couple of subdirectories,
which in turn have more subdirectories.

I copied the whole drive content (directory tree) to an
identical 100 GB drive but formated in NTFS.

The first thing I noticed was that while it only took 24
sec to find out the directory size (right click directory
and properties) on the FAT32 drive, it took several
minutes on the NTFS drive (I cancelled before done).

I next opened an image directory (using ACDSee to display
the files in the directory) on the FAT32 drive, which took
8.7 sec. In contrast, opening the identical directory on
the NTFS drive took 50.6 sec!

I disabeled the indexing on the NTFS drive, but it did not
help (not sure even if it is relevant).

What is going on here? Should I convert all my drives to
FAT32?

p.s. There are ~20,000 files in each subdirectory. Maybe
FAT32 is faster for large file collections?
.

.

 

[Daniel Billingsley]

Is everybody missing the obvious point here that NTFS *IS* a journaled file
system, after all? All else being equal (and I'm not sure how equal things
could be in this kind of test) doesn't that cost some performance?

 

[Bill Todd]

Is everybody missing the obvious point here that NTFS *IS* a journaled file
system, after all? All else being equal (and I'm not sure how equal things
could be in this kind of test) doesn't that cost some performance?

As usual, it depends.

For reading (rather than writing), journaling makes no difference.

If the disk has write-back caching enabled and it's an ATA/IDE disk,
journaling should effectively cost nothing on writes but may also lead to
file system corruption if power fails (SCSI has facilities to protect
against this) - which *may* be caught by running chkdsk on restart if NT
does this after an unclean shutdown with an IDE drive.

If NTFS were optimally implemented, journaling could well be faster than a
non-journaled file system that provided equal consistency guarantees.
Possibly even faster than a non-journaled file system like FAT (which
provides considerably weaker consistency guarantees).

But as currently implemented, NTFS probably is in general somewhat slower
than FAT for writes.

- bill

 

[fsdf]

Thank you BT and Dmitry for you comments.
I have now done more experiments. Note that the NTFS
partition is defragmented, and I am using identical (WD)
drives. I have tried connecting the drives as external
firewire drives (on a 700 MHz P3, W2K machine) and on the
Raid controller on my Abit motherboard (2 GHz P4, XP
machine; 80 pin cables). The results are qualitatively
very similar on both machines and OSs.

I am reporting the results on the XP machine here. In all
cases ACDSee was used to open a directory and list all the
18,864 image files (2.1 GB; detail view). The times listed
are measured with a stopwatch from the time I dbl-clicked
on the directory until the file listing displayed.
...

I was surprised at how slow NTFS was when I switched. I found the
file tree window in ACDsee is what slows it down. If you close the
file tree and open shortcuts instead, ACDsee will access directories
much faster, but you have to make a lot of shortcuts.

It also seems that using subdirectories helped a little.

 

[Guest]

In my opinion, the speed of NTFS and FAT32 file systems is highly dependent on your individual system configuration. You are also trying to compare an apple with an orange. To the best of my knowledge, the NTFS file system was originally designed with a coporate environment in mind. NTFS offers many features which are ideal in a corporate/business environment such as advanced indexing services, advanced security policies, integrated key-based encryption and decryption, and increased reliability. Yes, FAT32 may render faster performance, but does not offer the capabilities of NTFS.

Because FAT32 uses a smaller cluster size by default, it is also more prone to defragmentation. I have also noticed some indexing problems with FAT32 volumes (at least from within Windows 2k). When I use the command prompt to list directories and or files on a FAT32 volume, files would always be displayed out of alphabetical order. Adding the proper command extension to force alphabetical order rendered the same results. I soon reformatted the drive using NTFS and moved the files back over. Then I tried listing the same directory, and every directory and file displayed alphabetically as it should. This is critical for generating text files with such information. After that, I no longer used FAT32!