Several words on File Fragmentation and performance

[Leythos]

Over the last couple days we've had a thread about Fragmentation and
Performance, as most of the long time administrators know how it impacts
performance in the real world.

For those of you that still are unsure, where are articles explaining
it, even the mechanical aspect of it, for you to review.


http://searchwindowssecurity.techtarget.com/whitepaperPage/0,293857,sid4
5_gci997657,00.html

System performance and file fragmentation in Windows NT

Date Published:
01 OCT 1999
About the White Paper:
Contrary to early conventional wisdom about Windows NT, its file systems
do become fragmented. This fragmentation occurs in the normal course of
using the operating system. Theoretical analysis and real-world
performance testing demonstrate that fragmentation has an adverse impact
on system performance. Special characteristics of the NTFS file system,
such as the paging file, directories and the Master File Table, are
especially vulnerable to fragmentation, and allowing them to become
fragmented is a guarantee of a decrease in overall system performance.
Other NTFS features, such as file system compression, inherently create
fragmentation. The best way to avoid these worst-case fragmentation
problems, and to keep the system running at optimal performance, is to
run a defragmentation system on a regularly scheduled basis. Both
Windows NT workstations and NT servers are subject to these problems,
and both can improve system performance through regular defragmentation.

*********************

http://www.executive.com/whats-new/whitepaper.asp#_Toc463769971
NTFS Does Get Fragmented

The Windows NTFS File System Driver uses a special file called the
Master File Table (MFT) to track all files on that volume. The MFT
starts out with some free space to allow new files to be tracked, but on
a very busy system it too can run out of space. At this point NTFS
extends the MFT itself, creating new stretches of it for new
allocations. This situation is precipitated most often by fragmentation
in the file system itself, as file system fragments consume entries in
the MFT. If these new stretches are not contiguous, the MFT itself
becomes fragmented.

There are other files, such as the paging file used by Windows NT=3Fs
virtual memory subsystem, which can also become fragmented with
unpleasant implications for performance. The solution to these problems,
as we will see, it to prevent them from happening by keeping your system
defragmented.

Lastly, directories in NTFS are allocated similarly to files, but
defragmentation of them can be difficult.

Performance Degradations Can Impede Productivity

Windows NT does a good job of allowing the system to continue operation
even as programs wait for disk I/O, but some inefficiency cannot be
hidden forever. Especially on a mission-critical server, on which many
users rely, inefficiencies in the file system can lead to performance
degradation that impedes user productivity.

These problems are not always apparent, and are frequently cavalierly
blamed on other sources; perhaps the computer=3Fs just too slow, needs
more memory, or some program being run needs an upgrade. Overall system
performance is a complex phenomenon, and even experienced system
administrators may not recognize fragmentation in a file system. After
all, it can occur with large amounts of free space on the disk. But the
main reason users don=3Ft recognize fragmentation is because Windows NT
comes with no tools to identify it.

Heavily used systems, which are by definition mission-critical systems
for an organization, will become fragmented over time under normal usage
in Windows NT. As performance decreases in such systems and users are
forced to wait, productivity is thereby impeded.

*********************

http://www.microsoft.com/windows2000/techinfo/administration/fileandprin
t/defrag.asp

File Fragmentation

A file with all its parts stored in one location on a disk is described
as "contiguous." If a file is not contiguous, it=3Fs fragmented; broken
into pieces that are scattered throughout the disk. All Windows NT® and
Windows 2000 file types=3FFile Allocation Table (FAT) and NTFS file system
(NTFS)=3Fare susceptible to fragmentation.

File fragmentation has a negative effect on disk performance because the
disk head requires more time to move around to different points on the
disk to read scattered file parts. This is a primary reason for the
gradual degradation of system performance=3Fand the specific cause of
longer reads and extended reboots.

 

[David H. Lipman]

I have always described the concept of file fragmentation as thus...

Say you are reading a Newspaper. You find an article on page 2 and read a couple of
paragraphs and it then says "go to page 5". You go to page 5 and read another couple of
paragraphs and it now says "go to page 21". You go to page 21 and read a few more
paragraphs and it now says "go to page 12". You go to page 12 and read a few more
paragraphs and the article is completed.

Defragmentation is akin to taking all those paragraphs on other pages and assembling them
all in one contiguous article on page 2.

--
Dave




Over the last couple days we've had a thread about Fragmentation and
Performance, as most of the long time administrators know how it impacts
performance in the real world.

For those of you that still are unsure, where are articles explaining
it, even the mechanical aspect of it, for you to review.


http://searchwindowssecurity.techtarget.com/whitepaperPage/0,293857,sid4
5_gci997657,00.html

System performance and file fragmentation in Windows NT

Date Published:
01 OCT 1999
About the White Paper:
Contrary to early conventional wisdom about Windows NT, its file systems
do become fragmented. This fragmentation occurs in the normal course of
using the operating system. Theoretical analysis and real-world
performance testing demonstrate that fragmentation has an adverse impact
on system performance. Special characteristics of the NTFS file system,
such as the paging file, directories and the Master File Table, are
especially vulnerable to fragmentation, and allowing them to become
fragmented is a guarantee of a decrease in overall system performance.
Other NTFS features, such as file system compression, inherently create
fragmentation. The best way to avoid these worst-case fragmentation
problems, and to keep the system running at optimal performance, is to
run a defragmentation system on a regularly scheduled basis. Both
Windows NT workstations and NT servers are subject to these problems,
and both can improve system performance through regular defragmentation.

*********************

http://www.executive.com/whats-new/whitepaper.asp#_Toc463769971
NTFS Does Get Fragmented

The Windows NTFS File System Driver uses a special file called the
Master File Table (MFT) to track all files on that volume. The MFT
starts out with some free space to allow new files to be tracked, but on
a very busy system it too can run out of space. At this point NTFS
extends the MFT itself, creating new stretches of it for new
allocations. This situation is precipitated most often by fragmentation
in the file system itself, as file system fragments consume entries in
the MFT. If these new stretches are not contiguous, the MFT itself
becomes fragmented.

There are other files, such as the paging file used by Windows NT=3Fs
virtual memory subsystem, which can also become fragmented with
unpleasant implications for performance. The solution to these problems,
as we will see, it to prevent them from happening by keeping your system
defragmented.

Lastly, directories in NTFS are allocated similarly to files, but
defragmentation of them can be difficult.

Performance Degradations Can Impede Productivity

Windows NT does a good job of allowing the system to continue operation
even as programs wait for disk I/O, but some inefficiency cannot be
hidden forever. Especially on a mission-critical server, on which many
users rely, inefficiencies in the file system can lead to performance
degradation that impedes user productivity.

These problems are not always apparent, and are frequently cavalierly
blamed on other sources; perhaps the computer=3Fs just too slow, needs
more memory, or some program being run needs an upgrade. Overall system
performance is a complex phenomenon, and even experienced system
administrators may not recognize fragmentation in a file system. After
all, it can occur with large amounts of free space on the disk. But the
main reason users don=3Ft recognize fragmentation is because Windows NT
comes with no tools to identify it.

Heavily used systems, which are by definition mission-critical systems
for an organization, will become fragmented over time under normal usage
in Windows NT. As performance decreases in such systems and users are
forced to wait, productivity is thereby impeded.

*********************

http://www.microsoft.com/windows2000/techinfo/administration/fileandprin
t/defrag.asp

File Fragmentation

A file with all its parts stored in one location on a disk is described
as "contiguous." If a file is not contiguous, it=3Fs fragmented; broken
into pieces that are scattered throughout the disk. All Windows NT® and
Windows 2000 file types=3FFile Allocation Table (FAT) and NTFS file system
(NTFS)=3Fare susceptible to fragmentation.

File fragmentation has a negative effect on disk performance because the
disk head requires more time to move around to different points on the
disk to read scattered file parts. This is a primary reason for the
gradual degradation of system performance=3Fand the specific cause of
longer reads and extended reboots.

 

[Ron Martell]

There are other files, such as the paging file used by Windows NT=3Fs
virtual memory subsystem, which can also become fragmented with
unpleasant implications for performance. The solution to these problems,
as we will see, it to prevent them from happening by keeping your system
defragmented.

Paging file fragmentation, while not totally irrelevant, is pretty
much a non-issue simply because of the way that paging is done.

Paging out is done based on lack of usage of each individual page
(4K) and there is no requirement or attempt to ensure that pages moved
out to the paging file are in any way related to each other. So if
100 pages (400k) is being moved from RAM to the paging file these 100
pages could be parts of a dozen or more different applications, device
drivers, Windows components, or data files.

If it is subsequently decided to move more items from RAM to the
paging file the items being moved could again represent parts of many
different items, and it is quite possible in fact even likely that
some pages moved out in the second instance will be from the same
items as were some of the pages moved out in the first instance.

And then when one of these items becomes active again it will be
necessary to reload the paged out pages for that item from the paging
file. This could involve pages moved out in two or more different
paging out operations and therefore the items being paged back in will
not be contiguous even if the paging file is totally unfragmented.


Ron Martell Duncan B.C. Canada
--
Microsoft MVP
On-Line Help Computer Service
http://onlinehelp.bc.ca

"The reason computer chips are so small is computers don't eat much."

 

[Raymond J. Johnson Jr.]

Over the last couple days we've had a thread about Fragmentation and
Performance, as most of the long time administrators know how it impacts
performance in the real world.

For those of you that still are unsure, where are articles explaining
it, even the mechanical aspect of it, for you to review.


http://searchwindowssecurity.techtarget.com/whitepaperPage/0,293857,sid4
5_gci997657,00.html

System performance and file fragmentation in Windows NT

Date Published:
01 OCT 1999
About the White Paper:
Contrary to early conventional wisdom about Windows NT, its file systems
do become fragmented. This fragmentation occurs in the normal course of
using the operating system. Theoretical analysis and real-world
performance testing demonstrate that fragmentation has an adverse impact
on system performance. Special characteristics of the NTFS file system,
such as the paging file, directories and the Master File Table, are
especially vulnerable to fragmentation, and allowing them to become
fragmented is a guarantee of a decrease in overall system performance.
Other NTFS features, such as file system compression, inherently create
fragmentation. The best way to avoid these worst-case fragmentation
problems, and to keep the system running at optimal performance, is to
run a defragmentation system on a regularly scheduled basis. Both
Windows NT workstations and NT servers are subject to these problems,
and both can improve system performance through regular defragmentation.

*********************

http://www.executive.com/whats-new/whitepaper.asp#_Toc463769971
NTFS Does Get Fragmented

The Windows NTFS File System Driver uses a special file called the
Master File Table (MFT) to track all files on that volume. The MFT
starts out with some free space to allow new files to be tracked, but on
a very busy system it too can run out of space. At this point NTFS
extends the MFT itself, creating new stretches of it for new
allocations. This situation is precipitated most often by fragmentation
in the file system itself, as file system fragments consume entries in
the MFT. If these new stretches are not contiguous, the MFT itself
becomes fragmented.

There are other files, such as the paging file used by Windows NT=3Fs
virtual memory subsystem, which can also become fragmented with
unpleasant implications for performance. The solution to these problems,
as we will see, it to prevent them from happening by keeping your system
defragmented.

Lastly, directories in NTFS are allocated similarly to files, but
defragmentation of them can be difficult.

Performance Degradations Can Impede Productivity

Windows NT does a good job of allowing the system to continue operation
even as programs wait for disk I/O, but some inefficiency cannot be
hidden forever. Especially on a mission-critical server, on which many
users rely, inefficiencies in the file system can lead to performance
degradation that impedes user productivity.

These problems are not always apparent, and are frequently cavalierly
blamed on other sources; perhaps the computer=3Fs just too slow, needs
more memory, or some program being run needs an upgrade. Overall system
performance is a complex phenomenon, and even experienced system
administrators may not recognize fragmentation in a file system. After
all, it can occur with large amounts of free space on the disk. But the
main reason users don=3Ft recognize fragmentation is because Windows NT
comes with no tools to identify it.

Heavily used systems, which are by definition mission-critical systems
for an organization, will become fragmented over time under normal usage
in Windows NT. As performance decreases in such systems and users are
forced to wait, productivity is thereby impeded.

*********************

http://www.microsoft.com/windows2000/techinfo/administration/fileandprin
t/defrag.asp

File Fragmentation

A file with all its parts stored in one location on a disk is described
as "contiguous." If a file is not contiguous, it=3Fs fragmented; broken
into pieces that are scattered throughout the disk. All Windows NT® and
Windows 2000 file types=3FFile Allocation Table (FAT) and NTFS file system
(NTFS)=3Fare susceptible to fragmentation.

File fragmentation has a negative effect on disk performance because the
disk head requires more time to move around to different points on the
disk to read scattered file parts. This is a primary reason for the
gradual degradation of system performance=3Fand the specific cause of
longer reads and extended reboots.

Your first two references come from a seller of defrag software (hardly
an unbiased source) and the third, from MS, includes nothing but
unsupported assertions and irrelevant (to this discussion)
generalizations. If this kind of thing impresses you, I have a good
idea now how Dubya got reelected.

 

[SlowJet]

Even though fragmentation occurs there are two amin cases where it matters
much less today than evne 3 years ago.

1. Hugh disks for the end user and very fast CPU and disks.

2. Servers have specific volumes for appiclation use, and data use.
Only those ares of high volitility need to be defragment in the back
ground, and depeneing on the volitility (20% change, 10% Add, 10% Delete)
vs. 40% change, 25% Add, 20% change) may indicate that defraging will occurs
naturely over time in a constance fragmented pattern.

The world id different now, find another hobby.



SJ
Over the last couple days we've had a thread about Fragmentation and
Performance, as most of the long time administrators know how it impacts
performance in the real world.

For those of you that still are unsure, where are articles explaining
it, even the mechanical aspect of it, for you to review.


http://searchwindowssecurity.techtarget.com/whitepaperPage/0,293857,sid4
5_gci997657,00.html

System performance and file fragmentation in Windows NT

Date Published:
01 OCT 1999
About the White Paper:
Contrary to early conventional wisdom about Windows NT, its file systems
do become fragmented. This fragmentation occurs in the normal course of
using the operating system. Theoretical analysis and real-world
performance testing demonstrate that fragmentation has an adverse impact
on system performance. Special characteristics of the NTFS file system,
such as the paging file, directories and the Master File Table, are
especially vulnerable to fragmentation, and allowing them to become
fragmented is a guarantee of a decrease in overall system performance.
Other NTFS features, such as file system compression, inherently create
fragmentation. The best way to avoid these worst-case fragmentation
problems, and to keep the system running at optimal performance, is to
run a defragmentation system on a regularly scheduled basis. Both
Windows NT workstations and NT servers are subject to these problems,
and both can improve system performance through regular defragmentation.

*********************

http://www.executive.com/whats-new/whitepaper.asp#_Toc463769971
NTFS Does Get Fragmented

The Windows NTFS File System Driver uses a special file called the
Master File Table (MFT) to track all files on that volume. The MFT
starts out with some free space to allow new files to be tracked, but on
a very busy system it too can run out of space. At this point NTFS
extends the MFT itself, creating new stretches of it for new
allocations. This situation is precipitated most often by fragmentation
in the file system itself, as file system fragments consume entries in
the MFT. If these new stretches are not contiguous, the MFT itself
becomes fragmented.

There are other files, such as the paging file used by Windows NT=3Fs
virtual memory subsystem, which can also become fragmented with
unpleasant implications for performance. The solution to these problems,
as we will see, it to prevent them from happening by keeping your system
defragmented.

Lastly, directories in NTFS are allocated similarly to files, but
defragmentation of them can be difficult.

Performance Degradations Can Impede Productivity

Windows NT does a good job of allowing the system to continue operation
even as programs wait for disk I/O, but some inefficiency cannot be
hidden forever. Especially on a mission-critical server, on which many
users rely, inefficiencies in the file system can lead to performance
degradation that impedes user productivity.

These problems are not always apparent, and are frequently cavalierly
blamed on other sources; perhaps the computer=3Fs just too slow, needs
more memory, or some program being run needs an upgrade. Overall system
performance is a complex phenomenon, and even experienced system
administrators may not recognize fragmentation in a file system. After
all, it can occur with large amounts of free space on the disk. But the
main reason users don=3Ft recognize fragmentation is because Windows NT
comes with no tools to identify it.

Heavily used systems, which are by definition mission-critical systems
for an organization, will become fragmented over time under normal usage
in Windows NT. As performance decreases in such systems and users are
forced to wait, productivity is thereby impeded.

*********************

http://www.microsoft.com/windows2000/techinfo/administration/fileandprin
t/defrag.asp

File Fragmentation

A file with all its parts stored in one location on a disk is described
as "contiguous." If a file is not contiguous, it=3Fs fragmented; broken
into pieces that are scattered throughout the disk. All Windows NT® and
Windows 2000 file types=3FFile Allocation Table (FAT) and NTFS file system
(NTFS)=3Fare susceptible to fragmentation.

File fragmentation has a negative effect on disk performance because the
disk head requires more time to move around to different points on the
disk to read scattered file parts. This is a primary reason for the
gradual degradation of system performance=3Fand the specific cause of
longer reads and extended reboots.

 

[Alex Nichol]

I have always described the concept of file fragmentation as thus...

Say you are reading a Newspaper. You find an article on page 2 and read a couple of
paragraphs and it then says "go to page 5". You go to page 5 and read another couple of
paragraphs and it now says "go to page 21". You go to page 21 and read a few more
paragraphs and it now says "go to page 12". You go to page 12 and read a few more
paragraphs and the article is completed.

Defragmentation is akin to taking all those paragraphs on other pages and assembling them
all in one contiguous article on page 2.

That is a very nice analogy, Dave. Thank you

 

[Raymond J. Johnson Jr.]

David H. Lipman wrote:




That is a very nice analogy, Dave. Thank you

Actually, in the context of the discussion, it's not. While it serves to
illustrate what happens in reading a fragmented file (and for that
purpose it is indeed a good analogy) it is a bit misleading in this
context, because paging around in a newspaper to read an article is a
PITA, while the analogous action of the hard drive will probably be
unnoticeable.

 

[Leythos]

Actually, in the context of the discussion, it's not. While it serves to
illustrate what happens in reading a fragmented file (and for that

It also impacts write performance.

purpose it is indeed a good analogy) it is a bit misleading in this
context, because paging around in a newspaper to read an article is a
PITA, while the analogous action of the hard drive will probably be
unnoticeable.

I don't really think you are ever going to understand performance and
drives.

 

[cquirke (MVP Win9x)]

On Thu, 23 Dec 2004 16:13:28 -0600, "Raymond J. Johnson Jr."

Actually, in the context of the discussion, it's not. While it serves to
illustrate what happens in reading a fragmented file (and for that
purpose it is indeed a good analogy) it is a bit misleading in this
context, because paging around in a newspaper to read an article is a
PITA, while the analogous action of the hard drive will probably be
unnoticeable.

Unless you are waiting for a modem, you are generally waiting for the
HD, which is in effect not reading one file in several bits, but lots
of files in several bits. So if fragmentation makes each one of those
file acesses take longer, then yes; you could feel it.

---------- ----- ---- --- -- - - - -

Proverbs Unscrolled #37
"Build it and they will come and break it"

 

[Raymond J. Johnson Jr.]

On Thu, 23 Dec 2004 16:13:28 -0600, "Raymond J. Johnson Jr."




Unless you are waiting for a modem, you are generally waiting for the
HD, which is in effect not reading one file in several bits, but lots
of files in several bits. So if fragmentation makes each one of those
file acesses take longer, then yes; you could feel it.

No one here can read, apparently. I NEVER SAID that fragmentation will
NEVER cause noticeable problems. The idiot Leythos keeps trying to read
that though, and now it looks like you are too. Here you go--let's see
who can understand plain English: For most users, and "most users" does
not include people who work with huge database or image files
constantly, regular and habitual defragging is a waste of time. For MOST
users, defragging should be done if the user is anal retentive and feels
an irresistible compulsion to do it, or if there is a performance issue
that might be ameliorated by defragging.

 

[Leythos]

No one here can read, apparently. I NEVER SAID that fragmentation will
NEVER cause noticeable problems. The idiot Leythos keeps trying to read

I didn't have to "read to that", you've misstated how fragmentation
impacts performance a number of times, and have said it's not noticeable
to most people a number of times, and you've clearly never managed a
server or workstation that has any real level of use.

that though, and now it looks like you are too. Here you go--let's see
who can understand plain English:

You've tried to tell us all you know about Fragmentation and
performance, and you've been wrong at almost every turn - this was the
only time you got it right:

For most users, and "most users" does
not include people who work with huge database or image files
constantly, regular and habitual defragging is a waste of time.

With one exception, those with smaller drives, or drives that are more
than 60% full, will benefit from a full defrag, but the frequency of the
defrag should be based on actual system use and is not the same for
every user.

A better way would have been to say:

For users that frequently add/delete files of any size from a system, or
that have large files that are frequently edited, or for users that work
with large files on a frequent basis, "you" would "most likely" benefit
from running a full defrag (one that also packs) on your drive every
month or so.

For those of you with LARGE drives that only do email and browsing, your
are very unlikely to benefit from a monthly defrag to the level that
"you" might "notice" a performance increase.

 

[cquirke (MVP Win9x)]

On Thu, 23 Dec 2004 21:36:05 -0600, "Raymond J. Johnson Jr."

For most users, and "most users" does not include people who
work with huge database or image files constantly, regular and
habitual defragging is a waste of time.

Most users wait for HD, and therefore will benefit from defragging. I
do agree with you that regular defragging, when nothing much has
changed on the system, is an ineffective way of managing this.

I would defrag *only* if the PC is known to be reliable and stable,
because to defrg on a flaky system is to risk installation or data
loss. Even then, I'd usually do it only at these times:

1) After deleting or uninstalling a mass of stuff

2) Before installing or adding a mass of stuff

3) Follow-up defrag a week after changing core code

Because Win98 and later track module usage during system use, to know
what often-used code to locate for best speed, you may benefit from
(3). For example, if you install a new version of IE, the new files
aren't listed in AppLog (Win98/SE/ME) or Prefetch (XP) and thus won't
be located for best speed. After a week's use, they will have been
identified by the system to be particularly optimised.

More effective strategies to reduce fragmentation impact include:

a) Kill off massive web browser caches

Web cache is supposed to speed up slow connections by holding
most-recently-used web content. A connection that can populate a 256M
cache within "recent" usage is fast enough not to need caching. A
connection slow enough to need caching will take so long to populate
more than 60M or so, that the oldest stuff won't be "recent" anymore.

Web cache contains lots of small files, which are deleted when "old".
If that takes months, you create fragmentation by opening up holes in
past areas of use. Also, whenever IE has to check all names in a
cache branch to ensure a new item doesn't overwrite and existing one,
this will be slow on FATxx as it has to traverse a large directory
that is itself fragmented. Ouch! So I usually limit cache to 20M.

b) Keep C: small and lean

By partitioning, and more to the point, being smart about what goes on
which partition, you can keep a system running as fast with 60G of
stuff on it as it was when there was only 10G of stuff there.

When 90% of disk accesses (temp, swap, OS code, core apps) are in a C:
that spans only 10% of the HD, your head travel will usually be no
more than 10% of the span - no matter how fragged C: gets.

Much smarter than having to step over your 40G MP3 collection, when
stepping from first-installed OS code at the 'front" of C: to the
fresh temp files created at the "end" of C:.

The other payoff is reduced risk of data corruption, and faster
maintenance. Files get corrupted during file system writes, and if
your data is somewhere that avoids 90% of write traffic, it is that
much safer. After a bad exit, you may typically find only C: has to
be checked, and waiting for 8G to be checked is better than 60G.

---------- ----- ---- --- -- - - - -

Proverbs Unscrolled #37
"Build it and they will come and break it"