Reformatting to non-standard NTFS cluster sizes?

  • Thread starter Thread starter Yousuf Khan
  • Start date Start date
Y

Yousuf Khan

Has anyone experimented with non-standard cluster sizes on NTFS
partitions? Typically, the default behaviour is that when creating the
file system, the format utility chooses the ideal cluster size based on
the size of the partition. But typically the highest it ever chooses is
4KB.

http://technet.microsoft.com/en-us/library/cc767961.aspx

But there are options for choosing upto 64KB, which never get chosen by
default. I have a file system that I calculated has a simple average
file size of 115MB! I was thinking for files that large, I should choose
the highest possible cluster size available.

Yousuf Khan
 
On Sun, 15 Apr 2012 10:47:48 -0400 "Yousuf Khan" <bbbl67
@spammenot.yahoo.com> wrote in article said:
Has anyone experimented with non-standard cluster sizes on NTFS
partitions? Typically, the default behaviour is that when creating the
file system, the format utility chooses the ideal cluster size based on
the size of the partition. But typically the highest it ever chooses is
4KB.

http://technet.microsoft.com/en-us/library/cc767961.aspx

But there are options for choosing upto 64KB, which never get chosen by
default. I have a file system that I calculated has a simple average
file size of 115MB! I was thinking for files that large, I should choose
the highest possible cluster size available.

Yousuf Khan
Click to expand...

I have partitions with large numbers of enormous .wav files I've
recorded. I have experimented with cluster sizes all the way to 64k but
see very little, if any, change in performance. That surprises me. I also
have several external HDD's (USB- and eSATA-attached) for backup and have
done the same tests with the same result.
 
NTFS was originally called HPFS, for High Performance File System. It
was designed to be fast and reliable with low cluster sizes, so as to
reduce blank space. NTFS inherited that ability, so you will see small
differences in performance when you sue larger clusters. Whether those
small diffs are worth it is your call. IMO, the advantage of larger
cluster sizes has been overtaken by improved hardware performance.

HPFS was devised was back the days when MS was creating OS/2 for IBM
(and itself). After the split, MS renamed its version NT. Up to NT 3.5,
both OS/2 and NT could read and write each other's file systems. Up to
Win 2000 there was still a directory labelled OS/2 buried in the
Windows/System tree.
Click to expand...

My question would be, does it affect the amount of data that can be stored?

Logically, the smaller the cluster size, it would require more entries
on the disk to keep track of which cluster belonged to which file, and
the order of the clusters. Larger sized clusters would require fewer
entries.

If your files are relatively small, I wouldn't think there'd be much
difference. But, what if you have huge amounts of large and larger
video, graphic, and audio files?


--
Ken

Mac OS X 10.6.8
Firefox 11.0
Thunderbird 11.0.1
LibreOffice 3.5.1.2
 
On Sun, 15 Apr 2012 10:47:48 -0400 "Yousuf Khan" <bbbl67


I have partitions with large numbers of enormous .wav files I've
recorded. I have experimented with cluster sizes all the way to 64k but
see very little, if any, change in performance. That surprises me. I also
have several external HDD's (USB- and eSATA-attached) for backup and have
done the same tests with the same result.
Click to expand...
It really has nothing to do with differences in performance, and more to do
with if you choose the wrong cluster size for the files you intend to store,
but the more wasted space per file.
Youd want a smaller clsuetr size if you were storing thousands of small MIDI
files, and larger sized clusters for large backup files or DV video captures.
 
Ken Springer said:
My question would be, does it affect the amount of data that can be
stored?

Logically, the smaller the cluster size, it would require more entries on
the disk to keep track of which cluster belonged to which file, and the
order of the clusters. Larger sized clusters would require fewer entries.

If your files are relatively small, I wouldn't think there'd be much
difference. But, what if you have huge amounts of large and larger video,
graphic, and audio files?
Click to expand...

But those drives by definition arent likely to be where you are obsessed
about space use.
 
I have partitions with large numbers of enormous .wav files I've
recorded. I have experimented with cluster sizes all the way to 64k but
see very little, if any, change in performance. That surprises me. I also
have several external HDD's (USB- and eSATA-attached) for backup and have
done the same tests with the same result.
Click to expand...

Well, I have gone ahead done the reformat to one my partitions, I'm now
using 64K clusters on that. I'm in the middle of some rearrangement of
data on my hard drives. So what I've been going around doing is copying
and moving entire file systems worth of data from one drive to another.
When I did the transfer between a default 4K cluster drive and a 64K
cluster drive, the speed of transfer is not hugely different. However,
there was a huge difference in responsiveness of the drives. I was
watching the transfers (which lasted a couple of hours each) on the
Windows 7 Resource Monitor, and you could see that the 4K drives were
absolutely pegged at 100% disk activity, with disk queues that were
hovering >=1.0. But the drive that was using 64K clusters had disk
activity of 40-50%, and disk queues well <1.0. This worked in both
directions, whether I was copying off of the 64K drive, or copying to
the 64K drive, it was the same way.

Yousuf Khan
 
GMAN said:
It really has nothing to do with differences in performance, and more to
do
with if you choose the wrong cluster size for the files you intend to
store,
but the more wasted space per file.
Youd want a smaller clsuetr size if you were storing thousands of small
MIDI
files, and larger sized clusters for large backup files or DV video
captures.
Click to expand...

But again, with that last situation, you are likely to be choosing between
2TB and
3TB drives base on $/GB and the small saving in free space isnt likely to
matter.
All that really does is see you be able to squeeze one more file on the
drive at most.
 
My question would be, does it affect the amount of data that can be stored?

Logically, the smaller the cluster size, it would require more entries
on the disk to keep track of which cluster belonged to which file, and
the order of the clusters. Larger sized clusters would require fewer
entries.
Click to expand...

You're thinking of the FAT system which has a global bitmap of clusters
assigned to files and folders. The bigger the drive and the smaller the
cluster size, the bigger the bitmap gets. In NTFS, it's more of an
extent-based system, which basically contains a starting and ending
cluster address of all extents belonging to a file. The metadata on the
disk only grows with the number of files, but not necessarily with the
size of the file system. So the size of the clusters won't make a
difference to the size of the metadata, just the number of files you put
into it.
If your files are relatively small, I wouldn't think there'd be much
difference. But, what if you have huge amounts of large and larger
video, graphic, and audio files?
Click to expand...

However, you will get more slack with small files in a large-cluster
filesystem. Even the smallest files will take up a minimum of 64K with a
filesystem with 64K clusters.

However, I've found that the responsiveness of a large-cluster-size
filesystem, is much greater when you also have large files.

Yousuf Khan

Yousuf Khan
 
lately I've always formatted my large external USB2 disks as NTFS with 32K
clusters.

It works fine, but I don't know if it really provides any benefit.
Click to expand...

But there must've been a reason you chose that size? Do you remember why?

Yousuf Khan
 
In comp.sys.ibm.pc.hardware.storage Yousuf Khan said:
Has anyone experimented with non-standard cluster sizes on NTFS
partitions? Typically, the default behaviour is that when creating the
file system, the format utility chooses the ideal cluster size based on
the size of the partition. But typically the highest it ever chooses is
4KB.

But there are options for choosing upto 64KB, which never get chosen by
default. I have a file system that I calculated has a simple average
file size of 115MB! I was thinking for files that large, I should choose
the highest possible cluster size available.
Click to expand...
Yousuf Khan
Click to expand...

First find out whether it makes a difference. I expect it does not,
as most of these clusters will be placed on disk consecutively
anyways. Usually, larger clusters are just to reduce administration
information (space gain) or to make large filesystems possible (such
as for FAT). My guess would be it causes more trouble than it is
worth.

Arno
 
First find out whether it makes a difference. I expect it does not,
as most of these clusters will be placed on disk consecutively
anyways. Usually, larger clusters are just to reduce administration
information (space gain) or to make large filesystems possible (such
as for FAT). My guess would be it causes more trouble than it is
worth.
Click to expand...

So far, it's been working pretty well, it seems to reduce disk activity
immensely. I've got several disks that have nothing but huge files on
them, I'm thinking of converting them all over now, but that would
involve having other places to store the files that they got on them now
until the reformat can be done. I've converted about 3 file systems to
64K clusters already, but those were the smaller disks, which could be
temporarily accommodated on larger disks, but now the larger disks are
the ones that require it.

Yousuf Khan
 
So far, it's been working pretty well, it seems to reduce disk activity
immensely. I've got several disks that have nothing but huge files on
them, I'm thinking of converting them all over now, but that would
involve having other places to store the files that they got on them now
until the reformat can be done. I've converted about 3 file systems to
64K clusters already, but those were the smaller disks, which could be
temporarily accommodated on larger disks, but now the larger disks are
the ones that require it.
Click to expand...

Acronis Disk Director, and I presume other disk partitioning tools,
will allow you to change the cluster size nondestructively, so there
would be no need to move the data off of the drive. I'll leave the
risk/reward analysis up to the reader.
 
Acronis Disk Director, and I presume other disk partitioning tools,
will allow you to change the cluster size nondestructively, so there
would be no need to move the data off of the drive. I'll leave the
risk/reward analysis up to the reader.
Click to expand...

Is it free?

Yousuf Khan
 
I think I just chose the same size as the allocation blocks were on some
FAT32 drives I had.
Click to expand...

Oh-ok, it was just chosen based on precedent and tradition: "In my day,
we _always_ chose 32K clusters". :)

Yousuf Khan
 
Back
Top