Defragging a very large, heavily fragmented drive

  • Thread starter Thread starter Brad
  • Start date Start date
B

Brad

I have a 15TB disk array (3ware 24-port array contoller with 750GB disks in
RAID 5 with a hot spare) that I think is very fragmented. Recently I have
started getting the error "Not enough server storage is available to process
this command" when I try to access certain files or directories on the disk
over the network or locally. From what I have read it seems that heavy
fragmentation can be a problem that causes the disk to act like it is full
when it is not. This server is over 2 years old and has never been
defragged. The disk is 8% empty right now, but was as little as 7% recently
when the problem started. I'm wondering what you thinks is the best course
of action. There is no backup of a lot of the data so that is not an option.
I am currently moving as many files as I can, but that is going very slowly.
The array is not the boot drive on the server. The server is Windows Server
2003 Standard (32-bit). There are two main directory trees on the disk. One
has about 16 million JPGs in it (in a tree, not all in one directory) that
average 670K in size and the other has about 60 million JPGs that average
around 30K. The directory with the smaller images is not critical because
they can be regenerated.
 
Brad said:
I have a 15TB disk array (3ware 24-port array contoller with 750GB disks in
RAID 5 with a hot spare) that I think is very fragmented. Recently I have
started getting the error "Not enough server storage is available to
process
this command" when I try to access certain files or directories on the
disk
over the network or locally. From what I have read it seems that heavy
fragmentation can be a problem that causes the disk to act like it is full
when it is not. This server is over 2 years old and has never been
defragged. The disk is 8% empty right now, but was as little as 7%
recently
when the problem started. I'm wondering what you thinks is the best
course
of action. There is no backup of a lot of the data so that is not an
option.
I am currently moving as many files as I can, but that is going very
slowly.
The array is not the boot drive on the server. The server is Windows
Server
2003 Standard (32-bit). There are two main directory trees on the disk.
One
has about 16 million JPGs in it (in a tree, not all in one directory) that
average 670K in size and the other has about 60 million JPGs that average
around 30K. The directory with the smaller images is not critical because
they can be regenerated.
Click to expand...

When you say "The disk is 8% empty right now", do you mean to say that the
disk is 92% full? If so then I would describe it as an overly full disk.
IMHO disk should never be allowed to go beyond the 80% limit. When you're
above this limit then defragging will be extremely slow and quite
ineffective.

Note also that defragging will NOT increase the amount of free disk space.
It merely re-arranges the various file fragments.

From other posts I have collected the notes below in relation to the error
message "Not enough storage . . .":
- Usually resolved by applying the latest service pack
- Sometimes caused by NAV or IBM AV:
http://support.microsoft.com/?scid=kb;en-us;177078
- Set
HKLM\SYSTEM\CurrentControlSet\Services\lanmanserver\parameter\IRPStackSize
to 15..21 decimal
- Sometimes requires server service configuration and tuning
http://support.microsoft.com/default.aspx?scid=kb;[LN];128167
- PoolUsageMaximum set too low:
http://support.microsoft.com/default.aspx?scid=kb;EN-US;304101
 
I have a 15TB disk array (3ware 24-port array contoller with 750GB disks in
RAID 5 with a hot spare) that I think is very fragmented. Recently I have
started getting the error "Not enough server storage is available to process
this command"
Click to expand...


Read the message carefully, look at the event code if there is one and
google it.

It's been a long time but there used to be a message that was related
to OS process data structures (not disks and disk space) and the fix,
in my case, was to re-run the latest service pack to fix skewed
versions of DLLs on the system.
 
I realize that the disk is way too full. We are not writing any data to it
at this point, but I am moving files off of it and some files are being read
at the same time. I guess I'm just looking for a best recommendation at this
point.

One of the weird issues is that if I issue a Dir D:\01\23\45\ I get the "Not
enough server storage" error, but if I call Dir D:\01\23\45 I get
Directory of \\server\D$\01\23\45

09/26/2008 08:40 AM <DIR> 00
0 File(s) 0 bytes
1 Dir(s) 1,324,973,830,144 bytes free

instead of getting what is inside the directory. The OS doesn't seem to be
able to look inside because of some kind of corruption.

Pegasus said:
Brad said:
I have a 15TB disk array (3ware 24-port array contoller with 750GB disks in
RAID 5 with a hot spare) that I think is very fragmented. Recently I have
started getting the error "Not enough server storage is available to
process
this command" when I try to access certain files or directories on the
disk
over the network or locally. From what I have read it seems that heavy
fragmentation can be a problem that causes the disk to act like it is full
when it is not. This server is over 2 years old and has never been
defragged. The disk is 8% empty right now, but was as little as 7%
recently
when the problem started. I'm wondering what you thinks is the best
course
of action. There is no backup of a lot of the data so that is not an
option.
I am currently moving as many files as I can, but that is going very
slowly.
The array is not the boot drive on the server. The server is Windows
Server
2003 Standard (32-bit). There are two main directory trees on the disk.
One
has about 16 million JPGs in it (in a tree, not all in one directory) that
average 670K in size and the other has about 60 million JPGs that average
around 30K. The directory with the smaller images is not critical because
they can be regenerated.
Click to expand...

When you say "The disk is 8% empty right now", do you mean to say that the
disk is 92% full? If so then I would describe it as an overly full disk.
IMHO disk should never be allowed to go beyond the 80% limit. When you're
above this limit then defragging will be extremely slow and quite
ineffective.

Note also that defragging will NOT increase the amount of free disk space.
It merely re-arranges the various file fragments.

From other posts I have collected the notes below in relation to the error
message "Not enough storage . . .":
- Usually resolved by applying the latest service pack
- Sometimes caused by NAV or IBM AV:
http://support.microsoft.com/?scid=kb;en-us;177078
- Set
HKLM\SYSTEM\CurrentControlSet\Services\lanmanserver\parameter\IRPStackSize
to 15..21 decimal
- Sometimes requires server service configuration and tuning
http://support.microsoft.com/default.aspx?scid=kb;[LN];128167
- PoolUsageMaximum set too low:
http://support.microsoft.com/default.aspx?scid=kb;EN-US;304101
Click to expand...
 
Brad said:
I realize that the disk is way too full. We are not writing any data to it
at this point, but I am moving files off of it and some files are being
read
at the same time. I guess I'm just looking for a best recommendation at
this
point.

One of the weird issues is that if I issue a Dir D:\01\23\45\ I get the
"Not
enough server storage" error, but if I call Dir D:\01\23\45 I get
Directory of \\server\D$\01\23\45

09/26/2008 08:40 AM <DIR> 00
0 File(s) 0 bytes
1 Dir(s) 1,324,973,830,144 bytes free

instead of getting what is inside the directory. The OS doesn't seem to
be
able to look inside because of some kind of corruption.
Click to expand...

You can, of course, run chkdsk.exe but I'm always reluctant to do this
unless I have a confirmed saved copy of all files. Chkdsk might thrash a few
files or folder that are perfectly readable right now.

About the disk getting way too full: With servers it is a good idea to
generate weekly management reports about a number of key parameters - and to
read them!
 
I'm not an advocate for defragging. Quite contrary.
IMO,
a) If you have enough free storage space, fragmentation is not the issue.
b) If you don't have enough free storage space, fragmentation is the issue,
but defragging will not help. The only solution is to increase free storage
space.
c) If you start defragging the full disk, you risk loss of data (Exchange
database was known to be sensitive to defragging).
d) Defragging the full disk increases the disk I/O so that other services
suffer significantly.

In your particular case I'd use "divide and conquer" strategy.
I would split this large space to something more manageable.For example you
can use mounting points where you can mount smaller storage units into your
directory tree structure.
 
I have a 15TB disk array (3ware 24-port array contoller with 750GB disks in
RAID 5 with a hot spare) that I think is very fragmented. Recently I have
started getting the error "Not enough server storage is available to process
this command" when I try to access certain files or directories on the disk
over the network or locally. From what I have read it seems that heavy
fragmentation can be a problem that causes the disk to act like it is full
when it is not. This server is over 2 years old and has never been
defragged. The disk is 8% empty right now, but was as little as 7% recently
when the problem started. I'm wondering what you thinks is the best course
of action. There is no backup of a lot of the data so that is not an option.
I am currently moving as many files as I can, but that is going very slowly.
The array is not the boot drive on the server. The server is Windows Server
2003 Standard (32-bit). There are two main directory trees on the disk. One
has about 16 million JPGs in it (in a tree, not all in one directory) that
average 670K in size and the other has about 60 million JPGs that average
around 30K. The directory with the smaller images is not critical because
they can be regenerated.
Click to expand...

Download JKDefrag and let it run overnight, it's free and does a good
job on overly filled drives.
 
Download JKDefrag and let it run overnight, it's free and does a good
job on overly filled drives.
Click to expand...

I'll throw in a 3rd vote for JkDefrag
(http://www.kessels.com/JkDefrag/). After a colleague turned me on to
it about 9 months ago, I've switched to it and not looked back.

As far as the drive not having much free space, that's not as much of a
problem for JkDefrag as you might think. Run JkDefrag with the option
-a 6 to move all the files to the end of the drive, including all
fragments. This will open up as much free space as possible at the
start of the drive, thus facilitating defragmenting the rest of the
drive. I typically use the following modes 6, 8, and 3 to clean systems up.

Though with 13+ TB of data on a 15 TB file system it's going to take a
VERY long time to do. I would not be surprised if it would take a
weekend or more to run. Though, JkDefrag has an option to help with
that too. You can tell JkDefrag to slow down to a percentage of it's
normal operating speed, enough so that you can probably start it and let
it run while you are reading from the drive.

You mentioned that the directory with the smaller JPGs is not important
because they can be regenerated. If this is indeed the case, I'd
strongly urge you to consider moving them of the drive or out and out
deleting and subsequently regenerating them. Removing the smaller
regenerate able files will free up space and reduce the amount of data
that has to be handled, both of which will speed things up / reduce the
length of time required to complete the operation.

Before I tackle a file system problem like you are describing (though
usually on a much smaller scale) I typically like to have a robocopy (or
the likes) of the files on the file system elsewhere, perform a full 5
stage chkdsk (/f /r /x) to make sure that the file system is not
corrupt. If chkdsk found any thing I'd re-do the backup (after making
sure things are happy) and then and only then turn JkDefrag loose.

Also, seeing that you have so much data to work with and how long it
will likely take to do so, I'd *STRONGLY* suggest that you invest in a
UPS that is capable of shutting the server down gracefully at the lose
of power. Doing so will greatly help avoid a potential file system
corruption (caused by a power failure during a disk intensive operation)
worse than what you are trying to solve.



Grant. . . .
 
In your particular case I'd use "divide and conquer" strategy.
I would split this large space to something more manageable.For example
you can use mounting points where you can mount smaller storage units
into your directory tree structure.
Click to expand...

Be careful doing this.

Despite all the wonderful things that you can do with DDO, you may be
shooting your self in the foot. Some applications that don't understand
DDO will mis-read the amount of free space on the root of the drive that
your directory structure(s) is (are) on not realizing that the directory
structure(s) may actually be on a different file system.



Grant. . . .
 
Back
Top