Help with excessive I/O

[Mr. Land]

(Please advise if this isn't the appropriate group for this post, and
I will repost it elsewhere, thank you.)

I'm trying to determine the cause of a sudden, drastic decrease in
performance on my Windows 2003 Server box.

Any action at all, opening an application or an Explorer window,
logging in, causes a huge amount of I/O to happen against the system
drive (drive 0, the C: drive). For instance, when logging in, there
is a greater than 3 minute delay between the time I OK the login
dialog until Windows displays the desktop icons. During this long
delay the drive activity LED is steadily on.

At first I thought Windows might be thrashing due to low memory, but
the physical memory usage seems to only be about 40%. I did not see
any appreciable paging activity when running perfmon, either. Just to
be sure, I increased the physical memory from 1 to 2GB in the machine,
but that had no effect whatsoever.

I checked the disk for errors via Windows Explorer - none were
reported. I tried running Process Monitor to see the I/O being
performed - there isn't one process that seems to be doing a majority
of the I/O (although a process named lsass.exe seems to be doing quite
a bit.) I did notice that all of the I/O's seem to have multiple
sequential entries in PM's log - not sure if this is normal or there
is heavy re-trying happening.

I used some low-level disk checking utilities on the drive (Maxtor and
SpinRite) - neither reported any errors on the disk.

I set up a perfmon session to watch the following counters on Drive 0:
Avg. Disk Queue Length, Avg Disk sec/Read, Avg Disk sec/Write.

These all exhibit spikey, very high values. For instance the avg time
per write on the disk repeatedly spikes up to 5-7 secs per write (yes,
I'm sure of this value, I've tripled-checked the scaling and my
interpretation.) The values for all three counters repeatedly
oscillate from very low (normal) values to extremely high values over
time, forming what almost looks like a sawtooth wave on perfmon's
graph display. The period between these spikes hovers around 3-4
secs.

So, it seems that my drive does not have any errors on it, yet I/O to
it has suddenly and mysteriously become abysmal.

I'm running out of ideas - can anyone help?

Thanks very much.

 

[Rod Speed]

(Please advise if this isn't the appropriate group for this post, and
I will repost it elsewhere, thank you.)

I'm trying to determine the cause of a sudden, drastic decrease in
performance on my Windows 2003 Server box.

Any action at all, opening an application or an Explorer window,
logging in, causes a huge amount of I/O to happen against the system
drive (drive 0, the C: drive). For instance, when logging in, there
is a greater than 3 minute delay between the time I OK the login
dialog until Windows displays the desktop icons. During this long
delay the drive activity LED is steadily on.

At first I thought Windows might be thrashing due to low memory, but
the physical memory usage seems to only be about 40%. I did not see
any appreciable paging activity when running perfmon, either. Just to
be sure, I increased the physical memory from 1 to 2GB in the machine,
but that had no effect whatsoever.

I checked the disk for errors via Windows Explorer - none were
reported. I tried running Process Monitor to see the I/O being
performed - there isn't one process that seems to be doing a majority
of the I/O (although a process named lsass.exe seems to be doing quite
a bit.) I did notice that all of the I/O's seem to have multiple
sequential entries in PM's log - not sure if this is normal or there
is heavy re-trying happening.

I used some low-level disk checking utilities on the drive (Maxtor and
SpinRite) - neither reported any errors on the disk.

I set up a perfmon session to watch the following counters on Drive 0:
Avg. Disk Queue Length, Avg Disk sec/Read, Avg Disk sec/Write.

These all exhibit spikey, very high values. For instance the avg time
per write on the disk repeatedly spikes up to 5-7 secs per write (yes,
I'm sure of this value, I've tripled-checked the scaling and my
interpretation.) The values for all three counters repeatedly
oscillate from very low (normal) values to extremely high values over
time, forming what almost looks like a sawtooth wave on perfmon's
graph display. The period between these spikes hovers around 3-4
secs.

So, it seems that my drive does not have any errors on it, yet I/O to
it has suddenly and mysteriously become abysmal.

I'm running out of ideas - can anyone help?

Post the Everest SMART report.
http://www.majorgeeks.com/download.php?det=4181

One possibility is that you have some marginal sectors and the drive is trying on those
and eventually getting the data successfully, but the retrys are slowing things down.

 

[Bob Willard]

(Please advise if this isn't the appropriate group for this post, and
I will repost it elsewhere, thank you.)

I'm trying to determine the cause of a sudden, drastic decrease in
performance on my Windows 2003 Server box.

Any action at all, opening an application or an Explorer window,
logging in, causes a huge amount of I/O to happen against the system
drive (drive 0, the C: drive). For instance, when logging in, there
is a greater than 3 minute delay between the time I OK the login
dialog until Windows displays the desktop icons. During this long
delay the drive activity LED is steadily on.

At first I thought Windows might be thrashing due to low memory, but
the physical memory usage seems to only be about 40%. I did not see
any appreciable paging activity when running perfmon, either. Just to
be sure, I increased the physical memory from 1 to 2GB in the machine,
but that had no effect whatsoever.

I checked the disk for errors via Windows Explorer - none were
reported. I tried running Process Monitor to see the I/O being
performed - there isn't one process that seems to be doing a majority
of the I/O (although a process named lsass.exe seems to be doing quite
a bit.) I did notice that all of the I/O's seem to have multiple
sequential entries in PM's log - not sure if this is normal or there
is heavy re-trying happening.

I used some low-level disk checking utilities on the drive (Maxtor and
SpinRite) - neither reported any errors on the disk.

I set up a perfmon session to watch the following counters on Drive 0:
Avg. Disk Queue Length, Avg Disk sec/Read, Avg Disk sec/Write.

These all exhibit spikey, very high values. For instance the avg time
per write on the disk repeatedly spikes up to 5-7 secs per write (yes,
I'm sure of this value, I've tripled-checked the scaling and my
interpretation.) The values for all three counters repeatedly
oscillate from very low (normal) values to extremely high values over
time, forming what almost looks like a sawtooth wave on perfmon's
graph display. The period between these spikes hovers around 3-4
secs.

So, it seems that my drive does not have any errors on it, yet I/O to
it has suddenly and mysteriously become abysmal.

I'm running out of ideas - can anyone help?

Thanks very much.

1. Because it is free, not because it is terribly likely, reseat all of
the cables (not to be insulting, but with the power off). While you're
at it, examine the cables and connectors to see if one has been nicked.

2. If you've added any hardware recently, remove it. It is possible that
adding something has overloaded the power supply, causing sags during
heavy I/O and retrys. And, if feasible, you might swap the PS with one
that is known good and that delivers more juice, for the same reason.

3. Run HDtach on the whole HD(s), to see if there are repeatable read spikes,
indicating bad sectors. Or run HDtune's HealthCheck. Or try both.