OUCH! Disconnecting USB drive. How long before write cache is flushedautomatically??????

T

Thomas G. Marshall

Most write caches that I've had experience had a timeout associated
with them beyond which whatever was there pending output was flushed.

But I recently disconnected a USB drive after 45 minutes of inactivity
(after a big write) and it kicked out a "delayed write" error. Huh?

What is the time limit for this?
 
T

Twayne

Most write caches that I've had experience had a timeout associated
with them beyond which whatever was there pending output was flushed.

But I recently disconnected a USB drive after 45 minutes of inactivity
(after a big write) and it kicked out a "delayed write" error. Huh?

What is the time limit for this?

I don't believe there is one. The cache can sit there in memory
practically forever if the machine just sits idle. That's why the
Remove Hardware icon was created. Or, in the device's properties, you
can set it for "quick removal" which turns off the write-behind
operation and you won't need the Remove Hardware icon.
It sounds like you did not use Remove Hardware, right? You should
clean out the USB drive if the write left pieces on it, and do the write
over again. This time use the Remove Hardware icon to shut down the
drive before you disconnect it.

If that's not your issues, please explain further.

HTH

Twayne
 
T

Thomas G. Marshall

I don't believe there is one.  The cache can sit there in memory
practically forever if the machine just sits idle.  That's why the
Remove Hardware icon was created.  Or, in the device's properties, you
can set it for "quick removal" which turns off the write-behind
operation and you won't need the Remove Hardware icon.
   It sounds like you did not use Remove Hardware, right?  You should
clean out the USB drive if the write left pieces on it, and do the write
over again.  This time use the Remove Hardware icon to shut down the
drive before you disconnect it.

If that's not your issues, please explain further.

HTH

Twayne


Hmmmm..... Well I looked and there *was no* remove hardware icon in
the notification area. Which is what I expected because I had
disabled the caching my usb drive. But this was another person's
drive. Could it really be the case that different drives are using
different USB drivers?

Or perhaps was it my recent set of updates that somehow turned it back
on? And I've never had to do this for my drive anyway. This only
happened on another persons.

Further (BTW): IMO it totally violates the law of least surprise to
have a write cache never timeout. That just bothers me. For example,
if I set up a large write, go on vacation for 3 weeks, come back and
pull the plug I can lose data? Jeez....

How much data could I expect to be lost? 1/2, 1, 3 MB or what?

Thanks so much.
 
T

Thomas G. Marshall

Hmmmm.....  Well I looked and there *was no* remove hardware icon in
the notification area.  Which is what I expected because I had
disabled the caching my usb drive.  But this was another person's
drive.  Could it really be the case that different drives are using
different USB drivers?

Or perhaps was it my recent set of updates that somehow turned it back
on?  And I've never had to do this for my drive anyway.  This only
happened on another persons.

Further (BTW): IMO it totally violates the law of least surprise to
have a write cache never timeout.  That just bothers me.  For example,
if I set up a large write, go on vacation for 3 weeks, come back and
pull the plug I can lose data?  Jeez....

How much data could I expect to be lost?  1/2, 1, 3 MB or what?

Thanks so much.

Further, can this corrupt the filesystem????
 
B

Bob I

Thomas said:
Further, can this corrupt the filesystem????

If there is software on the system the is checking/scanning the drive it
never gets close. Sudden power offs ALWAYS provide a possibility of
corruption.
 
T

Thomas G. Marshall

If there is software on the system the is checking/scanning the drive it
never gets close. Sudden power offs ALWAYS provide a possibility of
corruption.

That's really unfortunate because this particular one was powered
soley by the USB port. Criminey.

What's the size of the write cache typically?????
 
B

Bob I

Thomas said:
That's really unfortunate because this particular one was powered
soley by the USB port. Criminey.

What's the size of the write cache typically?????

Don't know, I believe that's one of the areas where Windows will resize
to provide a balance of need vs. performance. I think you need to review
that installation. Also a chkdsk against the drive will tell you if you
have hosed it or not and provide some reassurance as you continued use.
 
T

Thomas G. Marshall

Don't know, I believe that's one of the areas where Windows will resize
to provide a balance of need vs. performance. I think you need to review
that installation. Also a chkdsk against the drive will tell you if you
have hosed it or not and provide some reassurance as you continued use.

I know, thanks. Related note: I've got a current post into the
UBCD4Win forum as to whether or not that "fake" XP boots up with write
caching enabled for USB disks. When I get a moment, I'll report back
to this thread what I find out.
 
T

Twayne

Hmmmm..... Well I looked and there *was no* remove hardware icon in
the notification area. Which is what I expected because I had
disabled the caching my usb drive. But this was another person's
drive. Could it really be the case that different drives are using
different USB drivers?

There are so many buffers/caches in windows it's hard to give that a
yes/no. IF you have turned off the write-behind feature in Properties,
then it won't cache data and will write until everything is on disk
before the write stops.
Note that this CAN still take some time if it's a large file, so if
you were in a hurry, it would still be possible to turn OFF the drive
before the write were completed, even if it's not cached in write-behind
but only in disk buffers.
When there is no remove hardware icon in the ssytem tray, that's a
decent indication that there is no write-behind enabled. If it isn't
needed, the icon isn't called into the tray. I just tried it; the icon
comes and goes with whether it's enabled or not. Of course, the icon
appearance depends on the related services having been started. If in
doubt, I'd make the changes to be sure it appears when it's needed and
disappears when not needed (write-behind turned OFF). I've no idea which
services it depends on.
Or perhaps was it my recent set of updates that somehow turned it back
on? And I've never had to do this for my drive anyway. This only
happened on another persons.

Possibly; stranger things have happened. Don't forget though, even
without it, the bottleneck of the drive comms still has to allow the
entire file to pass thru it and it can take time if it's a large file or
worse, several files.
Further (BTW): IMO it totally violates the law of least surprise to
have a write cache never timeout. That just bothers me. For example,
if I set up a large write, go on vacation for 3 weeks, come back and
pull the plug I can lose data? Jeez....

That "should" be the way it works, but then again you shouldn't be going
off on vacation and leaving that app wide open either, just sitting
there waiting for more input from the keyboard/mouse, whatever. Closing
the application should flush the buffer as long as there is enough idle
time for the cache to finish writing to the disk. While the file is
open though, it does seem like they sit there "forever" but I've never
had occasion to leave a program open like that, or even overnight. It's
just too prone to corruption of whatever kind may be able to get into
the machnine or that XP does on its own occasionally.
How much data could I expect to be lost? 1/2, 1, 3 MB or what?

That I've no idea. I'm sure it'll vary depending on what it was that
stopped the disk write in the first place. If nothing else asked for
the processor, then it'd empty the cache it seems. But somethign as
simple as indexing, if it occurred immediately, it seems could stop the
write and hold it in the write-behind cache. I've no idea how much you
could lose; I've never had the occasion and don't intend to<g>.
I really have the feeling, going thru this a piece at a time, that
you may have sent a huge, or several, file/s to the drive, and it was
simply not finished writing to the disk because of the comms data
bottlenect. I can't be sure, but ... that's my take on it at least.
Believe me, I'm no guru on these things<g>!

HTH

Twayne
 
T

Twayne

Further, can this corrupt the filesystem????


TGM,

From what little I know of the NTFS table, I don't think so. NTFS,
unlike FAT, doesn't depend on end of file markers and all that good
stuff. I mean, obviously it could trash whatever file/s end up
incomplete and that were still open, but I don't think it would corrupt
anything outside the file. I say that mostly because I haven't seen it
in my travels, reading and research: Not because I actually know it for
a fact. Sorry.

"Law of Surprise": I like that phrase, it's so few words to express
such an important idea. I'm sure MS would argue the case it's not a
surprise, but I don't; they surprise me every time I turn around, it
seems.

Twayne
 
T

Twayne

Well, simple reads wouldn't cause any problems, but sure, any write in
process that gets interrupted could be a problem, especially when you
don't know WHERE the write/s is/are taking place. But that's the case
with any operating system and any disk drive so it's not specific to
windows or ext drives, etc..
It is worse in XP though, because the registry is under constant
activity, even when there are no applications running on the computer.
A LOT is going on behind the scenes that the user is seldom aware of.
As long as a machine is powered down properly though, it's not a
problem. Useing Task Manager to End a Task causes no problems since,
although it forces and end to the service/program, it does so
gracefully. Loss of ac power though gives you something better than a
50-50 chance something will corrupt. That's why chkdsk often runs
automatically after such an event; to straighten out the tables and
areas it has dominion over in order to at least let the machine boot.
Or not<g>.


TGM said:
That's really unfortunate because this particular one was powered
soley by the USB port. Criminey.

Don't quote me here: Anyone who pipes in will probably be more accurate
than I:
Proper Shut Down of the computer though would have allowed a proper shut
down of the USB device in most cases. It will also flush buffers on the
way down, IF there aren't programs holding them open, as in a file/s
save. In that case things can get out of order, and by the time it tries
to save the pending write, it's already too late. Had the program
been closed and the file properly stated, then the cache would be
flushed onto the drive and all would be fine. Apparently it gets real
iffy in this area and you may or may not have corruption as a result of
the program holding a buffer open while the registry thinks it's been
written to, close it, and then the data never gets to the drive.
What's the size of the write cache typically?????

I think there is a max size, based on installed RAM and pagefile, but
IME the buffer isn't usually maxxed out. The buffer will only have in
it what was left to write before it lost the processor's attention. And
that activity flashes back and forth a bunch of times, write a little,
go do someting higher pri, write a little more, hi pri again, and so
forth. It's when it can't get the processor's attention any more it
sits there and waits. As in, the program is open, so ... no rush to
write this stuff, it lets other tasks run instead. Closing the file
SHOULD flush the buffer but I know lots of times I actually have to
close the application before it flushes. It that's by design, which I
doubt, it's a crummy process. IMO it's a "feature" they didn't fix.

Twayne
 
T

Thomas G. Marshall

There are so many buffers/caches in windows it's hard to give that a
yes/no.  IF you have turned off the write-behind feature in Properties,
then it won't cache data and will write until everything is on disk
before the write stops.
   Note that this CAN still take some time if it's a large file, so if
you were in a hurry, it would still be possible to turn OFF the drive
before the write were completed, even if it's not cached in write-behind
but only in disk buffers.
   When there is no remove hardware icon in the ssytem tray, that's a
decent indication that there is no write-behind enabled.  If it isn't
needed, the icon isn't called into the tray.  I just tried it; the icon
comes and goes with whether it's enabled or not.  Of course, the icon
appearance depends on the related services having been started.  If in
doubt, I'd make the changes to be sure it appears when it's needed and
disappears when not needed (write-behind turned OFF). I've no idea which
services it depends on.




Possibly; stranger things have happened.  Don't forget though, even
without it, the bottleneck of the drive comms still has to allow the
entire file to pass thru it and it can take time if it's a large file or
worse, several files.




That "should" be the way it works, but then again you shouldn't be going
off on vacation and leaving that app wide open either, just sitting
there waiting for more input from the keyboard/mouse, whatever.  Closing
the application should flush the buffer as long as there is enough idle
time for the cache to finish writing to the disk.  While the file is
open though, it does seem like they sit there "forever" but I've never
had occasion to leave a program open like that, or even overnight.  It's
just too prone to corruption of whatever kind may be able to get into
the machnine or that XP does on its own occasionally.




That I've no idea.  I'm sure it'll vary depending on what it was that
stopped the disk write in the first place.  If nothing else asked for
the processor, then it'd empty the cache it seems.  But somethign as
simple as indexing, if it occurred immediately, it seems could stop the
write and hold it in the write-behind cache.  I've no idea how much you
could lose; I've never had the occasion and don't intend to<g>.
   I really have the feeling, going thru this a piece at a time, that
you may have sent a huge, or several, file/s to the drive, and it was
simply not finished writing to the disk because of the comms data
bottlenect.  I can't be sure, but ... that's my take on it at least.
Believe me, I'm no guru on these things<g>!

HTH

Twayne

Well it was easily 45 minutes or more since the bulk copy of a 28 gig
folder claimed to be over. There was no icon in the tray. Zip nada.
Pulled the cable and >poof<, that foreboding message of delayed write
failure. :( Grrrrr.......... Still not quite grokking this.

I mean, 45 minutes? The little red LED on the HD turned green, which
indicates that from it's point of view it had finished receiving
data. {shrug}. So I'm unclear on something: If I r-click on a large
28 gig folder tree and ask for properties (and watch it whirr away
gathering information), does it *write* any of that information back
to the hard-drive so save on future properties requests?

It seems the more questions I have answered here the less sense it
makes.
 
T

Thomas G. Marshall

Well, simple reads wouldn't cause any problems, but sure, any write in
process that gets interrupted could be a problem, especially when you
don't know WHERE the write/s is/are taking place.   But that's the case
with any operating system and any disk drive so it's not specific to
windows or ext drives, etc..
   It is worse in XP though, because the registry is under constant
activity, even when there are no applications running on the computer.
A LOT is going on behind the scenes that the user is seldom aware of.
As long as a machine is powered down properly though, it's not a
problem.  Useing Task Manager to End a Task causes no problems since,
although it forces and end to the service/program, it does so
gracefully.  Loss of ac power though gives you something better than a
50-50 chance something will corrupt.  That's why chkdsk often runs
automatically after such an event; to straighten out the tables and
areas it has dominion over in order to at least let the machine boot.
Or not<g>.

TGM said:




Don't quote me here:  Anyone who pipes in will probably be more accurate
than I:
Proper Shut Down of the computer though would have allowed a proper shut
down of the USB device in most cases.  It will also flush buffers on the
way down, IF there aren't programs holding them open, as in a file/s
save. In that case things can get out of order, and by the time it tries
to save the pending write, it's already too late.    Had the program
been closed and the file properly stated, then the cache would be
flushed onto the drive and all would be fine.  Apparently it gets real
iffy in this area and you may or may not have corruption as a result of
the program holding a buffer open while the registry thinks it's been
written to, close it, and then the data never gets to the drive.




I think there is a max size, based on installed RAM and pagefile, but
IME the buffer isn't usually maxxed out.  The buffer will only have in
it what was left to write before it lost the processor's attention.  And
that activity flashes back and forth a bunch of times, write a little,
go do someting higher pri, write a little more, hi pri again, and so
forth.  It's when it can't get the processor's attention any more it
sits there and waits.  As in, the program is open, so ... no rush to
write this stuff, it lets other tasks run instead.  Closing the file
SHOULD flush the buffer but I know lots of times I actually have to
close the application before it flushes.  It that's by design, which I
doubt, it's a crummy process.  IMO it's a "feature" they didn't fix.

Twayne

All of this needs to be predicated by this simple notion: All write
caches really *should* be flushing themselves at least after several
seconds after the stop of user space writes. Unless of course you're
dealing with something *dog* slow like an old tape drive.

Unless I was writing to a USB papertape punch :) this shouldn't
happen.
LOL...
 
G

Guest

Thomas said:
Well it was easily 45 minutes or more since the bulk copy of a 28 gig
folder claimed to be over. There was no icon in the tray. Zip nada.
Pulled the cable and >poof<, that foreboding message of delayed write
failure. :( Grrrrr.......... Still not quite grokking this.

I mean, 45 minutes? The little red LED on the HD turned green, which
indicates that from it's point of view it had finished receiving
data. {shrug}. So I'm unclear on something: If I r-click on a large
28 gig folder tree and ask for properties (and watch it whirr away
gathering information), does it *write* any of that information back
to the hard-drive so save on future properties requests?

It seems the more questions I have answered here the less sense it
makes.

I have a similar problem on SOME usb drives, but not all. Seems to
be random. I do have a disconnect icon, but it says the drive is in use
and won't.

One thing that works is to run chkdsk /F on the drive
to force dismount. If "dimount" means flush caches and orderly
dismount, I'm in good shape. If "force dismount" means "pull the plug"
it's not so nice.
 
T

Thomas G. Marshall

Thomas G. Marshall wrote:

....[snipity doo dah]...
I have a similar problem on SOME usb drives, but not all.  Seems to
be random.  I do have a disconnect icon, but it says the drive is in use
and won't.

One thing that works is to run chkdsk /F on the drive
to force dismount.  If "dimount" means flush caches and orderly
dismount, I'm in good shape.  If "force dismount" means "pull the plug"
it's not so nice.

Hmmmm... Do you mean "chkdsk /X"? That only dismounts "if necessary"
for the chkdsk. Here's what chkdsk /? provides:

CHKDSK [volume[[path]filename]]] [/F] [/V] [/R] [/X] [/I] [/C] [/L
[:size]]

/F Fixes errors on the disk.
/X Forces the volume to dismount first if necessary.
All opened handles to the volume would then be
invalid
(implies /F).

Either way, there are two potentially huge oversights here:

1. This is possibly something that is not well defined anywhere, may
be random, and in fact may actually be an infinite flush timeout.

2. This seems certainly something that is difficult for nearly anyone
to find documentation for.

I'm not a classic MS basher, but I'm just a little frustrated at
something being so vaguely understood like this that can cause such
dire results.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Top