Vista SP1 - Error copying very large files

[Jorge Coelho]

I have this very large Outlook 2003 PST file (2.97 GB) which I must
regularly transfer from my desktop into my laptop when I go on the road and
back to my desktop when I return.

The desktop is a Core2 Duo at 3.0 Ghz with 4 GB of RAM and a very fast WD
Raptor 150 GB running Vista Home Premium English with SP1, the laptop is a
Toshiba Core2 Duo at 2.40 Ghz with 4 GB RAM running Vista Home Premium
Portuguese *without* SP1 (since it hasn't been released for this language
yet). I'm using a *wired* LAN at home with a LinkSys Router. Both the laptop
and the desktop are protected by ESET's Smart Security suite (NOD32 AV +
Firewall).

Prior to installing SP1 on the desktop I was already getting sporadic
network errors when transfering the file either to the laptop or from it.
The error is always the same and would occur midway through the transfer:
'network location no longer available'. However, clicking the 'Try Again'
button once usually seemed to temporarily 'fix' the issue.

This time, however, clicking 'Try Again' when transfering the PST file from
the laptop would NOT fix the problem, no matter how many times I clicked it.
The transfer would resume FROM THE BEGINING (doh?!) and would eventualy
result in another 'network location no longer available' error. Rebooting
the desktop did NOT fix the problem (I didn't try rebooting the laptop
because I had just turned it on).

Using the network bandwith monitor module of Winstep Xtreme, I could see the
transfer going at full speed (11+ MB per second on the 100 Mb/s LAN) when it
would suddenly drop to 8 or 12 KB/s for a few seconds, then go back to full
speed, etc, until it would drop to 0 KB/s and stay there, with the error
message popping up a few seconds later.

The only way I could get the transfer to finally succeed was by disabling
the Firewall on BOTH computers - but do not be to quick to blame the
firewall!!! With the firewall disabled, I was still seeing the transfer
speed 'hiccup' as regularly as before.

One thing I found really odd was that while the hard disk led on the laptop
only lit up a couple of times a second, the led on the desktop (the one with
the very fast 10,000 RPM WD Raptor) was flashing constantly! Didn't add up,
so I fired up the Resource Monitor applet you can find on the Windows Vista
Task Manager dialog. What I found out was that not only was the Outlook.pst
file being constantly written to, but so was some file with a LOT of
alphanumeric digits under C:\System Volume Information\. In a way, it was
like the information was being 'mirrored' into another file.

I suspect the Volume Shadow Copy feature might have something to do with
this, even though this is a Vista Home Premium installation, but, as I said,
it's only a suspicion. I also suspect that the real problem has something to
do with the 'mirroring' operation maxing out the hard disk tranfer rate,
which makes the data comming from the network fill up all the available RAM.
Once this happens, the network transfer basically stops until the hard disk
'catches up' (the reason why the network transfer speed suddenly drops down
to 8 Kb/s or less for a while and then goes back to full speed again). I
also think the only reason disabling the firewall on both systems changed
the outcome was because the smaller overhead 'bought' enough time for the
transfer *not* to time out.

Anybody here has seen this problem or care to discuss the issue?

 

[Charles W Davis]

I have this very large Outlook 2003 PST file (2.97 GB) which I must
regularly transfer from my desktop into my laptop when I go on the road and
back to my desktop when I return.

The desktop is a Core2 Duo at 3.0 Ghz with 4 GB of RAM and a very fast WD
Raptor 150 GB running Vista Home Premium English with SP1, the laptop is a
Toshiba Core2 Duo at 2.40 Ghz with 4 GB RAM running Vista Home Premium
Portuguese *without* SP1 (since it hasn't been released for this language
yet). I'm using a *wired* LAN at home with a LinkSys Router. Both the
laptop and the desktop are protected by ESET's Smart Security suite (NOD32
AV + Firewall).

Prior to installing SP1 on the desktop I was already getting sporadic
network errors when transfering the file either to the laptop or from it.
The error is always the same and would occur midway through the transfer:
'network location no longer available'. However, clicking the 'Try Again'
button once usually seemed to temporarily 'fix' the issue.

This time, however, clicking 'Try Again' when transfering the PST file
from the laptop would NOT fix the problem, no matter how many times I
clicked it. The transfer would resume FROM THE BEGINING (doh?!) and would
eventualy result in another 'network location no longer available' error.
Rebooting the desktop did NOT fix the problem (I didn't try rebooting the
laptop because I had just turned it on).

Using the network bandwith monitor module of Winstep Xtreme, I could see
the transfer going at full speed (11+ MB per second on the 100 Mb/s LAN)
when it would suddenly drop to 8 or 12 KB/s for a few seconds, then go
back to full speed, etc, until it would drop to 0 KB/s and stay there,
with the error message popping up a few seconds later.

The only way I could get the transfer to finally succeed was by disabling
the Firewall on BOTH computers - but do not be to quick to blame the
firewall!!! With the firewall disabled, I was still seeing the transfer
speed 'hiccup' as regularly as before.

One thing I found really odd was that while the hard disk led on the
laptop only lit up a couple of times a second, the led on the desktop (the
one with the very fast 10,000 RPM WD Raptor) was flashing constantly!
Didn't add up, so I fired up the Resource Monitor applet you can find on
the Windows Vista Task Manager dialog. What I found out was that not only
was the Outlook.pst file being constantly written to, but so was some file
with a LOT of alphanumeric digits under C:\System Volume Information\. In
a way, it was like the information was being 'mirrored' into another file.

I suspect the Volume Shadow Copy feature might have something to do with
this, even though this is a Vista Home Premium installation, but, as I
said, it's only a suspicion. I also suspect that the real problem has
something to do with the 'mirroring' operation maxing out the hard disk
tranfer rate, which makes the data comming from the network fill up all
the available RAM. Once this happens, the network transfer basically stops
until the hard disk 'catches up' (the reason why the network transfer
speed suddenly drops down to 8 Kb/s or less for a while and then goes back
to full speed again). I also think the only reason disabling the firewall
on both systems changed the outcome was because the smaller overhead
'bought' enough time for the transfer *not* to time out.

Anybody here has seen this problem or care to discuss the issue?

You have probably never converted your PST file format to the Personal
Folders File format and have reached the maximum size in the older format.

Outlook Help returned this:

On the File menu, click Data File Management.
On the Data Files tab, click the Personal Folders file (.pst) data file that
you want to check, and then click Settings.
View the Format field to see whether it contains the text Personal Folders
File or Personal Folders File (97-2002).

Personal Folders File indicates the new format in Microsoft Office Outlook
2003 and Microsoft Office Outlook 2007 that supports multilingual Unicode
data and offers greater storage capacity for items and folders. A file in
the Personal Folders file format is not compatible with Microsoft Outlook
97-2002.

Personal Folders File (97-2002) indicates the .pst file format that is
compatible with the versions of Outlook prior to Outlook 2003. Personal
Folders file (97-2002) does not support Unicode and offers the same storage
capacity for items and folders that was available in the previous versions
of Outlook.

 

[Jorge Coelho]

You have probably never converted your PST file format to the Personal
Folders File format and have reached the maximum size in the older format.

Hmm, thanks for the reply, but I guess you didn't actually read the whole
post. ;-) I have no problem with Outlook itself, it's *manually* copying the
Outlook PST file from one system to another over the LAN that causes the
problem. The file copy operation fails with the described error. Current
Outlook data file format is correct and goes up to 3 Terabytes maximum,
IIRC.

What I do when I go 'on the road' is: I close Outlook on the desktop machine
to prevent it from receiving more messages and locking the Outlook.pst data
file, copy the PST file over the LAN into the laptop system (overwritting
any older version there), and from then on - and until I come back - I use
the Outlook 2003 version installed on the laptop to receive mail. When I
come back home, I do the reverse.

Unfortunately I have to do it this way because Outlook has no mechanism to
synchronize email and, although I use an IMAP account for my company emails,
my personal email accounts MUST use POP3 (i.e. the email is retrieved from
the server, stored locally and deleted from the server) and I need to keep a
record of all the personal messages I ever sent and received.

 

[Julian]

One thing I found really odd was that while the hard disk led on the
laptop only lit up a couple of times a second, the led on the desktop (the
one with the very fast 10,000 RPM WD Raptor) was flashing constantly!
Didn't add up, so I fired up the Resource Monitor applet you can find on
the Windows Vista Task Manager dialog. What I found out was that not only
was the Outlook.pst file being constantly written to, but so was some file
with a LOT of alphanumeric digits under C:\System Volume Information\. In
a way, it was like the information was being 'mirrored' into another file.

I suspect the Volume Shadow Copy feature might have something to do with
this, even though this is a Vista Home Premium installation, but, as I
said, it's only a suspicion.

If the file in SVI ends with "{3808876b-c176-4e48-b7ae-04046e6cc752}" then,
yes, this is a ShadowCopy storage file (the long string is the CLSID (?) of
the default VSS writer... you can find the value in the registry).
ShadowStorage is availablein all Vista versions and provides System Restore
for Basic/Home Premium systems and SR and "Previous Versions" (file and
folder backtracking) in Business/Enterprise/Ultimate.

It could be worthwhile moving the PST to another partition without System
Restore/System Protection enabled; it would be a long explanation but this
is the only way to completely exclude a file from getting (at least
partially) copied to ShadowStorage. However, ShadowCopies can be accessed
even on Vista HP (e.g. with the aid of ShadowExplorer), so if you feel the
recovery feature might be useful you may want to stick with the current
setup.

I have no idea why the PST should be being written to, but, is Outlook still
running? If so, copying might be attempted via a shadow copy of the file...
Have you tried the copy process after having quit Outlook?

 

[Jorge Coelho]

Hey Julian!

If the file in SVI ends with "{3808876b-c176-4e48-b7ae-04046e6cc752}"
then, yes, this is a ShadowCopy storage file (the long string is the CLSID
(?) of the default VSS writer... you can find the value in the registry).

Yes, I remember the 752 at the end of the CLSID. Thanks for confirming it
was indeed the VSC.

ShadowStorage is availablein all Vista versions and provides System
Restore for Basic/Home Premium systems and SR and "Previous Versions"
(file and folder backtracking) in Business/Enterprise/Ultimate.

That's what gets me... Seems like the VSC service is doing its job of
preserving older versions even on the Vista Home editions, you just don't
get the 'Previous Version' option in the properties of a file or document. I
mean, I know VSC itself is available for all Vista editions since the System
Restore service and backup applications depend on it to access locked files,
but I thought the service was only started when required by one of those
two.

It could be worthwhile moving the PST to another partition without System
Restore/System Protection enabled; it would be a long explanation but this
is the only way to completely exclude a file from getting (at least
partially) copied to ShadowStorage.

You have a point, but then I bet Acronis True Image would be unable to
automatically backup the PST file with Outlook running. I keep my system on
24/7.

Anyway, my aim is not to prevent the PST file from being copied to
ShadowStorage, but to get the damn thing to copy correctly and without
errors across the LAN.

However, ShadowCopies can be accessed even on Vista HP (e.g. with the aid
of ShadowExplorer)

Ah, there you go, didn't know that! Good tip, thanks, I'll google for
ShadowExplorer.

I have no idea why the PST should be being written to, but, is Outlook
still running? If so, copying might be attempted via a shadow copy of the
file... Have you tried the copy process after having quit Outlook?

When I copy the Outlook.pst file, Outlook is NOT running, neither on the
laptop nor on the desktop. I just use Explorer together with drag & drop to
manually copy the file from the laptop into the desktop and vice-versa.

In the first attempt, the old version is going to be written over, so that
would explain VSC saving it to shaddow storage, but after the first error
the file is no longer there (i.e. the pst version that was being overwritten
is deleted when the copy opeation fails for the first time), therefore I
don't really understand what VSC is doing. For me it just looks like the
combination of very large files + Vista file copy method + VSC ends up
causing a network timeout. It wouldn't be so bad if the copy operation could
be resumed at the point where it was interrupted, but alas, it goes back to
the very begining. :-(

 

[Julian]

news:%[email protected]...

Anyway, my aim is not to prevent the PST file from being copied to
ShadowStorage, but to get the damn thing to copy correctly and without
errors across the LAN.


Ah, there you go, didn't know that! Good tip, thanks, I'll google for
ShadowExplorer.

There are other ways too - see Adi Oltean's Blog entry on "A Simple Way to
Access Shadow Copies in Vista" at
http://blogs.msdn.com/adioltean/arc...ple-way-to-access-shadow-copies-in-vista.aspx

When I copy the Outlook.pst file, Outlook is NOT running, neither on the
laptop nor on the desktop. I just use Explorer together with drag & drop
to manually copy the file from the laptop into the desktop and vice-versa.

In the first attempt, the old version is going to be written over, so that
would explain VSC saving it to shaddow storage, but after the first error
the file is no longer there (i.e. the pst version that was being
overwritten is deleted when the copy opeation fails for the first time),
therefore I don't really understand what VSC is doing.

Just because the destination PST was deleted doesn't mean VSS (Volume
ShadowCopy Service) isn't attending to it... it works at the block level,
not the file level, so even though it is deleted changes to the blocks it
occupied will still be monitored and original blocks written to
ShadowStorage when they are overwritten

For me it just looks like the combination of very large files + Vista file
copy method + VSC ends up causing a network timeout. It wouldn't be so bad
if the copy operation could be resumed at the point where it was
interrupted, but alas, it goes back to the very begining. :-(

You don't by any chance have wireless and wired lans in operation
simultaneously do you? I noticed that with my wireless router (Netgear)
large file transfers (~GB) would nearly always fail over the wireless... if
you have both, Vista might be using wireless even if faster wired is also
available - you can override the metric on the connection to force the wired
preference though (but you shouldn't have had to <g>)

One other thing about VSS... if your PST file is ~3GB it is almost certain
to cause the ShadowStorage area on the laptop to run out... Vista will then
create a new System Restore point automatically - and that will take time
etc. which might perhaps affect network timeout - it might not be able to
handle the shadowstorage I/O for your file while it is doing the new
snapshot initialisation...

If you have space, you could try renaming the PST before copying, copy,
delete the original on the laptop and rename the copy - I'd be interested to
know if/how well that works at improving things

One other thought - Have you checked that IPv6 is off if you don't use it?
Could be a contributory factor...

HTH

 

[Julian]

Odd... this is in the sent items of a couple of days ago but hasn't
appeared... hope it arrives this time

news:%[email protected]...

Anyway, my aim is not to prevent the PST file from being copied to
ShadowStorage, but to get the damn thing to copy correctly and without
errors across the LAN.


Ah, there you go, didn't know that! Good tip, thanks, I'll google for
ShadowExplorer.

There are other ways too - see Adi Oltean's Blog entry on "A Simple Way to
Access Shadow Copies in Vista" at
http://blogs.msdn.com/adioltean/arc...ple-way-to-access-shadow-copies-in-vista.aspx

When I copy the Outlook.pst file, Outlook is NOT running, neither on the
laptop nor on the desktop. I just use Explorer together with drag & drop
to manually copy the file from the laptop into the desktop and vice-versa.

In the first attempt, the old version is going to be written over, so that
would explain VSC saving it to shaddow storage, but after the first error
the file is no longer there (i.e. the pst version that was being
overwritten is deleted when the copy opeation fails for the first time),
therefore I don't really understand what VSC is doing.

Just because the destination PST was deleted doesn't mean VSS (Volume
ShadowCopy Service) isn't attending to it... it works at the block level,
not the file level, so even though it is deleted changes to the blocks it
occupied will still be monitored and original blocks written to
ShadowStorage when they are overwritten [because it might be UNdeleted
later...]

For me it just looks like the combination of very large files + Vista file
copy method + VSC ends up causing a network timeout. It wouldn't be so bad
if the copy operation could be resumed at the point where it was
interrupted, but alas, it goes back to the very begining. :-(

You don't by any chance have wireless and wired lans in operation
simultaneously do you? I noticed that with my wireless router (Netgear)
large file transfers (~GB) would nearly always fail over the wireless... if
you have both, Vista might be using wireless even if faster wired is also
available - you can override the metric on the connection to force the wired
preference though (but you shouldn't have had to <g>)

One other thing about VSS... if your PST file is ~3GB it is almost certain
to cause the ShadowStorage area on the laptop to run out... Vista will then
create a new System Restore point automatically - and that will take time
etc. which might perhaps affect network timeout - it might not be able to
handle the shadowstorage I/O for your file while it is doing the new
snapshot initialisation...

If you have space, you could try renaming the source PST before copying,
copy it,
delete the original PST on the laptop and rename the copy - I'd be
interested to
know if/how well that works at improving things

One other thought - Have you checked that IPv6 is off if you don't use it?
Could be a contributory factor...

HTH

 

[Jorge Coelho]

Ok, sorry for not responding in the last couple of days, but I was away and
also wanted to make sure it was not a cable problem first.

I also archived a lot of older messages in Outlook, so the Outlook.pst file
is now down to 1.5 GB from 3 GB.

So, when I came back home today, I got the same error several times when
transfering the Outlook.pst file from the laptop to the desktop. I then used
another known good CAT5 cable to connect the laptop to the router, just in
case... at first things appeared to be going fine and I was going like
'ooops, bad cable, now I'm embarassed', then, bang, same thing, network
transfer goes down to zero for several seconds, resumes for a couple of
seconds, back to zero and error. So the bad cable theory was out.

Since I was in a hurry, and because disabling the Eset firewall had enabled
the transfer to complete the other day, I disabled the firewall in both
computers. This time the whole 1.5 GBs went in a single gulp at maximum
transfer speed (about 11 MB/s)! Hmmm...

So this time I checked the firewall logs. Guess what? On the laptop computer
(the SOURCE computer) there are several entries (one for each failed file
transfer) stating the following:

Detected TCP Desynchronization Attack

In the end this seems to be a problem between the Eset firewall and the
method Vista uses to transfer large files over the network. Since I am an
Eset beta tester, the next step is to let them know about the problem.

There are other ways too - see Adi Oltean's Blog entry on "A Simple Way to
Access Shadow Copies in Vista" at
http://blogs.msdn.com/adioltean/arc...ple-way-to-access-shadow-copies-in-vista.aspx
Thanks!

Just because the destination PST was deleted doesn't mean VSS (Volume
ShadowCopy Service) isn't attending to it... it works at the block level,

Makes all the sense in the world, thanks!

You don't by any chance have wireless and wired lans in operation
simultaneously do you?

At home I only have a WIRED connection.

One other thought - Have you checked that IPv6 is off if you don't use it?
Could be a contributory factor...

IPv6 is enabled. But given the above, I guess this was not a contributing
factor in this case...

Thanks for the help, Julian!

 

[Julian]

Ok, sorry for not responding in the last couple of days, but I was away
and also wanted to make sure it was not a cable problem first.

LOL... I can understand that would have been a red-face moment, but if it
sort of worked I wouldn't have suspected the cable at first

Since I was in a hurry, and because disabling the Eset firewall had
enabled the transfer to complete the other day, I disabled the firewall in
both computers. This time the whole 1.5 GBs went in a single gulp at
maximum transfer speed (about 11 MB/s)! Hmmm...

So this time I checked the firewall logs. Guess what? On the laptop
computer (the SOURCE computer) there are several entries (one for each
failed file transfer) stating the following:

Detected TCP Desynchronization Attack

In the end this seems to be a problem between the Eset firewall and the
method Vista uses to transfer large files over the network. Since I am an
Eset beta tester, the next step is to let them know about the problem.

<snip>

You're most welcome... not that I actually *did* anything to resolve the
problem, sometimes just "being there" helps...