Since weekend update, small files up to 4096 bytes get corrupted

[Guest]

Hi,

I have two different servers (different models and manufacturing years), but
both of them running Windows 2000 server with software mirrored NTFS drives,
which suddenly corrupt small image thumbnail files after an hour or two.

No larger file sizes seem to be effected. The file content is set to all
binary 0xdf.

If we replace those files with valid copies, then those files will appear to
be correct for a while (probably read back from cache) but after some time
they will "revert" back to all 0xdf.

I have since been in contact with another customer who has the exact same
problem with his Windows 2000 server also since around the past weekend. In
the days prior we did install recent security fixes and Microsoft product
updates that were released in the past month.

Is there anyone else out there with the same problem?

 

[Pegasus \(MVP\)]

Multi-posted - see your other posts. Please use cross-posting
if you wish to address several newsgroups. Multiposting wastes
everybody's time.

 

[KeithMail]

YES I have been in a losing battle with Msoft support since thursday last
week.

For me, Its KB920958 and it can be forced to happen if the data folder is
compressed (with explorer) and trigger it to happen immediately with a
chkdsk c: /f and a reboot
otherwise you have to wait for a while for it to go bad. We assume the
compressoin is done by a LAZY process and sin't immediate.
I don't think its the length as once the file is corrupt I have edited the
contents to other values and replaced it without a problem, there must be a
trigger byte sequence or something.
If you look at the file with the 2000 resource kit dskprobe sector editor it
complains about the length.
Are you using compressed folders for yourr data cos I can't get it to fail
otherwise and that seems to be my key trigger.
We have disabled antivirus and gone through hoops to prove the point.
Keith

 

[Guest]

Hi Keith,

Let's please try to compare commonalities:

- Yes, on both servers those folders and parent folders have the compression
flag on.

- On both effected servers, the disks are defined as "dynamic"
- They are software mirrored (using Windows 2000' Disk Management)
- On both servers the files show all 0xDF characters (which shows up as the
German special 's', looks a little like a B or a beta character.)
- We run NetShield 4.5 with the 5.1.00 scan engine

How does that compare with your environment?

Andy
at Argos.net

 

[Guest]

PS:

Does that mean you have removed this Hotfix and your problems disappeared?

 

[KeithMail]

The problem has now been seen on over 7 systems here in uk and usa, both
win2000server and Pro across differing machine manufacturers so I have
pretty well eliminated hard disk controllers, device drivers etc.
Simply put if a folder is compressed and one of these (<4096 bytes) files is
dropped into the compressed folder, then either after a period of time
(hence my question of a "lazy" compression process) or by forcing with a
chkdsk it will be corrupt immediately you reboot - this saves time waiting!

I do have mirrored disks on the 2000 server but just standard basic disks on
my 2000pros

If the Kb920958 is removed, then no amount of file placement and chkdsk will
cause the problem, all files already corrupt stay corrupt but no new damage
occurs.
If I confirm my compressed folder is good and then re add the kb920958 ,
followed by chkdsk /f etc the file corruption process occurs again.

I have checked the files with 3 hex editors (Cygnus, win2kresource kit
dskprobe and our own company proprietary viewer)
The problem as far as I can see is
KB920958
With Compressed folders
and only CERTAIN files smaller than 4096 bytes (interesting match to cluster
size perhaps?) I have dropped in text files of a k or so and they seem to be
impervious.

I have disabled all AV (was sophos), and other adaware checkers etc (that's
the beauty of being in a test lab I can pull systems apart as I see fit)

The most difficult question that our Msoft security case engineer wants
answering is what other file types are affected. Until you popped up I was
dealing with one of our own companies proprietary data storage types which
should, as far as the O/S is concerned, be just a binary file.

Once a file has gone bad then it never seems to "comes back", if I hexedit
the 0xDF file content to another hex value the bad file doesn't seem to get
made bad again (go back to 0xdf). This is why I was looking for a string of
bytes as a trigger. Still , I don't have access to Msoft O/s Code and that's
why I hoped raising a "case" with them would get it fixed.

In answer to your other post (17:16) then
At the moment, the solution for me is to
Take off the kb920958 patch
Go through storage and restore all bad files
Wait for a fix.
If the KB is necessary for you then I have not found corruption on a non
compressed folder so maybe the option would be to remove compression, then
restore the bad files
In essence if you need the 920958 then don't enable compression.

Keith

 

[Guest]

Hi Keith,

okay, I will attempt to uninstall Kb920958 tonight! That's the best lead yet.

I will also test the "compression" theory as soon as one of my customers
reports another failure with one of their files.

I definitely have seen at LEAST two files that are larger than 4096.
However, both are very close to a MULTIPLE of 4096 (close to 8K in one case,
in the other case 167,564 bytes = 163K). I inspected the files. In each case
the beginning of the files are good, and ONLY the last 4000 or so bytes have
the 0xDF byte pattern.

I too created a small TXT file (just one line) and it never corrupted, while
other files around it repeatedly did.

I can say, that I have seen corruptions with the following file types:
..JPG
..GIF
..SWF

where the vast majority are small .JPG and .GIF thumbnails and so far I've
only found few files that are larger. But that may be deceiving, because you
simply may not notice the end of a larger file being corrupted, because the
it "starts" out correctly!

Best Regards,
Andy

 

[Guest]

I just removed KB920958.
Its has not been more then an hour so its too early to tell.

We are having problems with jpg files.
Small thumbnails that we create is being corrupted.

Heimir

 

[Guest]

Hi Keith,

you were "right on". Disabling compression on certain folders seem to fix
it temporarily - and after uninstalling the hotfix last night on all servers
with compressed disks I have had no more complaints thus far.

Thank you for sharing your insight and for researching this so thoroughly.

Let's see how long it takes Microsoft to take your bug report and
proactively warn its customers that their hotfix can cause loss of data!

Best Regards,
Andy

 

[Guest]

27 hours since I removed the update and we have not have any problems since.

Heimir

 

[Guest]

Out of curiosity, does Automatic Updates try to re-apply the patch? Did you
have to turn that off?

-jl

 

[KeithMail]

I have auto update disabled as I need control of my environment but after I
removed the patch it re downloaded it and notified me that it was there and
waiting to be applied so I assume the answer is yes to your question.
It is always a difficult compromise in an IT environment, do you want
platform stability or up to date security, its a pity this has shown you
can't always have both.
As of this date I have still NO confirmation from the patch authors to
acknowledge this as happening.
So, either we are imagining this problem or this has bigger repercussions
than we have discovered ourselves?

 

[Guest]

Hey Keith, thanks for the reply, I actually discovered this answer myself. I
have 2 production environment that house most of my clients' files and to
test I uncompressed one, and I uninstalled the patch from the other (which
subsequently auto applied the patch again, hehe). It is a shame that I must
choose between manual patching my application, and therefore violating my own
security policy, or to uncompress many gigs worth of data, costing me 20-30%
additional space now.

I have to believe the lack of acknowledgement is out of embarassment or fear
of the bug's effects. I have over 300 clients who upload images all the time
and I've personally counted around 800 images (especially thumbnails) that
kept getting corrupt during that week from Hell. Luckily, my business isn't
file storage or graphic files. But if it was, this bug could of been
catastrophic. And that's just a with a small business...

If this isn't a Sev 1 bug within MS's system, I would be completedly
shocked. My guess is the next round of patches will fix it and it will be
done very quietly.

-jl

 

[frank]

Hi Keith.

I'm posting over here as you requested. Right now I'm generating a list
of all the files which get altered through the copy operation (I
understand it can happen whenever *any* small file gets written or
altered?). I'll be able to give you a list of filetypes and filesize
limits as well. After that, I'll be trying to roll back the KB 920958
and see if that fixes my problem.

Thanks!

Frank

 

[frank]

Ok, so this is what I've got. After checking some one million files of
about 2500 different file types, I found evidence of corruption in some
five thousand files, of about 150 different filetypes. That's about
seven percent of all the filetypes listed. Contrary to what was seen
earlier, I saw *no* 4k limit on file size.

The largest file corrupted was an mpg file, some 41 mb in size. I also
saw corruption in mp3's and other multimedia files (m4a, wav, etc.),
and even gzipped and zipped files. conversely, even though a lot of
jpegs were hit, out of all available jpegs, only about 3% were
corrupted.

Now I'll be rolling back that patch, and confirm if the problem goes
away for all cases.

 

[JP]

im sending zip files to a customer's windows ftp server. they've been
complaining intermittenly since 2006.08.09, that the zip files we're
sending are occasionally corrupted.

upon inspection, the bad versions are ending on 0xdfs. when i create
a hexdump of both the bad and good copy and then diff them:

---

 

[frank]

I saw the same thing with my files. removing the kb patch removed the
problem.

 

[lily]

We are experiencing something very similar at my company, but it's our
customers that are having the problem, not us.

Our product is a Windows-based product that is currently being
supported back to Windows 98. One of the things we do is save
information (data, graphs, etc.) using OLE structured storage. We
compress some of the larger data streams using a third party package.
Since Sept 5, 2006 our Tech Support staff has received three corrupted
files which, upon further review by the Software Development team, show
the presence of 0xdf at the end of the compressed stream.

Some information about the occurrence and amount of 0xdf:
Corruption 1: Compressed data stream size - 214749 bytes
0xdf starts at byte 210944
Total 0xdf bytes - 3805
Corruption 2: Compressed data stream size - 901867 bytes
0xdf starts at byte 898048
Total 0xdf bytes - 3819
Corruption 3: Compressed data stream size - 98889 bytes
0xdf starts at byte 95232
Total 0xdf bytes - 3657

This is devastating to our customers because subsequent attempts by our
software to open the saved file fail because the "decompression"
algorithm fails.

We are currently gathering information from the affected customers to
see if hot fix KB920958 is a possibility.

 

[Guest]

Microsoft Support confirmed Hotfix 920958 is bad:

"We are aware the issue you are experiencing. A corresponding bugcheck
request is currently open, and the develop team is working on this issue.
However, the hotfix for this issue is not ready.

0xDF is the data pattern that NTFS returns when it has problem to decompress
the file (e.g.. the compression fragments are corrupted and can't be
decompressed). Based on my research, the actual raw data on the disk is not
changed, it shows as 0xDF because the system cannot decompress the file and
display the data correctly. So the corrupt is not permanent.

Further more, the issue only occurs on files which containing Hexadecimal
codes."

 

[mrkiwi]

" Further more, the issue only occurs on files which containing
Hexadecimal codes."

This must be why you (or others?) couldnt get a text file to be
corrupted.