mysterious discrepancy in the reported free space on two identicalusb drives

[sobriquet]

http://img237.imageshack.us/img237/2852/passportdiscrepancy.jpg

I have two identical WD 1tb passport usb drives filled with identical
data.
When I select all files (including hidden and system files) and
display its properties, the listed filesizes match up between the two
drives. But when I go to 'my computer' and display the properties of
each of the two passport drives, they differ in the size of the data
stored on the disk. Can anyone explain this discrepancy in the
reported data/free space between the two identical usb drives with
identical data?

The recycle bin is empty on both drives.

Kind regards and thanks in advance for any suggestions, Niek

 

[Arno]

http://img237.imageshack.us/img237/2852/passportdiscrepancy.jpg

I have two identical WD 1tb passport usb drives filled with identical
data.
When I select all files (including hidden and system files) and
display its properties, the listed filesizes match up between the two
drives. But when I go to 'my computer' and display the properties of
each of the two passport drives, they differ in the size of the data
stored on the disk. Can anyone explain this discrepancy in the
reported data/free space between the two identical usb drives with
identical data?

The recycle bin is empty on both drives.

Kind regards and thanks in advance for any suggestions, Niek

One thing is that directories grow when more files are put in them
but they do not automatically shrink. Then there is the possibility
that both drives have different filesystem parameters, e.g one can
have a larger cluster size, which results in more wasted space
at the end of the files for files that do not exactly end
on a cluster boundary.

Other possibilities exist, What are the concrete numbers you
are seeing.

Arno

 

[sobriquet]

One thing is that directories grow when more files are put in them
but they do not automatically shrink. Then there is the possibility
that both drives have different filesystem parameters, e.g one can
have   a larger cluster size, which results in more wasted space
at the end of the files for files that do not exactly end
on a cluster boundary.

Other possibilities exist, What are the concrete numbers you
are seeing.

You can see the numbers in the screenshot at the top of my posting. As
far as I know, both drives have been formatted in an identical way
(NTFS is the default filesystem).

 

[Rod Speed]

http://img237.imageshack.us/img237/2852/passportdiscrepancy.jpg

I have two identical WD 1tb passport usb drives filled with identical
data. When I select all files (including hidden and system files) and
display its properties, the listed filesizes match up between the two
drives. But when I go to 'my computer' and display the properties of
each of the two passport drives, they differ in the size of the data
stored on the disk. Can anyone explain this discrepancy in the
reported data/free space between the two identical usb drives with
identical data?

You're displaying the contents of the recycle bin with the top display, not the entire drive.

The recycle bin is empty on both drives.

No it is not.

 

[Arno]

You can see the numbers in the screenshot at the top of my posting. As
far as I know, both drives have been formatted in an identical way
(NTFS is the default filesystem).

The discrepancy seen is well within potential differences
in directory (or other metadata) size. Don't worry about it.

Worry if you see differences in the combined file size.

As you can, see the metadata is about 156MB. It size is not
as well-defined as the file size and may even depend on the
order the files were put on disk. The difference of 29kB
is pretty small.

Arno

 

[Jonathan de Boyne Pollard]

http://img237.imageshack.us/img237/2852/passportdiscrepancy.jpg

I have two identical WD 1tb passport usb drives filled with identical data. [...]




... but not necessarily identical metadata.  That 28KiB difference is a mere 28 deleted MFT records, for example.  Or it could be journal entries, security descriptor records, or quite a number of other things.

 

[sobriquet]

You're displaying the contents of the recycle bin with the top display, not the entire drive.

It might look like that, because the windows operating system (vista
in particular) was designed by a bunch of retards, but I can assure
you that I'm displaying the entire contents of the drive. If you look
carefully, you can
also see this in the top address bar that is pointing to the root
folder on both sides in the two explorer windows.

No it is not.

Well, I double checked it again and the recycle bin is empty on both
drives.

 

[sobriquet]

http://img237.imageshack.us/img237/2852/passportdiscrepancy.jpg
I have two identical WD 1tb passport usb drives filled with identical data. [...]



... but not necessarily identical metadata.  That 28KiB difference is amere 28 deleted MFT records, for example.  Or it could be journal entries, security descriptor records, or quite a number of other things.

But isn't this the kind of meta data that is supposed to disappear
when the recyclebin is emptied?
Is there any other way to clear any superfluous data somehow?
If two drives have the same data and one has more free space, surely,
somehow, it must be possible to free up
the same space on the other drive, because we're talking about two
identical drives and identical data that is supposed to be stored on
the drive.

I don't care about the minimal filespace that would be obtained this
way, I'm just interested in figuring out exactly what's responsible
for this discrepancy between the two drives in reported free space
while they contain identical data that has been stored on the drives
in an identical fashion.

 

[Jonathan de Boyne Pollard]

http://img237.imageshack.us/img237/2852/passportdiscrepancy.jpg
I have two identical WD 1tb passport usb drives filled with identical data. [...]




... but not necessarily identical metadata.  That 28KiB difference is a mere 28 deleted MFT records, for example.  Or it could be journal entries, security descriptor records, or quite a number of other things.



But isn't this the kind of meta data that is supposed to disappear when the recyclebin is emptied?



Put simply: No.  Deleted MFT records are nothing to do with files in the recycle bin, for example.



Is there any other way to clear any superfluous data somehow?



Put simply: Short of drastic measures such as reformatting the volume, no.  The MFT doesn't shrink in normal operation, for example.  And the security descriptor stream is only compacted by chkdsk.  (See MSKB 919241.)



we're talking about two identical drives and identical data that is supposed to be stored on the drive.



No, we're not.  As I said, the metadata are not necessarily identical.

 

[sobriquet]

http://img237.imageshack.us/img237/2852/passportdiscrepancy.jpg
I have two identical WD 1tb passport usb drives filled with identical data. [...]



... but not necessarily identical metadata.  That 28KiB difference is amere 28 deleted MFT records, for example.  Or it could be journal entries, security descriptor records, or quite a number of other things.



But isn't this the kind of meta data that is supposed to disappear when the recyclebin is emptied?



Put simply: No.  Deleted MFT records are nothing to do with files in the recycle bin, for example.



Is there any other way to clear any superfluous data somehow?



Put simply: Short of drastic measures such as reformatting the volume, no..  The MFT doesn't shrink in normal operation, for example.  And the security descriptor stream is only compacted bychkdsk.  (See MSKB 919241.)



we're talking about two identical drives and identical data that is supposed to be stored on the drive.



No, we're not.  As I said, the metadata are not necessarily identical.

I see. Could a virus or malware somehow gain access to this space
where this metadata is stored to hide a copy of itself there?
Can I use a diskeditor like HxD or DiskExplorer for NTFS to view this
metadata somehow?

 

[Arno]

http://img237.imageshack.us/img237/2852/passportdiscrepancy.jpg
I have two identical WD 1tb passport usb drives filled with identical data. [...]



... but not necessarily identical metadata.? That 28KiB difference is a mere 28 deleted MFT records, for example.? Or it could be journal entries, security descriptor records, or quite a number of other things.



But isn't this the kind of meta data that is supposed to disappear when the recyclebin is emptied?



Put simply: No.? Deleted MFT records are nothing to do with files in the recycle bin, for example.



Is there any other way to clear any superfluous data somehow?



Put simply: Short of drastic measures such as reformatting the volume, no.? The MFT doesn't shrink in normal operation, for example.? And the security descriptor stream is only compacted bychkdsk.? (See MSKB 919241.)



we're talking about two identical drives and identical data that is supposed to be stored on the drive.



No, we're not.? As I said, the metadata are not necessarily identical.

I see. Could a virus or malware somehow gain access to this space
where this metadata is stored to hide a copy of itself there?
Can I use a diskeditor like HxD or DiskExplorer for NTFS to view this
metadata somehow?

Very, very unlikely as this is only possible if the malware
has a very good unsderstanding of NTFS. This would be hard to
do and make the malware large, hence easy to detect. Malware
can hide in other places thogh, for example the partially
used clusters at file ends or brazenly in seemingly unused
space.

This is almost certainly not malware. Also the extra
space may well be used with the emtadata just being a bit
more compact on the one drive.

As I said, don't worry about this, a bit of uncertainity
in metadata size is expected in modern filesystems. If
you look at the actual size difference, you can understand
why nobody invested a lot of effort to optimize this.
It is just not worth the effort.

If you really want to make both drives the same size, the
only way I see is to format them and then put all files
on both using exactly the same procedure. This may still
not work, as the metadata is always slightly different,
for example in the timestamps.

If you really want to look at the metadata, good luck.
I expect analyzing these drives manually in detail might
take more than a month of time, possible much more.

Arno

 

[Rod Speed]

sobriquet wrote

http://img237.imageshack.us/img237/2852/passportdiscrepancy.jpg
I have two identical WD 1tb passport usb drives filled with identical data. [...]
... but not necessarily identical metadata. That 28KiB difference is
a mere 28 deleted MFT records, for example. Or it could be journal
entries, security descriptor records, or quite a number of other things.
But isn't this the kind of meta data that is supposed
to disappear when the recyclebin is emptied?
Put simply: No. Deleted MFT records are nothing to do with files in
the recycle bin, for example.
Is there any other way to clear any superfluous data somehow?
Put simply: Short of drastic measures such as reformatting the
volume, no. The MFT doesn't shrink in normal operation, for example.
And the security descriptor stream is only compacted bychkdsk. (See
MSKB 919241.)
we're talking about two identical drives and identical data that is supposed to be stored on the drive.
No, we're not. As I said, the metadata are not necessarily identical.

I see. Could a virus or malware somehow gain access to this space
where this metadata is stored to hide a copy of itself there?

Yes, it might be able to hide a copy of itself, but
there is no way to get it executed from there.

Can I use a diskeditor like HxD or DiskExplorer for
NTFS to view this metadata somehow?

Yes, anything that can dump the contents of sectors you specify can do that.

Interpreting what you see tho is much harder.

Quite a bit of the detail of NTFS structures have never been formally documented.

 

[Yousuf Khan]

If you really want to make both drives the same size, the
only way I see is to format them and then put all files
on both using exactly the same procedure. This may still
not work, as the metadata is always slightly different,
for example in the timestamps.

Something like a RAID-based mirroring system which copies data at a much
lower level than the filesystem.

Yousuf Khan

 

[Arno]

Something like a RAID-based mirroring system which copies data at a much
lower level than the filesystem.

Yes. Or making a copy with a sector imager. Don't use both
drives at the same time aftert this, as the GUIDs will
allso have been copied.

Arno

 

[Jonathan de Boyne Pollard]

Or making a copy with a sector imager. Don't use both drives at the
same time aftert this, as the GUIDs will also have been copied.

That's only the case if one copies the entire disc, rather than copies
just the contents of a specific volume (as tools such as DISKCOPY and
its equivalents do). When copying a specific volume, one only need
worry about volume serial numbers; and the failure mode that duplicate
volume serial numbers predominantly cause (failure to recognize
removable disc changes in some operating systems) won't occur with both
discs present simultaneously, and won't occur with non-removable media,
in any case.