EWF and NTFS free list behavior

[Mark K Vallevand]

When a file is deleted, do its sectors go onto the NTFS free list at the
beginning or the end?

We think we are seeing EWF memory usage growth when there shouldn't be any.
Consider copying a large file to the disk and deleting it, then copying it
again. The second copy should get sectors from the free list that were just
freed (ideally) so EWF memory usage shouldn't grow. But, it seems the
opposite. It seems like this:
ewfmgr c:
copy c:\tengig c:\tengig1
del c:\tengig1
copy c:\tengig c:\tengig2
del c:\tengig2
ewfmgr c:
Twenty gig of EWF memory usage growth.

I'm going to test this to see if the report is accurate. But, if anyone
knows how it really works, I'd like to hear it.
--
Regards.
Mark K Vallevand (e-mail address removed)

Beer is living proof that God loves us and wants us to be happy.
- Benjamin Franklin


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you received
this in error, please contact the sender and delete the e-mail and its
attachments from all computers.

 

[Slobodan Brcin \(eMVP\)]

Hi Mark,

EWF driver is not FS filter driver so it has no idea what you do with your
or even RAW data.
EWF look only at what you tried to write and whether this sector is already
in overlay if it is then its content is changed if not it will allocate
little more of your memory :-(
EWF overlay usage will never decrease (unless you reboot computer).

Even EwfMgrCommitFile will not reduce memory overlay usage :-( But on the
bright side at least this command will allow to EWF to reuse this allocated
memory for other sectors.

During the last week or two I have been bugging the MS about this and few
other missing features.
I wanted to make application "for my personal fun" that would decrease EWF
overlay usage just for cases that you described.

Regards,
Slobodan

 

[KM]

I think Mark was heading to the fact that NTFS should reuse the most recent MFT records for the disk writes.

The MFT keeps a list of all files written to disk and never removes an entry, just marks it as deleted. When a file is marked for
deletion it's location on disk and it's contents are not destroyed; the space occupied is released for re-use. The more recent the
file entry in the MFT, the more likely it is that it will be overwritten by the next write to disk. Data with an older entry in the
MFT, is less likely to be overwritten by new disk writes.
A file that is created and then immediately deleted is highly likely to be overwritten by general system usage, based on its entry
in the MFT.

Therefore the EWF usage should not grow very fast if you just do the copy/del test on the same size files. (I mean it may still grow
but not very fast).

 

[Slobodan Brcin \(eMVP\)]

Konstantin you are right about what Mark said

BTW:
You are probably right about MFT entries, but how does that add up with data
clusters which consume space?

As you know on NTFS you have bitmap entry that tell you usage/availability
of each cluster on disk. How NTFS will pick up free clusters is really
mystery. It might even try to use different clusters that recently used just
to reduce writes to same place on disk.
Do you have some MS link with official information's on this?

Regards,
Slobodan

 

[KM]

Slobodan,

BTW:
You are probably right about MFT entries, but how does that add up with data
clusters which consume space?

I read a little bit more and I am about to change my statement.
Seems like NTFS will use the MFT reconds marked as deleted as last resort. Basically the MFT Zone is not subtracted from available
(free) drive space used for user data files, it is only space that is used last.
http://support.microsoft.com/default.aspx?scid=kb;en-us;174619

As you know on NTFS you have bitmap entry that tell you usage/availability
of each cluster on disk. How NTFS will pick up free clusters is really mystery.
It might even try to use different clusters that recently used just
to reduce writes to same place on disk.

Hey.. that would be a "CF'ed" version of NTFS
I never heard NTFS was anyhow adopted for an embedded usage.
But as I outlined above, it may appear that NTFS tries to create new MFT records for new files until MFT Zone is filled up instead
of reusing the one marked deleted.

Do you have some MS link with official information's on this?

I do not. :-(
Otherwise I'd already post it here since Mark seems to be struggling with the issue for quite a while now

 

[Slobodan Brcin \(eMVP\)]

Konstantin,

How NTFS uses MFT records probably is not interested to us anyhow since they
are smaller than one sector.

Hey.. that would be a "CF'ed" version of NTFS
I never heard NTFS was anyhow adopted for an embedded usage.

I never heard that either, but algorithm that XPe uses to allocate available
clusters will remain mystery.
They could always analyze available cluster bitmap or they could have some
internal memory lists of jointed clusters or many other simpler algorithms
for allocating data clusters :-(

Regards,
Slobodan

 

[KM]

Slobodan,

How NTFS uses MFT records probably is not interested to us anyhow since they
are smaller than one sector.

You are right. But I wasn't clear in my post above. I actually meant that NTFS might reuse the used clusters last to support more
reliable file recovery process on NTFS volume. Basically, they may be trying to preserve deleted file data as long as it is
possible.

I never heard that either, but algorithm that XPe uses to allocate available
clusters will remain mystery.
They could always analyze available cluster bitmap or they could have some
internal memory lists of jointed clusters or many other simpler algorithms
for allocating data clusters :-(

True. And it may be even there (NTFS evolved very much since last few years) but we just don't know the right key to turn on that
behaviour
Konstantin

 

[Matt Kellner \(MS\)]

My understanding is that EWF just stores a literal record of all write
operations to the protected volume, regardless of what those operations are,
and it does not have a concept (or a very good one, anyway) of what the
actual file system is doing or how it allocates space. So it might take
Mark's scenario and store a list of operations as:

Write file X (20 MB) ... Impact = 20 MB
Delete file X (Deallocate sectors) ... Impact = 1 KB
Write file Y (20 MB) ... Impact = 20 MB
....

I don't believe EWF is sophisticated enough to deallocate and reuse space
within the overlay in the same way space is reclaimed in the file system.

--
Matt Kellner ([email protected])
STE, Windows Embedded Group

This posting is provided "AS IS" with no warranties, and confers no rights.
===============================

 

[Slobodan Brcin \(eMVP\)]

Hi Matt,

Yup EWF has no idea about what FS is doing or whether there is FS present at
all. I have been testing it without FS present also.
I have been bugging Troy Shaw about CompareBeforeAlloc CompareBeforeWrite
optimization registry keys.
Unfortunately CompareBeforeWrite is not implemented and it would save a day
for my application.

CompareBeforeAlloc has some write size block limitation. (Not a big problem
but still).
Second problem with this (which is a bigger problem BTW) is that once when
sector enter overlay EWF will not compare new changes to that sector with
the data on disk, and in case that there is a data match EWF won't free
memory for that sector.

If that option worked I would be able to make a tool that would figure out
cluster usage and decommit memory by writing same data that is stored on
physical disk.

Regards,
Slobodan