RAM Reg EWF with hibernation questions

[Sean Liming \(eMVP\)]

After thinking about this a bit...

Isn't the real problem here the FS information stored in the hiberfile?

Let me spin the idea a different way:

Commit the file to the disk using EWFMGRCommitFile - I would assume that
this updates the FS tables on the disk. Then it should be a matter of update
the FS table in the hiberfile with the update FS tables on the disk. Would
that work?

Regards,

Sean

 

[Slobodan Brcin \(eMVP\)]

Hi Sean,

Isn't the real problem here the FS information stored in the hiberfile?

Of course it is, this was always a problem that prevent updates and that need from us to use stateless operations.

Commit the file to the disk using EWFMGRCommitFile - I would assume that
this updates the FS tables on the disk. Then it should be a matter of update
the FS table in the hiberfile with the update FS tables on the disk. Would
that work?

First problem here would be how can we update hibrfile. (We do not know anything about compression and data structure in it.)
Second problem would be that even if we could update FS cached in hiberfile we could not update information's about open files hold
by applications/driver that were running when we made an original image. So my best guess is that this approach is impossible even
in theory.

BTW: Let me simplify description of my approach so developers that read this in future can understand it: (I will abstract things so
that no optimizations are mentioned here.)

Field device update procedure.
Step 1. On running XPe we force RAM EWF driver to load whole protected partition data to memory. So all further read request will be
satisfied from memory instead from disk.
Step 2. We use image file of new/updated XPe and write it on disk level (bypassing EWF) over the current OS partition.

After the reboot we will have brand new OS that will do what ever it is made to do. So basically we replaced whole underlying OS.
Previous posts were explaining how can we minimize image size and RAM overlay usage during these operations.

Regards,
Slobodan

 

[KM]

Slobodan,

One question I have that may be easy for you to answer.

You are going to bypass EWF to update the hibernate file through the direct disk access.
So instaed of the scenerio you oulined below, wouldn't that be easier to update all the changed sectors the same way (direct disk
access) and then just reboot. The update will include all the sectors that were different between the master image and current
client image and the hiber.sys file sector that were changed.

Also, another question to understand your scenerio.. You basically use EWF overlay in RAM to cover all the changed files (actually
sectors) so that on next boot it won't matter the differences between real FS and the FS cache in hiber file, right?
Anyway, if you are going to make a few such update later on, aren't you going to end up with the whole FS redirected to EWF RAM
overlay? And, therefore, the EWF RAM usage would be huge and this all may not worth the efforts?

Interesting ideas, btw

Konstantin

 

[Slobodan Brcin \(eMVP\)]

Konstantin,

You are going to bypass EWF to update the hibernate file through the direct disk access.
So instaed of the scenerio you oulined below, wouldn't that be easier to update all the changed sectors the same way (direct disk
access) and then just reboot. The update will include all the sectors that were different between the master image and current
client image and the hiber.sys file sector that were changed.

Easier, of course. But I would like to keep image file size on minimum, and that require some file analysis.

We have three different types of regions on disk.
1. Data area occupied by hiberfile
2. FS area with valid data in it.
3. FS available clusters of data not assigned to any file in FS.

If we have 1GB of ram hiberfile would be 1GB. But internal hiberfile usage could be less than 50 MB. And we would need only to save
valid data to image file. (Or some other form of specific analysis of this file could be also used)

Also, another question to understand your scenario.. You basically use EWF overlay in RAM to cover all the changed files (actually
sectors) so that on next boot it won't matter the differences between real FS and the FS cache in hiber file, right?

After reboot both FS and hiber file would be updated. Also steps 2 and 3 would allow us not to overload RAM overlay with sectors
that are not used by FS.

Anyway, if you are going to make a few such update later on, aren't you going to end up with the whole FS redirected to EWF RAM
overlay? And, therefore, the EWF RAM usage would be huge and this all may not worth the efforts?

No, I would not
I do not expect heavy usage of RAM overlay during this procedure at all. (Unless you update your image from SP1 to SP2)

Regards,
Slobodan

 

[KM]

Slobodan,

Easier, of course. But I would like to keep image file size on minimum, and that require some file analysis.

We have three different types of regions on disk.
1. Data area occupied by hiberfile
2. FS area with valid data in it.

These two above I was actually referring to. But you are right, that would require some FS analysis.

3. FS available clusters of data not assigned to any file in FS.

If we have 1GB of ram hiberfile would be 1GB. But internal hiberfile usage could be less than 50 MB. And we would need only to

save valid data to image file. (Or some other form of specific analysis of this file could be also used)

Because of compression it could be hard to capture only changed blocks in hiber file unless you know the hiber file format.

After reboot both FS and hiber file would be updated. Also steps 2 and 3 would allow us not to overload RAM overlay with sectors
that are not used by FS.

I probalby missed the step from your list where you update the files (to be commited) on the disk through direct access.

No, I would not
I do not expect heavy usage of RAM overlay during this procedure at all. (Unless you update your image from SP1 to SP2)

I have just re-read your previous message (with fresh head now ) and finally found the missing peice because of which I could
not undertand why you are trying to "trick EWF".
I understand it now. You basically just want to create a situation where any OS file reads/writes to the sectors that we need to
update on the client will go to EWF overlay. Then any changes that go directly to the disk for those sectors won't break anything
from OS/FS point of view because FS will continue using EWF RAM OVerlay version of these sectors during this session anyway.

With that in mind, my question about the EWF RAM usage does not make much sense

Interesting solution.. Although I still believe this would worth efforts if you could lower the number of sectors to be updated in
hiber.sys file. Who wants to dowload 1G file when you can just create it locally (and this will add one more reboot).

Btw, you coudl use your idea in SP1 where EWF does not support committting particular files (sectors).
Right now in the filed you have to do a reboot on the client to clean up the EWF overlay from all the unnecesary user changes. then
you do the admin required changes and commit them to the disk, do the reboot again.
Instead of all this, we can just "trick the EWF" following your method and then update the changed sectors directly on disk level
and do only one reboot.

Thanks,
Konstantin

 

[Slobodan Brcin \(eMVP\)]

Konstantin,

Yup you understood it correctly

Interesting solution.. Although I still believe this would worth efforts if you could lower the number of sectors to be updated in
hiber.sys file. Who wants to dowload 1G file when you can just create it locally (and this will add one more reboot).

Hey I just hope that someone from MS will do all this ;-) After all they have access to hiberfile specifications and DUA server
components already available, they could integrate all this.
If not I'm certain that with little effort we could figure out layout of hibernation file, but since I do not need this option at
the moment, I guess that this will stay only as an idea for a XPe improvement,

Regards,
Slobodan

 

[KM]

Slobodan,

Hey I just hope that someone from MS will do all this ;-) After all they have access to hiberfile specifications and DUA server
components already available, they could integrate all this.

Ah, that's your idea

If not I'm certain that with little effort we could figure out layout of hibernation file, but since I do not need this option at
the moment, I guess that this will stay only as an idea for a XPe improvement,

For the moment all I remember is that kernel compressing very 64Kb chunk of the hiber file to have the overlap between the
compression and disk I/O operations. The same on resume.
Any compression algorithm coould be reverse engineering if you have a sample file (although, it is just a compression, not no
encryption) but then it will stumble over legal aspects on using the final solution.
So I guess you are right - this is MS call.

Regards,
Konstantin