EWF RAM Reg diagnostic

[beeb]

Hi,

We will use for the first time XPE on a compact flash with EWF RAM Reg.
The CF will be write protected with EWF, a hard disk (not protected)
will be used to store application data.

I have tested the amount of RAM used for the EWF, the size grows then
seems to be stabilized . But I'm not sure how it will be on the customer
plant: the application will be a 24h/24h 7/7 application.

I try to make (or find) a tool that:
- Launch at startup and display a warning if the CF is no more write
protected (I know how to do using EWF API).

- Check the amount of RAM used by EWF at regular interval (I know how to
do but what will be the best interval: each 24 hours ?)

- If the available RAM becomes critically low, write a log on the
unprotected partition of the hard drive and reboot (a reboot seems
better than a crash). So I need to also check the total amount of
available RAM. What will be the limit for "critically low"?
Maybe I can get this information directly from XPE as an event; this
will avoid my tool to continuously check the RAM.

- The log created before the reboot should report the actual amount of
RAM used for EWF and the list of the files that are in the RAM overlay.
This last point is the most important for me to be able to solve the
problem but I do not have really any idea how to make that.

(We buy an industrial PC with XPE already installed on the CF and we do
not have the possibility of creating a new XPE image)

regards
Beeb

 

[David D]

It will all depend on how much writing you do to your C drive.
That was a concern of my application as well. Since I have no writing to my
C drive in my application, all the writes to the C drive come from the OS.
You need to find those and minimize them. (Internet Explorer caching images
etc).

David

 

[KM]

Beeb,

Please see my comments inline..

We will use for the first time XPE on a compact flash with EWF RAM Reg.
The CF will be write protected with EWF, a hard disk (not protected) will be used to store application data.

I have tested the amount of RAM used for the EWF, the size grows then seems to be stabilized . But I'm not sure how it will be on
the customer plant: the application will be a 24h/24h 7/7 application.

First EWF RAM Reg tip. Try to minimize its RAM usage as much as it is possible in your system design:
http://msdn2.microsoft.com/en-us/embedded/aa731207.aspx

The max limit for the EWF RAM usage is the entire CF size. Basically the situation where all the CF blocks are redirected to RAM is
the worse.

I try to make (or find) a tool that:
- Launch at startup and display a warning if the CF is no more write protected (I know how to do using EWF API).

Long time ago I posted EwfTray component on www.xpefiles.com. It shows the status of the EWF protecttion as a green/red button in
SysTray area. This app (or the idea) may be useful to you if your image runs Explorer Shell.

- Check the amount of RAM used by EWF at regular interval (I know how to do but what will be the best interval: each 24 hours ?)

I am wondering how you are going to check the RAM usage of EWF?
Using EwfMgrRegisterLowSpaceNotification API or something else?

- If the available RAM becomes critically low, write a log on the unprotected partition of the hard drive and reboot (a reboot
seems better than a crash). So I need to also check the total amount of available RAM. What will be the limit for "critically
low"?

Hard to tell and very much dependent on what's included in your image on OS side as well as your own applicaiton(s). Probably an
extensive testing would show the right numbers.
To "help out" your system and the apps you can always implement a pagefile created on the unprotected area (hard disk).

Maybe I can get this information directly from XPE as an event; this will avoid my tool to continuously check the RAM.

IIRC, we asked for a event type notification long time ago and it hasn't yet been implemented within EWF. But there is FBWF (see
below).

- The log created before the reboot should report the actual amount of RAM used for EWF and the list of the files that are in the
RAM overlay.
This last point is the most important for me to be able to solve the problem but I do not have really any idea how to make that.

From your description and troubleshooting at runtime it sounded like FBWF would suit better for your device.
You may want to take a look at it: http://msdn2.microsoft.com/en-us/library/aa940926.aspx.
Especially the API functions such as: FbwfCacheThresholdNotification, FbwfSetCacheThreshold.

(We buy an industrial PC with XPE already installed on the CF and we do not have the possibility of creating a new XPE image)

But you can always "tweak" the image - not by adding components but by removing/disabling some Dlls or functionality, right?

 

[beeb]

First EWF RAM Reg tip. Try to minimize its RAM usage as much as it is possible in your system design:

http://msdn2.microsoft.com/en-us/embedded/aa731207.aspx

I have already followed theses recommandations.
I have some third party products (OPC server,...) and I do not know how
they will behave 24/24 7/7 and under unusual circumstances.

Long time ago I posted EwfTray component on www.xpefiles.com.

I use it: it is very usefull. Thanks.

I am wondering how you are going to check the RAM usage of EWF?
Using EwfMgrRegisterLowSpaceNotification API or something else?

I'm using EWF_VOLUME_CONFIG.RamOverlay.RamDataSize

But there is FBWF (see
below).

But you can always "tweak" the image - not by adding components but by removing/disabling some Dlls or functionality, right?

The XPE image we have is SP2 without Feature Pack 2007 (not FBWF but
EWF). Do you think it's possible to upgrade my XPE image without
rebuilding it ?

For my log file, I was thinking to something like parsing the EWF map
area and find the list of the files that are partially in the RAM
overlay (with the size of this part)

 

[KM]

I have some third party products (OPC server,...) and I do not know how they will behave 24/24 7/7 and under unusual

circumstances.

Well, if you don't do enough testing you can always miss something that is hard to predict or pre-program.

I'm using EWF_VOLUME_CONFIG.RamOverlay.RamDataSize

I see. I don't think using that method is very sufficient - you have to pull that informtation structure in timely manner and often
enough to not miss the big RAM usage increase which might happen easy if end user copies over a big file to an unprotected
partition.
However, beside the EwfMgrRegisterLowSpaceNotification and the method you are using I can't suggest anything else for EWF here (only
checking the system memory usage but that is also insufficient).

The XPE image we have is SP2 without Feature Pack 2007 (not FBWF but EWF). Do you think it's possible to upgrade my XPE image
without rebuilding it ?

Technically it is possible to "upgrade" the image.
First of all, you can use the Image Difference Tool release by Microsoft.
Second, you can go by and update the image on component basis. E.g., update EWF binaries (making sure to stop the driver service
first) and add FBWF and etc.
Just FYI. There were some fixes in EWF area for FP2007 version.

However, there is another side of it - legal.
If you bought an industrial PC with XPe image pre-installed there you are likely obligated to not mess with the image unless it
states otherwise in the contract between you and your device manufacturer. Please contact them and make sure you are on the right
path.

For my log file, I was thinking to something like parsing the EWF map area and find the list of the files that are partially in
the RAM overlay (with the size of this part)

Could you clarify what you meant by "parsing the EWF map"?

 

[beeb]

However, beside the EwfMgrRegisterLowSpaceNotification and the method you are using I can't suggest anything else for EWF here (only

checking the system memory usage but that is also insufficient).

In the EWF API documentation, it is written that
EwfMgrRegisterLowSpaceNotification works only for disk overlay and I'm
using RAM Reg. I will use EWF_VOLUME_CONFIG.RamOverlay.RamDataSize and
call EwfMgrGetProtectedVolumeConfig periodically. I hope that it not
disturb to much the EWF service.

Technically it is possible to "upgrade" the image.
First of all, you can use the Image Difference Tool release by Microsoft.
Second, you can go by and update the image on component basis. E.g., update EWF binaries (making sure to stop the driver service
first) and add FBWF and etc.

To use Image Difference Tool, I need to have an other image that have
already FBWF, right ? (I don't know this tool)

I trained me with the evaluation version of the XP embedded toolkit

Do you think it's possible to do that (technically and from a legal
point of view)
- upgrade my evaluation version with FP2007 (and SP2)
- use Tap.exe on the PC with the existing XPE image
- Import the result on the toolkit to have almost the same target.
- Modify the target to add FBWF
- build the new image and install it on an other PC (XPE will be time
limited on this PC)
- use Image Difference Tool with the old and the new image.
- Copy new and modified binaries on the PC with the old image, apply
also the registry changes.
(theses binaries comes from the evaluation toolkit, are they also time
limited ?)

Just FYI. There were some fixes in EWF area for FP2007 version.

Interresting.
Is it officialy possible to upgrade my image without rebuilding it (some
kind of patch) or do I need to use the same procedure as to implement
the FBWF ?
Do you know what are the fixes ?

However, there is another side of it - legal.
If you bought an industrial PC with XPe image pre-installed there you are likely obligated to not mess with the image unless it
states otherwise in the contract between you and your device manufacturer. Please contact them and make sure you are on the right
path.

I had not thought of that, I will ask them.

Could you clarify what you meant by "parsing the EWF map"?

The goal is to emulate the fbfwmgr /overlaydetail of FBWF (or
fbwfFindFirst / fbwfFindNext)

The solution could be:

- examine the contents of the card field, retrieve the list of the
blocks of the hard disk which are redirected in memory.
- examine the hard disk and find to which files these blocks belong

I have doubts about feasibility:
the location of the map area in the RAM is unknown
The structure of the map area is unknown.

 

[KM]

In the EWF API documentation, it is written that EwfMgrRegisterLowSpaceNotification works only for disk overlay and I'm

And that is true. I was just referring to all methods available while using EWF in general.

using RAM Reg. I will use EWF_VOLUME_CONFIG.RamOverlay.RamDataSize and call EwfMgrGetProtectedVolumeConfig periodically. I hope
that it not disturb to much the EWF service.

Well, the load is not for EWF but for the system in overall.
In a sense, if you make the checking interval too long you may happen to miss an large step in EWF overlay usage.
On the other hands, increasing the frequency will lead to system performance degradation.
This is in general the trade-off you have to make for any intensive monitoring approach.

To use Image Difference Tool, I need to have an other image that have already FBWF, right ? (I don't know this tool)
Yup.

I trained me with the evaluation version of the XP embedded toolkit

Oh, that's no good.

Do you think it's possible to do that (technically and from a legal point of view)
- upgrade my evaluation version with FP2007 (and SP2)
- use Tap.exe on the PC with the existing XPE image
- Import the result on the toolkit to have almost the same target.
- Modify the target to add FBWF
- build the new image and install it on an other PC (XPE will be time limited on this PC)
- use Image Difference Tool with the old and the new image.
- Copy new and modified binaries on the PC with the old image, apply also the registry changes.
(theses binaries comes from the evaluation toolkit, are they also time limited ?)

If you are meaning here to upgrade from eval to full version of the image. Or from eval to eval?
In either case, I don't think the Image Diff tool will handle that well. Not sure though.

The time restrictions there is not about the binaries but about the registry database.

If you follow the steps you just described, I think you will end up replacing the entire registry hives anyway (or at least some
reg.hives like system, software and sam).

Interresting.
Is it officialy possible to upgrade my image without rebuilding it (some kind of patch) or do I need to use the same procedure as
to implement the FBWF ?

It is certainly possible. You basically have to capture all the files and registry that have to changed. For instance, EWF binaries
will change.
If EWF RAM Reg mode used there is no other changes required. However, if EWF RAM or EWF Disk used - the EWF Config partition will
likely have to be re-created.

Do you know what are the fixes ?

I don't know all of them but some of them were mentioned in this newsgroup and product documentaiton.
There are some better integration with EWF RAM Reg mode, HORM specific commands, etc.

The goal is to emulate the fbfwmgr /overlaydetail of FBWF (or fbwfFindFirst / fbwfFindNext)

The solution could be:

- examine the contents of the card field, retrieve the list of the blocks of the hard disk which are redirected in memory.
- examine the hard disk and find to which files these blocks belong

Wow.. Sounds like you are planning to implement your own filter driver.
Not sure why you want to mess with undocumented implementation details of EWF.
Nothing is impossible, of course, but that may appear to be a huge amount of work to reverse engineer the missing parts you don't
know about EWF implementation.

 

[beeb]

I trained me with the evaluation version of the XP embedded toolkit

Oh, that's no good.

Is it not the goal of the evaluation version ?
I was under Windows XP and experimented a lot of hard disk failures or
corruptions (users was powerring off the PC without shuting down Windows).
I needed to know if XPE was the solution. With the evaluation version, I
have tested that all was Ok (Applications still running, CF OK,...).
Now we buy the PC with XPE full version pre installed.

If you are meaning here to upgrade from eval to full version of the image. Or from eval to eval?
In either case, I don't think the Image Diff tool will handle that well. Not sure though.

In my case, I mean: upgrade a full version with some binaries comming
from the eval version

I don't know all of them but some of them were mentioned in this newsgroup and product documentaiton.
There are some better integration with EWF RAM Reg mode, HORM specific commands, etc.

You mean that the EWF fixes are in the target builder tool, not in the
binaries of the image ?

 

[KM]

Is it not the goal of the evaluation version ?

I was under Windows XP and experimented a lot of hard disk failures or corruptions (users was powerring off the PC without shuting
down Windows).
I needed to know if XPE was the solution. With the evaluation version, I have tested that all was Ok (Applications still running,
CF OK,...).
Now we buy the PC with XPE full version pre installed.

I see. Just that through the thread I got lost what version of the XPe you are currently playing with

In my case, I mean: upgrade a full version with some binaries comming from the eval version

That should be fine assuming you are covered on the legal side - your agrement with MS and the device manufacturer must allow
changing the XPe image on the devices and using newest feature pack binaries).

You mean that the EWF fixes are in the target builder tool, not in the binaries of the image ?

Hmm.. Even for the HORM commands (and the new API functions) the EWF binaries have changed.
I haven't seen the list of all fixes for EWF posted by Microsoft in public though.

 

[beeb]

Thank a lot for your help.