Program writing to disk loses data

I have a fairly simple test pgm that open() a file and write() to end
(append) and then seeks to beginning and writes again. Continuous.
If I power off/on, the file is very large according to "dir", but the file
is empty. If I just end my program, the file is OK. Totally reproducible.
We do NOT have EWF or FBWF enabled at all.

Our "disk" is a Compact Flash - 1gb.
The OS is Windows XPE w/Service Pack 2.

This doesn't happen with our normal WinXP systems, so am sure it has to do
either with WinXPe or the Compact flash.

Help!!!

You may want to give us a little bit more info about your XPe image.
Winlogon or Winlogon based?
What shell?

Also, how exactly you are powering off the target system?

Since you are using CRT calls and probably don't want to mess with API,
make sure you issue fflush() call on the file handle after each write. (or
do _flushall() call)

It should also help if you turn off write caching on your CF. (assuming the
card is marked as fixed)

Regards,
KM


>I have a fairly simple test pgm that open() a file and write() to end
> (append) and then seeks to beginning and writes again. Continuous.
> If I power off/on, the file is very large according to "dir", but the file
> is empty. If I just end my program, the file is OK. Totally
> reproducible.
> We do NOT have EWF or FBWF enabled at all.
>
> Our "disk" is a Compact Flash - 1gb.
> The OS is Windows XPE w/Service Pack 2.
>
> This doesn't happen with our normal WinXP systems, so am sure it has to do
> either with WinXPe or the Compact flash.
>
> Help!!!

j1mw3b wrote:

> I have a fairly simple test pgm that open() a file and write() to end
> (append) and then seeks to beginning and writes again. Continuous.
> If I power off/on, the file is very large according to "dir", but the
> file is empty. If I just end my program, the file is OK. Totally
> reproducible. We do NOT have EWF or FBWF enabled at all.
>
> Our "disk" is a Compact Flash - 1gb.
> The OS is Windows XPE w/Service Pack 2.
>
> This doesn't happen with our normal WinXP systems, so am sure it has
> to do either with WinXPe or the Compact flash.
>
> Help!!!

I suspect it has a lot to do with the much slower write time of the CF.

I have a system in place where there are always 2 copies of my file and
I put a 30 second delay between writing to the primary and secondary.

On startup or my program I attempt to load the primary file. If that
fails, or the checksum is incorrect, I recreate the primary from the
data in the secondary.

This is transparent to the user and I haven't had any reports of lost
data since I did this over 2 years ago.

It's still possible to loose the last entry but that's not critical in
my situation.

--
-Mike

I am not sure what is meant by the XPe image being Winlogon or Winlogon based.
I didn't install it, and this is first experience with XPe. It just boots
up into a gui screen with no login.
My program is a short loop using open(), write(), lseek(), write(), lseek(),
loop to first write. It's simulating a much larger program that originally
failed. fflush() would not apply, but I do use a _commit() call that I found.

I power off the system by turning off power strip - this simulates what
might happen in real world.

I did indeed turn off write caching in the device mgr. Didn't look to see
if the card is marked as fixed (machine is at work, I am at home now).

As far as compact flash speed being a factor... maybe. I have run the same
program on a MontaVista embedded Linux system and it is relatively fast.
Never suffered much data loss on the Linux. Maybe a few recs at the end
until data journaling was enabled - then NO loss.
I'm pretty sure the little program is ok. If I just control-break it (no
signal handler), the data sticks.

This is just totally weird to run a program for about 1 minute, then power
off and on and the file is there (directory entry), but the entire file is
filled with binary zeroes.
I am baffled.

thanks,

Jim

"KM" wrote:

> You may want to give us a little bit more info about your XPe image.
> Winlogon or Winlogon based?
> What shell?
>
> Also, how exactly you are powering off the target system?
>
> Since you are using CRT calls and probably don't want to mess with API,
> make sure you issue fflush() call on the file handle after each write. (or
> do _flushall() call)
>
> It should also help if you turn off write caching on your CF. (assuming the
> card is marked as fixed)
>
> Regards,
> KM
>
>
> >I have a fairly simple test pgm that open() a file and write() to end
> > (append) and then seeks to beginning and writes again. Continuous.
> > If I power off/on, the file is very large according to "dir", but the file
> > is empty. If I just end my program, the file is OK. Totally
> > reproducible.
> > We do NOT have EWF or FBWF enabled at all.
> >
> > Our "disk" is a Compact Flash - 1gb.
> > The OS is Windows XPE w/Service Pack 2.
> >
> > This doesn't happen with our normal WinXP systems, so am sure it has to do
> > either with WinXPe or the Compact flash.
> >
> > Help!!!
>
>
>

Jim,

I meant Minlogon there. My apology for the mistake.

You may probably want to first read about the XP Embedded product:
http://msdn2.microsoft.com/en-us/library/aa460432.aspx. There you will find
sections about Minlogon and Winlogon. Also, you can find there info on how
you could protect your CF card from sudden power loss on target machine.

Just a few more comments on your application:
- What components included in your XPe image is important and sometimes
may have an impact on how data is cached and flushed in the OS. This is
especially important what shell, what logon and what files system (FS)
components you are running. FAT is more deterministic when it comes to
flushing file data to disk than NTFS.

- Unless you want to set this through API, you can use Resource Kit
tools to deal set up disk cache. E.g., dskcache:
http://support.microsoft.com/kb/811392/EN-US
Please read this thread:
http://groups.google.com/group/micr...e1e7d737?hl=en&
http://groups.google.com/group/micr...4be8756e1e7d737

- Since you are using low-level file IO of CRT you are good to call
_commit.
But it is still different what's going on what you ctrol break an app
vs turning off the power. If an app exits or even terminated, the OS will
take care of closing all the handles and flushing file buffers.

- Handling sudden power loss with a solid state device used as storage
it what XPE EEF (embedded enabled features) will help you with. please read
the docs about EWF and FBWF features. Otherwise you will always end up with
some data loss.

- Similar issues have been discussed in this NG numerous times. You may
want to read through some relevant threads:
http://groups.google.com/group/micr...isk+cache+flush

Regards,
KM

>I am not sure what is meant by the XPe image being Winlogon or Winlogon
>based.
> I didn't install it, and this is first experience with XPe. It just boots
> up into a gui screen with no login.
> My program is a short loop using open(), write(), lseek(), write(),
> lseek(),
> loop to first write. It's simulating a much larger program that
> originally
> failed. fflush() would not apply, but I do use a _commit() call that I
> found.
>
> I power off the system by turning off power strip - this simulates what
> might happen in real world.
>
> I did indeed turn off write caching in the device mgr. Didn't look to see
> if the card is marked as fixed (machine is at work, I am at home now).
>
> As far as compact flash speed being a factor... maybe. I have run the
> same
> program on a MontaVista embedded Linux system and it is relatively fast.
> Never suffered much data loss on the Linux. Maybe a few recs at the end
> until data journaling was enabled - then NO loss.
> I'm pretty sure the little program is ok. If I just control-break it (no
> signal handler), the data sticks.
>
> This is just totally weird to run a program for about 1 minute, then power
> off and on and the file is there (directory entry), but the entire file is
> filled with binary zeroes.
> I am baffled.
>
> thanks,
>
> Jim
>
> "KM" wrote:
>
>> You may want to give us a little bit more info about your XPe image.
>> Winlogon or Winlogon based?
>> What shell?
>>
>> Also, how exactly you are powering off the target system?
>>
>> Since you are using CRT calls and probably don't want to mess with API,
>> make sure you issue fflush() call on the file handle after each write.
>> (or
>> do _flushall() call)
>>
>> It should also help if you turn off write caching on your CF. (assuming
>> the
>> card is marked as fixed)
>>
>> Regards,
>> KM
>>
>>
>> >I have a fairly simple test pgm that open() a file and write() to end
>> > (append) and then seeks to beginning and writes again. Continuous.
>> > If I power off/on, the file is very large according to "dir", but the
>> > file
>> > is empty. If I just end my program, the file is OK. Totally
>> > reproducible.
>> > We do NOT have EWF or FBWF enabled at all.
>> >
>> > Our "disk" is a Compact Flash - 1gb.
>> > The OS is Windows XPE w/Service Pack 2.
>> >
>> > This doesn't happen with our normal WinXP systems, so am sure it has to
>> > do
>> > either with WinXPe or the Compact flash.
>> >
>> > Help!!!

KM, thanks a whole lot. OK. I went and read most of those links (not all of
them yet!). I changed to WINApi (CreateFile(), WriteFile, etc. and stopped
losing data. Then realized that my _commit() wasn't in the right places -
went back to low-level I/O w/_commit() and get no data loss now using that.

However, as expected, it is verrrrrryyy slooooow. We had similar slowness
on Linux embedded CF devices with fsync(), which is why we switched to data
journaling which solved the speed and data loss.

Hopefully, some more reading may shed some light on this with XPe.
One post I saw mentioned a feature that lets the disk flush it's cache
before power actually goes out. Maybe that will help.

Anyway, thanks again. Your were very helpful.
The "dark side" of Win32 is getting a little bit lighter....

"KM" wrote:

> Jim,
>
> I meant Minlogon there. My apology for the mistake.
>
> You may probably want to first read about the XP Embedded product:
> http://msdn2.microsoft.com/en-us/library/aa460432.aspx. There you will find
> sections about Minlogon and Winlogon. Also, you can find there info on how
> you could protect your CF card from sudden power loss on target machine.
>
> Just a few more comments on your application:
> - What components included in your XPe image is important and sometimes
> may have an impact on how data is cached and flushed in the OS. This is
> especially important what shell, what logon and what files system (FS)
> components you are running. FAT is more deterministic when it comes to
> flushing file data to disk than NTFS.
>
> - Unless you want to set this through API, you can use Resource Kit
> tools to deal set up disk cache. E.g., dskcache:
> http://support.microsoft.com/kb/811392/EN-US
> Please read this thread:
> http://groups.google.com/group/micr...e1e7d737?hl=en&
> http://groups.google.com/group/micr...4be8756e1e7d737
>
> - Since you are using low-level file IO of CRT you are good to call
> _commit.
> But it is still different what's going on what you ctrol break an app
> vs turning off the power. If an app exits or even terminated, the OS will
> take care of closing all the handles and flushing file buffers.
>
> - Handling sudden power loss with a solid state device used as storage
> it what XPE EEF (embedded enabled features) will help you with. please read
> the docs about EWF and FBWF features. Otherwise you will always end up with
> some data loss.
>
> - Similar issues have been discussed in this NG numerous times. You may
> want to read through some relevant threads:
> http://groups.google.com/group/micr...isk+cache+flush
>
> Regards,
> KM
>
> >I am not sure what is meant by the XPe image being Winlogon or Winlogon
> >based.
> > I didn't install it, and this is first experience with XPe. It just boots
> > up into a gui screen with no login.
> > My program is a short loop using open(), write(), lseek(), write(),
> > lseek(),
> > loop to first write. It's simulating a much larger program that
> > originally
> > failed. fflush() would not apply, but I do use a _commit() call that I
> > found.
> >
> > I power off the system by turning off power strip - this simulates what
> > might happen in real world.
> >
> > I did indeed turn off write caching in the device mgr. Didn't look to see
> > if the card is marked as fixed (machine is at work, I am at home now).
> >
> > As far as compact flash speed being a factor... maybe. I have run the
> > same
> > program on a MontaVista embedded Linux system and it is relatively fast.
> > Never suffered much data loss on the Linux. Maybe a few recs at the end
> > until data journaling was enabled - then NO loss.
> > I'm pretty sure the little program is ok. If I just control-break it (no
> > signal handler), the data sticks.
> >
> > This is just totally weird to run a program for about 1 minute, then power
> > off and on and the file is there (directory entry), but the entire file is
> > filled with binary zeroes.
> > I am baffled.
> >
> > thanks,
> >
> > Jim
> >
> > "KM" wrote:
> >
> >> You may want to give us a little bit more info about your XPe image.
> >> Winlogon or Winlogon based?
> >> What shell?
> >>
> >> Also, how exactly you are powering off the target system?
> >>
> >> Since you are using CRT calls and probably don't want to mess with API,
> >> make sure you issue fflush() call on the file handle after each write.
> >> (or
> >> do _flushall() call)
> >>
> >> It should also help if you turn off write caching on your CF. (assuming
> >> the
> >> card is marked as fixed)
> >>
> >> Regards,
> >> KM
> >>
> >>
> >> >I have a fairly simple test pgm that open() a file and write() to end
> >> > (append) and then seeks to beginning and writes again. Continuous.
> >> > If I power off/on, the file is very large according to "dir", but the
> >> > file
> >> > is empty. If I just end my program, the file is OK. Totally
> >> > reproducible.
> >> > We do NOT have EWF or FBWF enabled at all.
> >> >
> >> > Our "disk" is a Compact Flash - 1gb.
> >> > The OS is Windows XPE w/Service Pack 2.
> >> >
> >> > This doesn't happen with our normal WinXP systems, so am sure it has to
> >> > do
> >> > either with WinXPe or the Compact flash.
> >> >
> >> > Help!!!
>
>
>