Memory leak during file writes with IO perfromance loss, help !

[Geoff]

I have an application that is attempting to read data in from a capture card
at high speed, process it and write it back out to disc. Writes to disc are
capable of sustaining in the order of 600Mbytes/sec with the set-up I have.
The input data rate to the capture card is 200Mbits/s. The data is DMA ed
from the capture card across the PCI bus to memory. An application then
picks up this data processes it and writes it out to disc. Size of buffers
transferred both in and out to disc is 128Kbytes.

I use the standard fwrite, FileWrite and other such standard C & C++ methods
to stream the data to disc. I have found however that under Windows 2000 &
XP, there is a problem with writing data to disc. Due to the fact that I am
creating files in the order of 100Gbytes I have spotted what might be a bug
in the kernel code. During writes there is a memory loss of approx.
4kBytes/sec and IO performance significantly degrades with time. This memory
loss is also seen under Windows NT, but the behaviour is different, since
when the loss reaches a threshold of approx. 4Mbytes the memory is released
and over time there is no perceptible loss of IO performance.

Does anybody know if this is a bug and why this loss of memory and IO
performance occurs ? Is there a cure to this ? Is there another way of
writing data to disc at high speed over a sustained period of time ?

Any advice or help appreciated.

Geoff

 

[ScottD]

You might look into the ReleaseWorkingSet API. Call that func every so
often, and it might keep your system from bogging down.

ScottD