Collection memory usage in debug build

[greg.merideth]

I'm working on a service for a project that is producing some bizarre
behavior. In debug mode while the service is running, the memory usage
of the service (watching with process explorer) goes from 18MB to 34MB
to 711+MB back down to 51MB then down to 18MB every 1-2 minutes. The
service maintains a List<t,v> of 4400 objects that are being processed
throughout the services life. No secondary objects are created nor
released.

I can understand the 18MB of usage but why on earth does the service
"jump" to over 700MB of memory usage then back down to 50 and then to
18. What could the GC being doing with 700+MB of memory than the
service even uses? In release mode the service consumes 18-25MB on
average over a 12 hour run, it's just in debug that things go insane.

What could possibly be going on?

Note : This didn't happen when we built in VS2003. Since going to
VS2005 and building with generics we get this debug memory nuttyness.

 

[JoSkillz]

Can you give us an example of your code implmentation?

 

[greg.merideth]

After blocking out code segments and monitoring I've been able to
whiddle down the part that appears to be the culprit and I've included
the two methods below. CRCGenerationT runs as a thread and calls
GenerateCRC32 around 4-5 times/sec. I didn't create the CRC part I'm
just trying to make it work.

After adding a GC.Collect at the end of the GenerateCRC32 I've brought
memory consumption during debug builds down to ~33MB and I have yet to
see any of those 700+ MB spikes, remove the garbage collection and
bingo, 700-900MB memory spikes.

Here's the code snippet - hopefully it's enough to point out what I
need to rewrite. I'm looking for the code for the Inc and Dec
UpdateFileIOCounter's but all they do from looking at the MSIL is call
[performancecounter].Increment(). Thread wise I dont know if thats
causing an issue.

private void CRCGenerationT() {
int TSleepDuration = GetMSSleepDuration(idleTiming);

while(ProcessFileCRCs) { // thread flag from baseio.cs
if(fileCRCProcessedIndex != fileCollection.Count) {
GenerateCRC32();
++fileCRCProcessedIndex;
Thread.Sleep(TSleepDuration);
}

if(fileCRCProcessedIndex == fileCollection.Count &&
IsScanningComplete)
ProcessFileCRCs = false;
}
this.TCRC32ThreadDequeue = false;
}

private void GenerateCRC32() {
try {
FileStream CRCFileStream = new
FileStream(IFileCol[fileCRCProcessedIndex].FullIOName, FileMode.Open);
StreamReader CRCStreamReader = new StreamReader(CRCFileStream);
crcReader = CRCStreamReader.ReadToEnd();
CRC.Init(CRCTool.CRCCode.CRC32);
byte[] rawBytes =
System.Text.ASCIIEncoding.ASCII.GetBytes(crcReader);
ushort usCrc32 = (ushort)compCRC.crctablefast(rawBytes);

fileCollection[fileCRCProcessedIndex].Hash =
usCrc32.ToString();
IncUpdateFileIOCounter(Counters.Methods.GenCRC32);
}
catch {
fileCollection[fileCRCProcessedIndex].Hash = string.Empty;
DecUpdateFileIOCounter(Counters.Methods.GenCRC32);
}

GC.Collect();
}

 

[JoSkillz]

Off the back anytime you are using ANY stream object be sure to wrap it
in a "using" statement. This will insure that the stream is closed and
properly disposed of when you it falls out of scope.

http://msdn.microsoft.com/library/d...ry/en-us/csspec/html/vclrfcsharpspec_8_13.asp

http://www.c-sharpcorner.com/Code/2002/Dec/UsingStatement.asp

Hope this helps.

 

[greg.merideth]

I'll wrap a using block around it now and run some tests. Thats just
one of those things that after days of staring at code one just kind of
forgets. Thanks.

 

[JoSkillz]

Happens to all of us!

Good luck