Memory grow and out of memory

[Florin]

Hi all,

I have a problem related to memory grow on a server application
which is basically stateless (I have some static info loaded). The
client accesses this server using remoting and it has worked for about
2 years without problems.
The problem I have encountered lately is that the memory on server
side started to grow and not to be released. I have checked first new
functionalities but after isolating these on separate server, the
memory did not grow.
I have also used .Net Memory profiler to find memory leaks on
server and I could not find something that would explain this grows.
With the dump file created with windbg I see that there is memory
allocated of about 10 Mb and the rest 80-120 Mb is occupied by Free
Object (which I understand is the space ready for allocation). These
tests were done on our test environment and not on production so they
may not be conculdent.
Now, the service on production takes up to 600-800 Mb and then I
receive Out Of Memory exception. There physical Memory is about 2 Gb
and this is dedicated server, so it should be able to allocate more
memory. I suspect that the blame is on the memory fragmentation, but I
cannot be sure.
Before posting this, I have checked several issues on this group,
but none of them has helped me so far. I know about the STA and MTA
problem, and I not this .

The good know is that I have been able to isolate this and I know
that on Windows 2003 is working correctly, and the memory is
deallocated, and the problem I have is on Windows 2000 (sp4 and fwk 1.1
sp1). What is also interesting is that the memory allocation pattern is
like this: first it allocates aprox. 100 Mb, and than stays with this
allocation for about 4h, than allocates another 100 Mb and again it
stays about 4h etc. After 2-3 days I get the out of memory exception.
On the 4h period, I see that memory grows slowly and then decreases to
the same level, but after the next 100Mb allocation, it is never
released.

Does anyone have an explanation for this?

Thanks In Advance,
Florin

 

[Chris Mullins]

[Out of Memory]

The problem I have encountered lately is that the memory on server
side started to grow and not to be released. I have checked first new
functionalities but after isolating these on separate server, the
memory did not grow.

Sounds like either a classic memory leak or a heap fragmentation issue then.

Are you calling into Win32 or COM on a regular basis?

I have also used .Net Memory profiler to find memory leaks on
server and I could not find something that would explain this grows.

As a quick aside, the SciTech memory profiler is light-years ahead of the
..Net memory profiler. I've done quite a bit of memory analysis for .Net
Server applications, and found this to be by far the best tool.

http://www.scitech.se/memprofiler/

Now, the service on production takes up to 600-800 Mb and then I
receive Out Of Memory exception. There physical Memory is about 2 Gb
and this is dedicated server, so it should be able to allocate more
memory. I suspect that the blame is on the memory fragmentation, but I
cannot be sure.

Have you looked much as the GC performance counters? In which heap is it
saying most of your memory is allocated? Gen2? Large Object?

Other things to think about: Are you running the Workstation or Server CLR?
If you're running the Server CLR (which means you're running on a
Multi-Processor box) you'll see out-of-memory much sooner due to
fragmentation than you would on the workstation version. This is due to the
heaps being divided up for the various processors, so each heap is smaller.

Does anyone have an explanation for this?

It sounds like:
1 - Something is leaking, and it's a classic GC related memory leak. Debug
this with the Scitech tool - run your app, take a snapshot, let it run for a
while longer, take another snapshot. Compare. This will take several hours
and hurt your head.

.... or ...

2 - Many somethings are being pinned in memory causing fragmentation. Debug
this using Son of Strike and WinDebug. The overall problem is discussed
here - https://blogs.msdn.com/yunjin/archive/2004/01/27/63642.aspx

This approach will make you swear and curse, and really, really hurt your
head. Then you'll curse MS for cutting Managed Debugging for Crashdumps out
of VS.Net.

 

[Florin]

Hi Chris,

First of all, thank you for posting you reply. I will answer here to
all your questions:
1) I am not using any calls to Win32 or COM.
2) I have used the same application (http://www.scitech.se/memprofiler/
) as you recommended. I know about it, is very good and we bought it.
It has helped us here to find the STA/MTA problem on server side.
Unfortunately it has a defect: I am able to record the calls to the
server, and then replay them. At high load, the .NET Memory Profiler
crashes along with the application is hosting and I cannot use it on
production for example. This time I used it for a set of
functionalities and I could not find and excessive grow. I saw some
bytes[] grow, but not significant (and I am using some statis fields on
server)... I will check them, but with the dump file (according to the
article you gave me). The test I have done with this application where
on win 2003 server, but on this type on OS is working without problem
(I tested this on production - same application, different behavior on
different OS 2000 vs 2003).
3) The counters, yes, I checked them:
- the heap that grows is Gen2, and not large object
- the Private bytes grows in the same time with .NET Bytes in all
Heaps. This suggests managed memory leak/problem. But I could not find
it..
4) I am running in workstation mode. I was planning to suggest server
mode for tests.. So you say is not a good choice..
5) Regarding the memory leak.. I tried, nothing got my attention... And
is working correctly on 2003, so no leaks here. But I will keep
searching.
6) Regarding WinDbg, I made a dump a while ago. I saw only free object
followed by another allocated object. I will try to read it according
to the article you posted.

Also one thing that I not mentioned: Because of some internal
decisions, we are deploying files that are complied in debug mode. I
understand that GC is behaving differently in this situation. Next
builds, I will try to make it as release and see what this brings,
maybe more problems ...

Thanks for the reply,
Florin

 

[Chris Mullins]

Florin,

I wished I had answers for you. It sounds like a memory leak still, as you
are seeing the Gen2 heap grow without bounds. If you can't debug it with the
SciTech profiler, I'm not sure where to go from here. If you can make the
app crash, the crashdump files will probably be your bes bet - but analyzing
them is hard.

I would open up an MSDN Support incident, (with will cost a few hundred
dollars) and ask them for their help. They'll want a crashdump file (or
two). I've had very positive experiences with the MS folk on a
pay-per-incident basis.

If you're an MSDN Subscriber (Profssional? Universal? Preferred? I can't
keep the new things straight) you are entitled to a few support calls as
part of your subscription.

As to Win32 and Com, sometimes it's not at all obvious. For example, most of
my code is "pure managed code", but I call into the Sockets class. This
causes memory to be pinned while the sockets class in turn calls into Win32
land. My code wasn't doing any Win32 stuff, but under the hood it sure was.

--
Chris Mullins

Hi Chris,

First of all, thank you for posting you reply. I will answer here to
all your questions:
1) I am not using any calls to Win32 or COM.
2) I have used the same application (http://www.scitech.se/memprofiler/
) as you recommended. I know about it, is very good and we bought it.
It has helped us here to find the STA/MTA problem on server side.
Unfortunately it has a defect: I am able to record the calls to the
server, and then replay them. At high load, the .NET Memory Profiler
crashes along with the application is hosting and I cannot use it on
production for example. This time I used it for a set of
functionalities and I could not find and excessive grow. I saw some
bytes[] grow, but not significant (and I am using some statis fields on
server)... I will check them, but with the dump file (according to the
article you gave me). The test I have done with this application where
on win 2003 server, but on this type on OS is working without problem
(I tested this on production - same application, different behavior on
different OS 2000 vs 2003).
3) The counters, yes, I checked them:
- the heap that grows is Gen2, and not large object
- the Private bytes grows in the same time with .NET Bytes in all
Heaps. This suggests managed memory leak/problem. But I could not find
it..
4) I am running in workstation mode. I was planning to suggest server
mode for tests.. So you say is not a good choice..
5) Regarding the memory leak.. I tried, nothing got my attention... And
is working correctly on 2003, so no leaks here. But I will keep
searching.
6) Regarding WinDbg, I made a dump a while ago. I saw only free object
followed by another allocated object. I will try to read it according
to the article you posted.

Also one thing that I not mentioned: Because of some internal
decisions, we are deploying files that are complied in debug mode. I
understand that GC is behaving differently in this situation. Next
builds, I will try to make it as release and see what this brings,
maybe more problems ...

Thanks for the reply,
Florin