Why is OutOfMemoryException thrown?

L

Lamont Sanford

I can't figure out why once or twice per week, my C# server application
throws OutOfMemoryExceptions. There is plenty of memory in the machine. As
far as I can tell, when the exception is thrown, only 400 megs of 1gig total
is in use.

The exception is thrown while trying to update List<>. Here's a bit of the
stack trace:

===
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException'
was thrown.
at System.Collections.Generic.List`1.set_Capacity(Int32 value)
at System.Collections.Generic.List`1.EnsureCapacity(Int32 min)
at System.Collections.Generic.List`1.Add(T item)
at Mosaic.Core.StatUtils.ExtractRORs(MiniPerfItem[] perfItems, Int32
startIdx, Int32 endIdx, Boolean useNAOnNonPosBal) in
C:\Development\Mosaic\Mosaic-AM\Mosaic.Core.Shared\Code\Utilities\Statistical\StatUtils.cs:line
38
===

I have a lot of small objects in memory at the time of the exception but I'm
nowhere close to exhausting available RAM. Any ideas?
 
J

Jon Skeet [C# MVP]

Lamont Sanford said:
I can't figure out why once or twice per week, my C# server application
throws OutOfMemoryExceptions. There is plenty of memory in the machine. As
far as I can tell, when the exception is thrown, only 400 megs of 1gig total
is in use.

The exception is thrown while trying to update List<>. Here's a bit of the
stack trace:

===
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException'
was thrown.
at System.Collections.Generic.List`1.set_Capacity(Int32 value)
at System.Collections.Generic.List`1.EnsureCapacity(Int32 min)
at System.Collections.Generic.List`1.Add(T item)
at Mosaic.Core.StatUtils.ExtractRORs(MiniPerfItem[] perfItems, Int32
startIdx, Int32 endIdx, Boolean useNAOnNonPosBal) in
C:\Development\Mosaic\Mosaic-AM\Mosaic.Core.Shared\Code\Utilities\Statistical\StatUtils.cs:line
38
===

I have a lot of small objects in memory at the time of the exception but I'm
nowhere close to exhausting available RAM. Any ideas?
Click to expand...

Do you have any idea how big the list is at the point of failure?
 
L

Lamont Sanford

Do you have any idea how big the list is at the point of failure?
At max it could get up to the tens of thousands, but certainly never over
100k...
 
G

Guest

Most likely - improperly closed / disposed connection objects means you run
out of Connection Pool connections, which can cause exceptions similar to
what you're seeing - especially if combined with other code.

1) use the same connection string througout your app.
2) open a connection just before you need to do database call.
3) close the connection immediately after - this returns it to the pool
4) be very careful with DataReaders as they hold open a connection.
-- Peter
Recursion: see Recursion
site: http://www.eggheadcafe.com
unBlog: http://petesbloggerama.blogspot.com
BlogMetaFinder: http://www.blogmetafinder.com
 
J

Jon Skeet [C# MVP]

Lamont Sanford said:
At max it could get up to the tens of thousands, but certainly never over
100k...
Click to expand...

That sounds unlikely to be the issue then.

Can I suggest that you use perfmon to look at both the heap and the
handle count of your process?
 
P

Przemek Celej

Lamont Sanford pisze:
I can't figure out why once or twice per week, my C# server application
throws OutOfMemoryExceptions. There is plenty of memory in the machine. As
far as I can tell, when the exception is thrown, only 400 megs of 1gig total
is in use.

The exception is thrown while trying to update List<>. Here's a bit of the
stack trace:

===
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException'
was thrown.
at System.Collections.Generic.List`1.set_Capacity(Int32 value)
at System.Collections.Generic.List`1.EnsureCapacity(Int32 min)
at System.Collections.Generic.List`1.Add(T item)
at Mosaic.Core.StatUtils.ExtractRORs(MiniPerfItem[] perfItems, Int32
startIdx, Int32 endIdx, Boolean useNAOnNonPosBal) in
C:\Development\Mosaic\Mosaic-AM\Mosaic.Core.Shared\Code\Utilities\Statistical\StatUtils.cs:line
38
===

I have a lot of small objects in memory at the time of the exception but I'm
nowhere close to exhausting available RAM. Any ideas?
Click to expand...

Maybe due to extensively allocating and freeing memory, your memory is
too much fragmentized ?

Regards
 
L

Lamont Sanford

That sounds unlikely to be the issue then.

Can I suggest that you use perfmon to look at both the heap and the
handle count of your process?
Click to expand...

Sure, I'll check it out. I've never used permon before. What signs/symptoms
should I be looking for?
 
L

Lamont Sanford

Maybe due to extensively allocating and freeing memory, your memory is
too much fragmentized ?
Click to expand...

How would I go about defragmenting it? Should I manually run a garbage
collection?
 
J

Jon Skeet [C# MVP]

Sure, I'll check it out. I've never used permon before. What signs/symptoms
should I be looking for?
Click to expand...

Handles or heap memory rising constantly would be generally bad - but
you can also add performance traces for various .NET aspects, which
can be handy.

Jon
 
W

Willy Denoyette [MVP]

Lamont Sanford said:
How would I go about defragmenting it? Should I manually run a garbage
collection?
Click to expand...


No, there is nothing you can do about a fragmented heap (both managed and
process heap), other than restart the server.
Of course it's up to you to prevent possible heap fragmentation as much as
possible.
Therefore, I would like to know what version of the Framework and OS you
are running this on?
How large are the Lists in bytes, if you say up to 100k, do you mean 100kb
or something else, how did you measure this size?
How many of these list do you possibly create simultaneously?
Are you using Sockets connections?
Do you explicitly pin objects, or do you call into unmanaged code?

Willy.
 
Z

Zhangming Su

I got the same problem before. Then I rewrited the application by replacing
DataSet by SqlDataReader.
 
N

not_a_commie

Try using LinkedList<> instead of List<> if you can. It won't be as
efficient at stopping memory fragmentation, but it will not call the
EnsureCapacity function, which, in essence, momentarily triples the
amount of memory used so that it can double the amount of memory used.
Okay, so it's not really "memory used" unless you're using structs.
It's memory used for pointers, which on a 32bit system for 100k should
400kilobytes -- not much! (Tripling that should be fine which makes me
wonder if it's a list of structs.)
 
L

Lamont Sanford

Therefore, I would like to know what version of the Framework and OS you
are running this on?
Click to expand...

I'm running .NET Framework 2.0 (VS 2005) on Win2003.
How large are the Lists in bytes, if you say up to 100k, do you mean 100kb
or something else, how did you measure this size?
Click to expand...

I just meant that the size of the list (List<T>.Count) would never exceed
100,000. If type 'T' is a double and double requires 8 bytes, the total
required memory shouldn't exceed 800,000 bytes... and I have hundreds of
gigs free.
How many of these list do you possibly create simultaneously?
Click to expand...

Only one can exist at a time, but these lists will be created over and over
again with different values.When a list is no longer needed, it isn't
explicitly Clear()'d, but it does fall out of scope so it should be GCd.
Are you using Sockets connections?
Click to expand...

Yes, sockets are used. No more than 30 inbound connections are active at any
given time.
Do you explicitly pin objects, or do you call into unmanaged code?
Click to expand...

What does "pin" mean? No, I don't make any calls into unmanaged code.
 
W

Willy Denoyette [MVP]

Lamont Sanford said:
I'm running .NET Framework 2.0 (VS 2005) on Win2003.


I just meant that the size of the list (List<T>.Count) would never exceed
100,000. If type 'T' is a double and double requires 8 bytes, the total
required memory shouldn't exceed 800,000 bytes... and I have hundreds of
gigs free.
Click to expand...

No you don't? The virtual memory space attributed to your process is only
2GB (supposing W2K3 32 bit). That means that you cannot allocate more than
this space for managed objects minus the space allocated by the application
the runtime and the Framework data and code, this leaves you with less than
1.6GB (non contiguous) in the most optimal case at process creation time.
Now, 800 Kb is not a big deal, but as these objects are larger than 85Kb,
they end on the LOH and as this one is never compacted, you may end with a
highly fragmented heap, if you don't tput an eye on object life time.
Only one can exist at a time, but these lists will be created over and
over again with different values.When a list is no longer needed, it isn't
explicitly Clear()'d, but it does fall out of scope so it should be GCd.
Click to expand...
Yes, but that doesn't mean that only one can exist at a time, the GC doesn't
run deterministically.
Yes, sockets are used. No more than 30 inbound connections are active at
any given time.


What does "pin" mean? No, I don't make any calls into unmanaged code.
Click to expand...

Yes, you do (indirectly), there is no single managed application in the
world that don't call in unmanaged code. The socket classes need to pin the
send/reeive buffers when calling into the native Winsock API's. Pinning is
there to prevent the heap allocated object to move in memory when the GC
runs, however, pinning also prevents compactation of the GC heap.
Does it happen that these buffers have something in common with the above
mentioned List's?

Anyway, I would suggest you to inspect your memory allocation pattern by
running your code against a memory profiler.

Willy.
 
L

Lamont Sanford

I just meant that the size of the list (List said:
No you don't?
Click to expand...

Ooops, I meant hundreds of megs, not hundreds of gigs... sorry for the
typeo.

Thanks for the hints. I'll take your advice and hopefully I can get to the
bottom of this.
 
B

bob

OK.
Not what I thought.
I had weird problem with SQL Anywhere Version 9 that appeared as a
memory problem.
Sorry no other ideas.
Bob
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

LoadDataRow + OutOfMemoryException 1
Out-of-memory errors and caching errors. 2
OutOfMemoryException using .NewRow 1
"System.Web.HttpException: External component has thrown an exception" if OnSortCommand attribute is 0

Top