Fatal out of memory error

G

Guest

I've written a .NET application that typically uses about 500MB of memory. After running for anywhere from 20 minutes to a few hours, the application crashes with "Fatal out of memory error" printed on the console. The CLR does not appear to raise the OutOfMemoryException before the application crashes. The machines running this application all have 1GB physical memory. Since the application simply quits (even while being debugged) I am unsure of how to best debug the problem. A quick search of the newsgroups has turned up little help. Any suggestions would be appreciated. Thank you

Regards

- Ben Blai
{My Name} {at} acm.org
 
M

Michael Giagnocavo [MVP]

Well, are you using more memory than the machine has? :) I'm only asking
because you said typically uses 500MB, leading me to think that there could
be a bug in the code that makes the memory usage spiral up and boom.

This was just posted in another newsgroup (OutOfMemoryExceptions on V1.1 of
the framework). Maybe your code is experiencing it?

MORE INFORMATION
The problem is not limited to the RichTextBox control. The problem may also
occur with certain large memory allocations. For example, the following code
also results in an "Out of memory" exception error message:
byte[] arr = new byte[16773053];
To work around this problem, add a single byte to this allocation.

http://support.microsoft.com/?kbid=826981

-mike
MVP
 
T

Tian Min Huang

Hello Ben,

Thanks for your post. As I understand, the problem you are facaing is that
your application crashes with "Tatal out of memory error" sometimes. I'd
like to share the following information with you:

1. How much memory is your application consuming when it crashes? Based on
my experience, when a process reaches a memory consumption of about 800MB,
it is possible for it to begin to experience the OutOfMemory problem due to
virtual memory fragmentation. Virtual memory fragmentation is a condition
where virtual memory is available for a process, but none of the virtual
memory blocks that are available are of a significant size. Memory
fragmentation occurs over time because of the varying size of memory
allocations and the varying lifetimes of each allocation.

2. In addition, I suggest that you can use ADPlus to create a mini dump
file and then debug it:

HOW TO: Use ADPlus to Troubleshoot "Hangs" and "Crashes"
http://support.microsoft.com/default.aspx?scid=kb;en-us;286350

Hope this helps.

Regards,

HuangTM
Microsoft Online Partner Support
MCSE/MCSD

Get Secure! -- www.microsoft.com/security
This posting is provided "as is" with no warranties and confers no rights.
 
G

Guest

HuangTM

Thanks for your reply. Memory fragmentation seems like a plausible explanation. To be clear, though, I don't get .Net OutOfMemoryException exceptions thrown. Rather, the application simply terminates and prints "Fatal out of memory error" to the console. I was not aware of ADPlus, but it looks like just what I need to help track down the source of the problem. As I said in my response to Mike's post (above), I have tried freeing about 200 MB worth of objects that are not as necessary as the others. So far, this seems to have worked, but it will be a few days before I can be sure. If it doesn't work, I will try using ADPlus, to track down the problem more directly

If it does turn out to be a memory fragmentation problem, can you suggest a way to fix it? The objects I allocate are not huge (<1K) and most of that does not need to be contiguous. I would think that if there wasn't enough physical memory available to handle an allocation request, the memory manager would simply write a few more pages to disk to free some space. Since the program crashes when the total memory being used by the system is less than 600 MB of the 1GB physical memory available, it would be surprising if the memory manager couldn't find some contiguous memory.

Anyway, thanks for the response. It's good to know about the ADPlus debuging script

Regards

- Ben Blai
{my name} {at} acm.or

----- Tian Min Huang wrote: ----

Hello Ben

Thanks for your post. As I understand, the problem you are facaing is that
your application crashes with "Tatal out of memory error" sometimes. I'd
like to share the following information with you

1. How much memory is your application consuming when it crashes? Based on
my experience, when a process reaches a memory consumption of about 800MB,
it is possible for it to begin to experience the OutOfMemory problem due to
virtual memory fragmentation. Virtual memory fragmentation is a condition
where virtual memory is available for a process, but none of the virtual
memory blocks that are available are of a significant size. Memory
fragmentation occurs over time because of the varying size of memory
allocations and the varying lifetimes of each allocation

2. In addition, I suggest that you can use ADPlus to create a mini dump
file and then debug it

HOW TO: Use ADPlus to Troubleshoot "Hangs" and "Crashes
http://support.microsoft.com/default.aspx?scid=kb;en-us;28635

Hope this helps

Regards

HuangT
Microsoft Online Partner Suppor
MCSE/MCS

Get Secure! -- www.microsoft.com/securit
This posting is provided "as is" with no warranties and confers no rights
 
G

Guest

Mike

After changing my program to allocate fewer objects, the probelm seems to be resolved. However, when I add some functionality that we will need in the near future and thereby increase the memory usage, I receive OutOfMemoryException exceptions after reaching about 400 MB of VM Memory usage by my process. I took a look at the KB article you referred to and I may be having a similar problem. As a test, I wrote the following console application

public static void Main(string [] args

tr

byte[] bytes = new byte[300000000]
//byte[] bytes = new byte[218099999]
Console.ReadLine()

catch(Exception ex

Console.WriteLine(ex)



If run as written, the application throws an OutOfMemoryException. However, if the commented out line is commented in and the previous line is commented out (thereby reducing the memory allocation), I get an OutOfMemoryException. Adding one (as the article suggests) to the allocation amount does not fix the problem. If I were simply getting errors when allocating byte arrays, I could change them until I found a number that worked, but my OutOfMemoryException errors are coming from framework code (Dictionary.Add and SqlConnection.CheckForDroppedConnection, for example). Any idea if the hotfix mentioned in the article will fix my problems? Thanks

Regards

- Ben Blai
{my name} {at} acm.or

----- Michael Giagnocavo [MVP] wrote: ----

Well, are you using more memory than the machine has? :) I'm only asking
because you said typically uses 500MB, leading me to think that there could
be a bug in the code that makes the memory usage spiral up and boom

This was just posted in another newsgroup (OutOfMemoryExceptions on V1.1 of
the framework). Maybe your code is experiencing it

MORE INFORMATIO
The problem is not limited to the RichTextBox control. The problem may also
occur with certain large memory allocations. For example, the following code
also results in an "Out of memory" exception error message
byte[] arr = new byte[16773053]
To work around this problem, add a single byte to this allocation

http://support.microsoft.com/?kbid=82698

-mik
MV
 
M

Michael Giagnocavo [MVP]

Very odd. I can repro it on v1.1 but not 2.0. I have a P4c @ 3GHz and 1.5GB
of RAM. I'll look into it.

You mention you allocate a lot of small objects. How are they dying? If you
introduce a GC.Collect(2) every now and then, do you still run into the
error?

Thanks,
-mike
MVP

Ben Blair said:
Mike,

After changing my program to allocate fewer objects, the probelm seems to
be resolved. However, when I add some functionality that we will need in
the near future and thereby increase the memory usage, I receive
OutOfMemoryException exceptions after reaching about 400 MB of VM Memory
usage by my process. I took a look at the KB article you referred to and I
may be having a similar problem. As a test, I wrote the following console
application:

public static void Main(string [] args)
{
try
{
byte[] bytes = new byte[300000000];
//byte[] bytes = new byte[218099999];
Console.ReadLine();
}
catch(Exception ex)
{
Console.WriteLine(ex);
}
}

If run as written, the application throws an OutOfMemoryException.
However, if the commented out line is commented in and the previous line
is commented out (thereby reducing the memory allocation), I get an
OutOfMemoryException. Adding one (as the article suggests) to the
allocation amount does not fix the problem. If I were simply getting
errors when allocating byte arrays, I could change them until I found a
number that worked, but my OutOfMemoryException errors are coming from
framework code (Dictionary.Add and
SqlConnection.CheckForDroppedConnection, for example). Any idea if the
hotfix mentioned in the article will fix my problems? Thanks!

Regards,

- Ben Blair
{my name} {at} acm.org

----- Michael Giagnocavo [MVP] wrote: -----

Well, are you using more memory than the machine has? :) I'm only
asking
because you said typically uses 500MB, leading me to think that there
could
be a bug in the code that makes the memory usage spiral up and boom.

This was just posted in another newsgroup (OutOfMemoryExceptions on
V1.1 of
the framework). Maybe your code is experiencing it?

MORE INFORMATION
The problem is not limited to the RichTextBox control. The problem may
also
occur with certain large memory allocations. For example, the
following code
also results in an "Out of memory" exception error message:
byte[] arr = new byte[16773053];
To work around this problem, add a single byte to this allocation.

http://support.microsoft.com/?kbid=826981

-mike
MVP

Ben Blair said:
I've written a .NET application that typically uses about 500MB of memory.
After running for anywhere from 20 minutes to a few hours, the application
crashes with "Fatal out of memory error" printed on the console. The CLR
does not appear to raise the OutOfMemoryException before the application
crashes. The machines running this application all have 1GB physical
memory. Since the application simply quits (even while being debugged) I
am unsure of how to best debug the problem. A quick search of the
newsgroups has turned up little help. Any suggestions would be
appreciated. Thank you.
{My Name} {at} acm.org
 
T

Tian Min Huang

Hello Ben,
suggest a way to fix it?

If you are running .NET Framework 1.1 on some Windows systems, say, Windows
2000 Advanced Server or Windows 2003 Server, you can enable the /3GB switch
(by appending the /3GB switch to the desired entry in your system's
BOOT.INI file) on the OS level to increase the VM space by 1GB. And
subsequently, the OOM should be much less frequent and a process should be
able to run stably at 1.4GB to 1.6 GB. .NET Framework 1.0 does not support
the /3GB switch.

Hope this helps.

Regards,

HuangTM
Microsoft Online Partner Support
MCSE/MCSD

Get Secure! -- www.microsoft.com/security
This posting is provided "as is" with no warranties and confers no rights.
 
G

Guest

Mike

Yea, I thought that was odd too. Thanks for looking into it

I allocate lots of (relatively) small objects, but the vast majority of them do not and cannot die. My application is an option trading system that watches about 2500 stocks and 500,000+ options in realtime. Probably the best way to cut my memory use is to shave a bit off of each of the options, which I'm trying to do now. But it's frustrating that I've got a box with 1GB of physical ram and .NET says I'm out of memory after using less than two thirds of that. Any further sugestions you have would be greatly appreciated

Thanks

- Ben Blai
{my name} {at} acm.or

----- Michael Giagnocavo [MVP] wrote: ----

Very odd. I can repro it on v1.1 but not 2.0. I have a P4c @ 3GHz and 1.5GB
of RAM. I'll look into it

You mention you allocate a lot of small objects. How are they dying? If you
introduce a GC.Collect(2) every now and then, do you still run into the
error

Thanks
-mik
MV

Ben Blair said:
Mike
After changing my program to allocate fewer objects, the probelm seems to
be resolved. However, when I add some functionality that we will need in
the near future and thereby increase the memory usage, I receive
OutOfMemoryException exceptions after reaching about 400 MB of VM Memory
usage by my process. I took a look at the KB article you referred to and I
may be having a similar problem. As a test, I wrote the following console
application
public static void Main(string [] args

tr

byte[] bytes = new byte[300000000]
//byte[] bytes = new byte[218099999]
Console.ReadLine()

catch(Exception ex

Console.WriteLine(ex)

If run as written, the application throws an OutOfMemoryException.
However, if the commented out line is commented in and the previous line
is commented out (thereby reducing the memory allocation), I get an
OutOfMemoryException. Adding one (as the article suggests) to the
allocation amount does not fix the problem. If I were simply getting
errors when allocating byte arrays, I could change them until I found a
number that worked, but my OutOfMemoryException errors are coming from
framework code (Dictionary.Add and
SqlConnection.CheckForDroppedConnection, for example). Any idea if the
hotfix mentioned in the article will fix my problems? Thanks
Regards
- Ben Blai {my name} {at} acm.or
----- Michael Giagnocavo [MVP] wrote: ----
Well, are you using more memory than the machine has? :) I'm only
askin
because you said typically uses 500MB, leading me to think that there
coul
be a bug in the code that makes the memory usage spiral up and boom
This was just posted in another newsgroup (OutOfMemoryExceptions on
V1.1 o
the framework). Maybe your code is experiencing it
MORE INFORMATIO
The problem is not limited to the RichTextBox control. The problem may
als
occur with certain large memory allocations. For example, the
following cod
also results in an "Out of memory" exception error message
byte[] arr = new byte[16773053]
To work around this problem, add a single byte to this allocation
http://support.microsoft.com/?kbid=82698
-mik MV
I've written a .NET application that typically uses about 500MB of memory
After running for anywhere from 20 minutes to a few hours, the applicatio
crashes with "Fatal out of memory error" printed on the console. The CLR
does not appear to raise the OutOfMemoryException before the application
crashes. The machines running this application all have 1GB physical
memory. Since the application simply quits (even while being debugged) I
am unsure of how to best debug the problem. A quick search of the
newsgroups has turned up little help. Any suggestions would be
appreciated. Thank you.
{My Name} {at} acm.org
 
W

Willy Denoyette [MVP]

Ben Blair said:
Mike,

Yea, I thought that was odd too. Thanks for looking into it.

I allocate lots of (relatively) small objects, but the vast majority of
them do not and cannot die. My application is an option trading system
that watches about 2500 stocks and 500,000+ options in realtime. Probably
the best way to cut my memory use is to shave a bit off of each of the
options, which I'm trying to do now. But it's frustrating that I've got a
box with 1GB of physical ram and .NET says I'm out of memory after using
less than two thirds of that. Any further sugestions you have would be
greatly appreciated.


1GB of RAM, but how large is the Virtual memory (page file size + RAM)?
Let's suppose the Pagefile is 1GB too, you should be able to allocate 1.6 GB
for your application, but your application fails with far less, so I guess
you have a fragmentation problem, or you are pinning a lot of objects so
that the GC heap becomes fragmented and cannot be compacted. Note that
pinning occurs automatically when using PInvoke or when you call into
unmanaged code using COM interop. The only way to investigate this is by
attaching a debugger (native code) and load sos.dll, now you should take
some snapshots of the GC heap and see if there are no pinned objects that
remain allocated (creating sandbars).

Willy.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Top