How to use a huge memory in C#?

[fAnSKyer/C# newbie]

My system has 4GB memory and My program in C# is really memory
consuming. and I noticed that when the memory I used is more than 2GB I
will get an exception.

How to solve this? thanks a lot

 

[Greg Young]

That is a windows issue not a C# issue ..There is a 2gb process limit in
windows ..

see
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnpag/html/scalenetchapt17.asp
search for /3 gb for instructions for how to make this larger.

Cheers,

Greg Young
MVP - C#
http://codebetter.com/blogs/gregyoung

 

[Willy Denoyette [MVP]]

| My system has 4GB memory and My program in C# is really memory
| consuming. and I noticed that when the memory I used is more than 2GB I
| will get an exception.
|
| How to solve this? thanks a lot
|

Why does your program uses that much memory? Is it by design or are you
chasing a bug?
If it's the first, you'll have to change your design or move to a 64 bit OS,
if it's a bug, you will have to profile your application.

Willy.

 

[Chris Mullins]

I'm pretty sure .Net doesn't respects the /3gb switch.

I haven't looked in a while, but I know we did quite a bit of performance
tuning in 32-bit land a while back and were very crabby about this.

Fortunatly 64-bit hardware came along, and life was good. I love allocating
20+gb of memory!

 

[Chris Mullins]

You're not hitting .Net limits, you're hitting 32-bit Windows Process
limits. In 32-bit land, you're pretty much stuck.

You're going to need to go buy 64-bit hardware, install a 64-bit O/S (XP,
Win2k3, Vista, Longhorn), and run your application as a 64-bit app.

 

[Willy Denoyette [MVP]]

Sure it does, .NET (the CLR) is LARGEADRESSAWARE since v1.1. The only
problem is that the C# compiler doesn't have an option to set the
largeaddressaware flag, you have to edit the PE header using editbin.

editbin /largeaddressaware <some.exe>


Another option (framework V2)is to compile all modules as netmodules using
the /t:module compiler flag and use the linker (link.exe) to build the PE
file.
here is a sample of such linker command...

link /entry:someNamespace.someClass.Main /subsystem:console
/largeaddressaware some.netmodule someother.netmodule

in both cases you should see something like following when running dumpbin

File Type: EXECUTABLE IMAGE

FILE HEADER VALUES
14C machine (x86)
3 number of sections
452C1EAA time date stamp Wed Oct 11 00:28:58 2006
0 file pointer to symbol table
0 number of symbols
E0 size of optional header
122 characteristics
Executable
Application can handle large (>2GB) addresses
32 bit word machine
Willy.

| I'm pretty sure .Net doesn't respects the /3gb switch.
|
| I haven't looked in a while, but I know we did quite a bit of performance
| tuning in 32-bit land a while back and were very crabby about this.
|
| Fortunatly 64-bit hardware came along, and life was good. I love
allocating
| 20+gb of memory!
|
| --
| Chris Mullins, MCSD.NET, MCPD:Enterprise
| http://www.coversant.net/blogs/cmullins
|
|
|
| | > That is a windows issue not a C# issue ..There is a 2gb process limit in
| > windows ..
| >
| > see
| >
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnpag/html/scalenetchapt17.asp
| > search for /3 gb for instructions for how to make this larger.
| >
| > Cheers,
| >
| > Greg Young
| > MVP - C#
| > http://codebetter.com/blogs/gregyoung
| >
| > | >> My system has 4GB memory and My program in C# is really memory
| >> consuming. and I noticed that when the memory I used is more than 2GB I
| >> will get an exception.
| >>
| >> How to solve this? thanks a lot
| >>
| >
| >
|
|

 

[Chris Mullins]

Sure enough. A quick Google Search agrees with you.

I wonder if the last time I tried was v1.0? I don't think it's been that
long.

I do remember at that time (we were doing extensive profiling), we wanted to
host our Server using the Server CLR, not the Workstation CLR. I needed to
write a C++ hosting exe to do this. If I remember right, that means it must
have been 1.0, since in 1.1 CLR selection was a config file entry.

Ah well. I'm probably just confused. We've been 64-bit for a while now, so
this just doesn't come up for me anymore. Anytime a customer asks "how big
can it scale?" we just give out the 64-bit numbers.

 

[fAnSKyer/C# newbie]

Thanks a lot for give me great help.
I will try the /3GB to see if it solve the problem

But I am quite curious that a 32-bit can search a 4GB memory, why the
bottleneck is 2GB?

This is not a bug, but I am wonder if I can do it better because I
write 2 memory consuming thread in one program. I may separate them to
two program and using .NET remoting to control these two to solve this?

Thanks again
fAnSKyer

 

[Chris Mullins]

That's just the way Win32 works. This is why the move to 64 bit computing
has been so welcomed by database providers and other server application
authors.

For more detail:
http://msdn.microsoft.com/msdnmag/issues/0700/hood/
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dngenlib/html/msdn_virtmm.asp

 

[Greg Young]

This is a prime example of why multiple process is preferred in many cases
over multi-thread

Cheers,

Greg

 

[Willy Denoyette [MVP]]

| Thanks a lot for give me great help.
| I will try the /3GB to see if it solve the problem
|
| But I am quite curious that a 32-bit can search a 4GB memory, why the
| bottleneck is 2GB?
|
| This is not a bug, but I am wonder if I can do it better because I
| write 2 memory consuming thread in one program. I may separate them to
| two program and using .NET remoting to control these two to solve this?
|

You won't solve anything, first because remoting is not meant to 'share'
memory space, second you will introduce something real bad that is "paging".
The reason is simple, you have 4GB of physical memory in your system, great
but this 4GB isn't available to the OS because some part of it is 'stolen'
by the HW to map things like motherborad and adapter BIOS and more
importantly graphics adapter memory.
So let's pretend you keep 3.5 GB available to the SW, this has to be shared
between all running processes and the OS. Say you have two processes that
each try to commit 2GB of memory, that make 4GB and add to this the other
processes comitted memory space (more precisely their working sets), but you
simply don't have the physical memory available you and you will find
yourself in a painfull world of heavy paging.
The only solution is to move to a 64 bit environment, that is 64 bit HW, 64
bit OS and MORE than 4GB of RAM installed. But may I ask you again, why do
you need this huge amount of memory, what kind of application is this?

Willy.

 

[Willy Denoyette [MVP]]

| Sure enough. A quick Google Search agrees with you.
|
| I wonder if the last time I tried was v1.0? I don't think it's been that
| long.
|
Could be, but it could be something else....

| I do remember at that time (we were doing extensive profiling), we wanted
to
| host our Server using the Server CLR, not the Workstation CLR. I needed to
| write a C++ hosting exe to do this. If I remember right, that means it
must
| have been 1.0, since in 1.1 CLR selection was a config file entry.
|
Keep in mind that the /4GT RAM tuning introduces another serious issue, that
is OS resource exhaustion, most of the time it doesn't solve anything, this
trick was invented by MSFT specifically to give MSQL and Exchange some more
headroom.
Also, keep i mind is that this extra GB is separated from the lower 2GB by
the loaded modules (DLL's) and an additional 64KB gap. That means that you
don't get a contigious memory block of 3GB to allocate, so if you think you
can now create a 2GB byte array, you will soon find out that you can't.
Fragmentation bytes again ;-)

| Ah well. I'm probably just confused. We've been 64-bit for a while now, so
| this just doesn't come up for me anymore. Anytime a customer asks "how big
| can it scale?" we just give out the 64-bit numbers.

Absolutely right, whenever your application needs more than 2GB, move to 64
bit.

Willy.

 

[fAnSKyer/C# newbie]

Thanks for your really helpful advice.
the program is a speech search engine and really memory consuming, But
I think I can optimize it to decrese the useage of memory.
using remote may work because it used about 1.5GB memory, so plus it is
3.0GB : )

However, I got in some problems

1) I try to use /3GB but it doesn't work, I modified my boot.ini file
as follows
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows Server 2003,
Standard" /noexecute=optout /fastdetect /3GB

2) if there is anyway to upgrade the system to 64bit? I am not sure if
a 32bit CPU gan run a 64bit System


Thanks a lot

 

[Willy Denoyette [MVP]]

| Thanks for your really helpful advice.
| the program is a speech search engine and really memory consuming, But
| I think I can optimize it to decrese the useage of memory.
| using remote may work because it used about 1.5GB memory, so plus it is
| 3.0GB : )
|

This is a rather simplistic assumption, you forget the OS memory
requirements, neither do I see how you could split your memory needs over
two processes.

| However, I got in some problems
|
| 1) I try to use /3GB but it doesn't work, I modified my boot.ini file
| as follows
| multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows Server 2003,
| Standard" /noexecute=optout /fastdetect /3GB

This is not the only requirement, you also have to enable your program to be
"largeaddressaware", see one of my previous answers.
But again this is not going to help you if your requirement is to get a
larger free block of memory, all you get is an extra block, but if you try
to allocate a large array of say >1.5 GB it will fail even with /3GB and
largeaware enabled.
Honestly, you need to redesign or switch to 64 bit.


|
| 2) if there is anyway to upgrade the system to 64bit? I am not sure if
| a 32bit CPU gan run a 64bit System
|
Nope, you need 64 bit hardware and OS.

Willy.