No memory although more than 1GB free

  • Thread starter Thread starter Klaus Bonadt
  • Start date Start date
K

Klaus Bonadt

I have an application, which makes heavily use of stl and memory
consumption.
After a while the application is not be able to allocate memory from the
system anymore. The following functions fail for claiming 0.5MB:
HeapAlloc with current default heap
creating a new heap with CreateHeap
using new operator

I use XP pro with 2GB physical RAM and fixed 1GB page file.
Task Manager provide the following information at this point for Commit
Charge in KBytes:
Total: 1631776
Limit: 2967716

Process Viewer shows for this application:
Working Set: 289036 KB
Heap Usage: 1421896 KB

What's strange is, I am still be able to start another program, which
allocates in a loop chunks of 1MB RAM. This is possible for approximately
further 1.2 GB until the limit 2967716 KB is reached.

Why is it not possible for my application to allocate more memory while
another process is able to do so?

Regards,
Klaus
 
Klaus Bonadt said:
What's strange is, I am still be able to start another program, which
allocates in a loop chunks of 1MB RAM. This is possible for approximately
further 1.2 GB until the limit 2967716 KB is reached.

Why is it not possible for my application to allocate more memory while
another process is able to do so?
Click to expand...

It looks like your application runs out of virtual address space, and/or
virtual memory gets fragmented. You see, you may have plenty of physical
memory, but no free virtual address window to map it to.
--
With best wishes,
Igor Tandetnik

"For every complex problem, there is a solution that is simple, neat,
and wrong." H.L. Mencken
 
It seems that you could have that case. You could try to walk the virtual
address space and see what's going on. Check out VirtualQueryEx().

Nick
 
Why is it not possible for my application to allocate more memory
while

It looks like your application runs out of virtual address space, and/or
virtual memory gets fragmented. You see, you may have plenty of physical
memory, but no free virtual address window to map it to.
Click to expand...

What do you mean with "virtual address space"? If it means "memory", even
defragmented memory, any other application should not get memory in huge
chunks anymore?
Does it mean something like "page handles"? Maybe this number is restricted?

Thanks for clarifying.
Regards,
Klaus
 
It seems that you could have that case. You could try to walk the virtual
address space and see what's going on. Check out VirtualQueryEx().
Click to expand...

DWORD VirtualQuery(
LPCVOID lpAddress, // address of region
PMEMORY_BASIC_INFORMATION lpBuffer, // information buffer
SIZE_T dwLength // size of buffer
);

Two questions:
1.
How to set lpAddress initially?

2.
I will collect sets of memory regions with different attributes. However, I
do not know how to interpret these sets.
Furthermore, I guess I will end up with the same amount of memory, which
"Process viewer" is presenting.

Regards,
Klaus
 
Klaus Bonadt said:
What do you mean with "virtual address space"? If it means "memory", even
defragmented memory, any other application should not get memory in huge
chunks anymore?
Click to expand...

I mean what I said - address space. Each process is allocated a flat 4GB
address space - that's as much as you can address with a 32-bit pointer.
Upper 2GB are reserved for the system (note - this does not mean the
system uses up 2GB of RAM, it just means that you cannot allocate memory
with an address in the upper 2GB). The bottom 2GB are partially occupied
by your executable code and any DLLs it may load. The rest is available
for allocation.

Now, with very large amounts of RAM, it may so happen that the process
runs out of addresses before it runs out of physical memory (in fact, if
you have more than 2GB of RAM you simply cannot address it all as a flat
space). That's the primary motivation for moving to 64bit processors.

Even if you have enough address space, it may be fragmented. That is,
there are many small stretches of unused address space, but none large
enough to accomodate your allocation request.
Does it mean something like "page handles"? Maybe this number is
Click to expand...
restricted?

I don't know what you are talking about.
--
With best wishes,
Igor Tandetnik

"For every complex problem, there is a solution that is simple, neat,
and wrong." H.L. Mencken
 
Klaus Bonadt said:
DWORD VirtualQuery(
LPCVOID lpAddress, // address of region
PMEMORY_BASIC_INFORMATION lpBuffer, // information buffer
SIZE_T dwLength // size of buffer
);

Two questions:
1.
How to set lpAddress initially?
Click to expand...

You can start it at 0 and then increment it based on the info returned in
lpBuffer (i.e. look up MEMORY_BASIC_INFORMATION).
2.
I will collect sets of memory regions with different attributes. However, I
do not know how to interpret these sets.
Furthermore, I guess I will end up with the same amount of memory, which
"Process viewer" is presenting.
Click to expand...

Yes, but this will give you more info than a simple sum. To understand what
you're being given again see MEMORY_BASIC_INFORMATION.

BTW, what failture code to you get from the allocation functions that you
call?

HTH,
Nick
 
One other tool useful for a meaningful dump of the process address space
is the '!address' command in cdb/ntsd/windbg.
If will also try to give the usage of the allocated regions, their size,
flags, etc, etc.
 
I mean what I said - address space. Each process is allocated a flat 4GB
address space - that's as much as you can address with a 32-bit pointer.
Upper 2GB are reserved for the system (note - this does not mean the
system uses up 2GB of RAM, it just means that you cannot allocate memory
with an address in the upper 2GB). The bottom 2GB are partially occupied
by your executable code and any DLLs it may load. The rest is available
for allocation.
Click to expand...

Thus, each process is able to allocate 2GB at maximum. Indeed, I can almost
allocate 2GB with my test program, allocating chunks of 1MB.
However, when my crucial application runs out of memory, Task Manager tells
that there is only 1,6GB in use (see my first mail). Furthermore, starting
my test program at this point in time, this program is able to allocate
further 1,2 GB until my whole virtual memory is allocated. Thus, all other
processes (system and my application) share 3GB - 1,2GB = 1,8GB. One more
indication that my crucial application was still not be able to allocate
nearly 2GB.

My question is, why could my application allocate only 1,4GB (this is what
Process Viewer (Dev Studio6 tools) says) although it should be able to
allocate 2GB.
Now, with very large amounts of RAM, it may so happen that the process
runs out of addresses before it runs out of physical memory (in fact, if
you have more than 2GB of RAM you simply cannot address it all as a flat
space). That's the primary motivation for moving to 64bit processors.
Click to expand...

I have the AMD 64 processor, but I need a special 64-bit XP system, which is
not yet available for AMD, correct?
Even if you have enough address space, it may be fragmented. That is,
there are many small stretches of unused address space, but none large
enough to accomodate your allocation request.
Click to expand...

What are PVIEW and Task Manager showing, the sum of heap sizes (including
unused space due to fragmentation) or the sum of actual allocated memory
(HeapAlloc) which was not freed afterwards?
I guess the first case, which means the sum of memory which is reserved for
the process even the process is not able to allocate due to fragmentation?
But anyway, as I have mentioned above there must be at least 2 - 1,8 = 0,2GB
unfragmented memory available, otherwise my test program could not allocate
1,2GB until the whole virtual memory is occupied.
restricted?
Click to expand...
Maybe there is another limiting resource, for instance the amount of allocs.
I just wrote another test program to clarify this. The test program
allocates only 2Bytes with HeapAlloc() in a loop. It was able to allocate
133,164,202 * 2Bytes = 266,328,404 Bytes, which is 254MB. However, process
viewer shows for this process a heap usage of 2080848KB, which is almost
2GB.
It seems that for every HeapAlloc there are 8 Bytes additional costs in
terms of memory, but the number of allocations
seem to be not limited.

Regards,
Klaus
 
Klaus Bonadt said:
I have the AMD 64 processor, but I need a special 64-bit XP system, which is
not yet available for AMD, correct?
Click to expand...

You can download a preview version for free from the MS site.
But anyway, as I have mentioned above there must be at least 2 - 1,8 = 0,2GB
unfragmented memory available, otherwise my test program could not allocate
1,2GB until the whole virtual memory is occupied.
Click to expand...

Each process has a separate virtual address space so fragmentation in one
would not affect another.
It seems that for every HeapAlloc there are 8 Bytes additional costs in
terms of memory, but the number of allocations
seem to be not limited.
Click to expand...

Of course. The OS needs to do some bookkeeping for each allocation.

Nick
 
In microsoft.public.vc.stl Nick Savoiu said:
You can download a preview version for free from the MS site.


Each process has a separate virtual address space so fragmentation in
one would not affect another.
Click to expand...

It might be a bit off-topic by now, but what is the largest chunk of
'flat memory' I can expect to get on a 32bit Intel system stuffed with
e.g. 3GB of RAM and running not much else than XP?

Andre'
 
Klaus Bonadt said:
Thus, each process is able to allocate 2GB at maximum. Indeed, I can almost
allocate 2GB with my test program, allocating chunks of 1MB.
However, when my crucial application runs out of memory, Task Manager tells
that there is only 1,6GB in use (see my first mail).
Click to expand...

That's why I think there's fragmentation at work. Imagine the
pathological scenario: your whole memory is occupied by 1K allocated
chunk followed by 1023K free chunk, and so on. 1024 such pairs will eat
up 1GB of RAM in a way that only 1MB is actually used, but you cannot
allocate another 1MB chunk. It's an artificial example of course, it's
highly unlikely to occur in practice, but it demonstrates the idea of
fragmentation nicely.
Furthermore, starting
my test program at this point in time, this program is able to allocate
further 1,2 GB until my whole virtual memory is allocated.
Click to expand...

Your test program is a separate process that has its own address space.
Remember - you are not running out of physical RAM, you are running out
of addresses to map it to. Each process has its very own 4GB address
space, independent of any other process. Physical RAM gets mapped to a
process' address space on as-needed basis.

By the way, with Windows virtual memory management, you cannot fragment
physical memory at all, you can only fragment virtual address space. The
way it works, all physical memory is broken into pages 4KB (on some
systems 8KB) large. So is virtual memory. When a region of virual memory
is allocated, each page of virtual memory is backed by a page of
physical memory. Now, virtual addresses within this region need to be
consequtive (because your program expects the address arithmetic to
work), but physical RAM pages don't need to be. The system just picks
free RAM pages lying around and maps them to virtual pages. Moreover, if
the system runs out of RAM, it picks a physical page (mapped to some
virtual page in some process A), saves its contents to disk and reuses
it for another virtual page possibly in a different process B. If
process A later needs to refer to that virtual page, some other physical
page may be assigned to it and contents read back from disk.

Bottom line: there is no permanent attachment of virtual addresses to
physical RAM pages, and there is no requirement that a contiguous
stretch of virtual addresses be backed by a contiguous stretch of
physical addresses. Thus RAM cannot become fragmented, but virtual
address space can.
I have the AMD 64 processor, but I need a special 64-bit XP system, which is
not yet available for AMD, correct?
Click to expand...

Sorry, I don't know.
What are PVIEW and Task Manager showing, the sum of heap sizes (including
unused space due to fragmentation) or the sum of actual allocated memory
(HeapAlloc) which was not freed afterwards?
Click to expand...

I believe they show a working set, which is memory that is a) actually
allocated and b) actually loaded in RAM, rather than being swapped to
hard drive. Well, in your case of huge amounts of RAM available, you can
assume that a working set is very close to actual RAM consumption. They
don't show the unused space, because it is, well, unused.
I guess the first case, which means the sum of memory which is reserved for
the process even the process is not able to allocate due to
Click to expand...
fragmentation?

If this were the case, they would always show 2GB - that's the amount of
address space given to every process. Such a display would be pretty
useless, don't you think?
--
With best wishes,
Igor Tandetnik

"For every complex problem, there is a solution that is simple, neat,
and wrong." H.L. Mencken
 
But anyway, as I have mentioned above there must be at least 2 - 1,8 =
0,2GB

Each process has a separate virtual address space so fragmentation in one
would not affect another.
Click to expand...

In my current situation, my application runs out of memory due to a memory
claim of 0.5MB RAM.
Let us assume this is due to fragmented memory, i.e. there is no contiguous
range with more than 0.5MB RAM in the lower 2 GB of virtual address space.
The application stops with popping a message box.
Now I start another application, which allocates in a loop chunks of memory,
each chunk is 1MB. The application does this up to 1.2 GB RAM. I have 3 GB
RAM in total (fixed 1 GB page size). Thus, there was 3 - 1.2 = 1.8GB for my
crucial application plus operating system.
Even the operating system does not need any memory, there are at least 2GB -
1.8GB = 0.2GB in the lower address space available with memory chunks
greater or equal 1MB (otherwise the test program would not be able to
allocate in this area 1MB chunks)!

Excuse me for my questions, but I still do not understand what is going on.
Of course. The OS needs to do some bookkeeping for each allocation.
Click to expand...

OK, but 133,164,202 allocations is probably more than enough, it is
sufficient for allocating the maximum number of chunks with 2Byte size.
Only when freeing memory does not result in a free'd entry in the book, this
could be a limited resource. I could try to simulate memory fragmentation
with my test program.

Thanks for your patience with me!
Regards,
Klaus
 
Klaus Bonadt said:
Each process has a separate virtual address space so fragmentation in one
would not affect another.
Click to expand...

In my current situation, my application runs out of memory due to a memory
claim of 0.5MB RAM. [...]
Even the operating system does not need any memory, there are at least 2GB -
1.8GB = 0.2GB in the lower address space available with memory chunks
greater or equal 1MB (otherwise the test program would not be able to
allocate in this area 1MB chunks)!
Excuse me for my questions, but I still do not understand what is going
Click to expand...
on.

Not a problem. We're all here to hopefully learn something. Try re-reading
Igor's posting again. He explains quite well what goes on with virtual/real
memory and address spaces. If that doesn't clear things up I'd suggest
picking up an OS design book and reading up on virtual memory and such.
I could try to simulate memory fragmentation with my test program.
Click to expand...

This might be not be easy to do as you do not directly control the virtual
addresses at which you make allocations. So the OS could be playing tricks
behind your back.

Nick
 
In microsoft.public.vc.stl Igor Tandetnik said:
That's why I think there's fragmentation at work. Imagine the
pathological scenario: your whole memory is occupied by 1K allocated
chunk followed by 1023K free chunk, and so on. 1024 such pairs will eat
up 1GB of RAM in a way that only 1MB is actually used, but you cannot
allocate another 1MB chunk. It's an artificial example of course, it's
highly unlikely to occur in practice, but it demonstrates the idea of
fragmentation nicely.
Click to expand...

Does that mean the allocator does _not_ try to collect chunks of 'almost
the same size' in one place and different sizes in another one? [I am
talking about virtual adresses here]

Andre'
 
By the way, with Windows virtual memory management, you cannot fragment
physical memory at all, you can only fragment virtual address space. The
way it works, all physical memory is broken into pages 4KB (on some
systems 8KB) large. So is virtual memory. When a region of virual memory
is allocated, each page of virtual memory is backed by a page of
physical memory. Now, virtual addresses within this region need to be
consequtive (because your program expects the address arithmetic to
work), but physical RAM pages don't need to be. The system just picks
free RAM pages lying around and maps them to virtual pages. Moreover, if
the system runs out of RAM, it picks a physical page (mapped to some
virtual page in some process A), saves its contents to disk and reuses
it for another virtual page possibly in a different process B. If
process A later needs to refer to that virtual page, some other physical
page may be assigned to it and contents read back from disk.
Click to expand...

Thanks a lot, Igor, I got it!
 
Klaus Bonadt said:
In my current situation, my application runs out of memory due to a memory
claim of 0.5MB RAM.
Let us assume this is due to fragmented memory, i.e. there is no contiguous
range with more than 0.5MB RAM in the lower 2 GB of virtual address space.
The application stops with popping a message box.
Now I start another application, which allocates in a loop chunks of memory,
each chunk is 1MB. The application does this up to 1.2 GB RAM.
Click to expand...

Which part of "each process has a separate virtual address space" do you
have difficulty understanding? A success or failure of a memory
allocation in one process tells you nothing about the state of the
virtual address space in another. It's like saying: "I have machine A
where memory allocation fails. Then I run a test program on machine B,
and it can successfully allocate plenty of memory. So enough memory
should be avaiable on machine A, right?"
--
With best wishes,
Igor Tandetnik

"For every complex problem, there is a solution that is simple, neat,
and wrong." H.L. Mencken
 
Does that mean the allocator does _not_ try to collect chunks of 'almost
the same size' in one place and different sizes in another one? [I am
talking about virtual adresses here]
Click to expand...

At least it would be worthwile in order to reduce fragmentation to think
about creating different heaps, each responding for allocating chunks with
similar size?

If I create two growable heaps, how the system does position these heaps in
the virtual address space? If the second heap starts at the end of the first
heap, the first heap is actually not growable?
On the other hand, the space between heaps will also cause fragmentation.
Any ideas for balancing?

Regards,
Klaus
 
Back
Top