large (>2G) addresses

[Sergei]

Dear staff
Can I get your assistance with \3GB (LARGEADDRESSAWARE) switch in mixed mode
process built by VS 2008, please?

I have a mixed mode application: C# GUI calling native C++ DLL through
managed C++ wrapper. And I want to give the native C++ code access to large
(>2G) addresses; but (if it’s possible) I do not want the managed code to use
this extra GB of virtual memory.

Questions are:
1. Do I need to alter the header of the mixed mode C# executable to
activate large (>2G) addresses in native C++ DLL which is built using
LARGEADDRESSAWARE?
2. And if I didn’t change the executable header would the managed code still
be confined to 2GB space?
3. Does same logic applied to larger (>4G) memory allocation in win32 mixed
mode process?
4. Is there any information (guide) on how mixed mode process handles large
virtual memory?

Your help is appreciated. Sergei

 

[Jeroen Mostert]

Dear staff

There's no staff here, just a bunch of volunteers. (Well, some of the
volunteers *are* Microsoft staff, and some might not even be volunteers, but
that's neither here nor there.)

Can I get your assistance with \3GB (LARGEADDRESSAWARE) switch in mixed mode
process built by VS 2008, please?

And you don't need to ask to ask.

I have a mixed mode application: C# GUI calling native C++ DLL through
managed C++ wrapper. And I want to give the native C++ code access to large
(>2G) addresses; but (if it’s possible) I do not want the managed code to use
this extra GB of virtual memory.

Why not? If anything the *unmanaged* code might have problems with large
addresses. The managed code should be fine. The CLR has no problem with
large addresses.

Questions are:
1. Do I need to alter the header of the mixed mode C# executable to
activate large (>2G) addresses in native C++ DLL which is built using
LARGEADDRESSAWARE?

Yes. In fact, /LARGEADDRESSAWARE has no effect on DLLs other than possibly
to signal to developers that the DLL has been vetted for large address
correctness. Whether or not large addresses are used depends entirely on the
hosting executable. This is really the only scenario that makes sense; a
loaded DLL becomes part of the application's address space, so it has to
handle the same addresses.

2. And if I didn’t change the executable header would the managed code still
be confined to 2GB space?

Yes, as would any other code running in the same address space, managed or not.

3. Does same logic applied to larger (>4G) memory allocation in win32 mixed
mode process?

Your question is ambiguous.

4G *addresses* are only possible for 64-bit applications. Those are

automatically all "large address aware", since you cannot mix 32-bit and
64-bit code. On a 32-bit system, you cannot have addresses >3G in a 32-bit
application; on a 64-bit system this goes up to 4G. A 32-bit application can
never have >4G addresses simply because that's as much as 32 bits will hold.

Using >4G *memory* is possible for 32-bit applications if they use clumsy
and non-transparent mechanisms like AWE to access the extra memory (since
they don't have >4G addresses, they can't access all the memory at once),
and through some creative uses of memory-mapped files. For 64-bit
applications the memory is allocated as usual, and there's nothing special
about allocating >4G.

Finally, a single *allocation* >4G could only be done by a 64-bit process.

4. Is there any information (guide) on how mixed mode process handles large
virtual memory?

It's really not much different from how unmanaged processes handle it, with
the provision that the managed part should take care of itself, as the CLR
is large address aware (even though managed applications are not marked as
such by default).

 

[Sergei]

Thank you, Jeroen; it was nice of you to walk through all my questions. And
indeed, I had same assumptions with regard to LARGEADDRESSAWARE in DLL and
calling process. But there is a contradicting observation.

I have a large address aware C++ DLL (Enable Large Addresses is ON) and C#
winform executable whose header is not modified to access large addresses

2GB and <3GB (dumpbin \headers doesn’t say “Application can handle large

addressesâ€). The following code running inside of the DLL allows me to query
and to allocate virtual memory at > 2GB and < 3GB addresses:

MEMORY_BASIC_INFORMATION memory_info;
memory_info.BaseAddress = NULL;
while (VirtualQuery (memory_info.BaseAddress, &memory_info, sizeof
(memory_info)))
{
// Region is free, and it can be well aligned and big enough: we are done
if (memory_info.State == MEM_FREE )
BYTE * m_pDIB = (BYTE*)VirtualAlloc(
memory_info.BaseAddress // system selects address
, lRegionSize // page size, in bytes
, MEM_COMMIT // allocate a committed page
, PAGE_READWRITE); // read/write access

// Recompute BaseAddress
memory_info.BaseAddress = (char *) memory_info.BaseAddress +
memory_info.RegionSize;
}

And that is how I manage memory for large arrays.

I want to know will managed code be able to fragment the large addresses and
is AWE is a better performance wise approach handling large memory in win32
application.

Cheers Sergei

 

[Willy Denoyette [MVP]]

Thank you, Jeroen; it was nice of you to walk through all my questions.
And
indeed, I had same assumptions with regard to LARGEADDRESSAWARE in DLL and
calling process. But there is a contradicting observation.

I have a large address aware C++ DLL (Enable Large Addresses is ON) and C#
winform executable whose header is not modified to access large addresses
addressesâ€). The following code running inside of the DLL allows me to
query
and to allocate virtual memory at > 2GB and < 3GB addresses:

MEMORY_BASIC_INFORMATION memory_info;
memory_info.BaseAddress = NULL;
while (VirtualQuery (memory_info.BaseAddress, &memory_info, sizeof
(memory_info)))
{
// Region is free, and it can be well aligned and big enough: we are done
if (memory_info.State == MEM_FREE )
BYTE * m_pDIB = (BYTE*)VirtualAlloc(
memory_info.BaseAddress // system selects address
, lRegionSize // page size, in bytes
, MEM_COMMIT // allocate a committed page
, PAGE_READWRITE); // read/write access

// Recompute BaseAddress
memory_info.BaseAddress = (char *) memory_info.BaseAddress +
memory_info.RegionSize;
}

What makes you think the above code allocates memory from the range >2GB -
<3G?
It doesn't, just try to allocate from 0x80000000, you'll see VirtualQuery
will return 0, which means that you are allocating from kernel space!

You can't allocate memory from the virtual address range above 2GB in 32
bit mode without LARGEADDREASSAWARE, the OS loader uses this bit from the
executable image when he reserves the address space for the process, setting
this bit in a DLL makes no sense.
That means that you enable this for the whole process, which also means that
the CLR can allocate from the extended area, unless you pre-allocate from
0x80000000 early in the process.
Note also that you should have very good reasons to enable this on system
running desktop applications, most probably you will run into issues because
the memory space for the system is now limited to 1GB, LARGEADDRESSAWARE is
something that only works well for SQL and Exchange servers, provided they
run on dedicated servers.


Willy.




Willy.

 

[Jeroen Mostert]

Thank you, Jeroen; it was nice of you to walk through all my questions. And
indeed, I had same assumptions with regard to LARGEADDRESSAWARE in DLL and
calling process. But there is a contradicting observation.

I have a large address aware C++ DLL (Enable Large Addresses is ON) and C#
winform executable whose header is not modified to access large addresses
addressesâ€). The following code running inside of the DLL allows me to query
and to allocate virtual memory at > 2GB and < 3GB addresses:

MEMORY_BASIC_INFORMATION memory_info;
memory_info.BaseAddress = NULL;
while (VirtualQuery (memory_info.BaseAddress, &memory_info, sizeof
(memory_info)))
{
// Region is free, and it can be well aligned and big enough: we are done
if (memory_info.State == MEM_FREE )
BYTE * m_pDIB = (BYTE*)VirtualAlloc(
memory_info.BaseAddress // system selects address
, lRegionSize // page size, in bytes
, MEM_COMMIT // allocate a committed page
, PAGE_READWRITE); // read/write access

// Recompute BaseAddress
memory_info.BaseAddress = (char *) memory_info.BaseAddress +
memory_info.RegionSize;
}

I have no idea what you're trying to achieve here. Is this just
demonstration code or a simplified version of what you're actually using? It
certainly doesn't compile, and even with the obvious changes it seems pointless.

There's no point in looking for free memory yourself. If you want that, you
should simply pass NULL for the address in VirtualAlloc(). Also, you must
pass MEM_COMMIT | MEM_RESERVE if you want to both reserve and commit memory;
just specifying MEM_COMMIT only works if the memory is already reserved. If
you just want a big region of memory where you can do contiguous allocations
of your own, use MEM_RESERVE to reserve memory and MEM_COMMIT to commit
pages within that region.

I wrote a demonstration program of my own. It does a few VirtualAlloc()s
from both the main EXE and the DLL, freeing each intermediate result. The
results are consistent: the behavior of VirtualAlloc() depends *only* on
whether the executable is marked LAA; how the DLL is marked is irrelevant.
To be certain I repeated the test with a dynamically loaded DLL; the results
are the same.

Here's the output for a 32-bit app running on 64-bit Windows when the
executable is *not* marked /LARGEADDRESSAWARE:

VirtualAlloc(NULL, 0x1000, MEM_COMMIT | MEM_RESERVE | MEM_TOP_DOWN,
PAGE_READWRITE):
7EFA0000
VirtualAlloc(NULL, 0x1000, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE):
00520000
VirtualAlloc(0x01000000, 0x1000, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE):
01000000
VirtualAlloc(0x81000000, 0x1000, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE):
FAILED: Attempt to access invalid address.

---DLL---
VirtualAlloc(NULL, 0x1000, MEM_COMMIT | MEM_RESERVE | MEM_TOP_DOWN,
PAGE_READWRITE):
7EFA0000
VirtualAlloc(NULL, 0x1000, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE):
00520000
VirtualAlloc(0x01000000, 0x1000, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE):
01000000
VirtualAlloc(0x81000000, 0x1000, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE):
FAILED: Attempt to access invalid address.


And here's the output when the EXE *is* marked LAA:

---EXE---
VirtualAlloc(NULL, 0x1000, MEM_COMMIT | MEM_RESERVE | MEM_TOP_DOWN,
PAGE_READWRITE):
FFFA0000
VirtualAlloc(NULL, 0x1000, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE):
00520000
VirtualAlloc(0x01000000, 0x1000, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE):
01000000
VirtualAlloc(0x81000000, 0x1000, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE):
81000000

---DLL---
VirtualAlloc(NULL, 0x1000, MEM_COMMIT | MEM_RESERVE | MEM_TOP_DOWN,
PAGE_READWRITE):
FFFA0000
VirtualAlloc(NULL, 0x1000, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE):
00520000
VirtualAlloc(0x01000000, 0x1000, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE):
01000000
VirtualAlloc(0x81000000, 0x1000, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE):
81000000

Results were obtained on Windows Server 2003 R2 x64.

I want to know will managed code be able to fragment the large addresses and
is AWE is a better performance wise approach handling large memory in win32
application.

I don't know what you mean by "fragmenting" the addresses, but the CLR will
be able to access any memory you can allocate just fine.

As for AWE: don't bother. If you need >4G memory (which is really the point
where AWE becomes necessary) you should move to 64-bit and leave the
restrictions of 32-bit behind you altogether. I'm fairly certain the CLR has
no support for AWE, so managing it would have to be done entirely from
unmanaged code, which negates a lot of the advantages of garbage collection
(a big plus the CLR offers). Performance is not the issue here: simply being
able to use memory at all is.