PC 4GB RAM limit

[Tim Anderson]

The cheaper motherboards (ie. most of them) for P4 or Athlon 64 support a
maximum of 4GB RAM, even if the motherboards and processors support the
AMD64 extensions that can address sqillions of GB. On the other hand, more
upmarket boards support more RAM - up to 24GB or maybe more - for Xeon and
Opteron.

If you buy one of these 4GB boards and install 4GB RAM, you don't get the
use of all of it. The top of the 4GB address space gets shadowed by system
functions such as PCI Express addressing. This is not just a small detail
- typically you lose 1GB of your 4GB. See:

http://www.itwriting.com/blog/?postid=152

I can't at the moment find a clear explanation of this. I understand about
the shadowing, but the question of course is why a modern board can't use
a higher range of addresses to make the full 4GB available to the OS. The
manufacturers mutter about "PC Architecture", but then again they also
make boards that *do* overcome this limit.

With PAE, PC processors have been able to address more than 4GB for years.
So why are we still running into this limit?

Tim

 

[Rick]

The cheaper motherboards (ie. most of them) for P4 or Athlon 64 support a
maximum of 4GB RAM, even if the motherboards and processors support the
AMD64 extensions that can address sqillions of GB. On the other hand, more
upmarket boards support more RAM - up to 24GB or maybe more - for Xeon and
Opteron.

If you buy one of these 4GB boards and install 4GB RAM, you don't get the
use of all of it. The top of the 4GB address space gets shadowed by system
functions such as PCI Express addressing. This is not just a small detail
- typically you lose 1GB of your 4GB. See:

http://www.itwriting.com/blog/?postid=152

I can't at the moment find a clear explanation of this. I understand about
the shadowing, but the question of course is why a modern board can't use
a higher range of addresses to make the full 4GB available to the OS. The
manufacturers mutter about "PC Architecture", but then again they also
make boards that *do* overcome this limit.

With PAE, PC processors have been able to address more than 4GB for years.
So why are we still running into this limit?

Because *applications* have to be written to be PAE-aware,
and have to be written in 64-bit address space to make use of
anything over 4GB. This was (and still is) very much a niche
market -- very few desktop/workstation users will come close
to using 2GB, let alone 4, which is why MS originally split the
4GB address range between 2GB user space and 2GB
system space.

 

[Tim Anderson]

Because *applications* have to be written to be PAE-aware,
and have to be written in 64-bit address space to make use of
anything over 4GB. This was (and still is) very much a niche
market -- very few desktop/workstation users will come close
to using 2GB, let alone 4, which is why MS originally split the
4GB address range between 2GB user space and 2GB
system space.

Rick,

Thanks for the response. I understand about that; what I'm talking about is
the hardware issue. When you boot with 4GB installed on one of these
motherboards, typically 1GB is not made available to the OS at all; it is
"consumed" before the OS boots (because of address shadowing). Even with an
AMD64 (EM64T) CPU installed.

Tim

 

[Rick]

Rick,

Thanks for the response. I understand about that; what I'm talking about is
the hardware issue. When you boot with 4GB installed on one of these
motherboards, typically 1GB is not made available to the OS at all; it is
"consumed" before the OS boots (because of address shadowing). Even with an
AMD64 (EM64T) CPU installed.

The Intel D915PBL (the board referenced in your link) is a desktop
board:
ftp://download.intel.com/design/motherbd/bl/C6859901.pdf

The board utilizes 4 GB of addressable system memory. Typically
the address space that is allocated for PCI Conventional bus add-in
cards, PCI Express configuration space, BIOS (firmware hub), and
chipset overhead resides above the top of DRAM (total system
memory). On a system that has 4 GB of system memory installed, it
is not possible to use all of the installed memory due to system
address space being allocated for other system critical functions.

These functions include the following:

BIOS/firmware hub (2 MB)
Local APIC (19 MB)
Digital Media Interface (40 MB)
Front side bus interrupts (17 MB)
PCI Express configuration space (256 MB)
MCH base address registers, internal graphics ranges, PCI Express ports (up to 512 MB)
Memory-mapped I/O that is dynamically allocated for PCI Conventional and
PCI Express add-in cards

The amount of installed memory that can be used will vary based
on add-in cards and BIOS settings. Figure 14 shows a schematic
of the system memory map. All installed system memory can be
used when there is no overlap of system addresses.

 

[Tim Anderson]

Thanks again Rick; yes, I'm familiar with that document and I've read it
before.

My question was, to quote from my original message:

"I can't at the moment find a clear explanation of this. I understand about
the shadowing, but the question of course is why a modern board can't use
a higher range of addresses to make the full 4GB available to the OS. The
manufacturers mutter about "PC Architecture", but then again they also
make boards that *do* overcome this limit.

With PAE, PC processors have been able to address more than 4GB for years.
So why are we still running into this limit?"

Tim

 

[Al Dykes]

Because *applications* have to be written to be PAE-aware,
and have to be written in 64-bit address space to make use of
anything over 4GB. This was (and still is) very much a niche
market -- very few desktop/workstation users will come close
to using 2GB, let alone 4, which is why MS originally split the
4GB address range between 2GB user space and 2GB
system space.

Real (tm) operating systems, such as VMS, have done something
splitting the address space this for decades.

What OS were you running?

So what happens on a dual CPU opteron system with 8 memory slots
(and a 64 bit OS)?

I think you'll find that on a PAE mobo the CPU never sees a hardware
address higher than the 4GB limit. Hardware registers external to the
CPU are set by the PAE drivers and memory mapping is done to show
different segments of physical memory to the CPU.

What OS are you using,

 

[Tim Anderson]

Real (tm) operating systems, such as VMS, have done something
splitting the address space this for decades.

What OS were you running?

This is not an OS issue, the memory is made unavailable before the OS boots.

But for the record, dual boot Win XP x64 and Suse Linux 9.3 64-bit.

So what happens on a dual CPU opteron system with 8 memory slots
(and a 64 bit OS)?

That's what I'm wondering, as these systems support more than 4GB RAM
successfully.

Tim

 

[Mxsmanic]

These functions include the following:

BIOS/firmware hub (2 MB)
Local APIC (19 MB)
Digital Media Interface (40 MB)
Front side bus interrupts (17 MB)
PCI Express configuration space (256 MB)
MCH base address registers, internal graphics ranges, PCI Express ports (up to 512 MB)
Memory-mapped I/O that is dynamically allocated for PCI Conventional and
PCI Express add-in cards

Looks like PCI Express is a staggering waste of memory space.

 

[Tim Anderson]

Rick writes:

Looks like PCI Express is a staggering waste of memory space.

Yes, PCI Express is the main guilty party here.

But *why* does this address space have to be in the first 4GB? Sounds like a
32-bit issue; but especially for boards that support AMD64 this is all very
unfortunate.

Tim

 

[CBFalconer]

Thanks for the response. I understand about that; what I'm talking
about is the hardware issue. When you boot with 4GB installed on
one of these motherboards, typically 1GB is not made available to
the OS at all; it is "consumed" before the OS boots (because of
address shadowing). Even with an AMD64 (EM64T) CPU installed.

Because once you assign a group of memory addresses for non-memory
(i.e. i/o) use you have to decode that range. Not only do you have
to designate a range for i/o, you also have to designate the
complementary range for actual memory. The decoders involve added
loads on the address lines, and added propagation delay. So you
want to minimize the bits used in this, and possibly strengthen
their output buffers on the CPU chip.

All of which costs time, power, and performance.

I expect the main problems are due to the added capacitance from
added fan-out. This is similar to the often encountered
difficulties when further memory modules are added to a system.

All of this deals with the physical addresses, not the virtual
addresses that the software sees.

 

[Rick]

Yes, PCI Express is the main guilty party here.

But *why* does this address space have to be in the first 4GB?

To maintain compatibility with OS's that are limited to 4GB.
I came across this note in one of Intel's server motherboard
manuals (which supports up to 8GB):
ftp://download.intel.com/support/motherboards/server/sb/sp2gp2tps.pdf

Note: Memory between 4GB and 4GB minus 512MB will not
be accessible for use by the operating system and may be lost
to the user, because this area is reserved for BIOS, APIC
configuration space, PCI adapter interface, and virtual video
memory space. This means that if 4GB of memory is installed,
3.5GB of this memory is usable. The chipset should allow the
remapping of unused memory above the 4GB address, but
this memory may not be accessible to an operating system that
has a 4GB memory limit.

 

[Tim Anderson]

3.5GB of this memory is usable. The chipset should allow the
remapping of unused memory above the 4GB address, but
this memory may not be accessible to an operating system that
has a 4GB memory limit.

Yes, this is helpful. Thanks. So the proper question may be this: Why don't
these 4GB boards remap this memory to make it usable by 64-bit operating
systems?

Tim

 

[Tim Anderson]

All of this deals with the physical addresses, not the virtual
addresses that the software sees.

Thanks Chuck. Can you expand on that last remark? How difficult would it be
to make this wasted RAM available to the OS? For the board deisgner, that
is. I'm not expecting any workaround for the motherboard I'm using.

Tim

 

[CBFalconer]

Yes, this is helpful. Thanks. So the proper question may be this:
Why don't these 4GB boards remap this memory to make it usable by
64-bit operating systems?

Because it is an unnecessary expense. While there may be uses for
addressing over 4GB to conveniently handle large databases etc.,
there is virtually no real need for that much actual memory.

The function can be efficiently served by virtual memory systems at
little performance cost. About the only area where I can conceive
such real memory to be useful is in high performance video
processing.

--
Some informative links:
http://www.geocities.com/nnqweb/

http://www.caliburn.nl/topposting.html
http://www.netmeister.org/news/learn2quote.html

 

[CBFalconer]

Thanks Chuck. Can you expand on that last remark? How difficult
would it be to make this wasted RAM available to the OS? For the
board deisgner, that is. I'm not expecting any workaround for the
motherboard I'm using.

If you don't install memory in a physical area it isn't wasted.
Virtual memory systems involve remapping memory anyhow, so the
software doesn't see any gaps. There is no real problem.

--
Some informative links:
http://www.geocities.com/nnqweb/

http://www.caliburn.nl/topposting.html
http://www.netmeister.org/news/learn2quote.html

 

[Rick]

Because it is an unnecessary expense. While there may be uses for
addressing over 4GB to conveniently handle large databases etc.,
there is virtually no real need for that much actual memory.

The function can be efficiently served by virtual memory systems at
little performance cost. About the only area where I can conceive
such real memory to be useful is in high performance video
processing.

Even for that, apps (at least if they're written properly) process
smaller chunks of ram at a time.

 

[Dorothy Bradbury]

There are several other reasons for the 4GB limit:
o Historically, motherboards were limited by RAM VRM
---- lots of RAM sockets = lots of DIMMs = lots of watts = 30W+
---- RAM VRMs are limited - indeed BTX tackles their cooling
o Historically, memory is relatively expensive
---- PC industry focuses products around usage segments
---- multi-GB buyers tended to buy servers, not general PCs

As always, the computer industry migrates downwards.

Applications using multi-GB were once small & niche:
o dBase servers = RAM determinant
---- more RAM = more index keys, more caching
---- contrast with slow electromechanical HD rotation & seek
---- however such PCs are 1) Servers 2) 64-bit
o Photoshop production machines = RAM determinant
---- historically most pre-press is done on Apples
---- recently reinforced by the Dual G5 Apple re OS-X + 8GB+
---- Photoshop on the PC has deteriorating benefit of >2GB

Remember the focus of the IT industry:
o Create as many interfaces with buyers as possible per unit time
o Eg, CD, DVD, DVD-II speed & iteration, or mobile phone iterations

So some limits are based on guesswork "no-one will need >640KB"
and also one of creating a build-up of pressure for an obsolescence
replacement cycle as opposed to the migratory upgrade cycle.

So there are inherent assumptions in the PC architecture at work:
o Hardware both "internal" & from plug-in devices needs address space
---- efficiency has not been great here - PCI-Express has high demands
o Assumptions by the O/S come into play
---- XP may be based on NT, but it still makes relatively "old" assumptions

A lot of the assumptions are still valid somewhat:
o Most economic buyers of multi-GB = 64-bit = Dual-Opteron/Xeon64 Servers
o Many multi-GB buyers are after somewhat niche machines
---- Apple Dual G5 -- re Photoshop pre-press
---- downsized *nix box -- re displacing IBM/SUN/SGI boxes
o Eventually multi-GB buyers will migrate down to the commodity PC

It comes down to usage/economics:
o Yes CPUs can gobble memory vastly faster than a HD can move it
---- a typical RAID-10 array can manage 150MB/sec
---- a typical P4 CPU can manage 40x that on RAM to CPU transfers
o In reality very few applications run into that as a bottleneck
---- it's a tail-end of the usage distribution - few buyers, high cost, low margin
---- an error is to assume it remains that way - it never does

Adding lots of memory to PCs has various electrical limitations, RAM VRM,
and various electrical dependencies required to maintain a stable I/O system.
That moves you away from mainstream buyer requirements re what they pay.

 

[Al Dykes]

There are several other reasons for the 4GB limit:
o Historically, motherboards were limited by RAM VRM
---- lots of RAM sockets = lots of DIMMs = lots of watts = 30W+
---- RAM VRMs are limited - indeed BTX tackles their cooling
o Historically, memory is relatively expensive
---- PC industry focuses products around usage segments
---- multi-GB buyers tended to buy servers, not general PCs

Are we talking virtual address space or physical memory here? I think
the latter.

Processes on a 64 bit OS see the same huge address space no matter how
much physical RAM you have. It's CPU and Mobo model implementation
specific.)

I just looked at amd.com and see that one of their Opteron chips has a
48 bit virtual address space and a 40 bit physical space.

The amount of memory you need depends on the working set of your
application mix.

 

[Dorothy Bradbury]

I think the latter.

Yes - physical memory.

The amount of memory you need depends on the working set of your
application mix.

Which currently still makes multi-GB requirements somewhat niche.
Dbase servers, Photoshop pre-press - few boxes versus generic PCs.

Some Photoshop filters do require physical RAM to be a quite high
multiple of the image size for completion. Upgrading from 2GB to 4GB
can be frustrating in that considerably less than 4GB is realisable.

Still XP 2012 probably has some obscene memory requirements.

 

[Michael Brown]

The cheaper motherboards (ie. most of them) for P4 or Athlon 64
support a maximum of 4GB RAM, even if the motherboards and processors
support the AMD64 extensions that can address sqillions of GB. On the
other hand, more upmarket boards support more RAM - up to 24GB or
maybe more - for Xeon and Opteron.

If you buy one of these 4GB boards and install 4GB RAM, you don't get
the use of all of it. The top of the 4GB address space gets shadowed
by system functions such as PCI Express addressing. This is not just
a small detail - typically you lose 1GB of your 4GB. See:

http://www.itwriting.com/blog/?postid=152

I can't at the moment find a clear explanation of this. I understand
about the shadowing, but the question of course is why a modern board
can't use a higher range of addresses to make the full 4GB available
to the OS. The manufacturers mutter about "PC Architecture", but then
again they also make boards that *do* overcome this limit.

With PAE, PC processors have been able to address more than 4GB for
years. So why are we still running into this limit?

Three reasons:
1) Chipset support
2) BIOS support
3) Application support

Because we still have (or rather, had) a 32-bit virtual address space, and
there are very few apps which can use PAE. Use of addresses above the 4GB
mark requires a bit of fancy footwork from the OS, and memory above this
limit cannot be used in a general-purpose way. For example, it cannot be
shared among processes. Because of this, it has to be
allocated/deallocated/mapped using a different API than the usual malloc()
or whatever your language uses. Large database application and the like
support PAE, but your normal "consumer" application does not, and hence all
your consumer applications still have to fit into the original 4GB address
space.

On top of this, many chipsets simply do not support PAE. To support it, they
must support mapping memory banks above the 4GB line, and some only support
32-bit addresses for bank base addresses. Finally, the motherboard
manufacturer has to set up things correctly in the BIOS and due to the lack
of demand for >4GB RAM in the consumer space, I suspect it hasn't been at
the top of their list.

With a larger virtual address space (and the larger physical address space
for PCIe), most of these problems are removed. I don't know about support in
NF4 and other chipsets (manufacturers won't even tell you the most basic
interface details nowadays ) but given the NF4 Pro motherboards support a
large address space, I suspect it's more of a BIOS issue than anything.