Help, 64 bit Linux only sees 3.5G of RAM on an A64 X2

[General Schvantzkoph]

Just got an Athlon 64 X2 system with 4G of RAM. The motherboard is an MSI
K8N Neo4 Platinum (NForce 4). I'm using 64 bit Fedora Core 3 with a
2.6.11.12 kernel compiled for the Athlon 64.

The BIOS has a switch called S/W memory hole remapping, when OFF the BIOS
sees 3.5G of RAM as does Linux. When ON the BIOS sees all 4G of RAM but
Linux sees only 3G. I've given a mem=4096M to GRUB,

title Fedora Core (2.6.11.12)
root (hd0,0)
kernel /boot/vmlinuz-2.6.11.12 ro root=LABEL=/1 mem=4096M rhgb quiet
initrd /boot/initrd-2.6.11.12.img

This doesn't help, is my grub config correct?

With the memory hole remapping ON the mem switch is required, if it's not
there I get a kernel panic.

Is there another switch that I need? Is there a kernel config switch that
I need to throw.

BTW this a custom kernel based on the .config file for the FC3 2.6.12
kernel. The current FC3 2.6.12 kernel panics on boot because they left our
the nv_sata driver. I built a standard 2.6.12.3 kernel but it panics
because of a problem with the Silicon Image RAID driver, the 2.6.11.12
kernel boots fine, however I had to install Nvidia's networking drivers to
get the ethernet controller to work, the kernel 2.6.11.12 drivers don't
seem to work although the drivers in the old 2.6.9 kernel that comes with
FC3 work fine.

 

[joe]

Just got an Athlon 64 X2 system with 4G of RAM. The motherboard is an MSI
K8N Neo4 Platinum (NForce 4). I'm using 64 bit Fedora Core 3 with a
2.6.11.12 kernel compiled for the Athlon 64.
Cool!

Is there another switch that I need? Is there a kernel config switch that
I need to throw.

You bet there is!

BTW this a custom kernel based on the .config file for the FC3 2.6.12
kernel. The current FC3 2.6.12 kernel panics on boot because they left our

You're doing WHAT?!?!? You're using 2.6.11 with a config lifted from a
broken 2.6.12?!?!? What were you thinking?!?!?

You need to find an old config for 2.6.11 and make menuconfig from it,
customizing as necessary. If you can't find such a file, you should
start from scratch or even reinstall the 2.6.11 source package.

The config for 2.6.11 contains a line called CONFIG_NOHIGHMEM. In
menuconfig, this appears as a list of three options: off, 4GB, 64GB.
This is deprecated in 2.6.12 so you have essentially had it set to off
(because it was missing altogether).

You should never try to use a config file from one version with another
version. Do "make oldconfig" and it will walk you through the upgrade
from one kernel version to another. Unfortunately, "make oldconfig"
just tells you the name of the new features added to the upgraded
kernel; it doesn't explain what they mean. With a little slight of
hand, you can open a copy of the new kernel version's config file in
"make menuconfig" so that you'll have explanations of all the new
features. This will let you do "make oldconfig" in another terminal and
you'll know how to answer the new lines. When you're done, you will
have an upgraded config file appropriate for the new kernel version, but
retaining all of your old settings (if they still apply in the new version).

Since you have 4GB and that is an edge condition, you will want to test
a kernel built with CONFIG_NOHIGHMEM set at 4GB. If it still reports
less than 4GB, you may need to try setting it to 64GB. I know it sounds
like overkill but what can you do.

Well, actually there is something you can do. Upgrade to 2.6.12 and get
it working. I'm pretty sure this HIGHMEM stuff is handled automatically
by the more recent kernels.

 

[General Schvantzkoph]

You bet there is!


You're doing WHAT?!?!? You're using 2.6.11 with a config lifted from a
broken 2.6.12?!?!? What were you thinking?!?!?

You need to find an old config for 2.6.11 and make menuconfig from it,
customizing as necessary. If you can't find such a file, you should
start from scratch or even reinstall the 2.6.11 source package.

The config for 2.6.11 contains a line called CONFIG_NOHIGHMEM. In
menuconfig, this appears as a list of three options: off, 4GB, 64GB.
This is deprecated in 2.6.12 so you have essentially had it set to off
(because it was missing altogether).

You should never try to use a config file from one version with another
version. Do "make oldconfig" and it will walk you through the upgrade
from one kernel version to another. Unfortunately, "make oldconfig"
just tells you the name of the new features added to the upgraded
kernel; it doesn't explain what they mean. With a little slight of
hand, you can open a copy of the new kernel version's config file in
"make menuconfig" so that you'll have explanations of all the new
features. This will let you do "make oldconfig" in another terminal and
you'll know how to answer the new lines. When you're done, you will
have an upgraded config file appropriate for the new kernel version, but
retaining all of your old settings (if they still apply in the new version).

Since you have 4GB and that is an edge condition, you will want to test
a kernel built with CONFIG_NOHIGHMEM set at 4GB. If it still reports
less than 4GB, you may need to try setting it to 64GB. I know it sounds
like overkill but what can you do.

Well, actually there is something you can do. Upgrade to 2.6.12 and get
it working. I'm pretty sure this HIGHMEM stuff is handled automatically
by the more recent kernels.

I've built a 32 bit 2.6.11.12 kernel with highmem on, it does the same
thing as the 64 bit kernel. The 2.6.12 kernel has a broken driver for the
Silicon Image SATA raid chip which is on my motherboard. I've been trying
to get a 2.6.13.rc3 kernel to boot. The 2.6.13 kernel has a bunch of new
memory modes, flat and a couple of NUMA modes that have different
mechanisms for handling discontinuous memory. I tried the NUMA kernel,
wouldn't boot at all, it gets to the booting kernel line and freezes. I'm
now building a non-NUMA kernel with the flat memory model. The X2 is a
shared memory multi-processor, it's not NUMA, but architecturally it's
nearly identical to two Opterons so I'm surprised that the NUMA mode
doesn't boot but then again the BIOS is basically a uniprocessor BIOS so
maybe that explains it.

I have another data point. I ran Memtest86. It has a couple of ways of
sizing memory. The default mechanism is to ask the BIOS, in that mode it
sees 4G if the BIOS has memory hole remapping turned on, and 3.5G if
off. It also has a memory probe method. With the memory probe method it
gets exactly the same results as Linux, 3.5G if memory hole mapping is
OFF, 3G if it's ON. Memtest86 crashes if you run the test using the memory
probe method, it runs fine on all 4G with no errors if you use the BIOS
method of memory sizing.

 

[General Schvantzkoph]

On Wed, 27 Jul 2005 15:19:06 -0400, General Schvantzkoph wrote:

I got some e-mail from MSI, it's a design defect and they aren't going to
fix it. Anyone wanting 4G or more of RAM should pick another brand, the
MSI K8N Neo4 can only see 3.5G.

Here is the notice on their website.

Due to the South Bridge resource deployment, the system density will only
be detected up to 3+ GB (not full 4GB) when each DIMM is installed with an
1GB memory module

 

[Doug Lynn]

Hi, the chipset and cpu are probably reserving part of high memory, its
always been that way with 4G of memory.

doug
www.lynncomp.com

 

[Hari Seldon]

Il Sat, 30 Jul 2005 22:50:39 -0400, Doug Lynn ha scritto

Hi, the chipset and cpu are probably reserving part of high memory, its
always been that way with 4G of memory.

It's a 64 bit system, the statement doesn't make any sense.

Moreover, the OP discovered what went wrong. It's board's fault. Msi
admitted they did something wrong, that mb won't support more than
3.5GB.

 

[General Schvantzkoph]

Il Sat, 30 Jul 2005 22:50:39 -0400, Doug Lynn ha scritto

It's a 64 bit system, the statement doesn't make any sense.

Moreover, the OP discovered what went wrong. It's board's fault. Msi
admitted they did something wrong, that mb won't support more than
3.5GB.

Right. The AMD64 has a 64 bit Virtual Address and more importantly it has
a 40 bit physical address so it makes no sense to map the chipset into the
same physical space as the RAM. Obviously this is a solvable problem
because Opteron systems support 8G per processor. I'm guessing that this
is really a BIOS problem because the Award BIOS was developed for 32 bit
systems and they haven't gotten around to rewriting it for 64 bit chips.
The memory controller on the A64 is on the CPU not in the chipset so I
don't see what they could have done to actually block access to all 4G.
The RAM could be mapped anywhere in the 40 bit physical address space so
there is no excuse for mapping it into the same space as the bridge chip.

 

[Hari Seldon]

Il Sun, 31 Jul 2005 08:54:48 -0400, General Schvantzkoph ha scritto

Right. The AMD64 has a 64 bit Virtual Address and more importantly it has
a 40 bit physical address so it makes no sense to map the chipset into the
same physical space as the RAM. Obviously this is a solvable problem
because Opteron systems support 8G per processor. I'm guessing that this
is really a BIOS problem because the Award BIOS was developed for 32 bit
systems and they haven't gotten around to rewriting it for 64 bit chips.
The memory controller on the A64 is on the CPU not in the chipset so I
don't see what they could have done to actually block access to all 4G.
The RAM could be mapped anywhere in the 40 bit physical address space so
there is no excuse for mapping it into the same space as the bridge chip.

Actually there still is a part of northbridge: the one necessary for the
agp/pci-e video interface is still there. Could *this* one be the culprit
somehow? Or the ACPI interface?
I'm not quite convinced about the bios part. Does Opteron have something
totally different from the usual award/ami/whatever stuff?

 

[General Schvantzkoph]

Il Sun, 31 Jul 2005 08:54:48 -0400, General Schvantzkoph ha scritto


Actually there still is a part of northbridge: the one necessary for the
agp/pci-e video interface is still there. Could *this* one be the culprit
somehow? Or the ACPI interface?
I'm not quite convinced about the bios part. Does Opteron have something
totally different from the usual award/ami/whatever stuff?

Server cards have had to deal with > 4G for a long time because the
physical address space on a Xeon is 36 bits, so they must have solved the
problem there. So you would expect a server BIOS to be able to
accommodate a large address space. 4G on desktop systems is brand new
although it can't be a surprise to anyone designing a chipset or writing
a BIOS.

As for the physical address of each DIMM it can be anywhere in the 40 bit
space as long as the OS knows about it. The memory management code in the
OS takes care of mapping virtual memory to physical memory and as already
stated the MTU entries in the Athlon64 accommodate a 40 bit address. Even
the legacy modes accommodate 36 bits because that's what the X86s have
used for years. Also the memory controller is on the CPU, it's completely
divorced from the bridge chip so how is it possible for there to be
anything on the board that prevents the BIOS from recognizing all 4G and
more importantly passing the info to the OS.

The problem could be a Linux issue. The BIOS does see all 4G if you use
memory hole mapping. However in this mode Linux sees only 3G, it also
requires a mem= statement in order to boot. I've tried lots of kernels
including the latest release candidate for 2.6.12, they all behave
identically. I don't own a copy of XP so I have no idea if it can handle
memory hole mapping. I'd be curious to hear if anyone with a 4G XP system
has the same issue.

 

[dead kitty]

Server cards have had to deal with > 4G for a long time because the
physical address space on a Xeon is 36 bits, so they must have solved
the problem there. So you would expect a server BIOS to be able to
accommodate a large address space. 4G on desktop systems is brand new
although it can't be a surprise to anyone designing a chipset or
writing a BIOS.

As for the physical address of each DIMM it can be anywhere in the 40
bit space as long as the OS knows about it. The memory management code
in the OS takes care of mapping virtual memory to physical memory and
as already stated the MTU entries in the Athlon64 accommodate a 40 bit
address. Even the legacy modes accommodate 36 bits because that's what
the X86s have used for years. Also the memory controller is on the
CPU, it's completely divorced from the bridge chip so how is it
possible for there to be anything on the board that prevents the BIOS
from recognizing all 4G and more importantly passing the info to the
OS.

The problem could be a Linux issue. The BIOS does see all 4G if you
use memory hole mapping. However in this mode Linux sees only 3G, it
also requires a mem= statement in order to boot. I've tried lots of
kernels including the latest release candidate for 2.6.12, they all
behave identically. I don't own a copy of XP so I have no idea if it
can handle memory hole mapping. I'd be curious to hear if anyone with
a 4G XP system has the same issue.

I have the 4400+, 4GB PC3200, Abit Fatal1ty SLi, XP x64.
I see 4.00GB displayed in Windows properties.

Running XP 32-bit I only see 2.75GB of my 4GB. Is your Linux 32-bit?

 

[General Schvantzkoph]

I have the 4400+, 4GB PC3200, Abit Fatal1ty SLi, XP x64.
I see 4.00GB displayed in Windows properties.

Running XP 32-bit I only see 2.75GB of my 4GB. Is your Linux 32-bit?

I'm running 64 bit Linux, latest kernel. I've also tried a 32 bit large
mem kernel. Do you have memory hole mapping turned on in your BIOS?

 

[Scott Lurndal]

I have another data point. I ran Memtest86. It has a couple of ways of
sizing memory. The default mechanism is to ask the BIOS, in that mode it
sees 4G if the BIOS has memory hole remapping turned on, and 3.5G if
off. It also has a memory probe method. With the memory probe method it
gets exactly the same results as Linux, 3.5G if memory hole mapping is
OFF, 3G if it's ON. Memtest86 crashes if you run the test using the memory
probe method, it runs fine on all 4G with no errors if you use the BIOS
method of memory sizing.

"Memory Hole Mapping"

The northbridge on amd64 processors supports a concept called
memory hoisting. The Memory Mapped I/O space in 32-bit environments
is often in the upper 512MB of the 32-bit address space. When you
have 4GB of DRAM, the upper 512MB of the DRAM conflicts with the
MMIO range, so the BIOS can set a bit in the northbridge configuration
that will "hoist" the last 512MB of DRAM and re-base it at 4GB (which
makes the highest physical DRAM address (or Top Of Memory in amd-ese)
4.5GB, with a hole (occupied by the MMIO space) from 3.5GB to 4.0GB).

Memtest will crash using the probe method because it is writing to
MMIO space (i.e pci device configuration and operation registers) between
3.5GB and 4.0GB.

64-bit linux only seeing 3.0GB when hoisting is enabled sounds like
either a bug in the linux sizing code, or a bug in the BIOS when it exposes
the memory configuration through ACPI and/or E820.

A 32-bit kernel would need to use PAE to access the high 512MB when
hoisting is enabled - config for 64G should enable PAE support.

The early dmesg(1m) output should show the values of the E820 BIOS memory
map that the kernel operates from, can you post that?

scott

 

[General Schvantzkoph]

"Memory Hole Mapping"

The northbridge on amd64 processors supports a concept called
memory hoisting. The Memory Mapped I/O space in 32-bit environments
is often in the upper 512MB of the 32-bit address space. When you
have 4GB of DRAM, the upper 512MB of the DRAM conflicts with the
MMIO range, so the BIOS can set a bit in the northbridge configuration
that will "hoist" the last 512MB of DRAM and re-base it at 4GB (which
makes the highest physical DRAM address (or Top Of Memory in amd-ese)
4.5GB, with a hole (occupied by the MMIO space) from 3.5GB to 4.0GB).

Memtest will crash using the probe method because it is writing to
MMIO space (i.e pci device configuration and operation registers) between
3.5GB and 4.0GB.

64-bit linux only seeing 3.0GB when hoisting is enabled sounds like
either a bug in the linux sizing code, or a bug in the BIOS when it exposes
the memory configuration through ACPI and/or E820.

A 32-bit kernel would need to use PAE to access the high 512MB when
hoisting is enabled - config for 64G should enable PAE support.

The early dmesg(1m) output should show the values of the E820 BIOS memory
map that the kernel operates from, can you post that?

scott

I'll put the 2G back into the system (I swapped it into another system)
and post the dmesg info.

 

[Ville Muikkula]

The BIOS has a switch called S/W memory hole remapping, when OFF the
BIOS sees 3.5G of RAM as does Linux. When ON the BIOS sees all 4G of RAM
but Linux sees only 3G. I've given a mem=4096M to GRUB

You could try to use the mem= boot option like this (with values that
are correct for your system):
mem=exactmap mem=640K@0 mem=3071M@1M mem=1024M@4096M

 

[General Schvantzkoph]

On Sat, 06 Aug 2005 03:11:34 +0000, Scott Lurndal wrote:

Here is the dmesg info from a 32 bit 64G highmem kernel, I'll have to
reinstall 64 bit FC3 to boot a 64 bit kernel.

Linux version 2.6.11.12BigMem ([email protected]) (gcc version 3.4.4 20050721 (Red Hat 3.4.4-2)) #2 SMP Sat Aug 6 09:20:51 EDT 2005
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009f400 (usable)
BIOS-e820: 000000000009f400 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 00000000bfff0000 (usable)
BIOS-e820: 00000000bfff0000 - 00000000bfff3000 (ACPI NVS)
BIOS-e820: 00000000bfff3000 - 00000000c0000000 (ACPI data)
BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
BIOS-e820: 00000000fec00000 - 0000000100000000 (reserved)
BIOS-e820: 0000000100000000 - 0000000140000000 (usable)
user-defined physical RAM map:
user: 0000000000000000 - 000000000009f400 (usable)
user: 000000000009f400 - 00000000000a0000 (reserved)
user: 00000000000f0000 - 0000000000100000 (reserved)
user: 0000000000100000 - 00000000bfff0000 (usable)
user: 00000000bfff0000 - 00000000bfff3000 (ACPI NVS)
user: 00000000bfff3000 - 00000000c0000000 (ACPI data)
user: 00000000e0000000 - 00000000f0000000 (reserved)
user: 00000000fec00000 - 0000000100000000 (reserved)
user: 0000000100000000 - 0000000100000000 (usable)
2175MB HIGHMEM available.
896MB LOWMEM available.
found SMP MP-table at 000f5300
NX (Execute Disable) protection: active
On node 0 totalpages: 786416
DMA zone: 4096 pages, LIFO batch:1
Normal zone: 225280 pages, LIFO batch:16
HighMem zone: 557040 pages, LIFO batch:16
DMI 2.3 present.

free
total used free shared buffers cached
Mem: 3116196 519532 2596664 0 32832 409304
-/+ buffers/cache: 77396 3038800
Swap: 0 0 0

 

[General Schvantzkoph]

Here is the dmseg results for a 64 bit kernel, looks the same as for the
highmem 32 bit kernel.

Bootdata ok (command line is ro root=LABEL=/1 mem=4g rhgb quiet)
Linux version 2.6.12-1.1372_FC3smp ([email protected]) (gcc version 3.4.3 20050227 (Red Hat 3.4.3-22)) #1 SMP Fri Jul 15 01:08:54 EDT 2005
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009f400 (usable)
BIOS-e820: 000000000009f400 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 00000000bfff0000 (usable)
BIOS-e820: 00000000bfff0000 - 00000000bfff3000 (ACPI NVS)
BIOS-e820: 00000000bfff3000 - 00000000c0000000 (ACPI data)
BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
BIOS-e820: 00000000fec00000 - 0000000100000000 (reserved)
BIOS-e820: 0000000100000000 - 0000000140000000 (usable)
ACPI: RSDP (v000 Nvidia ) @ 0x00000000000f91c0
ACPI: RSDT (v001 Nvidia AWRDACPI 0x42302e31 AWRD 0x00000000) @ 0x00000000bfff3040
ACPI: FADT (v001 Nvidia AWRDACPI 0x42302e31 AWRD 0x00000000) @ 0x00000000bfff30c0
ACPI: SSDT (v001 PTLTD POWERNOW 0x00000001 LTP 0x00000001) @ 0x00000000bfff9680
ACPI: SRAT (v001 AMD HAMMER 0x00000001 AMD 0x00000001) @ 0x00000000bfff98c0
ACPI: MCFG (v001 Nvidia AWRDACPI 0x42302e31 AWRD 0x00000000) @ 0x00000000bfff9a00
ACPI: MADT (v001 Nvidia AWRDACPI 0x42302e31 AWRD 0x00000000) @ 0x00000000bfff95c0
ACPI: DSDT (v001 NVIDIA AWRDACPI 0x00001000 MSFT 0x0100000e) @ 0x0000000000000000
SRAT: PXM 0 -> APIC 0 -> Node 0
SRAT: PXM 0 -> APIC 1 -> Node 0
SRAT: Node 0 PXM 0 0-9ffff
SRAT: Node 0 PXM 0 0-bfffffff
SRAT: Node 0 PXM 0 0-13fffffff
Bootmem setup node 0 0000000000000000-00000000ffffffff
On node 0 totalpages: 1048575
DMA zone: 4096 pages, LIFO batch:1
Normal zone: 1044479 pages, LIFO batch:31
HighMem zone: 0 pages, LIFO batch:1

 

[General Schvantzkoph]

Found the solution. Set the BIOS to Fill Memory Hole and then told Linux
it had 5G, i.e. mem=5G on the boot line.

BTW a mem= line is required if when you set the BIOS to Fill Memory Hole,
without it you'll get a kernel panic when you boot. The FC3 installer just
does a straight boot so it will panic on a machine with the Fill Memory
Hole option set in the BIOS. The work around is to disable the option when
you do an install, alternatively I suppose you could boot the installer
with a mem=.