Running a large Java program

[masonmarc]

System specifics:

AMD x86-64
128 GB of physical memory
Red Hat Enterprise Linux 3 - 64-bit
kernel 2.4.21-47.ELsmp x86_64
8 cpus

I am trying to run a Java program that will take a lot of memory. In
one instance I give the system 90GB of heap space with:

java -Xmx90g JavaProgam

The program runs fine until top reports that it is using approximately
43GB and then it starts swapping. top reported about 62 - 63 GB total
used on the system (not sure what else is using the memory), but I
still have another 64GB. Why is the system swapping when I still have
memory? Do I need to do something to configure the system to use the
whole 128GB? Can one process use most of the 128GB?

Thanks.

 

[Scott Lurndal]

System specifics:

AMD x86-64
128 GB of physical memory
Red Hat Enterprise Linux 3 - 64-bit
kernel 2.4.21-47.ELsmp x86_64
8 cpus

I am trying to run a Java program that will take a lot of memory. In
one instance I give the system 90GB of heap space with:

java -Xmx90g JavaProgam

The program runs fine until top reports that it is using approximately
43GB and then it starts swapping. top reported about 62 - 63 GB total
used on the system (not sure what else is using the memory), but I
still have another 64GB. Why is the system swapping when I still have
memory? Do I need to do something to configure the system to use the
whole 128GB? Can one process use most of the 128GB?

Thanks.

Post the results of

$ cat /proc/meminfo


scott

 

[masonmarc]

Should I post the output of the meminfo when it is starting to swap or
would any point in time be O.K.? It takes a while to get to the point
of swapping.

Thanks,
Marc

 

[masonmarc]

Should I post the output of the meminfo when it is starting to swap or
would any point in time be O.K.? It takes a while to get to the point
of swapping.

Thanks,
Marc

Here is the output of another run at about 20GB of memory used by the
process and other processes using memory and the system is swapping.
I freed other memory and the program continued, but if I hadn't I've
seen cases where the program halts with no message or other times it
uses all the swap and the system is unusable.

-bash-3.00$ cat /proc/meminfo
total: used: free: shared: buffers: cached:
Mem: 1333383839744 66657845248 66725994496 0 174751744 21924777984
Swap: 4188856768 4186075136 581632
MemTotal: 130257656 kB
MemFree: 65162104 kB
MemShared: 0 kB
Buffers: 170656 kB
Cached: 21410780 kB
SwapCached: 136 kB
Active: 42317496 kB
ActiveAnon: 40921760 kB
ActiveCache: 1395736 kB
Inact_dirty: 7807336 kB
Inact_laundry: 7362508 kB
Inact_clean: 5877816 kB
Inact_target: 12673028 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 130257656 kB
LowFree: 65162104 kB
SwapTotal: 4088532 kB
SwapFree: 586 kB
CommitLimit: 69217360 kB
Committed_AS: 57812004 kB
HugePages_Total: 0
HugePages_Free: 0
Hugepagesize: 2048 kB

Marc

 

[Scott Lurndal]

Here is the output of another run at about 20GB of memory used by the
process and other processes using memory and the system is swapping.
I freed other memory and the program continued, but if I hadn't I've
seen cases where the program halts with no message or other times it
uses all the swap and the system is unusable.

-bash-3.00$ cat /proc/meminfo
total: used: free: shared: buffers: cached:
Mem: 1333383839744 66657845248 66725994496 0 174751744 21924777984
Swap: 4188856768 4186075136 581632
MemTotal: 130257656 kB
MemFree: 65162104 kB
MemShared: 0 kB
Buffers: 170656 kB
Cached: 21410780 kB
SwapCached: 136 kB
Active: 42317496 kB
ActiveAnon: 40921760 kB
ActiveCache: 1395736 kB
Inact_dirty: 7807336 kB
Inact_laundry: 7362508 kB
Inact_clean: 5877816 kB
Inact_target: 12673028 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 130257656 kB
LowFree: 65162104 kB
SwapTotal: 4088532 kB
SwapFree: 586 kB
CommitLimit: 69217360 kB
Committed_AS: 57812004 kB
HugePages_Total: 0
HugePages_Free: 0
Hugepagesize: 2048 kB

Marc

Folks usually have swap size == memory size. Here you have only 4GB swap backing 128GB
of memory. Note that at the point at which you snapshot meminfo, there is 60+GB of
free memory.

I'd create another swap area of say, 36GB.

scott

 

[masonmarc]

Folks usually have swap size == memory size. Here you have only 4GB swap backing 128GB
of memory. Note that at the point at which you snapshot meminfo, there is 60+GB of
free memory.

I'd create another swap area of say, 36GB.

scott

I could do that, but it really doesn't explain why I am swapping at
all with 60+ GB of free memory.

Anyone have any ideas?

Thanks,
Marc

 

[masonmarc]

Folks usually have swap size == memory size. Here you have only 4GB swap backing 128GB
of memory. Note that at the point at which you snapshot meminfo, there is 60+GB of
free memory.

I'd create another swap area of say, 36GB.

scott

I could do that, but it really doesn't explain why I am swapping at
all with 60+ GB of free memory.

Anyone have any ideas?

Thanks,
Marc

 

[Wes Newell]

I could do that, but it really doesn't explain why I am swapping at
all with 60+ GB of free memory.

Anyone have any ideas?

I thought Linux only supported 64GB. Since I've never had more than 1 I
haven't worried about it. I notice you don't have any hugepages and your
hugepage size is the default 2MB. I know the hugepage size goes to at
least 16MB from the little research I've done on them, but I'm still not
sure exactly what they are for. Perhaps they are for using more more than
64G of ram. This is all probably worthless wag info but I thought I'd
throw it out anyway. I wouldn't even think you'd need a swap file with
that much ram, but if the swapping is what's slowing you down it might be
time to look into a real ram drive system for the swap file if you can't
figure out how to use more than 64GB internally. I'll now get lost.

 

[Scott Lurndal]

I thought Linux only supported 64GB. Since I've never had more than 1 I

32-bit linux is limited to 64GB on ia32 due to PAE. On AMD64 or EMT64
64-bit linux is limited to 1TB by the physical bus width on the current processor
generations. Next gen Opteron will support more.

Hugepages are only used to reduce TLB usage for applications with large
datasets. The size of a hugepage is an architecture dependency. Large pages
on ia32 are 4MB. on x86_64, they are 2MB.

scott

 

[Scott Lurndal]

I could do that, but it really doesn't explain why I am swapping at
all with 60+ GB of free memory.

Anyone have any ideas?

Thanks,
Marc

I suspect a 2.4 kernel limitation. Try with a 2.6 kernel (e.g. RHEL4 or RHEL5).

scott