J
Jim Lewandowski
I've been scouring newsgroups for the past week to see what's going on out there with
regard to XP's memory management. I've also read Russinovich's Windows Internal book.
This is a long post for those so-inclined to not like to read verbose posts.
FWIW, my background comes from 25 years of machine/assembler-level coding on 4 different
hardware platforms. Memory algorithms and memory management, low-level disk I/O and
optimization (including buffering algorithms) are my forte'.
I've seen not ONE newsgroup post from anyone with specific examples. All the
newsgroup/regular web stuff I read talks about conceptuals. I'd like to move the
discussion to conceptual EXAMPLES (to start) and eventually move to real-life examples (by
discussing conceptual examples we can then hopefully agree on WHAT types of real-life
tests to perform).
I've also seen MVPs constantly point to various links/URLs which do NOT address the
specific questions being asked: NEVER. The URLs/links in question NEVER answer the 3
questions below.
1) CUSTOM page file not STICKING to what the user requests
2) WHY XP seems to not want to USE a lot of RAM (I have 1G on my machine) even when HALF
of it is still available (i.e. pagefile usage grows)
3) WHAT the actual memory management algorithms ARE (I have a hunch which we'll discuss at
the end)
***
SOME of the below is assumed to work a certain way based on my experience with the large
IBM mainframe OS (which is 1000x more complex than Windows) and has been evolving since
the 60's (NASA was a big user in the early days).
***
Gigabyte/Gig/GB and Megabyte/Meg/MB and kilobyte/KB/K are all powers of 2 (i.e. K is 1024,
MB is 1024*1024).
VIRTUAL MEMORY
Virtual memory is like having MORE memory than what your RAM (not disk, not anything else,
but RAM) will hold. The reasoning behind this is that programs use only a SUBSET of their
memory ALLOCATION request. Thus, for the 80% of the memory pages that they DON'T access
(whether DATA or CODE), those can either be (discussion later) discarded COMPLETELY out of
memory OR put onto a paging file if there's stress for RAM usage/needs.
XP (Windows, the CPU chips, however you want to slice and dice this) is designed to allow
each PROCESS to use a 4 GB address space (2 to the power 32) which is numbered in hex from
00000000 to FFFFFFFF. However, only HALF of that address space is USABLE by joe-average
application program. IOW, let's discard system-level (kernal, drivers) from the
discussion at this point. FTR, the USER accessible VIRTUAL address space area resides from
00000000-7FFFFFFF. Let's not discuss the special case of 3 GB user address spaces (of the
4 GB total when the system only gets 1 GB).
ALL referenced storage to an average non-system/driver/kernal program is VIRTUAL in
nature. EACH address space (process) has it's OWN 00000000-7FFFFFFF area of VIRTUAL
storage to play with (2 GB). IOW, process A has a virtual address at 0x00001000 as does
process B and process C. Those are eventually mapped (backed) by REAL RAM to allow the
CPU to execute them or the data bus to ADDRESS/access them.
WORKING SET
The working set for a process is ONLY VIRTUAL storage/memory that is backed in REAL RAM at
this instant in time. If you add up all the exe/dll storage for a process AND its data
storage, this number is usually MUCH small than the AGGREGATE (total) VIRTUAL storage in
use within the address space. IOW, within the WORD application program, if you don't use
any of the HELP or touch ANY functions other than start typing away in the default font,
the working set (ACTUAL code executed within the mass of exe and dlls) CAN BE quite small
(i.e. it WILL be small if XP is under a RAM shortage and starts trimming pages from RAM).
PAGING FILE
There is some disagreement as to whether the XP/Windows paging file will, by design, ever
hold NON-modified pages (i.e. exe and dll code). We'll discuss this later in this post as
we get to conceptual examples that will hopefully illuminate how things work AND how to
test HOW we THINK things work.
FTR, if you specify a CUSTOM-sized pagefile and then upon boot see a 1.5x RAM-sized one,
it is likely that your initial (primary) size was TOO SMALL and XP took it upon himself to
discard the CUSTOM settings. There are supposedly some Norton Anti-Virus issues related
to the CUSTOM setting of a pagefile NOT being used by XP. In my case, with NAV disabled
on boot, a too-small CUSTOM primary/minimum setting still reverted to 1.5x RAM. After
boot, about 20 MB of my pagefile is in use ALREADY (this implies XP writes some
nucleus/kernal code out there that will likely NEVER be used for this boot sequence).
This is MY experience and yours is likely to be the same.
CONCEPTUAL EXAMPLE to illustrate the above terms
Program A.exe loads from your C: drive and is 100K in size. It will reside in its own
address space (PROCESS A) and consume VIRTUAL addresses 00000000-00018FFF. Program A then
asks XP, I'd like 100MB of VIRTUAL storage in one contiguous VIRTUAL chunk. This VIRTUAL
memory allocation consumes VIRTUAL addresses from 00019000-06418FFF. NONE of this virtual
memory allocation (100MB residing on VIRTUAL addresses > 00019000) has been TOUCHED in any
way by the application. XP does NOTHING other than account for it (i.e. PROCESS A has
used 0-06418FFF VIRTUAL so the next VIRTUAL address I will hand PROCESS A will be
06419000) AND increment the PF usage (XP Task Manager) by the SAME storage amount (100M)
amount EVEN THOUGH none of it is IN REAL RAM YET!
Viewing the SYSTEM MONITOR in XP (Admin tools > Performance > System Monitor) and viewing
PROCESS (ALL FIELDS) shows for Process A:
PRIVATE BYTES (100K + 100M)
PAGE FILE BYTES (100K + 100M)
The 2 terms are synoymous save for the exception, IIRC, of the SYSTEM address space.
This newly asked-for memory is NOT written to the pagefile. The PF USAGE in TM is only
accumulating the POTENTIAL virtual memory in use by ALL processes. It's a poor-man's way
of showing YOU the user what the ACTUAL memory demands of your system are.
In the original program A.exe, ONLY code within the first 4K has been executed at this
point in time. However, the WORKING SET is STILL 100K (A.exe) because XP hasn't done any
storage trimming, etc.
The working set currently sits at 100K (A.exe). As A.exe starts TOUCHING (i.e. modifying
and/or reading) VIRTUAL addresses within its 2 Gigabyte (Gig, GB) address space, those
result in a machine exception (i.e. virtual address Q for a given address space is NOT
backed in REAL RAM) which results in XP allocating a 4K memory page in REAL RAM with the
appropriate system-level housekeeping to keep track of WHICH REAL RAM page contains
virtual address Q for process A. Let's say A.exe only touches the first 100K of this
100MB that it has asked for. Thus, the working set at this instant would be 200K (100K
for A.exe + 100K of touched/POPULATED WITH DATA virtual pages). This would show as
WORKING SET in System Monitor for Process A. Note: SM shows things in DECIMAL (unless
noted IN SM's field descriptors).
Questions to ask ourselves:
If XP loads a 100K .exe program into storage, it DOES have to page-fault 100K of REAL RAM
to do so. Is this exe/dll REAL RAM "modified" as to pagefile potential?
If a process populates 100K of data storage, to XP are BOTH equal in terms of
"modifiability"? IOW, exe code rarely changes on the fly (self-modifying code is not
likely with C/C#/C+/C++ programmers).
Storage had to be "backed in REAL RAM" to hold BOTH the exe code pages AND the then-asked
for DATA pages. IF, as I've read, exe/dll pages are NEVER paged to a pagefile (only
MODIFIED pages are ever written to the pagefile), again, aren't BOTH the exe/dll code
pages AND the DATA pages "modified" in that they were changed/populated with something
other than binary 0's?
IF XP won't page exe/dll code, it would then have to re-read the exe/dll and then re-back
in REAL RAM/page fault REAL RAM to then hold them. This also begs the question, in our
example, of the original 100K of exe code, is it BLOCK-paged (i.e. removed from RAM if
there's a storage crisis/shortage as far as XP is concerned)? I can't believe XP would
only discard REAL RAM pages, as AN example for our case, 1, 35, 100, 103, and 106 then
ONLY repopulate THOSE pages (i.e. XP would read the exe disk block/page by disk block/page
and ONLY re-back in REAL RAM pages 1,35,100,103,106). It would be FAR simpler to have
them paged to the pagefile (after all that's what it is there for - to hold MORE virtual
addresses than what RAM CAN hold). What difference does it make whether they go to the
pagefile when they're on disk already? Well, it would eat up more pagefile territory, but
exe/dll storage is quite small relative to DATA storage (typically).
I've noticed at work (XP Pro) and at home (XP Home), the % PAGE FILE IN USE grows quite
quickly at certain points in time. It almost seems that XP is PREEMPTIVELY paging when
likely YOUR workload on YOUR PC doesn't change much from boot sequence to boot. It COULD
be that this is because in 2001 when XP was released, disk drives that hold page files
were slow (5400 RPM or lower) and they wanted to stay "ahead of the RAM demand curve" by
paging older, modified, but NOT recently referenced pages to the pagefile to clear REAL
RAM for "potential" upcoming uses. This seems possible but it would seem a WASTE of time
to preemptively PREDICT what future workload an OS might experience. Completely, utterly
a waste of time. Predicting DISK ACCESS patterns for a specific file: very doable based
on "hints" from open time. Predicting when and IF you'll start MSWord seems wasteful.
I've attempted to use Sysinternals.com (Russinovich) FILEMON to trace pagefile.sys I/Os
and it's hard to catch them at those points in time when the % PF in usage DOES grow
quickly.
REAL-LIFE EXAMPLE
PF usage in Task Manager shows 250MB (VERY normal +/0 a few MB). I open a 20 MB JPG and
surprisingly see it grow to 850 MB (how a 20 MB JPG could take that much RAM for Photoshop
is beyond me). My AVAILABLE (Task Manager PHYSICAL MEMORY K) memory never dropped below
150 MB. I then opened another JPG (xxx) and resized the resolution and Photoshop just sat
there for 30 seconds. TM PF usage did NOT increase. Physical Memory AVAILABLE never
DECREASED. Filemon showed no pagefile accesses. CPU was NOT screaming away (some in
SYSTEM, little in Photoshop). Photoshop never came back so I tried to flush it using END
PROCESS in TM and it still took quite a while. When the desktop eventually was shown
again, it took FOREVER to repaint the screen (like a VERY slow TV picture paint,
resolution line by resolution line).
Again, AVAILABLE memory never dropped below 150 MB and PF usage never climbed ABOVE 850
MB. Makes NO sense whatsover.
Someone asked WHY anyone would want to play with the pagefile size. I have 2 reasons:
1) when I backup my PC using 3rd party software, it doesn't allow EXCLUDE files so a 1.5
GB pagefile.sys file takes about 1/4 of the backed up image.
2) my C: partition is only 10 GB and I don't want 1.5 GB of that eaten up by a pagefile
that's useless since THAT disk space might be needed for burning a CD or defragging the C:
partition.
If one does NOT have a pagefile, the only real drawback seems to be the 20 MB that's
parked there RIGHT after boot? If that were to remain in REAL RAM in my case, that would
be NO problem when typically I have 750 MB of REAL RAM left. If I exclude ALL that
pagefile writing when my REAL RAM is STILL above a few hundred MB, frankly I don't see ANY
downside. If I ever open a massive JPG in photoshop, not having a pagefile could be a
problem. However, in the instance when I did the real-life example above, the PF usage
went to 99.5% (100 MB) and never seemed to force an ADDITIONAL pagefile allocation. Very,
very mysterious.
This post is just for starters. I'll gladly welcome discussion (whether terse and
confrontational or otherwise). Eventually, we'll get into cache fast-write options (for
your disk drive) and how they impact the page file. As well, rarely if ever would only
ONE 4K page from REAL RAM be written to the pagefile in cases of RAM shortage/stress. A
whole track's worth would seem the most worthwhile as little additional delay would occur
(cache fast-write, etc.). This can be turned on via SYSTEM > DEVICE MANAGER > DISK DRIVES
(policies tab).
JL
regard to XP's memory management. I've also read Russinovich's Windows Internal book.
This is a long post for those so-inclined to not like to read verbose posts.
FWIW, my background comes from 25 years of machine/assembler-level coding on 4 different
hardware platforms. Memory algorithms and memory management, low-level disk I/O and
optimization (including buffering algorithms) are my forte'.
I've seen not ONE newsgroup post from anyone with specific examples. All the
newsgroup/regular web stuff I read talks about conceptuals. I'd like to move the
discussion to conceptual EXAMPLES (to start) and eventually move to real-life examples (by
discussing conceptual examples we can then hopefully agree on WHAT types of real-life
tests to perform).
I've also seen MVPs constantly point to various links/URLs which do NOT address the
specific questions being asked: NEVER. The URLs/links in question NEVER answer the 3
questions below.
1) CUSTOM page file not STICKING to what the user requests
2) WHY XP seems to not want to USE a lot of RAM (I have 1G on my machine) even when HALF
of it is still available (i.e. pagefile usage grows)
3) WHAT the actual memory management algorithms ARE (I have a hunch which we'll discuss at
the end)
***
SOME of the below is assumed to work a certain way based on my experience with the large
IBM mainframe OS (which is 1000x more complex than Windows) and has been evolving since
the 60's (NASA was a big user in the early days).
***
Gigabyte/Gig/GB and Megabyte/Meg/MB and kilobyte/KB/K are all powers of 2 (i.e. K is 1024,
MB is 1024*1024).
VIRTUAL MEMORY
Virtual memory is like having MORE memory than what your RAM (not disk, not anything else,
but RAM) will hold. The reasoning behind this is that programs use only a SUBSET of their
memory ALLOCATION request. Thus, for the 80% of the memory pages that they DON'T access
(whether DATA or CODE), those can either be (discussion later) discarded COMPLETELY out of
memory OR put onto a paging file if there's stress for RAM usage/needs.
XP (Windows, the CPU chips, however you want to slice and dice this) is designed to allow
each PROCESS to use a 4 GB address space (2 to the power 32) which is numbered in hex from
00000000 to FFFFFFFF. However, only HALF of that address space is USABLE by joe-average
application program. IOW, let's discard system-level (kernal, drivers) from the
discussion at this point. FTR, the USER accessible VIRTUAL address space area resides from
00000000-7FFFFFFF. Let's not discuss the special case of 3 GB user address spaces (of the
4 GB total when the system only gets 1 GB).
ALL referenced storage to an average non-system/driver/kernal program is VIRTUAL in
nature. EACH address space (process) has it's OWN 00000000-7FFFFFFF area of VIRTUAL
storage to play with (2 GB). IOW, process A has a virtual address at 0x00001000 as does
process B and process C. Those are eventually mapped (backed) by REAL RAM to allow the
CPU to execute them or the data bus to ADDRESS/access them.
WORKING SET
The working set for a process is ONLY VIRTUAL storage/memory that is backed in REAL RAM at
this instant in time. If you add up all the exe/dll storage for a process AND its data
storage, this number is usually MUCH small than the AGGREGATE (total) VIRTUAL storage in
use within the address space. IOW, within the WORD application program, if you don't use
any of the HELP or touch ANY functions other than start typing away in the default font,
the working set (ACTUAL code executed within the mass of exe and dlls) CAN BE quite small
(i.e. it WILL be small if XP is under a RAM shortage and starts trimming pages from RAM).
PAGING FILE
There is some disagreement as to whether the XP/Windows paging file will, by design, ever
hold NON-modified pages (i.e. exe and dll code). We'll discuss this later in this post as
we get to conceptual examples that will hopefully illuminate how things work AND how to
test HOW we THINK things work.
FTR, if you specify a CUSTOM-sized pagefile and then upon boot see a 1.5x RAM-sized one,
it is likely that your initial (primary) size was TOO SMALL and XP took it upon himself to
discard the CUSTOM settings. There are supposedly some Norton Anti-Virus issues related
to the CUSTOM setting of a pagefile NOT being used by XP. In my case, with NAV disabled
on boot, a too-small CUSTOM primary/minimum setting still reverted to 1.5x RAM. After
boot, about 20 MB of my pagefile is in use ALREADY (this implies XP writes some
nucleus/kernal code out there that will likely NEVER be used for this boot sequence).
This is MY experience and yours is likely to be the same.
CONCEPTUAL EXAMPLE to illustrate the above terms
Program A.exe loads from your C: drive and is 100K in size. It will reside in its own
address space (PROCESS A) and consume VIRTUAL addresses 00000000-00018FFF. Program A then
asks XP, I'd like 100MB of VIRTUAL storage in one contiguous VIRTUAL chunk. This VIRTUAL
memory allocation consumes VIRTUAL addresses from 00019000-06418FFF. NONE of this virtual
memory allocation (100MB residing on VIRTUAL addresses > 00019000) has been TOUCHED in any
way by the application. XP does NOTHING other than account for it (i.e. PROCESS A has
used 0-06418FFF VIRTUAL so the next VIRTUAL address I will hand PROCESS A will be
06419000) AND increment the PF usage (XP Task Manager) by the SAME storage amount (100M)
amount EVEN THOUGH none of it is IN REAL RAM YET!
Viewing the SYSTEM MONITOR in XP (Admin tools > Performance > System Monitor) and viewing
PROCESS (ALL FIELDS) shows for Process A:
PRIVATE BYTES (100K + 100M)
PAGE FILE BYTES (100K + 100M)
The 2 terms are synoymous save for the exception, IIRC, of the SYSTEM address space.
This newly asked-for memory is NOT written to the pagefile. The PF USAGE in TM is only
accumulating the POTENTIAL virtual memory in use by ALL processes. It's a poor-man's way
of showing YOU the user what the ACTUAL memory demands of your system are.
In the original program A.exe, ONLY code within the first 4K has been executed at this
point in time. However, the WORKING SET is STILL 100K (A.exe) because XP hasn't done any
storage trimming, etc.
The working set currently sits at 100K (A.exe). As A.exe starts TOUCHING (i.e. modifying
and/or reading) VIRTUAL addresses within its 2 Gigabyte (Gig, GB) address space, those
result in a machine exception (i.e. virtual address Q for a given address space is NOT
backed in REAL RAM) which results in XP allocating a 4K memory page in REAL RAM with the
appropriate system-level housekeeping to keep track of WHICH REAL RAM page contains
virtual address Q for process A. Let's say A.exe only touches the first 100K of this
100MB that it has asked for. Thus, the working set at this instant would be 200K (100K
for A.exe + 100K of touched/POPULATED WITH DATA virtual pages). This would show as
WORKING SET in System Monitor for Process A. Note: SM shows things in DECIMAL (unless
noted IN SM's field descriptors).
Questions to ask ourselves:
If XP loads a 100K .exe program into storage, it DOES have to page-fault 100K of REAL RAM
to do so. Is this exe/dll REAL RAM "modified" as to pagefile potential?
If a process populates 100K of data storage, to XP are BOTH equal in terms of
"modifiability"? IOW, exe code rarely changes on the fly (self-modifying code is not
likely with C/C#/C+/C++ programmers).
Storage had to be "backed in REAL RAM" to hold BOTH the exe code pages AND the then-asked
for DATA pages. IF, as I've read, exe/dll pages are NEVER paged to a pagefile (only
MODIFIED pages are ever written to the pagefile), again, aren't BOTH the exe/dll code
pages AND the DATA pages "modified" in that they were changed/populated with something
other than binary 0's?
IF XP won't page exe/dll code, it would then have to re-read the exe/dll and then re-back
in REAL RAM/page fault REAL RAM to then hold them. This also begs the question, in our
example, of the original 100K of exe code, is it BLOCK-paged (i.e. removed from RAM if
there's a storage crisis/shortage as far as XP is concerned)? I can't believe XP would
only discard REAL RAM pages, as AN example for our case, 1, 35, 100, 103, and 106 then
ONLY repopulate THOSE pages (i.e. XP would read the exe disk block/page by disk block/page
and ONLY re-back in REAL RAM pages 1,35,100,103,106). It would be FAR simpler to have
them paged to the pagefile (after all that's what it is there for - to hold MORE virtual
addresses than what RAM CAN hold). What difference does it make whether they go to the
pagefile when they're on disk already? Well, it would eat up more pagefile territory, but
exe/dll storage is quite small relative to DATA storage (typically).
I've noticed at work (XP Pro) and at home (XP Home), the % PAGE FILE IN USE grows quite
quickly at certain points in time. It almost seems that XP is PREEMPTIVELY paging when
likely YOUR workload on YOUR PC doesn't change much from boot sequence to boot. It COULD
be that this is because in 2001 when XP was released, disk drives that hold page files
were slow (5400 RPM or lower) and they wanted to stay "ahead of the RAM demand curve" by
paging older, modified, but NOT recently referenced pages to the pagefile to clear REAL
RAM for "potential" upcoming uses. This seems possible but it would seem a WASTE of time
to preemptively PREDICT what future workload an OS might experience. Completely, utterly
a waste of time. Predicting DISK ACCESS patterns for a specific file: very doable based
on "hints" from open time. Predicting when and IF you'll start MSWord seems wasteful.
I've attempted to use Sysinternals.com (Russinovich) FILEMON to trace pagefile.sys I/Os
and it's hard to catch them at those points in time when the % PF in usage DOES grow
quickly.
REAL-LIFE EXAMPLE
PF usage in Task Manager shows 250MB (VERY normal +/0 a few MB). I open a 20 MB JPG and
surprisingly see it grow to 850 MB (how a 20 MB JPG could take that much RAM for Photoshop
is beyond me). My AVAILABLE (Task Manager PHYSICAL MEMORY K) memory never dropped below
150 MB. I then opened another JPG (xxx) and resized the resolution and Photoshop just sat
there for 30 seconds. TM PF usage did NOT increase. Physical Memory AVAILABLE never
DECREASED. Filemon showed no pagefile accesses. CPU was NOT screaming away (some in
SYSTEM, little in Photoshop). Photoshop never came back so I tried to flush it using END
PROCESS in TM and it still took quite a while. When the desktop eventually was shown
again, it took FOREVER to repaint the screen (like a VERY slow TV picture paint,
resolution line by resolution line).
Again, AVAILABLE memory never dropped below 150 MB and PF usage never climbed ABOVE 850
MB. Makes NO sense whatsover.
Someone asked WHY anyone would want to play with the pagefile size. I have 2 reasons:
1) when I backup my PC using 3rd party software, it doesn't allow EXCLUDE files so a 1.5
GB pagefile.sys file takes about 1/4 of the backed up image.
2) my C: partition is only 10 GB and I don't want 1.5 GB of that eaten up by a pagefile
that's useless since THAT disk space might be needed for burning a CD or defragging the C:
partition.
If one does NOT have a pagefile, the only real drawback seems to be the 20 MB that's
parked there RIGHT after boot? If that were to remain in REAL RAM in my case, that would
be NO problem when typically I have 750 MB of REAL RAM left. If I exclude ALL that
pagefile writing when my REAL RAM is STILL above a few hundred MB, frankly I don't see ANY
downside. If I ever open a massive JPG in photoshop, not having a pagefile could be a
problem. However, in the instance when I did the real-life example above, the PF usage
went to 99.5% (100 MB) and never seemed to force an ADDITIONAL pagefile allocation. Very,
very mysterious.
This post is just for starters. I'll gladly welcome discussion (whether terse and
confrontational or otherwise). Eventually, we'll get into cache fast-write options (for
your disk drive) and how they impact the page file. As well, rarely if ever would only
ONE 4K page from REAL RAM be written to the pagefile in cases of RAM shortage/stress. A
whole track's worth would seem the most worthwhile as little additional delay would occur
(cache fast-write, etc.). This can be turned on via SYSTEM > DEVICE MANAGER > DISK DRIVES
(policies tab).
JL