File Mapping

[Doug G]

What are the ramifications of using Win32 File Mapping on XPE with
disk-based EWF? I am not concerned about saving the data between bootups at
all -- it would just be used for data sharing among multiple processes.
However, I don't want the data to actually be paged out to the disk since
I'm trying to avoid disk activity (the disk should normally stay spun down
via power savings settings). I would probably just create the mapped section
directly in the system page file.

Am I right in that, if nothing else causes a need for paging, the mapped
section will just be updated in RAM without the changes being flushed out to
the disk? IIRC from the theory of virtual memory operations, paging should
only occur if the physical memory used by the section has to be swapped for
some other set of pages, and in our app with plenty of RAM this should never
happen. Anyone say otherwise?

Doug

 

[Slobodan Brcin \(eMVP\)]

First I would read and use:
http://msdn.microsoft.com/embedded/community/tips/xp/renamert/default.aspx

Then file mapping (if you don't write anything to disk won't affect EWF).
If you are writing to disk then it will use EWF.

You are using file mapping in memory only, right?
So you should move page file to unprotected partition and you will be ok
regarding EWF but your problem remains, so:

1. You have plenty of RAM solution would be to disable page file, and to use
RAM EWF. This will guaranty you that HDD will be accessed only for read
requests not for write.
2. You can always use approach similar to VirtualLock function that will
guarantee to you that this memory will be never swapped.

Regards,
Slobodan

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Have an opinion on the effectiveness of Microsoft Embedded newsgroups? Tell
Microsoft!
https://www.windowsembeddedeval.com/community/newsgroups
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 

[KM]

Just to add to Slobodan's detail message...
Sometimes it may be a good idea to call SetProcessWorkingSetSize API in conjunction with VirtualLock to increase the upper and lower
limits of the particular process's working set. VirtualLock() only locks pages into the process working set, it does not lock them
absolutely into physical memory. The API may fail it the working set is not enough to fit your app memory requests.

 

[Slobodan Brcin \(eMVP\)]

Hi Konstantin,

I said similar approach to VirtualLock, so I have not given any details
(VirtualLock is described very well in SDK).
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/memory/base
/virtuallock.asp

Sometimes it may be a good idea to call SetProcessWorkingSetSize API in

conjunction with VirtualLock to increase the upper and lower

limits of the particular process's working set.

I agree that this is what should be done. (Unless he will use user account
or small amount of memory like 4096 bytes.)

VirtualLock() only locks pages into the process working set, it does not lock them
absolutely into physical memory.

Are you sure? If you are, then my English is much worse than I thought :'(

Quote:
The VirtualLock function locks the specified region of the process's virtual
address space into physical memory, ensuring that subsequent access to the
region will not incur a page fault.

Pages that a process has locked remain in physical memory until the process
unlocks them or terminates.

What do you think?
Slobodan

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Have an opinion on the effectiveness of Microsoft Embedded newsgroups? Tell
Microsoft!
https://www.windowsembeddedeval.com/community/newsgroups
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Just to add to Slobodan's detail message...

VirtualLock() only locks pages into the process working set, it does not
lock them

absolutely into physical memory. The API may fail it the working set is

not enough to fit your app memory requests.

 

[KM]

Slobodan,

To be frankly, there is an information on VirtualLock in MSDN that is a bit conflicting to another MSDN articles.
In the VirtualLock API description it says that the pages are *absolutely* locked into physical memory.
However, in the Article Q94996 (it was written for NT 3.1-4.0 but my guess it should still apply to 2000/XP as the API didn't change
much) it says the following:
-----------------------------------------------------------------------------

SUMMARY
VirtualLock() causes pages to be locked into an application's working set (virtual memory); it does not lock them absolutely into
physical memory. VirtualLock() essentially means "this page is always part of the process's working set."

The system is free to swap out any virtually locked pages if it swaps out the whole process. And when the system swaps the process
back in, the virtually locked pages (similar to any virtual pages) may end up residing in different real pages.

It is wise to use VirtualLock() very sparingly because it reduces the flexibility of the system.

Depending upon memory demands on the system, the memory manager may vary the number of pages a process can lock. Under typical
conditions you can expect to be able to VirtualLock() approximately 28 to 32 pages.

In Windows NT 3.5, you can use SetProcessWorkingSetSize() to increase the size of the working set, and therefore increase the number
of pages that VirtualLock() can lock.

-----------------------------------------------------------------------------


I failed to find the article on MSDN MS website but I still have VC 6.0 old MSDN archive saved locally (sometimes it helps ).
Probably, the answer would be in experimenting with the API: allocating a big chunk of memory, loading up the system with various
apps and querying the number of pages committed.
Or MS which have the access to the source code of VirtualLock may easy answer the question.

 

[Slobodan Brcin \(eMVP\)]

Hi Konstantin,

This is creepy. But as I understood from link below it that it will only
happen if whole process is paged out. (This will never happen). And that
would be critical if we really used physical memory for let say DMA
transfers.

http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&threadm=Xns92065
1D88F248JochenKalmbachholzma%40127.0.0.1&rnum=1&prev=/groups%3Fq%3DQ94996%26
hl%3Den%26lr%3D%26ie%3DUTF-8%26oe%3DUTF-8%26selm%3DXns920651D88F248JochenKal
mbachholzma%2540127.0.0.1%26rnum%3D1

Regards,
Slobodan

Slobodan,

To be frankly, there is an information on VirtualLock in MSDN that is a

bit conflicting to another MSDN articles.

In the VirtualLock API description it says that the pages are *absolutely* locked into physical memory.
However, in the Article Q94996 (it was written for NT 3.1-4.0 but my guess

it should still apply to 2000/XP as the API didn't change

much) it says the following:
-------------------------------------------------------------------------- ---

SUMMARY
VirtualLock() causes pages to be locked into an application's working set

(virtual memory); it does not lock them absolutely into

physical memory. VirtualLock() essentially means "this page is always part of the process's working set."

The system is free to swap out any virtually locked pages if it swaps out

the whole process. And when the system swaps the process

back in, the virtually locked pages (similar to any virtual pages) may end

up residing in different real pages.

It is wise to use VirtualLock() very sparingly because it reduces the flexibility of the system.

Depending upon memory demands on the system, the memory manager may vary

the number of pages a process can lock. Under typical

conditions you can expect to be able to VirtualLock() approximately 28 to 32 pages.

In Windows NT 3.5, you can use SetProcessWorkingSetSize() to increase the

size of the working set, and therefore increase the number

of pages that VirtualLock() can lock.

-------------------------------------------------------------------------- ---


I failed to find the article on MSDN MS website but I still have VC 6.0

old MSDN archive saved locally (sometimes it helps ).

Probably, the answer would be in experimenting with the API: allocating a

big chunk of memory, loading up the system with various

 

[KM]

Slobodan,

Yup, that thread is confirming (and mentioning) the article.
The article says : "The system is free to swap out any virtually locked pages if it swaps out the whole process. And when the system
swaps the process back in, the virtually locked pages (similar to any virtual pages) may end up residing in different real pages.".
The last statement is really creepy (agree, critical for DMA transfers).

Anyway.. they pointed out the solution we are discussing here:
Q: How can I lock a page in memory, to prevent it from being swapped out? How can I apply this to my programs to lock the contents
of the program in
memory?
A: 1) Set the Page File size to 0; or
2) use the following functions: VirtualQuery, VirtualLock, GetProcessWorkingSetSize and SetProcessWorkingSetSize.

Btw, MS seem to have removed the article from MSDN

 

[Slobodan Brcin \(eMVP\)]

Konstantin,

Well my DMA example was stupid since we would do it from kernel mode and
there we have functions that behave well.

Considering this function.
MS might removed that article since.
a) they solved the problem in all OS-es starting with some service pack for
NT.
b) that was only theoretical problem that was never and issue.

Who knows.

But not having page file would be the best solution in any case.

Regards,
Slobodan

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Have an opinion on the effectiveness of Microsoft Embedded newsgroups? Tell
Microsoft!
https://www.windowsembeddedeval.com/community/newsgroups
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Slobodan,

Yup, that thread is confirming (and mentioning) the article.
The article says : "The system is free to swap out any virtually locked

pages if it swaps out the whole process. And when the system

swaps the process back in, the virtually locked pages (similar to any

virtual pages) may end up residing in different real pages.".

The last statement is really creepy (agree, critical for DMA transfers).

Anyway.. they pointed out the solution we are discussing here:
Q: How can I lock a page in memory, to prevent it from being swapped out?

How can I apply this to my programs to lock the contents

of the program in
memory?
A: 1) Set the Page File size to 0; or
2) use the following functions: VirtualQuery, VirtualLock,

GetProcessWorkingSetSize and SetProcessWorkingSetSize.

Btw, MS seem to have removed the article from MSDN

--
KM,
BSquare Corporation

Hi Konstantin,

This is creepy. But as I understood from link below it that it will only
happen if whole process is paged out. (This will never happen). And that
would be critical if we really used physical memory for let say DMA
transfers.

http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&threadm=Xns92065
1D88F248JochenKalmbachholzma%40127.0.0.1&rnum=1&prev=/groups%3Fq%3DQ94996%26
hl%3Den%26lr%3D%26ie%3DUTF-8%26oe%3DUTF-8%26selm%3DXns920651D88F248JochenKal
mbachholzma%2540127.0.0.1%26rnum%3D1

Regards,
Slobodan

Slobodan,

To be frankly, there is an information on VirtualLock in MSDN that is

a
bit conflicting to another MSDN articles.

In the VirtualLock API description it says that the pages are

*absolutely*
locked into physical memory.

However, in the Article Q94996 (it was written for NT 3.1-4.0 but my

guess
it should still apply to 2000/XP as the API didn't change

much) it says the following:

-------------------------------------------------------------------------- set
(virtual memory); it does not lock them absolutely into

physical memory. VirtualLock() essentially means "this page is always

part
of the process's working set."

The system is free to swap out any virtually locked pages if it swaps

out
the whole process. And when the system swaps the process

back in, the virtually locked pages (similar to any virtual pages) may

end
up residing in different real pages.

It is wise to use VirtualLock() very sparingly because it reduces the flexibility of the system.

Depending upon memory demands on the system, the memory manager may

vary
the number of pages a process can lock. Under typical

conditions you can expect to be able to VirtualLock() approximately 28

to
32 pages.

In Windows NT 3.5, you can use SetProcessWorkingSetSize() to increase

the
size of the working set, and therefore increase the number

of pages that VirtualLock() can lock.

-------------------------------------------------------------------------- 6.0
old MSDN archive saved locally (sometimes it helps ).

Probably, the answer would be in experimenting with the API:

allocating a
big chunk of memory, loading up the system with various

apps and querying the number of pages committed.
Or MS which have the access to the source code of VirtualLock may easy answer the question.

--
KM,
BSquare Corporation

Hi Konstantin,

I said similar approach to VirtualLock, so I have not given any details
(VirtualLock is described very well in SDK).

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/memory/base

/virtuallock.asp

Sometimes it may be a good idea to call SetProcessWorkingSetSize

API
in

conjunction with VirtualLock to increase the upper and lower
limits of the particular process's working set.

I agree that this is what should be done. (Unless he will use user account
or small amount of memory like 4096 bytes.)

VirtualLock() only locks pages into the process working set, it

does
not

lock them
absolutely into physical memory.

Are you sure? If you are, then my English is much worse than I

thought
:'(

Quote:
The VirtualLock function locks the specified region of the process's virtual
address space into physical memory, ensuring that subsequent access

to
the

region will not incur a page fault.

Pages that a process has locked remain in physical memory until the process
unlocks them or terminates.

What do you think?
Slobodan

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Have an opinion on the effectiveness of Microsoft Embedded

newsgroups?
Tell

Microsoft!
https://www.windowsembeddedeval.com/community/newsgroups
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Just to add to Slobodan's detail message...
VirtualLock() only locks pages into the process working set, it

does
not

lock them
absolutely into physical memory. The API may fail it the working

set
is

not enough to fit your app memory requests.

--
KM,
BSquare Corporation

First I would read and use:

http://msdn.microsoft.com/embedded/community/tips/xp/renamert/default.aspx

Then file mapping (if you don't write anything to disk won't affect
EWF).
If you are writing to disk then it will use EWF.

You are using file mapping in memory only, right?
So you should move page file to unprotected partition and you

will
be ok

regarding EWF but your problem remains, so:

1. You have plenty of RAM solution would be to disable page

file,
and to

use
RAM EWF. This will guaranty you that HDD will be accessed only

for
read

requests not for write.
2. You can always use approach similar to VirtualLock function

that
will

guarantee to you that this memory will be never swapped.

Regards,
Slobodan

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Have an opinion on the effectiveness of Microsoft Embedded newsgroups?
Tell
Microsoft!
https://www.windowsembeddedeval.com/community/newsgroups
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

What are the ramifications of using Win32 File Mapping on XPE with
disk-based EWF? I am not concerned about saving the data between
bootups
at
all -- it would just be used for data sharing among multiple
processes.
However, I don't want the data to actually be paged out to the disk
since
I'm trying to avoid disk activity (the disk should normally

stay
spun

down
via power savings settings). I would probably just create the mapped
section
directly in the system page file.

Am I right in that, if nothing else causes a need for paging, the
mapped
section will just be updated in RAM without the changes being flushed
out
to
the disk? IIRC from the theory of virtual memory operations, paging
should
only occur if the physical memory used by the section has to be
swapped
for
some other set of pages, and in our app with plenty of RAM

this
should

never
happen. Anyone say otherwise?

Doug

 

[KM]

Slobodan,

I don't think MS has changed VMM much since NT. Although the precise behaviour of VirtualLock() on NT/2K/XP isn't known, it appears
that it acts as a form of advisory lock which tells the operating system to keep the pages resident for as long as possible before
swapping them out.

This may be critical to high secure applications that wants to prevent sensitive material such as private keys being written to
disk.

Although I found an interesting thread about the problem on Internet that shows that MS has tested VirtualLock enough to trust:
"Actually not trusting VirtualLock may no longer be totally accurate. I had a long discussion about this with some MS technical
people and it appears that (at least in Win2K and newer) VirtualLock is more effective than expected. One of the MS guys tried
everything possible to get VirtualLocked'd memory paged and couldn't get it swapped out. Unfortunately the info for different
Windows versions and at different times has been conflicting, but it appears that with newer versions of Windows it has at least
some positive effect.".
I still think there is a possibility if a process has no nonpageable kernel stacks, it "outswaps" the process (if paging activity
high or memory needed).
Check out this doc: http://www.i.u-tokyo.ac.jp/ss/lecture/new-documents/Lectures/02-VirtualMemory/VirtualMemory.pdf

 

[Doug G]

Slobodan:
I am clear on how virtual memory works as far as issues like locking pages
into a workings set vs. locking a working set in memory. I am sure that no
process swapping will be going on!

What I am not clear on is what moving the page file to a non-EWF partition
vs. EWF would have to do with my issue. For various reasons, I have to use
disk-based EWF, by the way. This file mapping is only for inter-process
memory sharing, so nothing has to be saved between bootups. So I am
wondering, when using a memory mapped "file", under what circumstances a
physical write to the HDD would take place.

Writing to a memory-mapped file is not like using "fputs" where you are
actually trying to update the disk and your write eventually gets flushed
out to the HDD. If I have a pointer to the mapped region and copy some data
there, that's not going to cause a disk update, correct? I would assume that
a HDD update for a mapped file would only occur if normal system paging or
swapping takes place; if the section is unmapped (explicitly or implicitly);
when the system does a soft shutdown; etc. So I think that using VirtualLock
in the sharing processes that are not ever swapped should do the trick.

Anyone disagree with this? I might whip up a simple test (famous last words)
to try this out before making a commitment to my design.

Doug Gordon

 

[Slobodan Brcin \(eMVP\)]

Hi Doug,

I am sure that no process swapping will be going on!

Of course you are since this process will probably consume 99% of your
device work then it can't be paged out.

What I am not clear on is what moving the page file to a non-EWF partition
vs. EWF would have to do with my issue.

It has nothing to do with your issue, but it has everything with your
concept.
Are you sure that you need virtual memory?

is file mapping is only for inter-process
memory sharing, so nothing has to be saved between bootups. So I am
wondering, when using a memory mapped "file", under what circumstances a
physical write to the HDD would take place.

I'm using file mapping for interposes communication as well, but there are
two different approaches how you can use this function as you probably know.
1. Use shared memory.
2. Use shared file.

So you should use 1, and under no circumstances paging to disk will occur as
long as system do not run out of memory.

But like Konstantin said this question is not for this NG you should ask it
in SDK/DDK related groups.

Regards,
Slobodan

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Have an opinion on the effectiveness of Microsoft Embedded newsgroups? Tell
Microsoft!
https://www.windowsembeddedeval.com/community/newsgroups
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 

[Doug G]

FWIW, I tried my idea and it works -- I can freely access the shared memory
and the HDD stays spun down. Also, in Win32, "shared memory" really is a
"shared file", and it's the only easy way that totally independent processes
can have direct access to a shared piece of memory.

Here's all it takes in the code for single-process access (for multiple
processes it's a little more complex due to security descriptor issues):

hMap = CreateFileMapping(INVALID_HANDLE_VALUE, NULL, PAGE_READWRITE, 0,
1024, "Global\\TestMap");
if (!hMap) {
printf("CreateFileMapping failed\n");
return(GetLastError());
}

pMap = (unsigned char *) MapViewOfFile(hMap, FILE_ALL_ACCESS, 0, 0, 1024);
if (!pMap) {
printf("MapViewOfFile failed\n");
return(GetLastError());
}

At this point, "pMap" points to a 2K block of memory that can be shared with
another process. No file is actually created since I passed
INVALID_HANDLE_VALUE in the first call.

Doug

 

[KM]

Doug,

I guess Slobodan was referring to "CreateFileMapping(INVALID_HANDLE_VALUE,..." as a "shared memory". The function creates a
file-mapping object of the specified size backed by the operating-system paging file. So, calling to VirtualLock here is still a
good idea (regardless how trustable the VirtualLock API is).

The "shared file" would be "hFile=CreateFile --> CreateFileMapping(hFile,...) .... --> ". Her eyou will need to call FlushViewOfFile
to flush a mapped view dirty pages. However, this is not your case.

 

[Slobodan Brcin \(eMVP\)]

Yep.

Doug,

I guess Slobodan was referring to

"CreateFileMapping(INVALID_HANDLE_VALUE,..." as a "shared memory". The
function creates a

file-mapping object of the specified size backed by the operating-system

paging file. So, calling to VirtualLock here is still a

good idea (regardless how trustable the VirtualLock API is).

The "shared file" would be "hFile=CreateFile -->

CreateFileMapping(hFile,...) .... --> ". Her eyou will need to call
FlushViewOfFile