[Xenomai-help] avoiding page faults

[Xenomai-help] avoiding page faults

[Jeff Weber]

[-- Attachment #1: Type: text/plain, Size: 870 bytes --]

I wish to avoid unwanted transitions of POSIX pthreads to secondary mode due
to page faults.

Does calling
mlockall(MCL_CURRENT | MCL_FUTURE)

lock pages when the memory is allocated, or referenced?

>From reviewing Xenomai, Linux RT_PREEMPT, and RTAI code, it appears the
answer is that memory is locked when the page is "referenced" (not
allocated), and that referencing by write is preferred over referencing by
read.  xeno_fault_stack() appears to pre-fault a subset of the stack when
turning the calling thread into a Xenomai thread.

If true, then I should allocate my own stack for each pthread created in my
program, and write every page ahead of time, then
call pthread_attr_setstack() prior to thread creation.  I should also write
every page of every dynamically allocated memory object.

Please help me verify if this understanding is correct.

thanks,
Jeff

[-- Attachment #2: Type: text/html, Size: 1077 bytes --]

Re: [Xenomai-help] avoiding page faults

[Gilles Chanteperdrix]

Jeff Weber wrote:
> I wish to avoid unwanted transitions of POSIX pthreads to secondary mode due
> to page faults.
> 
> Does calling
> mlockall(MCL_CURRENT | MCL_FUTURE)
> 
> lock pages when the memory is allocated, or referenced?
> 
> From reviewing Xenomai, Linux RT_PREEMPT, and RTAI code, it appears the
> answer is that memory is locked when the page is "referenced" (not
> allocated), and that referencing by write is preferred over referencing by
> read.  xeno_fault_stack() appears to pre-fault a subset of the stack when
> turning the calling thread into a Xenomai thread.
> 
> If true, then I should allocate my own stack for each pthread created in my
> program, and write every page ahead of time, then
> call pthread_attr_setstack() prior to thread creation.  I should also write
> every page of every dynamically allocated memory object.
> 
> Please help me verify if this understanding is correct.

mlockall(MCL_CURRENT) locks all the memory currently mapped.
mlockall(MCL_FUTURE) will lock all the memory when it is allocated/mapped.

The only thing which remains allocated on demand is the main thread
stack. This explains why we use xeno_fault_stack(), which faults a
"reasonable" amount of stack, in order to get the kernel to allocate and
map memory.

-- 
					    Gilles.

Re: [Xenomai-help] avoiding page faults

[Jeff Weber]

[-- Attachment #1: Type: text/plain, Size: 3643 bytes --]

On Thu, Apr 14, 2011 at 7:18 AM, Gilles Chanteperdrix <
gilles.chanteperdrix@xenomai.org> wrote:

> Jeff Weber wrote:
> > I wish to avoid unwanted transitions of POSIX pthreads to secondary mode
> due
> > to page faults.
> >
> > Does calling
> > mlockall(MCL_CURRENT | MCL_FUTURE)
> >
> > lock pages when the memory is allocated, or referenced?
> >
> > From reviewing Xenomai, Linux RT_PREEMPT, and RTAI code, it appears the
> > answer is that memory is locked when the page is "referenced" (not
> > allocated), and that referencing by write is preferred over referencing
> by
> > read.  xeno_fault_stack() appears to pre-fault a subset of the stack when
> > turning the calling thread into a Xenomai thread.
> >
> > If true, then I should allocate my own stack for each pthread created in
> my
> > program, and write every page ahead of time, then
> > call pthread_attr_setstack() prior to thread creation.  I should also
> write
> > every page of every dynamically allocated memory object.
> >
> > Please help me verify if this understanding is correct.
>
> mlockall(MCL_CURRENT) locks all the memory currently mapped.
> mlockall(MCL_FUTURE) will lock all the memory when it is allocated/mapped.
>
> Here's an strace of my pthread library starting up a thread, which will
become a Xenomai posix thread:
...
25210 mlockall(MCL_CURRENT|MCL_FUTURE)  = 0
25210 open("/dev/null", O_RDWR)         = 3
25210 open("/dev/null", O_RDWR)         = 4
25210 sched_get_priority_max(SCHED_FIFO) = 99
25210 sched_get_priority_min(SCHED_FIFO) = 1
25210 sched_get_priority_max(SCHED_FIFO) = 99
25210 mmap2(NULL, 8392704, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS|MAP_S
TACK, -1, 0) = 0xb6dd7000
25210 mprotect(0xb6dd7000, 4096, PROT_NONE) = 0
25210 clone(child_stack=0xb75d7474,
flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SI
GHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CL
EARTID, parent_tidptr=0xb75d7bd8, {entry_number:6, base_addr:0xb75d7b70,
limit:1
048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1,
seg_not_pre
sent:0, useable:1}, child_tidptr=0xb75d7bd8) = 25212
...

The child thread stack is allocated/mapped by the mmap2() call above, in the
parent thread.  Since mlock(MCL_FUTURE) was called prior, the child stack
should be completely locked into parent memory.  However, it is not clear if
the stack memory locks are lost in the new child thread across the
clone().  The mlock() man page says these are definitely lost across a
fork().  The clone() CLONE_VM flag indicates the the parent and child thread
share the same memory mappings. I'm not a Linux virtual memory wizard.  Does
this also imply parent memory locks are inherited by a cloned child?

My mlock() man page has this:
       Real-time processes that are using mlockall() to prevent delays on
page
       faults should reserve enough locked stack  pages  before  entering
 the
       time-critical  section, so that no page fault can be caused by
function
       calls.  This can be achieved by calling a  function  that  allocates
 a
       sufficiently large automatic variable (an array) and writes to the
mem-
       ory occupied by this array in order to touch these stack  pages.
This
       way,  enough  pages will be mapped for the stack and can be locked
into
       RAM.  The dummy writes ensure that not even copy-on-write  page
 faults
       can occur in the critical section.

So does calling mlockall(MCL_FUTURE) early in the parent permanently lock
memory for all subsequent threads, or do I need to touch all pages in all
child threads to guarantee they are locked?

My architecture is x86 32-bit.

TIA,
Jeff

[-- Attachment #2: Type: text/html, Size: 4597 bytes --]

Re: [Xenomai-help] avoiding page faults

[Gilles Chanteperdrix]

Jeff Weber wrote:
> On Thu, Apr 14, 2011 at 7:18 AM, Gilles Chanteperdrix <
> gilles.chanteperdrix@xenomai.org> wrote:
> 
>> Jeff Weber wrote:
>>> I wish to avoid unwanted transitions of POSIX pthreads to secondary mode
>> due
>>> to page faults.
>>>
>>> Does calling
>>> mlockall(MCL_CURRENT | MCL_FUTURE)
>>>
>>> lock pages when the memory is allocated, or referenced?
>>>
>>> From reviewing Xenomai, Linux RT_PREEMPT, and RTAI code, it appears the
>>> answer is that memory is locked when the page is "referenced" (not
>>> allocated), and that referencing by write is preferred over referencing
>> by
>>> read.  xeno_fault_stack() appears to pre-fault a subset of the stack when
>>> turning the calling thread into a Xenomai thread.
>>>
>>> If true, then I should allocate my own stack for each pthread created in
>> my
>>> program, and write every page ahead of time, then
>>> call pthread_attr_setstack() prior to thread creation.  I should also
>> write
>>> every page of every dynamically allocated memory object.
>>>
>>> Please help me verify if this understanding is correct.
>> mlockall(MCL_CURRENT) locks all the memory currently mapped.
>> mlockall(MCL_FUTURE) will lock all the memory when it is allocated/mapped.
>>
>> Here's an strace of my pthread library starting up a thread, which will
> become a Xenomai posix thread:
> ...
> 25210 mlockall(MCL_CURRENT|MCL_FUTURE)  = 0
> 25210 open("/dev/null", O_RDWR)         = 3
> 25210 open("/dev/null", O_RDWR)         = 4
> 25210 sched_get_priority_max(SCHED_FIFO) = 99
> 25210 sched_get_priority_min(SCHED_FIFO) = 1
> 25210 sched_get_priority_max(SCHED_FIFO) = 99
> 25210 mmap2(NULL, 8392704, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_ANONYMOUS|MAP_S
> TACK, -1, 0) = 0xb6dd7000
> 25210 mprotect(0xb6dd7000, 4096, PROT_NONE) = 0
> 25210 clone(child_stack=0xb75d7474,
> flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SI
> GHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CL
> EARTID, parent_tidptr=0xb75d7bd8, {entry_number:6, base_addr:0xb75d7b70,
> limit:1
> 048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1,
> seg_not_pre
> sent:0, useable:1}, child_tidptr=0xb75d7bd8) = 25212
> ...
> 
> The child thread stack is allocated/mapped by the mmap2() call above, in the
> parent thread.  Since mlock(MCL_FUTURE) was called prior, the child stack
> should be completely locked into parent memory.  However, it is not clear if
> the stack memory locks are lost in the new child thread across the
> clone().  The mlock() man page says these are definitely lost across a
> fork().  The clone() CLONE_VM flag indicates the the parent and child thread
> share the same memory mappings. I'm not a Linux virtual memory wizard.  Does
> this also imply parent memory locks are inherited by a cloned child?

Yes, it does. As I said, from what I have seen in the systems where I
run Xenomai until now. The only stack which is an issue is the main
thread stack.

-- 
					    Gilles.