[Xenomai-help] Switch to secondary mode

[Xenomai-help] Switch to secondary mode

[Johan Borkhuis]

Hello,

I am having some problems with an application switching to secondary 
mode. One of my applications is switching to secondary mode when a 
function is called in a library. This function itself is empty (only 
returning 0).
When I move this function out of the library into the main C-file the 
program stays in primary mode. What could be happening here?

This is the output of a backtrace:
./dsAuxEvQueTest1[0x10001858]
/lib/ld.so.1[0x3001eaf8]
[0x0]
./dsAuxEvQueTest1[0x100018f0]
./dsAuxEvQueTest1[0x10005c9c]
/lib/libpthread.so.0[0xffd32fc]
/lib/libc.so.6(__clone+0x84)[0xfe751e8]

Do I need to use some specific compile or link commandline arguments?

My architecture is ppc, Xenomai version 2.3.1 and kernel 2.6.14.

Kind regards,
    Johan Borkhuis

Re: [Xenomai-help] Switch to secondary mode

[Philippe Gerum]

On Mon, 2007-07-02 at 13:25 +0200, Johan Borkhuis wrote:
> Hello,
> 
> I am having some problems with an application switching to secondary 
> mode. One of my applications is switching to secondary mode when a 
> function is called in a library. This function itself is empty (only 
> returning 0).
> When I move this function out of the library into the main C-file the 
> program stays in primary mode. What could be happening here?
> 

Late binding to functions performed on behalf of the dynamic loader
against shared libraries shall need the kernel during symbol resolution
(internal syscalls) or execution (e.g. demand loading, COW), hence the
switch. Unfortunately, the I-pipe patch for PPC does not support
disabling all on-demand memory mappings for selected Linux tasks (only
the x86 and ARM patches support this feature so far).

> This is the output of a backtrace:
> ./dsAuxEvQueTest1[0x10001858]
> /lib/ld.so.1[0x3001eaf8]
> [0x0]
> ./dsAuxEvQueTest1[0x100018f0]
> ./dsAuxEvQueTest1[0x10005c9c]
> /lib/libpthread.so.0[0xffd32fc]
> /lib/libc.so.6(__clone+0x84)[0xfe751e8]
> 
> Do I need to use some specific compile or link commandline arguments?
> 
> My architecture is ppc, Xenomai version 2.3.1 and kernel 2.6.14.
> 
> Kind regards,
>     Johan Borkhuis
> 
> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help
-- 
Philippe.

Re: [Xenomai-help] Switch to secondary mode

[Johan Borkhuis]

Philippe,

Philippe Gerum wrote:
> Late binding to functions performed on behalf of the dynamic loader
> against shared libraries shall need the kernel during symbol resolution
> (internal syscalls) or execution (e.g. demand loading, COW), hence the
> switch. Unfortunately, the I-pipe patch for PPC does not support
> disabling all on-demand memory mappings for selected Linux tasks (only
> the x86 and ARM patches support this feature so far).
>   

Thank you for you answer.

Just for me to make sure I understand this correctly:
We are not using shared libraries for our application, our applications 
are linked against .a files, which are included in the final application 
(all symbols are resolved and available in the executable, so I expected 
that all symbols would have been resolved. This does apply to that as 
well to this type of applications?
(I tried linking using -static,  but that does still give the same problem.)

How can I avoid this problem? Is there a way to make sure all symbols 
are available in memory, or is there a way to instruct the 
compiler/linker/loader to perform the symbol-resolving during 
compile-time or on load?
One thing I am thinking of (but this may be completely wrong) is to have 
a Init function in all modules that are part of the library. When a 
library module is used, this Init function needs to be called before 
switching to RT-mode, so the symbols are made available to the 
application. Would this work (and solve the problem), or would this 
create a lot of other problems?

Where can I find some more information on this issue?

Kind regards,
    Johan Borkhuis

Re: [Xenomai-help] Switch to secondary mode

[Johan Borkhuis]

One more question.

I tried finding the cause of the switch using the backtrace commands, 
but this does not give a very clear picture of what is causing the 
switch. Are there other commands that give a better view, or is there 
some instruction (or just a direction) on how to interpret the backtrace?

Kind regards,
    Johan Borkhuis

Re: [Xenomai-help] Switch to secondary mode

[Philippe Gerum]

On Mon, 2007-07-02 at 16:18 +0200, Johan Borkhuis wrote:
> Philippe,
> 
> Philippe Gerum wrote:
> > Late binding to functions performed on behalf of the dynamic loader
> > against shared libraries shall need the kernel during symbol resolution
> > (internal syscalls) or execution (e.g. demand loading, COW), hence the
> > switch. Unfortunately, the I-pipe patch for PPC does not support
> > disabling all on-demand memory mappings for selected Linux tasks (only
> > the x86 and ARM patches support this feature so far).
> >   
> 
> Thank you for you answer.
> 
> Just for me to make sure I understand this correctly:
> We are not using shared libraries for our application, our applications 
> are linked against .a files, which are included in the final application

In such a case, you have likely hit an illustration of the latter issue
which the I-pipe/ppc implementation still suffers from: some page table
entries are missed during real-time operations. As a consequence of
this, the nucleus catches page faults on behalf of RT threads in primary
mode, then switches these threads back to secondary in order to process
the faults, and eventually wire the missing PTEs in. This is something
calling mlockall() does not prevent the application from (like COW).

The shared lib problem would be another issue, even if it relates to the
same general topic (i.e. lazy/on-demand mapping of memory resources the
kernel performs).

>  
> (all symbols are resolved and available in the executable, so I expected 
> that all symbols would have been resolved. This does apply to that as 
> well to this type of applications?

> (I tried linking using -static,  but that does still give the s
> ame problem.)
> 

That is expected. If you switch the nucleus debug option on, you should
see Xenomai whining about secondary mode switches from code locations in
kernel space. This would confirm the fact that you have been hitting
this problem.

> How can I avoid this problem?

There is not much to be done except improving the I-pipe/ppc support so
that it provides a way to pin down any PTE an application might refer
to. There might be other related issues beyond this one though.
Fortunately, mode transitions for dealing for such faults normally don't
lead to bad latencies on this arch. Do you confirm that, or are you
unlucky regarding this?

>  Is there a way to make sure all symbols 
> are available in memory, or is there a way to instruct the 
> compiler/linker/loader to perform the symbol-resolving during 
> compile-time or on load?
> One thing I am thinking of (but this may be completely wrong) is to have 
> a Init function in all modules that are part of the library. When a 
> library module is used, this Init function needs to be called before 
> switching to RT-mode, so the symbols are made available to the 
> application. Would this work (and solve the problem), or would this 
> create a lot of other problems?
> 

The answer is not at user-space level, unfortunately.

> Where can I find some more information on this issue?
> 
> Kind regards,
>     Johan Borkhuis
-- 
Philippe.

Re: [Xenomai-help] Switch to secondary mode

[Johan Borkhuis]

Philippe,

Again, thank you for very extensive answer. It is starting to get 
clearer what is happening here but I still have some questions.

The first one is probably an easy one: what do you mean by COW? I did 
try to find what this means, but I was only able to find references to 
farm animals..... (and some references to lazy linking etc. but there 
COW was also not explained).

Philippe Gerum wrote:
> On Mon, 2007-07-02 at 16:18 +0200, Johan Borkhuis wrote:
>   
>> Philippe,
>>
>> Philippe Gerum wrote:
>>     
>>> Late binding to functions performed on behalf of the dynamic loader
>>> against shared libraries shall need the kernel during symbol resolution
>>> (internal syscalls) or execution (e.g. demand loading, COW), hence the
>>> switch. Unfortunately, the I-pipe patch for PPC does not support
>>> disabling all on-demand memory mappings for selected Linux tasks (only
>>> the x86 and ARM patches support this feature so far).
>>>   
>>>       
>> Thank you for you answer.
>>
>> Just for me to make sure I understand this correctly:
>> We are not using shared libraries for our application, our applications 
>> are linked against .a files, which are included in the final application
>>     
>
> In such a case, you have likely hit an illustration of the latter issue
> which the I-pipe/ppc implementation still suffers from: some page table
> entries are missed during real-time operations. As a consequence of
> this, the nucleus catches page faults on behalf of RT threads in primary
> mode, then switches these threads back to secondary in order to process
> the faults, and eventually wire the missing PTEs in. This is something
> calling mlockall() does not prevent the application from (like COW).
>   
As some PTE's are missed, does this mean that not the complete program 
was loaded into memory?

What I understand until now about this process is the following:
The program is executed. Not everything is loaded into memory by the 
dynamic loader. As the functions that are not in memory are accessed a 
page fault is created, and the page is made available. (Or is the page 
already in memory, but not made available to the application?)
Is my assumption correct that once a page is accessed mlockall will take 
care that this page stays active, or is it possible that the page is 
moved out, and that another page fault occurs for the same page?

> .....
>   
> That is expected. If you switch the nucleus debug option on, you should
> see Xenomai whining about secondary mode switches from code locations in
> kernel space. This would confirm the fact that you have been hitting
> this problem.
>   

When looking at the nucleus debug output, I see a number of switches 
coming from user space, like this one:

Jul  3 06:51:10 MVME3100-198 kernel: Xenomai: Switching testTask to 
secondary mode after exception #1025 from user-space at 0x10005f2c (pid 
1069)

I did try to find what exception 1025 is, but I could not find this 
reference. I expect indeed that this is a page-fault, but I am not sure.
The location 0x10005f2c is the start of a function in one of the 
statically linked objects from one of our archives. You are referring to 
kernel space, but these exceptions are generated from user space. Is 
this different from what you are referring to?
> There is not much to be done except improving the I-pipe/ppc support so
> that it provides a way to pin down any PTE an application might refer
> to. There might be other related issues beyond this one though.
> Fortunately, mode transitions for dealing for such faults normally don't
> lead to bad latencies on this arch. Do you confirm that, or are you
> unlucky regarding this?
>   

As this would normally happen only on startup it is not such a big deal. 
After the first couple of cycles this should not occur anymore, so 
operational RT performance is not compromised.
The only problem is that the application switches to secondary mode, and 
that I have to switch it back to primary mode manually (or by doing a 
Xenomai system call). Is there a possibility to automatically switch 
back to primary mode when such a fault occurs?

I would expect that this problem was popping up on other ppc platforms 
as well, but I did not find references to this. Is this something that 
specific to this architecture, this platform, or are other people just 
ignoring this problem?

And now for some personal thoughts.
To be honest, I am a bit surprised by this problem (but that is caused 
by my lack of knowledge in this area).
I am running this on a system without swap and with CONFIG_SWAP disabled 
in the kernel. My applications are linked statically (ldd gives: "not a 
dynamic executable"), so I would expect that everything is available in 
memory. But it looks like there are some things going on below the 
surface that I am not aware of (but that I am curious to find out).

Kind regards,
    Johan Borkhuis

Re: [Xenomai-help] Switch to secondary mode

[Philippe Gerum]

On Tue, 2007-07-03 at 11:09 +0200, Johan Borkhuis wrote:
> Philippe,
> 
> Again, thank you for very extensive answer. It is starting to get 
> clearer what is happening here but I still have some questions.
> 
> The first one is probably an easy one: what do you mean by COW? I did 
> try to find what this means, but I was only able to find references to 
> farm animals.....

Ok, that's a good start.

>  (and some references to lazy linking etc. but there 
> COW was also not explained).

Search for "copy-on-write".

> 
> Philippe Gerum wrote:
> > On Mon, 2007-07-02 at 16:18 +0200, Johan Borkhuis wrote:
> >   
> >> Philippe,
> >>
> >> Philippe Gerum wrote:
> >>     
> >>> Late binding to functions performed on behalf of the dynamic loader
> >>> against shared libraries shall need the kernel during symbol resolution
> >>> (internal syscalls) or execution (e.g. demand loading, COW), hence the
> >>> switch. Unfortunately, the I-pipe patch for PPC does not support
> >>> disabling all on-demand memory mappings for selected Linux tasks (only
> >>> the x86 and ARM patches support this feature so far).
> >>>   
> >>>      
> >> Thank you for you answer.
> >>
> >> Just for me to make sure I understand this correctly:
> >> We are not using shared libraries for our application, our applications 
> >> are linked against .a files, which are included in the final application
> >>     
> >
> > In such a case, you have likely hit an illustration of the latter issue
> > which the I-pipe/ppc implementation still suffers from: some page table
> > entries are missed during real-time operations. As a consequence of
> > this, the nucleus catches page faults on behalf of RT threads in primary
> > mode, then switches these threads back to secondary in order to process
> > the faults, and eventually wire the missing PTEs in. This is something
> > calling mlockall() does not prevent the application from (like COW).
> >   
> As some PTE's are missed, does this mean that not the complete program 
> was loaded into memory?
> 
> What I understand until now about this process is the following:
> The program is executed. Not everything is loaded into memory by the 
> dynamic loader. As the functions that are not in memory are accessed a 
> page fault is created, and the page is made available. (Or is the page 
> already in memory, but not made available to the application?)
> Is my assumption correct that once a page is accessed mlockall will take 
> care that this page stays active, or is it possible that the page is 
> moved out, and that another page fault occurs for the same page?
> 
> > .....
> >   
> > That is expected. If you switch the nucleus debug option on, you should
> > see Xenomai whining about secondary mode switches from code locations in
> > kernel space. This would confirm the fact that you have been hitting
> > this problem.
> >   
> 
> When looking at the nucleus debug output, I see a number of switches 
> coming from user space, like this one:
> 
> Jul  3 06:51:10 MVME3100-198 kernel: Xenomai: Switching testTask to 
> secondary mode after exception #1025 from user-space at 0x10005f2c (pid 
> 1069)
> 
> I did try to find what exception 1025 is, but I could not find this 
> reference. I expect indeed that this is a page-fault, but I am not sure.
> The location 0x10005f2c is the start of a function in one of the 
> statically linked objects from one of our archives. You are referring to 
> kernel space, but these exceptions are generated from user space. Is 
> this different from what you are referring to?
> > There is not much to be done except improving the I-pipe/ppc support so
> > that it provides a way to pin down any PTE an application might refer
> > to. There might be other related issues beyond this one though.
> > Fortunately, mode transitions for dealing for such faults normally don't
> > lead to bad latencies on this arch. Do you confirm that, or are you
> > unlucky regarding this?
> >   
> 
> As this would normally happen only on startup it is not such a big deal. 
> After the first couple of cycles this should not occur anymore, so 
> operational RT performance is not compromised.
> The only problem is that the application switches to secondary mode, and 
> that I have to switch it back to primary mode manually (or by doing a 
> Xenomai system call). Is there a possibility to automatically switch 
> back to primary mode when such a fault occurs?
> 
> I would expect that this problem was popping up on other ppc platforms 
> as well, but I did not find references to this. Is this something that 
> specific to this architecture, this platform, or are other people just 
> ignoring this problem?
> 
> And now for some personal thoughts.
> To be honest, I am a bit surprised by this problem (but that is caused 
> by my lack of knowledge in this area).
> I am running this on a system without swap and with CONFIG_SWAP disabled 
> in the kernel. My applications are linked statically (ldd gives: "not a 
> dynamic executable"), so I would expect that everything is available in 
> memory. But it looks like there are some things going on below the 
> surface that I am not aware of (but that I am curious to find out).
> 
> Kind regards,
>     Johan Borkhuis
-- 
Philippe.

Re: [Xenomai-help] Switch to secondary mode

[Jeff Koftinoff]

Philippe Gerum wrote:
> In such a case, you have likely hit an illustration of the latter issue
> which the I-pipe/ppc implementation still suffers from: some page table
> entries are missed during real-time operations. As a consequence of
> this, the nucleus catches page faults on behalf of RT threads in primary
> mode, then switches these threads back to secondary in order to process
> the faults, and eventually wire the missing PTEs in. This is something
> calling mlockall() does not prevent the application from (like COW).
>
> The shared lib problem would be another issue, even if it relates to the
> same general topic (i.e. lazy/on-demand mapping of memory resources the
> kernel performs).
>
>   
Hi Philippe.

Very soon I am going to be using Xenomai on PPC with SMP...

Under what situations are 'page table entries missed'? what exactly does 
that mean?

Regards,
Jeff Koftinoff

www.jdkoftinoff.com

Re: [Xenomai-help] Switch to secondary mode

[Philippe Gerum]

On Tue, 2007-07-03 at 12:53 -0700, Jeff Koftinoff wrote:
> Philippe Gerum wrote:
> > In such a case, you have likely hit an illustration of the latter issue
> > which the I-pipe/ppc implementation still suffers from: some page table
> > entries are missed during real-time operations. As a consequence of
> > this, the nucleus catches page faults on behalf of RT threads in primary
> > mode, then switches these threads back to secondary in order to process
> > the faults, and eventually wire the missing PTEs in. This is something
> > calling mlockall() does not prevent the application from (like COW).
> >
> > The shared lib problem would be another issue, even if it relates to the
> > same general topic (i.e. lazy/on-demand mapping of memory resources the
> > kernel performs).
> >
> >   
> Hi Philippe.
> 
> Very soon I am going to be using Xenomai on PPC with SMP...

You mean ppc64?

> 
> Under what situations are 'page table entries missed'? what exactly does 
> that mean?
> 

For instance, the MMU service code constantly needs to populate an
internal cache holding the most recently used PTEs, but also to evict
some page table entries, when it is under pressure to get space into the
cache to hold other - hotter - entries. In that case, any subsequent
(virtual) memory reference which needs the missing PTE for translation
would cause a minor fault on a CPU with no hardware assistance for
handling PTE misses. This fault is intercepted by the kernel to fetch
back the needed PTE, and make the memory reference possible. The code
would resume at the faulted code location without even knowing of the
former issue.

The interrupt pipeline relays such fault event to Xenomai, which in turn
switches the faulting RT thread to plain Linux mode (i.e. "secondary"
mode from Xenomai's POV), so that Linux's fault handler may do the
housekeeping chores as needed (i.e. fetch the PTE back). We have to do
that because a RT thread might have preempted the vanilla kernel code in
the middle of nowhere, to serve a high priority RT interrupt. In such a
case, "nowhere" could be anywhere, including in some critical section
Linux-wise (but still preemptible by Xenomai threads), and we don't want
to re-enter Linux in an unsafe manner. By switching to secondary mode,
we ensure that the fault shall be processed only when Linux allows such
faults to be processed (i.e. outside of any critical section from its
own POV), because we restore the normal Linux execution flow before the
fault is handled by the kernel.

Even if I'm not satisfied with the current state of affairs regarding
this issue, PTE misses on PPC do not seem to bring unbearable latencies,
even if they do increase the wost-case latency. AFAICT, on a Freescale
lite5200/64mb for instance, PTE entries bring an additional jitter
estimated at ~20 us (~55 us instead of ~35 us when running a 10Khz loop
in user-space). On a mpc85xx/128Mb, worst-case is still in the 35 us
range in user-space context, which is quite acceptable. The same way,
the worst-case was quite low on x86, even before we did provide (Gilles
did actually) for pinning down ranges of PTE entries there, so that some
Linux tasks would never endure PTE misses anymore once they get promoted
to Xenomai RT threads. We likely benefit from the continuous effort the
upstream folks are making toward an always finer kernel granularity,
there - i.e. switching mode on recent kernels is becoming reasonably
short and bounded. Memory pressure is a significant factor increasing
the PTE issue too, which has more impact on low-end than mid-range or
high-end embedded hardware.

The proper cure - mainly for low-end hardware - would require to have
the I-pipe/ppc patch provide a particular feature the latest x86 and ARM
patches do, which basically makes sure that all on-demand memory
strategies (copy-on-write, lazy mapping of kernel memory resources etc.)
are fully disabled for a certain class of Linux tasks (which would need
to cover the PTE case too if possible). Probably non-trivial.

> Regards,
> Jeff Koftinoff
> 
> www.jdkoftinoff.com
> 
-- 
Philippe.