[Xenomai-help] cannot make system() calls anymore

[Xenomai-help] cannot make system() calls anymore

[Jerome Pasquero]

[-- Attachment #1: Type: text/plain, Size: 2548 bytes --]

Hi,

I recently upgraded from xenomai 2.3.0 under linux 2.6.17.14 to the latest
stable version of xenomai 2.4.1 under linux 2.6.23.12.
Everything seems to be working just fine, except that I seem to  no longer
be able to make system calls inside a real-time task. It simply freezes
everything and I need to restart my computer.
I don't know if it was just a fluke that it was working with 2.3.0 and that
I am not supposed to do that at all. Could it be that I forgot to include an
option when recompiling the kernel?

Note that I have no problem making system calls from outside a real time
task (for instance, right before a rt_task_shadow() call).
Also, it seems that the problem only occurs when other libraries are used.
For instance, if one compiles the code below with no other libraries than
the xenomai libs (and stdc++), it works just fine:
gcc  `/usr/xenomai/bin/xeno-config --xeno-cflags`
`/usr/xenomai/bin/xeno-config --xeno-ldflags` -lstdc++ -lnative main.cpp

However, as soon as I add the libxml2 library, it crashes:
gcc  -D_GNU_SOURCE -D_REENTRANT -D__XENO__  -I/usr/include/libxml2
-I/usr/xenomai-2.4.1/include  -L/usr/local/lib -L/usr/xenomai-2.4.1/lib
-lxml2 -lnative -lpthread `/usr/xenomai/bin/xeno-config --xeno-cflags`
`/usr/xenomai/bin/xeno-config --xeno-ldflags` -lstdc++ -lnative main.cp

This is true with other libraries as well (e.g. gsl lib) but not of all
libraries (for instance, it seems to work with the math library i.e. -lm)

Am I doing something wrong?
In case you were wondering, I want to be able to make system calls from
inside real time threads cause I want to load/unload realtime modules
directly from my application. In the code snippet below, I am simply trying
to do a  'ls'  command to keep it as simple as possible.

Thanks for your help in advance.


#include <iostream>
#include <sys/mman.h>
#include <signal.h>
#include <native/mutex.h>
#include <native/timer.h>
#include <native/task.h>

void cleanup(int unused)
{
    // delete current task
    rt_task_delete(NULL);
}

int main(int argc, char *argv[])
{
    signal(SIGABRT, cleanup);
    signal(SIGKILL, cleanup);
    signal(SIGINT, cleanup);
    signal(SIGTERM, cleanup);
    signal(SIGHUP, cleanup);
    signal(SIGALRM, cleanup);


    RT_TASK mainTask;

    mlockall(MCL_CURRENT | MCL_FUTURE);
    rt_task_shadow(&mainTask, "gui", 50, T_FPU);

    rt_task_sleep((RTIME) 1E9);

    system("ls"); // this is what causes the problem...

    cleanup(0);

    return 0;
}


-jerome

-- 
Jerome Pasquero
Ph.D. Student
McGill University
Montreal

[-- Attachment #2: Type: text/html, Size: 3284 bytes --]

Re: [Xenomai-help] cannot make system() calls anymore

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 3189 bytes --]

Jerome Pasquero wrote:
> Hi,
> 
> I recently upgraded from xenomai 2.3.0 under linux 2.6.17.14 to the latest
> stable version of xenomai 2.4.1 under linux 2.6.23.12.
> Everything seems to be working just fine, except that I seem to  no longer
> be able to make system calls inside a real-time task. It simply freezes
> everything and I need to restart my computer.
> I don't know if it was just a fluke that it was working with 2.3.0 and that
> I am not supposed to do that at all. Could it be that I forgot to include an
> option when recompiling the kernel?

A system freeze is not a valid reaction of Xenomai on invalid user 
activity - unless that freeze just means one of your high-prio tasks 
decided to enter an endless loop. Have you already tried to enable the 
Xenomai watchdog?

> 
> Note that I have no problem making system calls from outside a real time
> task (for instance, right before a rt_task_shadow() call).
> Also, it seems that the problem only occurs when other libraries are used.
> For instance, if one compiles the code below with no other libraries than
> the xenomai libs (and stdc++), it works just fine:
> gcc  `/usr/xenomai/bin/xeno-config --xeno-cflags`
> `/usr/xenomai/bin/xeno-config --xeno-ldflags` -lstdc++ -lnative main.cpp
> 
> However, as soon as I add the libxml2 library, it crashes:
> gcc  -D_GNU_SOURCE -D_REENTRANT -D__XENO__  -I/usr/include/libxml2
> -I/usr/xenomai-2.4.1/include  -L/usr/local/lib -L/usr/xenomai-2.4.1/lib
> -lxml2 -lnative -lpthread `/usr/xenomai/bin/xeno-config --xeno-cflags`
> `/usr/xenomai/bin/xeno-config --xeno-ldflags` -lstdc++ -lnative main.cp
> 
> This is true with other libraries as well (e.g. gsl lib) but not of all
> libraries (for instance, it seems to work with the math library i.e. -lm)
> 
> Am I doing something wrong?
> In case you were wondering, I want to be able to make system calls from
> inside real time threads cause I want to load/unload realtime modules
> directly from my application. In the code snippet below, I am simply trying
> to do a  'ls'  command to keep it as simple as possible.

Thanks for the demo. This one already bricks your box? But only when 
linked against libxml?

> 
> Thanks for your help in advance.
> 
> 
> #include <iostream>
> #include <sys/mman.h>
> #include <signal.h>
> #include <native/mutex.h>
> #include <native/timer.h>
> #include <native/task.h>
> 
> void cleanup(int unused)
> {
>     // delete current task
>     rt_task_delete(NULL);
> }
> 
> int main(int argc, char *argv[])
> {
>     signal(SIGABRT, cleanup);
>     signal(SIGKILL, cleanup);
>     signal(SIGINT, cleanup);
>     signal(SIGTERM, cleanup);
>     signal(SIGHUP, cleanup);
>     signal(SIGALRM, cleanup);
> 
> 
>     RT_TASK mainTask;
> 
>     mlockall(MCL_CURRENT | MCL_FUTURE);
>     rt_task_shadow(&mainTask, "gui", 50, T_FPU);
> 
>     rt_task_sleep((RTIME) 1E9);
> 
>     system("ls"); // this is what causes the problem...

Ah, you mean "system" with system call - or actually any call of 
Standard-Linux kernel services?

In any case, let's first check for runaway threads with the watchdog.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-help] cannot make system() calls anymore

[Gilles Chanteperdrix]

On Jan 11, 2008 7:22 PM, Jan Kiszka <jan.kiszka@domain.hid> wrote:
> Jerome Pasquero wrote:
> > Hi,
> >
> > I recently upgraded from xenomai 2.3.0 under linux 2.6.17.14 to the latest
> > stable version of xenomai 2.4.1 under linux 2.6.23.12.
> > Everything seems to be working just fine, except that I seem to  no longer
> > be able to make system calls inside a real-time task. It simply freezes
> > everything and I need to restart my computer.
> > I don't know if it was just a fluke that it was working with 2.3.0 and that
> > I am not supposed to do that at all. Could it be that I forgot to include an
> > option when recompiling the kernel?
>
> A system freeze is not a valid reaction of Xenomai on invalid user
> activity - unless that freeze just means one of your high-prio tasks
> decided to enter an endless loop. Have you already tried to enable the
> Xenomai watchdog?

A freeze on fork (system calls fork) is typical of the "nocow" stuff
not working correctly.

-- 
                                               Gilles Chanteperdrix

Re: [Xenomai-help] cannot make system() calls anymore

[Jerome Pasquero]

[-- Attachment #1: Type: text/plain, Size: 4129 bytes --]

On Jan 11, 2008 1:22 PM, Jan Kiszka <jan.kiszka@domain.hid> wrote:

> Jerome Pasquero wrote:
> > Hi,
> >
> > I recently upgraded from xenomai 2.3.0 under linux 2.6.17.14 to the
> latest
> > stable version of xenomai 2.4.1 under linux 2.6.23.12.
> > Everything seems to be working just fine, except that I seem to  no
> longer
> > be able to make system calls inside a real-time task. It simply freezes
> > everything and I need to restart my computer.
> > I don't know if it was just a fluke that it was working with 2.3.0 and
> that
> > I am not supposed to do that at all. Could it be that I forgot to
> include an
> > option when recompiling the kernel?
>
> A system freeze is not a valid reaction of Xenomai on invalid user
> activity - unless that freeze just means one of your high-prio tasks
> decided to enter an endless loop. Have you already tried to enable the
> Xenomai watchdog?
>

No, I haven't tried to enable the watchdog. I'll try to take a look at that.


>
> >
> > Note that I have no problem making system calls from outside a real time
> > task (for instance, right before a rt_task_shadow() call).
> > Also, it seems that the problem only occurs when other libraries are
> used.
> > For instance, if one compiles the code below with no other libraries
> than
> > the xenomai libs (and stdc++), it works just fine:
> > gcc  `/usr/xenomai/bin/xeno-config --xeno-cflags`
> > `/usr/xenomai/bin/xeno-config --xeno-ldflags` -lstdc++ -lnative main.cpp
> >
> > However, as soon as I add the libxml2 library, it crashes:
> > gcc  -D_GNU_SOURCE -D_REENTRANT -D__XENO__  -I/usr/include/libxml2
> > -I/usr/xenomai-2.4.1/include  -L/usr/local/lib -L/usr/xenomai-2.4.1/lib
> > -lxml2 -lnative -lpthread `/usr/xenomai/bin/xeno-config --xeno-cflags`
> > `/usr/xenomai/bin/xeno-config --xeno-ldflags` -lstdc++ -lnative main.cp
> >
> > This is true with other libraries as well (e.g. gsl lib) but not of all
> > libraries (for instance, it seems to work with the math library i.e.
> -lm)
> >
> > Am I doing something wrong?
> > In case you were wondering, I want to be able to make system calls from
> > inside real time threads cause I want to load/unload realtime modules
> > directly from my application. In the code snippet below, I am simply
> trying
> > to do a  'ls'  command to keep it as simple as possible.
>
> Thanks for the demo. This one already bricks your box? But only when
> linked against libxml?
>
That's exactly it. I tried to reduce the problem to a very simple case. Note
that it also bricks my box when I link it to other libraries but not to all
libraries (e.g. it works with the math library but not with the gsl).



>
> >
> > Thanks for your help in advance.
> >
> >
> > #include <iostream>
> > #include <sys/mman.h>
> > #include <signal.h>
> > #include <native/mutex.h>
> > #include <native/timer.h>
> > #include <native/task.h>
> >
> > void cleanup(int unused)
> > {
> >     // delete current task
> >     rt_task_delete(NULL);
> > }
> >
> > int main(int argc, char *argv[])
> > {
> >     signal(SIGABRT, cleanup);
> >     signal(SIGKILL, cleanup);
> >     signal(SIGINT, cleanup);
> >     signal(SIGTERM, cleanup);
> >     signal(SIGHUP, cleanup);
> >     signal(SIGALRM, cleanup);
> >
> >
> >     RT_TASK mainTask;
> >
> >     mlockall(MCL_CURRENT | MCL_FUTURE);
> >     rt_task_shadow(&mainTask, "gui", 50, T_FPU);
> >
> >     rt_task_sleep((RTIME) 1E9);
> >
> >     system("ls"); // this is what causes the problem...
>
> Ah, you mean "system" with system call - or actually any call of
> Standard-Linux kernel services?


I mean any call with the function system() made directly from the realtime
task. If I open a shell and make a system call, everything seems to work
fine.


>
>
> In any case, let's first check for runaway threads with the watchdog.


>
> Jan
>
>


-- 
Jerome Pasquero
Ph.D. Student
McGill University
Haptics Lab
3480 University Street Room 410
Montréal, Québec, Canada
H3A 2A7
T: (514) 398-8206
F: (514) 398-7348
www.cim.mcgill.ca/~jay

[-- Attachment #2: Type: text/html, Size: 5788 bytes --]

Re: [Xenomai-help] cannot make system() calls anymore

[Jan Kiszka]

[-- Attachment #1.1: Type: text/plain, Size: 1780 bytes --]

Gilles Chanteperdrix wrote:
> On Jan 11, 2008 7:22 PM, Jan Kiszka <jan.kiszka@domain.hid> wrote:
>> Jerome Pasquero wrote:
>>> Hi,
>>>
>>> I recently upgraded from xenomai 2.3.0 under linux 2.6.17.14 to the latest
>>> stable version of xenomai 2.4.1 under linux 2.6.23.12.
>>> Everything seems to be working just fine, except that I seem to  no longer
>>> be able to make system calls inside a real-time task. It simply freezes
>>> everything and I need to restart my computer.
>>> I don't know if it was just a fluke that it was working with 2.3.0 and that
>>> I am not supposed to do that at all. Could it be that I forgot to include an
>>> option when recompiling the kernel?
>> A system freeze is not a valid reaction of Xenomai on invalid user
>> activity - unless that freeze just means one of your high-prio tasks
>> decided to enter an endless loop. Have you already tried to enable the
>> Xenomai watchdog?
> 
> A freeze on fork (system calls fork) is typical of the "nocow" stuff
> not working correctly.

Yep, you are most probably right: The page table walks in
ipipe_disable_ondemand_mappings and below is not safe /wrt unused and
bad entries. A test for p?d_none_or_clear_bad is missing, see attached
patch. Jerome, could you try this one and report if it helps? Please
also watch out for kernel messages that might pop up with this patch.

I'm not %100 sure if we should use clear_bad here or better just ignore
bad pages. Once and only once during the last tests, I had a warning
about a bad page in my syslog after running Jerome's test. On the other
hand, most other code in mm/memory.c does it like that. Opinions?

Jerome's test triggered another, less critical I-pipe bug (SMP-related)
I'm going to report separately.

Jan

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: safe-pagetable-walks.patch --]
[-- Type: text/x-patch; name="safe-pagetable-walks.patch", Size: 1238 bytes --]

---
 mm/memory.c |    6 ++++++
 1 file changed, 6 insertions(+)

Index: linux-2.6.24-rc6-xeno_64/mm/memory.c
===================================================================
--- linux-2.6.24-rc6-xeno_64.orig/mm/memory.c
+++ linux-2.6.24-rc6-xeno_64/mm/memory.c
@@ -2827,6 +2827,8 @@ static inline int ipipe_pin_pmd_range(st
 	pmd = pmd_offset(pud, addr);
 	do {
 		next = pmd_addr_end(addr, end);
+		if (pmd_none(*pmd) || unlikely(pmd_bad(*pmd)))
+			continue;
 		if (ipipe_pin_pte_range(mm, pmd, vma, addr, end))
 			return -ENOMEM;
 	} while (pmd++, addr = next, addr != end);
@@ -2843,6 +2845,8 @@ static inline int ipipe_pin_pud_range(st
 	pud = pud_offset(pgd, addr);
 	do {
 		next = pud_addr_end(addr, end);
+		if (pud_none(*pud) || unlikely(pud_bad(*pud)))
+			continue;
 		if (ipipe_pin_pmd_range(mm, pud, vma, addr, end))
 			return -ENOMEM;
 	} while (pud++, addr = next, addr != end);
@@ -2875,6 +2879,8 @@ int ipipe_disable_ondemand_mappings(stru
 		pgd = pgd_offset(mm, addr);
 		do {
 			next = pgd_addr_end(addr, end);
+			if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd)))
+				continue;
 			if (ipipe_pin_pud_range(mm, pgd, vma, addr, next)) {
 				result = -ENOMEM;
 				goto done_mm;

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 254 bytes --]

Re: [Xenomai-help] cannot make system() calls anymore

[Philippe Gerum]

Jan Kiszka wrote:
> Gilles Chanteperdrix wrote:
>> On Jan 11, 2008 7:22 PM, Jan Kiszka <jan.kiszka@domain.hid> wrote:
>>> Jerome Pasquero wrote:
>>>> Hi,
>>>>
>>>> I recently upgraded from xenomai 2.3.0 under linux 2.6.17.14 to the latest
>>>> stable version of xenomai 2.4.1 under linux 2.6.23.12.
>>>> Everything seems to be working just fine, except that I seem to  no longer
>>>> be able to make system calls inside a real-time task. It simply freezes
>>>> everything and I need to restart my computer.
>>>> I don't know if it was just a fluke that it was working with 2.3.0 and that
>>>> I am not supposed to do that at all. Could it be that I forgot to include an
>>>> option when recompiling the kernel?
>>> A system freeze is not a valid reaction of Xenomai on invalid user
>>> activity - unless that freeze just means one of your high-prio tasks
>>> decided to enter an endless loop. Have you already tried to enable the
>>> Xenomai watchdog?
>> A freeze on fork (system calls fork) is typical of the "nocow" stuff
>> not working correctly.
> 
> Yep, you are most probably right: The page table walks in
> ipipe_disable_ondemand_mappings and below is not safe /wrt unused and
> bad entries. A test for p?d_none_or_clear_bad is missing, see attached
> patch. Jerome, could you try this one and report if it helps? Please
> also watch out for kernel messages that might pop up with this patch.
> 
> I'm not %100 sure if we should use clear_bad here or better just ignore
> bad pages. Once and only once during the last tests, I had a warning
> about a bad page in my syslog after running Jerome's test. On the other
> hand, most other code in mm/memory.c does it like that. Opinions?
> 

It seems that anyone detecting bad entries during walks should clear
them since this code very looks like a safety belt to trap unexpected
situations. The only minor issue for us to do more than ignoring, would
be to have the error message point the finger at the non-vanilla nocow
code, albeit it would not be responsible for this situation.

> Jerome's test triggered another, less critical I-pipe bug (SMP-related)
> I'm going to report separately.
> 
> Jan
> 


-- 
Philippe.