[Xenomai-core] latency kernel part crashes on ppc64

[Xenomai-core] latency kernel part crashes on ppc64

[Heikki Lindholm]

Hi,

Some recent changes (*cough* RTDM benchmark driver *cough*) broke kernel 
mode benchmarking for ppc64. Previously klatency worked fine, but now 
latency -t 1 crashes somewhere in xnpod_schedule. Jan, any pending 
patches a comin'?

-- Heikki Lindholm

Re: [Xenomai-core] latency kernel part crashes on ppc64

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 953 bytes --]

Heikki Lindholm wrote:
> Hi,
> 
> Some recent changes (*cough* RTDM benchmark driver *cough*) broke kernel
> mode benchmarking for ppc64. Previously klatency worked fine, but now
> latency -t 1 crashes somewhere in xnpod_schedule. Jan, any pending
> patches a comin'?

Nope, it should work as it is. But as Stelian also reported problems on
his fresh ARM port with the in-kernel test, I cannot exclude that there
/might/ be a problem in the benchmark.

As I don't have any ppc64 hanging around somewhere, we will have to go
through this together. Things I would like to know:

 o When and how does it crash? At start-up immediately? Or after a
   while?
 o Are there any details / backtraces available with the crash?
 o Does -t2 work?
 o What happens if your disable "rtdm_event_pulse(&ctx->result_event);"
   in eval_outer_loop (thus no signalling of intermediate results during
   the test)? Does it still crash, maybe later during cleanup now?

Jan

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 254 bytes --]

Re: [Xenomai-core] latency kernel part crashes on ppc64

[Heikki Lindholm]

Jan Kiszka kirjoitti:
> Heikki Lindholm wrote:
> 
>>Hi,
>>
>>Some recent changes (*cough* RTDM benchmark driver *cough*) broke kernel
>>mode benchmarking for ppc64. Previously klatency worked fine, but now
>>latency -t 1 crashes somewhere in xnpod_schedule. Jan, any pending
>>patches a comin'?
> 
> 
> Nope, it should work as it is. But as Stelian also reported problems on
> his fresh ARM port with the in-kernel test, I cannot exclude that there
> /might/ be a problem in the benchmark.
> 
> As I don't have any ppc64 hanging around somewhere, we will have to go
> through this together. Things I would like to know:

Dammit, I hoped you'd whip up a fix just from me noting a problem. Well, 
all right then, I'll play along...;)

>  o When and how does it crash? At start-up immediately? Or after a
>    while?

I inserted some serial debug prints and it gets two passes to 
eval_outer_loop done (enter/exit function). After that it freezes. 
Without the debug printing it dies with kernel access of illegal memory 
at xnpod_schedule, which btw. has been quite a common place to die.

>  o Are there any details / backtraces available with the crash?

Becaktrace limits to xnpod_schedule if I remember right.

>  o Does -t2 work?

Umm. Probably not. See below.

>  o What happens if your disable "rtdm_event_pulse(&ctx->result_event);"
>    in eval_outer_loop (thus no signalling of intermediate results during
>    the test)? Does it still crash, maybe later during cleanup now?

Doesn't freeze and can be exited with ctrl-c and even re-run.

One odd thing (probably unrelated) is that the first two ioctls get 
called in what seems like wrong order, eg. START_TMTEST first ends up in 
tmbench_ioctl_rt and then _nrt and INTERM_RESULT ends up first in _nrt 
and then _rt.

-- Heikki Lindholm

Re: [Xenomai-core] latency kernel part crashes on ppc64

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2848 bytes --]

Heikki Lindholm wrote:
> Jan Kiszka kirjoitti:
> 
>> Heikki Lindholm wrote:
>>
>>> Hi,
>>>
>>> Some recent changes (*cough* RTDM benchmark driver *cough*) broke kernel
>>> mode benchmarking for ppc64. Previously klatency worked fine, but now
>>> latency -t 1 crashes somewhere in xnpod_schedule. Jan, any pending
>>> patches a comin'?
>>

To get this clearly: You tested the old klatency(+front-end) on latest
xeno and it worked? Or does this parse "the old klatency worked over old
xeno on PPC64"?

Comparing the old test with the new framework, the major difference is
that the old one only knew a single kernel RT-task. Its front-end was
reading from a pipe and was therefore a pure linux program. Now we have
two RT-tasks, one is even a shadow, and they use RT-IPC. Not sure if
this really means that the bug must be in the benchmark suite...

>>
>>
>> Nope, it should work as it is. But as Stelian also reported problems on
>> his fresh ARM port with the in-kernel test, I cannot exclude that there
>> /might/ be a problem in the benchmark.
>>
>> As I don't have any ppc64 hanging around somewhere, we will have to go
>> through this together. Things I would like to know:
> 
> 
> Dammit, I hoped you'd whip up a fix just from me noting a problem. Well,
> all right then, I'll play along...;)
> 
>>  o When and how does it crash? At start-up immediately? Or after a
>>    while?
> 
> 
> I inserted some serial debug prints and it gets two passes to
> eval_outer_loop done (enter/exit function). After that it freezes.
> Without the debug printing it dies with kernel access of illegal memory
> at xnpod_schedule, which btw. has been quite a common place to die.
> 
>>  o Are there any details / backtraces available with the crash?
> 
> 
> Becaktrace limits to xnpod_schedule if I remember right.
> 
>>  o Does -t2 work?
> 
> 
> Umm. Probably not. See below.

Arrgh, "probably" - when it's so easy to test...

When you are already on it: pure user-space (-t0) also works?

> 
>>  o What happens if your disable "rtdm_event_pulse(&ctx->result_event);"
>>    in eval_outer_loop (thus no signalling of intermediate results during
>>    the test)? Does it still crash, maybe later during cleanup now?
> 
> 
> Doesn't freeze and can be exited with ctrl-c and even re-run.
> 
> One odd thing (probably unrelated) is that the first two ioctls get
> called in what seems like wrong order, eg. START_TMTEST first ends up in
> tmbench_ioctl_rt and then _nrt and INTERM_RESULT ends up first in _nrt
> and then _rt.

This takes me back to the number of active real-time tasks during the
test. This disabling reduces the scenario basically again to old
klatency times.

I looked at my code again, but - maybe I'm too blind now - I cannot find
even a potential pointer bug, especially when the histogram feature (-h)
is not used. I need more input! ;)

Jan

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 254 bytes --]

Re: [Xenomai-core] latency kernel part crashes on ppc64

[Heikki Lindholm]

Jan Kiszka kirjoitti:
> Heikki Lindholm wrote:
> 
>>Jan Kiszka kirjoitti:
>>
>>
>>>Heikki Lindholm wrote:
>>>
>>>
>>>>Hi,
>>>>
>>>>Some recent changes (*cough* RTDM benchmark driver *cough*) broke kernel
>>>>mode benchmarking for ppc64. Previously klatency worked fine, but now
>>>>latency -t 1 crashes somewhere in xnpod_schedule. Jan, any pending
>>>>patches a comin'?
>>>
> 
> To get this clearly: You tested the old klatency(+front-end) on latest
> xeno and it worked? Or does this parse "the old klatency worked over old
> xeno on PPC64"?

"Previously" as in ... well ... previously, so it means the old xenomai 
with klatency intact.

> Comparing the old test with the new framework, the major difference is
> that the old one only knew a single kernel RT-task. Its front-end was
> reading from a pipe and was therefore a pure linux program. Now we have
> two RT-tasks, one is even a shadow, and they use RT-IPC. Not sure if
> this really means that the bug must be in the benchmark suite...

Right. I'll have to see if there's a problem with any of these.

>>> o Does -t2 work?
>>
>>
>>Umm. Probably not. See below.
> 
> 
> Arrgh, "probably" - when it's so easy to test...

Well, it's one compile and boot cycle more with my current situation. I 
try to view laziness as a gift...

> When you are already on it: pure user-space (-t0) also works?

Pure user-space works fine.

>>> o What happens if your disable "rtdm_event_pulse(&ctx->result_event);"
>>>   in eval_outer_loop (thus no signalling of intermediate results during
>>>   the test)? Does it still crash, maybe later during cleanup now?
>>
>>
>>Doesn't freeze and can be exited with ctrl-c and even re-run.
>>
>>One odd thing (probably unrelated) is that the first two ioctls get
>>called in what seems like wrong order, eg. START_TMTEST first ends up in
>>tmbench_ioctl_rt and then _nrt and INTERM_RESULT ends up first in _nrt
>>and then _rt.
> 
> 
> This takes me back to the number of active real-time tasks during the
> test. This disabling reduces the scenario basically again to old
> klatency times.
> 
> I looked at my code again, but - maybe I'm too blind now - I cannot find
> even a potential pointer bug, especially when the histogram feature (-h)
> is not used. I need more input! ;)

Hehe. The histogram was the first thing I peered at, only to find out 
it's not even used. This might be a kernel thread switching bug in my 
code, but I find it hard to believe, because then even one thread 
probably wouldn't work.

-- Heikki Lindholm

Re: [Xenomai-core] latency kernel part crashes on ppc64

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 3368 bytes --]

Heikki Lindholm wrote:
> Jan Kiszka kirjoitti:
> 
>> Heikki Lindholm wrote:
>>
>>> Jan Kiszka kirjoitti:
>>>
>>>
>>>> Heikki Lindholm wrote:
>>>>
>>>>
>>>>> Hi,
>>>>>
>>>>> Some recent changes (*cough* RTDM benchmark driver *cough*) broke
>>>>> kernel
>>>>> mode benchmarking for ppc64. Previously klatency worked fine, but now
>>>>> latency -t 1 crashes somewhere in xnpod_schedule. Jan, any pending
>>>>> patches a comin'?
>>>>
>>>>
>>
>> To get this clearly: You tested the old klatency(+front-end) on latest
>> xeno and it worked? Or does this parse "the old klatency worked over old
>> xeno on PPC64"?
> 
> 
> "Previously" as in ... well ... previously, so it means the old xenomai
> with klatency intact.
> 
>> Comparing the old test with the new framework, the major difference is
>> that the old one only knew a single kernel RT-task. Its front-end was
>> reading from a pipe and was therefore a pure linux program. Now we have
>> two RT-tasks, one is even a shadow, and they use RT-IPC. Not sure if
>> this really means that the bug must be in the benchmark suite...
> 
> 
> Right. I'll have to see if there's a problem with any of these.
> 
>>>> o Does -t2 work?
>>>
>>>
>>>
>>> Umm. Probably not. See below.
>>
>>
>>
>> Arrgh, "probably" - when it's so easy to test...
> 
> 
> Well, it's one compile and boot cycle more with my current situation. I
> try to view laziness as a gift...
> 

Rebooting - sure, but what do you compile each time?

>> When you are already on it: pure user-space (-t0) also works?
> 
> 
> Pure user-space works fine.

Hmm, that's not so different from the POV of used nucleus mechanisms...

> 
>>>> o What happens if your disable "rtdm_event_pulse(&ctx->result_event);"
>>>>   in eval_outer_loop (thus no signalling of intermediate results during
>>>>   the test)? Does it still crash, maybe later during cleanup now?
>>>
>>>
>>>
>>> Doesn't freeze and can be exited with ctrl-c and even re-run.
>>>
>>> One odd thing (probably unrelated) is that the first two ioctls get
>>> called in what seems like wrong order, eg. START_TMTEST first ends up in
>>> tmbench_ioctl_rt and then _nrt and INTERM_RESULT ends up first in _nrt
>>> and then _rt.

Forgot to comment on this: It's a normal behaviour. The caller comes
across the first IOCTL in primary mode. But as this service is only
available to secondary mode, the invocation is restarted after a mode
switch. Same happens to the second IOCTL, just the other way around. If
you weren't lazy, I would suggest watching the different flow when
switching the mode manually first. ;)

>>
>>
>>
>> This takes me back to the number of active real-time tasks during the
>> test. This disabling reduces the scenario basically again to old
>> klatency times.
>>
>> I looked at my code again, but - maybe I'm too blind now - I cannot find
>> even a potential pointer bug, especially when the histogram feature (-h)
>> is not used. I need more input! ;)
> 
> 
> Hehe. The histogram was the first thing I peered at, only to find out
> it's not even used. This might be a kernel thread switching bug in my
> code, but I find it hard to believe, because then even one thread
> probably wouldn't work.

... and the userspace test with two threads should fail.

Another player in this game is the RTDM layer itself. But there is no
much of it involved, and it's not really arch-dependent code.

Jan

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 254 bytes --]

Re: [Xenomai-core] latency kernel part crashes on ppc64

[Heikki Lindholm]

Jan Kiszka kirjoitti:
> Heikki Lindholm wrote:
> 
>>Jan Kiszka kirjoitti:
>>
>>
>>>Heikki Lindholm wrote:
>>>
>>>
>>>>Hi,
>>>>
>>>>Some recent changes (*cough* RTDM benchmark driver *cough*) broke kernel
>>>>mode benchmarking for ppc64. Previously klatency worked fine, but now
>>>>latency -t 1 crashes somewhere in xnpod_schedule. Jan, any pending
>>>>patches a comin'?
>>>
> 
> To get this clearly: You tested the old klatency(+front-end) on latest
> xeno and it worked? Or does this parse "the old klatency worked over old
> xeno on PPC64"?
> 
> Comparing the old test with the new framework, the major difference is
> that the old one only knew a single kernel RT-task. Its front-end was
> reading from a pipe and was therefore a pure linux program. Now we have
> two RT-tasks, one is even a shadow, and they use RT-IPC. Not sure if
> this really means that the bug must be in the benchmark suite...
> 
> 
>>>
>>>Nope, it should work as it is. But as Stelian also reported problems on
>>>his fresh ARM port with the in-kernel test, I cannot exclude that there
>>>/might/ be a problem in the benchmark.
>>>
>>>As I don't have any ppc64 hanging around somewhere, we will have to go
>>>through this together. Things I would like to know:
>>
>>
>>Dammit, I hoped you'd whip up a fix just from me noting a problem. Well,
>>all right then, I'll play along...;)
>>
>>
>>> o When and how does it crash? At start-up immediately? Or after a
>>>   while?
>>
>>
>>I inserted some serial debug prints and it gets two passes to
>>eval_outer_loop done (enter/exit function). After that it freezes.
>>Without the debug printing it dies with kernel access of illegal memory
>>at xnpod_schedule, which btw. has been quite a common place to die.
>>
>>
>>> o Are there any details / backtraces available with the crash?
>>
>>
>>Becaktrace limits to xnpod_schedule if I remember right.
>>
>>
>>> o Does -t2 work?
>>
>>
>>Umm. Probably not. See below.
> 
> 
> Arrgh, "probably" - when it's so easy to test...

Shoot! The "probably" operator was incorrect, "-t 2" did work.

-- hl

Re: [Xenomai-core] latency kernel part crashes on ppc64

[Gilles Chanteperdrix]

Heikki Lindholm wrote:
 > >  o When and how does it crash? At start-up immediately? Or after a
 > >    while?
 > 
 > I inserted some serial debug prints and it gets two passes to 
 > eval_outer_loop done (enter/exit function). After that it freezes. 
 > Without the debug printing it dies with kernel access of illegal memory 
 > at xnpod_schedule, which btw. has been quite a common place to die.

A way to know a bit more about this error would be to have a look at the
exact address given by the kernel oops in the disassembly of
xnpod_schedule and try and find which pointer is invalid. This is not
very subtle, but sometimes works.

One reason why there used to be some invalid pointers accesses in
xnpod_schedule was when the context switch routine (on x86) forgot to
declare a few output or clobbered registers. In this case, the compiler
may assume that the contents of the registers are correct, whereas the
context switch clobbered them.

-- 


					    Gilles Chanteperdrix.

Re: [Xenomai-core] latency kernel part crashes on ppc64

[Stelian Pop]

Le dimanche 08 janvier 2006 à 18:56 +0200, Heikki Lindholm a écrit :

> >>Some recent changes (*cough* RTDM benchmark driver *cough*) broke kernel
> >>mode benchmarking for ppc64. Previously klatency worked fine, but now
> >>latency -t 1 crashes somewhere in xnpod_schedule. Jan, any pending
> >>patches a comin'?

So it seems I'm not alone. 

I have done some additionnal debugging on this issue in the last days. I
still haven't find the bug but I narrowed it down a bit.
> 
> > 
> > Nope, it should work as it is. But as Stelian also reported problems on
> > his fresh ARM port with the in-kernel test, I cannot exclude that there
> > /might/ be a problem in the benchmark.
> > 
> > As I don't have any ppc64 hanging around somewhere, we will have to go
> > through this together. Things I would like to know:
> 
> Dammit, I hoped you'd whip up a fix just from me noting a problem. Well, 
> all right then, I'll play along...;)
> 
> >  o When and how does it crash? At start-up immediately? Or after a
> >    while?
> 
> I inserted some serial debug prints and it gets two passes to 
> eval_outer_loop done (enter/exit function). After that it freezes. 

It freezes exactly upon the invocation of rtdm_event_pulse() which
causes a scheduling. In xnpod_schedule, the scheduler queue has been
corrupted and this causes the illegal accesses.

> Without the debug printing it dies with kernel access of illegal memory 
> at xnpod_schedule, which btw. has been quite a common place to die.

Same for me.

> >  o Are there any details / backtraces available with the crash?
> 
> Becaktrace limits to xnpod_schedule if I remember right.

Same for me. But very often I don't even get a backtrace, it just hangs.

> >  o Does -t2 work?
> 
> Umm. Probably not. See below.

Heikki said in a later mail that it works for him, and so it does for me
too.

> >  o What happens if your disable "rtdm_event_pulse(&ctx->result_event);"
> >    in eval_outer_loop (thus no signalling of intermediate results during
> >    the test)? Does it still crash, maybe later during cleanup now?

> Doesn't freeze and can be exited with ctrl-c and even re-run.

Same for me.

Some additionnal information: I've disabled FPU handling in Xeno and it
doesn't change anything, it still crashes.

As I said before, the old klatency test does work reliably for me, with
the latest Xenomai.

I tried moving the 'display' thread into the kernel, and in this
configuration it does no longer crash.

I've started simplifying the code trying to get to the simplest code
which does have the problem. The results is at
http://www.popies.net/tmp/xenobug/bug.tgz if somebody wants to take a
look.

I'll be working on this again tomorrow...

Stelian.
-- 
Stelian Pop <stelian.pop@domain.hid>
Open Wide

Re: [Xenomai-core] latency kernel part crashes on ppc64

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 3868 bytes --]

Stelian Pop wrote:
> Le dimanche 08 janvier 2006 à 18:56 +0200, Heikki Lindholm a écrit :
> 
> 
>>>>Some recent changes (*cough* RTDM benchmark driver *cough*) broke kernel
>>>>mode benchmarking for ppc64. Previously klatency worked fine, but now
>>>>latency -t 1 crashes somewhere in xnpod_schedule. Jan, any pending
>>>>patches a comin'?
> 
> 
> So it seems I'm not alone. 
> 
> I have done some additionnal debugging on this issue in the last days. I
> still haven't find the bug but I narrowed it down a bit.
> 
>>>Nope, it should work as it is. But as Stelian also reported problems on
>>>his fresh ARM port with the in-kernel test, I cannot exclude that there
>>>/might/ be a problem in the benchmark.
>>>
>>>As I don't have any ppc64 hanging around somewhere, we will have to go
>>>through this together. Things I would like to know:
>>
>>Dammit, I hoped you'd whip up a fix just from me noting a problem. Well, 
>>all right then, I'll play along...;)
>>
>>
>>> o When and how does it crash? At start-up immediately? Or after a
>>>   while?
>>
>>I inserted some serial debug prints and it gets two passes to 
>>eval_outer_loop done (enter/exit function). After that it freezes. 
> 
> 
> It freezes exactly upon the invocation of rtdm_event_pulse() which
> causes a scheduling. In xnpod_schedule, the scheduler queue has been
> corrupted and this causes the illegal accesses.

Do you mean the synch-queue inside result_event or the global run-queue?
I saw in your test that you even moved result_event out of the context
structure and turned it into a global variable. So it seems that the
timerbench itself does not overwrite it.

Does it crash on the second or already first invocation of
rtdm_event_flush()?

> 
> 
>>Without the debug printing it dies with kernel access of illegal memory 
>>at xnpod_schedule, which btw. has been quite a common place to die.
> 
> 
> Same for me.
> 
> 
>>> o Are there any details / backtraces available with the crash?
>>
>>Becaktrace limits to xnpod_schedule if I remember right.
> 
> 
> Same for me. But very often I don't even get a backtrace, it just hangs.
> 
> 
>>> o Does -t2 work?
>>
>>Umm. Probably not. See below.
> 
> 
> Heikki said in a later mail that it works for him, and so it does for me
> too.
> 
> 
>>> o What happens if your disable "rtdm_event_pulse(&ctx->result_event);"
>>>   in eval_outer_loop (thus no signalling of intermediate results during
>>>   the test)? Does it still crash, maybe later during cleanup now?
> 
> 
>>Doesn't freeze and can be exited with ctrl-c and even re-run.
> 
> 
> Same for me.
> 
> Some additionnal information: I've disabled FPU handling in Xeno and it
> doesn't change anything, it still crashes.
> 
> As I said before, the old klatency test does work reliably for me, with
> the latest Xenomai.
> 
> I tried moving the 'display' thread into the kernel, and in this
> configuration it does no longer crash.

Hmm, the RTDM kernel entry/exit path? But what evil thing should happen
there? Weird.

> 
> I've started simplifying the code trying to get to the simplest code
> which does have the problem. The results is at
> http://www.popies.net/tmp/xenobug/bug.tgz if somebody wants to take a
> look.

Re-checked your test on x86, no problems (this was obvious, I did a lot
of tests the last week with that setup and that particular piece of code).

> 
> I'll be working on this again tomorrow...
> 

When you know now what piece of memory gets corrupted, you may add
checks or outputs of the content to the code, seeing which is the
offending function. And there is also a nice switch called
CONFIG_XENO_OPT_DEBUG which may provide some hints regarding incorrect
queue usages (although such things should have been triggered on other
archs as well).

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: [Xenomai-core] latency kernel part crashes on ppc64

[Philippe Gerum]

Jan Kiszka wrote:
> Stelian Pop wrote:
> 
>>Le dimanche 08 janvier 2006 à 18:56 +0200, Heikki Lindholm a écrit :
>>
>>
>>
>>>>>Some recent changes (*cough* RTDM benchmark driver *cough*) broke kernel
>>>>>mode benchmarking for ppc64. Previously klatency worked fine, but now
>>>>>latency -t 1 crashes somewhere in xnpod_schedule. Jan, any pending
>>>>>patches a comin'?
>>
>>
>>So it seems I'm not alone. 
>>
>>I have done some additionnal debugging on this issue in the last days. I
>>still haven't find the bug but I narrowed it down a bit.
>>
>>
>>>>Nope, it should work as it is. But as Stelian also reported problems on
>>>>his fresh ARM port with the in-kernel test, I cannot exclude that there
>>>>/might/ be a problem in the benchmark.
>>>>
>>>>As I don't have any ppc64 hanging around somewhere, we will have to go
>>>>through this together. Things I would like to know:
>>>

Hi guys, it's "/me too" time here for ppc. My mpc5200 board is jumping out of the 
window when running "./latency -t1", while "./latency -t2" works. No trace dump, 
hard freeze or spontaneous reboot, at your option. OPT_NUCLEUS_DEBUG spits 
nothing. No issue with the regular user-space test though.

x86 has no problem at all.

-- 

Philippe.

Re: [Xenomai-core] latency kernel part crashes on ppc64

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1578 bytes --]

Philippe Gerum wrote:
> Jan Kiszka wrote:
> 
>> Stelian Pop wrote:
>>
>>> Le dimanche 08 janvier 2006 à 18:56 +0200, Heikki Lindholm a écrit :
>>>
>>>
>>>
>>>>>> Some recent changes (*cough* RTDM benchmark driver *cough*) broke
>>>>>> kernel
>>>>>> mode benchmarking for ppc64. Previously klatency worked fine, but now
>>>>>> latency -t 1 crashes somewhere in xnpod_schedule. Jan, any pending
>>>>>> patches a comin'?
>>>
>>>
>>>
>>> So it seems I'm not alone.
>>> I have done some additionnal debugging on this issue in the last days. I
>>> still haven't find the bug but I narrowed it down a bit.
>>>
>>>
>>>>> Nope, it should work as it is. But as Stelian also reported
>>>>> problems on
>>>>> his fresh ARM port with the in-kernel test, I cannot exclude that
>>>>> there
>>>>> /might/ be a problem in the benchmark.
>>>>>
>>>>> As I don't have any ppc64 hanging around somewhere, we will have to go
>>>>> through this together. Things I would like to know:
>>>>
>>>>
> 
> Hi guys, it's "/me too" time here for ppc. My mpc5200 board is jumping
> out of the window when running "./latency -t1", while "./latency -t2"
> works. No trace dump, hard freeze or spontaneous reboot, at your option.
> OPT_NUCLEUS_DEBUG spits nothing. No issue with the regular user-space
> test though.
> 
> x86 has no problem at all.
> 

Ok, three smart guys, all having the same bug on their systems - who
will be first shouting "I got it"? If it's an issue in my code, I'm
going to pay you a beer (which I likely have to schedule already). ;)

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 254 bytes --]

Re: [Xenomai-core] latency kernel part crashes on ppc64

[Philippe Gerum]

Jan Kiszka wrote:
> Philippe Gerum wrote:
> 
>>Jan Kiszka wrote:
>>
>>
>>>Stelian Pop wrote:
>>>
>>>
>>>>Le dimanche 08 janvier 2006 à 18:56 +0200, Heikki Lindholm a écrit :
>>>>
>>>>
>>>>
>>>>
>>>>>>>Some recent changes (*cough* RTDM benchmark driver *cough*) broke
>>>>>>>kernel
>>>>>>>mode benchmarking for ppc64. Previously klatency worked fine, but now
>>>>>>>latency -t 1 crashes somewhere in xnpod_schedule. Jan, any pending
>>>>>>>patches a comin'?
>>>>
>>>>
>>>>
>>>>So it seems I'm not alone.
>>>>I have done some additionnal debugging on this issue in the last days. I
>>>>still haven't find the bug but I narrowed it down a bit.
>>>>
>>>>
>>>>
>>>>>>Nope, it should work as it is. But as Stelian also reported
>>>>>>problems on
>>>>>>his fresh ARM port with the in-kernel test, I cannot exclude that
>>>>>>there
>>>>>>/might/ be a problem in the benchmark.
>>>>>>
>>>>>>As I don't have any ppc64 hanging around somewhere, we will have to go
>>>>>>through this together. Things I would like to know:
>>>>>
>>>>>
>>Hi guys, it's "/me too" time here for ppc. My mpc5200 board is jumping
>>out of the window when running "./latency -t1", while "./latency -t2"
>>works. No trace dump, hard freeze or spontaneous reboot, at your option.
>>OPT_NUCLEUS_DEBUG spits nothing. No issue with the regular user-space
>>test though.
>>
>>x86 has no problem at all.
>>
> 
> 
> Ok, three smart guys, all having the same bug on their systems - who
> will be first shouting "I got it"? If it's an issue in my code, I'm
> going to pay you a beer (which I likely have to schedule already). ;)
> 

Likely not this time (keep it cold anyway, who knows); I strongly suspect a bug in 
xnarch_switch_to() for all archs but x86 which causes the sampling task code to be 
spuriously rescheduled over the root thread after a few rounds of 
rtdm_task_sleep_until. This might happen when a user-space shadow attempts to 
switch to a kernel-based task. Sounds rather worrying eh?! More later.

-- 

Philippe.

Re: [Xenomai-core] latency kernel part crashes on ppc64

[Gilles Chanteperdrix]

[-- Attachment #1: message body and .signature --]
[-- Type: text/plain, Size: 573 bytes --]

Philippe Gerum wrote:
 > Likely not this time (keep it cold anyway, who knows); I strongly suspect a bug in 
 > xnarch_switch_to() for all archs but x86

Now that you are talking about it, I may be the one who owes a beer to
everyone by first having put a bug in ia64 context-switch code... If I
am not wrong, the bug should be observed on ia64 too. Unfortunately, I
am unable to compile my ia64 kernel right now, so I wrote a patch for
power PC, and would be glad if some power PC owner could try it.

It should work on PPC 64 too...

-- 


					    Gilles Chanteperdrix.

[-- Attachment #2: xeno-ppc-switch.diff --]
[-- Type: text/plain, Size: 1735 bytes --]

Index: include/asm-powerpc/system.h
===================================================================
--- include/asm-powerpc/system.h	(revision 410)
+++ include/asm-powerpc/system.h	(working copy)
@@ -182,13 +182,19 @@
 {
     struct task_struct *prev = out_tcb->active_task;
     struct task_struct *next = in_tcb->user_task;
+    static unsigned long last_ksp;
+    static xnarchtcb_t *last_tcb;
 
+    last_tcb = out_tcb;
     in_tcb->active_task = next ?: prev;
 
     if (next && next != prev) /* Switch to new user-space thread? */
 	{
 	struct mm_struct *mm = next->active_mm;
 
+        if (prev != out_tcb->user_task)
+            last_ksp = prev->thread.ksp;
+
 	/* Switch the mm context.*/
 
 #ifdef CONFIG_PPC64
@@ -245,6 +251,22 @@
     else
         /* Kernel-to-kernel context switch. */
         rthal_switch_context(out_tcb->kspp,in_tcb->kspp);
+
+    /* If we are not the epilogue of a regular linux schedule(),... */
+    if (likely(test_bit(xnarch_current_cpu(),&rthal_cpu_realtime)) &&
+        /* we are a user-space thread,... */
+        out_tcb->user_task &&
+        /* the last context switch used linux switch routine,... */
+        last_tcb->active_task != out_tcb->user_task &&
+        /* but the last xenomai context was a kernel thread,... */
+        last_tcb->user_task != last_tcb->active_task)
+        {
+        /* then linux context switch routine saved kernel thread stack pointer
+           in last user-space context. We hence put the stack pointers in the
+           right place. */
+        last_tcb->ksp = last_tcb->active_task->thread.ksp;
+        last_tcb->active_task->thread.ksp = last_ksp;
+        }
 }
 
 static inline void xnarch_finalize_and_switch (xnarchtcb_t *dead_tcb,

Re: [Xenomai-core] latency kernel part crashes on ppc64

[Philippe Gerum]

Gilles Chanteperdrix wrote:
> Philippe Gerum wrote:
>  > Likely not this time (keep it cold anyway, who knows); I strongly suspect a bug in 
>  > xnarch_switch_to() for all archs but x86
> 
> Now that you are talking about it, I may be the one who owes a beer to
> everyone by first having put a bug in ia64 context-switch code... If I
> am not wrong, the bug should be observed on ia64 too. Unfortunately, I
> am unable to compile my ia64 kernel right now, so I wrote a patch for
> power PC, and would be glad if some power PC owner could try it.
> 

Nope, same lockup with hybrid scheduling.

> It should work on PPC 64 too...
> 
> 
> 
> ------------------------------------------------------------------------
> 
> Index: include/asm-powerpc/system.h
> ===================================================================
> --- include/asm-powerpc/system.h	(revision 410)
> +++ include/asm-powerpc/system.h	(working copy)
> @@ -182,13 +182,19 @@
>  {
>      struct task_struct *prev = out_tcb->active_task;
>      struct task_struct *next = in_tcb->user_task;
> +    static unsigned long last_ksp;
> +    static xnarchtcb_t *last_tcb;
>  
> +    last_tcb = out_tcb;
>      in_tcb->active_task = next ?: prev;
>  
>      if (next && next != prev) /* Switch to new user-space thread? */
>  	{
>  	struct mm_struct *mm = next->active_mm;
>  
> +        if (prev != out_tcb->user_task)
> +            last_ksp = prev->thread.ksp;
> +
>  	/* Switch the mm context.*/
>  
>  #ifdef CONFIG_PPC64
> @@ -245,6 +251,22 @@
>      else
>          /* Kernel-to-kernel context switch. */
>          rthal_switch_context(out_tcb->kspp,in_tcb->kspp);
> +
> +    /* If we are not the epilogue of a regular linux schedule(),... */
> +    if (likely(test_bit(xnarch_current_cpu(),&rthal_cpu_realtime)) &&
> +        /* we are a user-space thread,... */
> +        out_tcb->user_task &&
> +        /* the last context switch used linux switch routine,... */
> +        last_tcb->active_task != out_tcb->user_task &&
> +        /* but the last xenomai context was a kernel thread,... */
> +        last_tcb->user_task != last_tcb->active_task)
> +        {
> +        /* then linux context switch routine saved kernel thread stack pointer
> +           in last user-space context. We hence put the stack pointers in the
> +           right place. */
> +        last_tcb->ksp = last_tcb->active_task->thread.ksp;
> +        last_tcb->active_task->thread.ksp = last_ksp;
> +        }
>  }
>  
>  static inline void xnarch_finalize_and_switch (xnarchtcb_t *dead_tcb,


-- 

Philippe.

Re: [Xenomai-core] latency kernel part crashes on ppc64

[Gilles Chanteperdrix]

Philippe Gerum wrote:
 > Gilles Chanteperdrix wrote:
 > > Philippe Gerum wrote:
 > >  > Likely not this time (keep it cold anyway, who knows); I strongly suspect a bug in 
 > >  > xnarch_switch_to() for all archs but x86
 > > 
 > > Now that you are talking about it, I may be the one who owes a beer to
 > > everyone by first having put a bug in ia64 context-switch code... If I
 > > am not wrong, the bug should be observed on ia64 too. Unfortunately, I
 > > am unable to compile my ia64 kernel right now, so I wrote a patch for
 > > power PC, and would be glad if some power PC owner could try it.
 > > 
 > 
 > Nope, same lockup with hybrid scheduling.

The latency -t 1 crash may be observed on ia64 too.

But it does not seem to be due to the bug I was suspecting in context
switch code.

-- 


					    Gilles Chanteperdrix.

Re: [Xenomai-core] latency kernel part crashes on ppc64

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1021 bytes --]

Gilles Chanteperdrix wrote:
> Philippe Gerum wrote:
>  > Gilles Chanteperdrix wrote:
>  > > Philippe Gerum wrote:
>  > >  > Likely not this time (keep it cold anyway, who knows); I strongly suspect a bug in 
>  > >  > xnarch_switch_to() for all archs but x86
>  > > 
>  > > Now that you are talking about it, I may be the one who owes a beer to
>  > > everyone by first having put a bug in ia64 context-switch code... If I
>  > > am not wrong, the bug should be observed on ia64 too. Unfortunately, I
>  > > am unable to compile my ia64 kernel right now, so I wrote a patch for
>  > > power PC, and would be glad if some power PC owner could try it.
>  > > 
>  > 
>  > Nope, same lockup with hybrid scheduling.
> 
> The latency -t 1 crash may be observed on ia64 too.
> 
> But it does not seem to be due to the bug I was suspecting in context
> switch code.
> 

What about the PPC world? I saw some SVN commits claiming to fix this
(also for blackfin) - but there was no "Hey, it's fixed now!" on the
mailing list.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 254 bytes --]

[Xenomai-core] latency kernel part fixed

[Philippe Gerum]

Jan Kiszka wrote:
> Gilles Chanteperdrix wrote:
> 
>>Philippe Gerum wrote:
>> > Gilles Chanteperdrix wrote:
>> > > Philippe Gerum wrote:
>> > >  > Likely not this time (keep it cold anyway, who knows); I strongly suspect a bug in 
>> > >  > xnarch_switch_to() for all archs but x86
>> > > 
>> > > Now that you are talking about it, I may be the one who owes a beer to
>> > > everyone by first having put a bug in ia64 context-switch code... If I
>> > > am not wrong, the bug should be observed on ia64 too. Unfortunately, I
>> > > am unable to compile my ia64 kernel right now, so I wrote a patch for
>> > > power PC, and would be glad if some power PC owner could try it.
>> > > 
>> > 
>> > Nope, same lockup with hybrid scheduling.
>>
>>The latency -t 1 crash may be observed on ia64 too.
>>
>>But it does not seem to be due to the bug I was suspecting in context
>>switch code.
>>
> 
> 
> What about the PPC world? I saw some SVN commits claiming to fix this
> (also for blackfin) - but there was no "Hey, it's fixed now!" on the
> mailing list.
>

Ok... "Hey, it's fixed now!"

Hybrid scheduling (kernel and user-space Xeno threads combination) has been fixed 
for ppc32, ppc64, ia64 and the Blackfin. Issue fixed on the ARM port to come too. 
Some tests remain to be conducted on the ppc64 though. The rest has already been 
validated. Fixes are available from the SVN trunk/.

This said, here is my next Xmas wishlist I'd really appreciate you guys anticipate 
from, say 11 months and a couple of weeks: if you have any of the above archs, 
please run both the kernel and user-space latency tests on such hw:

Basically, this boils down to building Jan's timerbench test into the kernel or as 
a separate module, then run the following tests on the said kernel:

$ sudo ./latency -t0
$ sudo ./latency -t1
$ sudo ./latency -t2

Special note to the ppc32 people: I would be interested in having your feedback 
about switching on the ALTIVEC and SPE supports at kernel level, on platforms that 
do not have those beasts, and see if all tests survive this.

In any case, I'm eventually going to warn about such configurations at the very 
least, since relying on FTR fixups right in the middle of the fast path for all 
context switches is kind of, mmh, silly?! Nevertheless, knowing about the result 
might be important in the future.

TIA,

-- 

Philippe.

Re: [Xenomai-core] latency kernel part fixed

[Wolfgang Grandegger]

Philippe Gerum wrote:
> Jan Kiszka wrote:
> 
>> Gilles Chanteperdrix wrote:
>>
>>> Philippe Gerum wrote:
>>> > Gilles Chanteperdrix wrote:
>>> > > Philippe Gerum wrote:
>>> > >  > Likely not this time (keep it cold anyway, who knows); I 
>>> strongly suspect a bug in > >  > xnarch_switch_to() for all archs but 
>>> x86
>>> > > > > Now that you are talking about it, I may be the one who owes 
>>> a beer to
>>> > > everyone by first having put a bug in ia64 context-switch code... 
>>> If I
>>> > > am not wrong, the bug should be observed on ia64 too. 
>>> Unfortunately, I
>>> > > am unable to compile my ia64 kernel right now, so I wrote a patch 
>>> for
>>> > > power PC, and would be glad if some power PC owner could try it.
>>> > > > > Nope, same lockup with hybrid scheduling.
>>>
>>> The latency -t 1 crash may be observed on ia64 too.
>>>
>>> But it does not seem to be due to the bug I was suspecting in context
>>> switch code.
>>>
>>
>>
>> What about the PPC world? I saw some SVN commits claiming to fix this
>> (also for blackfin) - but there was no "Hey, it's fixed now!" on the
>> mailing list.
>>
> 
> Ok... "Hey, it's fixed now!"
> 
> Hybrid scheduling (kernel and user-space Xeno threads combination) has 
> been fixed for ppc32, ppc64, ia64 and the Blackfin. Issue fixed on the 
> ARM port to come too. Some tests remain to be conducted on the ppc64 
> though. The rest has already been validated. Fixes are available from 
> the SVN trunk/.
> 
> This said, here is my next Xmas wishlist I'd really appreciate you guys 
> anticipate from, say 11 months and a couple of weeks: if you have any of 
> the above archs, please run both the kernel and user-space latency tests 
> on such hw:

OK, I will do tests on a few PowerPC boards (4xx, 8xx, 8260).

> Basically, this boils down to building Jan's timerbench test into the 
> kernel or as a separate module, then run the following tests on the said 
> kernel:
> 
> $ sudo ./latency -t0
> $ sudo ./latency -t1
> $ sudo ./latency -t2
> 
> Special note to the ppc32 people: I would be interested in having your 
> feedback about switching on the ALTIVEC and SPE supports at kernel 
> level, on platforms that do not have those beasts, and see if all tests 
> survive this.
> 
> In any case, I'm eventually going to warn about such configurations at 
> the very least, since relying on FTR fixups right in the middle of the 
> fast path for all context switches is kind of, mmh, silly?! 
> Nevertheless, knowing about the result might be important in the future.
> 
> TIA,
> 

Re: [Xenomai-core] latency kernel part crashes on ppc64

[Gilles Chanteperdrix]

Jan Kiszka wrote:
 > Gilles Chanteperdrix wrote:
 > > But it does not seem to be due to the bug I was suspecting in context
 > > switch code.
 > > 
 > 
 > What about the PPC world? I saw some SVN commits claiming to fix this
 > (also for blackfin) - but there was no "Hey, it's fixed now!" on the
 > mailing list.

Actually, the bug I suspected was the real cause of trouble on
ia64. But, if you look at Philippe's commit, you will see that there
were a couple of other bugs (probably all my work) on this architecture,
hence my previous, wrong, claim.

-- 


					    Gilles Chanteperdrix.