[Xenomai-core] [BUG] recursive fault on cyclictest termination -- scalable sched?

[Xenomai-core] [BUG] recursive fault on cyclictest termination -- scalable sched?

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 631 bytes --]

People, we have some new troubles:

I'm reproducibly getting recursive faults on termination of cyclictest
via ^C. It's all standard here: ipipe 1.8-02, Xenomai trunk #2469, no
weird patches of mine.

Something goes utterly wrong, the debugger currently points into
xnshadow_relax->rpi_push, and there into some queuing operation. Note
that I have XENO_OPT_SCALABLE_SCHED on in my config, also
XENO_OPT_PRIOCPL. Without XENO_OPT_SCALABLE_SCHED, things seems to work
find. Who did last work on this? What was fixed?

Jan


PS: Found a few more bugs in my latest instrumentation patches. Grmbl.
Fixes will follow.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-core] [BUG] recursive fault on cyclictest termination -- scalable sched?

[Gilles Chanteperdrix]

Jan Kiszka wrote:
> People, we have some new troubles:
> 
> I'm reproducibly getting recursive faults on termination of cyclictest
> via ^C. It's all standard here: ipipe 1.8-02, Xenomai trunk #2469, no
> weird patches of mine.
> 
> Something goes utterly wrong, the debugger currently points into
> xnshadow_relax->rpi_push, and there into some queuing operation. Note
> that I have XENO_OPT_SCALABLE_SCHED on in my config, also
> XENO_OPT_PRIOCPL. Without XENO_OPT_SCALABLE_SCHED, things seems to work
> find. Who did last work on this? What was fixed?

Looking at svn log, you did the last modification. What about activating
queue debugging ?

-- 
                                                 Gilles Chanteperdrix

Re: [Xenomai-core] [BUG] recursive fault on cyclictest termination -- scalable sched?

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2246 bytes --]

Gilles Chanteperdrix wrote:
> Jan Kiszka wrote:
>> People, we have some new troubles:
>>
>> I'm reproducibly getting recursive faults on termination of cyclictest
>> via ^C. It's all standard here: ipipe 1.8-02, Xenomai trunk #2469, no
>> weird patches of mine.
>>
>> Something goes utterly wrong, the debugger currently points into
>> xnshadow_relax->rpi_push, and there into some queuing operation. Note
>> that I have XENO_OPT_SCALABLE_SCHED on in my config, also
>> XENO_OPT_PRIOCPL. Without XENO_OPT_SCALABLE_SCHED, things seems to work
>> find. Who did last work on this? What was fixed?
> 
> Looking at svn log, you did the last modification. What about activating
> queue debugging ?
> 

Well, I remember that 64-bit issue now, but it was my patch IIRC.

Anyway, good suggestion:

T: 0 (  824) P:99 I:    1000 C:    1483 Min:       3 Act:     349 Avg:    1602 Max:    3240
[  474.912841] Xenomai: fatal: corrupted queue, qslot->elems=0, qslot=c0614a28 at include/xenomai/nucleus/queue.h:684
[  474.912921]  CPU  PID    PRI      TIMEOUT  STAT      NAME
[  474.912941]    0  0       -1      0        00500088  ROOT
[  474.912960]    0  823      0      0        00300380  cyclictest
[  474.912980] >  0  824      0      0        00300180  cyclictest
[  474.913000] Master time base: clock=1043585105469
[  474.913017] 
[  474.915241]        c11e7eb4 00000000 00000000 c0614a28 c11e7ed8 c0104c6e c033f860 00000000 
[  474.915762]        00000103 c11e7f18 c0154f02 c033fd86 c115c000 c0614a28 c03428c4 000002ac 
[  474.916378]        c015657d 00000000 c1180b30 ffffffff c1180d94 00000000 c1180b30 00000000 
[  474.916874] Call Trace:
[  474.917034]  [<c010451a>] show_trace_log_lvl+0x1a/0x30
[  474.917422]  [<c01045e1>] show_stack_log_lvl+0xb1/0xe0
[  474.917693]  [<c0104c6e>] show_stack+0x2e/0x40
[  474.917920]  [<c0154f02>] rpi_push+0x192/0x3a0
[  474.918146]  [<c0156a30>] xnshadow_relax+0x50/0x1c0
[  474.918394]  [<c01570c0>] hisyscall_event+0xd0/0x290
[  474.918647]  [<c0142a2e>] __ipipe_dispatch_event+0x8e/0x140
[  474.918917]  [<c010d97e>] __ipipe_syscall_root+0x3e/0xf0
[  474.919292]  [<c0102e59>] system_call+0x29/0x41
[  474.919534]  =======================

Any comment?

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-core] [BUG] recursive fault on cyclictest termination -- scalable sched?

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 905 bytes --]

Jan Kiszka wrote:
> Gilles Chanteperdrix wrote:
>> Jan Kiszka wrote:
>>> People, we have some new troubles:
>>>
>>> I'm reproducibly getting recursive faults on termination of cyclictest
>>> via ^C. It's all standard here: ipipe 1.8-02, Xenomai trunk #2469, no
>>> weird patches of mine.
>>>
>>> Something goes utterly wrong, the debugger currently points into
>>> xnshadow_relax->rpi_push, and there into some queuing operation. Note
>>> that I have XENO_OPT_SCALABLE_SCHED on in my config, also
>>> XENO_OPT_PRIOCPL. Without XENO_OPT_SCALABLE_SCHED, things seems to work
>>> find. Who did last work on this? What was fixed?
>> Looking at svn log, you did the last modification. What about activating
>> queue debugging ?
>>
> 
> Well, I remember that 64-bit issue now, but it was my patch IIRC.
                                                   ^^^not
(hell, I need the weekend)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-core] [BUG] recursive fault on cyclictest termination -- scalable sched?

[Philippe Gerum]

On Fri, 2007-05-25 at 15:55 +0200, Jan Kiszka wrote:
> Gilles Chanteperdrix wrote:
> > Jan Kiszka wrote:
> >> People, we have some new troubles:
> >>
> >> I'm reproducibly getting recursive faults on termination of cyclictest
> >> via ^C. It's all standard here: ipipe 1.8-02, Xenomai trunk #2469, no
> >> weird patches of mine.
> >>
> >> Something goes utterly wrong, the debugger currently points into
> >> xnshadow_relax->rpi_push, and there into some queuing operation. Note
> >> that I have XENO_OPT_SCALABLE_SCHED on in my config, also
> >> XENO_OPT_PRIOCPL. Without XENO_OPT_SCALABLE_SCHED, things seems to work
> >> find. Who did last work on this? What was fixed?
> > 
> > Looking at svn log, you did the last modification. What about activating
> > queue debugging ?
> > 
> 
> Well, I remember that 64-bit issue now, but it was my patch IIRC.
> 
> Anyway, good suggestion:
> 
> T: 0 (  824) P:99 I:    1000 C:    1483 Min:       3 Act:     349 Avg:    1602 Max:    3240
> [  474.912841] Xenomai: fatal: corrupted queue, qslot->elems=0, qslot=c0614a28 at include/xenomai/nucleus/queue.h:684
> [  474.912921]  CPU  PID    PRI      TIMEOUT  STAT      NAME
> [  474.912941]    0  0       -1      0        00500088  ROOT
> [  474.912960]    0  823      0      0        00300380  cyclictest
> [  474.912980] >  0  824      0      0        00300180  cyclictest
> [  474.913000] Master time base: clock=1043585105469
> [  474.913017] 
> [  474.915241]        c11e7eb4 00000000 00000000 c0614a28 c11e7ed8 c0104c6e c033f860 00000000 
> [  474.915762]        00000103 c11e7f18 c0154f02 c033fd86 c115c000 c0614a28 c03428c4 000002ac 
> [  474.916378]        c015657d 00000000 c1180b30 ffffffff c1180d94 00000000 c1180b30 00000000 
> [  474.916874] Call Trace:
> [  474.917034]  [<c010451a>] show_trace_log_lvl+0x1a/0x30
> [  474.917422]  [<c01045e1>] show_stack_log_lvl+0xb1/0xe0
> [  474.917693]  [<c0104c6e>] show_stack+0x2e/0x40
> [  474.917920]  [<c0154f02>] rpi_push+0x192/0x3a0
> [  474.918146]  [<c0156a30>] xnshadow_relax+0x50/0x1c0
> [  474.918394]  [<c01570c0>] hisyscall_event+0xd0/0x290
> [  474.918647]  [<c0142a2e>] __ipipe_dispatch_event+0x8e/0x140
> [  474.918917]  [<c010d97e>] __ipipe_syscall_root+0x3e/0xf0
> [  474.919292]  [<c0102e59>] system_call+0x29/0x41
> [  474.919534]  =======================
> 
> Any comment?
> 

Yeah. This is the exact bug I told you I was chasing some moons ago on
qemu/x86_64 and which I can't reproduce anywhere else (on real hw for
instance), glad to see I'm not alone in the twilight zone anymore. :o)
(Btw, this issue predates any recent change; this is something I've seen
popping up more than six weeks ago on my setup).

> Jan
> 
> _______________________________________________
> Xenomai-core mailing list
> Xenomai-core@domain.hid
> https://mail.gna.org/listinfo/xenomai-core
-- 
Philippe.

Re: [Xenomai-core] [BUG] recursive fault on cyclictest termination -- scalable sched?

[Gilles Chanteperdrix]

Philippe Gerum wrote:
> On Fri, 2007-05-25 at 15:55 +0200, Jan Kiszka wrote:
> 
>>Gilles Chanteperdrix wrote:
>>
>>>Jan Kiszka wrote:
>>>
>>>>People, we have some new troubles:
>>>>
>>>>I'm reproducibly getting recursive faults on termination of cyclictest
>>>>via ^C. It's all standard here: ipipe 1.8-02, Xenomai trunk #2469, no
>>>>weird patches of mine.
>>>>
>>>>Something goes utterly wrong, the debugger currently points into
>>>>xnshadow_relax->rpi_push, and there into some queuing operation. Note
>>>>that I have XENO_OPT_SCALABLE_SCHED on in my config, also
>>>>XENO_OPT_PRIOCPL. Without XENO_OPT_SCALABLE_SCHED, things seems to work
>>>>find. Who did last work on this? What was fixed?
>>>
>>>Looking at svn log, you did the last modification. What about activating
>>>queue debugging ?
>>>
>>
>>Well, I remember that 64-bit issue now, but it was my patch IIRC.
>>
>>Anyway, good suggestion:
>>
>>T: 0 (  824) P:99 I:    1000 C:    1483 Min:       3 Act:     349 Avg:    1602 Max:    3240
>>[  474.912841] Xenomai: fatal: corrupted queue, qslot->elems=0, qslot=c0614a28 at include/xenomai/nucleus/queue.h:684
>>[  474.912921]  CPU  PID    PRI      TIMEOUT  STAT      NAME
>>[  474.912941]    0  0       -1      0        00500088  ROOT
>>[  474.912960]    0  823      0      0        00300380  cyclictest
>>[  474.912980] >  0  824      0      0        00300180  cyclictest
>>[  474.913000] Master time base: clock=1043585105469
>>[  474.913017] 
>>[  474.915241]        c11e7eb4 00000000 00000000 c0614a28 c11e7ed8 c0104c6e c033f860 00000000 
>>[  474.915762]        00000103 c11e7f18 c0154f02 c033fd86 c115c000 c0614a28 c03428c4 000002ac 
>>[  474.916378]        c015657d 00000000 c1180b30 ffffffff c1180d94 00000000 c1180b30 00000000 
>>[  474.916874] Call Trace:
>>[  474.917034]  [<c010451a>] show_trace_log_lvl+0x1a/0x30
>>[  474.917422]  [<c01045e1>] show_stack_log_lvl+0xb1/0xe0
>>[  474.917693]  [<c0104c6e>] show_stack+0x2e/0x40
>>[  474.917920]  [<c0154f02>] rpi_push+0x192/0x3a0
>>[  474.918146]  [<c0156a30>] xnshadow_relax+0x50/0x1c0
>>[  474.918394]  [<c01570c0>] hisyscall_event+0xd0/0x290
>>[  474.918647]  [<c0142a2e>] __ipipe_dispatch_event+0x8e/0x140
>>[  474.918917]  [<c010d97e>] __ipipe_syscall_root+0x3e/0xf0
>>[  474.919292]  [<c0102e59>] system_call+0x29/0x41
>>[  474.919534]  =======================
>>
>>Any comment?
>>
> 
> 
> Yeah. This is the exact bug I told you I was chasing some moons ago on
> qemu/x86_64 and which I can't reproduce anywhere else (on real hw for
> instance), glad to see I'm not alone in the twilight zone anymore. :o)
> (Btw, this issue predates any recent change; this is something I've seen
> popping up more than six weeks ago on my setup).

A recent change that might change the behaviour is the direct tsc access
by clock_gettime in user-space. This may make the bug win a race.

-- 
                                                 Gilles Chanteperdrix

Re: [Xenomai-core] [BUG] recursive fault on cyclictest termination -- scalable sched?

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 282 bytes --]

Daniel Schnell wrote:
> Well, that may be related to the bug I am seeing when I am ending cyclictest with Ctrl-C, Recursive kernel Ooops and this is on plain 2.3.1

My tests all ran over trunk, but you may easily verify the relation by
enabling XENO_OPT_DEBUG_QUEUES.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-core] [BUG] recursive fault on cyclictest termination -- scalable sched?

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 928 bytes --]

Daniel Schnell wrote:
> Well, that may be related to the bug I am seeing when I am ending cyclictest with Ctrl-C, Recursive kernel Ooops and this is on plain 2.3.1
> 

2.3.1 is definitely affected by this bug as well, as it also gained the
prio-coupling rework. But you have to have SCALABLE_SCHED enabled.

I got it. It is - as usual - a trivial one-liner:

--- xenomai.orig/ksrc/nucleus/shadow.c
+++ xenomai/ksrc/nucleus/shadow.c
@@ -186,7 +186,7 @@ static inline void rpi_init(void)
 
 static inline void rpi_init_gk(struct __gatekeeper *gk)
 {
-	sched_initpq(&gk->rpislot.threadq, xnqueue_down, XNCORE_NR_PRIO);
+	sched_initpq(&gk->rpislot.threadq, xnqueue_down, XNCORE_MAX_PRIO);
 }
 
 static inline void rpi_none(xnthread_t *thread)


Happy weekend, =8)

Jan


PS: The maximum priority level has initialised to a wrong value, causing
threads with prio 0 to be placed out of the queue array.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-core] [BUG] recursive fault on cyclictest termination -- scalable sched?

[Philippe Gerum]

On Fri, 2007-05-25 at 17:32 +0200, Jan Kiszka wrote:
> Daniel Schnell wrote:
> > Well, that may be related to the bug I am seeing when I am ending cyclictest with Ctrl-C, Recursive kernel Ooops and this is on plain 2.3.1
> > 
> 
> 2.3.1 is definitely affected by this bug as well, as it also gained the
> prio-coupling rework. But you have to have SCALABLE_SCHED enabled.
> 
> I got it. It is - as usual - a trivial one-liner:
> 

Actually, whatever we passed to the multi-level queue constructor was
lacking normalization wrt the priority scale currently used by the
client skin. I've sanitize this code for the most painful part, but we
need more than this for 2.4. Hey! nice, the bottom of my TODO list has
just reached the basement of my house...

> --- xenomai.orig/ksrc/nucleus/shadow.c
> +++ xenomai/ksrc/nucleus/shadow.c
> @@ -186,7 +186,7 @@ static inline void rpi_init(void)
>  
>  static inline void rpi_init_gk(struct __gatekeeper *gk)
>  {
> -	sched_initpq(&gk->rpislot.threadq, xnqueue_down, XNCORE_NR_PRIO);
> +	sched_initpq(&gk->rpislot.threadq, xnqueue_down, XNCORE_MAX_PRIO);
>  }
>  
>  static inline void rpi_none(xnthread_t *thread)
> 
> 
> Happy weekend, =8)
> 
> Jan
> 
> 
> PS: The maximum priority level has initialised to a wrong value, causing
> threads with prio 0 to be placed out of the queue array.
> 
-- 
Philippe.

Re: [Xenomai-core] [BUG] recursive fault on cyclictest termination -- scalable sched?

[Gilles Chanteperdrix]

Jan Kiszka wrote:
> People, we have some new troubles:
> 
> I'm reproducibly getting recursive faults on termination of cyclictest
> via ^C. It's all standard here: ipipe 1.8-02, Xenomai trunk #2469, no
> weird patches of mine.
> 
> Something goes utterly wrong, the debugger currently points into
> xnshadow_relax->rpi_push, and there into some queuing operation. Note
> that I have XENO_OPT_SCALABLE_SCHED on in my config, also
> XENO_OPT_PRIOCPL. Without XENO_OPT_SCALABLE_SCHED, things seems to work
> find. Who did last work on this? What was fixed?

cyclictest termination is rather strange, to say the least. The first
SIGTERM terminates the application, the other SIGTERM never get sent,
pthread_join never get called. I would recommend replacing the SIGTERM,
join loop, with a pthread_cancel loop followed by a pthread_join loop
(running a loop with pthread_cancel followed by pthread_join may lead to
deadlocks).

-- 
                                                 Gilles Chanteperdrix