Re: [Xenomai-core] Xenomai v2.4-rc4: freeze with RTAI skin, fine with other skins

Re: [Xenomai-core] Xenomai v2.4-rc4: freeze with RTAI skin, fine with other skins

[Fillod Stephane]

Philippe Gerum wrote:
>Fillod Stephane wrote:
>> For the legacy RTAI application to load, the attached patch was
>> necessary.
>> The patch against ksrc/skins/rtai/shm.c is somewhat defeating the
>> purpose
>> of a lower XNCORE_PAGE_SIZE, so a better fix might be expected.
>> 
>
>This one should prevent -EINVAL from being returned. Hopefully.

Nope, it doesn't :-(
Most probably because still (hdrsize + 2 * pagesize > heapsize).

[..] 
>> In the mean time, does anyone have a clue where to look particularly?
>> 
>
>Hard to say at this point. Since the nucleus watchdog does not trigger,
>you may want to try disabling X86_UP_IOAPIC while keeping X86_UP_APIC,
>and arm the kernel NMI watchdog on the LAPIC (nmi_watchdog=2). You may
>be lucky and have a backtrace after the freeze.
>
>PS: maybe enabling all the nucleus debug options would catch something
too.

Thanks for the ideas. It looks like part of the problem came from memory
corruption (access beyond heap) in the application (not so stable app
in the end, he!). I'm still experiencing random freeze when freeing
resources when stopping application, but runtime is stable.
To be honest, the freeze was not a perfect freeze, I got just a one line
trace from do_page_fault(), with no backtrace or pointer. Unhelpful.
I have updated http://xenomai.org/index.php/FAQs

-- 
Stephane

Re: [Xenomai-core] Xenomai v2.4-rc4: freeze with RTAI skin, fine with other skins

[Philippe Gerum]

Fillod Stephane wrote:
> Philippe Gerum wrote:
>> Fillod Stephane wrote:
>>> For the legacy RTAI application to load, the attached patch was
>>> necessary.
>>> The patch against ksrc/skins/rtai/shm.c is somewhat defeating the
>>> purpose
>>> of a lower XNCORE_PAGE_SIZE, so a better fix might be expected.
>>>
>> This one should prevent -EINVAL from being returned. Hopefully.
> 
> Nope, it doesn't :-(
> Most probably because still (hdrsize + 2 * pagesize > heapsize).
> 

This rounding was missing too. We need the previous one for kernel local
 heaps, and the one below to meet the stricter PAGE_SIZE constraint for
shareable heaps.

--- ksrc/nucleus/heap.c	(revision 3095)
+++ ksrc/nucleus/heap.c	(working copy)
@@ -1103,7 +1103,7 @@
 	spl_t s;
 	int err;

-	heapsize = PAGE_ALIGN(heapsize);
+	heapsize = xnheap_rounded_size(heapsize, PAGE_SIZE);
 	heapbase = __alloc_and_reserve_heap(heapsize, memflags);

 	if (!heapbase)

-- 
Philippe.

Re: [Xenomai-core] Xenomai v2.4-rc4: freeze with RTAI skin, fine with other skins

[Fillod Stephane]

Philippe Gerum wrote:
[...]
>This rounding was missing too. We need the previous one for kernel
local
> heaps, and the one below to meet the stricter PAGE_SIZE constraint for
> shareable heaps.
>
>--- ksrc/nucleus/heap.c	(revision 3095)
>+++ ksrc/nucleus/heap.c	(working copy)
>@@ -1103,7 +1103,7 @@
> 	spl_t s;
> 	int err;
>
>-	heapsize = PAGE_ALIGN(heapsize);
>+	heapsize = xnheap_rounded_size(heapsize, PAGE_SIZE);
> 	heapbase = __alloc_and_reserve_heap(heapsize, memflags);
>
> 	if (!heapbase)

Nope, still doesn't in -rc5  :-(
Most probably because it should be at least _2_ times the page size.


The following patch missed the -rc5, can it please make it for -rc6?

--- ksrc/skins/rtai/task.c	29 Oct 2007 08:45:27 -0000	1.3
+++ ksrc/skins/rtai/task.c	30 Oct 2007 15:04:08 -0000
@@ -139,6 +139,9 @@
	task->body = body;
	task->sigfn = sigfn;

+	if (xnarch_cpus_empty(task->affinity))
+		task->affinity = XNPOD_ALL_CPUS;
+
	xnlock_get_irqsave(&nklock, s);

	err = xnpod_start_thread(&task->thread_base, XNSUSP,    /*
Suspend on startup. */

-- 
Stephane

Re: [Xenomai-core] Xenomai v2.4-rc4: freeze with RTAI skin, fine with other skins

[Philippe Gerum]

Fillod Stephane wrote:
> Philippe Gerum wrote:
> [...]
>> This rounding was missing too. We need the previous one for kernel
> local
>> heaps, and the one below to meet the stricter PAGE_SIZE constraint for
>> shareable heaps.
>>
>> --- ksrc/nucleus/heap.c	(revision 3095)
>> +++ ksrc/nucleus/heap.c	(working copy)
>> @@ -1103,7 +1103,7 @@
>> 	spl_t s;
>> 	int err;
>>
>> -	heapsize = PAGE_ALIGN(heapsize);
>> +	heapsize = xnheap_rounded_size(heapsize, PAGE_SIZE);
>> 	heapbase = __alloc_and_reserve_heap(heapsize, memflags);
>>
>> 	if (!heapbase)
> 
> Nope, still doesn't in -rc5  :-(
> Most probably because it should be at least _2_ times the page size.
>

Damnit. Half of the logic is indeed missing. Ok, will re-re-re-fix, ...
the fix.

> 
> The following patch missed the -rc5, can it please make it for -rc6?
> 

Yes. Thanks.

> --- ksrc/skins/rtai/task.c	29 Oct 2007 08:45:27 -0000	1.3
> +++ ksrc/skins/rtai/task.c	30 Oct 2007 15:04:08 -0000
> @@ -139,6 +139,9 @@
> 	task->body = body;
> 	task->sigfn = sigfn;
> 
> +	if (xnarch_cpus_empty(task->affinity))
> +		task->affinity = XNPOD_ALL_CPUS;
> +
> 	xnlock_get_irqsave(&nklock, s);
> 
> 	err = xnpod_start_thread(&task->thread_base, XNSUSP,    /*
> Suspend on startup. */
> 


-- 
Philippe.

Re: [Xenomai-core] Xenomai v2.4-rc4: freeze with RTAI skin, fine with other skins

[Fillod Stephane]

Hi Gilles,

Thanks for the quick reply.

Gilles Chanteperdrix wrote:
> A case of freeze is a system call called in a loop which fails without
its return value being checked.

I forget to say that the RTAI application is running in kernel land,
because
no port of the RTAI skin has been made yet to user land (in fact, only
shm access).
So, can it still be a system call in a loop issue? Besides, the xeno 
watchdog is not kicking.
Once freezed, the box does not respond to ping, so this is not a
SCHED_FIFO task stuck either.
Still searching..
-- 
Stephane

Re: [Xenomai-core] Xenomai v2.4-rc4: freeze with RTAI skin, fine with other skins

[Philippe Gerum]

Fillod Stephane wrote:
> Hi Gilles,
> 
> Thanks for the quick reply.
> 
> Gilles Chanteperdrix wrote:
>> A case of freeze is a system call called in a loop which fails without
> its return value being checked.
> 
> I forget to say that the RTAI application is running in kernel land,
> because
> no port of the RTAI skin has been made yet to user land (in fact, only
> shm access).
> So, can it still be a system call in a loop issue? Besides, the xeno 
> watchdog is not kicking.
> Once freezed, the box does not respond to ping, so this is not a
> SCHED_FIFO task stuck either.
> Still searching..

Stack space overflow issue now biting some real-time kernel thread?
Spurious delay/timeout values computed by the application (specifically:
too short), as a side-effect of the timebase introduction?

-- 
Philippe.

Re: [Xenomai-core] Xenomai v2.4-rc4: freeze with RTAI skin, fine with other skins

[Gilles Chanteperdrix]

On 10/22/07, Fillod Stephane <stephane.fillod@domain.hid> wrote:
> Hi Gilles,
>
> Thanks for the quick reply.
>
> Gilles Chanteperdrix wrote:
> > A case of freeze is a system call called in a loop which fails without
> its return value being checked.
>
> I forget to say that the RTAI application is running in kernel land,
> because
> no port of the RTAI skin has been made yet to user land (in fact, only
> shm access).
> So, can it still be a system call in a loop issue? Besides, the xeno
> watchdog is not kicking.
> Once freezed, the box does not respond to ping, so this is not a
> SCHED_FIFO task stuck either.
> Still searching..

I meant to say a blocking call. But you are right, in this case the
watchdog would trigger.

-- 
                                               Gilles Chanteperdrix

[Xenomai-core] Xenomai v2.4-rc4: freeze with RTAI skin, fine with other skins

[Fillod Stephane]

[-- Attachment #1: Type: text/plain, Size: 1143 bytes --]

Hi,

As Philippe has been suggesting it, I've been testing v2.4-rc4 on x86 
(not favourite board though :) with the RTAI skin (not my favourite skin
too).
Anyway, a legacy RTAI application which was fine with v2.3.2/2.6.20, is
now 
freezing the box randomly in the first 30 seconds. On the other hand, 
the programs from the testsuite are running stable. So we may rule out 
a problem in the I-Pipe/kernel.

For the legacy RTAI application to load, the attached patch was
necessary.
The patch against ksrc/skins/rtai/shm.c is somewhat defeating the
purpose
of a lower XNCORE_PAGE_SIZE, so a better fix might be expected.

The box is running FC5 on a pentium D, but kernel is compiled for UP.
$ cat /proc/ipipe/version
1.10-09
$ cat /proc/xenomai/version
2.4-rc4
$ cat /proc/version
Linux version 2.6.23.1 () (gcc version 4.1.1 20070105 (Red Hat
4.1.1-51))

The kernel config file is attached.

I'm currently stripping down the RTAI code out of the application
in order to have a simple testbed reproducing the freeze.

In the mean time, does anyone have a clue where to look particularly?

TIA
-- 
Stephane

[-- Attachment #2: rtai-xeno-2.4rc4.patch --]
[-- Type: application/octet-stream, Size: 1129 bytes --]

diff -ru xenomai-2.4-rc4.orig/ksrc/skins/rtai/shm.c xenomai-2.4-rc4/ksrc/skins/rtai/shm.c
--- xenomai-2.4-rc4.orig/ksrc/skins/rtai/shm.c	2007-09-16 19:20:37.000000000 +0200
+++ xenomai-2.4-rc4/ksrc/skins/rtai/shm.c	2007-10-22 10:47:28.459171000 +0200
@@ -144,8 +144,8 @@
 	/* Make sure we won't hit trivial argument errors when calling
 	   xnheap_init(). */
 
-	if (heapsize < 2 * XNCORE_PAGE_SIZE)
-		heapsize = 2 * XNCORE_PAGE_SIZE;
+	if (heapsize < 2 * PAGE_SIZE)
+		heapsize = 2 * PAGE_SIZE;
 
 	/* Account for the overhead so that the actual free space is
 	   large enough to match the requested size. */
diff -ru xenomai-2.4-rc4.orig/ksrc/skins/rtai/task.c xenomai-2.4-rc4/ksrc/skins/rtai/task.c
--- xenomai-2.4-rc4.orig/ksrc/skins/rtai/task.c	2007-09-16 19:20:37.000000000 +0200
+++ xenomai-2.4-rc4/ksrc/skins/rtai/task.c	2007-10-22 10:49:57.100243000 +0200
@@ -139,6 +139,9 @@
 	task->body = body;
 	task->sigfn = sigfn;
 
+	if (xnarch_cpus_empty(task->affinity))
+		task->affinity = XNPOD_ALL_CPUS;
+
 	xnlock_get_irqsave(&nklock, s);
 
 	err = xnpod_start_thread(&task->thread_base, XNSUSP,	/* Suspend on startup. */

[-- Attachment #3: config-2.4rc4-rtai-freeze.gz --]
[-- Type: application/x-gzip, Size: 19631 bytes --]

Re: [Xenomai-core] Xenomai v2.4-rc4: freeze with RTAI skin, fine with other skins

[Gilles Chanteperdrix]

On 10/22/07, Fillod Stephane <stephane.fillod@domain.hid> wrote:
> In the mean time, does anyone have a clue where to look particularly?

A case of freeze is a system call called in a loop which fails without
its return value being checked.

-- 
                                               Gilles Chanteperdrix

Re: [Xenomai-core] Xenomai v2.4-rc4: freeze with RTAI skin, fine with other skins

[Philippe Gerum]

Fillod Stephane wrote:
> Hi,
> 
> As Philippe has been suggesting it, I've been testing v2.4-rc4 on x86 
> (not favourite board though :) with the RTAI skin (not my favourite skin
> too).
> Anyway, a legacy RTAI application which was fine with v2.3.2/2.6.20, is
> now 
> freezing the box randomly in the first 30 seconds. On the other hand, 
> the programs from the testsuite are running stable. So we may rule out 
> a problem in the I-Pipe/kernel.
> 
> For the legacy RTAI application to load, the attached patch was
> necessary.
> The patch against ksrc/skins/rtai/shm.c is somewhat defeating the
> purpose
> of a lower XNCORE_PAGE_SIZE, so a better fix might be expected.
> 

This one should prevent -EINVAL from being returned. Hopefully.

--- ksrc/skins/rtai/shm.c	(revision 3095)
+++ ksrc/skins/rtai/shm.c	(working copy)
@@ -150,8 +150,7 @@
 	/* Account for the overhead so that the actual free space is
 	   large enough to match the requested size. */

-	heapsize += xnheap_overhead(heapsize, XNCORE_PAGE_SIZE);
-	heapsize = XNCORE_PAGE_ALIGN(heapsize);
+	heapsize = xnheap_rounded_size(heapsize, XNCORE_PAGE_SIZE);

 #ifdef CONFIG_XENO_OPT_PERVASIVE
 	err = xnheap_init_mapped(p->heap,

> The box is running FC5 on a pentium D, but kernel is compiled for UP.
> $ cat /proc/ipipe/version
> 1.10-09
> $ cat /proc/xenomai/version
> 2.4-rc4
> $ cat /proc/version
> Linux version 2.6.23.1 () (gcc version 4.1.1 20070105 (Red Hat
> 4.1.1-51))
> 
> The kernel config file is attached.
> 
> I'm currently stripping down the RTAI code out of the application
> in order to have a simple testbed reproducing the freeze.
> 
> In the mean time, does anyone have a clue where to look particularly?
> 

Hard to say at this point. Since the nucleus watchdog does not trigger,
you may want to try disabling X86_UP_IOAPIC while keeping X86_UP_APIC,
and arm the kernel NMI watchdog on the LAPIC (nmi_watchdog=2). You may
be lucky and have a backtrace after the freeze.

PS: maybe enabling all the nucleus debug options would catch something too.

> TIA
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Xenomai-core mailing list
> Xenomai-core@domain.hid
> https://mail.gna.org/listinfo/xenomai-core


-- 
Philippe.