[Xenomai-help] Re: Re: What happens if task entry function returns?

[Xenomai-help] Re: Re: What happens if task entry function returns?

[M. Koehrer]

Hi Jan, 

I found out, that is issue seems to be related to the NPTL (native posix thread library).
The pthread library in use was 
/lib/tls/i686/cmov/libpthread.so.0
which is part of the Debian (testing) package libc6-i686.
(see http://packages.debian.org/testing/libs/libc6-i686)

When renaming the lib/tls to lib/tls.disabled the pthread library in use is
/lib/libpthread.so.0
and then it is working perfectly!

It seems that the i686 optimized libc is not fully compatible with Xenomai.
I read about this library at Romain Lenglet's web page:
http://www.csg.is.titech.ac.jp/~lenglet/howtos/realtimelinuxhowto/index.html
I think this library uses the NPTL (Native Posix Thread Lib).
The i686 optimized library shows the value
NPTL 2.3.6  
at 
getconf GNU_LIBPTHREAD_VERSION
The "old" pthread lib shows the value:
linuxthreads-0.10

Now, I tried to reconfigure Xenomai with --enable-x86-sep to use
the NPTL but this did not help either.
When I rename now  /lib/tls to /lib/tls.disabled, the application complains
about the missing NPTL support in the library.
Well, I have now a solution that works:
I can use the standard pthread lib and not the i686 optimized NPTL.
However, as NPTL seems to be a performance benefit, I think it could be
worth to get this combination up and running.

Is there anybody out there that uses a Debian based system with the i686
optimized libc that works perfectly?

BTW: I am using xenomai 2.2.4 from end of October.

Thanks for any feedback on this

Mathias
> > I am just playing around with Xenomai to understand it.
> > And I have one question concerning the entry function that will be called
> from
> > rt_task_start (Native API).
> > What happens if this entry function reaches its end or a return within the
> function is
> > called?
> > 
> > I have made a simple example that lead to strange effects.
> > In my example (see complete C Code in attachement) I create to tasks
> > that do the same.
> > The entry functions look like:
> > 
> > void taska(void *cookie)
> > {
> >     RTIME delay;
> >     int i;
> >     printf("Hi, I am task A %s\n", (char*)cookie);
> >     delay = 100000;
> > 
> >     for (i=0; i<200; i++)
> >     {
> >         rt_task_sleep(delay);
> >     }
> > 
> >     printf("This is the end of A\n");
> >     // rt_task_delete(0);
> > }
> > When I do not place the rt_task_delete(0) at the end of the function
> > my second task will never reach the end with the second printf.
> > Whenever I use rt_task_delete(0) at the end of the function it works
> perfectly.
> 
> Quite inconsistent, sounds like a bug. Does the second rt_task_join just
> rush through and main terminates?
> 
> > 
> > My question is now: What happens if the task's entry function returns?
> 
> pthread_exit() is invoked implicitly, which implies rt_task_delete(NULL).
> 
> > 
> > Thanks for any feedback on that question.
> 
> Is this issue SMP-related? What happens if you force all your tasks to
> the same CPU? Or if you run on a !CONFIG_SMP box?
> 
> Jan
> 

-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

[Xenomai-help] Re: What happens if task entry function returns?

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2079 bytes --]

M. Koehrer wrote:
> Hi Jan, 
> 
> I found out, that is issue seems to be related to the NPTL (native posix thread library).
> The pthread library in use was 
> /lib/tls/i686/cmov/libpthread.so.0
> which is part of the Debian (testing) package libc6-i686.
> (see http://packages.debian.org/testing/libs/libc6-i686)
> 
> When renaming the lib/tls to lib/tls.disabled the pthread library in use is
> /lib/libpthread.so.0
> and then it is working perfectly!

...which does not yet explain WHY the other scenario fails.

> 
> It seems that the i686 optimized libc is not fully compatible with Xenomai.

That's not true in such a common form. We are using NTPL libs for i586
and i686 (SuSE builds) without /general/ issues. Still, specific
problems like this may sleep somewhere.

> I read about this library at Romain Lenglet's web page:
> http://www.csg.is.titech.ac.jp/~lenglet/howtos/realtimelinuxhowto/index.html
> I think this library uses the NPTL (Native Posix Thread Lib).
> The i686 optimized library shows the value
> NPTL 2.3.6  
> at 
> getconf GNU_LIBPTHREAD_VERSION
> The "old" pthread lib shows the value:
> linuxthreads-0.10
> 
> Now, I tried to reconfigure Xenomai with --enable-x86-sep to use
> the NPTL but this did not help either.

That only influences the way syscalls are passed down into the kernel.

> When I rename now  /lib/tls to /lib/tls.disabled, the application complains
> about the missing NPTL support in the library.
> Well, I have now a solution that works:
> I can use the standard pthread lib and not the i686 optimized NPTL.
> However, as NPTL seems to be a performance benefit, I think it could be
> worth to get this combination up and running.
> 
> Is there anybody out there that uses a Debian based system with the i686
> optimized libc that works perfectly?
> 
> BTW: I am using xenomai 2.2.4 from end of October.
> 

Again my question: is the issue SMP related?

Reproducing your problem over a qemu box (with or without SMP) failed so
far for me. But I'm not on your glibc.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

[Xenomai-help] Re: Re: What happens if task entry function returns?

[M. Koehrer]

Hi Jan, 

First, I give you some more information about the error.
The output is the following:
Hi, I am task A Hello
Hi, I am task B World
Main waits for A
This is the end of A
Main waits for B

Then the system hangs.

I have done two tests to verify if there is an SMP issue.
First, I have set all CPUs in use with the application to CPU 0.
There was no change. However, I am able to stop the application with CTRL-C
and everything is cleaned up fine.

Then, I have modified the BIOS settings of the PC to disable Hyperthreading.
Again, no change. The same effect as with the test before.

The strange thing here is, when I use the original test (main on CPU 0, rt tasks on CPU 1)
on a SMP machine, I am not able to terminate the application with CTRL-C.
I have to reboot the PC.

That means, there seems to be a relation to SMP...

Another thing I have detected:
When I modify the program to replace the final rt_task_delete by
a sleep(1) call everything runs perfectly again.

Mathias

> > I found out, that is issue seems to be related to the NPTL (native posix
> thread library).
> > The pthread library in use was 
> > /lib/tls/i686/cmov/libpthread.so.0
> > which is part of the Debian (testing) package libc6-i686.
> > (see http://packages.debian.org/testing/libs/libc6-i686)
> > 
> > When renaming the lib/tls to lib/tls.disabled the pthread library in use
> is
> > /lib/libpthread.so.0
> > and then it is working perfectly!
> 
> ...which does not yet explain WHY the other scenario fails.
> 
> > 
> > It seems that the i686 optimized libc is not fully compatible with
> Xenomai.
> 
> That's not true in such a common form. We are using NTPL libs for i586
> and i686 (SuSE builds) without /general/ issues. Still, specific
> problems like this may sleep somewhere.
> 
> > I read about this library at Romain Lenglet's web page:
> >
> http://www.csg.is.titech.ac.jp/~lenglet/howtos/realtimelinuxhowto/index.html
> 
> > I think this library uses the NPTL (Native Posix Thread Lib).
> > The i686 optimized library shows the value
> > NPTL 2.3.6  
> > at 
> > getconf GNU_LIBPTHREAD_VERSION
> > The "old" pthread lib shows the value:
> > linuxthreads-0.10
> > 
> > Now, I tried to reconfigure Xenomai with --enable-x86-sep to use
> > the NPTL but this did not help either.
> 
> That only influences the way syscalls are passed down into the kernel.
> 
> > When I rename now  /lib/tls to /lib/tls.disabled, the application
> complains
> > about the missing NPTL support in the library.
> > Well, I have now a solution that works:
> > I can use the standard pthread lib and not the i686 optimized NPTL.
> > However, as NPTL seems to be a performance benefit, I think it could be
> > worth to get this combination up and running.
> > 
> > Is there anybody out there that uses a Debian based system with the i686
> > optimized libc that works perfectly?
> > 
> > BTW: I am using xenomai 2.2.4 from end of October.
> > 
> 
> Again my question: is the issue SMP related?
> 
> Reproducing your problem over a qemu box (with or without SMP) failed so
> far for me. But I'm not on your glibc.
> 
> Jan


-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

Re: [Xenomai-help] Re: Re: What happens if task entry function returns?

[Jan Kiszka]

M. Koehrer wrote:
> Hi Jan, 
> 
> First, I give you some more information about the error.
> The output is the following:
> Hi, I am task A Hello
> Hi, I am task B World
> Main waits for A
> This is the end of A
> Main waits for B
> 
> Then the system hangs.

You mean completely? Or just that application? Please enable the Xenomai
watchdog to see if we have a run-away RT thread in that case.

> 
> I have done two tests to verify if there is an SMP issue.
> First, I have set all CPUs in use with the application to CPU 0.
> There was no change. However, I am able to stop the application with CTRL-C
> and everything is cleaned up fine.
> 
> Then, I have modified the BIOS settings of the PC to disable Hyperthreading.

[Which is recommended anyway to keep latencies bounded - HT is evil for RT!]

> Again, no change. The same effect as with the test before.
> 
> The strange thing here is, when I use the original test (main on CPU 0, rt tasks on CPU 1)
> on a SMP machine, I am not able to terminate the application with CTRL-C.
> I have to reboot the PC.
> 
> That means, there seems to be a relation to SMP...

Probably in so far that the symptom changes. But !SMP doesn't make it go
away.

> 
> Another thing I have detected:
> When I modify the program to replace the final rt_task_delete by
> a sleep(1) call everything runs perfectly again.

Ok, could you try on a non-hanging setup to grab what is happening via
strace?

Thanks,
Jan

Re: Re: [Xenomai-help] Re: Re: What happens if task entry function returns?

[M. Koehrer]

[-- Attachment #1.1: Type: text/plain, Size: 2374 bytes --]

Hi Jan!

I have modified my test to use a sleep(1) at the end of the task functions.
I have enclosed the strace logs to this mail.
The file strace.tls is the one that uses the /lib/tls/... libraries.
The file strace is the one that uses the /lib/... libraries.

I hope that helps...
BTW: I have updated to the latest SVN version of Xenomai (kernel is 2.6.17.7) - the same effect here.

Thanks for the feedback!

Regards

Mathias

> > First, I give you some more information about the error.
> > The output is the following:
> > Hi, I am task A Hello
> > Hi, I am task B World
> > Main waits for A
> > This is the end of A
> > Main waits for B
> > 
> > Then the system hangs.
> 
> You mean completely? Or just that application? Please enable the Xenomai
> watchdog to see if we have a run-away RT thread in that case.
Only the application hangs!

> 
> > 
> > I have done two tests to verify if there is an SMP issue.
> > First, I have set all CPUs in use with the application to CPU 0.
> > There was no change. However, I am able to stop the application with
> CTRL-C
> > and everything is cleaned up fine.
> > 
> > Then, I have modified the BIOS settings of the PC to disable
> Hyperthreading.
> 
> [Which is recommended anyway to keep latencies bounded - HT is evil for
> RT!]
> 
> > Again, no change. The same effect as with the test before.
> > 
> > The strange thing here is, when I use the original test (main on CPU 0, rt
> tasks on CPU 1)
> > on a SMP machine, I am not able to terminate the application with CTRL-C.
> > I have to reboot the PC.
> > 
> > That means, there seems to be a relation to SMP...
> 
> Probably in so far that the symptom changes. But !SMP doesn't make it go
> away.
> 
> > 
> > Another thing I have detected:
> > When I modify the program to replace the final rt_task_delete by
> > a sleep(1) call everything runs perfectly again.
> 
> Ok, could you try on a non-hanging setup to grab what is happening via
> strace?


-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

[-- Attachment #2: strace.tgz --]
[-- Type: application/octet-stream, Size: 2102 bytes --]

Re: [Xenomai-help] Re: Re: What happens if task entry function returns?

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 428 bytes --]

M. Koehrer wrote:
> Hi Jan!
> 
> I have modified my test to use a sleep(1) at the end of the task functions.
> I have enclosed the strace logs to this mail.
> The file strace.tls is the one that uses the /lib/tls/... libraries.
> The file strace is the one that uses the /lib/... libraries.

Ok. And now a trace of the all-cpu-0 scenario would be good, as it is
broken but should still dump some information.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-help] Re: Re: What happens if task entry function returns?

[Philippe Gerum]

On Fri, 2006-11-17 at 15:17 +0100, M. Koehrer wrote:
> Hi Jan, 
> 
> First, I give you some more information about the error.
> The output is the following:
> Hi, I am task A Hello
> Hi, I am task B World
> Main waits for A
> This is the end of A
> Main waits for B
> 
> Then the system hangs.
> 

Confirmed here on a two-way PIII. Well, _this_ was a nice one.
Basically, the softlock was due to a missing rescheduling call after
task A has exited, leaving task B in ready state, somewhere in the
Twilight Zone. The patch below fixes it on my setup:

--- ksrc/nucleus/shadow.c	(revision 1854)
+++ ksrc/nucleus/shadow.c	(working copy)
@@ -1631,7 +1631,8 @@
 
 static inline void do_taskexit_event(struct task_struct *p)
 {
-	xnthread_t *thread = xnshadow_thread(p);	/* p == current */
+	xnthread_t *thread = xnshadow_thread(p); /* p == current */
+	spl_t s;
 
 	if (!thread)
 		return;
@@ -1639,12 +1640,15 @@
 	if (xnpod_shadow_p())
 		xnshadow_relax(0);
 
-	/* So that we won't attempt to further wakeup the exiting task in
-	   xnshadow_unmap(). */
-
+	xnlock_get_irqsave(&nklock, s);
+	/* Prevent wakeup call from xnshadow_unmap(). */
 	xnshadow_thrptd(p) = NULL;
 	xnthread_archtcb(thread)->user_task = NULL;
-	xnpod_delete_thread(thread);	/* Should indirectly call xnshadow_unmap(). */
+	/* xnpod_delete_thread() -> hook -> xnshadow_unmap(). */
+	xnpod_delete_thread(thread);
+	xnsched_set_resched(thread->sched);
+	xnpod_schedule();
+	xnlock_put_irqrestore(&nklock, s);
 
 	xnltt_log_event(xeno_ev_shadowexit, thread->name);
 }

-- 
Philippe.

Re: Re: [Xenomai-help] Re: Re: What happens if task entry function

[M. Koehrer]

Hi Philppe,

thanks for the patch. Using this patch, my test application is working now!
I have also update to the latest SVN version (1882) of Xenomai.
Using this version, the application works perfectly, too.

Regards

Mathias

> > First, I give you some more information about the error.
> > The output is the following:
> > Hi, I am task A Hello
> > Hi, I am task B World
> > Main waits for A
> > This is the end of A
> > Main waits for B
> > 
> > Then the system hangs.
> > 
> 
> Confirmed here on a two-way PIII. Well, _this_ was a nice one.
> Basically, the softlock was due to a missing rescheduling call after
> task A has exited, leaving task B in ready state, somewhere in the
> Twilight Zone. The patch below fixes it on my setup:
> 
> --- ksrc/nucleus/shadow.c	(revision 1854)
> +++ ksrc/nucleus/shadow.c	(working copy)
> @@ -1631,7 +1631,8 @@
>  
>  static inline void do_taskexit_event(struct task_struct *p)
>  {
> -	xnthread_t *thread = xnshadow_thread(p);	/* p == current */
> +	xnthread_t *thread = xnshadow_thread(p); /* p == current */
> +	spl_t s;
>  
>  	if (!thread)
>  		return;
> @@ -1639,12 +1640,15 @@
>  	if (xnpod_shadow_p())
>  		xnshadow_relax(0);
>  
> -	/* So that we won't attempt to further wakeup the exiting task in
> -	   xnshadow_unmap(). */
> -
> +	xnlock_get_irqsave(&nklock, s);
> +	/* Prevent wakeup call from xnshadow_unmap(). */
>  	xnshadow_thrptd(p) = NULL;
>  	xnthread_archtcb(thread)->user_task = NULL;
> -	xnpod_delete_thread(thread);	/* Should indirectly call xnshadow_unmap().
> */
> +	/* xnpod_delete_thread() -> hook -> xnshadow_unmap(). */
> +	xnpod_delete_thread(thread);
> +	xnsched_set_resched(thread->sched);
> +	xnpod_schedule();
> +	xnlock_put_irqrestore(&nklock, s);
>  
>  	xnltt_log_event(xeno_ev_shadowexit, thread->name);
>  }
> 
> -- 
> Philippe.
> 
> 
> 

-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2