[Xenomai-core] yet another test tool

[Xenomai-core] yet another test tool

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 767 bytes --]

Hi,

as I already mentioned, I experimented with the cyclictest-0.5 by Thomas
Gleixner (http://www.tglx.de/projects/misc/cyclictest), one of the
PREEMPT_RT developers. The attached patch fixes the scheduling policy
setup and locks the whole test into memory.

This tool is quite handy for running more than one timed thread, and for
basic testing of the POSIX skin. As Xenomai does not support hard-RT
signal delivery yet, the only relevant mode is -n, i.e. delaying via
clock_nanosleep. From the bugs I fixed I would say that not every
feature may work yet, but running with -n, -p 99 (highest priority
used), and -t 10 (create e.g. 10 cascading threads) looks fine to me.

What do you think, is it worth including as a POSIX counterpart for
testsuite/latency?

Jan

[-- Attachment #2: cyclic.patch --]
[-- Type: text/plain, Size: 1735 bytes --]

--- cyclictest.c.orig	2005-11-24 13:33:21.000000000 +0100
+++ cyclictest.c	2006-03-17 10:50:26.000000000 +0100
@@ -24,12 +24,14 @@
 #include <string.h>
 #include <time.h>
 #include <unistd.h>
+#include <limits.h>
 
 #include <linux/unistd.h>
 
 #include <sys/stat.h>
 #include <sys/types.h>
 #include <sys/time.h>
+#include <sys/mman.h>
 
 /* Ugly, but .... */
 #define gettid() syscall(__NR_gettid)
@@ -158,7 +160,7 @@ void *timerthread(void *param)
 
 	memset(&schedp, 0, sizeof(schedp));
 	schedp.sched_priority = par->prio;
-	sched_setscheduler(0, policy, &schedp);
+	pthread_setschedparam(pthread_self(), policy, &schedp);
 
 	/* Get current time */
 	clock_gettime(par->clock, &now);
@@ -265,7 +267,7 @@ out:		
 
 	/* switch to normal */
 	schedp.sched_priority = 0;
-	sched_setscheduler(0, SCHED_OTHER, &schedp);	
+	pthread_setschedparam(pthread_self(), SCHED_OTHER, &schedp);
 
 	stat->threadstarted = -1;
 
@@ -396,6 +398,7 @@ int main(int argc, char **argv)
 	int mode;
 	struct thread_param *par;
 	struct thread_stat *stat;
+	pthread_attr_t thattr;
 	int i, ret = -1;
 
 	if (geteuid()) {
@@ -403,6 +406,8 @@ int main(int argc, char **argv)
 		exit(-1);
 	}
 
+	mlockall(MCL_CURRENT | MCL_FUTURE);
+
 	process_options(argc, argv);
 
 	mode = use_nanosleep + use_system;
@@ -442,7 +447,9 @@ int main(int argc, char **argv)
 		par[i].stats = &stat[i];
 		stat[i].min = 1000000;
 		stat[i].max = -1000000;
-		pthread_create(&stat[i].thread, NULL, timerthread, &par[i]);
+		pthread_attr_init(&thattr);
+		pthread_attr_setstacksize(&thattr, PTHREAD_STACK_MIN);
+		pthread_create(&stat[i].thread, &thattr, timerthread, &par[i]);
 		stat[i].threadstarted = 1;
 	}
 	

Re: [Xenomai-core] yet another test tool

[Philippe Gerum]

Jan Kiszka wrote:
> Hi,
> 
> as I already mentioned, I experimented with the cyclictest-0.5 by Thomas
> Gleixner (http://www.tglx.de/projects/misc/cyclictest), one of the
> PREEMPT_RT developers. The attached patch fixes the scheduling policy
> setup and locks the whole test into memory.
> 
> This tool is quite handy for running more than one timed thread, and for
> basic testing of the POSIX skin. As Xenomai does not support hard-RT
> signal delivery yet, the only relevant mode is -n, i.e. delaying via
> clock_nanosleep. From the bugs I fixed I would say that not every
> feature may work yet, but running with -n, -p 99 (highest priority
> used), and -t 10 (create e.g. 10 cascading threads) looks fine to me.
> 
> What do you think, is it worth including as a POSIX counterpart for
> testsuite/latency?

Yes indeed, e.g. testsuite/cyclic.

> 
> Jan
> 
> 
> ------------------------------------------------------------------------
> 
> --- cyclictest.c.orig	2005-11-24 13:33:21.000000000 +0100
> +++ cyclictest.c	2006-03-17 10:50:26.000000000 +0100
> @@ -24,12 +24,14 @@
>  #include <string.h>
>  #include <time.h>
>  #include <unistd.h>
> +#include <limits.h>
>  
>  #include <linux/unistd.h>
>  
>  #include <sys/stat.h>
>  #include <sys/types.h>
>  #include <sys/time.h>
> +#include <sys/mman.h>
>  
>  /* Ugly, but .... */
>  #define gettid() syscall(__NR_gettid)
> @@ -158,7 +160,7 @@ void *timerthread(void *param)
>  
>  	memset(&schedp, 0, sizeof(schedp));
>  	schedp.sched_priority = par->prio;
> -	sched_setscheduler(0, policy, &schedp);
> +	pthread_setschedparam(pthread_self(), policy, &schedp);
>  
>  	/* Get current time */
>  	clock_gettime(par->clock, &now);
> @@ -265,7 +267,7 @@ out:		
>  
>  	/* switch to normal */
>  	schedp.sched_priority = 0;
> -	sched_setscheduler(0, SCHED_OTHER, &schedp);	
> +	pthread_setschedparam(pthread_self(), SCHED_OTHER, &schedp);
>  
>  	stat->threadstarted = -1;
>  
> @@ -396,6 +398,7 @@ int main(int argc, char **argv)
>  	int mode;
>  	struct thread_param *par;
>  	struct thread_stat *stat;
> +	pthread_attr_t thattr;
>  	int i, ret = -1;
>  
>  	if (geteuid()) {
> @@ -403,6 +406,8 @@ int main(int argc, char **argv)
>  		exit(-1);
>  	}
>  
> +	mlockall(MCL_CURRENT | MCL_FUTURE);
> +
>  	process_options(argc, argv);
>  
>  	mode = use_nanosleep + use_system;
> @@ -442,7 +447,9 @@ int main(int argc, char **argv)
>  		par[i].stats = &stat[i];
>  		stat[i].min = 1000000;
>  		stat[i].max = -1000000;
> -		pthread_create(&stat[i].thread, NULL, timerthread, &par[i]);
> +		pthread_attr_init(&thattr);
> +		pthread_attr_setstacksize(&thattr, PTHREAD_STACK_MIN);
> +		pthread_create(&stat[i].thread, &thattr, timerthread, &par[i]);
>  		stat[i].threadstarted = 1;
>  	}
>  	
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Xenomai-core mailing list
> Xenomai-core@domain.hid
> https://mail.gna.org/listinfo/xenomai-core


-- 

Philippe.

Re: [Xenomai-core] yet another test tool

[Gilles Chanteperdrix]

Jan Kiszka wrote:
 > (...)As Xenomai does not support hard-RT signal delivery yet (...)

This is the next feature missing to the POSIX skin. I would like to
implement this, but I am not sure which way to go :
- either, if it is possible, getting Linux signals services to run in
  every domain at Adeos level, by replacing spinlocks with spinlocks_hw
  and such kind of tricks;
- or adding a generic service at the adeos layer (a hook called when
  returning to user-space), building a generic user-space signals
  service at the nucleus level, and finally building all posix signals
  services on top of this.

The first approach guarantees the best integration with Linux, but
potentially add sections in Linux that are not preemptible by any
Xenomai skin. With the second approach, all services related to signals
have to be reimplemented plus some shortcuts to have standard user-space
tools such as "kill" working.


 > 
 > What do you think, is it worth including as a POSIX counterpart for
 > testsuite/latency?

There are a few details that I do not like about this tool, but we may
take it, and fix the details later.


-- 


					    Gilles Chanteperdrix.

Re: [Xenomai-core] yet another test tool

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1958 bytes --]

Gilles Chanteperdrix wrote:
> Jan Kiszka wrote:
>  > (...)As Xenomai does not support hard-RT signal delivery yet (...)
> 
> This is the next feature missing to the POSIX skin. I would like to
> implement this, but I am not sure which way to go :
> - either, if it is possible, getting Linux signals services to run in
>   every domain at Adeos level, by replacing spinlocks with spinlocks_hw
>   and such kind of tricks;
> - or adding a generic service at the adeos layer (a hook called when
>   returning to user-space), building a generic user-space signals
>   service at the nucleus level, and finally building all posix signals
>   services on top of this.
> 
> The first approach guarantees the best integration with Linux, but
> potentially add sections in Linux that are not preemptible by any
> Xenomai skin. With the second approach, all services related to signals
> have to be reimplemented plus some shortcuts to have standard user-space
> tools such as "kill" working.
> 

The preemptibility of the Linux signal code path heavily depends on the
tasklist_lock, and that was still an issue even for PREEMPT_RT with all
their modifications the last time I checked their code more thoroughly
(~2 months ago). Meanwhile, things may have improved for that tree, but
I doubt that this is already in mainline, not to speak of older 2.6 or
even 2.4 kernels.

As a first step, I would vote for establishing that generic service to
redirect the userspace return path to some arbitrary handler in hard-RT
context. Then we can think about how to handle signal injection from
Linux vs. injection from Xenomai gracefully.

> 
>  > 
>  > What do you think, is it worth including as a POSIX counterpart for
>  > testsuite/latency?
> 
> There are a few details that I do not like about this tool, but we may
> take it, and fix the details later.

It's a real hack, isn't it ;)? But what precisely do you mean?

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [Xenomai-core] yet another test tool

[Gilles Chanteperdrix]

Jan Kiszka wrote:
 > As a first step, I would vote for establishing that generic service to
 > redirect the userspace return path to some arbitrary handler in hard-RT
 > context. Then we can think about how to handle signal injection from
 > Linux vs. injection from Xenomai gracefully.

Ok, the first idea was a bad idea. Will try the second, or the
alternative proposed by Philippe.

 > 
 > > 
 > >  > 
 > >  > What do you think, is it worth including as a POSIX counterpart for
 > >  > testsuite/latency?
 > > 
 > > There are a few details that I do not like about this tool, but we may
 > > take it, and fix the details later.
 > 
 > It's a real hack, isn't it ;)? But what precisely do you mean?

What I dislike most, is the lack of return values check. This may work
well with Linux, but not with Xenomai. I would also prefer a clean
shutdown using pthread_cancel, after all sigwait and *nanosleep are
cancellation points, so it should work even without resorting to cleanup
handlers.

-- 


					    Gilles Chanteperdrix.

Re: [Xenomai-core] yet another test tool

[Philippe Gerum]

Gilles Chanteperdrix wrote:
> Jan Kiszka wrote:
>  > As a first step, I would vote for establishing that generic service to
>  > redirect the userspace return path to some arbitrary handler in hard-RT
>  > context. Then we can think about how to handle signal injection from
>  > Linux vs. injection from Xenomai gracefully.
> 
> Ok, the first idea was a bad idea. Will try the second, or the
> alternative proposed by Philippe.
> 
>  > 
>  > > 
>  > >  > 
>  > >  > What do you think, is it worth including as a POSIX counterpart for
>  > >  > testsuite/latency?
>  > > 
>  > > There are a few details that I do not like about this tool, but we may
>  > > take it, and fix the details later.
>  > 
>  > It's a real hack, isn't it ;)? But what precisely do you mean?
> 
> What I dislike most, is the lack of return values check. This may work
> well with Linux, but not with Xenomai. I would also prefer a clean
> shutdown using pthread_cancel, after all sigwait and *nanosleep are
> cancellation points, so it should work even without resorting to cleanup
> handlers.
> 

The current implementation is one thing (we could fix it), the purpose 
of the tool is another, and actually, this is the latter which seems 
useful to me. By sharing some common tests between native preemption and 
real-time sub-systems like Xeno, we would make performance comparisons 
more relevant. Additionally, I'm convinced that the POSIX skin is an 
underutilized goodie albeit it works damn well and obviously favours a 
close integration within the Linux environment, simply because there is 
a lack of explanation about it. Illustrating how one could leverage it 
is always a good thing.

-- 

Philippe.

Re: [Xenomai-core] yet another test tool

[Gilles Chanteperdrix]

Philippe Gerum wrote:
 > The current implementation is one thing (we could fix it), the purpose 
 > of the tool is another, and actually, this is the latter which seems 
 > useful to me. By sharing some common tests between native preemption and 
 > real-time sub-systems like Xeno, we would make performance comparisons 
 > more relevant. 

Ok. I added the tool. I also had to add some cruft to configure.in to
get the --wrap flags in src/testsuite/cyclic/Makefile.

Now, running this test seem to have uncovered a bug with the binary heap
based timers that the latency test did not show. I am investigating on
this, but in the mean time, please do not enable the binary heap timers
option.

 > Additionally, I'm convinced that the POSIX skin is an 
 > underutilized goodie albeit it works damn well and obviously favours a 
 > close integration within the Linux environment, simply because there is 
 > a lack of explanation about it. Illustrating how one could leverage it 
 > is always a good thing.

Well, thanks. I believe that if the POSIX skin works well, it is simply
because its testsuite has a good coverage. As for the documentation, you
may have noticed that I am in the process of documenting the POSIX API,
you can follow the progression here :

http://download.gna.org/xenomai/documentation/trunk/html/api/group__posix.html

Please, everybody, feel free to send comments, critics, patches.

I guess you would like something like a POSIX API tour, but in order
to write that, we would need a concrete example, such as, for example,
playing .wav files with a PC speaker.

-- 


					    Gilles Chanteperdrix.

Re: [Xenomai-core] yet another test tool

[Gilles Chanteperdrix]

Gilles Chanteperdrix wrote:
 > Philippe Gerum wrote:
 >  > The current implementation is one thing (we could fix it), the purpose 
 >  > of the tool is another, and actually, this is the latter which seems 
 >  > useful to me. By sharing some common tests between native preemption and 
 >  > real-time sub-systems like Xeno, we would make performance comparisons 
 >  > more relevant. 
 > 
 > Ok. I added the tool. I also had to add some cruft to configure.in to
 > get the --wrap flags in src/testsuite/cyclic/Makefile.
 > 
 > Now, running this test seem to have uncovered a bug with the binary heap
 > based timers that the latency test did not show. I am investigating on
 > this, but in the mean time, please do not enable the binary heap timers
 > option.

The bug was not due to the binary heap timers list, but to the fact that
the POSIX skin module (and other skins modules) did call
xnpod_stop_timer.

I removed all such calls of xnpod_stop_timer, since xnpod_shutdown calls
xnpod_stop_timer when needed (that is, only when shutting down the last
skin pod).

-- 


					    Gilles Chanteperdrix.

Re: [Xenomai-core] yet another test tool

[Philippe Gerum]

Gilles Chanteperdrix wrote:
> Jan Kiszka wrote:
>  > (...)As Xenomai does not support hard-RT signal delivery yet (...)
> 
> This is the next feature missing to the POSIX skin. I would like to
> implement this, but I am not sure which way to go :
> - either, if it is possible, getting Linux signals services to run in
>   every domain at Adeos level, by replacing spinlocks with spinlocks_hw
>   and such kind of tricks;

Would be a nightmare, I think. Way too many paths are involved in the 
vanilla kernel, and this would be overkill wrt what we want to do. 
Actually, what we need is basically exposing the nucleus signal 
interface to user-space, and map Linux RT signals over nucleus signals. 
Other (non-RT) Linux signals would keep on being handled in secondary 
mode the way they are now.

> - or adding a generic service at the adeos layer (a hook called when
>   returning to user-space), building a generic user-space signals
>   service at the nucleus level, and finally building all posix signals
>   services on top of this.

A (maybe easier) third option would be to generalize some kind of 
pseudo-asynchronous call support, with a worker thread operating on a 
dedicated priority level inside applications registering for 
asynchronous notifications. The kernel-side would handle the server 
wakeups, providing a unified interface for pending on hooks, signals, 
watchdogs etc. It would also need to properly multiplex those events 
notified from within the skins, and wake up the right pending server in 
user-space, which would in turn fire the user provided handler, all in 
primary mode. In any case, this would not be more costly latency-wise 
than implementing mere callouts, since most of the switching cost comes 
from the MMU switch, which we would have to do in both cases, anyway.

> 
> The first approach guarantees the best integration with Linux, but
> potentially add sections in Linux that are not preemptible by any
> Xenomai skin. With the second approach, all services related to signals
> have to be reimplemented plus some shortcuts to have standard user-space
> tools such as "kill" working.
> 
> 
>  > 
>  > What do you think, is it worth including as a POSIX counterpart for
>  > testsuite/latency?
> 
> There are a few details that I do not like about this tool, but we may
> take it, and fix the details later.
> 
> 


-- 

Philippe.

Re: [Xenomai-core] yet another test tool

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2246 bytes --]

Philippe Gerum wrote:
> Gilles Chanteperdrix wrote:
>> Jan Kiszka wrote:
>>  > (...)As Xenomai does not support hard-RT signal delivery yet (...)
>>
>> This is the next feature missing to the POSIX skin. I would like to
>> implement this, but I am not sure which way to go :
>> - either, if it is possible, getting Linux signals services to run in
>>   every domain at Adeos level, by replacing spinlocks with spinlocks_hw
>>   and such kind of tricks;
> 
> Would be a nightmare, I think. Way too many paths are involved in the
> vanilla kernel, and this would be overkill wrt what we want to do.
> Actually, what we need is basically exposing the nucleus signal
> interface to user-space, and map Linux RT signals over nucleus signals.
> Other (non-RT) Linux signals would keep on being handled in secondary
> mode the way they are now.
> 
>> - or adding a generic service at the adeos layer (a hook called when
>>   returning to user-space), building a generic user-space signals
>>   service at the nucleus level, and finally building all posix signals
>>   services on top of this.
> 
> A (maybe easier) third option would be to generalize some kind of
> pseudo-asynchronous call support, with a worker thread operating on a
> dedicated priority level inside applications registering for
> asynchronous notifications. The kernel-side would handle the server
> wakeups, providing a unified interface for pending on hooks, signals,
> watchdogs etc. It would also need to properly multiplex those events
> notified from within the skins, and wake up the right pending server in
> user-space, which would in turn fire the user provided handler, all in
> primary mode. In any case, this would not be more costly latency-wise
> than implementing mere callouts, since most of the switching cost comes
> from the MMU switch, which we would have to do in both cases, anyway.

We would need a "shadow" priority level for each real one so that those
handlers do not cause any priority inversions (the main RT issue of
servers). Moreover, it would require a bulk of extra threads, actually
one per used prio level, to handle all those calls with the correct
priority. My feeling: too costly, memory-wise.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [Xenomai-core] yet another test tool

[Philippe Gerum]

Jan Kiszka wrote:
> Philippe Gerum wrote:
> 
>>Gilles Chanteperdrix wrote:
>>
>>>Jan Kiszka wrote:
>>> > (...)As Xenomai does not support hard-RT signal delivery yet (...)
>>>
>>>This is the next feature missing to the POSIX skin. I would like to
>>>implement this, but I am not sure which way to go :
>>>- either, if it is possible, getting Linux signals services to run in
>>>  every domain at Adeos level, by replacing spinlocks with spinlocks_hw
>>>  and such kind of tricks;
>>
>>Would be a nightmare, I think. Way too many paths are involved in the
>>vanilla kernel, and this would be overkill wrt what we want to do.
>>Actually, what we need is basically exposing the nucleus signal
>>interface to user-space, and map Linux RT signals over nucleus signals.
>>Other (non-RT) Linux signals would keep on being handled in secondary
>>mode the way they are now.
>>
>>
>>>- or adding a generic service at the adeos layer (a hook called when
>>>  returning to user-space), building a generic user-space signals
>>>  service at the nucleus level, and finally building all posix signals
>>>  services on top of this.
>>
>>A (maybe easier) third option would be to generalize some kind of
>>pseudo-asynchronous call support, with a worker thread operating on a
>>dedicated priority level inside applications registering for
>>asynchronous notifications. The kernel-side would handle the server
>>wakeups, providing a unified interface for pending on hooks, signals,
>>watchdogs etc. It would also need to properly multiplex those events
>>notified from within the skins, and wake up the right pending server in
>>user-space, which would in turn fire the user provided handler, all in
>>primary mode. In any case, this would not be more costly latency-wise
>>than implementing mere callouts, since most of the switching cost comes
>>from the MMU switch, which we would have to do in both cases, anyway.
> 
> 
> We would need a "shadow" priority level for each real one so that those
> handlers do not cause any priority inversions (the main RT issue of
> servers).

Not if the worker threads inherit the priority of the target threads 
dynamically.

  Moreover, it would require a bulk of extra threads, actually
> one per used prio level, to handle all those calls with the correct
> priority. My feeling: too costly, memory-wise.

You would just need one worker thread per-application and per-cpu. Async 
calls are always serialized, at least on a given CPU. The idea is to 
provide some event multiplexing capabilities to the kernel-side, instead 
of implementing a worker for each and every pended event.

The only thing that would differ from a real signal handling is the 
actual backtrace of the handler.

Another option without worker thread would be to implement the same kind 
of pending event detection the kernel does on the return path from 
Xenomai syscalls using some reserved CPU register, and ask the kernel 
side about the callout to fire and the argument to pass the handler 
whenever some pending event is detected.

> 
> Jan
> 


-- 

Philippe.