[Xenomai-help] expected output and runtime of switchtest ?

[Xenomai-help] expected output and runtime of switchtest ?

[Jim Cromie]

soekris:/usr/xenomai/testsuite/switchtest# modprobe xeno_switchtest
[  160.221018] Xenomai: starting RTDM services.
soekris:/usr/xenomai/testsuite/switchtest#
soekris:/usr/xenomai/testsuite/switchtest# switch -n
soekris:/usr/xenomai/testsuite/switchtest# switch rtk0
cpu 0: 498
cpu 0: 998
cpu 0: 1498
cpu 0: 2000
cpu 0: 2496
cpu 0: 2998
cpu 0: 3498
cpu 0: 3998
cpu 0: 4500

...

This prog has been running for atleast 1/2 hr,
with minimum args..

what should it be doing ?

Re: [Xenomai-help] expected output and runtime of switchtest ?

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 749 bytes --]

Jim Cromie wrote:
> 
> 
> soekris:/usr/xenomai/testsuite/switchtest# modprobe xeno_switchtest
> [  160.221018] Xenomai: starting RTDM services.
> soekris:/usr/xenomai/testsuite/switchtest#
> soekris:/usr/xenomai/testsuite/switchtest# switch -n
> soekris:/usr/xenomai/testsuite/switchtest# switch rtk0
> cpu 0: 498
> cpu 0: 998
> cpu 0: 1498
> cpu 0: 2000
> cpu 0: 2496
> cpu 0: 2998
> cpu 0: 3498
> cpu 0: 3998
> cpu 0: 4500
> 
> ...
> 
> This prog has been running for atleast 1/2 hr,
> with minimum args..
> 
> what should it be doing ?

This is a regression test. You should see an error message or a system
crash if something goes wrong. The output above looks ok (number of
passed loops, kind of lifesign).

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-core] expected output and runtime of switchtest ?

[Jim Cromie]

[-- Attachment #1: Type: text/plain, Size: 2678 bytes --]

Jan Kiszka wrote:
> Jim Cromie wrote:
>   
>> soekris:/usr/xenomai/testsuite/switchtest# modprobe xeno_switchtest
>> [  160.221018] Xenomai: starting RTDM services.
>> soekris:/usr/xenomai/testsuite/switchtest#
>> soekris:/usr/xenomai/testsuite/switchtest# switch -n
>> soekris:/usr/xenomai/testsuite/switchtest# switch rtk0
>> cpu 0: 498
>> cpu 0: 998
>> cpu 0: 1498
>> cpu 0: 2000
>> cpu 0: 2496
>> cpu 0: 2998
>> cpu 0: 3498
>> cpu 0: 3998
>> cpu 0: 4500
>>
>> ...
>>
>> This prog has been running for atleast 1/2 hr,
>> with minimum args..
>>
>> what should it be doing ?
>>     
>
> This is a regression test. You should see an error message or a system
> crash if something goes wrong. The output above looks ok (number of
> passed loops, kind of lifesign).
>
>   
OK thanks.

FWIW, I noted that xeno-test is not running these:
- switchbench
- switchtest
- irqbench

Im not sure they belong in xeno-test though, since they dont
appear to produce output that shows good vs bad performance,
only an informal 'sanity' check.

And technically, dont regression tests have to yield
 a PASS / FAIL decision ?  ;-)


Speaking more broadly, there are 3 possible kinds of test-progs

- regression tests
    PASS / FAIL - either by exit(rc),
       or by printf( "%s\n", rc ? "not-ok" : "ok")
       see perl's regression test suite ( 100k separate tests )
       usually test details, are not tutorial

- performance tests
    progs stress xenomai, print performance data
    should help see performance problems, expose bugs
    xeno-test aims to collect performance data
    performance data must be expressive
       (switchtest is perhaps insufficient here)

- examples / tutorials
    ex: satch.c 
    simple, clear progs (low feature clutter, etc)
    Id like to see all demo/**/ progs in single dir
       forex satch-native, satch-vxworks, etc ..
       makes for easier browsing
    simple makefile
       builds out-of-tree
       handles kernel-modules and user-progs
       (Ive seen some clean ones, cant find now.  Mine are crufty:-(
    'patterns' of usage
       IWBGreat if we had common usage patterns isolated,
       named, and described


Towards this last item, Ive done 2 things:

- poached code from Hannes Mayer :-)
    http://www.captain.at/xenomai.php
    task-timers.c does periodic timer 3 ways:
       sleeper, waiter, alarm.

- scrounged old rtai/fusion code (ls -l says Jul 05 ;-)
    cleaned up, 1/2 compile now
    maybe theres examples-tutorials-patterns fodder in here.


attached tarball has these in 2 top-level dirs.
Id like to see if theres a place for them long-term, and clean
them up so theyre correct and helpful.


> Jan
>

thanks
-jimc

[-- Attachment #2: xeno-examples-tuts.tgz --]
[-- Type: application/gzip, Size: 16676 bytes --]

[Xenomai-core] Test, benchmark, demo frameworks (was: expected output and runtime of switchtest ?)

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 5103 bytes --]

Hi all,

Jim raised these issues nicely to a generic level. I would like to pick
it up and add some thoughts.

Jim Cromie wrote:
> ...
> FWIW, I noted that xeno-test is not running these:
> - switchbench
> - switchtest
> - irqbench
> 
> Im not sure they belong in xeno-test though, since they dont
> appear to produce output that shows good vs bad performance,
> only an informal 'sanity' check.

Including switchtest depends on if xeno-test should also do some
elementary stability tests. This can be derived from performance tests
as well, but Gilles' switchtest does it for the various switching
constellations more systematically.

Including irqbench is more tricky as "real" hardware and a second box
are always involved here (so far it only works over null-modem, need to
be extended to some GPIOs or parallel port).

Regarding the output of the various benchmarks I would like to cite
myself here:

https://mail.gna.org/public/xenomai-core/2006-06/msg00195.html

[And as one of the major xeno-test contributors, you may feel included
by the term "test team". ;)]

> 
> And technically, dont regression tests have to yield
> a PASS / FAIL decision ?  ;-)
> 

Simple regular output is a good idea whenever the result is simple to
express. A fatally crashing switch test due to broken support on arch
XYZ will make it hard to issue "FAIL"... :)

> 
> Speaking more broadly, there are 3 possible kinds of test-progs
> 
> - regression tests
>    PASS / FAIL - either by exit(rc),
>       or by printf( "%s\n", rc ? "not-ok" : "ok")
>       see perl's regression test suite ( 100k separate tests )
>       usually test details, are not tutorial

Have you checked what is already under sim/skins/*/testsuite? I must
confess I don't know if it is easily compilable for non-simulated
execution as well. The best thing would be a test framework that builds
both for the simulator and for "real" usage on the target.

> 
> - performance tests
>    progs stress xenomai, print performance data
>    should help see performance problems, expose bugs
>    xeno-test aims to collect performance data
>    performance data must be expressive
>       (switchtest is perhaps insufficient here)

See my note above. I think some approach with a generic data collection
suite + various data generators would be really fantastic! Just takes
some brain(s) to design it and some hands to hack it...

> 
> - examples / tutorials
>    ex: satch.c    simple, clear progs (low feature clutter, etc)
>    Id like to see all demo/**/ progs in single dir
>       forex satch-native, satch-vxworks, etc ..
>       makes for easier browsing
>    simple makefile
>       builds out-of-tree
>       handles kernel-modules and user-progs
>       (Ive seen some clean ones, cant find now.  Mine are crufty:-(
>    'patterns' of usage
>       IWBGreat if we had common usage patterns isolated,
>       named, and described

Basically full ack! Wolfgang and I had this discussion recently in a
private thread on how to deal with demos for the new CAN stack. Xenomai
really needs more of this, and it needs it in a structured, regularly
organised way. Some of my thoughts on a demo repository:

 o Should be located in the same SVN repos, maybe under /demo, in order
   to keep it in sync with ongoing API enhancements/modifications.
   But the distribution should be separate from the release (just like
   the simulator).

 o Must be well commented, i.e. should not only show how things can be
   done, but also why (or why not). That matches very well with your
   patterns.

 o Should include both simple beginners examples as well as a few more
   complex demos.

 o Full ack to your Makefile requirement: stand-alone kernel (Hannes has
   a nice generic one for both 2.4 and 2.6) and user-space makefiles. No
   autotool stuff here. At the same time, we need a script to build and
   maybe even run them automatically to detect breakages.

This will certainly be a longer process and will take some helping
hands, but we first of all need to agree on the basic structure and
requirements, and then give it an (even small) start.

> 
> 
> Towards this last item, Ive done 2 things:
> 
> - poached code from Hannes Mayer :-)
>    http://www.captain.at/xenomai.php
>    task-timers.c does periodic timer 3 ways:
>       sleeper, waiter, alarm.
> 
> - scrounged old rtai/fusion code (ls -l says Jul 05 ;-)
>    cleaned up, 1/2 compile now
>    maybe theres examples-tutorials-patterns fodder in here.
> 
> 
> attached tarball has these in 2 top-level dirs.
> Id like to see if theres a place for them long-term, and clean
> them up so theyre correct and helpful.
> 

They can serve as a source, but may need to be aligned to a fairly
strict quality level we should apply on all examples. You know, the
clearer the code is, the better the user can adopt it, the less
questions pop up on xenomai-help - and the more time we have to hack
nice new features instead. =8)

So, how to proceed?

Jan




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

[Xenomai-core] Re: Test, benchmark, demo frameworks

[Jim Cromie]

Jan Kiszka wrote:
> Hi all,
>
> Jim raised these issues nicely to a generic level. I would like to pick
> it up and add some thoughts.
>
> Jim Cromie wrote:
>   
>> ...
>> FWIW, I noted that xeno-test is not running these:
>> - switchbench
>> - switchtest
>> - irqbench
>>
>> Im not sure they belong in xeno-test though, since they dont
>> appear to produce output that shows good vs bad performance,
>> only an informal 'sanity' check.
>>     
>
> Including switchtest depends on if xeno-test should also do some
> elementary stability tests. This can be derived from performance tests
> as well, but Gilles' switchtest does it for the various switching
> constellations more systematically.
>
>   
elementary stability says make it 1st test. before longer running 
latency tests. TBD..

> Including irqbench is more tricky as "real" hardware and a second box
> are always involved here (so far it only works over null-modem, need to
> be extended to some GPIOs or parallel port).
>
> Regarding the output of the various benchmarks I would like to cite
> myself here:
>
> https://mail.gna.org/public/xenomai-core/2006-06/msg00195.html
>
> [And as one of the major xeno-test contributors, you may feel included
> by the term "test team". ;)]
>
>   
LOL at 2nd-to-last paragraph.

wrt data collection, any updates on LTT or relayfs ?
iirc LTT was split to create relayfs and LTT++, but the latter is WIP.
With them, data-collection becomes comparatively limitless.

also, Niklaus will be happy to hear I feel ownership (ie guilt) about a 
xeno-test bug
where workloads get orphaned by middle 'workload-manager' shell not catching
a terminating condition and cleaning up.  Im not thrilled about bashing my
way thru this jobctl problem, but I'll knuckle down someday (soon?), 
reduce it
to an context-free bash script/apparatus for us to kick tires on 
busybox, etc..
Then fold into xeno-test and submit


>> And technically, dont regression tests have to yield
>> a PASS / FAIL decision ?  ;-)
>>
>>     
>
> Simple regular output is a good idea whenever the result is simple to
> express. A fatally crashing switch test due to broken support on arch
> XYZ will make it hard to issue "FAIL"... :)
>
>   
True, but that tells us something, doesnt it ?

presume a regression test that prints this --

    1..4
    ok 1 - Creating test program
    ok 2 - Test program runs, no error
    not ok 3 - infinite loop # TODO halting problem unsolved
    not ok 4 - infinite loop 2 # TODO halting problem unsolved

we can know:
- prog expects to complete 4 tests, and does so. (no segfault)
- fails 2 of them - and which ones.

http://jc.ngo.org.uk/trac-bin/trac.cgi/wiki/LibTap
has a nice code sample, which, at its core, is:
   

 #include "tap.h"
 plan_tests(4);
 ok(0, "Creating test prog");
 ok(some_function(), "Test program runs, no error\n");
  ...


I havent looked, but it looks purely header / macros / static inlines.
Theres a full TAP model, but we can use just the basics.

http://search.cpan.org/dist/Test-Harness/lib/Test/Harness/TAP.pod#Got_spare_tuits%3F

One aspect we might reject is the rule about other print output starting 
with "# ".
such other output is allowed by test harness, which complains otherwize.


>> Speaking more broadly, there are 3 possible kinds of test-progs
>>
>> - regression tests
>>    PASS / FAIL - either by exit(rc),
>>       or by printf( "%s\n", rc ? "not-ok" : "ok")
>>       see perl's regression test suite ( 100k separate tests )
>>       usually test details, are not tutorial
>>     
>
> Have you checked what is already under sim/skins/*/testsuite? I must
> confess I don't know if it is easily compilable for non-simulated
> execution as well. The best thing would be a test framework that builds
> both for the simulator and for "real" usage on the target.
>
>   
<sheepishly> never have tried the sim, beyond 1,2x, punted on some 
dependency issues..
Obviously thats no longer sufficient :-}
>> - performance tests
>>    progs stress xenomai, print performance data
>>    should help see performance problems, expose bugs
>>    xeno-test aims to collect performance data
>>    performance data must be expressive
>>       (switchtest is perhaps insufficient here)
>>     
>
> See my note above. I think some approach with a generic data collection
> suite + various data generators would be really fantastic! Just takes
> some brain(s) to design it and some hands to hack it...
>
>   
Taking the vision apart (for inspection), we have :

- xeno-test - shell based, semi-primitive,
     captures logs while running:
     machine-factors probes/reports, and performance tests
    workload management (semi-broken)
    semi-functional data delivery service (environmental challenges)
       email delivery wont work for me, others w/o local mail set up.

- Niklaus' ruby-on-rails ideas 
    (his xeno-test++ code to list was tantalizing, but Ill admit, havent 
looked since :-(

- big issue - server side availability

- klive.org
    python based client & server system
    uses twisted library - very network apps centric.
    implies server-side sophistication
    maybe even extensible, perhaps such that we could piggyback on their 
service (pipe dream ?)

- analysing the data (phase 2)
    big brains - think here !
    gnuplot has capabilties, but ideosyncracys too.
       also is flat-file centric, and *needs* massaging scripts.  
tedious, lots of moving parts.
    'R' is an analytical system
       big, powerful, complex, -ENOTIME
    punt on this:
       hacker exploration project(s)
       if we can just deliver good data regularly from a bunch of 
machines,  phase 1 is done.



>> - examples / tutorials
>>    ex: satch.c    simple, clear progs (low feature clutter, etc)
>>    Id like to see all demo/**/ progs in single dir
>>       forex satch-native, satch-vxworks, etc ..
>>       makes for easier browsing
>>    simple makefile
>>       builds out-of-tree
>>       handles kernel-modules and user-progs
>>       (Ive seen some clean ones, cant find now.  Mine are crufty:-(
>>    'patterns' of usage
>>       IWBGreat if we had common usage patterns isolated,
>>       named, and described
>>     
>
> Basically full ack! Wolfgang and I had this discussion recently in a
> private thread on how to deal with demos for the new CAN stack. Xenomai
> really needs more of this, and it needs it in a structured, regularly
> organised way. Some of my thoughts on a demo repository:
>
>  o Should be located in the same SVN repos, maybe under /demo, in order
>    to keep it in sync with ongoing API enhancements/modifications.
>    But the distribution should be separate from the release (just like
>    the simulator).
>
>   
im just happy to pull it from SVN.
a separate cd demos ; make is fine.

>  o Must be well commented, i.e. should not only show how things can be
>    done, but also why (or why not). That matches very well with your
>    patterns.
>
>  o Should include both simple beginners examples as well as a few more
>    complex demos.
>   
>  o Full ack to your Makefile requirement: stand-alone kernel (Hannes has
>    a nice generic one for both 2.4 and 2.6) and user-space makefiles. No
>    autotool stuff here. At the same time, we need a script to build and
>    maybe even run them automatically to detect breakages.
>
> This will certainly be a longer process and will take some helping
> hands, but we first of all need to agree on the basic structure and
> requirements, and then give it an (even small) start.
>
>   
>> Towards this last item, Ive done 2 things:
>>
>> - poached code from Hannes Mayer :-)
>>    http://www.captain.at/xenomai.php
>>    task-timers.c does periodic timer 3 ways:
>>       sleeper, waiter, alarm.
>>
>> - scrounged old rtai/fusion code (ls -l says Jul 05 ;-)
>>    cleaned up, 1/2 compile now
>>    maybe theres examples-tutorials-patterns fodder in here.
>>
>>
>> attached tarball has these in 2 top-level dirs.
>> Id like to see if theres a place for them long-term, and clean
>> them up so theyre correct and helpful.
>>
>>     
>
> They can serve as a source, but may need to be aligned to a fairly
> strict quality level we should apply on all examples. You know, the
> clearer the code is, the better the user can adopt it, the less
> questions pop up on xenomai-help - and the more time we have to hack
> nice new features instead. =8)
>
> So, how to proceed?
>   
<aside> wiki++ </>

1 file at a time - I suppose..

I'll start by poaching Hannes' Makefile, bundling it into an examples/ 
dir with
my 3-way version of his timer programs.  Id like see the target files 
appear as 0 len files,
( presuming this necessary for svn diff to see code I drop in )

Then we can discuss the more nuanced improvements /* adding tutorial, 
guidance */


After that, its just step and repeat, tossing crap along the way.

Of course, if anyone wants to pick up bits of the tarball, feel welcome :-D.
I stopped fixing compile errors where they 1st became unobvious.


> Jan
>
>
>
>   

-jimc

[Xenomai-core] Re: Test, benchmark, demo frameworks

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1673 bytes --]

Jim Cromie wrote:
> Jan Kiszka wrote:
>> ...
>> So, how to proceed?
>>   
> <aside> wiki++ </>
> 
> 1 file at a time - I suppose..
> 
> I'll start by poaching Hannes' Makefile, bundling it into an examples/
> dir with
> my 3-way version of his timer programs.  Id like see the target files
> appear as 0 len files,
> ( presuming this necessary for svn diff to see code I drop in )
> 
> Then we can discuss the more nuanced improvements /* adding tutorial,
> guidance */
> 
> After that, its just step and repeat, tossing crap along the way.
> 

Ok, to ease pushing things into folders, I would suggest creating this
structure in the root dir of trunk:

examples/native/...
examples/native/kernel/...
examples/posix/...
examples/...
examples/drivers/can/...
examples/drivers/serial/...

All stuff self-contained, i.e. one should simply cd to the folder type
make, maybe additionally providing some information on the xenomai
installation dir / kernel source path, and some binary pops up, ready to
run or insmod.

I would furthermore suggest to have some kind of hello-xenomai.c in
every skin example folder that is intended as the beginners tutorial,
providing an elementary framework to start hacking.


Reorganising the testsuites is another topic. I currently have no idea
how to sort things. We have some stuff under sim/skins/*/testsuite that
only builds for the simulator, there is src/testsuite, intended to be
distributed and installed on the target, and we may want to have a
larger testsuite that can be downloaded separately (my concern is also
about driver tests here). Anyone any ideas/comments on this?

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 249 bytes --]

[Xenomai-core] Re: Test, benchmark

[Jim Cromie]

[-- Attachment #1: Type: text/plain, Size: 3152 bytes --]

Jan Kiszka wrote:
> Hi all,
>
> Jim raised these issues nicely to a generic level. I would like to pick
> it up and add some thoughts.
>
> Jim Cromie wrote:
>   
>> ...
>> FWIW, I noted that xeno-test is not running these:
>> - switchbench
>> - switchtest
>> - irqbench
>>
>> Im not sure they belong in xeno-test though, since they dont
>> appear to produce output that shows good vs bad performance,
>> only an informal 'sanity' check.
>>     
>
> Including switchtest depends on if xeno-test should also do some
> elementary stability tests. This can be derived from performance tests
> as well, but Gilles' switchtest does it for the various switching
> constellations more systematically.
>
> Including irqbench is more tricky as "real" hardware and a second box
> are always involved here (so far it only works over null-modem, need to
> be extended to some GPIOs or parallel port).
>
>   

just responding to small part now..

this patch adds switchtest, switchbench (and drops switch) and irqbench.

each test-prog has a corresponding $XENOT_<progname>
with which you can inject new test arguments individually.
Most of these can be undef'd, except for XENOT_IRQBENCH,
which needs to be set in order for test to run (since the test requires
additional resources, as you noted above)

WRT switchtest, the -T option is useful, and makes its inclusion 
possible :-)
xeno-test adds -T 120, which you can override as follows

XENOT_SWITCHTEST='-T 300'
# other useful ones
XENOT_CYCLIC='-v'   # make it verbose

Fri Aug 18 07:06:20 MDT 2006
running: ./run -- -n -T 120 # switchtest
*
*
* Type ^C to stop this application.
*
*
[ 1574.162754] Xenomai: starting RTDM services.
cpu 0: 2079 context switches.
cpu 0: 4212 context switches.
cpu 0: 6336 context switches.
cpu 0: 8442 context switches.
...
cpu 0: 246981 context switches.
cpu 0: 249096 context switches.
cpu 0: 250263 context switches.
[ 1698.479703] Xenomai: stopping RTDM services.

wrt the data emitted, what can we learn from the numbers ?
They look to be increasing linearly, with some noise/perturbations.
We could do some statistics, but whats useful ?
Histogramming, averaging the delta-context-switches ?

Also, I see from the help-text that it does many kinds of context switches.
Does it make sense to run each kind for a bunch of samples,
so that we can see # and variation for each kind of switch ?


Other things (for other emails)

1 - one more stats/histogram suggests that it should be in a library
or at least a separate object-file.    Any thoughts / prefs / advice ?
Perhaps a 'do it the way I did in <X>'

2 - I believe Ive tracked down xeno-test's problem cleaning up workloads.
mkload is missing an 'exec', so the collected pid is that of an intermediate
shell, which is either killed, or goes away by itself, leaving the actual
workload reparented to init.  Ive got the spawn-cleanup mechanics working
in a separate script that works -
a -  it respawns tasks that have finished
    forex  dd if=/dev/hda1 of=/dev/null
    will complete, since hda1 is a finite device ( unlike /dev/zero )
b - it kills the tasks it started before it exits.
c - but needs more testing..


[-- Attachment #2: diff-xt-newtests --]
[-- Type: text/plain, Size: 1886 bytes --]

Index: scripts/xeno-test.in
===================================================================
--- scripts/xeno-test.in	(revision 1453)
+++ scripts/xeno-test.in	(working copy)
@@ -1,7 +1,9 @@
 #! /bin/sh
-# Adapted to be run also under the BusyBox. If you want to test it under the BusyBox use
-# busybox sh xeno-test 
-# A BusyBox >= 1.1.3 with a make defconfig should provide all needed applets.
+
+# Adapted to be run also under the BusyBox. 
+# If you want to test it this way, do: sh xeno-test
+# BusyBox >= 1.1.3 with a make defconfig should provide all needed applets.
+
 myusage() {
     cat >&1 <<EOF
 xeno-test [options]
@@ -190,16 +192,30 @@
     boxstatus
     (
         cd `dirname $0`/../testsuite/latency
-	loudly ./run --  $opts -t0
-	loudly ./run --  $opts -t1
-	loudly ./run --  $opts -t2
+	loudly ./run -- $XENOT_LATENCY $opts -t0 '# latency'
+	loudly ./run -- $XENOT_LATENCY $opts -t1 '# latency'
+	loudly ./run -- $XENOT_LATENCY $opts -t2 '# latency'
     )
-    (	cd `dirname $0`/../testsuite/switch
-	loudly ./run --  '# switch'
+    (	cd `dirname $0`/../testsuite/switchtest
+	loudly ./run -- -n -T 120 $XENOT_SWITCHTEST '# switchtest'
     )
+    (	cd `dirname $0`/../testsuite/switchbench
+	loudly ./run -- -p 10 -n -l 1000 $XENOT_SWITCHBENCH '# switchbench'
+    )
     (	cd `dirname $0`/../testsuite/cyclic
-	loudly ./run -- -p 10 -n -l 1000 '# cyclictest'
+	loudly ./run -- -p 10 -n -l 1000 $XENOT_CYCLIC '# cyclictest'
     )
+
+    if [ "$XENOT_IRQBENCH" != "" ] ; then
+	(
+	    cd `dirname $0`/../testsuite/irqbench
+	    loudly ./run -- -P 10 $XENOT_IRQBENCH -t0 '# irqbench user'
+	    loudly ./run -- -P 10 $XENOT_IRQBENCH -t1 '# irqbench kernel'
+	    loudly ./run -- -P 10 $XENOT_IRQBENCH -t2 '# irqbench irq-handler'
+	    loudly ./run -- -P 10 $XENOT_IRQBENCH -t3 '# irqbench hard-irq-handler'
+	)
+    fi
+
     boxstatus
     cleanup_load
 }

Re: [Xenomai-core] Re: Test, benchmark

[Gilles Chanteperdrix]

Jim Cromie wrote:
 > [ 1574.162754] Xenomai: starting RTDM services.
 > cpu 0: 2079 context switches.
 > cpu 0: 4212 context switches.
 > cpu 0: 6336 context switches.
 > cpu 0: 8442 context switches.
 > ...
 > cpu 0: 246981 context switches.
 > cpu 0: 249096 context switches.
 > cpu 0: 250263 context switches.
 > [ 1698.479703] Xenomai: stopping RTDM services.
 > 
 > wrt the data emitted, what can we learn from the numbers ?
 > They look to be increasing linearly, with some noise/perturbations.
 > We could do some statistics, but whats useful ?
 > Histogramming, averaging the delta-context-switches ?
 > 
 > Also, I see from the help-text that it does many kinds of context switches.
 > Does it make sense to run each kind for a bunch of samples,
 > so that we can see # and variation for each kind of switch ?

No, because one of the threads in the chain of context switches is
sleeping, otherwise the program would completely block your box. So, the
figures are largely irrelevant, the only important thing about them is
that they are increasing, it proves that the test is really switching
contexts. But the test fails if the value stop increasing, so no, the
output is useless. Note that if you add the -q option, the program will
be silent and only print the final count of context switches.

A question: I see that you always use the -n option, do you have
problems running the test without this option ? When launched with the
-n option switchtest does not test cpu context switches.

-- 


					    Gilles Chanteperdrix.

Re: [Xenomai-core] Re: Test, benchmark

[Gilles Chanteperdrix]

Gilles Chanteperdrix wrote:
 > A question: I see that you always use the -n option, do you have
 > problems running the test without this option ? When launched with the
 > -n option switchtest does not test cpu context switches.

s/cpu/FPU/


-- 


					    Gilles Chanteperdrix.

Re: [Xenomai-core] Re: Test, benchmark

[Jim Cromie]

Gilles Chanteperdrix wrote:
> Jim Cromie wrote:
>  > [ 1574.162754] Xenomai: starting RTDM services.
>  > cpu 0: 2079 context switches.
>  > cpu 0: 4212 context switches.
>  > cpu 0: 6336 context switches.
>  > cpu 0: 8442 context switches.
>  > ...
>  > cpu 0: 246981 context switches.
>  > cpu 0: 249096 context switches.
>  > cpu 0: 250263 context switches.
>  > [ 1698.479703] Xenomai: stopping RTDM services.
>  > 
>  > wrt the data emitted, what can we learn from the numbers ?
>  > They look to be increasing linearly, with some noise/perturbations.
>  > We could do some statistics, but whats useful ?
>  > Histogramming, averaging the delta-context-switches ?
>  > 
>  > Also, I see from the help-text that it does many kinds of context switches.
>  > Does it make sense to run each kind for a bunch of samples,
>  > so that we can see # and variation for each kind of switch ?
>
> No, because one of the threads in the chain of context switches is
> sleeping, otherwise the program would completely block your box. So, the
> figures are largely irrelevant, the only important thing about them is
> that they are increasing, it proves that the test is really switching
> contexts. But the test fails if the value stop increasing, so no, the
> output is useless. Note that if you add the -q option, the program will
> be silent and only print the final count of context switches.
>
> A question: I see that you always use the -n option, do you have
> problems running the test without this option ? When launched with the
> -n option switchtest does not test cpu context switches.
>
>   

I think I added it at some point when I wasnt getting output.
It works without the -n too,  which should be added via XENOT_SWITCHTEST,
not stuffed in by default.   You could just edit it out of patch, if 
otherwize satisfied...

fwiw, Im not averse to expanding to XENOTEST_OPTS_<foo>,
if you think thats better  (its probly more self-explanatory when found 
in an .rc file)

(its obviously trivial to redo the patch, lemme know)

BTW, running without -n, and with -T 120 (same as before)
I get more total context switches:


[ 1075.064980] Xenomai: starting RTDM services.
Testing FPU check routines...
r0: 1 != 2
r1: 1 != 2
r2: 1 != 2
r3: 1 != 2
r4: 1 != 2
r5: 1 != 2
r6: 1 != 2
r7: 1 != 2
FPU check routines OK.
cpu 0: 5727 context switches.
cpu 0: 11477 context switches.
cpu 0: 17227 context switches.
cpu 0: 23000 context switches.
...
cpu 0: 673164 context switches.
cpu 0: 678914 context switches.
cpu 0: 684664 context switches.
cpu 0: 688620 context switches.
[  708.976879] Xenomai: stopping RTDM services.

Offhand, that seems counterintuitive that -n gives lower growth-rate of 
total switches,
since its equivalent to a shorter list of 'threadspec's


Also, 2 possible output change requests:

a - print per-sample measures, not accumulating ones.
    this is more consistent with latency, which prints the latencies
    seen over the 1-sec sample period
    This also feeds better into histogram, w/o adding 'delta' logic to 
histogrammer

b - label output lines more in spirit of latency

RTT|  00:00:01  (in-kernel periodic task, 100 us period, priority 99)
RTH|-----lat min|-----lat avg|-----lat max|-overrun|----lat best|---lat 
worst
RTD|       4.733|      13.145|      18.518|       0|       4.733|      
18.518
RTD|       4.868|      13.169|      24.735|       0|       4.733|      
24.735
RTD|       5.014|      13.104|      36.270|       0|       4.733|      
36.270
RTD|       4.905|      13.111|      36.394|       0|       4.733|      
36.394

I have some misgivings about asking for this :

- current output needs massaging, esp b4 feeding to gnuplot
    Someday, I'll sit down and write a script to reformat the data the 
way gnuplot wants it,
    then we'll have some idea whether current form is sub-optimal.

- Ideally, we'd be using relayfs to collect data.
    More library-fodder ??


Lastly, we once had testsuite/README, I suppose it was dropped as being
fatally-out-of-date.  Presuming we want a new fresh one, could you add
an empty file to svn, so that we can patch against it ?

thanks
jimc

Re: [Xenomai-core] Re: Test, benchmark

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 933 bytes --]

Jim Cromie wrote:
> ...
> Also, 2 possible output change requests:
> 
> a - print per-sample measures, not accumulating ones.
>    this is more consistent with latency, which prints the latencies
>    seen over the 1-sec sample period
>    This also feeds better into histogram, w/o adding 'delta' logic to
> histogrammer
> 
> b - label output lines more in spirit of latency

My idea on this is to get that part (the formatting and dumping) out of
the individual test, into a shared benchmark library. That would
consolidate things, simplify the individual tests (irqbench is lacking
it therefore), and help to modify things later when requirements for the
data processing change/enhance.

Would you like to work on this? Of course, you can count on useless to
weird comments by us when you'll post any API design or patch. ;)

Jan


PS: But I'm not sure if that library would apply well to switchtest...


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-core] Re: Test, benchmark

[Gilles Chanteperdrix]

Jim Cromie wrote:
 > Offhand, that seems counterintuitive that -n gives lower growth-rate of 
 > total switches,
 > since its equivalent to a shorter list of 'threadspec's

In addition to the threads specified by the 'threadspec's list,
switchtest create a thread (named sleeper) that sleep 10 ms every time
some other thread switches to it. If there are more threads, this
sleeper thread is less likely to run, so there are more context
switches per second.

 > 
 > 
 > Also, 2 possible output change requests:
 > 
 > a - print per-sample measures, not accumulating ones.
 >     this is more consistent with latency, which prints the latencies
 >     seen over the 1-sec sample period
 >     This also feeds better into histogram, w/o adding 'delta' logic to 
 > histogrammer
 > 
 > b - label output lines more in spirit of latency

Ok, I will change the output.

-- 


					    Gilles Chanteperdrix.

Re: [Xenomai-core] Re: Test, benchmark

[Gilles Chanteperdrix]

Jim Cromie wrote:
 > I think I added it at some point when I wasnt getting output.
 > It works without the -n too,  which should be added via XENOT_SWITCHTEST,
 > not stuffed in by default.   You could just edit it out of patch, if 
 > otherwize satisfied...

Patch applied without -n, thanks.

-- 


					    Gilles Chanteperdrix.

[Xenomai-core] Re: Test, benchmark

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 750 bytes --]

Jim Cromie wrote:
> ...
> just responding to small part now..
> 
> this patch adds switchtest, switchbench (and drops switch) and irqbench.
> 
> each test-prog has a corresponding $XENOT_<progname>
> with which you can inject new test arguments individually.
> Most of these can be undef'd, except for XENOT_IRQBENCH,
> which needs to be set in order for test to run (since the test requires
> additional resources, as you noted above)

I doesn't necessarily require additional parameters (default is the
first serial port on PCs).

Why not adding a switch to xeno_test instead? Something like "-a
<additional-tests>" (e.g. "-a irqbench,whateverbench"). However, please
don't forget to document this extension (man page?).

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

[Xenomai-core] Re: Test, benchmark

[Jim Cromie]

Jan Kiszka wrote:
> Jim Cromie wrote:
>   
>> ...
>> just responding to small part now..
>>
>> this patch adds switchtest, switchbench (and drops switch) and irqbench.
>>
>> each test-prog has a corresponding $XENOT_<progname>
>> with which you can inject new test arguments individually.
>> Most of these can be undef'd, except for XENOT_IRQBENCH,
>> which needs to be set in order for test to run (since the test requires
>> additional resources, as you noted above)
>>     
>
> I doesn't necessarily require additional parameters (default is the
> first serial port on PCs).
>
> Why not adding a switch to xeno_test instead? Something like "-a
> <additional-tests>" (e.g. "-a irqbench,whateverbench"). However, please
> don't forget to document this extension (man page?).
>
> Jan
>
>   
several reasons for my preference:

- xeno-test is already cluttered with options, many of which propagate 
down to tests
- as more tests are added, more option-clashing is inevitable.
    this makes pass-downs more complicated.
- prog-specific options isolate us from option clashes / churn
- assuming -a <prog>,
    need to handle multiples on line (I havent done that with shell getopts)
    I have to wonder if all shells have uniform getopt behavior -
       we want to work on bash, sh, ash, dash ...
- I get to duck the manpage update ;-)

[Xenomai-core] Re: Test, benchmark

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1794 bytes --]

Jim Cromie wrote:
> Jan Kiszka wrote:
>> Jim Cromie wrote:
>>  
>>> ...
>>> just responding to small part now..
>>>
>>> this patch adds switchtest, switchbench (and drops switch) and irqbench.
>>>
>>> each test-prog has a corresponding $XENOT_<progname>
>>> with which you can inject new test arguments individually.
>>> Most of these can be undef'd, except for XENOT_IRQBENCH,
>>> which needs to be set in order for test to run (since the test requires
>>> additional resources, as you noted above)
>>>     
>>
>> I doesn't necessarily require additional parameters (default is the
>> first serial port on PCs).
>>
>> Why not adding a switch to xeno_test instead? Something like "-a
>> <additional-tests>" (e.g. "-a irqbench,whateverbench"). However, please
>> don't forget to document this extension (man page?).
>>
>> Jan
>>
>>   
> several reasons for my preference:
> 
> - xeno-test is already cluttered with options, many of which propagate
> down to tests
> - as more tests are added, more option-clashing is inevitable.
>    this makes pass-downs more complicated.
> - prog-specific options isolate us from option clashes / churn
> - assuming -a <prog>,
>    need to handle multiples on line (I havent done that with shell getopts)
>    I have to wonder if all shells have uniform getopt behavior -
>       we want to work on bash, sh, ash, dash ...

Maybe there are even simpler solutions, but this one seems to be
busybox-safe (sed must be on):

list="a,b,c,d"
for option in `echo $list|sed -e's/,/\n/g'`; do
	case $option in
	a)
		do_this
		;;
	b)
		do_that
		;;
	...
	esac
done

But that's just a suggestion. Your approach is simpler and, given good
documentation, is ok as well.

> - I get to duck the manpage update ;-)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 249 bytes --]

Re: [Xenomai-core] Re: Test, benchmark

[Gilles Chanteperdrix]

Jan Kiszka wrote:
 > list="a,b,c,d"
 > for option in `echo $list|sed -e's/,/\n/g'`; do
 > 	case $option in
 > 	a)
 > 		do_this
 > 		;;
 > 	b)
 > 		do_that
 > 		;;
 > 	...
 > 	esac
 > done


What about:

save_ifs="$IFS"
IFS=,
for option in $list; do
     case $option in
...
     esac
done
IFS="$save_ifs"

-- 


					    Gilles Chanteperdrix.

Re: [Xenomai-core] expected output and runtime of switchtest ?

[Gilles Chanteperdrix]

Jim Cromie wrote:
 > Jan Kiszka wrote:
 > > This is a regression test. You should see an error message or a system
 > > crash if something goes wrong. The output above looks ok (number of
 > > passed loops, kind of lifesign).
 > >
 > >   
 > OK thanks.
 > 
 > FWIW, I noted that xeno-test is not running these:
 > - switchbench
 > - switchtest
 > - irqbench
 > 
 > Im not sure they belong in xeno-test though, since they dont
 > appear to produce output that shows good vs bad performance,
 > only an informal 'sanity' check.
 > 
 > And technically, dont regression tests have to yield
 >  a PASS / FAIL decision ?  ;-)

switchtest now has a -q and -T option, like latency, that should make it
more xeno-test friendly. Running the test, say, 5 seconds, should be
enough to know if the context switches routines are working. You can
also rely on the program exit status to know if the test passed or
failed, all identified cases of failure cause switchtest to exit with an
error.

-- 


					    Gilles Chanteperdrix.