[Xenomai-core] Test, benchmark, demo frameworks (was: expected output and runtime of switchtest ?)

[Jan Kiszka <jan.kiszka@domain.hid>]

[-- Attachment #1: Type: text/plain, Size: 5103 bytes --]

Hi all,

Jim raised these issues nicely to a generic level. I would like to pick
it up and add some thoughts.

Jim Cromie wrote:
> ...
> FWIW, I noted that xeno-test is not running these:
> - switchbench
> - switchtest
> - irqbench
> 
> Im not sure they belong in xeno-test though, since they dont
> appear to produce output that shows good vs bad performance,
> only an informal 'sanity' check.

Including switchtest depends on if xeno-test should also do some
elementary stability tests. This can be derived from performance tests
as well, but Gilles' switchtest does it for the various switching
constellations more systematically.

Including irqbench is more tricky as "real" hardware and a second box
are always involved here (so far it only works over null-modem, need to
be extended to some GPIOs or parallel port).

Regarding the output of the various benchmarks I would like to cite
myself here:

https://mail.gna.org/public/xenomai-core/2006-06/msg00195.html

[And as one of the major xeno-test contributors, you may feel included
by the term "test team". ;)]

> 
> And technically, dont regression tests have to yield
> a PASS / FAIL decision ?  ;-)
> 

Simple regular output is a good idea whenever the result is simple to
express. A fatally crashing switch test due to broken support on arch
XYZ will make it hard to issue "FAIL"... :)

> 
> Speaking more broadly, there are 3 possible kinds of test-progs
> 
> - regression tests
>    PASS / FAIL - either by exit(rc),
>       or by printf( "%s\n", rc ? "not-ok" : "ok")
>       see perl's regression test suite ( 100k separate tests )
>       usually test details, are not tutorial

Have you checked what is already under sim/skins/*/testsuite? I must
confess I don't know if it is easily compilable for non-simulated
execution as well. The best thing would be a test framework that builds
both for the simulator and for "real" usage on the target.

> 
> - performance tests
>    progs stress xenomai, print performance data
>    should help see performance problems, expose bugs
>    xeno-test aims to collect performance data
>    performance data must be expressive
>       (switchtest is perhaps insufficient here)

See my note above. I think some approach with a generic data collection
suite + various data generators would be really fantastic! Just takes
some brain(s) to design it and some hands to hack it...

> 
> - examples / tutorials
>    ex: satch.c    simple, clear progs (low feature clutter, etc)
>    Id like to see all demo/**/ progs in single dir
>       forex satch-native, satch-vxworks, etc ..
>       makes for easier browsing
>    simple makefile
>       builds out-of-tree
>       handles kernel-modules and user-progs
>       (Ive seen some clean ones, cant find now.  Mine are crufty:-(
>    'patterns' of usage
>       IWBGreat if we had common usage patterns isolated,
>       named, and described

Basically full ack! Wolfgang and I had this discussion recently in a
private thread on how to deal with demos for the new CAN stack. Xenomai
really needs more of this, and it needs it in a structured, regularly
organised way. Some of my thoughts on a demo repository:

 o Should be located in the same SVN repos, maybe under /demo, in order
   to keep it in sync with ongoing API enhancements/modifications.
   But the distribution should be separate from the release (just like
   the simulator).

 o Must be well commented, i.e. should not only show how things can be
   done, but also why (or why not). That matches very well with your
   patterns.

 o Should include both simple beginners examples as well as a few more
   complex demos.

 o Full ack to your Makefile requirement: stand-alone kernel (Hannes has
   a nice generic one for both 2.4 and 2.6) and user-space makefiles. No
   autotool stuff here. At the same time, we need a script to build and
   maybe even run them automatically to detect breakages.

This will certainly be a longer process and will take some helping
hands, but we first of all need to agree on the basic structure and
requirements, and then give it an (even small) start.

> 
> 
> Towards this last item, Ive done 2 things:
> 
> - poached code from Hannes Mayer :-)
>    http://www.captain.at/xenomai.php
>    task-timers.c does periodic timer 3 ways:
>       sleeper, waiter, alarm.
> 
> - scrounged old rtai/fusion code (ls -l says Jul 05 ;-)
>    cleaned up, 1/2 compile now
>    maybe theres examples-tutorials-patterns fodder in here.
> 
> 
> attached tarball has these in 2 top-level dirs.
> Id like to see if theres a place for them long-term, and clean
> them up so theyre correct and helpful.
> 

They can serve as a source, but may need to be aligned to a fairly
strict quality level we should apply on all examples. You know, the
clearer the code is, the better the user can adopt it, the less
questions pop up on xenomai-help - and the more time we have to hack
nice new features instead. =8)

So, how to proceed?

Jan




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]