[Xenomai-core] Re: Benchmarking Plan

[Jim Cromie <jim.cromie@domain.hid>]

Philippe Gerum wrote:

>
>>> This is a partial roadmap for the project, composed of the currently
>>
> Ah! I just _knew_ you would jump in as expected. The teasing worked :o)
>
well done !   Its the mark of a great leader to get folks to do what he 
wants,
while making them think its their idea ;-)

(and I imagine thats why you ccd Takis too :-)

[lots of snippage, thruout]

>>
>> LiveCD has a few weaknesses though:
>>
>> - cant test platforms w/o cdrom
>
>
> I also think that's a serious issue. Aside of the hw availability 
> problem (e.g. non-x86 eval boards), having to burn the CD is one step 
> too many when time is a scarce resource. It often prevents to run it 
> as a fast check procedure even in the absence of any noticeable 
> problem. IOW, you won't burn a CD to run the tests unless you are 
> really stuck with some issue. So a significant part of the interest of 
> having a generic testsuite is lost: you just don't discover potential 
> problems before the serious breakage is already in the wild.
>
One thing that would help expand LiveCD's usefullness is to be able to :

- mount pirt.iso in loopback on a host (my laptop),
- export it via NFS to box-under-test,
- use pxelinux to feed LiveCD's kernel(s?)  to box when it boots.

I tried to do this, and IIRC ran into trouble with absolute symlinks
from /etc.ro to /etc.   The absoluteness fouls things when the ISO
is mounted on forex: /media/cd.

I poked a bit at trying to convince NFS to resolve them as if they
were used within a chroot jail, but I dont know enough about that.



>> - manual re-entry of data is tedious,
>> - no collection of platform data (available for automation)
>> - spotty info about cpu, memory, mobo, etc
>
which is largely user-supplied, so it can be wrong.

>> - no unattended test (still true?)
>
>
> - unfiltered preposterous data. Sometimes, data sent are just rubbish 
> because of well-known hw-related dysfunctioning or misuse of the 
> LiveCD. This perturbates the results uselessly.
>
Any ideas on how to reject these outliers ?
(defer til we have statistical analysis in place ?)

> - difficulties so far to really get a sensible digested information 
> out of the zillions of results, aside of very general figures (e.g. 
> best performer). But this is more an issue of lack of data 
> post-processors than of the LiveCD infrastructure itself.
>
yep.  And we *need* platform data to start to categorize them by platform,
important config choices, etc. We should see narrower ranges of results,
and be more able to reject the junk.

<snip>

> Additionally, LiveCD is a really great tool when it comes to help 
> people figuring out whether their respective box or brain have a 
> problem with the tested software, i.e. by automatically providing a 
> sane software (kernel+rtos) configuration and the proper way to run it 
> quite easily, a number of people could determine if their current lack 
> of luck comes from their software configuration, or rather from a more 
> serious problem.
>
yeah.  pre-built world saves a lot of early thrashing.



>> - testsuite/cruncher ?
>>
>
> The cruncher measures the impact of using the interrupt shield, but 
> this setting is now configured out by default since a majority of 
> people don't currently need it. Shield cost/performances are still 
> useful to know though.
>
OK.  adding 1 call to cruncher is simple.  Over time we *may* collect 
enough data to
make some A (shields up!)  vs B (shields down!) comparisons. 
But I dont see the data to distinguish A, B - dont we need the 
xeno/ipipe equivalent
of /proc/config.gz to do this ?

wrt testsuite/README cruncher notes, is this useful info ?

(manual insmods here...)
soekris:/usr/realtime/2.6.14-ski9-v1/testsuite/cruncher# cruncher
Calibrating cruncher...11773, done -- ideal computation time = 10023 us.
1000 samples, 1000 hz freq (pid=4183, policy=SCHED_FIFO, prio=99)
--------
Nanosleep jitter: min = 60 us, max = 192 us, avg = 77 us
Execution jitter: min = 39 us (0%), max = 72 us (0%), avg = 51 us (0%)
--------
Segmentation fault

soekris:/usr/realtime/2.6.14-ski9-v1/testsuite/cruncher# run
*
*
* Type ^C to stop this application.
*
*
Calibrating cruncher...11769, done -- ideal computation time = 10018 us.
1000 samples, 1000 hz freq (pid=4260, policy=SCHED_FIFO, prio=99)
--------
Nanosleep jitter: min = 62 us, max = 195 us, avg = 79 us
Execution jitter: min = 46 us (0%), max = 77 us (0%), avg = 57 us (0%)
--------



>> 2. send your results to xenomai.testout-at-gmail.com
>> Obviously, an official gna.org ML might be more appropriate.
>>
>
> Will appear soon.

should this wait til xeno-test is upgraded to produce good data ?
ie prevent early bogus data from being submitted.


<snip>

> As said before, the problem that currently exists with LiveCD's data, 
> is that the results are cripled with irrelevant stuff, either because 
> some people just tried it out over a simulator (ahem...), or had a 
> serious hw-generated latency issue that basically made the whole run 
> useless (mostly x86 issues: e.g. SMI stuff, legacy USB emulation, 
> powermgmt, cpufreq artefacts etc.).
>
I added a few /proc/config.gz related checks for CPU-FREQ, X86-GENERIC,
can you suggest additional checks ?




>>
>> 4. xeno-test output parser
>>
>> - /proc/ipipe/Linux-stats parse into pairs of IRQ => CPU0 prop-times
>> - such data is only comparable across kernels with eq IRQ maps
>> - currently wont handle CPU1, SMP data
>> - /proc/interrupts is slightly better parsed.
>> - no detail-parse at all for top-data, needed?
>>
>
> I'm not sure that per-process data would help, just because those are 
> way too volatile and fragmented to be interpreted rationally over a 
> long test period; maybe using per-subsystem data (e.g. /proc/sys 
> crowd) at some point in time would better help.
>



>> prototype only, but its hackable (perl), and Im happy to graft all
>> sorts of horrible experiments on it provisionally to see whats useful.
>> Hopefully a plugin refactoring will become obvious wo too much work.
>>
>
> Warning people: JimC belongs to some kind of hybridization between a 
> Perl Monger and a Real-timer; and the resulting entity is about to go 
> wild... :o>
>
go off the deep end ?  into shark infested waters ?


>
> Generally speaking, I guess that your idea is to collect sensible raw 
> data first, and devise how to process combiantions of them later. 
> Sounds ok for me, and I especially like the idea of providing a 
> specialized ML for that which would be processed by a bot', since 
> anyone would have unlimited access to the data, which might trigger 
> some incentive for anyone to craft other/better digested figures.
>
yup.  inspired by LiveCD, and your reaction to it.

>
> We should make sure to not base all the reasoning on a lo latency / hi 
> cpufreq correlation: this just happens to be wrong, especially 
> x86-wise. Actually, a lot of recent x86 platforms with insanely high 
> CPU freqs are really out of luck when it comes to perform decently in 
> real-time mode, just because the trend of "optimization" is just about 
> killing any determinism one would expect from his hw, by various ugly 
> tricks often aimed at making gamers happy.
>
pentium 4's   31 stage instruction-processing pipeline ? :-O

Im not suggesting its a good measure, but that it would make an 
interesting graph.
latency vs mhz,  with data-points colored per the CPU type.
K6 - navy-blue, K7-royal-blue, K8- sky-blue, P2 - lime-green, P3 - 
mint-green, P4 - forest green


>
> I understand "the plan behind the plan" to be able to somehow predict 
> that some particular sw / hw combo would work and help people figuring 
> out which platform they might want to build their RT solution over 
> using Xeno, and it would be quite an achievement to do that.
>
> For the time being though, I'd suggest that we focus on gathering raw 
> data and digest them according to a few simple metrics first; I'm 
> pretty sure that once a sane and simple infrastructure to do that is 
> in place, we should be able to flesh out the available results. As 
> usual, the key issue is to make such process of producing and using 
> this data becoming a routine; once people get used to something, they 
> tend to improve it quite naturally.


agreed.  Its all blue-sky dreaming atm, and subject to ongoing reality 
checks,
and ongoing discussion ( in little trickles )

jimc