Rt-preempt testing best practices

Rt-preempt testing best practices

[Tim Bird]

Hi all,

I'm writing to tell you about a project I'm working on.
I'm trying to collect as much information as I can about
testing real time performance of Linux.

I've started a wiki page at:
http://tree.celinuxforum.org/CelfPubWiki/RealtimeTestingBestPractices
(which is on CELF's wiki)

However, I think this material would be better placed
on the Linux RT wiki at: http://rt.wiki.kernel.org/index.php/Main_Page
Does anyone have any problem with me moving the content to there,
and continuing my attempt to document this on that site?

Note that there's some duplication of content with other
pages on the RT wiki.  I'll try to clean this up during the move.

Any comments or suggestions are welcome.

If you have done RT testing, and especially if you have published
information I can look at (and link to), please contact me.

Thanks!
 -- Tim

P.S. This is based on a discussion we had at the RT BOF at
ELC last month.

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Electronics
=============================

Re: [Celinux-dev] Rt-preempt testing best practices

[Nicholas Mc Guire]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

>
> I'm writing to tell you about a project I'm working on.
> I'm trying to collect as much information as I can about
> testing real time performance of Linux.
>
> I've started a wiki page at:
> http://tree.celinuxforum.org/CelfPubWiki/RealtimeTestingBestPractices
> (which is on CELF's wiki)
>
The tests noted in the LKML post on this page are very problematic, ping 
- -f is not testing RT at all, it keeps the kernel in a very small active
page set thus reducing page related penalties, the while loop using dd
is also not too helpfull as it will de-facto run only in memory and cause
absolutely no disk/mass-storage related interaction (try the same with
mount -o remount,sync /  first and it will be devastating ! (limited to 
ext2/ext3/ufs))

The big problem with RT tests published is that they are all looking at 
the good case, they are loading the system but assuming successfull 
operations. The worst cases pup up when you run in the error paths of the 
kernel - then a trivial application can induce very large jitter in the 
system (run crashme in the background and rerun the tests...)

Also lmbench can give a statistic view of things (and not even that very 
precisely in some case i.e. context switch measurements are flawed) so 
this is not of much help for descision makers which variant to use - it 
does not help if the average performance is good but the mobile phone or 
mp3 klicks at 1s intervales "deterministically" - so I guess RT benchmarks 
need a notion of usage-profile to be of value.


> However, I think this material would be better placed
> on the Linux RT wiki at: http://rt.wiki.kernel.org/index.php/Main_Page
> Does anyone have any problem with me moving the content to there,
> and continuing my attempt to document this on that site?
>

Note that this page is dedicated to RT-preempt and will most likely not 
cover the other variants of RT Linux availabe - so it might well be a
good idea to maintain an independant page but cross-link them ?

> Note that there's some duplication of content with other
> pages on the RT wiki.  I'll try to clean this up during the move.
>
> Any comments or suggestions are welcome.
>
> If you have done RT testing, and especially if you have published
> information I can look at (and link to), please contact me.

There are a number of publications related to both benchmarking and 
analysis of hardware related artefacts (cache,BTB,TLB,etc.) which were 
published at the real-time Linux Workshops - should I collect them 
together and send you those as tar.bz2 or just the links ?

hofrat
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFGSnkHnU7rXZKfY2oRAhAkAJ9OYUionz+BULkA90ezFz7QRcHAvACgofnY
nZbizb6oIon3AYr5dSJ0ubk=
=gPtS
-----END PGP SIGNATURE-----

Re: [Celinux-dev] Rt-preempt testing best practices

[K.R. Foley]

Nicholas Mc Guire wrote:
> 
>> I'm writing to tell you about a project I'm working on.
>> I'm trying to collect as much information as I can about
>> testing real time performance of Linux.
> 
>> I've started a wiki page at:
>> http://tree.celinuxforum.org/CelfPubWiki/RealtimeTestingBestPractices
>> (which is on CELF's wiki)
> 
> The tests noted in the LKML post on this page are very problematic, ping
> -f is not testing RT at all, it keeps the kernel in a very small active
> page set thus reducing page related penalties, the while loop using dd
> is also not too helpfull as it will de-facto run only in memory and cause
> absolutely no disk/mass-storage related interaction (try the same with
> mount -o remount,sync /  first and it will be devastating ! (limited to
> ext2/ext3/ufs))
> 
> The big problem with RT tests published is that they are all looking at
> the good case, they are loading the system but assuming successfull
> operations. The worst cases pup up when you run in the error paths of
> the kernel - then a trivial application can induce very large jitter in
> the system (run crashme in the background and rerun the tests...)
> 
> Also lmbench can give a statistic view of things (and not even that very
> precisely in some case i.e. context switch measurements are flawed) so
> this is not of much help for descision makers which variant to use - it
> does not help if the average performance is good but the mobile phone or
> mp3 klicks at 1s intervales "deterministically" - so I guess RT
> benchmarks need a notion of usage-profile to be of value.
> 
> 
>> However, I think this material would be better placed
>> on the Linux RT wiki at: http://rt.wiki.kernel.org/index.php/Main_Page
>> Does anyone have any problem with me moving the content to there,
>> and continuing my attempt to document this on that site?
> 
> 
> Note that this page is dedicated to RT-preempt and will most likely not
> cover the other variants of RT Linux availabe - so it might well be a
> good idea to maintain an independant page but cross-link them ?
> 
>> Note that there's some duplication of content with other
>> pages on the RT wiki.  I'll try to clean this up during the move.
> 
>> Any comments or suggestions are welcome.
> 
>> If you have done RT testing, and especially if you have published
>> information I can look at (and link to), please contact me.
> 
> There are a number of publications related to both benchmarking and
> analysis of hardware related artefacts (cache,BTB,TLB,etc.) which were
> published at the real-time Linux Workshops - should I collect them
> together and send you those as tar.bz2 or just the links ?
> 

I would definitely be interested in such material myself. We are
actually using the RT patches in real world scenarios on probably 50+
systems (some are development workstations and others are
hardware-in-the-loop simulation test systems) and they perform very
well. However, it would be nice to have a battery of tests (both
correctness and determinism)that could be used to compare future versions.

> hofrat
-
To unsubscribe from this list: send the line "unsubscribe
linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
   kr

Re: [Celinux-dev] Rt-preempt testing best practices

[Tim Bird]

Nicholas Mc Guire wrote:
> The tests noted in the LKML post on this page are very problematic...
Thanks, Nicholas, this is very helpful information.  I remember
you mentioning some of this at ELC, and this is exactly the
type of information I want to record.

> Note that this page is dedicated to RT-preempt and will most likely not
> cover the other variants of RT Linux availabe - so it might well be a
> good idea to maintain an independant page but cross-link them ?

It claims to cover more than just Rt-preempt, but I was hoping
for feedback from someone running the site.  I'd prefer that my
"Best Practices" doc not be limited to just RT-preempt testing,
so if the rt.wiki.kernel.org site is really just RT-preempt
maybe it would make sense to put it on the embedded linux wiki
instead.

Anyone else care to comment?  It's just wiki stuff in any event, so
it may not matter that much where it lives, as long as people
can get to it easily.

> There are a number of publications related to both benchmarking and
> analysis of hardware related artefacts (cache,BTB,TLB,etc.) which were
> published at the real-time Linux Workshops - should I collect them
> together and send you those as tar.bz2 or just the links ?

A list of links would be great!  Thanks,
 -- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Electronics
=============================