* [Xenomai-help] Results of xenomai's latency test vs. RTAI's latency test @ 2006-10-30 10:05 M. Koehrer 2006-10-30 10:22 ` Jan Kiszka 2006-10-30 11:37 ` [Xenomai-help] Results of xenomai's latency test vs. RTAI's latency test Philippe Gerum 0 siblings, 2 replies; 12+ messages in thread From: M. Koehrer @ 2006-10-30 10:05 UTC (permalink / raw) To: xenomai Hi everybody, I am currently checking XENOMAI (V2.2.3 on a 2.6.17.7 kernel P4) to see if I can use it as replacement for a RTAI 3.3-cv application. The first thing I did was to run the latency test in the the xenomai's testsuite directory. The results of the worst time latency are really ugly - about 40µs! On the very same PC I got a value of about 5µs using RTAI 3.3-cv running the RTAI's user/latency test. My question is now: Why can there be such a huge difference between the two systems on the very same hardware?? Is there a way to improve this value? The RTAI system uses a 2.4.33 kernel, the XENOMAI uses the 2.6.17.7 kernel. Could this be an issue? Thanks for any feedback on that issue! Mathias -- Mathias Koehrer mathias_koehrer@domain.hid Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer, nur 44,85 inkl. DSL- und ISDN-Grundgebühr! http://www.arcor.de/rd/emf-dsl-2 ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Xenomai-help] Results of xenomai's latency test vs. RTAI's latency test 2006-10-30 10:05 [Xenomai-help] Results of xenomai's latency test vs. RTAI's latency test M. Koehrer @ 2006-10-30 10:22 ` Jan Kiszka 2006-10-30 11:05 ` Re: [Xenomai-help] Results of xenomai's latency test vs. RTAI's latency M. Koehrer 2006-10-30 11:37 ` [Xenomai-help] Results of xenomai's latency test vs. RTAI's latency test Philippe Gerum 1 sibling, 1 reply; 12+ messages in thread From: Jan Kiszka @ 2006-10-30 10:22 UTC (permalink / raw) To: M. Koehrer; +Cc: xenomai [-- Attachment #1: Type: text/plain, Size: 1590 bytes --] M. Koehrer wrote: > Hi everybody, > > I am currently checking XENOMAI (V2.2.3 on a 2.6.17.7 kernel P4) to see if I can use it > as replacement for a RTAI 3.3-cv application. > The first thing I did was to run the latency test in the the xenomai's testsuite directory. > The results of the worst time latency are really ugly - about 40µs! > On the very same PC I got a value of about 5µs using RTAI 3.3-cv running the RTAI's > user/latency test. I'm _very_ sceptical about your 5 us. Could you elaborate on how you load your box and how long those tests ran? See also TROUBLESHOOTING in the Xenomai source tree on appropriate load for triggering the worst case. The worst difference I once measured on a Pentium 133 MHz was 10% better maximum latency for a timed task under RTAI. IIRC, it took longer for RTAI to expose this than for Xenomai. > > My question is now: Why can there be such a huge difference between the two systems on the very same > hardware?? > Is there a way to improve this value? You may what to have a look at the I-pipe tracer to analyse the worst case - also under RTAI (requires a bit hacking to obtain the same freezing feature with its latency test). It is helpful to see what scenario both system faced when running on the maximum delay. > > The RTAI system uses a 2.4.33 kernel, the XENOMAI uses the 2.6.17.7 kernel. Could this > be an issue? Nope, the latency differences between 2.4 and 2.6 are insignificant on x86 last time I checked. > > Thanks for any feedback on that issue! > > Mathias > > Jan [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 250 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Re: [Xenomai-help] Results of xenomai's latency test vs. RTAI's latency 2006-10-30 10:22 ` Jan Kiszka @ 2006-10-30 11:05 ` M. Koehrer 2006-10-30 11:39 ` Jan Kiszka 0 siblings, 1 reply; 12+ messages in thread From: M. Koehrer @ 2006-10-30 11:05 UTC (permalink / raw) To: jan.kiszka, mathias_koehrer; +Cc: xenomai Hi Jan, thanks for your reply. > > I am currently checking XENOMAI (V2.2.3 on a 2.6.17.7 kernel P4) to see if > I can use it > > as replacement for a RTAI 3.3-cv application. > > The first thing I did was to run the latency test in the the xenomai's > testsuite directory. > > The results of the worst time latency are really ugly - about 40µs! > > On the very same PC I got a value of about 5µs using RTAI 3.3-cv running > the RTAI's > > user/latency test. > > I'm _very_ sceptical about your 5 us. Could you elaborate on how you > load your box and how long those tests ran? See also TROUBLESHOOTING in > the Xenomai source tree on appropriate load for triggering the worst case. My test setup was the same in both cases. I connect to the RTAI/Xenomai PC via telnet and let the latency test run for about 5 minutes. No other activity happens (well, there are a couple of services running like apache, ... - but they should not cause any load). The strange thing, is that in Xenomai even after a few seconds a latency in the range of the maximum latency is shown. I can retry the tests using a background stress situation. -- Mathias Koehrer mathias_koehrer@domain.hid Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer, nur 44,85 inkl. DSL- und ISDN-Grundgebühr! http://www.arcor.de/rd/emf-dsl-2 ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Xenomai-help] Results of xenomai's latency test vs. RTAI's latency 2006-10-30 11:05 ` Re: [Xenomai-help] Results of xenomai's latency test vs. RTAI's latency M. Koehrer @ 2006-10-30 11:39 ` Jan Kiszka 0 siblings, 0 replies; 12+ messages in thread From: Jan Kiszka @ 2006-10-30 11:39 UTC (permalink / raw) To: M. Koehrer; +Cc: xenomai [-- Attachment #1: Type: text/plain, Size: 1727 bytes --] M. Koehrer wrote: > Hi Jan, > > thanks for your reply. > >>> I am currently checking XENOMAI (V2.2.3 on a 2.6.17.7 kernel P4) to see if >> I can use it >>> as replacement for a RTAI 3.3-cv application. >>> The first thing I did was to run the latency test in the the xenomai's >> testsuite directory. >>> The results of the worst time latency are really ugly - about 40µs! >>> On the very same PC I got a value of about 5µs using RTAI 3.3-cv running >> the RTAI's >>> user/latency test. >> I'm _very_ sceptical about your 5 us. Could you elaborate on how you >> load your box and how long those tests ran? See also TROUBLESHOOTING in >> the Xenomai source tree on appropriate load for triggering the worst case. > > My test setup was the same in both cases. > I connect to the RTAI/Xenomai PC via telnet and let the latency test run for about 5 minutes. 5 min. are far too short. It takes a while to "arrange" the worst case, i.e. a dirty cache, one unrelated IRQ or atomic section running while the timer (or any other event of interest) occurs, and then one or more further IRQs right after the woken-up task re-enabled interrupts. If you hit such scenario can be observed via the latency tracer (which has some impact on the numbers, but not on the structuring). > No other activity happens (well, there are a couple of services running like > apache, ... - but they should not cause any load). > The strange thing, is that in Xenomai even after a few seconds a latency in the range of the > maximum latency is shown. Such a test on an unloaded system says almost nothing about a hard real-time system. > > I can retry the tests using a background stress situation. Jan [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 250 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Xenomai-help] Results of xenomai's latency test vs. RTAI's latency test 2006-10-30 10:05 [Xenomai-help] Results of xenomai's latency test vs. RTAI's latency test M. Koehrer 2006-10-30 10:22 ` Jan Kiszka @ 2006-10-30 11:37 ` Philippe Gerum 2006-10-30 14:15 ` Re: [Xenomai-help] Results of xenomai's latency test vs. RTAI's M. Koehrer 1 sibling, 1 reply; 12+ messages in thread From: Philippe Gerum @ 2006-10-30 11:37 UTC (permalink / raw) To: M. Koehrer; +Cc: xenomai On Mon, 2006-10-30 at 11:05 +0100, M. Koehrer wrote: > Hi everybody, > > I am currently checking XENOMAI (V2.2.3 on a 2.6.17.7 kernel P4) to see if I can use it > as replacement for a RTAI 3.3-cv application. > The first thing I did was to run the latency test in the the xenomai's testsuite directory. > The results of the worst time latency are really ugly - about 40µs! > On the very same PC I got a value of about 5µs using RTAI 3.3-cv running the RTAI's > user/latency test. > > My question is now: Why can there be such a huge difference between the two systems on the very same > hardware?? > Is there a way to improve this value? > > The RTAI system uses a 2.4.33 kernel, the XENOMAI uses the 2.6.17.7 kernel. Could this > be an issue? > Several issues there: - are you 100% sure that your kernel config file for 2.4.33 is perfectly recycled for 2.6.17, e.g. are all latency killer options as listed in the TROUBLESHOOTING file really disabled? P4 configurations are jitter-prone; some options are know to induce bad latencies. - worst-case figures do not depend on how fast you get them, the latter information gives you nothing to interpret from. You may want to stress test the box for a longer period of time, using things like e.g. a dd loop, and a compilation in the background (dd if=/dev/zero of=/somefile count=500 bs=1M). "ping" network test is not the worst latency raiser, actually, under some circumstances, it could even hide some latency issues, basically because it favours the code locality in i-cache. Also make sure to measure without X-window interaction in both cases; some graphic card drivers induce latencies, some don't. YMMV. This said, 40us on a P4 class machine is not an expected value for Xenomai/x86. To give you some reference figures, I have a uniprocessor 2.8Ghz Xeon box (no HT) performing at 15 us worst-case, and an older dual 2.4Ghz Xeon SMP which honours 25 us worst-case in SMP mode. Fact is that to get that, I need to disable a number of ACPI options (except those which are needed to boot properly in SMP mode), and activate the SMI work-around. And above all, a good starting point would be to send your kernel configuration file, so that we could discuss about facts. Additionally, you may want to upgrade to 2.2.4; 2.2.3 has a FPU issue on some hw. > Thanks for any feedback on that issue! > > Mathias > > -- Philippe. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Re: [Xenomai-help] Results of xenomai's latency test vs. RTAI's 2006-10-30 11:37 ` [Xenomai-help] Results of xenomai's latency test vs. RTAI's latency test Philippe Gerum @ 2006-10-30 14:15 ` M. Koehrer 2006-10-30 14:40 ` Philippe Gerum 2006-10-30 16:39 ` Jan Kiszka 0 siblings, 2 replies; 12+ messages in thread From: M. Koehrer @ 2006-10-30 14:15 UTC (permalink / raw) To: rpm, mathias_koehrer; +Cc: xenomai [-- Attachment #1.1: Type: text/plain, Size: 3379 bytes --] Hi Philippe, I have rebuild my system using V2.2.4 of xenomai. Also, I had a look to reuse the kernel configuration from the 2.4.33 RTAI system. However, the results are more or less the same. The only kernel module I have loaded is the "tg3" network driver, the chipset of the PC (ICH6) is supported by the SMI detection and it is activated. I have enclosed my kernel configuration, perhaps this can help to identify an issue. I do not have X11 running, it is a text-mode only system that is controlled via telnet from another PC. I'll try to do a stress test on the RTAI system as well. Regards Mathias > > I am currently checking XENOMAI (V2.2.3 on a 2.6.17.7 kernel P4) to see if > I can use it > > as replacement for a RTAI 3.3-cv application. > > The first thing I did was to run the latency test in the the xenomai's > testsuite directory. > > The results of the worst time latency are really ugly - about 40µs! > > On the very same PC I got a value of about 5µs using RTAI 3.3-cv running > the RTAI's > > user/latency test. > > > > My question is now: Why can there be such a huge difference between the > two systems on the very same > > hardware?? > > Is there a way to improve this value? > > > > The RTAI system uses a 2.4.33 kernel, the XENOMAI uses the 2.6.17.7 > kernel. Could this > > be an issue? > > > > Several issues there: > > - are you 100% sure that your kernel config file for 2.4.33 is perfectly > recycled for 2.6.17, e.g. are all latency killer options as listed in > the TROUBLESHOOTING file really disabled? P4 configurations are > jitter-prone; some options are know to induce bad latencies. > > - worst-case figures do not depend on how fast you get them, the latter > information gives you nothing to interpret from. You may want to stress > test the box for a longer period of time, using things like e.g. a dd > loop, and a compilation in the background (dd if=/dev/zero of=/somefile > count=500 bs=1M). "ping" network test is not the worst latency raiser, > actually, under some circumstances, it could even hide some latency > issues, basically because it favours the code locality in i-cache. Also > make sure to measure without X-window interaction in both cases; some > graphic card drivers induce latencies, some don't. YMMV. > > This said, 40us on a P4 class machine is not an expected value for > Xenomai/x86. To give you some reference figures, I have a uniprocessor > 2.8Ghz Xeon box (no HT) performing at 15 us worst-case, and an older > dual 2.4Ghz Xeon SMP which honours 25 us worst-case in SMP mode. Fact is > that to get that, I need to disable a number of ACPI options (except > those which are needed to boot properly in SMP mode), and activate the > SMI work-around. > > And above all, a good starting point would be to send your kernel > configuration file, so that we could discuss about facts. Additionally, > you may want to upgrade to 2.2.4; 2.2.3 has a FPU issue on some hw. > -- Mathias Koehrer mathias_koehrer@domain.hid Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer, nur 44,85 inkl. DSL- und ISDN-Grundgebühr! http://www.arcor.de/rd/emf-dsl-2 [-- Attachment #2: config_linux_2.6.17.7.gz --] [-- Type: application/octetstream, Size: 7228 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Re: [Xenomai-help] Results of xenomai's latency test vs. RTAI's 2006-10-30 14:15 ` Re: [Xenomai-help] Results of xenomai's latency test vs. RTAI's M. Koehrer @ 2006-10-30 14:40 ` Philippe Gerum 2006-10-30 15:34 ` M. Koehrer 2006-10-30 16:39 ` Jan Kiszka 1 sibling, 1 reply; 12+ messages in thread From: Philippe Gerum @ 2006-10-30 14:40 UTC (permalink / raw) To: M. Koehrer; +Cc: xenomai On Mon, 2006-10-30 at 15:15 +0100, M. Koehrer wrote: > Hi Philippe, > > I have rebuild my system using V2.2.4 of xenomai. > Also, I had a look to reuse the kernel configuration from the 2.4.33 RTAI system. > However, the results are more or less the same. > The only kernel module I have loaded is the "tg3" network driver, the chipset of the > PC (ICH6) is supported by the SMI detection and it is activated. > On such hw, you may want to enable the following: CONFIG_BLK_DEV_IDEDMA_PCI=y CONFIG_BLK_DEV_IDEDMA=y CONFIG_IDEDMA_AUTO=y > I have enclosed my kernel configuration, perhaps this can help to identify an issue. > I do not have X11 running, it is a text-mode only system that is controlled via telnet from > another PC. > > I'll try to do a stress test on the RTAI system as well. > You could try running RTAI 3.3 over 2.6.17 with the very same config file too (Xenomai opts aside, of course), so you could rule out any configuration problem if the gap still exists. > Regards > > Mathias > > > > I am currently checking XENOMAI (V2.2.3 on a 2.6.17.7 kernel P4) to see if > > I can use it > > > as replacement for a RTAI 3.3-cv application. > > > The first thing I did was to run the latency test in the the xenomai's > > testsuite directory. > > > The results of the worst time latency are really ugly - about 40µs! > > > On the very same PC I got a value of about 5µs using RTAI 3.3-cv running > > the RTAI's > > > user/latency test. > > > > > > My question is now: Why can there be such a huge difference between the > > two systems on the very same > > > hardware?? > > > Is there a way to improve this value? > > > > > > The RTAI system uses a 2.4.33 kernel, the XENOMAI uses the 2.6.17.7 > > kernel. Could this > > > be an issue? > > > > > > > Several issues there: > > > > - are you 100% sure that your kernel config file for 2.4.33 is perfectly > > recycled for 2.6.17, e.g. are all latency killer options as listed in > > the TROUBLESHOOTING file really disabled? P4 configurations are > > jitter-prone; some options are know to induce bad latencies. > > > > - worst-case figures do not depend on how fast you get them, the latter > > information gives you nothing to interpret from. You may want to stress > > test the box for a longer period of time, using things like e.g. a dd > > loop, and a compilation in the background (dd if=/dev/zero of=/somefile > > count=500 bs=1M). "ping" network test is not the worst latency raiser, > > actually, under some circumstances, it could even hide some latency > > issues, basically because it favours the code locality in i-cache. Also > > make sure to measure without X-window interaction in both cases; some > > graphic card drivers induce latencies, some don't. YMMV. > > > > This said, 40us on a P4 class machine is not an expected value for > > Xenomai/x86. To give you some reference figures, I have a uniprocessor > > 2.8Ghz Xeon box (no HT) performing at 15 us worst-case, and an older > > dual 2.4Ghz Xeon SMP which honours 25 us worst-case in SMP mode. Fact is > > that to get that, I need to disable a number of ACPI options (except > > those which are needed to boot properly in SMP mode), and activate the > > SMI work-around. > > > > And above all, a good starting point would be to send your kernel > > configuration file, so that we could discuss about facts. Additionally, > > you may want to upgrade to 2.2.4; 2.2.3 has a FPU issue on some hw. > > > > > -- > Mathias Koehrer > mathias_koehrer@domain.hid > > > Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren > ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig > und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer, > nur 44,85 € inkl. DSL- und ISDN-Grundgebühr! > http://www.arcor.de/rd/emf-dsl-2 -- Philippe. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Xenomai-help] Results of xenomai's latency test vs. RTAI's 2006-10-30 14:40 ` Philippe Gerum @ 2006-10-30 15:34 ` M. Koehrer 2006-10-30 16:23 ` Jan Kiszka ` (2 more replies) 0 siblings, 3 replies; 12+ messages in thread From: M. Koehrer @ 2006-10-30 15:34 UTC (permalink / raw) To: rpm, mathias_koehrer; +Cc: xenomai Hi everybody, I have repeated the test using a parallel stress test (dd if=/dev/sda of=/dev/null, I have a SATA harddisk). The results are that my RTAI latency of 4µs is not valid, here the latency is about 26µs. However, xenomai shows a higher latency of about 40µs. The interesting thing is, that without load RTAI seems to run very long in a very low latency region, at the same setup xenomai shows large latencies earlier. Well, I have repeated these tests on a different PC that has a server mainboard and a Pentium D 4 (real dual core) CPU. The results (measured under load) are much better here: RTAI shows a latency of about 6µs Xenomai shows a latency of about 10µs. Both values are acceptable, however I wonder where the difference comes from? Thanks for all the feedback on that issue! Regards Mathias > > I have rebuild my system using V2.2.4 of xenomai. > > Also, I had a look to reuse the kernel configuration from the 2.4.33 RTAI > system. > > However, the results are more or less the same. > > The only kernel module I have loaded is the "tg3" network driver, the > chipset of the > > PC (ICH6) is supported by the SMI detection and it is activated. > > > > On such hw, you may want to enable the following: > CONFIG_BLK_DEV_IDEDMA_PCI=y > CONFIG_BLK_DEV_IDEDMA=y > CONFIG_IDEDMA_AUTO=y > > > I have enclosed my kernel configuration, perhaps this can help to identify > an issue. > > I do not have X11 running, it is a text-mode only system that is > controlled via telnet from > > another PC. > > > > I'll try to do a stress test on the RTAI system as well. > > > > You could try running RTAI 3.3 over 2.6.17 with the very same config > file too (Xenomai opts aside, of course), so you could rule out any > configuration problem if the gap still exists. -- Mathias Koehrer mathias_koehrer@domain.hid Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer, nur 44,85 inkl. DSL- und ISDN-Grundgebühr! http://www.arcor.de/rd/emf-dsl-2 ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Xenomai-help] Results of xenomai's latency test vs. RTAI's 2006-10-30 15:34 ` M. Koehrer @ 2006-10-30 16:23 ` Jan Kiszka 2006-10-30 16:24 ` Gilles Chanteperdrix 2006-10-30 17:44 ` Philippe Gerum 2 siblings, 0 replies; 12+ messages in thread From: Jan Kiszka @ 2006-10-30 16:23 UTC (permalink / raw) To: M. Koehrer; +Cc: xenomai [-- Attachment #1: Type: text/plain, Size: 1972 bytes --] M. Koehrer wrote: > Hi everybody, > > I have repeated the test using a parallel stress test (dd if=/dev/sda of=/dev/null, I have > a SATA harddisk). > The results are that my RTAI latency of 4µs is not valid, here the latency is about 26µs. > However, xenomai shows a higher latency of about 40µs. How long did the tests run this time? To my experience anything below 30 min. is not meaningful. And the faster the machines get, the lower the probability for certain constellations becomes. So each additional hour runtime gives a little bit more confidence. > The interesting thing is, that without load RTAI seems to run very long in a very > low latency region, at the same setup xenomai shows large latencies earlier. > > > Well, I have repeated these tests on a different PC that has a server mainboard and > a Pentium D 4 (real dual core) CPU. > The results (measured under load) are much better here: > RTAI shows a latency of about 6µs > Xenomai shows a latency of about 10µs. > Both values are acceptable, however I wonder where the difference comes from? RTAI contains a few shortcuts that may explain parts of this gap, but not all. One of these shortcuts is the immediate IRQ dispatching mode (the reason why they have their own patch...). In that mode the registered timer IRQ handler gets called without caring about any I-pipe abstraction. Some others I recall are a missing clean timer subsystem abstraction (tasked are timed, and then there are also some kind of timer tasklets) and the immediate reschedule from IRQ context instead of doing this on IRQ return. Some of those features should be controllable via magic knobs, maybe it's worth a try what they actually buy you here. In any case, to really understand the differences, it takes the tracer. I will check if I can find my old RTAI hack to instrument their latency tool - if you are still interested in digging deeper on your hardware. Jan [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 250 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Xenomai-help] Results of xenomai's latency test vs. RTAI's 2006-10-30 15:34 ` M. Koehrer 2006-10-30 16:23 ` Jan Kiszka @ 2006-10-30 16:24 ` Gilles Chanteperdrix 2006-10-30 17:44 ` Philippe Gerum 2 siblings, 0 replies; 12+ messages in thread From: Gilles Chanteperdrix @ 2006-10-30 16:24 UTC (permalink / raw) To: M. Koehrer; +Cc: xenomai M. Koehrer wrote: > Hi everybody, > > I have repeated the test using a parallel stress test (dd if=/dev/sda of=/dev/null, I have > a SATA harddisk). > The results are that my RTAI latency of 4µs is not valid, here the latency is about 26µs. > However, xenomai shows a higher latency of about 40µs. Please show us your .config. -- Gilles Chanteperdrix ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Xenomai-help] Results of xenomai's latency test vs. RTAI's 2006-10-30 15:34 ` M. Koehrer 2006-10-30 16:23 ` Jan Kiszka 2006-10-30 16:24 ` Gilles Chanteperdrix @ 2006-10-30 17:44 ` Philippe Gerum 2 siblings, 0 replies; 12+ messages in thread From: Philippe Gerum @ 2006-10-30 17:44 UTC (permalink / raw) To: M. Koehrer; +Cc: xenomai On Mon, 2006-10-30 at 16:34 +0100, M. Koehrer wrote: > Hi everybody, > > I have repeated the test using a parallel stress test (dd if=/dev/sda of=/dev/null, I have > a SATA harddisk). > The results are that my RTAI latency of 4µs is not valid, here the latency is about 26µs. > However, xenomai shows a higher latency of about 40µs. > The interesting thing is, that without load RTAI seems to run very long in a very > low latency region, at the same setup xenomai shows large latencies earlier. > It has always been the case. A possible explanation could be the immediate impact of the full interrupt/event virtualization Adeos performs for Xenomai. The first contribution I once made to the RTAI project back in late 2002, was to port the 24.1.x release over Adeos/x86, in order to replace the former RTHAL. Doing so, I noticed that when running over Adeos, RTAI quickly reached its latency peak/worst-case after some time, but usually exhibited a lower variance of the mean latency. The RTHAL had shorter latencies (5-10%) which tended to converge close to the Adeos figures after a longer time (and hours of debugging), but had higher variance figures on average. Since the RTAI project has decided to go back to the RTHAL approach circa 3.2 instead of using the vanilla Adeos support anymore, this phenomenon is likely to have re-appeared when comparing a pure Adeos-based implementation (i.e. Xenomai), and a raw/direct interrupt handling from the x86 IDT to the RTOS innards. > > Well, I have repeated these tests on a different PC that has a server mainboard and > a Pentium D 4 (real dual core) CPU. > The results (measured under load) are much better here: > RTAI shows a latency of about 6µs > Xenomai shows a latency of about 10µs. > Both values are acceptable, however I wonder where the difference comes from? > Again, it's difficult to discuss this matter without figures and instrumentation at hand, but some facts are known. Among those, there is more code traversed in the Xenomai case, from the receipt of an IRQ, to the actual scheduling of the processing task, and the I-cache penalty is larger for Xenomai, which benefits less from code locality than RTAI does when it comes to measuring the performances using a basic latency test. Basically, the code in question implements the "abstract RTOS", Xenomai APIs run over. For instance, each Xenomai API having to block a thread calls a particular function into the abstract core to do so (i.e. xnpod_suspend_thread()), and this is the one and only way to suspend a thread (i.e. the API never accesses the scheduler innards directly). By contrast, each synchronization/scheduling primitive RTAI defines directly manages the scheduler internals on its own (e.g. sema4, queue, fifos, any IPC in general, and the task management services, all fiddle with the scheduler's data structures locally in the body of their own routines). A possible explanation for the (reasonably small) slowdown is that Xenomai's abstract RTOS code: - 1) needs to handle more situations since it could be called from many more contexts, than a particular RTAI service could expect; this directly translates into more complexity and CPU cycles. The good news is that we don't have to fix core issues into each and every API when the weather gets rainy, but only once in the abstract RTOS if needed (e.g. think of something weird happening in the priority inheritance code). - 2) relies on a strict functional layering, which must have some impact on the code locality at the very least, as more internal interfaces are involved. On the other hand, the advantage is that Xenomai has been successfully ported over five new architectures in less than two years, all being maintained at the very same functional level (i.e. full kernel and user-space support, tight integration within the Linux programming model). - 3) deals with the vanilla Linux kernel as a partner, not as an enemy. For instance, an additional level of complexity is induced in order to properly manage regular Linux signals. This is what it takes to be able to gracefully end any RT application running in user-space through ^C, or debugging it over GDB, without the box jumping of out the window, screaming. This "integration with Linux" part does have an indirect cost in terms of footprints, but from the user POV, it's worth the handful of microseconds spent ensuring this. And to achieve that reliably over a co-kernel technology, we need the Adeos model. Deeply. Said differently, both projects have radically different views on key key aspects: - RTAI wants to eliminate Linux from the picture (so it uses radical sideways to do so), Xenomai wants to share the picture with it (so it conforms to all Linux semantics when integration is an issue, e.g. signals). - RTAI promotes its own API, Xenomai is API-agnostic because it was born as a portability platform from traditional RTOSes to Linux (so we must rely on some RTOS abstraction, not to reinvent the wheel each time we add a new interface/emulator). - RTAI is fundamentally tied to a co-kernel technology, because of #1. Xenomai is not, because of #1, since the real-time enabling layer is isolated from the RTOS that needs its services to operate the various APIs properly. All this translates in microseconds, because nothing is for free, especially a design choice. This said, a margin of improvement still exists, but reliability will always have precedence over raw latency figures. > Thanks for all the feedback on that issue! > > Regards > > Mathias > > > I have rebuild my system using V2.2.4 of xenomai. > > > Also, I had a look to reuse the kernel configuration from the 2.4.33 RTAI > > system. > > > However, the results are more or less the same. > > > The only kernel module I have loaded is the "tg3" network driver, the > > chipset of the > > > PC (ICH6) is supported by the SMI detection and it is activated. > > > > > > > On such hw, you may want to enable the following: > > CONFIG_BLK_DEV_IDEDMA_PCI=y > > CONFIG_BLK_DEV_IDEDMA=y > > CONFIG_IDEDMA_AUTO=y > > > > > I have enclosed my kernel configuration, perhaps this can help to identify > > an issue. > > > I do not have X11 running, it is a text-mode only system that is > > controlled via telnet from > > > another PC. > > > > > > I'll try to do a stress test on the RTAI system as well. > > > > > > > You could try running RTAI 3.3 over 2.6.17 with the very same config > > file too (Xenomai opts aside, of course), so you could rule out any > > configuration problem if the gap still exists. > > -- Philippe. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Xenomai-help] Results of xenomai's latency test vs. RTAI's 2006-10-30 14:15 ` Re: [Xenomai-help] Results of xenomai's latency test vs. RTAI's M. Koehrer 2006-10-30 14:40 ` Philippe Gerum @ 2006-10-30 16:39 ` Jan Kiszka 1 sibling, 0 replies; 12+ messages in thread From: Jan Kiszka @ 2006-10-30 16:39 UTC (permalink / raw) To: M. Koehrer; +Cc: xenomai [-- Attachment #1: Type: text/plain, Size: 528 bytes --] M. Koehrer wrote: > ... > I have enclosed my kernel configuration, perhaps this can help to identify an issue. CONFIG_XENO_OPT_STATS is on which can degrade the performance slightly. Try to turn this off. I would be interested in the difference on your boxes (a few micros should be noticeable on the slower one). I don't know if RTAI has comparable accounting. If yes, check if it's off as well to keep the conditions fair. I guess CONFIG_REGPARM is not enabled for RTAI then (LXRT should break, right?). Jan [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 250 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2006-10-30 17:44 UTC | newest] Thread overview: 12+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2006-10-30 10:05 [Xenomai-help] Results of xenomai's latency test vs. RTAI's latency test M. Koehrer 2006-10-30 10:22 ` Jan Kiszka 2006-10-30 11:05 ` Re: [Xenomai-help] Results of xenomai's latency test vs. RTAI's latency M. Koehrer 2006-10-30 11:39 ` Jan Kiszka 2006-10-30 11:37 ` [Xenomai-help] Results of xenomai's latency test vs. RTAI's latency test Philippe Gerum 2006-10-30 14:15 ` Re: [Xenomai-help] Results of xenomai's latency test vs. RTAI's M. Koehrer 2006-10-30 14:40 ` Philippe Gerum 2006-10-30 15:34 ` M. Koehrer 2006-10-30 16:23 ` Jan Kiszka 2006-10-30 16:24 ` Gilles Chanteperdrix 2006-10-30 17:44 ` Philippe Gerum 2006-10-30 16:39 ` Jan Kiszka
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.