Configuration of real-time tasks in PREEMPT_RT kernels > 2.6.33

Configuration of real-time tasks in PREEMPT_RT kernels > 2.6.33

[Wolfgang Wallner]

Dear RT community,

I have a question regarding the configuration of real-time tasks in PREEMPT
_RT kernels > 2.6.33.
Up to now, I only have experience with 2.6.33-rt.

I use the openPOWERLINK[1] real-time ethernet stack, and have it running on an x86-box with Ubuntu 13.04.
I try to use it with two different kernel versions:
* 2.6.33.7.2-rt30
* 3.2.48-rt69

When I use kernel version 2.6.33, I get very good results (jitter of ethernet frames on the network < +/-10us).
But when I use kernel version 3.2, the results are much worse. The jitter on the network increases up to +/-50us.

In my configuration, the openPOWERLINK stack consists of a kernel module (data link layer) and a user space application.

I know that in 2.6.33 there were splitted softirqs, and that it was possible to configure each thread with its own priority.
What are the steps that need to be done in 3.x to achieve similar results as in 2.6.33?

kind regards,
Wolfgang Wallner

[1] http://sourceforge.net/projects/openpowerlink/

PS: Running cyclictest (e.g. sudo cyclictest -p 99 -n -i 500) on both kernels shows very similar results (avg latency: 15us, max latency 35us). 

SV: Configuration of real-time tasks in PREEMPT_RT kernels > 2.6.33

[Simon Falsig]

-----Oprindelig meddelelse-----
Fra: linux-rt-users-owner@vger.kernel.org
[mailto:linux-rt-users-owner@vger.kernel.org] På vegne af Wolfgang Wallner
Sendt: 26. august 2013 14:29
Til: linux-rt-users@vger.kernel.org
Emne: Configuration of real-time tasks in PREEMPT_RT kernels > 2.6.33
>
>Dear RT community,
>
>I have a question regarding the configuration of real-time tasks in
PREEMPT _RT kernels > 2.6.33.
>Up to now, I only have experience with 2.6.33-rt.
>
>I use the openPOWERLINK[1] real-time ethernet stack, and have it running
on an x86-box with Ubuntu 13.04.
>I try to use it with two different kernel versions:
>* 2.6.33.7.2-rt30
>* 3.2.48-rt69
>
>When I use kernel version 2.6.33, I get very good results (jitter of
ethernet frames on the network < +/-10us).
>But when I use kernel version 3.2, the results are much worse. The jitter
on the network increases up to +/-50us.
>
>In my configuration, the openPOWERLINK stack consists of a kernel module
(data link layer) and a user space application.
>
>I know that in 2.6.33 there were splitted softirqs, and that it was
possible to configure each thread with its own priority.
>What are the steps that need to be done in 3.x to achieve similar results
as in 2.6.33?
>
>kind regards,
>Wolfgang Wallner
>
>[1] http://sourceforge.net/projects/openpowerlink/
>
>PS: Running cyclictest (e.g. sudo cyclictest -p 99 -n -i 500) on both
kernels shows very similar results (avg latency: 15us, max latency 35us).

Hi Wolfgang,

From the openPOWERLINK documentation (linux-x86.txt):

Kernel 3.x
   The behaviour of a 3.X real-time kernel changed. Splitted softirq
threads
   are no longer available. To get a good real-time behaviour the
following
   steps have to be made:
   1. A multicore processor is required for these optimizations!
   2. Ensure that the following configuration options are set for your
      real-time kernel:
      CONFIG_RT_GROUP_SCHED is not set
      CONFIG_RCU_BOOST=y
      CONFIG_RCU_BOOST_PRIO=99
   3. Isolate the second core of the multi-core processor to be used
      exclusively for openPOWERLINK by specifying the kernel-commandline
parameter
      isolcpus=1
   4. Set the default interrupt affinity to all other cores, e.g. for a
      dual-core system to core 0 by setting the kernel-commandline
parameter
      irqaffinity=0
   5. Disable IRQ balancing by disable irqbalance. This depends on your
      Linux distribution. For example in Ubuntu edit
/etc/default/irqbalance

So to get similar performance on 3.x, it seems that you'll need to assign
a core solely for the openPOWERLINK stack. I've wondered a bit about
whether there might be a different solution myself, but hadn't had the
time to investigate much further.

Best regards,
Simon Falsig
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Configuration of real-time tasks in PREEMPT_RT kernels > 2.6.33

[Jeremy Jongepier]

[-- Attachment #1: Type: text/plain, Size: 449 bytes --]

On 08/26/2013 02:28 PM, Wolfgang Wallner wrote:
> I know that in 2.6.33 there were splitted softirqs, and that it was possible to configure each thread with its own priority.
> What are the steps that need to be done in 3.x to achieve similar results as in 2.6.33?

Isn't it still possible to configure each thread with its own priority?
Afaik IRQs still get assigned a bottom-half. I'm using rtirq myself to
prioritize softirqs.

Jeremy


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Configuration of real-time tasks in PREEMPT_RT kernels > 2.6.33

[Carsten Emde]

Wolfgang,

> I know that in 2.6.33 there were splitted softirqs, and that it was possible to configure each thread with its own priority.
> What are the steps that need to be done in 3.x to achieve similar results as in 2.6.33?
The full story is here -> https://www.osadl.org/?id=930. Among others, 
it says "Thomas Gleixner found the most elegant solution that directly 
runs the network task in the context of the IRQ thread of the related 
device and, thus, implicitly adopts its priority." BTW: Thomas did this 
work when he ported RT to kernel version 3.6, Sebastian posted his 
backport to 3.2 and 3.4 half a year ago. The missed softirq split is 
history.

	-Carsten.

Re: Configuration of real-time tasks in PREEMPT_RT kernels > 2.6.33

[Wolfgang Wallner]

Hello Carsten,

>> I know that in 2.6.33 there were splitted softirqs, and that it was possible to configure each thread with its own priority.
>> What are the steps that need to be done in 3.x to achieve similar results as in 2.6.33?
>The full story is here -> https://www.osadl.org/?id=930.

Thanks for the info, I will have a look at the 3.2 feature branch.

kind regards,
Wolfgang Wallner

SV: Configuration of real-time tasks in PREEMPT_RT kernels > 2.6.33

[Wolfgang Wallner]

Hello Simon,
> From the openPOWERLINK documentation (linux-x86.txt):

Thanks for the answer, but I already knew this documentation :)
I have applied the described steps, and I think they work as expected.
My box has two cores, and I can see in htop that nothing is running on the second cpu, except my tasks which I start with the tool taskset.
Also in /proc/interrupts I can see that all IRQs are handled by the first cpu, except the POWERLINK-related ones, which are handled by the second cpu.
So as far as I know everything is configured as expected, however, I still see non-satisfying results with kernel 3.2-rt.

I have uploaded two images with measurements results for easy comparison:
2.6.33-rt: http://imageshack.us/photo/my-images/818/q4de.png/
3.2-rt: http://imageshack.us/photo/my-images/845/nqdw.png/

Both images show the reaction of the openPOWERLINK-stack on different workloads on the two kernels.
Any further hints on how to improve the 3.2-rt results are greatly appreciated :)

kind regards, Wolfgang Wallner