Improving ARM7 platform performance with CONFIG_PREEMPT_RT

Improving ARM7 platform performance with CONFIG_PREEMPT_RT

[Hartmut Behrens]

Hi,

I have patched a 3.2 kernel for a OMAP compatible processor (ARM
Cortex A8) to be used on an Gumstix Overo with the CONFIG_PREEMPT_RT
patch.

I have noticed that the latency of the patched kernel on this platform
is <300usec (using cyclictest), similar to what is reported for the
ARM9 platforms at
https://rt.wiki.kernel.org/index.php/CONFIG_PREEMPT_RT_Patch

Is this the best performance that can be expected from ARM7/9 platforms?

Could it be improved by using different kernel config settings - e.g.
CONFIG_LATENCY_TRACE=n  -
https://rt.wiki.kernel.org/index.php/Cyclictest mentions that latency
tracing adds significant kernel overhead?

Re: Improving ARM7 platform performance with CONFIG_PREEMPT_RT

[Tim Sander]

Hi Hartmut
> Is this the best performance that can be expected from ARM7/9 platforms?
Look there 
https://www.osadl.org/Latency-plot-of-system-in-rack-2-slot.qa-latencyplot-r2s3.0.html
There are also other OMAP devices in the nice testrack of the OSADL guys.

Further i would suppose to switch of power management (look also for the cpu idle stuff).
The advanced power saving mechanisms can create some unwanted latency.

Best regards
Tim

Hottinger Baldwin Messtechnik GmbH, Im Tiefen See 45, 64293 Darmstadt, Germany | www.hbm.com 

Registered as GmbH (German limited liability corporation) in the commercial register at the local court of Darmstadt, HRB 1147  
Company domiciled in Darmstadt | CEO: Andreas Huellhorst | Chairman of the board: James Charles Webster

Als Gesellschaft mit beschraenkter Haftung eingetragen im Handelsregister des Amtsgerichts Darmstadt unter HRB 1147 
Sitz der Gesellschaft: Darmstadt | Geschaeftsfuehrung: Andreas Huellhorst | Aufsichtsratsvorsitzender: James Charles Webster

The information in this email is confidential. It is intended solely for the addressee. If you are not the intended recipient, please let me know and delete this email.

Die in dieser E-Mail enthaltene Information ist vertraulich und lediglich fur den Empfaenger bestimmt. Sollten Sie nicht der eigentliche Empfaenger sein, informieren Sie mich bitte kurz und loeschen diese E-Mail.

Re: Improving ARM7 platform performance with CONFIG_PREEMPT_RT

[Carsten Emde]

Hartmut,

> I have patched a 3.2 kernel for a OMAP compatible processor (ARM
> Cortex A8) to be used on an Gumstix Overo with the CONFIG_PREEMPT_RT
> patch.
>
> I have noticed that the latency of the patched kernel on this platform
> is<300usec (using cyclictest), similar to what is reported for the
> ARM9 platforms at
> https://rt.wiki.kernel.org/index.php/CONFIG_PREEMPT_RT_Patch
>
> Is this the best performance that can be expected from ARM7/9 platforms?
No, some of them have latencies below 100 us.

For a comparison of the various ARM platforms, revisions and versions, 
you may refer to the OSADL QA Farm.

Relatively long latencies (this is a know CPU cache issue):
- Rack #2, Slot #1
   ARM926EJ-S rev 4 (v5l), Phytec/phyCARD-i.MX27
   Latency plot -> https://www.osadl.org/?id=990
- Rack #2, Slot #2
   ARMv7 rev 3 (v7l), Phytec/phyCARD-L OMAP3525
   Latency plot -> https://www.osadl.org/?id=988

Short latencies:
- Rack #0, Slot #5
   ARMv7 rev 5 (v7l), Freescale/MX53 Quickstart Board (LOCO)
   Latency plot -> https://www.osadl.org/?id=1351
- Rack #1, Slot #7
   ARMv6-compatible rev 3 (v6l), (undisclosed)
   Latency plot -> https://www.osadl.org/?id=1335
- Rack #2, Slot #3
   ARMv7 rev 7 (v7l), Texas Instruments/OMAP3517/AM3517 EVM
   Latency plot -> https://www.osadl.org/?id=887
- Rack #2, Slot #8
   ARMv6-compatible rev 3 (v6l), Phytec/PhyCARD-M pca101
   Latency plot -> https://www.osadl.org/?id=920
- Rack #4, Slot #4
   ARMv7 Processor rev 10 (v7l)x2, Texas Instruments OMAP4/Pandaboard
   Latency plot -> https://www.osadl.org/?id=911
- Rack #5, Slot #5
   Feroceon 88FR131 rev 1 (v5l), Marvell/SheevaPlug
   Latency plot -> https://www.osadl.org/?id=881

> Could it be improved by using different kernel config settings - e.g.
> CONFIG_LATENCY_TRACE=n  -
> https://rt.wiki.kernel.org/index.php/Cyclictest mentions that latency
> tracing adds significant kernel overhead?
CONFIG_LATENCY_TRACE no longer is a valid kernel configuration item. But 
even if you configure a valid tracing configuration, this will not lead 
to a relevant increase of the system's latencies, since tracing must 
explicitly be enabled to become effective, such as
# echo <your tracer here> >/sys/kernel/debug/tracing/current_tracer
or using cyclictest's trace options.

Of course, a particular kernel config may lead to long latencies. To 
check your kernel configuration, you may compare it to the kernel 
configurations of the OSADL QA Farm systems that are given at the bottom 
of the system profiles, the overview is here -> 
https://www.osadl.org/?id=1084.

Hope this helps,
	-Carsten.

Re: Improving ARM7 platform performance with CONFIG_PREEMPT_RT

[Hartmut Behrens]

Thank you - that did the trick.

By disabling  CONFIG_OMAP_32K_TIMER and the "frequency scaling power
saving stuff" I now have an average latency around 26usec and max
around 40usec.


On Wed, Sep 12, 2012 at 1:39 PM, Carsten Emde <C.Emde@osadl.org> wrote:
> Hartmut,
>
>
>> I have patched a 3.2 kernel for a OMAP compatible processor (ARM
>> Cortex A8) to be used on an Gumstix Overo with the CONFIG_PREEMPT_RT
>> patch.
>>
>> I have noticed that the latency of the patched kernel on this platform
>> is<300usec (using cyclictest), similar to what is reported for the
>> ARM9 platforms at
>> https://rt.wiki.kernel.org/index.php/CONFIG_PREEMPT_RT_Patch
>>
>> Is this the best performance that can be expected from ARM7/9 platforms?
>
> No, some of them have latencies below 100 us.
>
> For a comparison of the various ARM platforms, revisions and versions, you
> may refer to the OSADL QA Farm.
>
> Relatively long latencies (this is a know CPU cache issue):
> - Rack #2, Slot #1
>   ARM926EJ-S rev 4 (v5l), Phytec/phyCARD-i.MX27
>   Latency plot -> https://www.osadl.org/?id=990
> - Rack #2, Slot #2
>   ARMv7 rev 3 (v7l), Phytec/phyCARD-L OMAP3525
>   Latency plot -> https://www.osadl.org/?id=988
>
> Short latencies:
> - Rack #0, Slot #5
>   ARMv7 rev 5 (v7l), Freescale/MX53 Quickstart Board (LOCO)
>   Latency plot -> https://www.osadl.org/?id=1351
> - Rack #1, Slot #7
>   ARMv6-compatible rev 3 (v6l), (undisclosed)
>   Latency plot -> https://www.osadl.org/?id=1335
> - Rack #2, Slot #3
>   ARMv7 rev 7 (v7l), Texas Instruments/OMAP3517/AM3517 EVM
>   Latency plot -> https://www.osadl.org/?id=887
> - Rack #2, Slot #8
>   ARMv6-compatible rev 3 (v6l), Phytec/PhyCARD-M pca101
>   Latency plot -> https://www.osadl.org/?id=920
> - Rack #4, Slot #4
>   ARMv7 Processor rev 10 (v7l)x2, Texas Instruments OMAP4/Pandaboard
>   Latency plot -> https://www.osadl.org/?id=911
> - Rack #5, Slot #5
>   Feroceon 88FR131 rev 1 (v5l), Marvell/SheevaPlug
>   Latency plot -> https://www.osadl.org/?id=881
>
>
>> Could it be improved by using different kernel config settings - e.g.
>> CONFIG_LATENCY_TRACE=n  -
>> https://rt.wiki.kernel.org/index.php/Cyclictest mentions that latency
>> tracing adds significant kernel overhead?
>
> CONFIG_LATENCY_TRACE no longer is a valid kernel configuration item. But
> even if you configure a valid tracing configuration, this will not lead to a
> relevant increase of the system's latencies, since tracing must explicitly
> be enabled to become effective, such as
> # echo <your tracer here> >/sys/kernel/debug/tracing/current_tracer
> or using cyclictest's trace options.
>
> Of course, a particular kernel config may lead to long latencies. To check
> your kernel configuration, you may compare it to the kernel configurations
> of the OSADL QA Farm systems that are given at the bottom of the system
> profiles, the overview is here -> https://www.osadl.org/?id=1084.
>
> Hope this helps,
>         -Carsten.

Re: Improving ARM7 platform performance with CONFIG_PREEMPT_RT

[Sven-Thorsten Dietrich]

On Sep 12, 2012, at 3:20 AM, Hartmut Behrens <hartmut.behrens@gmail.com> wrote:

> Hi,
> 
> I have patched a 3.2 kernel for a OMAP compatible processor (ARM
> Cortex A8) to be used on an Gumstix Overo with the CONFIG_PREEMPT_RT
> patch.
> 
> I have noticed that the latency of the patched kernel on this platform
> is <300usec (using cyclictest), similar to what is reported for the
> ARM9 platforms at
> https://rt.wiki.kernel.org/index.php/CONFIG_PREEMPT_RT_Patch
> 
> Is this the best performance that can be expected from ARM7/9 platforms?
> 
> Could it be improved by using different kernel config settings - e.g.
> CONFIG_LATENCY_TRACE=n  -
> https://rt.wiki.kernel.org/index.php/Cyclictest mentions that latency
> tracing adds significant kernel overhead?


There are a lot of options that can be eliminated / optimized to reduce memory footprint and overhead.

If you are not using large scale SMP features, things like cgroups, name spaces and so on may shave a few nanoseconds here and there, and that adds up.

You'll have to experiment with the configs, but for typical embedded, less is more.

Historically, if you can run a UP configured Kernel, that tends to be much more lightweight.

The OSADL configs Carsten referenced in separate email are certainly a good starting point.

If you find compile errors as you twiddle with options, please do send the relevant snippets to this list.

Thanks

Sven



> --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Improving ARM7 platform performance with CONFIG_PREEMPT_RT

[Sven-Thorsten Dietrich]

On Sep 12, 2012, at 3:20 AM, Hartmut Behrens <hartmut.behrens@gmail.com> wrote:

> Hi,
> 
> I have patched a 3.2 kernel for a OMAP compatible processor (ARM
> Cortex A8) to be used on an Gumstix Overo with the CONFIG_PREEMPT_RT
> patch.
> 

Any interest in sending the patches to the list?

Thanks

Sven

Re: Improving ARM7 platform performance with CONFIG_PREEMPT_RT

[Gilles Chanteperdrix]

On 09/12/2012 09:01 PM, Sven-Thorsten Dietrich wrote:

> 
> On Sep 12, 2012, at 3:20 AM, Hartmut Behrens
> <hartmut.behrens@gmail.com> wrote:
> 
>> Hi,
>> 
>> I have patched a 3.2 kernel for a OMAP compatible processor (ARM 
>> Cortex A8) to be used on an Gumstix Overo with the
>> CONFIG_PREEMPT_RT patch.
>> 
>> I have noticed that the latency of the patched kernel on this
>> platform is <300usec (using cyclictest), similar to what is
>> reported for the ARM9 platforms at 
>> https://rt.wiki.kernel.org/index.php/CONFIG_PREEMPT_RT_Patch
>> 
>> Is this the best performance that can be expected from ARM7/9
>> platforms?
>> 
>> Could it be improved by using different kernel config settings -
>> e.g. CONFIG_LATENCY_TRACE=n  - 
>> https://rt.wiki.kernel.org/index.php/Cyclictest mentions that
>> latency tracing adds significant kernel overhead?
> 
> 
> There are a lot of options that can be eliminated / optimized to
> reduce memory footprint and overhead.


One of these options is, maybe surprisingly, CONFIG_THUMB2_KERNEL.

-- 
                                                                Gilles.

Re: Improving ARM7 platform performance with CONFIG_PREEMPT_RT

[Hartmut Behrens]

It's really quite trivial:

The kernel in question i used is available at
git://www.sakoman.com/git/linux-omap-2.6.git.
I used the omap-3.2 branch, commit 33128932803c3f8c35fe8dae257901deb60db2aa

I applied patch-3.2.28 from http://www.kernel.org/pub/linux/kernel/v3.x/

The only change required to patch-3.2.28 in order for it to apply cleanly is:

--- patch-3.2.28        2012-09-13 06:15:07.000000000 +0200
+++ patch-3.2.28.patch  2012-09-13 06:15:09.000000000 +0200
@@ -3176,6 +3176,7 @@
        outer_cache.inv_all = l2x0_inv_all;
        outer_cache.disable = l2x0_disable;
 -      outer_cache.set_debug = l2x0_set_debug;
+       outer_cache.clean_all = l2x0_clean_all;

        printk(KERN_INFO "%s cache controller enabled\n", type);
        printk(KERN_INFO "l2x0: %d ways, CACHE_ID 0x%08x, AUX_CTRL
0x%08x, Cache size: %d B\n",

Following this, the patch-3.2.28-rt42 from
http://www.kernel.org/pub/linux/kernel/projects/rt/3.2/ applies
cleanly as well.

On Wed, Sep 12, 2012 at 10:22 PM, Sven-Thorsten Dietrich
<thebigcorporation@gmail.com> wrote:
>
> On Sep 12, 2012, at 3:20 AM, Hartmut Behrens <hartmut.behrens@gmail.com> wrote:
>
>> Hi,
>>
>> I have patched a 3.2 kernel for a OMAP compatible processor (ARM
>> Cortex A8) to be used on an Gumstix Overo with the CONFIG_PREEMPT_RT
>> patch.
>>
>
> Any interest in sending the patches to the list?
>
> Thanks
>
> Sven