Kernel config options that impact latency

Kernel config options that impact latency

[Hindman, Gavin]

Since we seem to be in knowledge-sharing mode, is there an existing list of kernel config parameters, outside of those required to enable preempt_rt, known to impact or improve general system latency?  I didn't find anything prowling the various wikis.

Thanks,
Gavin Hindman
Intel Open-Source Technology Center

Re: Kernel config options that impact latency

[Austin Schuh]

On Thu, Jan 4, 2018 at 10:29 AM, Hindman, Gavin <gavin.hindman@intel.com> wrote:
> Since we seem to be in knowledge-sharing mode, is there an existing list of kernel config parameters, outside of those required to enable preempt_rt, known to impact or improve general system latency?  I didn't find anything prowling the various wikis.

For ARM platforms, we've found turning off power management helps.
The book-keeping was costing a lot of time in latency critical spots.

Austin

Re: Kernel config options that impact latency

[Tim Sander]

Hi

Am Donnerstag, 4. Januar 2018, 11:24:37 CET schrieben Sie:
> On Thu, Jan 4, 2018 at 10:29 AM, Hindman, Gavin <gavin.hindman@intel.com> wrote:
> For ARM platforms, we've found turning off power management helps.
> The book-keeping was costing a lot of time in latency critical spots.
Its not only the book-keeping. I also patched out wfi on i.mx3x since waking
up from sleeping took some considerable delays. But that depends on your latency
requirements and your actual cpu.

The patch below is for an older 3.4.x kernel as example for further tuning power vs. latency.

Best regards
Tim

diff --git a/arch/arm/mach-imx/mm-imx3.c b/arch/arm/mach-imx/mm-imx3.c
index 7412738..7b3d312 100644
--- a/arch/arm/mach-imx/mm-imx3.c
+++ b/arch/arm/mach-imx/mm-imx3.c
@@ -33,33 +33,6 @@
 
 static void imx3_idle(void)
 {
-       unsigned long reg = 0;
-
-       mx3_cpu_lp_set(MX3_WAIT);
-
-       __asm__ __volatile__(
-               /* disable I and D cache */
-               "mrc p15, 0, %0, c1, c0, 0\n"
-               "bic %0, %0, #0x00001000\n"
-               "bic %0, %0, #0x00000004\n"
-               "mcr p15, 0, %0, c1, c0, 0\n"
-               /* invalidate I cache */
-               "mov %0, #0\n"
-               "mcr p15, 0, %0, c7, c5, 0\n"
-               /* clear and invalidate D cache */
-               "mov %0, #0\n"
-               "mcr p15, 0, %0, c7, c14, 0\n"
-               /* WFI */
-               "mov %0, #0\n"
-               "mcr p15, 0, %0, c7, c0, 4\n"
-               "nop\n" "nop\n" "nop\n" "nop\n"
-               "nop\n" "nop\n" "nop\n"
-               /* enable I and D cache */
-               "mrc p15, 0, %0, c1, c0, 0\n"
-               "orr %0, %0, #0x00001000\n"
-               "orr %0, %0, #0x00000004\n"
-               "mcr p15, 0, %0, c1, c0, 0\n"
-               : "=r" (reg));
 }
 
 static void __iomem *imx3_ioremap_caller(unsigned long phys_addr, size_t size,

Hottinger Baldwin Messtechnik GmbH, Im Tiefen See 45, 64293 Darmstadt, Germany | www.hbm.com 

Registered as GmbH (German limited liability corporation) in the commercial register at the local court of Darmstadt, HRB 1147  
Company domiciled in Darmstadt | Managing Director: Andreas Huellhorst | Chairman of the board: Eoghan O?Lionaird

Als Gesellschaft mit beschraenkter Haftung eingetragen im Handelsregister des Amtsgerichts Darmstadt unter HRB 1147 
Sitz der Gesellschaft: Darmstadt | Geschaeftsfuehrung: Andreas Huellhorst | Aufsichtsratsvorsitzender: Eoghan O?Lionaird

The information in this email is confidential. It is intended solely for the addressee. If you are not the intended recipient, please let me know and delete this email.

Die in dieser E-Mail enthaltene Information ist vertraulich und lediglich fuer den Empfaenger bestimmt. Sollten Sie nicht der eigentliche Empfaenger sein, informieren Sie mich bitte kurz und loeschen diese E-Mail.

Re: Kernel config options that impact latency

[Nicholas Mc Guire]

On Thu, Jan 04, 2018 at 06:29:37PM +0000, Hindman, Gavin wrote:
> Since we seem to be in knowledge-sharing mode, is there an existing list of kernel config parameters, outside of those required to enable preempt_rt, known to impact or improve general system latency?  I didn't find anything prowling the various wikis.
>

Depending on what you mean with general system latency -
isolation related config items will allow you to improve the effectiveness
of pinning rt tasks to specific cores (e.g. cgroups) further
physical isolation like PALLOC (out of tree patch at
https://github.com/heechul/palloc) allow reducing cache side-effects
and L2/L3 sharing which can be a significant latency source.
of course these config items have no effect just by turning them on
but they allow to configure system software architectures that are
more suitable/robust for mixed rt/non-rt workloads.

thx!
hofrat

Re: Kernel config options that impact latency

[Reinette Chatre]

Hi Nicholas,

On 1/4/2018 11:37 AM, Nicholas Mc Guire wrote:
> On Thu, Jan 04, 2018 at 06:29:37PM +0000, Hindman, Gavin wrote:
>> Since we seem to be in knowledge-sharing mode, is there an existing list of kernel config parameters, outside of those required to enable preempt_rt, known to impact or improve general system latency?  I didn't find anything prowling the various wikis.
>>
> 
> Depending on what you mean with general system latency -
> isolation related config items will allow you to improve the effectiveness
> of pinning rt tasks to specific cores (e.g. cgroups) further
> physical isolation like PALLOC (out of tree patch at
> https://github.com/heechul/palloc) allow reducing cache side-effects
> and L2/L3 sharing which can be a significant latency source.
> of course these config items have no effect just by turning them on
> but they allow to configure system software architectures that are
> more suitable/robust for mixed rt/non-rt workloads.

If you do have the option to use Intel hardware then please do consider
Cache Allocation Technology (CAT) instead. With CAT it is possible to
isolate portions of cache and assign cores or tasks to the different
portions in order to control the amount of cache space into which these
entities can fill. CAT thus provides hardware support for the isolation
and prioritization of key tasks you seek. Additional benefits of CAT are
that it is supported in the Linux kernel from v4.10 and there is no
guessing needed from user space on which address bits determine cache
sets (which is increasingly hard to do).

More details about Linux support of CAT can be found in
Documentation/x86/intel_rdt_ui.txt within the kernel source (v4.10+). A
technical blog about the Linux implementation is at
https://01.org/intel-rdt-linux/blogs/fyu1/2017/resource-allocation-intel%C2%AE-resource-director-technology

Reinette

Re: Kernel config options that impact latency

[Grygorii Strashko]

On 01/04/2018 12:29 PM, Hindman, Gavin wrote:
> Since we seem to be in knowledge-sharing mode, is there an existing list of kernel config parameters,
 outside of those required to enable preempt_rt, known to impact or improve general system latency?
  I didn't find anything prowling the various wikis.
> 

You can pay attention on 
CONFIG_RCU_BOOST
CONFIG_RCU_NOCB_xxx

Also, as mentioned, disabling PM, especially CPU PM is helpful.
CONFIG_CPU_IDLE=n
CONFIG_CPU_FREQ=n
Note. you can also use CONFIG_GENERIC_IDLE_POLL_SETUP
and "nohlt" command line parameter, but you might hit overheating ;)

!!! And, of course, disable most of debug options unless you really need them. 


-- 
regards,
-grygorii