* Kernel config options that impact latency
@ 2018-01-04 18:29 Hindman, Gavin
2018-01-04 19:24 ` Austin Schuh
` (2 more replies)
0 siblings, 3 replies; 6+ messages in thread
From: Hindman, Gavin @ 2018-01-04 18:29 UTC (permalink / raw)
To: linux-rt-users
Since we seem to be in knowledge-sharing mode, is there an existing list of kernel config parameters, outside of those required to enable preempt_rt, known to impact or improve general system latency? I didn't find anything prowling the various wikis.
Thanks,
Gavin Hindman
Intel Open-Source Technology Center
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Kernel config options that impact latency
2018-01-04 18:29 Kernel config options that impact latency Hindman, Gavin
@ 2018-01-04 19:24 ` Austin Schuh
2018-01-05 10:09 ` Tim Sander
2018-01-04 19:37 ` Nicholas Mc Guire
2018-01-12 20:01 ` Grygorii Strashko
2 siblings, 1 reply; 6+ messages in thread
From: Austin Schuh @ 2018-01-04 19:24 UTC (permalink / raw)
To: Hindman, Gavin; +Cc: linux-rt-users
On Thu, Jan 4, 2018 at 10:29 AM, Hindman, Gavin <gavin.hindman@intel.com> wrote:
> Since we seem to be in knowledge-sharing mode, is there an existing list of kernel config parameters, outside of those required to enable preempt_rt, known to impact or improve general system latency? I didn't find anything prowling the various wikis.
For ARM platforms, we've found turning off power management helps.
The book-keeping was costing a lot of time in latency critical spots.
Austin
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Kernel config options that impact latency
2018-01-04 18:29 Kernel config options that impact latency Hindman, Gavin
2018-01-04 19:24 ` Austin Schuh
@ 2018-01-04 19:37 ` Nicholas Mc Guire
2018-02-01 18:40 ` Reinette Chatre
2018-01-12 20:01 ` Grygorii Strashko
2 siblings, 1 reply; 6+ messages in thread
From: Nicholas Mc Guire @ 2018-01-04 19:37 UTC (permalink / raw)
To: Hindman, Gavin; +Cc: linux-rt-users
On Thu, Jan 04, 2018 at 06:29:37PM +0000, Hindman, Gavin wrote:
> Since we seem to be in knowledge-sharing mode, is there an existing list of kernel config parameters, outside of those required to enable preempt_rt, known to impact or improve general system latency? I didn't find anything prowling the various wikis.
>
Depending on what you mean with general system latency -
isolation related config items will allow you to improve the effectiveness
of pinning rt tasks to specific cores (e.g. cgroups) further
physical isolation like PALLOC (out of tree patch at
https://github.com/heechul/palloc) allow reducing cache side-effects
and L2/L3 sharing which can be a significant latency source.
of course these config items have no effect just by turning them on
but they allow to configure system software architectures that are
more suitable/robust for mixed rt/non-rt workloads.
thx!
hofrat
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Kernel config options that impact latency
2018-01-04 19:24 ` Austin Schuh
@ 2018-01-05 10:09 ` Tim Sander
0 siblings, 0 replies; 6+ messages in thread
From: Tim Sander @ 2018-01-05 10:09 UTC (permalink / raw)
To: linux-rt-users; +Cc: Hindman, Gavin
Hi
Am Donnerstag, 4. Januar 2018, 11:24:37 CET schrieben Sie:
> On Thu, Jan 4, 2018 at 10:29 AM, Hindman, Gavin <gavin.hindman@intel.com> wrote:
> For ARM platforms, we've found turning off power management helps.
> The book-keeping was costing a lot of time in latency critical spots.
Its not only the book-keeping. I also patched out wfi on i.mx3x since waking
up from sleeping took some considerable delays. But that depends on your latency
requirements and your actual cpu.
The patch below is for an older 3.4.x kernel as example for further tuning power vs. latency.
Best regards
Tim
diff --git a/arch/arm/mach-imx/mm-imx3.c b/arch/arm/mach-imx/mm-imx3.c
index 7412738..7b3d312 100644
--- a/arch/arm/mach-imx/mm-imx3.c
+++ b/arch/arm/mach-imx/mm-imx3.c
@@ -33,33 +33,6 @@
static void imx3_idle(void)
{
- unsigned long reg = 0;
-
- mx3_cpu_lp_set(MX3_WAIT);
-
- __asm__ __volatile__(
- /* disable I and D cache */
- "mrc p15, 0, %0, c1, c0, 0\n"
- "bic %0, %0, #0x00001000\n"
- "bic %0, %0, #0x00000004\n"
- "mcr p15, 0, %0, c1, c0, 0\n"
- /* invalidate I cache */
- "mov %0, #0\n"
- "mcr p15, 0, %0, c7, c5, 0\n"
- /* clear and invalidate D cache */
- "mov %0, #0\n"
- "mcr p15, 0, %0, c7, c14, 0\n"
- /* WFI */
- "mov %0, #0\n"
- "mcr p15, 0, %0, c7, c0, 4\n"
- "nop\n" "nop\n" "nop\n" "nop\n"
- "nop\n" "nop\n" "nop\n"
- /* enable I and D cache */
- "mrc p15, 0, %0, c1, c0, 0\n"
- "orr %0, %0, #0x00001000\n"
- "orr %0, %0, #0x00000004\n"
- "mcr p15, 0, %0, c1, c0, 0\n"
- : "=r" (reg));
}
static void __iomem *imx3_ioremap_caller(unsigned long phys_addr, size_t size,
Hottinger Baldwin Messtechnik GmbH, Im Tiefen See 45, 64293 Darmstadt, Germany | www.hbm.com
Registered as GmbH (German limited liability corporation) in the commercial register at the local court of Darmstadt, HRB 1147
Company domiciled in Darmstadt | Managing Director: Andreas Huellhorst | Chairman of the board: Eoghan O?Lionaird
Als Gesellschaft mit beschraenkter Haftung eingetragen im Handelsregister des Amtsgerichts Darmstadt unter HRB 1147
Sitz der Gesellschaft: Darmstadt | Geschaeftsfuehrung: Andreas Huellhorst | Aufsichtsratsvorsitzender: Eoghan O?Lionaird
The information in this email is confidential. It is intended solely for the addressee. If you are not the intended recipient, please let me know and delete this email.
Die in dieser E-Mail enthaltene Information ist vertraulich und lediglich fuer den Empfaenger bestimmt. Sollten Sie nicht der eigentliche Empfaenger sein, informieren Sie mich bitte kurz und loeschen diese E-Mail.
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: Kernel config options that impact latency
2018-01-04 18:29 Kernel config options that impact latency Hindman, Gavin
2018-01-04 19:24 ` Austin Schuh
2018-01-04 19:37 ` Nicholas Mc Guire
@ 2018-01-12 20:01 ` Grygorii Strashko
2 siblings, 0 replies; 6+ messages in thread
From: Grygorii Strashko @ 2018-01-12 20:01 UTC (permalink / raw)
To: Hindman, Gavin, linux-rt-users
On 01/04/2018 12:29 PM, Hindman, Gavin wrote:
> Since we seem to be in knowledge-sharing mode, is there an existing list of kernel config parameters,
outside of those required to enable preempt_rt, known to impact or improve general system latency?
I didn't find anything prowling the various wikis.
>
You can pay attention on
CONFIG_RCU_BOOST
CONFIG_RCU_NOCB_xxx
Also, as mentioned, disabling PM, especially CPU PM is helpful.
CONFIG_CPU_IDLE=n
CONFIG_CPU_FREQ=n
Note. you can also use CONFIG_GENERIC_IDLE_POLL_SETUP
and "nohlt" command line parameter, but you might hit overheating ;)
!!! And, of course, disable most of debug options unless you really need them.
--
regards,
-grygorii
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Kernel config options that impact latency
2018-01-04 19:37 ` Nicholas Mc Guire
@ 2018-02-01 18:40 ` Reinette Chatre
0 siblings, 0 replies; 6+ messages in thread
From: Reinette Chatre @ 2018-02-01 18:40 UTC (permalink / raw)
To: Nicholas Mc Guire, Hindman, Gavin; +Cc: linux-rt-users
Hi Nicholas,
On 1/4/2018 11:37 AM, Nicholas Mc Guire wrote:
> On Thu, Jan 04, 2018 at 06:29:37PM +0000, Hindman, Gavin wrote:
>> Since we seem to be in knowledge-sharing mode, is there an existing list of kernel config parameters, outside of those required to enable preempt_rt, known to impact or improve general system latency? I didn't find anything prowling the various wikis.
>>
>
> Depending on what you mean with general system latency -
> isolation related config items will allow you to improve the effectiveness
> of pinning rt tasks to specific cores (e.g. cgroups) further
> physical isolation like PALLOC (out of tree patch at
> https://github.com/heechul/palloc) allow reducing cache side-effects
> and L2/L3 sharing which can be a significant latency source.
> of course these config items have no effect just by turning them on
> but they allow to configure system software architectures that are
> more suitable/robust for mixed rt/non-rt workloads.
If you do have the option to use Intel hardware then please do consider
Cache Allocation Technology (CAT) instead. With CAT it is possible to
isolate portions of cache and assign cores or tasks to the different
portions in order to control the amount of cache space into which these
entities can fill. CAT thus provides hardware support for the isolation
and prioritization of key tasks you seek. Additional benefits of CAT are
that it is supported in the Linux kernel from v4.10 and there is no
guessing needed from user space on which address bits determine cache
sets (which is increasingly hard to do).
More details about Linux support of CAT can be found in
Documentation/x86/intel_rdt_ui.txt within the kernel source (v4.10+). A
technical blog about the Linux implementation is at
https://01.org/intel-rdt-linux/blogs/fyu1/2017/resource-allocation-intel%C2%AE-resource-director-technology
Reinette
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2018-02-01 18:40 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2018-01-04 18:29 Kernel config options that impact latency Hindman, Gavin
2018-01-04 19:24 ` Austin Schuh
2018-01-05 10:09 ` Tim Sander
2018-01-04 19:37 ` Nicholas Mc Guire
2018-02-01 18:40 ` Reinette Chatre
2018-01-12 20:01 ` Grygorii Strashko
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).