The Linux Kernel Mailing List
 help / color / mirror / Atom feed
* [PATCH v2] Documentation: real-time: Add boot parameters and task isolation section
@ 2026-11-20  7:06 Zixin Wang
  2026-11-20 11:35 ` Sebastian Andrzej Siewior
  0 siblings, 1 reply; 2+ messages in thread
From: Zixin Wang @ 2026-11-20  7:06 UTC (permalink / raw)
  To: linux-doc
  Cc: linux-rt-devel, bigeasy, clrkwllms, rostedt, corbet, linux-kernel,
	Zixin Wang

Document the boot parameters and runtime tunables that affect the
worst-case latency of a real-time system, alongside the existing
kernel configuration guidance.

Cover CPU isolation, nohz_full, RCU callback offloading, interrupt
affinity, idle states, and real-time throttling, with references to
the relevant documentation.

Link: https://bugzilla.kernel.org/show_bug.cgi?id=220633
Link: https://github.com/elisa-tech/wg-aerospace/issues/222
Signed-off-by: Zixin Wang <w1355457260@gmail.com>
---
v1 -> v2:
  - Fix the kernel.sched_rt_runtime_us description: the default runtime
    is 1000000 us and, without CONFIG_RT_GROUP_SCHED, SCHED_FIFO/RR tasks
    are no longer throttled through this sysctl.  Describe the per-CPU
    fair deadline server (50 ms per 1000 ms) instead.  Reported by
    Sashiko.
  - Clarify the isolcpus= flag semantics: only the domain flag removes
    CPUs from load balancing, and the remaining flags do not affect
    unrelated kernel threads, timers, or interrupts.  Review feedback.

 .../real-time/kernel-configuration.rst        | 141 ++++++++++++++++++
 1 file changed, 141 insertions(+)

diff --git a/Documentation/core-api/real-time/kernel-configuration.rst b/Documentation/core-api/real-time/kernel-configuration.rst
index d7f08e7b..995674b9 100644
--- a/Documentation/core-api/real-time/kernel-configuration.rst
+++ b/Documentation/core-api/real-time/kernel-configuration.rst
@@ -216,6 +216,147 @@ just by being enabled.
 These tracers do incur measurable latency overhead even when tracing is not
 currently active.
 
+Boot parameters and task isolation
+==================================
+
+Kernel configuration options alone do not determine a real-time system's
+worst-case latency.  Integrators commonly rely on boot parameters and
+runtime tunables as well, to keep kernel housekeeping work away from
+latency-sensitive tasks.  The parameters below complement the options
+described above.  Each one is documented in detail in
+Documentation/admin-guide/kernel-parameters.rst; the notes here focus on
+the aspects that matter for real-time latency.  For the full CPU
+isolation model and its limitations, see
+Documentation/admin-guide/cpu-isolation.rst.
+
+.. caution::
+
+  CPU isolation is best effort.  Unbound workqueues, kernel threads,
+  timers, and interrupts may still run on isolated CPUs unless they are
+  explicitly steered away.  Measure the resulting latency with tools
+  such as ``osnoise``, ``timerlat``, and ``rtla`` rather than assuming
+  that an isolated CPU is free of kernel noise.  See
+  Documentation/timers/no_hz.rst and Documentation/tools/rtla/index.rst.
+
+CPU isolation
+-------------
+
+``isolcpus=``
+  Format: ``[flag-list,]<cpu-list>``.  Isolates the listed CPUs from the
+  disturbances selected by the flag list (default: ``domain``):
+
+  ``domain``
+    Remove the CPUs from the scheduler domains.  This is the default
+    when no flag is given.  Domain isolation cannot be undone at
+    runtime; use cpusets when the isolated CPU set must change
+    dynamically.  Isolated CPUs remain usable through CPU affinity or
+    cpusets.
+
+  ``nohz``
+    Stop the tick when a single task runs and offload kernel noise,
+    including RCU callbacks.  This is equivalent to ``nohz_full=``.
+    The residual 1 Hz tick is offloaded to workqueues, which must be
+    affined to housekeeping CPUs through the global workqueue
+    ``cpumask`` (/sys/devices/virtual/workqueue/cpumask); by default
+    that workqueue may run on any CPU.
+
+  ``managed_irq``
+    Prevent managed interrupts from being targeted at isolated CPUs
+    when their automatically computed affinity mask also contains
+    housekeeping CPUs.  This is best effort and has no effect when the
+    mask contains isolated CPUs only.  See
+    Documentation/core-api/irq/managed_irq.rst.
+
+  Apart from the best-effort managed IRQ redirection, these flags do
+  not stop kernel threads, timers, or interrupts that are already
+  affined to the isolated CPUs.  A practical configuration therefore
+  combines ``isolcpus=`` with ``nohz_full=``, ``rcu_nocbs=``, an
+  interrupt affinity mask, and a C-state limit.
+``nohz_full=``
+  Format: ``<cpu-list>``.  Requires ``CONFIG_NO_HZ_FULL=y``.  Stops the
+  scheduler tick on the listed CPUs whenever possible.  The boot CPU is
+  forced outside the range to maintain timekeeping, and the RCU
+  callbacks of the listed CPUs are offloaded as if they had been
+  passed to ``rcu_nocbs=``.  This parameter takes precedence over
+  ``CONFIG_RCU_NOCB_CPU_DEFAULT_ALL``.
+
+  Tickless operation can increase the kernel-to-userspace transition
+  latency, so it is most useful for workloads that execute in
+  userspace for long periods with few syscalls.  Periodic workloads
+  with short periods may prefer a consistent tick instead.
+
+RCU callback offloading
+-----------------------
+
+``rcu_nocbs[=cpu-list]``
+  In kernels built with ``CONFIG_RCU_NOCB_CPU=y``, move RCU callback
+  invocation for the listed CPUs out of softirq context and into
+  per-CPU ``rcuo`` kthreads.  This reduces OS jitter on the offloaded
+  CPUs.  If the argument is omitted, no CPU is offloaded at boot, but
+  the mode can be toggled at runtime through cpusets.  ``nohz_full=``
+  implies the same offload for the CPUs it lists.
+
+``rcu_nocb_poll``
+  Make the ``rcuo`` kthreads poll for callbacks instead of requiring
+  the offloaded CPUs to wake them.  This improves the real-time
+  response at the cost of higher power consumption, because the
+  kthreads wake up periodically to do the polling.
+
+Interrupt affinity
+------------------
+
+``irqaffinity=``
+  Format: ``<cpu-list>``.  Sets the default affinity mask for
+  interrupts that do not have an explicit affinity.  Managed
+  interrupts, such as MSI-X queue interrupts, derive their affinity
+  from the device queue mask instead and cannot be changed through
+  ``/proc/irq/*``.  Combine ``irqaffinity=`` with the ``managed_irq``
+  flag of ``isolcpus=`` to keep interrupts away from isolated CPUs
+  where possible.  See Documentation/core-api/irq/managed_irq.rst.
+
+Idle state limits
+-----------------
+
+``processor.max_cstate=``
+  ACPI platforms only.  Limits the deepest C-state the processor may
+  enter.  ``processor.max_cstate=1`` caps the processor at C1 and
+  avoids the entry and exit latency, cache flushing, and clock effects
+  of deeper states.  A value of 9 overrides any DMI blacklist limit.
+
+``intel_idle.max_cstate=``
+  x86 platforms using ``intel_idle`` only.  A value between 1 and 9
+  specifies the maximum C-state depth.  A value of 0 disables
+  ``intel_idle`` and falls back to ``acpi_idle``; it does not disable
+  C-states.
+
+Deeper C-states save power but increase wakeup latency.  Choose the
+shallowest state that still meets the platform's power and thermal
+requirements, and validate the result with the tools listed above.
+
+Real-time throttling
+--------------------
+
+``kernel.sched_rt_runtime_us``
+  Together with ``kernel.sched_rt_period_us``, this sysctl defines the
+  total bandwidth available to SCHED_DEADLINE entities.  When
+  ``CONFIG_RT_GROUP_SCHED`` is enabled, it also limits the bandwidth of
+  the real-time scheduling groups.  The default runtime equals the
+  period (1000000 us), so no bandwidth is subtracted from deadline
+  tasks by default.
+
+  Without ``CONFIG_RT_GROUP_SCHED``, SCHED_FIFO and SCHED_RR tasks are
+  no longer throttled through this interface.  Instead, a per-CPU fair
+  deadline server keeps non-real-time tasks running by providing
+  50 ms of runtime per 1000 ms period.  That reservation is not
+  disabled by setting this sysctl to -1; the server parameters are
+  exposed under /sys/kernel/debug/sched/fair_server/cpuX/.
+
+  A value of -1 removes the total deadline bandwidth limit, that is,
+  runtime equals period.  Do this only after validating the workload,
+  because runaway real-time tasks can then starve the rest of the
+  system.  See Documentation/scheduler/sched-rt-group.rst for RT group
+  scheduling details.
+
 Kernel Debug Options
 ====================
 
-- 
2.54.0


^ permalink raw reply related	[flat|nested] 2+ messages in thread

* Re: [PATCH v2] Documentation: real-time: Add boot parameters and task isolation section
  2026-11-20  7:06 [PATCH v2] Documentation: real-time: Add boot parameters and task isolation section Zixin Wang
@ 2026-11-20 11:35 ` Sebastian Andrzej Siewior
  0 siblings, 0 replies; 2+ messages in thread
From: Sebastian Andrzej Siewior @ 2026-11-20 11:35 UTC (permalink / raw)
  To: Zixin Wang
  Cc: linux-doc, linux-rt-devel, clrkwllms, rostedt, corbet,
	linux-kernel, Frederic Weisbecker, Daniel Wagner, Aaron Tomlin

On 2026-11-20 15:06:20 [+0800], Zixin Wang wrote:
> --- a/Documentation/core-api/real-time/kernel-configuration.rst
> +++ b/Documentation/core-api/real-time/kernel-configuration.rst
> @@ -216,6 +216,147 @@ just by being enabled.
>  These tracers do incur measurable latency overhead even when tracing is not
>  currently active.
>  
> +Boot parameters and task isolation
> +==================================
> +
> +Kernel configuration options alone do not determine a real-time system's
> +worst-case latency.  Integrators commonly rely on boot parameters and
> +runtime tunables as well, to keep kernel housekeeping work away from
> +latency-sensitive tasks.  The parameters below complement the options
> +described above.  Each one is documented in detail in
> +Documentation/admin-guide/kernel-parameters.rst; the notes here focus on
> +the aspects that matter for real-time latency.  For the full CPU
> +isolation model and its limitations, see
> +Documentation/admin-guide/cpu-isolation.rst.
> +
> +.. caution::
> +
> +  CPU isolation is best effort.  Unbound workqueues, kernel threads,

An unbound workqueue is not restricted to a certain to CPU therefore it
will not run on an isolated CPU. The same is true for kernel threads as
long as they are not restricted to the isolated CPU.


> +  timers, and interrupts may still run on isolated CPUs unless they are
> +  explicitly steered away.  Measure the resulting latency with tools
> +  such as ``osnoise``, ``timerlat``, and ``rtla`` rather than assuming
> +  that an isolated CPU is free of kernel noise.  See
> +  Documentation/timers/no_hz.rst and Documentation/tools/rtla/index.rst.
> +
> +CPU isolation
> +-------------
> +
> +``isolcpus=``
> +  Format: ``[flag-list,]<cpu-list>``.  Isolates the listed CPUs from the
> +  disturbances selected by the flag list (default: ``domain``):
> +
> +  ``domain``
> +    Remove the CPUs from the scheduler domains.  This is the default
> +    when no flag is given.  Domain isolation cannot be undone at
> +    runtime; use cpusets when the isolated CPU set must change
> +    dynamically.  Isolated CPUs remain usable through CPU affinity or
> +    cpusets.
> +
> +  ``nohz``
> +    Stop the tick when a single task runs and offload kernel noise,
> +    including RCU callbacks.  This is equivalent to ``nohz_full=``.
> +    The residual 1 Hz tick is offloaded to workqueues, which must be
> +    affined to housekeeping CPUs through the global workqueue
> +    ``cpumask`` (/sys/devices/virtual/workqueue/cpumask); by default
> +    that workqueue may run on any CPU.

By default, yes, but that options changes it.

> +  ``managed_irq``
> +    Prevent managed interrupts from being targeted at isolated CPUs
> +    when their automatically computed affinity mask also contains
> +    housekeeping CPUs.  This is best effort and has no effect when the
> +    mask contains isolated CPUs only.  See
> +    Documentation/core-api/irq/managed_irq.rst.

It is currently worked on managed_irq. Apart from this it might be
debatable that if you perform I/O on an isolated CPU should you or
should you not receive a completion interrupt on the isolated CPU
initiating the I/O. Any statement from your side?

Frederic, does this fit here from your perspective?

> +
> +  Apart from the best-effort managed IRQ redirection, these flags do
> +  not stop kernel threads, timers, or interrupts that are already
> +  affined to the isolated CPUs.  A practical configuration therefore

timers. Unless a timer is explicitly set to target a certain CPU it will
avoid to fire on the isolated CPU even if "setup" on the isolated CPU.

> +  combines ``isolcpus=`` with ``nohz_full=``, ``rcu_nocbs=``, an
> +  interrupt affinity mask, and a C-state limit.

C-state is x86 specific.

> +``nohz_full=``
> +  Format: ``<cpu-list>``.  Requires ``CONFIG_NO_HZ_FULL=y``.  Stops the
> +  scheduler tick on the listed CPUs whenever possible.  The boot CPU is
> +  forced outside the range to maintain timekeeping, and the RCU
> +  callbacks of the listed CPUs are offloaded as if they had been
> +  passed to ``rcu_nocbs=``.  This parameter takes precedence over
> +  ``CONFIG_RCU_NOCB_CPU_DEFAULT_ALL``.
> +
> +  Tickless operation can increase the kernel-to-userspace transition
> +  latency, so it is most useful for workloads that execute in
> +  userspace for long periods with few syscalls.  Periodic workloads
> +  with short periods may prefer a consistent tick instead.
> +
> +RCU callback offloading
> +-----------------------
> +
> +``rcu_nocbs[=cpu-list]``
> +  In kernels built with ``CONFIG_RCU_NOCB_CPU=y``, move RCU callback
> +  invocation for the listed CPUs out of softirq context and into
> +  per-CPU ``rcuo`` kthreads.  This reduces OS jitter on the offloaded
> +  CPUs.  If the argument is omitted, no CPU is offloaded at boot, but
> +  the mode can be toggled at runtime through cpusets.  ``nohz_full=``
> +  implies the same offload for the CPUs it lists.

This sounds very familiar with the wording in kernel-parameters.txt.
With PREEMPT_RT enabled the callbacks are not pulled out of the softirq
context and this is an additional detail. There is
rcutree.use_softirq=false on PREEMPT_RT.
What it actually does is that the callbacks are no longer invoked on the
CPU that enqueued them but in the offloaded thread which can run on the
housekeeping CPU.

> +
> +``rcu_nocb_poll``
> +  Make the ``rcuo`` kthreads poll for callbacks instead of requiring
> +  the offloaded CPUs to wake them.  This improves the real-time
> +  response at the cost of higher power consumption, because the
> +  kthreads wake up periodically to do the polling.

At this point I am not sure if you explain (and copy original
description) or collect option which are advised/ recommended.

Maybe it is a list of options worth considering.

> +
> +Interrupt affinity
> +------------------
> +
> +``irqaffinity=``
> +  Format: ``<cpu-list>``.  Sets the default affinity mask for
> +  interrupts that do not have an explicit affinity.  Managed
> +  interrupts, such as MSI-X queue interrupts, derive their affinity

this "such as MSI-X queue interrupts" probably does not belong here. The
affinity of managed-interrupts can not be changed. How they are placed
is a different story.

> +  from the device queue mask instead and cannot be changed through
> +  ``/proc/irq/*``.  Combine ``irqaffinity=`` with the ``managed_irq``
> +  flag of ``isolcpus=`` to keep interrupts away from isolated CPUs
> +  where possible.  See Documentation/core-api/irq/managed_irq.rst.
> +
> +Idle state limits
> +-----------------
> +
> +``processor.max_cstate=``
> +  ACPI platforms only.  Limits the deepest C-state the processor may
> +  enter.  ``processor.max_cstate=1`` caps the processor at C1 and
> +  avoids the entry and exit latency, cache flushing, and clock effects
> +  of deeper states.  A value of 9 overrides any DMI blacklist limit.
> +
> +``intel_idle.max_cstate=``
> +  x86 platforms using ``intel_idle`` only.  A value between 1 and 9
> +  specifies the maximum C-state depth.  A value of 0 disables
> +  ``intel_idle`` and falls back to ``acpi_idle``; it does not disable
> +  C-states.
> +
> +Deeper C-states save power but increase wakeup latency.  Choose the
> +shallowest state that still meets the platform's power and thermal
> +requirements, and validate the result with the tools listed above.
> +
> +Real-time throttling
> +--------------------
> +
> +``kernel.sched_rt_runtime_us``
> +  Together with ``kernel.sched_rt_period_us``, this sysctl defines the
> +  total bandwidth available to SCHED_DEADLINE entities.  When
> +  ``CONFIG_RT_GROUP_SCHED`` is enabled, it also limits the bandwidth of
> +  the real-time scheduling groups.  The default runtime equals the
> +  period (1000000 us), so no bandwidth is subtracted from deadline
> +  tasks by default.

I would throw this away. First of sched_rt_runtime_us is not related to
throttling anymore and then I try to get rid of CONFIG_RT_GROUP_SCHED
(and this option). And I doubt anyone seriously uses this option.

> +  Without ``CONFIG_RT_GROUP_SCHED``, SCHED_FIFO and SCHED_RR tasks are
> +  no longer throttled through this interface.  Instead, a per-CPU fair
> +  deadline server keeps non-real-time tasks running by providing
> +  50 ms of runtime per 1000 ms period.  That reservation is not
> +  disabled by setting this sysctl to -1; the server parameters are
> +  exposed under /sys/kernel/debug/sched/fair_server/cpuX/.
> +
> +  A value of -1 removes the total deadline bandwidth limit, that is,
> +  runtime equals period.  Do this only after validating the workload,
> +  because runaway real-time tasks can then starve the rest of the
> +  system.  See Documentation/scheduler/sched-rt-group.rst for RT group
> +  scheduling details.

I do try to move it someplace else, see 
	https://lore.kernel.org/all/20261113-sched_dl-v3-0-a76ffa1cdeee@linutronix.de/

The basic information is that we have throttling in place and one needs
to disable it if it interferes.

Overall I don't mind having a collection of useful options worth
considering. I'm just not sure if the format of explaining the
parameters again is the right approach. Maybe it is worth considering
why a certain configuration step should be considered. Like should a
kernel timer expire on the CPU that programmed it or on the
house-keeping CPU instead? That would be the knob for it.

> +
>  Kernel Debug Options
>  ====================
>  

Sebastian

^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2026-11-20 11:36 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-20  7:06 [PATCH v2] Documentation: real-time: Add boot parameters and task isolation section Zixin Wang
2026-11-20 11:35 ` Sebastian Andrzej Siewior

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox