Linux Documentation
 help / color / mirror / Atom feed
* [PATCH v3] Documentation: real-time: Add boot parameters and task isolation section
@ 2026-11-20 15:28 Zixin Wang
  2026-11-20 15:48 ` Sebastian Andrzej Siewior
  0 siblings, 1 reply; 2+ messages in thread
From: Zixin Wang @ 2026-11-20 15:28 UTC (permalink / raw)
  To: linux-doc
  Cc: linux-rt-devel, bigeasy, clrkwllms, rostedt, corbet, frederic,
	wagi, atomlin, linux-kernel, Zixin Wang

Document the boot parameters and runtime tunables that affect the
worst-case latency of a real-time system, alongside the existing
kernel configuration guidance.

Cover CPU isolation, nohz_full, RCU callback offloading, interrupt
affinity, and idle state limits, with references to the relevant
documentation.

Link: https://bugzilla.kernel.org/show_bug.cgi?id=220633
Link: https://github.com/elisa-tech/wg-aerospace/issues/222
Signed-off-by: Zixin Wang <w1355457260@gmail.com>
---
Thanks Sebastian for the review and for the pointer to the sched_dl
series.

v2 -> v3:
  - Rework the isolation caveat: explicitly bound work, such as per-CPU
    kthreads, bound workqueues, and interrupts, can still run on
    isolated CPUs; unbound work is steered away.
  - Clarify the flag responsibilities: domain isolation confines
    unbound workqueues and kernel threads to the housekeeping CPUs,
    while nohz/nohz_full handles the tick, timers, and RCU callback
    offload.  This was calibrated with local QEMU boots (domain-only,
    nohz-only, and combined configurations).
  - Describe rcu_nocbs by the CPU the callbacks run on, and mention
    rcutree.use_softirq=false on PREEMPT_RT.
  - Drop the managed_irq flag description while that area is being
    reworked; no position is taken on where I/O completion interrupts
    should be delivered.
  - Remove the MSI-X example from irqaffinity and state that managed
    interrupts are not affected.
  - Qualify the C-state limit as x86/ACPI specific.
  - Drop the kernel.sched_rt_runtime_us subsection; that documentation
    is being reworked in the sched_dl series.
  - Frame the section around the decisions an integrator has to make,
    and spell out the timer placement choice: a migratable timer
    expires on the CPU that armed it unless nohz/nohz_full migrates it
    to a housekeeping CPU.
  - Add when-to-use guidance for rcu_nocbs, irqaffinity and C-state
    limits, and state that options should remain disabled when no
    corresponding interference is measurable.

Testing:
  QEMU/KVM with 8 vCPUs and a virtio-scsi device with two queues,
  based on linux 7.3.5 with NO_HZ_FULL and RCU_NOCB_CPU enabled:

  The tests that check per-CPU behavior were pinned to CPU 6 and the
  affinity was verified from /proc/self/status.  These measurements
  verify the knob behavior; the guidance on when to use each option
  comes from the referenced documentation.

  - Domain-only and domain+nohz confine the unbound workqueue cpumask
    to the housekeeping CPUs; nohz-only leaves it unrestricted.
  - rcuo kthreads run on housekeeping CPUs when domain isolation is
    active; rcu_nocbs without domain isolation leaves them
    unrestricted.
  - Timers and hrtimers created on a nohz-isolated CPU expire on
    housekeeping CPUs; without nohz they expire on the isolated CPU.
  - With nohz, the per-CPU tick handler did not run on the isolated
    CPU while a pinned task was running; without nohz it did.
  - isolcpus=managed_irq moves a virtio-scsi queue interrupt from an
    isolated CPU to a housekeeping CPU.
  - The fair server reports 50 ms of runtime per 1000 ms period and
    sched_rt_runtime_us defaults to 1000000.
  - A PREEMPT_RT build boots as PREEMPT_RT; rcutree.use_softirq is
    false in that build (its module parameter is not exposed on RT).

  processor.max_cstate and intel_idle.max_cstate could not be
  exercised: QEMU exposes no ACPI _CST states, so no cpuidle states are
  registered, and intel_idle is not built for this AMD test setup.

One open question about the idle state limits section: I could not
exercise processor.max_cstate or intel_idle.max_cstate locally.  QEMU
exposes no ACPI _CST states, so the kernel registers no cpuidle states,
and intel_idle is not built for this AMD test host.  Should the section
stay as documentation-only guidance, or would you prefer that I drop or
rework it (for example, by pointing to
Documentation/admin-guide/pm/cpuidle.rst)?  I am happy to adjust it.

 .../real-time/kernel-configuration.rst        | 132 ++++++++++++++++++
 1 file changed, 132 insertions(+)

diff --git a/Documentation/core-api/real-time/kernel-configuration.rst b/Documentation/core-api/real-time/kernel-configuration.rst
index d7f08e7b..0e28dd94 100644
--- a/Documentation/core-api/real-time/kernel-configuration.rst
+++ b/Documentation/core-api/real-time/kernel-configuration.rst
@@ -216,6 +216,138 @@ just by being enabled.
 These tracers do incur measurable latency overhead even when tracing is not
 currently active.
 
+Boot parameters and task isolation
+==================================
+
+Kernel configuration options alone do not determine a real-time system's
+worst-case latency.  Integrators commonly rely on boot parameters and
+runtime tunables as well, to keep kernel housekeeping work away from
+latency-sensitive tasks.  The parameters below complement the options
+described above.  Each one is documented in detail in
+Documentation/admin-guide/kernel-parameters.rst; the notes here focus on
+the aspects that matter for real-time latency.  Consider these options
+only when the workload suffers from the corresponding interference;
+leave them disabled otherwise, as each of them trades flexibility or
+power for a lower worst-case latency.  The notes below follow the
+decisions an integrator has to make: whether to keep tasks off a CPU,
+whether to stop its tick, where RCU callbacks and interrupts should
+run, and whether deep idle states are acceptable.  For the full CPU
+isolation model and its limitations, see
+Documentation/admin-guide/cpu-isolation.rst.
+
+.. caution::
+
+  CPU isolation is best effort.  Work that is explicitly bound to an
+  isolated CPU, such as per-CPU kernel threads, bound workqueues, and
+  interrupts, may still run there unless it is steered away.  Measure
+  the resulting latency with tools such as ``osnoise``, ``timerlat``,
+  and ``rtla`` rather than assuming that an isolated CPU is free of
+  kernel noise.  See Documentation/timers/no_hz.rst and
+  Documentation/tools/rtla/index.rst.
+
+CPU isolation
+-------------
+
+``isolcpus=``
+  Format: ``[flag-list,]<cpu-list>``.  Isolates the listed CPUs from the
+  disturbances selected by the flag list (default: ``domain``):
+
+  ``domain``
+    Remove the CPUs from the scheduler domains.  This is the default
+    when no flag is given.  Domain isolation cannot be undone at
+    runtime; use cpusets when the isolated CPU set must change
+    dynamically.  Isolated CPUs remain usable through CPU affinity or
+    cpusets.  Unbound workqueues and unbound kernel threads are
+    confined to the housekeeping CPUs as a side effect.  Consider it
+    for CPUs dedicated to a latency-sensitive workload.
+
+  ``nohz``
+    Stop the tick when a single task runs and offload kernel noise,
+    including RCU callbacks.  This is equivalent to ``nohz_full=``.
+    The residual 1 Hz tick is offloaded to workqueues.  The global
+    workqueue ``cpumask`` (/sys/devices/virtual/workqueue/cpumask) can
+    further restrict where unbound work runs.
+
+  Domain isolation steers unbound workqueues and kernel threads away
+  from the isolated CPUs, while ``nohz`` (or ``nohz_full=``) moves the
+  tick, timers, and RCU callback processing off them; work explicitly
+  bound to an isolated CPU is not affected.  A practical configuration
+  therefore combines ``isolcpus=`` with ``nohz_full=``, ``rcu_nocbs=``,
+  an interrupt affinity mask, and, on x86 systems, a C-state limit.
+
+  Timer placement follows the same choice.  By default a migratable
+  timer expires on the CPU that armed it; with ``nohz`` (or
+  ``nohz_full=``), timers and hrtimers armed on the isolated CPUs are
+  migrated to housekeeping CPUs.
+
+``nohz_full=``
+  Format: ``<cpu-list>``.  Requires ``CONFIG_NO_HZ_FULL=y``.  Stops the
+  scheduler tick on the listed CPUs whenever possible.  The boot CPU is
+  forced outside the range to maintain timekeeping, and the RCU
+  callbacks of the listed CPUs are offloaded as if they had been
+  passed to ``rcu_nocbs=``.  This parameter takes precedence over
+  ``CONFIG_RCU_NOCB_CPU_DEFAULT_ALL``.
+
+  Tickless operation can increase the kernel-to-userspace transition
+  latency, so it is most useful for workloads that execute in
+  userspace for long periods with few syscalls.  Periodic workloads
+  with short periods may prefer a consistent tick instead.
+
+RCU callback offloading
+-----------------------
+
+``rcu_nocbs[=cpu-list]``
+  In kernels built with ``CONFIG_RCU_NOCB_CPU=y``, offload RCU callback
+  invocation from the listed CPUs to the corresponding ``rcuo``
+  kthread, which can run on a housekeeping CPU.  This keeps callbacks
+  from being invoked on the CPU that enqueued them.  On a non-real-time
+  kernel this also moves invocation out of softirq context; with
+  PREEMPT_RT, callbacks already run in kthread context because
+  ``rcutree.use_softirq`` is false, and the offload only changes the
+  CPU they run on.  Consider it when RCU callback processing causes
+  jitter on an isolated CPU.  If omitted, no CPU is offloaded at boot;
+  the mode can still be toggled at runtime through cpusets.
+  ``nohz_full=`` implies the same offload for the CPUs it lists.
+
+``rcu_nocb_poll``
+  Make the ``rcuo`` kthreads poll for callbacks instead of requiring
+  the offloaded CPUs to wake them.  This improves the real-time
+  response at the cost of higher power consumption, because the
+  kthreads wake up periodically to do the polling.  Consider this
+  option only when wakeup latency matters more than power.
+
+Interrupt affinity
+------------------
+
+``irqaffinity=``
+  Format: ``<cpu-list>``.  Sets the default affinity mask for
+  interrupts that do not have an explicit affinity, steering them to
+  the housekeeping CPUs.  Managed interrupts are not affected by this
+  setting and cannot be redirected through ``/proc/irq/*``.  Verify
+  the resulting placement in ``/proc/interrupts``.  See
+  Documentation/core-api/irq/managed_irq.rst.
+
+Idle state limits
+-----------------
+
+``processor.max_cstate=``
+  ACPI platforms only.  Limits the deepest C-state the processor may
+  enter.  ``processor.max_cstate=1`` caps the processor at C1 and
+  avoids the entry and exit latency, cache flushing, and clock effects
+  of deeper states.  A value of 9 overrides any DMI blacklist limit.
+
+``intel_idle.max_cstate=``
+  x86 platforms using ``intel_idle`` only.  A value between 1 and 9
+  specifies the maximum C-state depth.  A value of 0 disables
+  ``intel_idle`` and falls back to ``acpi_idle``; it does not disable
+  C-states.
+
+Deeper C-states save power but increase wakeup latency.  Consider
+limiting them only when wakeup latency matters; otherwise leave the
+platform's power management unchanged.  Choose the shallowest state
+that still meets the platform's power and thermal requirements, and
+validate the result with the tools listed above.
+
 Kernel Debug Options
 ====================
 
-- 
2.54.0


^ permalink raw reply related	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2026-11-20 15:48 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-20 15:28 [PATCH v3] Documentation: real-time: Add boot parameters and task isolation section Zixin Wang
2026-11-20 15:48 ` Sebastian Andrzej Siewior

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox