* [PATCH v3] Documentation: real-time: Add boot parameters and task isolation section
@ 2026-11-20 15:28 Zixin Wang
2026-11-20 15:48 ` Sebastian Andrzej Siewior
0 siblings, 1 reply; 2+ messages in thread
From: Zixin Wang @ 2026-11-20 15:28 UTC (permalink / raw)
To: linux-doc
Cc: linux-rt-devel, bigeasy, clrkwllms, rostedt, corbet, frederic,
wagi, atomlin, linux-kernel, Zixin Wang
Document the boot parameters and runtime tunables that affect the
worst-case latency of a real-time system, alongside the existing
kernel configuration guidance.
Cover CPU isolation, nohz_full, RCU callback offloading, interrupt
affinity, and idle state limits, with references to the relevant
documentation.
Link: https://bugzilla.kernel.org/show_bug.cgi?id=220633
Link: https://github.com/elisa-tech/wg-aerospace/issues/222
Signed-off-by: Zixin Wang <w1355457260@gmail.com>
---
Thanks Sebastian for the review and for the pointer to the sched_dl
series.
v2 -> v3:
- Rework the isolation caveat: explicitly bound work, such as per-CPU
kthreads, bound workqueues, and interrupts, can still run on
isolated CPUs; unbound work is steered away.
- Clarify the flag responsibilities: domain isolation confines
unbound workqueues and kernel threads to the housekeeping CPUs,
while nohz/nohz_full handles the tick, timers, and RCU callback
offload. This was calibrated with local QEMU boots (domain-only,
nohz-only, and combined configurations).
- Describe rcu_nocbs by the CPU the callbacks run on, and mention
rcutree.use_softirq=false on PREEMPT_RT.
- Drop the managed_irq flag description while that area is being
reworked; no position is taken on where I/O completion interrupts
should be delivered.
- Remove the MSI-X example from irqaffinity and state that managed
interrupts are not affected.
- Qualify the C-state limit as x86/ACPI specific.
- Drop the kernel.sched_rt_runtime_us subsection; that documentation
is being reworked in the sched_dl series.
- Frame the section around the decisions an integrator has to make,
and spell out the timer placement choice: a migratable timer
expires on the CPU that armed it unless nohz/nohz_full migrates it
to a housekeeping CPU.
- Add when-to-use guidance for rcu_nocbs, irqaffinity and C-state
limits, and state that options should remain disabled when no
corresponding interference is measurable.
Testing:
QEMU/KVM with 8 vCPUs and a virtio-scsi device with two queues,
based on linux 7.3.5 with NO_HZ_FULL and RCU_NOCB_CPU enabled:
The tests that check per-CPU behavior were pinned to CPU 6 and the
affinity was verified from /proc/self/status. These measurements
verify the knob behavior; the guidance on when to use each option
comes from the referenced documentation.
- Domain-only and domain+nohz confine the unbound workqueue cpumask
to the housekeeping CPUs; nohz-only leaves it unrestricted.
- rcuo kthreads run on housekeeping CPUs when domain isolation is
active; rcu_nocbs without domain isolation leaves them
unrestricted.
- Timers and hrtimers created on a nohz-isolated CPU expire on
housekeeping CPUs; without nohz they expire on the isolated CPU.
- With nohz, the per-CPU tick handler did not run on the isolated
CPU while a pinned task was running; without nohz it did.
- isolcpus=managed_irq moves a virtio-scsi queue interrupt from an
isolated CPU to a housekeeping CPU.
- The fair server reports 50 ms of runtime per 1000 ms period and
sched_rt_runtime_us defaults to 1000000.
- A PREEMPT_RT build boots as PREEMPT_RT; rcutree.use_softirq is
false in that build (its module parameter is not exposed on RT).
processor.max_cstate and intel_idle.max_cstate could not be
exercised: QEMU exposes no ACPI _CST states, so no cpuidle states are
registered, and intel_idle is not built for this AMD test setup.
One open question about the idle state limits section: I could not
exercise processor.max_cstate or intel_idle.max_cstate locally. QEMU
exposes no ACPI _CST states, so the kernel registers no cpuidle states,
and intel_idle is not built for this AMD test host. Should the section
stay as documentation-only guidance, or would you prefer that I drop or
rework it (for example, by pointing to
Documentation/admin-guide/pm/cpuidle.rst)? I am happy to adjust it.
.../real-time/kernel-configuration.rst | 132 ++++++++++++++++++
1 file changed, 132 insertions(+)
diff --git a/Documentation/core-api/real-time/kernel-configuration.rst b/Documentation/core-api/real-time/kernel-configuration.rst
index d7f08e7b..0e28dd94 100644
--- a/Documentation/core-api/real-time/kernel-configuration.rst
+++ b/Documentation/core-api/real-time/kernel-configuration.rst
@@ -216,6 +216,138 @@ just by being enabled.
These tracers do incur measurable latency overhead even when tracing is not
currently active.
+Boot parameters and task isolation
+==================================
+
+Kernel configuration options alone do not determine a real-time system's
+worst-case latency. Integrators commonly rely on boot parameters and
+runtime tunables as well, to keep kernel housekeeping work away from
+latency-sensitive tasks. The parameters below complement the options
+described above. Each one is documented in detail in
+Documentation/admin-guide/kernel-parameters.rst; the notes here focus on
+the aspects that matter for real-time latency. Consider these options
+only when the workload suffers from the corresponding interference;
+leave them disabled otherwise, as each of them trades flexibility or
+power for a lower worst-case latency. The notes below follow the
+decisions an integrator has to make: whether to keep tasks off a CPU,
+whether to stop its tick, where RCU callbacks and interrupts should
+run, and whether deep idle states are acceptable. For the full CPU
+isolation model and its limitations, see
+Documentation/admin-guide/cpu-isolation.rst.
+
+.. caution::
+
+ CPU isolation is best effort. Work that is explicitly bound to an
+ isolated CPU, such as per-CPU kernel threads, bound workqueues, and
+ interrupts, may still run there unless it is steered away. Measure
+ the resulting latency with tools such as ``osnoise``, ``timerlat``,
+ and ``rtla`` rather than assuming that an isolated CPU is free of
+ kernel noise. See Documentation/timers/no_hz.rst and
+ Documentation/tools/rtla/index.rst.
+
+CPU isolation
+-------------
+
+``isolcpus=``
+ Format: ``[flag-list,]<cpu-list>``. Isolates the listed CPUs from the
+ disturbances selected by the flag list (default: ``domain``):
+
+ ``domain``
+ Remove the CPUs from the scheduler domains. This is the default
+ when no flag is given. Domain isolation cannot be undone at
+ runtime; use cpusets when the isolated CPU set must change
+ dynamically. Isolated CPUs remain usable through CPU affinity or
+ cpusets. Unbound workqueues and unbound kernel threads are
+ confined to the housekeeping CPUs as a side effect. Consider it
+ for CPUs dedicated to a latency-sensitive workload.
+
+ ``nohz``
+ Stop the tick when a single task runs and offload kernel noise,
+ including RCU callbacks. This is equivalent to ``nohz_full=``.
+ The residual 1 Hz tick is offloaded to workqueues. The global
+ workqueue ``cpumask`` (/sys/devices/virtual/workqueue/cpumask) can
+ further restrict where unbound work runs.
+
+ Domain isolation steers unbound workqueues and kernel threads away
+ from the isolated CPUs, while ``nohz`` (or ``nohz_full=``) moves the
+ tick, timers, and RCU callback processing off them; work explicitly
+ bound to an isolated CPU is not affected. A practical configuration
+ therefore combines ``isolcpus=`` with ``nohz_full=``, ``rcu_nocbs=``,
+ an interrupt affinity mask, and, on x86 systems, a C-state limit.
+
+ Timer placement follows the same choice. By default a migratable
+ timer expires on the CPU that armed it; with ``nohz`` (or
+ ``nohz_full=``), timers and hrtimers armed on the isolated CPUs are
+ migrated to housekeeping CPUs.
+
+``nohz_full=``
+ Format: ``<cpu-list>``. Requires ``CONFIG_NO_HZ_FULL=y``. Stops the
+ scheduler tick on the listed CPUs whenever possible. The boot CPU is
+ forced outside the range to maintain timekeeping, and the RCU
+ callbacks of the listed CPUs are offloaded as if they had been
+ passed to ``rcu_nocbs=``. This parameter takes precedence over
+ ``CONFIG_RCU_NOCB_CPU_DEFAULT_ALL``.
+
+ Tickless operation can increase the kernel-to-userspace transition
+ latency, so it is most useful for workloads that execute in
+ userspace for long periods with few syscalls. Periodic workloads
+ with short periods may prefer a consistent tick instead.
+
+RCU callback offloading
+-----------------------
+
+``rcu_nocbs[=cpu-list]``
+ In kernels built with ``CONFIG_RCU_NOCB_CPU=y``, offload RCU callback
+ invocation from the listed CPUs to the corresponding ``rcuo``
+ kthread, which can run on a housekeeping CPU. This keeps callbacks
+ from being invoked on the CPU that enqueued them. On a non-real-time
+ kernel this also moves invocation out of softirq context; with
+ PREEMPT_RT, callbacks already run in kthread context because
+ ``rcutree.use_softirq`` is false, and the offload only changes the
+ CPU they run on. Consider it when RCU callback processing causes
+ jitter on an isolated CPU. If omitted, no CPU is offloaded at boot;
+ the mode can still be toggled at runtime through cpusets.
+ ``nohz_full=`` implies the same offload for the CPUs it lists.
+
+``rcu_nocb_poll``
+ Make the ``rcuo`` kthreads poll for callbacks instead of requiring
+ the offloaded CPUs to wake them. This improves the real-time
+ response at the cost of higher power consumption, because the
+ kthreads wake up periodically to do the polling. Consider this
+ option only when wakeup latency matters more than power.
+
+Interrupt affinity
+------------------
+
+``irqaffinity=``
+ Format: ``<cpu-list>``. Sets the default affinity mask for
+ interrupts that do not have an explicit affinity, steering them to
+ the housekeeping CPUs. Managed interrupts are not affected by this
+ setting and cannot be redirected through ``/proc/irq/*``. Verify
+ the resulting placement in ``/proc/interrupts``. See
+ Documentation/core-api/irq/managed_irq.rst.
+
+Idle state limits
+-----------------
+
+``processor.max_cstate=``
+ ACPI platforms only. Limits the deepest C-state the processor may
+ enter. ``processor.max_cstate=1`` caps the processor at C1 and
+ avoids the entry and exit latency, cache flushing, and clock effects
+ of deeper states. A value of 9 overrides any DMI blacklist limit.
+
+``intel_idle.max_cstate=``
+ x86 platforms using ``intel_idle`` only. A value between 1 and 9
+ specifies the maximum C-state depth. A value of 0 disables
+ ``intel_idle`` and falls back to ``acpi_idle``; it does not disable
+ C-states.
+
+Deeper C-states save power but increase wakeup latency. Consider
+limiting them only when wakeup latency matters; otherwise leave the
+platform's power management unchanged. Choose the shallowest state
+that still meets the platform's power and thermal requirements, and
+validate the result with the tools listed above.
+
Kernel Debug Options
====================
--
2.54.0
^ permalink raw reply related [flat|nested] 2+ messages in thread* Re: [PATCH v3] Documentation: real-time: Add boot parameters and task isolation section
2026-11-20 15:28 [PATCH v3] Documentation: real-time: Add boot parameters and task isolation section Zixin Wang
@ 2026-11-20 15:48 ` Sebastian Andrzej Siewior
0 siblings, 0 replies; 2+ messages in thread
From: Sebastian Andrzej Siewior @ 2026-11-20 15:48 UTC (permalink / raw)
To: Zixin Wang
Cc: linux-doc, linux-rt-devel, clrkwllms, rostedt, corbet, frederic,
wagi, atomlin, linux-kernel
On 2026-11-20 23:28:36 [+0800], Zixin Wang wrote:
> One open question about the idle state limits section: I could not
> exercise processor.max_cstate or intel_idle.max_cstate locally. QEMU
> exposes no ACPI _CST states, so the kernel registers no cpuidle states,
> and intel_idle is not built for this AMD test host. Should the section
> stay as documentation-only guidance, or would you prefer that I drop or
> rework it (for example, by pointing to
> Documentation/admin-guide/pm/cpuidle.rst)? I am happy to adjust it.
processor.max_cstate should work on AMD unless it does not expose
certain things. But then it might be an embedded board, not having this?
Does cpupower report anything?
And I don't thing this is exposed on qemu since the guest goes only idle
and the host decides what to do.
The CONFIG_CPU_IDLE section already mentions processor.max_cstate=1 and
refers to cpupower among other points.
I don't know how you want to structure this information but parts of it
are already there. You could go on an add some settings for i915 for
instance. But then it feels like it should be within the DRM section
because it somehow belongs together.
And there is a CONFIG_NO_HZ/ CONFIG_NO_HZ_FULL section for instance.
It also says that the transition is a bit more expensive. This repeated
again later on.
Maybe it wouldn't be that bad if the config switches would have a
suggestion for the comandline argument that might be helpful/
considered. That way the information pieces stay together.
What do you think?
Sebastian
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2026-11-20 15:48 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-20 15:28 [PATCH v3] Documentation: real-time: Add boot parameters and task isolation section Zixin Wang
2026-11-20 15:48 ` Sebastian Andrzej Siewior
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox