From: Zixin Wang <w1355457260@gmail.com>
To: linux-doc@vger.kernel.org
Cc: linux-rt-devel@lists.linux.dev, bigeasy@linutronix.de,
clrkwllms@kernel.org, rostedt@goodmis.org, corbet@lwn.net,
frederic@kernel.org, wagi@monom.org, atomlin@atomlin.com,
linux-kernel@vger.kernel.org, Zixin Wang <w1355457260@gmail.com>
Subject: [PATCH v3] Documentation: real-time: Add boot parameters and task isolation section
Date: Fri, 20 Nov 2026 23:28:36 +0800 [thread overview]
Message-ID: <20261120152836.100638-1-w1355457260@gmail.com> (raw)
Document the boot parameters and runtime tunables that affect the
worst-case latency of a real-time system, alongside the existing
kernel configuration guidance.
Cover CPU isolation, nohz_full, RCU callback offloading, interrupt
affinity, and idle state limits, with references to the relevant
documentation.
Link: https://bugzilla.kernel.org/show_bug.cgi?id=220633
Link: https://github.com/elisa-tech/wg-aerospace/issues/222
Signed-off-by: Zixin Wang <w1355457260@gmail.com>
---
Thanks Sebastian for the review and for the pointer to the sched_dl
series.
v2 -> v3:
- Rework the isolation caveat: explicitly bound work, such as per-CPU
kthreads, bound workqueues, and interrupts, can still run on
isolated CPUs; unbound work is steered away.
- Clarify the flag responsibilities: domain isolation confines
unbound workqueues and kernel threads to the housekeeping CPUs,
while nohz/nohz_full handles the tick, timers, and RCU callback
offload. This was calibrated with local QEMU boots (domain-only,
nohz-only, and combined configurations).
- Describe rcu_nocbs by the CPU the callbacks run on, and mention
rcutree.use_softirq=false on PREEMPT_RT.
- Drop the managed_irq flag description while that area is being
reworked; no position is taken on where I/O completion interrupts
should be delivered.
- Remove the MSI-X example from irqaffinity and state that managed
interrupts are not affected.
- Qualify the C-state limit as x86/ACPI specific.
- Drop the kernel.sched_rt_runtime_us subsection; that documentation
is being reworked in the sched_dl series.
- Frame the section around the decisions an integrator has to make,
and spell out the timer placement choice: a migratable timer
expires on the CPU that armed it unless nohz/nohz_full migrates it
to a housekeeping CPU.
- Add when-to-use guidance for rcu_nocbs, irqaffinity and C-state
limits, and state that options should remain disabled when no
corresponding interference is measurable.
Testing:
QEMU/KVM with 8 vCPUs and a virtio-scsi device with two queues,
based on linux 7.3.5 with NO_HZ_FULL and RCU_NOCB_CPU enabled:
The tests that check per-CPU behavior were pinned to CPU 6 and the
affinity was verified from /proc/self/status. These measurements
verify the knob behavior; the guidance on when to use each option
comes from the referenced documentation.
- Domain-only and domain+nohz confine the unbound workqueue cpumask
to the housekeeping CPUs; nohz-only leaves it unrestricted.
- rcuo kthreads run on housekeeping CPUs when domain isolation is
active; rcu_nocbs without domain isolation leaves them
unrestricted.
- Timers and hrtimers created on a nohz-isolated CPU expire on
housekeeping CPUs; without nohz they expire on the isolated CPU.
- With nohz, the per-CPU tick handler did not run on the isolated
CPU while a pinned task was running; without nohz it did.
- isolcpus=managed_irq moves a virtio-scsi queue interrupt from an
isolated CPU to a housekeeping CPU.
- The fair server reports 50 ms of runtime per 1000 ms period and
sched_rt_runtime_us defaults to 1000000.
- A PREEMPT_RT build boots as PREEMPT_RT; rcutree.use_softirq is
false in that build (its module parameter is not exposed on RT).
processor.max_cstate and intel_idle.max_cstate could not be
exercised: QEMU exposes no ACPI _CST states, so no cpuidle states are
registered, and intel_idle is not built for this AMD test setup.
One open question about the idle state limits section: I could not
exercise processor.max_cstate or intel_idle.max_cstate locally. QEMU
exposes no ACPI _CST states, so the kernel registers no cpuidle states,
and intel_idle is not built for this AMD test host. Should the section
stay as documentation-only guidance, or would you prefer that I drop or
rework it (for example, by pointing to
Documentation/admin-guide/pm/cpuidle.rst)? I am happy to adjust it.
.../real-time/kernel-configuration.rst | 132 ++++++++++++++++++
1 file changed, 132 insertions(+)
diff --git a/Documentation/core-api/real-time/kernel-configuration.rst b/Documentation/core-api/real-time/kernel-configuration.rst
index d7f08e7b..0e28dd94 100644
--- a/Documentation/core-api/real-time/kernel-configuration.rst
+++ b/Documentation/core-api/real-time/kernel-configuration.rst
@@ -216,6 +216,138 @@ just by being enabled.
These tracers do incur measurable latency overhead even when tracing is not
currently active.
+Boot parameters and task isolation
+==================================
+
+Kernel configuration options alone do not determine a real-time system's
+worst-case latency. Integrators commonly rely on boot parameters and
+runtime tunables as well, to keep kernel housekeeping work away from
+latency-sensitive tasks. The parameters below complement the options
+described above. Each one is documented in detail in
+Documentation/admin-guide/kernel-parameters.rst; the notes here focus on
+the aspects that matter for real-time latency. Consider these options
+only when the workload suffers from the corresponding interference;
+leave them disabled otherwise, as each of them trades flexibility or
+power for a lower worst-case latency. The notes below follow the
+decisions an integrator has to make: whether to keep tasks off a CPU,
+whether to stop its tick, where RCU callbacks and interrupts should
+run, and whether deep idle states are acceptable. For the full CPU
+isolation model and its limitations, see
+Documentation/admin-guide/cpu-isolation.rst.
+
+.. caution::
+
+ CPU isolation is best effort. Work that is explicitly bound to an
+ isolated CPU, such as per-CPU kernel threads, bound workqueues, and
+ interrupts, may still run there unless it is steered away. Measure
+ the resulting latency with tools such as ``osnoise``, ``timerlat``,
+ and ``rtla`` rather than assuming that an isolated CPU is free of
+ kernel noise. See Documentation/timers/no_hz.rst and
+ Documentation/tools/rtla/index.rst.
+
+CPU isolation
+-------------
+
+``isolcpus=``
+ Format: ``[flag-list,]<cpu-list>``. Isolates the listed CPUs from the
+ disturbances selected by the flag list (default: ``domain``):
+
+ ``domain``
+ Remove the CPUs from the scheduler domains. This is the default
+ when no flag is given. Domain isolation cannot be undone at
+ runtime; use cpusets when the isolated CPU set must change
+ dynamically. Isolated CPUs remain usable through CPU affinity or
+ cpusets. Unbound workqueues and unbound kernel threads are
+ confined to the housekeeping CPUs as a side effect. Consider it
+ for CPUs dedicated to a latency-sensitive workload.
+
+ ``nohz``
+ Stop the tick when a single task runs and offload kernel noise,
+ including RCU callbacks. This is equivalent to ``nohz_full=``.
+ The residual 1 Hz tick is offloaded to workqueues. The global
+ workqueue ``cpumask`` (/sys/devices/virtual/workqueue/cpumask) can
+ further restrict where unbound work runs.
+
+ Domain isolation steers unbound workqueues and kernel threads away
+ from the isolated CPUs, while ``nohz`` (or ``nohz_full=``) moves the
+ tick, timers, and RCU callback processing off them; work explicitly
+ bound to an isolated CPU is not affected. A practical configuration
+ therefore combines ``isolcpus=`` with ``nohz_full=``, ``rcu_nocbs=``,
+ an interrupt affinity mask, and, on x86 systems, a C-state limit.
+
+ Timer placement follows the same choice. By default a migratable
+ timer expires on the CPU that armed it; with ``nohz`` (or
+ ``nohz_full=``), timers and hrtimers armed on the isolated CPUs are
+ migrated to housekeeping CPUs.
+
+``nohz_full=``
+ Format: ``<cpu-list>``. Requires ``CONFIG_NO_HZ_FULL=y``. Stops the
+ scheduler tick on the listed CPUs whenever possible. The boot CPU is
+ forced outside the range to maintain timekeeping, and the RCU
+ callbacks of the listed CPUs are offloaded as if they had been
+ passed to ``rcu_nocbs=``. This parameter takes precedence over
+ ``CONFIG_RCU_NOCB_CPU_DEFAULT_ALL``.
+
+ Tickless operation can increase the kernel-to-userspace transition
+ latency, so it is most useful for workloads that execute in
+ userspace for long periods with few syscalls. Periodic workloads
+ with short periods may prefer a consistent tick instead.
+
+RCU callback offloading
+-----------------------
+
+``rcu_nocbs[=cpu-list]``
+ In kernels built with ``CONFIG_RCU_NOCB_CPU=y``, offload RCU callback
+ invocation from the listed CPUs to the corresponding ``rcuo``
+ kthread, which can run on a housekeeping CPU. This keeps callbacks
+ from being invoked on the CPU that enqueued them. On a non-real-time
+ kernel this also moves invocation out of softirq context; with
+ PREEMPT_RT, callbacks already run in kthread context because
+ ``rcutree.use_softirq`` is false, and the offload only changes the
+ CPU they run on. Consider it when RCU callback processing causes
+ jitter on an isolated CPU. If omitted, no CPU is offloaded at boot;
+ the mode can still be toggled at runtime through cpusets.
+ ``nohz_full=`` implies the same offload for the CPUs it lists.
+
+``rcu_nocb_poll``
+ Make the ``rcuo`` kthreads poll for callbacks instead of requiring
+ the offloaded CPUs to wake them. This improves the real-time
+ response at the cost of higher power consumption, because the
+ kthreads wake up periodically to do the polling. Consider this
+ option only when wakeup latency matters more than power.
+
+Interrupt affinity
+------------------
+
+``irqaffinity=``
+ Format: ``<cpu-list>``. Sets the default affinity mask for
+ interrupts that do not have an explicit affinity, steering them to
+ the housekeeping CPUs. Managed interrupts are not affected by this
+ setting and cannot be redirected through ``/proc/irq/*``. Verify
+ the resulting placement in ``/proc/interrupts``. See
+ Documentation/core-api/irq/managed_irq.rst.
+
+Idle state limits
+-----------------
+
+``processor.max_cstate=``
+ ACPI platforms only. Limits the deepest C-state the processor may
+ enter. ``processor.max_cstate=1`` caps the processor at C1 and
+ avoids the entry and exit latency, cache flushing, and clock effects
+ of deeper states. A value of 9 overrides any DMI blacklist limit.
+
+``intel_idle.max_cstate=``
+ x86 platforms using ``intel_idle`` only. A value between 1 and 9
+ specifies the maximum C-state depth. A value of 0 disables
+ ``intel_idle`` and falls back to ``acpi_idle``; it does not disable
+ C-states.
+
+Deeper C-states save power but increase wakeup latency. Consider
+limiting them only when wakeup latency matters; otherwise leave the
+platform's power management unchanged. Choose the shallowest state
+that still meets the platform's power and thermal requirements, and
+validate the result with the tools listed above.
+
Kernel Debug Options
====================
--
2.54.0
next reply other threads:[~2026-11-20 15:29 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-20 15:28 Zixin Wang [this message]
2026-11-20 15:48 ` [PATCH v3] Documentation: real-time: Add boot parameters and task isolation section Sebastian Andrzej Siewior
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261120152836.100638-1-w1355457260@gmail.com \
--to=w1355457260@gmail.com \
--cc=atomlin@atomlin.com \
--cc=bigeasy@linutronix.de \
--cc=clrkwllms@kernel.org \
--cc=corbet@lwn.net \
--cc=frederic@kernel.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rt-devel@lists.linux.dev \
--cc=rostedt@goodmis.org \
--cc=wagi@monom.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox