Linux Documentation
 help / color / mirror / Atom feed
From: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
To: Zixin Wang <w1355457260@gmail.com>
Cc: linux-doc@vger.kernel.org, linux-rt-devel@lists.linux.dev,
	clrkwllms@kernel.org, rostedt@goodmis.org, corbet@lwn.net,
	linux-kernel@vger.kernel.org,
	Frederic Weisbecker <frederic@kernel.org>,
	Daniel Wagner <wagi@monom.org>,
	Aaron Tomlin <atomlin@atomlin.com>
Subject: Re: [PATCH v2] Documentation: real-time: Add boot parameters and task isolation section
Date: Fri, 20 Nov 2026 12:35:59 +0100	[thread overview]
Message-ID: <20261120113559.VCQRFh0b@linutronix.de> (raw)
In-Reply-To: <20261120070620.535693-1-w1355457260@gmail.com>

On 2026-11-20 15:06:20 [+0800], Zixin Wang wrote:
> --- a/Documentation/core-api/real-time/kernel-configuration.rst
> +++ b/Documentation/core-api/real-time/kernel-configuration.rst
> @@ -216,6 +216,147 @@ just by being enabled.
>  These tracers do incur measurable latency overhead even when tracing is not
>  currently active.
>  
> +Boot parameters and task isolation
> +==================================
> +
> +Kernel configuration options alone do not determine a real-time system's
> +worst-case latency.  Integrators commonly rely on boot parameters and
> +runtime tunables as well, to keep kernel housekeeping work away from
> +latency-sensitive tasks.  The parameters below complement the options
> +described above.  Each one is documented in detail in
> +Documentation/admin-guide/kernel-parameters.rst; the notes here focus on
> +the aspects that matter for real-time latency.  For the full CPU
> +isolation model and its limitations, see
> +Documentation/admin-guide/cpu-isolation.rst.
> +
> +.. caution::
> +
> +  CPU isolation is best effort.  Unbound workqueues, kernel threads,

An unbound workqueue is not restricted to a certain to CPU therefore it
will not run on an isolated CPU. The same is true for kernel threads as
long as they are not restricted to the isolated CPU.


> +  timers, and interrupts may still run on isolated CPUs unless they are
> +  explicitly steered away.  Measure the resulting latency with tools
> +  such as ``osnoise``, ``timerlat``, and ``rtla`` rather than assuming
> +  that an isolated CPU is free of kernel noise.  See
> +  Documentation/timers/no_hz.rst and Documentation/tools/rtla/index.rst.
> +
> +CPU isolation
> +-------------
> +
> +``isolcpus=``
> +  Format: ``[flag-list,]<cpu-list>``.  Isolates the listed CPUs from the
> +  disturbances selected by the flag list (default: ``domain``):
> +
> +  ``domain``
> +    Remove the CPUs from the scheduler domains.  This is the default
> +    when no flag is given.  Domain isolation cannot be undone at
> +    runtime; use cpusets when the isolated CPU set must change
> +    dynamically.  Isolated CPUs remain usable through CPU affinity or
> +    cpusets.
> +
> +  ``nohz``
> +    Stop the tick when a single task runs and offload kernel noise,
> +    including RCU callbacks.  This is equivalent to ``nohz_full=``.
> +    The residual 1 Hz tick is offloaded to workqueues, which must be
> +    affined to housekeeping CPUs through the global workqueue
> +    ``cpumask`` (/sys/devices/virtual/workqueue/cpumask); by default
> +    that workqueue may run on any CPU.

By default, yes, but that options changes it.

> +  ``managed_irq``
> +    Prevent managed interrupts from being targeted at isolated CPUs
> +    when their automatically computed affinity mask also contains
> +    housekeeping CPUs.  This is best effort and has no effect when the
> +    mask contains isolated CPUs only.  See
> +    Documentation/core-api/irq/managed_irq.rst.

It is currently worked on managed_irq. Apart from this it might be
debatable that if you perform I/O on an isolated CPU should you or
should you not receive a completion interrupt on the isolated CPU
initiating the I/O. Any statement from your side?

Frederic, does this fit here from your perspective?

> +
> +  Apart from the best-effort managed IRQ redirection, these flags do
> +  not stop kernel threads, timers, or interrupts that are already
> +  affined to the isolated CPUs.  A practical configuration therefore

timers. Unless a timer is explicitly set to target a certain CPU it will
avoid to fire on the isolated CPU even if "setup" on the isolated CPU.

> +  combines ``isolcpus=`` with ``nohz_full=``, ``rcu_nocbs=``, an
> +  interrupt affinity mask, and a C-state limit.

C-state is x86 specific.

> +``nohz_full=``
> +  Format: ``<cpu-list>``.  Requires ``CONFIG_NO_HZ_FULL=y``.  Stops the
> +  scheduler tick on the listed CPUs whenever possible.  The boot CPU is
> +  forced outside the range to maintain timekeeping, and the RCU
> +  callbacks of the listed CPUs are offloaded as if they had been
> +  passed to ``rcu_nocbs=``.  This parameter takes precedence over
> +  ``CONFIG_RCU_NOCB_CPU_DEFAULT_ALL``.
> +
> +  Tickless operation can increase the kernel-to-userspace transition
> +  latency, so it is most useful for workloads that execute in
> +  userspace for long periods with few syscalls.  Periodic workloads
> +  with short periods may prefer a consistent tick instead.
> +
> +RCU callback offloading
> +-----------------------
> +
> +``rcu_nocbs[=cpu-list]``
> +  In kernels built with ``CONFIG_RCU_NOCB_CPU=y``, move RCU callback
> +  invocation for the listed CPUs out of softirq context and into
> +  per-CPU ``rcuo`` kthreads.  This reduces OS jitter on the offloaded
> +  CPUs.  If the argument is omitted, no CPU is offloaded at boot, but
> +  the mode can be toggled at runtime through cpusets.  ``nohz_full=``
> +  implies the same offload for the CPUs it lists.

This sounds very familiar with the wording in kernel-parameters.txt.
With PREEMPT_RT enabled the callbacks are not pulled out of the softirq
context and this is an additional detail. There is
rcutree.use_softirq=false on PREEMPT_RT.
What it actually does is that the callbacks are no longer invoked on the
CPU that enqueued them but in the offloaded thread which can run on the
housekeeping CPU.

> +
> +``rcu_nocb_poll``
> +  Make the ``rcuo`` kthreads poll for callbacks instead of requiring
> +  the offloaded CPUs to wake them.  This improves the real-time
> +  response at the cost of higher power consumption, because the
> +  kthreads wake up periodically to do the polling.

At this point I am not sure if you explain (and copy original
description) or collect option which are advised/ recommended.

Maybe it is a list of options worth considering.

> +
> +Interrupt affinity
> +------------------
> +
> +``irqaffinity=``
> +  Format: ``<cpu-list>``.  Sets the default affinity mask for
> +  interrupts that do not have an explicit affinity.  Managed
> +  interrupts, such as MSI-X queue interrupts, derive their affinity

this "such as MSI-X queue interrupts" probably does not belong here. The
affinity of managed-interrupts can not be changed. How they are placed
is a different story.

> +  from the device queue mask instead and cannot be changed through
> +  ``/proc/irq/*``.  Combine ``irqaffinity=`` with the ``managed_irq``
> +  flag of ``isolcpus=`` to keep interrupts away from isolated CPUs
> +  where possible.  See Documentation/core-api/irq/managed_irq.rst.
> +
> +Idle state limits
> +-----------------
> +
> +``processor.max_cstate=``
> +  ACPI platforms only.  Limits the deepest C-state the processor may
> +  enter.  ``processor.max_cstate=1`` caps the processor at C1 and
> +  avoids the entry and exit latency, cache flushing, and clock effects
> +  of deeper states.  A value of 9 overrides any DMI blacklist limit.
> +
> +``intel_idle.max_cstate=``
> +  x86 platforms using ``intel_idle`` only.  A value between 1 and 9
> +  specifies the maximum C-state depth.  A value of 0 disables
> +  ``intel_idle`` and falls back to ``acpi_idle``; it does not disable
> +  C-states.
> +
> +Deeper C-states save power but increase wakeup latency.  Choose the
> +shallowest state that still meets the platform's power and thermal
> +requirements, and validate the result with the tools listed above.
> +
> +Real-time throttling
> +--------------------
> +
> +``kernel.sched_rt_runtime_us``
> +  Together with ``kernel.sched_rt_period_us``, this sysctl defines the
> +  total bandwidth available to SCHED_DEADLINE entities.  When
> +  ``CONFIG_RT_GROUP_SCHED`` is enabled, it also limits the bandwidth of
> +  the real-time scheduling groups.  The default runtime equals the
> +  period (1000000 us), so no bandwidth is subtracted from deadline
> +  tasks by default.

I would throw this away. First of sched_rt_runtime_us is not related to
throttling anymore and then I try to get rid of CONFIG_RT_GROUP_SCHED
(and this option). And I doubt anyone seriously uses this option.

> +  Without ``CONFIG_RT_GROUP_SCHED``, SCHED_FIFO and SCHED_RR tasks are
> +  no longer throttled through this interface.  Instead, a per-CPU fair
> +  deadline server keeps non-real-time tasks running by providing
> +  50 ms of runtime per 1000 ms period.  That reservation is not
> +  disabled by setting this sysctl to -1; the server parameters are
> +  exposed under /sys/kernel/debug/sched/fair_server/cpuX/.
> +
> +  A value of -1 removes the total deadline bandwidth limit, that is,
> +  runtime equals period.  Do this only after validating the workload,
> +  because runaway real-time tasks can then starve the rest of the
> +  system.  See Documentation/scheduler/sched-rt-group.rst for RT group
> +  scheduling details.

I do try to move it someplace else, see 
	https://lore.kernel.org/all/20261113-sched_dl-v3-0-a76ffa1cdeee@linutronix.de/

The basic information is that we have throttling in place and one needs
to disable it if it interferes.

Overall I don't mind having a collection of useful options worth
considering. I'm just not sure if the format of explaining the
parameters again is the right approach. Maybe it is worth considering
why a certain configuration step should be considered. Like should a
kernel timer expire on the CPU that programmed it or on the
house-keeping CPU instead? That would be the knob for it.

> +
>  Kernel Debug Options
>  ====================
>  

Sebastian

      reply	other threads:[~2026-11-20 11:36 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-20  7:06 [PATCH v2] Documentation: real-time: Add boot parameters and task isolation section Zixin Wang
2026-11-20 11:35 ` Sebastian Andrzej Siewior [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261120113559.VCQRFh0b@linutronix.de \
    --to=bigeasy@linutronix.de \
    --cc=atomlin@atomlin.com \
    --cc=clrkwllms@kernel.org \
    --cc=corbet@lwn.net \
    --cc=frederic@kernel.org \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-rt-devel@lists.linux.dev \
    --cc=rostedt@goodmis.org \
    --cc=w1355457260@gmail.com \
    --cc=wagi@monom.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox