All of lore.kernel.org
 help / color / mirror / Atom feed
From: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
To: Zixin Wang <w1355457260@gmail.com>
Cc: linux-doc@vger.kernel.org, linux-rt-devel@lists.linux.dev,
	clrkwllms@kernel.org, rostedt@goodmis.org, corbet@lwn.net,
	linux-kernel@vger.kernel.org,
	Frederic Weisbecker <frederic@kernel.org>,
	Daniel Wagner <wagi@monom.org>,
	Aaron Tomlin <atomlin@atomlin.com>
Subject: Re: [PATCH v2] Documentation: real-time: Add boot parameters and task isolation section
Date: Fri, 20 Nov 2026 12:35:59 +0100	[thread overview]
Message-ID: <20261120113559.VCQRFh0b@linutronix.de> (raw)
In-Reply-To: <20261120070620.535693-1-w1355457260@gmail.com>

On 2026-11-20 15:06:20 [+0800], Zixin Wang wrote:
> --- a/Documentation/core-api/real-time/kernel-configuration.rst
> +++ b/Documentation/core-api/real-time/kernel-configuration.rst
> @@ -216,6 +216,147 @@ just by being enabled.
>  These tracers do incur measurable latency overhead even when tracing is not
>  currently active.
>  
> +Boot parameters and task isolation
> +==================================
> +
> +Kernel configuration options alone do not determine a real-time system's
> +worst-case latency.  Integrators commonly rely on boot parameters and
> +runtime tunables as well, to keep kernel housekeeping work away from
> +latency-sensitive tasks.  The parameters below complement the options
> +described above.  Each one is documented in detail in
> +Documentation/admin-guide/kernel-parameters.rst; the notes here focus on
> +the aspects that matter for real-time latency.  For the full CPU
> +isolation model and its limitations, see
> +Documentation/admin-guide/cpu-isolation.rst.
> +
> +.. caution::
> +
> +  CPU isolation is best effort.  Unbound workqueues, kernel threads,

An unbound workqueue is not restricted to a certain to CPU therefore it
will not run on an isolated CPU. The same is true for kernel threads as
long as they are not restricted to the isolated CPU.


> +  timers, and interrupts may still run on isolated CPUs unless they are
> +  explicitly steered away.  Measure the resulting latency with tools
> +  such as ``osnoise``, ``timerlat``, and ``rtla`` rather than assuming
> +  that an isolated CPU is free of kernel noise.  See
> +  Documentation/timers/no_hz.rst and Documentation/tools/rtla/index.rst.
> +
> +CPU isolation
> +-------------
> +
> +``isolcpus=``
> +  Format: ``[flag-list,]<cpu-list>``.  Isolates the listed CPUs from the
> +  disturbances selected by the flag list (default: ``domain``):
> +
> +  ``domain``
> +    Remove the CPUs from the scheduler domains.  This is the default
> +    when no flag is given.  Domain isolation cannot be undone at
> +    runtime; use cpusets when the isolated CPU set must change
> +    dynamically.  Isolated CPUs remain usable through CPU affinity or
> +    cpusets.
> +
> +  ``nohz``
> +    Stop the tick when a single task runs and offload kernel noise,
> +    including RCU callbacks.  This is equivalent to ``nohz_full=``.
> +    The residual 1 Hz tick is offloaded to workqueues, which must be
> +    affined to housekeeping CPUs through the global workqueue
> +    ``cpumask`` (/sys/devices/virtual/workqueue/cpumask); by default
> +    that workqueue may run on any CPU.

By default, yes, but that options changes it.

> +  ``managed_irq``
> +    Prevent managed interrupts from being targeted at isolated CPUs
> +    when their automatically computed affinity mask also contains
> +    housekeeping CPUs.  This is best effort and has no effect when the
> +    mask contains isolated CPUs only.  See
> +    Documentation/core-api/irq/managed_irq.rst.

It is currently worked on managed_irq. Apart from this it might be
debatable that if you perform I/O on an isolated CPU should you or
should you not receive a completion interrupt on the isolated CPU
initiating the I/O. Any statement from your side?

Frederic, does this fit here from your perspective?

> +
> +  Apart from the best-effort managed IRQ redirection, these flags do
> +  not stop kernel threads, timers, or interrupts that are already
> +  affined to the isolated CPUs.  A practical configuration therefore

timers. Unless a timer is explicitly set to target a certain CPU it will
avoid to fire on the isolated CPU even if "setup" on the isolated CPU.

> +  combines ``isolcpus=`` with ``nohz_full=``, ``rcu_nocbs=``, an
> +  interrupt affinity mask, and a C-state limit.

C-state is x86 specific.

> +``nohz_full=``
> +  Format: ``<cpu-list>``.  Requires ``CONFIG_NO_HZ_FULL=y``.  Stops the
> +  scheduler tick on the listed CPUs whenever possible.  The boot CPU is
> +  forced outside the range to maintain timekeeping, and the RCU
> +  callbacks of the listed CPUs are offloaded as if they had been
> +  passed to ``rcu_nocbs=``.  This parameter takes precedence over
> +  ``CONFIG_RCU_NOCB_CPU_DEFAULT_ALL``.
> +
> +  Tickless operation can increase the kernel-to-userspace transition
> +  latency, so it is most useful for workloads that execute in
> +  userspace for long periods with few syscalls.  Periodic workloads
> +  with short periods may prefer a consistent tick instead.
> +
> +RCU callback offloading
> +-----------------------
> +
> +``rcu_nocbs[=cpu-list]``
> +  In kernels built with ``CONFIG_RCU_NOCB_CPU=y``, move RCU callback
> +  invocation for the listed CPUs out of softirq context and into
> +  per-CPU ``rcuo`` kthreads.  This reduces OS jitter on the offloaded
> +  CPUs.  If the argument is omitted, no CPU is offloaded at boot, but
> +  the mode can be toggled at runtime through cpusets.  ``nohz_full=``
> +  implies the same offload for the CPUs it lists.

This sounds very familiar with the wording in kernel-parameters.txt.
With PREEMPT_RT enabled the callbacks are not pulled out of the softirq
context and this is an additional detail. There is
rcutree.use_softirq=false on PREEMPT_RT.
What it actually does is that the callbacks are no longer invoked on the
CPU that enqueued them but in the offloaded thread which can run on the
housekeeping CPU.

> +
> +``rcu_nocb_poll``
> +  Make the ``rcuo`` kthreads poll for callbacks instead of requiring
> +  the offloaded CPUs to wake them.  This improves the real-time
> +  response at the cost of higher power consumption, because the
> +  kthreads wake up periodically to do the polling.

At this point I am not sure if you explain (and copy original
description) or collect option which are advised/ recommended.

Maybe it is a list of options worth considering.

> +
> +Interrupt affinity
> +------------------
> +
> +``irqaffinity=``
> +  Format: ``<cpu-list>``.  Sets the default affinity mask for
> +  interrupts that do not have an explicit affinity.  Managed
> +  interrupts, such as MSI-X queue interrupts, derive their affinity

this "such as MSI-X queue interrupts" probably does not belong here. The
affinity of managed-interrupts can not be changed. How they are placed
is a different story.

> +  from the device queue mask instead and cannot be changed through
> +  ``/proc/irq/*``.  Combine ``irqaffinity=`` with the ``managed_irq``
> +  flag of ``isolcpus=`` to keep interrupts away from isolated CPUs
> +  where possible.  See Documentation/core-api/irq/managed_irq.rst.
> +
> +Idle state limits
> +-----------------
> +
> +``processor.max_cstate=``
> +  ACPI platforms only.  Limits the deepest C-state the processor may
> +  enter.  ``processor.max_cstate=1`` caps the processor at C1 and
> +  avoids the entry and exit latency, cache flushing, and clock effects
> +  of deeper states.  A value of 9 overrides any DMI blacklist limit.
> +
> +``intel_idle.max_cstate=``
> +  x86 platforms using ``intel_idle`` only.  A value between 1 and 9
> +  specifies the maximum C-state depth.  A value of 0 disables
> +  ``intel_idle`` and falls back to ``acpi_idle``; it does not disable
> +  C-states.
> +
> +Deeper C-states save power but increase wakeup latency.  Choose the
> +shallowest state that still meets the platform's power and thermal
> +requirements, and validate the result with the tools listed above.
> +
> +Real-time throttling
> +--------------------
> +
> +``kernel.sched_rt_runtime_us``
> +  Together with ``kernel.sched_rt_period_us``, this sysctl defines the
> +  total bandwidth available to SCHED_DEADLINE entities.  When
> +  ``CONFIG_RT_GROUP_SCHED`` is enabled, it also limits the bandwidth of
> +  the real-time scheduling groups.  The default runtime equals the
> +  period (1000000 us), so no bandwidth is subtracted from deadline
> +  tasks by default.

I would throw this away. First of sched_rt_runtime_us is not related to
throttling anymore and then I try to get rid of CONFIG_RT_GROUP_SCHED
(and this option). And I doubt anyone seriously uses this option.

> +  Without ``CONFIG_RT_GROUP_SCHED``, SCHED_FIFO and SCHED_RR tasks are
> +  no longer throttled through this interface.  Instead, a per-CPU fair
> +  deadline server keeps non-real-time tasks running by providing
> +  50 ms of runtime per 1000 ms period.  That reservation is not
> +  disabled by setting this sysctl to -1; the server parameters are
> +  exposed under /sys/kernel/debug/sched/fair_server/cpuX/.
> +
> +  A value of -1 removes the total deadline bandwidth limit, that is,
> +  runtime equals period.  Do this only after validating the workload,
> +  because runaway real-time tasks can then starve the rest of the
> +  system.  See Documentation/scheduler/sched-rt-group.rst for RT group
> +  scheduling details.

I do try to move it someplace else, see 
	https://lore.kernel.org/all/20261113-sched_dl-v3-0-a76ffa1cdeee@linutronix.de/

The basic information is that we have throttling in place and one needs
to disable it if it interferes.

Overall I don't mind having a collection of useful options worth
considering. I'm just not sure if the format of explaining the
parameters again is the right approach. Maybe it is worth considering
why a certain configuration step should be considered. Like should a
kernel timer expire on the CPU that programmed it or on the
house-keeping CPU instead? That would be the knob for it.

> +
>  Kernel Debug Options
>  ====================
>  

Sebastian

      reply	other threads:[~2026-11-20 11:36 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-20  7:06 [PATCH v2] Documentation: real-time: Add boot parameters and task isolation section Zixin Wang
2026-11-20 11:35 ` Sebastian Andrzej Siewior [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261120113559.VCQRFh0b@linutronix.de \
    --to=bigeasy@linutronix.de \
    --cc=atomlin@atomlin.com \
    --cc=clrkwllms@kernel.org \
    --cc=corbet@lwn.net \
    --cc=frederic@kernel.org \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-rt-devel@lists.linux.dev \
    --cc=rostedt@goodmis.org \
    --cc=w1355457260@gmail.com \
    --cc=wagi@monom.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.