[PATCH v5 0/1] Documentation: Add real-time kernel configuration guide

[PATCH v5 0/1] Documentation: Add real-time kernel configuration guide

[Ahmed S. Darwish]

Hi,

There is a no "one size fits all" solution for configuring a PREEMPT_RT
kernel.

Intorduce a PREEMPT_RT kernel configuration guide to better help system
developers and integrators.

Changelog v5
------------

Handle Sebastian remarks:

- Reference other documents as Jonathan prefers; i.e., no :doc: tags.
- Coalesce the CPU frequency governors text, and the tracing text.
- Add might_sleep() note to CONFIG_DEBUG_ATOMIC_SLEEP.
- Mention debugfs only as "allowed".
- Mention CONFIG_DEBUG_BUGVERBOSE.
- Do not mention DEBUG_INFO as it's not selectable; mention the
  CONFIG_DEBUG_INFO* selectable variants.

Changelog v4
------------

https://msgid.link/20260716195716.256025-1-darwi@linutronix.de

Handle Sashiko review: Remove a no-longer-referenced link.  Remove
CONFIG_WQ_SYS reference, which I thought it exists due to a quick
misunderstanding of core-api/workqueue.

Changelog v3
------------

https://msgid.link/20260715150147.225715-1-darwi@linutronix.de

CONFIG_EFI_DISABLE_RUNTIME: Mention that system integrators can now assign
the efi_runtime workqueue CPU affinity, then safely pin RT tasks on a
different CPU range.  This is due to:

  https://msgid.link/20260227170103.4042157-1-bigeasy@linutronix.de

now being merged mainline.

Changelog v2
------------

https://msgid.link/20260414174159.1271171-1-darwi@linutronix.de

Handle Rostedt remarks:

- Better reword certain paragraphs and statements
- Warn about enabling CONFIG_IRQSOFF_TRACER and CONFIG_PREEMPT_TRACER

Handle Wilcox remarks:

- Remove ToC comment + minor rewording

Changelog v1
------------

https://msgid.link/20260305205023.361530-1-darwi@linutronix.de

Thanks,

8<-----

Ahmed S. Darwish (1):
  Documentation: real-time: Add kernel configuration guide

 Documentation/core-api/real-time/index.rst    |   1 +
 .../real-time/kernel-configuration.rst        | 307 ++++++++++++++++++
 2 files changed, 308 insertions(+)
 create mode 100644 Documentation/core-api/real-time/kernel-configuration.rst

base-commit: 075b74841bd0065a3bda3440873c747938e69b68
-- 
2.55.0

[PATCH v5 1/1] Documentation: real-time: Add kernel configuration guide

[Ahmed S. Darwish]

Add a configuration guide for real-time kernels.

List all Kconfig options that are recommended to be either enabled or
disabled.  Explicitly add a table of contents at the top of the document,
so that all the options can be seen in a glance.

Whenever appropriate, link to other kernel guides; e.g. cpuidle, cpufreq,
power management, workqueues, and no_hz.

Add a summary at the end of the document warning users that there is a no
"one size fits all solution" for configuring a real-time system.

Signed-off-by: Ahmed S. Darwish <darwi@linutronix.de>
---
 Documentation/core-api/real-time/index.rst    |   1 +
 .../real-time/kernel-configuration.rst        | 307 ++++++++++++++++++
 2 files changed, 308 insertions(+)
 create mode 100644 Documentation/core-api/real-time/kernel-configuration.rst

diff --git a/Documentation/core-api/real-time/index.rst b/Documentation/core-api/real-time/index.rst
index f08d2395a22c..a17a3dec535c 100644
--- a/Documentation/core-api/real-time/index.rst
+++ b/Documentation/core-api/real-time/index.rst
@@ -15,3 +15,4 @@ the required changes compared to a non-PREEMPT_RT configuration.
    differences
    hardware
    architecture-porting
+   kernel-configuration
diff --git a/Documentation/core-api/real-time/kernel-configuration.rst b/Documentation/core-api/real-time/kernel-configuration.rst
new file mode 100644
index 000000000000..72f7e8453bcc
--- /dev/null
+++ b/Documentation/core-api/real-time/kernel-configuration.rst
@@ -0,0 +1,307 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+==============================
+Real-Time Kernel configuration
+==============================
+
+.. contents:: Table of Contents
+   :depth: 3
+   :local:
+
+Introduction
+============
+
+This document lists the kernel configuration options that might affect a
+real-time kernel's worst-case latency.  It is intended for system integrators.
+
+Configuration options
+=====================
+
+.. Please keep the configuration listings alphabetically ordered
+
+CPU frequency governors
+-----------------------
+
+``CONFIG_CPU_FREQ``
+^^^^^^^^^^^^^^^^^^^
+
+:Expectation: enabled
+:Severity: *high*
+
+The CPU frequency scaling subsystem ensures that the processor can operate at
+its maximum supported frequency.  While, in general, bootloaders are tasked
+with setting the CPU clock to the highest speed on boot, some do not.  It is
+thus desirable to keep this option enabled.
+
+.. caution::
+
+  A real-time kernel is not about being "as fast as possible", however
+  real-time requirements may demand that the CPU is clocked at a particular
+  speed.
+
+``CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+:Expectation: enabled
+:Severity: *high*
+
+Real-Time workloads expect a fixed CPU frequency during execution.  Using the
+performance governor is an easy way to achieve that purely from kernel
+configuration.
+
+This is not an absolute rule.  Some setups might prefer to clock the CPU to
+lower speeds due to thermal packaging or other requirements.  The key is that
+the CPU frequency remains constant once set.
+
+Non-performance CPU frequency governors
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+:Expectation: disabled
+:Severity: *medium*
+
+To ensure reproducible system latency measurements, disable the
+non-``PERFORMANCE`` CPU frequency governors whenever possible.  This avoids
+the risk of unknown userspace tasks implicitly or explicitly setting a
+different CPU frequency governor, and and thereby changing latency behavior
+while the system is running.
+
+If disabling other frequency governors is not an option, use a governor that
+keeps the CPU frequency fixed.  For example,
+``CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE`` can be enabled when userspace is
+responsible for setting a *stable* frequency during system initialization.
+
+If a low CPU frequency is desired, then
+``CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE`` can be set.
+
+The ``ONDEMAND`` governor should not be enabled on a real-time system.  Its
+frequency changes depend on workload behavior and can significantly harm
+determinism.
+
+For more information, see Documentation/admin-guide/pm/cpufreq.rst
+
+``CONFIG_CPU_IDLE``
+-------------------
+
+:Expectation: enabled
+:Severity: *info*
+
+CPU idle states (C-states) allow the processor to enter low-power modes during
+periods of inactivity.  Very-low CPU idle states may require flushing the CPU
+caches and lowering or disabling the clocking.  This can lower power
+consumption, but it also increases the entry and exit latency from such
+states.
+
+While disabling this option eliminates cpuidle-related latencies, doing so can
+significantly impact hardware longevity, warranty, and thermal behavior.
+Users should cap the maximum C-state to C1 instead.  For ACPI platforms, this
+can be achieved by using the boot parameter [1]_::
+
+  processor.max_cstate=1
+
+Higher C-states can be acceptable depending on the user workload's latency
+requirements.  For ACPI-based platforms, use the ``cpupower idle-info``
+command to inspect the available idle states.
+
+For more information, please see:
+
+- ``linux/tools/power/cpupower``
+- Documentation/admin-guide/pm/cpuidle.rst
+- Documentation/admin-guide/pm/index.rst
+
+``CONFIG_DRM``
+--------------
+
+:Expectation: disabled
+:Severity: *info*
+
+GPU-accelerated workloads can share system resources with the CPU, including
+last-level cache (LLC) and memory bandwidth.  Modern integrated GPUs optimize
+graphics performance at the expense of CPU determinism.
+
+Examples of affected platforms:
+
+- Intel processors with integrated graphics (Gen9 and later)
+- AMD APUs with Radeon Graphics
+- Xilinx Zynq UltraScale+ MPSoC EG/EV series
+
+If graphics workloads must run alongside real-time tasks, users must conduct
+thorough stress testing using tools like ``glmark2`` while measuring the
+overall system latency.
+
+For more information, please check:
+
+- Documentation/core-api/real-time/hardware.rst ("Regarding hardware" section)
+- Documentation/filesystems/resctrl.rst
+- `Real-Time and Graphics: A Contradiction? <https://web.archive.org/web/20221025085614/https://linutronix.de/PDF/Realtime_and_graphics-acontradiction2021.pdf>`_
+
+``CONFIG_EFI_DISABLE_RUNTIME``
+------------------------------
+
+:Expectation: enabled
+:Severity: *medium*
+
+EFI is the standard boot and firmware interface for multiple architectures.
+EFI runtime services provide callback functions to be called from the kernel;
+e.g., as utilized by (``CONFIG_EFI_VARS*``) or (``CONFIG_RTC_DRV_EFI``).  For
+the former, the kernel calls into EFI to update the EFI variables.
+
+Calling into EFI means invoking firmware callbacks.  During such invocations,
+the system might not be able to react to interrupts and will thus not be able
+to perform a context switch.  This can cause significant latency spikes for
+the real-time system.
+
+``CONFIG_PREEMPT_RT`` enables this option by default.  If this option is
+manually disabled at build time, the following boot parameter [1]_ may be used
+to disable EFI runtime at boot up::
+
+  efi=noruntime
+
+Alternatively, confine EFI runtime service calls to a housekeeping CPU by
+restricting the ``efi_runtime`` workqueue CPU affinity.  For example, set that
+workqueue's affinity to CPU #0 and pin your RT tasks to a different CPU range.
+See Documentation/core-api/workqueue.rst
+
+``CONFIG_NO_HZ`` / ``CONFIG_NO_HZ_FULL``
+----------------------------------------
+
+:Expectation: disabled
+:Severity: *medium*
+
+Tickless operation can increase kernel-to-userspace transition latency due to
+the extra accounting and state book-keeping.
+
+*Guidance by real-time workload type:*
+
+- For periodic workloads; e.g., control loops executing every 100 µs, avoid
+  ``NO_HZ`` modes.  Consistent kernel ticks are preferable.
+
+- For computation-intensive workloads; e.g. extended userspace execution,
+  ``NO_HZ_FULL`` may be beneficial.  In such cases, users should offload the
+  kernel housekeeping to dedicated CPUs and isolate compute cores.
+
+See also Documentation/timers/no_hz.rst
+
+``CONFIG_PREEMPT_RT``
+---------------------
+
+:Expectation: enabled
+:Severity: **fatal**
+
+This option must be enabled, or the resulting kernel will not be fully
+preemptible and real-time capable.
+
+``CONFIG_TRACING`` (and tracing options)
+----------------------------------------
+
+:Expectation: enabled
+:Severity: *info*
+
+Shipping kernels with tracing support enabled (but not actively running) is
+highly recommended.  This will allow the users to extract more information if
+latency problems arise.  Nonetheless, some tracers do incur latency overhead
+by just being enable.
+
+.. caution::
+
+  Users should *not* make use of tracers or trace events during production
+  real-time kernel operation as they can add considerable overhead and degrade
+  the system's latency.
+
+``CONFIG_IRQSOFF_TRACER`` and ``CONFIG_PREEMPT_TRACER``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+:Expectation: disabled
+:Severity: *high*
+
+These tracers do incur measurable latency overhead even when tracing is not
+currently active.
+
+Kernel Debug Options
+====================
+
+Most kernel debug options add runtime overhead that increases the worst-case
+latency.
+
+.. caution::
+
+  During development and early testing, users are encouraged to run their
+  real-time workloads and peripherals with lockdep (:ref:`lockdep`) and other
+  kernel debug options enabled, for a considerable amount of time.  Such
+  workloads might trigger kernel code paths that were not triggered during the
+  internal Linux real-time kernel development, thus helping to uncover locking
+  and other types of kernel bugs.
+
+``CONFIG_DEBUG_ATOMIC_SLEEP``
+-----------------------------
+
+:Expectation: allowed
+
+This sanity check catches common kernel programming errors with a tolerable
+latency cost.  It also increases overall scheduling as each ``might_sleep()``
+can lead to a context switch.
+
+``CONFIG_DEBUG_BUGVERBOSE`` and ``CONFIG_DEBUG_INFO*``
+------------------------------------------------------
+
+:Expectation: allowed
+
+These options increase the kernel image size but have no latency impact.  They
+are also essential for meaningful BUG logs, crash dumps, and profiling.
+
+``CONFIG_DEBUG_FS``
+-------------------
+
+:Expectation: allowed
+
+This is safe to include in real-time kernels, *provided that debugfs is not
+accessed during production runtime*.
+
+``CONFIG_DEBUG_KERNEL``
+-----------------------
+
+:Expectation: allowed
+
+Meta-option which allows debug features to be enabled.  It has no runtime
+impact, but beware of any debug features that it may have implicitly enabled.
+
+``CONFIG_LOCKUP_DETECTOR``
+--------------------------
+
+:Expectation: disabled
+:Severity: *high*
+
+The lockup detector creates kernel timer callbacks that execute every few
+seconds, in hard-IRQ context, even on real-time kernels.  These periodic
+interrupts can cause latency spikes.
+
+Users should use hardware watchdogs instead, which will provide a similar
+functionality without the software-induced latency.
+
+.. _lockdep:
+
+``CONFIG_PROVE_LOCKING``
+------------------------
+
+:Expectation: disabled
+:Severity: *high*
+
+Proving the correctness of all kernel locking adds substantial overhead and
+significantly increases worst-case latency.
+
+Summary
+=======
+
+There is no "one size fits all" solution for configuring a real-time Linux
+system.  Beginning with the system real-time requirements, integrators must
+consider the features and functions of the system's hardware, kernel, and
+userspace.  All such components must be properly configured in order to
+establish and constrain the system's maximum latency.
+
+With that in mind, any incorrect real-time kernel configuration could cause a
+new maximum latency that shows up at the wrong time and is catastrophic for
+the real-time system's latency.
+
+References
+==========
+
+.. [1] See Documentation/admin-guide/kernel-parameters.rst
-- 
2.55.0

Re: [PATCH v5 1/1] Documentation: real-time: Add kernel configuration guide

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Typographical errors in the documentation text and the commit message.
--

--- Patch [1]: [PATCH v5 1/1] Documentation: real-time: Add kernel configuration guide ---
commit fc9a84c888f9e6c7081f477ffb83f1ff5bc1b0aa
Author: Ahmed S. Darwish <darwi@linutronix.de>

Documentation: real-time: Add kernel configuration guide
    
This commit adds a new kernel configuration guide for real-time systems,
detailing recommended Kconfig options. It lists options for CPU governors,
cpuidle, graphics, EFI, NO_HZ, and debugging features, explaining the
latency impacts for each.

> Add a summary at the end of the document warning users that there is a no
> "one size fits all solution" for configuring a real-time system.

[Severity: Low]
This isn't a bug, but there is a typo in the commit message: "a no"
should probably just be "no".

> diff --git a/Documentation/core-api/real-time/kernel-configuration.rst b/Documentation/core-api/real-time/kernel-configuration.rst
> new file mode 100644
> index 0000000000000..72f7e8453bcc5
> --- /dev/null
> +++ b/Documentation/core-api/real-time/kernel-configuration.rst
> @@ -0,0 +1,307 @@

[ ... ]

> +Non-performance CPU frequency governors
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +:Expectation: disabled
> +:Severity: *medium*
> +
> +To ensure reproducible system latency measurements, disable the
> +non-``PERFORMANCE`` CPU frequency governors whenever possible.  This avoids
> +the risk of unknown userspace tasks implicitly or explicitly setting a
> +different CPU frequency governor, and and thereby changing latency behavior
> +while the system is running.

[Severity: Low]
This isn't a bug, but there is a duplicated "and" in the sentence above.

[ ... ]

> +``CONFIG_TRACING`` (and tracing options)
> +----------------------------------------
> +
> +:Expectation: enabled
> +:Severity: *info*
> +
> +Shipping kernels with tracing support enabled (but not actively running) is
> +highly recommended.  This will allow the users to extract more information if
> +latency problems arise.  Nonetheless, some tracers do incur latency overhead
> +by just being enable.

[Severity: Low]
This isn't a bug, but "enable" here should probably be "enabled".

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260804133746.744212-2-darwi@linutronix.de?part=1