[PATCH v1 0/1] Documentation: Add real-time kernel configuration guide

[PATCH v1 0/1] Documentation: Add real-time kernel configuration guide

[Ahmed S. Darwish]

Hi,

There is a no "one size fits all" solution for configuring a PREEMPT_RT
kernel.  Intorduce a PREEMPT_RT kernel configuration guide to better help
system developers and integrators.

Thanks,

8<-----

Ahmed S. Darwish (1):
  Documentation: real-time: Add kernel configuration guide

 Documentation/core-api/real-time/index.rst    |   1 +
 .../real-time/kernel-configuration.rst        | 297 ++++++++++++++++++
 2 files changed, 298 insertions(+)
 create mode 100644 Documentation/core-api/real-time/kernel-configuration.rst

base-commit: 87fde737174a006166a9a3a197e124917eee0c55
-- 
2.53.0

[PATCH v1 1/1] Documentation: real-time: Add kernel configuration guide

[Ahmed S. Darwish]

Add a configuration guide for real-time kernels.

List all Kconfig options that are recommended to be either enabled or
disabled.  Explicitly add a table of contents at the top of the document,
so that all the options can be seen in a glance.

Whenever appropriate, link to other kernel guides; e.g. cpuidle, cpufreq,
power management, and no_hz.

Add a summary at the end of the document warning users that there is a no
"one size fits all solution" for configuring a real-time system.

Signed-off-by: Ahmed S. Darwish <darwi@linutronix.de>
---
 Documentation/core-api/real-time/index.rst    |   1 +
 .../real-time/kernel-configuration.rst        | 297 ++++++++++++++++++
 2 files changed, 298 insertions(+)
 create mode 100644 Documentation/core-api/real-time/kernel-configuration.rst

diff --git a/Documentation/core-api/real-time/index.rst b/Documentation/core-api/real-time/index.rst
index 7e14c4ea3d59..2376c7754d8e 100644
--- a/Documentation/core-api/real-time/index.rst
+++ b/Documentation/core-api/real-time/index.rst
@@ -14,3 +14,4 @@ the required changes compared to a non-PREEMPT_RT configuration.
    theory
    differences
    architecture-porting
+   kernel-configuration
diff --git a/Documentation/core-api/real-time/kernel-configuration.rst b/Documentation/core-api/real-time/kernel-configuration.rst
new file mode 100644
index 000000000000..8f47f2e2aaf7
--- /dev/null
+++ b/Documentation/core-api/real-time/kernel-configuration.rst
@@ -0,0 +1,297 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+==============================
+Real-Time Kernel configuration
+==============================
+
+.. Add a ToC so that all options can be seen in a glance
+.. contents:: Table of Contents
+   :depth: 3
+   :local:
+
+Introduction
+============
+
+This document lists the kernel configuration options that might affect a
+real-time kernel's worst-case latency.  It is intended for system integrators.
+
+Configuration options
+=====================
+
+``CONFIG_CPU_FREQ``
+-------------------
+
+:Expectation: enabled
+:Severity: *high*
+
+The CPU frequency scaling subsystem ensures that the processor can operate
+at its maximum supported frequency.  While, in general, bootloaders are
+tasked with setting the CPU clock to the highest speed on boot, some do
+not.  It is thus desirable to keep this option enabled.
+
+.. caution::
+
+  A real-time kernel is not about being "as fast as possible", however
+  real-time requirements may demand that the CPU is clocked at a
+  particular speed.
+
+``CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE``
+-------------------------------------------
+
+:Expectation: enabled
+:Severity: *high*
+
+Real-Time workloads expect a fixed CPU frequency during execution.  Using
+the performance governor is an easy way to achieve that purely from kernel
+configuration.
+
+This is not a blanket rule.  Some setups might prefer to clock the CPU to
+lower speeds due to thermal packaging or other requirements.  The key is
+that the CPU frequency remains constant once set.
+
+``CONFIG_CPU_IDLE``
+-------------------
+
+:Expectation: enabled
+:Severity: *info*
+
+CPU idle states (C-states) allow the processor to enter low-power modes
+during periods of inactivity.  Very-low CPU idle states may require
+flushing the CPU caches and lowering or disabling the clocking.  This can
+lower power consumption, but it also increases the entry and exit latency
+from such states.
+
+While disabling this option eliminates cpuidle-related latencies, doing so
+can significantly impact hardware longevity, warranty, and thermal
+behavior.  Users should cap the maximum C-state to C1 instead.  For ACPI
+platforms, this can be achieved by using the boot parameter [1]_::
+
+  processor.max_cstate=1
+
+Higher C-states can be acceptable depending on the user workload's latency
+requirements.  For ACPI-based platforms, use the ``cpupower idle-info``
+command to inspect the available idle states.
+
+For more information, please see:
+
+- ``linux/tools/power/cpupower``
+- :doc:`/admin-guide/pm/cpuidle`
+- :doc:`/admin-guide/pm/index`
+
+``CONFIG_DRM``
+--------------
+
+:Expectation: disabled
+:Severity: *info*
+
+GPU-accelerated workloads can share system resources with the CPU,
+including last-level cache (LLC) and memory bandwidth.  Modern integrated
+GPUs optimize graphics performance at the expense of CPU determinism.
+
+Examples of affected platforms:
+
+- Intel processors with integrated graphics (Gen9 and later)
+- AMD APUs with Radeon Graphics
+- Xilinx Zynq UltraScale+ MPSoC EG/EV series
+
+If graphics workloads must run alongside real-time tasks, users must
+conduct thorough stress testing using tools like ``glmark2`` while
+measuring the overall system latency.
+
+For more information, please check:
+
+- :doc:`Regarding hardware (System memory and cache) </core-api/real-time/hardware>`
+- :doc:`/filesystems/resctrl`
+- `Real-Time and Graphics: A Contradiction?`_
+
+``CONFIG_EFI_DISABLE_RUNTIME``
+------------------------------
+
+:Expectation: enabled
+:Severity: *medium*
+
+EFI is the standard boot and firmware interface for multiple
+architectures.  EFI runtime services provide callback functions to be
+called from the kernel; e.g., as utilized by (``CONFIG_EFI_VARS*``) or
+(``CONFIG_RTC_DRV_EFI``).  For the former, the kernel calls into EFI to
+update the EFI variables.
+
+Calling into EFI means invoking firmware callbacks.  During such
+invocations, the system might not be able to react to interrupts and will
+thus not be able to perform a context switch.  This can cause significant
+latency spikes for the real-time system.
+
+``CONFIG_PREEMPT_RT`` enables this option by default.  If this option is
+disabled during the kernel build, pass the following boot parameter [1]_::
+
+  efi=noruntime
+
+There is ongoing `development work`_ to allow EFI variables access for a
+real-time Linux system.
+
+``CONFIG_NO_HZ`` / ``CONFIG_NO_HZ_FULL``
+----------------------------------------
+
+:Expectation: disabled
+:Severity: *medium*
+
+Tickless operation can increase kernel-to-userspace transition latency due
+to the extra accounting and state book-keeping.
+
+*Guidance by real-time workload type:*
+
+- For periodic workloads; e.g., control loops executing every 100 µs, avoid
+  ``NO_HZ`` modes.  Consistent kernel ticks are preferable.
+
+- For computation-intensive workloads; e.g. extended userspace execution,
+  ``NO_HZ_FULL`` may be beneficial.  In such cases, users should offload
+  the kernel housekeeping to dedicated CPUs and isolate compute cores.
+
+See also :doc:`/timers/no_hz`.
+
+``CONFIG_PREEMPT_RT``
+---------------------
+
+:Expectation: enabled
+:Severity: **fatal**
+
+This option must be enabled, or the resulting kernel will not be fully
+preemptible and real-time capable.
+
+``CONFIG_TRACING`` (and tracing options)
+----------------------------------------
+
+:Expectation: enabled
+:Severity: *info*
+
+Shipping kernels with tracing support enabled (but not actively running)
+is highly recommended.  This will allow the users to extract more
+information if latency problems arise.
+
+.. caution::
+
+  Users should *not* make use of tracers or trace events during production
+  real-time kernel operation as they can add considerable overhead and
+  degrade the system's latency.
+
+Non-performance CPU frequency governors
+---------------------------------------
+
+:Expectation: disabled
+:Severity: *medium*
+
+To ensure reproducible system latency measurements, disable the
+non-``PERFORMANCE`` CPU frequency governors when possible.  This avoids the
+risk of unknown userspace tasks implicitly or explicitly setting a
+different CPU frequency governor, and thus achieving different latency
+results across the system's runtime.
+
+If disabling other frequency governors is not an option, then
+``CPU_FREQ_DEFAULT_GOV_USERSPACE`` should be enabled.  In that case, users
+should set a *stable* CPU frequency setting during the system runtime, as
+changing the CPU frequency will increase the system latency and affect
+latency measurements reproducibility.  If a lower CPU frequency is desired,
+then ``CPU_FREQ_DEFAULT_GOV_POWERSAVE`` should be set.
+
+The ``ONDEMAND`` CPU frequency governor should *not* be enabled in a
+real-time system since in dramatically affects determinism depending on the
+workload.
+
+For more information, please check :doc:`/admin-guide/pm/cpufreq`.
+
+Kernel Debug Options
+====================
+
+Most kernel debug options add runtime overhead that increases the
+worst-case latency.
+
+.. caution::
+
+  During development and early testing, users are encouraged to run their
+  real-time workloads and peripherals with lockdep and other kernel debug
+  options enabled, for a considerable amount of time.  Such workloads
+  might trigger kernel code paths that were not triggered during the
+  internal Linux real-time kernel development, thus helping to uncover any
+  real-time latency issues in the kernel.
+
+Problematic debug options
+-------------------------
+
+``CONFIG_LOCKUP_DETECTOR``
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Severity: *high*
+
+The lockup detector creates kernel timer callbacks that execute every few
+seconds, in hard-IRQ context, even on real-time kernels.  These periodic
+interrupts can cause latency spikes.
+
+Users should use hardware watchdogs instead, which will provide a similar
+functionality without the software-induced latency.
+
+``CONFIG_PROVE_LOCKING``
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+Severity: *high*
+
+Proving the correctness of all kernel locking adds substantial overhead
+and significantly increases worst-case latency.
+
+Allowed kernel debug options
+----------------------------
+
+Kernel debug options which are not included in this list should be enabled
+with caution, after extensive auditing of their impact on system latency.
+
+``CONFIG_DEBUG_ATOMIC_SLEEP``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This sanity check catches common kernel programming errors with
+a tolerable latency cost.
+
+``CONFIG_DEBUG_BUGVERBOSE``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This improves the debugging capabilities without affecting normal
+operation latency.
+
+``CONFIG_DEBUG_FS``
+^^^^^^^^^^^^^^^^^^^
+
+This is safe to include in real-time kernels, *provided that debugfs is
+not accessed during production runtime*.
+
+``CONFIG_DEBUG_INFO``
+^^^^^^^^^^^^^^^^^^^^^
+
+This increases the kernel image size but has no latency impact.  It is
+also essential for meaningful crash dumps and profiling.
+
+``CONFIG_DEBUG_KERNEL``
+^^^^^^^^^^^^^^^^^^^^^^^
+
+Meta-option which allows debug features to be enabled.  This configuration
+option has no runtime impact, but be aware of any debug features that it
+may have allowed to be enabled.
+
+Summary
+=======
+
+There is no "one size fits all" solution for configuring a real-time Linux
+system.  Beginning with the system real-time requirements, integrators
+must consider the features and functions of the system's hardware, kernel,
+and userspace.  All such components must be properly configured in order
+to establish and constrain the system's maximum latency.
+
+With that in mind, any false real-time kernel configuration could cause a
+new maximum latency that shows up at the wrong time and is catastrophic
+for the real-time system's latency.
+
+References
+==========
+
+.. [1] See :doc:`/admin-guide/kernel-parameters`
+
+.. _development work: https://lore.kernel.org/r/20260205115559.1625236-1-bigeasy@linutronix.de
+
+.. _Real-Time and Graphics\: A Contradiction?: https://web.archive.org/web/20221025085614/https://linutronix.de/PDF/Realtime_and_graphics-acontradiction2021.pdf
-- 
2.53.0

Re: [PATCH v1 1/1] Documentation: real-time: Add kernel configuration guide

[Matthew Wilcox]

On Thu, Mar 05, 2026 at 09:50:12PM +0100, Ahmed S. Darwish wrote:
> +==============================
> +Real-Time Kernel configuration
> +==============================
> +
> +.. Add a ToC so that all options can be seen in a glance

Is this a useful comment?

> +.. contents:: Table of Contents
> +   :depth: 3
> +   :local:

[...]

> +``CONFIG_CPU_FREQ``
> +-------------------
> +
> +:Expectation: enabled
> +:Severity: *high*

Importance rather than Severity, perhaps?

And high/medium/low rather than high/medium/info.

> +With that in mind, any false real-time kernel configuration could cause a

incorrect rather than false, perhaps?

> +new maximum latency that shows up at the wrong time and is catastrophic
> +for the real-time system's latency.

Re: [PATCH v1 1/1] Documentation: real-time: Add kernel configuration guide

[Ahmed S. Darwish]

Hi Matthew,

On Thu, 05 Mar 2026, Matthew Wilcox wrote:
>
> On Thu, Mar 05, 2026 at 09:50:12PM +0100, Ahmed S. Darwish wrote:
> > +==============================
> > +Real-Time Kernel configuration
> > +==============================
> > +
> > +.. Add a ToC so that all options can be seen in a glance
>
> Is this a useful comment?
>

I thought I'm the only one explicitly adding a ToC.  A quick grep shows
that to be false so I'll remove the comment.

> > +.. contents:: Table of Contents
> > +   :depth: 3
> > +   :local:
>
> [...]
>
> > +``CONFIG_CPU_FREQ``
> > +-------------------
> > +
> > +:Expectation: enabled
> > +:Severity: *high*
>
> Importance rather than Severity, perhaps?
>
> And high/medium/low rather than high/medium/info.
>

The info sections were meant to provide extra info on the kconfig options
w.r.t. PREEMPT_RT.  So, the CONFIG_DRM entry is not of low importance as
"Importance: low" would imply.  I'll think of something here.

> > +With that in mind, any false real-time kernel configuration could cause a
>
> incorrect rather than false, perhaps?
>

will do.

Thanks a lot,
Ahmed

Re: [PATCH v1 1/1] Documentation: real-time: Add kernel configuration guide

[Steven Rostedt]

On Thu, 5 Mar 2026 21:05:15 +0000
Matthew Wilcox <willy@infradead.org> wrote:

> > +With that in mind, any false real-time kernel configuration could cause a  
> 
> incorrect rather than false, perhaps?

Heh, I just read this now after making the same comment.

-- Steve

Re: [PATCH v1 1/1] Documentation: real-time: Add kernel configuration guide

[Steven Rostedt]

On Thu,  5 Mar 2026 21:50:12 +0100
"Ahmed S. Darwish" <darwi@linutronix.de> wrote:

> Add a configuration guide for real-time kernels.
> 
> List all Kconfig options that are recommended to be either enabled or
> disabled.  Explicitly add a table of contents at the top of the document,
> so that all the options can be seen in a glance.
> 
> Whenever appropriate, link to other kernel guides; e.g. cpuidle, cpufreq,
> power management, and no_hz.
> 
> Add a summary at the end of the document warning users that there is a no
> "one size fits all solution" for configuring a real-time system.
> 

Very nice document!

> Signed-off-by: Ahmed S. Darwish <darwi@linutronix.de>
> ---
>  Documentation/core-api/real-time/index.rst    |   1 +
>  .../real-time/kernel-configuration.rst        | 297 ++++++++++++++++++
>  2 files changed, 298 insertions(+)
>  create mode 100644 Documentation/core-api/real-time/kernel-configuration.rst
> 

> +``CONFIG_EFI_DISABLE_RUNTIME``
> +------------------------------
> +
> +:Expectation: enabled
> +:Severity: *medium*
> +
> +EFI is the standard boot and firmware interface for multiple
> +architectures.  EFI runtime services provide callback functions to be
> +called from the kernel; e.g., as utilized by (``CONFIG_EFI_VARS*``) or
> +(``CONFIG_RTC_DRV_EFI``).  For the former, the kernel calls into EFI to
> +update the EFI variables.
> +
> +Calling into EFI means invoking firmware callbacks.  During such
> +invocations, the system might not be able to react to interrupts and will
> +thus not be able to perform a context switch.  This can cause significant
> +latency spikes for the real-time system.


> +
> +``CONFIG_PREEMPT_RT`` enables this option by default.  If this option is
> +disabled during the kernel build, pass the following boot parameter [1]_::
> +
> +  efi=noruntime

The above reads a bit funny. Maybe reword it to:

  ``CONFIG_PREEMPT_RT`` enables this option by default. If this option is
  manually disabled at build time, the following boot parameter [1]_ may
  be used to disable EFI runtime at boot up::

Or something like that.

> +
> +There is ongoing `development work`_ to allow EFI variables access for a
> +real-time Linux system.

  .. to allow access to EFI variables for a real-time Linux system

 ?

> +``CONFIG_TRACING`` (and tracing options)
> +----------------------------------------
> +
> +:Expectation: enabled
> +:Severity: *info*
> +
> +Shipping kernels with tracing support enabled (but not actively running)
> +is highly recommended.  This will allow the users to extract more
> +information if latency problems arise.
> +
> +.. caution::
> +
> +  Users should *not* make use of tracers or trace events during production
> +  real-time kernel operation as they can add considerable overhead and
> +  degrade the system's latency.

I wonder if a special note should be called out for:

  CONFIG_IRQSOFF_TRACER and CONFIG_PREEMPT_TRACER should be avoided as they
  do incur measurable overhead even when tracing is not currently active.

Maybe the above should be added in the "Problematic debug options"?


> +Kernel Debug Options
> +====================
> +
> +Most kernel debug options add runtime overhead that increases the
> +worst-case latency.
> +
> +.. caution::
> +
> +  During development and early testing, users are encouraged to run their
> +  real-time workloads and peripherals with lockdep and other kernel debug
> +  options enabled, for a considerable amount of time.  Such workloads
> +  might trigger kernel code paths that were not triggered during the
> +  internal Linux real-time kernel development, thus helping to uncover any
> +  real-time latency issues in the kernel.

Hmm, perhaps there should be some note that connects the use of "lockdep"
with CONFIG_PROVE_LOCKING below (as that is what enables lockdep). The last
sentence above makes it sound like lockdep can uncover latency issues, but
it will most likely cause latency issues. Perhaps a bit more explanation
should be used here.

> +
> +Problematic debug options
> +-------------------------
> +
> +``CONFIG_LOCKUP_DETECTOR``
> +^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +Severity: *high*
> +
> +The lockup detector creates kernel timer callbacks that execute every few
> +seconds, in hard-IRQ context, even on real-time kernels.  These periodic
> +interrupts can cause latency spikes.
> +
> +Users should use hardware watchdogs instead, which will provide a similar
> +functionality without the software-induced latency.
> +
> +``CONFIG_PROVE_LOCKING``
> +^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +Severity: *high*
> +
> +Proving the correctness of all kernel locking adds substantial overhead
> +and significantly increases worst-case latency.



> +Summary
> +=======
> +
> +There is no "one size fits all" solution for configuring a real-time Linux
> +system.  Beginning with the system real-time requirements, integrators
> +must consider the features and functions of the system's hardware, kernel,
> +and userspace.  All such components must be properly configured in order
> +to establish and constrain the system's maximum latency.
> +
> +With that in mind, any false real-time kernel configuration could cause a

  s/false/incorrect/ ?

> +new maximum latency that shows up at the wrong time and is catastrophic
> +for the real-time system's latency.
> +
> +References
> +==========
> +
> +.. [1] See :doc:`/admin-guide/kernel-parameters`
> +
> +.. _development work: https://lore.kernel.org/r/20260205115559.1625236-1-bigeasy@linutronix.de
> +
> +.. _Real-Time and Graphics\: A Contradiction?: https://web.archive.org/web/20221025085614/https://linutronix.de/PDF/Realtime_and_graphics-acontradiction2021.pdf

Nice job!

-- Steve

Re: [PATCH v1 1/1] Documentation: real-time: Add kernel configuration guide

[Ahmed S. Darwish]

On Thu, 05 Mar 2026, Steven Rostedt wrote:
>
> Very nice document!
>

Oh, thanks :)

>
> > +
> > +``CONFIG_PREEMPT_RT`` enables this option by default.  If this option is
> > +disabled during the kernel build, pass the following boot parameter [1]_::
> > +
> > +  efi=noruntime
>
> The above reads a bit funny. Maybe reword it to:
>
>   ``CONFIG_PREEMPT_RT`` enables this option by default. If this option is
>   manually disabled at build time, the following boot parameter [1]_ may
>   be used to disable EFI runtime at boot up::
>
> Or something like that.
>

Yes, much better; will do.

> > +
> > +There is ongoing `development work`_ to allow EFI variables access for a
> > +real-time Linux system.
>
>   .. to allow access to EFI variables for a real-time Linux system
>
>  ?
>

Yup.

> > +``CONFIG_TRACING`` (and tracing options)
> > +----------------------------------------
> > +
> > +:Expectation: enabled
> > +:Severity: *info*
> > +
> > +Shipping kernels with tracing support enabled (but not actively running)
> > +is highly recommended.  This will allow the users to extract more
> > +information if latency problems arise.
> > +
> > +.. caution::
> > +
> > +  Users should *not* make use of tracers or trace events during production
> > +  real-time kernel operation as they can add considerable overhead and
> > +  degrade the system's latency.
>
> I wonder if a special note should be called out for:
>
>   CONFIG_IRQSOFF_TRACER and CONFIG_PREEMPT_TRACER should be avoided as they
>   do incur measurable overhead even when tracing is not currently active.
>
> Maybe the above should be added in the "Problematic debug options"?
>

Oh, didn't know about that; I'll add them for sure.

>
> > +Kernel Debug Options
> > +====================
> > +
> > +Most kernel debug options add runtime overhead that increases the
> > +worst-case latency.
> > +
> > +.. caution::
> > +
> > +  During development and early testing, users are encouraged to run their
> > +  real-time workloads and peripherals with lockdep and other kernel debug
> > +  options enabled, for a considerable amount of time.  Such workloads
> > +  might trigger kernel code paths that were not triggered during the
> > +  internal Linux real-time kernel development, thus helping to uncover any
> > +  real-time latency issues in the kernel.
>
> Hmm, perhaps there should be some note that connects the use of "lockdep"
> with CONFIG_PROVE_LOCKING below (as that is what enables lockdep). The last
> sentence above makes it sound like lockdep can uncover latency issues, but
> it will most likely cause latency issues. Perhaps a bit more explanation
> should be used here.
>

will do.

> > +
> > +With that in mind, any false real-time kernel configuration could cause a
>
>   s/false/incorrect/ ?
>

Yup.

>
> Nice job!
>

Thanks!  I'll send a v2 on Monday.

All the best,
Ahmed