* [PATCH v4 0/1] Documentation: Add real-time kernel configuration guide @ 2026-07-16 19:57 Ahmed S. Darwish 2026-07-16 19:57 ` [PATCH v4 1/1] Documentation: real-time: Add " Ahmed S. Darwish 0 siblings, 1 reply; 13+ messages in thread From: Ahmed S. Darwish @ 2026-07-16 19:57 UTC (permalink / raw) To: Jonathan Corbet, Clark Williams, Steven Rostedt, linux-rt-devel Cc: Matthew Wilcox, Sebastian Andrzej Siewior, John Ogness, Derek Barbosa, linux-doc, linux-kernel, Ahmed S. Darwish Hi, There is a no "one size fits all" solution for configuring a PREEMPT_RT kernel. Intorduce a PREEMPT_RT kernel configuration guide to better help system developers and integrators. Changelog v4 ------------ Handle Sashiko review: Remove a no-longer-referenced link. Remove CONFIG_WQ_SYS reference, which I thought it exists due to a quick misunderstanding of core-api/workqueue. Changelog v3 ------------ CONFIG_EFI_DISABLE_RUNTIME: Mention that system integrators can now assign the efi_runtime workqueue CPU affinity, then safely pin RT tasks on a different CPU range. Changelog v2 ------------ https://msgid.link/20260414174159.1271171-1-darwi@linutronix.de Handle Rostedt remarks: - Better reword certain paragraphs and statements - Warn about enabling CONFIG_IRQSOFF_TRACER and CONFIG_PREEMPT_TRACER Handle Wilcox remarks: - Remove ToC comment + minor rewording Changelog v1 ------------ https://msgid.link/20260305205023.361530-1-darwi@linutronix.de Thanks, 8<----- Ahmed S. Darwish (1): Documentation: real-time: Add kernel configuration guide Documentation/core-api/real-time/index.rst | 1 + .../real-time/kernel-configuration.rst | 310 ++++++++++++++++++ 2 files changed, 311 insertions(+) create mode 100644 Documentation/core-api/real-time/kernel-configuration.rst base-commit: a13c140cc289c0b7b3770bce5b3ad42ab35074aa -- 2.54.0 ^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH v4 1/1] Documentation: real-time: Add kernel configuration guide 2026-07-16 19:57 [PATCH v4 0/1] Documentation: Add real-time kernel configuration guide Ahmed S. Darwish @ 2026-07-16 19:57 ` Ahmed S. Darwish 2026-07-21 13:02 ` Sebastian Andrzej Siewior 2026-07-21 13:05 ` Sebastian Andrzej Siewior 0 siblings, 2 replies; 13+ messages in thread From: Ahmed S. Darwish @ 2026-07-16 19:57 UTC (permalink / raw) To: Jonathan Corbet, Clark Williams, Steven Rostedt, linux-rt-devel Cc: Matthew Wilcox, Sebastian Andrzej Siewior, John Ogness, Derek Barbosa, linux-doc, linux-kernel, Ahmed S. Darwish Add a configuration guide for real-time kernels. List all Kconfig options that are recommended to be either enabled or disabled. Explicitly add a table of contents at the top of the document, so that all the options can be seen in a glance. Whenever appropriate, link to other kernel guides; e.g. cpuidle, cpufreq, power management, workqueues, and no_hz. Add a summary at the end of the document warning users that there is a no "one size fits all solution" for configuring a real-time system. Signed-off-by: Ahmed S. Darwish <darwi@linutronix.de> --- Documentation/core-api/real-time/index.rst | 1 + .../real-time/kernel-configuration.rst | 310 ++++++++++++++++++ 2 files changed, 311 insertions(+) create mode 100644 Documentation/core-api/real-time/kernel-configuration.rst diff --git a/Documentation/core-api/real-time/index.rst b/Documentation/core-api/real-time/index.rst index f08d2395a22c..a17a3dec535c 100644 --- a/Documentation/core-api/real-time/index.rst +++ b/Documentation/core-api/real-time/index.rst @@ -15,3 +15,4 @@ the required changes compared to a non-PREEMPT_RT configuration. differences hardware architecture-porting + kernel-configuration diff --git a/Documentation/core-api/real-time/kernel-configuration.rst b/Documentation/core-api/real-time/kernel-configuration.rst new file mode 100644 index 000000000000..99dfc73aa4d3 --- /dev/null +++ b/Documentation/core-api/real-time/kernel-configuration.rst @@ -0,0 +1,310 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============================== +Real-Time Kernel configuration +============================== + +.. contents:: Table of Contents + :depth: 3 + :local: + +Introduction +============ + +This document lists the kernel configuration options that might affect a +real-time kernel's worst-case latency. It is intended for system integrators. + +Configuration options +===================== + +``CONFIG_CPU_FREQ`` +------------------- + +:Expectation: enabled +:Severity: *high* + +The CPU frequency scaling subsystem ensures that the processor can operate +at its maximum supported frequency. While, in general, bootloaders are +tasked with setting the CPU clock to the highest speed on boot, some do +not. It is thus desirable to keep this option enabled. + +.. caution:: + + A real-time kernel is not about being "as fast as possible", however + real-time requirements may demand that the CPU is clocked at a + particular speed. + +``CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE`` +------------------------------------------- + +:Expectation: enabled +:Severity: *high* + +Real-Time workloads expect a fixed CPU frequency during execution. Using +the performance governor is an easy way to achieve that purely from kernel +configuration. + +This is not a blanket rule. Some setups might prefer to clock the CPU to +lower speeds due to thermal packaging or other requirements. The key is +that the CPU frequency remains constant once set. + +``CONFIG_CPU_IDLE`` +------------------- + +:Expectation: enabled +:Severity: *info* + +CPU idle states (C-states) allow the processor to enter low-power modes +during periods of inactivity. Very-low CPU idle states may require +flushing the CPU caches and lowering or disabling the clocking. This can +lower power consumption, but it also increases the entry and exit latency +from such states. + +While disabling this option eliminates cpuidle-related latencies, doing so +can significantly impact hardware longevity, warranty, and thermal +behavior. Users should cap the maximum C-state to C1 instead. For ACPI +platforms, this can be achieved by using the boot parameter [1]_:: + + processor.max_cstate=1 + +Higher C-states can be acceptable depending on the user workload's latency +requirements. For ACPI-based platforms, use the ``cpupower idle-info`` +command to inspect the available idle states. + +For more information, please see: + +- ``linux/tools/power/cpupower`` +- :doc:`/admin-guide/pm/cpuidle` +- :doc:`/admin-guide/pm/index` + +``CONFIG_DRM`` +-------------- + +:Expectation: disabled +:Severity: *info* + +GPU-accelerated workloads can share system resources with the CPU, +including last-level cache (LLC) and memory bandwidth. Modern integrated +GPUs optimize graphics performance at the expense of CPU determinism. + +Examples of affected platforms: + +- Intel processors with integrated graphics (Gen9 and later) +- AMD APUs with Radeon Graphics +- Xilinx Zynq UltraScale+ MPSoC EG/EV series + +If graphics workloads must run alongside real-time tasks, users must +conduct thorough stress testing using tools like ``glmark2`` while +measuring the overall system latency. + +For more information, please check: + +- :doc:`Regarding hardware (System memory and cache) </core-api/real-time/hardware>` +- :doc:`/filesystems/resctrl` +- `Real-Time and Graphics: A Contradiction?`_ + +``CONFIG_EFI_DISABLE_RUNTIME`` +------------------------------ + +:Expectation: enabled +:Severity: *medium* + +EFI is the standard boot and firmware interface for multiple +architectures. EFI runtime services provide callback functions to be +called from the kernel; e.g., as utilized by (``CONFIG_EFI_VARS*``) or +(``CONFIG_RTC_DRV_EFI``). For the former, the kernel calls into EFI to +update the EFI variables. + +Calling into EFI means invoking firmware callbacks. During such +invocations, the system might not be able to react to interrupts and will +thus not be able to perform a context switch. This can cause significant +latency spikes for the real-time system. + +``CONFIG_PREEMPT_RT`` enables this option by default. If this option is +manually disabled at build time, the following boot parameter [1]_ may be +used to disable EFI runtime at boot up:: + + efi=noruntime + +Alternatively, confine EFI runtime service calls to a housekeeping CPU by +restricting the ``efi_runtime`` workqueue CPU affinity. For example, set +that workqueue's affinity to CPU #0 and pin your RT tasks to a different +CPU range. See :doc:`/core-api/workqueue`. + +``CONFIG_NO_HZ`` / ``CONFIG_NO_HZ_FULL`` +---------------------------------------- + +:Expectation: disabled +:Severity: *medium* + +Tickless operation can increase kernel-to-userspace transition latency due +to the extra accounting and state book-keeping. + +*Guidance by real-time workload type:* + +- For periodic workloads; e.g., control loops executing every 100 µs, avoid + ``NO_HZ`` modes. Consistent kernel ticks are preferable. + +- For computation-intensive workloads; e.g. extended userspace execution, + ``NO_HZ_FULL`` may be beneficial. In such cases, users should offload + the kernel housekeeping to dedicated CPUs and isolate compute cores. + +See also :doc:`/timers/no_hz`. + +``CONFIG_PREEMPT_RT`` +--------------------- + +:Expectation: enabled +:Severity: **fatal** + +This option must be enabled, or the resulting kernel will not be fully +preemptible and real-time capable. + +``CONFIG_TRACING`` (and tracing options) +---------------------------------------- + +:Expectation: enabled +:Severity: *info* + +Shipping kernels with tracing support enabled (but not actively running) is +highly recommended. This will allow the users to extract more information +if latency problems arise. Nonetheless, some tracers do incur latency +overhead by just being enabled; see :ref:`tracers`. + +.. caution:: + + Users should *not* make use of tracers or trace events during production + real-time kernel operation as they can add considerable overhead and + degrade the system's latency. + +Non-performance CPU frequency governors +--------------------------------------- + +:Expectation: disabled +:Severity: *medium* + +To ensure reproducible system latency measurements, disable the +non-``PERFORMANCE`` CPU frequency governors when possible. This avoids the +risk of unknown userspace tasks implicitly or explicitly setting a +different CPU frequency governor, and thus achieving different latency +results across the system's runtime. + +If disabling other frequency governors is not an option, then +``CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE`` should be enabled. In that case, +users should set a *stable* CPU frequency setting during the system +runtime, as changing the CPU frequency will increase the system latency and +affect latency measurements reproducibility. If a lower CPU frequency is +desired, then ``CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE`` should be set. + +The ``ONDEMAND`` CPU frequency governor should *not* be enabled in a +real-time system since it dramatically affects determinism depending on the +workload. + +For more information, please check :doc:`/admin-guide/pm/cpufreq`. + +Kernel Debug Options +==================== + +Most kernel debug options add runtime overhead that increases the +worst-case latency. + +.. caution:: + + During development and early testing, users are encouraged to run their + real-time workloads and peripherals with lockdep (:ref:`lockdep`) and + other kernel debug options enabled, for a considerable amount of time. + Such workloads might trigger kernel code paths that were not triggered + during the internal Linux real-time kernel development, thus helping to + uncover locking and other types of kernel bugs. + +Problematic debug options +------------------------- + +.. _tracers: + +``CONFIG_IRQSOFF_TRACER`` and ``CONFIG_PREEMPT_TRACER`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Severity: *high* + +These tracers do incur measurable latency overhead even when tracing is not +currently active. + +``CONFIG_LOCKUP_DETECTOR`` +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Severity: *high* + +The lockup detector creates kernel timer callbacks that execute every few +seconds, in hard-IRQ context, even on real-time kernels. These periodic +interrupts can cause latency spikes. + +Users should use hardware watchdogs instead, which will provide a similar +functionality without the software-induced latency. + +.. _lockdep: + +``CONFIG_PROVE_LOCKING`` +^^^^^^^^^^^^^^^^^^^^^^^^ + +Severity: *high* + +Proving the correctness of all kernel locking adds substantial overhead +and significantly increases worst-case latency. + +Allowed kernel debug options +---------------------------- + +Kernel debug options which are not included in this list should be enabled +with caution, after extensive auditing of their impact on system latency. + +``CONFIG_DEBUG_ATOMIC_SLEEP`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This sanity check catches common kernel programming errors with +a tolerable latency cost. + +``CONFIG_DEBUG_BUGVERBOSE`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This improves the debugging capabilities without affecting normal +operation latency. + +``CONFIG_DEBUG_FS`` +^^^^^^^^^^^^^^^^^^^ + +This is safe to include in real-time kernels, *provided that debugfs is +not accessed during production runtime*. + +``CONFIG_DEBUG_INFO`` +^^^^^^^^^^^^^^^^^^^^^ + +This increases the kernel image size but has no latency impact. It is +also essential for meaningful crash dumps and profiling. + +``CONFIG_DEBUG_KERNEL`` +^^^^^^^^^^^^^^^^^^^^^^^ + +Meta-option which allows debug features to be enabled. This configuration +option has no runtime impact, but be aware of any debug features that it +may have allowed to be enabled. + +Summary +======= + +There is no "one size fits all" solution for configuring a real-time Linux +system. Beginning with the system real-time requirements, integrators +must consider the features and functions of the system's hardware, kernel, +and userspace. All such components must be properly configured in order +to establish and constrain the system's maximum latency. + +With that in mind, any incorrect real-time kernel configuration could cause +a new maximum latency that shows up at the wrong time and is catastrophic +for the real-time system's latency. + +References +========== + +.. [1] See :doc:`/admin-guide/kernel-parameters` + +.. _Real-Time and Graphics\: A Contradiction?: https://web.archive.org/web/20221025085614/https://linutronix.de/PDF/Realtime_and_graphics-acontradiction2021.pdf -- 2.54.0 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [PATCH v4 1/1] Documentation: real-time: Add kernel configuration guide 2026-07-16 19:57 ` [PATCH v4 1/1] Documentation: real-time: Add " Ahmed S. Darwish @ 2026-07-21 13:02 ` Sebastian Andrzej Siewior 2026-07-21 14:34 ` John Ogness 2026-07-22 19:40 ` Ahmed S. Darwish 2026-07-21 13:05 ` Sebastian Andrzej Siewior 1 sibling, 2 replies; 13+ messages in thread From: Sebastian Andrzej Siewior @ 2026-07-21 13:02 UTC (permalink / raw) To: Ahmed S. Darwish Cc: Jonathan Corbet, Clark Williams, Steven Rostedt, linux-rt-devel, Matthew Wilcox, John Ogness, Derek Barbosa, linux-doc, linux-kernel On 2026-07-16 21:57:14 [+0200], Ahmed S. Darwish wrote: > diff --git a/Documentation/core-api/real-time/kernel-configuration.rst b/Documentation/core-api/real-time/kernel-configuration.rst > new file mode 100644 > index 000000000000..99dfc73aa4d3 > --- /dev/null > +++ b/Documentation/core-api/real-time/kernel-configuration.rst > @@ -0,0 +1,310 @@ … > +``CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE`` > +------------------------------------------- > + > +:Expectation: enabled > +:Severity: *high* > + > +Real-Time workloads expect a fixed CPU frequency during execution. Using > +the performance governor is an easy way to achieve that purely from kernel > +configuration. > + > +This is not a blanket rule. Some setups might prefer to clock the CPU to > +lower speeds due to thermal packaging or other requirements. The key is > +that the CPU frequency remains constant once set. A dynamic governor such as CONFIG_CPU_FREQ_GOV_POWERSAVE must be avoided. … > +For more information, please see: > + > +- ``linux/tools/power/cpupower`` > +- :doc:`/admin-guide/pm/cpuidle` > +- :doc:`/admin-guide/pm/index` > + > +``CONFIG_DRM`` > +-------------- > + > +:Expectation: disabled > +:Severity: *info* > + > +GPU-accelerated workloads can share system resources with the CPU, > +including last-level cache (LLC) and memory bandwidth. Modern integrated > +GPUs optimize graphics performance at the expense of CPU determinism. > + > +Examples of affected platforms: > + > +- Intel processors with integrated graphics (Gen9 and later) > +- AMD APUs with Radeon Graphics > +- Xilinx Zynq UltraScale+ MPSoC EG/EV series > + > +If graphics workloads must run alongside real-time tasks, users must > +conduct thorough stress testing using tools like ``glmark2`` while > +measuring the overall system latency. > + > +For more information, please check: > + > +- :doc:`Regarding hardware (System memory and cache) </core-api/real-time/hardware>` > +- :doc:`/filesystems/resctrl` I've been told by Jonathan to avoid this and just use plain file references such as - Documentation/admin-guide/pm/cpuidle.rst - Documentation/admin-guide/pm/index.rst > +- `Real-Time and Graphics: A Contradiction?`_ And this might be included inline, too. > +``CONFIG_TRACING`` (and tracing options) > +---------------------------------------- > + > +:Expectation: enabled > +:Severity: *info* > + > +Shipping kernels with tracing support enabled (but not actively running) is > +highly recommended. This will allow the users to extract more information > +if latency problems arise. Nonetheless, some tracers do incur latency > +overhead by just being enabled; see :ref:`tracers`. Instead of this jumping forth and back while reading, couldn't the CONFIG_IRQSOFF_TRACER and CONFIG_PREEMPT_TRACER be added here? > +.. caution:: > + > + Users should *not* make use of tracers or trace events during production > + real-time kernel operation as they can add considerable overhead and > + degrade the system's latency. > > +Non-performance CPU frequency governors > +--------------------------------------- so here it is. > +:Expectation: disabled > +:Severity: *medium* > + > +To ensure reproducible system latency measurements, disable the > +non-``PERFORMANCE`` CPU frequency governors when possible. This avoids the > +risk of unknown userspace tasks implicitly or explicitly setting a > +different CPU frequency governor, and thus achieving different latency > +results across the system's runtime. > + > +If disabling other frequency governors is not an option, then > +``CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE`` should be enabled. In that case, > +users should set a *stable* CPU frequency setting during the system > +runtime, as changing the CPU frequency will increase the system latency and > +affect latency measurements reproducibility. If a lower CPU frequency is > +desired, then ``CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE`` should be set. > + > +The ``ONDEMAND`` CPU frequency governor should *not* be enabled in a > +real-time system since it dramatically affects determinism depending on the > +workload. > + > +For more information, please check :doc:`/admin-guide/pm/cpufreq`. Could we please merge this in the previous section so that we don't have two times the CPU frequency governors. > +Kernel Debug Options > +==================== > + > +Most kernel debug options add runtime overhead that increases the > +worst-case latency. > + > +.. caution:: > + > + During development and early testing, users are encouraged to run their > + real-time workloads and peripherals with lockdep (:ref:`lockdep`) and > + other kernel debug options enabled, for a considerable amount of time. > + Such workloads might trigger kernel code paths that were not triggered > + during the internal Linux real-time kernel development, thus helping to > + uncover locking and other types of kernel bugs. > + > +Problematic debug options > +------------------------- … > +``CONFIG_PROVE_LOCKING`` > +^^^^^^^^^^^^^^^^^^^^^^^^ > + > +Severity: *high* > + > +Proving the correctness of all kernel locking adds substantial overhead > +and significantly increases worst-case latency. Could we merge the caution box from before here somehow? The statement is that lockdep is very helpful in validating lockchains and so on especially on paths which are only seen if the real-time workload is active. However, since it can easily introduce scheduling latencies of 10ms it is hardly an option that ca be enabled in production. > +Allowed kernel debug options > +---------------------------- > + > +Kernel debug options which are not included in this list should be enabled > +with caution, after extensive auditing of their impact on system latency. > + > +``CONFIG_DEBUG_ATOMIC_SLEEP`` > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > + > +This sanity check catches common kernel programming errors with > +a tolerable latency cost. It also increases scheduling since each might_sleep() can lead to context switch. > + > +``CONFIG_DEBUG_BUGVERBOSE`` > +^^^^^^^^^^^^^^^^^^^^^^^^^^^ > + > +This improves the debugging capabilities without affecting normal > +operation latency. > + > +``CONFIG_DEBUG_FS`` > +^^^^^^^^^^^^^^^^^^^ > + > +This is safe to include in real-time kernels, *provided that debugfs is > +not accessed during production runtime*. It is safe to have as long as you don't use it. I would drop it. That filesystem is not dangerous on its own. It depends on the options you change and how this affect the system. We used to have it always enabled because you needed it for tracing. This is no longer the case as it has its own tracefs which has its own live outside of debugfs. So I don't think it is needed to mention it. > +``CONFIG_DEBUG_INFO`` > +^^^^^^^^^^^^^^^^^^^^^ > + > +This increases the kernel image size but has no latency impact. It is > +also essential for meaningful crash dumps and profiling. This is also the case for CONFIG_DEBUG_BUGVERBOSE. However this option can't be selected on its own. It is selected by other options such as DEBUG_INFO_REDUCED or if you select the compression method of it. > + > +``CONFIG_DEBUG_KERNEL`` > +^^^^^^^^^^^^^^^^^^^^^^^ > + > +Meta-option which allows debug features to be enabled. This configuration > +option has no runtime impact, but be aware of any debug features that it > +may have allowed to be enabled. This one should come first (if at all) because it enables to access of few of the options mentioned in this section. > +Summary > +======= > + > +There is no "one size fits all" solution for configuring a real-time Linux > +system. Beginning with the system real-time requirements, integrators > +must consider the features and functions of the system's hardware, kernel, > +and userspace. All such components must be properly configured in order > +to establish and constrain the system's maximum latency. > + > +With that in mind, any incorrect real-time kernel configuration could cause > +a new maximum latency that shows up at the wrong time and is catastrophic > +for the real-time system's latency. Basically everything could be turned on during testing but later it must go. Certain things like CONFIG_DEBUG_LIST are probably good have but others, such as those you mentioned, need to go in order to have a usable system. Sebastian ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v4 1/1] Documentation: real-time: Add kernel configuration guide 2026-07-21 13:02 ` Sebastian Andrzej Siewior @ 2026-07-21 14:34 ` John Ogness 2026-07-21 14:36 ` Sebastian Andrzej Siewior 2026-07-22 19:40 ` Ahmed S. Darwish 1 sibling, 1 reply; 13+ messages in thread From: John Ogness @ 2026-07-21 14:34 UTC (permalink / raw) To: Sebastian Andrzej Siewior, Ahmed S. Darwish Cc: Jonathan Corbet, Clark Williams, Steven Rostedt, linux-rt-devel, Matthew Wilcox, Derek Barbosa, linux-doc, linux-kernel On 2026-07-21, Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote: >> +``CONFIG_DEBUG_FS`` >> +^^^^^^^^^^^^^^^^^^^ >> + >> +This is safe to include in real-time kernels, *provided that debugfs is >> +not accessed during production runtime*. > > It is safe to have as long as you don't use it. > > I would drop it. That filesystem is not dangerous on its own. It depends > on the options you change and how this affect the system. > We used to have it always enabled because you needed it for tracing. > This is no longer the case as it has its own tracefs which has its own > live outside of debugfs. So I don't think it is needed to mention it. Unfortunately, debugfs is needed to configure the fair deadline server: /sys/kernel/debug/sched/fair_server/cpu*/runtime John Ogness ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v4 1/1] Documentation: real-time: Add kernel configuration guide 2026-07-21 14:34 ` John Ogness @ 2026-07-21 14:36 ` Sebastian Andrzej Siewior 2026-07-21 14:48 ` Steven Rostedt 0 siblings, 1 reply; 13+ messages in thread From: Sebastian Andrzej Siewior @ 2026-07-21 14:36 UTC (permalink / raw) To: John Ogness Cc: Ahmed S. Darwish, Jonathan Corbet, Clark Williams, Steven Rostedt, linux-rt-devel, Matthew Wilcox, Derek Barbosa, linux-doc, linux-kernel On 2026-07-21 16:40:10 [+0206], John Ogness wrote: > On 2026-07-21, Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote: > >> +``CONFIG_DEBUG_FS`` > >> +^^^^^^^^^^^^^^^^^^^ > >> + > >> +This is safe to include in real-time kernels, *provided that debugfs is > >> +not accessed during production runtime*. > > > > It is safe to have as long as you don't use it. > > > > I would drop it. That filesystem is not dangerous on its own. It depends > > on the options you change and how this affect the system. > > We used to have it always enabled because you needed it for tracing. > > This is no longer the case as it has its own tracefs which has its own > > live outside of debugfs. So I don't think it is needed to mention it. > > Unfortunately, debugfs is needed to configure the fair deadline server: > > /sys/kernel/debug/sched/fair_server/cpu*/runtime So drop the reference to it. It is not "bad" to have it enabled. It is just an interface to other things. Same as sysfs except that debugfs has no stable ABI unlike sysfs. > John Ogness Sebastian ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v4 1/1] Documentation: real-time: Add kernel configuration guide 2026-07-21 14:36 ` Sebastian Andrzej Siewior @ 2026-07-21 14:48 ` Steven Rostedt 2026-07-21 15:21 ` Sebastian Andrzej Siewior 0 siblings, 1 reply; 13+ messages in thread From: Steven Rostedt @ 2026-07-21 14:48 UTC (permalink / raw) To: Sebastian Andrzej Siewior Cc: John Ogness, Ahmed S. Darwish, Jonathan Corbet, Clark Williams, linux-rt-devel, Matthew Wilcox, Derek Barbosa, linux-doc, linux-kernel On Tue, 21 Jul 2026 16:36:06 +0200 Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote: > > Unfortunately, debugfs is needed to configure the fair deadline server: > > > > /sys/kernel/debug/sched/fair_server/cpu*/runtime > > So drop the reference to it. It is not "bad" to have it enabled. It is I believe that some distros feel that it is bad to enable because there's no real auditing of its interface. Although, it is now only accessible by root. > just an interface to other things. Same as sysfs except that debugfs has > no stable ABI unlike sysfs. I found out the hard way that it can most definitely be a stable ABI :-p -- Steve ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v4 1/1] Documentation: real-time: Add kernel configuration guide 2026-07-21 14:48 ` Steven Rostedt @ 2026-07-21 15:21 ` Sebastian Andrzej Siewior 2026-07-21 15:52 ` John Ogness 0 siblings, 1 reply; 13+ messages in thread From: Sebastian Andrzej Siewior @ 2026-07-21 15:21 UTC (permalink / raw) To: Steven Rostedt Cc: John Ogness, Ahmed S. Darwish, Jonathan Corbet, Clark Williams, linux-rt-devel, Matthew Wilcox, Derek Barbosa, linux-doc, linux-kernel On 2026-07-21 10:48:06 [-0400], Steven Rostedt wrote: > On Tue, 21 Jul 2026 16:36:06 +0200 > Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote: > > > > Unfortunately, debugfs is needed to configure the fair deadline server: > > > > > > /sys/kernel/debug/sched/fair_server/cpu*/runtime > > > > So drop the reference to it. It is not "bad" to have it enabled. It is > > I believe that some distros feel that it is bad to enable because there's > no real auditing of its interface. Although, it is now only accessible by > root. Yeah. Either way, there shouldn't be anything real-time specific that requires the user have it enabled or disabled. > -- Steve Sebastian ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v4 1/1] Documentation: real-time: Add kernel configuration guide 2026-07-21 15:21 ` Sebastian Andrzej Siewior @ 2026-07-21 15:52 ` John Ogness 2026-07-21 18:08 ` Sebastian Andrzej Siewior 0 siblings, 1 reply; 13+ messages in thread From: John Ogness @ 2026-07-21 15:52 UTC (permalink / raw) To: Sebastian Andrzej Siewior, Steven Rostedt Cc: Ahmed S. Darwish, Jonathan Corbet, Clark Williams, linux-rt-devel, Matthew Wilcox, Derek Barbosa, linux-doc, linux-kernel On 2026-07-21, Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote: > On 2026-07-21 10:48:06 [-0400], Steven Rostedt wrote: >> On Tue, 21 Jul 2026 16:36:06 +0200 >> Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote: >> >>>> Unfortunately, debugfs is needed to configure the fair deadline server: >>>> >>>> /sys/kernel/debug/sched/fair_server/cpu*/runtime >>> >>> So drop the reference to it. It is not "bad" to have it enabled. It is >> >> I believe that some distros feel that it is bad to enable because there's >> no real auditing of its interface. Although, it is now only accessible by >> root. > > Yeah. Either way, there shouldn't be anything real-time specific that > requires the user have it enabled or disabled. The fair deadline server is certainly real-time specific. It is the new/additional RT throttling mechanism. Generally speaking, RT systems will want to disable it or at least reduce the 50ms default. This document is not about runtime settings, but if debugfs is not enabled, the user will have no way to configure the fair deadline server at runtime. I am OK if it is completely dropped from this document. But if there are distros that think it is bad, it might be worth mentioning that this config is important (if not critical) for real-time systems. John ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v4 1/1] Documentation: real-time: Add kernel configuration guide 2026-07-21 15:52 ` John Ogness @ 2026-07-21 18:08 ` Sebastian Andrzej Siewior 0 siblings, 0 replies; 13+ messages in thread From: Sebastian Andrzej Siewior @ 2026-07-21 18:08 UTC (permalink / raw) To: John Ogness Cc: Steven Rostedt, Ahmed S. Darwish, Jonathan Corbet, Clark Williams, linux-rt-devel, Matthew Wilcox, Derek Barbosa, linux-doc, linux-kernel On 2026-07-21 17:58:00 [+0206], John Ogness wrote: > On 2026-07-21, Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote: > > On 2026-07-21 10:48:06 [-0400], Steven Rostedt wrote: > >> On Tue, 21 Jul 2026 16:36:06 +0200 > >> Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote: > >> > >>>> Unfortunately, debugfs is needed to configure the fair deadline server: > >>>> > >>>> /sys/kernel/debug/sched/fair_server/cpu*/runtime > >>> > >>> So drop the reference to it. It is not "bad" to have it enabled. It is > >> > >> I believe that some distros feel that it is bad to enable because there's > >> no real auditing of its interface. Although, it is now only accessible by > >> root. > > > > Yeah. Either way, there shouldn't be anything real-time specific that > > requires the user have it enabled or disabled. > > The fair deadline server is certainly real-time specific. It is the > new/additional RT throttling mechanism. Generally speaking, RT systems > will want to disable it or at least reduce the 50ms default. > > This document is not about runtime settings, but if debugfs is not > enabled, the user will have no way to configure the fair deadline server > at runtime. I would suggest to delegate it to the proper section. Also, maybe we get a proper interface outside of debugfs. > I am OK if it is completely dropped from this document. But if there are > distros that think it is bad, it might be worth mentioning that this > config is important (if not critical) for real-time systems. > > John Sebastian ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v4 1/1] Documentation: real-time: Add kernel configuration guide 2026-07-21 13:02 ` Sebastian Andrzej Siewior 2026-07-21 14:34 ` John Ogness @ 2026-07-22 19:40 ` Ahmed S. Darwish 1 sibling, 0 replies; 13+ messages in thread From: Ahmed S. Darwish @ 2026-07-22 19:40 UTC (permalink / raw) To: Sebastian Andrzej Siewior Cc: Jonathan Corbet, Clark Williams, Steven Rostedt, linux-rt-devel, Matthew Wilcox, John Ogness, Derek Barbosa, linux-doc, linux-kernel On Tue, 21 Jul 2026, Sebastian Andrzej Siewior wrote: > > On 2026-07-16 21:57:14 [+0200], Ahmed S. Darwish wrote: > > +``CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE`` > > +------------------------------------------- > > + > > +:Expectation: enabled > > +:Severity: *high* > > + > > +Real-Time workloads expect a fixed CPU frequency during execution. Using > > +the performance governor is an easy way to achieve that purely from kernel > > +configuration. > > + > > +This is not a blanket rule. Some setups might prefer to clock the CPU to > > +lower speeds due to thermal packaging or other requirements. The key is > > +that the CPU frequency remains constant once set. > > A dynamic governor such as CONFIG_CPU_FREQ_GOV_POWERSAVE must be > avoided. > ACK. > > > … > > + > > +- :doc:`Regarding hardware (System memory and cache) </core-api/real-time/hardware>` > > +- :doc:`/filesystems/resctrl`> > > I've been told by Jonathan to avoid this and just use plain file > references such as > - Documentation/admin-guide/pm/cpuidle.rst > - Documentation/admin-guide/pm/index.rst > ACK. > > +- `Real-Time and Graphics: A Contradiction?`_ > > And this might be included inline, too. > > > +``CONFIG_TRACING`` (and tracing options) > > +---------------------------------------- > > + > > +:Expectation: enabled > > +:Severity: *info* > > + > > +Shipping kernels with tracing support enabled (but not actively running) is > > +highly recommended. This will allow the users to extract more information > > +if latency problems arise. Nonetheless, some tracers do incur latency > > +overhead by just being enabled; see :ref:`tracers`. > > Instead of this jumping forth and back while reading, couldn't the > CONFIG_IRQSOFF_TRACER and CONFIG_PREEMPT_TRACER be added here? > I'll think of something. > > > > +Non-performance CPU frequency governors > > +--------------------------------------- > > so here it is. > > … > > … > > Could we please merge this in the previous section so that we don't have > two times the CPU frequency governors. > OK. > > Could we merge the caution box from before here somehow? The statement > is that lockdep is very helpful in validating lockchains and so on > especially on paths which are only seen if the real-time workload is > active. However, since it can easily introduce scheduling latencies of > 10ms it is hardly an option that ca be enabled in production. > I'll think of something. > > + > > +``CONFIG_DEBUG_FS`` > > +^^^^^^^^^^^^^^^^^^^ > > + > > +This is safe to include in real-time kernels, *provided that debugfs is > > +not accessed during production runtime*. > > It is safe to have as long as you don't use it. which is what I said above. > > I would drop it. > Well, earlier I stated that debug options need to be disabled for a real-time kernel, except the explicitly list debug options which are deemed OK. So I need to mention that CONFIG_DEBUG_FS is deemed OK. The list above is not saying, "enable it for RT". It rather says, "if you need it, it's not problematic for RT." > > > + > > +``CONFIG_DEBUG_KERNEL`` > > +^^^^^^^^^^^^^^^^^^^^^^^ > > + > > +Meta-option which allows debug features to be enabled. This configuration > > +option has no runtime impact, but be aware of any debug features that it > > +may have allowed to be enabled. > > This one should come first (if at all) because it enables to access of > few of the options mentioned in this section. > All the options are alphabetically listed, so that the 3-level table of contents on top (which lists all options) is clear and users cannot miss anything. I'll think about this. Thanks, Ahmed ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v4 1/1] Documentation: real-time: Add kernel configuration guide 2026-07-16 19:57 ` [PATCH v4 1/1] Documentation: real-time: Add " Ahmed S. Darwish 2026-07-21 13:02 ` Sebastian Andrzej Siewior @ 2026-07-21 13:05 ` Sebastian Andrzej Siewior 1 sibling, 0 replies; 13+ messages in thread From: Sebastian Andrzej Siewior @ 2026-07-21 13:05 UTC (permalink / raw) To: Ahmed S. Darwish, Rafael J. Wysocki Cc: Jonathan Corbet, Clark Williams, Steven Rostedt, linux-rt-devel, Matthew Wilcox, John Ogness, Derek Barbosa, linux-doc, linux-kernel Rafael, any comments from you? Maybe on the CPU FREQ/ IDLE part? Anything that is missing from your point of view? Here is slim version of what we already have ;) On 2026-07-16 21:57:14 [+0200], Ahmed S. Darwish wrote: > new file mode 100644 > index 000000000000..99dfc73aa4d3 > --- /dev/null > +++ b/Documentation/core-api/real-time/kernel-configuration.rst > @@ -0,0 +1,310 @@ > +.. SPDX-License-Identifier: GPL-2.0 > + > +============================== > +Real-Time Kernel configuration > +============================== > + > +.. contents:: Table of Contents > + :depth: 3 > + :local: > + > +Introduction > +============ > + > +This document lists the kernel configuration options that might affect a > +real-time kernel's worst-case latency. It is intended for system integrators. > + > +Configuration options > +===================== > + > +``CONFIG_CPU_FREQ`` > +------------------- > + > +:Expectation: enabled > +:Severity: *high* > + > +The CPU frequency scaling subsystem ensures that the processor can operate > +at its maximum supported frequency. While, in general, bootloaders are > +tasked with setting the CPU clock to the highest speed on boot, some do > +not. It is thus desirable to keep this option enabled. > + > +.. caution:: > + > + A real-time kernel is not about being "as fast as possible", however > + real-time requirements may demand that the CPU is clocked at a > + particular speed. > + > +``CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE`` > +------------------------------------------- > + > +:Expectation: enabled > +:Severity: *high* > + > +Real-Time workloads expect a fixed CPU frequency during execution. Using > +the performance governor is an easy way to achieve that purely from kernel > +configuration. > + > +This is not a blanket rule. Some setups might prefer to clock the CPU to > +lower speeds due to thermal packaging or other requirements. The key is > +that the CPU frequency remains constant once set. > + > +``CONFIG_CPU_IDLE`` > +------------------- > + > +:Expectation: enabled > +:Severity: *info* > + > +CPU idle states (C-states) allow the processor to enter low-power modes > +during periods of inactivity. Very-low CPU idle states may require > +flushing the CPU caches and lowering or disabling the clocking. This can > +lower power consumption, but it also increases the entry and exit latency > +from such states. > + > +While disabling this option eliminates cpuidle-related latencies, doing so > +can significantly impact hardware longevity, warranty, and thermal > +behavior. Users should cap the maximum C-state to C1 instead. For ACPI > +platforms, this can be achieved by using the boot parameter [1]_:: > + > + processor.max_cstate=1 > + > +Higher C-states can be acceptable depending on the user workload's latency > +requirements. For ACPI-based platforms, use the ``cpupower idle-info`` > +command to inspect the available idle states. > + > +For more information, please see: > + > +- ``linux/tools/power/cpupower`` > +- :doc:`/admin-guide/pm/cpuidle` > +- :doc:`/admin-guide/pm/index` > + … > +``CONFIG_NO_HZ`` / ``CONFIG_NO_HZ_FULL`` > +---------------------------------------- > + > +:Expectation: disabled > +:Severity: *medium* > + > +Tickless operation can increase kernel-to-userspace transition latency due > +to the extra accounting and state book-keeping. > + > +*Guidance by real-time workload type:* > + > +- For periodic workloads; e.g., control loops executing every 100 µs, avoid > + ``NO_HZ`` modes. Consistent kernel ticks are preferable. > + > +- For computation-intensive workloads; e.g. extended userspace execution, > + ``NO_HZ_FULL`` may be beneficial. In such cases, users should offload > + the kernel housekeeping to dedicated CPUs and isolate compute cores. > + > +See also :doc:`/timers/no_hz`. > + … > + > +Non-performance CPU frequency governors > +--------------------------------------- > + > +:Expectation: disabled > +:Severity: *medium* > + > +To ensure reproducible system latency measurements, disable the > +non-``PERFORMANCE`` CPU frequency governors when possible. This avoids the > +risk of unknown userspace tasks implicitly or explicitly setting a > +different CPU frequency governor, and thus achieving different latency > +results across the system's runtime. > + > +If disabling other frequency governors is not an option, then > +``CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE`` should be enabled. In that case, > +users should set a *stable* CPU frequency setting during the system > +runtime, as changing the CPU frequency will increase the system latency and > +affect latency measurements reproducibility. If a lower CPU frequency is > +desired, then ``CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE`` should be set. > + > +The ``ONDEMAND`` CPU frequency governor should *not* be enabled in a > +real-time system since it dramatically affects determinism depending on the > +workload. > + > +For more information, please check :doc:`/admin-guide/pm/cpufreq`. Sebastian ^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH v2 0/1] Documentation: Add real-time kernel configuration guide @ 2026-04-14 17:41 Ahmed S. Darwish 2026-04-14 17:41 ` [PATCH v2 1/1] Documentation: real-time: Add " Ahmed S. Darwish 0 siblings, 1 reply; 13+ messages in thread From: Ahmed S. Darwish @ 2026-04-14 17:41 UTC (permalink / raw) To: Jonathan Corbet, Clark Williams, Steven Rostedt, linux-rt-devel Cc: Matthew Wilcox, Sebastian Andrzej Siewior, John Ogness, Derek Barbosa, linux-doc, linux-kernel, Ahmed S. Darwish Hi, There is a no "one size fits all" solution for configuring a PREEMPT_RT kernel. Intorduce a PREEMPT_RT kernel configuration guide to better help system developers and integrators. Changelog v2 ------------ Handle Rostedt remarks: - Better reword certain paragraphs and statements - Warn about enabling CONFIG_IRQSOFF_TRACER and CONFIG_PREEMPT_TRACER Handle Wilcox remarks: - Remove ToC comment + minor rewording Changelog v1 ------------ https://lore.kernel.org/lkml/20260305205023.361530-1-darwi@linutronix.de Thanks, 8<----- Documentation/core-api/real-time/index.rst | 1 + .../real-time/kernel-configuration.rst | 313 ++++++++++++++++++ 2 files changed, 314 insertions(+) create mode 100644 Documentation/core-api/real-time/kernel-configuration.rst base-commit: 028ef9c96e96197026887c0f092424679298aae8 -- 2.53.0 ^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH v2 1/1] Documentation: real-time: Add kernel configuration guide 2026-04-14 17:41 [PATCH v2 0/1] Documentation: Add real-time " Ahmed S. Darwish @ 2026-04-14 17:41 ` Ahmed S. Darwish 2026-04-14 17:54 ` [PATCH v3 " Ahmed S. Darwish 0 siblings, 1 reply; 13+ messages in thread From: Ahmed S. Darwish @ 2026-04-14 17:41 UTC (permalink / raw) To: Jonathan Corbet, Clark Williams, Steven Rostedt, linux-rt-devel Cc: Matthew Wilcox, Sebastian Andrzej Siewior, John Ogness, Derek Barbosa, linux-doc, linux-kernel, Ahmed S. Darwish Add a configuration guide for real-time kernels. List all Kconfig options that are recommended to be either enabled or disabled. Explicitly add a table of contents at the top of the document, so that all the options can be seen in a glance. Whenever appropriate, link to other kernel guides; e.g. cpuidle, cpufreq, power management, and no_hz. Add a summary at the end of the document warning users that there is a no "one size fits all solution" for configuring a real-time system. Signed-off-by: Ahmed S. Darwish <darwi@linutronix.de> --- Documentation/core-api/real-time/index.rst | 1 + .../real-time/kernel-configuration.rst | 313 ++++++++++++++++++ 2 files changed, 314 insertions(+) create mode 100644 Documentation/core-api/real-time/kernel-configuration.rst diff --git a/Documentation/core-api/real-time/index.rst b/Documentation/core-api/real-time/index.rst index f08d2395a22c..a17a3dec535c 100644 --- a/Documentation/core-api/real-time/index.rst +++ b/Documentation/core-api/real-time/index.rst @@ -15,3 +15,4 @@ the required changes compared to a non-PREEMPT_RT configuration. differences hardware architecture-porting + kernel-configuration diff --git a/Documentation/core-api/real-time/kernel-configuration.rst b/Documentation/core-api/real-time/kernel-configuration.rst new file mode 100644 index 000000000000..4310ca85f014 --- /dev/null +++ b/Documentation/core-api/real-time/kernel-configuration.rst @@ -0,0 +1,313 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============================== +Real-Time Kernel configuration +============================== + +.. contents:: Table of Contents + :depth: 3 + :local: + +Introduction +============ + +This document lists the kernel configuration options that might affect a +real-time kernel's worst-case latency. It is intended for system integrators. + +Configuration options +===================== + +``CONFIG_CPU_FREQ`` +------------------- + +:Expectation: enabled +:Severity: *high* + +The CPU frequency scaling subsystem ensures that the processor can operate +at its maximum supported frequency. While, in general, bootloaders are +tasked with setting the CPU clock to the highest speed on boot, some do +not. It is thus desirable to keep this option enabled. + +.. caution:: + + A real-time kernel is not about being "as fast as possible", however + real-time requirements may demand that the CPU is clocked at a + particular speed. + +``CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE`` +------------------------------------------- + +:Expectation: enabled +:Severity: *high* + +Real-Time workloads expect a fixed CPU frequency during execution. Using +the performance governor is an easy way to achieve that purely from kernel +configuration. + +This is not a blanket rule. Some setups might prefer to clock the CPU to +lower speeds due to thermal packaging or other requirements. The key is +that the CPU frequency remains constant once set. + +``CONFIG_CPU_IDLE`` +------------------- + +:Expectation: enabled +:Severity: *info* + +CPU idle states (C-states) allow the processor to enter low-power modes +during periods of inactivity. Very-low CPU idle states may require +flushing the CPU caches and lowering or disabling the clocking. This can +lower power consumption, but it also increases the entry and exit latency +from such states. + +While disabling this option eliminates cpuidle-related latencies, doing so +can significantly impact hardware longevity, warranty, and thermal +behavior. Users should cap the maximum C-state to C1 instead. For ACPI +platforms, this can be achieved by using the boot parameter [1]_:: + + processor.max_cstate=1 + +Higher C-states can be acceptable depending on the user workload's latency +requirements. For ACPI-based platforms, use the ``cpupower idle-info`` +command to inspect the available idle states. + +For more information, please see: + +- ``linux/tools/power/cpupower`` +- :doc:`/admin-guide/pm/cpuidle` +- :doc:`/admin-guide/pm/index` + +``CONFIG_DRM`` +-------------- + +:Expectation: disabled +:Severity: *info* + +GPU-accelerated workloads can share system resources with the CPU, +including last-level cache (LLC) and memory bandwidth. Modern integrated +GPUs optimize graphics performance at the expense of CPU determinism. + +Examples of affected platforms: + +- Intel processors with integrated graphics (Gen9 and later) +- AMD APUs with Radeon Graphics +- Xilinx Zynq UltraScale+ MPSoC EG/EV series + +If graphics workloads must run alongside real-time tasks, users must +conduct thorough stress testing using tools like ``glmark2`` while +measuring the overall system latency. + +For more information, please check: + +- :doc:`Regarding hardware (System memory and cache) </core-api/real-time/hardware>` +- :doc:`/filesystems/resctrl` +- `Real-Time and Graphics: A Contradiction?`_ + +``CONFIG_EFI_DISABLE_RUNTIME`` +------------------------------ + +:Expectation: enabled +:Severity: *medium* + +EFI is the standard boot and firmware interface for multiple +architectures. EFI runtime services provide callback functions to be +called from the kernel; e.g., as utilized by (``CONFIG_EFI_VARS*``) or +(``CONFIG_RTC_DRV_EFI``). For the former, the kernel calls into EFI to +update the EFI variables. + +Calling into EFI means invoking firmware callbacks. During such +invocations, the system might not be able to react to interrupts and will +thus not be able to perform a context switch. This can cause significant +latency spikes for the real-time system. + +``CONFIG_PREEMPT_RT`` enables this option by default. If this option is +manually disabled at build time, the following boot parameter [1]_ may be +used to disable EFI runtime at boot up:: + + efi=noruntime + +There is ongoing `development work`_ to allow access to EFI variables for a +real-time Linux system. + +``CONFIG_NO_HZ`` / ``CONFIG_NO_HZ_FULL`` +---------------------------------------- + +:Expectation: disabled +:Severity: *medium* + +Tickless operation can increase kernel-to-userspace transition latency due +to the extra accounting and state book-keeping. + +*Guidance by real-time workload type:* + +- For periodic workloads; e.g., control loops executing every 100 µs, avoid + ``NO_HZ`` modes. Consistent kernel ticks are preferable. + +- For computation-intensive workloads; e.g. extended userspace execution, + ``NO_HZ_FULL`` may be beneficial. In such cases, users should offload + the kernel housekeeping to dedicated CPUs and isolate compute cores. + +See also :doc:`/timers/no_hz`. + +``CONFIG_PREEMPT_RT`` +--------------------- + +:Expectation: enabled +:Severity: **fatal** + +This option must be enabled, or the resulting kernel will not be fully +preemptible and real-time capable. + +``CONFIG_TRACING`` (and tracing options) +---------------------------------------- + +:Expectation: enabled +:Severity: *info* + +Shipping kernels with tracing support enabled (but not actively running) is +highly recommended. This will allow the users to extract more information +if latency problems arise. Nonetheless, some tracers do incur latency +overhead by just being enabled; see :ref:`tracers`. + +.. caution:: + + Users should *not* make use of tracers or trace events during production + real-time kernel operation as they can add considerable overhead and + degrade the system's latency. + +Non-performance CPU frequency governors +--------------------------------------- + +:Expectation: disabled +:Severity: *medium* + +To ensure reproducible system latency measurements, disable the +non-``PERFORMANCE`` CPU frequency governors when possible. This avoids the +risk of unknown userspace tasks implicitly or explicitly setting a +different CPU frequency governor, and thus achieving different latency +results across the system's runtime. + +If disabling other frequency governors is not an option, then +``CPU_FREQ_DEFAULT_GOV_USERSPACE`` should be enabled. In that case, users +should set a *stable* CPU frequency setting during the system runtime, as +changing the CPU frequency will increase the system latency and affect +latency measurements reproducibility. If a lower CPU frequency is desired, +then ``CPU_FREQ_DEFAULT_GOV_POWERSAVE`` should be set. + +The ``ONDEMAND`` CPU frequency governor should *not* be enabled in a +real-time system since in dramatically affects determinism depending on the +workload. + +For more information, please check :doc:`/admin-guide/pm/cpufreq`. + +Kernel Debug Options +==================== + +Most kernel debug options add runtime overhead that increases the +worst-case latency. + +.. TODO: Connect lockdep with PROVE_LOCKING. Make it clear that it does +.. not uncover latency issues. + +.. caution:: + + During development and early testing, users are encouraged to run their + real-time workloads and peripherals with lockdep (:ref:`lockdep`) and + other kernel debug options enabled, for a considerable amount of time. + Such workloads might trigger kernel code paths that were not triggered + during the internal Linux real-time kernel development, thus helping to + uncover locking bugs and any real-time latency issues in the kernel. + +Problematic debug options +------------------------- + +``CONFIG_LOCKUP_DETECTOR`` +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Severity: *high* + +The lockup detector creates kernel timer callbacks that execute every few +seconds, in hard-IRQ context, even on real-time kernels. These periodic +interrupts can cause latency spikes. + +Users should use hardware watchdogs instead, which will provide a similar +functionality without the software-induced latency. + +.. _lockdep: + +``CONFIG_PROVE_LOCKING`` +^^^^^^^^^^^^^^^^^^^^^^^^ + +Severity: *high* + +Proving the correctness of all kernel locking adds substantial overhead +and significantly increases worst-case latency. + +.. _tracers: + +``CONFIG_IRQSOFF_TRACER`` and ``CONFIG_PREEMPT_TRACER`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Severity: *high* + +These tracers do incur measurable latency overhead even when tracing is not +currently active. + +Allowed kernel debug options +---------------------------- + +Kernel debug options which are not included in this list should be enabled +with caution, after extensive auditing of their impact on system latency. + +``CONFIG_DEBUG_ATOMIC_SLEEP`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This sanity check catches common kernel programming errors with +a tolerable latency cost. + +``CONFIG_DEBUG_BUGVERBOSE`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This improves the debugging capabilities without affecting normal +operation latency. + +``CONFIG_DEBUG_FS`` +^^^^^^^^^^^^^^^^^^^ + +This is safe to include in real-time kernels, *provided that debugfs is +not accessed during production runtime*. + +``CONFIG_DEBUG_INFO`` +^^^^^^^^^^^^^^^^^^^^^ + +This increases the kernel image size but has no latency impact. It is +also essential for meaningful crash dumps and profiling. + +``CONFIG_DEBUG_KERNEL`` +^^^^^^^^^^^^^^^^^^^^^^^ + +Meta-option which allows debug features to be enabled. This configuration +option has no runtime impact, but be aware of any debug features that it +may have allowed to be enabled. + +Summary +======= + +There is no "one size fits all" solution for configuring a real-time Linux +system. Beginning with the system real-time requirements, integrators +must consider the features and functions of the system's hardware, kernel, +and userspace. All such components must be properly configured in order +to establish and constrain the system's maximum latency. + +With that in mind, any incorrect real-time kernel configuration could cause +a new maximum latency that shows up at the wrong time and is catastrophic +for the real-time system's latency. + +References +========== + +.. [1] See :doc:`/admin-guide/kernel-parameters` + +.. _development work: https://lore.kernel.org/r/20260205115559.1625236-1-bigeasy@linutronix.de + +.. _Real-Time and Graphics\: A Contradiction?: https://web.archive.org/web/20221025085614/https://linutronix.de/PDF/Realtime_and_graphics-acontradiction2021.pdf -- 2.53.0 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH v3 1/1] Documentation: real-time: Add kernel configuration guide 2026-04-14 17:41 ` [PATCH v2 1/1] Documentation: real-time: Add " Ahmed S. Darwish @ 2026-04-14 17:54 ` Ahmed S. Darwish 2026-04-14 18:12 ` [PATCH v4 " Ahmed S. Darwish 0 siblings, 1 reply; 13+ messages in thread From: Ahmed S. Darwish @ 2026-04-14 17:54 UTC (permalink / raw) To: Jonathan Corbet, Clark Williams, Steven Rostedt, linux-rt-devel Cc: Matthew Wilcox, Sebastian Andrzej Siewior, John Ogness, Derek Barbosa, linux-doc, linux-kernel Add a configuration guide for real-time kernels. List all Kconfig options that are recommended to be either enabled or disabled. Explicitly add a table of contents at the top of the document, so that all the options can be seen in a glance. Whenever appropriate, link to other kernel guides; e.g. cpuidle, cpufreq, power management, and no_hz. Add a summary at the end of the document warning users that there is a no "one size fits all solution" for configuring a real-time system. Signed-off-by: Ahmed S. Darwish <darwi@linutronix.de> --- * Changelog-v3 Order the "Problematic debug options" section alphabetically, thus matching rest of the document. Link to v2 of bigeasy EFI runtime services work, instead of v1. Documentation/core-api/real-time/index.rst | 1 + .../real-time/kernel-configuration.rst | 313 ++++++++++++++++++ 2 files changed, 314 insertions(+) create mode 100644 Documentation/core-api/real-time/kernel-configuration.rst diff --git a/Documentation/core-api/real-time/index.rst b/Documentation/core-api/real-time/index.rst index f08d2395a22c..a17a3dec535c 100644 --- a/Documentation/core-api/real-time/index.rst +++ b/Documentation/core-api/real-time/index.rst @@ -15,3 +15,4 @@ the required changes compared to a non-PREEMPT_RT configuration. differences hardware architecture-porting + kernel-configuration diff --git a/Documentation/core-api/real-time/kernel-configuration.rst b/Documentation/core-api/real-time/kernel-configuration.rst new file mode 100644 index 000000000000..ab06ec2c6ef8 --- /dev/null +++ b/Documentation/core-api/real-time/kernel-configuration.rst @@ -0,0 +1,313 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============================== +Real-Time Kernel configuration +============================== + +.. contents:: Table of Contents + :depth: 3 + :local: + +Introduction +============ + +This document lists the kernel configuration options that might affect a +real-time kernel's worst-case latency. It is intended for system integrators. + +Configuration options +===================== + +``CONFIG_CPU_FREQ`` +------------------- + +:Expectation: enabled +:Severity: *high* + +The CPU frequency scaling subsystem ensures that the processor can operate +at its maximum supported frequency. While, in general, bootloaders are +tasked with setting the CPU clock to the highest speed on boot, some do +not. It is thus desirable to keep this option enabled. + +.. caution:: + + A real-time kernel is not about being "as fast as possible", however + real-time requirements may demand that the CPU is clocked at a + particular speed. + +``CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE`` +------------------------------------------- + +:Expectation: enabled +:Severity: *high* + +Real-Time workloads expect a fixed CPU frequency during execution. Using +the performance governor is an easy way to achieve that purely from kernel +configuration. + +This is not a blanket rule. Some setups might prefer to clock the CPU to +lower speeds due to thermal packaging or other requirements. The key is +that the CPU frequency remains constant once set. + +``CONFIG_CPU_IDLE`` +------------------- + +:Expectation: enabled +:Severity: *info* + +CPU idle states (C-states) allow the processor to enter low-power modes +during periods of inactivity. Very-low CPU idle states may require +flushing the CPU caches and lowering or disabling the clocking. This can +lower power consumption, but it also increases the entry and exit latency +from such states. + +While disabling this option eliminates cpuidle-related latencies, doing so +can significantly impact hardware longevity, warranty, and thermal +behavior. Users should cap the maximum C-state to C1 instead. For ACPI +platforms, this can be achieved by using the boot parameter [1]_:: + + processor.max_cstate=1 + +Higher C-states can be acceptable depending on the user workload's latency +requirements. For ACPI-based platforms, use the ``cpupower idle-info`` +command to inspect the available idle states. + +For more information, please see: + +- ``linux/tools/power/cpupower`` +- :doc:`/admin-guide/pm/cpuidle` +- :doc:`/admin-guide/pm/index` + +``CONFIG_DRM`` +-------------- + +:Expectation: disabled +:Severity: *info* + +GPU-accelerated workloads can share system resources with the CPU, +including last-level cache (LLC) and memory bandwidth. Modern integrated +GPUs optimize graphics performance at the expense of CPU determinism. + +Examples of affected platforms: + +- Intel processors with integrated graphics (Gen9 and later) +- AMD APUs with Radeon Graphics +- Xilinx Zynq UltraScale+ MPSoC EG/EV series + +If graphics workloads must run alongside real-time tasks, users must +conduct thorough stress testing using tools like ``glmark2`` while +measuring the overall system latency. + +For more information, please check: + +- :doc:`Regarding hardware (System memory and cache) </core-api/real-time/hardware>` +- :doc:`/filesystems/resctrl` +- `Real-Time and Graphics: A Contradiction?`_ + +``CONFIG_EFI_DISABLE_RUNTIME`` +------------------------------ + +:Expectation: enabled +:Severity: *medium* + +EFI is the standard boot and firmware interface for multiple +architectures. EFI runtime services provide callback functions to be +called from the kernel; e.g., as utilized by (``CONFIG_EFI_VARS*``) or +(``CONFIG_RTC_DRV_EFI``). For the former, the kernel calls into EFI to +update the EFI variables. + +Calling into EFI means invoking firmware callbacks. During such +invocations, the system might not be able to react to interrupts and will +thus not be able to perform a context switch. This can cause significant +latency spikes for the real-time system. + +``CONFIG_PREEMPT_RT`` enables this option by default. If this option is +manually disabled at build time, the following boot parameter [1]_ may be +used to disable EFI runtime at boot up:: + + efi=noruntime + +There is ongoing `development work`_ to allow access to EFI variables for a +real-time Linux system. + +``CONFIG_NO_HZ`` / ``CONFIG_NO_HZ_FULL`` +---------------------------------------- + +:Expectation: disabled +:Severity: *medium* + +Tickless operation can increase kernel-to-userspace transition latency due +to the extra accounting and state book-keeping. + +*Guidance by real-time workload type:* + +- For periodic workloads; e.g., control loops executing every 100 µs, avoid + ``NO_HZ`` modes. Consistent kernel ticks are preferable. + +- For computation-intensive workloads; e.g. extended userspace execution, + ``NO_HZ_FULL`` may be beneficial. In such cases, users should offload + the kernel housekeeping to dedicated CPUs and isolate compute cores. + +See also :doc:`/timers/no_hz`. + +``CONFIG_PREEMPT_RT`` +--------------------- + +:Expectation: enabled +:Severity: **fatal** + +This option must be enabled, or the resulting kernel will not be fully +preemptible and real-time capable. + +``CONFIG_TRACING`` (and tracing options) +---------------------------------------- + +:Expectation: enabled +:Severity: *info* + +Shipping kernels with tracing support enabled (but not actively running) is +highly recommended. This will allow the users to extract more information +if latency problems arise. Nonetheless, some tracers do incur latency +overhead by just being enabled; see :ref:`tracers`. + +.. caution:: + + Users should *not* make use of tracers or trace events during production + real-time kernel operation as they can add considerable overhead and + degrade the system's latency. + +Non-performance CPU frequency governors +--------------------------------------- + +:Expectation: disabled +:Severity: *medium* + +To ensure reproducible system latency measurements, disable the +non-``PERFORMANCE`` CPU frequency governors when possible. This avoids the +risk of unknown userspace tasks implicitly or explicitly setting a +different CPU frequency governor, and thus achieving different latency +results across the system's runtime. + +If disabling other frequency governors is not an option, then +``CPU_FREQ_DEFAULT_GOV_USERSPACE`` should be enabled. In that case, users +should set a *stable* CPU frequency setting during the system runtime, as +changing the CPU frequency will increase the system latency and affect +latency measurements reproducibility. If a lower CPU frequency is desired, +then ``CPU_FREQ_DEFAULT_GOV_POWERSAVE`` should be set. + +The ``ONDEMAND`` CPU frequency governor should *not* be enabled in a +real-time system since in dramatically affects determinism depending on the +workload. + +For more information, please check :doc:`/admin-guide/pm/cpufreq`. + +Kernel Debug Options +==================== + +Most kernel debug options add runtime overhead that increases the +worst-case latency. + +.. TODO: Connect lockdep with PROVE_LOCKING. Make it clear that it does +.. not uncover latency issues. + +.. caution:: + + During development and early testing, users are encouraged to run their + real-time workloads and peripherals with lockdep (:ref:`lockdep`) and + other kernel debug options enabled, for a considerable amount of time. + Such workloads might trigger kernel code paths that were not triggered + during the internal Linux real-time kernel development, thus helping to + uncover locking bugs and any real-time latency issues in the kernel. + +Problematic debug options +------------------------- + +.. _tracers: + +``CONFIG_IRQSOFF_TRACER`` and ``CONFIG_PREEMPT_TRACER`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Severity: *high* + +These tracers do incur measurable latency overhead even when tracing is not +currently active. + +``CONFIG_LOCKUP_DETECTOR`` +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Severity: *high* + +The lockup detector creates kernel timer callbacks that execute every few +seconds, in hard-IRQ context, even on real-time kernels. These periodic +interrupts can cause latency spikes. + +Users should use hardware watchdogs instead, which will provide a similar +functionality without the software-induced latency. + +.. _lockdep: + +``CONFIG_PROVE_LOCKING`` +^^^^^^^^^^^^^^^^^^^^^^^^ + +Severity: *high* + +Proving the correctness of all kernel locking adds substantial overhead +and significantly increases worst-case latency. + +Allowed kernel debug options +---------------------------- + +Kernel debug options which are not included in this list should be enabled +with caution, after extensive auditing of their impact on system latency. + +``CONFIG_DEBUG_ATOMIC_SLEEP`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This sanity check catches common kernel programming errors with +a tolerable latency cost. + +``CONFIG_DEBUG_BUGVERBOSE`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This improves the debugging capabilities without affecting normal +operation latency. + +``CONFIG_DEBUG_FS`` +^^^^^^^^^^^^^^^^^^^ + +This is safe to include in real-time kernels, *provided that debugfs is +not accessed during production runtime*. + +``CONFIG_DEBUG_INFO`` +^^^^^^^^^^^^^^^^^^^^^ + +This increases the kernel image size but has no latency impact. It is +also essential for meaningful crash dumps and profiling. + +``CONFIG_DEBUG_KERNEL`` +^^^^^^^^^^^^^^^^^^^^^^^ + +Meta-option which allows debug features to be enabled. This configuration +option has no runtime impact, but be aware of any debug features that it +may have allowed to be enabled. + +Summary +======= + +There is no "one size fits all" solution for configuring a real-time Linux +system. Beginning with the system real-time requirements, integrators +must consider the features and functions of the system's hardware, kernel, +and userspace. All such components must be properly configured in order +to establish and constrain the system's maximum latency. + +With that in mind, any incorrect real-time kernel configuration could cause +a new maximum latency that shows up at the wrong time and is catastrophic +for the real-time system's latency. + +References +========== + +.. [1] See :doc:`/admin-guide/kernel-parameters` + +.. _development work: https://lore.kernel.org/r/20260227170103.4042157-1-bigeasy@linutronix.de + +.. _Real-Time and Graphics\: A Contradiction?: https://web.archive.org/web/20221025085614/https://linutronix.de/PDF/Realtime_and_graphics-acontradiction2021.pdf -- 2.53.0 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH v4 1/1] Documentation: real-time: Add kernel configuration guide 2026-04-14 17:54 ` [PATCH v3 " Ahmed S. Darwish @ 2026-04-14 18:12 ` Ahmed S. Darwish 2026-05-13 20:42 ` Ahmed S. Darwish 0 siblings, 1 reply; 13+ messages in thread From: Ahmed S. Darwish @ 2026-04-14 18:12 UTC (permalink / raw) To: Jonathan Corbet, Clark Williams, Steven Rostedt, linux-rt-devel Cc: Matthew Wilcox, Sebastian Andrzej Siewior, John Ogness, Derek Barbosa, linux-doc, linux-kernel Add a configuration guide for real-time kernels. List all Kconfig options that are recommended to be either enabled or disabled. Explicitly add a table of contents at the top of the document, so that all the options can be seen in a glance. Whenever appropriate, link to other kernel guides; e.g. cpuidle, cpufreq, power management, and no_hz. Add a summary at the end of the document warning users that there is a no "one size fits all solution" for configuring a real-time system. Signed-off-by: Ahmed S. Darwish <darwi@linutronix.de> --- * Changelog v4 Handle Sashiko's review remarks at https://sashiko.dev/#/patchset/ad5_XCnVDlC9Hvup%40lx-t490 Documentation/core-api/real-time/index.rst | 1 + .../real-time/kernel-configuration.rst | 310 ++++++++++++++++++ 2 files changed, 311 insertions(+) create mode 100644 Documentation/core-api/real-time/kernel-configuration.rst diff --git a/Documentation/core-api/real-time/index.rst b/Documentation/core-api/real-time/index.rst index f08d2395a22c..a17a3dec535c 100644 --- a/Documentation/core-api/real-time/index.rst +++ b/Documentation/core-api/real-time/index.rst @@ -15,3 +15,4 @@ the required changes compared to a non-PREEMPT_RT configuration. differences hardware architecture-porting + kernel-configuration diff --git a/Documentation/core-api/real-time/kernel-configuration.rst b/Documentation/core-api/real-time/kernel-configuration.rst new file mode 100644 index 000000000000..73f7730d468e --- /dev/null +++ b/Documentation/core-api/real-time/kernel-configuration.rst @@ -0,0 +1,310 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============================== +Real-Time Kernel configuration +============================== + +.. contents:: Table of Contents + :depth: 3 + :local: + +Introduction +============ + +This document lists the kernel configuration options that might affect a +real-time kernel's worst-case latency. It is intended for system integrators. + +Configuration options +===================== + +``CONFIG_CPU_FREQ`` +------------------- + +:Expectation: enabled +:Severity: *high* + +The CPU frequency scaling subsystem ensures that the processor can operate +at its maximum supported frequency. While, in general, bootloaders are +tasked with setting the CPU clock to the highest speed on boot, some do +not. It is thus desirable to keep this option enabled. + +.. caution:: + + A real-time kernel is not about being "as fast as possible", however + real-time requirements may demand that the CPU is clocked at a + particular speed. + +``CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE`` +------------------------------------------- + +:Expectation: enabled +:Severity: *high* + +Real-Time workloads expect a fixed CPU frequency during execution. Using +the performance governor is an easy way to achieve that purely from kernel +configuration. + +This is not a blanket rule. Some setups might prefer to clock the CPU to +lower speeds due to thermal packaging or other requirements. The key is +that the CPU frequency remains constant once set. + +``CONFIG_CPU_IDLE`` +------------------- + +:Expectation: enabled +:Severity: *info* + +CPU idle states (C-states) allow the processor to enter low-power modes +during periods of inactivity. Very-low CPU idle states may require +flushing the CPU caches and lowering or disabling the clocking. This can +lower power consumption, but it also increases the entry and exit latency +from such states. + +While disabling this option eliminates cpuidle-related latencies, doing so +can significantly impact hardware longevity, warranty, and thermal +behavior. Users should cap the maximum C-state to C1 instead. For ACPI +platforms, this can be achieved by using the boot parameter [1]_:: + + processor.max_cstate=1 + +Higher C-states can be acceptable depending on the user workload's latency +requirements. For ACPI-based platforms, use the ``cpupower idle-info`` +command to inspect the available idle states. + +For more information, please see: + +- ``linux/tools/power/cpupower`` +- :doc:`/admin-guide/pm/cpuidle` +- :doc:`/admin-guide/pm/index` + +``CONFIG_DRM`` +-------------- + +:Expectation: disabled +:Severity: *info* + +GPU-accelerated workloads can share system resources with the CPU, +including last-level cache (LLC) and memory bandwidth. Modern integrated +GPUs optimize graphics performance at the expense of CPU determinism. + +Examples of affected platforms: + +- Intel processors with integrated graphics (Gen9 and later) +- AMD APUs with Radeon Graphics +- Xilinx Zynq UltraScale+ MPSoC EG/EV series + +If graphics workloads must run alongside real-time tasks, users must +conduct thorough stress testing using tools like ``glmark2`` while +measuring the overall system latency. + +For more information, please check: + +- :doc:`Regarding hardware (System memory and cache) </core-api/real-time/hardware>` +- :doc:`/filesystems/resctrl` +- `Real-Time and Graphics: A Contradiction?`_ + +``CONFIG_EFI_DISABLE_RUNTIME`` +------------------------------ + +:Expectation: enabled +:Severity: *medium* + +EFI is the standard boot and firmware interface for multiple +architectures. EFI runtime services provide callback functions to be +called from the kernel; e.g., as utilized by (``CONFIG_EFI_VARS*``) or +(``CONFIG_RTC_DRV_EFI``). For the former, the kernel calls into EFI to +update the EFI variables. + +Calling into EFI means invoking firmware callbacks. During such +invocations, the system might not be able to react to interrupts and will +thus not be able to perform a context switch. This can cause significant +latency spikes for the real-time system. + +``CONFIG_PREEMPT_RT`` enables this option by default. If this option is +manually disabled at build time, the following boot parameter [1]_ may be +used to disable EFI runtime at boot up:: + + efi=noruntime + +There is ongoing `development work`_ to allow access to EFI variables for a +real-time Linux system. + +``CONFIG_NO_HZ`` / ``CONFIG_NO_HZ_FULL`` +---------------------------------------- + +:Expectation: disabled +:Severity: *medium* + +Tickless operation can increase kernel-to-userspace transition latency due +to the extra accounting and state book-keeping. + +*Guidance by real-time workload type:* + +- For periodic workloads; e.g., control loops executing every 100 µs, avoid + ``NO_HZ`` modes. Consistent kernel ticks are preferable. + +- For computation-intensive workloads; e.g. extended userspace execution, + ``NO_HZ_FULL`` may be beneficial. In such cases, users should offload + the kernel housekeeping to dedicated CPUs and isolate compute cores. + +See also :doc:`/timers/no_hz`. + +``CONFIG_PREEMPT_RT`` +--------------------- + +:Expectation: enabled +:Severity: **fatal** + +This option must be enabled, or the resulting kernel will not be fully +preemptible and real-time capable. + +``CONFIG_TRACING`` (and tracing options) +---------------------------------------- + +:Expectation: enabled +:Severity: *info* + +Shipping kernels with tracing support enabled (but not actively running) is +highly recommended. This will allow the users to extract more information +if latency problems arise. Nonetheless, some tracers do incur latency +overhead by just being enabled; see :ref:`tracers`. + +.. caution:: + + Users should *not* make use of tracers or trace events during production + real-time kernel operation as they can add considerable overhead and + degrade the system's latency. + +Non-performance CPU frequency governors +--------------------------------------- + +:Expectation: disabled +:Severity: *medium* + +To ensure reproducible system latency measurements, disable the +non-``PERFORMANCE`` CPU frequency governors when possible. This avoids the +risk of unknown userspace tasks implicitly or explicitly setting a +different CPU frequency governor, and thus achieving different latency +results across the system's runtime. + +If disabling other frequency governors is not an option, then +``CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE`` should be enabled. In that case, +users should set a *stable* CPU frequency setting during the system +runtime, as changing the CPU frequency will increase the system latency and +affect latency measurements reproducibility. If a lower CPU frequency is +desired, then ``CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE`` should be set. + +The ``ONDEMAND`` CPU frequency governor should *not* be enabled in a +real-time system since it dramatically affects determinism depending on the +workload. + +For more information, please check :doc:`/admin-guide/pm/cpufreq`. + +Kernel Debug Options +==================== + +Most kernel debug options add runtime overhead that increases the +worst-case latency. + +.. caution:: + + During development and early testing, users are encouraged to run their + real-time workloads and peripherals with lockdep (:ref:`lockdep`) and + other kernel debug options enabled, for a considerable amount of time. + Such workloads might trigger kernel code paths that were not triggered + during the internal Linux real-time kernel development, thus helping to + uncover locking and other types of kernel bugs. + +Problematic debug options +------------------------- + +.. _tracers: + +``CONFIG_IRQSOFF_TRACER`` and ``CONFIG_PREEMPT_TRACER`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Severity: *high* + +These tracers do incur measurable latency overhead even when tracing is not +currently active. + +``CONFIG_LOCKUP_DETECTOR`` +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Severity: *high* + +The lockup detector creates kernel timer callbacks that execute every few +seconds, in hard-IRQ context, even on real-time kernels. These periodic +interrupts can cause latency spikes. + +Users should use hardware watchdogs instead, which will provide a similar +functionality without the software-induced latency. + +.. _lockdep: + +``CONFIG_PROVE_LOCKING`` +^^^^^^^^^^^^^^^^^^^^^^^^ + +Severity: *high* + +Proving the correctness of all kernel locking adds substantial overhead +and significantly increases worst-case latency. + +Allowed kernel debug options +---------------------------- + +Kernel debug options which are not included in this list should be enabled +with caution, after extensive auditing of their impact on system latency. + +``CONFIG_DEBUG_ATOMIC_SLEEP`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This sanity check catches common kernel programming errors with +a tolerable latency cost. + +``CONFIG_DEBUG_BUGVERBOSE`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This improves the debugging capabilities without affecting normal +operation latency. + +``CONFIG_DEBUG_FS`` +^^^^^^^^^^^^^^^^^^^ + +This is safe to include in real-time kernels, *provided that debugfs is +not accessed during production runtime*. + +``CONFIG_DEBUG_INFO`` +^^^^^^^^^^^^^^^^^^^^^ + +This increases the kernel image size but has no latency impact. It is +also essential for meaningful crash dumps and profiling. + +``CONFIG_DEBUG_KERNEL`` +^^^^^^^^^^^^^^^^^^^^^^^ + +Meta-option which allows debug features to be enabled. This configuration +option has no runtime impact, but be aware of any debug features that it +may have allowed to be enabled. + +Summary +======= + +There is no "one size fits all" solution for configuring a real-time Linux +system. Beginning with the system real-time requirements, integrators +must consider the features and functions of the system's hardware, kernel, +and userspace. All such components must be properly configured in order +to establish and constrain the system's maximum latency. + +With that in mind, any incorrect real-time kernel configuration could cause +a new maximum latency that shows up at the wrong time and is catastrophic +for the real-time system's latency. + +References +========== + +.. [1] See :doc:`/admin-guide/kernel-parameters` + +.. _development work: https://lore.kernel.org/r/20260227170103.4042157-1-bigeasy@linutronix.de + +.. _Real-Time and Graphics\: A Contradiction?: https://web.archive.org/web/20221025085614/https://linutronix.de/PDF/Realtime_and_graphics-acontradiction2021.pdf -- 2.53.0 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [PATCH v4 1/1] Documentation: real-time: Add kernel configuration guide 2026-04-14 18:12 ` [PATCH v4 " Ahmed S. Darwish @ 2026-05-13 20:42 ` Ahmed S. Darwish 0 siblings, 0 replies; 13+ messages in thread From: Ahmed S. Darwish @ 2026-05-13 20:42 UTC (permalink / raw) To: Jonathan Corbet, Clark Williams, Steven Rostedt, linux-rt-devel Cc: Matthew Wilcox, Sebastian Andrzej Siewior, John Ogness, Derek Barbosa, linux-doc, linux-kernel On Tue, 14 Apr 2026, Ahmed S. Darwish wrote: > > Add a configuration guide for real-time kernels. > Kind reminder. ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2026-07-22 19:40 UTC | newest] Thread overview: 13+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-07-16 19:57 [PATCH v4 0/1] Documentation: Add real-time kernel configuration guide Ahmed S. Darwish 2026-07-16 19:57 ` [PATCH v4 1/1] Documentation: real-time: Add " Ahmed S. Darwish 2026-07-21 13:02 ` Sebastian Andrzej Siewior 2026-07-21 14:34 ` John Ogness 2026-07-21 14:36 ` Sebastian Andrzej Siewior 2026-07-21 14:48 ` Steven Rostedt 2026-07-21 15:21 ` Sebastian Andrzej Siewior 2026-07-21 15:52 ` John Ogness 2026-07-21 18:08 ` Sebastian Andrzej Siewior 2026-07-22 19:40 ` Ahmed S. Darwish 2026-07-21 13:05 ` Sebastian Andrzej Siewior -- strict thread matches above, loose matches on Subject: below -- 2026-04-14 17:41 [PATCH v2 0/1] Documentation: Add real-time " Ahmed S. Darwish 2026-04-14 17:41 ` [PATCH v2 1/1] Documentation: real-time: Add " Ahmed S. Darwish 2026-04-14 17:54 ` [PATCH v3 " Ahmed S. Darwish 2026-04-14 18:12 ` [PATCH v4 " Ahmed S. Darwish 2026-05-13 20:42 ` Ahmed S. Darwish
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox