From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8EE74313535 for ; Tue, 14 Apr 2026 17:54:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776189282; cv=none; b=OlvSKp0orGlSMGtp9fXNN0+bWSMCaOXLRDUEZs+KXHoCRIkTF6kHyHqZhzF0vRN+lzyivI9wEoG25dFmAGWSCYVIItIzxXxnQDwMbvB67AeCfShj9xMcUA+YOIrwqi01KpnqZWaLdNDi/uroLjdF9z0uI7wJjR+b1txRZr4uF9o= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776189282; c=relaxed/simple; bh=1PZquthFlD4NhUncEynrn/5l/381yUGvaip+EZHQ7ac=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=b3n1YBheDmOzXRpVRm9l6q6OCgQsrVJvik3jeVsxjrGmvQejSaT5cCxbZ8HNCArNjSaJRHySB73nGzSjH2ZHhSmg7NHRz0jxHgx05+Jws0JilnyGbHBu21Pj/vGgBZB7ELlLtBRGXQORU7d6BE8m/o2l3r30Yy/mgIi/yM7QoQ8= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=TyKJpiwt; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=60OmTS7X; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="TyKJpiwt"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="60OmTS7X" Date: Tue, 14 Apr 2026 19:54:36 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1776189278; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2uRgf8HbYCi0CdeRFmPwYr+lTkRGbzbXNVVw9fK437o=; b=TyKJpiwtCCBXN2c2BH+fy+wPvkh5fght+E1nzutlZxvr2pcmGmnV/2JuQfYY5p/UrFpCBm gRS6uV5pNgAb++F2sb9ncj/1lkTkL6GHkcnZdFGnsE4UEjwL1MPcDpvjRqwjECoim14Axv s4VHZM/j0UbRdkntIFBn4rW8/bP2KW5kOrLDTWdwv0mWYlxdZHJeSPFUQzcppYV0h+uaR2 unE0uGrgXIKviAXbJyM1YVnFJQL7pyObTxwfHZ+x87oSFPKhZiepflqN+MX7lMTCo5Dl0G BMrj89q6qDO1EokPD7Cs/YHnMdXTj1gS8Ur1vwB6Pom47I+zQyF4cnfq9dEE2A== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1776189278; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2uRgf8HbYCi0CdeRFmPwYr+lTkRGbzbXNVVw9fK437o=; b=60OmTS7XZF+xahlEAJfSSDLieHLHoCFM8qVzmFZG2+eJV7ERY1txvrGhzfl858ylTFKeqA ZHgXm/QbZERnXDDw== From: "Ahmed S. Darwish" To: Jonathan Corbet , Clark Williams , Steven Rostedt , linux-rt-devel@lists.linux.dev Cc: Matthew Wilcox , Sebastian Andrzej Siewior , John Ogness , Derek Barbosa , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 1/1] Documentation: real-time: Add kernel configuration guide Message-ID: References: <20260414174159.1271171-1-darwi@linutronix.de> <20260414174159.1271171-2-darwi@linutronix.de> Precedence: bulk X-Mailing-List: linux-rt-devel@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20260414174159.1271171-2-darwi@linutronix.de> Add a configuration guide for real-time kernels. List all Kconfig options that are recommended to be either enabled or disabled. Explicitly add a table of contents at the top of the document, so that all the options can be seen in a glance. Whenever appropriate, link to other kernel guides; e.g. cpuidle, cpufreq, power management, and no_hz. Add a summary at the end of the document warning users that there is a no "one size fits all solution" for configuring a real-time system. Signed-off-by: Ahmed S. Darwish --- * Changelog-v3 Order the "Problematic debug options" section alphabetically, thus matching rest of the document. Link to v2 of bigeasy EFI runtime services work, instead of v1. Documentation/core-api/real-time/index.rst | 1 + .../real-time/kernel-configuration.rst | 313 ++++++++++++++++++ 2 files changed, 314 insertions(+) create mode 100644 Documentation/core-api/real-time/kernel-configuration.rst diff --git a/Documentation/core-api/real-time/index.rst b/Documentation/core-api/real-time/index.rst index f08d2395a22c..a17a3dec535c 100644 --- a/Documentation/core-api/real-time/index.rst +++ b/Documentation/core-api/real-time/index.rst @@ -15,3 +15,4 @@ the required changes compared to a non-PREEMPT_RT configuration. differences hardware architecture-porting + kernel-configuration diff --git a/Documentation/core-api/real-time/kernel-configuration.rst b/Documentation/core-api/real-time/kernel-configuration.rst new file mode 100644 index 000000000000..ab06ec2c6ef8 --- /dev/null +++ b/Documentation/core-api/real-time/kernel-configuration.rst @@ -0,0 +1,313 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============================== +Real-Time Kernel configuration +============================== + +.. contents:: Table of Contents + :depth: 3 + :local: + +Introduction +============ + +This document lists the kernel configuration options that might affect a +real-time kernel's worst-case latency. It is intended for system integrators. + +Configuration options +===================== + +``CONFIG_CPU_FREQ`` +------------------- + +:Expectation: enabled +:Severity: *high* + +The CPU frequency scaling subsystem ensures that the processor can operate +at its maximum supported frequency. While, in general, bootloaders are +tasked with setting the CPU clock to the highest speed on boot, some do +not. It is thus desirable to keep this option enabled. + +.. caution:: + + A real-time kernel is not about being "as fast as possible", however + real-time requirements may demand that the CPU is clocked at a + particular speed. + +``CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE`` +------------------------------------------- + +:Expectation: enabled +:Severity: *high* + +Real-Time workloads expect a fixed CPU frequency during execution. Using +the performance governor is an easy way to achieve that purely from kernel +configuration. + +This is not a blanket rule. Some setups might prefer to clock the CPU to +lower speeds due to thermal packaging or other requirements. The key is +that the CPU frequency remains constant once set. + +``CONFIG_CPU_IDLE`` +------------------- + +:Expectation: enabled +:Severity: *info* + +CPU idle states (C-states) allow the processor to enter low-power modes +during periods of inactivity. Very-low CPU idle states may require +flushing the CPU caches and lowering or disabling the clocking. This can +lower power consumption, but it also increases the entry and exit latency +from such states. + +While disabling this option eliminates cpuidle-related latencies, doing so +can significantly impact hardware longevity, warranty, and thermal +behavior. Users should cap the maximum C-state to C1 instead. For ACPI +platforms, this can be achieved by using the boot parameter [1]_:: + + processor.max_cstate=1 + +Higher C-states can be acceptable depending on the user workload's latency +requirements. For ACPI-based platforms, use the ``cpupower idle-info`` +command to inspect the available idle states. + +For more information, please see: + +- ``linux/tools/power/cpupower`` +- :doc:`/admin-guide/pm/cpuidle` +- :doc:`/admin-guide/pm/index` + +``CONFIG_DRM`` +-------------- + +:Expectation: disabled +:Severity: *info* + +GPU-accelerated workloads can share system resources with the CPU, +including last-level cache (LLC) and memory bandwidth. Modern integrated +GPUs optimize graphics performance at the expense of CPU determinism. + +Examples of affected platforms: + +- Intel processors with integrated graphics (Gen9 and later) +- AMD APUs with Radeon Graphics +- Xilinx Zynq UltraScale+ MPSoC EG/EV series + +If graphics workloads must run alongside real-time tasks, users must +conduct thorough stress testing using tools like ``glmark2`` while +measuring the overall system latency. + +For more information, please check: + +- :doc:`Regarding hardware (System memory and cache) ` +- :doc:`/filesystems/resctrl` +- `Real-Time and Graphics: A Contradiction?`_ + +``CONFIG_EFI_DISABLE_RUNTIME`` +------------------------------ + +:Expectation: enabled +:Severity: *medium* + +EFI is the standard boot and firmware interface for multiple +architectures. EFI runtime services provide callback functions to be +called from the kernel; e.g., as utilized by (``CONFIG_EFI_VARS*``) or +(``CONFIG_RTC_DRV_EFI``). For the former, the kernel calls into EFI to +update the EFI variables. + +Calling into EFI means invoking firmware callbacks. During such +invocations, the system might not be able to react to interrupts and will +thus not be able to perform a context switch. This can cause significant +latency spikes for the real-time system. + +``CONFIG_PREEMPT_RT`` enables this option by default. If this option is +manually disabled at build time, the following boot parameter [1]_ may be +used to disable EFI runtime at boot up:: + + efi=noruntime + +There is ongoing `development work`_ to allow access to EFI variables for a +real-time Linux system. + +``CONFIG_NO_HZ`` / ``CONFIG_NO_HZ_FULL`` +---------------------------------------- + +:Expectation: disabled +:Severity: *medium* + +Tickless operation can increase kernel-to-userspace transition latency due +to the extra accounting and state book-keeping. + +*Guidance by real-time workload type:* + +- For periodic workloads; e.g., control loops executing every 100 µs, avoid + ``NO_HZ`` modes. Consistent kernel ticks are preferable. + +- For computation-intensive workloads; e.g. extended userspace execution, + ``NO_HZ_FULL`` may be beneficial. In such cases, users should offload + the kernel housekeeping to dedicated CPUs and isolate compute cores. + +See also :doc:`/timers/no_hz`. + +``CONFIG_PREEMPT_RT`` +--------------------- + +:Expectation: enabled +:Severity: **fatal** + +This option must be enabled, or the resulting kernel will not be fully +preemptible and real-time capable. + +``CONFIG_TRACING`` (and tracing options) +---------------------------------------- + +:Expectation: enabled +:Severity: *info* + +Shipping kernels with tracing support enabled (but not actively running) is +highly recommended. This will allow the users to extract more information +if latency problems arise. Nonetheless, some tracers do incur latency +overhead by just being enabled; see :ref:`tracers`. + +.. caution:: + + Users should *not* make use of tracers or trace events during production + real-time kernel operation as they can add considerable overhead and + degrade the system's latency. + +Non-performance CPU frequency governors +--------------------------------------- + +:Expectation: disabled +:Severity: *medium* + +To ensure reproducible system latency measurements, disable the +non-``PERFORMANCE`` CPU frequency governors when possible. This avoids the +risk of unknown userspace tasks implicitly or explicitly setting a +different CPU frequency governor, and thus achieving different latency +results across the system's runtime. + +If disabling other frequency governors is not an option, then +``CPU_FREQ_DEFAULT_GOV_USERSPACE`` should be enabled. In that case, users +should set a *stable* CPU frequency setting during the system runtime, as +changing the CPU frequency will increase the system latency and affect +latency measurements reproducibility. If a lower CPU frequency is desired, +then ``CPU_FREQ_DEFAULT_GOV_POWERSAVE`` should be set. + +The ``ONDEMAND`` CPU frequency governor should *not* be enabled in a +real-time system since in dramatically affects determinism depending on the +workload. + +For more information, please check :doc:`/admin-guide/pm/cpufreq`. + +Kernel Debug Options +==================== + +Most kernel debug options add runtime overhead that increases the +worst-case latency. + +.. TODO: Connect lockdep with PROVE_LOCKING. Make it clear that it does +.. not uncover latency issues. + +.. caution:: + + During development and early testing, users are encouraged to run their + real-time workloads and peripherals with lockdep (:ref:`lockdep`) and + other kernel debug options enabled, for a considerable amount of time. + Such workloads might trigger kernel code paths that were not triggered + during the internal Linux real-time kernel development, thus helping to + uncover locking bugs and any real-time latency issues in the kernel. + +Problematic debug options +------------------------- + +.. _tracers: + +``CONFIG_IRQSOFF_TRACER`` and ``CONFIG_PREEMPT_TRACER`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Severity: *high* + +These tracers do incur measurable latency overhead even when tracing is not +currently active. + +``CONFIG_LOCKUP_DETECTOR`` +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Severity: *high* + +The lockup detector creates kernel timer callbacks that execute every few +seconds, in hard-IRQ context, even on real-time kernels. These periodic +interrupts can cause latency spikes. + +Users should use hardware watchdogs instead, which will provide a similar +functionality without the software-induced latency. + +.. _lockdep: + +``CONFIG_PROVE_LOCKING`` +^^^^^^^^^^^^^^^^^^^^^^^^ + +Severity: *high* + +Proving the correctness of all kernel locking adds substantial overhead +and significantly increases worst-case latency. + +Allowed kernel debug options +---------------------------- + +Kernel debug options which are not included in this list should be enabled +with caution, after extensive auditing of their impact on system latency. + +``CONFIG_DEBUG_ATOMIC_SLEEP`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This sanity check catches common kernel programming errors with +a tolerable latency cost. + +``CONFIG_DEBUG_BUGVERBOSE`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This improves the debugging capabilities without affecting normal +operation latency. + +``CONFIG_DEBUG_FS`` +^^^^^^^^^^^^^^^^^^^ + +This is safe to include in real-time kernels, *provided that debugfs is +not accessed during production runtime*. + +``CONFIG_DEBUG_INFO`` +^^^^^^^^^^^^^^^^^^^^^ + +This increases the kernel image size but has no latency impact. It is +also essential for meaningful crash dumps and profiling. + +``CONFIG_DEBUG_KERNEL`` +^^^^^^^^^^^^^^^^^^^^^^^ + +Meta-option which allows debug features to be enabled. This configuration +option has no runtime impact, but be aware of any debug features that it +may have allowed to be enabled. + +Summary +======= + +There is no "one size fits all" solution for configuring a real-time Linux +system. Beginning with the system real-time requirements, integrators +must consider the features and functions of the system's hardware, kernel, +and userspace. All such components must be properly configured in order +to establish and constrain the system's maximum latency. + +With that in mind, any incorrect real-time kernel configuration could cause +a new maximum latency that shows up at the wrong time and is catastrophic +for the real-time system's latency. + +References +========== + +.. [1] See :doc:`/admin-guide/kernel-parameters` + +.. _development work: https://lore.kernel.org/r/20260227170103.4042157-1-bigeasy@linutronix.de + +.. _Real-Time and Graphics\: A Contradiction?: https://web.archive.org/web/20221025085614/https://linutronix.de/PDF/Realtime_and_graphics-acontradiction2021.pdf -- 2.53.0