[PATCH] docs: timers: hrtimers: clarify expiry modes and ktimersd on PREEMPT_RT

[PATCH] docs: timers: hrtimers: clarify expiry modes and ktimersd on PREEMPT_RT

[Liang Hao]

Documentation/timers/hrtimers.rst described the high-resolution timer
subsystem but did not cover the PREEMPT_RT expiry-mode semantics.  On a
PREEMPT_RT kernel a timer that is not explicitly marked
HRTIMER_MODE_HARD is forced into softirq expiry and its callback runs on
the per-CPU ktimers/%u thread at SCHED_FIFO priority 1, regardless of
the priority of the task that armed it -- a SCHED_FIFO task's priority
does not extend into its timer callback.  This is a recurring source of
hard-to-diagnose latency for RT/DL authors who assume the opposite.

Add a dedicated "Expiry modes and PREEMPT_RT" section that documents:

 - the distinction between HRTIMER_MODE_HARD and HRTIMER_MODE_SOFT;
 - the fact that unmarked timers are forced into softirq expiry on
   PREEMPT_RT (__hrtimer_setup());
 - the role of the per-CPU ktimers/%u thread and its fixed SCHED_FIFO
   priority 1 (sched_set_fifo_low());
 - the absence of priority inheritance between the arming task and the
   timer callback on RT;
 - the sleeper exception, where hrtimer_setup_sleeper() automatically
   marks RT/DL-armed timers HRTIMER_MODE_HARD
   (__hrtimer_setup_sleeper());
 - the critical distinction between the *requested* expiry mode (passed
   by the caller) and the *effective* execution context (chosen by the
   kernel and stored in timer->is_soft).

The hrtimer_start tracepoint logs the *requested* mode, not the
effective one.  On PREEMPT_RT a timer armed with the default mode
(e.g. ABS or REL without HARD/SOFT flags) is implicitly forced into
softirq expiry, so the effective soft nature must be inferred from the
*absence* of explicit mode flags in the trace rather than read directly.

Documentation only; no code or behaviour change.

Signed-off-by: Liang Hao <haohlliang@gmail.com>
---
 Documentation/timers/hrtimers.rst | 66 +++++++++++++++++++++++++++++++
 1 file changed, 66 insertions(+)

diff --git a/Documentation/timers/hrtimers.rst b/Documentation/timers/hrtimers.rst
index f88ff8bae89c..ff42377668c1 100644
--- a/Documentation/timers/hrtimers.rst
+++ b/Documentation/timers/hrtimers.rst
@@ -171,3 +171,69 @@ hrtimers-based high-resolution clock implementation, so the hrtimers
 code got a healthy amount of testing and use in practice.
 
 	Thomas Gleixner, Ingo Molnar
+
+
+Expiry modes and PREEMPT_RT
+---------------------------
+
+Each hrtimer carries an expiry mode that determines the execution context
+of its callback:
+
+  * ``HRTIMER_MODE_HARD`` -- the callback runs in hard interrupt context.
+    It must be hardirq-safe (no sleeping locks, no allocations, no
+    scheduling).
+  * ``HRTIMER_MODE_SOFT`` -- the callback runs in softirq context and may
+    use operations that are not hardirq-safe.
+  * Default (neither flag) -- the mode is selected by the subsystem or
+    the kernel configuration.
+
+On ``CONFIG_PREEMPT_RT`` the choice is not optional for most timers:
+any timer not explicitly marked ``HRTIMER_MODE_HARD`` is forced into
+softirq expiry (see ``__hrtimer_setup()``). Instead of executing in
+hardirq context or within the context of the task that armed it, the
+callback runs on the per-CPU ``ktimers/%u`` thread.
+
+The ``ktimersd`` thread operates at ``SCHED_FIFO`` priority 1 (established
+via ``sched_set_fifo_low()``). This priority is fixed and **does not
+inherit the priority of the task that armed the timer**. The effective
+execution context is determined internally by the hrtimer subsystem at
+setup time and is reflected in ``timer->is_soft``; it is not directly
+visible as a mode flag at arming time.
+
+Consequently, even if a timer is armed by a ``SCHED_FIFO`` task with
+priority 99, its callback will execute only when the ``ktimersd`` thread
+(priority 1) is selected to run.
+
+This design ensures that timer processing does not interfere with
+higher-priority real-time workloads, while still providing bounded
+latency relative to ``SCHED_OTHER`` tasks. However, it also means that
+latency-sensitive processing must not rely on implicit priority
+inheritance through the timer arming path.
+
+A notable exception is the sleeper path: a timer set up via
+``hrtimer_setup_sleeper()`` (used by ``clock_nanosleep()`` and similar)
+that is armed by an RT or DEADLINE task is automatically marked
+``HRTIMER_MODE_HARD``, so its wakeup runs in hardirq context and does
+not go through ``ktimersd`` (see ``__hrtimer_setup_sleeper()``).
+
+**Guidelines for RT authors:**
+
+- If the timer callback contains logic that must execute at the priority
+  of the owning RT task, the timer must be declared as
+  ``HRTIMER_MODE_HARD``. Ensure the callback adheres to hardirq context
+  constraints.
+- Alternatively, move the latency-sensitive logic out of the timer
+  callback and into a dedicated, properly prioritized kthread which is
+  woken by the timer.
+
+**Debugging context:** The ``hrtimer_start`` tracepoint logs the
+*requested* expiry mode passed by the caller (e.g. ``ABS``, ``REL``,
+``ABS|HARD``), not the effective mode chosen by the kernel. On
+PREEMPT_RT, any timer armed with the default mode (i.e. the trace shows
+``ABS`` or ``REL`` without ``|SOFT`` or ``|HARD``) is implicitly forced
+into softirq expiry. When such a timer is armed by an RT or DEADLINE
+task, the consequence -- the callback running at ``ktimersd`` priority
+rather than the arming task's -- is the case to watch. The effective
+soft nature of such timers is inferred from the *absence* of an explicit
+mode flag in the trace, combined with the ``CONFIG_PREEMPT_RT``
+configuration.
-- 
2.50.1 (Apple Git-155)

[PATCH v2] docs: timers: hrtimers: clarify expiry modes and ktimersd on PREEMPT_RT

[Liang Hao]

Documentation/timers/hrtimers.rst did not cover the PREEMPT_RT
expiry-mode semantics.  On a PREEMPT_RT kernel a timer that is not
explicitly marked HRTIMER_MODE_HARD is forced into softirq expiry and
its callback runs on the per-CPU ktimers/%u thread at the lowest
SCHED_FIFO priority (sched_set_fifo_low), regardless of the priority of
the task that armed it -- a SCHED_FIFO task running at priority 99 that
starts an unmarked timer still expires on ktimers/%u (lowest SCHED_FIFO
priority), not at priority 99.

Add an "Expiry modes and PREEMPT_RT" section that, rather than
duplicating the default-context description in
Documentation/core-api/real-time/differences.rst (Timers),
cross-references it and focuses on what that document does not spell
out:

 - the callback does not inherit the arming task's priority, and
   priority inheritance on PREEMPT_RT is used for the cancel handshake,
   not the arming path (the "Spin until ready" section of the same
   document);
 - the sleeper exception: hrtimer_setup_sleeper() marks RT/DL-armed
   timers HRTIMER_MODE_HARD, so their wakeups do not go through
   ktimers/%u.

Documentation only; no code or behaviour change.

Signed-off-by: Liang Hao <haohlliang@gmail.com>
---
v1 -> v2:
- shorten the RT overview; link to real-time/differences
- state the arming-path priority consequence (priority not inherited;
  PI is for the cancel handshake)
- drop the hrtimer_start trace debugging section
- use the ktimers/%u thread name, with ktimersd as its doc alias

 Documentation/timers/hrtimers.rst | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/Documentation/timers/hrtimers.rst b/Documentation/timers/hrtimers.rst
index f88ff8bae89c..fae191550c5d 100644
--- a/Documentation/timers/hrtimers.rst
+++ b/Documentation/timers/hrtimers.rst
@@ -171,3 +171,31 @@ hrtimers-based high-resolution clock implementation, so the hrtimers
 code got a healthy amount of testing and use in practice.
 
 	Thomas Gleixner, Ingo Molnar
+
+
+Expiry modes and PREEMPT_RT
+---------------------------
+
+The default expiry context on PREEMPT_RT and the role of the ktimersd
+thread are documented in :doc:`/core-api/real-time/differences`
+(Timers).  Those details are not repeated here.
+
+The per-CPU ``ktimers/%u`` thread (referred to as ktimersd in that
+document) runs at the lowest ``SCHED_FIFO`` priority via
+``sched_set_fifo_low()``.  That priority is fixed: the callback does
+not inherit the priority of the task that armed the timer.  A
+``SCHED_FIFO`` task running at priority 99 that starts an unmarked
+timer still expires on ``ktimers/%u`` (lowest ``SCHED_FIFO`` priority),
+not at priority 99.
+Priority inheritance on PREEMPT_RT is used for the cancel handshake,
+not for the arming path; see the "Spin until ready" section of the same
+document.
+
+``hrtimer_setup_sleeper()`` (used by ``clock_nanosleep()`` and similar)
+is an exception: when armed by an RT or DEADLINE task it is marked
+``HRTIMER_MODE_HARD``, so the wakeup runs in hardirq context and does
+not go through ``ktimers/%u``.
+
+If callback work must run at the owning task's RT priority, either
+mark the timer ``HRTIMER_MODE_HARD`` (and keep the callback
+hardirq-safe) or wake a dedicated kthread from the callback.
-- 
2.50.1 (Apple Git-155)

[PATCH v3] docs: timers: hrtimers: clarify expiry modes and ktimersd on PREEMPT_RT

[Liang Hao]

Documentation/timers/hrtimers.rst did not cover the PREEMPT_RT
expiry-mode semantics.  On a PREEMPT_RT kernel a timer that is not
explicitly marked HRTIMER_MODE_HARD is forced into softirq expiry and
its callback runs on the per-CPU ktimers/%u thread at the lowest
SCHED_FIFO priority (sched_set_fifo_low), regardless of the priority of
the task that armed it -- a SCHED_FIFO task running at priority 99 that
starts an unmarked timer still expires on ktimers/%u (lowest SCHED_FIFO
priority), not at priority 99.

Add an "Expiry modes and PREEMPT_RT" section that, rather than
duplicating the default-context description in
Documentation/core-api/real-time/differences.rst (Timers),
cross-references it and focuses on what that document does not spell
out:

 - the callback does not inherit the arming task's priority, and
   priority inheritance on PREEMPT_RT is used for the cancel handshake,
   not the arming path (the "Spin until ready" section of the same
   document);
 - the sleeper exception: hrtimer_setup_sleeper_on_stack() marks
   RT/DL-armed timers HRTIMER_MODE_HARD, so their wakeups do not go
   through ktimers/%u.

Documentation only; no code or behaviour change.

Signed-off-by: Liang Hao <haohlliang@gmail.com>
---
v2 -> v3:
- use hrtimer_setup_sleeper_on_stack(); there is no hrtimer_setup_sleeper()

 Documentation/timers/hrtimers.rst | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/Documentation/timers/hrtimers.rst b/Documentation/timers/hrtimers.rst
index f88ff8bae89c..b81785e0310e 100644
--- a/Documentation/timers/hrtimers.rst
+++ b/Documentation/timers/hrtimers.rst
@@ -171,3 +171,31 @@ hrtimers-based high-resolution clock implementation, so the hrtimers
 code got a healthy amount of testing and use in practice.
 
 	Thomas Gleixner, Ingo Molnar
+
+
+Expiry modes and PREEMPT_RT
+---------------------------
+
+The default expiry context on PREEMPT_RT and the role of the ktimersd
+thread are documented in :doc:`/core-api/real-time/differences`
+(Timers).  Those details are not repeated here.
+
+The per-CPU ``ktimers/%u`` thread (referred to as ktimersd in that
+document) runs at the lowest ``SCHED_FIFO`` priority via
+``sched_set_fifo_low()``.  That priority is fixed: the callback does
+not inherit the priority of the task that armed the timer.  A
+``SCHED_FIFO`` task running at priority 99 that starts an unmarked
+timer still expires on ``ktimers/%u`` (lowest ``SCHED_FIFO`` priority),
+not at priority 99.
+Priority inheritance on PREEMPT_RT is used for the cancel handshake,
+not for the arming path; see the "Spin until ready" section of the same
+document.
+
+``hrtimer_setup_sleeper_on_stack()`` (used by ``clock_nanosleep()`` and similar)
+is an exception: when armed by an RT or DEADLINE task it is marked
+``HRTIMER_MODE_HARD``, so the wakeup runs in hardirq context and does
+not go through ``ktimers/%u``.
+
+If callback work must run at the owning task's RT priority, either
+mark the timer ``HRTIMER_MODE_HARD`` (and keep the callback
+hardirq-safe) or wake a dedicated kthread from the callback.
-- 
2.50.1 (Apple Git-155)

Re: [PATCH v3] docs: timers: hrtimers: clarify expiry modes and ktimersd on PREEMPT_RT

[Sebastian Andrzej Siewior]

On 2026-08-13 22:57:46 [+0800], Liang Hao wrote:
> --- a/Documentation/timers/hrtimers.rst
> +++ b/Documentation/timers/hrtimers.rst
> @@ -171,3 +171,31 @@ hrtimers-based high-resolution clock implementation, so the hrtimers
>  code got a healthy amount of testing and use in practice.
>  
>  	Thomas Gleixner, Ingo Molnar
> +
> +
> +Expiry modes and PREEMPT_RT
> +---------------------------
> +
> +The default expiry context on PREEMPT_RT and the role of the ktimersd
> +thread are documented in :doc:`/core-api/real-time/differences`
> +(Timers).  Those details are not repeated here.

Interesting to say.

> +The per-CPU ``ktimers/%u`` thread (referred to as ktimersd in that

Maybe the other document could be updated so it ktimers everywhere.

> +document) runs at the lowest ``SCHED_FIFO`` priority via
> +``sched_set_fifo_low()``.  That priority is fixed: the callback does
> +not inherit the priority of the task that armed the timer.  A
> +``SCHED_FIFO`` task running at priority 99 that starts an unmarked
> +timer still expires on ``ktimers/%u`` (lowest ``SCHED_FIFO`` priority),
> +not at priority 99.

Right. Why would one expect that to happen?

> +Priority inheritance on PREEMPT_RT is used for the cancel handshake,
> +not for the arming path; see the "Spin until ready" section of the same
> +document.
> +
> +``hrtimer_setup_sleeper_on_stack()`` (used by ``clock_nanosleep()`` and similar)

I wouldn't say similar because what is similar? Does the timeout passed
to select() count as similar? Any of the POSIX timers?

> +is an exception: when armed by an RT or DEADLINE task it is marked
> +``HRTIMER_MODE_HARD``, so the wakeup runs in hardirq context and does
> +not go through ``ktimers/%u``.
> +
> +If callback work must run at the owning task's RT priority, either
> +mark the timer ``HRTIMER_MODE_HARD`` (and keep the callback
> +hardirq-safe) or wake a dedicated kthread from the callback.

I don't think this belongs here. Anything that general hrtimer related
could be added here. The flags parameters such as HRTIMER_MODE_REL,
HRTIMER_MODE_HARD, HRTIMER_MODE_SOFT are only documented in their
kernel-doc of the hrtimer_mode. 

This document does not cover where or in which context the timer
expires. This is also true for timer_list timers. Those are not affected
by this and expire always via ktimers/.

I would suggest that you extend the existing document that you refer to
instead adding RT bits here and refer to the other document. If there is
a need to mention the default context, hrtimer_mode would be the place.

> -- 
> 2.50.1 (Apple Git-155)

Sebastian

[PATCH v4] docs: real-time: mention the hrtimer sleeper HARD path

[Liang Hao]

The Timers section of Documentation/core-api/real-time/differences.rst
describes the PREEMPT_RT default (softirq / ktimers) and HRTIMER_MODE_HARD,
but not the sleeper helper: hrtimer_setup_sleeper_on_stack() marks the
timer HRTIMER_MODE_HARD when the current task is RT or DEADLINE and
HRTIMER_MODE_SOFT was not requested, so the wakeup runs in hard interrupt
context.

Document that behaviour. Also rename "ktimersd" to "ktimers/%u" to match
the per-CPU thread name.

No code or behaviour change.

Signed-off-by: Liang Hao <haohlliang@gmail.com>
---
v3 -> v4:
- drop the Documentation/timers/hrtimers.rst section; extend differences.rst
  instead, per review
- add only the sleeper HARD wording and the ktimers/%u rename
- do not restate lowest-priority / cancel-PI points already covered elsewhere
- avoid vague "and similar" caller lists

 Documentation/core-api/real-time/differences.rst | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/Documentation/core-api/real-time/differences.rst b/Documentation/core-api/real-time/differences.rst
index a129570dab5a..c8acfba051eb 100644
--- a/Documentation/core-api/real-time/differences.rst
+++ b/Documentation/core-api/real-time/differences.rst
@@ -119,12 +119,17 @@ timers initialized with the HRTIMER_MODE_SOFT flag, which are executed in
 softirq context.
 
 On a PREEMPT_RT kernel, this behavior is reversed: hrtimers are executed in
-softirq context by default, typically within the ktimersd thread. This thread
-runs at the lowest real-time priority, ensuring it executes before any
-SCHED_OTHER tasks but does not interfere with higher-priority real-time
+softirq context by default, typically within the per-CPU ktimers/%u thread.
+This thread runs at the lowest real-time priority, ensuring it executes before
+any SCHED_OTHER tasks but does not interfere with higher-priority real-time
 threads. To explicitly request execution in hard interrupt context on
 PREEMPT_RT, the timer must be marked with the HRTIMER_MODE_HARD flag.
 
+hrtimer_setup_sleeper_on_stack() marks the sleeper HRTIMER_MODE_HARD when the
+current task is in a real-time or deadline scheduling class and
+HRTIMER_MODE_SOFT was not requested, so the wakeup runs in hard interrupt
+context rather than on ktimers/%u.
+
 Memory allocation
 -----------------
 
-- 
2.50.1 (Apple Git-155)