Re: [RFC] cpuidle: runtime exit latency prediction and timer compensation

[Christian Loehle <christian.loehle@arm.com>]

On 10/16/26 11:41, Cong Zhang wrote:
> Hi all,
> 
> 1. Abstract
> -----------
> 
> This RFC proposes a way to measure and predict CPU idle exit latency at run
> time in the kernel. It also discusses how to use the prediction to reduce the
> delay between the expected hrtimer expiry time, called hrtimer rooster timer
> solution.
> 
> This idea was presented at the 2026 Linux Plumbers Conference [1]:
> [1]https://lpc.events/event/20/contributions/2595/
> We would like to continue the discussion of this approach here in this RFC.
> 
> 2. Background
> ------------
> 
> An hrtimer callback can run later than hrtimer next event, the expected expiry
> time. This is a known behavior in the kernel. Hrtimer expiry only means that
> the IRQ will be generated at the corresponding time. The callback still waits
> for wakeup, resume, and the rest of the timer IRQ handling path. This can
> delay its execution.
> 
> When an hrtimer expires while a CPU is idle, the timer IRQ starts waking the
> CPU. The hardware resumes first. The firmware resumes next. The kernel can
> start running after that. We call the period start from receiving the IRQ to
> the entry of kernel execution as idle exit latency. Different CPU
> architectures and platforms have this kind of idle exit latency.
> 
> The kernel currently uses a static idle exit latency from the device tree.
> This value describes the worst case. It does not directly show the idle exit
> latency under current conditions. A more accurate idle exit latency estimation
> will cause less confusion and less power impact in real use cases. A similar
> upstream discussion also points out that using a static worst-case value to
> estimate typical latency can cause errors [2]:
> [2]https://lore.kernel.org/all/44c4ee5a-34ec-4b23-b06b-05bd0fda6585@arm.com/
> 
> Other related upstream discussions cover genpd CPU latency QoS [3]:
> [3]https://lore.kernel.org/all/20250709-pmdomain_qos-v2-1-976b12257899@oss.qualcomm.com/
> and IRQ timing infrastructure removal [4]:
> [4]https://lore.kernel.org/all/20251210082242.360936-2-maz@kernel.org/
> 
> This RFC therefore discusses how to measure and predict idle exit latency in
> the Kernel context, and how the prediction may be used for hrtimer handle
> latency accommodation and other kernel Decisions needed to consider the idle
> exit latency.
> 
> 3. Runtime measurement and prediction
> -------------------------------------
> 
> Different than other interrupts, timer interrupts will have the info of
> expected time to be triggered. So, We assume the hrtimer expire time and it’s
> handle latency can be used for specific idle exit latency runtime measurement.
> We identify the cpu idle level, different cpu, cluster idle level is the key
> difference of the idle exit latency variation. This section describes how the
> kernel collects runtime samples and gets a typical exit latency under current
> conditions. The collection step records valid samples. Prediction uses a
> sample window to make a dynamic estimate for each condition.
> 
> 3.1 CPU sample collection
> -------------------------
> 
> When the CPU enters idle, record the CPU idle level it actually enters and the
> current programmed hrtimer next event. The idle exit path records when the CPU
> is awake and running again. This point is before any IRQ handler or hrtimer
> handler.
> 
> In this RFC, observed exit latency is defined as the time when the CPU is exit
> from idle in kernel idle driver minus the programmed hrtimer expire time:
> 
>     cpu idle exit time stamp - programmed hrtimer expired time
> 
> Keep separate records for each CPU ID and idle level.
> 
> There is much noise that will affect this period, for example other
> unpredicted interrupt coming at the same time. We only take the measured and
> expected time into consideration. A sample is valid only when all these
> conditions are held:
> 
>   - The target CPU entered the target idle level.
>   - The target hrtimer has expired.
>   - The target CPU had only the target timer IRQ pending.
> 
> These conditions make sure that the recorded resume time belongs to the target
> hrtimer, not to an early wakeup caused by another interrupt. Samples that fail
> these conditions are marked invalid and excluded from exit latency statistics.
> 
> 3.2 Cluster exit latency
> ------------------------
> 
> A single CPU chooses the CPU idle level, while multiple CPUs choose the
> cluster idle level together. When the first CPU wakes, the resume process
> includes the resume time for the whole cluster. We use the exit latency of the
> first woken CPU as the cluster exit latency.
> 
> The cluster case uses the same measurement method as the CPU case. The last
> CPU that makes the cluster enter sleep records the cluster ID and cluster idle
> level for this entry. At exit, the first CPU in the cluster resumes after
> wakeup represents the sample.
> 
> The validity conditions for a cluster sample are similar to those for a CPU
> sample:
> 
>   - The cluster entered the recorded idle level.
>   - The target hrtimer has expired.
>   - The target CPU had only the target timer IRQ pending.
>   - Other CPUs in the cluster had no pending IRQ except timer IRQ.
> 
> Mark samples that fail these conditions as invalid. Exclude them from cluster
> exit latency statistics and prediction.
> 
> 3.3 Why measure in the kernel
> -----------------------------
> 
> A user-space program can use a periodic timer to measure the time from hrtimer
> next event until the thread runs again. This time does not accurately
> represent CPU exit latency for these reasons:
> 
>   - The time when the thread runs again includes scheduler and user-space path
>     overhead. It cannot show the exact time when the CPU resumed after wakeup.
>   - User space usually does not know which CPU idle level the CPU entered, or
>     which cluster idle level the whole cluster entered.
>   - User space cannot reliably tell whether the target hrtimer, an IPI, a
>     device
>     IRQ, or another pending interrupt woke the CPU.
>   - To send test results back to the kernel in real time, user space would
>     need
>     an extra user-to-kernel interface. The results cannot then be sent
>     directly and continuously to a running cpuidle governor, or used directly
>     for hardware timer programming and compensation.
> 
> 3.4 Prediction from historical samples
> ---------------------------------------
> 
> For current experiment, we use a simple way to do the prediction to the valid
> history samples. The predictor keeps a separate fixed-length window of recent
> samples for every CPU and idle level. It keeps another window for every
> cluster and cluster idle level. The current prediction is the average of the
> samples in the window.
> 
> 4. Possible use cases
> --------------------
> The “more accurate” idle exit latency can be used in different areas. It can
> replace the current static idle exit latency used in current kernel
> components. And One another possible use is hrtimer compensation. We will take
> this as a specific example.
> 
> 4.1 hrtimer compensation
> ------------------------
> 
> Idle Exit latency is one important part of the delay before an hrtimer
> callback. When cpu is about to go to idle, we can predict the possible current
> idle exit latency and do compensate. the prediction can set the hardware
> wakeup time before hrtimer next event. This can compensate for the delay
> caused by idle exit latency.
> 
> We call this early wakeup interrupt Rooster IRQ. Its lead time covers the time
> needed for hardware and firmware to resume. Rooster IRQ fires before the
> original hrtimer expiry time, so the CPU can resume close to the expected
> time. It does not change the software hrtimer expiry time or run the hrtimer
> callback early.
> 
> Rooster IRQ only adds a new hardware timer used for wakeup. This can use The
> cpu local arch timer and a global/broadcast timer. The rooster timer is set
> before hrtimer next expire time.
> 
> In the CPU case, use the prediction for the CPU and idle level to set that
> CPU's Local timer. The local timer wakes the CPU early.
> 
> In the cluster case, a platform may have several global timer frames. A
> separate frame can be assigned to each cluster that needs an early wakeup. The
> last CPU that makes the cluster enter sleep sets a rooster cluser timer. It
> uses the prediction for the cluster idle level to set the rooster wakeup time.
> The rooster cluster timer directly wakes the first CPU in the target cluster
> that needs to resume.
> 
> If the platform has only one global timer, or if the local timer stops at the
> target idle level, a global/broadcast timer can wake an intermediate CPU
> first. That CPU then sends the wakeup interrupt to the target CPU through an
> IPI. The rooster timer lead time must cover the exit latency of the
> intermediate CPU's cluster and the target CPU or cluster.
> 
> 5. Prototype status and results
> ------------------------------
> 
> The measurement and prediction mechanism has been validated with real hardware
> on Qualcomm's platform.
> 
> The LPC slides show sample results over a period. The sampled exit latency
> stays fairly stable during that period. A prediction based on recent sample
> averages also follows changes in the samples reasonably well.
> 
> On device, we enabled compensation with one global timer and broadcast. We
> measured the difference between the actual CPU resume time and the expected
> hrtimer expiry time. t1 is the time when the CPU resumes kernel execution. t0
> is the original hrtimer expiry time. In the table, "Before compensation" is
> the reference with the same definition. "After compensation" is the result
> with this compensation path enabled. The unit is us:
> 
>     Metric: mean absolute value of t1 - t0, us
>     State               | CPU C4 | Cluster CL4 | Cluster CL5
>     -------------------+--------+-------------+------------
>     Before compensation | 19.824 | 61.991      | 1929.767
>     After compensation  |  0.073 |  0.191      |   10.738
> 
> 6. Discussion
> -------------
> 
> The prototype code is still being cleaned up. We would like to discuss the
> design and get community feedback before we finalize the code. The feedback
> will guide the next implementation direction.
> 

So a few thoughts that are still in my head from the LPC talk:

1. Existing hardware mitigations.
For example Intel accomplishes this with a hardware feature called
C-state pre-wake, how will these co-exist?

2. What about platforms without OS-coordinated cluster states?
What about the platform influencing idle states more generally?
(In particular with platform-coordinated cluster-off you should
see two range of latencies, one which was true cluster-off and
the other was unrealised cluster-off that is really just CPU off).

3. Does any workload actually care? What's the performance improvement?
How does it affect power? Especially w.r.t existing latency constaints

I have a bunch of methodological questions about the sampling, but
presumably they would all be answered by the code.
(Are we still sampling while the mitigation is applied? If so how is
exit latency observed distinguished from prediction error? What if the
first waking CPU is not the cluster-wake-initiating? What if the
CPU wakes up too early because of the mitigation? (if this enters the
idle loop again, this influences the governor massively of course).)

Also if "replace the current static idle exit latency" actually means
for the QoS constaints too, then I think we shouldn't do this, it's
deliberately a worst-case.