[PATCH] block: invalidate cached plug timestamp after task switch

[PATCH] block: invalidate cached plug timestamp after task switch

[Usama Arif]

blk_time_get_ns() caches ktime_get_ns() in current->plug->cur_ktime
and marks the task with PF_BLOCK_TS. That cache is only valid while the
task keeps running; if the task is switched out, wall-clock time
advances and the cached value must not be reused when the task runs again.

The existing invalidation covers explicit plug flushes through
__blk_flush_plug(), and the schedule() / rtmutex paths through
sched_update_worker(). It does not cover in-kernel preemption paths such
as preempt_schedule(), preempt_schedule_notrace(), and
preempt_schedule_irq(), which enter __schedule(SM_PREEMPT) directly and
return without calling sched_update_worker().

As a result, a task preempted while holding a plug with PF_BLOCK_TS set
can reuse a stale plug->cur_ktime after it is scheduled back in. blk-iocost
then consumes that stale timestamp through ioc_now(), producing stale vnow
values for throttle decisions, and through ioc_rqos_done(), inflating
on-queue time and feeding false missed-QoS samples into vrate
adjustment.

Move the schedule-side invalidation to finish_task_switch(), which runs
for the scheduled-in task after every actual context switch regardless
of which schedule entry point was used. Keep __blk_flush_plug() as the
explicit flush/finish-plug invalidation path, and remove only the
PF_BLOCK_TS handling from sched_update_worker().

Fixes: 06b23f92af87 ("block: update cached timestamp post schedule/preemption")
Cc: stable@vger.kernel.org
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
 kernel/sched/core.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 8b791e9e9f67..bf024ca115ff 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -5368,6 +5368,13 @@ static struct rq *finish_task_switch(struct task_struct *prev)
 	 */
 	kmap_local_sched_in();
 
+	/*
+	 * Any cached block-layer timestamp (plug->cur_ktime) is stale now,
+	 * invalidate it.
+	 */
+	if (unlikely(current->flags & PF_BLOCK_TS))
+		blk_plug_invalidate_ts(current);
+
 	fire_sched_in_preempt_notifiers(current);
 	/*
 	 * When switching through a kernel thread, the loop in
@@ -7290,12 +7297,10 @@ static inline void sched_submit_work(struct task_struct *tsk)
 
 static void sched_update_worker(struct task_struct *tsk)
 {
-	if (tsk->flags & (PF_WQ_WORKER | PF_IO_WORKER | PF_BLOCK_TS)) {
-		if (tsk->flags & PF_BLOCK_TS)
-			blk_plug_invalidate_ts(tsk);
+	if (tsk->flags & (PF_WQ_WORKER | PF_IO_WORKER)) {
 		if (tsk->flags & PF_WQ_WORKER)
 			wq_worker_running(tsk);
-		else if (tsk->flags & PF_IO_WORKER)
+		else
 			io_wq_worker_running(tsk);
 	}
 }
-- 
2.53.0-Meta

Re: [PATCH] block: invalidate cached plug timestamp after task switch

[Peter Zijlstra]

On Thu, Jun 11, 2026 at 04:14:28PM -0700, Usama Arif wrote:
> blk_time_get_ns() caches ktime_get_ns() in current->plug->cur_ktime
> and marks the task with PF_BLOCK_TS. That cache is only valid while the
> task keeps running; if the task is switched out, wall-clock time
> advances and the cached value must not be reused when the task runs again.
> 
> The existing invalidation covers explicit plug flushes through
> __blk_flush_plug(), and the schedule() / rtmutex paths through
> sched_update_worker(). It does not cover in-kernel preemption paths such
> as preempt_schedule(), preempt_schedule_notrace(), and
> preempt_schedule_irq(), which enter __schedule(SM_PREEMPT) directly and
> return without calling sched_update_worker().
> 
> As a result, a task preempted while holding a plug with PF_BLOCK_TS set
> can reuse a stale plug->cur_ktime after it is scheduled back in. blk-iocost
> then consumes that stale timestamp through ioc_now(), producing stale vnow
> values for throttle decisions, and through ioc_rqos_done(), inflating
> on-queue time and feeding false missed-QoS samples into vrate
> adjustment.
> 
> Move the schedule-side invalidation to finish_task_switch(), which runs
> for the scheduled-in task after every actual context switch regardless
> of which schedule entry point was used. Keep __blk_flush_plug() as the
> explicit flush/finish-plug invalidation path, and remove only the
> PF_BLOCK_TS handling from sched_update_worker().
> 
> Fixes: 06b23f92af87 ("block: update cached timestamp post schedule/preemption")
> Cc: stable@vger.kernel.org
> Signed-off-by: Usama Arif <usama.arif@linux.dev>
> ---
>  kernel/sched/core.c | 13 +++++++++----
>  1 file changed, 9 insertions(+), 4 deletions(-)
> 
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 8b791e9e9f67..bf024ca115ff 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -5368,6 +5368,13 @@ static struct rq *finish_task_switch(struct task_struct *prev)
>  	 */
>  	kmap_local_sched_in();
>  
> +	/*
> +	 * Any cached block-layer timestamp (plug->cur_ktime) is stale now,
> +	 * invalidate it.
> +	 */
> +	if (unlikely(current->flags & PF_BLOCK_TS))
> +		blk_plug_invalidate_ts(current);

Can you make that just blk_plug_invalidate_ts() and move the branch into
the function itself, which is already inline anyway (but perhaps upgrade
it to __always_inline).