[PATCH v2] block: invalidate cached plug timestamp after task switch

[PATCH v2] block: invalidate cached plug timestamp after task switch

[Usama Arif]

blk_time_get_ns() caches ktime_get_ns() in current->plug->cur_ktime
and marks the task with PF_BLOCK_TS. That cache is only valid while the
task keeps running; if the task is switched out, wall-clock time
advances and the cached value must not be reused when the task runs again.

The existing invalidation covers explicit plug flushes through
__blk_flush_plug(), and the schedule() / rtmutex paths through
sched_update_worker(). It does not cover in-kernel preemption paths such
as preempt_schedule(), preempt_schedule_notrace(), and
preempt_schedule_irq(), which enter __schedule(SM_PREEMPT) directly and
return without calling sched_update_worker().

As a result, a task preempted while holding a plug with PF_BLOCK_TS set
can reuse a stale plug->cur_ktime after it is scheduled back in. blk-iocost
then consumes that stale timestamp through ioc_now(), producing stale vnow
values for throttle decisions, and through ioc_rqos_done(), inflating
on-queue time and feeding false missed-QoS samples into vrate
adjustment.

Move the schedule-side invalidation to finish_task_switch(), which runs
for the scheduled-in task after every actual context switch regardless
of which schedule entry point was used. Keep __blk_flush_plug() as the
explicit flush/finish-plug invalidation path, and remove only the
PF_BLOCK_TS handling from sched_update_worker().

Fixes: 06b23f92af87 ("block: update cached timestamp post schedule/preemption")
Cc: stable@vger.kernel.org
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
v1 -> v2: https://lore.kernel.org/all/20260611231428.345098-1-usama.arif@linux.dev/
- Make the function just blk_plug_invalidate_ts(), move the check for
PF_BLOCK_TS flag into blk_plug_invalidate_ts and make it __always_inline
(Peter Zijlstra).
---
 include/linux/blkdev.h | 17 ++++++++---------
 kernel/sched/core.c    | 12 ++++++++----
 2 files changed, 16 insertions(+), 13 deletions(-)

diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 57e84d59a642..1c1fd31ce187 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -1216,16 +1216,15 @@ static inline void blk_flush_plug(struct blk_plug *plug, bool async)
 		__blk_flush_plug(plug, async);
 }
 
-/*
- * tsk == current here
- */
-static inline void blk_plug_invalidate_ts(struct task_struct *tsk)
+static __always_inline void blk_plug_invalidate_ts(void)
 {
-	struct blk_plug *plug = tsk->plug;
+	if (unlikely(current->flags & PF_BLOCK_TS)) {
+		struct blk_plug *plug = current->plug;
 
-	if (plug)
-		plug->cur_ktime = 0;
-	current->flags &= ~PF_BLOCK_TS;
+		if (plug)
+			plug->cur_ktime = 0;
+		current->flags &= ~PF_BLOCK_TS;
+	}
 }
 
 int blkdev_issue_flush(struct block_device *bdev);
@@ -1251,7 +1250,7 @@ static inline void blk_flush_plug(struct blk_plug *plug, bool async)
 {
 }
 
-static inline void blk_plug_invalidate_ts(struct task_struct *tsk)
+static inline void blk_plug_invalidate_ts(void)
 {
 }
 
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 8b791e9e9f67..e97e98c33be5 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -5368,6 +5368,12 @@ static struct rq *finish_task_switch(struct task_struct *prev)
 	 */
 	kmap_local_sched_in();
 
+	/*
+	 * Any cached block-layer timestamp (plug->cur_ktime) is stale now,
+	 * invalidate it.
+	 */
+	blk_plug_invalidate_ts();
+
 	fire_sched_in_preempt_notifiers(current);
 	/*
 	 * When switching through a kernel thread, the loop in
@@ -7290,12 +7296,10 @@ static inline void sched_submit_work(struct task_struct *tsk)
 
 static void sched_update_worker(struct task_struct *tsk)
 {
-	if (tsk->flags & (PF_WQ_WORKER | PF_IO_WORKER | PF_BLOCK_TS)) {
-		if (tsk->flags & PF_BLOCK_TS)
-			blk_plug_invalidate_ts(tsk);
+	if (tsk->flags & (PF_WQ_WORKER | PF_IO_WORKER)) {
 		if (tsk->flags & PF_WQ_WORKER)
 			wq_worker_running(tsk);
-		else if (tsk->flags & PF_IO_WORKER)
+		else
 			io_wq_worker_running(tsk);
 	}
 }
-- 
2.53.0-Meta

Re: [PATCH v2] block: invalidate cached plug timestamp after task switch

[Peter Zijlstra]

On Fri, Jun 12, 2026 at 02:40:42AM -0700, Usama Arif wrote:

> +static __always_inline void blk_plug_invalidate_ts(void)
>  {
> +	if (unlikely(current->flags & PF_BLOCK_TS)) {
> +		struct blk_plug *plug = current->plug;
>  
> +		if (plug)
> +			plug->cur_ktime = 0;
> +		current->flags &= ~PF_BLOCK_TS;
> +	}
>  }

If you can guarantee PF_BLOCK_TS is only ever set when current->plug,
this can be reduced further.

Re: [PATCH v2] block: invalidate cached plug timestamp after task switch

[Usama Arif]

On 12/06/2026 10:45, Peter Zijlstra wrote:
> On Fri, Jun 12, 2026 at 02:40:42AM -0700, Usama Arif wrote:
> 
>> +static __always_inline void blk_plug_invalidate_ts(void)
>>  {
>> +	if (unlikely(current->flags & PF_BLOCK_TS)) {
>> +		struct blk_plug *plug = current->plug;
>>  
>> +		if (plug)
>> +			plug->cur_ktime = 0;
>> +		current->flags &= ~PF_BLOCK_TS;
>> +	}
>>  }
> 
> If you can guarantee PF_BLOCK_TS is only ever set when current->plug,
> this can be reduced further.

Thanks for the reviews!

The invariant holds at set time (the only set in blk_time_get_ns() is
gated by if (!plug)) and through the only legitimate plug clear in
blk_finish_plug() (which goes through __blk_flush_plug() that clears
PF_BLOCK_TS first).

However, copy_process() sets p->plug = NULL for the child but doesn't
strip PF_BLOCK_TS from the inherited flags.

I think the if(plug) is a good defensive check, but can also do the below
if you prefer?

diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 1c1fd31ce187..c285a4d9837d 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -1219,10 +1219,7 @@ static inline void blk_flush_plug(struct blk_plug *plug, bool async)
 static __always_inline void blk_plug_invalidate_ts(void)
 {
        if (unlikely(current->flags & PF_BLOCK_TS)) {
-               struct blk_plug *plug = current->plug;
-
-               if (plug)
-                       plug->cur_ktime = 0;
+               current->plug->cur_ktime = 0;
                current->flags &= ~PF_BLOCK_TS;
        }
 }
diff --git a/kernel/fork.c b/kernel/fork.c
index 892a95214c54..9a062149e0d8 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -2167,7 +2167,8 @@ __latent_entropy struct task_struct *copy_process(
                goto bad_fork_cleanup_count;
 
        delayacct_tsk_init(p);  /* Must remain after dup_task_struct() */
-       p->flags &= ~(PF_SUPERPRIV | PF_WQ_WORKER | PF_IDLE | PF_NO_SETAFFINITY);
+       p->flags &= ~(PF_SUPERPRIV | PF_WQ_WORKER | PF_IDLE | PF_NO_SETAFFINITY |
+                     PF_BLOCK_TS);
        p->flags |= PF_FORKNOEXEC;
        INIT_LIST_HEAD(&p->children);
        INIT_LIST_HEAD(&p->sibling);