[PATCH 0/1] sched/psi: skip irqtime accounting when no new irq time has elapsed

[PATCH 0/1] sched/psi: skip irqtime accounting when no new irq time has elapsed

[Usama Arif]

psi_account_irqtime() reads irq_time_read() into a per-rq cumulative
counter and only bails out when the delta vs. the previously accounted
amount is negative.  A delta of exactly zero is treated as "do the
work": psi_write_begin() is taken, cpu_clock(cpu) is read (which on
x86 ends up in native_sched_clock() / rdtsc) and the cgroup ancestor
chain is walked to add zero to every group's PSI_IRQ_FULL bucket.

The zero-delta case is common in practice -- it fires every time a
context switch crosses a PSI group boundary on a CPU that hasn't
serviced an interrupt between the two switches.

To find out how often this actually fires in the wild, I attached a
bpftrace probe to psi_account_irqtime() on a 176-thread AMD EPYC 9D64
server running an compute intensive workload.
The probe also reads irq_time_read(cpu) directly from the per-CPU
cpu_irqtime variable so it can separate delta == 0 from delta < 0.
The bpftrace script was generated by claude and is at the end
of the coverletter.

Over a 30 s window under steady-state load:

  @total              17,229,311    (100.0%)
  @ret_curr_swapper    7,864,195    ( 45.6%)  curr->pid == 0
  @ret_samegrp           323,299    (  1.9%)  same cgroup as prev
  @reached_delta       9,041,817    ( 52.5%)
    @delta_positive    6,358,192    ( 36.9%)  real work
    @delta_zero        2,683,625    ( 15.6%)  work wasted (this patch)
    @delta_negative          (0)    (  0.0%)  monotonic clock

15.6 % of all psi_account_irqtime() calls -- and 29.7 % of the calls
that get past the early returns -- hit the delta == 0 case.  delta < 0
did not occur once in the 30 s window.  That works out to ~89 k calls/sec
on this host that today take the full seqcount write + cpu_clock() +
cgroup-chain walk purely to add 0 to every group's PSI_IRQ_FULL counter.

Extend the early-return to also cover delta == 0.  rq->psi_irq_time
does not need updating in that case (it would store the same value
back) and no PSI bucket would change.  The existing behaviour for
delta > 0 is untouched.

--------- psi_delta_exact.bt -------
#!/usr/bin/env bpftrace

#include <linux/sched.h>

kprobe:psi_account_irqtime
{
	$rq = (struct rq *)arg0;
	$curr = (struct task_struct *)arg1;
	$prev = (struct task_struct *)arg2;

	@total = count();

	if ($curr->pid == 0) {
		@ret_curr_swapper = count();
		return;
	}

	if ($prev != 0) {
		$pg_curr = $curr->cgroups->dfl_cgrp;
		$pg_prev = $prev->cgroups->dfl_cgrp;
		if ($pg_curr == $pg_prev) {
			@ret_samegrp = count();
			return;
		}
	}

	@reached_delta = count();

	$pcpu_off  = *(uint64 *)(kaddr("__per_cpu_offset") + cpu * 8);
	$irq_time  = *(uint64 *)(kaddr("cpu_irqtime") + $pcpu_off);
	$prev_time = $rq->psi_irq_time;
	$delta     = (int64)($irq_time - $prev_time);

	if ($delta == 0) {
		@delta_zero = count();
	} else if ($delta > 0) {
		@delta_positive    = count();
	} else {
		@delta_negative = count();
	}
}

interval:s:30 { exit(); }
--------- end bpftrace ---------
 
Usama Arif (1):
  sched/psi: skip irqtime accounting when no new irq time has elapsed

 kernel/sched/psi.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
2.53.0-Meta

[PATCH 1/1] sched/psi: skip irqtime accounting when no new irq time has elapsed

[Usama Arif]

psi_account_irqtime() reads irq_time_read() into a per-rq cumulative
counter and only bails out when the delta vs. the previously accounted
amount is negative.  A delta of exactly zero is treated as "do the
work": psi_write_begin() is taken, cpu_clock(cpu) is read (which on
x86 ends up in native_sched_clock() / rdtsc) and the cgroup ancestor
chain is walked to add zero to every group's PSI_IRQ_FULL bucket.

The zero-delta case is common in practice -- it fires every time a
context switch crosses a PSI group boundary on a CPU that hasn't
serviced an interrupt between the two switches.

Measured on a 176-thread AMD EPYC 9D64 server running a compute
intensive production workload, instrumented with bpftrace over a 30s
window (irq_time_read() read directly from the per-CPU cpu_irqtime so
that delta == 0 and delta < 0 could be separated):

  @total              17,229,311    (100.0%)
  @ret_curr_swapper    7,864,195    ( 45.6%)  curr->pid == 0
  @ret_samegrp           323,299    (  1.9%)  same cgroup as prev
  @reached_delta       9,041,817    ( 52.5%)
    @delta_positive    6,358,192    ( 36.9%)  real work
    @delta_zero        2,683,625    ( 15.6%)  work wasted (this patch)
    @delta_negative          (0)    (  0.0%)  monotonic clock

So 15.6 % of all psi_account_irqtime() calls - and 29.7 % of the
calls that get past the early returns - hit the delta == 0 case;
delta < 0 did not occur once in the 30 s window.  Under the current
code each of those ~89 k calls per second performs the full seqcount
write + cpu_clock() read + cgroup-chain walk just to add 0 to every
group's PSI_IRQ_FULL counter.

Extend the early-return to also cover delta == 0.  rq->psi_irq_time
does not need updating in that case (it would store the same value
back) and no PSI bucket would change.  The existing behaviour for
delta > 0 is untouched.

Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
 kernel/sched/psi.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/sched/psi.c b/kernel/sched/psi.c
index d9c9d9480a45..848955f8893d 100644
--- a/kernel/sched/psi.c
+++ b/kernel/sched/psi.c
@@ -1023,7 +1023,7 @@ void psi_account_irqtime(struct rq *rq, struct task_struct *curr, struct task_st
 
 	irq = irq_time_read(cpu);
 	delta = (s64)(irq - rq->psi_irq_time);
-	if (delta < 0)
+	if (delta <= 0)
 		return;
 	rq->psi_irq_time = irq;
 
-- 
2.53.0-Meta