All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH RFC] sched/fair: decline WF_SYNC stacking when waker LLC is the busier share
@ 2026-08-04 23:14 Vinicius Costa Gomes
  2026-08-05  2:14 ` Hillf Danton
  0 siblings, 1 reply; 6+ messages in thread
From: Vinicius Costa Gomes @ 2026-08-04 23:14 UTC (permalink / raw)
  To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
	Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
	Valentin Schneider, K Prateek Nayak
  Cc: Chen Yu, Tim Chen, Srikar Dronamraju, Shubhang Kaushik,
	Christoph Lameter, linux-kernel, Vinicius Costa Gomes

Since commit 900bbaae67e9 ("epoll: Add synchronous wakeup support for
ep_poll_callback"), epoll driven WF_SYNC wakeups have been "too
strong" and could cause tasks to stack on a busy NUMA node while other
nodes are relatively idle.

As commit 900bbaae67e9 ("epoll: Add synchronous wakeup support for
ep_poll_callback") improves real workloads a revert is not the answer.
The fix is to make the WF_SYNC "stack on waker" shortcut take into
account the load on this and prev's LLC, rejecting the shortcut only
when the waker (this) LLC is fully loaded and prev's LLC is less
loaded than the waker's.

Signed-off-by: Vinicius Costa Gomes <vinicius.gomes@intel.com>
---
We received a report of a regression on a openresty based
workload (the main metric being tail latencies) on a CWF SNC3 single
socket system, the main symptom that we could measure was one node
being overloaded while the other nodes were relatively idle.

Further investigation showed that spreading the NIC RX interrupts over
all NUMA nodes helped. Reverting commit 900bbaae67e9 ("epoll: Add
synchronous wakeup support for ep_poll_callback") also helped.

Commit 900bbaae67e9 gave the hint, that perhaps for the WF_SYNC wakeup
path the 'sync' flag was too strong and was not taking the load into
account, which for workloads that are too quick for the NUMA load
balancer could perhaps cause this "over stacking".

The idea of this patch is to reject the stacking if it would overload
a LLC. I couldn't find a way to make it NUMA-only and only measured on
NUMA-like machines. One more reason for sending it as a RFC.

(I just became aware of [1], I haven't tested the series, but seems
that it's the same problem but approached from the "other side"; And
from [2], it's interesting, for this workload WF_SYNC is "too strong"
for others, it's too weak. Further documentation about the expected
behavior might be good indeed.)

To try and emulate the workload, which the wakeups happen mostly on
one node, I tried memcached + memtier_benchmark, with memcached
unpinned and memtier pinned to one of the NUMA nodes. As the customer
machine was single socket one, I offlined half the cores of the
machine I have access. (AI helped with this)

Kernels
-------
  base   master fc46aed51f62 (post-v7.2-rc5) vanilla
  rfc    base + sched/fair: decline WF_SYNC stacking when waker LLC is the
         busier share

Results (memcached + memtier_benchmark, 10 repetitions, medians +/- stdev)
---------------------------------------------------------------

Results (memcached + memtier_benchmark, 10 repetitions, mean +/- stddev)
------------------------------------------------------------------------
Client (memtier, the wakers) pinned to one SNC node; memcached worker
threads span the whole socket. Latencies in milliseconds.

Load 64x32 (memtier --threads=64 --clients=32)

  Throughput (higher is better)

      metric       base              rfc               delta
      -------------------------------------------------------------
      ops/sec      3.179M +/- 0.027  3.072M +/- 0.039  -3.4%

  Latency (lower is better)

      pctile       base              rfc               delta
      -------------------------------------------------------------
      p50 (ms)     0.488  +/- 0.006  0.493  +/- 0.013  +1.0%
      p75          0.783  +/- 0.009  0.809  +/- 0.011  +3.3%
      p90          1.265  +/- 0.010  1.312  +/- 0.017  +3.7%
      p99          2.756  +/- 0.026  2.897  +/- 0.028  +5.1%
      p99.9        3.753  +/- 0.064  4.005  +/- 0.117  +6.7%
      p99.99       5.993  +/- 0.308  5.938  +/- 0.322  -0.9%

Load 96x32 (memtier --threads=96 --clients=32)

  Throughput (higher is better)

      metric       base              rfc               delta
      -------------------------------------------------------------
      ops/sec      2.788M +/- 0.028  2.693M +/- 0.040  -3.4%

  Latency (lower is better)

      pctile       base              rfc               delta
      -------------------------------------------------------------
      p50 (ms)     0.773  +/- 0.020  1.109  +/- 0.013  +43.5%
      p75          1.321  +/- 0.013  1.135  +/- 0.016  -14.1%
      p90          2.442  +/- 0.028  1.188  +/- 0.070  -51.4%
      p99          4.857  +/- 0.020  1.974  +/- 0.141  -59.4%
      p99.9        7.145  +/- 0.048  2.722  +/- 0.699  -61.9%
      p99.99       9.637  +/- 0.230  4.324  +/- 0.888  -55.1%

[1] https://lore.kernel.org/all/20260714013940.4068189-4-srikar@linux.ibm.com/

[2] https://lore.kernel.org/all/20260803-b4-sched-sync-wakeup-v4-1-52333b0cfb79@gentwo.org/
---
 kernel/sched/fair.c | 55 +++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 53 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index d78467ec6ee1..c0c6cdaefda4 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -8185,6 +8185,56 @@ static int wake_wide(struct task_struct *p)
 	return 1;
 }
 
+/*
+ * Decline WF_SYNC "stack on waker" wakeup when it would overload this LLC.
+ *
+ * Only decline when prev and this do not share a cache, this LLC has
+ * run out of idle headroom and prev is actually less loaded.
+ *
+ * Returns true when WF_SYNC should be declined.
+ */
+static bool sched_llc_over_commited(int this_cpu, int prev_cpu)
+{
+	struct sched_domain_shared *sds;
+	int this_busy, this_size, prev_busy, prev_size;
+	int headroom;
+
+	/* Same LLC: stacking cannot spread anywhere. */
+	if (cpus_share_cache(this_cpu, prev_cpu))
+		return false;
+
+	sds = rcu_dereference_all(per_cpu(sd_llc_shared, this_cpu));
+	if (!sds)
+		return false;
+	/* FIXME: if NO_HZ nr_busy_cpus is not updated, always accept? */
+	this_busy = atomic_read(&sds->nr_busy_cpus);
+	this_size = per_cpu(sd_llc_size, this_cpu);
+
+	/*
+	 * Decline only once the waker LLC has no spare CPU beyond the
+	 * waker's own, which is going to be replaced by this sync
+	 * wakeup (the caller already made sure that nr_running == 1).
+	 * If at least another idle CPU remains, keep stacking.
+	 */
+	headroom = 1;
+
+	if (this_size - this_busy > headroom)
+		return false;
+
+	sds = rcu_dereference_all(per_cpu(sd_llc_shared, prev_cpu));
+	if (!sds)
+		return false;
+	prev_busy = atomic_read(&sds->nr_busy_cpus);
+	prev_size = per_cpu(sd_llc_size, prev_cpu);
+
+	/*
+	 * Waker LLC is near saturation: spread only if prev is the less
+	 * busy fraction (cross multiplied to take into account different
+	 * LLC sizes).
+	 */
+	return (u64)this_busy * prev_size > (u64)prev_busy * this_size;
+}
+
 /*
  * The purpose of wake_affine() is to quickly determine on which CPU we can run
  * soonest. For the purpose of speed we only consider the waking and previous
@@ -8218,7 +8268,8 @@ wake_affine_idle(int this_cpu, int prev_cpu, int sync)
 	if (sync) {
 		struct rq *rq = cpu_rq(this_cpu);
 
-		if ((rq->nr_running - cfs_h_nr_delayed(rq)) == 1)
+		if ((rq->nr_running - cfs_h_nr_delayed(rq)) == 1 &&
+		    !sched_llc_over_commited(this_cpu, prev_cpu))
 			return this_cpu;
 	}
 
@@ -8237,7 +8288,7 @@ wake_affine_weight(struct sched_domain *sd, struct task_struct *p,
 
 	this_eff_load = cpu_load(cpu_rq(this_cpu));
 
-	if (sync) {
+	if (sync && !sched_llc_over_commited(this_cpu, prev_cpu)) {
 		unsigned long current_load = task_h_load(current);
 
 		if (current_load > this_eff_load)

---
base-commit: c21bb4193868a8de71fc4693fa741e195fdf5d86
change-id: 20260723-epoll-wasync-fix-v1-28d2531033c7

Best regards,
--  
Vinicius


^ permalink raw reply related	[flat|nested] 6+ messages in thread
end of thread, other threads:[~2026-08-07  5:18 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-08-04 23:14 [PATCH RFC] sched/fair: decline WF_SYNC stacking when waker LLC is the busier share Vinicius Costa Gomes
2026-08-05  2:14 ` Hillf Danton
2026-08-06 17:44   ` Vinicius Costa Gomes
2026-08-06 22:53     ` Hillf Danton
2026-08-06 23:22       ` Vinicius Costa Gomes
2026-08-07  5:17         ` Hillf Danton

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.