From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A072232E6BD for ; Tue, 4 Aug 2026 23:14:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.17 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785885266; cv=none; b=m3Y5lJW/b0RyHAEXwiWfcZhpZ2uSMANbktXW6NJ9VpmQbmeJChsxwmTvMMrwCV9DkkpNRz3w+roZ8oiLXo3CyEPjprDjwPIb79giDN1Cm9Cmj+XHwG9Y/iIDn48ORrTBkzTb2W+zS6VJkG5B9KzBHEXKCkpv09Z/j5io/DwK3Vc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785885266; c=relaxed/simple; bh=JmiNLominLQiLT0xY4gUs51V0DRYFpSgcSbpSBhEKV0=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:To:Cc; b=Wuwj/EPngOm5KQ0Bm4t+6M1qslr1GvySwPKH+YCSYeBHnYQfr4ogKQ7sNF8aYxb1YWQGU5cSpSX4DAUv6neNtoRPgOYrpQGQb/wSlXQF92iutsK9ztYsBsR+4RPvH6Dg5gkYNPMhas1L8dwrjLL/b6Zm3wZT+jETKYQz0sVz3q0= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=IFE08jNP; arc=none smtp.client-ip=198.175.65.17 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="IFE08jNP" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1785885263; x=1817421263; h=from:date:subject:mime-version:content-transfer-encoding: message-id:to:cc; bh=JmiNLominLQiLT0xY4gUs51V0DRYFpSgcSbpSBhEKV0=; b=IFE08jNPCjwec4FxUm9z+JKfvlZk78yiL0/LWhFJPfyV1rJ4FhvIXPkS yX4VGGxiYR1NZ6YYymLCRlouu7n+7742+T6V9aMUePKKNlRr7vCKKjKhb IAswkPYn+QpRwdBxsUSxcS1R4vlFpN2zAndvgkymhKjK+h5KsNs2fYXE+ 3SxCXS9AC6w8Fmyz2wBeyWH6Br68ShhtWutVYjhDD4ZjTNNr/J6TJHXtD fLPehF3CL25iW0+TZbww5z1Coj8XuldIRb5B2kdgndPHN/NvMW7uUwA1a PeqVqeCw+x2tzP97iqEaigXrANUr8Rt2GgCtxRos5WdCKHwZh9N9OnnHp Q==; X-CSE-ConnectionGUID: zIyz5WS4RnOoRhMABFHqRg== X-CSE-MsgGUID: R9WTKUQiRQ60WUTQ6Tspsg== X-IronPort-AV: E=McAfee;i="6800,10657,11865"; a="86455544" X-IronPort-AV: E=Sophos;i="6.25,205,1779174000"; d="scan'208";a="86455544" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by orvoesa109.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Aug 2026 16:14:23 -0700 X-CSE-ConnectionGUID: BMv9Mb/KRZW1Svk5Am6xFA== X-CSE-MsgGUID: onyAsN7+SCKyzP5xZ7GqHQ== X-ExtLoop1: 1 Received: from vcostago-desk1.jf.intel.com (HELO [10.88.27.144]) ([10.88.27.144]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Aug 2026 16:14:22 -0700 From: Vinicius Costa Gomes Date: Tue, 04 Aug 2026 16:14:05 -0700 Subject: [PATCH RFC] sched/fair: decline WF_SYNC stacking when waker LLC is the busier share Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20260804-epoll-wasync-fix-v1-v1-1-a83b4c8e08ba@intel.com> X-B4-Tracking: v=1; b=H4sIAAAAAAAC/6tWKk4tykwtVrJSqFYqSi3LLM7MzwNyDHUUlJIzE vPSU3UzU4B8JSMDIzMDcyNj3dSC/Jwc3fLE4sq8ZN20zArdMkNdI4sUI1NjQwNj42RzJaDOgqJ UoAzY1GilIDdnpViIYHFpUlZqcgnIPKXaWgCPKF0dfAAAAA== X-Change-ID: 20260723-epoll-wasync-fix-v1-28d2531033c7 To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak Cc: Chen Yu , Tim Chen , Srikar Dronamraju , Shubhang Kaushik , Christoph Lameter , linux-kernel@vger.kernel.org, Vinicius Costa Gomes X-Mailer: b4 0.16-dev-4217c X-Developer-Signature: v=1; a=ed25519-sha256; t=1785885262; l=7621; i=vinicius.gomes@intel.com; s=20230921; h=from:subject:message-id; bh=JmiNLominLQiLT0xY4gUs51V0DRYFpSgcSbpSBhEKV0=; b=Sde9ZO4wOy2mx7avdpqd+Zx22SUbfq6PijTUbhifr5So3tGOl2o6L/Ml1ms1zArLKfcXwd5YM 54APq9HyZWJC6qBO9gOMaicBOaya1dkRPHMhQq6i23S2rDI2kep3v7d X-Developer-Key: i=vinicius.gomes@intel.com; a=ed25519; pk=aJkrtgqgT6TZ8iIHSG8/rTPsmlYnjMrUjCsMYvCzntk= Since commit 900bbaae67e9 ("epoll: Add synchronous wakeup support for ep_poll_callback"), epoll driven WF_SYNC wakeups have been "too strong" and could cause tasks to stack on a busy NUMA node while other nodes are relatively idle. As commit 900bbaae67e9 ("epoll: Add synchronous wakeup support for ep_poll_callback") improves real workloads a revert is not the answer. The fix is to make the WF_SYNC "stack on waker" shortcut take into account the load on this and prev's LLC, rejecting the shortcut only when the waker (this) LLC is fully loaded and prev's LLC is less loaded than the waker's. Signed-off-by: Vinicius Costa Gomes --- We received a report of a regression on a openresty based workload (the main metric being tail latencies) on a CWF SNC3 single socket system, the main symptom that we could measure was one node being overloaded while the other nodes were relatively idle. Further investigation showed that spreading the NIC RX interrupts over all NUMA nodes helped. Reverting commit 900bbaae67e9 ("epoll: Add synchronous wakeup support for ep_poll_callback") also helped. Commit 900bbaae67e9 gave the hint, that perhaps for the WF_SYNC wakeup path the 'sync' flag was too strong and was not taking the load into account, which for workloads that are too quick for the NUMA load balancer could perhaps cause this "over stacking". The idea of this patch is to reject the stacking if it would overload a LLC. I couldn't find a way to make it NUMA-only and only measured on NUMA-like machines. One more reason for sending it as a RFC. (I just became aware of [1], I haven't tested the series, but seems that it's the same problem but approached from the "other side"; And from [2], it's interesting, for this workload WF_SYNC is "too strong" for others, it's too weak. Further documentation about the expected behavior might be good indeed.) To try and emulate the workload, which the wakeups happen mostly on one node, I tried memcached + memtier_benchmark, with memcached unpinned and memtier pinned to one of the NUMA nodes. As the customer machine was single socket one, I offlined half the cores of the machine I have access. (AI helped with this) Kernels ------- base master fc46aed51f62 (post-v7.2-rc5) vanilla rfc base + sched/fair: decline WF_SYNC stacking when waker LLC is the busier share Results (memcached + memtier_benchmark, 10 repetitions, medians +/- stdev) --------------------------------------------------------------- Results (memcached + memtier_benchmark, 10 repetitions, mean +/- stddev) ------------------------------------------------------------------------ Client (memtier, the wakers) pinned to one SNC node; memcached worker threads span the whole socket. Latencies in milliseconds. Load 64x32 (memtier --threads=64 --clients=32) Throughput (higher is better) metric base rfc delta ------------------------------------------------------------- ops/sec 3.179M +/- 0.027 3.072M +/- 0.039 -3.4% Latency (lower is better) pctile base rfc delta ------------------------------------------------------------- p50 (ms) 0.488 +/- 0.006 0.493 +/- 0.013 +1.0% p75 0.783 +/- 0.009 0.809 +/- 0.011 +3.3% p90 1.265 +/- 0.010 1.312 +/- 0.017 +3.7% p99 2.756 +/- 0.026 2.897 +/- 0.028 +5.1% p99.9 3.753 +/- 0.064 4.005 +/- 0.117 +6.7% p99.99 5.993 +/- 0.308 5.938 +/- 0.322 -0.9% Load 96x32 (memtier --threads=96 --clients=32) Throughput (higher is better) metric base rfc delta ------------------------------------------------------------- ops/sec 2.788M +/- 0.028 2.693M +/- 0.040 -3.4% Latency (lower is better) pctile base rfc delta ------------------------------------------------------------- p50 (ms) 0.773 +/- 0.020 1.109 +/- 0.013 +43.5% p75 1.321 +/- 0.013 1.135 +/- 0.016 -14.1% p90 2.442 +/- 0.028 1.188 +/- 0.070 -51.4% p99 4.857 +/- 0.020 1.974 +/- 0.141 -59.4% p99.9 7.145 +/- 0.048 2.722 +/- 0.699 -61.9% p99.99 9.637 +/- 0.230 4.324 +/- 0.888 -55.1% [1] https://lore.kernel.org/all/20260714013940.4068189-4-srikar@linux.ibm.com/ [2] https://lore.kernel.org/all/20260803-b4-sched-sync-wakeup-v4-1-52333b0cfb79@gentwo.org/ --- kernel/sched/fair.c | 55 +++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 53 insertions(+), 2 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index d78467ec6ee1..c0c6cdaefda4 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -8185,6 +8185,56 @@ static int wake_wide(struct task_struct *p) return 1; } +/* + * Decline WF_SYNC "stack on waker" wakeup when it would overload this LLC. + * + * Only decline when prev and this do not share a cache, this LLC has + * run out of idle headroom and prev is actually less loaded. + * + * Returns true when WF_SYNC should be declined. + */ +static bool sched_llc_over_commited(int this_cpu, int prev_cpu) +{ + struct sched_domain_shared *sds; + int this_busy, this_size, prev_busy, prev_size; + int headroom; + + /* Same LLC: stacking cannot spread anywhere. */ + if (cpus_share_cache(this_cpu, prev_cpu)) + return false; + + sds = rcu_dereference_all(per_cpu(sd_llc_shared, this_cpu)); + if (!sds) + return false; + /* FIXME: if NO_HZ nr_busy_cpus is not updated, always accept? */ + this_busy = atomic_read(&sds->nr_busy_cpus); + this_size = per_cpu(sd_llc_size, this_cpu); + + /* + * Decline only once the waker LLC has no spare CPU beyond the + * waker's own, which is going to be replaced by this sync + * wakeup (the caller already made sure that nr_running == 1). + * If at least another idle CPU remains, keep stacking. + */ + headroom = 1; + + if (this_size - this_busy > headroom) + return false; + + sds = rcu_dereference_all(per_cpu(sd_llc_shared, prev_cpu)); + if (!sds) + return false; + prev_busy = atomic_read(&sds->nr_busy_cpus); + prev_size = per_cpu(sd_llc_size, prev_cpu); + + /* + * Waker LLC is near saturation: spread only if prev is the less + * busy fraction (cross multiplied to take into account different + * LLC sizes). + */ + return (u64)this_busy * prev_size > (u64)prev_busy * this_size; +} + /* * The purpose of wake_affine() is to quickly determine on which CPU we can run * soonest. For the purpose of speed we only consider the waking and previous @@ -8218,7 +8268,8 @@ wake_affine_idle(int this_cpu, int prev_cpu, int sync) if (sync) { struct rq *rq = cpu_rq(this_cpu); - if ((rq->nr_running - cfs_h_nr_delayed(rq)) == 1) + if ((rq->nr_running - cfs_h_nr_delayed(rq)) == 1 && + !sched_llc_over_commited(this_cpu, prev_cpu)) return this_cpu; } @@ -8237,7 +8288,7 @@ wake_affine_weight(struct sched_domain *sd, struct task_struct *p, this_eff_load = cpu_load(cpu_rq(this_cpu)); - if (sync) { + if (sync && !sched_llc_over_commited(this_cpu, prev_cpu)) { unsigned long current_load = task_h_load(current); if (current_load > this_eff_load) --- base-commit: c21bb4193868a8de71fc4693fa741e195fdf5d86 change-id: 20260723-epoll-wasync-fix-v1-28d2531033c7 Best regards, -- Vinicius