* [PATCH RFC] sched/fair: decline WF_SYNC stacking when waker LLC is the busier share
@ 2026-08-04 23:14 Vinicius Costa Gomes
2026-08-05 2:14 ` Hillf Danton
0 siblings, 1 reply; 6+ messages in thread
From: Vinicius Costa Gomes @ 2026-08-04 23:14 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak
Cc: Chen Yu, Tim Chen, Srikar Dronamraju, Shubhang Kaushik,
Christoph Lameter, linux-kernel, Vinicius Costa Gomes
Since commit 900bbaae67e9 ("epoll: Add synchronous wakeup support for
ep_poll_callback"), epoll driven WF_SYNC wakeups have been "too
strong" and could cause tasks to stack on a busy NUMA node while other
nodes are relatively idle.
As commit 900bbaae67e9 ("epoll: Add synchronous wakeup support for
ep_poll_callback") improves real workloads a revert is not the answer.
The fix is to make the WF_SYNC "stack on waker" shortcut take into
account the load on this and prev's LLC, rejecting the shortcut only
when the waker (this) LLC is fully loaded and prev's LLC is less
loaded than the waker's.
Signed-off-by: Vinicius Costa Gomes <vinicius.gomes@intel.com>
---
We received a report of a regression on a openresty based
workload (the main metric being tail latencies) on a CWF SNC3 single
socket system, the main symptom that we could measure was one node
being overloaded while the other nodes were relatively idle.
Further investigation showed that spreading the NIC RX interrupts over
all NUMA nodes helped. Reverting commit 900bbaae67e9 ("epoll: Add
synchronous wakeup support for ep_poll_callback") also helped.
Commit 900bbaae67e9 gave the hint, that perhaps for the WF_SYNC wakeup
path the 'sync' flag was too strong and was not taking the load into
account, which for workloads that are too quick for the NUMA load
balancer could perhaps cause this "over stacking".
The idea of this patch is to reject the stacking if it would overload
a LLC. I couldn't find a way to make it NUMA-only and only measured on
NUMA-like machines. One more reason for sending it as a RFC.
(I just became aware of [1], I haven't tested the series, but seems
that it's the same problem but approached from the "other side"; And
from [2], it's interesting, for this workload WF_SYNC is "too strong"
for others, it's too weak. Further documentation about the expected
behavior might be good indeed.)
To try and emulate the workload, which the wakeups happen mostly on
one node, I tried memcached + memtier_benchmark, with memcached
unpinned and memtier pinned to one of the NUMA nodes. As the customer
machine was single socket one, I offlined half the cores of the
machine I have access. (AI helped with this)
Kernels
-------
base master fc46aed51f62 (post-v7.2-rc5) vanilla
rfc base + sched/fair: decline WF_SYNC stacking when waker LLC is the
busier share
Results (memcached + memtier_benchmark, 10 repetitions, medians +/- stdev)
---------------------------------------------------------------
Results (memcached + memtier_benchmark, 10 repetitions, mean +/- stddev)
------------------------------------------------------------------------
Client (memtier, the wakers) pinned to one SNC node; memcached worker
threads span the whole socket. Latencies in milliseconds.
Load 64x32 (memtier --threads=64 --clients=32)
Throughput (higher is better)
metric base rfc delta
-------------------------------------------------------------
ops/sec 3.179M +/- 0.027 3.072M +/- 0.039 -3.4%
Latency (lower is better)
pctile base rfc delta
-------------------------------------------------------------
p50 (ms) 0.488 +/- 0.006 0.493 +/- 0.013 +1.0%
p75 0.783 +/- 0.009 0.809 +/- 0.011 +3.3%
p90 1.265 +/- 0.010 1.312 +/- 0.017 +3.7%
p99 2.756 +/- 0.026 2.897 +/- 0.028 +5.1%
p99.9 3.753 +/- 0.064 4.005 +/- 0.117 +6.7%
p99.99 5.993 +/- 0.308 5.938 +/- 0.322 -0.9%
Load 96x32 (memtier --threads=96 --clients=32)
Throughput (higher is better)
metric base rfc delta
-------------------------------------------------------------
ops/sec 2.788M +/- 0.028 2.693M +/- 0.040 -3.4%
Latency (lower is better)
pctile base rfc delta
-------------------------------------------------------------
p50 (ms) 0.773 +/- 0.020 1.109 +/- 0.013 +43.5%
p75 1.321 +/- 0.013 1.135 +/- 0.016 -14.1%
p90 2.442 +/- 0.028 1.188 +/- 0.070 -51.4%
p99 4.857 +/- 0.020 1.974 +/- 0.141 -59.4%
p99.9 7.145 +/- 0.048 2.722 +/- 0.699 -61.9%
p99.99 9.637 +/- 0.230 4.324 +/- 0.888 -55.1%
[1] https://lore.kernel.org/all/20260714013940.4068189-4-srikar@linux.ibm.com/
[2] https://lore.kernel.org/all/20260803-b4-sched-sync-wakeup-v4-1-52333b0cfb79@gentwo.org/
---
kernel/sched/fair.c | 55 +++++++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 53 insertions(+), 2 deletions(-)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index d78467ec6ee1..c0c6cdaefda4 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -8185,6 +8185,56 @@ static int wake_wide(struct task_struct *p)
return 1;
}
+/*
+ * Decline WF_SYNC "stack on waker" wakeup when it would overload this LLC.
+ *
+ * Only decline when prev and this do not share a cache, this LLC has
+ * run out of idle headroom and prev is actually less loaded.
+ *
+ * Returns true when WF_SYNC should be declined.
+ */
+static bool sched_llc_over_commited(int this_cpu, int prev_cpu)
+{
+ struct sched_domain_shared *sds;
+ int this_busy, this_size, prev_busy, prev_size;
+ int headroom;
+
+ /* Same LLC: stacking cannot spread anywhere. */
+ if (cpus_share_cache(this_cpu, prev_cpu))
+ return false;
+
+ sds = rcu_dereference_all(per_cpu(sd_llc_shared, this_cpu));
+ if (!sds)
+ return false;
+ /* FIXME: if NO_HZ nr_busy_cpus is not updated, always accept? */
+ this_busy = atomic_read(&sds->nr_busy_cpus);
+ this_size = per_cpu(sd_llc_size, this_cpu);
+
+ /*
+ * Decline only once the waker LLC has no spare CPU beyond the
+ * waker's own, which is going to be replaced by this sync
+ * wakeup (the caller already made sure that nr_running == 1).
+ * If at least another idle CPU remains, keep stacking.
+ */
+ headroom = 1;
+
+ if (this_size - this_busy > headroom)
+ return false;
+
+ sds = rcu_dereference_all(per_cpu(sd_llc_shared, prev_cpu));
+ if (!sds)
+ return false;
+ prev_busy = atomic_read(&sds->nr_busy_cpus);
+ prev_size = per_cpu(sd_llc_size, prev_cpu);
+
+ /*
+ * Waker LLC is near saturation: spread only if prev is the less
+ * busy fraction (cross multiplied to take into account different
+ * LLC sizes).
+ */
+ return (u64)this_busy * prev_size > (u64)prev_busy * this_size;
+}
+
/*
* The purpose of wake_affine() is to quickly determine on which CPU we can run
* soonest. For the purpose of speed we only consider the waking and previous
@@ -8218,7 +8268,8 @@ wake_affine_idle(int this_cpu, int prev_cpu, int sync)
if (sync) {
struct rq *rq = cpu_rq(this_cpu);
- if ((rq->nr_running - cfs_h_nr_delayed(rq)) == 1)
+ if ((rq->nr_running - cfs_h_nr_delayed(rq)) == 1 &&
+ !sched_llc_over_commited(this_cpu, prev_cpu))
return this_cpu;
}
@@ -8237,7 +8288,7 @@ wake_affine_weight(struct sched_domain *sd, struct task_struct *p,
this_eff_load = cpu_load(cpu_rq(this_cpu));
- if (sync) {
+ if (sync && !sched_llc_over_commited(this_cpu, prev_cpu)) {
unsigned long current_load = task_h_load(current);
if (current_load > this_eff_load)
---
base-commit: c21bb4193868a8de71fc4693fa741e195fdf5d86
change-id: 20260723-epoll-wasync-fix-v1-28d2531033c7
Best regards,
--
Vinicius
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH RFC] sched/fair: decline WF_SYNC stacking when waker LLC is the busier share
2026-08-04 23:14 [PATCH RFC] sched/fair: decline WF_SYNC stacking when waker LLC is the busier share Vinicius Costa Gomes
@ 2026-08-05 2:14 ` Hillf Danton
2026-08-06 17:44 ` Vinicius Costa Gomes
0 siblings, 1 reply; 6+ messages in thread
From: Hillf Danton @ 2026-08-05 2:14 UTC (permalink / raw)
To: Vinicius Costa Gomes
Cc: Peter Zijlstra, K Prateek Nayak, Christoph Lameter, linux-kernel
On Tue, 04 Aug 2026 16:14:05 -0700 Vinicius Costa Gomes wrote:
> Since commit 900bbaae67e9 ("epoll: Add synchronous wakeup support for
> ep_poll_callback"), epoll driven WF_SYNC wakeups have been "too
> strong" and could cause tasks to stack on a busy NUMA node while other
> nodes are relatively idle.
>
> As commit 900bbaae67e9 ("epoll: Add synchronous wakeup support for
> ep_poll_callback") improves real workloads a revert is not the answer.
> The fix is to make the WF_SYNC "stack on waker" shortcut take into
> account the load on this and prev's LLC, rejecting the shortcut only
> when the waker (this) LLC is fully loaded and prev's LLC is less
> loaded than the waker's.
>
> Signed-off-by: Vinicius Costa Gomes <vinicius.gomes@intel.com>
> ---
> We received a report of a regression on a openresty based
> workload (the main metric being tail latencies) on a CWF SNC3 single
> socket system, the main symptom that we could measure was one node
> being overloaded while the other nodes were relatively idle.
>
> Further investigation showed that spreading the NIC RX interrupts over
> all NUMA nodes helped. Reverting commit 900bbaae67e9 ("epoll: Add
> synchronous wakeup support for ep_poll_callback") also helped.
>
The irq approach is prefered because anything that gets the eevdf offloaded
is good, you see it is near to the knowall point, needless to say that they
lie in different layers and from the scheduling-cpu pov irq is a gray rhino
in the room while WF_SYNC is a tiger mosquito in the corner at best in your
case where the mosquito failed to understand your workload.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH RFC] sched/fair: decline WF_SYNC stacking when waker LLC is the busier share
2026-08-05 2:14 ` Hillf Danton
@ 2026-08-06 17:44 ` Vinicius Costa Gomes
2026-08-06 22:53 ` Hillf Danton
0 siblings, 1 reply; 6+ messages in thread
From: Vinicius Costa Gomes @ 2026-08-06 17:44 UTC (permalink / raw)
To: Hillf Danton
Cc: Peter Zijlstra, K Prateek Nayak, Christoph Lameter, linux-kernel
Hillf Danton <hdanton@sina.com> writes:
> On Tue, 04 Aug 2026 16:14:05 -0700 Vinicius Costa Gomes wrote:
>> Since commit 900bbaae67e9 ("epoll: Add synchronous wakeup support for
>> ep_poll_callback"), epoll driven WF_SYNC wakeups have been "too
>> strong" and could cause tasks to stack on a busy NUMA node while other
>> nodes are relatively idle.
>>
>> As commit 900bbaae67e9 ("epoll: Add synchronous wakeup support for
>> ep_poll_callback") improves real workloads a revert is not the answer.
>> The fix is to make the WF_SYNC "stack on waker" shortcut take into
>> account the load on this and prev's LLC, rejecting the shortcut only
>> when the waker (this) LLC is fully loaded and prev's LLC is less
>> loaded than the waker's.
>>
>> Signed-off-by: Vinicius Costa Gomes <vinicius.gomes@intel.com>
>> ---
>> We received a report of a regression on a openresty based
>> workload (the main metric being tail latencies) on a CWF SNC3 single
>> socket system, the main symptom that we could measure was one node
>> being overloaded while the other nodes were relatively idle.
>>
>> Further investigation showed that spreading the NIC RX interrupts over
>> all NUMA nodes helped. Reverting commit 900bbaae67e9 ("epoll: Add
>> synchronous wakeup support for ep_poll_callback") also helped.
>>
> The irq approach is prefered because anything that gets the eevdf offloaded
> is good, you see it is near to the knowall point, needless to say that they
> lie in different layers and from the scheduling-cpu pov irq is a gray rhino
> in the room while WF_SYNC is a tiger mosquito in the corner at best in your
> case where the mosquito failed to understand your workload.
I don't think irq spreading across NUMA nodes is that good of an idea on
low loads/by default, as it loses the locality that the kernel (even
with irqbalance) try to maintain. I used that as a hackish way of
testing "if I spread tasks, does it improve the tail latencies?".
Note that the "local-only reproducer" workload (memcached +
memtier_benchmark) runs over loopback (no NIC irqs here), I pin memtier
(the client) to one NUMA node, leave the server unpinned and I am able
to reproduce the issue: with the RFC patch the tail latencies reduce by
2-3x. (on the customer workload the impact is even higher)
My expectation was that the scheduler would be able to say: "ugh, even
though respecting WF_SYNC is good most of the cases, this isn't one of
them". And this is the spirit of the RFC, showing that those cases
exist and their impact.
Cheers,
--
Vinicius
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH RFC] sched/fair: decline WF_SYNC stacking when waker LLC is the busier share
2026-08-06 17:44 ` Vinicius Costa Gomes
@ 2026-08-06 22:53 ` Hillf Danton
2026-08-06 23:22 ` Vinicius Costa Gomes
0 siblings, 1 reply; 6+ messages in thread
From: Hillf Danton @ 2026-08-06 22:53 UTC (permalink / raw)
To: Vinicius Costa Gomes
Cc: Peter Zijlstra, K Prateek Nayak, Christoph Lameter, linux-kernel
On Thu, 06 Aug 2026 10:44:18 -0700 Vinicius Costa Gomes wrote:
>Hillf Danton <hdanton@sina.com> writes:
>> On Tue, 04 Aug 2026 16:14:05 -0700 Vinicius Costa Gomes wrote:
>>> Since commit 900bbaae67e9 ("epoll: Add synchronous wakeup support for
>>> ep_poll_callback"), epoll driven WF_SYNC wakeups have been "too
>>> strong" and could cause tasks to stack on a busy NUMA node while other
>>> nodes are relatively idle.
>>>
>>> As commit 900bbaae67e9 ("epoll: Add synchronous wakeup support for
>>> ep_poll_callback") improves real workloads a revert is not the answer.
>>> The fix is to make the WF_SYNC "stack on waker" shortcut take into
>>> account the load on this and prev's LLC, rejecting the shortcut only
>>> when the waker (this) LLC is fully loaded and prev's LLC is less
>>> loaded than the waker's.
>>>
>>> Signed-off-by: Vinicius Costa Gomes <vinicius.gomes@intel.com>
>>> ---
>>> We received a report of a regression on a openresty based
>>> workload (the main metric being tail latencies) on a CWF SNC3 single
>>> socket system, the main symptom that we could measure was one node
>>> being overloaded while the other nodes were relatively idle.
>>>
>>> Further investigation showed that spreading the NIC RX interrupts over
>>> all NUMA nodes helped. Reverting commit 900bbaae67e9 ("epoll: Add
>>> synchronous wakeup support for ep_poll_callback") also helped.
>>>
>> The irq approach is prefered because anything that gets the eevdf offloaded
>> is good, you see it is near to the knowall point, needless to say that they
>> lie in different layers and from the scheduling-cpu pov irq is a gray rhino
>> in the room while WF_SYNC is a tiger mosquito in the corner at best in your
>> case where the mosquito failed to understand your workload.
>
> I don't think irq spreading across NUMA nodes is that good of an idea on
> low loads/by default, as it loses the locality that the kernel (even
> with irqbalance) try to maintain. I used that as a hackish way of
> testing "if I spread tasks, does it improve the tail latencies?".
>
Can you specify the root cause of the tail latencies, particularly after
spending two minutes thinking if changing config in user space to solve the
issue is a light year better than adding a couple lines of code in the
wakeup path?
> Note that the "local-only reproducer" workload (memcached +
> memtier_benchmark) runs over loopback (no NIC irqs here), I pin memtier
> (the client) to one NUMA node, leave the server unpinned and I am able
> to reproduce the issue: with the RFC patch the tail latencies reduce by
> 2-3x. (on the customer workload the impact is even higher)
>
> My expectation was that the scheduler would be able to say: "ugh, even
> though respecting WF_SYNC is good most of the cases, this isn't one of
No comment before root cause specified, even if I suspect it sounds like
that two CPU cores could not provide line speed 24-port 1000MB ether switch
before 2010 while 12 cores could.
> them". And this is the spirit of the RFC, showing that those cases
> exist and their impact.
>
>
>Cheers,
>--
>Vinicius
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH RFC] sched/fair: decline WF_SYNC stacking when waker LLC is the busier share
2026-08-06 22:53 ` Hillf Danton
@ 2026-08-06 23:22 ` Vinicius Costa Gomes
2026-08-07 5:17 ` Hillf Danton
0 siblings, 1 reply; 6+ messages in thread
From: Vinicius Costa Gomes @ 2026-08-06 23:22 UTC (permalink / raw)
To: Hillf Danton
Cc: Peter Zijlstra, K Prateek Nayak, Christoph Lameter, linux-kernel
Hillf Danton <hdanton@sina.com> writes:
> On Thu, 06 Aug 2026 10:44:18 -0700 Vinicius Costa Gomes wrote:
>>Hillf Danton <hdanton@sina.com> writes:
>>> On Tue, 04 Aug 2026 16:14:05 -0700 Vinicius Costa Gomes wrote:
>>>> Since commit 900bbaae67e9 ("epoll: Add synchronous wakeup support for
>>>> ep_poll_callback"), epoll driven WF_SYNC wakeups have been "too
>>>> strong" and could cause tasks to stack on a busy NUMA node while other
>>>> nodes are relatively idle.
>>>>
>>>> As commit 900bbaae67e9 ("epoll: Add synchronous wakeup support for
>>>> ep_poll_callback") improves real workloads a revert is not the answer.
>>>> The fix is to make the WF_SYNC "stack on waker" shortcut take into
>>>> account the load on this and prev's LLC, rejecting the shortcut only
>>>> when the waker (this) LLC is fully loaded and prev's LLC is less
>>>> loaded than the waker's.
>>>>
>>>> Signed-off-by: Vinicius Costa Gomes <vinicius.gomes@intel.com>
>>>> ---
>>>> We received a report of a regression on a openresty based
>>>> workload (the main metric being tail latencies) on a CWF SNC3 single
>>>> socket system, the main symptom that we could measure was one node
>>>> being overloaded while the other nodes were relatively idle.
>>>>
>>>> Further investigation showed that spreading the NIC RX interrupts over
>>>> all NUMA nodes helped. Reverting commit 900bbaae67e9 ("epoll: Add
>>>> synchronous wakeup support for ep_poll_callback") also helped.
>>>>
>>> The irq approach is prefered because anything that gets the eevdf offloaded
>>> is good, you see it is near to the knowall point, needless to say that they
>>> lie in different layers and from the scheduling-cpu pov irq is a gray rhino
>>> in the room while WF_SYNC is a tiger mosquito in the corner at best in your
>>> case where the mosquito failed to understand your workload.
>>
>> I don't think irq spreading across NUMA nodes is that good of an idea on
>> low loads/by default, as it loses the locality that the kernel (even
>> with irqbalance) try to maintain. I used that as a hackish way of
>> testing "if I spread tasks, does it improve the tail latencies?".
>>
> Can you specify the root cause of the tail latencies, particularly after
> spending two minutes thinking if changing config in user space to solve the
> issue is a light year better than adding a couple lines of code in the
> wakeup path?
>
On a machine running the customer workload, I ran a bpftrace script that
tracks the time "openresty" tasks stay on the runqueue waiting to be run.
(this was a using an earlier version of the patch, on top of v7.1-rc7, I
don't have easy access to the machine anymore)
Before:
@runq_us:
[0] 5376 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1] 5122 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[2, 4) 5073 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[4, 8) 3821 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[8, 16) 1649 |@@@@@@@@@@@@@@@ |
[16, 32) 747 |@@@@@@@ |
[32, 64) 552 |@@@@@ |
[64, 128) 668 |@@@@@@ |
[128, 256) 907 |@@@@@@@@ |
[256, 512) 1235 |@@@@@@@@@@@ |
[512, 1K) 1946 |@@@@@@@@@@@@@@@@@@ |
[1K, 2K) 1832 |@@@@@@@@@@@@@@@@@ |
[2K, 4K) 1130 |@@@@@@@@@@ |
[4K, 8K) 561 |@@@@@ |
[8K, 16K) 340 |@@@ |
[16K, 32K) 250 |@@ |
[32K, 64K) 108 |@ |
[64K, 128K) 33 | |
[128K, 256K) 0 | |
[256K, 512K) 0 | |
[512K, 1M) 1 | |
After:
@runq_us:
[0] 1781353 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1] 450223 |@@@@@@@@@@@@@ |
[2, 4) 71505 |@@ |
[4, 8) 21345 | |
[8, 16) 22406 | |
[16, 32) 14792 | |
[32, 64) 7336 | |
[64, 128) 3867 | |
[128, 256) 4607 | |
[256, 512) 6886 | |
[512, 1K) 5862 | |
[1K, 2K) 2829 | |
[2K, 4K) 876 | |
[4K, 8K) 11 | |
This made me think that the almost unconditional stacking shortcut that
WF_SYNC promotes was causing tasks to wait on already busy CPUs, while
there were idle CPUs around.
That was as close to a root cause that I got.
>> Note that the "local-only reproducer" workload (memcached +
>> memtier_benchmark) runs over loopback (no NIC irqs here), I pin memtier
>> (the client) to one NUMA node, leave the server unpinned and I am able
>> to reproduce the issue: with the RFC patch the tail latencies reduce by
>> 2-3x. (on the customer workload the impact is even higher)
>>
>> My expectation was that the scheduler would be able to say: "ugh, even
>> though respecting WF_SYNC is good most of the cases, this isn't one of
>
> No comment before root cause specified, even if I suspect it sounds like
> that two CPU cores could not provide line speed 24-port 1000MB ether switch
> before 2010 while 12 cores could.
>
>> them". And this is the spirit of the RFC, showing that those cases
>> exist and their impact.
>>
>>
>>Cheers,
>>--
>>Vinicius
--
Vinicius
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH RFC] sched/fair: decline WF_SYNC stacking when waker LLC is the busier share
2026-08-06 23:22 ` Vinicius Costa Gomes
@ 2026-08-07 5:17 ` Hillf Danton
0 siblings, 0 replies; 6+ messages in thread
From: Hillf Danton @ 2026-08-07 5:17 UTC (permalink / raw)
To: Vinicius Costa Gomes
Cc: Peter Zijlstra, K Prateek Nayak, Christoph Lameter, linux-kernel
On Thu, 06 Aug 2026 16:22:14 -0700 Vinicius Costa Gomes wrote:
>Hillf Danton <hdanton@sina.com> writes:
>> On Thu, 06 Aug 2026 10:44:18 -0700 Vinicius Costa Gomes wrote:
>>>Hillf Danton <hdanton@sina.com> writes:
>>>> On Tue, 04 Aug 2026 16:14:05 -0700 Vinicius Costa Gomes wrote:
>>>>> Since commit 900bbaae67e9 ("epoll: Add synchronous wakeup support for
>>>>> ep_poll_callback"), epoll driven WF_SYNC wakeups have been "too
>>>>> strong" and could cause tasks to stack on a busy NUMA node while other
>>>>> nodes are relatively idle.
>>>>>
>>>>> As commit 900bbaae67e9 ("epoll: Add synchronous wakeup support for
>>>>> ep_poll_callback") improves real workloads a revert is not the answer.
>>>>> The fix is to make the WF_SYNC "stack on waker" shortcut take into
>>>>> account the load on this and prev's LLC, rejecting the shortcut only
>>>>> when the waker (this) LLC is fully loaded and prev's LLC is less
>>>>> loaded than the waker's.
>>>>>
>>>>> Signed-off-by: Vinicius Costa Gomes <vinicius.gomes@intel.com>
>>>>> ---
>>>>> We received a report of a regression on a openresty based
>>>>> workload (the main metric being tail latencies) on a CWF SNC3 single
>>>>> socket system, the main symptom that we could measure was one node
>>>>> being overloaded while the other nodes were relatively idle.
>>>>>
>>>>> Further investigation showed that spreading the NIC RX interrupts over
>>>>> all NUMA nodes helped. Reverting commit 900bbaae67e9 ("epoll: Add
>>>>> synchronous wakeup support for ep_poll_callback") also helped.
>>>>>
>>>> The irq approach is prefered because anything that gets the eevdf offloaded
>>>> is good, you see it is near to the knowall point, needless to say that they
>>>> lie in different layers and from the scheduling-cpu pov irq is a gray rhino
>>>> in the room while WF_SYNC is a tiger mosquito in the corner at best in your
>>>> case where the mosquito failed to understand your workload.
>>>
>>> I don't think irq spreading across NUMA nodes is that good of an idea on
>>> low loads/by default, as it loses the locality that the kernel (even
>>> with irqbalance) try to maintain. I used that as a hackish way of
>>> testing "if I spread tasks, does it improve the tail latencies?".
>>>
>> Can you specify the root cause of the tail latencies, particularly after
>> spending two minutes thinking if changing config in user space to solve the
>> issue is a light year better than adding a couple lines of code in the
>> wakeup path?
>>
> On a machine running the customer workload, I ran a bpftrace script that
> tracks the time "openresty" tasks stay on the runqueue waiting to be run.
>
> (this was a using an earlier version of the patch, on top of v7.1-rc7, I
> don't have easy access to the machine anymore)
>
> Before:
>
> @runq_us:
> [0] 5376 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
> [1] 5122 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
> [2, 4) 5073 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
> [4, 8) 3821 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
> [8, 16) 1649 |@@@@@@@@@@@@@@@ |
> [16, 32) 747 |@@@@@@@ |
> [32, 64) 552 |@@@@@ |
> [64, 128) 668 |@@@@@@ |
> [128, 256) 907 |@@@@@@@@ |
> [256, 512) 1235 |@@@@@@@@@@@ |
> [512, 1K) 1946 |@@@@@@@@@@@@@@@@@@ |
> [1K, 2K) 1832 |@@@@@@@@@@@@@@@@@ |
> [2K, 4K) 1130 |@@@@@@@@@@ |
> [4K, 8K) 561 |@@@@@ |
> [8K, 16K) 340 |@@@ |
> [16K, 32K) 250 |@@ |
> [32K, 64K) 108 |@ |
> [64K, 128K) 33 | |
> [128K, 256K) 0 | |
> [256K, 512K) 0 | |
> [512K, 1M) 1 | |
>
> After:
>
> @runq_us:
> [0] 1781353 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
> [1] 450223 |@@@@@@@@@@@@@ |
> [2, 4) 71505 |@@ |
> [4, 8) 21345 | |
> [8, 16) 22406 | |
> [16, 32) 14792 | |
> [32, 64) 7336 | |
> [64, 128) 3867 | |
> [128, 256) 4607 | |
> [256, 512) 6886 | |
> [512, 1K) 5862 | |
> [1K, 2K) 2829 | |
> [2K, 4K) 876 | |
> [4K, 8K) 11 | |
>
> This made me think that the almost unconditional stacking shortcut that
> WF_SYNC promotes was causing tasks to wait on already busy CPUs, while
> there were idle CPUs around.
>
Then like the line-speed ether switch below, the CPU cycles per tick, CPT,
instead of LLC is the critical point, and increasing CPT is the correct pill, no?
What is not unusual from Monday to Friday is code is added in kernel for
mis-configured boxes.
> That was as close to a root cause that I got.
>
>>> Note that the "local-only reproducer" workload (memcached +
>>> memtier_benchmark) runs over loopback (no NIC irqs here), I pin memtier
>>> (the client) to one NUMA node, leave the server unpinned and I am able
>>> to reproduce the issue: with the RFC patch the tail latencies reduce by
>>> 2-3x. (on the customer workload the impact is even higher)
>>>
>>> My expectation was that the scheduler would be able to say: "ugh, even
>>> though respecting WF_SYNC is good most of the cases, this isn't one of
>>
>> No comment before root cause specified, even if I suspect it sounds like
>> that two CPU cores could not provide line speed 24-port 1000MB ether switch
>> before 2010 while 12 cores could.
>>
>>> them". And this is the spirit of the RFC, showing that those cases
>>> exist and their impact.
>>>
>>>
>>>Cheers,
>>>--
>>>Vinicius
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2026-08-07 5:18 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-08-04 23:14 [PATCH RFC] sched/fair: decline WF_SYNC stacking when waker LLC is the busier share Vinicius Costa Gomes
2026-08-05 2:14 ` Hillf Danton
2026-08-06 17:44 ` Vinicius Costa Gomes
2026-08-06 22:53 ` Hillf Danton
2026-08-06 23:22 ` Vinicius Costa Gomes
2026-08-07 5:17 ` Hillf Danton
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox