Re: [PATCH RFC] sched/fair: decline WF_SYNC stacking when waker LLC is the busier share

[Hillf Danton <hdanton@sina.com>]

On Thu, 06 Aug 2026 16:22:14 -0700 Vinicius Costa Gomes wrote:
>Hillf Danton <hdanton@sina.com> writes:
>> On Thu, 06 Aug 2026 10:44:18 -0700 Vinicius Costa Gomes wrote:
>>>Hillf Danton <hdanton@sina.com> writes:
>>>> On Tue, 04 Aug 2026 16:14:05 -0700 Vinicius Costa Gomes wrote:
>>>>> Since commit 900bbaae67e9 ("epoll: Add synchronous wakeup support for
>>>>> ep_poll_callback"), epoll driven WF_SYNC wakeups have been "too
>>>>> strong" and could cause tasks to stack on a busy NUMA node while other
>>>>> nodes are relatively idle.
>>>>> 
>>>>> As commit 900bbaae67e9 ("epoll: Add synchronous wakeup support for
>>>>> ep_poll_callback") improves real workloads a revert is not the answer.
>>>>> The fix is to make the WF_SYNC "stack on waker" shortcut take into
>>>>> account the load on this and prev's LLC, rejecting the shortcut only
>>>>> when the waker (this) LLC is fully loaded and prev's LLC is less
>>>>> loaded than the waker's.
>>>>> 
>>>>> Signed-off-by: Vinicius Costa Gomes <vinicius.gomes@intel.com>
>>>>> ---
>>>>> We received a report of a regression on a openresty based
>>>>> workload (the main metric being tail latencies) on a CWF SNC3 single
>>>>> socket system, the main symptom that we could measure was one node
>>>>> being overloaded while the other nodes were relatively idle.
>>>>> 
>>>>> Further investigation showed that spreading the NIC RX interrupts over
>>>>> all NUMA nodes helped. Reverting commit 900bbaae67e9 ("epoll: Add
>>>>> synchronous wakeup support for ep_poll_callback") also helped.
>>>>> 
>>>> The irq approach is prefered because anything that gets the eevdf offloaded
>>>> is good, you see it is near to the knowall point, needless to say that they
>>>> lie in different layers and from the scheduling-cpu pov irq is a gray rhino
>>>> in the room while WF_SYNC is a tiger mosquito in the corner at best in your
>>>> case where the mosquito failed to understand your workload.
>>>
>>> I don't think irq spreading across NUMA nodes is that good of an idea on
>>> low loads/by default, as it loses the locality that the kernel (even
>>> with irqbalance) try to maintain. I used that as a hackish way of
>>> testing "if I spread tasks, does it improve the tail latencies?".
>>> 
>> Can you specify the root cause of the tail latencies, particularly after
>> spending two minutes thinking if changing config in user space to solve the
>> issue is a light year better than adding a couple lines of code in the
>> wakeup path?
>>
> On a machine running the customer workload, I ran a bpftrace script that
> tracks the time "openresty" tasks stay on the runqueue waiting to be run.
> 
> (this was a using an earlier version of the patch, on top of v7.1-rc7, I
> don't have easy access to the machine anymore)
> 
> Before:
> 
> @runq_us:
> [0]                 5376 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
> [1]                 5122 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@   |
> [2, 4)              5073 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@   |
> [4, 8)              3821 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                |
> [8, 16)             1649 |@@@@@@@@@@@@@@@                                     |
> [16, 32)             747 |@@@@@@@                                             |
> [32, 64)             552 |@@@@@                                               |
> [64, 128)            668 |@@@@@@                                              |
> [128, 256)           907 |@@@@@@@@                                            |
> [256, 512)          1235 |@@@@@@@@@@@                                         |
> [512, 1K)           1946 |@@@@@@@@@@@@@@@@@@                                  |
> [1K, 2K)            1832 |@@@@@@@@@@@@@@@@@                                   |
> [2K, 4K)            1130 |@@@@@@@@@@                                          |
> [4K, 8K)             561 |@@@@@                                               |
> [8K, 16K)            340 |@@@                                                 |
> [16K, 32K)           250 |@@                                                  |
> [32K, 64K)           108 |@                                                   |
> [64K, 128K)           33 |                                                    |
> [128K, 256K)           0 |                                                    |
> [256K, 512K)           0 |                                                    |
> [512K, 1M)             1 |                                                    |
> 
> After:
> 
> @runq_us:
> [0]              1781353 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
> [1]               450223 |@@@@@@@@@@@@@                                       |
> [2, 4)             71505 |@@                                                  |
> [4, 8)             21345 |                                                    |
> [8, 16)            22406 |                                                    |
> [16, 32)           14792 |                                                    |
> [32, 64)            7336 |                                                    |
> [64, 128)           3867 |                                                    |
> [128, 256)          4607 |                                                    |
> [256, 512)          6886 |                                                    |
> [512, 1K)           5862 |                                                    |
> [1K, 2K)            2829 |                                                    |
> [2K, 4K)             876 |                                                    |
> [4K, 8K)              11 |                                                    |
> 
> This made me think that the almost unconditional stacking shortcut that
> WF_SYNC promotes was causing tasks to wait on already busy CPUs, while
> there were idle CPUs around.
> 
Then like the line-speed ether switch below, the CPU cycles per tick, CPT,
instead of LLC is the critical point, and increasing CPT is the correct pill, no?
What is not unusual from Monday to Friday is code is added in kernel for
mis-configured boxes.

> That was as close to a root cause that I got.
> 
>>> Note that the "local-only reproducer" workload (memcached +
>>> memtier_benchmark) runs over loopback (no NIC irqs here), I pin memtier
>>> (the client) to one NUMA node, leave the server unpinned and I am able
>>> to reproduce the issue: with the RFC patch the tail latencies reduce by
>>> 2-3x. (on the customer workload the impact is even higher)
>>> 
>>> My expectation was that the scheduler would be able to say: "ugh, even
>>> though respecting WF_SYNC is good most of the cases, this isn't one of
>>
>> No comment before root cause specified, even if I suspect it sounds like
>> that two CPU cores could not provide line speed 24-port 1000MB ether switch
>> before 2010 while 12 cores could.
>>
>>> them". And this is the spirit of the RFC, showing that those cases
>>> exist and their impact.
>>>
>>>
>>>Cheers,
>>>-- 
>>>Vinicius