Re: [PATCH v4] sched/fair: Preserve wake-affine CPU for non-SMT reciprocal sync wakeups

[K Prateek Nayak <kprateek.nayak@amd.com>]

Hello Shrikanth,

On 8/4/2026 4:10 PM, Shrikanth Hegde wrote:
> Hi Prateek.
> 
> On 8/4/26 2:12 PM, K Prateek Nayak wrote:
>> Hello Shrikanth,
>>
>> On 8/4/2026 10:10 AM, Shrikanth Hegde wrote:
>>> As I said in v3, before we add bells/whistles to sync path, i want
>>> to know what is expected of sync behavior today.
>>> And that should be documented in Documentation/scheduler/
>>>
>>> Be it,
>>> - current way of hint only and scheduler can still choose an idle core/idle cpu etc.
>>> - Should it be enforcing it to waker cpu if waker cpu has only one task.
>>> - Whatever the policy maybe.
>>>
>>> Current api usage is tricky to use and effect is visible in real life workloads.
>>> The case I mentioned in v3 of networking code using sync api leads to strange
>>> results due to sync mechanism.
>>> - It depends whether waker/wakee are running on same node.
>>> - Result of wake_wide.
>>> In other end, user sees inconsistent latency/throughput.
>>>
>>> We can keep on adding minor changes to sync api path,
>>> but one benchmark will benefit and one will suffer.
>>> Having the behavior documented is a good start.
>>>
>>> Peter, Ingo, Vincent, Mel, Prateek,
>>> What do you guys think?
>>
>> Currently it is very arbitrary and WF_SYNC may, or may not, indicate a
>> true voluntary blocking behavior. For example, anon_pipe_read() uses a
>> wake_up_interruptible_sync_poll() to wake up writers once reader has
>> drained the pipe but if you think about it, why would the reader block
>> soon after just having the data it needed?
> 
> Doesn't "perf bench sched pipe" also use anon_pipe_read/write?
> 
> -   27.49%     0.32%  sched-pipe       [kernel.kallsyms]                 [k] ksys_read
>    - 27.17% ksys_read
>       - 26.57% vfs_read
>          - 22.33% anon_pipe_read

Exactly! Highly depends on the workload - if you are using pipe
for a signal, great, but if you are piping gigabytes of data,
and there is a continuous consumption, then co-locating the
readers and writers makes sense. This is probably why wake_wide
doesn't even care about the sync hint and makes a call purely on
waker_flips.

> 
>>
>> Here are the results on my Zen4 system from running perf bench
>> sched messaging (threads + pipes) at varying worker counts with
>> all wake_up_interruptible_sync_poll converted to
>> wake_up_interruptible_poll:
>>
>> Test:                   tip                     no_sync
>>   1-groups:         3.79 (0.00 pct)         3.35 (11.60 pct)
>>   2-groups:         3.85 (0.00 pct)         3.41 (11.42 pct)
>>   4-groups:         4.02 (0.00 pct)         3.32 (17.41 pct)
>>   8-groups:         4.33 (0.00 pct)         4.38 (-1.15 pct)
>> 16-groups:         6.09 (0.00 pct)         6.12 (-0.49 pct)
>> ---
>>
>> So seems like WF_SYNC hint on this machine with perf bench sched
>> messaging (thread + pipes) pattern is actually holding it back.
>> Lemme check processes ...
>>
>> Test:                   tip                     no_sync
>>   1-groups:         3.48 (0.00 pct)         3.08 (11.49 pct)
>>   2-groups:         3.80 (0.00 pct)         3.07 (19.21 pct)
>>   4-groups:         3.91 (0.00 pct)         3.09 (20.97 pct)
>>   8-groups:         4.13 (0.00 pct)         4.10 (0.72 pct)
>> 16-groups:         5.81 (0.00 pct)         5.74 (1.20 pct)
>>
>> Similar stuff. At some point it was pretty bad for Zen3 but
>> situation might have changed since ¯\_(ツ)_/¯ I'll let you
>> know once I have a machine.
>>
>> But ... If I have true 1:1 waiting on pipe as in the case of
>> "perf bench sched pipe -l 1000000" I go from ~2.5usecs/op on
>> average to ~4.2usecs/op which is close to a 50% increase in the
>> benchmark time so that WF_SYNC hint can also help if all we have
>> is looping over a read waiting for one page worth of write.
>>
>> The way I look at WF_SYNC nowadays is that it indicates a local LLC
>> wakeup is beneficial. wake_wide() doesn't even consider WF_SYNc and
>> simply uses wake-wakee flips and then only at want_affine() do we
>> actually check the sync hint.
>>
> 
> Yes, it is difficult to say when would sync actually kick in.
> Also, if we say hint, onus now falls on scheduler to optimize all
> call sites.
> 
> Clearly comments around __wake_up_sync_key are outdated.

At some point (1da177e4c3f4), sync + wakeup on waker's CPU also
inhibited resched_curr() and allowed the waker to naturally yield
the CPU (which is the reason for the UP comment) but today, we do
a resched_curr() unconditionally. Signs of different times.

> 
>> Most benefit come from wake_affine_idle() for the 1 task case where
>> target is set to current and select_idle_sibling() uses that as the
>> target from there on.
>>
>> It could purely be a coincidence that it benefits at all - most of
>> these microbenchmark we have always hit the same two syscall (mostly
>> read() and write() in turns) and a most of benefit for those comes
>> from kernel instructions being primed in cache.
>>
>> I remember a while back, removing the effect of WF_SYNC on the
>> networking side hampered a lot of performance - especially for
>> localhost communications. This one specifically
>> https://lore.kernel.org/lkml/20220711224704.1672831-1-libo.chen@oracle.com/
>>
>> Let me see if things have miraculously changed there too but for
>> TCP sockets in real world, with blocking for ACKs, I think
>> WF_SYNC still makes sense there but I feel most of the benefits
>> from sync are a second order effect.
>>
> 
> But today it calls sync even for non-blocking.

Ack. I think it just got carried over and for most real world
scenarios, it probably didn't made a difference until the processor
topologies diverged and we have many machines with many small LLCs
and many more machines with many large LLCs on the same socket.

-- 
Thanks and Regards,
Prateek