Re: [PATCH v4] sched/fair: Preserve wake-affine CPU for non-SMT reciprocal sync wakeups

[Shrikanth Hegde <sshegde@linux.ibm.com>]

Hi Prateek.

On 8/4/26 2:12 PM, K Prateek Nayak wrote:
> Hello Shrikanth,
> 
> On 8/4/2026 10:10 AM, Shrikanth Hegde wrote:
>> As I said in v3, before we add bells/whistles to sync path, i want
>> to know what is expected of sync behavior today.
>> And that should be documented in Documentation/scheduler/
>>
>> Be it,
>> - current way of hint only and scheduler can still choose an idle core/idle cpu etc.
>> - Should it be enforcing it to waker cpu if waker cpu has only one task.
>> - Whatever the policy maybe.
>>
>> Current api usage is tricky to use and effect is visible in real life workloads.
>> The case I mentioned in v3 of networking code using sync api leads to strange
>> results due to sync mechanism.
>> - It depends whether waker/wakee are running on same node.
>> - Result of wake_wide.
>> In other end, user sees inconsistent latency/throughput.
>>
>> We can keep on adding minor changes to sync api path,
>> but one benchmark will benefit and one will suffer.
>> Having the behavior documented is a good start.
>>
>> Peter, Ingo, Vincent, Mel, Prateek,
>> What do you guys think?
> 
> Currently it is very arbitrary and WF_SYNC may, or may not, indicate a
> true voluntary blocking behavior. For example, anon_pipe_read() uses a
> wake_up_interruptible_sync_poll() to wake up writers once reader has
> drained the pipe but if you think about it, why would the reader block
> soon after just having the data it needed?

Doesn't "perf bench sched pipe" also use anon_pipe_read/write?

-   27.49%     0.32%  sched-pipe       [kernel.kallsyms]                 [k] ksys_read
    - 27.17% ksys_read
       - 26.57% vfs_read
          - 22.33% anon_pipe_read

> 
> Here are the results on my Zen4 system from running perf bench
> sched messaging (threads + pipes) at varying worker counts with
> all wake_up_interruptible_sync_poll converted to
> wake_up_interruptible_poll:
> 
> Test:                   tip                     no_sync
>   1-groups:         3.79 (0.00 pct)         3.35 (11.60 pct)
>   2-groups:         3.85 (0.00 pct)         3.41 (11.42 pct)
>   4-groups:         4.02 (0.00 pct)         3.32 (17.41 pct)
>   8-groups:         4.33 (0.00 pct)         4.38 (-1.15 pct)
> 16-groups:         6.09 (0.00 pct)         6.12 (-0.49 pct)
> ---
> 
> So seems like WF_SYNC hint on this machine with perf bench sched
> messaging (thread + pipes) pattern is actually holding it back.
> Lemme check processes ...
> 
> Test:                   tip                     no_sync
>   1-groups:         3.48 (0.00 pct)         3.08 (11.49 pct)
>   2-groups:         3.80 (0.00 pct)         3.07 (19.21 pct)
>   4-groups:         3.91 (0.00 pct)         3.09 (20.97 pct)
>   8-groups:         4.13 (0.00 pct)         4.10 (0.72 pct)
> 16-groups:         5.81 (0.00 pct)         5.74 (1.20 pct)
> 
> Similar stuff. At some point it was pretty bad for Zen3 but
> situation might have changed since ¯\_(ツ)_/¯ I'll let you
> know once I have a machine.
> 
> But ... If I have true 1:1 waiting on pipe as in the case of
> "perf bench sched pipe -l 1000000" I go from ~2.5usecs/op on
> average to ~4.2usecs/op which is close to a 50% increase in the
> benchmark time so that WF_SYNC hint can also help if all we have
> is looping over a read waiting for one page worth of write.
> 
> The way I look at WF_SYNC nowadays is that it indicates a local LLC
> wakeup is beneficial. wake_wide() doesn't even consider WF_SYNc and
> simply uses wake-wakee flips and then only at want_affine() do we
> actually check the sync hint.
> 

Yes, it is difficult to say when would sync actually kick in.
Also, if we say hint, onus now falls on scheduler to optimize all
call sites.

Clearly comments around __wake_up_sync_key are outdated.

> Most benefit come from wake_affine_idle() for the 1 task case where
> target is set to current and select_idle_sibling() uses that as the
> target from there on.
> 
> It could purely be a coincidence that it benefits at all - most of
> these microbenchmark we have always hit the same two syscall (mostly
> read() and write() in turns) and a most of benefit for those comes
> from kernel instructions being primed in cache.
> 
> I remember a while back, removing the effect of WF_SYNC on the
> networking side hampered a lot of performance - especially for
> localhost communications. This one specifically
> https://lore.kernel.org/lkml/20220711224704.1672831-1-libo.chen@oracle.com/
> 
> Let me see if things have miraculously changed there too but for
> TCP sockets in real world, with blocking for ACKs, I think
> WF_SYNC still makes sense there but I feel most of the benefits
> from sync are a second order effect.
> 

But today it calls sync even for non-blocking.