Re: [PATCH v4] sched/fair: Preserve wake-affine CPU for non-SMT reciprocal sync wakeups

[Shrikanth Hegde <sshegde@linux.ibm.com>]

Hi Shubhang.

Please give time for discussion/reply for the people looking at
your patches. Even before I could respond to your v3, you have sent v4.
And your v4 doesn't addresses the concerns raised in v3.

On 8/4/26 7:15 AM, Shubhang Kaushik (Ampere) wrote:
> For WF_SYNC wakeups, wake_affine() may select the waker CPU, but the CFS
> wakeup path still passes that target to select_idle_sibling(). The idle
> CPU search can then move the wakee away from the wake-affine target.
> 
> Pipe-style ping-pong workloads expose this because the wakee is handed
> back and forth between two tasks. In that case, moving the wakee to
> another idle CPU can cost more than preserving the wake-affine waker CPU.
> 
> Use the existing last_wakee and wake_wide() state to identify narrow
> reciprocal WF_SYNC wakeups:
> 
>    A wakes B
>    B wakes A
>    A wakes B
>    ...
> 
> Handle only this narrow reciprocal case on non-SMT systems. Once the
> wake-affine path has selected or kept the waker CPU, preserve that target
> when the waker rq has no other runnable fair task. Return the waker CPU
> before select_idle_sibling() so the idle CPU search does not move this
> handoff away from the wake-affine target.
> 

Why on non-SMT? Why the same problem cannot happen in SMT systems?


> This does not define a generic WF_SYNC placement rule. Generic WF_SYNC
> wakeups continue through the existing wake_affine() and
> select_idle_sibling() behavior. SMT systems also continue through
> select_idle_sibling(), where idle sibling/core placement can be handled
> with SMT topology visible.
> 
> On asymmetric-capacity systems, still require the wakee to fit on the
> waker CPU.
> 
> Signed-off-by: Shubhang Kaushik (Ampere) <sh@gentwo.org>
> ---
> Tested on 80-core non-SMT Ampere Altra, tip:sched/core baseline.

Where is your LLC? Does it has multiple cores and currently
you end up choosing an idle core?
Your numbers below pretty much tell the same story.

> 
> perf bench sched pipe -l 1000000, 20 runs:

IIUC, sched pipe doesn't do any work apart from ping-pong.

> default:
>    3.985 -> 3.187 usec/op mean,   about 20.0% improvement
>    4.026 -> 3.181 usec/op median, about 21.0% improvement
> 
> taskset -c 78,79:
>    3.851 -> 3.144 usec/op mean,   about 18.4% improvement
>    3.804 -> 3.140 usec/op median, about 17.4% improvement
> 
> taskset -c 79:
>    3.055 -> 3.113 usec/op mean,   about 1.9% slower
>    3.045 -> 3.109 usec/op median, about 2.1% slower
> 

Which means you get the best result when it runs on same CPU.
The rest of the changes likely enforce that behavior. Then same issue is
prevalent in SMT world too.

> Hackbench process/thread pipe cases with 1/2/4/8 groups were within
> noise, with mean deltas from -1.8% to +3.7% over 10 runs.
> 
> Schbench normal mode at 8/40/80/240 workers and schbench pipe mode at
> 1/2/4/8 workers showed no material regression.
> 
> Baseline: tip/sched/core at 5186ef36909c

As I said in v3, before we add bells/whistles to sync path, i want
to know what is expected of sync behavior today.
And that should be documented in Documentation/scheduler/

Be it,
- current way of hint only and scheduler can still choose an idle core/idle cpu etc.
- Should it be enforcing it to waker cpu if waker cpu has only one task.
- Whatever the policy maybe.

Current api usage is tricky to use and effect is visible in real life workloads.
The case I mentioned in v3 of networking code using sync api leads to strange
results due to sync mechanism.
- It depends whether waker/wakee are running on same node.
- Result of wake_wide.
In other end, user sees inconsistent latency/throughput.

We can keep on adding minor changes to sync api path,
but one benchmark will benefit and one will suffer.
Having the behavior documented is a good start.

Peter, Ingo, Vincent, Mel, Prateek,
What do you guys think?