Re: [RFC PATCH 7/8] sched/fair: Retrieve cached group stats from sg_lb_stats_prop

[K Prateek Nayak <kprateek.nayak@amd.com>]

Hello Chenyu,

Thank you for taking a look at the series.

On 3/17/2025 11:34 PM, Chen, Yu C wrote:
> On 3/13/2025 5:37 PM, K Prateek Nayak wrote:
>> Allow update_sg_lb_stats() to retrieve the group stats cached in
>> sg_lb_stats_prop saved by another CPU performing load balancing around
>> the same time (same jiffy)
>>
> 
> If I understand correctly, we allow update_sg_lb_stats() to retrieve
> cached sg stats if another CPU in the same sched group has just done
> load balance within a jiffy ago, say 10ms for CONFIG_100_HZ.

Quick disclaimer: All of this is best effort currently.

Periodic load balancing is easy to start since it only happens once a
jiffy at the maximum so "last_update" as a jiffy counter should be
good enough (in most cases).

Secondly, and this is slightly harder to solve for, is to get all the
CPUs to actually sync. Currently it is a best effort case since the
tick can fire late due to disabled interrupts on CPU, SCHED_SOFTIRQ
may run at different times depending on how much work is done at tick
prior to reaching the softirq handler etc.

But assuming some amount of sync, I would like:

- During busy balance only one CPU gets to proceed as per
   should_we_balance() heuristics. In addition to that, since all CPUs
   are busy (should_we_balance() would have allowed the first idle CPU
   to go ahead otherwise) the "idle_cpus" and "overloaded" situations
   may change and those are hard to propagate.

- By the time this CPU does busy balancing, other groups below it
   hopefully had enough time to reach update_sd_lb_stats() and cache
   their copy for this jiffy in there. If not - the load balancing CPU
   will recompute.

- Since stats at a higher domain is used only once, there was no need
   to invalidate them which I couldn't get right back then (or maybe
   even now :)

> 
> There are two roles,  writer who updates the cached stats,
> the reader who reads the cache stats. For both cache writer and
> the cache reader, do we trigger them only when it is in busy periodic
> load balance? If yes, consider the periodic load balance is usually
> triggered on 1 CPU in each SD(should_we_balance()),  and the
> interval increases with the number of CPUs in that sd, just wonder
> if 10 ms is a little short to find a cached stats on large LLC?

So the reader is always the CPU going to the higher domain and
recomputing stats. The writer should have updated the stats by then
or the reader won't care and recompute it.

At the very least, since the CPU has to look at local stats too, the
logic ensures at least that is reused and not recomputed.

Beyond the annotated PATCH 9, I've moved to a versioning scheme that
could also be reused for newidle balancing with stats invalidation
and that should help reuse stats more. There are some stats on the
empty PATCH 9.

-- 
Thanks and Regards,
Prateek

> thanks,
> Chenyu
> 
> 
>> Current implementation without invalidation of cached stats have few
>> limitations namely that the stats reuse is limited to busy load
>> balancing since stats can only be updated once a jiffy. Newidle Balance
>> can happen frequently and concurrently on many CPUs which can result in
>> readers seeing inconsitent values for the propagated stats.
>>
>> For this iteration, the focus is to reduce the time taken for busy load
>> balancing allowing the busy CPU to resume renning the task as quickly as
>> possible.
>>
>> Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
>> ---