Re: [RFC][PATCH] sched: Cache aware load-balancing

Chen, Yu C
2025-03-26T09:15:47Z

Hi Prateek,

On 3/26/2025 2:18 PM, K Prateek Nayak wrote:
> Hello Peter, Chenyu,
>
> On 3/26/2025 12:14 AM, Peter Zijlstra wrote:
>> On Tue, Mar 25, 2025 at 11:19:52PM +0800, Chen, Yu C wrote:
>>>
>>> Hi Peter,
>>>
>>> Thanks for sending this out,
>>>
>>> On 3/25/2025 8:09 PM, Peter Zijlstra wrote:
>>>> Hi all,
>>>>
>>>> One of the many things on the eternal todo list has been finishing the
>>>> below hackery.
>>>>
>>>> It is an attempt at modelling cache affinity -- and while the patch
>>>> really only targets LLC, it could very well be extended to also
>>>> apply to
>>>> clusters (L2). Specifically any case of multiple cache domains inside a
>>>> node.
>>>>
>>>> Anyway, I wrote this about a year ago, and I mentioned this at the
>>>> recent OSPM conf where Gautham and Prateek expressed interest in
>>>> playing
>>>> with this code.
>>>>
>>>> So here goes, very rough and largely unproven code ahead :-)
>>>>
>>>> It applies to current tip/master, but I know it will fail the __percpu
>>>> validation that sits in -next, although that shouldn't be terribly hard
>>>> to fix up.
>>>>
>>>> As is, it only computes a CPU inside the LLC that has the highest
>>>> recent
>>>> runtime, this CPU is then used in the wake-up path to steer towards
>>>> this
>>>> LLC and in task_hot() to limit migrations away from it.
>>>>
>>>> More elaborate things could be done, notably there is an XXX in there
>>>> somewhere about finding the best LLC inside a NODE (interaction with
>>>> NUMA_BALANCING).
>>>>
>>>
>>> Besides the control provided by CONFIG_SCHED_CACHE, could we also
>>> introduce
>>> sched_feat(SCHED_CACHE) to manage this feature, facilitating dynamic
>>> adjustments? Similarly we can also introduce other sched_feats for load
>>> balancing and NUMA balancing for fine-grain control.
>>
>> We can do all sorts, but the very first thing is determining if this is
>> worth it at all. Because if we can't make this work at all, all those
>> things are a waste of time.
>>
>> This patch is not meant to be merged, it is meant for testing and
>> development. We need to first make it actually improve workloads. If it
>> then turns out it regresses workloads (likely, things always do), then
>> we can look at how to best do that.
>>
>
> Thank you for sharing the patch and the initial review from Chenyu
> pointing to issues that need fixing. I'll try to take a good look at it
> this week and see if I can improve some trivial benchmarks that regress
> currently with RFC as is.
>
> In its current form I think this suffers from the same problem as
> SIS_NODE where wakeups redirect to same set of CPUs and a good deal of
> additional work is being done without any benefit.
>
> I'll leave the results from my initial testing on the 3rd Generation
> EPYC platform below and will evaluate what is making the benchmarks
> unhappy. I'll return with more data when some of these benchmarks
> are not as unhappy as they are now.
>
> Thank you both for the RFC and the initial feedback. Following are
> the initial results for the RFC as is:
>
>   ==================================================================
>   Test          : hackbench
>   Units         : Normalized time in seconds
>   Interpretation: Lower is better
>   Statistic     : AMean
>   ==================================================================
>   Case:           tip[pct imp](CV)      sched_cache[pct imp](CV)
>    1-groups     1.00 [ -0.00](10.12)     1.01 [ -0.89]( 2.84)
>    2-groups     1.00 [ -0.00]( 6.92)     1.83 [-83.15]( 1.61)
>    4-groups     1.00 [ -0.00]( 3.14)     3.00 [-200.21]( 3.13)
>    8-groups     1.00 [ -0.00]( 1.35)     3.44 [-243.75]( 2.20)
>   16-groups     1.00 [ -0.00]( 1.32)     2.59 [-158.98]( 4.29)
>
>
>   ==================================================================
>   Test          : tbench
>   Units         : Normalized throughput
>   Interpretation: Higher is better
>   Statistic     : AMean
>   ==================================================================
>   Clients:    tip[pct imp](CV)     sched_cache[pct imp](CV)
>       1     1.00 [  0.00]( 0.43)     0.96 [ -3.54]( 0.56)
>       2     1.00 [  0.00]( 0.58)     0.99 [ -1.32]( 1.40)
>       4     1.00 [  0.00]( 0.54)     0.98 [ -2.34]( 0.78)
>       8     1.00 [  0.00]( 0.49)     0.96 [ -3.91]( 0.54)
>      16     1.00 [  0.00]( 1.06)     0.97 [ -3.22]( 1.82)
>      32     1.00 [  0.00]( 1.27)     0.95 [ -4.74]( 2.05)
>      64     1.00 [  0.00]( 1.54)     0.93 [ -6.65]( 0.63)
>     128     1.00 [  0.00]( 0.38)     0.93 [ -6.91]( 1.18)
>     256     1.00 [  0.00]( 1.85)     0.99 [ -0.50]( 1.34)
>     512     1.00 [  0.00]( 0.31)     0.98 [ -2.47]( 0.14)
>    1024     1.00 [  0.00]( 0.19)     0.97 [ -3.06]( 0.39)
>
>
>   ==================================================================
>   Test          : stream-10
>   Units         : Normalized Bandwidth, MB/s
>   Interpretation: Higher is better
>   Statistic     : HMean
>   ==================================================================
>   Test:       tip[pct imp](CV)     sched_cache[pct imp](CV)
>    Copy     1.00 [  0.00](11.31)     0.34 [-65.89](72.77)
>   Scale     1.00 [  0.00]( 6.62)     0.32 [-68.09](72.49)
>     Add     1.00 [  0.00]( 7.06)     0.34 [-65.56](70.56)
>   Triad     1.00 [  0.00]( 8.91)     0.34 [-66.47](72.70)
>
>
>   ==================================================================
>   Test          : stream-100
>   Units         : Normalized Bandwidth, MB/s
>   Interpretation: Higher is better
>   Statistic     : HMean
>   ==================================================================
>   Test:       tip[pct imp](CV)     sched_cache[pct imp](CV)
>    Copy     1.00 [  0.00]( 2.01)     0.83 [-16.96](24.55)
>   Scale     1.00 [  0.00]( 1.49)     0.79 [-21.40](24.10)
>     Add     1.00 [  0.00]( 2.67)     0.79 [-21.33](25.39)
>   Triad     1.00 [  0.00]( 2.19)     0.81 [-19.19](25.55)
>
>
>   ==================================================================
>   Test          : netperf
>   Units         : Normalized Througput
>   Interpretation: Higher is better
>   Statistic     : AMean
>   ==================================================================
>   Clients:         tip[pct imp](CV)     sched_cache[pct imp](CV)
>    1-clients     1.00 [  0.00]( 1.43)     0.98 [ -2.22]( 0.26)
>    2-clients     1.00 [  0.00]( 1.02)     0.97 [ -2.55]( 0.89)
>    4-clients     1.00 [  0.00]( 0.83)     0.98 [ -2.27]( 0.46)
>    8-clients     1.00 [  0.00]( 0.73)     0.98 [ -2.45]( 0.80)
>   16-clients     1.00 [  0.00]( 0.97)     0.97 [ -2.90]( 0.88)
>   32-clients     1.00 [  0.00]( 0.88)     0.95 [ -5.29]( 1.69)
>   64-clients     1.00 [  0.00]( 1.49)     0.91 [ -8.70]( 1.95)
>   128-clients    1.00 [  0.00]( 1.05)     0.92 [ -8.39]( 4.25)
>   256-clients    1.00 [  0.00]( 3.85)     0.92 [ -8.33]( 2.45)
>   512-clients    1.00 [  0.00](59.63)     0.92 [ -7.83](51.19)
>
>
>   ==================================================================
>   Test          : schbench
>   Units         : Normalized 99th percentile latency in us
>   Interpretation: Lower is better
>   Statistic     : Median
>   ==================================================================
>   #workers: tip[pct imp](CV)       sched_cache[pct imp](CV)
>     1     1.00 [ -0.00]( 6.67)      0.38 [ 62.22]    ( 5.88)
>     2     1.00 [ -0.00](10.18)      0.43 [ 56.52]    ( 2.94)
>     4     1.00 [ -0.00]( 4.49)      0.60 [ 40.43]    ( 5.52)
>     8     1.00 [ -0.00]( 6.68)    113.96 [-11296.23] (12.91)
>    16     1.00 [ -0.00]( 1.87)    359.34 [-35834.43] (20.02)
>    32     1.00 [ -0.00]( 4.01)    217.67 [-21667.03] ( 5.48)
>    64     1.00 [ -0.00]( 3.21)     97.43 [-9643.02]  ( 4.61)
>   128     1.00 [ -0.00](44.13)     41.36 [-4036.10]  ( 6.92)
>   256     1.00 [ -0.00](14.46)      2.69 [-169.31]   ( 1.86)
>   512     1.00 [ -0.00]( 1.95)      1.89 [-89.22]    ( 2.24)
>
>
>   ==================================================================
>   Test          : new-schbench-requests-per-second
>   Units         : Normalized Requests per second
>   Interpretation: Higher is better
>   Statistic     : Median
>   ==================================================================
>   #workers: tip[pct imp](CV)      sched_cache[pct imp](CV)
>     1     1.00 [  0.00]( 0.46)     0.96 [ -4.14]( 0.00)
>     2     1.00 [  0.00]( 0.15)     0.95 [ -5.27]( 2.29)
>     4     1.00 [  0.00]( 0.15)     0.88 [-12.01]( 0.46)
>     8     1.00 [  0.00]( 0.15)     0.55 [-45.47]( 1.23)
>    16     1.00 [  0.00]( 0.00)     0.54 [-45.62]( 0.50)
>    32     1.00 [  0.00]( 3.40)     0.63 [-37.48]( 6.37)
>    64     1.00 [  0.00]( 7.09)     0.67 [-32.73]( 0.59)
>   128     1.00 [  0.00]( 0.00)     0.99 [ -0.76]( 0.34)
>   256     1.00 [  0.00]( 1.12)     1.06 [  6.32]( 1.55)
>   512     1.00 [  0.00]( 0.22)     1.06 [  6.08]( 0.92)
>
>
>   ==================================================================
>   Test          : new-schbench-wakeup-latency
>   Units         : Normalized 99th percentile latency in us
>   Interpretation: Lower is better
>   Statistic     : Median
>   ==================================================================
>   #workers: tip[pct imp](CV)       sched_cache[pct imp](CV)
>     1     1.00 [ -0.00](19.72)     0.85  [ 15.38]    ( 8.13)
>     2     1.00 [ -0.00](15.96)     1.09  [ -9.09]    (18.20)
>     4     1.00 [ -0.00]( 3.87)     1.00  [ -0.00]    ( 0.00)
>     8     1.00 [ -0.00]( 8.15)    118.17 [-11716.67] ( 0.58)
>    16     1.00 [ -0.00]( 3.87)    146.62 [-14561.54] ( 4.64)
>    32     1.00 [ -0.00](12.99)    141.60 [-14060.00] ( 5.64)
>    64     1.00 [ -0.00]( 6.20)    78.62  [-7762.50]  ( 1.79)
>   128     1.00 [ -0.00]( 0.96)    11.36  [-1036.08]  ( 3.41)
>   256     1.00 [ -0.00]( 2.76)     1.11  [-11.22]    ( 3.28)
>   512     1.00 [ -0.00]( 0.20)     1.21  [-20.81]    ( 0.91)
>
>
>   ==================================================================
>   Test          : new-schbench-request-latency
>   Units         : Normalized 99th percentile latency in us
>   Interpretation: Lower is better
>   Statistic     : Median
>   ==================================================================
>   #workers: tip[pct imp](CV)      sched_cache[pct imp](CV)
>     1     1.00 [ -0.00]( 1.07)     1.11 [-10.66]  ( 2.76)
>     2     1.00 [ -0.00]( 0.14)     1.20 [-20.40]  ( 1.73)
>     4     1.00 [ -0.00]( 1.39)     2.04 [-104.20] ( 0.96)
>     8     1.00 [ -0.00]( 0.36)     3.94 [-294.20] ( 2.85)
>    16     1.00 [ -0.00]( 1.18)     4.56 [-356.16] ( 1.19)
>    32     1.00 [ -0.00]( 8.42)     3.02 [-201.67] ( 8.93)
>    64     1.00 [ -0.00]( 4.85)     1.51 [-51.38]  ( 0.80)
>   128     1.00 [ -0.00]( 0.28)     1.83 [-82.77]  ( 1.21)
>   256     1.00 [ -0.00](10.52)     1.43 [-43.11]  (10.67)
>   512     1.00 [ -0.00]( 0.69)     1.25 [-24.96]  ( 6.24)
>
>
>   ==================================================================
>   Test          : Various longer running benchmarks
>   Units         : %diff in throughput reported
>   Interpretation: Higher is better
>   Statistic     : Median
>   ==================================================================
>   Benchmarks:                 %diff
>   ycsb-cassandra             -10.70%
>   ycsb-mongodb               -13.66%
>
>   deathstarbench-1x           13.87%
>   deathstarbench-2x            1.70%
>   deathstarbench-3x           -8.44%
>   deathstarbench-6x           -3.12%
>
>   hammerdb+mysql 16VU        -33.50%
>   hammerdb+mysql 64VU        -33.22%
>
> ---
>
> I'm planning on taking hackbench and schbench as two extreme cases for
> throughput and tail latency and later look at Stream from a "high
> bandwidth, don't consolidate" standpoint. I hope once those cases
> aren't as much in the reds, the larger benchmarks will be happier too.
>

Thanks for running the test. I think hackbenc/schbench would be the good
benchmarks to start with. I remember that you and Gautham mentioned that
schbench prefers to be aggregated in a single LLC in LPC2021 or 2022. I
ran a schbench test using mmtests on a Xeon server which has 4 NUMA
nodes. Each node has 80 cores (with SMT disabled). The numa=off option
was appended to the boot commandline, so there are 4 "LLCs" within each
node.

BASELIN SCHED_CACH
BASELINE SCHED_CACHE
Lat 50.0th-qrtle-1 8.00 ( 0.00%) 5.00 ( 37.50%)
Lat 90.0th-qrtle-1 9.00 ( 0.00%) 5.00 ( 44.44%)
Lat 99.0th-qrtle-1 13.00 ( 0.00%) 10.00 ( 23.08%)
Lat 99.9th-qrtle-1 21.00 ( 0.00%) 19.00 ( 9.52%)*
Lat 20.0th-qrtle-1 404.00 ( 0.00%) 411.00 ( -1.73%)
Lat 50.0th-qrtle-2 8.00 ( 0.00%) 5.00 ( 37.50%)
Lat 90.0th-qrtle-2 11.00 ( 0.00%) 8.00 ( 27.27%)
Lat 99.0th-qrtle-2 16.00 ( 0.00%) 11.00 ( 31.25%)
Lat 99.9th-qrtle-2 27.00 ( 0.00%) 17.00 ( 37.04%)*
Lat 20.0th-qrtle-2 823.00 ( 0.00%) 821.00 ( 0.24%)
Lat 50.0th-qrtle-4 10.00 ( 0.00%) 5.00 ( 50.00%)
Lat 90.0th-qrtle-4 12.00 ( 0.00%) 6.00 ( 50.00%)
Lat 99.0th-qrtle-4 18.00 ( 0.00%) 9.00 ( 50.00%)
Lat 99.9th-qrtle-4 29.00 ( 0.00%) 16.00 ( 44.83%)*
Lat 20.0th-qrtle-4 1650.00 ( 0.00%) 1598.00 ( 3.15%)
Lat 50.0th-qrtle-8 9.00 ( 0.00%) 4.00 ( 55.56%)
Lat 90.0th-qrtle-8 11.00 ( 0.00%) 6.00 ( 45.45%)
Lat 99.0th-qrtle-8 16.00 ( 0.00%) 9.00 ( 43.75%)
Lat 99.9th-qrtle-8 28.00 ( 0.00%) 188.00 (-571.43%)*
Lat 20.0th-qrtle-8 3316.00 ( 0.00%) 3100.00 ( 6.51%)
Lat 50.0th-qrtle-16 10.00 ( 0.00%) 5.00 ( 50.00%)
Lat 90.0th-qrtle-16 13.00 ( 0.00%) 7.00 ( 46.15%)
Lat 99.0th-qrtle-16 19.00 ( 0.00%) 12.00 ( 36.84%)
Lat 99.9th-qrtle-16 28.00 ( 0.00%) 2034.00 (-7164.29%)*
Lat 20.0th-qrtle-16 6632.00 ( 0.00%) 5800.00 ( 12.55%)
Lat 50.0th-qrtle-32 7.00 ( 0.00%) 12.00 ( -71.43%)
Lat 90.0th-qrtle-32 10.00 ( 0.00%) 62.00 (-520.00%)
Lat 99.0th-qrtle-32 14.00 ( 0.00%) 841.00 (-5907.14%)
Lat 99.9th-qrtle-32 23.00 ( 0.00%) 1862.00 (-7995.65%)*
Lat 20.0th-qrtle-32 13264.00 ( 0.00%) 10608.00 ( 20.02%)
Lat 50.0th-qrtle-64 7.00 ( 0.00%) 64.00 (-814.29%)
Lat 90.0th-qrtle-64 12.00 ( 0.00%) 709.00 (-5808.33%)
Lat 99.0th-qrtle-64 18.00 ( 0.00%) 2260.00 (-12455.56%)
Lat 99.9th-qrtle-64 26.00 ( 0.00%) 3572.00 (-13638.46%)*
Lat 20.0th-qrtle-64 26528.00 ( 0.00%) 14064.00 ( 46.98%)
Lat 50.0th-qrtle-128 7.00 ( 0.00%) 115.00 (-1542.86%)
Lat 90.0th-qrtle-128 11.00 ( 0.00%) 1626.00 (-14681.82%)
Lat 99.0th-qrtle-128 17.00 ( 0.00%) 4472.00 (-26205.88%)
Lat 99.9th-qrtle-128 27.00 ( 0.00%) 8088.00 (-29855.56%)*
Lat 20.0th-qrtle-128 53184.00 ( 0.00%) 17312.00 ( 67.45%)
Lat 50.0th-qrtle-256 172.00 ( 0.00%) 255.00 ( -48.26%)
Lat 90.0th-qrtle-256 2092.00 ( 0.00%) 1482.00 ( 29.16%)
Lat 99.0th-qrtle-256 2684.00 ( 0.00%) 3148.00 ( -17.29%)
Lat 99.9th-qrtle-256 4504.00 ( 0.00%) 6008.00 ( -33.39%)*
Lat 20.0th-qrtle-256 53056.00 ( 0.00%) 48064.00 ( 9.41%)
Lat 50.0th-qrtle-319 375.00 ( 0.00%) 478.00 ( -27.47%)
Lat 90.0th-qrtle-319 2420.00 ( 0.00%) 2244.00 ( 7.27%)
Lat 99.0th-qrtle-319 4552.00 ( 0.00%) 4456.00 ( 2.11%)
Lat 99.9th-qrtle-319 6072.00 ( 0.00%) 7656.00 ( -26.09%)*
Lat 20.0th-qrtle-319 47936.00 ( 0.00%) 47808.00 ( 0.27%)

We can see that, when the system is under-load, the 99.9th wakeup
latency improves. But when the system gets busier, say, from thread
number 8 to 319, the wakeup latency suffers.

The following change could mitigate the issue, which is intended to
avoid task migration/stacking:

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index cddd67100a91..a492463aed71 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -8801,6 +8801,7 @@ static long __migrate_degrades_locality(struct
task_struct *p, int src_cpu, int
static int select_cache_cpu(struct task_struct *p, int prev_cpu)
{
struct mm_struct *mm = p->mm;
+ struct sched_domain *sd;
int cpu;

if (!sched_feat(SCHED_CACHE))
@@ -8813,6 +8814,8 @@ static int select_cache_cpu(struct task_struct *p,
int prev_cpu)
if (cpu < 0)
return prev_cpu;

+ if (cpus_share_cache(prev_cpu, cpu))
+ return prev_cpu;

if (static_branch_likely(&sched_numa_balancing) &&
__migrate_degrades_locality(p, prev_cpu, cpu, false) > 0) {
@@ -8822,6 +8825,10 @@ static int select_cache_cpu(struct task_struct
*p, int prev_cpu)
return prev_cpu;
}

+ sd = rcu_dereference(per_cpu(sd_llc, cpu));
+ if (likely(sd))
+ return cpumask_any(sched_domain_span(sd));
+
return cpu;
}

BASELINE_s SCHED_CACHE_s
BASELINE_sc SCHED_CACHE_sc
Lat 50.0th-qrtle-1 5.00 ( 0.00%) 5.00 ( 0.00%)
Lat 90.0th-qrtle-1 8.00 ( 0.00%) 5.00 ( 37.50%)
Lat 99.0th-qrtle-1 10.00 ( 0.00%) 10.00 ( 0.00%)
Lat 99.9th-qrtle-1 20.00 ( 0.00%) 20.00 ( 0.00%)*
Lat 20.0th-qrtle-1 409.00 ( 0.00%) 406.00 ( 0.73%)
Lat 50.0th-qrtle-2 8.00 ( 0.00%) 4.00 ( 50.00%)
Lat 90.0th-qrtle-2 11.00 ( 0.00%) 5.00 ( 54.55%)
Lat 99.0th-qrtle-2 16.00 ( 0.00%) 11.00 ( 31.25%)
Lat 99.9th-qrtle-2 29.00 ( 0.00%) 16.00 ( 44.83%)*
Lat 20.0th-qrtle-2 819.00 ( 0.00%) 825.00 ( -0.73%)
Lat 50.0th-qrtle-4 10.00 ( 0.00%) 4.00 ( 60.00%)
Lat 90.0th-qrtle-4 12.00 ( 0.00%) 4.00 ( 66.67%)
Lat 99.0th-qrtle-4 18.00 ( 0.00%) 6.00 ( 66.67%)
Lat 99.9th-qrtle-4 30.00 ( 0.00%) 15.00 ( 50.00%)*
Lat 20.0th-qrtle-4 1658.00 ( 0.00%) 1670.00 ( -0.72%)
Lat 50.0th-qrtle-8 9.00 ( 0.00%) 3.00 ( 66.67%)
Lat 90.0th-qrtle-8 11.00 ( 0.00%) 4.00 ( 63.64%)
Lat 99.0th-qrtle-8 16.00 ( 0.00%) 6.00 ( 62.50%)
Lat 99.9th-qrtle-8 29.00 ( 0.00%) 13.00 ( 55.17%)*
Lat 20.0th-qrtle-8 3308.00 ( 0.00%) 3340.00 ( -0.97%)
Lat 50.0th-qrtle-16 9.00 ( 0.00%) 4.00 ( 55.56%)
Lat 90.0th-qrtle-16 12.00 ( 0.00%) 4.00 ( 66.67%)
Lat 99.0th-qrtle-16 18.00 ( 0.00%) 6.00 ( 66.67%)
Lat 99.9th-qrtle-16 31.00 ( 0.00%) 12.00 ( 61.29%)*
Lat 20.0th-qrtle-16 6616.00 ( 0.00%) 6680.00 ( -0.97%)
Lat 50.0th-qrtle-32 8.00 ( 0.00%) 4.00 ( 50.00%)
Lat 90.0th-qrtle-32 11.00 ( 0.00%) 5.00 ( 54.55%)
Lat 99.0th-qrtle-32 17.00 ( 0.00%) 8.00 ( 52.94%)
Lat 99.9th-qrtle-32 27.00 ( 0.00%) 11.00 ( 59.26%)*
Lat 20.0th-qrtle-32 13296.00 ( 0.00%) 13328.00 ( -0.24%)
Lat 50.0th-qrtle-64 9.00 ( 0.00%) 46.00 (-411.11%)
Lat 90.0th-qrtle-64 14.00 ( 0.00%) 1198.00 (-8457.14%)
Lat 99.0th-qrtle-64 20.00 ( 0.00%) 2252.00 (-11160.00%)
Lat 99.9th-qrtle-64 31.00 ( 0.00%) 2844.00 (-9074.19%)*
Lat 20.0th-qrtle-64 26528.00 ( 0.00%) 15504.00 ( 41.56%)
Lat 50.0th-qrtle-128 7.00 ( 0.00%) 26.00 (-271.43%)
Lat 90.0th-qrtle-128 11.00 ( 0.00%) 2244.00 (-20300.00%)
Lat 99.0th-qrtle-128 17.00 ( 0.00%) 4488.00 (-26300.00%)
Lat 99.9th-qrtle-128 27.00 ( 0.00%) 5752.00 (-21203.70%)*
Lat 20.0th-qrtle-128 53184.00 ( 0.00%) 24544.00 ( 53.85%)
Lat 50.0th-qrtle-256 172.00 ( 0.00%) 135.00 ( 21.51%)
Lat 90.0th-qrtle-256 2084.00 ( 0.00%) 2022.00 ( 2.98%)
Lat 99.0th-qrtle-256 2780.00 ( 0.00%) 3908.00 ( -40.58%)
Lat 99.9th-qrtle-256 4536.00 ( 0.00%) 5832.00 ( -28.57%)*
Lat 20.0th-qrtle-256 53568.00 ( 0.00%) 51904.00 ( 3.11%)
Lat 50.0th-qrtle-319 369.00 ( 0.00%) 358.00 ( 2.98%)
Lat 90.0th-qrtle-319 2428.00 ( 0.00%) 2436.00 ( -0.33%)
Lat 99.0th-qrtle-319 4552.00 ( 0.00%) 4664.00 ( -2.46%)
Lat 99.9th-qrtle-319 6104.00 ( 0.00%) 6632.00 ( -8.65%)*
Lat 20.0th-qrtle-319 48192.00 ( 0.00%) 48832.00 ( -1.33%)

We can see wakeup latency improvement in a wider range when running
different number of threads. But there is still regression starting from
thread number 64 - maybe the benefit of LLC locality is offset by the
task stacking on 1 LLC. One possible direction I'm thinking of is that,
we can get a snapshot of LLC status in load balance, check if the LLC is
overloaded, if yes, do not enable this LLC aggregation during task
wakeup - but do it in the load balancer, which is less frequent.

thanks,
Chenyu