* [PATCH v2 0/2] sched: Enable preferred SMT siblings on NVIDIA Olympus
@ 2026-09-04 9:18 Andrea Righi
2026-09-04 9:18 ` [PATCH 1/2] arm64: topology: Prefer PE0 on NVIDIA Olympus SMT cores Andrea Righi
2026-09-04 9:18 ` [PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection Andrea Righi
0 siblings, 2 replies; 6+ messages in thread
From: Andrea Righi @ 2026-09-04 9:18 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Catalin Marinas, Will Deacon
Cc: Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak, Mark Rutland,
Christian Loehle, Shrikanth Hegde, Phil Auld, Breno Leitao,
linux-arm-kernel, linux-kernel
NVIDIA Olympus implements SMT with two symmetric processing elements (PEs).
When only one PE is active, the core operates in single-thread mode and
that PE can use the full core resources. When both PEs are active, the core
operates in two-thread mode and the PEs share those resources. This
behavior is common to SMT implementations, but Olympus is particularly
sensitive to brief sibling activations because returning from two-thread
mode to single-thread mode after a sibling becomes idle is not immediate.
As described by commit 293f9611ae735 ("sched/fair: Prefer fully idle cores
for NOHZ balancing"):
Briefly activating an otherwise idle sibling can reduce the performance
available to the other sibling and this effect does not necessarily end
once the activated sibling becomes idle: after the ILB finishes and its
CPU enters WFI, full single-thread performance is restored only after the
sibling has remained idle for a qualification interval (10 Ki cycles on
the tested Vera system).
That change prevents the NOHZ idle load balancer from unnecessarily waking
a sibling of a busy PE. However, ordinary task placement can still select
either sibling of an idle core and repeated changes of the active PE can
keep Olympus cores in two-thread mode despite little or no useful overlap
between the siblings.
This series makes PE0 the preferred sibling of an Olympus core using
SD_ASYM_PACKING and teaches the fair scheduler's idle-selection paths to
honor asymmetric SMT priority. The scheduler first selects an idle core
according to its existing placement and capacity rules, then chooses the
highest-priority available sibling within that core. The generic scheduler
behavior is enabled only when an architecture supplies an SD_ASYM_PACKING
SMT domain.
PE0 and PE1 have equal steady-state capacity, the preference does not
identify a faster PE. PE0 is used only as a canonical choice when both
siblings are available. Consistently selecting the same sibling avoids
alternating the active PE across wakeups, lets PE1 remain idle for longer,
and allows more cores to remain in, or return to, full-resource
single-thread mode.
The series was tested on a two-node Vera system using an 88-thread
single-precision GEMM on the 88 physical cores of NUMA node 0.
With the workload allowed to choose either sibling of every core, observed
throughput improved from approximately 9.4 TFLOP/s on the baseline kernel
to approximately 10.1 TFLOP/s with this series applied. Repeated runs also
became more predictable because the workload consistently settled on PE0
while PE1 remained quiet.
Changes in v2:
- Clarify that the generic scheduler change also covers POWER7 (Dietmar
Eggemann)
- Simplify sched_smt_asym_prefer() by inspecting the lowest scheduling
domain directly (Dietmar Eggemann)
- Link to v1: https://lore.kernel.org/r/20260831181800.1668646-1-arighi@nvidia.com
Andrea Righi (2):
arm64: topology: Prefer PE0 on NVIDIA Olympus SMT cores
sched/fair: Honor asymmetric SMT priority in idle selection
arch/arm64/include/asm/topology.h | 1 +
arch/arm64/kernel/smp.c | 1 +
arch/arm64/kernel/topology.c | 62 ++++++++++++++++++++++++++++++
kernel/sched/fair.c | 79 ++++++++++++++++++++++++++++++++++-----
kernel/sched/sched.h | 6 +++
kernel/sched/topology.c | 36 ++++++++++++++++++
6 files changed, 176 insertions(+), 9 deletions(-)
^ permalink raw reply [flat|nested] 6+ messages in thread* [PATCH 1/2] arm64: topology: Prefer PE0 on NVIDIA Olympus SMT cores 2026-09-04 9:18 [PATCH v2 0/2] sched: Enable preferred SMT siblings on NVIDIA Olympus Andrea Righi @ 2026-09-04 9:18 ` Andrea Righi 2026-09-04 9:18 ` [PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection Andrea Righi 1 sibling, 0 replies; 6+ messages in thread From: Andrea Righi @ 2026-09-04 9:18 UTC (permalink / raw) To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot, Catalin Marinas, Will Deacon Cc: Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman, Valentin Schneider, K Prateek Nayak, Mark Rutland, Christian Loehle, Shrikanth Hegde, Phil Auld, Breno Leitao, linux-arm-kernel, linux-kernel NVIDIA Olympus implements spatial SMT with symmetric steady-state PE capacity but two different resource modes. One-Thread Active mode gives one PE the full core, while waking the other PE restores Two-Thread Active mode and partitions decode, issue, cache, TLB, and vector resources. Returning to full-resource mode requires the sibling to remain in WFI for 10 Ki cycles. Measurements show that pinned workloads perform equally on either PE, but freely migratable workloads lose substantial throughput when they alternate between PE identities. Consistently selecting PE0 keeps PE1 idle, avoids repeated SMT repartitioning, and restores one-thread-per-core performance. Describe this scheduling preference with SD_ASYM_PACKING and give PE0, identified by MPIDR_EL1.Aff0, the higher arch_asym_cpu_priority(). This is independent of SD_ASYM_CPUCAPACITY: SMT siblings retain equal capacity, while physical cores with different maximum frequencies are handled by a higher scheduling domain. Firmware currently provides no interface for describing the preferred SMT sibling. Detect Olympus by MIDR until such an interface is available. Signed-off-by: Andrea Righi <arighi@nvidia.com> --- arch/arm64/include/asm/topology.h | 1 + arch/arm64/kernel/smp.c | 1 + arch/arm64/kernel/topology.c | 62 +++++++++++++++++++++++++++++++ 3 files changed, 64 insertions(+) diff --git a/arch/arm64/include/asm/topology.h b/arch/arm64/include/asm/topology.h index b9eaf4ad70850..edc1c59b3448d 100644 --- a/arch/arm64/include/asm/topology.h +++ b/arch/arm64/include/asm/topology.h @@ -18,6 +18,7 @@ int pcibus_to_node(struct pci_bus *bus); #include <linux/arch_topology.h> void update_freq_counters_refs(void); +void arm64_init_sched_topology(void); /* Replace task scheduler's default frequency-invariant accounting */ #define arch_scale_freq_tick topology_scale_freq_tick diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index a61dc3016a117..0135ac4eea8bd 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -443,6 +443,7 @@ void __init smp_cpus_done(unsigned int max_cpus) hyp_mode_check(); setup_system_features(); setup_user_features(); + arm64_init_sched_topology(); mark_linear_text_alias_ro(); } diff --git a/arch/arm64/kernel/topology.c b/arch/arm64/kernel/topology.c index d28438f8b83f1..0dd9eec1c4946 100644 --- a/arch/arm64/kernel/topology.c +++ b/arch/arm64/kernel/topology.c @@ -19,6 +19,8 @@ #include <linux/init.h> #include <linux/percpu.h> #include <linux/sched/isolation.h> +#include <linux/sched/topology.h> +#include <linux/smp.h> #include <linux/xarray.h> #include <asm/cpu.h> @@ -44,6 +46,66 @@ static DEFINE_PER_CPU_READ_MOSTLY(unsigned long, arch_max_freq_scale) = 1UL << (2 * SCHED_CAPACITY_SHIFT); static cpumask_var_t amu_fie_cpus; +/* + * Switching the active PE on an NVIDIA Olympus SMT core can keep the core in + * two-thread active mode, with resources partitioned between the PEs. + * + * Prefer PE0 so PE1 can remain idle and the core can stay in full-resource + * mode. Firmware does not currently describe this preference, so detect + * Olympus by MIDR until a firmware interface is available. + */ +static bool olympus_prefer_pe0 __ro_after_init; + +#ifdef CONFIG_SCHED_SMT +static int arm64_smt_flags(void) +{ + int flags = cpu_smt_flags(); + + if (olympus_prefer_pe0) + flags |= SD_ASYM_PACKING; + + return flags; +} +#endif + +static struct sched_domain_topology_level arm64_asym_smt_topology[] = { +#ifdef CONFIG_SCHED_SMT + SDTL_INIT(tl_smt_mask, arm64_smt_flags, SMT), +#endif +#ifdef CONFIG_SCHED_CLUSTER + SDTL_INIT(tl_cls_mask, cpu_cluster_flags, CLS), +#endif +#ifdef CONFIG_SCHED_MC + SDTL_INIT(tl_mc_mask, cpu_core_flags, MC), +#endif + SDTL_INIT(tl_pkg_mask, NULL, PKG), + { NULL, }, +}; + +void __init arm64_init_sched_topology(void) +{ + if (!IS_ENABLED(CONFIG_SCHED_SMT)) + return; + + if ((read_cpuid_id() & MIDR_CPU_MODEL_MASK) != MIDR_NVIDIA_OLYMPUS) + return; + + if (!topology_core_has_smt(smp_processor_id())) + return; + + olympus_prefer_pe0 = true; + set_sched_topology(arm64_asym_smt_topology); + pr_info("Enabling PE0 SMT preference for NVIDIA Olympus\n"); +} + +int arch_asym_cpu_priority(int cpu) +{ + if (!olympus_prefer_pe0) + return 0; + + return MPIDR_AFFINITY_LEVEL(cpu_logical_map(cpu), 0) == 0; +} + struct amu_cntr_sample { u64 arch_const_cycles_prev; u64 arch_core_cycles_prev; -- 2.55.0 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection 2026-09-04 9:18 [PATCH v2 0/2] sched: Enable preferred SMT siblings on NVIDIA Olympus Andrea Righi 2026-09-04 9:18 ` [PATCH 1/2] arm64: topology: Prefer PE0 on NVIDIA Olympus SMT cores Andrea Righi @ 2026-09-04 9:18 ` Andrea Righi 1 sibling, 0 replies; 6+ messages in thread From: Andrea Righi @ 2026-09-04 9:18 UTC (permalink / raw) To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot, Catalin Marinas, Will Deacon Cc: Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman, Valentin Schneider, K Prateek Nayak, Mark Rutland, Christian Loehle, Shrikanth Hegde, Phil Auld, Breno Leitao, linux-arm-kernel, linux-kernel POWER7 and NVIDIA Olympus use SD_ASYM_PACKING at the shared-capacity SMT level to order hardware threads. Idle CPU selection does not consult that order, so a task can wake on an arbitrary sibling and remain there until load balancing corrects the placement. On these systems, that initial choice can prevent the core from entering its preferred lower-thread resource mode and cause a large and persistent performance loss. When idle selection finds an available CPU in an SMT core, choose the highest-priority available sibling. On SMT2 Olympus this only changes selection on fully idle cores. A partially idle core has only one available CPU. On wider SMT systems such as POWER7, it also fills available siblings in priority order while the core is partially busy. Apply the preference to idle-core and idle-CPU scans, asymmetric-capacity scans, and the target, previous, and recently-used CPU fast paths. Inspect the lowest scheduling domain directly, but require both CPUs to share its span because isolcpus can split hardware siblings across scheduling domains. Keep physical-core capacity selection independent from SMT sibling ordering. SD_ASYM_CPUCAPACITY first selects among cores with different maximum capacities, then SD_ASYM_PACKING selects the preferred available sibling inside the chosen core, whose siblings continue to share equal capacity. Signed-off-by: Andrea Righi <arighi@nvidia.com> --- kernel/sched/fair.c | 79 ++++++++++++++++++++++++++++++++++++----- kernel/sched/sched.h | 6 ++++ kernel/sched/topology.c | 36 +++++++++++++++++++ 3 files changed, 112 insertions(+), 9 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index b8bd308c2d5b1..ff9a7b1fcbe7f 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -8587,6 +8587,63 @@ static inline bool test_idle_cores(int cpu) return false; } +/* + * Return true when @cpu has a higher asymmetric-packing priority than + * @other in their shared SMT scheduling domain. + */ +static bool sched_smt_asym_prefer(int cpu, int other) +{ + struct sched_domain *sd = rcu_dereference_all(cpu_rq(cpu)->sd); + + if (!sd) + return false; + + if (!(sd->flags & SD_SHARE_CPUCAPACITY) || + !(sd->flags & SD_ASYM_PACKING)) + return false; + + if (!cpumask_test_cpu(other, sched_domain_span(sd))) + return false; + + return sched_asym_prefer(cpu, other); +} + +/* + * Return the highest-priority available CPU in @cpu's SMT core that is also in @cpus. + */ +static int __select_idle_smt_cpu(struct task_struct *p, int cpu, const struct cpumask *cpus) +{ + int best = cpu; + int sibling; + + for_each_cpu_and(sibling, cpu_smt_mask(cpu), cpus) { + if (sibling == best || !choose_idle_cpu(sibling, p)) + continue; + + if (sched_smt_asym_prefer(sibling, best)) + best = sibling; + } + + return best; +} + +static inline int +select_idle_smt_cpu(struct task_struct *p, int cpu, const struct cpumask *cpus) +{ + if (!sched_smt_asym_active()) + return cpu; + + return __select_idle_smt_cpu(p, cpu, cpus); +} + +/* + * Redirect an available SMT CPU to a higher-priority available sibling allowed by task affinity. + */ +static inline int select_idle_smt_priority(struct task_struct *p, int cpu) +{ + return select_idle_smt_cpu(p, cpu, p->cpus_ptr); +} + /* * Scans the local SMT mask to see if the entire core is idle, and records this * information in sd_balance_shared->has_idle_cores. @@ -8645,7 +8702,7 @@ static int select_idle_core(struct task_struct *p, int core, struct cpumask *cpu } if (idle) - return core; + return select_idle_smt_cpu(p, core, cpus); cpumask_andnot(cpus, cpus, cpu_smt_mask(core)); return -1; @@ -8668,7 +8725,7 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t if (!cpumask_test_cpu(cpu, sched_domain_span(sd))) continue; if (choose_idle_cpu(cpu, p)) - return cpu; + return select_idle_smt_priority(p, cpu); } return -1; @@ -8720,7 +8777,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool return -1; idle_cpu = __select_idle_cpu(cpu, p); if ((unsigned int)idle_cpu < nr_cpumask_bits) - return idle_cpu; + return select_idle_smt_priority(p, idle_cpu); } } cpumask_andnot(cpus, cpus, sched_group_span(sg)); @@ -8745,7 +8802,8 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool if (has_idle_core) set_idle_cores(target, false); - return idle_cpu; + return (unsigned int)idle_cpu < nr_cpumask_bits ? + select_idle_smt_priority(p, idle_cpu) : idle_cpu; } /* @@ -8858,7 +8916,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) * immediately. */ if (fits > 0 && preferred_core) - return cpu; + return select_idle_smt_cpu(p, cpu, cpus); /* * Only the min performance hint (i.e. uclamp_min) doesn't fit. * Look for the CPU with best capacity. @@ -8915,6 +8973,9 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) if (has_idle_core && best_fits > ASYM_IDLE_COMPLETE_MISFIT) set_idle_cores(target, false); + if (best_cpu >= 0) + best_cpu = select_idle_smt_cpu(p, best_cpu, cpus); + return best_cpu; } @@ -8971,7 +9032,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) if (choose_idle_cpu(target, p) && asym_fits_cpu(task_util, util_min, util_max, target)) - return target; + return select_idle_smt_priority(p, target); /* * If the previous CPU is cache affine and idle, don't be stupid: @@ -8982,9 +9043,9 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) if (!static_branch_unlikely(&sched_cluster_active) || cpus_share_resources(prev, target)) - return prev; + return select_idle_smt_priority(p, prev); - prev_aff = prev; + prev_aff = select_idle_smt_priority(p, prev); } /* @@ -9015,7 +9076,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) if (!static_branch_unlikely(&sched_cluster_active) || cpus_share_resources(recent_used_cpu, target)) - return recent_used_cpu; + return select_idle_smt_priority(p, recent_used_cpu); } else { recent_used_cpu = -1; diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 6c3ad70e58b8e..568cb1ed2dd6b 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2240,6 +2240,7 @@ DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing); DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity); extern struct static_key_false sched_asym_cpucapacity; +extern struct static_key_false sched_smt_asym_packing; extern struct static_key_false sched_cluster_active; static __always_inline bool sched_asym_cpucap_active(void) @@ -2247,6 +2248,11 @@ static __always_inline bool sched_asym_cpucap_active(void) return static_branch_unlikely(&sched_asym_cpucapacity); } +static __always_inline bool sched_smt_asym_active(void) +{ + return static_branch_unlikely(&sched_smt_asym_packing); +} + struct sched_group_capacity { atomic_t ref; /* diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 0248227d983a7..06c40eb5932af 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -683,8 +683,24 @@ DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing); DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity); DEFINE_STATIC_KEY_FALSE(sched_asym_cpucapacity); +DEFINE_STATIC_KEY_FALSE(sched_smt_asym_packing); DEFINE_STATIC_KEY_FALSE(sched_cluster_active); +static bool has_asym_smt_domain(int cpu) +{ + struct sched_domain *sd; + + for_each_domain(cpu, sd) { + if (!(sd->flags & SD_SHARE_CPUCAPACITY)) + break; + + if (sd->flags & SD_ASYM_PACKING) + return true; + } + + return false; +} + static void update_top_cache_domain(int cpu) { struct sched_domain_shared *sds = NULL; @@ -3084,6 +3100,7 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att struct rq *rq = NULL; int i, ret = -ENOMEM; bool has_asym = false; + bool has_asym_smt = false; bool has_cluster = false; if (WARN_ON(cpumask_empty(cpu_map))) @@ -3202,6 +3219,9 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att cpu_attach_domain(sd, d.rd, i); + if (has_asym_smt_domain(i)) + has_asym_smt = true; + if (lowest_flag_domain(i, SD_CLUSTER)) has_cluster = true; } @@ -3210,6 +3230,9 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att if (has_asym) static_branch_inc_cpuslocked(&sched_asym_cpucapacity); + if (has_asym_smt) + static_branch_inc_cpuslocked(&sched_smt_asym_packing); + if (has_cluster) static_branch_inc_cpuslocked(&sched_cluster_active); @@ -3310,11 +3333,24 @@ int __init sched_init_domains(const struct cpumask *cpu_map) static void detach_destroy_domains(const struct cpumask *cpu_map) { unsigned int cpu = cpumask_any(cpu_map); + bool has_asym_smt = false; int i; + rcu_read_lock(); + for_each_cpu(i, cpu_map) { + if (has_asym_smt_domain(i)) { + has_asym_smt = true; + break; + } + } + rcu_read_unlock(); + if (rcu_access_pointer(per_cpu(sd_asym_cpucapacity, cpu))) static_branch_dec_cpuslocked(&sched_asym_cpucapacity); + if (has_asym_smt) + static_branch_dec_cpuslocked(&sched_smt_asym_packing); + if (static_branch_unlikely(&sched_cluster_active)) static_branch_dec_cpuslocked(&sched_cluster_active); -- 2.55.0 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 0/2] sched: Enable preferred SMT siblings on NVIDIA Olympus
@ 2026-08-31 18:10 Andrea Righi
2026-08-31 18:10 ` [PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection Andrea Righi
0 siblings, 1 reply; 6+ messages in thread
From: Andrea Righi @ 2026-08-31 18:10 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Catalin Marinas, Will Deacon
Cc: Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, K Prateek Nayak, Mark Rutland,
Christian Loehle, Shrikanth Hegde, Phil Auld, Breno Leitao,
linux-arm-kernel, linux-kernel
NVIDIA Olympus implements SMT with two symmetric processing elements (PEs).
When only one PE is active, the core operates in single-thread mode and
that PE can use the full core resources. When both PEs are active, the core
operates in two-thread mode and the PEs share those resources. This
behavior is common to SMT implementations, but Olympus is particularly
sensitive to brief sibling activations because returning from two-thread
mode to single-thread mode after a sibling becomes idle is not immediate.
As described by commit 293f9611ae735 ("sched/fair: Prefer fully idle cores
for NOHZ balancing"):
Briefly activating an otherwise idle sibling can reduce the performance
available to the other sibling and this effect does not necessarily end
once the activated sibling becomes idle: after the ILB finishes and its
CPU enters WFI, full single-thread performance is restored only after the
sibling has remained idle for a qualification interval (10 Ki cycles on
the tested Vera system).
That change prevents the NOHZ idle load balancer from unnecessarily waking
a sibling of a busy PE. However, ordinary task placement can still select
either sibling of an idle core and repeated changes of the active PE can
keep Olympus cores in two-thread mode despite little or no useful overlap
between the siblings.
This series makes PE0 the preferred sibling of an Olympus core using
SD_ASYM_PACKING and teaches the fair scheduler's idle-selection paths to
honor asymmetric SMT priority. The scheduler first selects an idle core
according to its existing placement and capacity rules, then chooses the
highest-priority available sibling within that core. The generic scheduler
behavior is enabled only when an architecture supplies an SD_ASYM_PACKING
SMT domain.
PE0 and PE1 have equal steady-state capacity, the preference does not
identify a faster PE. PE0 is used only as a canonical choice when both
siblings are available. Consistently selecting the same sibling avoids
alternating the active PE across wakeups, lets PE1 remain idle for longer,
and allows more cores to remain in, or return to, full-resource
single-thread mode.
The series was tested on a two-node Vera system using an 88-thread
single-precision GEMM on the 88 physical cores of NUMA node 0.
With the workload allowed to choose either sibling of every core, observed
throughput improved from approximately 9.4 TFLOP/s on the baseline kernel
to approximately 10.1 TFLOP/s with this series applied. Repeated runs also
became more predictable because the workload consistently settled on PE0
while PE1 remained quiet.
Andrea Righi (2):
arm64: topology: Prefer PE0 on NVIDIA Olympus SMT cores
sched/fair: Honor asymmetric SMT priority in idle selection
arch/arm64/include/asm/topology.h | 1 +
arch/arm64/kernel/smp.c | 1 +
arch/arm64/kernel/topology.c | 62 +++++++++++++++++++++++++++++++++++++++
kernel/sched/fair.c | 81 +++++++++++++++++++++++++++++++++++++++++++------
kernel/sched/sched.h | 6 ++++
kernel/sched/topology.c | 36 ++++++++++++++++++++++
6 files changed, 178 insertions(+), 9 deletions(-)
^ permalink raw reply [flat|nested] 6+ messages in thread* [PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection 2026-08-31 18:10 [PATCH 0/2] sched: Enable preferred SMT siblings on NVIDIA Olympus Andrea Righi @ 2026-08-31 18:10 ` Andrea Righi 2026-09-03 10:59 ` Dietmar Eggemann 0 siblings, 1 reply; 6+ messages in thread From: Andrea Righi @ 2026-08-31 18:10 UTC (permalink / raw) To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot, Catalin Marinas, Will Deacon Cc: Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman, Valentin Schneider, K Prateek Nayak, Mark Rutland, Christian Loehle, Shrikanth Hegde, Phil Auld, Breno Leitao, linux-arm-kernel, linux-kernel SD_ASYM_PACKING orders CPUs that share an SMT core, but idle CPU selection does not consult that order. A task can therefore wake on an arbitrary sibling and remain there until load balancing corrects the placement. On SMT implementations where changing the active sibling repartitions core resources, that initial choice can cause a large and persistent performance loss. When idle selection finds an available CPU in an SMT core, choose the highest-priority available sibling. On SMT2 this only changes selection on fully idle cores, because a partially idle core has only one available CPU. On wider SMT cores it also fills available siblings in priority order while the core is partially busy. Apply the preference to idle-core and idle-CPU scans, asymmetric capacity scans, and the target, previous, and recently-used CPU fast paths. Keep physical-core capacity selection independent from SMT sibling ordering. SD_ASYM_CPUCAPACITY can first select among cores with different maximum capacities, SD_ASYM_PACKING then selects the preferred available sibling inside the chosen core, whose siblings continue to share equal capacity. Signed-off-by: Andrea Righi <arighi@nvidia.com> --- kernel/sched/fair.c | 81 ++++++++++++++++++++++++++++++++++++----- kernel/sched/sched.h | 6 +++ kernel/sched/topology.c | 36 ++++++++++++++++++ 3 files changed, 114 insertions(+), 9 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 8dff37059faf7..3c49aa63742cb 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -8587,6 +8587,65 @@ static inline bool test_idle_cores(int cpu) return false; } +/* + * Return true when @cpu has a higher asymmetric-packing priority than @other in their SMT + * scheduling domain. + */ +static bool sched_smt_asym_prefer(int cpu, int other) +{ + struct sched_domain *sd; + + for_each_domain(cpu, sd) { + /* + * Only honor priorities declared at shared-capacity SMT levels. + * SD_ASYM_PACKING at higher levels may describe core ordering. + */ + if (!(sd->flags & SD_SHARE_CPUCAPACITY)) + break; + + if ((sd->flags & SD_ASYM_PACKING) && cpumask_test_cpu(other, sched_domain_span(sd))) + return sched_asym_prefer(cpu, other); + } + + return false; +} + +/* + * Return the highest-priority available CPU in @cpu's SMT core that is also in @cpus. + */ +static int __select_idle_smt_cpu(struct task_struct *p, int cpu, const struct cpumask *cpus) +{ + int best = cpu; + int sibling; + + for_each_cpu_and(sibling, cpu_smt_mask(cpu), cpus) { + if (sibling == best || !choose_idle_cpu(sibling, p)) + continue; + + if (sched_smt_asym_prefer(sibling, best)) + best = sibling; + } + + return best; +} + +static inline int +select_idle_smt_cpu(struct task_struct *p, int cpu, const struct cpumask *cpus) +{ + if (!sched_smt_asym_active()) + return cpu; + + return __select_idle_smt_cpu(p, cpu, cpus); +} + +/* + * Redirect an available SMT CPU to a higher-priority available sibling allowed by task affinity. + */ +static inline int select_idle_smt_priority(struct task_struct *p, int cpu) +{ + return select_idle_smt_cpu(p, cpu, p->cpus_ptr); +} + /* * Scans the local SMT mask to see if the entire core is idle, and records this * information in sd_balance_shared->has_idle_cores. @@ -8645,7 +8704,7 @@ static int select_idle_core(struct task_struct *p, int core, struct cpumask *cpu } if (idle) - return core; + return select_idle_smt_cpu(p, core, cpus); cpumask_andnot(cpus, cpus, cpu_smt_mask(core)); return -1; @@ -8668,7 +8727,7 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t if (!cpumask_test_cpu(cpu, sched_domain_span(sd))) continue; if (choose_idle_cpu(cpu, p)) - return cpu; + return select_idle_smt_priority(p, cpu); } return -1; @@ -8720,7 +8779,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool return -1; idle_cpu = __select_idle_cpu(cpu, p); if ((unsigned int)idle_cpu < nr_cpumask_bits) - return idle_cpu; + return select_idle_smt_priority(p, idle_cpu); } } cpumask_andnot(cpus, cpus, sched_group_span(sg)); @@ -8745,7 +8804,8 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool if (has_idle_core) set_idle_cores(target, false); - return idle_cpu; + return (unsigned int)idle_cpu < nr_cpumask_bits ? + select_idle_smt_priority(p, idle_cpu) : idle_cpu; } /* @@ -8858,7 +8918,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) * immediately. */ if (fits > 0 && preferred_core) - return cpu; + return select_idle_smt_cpu(p, cpu, cpus); /* * Only the min performance hint (i.e. uclamp_min) doesn't fit. * Look for the CPU with best capacity. @@ -8915,6 +8975,9 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) if (has_idle_core && best_fits > ASYM_IDLE_COMPLETE_MISFIT) set_idle_cores(target, false); + if (best_cpu >= 0) + best_cpu = select_idle_smt_cpu(p, best_cpu, cpus); + return best_cpu; } @@ -8971,7 +9034,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) if (choose_idle_cpu(target, p) && asym_fits_cpu(task_util, util_min, util_max, target)) - return target; + return select_idle_smt_priority(p, target); /* * If the previous CPU is cache affine and idle, don't be stupid: @@ -8982,9 +9045,9 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) if (!static_branch_unlikely(&sched_cluster_active) || cpus_share_resources(prev, target)) - return prev; + return select_idle_smt_priority(p, prev); - prev_aff = prev; + prev_aff = select_idle_smt_priority(p, prev); } /* @@ -9015,7 +9078,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) if (!static_branch_unlikely(&sched_cluster_active) || cpus_share_resources(recent_used_cpu, target)) - return recent_used_cpu; + return select_idle_smt_priority(p, recent_used_cpu); } else { recent_used_cpu = -1; diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index e656c7059bf86..73731e9439b97 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2240,6 +2240,7 @@ DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing); DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity); extern struct static_key_false sched_asym_cpucapacity; +extern struct static_key_false sched_smt_asym_packing; extern struct static_key_false sched_cluster_active; static __always_inline bool sched_asym_cpucap_active(void) @@ -2247,6 +2248,11 @@ static __always_inline bool sched_asym_cpucap_active(void) return static_branch_unlikely(&sched_asym_cpucapacity); } +static __always_inline bool sched_smt_asym_active(void) +{ + return static_branch_unlikely(&sched_smt_asym_packing); +} + struct sched_group_capacity { atomic_t ref; /* diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 0248227d983a7..06c40eb5932af 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -683,8 +683,24 @@ DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing); DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity); DEFINE_STATIC_KEY_FALSE(sched_asym_cpucapacity); +DEFINE_STATIC_KEY_FALSE(sched_smt_asym_packing); DEFINE_STATIC_KEY_FALSE(sched_cluster_active); +static bool has_asym_smt_domain(int cpu) +{ + struct sched_domain *sd; + + for_each_domain(cpu, sd) { + if (!(sd->flags & SD_SHARE_CPUCAPACITY)) + break; + + if (sd->flags & SD_ASYM_PACKING) + return true; + } + + return false; +} + static void update_top_cache_domain(int cpu) { struct sched_domain_shared *sds = NULL; @@ -3084,6 +3100,7 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att struct rq *rq = NULL; int i, ret = -ENOMEM; bool has_asym = false; + bool has_asym_smt = false; bool has_cluster = false; if (WARN_ON(cpumask_empty(cpu_map))) @@ -3202,6 +3219,9 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att cpu_attach_domain(sd, d.rd, i); + if (has_asym_smt_domain(i)) + has_asym_smt = true; + if (lowest_flag_domain(i, SD_CLUSTER)) has_cluster = true; } @@ -3210,6 +3230,9 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att if (has_asym) static_branch_inc_cpuslocked(&sched_asym_cpucapacity); + if (has_asym_smt) + static_branch_inc_cpuslocked(&sched_smt_asym_packing); + if (has_cluster) static_branch_inc_cpuslocked(&sched_cluster_active); @@ -3310,11 +3333,24 @@ int __init sched_init_domains(const struct cpumask *cpu_map) static void detach_destroy_domains(const struct cpumask *cpu_map) { unsigned int cpu = cpumask_any(cpu_map); + bool has_asym_smt = false; int i; + rcu_read_lock(); + for_each_cpu(i, cpu_map) { + if (has_asym_smt_domain(i)) { + has_asym_smt = true; + break; + } + } + rcu_read_unlock(); + if (rcu_access_pointer(per_cpu(sd_asym_cpucapacity, cpu))) static_branch_dec_cpuslocked(&sched_asym_cpucapacity); + if (has_asym_smt) + static_branch_dec_cpuslocked(&sched_smt_asym_packing); + if (static_branch_unlikely(&sched_cluster_active)) static_branch_dec_cpuslocked(&sched_cluster_active); -- 2.55.0 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection 2026-08-31 18:10 ` [PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection Andrea Righi @ 2026-09-03 10:59 ` Dietmar Eggemann 2026-09-04 5:59 ` Andrea Righi 0 siblings, 1 reply; 6+ messages in thread From: Dietmar Eggemann @ 2026-09-03 10:59 UTC (permalink / raw) To: Andrea Righi, Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot, Catalin Marinas, Will Deacon Cc: Steven Rostedt, Ben Segall, Mel Gorman, Valentin Schneider, K Prateek Nayak, Mark Rutland, Christian Loehle, Shrikanth Hegde, Phil Auld, Breno Leitao, linux-arm-kernel, linux-kernel On 31.08.26 20:10, Andrea Righi wrote: > SD_ASYM_PACKING orders CPUs that share an SMT core, but idle CPU > selection does not consult that order. A task can therefore wake on an > arbitrary sibling and remain there until load balancing corrects the > placement. On SMT implementations where changing the active sibling > repartitions core resources, that initial choice can cause a large and > persistent performance loss. I assume this sentence refers to Olympus/Vera and Power7? [...] > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c > index 8dff37059faf7..3c49aa63742cb 100644 > --- a/kernel/sched/fair.c > +++ b/kernel/sched/fair.c > @@ -8587,6 +8587,65 @@ static inline bool test_idle_cores(int cpu) > return false; > } > > +/* > + * Return true when @cpu has a higher asymmetric-packing priority than @other in their SMT > + * scheduling domain. > + */ > +static bool sched_smt_asym_prefer(int cpu, int other) > +{ > + struct sched_domain *sd; > + > + for_each_domain(cpu, sd) { > + /* > + * Only honor priorities declared at shared-capacity SMT levels. > + * SD_ASYM_PACKING at higher levels may describe core ordering. > + */ > + if (!(sd->flags & SD_SHARE_CPUCAPACITY)) > + break; > + > + if ((sd->flags & SD_ASYM_PACKING) && cpumask_test_cpu(other, sched_domain_span(sd))) Looks like 'other' is always part of the mask? > + return sched_asym_prefer(cpu, other); > + } > + > + return false; > +} SMT will always the lowest SD, so for_each_domain() is not necessary: static bool sched_smt_asym_prefer(int cpu, int other) { struct sched_domain *sd = rcu_dereference_all(cpu_rq(cpu)->sd); if (sd && ((sd->flags & (SD_SHARE_CPUCAPACITY | SD_ASYM_PACKING)) == (SD_SHARE_CPUCAPACITY | SD_ASYM_PACKING))) return sched_asym_prefer(cpu, other); return false; } [...] > @@ -8668,7 +8727,7 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t > if (!cpumask_test_cpu(cpu, sched_domain_span(sd))) > continue; > if (choose_idle_cpu(cpu, p)) > - return cpu; > + return select_idle_smt_priority(p, cpu); This one is weird for SMT2. AFAICS, select_idle_smt() is called when there are no idle cores. So if you find an idle CPU this is what you will return anyway. I guess your tests on Olympus/Vera do wakeups via select_idle_capacity() so you haven't touched this one. [...] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection 2026-09-03 10:59 ` Dietmar Eggemann @ 2026-09-04 5:59 ` Andrea Righi 0 siblings, 0 replies; 6+ messages in thread From: Andrea Righi @ 2026-09-04 5:59 UTC (permalink / raw) To: Dietmar Eggemann Cc: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot, Catalin Marinas, Will Deacon, Steven Rostedt, Ben Segall, Mel Gorman, Valentin Schneider, K Prateek Nayak, Mark Rutland, Christian Loehle, Shrikanth Hegde, Phil Auld, Breno Leitao, linux-arm-kernel, linux-kernel Hi Dietmar, On Thu, Sep 03, 2026 at 12:59:59PM +0200, Dietmar Eggemann wrote: > On 31.08.26 20:10, Andrea Righi wrote: > > SD_ASYM_PACKING orders CPUs that share an SMT core, but idle CPU > > selection does not consult that order. A task can therefore wake on an > > arbitrary sibling and remain there until load balancing corrects the > > placement. On SMT implementations where changing the active sibling > > repartitions core resources, that initial choice can cause a large and > > persistent performance loss. > > I assume this sentence refers to Olympus/Vera and Power7? Yes. Olympus/Vera is the platform motivating this series, but this is affecting POWER7 as well, since it's also using SD_ASYM_PACKING at the SMT level. I'll change the description to make that scope explicit. > > [...] > > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c > > index 8dff37059faf7..3c49aa63742cb 100644 > > --- a/kernel/sched/fair.c > > +++ b/kernel/sched/fair.c > > @@ -8587,6 +8587,65 @@ static inline bool test_idle_cores(int cpu) > > return false; > > } > > > > +/* > > + * Return true when @cpu has a higher asymmetric-packing priority than @other in their SMT > > + * scheduling domain. > > + */ > > +static bool sched_smt_asym_prefer(int cpu, int other) > > +{ > > + struct sched_domain *sd; > > + > > + for_each_domain(cpu, sd) { > > + /* > > + * Only honor priorities declared at shared-capacity SMT levels. > > + * SD_ASYM_PACKING at higher levels may describe core ordering. > > + */ > > + if (!(sd->flags & SD_SHARE_CPUCAPACITY)) > > + break; > > + > > + if ((sd->flags & SD_ASYM_PACKING) && cpumask_test_cpu(other, sched_domain_span(sd))) > > Looks like 'other' is always part of the mask? It's always in the same hardware SMT sibling mask, but it's not necessarily in the same scheduling-domain span, isolcpus could split siblings across scheduling domain, select_idle_smt() has the same sched_domain_span() check for the same resason IIUC. > > > + return sched_asym_prefer(cpu, other); > > + } > > + > > + return false; > > +} > > SMT will always the lowest SD, so for_each_domain() is not necessary: Agreed, walking the domain hierarchy is unnecessary. I'll use the lowest domain directly. > > static bool sched_smt_asym_prefer(int cpu, int other) > { > struct sched_domain *sd = rcu_dereference_all(cpu_rq(cpu)->sd); > > if (sd && ((sd->flags & (SD_SHARE_CPUCAPACITY | > SD_ASYM_PACKING)) == (SD_SHARE_CPUCAPACITY | SD_ASYM_PACKING))) > return sched_asym_prefer(cpu, other); > > return false; > } > > [...] > > > @@ -8668,7 +8727,7 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t > > if (!cpumask_test_cpu(cpu, sched_domain_span(sd))) > > continue; > > if (choose_idle_cpu(cpu, p)) > > - return cpu; > > + return select_idle_smt_priority(p, cpu); > > This one is weird for SMT2. AFAICS, select_idle_smt() is called when > there are no idle cores. So if you find an idle CPU this is what you > will return anyway. Correcdt for SMT2: once the core is partially idle, there's only one idle sibling and the priority lookup returns that same CPU. The hook is useful for wider asym SMT though. POWER7 uses SMT4 with SD_ASYM_PACKING at the SMT level, so a partially idle core can still have multiple idle threads. In that case select_idle_smt() would otherwise return the first idle thread rather than the highest-priority one. > > I guess your tests on Olympus/Vera do wakeups via select_idle_capacity() > so you haven't touched this one. Correct, Vera also has SD_ASYM_CPUCAPACITY, so the scan path used by these tests is select_idle_capacity(). Thanks for taking a look at this! -Andrea ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2026-09-04 9:19 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-09-04 9:18 [PATCH v2 0/2] sched: Enable preferred SMT siblings on NVIDIA Olympus Andrea Righi 2026-09-04 9:18 ` [PATCH 1/2] arm64: topology: Prefer PE0 on NVIDIA Olympus SMT cores Andrea Righi 2026-09-04 9:18 ` [PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection Andrea Righi -- strict thread matches above, loose matches on Subject: below -- 2026-08-31 18:10 [PATCH 0/2] sched: Enable preferred SMT siblings on NVIDIA Olympus Andrea Righi 2026-08-31 18:10 ` [PATCH 2/2] sched/fair: Honor asymmetric SMT priority in idle selection Andrea Righi 2026-09-03 10:59 ` Dietmar Eggemann 2026-09-04 5:59 ` Andrea Righi
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox