* [PATCH v2 1/8] x86/realmode: Remove unnecessary WBINVD from AP startup trampoline
2026-10-19 1:14 [PATCH v2 0/8] x86, sched, mm: Speed up CPU hotplug on large systems Pasha Tatashin
@ 2026-10-19 1:14 ` Pasha Tatashin
2026-10-20 1:15 ` sashiko-bot
2026-10-19 1:14 ` [PATCH v2 2/8] x86/smp: Remove unnecessary WBINVD from mwait_play_dead() Pasha Tatashin
` (8 subsequent siblings)
9 siblings, 1 reply; 20+ messages in thread
From: Pasha Tatashin @ 2026-10-19 1:14 UTC (permalink / raw)
To: kees, rppt, juri.lelli, nik.borisov, tglx, vschneid,
kprateek.nayak, vincent.guittot, bp, jpoimboe, cgroups, tony.luck,
david, mingo, akpm, linux-mm, seanjc, ljs, zhangguopeng,
dietmar.eggemann, mhocko, vinicius.gomes, vbabka, linux-kernel,
peterz, surenb, hpa, x86, pasha.tatashin, dmatlack, tim.c.chen,
kexec, pratyush, dave.hansen, rostedt, gmonaco, jgross, mgorman,
graf, linux-edac, liam, ridong.chen, blum, longman, bsegall
trampoline_start executes WBINVD in 16-bit real mode before switching to
protected mode. This WBINVD was originally added in the 1990s for
Sequent NUMA-Q hardware (as noted by the comment in trampoline_32.S:
"Needed for NUMA-Q should be harmless for others"), whose cross-quad
caches came up non-coherent prior to OS initialization. Sequent NUMA-Q
support (CONFIG_X86_NUMAQ) was removed in commit b5660ba76b41
("x86, platforms: Remove NUMAQ").
Per the Intel SDM (Vol. 3A, Table 10-1, note 2) and AMD APM, the INIT
IPI used to wake secondary CPUs preserves cache contents and leaves
CR0.CD and CR0.NW unchanged (0, as already enabled by firmware or the
kernel). Neither sev_es_trampoline_start nor trampoline_start64
executes WBINVD.
On modern multi-core processors with large shared last-level caches
(such as 504 MB shared L3 per socket on Intel Xeon 6985P-C), WBINVD is
not harmless: each execution flushes and invalidates the entire
coherence domain's caches and takes ~85-115 ms per CPU while disturbing
active workloads on online CPUs.
Remove the obsolete WBINVD from trampoline_start.
2-socket, 480-CPU Intel Xeon 6985P-C (N=479, mean of 10 runs):
Offline: 40.40 s -> 40.46 s
Online: 65.83 s -> 11.14 s (-83.1%)
Total: 106.23 s -> 51.60 s (-51.4%)
Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
arch/x86/realmode/rm/trampoline_32.S | 2 --
arch/x86/realmode/rm/trampoline_64.S | 1 -
2 files changed, 3 deletions(-)
diff --git a/arch/x86/realmode/rm/trampoline_32.S b/arch/x86/realmode/rm/trampoline_32.S
index 3fad907a179f..14e56ea95416 100644
--- a/arch/x86/realmode/rm/trampoline_32.S
+++ b/arch/x86/realmode/rm/trampoline_32.S
@@ -30,8 +30,6 @@
.balign PAGE_SIZE
SYM_CODE_START(trampoline_start)
- wbinvd # Needed for NUMA-Q should be harmless for others
-
LJMPW_RM(1f)
1:
mov %cs, %ax # Code and data in the same place
diff --git a/arch/x86/realmode/rm/trampoline_64.S b/arch/x86/realmode/rm/trampoline_64.S
index 14d9c7daf90f..e5a3f03d6221 100644
--- a/arch/x86/realmode/rm/trampoline_64.S
+++ b/arch/x86/realmode/rm/trampoline_64.S
@@ -60,7 +60,6 @@
.balign PAGE_SIZE
SYM_CODE_START(trampoline_start)
cli # We should be safe anyway
- wbinvd
LJMPW_RM(1f)
1:
--
2.56.0.176.g3575e65682-goog
^ permalink raw reply related [flat|nested] 20+ messages in thread* Re: [PATCH v2 1/8] x86/realmode: Remove unnecessary WBINVD from AP startup trampoline
2026-10-19 1:14 ` [PATCH v2 1/8] x86/realmode: Remove unnecessary WBINVD from AP startup trampoline Pasha Tatashin
@ 2026-10-20 1:15 ` sashiko-bot
0 siblings, 0 replies; 20+ messages in thread
From: sashiko-bot @ 2026-10-20 1:15 UTC (permalink / raw)
To: Pasha Tatashin, kees, rppt, juri.lelli, nik.borisov, tglx,
vschneid, kprateek.nayak, vincent.guittot, bp, jpoimboe,
tony.luck, david, mingo, akpm, seanjc, ljs, zhangguopeng,
dietmar.eggemann, mhocko, vinicius.gomes, vbabka, peterz, surenb,
hpa, x86, dmatlack, tim.c.chen, kexec, pratyush, dave.hansen,
rostedt, gmonaco, jgross, mgorman, graf, liam, ridong.chen, blum,
longman, bsegall
> trampoline_start executes WBINVD in 16-bit real mode before switching to
> protected mode. This WBINVD was originally added in the 1990s for
> Sequent NUMA-Q hardware (as noted by the comment in trampoline_32.S:
> "Needed for NUMA-Q should be harmless for others"), whose cross-quad
> caches came up non-coherent prior to OS initialization. Sequent NUMA-Q
> support (CONFIG_X86_NUMAQ) was removed in commit b5660ba76b41
> ("x86, platforms: Remove NUMAQ").
>
> Per the Intel SDM (Vol. 3A, Table 10-1, note 2) and AMD APM, the INIT
> IPI used to wake secondary CPUs preserves cache contents and leaves
> CR0.CD and CR0.NW unchanged (0, as already enabled by firmware or the
> kernel). Neither sev_es_trampoline_start nor trampoline_start64
> executes WBINVD.
>
> On modern multi-core processors with large shared last-level caches
> (such as 504 MB shared L3 per socket on Intel Xeon 6985P-C), WBINVD is
> not harmless: each execution flushes and invalidates the entire
> coherence domain's caches and takes ~85-115 ms per CPU while disturbing
> active workloads on online CPUs.
>
> Remove the obsolete WBINVD from trampoline_start.
>
> 2-socket, 480-CPU Intel Xeon 6985P-C (N=479, mean of 10 runs):
> Offline: 40.40 s -> 40.46 s
> Online: 65.83 s -> 11.14 s (-83.1%)
> Total: 106.23 s -> 51.60 s (-51.4%)
>
> Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261019011446.834487-1-pasha.tatashin@soleen.com?part=1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v2 2/8] x86/smp: Remove unnecessary WBINVD from mwait_play_dead()
2026-10-19 1:14 [PATCH v2 0/8] x86, sched, mm: Speed up CPU hotplug on large systems Pasha Tatashin
2026-10-19 1:14 ` [PATCH v2 1/8] x86/realmode: Remove unnecessary WBINVD from AP startup trampoline Pasha Tatashin
@ 2026-10-19 1:14 ` Pasha Tatashin
2026-10-20 1:15 ` sashiko-bot
2026-10-19 1:14 ` [PATCH v2 3/8] x86/mce/intel: Bound cmci_rediscover() to a single topology peer Pasha Tatashin
` (7 subsequent siblings)
9 siblings, 1 reply; 20+ messages in thread
From: Pasha Tatashin @ 2026-10-19 1:14 UTC (permalink / raw)
To: kees, rppt, juri.lelli, nik.borisov, tglx, vschneid,
kprateek.nayak, vincent.guittot, bp, jpoimboe, cgroups, tony.luck,
david, mingo, akpm, linux-mm, seanjc, ljs, zhangguopeng,
dietmar.eggemann, mhocko, vinicius.gomes, vbabka, linux-kernel,
peterz, surenb, hpa, x86, pasha.tatashin, dmatlack, tim.c.chen,
kexec, pratyush, dave.hansen, rostedt, gmonaco, jgross, mgorman,
graf, linux-edac, liam, ridong.chen, blum, longman, bsegall
When a logical CPU goes offline via mwait_play_dead(), it executes
WBINVD before entering the MWAIT loop. On processors with large shared
last-level caches (such as Intel Xeon 6985P-C with 504 MB L3 per
socket), each WBINVD takes ~85-115 ms, flushes and invalidates the
entire socket's shared caches, and stalls the physical core (including
any online SMT sibling on that core) for the duration of the flush.
mwait_play_dead() places the logical processor into an architectural
MWAIT C-state—the same hardware C-states entered by intel_idle during
normal idle without executing WBINVD—where C-states deeper than C1/C1E
flush core-private L1/L2 caches to the coherent LLC on C-state entry
and maintain cache coherency while the package remains powered.
As long as at least one CPU sharing the same LLC on the package remains
online, system-wide cache flushes (wbinvd_on_all_cpus()), system sleep,
and kexec will still flush that shared LLC via the online CPU(s).
However, when the last online CPU in an LLC goes offline, no online CPU
remains to service wbinvd_on_all_cpus() or flush the package before
S3/S4 power removal. Similarly, if the target MWAIT state is C1/C1E
(which does not flush core-private caches) or cache_state_incoherent is
set (e.g., SME), the CPU must flush its caches before entering MWAIT.
Skip WBINVD in mwait_play_dead() when entering a C-state deeper than
C1/C1E while another CPU sharing the same package and LLC remains
online (unless cache_state_incoherent is set).
2-socket, 480-CPU Intel Xeon 6985P-C (N=479, mean of 10 runs):
Offline: 40.46 s -> 11.42 s (-71.8%)
Online: 11.14 s -> 11.08 s (-0.5%)
Total: 51.60 s -> 22.50 s (-56.4%)
Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
arch/x86/kernel/smpboot.c | 46 +++++++++++++++++++++++++++++++++------
1 file changed, 39 insertions(+), 7 deletions(-)
diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
index ba01a9e919b7..0dccd2f927bb 100644
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -1414,10 +1414,28 @@ void play_dead_common(void)
local_irq_disable();
}
-/*
- * We need to flush the caches before going to sleep, lest we have
- * dirty data in our caches when we come back up.
- */
+static bool mwait_play_dead_needs_wbinvd(unsigned int eax_hint)
+{
+ unsigned int cpu = smp_processor_id();
+ u32 pkg_id = topology_physical_package_id(cpu);
+ u32 llc_id = per_cpu_llc_id(cpu);
+ unsigned int i;
+
+ if (cpuhp_tasks_frozen || MWAIT_HINT2CSTATE(eax_hint) < 2 ||
+ llc_id == BAD_APICID || this_cpu_read(cache_state_incoherent)) {
+ return true;
+ }
+
+ for_each_online_cpu(i) {
+ if (topology_physical_package_id(i) == pkg_id &&
+ per_cpu_llc_id(i) == llc_id) {
+ return false;
+ }
+ }
+
+ return true;
+}
+
void __noreturn mwait_play_dead(unsigned int eax_hint)
{
struct mwait_cpu_dead *md = this_cpu_ptr(&mwait_cpu_dead);
@@ -1426,15 +1444,29 @@ void __noreturn mwait_play_dead(unsigned int eax_hint)
md->status = CPUDEAD_MWAIT_WAIT;
md->control = CPUDEAD_MWAIT_WAIT;
- wbinvd();
+ /*
+ * Entering MWAIT C-states deeper than C1/C1E/C2 writes back and
+ * invalidates core-private L1/L2 caches into the shared LLC before
+ * power-gating the core. While the CPU is still in C0 prior to MWAIT,
+ * its L1/L2 caches remain coherent and snoop external cache flushes,
+ * and once in MWAIT its L1/L2 caches are empty and all dirty lines
+ * reside in the shared LLC. Thus, as long as at least one CPU sharing
+ * this LLC remains online, any concurrent or subsequent
+ * wbinvd_on_all_cpus(), system sleep, or kexec will flush the shared
+ * LLC to memory.
+ *
+ * Only flush here during system sleep (cpuhp_tasks_frozen), when
+ * entering C1/C1E/C2, when the last online CPU in the LLC goes
+ * offline, or if the CPU's cache state is marked incoherent (e.g. SME).
+ */
+ if (mwait_play_dead_needs_wbinvd(eax_hint))
+ wbinvd();
while (1) {
/*
* The CLFLUSH is a workaround for erratum AAI65 for
* the Xeon 7400 series. It's not clear it is actually
* needed, but it should be harmless in either case.
- * The WBINVD is insufficient due to the spurious-wakeup
- * case where we return around the loop.
*/
mb();
clflush(md);
--
2.56.0.176.g3575e65682-goog
^ permalink raw reply related [flat|nested] 20+ messages in thread* Re: [PATCH v2 2/8] x86/smp: Remove unnecessary WBINVD from mwait_play_dead()
2026-10-19 1:14 ` [PATCH v2 2/8] x86/smp: Remove unnecessary WBINVD from mwait_play_dead() Pasha Tatashin
@ 2026-10-20 1:15 ` sashiko-bot
0 siblings, 0 replies; 20+ messages in thread
From: sashiko-bot @ 2026-10-20 1:15 UTC (permalink / raw)
To: Pasha Tatashin, kees, rppt, juri.lelli, nik.borisov, tglx,
vschneid, kprateek.nayak, vincent.guittot, bp, jpoimboe,
tony.luck, david, mingo, akpm, linux-mm, seanjc, ljs,
zhangguopeng, dietmar.eggemann, mhocko, vinicius.gomes, vbabka,
peterz, surenb, hpa, x86, dmatlack, tim.c.chen, kexec, pratyush,
dave.hansen, rostedt, gmonaco, jgross, mgorman, graf, liam,
ridong.chen, blum, longman, bsegall
Cc: tj, hannes, mkoutny
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] x86/smp: skipping wbinvd() in mwait_play_dead() breaks KVM wbinvd_dirty_mask
--
commit 52dafb51edf021a7aaa6017af5d2ddd2b493e51b
Author: Pasha Tatashin <pasha.tatashin@soleen.com>
x86/smp: Remove unnecessary WBINVD from mwait_play_dead()
This commit skips the wbinvd() call in mwait_play_dead() when entering an
MWAIT C-state deeper than C1/C1E while another CPU sharing the same
package and LLC remains online.
> diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
> index ba01a9e919b78..0dccd2f927bb6 100644
> --- a/arch/x86/kernel/smpboot.c
> +++ b/arch/x86/kernel/smpboot.c
> @@ -1414,10 +1414,28 @@ void play_dead_common(void)
> local_irq_disable();
> }
>
> -/*
> - * We need to flush the caches before going to sleep, lest we have
> - * dirty data in our caches when we come back up.
> - */
> +static bool mwait_play_dead_needs_wbinvd(unsigned int eax_hint)
> +{
> + unsigned int cpu = smp_processor_id();
> + u32 pkg_id = topology_physical_package_id(cpu);
> + u32 llc_id = per_cpu_llc_id(cpu);
> + unsigned int i;
> +
> + if (cpuhp_tasks_frozen || MWAIT_HINT2CSTATE(eax_hint) < 2 ||
> + llc_id == BAD_APICID || this_cpu_read(cache_state_incoherent)) {
> + return true;
> + }
> +
> + for_each_online_cpu(i) {
> + if (topology_physical_package_id(i) == pkg_id &&
> + per_cpu_llc_id(i) == llc_id) {
> + return false;
> + }
> + }
[Severity: High]
Can skipping wbinvd() in mwait_play_dead_needs_wbinvd() when another CPU in
the same LLC is online leave a KVM guest's dirty cache lines unflushed in
the shared LLC when emulating guest WBINVD?
For KVM vCPUs with non-coherent DMA enabled (need_emulate_wbinvd(vcpu)),
KVM does not use wbinvd_on_all_cpus(). Instead, kvm_arch_vcpu_load() tracks
only the specific physical CPU where the vCPU ran in
vcpu->arch.wbinvd_dirty_mask (or flushes vcpu->cpu via wbinvd_on_cpu()):
arch/x86/kvm/x86.c:kvm_arch_vcpu_load() {
...
if (need_emulate_wbinvd(vcpu)) {
if (kvm_x86_call(has_wbinvd_exit)())
cpumask_set_cpu(cpu, vcpu->arch.wbinvd_dirty_mask);
else if (vcpu->cpu != -1 && vcpu->cpu != cpu)
wbinvd_on_cpu(vcpu->cpu);
}
...
}
When the guest later executes WBINVD, kvm_emulate_wbinvd_noskip() flushes
only the CPUs recorded in vcpu->arch.wbinvd_dirty_mask and clears the mask:
arch/x86/kvm/x86.c:kvm_emulate_wbinvd_noskip() {
...
if (kvm_x86_call(has_wbinvd_exit)()) {
int cpu = get_cpu();
cpumask_set_cpu(cpu, vcpu->arch.wbinvd_dirty_mask);
wbinvd_on_cpus_mask(vcpu->arch.wbinvd_dirty_mask);
put_cpu();
cpumask_clear(vcpu->arch.wbinvd_dirty_mask);
} else
wbinvd();
...
}
wbinvd_on_cpus_mask() calls on_each_cpu_mask() ->
smp_call_function_many_cond(), which filters the target mask against
cpu_online_mask:
kernel/smp.c:smp_call_function_many_cond() {
...
if (cpumask_any_and_but(mask, cpu_online_mask, this_cpu) < nr_cpu_ids) {
cpumask_and(cpumask, mask, cpu_online_mask);
...
}
If a vCPU runs on physical CPU A (in LLC 0), migrates to CPU B (in LLC 1),
and CPU A goes offline via mwait_play_dead() while sibling CPU C in LLC 0
remains online, CPU A will now skip wbinvd() and only flush its core
caches into LLC 0 (or leave lines in shared L1/L2 if CPU C is an SMT
sibling).
If the vCPU never ran on CPU C, CPU C is not in
vcpu->arch.wbinvd_dirty_mask. When the guest subsequently executes WBINVD,
wbinvd_on_cpus_mask() ignores the offline CPU A and never sends an IPI to
CPU C, leaving the guest's dirty lines in LLC 0 unflushed to DRAM before
clearing vcpu->arch.wbinvd_dirty_mask.
> +
> + return true;
> +}
[ ... ]
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261019011446.834487-1-pasha.tatashin@soleen.com?part=2
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v2 3/8] x86/mce/intel: Bound cmci_rediscover() to a single topology peer
2026-10-19 1:14 [PATCH v2 0/8] x86, sched, mm: Speed up CPU hotplug on large systems Pasha Tatashin
2026-10-19 1:14 ` [PATCH v2 1/8] x86/realmode: Remove unnecessary WBINVD from AP startup trampoline Pasha Tatashin
2026-10-19 1:14 ` [PATCH v2 2/8] x86/smp: Remove unnecessary WBINVD from mwait_play_dead() Pasha Tatashin
@ 2026-10-19 1:14 ` Pasha Tatashin
2026-10-20 1:15 ` sashiko-bot
2026-10-19 1:14 ` [PATCH v2 4/8] x86/cacheinfo, mtrr: Skip stop_machine rendezvous in cache_ap_online() when MTRRs match Pasha Tatashin
` (6 subsequent siblings)
9 siblings, 1 reply; 20+ messages in thread
From: Pasha Tatashin @ 2026-10-19 1:14 UTC (permalink / raw)
To: kees, rppt, juri.lelli, nik.borisov, tglx, vschneid,
kprateek.nayak, vincent.guittot, bp, jpoimboe, cgroups, tony.luck,
david, mingo, akpm, linux-mm, seanjc, ljs, zhangguopeng,
dietmar.eggemann, mhocko, vinicius.gomes, vbabka, linux-kernel,
peterz, surenb, hpa, x86, pasha.tatashin, dmatlack, tim.c.chen,
kexec, pratyush, dave.hansen, rostedt, gmonaco, jgross, mgorman,
graf, linux-edac, liam, ridong.chen, blum, longman, bsegall
When a CPU goes offline, cmci_clear() disables CMCI on any MCA banks
owned by that CPU, and mce_cpu_dead() calls cmci_rediscover(). Currently
cmci_rediscover() invokes on_each_cpu(cmci_rediscover_work_func, NULL, 1),
sending a synchronous broadcast IPI to every online CPU in the system,
all of which contend on the global cmci_discover_lock while reading up
to 32 MSR_IA32_MCx_CTL2 MSRs.
Any bank disabled by cmci_clear() is either thread-private, core-scoped
(shared between SMT threads of the same physical core), L2-cluster-scoped,
die-scoped, or package-scoped:
1. Any bank index that is already owned by the target peer in its own
mce_banks_owned bitmap is private to a narrower scope than the shared
level and will be skipped by cmci_skip_bank() on the target anyway.
2. Core- or cluster-shared MCA banks can only be claimed by an online
SMT sibling or L2 cluster peer.
3. Shared die-scoped or package-scoped uncore banks can be claimed in a
single pass by one online peer on the same die or package.
Record the bitmap of banks disabled by __cmci_disable_bank() in
cmci_orphaned_banks, select a single online topology peer
cmci_orphaned_target (SMT sibling, L2 cluster peer, die peer, or package
peer) in cmci_clear(), and mask out any bank indices already owned by
that target peer. In cmci_rediscover(), return immediately when
cmci_orphaned_banks is empty; otherwise probe only cmci_orphaned_target
via a single smp_call_function_single() call.
2-socket, 480-CPU Intel Xeon 6985P-C (N=479, mean of 10 runs):
Offline: 11.42 s -> 9.63 s (-15.7%)
Online: 11.08 s -> 11.11 s
Total: 22.50 s -> 20.74 s (-7.8%)
cmci_rediscover() (cpu479 offline):
7,720.6 us -> 0.6 us (-99.99%)
Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
arch/x86/kernel/cpu/mce/intel.c | 57 +++++++++++++++++++++++++++++++--
1 file changed, 55 insertions(+), 2 deletions(-)
diff --git a/arch/x86/kernel/cpu/mce/intel.c b/arch/x86/kernel/cpu/mce/intel.c
index 4655223ba560..f57718b097d9 100644
--- a/arch/x86/kernel/cpu/mce/intel.c
+++ b/arch/x86/kernel/cpu/mce/intel.c
@@ -11,6 +11,7 @@
#include <linux/percpu.h>
#include <linux/sched.h>
#include <linux/cpumask.h>
+#include <linux/topology.h>
#include <asm/apic.h>
#include <asm/cpufeature.h>
#include <asm/cpu_device_id.h>
@@ -40,6 +41,8 @@
* owned by the offlined cpu.
*/
static DEFINE_PER_CPU(mce_banks_t, mce_banks_owned);
+static mce_banks_t cmci_orphaned_banks;
+static unsigned int cmci_orphaned_target;
/*
* cmci_discover_lock protects against parallel discovery attempts
@@ -244,6 +247,7 @@ static void cmci_claim_bank(int bank, u64 val, int bios_zero_thresh, int *bios_w
/* This CPU successfully set the enable bit. */
set_bit(bank, (void *)this_cpu_ptr(&mce_banks_owned));
+ __clear_bit(bank, cmci_orphaned_banks);
if ((val & MCI_CTL2_CMCI_THRESHOLD_MASK) == CMCI_STORM_THRESHOLD) {
pr_notice("CPU%d BANK%d CMCI inherited storm\n", smp_processor_id(), bank);
@@ -328,6 +332,7 @@ static void __cmci_disable_bank(int bank)
val &= ~MCI_CTL2_CMCI_EN;
wrmsrq(MSR_IA32_MCx_CTL2(bank), val);
__clear_bit(bank, this_cpu_ptr(mce_banks_owned));
+ __set_bit(bank, cmci_orphaned_banks);
if ((val & MCI_CTL2_CMCI_THRESHOLD_MASK) == CMCI_STORM_THRESHOLD)
cmci_storm_end(bank);
@@ -339,6 +344,8 @@ static void __cmci_disable_bank(int bank)
*/
void cmci_clear(void)
{
+ unsigned int cpu = smp_processor_id();
+ unsigned int target;
unsigned long flags;
int i;
int banks;
@@ -346,8 +353,31 @@ void cmci_clear(void)
if (!cmci_supported(&banks))
return;
raw_spin_lock_irqsave(&cmci_discover_lock, flags);
+ bitmap_zero(cmci_orphaned_banks, MAX_NR_BANKS);
+ /*
+ * A bank is shared by the CPUs of one SMT core, L2 cluster, die or
+ * package, and these scopes nest. The first online peer found from
+ * the innermost scope outwards is therefore part of every sharing
+ * scope that still has an online CPU, so it alone can claim every
+ * orphaned bank that is still claimable. If it already owns an index,
+ * the instance released here is not shared with it, hence not with
+ * any online CPU, and that index can be dropped from the rediscovery.
+ */
+ target = cpumask_any_but(topology_sibling_cpumask(cpu), cpu);
+ if (target >= nr_cpu_ids)
+ target = cpumask_any_but(topology_cluster_cpumask(cpu), cpu);
+ if (target >= nr_cpu_ids)
+ target = cpumask_any_but(topology_die_cpumask(cpu), cpu);
+ if (target >= nr_cpu_ids)
+ target = cpumask_any_but(topology_core_cpumask(cpu), cpu);
+ cmci_orphaned_target = target;
for (i = 0; i < banks; i++)
__cmci_disable_bank(i);
+ if (target < nr_cpu_ids)
+ bitmap_andnot(cmci_orphaned_banks, cmci_orphaned_banks,
+ per_cpu(mce_banks_owned, target), MAX_NR_BANKS);
+ else
+ bitmap_zero(cmci_orphaned_banks, MAX_NR_BANKS);
raw_spin_unlock_irqrestore(&cmci_discover_lock, flags);
}
@@ -360,15 +390,38 @@ static void cmci_rediscover_work_func(void *arg)
cmci_discover(banks);
}
-/* After a CPU went down cycle through all the others and rediscover */
+/*
+ * After a CPU went down, rediscover orphaned shared banks on a single
+ * topology peer (SMT sibling, L2 cluster peer, die peer, or package peer)
+ * selected in cmci_clear(). Core-private banks whose core is now completely
+ * offline cannot be claimed by any other core, so probing more than one
+ * peer is redundant.
+ */
void cmci_rediscover(void)
{
+ unsigned int target;
+ unsigned long flags;
+ bool orphaned;
int banks;
if (!cmci_supported(&banks))
return;
- on_each_cpu(cmci_rediscover_work_func, NULL, 1);
+ raw_spin_lock_irqsave(&cmci_discover_lock, flags);
+ orphaned = !bitmap_empty(cmci_orphaned_banks, MAX_NR_BANKS);
+ target = cmci_orphaned_target;
+ raw_spin_unlock_irqrestore(&cmci_discover_lock, flags);
+
+ if (!orphaned)
+ return;
+
+ if (target < nr_cpu_ids && cpu_online(target))
+ smp_call_function_single(target, cmci_rediscover_work_func,
+ NULL, 1);
+
+ raw_spin_lock_irqsave(&cmci_discover_lock, flags);
+ bitmap_zero(cmci_orphaned_banks, MAX_NR_BANKS);
+ raw_spin_unlock_irqrestore(&cmci_discover_lock, flags);
}
/*
--
2.56.0.176.g3575e65682-goog
^ permalink raw reply related [flat|nested] 20+ messages in thread* Re: [PATCH v2 3/8] x86/mce/intel: Bound cmci_rediscover() to a single topology peer
2026-10-19 1:14 ` [PATCH v2 3/8] x86/mce/intel: Bound cmci_rediscover() to a single topology peer Pasha Tatashin
@ 2026-10-20 1:15 ` sashiko-bot
0 siblings, 0 replies; 20+ messages in thread
From: sashiko-bot @ 2026-10-20 1:15 UTC (permalink / raw)
To: Pasha Tatashin, kees, rppt, juri.lelli, nik.borisov, tglx,
vschneid, kprateek.nayak, vincent.guittot, bp, jpoimboe,
tony.luck, david, mingo, akpm, linux-mm, seanjc, ljs,
zhangguopeng, dietmar.eggemann, mhocko, vinicius.gomes, vbabka,
peterz, surenb, hpa, x86, dmatlack, tim.c.chen, kexec, pratyush,
dave.hansen, rostedt, gmonaco, jgross, mgorman, graf, liam,
ridong.chen, blum, longman, bsegall
Cc: tj, hannes, mkoutny
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] x86/mce/intel: missing LLC topology level in cmci_clear() peer selection
- [Medium] x86/mce/intel: cmci_rediscover() leaves shared banks unclaimed on hybrid CPUs
--
commit cb9d97fffee1a97a8d921b6c7ed854215ee3231b
Author: Pasha Tatashin <pasha.tatashin@soleen.com>
x86/mce/intel: Bound cmci_rediscover() to a single topology peer
This commit optimizes CPU offline CMCI bank rediscovery by recording
orphaned banks in cmci_clear(), selecting a single online topology peer,
and probing only that target peer via smp_call_function_single() in
cmci_rediscover() instead of broadcasting to all online CPUs with
on_each_cpu().
> diff --git a/arch/x86/kernel/cpu/mce/intel.c b/arch/x86/kernel/cpu/mce/intel.c
> index 4655223ba5606..f57718b097d9b 100644
> --- a/arch/x86/kernel/cpu/mce/intel.c
> +++ b/arch/x86/kernel/cpu/mce/intel.c
[ ... ]
> @@ -346,8 +353,31 @@ void cmci_clear(void)
> if (!cmci_supported(&banks))
> return;
> raw_spin_lock_irqsave(&cmci_discover_lock, flags);
> + bitmap_zero(cmci_orphaned_banks, MAX_NR_BANKS);
> + /*
> + * A bank is shared by the CPUs of one SMT core, L2 cluster, die or
> + * package, and these scopes nest. The first online peer found from
> + * the innermost scope outwards is therefore part of every sharing
> + * scope that still has an online CPU, so it alone can claim every
> + * orphaned bank that is still claimable. If it already owns an index,
> + * the instance released here is not shared with it, hence not with
> + * any online CPU, and that index can be dropped from the rediscovery.
> + */
> + target = cpumask_any_but(topology_sibling_cpumask(cpu), cpu);
> + if (target >= nr_cpu_ids)
> + target = cpumask_any_but(topology_cluster_cpumask(cpu), cpu);
> + if (target >= nr_cpu_ids)
> + target = cpumask_any_but(topology_die_cpumask(cpu), cpu);
> + if (target >= nr_cpu_ids)
> + target = cpumask_any_but(topology_core_cpumask(cpu), cpu);
> + cmci_orphaned_target = target;
> for (i = 0; i < banks; i++)
> __cmci_disable_bank(i);
> + if (target < nr_cpu_ids)
> + bitmap_andnot(cmci_orphaned_banks, cmci_orphaned_banks,
> + per_cpu(mce_banks_owned, target), MAX_NR_BANKS);
[Severity: High]
Should the outward peer search in cmci_clear() also check
cpu_coregroup_mask(cpu) (or cpu_llc_shared_mask(cpu)) between
topology_cluster_cpumask(cpu) and topology_die_cpumask(cpu)?
In arch/x86/kernel/smpboot.c:set_cpu_sibling_map(), x86 populates
cpu_llc_shared_mask as a distinct topology level between
cpu_l2c_shared_mask (topology_cluster_cpumask) and topology_die_cpumask:
if ((i == cpu) || (has_smt && match_smt(c, o)))
link_mask(topology_sibling_cpumask, cpu, i);
if ((i == cpu) || (has_mp && match_llc(c, o)))
link_mask(cpu_llc_shared_mask, cpu, i);
if ((i == cpu) || (has_mp && match_l2c(c, o)))
link_mask(cpu_l2c_shared_mask, cpu, i);
if ((i == cpu) || (has_mp && match_die(c, o)))
link_mask(topology_die_cpumask, cpu, i);
On processors where cpu_coregroup_mask(cpu) is narrower than
topology_die_cpumask(cpu) (for example, processors that enumerate tiles via
CPUID leaf 0x1f tile domain or leaf 0x4 L3 cache sharing rather than die
domain, such as Meteor Lake and Arrow Lake where compute tile cores and SoC
tile LP E-cores have distinct llc_id values while sharing die_id, or Xeon
systems in SNC mode), offlining the last online CPU of an L2 cluster while
other CPUs in the same LLC/tile remain online will skip directly to
topology_die_cpumask(cpu).
If cpumask_any_but(topology_die_cpumask(cpu), cpu) picks a target on a
different LLC/tile, could an MCA bank scoped to the offlined CPU's LLC/tile
either be masked out of cmci_orphaned_banks by bitmap_andnot() (because the
remote-tile target owns its own tile's instance of that bank index) or fail
to be claimed by the remote target in cmci_rediscover(), leaving the bank
with CMCI disabled and unpolled while online peers in cpu_coregroup_mask(cpu)
are never probed?
> + else
> + bitmap_zero(cmci_orphaned_banks, MAX_NR_BANKS);
> raw_spin_unlock_irqrestore(&cmci_discover_lock, flags);
> }
>
> @@ -360,15 +390,38 @@ static void cmci_rediscover_work_func(void *arg)
> cmci_discover(banks);
> }
>
> -/* After a CPU went down cycle through all the others and rediscover */
> +/*
> + * After a CPU went down, rediscover orphaned shared banks on a single
> + * topology peer (SMT sibling, L2 cluster peer, die peer, or package peer)
> + * selected in cmci_clear(). Core-private banks whose core is now completely
> + * offline cannot be claimed by any other core, so probing more than one
> + * peer is redundant.
> + */
> void cmci_rediscover(void)
> {
> + unsigned int target;
> + unsigned long flags;
> + bool orphaned;
> int banks;
>
> if (!cmci_supported(&banks))
> return;
>
> - on_each_cpu(cmci_rediscover_work_func, NULL, 1);
> + raw_spin_lock_irqsave(&cmci_discover_lock, flags);
> + orphaned = !bitmap_empty(cmci_orphaned_banks, MAX_NR_BANKS);
> + target = cmci_orphaned_target;
> + raw_spin_unlock_irqrestore(&cmci_discover_lock, flags);
> +
> + if (!orphaned)
> + return;
> +
> + if (target < nr_cpu_ids && cpu_online(target))
> + smp_call_function_single(target, cmci_rediscover_work_func,
> + NULL, 1);
> +
> + raw_spin_lock_irqsave(&cmci_discover_lock, flags);
> + bitmap_zero(cmci_orphaned_banks, MAX_NR_BANKS);
> + raw_spin_unlock_irqrestore(&cmci_discover_lock, flags);
> }
[Severity: Medium]
Can probing only a single target CPU in cmci_rediscover() leave shared MCA
banks unclaimed on hybrid processors where CPUs on the same die or package
have different MCA bank counts or per-bank CMCI capabilities?
In cmci_rediscover_work_func(), cmci_supported(&banks) re-reads
MSR_IA32_MCG_CAP on the target CPU because CPUs do not necessarily all have
the same bank count:
/* Recheck banks in case CPUs don't all have the same */
if (cmci_supported(&banks))
cmci_discover(banks);
On hybrid systems (such as Alder Lake, Raptor Lake, Meteor Lake, or Arrow
Lake), P-cores and E-cores share topology_die_cpumask(cpu) and
topology_core_cpumask(cpu) while exposing different bank counts or bank
layouts. When the last thread of a P-core goes offline through
mce_cpu_pre_down() -> mce_disable_cpu() -> cmci_clear() and
mce_cpu_dead() -> cmci_rediscover(), cpumask_any_but() on the die or package
mask can select an E-core with a smaller bank count as cmci_orphaned_target
even when another P-core is still online.
When that happens, cmci_discover(banks) on target only iterates up to target's
bank count, and cmci_rediscover() then unconditionally clears
cmci_orphaned_banks with bitmap_zero() without falling back to other online
CPUs if unclaimed bits remain. Since other CPUs sharing those higher-numbered
banks already cleared them from their per-CPU mce_poll_banks bitmaps in
cmci_skip_bank() during initial discovery, wouldn't those shared banks remain
both unowned for CMCI and unpolled?
титься
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261019011446.834487-1-pasha.tatashin@soleen.com?part=3
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v2 4/8] x86/cacheinfo, mtrr: Skip stop_machine rendezvous in cache_ap_online() when MTRRs match
2026-10-19 1:14 [PATCH v2 0/8] x86, sched, mm: Speed up CPU hotplug on large systems Pasha Tatashin
` (2 preceding siblings ...)
2026-10-19 1:14 ` [PATCH v2 3/8] x86/mce/intel: Bound cmci_rediscover() to a single topology peer Pasha Tatashin
@ 2026-10-19 1:14 ` Pasha Tatashin
2026-10-20 1:15 ` sashiko-bot
2026-10-19 1:14 ` [PATCH v2 5/8] sched/topology: Avoid redundant O(N^3) group capacity walks and unused allocations in build_sched_domains() Pasha Tatashin
` (5 subsequent siblings)
9 siblings, 1 reply; 20+ messages in thread
From: Pasha Tatashin @ 2026-10-19 1:14 UTC (permalink / raw)
To: kees, rppt, juri.lelli, nik.borisov, tglx, vschneid,
kprateek.nayak, vincent.guittot, bp, jpoimboe, cgroups, tony.luck,
david, mingo, akpm, linux-mm, seanjc, ljs, zhangguopeng,
dietmar.eggemann, mhocko, vinicius.gomes, vbabka, linux-kernel,
peterz, surenb, hpa, x86, pasha.tatashin, dmatlack, tim.c.chen,
kexec, pratyush, dave.hansen, rostedt, gmonaco, jgross, mgorman,
graf, linux-edac, liam, ridong.chen, blum, longman, bsegall
When a single CPU is brought online at runtime
(cache_aps_delayed_init == false), cache_ap_online() unconditionally
calls stop_machine_from_inactive_cpu() to rendezvous all online CPUs
while the incoming AP runs cache_cpu_init() to disable caches
(CR0.CD = 1, plus wbinvd() when X86_FEATURE_SELFSNOOP is not set),
reprogram MTRRs via mtrr_generic_set_state(), and re-enable caches.
During initial boot (cache_aps_init()) and across warm CPU offline/online
cycles, the AP's hardware MTRR MSRs have already been programmed to match
the saved mtrr_state. Stopping all online CPUs in stop_machine (and
flushing caches on the AP) when the AP's MTRRs already match mtrr_state
is unnecessary.
Add mtrr_generic_state_matches() to compare the local CPU's
MSR_MTRRdefType, variable MTRR ranges, and fixed MTRR ranges against
mtrr_state (including k8_check_syscfg_dram_mod_en() on AMD), and check it
on the AP in cache_ap_online() before invoking
stop_machine_from_inactive_cpu(). If the hardware MTRRs already match
mtrr_state, initialize PAT locally on the AP and return.
2-socket, 480-CPU Intel Xeon 6985P-C (N=479, mean of 10 runs):
Offline: 9.63 s -> 9.85 s
Online: 11.11 s -> 8.69 s (-21.8%)
Total: 20.74 s -> 18.54 s (-10.6%)
cache_ap_online() (cpu479 online):
7,704.0 us -> 4.7 us (-99.94%)
2-socket, 512-CPU AMD EPYC 9B45 (N=511, mean of 10 runs):
Offline: 4.53 s -> 4.51 s (-0.4%)
Online: 3.92 s -> 3.15 s (-19.6%)
Total: 8.45 s -> 7.66 s (-9.3%)
cache_ap_online() (cpu511 online):
1,354.6 us -> 6.4 us (-99.53%)
Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
arch/x86/include/asm/mtrr.h | 2 ++
arch/x86/kernel/cpu/cacheinfo.c | 7 +++++
arch/x86/kernel/cpu/mtrr/generic.c | 49 ++++++++++++++++++++++++++++++
3 files changed, 58 insertions(+)
diff --git a/arch/x86/include/asm/mtrr.h b/arch/x86/include/asm/mtrr.h
index 76b95bd1a405..0527e216808c 100644
--- a/arch/x86/include/asm/mtrr.h
+++ b/arch/x86/include/asm/mtrr.h
@@ -61,6 +61,7 @@ extern int amd_special_default_mtrr(void);
void mtrr_disable(void);
void mtrr_enable(void);
void mtrr_generic_set_state(void);
+bool mtrr_generic_state_matches(void);
# else
static inline void guest_force_mtrr_state(struct mtrr_var_range *var,
unsigned int num_var,
@@ -106,6 +107,7 @@ static inline int mtrr_trim_uncached_memory(unsigned long end_pfn)
#define mtrr_disable() do {} while (0)
#define mtrr_enable() do {} while (0)
#define mtrr_generic_set_state() do {} while (0)
+static inline bool mtrr_generic_state_matches(void) { return true; }
# endif
#ifdef CONFIG_COMPAT
diff --git a/arch/x86/kernel/cpu/cacheinfo.c b/arch/x86/kernel/cpu/cacheinfo.c
index 13ed16527905..64796d5b34fd 100644
--- a/arch/x86/kernel/cpu/cacheinfo.c
+++ b/arch/x86/kernel/cpu/cacheinfo.c
@@ -770,6 +770,13 @@ static int cache_ap_online(unsigned int cpu)
if (!memory_caching_control || get_cache_aps_delayed_init())
return 0;
+ if (!(memory_caching_control & CACHE_MTRR) ||
+ mtrr_generic_state_matches()) {
+ if (memory_caching_control & CACHE_PAT)
+ pat_cpu_init();
+ return 0;
+ }
+
/*
* Ideally we should hold mtrr_mutex here to avoid MTRR entries
* changed, but this routine will be called in CPU boot time,
diff --git a/arch/x86/kernel/cpu/mtrr/generic.c b/arch/x86/kernel/cpu/mtrr/generic.c
index 67cf69f24b00..8760a7b4dc38 100644
--- a/arch/x86/kernel/cpu/mtrr/generic.c
+++ b/arch/x86/kernel/cpu/mtrr/generic.c
@@ -981,6 +981,55 @@ void mtrr_generic_set_state(void)
}
}
+bool mtrr_generic_state_matches(void)
+{
+ struct msr val;
+ unsigned int i;
+
+ if (!cpu_feature_enabled(X86_FEATURE_MTRR))
+ return true;
+
+ rdmsrq(MSR_MTRRdefType, val.q);
+ if ((val.l & MTRR_DEF_TYPE_TYPE) != mtrr_state.def_type ||
+ ((val.l & MTRR_DEF_TYPE_ENABLE) >> MTRR_STATE_SHIFT) != mtrr_state.enabled)
+ return false;
+
+ for (i = 0; i < num_var_ranges; i++) {
+ struct mtrr_var_range *vr = &mtrr_state.var_ranges[i];
+
+ rdmsrq(MTRRphysBase_MSR(i), val.q);
+ if ((vr->base_lo & ~MTRR_PHYSBASE_RSVD) != (val.l & ~MTRR_PHYSBASE_RSVD) ||
+ (vr->base_hi & ~phys_hi_rsvd) != (val.h & ~phys_hi_rsvd))
+ return false;
+
+ rdmsrq(MTRRphysMask_MSR(i), val.q);
+ if ((vr->mask_lo & ~MTRR_PHYSMASK_RSVD) != (val.l & ~MTRR_PHYSMASK_RSVD) ||
+ (vr->mask_hi & ~phys_hi_rsvd) != (val.h & ~phys_hi_rsvd))
+ return false;
+ }
+
+ if (mtrr_state.have_fixed) {
+ u64 *saved = (u64 *)mtrr_state.fixed_ranges;
+ int block = -1, range;
+
+ /*
+ * Ensure MtrrFixDramModEn is clear so fixed MTRR reads do not
+ * expose K8 RdMem/WrMem bits and cause false mismatches.
+ */
+ k8_check_syscfg_dram_mod_en();
+
+ while (fixed_range_blocks[++block].ranges) {
+ for (range = 0; range < fixed_range_blocks[block].ranges; range++) {
+ rdmsrq(fixed_range_blocks[block].base_msr + range, val.q);
+ if (val.q != *saved++)
+ return false;
+ }
+ }
+ }
+
+ return true;
+}
+
/**
* generic_set_mtrr - set variable MTRR register on the local CPU.
*
--
2.56.0.176.g3575e65682-goog
^ permalink raw reply related [flat|nested] 20+ messages in thread* Re: [PATCH v2 4/8] x86/cacheinfo, mtrr: Skip stop_machine rendezvous in cache_ap_online() when MTRRs match
2026-10-19 1:14 ` [PATCH v2 4/8] x86/cacheinfo, mtrr: Skip stop_machine rendezvous in cache_ap_online() when MTRRs match Pasha Tatashin
@ 2026-10-20 1:15 ` sashiko-bot
0 siblings, 0 replies; 20+ messages in thread
From: sashiko-bot @ 2026-10-20 1:15 UTC (permalink / raw)
To: Pasha Tatashin, kees, rppt, juri.lelli, nik.borisov, tglx,
vschneid, kprateek.nayak, vincent.guittot, bp, jpoimboe,
tony.luck, david, mingo, akpm, seanjc, ljs, zhangguopeng,
dietmar.eggemann, mhocko, vinicius.gomes, vbabka, peterz, surenb,
hpa, x86, dmatlack, tim.c.chen, kexec, pratyush, dave.hansen,
rostedt, gmonaco, jgross, mgorman, graf, liam, ridong.chen, blum,
longman, bsegall
> When a single CPU is brought online at runtime
> (cache_aps_delayed_init == false), cache_ap_online() unconditionally
> calls stop_machine_from_inactive_cpu() to rendezvous all online CPUs
> while the incoming AP runs cache_cpu_init() to disable caches
> (CR0.CD = 1, plus wbinvd() when X86_FEATURE_SELFSNOOP is not set),
> reprogram MTRRs via mtrr_generic_set_state(), and re-enable caches.
>
> During initial boot (cache_aps_init()) and across warm CPU offline/online
> cycles, the AP's hardware MTRR MSRs have already been programmed to match
> the saved mtrr_state. Stopping all online CPUs in stop_machine (and
> flushing caches on the AP) when the AP's MTRRs already match mtrr_state
> is unnecessary.
>
> Add mtrr_generic_state_matches() to compare the local CPU's
> MSR_MTRRdefType, variable MTRR ranges, and fixed MTRR ranges against
> [ ... ]
> Total: 8.45 s -> 7.66 s (-9.3%)
> cache_ap_online() (cpu511 online):
> 1,354.6 us -> 6.4 us (-99.53%)
>
> Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261019011446.834487-1-pasha.tatashin@soleen.com?part=4
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v2 5/8] sched/topology: Avoid redundant O(N^3) group capacity walks and unused allocations in build_sched_domains()
2026-10-19 1:14 [PATCH v2 0/8] x86, sched, mm: Speed up CPU hotplug on large systems Pasha Tatashin
` (3 preceding siblings ...)
2026-10-19 1:14 ` [PATCH v2 4/8] x86/cacheinfo, mtrr: Skip stop_machine rendezvous in cache_ap_online() when MTRRs match Pasha Tatashin
@ 2026-10-19 1:14 ` Pasha Tatashin
2026-10-20 1:15 ` sashiko-bot
2026-10-19 1:14 ` [PATCH v2 6/8] sched/deadline, topology: Avoid O(N^2) dl_server bandwidth walks during root_domain rebuilds Pasha Tatashin
` (4 subsequent siblings)
9 siblings, 1 reply; 20+ messages in thread
From: Pasha Tatashin @ 2026-10-19 1:14 UTC (permalink / raw)
To: kees, rppt, juri.lelli, nik.borisov, tglx, vschneid,
kprateek.nayak, vincent.guittot, bp, jpoimboe, cgroups, tony.luck,
david, mingo, akpm, linux-mm, seanjc, ljs, zhangguopeng,
dietmar.eggemann, mhocko, vinicius.gomes, vbabka, linux-kernel,
peterz, surenb, hpa, x86, pasha.tatashin, dmatlack, tim.c.chen,
kexec, pratyush, dave.hansen, rostedt, gmonaco, jgross, mgorman,
graf, linux-edac, liam, ridong.chen, blum, longman, bsegall
During sequential CPU hotplug on large systems, build_sched_domains() is
invoked for every CPU offline and online step. Two bottlenecks in the
domain construction path scale poorly with CPU count:
1. On non-overlapping topology levels, all CPUs sharing a domain span
share the same circular ring of sched_group instances, and on
overlapping NUMA levels, groups with identical spans share
sched_group_capacity (sgc). However, init_sched_groups_capacity()
walks the entire group ring for every CPU in the domain and repeatedly
recomputes group_weight, cores (walking the group cpumask and clearing
cpu_smt_mask), and asym_prefer_cpu (O(N^3) bitmap operations per
hotplug step). Similarly, init_overlap_sched_group() recomputes
build_balance_mask() and cpumask_weight(sg_span) even when the shared
sgc was already initialized.
2. __sdt_alloc() and __sds_alloc() upfront-allocate a sched_group,
sched_group_capacity, and sched_domain_shared for every CPU in cpu_map
at every topology level, even though get_group() only uses the entry
for the first CPU of each child domain and d.sds is only used once per
LLC (plus at most once per SD_ASYM_CPUCAPACITY_FULL domain). All other
per-CPU allocations are immediately freed unused by
__free_domain_allocs().
Cache group_weight, cores, and asym_prefer_cpu on struct
sched_group_capacity so init_sched_groups_capacity() computes them once
per shared group capacity, reuse already-initialized sgc instances in
init_overlap_sched_group(), and allocate sdd->sg, sdd->sgc, and d->sds
lazily on demand.
2-socket, 480-CPU Intel Xeon 6985P-C (N=479, mean of 10 runs):
Offline: 9.85 s -> 5.15 s (-47.7%)
Online: 8.69 s -> 4.05 s (-53.4%)
Total: 18.54 s -> 9.20 s (-50.4%)
build_sched_domains() (cpu479 offline / online):
35.65 ms -> 10.39 ms (-70.9%) / 35.68 ms -> 9.58 ms (-73.2%)
2-socket, 512-CPU AMD EPYC 9B45 (N=511, mean of 10 runs):
Offline: 4.51 s -> 3.26 s (-27.7%)
Online: 3.15 s -> 2.03 s (-35.6%)
Total: 7.66 s -> 5.29 s (-30.9%)
build_sched_domains() (cpu511 offline / online):
11.69 ms -> 5.61 ms (-52.0%) / 12.24 ms -> 5.84 ms (-52.3%)
Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
kernel/sched/sched.h | 9 ++
kernel/sched/topology.c | 218 ++++++++++++++++++++++++++--------------
2 files changed, 150 insertions(+), 77 deletions(-)
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 499a5649582e..314fcad94668 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2262,6 +2262,15 @@ struct sched_group_capacity {
int id;
+ /*
+ * Build-time cache shared by all groups using this sgc while
+ * build_sched_domains() runs; not maintained afterwards (e.g. by
+ * sched_update_asym_prefer_cpu()). Use the sched_group copies.
+ */
+ unsigned int group_weight;
+ unsigned int cores;
+ int asym_prefer_cpu;
+
unsigned long cpumask[]; /* Balance mask */
};
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 919d0fb00bd9..a5a5d1378ce2 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -1207,6 +1207,33 @@ int group_balance_cpu(struct sched_group *sg)
*/
+/* Find the group_balance_cpu() */
+static int find_group_balance_cpu(struct sched_domain *sd, struct sched_group *sg)
+{
+ const struct cpumask *sg_span = sched_group_span(sg);
+ struct sd_data *sdd = sd->private;
+ struct sched_domain *sibling;
+ int i;
+
+ for_each_cpu(i, sg_span) {
+ sibling = *per_cpu_ptr(sdd->sd, i);
+
+ /*
+ * Can happen in the asymmetric case, where these siblings are
+ * unused. The mask will not be empty because those CPUs that
+ * do have the top domain _should_ span the domain.
+ */
+ if (!sibling->child)
+ continue;
+
+ /* If we would not end up here, we can't continue from here */
+ if (cpumask_equal(sg_span, sched_domain_span(sibling->child)))
+ return i;
+ }
+
+ return nr_cpu_ids;
+}
+
/*
* Build the balance mask; it contains only those CPUs that can arrive at this
* group and should be considered to continue balancing.
@@ -1216,17 +1243,19 @@ int group_balance_cpu(struct sched_group *sg)
* can fully construct this using the sched_domain bits (which are already
* complete).
*/
-static void
-build_balance_mask(struct sched_domain *sd, struct sched_group *sg, struct cpumask *mask)
+static void build_balance_mask(int group_balance_cpu,
+ struct sched_domain *sd,
+ struct sched_group *sg,
+ struct cpumask *mask)
{
const struct cpumask *sg_span = sched_group_span(sg);
struct sd_data *sdd = sd->private;
struct sched_domain *sibling;
- int i;
+ int i = group_balance_cpu;
cpumask_clear(mask);
- for_each_cpu(i, sg_span) {
+ for_each_cpu_from(i, sg_span) {
sibling = *per_cpu_ptr(sdd->sd, i);
/*
@@ -1277,32 +1306,51 @@ build_group_from_child_sched_domain(struct sched_domain *sd, int cpu)
return sg;
}
-static void init_overlap_sched_group(struct sched_domain *sd,
- struct sched_group *sg)
+static int init_overlap_sched_group(struct sched_domain *sd,
+ struct sched_group *sg)
{
struct cpumask *mask = sched_domains_tmpmask2;
struct sd_data *sdd = sd->private;
- struct cpumask *sg_span;
+ struct cpumask *sg_span = sched_group_span(sg);
+ struct sched_group_capacity *sgc;
int cpu;
- build_balance_mask(sd, sg, mask);
- cpu = cpumask_first(mask);
+ cpu = find_group_balance_cpu(sd, sg);
+ if (WARN_ON_ONCE(cpu >= nr_cpu_ids))
+ return -ENOENT;
- sg->sgc = *per_cpu_ptr(sdd->sgc, cpu);
- if (atomic_inc_return(&sg->sgc->ref) == 1)
- cpumask_copy(group_balance_mask(sg), mask);
- else
- WARN_ON_ONCE(!cpumask_equal(group_balance_mask(sg), mask));
+ sgc = *per_cpu_ptr(sdd->sgc, cpu);
+ if (sgc && atomic_read(&sgc->ref)) {
+ atomic_inc(&sgc->ref);
+ sg->sgc = sgc;
+ return 0;
+ }
+
+ if (!sgc) {
+ sgc = kzalloc_node(sizeof(*sgc) + cpumask_size(),
+ GFP_KERNEL, cpu_to_node(cpu));
+ if (!sgc)
+ return -ENOMEM;
+ sgc->id = cpu;
+ *per_cpu_ptr(sdd->sgc, cpu) = sgc;
+ }
+
+ build_balance_mask(cpu, sd, sg, mask);
+
+ sg->sgc = sgc;
+ atomic_set(&sgc->ref, 1);
+ cpumask_copy(group_balance_mask(sg), mask);
/*
* Initialize sgc->capacity such that even if we mess up the
* domains and no possible iteration will get us here, we won't
* die on a /0 trap.
*/
- sg_span = sched_group_span(sg);
- sg->sgc->capacity = SCHED_CAPACITY_SCALE * cpumask_weight(sg_span);
- sg->sgc->min_capacity = SCHED_CAPACITY_SCALE;
- sg->sgc->max_capacity = SCHED_CAPACITY_SCALE;
+ sgc->capacity = SCHED_CAPACITY_SCALE * cpumask_weight(sg_span);
+ sgc->min_capacity = SCHED_CAPACITY_SCALE;
+ sgc->max_capacity = SCHED_CAPACITY_SCALE;
+
+ return 0;
}
static struct sched_domain *
@@ -1405,7 +1453,10 @@ build_overlap_sched_groups(struct sched_domain *sd, int cpu)
sg_span = sched_group_span(sg);
cpumask_or(covered, covered, sg_span);
- init_overlap_sched_group(sibling, sg);
+ if (init_overlap_sched_group(sibling, sg)) {
+ kfree(sg);
+ goto fail;
+ }
if (!first)
first = sg;
@@ -1507,6 +1558,25 @@ static struct sched_group *get_group(int cpu, struct sd_data *sdd)
cpu = cpumask_first(sched_domain_span(child));
sg = *per_cpu_ptr(sdd->sg, cpu);
+ if (!sg) {
+ sg = kzalloc_node(sizeof(*sg) + cpumask_size(),
+ GFP_KERNEL, cpu_to_node(cpu));
+ if (!sg)
+ return NULL;
+ sg->next = sg;
+ *per_cpu_ptr(sdd->sg, cpu) = sg;
+ }
+
+ if (!*per_cpu_ptr(sdd->sgc, cpu)) {
+ struct sched_group_capacity *sgc;
+
+ sgc = kzalloc_node(sizeof(*sgc) + cpumask_size(),
+ GFP_KERNEL, cpu_to_node(cpu));
+ if (!sgc)
+ return NULL;
+ sgc->id = cpu;
+ *per_cpu_ptr(sdd->sgc, cpu) = sgc;
+ }
sg->sgc = *per_cpu_ptr(sdd->sgc, cpu);
/* Increase refcounts for claim_allocations: */
@@ -1562,6 +1632,8 @@ build_sched_groups(struct sched_domain *sd, int cpu)
continue;
sg = get_group(i, sdd);
+ if (!sg)
+ return -ENOMEM;
cpumask_or(covered, covered, sched_group_span(sg));
@@ -1595,27 +1667,40 @@ static void init_sched_groups_capacity(int cpu, struct sched_domain *sd)
WARN_ON(!sg);
do {
- int cpu, cores = 0, max_cpu = -1;
+ int i, cores = 0, max_cpu = -1;
+
+ if (sg->group_weight)
+ goto next;
+
+ if (sg->sgc->group_weight) {
+ sg->group_weight = sg->sgc->group_weight;
+ sg->cores = sg->sgc->cores;
+ sg->asym_prefer_cpu = sg->sgc->asym_prefer_cpu;
+ goto next;
+ }
sg->group_weight = cpumask_weight(sched_group_span(sg));
cpumask_copy(mask, sched_group_span(sg));
- for_each_cpu(cpu, mask) {
+ for_each_cpu(i, mask) {
cores++;
- cpumask_andnot(mask, mask, cpu_smt_mask(cpu));
+ cpumask_andnot(mask, mask, cpu_smt_mask(i));
}
sg->cores = cores;
- if (!(sd->flags & SD_ASYM_PACKING))
- goto next;
-
- for_each_cpu(cpu, sched_group_span(sg)) {
- if (max_cpu < 0)
- max_cpu = cpu;
- else if (sched_asym_prefer(cpu, max_cpu))
- max_cpu = cpu;
+ if (sd->flags & SD_ASYM_PACKING) {
+ for_each_cpu(i, sched_group_span(sg)) {
+ if (max_cpu < 0)
+ max_cpu = i;
+ else if (sched_asym_prefer(i, max_cpu))
+ max_cpu = i;
+ }
+ sg->asym_prefer_cpu = max_cpu;
}
- sg->asym_prefer_cpu = max_cpu;
+
+ sg->sgc->group_weight = sg->group_weight;
+ sg->sgc->cores = sg->cores;
+ sg->sgc->asym_prefer_cpu = sg->asym_prefer_cpu;
next:
sg = sg->next;
@@ -1889,7 +1974,8 @@ static void claim_allocations(int cpu, struct s_data *d)
{
struct sched_domain *sd;
- if (atomic_read(&(*per_cpu_ptr(d->sds, cpu))->ref))
+ if (*per_cpu_ptr(d->sds, cpu) &&
+ atomic_read(&(*per_cpu_ptr(d->sds, cpu))->ref))
*per_cpu_ptr(d->sds, cpu) = NULL;
for (sd = *per_cpu_ptr(d->sd, cpu); sd; sd = sd->parent) {
@@ -1898,10 +1984,12 @@ static void claim_allocations(int cpu, struct s_data *d)
WARN_ON_ONCE(*per_cpu_ptr(sdd->sd, cpu) != sd);
*per_cpu_ptr(sdd->sd, cpu) = NULL;
- if (atomic_read(&(*per_cpu_ptr(sdd->sg, cpu))->ref))
+ if (*per_cpu_ptr(sdd->sg, cpu) &&
+ atomic_read(&(*per_cpu_ptr(sdd->sg, cpu))->ref))
*per_cpu_ptr(sdd->sg, cpu) = NULL;
- if (atomic_read(&(*per_cpu_ptr(sdd->sgc, cpu))->ref))
+ if (*per_cpu_ptr(sdd->sgc, cpu) &&
+ atomic_read(&(*per_cpu_ptr(sdd->sgc, cpu))->ref))
*per_cpu_ptr(sdd->sgc, cpu) = NULL;
}
}
@@ -2700,8 +2788,6 @@ static int __sdt_alloc(const struct cpumask *cpu_map)
for_each_cpu(j, cpu_map) {
struct sched_domain *sd;
- struct sched_group *sg;
- struct sched_group_capacity *sgc;
sd = kzalloc_node(sizeof(struct sched_domain) + cpumask_size(),
GFP_KERNEL, cpu_to_node(j));
@@ -2709,24 +2795,6 @@ static int __sdt_alloc(const struct cpumask *cpu_map)
return -ENOMEM;
*per_cpu_ptr(sdd->sd, j) = sd;
-
- sg = kzalloc_node(sizeof(struct sched_group) + cpumask_size(),
- GFP_KERNEL, cpu_to_node(j));
- if (!sg)
- return -ENOMEM;
-
- sg->next = sg;
-
- *per_cpu_ptr(sdd->sg, j) = sg;
-
- sgc = kzalloc_node(sizeof(struct sched_group_capacity) + cpumask_size(),
- GFP_KERNEL, cpu_to_node(j));
- if (!sgc)
- return -ENOMEM;
-
- sgc->id = j;
-
- *per_cpu_ptr(sdd->sgc, j) = sgc;
}
}
@@ -2767,23 +2835,10 @@ static void __sdt_free(const struct cpumask *cpu_map)
static int __sds_alloc(struct s_data *d, const struct cpumask *cpu_map)
{
- int j;
-
d->sds = alloc_percpu(struct sched_domain_shared *);
if (!d->sds)
return -ENOMEM;
- for_each_cpu(j, cpu_map) {
- struct sched_domain_shared *sds;
-
- sds = kzalloc_node(sizeof(struct sched_domain_shared),
- GFP_KERNEL, cpu_to_node(j));
- if (!sds)
- return -ENOMEM;
-
- *per_cpu_ptr(d->sds, j) = sds;
- }
-
return 0;
}
@@ -2954,7 +3009,7 @@ static void adjust_numa_imbalance(struct sched_domain *sd_llc)
}
}
-static void
+static int
init_sched_domain_shared(struct s_data *d, struct sched_domain *sd, int flags)
{
struct sched_domain_shared *sds = NULL;
@@ -2976,6 +3031,13 @@ init_sched_domain_shared(struct s_data *d, struct sched_domain *sd, int flags)
*/
for_each_cpu(cpu, sched_domain_span(sd)) {
sds = *per_cpu_ptr(d->sds, cpu);
+ if (!sds) {
+ sds = kzalloc_node(sizeof(*sds),
+ GFP_KERNEL, cpu_to_node(cpu));
+ if (!sds)
+ return -ENOMEM;
+ *per_cpu_ptr(d->sds, cpu) = sds;
+ }
if (!sds->alloc_flags ||
sd->span_weight == 1 ||
@@ -3000,6 +3062,8 @@ init_sched_domain_shared(struct s_data *d, struct sched_domain *sd, int flags)
*/
atomic_set(&sd->shared->nr_busy_cpus, sd->span_weight);
atomic_inc(&sd->shared->ref);
+
+ return 0;
}
/*
@@ -3015,25 +3079,24 @@ init_sched_domain_shared(struct s_data *d, struct sched_domain *sd, int flags)
* degenerate into a single-capacity group. The relevant edge cases are instead
* covered by the caveats above.
*
- * Return true if this CPU's asym path claimed sd->shared, false otherwise.
+ * Return 0 on success or if not claimed, -ENOMEM on allocation failure.
*/
-static bool claim_asym_sched_domain_shared(struct s_data *d, int cpu)
+static int claim_asym_sched_domain_shared(struct s_data *d, int cpu)
{
struct sched_domain *sd = *per_cpu_ptr(d->sd, cpu);
struct sched_domain *sd_asym;
if (!sd)
- return false;
+ return 0;
sd_asym = sd;
while (sd_asym && !(sd_asym->flags & SD_ASYM_CPUCAPACITY_FULL))
sd_asym = sd_asym->parent;
if (!sd_asym || (sd_asym->flags & SD_NUMA))
- return false;
+ return 0;
- init_sched_domain_shared(d, sd_asym, SD_ASYM_CPUCAPACITY);
- return true;
+ return init_sched_domain_shared(d, sd_asym, SD_ASYM_CPUCAPACITY);
}
static int __sched_domains_alloc_llc_id(void)
@@ -3179,15 +3242,16 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
if (!sd)
continue;
- if (has_asym)
- claim_asym_sched_domain_shared(&d, i);
+ if (has_asym && claim_asym_sched_domain_shared(&d, i))
+ goto error;
/* First, find the topmost SD_SHARE_LLC domain */
while (sd->parent && (sd->parent->flags & SD_SHARE_LLC))
sd = sd->parent;
if (sd->flags & SD_SHARE_LLC) {
- init_sched_domain_shared(&d, sd, SD_SHARE_LLC);
+ if (init_sched_domain_shared(&d, sd, SD_SHARE_LLC))
+ goto error;
/*
* In presence of higher domains, adjust the
--
2.56.0.176.g3575e65682-goog
^ permalink raw reply related [flat|nested] 20+ messages in thread* Re: [PATCH v2 5/8] sched/topology: Avoid redundant O(N^3) group capacity walks and unused allocations in build_sched_domains()
2026-10-19 1:14 ` [PATCH v2 5/8] sched/topology: Avoid redundant O(N^3) group capacity walks and unused allocations in build_sched_domains() Pasha Tatashin
@ 2026-10-20 1:15 ` sashiko-bot
0 siblings, 0 replies; 20+ messages in thread
From: sashiko-bot @ 2026-10-20 1:15 UTC (permalink / raw)
To: Pasha Tatashin, kees, rppt, juri.lelli, nik.borisov, tglx,
vschneid, kprateek.nayak, vincent.guittot, bp, jpoimboe,
tony.luck, david, mingo, akpm, seanjc, ljs, zhangguopeng,
dietmar.eggemann, mhocko, vinicius.gomes, vbabka, peterz, surenb,
hpa, x86, dmatlack, tim.c.chen, kexec, pratyush, dave.hansen,
rostedt, gmonaco, jgross, mgorman, graf, liam, ridong.chen, blum,
longman, bsegall
> During sequential CPU hotplug on large systems, build_sched_domains() is
> invoked for every CPU offline and online step. Two bottlenecks in the
> domain construction path scale poorly with CPU count:
>
> 1. On non-overlapping topology levels, all CPUs sharing a domain span
> share the same circular ring of sched_group instances, and on
> overlapping NUMA levels, groups with identical spans share
> sched_group_capacity (sgc). However, init_sched_groups_capacity()
> walks the entire group ring for every CPU in the domain and repeatedly
> recomputes group_weight, cores (walking the group cpumask and clearing
> cpu_smt_mask), and asym_prefer_cpu (O(N^3) bitmap operations per
> hotplug step). Similarly, init_overlap_sched_group() recomputes
> build_balance_mask() and cpumask_weight(sg_span) even when the shared
> sgc was already initialized.
>
> [ ... ]
> Total: 7.66 s -> 5.29 s (-30.9%)
> build_sched_domains() (cpu511 offline / online):
> 11.69 ms -> 5.61 ms (-52.0%) / 12.24 ms -> 5.84 ms (-52.3%)
>
> Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261019011446.834487-1-pasha.tatashin@soleen.com?part=5
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v2 6/8] sched/deadline, topology: Avoid O(N^2) dl_server bandwidth walks during root_domain rebuilds
2026-10-19 1:14 [PATCH v2 0/8] x86, sched, mm: Speed up CPU hotplug on large systems Pasha Tatashin
` (4 preceding siblings ...)
2026-10-19 1:14 ` [PATCH v2 5/8] sched/topology: Avoid redundant O(N^3) group capacity walks and unused allocations in build_sched_domains() Pasha Tatashin
@ 2026-10-19 1:14 ` Pasha Tatashin
2026-10-20 1:15 ` sashiko-bot
2026-10-19 1:14 ` [PATCH v2 7/8] sched/topology: Avoid redundant percpu allocations and bitmap walks in build_sched_domains() Pasha Tatashin
` (3 subsequent siblings)
9 siblings, 1 reply; 20+ messages in thread
From: Pasha Tatashin @ 2026-10-19 1:14 UTC (permalink / raw)
To: kees, rppt, juri.lelli, nik.borisov, tglx, vschneid,
kprateek.nayak, vincent.guittot, bp, jpoimboe, cgroups, tony.luck,
david, mingo, akpm, linux-mm, seanjc, ljs, zhangguopeng,
dietmar.eggemann, mhocko, vinicius.gomes, vbabka, linux-kernel,
peterz, surenb, hpa, x86, pasha.tatashin, dmatlack, tim.c.chen,
kexec, pratyush, dave.hansen, rostedt, gmonaco, jgross, mgorman,
graf, linux-edac, liam, ridong.chen, blum, longman, bsegall
Every CPU runqueue has a fair_server (and optionally an ext_server)
deadline entity enabled by default. Whenever a root_domain is rebuilt on
CPU hotplug or cpuset changes, partition_sched_domains_locked() currently
performs three O(N^2) remote runqueue bandwidth updates across all N
active CPUs (O(N^3) across sequential hotplug of N CPUs):
1. detach_destroy_domains() calls
cpu_attach_domain(NULL, &def_root_domain, i) -> rq_attach_root() ->
__dl_server_attach_root() for each of the N CPUs. Each call recomputes
cpumask_weight_and(rd->span, cpu_active_mask) twice and calls
__dl_add() -> __dl_update(), walking all active CPUs attached to
def_root_domain so far to update rq->dl.extra_bw, only for
def_root_domain to be detached milliseconds later.
2. build_sched_domains() calls cpu_attach_domain(sd, d.rd, i) ->
rq_attach_root() -> __dl_server_attach_root() for each of the N CPUs,
performing another N*(N+1)/2 remote rq->dl.extra_bw updates using
intermediate partial span weights (1..N) that are immediately
discarded at the end of partition_sched_domains_locked().
3. dl_rebuild_rd_accounting() -> dl_clear_root_domain() resets
rd->dl_bw.total_bw = 0 and cpu_rq(i)->dl.extra_bw = max_bw, and then
calls dl_server_add_bw(rd, i) -> __dl_add() -> __dl_update() in a
nested O(N^2) loop over rd->span with interrupts disabled.
Remove the redundant __dl_server_attach_root() calls from rq_attach_root()
(since partition_sched_domains_locked() always rebuilds root_domain
accounting via dl_rebuild_rd_accounting() once domain attachment is
complete), and replace the nested O(N^2) loop in dl_clear_root_domain()
with a single O(N) accumulation pass and a single O(N) extra_bw update
pass.
Also provide a !CONFIG_CPUSETS implementation of
dl_rebuild_rd_accounting() (using a lockless dl_prio(READ_ONCE(p->prio))
pre-check before dl_add_task_root_domain(p)). Previously,
dl_rebuild_rd_accounting() was a no-op stub when CONFIG_CPUSETS=n, which
left root_domain rebuilds without SCHED_DEADLINE task bandwidth
accounting and left inconsistent per-CPU extra_bw values computed from
partial active CPU counts in rq_attach_root().
2-socket, 480-CPU Intel Xeon 6985P-C (N=479, mean of 10 runs):
Offline: 5.15 s -> 3.88 s (-24.7%)
Online: 4.05 s -> 2.51 s (-38.0%)
Total: 9.20 s -> 6.39 s (-30.5%)
dl_clear_root_domain() / dl_rebuild_rd_accounting() (cpu479):
5,537.0 us -> 58.1 us (-99.0%) / 5,176.2 us -> 61.9 us (-98.8%)
2-socket, 512-CPU AMD EPYC 9B45 (N=511, mean of 10 runs):
Offline: 3.26 s -> 2.77 s (-15.0%)
Online: 2.03 s -> 1.53 s (-24.6%)
Total: 5.29 s -> 4.30 s (-18.7%)
dl_clear_root_domain() / dl_rebuild_rd_accounting() (cpu511):
1,880.5 us -> 27.5 us (-98.5%) / 1,762.3 us -> 22.6 us (-98.7%)
Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
include/linux/cpuset.h | 23 ++++++---
include/linux/sched/deadline.h | 1 -
kernel/sched/deadline.c | 93 +++++++++++++++++++++-------------
kernel/sched/sched.h | 1 -
kernel/sched/topology.c | 13 -----
5 files changed, 73 insertions(+), 58 deletions(-)
diff --git a/include/linux/cpuset.h b/include/linux/cpuset.h
index 914983387e95..b99f329551fa 100644
--- a/include/linux/cpuset.h
+++ b/include/linux/cpuset.h
@@ -129,7 +129,7 @@ static inline int cpuset_do_page_mem_spread(void)
extern bool current_cpuset_is_being_rebound(void);
-extern void dl_rebuild_rd_accounting(void);
+void dl_rebuild_rd_accounting(void);
extern void rebuild_sched_domains(void);
extern void cpuset_print_current_mems_allowed(void);
@@ -196,9 +196,20 @@ static inline void cpuset_update_active_cpus(void)
static inline void inc_dl_tasks_cs(struct task_struct *task) { }
static inline void dec_dl_tasks_cs(struct task_struct *task) { }
-static inline void cpuset_lock(void) { }
-static inline void cpuset_unlock(void) { }
-static inline void lockdep_assert_cpuset_lock_held(void) { }
+static inline void cpuset_lock(void)
+{
+ sched_domains_mutex_lock();
+}
+
+static inline void cpuset_unlock(void)
+{
+ sched_domains_mutex_unlock();
+}
+
+static inline void lockdep_assert_cpuset_lock_held(void)
+{
+ lockdep_assert_held(&sched_domains_mutex);
+}
static inline void cpuset_cpus_allowed_locked(struct task_struct *p,
struct cpumask *mask)
@@ -273,9 +284,7 @@ static inline bool current_cpuset_is_being_rebound(void)
return false;
}
-static inline void dl_rebuild_rd_accounting(void)
-{
-}
+void dl_rebuild_rd_accounting(void);
static inline void rebuild_sched_domains(void)
{
diff --git a/include/linux/sched/deadline.h b/include/linux/sched/deadline.h
index 273538200a44..6f03357f05c7 100644
--- a/include/linux/sched/deadline.h
+++ b/include/linux/sched/deadline.h
@@ -31,7 +31,6 @@ static inline bool dl_time_before(u64 a, u64 b)
struct root_domain;
extern void dl_add_task_root_domain(struct task_struct *p);
-extern void dl_clear_root_domain(struct root_domain *rd);
extern void dl_clear_root_domain_cpu(int cpu);
/*
* Return whether moving DL task @p to @new_mask requires moving DL
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index c0ebdcde5fe5..c8847e1ea55a 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -1885,24 +1885,6 @@ void sched_init_dl_servers(void)
}
}
-void __dl_server_attach_root(struct sched_dl_entity *dl_se, struct rq *rq)
-{
- u64 new_bw = dl_se->dl_bw;
- int cpu = cpu_of(rq);
- struct dl_bw *dl_b;
-
- if (!dl_se->dl_bw_attached)
- return;
-
- dl_b = dl_bw_of(cpu_of(rq));
- guard(raw_spinlock)(&dl_b->lock);
-
- if (!dl_bw_cpus(cpu))
- return;
-
- __dl_add(dl_b, new_bw, dl_bw_cpus(cpu));
-}
-
int dl_server_apply_params(struct sched_dl_entity *dl_se, u64 runtime, u64 period, bool init)
{
u64 old_bw = (init || !dl_se->dl_bw_attached) ? 0 :
@@ -3433,7 +3415,8 @@ void dl_add_task_root_domain(struct task_struct *p)
struct cpumask *msk;
raw_spin_lock_irqsave(&p->pi_lock, rf.flags);
- if (!dl_task(p) || dl_entity_is_special(&p->dl)) {
+ if (!dl_task(p) || dl_entity_is_special(&p->dl) ||
+ (READ_ONCE(p->__state) == TASK_DEAD && !cpu_active(task_cpu(p)))) {
raw_spin_unlock_irqrestore(&p->pi_lock, rf.flags);
return;
}
@@ -3451,18 +3434,22 @@ void dl_add_task_root_domain(struct task_struct *p)
raw_spin_unlock_irqrestore(&p->pi_lock, rf.flags);
}
-static void dl_server_add_bw(struct root_domain *rd, int cpu)
+static void dl_server_sum_bw(int cpu, int cpus, u64 *total_bw, s64 *extra_bw)
{
struct sched_dl_entity *dl_se;
dl_se = &cpu_rq(cpu)->fair_server;
- if (dl_server(dl_se) && dl_se->dl_bw_attached && cpu_active(cpu))
- __dl_add(&rd->dl_bw, dl_se->dl_bw, dl_bw_cpus(cpu));
+ if (dl_server(dl_se) && dl_se->dl_bw_attached) {
+ *total_bw += dl_se->dl_bw;
+ *extra_bw -= (s32)dl_se->dl_bw / cpus;
+ }
#ifdef CONFIG_SCHED_CLASS_EXT
dl_se = &cpu_rq(cpu)->ext_server;
- if (dl_server(dl_se) && dl_se->dl_bw_attached && cpu_active(cpu))
- __dl_add(&rd->dl_bw, dl_se->dl_bw, dl_bw_cpus(cpu));
+ if (dl_server(dl_se) && dl_se->dl_bw_attached) {
+ *total_bw += dl_se->dl_bw;
+ *extra_bw -= (s32)dl_se->dl_bw / cpus;
+ }
#endif
}
@@ -3483,26 +3470,33 @@ static u64 dl_server_read_bw(int cpu)
return dl_bw;
}
-void dl_clear_root_domain(struct root_domain *rd)
+static void dl_clear_root_domain(struct root_domain *rd)
{
- int i;
+ u64 total_bw = 0;
+ s64 extra_bw = 0;
+ int i, cpus;
guard(raw_spinlock_irqsave)(&rd->dl_bw.lock);
- /*
- * Reset total_bw to zero and extra_bw to max_bw so that next
- * loop will add dl-servers contributions back properly,
- */
- rd->dl_bw.total_bw = 0;
- for_each_cpu(i, rd->span)
- cpu_rq(i)->dl.extra_bw = cpu_rq(i)->dl.max_bw;
-
/*
* dl_servers are not tasks. Since dl_add_task_root_domain ignores
* them, we need to account for them here explicitly.
*/
- for_each_cpu(i, rd->span)
- dl_server_add_bw(rd, i);
+ cpus = cpumask_weight_and(rd->span, cpu_active_mask);
+ if (cpus) {
+ for_each_cpu_and(i, rd->span, cpu_active_mask)
+ dl_server_sum_bw(i, cpus, &total_bw, &extra_bw);
+ }
+
+ rd->dl_bw.total_bw = total_bw;
+ for_each_cpu(i, rd->span) {
+ struct rq *rq = cpu_rq(i);
+ u64 bw = rq->dl.max_bw;
+
+ if (cpu_active(i))
+ bw += extra_bw;
+ WRITE_ONCE(rq->dl.extra_bw, bw);
+ }
}
void dl_clear_root_domain_cpu(int cpu)
@@ -3510,6 +3504,33 @@ void dl_clear_root_domain_cpu(int cpu)
dl_clear_root_domain(cpu_rq(cpu)->rd);
}
+#ifndef CONFIG_CPUSETS
+void dl_rebuild_rd_accounting(void)
+{
+ struct task_struct *g, *p;
+ u64 cookie = ++dl_cookie;
+ int cpu;
+
+ lockdep_assert_cpus_held();
+ lockdep_assert_held(&sched_domains_mutex);
+
+ rcu_read_lock();
+ for_each_possible_cpu(cpu) {
+ if (dl_bw_visited(cpu, cookie))
+ continue;
+
+ dl_clear_root_domain_cpu(cpu);
+ }
+ for_each_process_thread(g, p) {
+ /* Racy pre-check; dl_add_task_root_domain() rechecks under pi_lock. */
+ if (!dl_prio(READ_ONCE(p->prio)))
+ continue;
+ dl_add_task_root_domain(p);
+ }
+ rcu_read_unlock();
+}
+#endif
+
static void switched_from_dl(struct rq *rq, struct task_struct *p)
{
/*
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 314fcad94668..3225cda0f9f6 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -418,7 +418,6 @@ extern void sched_init_dl_servers(void);
extern void fair_server_init(struct rq *rq);
extern void ext_server_init(struct rq *rq);
-extern void __dl_server_attach_root(struct sched_dl_entity *dl_se, struct rq *rq);
extern int dl_server_apply_params(struct sched_dl_entity *dl_se,
u64 runtime, u64 period, bool init);
extern int dl_server_attach_bw(struct sched_dl_entity *dl_se);
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index a5a5d1378ce2..b5dec7f486b5 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -516,19 +516,6 @@ void rq_attach_root(struct rq *rq, struct root_domain *rd)
if (cpumask_test_cpu(rq->cpu, cpu_active_mask))
set_rq_online(rq);
- /*
- * Because the rq is not a task, dl_add_task_root_domain() did not
- * move the fair server bw to the rd if it already started.
- * Add it now.
- */
- if (rq->fair_server.dl_server)
- __dl_server_attach_root(&rq->fair_server, rq);
-
-#ifdef CONFIG_SCHED_CLASS_EXT
- if (rq->ext_server.dl_server)
- __dl_server_attach_root(&rq->ext_server, rq);
-#endif
-
rq_unlock_irqrestore(rq, &rf);
if (old_rd)
--
2.56.0.176.g3575e65682-goog
^ permalink raw reply related [flat|nested] 20+ messages in thread* Re: [PATCH v2 6/8] sched/deadline, topology: Avoid O(N^2) dl_server bandwidth walks during root_domain rebuilds
2026-10-19 1:14 ` [PATCH v2 6/8] sched/deadline, topology: Avoid O(N^2) dl_server bandwidth walks during root_domain rebuilds Pasha Tatashin
@ 2026-10-20 1:15 ` sashiko-bot
0 siblings, 0 replies; 20+ messages in thread
From: sashiko-bot @ 2026-10-20 1:15 UTC (permalink / raw)
To: Pasha Tatashin, kees, rppt, juri.lelli, nik.borisov, tglx,
vschneid, kprateek.nayak, vincent.guittot, bp, jpoimboe,
tony.luck, david, mingo, akpm, linux-mm, seanjc, ljs,
zhangguopeng, dietmar.eggemann, mhocko, vinicius.gomes, vbabka,
peterz, surenb, hpa, x86, dmatlack, tim.c.chen, kexec, pratyush,
dave.hansen, rostedt, gmonaco, jgross, mgorman, graf, liam,
ridong.chen, blum, longman, bsegall
Cc: tj, hannes, mkoutny
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] sched/deadline: dl_rebuild_rd_accounting() drops TASK_DEAD inactive_timer bw
- [High] sched/deadline: unskipped zero-dl_bw tasks in dl_add_task_root_domain()
--
commit e4cc8155c8ec900e6f7e52f8baebd0b6ab370fff
Author: Pasha Tatashin <pasha.tatashin@soleen.com>
sched/deadline, topology: Avoid O(N^2) dl_server bandwidth walks during root_domain rebuilds
This commit removes redundant __dl_server_attach_root() calls from
rq_attach_root() and replaces the nested O(N^2) loop in dl_clear_root_domain()
with an O(N) accumulation and update pass. It also adds a !CONFIG_CPUSETS
implementation of dl_rebuild_rd_accounting() and skips TASK_DEAD tasks on
inactive CPUs in dl_add_task_root_domain().
> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index c0ebdcde5fe5d..c8847e1ea55a0 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
[ ... ]
> @@ -3433,7 +3415,8 @@ void dl_add_task_root_domain(struct task_struct *p)
> struct cpumask *msk;
>
> raw_spin_lock_irqsave(&p->pi_lock, rf.flags);
> - if (!dl_task(p) || dl_entity_is_special(&p->dl)) {
> + if (!dl_task(p) || dl_entity_is_special(&p->dl) ||
> + (READ_ONCE(p->__state) == TASK_DEAD && !cpu_active(task_cpu(p)))) {
> raw_spin_unlock_irqrestore(&p->pi_lock, rf.flags);
> return;
> }
[Severity: High]
Can this trigger the BUG_ON(cpu >= nr_cpu_ids) just below in
dl_add_task_root_domain() for tasks where dl_task(p) is true, but
p->dl.dl_bw == 0?
During CPU hotplug or sched domain rebuilds, partition_sched_domains_locked()
calls dl_rebuild_rd_accounting() -> dl_add_task_root_domain(p) on tasks where
dl_prio(p->prio) / dl_task(p) is true:
kernel/sched/deadline.c:dl_add_task_root_domain() {
...
msk = this_cpu_cpumask_var_ptr(local_cpu_mask_dl);
dl_get_task_effective_cpus(p, msk);
cpu = cpumask_first_and(cpu_active_mask, msk);
BUG_ON(cpu >= nr_cpu_ids);
}
In dl_get_task_effective_cpus(), if p->cpus_ptr only intersects with
isolcpus=domain CPUs, it returns the active isolated CPUs:
kernel/sched/deadline.c:dl_get_task_effective_cpus() {
hk_msk = housekeeping_cpumask(HK_TYPE_DOMAIN);
if (housekeeping_enabled(HK_TYPE_DOMAIN)) {
if (!cpumask_intersects(p->cpus_ptr, hk_msk)) {
cpumask_andnot(cpus, cpu_active_mask, hk_msk);
return;
}
}
...
}
Normally, dl_bw_deactivate() blocks deactivating the last CPU in that domain
while reserved task bandwidth (total_bw - dl_server_bw) is non-zero. However,
dl_task(p) can be true while p->dl.dl_bw == 0 in two situations:
1. A non-deadline task (!task_has_dl_policy(p)) is PI-boosted by a
SCHED_DEADLINE waiter via rt_mutex_setprio(), which sets
p->prio = MAX_DL_PRIO - 1 while p->dl.dl_bw remains 0.
2. An exiting SCHED_DEADLINE task (p->__state == TASK_DEAD) has already had
its bandwidth subtracted and cleared to p->dl.dl_bw = 0 by
__dl_clear_params(&p->dl) in task_non_contending() or
inactive_task_timer(), leaving p->prio as MAX_DL_PRIO - 1 while the
unreaped zombie remains on the process/thread list.
Because p->dl.dl_bw == 0, dl_bw_deactivate() sees total_bw - dl_server_bw == 0
and permits all isolcpus=domain CPUs to be taken offline.
The new check only skips TASK_DEAD tasks when !cpu_active(task_cpu(p)). It
does not filter out PI-boosted non-DL tasks (where p->__state != TASK_DEAD,
such as a task blocked on a PI futex while affine to an isolated CPU) or
TASK_DEAD zombies whose task_cpu(p) is still active while p->cpus_ptr is
restricted to now-offline isolcpus=domain CPUs. For those tasks,
dl_get_task_effective_cpus() returns an empty mask and cpumask_first_and()
returns nr_cpu_ids, hitting BUG_ON(cpu >= nr_cpu_ids) just to add
p->dl.dl_bw == 0.
Should dl_add_task_root_domain() check !p->dl.dl_bw (or !task_has_dl_policy(p))
before computing the effective CPU mask?
[ ... ]
> @@ -3510,6 +3504,33 @@ void dl_clear_root_domain_cpu(int cpu)
> dl_clear_root_domain(cpu_rq(cpu)->rd);
> }
>
> +#ifndef CONFIG_CPUSETS
> +void dl_rebuild_rd_accounting(void)
> +{
> + struct task_struct *g, *p;
> + u64 cookie = ++dl_cookie;
> + int cpu;
> +
> + lockdep_assert_cpus_held();
> + lockdep_assert_held(&sched_domains_mutex);
> +
> + rcu_read_lock();
> + for_each_possible_cpu(cpu) {
> + if (dl_bw_visited(cpu, cookie))
> + continue;
> +
> + dl_clear_root_domain_cpu(cpu);
> + }
> + for_each_process_thread(g, p) {
> + /* Racy pre-check; dl_add_task_root_domain() rechecks under pi_lock. */
> + if (!dl_prio(READ_ONCE(p->prio)))
> + continue;
> + dl_add_task_root_domain(p);
> + }
> + rcu_read_unlock();
> +}
> +#endif
[Severity: High]
Can dl_rebuild_rd_accounting() drop the bandwidth of a TASK_DEAD
SCHED_DEADLINE task whose p->dl.inactive_timer is still armed, leading to an
underflow of rd->dl_bw.total_bw or a divide-by-zero in inactive_task_timer()?
When a running SCHED_DEADLINE task p exits via do_exit() -> do_task_dead() ->
__schedule() -> dequeue_task_dl() -> dequeue_dl_entity() ->
task_non_contending(&p->dl, true), if its 0-lag time is in the future,
task_non_contending() does not subtract p->dl.dl_bw immediately. Instead, it
sets p->dl.dl_non_contending = 1, takes a reference with get_task_struct(p),
and arms p->dl.inactive_timer.
When p->dl.inactive_timer later fires, inactive_task_timer() unconditionally
subtracts p->dl.dl_bw:
kernel/sched/deadline.c:inactive_task_timer() {
...
if (!dl_task(p) || READ_ONCE(p->__state) == TASK_DEAD) {
struct dl_bw *dl_b = dl_bw_of(task_cpu(p));
if (READ_ONCE(p->__state) == TASK_DEAD && dl_se->dl_non_contending) {
sub_running_bw(&p->dl, dl_rq_of_se(&p->dl));
sub_rq_bw(&p->dl, dl_rq_of_se(&p->dl));
dl_se->dl_non_contending = 0;
}
raw_spin_lock(&dl_b->lock);
__dl_sub(dl_b, p->dl.dl_bw, dl_bw_cpus(task_cpu(p)));
raw_spin_unlock(&dl_b->lock);
__dl_clear_params(dl_se);
goto unlock;
}
}
There appear to be two ways a root domain rebuild loses p->dl.dl_bw while
p->dl.inactive_timer is still pending:
1. In the new !CONFIG_CPUSETS dl_rebuild_rd_accounting(),
dl_clear_root_domain() resets rd->dl_bw.total_bw to only the dl_server
bandwidth and rebuilds task bandwidth via for_each_process_thread(g, p).
If p was a sub-thread, autoreaped, or already reaped by wait(),
release_task() -> __exit_signal() -> __unhash_process() has already removed
p from the process/thread lists before p->dl.inactive_timer expires.
for_each_process_thread(g, p) will not visit p, so p->dl.dl_bw is omitted
from rd->dl_bw.total_bw, and when inactive_task_timer() later calls
__dl_sub(), it underflows rd->dl_bw.total_bw and inflates rq->dl.extra_bw.
2. Even if p is still an unwaited zombie in for_each_process_thread(g, p),
the new (READ_ONCE(p->__state) == TASK_DEAD && !cpu_active(task_cpu(p)))
check in dl_add_task_root_domain() skips p when task_cpu(p) is deactivated
during CPU offline while p->dl.inactive_timer is armed. That drops
p->dl.dl_bw from the active root domain on rebuild, and if
inactive_task_timer() fires after task_cpu(p) is attached to
def_root_domain, __dl_sub() underflows def_root_domain.dl_bw.total_bw and
divides by dl_bw_cpus(task_cpu(p)) == 0 when no isolated CPUs are active in
def_root_domain.
_________________
_________________
_________________
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261019011446.834487-1-pasha.tatashin@soleen.com?part=6
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v2 7/8] sched/topology: Avoid redundant percpu allocations and bitmap walks in build_sched_domains()
2026-10-19 1:14 [PATCH v2 0/8] x86, sched, mm: Speed up CPU hotplug on large systems Pasha Tatashin
` (5 preceding siblings ...)
2026-10-19 1:14 ` [PATCH v2 6/8] sched/deadline, topology: Avoid O(N^2) dl_server bandwidth walks during root_domain rebuilds Pasha Tatashin
@ 2026-10-19 1:14 ` Pasha Tatashin
2026-10-20 1:15 ` sashiko-bot
2026-10-19 1:14 ` [PATCH v2 8/8] mm/vmstat: Avoid redundant per-CPU stat_threshold writes in refresh_zone_stat_thresholds() Pasha Tatashin
` (2 subsequent siblings)
9 siblings, 1 reply; 20+ messages in thread
From: Pasha Tatashin @ 2026-10-19 1:14 UTC (permalink / raw)
To: kees, rppt, juri.lelli, nik.borisov, tglx, vschneid,
kprateek.nayak, vincent.guittot, bp, jpoimboe, cgroups, tony.luck,
david, mingo, akpm, linux-mm, seanjc, ljs, zhangguopeng,
dietmar.eggemann, mhocko, vinicius.gomes, vbabka, linux-kernel,
peterz, surenb, hpa, x86, pasha.tatashin, dmatlack, tim.c.chen,
kexec, pratyush, dave.hansen, rostedt, gmonaco, jgross, mgorman,
graf, linux-edac, liam, ridong.chen, blum, longman, bsegall
During CPU hotplug, build_sched_domains() is invoked on every CPU
activation and deactivation. Profiling build_sched_domains() on a
480-CPU Intel Xeon 6985P-C and a 512-CPU AMD EPYC 9B45 shows several
sources of avoidable O(N^2) and O(N) work per step:
1. build_sched_groups() and build_overlap_sched_groups() iterate over
every bit of sched_domain_span(sd) via for_each_cpu_wrap(i, span, cpu)
and check cpumask_test_cpu(i, covered) one CPU at a time, even after
entire child groups (e.g., 120-CPU NUMA nodes) have been added to
covered. Instead, track the remaining uncovered CPUs
(cpumask_copy(uncovered, span)), clear each built group's span with
cpumask_andnot(), and advance directly to the next uncovered CPU.
2. __visit_domain_allocation_hell() and __free_domain_allocs() allocate
and free 3 * nr_levels + 2 temporary per-CPU pointer arrays via
alloc_percpu() / free_percpu() on every domain rebuild. Reuse the
per-CPU pointer arrays across domain rebuilds (serialized by
sched_domains_mutex), clearing entries in __free_domain_allocs() and
only freeing the topology-level arrays when sched_domain_topology is
replaced in sched_init_numa() / sched_reset_numa() (serialized
against domain rebuilds by the CPU hotplug lock).
3. sd_init() already computes sd_weight = cpumask_weight(sd_span), so
initialize sd->span_weight directly in sd_init() and reuse
child->span_weight in get_group() instead of recomputing
cpumask_weight() across all domains and groups.
4. Reuse the first LLC sibling's computed llc_bytes in alloc_sd_llc()
and sched_update_llc_bytes().
2-socket, 480-CPU Intel Xeon 6985P-C (N=479, mean of 10 runs):
Offline: 3.88 s -> 3.37 s (-13.1%)
Online: 2.51 s -> 2.06 s (-17.9%)
Total: 6.39 s -> 5.43 s (-15.0%)
build_sched_domains() (cpu479 offline / online):
7.91 ms -> 3.96 ms (-50.0%) / 7.51 ms -> 4.26 ms (-43.3%)
2-socket, 512-CPU AMD EPYC 9B45 (N=511, mean of 10 runs):
Offline: 2.77 s -> 2.34 s (-15.5%)
Online: 1.53 s -> 1.12 s (-26.8%)
Total: 4.30 s -> 3.46 s (-19.5%)
build_sched_domains() (cpu511 offline / online):
6.55 ms -> 3.04 ms (-53.7%) / 5.28 ms -> 3.20 ms (-39.4%)
Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
kernel/sched/topology.c | 171 ++++++++++++++++++++++++++++------------
1 file changed, 122 insertions(+), 49 deletions(-)
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index b5dec7f486b5..33210fd9dc2c 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -917,9 +917,15 @@ static bool alloc_sd_llc(const struct cpumask *cpu_map,
top_llc = parent;
if (top_llc->flags & SD_SHARE_LLC) {
+ int first = cpumask_first(sched_domain_span(top_llc));
+
sd->llc_max = max_lid + 1;
sd->llc_counts = p;
- sd->llc_bytes = get_effective_llc_bytes(i, top_llc);
+ /* All CPUs in top_llc share the same LLC size and weight. */
+ if (first < i)
+ sd->llc_bytes = (*per_cpu_ptr(d->sd, first))->llc_bytes;
+ else
+ sd->llc_bytes = get_effective_llc_bytes(i, top_llc);
} else {
/* avoid memory leak */
kfree(p);
@@ -1018,12 +1024,20 @@ void sched_update_llc_bytes(const struct cpumask *cpus)
* own LLC domain to scale its share of the physical cache.
*/
for_each_cpu(i, cpus) {
+ int first;
+
sdp = rcu_dereference_sched_domain(per_cpu(sd_llc, i));
if (!sdp)
continue;
sd = rcu_dereference_sched_domain(cpu_rq(i)->sd);
- if (sd)
+ if (!sd)
+ continue;
+
+ first = cpumask_first(sched_domain_span(sdp));
+ if (first < i)
+ sd->llc_bytes = rcu_dereference_sched_domain(cpu_rq(first)->sd)->llc_bytes;
+ else
sd->llc_bytes = get_effective_llc_bytes(i, sdp);
}
@@ -1370,19 +1384,18 @@ build_overlap_sched_groups(struct sched_domain *sd, int cpu)
{
struct sched_group *first = NULL, *last = NULL, *sg;
const struct cpumask *span = sched_domain_span(sd);
- struct cpumask *covered = sched_domains_tmpmask;
+ struct cpumask *uncovered = sched_domains_tmpmask;
struct sd_data *sdd = sd->private;
struct sched_domain *sibling;
int i;
- cpumask_clear(covered);
+ cpumask_copy(uncovered, span);
- for_each_cpu_wrap(i, span, cpu) {
+ for (i = cpu; i < nr_cpu_ids;
+ i = cpumask_next(i, uncovered),
+ i = i < nr_cpu_ids ? i : cpumask_first(uncovered)) {
struct cpumask *sg_span;
- if (cpumask_test_cpu(i, covered))
- continue;
-
sibling = *per_cpu_ptr(sdd->sd, i);
/*
@@ -1395,8 +1408,10 @@ build_overlap_sched_groups(struct sched_domain *sd, int cpu)
* Domains should always include the CPU they're built on, so
* check that.
*/
- if (!cpumask_test_cpu(i, sched_domain_span(sibling)))
+ if (!cpumask_test_cpu(i, sched_domain_span(sibling))) {
+ __cpumask_clear_cpu(i, uncovered);
continue;
+ }
/*
* Usually we build sched_group by sibling's child sched_domain
@@ -1438,7 +1453,7 @@ build_overlap_sched_groups(struct sched_domain *sd, int cpu)
goto fail;
sg_span = sched_group_span(sg);
- cpumask_or(covered, covered, sg_span);
+ cpumask_andnot(uncovered, uncovered, sg_span);
if (init_overlap_sched_group(sibling, sg)) {
kfree(sg);
@@ -1579,12 +1594,13 @@ static struct sched_group *get_group(int cpu, struct sd_data *sdd)
cpumask_copy(sched_group_span(sg), sched_domain_span(child));
cpumask_copy(group_balance_mask(sg), sched_group_span(sg));
sg->flags = child->flags;
+ sg->sgc->capacity = SCHED_CAPACITY_SCALE * child->span_weight;
} else {
cpumask_set_cpu(cpu, sched_group_span(sg));
cpumask_set_cpu(cpu, group_balance_mask(sg));
+ sg->sgc->capacity = SCHED_CAPACITY_SCALE;
}
- sg->sgc->capacity = SCHED_CAPACITY_SCALE * cpumask_weight(sched_group_span(sg));
sg->sgc->min_capacity = SCHED_CAPACITY_SCALE;
sg->sgc->max_capacity = SCHED_CAPACITY_SCALE;
@@ -1604,25 +1620,24 @@ build_sched_groups(struct sched_domain *sd, int cpu)
struct sched_group *first = NULL, *last = NULL;
struct sd_data *sdd = sd->private;
const struct cpumask *span = sched_domain_span(sd);
- struct cpumask *covered;
+ struct cpumask *uncovered;
int i;
lockdep_assert_held(&sched_domains_mutex);
- covered = sched_domains_tmpmask;
+ uncovered = sched_domains_tmpmask;
- cpumask_clear(covered);
+ cpumask_copy(uncovered, span);
- for_each_cpu_wrap(i, span, cpu) {
+ for (i = cpu; i < nr_cpu_ids;
+ i = cpumask_next(i, uncovered),
+ i = i < nr_cpu_ids ? i : cpumask_first(uncovered)) {
struct sched_group *sg;
- if (cpumask_test_cpu(i, covered))
- continue;
-
sg = get_group(i, sdd);
if (!sg)
return -ENOMEM;
- cpumask_or(covered, covered, sched_group_span(sg));
+ cpumask_andnot(uncovered, uncovered, sched_group_span(sg));
if (!first)
first = sg;
@@ -1911,16 +1926,21 @@ static int __sdt_alloc(const struct cpumask *cpu_map);
static void __sds_free(struct s_data *d, const struct cpumask *cpu_map);
static int __sds_alloc(struct s_data *d, const struct cpumask *cpu_map);
+static struct sched_domain * __percpu *sched_domains_sd;
+
static void __free_domain_allocs(struct s_data *d, enum s_alloc what,
const struct cpumask *cpu_map)
{
+ int j;
+
switch (what) {
case sa_rootdomain:
if (!atomic_read(&d->rd->refcount))
free_rootdomain(&d->rd->rcu);
fallthrough;
case sa_sd:
- free_percpu(d->sd);
+ for_each_cpu(j, cpu_map)
+ *per_cpu_ptr(d->sd, j) = NULL;
fallthrough;
case sa_sd_shared:
__sds_free(d, cpu_map);
@@ -1942,9 +1962,12 @@ __visit_domain_allocation_hell(struct s_data *d, const struct cpumask *cpu_map)
return sa_sd_storage;
if (__sds_alloc(d, cpu_map))
return sa_sd_shared;
- d->sd = alloc_percpu(struct sched_domain *);
- if (!d->sd)
- return sa_sd_shared;
+ if (!sched_domains_sd) {
+ sched_domains_sd = alloc_percpu(struct sched_domain *);
+ if (!sched_domains_sd)
+ return sa_sd_shared;
+ }
+ d->sd = sched_domains_sd;
d->rd = alloc_rootdomain();
if (!d->rd)
return sa_sd;
@@ -2072,6 +2095,7 @@ sd_init(struct sched_domain_topology_level *tl,
.last_balance = jiffies,
.balance_interval = sd_weight,
+ .span_weight = sd_weight,
/* 50% success rate */
.newidle_call = 512,
@@ -2212,6 +2236,24 @@ void __init set_sched_topology(struct sched_domain_topology_level *tl)
}
#ifdef CONFIG_NUMA
+static void __sdt_free_percpu(void)
+{
+ struct sched_domain_topology_level *tl;
+
+ lockdep_assert_cpus_held();
+
+ for_each_sd_topology(tl) {
+ struct sd_data *sdd = &tl->data;
+
+ free_percpu(sdd->sd);
+ sdd->sd = NULL;
+ free_percpu(sdd->sg);
+ sdd->sg = NULL;
+ free_percpu(sdd->sgc);
+ sdd->sgc = NULL;
+ }
+}
+
static int cpu_numa_flags(void)
{
return SD_NUMA;
@@ -2502,6 +2544,7 @@ void sched_init_numa(int offline_node)
/*
* Copy the default topology bits..
*/
+ __sdt_free_percpu();
for (i = 0; sched_domain_topology[i].mask; i++)
tl[i] = sched_domain_topology[i];
@@ -2560,6 +2603,7 @@ static void sched_reset_numa(void)
kfree(masks);
}
if (sched_domain_topology_saved) {
+ __sdt_free_percpu();
kfree(sched_domain_topology);
sched_domain_topology = sched_domain_topology_saved;
sched_domain_topology_saved = NULL;
@@ -2761,21 +2805,38 @@ static int __sdt_alloc(const struct cpumask *cpu_map)
for_each_sd_topology(tl) {
struct sd_data *sdd = &tl->data;
- sdd->sd = alloc_percpu(struct sched_domain *);
- if (!sdd->sd)
- return -ENOMEM;
+ if (!sdd->sd) {
+ sdd->sd = alloc_percpu(struct sched_domain *);
+ if (!sdd->sd)
+ return -ENOMEM;
+ }
- sdd->sg = alloc_percpu(struct sched_group *);
- if (!sdd->sg)
- return -ENOMEM;
+ if (!sdd->sg) {
+ sdd->sg = alloc_percpu(struct sched_group *);
+ if (!sdd->sg)
+ return -ENOMEM;
+ }
- sdd->sgc = alloc_percpu(struct sched_group_capacity *);
- if (!sdd->sgc)
- return -ENOMEM;
+ if (!sdd->sgc) {
+ sdd->sgc = alloc_percpu(struct sched_group_capacity *);
+ if (!sdd->sgc)
+ return -ENOMEM;
+ }
+ /*
+ * The slot arrays are kept between builds. Every slot for a CPU
+ * in cpu_map must be NULL here; lazy allocation treats NULL as
+ * "not allocated in this build". claim_allocations() and
+ * __sdt_free()/__sds_free() clear the slots, but only for
+ * cpu_map, so a build must not fill slots outside cpu_map.
+ */
for_each_cpu(j, cpu_map) {
struct sched_domain *sd;
+ WARN_ON_ONCE(*per_cpu_ptr(sdd->sd, j) ||
+ *per_cpu_ptr(sdd->sg, j) ||
+ *per_cpu_ptr(sdd->sgc, j));
+
sd = kzalloc_node(sizeof(struct sched_domain) + cpumask_size(),
GFP_KERNEL, cpu_to_node(j));
if (!sd)
@@ -2801,30 +2862,41 @@ static void __sdt_free(const struct cpumask *cpu_map)
if (sdd->sd) {
sd = *per_cpu_ptr(sdd->sd, j);
- if (sd && (sd->flags & SD_NUMA))
- free_sched_groups(sd->groups, 0);
- kfree(*per_cpu_ptr(sdd->sd, j));
+ if (sd) {
+ if (sd->flags & SD_NUMA)
+ free_sched_groups(sd->groups, 0);
+ kfree(sd);
+ *per_cpu_ptr(sdd->sd, j) = NULL;
+ }
}
- if (sdd->sg)
+ if (sdd->sg) {
kfree(*per_cpu_ptr(sdd->sg, j));
- if (sdd->sgc)
+ *per_cpu_ptr(sdd->sg, j) = NULL;
+ }
+ if (sdd->sgc) {
kfree(*per_cpu_ptr(sdd->sgc, j));
+ *per_cpu_ptr(sdd->sgc, j) = NULL;
+ }
}
- free_percpu(sdd->sd);
- sdd->sd = NULL;
- free_percpu(sdd->sg);
- sdd->sg = NULL;
- free_percpu(sdd->sgc);
- sdd->sgc = NULL;
}
}
+static struct sched_domain_shared * __percpu *sched_domains_sds;
+
static int __sds_alloc(struct s_data *d, const struct cpumask *cpu_map)
{
- d->sds = alloc_percpu(struct sched_domain_shared *);
- if (!d->sds)
- return -ENOMEM;
+ int j;
+
+ if (!sched_domains_sds) {
+ sched_domains_sds = alloc_percpu(struct sched_domain_shared *);
+ if (!sched_domains_sds)
+ return -ENOMEM;
+ }
+ d->sds = sched_domains_sds;
+
+ for_each_cpu(j, cpu_map)
+ WARN_ON_ONCE(*per_cpu_ptr(d->sds, j));
return 0;
}
@@ -2836,10 +2908,11 @@ static void __sds_free(struct s_data *d, const struct cpumask *cpu_map)
if (!d->sds)
return;
- for_each_cpu(j, cpu_map)
+ for_each_cpu(j, cpu_map) {
kfree(*per_cpu_ptr(d->sds, j));
+ *per_cpu_ptr(d->sds, j) = NULL;
+ }
- free_percpu(d->sds);
d->sds = NULL;
}
@@ -2863,6 +2936,7 @@ static struct sched_domain *build_sched_domain(struct sched_domain_topology_leve
cpumask_or(sched_domain_span(sd),
sched_domain_span(sd),
sched_domain_span(child));
+ sd->span_weight = cpumask_weight(sched_domain_span(sd));
}
}
@@ -3213,7 +3287,6 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
/* Build the groups for the domains */
for_each_cpu(i, cpu_map) {
for (sd = *per_cpu_ptr(d.sd, i); sd; sd = sd->parent) {
- sd->span_weight = cpumask_weight(sched_domain_span(sd));
if (sd->flags & SD_NUMA) {
if (build_overlap_sched_groups(sd, i))
goto error;
--
2.56.0.176.g3575e65682-goog
^ permalink raw reply related [flat|nested] 20+ messages in thread* Re: [PATCH v2 7/8] sched/topology: Avoid redundant percpu allocations and bitmap walks in build_sched_domains()
2026-10-19 1:14 ` [PATCH v2 7/8] sched/topology: Avoid redundant percpu allocations and bitmap walks in build_sched_domains() Pasha Tatashin
@ 2026-10-20 1:15 ` sashiko-bot
0 siblings, 0 replies; 20+ messages in thread
From: sashiko-bot @ 2026-10-20 1:15 UTC (permalink / raw)
To: Pasha Tatashin, kees, rppt, juri.lelli, nik.borisov, tglx,
vschneid, kprateek.nayak, vincent.guittot, bp, jpoimboe,
tony.luck, david, mingo, akpm, linux-mm, seanjc, ljs,
zhangguopeng, dietmar.eggemann, mhocko, vinicius.gomes, vbabka,
peterz, surenb, hpa, x86, dmatlack, tim.c.chen, kexec, pratyush,
dave.hansen, rostedt, gmonaco, jgross, mgorman, graf, liam,
ridong.chen, blum, longman, bsegall
Cc: tj, hannes, mkoutny
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] sched/topology: sched_update_llc_bytes() reuses llc_bytes outside cpus
--
commit d95533f0e2e71e5e08ee625a1ea2afb7ecbe1b86
Author: Pasha Tatashin <pasha.tatashin@soleen.com>
sched/topology: Avoid redundant percpu allocations and bitmap walks in build_sched_domains()
This commit optimizes build_sched_domains() during CPU hotplug by tracking
uncovered CPUs during sched_group construction, reusing temporary per-CPU
pointer arrays across rebuilds, initializing sd->span_weight in sd_init(),
and reusing the first LLC sibling's computed llc_bytes in alloc_sd_llc()
and sched_update_llc_bytes().
> diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
> index b5dec7f486b5c..33210fd9dc2c6 100644
> --- a/kernel/sched/topology.c
> +++ b/kernel/sched/topology.c
> @@ -917,9 +917,15 @@ static bool alloc_sd_llc(const struct cpumask *cpu_map,
> top_llc = parent;
>
> if (top_llc->flags & SD_SHARE_LLC) {
> + int first = cpumask_first(sched_domain_span(top_llc));
> +
> sd->llc_max = max_lid + 1;
> sd->llc_counts = p;
> - sd->llc_bytes = get_effective_llc_bytes(i, top_llc);
> + /* All CPUs in top_llc share the same LLC size and weight. */
> + if (first < i)
> + sd->llc_bytes = (*per_cpu_ptr(d->sd, first))->llc_bytes;
> + else
> + sd->llc_bytes = get_effective_llc_bytes(i, top_llc);
> } else {
> /* avoid memory leak */
> kfree(p);
> @@ -1018,12 +1024,20 @@ void sched_update_llc_bytes(const struct cpumask *cpus)
> * own LLC domain to scale its share of the physical cache.
> */
> for_each_cpu(i, cpus) {
> + int first;
> +
> sdp = rcu_dereference_sched_domain(per_cpu(sd_llc, i));
> if (!sdp)
> continue;
>
> sd = rcu_dereference_sched_domain(cpu_rq(i)->sd);
> - if (sd)
> + if (!sd)
> + continue;
> +
> + first = cpumask_first(sched_domain_span(sdp));
> + if (first < i)
> + sd->llc_bytes = rcu_dereference_sched_domain(cpu_rq(first)->sd)->llc_bytes;
> + else
> sd->llc_bytes = get_effective_llc_bytes(i, sdp);
> }
[Severity: Medium]
Can sched_update_llc_bytes() and alloc_sd_llc() copy an un-updated or
mismatched llc_bytes value when the topmost SD_SHARE_LLC domain spans
multiple cacheinfo LLC leaves?
When cacheinfo_cpu_online() or cacheinfo_cpu_pre_down() calls
sched_update_llc_bytes(cpus), cpus is a single cacheinfo leaf's
&llc->shared_cpu_map, whereas sdp (and top_llc in alloc_sd_llc()) is the
highest sched_domain with SD_SHARE_LLC, whose span comes from
cpu_coregroup_mask().
On topologies where cpu_coregroup_mask() is wider than the cacheinfo LLC
leaf's shared_cpu_map, sched_domain_span(sdp) spans multiple physical
cacheinfo LLCs:
1. On ARM64/RISC-V systems with clusters and per-core L2 caches (no shared
L3), cpu_coregroup_mask() in drivers/base/arch_topology.c extends the MC
domain to cluster_sibling:
if (IS_ENABLED(CONFIG_SCHED_CLUSTER) &&
cpumask_subset(core_mask, &cpu_topology[cpu].cluster_sibling))
core_mask = &cpu_topology[cpu].cluster_sibling;
2. On AMD/Hygon x86 systems without an L3 cache, legacy_set_llc() in
arch/x86/kernel/cpu/topology_amd.c sets llc_id to the die ID while
get_cpu_cacheinfo_llc() returns the per-core L2 cache leaf:
/* If none of the parsers set LLC ID then use the die ID for it. */
if (tscan->c->topo.llc_id == BAD_APICID)
tscan->c->topo.llc_id = apicid >> tscan->dom_shifts[TOPO_CORE_DOMAIN];
When sched_update_llc_bytes(cpus) is invoked for any core other than the
first core in sdp, first = cpumask_first(sched_domain_span(sdp)) is not in
cpus, so cpu_rq(first)->sd->llc_bytes is not updated by the
for_each_cpu(i, cpus) loop. Because first < i holds for every CPU i in cpus,
get_effective_llc_bytes() is skipped and sd->llc_bytes copies the un-updated
value from cpu_rq(first)->sd.
Similarly, in alloc_sd_llc(), copying (*per_cpu_ptr(d->sd, first))->llc_bytes
across all CPUs in top_llc reuses the first CPU's L2 cache calculation even
when other cores in top_llc have different cacheinfo shared_cpu_map weights
(for example, partially online SMT siblings) or cache sizes, which can cause
exceed_llc_capacity() in kernel/sched/fair.c to use the wrong capacity.
[ ... ]
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261019011446.834487-1-pasha.tatashin@soleen.com?part=7
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v2 8/8] mm/vmstat: Avoid redundant per-CPU stat_threshold writes in refresh_zone_stat_thresholds()
2026-10-19 1:14 [PATCH v2 0/8] x86, sched, mm: Speed up CPU hotplug on large systems Pasha Tatashin
` (6 preceding siblings ...)
2026-10-19 1:14 ` [PATCH v2 7/8] sched/topology: Avoid redundant percpu allocations and bitmap walks in build_sched_domains() Pasha Tatashin
@ 2026-10-19 1:14 ` Pasha Tatashin
2026-10-20 1:16 ` sashiko-bot
2026-10-19 4:14 ` [PATCH v2 0/8] x86, sched, mm: Speed up CPU hotplug on large systems Borislav Petkov
2026-10-19 18:22 ` Luck, Tony
9 siblings, 1 reply; 20+ messages in thread
From: Pasha Tatashin @ 2026-10-19 1:14 UTC (permalink / raw)
To: kees, rppt, juri.lelli, nik.borisov, tglx, vschneid,
kprateek.nayak, vincent.guittot, bp, jpoimboe, cgroups, tony.luck,
david, mingo, akpm, linux-mm, seanjc, ljs, zhangguopeng,
dietmar.eggemann, mhocko, vinicius.gomes, vbabka, linux-kernel,
peterz, surenb, hpa, x86, pasha.tatashin, dmatlack, tim.c.chen,
kexec, pratyush, dave.hansen, rostedt, gmonaco, jgross, mgorman,
graf, linux-edac, liam, ridong.chen, blum, longman, bsegall
During CPU hotplug, vmstat_cpu_online() and vmstat_cpu_dead() invoke
refresh_zone_stat_thresholds() on every CPU transition. Currently,
refresh_zone_stat_thresholds() first zeroes
per_cpu_nodestats->stat_threshold across all online CPUs and all online
nodes, and then unconditionally writes per_cpu_zonestats->stat_threshold
and per_cpu_nodestats->stat_threshold inside the zone loop for every
populated zone and every online CPU.
However, calculate_normal_threshold(zone) only changes when
fls(num_online_cpus()) changes (at powers of 2) and is clamped at 125
for large normal zones (>= 16 GB) once num_online_cpus() >= 64.
Unconditionally zeroing and rewriting every CPU's per-CPU stat_threshold
on every hotplug step repeatedly dirties remote per-CPU cachelines with
identical values (and briefly exposes stat_threshold == 0 on live CPUs).
Compute pgdat_threshold across each node's populated zones first, hoist
the per_cpu_nodestats->stat_threshold update out of the zone loop, and
only write stat_threshold when the value actually changes.
2-socket, 480-CPU Intel Xeon 6985P-C (N=479, mean of 10 runs):
Offline: 3.37 s -> 3.09 s (-8.3%)
Online: 2.06 s -> 1.98 s (-3.9%)
Total: 5.43 s -> 5.07 s (-6.6%)
refresh_zone_stat_thresholds() (cpu479 offline / online):
1,158.9 us -> 135.3 us (-88.3%) / 1,468.2 us -> 128.6 us (-91.2%)
2-socket, 512-CPU AMD EPYC 9B45 (N=511, mean of 10 runs):
Offline: 2.34 s -> 2.30 s (-1.7%)
Online: 1.12 s -> 1.08 s (-3.6%)
Total: 3.46 s -> 3.38 s (-2.3%)
refresh_zone_stat_thresholds() (cpu511 offline / online):
104.6 us -> 50.7 us (-51.6%) / 109.3 us -> 53.5 us (-51.1%)
Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
---
mm/vmstat.c | 58 ++++++++++++++++++++++++++++-------------------------
1 file changed, 31 insertions(+), 27 deletions(-)
diff --git a/mm/vmstat.c b/mm/vmstat.c
index cb57714539fb..c671e9669b43 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -280,41 +280,45 @@ void refresh_zone_stat_thresholds(void)
int cpu;
int threshold;
- /* Zero current pgdat thresholds */
for_each_online_pgdat(pgdat) {
- for_each_online_cpu(cpu) {
- per_cpu_ptr(pgdat->per_cpu_nodestats, cpu)->stat_threshold = 0;
- }
- }
+ int pgdat_threshold = 0;
- for_each_populated_zone(zone) {
- struct pglist_data *pgdat = zone->zone_pgdat;
- unsigned long max_drift, tolerate_drift;
+ for (int z = 0; z < MAX_NR_ZONES; z++) {
+ unsigned long max_drift, tolerate_drift;
- threshold = calculate_normal_threshold(zone);
+ zone = &pgdat->node_zones[z];
+ if (!populated_zone(zone))
+ continue;
- for_each_online_cpu(cpu) {
- int pgdat_threshold;
+ threshold = calculate_normal_threshold(zone);
+ /* Base nodestat threshold on the largest populated zone. */
+ pgdat_threshold = max(threshold, pgdat_threshold);
- per_cpu_ptr(zone->per_cpu_zonestats, cpu)->stat_threshold
- = threshold;
+ for_each_online_cpu(cpu) {
+ s8 *pz = &per_cpu_ptr(zone->per_cpu_zonestats, cpu)->stat_threshold;
- /* Base nodestat threshold on the largest populated zone. */
- pgdat_threshold = per_cpu_ptr(pgdat->per_cpu_nodestats, cpu)->stat_threshold;
- per_cpu_ptr(pgdat->per_cpu_nodestats, cpu)->stat_threshold
- = max(threshold, pgdat_threshold);
+ if (*pz != threshold)
+ *pz = threshold;
+ }
+
+ /*
+ * Only set percpu_drift_mark if there is a danger that
+ * NR_FREE_PAGES reports the low watermark is ok when in fact
+ * the min watermark could be breached by an allocation
+ */
+ tolerate_drift = low_wmark_pages(zone) - min_wmark_pages(zone);
+ max_drift = num_online_cpus() * threshold;
+ if (max_drift > tolerate_drift)
+ zone->percpu_drift_mark = high_wmark_pages(zone) +
+ max_drift;
}
- /*
- * Only set percpu_drift_mark if there is a danger that
- * NR_FREE_PAGES reports the low watermark is ok when in fact
- * the min watermark could be breached by an allocation
- */
- tolerate_drift = low_wmark_pages(zone) - min_wmark_pages(zone);
- max_drift = num_online_cpus() * threshold;
- if (max_drift > tolerate_drift)
- zone->percpu_drift_mark = high_wmark_pages(zone) +
- max_drift;
+ for_each_online_cpu(cpu) {
+ s8 *pn = &per_cpu_ptr(pgdat->per_cpu_nodestats, cpu)->stat_threshold;
+
+ if (*pn != pgdat_threshold)
+ *pn = pgdat_threshold;
+ }
}
}
--
2.56.0.176.g3575e65682-goog
^ permalink raw reply related [flat|nested] 20+ messages in thread* Re: [PATCH v2 8/8] mm/vmstat: Avoid redundant per-CPU stat_threshold writes in refresh_zone_stat_thresholds()
2026-10-19 1:14 ` [PATCH v2 8/8] mm/vmstat: Avoid redundant per-CPU stat_threshold writes in refresh_zone_stat_thresholds() Pasha Tatashin
@ 2026-10-20 1:16 ` sashiko-bot
0 siblings, 0 replies; 20+ messages in thread
From: sashiko-bot @ 2026-10-20 1:16 UTC (permalink / raw)
To: Pasha Tatashin, kees, rppt, juri.lelli, nik.borisov, tglx,
vschneid, kprateek.nayak, vincent.guittot, bp, jpoimboe,
tony.luck, david, mingo, akpm, seanjc, ljs, zhangguopeng,
dietmar.eggemann, mhocko, vinicius.gomes, vbabka, peterz, surenb,
hpa, x86, dmatlack, tim.c.chen, kexec, pratyush, dave.hansen,
rostedt, gmonaco, jgross, mgorman, graf, liam, ridong.chen, blum,
longman, bsegall
> During CPU hotplug, vmstat_cpu_online() and vmstat_cpu_dead() invoke
> refresh_zone_stat_thresholds() on every CPU transition. Currently,
> refresh_zone_stat_thresholds() first zeroes
> per_cpu_nodestats->stat_threshold across all online CPUs and all online
> nodes, and then unconditionally writes per_cpu_zonestats->stat_threshold
> and per_cpu_nodestats->stat_threshold inside the zone loop for every
> populated zone and every online CPU.
>
> However, calculate_normal_threshold(zone) only changes when
> fls(num_online_cpus()) changes (at powers of 2) and is clamped at 125
> for large normal zones (>= 16 GB) once num_online_cpus() >= 64.
> Unconditionally zeroing and rewriting every CPU's per-CPU stat_threshold
> on every hotplug step repeatedly dirties remote per-CPU cachelines with
> identical values (and briefly exposes stat_threshold == 0 on live CPUs).
>
> [ ... ]
> Total: 3.46 s -> 3.38 s (-2.3%)
> refresh_zone_stat_thresholds() (cpu511 offline / online):
> 104.6 us -> 50.7 us (-51.6%) / 109.3 us -> 53.5 us (-51.1%)
>
> Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261019011446.834487-1-pasha.tatashin@soleen.com?part=8
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v2 0/8] x86, sched, mm: Speed up CPU hotplug on large systems
2026-10-19 1:14 [PATCH v2 0/8] x86, sched, mm: Speed up CPU hotplug on large systems Pasha Tatashin
` (7 preceding siblings ...)
2026-10-19 1:14 ` [PATCH v2 8/8] mm/vmstat: Avoid redundant per-CPU stat_threshold writes in refresh_zone_stat_thresholds() Pasha Tatashin
@ 2026-10-19 4:14 ` Borislav Petkov
2026-10-19 18:22 ` Luck, Tony
9 siblings, 0 replies; 20+ messages in thread
From: Borislav Petkov @ 2026-10-19 4:14 UTC (permalink / raw)
To: Pasha Tatashin
Cc: kees, rppt, juri.lelli, nik.borisov, tglx, vschneid,
kprateek.nayak, vincent.guittot, jpoimboe, cgroups, tony.luck,
david, mingo, akpm, linux-mm, seanjc, ljs, zhangguopeng,
dietmar.eggemann, mhocko, vinicius.gomes, vbabka, linux-kernel,
peterz, surenb, hpa, x86, dmatlack, tim.c.chen, kexec, pratyush,
dave.hansen, rostedt, gmonaco, jgross, mgorman, graf, linux-edac,
liam, ridong.chen, blum, longman, bsegall
On Sun, Oct 18, 2026 at 09:14:38PM -0400, Pasha Tatashin wrote:
> For live update we require a fast CPU online and offline capability.
> However, sequentially offlining and onlining all application processors
> on modern high-core-count x86 servers exposes a series of hardware and
> algorithmic bottlenecks across arch/x86, the scheduler topology builder,
> deadline bandwidth accounting, and vmstat.
From: Documentation/process/submitting-patches.rst
"Don't get discouraged - or impatient
------------------------------------
After you have submitted your change, be patient and wait. Reviewers are
busy people and may not get to your patch right away.
Once upon a time, patches used to disappear into the void without comment,
but the development process works more smoothly than that now. You should
receive comments within a week or so; if that does not happen, make sure
that you have sent your patches to the right place. Wait for a minimum of
^^^^^^^^^^^^^^^^^^^^^
one week before resubmitting or pinging reviewers - possibly longer during
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
busy times like merge windows."
+ the merge window just opened.
So stop the spamming please.
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
^ permalink raw reply [flat|nested] 20+ messages in thread* RE: [PATCH v2 0/8] x86, sched, mm: Speed up CPU hotplug on large systems
2026-10-19 1:14 [PATCH v2 0/8] x86, sched, mm: Speed up CPU hotplug on large systems Pasha Tatashin
` (8 preceding siblings ...)
2026-10-19 4:14 ` [PATCH v2 0/8] x86, sched, mm: Speed up CPU hotplug on large systems Borislav Petkov
@ 2026-10-19 18:22 ` Luck, Tony
2026-10-19 19:11 ` Pasha Tatashin
9 siblings, 1 reply; 20+ messages in thread
From: Luck, Tony @ 2026-10-19 18:22 UTC (permalink / raw)
To: Pasha Tatashin, kees@kernel.org, rppt@kernel.org,
juri.lelli@redhat.com, nik.borisov@suse.com, tglx@kernel.org,
vschneid@redhat.com, kprateek.nayak@amd.com,
vincent.guittot@linaro.org, bp@alien8.de, jpoimboe@kernel.org,
cgroups@vger.kernel.org, david@kernel.org, mingo@redhat.com,
akpm@linux-foundation.org, linux-mm@kvack.org, seanjc@google.com,
ljs@kernel.org, zhangguopeng@kylinos.cn, dietmar.eggemann@arm.com,
mhocko@suse.com, Gomes, Vinicius, vbabka@kernel.org,
linux-kernel@vger.kernel.org, peterz@infradead.org,
surenb@google.com, hpa@zytor.com, x86@kernel.org,
dmatlack@google.com, tim.c.chen@linux.intel.com,
kexec@lists.infradead.org, pratyush@kernel.org,
dave.hansen@linux.intel.com, rostedt@goodmis.org,
gmonaco@redhat.com, jgross@suse.com, mgorman@suse.de,
Graf, Alexander, linux-edac@vger.kernel.org, liam@infradead.org,
ridong.chen@linux.dev, blum@kernel.org, longman@redhat.com,
bsegall@google.com
> Benchmark workload (mean of 10 runs per step):
>
> # echo 1 > /sys/kernel/rcu_expedited
> # for i in {1..N}; do echo 0 > /sys/devices/system/cpu/cpu$i/online; done
Side question. Have you had issues with this step failing on large systems with "smart" I/O
devices that support many interrupt vectors?
Some validation folks inside Intel had some tests that looped taking all CPUs offline and
bringing them back. They complained starting around Sapphire Rapids this their test
would get down to just CPU 0 and CPU 1 and fail to take CPU 1 offline because there
were too many interrupts to fit into just a single CPUs available vectors.
It was a while ago, so maybe something changed in interrupt code to handle this.
-Tony
^ permalink raw reply [flat|nested] 20+ messages in thread* Re: [PATCH v2 0/8] x86, sched, mm: Speed up CPU hotplug on large systems
2026-10-19 18:22 ` Luck, Tony
@ 2026-10-19 19:11 ` Pasha Tatashin
0 siblings, 0 replies; 20+ messages in thread
From: Pasha Tatashin @ 2026-10-19 19:11 UTC (permalink / raw)
To: Luck, Tony
Cc: kees@kernel.org, rppt@kernel.org, juri.lelli@redhat.com,
nik.borisov@suse.com, tglx@kernel.org, vschneid@redhat.com,
kprateek.nayak@amd.com, vincent.guittot@linaro.org, bp@alien8.de,
jpoimboe@kernel.org, cgroups@vger.kernel.org, david@kernel.org,
mingo@redhat.com, akpm@linux-foundation.org, linux-mm@kvack.org,
seanjc@google.com, ljs@kernel.org, zhangguopeng@kylinos.cn,
dietmar.eggemann@arm.com, mhocko@suse.com, Gomes, Vinicius,
vbabka@kernel.org, linux-kernel@vger.kernel.org,
peterz@infradead.org, surenb@google.com, hpa@zytor.com,
x86@kernel.org, dmatlack@google.com, tim.c.chen@linux.intel.com,
kexec@lists.infradead.org, pratyush@kernel.org,
dave.hansen@linux.intel.com, rostedt@goodmis.org,
gmonaco@redhat.com, jgross@suse.com, mgorman@suse.de,
Graf, Alexander, linux-edac@vger.kernel.org, liam@infradead.org,
ridong.chen@linux.dev, blum@kernel.org, longman@redhat.com,
bsegall@google.com
On Mon, Oct 19, 2026 at 2:22 PM Luck, Tony <tony.luck@intel.com> wrote:
>
> > Benchmark workload (mean of 10 runs per step):
> >
> > # echo 1 > /sys/kernel/rcu_expedited
> > # for i in {1..N}; do echo 0 > /sys/devices/system/cpu/cpu$i/online; done
>
> Side question. Have you had issues with this step failing on large systems with "smart" I/O
> devices that support many interrupt vectors?
>
> Some validation folks inside Intel had some tests that looped taking all CPUs offline and
> bringing them back. They complained starting around Sapphire Rapids this their test
> would get down to just CPU 0 and CPU 1 and fail to take CPU 1 offline because there
> were too many interrupts to fit into just a single CPUs available vectors.
>
> It was a while ago, so maybe something changed in interrupt code to handle this.
Hi Tony,
I didn't encounter that issue while collecting the benchmark data.
For the Live Update work I'm currently doing, I leave 8 CPUs on the
host to perform kexec while the AP CPUs are removed from the
scheduler. In the future, I plan to leave only one host CPU and have
the remaining CPUs manage the VMs in Caretaker mode.
Best,
Pasha
>
> -Tony
>
>
>
^ permalink raw reply [flat|nested] 20+ messages in thread