[PATCH 0/7] x86/microcode: Refactor NMI-based rendezvous mechanism to stop-machine

[PATCH 0/7] x86/microcode: Refactor NMI-based rendezvous mechanism to stop-machine

[Chang S. Bae]

Hi all,

David Kaplan previously proposed introducing an NMI-based stop-machine
mechanism as part of his dynamic mitigation series [1]. This "big hammer"
approach was also discussed during the last LPC x86 uconf [2].

This patchset uses the facility for the late-loading of Intel microcode
by refactoring existing NMI control mechanism. There is one bit to call
out for the review here -- offline CPU handling. Before highlighting the
case, let me first walk through the online case:

== Online CPU Handling ==

Currently, the microcode loader uses a NMI-based control flow, similar
to the proposed stop_machine_nmi(). When the NMI option is enabled, the
loader invokes stop-machine, and the callback establishes an IPI back to
the local CPU, which then performs the microcode update from NMI context.
Per-CPU flags are used throughout to ensure correct handling.

The new facility directly incorporates this existing control logic, with
the following components in place (mostly in PATCH1 unless noted
otherwise):

 1. A static key indicating the global window of stop-machine execution
    during rendezvous

 2. A control mechanism to fire NMIs and invoke a custom handler from
    the NMI path, implemented in architecture-specific code (PATCH2/3)

 3. A per-CPU notion of an active NMI handling window

Given this, it is natural to convert this to use the generic
stop_machine_nmi().

== Offline CPU Handling ==

With NMI-based microcode loading, soft-offlined CPUs are also brought
into the rendezvous to avoid unexpected behavior changes (e.g. related
to MWAIT) introduced by new microcode [3].

Today, the boot CPU explicitly fires IPIs to those offline CPUs, which
establishes yet another NMI control path. However, the new stop-machine
facility does not currently account for this case, as it is primarily
shaped for online CPUs.

This patch set is intended to introduce stop_machine_nmi() along with an
existing use case (Intel ucode late-loading), independent of the dynamic
mitigation series.

The commonality, nonetheless, between them lies on online CPU handling.
So, it was considered to make sense that the static key is just reused to
condition offline CPU handling as well (PATCH6).

== Patch Structure ==

 * PATCH 1–3: Establish the stop-machine NMI facility (pretty much those
   PATCH29–31 of the dynamic mitigation series [1])

 * PATCH 4–7: Incremental refactoring, with offline handling update.

The patch set is available in this repository:
 git://github.com/intel-staging/microcode.git nmi-stop-machine

Testing was performed using late-loading on Intel CPUs, which default to
the NMI-based update path.

Thanks,
Chang

[1]: https://lore.kernel.org/lkml/20251013143444.3999-1-david.kaplan@amd.com/
[2]: https://youtu.be/2eEPYt5XrCE?si=NUGF2pkqk8MUh9GR&t=1739
[3]: 8f849ff63bcb ("x86/microcode: Handle "offline" CPUs correctly")

Chang S. Bae (4):
  x86/microcode: Distinguish NMI control path on stop-machine callback
  x86/microcode: Use stop-machine NMI facility
  x86/nmi: Reference stop-machine static key for offline microcode
    handler
  x86/microcode: Remove microcode_nmi_handler_enable

David Kaplan (3):
  stop_machine: Introduce stop_machine_nmi()
  x86/apic: Implement self-NMI support
  x86/nmi: Support stop_machine_nmi() handler

 arch/x86/include/asm/microcode.h     | 11 +---
 arch/x86/kernel/apic/ipi.c           |  7 +++
 arch/x86/kernel/cpu/microcode/core.c | 35 +++---------
 arch/x86/kernel/nmi.c                |  7 ++-
 include/linux/stop_machine.h         | 50 +++++++++++++++++
 kernel/stop_machine.c                | 84 ++++++++++++++++++++++++++--
 6 files changed, 150 insertions(+), 44 deletions(-)


base-commit: 24d479d26b25bce5faea3ddd9fa8f3a6c3129ea7
-- 
2.51.0

[PATCH 1/7] stop_machine: Introduce stop_machine_nmi()

[Chang S. Bae]

From: David Kaplan <david.kaplan@amd.com>

stop_machine_nmi() is a variant of stop_machine() that runs the specified
function in NMI context.  This is useful for flows that cannot tolerate any
risk of interruption even due to an NMI.  Arch-specific code must handle
sending the actual NMI and running the stop_machine_nmi_handler().

Signed-off-by: David Kaplan <david.kaplan@amd.com>
Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com>
---
Update from the original version:
* Move static key handling into stop_machine_cpuslocked_nmi() to support
  core-code users that already hold cpu hotplug locks
* Tweak the subject to better reflect the new interface and changelog a
  bit as well
---
 include/linux/stop_machine.h | 50 +++++++++++++++++++++
 kernel/stop_machine.c        | 84 ++++++++++++++++++++++++++++++++++--
 2 files changed, 130 insertions(+), 4 deletions(-)

diff --git a/include/linux/stop_machine.h b/include/linux/stop_machine.h
index 72820503514c..86113084456a 100644
--- a/include/linux/stop_machine.h
+++ b/include/linux/stop_machine.h
@@ -141,6 +141,29 @@ int stop_machine(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus);
  */
 int stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus);
 
+/**
+ * stop_machine_nmi: freeze the machine and run this function in NMI context
+ * @fn: the function to run
+ * @data: the data ptr for the @fn()
+ * @cpus: the cpus to run the @fn() on (NULL = any online cpu)
+ *
+ * Like stop_machine() but runs the function in NMI context to avoid any risk of
+ * interruption due to NMIs.
+ *
+ * Protects against CPU hotplug.
+ */
+int stop_machine_nmi(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus);
+
+/**
+ * stop_machine_cpuslocked_nmi: freeze and run this function in NMI context
+ * @fn: the function to run
+ * @data: the data ptr for the @fn()
+ * @cpus: the cpus to run the @fn() on (NULL = any online cpu)
+ *
+ * Same as above. Must be called from within a cpus_read_lock() protected
+ * region. Avoids nested calls to cpus_read_lock().
+ */
+int stop_machine_cpuslocked_nmi(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus);
 /**
  * stop_core_cpuslocked: - stop all threads on just one core
  * @cpu: any cpu in the targeted core
@@ -160,6 +183,14 @@ int stop_core_cpuslocked(unsigned int cpu, cpu_stop_fn_t fn, void *data);
 
 int stop_machine_from_inactive_cpu(cpu_stop_fn_t fn, void *data,
 				   const struct cpumask *cpus);
+
+bool noinstr stop_machine_nmi_handler(void);
+DECLARE_STATIC_KEY_FALSE(stop_machine_nmi_handler_enable);
+static __always_inline bool stop_machine_nmi_handler_enabled(void)
+{
+	return static_branch_unlikely(&stop_machine_nmi_handler_enable);
+}
+
 #else	/* CONFIG_SMP || CONFIG_HOTPLUG_CPU */
 
 static __always_inline int stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
@@ -186,5 +217,24 @@ stop_machine_from_inactive_cpu(cpu_stop_fn_t fn, void *data,
 	return stop_machine(fn, data, cpus);
 }
 
+/* stop_machine_nmi() is only supported in SMP systems. */
+static __always_inline int stop_machine_nmi(cpu_stop_fn_t fn, void *data,
+					const struct cpumask *cpus)
+{
+	return -EINVAL;
+}
+
+static __always_inline bool stop_machine_nmi_handler_enabled(void)
+{
+	return false;
+}
+
+static __always_inline bool stop_machine_nmi_handler(void)
+{
+	return false;
+}
+
 #endif	/* CONFIG_SMP || CONFIG_HOTPLUG_CPU */
+
+void arch_send_self_nmi(void);
 #endif	/* _LINUX_STOP_MACHINE */
diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c
index 3fe6b0c99f3d..189b4b108d13 100644
--- a/kernel/stop_machine.c
+++ b/kernel/stop_machine.c
@@ -174,6 +174,8 @@ struct multi_stop_data {
 
 	enum multi_stop_state	state;
 	atomic_t		thread_ack;
+
+	bool			use_nmi;
 };
 
 static void set_state(struct multi_stop_data *msdata,
@@ -197,6 +199,42 @@ notrace void __weak stop_machine_yield(const struct cpumask *cpumask)
 	cpu_relax();
 }
 
+struct stop_machine_nmi_ctrl {
+	bool nmi_enabled;
+	struct multi_stop_data *msdata;
+	int err;
+};
+
+DEFINE_STATIC_KEY_FALSE(stop_machine_nmi_handler_enable);
+static DEFINE_PER_CPU(struct stop_machine_nmi_ctrl, stop_machine_nmi_ctrl);
+
+static void enable_nmi_handler(struct multi_stop_data *msdata)
+{
+	this_cpu_write(stop_machine_nmi_ctrl.msdata, msdata);
+	this_cpu_write(stop_machine_nmi_ctrl.nmi_enabled, true);
+}
+
+void __weak arch_send_self_nmi(void)
+{
+	/* Arch code must implement this to support stop_machine_nmi() */
+}
+
+bool noinstr stop_machine_nmi_handler(void)
+{
+	struct multi_stop_data *msdata;
+	int err;
+
+	if (!raw_cpu_read(stop_machine_nmi_ctrl.nmi_enabled))
+		return false;
+
+	raw_cpu_write(stop_machine_nmi_ctrl.nmi_enabled, false);
+
+	msdata = raw_cpu_read(stop_machine_nmi_ctrl.msdata);
+	err = msdata->fn(msdata->data);
+	raw_cpu_write(stop_machine_nmi_ctrl.err, err);
+	return true;
+}
+
 /* This is the cpu_stop function which stops the CPU. */
 static int multi_cpu_stop(void *data)
 {
@@ -234,8 +272,15 @@ static int multi_cpu_stop(void *data)
 				hard_irq_disable();
 				break;
 			case MULTI_STOP_RUN:
-				if (is_active)
-					err = msdata->fn(msdata->data);
+				if (is_active) {
+					if (msdata->use_nmi) {
+						enable_nmi_handler(msdata);
+						arch_send_self_nmi();
+						err = raw_cpu_read(stop_machine_nmi_ctrl.err);
+					} else {
+						err = msdata->fn(msdata->data);
+					}
+				}
 				break;
 			default:
 				break;
@@ -584,14 +629,15 @@ static int __init cpu_stop_init(void)
 }
 early_initcall(cpu_stop_init);
 
-int stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
-			    const struct cpumask *cpus)
+static int __stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
+			    const struct cpumask *cpus, bool use_nmi)
 {
 	struct multi_stop_data msdata = {
 		.fn = fn,
 		.data = data,
 		.num_threads = num_online_cpus(),
 		.active_cpus = cpus,
+		.use_nmi = use_nmi,
 	};
 
 	lockdep_assert_cpus_held();
@@ -620,6 +666,24 @@ int stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
 	return stop_cpus(cpu_online_mask, multi_cpu_stop, &msdata);
 }
 
+int stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
+			    const struct cpumask *cpus)
+{
+	return __stop_machine_cpuslocked(fn, data, cpus, false);
+}
+
+int stop_machine_cpuslocked_nmi(cpu_stop_fn_t fn, void *data,
+				const struct cpumask *cpus)
+{
+	int ret;
+
+	static_branch_enable_cpuslocked(&stop_machine_nmi_handler_enable);
+	ret = __stop_machine_cpuslocked(fn, data, cpus, true);
+	static_branch_disable_cpuslocked(&stop_machine_nmi_handler_enable);
+
+	return ret;
+}
+
 int stop_machine(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus)
 {
 	int ret;
@@ -632,6 +696,18 @@ int stop_machine(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus)
 }
 EXPORT_SYMBOL_GPL(stop_machine);
 
+int stop_machine_nmi(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus)
+{
+	int ret;
+
+	cpus_read_lock();
+	ret = stop_machine_cpuslocked_nmi(fn, data, cpus);
+	cpus_read_unlock();
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(stop_machine_nmi);
+
 #ifdef CONFIG_SCHED_SMT
 int stop_core_cpuslocked(unsigned int cpu, cpu_stop_fn_t fn, void *data)
 {
-- 
2.51.0

Re: [PATCH 1/7] stop_machine: Introduce stop_machine_nmi()

[kernel test robot]

Hi Chang,

kernel test robot noticed the following build errors:

[auto build test ERROR on 24d479d26b25bce5faea3ddd9fa8f3a6c3129ea7]

url:    https://github.com/intel-lab-lkp/linux/commits/Chang-S-Bae/stop_machine-Introduce-stop_machine_nmi/20260125-101013
base:   24d479d26b25bce5faea3ddd9fa8f3a6c3129ea7
patch link:    https://lore.kernel.org/r/20260125014224.249901-2-chang.seok.bae%40intel.com
patch subject: [PATCH 1/7] stop_machine: Introduce stop_machine_nmi()
config: x86_64-randconfig-r133-20260126 (https://download.01.org/0day-ci/archive/20260126/202601261901.Yj2kmvgI-lkp@intel.com/config)
compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260126/202601261901.Yj2kmvgI-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202601261901.Yj2kmvgI-lkp@intel.com/

All errors (new ones prefixed by >>):

>> vmlinux.o: error: objtool: stop_machine_nmi_handler+0x23: call to {dynamic}() leaves .noinstr.text section

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH 1/7] stop_machine: Introduce stop_machine_nmi()

[Borislav Petkov]

On Mon, Jan 26, 2026 at 07:51:11PM +0800, kernel test robot wrote:
> Hi Chang,
> 
> kernel test robot noticed the following build errors:
> 
> [auto build test ERROR on 24d479d26b25bce5faea3ddd9fa8f3a6c3129ea7]
> 
> url:    https://github.com/intel-lab-lkp/linux/commits/Chang-S-Bae/stop_machine-Introduce-stop_machine_nmi/20260125-101013
> base:   24d479d26b25bce5faea3ddd9fa8f3a6c3129ea7
> patch link:    https://lore.kernel.org/r/20260125014224.249901-2-chang.seok.bae%40intel.com
> patch subject: [PATCH 1/7] stop_machine: Introduce stop_machine_nmi()
> config: x86_64-randconfig-r133-20260126 (https://download.01.org/0day-ci/archive/20260126/202601261901.Yj2kmvgI-lkp@intel.com/config)
> compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)
> reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260126/202601261901.Yj2kmvgI-lkp@intel.com/reproduce)
> 
> If you fix the issue in a separate patch/commit (i.e. not just a new version of
> the same patch/commit), kindly add following tags
> | Reported-by: kernel test robot <lkp@intel.com>
> | Closes: https://lore.kernel.org/oe-kbuild-all/202601261901.Yj2kmvgI-lkp@intel.com/
> 
> All errors (new ones prefixed by >>):
> 
> >> vmlinux.o: error: objtool: stop_machine_nmi_handler+0x23: call to {dynamic}() leaves .noinstr.text section

This one would be hard to fix because that's the

	err = msdata->fn(msdata->data);

function pointer which objtool can't see through.

So my suggestion would be to do:

struct nmi_multi_stop_data {
	cpu_stop_safe_fn_t           fn;

note the different function pointer type which we will hopefully catch during
review in the sense that only *safe* functions should be callable by the
NMI-specific stomp_machine.

Along with:

diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c
index 189b4b108d13..55a350048d4c 100644
--- a/kernel/stop_machine.c
+++ b/kernel/stop_machine.c
@@ -230,7 +230,9 @@ bool noinstr stop_machine_nmi_handler(void)
 	raw_cpu_write(stop_machine_nmi_ctrl.nmi_enabled, false);
 
 	msdata = raw_cpu_read(stop_machine_nmi_ctrl.msdata);
+	instrumentation_begin();
 	err = msdata->fn(msdata->data);
+	instrumentation_end();
 	raw_cpu_write(stop_machine_nmi_ctrl.err, err);
 	return true;
 }

ofc.

Unless someone has a better idea ofc...

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH 1/7] stop_machine: Introduce stop_machine_nmi()

[Chang S. Bae]

On 1/27/2026 6:49 AM, Borislav Petkov wrote:
>>
>>>> vmlinux.o: error: objtool: stop_machine_nmi_handler+0x23: call to {dynamic}() leaves .noinstr.text section
> 
> This one would be hard to fix because that's the
> 
> 	err = msdata->fn(msdata->data);
> 
> function pointer which objtool can't see through.

Yes. Under certain config, the 0-day bot tirggers this with 
CONFIG_NOINSTR_VALIDATION=y.

Objtool ends up with the following path:

tools/objtool/check.c::validation_call()
-> noinstr_call_dest()
    {
	/*
	 * We can't deal with indirect function calls at present;
	 * assume they're instrumented.
	 */
	if (!func) {
		if (file->pv_ops)
			return pv_call_dest(file, insn);

		return false;
	}
     ...
    }

So every indirect call under noinstr is conservatively treated as 
invalid except for pv_ops which seems to be investigated further. In 
this series, however, the destination code is to in .noinstr.text.

Given this, I considered something to be adjusted on the tool side, e.g. 
by whitelisting the case along the lines of

   if (!func) {
     ...

     if (!strncmp(insn_func(insn)->name, "stop_machine_nmi_handler", 24))
       return true;

     return false;
   }

The existing pv_ops handling also looks somewhat misaligned with large 
objects like vmlinux.o.

> 
> So my suggestion would be to do:
> 
> struct nmi_multi_stop_data {
> 	cpu_stop_safe_fn_t           fn;
> 
> note the different function pointer type which we will hopefully catch during
> review in the sense that only *safe* functions should be callable by the
> NMI-specific stomp_machine.
> 
> Along with:
> 
> diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c
> index 189b4b108d13..55a350048d4c 100644
> --- a/kernel/stop_machine.c
> +++ b/kernel/stop_machine.c
> @@ -230,7 +230,9 @@ bool noinstr stop_machine_nmi_handler(void)
>   	raw_cpu_write(stop_machine_nmi_ctrl.nmi_enabled, false);
>   
>   	msdata = raw_cpu_read(stop_machine_nmi_ctrl.msdata);
> +	instrumentation_begin();
>   	err = msdata->fn(msdata->data);
> +	instrumentation_end();
>   	raw_cpu_write(stop_machine_nmi_ctrl.err, err);
>   	return true;
>   }

I also thought this change, but I was a bit hesitant as it could be seen 
as lifting up the no-instrumentation.

Now I suppose the net effect will be the same with other workaround, 
e.g. tool-side whitelisting. So I think I can go with this as you 
suggested (with a clear note on this).

> 
> ofc.
> 
> Unless someone has a better idea ofc...

Sure.

Thanks,
Chang

Re: [PATCH 1/7] stop_machine: Introduce stop_machine_nmi()

[Borislav Petkov]

On Sun, Jan 25, 2026 at 01:42:16AM +0000, Chang S. Bae wrote:
> +/**
> + * stop_machine_nmi: freeze the machine and run this function in NMI context
> + * @fn: the function to run
> + * @data: the data ptr for the @fn()
> + * @cpus: the cpus to run the @fn() on (NULL = any online cpu)

s/cpu/CPU/g in all text.

> + *
> + * Like stop_machine() but runs the function in NMI context to avoid any risk of
> + * interruption due to NMIs.
> + *
> + * Protects against CPU hotplug.
> + */
> +int stop_machine_nmi(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus);
> +
> +/**
> + * stop_machine_cpuslocked_nmi: freeze and run this function in NMI context
> + * @fn: the function to run
> + * @data: the data ptr for the @fn()
> + * @cpus: the cpus to run the @fn() on (NULL = any online cpu)
> + *
> + * Same as above. Must be called from within a cpus_read_lock() protected
> + * region. Avoids nested calls to cpus_read_lock().
> + */
> +int stop_machine_cpuslocked_nmi(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus);
>  /**
>   * stop_core_cpuslocked: - stop all threads on just one core
>   * @cpu: any cpu in the targeted core
> @@ -160,6 +183,14 @@ int stop_core_cpuslocked(unsigned int cpu, cpu_stop_fn_t fn, void *data);
>  
>  int stop_machine_from_inactive_cpu(cpu_stop_fn_t fn, void *data,
>  				   const struct cpumask *cpus);
> +
> +bool noinstr stop_machine_nmi_handler(void);
> +DECLARE_STATIC_KEY_FALSE(stop_machine_nmi_handler_enable);

Why is the static key in the header if you have an accessor below?

> +static __always_inline bool stop_machine_nmi_handler_enabled(void)
> +{
> +	return static_branch_unlikely(&stop_machine_nmi_handler_enable);
> +}

Just make the accessor the only thing that external code calls.

...

> +DEFINE_STATIC_KEY_FALSE(stop_machine_nmi_handler_enable);
> +static DEFINE_PER_CPU(struct stop_machine_nmi_ctrl, stop_machine_nmi_ctrl);
> +
> +static void enable_nmi_handler(struct multi_stop_data *msdata)
> +{
> +	this_cpu_write(stop_machine_nmi_ctrl.msdata, msdata);
> +	this_cpu_write(stop_machine_nmi_ctrl.nmi_enabled, true);
> +}

Why do we have to enable the NMI handler?

> +void __weak arch_send_self_nmi(void)
> +{
> +	/* Arch code must implement this to support stop_machine_nmi() */
> +}

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

RE: [PATCH 1/7] stop_machine: Introduce stop_machine_nmi()

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Borislav Petkov <bp@alien8.de>
> Sent: Tuesday, January 27, 2026 9:50 AM
> To: Chang S. Bae <chang.seok.bae@intel.com>
> Cc: linux-kernel@vger.kernel.org; x86@kernel.org; tglx@linutronix.de;
> mingo@redhat.com; dave.hansen@linux.intel.com; peterz@infradead.org;
> Kaplan, David <David.Kaplan@amd.com>
> Subject: Re: [PATCH 1/7] stop_machine: Introduce stop_machine_nmi()
>
> Caution: This message originated from an External Source. Use proper
> caution when opening attachments, clicking links, or responding.
>
>
> On Sun, Jan 25, 2026 at 01:42:16AM +0000, Chang S. Bae wrote:
> > +/**
> > + * stop_machine_nmi: freeze the machine and run this function in NMI
> context
> > + * @fn: the function to run
> > + * @data: the data ptr for the @fn()
> > + * @cpus: the cpus to run the @fn() on (NULL = any online cpu)
>
> s/cpu/CPU/g in all text.

The existing stop_machine() function description in the same header uses lowercase cpu.  This was intended to match.

>
> > + *
> > + * Like stop_machine() but runs the function in NMI context to avoid any
> risk of
> > + * interruption due to NMIs.
> > + *
> > + * Protects against CPU hotplug.
> > + */
> > +int stop_machine_nmi(cpu_stop_fn_t fn, void *data, const struct cpumask
> *cpus);
> > +
> > +/**
> > + * stop_machine_cpuslocked_nmi: freeze and run this function in NMI
> context
> > + * @fn: the function to run
> > + * @data: the data ptr for the @fn()
> > + * @cpus: the cpus to run the @fn() on (NULL = any online cpu)
> > + *
> > + * Same as above. Must be called from within a cpus_read_lock()
> protected
> > + * region. Avoids nested calls to cpus_read_lock().
> > + */
> > +int stop_machine_cpuslocked_nmi(cpu_stop_fn_t fn, void *data, const
> struct cpumask *cpus);
> >  /**
> >   * stop_core_cpuslocked: - stop all threads on just one core
> >   * @cpu: any cpu in the targeted core
> > @@ -160,6 +183,14 @@ int stop_core_cpuslocked(unsigned int cpu,
> cpu_stop_fn_t fn, void *data);
> >
> >  int stop_machine_from_inactive_cpu(cpu_stop_fn_t fn, void *data,
> >                                  const struct cpumask *cpus);
> > +
> > +bool noinstr stop_machine_nmi_handler(void);
> > +DECLARE_STATIC_KEY_FALSE(stop_machine_nmi_handler_enable);
>
> Why is the static key in the header if you have an accessor below?
>
> > +static __always_inline bool stop_machine_nmi_handler_enabled(void)
> > +{
> > +     return static_branch_unlikely(&stop_machine_nmi_handler_enable);
> > +}
>
> Just make the accessor the only thing that external code calls.

Not entirely sure I follow the suggestion, but keep in mind that stop_machine.c is only included in the build if CONFIG_SMP.  So I had to put some stuff in the header to ensure that a non-SMP build would not fail.

>
> ...
>
> > +DEFINE_STATIC_KEY_FALSE(stop_machine_nmi_handler_enable);
> > +static DEFINE_PER_CPU(struct stop_machine_nmi_ctrl,
> stop_machine_nmi_ctrl);
> > +
> > +static void enable_nmi_handler(struct multi_stop_data *msdata)
> > +{
> > +     this_cpu_write(stop_machine_nmi_ctrl.msdata, msdata);
> > +     this_cpu_write(stop_machine_nmi_ctrl.nmi_enabled, true);
> > +}
>
> Why do we have to enable the NMI handler?

This was in the existing CPU patching logic.  I believe it is intended to protect against the handler running multiple times.

In particular, there are some cases where NMIs can get unmasked early so there could be a risk I'd think of a second NMI coming in while the handler is running.  The Boolean protects against that.  Maybe Chang knows more history here.

That said, the whole point of stop_machine_nmi() is to avoid an NMI coming in, and for dynamic mitigations I explicitly made it mutually exclusive with DEBUG_ENTRY so this wouldn't happen...

--David Kaplan

Re: [PATCH 1/7] stop_machine: Introduce stop_machine_nmi()

[Borislav Petkov]

On Tue, Jan 27, 2026 at 04:00:42PM +0000, Kaplan, David wrote:
> The existing stop_machine() function description in the same header uses
> lowercase cpu.  This was intended to match.

You can change yours to "CPU" and I can convert the rest. Or I can do them all
- not a biggie.

> Not entirely sure I follow the suggestion, but keep in mind that
> stop_machine.c is only included in the build if CONFIG_SMP.  So I had to put
> some stuff in the header to ensure that a non-SMP build would not fail.

Something like this:

diff --git a/include/linux/stop_machine.h b/include/linux/stop_machine.h
index 86113084456a..933fe8ddbec9 100644
--- a/include/linux/stop_machine.h
+++ b/include/linux/stop_machine.h
@@ -184,13 +184,8 @@ int stop_core_cpuslocked(unsigned int cpu, cpu_stop_fn_t fn, void *data);
 int stop_machine_from_inactive_cpu(cpu_stop_fn_t fn, void *data,
 				   const struct cpumask *cpus);
 
+bool stop_machine_nmi_handler_enabled(void);
 bool noinstr stop_machine_nmi_handler(void);
-DECLARE_STATIC_KEY_FALSE(stop_machine_nmi_handler_enable);
-static __always_inline bool stop_machine_nmi_handler_enabled(void)
-{
-	return static_branch_unlikely(&stop_machine_nmi_handler_enable);
-}
-
 #else	/* CONFIG_SMP || CONFIG_HOTPLUG_CPU */
 
 static __always_inline int stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c
index 55a350048d4c..0ecec98d52a6 100644
--- a/kernel/stop_machine.c
+++ b/kernel/stop_machine.c
@@ -205,7 +205,13 @@ struct stop_machine_nmi_ctrl {
 	int err;
 };
 
-DEFINE_STATIC_KEY_FALSE(stop_machine_nmi_handler_enable);
+static DEFINE_STATIC_KEY_FALSE(stop_machine_nmi_handler_enable);
+
+bool stop_machine_nmi_handler_enabled(void)
+{
+	return static_branch_unlikely(&stop_machine_nmi_handler_enable);
+}
+
 static DEFINE_PER_CPU(struct stop_machine_nmi_ctrl, stop_machine_nmi_ctrl);
 
 static void enable_nmi_handler(struct multi_stop_data *msdata)

> This was in the existing CPU patching logic.  I believe it is intended to
> protect against the handler running multiple times.
> 
> In particular, there are some cases where NMIs can get unmasked early so
> there could be a risk I'd think of a second NMI coming in while the handler
> is running.  The Boolean protects against that.  Maybe Chang knows more
> history here.
> 
> That said, the whole point of stop_machine_nmi() is to avoid an NMI coming
> in, and for dynamic mitigations I explicitly made it mutually exclusive with
> DEBUG_ENTRY so this wouldn't happen...

Except that you don't need any of that fun. You can do:

	struct multi_stop_data {
        cpu_stop_fn_t           fn;
        void                    *data;
        /* Like num_online_cpus(), but hotplug cpu uses us, so we need this. */
        unsigned int            num_threads;
        const struct cpumask    *active_cpus;
 
        enum multi_stop_state   state;
        atomic_t                thread_ack;
 
        bool                    use_nmi;

	/* cpumask of CPUs which need to run an NMI handler */
	const struct cpumask	nmi_cpusmask;
	^^^^^^

add this thing

};

which you initialize to the cpumask you want NMIs to run on.

stop_machine_nmi_handler() tests its CPU ID against the mask, if the bit is
set, it clears it and runs the function pointer.

You don't need to enable anything as it is implicitly enabled when you have
that bit in the cpumask set and you don't need to test whether anything's
enabled because, well, also implicitly enabled for only once on the respective
CPU.

And then you can wrap all that logic:

                        case MULTI_STOP_RUN:
                                if (is_active) {
                                        if (msdata->use_nmi) {
                                                arch_send_self_nmi();
                                                err = raw_cpu_read(stop_machine_nmi_ctrl.err);
                                        } else {
                                                err = msdata->fn(msdata->data);
                                        }
                                }

into a single function which multi_cpu_stop() calls and in that function you
decide whether to do an NMI or do a normal function call.

Sounds good?

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

RE: [PATCH 1/7] stop_machine: Introduce stop_machine_nmi()

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Borislav Petkov <bp@alien8.de>
> Sent: Tuesday, January 27, 2026 12:50 PM
> To: Kaplan, David <David.Kaplan@amd.com>
> Cc: Chang S. Bae <chang.seok.bae@intel.com>; linux-kernel@vger.kernel.org;
> x86@kernel.org; tglx@linutronix.de; mingo@redhat.com;
> dave.hansen@linux.intel.com; peterz@infradead.org
> Subject: Re: [PATCH 1/7] stop_machine: Introduce stop_machine_nmi()
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On Tue, Jan 27, 2026 at 04:00:42PM +0000, Kaplan, David wrote:
> > The existing stop_machine() function description in the same header uses
> > lowercase cpu.  This was intended to match.
>
> You can change yours to "CPU" and I can convert the rest. Or I can do them all
> - not a biggie.
>
> > Not entirely sure I follow the suggestion, but keep in mind that
> > stop_machine.c is only included in the build if CONFIG_SMP.  So I had to put
> > some stuff in the header to ensure that a non-SMP build would not fail.
>
> Something like this:
>
> diff --git a/include/linux/stop_machine.h b/include/linux/stop_machine.h
> index 86113084456a..933fe8ddbec9 100644
> --- a/include/linux/stop_machine.h
> +++ b/include/linux/stop_machine.h
> @@ -184,13 +184,8 @@ int stop_core_cpuslocked(unsigned int cpu,
> cpu_stop_fn_t fn, void *data);
>  int stop_machine_from_inactive_cpu(cpu_stop_fn_t fn, void *data,
>                                    const struct cpumask *cpus);
>
> +bool stop_machine_nmi_handler_enabled(void);
>  bool noinstr stop_machine_nmi_handler(void);
> -DECLARE_STATIC_KEY_FALSE(stop_machine_nmi_handler_enable);
> -static __always_inline bool stop_machine_nmi_handler_enabled(void)
> -{
> -       return static_branch_unlikely(&stop_machine_nmi_handler_enable);
> -}
> -
>  #else  /* CONFIG_SMP || CONFIG_HOTPLUG_CPU */
>
>  static __always_inline int stop_machine_cpuslocked(cpu_stop_fn_t fn, void
> *data,
> diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c
> index 55a350048d4c..0ecec98d52a6 100644
> --- a/kernel/stop_machine.c
> +++ b/kernel/stop_machine.c
> @@ -205,7 +205,13 @@ struct stop_machine_nmi_ctrl {
>         int err;
>  };
>
> -DEFINE_STATIC_KEY_FALSE(stop_machine_nmi_handler_enable);
> +static DEFINE_STATIC_KEY_FALSE(stop_machine_nmi_handler_enable);
> +
> +bool stop_machine_nmi_handler_enabled(void)
> +{
> +       return static_branch_unlikely(&stop_machine_nmi_handler_enable);
> +}
> +
>  static DEFINE_PER_CPU(struct stop_machine_nmi_ctrl,
> stop_machine_nmi_ctrl);
>
>  static void enable_nmi_handler(struct multi_stop_data *msdata)

Ok, that's what I thought you were suggesting.  But as I noted, I don't think that works if CONFIG_SMP=n because stop_machine.c isn't built.

I think you could fix this by adding an #ifdef CONFIG_SMP in the header and a dummy definition of stop_machine_nmi_handler_enabled in that case, but is it worth adding another #ifdef/#else/#endif just to move the static key declaration?

>
> > This was in the existing CPU patching logic.  I believe it is intended to
> > protect against the handler running multiple times.
> >
> > In particular, there are some cases where NMIs can get unmasked early so
> > there could be a risk I'd think of a second NMI coming in while the handler
> > is running.  The Boolean protects against that.  Maybe Chang knows more
> > history here.
> >
> > That said, the whole point of stop_machine_nmi() is to avoid an NMI coming
> > in, and for dynamic mitigations I explicitly made it mutually exclusive with
> > DEBUG_ENTRY so this wouldn't happen...
>
> Except that you don't need any of that fun. You can do:
>
>         struct multi_stop_data {
>         cpu_stop_fn_t           fn;
>         void                    *data;
>         /* Like num_online_cpus(), but hotplug cpu uses us, so we need this. */
>         unsigned int            num_threads;
>         const struct cpumask    *active_cpus;
>
>         enum multi_stop_state   state;
>         atomic_t                thread_ack;
>
>         bool                    use_nmi;
>
>         /* cpumask of CPUs which need to run an NMI handler */
>         const struct cpumask    nmi_cpusmask;
>         ^^^^^^
>
> add this thing
>
> };
>
> which you initialize to the cpumask you want NMIs to run on.
>
> stop_machine_nmi_handler() tests its CPU ID against the mask, if the bit is
> set, it clears it and runs the function pointer.
>
> You don't need to enable anything as it is implicitly enabled when you have
> that bit in the cpumask set and you don't need to test whether anything's
> enabled because, well, also implicitly enabled for only once on the respective
> CPU.
>
> And then you can wrap all that logic:
>
>                         case MULTI_STOP_RUN:
>                                 if (is_active) {
>                                         if (msdata->use_nmi) {
>                                                 arch_send_self_nmi();
>                                                 err = raw_cpu_read(stop_machine_nmi_ctrl.err);
>                                         } else {
>                                                 err = msdata->fn(msdata->data);
>                                         }
>                                 }
>
> into a single function which multi_cpu_stop() calls and in that function you
> decide whether to do an NMI or do a normal function call.
>
> Sounds good?
>

I think that can also work, but just curious why that would be preferred?  It seems very similar in theory (test a bit to see if the handler hasn't been run yet on this cpu and if so clear it and run the handler).  It might be slightly less memory usage (I think) but creates more cache contention with every core trying to modify the one nmi_cpumask value.  And while the perf of any stop machine isn't meant to be the best, we still don't want the NMI handler to run longer than it has to and having hundreds of cores fighting over the same cacheline to clear their bit doesn't seem ideal.

Thanks
--David Kaplan

Re: [PATCH 1/7] stop_machine: Introduce stop_machine_nmi()

[Borislav Petkov]

On Wed, Jan 28, 2026 at 01:31:44AM +0000, Kaplan, David wrote:
> Ok, that's what I thought you were suggesting.  But as I noted, I don't
> think that works if CONFIG_SMP=n because stop_machine.c isn't built.
> 
> I think you could fix this by adding an #ifdef CONFIG_SMP in the header and
> a dummy definition of stop_machine_nmi_handler_enabled in that case, but is
> it worth adding another #ifdef/#else/#endif just to move the static key
> declaration?

That's already there:

#else	/* CONFIG_SMP || CONFIG_HOTPLUG_CPU */
...

static __always_inline bool stop_machine_nmi_handler_enabled(void)
{
        return false;
}

:-)

That's the usual pattern with those things.

But we might not need it anymore.

> I think that can also work, but just curious why that would be preferred?

Simpler code.

> It seems very similar in theory (test a bit to see if the handler hasn't
> been run yet on this cpu and if so clear it and run the handler).  It might
> be slightly less memory usage (I think) but creates more cache contention
> with every core trying to modify the one nmi_cpumask value.

We don't care about performance with stomp_machine, right? It is a big fat
hammer and bouncing a cacheline in the face of NMIs firing on every core is
probably meh...

> And while the perf of any stop machine isn't meant to be the best, we still
> don't want the NMI handler to run longer than it has to and having hundreds
> of cores fighting over the same cacheline to clear their bit doesn't seem
> ideal.

I'd say let's make the simplest variant work first and then go optimize it.

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH 1/7] stop_machine: Introduce stop_machine_nmi()

[Borislav Petkov]

On Wed, Jan 28, 2026 at 05:35:56PM +0100, Borislav Petkov wrote:
> I'd say let's make the simplest variant work first and then go optimize it.

IOW, something like this, ontop.

What I'm not sure about is:

        if (msdata->use_nmi) {
                this_cpu_write(stop_machine_nmi_ctrl.msdata, msdata);
                arch_send_self_nmi();

<--- we send the NMI IPI here...

                return raw_cpu_read(stop_machine_nmi_ctrl.err);

... and we read the err result immediately but what guarantees us that the NMI
handler on that CPU will have run and written err:

	raw_cpu_write(stop_machine_nmi_ctrl.err, err);

?

In any case, this looks simpler to me. You have a single struct
multi_stop_data.nmi_cpus which accounts for which CPU runs the NMI handler and
there's no need for any enable logic anymore.

---

diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c
index 0ecec98d52a6..02faf75aa974 100644
--- a/kernel/stop_machine.c
+++ b/kernel/stop_machine.c
@@ -176,8 +176,21 @@ struct multi_stop_data {
 	atomic_t		thread_ack;
 
 	bool			use_nmi;
+
+	/*
+	 * cpumask of CPUs on which to raise an NMI; used in the NMI
+	 * stomp_machine variant.
+	 */
+	struct cpumask		nmi_cpus;
 };
 
+struct stop_machine_nmi_ctrl {
+	struct multi_stop_data *msdata;
+	int err;
+};
+
+static DEFINE_PER_CPU(struct stop_machine_nmi_ctrl, stop_machine_nmi_ctrl);
+
 static void set_state(struct multi_stop_data *msdata,
 		      enum multi_stop_state newstate)
 {
@@ -199,27 +212,6 @@ notrace void __weak stop_machine_yield(const struct cpumask *cpumask)
 	cpu_relax();
 }
 
-struct stop_machine_nmi_ctrl {
-	bool nmi_enabled;
-	struct multi_stop_data *msdata;
-	int err;
-};
-
-static DEFINE_STATIC_KEY_FALSE(stop_machine_nmi_handler_enable);
-
-bool stop_machine_nmi_handler_enabled(void)
-{
-	return static_branch_unlikely(&stop_machine_nmi_handler_enable);
-}
-
-static DEFINE_PER_CPU(struct stop_machine_nmi_ctrl, stop_machine_nmi_ctrl);
-
-static void enable_nmi_handler(struct multi_stop_data *msdata)
-{
-	this_cpu_write(stop_machine_nmi_ctrl.msdata, msdata);
-	this_cpu_write(stop_machine_nmi_ctrl.nmi_enabled, true);
-}
-
 void __weak arch_send_self_nmi(void)
 {
 	/* Arch code must implement this to support stop_machine_nmi() */
@@ -227,15 +219,12 @@ void __weak arch_send_self_nmi(void)
 
 bool noinstr stop_machine_nmi_handler(void)
 {
-	struct multi_stop_data *msdata;
+	struct multi_stop_data *msdata = raw_cpu_read(stop_machine_nmi_ctrl.msdata);
 	int err;
 
-	if (!raw_cpu_read(stop_machine_nmi_ctrl.nmi_enabled))
+	if (!cpumask_test_and_clear_cpu(smp_processor_id(), &msdata->nmi_cpus))
 		return false;
 
-	raw_cpu_write(stop_machine_nmi_ctrl.nmi_enabled, false);
-
-	msdata = raw_cpu_read(stop_machine_nmi_ctrl.msdata);
 	instrumentation_begin();
 	err = msdata->fn(msdata->data);
 	instrumentation_end();
@@ -243,6 +232,17 @@ bool noinstr stop_machine_nmi_handler(void)
 	return true;
 }
 
+static int __multi_cpu_stop(struct multi_stop_data *msdata)
+{
+	if (msdata->use_nmi) {
+		this_cpu_write(stop_machine_nmi_ctrl.msdata, msdata);
+		arch_send_self_nmi();
+		return raw_cpu_read(stop_machine_nmi_ctrl.err);
+	} else {
+		return msdata->fn(msdata->data);
+	}
+}
+
 /* This is the cpu_stop function which stops the CPU. */
 static int multi_cpu_stop(void *data)
 {
@@ -280,15 +280,8 @@ static int multi_cpu_stop(void *data)
 				hard_irq_disable();
 				break;
 			case MULTI_STOP_RUN:
-				if (is_active) {
-					if (msdata->use_nmi) {
-						enable_nmi_handler(msdata);
-						arch_send_self_nmi();
-						err = raw_cpu_read(stop_machine_nmi_ctrl.err);
-					} else {
-						err = msdata->fn(msdata->data);
-					}
-				}
+				if (is_active)
+					err = __multi_cpu_stop(msdata);
 				break;
 			default:
 				break;
@@ -648,6 +641,9 @@ static int __stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
 		.use_nmi = use_nmi,
 	};
 
+	if (use_nmi)
+		cpumask_copy(&msdata.nmi_cpus, cpus);
+
 	lockdep_assert_cpus_held();
 
 	if (!stop_machine_initialized) {
@@ -683,13 +679,7 @@ int stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
 int stop_machine_cpuslocked_nmi(cpu_stop_fn_t fn, void *data,
 				const struct cpumask *cpus)
 {
-	int ret;
-
-	static_branch_enable_cpuslocked(&stop_machine_nmi_handler_enable);
-	ret = __stop_machine_cpuslocked(fn, data, cpus, true);
-	static_branch_disable_cpuslocked(&stop_machine_nmi_handler_enable);
-
-	return ret;
+	return __stop_machine_cpuslocked(fn, data, cpus, true);
 }
 
 int stop_machine(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus)

---

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH 1/7] stop_machine: Introduce stop_machine_nmi()

[Chang S. Bae]

On 1/29/2026 4:17 AM, Borislav Petkov wrote:
> On Wed, Jan 28, 2026 at 05:35:56PM +0100, Borislav Petkov wrote:>
> -
> -static DEFINE_STATIC_KEY_FALSE(stop_machine_nmi_handler_enable);
> -
> -bool stop_machine_nmi_handler_enabled(void)
> -{
> -	return static_branch_unlikely(&stop_machine_nmi_handler_enable);
> -}
Thanks for the detailed write-up! I just wanted to clarify a few points, 
if you don't mind:

First, on patch6, it is going to be used by microcode_offline_nmi_handler()
{
	if (!raw_cpu_read(ucode_ctrl.nmi_enabled))
		return;
	raw_cpu_write(ucode_ctrl.nmi_enabled, false);
	raw_cpu_write(ucode_ctrl.result, UCODE_OFFLINE);
	raw_atomic_inc(&offline_in_nmi);
	wait_for_ctrl();
}

I assume you consider the per-CPU nmi_enabled bit is enough going there. 
Then,

DEFINE_IDTENTRY_RAW(exc_nmi)
{
  ...
	if (arch_cpu_is_offline(smp_processor_id())) {
		microcode_offline_nmi_handler();
		return;
	}
...
}

Correct?

The other bit is possible bug after the removal. The NMI path 
effectively becomes:

static noinstr void default_do_nmi(struct pt_regs *regs)
...
+	if (stop_machine_nmi_handler())
+		goto out;

which means it could be invoked from an NMI unrelated to stop-machine.

>   bool noinstr stop_machine_nmi_handler(void)
>   {
> -	struct multi_stop_data *msdata;
> +	struct multi_stop_data *msdata = raw_cpu_read(stop_machine_nmi_ctrl.msdata);

In that case, msdata could still be NULL here.

>   	int err;
>   
> -	if (!raw_cpu_read(stop_machine_nmi_ctrl.nmi_enabled))
> +	if (!cpumask_test_and_clear_cpu(smp_processor_id(), &msdata->nmi_cpus))
>   		return false;
>   
> -	raw_cpu_write(stop_machine_nmi_ctrl.nmi_enabled, false);
> -
> -	msdata = raw_cpu_read(stop_machine_nmi_ctrl.msdata);
>   	instrumentation_begin();
>   	err = msdata->fn(msdata->data);
>   	instrumentation_end();

Also, I suppose

	this_cpu_write(stop_machine_nmi_ctrl.msdata, NULL);

> @@ -243,6 +232,17 @@ bool noinstr stop_machine_nmi_handler(void)
>   	return true;
>   }
>   

Finally, while I do appreciate the nmi_cpus approach, I could also think 
another simple msdata == NULL check could be a guard here too.

Thanks,
Chang

Re: [PATCH 1/7] stop_machine: Introduce stop_machine_nmi()

[Borislav Petkov]

On Thu, Jan 29, 2026 at 01:17:29PM +0100, Borislav Petkov wrote:
> What I'm not sure about is:
> 
>         if (msdata->use_nmi) {
>                 this_cpu_write(stop_machine_nmi_ctrl.msdata, msdata);
>                 arch_send_self_nmi();
> 
> <--- we send the NMI IPI here...
> 
>                 return raw_cpu_read(stop_machine_nmi_ctrl.err);
> 
> ... and we read the err result immediately but what guarantees us that the NMI
> handler on that CPU will have run and written err:
> 
> 	raw_cpu_write(stop_machine_nmi_ctrl.err, err);
> 
> ?

Yeah, I don't think we can rely on the NMI handler running immediately after
the ICR write... so I guess we will have to OR-in the retvals after all the
NMIs have been raised on all CPUs.

I had a silly idea about that: another CPU mask. See below.

This still doesn't handle what Chang and you mentioned that the NMI handler
needs to check whether it comes from a stomp_machine call or not. That
later...

Completely untested ofc:

---
diff --git a/include/linux/stop_machine.h b/include/linux/stop_machine.h
index 72820503514c..ce5f932443cd 100644
--- a/include/linux/stop_machine.h
+++ b/include/linux/stop_machine.h
@@ -141,6 +141,29 @@ int stop_machine(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus);
  */
 int stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus);
 
+/**
+ * stop_machine_nmi: freeze the machine and run this function in NMI context
+ * @fn: the function to run
+ * @data: the data ptr for the @fn()
+ * @cpus: the CPUs to run the @fn() on (NULL = any online CPU)
+ *
+ * Like stop_machine() but runs the function in NMI context to avoid any risk of
+ * interruption due to NMIs.
+ *
+ * Protects against CPU hotplug.
+ */
+int stop_machine_nmi(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus);
+
+/**
+ * stop_machine_cpuslocked_nmi: freeze and run this function in NMI context
+ * @fn: the function to run
+ * @data: the data ptr for the @fn()
+ * @cpus: the CPUs to run the @fn() on (NULL = any online CPU)
+ *
+ * Same as above. Must be called from within a cpus_read_lock() protected
+ * region. Avoids nested calls to cpus_read_lock().
+ */
+int stop_machine_cpuslocked_nmi(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus);
 /**
  * stop_core_cpuslocked: - stop all threads on just one core
  * @cpu: any cpu in the targeted core
@@ -160,6 +183,9 @@ int stop_core_cpuslocked(unsigned int cpu, cpu_stop_fn_t fn, void *data);
 
 int stop_machine_from_inactive_cpu(cpu_stop_fn_t fn, void *data,
 				   const struct cpumask *cpus);
+
+bool stop_machine_nmi_handler_enabled(void);
+bool noinstr stop_machine_nmi_handler(void);
 #else	/* CONFIG_SMP || CONFIG_HOTPLUG_CPU */
 
 static __always_inline int stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
@@ -186,5 +212,23 @@ stop_machine_from_inactive_cpu(cpu_stop_fn_t fn, void *data,
 	return stop_machine(fn, data, cpus);
 }
 
+/* stop_machine_nmi() is only supported in SMP systems. */
+static __always_inline int stop_machine_nmi(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus)
+{
+	return -EINVAL;
+}
+
+static __always_inline bool stop_machine_nmi_handler_enabled(void)
+{
+	return false;
+}
+
+static __always_inline bool stop_machine_nmi_handler(void)
+{
+	return false;
+}
+
 #endif	/* CONFIG_SMP || CONFIG_HOTPLUG_CPU */
+
+void arch_send_self_nmi(void);
 #endif	/* _LINUX_STOP_MACHINE */
diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c
index 3fe6b0c99f3d..c6c8afc4d03e 100644
--- a/kernel/stop_machine.c
+++ b/kernel/stop_machine.c
@@ -174,8 +174,26 @@ struct multi_stop_data {
 
 	enum multi_stop_state	state;
 	atomic_t		thread_ack;
+
+	bool			use_nmi;
+
+	/*
+	 * cpumasks of CPUs on which to raise an NMI; used in the NMI
+	 * stomp_machine variant. nmi_cpus_done is used for tracking
+	 * when the NMI handler has executed successfully.
+	 */
+	struct cpumask		nmi_cpus;
+	struct cpumask		nmi_cpus_done;
+
+};
+
+struct stop_machine_nmi_ctrl {
+	struct multi_stop_data *msdata;
+	int err;
 };
 
+static DEFINE_PER_CPU(struct stop_machine_nmi_ctrl, stop_machine_nmi_ctrl);
+
 static void set_state(struct multi_stop_data *msdata,
 		      enum multi_stop_state newstate)
 {
@@ -197,6 +215,41 @@ notrace void __weak stop_machine_yield(const struct cpumask *cpumask)
 	cpu_relax();
 }
 
+void __weak arch_send_self_nmi(void)
+{
+	/* Arch code must implement this to support stop_machine_nmi() */
+}
+
+bool noinstr stop_machine_nmi_handler(void)
+{
+	struct multi_stop_data *msdata = raw_cpu_read(stop_machine_nmi_ctrl.msdata);
+	unsigned int cpu = smp_processor_id();
+	int err;
+
+	if (!cpumask_test_and_clear_cpu(cpu, &msdata->nmi_cpus))
+		return false;
+
+	instrumentation_begin();
+	err = msdata->fn(msdata->data);
+	instrumentation_end();
+	raw_cpu_write(stop_machine_nmi_ctrl.err, err);
+
+	cpumask_set_cpu(cpu, &msdata->nmi_cpus_done);
+
+	return true;
+}
+
+static int __multi_cpu_stop(struct multi_stop_data *msdata)
+{
+	if (msdata->use_nmi) {
+		this_cpu_write(stop_machine_nmi_ctrl.msdata, msdata);
+		arch_send_self_nmi();
+		return 0;
+	} else {
+		return msdata->fn(msdata->data);
+	}
+}
+
 /* This is the cpu_stop function which stops the CPU. */
 static int multi_cpu_stop(void *data)
 {
@@ -235,7 +288,7 @@ static int multi_cpu_stop(void *data)
 				break;
 			case MULTI_STOP_RUN:
 				if (is_active)
-					err = msdata->fn(msdata->data);
+					err = __multi_cpu_stop(msdata);
 				break;
 			default:
 				break;
@@ -584,15 +637,22 @@ static int __init cpu_stop_init(void)
 }
 early_initcall(cpu_stop_init);
 
-int stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
-			    const struct cpumask *cpus)
+static int __stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
+			    const struct cpumask *cpus, bool use_nmi)
 {
 	struct multi_stop_data msdata = {
 		.fn = fn,
 		.data = data,
 		.num_threads = num_online_cpus(),
 		.active_cpus = cpus,
+		.use_nmi = use_nmi,
 	};
+	int ret, cpu;
+
+	if (use_nmi) {
+		cpumask_copy(&msdata.nmi_cpus, cpus);
+		cpumask_clear(&msdata.nmi_cpus_done);
+	}
 
 	lockdep_assert_cpus_held();
 
@@ -617,7 +677,32 @@ int stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
 
 	/* Set the initial state and stop all online cpus. */
 	set_state(&msdata, MULTI_STOP_PREPARE);
-	return stop_cpus(cpu_online_mask, multi_cpu_stop, &msdata);
+	ret = stop_cpus(cpu_online_mask, multi_cpu_stop, &msdata);
+
+	if (!use_nmi)
+		return ret;
+
+	if (!cpumask_equal(cpus, &msdata.nmi_cpus_done)) {
+		pr_err("Some CPUs didn't run the stomp_machine NMI handler\n");
+		return -EINVAL;
+	} else {
+		for_each_cpu(cpu, cpus)
+			ret |= per_cpu(stop_machine_nmi_ctrl.err, cpu);
+
+		return ret;
+	}
+}
+
+int stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
+			    const struct cpumask *cpus)
+{
+	return __stop_machine_cpuslocked(fn, data, cpus, false);
+}
+
+int stop_machine_cpuslocked_nmi(cpu_stop_fn_t fn, void *data,
+				const struct cpumask *cpus)
+{
+	return __stop_machine_cpuslocked(fn, data, cpus, true);
 }
 
 int stop_machine(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus)
@@ -632,6 +717,18 @@ int stop_machine(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus)
 }
 EXPORT_SYMBOL_GPL(stop_machine);
 
+int stop_machine_nmi(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus)
+{
+	int ret;
+
+	cpus_read_lock();
+	ret = stop_machine_cpuslocked_nmi(fn, data, cpus);
+	cpus_read_unlock();
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(stop_machine_nmi);
+
 #ifdef CONFIG_SCHED_SMT
 int stop_core_cpuslocked(unsigned int cpu, cpu_stop_fn_t fn, void *data)
 {
-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH 1/7] stop_machine: Introduce stop_machine_nmi()

[Chang S. Bae]

On 2/2/2026 2:54 AM, Borislav Petkov wrote:
...
> @@ -174,8 +174,26 @@ struct multi_stop_data {
>   
>   	enum multi_stop_state	state;
>   	atomic_t		thread_ack;
> +
> +	bool			use_nmi;
> +
> +	/*
> +	 * cpumasks of CPUs on which to raise an NMI; used in the NMI
> +	 * stomp_machine variant. nmi_cpus_done is used for tracking
> +	 * when the NMI handler has executed successfully.
> +	 */
> +	struct cpumask		nmi_cpus;
> +	struct cpumask		nmi_cpus_done;
> +
> +};

Looks like every stop_machine variant then will spend stack for these 
masks. It seems they could be cpumask_var_t.

Alternatively, to make it simple further, a per-CPU variable could 
achieve this if I understand correctly:

   struct stop_machine_nmi_ctrl {
	...
	bool	done;
   }

Then,

   for_each_cpu(cpu, cpus)
   	per_cpu(stop_machine_nmi_ctrl.done, cpu) = false;

   ...
   ret = stop_cpus(cpu_online_mask, multi_cpu_stop, &msdata);
   ...

   for_each_cpu(cpu, cpus) {
   	if (!per_cpu(stop_machine_nmi_ctrl.done, cpu)) {
		pr_err("Some CPUs didn't run ...\n");
		return -EINVAL;
	}
   }

Or, I guess even the msdata pointer alone could do that too -- setting 
it before and checking if NULL after.

> +static int __stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
> +			    const struct cpumask *cpus, bool use_nmi)
>   {
>   	struct multi_stop_data msdata = {
>   		.fn = fn,
>   		.data = data,
>   		.num_threads = num_online_cpus(),
>   		.active_cpus = cpus,
> +		.use_nmi = use_nmi,
>   	};
> +	int ret, cpu;
> +
> +	if (use_nmi) {
> +		cpumask_copy(&msdata.nmi_cpus, cpus);
> +		cpumask_clear(&msdata.nmi_cpus_done);
> +	}
>   
>   	lockdep_assert_cpus_held();
>   
> @@ -617,7 +677,32 @@ int stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
>   
>   	/* Set the initial state and stop all online cpus. */
>   	set_state(&msdata, MULTI_STOP_PREPARE);
> -	return stop_cpus(cpu_online_mask, multi_cpu_stop, &msdata);
> +	ret = stop_cpus(cpu_online_mask, multi_cpu_stop, &msdata);
> +
> +	if (!use_nmi)
> +		return ret;
> +
> +	if (!cpumask_equal(cpus, &msdata.nmi_cpus_done)) {
> +		pr_err("Some CPUs didn't run the stomp_machine NMI handler\n");
> +		return -EINVAL;
> +	} else {
> +		for_each_cpu(cpu, cpus)
> +			ret |= per_cpu(stop_machine_nmi_ctrl.err, cpu);

This error accumulation here makes sense to me though,

Currently, stop_machine() documents:

/*
  ...
  * Return: 0 if all invocations of @fn return zero.  Otherwise, the
  * value returned by an arbitrarily chosen member of the set of calls to
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  * @fn that returned non-zero.
  */

This behavior dates back to:

   commit 1142d810298e694 ("cpu_stop: implement stop_cpu[s]()")

where cpu_stopper_thread() does:

   struct cpu_stop_done *done = work->done;
   ...
   ret = fn(arg);
   if (ret)
   	done->ret = ret;

So the return value is overwritten instead of accumulation, while struct 
cpu_stop_done is shared like the note:

/*
  * Structure to determine completion condition and record errors.  May
  * be shared by works on different cpus.
  */

I don't know whether that was an intentional design choice or not. But, 
at least the NMI variant might have a slight different semantic in this 
regard.

Re: [PATCH 1/7] stop_machine: Introduce stop_machine_nmi()

[Borislav Petkov]

On Thu, Feb 05, 2026 at 06:14:39PM -0800, Chang S. Bae wrote:
> On 2/2/2026 2:54 AM, Borislav Petkov wrote:
> ...
> > @@ -174,8 +174,26 @@ struct multi_stop_data {
> >   	enum multi_stop_state	state;
> >   	atomic_t		thread_ack;
> > +
> > +	bool			use_nmi;
> > +
> > +	/*
> > +	 * cpumasks of CPUs on which to raise an NMI; used in the NMI
> > +	 * stomp_machine variant. nmi_cpus_done is used for tracking
> > +	 * when the NMI handler has executed successfully.
> > +	 */
> > +	struct cpumask		nmi_cpus;
> > +	struct cpumask		nmi_cpus_done;
> > +
> > +};
> 
> Looks like every stop_machine variant then will spend stack for these masks.
> It seems they could be cpumask_var_t.

I guess...

> Alternatively, to make it simple further, a per-CPU variable could achieve
> this if I understand correctly:
> 
>   struct stop_machine_nmi_ctrl {
> 	...
> 	bool	done;
>   }

The first mask - nmi_cpus guards from the NMI handler running again. The
second one checks whether all CPUs ran the NMI handler.

I guess simply checking whether the nmi_cpus mask is *not* empty, would tell
us that too so we probably are fine with a single mask only.

> I don't know whether that was an intentional design choice or not. But, at
> least the NMI variant might have a slight different semantic in this regard.

This current behavior doesn't make a whole lot of sense to me - at least from
what I'm reading. I think it is clearly better if the caller gets told when
some NMI handler failed instead of overwriting an error val.

But maybe we'll fix that while we're at it.

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH 1/7] stop_machine: Introduce stop_machine_nmi()

[Thomas Gleixner]

On Sun, Jan 25 2026 at 01:42, Chang S. Bae wrote:
> +/**
> + * stop_machine_nmi: freeze the machine and run this function in NMI context
> + * @fn: the function to run
> + * @data: the data ptr for the @fn()
> + * @cpus: the cpus to run the @fn() on (NULL = any online cpu)

Please format these tabular, use uppercase letters to start the
explanation, use CPU[s] all over the place and write out words instead
of using made up abbreviations. This is documentation not twitter.

 * @fn:		The function to invoke
 * @data:	The data pointer for @fn()
 * @cpus:	A cpumask containing the CPUs to run fn() on

Also this NULL == any online CPU is just made up. What's wrong with
cpu_online_mask?

> +
> +bool noinstr stop_machine_nmi_handler(void);
> +DECLARE_STATIC_KEY_FALSE(stop_machine_nmi_handler_enable);
> +static __always_inline bool stop_machine_nmi_handler_enabled(void)

Can you please separate the declarations from the inline with an empty
new line? This glued together way to write it is unreadable.

> +{
> +	return static_branch_unlikely(&stop_machine_nmi_handler_enable);
> +}
> +
>  #else	/* CONFIG_SMP || CONFIG_HOTPLUG_CPU */
>  
>  static __always_inline int stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
> @@ -186,5 +217,24 @@ stop_machine_from_inactive_cpu(cpu_stop_fn_t fn, void *data,
>  	return stop_machine(fn, data, cpus);
>  }
>  
> +/* stop_machine_nmi() is only supported in SMP systems. */
> +static __always_inline int stop_machine_nmi(cpu_stop_fn_t fn, void *data,
> +					const struct cpumask *cpus)

Align the second line argument with the first argument above.

See https://www.kernel.org/doc/html/latest/process/maintainer-tip.html

> +{
> +	return -EINVAL;
> +}
> +

> +
> +void arch_send_self_nmi(void);
>  #endif	/* _LINUX_STOP_MACHINE */
> diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c
> index 3fe6b0c99f3d..189b4b108d13 100644
> --- a/kernel/stop_machine.c
> +++ b/kernel/stop_machine.c
> @@ -174,6 +174,8 @@ struct multi_stop_data {
>  
>  	enum multi_stop_state	state;
>  	atomic_t		thread_ack;
> +
> +	bool			use_nmi;
>  };
>  
>  static void set_state(struct multi_stop_data *msdata,
> @@ -197,6 +199,42 @@ notrace void __weak stop_machine_yield(const struct cpumask *cpumask)
>  	cpu_relax();
>  }
>  
> +struct stop_machine_nmi_ctrl {
> +	bool nmi_enabled;
> +	struct multi_stop_data *msdata;
> +	int err;

Please align the struct member names tabular. See documentation.

> +};
> +
> +DEFINE_STATIC_KEY_FALSE(stop_machine_nmi_handler_enable);
> +static DEFINE_PER_CPU(struct stop_machine_nmi_ctrl, stop_machine_nmi_ctrl);
> +
> +static void enable_nmi_handler(struct multi_stop_data *msdata)
> +{
> +	this_cpu_write(stop_machine_nmi_ctrl.msdata, msdata);
> +	this_cpu_write(stop_machine_nmi_ctrl.nmi_enabled, true);
> +}
> +
> +void __weak arch_send_self_nmi(void)
> +{
> +	/* Arch code must implement this to support stop_machine_nmi() */

Architecture

> +}

Also this weak function is wrong.

All of this NMI mode needs to be guarded with a config option as it
otherwise is compiled in unconditionally and any accidental usage on an
architecture which does not support this will result in a undecodable
malfunction. There is a world outside of x86.

With that arch_send_self_nmi() becomes a plain declaration in a header.

> +
> +bool noinstr stop_machine_nmi_handler(void)
> +{
> +	struct multi_stop_data *msdata;
> +	int err;
> +
> +	if (!raw_cpu_read(stop_machine_nmi_ctrl.nmi_enabled))
> +		return false;
> +
> +	raw_cpu_write(stop_machine_nmi_ctrl.nmi_enabled, false);
> +
> +	msdata = raw_cpu_read(stop_machine_nmi_ctrl.msdata);
> +	err = msdata->fn(msdata->data);
> +	raw_cpu_write(stop_machine_nmi_ctrl.err, err);
> +	return true;
> +}
> +
>  /* This is the cpu_stop function which stops the CPU. */
>  static int multi_cpu_stop(void *data)
>  {
> @@ -234,8 +272,15 @@ static int multi_cpu_stop(void *data)
>  				hard_irq_disable();
>  				break;
>  			case MULTI_STOP_RUN:
> -				if (is_active)
> -					err = msdata->fn(msdata->data);
> +				if (is_active) {
> +					if (msdata->use_nmi) {
> +						enable_nmi_handler(msdata);
> +						arch_send_self_nmi();
> +						err = raw_cpu_read(stop_machine_nmi_ctrl.err);
> +					} else {
> +						err = msdata->fn(msdata->data);
> +					}

And this wants to become

    if (IS_ENABLED(CONFIG_STOMP_MACHINE_NMI) && msdata->use_nmi)
    	err = stop_this_cpu_nmi(msdata);
    else
	err = msdata->fn(msdata->data);

> +				}
>  				break;
>  			default:
>  				break;
> @@ -584,14 +629,15 @@ static int __init cpu_stop_init(void)
>  }
>  early_initcall(cpu_stop_init);
>  
> -int stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
> -			    const struct cpumask *cpus)
> +static int __stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
> +			    const struct cpumask *cpus, bool use_nmi)

The argument alignment was correct before....

>  {
>  	struct multi_stop_data msdata = {
>  		.fn = fn,
>  		.data = data,
>  		.num_threads = num_online_cpus(),
>  		.active_cpus = cpus,
> +		.use_nmi = use_nmi,
>  	};
>  
>  	lockdep_assert_cpus_held();
> @@ -620,6 +666,24 @@ int stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
>  	return stop_cpus(cpu_online_mask, multi_cpu_stop, &msdata);
>  }

> +int stop_machine_nmi(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus)
> +{
> +	int ret;
> +
> +	cpus_read_lock();
> +	ret = stop_machine_cpuslocked_nmi(fn, data, cpus);
> +	cpus_read_unlock();
> +
> +	return ret;
> +}
> +EXPORT_SYMBOL_GPL(stop_machine_nmi);

Why needs this to be exported? No module has any business with stomp
machine.

Thanks

        tglx

Re: [PATCH 1/7] stop_machine: Introduce stop_machine_nmi()

[Chang S. Bae]

On 1/28/2026 12:02 AM, Thomas Gleixner wrote:
> On Sun, Jan 25 2026 at 01:42, Chang S. Bae wrote:
>> +/**
>> + * stop_machine_nmi: freeze the machine and run this function in NMI context
>> + * @fn: the function to run
>> + * @data: the data ptr for the @fn()
>> + * @cpus: the cpus to run the @fn() on (NULL = any online cpu)
> 
> Please format these tabular, use uppercase letters to start the
> explanation, use CPU[s] all over the place and write out words instead
> of using made up abbreviations. This is documentation not twitter.
> 
>   * @fn:		The function to invoke
>   * @data:	The data pointer for @fn()
>   * @cpus:	A cpumask containing the CPUs to run fn() on
> 
> Also this NULL == any online CPU is just made up. What's wrong with
> cpu_online_mask?
> 
>> +
>> +bool noinstr stop_machine_nmi_handler(void);
>> +DECLARE_STATIC_KEY_FALSE(stop_machine_nmi_handler_enable);
>> +static __always_inline bool stop_machine_nmi_handler_enabled(void)
> 
> Can you please separate the declarations from the inline with an empty
> new line? This glued together way to write it is unreadable.

Yes, I fixed them all on my local right now.

> 
>> +{
>> +	return static_branch_unlikely(&stop_machine_nmi_handler_enable);
>> +}
>> +
>>   #else	/* CONFIG_SMP || CONFIG_HOTPLUG_CPU */
>>   
>>   static __always_inline int stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
>> @@ -186,5 +217,24 @@ stop_machine_from_inactive_cpu(cpu_stop_fn_t fn, void *data,
>>   	return stop_machine(fn, data, cpus);
>>   }
>>   
>> +/* stop_machine_nmi() is only supported in SMP systems. */
>> +static __always_inline int stop_machine_nmi(cpu_stop_fn_t fn, void *data,
>> +					const struct cpumask *cpus)
> 
> Align the second line argument with the first argument above.
> 
> See https://www.kernel.org/doc/html/latest/process/maintainer-tip.html

Sorry for lot of misalignment issues in this change that I missed out.

>> +{
>> +	return -EINVAL;
>> +}
>> +
> 
>> +
>> +void arch_send_self_nmi(void);
>>   #endif	/* _LINUX_STOP_MACHINE */
>> diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c
>> index 3fe6b0c99f3d..189b4b108d13 100644
>> --- a/kernel/stop_machine.c
>> +++ b/kernel/stop_machine.c
>> @@ -174,6 +174,8 @@ struct multi_stop_data {
>>   
>>   	enum multi_stop_state	state;
>>   	atomic_t		thread_ack;
>> +
>> +	bool			use_nmi;
>>   };
>>   
>>   static void set_state(struct multi_stop_data *msdata,
>> @@ -197,6 +199,42 @@ notrace void __weak stop_machine_yield(const struct cpumask *cpumask)
>>   	cpu_relax();
>>   }
>>   
>> +struct stop_machine_nmi_ctrl {
>> +	bool nmi_enabled;
>> +	struct multi_stop_data *msdata;
>> +	int err;
> 
> Please align the struct member names tabular. See documentation.

Fixed.

> 
>> +};
>> +
>> +DEFINE_STATIC_KEY_FALSE(stop_machine_nmi_handler_enable);
>> +static DEFINE_PER_CPU(struct stop_machine_nmi_ctrl, stop_machine_nmi_ctrl);
>> +
>> +static void enable_nmi_handler(struct multi_stop_data *msdata)
>> +{
>> +	this_cpu_write(stop_machine_nmi_ctrl.msdata, msdata);
>> +	this_cpu_write(stop_machine_nmi_ctrl.nmi_enabled, true);
>> +}
>> +
>> +void __weak arch_send_self_nmi(void)
>> +{
>> +	/* Arch code must implement this to support stop_machine_nmi() */
> 
> Architecture

Fixed.

> 
>> +}
> 
> Also this weak function is wrong.
> 
> All of this NMI mode needs to be guarded with a config option as it
> otherwise is compiled in unconditionally and any accidental usage on an
> architecture which does not support this will result in a undecodable
> malfunction. There is a world outside of x86.
> 
> With that arch_send_self_nmi() becomes a plain declaration in a header.

I see.

> 
>> +
>> +bool noinstr stop_machine_nmi_handler(void)
>> +{
>> +	struct multi_stop_data *msdata;
>> +	int err;
>> +
>> +	if (!raw_cpu_read(stop_machine_nmi_ctrl.nmi_enabled))
>> +		return false;
>> +
>> +	raw_cpu_write(stop_machine_nmi_ctrl.nmi_enabled, false);
>> +
>> +	msdata = raw_cpu_read(stop_machine_nmi_ctrl.msdata);
>> +	err = msdata->fn(msdata->data);
>> +	raw_cpu_write(stop_machine_nmi_ctrl.err, err);
>> +	return true;
>> +}
>> +
>>   /* This is the cpu_stop function which stops the CPU. */
>>   static int multi_cpu_stop(void *data)
>>   {
>> @@ -234,8 +272,15 @@ static int multi_cpu_stop(void *data)
>>   				hard_irq_disable();
>>   				break;
>>   			case MULTI_STOP_RUN:
>> -				if (is_active)
>> -					err = msdata->fn(msdata->data);
>> +				if (is_active) {
>> +					if (msdata->use_nmi) {
>> +						enable_nmi_handler(msdata);
>> +						arch_send_self_nmi();
>> +						err = raw_cpu_read(stop_machine_nmi_ctrl.err);
>> +					} else {
>> +						err = msdata->fn(msdata->data);
>> +					}
> 
> And this wants to become
> 
>      if (IS_ENABLED(CONFIG_STOMP_MACHINE_NMI) && msdata->use_nmi)
>      	err = stop_this_cpu_nmi(msdata);
>      else
> 	err = msdata->fn(msdata->data);

Although that config option is very clear and makes tons of sense, the 
latter reads like a (silent) fallback path for a stop_machine_nmi() 
invocation with CONFIG_STOMP_MACHINE_NMI=n.

Maybe this might be clear to reject the NMI option right away with 
something like:

stop_machine_cpuslocked_nmi(...)
{
	if (!IS_ENABLED(CONFIG_STOMP_MACHINE_NMI))
		return -EOPNOTSUPP;
	...
};

>>   
>> -int stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
>> -			    const struct cpumask *cpus)
>> +static int __stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
>> +			    const struct cpumask *cpus, bool use_nmi)
> 
> The argument alignment was correct before....

Sigh... fixed, again.

>> +int stop_machine_nmi(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus)
>> +{
>> +	int ret;
>> +
>> +	cpus_read_lock();
>> +	ret = stop_machine_cpuslocked_nmi(fn, data, cpus);
>> +	cpus_read_unlock();
>> +
>> +	return ret;
>> +}
>> +EXPORT_SYMBOL_GPL(stop_machine_nmi);
> 
> Why needs this to be exported? No module has any business with stomp
> machine.

Not at all. Removed.

I really appreciate your time and effort for the review!

Re: [PATCH 1/7] stop_machine: Introduce stop_machine_nmi()

[Thomas Gleixner]

On Thu, Jan 29 2026 at 09:07, Chang S. Bae wrote:
> On 1/28/2026 12:02 AM, Thomas Gleixner wrote:
>> And this wants to become
>> 
>>      if (IS_ENABLED(CONFIG_STOMP_MACHINE_NMI) && msdata->use_nmi)
>>      	err = stop_this_cpu_nmi(msdata);
>>      else
>> 	err = msdata->fn(msdata->data);
>
> Although that config option is very clear and makes tons of sense, the 
> latter reads like a (silent) fallback path for a stop_machine_nmi() 
> invocation with CONFIG_STOMP_MACHINE_NMI=n.
>
> Maybe this might be clear to reject the NMI option right away with 
> something like:
>
> stop_machine_cpuslocked_nmi(...)
> {
> 	if (!IS_ENABLED(CONFIG_STOMP_MACHINE_NMI))
> 		return -EOPNOTSUPP;
> 	...
> };

That function should not be exposed at all when the config switch is
off. Hide it behind #ifdef CONFIG...

It really should not be used in generic code at all.

Thanks,

        tglx

[PATCH 2/7] x86/apic: Implement self-NMI support

[Chang S. Bae]

From: David Kaplan <david.kaplan@amd.com>

Add function to send an NMI-to-self which is needed for stop_machine_nmi().
Note that send_IPI_self() cannot be used to send an NMI so send_IPI() is
used.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com>
---
Updates from the original version:
* Rename the subject to highlight arch-side "implementation"
---
 arch/x86/kernel/apic/ipi.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/arch/x86/kernel/apic/ipi.c b/arch/x86/kernel/apic/ipi.c
index 98a57cb4aa86..6d4e5aa27529 100644
--- a/arch/x86/kernel/apic/ipi.c
+++ b/arch/x86/kernel/apic/ipi.c
@@ -4,6 +4,7 @@
 #include <linux/delay.h>
 #include <linux/smp.h>
 #include <linux/string_choices.h>
+#include <linux/stop_machine.h>
 
 #include <asm/io_apic.h>
 
@@ -248,6 +249,12 @@ void default_send_IPI_self(int vector)
 	__default_send_IPI_shortcut(APIC_DEST_SELF, vector);
 }
 
+/* Self-NMI is used in stop_machine_nmi() */
+void arch_send_self_nmi(void)
+{
+	apic->send_IPI(smp_processor_id(), NMI_VECTOR);
+}
+
 #ifdef CONFIG_X86_32
 void default_send_IPI_mask_sequence_logical(const struct cpumask *mask, int vector)
 {
-- 
2.51.0

Re: [PATCH 2/7] x86/apic: Implement self-NMI support

[Thomas Gleixner]

On Sun, Jan 25 2026 at 01:42, Chang S. Bae wrote:
> From: David Kaplan <david.kaplan@amd.com>
>
> Add function to send an NMI-to-self which is needed for stop_machine_nmi().

Add _a_ function... It's not that hard to write proper sentences.

Re: [PATCH 2/7] x86/apic: Implement self-NMI support

[Chang S. Bae]

On 1/28/2026 12:05 AM, Thomas Gleixner wrote:
> On Sun, Jan 25 2026 at 01:42, Chang S. Bae wrote:
>> From: David Kaplan <david.kaplan@amd.com>
>>
>> Add function to send an NMI-to-self which is needed for stop_machine_nmi().
> 
> Add _a_ function... It's not that hard to write proper sentences.

My apologies. I should have caught the typo.

[PATCH 3/7] x86/nmi: Support stop_machine_nmi() handler

[Chang S. Bae]

From: David Kaplan <david.kaplan@amd.com>

Call stop_machine_nmi_handler() from the NMI path when the corresponding
static key is enabled, in order to support stop_machine_nmi().

Signed-off-by: David Kaplan <david.kaplan@amd.com>
Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com>
---
Update from the original posting:
* Massage the subject and expand changelog a little bit.
---
 arch/x86/kernel/nmi.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/arch/x86/kernel/nmi.c b/arch/x86/kernel/nmi.c
index 3d239ed12744..4bc4b49f1ea7 100644
--- a/arch/x86/kernel/nmi.c
+++ b/arch/x86/kernel/nmi.c
@@ -24,6 +24,7 @@
 #include <linux/export.h>
 #include <linux/atomic.h>
 #include <linux/sched/clock.h>
+#include <linux/stop_machine.h>
 #include <linux/kvm_types.h>
 
 #include <asm/cpu_entry_area.h>
@@ -382,6 +383,9 @@ static noinstr void default_do_nmi(struct pt_regs *regs)
 
 	instrumentation_begin();
 
+	if (stop_machine_nmi_handler_enabled() && stop_machine_nmi_handler())
+		goto out;
+
 	if (microcode_nmi_handler_enabled() && microcode_nmi_handler())
 		goto out;
 
-- 
2.51.0

[PATCH 4/7] x86/microcode: Distinguish NMI control path on stop-machine callback

[Chang S. Bae]

load_cpus_stopped() currently centralizes the stop_machine() callback for
both NMI and NMI-less rendezvous. microcode_update_handler() alone is
enough for the latter.

While the NMI-based rendezvous finally reaches the same update handler,
it requires additional logic to trigger and process NMIs. That machinery
will be replaced by stop_machine_nmi().

As preparation for that conversion, split the callback path to make
NMI-specific steps explicit and clear.

Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com>
---
 arch/x86/kernel/cpu/microcode/core.c | 28 ++++++++++++----------------
 1 file changed, 12 insertions(+), 16 deletions(-)

diff --git a/arch/x86/kernel/cpu/microcode/core.c b/arch/x86/kernel/cpu/microcode/core.c
index 68049f171860..28317176ae29 100644
--- a/arch/x86/kernel/cpu/microcode/core.c
+++ b/arch/x86/kernel/cpu/microcode/core.c
@@ -524,7 +524,7 @@ void noinstr microcode_offline_nmi_handler(void)
 	wait_for_ctrl();
 }
 
-static noinstr bool microcode_update_handler(void)
+static noinstr int microcode_update_handler(void *unused)
 {
 	unsigned int cpu = raw_smp_processor_id();
 
@@ -540,7 +540,7 @@ static noinstr bool microcode_update_handler(void)
 	touch_nmi_watchdog();
 	instrumentation_end();
 
-	return true;
+	return 0;
 }
 
 /*
@@ -561,19 +561,15 @@ bool noinstr microcode_nmi_handler(void)
 		return false;
 
 	raw_cpu_write(ucode_ctrl.nmi_enabled, false);
-	return microcode_update_handler();
+	return microcode_update_handler(NULL) == 0;
 }
 
 static int load_cpus_stopped(void *unused)
 {
-	if (microcode_ops->use_nmi) {
-		/* Enable the NMI handler and raise NMI */
-		this_cpu_write(ucode_ctrl.nmi_enabled, true);
-		apic->send_IPI(smp_processor_id(), NMI_VECTOR);
-	} else {
-		/* Just invoke the handler directly */
-		microcode_update_handler();
-	}
+	/* Enable the NMI handler and raise NMI */
+	this_cpu_write(ucode_ctrl.nmi_enabled, true);
+	apic->send_IPI(smp_processor_id(), NMI_VECTOR);
+
 	return 0;
 }
 
@@ -610,13 +606,13 @@ static int load_late_stop_cpus(bool is_safe)
 	 */
 	store_cpu_caps(&prev_info);
 
-	if (microcode_ops->use_nmi)
+	if (microcode_ops->use_nmi) {
 		static_branch_enable_cpuslocked(&microcode_nmi_handler_enable);
-
-	stop_machine_cpuslocked(load_cpus_stopped, NULL, cpu_online_mask);
-
-	if (microcode_ops->use_nmi)
+		stop_machine_cpuslocked(load_cpus_stopped, NULL, cpu_online_mask);
 		static_branch_disable_cpuslocked(&microcode_nmi_handler_enable);
+	} else {
+		stop_machine_cpuslocked(microcode_update_handler, NULL, cpu_online_mask);
+	}
 
 	/* Analyze the results */
 	for_each_cpu_and(cpu, cpu_present_mask, &cpus_booted_once_mask) {
-- 
2.51.0

Re: [PATCH 4/7] x86/microcode: Distinguish NMI control path on stop-machine callback

[Thomas Gleixner]

On Sun, Jan 25 2026 at 01:42, Chang S. Bae wrote:
>  static int load_cpus_stopped(void *unused)
>  {
> -	if (microcode_ops->use_nmi) {
> -		/* Enable the NMI handler and raise NMI */
> -		this_cpu_write(ucode_ctrl.nmi_enabled, true);
> -		apic->send_IPI(smp_processor_id(), NMI_VECTOR);
> -	} else {
> -		/* Just invoke the handler directly */
> -		microcode_update_handler();
> -	}
> +	/* Enable the NMI handler and raise NMI */
> +	this_cpu_write(ucode_ctrl.nmi_enabled, true);
> +	apic->send_IPI(smp_processor_id(), NMI_VECTOR);
> +

With this change the function name is completely bogus.

Re: [PATCH 4/7] x86/microcode: Distinguish NMI control path on stop-machine callback

[Chang S. Bae]

On 1/28/2026 12:11 AM, Thomas Gleixner wrote:
> On Sun, Jan 25 2026 at 01:42, Chang S. Bae wrote:
>>   static int load_cpus_stopped(void *unused)
>>   {
>> -	if (microcode_ops->use_nmi) {
>> -		/* Enable the NMI handler and raise NMI */
>> -		this_cpu_write(ucode_ctrl.nmi_enabled, true);
>> -		apic->send_IPI(smp_processor_id(), NMI_VECTOR);
>> -	} else {
>> -		/* Just invoke the handler directly */
>> -		microcode_update_handler();
>> -	}
>> +	/* Enable the NMI handler and raise NMI */
>> +	this_cpu_write(ucode_ctrl.nmi_enabled, true);
>> +	apic->send_IPI(smp_processor_id(), NMI_VECTOR);
>> +
> 
> With this change the function name is completely bogus.

Yes, you're right. The change should stand in a sensible shape if the 
series were to stop here. I think stop_this_cpu_nmi() that you mentioned 
on patch1 sounds aligned, or maybe stop_cpu_in_nmi() like that.

[PATCH 5/7] x86/microcode: Use stop-machine NMI facility

[Chang S. Bae]

The existing NMI-based loading logic explicitly sends NMIs to online CPUs
and invokes microcode_update_handler() from the NMI context.

The stop-machine NMI variant already provides the mechanism on x86.
Replace the custom NMI-based mechanism with the common facility to
simplify the loader code.

Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com>
---
 arch/x86/include/asm/microcode.h     |  1 -
 arch/x86/kernel/cpu/microcode/core.c | 19 +++----------------
 arch/x86/kernel/nmi.c                |  3 ---
 3 files changed, 3 insertions(+), 20 deletions(-)

diff --git a/arch/x86/include/asm/microcode.h b/arch/x86/include/asm/microcode.h
index 8b41f26f003b..946b0cebed87 100644
--- a/arch/x86/include/asm/microcode.h
+++ b/arch/x86/include/asm/microcode.h
@@ -77,7 +77,6 @@ static inline u32 intel_get_microcode_revision(void)
 }
 #endif /* !CONFIG_CPU_SUP_INTEL */
 
-bool microcode_nmi_handler(void);
 void microcode_offline_nmi_handler(void);
 
 #ifdef CONFIG_MICROCODE_LATE_LOADING
diff --git a/arch/x86/kernel/cpu/microcode/core.c b/arch/x86/kernel/cpu/microcode/core.c
index 28317176ae29..0facadea39d6 100644
--- a/arch/x86/kernel/cpu/microcode/core.c
+++ b/arch/x86/kernel/cpu/microcode/core.c
@@ -555,22 +555,9 @@ static noinstr int microcode_update_handler(void *unused)
  * path which must be NMI safe until the primary thread completed the
  * update.
  */
-bool noinstr microcode_nmi_handler(void)
+static noinstr int microcode_nmi_handler(void *data)
 {
-	if (!raw_cpu_read(ucode_ctrl.nmi_enabled))
-		return false;
-
-	raw_cpu_write(ucode_ctrl.nmi_enabled, false);
-	return microcode_update_handler(NULL) == 0;
-}
-
-static int load_cpus_stopped(void *unused)
-{
-	/* Enable the NMI handler and raise NMI */
-	this_cpu_write(ucode_ctrl.nmi_enabled, true);
-	apic->send_IPI(smp_processor_id(), NMI_VECTOR);
-
-	return 0;
+	return microcode_update_handler(data);
 }
 
 static int load_late_stop_cpus(bool is_safe)
@@ -608,7 +595,7 @@ static int load_late_stop_cpus(bool is_safe)
 
 	if (microcode_ops->use_nmi) {
 		static_branch_enable_cpuslocked(&microcode_nmi_handler_enable);
-		stop_machine_cpuslocked(load_cpus_stopped, NULL, cpu_online_mask);
+		stop_machine_cpuslocked_nmi(microcode_nmi_handler, NULL, cpu_online_mask);
 		static_branch_disable_cpuslocked(&microcode_nmi_handler_enable);
 	} else {
 		stop_machine_cpuslocked(microcode_update_handler, NULL, cpu_online_mask);
diff --git a/arch/x86/kernel/nmi.c b/arch/x86/kernel/nmi.c
index 4bc4b49f1ea7..642469f3be80 100644
--- a/arch/x86/kernel/nmi.c
+++ b/arch/x86/kernel/nmi.c
@@ -386,9 +386,6 @@ static noinstr void default_do_nmi(struct pt_regs *regs)
 	if (stop_machine_nmi_handler_enabled() && stop_machine_nmi_handler())
 		goto out;
 
-	if (microcode_nmi_handler_enabled() && microcode_nmi_handler())
-		goto out;
-
 	/*
 	 * CPU-specific NMI must be processed before non-CPU-specific
 	 * NMI, otherwise we may lose it, because the CPU-specific
-- 
2.51.0

[PATCH 6/7] x86/nmi: Reference stop-machine static key for offline microcode handler

[Chang S. Bae]

The microcode loader now uses stop-machine facility for NMI-based
loading. During the rendezvous, the stop-machine static key is enabled
and can be queried via stop_machine_nmi_handler_enabled(). Reference this
on the offline handling path.

Previously, the microcode-specific static key helper returned false when
CONFIG_MICROCODE_LATE_LOADING=n. Since the offline handler is also built
only with that option, the helper effectively acted as a guard to avoid
referencing the handler symbol when it was not present. With the removal,
to avoid build issues, introduce a one-line helper.

Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com>
---
Note on an alternative option:
Rather than keeping this loader-specific handler, stop-machine facility
could be extended further to call-back to handle offline cases. Then, the
microcode loader however will be the only user of that. Instead, just
switching to check the stop-machine helper looks to be simple enough.
---
 arch/x86/include/asm/microcode.h | 5 +++--
 arch/x86/kernel/nmi.c            | 4 ++--
 2 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/arch/x86/include/asm/microcode.h b/arch/x86/include/asm/microcode.h
index 946b0cebed87..a4e4ea8acf99 100644
--- a/arch/x86/include/asm/microcode.h
+++ b/arch/x86/include/asm/microcode.h
@@ -77,15 +77,16 @@ static inline u32 intel_get_microcode_revision(void)
 }
 #endif /* !CONFIG_CPU_SUP_INTEL */
 
-void microcode_offline_nmi_handler(void);
-
 #ifdef CONFIG_MICROCODE_LATE_LOADING
+void microcode_offline_nmi_handler(void);
+
 DECLARE_STATIC_KEY_FALSE(microcode_nmi_handler_enable);
 static __always_inline bool microcode_nmi_handler_enabled(void)
 {
 	return static_branch_unlikely(&microcode_nmi_handler_enable);
 }
 #else
+static __always_inline void microcode_offline_nmi_handler(void) { }
 static __always_inline bool microcode_nmi_handler_enabled(void) { return false; }
 #endif
 
diff --git a/arch/x86/kernel/nmi.c b/arch/x86/kernel/nmi.c
index 642469f3be80..52bc844354de 100644
--- a/arch/x86/kernel/nmi.c
+++ b/arch/x86/kernel/nmi.c
@@ -548,7 +548,7 @@ DEFINE_IDTENTRY_RAW(exc_nmi)
 		raw_atomic_long_inc(&nsp->idt_calls);
 
 	if (arch_cpu_is_offline(smp_processor_id())) {
-		if (microcode_nmi_handler_enabled())
+		if (stop_machine_nmi_handler_enabled())
 			microcode_offline_nmi_handler();
 		return;
 	}
@@ -711,7 +711,7 @@ DEFINE_FREDENTRY_NMI(exc_nmi)
 	irqentry_state_t irq_state;
 
 	if (arch_cpu_is_offline(smp_processor_id())) {
-		if (microcode_nmi_handler_enabled())
+		if (stop_machine_nmi_handler_enabled())
 			microcode_offline_nmi_handler();
 		return;
 	}
-- 
2.51.0

[PATCH 7/7] x86/microcode: Remove microcode_nmi_handler_enable

[Chang S. Bae]

Remove it, as there is no more user.

Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com>
---
 arch/x86/include/asm/microcode.h     | 7 -------
 arch/x86/kernel/cpu/microcode/core.c | 8 ++------
 2 files changed, 2 insertions(+), 13 deletions(-)

diff --git a/arch/x86/include/asm/microcode.h b/arch/x86/include/asm/microcode.h
index a4e4ea8acf99..b7e89fc45500 100644
--- a/arch/x86/include/asm/microcode.h
+++ b/arch/x86/include/asm/microcode.h
@@ -79,15 +79,8 @@ static inline u32 intel_get_microcode_revision(void)
 
 #ifdef CONFIG_MICROCODE_LATE_LOADING
 void microcode_offline_nmi_handler(void);
-
-DECLARE_STATIC_KEY_FALSE(microcode_nmi_handler_enable);
-static __always_inline bool microcode_nmi_handler_enabled(void)
-{
-	return static_branch_unlikely(&microcode_nmi_handler_enable);
-}
 #else
 static __always_inline void microcode_offline_nmi_handler(void) { }
-static __always_inline bool microcode_nmi_handler_enabled(void) { return false; }
 #endif
 
 #endif /* _ASM_X86_MICROCODE_H */
diff --git a/arch/x86/kernel/cpu/microcode/core.c b/arch/x86/kernel/cpu/microcode/core.c
index 0facadea39d6..35b1b5d88893 100644
--- a/arch/x86/kernel/cpu/microcode/core.c
+++ b/arch/x86/kernel/cpu/microcode/core.c
@@ -316,7 +316,6 @@ struct microcode_ctrl {
 	bool			nmi_enabled;
 };
 
-DEFINE_STATIC_KEY_FALSE(microcode_nmi_handler_enable);
 static DEFINE_PER_CPU(struct microcode_ctrl, ucode_ctrl);
 static atomic_t late_cpus_in, offline_in_nmi;
 static unsigned int loops_per_usec;
@@ -593,13 +592,10 @@ static int load_late_stop_cpus(bool is_safe)
 	 */
 	store_cpu_caps(&prev_info);
 
-	if (microcode_ops->use_nmi) {
-		static_branch_enable_cpuslocked(&microcode_nmi_handler_enable);
+	if (microcode_ops->use_nmi)
 		stop_machine_cpuslocked_nmi(microcode_nmi_handler, NULL, cpu_online_mask);
-		static_branch_disable_cpuslocked(&microcode_nmi_handler_enable);
-	} else {
+	else
 		stop_machine_cpuslocked(microcode_update_handler, NULL, cpu_online_mask);
-	}
 
 	/* Analyze the results */
 	for_each_cpu_and(cpu, cpu_present_mask, &cpus_booted_once_mask) {
-- 
2.51.0