[RFC PATCH 0/2] arm64/hyperv: Enable kexec reboot support

[RFC PATCH 0/2] arm64/hyperv: Enable kexec reboot support

[Shradha Gupta]

Kexec reboot consistently fails on ARM64 Hyper-V guests (Azure VMs).
During the kexec shutdown path, VMBus channels are never cleaned up,
so in the fresh kexec kernel startup hv_acpi_init() blocks because
the hypervisor still holds the old kernel's VMBus session open.

This is because ARM64 lacks the VMBus teardown that x86 performs during
kexec via hv_machine_shutdown(). On x86, machine_ops.shutdown is
overridden to send CHANNELMSG_UNLOAD and disable SynIC before CPUs
go offline. ARM64 has no equivalent mechanism.

Why existing notification mechanisms don't work:

- Reboot notifiers: Fire too early - they run inside
  kernel_restart_prepare(), BEFORE device_shutdown(). VMBus UNLOAD
  must happen AFTER device_shutdown() so that PCI drivers can clean
  up interrupt mappings (PCI_DELETE_INTERRUPT_MESSAGE) before UNLOAD
  force-closes channels. Sending UNLOAD first causes interrupt
  mapping leaks because the mappings aren't released on channel
  close.

- Device .shutdown callbacks: Too narrow - these handle per-device
  cleanup (e.g., individual VMBus channel teardown), but cannot
  perform bus-level operations like sending the global VMBus UNLOAD
  message or removing the SynIC CPU hotplug state via
  cpuhp_remove_state().

- VMBus parent device .shutdown callback: Also insufficient -
  cpuhp_remove_state() for SynIC teardown must run after all device
  shutdown completes, not during it. A parent .shutdown callback
  still executes within device_shutdown().

- The required window is: after device_shutdown() completes, after
  cpu_hotplug_enable(), but before smp_shutdown_nonboot_cpus(). That
  window exists inside machine_shutdown(), which currently has no hook.

The need for an ARM64 shutdown hook was previously discussed in [1]
but lacked a concrete failure case at the time. We now have one.

[1] https://lore.kernel.org/linux-arm-kernel/427a8277-49f0-4317-d6c3-4a15d7070e55@igalia.com/

This RFC proposes fixing kexec on ARM64 Hyper-V guests with:

Patch 1: A platform hook (arm64_pre_smp_shutdown_hook) in ARM64's
  machine_shutdown(), analogous to x86's machine_ops.shutdown. This
  runs after device_shutdown() and cpu_hotplug_enable(), allowing
  platform code to inject pre-shutdown logic at the right point in
  the kexec path.

  Design choices I'd like feedback on:
  - Single function pointer vs full machine_ops struct: ARM64 uses
    kernel-wide APIs (register_restart_handler, register_platform_power_off)
    for restart/poweroff rather than x86's monolithic machine_ops, so
    a targeted hook seemed more consistent with the ARM64 pattern.
  - On ARM64, machine_shutdown() is only called from kernel_kexec(),
    unlike x86/powerpc where it's also called from restart/halt/poweroff.

Patch 2: Uses this hook to call hv_kexec_handler() which performs:
  - vmbus_initiate_unload(): sends CHANNELMSG_UNLOAD to host
  - cpuhp_remove_state(): disables SynIC (SIMP, SIEFP, SINT) on
    all CPUs, ensuring the kexec'd kernel starts with clean state

Tested on ARM64 Azure VMs (Ubuntu 22.04, multiple vCPU configs):
- kexec reboot succeeds, VM comes back online with SSH
- Normal reboot/poweroff unaffected
- Multiple consecutive kexec cycles pass

Looking for feedback on:
1. Is the single function pointer hook acceptable for ARM64, or would
   the ARM64 maintainers prefer a registration API or __weak function?
2. Should this be arm64-specific or generic kexec infrastructure?

Shradha Gupta (2):
  arm64: Add pre-shutdown hook to machine_shutdown()
  arm64/hyperv: Add kexec handler using machine_shutdown hook

 arch/arm64/hyperv/mshyperv.c         | 37 ++++++++++++++++++++++++++++
 arch/arm64/include/asm/system_misc.h |  2 ++
 arch/arm64/kernel/process.c          | 13 ++++++++++
 3 files changed, 52 insertions(+)

-- 
2.43.0

[RFC PATCH 1/2] arm64: Add pre-shutdown hook to machine_shutdown()

[Shradha Gupta]

Add a function pointer hook (arm64_pre_smp_shutdown_hook) that is invoked
from machine_shutdown() before smp_shutdown_nonboot_cpus(). This allows
platform code (e.g., hypervisors) to perform cleanup that must happen
after device_shutdown() but before secondary CPUs go offline.

In the kexec path, the call sequence is:
  kernel_kexec()
    kernel_restart_prepare()
      device_shutdown()         // drivers shut down here
    migrate_to_reboot_cpu()
    cpu_hotplug_enable()
    machine_shutdown()
      arm64_pre_smp_shutdown_hook()  // new: platform cleanup
      smp_shutdown_nonboot_cpus()    // CPUs go offline

x86 achieves this via machine_ops.shutdown; ARM64 currently has no
equivalent mechanism. Rather than introducing the full machine_ops
structure, add a targeted hook for the shutdown path.

Signed-off-by: Shradha Gupta <shradhagupta@linux.microsoft.com>
---
 arch/arm64/include/asm/system_misc.h |  2 ++
 arch/arm64/kernel/process.c          | 13 +++++++++++++
 2 files changed, 15 insertions(+)

diff --git a/arch/arm64/include/asm/system_misc.h b/arch/arm64/include/asm/system_misc.h
index d316a804eb38..7c37b9f33e96 100644
--- a/arch/arm64/include/asm/system_misc.h
+++ b/arch/arm64/include/asm/system_misc.h
@@ -28,6 +28,8 @@ void arm64_notify_die(const char *str, struct pt_regs *regs,
 struct mm_struct;
 extern void __show_regs(struct pt_regs *);
 
+extern void (*arm64_pre_smp_shutdown_hook)(void);
+
 #endif	/* __ASSEMBLER__ */
 
 #endif	/* __ASM_SYSTEM_MISC_H */
diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
index 581f80e9b9b7..41c0e9840913 100644
--- a/arch/arm64/kernel/process.c
+++ b/arch/arm64/kernel/process.c
@@ -88,8 +88,21 @@ void __noreturn arch_cpu_idle_dead(void)
  * avoid any code or data used by any SW CPU pin loop. The CPU hotplug
  * functionality embodied in smpt_shutdown_nonboot_cpus() to achieve this.
  */
+
+/*
+ * Hook for platform code to perform cleanup after device_shutdown()
+ * but before secondary CPUs are offlined. This runs in the kexec path
+ * from kernel_kexec() after device_shutdown() and cpu_hotplug_enable()
+ * have been called, matching the point at which x86 invokes
+ * machine_ops.shutdown.
+ */
+void (*arm64_pre_smp_shutdown_hook)(void);
+
 void machine_shutdown(void)
 {
+	if (arm64_pre_smp_shutdown_hook)
+		arm64_pre_smp_shutdown_hook();
+
 	smp_shutdown_nonboot_cpus(reboot_cpu);
 }
 
-- 
2.43.0

Re: [RFC PATCH 1/2] arm64: Add pre-shutdown hook to machine_shutdown()

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The newly introduced global function pointer `arm64_pre_smp_shutdown_hook` is placed in writable memory without the `__ro_after_init` annotation, creating a potential target for control-flow hijacking.
--

--- Patch [1]: [RFC PATCH 1/2] arm64: Add pre-shutdown hook to machine_shutdown() ---
commit 438763f14354dbd91b5fa7bea807c334ada0c642
Author: Shradha Gupta <shradhagupta@linux.microsoft.com>

arm64: Add pre-shutdown hook to machine_shutdown()

This patch introduces a global function pointer invoked from
machine_shutdown() to allow platform-specific cleanup before secondary CPUs
go offline during kexec or reboot.

[ ... ]

> diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
> index 581f80e9b9b73..41c0e98409138 100644
> --- a/arch/arm64/kernel/process.c
> +++ b/arch/arm64/kernel/process.c
> @@ -88,8 +88,21 @@ void __noreturn arch_cpu_idle_dead(void)

[ ... ]

> +/*
> + * Hook for platform code to perform cleanup after device_shutdown()
> + * but before secondary CPUs are offlined. This runs in the kexec path
> + * from kernel_kexec() after device_shutdown() and cpu_hotplug_enable()
> + * have been called, matching the point at which x86 invokes
> + * machine_ops.shutdown.
> + */
> +void (*arm64_pre_smp_shutdown_hook)(void);

[Severity: Medium]
Should this global function pointer be annotated with __ro_after_init?

Without it, the pointer resides in writable memory, which could make it a
reliable target for control-flow hijacking if an arbitrary memory write
primitive is obtained elsewhere. Since the hook is presumably only set once
during platform initialization, making it read-only after init would improve
defense-in-depth on the shutdown/reboot path.

> +
>  void machine_shutdown(void)
>  {
> +	if (arm64_pre_smp_shutdown_hook)
> +		arm64_pre_smp_shutdown_hook();
> +
>  	smp_shutdown_nonboot_cpus(reboot_cpu);
>  }

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260814093133.3191250-1-shradhagupta@linux.microsoft.com?part=1

[RFC PATCH 2/2] arm64/hyperv: Add kexec handler using machine_shutdown hook

[Shradha Gupta]

ARM64 Hyper-V guests currently lack the VMBus teardown that x86 performs
during kexec via hv_machine_shutdown(). Without this, kexec fails because:

1. The kexec'd kernel's vmbus_connect() hangs because the hypervisor
   still has the old VMBus connection active.
2. Stale SynIC state (message pending flags, event flags) causes
   message delivery failures or NULL pointer dereferences in the
   kexec'd kernel.

The VMBus driver already provides a kexec cleanup callback via the
existing hv_setup_kexec_handler() / hv_remove_kexec_handler() API
(drivers/hv/vmbus_drv.c). These call into arch-specific overrides;
on x86, the override is in mshyperv.c, while ARM64 currently uses
the __weak no-op stubs from hv_common.c.

This patch provides the ARM64 override of hv_setup_kexec_handler()
and hv_remove_kexec_handler(), wiring the registered hv_kexec_handler
to the new arm64_pre_smp_shutdown_hook so that it is invoked during
machine_shutdown(). This matches x86's ordering: the handler runs
AFTER device_shutdown() (so PCI drivers can clean up interrupt
mappings via PCI_DELETE_INTERRUPT_MESSAGE before VMBus UNLOAD
force-closes channels) and AFTER cpu_hotplug_enable() (so
cpuhp_remove_state() can disable SynIC on all CPUs).

hv_kexec_handler() (registered by vmbus_drv.c via
hv_setup_kexec_handler()) performs:
  - vmbus_initiate_unload(false): sends CHANNELMSG_UNLOAD to host
  - cpuhp_remove_state(): disables SynIC (SIMP, SIEFP, SINT) on all
    CPUs, ensuring the kexec'd kernel starts with clean SynIC state

Signed-off-by: Shradha Gupta <shradhagupta@linux.microsoft.com>
---
 arch/arm64/hyperv/mshyperv.c | 37 ++++++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)

diff --git a/arch/arm64/hyperv/mshyperv.c b/arch/arm64/hyperv/mshyperv.c
index 4fdc26ade1d7..4176175060d3 100644
--- a/arch/arm64/hyperv/mshyperv.c
+++ b/arch/arm64/hyperv/mshyperv.c
@@ -15,10 +15,47 @@
 #include <linux/errno.h>
 #include <linux/version.h>
 #include <linux/cpuhotplug.h>
+#include <linux/kexec.h>
 #include <asm/mshyperv.h>
+#include <asm/system_misc.h>
 
 static bool hyperv_initialized;
 
+/*
+ * Kexec/shutdown handler for ARM64 Hyper-V guests.
+ *
+ * On x86, hv_machine_shutdown() overrides machine_ops.shutdown and
+ * runs after device_shutdown() and cpu_hotplug_enable(). ARM64 uses
+ * the arm64_pre_smp_shutdown_hook to achieve the same ordering.
+ *
+ * hv_kexec_handler() (set by vmbus_drv.c) performs:
+ *   1. vmbus_initiate_unload(false) - sends CHANNELMSG_UNLOAD
+ *   2. cpuhp_remove_state(hyperv_cpuhp_online) - disables SynIC per CPU
+ *
+ * By running after device_shutdown(), PCI drivers (NVMe, MANA) can
+ * send PCI_DELETE_INTERRUPT_MESSAGE and clean up MMIO/interrupt
+ * mappings before VMBus channels are force-closed by UNLOAD.
+ */
+static void (*hv_kexec_handler)(void);
+
+static void hv_machine_shutdown(void)
+{
+	if (kexec_in_progress && hv_kexec_handler)
+		hv_kexec_handler();
+}
+
+void hv_setup_kexec_handler(void (*handler)(void))
+{
+	hv_kexec_handler = handler;
+	arm64_pre_smp_shutdown_hook = hv_machine_shutdown;
+}
+
+void hv_remove_kexec_handler(void)
+{
+	arm64_pre_smp_shutdown_hook = NULL;
+	hv_kexec_handler = NULL;
+}
+
 int hv_get_hypervisor_version(union hv_hypervisor_version_info *info)
 {
 	hv_get_vpreg_128(HV_REGISTER_HYPERVISOR_VERSION,
-- 
2.43.0

Re: [RFC PATCH 2/2] arm64/hyperv: Add kexec handler using machine_shutdown hook

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [High] Concurrent module unload and kexec execution causes a Time-of-Check to Time-of-Use (TOCTOU) Use-After-Free/NULL pointer dereference.

Pre-existing issues:
- [High] vmbus_initiate_unload() ignores vmbus_post_msg() errors, hanging kexec indefinitely on wait_for_completion().
--

--- Patch [2]: [RFC PATCH 2/2] arm64/hyperv: Add kexec handler using machine_shutdown hook ---
commit 49c11bb6b5ac02d64fdd78fc6417342ccc8089ed
Author: Shradha Gupta <shradhagupta@linux.microsoft.com>

arm64/hyperv: Add kexec handler using machine_shutdown hook

This patch provides the ARM64 override of hv_setup_kexec_handler() and
hv_remove_kexec_handler(). It wires the registered hv_kexec_handler to
arm64_pre_smp_shutdown_hook to perform VMBus teardown and disable SynIC
during kexec.

> diff --git a/arch/arm64/hyperv/mshyperv.c b/arch/arm64/hyperv/mshyperv.c
> index 4fdc26ade1d74..4176175060d30 100644
> --- a/arch/arm64/hyperv/mshyperv.c
> +++ b/arch/arm64/hyperv/mshyperv.c

[ ... ]

> @@ -15,10 +15,47 @@
>  #include <linux/errno.h>
>  #include <linux/version.h>
>  #include <linux/cpuhotplug.h>
> +#include <linux/kexec.h>
>  #include <asm/mshyperv.h>
> +#include <asm/system_misc.h>
>  
>  static bool hyperv_initialized;
>  
> +/*
> + * Kexec/shutdown handler for ARM64 Hyper-V guests.
> + *
> + * On x86, hv_machine_shutdown() overrides machine_ops.shutdown and
> + * runs after device_shutdown() and cpu_hotplug_enable(). ARM64 uses
> + * the arm64_pre_smp_shutdown_hook to achieve the same ordering.
> + *
> + * hv_kexec_handler() (set by vmbus_drv.c) performs:
> + *   1. vmbus_initiate_unload(false) - sends CHANNELMSG_UNLOAD
> + *   2. cpuhp_remove_state(hyperv_cpuhp_online) - disables SynIC per CPU
> + *
> + * By running after device_shutdown(), PCI drivers (NVMe, MANA) can
> + * send PCI_DELETE_INTERRUPT_MESSAGE and clean up MMIO/interrupt
> + * mappings before VMBus channels are force-closed by UNLOAD.
> + */
> +static void (*hv_kexec_handler)(void);
> +
> +static void hv_machine_shutdown(void)
> +{
> +	if (kexec_in_progress && hv_kexec_handler)
> +		hv_kexec_handler();

[Severity: High]
Is there a race condition here if the hv_vmbus module is unloaded concurrently?

If sys_delete_module() calls hv_remove_kexec_handler(), it clears
hv_kexec_handler to NULL. If that happens after the check here but before
execution, could this jump to a NULL pointer? Or if the module memory is freed
before execution, could this cause a use-after-free panic? A simple READ_ONCE()
would prevent a compiler double-fetch, but might not prevent the underlying
use-after-free without proper synchronization like RCU or a mutex.

[Severity: High]
This is a pre-existing issue, but enabling this path on ARM64 exposes the
system to a potential indefinite hang during kexec.

When hv_kexec_handler() calls vmbus_initiate_unload(false), it ignores errors
from vmbus_post_msg() and unconditionally waits for completion:

drivers/hv/channel_mgmt.c:vmbus_initiate_unload() {
	...
	vmbus_post_msg(&hdr, sizeof(struct vmbus_channel_message_header),
		       !crash);

	/*
	 * vmbus_initiate_unload() is also called on crash and the crash can be
	 * happening in an interrupt context, where scheduling is impossible.
	 */
	if (!crash)
		wait_for_completion(&vmbus_connection.unload_event);
	...
}

If vmbus_post_msg() fails (e.g., returning -ENOBUFS if the hypervisor is out
of buffers), the unload message is never sent, the completion is never
signaled, and the thread waits forever. Is there a way to handle this error
instead of hanging the reboot process?

> +}
> +
> +void hv_setup_kexec_handler(void (*handler)(void))
> +{
> +	hv_kexec_handler = handler;
> +	arm64_pre_smp_shutdown_hook = hv_machine_shutdown;
> +}
> +
> +void hv_remove_kexec_handler(void)
> +{
> +	arm64_pre_smp_shutdown_hook = NULL;
> +	hv_kexec_handler = NULL;
> +}

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260814093133.3191250-1-shradhagupta@linux.microsoft.com?part=2