[PATCH] accel/mshv: mitigate early boot vcpu exit race

[PATCH] accel/mshv: mitigate early boot vcpu exit race

[Magnus Kulke]

When using the mshv accelerator, there is a high likelihood in early
boot for a QMP execute::quit command to return w/o the QEMU process
terminating.

The race is between SIG_IPI delivery and the vcpu thread's exec loop.
qemu_cpu_kick() relies on SIG_IPI to make MSHV_RUN_VP return -EINTR so
the vcpu thread can observe cpu->stop. During early boot, there are many
PIO intercept exit produce MshvVmExitIgnore. That means the inner loop
in mshv_cpu_exec() spins through RUN_VP ioctl -> handler -> RUN_VP
ioctl, without being able to consult cpu->exit_request of cpu->stop.

The qemu_cpu_kick_self() in sa_ipi_handler() was practically a no-op,
since it calls cpus_kick_thread() which itsself issues a SIG_IPI, in the
handler for that signal. Hence it has been removed a made a practial
noop. Once we have a facility in MSHV to signal immediate_exit to the
kernel it should be used in this handler.

To mitigate we register a custom kick_vcpu_thread() impl for mshv, that
will set cpu->exit_request before calling the generic kick_vcpu_thread()
implement that will raise SIG_IPI.

We then observe cpu->exit request in the mshv_cpu_exec() inner loop and
break with EXCP_INTERRUPT when set.

Signed-off-by: Magnus Kulke <magnuskulke@linux.microsoft.com>
---
 accel/mshv/mshv-all.c | 42 ++++++++++++++++++++++++++++++++++--------
 1 file changed, 34 insertions(+), 8 deletions(-)

diff --git a/accel/mshv/mshv-all.c b/accel/mshv/mshv-all.c
index 58af674bd9..a3593c67db 100644
--- a/accel/mshv/mshv-all.c
+++ b/accel/mshv/mshv-all.c
@@ -502,6 +502,18 @@ static int mshv_cpu_exec(CPUState *cpu)
     cpu_exec_start(cpu);
 
     do {
+        /*
+         * We consider a pending exit_request before re-entering the guest.
+         * This condition is set by mshv_kick_vcpu_thread(), so we unset
+         * thread_kicked to allow future kicks to interrupt the guest.
+         */
+        if (qatomic_load_acquire(&cpu->exit_request)) {
+            qatomic_set(&cpu->exit_request, false);
+            qatomic_set_mb(&cpu->thread_kicked, false);
+            ret = EXCP_INTERRUPT;
+            break;
+        }
+
         if (cpu->accel->dirty) {
             ret = mshv_arch_put_registers(cpu);
             if (ret) {
@@ -540,17 +552,17 @@ static int mshv_cpu_exec(CPUState *cpu)
 }
 
 /*
- * The signal handler is triggered when QEMU's main thread receives a SIG_IPI
- * (SIGUSR1). This signal causes the current CPU thread to be kicked, forcing a
- * VM exit on the CPU. The VM exit generates an exit reason that breaks the loop
- * (see mshv_cpu_exec). If the exit is due to a Ctrl+A+x command, the system
- * will shut down. For other cases, the system will continue running.
+ * SIG_IPI handler for the vCPU thread. It's a noop currently. A decision to
+ * leave the guest is communicated via cpu->exit_request, which is set by
+ * mshv_kick_vcpu_thread() before it raises a SIG_IPI.
+ *
+ * Note: MSHV currently lacks an immediate_exit equivalent to KVM, there
+ * remains a theoretical race window between userspace check and ioctl entry
+ * in the vcpu loop. Once MSHV supports immediate_exit semantics, we should
+ * invoke it here.
  */
 static void sa_ipi_handler(int sig)
 {
-    /* TODO: call IOCTL to set_immediate_exit, once implemented. */
-
-    qemu_cpu_kick_self();
 }
 
 static void init_signal(CPUState *cpu)
@@ -605,6 +617,19 @@ cleanup:
     return NULL;
 }
 
+/*
+ * mshv-custom kick implementation:
+ *
+ * We set cpu->exit_request before the SIG_IPI is delivered to the vcpu
+ * thread (as part of cpus_kick_thread()). It is consumed in the
+ * mshv_cpu_exec loop.
+ */
+static void mshv_kick_vcpu_thread(CPUState *cpu)
+{
+    qatomic_set_mb(&cpu->exit_request, true);
+    cpus_kick_thread(cpu);
+}
+
 static void mshv_start_vcpu_thread(CPUState *cpu)
 {
     char thread_name[VCPU_THREAD_NAME_SIZE];
@@ -719,6 +744,7 @@ static void mshv_accel_ops_class_init(ObjectClass *oc, const void *data)
     AccelOpsClass *ops = ACCEL_OPS_CLASS(oc);
 
     ops->create_vcpu_thread = mshv_start_vcpu_thread;
+    ops->kick_vcpu_thread = mshv_kick_vcpu_thread;
     ops->synchronize_post_init = mshv_cpu_synchronize_post_init;
     ops->synchronize_post_reset = mshv_cpu_synchronize_post_reset;
     ops->synchronize_state = mshv_cpu_synchronize;
-- 
2.34.1

Re: [PATCH] accel/mshv: mitigate early boot vcpu exit race

[Doru Blânzeanu]

On Thu, May 21, 2026 at 03:34:33PM +0200, Magnus Kulke wrote:
> When using the mshv accelerator, there is a high likelihood in early
> boot for a QMP execute::quit command to return w/o the QEMU process
> terminating.
> 
> The race is between SIG_IPI delivery and the vcpu thread's exec loop.
> qemu_cpu_kick() relies on SIG_IPI to make MSHV_RUN_VP return -EINTR so
> the vcpu thread can observe cpu->stop. During early boot, there are many
> PIO intercept exit produce MshvVmExitIgnore. That means the inner loop
> in mshv_cpu_exec() spins through RUN_VP ioctl -> handler -> RUN_VP
> ioctl, without being able to consult cpu->exit_request of cpu->stop.
> 
> The qemu_cpu_kick_self() in sa_ipi_handler() was practically a no-op,
> since it calls cpus_kick_thread() which itsself issues a SIG_IPI, in the
> handler for that signal. Hence it has been removed a made a practial
> noop. Once we have a facility in MSHV to signal immediate_exit to the
> kernel it should be used in this handler.
> 
> To mitigate we register a custom kick_vcpu_thread() impl for mshv, that
> will set cpu->exit_request before calling the generic kick_vcpu_thread()
> implement that will raise SIG_IPI.
> 
> We then observe cpu->exit request in the mshv_cpu_exec() inner loop and
> break with EXCP_INTERRUPT when set.
> 
> Signed-off-by: Magnus Kulke <magnuskulke@linux.microsoft.com>
> ---
>  accel/mshv/mshv-all.c | 42 ++++++++++++++++++++++++++++++++++--------
>  1 file changed, 34 insertions(+), 8 deletions(-)
> 
> diff --git a/accel/mshv/mshv-all.c b/accel/mshv/mshv-all.c
> index 58af674bd9..a3593c67db 100644
> --- a/accel/mshv/mshv-all.c
> +++ b/accel/mshv/mshv-all.c
> @@ -502,6 +502,18 @@ static int mshv_cpu_exec(CPUState *cpu)
>      cpu_exec_start(cpu);
>  
>      do {
> +        /*
> +         * We consider a pending exit_request before re-entering the guest.
> +         * This condition is set by mshv_kick_vcpu_thread(), so we unset
> +         * thread_kicked to allow future kicks to interrupt the guest.
> +         */
> +        if (qatomic_load_acquire(&cpu->exit_request)) {
> +            qatomic_set(&cpu->exit_request, false);
> +            qatomic_set_mb(&cpu->thread_kicked, false);
> +            ret = EXCP_INTERRUPT;
> +            break;
> +        }
> +
>          if (cpu->accel->dirty) {
>              ret = mshv_arch_put_registers(cpu);
>              if (ret) {
> @@ -540,17 +552,17 @@ static int mshv_cpu_exec(CPUState *cpu)
>  }
>  
>  /*
> - * The signal handler is triggered when QEMU's main thread receives a SIG_IPI
> - * (SIGUSR1). This signal causes the current CPU thread to be kicked, forcing a
> - * VM exit on the CPU. The VM exit generates an exit reason that breaks the loop
> - * (see mshv_cpu_exec). If the exit is due to a Ctrl+A+x command, the system
> - * will shut down. For other cases, the system will continue running.
> + * SIG_IPI handler for the vCPU thread. It's a noop currently. A decision to
> + * leave the guest is communicated via cpu->exit_request, which is set by
> + * mshv_kick_vcpu_thread() before it raises a SIG_IPI.
> + *
> + * Note: MSHV currently lacks an immediate_exit equivalent to KVM, there
> + * remains a theoretical race window between userspace check and ioctl entry
> + * in the vcpu loop. Once MSHV supports immediate_exit semantics, we should
> + * invoke it here.
>   */
>  static void sa_ipi_handler(int sig)
>  {
> -    /* TODO: call IOCTL to set_immediate_exit, once implemented. */
> -
> -    qemu_cpu_kick_self();
>  }
>  
>  static void init_signal(CPUState *cpu)
> @@ -605,6 +617,19 @@ cleanup:
>      return NULL;
>  }
>  
> +/*
> + * mshv-custom kick implementation:
> + *
> + * We set cpu->exit_request before the SIG_IPI is delivered to the vcpu
> + * thread (as part of cpus_kick_thread()). It is consumed in the
> + * mshv_cpu_exec loop.
> + */
> +static void mshv_kick_vcpu_thread(CPUState *cpu)
> +{
> +    qatomic_set_mb(&cpu->exit_request, true);
> +    cpus_kick_thread(cpu);
> +}
> +
>  static void mshv_start_vcpu_thread(CPUState *cpu)
>  {
>      char thread_name[VCPU_THREAD_NAME_SIZE];
> @@ -719,6 +744,7 @@ static void mshv_accel_ops_class_init(ObjectClass *oc, const void *data)
>      AccelOpsClass *ops = ACCEL_OPS_CLASS(oc);
>  
>      ops->create_vcpu_thread = mshv_start_vcpu_thread;
> +    ops->kick_vcpu_thread = mshv_kick_vcpu_thread;
>      ops->synchronize_post_init = mshv_cpu_synchronize_post_init;
>      ops->synchronize_post_reset = mshv_cpu_synchronize_post_reset;
>      ops->synchronize_state = mshv_cpu_synchronize;
> -- 
> 2.34.1

Reviewed-by: Doru Blânzeanu <dblanzeanu@linux.microsoft.com>

Re: [PATCH] accel/mshv: mitigate early boot vcpu exit race

[Paolo Bonzini]

On 5/21/26 15:34, Magnus Kulke wrote:
> When using the mshv accelerator, there is a high likelihood in early
> boot for a QMP execute::quit command to return w/o the QEMU process
> terminating.
> 
> The race is between SIG_IPI delivery and the vcpu thread's exec loop.
> qemu_cpu_kick() relies on SIG_IPI to make MSHV_RUN_VP return -EINTR so
> the vcpu thread can observe cpu->stop. During early boot, there are many
> PIO intercept exit produce MshvVmExitIgnore. That means the inner loop
> in mshv_cpu_exec() spins through RUN_VP ioctl -> handler -> RUN_VP
> ioctl, without being able to consult cpu->exit_request of cpu->stop.
> 
> The qemu_cpu_kick_self() in sa_ipi_handler() was practically a no-op,
> since it calls cpus_kick_thread() which itsself issues a SIG_IPI, in the
> handler for that signal. Hence it has been removed a made a practial
> noop. Once we have a facility in MSHV to signal immediate_exit to the
> kernel it should be used in this handler.
> 
> To mitigate we register a custom kick_vcpu_thread() impl for mshv, that
> will set cpu->exit_request before calling the generic kick_vcpu_thread()
> implement that will raise SIG_IPI.
> 
> We then observe cpu->exit request in the mshv_cpu_exec() inner loop and
> break with EXCP_INTERRUPT when set.
> 
> Signed-off-by: Magnus Kulke <magnuskulke@linux.microsoft.com>
> Reviewed-by: Doru Blânzeanu <dblanzeanu@linux.microsoft.com>
> ---
>   accel/mshv/mshv-all.c | 42 ++++++++++++++++++++++++++++++++++--------
>   1 file changed, 34 insertions(+), 8 deletions(-)
> 
> diff --git a/accel/mshv/mshv-all.c b/accel/mshv/mshv-all.c
> index 58af674bd9..a3593c67db 100644
> --- a/accel/mshv/mshv-all.c
> +++ b/accel/mshv/mshv-all.c
> @@ -502,6 +502,18 @@ static int mshv_cpu_exec(CPUState *cpu)
>       cpu_exec_start(cpu);
>   
>       do {
> +        /*
> +         * We consider a pending exit_request before re-entering the guest.
> +         * This condition is set by mshv_kick_vcpu_thread(), so we unset
> +         * thread_kicked to allow future kicks to interrupt the guest.
> +         */
> +        if (qatomic_load_acquire(&cpu->exit_request)) {
> +            qatomic_set(&cpu->exit_request, false);
> +            qatomic_set_mb(&cpu->thread_kicked, false);


These two should not be needed.  The sequence of events is:

- exit from here

- "while (!cpu->unplug || cpu_can_run(cpu))" is still true

- qemu_process_cpu_events() runs:
   - set exit_request == false
   - cpu_thread_is_idle(cpu) is false due to cpu->stop == true
   - qemu_process_cpu_events_common() runs:
     - sets cpu->thread_kicked == false
     - qemu_cpu_stop() sets cpu->stopped == true

- cpu_can_run(cpu) is now false, so mshv_cpu_exec() does not run

- ... but !cpu->unplug is true -> another run around the loop

- now cpu_thread_is_idle(cpu) is true, so the thread goes to sleep.

> +            ret = EXCP_INTERRUPT;
> +            break;
> +        }


Also, for cleanliness I'd put the lines after the "if 
(cpu->accel->dirty)" immediately below.


> +/*
> + * mshv-custom kick implementation:
> + *
> + * We set cpu->exit_request before the SIG_IPI is delivered to the vcpu
> + * thread (as part of cpus_kick_thread()). It is consumed in the
> + * mshv_cpu_exec loop.
> + */
> +static void mshv_kick_vcpu_thread(CPUState *cpu)
> +{
> +    qatomic_set_mb(&cpu->exit_request, true);
> +    cpus_kick_thread(cpu);
> +}
> +

This should not be necessary, either.  In recent versions of QEMU, 
cpu->exit_request is set appropriately by cpu_exit(); see commits 
ac6c8a390b4 and f8217ae54e4 and nearby.

Accelerators can write cpu->exit_request, but only within the 
accelerator thread if they want to kick themselves out of cpu_exec(). 
For example, KVM does this to run kvm_arch_process_async_events(); but 
MSHV does not have anything similar.

Paolo

Re: [PATCH] accel/mshv: mitigate early boot vcpu exit race

[Magnus Kulke]

On Thu, Jun 25, 2026 at 03:15:16PM +0200, Paolo Bonzini wrote:
> 
> These two should not be needed.  The sequence of events is:
> 
> - exit from here
> 
> - "while (!cpu->unplug || cpu_can_run(cpu))" is still true
> 
> - qemu_process_cpu_events() runs:
>   - set exit_request == false
>   - cpu_thread_is_idle(cpu) is false due to cpu->stop == true
>   - qemu_process_cpu_events_common() runs:
>     - sets cpu->thread_kicked == false
>     - qemu_cpu_stop() sets cpu->stopped == true
> 
> - cpu_can_run(cpu) is now false, so mshv_cpu_exec() does not run
> 
> - ... but !cpu->unplug is true -> another run around the loop
> 
> - now cpu_thread_is_idle(cpu) is true, so the thread goes to sleep.
> 
> > +            ret = EXCP_INTERRUPT;
> > +            break;
> > +        }
> 
> 
> Also, for cleanliness I'd put the lines after the "if (cpu->accel->dirty)"
> immediately below.
> 
> 
> > +/*
> > + * mshv-custom kick implementation:
> > + *
> > + * We set cpu->exit_request before the SIG_IPI is delivered to the vcpu
> > + * thread (as part of cpus_kick_thread()). It is consumed in the
> > + * mshv_cpu_exec loop.
> > + */
> > +static void mshv_kick_vcpu_thread(CPUState *cpu)
> > +{
> > +    qatomic_set_mb(&cpu->exit_request, true);
> > +    cpus_kick_thread(cpu);
> > +}
> > +
> 
> This should not be necessary, either.  In recent versions of QEMU,
> cpu->exit_request is set appropriately by cpu_exit(); see commits
> ac6c8a390b4 and f8217ae54e4 and nearby.
> 
> Accelerators can write cpu->exit_request, but only within the accelerator
> thread if they want to kick themselves out of cpu_exec(). For example, KVM
> does this to run kvm_arch_process_async_events(); but MSHV does not have
> anything similar.
> 
> Paolo

I see, we indeed just need to honor cpu->exit_request in the loop and
break out if it is set, this only seems to be address the issue that the
patch was attempting to fix. Since we also need to it for the migration
case, I submit a revised patch as part of a migration series, so we can
abandon this one.

thanks!

magnus