[PATCH v3 0/5] Implement CPU hotplug on Arm

[PATCH v3 0/5] Implement CPU hotplug on Arm

[Mykyta Poturai]

This series implements support for CPU hotplug/unplug on Arm. To achieve this,
several things need to be done:

1. XEN_SYSCTL_CPU_HOTPLUG_* calls implemented.
2. timer and GIC maintenance interrupts switched to static irqactions to remove
the need for freeing them during release_irq.
3. Enabled the build of xen-hptool on Arm.

Tested on QEMU.

v2->v3:
* add docs

v1->v2:
* see individual patches

Mykyta Poturai (5):
  arm/time: Use static irqaction
  arm/gic: Use static irqaction
  arm/sysctl: Implement cpu hotplug ops
  tools: Allow building xen-hptool without CONFIG_MIGRATE
  docs: Document CPU hotplug

 config/Tools.mk.in               |  1 +
 docs/misc/cpu-hotplug.txt        | 51 ++++++++++++++++++++++++++++++++
 tools/configure                  | 30 +++++++++++++++++++
 tools/configure.ac               |  1 +
 tools/libs/guest/Makefile.common |  4 +++
 tools/misc/Makefile              |  2 +-
 xen/arch/arm/gic.c               | 11 +++++--
 xen/arch/arm/sysctl.c            | 45 ++++++++++++++++++++++++++++
 xen/arch/arm/time.c              | 21 ++++++++++---
 9 files changed, 159 insertions(+), 7 deletions(-)
 create mode 100644 docs/misc/cpu-hotplug.txt
 mode change 100755 => 100644 tools/configure

-- 
2.34.1

[PATCH v3 1/5] arm/time: Use static irqaction

[Mykyta Poturai]

When stopping a core deinit_timer_interrupt is called in non-alloc
context, which causes xfree in release_irq to fail an assert.

To fix this, switch to a statically allocated irqaction that does not
need to be freed in release_irq.

Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>

v2->v3:
* no changes

v1->v2:
* Use percpu actions
---
 xen/arch/arm/time.c | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)

diff --git a/xen/arch/arm/time.c b/xen/arch/arm/time.c
index e74d30d258..59349467de 100644
--- a/xen/arch/arm/time.c
+++ b/xen/arch/arm/time.c
@@ -303,9 +303,15 @@ static void check_timer_irq_cfg(unsigned int irq, const char *which)
            "WARNING: %s-timer IRQ%u is not level triggered.\n", which, irq);
 }
 
+DEFINE_PER_CPU_READ_MOSTLY(struct irqaction, irq_hyp);
+DEFINE_PER_CPU_READ_MOSTLY(struct irqaction, irq_virt);
+
 /* Set up the timer interrupt on this CPU */
 void init_timer_interrupt(void)
 {
+    struct irqaction *hyp_action = &this_cpu(irq_hyp);
+    struct irqaction *virt_action = &this_cpu(irq_virt);
+
     /* Sensible defaults */
     WRITE_SYSREG64(0, CNTVOFF_EL2);     /* No VM-specific offset */
     /* Do not let the VMs program the physical timer, only read the physical counter */
@@ -314,10 +320,17 @@ void init_timer_interrupt(void)
     WRITE_SYSREG(0, CNTHP_CTL_EL2);   /* Hypervisor's timer disabled */
     isb();
 
-    request_irq(timer_irq[TIMER_HYP_PPI], 0, htimer_interrupt,
-                "hyptimer", NULL);
-    request_irq(timer_irq[TIMER_VIRT_PPI], 0, vtimer_interrupt,
-                   "virtimer", NULL);
+    hyp_action->name = "hyptimer";
+    hyp_action->handler = htimer_interrupt;
+    hyp_action->dev_id = NULL;
+    hyp_action->free_on_release = 0;
+    setup_irq(timer_irq[TIMER_HYP_PPI], 0, hyp_action);
+
+    virt_action->name = "virtimer";
+    virt_action->handler = vtimer_interrupt;
+    virt_action->dev_id = NULL;
+    virt_action->free_on_release = 0;
+    setup_irq(timer_irq[TIMER_VIRT_PPI], 0, virt_action);
 
     check_timer_irq_cfg(timer_irq[TIMER_HYP_PPI], "hypervisor");
     check_timer_irq_cfg(timer_irq[TIMER_VIRT_PPI], "virtual");
-- 
2.34.1

Re: [PATCH v3 1/5] arm/time: Use static irqaction

[Mykola Kvach]

Hi Mykyta,

On Fri, Oct 10, 2025 at 12:22 PM Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
>
> When stopping a core deinit_timer_interrupt is called in non-alloc
> context, which causes xfree in release_irq to fail an assert.
>
> To fix this, switch to a statically allocated irqaction that does not
> need to be freed in release_irq.
>
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
>
> v2->v3:
> * no changes
>
> v1->v2:
> * Use percpu actions
> ---
>  xen/arch/arm/time.c | 21 +++++++++++++++++----
>  1 file changed, 17 insertions(+), 4 deletions(-)
>
> diff --git a/xen/arch/arm/time.c b/xen/arch/arm/time.c
> index e74d30d258..59349467de 100644
> --- a/xen/arch/arm/time.c
> +++ b/xen/arch/arm/time.c
> @@ -303,9 +303,15 @@ static void check_timer_irq_cfg(unsigned int irq, const char *which)
>             "WARNING: %s-timer IRQ%u is not level triggered.\n", which, irq);
>  }
>
> +DEFINE_PER_CPU_READ_MOSTLY(struct irqaction, irq_hyp);
> +DEFINE_PER_CPU_READ_MOSTLY(struct irqaction, irq_virt);

Scope nit (MISRA C:2012 R8.8):
if irq_hyp and irq_virt are only used in this TU, give it internal linkage.

> +
>  /* Set up the timer interrupt on this CPU */
>  void init_timer_interrupt(void)
>  {
> +    struct irqaction *hyp_action = &this_cpu(irq_hyp);
> +    struct irqaction *virt_action = &this_cpu(irq_virt);
> +
>      /* Sensible defaults */
>      WRITE_SYSREG64(0, CNTVOFF_EL2);     /* No VM-specific offset */
>      /* Do not let the VMs program the physical timer, only read the physical counter */
> @@ -314,10 +320,17 @@ void init_timer_interrupt(void)
>      WRITE_SYSREG(0, CNTHP_CTL_EL2);   /* Hypervisor's timer disabled */
>      isb();
>
> -    request_irq(timer_irq[TIMER_HYP_PPI], 0, htimer_interrupt,
> -                "hyptimer", NULL);
> -    request_irq(timer_irq[TIMER_VIRT_PPI], 0, vtimer_interrupt,
> -                   "virtimer", NULL);
> +    hyp_action->name = "hyptimer";
> +    hyp_action->handler = htimer_interrupt;
> +    hyp_action->dev_id = NULL;
> +    hyp_action->free_on_release = 0;
> +    setup_irq(timer_irq[TIMER_HYP_PPI], 0, hyp_action);
> +
> +    virt_action->name = "virtimer";
> +    virt_action->handler = vtimer_interrupt;
> +    virt_action->dev_id = NULL;
> +    virt_action->free_on_release = 0;
> +    setup_irq(timer_irq[TIMER_VIRT_PPI], 0, virt_action);
>
>      check_timer_irq_cfg(timer_irq[TIMER_HYP_PPI], "hypervisor");
>      check_timer_irq_cfg(timer_irq[TIMER_VIRT_PPI], "virtual");
> --
> 2.34.1
>

Reviewed-by: Mykola Kvach <mykola_kvach@epam.com>

Best regards,
Mykola

Re: [PATCH v3 1/5] arm/time: Use static irqaction

[Julien Grall]

Hi Mykyta,

On 10/10/2025 10:21, Mykyta Poturai wrote:
> When stopping a core deinit_timer_interrupt is called in non-alloc
> context, which causes xfree in release_irq to fail an assert.
> 
> To fix this, switch to a statically allocated irqaction that does not
> need to be freed in release_irq.
> 
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>

Assuming Mykola's comment will be addressed:

Reviewed-by: Julien Grall <jgrall@amazon.com>

Cheers,

-- 
Julien Grall

[PATCH v3 3/5] arm/sysctl: Implement cpu hotplug ops

[Mykyta Poturai]

Implement XEN_SYSCTL_CPU_HOTPLUG_{ONLINE,OFFLINE} calls to allow for
enabling/disabling CPU cores in runtime.

Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>

v2->v3:
* no changes

v1->v2:
* remove SMT ops
* remove cpu == 0 checks
* add XSM hooks
* only implement for 64bit Arm
---
 xen/arch/arm/sysctl.c | 45 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 45 insertions(+)

diff --git a/xen/arch/arm/sysctl.c b/xen/arch/arm/sysctl.c
index 32cab4feff..fecd649db1 100644
--- a/xen/arch/arm/sysctl.c
+++ b/xen/arch/arm/sysctl.c
@@ -12,6 +12,8 @@
 #include <xen/dt-overlay.h>
 #include <xen/errno.h>
 #include <xen/hypercall.h>
+#include <xen/cpu.h>
+#include <xsm/xsm.h>
 #include <asm/arm64/sve.h>
 #include <public/sysctl.h>
 
@@ -23,6 +25,42 @@ void arch_do_physinfo(struct xen_sysctl_physinfo *pi)
                                        XEN_SYSCTL_PHYSCAP_ARM_SVE_MASK);
 }
 
+#ifdef CONFIG_ARM_64
+static long cpu_up_helper(void *data)
+{
+    unsigned long cpu = (unsigned long) data;
+    return cpu_up(cpu);
+}
+
+static long cpu_down_helper(void *data)
+{
+    unsigned long cpu = (unsigned long) data;
+    return cpu_down(cpu);
+}
+
+static long cpu_hotplug_sysctl(struct xen_sysctl_cpu_hotplug *hotplug)
+{
+    int ret;
+
+    switch (hotplug->op) {
+        case XEN_SYSCTL_CPU_HOTPLUG_ONLINE:
+            ret = xsm_resource_plug_core(XSM_HOOK);
+            if ( ret )
+                return ret;
+            return continue_hypercall_on_cpu(0, cpu_up_helper, _p(hotplug->cpu));
+
+        case XEN_SYSCTL_CPU_HOTPLUG_OFFLINE:
+            ret = xsm_resource_unplug_core(XSM_HOOK);
+            if ( ret )
+                return ret;
+            return continue_hypercall_on_cpu(0, cpu_down_helper, _p(hotplug->cpu));
+
+        default:
+            return -EOPNOTSUPP;
+    }
+}
+#endif
+
 long arch_do_sysctl(struct xen_sysctl *sysctl,
                     XEN_GUEST_HANDLE_PARAM(xen_sysctl_t) u_sysctl)
 {
@@ -34,6 +72,13 @@ long arch_do_sysctl(struct xen_sysctl *sysctl,
         ret = dt_overlay_sysctl(&sysctl->u.dt_overlay);
         break;
 
+/* CPU Hotplug only implemented for 64-bit Arm */
+#ifdef CONFIG_ARM_64
+    case XEN_SYSCTL_cpu_hotplug:
+        ret = cpu_hotplug_sysctl(&sysctl->u.cpu_hotplug);
+        break;
+#endif
+
     default:
         ret = -ENOSYS;
         break;
-- 
2.34.1

Re: [PATCH v3 3/5] arm/sysctl: Implement cpu hotplug ops

[Julien Grall]

Hi,

On 10/10/2025 10:21, Mykyta Poturai wrote:
> Implement XEN_SYSCTL_CPU_HOTPLUG_{ONLINE,OFFLINE} calls to allow for
> enabling/disabling CPU cores in runtime.
> 
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> 
> v2->v3:
> * no changes
> 
> v1->v2:
> * remove SMT ops
> * remove cpu == 0 checks
> * add XSM hooks
> * only implement for 64bit Arm

Can you add some details in the commit message explaining why the 
feature is only enabled for 32-bit Arm?

> ---
>   xen/arch/arm/sysctl.c | 45 +++++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 45 insertions(+)
> 
> diff --git a/xen/arch/arm/sysctl.c b/xen/arch/arm/sysctl.c
> index 32cab4feff..fecd649db1 100644
> --- a/xen/arch/arm/sysctl.c
> +++ b/xen/arch/arm/sysctl.c
> @@ -12,6 +12,8 @@
>   #include <xen/dt-overlay.h>
>   #include <xen/errno.h>
>   #include <xen/hypercall.h>
> +#include <xen/cpu.h>
> +#include <xsm/xsm.h>
>   #include <asm/arm64/sve.h>
>   #include <public/sysctl.h>
>   
> @@ -23,6 +25,42 @@ void arch_do_physinfo(struct xen_sysctl_physinfo *pi)
>                                          XEN_SYSCTL_PHYSCAP_ARM_SVE_MASK);
>   }
>   
> +#ifdef CONFIG_ARM_64
> +static long cpu_up_helper(void *data)
> +{
> +    unsigned long cpu = (unsigned long) data;
> +    return cpu_up(cpu);
> +}
> +
> +static long cpu_down_helper(void *data)
> +{
> +    unsigned long cpu = (unsigned long) data;
> +    return cpu_down(cpu);
> +}
> +
> +static long cpu_hotplug_sysctl(struct xen_sysctl_cpu_hotplug *hotplug)
> +{
> +    int ret;
> +
> +    switch (hotplug->op) {
> +        case XEN_SYSCTL_CPU_HOTPLUG_ONLINE:
> +            ret = xsm_resource_plug_core(XSM_HOOK);
> +            if ( ret )
> +                return ret;
> +            return continue_hypercall_on_cpu(0, cpu_up_helper, _p(hotplug->cpu));
> +
> +        case XEN_SYSCTL_CPU_HOTPLUG_OFFLINE:
> +            ret = xsm_resource_unplug_core(XSM_HOOK);
> +            if ( ret )
> +                return ret;
> +            return continue_hypercall_on_cpu(0, cpu_down_helper, _p(hotplug->cpu));
> +
> +        default:
> +            return -EOPNOTSUPP;
> +    }
> +}
> +#endif

The logic seems to be very similar to the x86 code. There are some 
slight differences in cpu_up_helper() and cpu_down_helper() but:

* For cpu_up_helper(), we could create an arch specific helper for the 
second if check.
* For cpu_down_helper(), it would be ok to call cpu_down() a second time 
on Arm.

Can you look at consolidating the code?

Cheers,

-- 
Julien Grall

[PATCH v3 2/5] arm/gic: Use static irqaction

[Mykyta Poturai]

When stopping a core cpu_gic_callback is called in non-alloc
context, which causes xfree in release_irq to fail an assert.

To fix this, switch to a statically allocated irqaction that does not
need to be freed in release_irq.

Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>

v2->v3:
* no changes

v1->v2:
* use percpu actions
---
 xen/arch/arm/gic.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
index 260ee64cca..ed6853bb32 100644
--- a/xen/arch/arm/gic.c
+++ b/xen/arch/arm/gic.c
@@ -386,10 +386,17 @@ void gic_dump_info(struct vcpu *v)
     gic_hw_ops->dump_state(v);
 }
 
+DEFINE_PER_CPU_READ_MOSTLY(struct irqaction, irq_maintenance);
+
 void init_maintenance_interrupt(void)
 {
-    request_irq(gic_hw_ops->info->maintenance_irq, 0, maintenance_interrupt,
-                "irq-maintenance", NULL);
+    struct irqaction *maintenance = &this_cpu(irq_maintenance);
+
+    maintenance->name = "irq-maintenance";
+    maintenance->handler = maintenance_interrupt;
+    maintenance->dev_id = NULL;
+    maintenance->free_on_release = 0;
+    setup_irq(gic_hw_ops->info->maintenance_irq, 0, maintenance);
 }
 
 int gic_make_hwdom_dt_node(const struct domain *d,
-- 
2.34.1

Re: [PATCH v3 2/5] arm/gic: Use static irqaction

[Mykola Kvach]

Hi Mykyta,

On Fri, Oct 10, 2025 at 12:22 PM Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
>
> When stopping a core cpu_gic_callback is called in non-alloc
> context, which causes xfree in release_irq to fail an assert.
>
> To fix this, switch to a statically allocated irqaction that does not
> need to be freed in release_irq.
>
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
>
> v2->v3:
> * no changes
>
> v1->v2:
> * use percpu actions
> ---
>  xen/arch/arm/gic.c | 11 +++++++++--
>  1 file changed, 9 insertions(+), 2 deletions(-)
>
> diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
> index 260ee64cca..ed6853bb32 100644
> --- a/xen/arch/arm/gic.c
> +++ b/xen/arch/arm/gic.c
> @@ -386,10 +386,17 @@ void gic_dump_info(struct vcpu *v)
>      gic_hw_ops->dump_state(v);
>  }
>
> +DEFINE_PER_CPU_READ_MOSTLY(struct irqaction, irq_maintenance);

Scope nit (MISRA C:2012 R8.8):
if irq_maintenance is only used in this TU, give it internal linkage.

> +
>  void init_maintenance_interrupt(void)
>  {
> -    request_irq(gic_hw_ops->info->maintenance_irq, 0, maintenance_interrupt,
> -                "irq-maintenance", NULL);
> +    struct irqaction *maintenance = &this_cpu(irq_maintenance);
> +
> +    maintenance->name = "irq-maintenance";
> +    maintenance->handler = maintenance_interrupt;
> +    maintenance->dev_id = NULL;
> +    maintenance->free_on_release = 0;
> +    setup_irq(gic_hw_ops->info->maintenance_irq, 0, maintenance);
>  }
>
>  int gic_make_hwdom_dt_node(const struct domain *d,
> --
> 2.34.1
>

Reviewed-by: Mykola Kvach <mykola_kvach@epam.com>

Best regards,
Mykola

Re: [PATCH v3 2/5] arm/gic: Use static irqaction

[Julien Grall]

Hi Mykyta,

On 10/10/2025 10:21, Mykyta Poturai wrote:
> When stopping a core cpu_gic_callback is called in non-alloc
> context, which causes xfree in release_irq to fail an assert.
> 
> To fix this, switch to a statically allocated irqaction that does not
> need to be freed in release_irq.
> 
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>

Assuming Mykola's comment is address:

Reviewed-by: Julien Grall <jgrall@amazon.com>

Cheers,

-- 
Julien Grall

[PATCH v3 4/5] tools: Allow building xen-hptool without CONFIG_MIGRATE

[Mykyta Poturai]

With CPU hotplug sysctls implemented on Arm it becomes useful to have a
tool for calling them. Introduce a new congifure option "hptool" to
allow building hptool separately from other migration tools, and enable
it by default.

Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>

v2->v3:
* no changes

v1->v2:
* switch to configure from legacy config
---
 config/Tools.mk.in               |  1 +
 tools/configure                  | 30 ++++++++++++++++++++++++++++++
 tools/configure.ac               |  1 +
 tools/libs/guest/Makefile.common |  4 ++++
 tools/misc/Makefile              |  2 +-
 5 files changed, 37 insertions(+), 1 deletion(-)
 mode change 100755 => 100644 tools/configure

diff --git a/config/Tools.mk.in b/config/Tools.mk.in
index e47ac23d11..eb4855d93d 100644
--- a/config/Tools.mk.in
+++ b/config/Tools.mk.in
@@ -49,6 +49,7 @@ CONFIG_LIBNL        := @libnl@
 CONFIG_GOLANG       := @golang@
 CONFIG_PYGRUB       := @pygrub@
 CONFIG_LIBFSIMAGE   := @libfsimage@
+CONFIG_HPTOOL       := @hptool@
 
 CONFIG_SYSTEMD      := @systemd@
 XEN_SYSTEMD_DIR     := @SYSTEMD_DIR@
diff --git a/tools/configure b/tools/configure
old mode 100755
new mode 100644
index 5abd44e21e..5cf5381c0a
--- a/tools/configure
+++ b/tools/configure
@@ -728,6 +728,7 @@ LD86
 AS86
 ipxe
 LINUX_BACKEND_MODULES
+hptool
 pygrub
 golang
 seabios
@@ -834,6 +835,7 @@ enable_ovmf
 enable_seabios
 enable_golang
 enable_pygrub
+enable_hptool
 with_linux_backend_modules
 enable_ipxe
 with_system_ipxe
@@ -1519,6 +1521,7 @@ Optional Features:
   --disable-seabios       Disable SeaBIOS (default is ENABLED)
   --disable-golang        Disable Go tools (default is ENABLED)
   --disable-pygrub        Disable pygrub (default is ENABLED)
+  --disable-hptool        Disable hptool (default is ENABLED)
   --enable-ipxe           Enable in-tree IPXE, (DEFAULT is off, see also
                           --with-system-ipxe)
   --enable-rombios        Enable ROMBIOS, (DEFAULT is on if ipxe is enabled,
@@ -4807,6 +4810,33 @@ pygrub=$ax_cv_pygrub
 
 
 
+# Check whether --enable-hptool was given.
+if test ${enable_hptool+y}
+then :
+  enableval=$enable_hptool;
+fi
+
+
+if test "x$enable_hptool" = "xno"
+then :
+
+    ax_cv_hptool="n"
+
+elif test "x$enable_hptool" = "xyes"
+then :
+
+    ax_cv_hptool="y"
+
+elif test -z $ax_cv_hptool
+then :
+
+    ax_cv_hptool="y"
+
+fi
+hptool=$ax_cv_hptool
+
+
+
 
 # Check whether --with-linux-backend-modules was given.
 if test ${with_linux_backend_modules+y}
diff --git a/tools/configure.ac b/tools/configure.ac
index dada1c3b15..3a0644ef89 100644
--- a/tools/configure.ac
+++ b/tools/configure.ac
@@ -90,6 +90,7 @@ AX_ARG_DEFAULT_DISABLE([ovmf], [Enable OVMF])
 AX_ARG_DEFAULT_ENABLE([seabios], [Disable SeaBIOS])
 AX_ARG_DEFAULT_ENABLE([golang], [Disable Go tools])
 AX_ARG_DEFAULT_ENABLE([pygrub], [Disable pygrub])
+AX_ARG_DEFAULT_ENABLE([hptool], [Disable hptool])
 
 AC_ARG_WITH([linux-backend-modules],
     AS_HELP_STRING([--with-linux-backend-modules="mod1 mod2"],
diff --git a/tools/libs/guest/Makefile.common b/tools/libs/guest/Makefile.common
index a026a2f662..774b1d5392 100644
--- a/tools/libs/guest/Makefile.common
+++ b/tools/libs/guest/Makefile.common
@@ -25,6 +25,10 @@ OBJS-y       += xg_core.o
 OBJS-$(CONFIG_X86) += xg_core_x86.o
 OBJS-$(CONFIG_ARM) += xg_core_arm.o
 
+ifneq (,$(filter y,$(CONFIG_MIGRATE)$(CONFIG_HPTOOL)))
+OBJS-y += xg_offline_page.o
+endif
+
 vpath %.c ../../../xen/common/libelf
 
 LIBELF_OBJS += libelf-tools.o libelf-loader.o
diff --git a/tools/misc/Makefile b/tools/misc/Makefile
index c26e544e83..f783f16ae6 100644
--- a/tools/misc/Makefile
+++ b/tools/misc/Makefile
@@ -16,7 +16,7 @@ INSTALL_BIN                    += xencov_split
 INSTALL_BIN += $(INSTALL_BIN-y)
 
 # Everything to be installed in regular sbin/
-INSTALL_SBIN-$(CONFIG_MIGRATE) += xen-hptool
+INSTALL_SBIN-$(CONFIG_HPTOOL)  += xen-hptool
 INSTALL_SBIN-$(CONFIG_X86)     += xen-hvmcrash
 INSTALL_SBIN-$(CONFIG_X86)     += xen-hvmctx
 INSTALL_SBIN-$(CONFIG_X86)     += xen-lowmemd
-- 
2.34.1

[PATCH v3 5/5] docs: Document CPU hotplug

[Mykyta Poturai]

Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>

v2->v3:
* patch introduced
---
 docs/misc/cpu-hotplug.txt | 51 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 51 insertions(+)
 create mode 100644 docs/misc/cpu-hotplug.txt

diff --git a/docs/misc/cpu-hotplug.txt b/docs/misc/cpu-hotplug.txt
new file mode 100644
index 0000000000..f2b340d2ae
--- /dev/null
+++ b/docs/misc/cpu-hotplug.txt
@@ -0,0 +1,51 @@
+CPU Hotplug
+===========
+
+CPU hotplug is a feature that allows pCPU cores to be added to or removed from a
+running system without requiring a reboot. It is supported on x86 and Arm64
+architectures.
+
+Implementation Details
+----------------------
+
+CPU hotplug is implemented through the `XEN_SYSCTL_CPU_HOTPLUG_*` sysctl calls.
+The specific calls are:
+
+- `XEN_SYSCTL_CPU_HOTPLUG_ONLINE`: Brings a pCPU online
+- `XEN_SYSCTL_CPU_HOTPLUG_OFFLINE`: Takes a pCPU offline
+- `XEN_SYSCTL_CPU_HOTPLUG_SMT_ENABLE`: Enables SMT threads (x86 only)
+- `XEN_SYSCTL_CPU_HOTPLUG_SMT_DISABLE`: Disables SMT threads (x86 only)
+
+All cores can be disabled, assuming hardware support, except for core 0. Sysctl
+calls are routed to core 0 before doing any actual up/down operations on other
+cores.
+
+Configuration
+-------------
+
+Sysctl handlers are enabled unconditionally on supported architectures. Building
+of the userspace tool "hptool" is controlled by the "hptool" flag in the
+configure script. It is enabled by default and can be disabled with
+--disable-hptool command line option.
+
+Usage
+-----
+
+Disable core:
+
+$ xen-hptool cpu-offline 2
+Prepare to offline CPU 2
+(XEN) Removing cpu 2 from runqueue 0
+CPU 2 offlined successfully
+
+Enable core:
+
+$ xen-hptool cpu-online 2
+Prepare to online CPU 2
+(XEN) Bringing up CPU2
+(XEN) GICv3: CPU2: Found redistributor in region 0 @00000a004005c000
+(XEN) CPU2: Guest atomics will try 1 times before pausing the domain
+(XEN) CPU 2 booted.
+(XEN) Adding cpu 2 to runqueue 0
+CPU 2 onlined successfully
+
-- 
2.34.1

Re: [PATCH v3 0/5] Implement CPU hotplug on Arm

[Mykola Kvach]

Hi Mykyta,

Thanks for the series.

It seems there might be issues here -- please take a look and let me
know if my concerns are valid:

1. FF-A notification IRQ: after a CPU down->up cycle the IRQ
configuration may be lost.

2. GICv3 LPIs: a CPU may fail to come back up unless its LPI pending
table exists (is allocated) on bring-up. See
gicv3_lpi_allocate_pendtable() and its call chain.

3. IRQ migration on CPU down: if an IRQ targets a CPU being offlined,
its affinity should be moved to an online CPU before completing the
offlining.

4. Race between the new hypercalls and disable/enable_nonboot_cpus():
disable_nonboot_cpus is called, enable_nonboot_cpus() reads
frozen_cpus, and before it calls cpu_up() a hypercall onlines the CPU.
cpu_up() then fails as "already online", but the CPU_RESUME_FAILED
path may still run for an already-online CPU, risking use-after-free
of per-CPU state (e.g. via free_percpu_area()) and other issues
related to CPU_RESUME_FAILED notification.



On Fri, Oct 10, 2025 at 12:36 PM Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
>
> This series implements support for CPU hotplug/unplug on Arm. To achieve this,
> several things need to be done:
>
> 1. XEN_SYSCTL_CPU_HOTPLUG_* calls implemented.
> 2. timer and GIC maintenance interrupts switched to static irqactions to remove
> the need for freeing them during release_irq.
> 3. Enabled the build of xen-hptool on Arm.
>
> Tested on QEMU.
>
> v2->v3:
> * add docs
>
> v1->v2:
> * see individual patches
>
> Mykyta Poturai (5):
>   arm/time: Use static irqaction
>   arm/gic: Use static irqaction
>   arm/sysctl: Implement cpu hotplug ops
>   tools: Allow building xen-hptool without CONFIG_MIGRATE
>   docs: Document CPU hotplug
>
>  config/Tools.mk.in               |  1 +
>  docs/misc/cpu-hotplug.txt        | 51 ++++++++++++++++++++++++++++++++
>  tools/configure                  | 30 +++++++++++++++++++
>  tools/configure.ac               |  1 +
>  tools/libs/guest/Makefile.common |  4 +++
>  tools/misc/Makefile              |  2 +-
>  xen/arch/arm/gic.c               | 11 +++++--
>  xen/arch/arm/sysctl.c            | 45 ++++++++++++++++++++++++++++
>  xen/arch/arm/time.c              | 21 ++++++++++---
>  9 files changed, 159 insertions(+), 7 deletions(-)
>  create mode 100644 docs/misc/cpu-hotplug.txt
>  mode change 100755 => 100644 tools/configure
>
> --
> 2.34.1
>

Best regards,
Mykola

Re: [PATCH v3 0/5] Implement CPU hotplug on Arm

[Mykyta Poturai]

On 15.10.25 20:30, Mykola Kvach wrote:
> Hi Mykyta,
> 
> Thanks for the series.
> 
> It seems there might be issues here -- please take a look and let me
> know if my concerns are valid:
> 
> 1. FF-A notification IRQ: after a CPU down->up cycle the IRQ
> configuration may be lost.

OPTEE and FFA are marked as unsupported.

> 2. GICv3 LPIs: a CPU may fail to come back up unless its LPI pending
> table exists (is allocated) on bring-up. See
> gicv3_lpi_allocate_pendtable() and its call chain.

ITS is marked as unsupported. I have a plan to deal with this, but it is 
out of scope of this series.

> 3. IRQ migration on CPU down: if an IRQ targets a CPU being offlined,
> its affinity should be moved to an online CPU before completing the
> offlining.

All guest tied IRQ migration is handled by the scheduler. Regarding the 
irqs used by Xen, I didn't find any with affinity to other CPUs than CPU 
0, which can't be disabled. I think theoretically it is possible for 
them to have different affinity, but it seems unlikely considering that 
x86 hotplug code also doesn't seem to do any Xen irq migration AFAIU.

> 4. Race between the new hypercalls and disable/enable_nonboot_cpus():
> disable_nonboot_cpus is called, enable_nonboot_cpus() reads
> frozen_cpus, and before it calls cpu_up() a hypercall onlines the CPU.
> cpu_up() then fails as "already online", but the CPU_RESUME_FAILED
> path may still run for an already-online CPU, risking use-after-free
> of per-CPU state (e.g. via free_percpu_area()) and other issues
> related to CPU_RESUME_FAILED notification.
> 

There don't seem to be any calls to disable/enable_nonboot_cpus() on 
Arm. If we take x86 as an example, then they are called with all domains 
already paused, and I don't see how paused domains can issue hypercalls.

> 
> Best regards,
> Mykola

-- 
Mykyta

Re: [PATCH v3 0/5] Implement CPU hotplug on Arm

[Mykola Kvach]

Hi Mykyta,

Thank you for your answers

On Mon, Oct 20, 2025 at 5:15 PM Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
>
> On 15.10.25 20:30, Mykola Kvach wrote:
> > Hi Mykyta,
> >
> > Thanks for the series.
> >
> > It seems there might be issues here -- please take a look and let me
> > know if my concerns are valid:
> >
> > 1. FF-A notification IRQ: after a CPU down->up cycle the IRQ
> > configuration may be lost.
>
> OPTEE and FFA are marked as unsupported.

Understood, thanks. Would it be worth documenting this?

>
> > 2. GICv3 LPIs: a CPU may fail to come back up unless its LPI pending
> > table exists (is allocated) on bring-up. See
> > gicv3_lpi_allocate_pendtable() and its call chain.
>
> ITS is marked as unsupported. I have a plan to deal with this, but it is
> out of scope of this series.

Thanks for the clarification. Should we document this somewhere?

>
> > 3. IRQ migration on CPU down: if an IRQ targets a CPU being offlined,
> > its affinity should be moved to an online CPU before completing the
> > offlining.
>
> All guest tied IRQ migration is handled by the scheduler. Regarding the
> irqs used by Xen, I didn't find any with affinity to other CPUs than CPU
> 0, which can't be disabled. I think theoretically it is possible for
> them to have different affinity, but it seems unlikely considering that
> x86 hotplug code also doesn't seem to do any Xen irq migration AFAIU.

What about arm_smmu_init_domain_context and its related call chains?
As far as I can see, some of these paths touch XEN_DOMCTL_* hypercalls,
and my understanding is they can be issued on any CPU. Should we add a
check that no enabled (e)SPIs owned by Xen are pinned to the offlining
CPU?

>
> > 4. Race between the new hypercalls and disable/enable_nonboot_cpus():
> > disable_nonboot_cpus is called, enable_nonboot_cpus() reads
> > frozen_cpus, and before it calls cpu_up() a hypercall onlines the CPU.
> > cpu_up() then fails as "already online", but the CPU_RESUME_FAILED
> > path may still run for an already-online CPU, risking use-after-free
> > of per-CPU state (e.g. via free_percpu_area()) and other issues
> > related to CPU_RESUME_FAILED notification.
> >
>
> There don't seem to be any calls to disable/enable_nonboot_cpus() on
> Arm. If we take x86 as an example, then they are called with all domains
> already paused, and I don't see how paused domains can issue hypercalls.

Agreed; this looks even less likely given that disable_* runs on CPU0 and
your new hypercalls execute on CPU0. The only plausible issue would be a
contrived case where code disables non-boot CPUs from CPU0 but enables them
from another CPU woken by a hypercall. That seems unrealistic.

>
> >
> > Best regards,
> > Mykola
>
> --
> Mykyta

Best regards,
Mykola

Re: [PATCH v3 0/5] Implement CPU hotplug on Arm

[Julien Grall]

Hi,

On 20/10/2025 19:00, Mykola Kvach wrote:
> Thank you for your answers
> 
> On Mon, Oct 20, 2025 at 5:15 PM Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
>>
>> On 15.10.25 20:30, Mykola Kvach wrote:
>>> Hi Mykyta,
>>>
>>> Thanks for the series.
>>>
>>> It seems there might be issues here -- please take a look and let me
>>> know if my concerns are valid:
>>>
>>> 1. FF-A notification IRQ: after a CPU down->up cycle the IRQ
>>> configuration may be lost.
>>
>> OPTEE and FFA are marked as unsupported.
> 
> Understood, thanks. Would it be worth documenting this?

This must be documented. OP-TEE, FFA and ITS will eventually be 
supported. So we need to know the gap.

I think it would also be worth to have a Kconfig indicating whether CPU 
hotplug (and soon suspend/resume) can be used with the documentation. So 
CPU hotplug will gracefully fail rather than putting the system in a 
undefined state.

>>
>>> 2. GICv3 LPIs: a CPU may fail to come back up unless its LPI pending
>>> table exists (is allocated) on bring-up. See
>>> gicv3_lpi_allocate_pendtable() and its call chain.
>>
>> ITS is marked as unsupported. I have a plan to deal with this, but it is
>> out of scope of this series.
 > > Thanks for the clarification. Should we document this somewhere?
> 
>>
>>> 3. IRQ migration on CPU down: if an IRQ targets a CPU being offlined,
>>> its affinity should be moved to an online CPU before completing the
>>> offlining.
>>
>> All guest tied IRQ migration is handled by the scheduler. Regarding the
>> irqs used by Xen, I didn't find any with affinity to other CPUs than CPU
>> 0, which can't be disabled. I think theoretically it is possible for
>> them to have different affinity, but it seems unlikely considering that
>> x86 hotplug code also doesn't seem to do any Xen irq migration AFAIU.
> 
> What about arm_smmu_init_domain_context and its related call chains?
> As far as I can see, some of these paths touch XEN_DOMCTL_* hypercalls,
> and my understanding is they can be issued on any CPU.

You are right. The SMMU can be configured from any pCPU. When 
request_irq() is called, it will route the IRQ to the current pCPU.

Those IRQs are not guest interrupts, so from my understanding, they 
would not be migrated.

> Should we add a
> check that no enabled (e)SPIs owned by Xen are pinned to the offlining
> CPU?

This would be good. But I also think we should aim to migrate those 
interrupts.

>>
>>> 4. Race between the new hypercalls and disable/enable_nonboot_cpus():
>>> disable_nonboot_cpus is called, enable_nonboot_cpus() reads
>>> frozen_cpus, and before it calls cpu_up() a hypercall onlines the CPU.
>>> cpu_up() then fails as "already online", but the CPU_RESUME_FAILED
>>> path may still run for an already-online CPU, risking use-after-free
>>> of per-CPU state (e.g. via free_percpu_area()) and other issues
>>> related to CPU_RESUME_FAILED notification.
>>>
>>
>> There don't seem to be any calls to disable/enable_nonboot_cpus() on
>> Arm.

Yet. There is a patch series to use the functions as part of 
suspend/resume. In fact this series is a pre-requisite for the 
suspend/resume series.

> If we take x86 as an example, then they are called with all domains
>> already paused, and I don't see how paused domains can issue hypercalls.

The Arm version will also freeze all the domains before calling 
disable_nonboot_cpus(). So there should be no race on Arm as well.

Cheers,

-- 
Julien Grall