[PATCH v5 0/5] Implement CPU hotplug on Arm

[PATCH v5 0/5] Implement CPU hotplug on Arm

[Mykyta Poturai]

This series implements support for CPU hotplug/unplug on Arm. To achieve this,
several things need to be done:

1. XEN_SYSCTL_CPU_HOTPLUG_* calls implemented on Arm64.
2. Enabled building of xen-hptool.
3. Migration of irqs from dying CPUs implemented.

Tested on QEMU.

v4->v5:
* drop merged patches
* combine "smp: Move cpu_up/down helpers to common code" with 
  "arm/sysctl: Implement cpu hotplug ops"
* see individual patches

v3->v4:
* add irq migration patches
* see individual patches

v2->v3:
* add docs

v1->v2:
* see individual patches

Mykyta Poturai (5):
  arm/irq: Keep track of irq affinities
  arm/irq: Migrate IRQs during CPU up/down operations
  arm/sysctl: Implement cpu hotplug ops
  tools: Allow building xen-hptool without CONFIG_MIGRATE
  docs: Document CPU hotplug

 SUPPORT.md                       |  1 +
 docs/misc/cpu-hotplug.txt        | 50 +++++++++++++++++++++++++
 tools/libs/guest/Makefile.common |  2 +-
 tools/misc/Makefile              |  2 +-
 xen/arch/arm/Kconfig             |  1 +
 xen/arch/arm/gic-vgic.c          |  2 +
 xen/arch/arm/include/asm/irq.h   |  2 +
 xen/arch/arm/irq.c               | 63 +++++++++++++++++++++++++++++++-
 xen/arch/arm/smp.c               |  9 +++++
 xen/arch/arm/smpboot.c           |  6 +++
 xen/arch/arm/vgic.c              | 14 ++++++-
 xen/arch/ppc/stubs.c             |  4 ++
 xen/arch/riscv/stubs.c           |  5 +++
 xen/arch/x86/Kconfig             |  1 +
 xen/arch/x86/include/asm/smp.h   |  3 --
 xen/arch/x86/smp.c               | 33 ++---------------
 xen/arch/x86/sysctl.c            | 12 ++----
 xen/common/Kconfig               |  3 ++
 xen/common/smp.c                 | 34 +++++++++++++++++
 xen/common/sysctl.c              | 45 +++++++++++++++++++++++
 xen/include/xen/smp.h            |  4 ++
 21 files changed, 248 insertions(+), 48 deletions(-)
 create mode 100644 docs/misc/cpu-hotplug.txt

-- 
2.51.2

[PATCH v5 1/5] arm/irq: Keep track of irq affinities

[Mykyta Poturai]

Currently on Arm the desc->affinity mask of an irq is never updated,
which makes it hard to know the actual affinity of an interrupt.

Fix this by updating the field in irq_set_affinity.

Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>

---
v4->v5:
* add locking

v3->v4:
* patch introduced
---
 xen/arch/arm/gic-vgic.c |  2 ++
 xen/arch/arm/irq.c      |  9 +++++++--
 xen/arch/arm/vgic.c     | 14 ++++++++++++--
 3 files changed, 21 insertions(+), 4 deletions(-)

diff --git a/xen/arch/arm/gic-vgic.c b/xen/arch/arm/gic-vgic.c
index ea48c5375a..5253caf002 100644
--- a/xen/arch/arm/gic-vgic.c
+++ b/xen/arch/arm/gic-vgic.c
@@ -232,7 +232,9 @@ static void gic_update_one_lr(struct vcpu *v, int i)
             if ( test_bit(GIC_IRQ_GUEST_MIGRATING, &p->status) )
             {
                 struct vcpu *v_target = vgic_get_target_vcpu(v, irq);
+                spin_lock(&p->desc->lock);
                 irq_set_affinity(p->desc, cpumask_of(v_target->processor));
+                spin_unlock(&p->desc->lock);
                 clear_bit(GIC_IRQ_GUEST_MIGRATING, &p->status);
             }
         }
diff --git a/xen/arch/arm/irq.c b/xen/arch/arm/irq.c
index 73e58a5108..7204bc2b68 100644
--- a/xen/arch/arm/irq.c
+++ b/xen/arch/arm/irq.c
@@ -216,10 +216,15 @@ static inline struct domain *irq_get_domain(struct irq_desc *desc)
     return irq_get_guest_info(desc)->d;
 }
 
+/* Must be called with desc->lock held */
 void irq_set_affinity(struct irq_desc *desc, const cpumask_t *mask)
 {
-    if ( desc != NULL )
-        desc->handler->set_affinity(desc, mask);
+    if ( desc == NULL )
+        return;
+
+    ASSERT(spin_is_locked(&desc->lock));
+    cpumask_copy(desc->affinity, mask);
+    desc->handler->set_affinity(desc, mask);
 }
 
 int request_irq(unsigned int irq, unsigned int irqflags,
diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
index 6647071ad4..c59f6873db 100644
--- a/xen/arch/arm/vgic.c
+++ b/xen/arch/arm/vgic.c
@@ -445,7 +445,9 @@ bool vgic_migrate_irq(struct vcpu *old, struct vcpu *new, unsigned int irq)
 
     if ( list_empty(&p->inflight) )
     {
+        spin_lock(&p->desc->lock);
         irq_set_affinity(p->desc, cpumask_of(new->processor));
+        spin_unlock(&p->desc->lock);
         spin_unlock_irqrestore(&old->arch.vgic.lock, flags);
         return true;
     }
@@ -453,7 +455,9 @@ bool vgic_migrate_irq(struct vcpu *old, struct vcpu *new, unsigned int irq)
     if ( !list_empty(&p->lr_queue) )
     {
         vgic_remove_irq_from_queues(old, p);
+        spin_lock(&p->desc->lock);
         irq_set_affinity(p->desc, cpumask_of(new->processor));
+        spin_unlock(&p->desc->lock);
         spin_unlock_irqrestore(&old->arch.vgic.lock, flags);
         vgic_inject_irq(new->domain, new, irq, true);
         return true;
@@ -473,6 +477,7 @@ void arch_move_irqs(struct vcpu *v)
     struct domain *d = v->domain;
     struct pending_irq *p;
     struct vcpu *v_target;
+    unsigned long flags;
     int i;
 
     /*
@@ -494,7 +499,13 @@ void arch_move_irqs(struct vcpu *v)
         p = irq_to_pending(v_target, virq);
 
         if ( v_target == v && !test_bit(GIC_IRQ_GUEST_MIGRATING, &p->status) )
+        {
+            if ( !p->desc )
+                continue;
+            spin_lock_irqsave(&p->desc->lock, flags);
             irq_set_affinity(p->desc, cpu_mask);
+            spin_unlock_irqrestore(&p->desc->lock, flags);
+        }
     }
 }
 
@@ -574,8 +585,8 @@ void vgic_enable_irqs(struct vcpu *v, uint32_t r, unsigned int n)
         spin_unlock_irqrestore(&v_target->arch.vgic.lock, flags);
         if ( p->desc != NULL )
         {
-            irq_set_affinity(p->desc, cpumask_of(v_target->processor));
             spin_lock_irqsave(&p->desc->lock, flags);
+            irq_set_affinity(p->desc, cpumask_of(v_target->processor));
             /*
              * The irq cannot be a PPI, we only support delivery of SPIs
              * to guests.
@@ -944,4 +955,3 @@ void vgic_check_inflight_irqs_pending(struct vcpu *v, unsigned int rank, uint32_
  * indent-tabs-mode: nil
  * End:
  */
-
-- 
2.51.2

[PATCH v5 2/5] arm/irq: Migrate IRQs during CPU up/down operations

[Mykyta Poturai]

Move IRQs from dying CPU to the online ones when a CPU is getting
offlined. When onlining, rebalance all IRQs in a round-robin fashion.
Guest-bound IRQs are already handled by scheduler in the process of
moving vCPUs to active pCPUs, so we only need to handle IRQs used by Xen
itself.

Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
v4->v5:
* handle CPU onlining as well
* more comments
* fix crash when ESPI is disabled
* don't assume CPU 0 is a boot CPU
* use insigned int for irq number
* remove assumption that all irqs a bound to CPU 0 by default from the
  commit message

v3->v4:
* patch introduced
---
 xen/arch/arm/include/asm/irq.h |  2 ++
 xen/arch/arm/irq.c             | 54 ++++++++++++++++++++++++++++++++++
 xen/arch/arm/smpboot.c         |  6 ++++
 3 files changed, 62 insertions(+)

diff --git a/xen/arch/arm/include/asm/irq.h b/xen/arch/arm/include/asm/irq.h
index 09788dbfeb..a0250bac85 100644
--- a/xen/arch/arm/include/asm/irq.h
+++ b/xen/arch/arm/include/asm/irq.h
@@ -126,6 +126,8 @@ bool irq_type_set_by_domain(const struct domain *d);
 void irq_end_none(struct irq_desc *irq);
 #define irq_end_none irq_end_none
 
+void rebalance_irqs(unsigned int from, bool up);
+
 #endif /* _ASM_HW_IRQ_H */
 /*
  * Local variables:
diff --git a/xen/arch/arm/irq.c b/xen/arch/arm/irq.c
index 7204bc2b68..a32dc729f8 100644
--- a/xen/arch/arm/irq.c
+++ b/xen/arch/arm/irq.c
@@ -158,6 +158,58 @@ static int init_local_irq_data(unsigned int cpu)
     return 0;
 }
 
+static int cpu_next;
+
+static void balance_irq(int irq, unsigned int from, bool up)
+{
+    struct irq_desc *desc = irq_to_desc(irq);
+    unsigned long flags;
+
+    ASSERT(!cpumask_empty(&cpu_online_map));
+
+    spin_lock_irqsave(&desc->lock, flags);
+    if ( likely(!desc->action) )
+        goto out;
+
+    if ( likely(test_bit(_IRQ_GUEST, &desc->status) ||
+                test_bit(_IRQ_MOVE_PENDING, &desc->status)) )
+        goto out;
+
+    /*
+     * Setting affinity to a mask of multiple CPUs causes the GIC drivers to
+     * select one CPU from that mask. If the dying CPU was included in the IRQ's
+     * affinity mask, we cannot determine exactly which CPU the interrupt is
+     * currently routed to, as GIC drivers lack a concrete get_affinity API. So
+     * to be safe we must reroute it to a new, definitely online, CPU. In the
+     * case of CPU going down, we move only the interrupt that could reside on
+     * it. Otherwise, we rearrange all interrupts in a round-robin fashion.
+     */
+    if ( !up && !cpumask_test_cpu(from, desc->affinity) )
+        goto out;
+
+    cpu_next = cpumask_cycle(cpu_next, &cpu_online_map);
+    irq_set_affinity(desc, cpumask_of(cpu_next));
+
+out:
+    spin_unlock_irqrestore(&desc->lock, flags);
+}
+
+void rebalance_irqs(unsigned int from, bool up)
+{
+    int irq;
+
+    if ( cpumask_empty(&cpu_online_map) )
+        return;
+
+    for ( irq = NR_LOCAL_IRQS; irq < NR_IRQS; irq++ )
+        balance_irq(irq, from, up);
+
+#ifdef CONFIG_GICV3_ESPI
+    for ( irq = ESPI_BASE_INTID; irq < ESPI_MAX_INTID; irq++ )
+        balance_irq(irq, from, up);
+#endif
+}
+
 static int cpu_callback(struct notifier_block *nfb, unsigned long action,
                         void *hcpu)
 {
@@ -172,6 +224,8 @@ static int cpu_callback(struct notifier_block *nfb, unsigned long action,
             printk(XENLOG_ERR "Unable to allocate local IRQ for CPU%u\n",
                    cpu);
         break;
+    case CPU_ONLINE:
+        rebalance_irqs(cpu, true);
     }
 
     return notifier_from_errno(rc);
diff --git a/xen/arch/arm/smpboot.c b/xen/arch/arm/smpboot.c
index 7f3cfa812e..e1b9f94458 100644
--- a/xen/arch/arm/smpboot.c
+++ b/xen/arch/arm/smpboot.c
@@ -425,6 +425,12 @@ void __cpu_disable(void)
 
     smp_mb();
 
+    /*
+     * Now that the interrupts are cleared and the CPU marked as offline,
+     * move interrupts out of it
+     */
+    rebalance_irqs(cpu, false);
+
     /* Return to caller; eventually the IPI mechanism will unwind and the 
      * scheduler will drop to the idle loop, which will call stop_cpu(). */
 }
-- 
2.51.2

[PATCH v5 3/5] arm/sysctl: Implement cpu hotplug ops

[Mykyta Poturai]

Move XEN_SYSCTL_CPU_HOTPLUG_{ONLINE,OFFLINE} handlers to common code to
allow for enabling/disabling CPU cores in runtime on Arm64.

SMT-disable enforcement check is moved into a separate
architecture-specific function.

For now this operations only support Arm64. For proper Arm32 support,
there needs to be a mechanism to free per-cpu page tables, allocated in
init_domheap_mappings.  Also, hotplug is not supported if ITS, FFA, or
TEE is enabled, as they use non-static IRQ actions.

Create a Kconfig option CPU_HOTPLUG that reflects this
constraints. On X86 the option is enabled unconditionally.

Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>

---
v4->v5:
* move handling to common code
* rename config to CPU_HOTPUG
* merge with "smp: Move cpu_up/down helpers to common code"

v3->v4:
* don't reimplement cpu_up/down helpers
* add Kconfig option
* fixup formatting

v2->v3:
* no changes

v1->v2:
* remove SMT ops
* remove cpu == 0 checks
* add XSM hooks
* only implement for 64bit Arm
---
 xen/arch/arm/Kconfig           |  1 +
 xen/arch/arm/smp.c             |  9 +++++++
 xen/arch/ppc/stubs.c           |  4 +++
 xen/arch/riscv/stubs.c         |  5 ++++
 xen/arch/x86/Kconfig           |  1 +
 xen/arch/x86/include/asm/smp.h |  3 ---
 xen/arch/x86/smp.c             | 33 +++----------------------
 xen/arch/x86/sysctl.c          | 12 +++------
 xen/common/Kconfig             |  3 +++
 xen/common/smp.c               | 34 +++++++++++++++++++++++++
 xen/common/sysctl.c            | 45 ++++++++++++++++++++++++++++++++++
 xen/include/xen/smp.h          |  4 +++
 12 files changed, 112 insertions(+), 42 deletions(-)

diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
index cf6af68299..5144e9c8d5 100644
--- a/xen/arch/arm/Kconfig
+++ b/xen/arch/arm/Kconfig
@@ -7,6 +7,7 @@ config ARM_64
 	def_bool y
 	depends on !ARM_32
 	select 64BIT
+	select CPU_HOTPLUG if !TEE && !FFA && !HAS_ITS
 	select HAS_FAST_MULTIPLY
 	select HAS_VPCI_GUEST_SUPPORT if PCI_PASSTHROUGH
 
diff --git a/xen/arch/arm/smp.c b/xen/arch/arm/smp.c
index b372472188..075da9aeb3 100644
--- a/xen/arch/arm/smp.c
+++ b/xen/arch/arm/smp.c
@@ -44,6 +44,15 @@ void smp_send_call_function_mask(const cpumask_t *mask)
     }
 }
 
+/*
+ * We currently don't support SMT on ARM so we don't need any special logic for
+ * CPU disabling
+ */
+bool arch_smt_cpu_disable(unsigned int cpu)
+{
+    return false;
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/arch/ppc/stubs.c b/xen/arch/ppc/stubs.c
index f7f6e7ed97..ed75d06dd9 100644
--- a/xen/arch/ppc/stubs.c
+++ b/xen/arch/ppc/stubs.c
@@ -101,6 +101,10 @@ void smp_send_call_function_mask(const cpumask_t *mask)
     BUG_ON("unimplemented");
 }
 
+bool arch_smt_cpu_disable(unsigned int cpu)
+{
+    BUG_ON("unimplemented");
+}
 /* irq.c */
 
 void irq_ack_none(struct irq_desc *desc)
diff --git a/xen/arch/riscv/stubs.c b/xen/arch/riscv/stubs.c
index 29bdb65afb..8a9503ec94 100644
--- a/xen/arch/riscv/stubs.c
+++ b/xen/arch/riscv/stubs.c
@@ -75,6 +75,11 @@ void smp_send_call_function_mask(const cpumask_t *mask)
     BUG_ON("unimplemented");
 }
 
+bool arch_smt_cpu_disable(unsigned int cpu)
+{
+    BUG_ON("unimplemented");
+}
+
 /* irq.c */
 
 void irq_ack_none(struct irq_desc *desc)
diff --git a/xen/arch/x86/Kconfig b/xen/arch/x86/Kconfig
index c808c989fc..826aa2512d 100644
--- a/xen/arch/x86/Kconfig
+++ b/xen/arch/x86/Kconfig
@@ -12,6 +12,7 @@ config X86
 	select ARCH_PAGING_MEMPOOL
 	select ARCH_SUPPORTS_INT128
 	imply CORE_PARKING
+	select CPU_HOTPLUG
 	select FUNCTION_ALIGNMENT_16B
 	select GENERIC_BUG_FRAME
 	select HAS_ALTERNATIVE
diff --git a/xen/arch/x86/include/asm/smp.h b/xen/arch/x86/include/asm/smp.h
index 3f16e62696..cb3e0fed19 100644
--- a/xen/arch/x86/include/asm/smp.h
+++ b/xen/arch/x86/include/asm/smp.h
@@ -50,9 +50,6 @@ int cpu_add(uint32_t apic_id, uint32_t acpi_id, uint32_t pxm);
 
 void __stop_this_cpu(void);
 
-long cf_check cpu_up_helper(void *data);
-long cf_check cpu_down_helper(void *data);
-
 long cf_check core_parking_helper(void *data);
 bool core_parking_remove(unsigned int cpu);
 uint32_t get_cur_idle_nums(void);
diff --git a/xen/arch/x86/smp.c b/xen/arch/x86/smp.c
index 7936294f5f..d64b533cc0 100644
--- a/xen/arch/x86/smp.c
+++ b/xen/arch/x86/smp.c
@@ -418,35 +418,8 @@ void cf_check call_function_interrupt(void)
     smp_call_function_interrupt();
 }
 
-long cf_check cpu_up_helper(void *data)
+bool arch_smt_cpu_disable(unsigned int cpu)
 {
-    unsigned int cpu = (unsigned long)data;
-    int ret = cpu_up(cpu);
-
-    /* Have one more go on EBUSY. */
-    if ( ret == -EBUSY )
-        ret = cpu_up(cpu);
-
-    if ( !ret && !opt_smt &&
-         cpu_data[cpu].compute_unit_id == INVALID_CUID &&
-         cpumask_weight(per_cpu(cpu_sibling_mask, cpu)) > 1 )
-    {
-        ret = cpu_down_helper(data);
-        if ( ret )
-            printk("Could not re-offline CPU%u (%d)\n", cpu, ret);
-        else
-            ret = -EPERM;
-    }
-
-    return ret;
-}
-
-long cf_check cpu_down_helper(void *data)
-{
-    int cpu = (unsigned long)data;
-    int ret = cpu_down(cpu);
-    /* Have one more go on EBUSY. */
-    if ( ret == -EBUSY )
-        ret = cpu_down(cpu);
-    return ret;
+    return !opt_smt && cpu_data[cpu].compute_unit_id == INVALID_CUID &&
+           cpumask_weight(per_cpu(cpu_sibling_mask, cpu)) > 1;
 }
diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
index 1b04947516..87a4b7ac63 100644
--- a/xen/arch/x86/sysctl.c
+++ b/xen/arch/x86/sysctl.c
@@ -115,7 +115,6 @@ long arch_do_sysctl(
 
     case XEN_SYSCTL_cpu_hotplug:
     {
-        unsigned int cpu = sysctl->u.cpu_hotplug.cpu;
         unsigned int op  = sysctl->u.cpu_hotplug.op;
         bool plug;
         long (*fn)(void *data);
@@ -124,15 +123,10 @@ long arch_do_sysctl(
         switch ( op )
         {
         case XEN_SYSCTL_CPU_HOTPLUG_ONLINE:
-            plug = true;
-            fn = cpu_up_helper;
-            hcpu = _p(cpu);
-            break;
-
         case XEN_SYSCTL_CPU_HOTPLUG_OFFLINE:
-            plug = false;
-            fn = cpu_down_helper;
-            hcpu = _p(cpu);
+            /* Handled by common code */
+            ASSERT_UNREACHABLE();
+            ret = -EOPNOTSUPP;
             break;
 
         case XEN_SYSCTL_CPU_HOTPLUG_SMT_ENABLE:
diff --git a/xen/common/Kconfig b/xen/common/Kconfig
index 38320b248a..1a28c2dafe 100644
--- a/xen/common/Kconfig
+++ b/xen/common/Kconfig
@@ -176,6 +176,9 @@ config LIBFDT
 config MEM_ACCESS_ALWAYS_ON
 	bool
 
+config CPU_HOTPLUG
+    bool
+
 config VM_EVENT
 	def_bool MEM_ACCESS_ALWAYS_ON
 	prompt "Memory Access and VM events" if !MEM_ACCESS_ALWAYS_ON
diff --git a/xen/common/smp.c b/xen/common/smp.c
index a011f541f1..8ff81197cb 100644
--- a/xen/common/smp.c
+++ b/xen/common/smp.c
@@ -16,6 +16,7 @@
  * GNU General Public License for more details.
  */
 
+#include <xen/cpu.h>
 #include <asm/hardirq.h>
 #include <asm/processor.h>
 #include <xen/spinlock.h>
@@ -104,6 +105,39 @@ void smp_call_function_interrupt(void)
     irq_exit();
 }
 
+#ifdef CONFIG_CPU_HOTPLUG
+long cf_check cpu_up_helper(void *data)
+{
+    unsigned int cpu = (unsigned long)data;
+    int ret = cpu_up(cpu);
+
+    /* Have one more go on EBUSY. */
+    if ( ret == -EBUSY )
+        ret = cpu_up(cpu);
+
+    if ( !ret && arch_smt_cpu_disable(cpu) )
+    {
+        ret = cpu_down_helper(data);
+        if ( ret )
+            printk("Could not re-offline CPU%u (%d)\n", cpu, ret);
+        else
+            ret = -EPERM;
+    }
+
+    return ret;
+}
+
+long cf_check cpu_down_helper(void *data)
+{
+    int cpu = (unsigned long)data;
+    int ret = cpu_down(cpu);
+    /* Have one more go on EBUSY. */
+    if ( ret == -EBUSY )
+        ret = cpu_down(cpu);
+    return ret;
+}
+#endif /* CONFIG_CPU_HOTPLUG */
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/common/sysctl.c b/xen/common/sysctl.c
index 5207664252..2acf47723d 100644
--- a/xen/common/sysctl.c
+++ b/xen/common/sysctl.c
@@ -483,6 +483,51 @@ long do_sysctl(XEN_GUEST_HANDLE_PARAM(xen_sysctl_t) u_sysctl)
             copyback = 1;
         break;
 
+#ifdef CONFIG_CPU_HOTPLUG
+    case XEN_SYSCTL_cpu_hotplug:
+    {
+        unsigned int cpu = op->u.cpu_hotplug.cpu;
+        unsigned int hp_op = op->u.cpu_hotplug.op;
+        bool plug;
+        long (*fn)(void *data);
+        void *hcpu;
+
+        switch ( hp_op )
+        {
+        case XEN_SYSCTL_CPU_HOTPLUG_ONLINE:
+            plug = true;
+            fn = cpu_up_helper;
+            hcpu = _p(cpu);
+            break;
+
+        case XEN_SYSCTL_CPU_HOTPLUG_OFFLINE:
+            plug = false;
+            fn = cpu_down_helper;
+            hcpu = _p(cpu);
+            break;
+
+        case XEN_SYSCTL_CPU_HOTPLUG_SMT_ENABLE:
+        case XEN_SYSCTL_CPU_HOTPLUG_SMT_DISABLE:
+            /* Use arch specific handlers as SMT is very arch-dependent */
+            ret = arch_do_sysctl(op, u_sysctl);
+            copyback = 0;
+            goto out;
+
+        default:
+            ret = -EOPNOTSUPP;
+            break;
+        }
+
+        if ( !ret )
+            ret = plug ? xsm_resource_plug_core(XSM_HOOK)
+                       : xsm_resource_unplug_core(XSM_HOOK);
+
+        if ( !ret )
+            ret = continue_hypercall_on_cpu(0, fn, hcpu);
+        break;
+    }
+#endif
+
     default:
         ret = arch_do_sysctl(op, u_sysctl);
         copyback = 0;
diff --git a/xen/include/xen/smp.h b/xen/include/xen/smp.h
index 2ca9ff1bfc..c734033bfb 100644
--- a/xen/include/xen/smp.h
+++ b/xen/include/xen/smp.h
@@ -76,4 +76,8 @@ extern void *stack_base[NR_CPUS];
 void initialize_cpu_data(unsigned int cpu);
 int setup_cpu_root_pgt(unsigned int cpu);
 
+bool arch_smt_cpu_disable(unsigned int cpu);
+long cf_check cpu_up_helper(void *data);
+long cf_check cpu_down_helper(void *data);
+
 #endif /* __XEN_SMP_H__ */
-- 
2.51.2

[PATCH v5 4/5] tools: Allow building xen-hptool without CONFIG_MIGRATE

[Mykyta Poturai]

With CPU hotplug sysctls implemented on Arm it becomes useful to have a
tool for calling them.

According to the commit history it seems that putting hptool under
config MIGRATE was a measure to fix IA64 build. As IA64 is no longer
supported it can now be brought back. So build it unconditionally.

Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
v4->v5:
* make hptool always build

v3->v4:
* no changes

v2->v3:
* no changes

v1->v2:
* switch to configure from legacy config
---
 tools/libs/guest/Makefile.common | 2 +-
 tools/misc/Makefile              | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools/libs/guest/Makefile.common b/tools/libs/guest/Makefile.common
index a026a2f662..db268da68e 100644
--- a/tools/libs/guest/Makefile.common
+++ b/tools/libs/guest/Makefile.common
@@ -7,6 +7,7 @@ OBJS-y += xg_private.o
 OBJS-y += xg_domain.o
 OBJS-y += xg_suspend.o
 OBJS-y += xg_resume.o
+OBJS-y += xg_offline_page.o
 ifeq ($(CONFIG_MIGRATE),y)
 OBJS-y += xg_sr_common.o
 OBJS-$(CONFIG_X86) += xg_sr_common_x86.o
@@ -17,7 +18,6 @@ OBJS-$(CONFIG_X86) += xg_sr_save_x86_pv.o
 OBJS-$(CONFIG_X86) += xg_sr_save_x86_hvm.o
 OBJS-y += xg_sr_restore.o
 OBJS-y += xg_sr_save.o
-OBJS-y += xg_offline_page.o
 else
 OBJS-y += xg_nomigrate.o
 endif
diff --git a/tools/misc/Makefile b/tools/misc/Makefile
index c26e544e83..6cd8785cf1 100644
--- a/tools/misc/Makefile
+++ b/tools/misc/Makefile
@@ -16,7 +16,6 @@ INSTALL_BIN                    += xencov_split
 INSTALL_BIN += $(INSTALL_BIN-y)
 
 # Everything to be installed in regular sbin/
-INSTALL_SBIN-$(CONFIG_MIGRATE) += xen-hptool
 INSTALL_SBIN-$(CONFIG_X86)     += xen-hvmcrash
 INSTALL_SBIN-$(CONFIG_X86)     += xen-hvmctx
 INSTALL_SBIN-$(CONFIG_X86)     += xen-lowmemd
@@ -34,6 +33,7 @@ INSTALL_SBIN                   += xenwatchdogd
 INSTALL_SBIN                   += xen-access
 INSTALL_SBIN                   += xen-livepatch
 INSTALL_SBIN                   += xen-diag
+INSTALL_SBIN                   += xen-hptool
 INSTALL_SBIN += $(INSTALL_SBIN-y)
 
 # Everything to be installed
-- 
2.51.2

[PATCH v5 5/5] docs: Document CPU hotplug

[Mykyta Poturai]

Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---

v4->v5:
* s/supported/implemented/
* update SUPPORT.md

v3->v4:
* update configuration section

v2->v3:
* patch introduced
---
 SUPPORT.md                |  1 +
 docs/misc/cpu-hotplug.txt | 50 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 51 insertions(+)
 create mode 100644 docs/misc/cpu-hotplug.txt

diff --git a/SUPPORT.md b/SUPPORT.md
index d441bccf37..7b93ae69e7 100644
--- a/SUPPORT.md
+++ b/SUPPORT.md
@@ -52,6 +52,7 @@ For the Cortex A77 r0p0 - r1p0, see Errata 1508412.
 ### ACPI CPU Hotplug
 
     Status, x86: Experimental
+    Status, Arm64: Experimental
 
 ### Physical Memory
 
diff --git a/docs/misc/cpu-hotplug.txt b/docs/misc/cpu-hotplug.txt
new file mode 100644
index 0000000000..c34fc66361
--- /dev/null
+++ b/docs/misc/cpu-hotplug.txt
@@ -0,0 +1,50 @@
+CPU Hotplug
+===========
+
+CPU hotplug is a feature that allows pCPU cores to be added to or removed from a
+running system without requiring a reboot. It is implemented on x86 and Arm64
+architectures.
+
+Implementation Details
+----------------------
+
+CPU hotplug is implemented through the `XEN_SYSCTL_CPU_HOTPLUG_*` sysctl calls.
+The specific calls are:
+
+- `XEN_SYSCTL_CPU_HOTPLUG_ONLINE`: Brings a pCPU online
+- `XEN_SYSCTL_CPU_HOTPLUG_OFFLINE`: Takes a pCPU offline
+- `XEN_SYSCTL_CPU_HOTPLUG_SMT_ENABLE`: Enables SMT threads (x86 only)
+- `XEN_SYSCTL_CPU_HOTPLUG_SMT_DISABLE`: Disables SMT threads (x86 only)
+
+All cores can be disabled, assuming hardware support, except for the boot core.
+Sysctl calls are routed to the boot core before doing any actual up/down
+operations on other cores.
+
+Configuration
+-------------
+
+The presence of the feature is controlled by CONFIG_CPU_HOTPLUG option. It is
+enabled unconditionally on x86 architecture. On Arm64, the option is enabled by
+default when ITS, FFA, and TEE configs are disabled.
+xen-hptool userspace tool is built unconditionally.
+
+Usage
+-----
+
+Disable core:
+
+$ xen-hptool cpu-offline 2
+Prepare to offline CPU 2
+(XEN) Removing cpu 2 from runqueue 0
+CPU 2 offlined successfully
+
+Enable core:
+
+$ xen-hptool cpu-online 2
+Prepare to online CPU 2
+(XEN) Bringing up CPU2
+(XEN) GICv3: CPU2: Found redistributor in region 0 @00000a004005c000
+(XEN) CPU2: Guest atomics will try 1 times before pausing the domain
+(XEN) CPU 2 booted.
+(XEN) Adding cpu 2 to runqueue 0
+CPU 2 onlined successfully
-- 
2.51.2

Re: [PATCH v5 4/5] tools: Allow building xen-hptool without CONFIG_MIGRATE

[Jan Beulich]

On 13.01.2026 09:45, Mykyta Poturai wrote:
> --- a/tools/misc/Makefile
> +++ b/tools/misc/Makefile
> @@ -16,7 +16,6 @@ INSTALL_BIN                    += xencov_split
>  INSTALL_BIN += $(INSTALL_BIN-y)
>  
>  # Everything to be installed in regular sbin/
> -INSTALL_SBIN-$(CONFIG_MIGRATE) += xen-hptool
>  INSTALL_SBIN-$(CONFIG_X86)     += xen-hvmcrash
>  INSTALL_SBIN-$(CONFIG_X86)     += xen-hvmctx
>  INSTALL_SBIN-$(CONFIG_X86)     += xen-lowmemd
> @@ -34,6 +33,7 @@ INSTALL_SBIN                   += xenwatchdogd
>  INSTALL_SBIN                   += xen-access
>  INSTALL_SBIN                   += xen-livepatch
>  INSTALL_SBIN                   += xen-diag
> +INSTALL_SBIN                   += xen-hptool
>  INSTALL_SBIN += $(INSTALL_SBIN-y)

As per [1] I think the line should be edited in place.

Jan

[1] https://lists.xen.org/archives/html/xen-devel/2026-01/msg00285.html

Re: [PATCH v5 3/5] arm/sysctl: Implement cpu hotplug ops

[Jan Beulich]

On 13.01.2026 09:45, Mykyta Poturai wrote:
> Move XEN_SYSCTL_CPU_HOTPLUG_{ONLINE,OFFLINE} handlers to common code to
> allow for enabling/disabling CPU cores in runtime on Arm64.
> 
> SMT-disable enforcement check is moved into a separate
> architecture-specific function.
> 
> For now this operations only support Arm64. For proper Arm32 support,
> there needs to be a mechanism to free per-cpu page tables, allocated in
> init_domheap_mappings.  Also, hotplug is not supported if ITS, FFA, or
> TEE is enabled, as they use non-static IRQ actions.

For all of these "not supported" cases, what if a user nevertheless tries?
Wouldn't the request better be outright denied, rather leaving the system in
a questionable state? Hmm, I see you ...

> --- a/xen/arch/arm/Kconfig
> +++ b/xen/arch/arm/Kconfig
> @@ -7,6 +7,7 @@ config ARM_64
>  	def_bool y
>  	depends on !ARM_32
>  	select 64BIT
> +	select CPU_HOTPLUG if !TEE && !FFA && !HAS_ITS

... make the select conditional. But do TEE, FFA, and HAS_ITS each mean the
feature is actually in use when the hypervisor runs?

> --- a/xen/common/Kconfig
> +++ b/xen/common/Kconfig
> @@ -176,6 +176,9 @@ config LIBFDT
>  config MEM_ACCESS_ALWAYS_ON
>  	bool
>  
> +config CPU_HOTPLUG
> +    bool

Nit: Indentation by a single tab please. See adjacent entries.

> @@ -104,6 +105,39 @@ void smp_call_function_interrupt(void)
>      irq_exit();
>  }
>  
> +#ifdef CONFIG_CPU_HOTPLUG
> +long cf_check cpu_up_helper(void *data)
> +{
> +    unsigned int cpu = (unsigned long)data;

Note this for the first comment below on cpu_down_helper().

> +    int ret = cpu_up(cpu);
> +
> +    /* Have one more go on EBUSY. */
> +    if ( ret == -EBUSY )
> +        ret = cpu_up(cpu);
> +
> +    if ( !ret && arch_smt_cpu_disable(cpu) )

As you validly note in a comment in do_sysctl(), SMT is an arch-specific concept
and perhaps even an arch-specific term. Hence using it in the name of an arch
hook feels inappropriate. Plus - the hook could be used for other purposes. What
the arch needs to indicate is whether the CPU that was brought up may actually
stay online. That more generic purpose is what imo the name wants to cover.

> +    {
> +        ret = cpu_down_helper(data);
> +        if ( ret )
> +            printk("Could not re-offline CPU%u (%d)\n", cpu, ret);
> +        else
> +            ret = -EPERM;
> +    }
> +
> +    return ret;
> +}
> +
> +long cf_check cpu_down_helper(void *data)
> +{
> +    int cpu = (unsigned long)data;

Why is this left as plain int? Yes, it was like this in the original code,
but wrongly so.

> +    int ret = cpu_down(cpu);
> +    /* Have one more go on EBUSY. */

Also please add the missing blank line after the declarations.

> --- a/xen/common/sysctl.c
> +++ b/xen/common/sysctl.c
> @@ -483,6 +483,51 @@ long do_sysctl(XEN_GUEST_HANDLE_PARAM(xen_sysctl_t) u_sysctl)
>              copyback = 1;
>          break;
>  
> +#ifdef CONFIG_CPU_HOTPLUG
> +    case XEN_SYSCTL_cpu_hotplug:

Please see the pretty recent
https://lists.xen.org/archives/html/xen-devel/2026-01/msg00329.html
(scroll down to the xen/arch/x86/platform_hypercall.c change).

> +    {
> +        unsigned int cpu = op->u.cpu_hotplug.cpu;
> +        unsigned int hp_op = op->u.cpu_hotplug.op;
> +        bool plug;
> +        long (*fn)(void *data);
> +        void *hcpu;
> +
> +        switch ( hp_op )
> +        {
> +        case XEN_SYSCTL_CPU_HOTPLUG_ONLINE:
> +            plug = true;
> +            fn = cpu_up_helper;
> +            hcpu = _p(cpu);
> +            break;
> +
> +        case XEN_SYSCTL_CPU_HOTPLUG_OFFLINE:
> +            plug = false;
> +            fn = cpu_down_helper;
> +            hcpu = _p(cpu);
> +            break;
> +
> +        case XEN_SYSCTL_CPU_HOTPLUG_SMT_ENABLE:
> +        case XEN_SYSCTL_CPU_HOTPLUG_SMT_DISABLE:
> +            /* Use arch specific handlers as SMT is very arch-dependent */
> +            ret = arch_do_sysctl(op, u_sysctl);
> +            copyback = 0;
> +            goto out;

I wonder if it wouldn't be neater for this and actually also ...

> +        default:
> +            ret = -EOPNOTSUPP;
> +            break;

... this to fall through to ...

> +        }
> +
> +        if ( !ret )
> +            ret = plug ? xsm_resource_plug_core(XSM_HOOK)
> +                       : xsm_resource_unplug_core(XSM_HOOK);
> +
> +        if ( !ret )
> +            ret = continue_hypercall_on_cpu(0, fn, hcpu);
> +        break;
> +    }
> +#endif
> +
>      default:
>          ret = arch_do_sysctl(op, u_sysctl);

... here. (Minimally the earlier default case wants uniformly forwarding to
the arch handler, or else arch-specific additions would always require
adjustment here.)

Jan

Re: [PATCH v5 3/5] arm/sysctl: Implement cpu hotplug ops

[Mykyta Poturai]

On 14.01.26 11:49, Jan Beulich wrote:
> On 13.01.2026 09:45, Mykyta Poturai wrote:
>> Move XEN_SYSCTL_CPU_HOTPLUG_{ONLINE,OFFLINE} handlers to common code to
>> allow for enabling/disabling CPU cores in runtime on Arm64.
>>
>> SMT-disable enforcement check is moved into a separate
>> architecture-specific function.
>>
>> For now this operations only support Arm64. For proper Arm32 support,
>> there needs to be a mechanism to free per-cpu page tables, allocated in
>> init_domheap_mappings.  Also, hotplug is not supported if ITS, FFA, or
>> TEE is enabled, as they use non-static IRQ actions.
> 
> For all of these "not supported" cases, what if a user nevertheless tries?
> Wouldn't the request better be outright denied, rather leaving the system in
> a questionable state? Hmm, I see you ...
> 
>> --- a/xen/arch/arm/Kconfig
>> +++ b/xen/arch/arm/Kconfig
>> @@ -7,6 +7,7 @@ config ARM_64
>>   	def_bool y
>>   	depends on !ARM_32
>>   	select 64BIT
>> +	select CPU_HOTPLUG if !TEE && !FFA && !HAS_ITS
> 
> ... make the select conditional. But do TEE, FFA, and HAS_ITS each mean the
> feature is actually in use when the hypervisor runs?
> 
The way interrupts are requested in these modules causes Xen to crash 
when trying to offline a cpu. It’s a fairly simple fix and I plan to 
send them eventually. I’ve decided to omit them now and do these fixes 
only for supported code to keep the series from ballooning with too many 
patches.

>> +    int ret = cpu_up(cpu);
>> +
>> +    /* Have one more go on EBUSY. */
>> +    if ( ret == -EBUSY )
>> +        ret = cpu_up(cpu);
>> +
>> +    if ( !ret && arch_smt_cpu_disable(cpu) )
> 
> As you validly note in a comment in do_sysctl(), SMT is an arch-specific concept
> and perhaps even an arch-specific term. Hence using it in the name of an arch
> hook feels inappropriate. Plus - the hook could be used for other purposes. What
> the arch needs to indicate is whether the CPU that was brought up may actually
> stay online. That more generic purpose is what imo the name wants to cover.
> 

Would arch_cpu_online_allowed() be okay, or maybe you have something 
more specific in mind?


>> +        case XEN_SYSCTL_CPU_HOTPLUG_SMT_ENABLE:
>> +        case XEN_SYSCTL_CPU_HOTPLUG_SMT_DISABLE:
>> +            /* Use arch specific handlers as SMT is very arch-dependent */
>> +            ret = arch_do_sysctl(op, u_sysctl);
>> +            copyback = 0;
>> +            goto out;
> 
> I wonder if it wouldn't be neater for this and actually also ...
> 
>> +        default:
>> +            ret = -EOPNOTSUPP;
>> +            break;
> 
> ... this to fall through to ...
> 
>> +        }
>> +
>> +        if ( !ret )
>> +            ret = plug ? xsm_resource_plug_core(XSM_HOOK)
>> +                       : xsm_resource_unplug_core(XSM_HOOK);
>> +
>> +        if ( !ret )
>> +            ret = continue_hypercall_on_cpu(0, fn, hcpu);
>> +        break;
>> +    }
>> +#endif
>> +
>>       default:
>>           ret = arch_do_sysctl(op, u_sysctl);
> 
> ... here. (Minimally the earlier default case wants uniformly forwarding to
> the arch handler, or else arch-specific additions would always require
> adjustment here.)
> 
> Jan

Would it be okay to mix goto and switch like this, or is it too convoluted?

         case XEN_SYSCTL_CPU_HOTPLUG_OFFLINE:
             plug = false;
             fn = cpu_down_helper;
             hcpu = _p(cpu);
             break;

         default:
             goto outer_default;
         }

         if ( !ret )
             ret = plug ? xsm_resource_plug_core(XSM_HOOK)
                        : xsm_resource_unplug_core(XSM_HOOK);

         if ( !ret )
             ret = continue_hypercall_on_cpu(0, fn, hcpu);
         break;
     }
#endif

     default:
outer_default:
         ret = arch_do_sysctl(op, u_sysctl);
         copyback = 0;
         break;
     }



-- 
Mykyta

Re: [PATCH v5 3/5] arm/sysctl: Implement cpu hotplug ops

[Jan Beulich]

On 03.02.2026 11:30, Mykyta Poturai wrote:
> On 14.01.26 11:49, Jan Beulich wrote:
>> On 13.01.2026 09:45, Mykyta Poturai wrote:
>>> Move XEN_SYSCTL_CPU_HOTPLUG_{ONLINE,OFFLINE} handlers to common code to
>>> allow for enabling/disabling CPU cores in runtime on Arm64.
>>>
>>> SMT-disable enforcement check is moved into a separate
>>> architecture-specific function.
>>>
>>> For now this operations only support Arm64. For proper Arm32 support,
>>> there needs to be a mechanism to free per-cpu page tables, allocated in
>>> init_domheap_mappings.  Also, hotplug is not supported if ITS, FFA, or
>>> TEE is enabled, as they use non-static IRQ actions.
>>
>> For all of these "not supported" cases, what if a user nevertheless tries?
>> Wouldn't the request better be outright denied, rather leaving the system in
>> a questionable state? Hmm, I see you ...
>>
>>> --- a/xen/arch/arm/Kconfig
>>> +++ b/xen/arch/arm/Kconfig
>>> @@ -7,6 +7,7 @@ config ARM_64
>>>   	def_bool y
>>>   	depends on !ARM_32
>>>   	select 64BIT
>>> +	select CPU_HOTPLUG if !TEE && !FFA && !HAS_ITS
>>
>> ... make the select conditional. But do TEE, FFA, and HAS_ITS each mean the
>> feature is actually in use when the hypervisor runs?
>>
> The way interrupts are requested in these modules causes Xen to crash 
> when trying to offline a cpu. It’s a fairly simple fix and I plan to 
> send them eventually. I’ve decided to omit them now and do these fixes 
> only for supported code to keep the series from ballooning with too many 
> patches.

I disagree with such an approach, but it'll be the Arm maintainers to judge here.

>>> +    int ret = cpu_up(cpu);
>>> +
>>> +    /* Have one more go on EBUSY. */
>>> +    if ( ret == -EBUSY )
>>> +        ret = cpu_up(cpu);
>>> +
>>> +    if ( !ret && arch_smt_cpu_disable(cpu) )
>>
>> As you validly note in a comment in do_sysctl(), SMT is an arch-specific concept
>> and perhaps even an arch-specific term. Hence using it in the name of an arch
>> hook feels inappropriate. Plus - the hook could be used for other purposes. What
>> the arch needs to indicate is whether the CPU that was brought up may actually
>> stay online. That more generic purpose is what imo the name wants to cover.
> 
> Would arch_cpu_online_allowed() be okay, or maybe you have something 
> more specific in mind?

The name is already much better, just that it gives the impression that it perhaps
rather would want using ahead of calling cpu_up().

>>> +        case XEN_SYSCTL_CPU_HOTPLUG_SMT_ENABLE:
>>> +        case XEN_SYSCTL_CPU_HOTPLUG_SMT_DISABLE:
>>> +            /* Use arch specific handlers as SMT is very arch-dependent */
>>> +            ret = arch_do_sysctl(op, u_sysctl);
>>> +            copyback = 0;
>>> +            goto out;
>>
>> I wonder if it wouldn't be neater for this and actually also ...
>>
>>> +        default:
>>> +            ret = -EOPNOTSUPP;
>>> +            break;
>>
>> ... this to fall through to ...
>>
>>> +        }
>>> +
>>> +        if ( !ret )
>>> +            ret = plug ? xsm_resource_plug_core(XSM_HOOK)
>>> +                       : xsm_resource_unplug_core(XSM_HOOK);
>>> +
>>> +        if ( !ret )
>>> +            ret = continue_hypercall_on_cpu(0, fn, hcpu);
>>> +        break;
>>> +    }
>>> +#endif
>>> +
>>>       default:
>>>           ret = arch_do_sysctl(op, u_sysctl);
>>
>> ... here. (Minimally the earlier default case wants uniformly forwarding to
>> the arch handler, or else arch-specific additions would always require
>> adjustment here.)
> 
> Would it be okay to mix goto and switch like this, or is it too convoluted?

I'm not a fan of using goto, but maybe it would be acceptable here. By ...

>          case XEN_SYSCTL_CPU_HOTPLUG_OFFLINE:
>              plug = false;
>              fn = cpu_down_helper;
>              hcpu = _p(cpu);
>              break;
> 
>          default:
>              goto outer_default;
>          }
> 
>          if ( !ret )
>              ret = plug ? xsm_resource_plug_core(XSM_HOOK)
>                         : xsm_resource_unplug_core(XSM_HOOK);
> 
>          if ( !ret )
>              ret = continue_hypercall_on_cpu(0, fn, hcpu);
>          break;

... wrapping everything past the switch() block in "if ( fn )" you'd already
get what is wanted.

>      }
> #endif
> 
>      default:
> outer_default:

Nit: See ./CODING_STYLE.

Jan

>          ret = arch_do_sysctl(op, u_sysctl);
>          copyback = 0;
>          break;
>      }
> 
> 
> 

Re: [PATCH v5 0/5] Implement CPU hotplug on Arm

[Bertrand Marquis]

Hi Mykyta,

> On 13 Jan 2026, at 09:45, Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
> 
> This series implements support for CPU hotplug/unplug on Arm. To achieve this,
> several things need to be done:
> 
> 1. XEN_SYSCTL_CPU_HOTPLUG_* calls implemented on Arm64.
> 2. Enabled building of xen-hptool.
> 3. Migration of irqs from dying CPUs implemented.
> 
> Tested on QEMU.

Do you have a way to validate this on real hardware ?
Doing hotplug/unplug might have side effects (caches, coherency, interrupts, etc) that
I would not be completely confident to have validated by only testing this in qemu.

Could you explain how you validated this (what kind of tests you did).

I would be interested to know how the system behave in the following cases:
- stopping cpu with guests pinned to specific cores
- stopping a cpu with a guest pinned and only running on that one
- checking with passthrough devices (for irq balancing)

Then a general remark i have is that most of this code (if not all) is always in Xen
even if CONFIG_CPU_HOTPLUG is not activated in xen config which will probably
generate some dead code. Could we protect some of the code not used when
HOTPLUG is not activated with if (is_enabled(CONFIG_CPU_HOTPLUG)) or
equivalent forms ?

Cheers
Bertrand

> 
> v4->v5:
> * drop merged patches
> * combine "smp: Move cpu_up/down helpers to common code" with 
>  "arm/sysctl: Implement cpu hotplug ops"
> * see individual patches
> 
> v3->v4:
> * add irq migration patches
> * see individual patches
> 
> v2->v3:
> * add docs
> 
> v1->v2:
> * see individual patches
> 
> Mykyta Poturai (5):
>  arm/irq: Keep track of irq affinities
>  arm/irq: Migrate IRQs during CPU up/down operations
>  arm/sysctl: Implement cpu hotplug ops
>  tools: Allow building xen-hptool without CONFIG_MIGRATE
>  docs: Document CPU hotplug
> 
> SUPPORT.md                       |  1 +
> docs/misc/cpu-hotplug.txt        | 50 +++++++++++++++++++++++++
> tools/libs/guest/Makefile.common |  2 +-
> tools/misc/Makefile              |  2 +-
> xen/arch/arm/Kconfig             |  1 +
> xen/arch/arm/gic-vgic.c          |  2 +
> xen/arch/arm/include/asm/irq.h   |  2 +
> xen/arch/arm/irq.c               | 63 +++++++++++++++++++++++++++++++-
> xen/arch/arm/smp.c               |  9 +++++
> xen/arch/arm/smpboot.c           |  6 +++
> xen/arch/arm/vgic.c              | 14 ++++++-
> xen/arch/ppc/stubs.c             |  4 ++
> xen/arch/riscv/stubs.c           |  5 +++
> xen/arch/x86/Kconfig             |  1 +
> xen/arch/x86/include/asm/smp.h   |  3 --
> xen/arch/x86/smp.c               | 33 ++---------------
> xen/arch/x86/sysctl.c            | 12 ++----
> xen/common/Kconfig               |  3 ++
> xen/common/smp.c                 | 34 +++++++++++++++++
> xen/common/sysctl.c              | 45 +++++++++++++++++++++++
> xen/include/xen/smp.h            |  4 ++
> 21 files changed, 248 insertions(+), 48 deletions(-)
> create mode 100644 docs/misc/cpu-hotplug.txt
> 
> -- 
> 2.51.2

Re: [PATCH v5 1/5] arm/irq: Keep track of irq affinities

[Bertrand Marquis]

Hi Mykyta,

> On 13 Jan 2026, at 09:45, Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
> 
> Currently on Arm the desc->affinity mask of an irq is never updated,
> which makes it hard to know the actual affinity of an interrupt.
> 
> Fix this by updating the field in irq_set_affinity.

You are changing the generic irq_set_affinity which now requires
the irq descriptor to be locked.

This is good because it puts arm at an equivalent state than x86
one.

By doing a quick check there might be places where you
forgot to add the lock/unlock:

vgic/vgic.c line 826 (arch_move_irqs)
vgic/vgic-mmio-v2.c line 173 (vgic_mmio_write_target)

both of them take the irq lock but not the desc lock as far as i can see.

Cheers
Bertrand

> 
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> 
> ---
> v4->v5:
> * add locking
> 
> v3->v4:
> * patch introduced
> ---
> xen/arch/arm/gic-vgic.c |  2 ++
> xen/arch/arm/irq.c      |  9 +++++++--
> xen/arch/arm/vgic.c     | 14 ++++++++++++--
> 3 files changed, 21 insertions(+), 4 deletions(-)
> 
> diff --git a/xen/arch/arm/gic-vgic.c b/xen/arch/arm/gic-vgic.c
> index ea48c5375a..5253caf002 100644
> --- a/xen/arch/arm/gic-vgic.c
> +++ b/xen/arch/arm/gic-vgic.c
> @@ -232,7 +232,9 @@ static void gic_update_one_lr(struct vcpu *v, int i)
>             if ( test_bit(GIC_IRQ_GUEST_MIGRATING, &p->status) )
>             {
>                 struct vcpu *v_target = vgic_get_target_vcpu(v, irq);
> +                spin_lock(&p->desc->lock);
>                 irq_set_affinity(p->desc, cpumask_of(v_target->processor));
> +                spin_unlock(&p->desc->lock);
>                 clear_bit(GIC_IRQ_GUEST_MIGRATING, &p->status);
>             }
>         }
> diff --git a/xen/arch/arm/irq.c b/xen/arch/arm/irq.c
> index 73e58a5108..7204bc2b68 100644
> --- a/xen/arch/arm/irq.c
> +++ b/xen/arch/arm/irq.c
> @@ -216,10 +216,15 @@ static inline struct domain *irq_get_domain(struct irq_desc *desc)
>     return irq_get_guest_info(desc)->d;
> }
> 
> +/* Must be called with desc->lock held */
> void irq_set_affinity(struct irq_desc *desc, const cpumask_t *mask)
> {
> -    if ( desc != NULL )
> -        desc->handler->set_affinity(desc, mask);
> +    if ( desc == NULL )
> +        return;
> +
> +    ASSERT(spin_is_locked(&desc->lock));
> +    cpumask_copy(desc->affinity, mask);
> +    desc->handler->set_affinity(desc, mask);
> }
> 
> int request_irq(unsigned int irq, unsigned int irqflags,
> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
> index 6647071ad4..c59f6873db 100644
> --- a/xen/arch/arm/vgic.c
> +++ b/xen/arch/arm/vgic.c
> @@ -445,7 +445,9 @@ bool vgic_migrate_irq(struct vcpu *old, struct vcpu *new, unsigned int irq)
> 
>     if ( list_empty(&p->inflight) )
>     {
> +        spin_lock(&p->desc->lock);
>         irq_set_affinity(p->desc, cpumask_of(new->processor));
> +        spin_unlock(&p->desc->lock);
>         spin_unlock_irqrestore(&old->arch.vgic.lock, flags);
>         return true;
>     }
> @@ -453,7 +455,9 @@ bool vgic_migrate_irq(struct vcpu *old, struct vcpu *new, unsigned int irq)
>     if ( !list_empty(&p->lr_queue) )
>     {
>         vgic_remove_irq_from_queues(old, p);
> +        spin_lock(&p->desc->lock);
>         irq_set_affinity(p->desc, cpumask_of(new->processor));
> +        spin_unlock(&p->desc->lock);
>         spin_unlock_irqrestore(&old->arch.vgic.lock, flags);
>         vgic_inject_irq(new->domain, new, irq, true);
>         return true;
> @@ -473,6 +477,7 @@ void arch_move_irqs(struct vcpu *v)
>     struct domain *d = v->domain;
>     struct pending_irq *p;
>     struct vcpu *v_target;
> +    unsigned long flags;
>     int i;
> 
>     /*
> @@ -494,7 +499,13 @@ void arch_move_irqs(struct vcpu *v)
>         p = irq_to_pending(v_target, virq);
> 
>         if ( v_target == v && !test_bit(GIC_IRQ_GUEST_MIGRATING, &p->status) )
> +        {
> +            if ( !p->desc )
> +                continue;
> +            spin_lock_irqsave(&p->desc->lock, flags);
>             irq_set_affinity(p->desc, cpu_mask);
> +            spin_unlock_irqrestore(&p->desc->lock, flags);
> +        }
>     }
> }
> 
> @@ -574,8 +585,8 @@ void vgic_enable_irqs(struct vcpu *v, uint32_t r, unsigned int n)
>         spin_unlock_irqrestore(&v_target->arch.vgic.lock, flags);
>         if ( p->desc != NULL )
>         {
> -            irq_set_affinity(p->desc, cpumask_of(v_target->processor));
>             spin_lock_irqsave(&p->desc->lock, flags);
> +            irq_set_affinity(p->desc, cpumask_of(v_target->processor));
>             /*
>              * The irq cannot be a PPI, we only support delivery of SPIs
>              * to guests.
> @@ -944,4 +955,3 @@ void vgic_check_inflight_irqs_pending(struct vcpu *v, unsigned int rank, uint32_
>  * indent-tabs-mode: nil
>  * End:
>  */
> -
> -- 
> 2.51.2

Re: [PATCH v5 0/5] Implement CPU hotplug on Arm

[Mykyta Poturai]

On 03.02.26 18:35, Bertrand Marquis wrote:
> Hi Mykyta,
> 
>> On 13 Jan 2026, at 09:45, Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
>>
>> This series implements support for CPU hotplug/unplug on Arm. To achieve this,
>> several things need to be done:
>>
>> 1. XEN_SYSCTL_CPU_HOTPLUG_* calls implemented on Arm64.
>> 2. Enabled building of xen-hptool.
>> 3. Migration of irqs from dying CPUs implemented.
>>
>> Tested on QEMU.
> 
> Do you have a way to validate this on real hardware ?
> Doing hotplug/unplug might have side effects (caches, coherency, interrupts, etc) that
> I would not be completely confident to have validated by only testing this in qemu.
> 
> Could you explain how you validated this (what kind of tests you did).
> 
> I would be interested to know how the system behave in the following cases:
> - stopping cpu with guests pinned to specific cores
> - stopping a cpu with a guest pinned and only running on that one
> - checking with passthrough devices (for irq balancing)
> 
> Then a general remark i have is that most of this code (if not all) is always in Xen
> even if CONFIG_CPU_HOTPLUG is not activated in xen config which will probably
> generate some dead code. Could we protect some of the code not used when
> HOTPLUG is not activated with if (is_enabled(CONFIG_CPU_HOTPLUG)) or
> equivalent forms ?
> 
> Cheers
> Bertrand
> 

Hi Bertrand

I’ve tested various pinning scenarios, and all of them work fine, as Xen 
forcibly unpins everything that was pinned to the offlined CPU.
I’ve also explicitly checked that interrupts are getting moved to and 
from cores when needed.

I will update the next cover letter to explain testing better and also 
try to test on some hardware to be sure.

Can you please elaborate a little on what exactly you mean by “checking 
with passthrough devices”? I suspect I am not fully getting the idea.


-- 
Mykyta

Re: [PATCH v5 0/5] Implement CPU hotplug on Arm

[Bertrand Marquis]

Hi Mykyta,

> On 4 Feb 2026, at 13:58, Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
> 
> On 03.02.26 18:35, Bertrand Marquis wrote:
>> Hi Mykyta,
>> 
>>> On 13 Jan 2026, at 09:45, Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
>>> 
>>> This series implements support for CPU hotplug/unplug on Arm. To achieve this,
>>> several things need to be done:
>>> 
>>> 1. XEN_SYSCTL_CPU_HOTPLUG_* calls implemented on Arm64.
>>> 2. Enabled building of xen-hptool.
>>> 3. Migration of irqs from dying CPUs implemented.
>>> 
>>> Tested on QEMU.
>> 
>> Do you have a way to validate this on real hardware ?
>> Doing hotplug/unplug might have side effects (caches, coherency, interrupts, etc) that
>> I would not be completely confident to have validated by only testing this in qemu.
>> 
>> Could you explain how you validated this (what kind of tests you did).
>> 
>> I would be interested to know how the system behave in the following cases:
>> - stopping cpu with guests pinned to specific cores
>> - stopping a cpu with a guest pinned and only running on that one
>> - checking with passthrough devices (for irq balancing)
>> 
>> Then a general remark i have is that most of this code (if not all) is always in Xen
>> even if CONFIG_CPU_HOTPLUG is not activated in xen config which will probably
>> generate some dead code. Could we protect some of the code not used when
>> HOTPLUG is not activated with if (is_enabled(CONFIG_CPU_HOTPLUG)) or
>> equivalent forms ?
>> 
>> Cheers
>> Bertrand
>> 
> 
> Hi Bertrand
> 
> I’ve tested various pinning scenarios, and all of them work fine, as Xen 
> forcibly unpins everything that was pinned to the offlined CPU.
> I’ve also explicitly checked that interrupts are getting moved to and 
> from cores when needed.
> 
> I will update the next cover letter to explain testing better and also 
> try to test on some hardware to be sure.

Thanks.

> 
> Can you please elaborate a little on what exactly you mean by “checking 
> with passthrough devices”? I suspect I am not fully getting the idea.

If a guest has an hardware device assigned using passthrough and one of
its core is removed, is the guest and hardware device continue to work
well after ?

Cheers
Bertrand

> 
> 
> -- 
> Mykyta

Re: [PATCH v5 2/5] arm/irq: Migrate IRQs during CPU up/down operations

[Bertrand Marquis]

Hi Mykyta.

> On 13 Jan 2026, at 09:45, Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
> 
> Move IRQs from dying CPU to the online ones when a CPU is getting
> offlined. When onlining, rebalance all IRQs in a round-robin fashion.
> Guest-bound IRQs are already handled by scheduler in the process of
> moving vCPUs to active pCPUs, so we only need to handle IRQs used by Xen
> itself.
> 
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> ---
> v4->v5:
> * handle CPU onlining as well
> * more comments
> * fix crash when ESPI is disabled
> * don't assume CPU 0 is a boot CPU
> * use insigned int for irq number
> * remove assumption that all irqs a bound to CPU 0 by default from the
>  commit message
> 
> v3->v4:
> * patch introduced
> ---
> xen/arch/arm/include/asm/irq.h |  2 ++
> xen/arch/arm/irq.c             | 54 ++++++++++++++++++++++++++++++++++
> xen/arch/arm/smpboot.c         |  6 ++++
> 3 files changed, 62 insertions(+)
> 
> diff --git a/xen/arch/arm/include/asm/irq.h b/xen/arch/arm/include/asm/irq.h
> index 09788dbfeb..a0250bac85 100644
> --- a/xen/arch/arm/include/asm/irq.h
> +++ b/xen/arch/arm/include/asm/irq.h
> @@ -126,6 +126,8 @@ bool irq_type_set_by_domain(const struct domain *d);
> void irq_end_none(struct irq_desc *irq);
> #define irq_end_none irq_end_none
> 
> +void rebalance_irqs(unsigned int from, bool up);
> +
> #endif /* _ASM_HW_IRQ_H */
> /*
>  * Local variables:
> diff --git a/xen/arch/arm/irq.c b/xen/arch/arm/irq.c
> index 7204bc2b68..a32dc729f8 100644
> --- a/xen/arch/arm/irq.c
> +++ b/xen/arch/arm/irq.c
> @@ -158,6 +158,58 @@ static int init_local_irq_data(unsigned int cpu)
>     return 0;
> }
> 
> +static int cpu_next;
> +
> +static void balance_irq(int irq, unsigned int from, bool up)
> +{
> +    struct irq_desc *desc = irq_to_desc(irq);
> +    unsigned long flags;
> +
> +    ASSERT(!cpumask_empty(&cpu_online_map));
> +
> +    spin_lock_irqsave(&desc->lock, flags);
> +    if ( likely(!desc->action) )
> +        goto out;
> +
> +    if ( likely(test_bit(_IRQ_GUEST, &desc->status) ||
> +                test_bit(_IRQ_MOVE_PENDING, &desc->status)) )
> +        goto out;
> +
> +    /*
> +     * Setting affinity to a mask of multiple CPUs causes the GIC drivers to
> +     * select one CPU from that mask. If the dying CPU was included in the IRQ's
> +     * affinity mask, we cannot determine exactly which CPU the interrupt is
> +     * currently routed to, as GIC drivers lack a concrete get_affinity API. So
> +     * to be safe we must reroute it to a new, definitely online, CPU. In the
> +     * case of CPU going down, we move only the interrupt that could reside on
> +     * it. Otherwise, we rearrange all interrupts in a round-robin fashion.
> +     */
> +    if ( !up && !cpumask_test_cpu(from, desc->affinity) )
> +        goto out;

I am a bit lost here on what you are trying to do in the case where
a cpu is coming up here, it feels like you are trying to change the
affinity of all interrupts in this case to cycle everything.
Is it really what is expected ?
If affinity was set by a VM on its interrupts, I would not expect
Xen to round-robin everything each time a cpu comes up.

> +
> +    cpu_next = cpumask_cycle(cpu_next, &cpu_online_map);
> +    irq_set_affinity(desc, cpumask_of(cpu_next));
> +
> +out:
> +    spin_unlock_irqrestore(&desc->lock, flags);
> +}
> +
> +void rebalance_irqs(unsigned int from, bool up)
> +{
> +    int irq;
> +
> +    if ( cpumask_empty(&cpu_online_map) )
> +        return;
> +
> +    for ( irq = NR_LOCAL_IRQS; irq < NR_IRQS; irq++ )
> +        balance_irq(irq, from, up);
> +
> +#ifdef CONFIG_GICV3_ESPI
> +    for ( irq = ESPI_BASE_INTID; irq < ESPI_MAX_INTID; irq++ )
> +        balance_irq(irq, from, up);
> +#endif
> +}
> +
> static int cpu_callback(struct notifier_block *nfb, unsigned long action,
>                         void *hcpu)
> {
> @@ -172,6 +224,8 @@ static int cpu_callback(struct notifier_block *nfb, unsigned long action,
>             printk(XENLOG_ERR "Unable to allocate local IRQ for CPU%u\n",
>                    cpu);
>         break;
> +    case CPU_ONLINE:
> +        rebalance_irqs(cpu, true);
>     }
> 
>     return notifier_from_errno(rc);
> diff --git a/xen/arch/arm/smpboot.c b/xen/arch/arm/smpboot.c
> index 7f3cfa812e..e1b9f94458 100644
> --- a/xen/arch/arm/smpboot.c
> +++ b/xen/arch/arm/smpboot.c
> @@ -425,6 +425,12 @@ void __cpu_disable(void)
> 
>     smp_mb();
> 
> +    /*
> +     * Now that the interrupts are cleared and the CPU marked as offline,
> +     * move interrupts out of it
> +     */
> +    rebalance_irqs(cpu, false);
> +

I would expect this to only be useful when HOTPLUG is enabled, maybe
we could have a static inline doing nothing when HOTPLUG is not on
and only do something if HOTPLUG is enabled here ?

Cheers
Bertrand

>     /* Return to caller; eventually the IPI mechanism will unwind and the 
>      * scheduler will drop to the idle loop, which will call stop_cpu(). */
> }
> -- 
> 2.51.2

Re: [PATCH v5 2/5] arm/irq: Migrate IRQs during CPU up/down operations

[Mykyta Poturai]

On 04.02.26 16:20, Bertrand Marquis wrote:
> Hi Mykyta.
> 
>> On 13 Jan 2026, at 09:45, Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
>>
>> Move IRQs from dying CPU to the online ones when a CPU is getting
>> offlined. When onlining, rebalance all IRQs in a round-robin fashion.
>> Guest-bound IRQs are already handled by scheduler in the process of
>> moving vCPUs to active pCPUs, so we only need to handle IRQs used by Xen
>> itself.
>>
>> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
>> ---
>> v4->v5:
>> * handle CPU onlining as well
>> * more comments
>> * fix crash when ESPI is disabled
>> * don't assume CPU 0 is a boot CPU
>> * use insigned int for irq number
>> * remove assumption that all irqs a bound to CPU 0 by default from the
>>   commit message
>>
>> v3->v4:
>> * patch introduced
>> ---
>> xen/arch/arm/include/asm/irq.h |  2 ++
>> xen/arch/arm/irq.c             | 54 ++++++++++++++++++++++++++++++++++
>> xen/arch/arm/smpboot.c         |  6 ++++
>> 3 files changed, 62 insertions(+)
>>
>> diff --git a/xen/arch/arm/include/asm/irq.h b/xen/arch/arm/include/asm/irq.h
>> index 09788dbfeb..a0250bac85 100644
>> --- a/xen/arch/arm/include/asm/irq.h
>> +++ b/xen/arch/arm/include/asm/irq.h
>> @@ -126,6 +126,8 @@ bool irq_type_set_by_domain(const struct domain *d);
>> void irq_end_none(struct irq_desc *irq);
>> #define irq_end_none irq_end_none
>>
>> +void rebalance_irqs(unsigned int from, bool up);
>> +
>> #endif /* _ASM_HW_IRQ_H */
>> /*
>>   * Local variables:
>> diff --git a/xen/arch/arm/irq.c b/xen/arch/arm/irq.c
>> index 7204bc2b68..a32dc729f8 100644
>> --- a/xen/arch/arm/irq.c
>> +++ b/xen/arch/arm/irq.c
>> @@ -158,6 +158,58 @@ static int init_local_irq_data(unsigned int cpu)
>>      return 0;
>> }
>>
>> +static int cpu_next;
>> +
>> +static void balance_irq(int irq, unsigned int from, bool up)
>> +{
>> +    struct irq_desc *desc = irq_to_desc(irq);
>> +    unsigned long flags;
>> +
>> +    ASSERT(!cpumask_empty(&cpu_online_map));
>> +
>> +    spin_lock_irqsave(&desc->lock, flags);
>> +    if ( likely(!desc->action) )
>> +        goto out;
>> +
>> +    if ( likely(test_bit(_IRQ_GUEST, &desc->status) ||
>> +                test_bit(_IRQ_MOVE_PENDING, &desc->status)) )
>> +        goto out;
>> +
>> +    /*
>> +     * Setting affinity to a mask of multiple CPUs causes the GIC drivers to
>> +     * select one CPU from that mask. If the dying CPU was included in the IRQ's
>> +     * affinity mask, we cannot determine exactly which CPU the interrupt is
>> +     * currently routed to, as GIC drivers lack a concrete get_affinity API. So
>> +     * to be safe we must reroute it to a new, definitely online, CPU. In the
>> +     * case of CPU going down, we move only the interrupt that could reside on
>> +     * it. Otherwise, we rearrange all interrupts in a round-robin fashion.
>> +     */
>> +    if ( !up && !cpumask_test_cpu(from, desc->affinity) )
>> +        goto out;
> 
> I am a bit lost here on what you are trying to do in the case where
> a cpu is coming up here, it feels like you are trying to change the
> affinity of all interrupts in this case to cycle everything.
> Is it really what is expected ?
> If affinity was set by a VM on its interrupts, I would not expect
> Xen to round-robin everything each time a cpu comes up.
> 

The idea is to evenly spread interrupts between CPUs when the new ones 
are being brought online. This is needed to prevent Xen-bound IRQs from 
piling up on CPU 0 when other cores are being offlined and then onlined 
back. It shouldn’t mess with guest affinities, as the code skips 
everything that is assigned to guests and leaves it to be handled by the 
scheduler/vgic. Performance-wise, it should also be okay, as from what 
I’ve seen, there are not many interrupts used by Xen, and I expect CPU 
hotplug operations to be fairly infrequent.

>> +
>> +    cpu_next = cpumask_cycle(cpu_next, &cpu_online_map);
>> +    irq_set_affinity(desc, cpumask_of(cpu_next));
>> +
>> +out:
>> +    spin_unlock_irqrestore(&desc->lock, flags);
>> +}
>> +
>> +void rebalance_irqs(unsigned int from, bool up)
>> +{
>> +    int irq;
>> +
>> +    if ( cpumask_empty(&cpu_online_map) )
>> +        return;
>> +
>> +    for ( irq = NR_LOCAL_IRQS; irq < NR_IRQS; irq++ )
>> +        balance_irq(irq, from, up);
>> +
>> +#ifdef CONFIG_GICV3_ESPI
>> +    for ( irq = ESPI_BASE_INTID; irq < ESPI_MAX_INTID; irq++ )
>> +        balance_irq(irq, from, up);
>> +#endif
>> +}
>> +
>> static int cpu_callback(struct notifier_block *nfb, unsigned long action,
>>                          void *hcpu)
>> {
>> @@ -172,6 +224,8 @@ static int cpu_callback(struct notifier_block *nfb, unsigned long action,
>>              printk(XENLOG_ERR "Unable to allocate local IRQ for CPU%u\n",
>>                     cpu);
>>          break;
>> +    case CPU_ONLINE:
>> +        rebalance_irqs(cpu, true);
>>      }
>>
>>      return notifier_from_errno(rc);
>> diff --git a/xen/arch/arm/smpboot.c b/xen/arch/arm/smpboot.c
>> index 7f3cfa812e..e1b9f94458 100644
>> --- a/xen/arch/arm/smpboot.c
>> +++ b/xen/arch/arm/smpboot.c
>> @@ -425,6 +425,12 @@ void __cpu_disable(void)
>>
>>      smp_mb();
>>
>> +    /*
>> +     * Now that the interrupts are cleared and the CPU marked as offline,
>> +     * move interrupts out of it
>> +     */
>> +    rebalance_irqs(cpu, false);
>> +
> 
> I would expect this to only be useful when HOTPLUG is enabled, maybe
> we could have a static inline doing nothing when HOTPLUG is not on
> and only do something if HOTPLUG is enabled here ?
> 

Yes I will add this in the next version.

> Cheers
> Bertrand
> 
>>      /* Return to caller; eventually the IPI mechanism will unwind and the
>>       * scheduler will drop to the idle loop, which will call stop_cpu(). */
>> }
>> -- 
>> 2.51.2
> 

-- 
Mykyta

Re: [PATCH v5 2/5] arm/irq: Migrate IRQs during CPU up/down operations

[Bertrand Marquis]

Hi Mykyta,

> On 5 Feb 2026, at 14:23, Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
> 
> On 04.02.26 16:20, Bertrand Marquis wrote:
>> Hi Mykyta.
>> 
>>> On 13 Jan 2026, at 09:45, Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
>>> 
>>> Move IRQs from dying CPU to the online ones when a CPU is getting
>>> offlined. When onlining, rebalance all IRQs in a round-robin fashion.
>>> Guest-bound IRQs are already handled by scheduler in the process of
>>> moving vCPUs to active pCPUs, so we only need to handle IRQs used by Xen
>>> itself.
>>> 
>>> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
>>> ---
>>> v4->v5:
>>> * handle CPU onlining as well
>>> * more comments
>>> * fix crash when ESPI is disabled
>>> * don't assume CPU 0 is a boot CPU
>>> * use insigned int for irq number
>>> * remove assumption that all irqs a bound to CPU 0 by default from the
>>>  commit message
>>> 
>>> v3->v4:
>>> * patch introduced
>>> ---
>>> xen/arch/arm/include/asm/irq.h |  2 ++
>>> xen/arch/arm/irq.c             | 54 ++++++++++++++++++++++++++++++++++
>>> xen/arch/arm/smpboot.c         |  6 ++++
>>> 3 files changed, 62 insertions(+)
>>> 
>>> diff --git a/xen/arch/arm/include/asm/irq.h b/xen/arch/arm/include/asm/irq.h
>>> index 09788dbfeb..a0250bac85 100644
>>> --- a/xen/arch/arm/include/asm/irq.h
>>> +++ b/xen/arch/arm/include/asm/irq.h
>>> @@ -126,6 +126,8 @@ bool irq_type_set_by_domain(const struct domain *d);
>>> void irq_end_none(struct irq_desc *irq);
>>> #define irq_end_none irq_end_none
>>> 
>>> +void rebalance_irqs(unsigned int from, bool up);
>>> +
>>> #endif /* _ASM_HW_IRQ_H */
>>> /*
>>>  * Local variables:
>>> diff --git a/xen/arch/arm/irq.c b/xen/arch/arm/irq.c
>>> index 7204bc2b68..a32dc729f8 100644
>>> --- a/xen/arch/arm/irq.c
>>> +++ b/xen/arch/arm/irq.c
>>> @@ -158,6 +158,58 @@ static int init_local_irq_data(unsigned int cpu)
>>>     return 0;
>>> }
>>> 
>>> +static int cpu_next;
>>> +
>>> +static void balance_irq(int irq, unsigned int from, bool up)
>>> +{
>>> +    struct irq_desc *desc = irq_to_desc(irq);
>>> +    unsigned long flags;
>>> +
>>> +    ASSERT(!cpumask_empty(&cpu_online_map));
>>> +
>>> +    spin_lock_irqsave(&desc->lock, flags);
>>> +    if ( likely(!desc->action) )
>>> +        goto out;
>>> +
>>> +    if ( likely(test_bit(_IRQ_GUEST, &desc->status) ||
>>> +                test_bit(_IRQ_MOVE_PENDING, &desc->status)) )
>>> +        goto out;
>>> +
>>> +    /*
>>> +     * Setting affinity to a mask of multiple CPUs causes the GIC drivers to
>>> +     * select one CPU from that mask. If the dying CPU was included in the IRQ's
>>> +     * affinity mask, we cannot determine exactly which CPU the interrupt is
>>> +     * currently routed to, as GIC drivers lack a concrete get_affinity API. So
>>> +     * to be safe we must reroute it to a new, definitely online, CPU. In the
>>> +     * case of CPU going down, we move only the interrupt that could reside on
>>> +     * it. Otherwise, we rearrange all interrupts in a round-robin fashion.
>>> +     */
>>> +    if ( !up && !cpumask_test_cpu(from, desc->affinity) )
>>> +        goto out;
>> 
>> I am a bit lost here on what you are trying to do in the case where
>> a cpu is coming up here, it feels like you are trying to change the
>> affinity of all interrupts in this case to cycle everything.
>> Is it really what is expected ?
>> If affinity was set by a VM on its interrupts, I would not expect
>> Xen to round-robin everything each time a cpu comes up.
>> 
> 
> The idea is to evenly spread interrupts between CPUs when the new ones 
> are being brought online. This is needed to prevent Xen-bound IRQs from 
> piling up on CPU 0 when other cores are being offlined and then onlined 
> back. It shouldn’t mess with guest affinities, as the code skips 
> everything that is assigned to guests and leaves it to be handled by the 
> scheduler/vgic. Performance-wise, it should also be okay, as from what 
> I’ve seen, there are not many interrupts used by Xen, and I expect CPU 
> hotplug operations to be fairly infrequent.

My fear here is a bit that by removing and adding a cpu we will completely
change irq affinities. I am not so sure that those kind of random assignments
are compatible with embedded or safety use cases and here there is no way
to configure this.

@Julien, Stefano and Michal: What do you think here ?

Cheers
Bertrand

> 
>>> +
>>> +    cpu_next = cpumask_cycle(cpu_next, &cpu_online_map);
>>> +    irq_set_affinity(desc, cpumask_of(cpu_next));
>>> +
>>> +out:
>>> +    spin_unlock_irqrestore(&desc->lock, flags);
>>> +}
>>> +
>>> +void rebalance_irqs(unsigned int from, bool up)
>>> +{
>>> +    int irq;
>>> +
>>> +    if ( cpumask_empty(&cpu_online_map) )
>>> +        return;
>>> +
>>> +    for ( irq = NR_LOCAL_IRQS; irq < NR_IRQS; irq++ )
>>> +        balance_irq(irq, from, up);
>>> +
>>> +#ifdef CONFIG_GICV3_ESPI
>>> +    for ( irq = ESPI_BASE_INTID; irq < ESPI_MAX_INTID; irq++ )
>>> +        balance_irq(irq, from, up);
>>> +#endif
>>> +}
>>> +
>>> static int cpu_callback(struct notifier_block *nfb, unsigned long action,
>>>                         void *hcpu)
>>> {
>>> @@ -172,6 +224,8 @@ static int cpu_callback(struct notifier_block *nfb, unsigned long action,
>>>             printk(XENLOG_ERR "Unable to allocate local IRQ for CPU%u\n",
>>>                    cpu);
>>>         break;
>>> +    case CPU_ONLINE:
>>> +        rebalance_irqs(cpu, true);
>>>     }
>>> 
>>>     return notifier_from_errno(rc);
>>> diff --git a/xen/arch/arm/smpboot.c b/xen/arch/arm/smpboot.c
>>> index 7f3cfa812e..e1b9f94458 100644
>>> --- a/xen/arch/arm/smpboot.c
>>> +++ b/xen/arch/arm/smpboot.c
>>> @@ -425,6 +425,12 @@ void __cpu_disable(void)
>>> 
>>>     smp_mb();
>>> 
>>> +    /*
>>> +     * Now that the interrupts are cleared and the CPU marked as offline,
>>> +     * move interrupts out of it
>>> +     */
>>> +    rebalance_irqs(cpu, false);
>>> +
>> 
>> I would expect this to only be useful when HOTPLUG is enabled, maybe
>> we could have a static inline doing nothing when HOTPLUG is not on
>> and only do something if HOTPLUG is enabled here ?
>> 
> 
> Yes I will add this in the next version.
> 
>> Cheers
>> Bertrand
>> 
>>>     /* Return to caller; eventually the IPI mechanism will unwind and the
>>>      * scheduler will drop to the idle loop, which will call stop_cpu(). */
>>> }
>>> -- 
>>> 2.51.2
>> 
> 
> -- 
> Mykyta

Re: [PATCH v5 2/5] arm/irq: Migrate IRQs during CPU up/down operations

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 5255 bytes --]

On Thu, 5 Feb 2026, Bertrand Marquis wrote:
> > On 5 Feb 2026, at 14:23, Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
> > 
> > On 04.02.26 16:20, Bertrand Marquis wrote:
> >> Hi Mykyta.
> >> 
> >>> On 13 Jan 2026, at 09:45, Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
> >>> 
> >>> Move IRQs from dying CPU to the online ones when a CPU is getting
> >>> offlined. When onlining, rebalance all IRQs in a round-robin fashion.
> >>> Guest-bound IRQs are already handled by scheduler in the process of
> >>> moving vCPUs to active pCPUs, so we only need to handle IRQs used by Xen
> >>> itself.
> >>> 
> >>> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> >>> ---
> >>> v4->v5:
> >>> * handle CPU onlining as well
> >>> * more comments
> >>> * fix crash when ESPI is disabled
> >>> * don't assume CPU 0 is a boot CPU
> >>> * use insigned int for irq number
> >>> * remove assumption that all irqs a bound to CPU 0 by default from the
> >>>  commit message
> >>> 
> >>> v3->v4:
> >>> * patch introduced
> >>> ---
> >>> xen/arch/arm/include/asm/irq.h |  2 ++
> >>> xen/arch/arm/irq.c             | 54 ++++++++++++++++++++++++++++++++++
> >>> xen/arch/arm/smpboot.c         |  6 ++++
> >>> 3 files changed, 62 insertions(+)
> >>> 
> >>> diff --git a/xen/arch/arm/include/asm/irq.h b/xen/arch/arm/include/asm/irq.h
> >>> index 09788dbfeb..a0250bac85 100644
> >>> --- a/xen/arch/arm/include/asm/irq.h
> >>> +++ b/xen/arch/arm/include/asm/irq.h
> >>> @@ -126,6 +126,8 @@ bool irq_type_set_by_domain(const struct domain *d);
> >>> void irq_end_none(struct irq_desc *irq);
> >>> #define irq_end_none irq_end_none
> >>> 
> >>> +void rebalance_irqs(unsigned int from, bool up);
> >>> +
> >>> #endif /* _ASM_HW_IRQ_H */
> >>> /*
> >>>  * Local variables:
> >>> diff --git a/xen/arch/arm/irq.c b/xen/arch/arm/irq.c
> >>> index 7204bc2b68..a32dc729f8 100644
> >>> --- a/xen/arch/arm/irq.c
> >>> +++ b/xen/arch/arm/irq.c
> >>> @@ -158,6 +158,58 @@ static int init_local_irq_data(unsigned int cpu)
> >>>     return 0;
> >>> }
> >>> 
> >>> +static int cpu_next;
> >>> +
> >>> +static void balance_irq(int irq, unsigned int from, bool up)
> >>> +{
> >>> +    struct irq_desc *desc = irq_to_desc(irq);
> >>> +    unsigned long flags;
> >>> +
> >>> +    ASSERT(!cpumask_empty(&cpu_online_map));
> >>> +
> >>> +    spin_lock_irqsave(&desc->lock, flags);
> >>> +    if ( likely(!desc->action) )
> >>> +        goto out;
> >>> +
> >>> +    if ( likely(test_bit(_IRQ_GUEST, &desc->status) ||
> >>> +                test_bit(_IRQ_MOVE_PENDING, &desc->status)) )
> >>> +        goto out;
> >>> +
> >>> +    /*
> >>> +     * Setting affinity to a mask of multiple CPUs causes the GIC drivers to
> >>> +     * select one CPU from that mask. If the dying CPU was included in the IRQ's
> >>> +     * affinity mask, we cannot determine exactly which CPU the interrupt is
> >>> +     * currently routed to, as GIC drivers lack a concrete get_affinity API. So
> >>> +     * to be safe we must reroute it to a new, definitely online, CPU. In the
> >>> +     * case of CPU going down, we move only the interrupt that could reside on
> >>> +     * it. Otherwise, we rearrange all interrupts in a round-robin fashion.
> >>> +     */
> >>> +    if ( !up && !cpumask_test_cpu(from, desc->affinity) )
> >>> +        goto out;
> >> 
> >> I am a bit lost here on what you are trying to do in the case where
> >> a cpu is coming up here, it feels like you are trying to change the
> >> affinity of all interrupts in this case to cycle everything.
> >> Is it really what is expected ?
> >> If affinity was set by a VM on its interrupts, I would not expect
> >> Xen to round-robin everything each time a cpu comes up.
> >> 
> > 
> > The idea is to evenly spread interrupts between CPUs when the new ones 
> > are being brought online. This is needed to prevent Xen-bound IRQs from 
> > piling up on CPU 0 when other cores are being offlined and then onlined 
> > back. It shouldn’t mess with guest affinities, as the code skips 
> > everything that is assigned to guests and leaves it to be handled by the 
> > scheduler/vgic. Performance-wise, it should also be okay, as from what 
> > I’ve seen, there are not many interrupts used by Xen, and I expect CPU 
> > hotplug operations to be fairly infrequent.
> 
> My fear here is a bit that by removing and adding a cpu we will completely
> change irq affinities. I am not so sure that those kind of random assignments
> are compatible with embedded or safety use cases and here there is no way
> to configure this.
> 
> @Julien, Stefano and Michal: What do you think here ?

I think Bertrand's concern is valid. We need to make sure that
balance_irq is not called in "normal" or safety use-cases. I am going by
the assumption that pCPU offline/online is not going to happen in safety
use-cases. So as long as balance_irq is not called unless CPU
offline/online happens, then I think we are good. It should definitely
not happen on normal boot.                                            
 
There is also the other thought that while I do not like the idea of
spreading the IRQ evenly, I also cannot suggest something better (unless
we want to keep track of which IRQ used to belong to which dead CPU, but
I think that is undesirable).