[PATCH v7 0/6] Implement CPU hotplug on Arm

[PATCH v7 0/6] Implement CPU hotplug on Arm

[Mykyta Poturai]

This series implements support for CPU hotplug/unplug on Arm. To achieve this,
several things need to be done:

1. XEN_SYSCTL_CPU_HOTPLUG_* calls implemented on Arm64.
2. Enabled building of xen-hptool.
3. Migration of irqs from dying CPUs implemented.

Tested on QEMU and R-Car Gen5 HW.

v6->v7:
* new patch "Kconfig: Make cpu hotplug configurable

v5->v6:
* see individual patches

v4->v5:
* drop merged patches
* combine "smp: Move cpu_up/down helpers to common code" with 
  "arm/sysctl: Implement cpu hotplug ops"
* see individual patches

v3->v4:
* add irq migration patches
* see individual patches

v2->v3:
* add docs

v1->v2:
* see individual patches

Mykyta Poturai (6):
  arm/irq: Keep track of irq affinities
  arm/irq: Migrate IRQs during CPU up/down operations
  Kconfig: Make cpu hotplug configurable
  arm/sysctl: Implement cpu hotplug ops
  tools: Allow building xen-hptool without CONFIG_MIGRATE
  docs: Document CPU hotplug

 SUPPORT.md                        |  1 +
 docs/misc/cpu-hotplug.txt         | 97 +++++++++++++++++++++++++++++++
 tools/libs/guest/Makefile.common  |  2 +-
 tools/misc/Makefile               |  2 +-
 xen/arch/arm/gic-vgic.c           |  2 +
 xen/arch/arm/include/asm/irq.h    |  6 ++
 xen/arch/arm/irq.c                | 68 +++++++++++++++++++++-
 xen/arch/arm/smp.c                |  9 +++
 xen/arch/arm/smpboot.c            |  7 +++
 xen/arch/arm/vgic.c               | 14 ++++-
 xen/arch/arm/vgic/vgic-mmio-v2.c  | 11 ++--
 xen/arch/arm/vgic/vgic.c          | 21 ++++---
 xen/arch/ppc/stubs.c              |  4 ++
 xen/arch/riscv/stubs.c            |  5 ++
 xen/arch/x86/include/asm/smp.h    |  3 -
 xen/arch/x86/platform_hypercall.c | 12 ++++
 xen/arch/x86/smp.c                | 33 +----------
 xen/arch/x86/sysctl.c             | 24 +++++---
 xen/common/Kconfig                |  8 +++
 xen/common/smp.c                  | 35 +++++++++++
 xen/common/sysctl.c               | 42 +++++++++++++
 xen/include/xen/smp.h             |  4 ++
 xen/xsm/flask/hooks.c             |  2 -
 23 files changed, 348 insertions(+), 64 deletions(-)
 create mode 100644 docs/misc/cpu-hotplug.txt

-- 
2.51.2

[PATCH v7 2/6] arm/irq: Migrate IRQs during CPU up/down operations

[Mykyta Poturai]

Move IRQs from dying CPU to the online ones when a CPU is getting
offlined. When onlining, rebalance all IRQs in a round-robin fashion.
Guest-bound IRQs are already handled by scheduler in the process of
moving vCPUs to active pCPUs, so we only need to handle IRQs used by Xen
itself.

Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
v6->v7:
* replace ifdef with IS_ENABLED

v5->v6:
* don't do any balancing on boot
* only do balancing when cpu hotplug is enabled

v4->v5:
* handle CPU onlining as well
* more comments
* fix crash when ESPI is disabled
* don't assume CPU 0 is a boot CPU
* use insigned int for irq number
* remove assumption that all irqs a bound to CPU 0 by default from the
  commit message

v3->v4:
* patch introduced
---
 xen/arch/arm/include/asm/irq.h |  6 ++++
 xen/arch/arm/irq.c             | 59 ++++++++++++++++++++++++++++++++++
 xen/arch/arm/smpboot.c         |  7 ++++
 3 files changed, 72 insertions(+)

diff --git a/xen/arch/arm/include/asm/irq.h b/xen/arch/arm/include/asm/irq.h
index 09788dbfeb..3ed55e02c3 100644
--- a/xen/arch/arm/include/asm/irq.h
+++ b/xen/arch/arm/include/asm/irq.h
@@ -126,6 +126,12 @@ bool irq_type_set_by_domain(const struct domain *d);
 void irq_end_none(struct irq_desc *irq);
 #define irq_end_none irq_end_none
 
+#ifdef CONFIG_CPU_HOTPLUG
+void rebalance_irqs(unsigned int from, bool up);
+#else
+static inline void rebalance_irqs(unsigned int from, bool up) {}
+#endif
+
 #endif /* _ASM_HW_IRQ_H */
 /*
  * Local variables:
diff --git a/xen/arch/arm/irq.c b/xen/arch/arm/irq.c
index 7204bc2b68..447bee428e 100644
--- a/xen/arch/arm/irq.c
+++ b/xen/arch/arm/irq.c
@@ -158,6 +158,60 @@ static int init_local_irq_data(unsigned int cpu)
     return 0;
 }
 
+#ifdef CONFIG_CPU_HOTPLUG
+static int cpu_next;
+
+static void balance_irq(int irq, unsigned int from, bool up)
+{
+    struct irq_desc *desc = irq_to_desc(irq);
+    unsigned long flags;
+
+    ASSERT(!cpumask_empty(&cpu_online_map));
+
+    spin_lock_irqsave(&desc->lock, flags);
+    if ( likely(!desc->action) )
+        goto out;
+
+    if ( likely(test_bit(_IRQ_GUEST, &desc->status) ||
+                test_bit(_IRQ_MOVE_PENDING, &desc->status)) )
+        goto out;
+
+    /*
+     * Setting affinity to a mask of multiple CPUs causes the GIC drivers to
+     * select one CPU from that mask. If the dying CPU was included in the IRQ's
+     * affinity mask, we cannot determine exactly which CPU the interrupt is
+     * currently routed to, as GIC drivers lack a concrete get_affinity API. So
+     * to be safe we must reroute it to a new, definitely online, CPU. In the
+     * case of CPU going down, we move only the interrupt that could reside on
+     * it. Otherwise, we rearrange all interrupts in a round-robin fashion.
+     */
+    if ( !up && !cpumask_test_cpu(from, desc->affinity) )
+        goto out;
+
+    cpu_next = cpumask_cycle(cpu_next, &cpu_online_map);
+    irq_set_affinity(desc, cpumask_of(cpu_next));
+
+out:
+    spin_unlock_irqrestore(&desc->lock, flags);
+}
+
+void rebalance_irqs(unsigned int from, bool up)
+{
+    int irq;
+
+    if ( cpumask_empty(&cpu_online_map) )
+        return;
+
+    for ( irq = NR_LOCAL_IRQS; irq < NR_IRQS; irq++ )
+        balance_irq(irq, from, up);
+
+#ifdef CONFIG_GICV3_ESPI
+    for ( irq = ESPI_BASE_INTID; irq < ESPI_MAX_INTID; irq++ )
+        balance_irq(irq, from, up);
+#endif
+}
+#endif /* CONFIG_CPU_HOTPLUG */
+
 static int cpu_callback(struct notifier_block *nfb, unsigned long action,
                         void *hcpu)
 {
@@ -172,6 +226,11 @@ static int cpu_callback(struct notifier_block *nfb, unsigned long action,
             printk(XENLOG_ERR "Unable to allocate local IRQ for CPU%u\n",
                    cpu);
         break;
+    case CPU_ONLINE:
+        if ( IS_ENABLED(CONFIG_CPU_HOTPLUG) &&
+             system_state >= SYS_STATE_active )
+            rebalance_irqs(cpu, true);
+        break;
     }
 
     return notifier_from_errno(rc);
diff --git a/xen/arch/arm/smpboot.c b/xen/arch/arm/smpboot.c
index 7f3cfa812e..7d877179c0 100644
--- a/xen/arch/arm/smpboot.c
+++ b/xen/arch/arm/smpboot.c
@@ -425,6 +425,13 @@ void __cpu_disable(void)
 
     smp_mb();
 
+    /*
+     * Now that the interrupts are cleared and the CPU marked as offline,
+     * move interrupts out of it
+     */
+    if ( IS_ENABLED(CONFIG_CPU_HOTPLUG) )
+        rebalance_irqs(cpu, false);
+
     /* Return to caller; eventually the IPI mechanism will unwind and the 
      * scheduler will drop to the idle loop, which will call stop_cpu(). */
 }
-- 
2.51.2

[PATCH v7 1/6] arm/irq: Keep track of irq affinities

[Mykyta Poturai]

Currently on Arm the desc->affinity mask of an irq is never updated,
which makes it hard to know the actual affinity of an interrupt. Fix
this by updating the field in irq_set_affinity. Changing desc->affinity
requires desc->lock to be held, so add an assertion to ensure that
callers of irq_set_affinity are doing so correctly.

With desc->lock now being required for irq_set_affinity, add locking
around calls to it where it was missing.

Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>

---
v6->v7:
* update commit message
* fix possible locking on null desc
* collect RBs

v5->v6:
* add missing locking around irq_set_affinity calls

v4->v5:
* add locking

v3->v4:
* patch introduced
---
 xen/arch/arm/gic-vgic.c          |  2 ++
 xen/arch/arm/irq.c               |  9 +++++++--
 xen/arch/arm/vgic.c              | 14 ++++++++++++--
 xen/arch/arm/vgic/vgic-mmio-v2.c | 11 +++++------
 xen/arch/arm/vgic/vgic.c         | 21 +++++++++++++--------
 5 files changed, 39 insertions(+), 18 deletions(-)

diff --git a/xen/arch/arm/gic-vgic.c b/xen/arch/arm/gic-vgic.c
index ea48c5375a..5253caf002 100644
--- a/xen/arch/arm/gic-vgic.c
+++ b/xen/arch/arm/gic-vgic.c
@@ -232,7 +232,9 @@ static void gic_update_one_lr(struct vcpu *v, int i)
             if ( test_bit(GIC_IRQ_GUEST_MIGRATING, &p->status) )
             {
                 struct vcpu *v_target = vgic_get_target_vcpu(v, irq);
+                spin_lock(&p->desc->lock);
                 irq_set_affinity(p->desc, cpumask_of(v_target->processor));
+                spin_unlock(&p->desc->lock);
                 clear_bit(GIC_IRQ_GUEST_MIGRATING, &p->status);
             }
         }
diff --git a/xen/arch/arm/irq.c b/xen/arch/arm/irq.c
index 73e58a5108..7204bc2b68 100644
--- a/xen/arch/arm/irq.c
+++ b/xen/arch/arm/irq.c
@@ -216,10 +216,15 @@ static inline struct domain *irq_get_domain(struct irq_desc *desc)
     return irq_get_guest_info(desc)->d;
 }
 
+/* Must be called with desc->lock held */
 void irq_set_affinity(struct irq_desc *desc, const cpumask_t *mask)
 {
-    if ( desc != NULL )
-        desc->handler->set_affinity(desc, mask);
+    if ( desc == NULL )
+        return;
+
+    ASSERT(spin_is_locked(&desc->lock));
+    cpumask_copy(desc->affinity, mask);
+    desc->handler->set_affinity(desc, mask);
 }
 
 int request_irq(unsigned int irq, unsigned int irqflags,
diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
index 6647071ad4..c59f6873db 100644
--- a/xen/arch/arm/vgic.c
+++ b/xen/arch/arm/vgic.c
@@ -445,7 +445,9 @@ bool vgic_migrate_irq(struct vcpu *old, struct vcpu *new, unsigned int irq)
 
     if ( list_empty(&p->inflight) )
     {
+        spin_lock(&p->desc->lock);
         irq_set_affinity(p->desc, cpumask_of(new->processor));
+        spin_unlock(&p->desc->lock);
         spin_unlock_irqrestore(&old->arch.vgic.lock, flags);
         return true;
     }
@@ -453,7 +455,9 @@ bool vgic_migrate_irq(struct vcpu *old, struct vcpu *new, unsigned int irq)
     if ( !list_empty(&p->lr_queue) )
     {
         vgic_remove_irq_from_queues(old, p);
+        spin_lock(&p->desc->lock);
         irq_set_affinity(p->desc, cpumask_of(new->processor));
+        spin_unlock(&p->desc->lock);
         spin_unlock_irqrestore(&old->arch.vgic.lock, flags);
         vgic_inject_irq(new->domain, new, irq, true);
         return true;
@@ -473,6 +477,7 @@ void arch_move_irqs(struct vcpu *v)
     struct domain *d = v->domain;
     struct pending_irq *p;
     struct vcpu *v_target;
+    unsigned long flags;
     int i;
 
     /*
@@ -494,7 +499,13 @@ void arch_move_irqs(struct vcpu *v)
         p = irq_to_pending(v_target, virq);
 
         if ( v_target == v && !test_bit(GIC_IRQ_GUEST_MIGRATING, &p->status) )
+        {
+            if ( !p->desc )
+                continue;
+            spin_lock_irqsave(&p->desc->lock, flags);
             irq_set_affinity(p->desc, cpu_mask);
+            spin_unlock_irqrestore(&p->desc->lock, flags);
+        }
     }
 }
 
@@ -574,8 +585,8 @@ void vgic_enable_irqs(struct vcpu *v, uint32_t r, unsigned int n)
         spin_unlock_irqrestore(&v_target->arch.vgic.lock, flags);
         if ( p->desc != NULL )
         {
-            irq_set_affinity(p->desc, cpumask_of(v_target->processor));
             spin_lock_irqsave(&p->desc->lock, flags);
+            irq_set_affinity(p->desc, cpumask_of(v_target->processor));
             /*
              * The irq cannot be a PPI, we only support delivery of SPIs
              * to guests.
@@ -944,4 +955,3 @@ void vgic_check_inflight_irqs_pending(struct vcpu *v, unsigned int rank, uint32_
  * indent-tabs-mode: nil
  * End:
  */
-
diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
index b7c2d7ce99..fc04741ca1 100644
--- a/xen/arch/arm/vgic/vgic-mmio-v2.c
+++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
@@ -159,24 +159,23 @@ static void vgic_mmio_write_target(struct vcpu *vcpu,
     for ( i = 0; i < len; i++ )
     {
         struct vgic_irq *irq = vgic_get_irq(vcpu->domain, NULL, intid + i);
+        struct irq_desc *desc = irq_to_desc(irq->hwintid);
 
-        spin_lock_irqsave(&irq->irq_lock, flags);
+        spin_lock_irqsave(&desc->lock, flags);
+        spin_lock(&irq->irq_lock);
 
         irq->targets = (val >> (i * 8)) & cpu_mask;
         if ( irq->targets )
         {
             irq->target_vcpu = vcpu->domain->vcpu[ffs(irq->targets) - 1];
             if ( irq->hw )
-            {
-                struct irq_desc *desc = irq_to_desc(irq->hwintid);
-
                 irq_set_affinity(desc, cpumask_of(irq->target_vcpu->processor));
-            }
         }
         else
             irq->target_vcpu = NULL;
 
-        spin_unlock_irqrestore(&irq->irq_lock, flags);
+        spin_unlock(&irq->irq_lock);
+        spin_unlock_irqrestore(&desc->lock, flags);
         vgic_put_irq(vcpu->domain, irq);
     }
 }
diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
index b2c0e1873a..1c44236c4f 100644
--- a/xen/arch/arm/vgic/vgic.c
+++ b/xen/arch/arm/vgic/vgic.c
@@ -812,22 +812,27 @@ void arch_move_irqs(struct vcpu *v)
     {
         struct vgic_irq *irq = vgic_get_irq(d, NULL, i + VGIC_NR_PRIVATE_IRQS);
         unsigned long flags;
+        irq_desc_t *desc;
 
         if ( !irq )
             continue;
 
-        spin_lock_irqsave(&irq->irq_lock, flags);
-
-        /* Only hardware mapped vIRQs that are targeting this vCPU. */
-        if ( irq->hw && irq->target_vcpu == v)
+        if ( !irq->hw )
         {
-            irq_desc_t *desc = irq_to_desc(irq->hwintid);
+            vgic_put_irq(d, irq);
+            continue;
+        }
 
+        desc = irq_to_desc(irq->hwintid);
+        spin_lock_irqsave(&desc->lock, flags);
+        spin_lock(&irq->irq_lock);
+
+        /* Only hardware mapped vIRQs that are targeting this vCPU. */
+        if ( irq->target_vcpu == v )
             irq_set_affinity(desc, cpumask_of(v->processor));
-        }
 
-        spin_unlock_irqrestore(&irq->irq_lock, flags);
-        vgic_put_irq(d, irq);
+        spin_unlock(&irq->irq_lock);
+        spin_unlock_irqrestore(&desc->lock, flags);
     }
 }
 
-- 
2.51.2

[PATCH v7 4/6] arm/sysctl: Implement cpu hotplug ops

[Mykyta Poturai]

SMT-disable enforcement check is moved into a separate
architecture-specific function.

For now this operations only support Arm64. For proper Arm32 support,
there needs to be a mechanism to free per-cpu page tables, allocated in
init_domheap_mappings. Also, hotplug is not supported if ITS enabled,
and partially supported FFA, or TEE is enabled, as they use non-static
IRQ actions.

Remove ifdef guards for x86 in flask, as cpu hotplug is now
supported on more architectures.

Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>

---
v6->v7:
* use IS_ENABLED istead of ifdef in more places
* remove unneded variables
* more explicit fallthrough in do_sysctl

v5->v6:
* fix style issues
* rename arch_smt_cpu_disable -> arch_cpu_can_stay_online and invert the
logic
* use IS_ENABLED istead of ifdef
* remove explicit list af arch-specific SYSCTL_CPU_HOTPLUG_* options
from the common handler
* fix flask issue

v4->v5:
* move handling to common code
* rename config to CPU_HOTPUG
* merge with "smp: Move cpu_up/down helpers to common code"

v3->v4:
* don't reimplement cpu_up/down helpers
* add Kconfig option
* fixup formatting

v2->v3:
* no changes

v1->v2:
* remove SMT ops
* remove cpu == 0 checks
* add XSM hooks
* only implement for 64bit Arm

Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
 xen/arch/arm/smp.c             |  9 ++++++++
 xen/arch/ppc/stubs.c           |  4 ++++
 xen/arch/riscv/stubs.c         |  5 ++++
 xen/arch/x86/include/asm/smp.h |  3 ---
 xen/arch/x86/smp.c             | 36 +++--------------------------
 xen/arch/x86/sysctl.c          | 13 ++++-------
 xen/common/Kconfig             |  6 ++---
 xen/common/smp.c               | 35 ++++++++++++++++++++++++++++
 xen/common/sysctl.c            | 42 ++++++++++++++++++++++++++++++++++
 xen/include/xen/smp.h          |  4 ++++
 xen/xsm/flask/hooks.c          |  2 --
 11 files changed, 109 insertions(+), 50 deletions(-)

diff --git a/xen/arch/arm/smp.c b/xen/arch/arm/smp.c
index b372472188..0ea64d2ee1 100644
--- a/xen/arch/arm/smp.c
+++ b/xen/arch/arm/smp.c
@@ -44,6 +44,15 @@ void smp_send_call_function_mask(const cpumask_t *mask)
     }
 }
 
+/*
+ * We currently don't support SMT on ARM so we don't need any special logic for
+ * CPU disabling
+ */
+inline bool arch_cpu_can_stay_online(unsigned int cpu)
+{
+    return true;
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/arch/ppc/stubs.c b/xen/arch/ppc/stubs.c
index a333f06119..8f280ba080 100644
--- a/xen/arch/ppc/stubs.c
+++ b/xen/arch/ppc/stubs.c
@@ -101,6 +101,10 @@ void smp_send_call_function_mask(const cpumask_t *mask)
     BUG_ON("unimplemented");
 }
 
+bool arch_cpu_can_stay_online(unsigned int cpu)
+{
+    BUG_ON("unimplemented");
+}
 /* irq.c */
 
 void irq_ack_none(struct irq_desc *desc)
diff --git a/xen/arch/riscv/stubs.c b/xen/arch/riscv/stubs.c
index daadff0138..7c3cda7bc5 100644
--- a/xen/arch/riscv/stubs.c
+++ b/xen/arch/riscv/stubs.c
@@ -70,6 +70,11 @@ void smp_send_call_function_mask(const cpumask_t *mask)
     BUG_ON("unimplemented");
 }
 
+bool arch_cpu_can_stay_online(unsigned int cpu)
+{
+    BUG_ON("unimplemented");
+}
+
 /* irq.c */
 
 void irq_ack_none(struct irq_desc *desc)
diff --git a/xen/arch/x86/include/asm/smp.h b/xen/arch/x86/include/asm/smp.h
index 3f16e62696..cb3e0fed19 100644
--- a/xen/arch/x86/include/asm/smp.h
+++ b/xen/arch/x86/include/asm/smp.h
@@ -50,9 +50,6 @@ int cpu_add(uint32_t apic_id, uint32_t acpi_id, uint32_t pxm);
 
 void __stop_this_cpu(void);
 
-long cf_check cpu_up_helper(void *data);
-long cf_check cpu_down_helper(void *data);
-
 long cf_check core_parking_helper(void *data);
 bool core_parking_remove(unsigned int cpu);
 uint32_t get_cur_idle_nums(void);
diff --git a/xen/arch/x86/smp.c b/xen/arch/x86/smp.c
index a49505fb57..b781e933f2 100644
--- a/xen/arch/x86/smp.c
+++ b/xen/arch/x86/smp.c
@@ -418,38 +418,8 @@ void cf_check call_function_interrupt(void)
     smp_call_function_interrupt();
 }
 
-#ifdef CONFIG_CPU_HOTPLUG
-long cf_check cpu_up_helper(void *data)
+bool arch_cpu_can_stay_online(unsigned int cpu)
 {
-    unsigned int cpu = (unsigned long)data;
-    int ret = cpu_up(cpu);
-
-    /* Have one more go on EBUSY. */
-    if ( ret == -EBUSY )
-        ret = cpu_up(cpu);
-
-    if ( !ret && !opt_smt &&
-         cpu_data[cpu].compute_unit_id == INVALID_CUID &&
-         cpumask_weight(per_cpu(cpu_sibling_mask, cpu)) > 1 )
-    {
-        ret = cpu_down_helper(data);
-        if ( ret )
-            printk("Could not re-offline CPU%u (%d)\n", cpu, ret);
-        else
-            ret = -EPERM;
-    }
-
-    return ret;
-}
-
-long cf_check cpu_down_helper(void *data)
-{
-    int cpu = (unsigned long)data;
-    int ret = cpu_down(cpu);
-
-    /* Have one more go on EBUSY. */
-    if ( ret == -EBUSY )
-        ret = cpu_down(cpu);
-    return ret;
+    return opt_smt || cpu_data[cpu].compute_unit_id != INVALID_CUID ||
+           cpumask_weight(per_cpu(cpu_sibling_mask, cpu)) <= 1;
 }
-#endif
diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
index bdad44fef1..072726debc 100644
--- a/xen/arch/x86/sysctl.c
+++ b/xen/arch/x86/sysctl.c
@@ -120,7 +120,6 @@ long arch_do_sysctl(
 
     case XEN_SYSCTL_cpu_hotplug:
     {
-        unsigned int cpu = sysctl->u.cpu_hotplug.cpu;
         unsigned int op  = sysctl->u.cpu_hotplug.op;
         bool plug;
         long (*fn)(void *data);
@@ -128,6 +127,7 @@ long arch_do_sysctl(
 
         if ( !IS_ENABLED(CONFIG_CPU_HOTPLUG) )
         {
+            ASSERT_UNREACHABLE();
             ret = -EOPNOTSUPP;
             break;
         }
@@ -135,15 +135,10 @@ long arch_do_sysctl(
         switch ( op )
         {
         case XEN_SYSCTL_CPU_HOTPLUG_ONLINE:
-            plug = true;
-            fn = cpu_up_helper;
-            hcpu = _p(cpu);
-            break;
-
         case XEN_SYSCTL_CPU_HOTPLUG_OFFLINE:
-            plug = false;
-            fn = cpu_down_helper;
-            hcpu = _p(cpu);
+            /* Handled by common code */
+            ASSERT_UNREACHABLE();
+            ret = -EOPNOTSUPP;
             break;
 
         case XEN_SYSCTL_CPU_HOTPLUG_SMT_ENABLE:
diff --git a/xen/common/Kconfig b/xen/common/Kconfig
index 0e5b4738a8..9e26217404 100644
--- a/xen/common/Kconfig
+++ b/xen/common/Kconfig
@@ -638,9 +638,9 @@ config SYSTEM_SUSPEND
 	  If unsure, say N.
 
 config CPU_HOTPLUG
-	bool "CPU online/offline support"
-	depends on X86
-	default y
+	bool "CPU online/offline support" if EXPERT || X86
+	depends on X86 || (ARM_64 && !HAS_ITS)
+	default y if X86
 	help
 	  Enable support for bringing CPUs online and offline at runtime. On
 	  X86 this is required for disabling SMT.
diff --git a/xen/common/smp.c b/xen/common/smp.c
index a011f541f1..e2bf82856e 100644
--- a/xen/common/smp.c
+++ b/xen/common/smp.c
@@ -16,6 +16,7 @@
  * GNU General Public License for more details.
  */
 
+#include <xen/cpu.h>
 #include <asm/hardirq.h>
 #include <asm/processor.h>
 #include <xen/spinlock.h>
@@ -104,6 +105,40 @@ void smp_call_function_interrupt(void)
     irq_exit();
 }
 
+#ifdef CONFIG_CPU_HOTPLUG
+long cf_check cpu_up_helper(void *data)
+{
+    unsigned int cpu = (unsigned long)data;
+    int ret = cpu_up(cpu);
+
+    /* Have one more go on EBUSY. */
+    if ( ret == -EBUSY )
+        ret = cpu_up(cpu);
+
+    if ( !ret && !arch_cpu_can_stay_online(cpu) )
+    {
+        ret = cpu_down_helper(data);
+        if ( ret )
+            printk("Could not re-offline CPU%u (%d)\n", cpu, ret);
+        else
+            ret = -EPERM;
+    }
+
+    return ret;
+}
+
+long cf_check cpu_down_helper(void *data)
+{
+    unsigned int cpu = (unsigned long)data;
+    int ret = cpu_down(cpu);
+
+    /* Have one more go on EBUSY. */
+    if ( ret == -EBUSY )
+        ret = cpu_down(cpu);
+    return ret;
+}
+#endif /* CONFIG_CPU_HOTPLUG */
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/common/sysctl.c b/xen/common/sysctl.c
index 5207664252..da95446039 100644
--- a/xen/common/sysctl.c
+++ b/xen/common/sysctl.c
@@ -483,6 +483,48 @@ long do_sysctl(XEN_GUEST_HANDLE_PARAM(xen_sysctl_t) u_sysctl)
             copyback = 1;
         break;
 
+    case XEN_SYSCTL_cpu_hotplug:
+    {
+        unsigned int hp_op = op->u.cpu_hotplug.op;
+        bool plug;
+        long (*fn)(void *data);
+        void *hcpu = _p(op->u.cpu_hotplug.cpu);
+
+        ret = -EOPNOTSUPP;
+        if ( !IS_ENABLED(CONFIG_CPU_HOTPLUG) )
+            break;
+
+        switch ( hp_op )
+        {
+        case XEN_SYSCTL_CPU_HOTPLUG_ONLINE:
+            plug = true;
+            fn = cpu_up_helper;
+            break;
+
+        case XEN_SYSCTL_CPU_HOTPLUG_OFFLINE:
+            plug = false;
+            fn = cpu_down_helper;
+            break;
+
+        default:
+            fn = NULL;
+            break;
+        }
+
+        if ( fn )
+        {
+            ret = plug ? xsm_resource_plug_core(XSM_HOOK)
+                       : xsm_resource_unplug_core(XSM_HOOK);
+
+            if ( !ret )
+                ret = continue_hypercall_on_cpu(0, fn, hcpu);
+
+            break;
+        }
+    }
+
+        /* Use the arch handler for cases not handled here */
+        fallthrough;
     default:
         ret = arch_do_sysctl(op, u_sysctl);
         copyback = 0;
diff --git a/xen/include/xen/smp.h b/xen/include/xen/smp.h
index 2ca9ff1bfc..04530738c9 100644
--- a/xen/include/xen/smp.h
+++ b/xen/include/xen/smp.h
@@ -76,4 +76,8 @@ extern void *stack_base[NR_CPUS];
 void initialize_cpu_data(unsigned int cpu);
 int setup_cpu_root_pgt(unsigned int cpu);
 
+bool arch_cpu_can_stay_online(unsigned int cpu);
+long cf_check cpu_up_helper(void *data);
+long cf_check cpu_down_helper(void *data);
+
 #endif /* __XEN_SMP_H__ */
diff --git a/xen/xsm/flask/hooks.c b/xen/xsm/flask/hooks.c
index b250b27065..01f9d50605 100644
--- a/xen/xsm/flask/hooks.c
+++ b/xen/xsm/flask/hooks.c
@@ -835,9 +835,7 @@ static int cf_check flask_sysctl(int cmd)
     case XEN_SYSCTL_getdomaininfolist:
     case XEN_SYSCTL_page_offline_op:
     case XEN_SYSCTL_scheduler_op:
-#ifdef CONFIG_X86
     case XEN_SYSCTL_cpu_hotplug:
-#endif
         return 0;
 
     case XEN_SYSCTL_tbuf_op:
-- 
2.51.2

[PATCH v7 3/6] Kconfig: Make cpu hotplug configurable

[Mykyta Poturai]

For the purposes of certification, we want as little code as possible to
be unconditionally compiled in. Make CPU hotplug and SMT operations
configurable to ease the process. This will also help with introducing
CPU hotplug on Arm, where it needs to be configurable.

Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
v6->v7:
* new patch
---
 xen/arch/x86/platform_hypercall.c | 12 ++++++++++++
 xen/arch/x86/smp.c                |  3 +++
 xen/arch/x86/sysctl.c             | 11 +++++++++++
 xen/common/Kconfig                |  8 ++++++++
 4 files changed, 34 insertions(+)

diff --git a/xen/arch/x86/platform_hypercall.c b/xen/arch/x86/platform_hypercall.c
index cd4f0ae5e5..e745151790 100644
--- a/xen/arch/x86/platform_hypercall.c
+++ b/xen/arch/x86/platform_hypercall.c
@@ -735,6 +735,12 @@ ret_t do_platform_op(
     {
         int cpu = op->u.cpu_ol.cpuid;
 
+        if ( !IS_ENABLED(CONFIG_CPU_HOTPLUG) )
+        {
+            ret = -EOPNOTSUPP;
+            break;
+        }
+
         ret = xsm_resource_plug_core(XSM_HOOK);
         if ( ret )
             break;
@@ -761,6 +767,12 @@ ret_t do_platform_op(
     {
         int cpu = op->u.cpu_ol.cpuid;
 
+        if ( !IS_ENABLED(CONFIG_CPU_HOTPLUG) )
+        {
+            ret = -EOPNOTSUPP;
+            break;
+        }
+
         ret = xsm_resource_unplug_core(XSM_HOOK);
         if ( ret )
             break;
diff --git a/xen/arch/x86/smp.c b/xen/arch/x86/smp.c
index 7936294f5f..a49505fb57 100644
--- a/xen/arch/x86/smp.c
+++ b/xen/arch/x86/smp.c
@@ -418,6 +418,7 @@ void cf_check call_function_interrupt(void)
     smp_call_function_interrupt();
 }
 
+#ifdef CONFIG_CPU_HOTPLUG
 long cf_check cpu_up_helper(void *data)
 {
     unsigned int cpu = (unsigned long)data;
@@ -445,8 +446,10 @@ long cf_check cpu_down_helper(void *data)
 {
     int cpu = (unsigned long)data;
     int ret = cpu_down(cpu);
+
     /* Have one more go on EBUSY. */
     if ( ret == -EBUSY )
         ret = cpu_down(cpu);
     return ret;
 }
+#endif
diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
index 1b04947516..bdad44fef1 100644
--- a/xen/arch/x86/sysctl.c
+++ b/xen/arch/x86/sysctl.c
@@ -53,6 +53,11 @@ static long cf_check smt_up_down_helper(void *data)
     unsigned int cpu, sibling_mask = boot_cpu_data.x86_num_siblings - 1;
     int ret = 0;
 
+    if ( !IS_ENABLED(CONFIG_CPU_HOTPLUG) )
+    {
+        ASSERT_UNREACHABLE();
+        return -EOPNOTSUPP;
+    }
     opt_smt = up;
 
     for_each_present_cpu ( cpu )
@@ -121,6 +126,12 @@ long arch_do_sysctl(
         long (*fn)(void *data);
         void *hcpu;
 
+        if ( !IS_ENABLED(CONFIG_CPU_HOTPLUG) )
+        {
+            ret = -EOPNOTSUPP;
+            break;
+        }
+
         switch ( op )
         {
         case XEN_SYSCTL_CPU_HOTPLUG_ONLINE:
diff --git a/xen/common/Kconfig b/xen/common/Kconfig
index d7e79e752a..0e5b4738a8 100644
--- a/xen/common/Kconfig
+++ b/xen/common/Kconfig
@@ -637,6 +637,14 @@ config SYSTEM_SUSPEND
 
 	  If unsure, say N.
 
+config CPU_HOTPLUG
+	bool "CPU online/offline support"
+	depends on X86
+	default y
+	help
+	  Enable support for bringing CPUs online and offline at runtime. On
+	  X86 this is required for disabling SMT.
+
 menu "Supported hypercall interfaces"
 	visible if EXPERT
 
-- 
2.51.2

[PATCH v7 5/6] tools: Allow building xen-hptool without CONFIG_MIGRATE

[Mykyta Poturai]

With CPU hotplug sysctls implemented on Arm it becomes useful to have a
tool for calling them.

According to the commit history it seems that putting hptool under
config MIGRATE was a measure to fix IA64 build. As IA64 is no longer
supported it can now be brought back. So build it unconditionally.

Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
v6->v7:
* no changes

v5->v6:
* don't change order in Makefile

v4->v5:
* make hptool always build

v3->v4:
* no changes

v2->v3:
* no changes

v1->v2:
* switch to configure from legacy config
---
 tools/libs/guest/Makefile.common | 2 +-
 tools/misc/Makefile              | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools/libs/guest/Makefile.common b/tools/libs/guest/Makefile.common
index b928a4a246..03dfcee7fa 100644
--- a/tools/libs/guest/Makefile.common
+++ b/tools/libs/guest/Makefile.common
@@ -7,6 +7,7 @@ OBJS-y += xg_private.o
 OBJS-y += xg_domain.o
 OBJS-y += xg_suspend.o
 OBJS-y += xg_resume.o
+OBJS-y += xg_offline_page.o
 ifeq ($(CONFIG_MIGRATE),y)
 OBJS-y += xg_sr_common.o
 OBJS-$(CONFIG_X86) += xg_sr_common_x86.o
@@ -17,7 +18,6 @@ OBJS-$(CONFIG_X86) += xg_sr_save_x86_pv.o
 OBJS-$(CONFIG_X86) += xg_sr_save_x86_hvm.o
 OBJS-y += xg_sr_restore.o
 OBJS-y += xg_sr_save.o
-OBJS-y += xg_offline_page.o
 else
 OBJS-y += xg_nomigrate.o
 endif
diff --git a/tools/misc/Makefile b/tools/misc/Makefile
index 6ee783f43e..5a206133f7 100644
--- a/tools/misc/Makefile
+++ b/tools/misc/Makefile
@@ -16,7 +16,7 @@ INSTALL_BIN                    += xencov_split
 INSTALL_BIN += $(INSTALL_BIN-y)
 
 # Everything to be installed in regular sbin/
-INSTALL_SBIN-$(CONFIG_MIGRATE) += xen-hptool
+INSTALL_SBIN                   += xen-hptool
 INSTALL_SBIN-$(CONFIG_X86)     += xen-hvmcrash
 INSTALL_SBIN-$(CONFIG_X86)     += xen-hvmctx
 INSTALL_SBIN-$(CONFIG_X86)     += xen-lowmemd
-- 
2.51.2

[PATCH v7 6/6] docs: Document CPU hotplug

[Mykyta Poturai]

Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
v6->v7:
* add testing and limitations

v5->v6:
* no changes

v4->v5:
* s/supported/implemented/
* update SUPPORT.md

v3->v4:
* update configuration section

v2->v3:
* patch introduced
---
 SUPPORT.md                |  1 +
 docs/misc/cpu-hotplug.txt | 97 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 98 insertions(+)
 create mode 100644 docs/misc/cpu-hotplug.txt

diff --git a/SUPPORT.md b/SUPPORT.md
index d441bccf37..7b93ae69e7 100644
--- a/SUPPORT.md
+++ b/SUPPORT.md
@@ -52,6 +52,7 @@ For the Cortex A77 r0p0 - r1p0, see Errata 1508412.
 ### ACPI CPU Hotplug
 
     Status, x86: Experimental
+    Status, Arm64: Experimental
 
 ### Physical Memory
 
diff --git a/docs/misc/cpu-hotplug.txt b/docs/misc/cpu-hotplug.txt
new file mode 100644
index 0000000000..09a2855873
--- /dev/null
+++ b/docs/misc/cpu-hotplug.txt
@@ -0,0 +1,97 @@
+CPU Hotplug
+===========
+
+CPU hotplug is a feature that allows pCPU cores to be added to or removed from a
+running system without requiring a reboot. It is implemented on x86 and Arm64
+architectures.
+
+Implementation Details
+----------------------
+
+CPU hotplug is implemented through the `XEN_SYSCTL_CPU_HOTPLUG_*` sysctl calls.
+The specific calls are:
+
+- `XEN_SYSCTL_CPU_HOTPLUG_ONLINE`: Brings a pCPU online
+- `XEN_SYSCTL_CPU_HOTPLUG_OFFLINE`: Takes a pCPU offline
+- `XEN_SYSCTL_CPU_HOTPLUG_SMT_ENABLE`: Enables SMT threads (x86 only)
+- `XEN_SYSCTL_CPU_HOTPLUG_SMT_DISABLE`: Disables SMT threads (x86 only)
+
+All cores can be disabled, assuming hardware support, except for the boot core.
+Sysctl calls are routed to the boot core before doing any actual up/down
+operations on other cores.
+
+If there are Xen-bound interrupts pinned to the pCPU being offlined, they will
+be automatically migrated to other online pCPUs. Interrupts used by guest
+domains are handled by the scheduler when it reschedules the vCPUs to a new,
+online, pCPU. When a pCPU is being onlined, some Xen-bound interrupts will get
+redistributed to the newly onlined pCPU to prevent imbalance.
+
+If pCPU being offlined has some vCPUs pinned to it, they will be automatically
+unpinned and migrated to other online pCPUs.
+
+Limitations
+-----------
+
+On Arm64 cpu hotplug is currently not compatible with ITS, due to an issues with
+the redistributor assignment.
+
+On Arm64 there can be problems with FFA if secure FW support notification ABI.
+
+Configuration
+-------------
+
+The presence of the feature is controlled by CONFIG_CPU_HOTPLUG option. It is
+enabled by default on x86 architecture. On Arm64, the option is disabled by
+default and marked as EXPERT.
+xen-hptool userspace tool is built unconditionally.
+
+Usage
+-----
+
+Disable core:
+
+$ xen-hptool cpu-offline 2
+Prepare to offline CPU 2
+(XEN) Removing cpu 2 from runqueue 0
+CPU 2 offlined successfully
+
+Enable core:
+
+$ xen-hptool cpu-online 2
+Prepare to online CPU 2
+(XEN) Bringing up CPU2
+(XEN) GICv3: CPU2: Found redistributor in region 0 @00000a004005c000
+(XEN) CPU2: Guest atomics will try 1 times before pausing the domain
+(XEN) CPU 2 booted.
+(XEN) Adding cpu 2 to runqueue 0
+CPU 2 onlined successfully
+
+Disabling a core with pinned vCPUs:
+
+$ xl vcpu-pin 0 3 3 3
+$ xl vcpu-pin 0 2 3 3
+$ xl vcpu-pin 0 1 3 3
+$ xl vcpu-pin 0 0 3 3
+$ xen-hptool cpu-offline 3
+Prepare to offline CPU 3
+(XEN) Breaking affinity for d0v0
+(XEN) Breaking affinity for d0v1
+(XEN) Breaking affinity for d0v2
+(XEN) Breaking affinity for d0v3
+(XEN) Removing cpu 3 from runqueue 0
+CPU 3 offlined successfully
+
+Testing
+-------
+
+The CPU hotplug feature has been tested on both x86 and Arm64 QEMU setups and on
+R-Car Gen5 (Arm64) hardware.
+
+The tests included:
+- Offlining and onlining cores with no pinned vCPUs
+- Offlining cores with pinned vCPUs
+- Offlining cores with Xen-bound interrupts
+- Offlining all cores except the boot core
+- Offlining the boot core (expected to fail)
+- Enabling and disabling SMT threads (x86 only)
+- Ofllining cores to which guests with passthrough devices are pinned
-- 
2.51.2

Re: [PATCH v7 3/6] Kconfig: Make cpu hotplug configurable

[Jan Beulich]

On 30.03.2026 13:59, Mykyta Poturai wrote:
> For the purposes of certification, we want as little code as possible to
> be unconditionally compiled in. Make CPU hotplug and SMT operations
> configurable to ease the process. This will also help with introducing
> CPU hotplug on Arm, where it needs to be configurable.
> 
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>

Looks largely okay from a technical pov; one nit and one (repeated) remark
below.

> --- a/xen/arch/x86/sysctl.c
> +++ b/xen/arch/x86/sysctl.c
> @@ -53,6 +53,11 @@ static long cf_check smt_up_down_helper(void *data)
>      unsigned int cpu, sibling_mask = boot_cpu_data.x86_num_siblings - 1;
>      int ret = 0;
>  
> +    if ( !IS_ENABLED(CONFIG_CPU_HOTPLUG) )
> +    {
> +        ASSERT_UNREACHABLE();
> +        return -EOPNOTSUPP;
> +    }
>      opt_smt = up;

Another blank line above this one perhaps?

> --- a/xen/common/Kconfig
> +++ b/xen/common/Kconfig
> @@ -637,6 +637,14 @@ config SYSTEM_SUSPEND
>  
>  	  If unsure, say N.
>  
> +config CPU_HOTPLUG
> +	bool "CPU online/offline support"
> +	depends on X86
> +	default y
> +	help
> +	  Enable support for bringing CPUs online and offline at runtime. On
> +	  X86 this is required for disabling SMT.

The name of this option may need input from others; I'm not quite convinced
that this is a good name, as there's no true "hot-plugging" involved here.
IOW I fear the present name is misleading.

Jan

Re: [PATCH v7 4/6] arm/sysctl: Implement cpu hotplug ops

[Jan Beulich]

On 30.03.2026 13:59, Mykyta Poturai wrote:
> --- a/xen/common/Kconfig
> +++ b/xen/common/Kconfig
> @@ -638,9 +638,9 @@ config SYSTEM_SUSPEND
>  	  If unsure, say N.
>  
>  config CPU_HOTPLUG
> -	bool "CPU online/offline support"
> -	depends on X86
> -	default y
> +	bool "CPU online/offline support" if EXPERT || X86

Why not just EXPERT?

> +	depends on X86 || (ARM_64 && !HAS_ITS)

The !HAS_ITS is puzzling, and it doesn't help that that option looks
misnamed (HAS_* shouldn't have prompts imo). The description says
something there, yes, but then also mentions FFA and TEE. Yet for
those the option remains available.

> +	default y if X86

Shorter as "default X86".

Jan

Re: [PATCH v7 5/6] tools: Allow building xen-hptool without CONFIG_MIGRATE

[Jan Beulich]

On 30.03.2026 13:59, Mykyta Poturai wrote:
> With CPU hotplug sysctls implemented on Arm it becomes useful to have a
> tool for calling them.
> 
> According to the commit history it seems that putting hptool under
> config MIGRATE was a measure to fix IA64 build. As IA64 is no longer
> supported it can now be brought back. So build it unconditionally.
> 
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> ---
> v6->v7:
> * no changes
> 
> v5->v6:
> * don't change order in Makefile
> 
> v4->v5:
> * make hptool always build
> 
> v3->v4:
> * no changes
> 
> v2->v3:
> * no changes
> 
> v1->v2:
> * switch to configure from legacy config
> ---
>  tools/libs/guest/Makefile.common | 2 +-
>  tools/misc/Makefile              | 2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/tools/libs/guest/Makefile.common b/tools/libs/guest/Makefile.common
> index b928a4a246..03dfcee7fa 100644
> --- a/tools/libs/guest/Makefile.common
> +++ b/tools/libs/guest/Makefile.common
> @@ -7,6 +7,7 @@ OBJS-y += xg_private.o
>  OBJS-y += xg_domain.o
>  OBJS-y += xg_suspend.o
>  OBJS-y += xg_resume.o
> +OBJS-y += xg_offline_page.o
>  ifeq ($(CONFIG_MIGRATE),y)
>  OBJS-y += xg_sr_common.o
>  OBJS-$(CONFIG_X86) += xg_sr_common_x86.o
> @@ -17,7 +18,6 @@ OBJS-$(CONFIG_X86) += xg_sr_save_x86_pv.o
>  OBJS-$(CONFIG_X86) += xg_sr_save_x86_hvm.o
>  OBJS-y += xg_sr_restore.o
>  OBJS-y += xg_sr_save.o
> -OBJS-y += xg_offline_page.o
>  else
>  OBJS-y += xg_nomigrate.o
>  endif

This looks wrong to me. There are x86-specifics in that file, which shouldn't
be built on Arm. And the name of the file also doesn't indicate any relation
to CPU management.

Jan

Re: [PATCH v7 6/6] docs: Document CPU hotplug

[Jan Beulich]

On 30.03.2026 13:59, Mykyta Poturai wrote:
> --- a/SUPPORT.md
> +++ b/SUPPORT.md
> @@ -52,6 +52,7 @@ For the Cortex A77 r0p0 - r1p0, see Errata 1508412.
>  ### ACPI CPU Hotplug
>  
>      Status, x86: Experimental
> +    Status, Arm64: Experimental

Are you sure? I didn't spot an ACPI connection in the patches (in particular
in patch 4).

Jan

Re: [PATCH v7 3/6] Kconfig: Make cpu hotplug configurable

[Mykyta Poturai]

On 3/30/26 15:19, Jan Beulich wrote:
> On 30.03.2026 13:59, Mykyta Poturai wrote:
>> For the purposes of certification, we want as little code as possible to
>> be unconditionally compiled in. Make CPU hotplug and SMT operations
>> configurable to ease the process. This will also help with introducing
>> CPU hotplug on Arm, where it needs to be configurable.
>>
>> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> 
> Looks largely okay from a technical pov; one nit and one (repeated) remark
> below.
> 
>> --- a/xen/arch/x86/sysctl.c
>> +++ b/xen/arch/x86/sysctl.c
>> @@ -53,6 +53,11 @@ static long cf_check smt_up_down_helper(void *data)
>>       unsigned int cpu, sibling_mask = boot_cpu_data.x86_num_siblings - 1;
>>       int ret = 0;
>>   
>> +    if ( !IS_ENABLED(CONFIG_CPU_HOTPLUG) )
>> +    {
>> +        ASSERT_UNREACHABLE();
>> +        return -EOPNOTSUPP;
>> +    }
>>       opt_smt = up;
> 
> Another blank line above this one perhaps?
> 
>> --- a/xen/common/Kconfig
>> +++ b/xen/common/Kconfig
>> @@ -637,6 +637,14 @@ config SYSTEM_SUSPEND
>>   
>>   	  If unsure, say N.
>>   
>> +config CPU_HOTPLUG
>> +	bool "CPU online/offline support"
>> +	depends on X86
>> +	default y
>> +	help
>> +	  Enable support for bringing CPUs online and offline at runtime. On
>> +	  X86 this is required for disabling SMT.
> 
> The name of this option may need input from others; I'm not quite convinced
> that this is a good name, as there's no true "hot-plugging" involved here.
> IOW I fear the present name is misleading.
> 
> Jan

My first idea was "CONFIG_RUNTIME_CPU_CONTROL" I can switch back to it.

-- 
Mykyta

Re: [PATCH v7 3/6] Kconfig: Make cpu hotplug configurable

[Jan Beulich]

On 08.04.2026 14:21, Mykyta Poturai wrote:
> On 3/30/26 15:19, Jan Beulich wrote:
>> On 30.03.2026 13:59, Mykyta Poturai wrote:
>>> For the purposes of certification, we want as little code as possible to
>>> be unconditionally compiled in. Make CPU hotplug and SMT operations
>>> configurable to ease the process. This will also help with introducing
>>> CPU hotplug on Arm, where it needs to be configurable.
>>>
>>> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
>>
>> Looks largely okay from a technical pov; one nit and one (repeated) remark
>> below.
>>
>>> --- a/xen/arch/x86/sysctl.c
>>> +++ b/xen/arch/x86/sysctl.c
>>> @@ -53,6 +53,11 @@ static long cf_check smt_up_down_helper(void *data)
>>>       unsigned int cpu, sibling_mask = boot_cpu_data.x86_num_siblings - 1;
>>>       int ret = 0;
>>>   
>>> +    if ( !IS_ENABLED(CONFIG_CPU_HOTPLUG) )
>>> +    {
>>> +        ASSERT_UNREACHABLE();
>>> +        return -EOPNOTSUPP;
>>> +    }
>>>       opt_smt = up;
>>
>> Another blank line above this one perhaps?
>>
>>> --- a/xen/common/Kconfig
>>> +++ b/xen/common/Kconfig
>>> @@ -637,6 +637,14 @@ config SYSTEM_SUSPEND
>>>   
>>>   	  If unsure, say N.
>>>   
>>> +config CPU_HOTPLUG
>>> +	bool "CPU online/offline support"
>>> +	depends on X86
>>> +	default y
>>> +	help
>>> +	  Enable support for bringing CPUs online and offline at runtime. On
>>> +	  X86 this is required for disabling SMT.
>>
>> The name of this option may need input from others; I'm not quite convinced
>> that this is a good name, as there's no true "hot-plugging" involved here.
>> IOW I fear the present name is misleading.
> 
> My first idea was "CONFIG_RUNTIME_CPU_CONTROL" I can switch back to it.

I could live with that, for at least not being misleading. CPU_ONLINE or
CPU_ONLINE_OFFLINE might be another option, possibly better suited to
later become a dependency (select) of a true CPU_HOTPLUG. As said, input
from others may be helpful.

Jan

Re: [PATCH v7 4/6] arm/sysctl: Implement cpu hotplug ops

[Mykyta Poturai]

On 3/30/26 15:28, Jan Beulich wrote:
> On 30.03.2026 13:59, Mykyta Poturai wrote:
>> --- a/xen/common/Kconfig
>> +++ b/xen/common/Kconfig
>> @@ -638,9 +638,9 @@ config SYSTEM_SUSPEND
>>   	  If unsure, say N.
>>   
>>   config CPU_HOTPLUG
>> -	bool "CPU online/offline support"
>> -	depends on X86
>> -	default y
>> +	bool "CPU online/offline support" if EXPERT || X86
> 
> Why not just EXPERT?

Should it be marked as EXPERT on x86? I considered that if the option 
was non configurable (always enabled), it should stay enabled by default 
and always visible.

>> +	depends on X86 || (ARM_64 && !HAS_ITS)
> 
> The !HAS_ITS is puzzling, and it doesn't help that that option looks
> misnamed (HAS_* shouldn't have prompts imo). The description says
> something there, yes, but then also mentions FFA and TEE. Yet for
> those the option remains available.
> 

HAS_ITS can be named better, but this is way out of scope for this 
series, and for now this is the only way to express this dependency. 
Regarding TEE and FFA, I removed them from dependencies because hotplug 
may work with some TEE OS configurations, but not with all of them. That 
is the main reason it is marked as EXPERT for Arm64.

>> +	default y if X86
> 
> Shorter as "default X86".
> 
> Jan

Got it.

-- 
Mykyta

Re: [PATCH v7 4/6] arm/sysctl: Implement cpu hotplug ops

[Jan Beulich]

On 09.04.2026 16:34, Mykyta Poturai wrote:
> On 3/30/26 15:28, Jan Beulich wrote:
>> On 30.03.2026 13:59, Mykyta Poturai wrote:
>>> --- a/xen/common/Kconfig
>>> +++ b/xen/common/Kconfig
>>> @@ -638,9 +638,9 @@ config SYSTEM_SUSPEND
>>>   	  If unsure, say N.
>>>   
>>>   config CPU_HOTPLUG
>>> -	bool "CPU online/offline support"
>>> -	depends on X86
>>> -	default y
>>> +	bool "CPU online/offline support" if EXPERT || X86
>>
>> Why not just EXPERT?
> 
> Should it be marked as EXPERT on x86? I considered that if the option 
> was non configurable (always enabled), it should stay enabled by default 
> and always visible.

As you say, it wasn't configurable before. Imo that's a good sign that, at
least for starters, we should limit its disabling to experts.

Jan

Re: [PATCH v7 3/6] Kconfig: Make cpu hotplug configurable

[Oleksandr Tyshchenko]

On 4/8/26 15:27, Jan Beulich wrote:
> On 08.04.2026 14:21, Mykyta Poturai wrote:

Hello all.

>> On 3/30/26 15:19, Jan Beulich wrote:
>>> On 30.03.2026 13:59, Mykyta Poturai wrote:
>>>> For the purposes of certification, we want as little code as possible to
>>>> be unconditionally compiled in. Make CPU hotplug and SMT operations
>>>> configurable to ease the process. This will also help with introducing
>>>> CPU hotplug on Arm, where it needs to be configurable.
>>>>
>>>> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
>>>
>>> Looks largely okay from a technical pov; one nit and one (repeated) remark
>>> below.
>>>
>>>> --- a/xen/arch/x86/sysctl.c
>>>> +++ b/xen/arch/x86/sysctl.c
>>>> @@ -53,6 +53,11 @@ static long cf_check smt_up_down_helper(void *data)
>>>>        unsigned int cpu, sibling_mask = boot_cpu_data.x86_num_siblings - 1;
>>>>        int ret = 0;
>>>>    
>>>> +    if ( !IS_ENABLED(CONFIG_CPU_HOTPLUG) )
>>>> +    {
>>>> +        ASSERT_UNREACHABLE();
>>>> +        return -EOPNOTSUPP;
>>>> +    }
>>>>        opt_smt = up;
>>>
>>> Another blank line above this one perhaps?
>>>
>>>> --- a/xen/common/Kconfig
>>>> +++ b/xen/common/Kconfig
>>>> @@ -637,6 +637,14 @@ config SYSTEM_SUSPEND
>>>>    
>>>>    	  If unsure, say N.
>>>>    
>>>> +config CPU_HOTPLUG
>>>> +	bool "CPU online/offline support"
>>>> +	depends on X86
>>>> +	default y
>>>> +	help
>>>> +	  Enable support for bringing CPUs online and offline at runtime. On
>>>> +	  X86 this is required for disabling SMT.
>>>
>>> The name of this option may need input from others; I'm not quite convinced
>>> that this is a good name, as there's no true "hot-plugging" involved here.
>>> IOW I fear the present name is misleading.
>>
>> My first idea was "CONFIG_RUNTIME_CPU_CONTROL" I can switch back to it.
> 
> I could live with that, for at least not being misleading. CPU_ONLINE or
> CPU_ONLINE_OFFLINE might be another option, possibly better suited to
> later become a dependency (select) of a true CPU_HOTPLUG. As said, input
> from others may be helpful.

To me, CONFIG_RUNTIME_CPU_CONTROL sounds a little bit vague. Although we 
are indeed controlling CPUs at runtime, "control" could also mean 
cpufreq, power management, affinity pinning. I think 
CONFIG_CPU_ONLINE_OFFLINE is more precise, as it is clear from the name 
that we are transitioning CPUs between online and offline states.


> 
> Jan
> 

Re: [PATCH v7 2/6] arm/irq: Migrate IRQs during CPU up/down operations

[Oleksandr Tyshchenko]

On 3/30/26 14:59, Mykyta Poturai wrote:

Hello Mykyta


> Move IRQs from dying CPU to the online ones when a CPU is getting
> offlined. When onlining, rebalance all IRQs in a round-robin fashion.
> Guest-bound IRQs are already handled by scheduler in the process of
> moving vCPUs to active pCPUs, so we only need to handle IRQs used by Xen
> itself.
> 
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> ---
> v6->v7:
> * replace ifdef with IS_ENABLED
> 
> v5->v6:
> * don't do any balancing on boot
> * only do balancing when cpu hotplug is enabled
> 
> v4->v5:
> * handle CPU onlining as well
> * more comments
> * fix crash when ESPI is disabled
> * don't assume CPU 0 is a boot CPU
> * use insigned int for irq number
> * remove assumption that all irqs a bound to CPU 0 by default from the
>    commit message
> 
> v3->v4:
> * patch introduced
> ---
>   xen/arch/arm/include/asm/irq.h |  6 ++++
>   xen/arch/arm/irq.c             | 59 ++++++++++++++++++++++++++++++++++
>   xen/arch/arm/smpboot.c         |  7 ++++
>   3 files changed, 72 insertions(+)
> 
> diff --git a/xen/arch/arm/include/asm/irq.h b/xen/arch/arm/include/asm/irq.h
> index 09788dbfeb..3ed55e02c3 100644
> --- a/xen/arch/arm/include/asm/irq.h
> +++ b/xen/arch/arm/include/asm/irq.h
> @@ -126,6 +126,12 @@ bool irq_type_set_by_domain(const struct domain *d);
>   void irq_end_none(struct irq_desc *irq);
>   #define irq_end_none irq_end_none
>   
> +#ifdef CONFIG_CPU_HOTPLUG
> +void rebalance_irqs(unsigned int from, bool up);
> +#else
> +static inline void rebalance_irqs(unsigned int from, bool up) {}
> +#endif
> +
>   #endif /* _ASM_HW_IRQ_H */
>   /*
>    * Local variables:
> diff --git a/xen/arch/arm/irq.c b/xen/arch/arm/irq.c
> index 7204bc2b68..447bee428e 100644
> --- a/xen/arch/arm/irq.c
> +++ b/xen/arch/arm/irq.c
> @@ -158,6 +158,60 @@ static int init_local_irq_data(unsigned int cpu)
>       return 0;
>   }
>   
> +#ifdef CONFIG_CPU_HOTPLUG
> +static int cpu_next;
> +
> +static void balance_irq(int irq, unsigned int from, bool up)
> +{
> +    struct irq_desc *desc = irq_to_desc(irq);
> +    unsigned long flags;
> +
> +    ASSERT(!cpumask_empty(&cpu_online_map));
> +
> +    spin_lock_irqsave(&desc->lock, flags);
> +    if ( likely(!desc->action) )
> +        goto out;
> +
> +    if ( likely(test_bit(_IRQ_GUEST, &desc->status) ||
> +                test_bit(_IRQ_MOVE_PENDING, &desc->status)) )
> +        goto out;
> +
> +    /*
> +     * Setting affinity to a mask of multiple CPUs causes the GIC drivers to
> +     * select one CPU from that mask. If the dying CPU was included in the IRQ's
> +     * affinity mask, we cannot determine exactly which CPU the interrupt is
> +     * currently routed to, as GIC drivers lack a concrete get_affinity API. So
> +     * to be safe we must reroute it to a new, definitely online, CPU. In the
> +     * case of CPU going down, we move only the interrupt that could reside on
> +     * it. Otherwise, we rearrange all interrupts in a round-robin fashion.
> +     */
> +    if ( !up && !cpumask_test_cpu(from, desc->affinity) )
> +        goto out;
> +
> +    cpu_next = cpumask_cycle(cpu_next, &cpu_online_map);
> +    irq_set_affinity(desc, cpumask_of(cpu_next));
> +
> +out:
> +    spin_unlock_irqrestore(&desc->lock, flags);
> +}
> +
> +void rebalance_irqs(unsigned int from, bool up)
> +{
> +    int irq;
> +
> +    if ( cpumask_empty(&cpu_online_map) )
> +        return;
> +
> +    for ( irq = NR_LOCAL_IRQS; irq < NR_IRQS; irq++ )
> +        balance_irq(irq, from, up);
> +
> +#ifdef CONFIG_GICV3_ESPI
> +    for ( irq = ESPI_BASE_INTID; irq < ESPI_MAX_INTID; irq++ )
> +        balance_irq(irq, from, up);

I think, here we have an inefficient iteration over the ESPI range.
Even when CONFIG_GICV3_ESPI=y, the GIC HW might not support ESPIs at 
runtime.

We should probably check if they are present before entering the loop so 
we do not waste cycles on 1024 unnecessary NULL lookups. Could we use 
something like ESPI_BASE_INTID + gic_number_espis() to set the loop 
boundary? What do you think?

> +#endif
> +}
> +#endif /* CONFIG_CPU_HOTPLUG */
> +
>   static int cpu_callback(struct notifier_block *nfb, unsigned long action,
>                           void *hcpu)
>   {
> @@ -172,6 +226,11 @@ static int cpu_callback(struct notifier_block *nfb, unsigned long action,
>               printk(XENLOG_ERR "Unable to allocate local IRQ for CPU%u\n",
>                      cpu);
>           break;
> +    case CPU_ONLINE:
> +        if ( IS_ENABLED(CONFIG_CPU_HOTPLUG) &&
> +             system_state >= SYS_STATE_active )
> +            rebalance_irqs(cpu, true);
> +        break;
>       }
>   
>       return notifier_from_errno(rc);
> diff --git a/xen/arch/arm/smpboot.c b/xen/arch/arm/smpboot.c
> index 7f3cfa812e..7d877179c0 100644
> --- a/xen/arch/arm/smpboot.c
> +++ b/xen/arch/arm/smpboot.c
> @@ -425,6 +425,13 @@ void __cpu_disable(void)
>   
>       smp_mb();
>   
> +    /*
> +     * Now that the interrupts are cleared and the CPU marked as offline,
> +     * move interrupts out of it
> +     */
> +    if ( IS_ENABLED(CONFIG_CPU_HOTPLUG) )
> +        rebalance_irqs(cpu, false);
> +
>       /* Return to caller; eventually the IPI mechanism will unwind and the
>        * scheduler will drop to the idle loop, which will call stop_cpu(). */
>   }

Re: [PATCH v7 5/6] tools: Allow building xen-hptool without CONFIG_MIGRATE

[Mykyta Poturai]

On 3/30/26 15:32, Jan Beulich wrote:
> On 30.03.2026 13:59, Mykyta Poturai wrote:
>> With CPU hotplug sysctls implemented on Arm it becomes useful to have a
>> tool for calling them.
>>
>> According to the commit history it seems that putting hptool under
>> config MIGRATE was a measure to fix IA64 build. As IA64 is no longer
>> supported it can now be brought back. So build it unconditionally.
>>
>> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
>> ---
>> v6->v7:
>> * no changes
>>
>> v5->v6:
>> * don't change order in Makefile
>>
>> v4->v5:
>> * make hptool always build
>>
>> v3->v4:
>> * no changes
>>
>> v2->v3:
>> * no changes
>>
>> v1->v2:
>> * switch to configure from legacy config
>> ---
>>   tools/libs/guest/Makefile.common | 2 +-
>>   tools/misc/Makefile              | 2 +-
>>   2 files changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/tools/libs/guest/Makefile.common b/tools/libs/guest/Makefile.common
>> index b928a4a246..03dfcee7fa 100644
>> --- a/tools/libs/guest/Makefile.common
>> +++ b/tools/libs/guest/Makefile.common
>> @@ -7,6 +7,7 @@ OBJS-y += xg_private.o
>>   OBJS-y += xg_domain.o
>>   OBJS-y += xg_suspend.o
>>   OBJS-y += xg_resume.o
>> +OBJS-y += xg_offline_page.o
>>   ifeq ($(CONFIG_MIGRATE),y)
>>   OBJS-y += xg_sr_common.o
>>   OBJS-$(CONFIG_X86) += xg_sr_common_x86.o
>> @@ -17,7 +18,6 @@ OBJS-$(CONFIG_X86) += xg_sr_save_x86_pv.o
>>   OBJS-$(CONFIG_X86) += xg_sr_save_x86_hvm.o
>>   OBJS-y += xg_sr_restore.o
>>   OBJS-y += xg_sr_save.o
>> -OBJS-y += xg_offline_page.o
>>   else
>>   OBJS-y += xg_nomigrate.o
>>   endif
> 
> This looks wrong to me. There are x86-specifics in that file, which shouldn't
> be built on Arm. And the name of the file also doesn't indicate any relation
> to CPU management.
> 
> Jan

xen-hptool requires xg_offline_page as it has both CPU and memory 
hotplug commands. Without building xg_offline_page it fails with

xen-hptool: symbol lookup error: xen-hptool: undefined symbol: 
xc_mark_page_offline, version libxenguest_4.22.0

when trying to do memory ops.

Is it an acceptable behavior? If so I guess we can build xg_offline page 
only on x86.

-- 
Mykyta

Re: [PATCH v7 5/6] tools: Allow building xen-hptool without CONFIG_MIGRATE

[Jan Beulich]

On 15.04.2026 16:51, Mykyta Poturai wrote:
> On 3/30/26 15:32, Jan Beulich wrote:
>> On 30.03.2026 13:59, Mykyta Poturai wrote:
>>> With CPU hotplug sysctls implemented on Arm it becomes useful to have a
>>> tool for calling them.
>>>
>>> According to the commit history it seems that putting hptool under
>>> config MIGRATE was a measure to fix IA64 build. As IA64 is no longer
>>> supported it can now be brought back. So build it unconditionally.
>>>
>>> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
>>> ---
>>> v6->v7:
>>> * no changes
>>>
>>> v5->v6:
>>> * don't change order in Makefile
>>>
>>> v4->v5:
>>> * make hptool always build
>>>
>>> v3->v4:
>>> * no changes
>>>
>>> v2->v3:
>>> * no changes
>>>
>>> v1->v2:
>>> * switch to configure from legacy config
>>> ---
>>>   tools/libs/guest/Makefile.common | 2 +-
>>>   tools/misc/Makefile              | 2 +-
>>>   2 files changed, 2 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/tools/libs/guest/Makefile.common b/tools/libs/guest/Makefile.common
>>> index b928a4a246..03dfcee7fa 100644
>>> --- a/tools/libs/guest/Makefile.common
>>> +++ b/tools/libs/guest/Makefile.common
>>> @@ -7,6 +7,7 @@ OBJS-y += xg_private.o
>>>   OBJS-y += xg_domain.o
>>>   OBJS-y += xg_suspend.o
>>>   OBJS-y += xg_resume.o
>>> +OBJS-y += xg_offline_page.o
>>>   ifeq ($(CONFIG_MIGRATE),y)
>>>   OBJS-y += xg_sr_common.o
>>>   OBJS-$(CONFIG_X86) += xg_sr_common_x86.o
>>> @@ -17,7 +18,6 @@ OBJS-$(CONFIG_X86) += xg_sr_save_x86_pv.o
>>>   OBJS-$(CONFIG_X86) += xg_sr_save_x86_hvm.o
>>>   OBJS-y += xg_sr_restore.o
>>>   OBJS-y += xg_sr_save.o
>>> -OBJS-y += xg_offline_page.o
>>>   else
>>>   OBJS-y += xg_nomigrate.o
>>>   endif
>>
>> This looks wrong to me. There are x86-specifics in that file, which shouldn't
>> be built on Arm. And the name of the file also doesn't indicate any relation
>> to CPU management.
> 
> xen-hptool requires xg_offline_page as it has both CPU and memory 
> hotplug commands. Without building xg_offline_page it fails with
> 
> xen-hptool: symbol lookup error: xen-hptool: undefined symbol: 
> xc_mark_page_offline, version libxenguest_4.22.0
> 
> when trying to do memory ops.
> 
> Is it an acceptable behavior?

I don't think so, no. The tool wouldn't, aiui, load at all then if built with
"bindnow" enabled.

> If so I guess we can build xg_offline page only on x86.

We still need to, imo. But the tool still needs to be usable no matter how
specifically it is built. It should avoid referencing xg_offline_page.c
functions when built for non-x86.

Jan

Re: [PATCH v7 5/6] tools: Allow building xen-hptool without CONFIG_MIGRATE

[Mykyta Poturai]

On 4/16/26 09:49, Jan Beulich wrote:
> On 15.04.2026 16:51, Mykyta Poturai wrote:
>> On 3/30/26 15:32, Jan Beulich wrote:
>>> On 30.03.2026 13:59, Mykyta Poturai wrote:
>>>> With CPU hotplug sysctls implemented on Arm it becomes useful to have a
>>>> tool for calling them.
>>>>
>>>> According to the commit history it seems that putting hptool under
>>>> config MIGRATE was a measure to fix IA64 build. As IA64 is no longer
>>>> supported it can now be brought back. So build it unconditionally.
>>>>
>>>> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
>>>> ---
>>>> v6->v7:
>>>> * no changes
>>>>
>>>> v5->v6:
>>>> * don't change order in Makefile
>>>>
>>>> v4->v5:
>>>> * make hptool always build
>>>>
>>>> v3->v4:
>>>> * no changes
>>>>
>>>> v2->v3:
>>>> * no changes
>>>>
>>>> v1->v2:
>>>> * switch to configure from legacy config
>>>> ---
>>>>    tools/libs/guest/Makefile.common | 2 +-
>>>>    tools/misc/Makefile              | 2 +-
>>>>    2 files changed, 2 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/tools/libs/guest/Makefile.common b/tools/libs/guest/Makefile.common
>>>> index b928a4a246..03dfcee7fa 100644
>>>> --- a/tools/libs/guest/Makefile.common
>>>> +++ b/tools/libs/guest/Makefile.common
>>>> @@ -7,6 +7,7 @@ OBJS-y += xg_private.o
>>>>    OBJS-y += xg_domain.o
>>>>    OBJS-y += xg_suspend.o
>>>>    OBJS-y += xg_resume.o
>>>> +OBJS-y += xg_offline_page.o
>>>>    ifeq ($(CONFIG_MIGRATE),y)
>>>>    OBJS-y += xg_sr_common.o
>>>>    OBJS-$(CONFIG_X86) += xg_sr_common_x86.o
>>>> @@ -17,7 +18,6 @@ OBJS-$(CONFIG_X86) += xg_sr_save_x86_pv.o
>>>>    OBJS-$(CONFIG_X86) += xg_sr_save_x86_hvm.o
>>>>    OBJS-y += xg_sr_restore.o
>>>>    OBJS-y += xg_sr_save.o
>>>> -OBJS-y += xg_offline_page.o
>>>>    else
>>>>    OBJS-y += xg_nomigrate.o
>>>>    endif
>>>
>>> This looks wrong to me. There are x86-specifics in that file, which shouldn't
>>> be built on Arm. And the name of the file also doesn't indicate any relation
>>> to CPU management.
>>
>> xen-hptool requires xg_offline_page as it has both CPU and memory
>> hotplug commands. Without building xg_offline_page it fails with
>>
>> xen-hptool: symbol lookup error: xen-hptool: undefined symbol:
>> xc_mark_page_offline, version libxenguest_4.22.0
>>
>> when trying to do memory ops.
>>
>> Is it an acceptable behavior?
> 
> I don't think so, no. The tool wouldn't, aiui, load at all then if built with
> "bindnow" enabled.
> 
>> If so I guess we can build xg_offline page only on x86.
> 
> We still need to, imo. But the tool still needs to be usable no matter how
> specifically it is built. It should avoid referencing xg_offline_page.c
> functions when built for non-x86.
> 
> Jan

As I understand, the usage of arch-specific compile time checks is 
heavily discouraged in tools. So I don’t think it would be approved by 
tools maintainers. Do we really need to omit this file if memory ops are 
already getting blocked by Xen on Arm anyway?

-- 
Mykyta

Re: [PATCH v7 5/6] tools: Allow building xen-hptool without CONFIG_MIGRATE

[Jan Beulich]

On 16.04.2026 10:22, Mykyta Poturai wrote:
> On 4/16/26 09:49, Jan Beulich wrote:
>> On 15.04.2026 16:51, Mykyta Poturai wrote:
>>> On 3/30/26 15:32, Jan Beulich wrote:
>>>> On 30.03.2026 13:59, Mykyta Poturai wrote:
>>>>> With CPU hotplug sysctls implemented on Arm it becomes useful to have a
>>>>> tool for calling them.
>>>>>
>>>>> According to the commit history it seems that putting hptool under
>>>>> config MIGRATE was a measure to fix IA64 build. As IA64 is no longer
>>>>> supported it can now be brought back. So build it unconditionally.
>>>>>
>>>>> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
>>>>> ---
>>>>> v6->v7:
>>>>> * no changes
>>>>>
>>>>> v5->v6:
>>>>> * don't change order in Makefile
>>>>>
>>>>> v4->v5:
>>>>> * make hptool always build
>>>>>
>>>>> v3->v4:
>>>>> * no changes
>>>>>
>>>>> v2->v3:
>>>>> * no changes
>>>>>
>>>>> v1->v2:
>>>>> * switch to configure from legacy config
>>>>> ---
>>>>>    tools/libs/guest/Makefile.common | 2 +-
>>>>>    tools/misc/Makefile              | 2 +-
>>>>>    2 files changed, 2 insertions(+), 2 deletions(-)
>>>>>
>>>>> diff --git a/tools/libs/guest/Makefile.common b/tools/libs/guest/Makefile.common
>>>>> index b928a4a246..03dfcee7fa 100644
>>>>> --- a/tools/libs/guest/Makefile.common
>>>>> +++ b/tools/libs/guest/Makefile.common
>>>>> @@ -7,6 +7,7 @@ OBJS-y += xg_private.o
>>>>>    OBJS-y += xg_domain.o
>>>>>    OBJS-y += xg_suspend.o
>>>>>    OBJS-y += xg_resume.o
>>>>> +OBJS-y += xg_offline_page.o
>>>>>    ifeq ($(CONFIG_MIGRATE),y)
>>>>>    OBJS-y += xg_sr_common.o
>>>>>    OBJS-$(CONFIG_X86) += xg_sr_common_x86.o
>>>>> @@ -17,7 +18,6 @@ OBJS-$(CONFIG_X86) += xg_sr_save_x86_pv.o
>>>>>    OBJS-$(CONFIG_X86) += xg_sr_save_x86_hvm.o
>>>>>    OBJS-y += xg_sr_restore.o
>>>>>    OBJS-y += xg_sr_save.o
>>>>> -OBJS-y += xg_offline_page.o
>>>>>    else
>>>>>    OBJS-y += xg_nomigrate.o
>>>>>    endif
>>>>
>>>> This looks wrong to me. There are x86-specifics in that file, which shouldn't
>>>> be built on Arm. And the name of the file also doesn't indicate any relation
>>>> to CPU management.
>>>
>>> xen-hptool requires xg_offline_page as it has both CPU and memory
>>> hotplug commands. Without building xg_offline_page it fails with
>>>
>>> xen-hptool: symbol lookup error: xen-hptool: undefined symbol:
>>> xc_mark_page_offline, version libxenguest_4.22.0
>>>
>>> when trying to do memory ops.
>>>
>>> Is it an acceptable behavior?
>>
>> I don't think so, no. The tool wouldn't, aiui, load at all then if built with
>> "bindnow" enabled.
>>
>>> If so I guess we can build xg_offline page only on x86.
>>
>> We still need to, imo. But the tool still needs to be usable no matter how
>> specifically it is built. It should avoid referencing xg_offline_page.c
>> functions when built for non-x86.
> 
> As I understand, the usage of arch-specific compile time checks is 
> heavily discouraged in tools. So I don’t think it would be approved by 
> tools maintainers. Do we really need to omit this file if memory ops are 
> already getting blocked by Xen on Arm anyway?

Nothing I can answer in a way that's definitive for your purpose. As you
say, the final word here is with Anthony. I've voiced my opinion.

Jan

Re: [PATCH v7 5/6] tools: Allow building xen-hptool without CONFIG_MIGRATE

[Anthony PERARD]

[-- Attachment #1: Type: text/plain, Size: 2496 bytes --]

On Thu, Apr 16, 2026 at 08:22:32AM +0000, Mykyta Poturai wrote:
> On 4/16/26 09:49, Jan Beulich wrote:
> > On 15.04.2026 16:51, Mykyta Poturai wrote:
> >> On 3/30/26 15:32, Jan Beulich wrote:
> >>> This looks wrong to me. There are x86-specifics in that file, which shouldn't
> >>> be built on Arm. And the name of the file also doesn't indicate any relation
> >>> to CPU management.
> >>
> >> xen-hptool requires xg_offline_page as it has both CPU and memory
> >> hotplug commands. Without building xg_offline_page it fails with
> >>
> >> xen-hptool: symbol lookup error: xen-hptool: undefined symbol:
> >> xc_mark_page_offline, version libxenguest_4.22.0
> >>
> >> when trying to do memory ops.
> >>
> >> Is it an acceptable behavior?
> > 
> > I don't think so, no. The tool wouldn't, aiui, load at all then if built with
> > "bindnow" enabled.
> > 
> >> If so I guess we can build xg_offline page only on x86.
> > 
> > We still need to, imo. But the tool still needs to be usable no matter how
> > specifically it is built. It should avoid referencing xg_offline_page.c
> > functions when built for non-x86.
> 
> As I understand, the usage of arch-specific compile time checks is 
> heavily discouraged in tools. So I don’t think it would be approved by 
> tools maintainers. Do we really need to omit this file if memory ops are 
> already getting blocked by Xen on Arm anyway?

So you are trying to modify a library and introduced untested
functionality just to be able to build a different tool? I don't think
that a good idea especially in this case where it's more than just glue
code between a binary and xen.

We could change the library to provide the missing symbols, but it is
probably best to keep it that way for now.

So, how about changing `xen-hptool` to have reduced functionality on
other platform, and keep the 'mem-*' command on x86 only? You could move
the function that implement the 'mem-*' command into a separate file,
that compile only on x86 (or more specifically when CONFIG_MIGRATE is
set) and just have a "#if defined(__i386__) || defined(__x86_64__)" in
the `main_options` array.

They are compile-time arch-specific check everywhere in tools. Arch
specific are often implemented in separated source file, this mean we
can limit the #ifdefs to a minimum and keep the code readable.

Thanks,


--
Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH v7 5/6] tools: Allow building xen-hptool without CONFIG_MIGRATE

[Mykyta Poturai]

On 4/29/26 17:33, Anthony PERARD wrote:
> On Thu, Apr 16, 2026 at 08:22:32AM +0000, Mykyta Poturai wrote:
>> On 4/16/26 09:49, Jan Beulich wrote:
>>> On 15.04.2026 16:51, Mykyta Poturai wrote:
>>>> On 3/30/26 15:32, Jan Beulich wrote:
>>>>> This looks wrong to me. There are x86-specifics in that file, which shouldn't
>>>>> be built on Arm. And the name of the file also doesn't indicate any relation
>>>>> to CPU management.
>>>>
>>>> xen-hptool requires xg_offline_page as it has both CPU and memory
>>>> hotplug commands. Without building xg_offline_page it fails with
>>>>
>>>> xen-hptool: symbol lookup error: xen-hptool: undefined symbol:
>>>> xc_mark_page_offline, version libxenguest_4.22.0
>>>>
>>>> when trying to do memory ops.
>>>>
>>>> Is it an acceptable behavior?
>>>
>>> I don't think so, no. The tool wouldn't, aiui, load at all then if built with
>>> "bindnow" enabled.
>>>
>>>> If so I guess we can build xg_offline page only on x86.
>>>
>>> We still need to, imo. But the tool still needs to be usable no matter how
>>> specifically it is built. It should avoid referencing xg_offline_page.c
>>> functions when built for non-x86.
>>
>> As I understand, the usage of arch-specific compile time checks is
>> heavily discouraged in tools. So I don’t think it would be approved by
>> tools maintainers. Do we really need to omit this file if memory ops are
>> already getting blocked by Xen on Arm anyway?
> 
> So you are trying to modify a library and introduced untested
> functionality just to be able to build a different tool? I don't think
> that a good idea especially in this case where it's more than just glue
> code between a binary and xen.
> 
> We could change the library to provide the missing symbols, but it is
> probably best to keep it that way for now.
> 
> So, how about changing `xen-hptool` to have reduced functionality on
> other platform, and keep the 'mem-*' command on x86 only? You could move
> the function that implement the 'mem-*' command into a separate file,
> that compile only on x86 (or more specifically when CONFIG_MIGRATE is
> set) and just have a "#if defined(__i386__) || defined(__x86_64__)" in
> the `main_options` array.
> 
> They are compile-time arch-specific check everywhere in tools. Arch
> specific are often implemented in separated source file, this mean we
> can limit the #ifdefs to a minimum and keep the code readable.
> 
> Thanks,
> 
> 

Should I also do the same thing for SMT operations?

-- 
Mykyta

Re: [PATCH v7 5/6] tools: Allow building xen-hptool without CONFIG_MIGRATE

[Anthony PERARD]

[-- Attachment #1: Type: text/plain, Size: 1430 bytes --]

On Wed, May 06, 2026 at 08:52:29AM +0000, Mykyta Poturai wrote:
> On 4/29/26 17:33, Anthony PERARD wrote:
> > So you are trying to modify a library and introduced untested
> > functionality just to be able to build a different tool? I don't think
> > that a good idea especially in this case where it's more than just glue
> > code between a binary and xen.
> > 
> > We could change the library to provide the missing symbols, but it is
> > probably best to keep it that way for now.
> > 
> > So, how about changing `xen-hptool` to have reduced functionality on
> > other platform, and keep the 'mem-*' command on x86 only? You could move
> > the function that implement the 'mem-*' command into a separate file,
> > that compile only on x86 (or more specifically when CONFIG_MIGRATE is
> > set) and just have a "#if defined(__i386__) || defined(__x86_64__)" in
> > the `main_options` array.
> > 
> > They are compile-time arch-specific check everywhere in tools. Arch
> > specific are often implemented in separated source file, this mean we
> > can limit the #ifdefs to a minimum and keep the code readable.
> 
> Should I also do the same thing for SMT operations?

I guess it wouldn't hurt they are x86 only. So yes, looks like a good
idea to do it for the smt op as well.

Cheers,


--
Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech