[PATCH V1 0/4] Arch agnostic ACPI changes to support vCPU Hotplug (on Archs like ARM)

[PATCH V1 0/4] Arch agnostic ACPI changes to support vCPU Hotplug (on Archs like ARM)

[Salil Mehta via]

Certain CPU architecture specifications [1][2][3] prohibit changes to the CPUs
*presence* after the kernel has booted. This is because many system
initializations depend on the exact CPU count at boot time and do not expect it
to change afterward. For example, components like interrupt controllers that are
closely coupled with CPUs, or various per-CPU features, may not support
configuration changes once the kernel has been initialized.

This requirement poses a challenge for virtualization features like vCPU
hotplug. To address this, changes to the ACPI AML are necessary to update the
`_STA.PRES` (presence) and `_STA.ENA` (enabled) bits accordingly during guest
initialization, as well as when vCPUs are hot-plugged or hot-unplugged. The
presence of unplugged vCPUs may need to be deliberately *simulated* at the ACPI
level to maintain a *persistent* view of vCPUs for the guest kernel.

This patch set introduces the following features:

1. ACPI Interface with Explicit PRESENT and ENABLED CPU States: It allows the
   guest kernel to evaluate these states using the `_STA` ACPI method.
   
2. Initialization of ACPI CPU States: These states are initialized during
   `machvirt_init` and when vCPUs are hot-(un)plugged. This enables hotpluggable
   vCPUs to be exposed to the guest kernel via ACPI.

3. Support for Migrating ACPI CPU States: The patch set ensures the migration of
   the newly introduced `is_{present,enabled}` ACPI CPU states to the
   destination VM.

The approach is flexible enough to accommodate ARM-like architectures that
intend to implement vCPU hotplug functionality. It is suitable for architectures
facing similar constraints to ARM or those that plan to implement vCPU
hotplugging independently of hardware support (if available).

This patch set is derived from the ARM-specific vCPU hotplug implementation [4]
and includes migration components adaptable to other architectures, following
suggestions [5] made by Igor Mammedov <imammedo@redhat.com>.

It can be applied independently, ensuring compatibility with existing hotplug
support in other architectures. I have tested this patch set in conjunction with
the ARM-specific vCPU hotplug changes (included in the upcoming RFC V5 [6]), and
everything worked as expected. I kindly request maintainers of other
architectures to provide a "Tested-by" after running their respective regression
tests.

Many thanks!


References:
[1] KVMForum 2023 Presentation: Challenges Revisited in Supporting Virt CPU Hotplug on
    architectures that don’t Support CPU Hotplug (like ARM64)
    a. Kernel Link: https://kvm-forum.qemu.org/2023/KVM-forum-cpu-hotplug_7OJ1YyJ.pdf
    b. Qemu Link:  https://kvm-forum.qemu.org/2023/Challenges_Revisited_in_Supporting_Virt_CPU_Hotplug_-__ii0iNb3.pdf
[2] KVMForum 2020 Presentation: Challenges in Supporting Virtual CPU Hotplug on
    SoC Based Systems (like ARM64)
    Link: https://kvmforum2020.sched.com/event/eE4m
[3] Check comment 5 in the bugzilla entry
    Link: https://bugzilla.tianocore.org/show_bug.cgi?id=4481#c5
[4] [PATCH RFC V4 00/33] Support of Virtual CPU Hotplug for ARMv8 Arch
    Link: https://lore.kernel.org/qemu-devel/20241009031815.250096-1-salil.mehta@huawei.com/T/#mf32be203baa568a871dc625b732f666a4c4f1e68
[5] Architecture agnostic ACPI VMSD state migration (Discussion)
    Link: https://lore.kernel.org/qemu-devel/20240715155436.577d34c5@imammedo.users.ipa.redhat.com/
[6] Upcoming RFC V5, Support of Virtual CPU Hotplug for ARMv8 Arch
    Link: https://github.com/salil-mehta/qemu/commits/virt-cpuhp-armv8/rfc-v5

Salil Mehta (4):
  hw/acpi: Initialize ACPI Hotplug CPU Status with Support for vCPU
    `Persistence`
  hw/acpi: Update ACPI CPU Status `is_{present, enabled}` during vCPU
    hot(un)plug
  hw/acpi: Reflect ACPI vCPU {present,enabled} states in ACPI
    _STA.{PRES,ENA} Bits
  hw/acpi: Populate vCPU Hotplug VMSD to migrate `is_{present,enabled}`
    states

 cpu-target.c         patches.vcpuhp.rfc-v5.arch.agnostic.acpi          |  1 +
 hw/acpi/cpu.c                  | 70 +++++++++++++++++++++++++++++++---
 hw/acpi/generic_event_device.c | 11 ++++++
 include/hw/acpi/cpu.h          | 21 ++++++++++
 include/hw/core/cpu.h          | 21 ++++++++++
 5 files changed, 119 insertions(+), 5 deletions(-)

-- 
2.34.1

[PATCH V1 1/4] hw/acpi: Initialize ACPI Hotplug CPU Status with Support for vCPU `Persistence`

[Salil Mehta via]

Certain CPU architecture specifications [1][2][3] prohibit changes to CPU
presence after the kernel has booted. This limitation exists because many system
initializations rely on the exact CPU count at boot time and do not expect it to
change later. For example, components like interrupt controllers, which are
closely tied to CPUs, or various per-CPU features, may not support configuration
changes once the kernel has been initialized. This presents a challenge for
virtualization features such as vCPU hotplug.

To address this issue, introduce an `is_enabled` state in the `AcpiCpuStatus`,
which reflects whether a vCPU has been hot-plugged or hot-unplugged in QEMU,
marking it as (un)available in the Guest Kernel. The `is_present` state should
be set based on the `acpi_persistent` flag. In cases where unplugged vCPUs need
to be deliberately simulated in the ACPI to maintain a persistent view of vCPUs,
this flag ensures the guest kernel continues to see those vCPUs.

Additionally, introduce an `acpi_persistent` property that can be used to
initialize the ACPI vCPU presence state accordingly. Architectures requiring
ACPI to expose a persistent view of vCPUs can override its default value. Refer
to the patch-set implelenting vCPU hotplug support for ARM for more details on
its usage.

References:
[1] KVMForum 2023 Presentation: Challenges Revisited in Supporting Virt CPU Hotplug on
    architectures that don’t Support CPU Hotplug (like ARM64)
    a. Kernel Link: https://kvm-forum.qemu.org/2023/KVM-forum-cpu-hotplug_7OJ1YyJ.pdf
    b. Qemu Link:  https://kvm-forum.qemu.org/2023/Challenges_Revisited_in_Supporting_Virt_CPU_Hotplug_-__ii0iNb3.pdf
[2] KVMForum 2020 Presentation: Challenges in Supporting Virtual CPU Hotplug on
    SoC Based Systems (like ARM64)
    Link: https://kvmforum2020.sched.com/event/eE4m
[3] Check comment 5 in the bugzilla entry
    Link: https://bugzilla.tianocore.org/show_bug.cgi?id=4481#c5

Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
---
 cpu-target.c          |  1 +
 hw/acpi/cpu.c         | 35 ++++++++++++++++++++++++++++++++++-
 include/hw/acpi/cpu.h | 21 +++++++++++++++++++++
 include/hw/core/cpu.h | 21 +++++++++++++++++++++
 4 files changed, 77 insertions(+), 1 deletion(-)

diff --git a/cpu-target.c b/cpu-target.c
index 499facf774..c8a29ab495 100644
--- a/cpu-target.c
+++ b/cpu-target.c
@@ -200,6 +200,7 @@ static Property cpu_common_props[] = {
      */
     DEFINE_PROP_LINK("memory", CPUState, memory, TYPE_MEMORY_REGION,
                      MemoryRegion *),
+    DEFINE_PROP_BOOL("acpi-persistent", CPUState, acpi_persistent, false),
 #endif
     DEFINE_PROP_END_OF_LIST(),
 };
diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
index 5cb60ca8bc..083c4010c2 100644
--- a/hw/acpi/cpu.c
+++ b/hw/acpi/cpu.c
@@ -225,7 +225,40 @@ void cpu_hotplug_hw_init(MemoryRegion *as, Object *owner,
     state->dev_count = id_list->len;
     state->devs = g_new0(typeof(*state->devs), state->dev_count);
     for (i = 0; i < id_list->len; i++) {
-        state->devs[i].cpu =  CPU(id_list->cpus[i].cpu);
+        struct CPUState *cpu = CPU(id_list->cpus[i].cpu);
+        /*
+         * In most architectures, CPUs that are marked as ACPI 'present' are
+         * also ACPI 'enabled' by default. These states remain consistent at
+         * both the QOM and ACPI levels.
+         */
+        if (cpu) {
+            state->devs[i].is_enabled = true;
+            state->devs[i].is_present = true;
+            state->devs[i].cpu = cpu;
+        } else {
+            state->devs[i].is_enabled = false;
+            /*
+             * In some architectures, even 'unplugged' or 'disabled' QOM CPUs
+             * may be exposed as ACPI 'present.' This approach provides a
+             * persistent view of the vCPUs to the guest kernel. This could be
+             * due to an architectural constraint that requires every per-CPU
+             * component to be present at boot time, meaning the exact count of
+             * vCPUs must be known and cannot be altered after the kernel has
+             * booted. As a result, the vCPU states at the QOM and ACPI levels
+             * might become inconsistent. However, in such cases, the presence
+             * of vCPUs has been deliberately simulated at the ACPI level.
+             */
+            if (acpi_persistent_cpu(first_cpu)) {
+                state->devs[i].is_present = true;
+                /*
+                 * `CPUHotplugState::AcpiCpuStatus::cpu` becomes insignificant
+                 * in this case
+                 */
+            } else {
+                state->devs[i].is_present = false;
+                state->devs[i].cpu = cpu;
+            }
+        }
         state->devs[i].arch_id = id_list->cpus[i].arch_id;
     }
     memory_region_init_io(&state->ctrl_reg, owner, &cpu_hotplug_ops, state,
diff --git a/include/hw/acpi/cpu.h b/include/hw/acpi/cpu.h
index 32654dc274..bd3f9973c9 100644
--- a/include/hw/acpi/cpu.h
+++ b/include/hw/acpi/cpu.h
@@ -26,6 +26,8 @@ typedef struct AcpiCpuStatus {
     uint64_t arch_id;
     bool is_inserting;
     bool is_removing;
+    bool is_present;
+    bool is_enabled;
     bool fw_remove;
     uint32_t ost_event;
     uint32_t ost_status;
@@ -75,4 +77,23 @@ extern const VMStateDescription vmstate_cpu_hotplug;
     VMSTATE_STRUCT(cpuhp, state, 1, \
                    vmstate_cpu_hotplug, CPUHotplugState)
 
+/**
+ * acpi_persistent_cpu:
+ * @cpu: The vCPU to check
+ *
+ * Checks if the vCPU state should always be reflected as *present* via ACPI
+ * to the Guest. By default, this is False on all architectures and has to be
+ * explicity set during initialization.
+ *
+ * Returns: True if it is ACPI 'persistent' CPU
+ *
+ */
+static inline bool acpi_persistent_cpu(CPUState *cpu)
+{
+    /*
+     * returns if 'Presence' of the vCPU is persistent and should be simulated
+     * via ACPI even after vCPUs have been unplugged in QOM
+     */
+    return cpu && cpu->acpi_persistent;
+}
 #endif
diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
index 04e9ad4996..299e96c45b 100644
--- a/include/hw/core/cpu.h
+++ b/include/hw/core/cpu.h
@@ -542,6 +542,27 @@ struct CPUState {
     CPUPluginState *plugin_state;
 #endif
 
+    /*
+     * To implement the vCPU hotplug feature (which simulates CPU hotplug
+     * behavior), we need to dynamically create and destroy QOM vCPU objects,
+     * and (de)associate them with pre-existing KVM vCPUs while (un)parking the
+     * KVM vCPU context. One challenge is ensuring that these dynamically
+     * appearing or disappearing QOM vCPU objects are accurately reflected
+     * through ACPI to the Guest Kernel. Due to architectural constraints,
+     * changing the number of vCPUs after the guest kernel has booted may not
+     * always be possible.
+     *
+     * In certain architectures, to provide the guest kernel with a *persistent*
+     * view of vCPU presence, even when the QOM does not have a corresponding
+     * vCPU object, ACPI may simulate the presence of vCPUs by marking them as
+     * ACPI-disabled. This is achieved by setting `_STA.PRES=True` and
+     * `_STA.Ena=False` for unplugged vCPUs in QEMU's QOM.
+     *
+     * By default, this flag is set to `FALSE`, and it must be explicitly set
+     * to `TRUE` for architectures like ARM.
+     */
+    bool acpi_persistent;
+
     /* TODO Move common fields from CPUArchState here. */
     int cpu_index;
     int cluster_index;
-- 
2.34.1

[PATCH V1 2/4] hw/acpi: Update ACPI CPU Status `is_{present, enabled}` during vCPU hot(un)plug

[Salil Mehta via]

Update the `AcpiCpuStatus` for `is_enabled` and `is_present` accordingly when
vCPUs are hot-plugged or hot-unplugged, taking into account the *persistence*
of the vCPUs.

Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
---
 hw/acpi/cpu.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
index 083c4010c2..700aa855e9 100644
--- a/hw/acpi/cpu.c
+++ b/hw/acpi/cpu.c
@@ -291,6 +291,8 @@ void acpi_cpu_plug_cb(HotplugHandler *hotplug_dev,
     }
 
     cdev->cpu = CPU(dev);
+    cdev->is_present = true;
+    cdev->is_enabled = true;
     if (dev->hotplugged) {
         cdev->is_inserting = true;
         acpi_send_event(DEVICE(hotplug_dev), ACPI_CPU_HOTPLUG_STATUS);
@@ -322,6 +324,11 @@ void acpi_cpu_unplug_cb(CPUHotplugState *cpu_st,
         return;
     }
 
+    cdev->is_enabled = false;
+    if (!acpi_persistent_cpu(CPU(dev))) {
+        cdev->is_present = false;
+    }
+
     cdev->cpu = NULL;
 }
 
-- 
2.34.1

[PATCH V1 3/4] hw/acpi: Reflect ACPI vCPU {present, enabled} states in ACPI _STA.{PRES, ENA} Bits

[Salil Mehta via]

Reflect the ACPI CPU hotplug `is_{present, enabled}` states in the `_STA.PRES`
(presence) and `_STA.ENA` (enabled) bits when the guest kernel evaluates the
ACPI `_STA` method during initialization, as well as when vCPUs are hot-plugged
or hot-unplugged. The presence of unplugged vCPUs may need to be deliberately
*simulated* at the ACPI level to maintain a *persistent* view of vCPUs for the
guest kernel.

Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
---
 hw/acpi/cpu.c | 26 ++++++++++++++++++++++----
 1 file changed, 22 insertions(+), 4 deletions(-)

diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
index 700aa855e9..23ea2b9c70 100644
--- a/hw/acpi/cpu.c
+++ b/hw/acpi/cpu.c
@@ -63,10 +63,11 @@ static uint64_t cpu_hotplug_rd(void *opaque, hwaddr addr, unsigned size)
     cdev = &cpu_st->devs[cpu_st->selector];
     switch (addr) {
     case ACPI_CPU_FLAGS_OFFSET_RW: /* pack and return is_* fields */
-        val |= cdev->cpu ? 1 : 0;
+        val |= cdev->is_enabled ? 1 : 0;
         val |= cdev->is_inserting ? 2 : 0;
         val |= cdev->is_removing  ? 4 : 0;
         val |= cdev->fw_remove  ? 16 : 0;
+        val |= cdev->is_present ? 32 : 0;
         trace_cpuhp_acpi_read_flags(cpu_st->selector, val);
         break;
     case ACPI_CPU_CMD_DATA_OFFSET_RW:
@@ -376,6 +377,7 @@ const VMStateDescription vmstate_cpu_hotplug = {
 #define CPU_REMOVE_EVENT  "CRMV"
 #define CPU_EJECT_EVENT   "CEJ0"
 #define CPU_FW_EJECT_EVENT "CEJF"
+#define CPU_PRESENT       "CPRS"
 
 void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts,
                     build_madt_cpu_fn build_madt_cpu, hwaddr base_addr,
@@ -436,7 +438,9 @@ void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts,
         aml_append(field, aml_named_field(CPU_EJECT_EVENT, 1));
         /* tell firmware to do device eject, write only */
         aml_append(field, aml_named_field(CPU_FW_EJECT_EVENT, 1));
-        aml_append(field, aml_reserved_field(3));
+        /* 1 if present, read only */
+        aml_append(field, aml_named_field(CPU_PRESENT, 1));
+        aml_append(field, aml_reserved_field(2));
         aml_append(field, aml_named_field(CPU_COMMAND, 8));
         aml_append(cpu_ctrl_dev, field);
 
@@ -466,6 +470,7 @@ void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts,
         Aml *ctrl_lock = aml_name("%s.%s", cphp_res_path, CPU_LOCK);
         Aml *cpu_selector = aml_name("%s.%s", cphp_res_path, CPU_SELECTOR);
         Aml *is_enabled = aml_name("%s.%s", cphp_res_path, CPU_ENABLED);
+        Aml *is_present = aml_name("%s.%s", cphp_res_path, CPU_PRESENT);
         Aml *cpu_cmd = aml_name("%s.%s", cphp_res_path, CPU_COMMAND);
         Aml *cpu_data = aml_name("%s.%s", cphp_res_path, CPU_DATA);
         Aml *ins_evt = aml_name("%s.%s", cphp_res_path, CPU_INSERT_EVENT);
@@ -494,13 +499,26 @@ void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts,
         {
             Aml *idx = aml_arg(0);
             Aml *sta = aml_local(0);
+            Aml *ifctx2;
+            Aml *else_ctx;
 
             aml_append(method, aml_acquire(ctrl_lock, 0xFFFF));
             aml_append(method, aml_store(idx, cpu_selector));
             aml_append(method, aml_store(zero, sta));
-            ifctx = aml_if(aml_equal(is_enabled, one));
+            ifctx = aml_if(aml_equal(is_present, one));
             {
-                aml_append(ifctx, aml_store(aml_int(0xF), sta));
+                ifctx2 = aml_if(aml_equal(is_enabled, one));
+                {
+                    /* cpu is present and enabled */
+                    aml_append(ifctx2, aml_store(aml_int(0xF), sta));
+                }
+                aml_append(ifctx, ifctx2);
+                else_ctx = aml_else();
+                {
+                    /* cpu is present but disabled */
+                    aml_append(else_ctx, aml_store(aml_int(0xD), sta));
+                }
+                aml_append(ifctx, else_ctx);
             }
             aml_append(method, ifctx);
             aml_append(method, aml_release(ctrl_lock));
-- 
2.34.1

[PATCH V1 4/4] hw/acpi: Populate vCPU Hotplug VMSD to migrate `is_{present, enabled}` states

[Salil Mehta via]

The ACPI CPU hotplug states `is_{present, enabled}` must be migrated alongside
other vCPU hotplug states to the destination VM. Therefore, they should be
integrated into the existing CPU Hotplug VM State Description (VMSD) table.
Depending on the architecture and its implementation of CPU hotplug events
(such as ACPI GED, etc.), the CPU hotplug states should be populated
appropriately within their corresponding subsections of the VMSD table.

    "acpi-ged (16)": {
        "ged_state": {
            "sel": "0x00000000"
        },
        [...]
        "acpi-ged/cpuhp": {
            "cpuhp_state": {
                "selector": "0x00000005",
                "command": "0x00",
                "devs": [
                    {
                        "is_inserting": false,
                        "is_removing": false,
                        "is_present": true,
                        "is_enabled": true,
                        "ost_event": "0x00000000",
                        "ost_status": "0x00000000"
                    },
                    {
                        "is_inserting": false,
                        "is_removing": false,
                        "is_present": true,
                        "is_enabled": true,
                        "ost_event": "0x00000000",
                        "ost_status": "0x00000000"
                    },
                    {
                        "is_inserting": false,
                        "is_removing": false,
                        "is_present": true,
                        "is_enabled": true,
                        "ost_event": "0x00000000",
                        "ost_status": "0x00000000"
                    },
                    {
                        "is_inserting": false,
                        "is_removing": false,
                        "is_present": true,
                        "is_enabled": true,
                        "ost_event": "0x00000000",
                        "ost_status": "0x00000000"
                    },
                    {
                        "is_inserting": false,
                        "is_removing": false,
                        "is_present": true,
                        "is_enabled": false,
                        "ost_event": "0x00000000",
                        "ost_status": "0x00000000"
                    },
                    {
                        "is_inserting": false,
                        "is_removing": false,
                        "is_present": true,
                        "is_enabled": false,
                        "ost_event": "0x00000000",
                        "ost_status": "0x00000000"
                    }
                ]
            }
        }
    },

Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
---
 hw/acpi/cpu.c                  |  2 ++
 hw/acpi/generic_event_device.c | 11 +++++++++++
 2 files changed, 13 insertions(+)

diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
index 23ea2b9c70..d34c1e601e 100644
--- a/hw/acpi/cpu.c
+++ b/hw/acpi/cpu.c
@@ -340,6 +340,8 @@ static const VMStateDescription vmstate_cpuhp_sts = {
     .fields = (const VMStateField[]) {
         VMSTATE_BOOL(is_inserting, AcpiCpuStatus),
         VMSTATE_BOOL(is_removing, AcpiCpuStatus),
+        VMSTATE_BOOL(is_present, AcpiCpuStatus),
+        VMSTATE_BOOL(is_enabled, AcpiCpuStatus),
         VMSTATE_UINT32(ost_event, AcpiCpuStatus),
         VMSTATE_UINT32(ost_status, AcpiCpuStatus),
         VMSTATE_END_OF_LIST()
diff --git a/hw/acpi/generic_event_device.c b/hw/acpi/generic_event_device.c
index 15b4c3ebbf..a4d78a534c 100644
--- a/hw/acpi/generic_event_device.c
+++ b/hw/acpi/generic_event_device.c
@@ -331,6 +331,16 @@ static const VMStateDescription vmstate_memhp_state = {
     }
 };
 
+static const VMStateDescription vmstate_cpuhp_state = {
+    .name = "acpi-ged/cpuhp",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .fields      = (VMStateField[]) {
+        VMSTATE_CPU_HOTPLUG(cpuhp_state, AcpiGedState),
+        VMSTATE_END_OF_LIST()
+    }
+};
+
 static const VMStateDescription vmstate_ged_state = {
     .name = "acpi-ged-state",
     .version_id = 1,
@@ -379,6 +389,7 @@ static const VMStateDescription vmstate_acpi_ged = {
     },
     .subsections = (const VMStateDescription * const []) {
         &vmstate_memhp_state,
+        &vmstate_cpuhp_state,
         &vmstate_ghes_state,
         NULL
     }
-- 
2.34.1

Re: [PATCH V1 0/4] Arch agnostic ACPI changes to support vCPU Hotplug (on Archs like ARM)

[maobibo]

With cpu-add/cpu-del command tested on LoongArch system, no migration 
tested. There is no negative influence with LoongArch cpu hotplug.

Regards
Bibo Mao

On 2024/10/15 上午3:22, Salil Mehta via wrote:
> Certain CPU architecture specifications [1][2][3] prohibit changes to the CPUs
> *presence* after the kernel has booted. This is because many system
> initializations depend on the exact CPU count at boot time and do not expect it
> to change afterward. For example, components like interrupt controllers that are
> closely coupled with CPUs, or various per-CPU features, may not support
> configuration changes once the kernel has been initialized.
> 
> This requirement poses a challenge for virtualization features like vCPU
> hotplug. To address this, changes to the ACPI AML are necessary to update the
> `_STA.PRES` (presence) and `_STA.ENA` (enabled) bits accordingly during guest
> initialization, as well as when vCPUs are hot-plugged or hot-unplugged. The
> presence of unplugged vCPUs may need to be deliberately *simulated* at the ACPI
> level to maintain a *persistent* view of vCPUs for the guest kernel.
> 
> This patch set introduces the following features:
> 
> 1. ACPI Interface with Explicit PRESENT and ENABLED CPU States: It allows the
>     guest kernel to evaluate these states using the `_STA` ACPI method.
>     
> 2. Initialization of ACPI CPU States: These states are initialized during
>     `machvirt_init` and when vCPUs are hot-(un)plugged. This enables hotpluggable
>     vCPUs to be exposed to the guest kernel via ACPI.
> 
> 3. Support for Migrating ACPI CPU States: The patch set ensures the migration of
>     the newly introduced `is_{present,enabled}` ACPI CPU states to the
>     destination VM.
> 
> The approach is flexible enough to accommodate ARM-like architectures that
> intend to implement vCPU hotplug functionality. It is suitable for architectures
> facing similar constraints to ARM or those that plan to implement vCPU
> hotplugging independently of hardware support (if available).
> 
> This patch set is derived from the ARM-specific vCPU hotplug implementation [4]
> and includes migration components adaptable to other architectures, following
> suggestions [5] made by Igor Mammedov <imammedo@redhat.com>.
> 
> It can be applied independently, ensuring compatibility with existing hotplug
> support in other architectures. I have tested this patch set in conjunction with
> the ARM-specific vCPU hotplug changes (included in the upcoming RFC V5 [6]), and
> everything worked as expected. I kindly request maintainers of other
> architectures to provide a "Tested-by" after running their respective regression
> tests.
> 
> Many thanks!
> 
> 
> References:
> [1] KVMForum 2023 Presentation: Challenges Revisited in Supporting Virt CPU Hotplug on
>      architectures that don’t Support CPU Hotplug (like ARM64)
>      a. Kernel Link: https://kvm-forum.qemu.org/2023/KVM-forum-cpu-hotplug_7OJ1YyJ.pdf
>      b. Qemu Link:  https://kvm-forum.qemu.org/2023/Challenges_Revisited_in_Supporting_Virt_CPU_Hotplug_-__ii0iNb3.pdf
> [2] KVMForum 2020 Presentation: Challenges in Supporting Virtual CPU Hotplug on
>      SoC Based Systems (like ARM64)
>      Link: https://kvmforum2020.sched.com/event/eE4m
> [3] Check comment 5 in the bugzilla entry
>      Link: https://bugzilla.tianocore.org/show_bug.cgi?id=4481#c5
> [4] [PATCH RFC V4 00/33] Support of Virtual CPU Hotplug for ARMv8 Arch
>      Link: https://lore.kernel.org/qemu-devel/20241009031815.250096-1-salil.mehta@huawei.com/T/#mf32be203baa568a871dc625b732f666a4c4f1e68
> [5] Architecture agnostic ACPI VMSD state migration (Discussion)
>      Link: https://lore.kernel.org/qemu-devel/20240715155436.577d34c5@imammedo.users.ipa.redhat.com/
> [6] Upcoming RFC V5, Support of Virtual CPU Hotplug for ARMv8 Arch
>      Link: https://github.com/salil-mehta/qemu/commits/virt-cpuhp-armv8/rfc-v5
> 
> Salil Mehta (4):
>    hw/acpi: Initialize ACPI Hotplug CPU Status with Support for vCPU
>      `Persistence`
>    hw/acpi: Update ACPI CPU Status `is_{present, enabled}` during vCPU
>      hot(un)plug
>    hw/acpi: Reflect ACPI vCPU {present,enabled} states in ACPI
>      _STA.{PRES,ENA} Bits
>    hw/acpi: Populate vCPU Hotplug VMSD to migrate `is_{present,enabled}`
>      states
> 
>   cpu-target.c         patches.vcpuhp.rfc-v5.arch.agnostic.acpi          |  1 +
>   hw/acpi/cpu.c                  | 70 +++++++++++++++++++++++++++++++---
>   hw/acpi/generic_event_device.c | 11 ++++++
>   include/hw/acpi/cpu.h          | 21 ++++++++++
>   include/hw/core/cpu.h          | 21 ++++++++++
>   5 files changed, 119 insertions(+), 5 deletions(-)
> 

RE: [PATCH V1 0/4] Arch agnostic ACPI changes to support vCPU Hotplug (on Archs like ARM)

[Salil Mehta via]

HI Bibo,

>  From: maobibo <maobibo@loongson.cn>
>  Sent: Tuesday, October 15, 2024 4:31 AM
>  To: Salil Mehta <salil.mehta@huawei.com>; qemu-devel@nongnu.org;
>  qemu-arm@nongnu.org; mst@redhat.com
>  
>  With cpu-add/cpu-del command tested on LoongArch system, no migration
>  tested. There is no negative influence with LoongArch cpu hotplug.

Thanks for the confirmation. 

Just curious are guys still using cpu-{add,del} interface for vCPU hotplug?
I thought it was deprecated for device_{add,del}?

https://wiki.qemu.org/Features/CPUHotplug
https://www.qemu.org/docs/master/system/cpu-hotplug.html


Thanks
Salil.

>  
>  Regards
>  Bibo Mao
>  
>  On 2024/10/15 上午3:22, Salil Mehta via wrote:
>  > Certain CPU architecture specifications [1][2][3] prohibit changes to
>  > the CPUs
>  > *presence* after the kernel has booted. This is because many system
>  > initializations depend on the exact CPU count at boot time and do not
>  > expect it to change afterward. For example, components like interrupt
>  > controllers that are closely coupled with CPUs, or various per-CPU
>  > features, may not support configuration changes once the kernel has
>  been initialized.
>  >
>  > This requirement poses a challenge for virtualization features like
>  > vCPU hotplug. To address this, changes to the ACPI AML are necessary
>  > to update the `_STA.PRES` (presence) and `_STA.ENA` (enabled) bits
>  > accordingly during guest initialization, as well as when vCPUs are
>  > hot-plugged or hot-unplugged. The presence of unplugged vCPUs may
>  need
>  > to be deliberately *simulated* at the ACPI level to maintain a *persistent*
>  view of vCPUs for the guest kernel.
>  >
>  > This patch set introduces the following features:
>  >
>  > 1. ACPI Interface with Explicit PRESENT and ENABLED CPU States: It allows
>  the
>  >     guest kernel to evaluate these states using the `_STA` ACPI method.
>  >
>  > 2. Initialization of ACPI CPU States: These states are initialized during
>  >     `machvirt_init` and when vCPUs are hot-(un)plugged. This enables
>  hotpluggable
>  >     vCPUs to be exposed to the guest kernel via ACPI.
>  >
>  > 3. Support for Migrating ACPI CPU States: The patch set ensures the
>  migration of
>  >     the newly introduced `is_{present,enabled}` ACPI CPU states to the
>  >     destination VM.
>  >
>  > The approach is flexible enough to accommodate ARM-like architectures
>  > that intend to implement vCPU hotplug functionality. It is suitable
>  > for architectures facing similar constraints to ARM or those that plan
>  > to implement vCPU hotplugging independently of hardware support (if
>  available).
>  >
>  > This patch set is derived from the ARM-specific vCPU hotplug
>  > implementation [4] and includes migration components adaptable to
>  > other architectures, following suggestions [5] made by Igor Mammedov
>  <imammedo@redhat.com>.
>  >
>  > It can be applied independently, ensuring compatibility with existing
>  > hotplug support in other architectures. I have tested this patch set
>  > in conjunction with the ARM-specific vCPU hotplug changes (included in
>  > the upcoming RFC V5 [6]), and everything worked as expected. I kindly
>  > request maintainers of other architectures to provide a "Tested-by"
>  > after running their respective regression tests.
>  >
>  > Many thanks!
>  >
>  >
>  > References:
>  > [1] KVMForum 2023 Presentation: Challenges Revisited in Supporting Virt
>  CPU Hotplug on
>  >      architectures that don’t Support CPU Hotplug (like ARM64)
>  >      a. Kernel Link: https://kvm-forum.qemu.org/2023/KVM-forum-cpu-
>  hotplug_7OJ1YyJ.pdf
>  >      b. Qemu Link:
>  > https://kvm-
>  forum.qemu.org/2023/Challenges_Revisited_in_Supporting_Vir
>  > t_CPU_Hotplug_-__ii0iNb3.pdf [2] KVMForum 2020 Presentation:
>  > Challenges in Supporting Virtual CPU Hotplug on
>  >      SoC Based Systems (like ARM64)
>  >      Link: https://kvmforum2020.sched.com/event/eE4m
>  > [3] Check comment 5 in the bugzilla entry
>  >      Link: https://bugzilla.tianocore.org/show_bug.cgi?id=4481#c5
>  > [4] [PATCH RFC V4 00/33] Support of Virtual CPU Hotplug for ARMv8 Arch
>  >      Link:
>  > https://lore.kernel.org/qemu-devel/20241009031815.250096-1-salil.mehta
>  > @huawei.com/T/#mf32be203baa568a871dc625b732f666a4c4f1e68
>  > [5] Architecture agnostic ACPI VMSD state migration (Discussion)
>  >      Link:
>  > https://lore.kernel.org/qemu-
>  devel/20240715155436.577d34c5@imammedo.us
>  > ers.ipa.redhat.com/ [6] Upcoming RFC V5, Support of Virtual CPU
>  > Hotplug for ARMv8 Arch
>  >      Link:
>  > https://github.com/salil-mehta/qemu/commits/virt-cpuhp-armv8/rfc-v5
>  >
>  > Salil Mehta (4):
>  >    hw/acpi: Initialize ACPI Hotplug CPU Status with Support for vCPU
>  >      `Persistence`
>  >    hw/acpi: Update ACPI CPU Status `is_{present, enabled}` during vCPU
>  >      hot(un)plug
>  >    hw/acpi: Reflect ACPI vCPU {present,enabled} states in ACPI
>  >      _STA.{PRES,ENA} Bits
>  >    hw/acpi: Populate vCPU Hotplug VMSD to migrate
>  `is_{present,enabled}`
>  >      states
>  >
>  >   cpu-target.c         patches.vcpuhp.rfc-v5.arch.agnostic.acpi          |  1 +
>  >   hw/acpi/cpu.c                  | 70 +++++++++++++++++++++++++++++++---
>  >   hw/acpi/generic_event_device.c | 11 ++++++
>  >   include/hw/acpi/cpu.h          | 21 ++++++++++
>  >   include/hw/core/cpu.h          | 21 ++++++++++
>  >   5 files changed, 119 insertions(+), 5 deletions(-)
>  >

Re: [PATCH V1 0/4] Arch agnostic ACPI changes to support vCPU Hotplug (on Archs like ARM)

[Miguel Luis]

Hi Salil,

I’ve ran the usual tests successfully of hotplug/unplug from the number of cold-booted cpus up to maxcpus and migration on ARM. Please feel free to add:

Tested-by: Miguel Luis <miguel.luis@oracle.com>

Thanks
Miguel

> On 14 Oct 2024, at 19:22, Salil Mehta <salil.mehta@huawei.com> wrote:
> 
> Certain CPU architecture specifications [1][2][3] prohibit changes to the CPUs
> *presence* after the kernel has booted. This is because many system
> initializations depend on the exact CPU count at boot time and do not expect it
> to change afterward. For example, components like interrupt controllers that are
> closely coupled with CPUs, or various per-CPU features, may not support
> configuration changes once the kernel has been initialized.
> 
> This requirement poses a challenge for virtualization features like vCPU
> hotplug. To address this, changes to the ACPI AML are necessary to update the
> `_STA.PRES` (presence) and `_STA.ENA` (enabled) bits accordingly during guest
> initialization, as well as when vCPUs are hot-plugged or hot-unplugged. The
> presence of unplugged vCPUs may need to be deliberately *simulated* at the ACPI
> level to maintain a *persistent* view of vCPUs for the guest kernel.
> 
> This patch set introduces the following features:
> 
> 1. ACPI Interface with Explicit PRESENT and ENABLED CPU States: It allows the
>   guest kernel to evaluate these states using the `_STA` ACPI method.
> 
> 2. Initialization of ACPI CPU States: These states are initialized during
>   `machvirt_init` and when vCPUs are hot-(un)plugged. This enables hotpluggable
>   vCPUs to be exposed to the guest kernel via ACPI.
> 
> 3. Support for Migrating ACPI CPU States: The patch set ensures the migration of
>   the newly introduced `is_{present,enabled}` ACPI CPU states to the
>   destination VM.
> 
> The approach is flexible enough to accommodate ARM-like architectures that
> intend to implement vCPU hotplug functionality. It is suitable for architectures
> facing similar constraints to ARM or those that plan to implement vCPU
> hotplugging independently of hardware support (if available).
> 
> This patch set is derived from the ARM-specific vCPU hotplug implementation [4]
> and includes migration components adaptable to other architectures, following
> suggestions [5] made by Igor Mammedov <imammedo@redhat.com>.
> 
> It can be applied independently, ensuring compatibility with existing hotplug
> support in other architectures. I have tested this patch set in conjunction with
> the ARM-specific vCPU hotplug changes (included in the upcoming RFC V5 [6]), and
> everything worked as expected. I kindly request maintainers of other
> architectures to provide a "Tested-by" after running their respective regression
> tests.
> 
> Many thanks!
> 
> 
> References:
> [1] KVMForum 2023 Presentation: Challenges Revisited in Supporting Virt CPU Hotplug on
>    architectures that don’t Support CPU Hotplug (like ARM64)
>    a. Kernel Link: https://kvm-forum.qemu.org/2023/KVM-forum-cpu-hotplug_7OJ1YyJ.pdf
>    b. Qemu Link:  https://kvm-forum.qemu.org/2023/Challenges_Revisited_in_Supporting_Virt_CPU_Hotplug_-__ii0iNb3.pdf
> [2] KVMForum 2020 Presentation: Challenges in Supporting Virtual CPU Hotplug on
>    SoC Based Systems (like ARM64)
>    Link: https://kvmforum2020.sched.com/event/eE4m
> [3] Check comment 5 in the bugzilla entry
>    Link: https://bugzilla.tianocore.org/show_bug.cgi?id=4481#c5
> [4] [PATCH RFC V4 00/33] Support of Virtual CPU Hotplug for ARMv8 Arch
>    Link: https://lore.kernel.org/qemu-devel/20241009031815.250096-1-salil.mehta@huawei.com/T/#mf32be203baa568a871dc625b732f666a4c4f1e68
> [5] Architecture agnostic ACPI VMSD state migration (Discussion)
>    Link: https://lore.kernel.org/qemu-devel/20240715155436.577d34c5@imammedo.users.ipa.redhat.com/
> [6] Upcoming RFC V5, Support of Virtual CPU Hotplug for ARMv8 Arch
>    Link: https://github.com/salil-mehta/qemu/commits/virt-cpuhp-armv8/rfc-v5
> 
> Salil Mehta (4):
>  hw/acpi: Initialize ACPI Hotplug CPU Status with Support for vCPU
>    `Persistence`
>  hw/acpi: Update ACPI CPU Status `is_{present, enabled}` during vCPU
>    hot(un)plug
>  hw/acpi: Reflect ACPI vCPU {present,enabled} states in ACPI
>    _STA.{PRES,ENA} Bits
>  hw/acpi: Populate vCPU Hotplug VMSD to migrate `is_{present,enabled}`
>    states
> 
> cpu-target.c         patches.vcpuhp.rfc-v5.arch.agnostic.acpi          |  1 +
> hw/acpi/cpu.c                  | 70 +++++++++++++++++++++++++++++++---
> hw/acpi/generic_event_device.c | 11 ++++++
> include/hw/acpi/cpu.h          | 21 ++++++++++
> include/hw/core/cpu.h          | 21 ++++++++++
> 5 files changed, 119 insertions(+), 5 deletions(-)
> 
> -- 
> 2.34.1
> 

Re: [PATCH V1 0/4] Arch agnostic ACPI changes to support vCPU Hotplug (on Archs like ARM)

[maobibo]

On 2024/10/15 下午10:31, Salil Mehta wrote:
> HI Bibo,
> 
>>   From: maobibo <maobibo@loongson.cn>
>>   Sent: Tuesday, October 15, 2024 4:31 AM
>>   To: Salil Mehta <salil.mehta@huawei.com>; qemu-devel@nongnu.org;
>>   qemu-arm@nongnu.org; mst@redhat.com
>>   
>>   With cpu-add/cpu-del command tested on LoongArch system, no migration
>>   tested. There is no negative influence with LoongArch cpu hotplug.
> 
> Thanks for the confirmation.
> 
> Just curious are guys still using cpu-{add,del} interface for vCPU hotplug?
> I thought it was deprecated for device_{add,del}?
I use the device_add/del hmp command, such as
   # device_add 
la464-loongarch-cpu,socket-id=0,core-id=1,thread-id=0,id=cpu-1
   # device_del cpu-1

Tested-by: Bibo Mao <maobibo@loongson.cn>

Regards
Bibo Mao
> 
> https://wiki.qemu.org/Features/CPUHotplug
> https://www.qemu.org/docs/master/system/cpu-hotplug.html
> 
> 
> Thanks
> Salil.
> 
>>   
>>   Regards
>>   Bibo Mao
>>   
>>   On 2024/10/15 上午3:22, Salil Mehta via wrote:
>>   > Certain CPU architecture specifications [1][2][3] prohibit changes to
>>   > the CPUs
>>   > *presence* after the kernel has booted. This is because many system
>>   > initializations depend on the exact CPU count at boot time and do not
>>   > expect it to change afterward. For example, components like interrupt
>>   > controllers that are closely coupled with CPUs, or various per-CPU
>>   > features, may not support configuration changes once the kernel has
>>   been initialized.
>>   >
>>   > This requirement poses a challenge for virtualization features like
>>   > vCPU hotplug. To address this, changes to the ACPI AML are necessary
>>   > to update the `_STA.PRES` (presence) and `_STA.ENA` (enabled) bits
>>   > accordingly during guest initialization, as well as when vCPUs are
>>   > hot-plugged or hot-unplugged. The presence of unplugged vCPUs may
>>   need
>>   > to be deliberately *simulated* at the ACPI level to maintain a *persistent*
>>   view of vCPUs for the guest kernel.
>>   >
>>   > This patch set introduces the following features:
>>   >
>>   > 1. ACPI Interface with Explicit PRESENT and ENABLED CPU States: It allows
>>   the
>>   >     guest kernel to evaluate these states using the `_STA` ACPI method.
>>   >
>>   > 2. Initialization of ACPI CPU States: These states are initialized during
>>   >     `machvirt_init` and when vCPUs are hot-(un)plugged. This enables
>>   hotpluggable
>>   >     vCPUs to be exposed to the guest kernel via ACPI.
>>   >
>>   > 3. Support for Migrating ACPI CPU States: The patch set ensures the
>>   migration of
>>   >     the newly introduced `is_{present,enabled}` ACPI CPU states to the
>>   >     destination VM.
>>   >
>>   > The approach is flexible enough to accommodate ARM-like architectures
>>   > that intend to implement vCPU hotplug functionality. It is suitable
>>   > for architectures facing similar constraints to ARM or those that plan
>>   > to implement vCPU hotplugging independently of hardware support (if
>>   available).
>>   >
>>   > This patch set is derived from the ARM-specific vCPU hotplug
>>   > implementation [4] and includes migration components adaptable to
>>   > other architectures, following suggestions [5] made by Igor Mammedov
>>   <imammedo@redhat.com>.
>>   >
>>   > It can be applied independently, ensuring compatibility with existing
>>   > hotplug support in other architectures. I have tested this patch set
>>   > in conjunction with the ARM-specific vCPU hotplug changes (included in
>>   > the upcoming RFC V5 [6]), and everything worked as expected. I kindly
>>   > request maintainers of other architectures to provide a "Tested-by"
>>   > after running their respective regression tests.
>>   >
>>   > Many thanks!
>>   >
>>   >
>>   > References:
>>   > [1] KVMForum 2023 Presentation: Challenges Revisited in Supporting Virt
>>   CPU Hotplug on
>>   >      architectures that don’t Support CPU Hotplug (like ARM64)
>>   >      a. Kernel Link: https://kvm-forum.qemu.org/2023/KVM-forum-cpu-
>>   hotplug_7OJ1YyJ.pdf
>>   >      b. Qemu Link:
>>   > https://kvm-
>>   forum.qemu.org/2023/Challenges_Revisited_in_Supporting_Vir
>>   > t_CPU_Hotplug_-__ii0iNb3.pdf [2] KVMForum 2020 Presentation:
>>   > Challenges in Supporting Virtual CPU Hotplug on
>>   >      SoC Based Systems (like ARM64)
>>   >      Link: https://kvmforum2020.sched.com/event/eE4m
>>   > [3] Check comment 5 in the bugzilla entry
>>   >      Link: https://bugzilla.tianocore.org/show_bug.cgi?id=4481#c5
>>   > [4] [PATCH RFC V4 00/33] Support of Virtual CPU Hotplug for ARMv8 Arch
>>   >      Link:
>>   > https://lore.kernel.org/qemu-devel/20241009031815.250096-1-salil.mehta
>>   > @huawei.com/T/#mf32be203baa568a871dc625b732f666a4c4f1e68
>>   > [5] Architecture agnostic ACPI VMSD state migration (Discussion)
>>   >      Link:
>>   > https://lore.kernel.org/qemu-
>>   devel/20240715155436.577d34c5@imammedo.us
>>   > ers.ipa.redhat.com/ [6] Upcoming RFC V5, Support of Virtual CPU
>>   > Hotplug for ARMv8 Arch
>>   >      Link:
>>   > https://github.com/salil-mehta/qemu/commits/virt-cpuhp-armv8/rfc-v5
>>   >
>>   > Salil Mehta (4):
>>   >    hw/acpi: Initialize ACPI Hotplug CPU Status with Support for vCPU
>>   >      `Persistence`
>>   >    hw/acpi: Update ACPI CPU Status `is_{present, enabled}` during vCPU
>>   >      hot(un)plug
>>   >    hw/acpi: Reflect ACPI vCPU {present,enabled} states in ACPI
>>   >      _STA.{PRES,ENA} Bits
>>   >    hw/acpi: Populate vCPU Hotplug VMSD to migrate
>>   `is_{present,enabled}`
>>   >      states
>>   >
>>   >   cpu-target.c         patches.vcpuhp.rfc-v5.arch.agnostic.acpi          |  1 +
>>   >   hw/acpi/cpu.c                  | 70 +++++++++++++++++++++++++++++++---
>>   >   hw/acpi/generic_event_device.c | 11 ++++++
>>   >   include/hw/acpi/cpu.h          | 21 ++++++++++
>>   >   include/hw/core/cpu.h          | 21 ++++++++++
>>   >   5 files changed, 119 insertions(+), 5 deletions(-)
>>   >
> 

Re: [PATCH V1 1/4] hw/acpi: Initialize ACPI Hotplug CPU Status with Support for vCPU `Persistence`

[Gustavo Romero]

Hi Salil,

On 10/14/24 16:22, Salil Mehta wrote:
> Certain CPU architecture specifications [1][2][3] prohibit changes to CPU
> presence after the kernel has booted. This limitation exists because many system
> initializations rely on the exact CPU count at boot time and do not expect it to
> change later. For example, components like interrupt controllers, which are
> closely tied to CPUs, or various per-CPU features, may not support configuration
> changes once the kernel has been initialized. This presents a challenge for
> virtualization features such as vCPU hotplug.
> 
> To address this issue, introduce an `is_enabled` state in the `AcpiCpuStatus`,
> which reflects whether a vCPU has been hot-plugged or hot-unplugged in QEMU,
> marking it as (un)available in the Guest Kernel. The `is_present` state should
> be set based on the `acpi_persistent` flag. In cases where unplugged vCPUs need
> to be deliberately simulated in the ACPI to maintain a persistent view of vCPUs,
> this flag ensures the guest kernel continues to see those vCPUs.
> 
> Additionally, introduce an `acpi_persistent` property that can be used to
> initialize the ACPI vCPU presence state accordingly. Architectures requiring
> ACPI to expose a persistent view of vCPUs can override its default value. Refer
> to the patch-set implelenting vCPU hotplug support for ARM for more details on

nit: implementation


Cheers,
Gustavo

> its usage.
> 
> References:
> [1] KVMForum 2023 Presentation: Challenges Revisited in Supporting Virt CPU Hotplug on
>      architectures that don’t Support CPU Hotplug (like ARM64)
>      a. Kernel Link: https://kvm-forum.qemu.org/2023/KVM-forum-cpu-hotplug_7OJ1YyJ.pdf
>      b. Qemu Link:  https://kvm-forum.qemu.org/2023/Challenges_Revisited_in_Supporting_Virt_CPU_Hotplug_-__ii0iNb3.pdf
> [2] KVMForum 2020 Presentation: Challenges in Supporting Virtual CPU Hotplug on
>      SoC Based Systems (like ARM64)
>      Link: https://kvmforum2020.sched.com/event/eE4m
> [3] Check comment 5 in the bugzilla entry
>      Link: https://bugzilla.tianocore.org/show_bug.cgi?id=4481#c5
> 
> Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> ---
>   cpu-target.c          |  1 +
>   hw/acpi/cpu.c         | 35 ++++++++++++++++++++++++++++++++++-
>   include/hw/acpi/cpu.h | 21 +++++++++++++++++++++
>   include/hw/core/cpu.h | 21 +++++++++++++++++++++
>   4 files changed, 77 insertions(+), 1 deletion(-)
> 
> diff --git a/cpu-target.c b/cpu-target.c
> index 499facf774..c8a29ab495 100644
> --- a/cpu-target.c
> +++ b/cpu-target.c
> @@ -200,6 +200,7 @@ static Property cpu_common_props[] = {
>        */
>       DEFINE_PROP_LINK("memory", CPUState, memory, TYPE_MEMORY_REGION,
>                        MemoryRegion *),
> +    DEFINE_PROP_BOOL("acpi-persistent", CPUState, acpi_persistent, false),
>   #endif
>       DEFINE_PROP_END_OF_LIST(),
>   };
> diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
> index 5cb60ca8bc..083c4010c2 100644
> --- a/hw/acpi/cpu.c
> +++ b/hw/acpi/cpu.c
> @@ -225,7 +225,40 @@ void cpu_hotplug_hw_init(MemoryRegion *as, Object *owner,
>       state->dev_count = id_list->len;
>       state->devs = g_new0(typeof(*state->devs), state->dev_count);
>       for (i = 0; i < id_list->len; i++) {
> -        state->devs[i].cpu =  CPU(id_list->cpus[i].cpu);
> +        struct CPUState *cpu = CPU(id_list->cpus[i].cpu);
> +        /*
> +         * In most architectures, CPUs that are marked as ACPI 'present' are
> +         * also ACPI 'enabled' by default. These states remain consistent at
> +         * both the QOM and ACPI levels.
> +         */
> +        if (cpu) {
> +            state->devs[i].is_enabled = true;
> +            state->devs[i].is_present = true;
> +            state->devs[i].cpu = cpu;
> +        } else {
> +            state->devs[i].is_enabled = false;
> +            /*
> +             * In some architectures, even 'unplugged' or 'disabled' QOM CPUs
> +             * may be exposed as ACPI 'present.' This approach provides a
> +             * persistent view of the vCPUs to the guest kernel. This could be
> +             * due to an architectural constraint that requires every per-CPU
> +             * component to be present at boot time, meaning the exact count of
> +             * vCPUs must be known and cannot be altered after the kernel has
> +             * booted. As a result, the vCPU states at the QOM and ACPI levels
> +             * might become inconsistent. However, in such cases, the presence
> +             * of vCPUs has been deliberately simulated at the ACPI level.
> +             */
> +            if (acpi_persistent_cpu(first_cpu)) {
> +                state->devs[i].is_present = true;
> +                /*
> +                 * `CPUHotplugState::AcpiCpuStatus::cpu` becomes insignificant
> +                 * in this case
> +                 */
> +            } else {
> +                state->devs[i].is_present = false;
> +                state->devs[i].cpu = cpu;
> +            }
> +        }
>           state->devs[i].arch_id = id_list->cpus[i].arch_id;
>       }
>       memory_region_init_io(&state->ctrl_reg, owner, &cpu_hotplug_ops, state,
> diff --git a/include/hw/acpi/cpu.h b/include/hw/acpi/cpu.h
> index 32654dc274..bd3f9973c9 100644
> --- a/include/hw/acpi/cpu.h
> +++ b/include/hw/acpi/cpu.h
> @@ -26,6 +26,8 @@ typedef struct AcpiCpuStatus {
>       uint64_t arch_id;
>       bool is_inserting;
>       bool is_removing;
> +    bool is_present;
> +    bool is_enabled;
>       bool fw_remove;
>       uint32_t ost_event;
>       uint32_t ost_status;
> @@ -75,4 +77,23 @@ extern const VMStateDescription vmstate_cpu_hotplug;
>       VMSTATE_STRUCT(cpuhp, state, 1, \
>                      vmstate_cpu_hotplug, CPUHotplugState)
>   
> +/**
> + * acpi_persistent_cpu:
> + * @cpu: The vCPU to check
> + *
> + * Checks if the vCPU state should always be reflected as *present* via ACPI
> + * to the Guest. By default, this is False on all architectures and has to be
> + * explicity set during initialization.
> + *
> + * Returns: True if it is ACPI 'persistent' CPU
> + *
> + */
> +static inline bool acpi_persistent_cpu(CPUState *cpu)
> +{
> +    /*
> +     * returns if 'Presence' of the vCPU is persistent and should be simulated
> +     * via ACPI even after vCPUs have been unplugged in QOM
> +     */
> +    return cpu && cpu->acpi_persistent;
> +}
>   #endif
> diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
> index 04e9ad4996..299e96c45b 100644
> --- a/include/hw/core/cpu.h
> +++ b/include/hw/core/cpu.h
> @@ -542,6 +542,27 @@ struct CPUState {
>       CPUPluginState *plugin_state;
>   #endif
>   
> +    /*
> +     * To implement the vCPU hotplug feature (which simulates CPU hotplug
> +     * behavior), we need to dynamically create and destroy QOM vCPU objects,
> +     * and (de)associate them with pre-existing KVM vCPUs while (un)parking the
> +     * KVM vCPU context. One challenge is ensuring that these dynamically
> +     * appearing or disappearing QOM vCPU objects are accurately reflected
> +     * through ACPI to the Guest Kernel. Due to architectural constraints,
> +     * changing the number of vCPUs after the guest kernel has booted may not
> +     * always be possible.
> +     *
> +     * In certain architectures, to provide the guest kernel with a *persistent*
> +     * view of vCPU presence, even when the QOM does not have a corresponding
> +     * vCPU object, ACPI may simulate the presence of vCPUs by marking them as
> +     * ACPI-disabled. This is achieved by setting `_STA.PRES=True` and
> +     * `_STA.Ena=False` for unplugged vCPUs in QEMU's QOM.
> +     *
> +     * By default, this flag is set to `FALSE`, and it must be explicitly set
> +     * to `TRUE` for architectures like ARM.
> +     */
> +    bool acpi_persistent;
> +
>       /* TODO Move common fields from CPUArchState here. */
>       int cpu_index;
>       int cluster_index;

Re: [PATCH V1 1/4] hw/acpi: Initialize ACPI Hotplug CPU Status with Support for vCPU `Persistence`

[Gavin Shan]

On 10/15/24 5:22 AM, Salil Mehta wrote:
> Certain CPU architecture specifications [1][2][3] prohibit changes to CPU
> presence after the kernel has booted. This limitation exists because many system
> initializations rely on the exact CPU count at boot time and do not expect it to
> change later. For example, components like interrupt controllers, which are
> closely tied to CPUs, or various per-CPU features, may not support configuration
> changes once the kernel has been initialized. This presents a challenge for
> virtualization features such as vCPU hotplug.
> 
> To address this issue, introduce an `is_enabled` state in the `AcpiCpuStatus`,
> which reflects whether a vCPU has been hot-plugged or hot-unplugged in QEMU,
> marking it as (un)available in the Guest Kernel. The `is_present` state should
> be set based on the `acpi_persistent` flag. In cases where unplugged vCPUs need
> to be deliberately simulated in the ACPI to maintain a persistent view of vCPUs,
> this flag ensures the guest kernel continues to see those vCPUs.
> 
> Additionally, introduce an `acpi_persistent` property that can be used to
> initialize the ACPI vCPU presence state accordingly. Architectures requiring
> ACPI to expose a persistent view of vCPUs can override its default value. Refer
> to the patch-set implelenting vCPU hotplug support for ARM for more details on
> its usage.
> 
> References:
> [1] KVMForum 2023 Presentation: Challenges Revisited in Supporting Virt CPU Hotplug on
>      architectures that don’t Support CPU Hotplug (like ARM64)
>      a. Kernel Link: https://kvm-forum.qemu.org/2023/KVM-forum-cpu-hotplug_7OJ1YyJ.pdf
>      b. Qemu Link:  https://kvm-forum.qemu.org/2023/Challenges_Revisited_in_Supporting_Virt_CPU_Hotplug_-__ii0iNb3.pdf
> [2] KVMForum 2020 Presentation: Challenges in Supporting Virtual CPU Hotplug on
>      SoC Based Systems (like ARM64)
>      Link: https://kvmforum2020.sched.com/event/eE4m
> [3] Check comment 5 in the bugzilla entry
>      Link: https://bugzilla.tianocore.org/show_bug.cgi?id=4481#c5
> 
> Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> ---
>   cpu-target.c          |  1 +
>   hw/acpi/cpu.c         | 35 ++++++++++++++++++++++++++++++++++-
>   include/hw/acpi/cpu.h | 21 +++++++++++++++++++++
>   include/hw/core/cpu.h | 21 +++++++++++++++++++++
>   4 files changed, 77 insertions(+), 1 deletion(-)
> 
> diff --git a/cpu-target.c b/cpu-target.c
> index 499facf774..c8a29ab495 100644
> --- a/cpu-target.c
> +++ b/cpu-target.c
> @@ -200,6 +200,7 @@ static Property cpu_common_props[] = {
>        */
>       DEFINE_PROP_LINK("memory", CPUState, memory, TYPE_MEMORY_REGION,
>                        MemoryRegion *),
> +    DEFINE_PROP_BOOL("acpi-persistent", CPUState, acpi_persistent, false),
>   #endif
>       DEFINE_PROP_END_OF_LIST(),
>   };

Would the property be consistent on all CPUs? I mean it's invalid for this
property to be true on the first CPU while it's false on other CPUs. If my
understanding is correct, this property would be part of MachineState instead
of CPUState, and it's not configurable. Is there a particular reason why the
property should be tied with CPUState and configurable?

Besides, the property name "acpi-persistent" isn't indicative enough. CPU
objects can be described through ACPI table or device tree node. So a more
generic property name may be needed, for example "always_present"? If we
need move this property from CPUState to MachineStaste or MachineClass,
the name would be "cpu_always_present" or something like that.

> diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
> index 5cb60ca8bc..083c4010c2 100644
> --- a/hw/acpi/cpu.c
> +++ b/hw/acpi/cpu.c
> @@ -225,7 +225,40 @@ void cpu_hotplug_hw_init(MemoryRegion *as, Object *owner,
>       state->dev_count = id_list->len;
>       state->devs = g_new0(typeof(*state->devs), state->dev_count);
>       for (i = 0; i < id_list->len; i++) {
> -        state->devs[i].cpu =  CPU(id_list->cpus[i].cpu);
> +        struct CPUState *cpu = CPU(id_list->cpus[i].cpu);
> +        /*
> +         * In most architectures, CPUs that are marked as ACPI 'present' are
> +         * also ACPI 'enabled' by default. These states remain consistent at
> +         * both the QOM and ACPI levels.
> +         */
> +        if (cpu) {
> +            state->devs[i].is_enabled = true;
> +            state->devs[i].is_present = true;
> +            state->devs[i].cpu = cpu;
> +        } else {
> +            state->devs[i].is_enabled = false;
> +            /*
> +             * In some architectures, even 'unplugged' or 'disabled' QOM CPUs
> +             * may be exposed as ACPI 'present.' This approach provides a
> +             * persistent view of the vCPUs to the guest kernel. This could be
> +             * due to an architectural constraint that requires every per-CPU
> +             * component to be present at boot time, meaning the exact count of
> +             * vCPUs must be known and cannot be altered after the kernel has
> +             * booted. As a result, the vCPU states at the QOM and ACPI levels
> +             * might become inconsistent. However, in such cases, the presence
> +             * of vCPUs has been deliberately simulated at the ACPI level.
> +             */
> +            if (acpi_persistent_cpu(first_cpu)) {
> +                state->devs[i].is_present = true;
> +                /*
> +                 * `CPUHotplugState::AcpiCpuStatus::cpu` becomes insignificant
> +                 * in this case
> +                 */
> +            } else {
> +                state->devs[i].is_present = false;
> +                state->devs[i].cpu = cpu;
> +            }
> +        }
>           state->devs[i].arch_id = id_list->cpus[i].arch_id;
>       }
>       memory_region_init_io(&state->ctrl_reg, owner, &cpu_hotplug_ops, state,

The code is already self-explaining and the comments explaining how the property
"acpi-persistent" is handled seems a bit duplicate. If you agree, lets make comments
short and concise.

> diff --git a/include/hw/acpi/cpu.h b/include/hw/acpi/cpu.h
> index 32654dc274..bd3f9973c9 100644
> --- a/include/hw/acpi/cpu.h
> +++ b/include/hw/acpi/cpu.h
> @@ -26,6 +26,8 @@ typedef struct AcpiCpuStatus {
>       uint64_t arch_id;
>       bool is_inserting;
>       bool is_removing;
> +    bool is_present;
> +    bool is_enabled;
>       bool fw_remove;
>       uint32_t ost_event;
>       uint32_t ost_status;
> @@ -75,4 +77,23 @@ extern const VMStateDescription vmstate_cpu_hotplug;
>       VMSTATE_STRUCT(cpuhp, state, 1, \
>                      vmstate_cpu_hotplug, CPUHotplugState)
>   
> +/**
> + * acpi_persistent_cpu:
> + * @cpu: The vCPU to check
> + *
> + * Checks if the vCPU state should always be reflected as *present* via ACPI
> + * to the Guest. By default, this is False on all architectures and has to be
> + * explicity set during initialization.
> + *
> + * Returns: True if it is ACPI 'persistent' CPU
> + *
> + */
> +static inline bool acpi_persistent_cpu(CPUState *cpu)
> +{
> +    /*
> +     * returns if 'Presence' of the vCPU is persistent and should be simulated
> +     * via ACPI even after vCPUs have been unplugged in QOM
> +     */
> +    return cpu && cpu->acpi_persistent;
> +}
>   #endif
> diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
> index 04e9ad4996..299e96c45b 100644
> --- a/include/hw/core/cpu.h
> +++ b/include/hw/core/cpu.h
> @@ -542,6 +542,27 @@ struct CPUState {
>       CPUPluginState *plugin_state;
>   #endif
>   
> +    /*
> +     * To implement the vCPU hotplug feature (which simulates CPU hotplug
> +     * behavior), we need to dynamically create and destroy QOM vCPU objects,
> +     * and (de)associate them with pre-existing KVM vCPUs while (un)parking the
> +     * KVM vCPU context. One challenge is ensuring that these dynamically
> +     * appearing or disappearing QOM vCPU objects are accurately reflected
> +     * through ACPI to the Guest Kernel. Due to architectural constraints,
> +     * changing the number of vCPUs after the guest kernel has booted may not
> +     * always be possible.
> +     *
> +     * In certain architectures, to provide the guest kernel with a *persistent*
> +     * view of vCPU presence, even when the QOM does not have a corresponding
> +     * vCPU object, ACPI may simulate the presence of vCPUs by marking them as
> +     * ACPI-disabled. This is achieved by setting `_STA.PRES=True` and
> +     * `_STA.Ena=False` for unplugged vCPUs in QEMU's QOM.
> +     *
> +     * By default, this flag is set to `FALSE`, and it must be explicitly set
> +     * to `TRUE` for architectures like ARM.
> +     */
> +    bool acpi_persistent;
> +

The comments can be short and concise either. Reader can refer to the
commit log for all the details.

>       /* TODO Move common fields from CPUArchState here. */
>       int cpu_index;
>       int cluster_index;

Thanks,
Gavin

Re: [PATCH V1 1/4] hw/acpi: Initialize ACPI Hotplug CPU Status with Support for vCPU `Persistence`

[Gavin Shan]

On 10/15/24 5:22 AM, Salil Mehta wrote:
> Certain CPU architecture specifications [1][2][3] prohibit changes to CPU
> presence after the kernel has booted. This limitation exists because many system
> initializations rely on the exact CPU count at boot time and do not expect it to
> change later. For example, components like interrupt controllers, which are
> closely tied to CPUs, or various per-CPU features, may not support configuration
> changes once the kernel has been initialized. This presents a challenge for
> virtualization features such as vCPU hotplug.
> 
> To address this issue, introduce an `is_enabled` state in the `AcpiCpuStatus`,
> which reflects whether a vCPU has been hot-plugged or hot-unplugged in QEMU,
> marking it as (un)available in the Guest Kernel. The `is_present` state should
> be set based on the `acpi_persistent` flag. In cases where unplugged vCPUs need
> to be deliberately simulated in the ACPI to maintain a persistent view of vCPUs,
> this flag ensures the guest kernel continues to see those vCPUs.
> 
> Additionally, introduce an `acpi_persistent` property that can be used to
> initialize the ACPI vCPU presence state accordingly. Architectures requiring
> ACPI to expose a persistent view of vCPUs can override its default value. Refer
> to the patch-set implelenting vCPU hotplug support for ARM for more details on
> its usage.
> 
> References:
> [1] KVMForum 2023 Presentation: Challenges Revisited in Supporting Virt CPU Hotplug on
>      architectures that don’t Support CPU Hotplug (like ARM64)
>      a. Kernel Link: https://kvm-forum.qemu.org/2023/KVM-forum-cpu-hotplug_7OJ1YyJ.pdf
>      b. Qemu Link:  https://kvm-forum.qemu.org/2023/Challenges_Revisited_in_Supporting_Virt_CPU_Hotplug_-__ii0iNb3.pdf
> [2] KVMForum 2020 Presentation: Challenges in Supporting Virtual CPU Hotplug on
>      SoC Based Systems (like ARM64)
>      Link: https://kvmforum2020.sched.com/event/eE4m
> [3] Check comment 5 in the bugzilla entry
>      Link: https://bugzilla.tianocore.org/show_bug.cgi?id=4481#c5
> 
> Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> ---
>   cpu-target.c          |  1 +
>   hw/acpi/cpu.c         | 35 ++++++++++++++++++++++++++++++++++-
>   include/hw/acpi/cpu.h | 21 +++++++++++++++++++++
>   include/hw/core/cpu.h | 21 +++++++++++++++++++++
>   4 files changed, 77 insertions(+), 1 deletion(-)
> 

[...]
    
> +/**
> + * acpi_persistent_cpu:
> + * @cpu: The vCPU to check
> + *
> + * Checks if the vCPU state should always be reflected as *present* via ACPI
> + * to the Guest. By default, this is False on all architectures and has to be
> + * explicity set during initialization.
       ^^^^^^^^^
       explicitly

> + *
> + * Returns: True if it is ACPI 'persistent' CPU
> + *
> + */
> +static inline bool acpi_persistent_cpu(CPUState *cpu)
> +{
> +    /*
> +     * returns if 'Presence' of the vCPU is persistent and should be simulated
> +     * via ACPI even after vCPUs have been unplugged in QOM
> +     */
> +    return cpu && cpu->acpi_persistent;
> +}
>   #endif

Thanks,
Gavin

Re: [PATCH V1 1/4] hw/acpi: Initialize ACPI Hotplug CPU Status with Support for vCPU `Persistence`

[Gustavo Romero]

Hi Salil,

On 10/14/24 16:22, Salil Mehta wrote:
> Certain CPU architecture specifications [1][2][3] prohibit changes to CPU
> presence after the kernel has booted. This limitation exists because many system
> initializations rely on the exact CPU count at boot time and do not expect it to
> change later. For example, components like interrupt controllers, which are
> closely tied to CPUs, or various per-CPU features, may not support configuration
> changes once the kernel has been initialized. This presents a challenge for
> virtualization features such as vCPU hotplug.
> 
> To address this issue, introduce an `is_enabled` state in the `AcpiCpuStatus`,
> which reflects whether a vCPU has been hot-plugged or hot-unplugged in QEMU,
> marking it as (un)available in the Guest Kernel. The `is_present` state should
> be set based on the `acpi_persistent` flag. In cases where unplugged vCPUs need
> to be deliberately simulated in the ACPI to maintain a persistent view of vCPUs,
> this flag ensures the guest kernel continues to see those vCPUs.
> 
> Additionally, introduce an `acpi_persistent` property that can be used to
> initialize the ACPI vCPU presence state accordingly. Architectures requiring
> ACPI to expose a persistent view of vCPUs can override its default value. Refer
> to the patch-set implelenting vCPU hotplug support for ARM for more details on
> its usage.
> 
> References:
> [1] KVMForum 2023 Presentation: Challenges Revisited in Supporting Virt CPU Hotplug on
>      architectures that don’t Support CPU Hotplug (like ARM64)
>      a. Kernel Link: https://kvm-forum.qemu.org/2023/KVM-forum-cpu-hotplug_7OJ1YyJ.pdf
>      b. Qemu Link:  https://kvm-forum.qemu.org/2023/Challenges_Revisited_in_Supporting_Virt_CPU_Hotplug_-__ii0iNb3.pdf
> [2] KVMForum 2020 Presentation: Challenges in Supporting Virtual CPU Hotplug on
>      SoC Based Systems (like ARM64)
>      Link: https://kvmforum2020.sched.com/event/eE4m
> [3] Check comment 5 in the bugzilla entry
>      Link: https://bugzilla.tianocore.org/show_bug.cgi?id=4481#c5
> 
> Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> ---
>   cpu-target.c          |  1 +
>   hw/acpi/cpu.c         | 35 ++++++++++++++++++++++++++++++++++-
>   include/hw/acpi/cpu.h | 21 +++++++++++++++++++++
>   include/hw/core/cpu.h | 21 +++++++++++++++++++++
>   4 files changed, 77 insertions(+), 1 deletion(-)
> 
> diff --git a/cpu-target.c b/cpu-target.c
> index 499facf774..c8a29ab495 100644
> --- a/cpu-target.c
> +++ b/cpu-target.c
> @@ -200,6 +200,7 @@ static Property cpu_common_props[] = {
>        */
>       DEFINE_PROP_LINK("memory", CPUState, memory, TYPE_MEMORY_REGION,
>                        MemoryRegion *),
> +    DEFINE_PROP_BOOL("acpi-persistent", CPUState, acpi_persistent, false),
>   #endif
>       DEFINE_PROP_END_OF_LIST(),
>   };
> diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
> index 5cb60ca8bc..083c4010c2 100644
> --- a/hw/acpi/cpu.c
> +++ b/hw/acpi/cpu.c
> @@ -225,7 +225,40 @@ void cpu_hotplug_hw_init(MemoryRegion *as, Object *owner,
>       state->dev_count = id_list->len;
>       state->devs = g_new0(typeof(*state->devs), state->dev_count);
>       for (i = 0; i < id_list->len; i++) {
> -        state->devs[i].cpu =  CPU(id_list->cpus[i].cpu);
> +        struct CPUState *cpu = CPU(id_list->cpus[i].cpu);
> +        /*
> +         * In most architectures, CPUs that are marked as ACPI 'present' are
> +         * also ACPI 'enabled' by default. These states remain consistent at
> +         * both the QOM and ACPI levels.
> +         */
> +        if (cpu) {
> +            state->devs[i].is_enabled = true;
> +            state->devs[i].is_present = true;
> +            state->devs[i].cpu = cpu;
> +        } else {
> +            state->devs[i].is_enabled = false;
> +            /*
> +             * In some architectures, even 'unplugged' or 'disabled' QOM CPUs
> +             * may be exposed as ACPI 'present.' This approach provides a
> +             * persistent view of the vCPUs to the guest kernel. This could be
> +             * due to an architectural constraint that requires every per-CPU
> +             * component to be present at boot time, meaning the exact count of
> +             * vCPUs must be known and cannot be altered after the kernel has
> +             * booted. As a result, the vCPU states at the QOM and ACPI levels
> +             * might become inconsistent. However, in such cases, the presence
> +             * of vCPUs has been deliberately simulated at the ACPI level.
> +             */
> +            if (acpi_persistent_cpu(first_cpu)) {
> +                state->devs[i].is_present = true;
> +                /*
> +                 * `CPUHotplugState::AcpiCpuStatus::cpu` becomes insignificant
> +                 * in this case
> +                 */
> +            } else {
> +                state->devs[i].is_present = false;
> +                state->devs[i].cpu = cpu;

I think it's better to set cpu here explicitly to NULL in both cases
(persistent and non-persistent cases). Also, 'cpu' here is always NULL
since it's inside the else block of "if (cpu)" conditional. So how about
setting cpu to NULL at the end of the else block:

diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
index d34c1e601e..b830c0e85b 100644
--- a/hw/acpi/cpu.c
+++ b/hw/acpi/cpu.c
@@ -251,14 +251,14 @@ void cpu_hotplug_hw_init(MemoryRegion *as, Object 
*owner,
               */
              if (acpi_persistent_cpu(first_cpu)) {
                  state->devs[i].is_present = true;
-                /*
-                 * `CPUHotplugState::AcpiCpuStatus::cpu` becomes 
insignificant
-                 * in this case
-                 */
              } else {
                  state->devs[i].is_present = false;
-                state->devs[i].cpu = cpu;
              }
+            /*
+             * `CPUHotplugState::AcpiCpuStatus::cpu` becomes insignificant
+             * in this case
+             */
+            state->devs[i].cpu = NULL;
          }
          state->devs[i].arch_id = id_list->cpus[i].arch_id;
      }


Cheers,
Gustavo

> +            }
> +        }
>           state->devs[i].arch_id = id_list->cpus[i].arch_id;
>       }
>       memory_region_init_io(&state->ctrl_reg, owner, &cpu_hotplug_ops, state,
> diff --git a/include/hw/acpi/cpu.h b/include/hw/acpi/cpu.h
> index 32654dc274..bd3f9973c9 100644
> --- a/include/hw/acpi/cpu.h
> +++ b/include/hw/acpi/cpu.h
> @@ -26,6 +26,8 @@ typedef struct AcpiCpuStatus {
>       uint64_t arch_id;
>       bool is_inserting;
>       bool is_removing;
> +    bool is_present;
> +    bool is_enabled;
>       bool fw_remove;
>       uint32_t ost_event;
>       uint32_t ost_status;
> @@ -75,4 +77,23 @@ extern const VMStateDescription vmstate_cpu_hotplug;
>       VMSTATE_STRUCT(cpuhp, state, 1, \
>                      vmstate_cpu_hotplug, CPUHotplugState)
>   
> +/**
> + * acpi_persistent_cpu:
> + * @cpu: The vCPU to check
> + *
> + * Checks if the vCPU state should always be reflected as *present* via ACPI
> + * to the Guest. By default, this is False on all architectures and has to be
> + * explicity set during initialization.
> + *
> + * Returns: True if it is ACPI 'persistent' CPU
> + *
> + */
> +static inline bool acpi_persistent_cpu(CPUState *cpu)
> +{
> +    /*
> +     * returns if 'Presence' of the vCPU is persistent and should be simulated
> +     * via ACPI even after vCPUs have been unplugged in QOM
> +     */
> +    return cpu && cpu->acpi_persistent;
> +}
>   #endif
> diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
> index 04e9ad4996..299e96c45b 100644
> --- a/include/hw/core/cpu.h
> +++ b/include/hw/core/cpu.h
> @@ -542,6 +542,27 @@ struct CPUState {
>       CPUPluginState *plugin_state;
>   #endif
>   
> +    /*
> +     * To implement the vCPU hotplug feature (which simulates CPU hotplug
> +     * behavior), we need to dynamically create and destroy QOM vCPU objects,
> +     * and (de)associate them with pre-existing KVM vCPUs while (un)parking the
> +     * KVM vCPU context. One challenge is ensuring that these dynamically
> +     * appearing or disappearing QOM vCPU objects are accurately reflected
> +     * through ACPI to the Guest Kernel. Due to architectural constraints,
> +     * changing the number of vCPUs after the guest kernel has booted may not
> +     * always be possible.
> +     *
> +     * In certain architectures, to provide the guest kernel with a *persistent*
> +     * view of vCPU presence, even when the QOM does not have a corresponding
> +     * vCPU object, ACPI may simulate the presence of vCPUs by marking them as
> +     * ACPI-disabled. This is achieved by setting `_STA.PRES=True` and
> +     * `_STA.Ena=False` for unplugged vCPUs in QEMU's QOM.
> +     *
> +     * By default, this flag is set to `FALSE`, and it must be explicitly set
> +     * to `TRUE` for architectures like ARM.
> +     */
> +    bool acpi_persistent;
> +
>       /* TODO Move common fields from CPUArchState here. */
>       int cpu_index;
>       int cluster_index;

Re: [PATCH V1 3/4] hw/acpi: Reflect ACPI vCPU {present,enabled} states in ACPI _STA.{PRES,ENA} Bits

[Zhao Liu]

Hi Salil,

On Mon, Oct 14, 2024 at 08:22:04PM +0100, Salil Mehta wrote:
> Date: Mon, 14 Oct 2024 20:22:04 +0100
> From: Salil Mehta <salil.mehta@huawei.com>
> Subject: [PATCH V1 3/4] hw/acpi: Reflect ACPI vCPU {present,enabled} states
>  in ACPI _STA.{PRES,ENA} Bits
> X-Mailer: git-send-email 2.34.1
> 
> Reflect the ACPI CPU hotplug `is_{present, enabled}` states in the `_STA.PRES`
> (presence) and `_STA.ENA` (enabled) bits when the guest kernel evaluates the
> ACPI `_STA` method during initialization, as well as when vCPUs are hot-plugged
> or hot-unplugged. The presence of unplugged vCPUs may need to be deliberately
> *simulated* at the ACPI level to maintain a *persistent* view of vCPUs for the
> guest kernel.
> 
> Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> ---
>  hw/acpi/cpu.c | 26 ++++++++++++++++++++++----
>  1 file changed, 22 insertions(+), 4 deletions(-)
> 

It seems this patch changes ACPI table layout and then breaks current
ACPI table qtest. I'm not sure how to do such modifications. Maybe you
should first disable the related checks, then modify the code, update
the qtest, and finally re-enable the checks for qtest. This can help
to avoid any qtest failure due to this patch?

I think it should get Igor's advice on this. :)

Attach the error I met:

▶   2/920 ERROR:../tests/qtest/bios-tables-test.c:553:test_acpi_asl: assertion failed: (all_tables_match) ERROR
▶   3/920 ERROR:../tests/qtest/bios-tables-test.c:553:test_acpi_asl: assertion failed: (all_tables_match) ERROR
  2/920 qemu:qtest+qtest-i386 / qtest-i386/bios-tables-test                                ERROR            1.24s   killed by signal 6 SIGABRT
>>> G_TEST_DBUS_DAEMON=/media/liuzhao/data/qemu-cook/tests/dbus-vmstate-daemon.sh ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 MESON_TEST_ITERATION=1 UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 QTEST_QEMU_BINARY=./qemu-system-i386 MALLOC_PERTURB_=142 MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon QTEST_QEMU_IMG=./qemu-img PYTHON=/media/liuzhao/data/qemu-cook/build/pyvenv/bin/python3 /media/liuzhao/data/qemu-cook/build/tests/qtest/bios-tables-test --tap -k
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
stderr:
acpi-test: Warning! DSDT binary file mismatch. Actual [aml:/tmp/aml-VRT5V2], Expected [aml:tests/data/acpi/x86/pc/DSDT].
See source file tests/qtest/bios-tables-test.c for instructions on how to update expected files.
acpi-test: Warning! DSDT mismatch. Actual [asl:/tmp/asl-TTT5V2.dsl, aml:/tmp/aml-VRT5V2], Expected [asl:/tmp/asl-XXM5V2.dsl, aml:tests/data/acpi/x86/pc/DSDT].
**
ERROR:../tests/qtest/bios-tables-test.c:553:test_acpi_asl: assertion failed: (all_tables_match)

(test program exited with status code -6)
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

  3/920 qemu:qtest+qtest-x86_64 / qtest-x86_64/bios-tables-test                            ERROR            1.25s   killed by signal 6 SIGABRT
>>> G_TEST_DBUS_DAEMON=/media/liuzhao/data/qemu-cook/tests/dbus-vmstate-daemon.sh ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 MESON_TEST_ITERATION=1 UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon QTEST_QEMU_IMG=./qemu-img PYTHON=/media/liuzhao/data/qemu-cook/build/pyvenv/bin/python3 MALLOC_PERTURB_=41 QTEST_QEMU_BINARY=./qemu-system-x86_64 /media/liuzhao/data/qemu-cook/build/tests/qtest/bios-tables-test --tap -k
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
stderr:
acpi-test: Warning! DSDT binary file mismatch. Actual [aml:/tmp/aml-D5K5V2], Expected [aml:tests/data/acpi/x86/pc/DSDT].
See source file tests/qtest/bios-tables-test.c for instructions on how to update expected files.
acpi-test: Warning! DSDT mismatch. Actual [asl:/tmp/asl-G6K5V2.dsl, aml:/tmp/aml-D5K5V2], Expected [asl:/tmp/asl-AQD5V2.dsl, aml:tests/data/acpi/x86/pc/DSDT].
**
ERROR:../tests/qtest/bios-tables-test.c:553:test_acpi_asl: assertion failed: (all_tables_match)

(test program exited with status code -6)


Regards,
Zhao

Re: [PATCH V1 1/4] hw/acpi: Initialize ACPI Hotplug CPU Status with Support for vCPU `Persistence`

[Igor Mammedov]

On Mon, 14 Oct 2024 20:22:02 +0100
Salil Mehta <salil.mehta@huawei.com> wrote:

> Certain CPU architecture specifications [1][2][3] prohibit changes to CPU
                                          ^^^^^^^^^
these do not point to specs but never mind

> presence after the kernel has booted. This limitation exists because many system
> initializations rely on the exact CPU count at boot time and do not expect it to
> change later. For example, components like interrupt controllers, which are
> closely tied to CPUs, or various per-CPU features, may not support configuration
> changes once the kernel has been initialized. This presents a challenge for
> virtualization features such as vCPU hotplug.

well, x86 (incl cpu,interrupt ctrls) also doesn't have architectural hotplug.
It's just OEM managed to implement it regardless and then bothered to make
OS changes to work with that.
It's just ARM community doesn't want to go there at this point of time
but using above reasoning as justification for this series doesn't look good to me.

So what ARM would like to support is not CPU hotplug but rather a fixed
system with standby CPUs (that can be powered on/off on demand).
With ACPI spec amended to support that (MADT present/enabled changes), it's
good enough reason to add 'enabled' handling to acpi cpu-hotplug code instead
of inventing alternative one that would do almost the same.

But lets get rid of (can't/may not) language above and use standby CPUs reasoning
to avoid any confusion.

PS:
I'm taking about hw hotplug (at QEMU level) and not kernel hotplug
(where it's at logical cpu level).

> To address this issue, introduce an `is_enabled` state in the `AcpiCpuStatus`,
> which reflects whether a vCPU has been hot-plugged or hot-unplugged in QEMU,
> marking it as (un)available in the Guest Kernel. 
good so far

> The `is_present` state should
> be set based on the `acpi_persistent` flag. In cases where unplugged vCPUs need
> to be deliberately simulated in the ACPI to maintain a persistent view of vCPUs,
> this flag ensures the guest kernel continues to see those vCPUs.

that's where I have to disagree, vCPU is present when a corresponding QOM object
exists. Faking it's presence will only confuse at the end.

I get that you want to reuse device_add/del interface, but that leads to
pulling the leg here and there to make thing fit. That in short therm
(while someone remembers all tricks) might work for some, but long therm
it's not sustainable).

Maybe instead of pushing device_add/del, we should rather implement
standby CPU model here, as ARM folks expect it to be.
i.e. instead of device_add/del add 'enabled' property to ARM vCPU,
and let management to turn on/off vCPUs on demand.
(and very likely this model could be reused by other sock based platforms)
For end-user it would be the same as device_add/del (i.e. vCPU becomes usable/unsable)

I'd bet it would simplify your ARM CPU hotplug series a lot,
since you won't have to fake vCPU object lifetime and do
non trivial tricks to make it all work.
Which it turn will make ARM hotplug series much more approachable
for reviewers /and whomever comes later across that code/.

Regardless of said, we would still need changes to ACPI cphp code,
see my comments inline.


> Additionally, introduce an `acpi_persistent` property that can be used to
> initialize the ACPI vCPU presence state accordingly. Architectures requiring
> ACPI to expose a persistent view of vCPUs can override its default value. Refer
> to the patch-set implelenting vCPU hotplug support for ARM for more details on
> its usage.
> 
> References:
> [1] KVMForum 2023 Presentation: Challenges Revisited in Supporting Virt CPU Hotplug on
>     architectures that don’t Support CPU Hotplug (like ARM64)
>     a. Kernel Link: https://kvm-forum.qemu.org/2023/KVM-forum-cpu-hotplug_7OJ1YyJ.pdf
>     b. Qemu Link:  https://kvm-forum.qemu.org/2023/Challenges_Revisited_in_Supporting_Virt_CPU_Hotplug_-__ii0iNb3.pdf
> [2] KVMForum 2020 Presentation: Challenges in Supporting Virtual CPU Hotplug on
>     SoC Based Systems (like ARM64)
>     Link: https://kvmforum2020.sched.com/event/eE4m
> [3] Check comment 5 in the bugzilla entry
>     Link: https://bugzilla.tianocore.org/show_bug.cgi?id=4481#c5
> 
> Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> ---
>  cpu-target.c          |  1 +
>  hw/acpi/cpu.c         | 35 ++++++++++++++++++++++++++++++++++-
>  include/hw/acpi/cpu.h | 21 +++++++++++++++++++++
>  include/hw/core/cpu.h | 21 +++++++++++++++++++++
>  4 files changed, 77 insertions(+), 1 deletion(-)
> 
> diff --git a/cpu-target.c b/cpu-target.c
> index 499facf774..c8a29ab495 100644
> --- a/cpu-target.c
> +++ b/cpu-target.c
> @@ -200,6 +200,7 @@ static Property cpu_common_props[] = {
>       */
>      DEFINE_PROP_LINK("memory", CPUState, memory, TYPE_MEMORY_REGION,
>                       MemoryRegion *),
> +    DEFINE_PROP_BOOL("acpi-persistent", CPUState, acpi_persistent, false),

I agree with Gavin, it's not CPU property/business, but a platform one.

Pass it as argument to cpu_hotplug_hw_init(),
and maybe rename to always_present.
Then make sure that it's configurable in GED (which calls the function),
and just turn it on for arm/virt machine.
Other platforms might want to use x86 approach with GED and have
vCPU actually disappearing. /loongson and maybe risc-v/

>  #endif
>      DEFINE_PROP_END_OF_LIST(),
>  };
> diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
> index 5cb60ca8bc..083c4010c2 100644
> --- a/hw/acpi/cpu.c
> +++ b/hw/acpi/cpu.c
> @@ -225,7 +225,40 @@ void cpu_hotplug_hw_init(MemoryRegion *as, Object *owner,
>      state->dev_count = id_list->len;
>      state->devs = g_new0(typeof(*state->devs), state->dev_count);
>      for (i = 0; i < id_list->len; i++) {
> -        state->devs[i].cpu =  CPU(id_list->cpus[i].cpu);
> +        struct CPUState *cpu = CPU(id_list->cpus[i].cpu);
> +        /*
> +         * In most architectures, CPUs that are marked as ACPI 'present' are
> +         * also ACPI 'enabled' by default. These states remain consistent at
> +         * both the QOM and ACPI levels.
> +         */
> +        if (cpu) {
> +            state->devs[i].is_enabled = true;
> +            state->devs[i].is_present = true;
> +            state->devs[i].cpu = cpu;
> +        } else {
> +            state->devs[i].is_enabled = false;
> +            /*
> +             * In some architectures, even 'unplugged' or 'disabled' QOM CPUs
> +             * may be exposed as ACPI 'present.' This approach provides a
> +             * persistent view of the vCPUs to the guest kernel. This could be
> +             * due to an architectural constraint that requires every per-CPU
> +             * component to be present at boot time, meaning the exact count of
> +             * vCPUs must be known and cannot be altered after the kernel has
> +             * booted. As a result, the vCPU states at the QOM and ACPI levels
> +             * might become inconsistent. However, in such cases, the presence
> +             * of vCPUs has been deliberately simulated at the ACPI level.
> +             */

if cpus are not 'simulated', you will not need comments explaining that all
over place and whole hunk would likely go away.

> +            if (acpi_persistent_cpu(first_cpu)) {
> +                state->devs[i].is_present = true;
> +                /*
> +                 * `CPUHotplugState::AcpiCpuStatus::cpu` becomes insignificant
> +                 * in this case
> +                 */
> +            } else {
> +                state->devs[i].is_present = false;
> +                state->devs[i].cpu = cpu;
> +            }
> +        }
>          state->devs[i].arch_id = id_list->cpus[i].arch_id;
>      }
>      memory_region_init_io(&state->ctrl_reg, owner, &cpu_hotplug_ops, state,
> diff --git a/include/hw/acpi/cpu.h b/include/hw/acpi/cpu.h
> index 32654dc274..bd3f9973c9 100644
> --- a/include/hw/acpi/cpu.h
> +++ b/include/hw/acpi/cpu.h
> @@ -26,6 +26,8 @@ typedef struct AcpiCpuStatus {
>      uint64_t arch_id;
>      bool is_inserting;
>      bool is_removing;
> +    bool is_present;
with always_present, it might be better to move field to CPUHotplugState
as it's not per vCPU anymore, and in standby case state->devs[i].cpu
should work as implicit present flag. (see below wrt doc first comment)

> +    bool is_enabled;
I'd move introduction of this field into a separate patch.

BTW: new ABI/fields accessible by guest should be described in
docs/specs/acpi_cpu_hotplug.rst.
It would be better to have the spec as patch 1st, that we all agree on
and then follow with implementation.
And also include there an expected workflow for standby case. 

>      bool fw_remove;
>      uint32_t ost_event;
>      uint32_t ost_status;
> @@ -75,4 +77,23 @@ extern const VMStateDescription vmstate_cpu_hotplug;
>      VMSTATE_STRUCT(cpuhp, state, 1, \
>                     vmstate_cpu_hotplug, CPUHotplugState)
>  
> +/**
> + * acpi_persistent_cpu:
> + * @cpu: The vCPU to check
> + *
> + * Checks if the vCPU state should always be reflected as *present* via ACPI
> + * to the Guest. By default, this is False on all architectures and has to be
> + * explicity set during initialization.
> + *
> + * Returns: True if it is ACPI 'persistent' CPU
> + *
> + */
> +static inline bool acpi_persistent_cpu(CPUState *cpu)
> +{
> +    /*
> +     * returns if 'Presence' of the vCPU is persistent and should be simulated
> +     * via ACPI even after vCPUs have been unplugged in QOM
> +     */
> +    return cpu && cpu->acpi_persistent;
> +}
>  #endif
> diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
> index 04e9ad4996..299e96c45b 100644
> --- a/include/hw/core/cpu.h
> +++ b/include/hw/core/cpu.h
> @@ -542,6 +542,27 @@ struct CPUState {
>      CPUPluginState *plugin_state;
>  #endif
>  
> +    /*
> +     * To implement the vCPU hotplug feature (which simulates CPU hotplug
> +     * behavior), we need to dynamically create and destroy QOM vCPU objects,
> +     * and (de)associate them with pre-existing KVM vCPUs while (un)parking the
> +     * KVM vCPU context. One challenge is ensuring that these dynamically
> +     * appearing or disappearing QOM vCPU objects are accurately reflected
> +     * through ACPI to the Guest Kernel. Due to architectural constraints,
> +     * changing the number of vCPUs after the guest kernel has booted may not
> +     * always be possible.
> +     *
> +     * In certain architectures, to provide the guest kernel with a *persistent*
> +     * view of vCPU presence, even when the QOM does not have a corresponding
> +     * vCPU object, ACPI may simulate the presence of vCPUs by marking them as
> +     * ACPI-disabled. This is achieved by setting `_STA.PRES=True` and
> +     * `_STA.Ena=False` for unplugged vCPUs in QEMU's QOM.
> +     *
> +     * By default, this flag is set to `FALSE`, and it must be explicitly set
> +     * to `TRUE` for architectures like ARM.
> +     */
> +    bool acpi_persistent;
> +
>      /* TODO Move common fields from CPUArchState here. */
>      int cpu_index;
>      int cluster_index;

Re: [PATCH V1 2/4] hw/acpi: Update ACPI CPU Status `is_{present, enabled}` during vCPU hot(un)plug

[Igor Mammedov]

On Mon, 14 Oct 2024 20:22:03 +0100
Salil Mehta <salil.mehta@huawei.com> wrote:

> Update the `AcpiCpuStatus` for `is_enabled` and `is_present` accordingly when
> vCPUs are hot-plugged or hot-unplugged, taking into account the *persistence*
> of the vCPUs.
> 
> Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> ---
>  hw/acpi/cpu.c | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
> index 083c4010c2..700aa855e9 100644
> --- a/hw/acpi/cpu.c
> +++ b/hw/acpi/cpu.c
> @@ -291,6 +291,8 @@ void acpi_cpu_plug_cb(HotplugHandler *hotplug_dev,
>      }
>  
>      cdev->cpu = CPU(dev);
> +    cdev->is_present = true;
> +    cdev->is_enabled = true;

hmm, if cpu is always present, then these fields are redundant
since
  (!cdev->cpu) == present
and
  then is_enabled could be fetched from cdev->cpu directly

>      if (dev->hotplugged) {
>          cdev->is_inserting = true;
>          acpi_send_event(DEVICE(hotplug_dev), ACPI_CPU_HOTPLUG_STATUS);
> @@ -322,6 +324,11 @@ void acpi_cpu_unplug_cb(CPUHotplugState *cpu_st,
>          return;
>      }
>  
> +    cdev->is_enabled = false;
> +    if (!acpi_persistent_cpu(CPU(dev))) {
> +        cdev->is_present = false;
> +    }

and other way around works as well.

then we don't have to carry around extra state and making sure that it's in sync/migrated.

> +
>      cdev->cpu = NULL;
>  }
>  

Re: [PATCH V1 3/4] hw/acpi: Reflect ACPI vCPU {present,enabled} states in ACPI _STA.{PRES,ENA} Bits

[Igor Mammedov]

On Fri, 18 Oct 2024 13:12:52 +0800
Zhao Liu <zhao1.liu@intel.com> wrote:

> Hi Salil,
> 
> On Mon, Oct 14, 2024 at 08:22:04PM +0100, Salil Mehta wrote:
> > Date: Mon, 14 Oct 2024 20:22:04 +0100
> > From: Salil Mehta <salil.mehta@huawei.com>
> > Subject: [PATCH V1 3/4] hw/acpi: Reflect ACPI vCPU {present,enabled} states
> >  in ACPI _STA.{PRES,ENA} Bits
> > X-Mailer: git-send-email 2.34.1
> > 
> > Reflect the ACPI CPU hotplug `is_{present, enabled}` states in the `_STA.PRES`
> > (presence) and `_STA.ENA` (enabled) bits when the guest kernel evaluates the
> > ACPI `_STA` method during initialization, as well as when vCPUs are hot-plugged
> > or hot-unplugged. The presence of unplugged vCPUs may need to be deliberately
> > *simulated* at the ACPI level to maintain a *persistent* view of vCPUs for the
> > guest kernel.
> > 
> > Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> > ---
> >  hw/acpi/cpu.c | 26 ++++++++++++++++++++++----
> >  1 file changed, 22 insertions(+), 4 deletions(-)
> >   
> 
> It seems this patch changes ACPI table layout and then breaks current
> ACPI table qtest. I'm not sure how to do such modifications. Maybe you
> should first disable the related checks, then modify the code, update
> the qtest, and finally re-enable the checks for qtest. This can help
> to avoid any qtest failure due to this patch?

see comment at the top of tests/qtest/bios-tables-test.c

> 
> I think it should get Igor's advice on this. :)
> 
> Attach the error I met:
> 
> ▶   2/920 ERROR:../tests/qtest/bios-tables-test.c:553:test_acpi_asl: assertion failed: (all_tables_match) ERROR
> ▶   3/920 ERROR:../tests/qtest/bios-tables-test.c:553:test_acpi_asl: assertion failed: (all_tables_match) ERROR
>   2/920 qemu:qtest+qtest-i386 / qtest-i386/bios-tables-test                                ERROR            1.24s   killed by signal 6 SIGABRT
> >>> G_TEST_DBUS_DAEMON=/media/liuzhao/data/qemu-cook/tests/dbus-vmstate-daemon.sh ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 MESON_TEST_ITERATION=1 UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 QTEST_QEMU_BINARY=./qemu-system-i386 MALLOC_PERTURB_=142 MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon QTEST_QEMU_IMG=./qemu-img PYTHON=/media/liuzhao/data/qemu-cook/build/pyvenv/bin/python3 /media/liuzhao/data/qemu-cook/build/tests/qtest/bios-tables-test --tap -k  
> ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
> stderr:
> acpi-test: Warning! DSDT binary file mismatch. Actual [aml:/tmp/aml-VRT5V2], Expected [aml:tests/data/acpi/x86/pc/DSDT].
> See source file tests/qtest/bios-tables-test.c for instructions on how to update expected files.
> acpi-test: Warning! DSDT mismatch. Actual [asl:/tmp/asl-TTT5V2.dsl, aml:/tmp/aml-VRT5V2], Expected [asl:/tmp/asl-XXM5V2.dsl, aml:tests/data/acpi/x86/pc/DSDT].
> **
> ERROR:../tests/qtest/bios-tables-test.c:553:test_acpi_asl: assertion failed: (all_tables_match)
> 
> (test program exited with status code -6)
> ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
> 
>   3/920 qemu:qtest+qtest-x86_64 / qtest-x86_64/bios-tables-test                            ERROR            1.25s   killed by signal 6 SIGABRT
> >>> G_TEST_DBUS_DAEMON=/media/liuzhao/data/qemu-cook/tests/dbus-vmstate-daemon.sh ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 MESON_TEST_ITERATION=1 UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon QTEST_QEMU_IMG=./qemu-img PYTHON=/media/liuzhao/data/qemu-cook/build/pyvenv/bin/python3 MALLOC_PERTURB_=41 QTEST_QEMU_BINARY=./qemu-system-x86_64 /media/liuzhao/data/qemu-cook/build/tests/qtest/bios-tables-test --tap -k  
> ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
> stderr:
> acpi-test: Warning! DSDT binary file mismatch. Actual [aml:/tmp/aml-D5K5V2], Expected [aml:tests/data/acpi/x86/pc/DSDT].
> See source file tests/qtest/bios-tables-test.c for instructions on how to update expected files.
> acpi-test: Warning! DSDT mismatch. Actual [asl:/tmp/asl-G6K5V2.dsl, aml:/tmp/aml-D5K5V2], Expected [asl:/tmp/asl-AQD5V2.dsl, aml:tests/data/acpi/x86/pc/DSDT].
> **
> ERROR:../tests/qtest/bios-tables-test.c:553:test_acpi_asl: assertion failed: (all_tables_match)
> 
> (test program exited with status code -6)
> 
> 
> Regards,
> Zhao
> 
> 

Re: [PATCH V1 3/4] hw/acpi: Reflect ACPI vCPU {present,enabled} states in ACPI _STA.{PRES,ENA} Bits

[Igor Mammedov]

On Mon, 14 Oct 2024 20:22:04 +0100
Salil Mehta <salil.mehta@huawei.com> wrote:

> Reflect the ACPI CPU hotplug `is_{present, enabled}` states in the `_STA.PRES`
> (presence) and `_STA.ENA` (enabled) bits when the guest kernel evaluates the
> ACPI `_STA` method during initialization, as well as when vCPUs are hot-plugged
> or hot-unplugged. The presence of unplugged vCPUs may need to be deliberately
> *simulated* at the ACPI level to maintain a *persistent* view of vCPUs for the
> guest kernel.

given questionable future of is_present/is_enabled fields,
it probably premature to review this part.
The only thing, I have to say here is repeating spec/doc
update patch describing how it should work should come 1st,
so that we could compare this impl. with it. 

> 
> Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> ---
>  hw/acpi/cpu.c | 26 ++++++++++++++++++++++----
>  1 file changed, 22 insertions(+), 4 deletions(-)
> 
> diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
> index 700aa855e9..23ea2b9c70 100644
> --- a/hw/acpi/cpu.c
> +++ b/hw/acpi/cpu.c
> @@ -63,10 +63,11 @@ static uint64_t cpu_hotplug_rd(void *opaque, hwaddr addr, unsigned size)
>      cdev = &cpu_st->devs[cpu_st->selector];
>      switch (addr) {
>      case ACPI_CPU_FLAGS_OFFSET_RW: /* pack and return is_* fields */
> -        val |= cdev->cpu ? 1 : 0;
> +        val |= cdev->is_enabled ? 1 : 0;
>          val |= cdev->is_inserting ? 2 : 0;
>          val |= cdev->is_removing  ? 4 : 0;
>          val |= cdev->fw_remove  ? 16 : 0;
> +        val |= cdev->is_present ? 32 : 0;
>          trace_cpuhp_acpi_read_flags(cpu_st->selector, val);
>          break;
>      case ACPI_CPU_CMD_DATA_OFFSET_RW:
> @@ -376,6 +377,7 @@ const VMStateDescription vmstate_cpu_hotplug = {
>  #define CPU_REMOVE_EVENT  "CRMV"
>  #define CPU_EJECT_EVENT   "CEJ0"
>  #define CPU_FW_EJECT_EVENT "CEJF"
> +#define CPU_PRESENT       "CPRS"
>  
>  void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts,
>                      build_madt_cpu_fn build_madt_cpu, hwaddr base_addr,
> @@ -436,7 +438,9 @@ void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts,
>          aml_append(field, aml_named_field(CPU_EJECT_EVENT, 1));
>          /* tell firmware to do device eject, write only */
>          aml_append(field, aml_named_field(CPU_FW_EJECT_EVENT, 1));
> -        aml_append(field, aml_reserved_field(3));
> +        /* 1 if present, read only */
> +        aml_append(field, aml_named_field(CPU_PRESENT, 1));
> +        aml_append(field, aml_reserved_field(2));
>          aml_append(field, aml_named_field(CPU_COMMAND, 8));
>          aml_append(cpu_ctrl_dev, field);
>  
> @@ -466,6 +470,7 @@ void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts,
>          Aml *ctrl_lock = aml_name("%s.%s", cphp_res_path, CPU_LOCK);
>          Aml *cpu_selector = aml_name("%s.%s", cphp_res_path, CPU_SELECTOR);
>          Aml *is_enabled = aml_name("%s.%s", cphp_res_path, CPU_ENABLED);
> +        Aml *is_present = aml_name("%s.%s", cphp_res_path, CPU_PRESENT);
>          Aml *cpu_cmd = aml_name("%s.%s", cphp_res_path, CPU_COMMAND);
>          Aml *cpu_data = aml_name("%s.%s", cphp_res_path, CPU_DATA);
>          Aml *ins_evt = aml_name("%s.%s", cphp_res_path, CPU_INSERT_EVENT);
> @@ -494,13 +499,26 @@ void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts,
>          {
>              Aml *idx = aml_arg(0);
>              Aml *sta = aml_local(0);
> +            Aml *ifctx2;
> +            Aml *else_ctx;
>  
>              aml_append(method, aml_acquire(ctrl_lock, 0xFFFF));
>              aml_append(method, aml_store(idx, cpu_selector));
>              aml_append(method, aml_store(zero, sta));
> -            ifctx = aml_if(aml_equal(is_enabled, one));
> +            ifctx = aml_if(aml_equal(is_present, one));
>              {
> -                aml_append(ifctx, aml_store(aml_int(0xF), sta));
> +                ifctx2 = aml_if(aml_equal(is_enabled, one));
> +                {
> +                    /* cpu is present and enabled */
> +                    aml_append(ifctx2, aml_store(aml_int(0xF), sta));
> +                }
> +                aml_append(ifctx, ifctx2);
> +                else_ctx = aml_else();
> +                {
> +                    /* cpu is present but disabled */
> +                    aml_append(else_ctx, aml_store(aml_int(0xD), sta));
> +                }
> +                aml_append(ifctx, else_ctx);
>              }
>              aml_append(method, ifctx);
>              aml_append(method, aml_release(ctrl_lock));

Re: [PATCH V1 4/4] hw/acpi: Populate vCPU Hotplug VMSD to migrate `is_{present,enabled}` states

[Igor Mammedov]

On Mon, 14 Oct 2024 20:22:05 +0100
Salil Mehta <salil.mehta@huawei.com> wrote:

> The ACPI CPU hotplug states `is_{present, enabled}` must be migrated alongside
> other vCPU hotplug states to the destination VM. Therefore, they should be
> integrated into the existing CPU Hotplug VM State Description (VMSD) table.
> Depending on the architecture and its implementation of CPU hotplug events
> (such as ACPI GED, etc.), the CPU hotplug states should be populated
> appropriately within their corresponding subsections of the VMSD table.
> 
>     "acpi-ged (16)": {
>         "ged_state": {
>             "sel": "0x00000000"
>         },
>         [...]
>         "acpi-ged/cpuhp": {
>             "cpuhp_state": {
>                 "selector": "0x00000005",
>                 "command": "0x00",
>                 "devs": [
>                     {
>                         "is_inserting": false,
>                         "is_removing": false,
>                         "is_present": true,
>                         "is_enabled": true,
>                         "ost_event": "0x00000000",
>                         "ost_status": "0x00000000"
>                     },
>                     {
>                         "is_inserting": false,
>                         "is_removing": false,
>                         "is_present": true,
>                         "is_enabled": true,
>                         "ost_event": "0x00000000",
>                         "ost_status": "0x00000000"
>                     },
>                     {
>                         "is_inserting": false,
>                         "is_removing": false,
>                         "is_present": true,
>                         "is_enabled": true,
>                         "ost_event": "0x00000000",
>                         "ost_status": "0x00000000"
>                     },
>                     {
>                         "is_inserting": false,
>                         "is_removing": false,
>                         "is_present": true,
>                         "is_enabled": true,
>                         "ost_event": "0x00000000",
>                         "ost_status": "0x00000000"
>                     },
>                     {
>                         "is_inserting": false,
>                         "is_removing": false,
>                         "is_present": true,
>                         "is_enabled": false,
>                         "ost_event": "0x00000000",
>                         "ost_status": "0x00000000"
>                     },
>                     {
>                         "is_inserting": false,
>                         "is_removing": false,
>                         "is_present": true,
>                         "is_enabled": false,
>                         "ost_event": "0x00000000",
>                         "ost_status": "0x00000000"
>                     }
>                 ]
>             }
>         }
>     },
> 
> Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> ---
>  hw/acpi/cpu.c                  |  2 ++
>  hw/acpi/generic_event_device.c | 11 +++++++++++
>  2 files changed, 13 insertions(+)
> 
> diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
> index 23ea2b9c70..d34c1e601e 100644
> --- a/hw/acpi/cpu.c
> +++ b/hw/acpi/cpu.c
> @@ -340,6 +340,8 @@ static const VMStateDescription vmstate_cpuhp_sts = {
>      .fields = (const VMStateField[]) {
>          VMSTATE_BOOL(is_inserting, AcpiCpuStatus),
>          VMSTATE_BOOL(is_removing, AcpiCpuStatus),
> +        VMSTATE_BOOL(is_present, AcpiCpuStatus),
> +        VMSTATE_BOOL(is_enabled, AcpiCpuStatus),

that's likely will break x86 migration,
but before bothering peterx, maybe we won't need this hunk if
is_enabled is migrated as part of vCPU state.

>          VMSTATE_UINT32(ost_event, AcpiCpuStatus),
>          VMSTATE_UINT32(ost_status, AcpiCpuStatus),
>          VMSTATE_END_OF_LIST()
> diff --git a/hw/acpi/generic_event_device.c b/hw/acpi/generic_event_device.c
> index 15b4c3ebbf..a4d78a534c 100644
> --- a/hw/acpi/generic_event_device.c
> +++ b/hw/acpi/generic_event_device.c
> @@ -331,6 +331,16 @@ static const VMStateDescription vmstate_memhp_state = {
>      }
>  };
>  
> +static const VMStateDescription vmstate_cpuhp_state = {
> +    .name = "acpi-ged/cpuhp",
> +    .version_id = 1,
> +    .minimum_version_id = 1,
> +    .fields      = (VMStateField[]) {
> +        VMSTATE_CPU_HOTPLUG(cpuhp_state, AcpiGedState),
> +        VMSTATE_END_OF_LIST()
> +    }
> +};

this subsection likely needs is_needed hook to avoid breaking
case where target doesn't have cpuhp support (older QEMU)

> +
>  static const VMStateDescription vmstate_ged_state = {
>      .name = "acpi-ged-state",
>      .version_id = 1,
> @@ -379,6 +389,7 @@ static const VMStateDescription vmstate_acpi_ged = {
>      },
>      .subsections = (const VMStateDescription * const []) {
>          &vmstate_memhp_state,
> +        &vmstate_cpuhp_state,
>          &vmstate_ghes_state,
>          NULL
>      }

Re: [PATCH V1 0/4] Arch agnostic ACPI changes to support vCPU Hotplug (on Archs like ARM)

[Igor Mammedov]

On Mon, 14 Oct 2024 20:22:01 +0100
Salil Mehta <salil.mehta@huawei.com> wrote:

> Certain CPU architecture specifications [1][2][3] prohibit changes to the CPUs
> *presence* after the kernel has booted. This is because many system
> initializations depend on the exact CPU count at boot time and do not expect it
> to change afterward. For example, components like interrupt controllers that are
> closely coupled with CPUs, or various per-CPU features, may not support
> configuration changes once the kernel has been initialized.
> 
> This requirement poses a challenge for virtualization features like vCPU
> hotplug. To address this, changes to the ACPI AML are necessary to update the
> `_STA.PRES` (presence) and `_STA.ENA` (enabled) bits accordingly during guest
> initialization, as well as when vCPUs are hot-plugged or hot-unplugged. The
> presence of unplugged vCPUs may need to be deliberately *simulated* at the ACPI
> level to maintain a *persistent* view of vCPUs for the guest kernel.

the series is peppered with *simulated* idea, which after looking at code
I read as 'fake'. While it's obvious to author why things need to be faked
at this time, it will be forgotten later on. And cause a lot swearing from
whoever will have to deal with this code down the road.

Salil, I'm sorry that review comes out as mostly negative but for me having to
repeat 'simulated' some many times, hints that the there is
something wrong with design and that we should re-evaluate the approach.

ps:
see comments on 1/4 for suggestions


> This patch set introduces the following features:
> 
> 1. ACPI Interface with Explicit PRESENT and ENABLED CPU States: It allows the
>    guest kernel to evaluate these states using the `_STA` ACPI method.
>    
> 2. Initialization of ACPI CPU States: These states are initialized during
>    `machvirt_init` and when vCPUs are hot-(un)plugged. This enables hotpluggable
>    vCPUs to be exposed to the guest kernel via ACPI.
> 
> 3. Support for Migrating ACPI CPU States: The patch set ensures the migration of
>    the newly introduced `is_{present,enabled}` ACPI CPU states to the
>    destination VM.
> 
> The approach is flexible enough to accommodate ARM-like architectures that
> intend to implement vCPU hotplug functionality. It is suitable for architectures
> facing similar constraints to ARM or those that plan to implement vCPU
> hotplugging independently of hardware support (if available).
> 
> This patch set is derived from the ARM-specific vCPU hotplug implementation [4]
> and includes migration components adaptable to other architectures, following
> suggestions [5] made by Igor Mammedov <imammedo@redhat.com>.
> 
> It can be applied independently, ensuring compatibility with existing hotplug
> support in other architectures. I have tested this patch set in conjunction with
> the ARM-specific vCPU hotplug changes (included in the upcoming RFC V5 [6]), and
> everything worked as expected. I kindly request maintainers of other
> architectures to provide a "Tested-by" after running their respective regression
> tests.
> 
> Many thanks!
> 
> 
> References:
> [1] KVMForum 2023 Presentation: Challenges Revisited in Supporting Virt CPU Hotplug on
>     architectures that don’t Support CPU Hotplug (like ARM64)
>     a. Kernel Link: https://kvm-forum.qemu.org/2023/KVM-forum-cpu-hotplug_7OJ1YyJ.pdf
>     b. Qemu Link:  https://kvm-forum.qemu.org/2023/Challenges_Revisited_in_Supporting_Virt_CPU_Hotplug_-__ii0iNb3.pdf
> [2] KVMForum 2020 Presentation: Challenges in Supporting Virtual CPU Hotplug on
>     SoC Based Systems (like ARM64)
>     Link: https://kvmforum2020.sched.com/event/eE4m
> [3] Check comment 5 in the bugzilla entry
>     Link: https://bugzilla.tianocore.org/show_bug.cgi?id=4481#c5
> [4] [PATCH RFC V4 00/33] Support of Virtual CPU Hotplug for ARMv8 Arch
>     Link: https://lore.kernel.org/qemu-devel/20241009031815.250096-1-salil.mehta@huawei.com/T/#mf32be203baa568a871dc625b732f666a4c4f1e68
> [5] Architecture agnostic ACPI VMSD state migration (Discussion)
>     Link: https://lore.kernel.org/qemu-devel/20240715155436.577d34c5@imammedo.users.ipa.redhat.com/
> [6] Upcoming RFC V5, Support of Virtual CPU Hotplug for ARMv8 Arch
>     Link: https://github.com/salil-mehta/qemu/commits/virt-cpuhp-armv8/rfc-v5
> 
> Salil Mehta (4):
>   hw/acpi: Initialize ACPI Hotplug CPU Status with Support for vCPU
>     `Persistence`
>   hw/acpi: Update ACPI CPU Status `is_{present, enabled}` during vCPU
>     hot(un)plug
>   hw/acpi: Reflect ACPI vCPU {present,enabled} states in ACPI
>     _STA.{PRES,ENA} Bits
>   hw/acpi: Populate vCPU Hotplug VMSD to migrate `is_{present,enabled}`
>     states
> 
>  cpu-target.c         patches.vcpuhp.rfc-v5.arch.agnostic.acpi          |  1 +
>  hw/acpi/cpu.c                  | 70 +++++++++++++++++++++++++++++++---
>  hw/acpi/generic_event_device.c | 11 ++++++
>  include/hw/acpi/cpu.h          | 21 ++++++++++
>  include/hw/core/cpu.h          | 21 ++++++++++
>  5 files changed, 119 insertions(+), 5 deletions(-)
> 

Re: [PATCH V1 0/4] Arch agnostic ACPI changes to support vCPU Hotplug (on Archs like ARM)

[Gustavo Romero]

Hi Miguel,

On 10/15/24 15:41, Miguel Luis wrote:
> Hi Salil,
> 
> I’ve ran the usual tests successfully of hotplug/unplug from the number of cold-booted cpus up to maxcpus and migration on ARM. Please feel free to add:

Do you mind sharing what cpus you used for your tests?

Did you use tcg or kvm for the tests?

Thanks!


Cheers,
Gustavo
  
> Tested-by: Miguel Luis <miguel.luis@oracle.com>
> 
> Thanks
> Miguel
> 
>> On 14 Oct 2024, at 19:22, Salil Mehta <salil.mehta@huawei.com> wrote:
>>
>> Certain CPU architecture specifications [1][2][3] prohibit changes to the CPUs
>> *presence* after the kernel has booted. This is because many system
>> initializations depend on the exact CPU count at boot time and do not expect it
>> to change afterward. For example, components like interrupt controllers that are
>> closely coupled with CPUs, or various per-CPU features, may not support
>> configuration changes once the kernel has been initialized.
>>
>> This requirement poses a challenge for virtualization features like vCPU
>> hotplug. To address this, changes to the ACPI AML are necessary to update the
>> `_STA.PRES` (presence) and `_STA.ENA` (enabled) bits accordingly during guest
>> initialization, as well as when vCPUs are hot-plugged or hot-unplugged. The
>> presence of unplugged vCPUs may need to be deliberately *simulated* at the ACPI
>> level to maintain a *persistent* view of vCPUs for the guest kernel.
>>
>> This patch set introduces the following features:
>>
>> 1. ACPI Interface with Explicit PRESENT and ENABLED CPU States: It allows the
>>    guest kernel to evaluate these states using the `_STA` ACPI method.
>>
>> 2. Initialization of ACPI CPU States: These states are initialized during
>>    `machvirt_init` and when vCPUs are hot-(un)plugged. This enables hotpluggable
>>    vCPUs to be exposed to the guest kernel via ACPI.
>>
>> 3. Support for Migrating ACPI CPU States: The patch set ensures the migration of
>>    the newly introduced `is_{present,enabled}` ACPI CPU states to the
>>    destination VM.
>>
>> The approach is flexible enough to accommodate ARM-like architectures that
>> intend to implement vCPU hotplug functionality. It is suitable for architectures
>> facing similar constraints to ARM or those that plan to implement vCPU
>> hotplugging independently of hardware support (if available).
>>
>> This patch set is derived from the ARM-specific vCPU hotplug implementation [4]
>> and includes migration components adaptable to other architectures, following
>> suggestions [5] made by Igor Mammedov <imammedo@redhat.com>.
>>
>> It can be applied independently, ensuring compatibility with existing hotplug
>> support in other architectures. I have tested this patch set in conjunction with
>> the ARM-specific vCPU hotplug changes (included in the upcoming RFC V5 [6]), and
>> everything worked as expected. I kindly request maintainers of other
>> architectures to provide a "Tested-by" after running their respective regression
>> tests.
>>
>> Many thanks!
>>
>>
>> References:
>> [1] KVMForum 2023 Presentation: Challenges Revisited in Supporting Virt CPU Hotplug on
>>     architectures that don’t Support CPU Hotplug (like ARM64)
>>     a. Kernel Link: https://kvm-forum.qemu.org/2023/KVM-forum-cpu-hotplug_7OJ1YyJ.pdf
>>     b. Qemu Link:  https://kvm-forum.qemu.org/2023/Challenges_Revisited_in_Supporting_Virt_CPU_Hotplug_-__ii0iNb3.pdf
>> [2] KVMForum 2020 Presentation: Challenges in Supporting Virtual CPU Hotplug on
>>     SoC Based Systems (like ARM64)
>>     Link: https://kvmforum2020.sched.com/event/eE4m
>> [3] Check comment 5 in the bugzilla entry
>>     Link: https://bugzilla.tianocore.org/show_bug.cgi?id=4481#c5
>> [4] [PATCH RFC V4 00/33] Support of Virtual CPU Hotplug for ARMv8 Arch
>>     Link: https://lore.kernel.org/qemu-devel/20241009031815.250096-1-salil.mehta@huawei.com/T/#mf32be203baa568a871dc625b732f666a4c4f1e68
>> [5] Architecture agnostic ACPI VMSD state migration (Discussion)
>>     Link: https://lore.kernel.org/qemu-devel/20240715155436.577d34c5@imammedo.users.ipa.redhat.com/
>> [6] Upcoming RFC V5, Support of Virtual CPU Hotplug for ARMv8 Arch
>>     Link: https://github.com/salil-mehta/qemu/commits/virt-cpuhp-armv8/rfc-v5
>>
>> Salil Mehta (4):
>>   hw/acpi: Initialize ACPI Hotplug CPU Status with Support for vCPU
>>     `Persistence`
>>   hw/acpi: Update ACPI CPU Status `is_{present, enabled}` during vCPU
>>     hot(un)plug
>>   hw/acpi: Reflect ACPI vCPU {present,enabled} states in ACPI
>>     _STA.{PRES,ENA} Bits
>>   hw/acpi: Populate vCPU Hotplug VMSD to migrate `is_{present,enabled}`
>>     states
>>
>> cpu-target.c         patches.vcpuhp.rfc-v5.arch.agnostic.acpi          |  1 +
>> hw/acpi/cpu.c                  | 70 +++++++++++++++++++++++++++++++---
>> hw/acpi/generic_event_device.c | 11 ++++++
>> include/hw/acpi/cpu.h          | 21 ++++++++++
>> include/hw/core/cpu.h          | 21 ++++++++++
>> 5 files changed, 119 insertions(+), 5 deletions(-)
>>
>> -- 
>> 2.34.1
>>
> 

Re: [PATCH V1 3/4] hw/acpi: Reflect ACPI vCPU {present,enabled} states in ACPI _STA.{PRES,ENA} Bits

[Gustavo Romero]

Hi Salil,

On 10/14/24 16:22, Salil Mehta wrote:
> Reflect the ACPI CPU hotplug `is_{present, enabled}` states in the `_STA.PRES`
> (presence) and `_STA.ENA` (enabled) bits when the guest kernel evaluates the
> ACPI `_STA` method during initialization, as well as when vCPUs are hot-plugged
> or hot-unplugged. The presence of unplugged vCPUs may need to be deliberately
> *simulated* at the ACPI level to maintain a *persistent* view of vCPUs for the
> guest kernel.
> 
> Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> ---
>   hw/acpi/cpu.c | 26 ++++++++++++++++++++++----
>   1 file changed, 22 insertions(+), 4 deletions(-)
> 
> diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
> index 700aa855e9..23ea2b9c70 100644
> --- a/hw/acpi/cpu.c
> +++ b/hw/acpi/cpu.c
> @@ -63,10 +63,11 @@ static uint64_t cpu_hotplug_rd(void *opaque, hwaddr addr, unsigned size)
>       cdev = &cpu_st->devs[cpu_st->selector];
>       switch (addr) {
>       case ACPI_CPU_FLAGS_OFFSET_RW: /* pack and return is_* fields */
> -        val |= cdev->cpu ? 1 : 0;
> +        val |= cdev->is_enabled ? 1 : 0;
>           val |= cdev->is_inserting ? 2 : 0;
>           val |= cdev->is_removing  ? 4 : 0;
>           val |= cdev->fw_remove  ? 16 : 0;
> +        val |= cdev->is_present ? 32 : 0;
>           trace_cpuhp_acpi_read_flags(cpu_st->selector, val);
>           break;
>       case ACPI_CPU_CMD_DATA_OFFSET_RW:
> @@ -376,6 +377,7 @@ const VMStateDescription vmstate_cpu_hotplug = {
>   #define CPU_REMOVE_EVENT  "CRMV"
>   #define CPU_EJECT_EVENT   "CEJ0"
>   #define CPU_FW_EJECT_EVENT "CEJF"
> +#define CPU_PRESENT       "CPRS"
>   
>   void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts,
>                       build_madt_cpu_fn build_madt_cpu, hwaddr base_addr,
> @@ -436,7 +438,9 @@ void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts,
>           aml_append(field, aml_named_field(CPU_EJECT_EVENT, 1));
>           /* tell firmware to do device eject, write only */
>           aml_append(field, aml_named_field(CPU_FW_EJECT_EVENT, 1));
> -        aml_append(field, aml_reserved_field(3));
> +        /* 1 if present, read only */
> +        aml_append(field, aml_named_field(CPU_PRESENT, 1));
> +        aml_append(field, aml_reserved_field(2));
>           aml_append(field, aml_named_field(CPU_COMMAND, 8));
>           aml_append(cpu_ctrl_dev, field);
>   
> @@ -466,6 +470,7 @@ void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts,
>           Aml *ctrl_lock = aml_name("%s.%s", cphp_res_path, CPU_LOCK);
>           Aml *cpu_selector = aml_name("%s.%s", cphp_res_path, CPU_SELECTOR);
>           Aml *is_enabled = aml_name("%s.%s", cphp_res_path, CPU_ENABLED);
> +        Aml *is_present = aml_name("%s.%s", cphp_res_path, CPU_PRESENT);
>           Aml *cpu_cmd = aml_name("%s.%s", cphp_res_path, CPU_COMMAND);
>           Aml *cpu_data = aml_name("%s.%s", cphp_res_path, CPU_DATA);
>           Aml *ins_evt = aml_name("%s.%s", cphp_res_path, CPU_INSERT_EVENT);
> @@ -494,13 +499,26 @@ void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts,
>           {
>               Aml *idx = aml_arg(0);
>               Aml *sta = aml_local(0);
> +            Aml *ifctx2;
> +            Aml *else_ctx;
>   
>               aml_append(method, aml_acquire(ctrl_lock, 0xFFFF));
>               aml_append(method, aml_store(idx, cpu_selector));
>               aml_append(method, aml_store(zero, sta));
> -            ifctx = aml_if(aml_equal(is_enabled, one));
> +            ifctx = aml_if(aml_equal(is_present, one));
>               {
> -                aml_append(ifctx, aml_store(aml_int(0xF), sta));
> +                ifctx2 = aml_if(aml_equal(is_enabled, one));
> +                {
> +                    /* cpu is present and enabled */
> +                    aml_append(ifctx2, aml_store(aml_int(0xF), sta));
> +                }
> +                aml_append(ifctx, ifctx2);
> +                else_ctx = aml_else();
> +                {
> +                    /* cpu is present but disabled */
> +                    aml_append(else_ctx, aml_store(aml_int(0xD), sta));

Here, the return value for _STA method is set to reflect the state of
CPU_PRESENT and CPU_ENABLED fields. I can't see these two fields being
mapped to AcpiCpuStatus.{is_present,is_enabled}. They look to be mapped
to the MMIO region (base_addr), which doesn't mapped to AcpiCpuStatus
afaics. So where CPU_PRESENT and CPU_ENABLED are set and where exactly
they reside?


Cheers,
Gustavo

> +                }
> +                aml_append(ifctx, else_ctx);
>               }
>               aml_append(method, ifctx);
>               aml_append(method, aml_release(ctrl_lock));

Re: [PATCH V1 0/4] Arch agnostic ACPI changes to support vCPU Hotplug (on Archs like ARM)

[Gustavo Romero]

Hi Igor,

On 10/18/24 11:46, Igor Mammedov wrote:
> On Mon, 14 Oct 2024 20:22:01 +0100
> Salil Mehta <salil.mehta@huawei.com> wrote:
> 
>> Certain CPU architecture specifications [1][2][3] prohibit changes to the CPUs
>> *presence* after the kernel has booted. This is because many system
>> initializations depend on the exact CPU count at boot time and do not expect it
>> to change afterward. For example, components like interrupt controllers that are
>> closely coupled with CPUs, or various per-CPU features, may not support
>> configuration changes once the kernel has been initialized.
>>
>> This requirement poses a challenge for virtualization features like vCPU
>> hotplug. To address this, changes to the ACPI AML are necessary to update the
>> `_STA.PRES` (presence) and `_STA.ENA` (enabled) bits accordingly during guest
>> initialization, as well as when vCPUs are hot-plugged or hot-unplugged. The
>> presence of unplugged vCPUs may need to be deliberately *simulated* at the ACPI
>> level to maintain a *persistent* view of vCPUs for the guest kernel.
> 
> the series is peppered with *simulated* idea, which after looking at code
> I read as 'fake'. While it's obvious to author why things need to be faked
> at this time, it will be forgotten later on. And cause a lot swearing from
> whoever will have to deal with this code down the road.
> 
> Salil, I'm sorry that review comes out as mostly negative but for me having to
> repeat 'simulated' some many times, hints that the there is
> something wrong with design and that we should re-evaluate the approach.

I think 'simulated' was indeed a badly chosen name because the series,
as read it, is merely saving the hotplugged/unplugged vCPU states to the
ACPI for persistence and later restoration, so I think s/simulated/saved/ is
reasonable here, i.e., ACPI now reflects the vCPU states after hotplug/unplug.
QEMU manipulates the ACPI tables and methods to reflect the machine config.
and this series is essentially doing the same. Why that's not ok?


Cheers,
Gustavo

> ps:
> see comments on 1/4 for suggestions
> 
> 
>> This patch set introduces the following features:
>>
>> 1. ACPI Interface with Explicit PRESENT and ENABLED CPU States: It allows the
>>     guest kernel to evaluate these states using the `_STA` ACPI method.
>>     
>> 2. Initialization of ACPI CPU States: These states are initialized during
>>     `machvirt_init` and when vCPUs are hot-(un)plugged. This enables hotpluggable
>>     vCPUs to be exposed to the guest kernel via ACPI.
>>
>> 3. Support for Migrating ACPI CPU States: The patch set ensures the migration of
>>     the newly introduced `is_{present,enabled}` ACPI CPU states to the
>>     destination VM.
>>
>> The approach is flexible enough to accommodate ARM-like architectures that
>> intend to implement vCPU hotplug functionality. It is suitable for architectures
>> facing similar constraints to ARM or those that plan to implement vCPU
>> hotplugging independently of hardware support (if available).
>>
>> This patch set is derived from the ARM-specific vCPU hotplug implementation [4]
>> and includes migration components adaptable to other architectures, following
>> suggestions [5] made by Igor Mammedov <imammedo@redhat.com>.
>>
>> It can be applied independently, ensuring compatibility with existing hotplug
>> support in other architectures. I have tested this patch set in conjunction with
>> the ARM-specific vCPU hotplug changes (included in the upcoming RFC V5 [6]), and
>> everything worked as expected. I kindly request maintainers of other
>> architectures to provide a "Tested-by" after running their respective regression
>> tests.
>>
>> Many thanks!
>>
>>
>> References:
>> [1] KVMForum 2023 Presentation: Challenges Revisited in Supporting Virt CPU Hotplug on
>>      architectures that don’t Support CPU Hotplug (like ARM64)
>>      a. Kernel Link: https://kvm-forum.qemu.org/2023/KVM-forum-cpu-hotplug_7OJ1YyJ.pdf
>>      b. Qemu Link:  https://kvm-forum.qemu.org/2023/Challenges_Revisited_in_Supporting_Virt_CPU_Hotplug_-__ii0iNb3.pdf
>> [2] KVMForum 2020 Presentation: Challenges in Supporting Virtual CPU Hotplug on
>>      SoC Based Systems (like ARM64)
>>      Link: https://kvmforum2020.sched.com/event/eE4m
>> [3] Check comment 5 in the bugzilla entry
>>      Link: https://bugzilla.tianocore.org/show_bug.cgi?id=4481#c5
>> [4] [PATCH RFC V4 00/33] Support of Virtual CPU Hotplug for ARMv8 Arch
>>      Link: https://lore.kernel.org/qemu-devel/20241009031815.250096-1-salil.mehta@huawei.com/T/#mf32be203baa568a871dc625b732f666a4c4f1e68
>> [5] Architecture agnostic ACPI VMSD state migration (Discussion)
>>      Link: https://lore.kernel.org/qemu-devel/20240715155436.577d34c5@imammedo.users.ipa.redhat.com/
>> [6] Upcoming RFC V5, Support of Virtual CPU Hotplug for ARMv8 Arch
>>      Link: https://github.com/salil-mehta/qemu/commits/virt-cpuhp-armv8/rfc-v5
>>
>> Salil Mehta (4):
>>    hw/acpi: Initialize ACPI Hotplug CPU Status with Support for vCPU
>>      `Persistence`
>>    hw/acpi: Update ACPI CPU Status `is_{present, enabled}` during vCPU
>>      hot(un)plug
>>    hw/acpi: Reflect ACPI vCPU {present,enabled} states in ACPI
>>      _STA.{PRES,ENA} Bits
>>    hw/acpi: Populate vCPU Hotplug VMSD to migrate `is_{present,enabled}`
>>      states
>>
>>   cpu-target.c         patches.vcpuhp.rfc-v5.arch.agnostic.acpi          |  1 +
>>   hw/acpi/cpu.c                  | 70 +++++++++++++++++++++++++++++++---
>>   hw/acpi/generic_event_device.c | 11 ++++++
>>   include/hw/acpi/cpu.h          | 21 ++++++++++
>>   include/hw/core/cpu.h          | 21 ++++++++++
>>   5 files changed, 119 insertions(+), 5 deletions(-)
>>
> 

Re: [PATCH V1 0/4] Arch agnostic ACPI changes to support vCPU Hotplug (on Archs like ARM)

[Miguel Luis]

Hi Gustavo,

> On 18 Oct 2024, at 17:57, Gustavo Romero <gustavo.romero@linaro.org> wrote:
> 
> Hi Miguel,
> 
> On 10/15/24 15:41, Miguel Luis wrote:
>> Hi Salil,
>> I’ve ran the usual tests successfully of hotplug/unplug from the number of cold-booted cpus up to maxcpus and migration on ARM. Please feel free to add:
> 
> Do you mind sharing what cpus you used for your tests?
> 

Not at all. I’ve used -cpu host and -cpu max. I had used RFC-v5 from [1] in order to test
this patchset as they are at the base of it.

> Did you use tcg or kvm for the tests?

I’ve used both in the following configurations:

-M virt -accel kvm -cpu host
-M virt,gic_version=3 -accel kvm -cpu host
-M virt,gic_version=3 -accel tcg -cpu max

And 

-M virt -accel tcg -cpu max

This last one presented defects in QEMU on behalf of RFC-V5 but, if you don’t
mind, instead of me transcribing everything here, could you please take a look
at [2] ?

I could have given more detail from start, apologies for any confusion on my part.

Thanks,
Miguel

[1]: https://github.com/salil-mehta/qemu/commits/virt-cpuhp-armv8/rfc-v5/
[2]: https://lore.kernel.org/qemu-devel/30478B18-932D-413C-BFF0-5FD70510D9D5@oracle.com/

> 
> Thanks!
> 
> 
> Cheers,
> Gustavo
> 
>> Tested-by: Miguel Luis <miguel.luis@oracle.com>
>> Thanks
>> Miguel
>>> On 14 Oct 2024, at 19:22, Salil Mehta <salil.mehta@huawei.com> wrote:
>>> 
>>> Certain CPU architecture specifications [1][2][3] prohibit changes to the CPUs
>>> *presence* after the kernel has booted. This is because many system
>>> initializations depend on the exact CPU count at boot time and do not expect it
>>> to change afterward. For example, components like interrupt controllers that are
>>> closely coupled with CPUs, or various per-CPU features, may not support
>>> configuration changes once the kernel has been initialized.
>>> 
>>> This requirement poses a challenge for virtualization features like vCPU
>>> hotplug. To address this, changes to the ACPI AML are necessary to update the
>>> `_STA.PRES` (presence) and `_STA.ENA` (enabled) bits accordingly during guest
>>> initialization, as well as when vCPUs are hot-plugged or hot-unplugged. The
>>> presence of unplugged vCPUs may need to be deliberately *simulated* at the ACPI
>>> level to maintain a *persistent* view of vCPUs for the guest kernel.
>>> 
>>> This patch set introduces the following features:
>>> 
>>> 1. ACPI Interface with Explicit PRESENT and ENABLED CPU States: It allows the
>>>   guest kernel to evaluate these states using the `_STA` ACPI method.
>>> 
>>> 2. Initialization of ACPI CPU States: These states are initialized during
>>>   `machvirt_init` and when vCPUs are hot-(un)plugged. This enables hotpluggable
>>>   vCPUs to be exposed to the guest kernel via ACPI.
>>> 
>>> 3. Support for Migrating ACPI CPU States: The patch set ensures the migration of
>>>   the newly introduced `is_{present,enabled}` ACPI CPU states to the
>>>   destination VM.
>>> 
>>> The approach is flexible enough to accommodate ARM-like architectures that
>>> intend to implement vCPU hotplug functionality. It is suitable for architectures
>>> facing similar constraints to ARM or those that plan to implement vCPU
>>> hotplugging independently of hardware support (if available).
>>> 
>>> This patch set is derived from the ARM-specific vCPU hotplug implementation [4]
>>> and includes migration components adaptable to other architectures, following
>>> suggestions [5] made by Igor Mammedov <imammedo@redhat.com>.
>>> 
>>> It can be applied independently, ensuring compatibility with existing hotplug
>>> support in other architectures. I have tested this patch set in conjunction with
>>> the ARM-specific vCPU hotplug changes (included in the upcoming RFC V5 [6]), and
>>> everything worked as expected. I kindly request maintainers of other
>>> architectures to provide a "Tested-by" after running their respective regression
>>> tests.
>>> 
>>> Many thanks!
>>> 
>>> 
>>> References:
>>> [1] KVMForum 2023 Presentation: Challenges Revisited in Supporting Virt CPU Hotplug on
>>>    architectures that don’t Support CPU Hotplug (like ARM64)
>>>    a. Kernel Link: https://kvm-forum.qemu.org/2023/KVM-forum-cpu-hotplug_7OJ1YyJ.pdf
>>>    b. Qemu Link:  https://kvm-forum.qemu.org/2023/Challenges_Revisited_in_Supporting_Virt_CPU_Hotplug_-__ii0iNb3.pdf
>>> [2] KVMForum 2020 Presentation: Challenges in Supporting Virtual CPU Hotplug on
>>>    SoC Based Systems (like ARM64)
>>>    Link: https://kvmforum2020.sched.com/event/eE4m
>>> [3] Check comment 5 in the bugzilla entry
>>>    Link: https://bugzilla.tianocore.org/show_bug.cgi?id=4481#c5
>>> [4] [PATCH RFC V4 00/33] Support of Virtual CPU Hotplug for ARMv8 Arch
>>>    Link: https://lore.kernel.org/qemu-devel/20241009031815.250096-1-salil.mehta@huawei.com/T/#mf32be203baa568a871dc625b732f666a4c4f1e68
>>> [5] Architecture agnostic ACPI VMSD state migration (Discussion)
>>>    Link: https://lore.kernel.org/qemu-devel/20240715155436.577d34c5@imammedo.users.ipa.redhat.com/
>>> [6] Upcoming RFC V5, Support of Virtual CPU Hotplug for ARMv8 Arch
>>>    Link: https://github.com/salil-mehta/qemu/commits/virt-cpuhp-armv8/rfc-v5
>>> 
>>> Salil Mehta (4):
>>>  hw/acpi: Initialize ACPI Hotplug CPU Status with Support for vCPU
>>>    `Persistence`
>>>  hw/acpi: Update ACPI CPU Status `is_{present, enabled}` during vCPU
>>>    hot(un)plug
>>>  hw/acpi: Reflect ACPI vCPU {present,enabled} states in ACPI
>>>    _STA.{PRES,ENA} Bits
>>>  hw/acpi: Populate vCPU Hotplug VMSD to migrate `is_{present,enabled}`
>>>    states
>>> 
>>> cpu-target.c         patches.vcpuhp.rfc-v5.arch.agnostic.acpi          |  1 +
>>> hw/acpi/cpu.c                  | 70 +++++++++++++++++++++++++++++++---
>>> hw/acpi/generic_event_device.c | 11 ++++++
>>> include/hw/acpi/cpu.h          | 21 ++++++++++
>>> include/hw/core/cpu.h          | 21 ++++++++++
>>> 5 files changed, 119 insertions(+), 5 deletions(-)
>>> 
>>> -- 
>>> 2.34.1
>>> 
> 

Re: [PATCH V1 1/4] hw/acpi: Initialize ACPI Hotplug CPU Status with Support for vCPU `Persistence`

[Salil Mehta]

[-- Attachment #1: Type: text/plain, Size: 8699 bytes --]

HI Gustavo,

On Wed, Oct 16, 2024 at 10:01 PM Gustavo Romero <gustavo.romero@linaro.org>
wrote:

> Hi Salil,
>
> On 10/14/24 16:22, Salil Mehta wrote:
> > Certain CPU architecture specifications [1][2][3] prohibit changes to CPU
> > presence after the kernel has booted. This limitation exists because
> many system
> > initializations rely on the exact CPU count at boot time and do not
> expect it to
> > change later. For example, components like interrupt controllers, which
> are
> > closely tied to CPUs, or various per-CPU features, may not support
> configuration
> > changes once the kernel has been initialized. This presents a challenge
> for
> > virtualization features such as vCPU hotplug.
> >
> > To address this issue, introduce an `is_enabled` state in the
> `AcpiCpuStatus`,
> > which reflects whether a vCPU has been hot-plugged or hot-unplugged in
> QEMU,
> > marking it as (un)available in the Guest Kernel. The `is_present` state
> should
> > be set based on the `acpi_persistent` flag. In cases where unplugged
> vCPUs need
> > to be deliberately simulated in the ACPI to maintain a persistent view
> of vCPUs,
> > this flag ensures the guest kernel continues to see those vCPUs.
> >
> > Additionally, introduce an `acpi_persistent` property that can be used to
> > initialize the ACPI vCPU presence state accordingly. Architectures
> requiring
> > ACPI to expose a persistent view of vCPUs can override its default
> value. Refer
> > to the patch-set implelenting vCPU hotplug support for ARM for more
> details on
>
> nit: implementation
>

Thanks


>
>
> Cheers,
> Gustavo
>
> > its usage.
> >
> > References:
> > [1] KVMForum 2023 Presentation: Challenges Revisited in Supporting Virt
> CPU Hotplug on
> >      architectures that don’t Support CPU Hotplug (like ARM64)
> >      a. Kernel Link:
> https://kvm-forum.qemu.org/2023/KVM-forum-cpu-hotplug_7OJ1YyJ.pdf
> >      b. Qemu Link:
> https://kvm-forum.qemu.org/2023/Challenges_Revisited_in_Supporting_Virt_CPU_Hotplug_-__ii0iNb3.pdf
> > [2] KVMForum 2020 Presentation: Challenges in Supporting Virtual CPU
> Hotplug on
> >      SoC Based Systems (like ARM64)
> >      Link: https://kvmforum2020.sched.com/event/eE4m
> > [3] Check comment 5 in the bugzilla entry
> >      Link: https://bugzilla.tianocore.org/show_bug.cgi?id=4481#c5
> >
> > Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> > ---
> >   cpu-target.c          |  1 +
> >   hw/acpi/cpu.c         | 35 ++++++++++++++++++++++++++++++++++-
> >   include/hw/acpi/cpu.h | 21 +++++++++++++++++++++
> >   include/hw/core/cpu.h | 21 +++++++++++++++++++++
> >   4 files changed, 77 insertions(+), 1 deletion(-)
> >
> > diff --git a/cpu-target.c b/cpu-target.c
> > index 499facf774..c8a29ab495 100644
> > --- a/cpu-target.c
> > +++ b/cpu-target.c
> > @@ -200,6 +200,7 @@ static Property cpu_common_props[] = {
> >        */
> >       DEFINE_PROP_LINK("memory", CPUState, memory, TYPE_MEMORY_REGION,
> >                        MemoryRegion *),
> > +    DEFINE_PROP_BOOL("acpi-persistent", CPUState, acpi_persistent,
> false),
> >   #endif
> >       DEFINE_PROP_END_OF_LIST(),
> >   };
> > diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
> > index 5cb60ca8bc..083c4010c2 100644
> > --- a/hw/acpi/cpu.c
> > +++ b/hw/acpi/cpu.c
> > @@ -225,7 +225,40 @@ void cpu_hotplug_hw_init(MemoryRegion *as, Object
> *owner,
> >       state->dev_count = id_list->len;
> >       state->devs = g_new0(typeof(*state->devs), state->dev_count);
> >       for (i = 0; i < id_list->len; i++) {
> > -        state->devs[i].cpu =  CPU(id_list->cpus[i].cpu);
> > +        struct CPUState *cpu = CPU(id_list->cpus[i].cpu);
> > +        /*
> > +         * In most architectures, CPUs that are marked as ACPI
> 'present' are
> > +         * also ACPI 'enabled' by default. These states remain
> consistent at
> > +         * both the QOM and ACPI levels.
> > +         */
> > +        if (cpu) {
> > +            state->devs[i].is_enabled = true;
> > +            state->devs[i].is_present = true;
> > +            state->devs[i].cpu = cpu;
> > +        } else {
> > +            state->devs[i].is_enabled = false;
> > +            /*
> > +             * In some architectures, even 'unplugged' or 'disabled'
> QOM CPUs
> > +             * may be exposed as ACPI 'present.' This approach provides
> a
> > +             * persistent view of the vCPUs to the guest kernel. This
> could be
> > +             * due to an architectural constraint that requires every
> per-CPU
> > +             * component to be present at boot time, meaning the exact
> count of
> > +             * vCPUs must be known and cannot be altered after the
> kernel has
> > +             * booted. As a result, the vCPU states at the QOM and ACPI
> levels
> > +             * might become inconsistent. However, in such cases, the
> presence
> > +             * of vCPUs has been deliberately simulated at the ACPI
> level.
> > +             */
> > +            if (acpi_persistent_cpu(first_cpu)) {
> > +                state->devs[i].is_present = true;
> > +                /*
> > +                 * `CPUHotplugState::AcpiCpuStatus::cpu` becomes
> insignificant
> > +                 * in this case
> > +                 */
> > +            } else {
> > +                state->devs[i].is_present = false;
> > +                state->devs[i].cpu = cpu;
> > +            }
> > +        }
> >           state->devs[i].arch_id = id_list->cpus[i].arch_id;
> >       }
> >       memory_region_init_io(&state->ctrl_reg, owner, &cpu_hotplug_ops,
> state,
> > diff --git a/include/hw/acpi/cpu.h b/include/hw/acpi/cpu.h
> > index 32654dc274..bd3f9973c9 100644
> > --- a/include/hw/acpi/cpu.h
> > +++ b/include/hw/acpi/cpu.h
> > @@ -26,6 +26,8 @@ typedef struct AcpiCpuStatus {
> >       uint64_t arch_id;
> >       bool is_inserting;
> >       bool is_removing;
> > +    bool is_present;
> > +    bool is_enabled;
> >       bool fw_remove;
> >       uint32_t ost_event;
> >       uint32_t ost_status;
> > @@ -75,4 +77,23 @@ extern const VMStateDescription vmstate_cpu_hotplug;
> >       VMSTATE_STRUCT(cpuhp, state, 1, \
> >                      vmstate_cpu_hotplug, CPUHotplugState)
> >
> > +/**
> > + * acpi_persistent_cpu:
> > + * @cpu: The vCPU to check
> > + *
> > + * Checks if the vCPU state should always be reflected as *present* via
> ACPI
> > + * to the Guest. By default, this is False on all architectures and has
> to be
> > + * explicity set during initialization.
> > + *
> > + * Returns: True if it is ACPI 'persistent' CPU
> > + *
> > + */
> > +static inline bool acpi_persistent_cpu(CPUState *cpu)
> > +{
> > +    /*
> > +     * returns if 'Presence' of the vCPU is persistent and should be
> simulated
> > +     * via ACPI even after vCPUs have been unplugged in QOM
> > +     */
> > +    return cpu && cpu->acpi_persistent;
> > +}
> >   #endif
> > diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
> > index 04e9ad4996..299e96c45b 100644
> > --- a/include/hw/core/cpu.h
> > +++ b/include/hw/core/cpu.h
> > @@ -542,6 +542,27 @@ struct CPUState {
> >       CPUPluginState *plugin_state;
> >   #endif
> >
> > +    /*
> > +     * To implement the vCPU hotplug feature (which simulates CPU
> hotplug
> > +     * behavior), we need to dynamically create and destroy QOM vCPU
> objects,
> > +     * and (de)associate them with pre-existing KVM vCPUs while
> (un)parking the
> > +     * KVM vCPU context. One challenge is ensuring that these
> dynamically
> > +     * appearing or disappearing QOM vCPU objects are accurately
> reflected
> > +     * through ACPI to the Guest Kernel. Due to architectural
> constraints,
> > +     * changing the number of vCPUs after the guest kernel has booted
> may not
> > +     * always be possible.
> > +     *
> > +     * In certain architectures, to provide the guest kernel with a
> *persistent*
> > +     * view of vCPU presence, even when the QOM does not have a
> corresponding
> > +     * vCPU object, ACPI may simulate the presence of vCPUs by marking
> them as
> > +     * ACPI-disabled. This is achieved by setting `_STA.PRES=True` and
> > +     * `_STA.Ena=False` for unplugged vCPUs in QEMU's QOM.
> > +     *
> > +     * By default, this flag is set to `FALSE`, and it must be
> explicitly set
> > +     * to `TRUE` for architectures like ARM.
> > +     */
> > +    bool acpi_persistent;
> > +
> >       /* TODO Move common fields from CPUArchState here. */
> >       int cpu_index;
> >       int cluster_index;
>
>

[-- Attachment #2: Type: text/html, Size: 11027 bytes --]

Re: [PATCH V1 1/4] hw/acpi: Initialize ACPI Hotplug CPU Status with Support for vCPU `Persistence`

[Salil Mehta]

[-- Attachment #1: Type: text/plain, Size: 10534 bytes --]

Hi Gavin,

On Thu, Oct 17, 2024 at 6:27 AM Gavin Shan <gshan@redhat.com> wrote:

> On 10/15/24 5:22 AM, Salil Mehta wrote:
> > Certain CPU architecture specifications [1][2][3] prohibit changes to CPU
> > presence after the kernel has booted. This limitation exists because
> many system
> > initializations rely on the exact CPU count at boot time and do not
> expect it to
> > change later. For example, components like interrupt controllers, which
> are
> > closely tied to CPUs, or various per-CPU features, may not support
> configuration
> > changes once the kernel has been initialized. This presents a challenge
> for
> > virtualization features such as vCPU hotplug.
> >
> > To address this issue, introduce an `is_enabled` state in the
> `AcpiCpuStatus`,
> > which reflects whether a vCPU has been hot-plugged or hot-unplugged in
> QEMU,
> > marking it as (un)available in the Guest Kernel. The `is_present` state
> should
> > be set based on the `acpi_persistent` flag. In cases where unplugged
> vCPUs need
> > to be deliberately simulated in the ACPI to maintain a persistent view
> of vCPUs,
> > this flag ensures the guest kernel continues to see those vCPUs.
> >
> > Additionally, introduce an `acpi_persistent` property that can be used to
> > initialize the ACPI vCPU presence state accordingly. Architectures
> requiring
> > ACPI to expose a persistent view of vCPUs can override its default
> value. Refer
> > to the patch-set implelenting vCPU hotplug support for ARM for more
> details on
> > its usage.
> >
> > References:
> > [1] KVMForum 2023 Presentation: Challenges Revisited in Supporting Virt
> CPU Hotplug on
> >      architectures that don’t Support CPU Hotplug (like ARM64)
> >      a. Kernel Link:
> https://kvm-forum.qemu.org/2023/KVM-forum-cpu-hotplug_7OJ1YyJ.pdf
> >      b. Qemu Link:
> https://kvm-forum.qemu.org/2023/Challenges_Revisited_in_Supporting_Virt_CPU_Hotplug_-__ii0iNb3.pdf
> > [2] KVMForum 2020 Presentation: Challenges in Supporting Virtual CPU
> Hotplug on
> >      SoC Based Systems (like ARM64)
> >      Link: https://kvmforum2020.sched.com/event/eE4m
> > [3] Check comment 5 in the bugzilla entry
> >      Link: https://bugzilla.tianocore.org/show_bug.cgi?id=4481#c5
> >
> > Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> > ---
> >   cpu-target.c          |  1 +
> >   hw/acpi/cpu.c         | 35 ++++++++++++++++++++++++++++++++++-
> >   include/hw/acpi/cpu.h | 21 +++++++++++++++++++++
> >   include/hw/core/cpu.h | 21 +++++++++++++++++++++
> >   4 files changed, 77 insertions(+), 1 deletion(-)
> >
> > diff --git a/cpu-target.c b/cpu-target.c
> > index 499facf774..c8a29ab495 100644
> > --- a/cpu-target.c
> > +++ b/cpu-target.c
> > @@ -200,6 +200,7 @@ static Property cpu_common_props[] = {
> >        */
> >       DEFINE_PROP_LINK("memory", CPUState, memory, TYPE_MEMORY_REGION,
> >                        MemoryRegion *),
> > +    DEFINE_PROP_BOOL("acpi-persistent", CPUState, acpi_persistent,
> false),
> >   #endif
> >       DEFINE_PROP_END_OF_LIST(),
> >   };
>
> Would the property be consistent on all CPUs? I mean it's invalid for this
> property to be true on the first CPU while it's false on other CPUs. If my
> understanding is correct, this property would be part of MachineState
> instead
> of CPUState, and it's not configurable. Is there a particular reason why
> the
> property should be tied with CPUState and configurable?
>


Yes, it might make more sense to keep it at the machine level or configured
via firmware.

It might not look very obvious but the idea of keeping ACPI persistence at
CPUState level was to extend this feature and have some group of vCPUs to be
hot pluggable while others could be forced to be non-hotpluggable.  This
could be useful if in future hot pluggability is extended to
per-die/per-numa
level. I also removed the 'CPUState::disabled' flag which was more of an
architecture specific thing.



> Besides, the property name "acpi-persistent" isn't indicative enough. CPU
> objects can be described through ACPI table or device tree node. So a more
> generic property name may be needed, for example "always_present"? If we
> need move this property from CPUState to MachineStaste or MachineClass,
> the name would be "cpu_always_present" or something like that.
>

This property is more related to the persistence of the vCPU state at
the ACPI
level. Therefore, I changed the name from 'always_present' which I used in
one
of the preliminary prototype presented at the KVM Forum 2023.


>
> > diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
> > index 5cb60ca8bc..083c4010c2 100644
> > --- a/hw/acpi/cpu.c
> > +++ b/hw/acpi/cpu.c
> > @@ -225,7 +225,40 @@ void cpu_hotplug_hw_init(MemoryRegion *as, Object
> *owner,
> >       state->dev_count = id_list->len;
> >       state->devs = g_new0(typeof(*state->devs), state->dev_count);
> >       for (i = 0; i < id_list->len; i++) {
> > -        state->devs[i].cpu =  CPU(id_list->cpus[i].cpu);
> > +        struct CPUState *cpu = CPU(id_list->cpus[i].cpu);
> > +        /*
> > +         * In most architectures, CPUs that are marked as ACPI
> 'present' are
> > +         * also ACPI 'enabled' by default. These states remain
> consistent at
> > +         * both the QOM and ACPI levels.
> > +         */
> > +        if (cpu) {
> > +            state->devs[i].is_enabled = true;
> > +            state->devs[i].is_present = true;
> > +            state->devs[i].cpu = cpu;
> > +        } else {
> > +            state->devs[i].is_enabled = false;
> > +            /*
> > +             * In some architectures, even 'unplugged' or 'disabled'
> QOM CPUs
> > +             * may be exposed as ACPI 'present.' This approach provides
> a
> > +             * persistent view of the vCPUs to the guest kernel. This
> could be
> > +             * due to an architectural constraint that requires every
> per-CPU
> > +             * component to be present at boot time, meaning the exact
> count of
> > +             * vCPUs must be known and cannot be altered after the
> kernel has
> > +             * booted. As a result, the vCPU states at the QOM and ACPI
> levels
> > +             * might become inconsistent. However, in such cases, the
> presence
> > +             * of vCPUs has been deliberately simulated at the ACPI
> level.
> > +             */
> > +            if (acpi_persistent_cpu(first_cpu)) {
> > +                state->devs[i].is_present = true;
> > +                /*
> > +                 * `CPUHotplugState::AcpiCpuStatus::cpu` becomes
> insignificant
> > +                 * in this case
> > +                 */
> > +            } else {
> > +                state->devs[i].is_present = false;
> > +                state->devs[i].cpu = cpu;
> > +            }
> > +        }
> >           state->devs[i].arch_id = id_list->cpus[i].arch_id;
> >       }
> >       memory_region_init_io(&state->ctrl_reg, owner, &cpu_hotplug_ops,
> state,
>
> The code is already self-explaining and the comments explaining how the
> property
> "acpi-persistent" is handled seems a bit duplicate. If you agree, lets
> make comments
> short and concise.
>


We can definitely make it more succinct


>
> > diff --git a/include/hw/acpi/cpu.h b/include/hw/acpi/cpu.h
> > index 32654dc274..bd3f9973c9 100644
> > --- a/include/hw/acpi/cpu.h
> > +++ b/include/hw/acpi/cpu.h
> > @@ -26,6 +26,8 @@ typedef struct AcpiCpuStatus {
> >       uint64_t arch_id;
> >       bool is_inserting;
> >       bool is_removing;
> > +    bool is_present;
> > +    bool is_enabled;
> >       bool fw_remove;
> >       uint32_t ost_event;
> >       uint32_t ost_status;
> > @@ -75,4 +77,23 @@ extern const VMStateDescription vmstate_cpu_hotplug;
> >       VMSTATE_STRUCT(cpuhp, state, 1, \
> >                      vmstate_cpu_hotplug, CPUHotplugState)
> >
> > +/**
> > + * acpi_persistent_cpu:
> > + * @cpu: The vCPU to check
> > + *
> > + * Checks if the vCPU state should always be reflected as *present* via
> ACPI
> > + * to the Guest. By default, this is False on all architectures and has
> to be
> > + * explicity set during initialization.
> > + *
> > + * Returns: True if it is ACPI 'persistent' CPU
> > + *
> > + */
> > +static inline bool acpi_persistent_cpu(CPUState *cpu)
> > +{
> > +    /*
> > +     * returns if 'Presence' of the vCPU is persistent and should be
> simulated
> > +     * via ACPI even after vCPUs have been unplugged in QOM
> > +     */
> > +    return cpu && cpu->acpi_persistent;
> > +}
> >   #endif
> > diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
> > index 04e9ad4996..299e96c45b 100644
> > --- a/include/hw/core/cpu.h
> > +++ b/include/hw/core/cpu.h
> > @@ -542,6 +542,27 @@ struct CPUState {
> >       CPUPluginState *plugin_state;
> >   #endif
> >
> > +    /*
> > +     * To implement the vCPU hotplug feature (which simulates CPU
> hotplug
> > +     * behavior), we need to dynamically create and destroy QOM vCPU
> objects,
> > +     * and (de)associate them with pre-existing KVM vCPUs while
> (un)parking the
> > +     * KVM vCPU context. One challenge is ensuring that these
> dynamically
> > +     * appearing or disappearing QOM vCPU objects are accurately
> reflected
> > +     * through ACPI to the Guest Kernel. Due to architectural
> constraints,
> > +     * changing the number of vCPUs after the guest kernel has booted
> may not
> > +     * always be possible.
> > +     *
> > +     * In certain architectures, to provide the guest kernel with a
> *persistent*
> > +     * view of vCPU presence, even when the QOM does not have a
> corresponding
> > +     * vCPU object, ACPI may simulate the presence of vCPUs by marking
> them as
> > +     * ACPI-disabled. This is achieved by setting `_STA.PRES=True` and
> > +     * `_STA.Ena=False` for unplugged vCPUs in QEMU's QOM.
> > +     *
> > +     * By default, this flag is set to `FALSE`, and it must be
> explicitly set
> > +     * to `TRUE` for architectures like ARM.
> > +     */
> > +    bool acpi_persistent;
> > +
>
> The comments can be short and concise either. Reader can refer to the
> commit log for all the details.
>

Yes, we can

thanks


>
> >       /* TODO Move common fields from CPUArchState here. */
> >       int cpu_index;
> >       int cluster_index;
>
> Thanks,
> Gavin
>
>

[-- Attachment #2: Type: text/html, Size: 13650 bytes --]

Re: [PATCH V1 1/4] hw/acpi: Initialize ACPI Hotplug CPU Status with Support for vCPU `Persistence`

[Salil Mehta]

[-- Attachment #1: Type: text/plain, Size: 9974 bytes --]

Hi Gustavo

On Thu, Oct 17, 2024 at 9:25 PM Gustavo Romero <gustavo.romero@linaro.org>
wrote:

> Hi Salil,
>
> On 10/14/24 16:22, Salil Mehta wrote:
> > Certain CPU architecture specifications [1][2][3] prohibit changes to CPU
> > presence after the kernel has booted. This limitation exists because
> many system
> > initializations rely on the exact CPU count at boot time and do not
> expect it to
> > change later. For example, components like interrupt controllers, which
> are
> > closely tied to CPUs, or various per-CPU features, may not support
> configuration
> > changes once the kernel has been initialized. This presents a challenge
> for
> > virtualization features such as vCPU hotplug.
> >
> > To address this issue, introduce an `is_enabled` state in the
> `AcpiCpuStatus`,
> > which reflects whether a vCPU has been hot-plugged or hot-unplugged in
> QEMU,
> > marking it as (un)available in the Guest Kernel. The `is_present` state
> should
> > be set based on the `acpi_persistent` flag. In cases where unplugged
> vCPUs need
> > to be deliberately simulated in the ACPI to maintain a persistent view
> of vCPUs,
> > this flag ensures the guest kernel continues to see those vCPUs.
> >
> > Additionally, introduce an `acpi_persistent` property that can be used to
> > initialize the ACPI vCPU presence state accordingly. Architectures
> requiring
> > ACPI to expose a persistent view of vCPUs can override its default
> value. Refer
> > to the patch-set implelenting vCPU hotplug support for ARM for more
> details on
> > its usage.
> >
> > References:
> > [1] KVMForum 2023 Presentation: Challenges Revisited in Supporting Virt
> CPU Hotplug on
> >      architectures that don’t Support CPU Hotplug (like ARM64)
> >      a. Kernel Link:
> https://kvm-forum.qemu.org/2023/KVM-forum-cpu-hotplug_7OJ1YyJ.pdf
> >      b. Qemu Link:
> https://kvm-forum.qemu.org/2023/Challenges_Revisited_in_Supporting_Virt_CPU_Hotplug_-__ii0iNb3.pdf
> > [2] KVMForum 2020 Presentation: Challenges in Supporting Virtual CPU
> Hotplug on
> >      SoC Based Systems (like ARM64)
> >      Link: https://kvmforum2020.sched.com/event/eE4m
> > [3] Check comment 5 in the bugzilla entry
> >      Link: https://bugzilla.tianocore.org/show_bug.cgi?id=4481#c5
> >
> > Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> > ---
> >   cpu-target.c          |  1 +
> >   hw/acpi/cpu.c         | 35 ++++++++++++++++++++++++++++++++++-
> >   include/hw/acpi/cpu.h | 21 +++++++++++++++++++++
> >   include/hw/core/cpu.h | 21 +++++++++++++++++++++
> >   4 files changed, 77 insertions(+), 1 deletion(-)
> >
> > diff --git a/cpu-target.c b/cpu-target.c
> > index 499facf774..c8a29ab495 100644
> > --- a/cpu-target.c
> > +++ b/cpu-target.c
> > @@ -200,6 +200,7 @@ static Property cpu_common_props[] = {
> >        */
> >       DEFINE_PROP_LINK("memory", CPUState, memory, TYPE_MEMORY_REGION,
> >                        MemoryRegion *),
> > +    DEFINE_PROP_BOOL("acpi-persistent", CPUState, acpi_persistent,
> false),
> >   #endif
> >       DEFINE_PROP_END_OF_LIST(),
> >   };
> > diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
> > index 5cb60ca8bc..083c4010c2 100644
> > --- a/hw/acpi/cpu.c
> > +++ b/hw/acpi/cpu.c
> > @@ -225,7 +225,40 @@ void cpu_hotplug_hw_init(MemoryRegion *as, Object
> *owner,
> >       state->dev_count = id_list->len;
> >       state->devs = g_new0(typeof(*state->devs), state->dev_count);
> >       for (i = 0; i < id_list->len; i++) {
> > -        state->devs[i].cpu =  CPU(id_list->cpus[i].cpu);
> > +        struct CPUState *cpu = CPU(id_list->cpus[i].cpu);
> > +        /*
> > +         * In most architectures, CPUs that are marked as ACPI
> 'present' are
> > +         * also ACPI 'enabled' by default. These states remain
> consistent at
> > +         * both the QOM and ACPI levels.
> > +         */
> > +        if (cpu) {
> > +            state->devs[i].is_enabled = true;
> > +            state->devs[i].is_present = true;
> > +            state->devs[i].cpu = cpu;
> > +        } else {
> > +            state->devs[i].is_enabled = false;
> > +            /*
> > +             * In some architectures, even 'unplugged' or 'disabled'
> QOM CPUs
> > +             * may be exposed as ACPI 'present.' This approach provides
> a
> > +             * persistent view of the vCPUs to the guest kernel. This
> could be
> > +             * due to an architectural constraint that requires every
> per-CPU
> > +             * component to be present at boot time, meaning the exact
> count of
> > +             * vCPUs must be known and cannot be altered after the
> kernel has
> > +             * booted. As a result, the vCPU states at the QOM and ACPI
> levels
> > +             * might become inconsistent. However, in such cases, the
> presence
> > +             * of vCPUs has been deliberately simulated at the ACPI
> level.
> > +             */
> > +            if (acpi_persistent_cpu(first_cpu)) {
> > +                state->devs[i].is_present = true;
> > +                /*
> > +                 * `CPUHotplugState::AcpiCpuStatus::cpu` becomes
> insignificant
> > +                 * in this case
> > +                 */
> > +            } else {
> > +                state->devs[i].is_present = false;
> > +                state->devs[i].cpu = cpu;
>
> I think it's better to set cpu here explicitly to NULL in both cases
> (persistent and non-persistent cases). Also, 'cpu' here is always NULL
> since it's inside the else block of "if (cpu)" conditional. So how about
> setting cpu to NULL at the end of the else block:
>

Good catch. Yes we can. It got under my hood after changes from RFC V4 to
RFC V5


>
> diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
> index d34c1e601e..b830c0e85b 100644
> --- a/hw/acpi/cpu.c
> +++ b/hw/acpi/cpu.c
> @@ -251,14 +251,14 @@ void cpu_hotplug_hw_init(MemoryRegion *as, Object
> *owner,
>                */
>               if (acpi_persistent_cpu(first_cpu)) {
>                   state->devs[i].is_present = true;
> -                /*
> -                 * `CPUHotplugState::AcpiCpuStatus::cpu` becomes
> insignificant
> -                 * in this case
> -                 */
>               } else {
>                   state->devs[i].is_present = false;
> -                state->devs[i].cpu = cpu;
>               }
> +            /*
> +             * `CPUHotplugState::AcpiCpuStatus::cpu` becomes insignificant
> +             * in this case
> +             */
> +            state->devs[i].cpu = NULL;
>           }
>           state->devs[i].arch_id = id_list->cpus[i].arch_id;
>       }
>
>
> Cheers,
> Gustavo
>
> > +            }
> > +        }
> >           state->devs[i].arch_id = id_list->cpus[i].arch_id;
> >       }
> >       memory_region_init_io(&state->ctrl_reg, owner, &cpu_hotplug_ops,
> state,
> > diff --git a/include/hw/acpi/cpu.h b/include/hw/acpi/cpu.h
> > index 32654dc274..bd3f9973c9 100644
> > --- a/include/hw/acpi/cpu.h
> > +++ b/include/hw/acpi/cpu.h
> > @@ -26,6 +26,8 @@ typedef struct AcpiCpuStatus {
> >       uint64_t arch_id;
> >       bool is_inserting;
> >       bool is_removing;
> > +    bool is_present;
> > +    bool is_enabled;
> >       bool fw_remove;
> >       uint32_t ost_event;
> >       uint32_t ost_status;
> > @@ -75,4 +77,23 @@ extern const VMStateDescription vmstate_cpu_hotplug;
> >       VMSTATE_STRUCT(cpuhp, state, 1, \
> >                      vmstate_cpu_hotplug, CPUHotplugState)
> >
> > +/**
> > + * acpi_persistent_cpu:
> > + * @cpu: The vCPU to check
> > + *
> > + * Checks if the vCPU state should always be reflected as *present* via
> ACPI
> > + * to the Guest. By default, this is False on all architectures and has
> to be
> > + * explicity set during initialization.
> > + *
> > + * Returns: True if it is ACPI 'persistent' CPU
> > + *
> > + */
> > +static inline bool acpi_persistent_cpu(CPUState *cpu)
> > +{
> > +    /*
> > +     * returns if 'Presence' of the vCPU is persistent and should be
> simulated
> > +     * via ACPI even after vCPUs have been unplugged in QOM
> > +     */
> > +    return cpu && cpu->acpi_persistent;
> > +}
> >   #endif
> > diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
> > index 04e9ad4996..299e96c45b 100644
> > --- a/include/hw/core/cpu.h
> > +++ b/include/hw/core/cpu.h
> > @@ -542,6 +542,27 @@ struct CPUState {
> >       CPUPluginState *plugin_state;
> >   #endif
> >
> > +    /*
> > +     * To implement the vCPU hotplug feature (which simulates CPU
> hotplug
> > +     * behavior), we need to dynamically create and destroy QOM vCPU
> objects,
> > +     * and (de)associate them with pre-existing KVM vCPUs while
> (un)parking the
> > +     * KVM vCPU context. One challenge is ensuring that these
> dynamically
> > +     * appearing or disappearing QOM vCPU objects are accurately
> reflected
> > +     * through ACPI to the Guest Kernel. Due to architectural
> constraints,
> > +     * changing the number of vCPUs after the guest kernel has booted
> may not
> > +     * always be possible.
> > +     *
> > +     * In certain architectures, to provide the guest kernel with a
> *persistent*
> > +     * view of vCPU presence, even when the QOM does not have a
> corresponding
> > +     * vCPU object, ACPI may simulate the presence of vCPUs by marking
> them as
> > +     * ACPI-disabled. This is achieved by setting `_STA.PRES=True` and
> > +     * `_STA.Ena=False` for unplugged vCPUs in QEMU's QOM.
> > +     *
> > +     * By default, this flag is set to `FALSE`, and it must be
> explicitly set
> > +     * to `TRUE` for architectures like ARM.
> > +     */
> > +    bool acpi_persistent;
> > +
> >       /* TODO Move common fields from CPUArchState here. */
> >       int cpu_index;
> >       int cluster_index;
>
>

[-- Attachment #2: Type: text/html, Size: 12544 bytes --]

Re: [PATCH V1 1/4] hw/acpi: Initialize ACPI Hotplug CPU Status with Support for vCPU `Persistence`

[Salil Mehta]

[-- Attachment #1: Type: text/plain, Size: 14456 bytes --]

Hi Igor,

Thanks for taking time to review and sorry for not being prompt. I was in
transit
due to some difficult personal situation.

On Fri, Oct 18, 2024 at 3:11 PM Igor Mammedov <imammedo@redhat.com> wrote:

> On Mon, 14 Oct 2024 20:22:02 +0100
> Salil Mehta <salil.mehta@huawei.com> wrote:
>
> > Certain CPU architecture specifications [1][2][3] prohibit changes to CPU
>                                           ^^^^^^^^^
> these do not point to specs but never mind
>
> > presence after the kernel has booted. This limitation exists because
> many system
> > initializations rely on the exact CPU count at boot time and do not
> expect it to
> > change later. For example, components like interrupt controllers, which
> are
> > closely tied to CPUs, or various per-CPU features, may not support
> configuration
> > changes once the kernel has been initialized. This presents a challenge
> for
> > virtualization features such as vCPU hotplug.
>
> well, x86 (incl cpu,interrupt ctrls) also doesn't have architectural
> hotplug.
> It's just OEM managed to implement it regardless and then bothered to make
> OS changes to work with that.
> It's just ARM community doesn't want to go there at this point of time
> but using above reasoning as justification for this series doesn't look
> good to me.
>


There is a difference though. vCPU presence cannot be changed after the
system has
initialized in the ARM world due to the Architectural constraint. I think
in the x86 world
it is allowed?


>
> So what ARM would like to support is not CPU hotplug but rather a fixed
> system with standby CPUs (that can be powered on/off on demand).
> With ACPI spec amended to support that (MADT present/enabled changes), it's
> good enough reason to add 'enabled' handling to acpi cpu-hotplug code
> instead
> of inventing alternative one that would do almost the same.
>


Not sure what you mean here but this is what is being done.



> But lets get rid of (can't/may not) language above and use standby CPUs
> reasoning
> to avoid any confusion.
>


I've to disagree here. Standy-by means you will have to keep all the vCPUs
states
realized and the objects will always exist. This is a problem for larger
systems
with vCPU hotplug support as it drastically affects the boot times. Please
check
the KVM Forum 2023 slides for objective values.

It's been a long time since this series was actually conceived and at that
time we wanted
to use it for kata containers but later with the gradual adoption of
various microvms
and the cloud hypervisor requirements have bit changed. Boot time still
remains one
of the major requirements. Not bounding boot time while using this feature
will
seriously affect performance if the number of vCPUs increases



>
> PS:
> I'm taking about hw hotplug (at QEMU level) and not kernel hotplug
> (where it's at logical cpu level).
>

sure


>
> > To address this issue, introduce an `is_enabled` state in the
> `AcpiCpuStatus`,
> > which reflects whether a vCPU has been hot-plugged or hot-unplugged in
> QEMU,
> > marking it as (un)available in the Guest Kernel.
> good so far
>
> > The `is_present` state should
> > be set based on the `acpi_persistent` flag. In cases where unplugged
> vCPUs need
> > to be deliberately simulated in the ACPI to maintain a persistent view
> of vCPUs,
> > this flag ensures the guest kernel continues to see those vCPUs.
>
> that's where I have to disagree, vCPU is present when a corresponding QOM
> object
> exists. Faking it's presence will only confuse at the end.
>


Not faking care of it will mean realization  of the unnecessary vCPU
threads during
the VM init time, which in turn will increase the boot time. Trade-off
between more
clean design and performance.


>
> I get that you want to reuse device_add/del interface, but that leads to
> pulling the leg here and there to make thing fit. That in short therm
> (while someone remembers all tricks) might work for some, but long therm
> it's not sustainable).
>


I do not understand why?


>
> Maybe instead of pushing device_add/del, we should rather implement
> standby CPU model here, as ARM folks expect it to be.
> i.e. instead of device_add/del add 'enabled' property to ARM vCPU,
> and let management to turn on/off vCPUs on demand.
> (and very likely this model could be reused by other sock based platforms)
> For end-user it would be the same as device_add/del (i.e. vCPU becomes
> usable/unsable)
>

One of the main goals is to facilitate seamless migration from the x86
world to
ARM world. Hence, preservation of the x86 interface is important. It is a
requirement.
Just imagine if Apple had to change end user interface experience after
migrating iOS
from x86 to ARM architecture. ?



>
> I'd bet it would simplify your ARM CPU hotplug series a lot,
> since you won't have to fake vCPU object lifetime and do
> non trivial tricks to make it all work.
> Which it turn will make ARM hotplug series much more approachable
> for reviewers /and whomever comes later across that code/.
>

Tricks are just for ACPI and GIC and nothing else. Rest is a
boilerplate code of the
vCPU hotplug framework which x86 is also using.


>
> Regardless of said, we would still need changes to ACPI cphp code,
> see my comments inline.
>

sure.


>
>
> > Additionally, introduce an `acpi_persistent` property that can be used to
> > initialize the ACPI vCPU presence state accordingly. Architectures
> requiring
> > ACPI to expose a persistent view of vCPUs can override its default
> value. Refer
> > to the patch-set implelenting vCPU hotplug support for ARM for more
> details on
> > its usage.
> >
> > References:
> > [1] KVMForum 2023 Presentation: Challenges Revisited in Supporting Virt
> CPU Hotplug on
> >     architectures that don’t Support CPU Hotplug (like ARM64)
> >     a. Kernel Link:
> https://kvm-forum.qemu.org/2023/KVM-forum-cpu-hotplug_7OJ1YyJ.pdf
> >     b. Qemu Link:
> https://kvm-forum.qemu.org/2023/Challenges_Revisited_in_Supporting_Virt_CPU_Hotplug_-__ii0iNb3.pdf
> > [2] KVMForum 2020 Presentation: Challenges in Supporting Virtual CPU
> Hotplug on
> >     SoC Based Systems (like ARM64)
> >     Link: https://kvmforum2020.sched.com/event/eE4m
> > [3] Check comment 5 in the bugzilla entry
> >     Link: https://bugzilla.tianocore.org/show_bug.cgi?id=4481#c5
> >
> > Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> > ---
> >  cpu-target.c          |  1 +
> >  hw/acpi/cpu.c         | 35 ++++++++++++++++++++++++++++++++++-
> >  include/hw/acpi/cpu.h | 21 +++++++++++++++++++++
> >  include/hw/core/cpu.h | 21 +++++++++++++++++++++
> >  4 files changed, 77 insertions(+), 1 deletion(-)
> >
> > diff --git a/cpu-target.c b/cpu-target.c
> > index 499facf774..c8a29ab495 100644
> > --- a/cpu-target.c
> > +++ b/cpu-target.c
> > @@ -200,6 +200,7 @@ static Property cpu_common_props[] = {
> >       */
> >      DEFINE_PROP_LINK("memory", CPUState, memory, TYPE_MEMORY_REGION,
> >                       MemoryRegion *),
> > +    DEFINE_PROP_BOOL("acpi-persistent", CPUState, acpi_persistent,
> false),
>
> I agree with Gavin, it's not CPU property/business, but a platform one.
>
> Pass it as argument to cpu_hotplug_hw_init(),
> and maybe rename to always_present.
> Then make sure that it's configurable in GED (which calls the function),
> and just turn it on for arm/virt machine.
> Other platforms might want to use x86 approach with GED and have
> vCPU actually disappearing. /loongson and maybe risc-v/
>

can we move it to Machine level or fetch it through firmware?


>
> >  #endif
> >      DEFINE_PROP_END_OF_LIST(),
> >  };
> > diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
> > index 5cb60ca8bc..083c4010c2 100644
> > --- a/hw/acpi/cpu.c
> > +++ b/hw/acpi/cpu.c
> > @@ -225,7 +225,40 @@ void cpu_hotplug_hw_init(MemoryRegion *as, Object
> *owner,
> >      state->dev_count = id_list->len;
> >      state->devs = g_new0(typeof(*state->devs), state->dev_count);
> >      for (i = 0; i < id_list->len; i++) {
> > -        state->devs[i].cpu =  CPU(id_list->cpus[i].cpu);
> > +        struct CPUState *cpu = CPU(id_list->cpus[i].cpu);
> > +        /*
> > +         * In most architectures, CPUs that are marked as ACPI
> 'present' are
> > +         * also ACPI 'enabled' by default. These states remain
> consistent at
> > +         * both the QOM and ACPI levels.
> > +         */
> > +        if (cpu) {
> > +            state->devs[i].is_enabled = true;
> > +            state->devs[i].is_present = true;
> > +            state->devs[i].cpu = cpu;
> > +        } else {
> > +            state->devs[i].is_enabled = false;
> > +            /*
> > +             * In some architectures, even 'unplugged' or 'disabled'
> QOM CPUs
> > +             * may be exposed as ACPI 'present.' This approach provides
> a
> > +             * persistent view of the vCPUs to the guest kernel. This
> could be
> > +             * due to an architectural constraint that requires every
> per-CPU
> > +             * component to be present at boot time, meaning the exact
> count of
> > +             * vCPUs must be known and cannot be altered after the
> kernel has
> > +             * booted. As a result, the vCPU states at the QOM and ACPI
> levels
> > +             * might become inconsistent. However, in such cases, the
> presence
> > +             * of vCPUs has been deliberately simulated at the ACPI
> level.
> > +             */
>
> if cpus are not 'simulated', you will not need comments explaining that all
> over place and whole hunk would likely go away.
>

I can reduce the content if you wish.


>
> > +            if (acpi_persistent_cpu(first_cpu)) {
> > +                state->devs[i].is_present = true;
> > +                /*
> > +                 * `CPUHotplugState::AcpiCpuStatus::cpu` becomes
> insignificant
> > +                 * in this case
> > +                 */
> > +            } else {
> > +                state->devs[i].is_present = false;
> > +                state->devs[i].cpu = cpu;
> > +            }
> > +        }
> >          state->devs[i].arch_id = id_list->cpus[i].arch_id;
> >      }
> >      memory_region_init_io(&state->ctrl_reg, owner, &cpu_hotplug_ops,
> state,
> > diff --git a/include/hw/acpi/cpu.h b/include/hw/acpi/cpu.h
> > index 32654dc274..bd3f9973c9 100644
> > --- a/include/hw/acpi/cpu.h
> > +++ b/include/hw/acpi/cpu.h
> > @@ -26,6 +26,8 @@ typedef struct AcpiCpuStatus {
> >      uint64_t arch_id;
> >      bool is_inserting;
> >      bool is_removing;
> > +    bool is_present;
> with always_present, it might be better to move field to CPUHotplugState
> as it's not per vCPU anymore, and in standby case state->devs[i].cpu
> should work as implicit present flag. (see below wrt doc first comment)
>

I'm still not convinced of the stand-by approach because of the performance
impact
it will have upon increasing the number of possible vCPUs and its worst
case is
unbounded. That's a major problem.



> > +    bool is_enabled;
> I'd move introduction of this field into a separate patch.
>
> BTW: new ABI/fields accessible by guest should be described in
> docs/specs/acpi_cpu_hotplug.rst.
> It would be better to have the spec as patch 1st, that we all agree on
> and then follow with implementation.
>

We can do that.


> And also include there an expected workflow for standby case.



We need more discussion on this as our requirements for which we floated
this
feature are not being met using stand-by cases.

Thanks!

Best regards
Salil.

>
>
> >      bool fw_remove;
> >      uint32_t ost_event;
> >      uint32_t ost_status;
> > @@ -75,4 +77,23 @@ extern const VMStateDescription vmstate_cpu_hotplug;
> >      VMSTATE_STRUCT(cpuhp, state, 1, \
> >                     vmstate_cpu_hotplug, CPUHotplugState)
> >
> > +/**
> > + * acpi_persistent_cpu:
> > + * @cpu: The vCPU to check
> > + *
> > + * Checks if the vCPU state should always be reflected as *present* via
> ACPI
> > + * to the Guest. By default, this is False on all architectures and has
> to be
> > + * explicity set during initialization.
> > + *
> > + * Returns: True if it is ACPI 'persistent' CPU
> > + *
> > + */
> > +static inline bool acpi_persistent_cpu(CPUState *cpu)
> > +{
> > +    /*
> > +     * returns if 'Presence' of the vCPU is persistent and should be
> simulated
> > +     * via ACPI even after vCPUs have been unplugged in QOM
> > +     */
> > +    return cpu && cpu->acpi_persistent;
> > +}
> >  #endif
> > diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
> > index 04e9ad4996..299e96c45b 100644
> > --- a/include/hw/core/cpu.h
> > +++ b/include/hw/core/cpu.h
> > @@ -542,6 +542,27 @@ struct CPUState {
> >      CPUPluginState *plugin_state;
> >  #endif
> >
> > +    /*
> > +     * To implement the vCPU hotplug feature (which simulates CPU
> hotplug
> > +     * behavior), we need to dynamically create and destroy QOM vCPU
> objects,
> > +     * and (de)associate them with pre-existing KVM vCPUs while
> (un)parking the
> > +     * KVM vCPU context. One challenge is ensuring that these
> dynamically
> > +     * appearing or disappearing QOM vCPU objects are accurately
> reflected
> > +     * through ACPI to the Guest Kernel. Due to architectural
> constraints,
> > +     * changing the number of vCPUs after the guest kernel has booted
> may not
> > +     * always be possible.
> > +     *
> > +     * In certain architectures, to provide the guest kernel with a
> *persistent*
> > +     * view of vCPU presence, even when the QOM does not have a
> corresponding
> > +     * vCPU object, ACPI may simulate the presence of vCPUs by marking
> them as
> > +     * ACPI-disabled. This is achieved by setting `_STA.PRES=True` and
> > +     * `_STA.Ena=False` for unplugged vCPUs in QEMU's QOM.
> > +     *
> > +     * By default, this flag is set to `FALSE`, and it must be
> explicitly set
> > +     * to `TRUE` for architectures like ARM.
> > +     */
> > +    bool acpi_persistent;
> > +
> >      /* TODO Move common fields from CPUArchState here. */
> >      int cpu_index;
> >      int cluster_index;
>
>

[-- Attachment #2: Type: text/html, Size: 19577 bytes --]

Re: [PATCH V1 0/4] Arch agnostic ACPI changes to support vCPU Hotplug (on Archs like ARM)

[Gustavo Romero]

Hi Miguel!

On 10/21/24 05:04, Miguel Luis wrote:
> Hi Gustavo,
> 
>> On 18 Oct 2024, at 17:57, Gustavo Romero <gustavo.romero@linaro.org> wrote:
>>
>> Hi Miguel,
>>
>> On 10/15/24 15:41, Miguel Luis wrote:
>>> Hi Salil,
>>> I’ve ran the usual tests successfully of hotplug/unplug from the number of cold-booted cpus up to maxcpus and migration on ARM. Please feel free to add:
>>
>> Do you mind sharing what cpus you used for your tests?
>>
> 
> Not at all. I’ve used -cpu host and -cpu max. I had used RFC-v5 from [1] in order to test
> this patchset as they are at the base of it.

ah, got it. Thanks for the clarification, that's what
I was wondering at first :)


>> Did you use tcg or kvm for the tests?
> 
> I’ve used both in the following configurations:
> 
> -M virt -accel kvm -cpu host
> -M virt,gic_version=3 -accel kvm -cpu host
> -M virt,gic_version=3 -accel tcg -cpu max
> 
> And
> 
> -M virt -accel tcg -cpu max
> 
> This last one presented defects in QEMU on behalf of RFC-V5 but, if you don’t
> mind, instead of me transcribing everything here, could you please take a look
> at [2] ?
> 
> I could have given more detail from start, apologies for any confusion on my part.

Sure, thanks for the pointers ;)


Cheers,
Gustavo

RE: [PATCH V1 2/4] hw/acpi: Update ACPI CPU Status `is_{present, enabled}` during vCPU hot(un)plug

[Salil Mehta via]

Hi Igor,

>  From: Igor Mammedov <imammedo@redhat.com>
>  Sent: Friday, October 18, 2024 3:18 PM
>  To: Salil Mehta <salil.mehta@huawei.com>
>  
>  On Mon, 14 Oct 2024 20:22:03 +0100
>  Salil Mehta <salil.mehta@huawei.com> wrote:
>  
>  > Update the `AcpiCpuStatus` for `is_enabled` and `is_present`
>  > accordingly when vCPUs are hot-plugged or hot-unplugged, taking into
>  > account the *persistence* of the vCPUs.
>  >
>  > Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
>  > ---
>  >  hw/acpi/cpu.c | 7 +++++++
>  >  1 file changed, 7 insertions(+)
>  >
>  > diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c index
>  > 083c4010c2..700aa855e9 100644
>  > --- a/hw/acpi/cpu.c
>  > +++ b/hw/acpi/cpu.c
>  > @@ -291,6 +291,8 @@ void acpi_cpu_plug_cb(HotplugHandler
>  *hotplug_dev,
>  >      }
>  >
>  >      cdev->cpu = CPU(dev);
>  > +    cdev->is_present = true;
>  > +    cdev->is_enabled = true;
>  
>  hmm, if cpu is always present, then these fields are redundant since
>    (!cdev->cpu) == present
>  and
>    then is_enabled could be fetched from cdev->cpu directly


I believe you are assuming, that all the QOM possible vCPUs objects will
exists (and hence realized) for the entire life time of the VM? If yes, then
we need to first debate on that as we cannot let presence of all the possible
vCPUs affect  the boot time. This is one of the key requirement from this
feature.


>  
>  >      if (dev->hotplugged) {
>  >          cdev->is_inserting = true;
>  >          acpi_send_event(DEVICE(hotplug_dev),
>  > ACPI_CPU_HOTPLUG_STATUS); @@ -322,6 +324,11 @@ void
>  acpi_cpu_unplug_cb(CPUHotplugState *cpu_st,
>  >          return;
>  >      }
>  >
>  > +    cdev->is_enabled = false;
>  > +    if (!acpi_persistent_cpu(CPU(dev))) {
>  > +        cdev->is_present = false;
>  > +    }
>  
>  and other way around works as well.
>  
>  then we don't have to carry around extra state and making sure that it's in
>  sync/migrated.
>  
>  > +
>  >      cdev->cpu = NULL;
>  >  }
>  >
>  

RE: [PATCH V1 4/4] hw/acpi: Populate vCPU Hotplug VMSD to migrate `is_{present,enabled}` states

[Salil Mehta via]

Hi Igor,

>  From: Igor Mammedov <imammedo@redhat.com>
>  Sent: Friday, October 18, 2024 3:31 PM
>  To: Salil Mehta <salil.mehta@huawei.com>
>  
>  On Mon, 14 Oct 2024 20:22:05 +0100
>  Salil Mehta <salil.mehta@huawei.com> wrote:
>  
>  > The ACPI CPU hotplug states `is_{present, enabled}` must be migrated
>  > alongside other vCPU hotplug states to the destination VM. Therefore,
>  > they should be integrated into the existing CPU Hotplug VM State
>  Description (VMSD) table.
>  > Depending on the architecture and its implementation of CPU hotplug
>  > events (such as ACPI GED, etc.), the CPU hotplug states should be
>  > populated appropriately within their corresponding subsections of the
>  VMSD table.
>  >
>  >     "acpi-ged (16)": {
>  >         "ged_state": {
>  >             "sel": "0x00000000"
>  >         },
>  >         [...]
>  >         "acpi-ged/cpuhp": {
>  >             "cpuhp_state": {
>  >                 "selector": "0x00000005",
>  >                 "command": "0x00",
>  >                 "devs": [
>  >                     {
>  >                         "is_inserting": false,
>  >                         "is_removing": false,
>  >                         "is_present": true,
>  >                         "is_enabled": true,
>  >                         "ost_event": "0x00000000",
>  >                         "ost_status": "0x00000000"
>  >                     },
>  >                     {
>  >                         "is_inserting": false,
>  >                         "is_removing": false,
>  >                         "is_present": true,
>  >                         "is_enabled": true,
>  >                         "ost_event": "0x00000000",
>  >                         "ost_status": "0x00000000"
>  >                     },
>  >                     {
>  >                         "is_inserting": false,
>  >                         "is_removing": false,
>  >                         "is_present": true,
>  >                         "is_enabled": true,
>  >                         "ost_event": "0x00000000",
>  >                         "ost_status": "0x00000000"
>  >                     },
>  >                     {
>  >                         "is_inserting": false,
>  >                         "is_removing": false,
>  >                         "is_present": true,
>  >                         "is_enabled": true,
>  >                         "ost_event": "0x00000000",
>  >                         "ost_status": "0x00000000"
>  >                     },
>  >                     {
>  >                         "is_inserting": false,
>  >                         "is_removing": false,
>  >                         "is_present": true,
>  >                         "is_enabled": false,
>  >                         "ost_event": "0x00000000",
>  >                         "ost_status": "0x00000000"
>  >                     },
>  >                     {
>  >                         "is_inserting": false,
>  >                         "is_removing": false,
>  >                         "is_present": true,
>  >                         "is_enabled": false,
>  >                         "ost_event": "0x00000000",
>  >                         "ost_status": "0x00000000"
>  >                     }
>  >                 ]
>  >             }
>  >         }
>  >     },
>  >
>  > Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
>  > ---
>  >  hw/acpi/cpu.c                  |  2 ++
>  >  hw/acpi/generic_event_device.c | 11 +++++++++++
>  >  2 files changed, 13 insertions(+)
>  >
>  > diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c index
>  > 23ea2b9c70..d34c1e601e 100644
>  > --- a/hw/acpi/cpu.c
>  > +++ b/hw/acpi/cpu.c
>  > @@ -340,6 +340,8 @@ static const VMStateDescription
>  vmstate_cpuhp_sts = {
>  >      .fields = (const VMStateField[]) {
>  >          VMSTATE_BOOL(is_inserting, AcpiCpuStatus),
>  >          VMSTATE_BOOL(is_removing, AcpiCpuStatus),
>  > +        VMSTATE_BOOL(is_present, AcpiCpuStatus),
>  > +        VMSTATE_BOOL(is_enabled, AcpiCpuStatus),
>  
>  that's likely will break x86 migration,


Point taken. It would helpful if you can elucidate further how that
might happen?


>  but before bothering peterx, maybe we won't need this hunk if is_enabled
>  is migrated as part of vCPU state.


Again. This entire argument hinges on the fact whether to keep all the
possible vCPUs realized or not. I think we need a thorough discussion
on that first so that pain of all the stakeholders can be addressed.



>  >          VMSTATE_UINT32(ost_event, AcpiCpuStatus),
>  >          VMSTATE_UINT32(ost_status, AcpiCpuStatus),
>  >          VMSTATE_END_OF_LIST()
>  > diff --git a/hw/acpi/generic_event_device.c
>  > b/hw/acpi/generic_event_device.c index 15b4c3ebbf..a4d78a534c 100644
>  > --- a/hw/acpi/generic_event_device.c
>  > +++ b/hw/acpi/generic_event_device.c
>  > @@ -331,6 +331,16 @@ static const VMStateDescription
>  vmstate_memhp_state = {
>  >      }
>  >  };
>  >
>  > +static const VMStateDescription vmstate_cpuhp_state = {
>  > +    .name = "acpi-ged/cpuhp",
>  > +    .version_id = 1,
>  > +    .minimum_version_id = 1,
>  > +    .fields      = (VMStateField[]) {
>  > +        VMSTATE_CPU_HOTPLUG(cpuhp_state, AcpiGedState),
>  > +        VMSTATE_END_OF_LIST()
>  > +    }
>  > +};
>  
>  this subsection likely needs is_needed hook to avoid breaking case where
>  target doesn't have cpuhp support (older QEMU)


Sure. AFAICS by checking the usage of this in the other parts of the code
and please correct me if I'm wrong here,

#1. it is used at the source VM
#2. used to decide whether to include certain subsection in the state
       being migrated.

Q1: How does the source VM know what version is there at the
       destination VM? or is it managed by the administrator?
Q2: Or maybe the is_needed hooks provides a way for the administrator
       to configure that at the source?
     

>  
>  > +
>  >  static const VMStateDescription vmstate_ged_state = {
>  >      .name = "acpi-ged-state",
>  >      .version_id = 1,
>  > @@ -379,6 +389,7 @@ static const VMStateDescription vmstate_acpi_ged
>  = {
>  >      },
>  >      .subsections = (const VMStateDescription * const []) {
>  >          &vmstate_memhp_state,
>  > +        &vmstate_cpuhp_state,
>  >          &vmstate_ghes_state,
>  >          NULL
>  >      }
>  

RE: [PATCH V1 3/4] hw/acpi: Reflect ACPI vCPU {present,enabled} states in ACPI _STA.{PRES,ENA} Bits

[Salil Mehta via]

Hi Zhao,

Sorry, for the late reply. I was away last week with only intermittent access
to the mails.

>  From: Zhao Liu <zhao1.liu@intel.com>
>  Sent: Friday, October 18, 2024 6:13 AM
>  To: Salil Mehta <salil.mehta@huawei.com>
>  
>  Hi Salil,
>  
>  On Mon, Oct 14, 2024 at 08:22:04PM +0100, Salil Mehta wrote:
>  > Date: Mon, 14 Oct 2024 20:22:04 +0100
>  > From: Salil Mehta <salil.mehta@huawei.com>
>  > Subject: [PATCH V1 3/4] hw/acpi: Reflect ACPI vCPU {present,enabled}
>  > states  in ACPI _STA.{PRES,ENA} Bits
>  > X-Mailer: git-send-email 2.34.1
>  >
>  > Reflect the ACPI CPU hotplug `is_{present, enabled}` states in the
>  > `_STA.PRES`
>  > (presence) and `_STA.ENA` (enabled) bits when the guest kernel
>  > evaluates the ACPI `_STA` method during initialization, as well as
>  > when vCPUs are hot-plugged or hot-unplugged. The presence of
>  unplugged
>  > vCPUs may need to be deliberately
>  > *simulated* at the ACPI level to maintain a *persistent* view of vCPUs
>  > for the guest kernel.
>  >
>  > Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
>  > ---
>  >  hw/acpi/cpu.c | 26 ++++++++++++++++++++++----
>  >  1 file changed, 22 insertions(+), 4 deletions(-)
>  >
>  
>  It seems this patch changes ACPI table layout and then breaks current ACPI
>  table qtest. I'm not sure how to do such modifications. Maybe you should
>  first disable the related checks, then modify the code, update the qtest, and
>  finally re-enable the checks for qtest. This can help to avoid any qtest failure
>  due to this patch?


Thanks for reporting. Let me get back to you on this.


>  
>  I think it should get Igor's advice on this. :)
>  
>  Attach the error I met:
>  
>  ▶   2/920 ERROR:../tests/qtest/bios-tables-test.c:553:test_acpi_asl:
>  assertion failed: (all_tables_match) ERROR
>  ▶   3/920 ERROR:../tests/qtest/bios-tables-test.c:553:test_acpi_asl:
>  assertion failed: (all_tables_match) ERROR
>    2/920 qemu:qtest+qtest-i386 / qtest-i386/bios-tables-test
>  ERROR            1.24s   killed by signal 6 SIGABRT
>  >>> G_TEST_DBUS_DAEMON=/media/liuzhao/data/qemu-
>  cook/tests/dbus-vmstate-
>  >>> daemon.sh
>  >>> ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1
>  >>> MESON_TEST_ITERATION=1
>  >>>
>  UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:pr
>  int
>  >>> _stacktrace=1 QTEST_QEMU_BINARY=./qemu-system-i386
>  >>> MALLOC_PERTURB_=142
>  >>>
>  MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:pri
>  nt_
>  >>> stacktrace=1
>  >>> QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-
>  daemon/qemu-storage-daemo
>  >>> n QTEST_QEMU_IMG=./qemu-img
>  >>> PYTHON=/media/liuzhao/data/qemu-cook/build/pyvenv/bin/python3
>  >>> /media/liuzhao/data/qemu-cook/build/tests/qtest/bios-tables-test
>  >>> --tap -k
>  ―――――――――――――――――――――――――――――――――
>  ―――――――――――――――――――――――――――――――――
>  ――――――― ✀
>  ―――――――――――――――――――――――――――――――――
>  ―――――――――――――――――――――――――――――――――
>  ―――――――
>  stderr:
>  acpi-test: Warning! DSDT binary file mismatch. Actual [aml:/tmp/aml-
>  VRT5V2], Expected [aml:tests/data/acpi/x86/pc/DSDT].
>  See source file tests/qtest/bios-tables-test.c for instructions on how to
>  update expected files.
>  acpi-test: Warning! DSDT mismatch. Actual [asl:/tmp/asl-TTT5V2.dsl,
>  aml:/tmp/aml-VRT5V2], Expected [asl:/tmp/asl-XXM5V2.dsl,
>  aml:tests/data/acpi/x86/pc/DSDT].
>  **
>  ERROR:../tests/qtest/bios-tables-test.c:553:test_acpi_asl: assertion failed:
>  (all_tables_match)
>  
>  (test program exited with status code -6)
>  ―――――――――――――――――――――――――――――――――
>  ―――――――――――――――――――――――――――――――――
>  ―――――――――――――――――――――――――――――――――
>  ―――――――――――――――――――――――――――――――――
>  ――――――――――――――――――
>  
>    3/920 qemu:qtest+qtest-x86_64 / qtest-x86_64/bios-tables-test
>  ERROR            1.25s   killed by signal 6 SIGABRT
>  >>> G_TEST_DBUS_DAEMON=/media/liuzhao/data/qemu-
>  cook/tests/dbus-vmstate-
>  >>> daemon.sh
>  >>> ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1
>  >>> MESON_TEST_ITERATION=1
>  >>>
>  UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:pr
>  int
>  >>> _stacktrace=1
>  >>>
>  MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:pri
>  nt_
>  >>> stacktrace=1
>  >>> QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-
>  daemon/qemu-storage-daemo
>  >>> n QTEST_QEMU_IMG=./qemu-img
>  >>> PYTHON=/media/liuzhao/data/qemu-cook/build/pyvenv/bin/python3
>  >>> MALLOC_PERTURB_=41 QTEST_QEMU_BINARY=./qemu-system-
>  x86_64
>  >>> /media/liuzhao/data/qemu-cook/build/tests/qtest/bios-tables-test
>  >>> --tap -k
>  ―――――――――――――――――――――――――――――――――
>  ―――――――――――――――――――――――――――――――――
>  ――――――― ✀
>  ―――――――――――――――――――――――――――――――――
>  ―――――――――――――――――――――――――――――――――
>  ―――――――
>  stderr:
>  acpi-test: Warning! DSDT binary file mismatch. Actual [aml:/tmp/aml-
>  D5K5V2], Expected [aml:tests/data/acpi/x86/pc/DSDT].
>  See source file tests/qtest/bios-tables-test.c for instructions on how to
>  update expected files.
>  acpi-test: Warning! DSDT mismatch. Actual [asl:/tmp/asl-G6K5V2.dsl,
>  aml:/tmp/aml-D5K5V2], Expected [asl:/tmp/asl-AQD5V2.dsl,
>  aml:tests/data/acpi/x86/pc/DSDT].
>  **
>  ERROR:../tests/qtest/bios-tables-test.c:553:test_acpi_asl: assertion failed:
>  (all_tables_match)
>  
>  (test program exited with status code -6)
>  
>  
>  Regards,
>  Zhao
>  
>  

RE: [PATCH V1 3/4] hw/acpi: Reflect ACPI vCPU {present,enabled} states in ACPI _STA.{PRES,ENA} Bits

[Salil Mehta via]

>  From: Igor Mammedov <imammedo@redhat.com>
>  Sent: Friday, October 18, 2024 3:19 PM
>  To: Zhao Liu <zhao1.liu@intel.com>
>  Cc: Salil Mehta <salil.mehta@huawei.com>; qemu-devel@nongnu.org;
>  qemu-arm@nongnu.org; mst@redhat.com; maz@kernel.org; jean-
>  philippe@linaro.org; Jonathan Cameron
>  <jonathan.cameron@huawei.com>; lpieralisi@kernel.org;
>  peter.maydell@linaro.org; richard.henderson@linaro.org;
>  andrew.jones@linux.dev; david@redhat.com; philmd@linaro.org;
>  eric.auger@redhat.com; will@kernel.org; ardb@kernel.org;
>  oliver.upton@linux.dev; pbonzini@redhat.com; gshan@redhat.com;
>  rafael@kernel.org; borntraeger@linux.ibm.com; alex.bennee@linaro.org;
>  npiggin@gmail.com; harshpb@linux.ibm.com; linux@armlinux.org.uk;
>  darren@os.amperecomputing.com; ilkka@os.amperecomputing.com;
>  vishnu@os.amperecomputing.com; karl.heubaum@oracle.com;
>  miguel.luis@oracle.com; salil.mehta@opnsrc.net; zhukeqian
>  <zhukeqian1@huawei.com>; wangxiongfeng (C)
>  <wangxiongfeng2@huawei.com>; wangyanan (Y)
>  <wangyanan55@huawei.com>; jiakernel2@gmail.com;
>  maobibo@loongson.cn; lixianglai@loongson.cn; shahuang@redhat.com;
>  Linuxarm <linuxarm@huawei.com>; gustavo.romero@linaro.org
>  Subject: Re: [PATCH V1 3/4] hw/acpi: Reflect ACPI vCPU {present,enabled}
>  states in ACPI _STA.{PRES,ENA} Bits
>  
>  On Fri, 18 Oct 2024 13:12:52 +0800
>  Zhao Liu <zhao1.liu@intel.com> wrote:
>  
>  > Hi Salil,
>  >
>  > On Mon, Oct 14, 2024 at 08:22:04PM +0100, Salil Mehta wrote:
>  > > Date: Mon, 14 Oct 2024 20:22:04 +0100
>  > > From: Salil Mehta <salil.mehta@huawei.com>
>  > > Subject: [PATCH V1 3/4] hw/acpi: Reflect ACPI vCPU {present,enabled}
>  > > states  in ACPI _STA.{PRES,ENA} Bits
>  > > X-Mailer: git-send-email 2.34.1
>  > >
>  > > Reflect the ACPI CPU hotplug `is_{present, enabled}` states in the
>  > > `_STA.PRES`
>  > > (presence) and `_STA.ENA` (enabled) bits when the guest kernel
>  > > evaluates the ACPI `_STA` method during initialization, as well as
>  > > when vCPUs are hot-plugged or hot-unplugged. The presence of
>  > > unplugged vCPUs may need to be deliberately
>  > > *simulated* at the ACPI level to maintain a *persistent* view of
>  > > vCPUs for the guest kernel.
>  > >
>  > > Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
>  > > ---
>  > >  hw/acpi/cpu.c | 26 ++++++++++++++++++++++----
>  > >  1 file changed, 22 insertions(+), 4 deletions(-)
>  > >
>  >
>  > It seems this patch changes ACPI table layout and then breaks current
>  > ACPI table qtest. I'm not sure how to do such modifications. Maybe you
>  > should first disable the related checks, then modify the code, update
>  > the qtest, and finally re-enable the checks for qtest. This can help
>  > to avoid any qtest failure due to this patch?
>  
>  see comment at the top of tests/qtest/bios-tables-test.c


Thanks for the pointers.


>  
>  >
>  > I think it should get Igor's advice on this. :)
>  >
>  > Attach the error I met:
>  >
>  > ▶   2/920 ERROR:../tests/qtest/bios-tables-test.c:553:test_acpi_asl:
>  assertion failed: (all_tables_match) ERROR
>  > ▶   3/920 ERROR:../tests/qtest/bios-tables-test.c:553:test_acpi_asl:
>  assertion failed: (all_tables_match) ERROR
>  >   2/920 qemu:qtest+qtest-i386 / qtest-i386/bios-tables-test
>  ERROR            1.24s   killed by signal 6 SIGABRT
>  > >>> G_TEST_DBUS_DAEMON=/media/liuzhao/data/qemu-
>  cook/tests/dbus-vmstat
>  > >>> e-daemon.sh
>  > >>>
>  ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1
>  > >>> MESON_TEST_ITERATION=1
>  > >>>
>  UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:pr
>  i
>  > >>> nt_stacktrace=1 QTEST_QEMU_BINARY=./qemu-system-i386
>  > >>> MALLOC_PERTURB_=142
>  > >>>
>  MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:pri
>  n
>  > >>> t_stacktrace=1
>  > >>> QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-
>  daemon/qemu-storage-dae
>  > >>> mon QTEST_QEMU_IMG=./qemu-img
>  > >>> PYTHON=/media/liuzhao/data/qemu-
>  cook/build/pyvenv/bin/python3
>  > >>> /media/liuzhao/data/qemu-cook/build/tests/qtest/bios-tables-test
>  > >>> --tap -k
>  >
>  ―――――――――――――――――――――――――――――――――
>  ―――――――――――――――――――――――――――――――――
>  ――――
>  > ――― ✀
>  >
>  ―――――――――――――――――――――――――――――――――
>  ―――――――――――――――――――――――――――――――――
>  ――――
>  > ―――
>  > stderr:
>  > acpi-test: Warning! DSDT binary file mismatch. Actual [aml:/tmp/aml-
>  VRT5V2], Expected [aml:tests/data/acpi/x86/pc/DSDT].
>  > See source file tests/qtest/bios-tables-test.c for instructions on how to
>  update expected files.
>  > acpi-test: Warning! DSDT mismatch. Actual [asl:/tmp/asl-TTT5V2.dsl,
>  aml:/tmp/aml-VRT5V2], Expected [asl:/tmp/asl-XXM5V2.dsl,
>  aml:tests/data/acpi/x86/pc/DSDT].
>  > **
>  > ERROR:../tests/qtest/bios-tables-test.c:553:test_acpi_asl: assertion
>  > failed: (all_tables_match)
>  >
>  > (test program exited with status code -6)
>  >
>  ―――――――――――――――――――――――――――――――――
>  ―――――――――――――――――――――――――――――――――
>  ――――
>  >
>  ―――――――――――――――――――――――――――――――――
>  ―――――――――――――――――――――――――――――――――
>  ――――
>  > ――――――――――
>  >
>  >   3/920 qemu:qtest+qtest-x86_64 / qtest-x86_64/bios-tables-test
>  ERROR            1.25s   killed by signal 6 SIGABRT
>  > >>> G_TEST_DBUS_DAEMON=/media/liuzhao/data/qemu-
>  cook/tests/dbus-vmstat
>  > >>> e-daemon.sh
>  > >>>
>  ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1
>  > >>> MESON_TEST_ITERATION=1
>  > >>>
>  UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:pr
>  i
>  > >>> nt_stacktrace=1
>  > >>>
>  MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:pri
>  n
>  > >>> t_stacktrace=1
>  > >>> QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-
>  daemon/qemu-storage-dae
>  > >>> mon QTEST_QEMU_IMG=./qemu-img
>  > >>> PYTHON=/media/liuzhao/data/qemu-
>  cook/build/pyvenv/bin/python3
>  > >>> MALLOC_PERTURB_=41 QTEST_QEMU_BINARY=./qemu-system-
>  x86_64
>  > >>> /media/liuzhao/data/qemu-cook/build/tests/qtest/bios-tables-test
>  > >>> --tap -k
>  >
>  ―――――――――――――――――――――――――――――――――
>  ―――――――――――――――――――――――――――――――――
>  ――――
>  > ――― ✀
>  >
>  ―――――――――――――――――――――――――――――――――
>  ―――――――――――――――――――――――――――――――――
>  ――――
>  > ―――
>  > stderr:
>  > acpi-test: Warning! DSDT binary file mismatch. Actual [aml:/tmp/aml-
>  D5K5V2], Expected [aml:tests/data/acpi/x86/pc/DSDT].
>  > See source file tests/qtest/bios-tables-test.c for instructions on how to
>  update expected files.
>  > acpi-test: Warning! DSDT mismatch. Actual [asl:/tmp/asl-G6K5V2.dsl,
>  aml:/tmp/aml-D5K5V2], Expected [asl:/tmp/asl-AQD5V2.dsl,
>  aml:tests/data/acpi/x86/pc/DSDT].
>  > **
>  > ERROR:../tests/qtest/bios-tables-test.c:553:test_acpi_asl: assertion
>  > failed: (all_tables_match)
>  >
>  > (test program exited with status code -6)
>  >
>  >
>  > Regards,
>  > Zhao
>  >
>  >
>  

RE: [PATCH V1 3/4] hw/acpi: Reflect ACPI vCPU {present,enabled} states in ACPI _STA.{PRES,ENA} Bits

[Salil Mehta via]

Hi Igor,

>  From: Igor Mammedov <imammedo@redhat.com>
>  Sent: Friday, October 18, 2024 3:25 PM
>  To: Salil Mehta <salil.mehta@huawei.com>
>  Cc: qemu-devel@nongnu.org; qemu-arm@nongnu.org; mst@redhat.com;
>  maz@kernel.org; jean-philippe@linaro.org; Jonathan Cameron
>  <jonathan.cameron@huawei.com>; lpieralisi@kernel.org;
>  peter.maydell@linaro.org; richard.henderson@linaro.org;
>  andrew.jones@linux.dev; david@redhat.com; philmd@linaro.org;
>  eric.auger@redhat.com; will@kernel.org; ardb@kernel.org;
>  oliver.upton@linux.dev; pbonzini@redhat.com; gshan@redhat.com;
>  rafael@kernel.org; borntraeger@linux.ibm.com; alex.bennee@linaro.org;
>  npiggin@gmail.com; harshpb@linux.ibm.com; linux@armlinux.org.uk;
>  darren@os.amperecomputing.com; ilkka@os.amperecomputing.com;
>  vishnu@os.amperecomputing.com; karl.heubaum@oracle.com;
>  miguel.luis@oracle.com; salil.mehta@opnsrc.net; zhukeqian
>  <zhukeqian1@huawei.com>; wangxiongfeng (C)
>  <wangxiongfeng2@huawei.com>; wangyanan (Y)
>  <wangyanan55@huawei.com>; jiakernel2@gmail.com;
>  maobibo@loongson.cn; lixianglai@loongson.cn; shahuang@redhat.com;
>  zhao1.liu@intel.com; Linuxarm <linuxarm@huawei.com>;
>  gustavo.romero@linaro.org
>  Subject: Re: [PATCH V1 3/4] hw/acpi: Reflect ACPI vCPU {present,enabled}
>  states in ACPI _STA.{PRES,ENA} Bits
>  
>  On Mon, 14 Oct 2024 20:22:04 +0100
>  Salil Mehta <salil.mehta@huawei.com> wrote:
>  
>  > Reflect the ACPI CPU hotplug `is_{present, enabled}` states in the
>  > `_STA.PRES`
>  > (presence) and `_STA.ENA` (enabled) bits when the guest kernel
>  > evaluates the ACPI `_STA` method during initialization, as well as
>  > when vCPUs are hot-plugged or hot-unplugged. The presence of
>  unplugged
>  > vCPUs may need to be deliberately
>  > *simulated* at the ACPI level to maintain a *persistent* view of vCPUs
>  > for the guest kernel.
>  
>  given questionable future of is_present/is_enabled fields, it probably
>  premature to review this part.


Sure, let us discuss the pain points. 


>  The only thing, I have to say here is repeating spec/doc update patch
>  describing how it should work should come 1st, so that we could compare
>  this impl. with it.


From the user perspective, there should not be any change in how he
plug/unplugs the ARM vCPUs. This is a requirement, as we have to
assists folks who wants to migrate from x86 world to ARM world
seamlessly (without affecting performance parameters) and this
feature is one small part of that bigger effort.


>  > Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
>  > ---
>  >  hw/acpi/cpu.c | 26 ++++++++++++++++++++++----
>  >  1 file changed, 22 insertions(+), 4 deletions(-)
>  >
>  > diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c index
>  > 700aa855e9..23ea2b9c70 100644
>  > --- a/hw/acpi/cpu.c
>  > +++ b/hw/acpi/cpu.c
>  > @@ -63,10 +63,11 @@ static uint64_t cpu_hotplug_rd(void *opaque,
>  hwaddr addr, unsigned size)
>  >      cdev = &cpu_st->devs[cpu_st->selector];
>  >      switch (addr) {
>  >      case ACPI_CPU_FLAGS_OFFSET_RW: /* pack and return is_* fields */
>  > -        val |= cdev->cpu ? 1 : 0;
>  > +        val |= cdev->is_enabled ? 1 : 0;
>  >          val |= cdev->is_inserting ? 2 : 0;
>  >          val |= cdev->is_removing  ? 4 : 0;
>  >          val |= cdev->fw_remove  ? 16 : 0;
>  > +        val |= cdev->is_present ? 32 : 0;
>  >          trace_cpuhp_acpi_read_flags(cpu_st->selector, val);
>  >          break;
>  >      case ACPI_CPU_CMD_DATA_OFFSET_RW:
>  > @@ -376,6 +377,7 @@ const VMStateDescription vmstate_cpu_hotplug =
>  {
>  > #define CPU_REMOVE_EVENT  "CRMV"
>  >  #define CPU_EJECT_EVENT   "CEJ0"
>  >  #define CPU_FW_EJECT_EVENT "CEJF"
>  > +#define CPU_PRESENT       "CPRS"
>  >
>  >  void build_cpus_aml(Aml *table, MachineState *machine,
>  CPUHotplugFeatures opts,
>  >                      build_madt_cpu_fn build_madt_cpu, hwaddr
>  > base_addr, @@ -436,7 +438,9 @@ void build_cpus_aml(Aml *table,
>  MachineState *machine, CPUHotplugFeatures opts,
>  >          aml_append(field, aml_named_field(CPU_EJECT_EVENT, 1));
>  >          /* tell firmware to do device eject, write only */
>  >          aml_append(field, aml_named_field(CPU_FW_EJECT_EVENT, 1));
>  > -        aml_append(field, aml_reserved_field(3));
>  > +        /* 1 if present, read only */
>  > +        aml_append(field, aml_named_field(CPU_PRESENT, 1));
>  > +        aml_append(field, aml_reserved_field(2));
>  >          aml_append(field, aml_named_field(CPU_COMMAND, 8));
>  >          aml_append(cpu_ctrl_dev, field);
>  >
>  > @@ -466,6 +470,7 @@ void build_cpus_aml(Aml *table, MachineState
>  *machine, CPUHotplugFeatures opts,
>  >          Aml *ctrl_lock = aml_name("%s.%s", cphp_res_path, CPU_LOCK);
>  >          Aml *cpu_selector = aml_name("%s.%s", cphp_res_path,
>  CPU_SELECTOR);
>  >          Aml *is_enabled = aml_name("%s.%s", cphp_res_path,
>  > CPU_ENABLED);
>  > +        Aml *is_present = aml_name("%s.%s", cphp_res_path,
>  > + CPU_PRESENT);
>  >          Aml *cpu_cmd = aml_name("%s.%s", cphp_res_path,
>  CPU_COMMAND);
>  >          Aml *cpu_data = aml_name("%s.%s", cphp_res_path, CPU_DATA);
>  >          Aml *ins_evt = aml_name("%s.%s", cphp_res_path,
>  > CPU_INSERT_EVENT); @@ -494,13 +499,26 @@ void build_cpus_aml(Aml
>  *table, MachineState *machine, CPUHotplugFeatures opts,
>  >          {
>  >              Aml *idx = aml_arg(0);
>  >              Aml *sta = aml_local(0);
>  > +            Aml *ifctx2;
>  > +            Aml *else_ctx;
>  >
>  >              aml_append(method, aml_acquire(ctrl_lock, 0xFFFF));
>  >              aml_append(method, aml_store(idx, cpu_selector));
>  >              aml_append(method, aml_store(zero, sta));
>  > -            ifctx = aml_if(aml_equal(is_enabled, one));
>  > +            ifctx = aml_if(aml_equal(is_present, one));
>  >              {
>  > -                aml_append(ifctx, aml_store(aml_int(0xF), sta));
>  > +                ifctx2 = aml_if(aml_equal(is_enabled, one));
>  > +                {
>  > +                    /* cpu is present and enabled */
>  > +                    aml_append(ifctx2, aml_store(aml_int(0xF), sta));
>  > +                }
>  > +                aml_append(ifctx, ifctx2);
>  > +                else_ctx = aml_else();
>  > +                {
>  > +                    /* cpu is present but disabled */
>  > +                    aml_append(else_ctx, aml_store(aml_int(0xD), sta));
>  > +                }
>  > +                aml_append(ifctx, else_ctx);
>  >              }
>  >              aml_append(method, ifctx);
>  >              aml_append(method, aml_release(ctrl_lock));
>  

RE: [PATCH V1 3/4] hw/acpi: Reflect ACPI vCPU {present,enabled} states in ACPI _STA.{PRES,ENA} Bits

[Salil Mehta via]

Hi Gustavo,

>  From: Gustavo Romero <gustavo.romero@linaro.org>
>  Sent: Monday, October 21, 2024 3:10 AM
>  To: Salil Mehta <salil.mehta@huawei.com>; qemu-devel@nongnu.org;
>  qemu-arm@nongnu.org; mst@redhat.com
>  
>  Hi Salil,
>  
>  On 10/14/24 16:22, Salil Mehta wrote:
>  > Reflect the ACPI CPU hotplug `is_{present, enabled}` states in the
>  > `_STA.PRES`
>  > (presence) and `_STA.ENA` (enabled) bits when the guest kernel
>  > evaluates the ACPI `_STA` method during initialization, as well as
>  > when vCPUs are hot-plugged or hot-unplugged. The presence of
>  unplugged
>  > vCPUs may need to be deliberately
>  > *simulated* at the ACPI level to maintain a *persistent* view of vCPUs
>  > for the guest kernel.
>  >
>  > Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
>  > ---
>  >   hw/acpi/cpu.c | 26 ++++++++++++++++++++++----
>  >   1 file changed, 22 insertions(+), 4 deletions(-)
>  >
>  > diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c index
>  > 700aa855e9..23ea2b9c70 100644
>  > --- a/hw/acpi/cpu.c
>  > +++ b/hw/acpi/cpu.c
>  > @@ -63,10 +63,11 @@ static uint64_t cpu_hotplug_rd(void *opaque,
>  hwaddr addr, unsigned size)
>  >       cdev = &cpu_st->devs[cpu_st->selector];
>  >       switch (addr) {
>  >       case ACPI_CPU_FLAGS_OFFSET_RW: /* pack and return is_* fields */
>  > -        val |= cdev->cpu ? 1 : 0;
>  > +        val |= cdev->is_enabled ? 1 : 0;
>  >           val |= cdev->is_inserting ? 2 : 0;
>  >           val |= cdev->is_removing  ? 4 : 0;
>  >           val |= cdev->fw_remove  ? 16 : 0;
>  > +        val |= cdev->is_present ? 32 : 0;
>  >           trace_cpuhp_acpi_read_flags(cpu_st->selector, val);
>  >           break;
>  >       case ACPI_CPU_CMD_DATA_OFFSET_RW:
>  > @@ -376,6 +377,7 @@ const VMStateDescription vmstate_cpu_hotplug =
>  {
>  >   #define CPU_REMOVE_EVENT  "CRMV"
>  >   #define CPU_EJECT_EVENT   "CEJ0"
>  >   #define CPU_FW_EJECT_EVENT "CEJF"
>  > +#define CPU_PRESENT       "CPRS"
>  >
>  >   void build_cpus_aml(Aml *table, MachineState *machine,
>  CPUHotplugFeatures opts,
>  >                       build_madt_cpu_fn build_madt_cpu, hwaddr
>  > base_addr, @@ -436,7 +438,9 @@ void build_cpus_aml(Aml *table,
>  MachineState *machine, CPUHotplugFeatures opts,
>  >           aml_append(field, aml_named_field(CPU_EJECT_EVENT, 1));
>  >           /* tell firmware to do device eject, write only */
>  >           aml_append(field, aml_named_field(CPU_FW_EJECT_EVENT, 1));
>  > -        aml_append(field, aml_reserved_field(3));
>  > +        /* 1 if present, read only */
>  > +        aml_append(field, aml_named_field(CPU_PRESENT, 1));
>  > +        aml_append(field, aml_reserved_field(2));
>  >           aml_append(field, aml_named_field(CPU_COMMAND, 8));
>  >           aml_append(cpu_ctrl_dev, field);
>  >
>  > @@ -466,6 +470,7 @@ void build_cpus_aml(Aml *table, MachineState
>  *machine, CPUHotplugFeatures opts,
>  >           Aml *ctrl_lock = aml_name("%s.%s", cphp_res_path, CPU_LOCK);
>  >           Aml *cpu_selector = aml_name("%s.%s", cphp_res_path,
>  CPU_SELECTOR);
>  >           Aml *is_enabled = aml_name("%s.%s", cphp_res_path,
>  > CPU_ENABLED);
>  > +        Aml *is_present = aml_name("%s.%s", cphp_res_path,
>  > + CPU_PRESENT);
>  >           Aml *cpu_cmd = aml_name("%s.%s", cphp_res_path,
>  CPU_COMMAND);
>  >           Aml *cpu_data = aml_name("%s.%s", cphp_res_path, CPU_DATA);
>  >           Aml *ins_evt = aml_name("%s.%s", cphp_res_path,
>  > CPU_INSERT_EVENT); @@ -494,13 +499,26 @@ void build_cpus_aml(Aml
>  *table, MachineState *machine, CPUHotplugFeatures opts,
>  >           {
>  >               Aml *idx = aml_arg(0);
>  >               Aml *sta = aml_local(0);
>  > +            Aml *ifctx2;
>  > +            Aml *else_ctx;
>  >
>  >               aml_append(method, aml_acquire(ctrl_lock, 0xFFFF));
>  >               aml_append(method, aml_store(idx, cpu_selector));
>  >               aml_append(method, aml_store(zero, sta));
>  > -            ifctx = aml_if(aml_equal(is_enabled, one));
>  > +            ifctx = aml_if(aml_equal(is_present, one));
>  >               {
>  > -                aml_append(ifctx, aml_store(aml_int(0xF), sta));
>  > +                ifctx2 = aml_if(aml_equal(is_enabled, one));
>  > +                {
>  > +                    /* cpu is present and enabled */
>  > +                    aml_append(ifctx2, aml_store(aml_int(0xF), sta));
>  > +                }
>  > +                aml_append(ifctx, ifctx2);
>  > +                else_ctx = aml_else();
>  > +                {
>  > +                    /* cpu is present but disabled */
>  > +                    aml_append(else_ctx, aml_store(aml_int(0xD),
>  > + sta));
>  
>  Here, the return value for _STA method is set to reflect the state of
>  CPU_PRESENT and CPU_ENABLED fields. I can't see these two fields being
>  mapped to AcpiCpuStatus.{is_present,is_enabled}. They look to be mapped
>  to the MMIO region (base_addr), which doesn't mapped to AcpiCpuStatus
>  afaics. So where CPU_PRESENT and CPU_ENABLED are set and where
>  exactly they reside?


You don’t set _STA field from guest for CPUs. We only fetch values being
reflected by the VMM. In general, ACPI _STA method is for fetching status
of a device. Please check below:

https://uefi.org/specs/ACPI/6.5/06_Device_Configuration.html?highlight=_sta#sta-device-status


We are banking on below status bits:

Bit [0] - Set if the device is present.
Bit [1] - Set if the device is enabled and decoding its resources.

Hence, AcpiCpuStatus::(is_present, is_enabled} are set in the VMM
and their status is shared to the guest kernel when the ACPI _STA
method is evaluated by the OSPM.


Patch 3/4: https://lore.kernel.org/qemu-devel/20241014192205.253479-4-salil.mehta@huawei.com/
#HUNK

diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
index 700aa855e9..23ea2b9c70 100644
--- a/hw/acpi/cpu.c
+++ b/hw/acpi/cpu.c
@@ -63,10 +63,11 @@ static uint64_t cpu_hotplug_rd(void *opaque, hwaddr addr, unsigned size)
     cdev = &cpu_st->devs[cpu_st->selector];---------> check this code excerpt
     switch (addr) {
     case ACPI_CPU_FLAGS_OFFSET_RW: /* pack and return is_* fields */
-        val |= cdev->cpu ? 1 : 0;
+        val |= cdev->is_enabled ? 1 : 0;---------------> this change
         val |= cdev->is_inserting ? 2 : 0;
         val |= cdev->is_removing  ? 4 : 0;
         val |= cdev->fw_remove  ? 16 : 0;
+        val |= cdev->is_present ? 32 : 0;-------------> and this change
         trace_cpuhp_acpi_read_flags(cpu_st->selector, val);
         break;
     case ACPI_CPU_CMD_DATA_OFFSET_RW:


Thanks

>  
>  
>  Cheers,
>  Gustavo
>  
>  > +                }
>  > +                aml_append(ifctx, else_ctx);
>  >               }
>  >               aml_append(method, ifctx);
>  >               aml_append(method, aml_release(ctrl_lock));

RE: [PATCH V1 0/4] Arch agnostic ACPI changes to support vCPU Hotplug (on Archs like ARM)

[Salil Mehta via]

Hi Igor,

Thanks for taking time to review the series. Please find my replies inline.

>  From: qemu-devel-bounces+salil.mehta=huawei.com@nongnu.org <qemu-
>  devel-bounces+salil.mehta=huawei.com@nongnu.org> On Behalf Of Igor
>  Mammedov
>  Sent: Friday, October 18, 2024 3:46 PM
>  To: Salil Mehta <salil.mehta@huawei.com>
>  
>  On Mon, 14 Oct 2024 20:22:01 +0100
>  Salil Mehta <salil.mehta@huawei.com> wrote:
>  
>  > Certain CPU architecture specifications [1][2][3] prohibit changes to
>  > the CPUs
>  > *presence* after the kernel has booted. This is because many system
>  > initializations depend on the exact CPU count at boot time and do not
>  > expect it to change afterward. For example, components like interrupt
>  > controllers that are closely coupled with CPUs, or various per-CPU
>  > features, may not support configuration changes once the kernel has
>  been initialized.
>  >
>  > This requirement poses a challenge for virtualization features like
>  > vCPU hotplug. To address this, changes to the ACPI AML are necessary
>  > to update the `_STA.PRES` (presence) and `_STA.ENA` (enabled) bits
>  > accordingly during guest initialization, as well as when vCPUs are
>  > hot-plugged or hot-unplugged. The presence of unplugged vCPUs may
>  need
>  > to be deliberately *simulated* at the ACPI level to maintain a *persistent*
>  view of vCPUs for the guest kernel.
>  
>  the series is peppered with *simulated* idea, which after looking at code I
>  read as 'fake'.


Got it! If "simulate" doesn't convey the meaning well, we can definitely switch
back to "fake." or something else. No issues at all.


> While it's obvious to author why things need to be faked at
>  this time, it will be forgotten later on. And cause a lot swearing from
>  whoever will have to deal with this code down the road.


Let's improve the design then. However, the boot time cannot be negotiated.
We cannot have 500 vCPUs spawned at boot time when we only need 10 vCPUs
to run. That’s a non-starter!

(please have a look at KVMForum 2023 slides for the measurements we did for
 128 vCPUs )

This also creates overhead during migration and will prolong the migration time.
The alternative you suggested was the first approach we experimented way back
in the year 2020. We then gradually moved to removing the threads but keeping
the QOM vCPU objects, and later even discarded the vCPU objects corresponding
to the `disabled` possible vCPUs. This change was made precisely at your request
last year in June.


>  
>  Salil, I'm sorry that review comes out as mostly negative but for me having
>  to repeat 'simulated' some many times, hints that the there is something
>  wrong with design and that we should re-evaluate the approach.


A lot of effort and time has gone into this project, involving many companies.

Therefore, let’s proceed, but it’s not fair to the many stakeholders who have
spent so much of their time over the past four years to simply hear that
“there is something wrong with the design.” It would be very helpful if you
could provide more details about your concerns so that we can work on
improving the `existing` design and help you understand our perspective.
Discussion around specifics are required, please!


>  
>  ps:
>  see comments on 1/4 for suggestions


Thanks for that. I've already replied with my perspective in context to that.
Please have a look.

Many thanks!

Best regards
Salil.

Re: [PATCH V1 1/4] hw/acpi: Initialize ACPI Hotplug CPU Status with Support for vCPU `Persistence`

[Igor Mammedov]

On Mon, 21 Oct 2024 22:50:40 +0100
Salil Mehta <salil.mehta@opnsrc.net> wrote:

> Hi Igor,
> 
> Thanks for taking time to review and sorry for not being prompt. I was in
> transit
> due to some difficult personal situation.
> 
> On Fri, Oct 18, 2024 at 3:11 PM Igor Mammedov <imammedo@redhat.com> wrote:
> 
> > On Mon, 14 Oct 2024 20:22:02 +0100
> > Salil Mehta <salil.mehta@huawei.com> wrote:
> >  
> > > Certain CPU architecture specifications [1][2][3] prohibit changes to CPU  
> >                                           ^^^^^^^^^
> > these do not point to specs but never mind
> >  
> > > presence after the kernel has booted. This limitation exists because  
> > many system  
> > > initializations rely on the exact CPU count at boot time and do not  
> > expect it to  
> > > change later. For example, components like interrupt controllers, which  
> > are  
> > > closely tied to CPUs, or various per-CPU features, may not support  
> > configuration  
> > > changes once the kernel has been initialized. This presents a challenge  
> > for  
> > > virtualization features such as vCPU hotplug.  
> >
> > well, x86 (incl cpu,interrupt ctrls) also doesn't have architectural
> > hotplug.
> > It's just OEM managed to implement it regardless and then bothered to make
> > OS changes to work with that.
> > It's just ARM community doesn't want to go there at this point of time
> > but using above reasoning as justification for this series doesn't look
> > good to me.
> >  
> 
> 
> There is a difference though. vCPU presence cannot be changed after the
> system has
> initialized in the ARM world due to the Architectural constraint. 
> I think
> in the x86 world
> it is allowed? 

As far as I know x86 doesn't have arch defined hotplug, but vendors
managed to implement it somehow and then made sure that OS running on top
could stomach it.

> > So what ARM would like to support is not CPU hotplug but rather a fixed
> > system with standby CPUs (that can be powered on/off on demand).
> > With ACPI spec amended to support that (MADT present/enabled changes), it's
> > good enough reason to add 'enabled' handling to acpi cpu-hotplug code
> > instead
> > of inventing alternative one that would do almost the same.
> >  
> 
> 
> Not sure what you mean here but this is what is being done.
it was ack for 'enabled' flag in cphp acpi abi.
The rest was digression wrt 'present' hack in ACPI code.
 
 
> > But lets get rid of (can't/may not) language above and use standby CPUs
> > reasoning
> > to avoid any confusion.
> >  
> 
> 
> I've to disagree here. Standy-by means you will have to keep all the vCPUs
> states
> realized and the objects will always exist. This is a problem for larger
> systems
> with vCPU hotplug support as it drastically affects the boot times.
>

see comment below about boot times.

> Please
> check
> the KVM Forum 2023 slides for objective values.

For sure we can conjure unicorns especially virtual ones,
it doesn't mean that we should do it though.

> It's been a long time since this series was actually conceived and at that
> time we wanted
> to use it for kata containers but later with the gradual adoption of
> various microvms
> and the cloud hypervisor requirements have bit changed. Boot time still
> remains one
> of the major requirements. Not bounding boot time while using this feature
> will
> seriously affect performance if the number of vCPUs increases

again wrt boot time, see comment below.


> > PS:
> > I'm taking about hw hotplug (at QEMU level) and not kernel hotplug
> > (where it's at logical cpu level).
> >  
> 
> sure
> 
> 
> >  
> > > To address this issue, introduce an `is_enabled` state in the  
> > `AcpiCpuStatus`,  
> > > which reflects whether a vCPU has been hot-plugged or hot-unplugged in  
> > QEMU,  
> > > marking it as (un)available in the Guest Kernel.  
> > good so far
> >  
> > > The `is_present` state should
> > > be set based on the `acpi_persistent` flag. In cases where unplugged  
> > vCPUs need  
> > > to be deliberately simulated in the ACPI to maintain a persistent view  
> > of vCPUs,  
> > > this flag ensures the guest kernel continues to see those vCPUs.  
> >
> > that's where I have to disagree, vCPU is present when a corresponding QOM
> > object
> > exists. Faking it's presence will only confuse at the end.
> >  
> 
> 
> Not faking care of it will mean realization  of the unnecessary vCPU
> threads during
> the VM init time, which in turn will increase the boot time. Trade-off
> between more
> clean design and performance.

I very much prefer clean design, which should result in
less code/less bugs/easier maintenance and less regressions
when someone fixes intentional hacks, with a good reasoning that
hardware doesn't work that way.

wrt boot time, see below.

> > I get that you want to reuse device_add/del interface, but that leads to
> > pulling the leg here and there to make thing fit. That in short therm
> > (while someone remembers all tricks) might work for some, but long therm
> > it's not sustainable).
> >  
> 
> 
> I do not understand why?

It's a complicated set of hacks all over place to make something that
do not exists in the real world work somehow.

While at the same time there is alternative approach that mimics ARM hardware
behavior and is supported by vendor (standby cpus).
That also will be much more simple way, while providing similar to hotplug
functionality. 


> > Maybe instead of pushing device_add/del, we should rather implement
> > standby CPU model here, as ARM folks expect it to be.
> > i.e. instead of device_add/del add 'enabled' property to ARM vCPU,
> > and let management to turn on/off vCPUs on demand.
> > (and very likely this model could be reused by other sock based platforms)
> > For end-user it would be the same as device_add/del (i.e. vCPU becomes
> > usable/unsable)
>
> 
> One of the main goals is to facilitate seamless migration from the x86
> world to
> ARM world. Hence, preservation of the x86 interface is important. It is a
> requirement.
> Just imagine if Apple had to change end user interface experience after
> migrating iOS
> from x86 to ARM architecture. ?

About that I wouldn't worry much as it's secondary.
Management can adapt to call 'qom set' instead of calling device_add/del.
It definitely shouldn't be the thing that turns design decisions
into the bad model.

> > I'd bet it would simplify your ARM CPU hotplug series a lot,
> > since you won't have to fake vCPU object lifetime and do
> > non trivial tricks to make it all work.
> > Which it turn will make ARM hotplug series much more approachable
> > for reviewers /and whomever comes later across that code/.
> >  
> 
> Tricks are just for ACPI and GIC and nothing else. Rest is a
> boilerplate code of the
> vCPU hotplug framework which x86 is also using.
 
looking at your v5 ARM cphp series, it does a bunch refactoring
to handle CPU absence whre there should be one by design.

While in standby cpus, that won't be needed.
(one would need a code to enable/disable CPUs, plus making ACPI
deliver those events, but that's pretty much all. i.e. do what real
hw can do)

> > Regardless of said, we would still need changes to ACPI cphp code,
> > see my comments inline.
> >  
> 
> sure.
> 
> 
> >
> >  
> > > Additionally, introduce an `acpi_persistent` property that can be used to
> > > initialize the ACPI vCPU presence state accordingly. Architectures  
> > requiring  
> > > ACPI to expose a persistent view of vCPUs can override its default  
> > value. Refer  
> > > to the patch-set implelenting vCPU hotplug support for ARM for more  
> > details on  
> > > its usage.
> > >
> > > References:
> > > [1] KVMForum 2023 Presentation: Challenges Revisited in Supporting Virt  
> > CPU Hotplug on  
> > >     architectures that don’t Support CPU Hotplug (like ARM64)
> > >     a. Kernel Link:  
> > https://kvm-forum.qemu.org/2023/KVM-forum-cpu-hotplug_7OJ1YyJ.pdf  
> > >     b. Qemu Link:  
> > https://kvm-forum.qemu.org/2023/Challenges_Revisited_in_Supporting_Virt_CPU_Hotplug_-__ii0iNb3.pdf  
> > > [2] KVMForum 2020 Presentation: Challenges in Supporting Virtual CPU  
> > Hotplug on  
> > >     SoC Based Systems (like ARM64)
> > >     Link: https://kvmforum2020.sched.com/event/eE4m
> > > [3] Check comment 5 in the bugzilla entry
> > >     Link: https://bugzilla.tianocore.org/show_bug.cgi?id=4481#c5
> > >
> > > Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> > > ---
> > >  cpu-target.c          |  1 +
> > >  hw/acpi/cpu.c         | 35 ++++++++++++++++++++++++++++++++++-
> > >  include/hw/acpi/cpu.h | 21 +++++++++++++++++++++
> > >  include/hw/core/cpu.h | 21 +++++++++++++++++++++
> > >  4 files changed, 77 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/cpu-target.c b/cpu-target.c
> > > index 499facf774..c8a29ab495 100644
> > > --- a/cpu-target.c
> > > +++ b/cpu-target.c
> > > @@ -200,6 +200,7 @@ static Property cpu_common_props[] = {
> > >       */
> > >      DEFINE_PROP_LINK("memory", CPUState, memory, TYPE_MEMORY_REGION,
> > >                       MemoryRegion *),
> > > +    DEFINE_PROP_BOOL("acpi-persistent", CPUState, acpi_persistent,  
> > false),
> >
> > I agree with Gavin, it's not CPU property/business, but a platform one.
> >
> > Pass it as argument to cpu_hotplug_hw_init(),
> > and maybe rename to always_present.
> > Then make sure that it's configurable in GED (which calls the function),
> > and just turn it on for arm/virt machine.
> > Other platforms might want to use x86 approach with GED and have
> > vCPU actually disappearing. /loongson and maybe risc-v/
> >  
> 
> can we move it to Machine level or fetch it through firmware?

following would do:
  1. add to ged  a new property to opt-in into this
  2. set the new property from arm's board_init where GED is created 
  3. when GED calls cpu_hotplug_hw_init(), pass property value as an argument

> > >  #endif
> > >      DEFINE_PROP_END_OF_LIST(),
> > >  };
> > > diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
> > > index 5cb60ca8bc..083c4010c2 100644
> > > --- a/hw/acpi/cpu.c
> > > +++ b/hw/acpi/cpu.c
> > > @@ -225,7 +225,40 @@ void cpu_hotplug_hw_init(MemoryRegion *as, Object  
> > *owner,  
> > >      state->dev_count = id_list->len;
> > >      state->devs = g_new0(typeof(*state->devs), state->dev_count);
> > >      for (i = 0; i < id_list->len; i++) {
> > > -        state->devs[i].cpu =  CPU(id_list->cpus[i].cpu);
> > > +        struct CPUState *cpu = CPU(id_list->cpus[i].cpu);
> > > +        /*
> > > +         * In most architectures, CPUs that are marked as ACPI  
> > 'present' are  
> > > +         * also ACPI 'enabled' by default. These states remain  
> > consistent at  
> > > +         * both the QOM and ACPI levels.
> > > +         */
> > > +        if (cpu) {
> > > +            state->devs[i].is_enabled = true;
> > > +            state->devs[i].is_present = true;
> > > +            state->devs[i].cpu = cpu;
> > > +        } else {
> > > +            state->devs[i].is_enabled = false;
> > > +            /*
> > > +             * In some architectures, even 'unplugged' or 'disabled'  
> > QOM CPUs  
> > > +             * may be exposed as ACPI 'present.' This approach provides  
> > a  
> > > +             * persistent view of the vCPUs to the guest kernel. This  
> > could be  
> > > +             * due to an architectural constraint that requires every  
> > per-CPU  
> > > +             * component to be present at boot time, meaning the exact  
> > count of  
> > > +             * vCPUs must be known and cannot be altered after the  
> > kernel has  
> > > +             * booted. As a result, the vCPU states at the QOM and ACPI  
> > levels  
> > > +             * might become inconsistent. However, in such cases, the  
> > presence  
> > > +             * of vCPUs has been deliberately simulated at the ACPI  
> > level.  
> > > +             */  
> >
> > if cpus are not 'simulated', you will not need comments explaining that all
> > over place and whole hunk would likely go away.
> >  
> 
> I can reduce the content if you wish.

you have those comments for a reason since the code does unexpected
'simulate' thing. Removing that would mask intentional behavior
for a reader later down the road.

  
> > > +            if (acpi_persistent_cpu(first_cpu)) {
> > > +                state->devs[i].is_present = true;
> > > +                /*
> > > +                 * `CPUHotplugState::AcpiCpuStatus::cpu` becomes  
> > insignificant  
> > > +                 * in this case
> > > +                 */
> > > +            } else {
> > > +                state->devs[i].is_present = false;
> > > +                state->devs[i].cpu = cpu;
> > > +            }
> > > +        }
> > >          state->devs[i].arch_id = id_list->cpus[i].arch_id;
> > >      }
> > >      memory_region_init_io(&state->ctrl_reg, owner, &cpu_hotplug_ops,  
> > state,  
> > > diff --git a/include/hw/acpi/cpu.h b/include/hw/acpi/cpu.h
> > > index 32654dc274..bd3f9973c9 100644
> > > --- a/include/hw/acpi/cpu.h
> > > +++ b/include/hw/acpi/cpu.h
> > > @@ -26,6 +26,8 @@ typedef struct AcpiCpuStatus {
> > >      uint64_t arch_id;
> > >      bool is_inserting;
> > >      bool is_removing;
> > > +    bool is_present;  
> > with always_present, it might be better to move field to CPUHotplugState
> > as it's not per vCPU anymore, and in standby case state->devs[i].cpu
> > should work as implicit present flag. (see below wrt doc first comment)
> >  
> 
> I'm still not convinced of the stand-by approach because of the performance
> impact
> it will have upon increasing the number of possible vCPUs and its worst
> case is
> unbounded. That's a major problem.

# of vCPUs is always bound by host capabilities and by modeled hardware.

From guest point of view both approaches should save time as
guest won't try to online non-present or disabled CPUs
(with large machines guest boot time usually dwarfs whatever
QEMU does before the guest, +UEFI adds to insult event more).
And fast booting large VM (with non present CPUs) initially,
is just postponing problem the the later time when hotplugging
CPUs causes worse timing (since guest effectively does stop_machine)
for 'online'-ing to happen.

Regardless, standby CPUs is what ARM folks have chosen to support.
In this case, I'd pick arch preferred way anytime vs some boot time
improvements.

If arm_cpu_realizefn() is still expensive for your case and better
init times are needed, fix it instead of working around it by faking
hotplug on QEMU side and moving issue to the later time.
That way will benefit not only hotplug case, but also huge VMs case.
(good example is x86, where it scales well without any hotplug)

PS:
Out of curiosity, I've just fed qemu to perf on Ampere host.
There are a number of low hanging fruits that would reduce consumed
CPU cycles, for those who wishes to improve performance.

Some would lead to overall improvements, other could be
done when vCPU is disabled.

ex: with -enable-kvm -smp 100
   - 26.18% arm_cpu_realizefn
      + 8.86% cpu_reset <- likely is not necessary for disabled vCPUs, as one has to call it again when enabling vCPU
      + 4.53% register_cp_regs_for_features
      + 4.32% arm_cpu_register_gdb_regs_for_features  <- do we need it when gdbstub is not enabled?
      + 4.09% init_cpreg_list  <- naive refactoring makes it a fraction of percent
      + 3.40% qemu_init_vcpu
        0.59% trace_init_vcpu

with above hacks it goes down to 11% (12% for x86_cpu_realizefn).
However wall-clock wise compared to x86, the arm/virt doesn't scale well.
So there is something else tgetting in the way (i.e. something stalls vCPU creation on ARM?).

And well, I' might be comparing apples to oranges here (old E5-2630v3 vs some 32core 3.3Ghz Ampere cpu).

PS2:
create-gic() is another candidate for improvement, it spends a lot
of time on setting GPIO properties.

> > > +    bool is_enabled;  
> > I'd move introduction of this field into a separate patch.
> >
> > BTW: new ABI/fields accessible by guest should be described in
> > docs/specs/acpi_cpu_hotplug.rst.
> > It would be better to have the spec as patch 1st, that we all agree on
> > and then follow with implementation.
> >  
> 
> We can do that.
> 
> 
> > And also include there an expected workflow for standby case.  
> 
> 
> 
> We need more discussion on this as our requirements for which we floated
                                      ^^^^^^^^^^^^^^
> this
> feature are not being met using stand-by cases.

A list of them would be a good starting point for discussion.
Perhaps requirements should be made more realistic and be re-evaluated.


> Thanks!
> 
> Best regards
> Salil.
> 
> >
> >  
> > >      bool fw_remove;
> > >      uint32_t ost_event;
> > >      uint32_t ost_status;
> > > @@ -75,4 +77,23 @@ extern const VMStateDescription vmstate_cpu_hotplug;
> > >      VMSTATE_STRUCT(cpuhp, state, 1, \
> > >                     vmstate_cpu_hotplug, CPUHotplugState)
> > >
> > > +/**
> > > + * acpi_persistent_cpu:
> > > + * @cpu: The vCPU to check
> > > + *
> > > + * Checks if the vCPU state should always be reflected as *present* via  
> > ACPI  
> > > + * to the Guest. By default, this is False on all architectures and has  
> > to be  
> > > + * explicity set during initialization.
> > > + *
> > > + * Returns: True if it is ACPI 'persistent' CPU
> > > + *
> > > + */
> > > +static inline bool acpi_persistent_cpu(CPUState *cpu)
> > > +{
> > > +    /*
> > > +     * returns if 'Presence' of the vCPU is persistent and should be  
> > simulated  
> > > +     * via ACPI even after vCPUs have been unplugged in QOM
> > > +     */
> > > +    return cpu && cpu->acpi_persistent;
> > > +}
> > >  #endif
> > > diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
> > > index 04e9ad4996..299e96c45b 100644
> > > --- a/include/hw/core/cpu.h
> > > +++ b/include/hw/core/cpu.h
> > > @@ -542,6 +542,27 @@ struct CPUState {
> > >      CPUPluginState *plugin_state;
> > >  #endif
> > >
> > > +    /*
> > > +     * To implement the vCPU hotplug feature (which simulates CPU  
> > hotplug  
> > > +     * behavior), we need to dynamically create and destroy QOM vCPU  
> > objects,  
> > > +     * and (de)associate them with pre-existing KVM vCPUs while  
> > (un)parking the  
> > > +     * KVM vCPU context. One challenge is ensuring that these  
> > dynamically  
> > > +     * appearing or disappearing QOM vCPU objects are accurately  
> > reflected  
> > > +     * through ACPI to the Guest Kernel. Due to architectural  
> > constraints,  
> > > +     * changing the number of vCPUs after the guest kernel has booted  
> > may not  
> > > +     * always be possible.
> > > +     *
> > > +     * In certain architectures, to provide the guest kernel with a  
> > *persistent*  
> > > +     * view of vCPU presence, even when the QOM does not have a  
> > corresponding  
> > > +     * vCPU object, ACPI may simulate the presence of vCPUs by marking  
> > them as  
> > > +     * ACPI-disabled. This is achieved by setting `_STA.PRES=True` and
> > > +     * `_STA.Ena=False` for unplugged vCPUs in QEMU's QOM.
> > > +     *
> > > +     * By default, this flag is set to `FALSE`, and it must be  
> > explicitly set  
> > > +     * to `TRUE` for architectures like ARM.
> > > +     */
> > > +    bool acpi_persistent;
> > > +
> > >      /* TODO Move common fields from CPUArchState here. */
> > >      int cpu_index;
> > >      int cluster_index;  
> >
> >  

RE: [PATCH V1 1/4] hw/acpi: Initialize ACPI Hotplug CPU Status with Support for vCPU `Persistence`

[Salil Mehta via]

Hi Igor,

Thanks for replying back and sorry for the late reply. I wanted to make sure
that I've addressed your comments internally and I've at-least one solution
fully tested and working with adjusted main series (upcoming RFC V6) after
addressing your major comments. I'm further open to your suggestions.

I've just floated V2 series of this ACPI part addressing your major concerns:

#1. Dropped `acpi_persistence` logic from the ACPI code. Now, all the vCPUs
      are always part of the `possible_cpus` list. This has reduced code.
#2. Removed the `ACPICpuStatus::is_present` state and corresponding ACPI
       CPU_PRESENT flag from the ACPI interface and it logic in the CPUs AML.
#3. Introduced the conditional inclusion of the CPU HP VMSD in the GED's
       VMSD using .needed() hook. 
#4.  Fixed the make-qtest/bios-acpi-test failure on x86/{q35,pc} platforms
       because of the change in the CPUs AML code
#5. Adjusted the main series (upcoming RFC V6) with above logic and have
       found it working. I've requested other stake holders to test the ACPI
       part and the complete RFC V6 as well.

At present I've not removed `is_enabled` part from the migration code which
you requested to make it part of the `CPUState`. I was wondering if there was
a way to keep it in the migration state without breaking x86 migration code?

A humble request:
If not, do you think we can go ahead with acceptance of rest of the patches
for this cycle and drop the last patch?


Arch agnostic ACPI V2 series:
https://lore.kernel.org/qemu-devel/20241031200502.3869-1-salil.mehta@huawei.com/T/#mf10104510269d5c290622a0471f0997ad058e397

Upcoming RFC V6, Support of Virtual CPU Hotplug for ARMv8 Arch
Link: https://github.com/salil-mehta/qemu/commits/virt-cpuhp-armv8/rfc-v6-rc4


Please find my replies inline for the rest of the comments. 


Many thanks!

Best regards
Salil.
 

>  From: Igor Mammedov <imammedo@redhat.com>
>  Sent: Friday, October 25, 2024 2:52 PM
>  To: Salil Mehta <salil.mehta@opnsrc.net>
>  Cc: Salil Mehta <salil.mehta@huawei.com>; qemu-devel@nongnu.org;
>  qemu-arm@nongnu.org; mst@redhat.com; maz@kernel.org; jean-
>  philippe@linaro.org; Jonathan Cameron
>  <jonathan.cameron@huawei.com>; lpieralisi@kernel.org;
>  peter.maydell@linaro.org; richard.henderson@linaro.org;
>  andrew.jones@linux.dev; david@redhat.com; philmd@linaro.org;
>  eric.auger@redhat.com; will@kernel.org; ardb@kernel.org;
>  oliver.upton@linux.dev; pbonzini@redhat.com; gshan@redhat.com;
>  rafael@kernel.org; borntraeger@linux.ibm.com; alex.bennee@linaro.org;
>  npiggin@gmail.com; harshpb@linux.ibm.com; linux@armlinux.org.uk;
>  darren@os.amperecomputing.com; ilkka@os.amperecomputing.com;
>  vishnu@os.amperecomputing.com; karl.heubaum@oracle.com;
>  miguel.luis@oracle.com; zhukeqian <zhukeqian1@huawei.com>;
>  wangxiongfeng (C) <wangxiongfeng2@huawei.com>; wangyanan (Y)
>  <wangyanan55@huawei.com>; jiakernel2@gmail.com;
>  maobibo@loongson.cn; lixianglai@loongson.cn; shahuang@redhat.com;
>  zhao1.liu@intel.com; Linuxarm <linuxarm@huawei.com>;
>  gustavo.romero@linaro.org
>  Subject: Re: [PATCH V1 1/4] hw/acpi: Initialize ACPI Hotplug CPU Status with
>  Support for vCPU `Persistence`
>  
>  On Mon, 21 Oct 2024 22:50:40 +0100
>  Salil Mehta <salil.mehta@opnsrc.net> wrote:
>  
>  > Hi Igor,
>  >
>  > Thanks for taking time to review and sorry for not being prompt. I was
>  > in transit due to some difficult personal situation.
>  >
>  > On Fri, Oct 18, 2024 at 3:11 PM Igor Mammedov
>  <imammedo@redhat.com> wrote:
>  >
>  > > On Mon, 14 Oct 2024 20:22:02 +0100
>  > > Salil Mehta <salil.mehta@huawei.com> wrote:
>  > >
>  > > > Certain CPU architecture specifications [1][2][3] prohibit changes
>  > > > to CPU
>  > >                                           ^^^^^^^^^ these do not
>  > > point to specs but never mind
>  > >
>  > > > presence after the kernel has booted. This limitation exists
>  > > > because
>  > > many system
>  > > > initializations rely on the exact CPU count at boot time and do
>  > > > not
>  > > expect it to
>  > > > change later. For example, components like interrupt controllers,
>  > > > which
>  > > are
>  > > > closely tied to CPUs, or various per-CPU features, may not support
>  > > configuration
>  > > > changes once the kernel has been initialized. This presents a
>  > > > challenge
>  > > for
>  > > > virtualization features such as vCPU hotplug.
>  > >
>  > > well, x86 (incl cpu,interrupt ctrls) also doesn't have architectural
>  > > hotplug.
>  > > It's just OEM managed to implement it regardless and then bothered
>  > > to make OS changes to work with that.
>  > > It's just ARM community doesn't want to go there at this point of
>  > > time but using above reasoning as justification for this series
>  > > doesn't look good to me.
>  > >
>  >
>  >
>  > There is a difference though. vCPU presence cannot be changed after
>  > the system has initialized in the ARM world due to the Architectural
>  > constraint.
>  > I think
>  > in the x86 world
>  > it is allowed?
>  
>  As far as I know x86 doesn't have arch defined hotplug, but vendors
>  managed to implement it somehow and then made sure that OS running on
>  top could stomach it.


sure, I understand. ARM specification imposes restrictions on any such
change. It is well controlled by the architecture team within ARM which
includes the hardware guys. We've worked very closely with ARM in the
past 4 years to be able to check the viability of any such change but it was
repeatedly rejected by the architecture team. 

In fact, Jean-phillipe from Linaro/ARM (CC'ed in this series) tried different
approach using this series but it required change in the specification. 
AFAIK, it was attempted but was rejected. Please follow below link:

https://op-lists.linaro.org/archives/list/linaro-open-discussions@op-lists.linaro.org/message/X74JS6P2N4AUWHHATJJVVFDI2EMDZJ74/


We've come to current stage after lots of discussions and attempts while
co-working with ARM/Linaro and other companies using Linaro-open-discussions
as a common platform for co-working for over past 3 years.


>  
>  > > So what ARM would like to support is not CPU hotplug but rather a
>  > > fixed system with standby CPUs (that can be powered on/off on
>  demand).
>  > > With ACPI spec amended to support that (MADT present/enabled
>  > > changes), it's good enough reason to add 'enabled' handling to acpi
>  > > cpu-hotplug code instead of inventing alternative one that would do
>  > > almost the same.
>  > >
>  >
>  >
>  > Not sure what you mean here but this is what is being done.
>  it was ack for 'enabled' flag in cphp acpi abi.
>  The rest was digression wrt 'present' hack in ACPI code.


Ok. I've incorporated your suggestions in V2 series


>  > > But lets get rid of (can't/may not) language above and use standby
>  > > CPUs reasoning to avoid any confusion.
>  > >
>  >
>  >
>  > I've to disagree here. Standy-by means you will have to keep all the vCPUs
>  > states
>  > realized and the objects will always exist. This is a problem for larger
>  > systems
>  > with vCPU hotplug support as it drastically affects the boot times.
>  >
>  
>  see comment below about boot times.
>  
>  > Please
>  > check
>  > the KVM Forum 2023 slides for objective values.
>  
>  For sure we can conjure unicorns especially virtual ones,
>  it doesn't mean that we should do it though.
>  
>  > It's been a long time since this series was actually conceived and at that
>  > time we wanted
>  > to use it for kata containers but later with the gradual adoption of
>  > various microvms
>  > and the cloud hypervisor requirements have bit changed. Boot time still
>  > remains one
>  > of the major requirements. Not bounding boot time while using this
>  feature
>  > will
>  > seriously affect performance if the number of vCPUs increases
>  
>  again wrt boot time, see comment below.
>  
>  
>  > > PS:
>  > > I'm taking about hw hotplug (at QEMU level) and not kernel hotplug
>  > > (where it's at logical cpu level).
>  > >
>  >
>  > sure
>  >
>  >
>  > >
>  > > > To address this issue, introduce an `is_enabled` state in the
>  > > `AcpiCpuStatus`,
>  > > > which reflects whether a vCPU has been hot-plugged or hot-
>  unplugged in
>  > > QEMU,
>  > > > marking it as (un)available in the Guest Kernel.
>  > > good so far
>  > >
>  > > > The `is_present` state should
>  > > > be set based on the `acpi_persistent` flag. In cases where unplugged
>  > > vCPUs need
>  > > > to be deliberately simulated in the ACPI to maintain a persistent view
>  > > of vCPUs,
>  > > > this flag ensures the guest kernel continues to see those vCPUs.
>  > >
>  > > that's where I have to disagree, vCPU is present when a corresponding
>  QOM
>  > > object
>  > > exists. Faking it's presence will only confuse at the end.
>  > >
>  >
>  >
>  > Not faking care of it will mean realization  of the unnecessary vCPU
>  > threads during
>  > the VM init time, which in turn will increase the boot time. Trade-off
>  > between more
>  > clean design and performance.
>  
>  I very much prefer clean design, which should result in
>  less code/less bugs/easier maintenance and less regressions
>  when someone fixes intentional hacks, with a good reasoning that
>  hardware doesn't work that way.


I've removed the `acpi_persistent` logic in the just floated V2 series.
Yes, code has reduced. Please have a look.


>  
>  wrt boot time, see below.
>  
>  > > I get that you want to reuse device_add/del interface, but that leads to
>  > > pulling the leg here and there to make thing fit. That in short therm
>  > > (while someone remembers all tricks) might work for some, but long
>  therm
>  > > it's not sustainable).
>  > >
>  >
>  >
>  > I do not understand why?
>  
>  It's a complicated set of hacks all over place to make something that
>  do not exists in the real world work somehow.


All vCPUs are now part of the `possible_cpus_list` and ACPI CPU state is
now consistent with QOM vCPU object state all the time.

>  
>  While at the same time there is alternative approach that mimics ARM
>  hardware
>  behavior and is supported by vendor (standby cpus).
>  That also will be much more simple way, while providing similar to hotplug
>  functionality.


You still need some changes in the ACPI specifications to allow delay online'ing
the vCPUs. Also, we need a way to avoid accesses to the GIC CPU Interfaces while
the so called `stand-by` vCPUs are not operational but are part of the larger
`possible_cpus_list`.  The existing changes in the GICv3 code and the ACPI
(GICC::flags::online-capable) part does exactly the same thing.

Hot(un)plug hooks are boiler plate code part of the hotplug framework which
x86 also implements.


>  > > Maybe instead of pushing device_add/del, we should rather implement
>  > > standby CPU model here, as ARM folks expect it to be.
>  > > i.e. instead of device_add/del add 'enabled' property to ARM vCPU,
>  > > and let management to turn on/off vCPUs on demand.
>  > > (and very likely this model could be reused by other sock based
>  platforms)
>  > > For end-user it would be the same as device_add/del (i.e. vCPU
>  becomes
>  > > usable/unsable)
>  >
>  >
>  > One of the main goals is to facilitate seamless migration from the x86
>  > world to
>  > ARM world. Hence, preservation of the x86 interface is important. It is a
>  > requirement.
>  > Just imagine if Apple had to change end user interface experience after
>  > migrating iOS
>  > from x86 to ARM architecture. ?
>  
>  About that I wouldn't worry much as it's secondary.
>  Management can adapt to call 'qom set' instead of calling device_add/del.
>  It definitely shouldn't be the thing that turns design decisions
>  into the bad model.


I'm using exactly the same model which x86 is also using. We want to emulate
x86 interface here. 


>  > > I'd bet it would simplify your ARM CPU hotplug series a lot,
>  > > since you won't have to fake vCPU object lifetime and do
>  > > non trivial tricks to make it all work.
>  > > Which it turn will make ARM hotplug series much more approachable
>  > > for reviewers /and whomever comes later across that code/.
>  > >
>  >
>  > Tricks are just for ACPI and GIC and nothing else. Rest is a
>  > boilerplate code of the
>  > vCPU hotplug framework which x86 is also using.
>  
>  looking at your v5 ARM cphp series, it does a bunch refactoring
>  to handle CPU absence whre there should be one by design.


I'll be separating out some common refactoring which can be floated
independently of the vCPU hotplug code. May be this might help in
alleviating your concerns. 


>  
>  While in standby cpus, that won't be needed.
>  (one would need a code to enable/disable CPUs, plus making ACPI
>  deliver those events, but that's pretty much all. i.e. do what real
>  hw can do)


By keeping all the objects as part of the `possible_cpus_list` we are indeed
creating stand-by CPUs only. Rest is the interface part, I've a prototype ready
for that as well but then that would be part of main series sometime later.

>  
>  > > Regardless of said, we would still need changes to ACPI cphp code,
>  > > see my comments inline.


Sure, thanks for your valuable comments. I've tried to address most of them.
Please have a look at V2 series and let me know if you still have concerns. I'll
try my best to address them.


[...]
>  > > > diff --git a/cpu-target.c b/cpu-target.c
>  > > > index 499facf774..c8a29ab495 100644
>  > > > --- a/cpu-target.c
>  > > > +++ b/cpu-target.c
>  > > > @@ -200,6 +200,7 @@ static Property cpu_common_props[] = {
>  > > >       */
>  > > >      DEFINE_PROP_LINK("memory", CPUState, memory,
>  TYPE_MEMORY_REGION,
>  > > >                       MemoryRegion *),
>  > > > +    DEFINE_PROP_BOOL("acpi-persistent", CPUState, acpi_persistent,
>  > > false),
>  > >
>  > > I agree with Gavin, it's not CPU property/business, but a platform one.
>  > >
>  > > Pass it as argument to cpu_hotplug_hw_init(),
>  > > and maybe rename to always_present.
>  > > Then make sure that it's configurable in GED (which calls the function),
>  > > and just turn it on for arm/virt machine.
>  > > Other platforms might want to use x86 approach with GED and have
>  > > vCPU actually disappearing. /loongson and maybe risc-v/
>  > >
>  >
>  > can we move it to Machine level or fetch it through firmware?
>  
>  following would do:
>    1. add to ged  a new property to opt-in into this
>    2. set the new property from arm's board_init where GED is created
>    3. when GED calls cpu_hotplug_hw_init(), pass property value as an
>  argument


Because I've dropped the `acpi_persistent` a special flag to distinguish the
disabled vCPUs is not required. Please check the V2 series.

[...]
>  
>  > > >  #endif
>  > > >      DEFINE_PROP_END_OF_LIST(),
>  > > >  };
>  > > > diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
>  > > > index 5cb60ca8bc..083c4010c2 100644
>  > > > --- a/hw/acpi/cpu.c
>  > > > +++ b/hw/acpi/cpu.c
>  > > > @@ -225,7 +225,40 @@ void cpu_hotplug_hw_init(MemoryRegion
>  *as, Object
>  > > *owner,
>  > > >      state->dev_count = id_list->len;
>  > > >      state->devs = g_new0(typeof(*state->devs), state->dev_count);
>  > > >      for (i = 0; i < id_list->len; i++) {
>  > > > -        state->devs[i].cpu =  CPU(id_list->cpus[i].cpu);
>  > > > +        struct CPUState *cpu = CPU(id_list->cpus[i].cpu);
>  > > > +        /*
>  > > > +         * In most architectures, CPUs that are marked as ACPI
>  > > 'present' are
>  > > > +         * also ACPI 'enabled' by default. These states remain
>  > > consistent at
>  > > > +         * both the QOM and ACPI levels.
>  > > > +         */
>  > > > +        if (cpu) {
>  > > > +            state->devs[i].is_enabled = true;
>  > > > +            state->devs[i].is_present = true;
>  > > > +            state->devs[i].cpu = cpu;
>  > > > +        } else {
>  > > > +            state->devs[i].is_enabled = false;
>  > > > +            /*
>  > > > +             * In some architectures, even 'unplugged' or 'disabled'
>  > > QOM CPUs
>  > > > +             * may be exposed as ACPI 'present.' This approach provides
>  > > a
>  > > > +             * persistent view of the vCPUs to the guest kernel. This
>  > > could be
>  > > > +             * due to an architectural constraint that requires every
>  > > per-CPU
>  > > > +             * component to be present at boot time, meaning the exact
>  > > count of
>  > > > +             * vCPUs must be known and cannot be altered after the
>  > > kernel has
>  > > > +             * booted. As a result, the vCPU states at the QOM and ACPI
>  > > levels
>  > > > +             * might become inconsistent. However, in such cases, the
>  > > presence
>  > > > +             * of vCPUs has been deliberately simulated at the ACPI
>  > > level.
>  > > > +             */
>  > >
>  > > if cpus are not 'simulated', you will not need comments explaining that
>  all
>  > > over place and whole hunk would likely go away.
>  > >
>  >
>  > I can reduce the content if you wish.
>  
>  you have those comments for a reason since the code does unexpected
>  'simulate' thing. Removing that would mask intentional behavior
>  for a reader later down the road.


Dropped the `acpi_persistent` logic and the related comments.


[...]

>  > > > diff --git a/include/hw/acpi/cpu.h b/include/hw/acpi/cpu.h
>  > > > index 32654dc274..bd3f9973c9 100644
>  > > > --- a/include/hw/acpi/cpu.h
>  > > > +++ b/include/hw/acpi/cpu.h
>  > > > @@ -26,6 +26,8 @@ typedef struct AcpiCpuStatus {
>  > > >      uint64_t arch_id;
>  > > >      bool is_inserting;
>  > > >      bool is_removing;
>  > > > +    bool is_present;
>  > > with always_present, it might be better to move field to
>  CPUHotplugState
>  > > as it's not per vCPU anymore, and in standby case state->devs[i].cpu
>  > > should work as implicit present flag. (see below wrt doc first comment)
>  > >
>  >
>  > I'm still not convinced of the stand-by approach because of the
>  performance
>  > impact
>  > it will have upon increasing the number of possible vCPUs and its worst
>  > case is
>  > unbounded. That's a major problem.
>  
>  # of vCPUs is always bound by host capabilities and by modeled hardware.
>  
>  From guest point of view both approaches should save time as
>  guest won't try to online non-present or disabled CPUs
>  (with large machines guest boot time usually dwarfs whatever
>  QEMU does before the guest, +UEFI adds to insult event more).
>  And fast booting large VM (with non present CPUs) initially,
>  is just postponing problem the the later time when hotplugging
>  CPUs causes worse timing (since guest effectively does stop_machine)
>  for 'online'-ing to happen.
>  
>  Regardless, standby CPUs is what ARM folks have chosen to support.
>  In this case, I'd pick arch preferred way anytime vs some boot time
>  improvements.


It's unfortunate for all of us but ARM folks are legally binded not to comment
on the Qemu code. But Jean-phillipe Brucker from Linaro has indeed gone
through the Qemu patches earlier in year 2021. 

https://op-lists.linaro.org/archives/list/linaro-open-discussions@op-lists.linaro.org/message/X74JS6P2N4AUWHHATJJVVFDI2EMDZJ74/

We had tried to solve the problem using just PSCI approach earlier.

`Stand-by` CPUs is just a cosmetic term. By having all the vCPUs part
of the `possible_cpus_list` and parking disabled vCPUs. We've indeed kept
the vCPUs on `stand-by`. This now is the existing approach in the upcoming
RFC V6. ACPI CPU state is consistent with QOM vCPUs state.


>  
>  If arm_cpu_realizefn() is still expensive for your case and better
>  init times are needed, fix it instead of working around it by faking
>  hotplug on QEMU side and moving issue to the later time.
>  That way will benefit not only hotplug case, but also huge VMs case.
>  (good example is x86, where it scales well without any hotplug)


Sure, agreed. that’s definitely one of the way to improve but it can also
be add-on?


>  
>  PS:
>  Out of curiosity, I've just fed qemu to perf on Ampere host.
>  There are a number of low hanging fruits that would reduce consumed
>  CPU cycles, for those who wishes to improve performance.
>  
>  Some would lead to overall improvements, other could be
>  done when vCPU is disabled.
>  
>  ex: with -enable-kvm -smp 100
>     - 26.18% arm_cpu_realizefn
>        + 8.86% cpu_reset <- likely is not necessary for disabled vCPUs, as one
>  has to call it again when enabling vCPU
>        + 4.53% register_cp_regs_for_features
>        + 4.32% arm_cpu_register_gdb_regs_for_features  <- do we need it
>  when gdbstub is not enabled?
>        + 4.09% init_cpreg_list  <- naive refactoring makes it a fraction of percent
>        + 3.40% qemu_init_vcpu
>          0.59% trace_init_vcpu
>  
>  with above hacks it goes down to 11% (12% for x86_cpu_realizefn).
>  However wall-clock wise compared to x86, the arm/virt doesn't scale well.
>  So there is something else tgetting in the way (i.e. something stalls vCPU
>  creation on ARM?).


Thanks for these leads but I will need to spend time in this direction to check
and verify these. I'll get back to you regarding your question.


>  
>  And well, I' might be comparing apples to oranges here (old E5-2630v3 vs
>  some 32core 3.3Ghz Ampere cpu).

True. 

>  
>  PS2:
>  create-gic() is another candidate for improvement, it spends a lot
>  of time on setting GPIO properties.

Yes, indeed. These are very helpful. I will ensure that I look into this direction
as well regardless of the vCPU hotplug patches. Maybe we can combine all
good things together.


>  
>  > > > +    bool is_enabled;
>  > > I'd move introduction of this field into a separate patch.
>  > >
>  > > BTW: new ABI/fields accessible by guest should be described in
>  > > docs/specs/acpi_cpu_hotplug.rst.
>  > > It would be better to have the spec as patch 1st, that we all agree on
>  > > and then follow with implementation.
>  > >
>  >
>  > We can do that.
>  >
>  >
>  > > And also include there an expected workflow for standby case.
>  >
>  >
>  >
>  > We need more discussion on this as our requirements for which we
>  floated
>                                        ^^^^^^^^^^^^^^
>  > this
>  > feature are not being met using stand-by cases.
>  
>  A list of them would be a good starting point for discussion.

Sure, agreed. I'll jot them down as part of the main series.

>  Perhaps requirements should be made more realistic and be re-evaluated.


Requirements are clearly indicated on the cover-letter of the main series.


Best regards
Salil.

RE: [PATCH V1 1/4] hw/acpi: Initialize ACPI Hotplug CPU Status with Support for vCPU `Persistence`

[Salil Mehta via]

Hi Igor,

I've fixed most of the x86 problems you had commented in the V1 patch-set in the
recently floated V3 ACPI patch-set. This includes removing the `is_{enabled,present}`
ACPI CPU States for which you expressed apprehensions.  Sorry, there was a miss
from my side in the V2 related to the CPUs AML code for x86 platform. I've fixed that
as well.

Please let me know if this patch-set addresses all your concerns. I am open to any
suggestions you deem necessary for its acceptance in this cycle. 

Acceptance of Arch agnostic ACPI changes in this cycle will really help all of us.

https://lore.kernel.org/qemu-devel/20241103102419.202225-1-salil.mehta@huawei.com/


Many thanks!

Best regards
Salil.

>  From: Salil Mehta
>  Sent: Friday, November 1, 2024 10:54 AM
>  To: 'Igor Mammedov' <imammedo@redhat.com>; Salil Mehta
>  <salil.mehta@opnsrc.net>
>  
>  Hi Igor,
>  
>  Thanks for replying back and sorry for the late reply. I wanted to make sure
>  that I've addressed your comments internally and I've at-least one solution
>  fully tested and working with adjusted main series (upcoming RFC V6) after
>  addressing your major comments. I'm further open to your suggestions.
>  
>  I've just floated V2 series of this ACPI part addressing your major concerns:
>  
>  #1. Dropped `acpi_persistence` logic from the ACPI code. Now, all the vCPUs
>        are always part of the `possible_cpus` list. This has reduced code.
>  #2. Removed the `ACPICpuStatus::is_present` state and corresponding ACPI
>         CPU_PRESENT flag from the ACPI interface and it logic in the CPUs AML.
>  #3. Introduced the conditional inclusion of the CPU HP VMSD in the GED's
>         VMSD using .needed() hook.
>  #4.  Fixed the make-qtest/bios-acpi-test failure on x86/{q35,pc} platforms
>         because of the change in the CPUs AML code #5. Adjusted the main
>  series (upcoming RFC V6) with above logic and have
>         found it working. I've requested other stake holders to test the ACPI
>         part and the complete RFC V6 as well.
>  
>  At present I've not removed `is_enabled` part from the migration code
>  which you requested to make it part of the `CPUState`. I was wondering if
>  there was a way to keep it in the migration state without breaking x86
>  migration code?
>  
>  A humble request:
>  If not, do you think we can go ahead with acceptance of rest of the patches
>  for this cycle and drop the last patch?
>  
>  
>  Arch agnostic ACPI V2 series:
>  https://lore.kernel.org/qemu-devel/20241031200502.3869-1-
>  salil.mehta@huawei.com/T/#mf10104510269d5c290622a0471f0997ad058e39
>  7
>  
>  Upcoming RFC V6, Support of Virtual CPU Hotplug for ARMv8 Arch
>  Link: https://github.com/salil-mehta/qemu/commits/virt-cpuhp-armv8/rfc-
>  v6-rc4
>  
>  
>  Please find my replies inline for the rest of the comments.
>  
>  
>  Many thanks!
>  
>  Best regards
>  Salil.
>  
>  
>  >  From: Igor Mammedov <imammedo@redhat.com>
>  >  Sent: Friday, October 25, 2024 2:52 PM
>  >  To: Salil Mehta <salil.mehta@opnsrc.net>
>  >  Cc: Salil Mehta <salil.mehta@huawei.com>; qemu-devel@nongnu.org;
>  > qemu-arm@nongnu.org; mst@redhat.com; maz@kernel.org; jean-
>  > philippe@linaro.org; Jonathan Cameron
>  <jonathan.cameron@huawei.com>;
>  > lpieralisi@kernel.org;  peter.maydell@linaro.org;
>  > richard.henderson@linaro.org;  andrew.jones@linux.dev;
>  > david@redhat.com; philmd@linaro.org;  eric.auger@redhat.com;
>  > will@kernel.org; ardb@kernel.org;  oliver.upton@linux.dev;
>  > pbonzini@redhat.com; gshan@redhat.com;  rafael@kernel.org;
>  > borntraeger@linux.ibm.com; alex.bennee@linaro.org;
>  npiggin@gmail.com;
>  > harshpb@linux.ibm.com; linux@armlinux.org.uk;
>  > darren@os.amperecomputing.com; ilkka@os.amperecomputing.com;
>  > vishnu@os.amperecomputing.com; karl.heubaum@oracle.com;
>  > miguel.luis@oracle.com; zhukeqian <zhukeqian1@huawei.com>;
>  > wangxiongfeng (C) <wangxiongfeng2@huawei.com>; wangyanan (Y)
>  > <wangyanan55@huawei.com>; jiakernel2@gmail.com;
>  maobibo@loongson.cn;
>  > lixianglai@loongson.cn; shahuang@redhat.com;  zhao1.liu@intel.com;
>  > Linuxarm <linuxarm@huawei.com>;  gustavo.romero@linaro.org
>  >  Subject: Re: [PATCH V1 1/4] hw/acpi: Initialize ACPI Hotplug CPU
>  > Status with  Support for vCPU `Persistence`
>  >
>  >  On Mon, 21 Oct 2024 22:50:40 +0100
>  >  Salil Mehta <salil.mehta@opnsrc.net> wrote:
>  >
>  >  > Hi Igor,
>  >  >
>  >  > Thanks for taking time to review and sorry for not being prompt. I
>  > was  > in transit due to some difficult personal situation.
>  >  >
>  >  > On Fri, Oct 18, 2024 at 3:11 PM Igor Mammedov
>  > <imammedo@redhat.com> wrote:
>  >  >
>  >  > > On Mon, 14 Oct 2024 20:22:02 +0100  > > Salil Mehta
>  > <salil.mehta@huawei.com> wrote:
>  >  > >
>  >  > > > Certain CPU architecture specifications [1][2][3] prohibit
>  > changes  > > > to CPU
>  >  > >                                           ^^^^^^^^^ these do not
>  >  > > point to specs but never mind
>  >  > >
>  >  > > > presence after the kernel has booted. This limitation exists  >
>  > > > because  > > many system  > > > initializations rely on the exact
>  > CPU count at boot time and do  > > > not  > > expect it to  > > >
>  > change later. For example, components like interrupt controllers,  > >
>  > > which  > > are  > > > closely tied to CPUs, or various per-CPU
>  > features, may not support  > > configuration  > > > changes once the
>  > kernel has been initialized. This presents a  > > > challenge  > > for
>  > > > > virtualization features such as vCPU hotplug.
>  >  > >
>  >  > > well, x86 (incl cpu,interrupt ctrls) also doesn't have
>  > architectural  > > hotplug.
>  >  > > It's just OEM managed to implement it regardless and then
>  > bothered  > > to make OS changes to work with that.
>  >  > > It's just ARM community doesn't want to go there at this point of
>  > > > time but using above reasoning as justification for this series  >
>  > > doesn't look good to me.
>  >  > >
>  >  >
>  >  >
>  >  > There is a difference though. vCPU presence cannot be changed after
>  > > the system has initialized in the ARM world due to the Architectural
>  > > constraint.
>  >  > I think
>  >  > in the x86 world
>  >  > it is allowed?
>  >
>  >  As far as I know x86 doesn't have arch defined hotplug, but vendors
>  > managed to implement it somehow and then made sure that OS running
>  on
>  > top could stomach it.
>  
>  
>  sure, I understand. ARM specification imposes restrictions on any such
>  change. It is well controlled by the architecture team within ARM which
>  includes the hardware guys. We've worked very closely with ARM in the
>  past 4 years to be able to check the viability of any such change but it was
>  repeatedly rejected by the architecture team.
>  
>  In fact, Jean-phillipe from Linaro/ARM (CC'ed in this series) tried different
>  approach using this series but it required change in the specification.
>  AFAIK, it was attempted but was rejected. Please follow below link:
>  
>  https://op-lists.linaro.org/archives/list/linaro-open-discussions@op-
>  lists.linaro.org/message/X74JS6P2N4AUWHHATJJVVFDI2EMDZJ74/
>  
>  
>  We've come to current stage after lots of discussions and attempts while co-
>  working with ARM/Linaro and other companies using Linaro-open-
>  discussions as a common platform for co-working for over past 3 years.
>  
>  
>  >
>  >  > > So what ARM would like to support is not CPU hotplug but rather a
>  > > > fixed system with standby CPUs (that can be powered on/off on
>  > demand).
>  >  > > With ACPI spec amended to support that (MADT present/enabled  > >
>  > changes), it's good enough reason to add 'enabled' handling to acpi  >
>  > > cpu-hotplug code instead of inventing alternative one that would do
>  > > > almost the same.
>  >  > >
>  >  >
>  >  >
>  >  > Not sure what you mean here but this is what is being done.
>  >  it was ack for 'enabled' flag in cphp acpi abi.
>  >  The rest was digression wrt 'present' hack in ACPI code.
>  
>  
>  Ok. I've incorporated your suggestions in V2 series
>  
>  
>  >  > > But lets get rid of (can't/may not) language above and use
>  > standby  > > CPUs reasoning to avoid any confusion.
>  >  > >
>  >  >
>  >  >
>  >  > I've to disagree here. Standy-by means you will have to keep all
>  > the vCPUs  > states  > realized and the objects will always exist.
>  > This is a problem for larger  > systems  > with vCPU hotplug support
>  > as it drastically affects the boot times.
>  >  >
>  >
>  >  see comment below about boot times.
>  >
>  >  > Please
>  >  > check
>  >  > the KVM Forum 2023 slides for objective values.
>  >
>  >  For sure we can conjure unicorns especially virtual ones,  it doesn't
>  > mean that we should do it though.
>  >
>  >  > It's been a long time since this series was actually conceived and
>  > at that  > time we wanted  > to use it for kata containers but later
>  > with the gradual adoption of  > various microvms  > and the cloud
>  > hypervisor requirements have bit changed. Boot time still  > remains
>  > one  > of the major requirements. Not bounding boot time while using
>  > this  feature  > will  > seriously affect performance if the number of
>  > vCPUs increases
>  >
>  >  again wrt boot time, see comment below.
>  >
>  >
>  >  > > PS:
>  >  > > I'm taking about hw hotplug (at QEMU level) and not kernel
>  > hotplug  > > (where it's at logical cpu level).
>  >  > >
>  >  >
>  >  > sure
>  >  >
>  >  >
>  >  > >
>  >  > > > To address this issue, introduce an `is_enabled` state in the
>  > > > `AcpiCpuStatus`,  > > > which reflects whether a vCPU has been
>  > hot-plugged or hot-  unplugged in  > > QEMU,  > > > marking it as
>  > (un)available in the Guest Kernel.
>  >  > > good so far
>  >  > >
>  >  > > > The `is_present` state should
>  >  > > > be set based on the `acpi_persistent` flag. In cases where
>  > unplugged  > > vCPUs need  > > > to be deliberately simulated in the
>  > ACPI to maintain a persistent view  > > of vCPUs,  > > > this flag
>  > ensures the guest kernel continues to see those vCPUs.
>  >  > >
>  >  > > that's where I have to disagree, vCPU is present when a
>  > corresponding  QOM  > > object  > > exists. Faking it's presence will
>  > only confuse at the end.
>  >  > >
>  >  >
>  >  >
>  >  > Not faking care of it will mean realization  of the unnecessary
>  > vCPU  > threads during  > the VM init time, which in turn will
>  > increase the boot time. Trade-off  > between more  > clean design and
>  > performance.
>  >
>  >  I very much prefer clean design, which should result in  less
>  > code/less bugs/easier maintenance and less regressions  when someone
>  > fixes intentional hacks, with a good reasoning that  hardware doesn't
>  > work that way.
>  
>  
>  I've removed the `acpi_persistent` logic in the just floated V2 series.
>  Yes, code has reduced. Please have a look.
>  
>  
>  >
>  >  wrt boot time, see below.
>  >
>  >  > > I get that you want to reuse device_add/del interface, but that
>  > leads to  > > pulling the leg here and there to make thing fit. That
>  > in short therm  > > (while someone remembers all tricks) might work
>  > for some, but long  therm  > > it's not sustainable).
>  >  > >
>  >  >
>  >  >
>  >  > I do not understand why?
>  >
>  >  It's a complicated set of hacks all over place to make something that
>  > do not exists in the real world work somehow.
>  
>  
>  All vCPUs are now part of the `possible_cpus_list` and ACPI CPU state is now
>  consistent with QOM vCPU object state all the time.
>  
>  >
>  >  While at the same time there is alternative approach that mimics ARM
>  > hardware  behavior and is supported by vendor (standby cpus).
>  >  That also will be much more simple way, while providing similar to
>  > hotplug  functionality.
>  
>  
>  You still need some changes in the ACPI specifications to allow delay
>  online'ing the vCPUs. Also, we need a way to avoid accesses to the GIC CPU
>  Interfaces while the so called `stand-by` vCPUs are not operational but are
>  part of the larger `possible_cpus_list`.  The existing changes in the GICv3
>  code and the ACPI
>  (GICC::flags::online-capable) part does exactly the same thing.
>  
>  Hot(un)plug hooks are boiler plate code part of the hotplug framework
>  which
>  x86 also implements.
>  
>  
>  >  > > Maybe instead of pushing device_add/del, we should rather
>  > implement  > > standby CPU model here, as ARM folks expect it to be.
>  >  > > i.e. instead of device_add/del add 'enabled' property to ARM
>  > vCPU,  > > and let management to turn on/off vCPUs on demand.
>  >  > > (and very likely this model could be reused by other sock based
>  >  platforms)
>  >  > > For end-user it would be the same as device_add/del (i.e. vCPU
>  > becomes  > > usable/unsable)  >  >  > One of the main goals is to
>  > facilitate seamless migration from the x86  > world to  > ARM world.
>  > Hence, preservation of the x86 interface is important. It is a  >
>  > requirement.
>  >  > Just imagine if Apple had to change end user interface experience
>  > after  > migrating iOS  > from x86 to ARM architecture. ?
>  >
>  >  About that I wouldn't worry much as it's secondary.
>  >  Management can adapt to call 'qom set' instead of calling device_add/del.
>  >  It definitely shouldn't be the thing that turns design decisions
>  > into the bad model.
>  
>  
>  I'm using exactly the same model which x86 is also using. We want to
>  emulate
>  x86 interface here.
>  
>  
>  >  > > I'd bet it would simplify your ARM CPU hotplug series a lot,  > >
>  > since you won't have to fake vCPU object lifetime and do  > > non
>  > trivial tricks to make it all work.
>  >  > > Which it turn will make ARM hotplug series much more approachable
>  > > > for reviewers /and whomever comes later across that code/.
>  >  > >
>  >  >
>  >  > Tricks are just for ACPI and GIC and nothing else. Rest is a  >
>  > boilerplate code of the  > vCPU hotplug framework which x86 is also
>  > using.
>  >
>  >  looking at your v5 ARM cphp series, it does a bunch refactoring  to
>  > handle CPU absence whre there should be one by design.
>  
>  
>  I'll be separating out some common refactoring which can be floated
>  independently of the vCPU hotplug code. May be this might help in
>  alleviating your concerns.
>  
>  
>  >
>  >  While in standby cpus, that won't be needed.
>  >  (one would need a code to enable/disable CPUs, plus making ACPI
>  > deliver those events, but that's pretty much all. i.e. do what real
>  > hw can do)
>  
>  
>  By keeping all the objects as part of the `possible_cpus_list` we are indeed
>  creating stand-by CPUs only. Rest is the interface part, I've a prototype
>  ready for that as well but then that would be part of main series sometime
>  later.
>  
>  >
>  >  > > Regardless of said, we would still need changes to ACPI cphp
>  > code,  > > see my comments inline.
>  
>  
>  Sure, thanks for your valuable comments. I've tried to address most of
>  them.
>  Please have a look at V2 series and let me know if you still have concerns. I'll
>  try my best to address them.
>  
>  
>  [...]
>  >  > > > diff --git a/cpu-target.c b/cpu-target.c  > > > index
>  > 499facf774..c8a29ab495 100644  > > > --- a/cpu-target.c  > > > +++
>  > b/cpu-target.c  > > > @@ -200,6 +200,7 @@ static Property
>  > cpu_common_props[] = {
>  >  > > >       */
>  >  > > >      DEFINE_PROP_LINK("memory", CPUState, memory,
>  >  TYPE_MEMORY_REGION,
>  >  > > >                       MemoryRegion *),
>  >  > > > +    DEFINE_PROP_BOOL("acpi-persistent", CPUState,
>  acpi_persistent,
>  >  > > false),
>  >  > >
>  >  > > I agree with Gavin, it's not CPU property/business, but a platform one.
>  >  > >
>  >  > > Pass it as argument to cpu_hotplug_hw_init(),  > > and maybe
>  > rename to always_present.
>  >  > > Then make sure that it's configurable in GED (which calls the
>  > function),  > > and just turn it on for arm/virt machine.
>  >  > > Other platforms might want to use x86 approach with GED and have
>  > > > vCPU actually disappearing. /loongson and maybe risc-v/  > >  >  >
>  > can we move it to Machine level or fetch it through firmware?
>  >
>  >  following would do:
>  >    1. add to ged  a new property to opt-in into this
>  >    2. set the new property from arm's board_init where GED is created
>  >    3. when GED calls cpu_hotplug_hw_init(), pass property value as an
>  > argument
>  
>  
>  Because I've dropped the `acpi_persistent` a special flag to distinguish the
>  disabled vCPUs is not required. Please check the V2 series.
>  
>  [...]
>  >
>  >  > > >  #endif
>  >  > > >      DEFINE_PROP_END_OF_LIST(),
>  >  > > >  };
>  >  > > > diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c  > > > index
>  > 5cb60ca8bc..083c4010c2 100644  > > > --- a/hw/acpi/cpu.c  > > > +++
>  > b/hw/acpi/cpu.c  > > > @@ -225,7 +225,40 @@ void
>  > cpu_hotplug_hw_init(MemoryRegion  *as, Object  > > *owner,
>  >  > > >      state->dev_count = id_list->len;
>  >  > > >      state->devs = g_new0(typeof(*state->devs), state->dev_count);
>  >  > > >      for (i = 0; i < id_list->len; i++) {
>  >  > > > -        state->devs[i].cpu =  CPU(id_list->cpus[i].cpu);
>  >  > > > +        struct CPUState *cpu = CPU(id_list->cpus[i].cpu);
>  >  > > > +        /*
>  >  > > > +         * In most architectures, CPUs that are marked as ACPI
>  >  > > 'present' are
>  >  > > > +         * also ACPI 'enabled' by default. These states remain
>  >  > > consistent at
>  >  > > > +         * both the QOM and ACPI levels.
>  >  > > > +         */
>  >  > > > +        if (cpu) {
>  >  > > > +            state->devs[i].is_enabled = true;
>  >  > > > +            state->devs[i].is_present = true;
>  >  > > > +            state->devs[i].cpu = cpu;
>  >  > > > +        } else {
>  >  > > > +            state->devs[i].is_enabled = false;
>  >  > > > +            /*
>  >  > > > +             * In some architectures, even 'unplugged' or 'disabled'
>  >  > > QOM CPUs
>  >  > > > +             * may be exposed as ACPI 'present.' This approach provides
>  >  > > a
>  >  > > > +             * persistent view of the vCPUs to the guest kernel. This
>  >  > > could be
>  >  > > > +             * due to an architectural constraint that requires every
>  >  > > per-CPU
>  >  > > > +             * component to be present at boot time, meaning the exact
>  >  > > count of
>  >  > > > +             * vCPUs must be known and cannot be altered after the
>  >  > > kernel has
>  >  > > > +             * booted. As a result, the vCPU states at the QOM and ACPI
>  >  > > levels
>  >  > > > +             * might become inconsistent. However, in such cases, the
>  >  > > presence
>  >  > > > +             * of vCPUs has been deliberately simulated at the ACPI
>  >  > > level.
>  >  > > > +             */
>  >  > >
>  >  > > if cpus are not 'simulated', you will not need comments
>  > explaining that  all  > > over place and whole hunk would likely go
>  > away.
>  >  > >
>  >  >
>  >  > I can reduce the content if you wish.
>  >
>  >  you have those comments for a reason since the code does unexpected
>  > 'simulate' thing. Removing that would mask intentional behavior  for a
>  > reader later down the road.
>  
>  
>  Dropped the `acpi_persistent` logic and the related comments.
>  
>  
>  [...]
>  
>  >  > > > diff --git a/include/hw/acpi/cpu.h b/include/hw/acpi/cpu.h  > >
>  > > index 32654dc274..bd3f9973c9 100644  > > > ---
>  > a/include/hw/acpi/cpu.h  > > > +++ b/include/hw/acpi/cpu.h  > > > @@
>  > -26,6 +26,8 @@ typedef struct AcpiCpuStatus {
>  >  > > >      uint64_t arch_id;
>  >  > > >      bool is_inserting;
>  >  > > >      bool is_removing;
>  >  > > > +    bool is_present;
>  >  > > with always_present, it might be better to move field to
>  > CPUHotplugState  > > as it's not per vCPU anymore, and in standby case
>  > state->devs[i].cpu  > > should work as implicit present flag. (see
>  > below wrt doc first comment)  > >  >  > I'm still not convinced of the
>  > stand-by approach because of the  performance  > impact  > it will
>  > have upon increasing the number of possible vCPUs and its worst  >
>  > case is  > unbounded. That's a major problem.
>  >
>  >  # of vCPUs is always bound by host capabilities and by modeled
>  hardware.
>  >
>  >  From guest point of view both approaches should save time as  guest
>  > won't try to online non-present or disabled CPUs  (with large machines
>  > guest boot time usually dwarfs whatever  QEMU does before the guest,
>  > +UEFI adds to insult event more).
>  >  And fast booting large VM (with non present CPUs) initially,  is just
>  > postponing problem the the later time when hotplugging  CPUs causes
>  > worse timing (since guest effectively does stop_machine)  for
>  > 'online'-ing to happen.
>  >
>  >  Regardless, standby CPUs is what ARM folks have chosen to support.
>  >  In this case, I'd pick arch preferred way anytime vs some boot time
>  > improvements.
>  
>  
>  It's unfortunate for all of us but ARM folks are legally binded not to
>  comment on the Qemu code. But Jean-phillipe Brucker from Linaro has
>  indeed gone through the Qemu patches earlier in year 2021.
>  
>  https://op-lists.linaro.org/archives/list/linaro-open-discussions@op-
>  lists.linaro.org/message/X74JS6P2N4AUWHHATJJVVFDI2EMDZJ74/
>  
>  We had tried to solve the problem using just PSCI approach earlier.
>  
>  `Stand-by` CPUs is just a cosmetic term. By having all the vCPUs part of the
>  `possible_cpus_list` and parking disabled vCPUs. We've indeed kept the
>  vCPUs on `stand-by`. This now is the existing approach in the upcoming RFC
>  V6. ACPI CPU state is consistent with QOM vCPUs state.
>  
>  
>  >
>  >  If arm_cpu_realizefn() is still expensive for your case and better
>  > init times are needed, fix it instead of working around it by faking
>  > hotplug on QEMU side and moving issue to the later time.
>  >  That way will benefit not only hotplug case, but also huge VMs case.
>  >  (good example is x86, where it scales well without any hotplug)
>  
>  
>  Sure, agreed. that’s definitely one of the way to improve but it can also be
>  add-on?
>  
>  
>  >
>  >  PS:
>  >  Out of curiosity, I've just fed qemu to perf on Ampere host.
>  >  There are a number of low hanging fruits that would reduce consumed
>  > CPU cycles, for those who wishes to improve performance.
>  >
>  >  Some would lead to overall improvements, other could be  done when
>  > vCPU is disabled.
>  >
>  >  ex: with -enable-kvm -smp 100
>  >     - 26.18% arm_cpu_realizefn
>  >        + 8.86% cpu_reset <- likely is not necessary for disabled
>  > vCPUs, as one  has to call it again when enabling vCPU
>  >        + 4.53% register_cp_regs_for_features
>  >        + 4.32% arm_cpu_register_gdb_regs_for_features  <- do we need
>  > it  when gdbstub is not enabled?
>  >        + 4.09% init_cpreg_list  <- naive refactoring makes it a fraction of
>  percent
>  >        + 3.40% qemu_init_vcpu
>  >          0.59% trace_init_vcpu
>  >
>  >  with above hacks it goes down to 11% (12% for x86_cpu_realizefn).
>  >  However wall-clock wise compared to x86, the arm/virt doesn't scale well.
>  >  So there is something else tgetting in the way (i.e. something stalls
>  > vCPU  creation on ARM?).
>  
>  
>  Thanks for these leads but I will need to spend time in this direction to check
>  and verify these. I'll get back to you regarding your question.
>  
>  
>  >
>  >  And well, I' might be comparing apples to oranges here (old E5-2630v3
>  > vs  some 32core 3.3Ghz Ampere cpu).
>  
>  True.
>  
>  >
>  >  PS2:
>  >  create-gic() is another candidate for improvement, it spends a lot
>  > of time on setting GPIO properties.
>  
>  Yes, indeed. These are very helpful. I will ensure that I look into this
>  direction as well regardless of the vCPU hotplug patches. Maybe we can
>  combine all good things together.
>  
>  
>  >
>  >  > > > +    bool is_enabled;
>  >  > > I'd move introduction of this field into a separate patch.
>  >  > >
>  >  > > BTW: new ABI/fields accessible by guest should be described in  >
>  > > docs/specs/acpi_cpu_hotplug.rst.
>  >  > > It would be better to have the spec as patch 1st, that we all
>  > agree on  > > and then follow with implementation.
>  >  > >
>  >  >
>  >  > We can do that.
>  >  >
>  >  >
>  >  > > And also include there an expected workflow for standby case.
>  >  >
>  >  >
>  >  >
>  >  > We need more discussion on this as our requirements for which we
>  > floated
>  >                                        ^^^^^^^^^^^^^^  > this  >
>  > feature are not being met using stand-by cases.
>  >
>  >  A list of them would be a good starting point for discussion.
>  
>  Sure, agreed. I'll jot them down as part of the main series.
>  
>  >  Perhaps requirements should be made more realistic and be re-
>  evaluated.
>  
>  
>  Requirements are clearly indicated on the cover-letter of the main series.
>  
>  
>  Best regards
>  Salil.