[PATCH v2 0/3] target/loongarch: advertise pv features per vCPU

[PATCH v2 0/3] target/loongarch: advertise pv features per vCPU

[Tao Cui]

From: Tao Cui <cuitao@kylinos.cn>

kvm_set_pv_features() programs the KVM_FEATURE cpucfg attribute, which is a
per-vCPU setting, but it was called from kvm_arch_put_registers() under a
function-local static guard and so ran only once for the whole VM: only the
first vCPU got its pv features pushed to KVM, and on SMP guests the others
never saw KVM_FEATURE_IPI / KVM_FEATURE_STEAL_TIME.

  1/3: drop the static guard and push pv features per vCPU under
       KVM_PUT_FULL_STATE, like kvm_set_stealtime(); host detection stays in
       kvm_arch_init_vcpu().
  2/3: add pv_features (the value of pseudo-register CPUCFG_KVM_FEATURE) to
       the vCPU VMState as a subsection so it travels across migration.
  3/3: add a no_pv_feature compat flag (LoongArchCPU + virt machine class)
       so the new behavior is gated per machine version: 11.1 keeps pv
       advertisement off, the new 11.2 enables it.

Compiled and boot-tested on a loongarch64 KVM host; migration verified
virt-11.2->virt-11.2 and virt-11.1->virt-11.1.

Changes in v2 (per Bibo Mao's review):
- Revert v1's move to kvm_arch_init_vcpu(); keep kvm_set_pv_features() in
  kvm_arch_put_registers(), drop the static guard, push per vCPU under
  KVM_PUT_FULL_STATE.
- Migrate pv_features (CPUCFG_KVM_FEATURE) via a VMState subsection.
- Add no_pv_feature compat flag: 11.1 off, 11.2 on.
- Split into three patches (fix / migrate / compat).

Tao Cui (3):
  target/loongarch/kvm: advertise pv features per vCPU
  target/loongarch: migrate pv_features in the vCPU VMState
  target/loongarch: add no_pv_feature compat flag

 hw/loongarch/virt.c         | 14 +++++++++++++-
 include/hw/loongarch/virt.h |  9 ++++++++-
 target/loongarch/cpu.h      |  1 +
 target/loongarch/kvm/kvm.c  | 20 +++++++++++---------
 target/loongarch/machine.c  | 19 +++++++++++++++++++
 5 files changed, 52 insertions(+), 11 deletions(-)

-- 
2.43.0

[PATCH v2 1/3] target/loongarch/kvm: advertise pv features per vCPU

[Tao Cui]

From: Tao Cui <cuitao@kylinos.cn>

kvm_set_pv_features() programs the KVM_FEATURE cpucfg attribute, which is a
per-vCPU setting. It was called from kvm_arch_put_registers() under a
function-local static guard, so it ran only once for the whole VM: only the
first vCPU got its pv features pushed to KVM, and on SMP guests the others
never saw KVM_FEATURE_IPI / KVM_FEATURE_STEAL_TIME.

Drop the static guard and push pv features per vCPU under
KVM_PUT_FULL_STATE, the same gate kvm_set_stealtime() already uses. Host
feature detection stays in kvm_arch_init_vcpu(); the per-vCPU state write
belongs in kvm_arch_put_registers().

Signed-off-by: Tao Cui <cuitao@kylinos.cn>
---
 target/loongarch/kvm/kvm.c | 15 ++++++---------
 1 file changed, 6 insertions(+), 9 deletions(-)

diff --git a/target/loongarch/kvm/kvm.c b/target/loongarch/kvm/kvm.c
index d6539c12ac..c557ee3c3d 100644
--- a/target/loongarch/kvm/kvm.c
+++ b/target/loongarch/kvm/kvm.c
@@ -816,7 +816,6 @@ int kvm_arch_get_registers(CPUState *cs, Error **errp)
 int kvm_arch_put_registers(CPUState *cs, KvmPutState level, Error **errp)
 {
     int ret;
-    static int once;
 
     ret = kvm_loongarch_put_regs_core(cs);
     if (ret) {
@@ -843,19 +842,17 @@ int kvm_arch_put_registers(CPUState *cs, KvmPutState level, Error **errp)
         return ret;
     }
 
-    if (!once) {
+    if (level >= KVM_PUT_FULL_STATE) {
+        /*
+         * pv_features and steal time are per-vCPU state. Push them on
+         * full-state sync so every vCPU gets its own settings; the kernel
+         * clears the steal-time guest_addr on KVM_PUT_RESET_STATE.
+         */
         ret = kvm_set_pv_features(cs);
         if (ret) {
             return ret;
         }
-        once = 1;
-    }
 
-    if (level >= KVM_PUT_FULL_STATE) {
-        /*
-         * only KVM_PUT_FULL_STATE is required, kvm kernel will clear
-         * guest_addr for KVM_PUT_RESET_STATE
-         */
         ret = kvm_set_stealtime(cs);
         if (ret) {
             return ret;
-- 
2.43.0

[PATCH v2 2/3] target/loongarch: migrate pv_features in the vCPU VMState

[Tao Cui]

From: Tao Cui <cuitao@kylinos.cn>

pv_features holds the value of pseudo-register CPUCFG_KVM_FEATURE and is now
pushed per vCPU. Add it to the vCPU VMState as a subsection (sent when
non-zero) so it travels with the vCPU across migration between host kernel
versions.

Signed-off-by: Tao Cui <cuitao@kylinos.cn>
---
 target/loongarch/machine.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/target/loongarch/machine.c b/target/loongarch/machine.c
index 931a5ca5ba..b8819bd44d 100644
--- a/target/loongarch/machine.c
+++ b/target/loongarch/machine.c
@@ -65,6 +65,24 @@ static const VMStateDescription vmstate_msgint = {
     },
 };
 
+static bool pv_features_needed(void *opaque)
+{
+    LoongArchCPU *cpu = opaque;
+
+    return cpu->env.pv_features != 0;
+}
+
+static const VMStateDescription vmstate_pv_features = {
+    .name = "cpu/pv_features",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .needed = pv_features_needed,
+    .fields = (const VMStateField[]) {
+        VMSTATE_UINT32(env.pv_features, LoongArchCPU),
+        VMSTATE_END_OF_LIST()
+    },
+};
+
 static const VMStateDescription vmstate_lsxh_reg = {
     .name = "lsxh_reg",
     .version_id = 1,
@@ -289,6 +307,7 @@ const VMStateDescription vmstate_loongarch_cpu = {
         &vmstate_lbt,
         &vmstate_msgint,
         &vmstate_pmu,
+        &vmstate_pv_features,
         NULL
     }
 };
-- 
2.43.0

[PATCH v2 3/3] target/loongarch: add no_pv_feature compat flag

[Tao Cui]

From: Tao Cui <cuitao@kylinos.cn>

Advertising pv features to the guest affects the migration stream. Add a
no_pv_feature compat flag so the behavior can be gated per machine version:
LoongArchCPU gets a bool field, and the virt machine gains a
LoongArchVirtMachineClass field that defaults the CPU flag per version.

kvm_cpu_check_pv_features() skips populating pv features when the flag is
set. The 11.1 machine type keeps pv advertisement off (no_pv_feature is
true, preserving the previous behavior); from the new 11.2 machine type it
is on by default.

Signed-off-by: Tao Cui <cuitao@kylinos.cn>
---
 hw/loongarch/virt.c         | 14 +++++++++++++-
 include/hw/loongarch/virt.h |  9 ++++++++-
 target/loongarch/cpu.h      |  1 +
 target/loongarch/kvm/kvm.c  |  5 +++++
 4 files changed, 27 insertions(+), 2 deletions(-)

diff --git a/hw/loongarch/virt.c b/hw/loongarch/virt.c
index 6693dea647..53c84c8def 100644
--- a/hw/loongarch/virt.c
+++ b/hw/loongarch/virt.c
@@ -955,6 +955,8 @@ static void virt_init(MachineState *machine)
                          machine->cpu_type);
             exit(EXIT_FAILURE);
         }
+        LOONGARCH_CPU(cpuobj)->no_pv_feature =
+            LOONGARCH_VIRT_MACHINE_GET_CLASS(lvms)->no_pv_feature;
         qdev_realize_and_unref(DEVICE(cpuobj), NULL, &error_fatal);
     }
     virt_check_dmsi(machine);
@@ -1584,6 +1586,7 @@ static const TypeInfo virt_machine_info = {
     .parent         = TYPE_MACHINE,
     .abstract       = true,
     .instance_size  = sizeof(LoongArchVirtMachineState),
+    .class_size     = sizeof(LoongArchVirtMachineClass),
     .class_init     = virt_class_init,
     .instance_init  = virt_initfn,
     .instance_finalize = virt_instance_finalize,
@@ -1601,6 +1604,15 @@ static void machvirt_machine_init(void)
 type_init(machvirt_machine_init);
 
 static void virt_machine_11_1_options(MachineClass *mc)
+{
+    LoongArchVirtMachineClass *vmc = LOONGARCH_VIRT_MACHINE_CLASS(mc);
+
+    /* pv feature advertisement is enabled from 11.2; keep 11.1 unchanged */
+    vmc->no_pv_feature = true;
+}
+DEFINE_VIRT_MACHINE(11, 1)
+
+static void virt_machine_11_2_options(MachineClass *mc)
 {
 }
-DEFINE_VIRT_MACHINE_AS_LATEST(11, 1)
+DEFINE_VIRT_MACHINE_AS_LATEST(11, 2)
diff --git a/include/hw/loongarch/virt.h b/include/hw/loongarch/virt.h
index d39a9bbf5d..cf8d9e1fd2 100644
--- a/include/hw/loongarch/virt.h
+++ b/include/hw/loongarch/virt.h
@@ -130,8 +130,15 @@ struct LoongArchVirtMachineState {
     bool highmem_mmio;
 };
 
+struct LoongArchVirtMachineClass {
+    MachineClass parent_obj;
+
+    bool no_pv_feature;   /* compat: 11.1 and older do not advertise pv features */
+};
+
 #define TYPE_LOONGARCH_VIRT_MACHINE  MACHINE_TYPE_NAME("virt")
-OBJECT_DECLARE_SIMPLE_TYPE(LoongArchVirtMachineState, LOONGARCH_VIRT_MACHINE)
+OBJECT_DECLARE_TYPE(LoongArchVirtMachineState, LoongArchVirtMachineClass,
+                    LOONGARCH_VIRT_MACHINE)
 void virt_acpi_setup(LoongArchVirtMachineState *lvms);
 void virt_fdt_setup(LoongArchVirtMachineState *lvms);
 
diff --git a/target/loongarch/cpu.h b/target/loongarch/cpu.h
index ad30c73167..11bbb5f36e 100644
--- a/target/loongarch/cpu.h
+++ b/target/loongarch/cpu.h
@@ -447,6 +447,7 @@ struct ArchCPU {
     OnOffAuto msgint;
     OnOffAuto kvm_pv_ipi;
     OnOffAuto kvm_steal_time;
+    bool no_pv_feature;   /* compat: do not advertise KVM pv features */
     int32_t socket_id;  /* socket-id of this CPU */
     int32_t core_id;    /* core-id of this CPU */
     int32_t thread_id;  /* thread-id of this CPU */
diff --git a/target/loongarch/kvm/kvm.c b/target/loongarch/kvm/kvm.c
index c557ee3c3d..878914928b 100644
--- a/target/loongarch/kvm/kvm.c
+++ b/target/loongarch/kvm/kvm.c
@@ -1119,6 +1119,11 @@ static int kvm_cpu_check_pv_features(CPUState *cs, Error **errp)
     CPULoongArchState *env = cpu_env(cs);
     bool kvm_supported;
 
+    /* compat: older machine types do not advertise pv features to the guest */
+    if (cpu->no_pv_feature) {
+        return 0;
+    }
+
     kvm_supported = kvm_feature_supported(cs, LOONGARCH_FEATURE_PV_IPI);
     if (cpu->kvm_pv_ipi == ON_OFF_AUTO_ON) {
         if (!kvm_supported) {
-- 
2.43.0

Re: [PATCH v2 1/3] target/loongarch/kvm: advertise pv features per vCPU

[Bibo Mao]

On 2026/7/10 上午8:50, Tao Cui wrote:
> From: Tao Cui <cuitao@kylinos.cn>
> 
> kvm_set_pv_features() programs the KVM_FEATURE cpucfg attribute, which is a
> per-vCPU setting. It was called from kvm_arch_put_registers() under a
> function-local static guard, so it ran only once for the whole VM: only the
> first vCPU got its pv features pushed to KVM, and on SMP guests the others
> never saw KVM_FEATURE_IPI / KVM_FEATURE_STEAL_TIME.
> 
> Drop the static guard and push pv features per vCPU under
> KVM_PUT_FULL_STATE, the same gate kvm_set_stealtime() already uses. Host
> feature detection stays in kvm_arch_init_vcpu(); the per-vCPU state write
> belongs in kvm_arch_put_registers().
> 
> Signed-off-by: Tao Cui <cuitao@kylinos.cn>
> ---
>   target/loongarch/kvm/kvm.c | 15 ++++++---------
>   1 file changed, 6 insertions(+), 9 deletions(-)
> 
> diff --git a/target/loongarch/kvm/kvm.c b/target/loongarch/kvm/kvm.c
> index d6539c12ac..c557ee3c3d 100644
> --- a/target/loongarch/kvm/kvm.c
> +++ b/target/loongarch/kvm/kvm.c
> @@ -816,7 +816,6 @@ int kvm_arch_get_registers(CPUState *cs, Error **errp)
>   int kvm_arch_put_registers(CPUState *cs, KvmPutState level, Error **errp)
>   {
>       int ret;
> -    static int once;
>   
>       ret = kvm_loongarch_put_regs_core(cs);
>       if (ret) {
> @@ -843,19 +842,17 @@ int kvm_arch_put_registers(CPUState *cs, KvmPutState level, Error **errp)
>           return ret;
>       }
>   
> -    if (!once) {
> +    if (level >= KVM_PUT_FULL_STATE) {
> +        /*
> +         * pv_features and steal time are per-vCPU state. Push them on
> +         * full-state sync so every vCPU gets its own settings; the kernel
> +         * clears the steal-time guest_addr on KVM_PUT_RESET_STATE.
> +         */
>           ret = kvm_set_pv_features(cs);
pv feature is a little different from steal-time. steal-time guest_addr 
is created from guest OS, pv feature is created from VMM at beginning. 
steal-time guest_addr can be set for many times, and there is bit 
KVM_STEAL_PHYS_VALID checking with steal-time guest_addr, however pv 
feature can be set only once with existing method.

Although I do not understand flow of VM migration, with 
KVM_PUT_FULL_STATE state changing, there are at least two places where 
this state is set, one is from cpu_common_realizefn() which calls 
cpu_synchronize_post_init(), the other is 
qemu_loadvm_state()/qemu_loadvm_state_main() which calls 
cpu_synchronize_all_post_init().

It seems that VM will fail to migrate since kvm_set_pv_features is 
called twice at least here. Do you test VM migration with this patch?

Regards
Bibo Mao
>           if (ret) {
>               return ret;
>           }
> -        once = 1;
> -    }
>   
> -    if (level >= KVM_PUT_FULL_STATE) {
> -        /*
> -         * only KVM_PUT_FULL_STATE is required, kvm kernel will clear
> -         * guest_addr for KVM_PUT_RESET_STATE
> -         */
>           ret = kvm_set_stealtime(cs);
>           if (ret) {
>               return ret;
> 

Re: [PATCH v2 2/3] target/loongarch: migrate pv_features in the vCPU VMState

[Bibo Mao]

On 2026/7/10 上午8:50, Tao Cui wrote:
> From: Tao Cui <cuitao@kylinos.cn>
> 
> pv_features holds the value of pseudo-register CPUCFG_KVM_FEATURE and is now
> pushed per vCPU. Add it to the vCPU VMState as a subsection (sent when
> non-zero) so it travels with the vCPU across migration between host kernel
> versions.
> 
> Signed-off-by: Tao Cui <cuitao@kylinos.cn>
> ---
>   target/loongarch/machine.c | 19 +++++++++++++++++++
>   1 file changed, 19 insertions(+)
> 
> diff --git a/target/loongarch/machine.c b/target/loongarch/machine.c
> index 931a5ca5ba..b8819bd44d 100644
> --- a/target/loongarch/machine.c
> +++ b/target/loongarch/machine.c
> @@ -65,6 +65,24 @@ static const VMStateDescription vmstate_msgint = {
>       },
>   };
>   
> +static bool pv_features_needed(void *opaque)
> +{
> +    LoongArchCPU *cpu = opaque;
> +
> +    return cpu->env.pv_features != 0;
I think cpu->no_pv_feature should be check here. For QEMU 11.1 
pv_feature is not supported for migration, for later version it 
supports, something like this:
        return !cpu->no_pv_feature;

Regards
Bibo Mao
> +}
> +
> +static const VMStateDescription vmstate_pv_features = {
> +    .name = "cpu/pv_features",
> +    .version_id = 1,
> +    .minimum_version_id = 1,
> +    .needed = pv_features_needed,
> +    .fields = (const VMStateField[]) {
> +        VMSTATE_UINT32(env.pv_features, LoongArchCPU),
> +        VMSTATE_END_OF_LIST()
> +    },
> +};
> +
>   static const VMStateDescription vmstate_lsxh_reg = {
>       .name = "lsxh_reg",
>       .version_id = 1,
> @@ -289,6 +307,7 @@ const VMStateDescription vmstate_loongarch_cpu = {
>           &vmstate_lbt,
>           &vmstate_msgint,
>           &vmstate_pmu,
> +        &vmstate_pv_features,
>           NULL
>       }
>   };
> 

Re: [PATCH v2 3/3] target/loongarch: add no_pv_feature compat flag

[Bibo Mao]

On 2026/7/10 上午8:50, Tao Cui wrote:
> From: Tao Cui <cuitao@kylinos.cn>
> 
> Advertising pv features to the guest affects the migration stream. Add a
> no_pv_feature compat flag so the behavior can be gated per machine version:
> LoongArchCPU gets a bool field, and the virt machine gains a
> LoongArchVirtMachineClass field that defaults the CPU flag per version.
> 
> kvm_cpu_check_pv_features() skips populating pv features when the flag is
> set. The 11.1 machine type keeps pv advertisement off (no_pv_feature is
> true, preserving the previous behavior); from the new 11.2 machine type it
> is on by default.
> 
> Signed-off-by: Tao Cui <cuitao@kylinos.cn>
> ---
>   hw/loongarch/virt.c         | 14 +++++++++++++-
>   include/hw/loongarch/virt.h |  9 ++++++++-
>   target/loongarch/cpu.h      |  1 +
>   target/loongarch/kvm/kvm.c  |  5 +++++
>   4 files changed, 27 insertions(+), 2 deletions(-)
> 
> diff --git a/hw/loongarch/virt.c b/hw/loongarch/virt.c
> index 6693dea647..53c84c8def 100644
> --- a/hw/loongarch/virt.c
> +++ b/hw/loongarch/virt.c
> @@ -955,6 +955,8 @@ static void virt_init(MachineState *machine)
>                            machine->cpu_type);
>               exit(EXIT_FAILURE);
>           }
> +        LOONGARCH_CPU(cpuobj)->no_pv_feature =
> +            LOONGARCH_VIRT_MACHINE_GET_CLASS(lvms)->no_pv_feature;
I think it will better to set in function virt_cpu_plug().
>           qdev_realize_and_unref(DEVICE(cpuobj), NULL, &error_fatal);
>       }
>       virt_check_dmsi(machine);
> @@ -1584,6 +1586,7 @@ static const TypeInfo virt_machine_info = {
>       .parent         = TYPE_MACHINE,
>       .abstract       = true,
>       .instance_size  = sizeof(LoongArchVirtMachineState),
> +    .class_size     = sizeof(LoongArchVirtMachineClass),
>       .class_init     = virt_class_init,
>       .instance_init  = virt_initfn,
>       .instance_finalize = virt_instance_finalize,
> @@ -1601,6 +1604,15 @@ static void machvirt_machine_init(void)
>   type_init(machvirt_machine_init);
>   
>   static void virt_machine_11_1_options(MachineClass *mc)
> +{
> +    LoongArchVirtMachineClass *vmc = LOONGARCH_VIRT_MACHINE_CLASS(mc);
> +
> +    /* pv feature advertisement is enabled from 11.2; keep 11.1 unchanged */
> +    vmc->no_pv_feature = true;
> +}
> +DEFINE_VIRT_MACHINE(11, 1)
> +
> +static void virt_machine_11_2_options(MachineClass *mc)
>   {
>   }
> -DEFINE_VIRT_MACHINE_AS_LATEST(11, 1)
> +DEFINE_VIRT_MACHINE_AS_LATEST(11, 2)
> diff --git a/include/hw/loongarch/virt.h b/include/hw/loongarch/virt.h
> index d39a9bbf5d..cf8d9e1fd2 100644
> --- a/include/hw/loongarch/virt.h
> +++ b/include/hw/loongarch/virt.h
> @@ -130,8 +130,15 @@ struct LoongArchVirtMachineState {
>       bool highmem_mmio;
>   };
>   
> +struct LoongArchVirtMachineClass {
> +    MachineClass parent_obj;
> +
> +    bool no_pv_feature;   /* compat: 11.1 and older do not advertise pv features */
> +};
> +
>   #define TYPE_LOONGARCH_VIRT_MACHINE  MACHINE_TYPE_NAME("virt")
> -OBJECT_DECLARE_SIMPLE_TYPE(LoongArchVirtMachineState, LOONGARCH_VIRT_MACHINE)
> +OBJECT_DECLARE_TYPE(LoongArchVirtMachineState, LoongArchVirtMachineClass,
> +                    LOONGARCH_VIRT_MACHINE)
>   void virt_acpi_setup(LoongArchVirtMachineState *lvms);
>   void virt_fdt_setup(LoongArchVirtMachineState *lvms);
>   
> diff --git a/target/loongarch/cpu.h b/target/loongarch/cpu.h
> index ad30c73167..11bbb5f36e 100644
> --- a/target/loongarch/cpu.h
> +++ b/target/loongarch/cpu.h
> @@ -447,6 +447,7 @@ struct ArchCPU {
>       OnOffAuto msgint;
>       OnOffAuto kvm_pv_ipi;
>       OnOffAuto kvm_steal_time;
> +    bool no_pv_feature;   /* compat: do not advertise KVM pv features */
>       int32_t socket_id;  /* socket-id of this CPU */
>       int32_t core_id;    /* core-id of this CPU */
>       int32_t thread_id;  /* thread-id of this CPU */
> diff --git a/target/loongarch/kvm/kvm.c b/target/loongarch/kvm/kvm.c
> index c557ee3c3d..878914928b 100644
> --- a/target/loongarch/kvm/kvm.c
> +++ b/target/loongarch/kvm/kvm.c
> @@ -1119,6 +1119,11 @@ static int kvm_cpu_check_pv_features(CPUState *cs, Error **errp)
>       CPULoongArchState *env = cpu_env(cs);
>       bool kvm_supported;
>   
> +    /* compat: older machine types do not advertise pv features to the guest */
> +    if (cpu->no_pv_feature) {
> +        return 0;
> +    }
> +
This piece of code is unnecessary, with QEMU-11.1 machine type, 
kvm_cpu_check_pv_features checking is skipped when CPU is created, why?

Regards
Bibo Mao
>       kvm_supported = kvm_feature_supported(cs, LOONGARCH_FEATURE_PV_IPI);
>       if (cpu->kvm_pv_ipi == ON_OFF_AUTO_ON) {
>           if (!kvm_supported) {
> 

Re: [PATCH v2 2/3] target/loongarch: migrate pv_features in the vCPU VMState

[Bibo Mao]

On 2026/7/10 上午11:02, Bibo Mao wrote:
> 
> 
> On 2026/7/10 上午8:50, Tao Cui wrote:
>> From: Tao Cui <cuitao@kylinos.cn>
>>
>> pv_features holds the value of pseudo-register CPUCFG_KVM_FEATURE and 
>> is now
>> pushed per vCPU. Add it to the vCPU VMState as a subsection (sent when
>> non-zero) so it travels with the vCPU across migration between host 
>> kernel
>> versions.
>>
>> Signed-off-by: Tao Cui <cuitao@kylinos.cn>
>> ---
>>   target/loongarch/machine.c | 19 +++++++++++++++++++
>>   1 file changed, 19 insertions(+)
>>
>> diff --git a/target/loongarch/machine.c b/target/loongarch/machine.c
>> index 931a5ca5ba..b8819bd44d 100644
>> --- a/target/loongarch/machine.c
>> +++ b/target/loongarch/machine.c
>> @@ -65,6 +65,24 @@ static const VMStateDescription vmstate_msgint = {
>>       },
>>   };
>> +static bool pv_features_needed(void *opaque)
>> +{
>> +    LoongArchCPU *cpu = opaque;
>> +
>> +    return cpu->env.pv_features != 0;
> I think cpu->no_pv_feature should be check here. For QEMU 11.1 
> pv_feature is not supported for migration, for later version it 
> supports, something like this:
>         return !cpu->no_pv_feature;
For successful migration, env.pv_features from source host should be 
subset of destination host, there should be such checking also.

> 
> Regards
> Bibo Mao
>> +}
>> +
>> +static const VMStateDescription vmstate_pv_features = {
>> +    .name = "cpu/pv_features",
>> +    .version_id = 1,
>> +    .minimum_version_id = 1,
>> +    .needed = pv_features_needed,
>> +    .fields = (const VMStateField[]) {
>> +        VMSTATE_UINT32(env.pv_features, LoongArchCPU),
>> +        VMSTATE_END_OF_LIST()
>> +    },
>> +};
>> +
>>   static const VMStateDescription vmstate_lsxh_reg = {
>>       .name = "lsxh_reg",
>>       .version_id = 1,
>> @@ -289,6 +307,7 @@ const VMStateDescription vmstate_loongarch_cpu = {
>>           &vmstate_lbt,
>>           &vmstate_msgint,
>>           &vmstate_pmu,
>> +        &vmstate_pv_features,
>>           NULL
>>       }
>>   };
>>
> 

Re: [PATCH v2 1/3] target/loongarch/kvm: advertise pv features per vCPU

[Tao Cui]

在 2026/7/10 10:58, Bibo Mao 写道:
> 
> 
> On 2026/7/10 上午8:50, Tao Cui wrote:
>> From: Tao Cui <cuitao@kylinos.cn>
>>
>> kvm_set_pv_features() programs the KVM_FEATURE cpucfg attribute, which is a
>> per-vCPU setting. It was called from kvm_arch_put_registers() under a
>> function-local static guard, so it ran only once for the whole VM: only the
>> first vCPU got its pv features pushed to KVM, and on SMP guests the others
>> never saw KVM_FEATURE_IPI / KVM_FEATURE_STEAL_TIME.
>>
>> Drop the static guard and push pv features per vCPU under
>> KVM_PUT_FULL_STATE, the same gate kvm_set_stealtime() already uses. Host
>> feature detection stays in kvm_arch_init_vcpu(); the per-vCPU state write
>> belongs in kvm_arch_put_registers().
>>
>> Signed-off-by: Tao Cui <cuitao@kylinos.cn>
>> ---
>>   target/loongarch/kvm/kvm.c | 15 ++++++---------
>>   1 file changed, 6 insertions(+), 9 deletions(-)
>>
>> diff --git a/target/loongarch/kvm/kvm.c b/target/loongarch/kvm/kvm.c
>> index d6539c12ac..c557ee3c3d 100644
>> --- a/target/loongarch/kvm/kvm.c
>> +++ b/target/loongarch/kvm/kvm.c
>> @@ -816,7 +816,6 @@ int kvm_arch_get_registers(CPUState *cs, Error **errp)
>>   int kvm_arch_put_registers(CPUState *cs, KvmPutState level, Error **errp)
>>   {
>>       int ret;
>> -    static int once;
>>         ret = kvm_loongarch_put_regs_core(cs);
>>       if (ret) {
>> @@ -843,19 +842,17 @@ int kvm_arch_put_registers(CPUState *cs, KvmPutState level, Error **errp)
>>           return ret;
>>       }
>>   -    if (!once) {
>> +    if (level >= KVM_PUT_FULL_STATE) {
>> +        /*
>> +         * pv_features and steal time are per-vCPU state. Push them on
>> +         * full-state sync so every vCPU gets its own settings; the kernel
>> +         * clears the steal-time guest_addr on KVM_PUT_RESET_STATE.
>> +         */
>>           ret = kvm_set_pv_features(cs);
> pv feature is a little different from steal-time. steal-time guest_addr is created from guest OS, pv feature is created from VMM at beginning. steal-time guest_addr can be set for many times, and there is bit KVM_STEAL_PHYS_VALID checking with steal-time guest_addr, however pv feature can be set only once with existing method.
> 

Hi Bibo,

Thanks for catching this — I hadn't fully considered the double-call path.

KVM_PUT_FULL_STATE fires both at realize (cpu_synchronize_post_init) and
on incoming migration load (cpu_synchronize_all_post_init), so
kvm_set_pv_features() runs twice on the destination.

> Although I do not understand flow of VM migration, with KVM_PUT_FULL_STATE state changing, there are at least two places where this state is set, one is from cpu_common_realizefn() which calls cpu_synchronize_post_init(), the other is qemu_loadvm_state()/qemu_loadvm_state_main() which calls cpu_synchronize_all_post_init().
> 
> It seems that VM will fail to migrate since kvm_set_pv_features is called twice at least here. Do you test VM migration with this patch?

I did test migration (virt-11.2 -> virt-11.2, virt-11.1 -> virt-11.1) and
it passed, but that was on a single host — source and destination computed
the same pv_features. The kernel (kvm_loongarch_cpucfg_set_attr) only
rejects a re-set when the value differs:

    if ((kvm->arch.pv_features & LOONGARCH_PV_FEAT_UPDATED) &&
        ((kvm->arch.pv_features & valid) != val))
        return -EINVAL;

So a cross-host migration where the two sides compute different pv_features
would indeed fail on the second set.

I'll add a per-vCPU guard so the push happens exactly once (at the first
FULL_STATE sync); subsequent syncs are skipped and the destination keeps
advertising the features its own host supports. Does that sound like the
right direction?

Thanks,
Tao

> 
> Regards
> Bibo Mao
>>           if (ret) {
>>               return ret;
>>           }
>> -        once = 1;
>> -    }
>>   -    if (level >= KVM_PUT_FULL_STATE) {
>> -        /*
>> -         * only KVM_PUT_FULL_STATE is required, kvm kernel will clear
>> -         * guest_addr for KVM_PUT_RESET_STATE
>> -         */
>>           ret = kvm_set_stealtime(cs);
>>           if (ret) {
>>               return ret;
>>
> 

Re: [PATCH v2 2/3] target/loongarch: migrate pv_features in the vCPU VMState

[Tao Cui]

在 2026/7/10 11:15, Bibo Mao 写道:
> 
> 
> On 2026/7/10 上午11:02, Bibo Mao wrote:
>>
>>
>> On 2026/7/10 上午8:50, Tao Cui wrote:
>>> From: Tao Cui <cuitao@kylinos.cn>
>>>
>>> pv_features holds the value of pseudo-register CPUCFG_KVM_FEATURE and is now
>>> pushed per vCPU. Add it to the vCPU VMState as a subsection (sent when
>>> non-zero) so it travels with the vCPU across migration between host kernel
>>> versions.
>>>
>>> Signed-off-by: Tao Cui <cuitao@kylinos.cn>
>>> ---
>>>   target/loongarch/machine.c | 19 +++++++++++++++++++
>>>   1 file changed, 19 insertions(+)
>>>
>>> diff --git a/target/loongarch/machine.c b/target/loongarch/machine.c
>>> index 931a5ca5ba..b8819bd44d 100644
>>> --- a/target/loongarch/machine.c
>>> +++ b/target/loongarch/machine.c
>>> @@ -65,6 +65,24 @@ static const VMStateDescription vmstate_msgint = {
>>>       },
>>>   };
>>> +static bool pv_features_needed(void *opaque)
>>> +{
>>> +    LoongArchCPU *cpu = opaque;
>>> +
>>> +    return cpu->env.pv_features != 0;
>> I think cpu->no_pv_feature should be check here. For QEMU 11.1 pv_feature is not supported for migration, for later version it supports, something like this:
>>         return !cpu->no_pv_feature;
> For successful migration, env.pv_features from source host should be subset of destination host, there should be such checking also.
> 
Agreed on both points.

1. I'll change .needed to `return !cpu->no_pv_feature` so the subsection
   is gated by machine version. Since it references no_pv_feature (added
   in the compat patch), I'll reorder so compat comes before vmstate.

2. For the subset check, I'll add a .post_load that calls
   kvm_feature_supported() for each PV feature to build the destination's
   supported mask, and rejects migration if the migrated pv_features has
   bits outside it. Does this look right?

Thanks,
Tao
>>
>> Regards
>> Bibo Mao
>>> +}
>>> +
>>> +static const VMStateDescription vmstate_pv_features = {
>>> +    .name = "cpu/pv_features",
>>> +    .version_id = 1,
>>> +    .minimum_version_id = 1,
>>> +    .needed = pv_features_needed,
>>> +    .fields = (const VMStateField[]) {
>>> +        VMSTATE_UINT32(env.pv_features, LoongArchCPU),
>>> +        VMSTATE_END_OF_LIST()
>>> +    },
>>> +};
>>> +
>>>   static const VMStateDescription vmstate_lsxh_reg = {
>>>       .name = "lsxh_reg",
>>>       .version_id = 1,
>>> @@ -289,6 +307,7 @@ const VMStateDescription vmstate_loongarch_cpu = {
>>>           &vmstate_lbt,
>>>           &vmstate_msgint,
>>>           &vmstate_pmu,
>>> +        &vmstate_pv_features,
>>>           NULL
>>>       }
>>>   };
>>>
>>
> 

Re: [PATCH v2 3/3] target/loongarch: add no_pv_feature compat flag

[Tao Cui]

在 2026/7/10 11:09, Bibo Mao 写道:
> 
> 
> On 2026/7/10 上午8:50, Tao Cui wrote:
>> From: Tao Cui <cuitao@kylinos.cn>
>>
>> Advertising pv features to the guest affects the migration stream. Add a
>> no_pv_feature compat flag so the behavior can be gated per machine version:
>> LoongArchCPU gets a bool field, and the virt machine gains a
>> LoongArchVirtMachineClass field that defaults the CPU flag per version.
>>
>> kvm_cpu_check_pv_features() skips populating pv features when the flag is
>> set. The 11.1 machine type keeps pv advertisement off (no_pv_feature is
>> true, preserving the previous behavior); from the new 11.2 machine type it
>> is on by default.
>>
>> Signed-off-by: Tao Cui <cuitao@kylinos.cn>
>> ---
>>   hw/loongarch/virt.c         | 14 +++++++++++++-
>>   include/hw/loongarch/virt.h |  9 ++++++++-
>>   target/loongarch/cpu.h      |  1 +
>>   target/loongarch/kvm/kvm.c  |  5 +++++
>>   4 files changed, 27 insertions(+), 2 deletions(-)
>>
>> diff --git a/hw/loongarch/virt.c b/hw/loongarch/virt.c
>> index 6693dea647..53c84c8def 100644
>> --- a/hw/loongarch/virt.c
>> +++ b/hw/loongarch/virt.c
>> @@ -955,6 +955,8 @@ static void virt_init(MachineState *machine)
>>                            machine->cpu_type);
>>               exit(EXIT_FAILURE);
>>           }
>> +        LOONGARCH_CPU(cpuobj)->no_pv_feature =
>> +            LOONGARCH_VIRT_MACHINE_GET_CLASS(lvms)->no_pv_feature;
> I think it will better to set in function virt_cpu_plug().
On virt_cpu_plug(): 
  agreed, that's cleaner — I'll move the propagation there.
>>           qdev_realize_and_unref(DEVICE(cpuobj), NULL, &error_fatal);
>>       }
>>       virt_check_dmsi(machine);
>> @@ -1584,6 +1586,7 @@ static const TypeInfo virt_machine_info = {
>>       .parent         = TYPE_MACHINE,
>>       .abstract       = true,
>>       .instance_size  = sizeof(LoongArchVirtMachineState),
>> +    .class_size     = sizeof(LoongArchVirtMachineClass),
>>       .class_init     = virt_class_init,
>>       .instance_init  = virt_initfn,
>>       .instance_finalize = virt_instance_finalize,
>> @@ -1601,6 +1604,15 @@ static void machvirt_machine_init(void)
>>   type_init(machvirt_machine_init);
>>     static void virt_machine_11_1_options(MachineClass *mc)
>> +{
>> +    LoongArchVirtMachineClass *vmc = LOONGARCH_VIRT_MACHINE_CLASS(mc);
>> +
>> +    /* pv feature advertisement is enabled from 11.2; keep 11.1 unchanged */
>> +    vmc->no_pv_feature = true;
>> +}
>> +DEFINE_VIRT_MACHINE(11, 1)
>> +
>> +static void virt_machine_11_2_options(MachineClass *mc)
>>   {
>>   }
>> -DEFINE_VIRT_MACHINE_AS_LATEST(11, 1)
>> +DEFINE_VIRT_MACHINE_AS_LATEST(11, 2)
>> diff --git a/include/hw/loongarch/virt.h b/include/hw/loongarch/virt.h
>> index d39a9bbf5d..cf8d9e1fd2 100644
>> --- a/include/hw/loongarch/virt.h
>> +++ b/include/hw/loongarch/virt.h
>> @@ -130,8 +130,15 @@ struct LoongArchVirtMachineState {
>>       bool highmem_mmio;
>>   };
>>   +struct LoongArchVirtMachineClass {
>> +    MachineClass parent_obj;
>> +
>> +    bool no_pv_feature;   /* compat: 11.1 and older do not advertise pv features */
>> +};
>> +
>>   #define TYPE_LOONGARCH_VIRT_MACHINE  MACHINE_TYPE_NAME("virt")
>> -OBJECT_DECLARE_SIMPLE_TYPE(LoongArchVirtMachineState, LOONGARCH_VIRT_MACHINE)
>> +OBJECT_DECLARE_TYPE(LoongArchVirtMachineState, LoongArchVirtMachineClass,
>> +                    LOONGARCH_VIRT_MACHINE)
>>   void virt_acpi_setup(LoongArchVirtMachineState *lvms);
>>   void virt_fdt_setup(LoongArchVirtMachineState *lvms);
>>   diff --git a/target/loongarch/cpu.h b/target/loongarch/cpu.h
>> index ad30c73167..11bbb5f36e 100644
>> --- a/target/loongarch/cpu.h
>> +++ b/target/loongarch/cpu.h
>> @@ -447,6 +447,7 @@ struct ArchCPU {
>>       OnOffAuto msgint;
>>       OnOffAuto kvm_pv_ipi;
>>       OnOffAuto kvm_steal_time;
>> +    bool no_pv_feature;   /* compat: do not advertise KVM pv features */
>>       int32_t socket_id;  /* socket-id of this CPU */
>>       int32_t core_id;    /* core-id of this CPU */
>>       int32_t thread_id;  /* thread-id of this CPU */
>> diff --git a/target/loongarch/kvm/kvm.c b/target/loongarch/kvm/kvm.c
>> index c557ee3c3d..878914928b 100644
>> --- a/target/loongarch/kvm/kvm.c
>> +++ b/target/loongarch/kvm/kvm.c
>> @@ -1119,6 +1119,11 @@ static int kvm_cpu_check_pv_features(CPUState *cs, Error **errp)
>>       CPULoongArchState *env = cpu_env(cs);
>>       bool kvm_supported;
>>   +    /* compat: older machine types do not advertise pv features to the guest */
>> +    if (cpu->no_pv_feature) {
>> +        return 0;
>> +    }
>> +
> This piece of code is unnecessary, with QEMU-11.1 machine type, kvm_cpu_check_pv_features checking is skipped when CPU is created, why?

On the check_pv_features guard: 
you're right, it's unnecessary. As you
pointed out in v1, check_pv_features() is host detection and should run
regardless of machine version. I'll remove the guard and check
no_pv_feature in kvm_set_pv_features() instead, so the push is skipped for
11.1 while detection still happens normally.

One timing concern: 
virt_cpu_plug() runs after qdev_realize(), but the
first FULL_STATE sync (post_init) fires during realize — so the push could
happen before no_pv_feature is set in plug. Should I keep setting the flag
before qdev_realize() (in virt_init) for now, or is there a better
mechanism?

> 
> Regards
> Bibo Mao
>>       kvm_supported = kvm_feature_supported(cs, LOONGARCH_FEATURE_PV_IPI);
>>       if (cpu->kvm_pv_ipi == ON_OFF_AUTO_ON) {
>>           if (!kvm_supported) {
>>
> 

Re: [PATCH v2 1/3] target/loongarch/kvm: advertise pv features per vCPU

[Bibo Mao]

On 2026/7/10 下午4:09, Tao Cui wrote:
> 
> 
> 在 2026/7/10 10:58, Bibo Mao 写道:
>>
>>
>> On 2026/7/10 上午8:50, Tao Cui wrote:
>>> From: Tao Cui <cuitao@kylinos.cn>
>>>
>>> kvm_set_pv_features() programs the KVM_FEATURE cpucfg attribute, which is a
>>> per-vCPU setting. It was called from kvm_arch_put_registers() under a
>>> function-local static guard, so it ran only once for the whole VM: only the
>>> first vCPU got its pv features pushed to KVM, and on SMP guests the others
>>> never saw KVM_FEATURE_IPI / KVM_FEATURE_STEAL_TIME.
>>>
>>> Drop the static guard and push pv features per vCPU under
>>> KVM_PUT_FULL_STATE, the same gate kvm_set_stealtime() already uses. Host
>>> feature detection stays in kvm_arch_init_vcpu(); the per-vCPU state write
>>> belongs in kvm_arch_put_registers().
>>>
>>> Signed-off-by: Tao Cui <cuitao@kylinos.cn>
>>> ---
>>>    target/loongarch/kvm/kvm.c | 15 ++++++---------
>>>    1 file changed, 6 insertions(+), 9 deletions(-)
>>>
>>> diff --git a/target/loongarch/kvm/kvm.c b/target/loongarch/kvm/kvm.c
>>> index d6539c12ac..c557ee3c3d 100644
>>> --- a/target/loongarch/kvm/kvm.c
>>> +++ b/target/loongarch/kvm/kvm.c
>>> @@ -816,7 +816,6 @@ int kvm_arch_get_registers(CPUState *cs, Error **errp)
>>>    int kvm_arch_put_registers(CPUState *cs, KvmPutState level, Error **errp)
>>>    {
>>>        int ret;
>>> -    static int once;
>>>          ret = kvm_loongarch_put_regs_core(cs);
>>>        if (ret) {
>>> @@ -843,19 +842,17 @@ int kvm_arch_put_registers(CPUState *cs, KvmPutState level, Error **errp)
>>>            return ret;
>>>        }
>>>    -    if (!once) {
>>> +    if (level >= KVM_PUT_FULL_STATE) {
>>> +        /*
>>> +         * pv_features and steal time are per-vCPU state. Push them on
>>> +         * full-state sync so every vCPU gets its own settings; the kernel
>>> +         * clears the steal-time guest_addr on KVM_PUT_RESET_STATE.
>>> +         */
>>>            ret = kvm_set_pv_features(cs);
>> pv feature is a little different from steal-time. steal-time guest_addr is created from guest OS, pv feature is created from VMM at beginning. steal-time guest_addr can be set for many times, and there is bit KVM_STEAL_PHYS_VALID checking with steal-time guest_addr, however pv feature can be set only once with existing method.
>>
> 
> Hi Bibo,
> 
> Thanks for catching this — I hadn't fully considered the double-call path.
> 
> KVM_PUT_FULL_STATE fires both at realize (cpu_synchronize_post_init) and
> on incoming migration load (cpu_synchronize_all_post_init), so
> kvm_set_pv_features() runs twice on the destination.
> 
>> Although I do not understand flow of VM migration, with KVM_PUT_FULL_STATE state changing, there are at least two places where this state is set, one is from cpu_common_realizefn() which calls cpu_synchronize_post_init(), the other is qemu_loadvm_state()/qemu_loadvm_state_main() which calls cpu_synchronize_all_post_init().
>>
>> It seems that VM will fail to migrate since kvm_set_pv_features is called twice at least here. Do you test VM migration with this patch?
> 
> I did test migration (virt-11.2 -> virt-11.2, virt-11.1 -> virt-11.1) and
> it passed, but that was on a single host — source and destination computed
> the same pv_features. The kernel (kvm_loongarch_cpucfg_set_attr) only
> rejects a re-set when the value differs:
> 
>      if ((kvm->arch.pv_features & LOONGARCH_PV_FEAT_UPDATED) &&
>          ((kvm->arch.pv_features & valid) != val))
>          return -EINVAL;
> 
> So a cross-host migration where the two sides compute different pv_features
> would indeed fail on the second set.
> 
> I'll add a per-vCPU guard so the push happens exactly once (at the first
> FULL_STATE sync); subsequent syncs are skipped and the destination keeps
> advertising the features its own host supports. Does that sound like the
> right direction?
I do not consider pv feature migration supports before. There are two 
points from my side when VM migrates from hostA to hostB, maybe there 
are other requirements out of my knowledge :(
   1. the pv feature value got from running vCPUs on hostA should be 
supported on hostB.
   2. if VM is rebooted or one vCPU is hot added, the pv feature value 
got from new added vCPU on hostB should be the same with boot CPU0, 
which is got from hostA.

Regards
Bibo Mao
> 
> Thanks,
> Tao
> 
>>
>> Regards
>> Bibo Mao
>>>            if (ret) {
>>>                return ret;
>>>            }
>>> -        once = 1;
>>> -    }
>>>    -    if (level >= KVM_PUT_FULL_STATE) {
>>> -        /*
>>> -         * only KVM_PUT_FULL_STATE is required, kvm kernel will clear
>>> -         * guest_addr for KVM_PUT_RESET_STATE
>>> -         */
>>>            ret = kvm_set_stealtime(cs);
>>>            if (ret) {
>>>                return ret;
>>>
>>

Re: [PATCH v2 1/3] target/loongarch/kvm: advertise pv features per vCPU

[Tao Cui]

在 2026/7/10 16:44, Bibo Mao 写道:
> 
> 
> On 2026/7/10 下午4:09, Tao Cui wrote:
>>
>>
>> 在 2026/7/10 10:58, Bibo Mao 写道:
>>>
>>>
>>> On 2026/7/10 上午8:50, Tao Cui wrote:
>>>> From: Tao Cui <cuitao@kylinos.cn>
>>>>
>>>> kvm_set_pv_features() programs the KVM_FEATURE cpucfg attribute, which is a
>>>> per-vCPU setting. It was called from kvm_arch_put_registers() under a
>>>> function-local static guard, so it ran only once for the whole VM: only the
>>>> first vCPU got its pv features pushed to KVM, and on SMP guests the others
>>>> never saw KVM_FEATURE_IPI / KVM_FEATURE_STEAL_TIME.
>>>>
>>>> Drop the static guard and push pv features per vCPU under
>>>> KVM_PUT_FULL_STATE, the same gate kvm_set_stealtime() already uses. Host
>>>> feature detection stays in kvm_arch_init_vcpu(); the per-vCPU state write
>>>> belongs in kvm_arch_put_registers().
>>>>
>>>> Signed-off-by: Tao Cui <cuitao@kylinos.cn>
>>>> ---
>>>>    target/loongarch/kvm/kvm.c | 15 ++++++---------
>>>>    1 file changed, 6 insertions(+), 9 deletions(-)
>>>>
>>>> diff --git a/target/loongarch/kvm/kvm.c b/target/loongarch/kvm/kvm.c
>>>> index d6539c12ac..c557ee3c3d 100644
>>>> --- a/target/loongarch/kvm/kvm.c
>>>> +++ b/target/loongarch/kvm/kvm.c
>>>> @@ -816,7 +816,6 @@ int kvm_arch_get_registers(CPUState *cs, Error **errp)
>>>>    int kvm_arch_put_registers(CPUState *cs, KvmPutState level, Error **errp)
>>>>    {
>>>>        int ret;
>>>> -    static int once;
>>>>          ret = kvm_loongarch_put_regs_core(cs);
>>>>        if (ret) {
>>>> @@ -843,19 +842,17 @@ int kvm_arch_put_registers(CPUState *cs, KvmPutState level, Error **errp)
>>>>            return ret;
>>>>        }
>>>>    -    if (!once) {
>>>> +    if (level >= KVM_PUT_FULL_STATE) {
>>>> +        /*
>>>> +         * pv_features and steal time are per-vCPU state. Push them on
>>>> +         * full-state sync so every vCPU gets its own settings; the kernel
>>>> +         * clears the steal-time guest_addr on KVM_PUT_RESET_STATE.
>>>> +         */
>>>>            ret = kvm_set_pv_features(cs);
>>> pv feature is a little different from steal-time. steal-time guest_addr is created from guest OS, pv feature is created from VMM at beginning. steal-time guest_addr can be set for many times, and there is bit KVM_STEAL_PHYS_VALID checking with steal-time guest_addr, however pv feature can be set only once with existing method.
>>>
>>
>> Hi Bibo,
>>
>> Thanks for catching this — I hadn't fully considered the double-call path.
>>
>> KVM_PUT_FULL_STATE fires both at realize (cpu_synchronize_post_init) and
>> on incoming migration load (cpu_synchronize_all_post_init), so
>> kvm_set_pv_features() runs twice on the destination.
>>
>>> Although I do not understand flow of VM migration, with KVM_PUT_FULL_STATE state changing, there are at least two places where this state is set, one is from cpu_common_realizefn() which calls cpu_synchronize_post_init(), the other is qemu_loadvm_state()/qemu_loadvm_state_main() which calls cpu_synchronize_all_post_init().
>>>
>>> It seems that VM will fail to migrate since kvm_set_pv_features is called twice at least here. Do you test VM migration with this patch?
>>
>> I did test migration (virt-11.2 -> virt-11.2, virt-11.1 -> virt-11.1) and
>> it passed, but that was on a single host — source and destination computed
>> the same pv_features. The kernel (kvm_loongarch_cpucfg_set_attr) only
>> rejects a re-set when the value differs:
>>
>>      if ((kvm->arch.pv_features & LOONGARCH_PV_FEAT_UPDATED) &&
>>          ((kvm->arch.pv_features & valid) != val))
>>          return -EINVAL;
>>
>> So a cross-host migration where the two sides compute different pv_features
>> would indeed fail on the second set.
>>
>> I'll add a per-vCPU guard so the push happens exactly once (at the first
>> FULL_STATE sync); subsequent syncs are skipped and the destination keeps
>> advertising the features its own host supports. Does that sound like the
>> right direction?
> I do not consider pv feature migration supports before. There are two points from my side when VM migrates from hostA to hostB, maybe there are other requirements out of my knowledge :(
>   1. the pv feature value got from running vCPUs on hostA should be supported on hostB.
>   2. if VM is rebooted or one vCPU is hot added, the pv feature value got from new added vCPU on hostB should be the same with boot CPU0, which is got from hostA.
> 

Thanks, those are good points — my set-once guard doesn't handle them
correctly.

For Point 2, the root issue is that kvm->arch.pv_features is VM-level:
the first vCPU locks the value, and all others (including hot-added) must
match. So after migrating hostA -> hostB, a hot-added vCPU on hostB must
still use hostA's value, not re-compute from hostB.

I've been thinking about how to handle the push. The challenge is that
there's no single QEMU hook that fires exactly once with the final
pv_features value for both fresh boot and migration:
  - post_load only fires on migration load, not on fresh boot;
  - KVM_PUT_FULL_STATE fires at both realize and load (the double-call).

One option that seems to work: push from the first KVM_PUT_RUNTIME_STATE
sync (when the guest first runs), gated by a per-vCPU flag. By that point
env->pv_features holds the correct value for both cases — host-detected on
fresh boot, or the migrated value after post_load. The flag ensures the
push happens only once per vCPU.

This still doesn't solve hot-add after cross-host migration: a newly
created vCPU would re-compute from hostB via check_pv_features, getting a
different value from CPU0. For that I think pv_features needs to be stored
at the VM level (matching kvm->arch.pv_features), so hot-added vCPUs
inherit it rather than re-detecting.

I've also checked a few other migration paths: savevm/snapshot restore
goes through the same post_load, and multi-hop migration keeps the
original host's value — the runtime-state push should cover both.
During testing I observed that cross-machine-type migration (virt-11.1 <->
virt-11.2) fails at QEMU's configuration check; I haven't investigated
whether pv_features adds further constraints there.

I also looked at how other architectures handle this. On x86, CPUID
(including KVM feature leaves) is set via KVM_SET_CPUID2 in
kvm_arch_init_vcpu(), not in put_registers(); the kernel allows
re-setting CPUID and validates the features against the host on each SET.
On arm, KVM_ARM_VCPU_INIT and device attributes (PMU, PVTIME IPA, etc.)
are set in init_vcpu / machine init; the kernel likewise permits
re-setting device attributes. Neither architecture pushes feature
configuration through put_registers.

On LoongArch, the kernel currently locks pv_features after the first SET
and rejects any different value. This is what makes the migration case
hard — the destination cannot override the host-detected value with the
migrated one. If the kernel could relax this (allow overriding while
validating that the new value is a subset of what the host supports,
similar to x86's KVM_SET_CPUID2), the QEMU side would be simpler: push
once in init_vcpu (host-detected), then override from post_load on
migration. I think adjusting both QEMU and KVM together would be simpler
than working around it entirely in QEMU — would that work?

I'm still going through the overall flow to check if there are other
edge cases I've missed.

Tao

> Regards
> Bibo Mao
>>
>> Thanks,
>> Tao
>>
>>>
>>> Regards
>>> Bibo Mao
>>>>            if (ret) {
>>>>                return ret;
>>>>            }
>>>> -        once = 1;
>>>> -    }
>>>>    -    if (level >= KVM_PUT_FULL_STATE) {
>>>> -        /*
>>>> -         * only KVM_PUT_FULL_STATE is required, kvm kernel will clear
>>>> -         * guest_addr for KVM_PUT_RESET_STATE
>>>> -         */
>>>>            ret = kvm_set_stealtime(cs);
>>>>            if (ret) {
>>>>                return ret;
>>>>
>>>
> 

Re: [PATCH v2 1/3] target/loongarch/kvm: advertise pv features per vCPU

[Bibo Mao]

On 2026/7/10 下午8:46, Tao Cui wrote:
> 
> 
> 在 2026/7/10 16:44, Bibo Mao 写道:
>>
>>
>> On 2026/7/10 下午4:09, Tao Cui wrote:
>>>
>>>
>>> 在 2026/7/10 10:58, Bibo Mao 写道:
>>>>
>>>>
>>>> On 2026/7/10 上午8:50, Tao Cui wrote:
>>>>> From: Tao Cui <cuitao@kylinos.cn>
>>>>>
>>>>> kvm_set_pv_features() programs the KVM_FEATURE cpucfg attribute, which is a
>>>>> per-vCPU setting. It was called from kvm_arch_put_registers() under a
>>>>> function-local static guard, so it ran only once for the whole VM: only the
>>>>> first vCPU got its pv features pushed to KVM, and on SMP guests the others
>>>>> never saw KVM_FEATURE_IPI / KVM_FEATURE_STEAL_TIME.
>>>>>
>>>>> Drop the static guard and push pv features per vCPU under
>>>>> KVM_PUT_FULL_STATE, the same gate kvm_set_stealtime() already uses. Host
>>>>> feature detection stays in kvm_arch_init_vcpu(); the per-vCPU state write
>>>>> belongs in kvm_arch_put_registers().
>>>>>
>>>>> Signed-off-by: Tao Cui <cuitao@kylinos.cn>
>>>>> ---
>>>>>     target/loongarch/kvm/kvm.c | 15 ++++++---------
>>>>>     1 file changed, 6 insertions(+), 9 deletions(-)
>>>>>
>>>>> diff --git a/target/loongarch/kvm/kvm.c b/target/loongarch/kvm/kvm.c
>>>>> index d6539c12ac..c557ee3c3d 100644
>>>>> --- a/target/loongarch/kvm/kvm.c
>>>>> +++ b/target/loongarch/kvm/kvm.c
>>>>> @@ -816,7 +816,6 @@ int kvm_arch_get_registers(CPUState *cs, Error **errp)
>>>>>     int kvm_arch_put_registers(CPUState *cs, KvmPutState level, Error **errp)
>>>>>     {
>>>>>         int ret;
>>>>> -    static int once;
>>>>>           ret = kvm_loongarch_put_regs_core(cs);
>>>>>         if (ret) {
>>>>> @@ -843,19 +842,17 @@ int kvm_arch_put_registers(CPUState *cs, KvmPutState level, Error **errp)
>>>>>             return ret;
>>>>>         }
>>>>>     -    if (!once) {
>>>>> +    if (level >= KVM_PUT_FULL_STATE) {
>>>>> +        /*
>>>>> +         * pv_features and steal time are per-vCPU state. Push them on
>>>>> +         * full-state sync so every vCPU gets its own settings; the kernel
>>>>> +         * clears the steal-time guest_addr on KVM_PUT_RESET_STATE.
>>>>> +         */
>>>>>             ret = kvm_set_pv_features(cs);
>>>> pv feature is a little different from steal-time. steal-time guest_addr is created from guest OS, pv feature is created from VMM at beginning. steal-time guest_addr can be set for many times, and there is bit KVM_STEAL_PHYS_VALID checking with steal-time guest_addr, however pv feature can be set only once with existing method.
>>>>
>>>
>>> Hi Bibo,
>>>
>>> Thanks for catching this — I hadn't fully considered the double-call path.
>>>
>>> KVM_PUT_FULL_STATE fires both at realize (cpu_synchronize_post_init) and
>>> on incoming migration load (cpu_synchronize_all_post_init), so
>>> kvm_set_pv_features() runs twice on the destination.
>>>
>>>> Although I do not understand flow of VM migration, with KVM_PUT_FULL_STATE state changing, there are at least two places where this state is set, one is from cpu_common_realizefn() which calls cpu_synchronize_post_init(), the other is qemu_loadvm_state()/qemu_loadvm_state_main() which calls cpu_synchronize_all_post_init().
>>>>
>>>> It seems that VM will fail to migrate since kvm_set_pv_features is called twice at least here. Do you test VM migration with this patch?
>>>
>>> I did test migration (virt-11.2 -> virt-11.2, virt-11.1 -> virt-11.1) and
>>> it passed, but that was on a single host — source and destination computed
>>> the same pv_features. The kernel (kvm_loongarch_cpucfg_set_attr) only
>>> rejects a re-set when the value differs:
>>>
>>>       if ((kvm->arch.pv_features & LOONGARCH_PV_FEAT_UPDATED) &&
>>>           ((kvm->arch.pv_features & valid) != val))
>>>           return -EINVAL;
>>>
>>> So a cross-host migration where the two sides compute different pv_features
>>> would indeed fail on the second set.
>>>
>>> I'll add a per-vCPU guard so the push happens exactly once (at the first
>>> FULL_STATE sync); subsequent syncs are skipped and the destination keeps
>>> advertising the features its own host supports. Does that sound like the
>>> right direction?
>> I do not consider pv feature migration supports before. There are two points from my side when VM migrates from hostA to hostB, maybe there are other requirements out of my knowledge :(
>>    1. the pv feature value got from running vCPUs on hostA should be supported on hostB.
>>    2. if VM is rebooted or one vCPU is hot added, the pv feature value got from new added vCPU on hostB should be the same with boot CPU0, which is got from hostA.
>>
> 
> Thanks, those are good points — my set-once guard doesn't handle them
> correctly.
> 
> For Point 2, the root issue is that kvm->arch.pv_features is VM-level:
> the first vCPU locks the value, and all others (including hot-added) must
> match. So after migrating hostA -> hostB, a hot-added vCPU on hostB must
> still use hostA's value, not re-compute from hostB.
yes, it is.

> 
> I've been thinking about how to handle the push. The challenge is that
> there's no single QEMU hook that fires exactly once with the final
> pv_features value for both fresh boot and migration:
>    - post_load only fires on migration load, not on fresh boot;
>    - KVM_PUT_FULL_STATE fires at both realize and load (the double-call).
> 
> One option that seems to work: push from the first KVM_PUT_RUNTIME_STATE
> sync (when the guest first runs), gated by a per-vCPU flag. By that point
> env->pv_features holds the correct value for both cases — host-detected on
> fresh boot, or the migrated value after post_load. The flag ensures the
> push happens only once per vCPU.
> 
> This still doesn't solve hot-add after cross-host migration: a newly
> created vCPU would re-compute from hostB via check_pv_features, getting a
> different value from CPU0. For that I think pv_features needs to be stored
> at the VM level (matching kvm->arch.pv_features), so hot-added vCPUs
> inherit it rather than re-detecting.
> 
> I've also checked a few other migration paths: savevm/snapshot restore
> goes through the same post_load, and multi-hop migration keeps the
> original host's value — the runtime-state push should cover both.
> During testing I observed that cross-machine-type migration (virt-11.1 <->
> virt-11.2) fails at QEMU's configuration check; I haven't investigated
> whether pv_features adds further constraints there.
> 
> I also looked at how other architectures handle this. On x86, CPUID
> (including KVM feature leaves) is set via KVM_SET_CPUID2 in
> kvm_arch_init_vcpu(), not in put_registers(); the kernel allows
> re-setting CPUID and validates the features against the host on each SET.
> On arm, KVM_ARM_VCPU_INIT and device attributes (PMU, PVTIME IPA, etc.)
> are set in init_vcpu / machine init; the kernel likewise permits
> re-setting device attributes. Neither architecture pushes feature
> configuration through put_registers.
> 
> On LoongArch, the kernel currently locks pv_features after the first SET
> and rejects any different value. This is what makes the migration case
> hard — the destination cannot override the host-detected value with the
> migrated one. If the kernel could relax this (allow overriding while
> validating that the new value is a subset of what the host supports,
> similar to x86's KVM_SET_CPUID2), the QEMU side would be simpler: push
> once in init_vcpu (host-detected), then override from post_load on
> migration. I think adjusting both QEMU and KVM together would be simpler
> than working around it entirely in QEMU — would that work?
yes, I think that kvm kernel side should be modified also, pv_feature 
should support be set for many times at least before the fist time of 
vCPU running or similar methods like x86's KVM_SET_CPUID2.
> 
> I'm still going through the overall flow to check if there are other
> edge cases I've missed.
Good, that is the best.

Regards
Bibo Mao
> 
> Tao
> 
>> Regards
>> Bibo Mao
>>>
>>> Thanks,
>>> Tao
>>>
>>>>
>>>> Regards
>>>> Bibo Mao
>>>>>             if (ret) {
>>>>>                 return ret;
>>>>>             }
>>>>> -        once = 1;
>>>>> -    }
>>>>>     -    if (level >= KVM_PUT_FULL_STATE) {
>>>>> -        /*
>>>>> -         * only KVM_PUT_FULL_STATE is required, kvm kernel will clear
>>>>> -         * guest_addr for KVM_PUT_RESET_STATE
>>>>> -         */
>>>>>             ret = kvm_set_stealtime(cs);
>>>>>             if (ret) {
>>>>>                 return ret;
>>>>>
>>>>
>>
>