[RFC 0/3] Mitigation of migration failures accross different host kernels

[RFC 0/3] Mitigation of migration failures accross different host kernels

[Eric Auger]

When migrating ARM guests accross same machines with different host
kernels we are likely to encounter failures such as:

"failed to load cpu:cpreg_vmstate_array_len"

This is due to the fact KVM exposes a different number of registers
to qemu on source and destination. When trying to migrate a bigger
register set to a smaller one, qemu cannot save the CPU state.

For example, recently we faced such kind of situations with:
- unconditionnal exposure of KVM_REG_ARM_VENDOR_HYP_BMAP_2 FW pseudo
  register from v6.16 onwards. Causes backward migration failure.
- removal of unconditionnal exposure of TCR2_EL1, PIRE0_EL1,  PIR_EL1
  from v6.13 onwards. Causes forward migration failure.

More details can be found in individual patches.

This situation is really problematic for distributions which want to
guarantee forward and backward migration of a given machine type
between different releases.

This small series tries to address that issue by introducing CPU
array properties that list the registers to ignore or to fake according
to the situation. An example is given to illustrate how those props
could be used to apply compats for machine types supposed to "see" the
same register set accross various host kernels.

Obviously this is a last resort situation and this situation should be
avoided as much as possible.

Eric Auger (3):
  target/arm/cpu: Add new CPU property for KVM regs to hide
  target/arm/kvm: Add new CPU property for KVM regs to enforce
  hw/arm/virt: [DO NOT UPSTREAM] Enforce compatibility with older
    kernels

 target/arm/cpu.h        | 15 +++++++
 hw/arm/virt.c           | 19 ++++++++
 target/arm/kvm.c        | 99 +++++++++++++++++++++++++++++++++++++++--
 target/arm/trace-events |  6 +++
 4 files changed, 135 insertions(+), 4 deletions(-)

-- 
2.49.0

[RFC 1/3] target/arm/cpu: Add new CPU property for KVM regs to hide

[Eric Auger]

New kernels sometimes expose new registers in an unconditionnal
 manner.  This situation breaks backward migration as qemu notices
there are more registers to store on guest than supported in the
destination kerenl. This leads to a "failed to load
cpu:cpreg_vmstate_array_len" error.

A good example is the introduction of KVM_REG_ARM_VENDOR_HYP_BMAP_2
pseudo FW register in v6.16 by commit C0000e58c74e (“KVM: arm64:
Introduce KVM_REG_ARM_VENDOR_HYP_BMAP_2”). Trying to do backward
migration from a host kernel which features the commit to a destination
host that doesn't fail.

Currently QEMU is not using that feature so ignoring this latter
is not a problem. An easy way to fix the migration issue is to teach
qemu we don't care about that register and we can simply ignore it,
including its state migration.

This patch introduces a CPU property, under the form of an array of
reg indices which indicates which registers can be ignored.

The goal then is to set this property in machine type compats such
as:
static GlobalProperty arm_virt_kernel_compat_10_1[] = {
    /* KVM_REG_ARM_VENDOR_HYP_BMAP_2 */
    { TYPE_ARM_CPU, "kvm-hidden-regs", "0x6030000000160003" },
}

Signed-off-by: Eric Auger <eric.auger@redhat.com>
---
 target/arm/cpu.h        |  4 ++++
 target/arm/kvm.c        | 36 ++++++++++++++++++++++++++++++++++--
 target/arm/trace-events |  2 ++
 3 files changed, 40 insertions(+), 2 deletions(-)

diff --git a/target/arm/cpu.h b/target/arm/cpu.h
index c15d79a106..121b4372b2 100644
--- a/target/arm/cpu.h
+++ b/target/arm/cpu.h
@@ -1031,6 +1031,10 @@ struct ArchCPU {
     /* KVM steal time */
     OnOffAuto kvm_steal_time;
 
+    /* KVM registers that must be ignored/hidden */
+    uint64_t *kvm_hidden_regs;
+    uint32_t nr_kvm_hidden_regs;
+
     /* Uniprocessor system with MP extensions */
     bool mp_is_up;
 
diff --git a/target/arm/kvm.c b/target/arm/kvm.c
index 6672344855..67675781f4 100644
--- a/target/arm/kvm.c
+++ b/target/arm/kvm.c
@@ -39,6 +39,8 @@
 #include "qemu/log.h"
 #include "hw/acpi/acpi.h"
 #include "hw/acpi/ghes.h"
+#include "hw/qdev-properties.h"
+#include "hw/qdev-properties-system.h"
 #include "target/arm/gtimer.h"
 #include "migration/blocker.h"
 
@@ -483,6 +485,10 @@ static void kvm_steal_time_set(Object *obj, bool value, Error **errp)
     ARM_CPU(obj)->kvm_steal_time = value ? ON_OFF_AUTO_ON : ON_OFF_AUTO_OFF;
 }
 
+static const Property arm_cpu_kvm_compat_hidden_regs_property =
+    DEFINE_PROP_ARRAY("kvm-hidden-regs", ARMCPU,
+                      nr_kvm_hidden_regs, kvm_hidden_regs, qdev_prop_uint64, uint64_t);
+
 /* KVM VCPU properties should be prefixed with "kvm-". */
 void kvm_arm_add_vcpu_properties(ARMCPU *cpu)
 {
@@ -504,6 +510,8 @@ void kvm_arm_add_vcpu_properties(ARMCPU *cpu)
                              kvm_steal_time_set);
     object_property_set_description(obj, "kvm-steal-time",
                                     "Set off to disable KVM steal time.");
+
+    qdev_property_add_static(DEVICE(obj), &arm_cpu_kvm_compat_hidden_regs_property);
 }
 
 bool kvm_arm_pmu_supported(void)
@@ -764,6 +772,26 @@ static bool kvm_arm_reg_syncs_via_cpreg_list(uint64_t regidx)
     }
 }
 
+/**
+ * kvm_vcpu_compat_hidden_reg:
+ * @cpu: ARMCPU
+ * @regidx: index of the register to check
+ *
+ * Depending on the CPU compat returns true if @regidx must be
+ * ignored during sync & migration
+ */
+static inline bool
+kvm_vcpu_compat_hidden_reg(ARMCPU *cpu, uint64_t regidx)
+{
+    for (int i = 0; i < cpu->nr_kvm_hidden_regs; i++) {
+        if (cpu->kvm_hidden_regs[i] == regidx) {
+            trace_kvm_vcpu_compat_hidden_reg(regidx);
+            return true;
+        }
+    }
+    return false;
+}
+
 /**
  * kvm_arm_init_cpreg_list:
  * @cpu: ARMCPU
@@ -798,7 +826,8 @@ static int kvm_arm_init_cpreg_list(ARMCPU *cpu)
     qsort(&rlp->reg, rlp->n, sizeof(rlp->reg[0]), compare_u64);
 
     for (i = 0, arraylen = 0; i < rlp->n; i++) {
-        if (!kvm_arm_reg_syncs_via_cpreg_list(rlp->reg[i])) {
+        if (!kvm_arm_reg_syncs_via_cpreg_list(rlp->reg[i]) ||
+            kvm_vcpu_compat_hidden_reg(cpu, rlp->reg[i])) {
             continue;
         }
         switch (rlp->reg[i] & KVM_REG_SIZE_MASK) {
@@ -814,6 +843,8 @@ static int kvm_arm_init_cpreg_list(ARMCPU *cpu)
         arraylen++;
     }
 
+    trace_kvm_arm_init_cpreg_list_arraylen(arraylen);
+
     cpu->cpreg_indexes = g_renew(uint64_t, cpu->cpreg_indexes, arraylen);
     cpu->cpreg_values = g_renew(uint64_t, cpu->cpreg_values, arraylen);
     cpu->cpreg_vmstate_indexes = g_renew(uint64_t, cpu->cpreg_vmstate_indexes,
@@ -825,7 +856,8 @@ static int kvm_arm_init_cpreg_list(ARMCPU *cpu)
 
     for (i = 0, arraylen = 0; i < rlp->n; i++) {
         uint64_t regidx = rlp->reg[i];
-        if (!kvm_arm_reg_syncs_via_cpreg_list(regidx)) {
+        if (!kvm_arm_reg_syncs_via_cpreg_list(regidx) ||
+            kvm_vcpu_compat_hidden_reg(cpu, regidx)) {
             continue;
         }
         cpu->cpreg_indexes[arraylen] = regidx;
diff --git a/target/arm/trace-events b/target/arm/trace-events
index 4438dce7be..1b4ab0c683 100644
--- a/target/arm/trace-events
+++ b/target/arm/trace-events
@@ -13,3 +13,5 @@ arm_gt_update_irq(int timer, int irqstate) "gt_update_irq: timer %d irqstate %d"
 
 # kvm.c
 kvm_arm_fixup_msi_route(uint64_t iova, uint64_t gpa) "MSI iova = 0x%"PRIx64" is translated into 0x%"PRIx64
+kvm_arm_init_cpreg_list_arraylen(uint32_t arraylen) "arraylen=%d"
+kvm_vcpu_compat_hidden_reg(uint64_t regidx) "0x%"PRIx64" is hidden"
-- 
2.49.0

Re: [RFC 1/3] target/arm/cpu: Add new CPU property for KVM regs to hide

[Sebastian Ott]

[-- Attachment #1: Type: text/plain, Size: 1562 bytes --]

On Thu, 11 Sep 2025, Eric Auger wrote:
> New kernels sometimes expose new registers in an unconditionnal
> manner.  This situation breaks backward migration as qemu notices
> there are more registers to store on guest than supported in the
> destination kerenl. This leads to a "failed to load
> cpu:cpreg_vmstate_array_len" error.
>
> A good example is the introduction of KVM_REG_ARM_VENDOR_HYP_BMAP_2
> pseudo FW register in v6.16 by commit C0000e58c74e (“KVM: arm64:
> Introduce KVM_REG_ARM_VENDOR_HYP_BMAP_2”). Trying to do backward
> migration from a host kernel which features the commit to a destination
> host that doesn't fail.
>
> Currently QEMU is not using that feature so ignoring this latter
> is not a problem. An easy way to fix the migration issue is to teach
> qemu we don't care about that register and we can simply ignore it,
> including its state migration.
>
> This patch introduces a CPU property, under the form of an array of
> reg indices which indicates which registers can be ignored.
>
> The goal then is to set this property in machine type compats such
> as:
> static GlobalProperty arm_virt_kernel_compat_10_1[] = {
>    /* KVM_REG_ARM_VENDOR_HYP_BMAP_2 */
>    { TYPE_ARM_CPU, "kvm-hidden-regs", "0x6030000000160003" },
> }
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> ---
> target/arm/cpu.h        |  4 ++++
> target/arm/kvm.c        | 36 ++++++++++++++++++++++++++++++++++--
> target/arm/trace-events |  2 ++
> 3 files changed, 40 insertions(+), 2 deletions(-)
>

Reviewed-by: Sebastian Ott <sebott@redhat.com>

Re: [RFC 1/3] target/arm/cpu: Add new CPU property for KVM regs to hide

[Sebastian Ott]

[-- Attachment #1: Type: text/plain, Size: 1659 bytes --]

On Thu, 11 Sep 2025, Eric Auger wrote:
> New kernels sometimes expose new registers in an unconditionnal
> manner.  This situation breaks backward migration as qemu notices
> there are more registers to store on guest than supported in the
> destination kerenl. This leads to a "failed to load
> cpu:cpreg_vmstate_array_len" error.
>
> A good example is the introduction of KVM_REG_ARM_VENDOR_HYP_BMAP_2
> pseudo FW register in v6.16 by commit C0000e58c74e (“KVM: arm64:
> Introduce KVM_REG_ARM_VENDOR_HYP_BMAP_2”). Trying to do backward
> migration from a host kernel which features the commit to a destination
> host that doesn't fail.
>
> Currently QEMU is not using that feature so ignoring this latter
> is not a problem. An easy way to fix the migration issue is to teach
> qemu we don't care about that register and we can simply ignore it,
> including its state migration.
>
> This patch introduces a CPU property, under the form of an array of
> reg indices which indicates which registers can be ignored.
>
> The goal then is to set this property in machine type compats such
> as:
> static GlobalProperty arm_virt_kernel_compat_10_1[] = {
>    /* KVM_REG_ARM_VENDOR_HYP_BMAP_2 */
>    { TYPE_ARM_CPU, "kvm-hidden-regs", "0x6030000000160003" },
> }

One thing worth noting - once this series lands:
https://lore.kernel.org/qemu-devel/20250801074730.28329-1-shameerkolothum@gmail.com/
we might need to add a bit more logic here. Either using the kvm
interfaces (only ignore KVM_REG_ARM_VENDOR_HYP_BMAP_2 when the register
value is 0) or qemu knowledge (only ignore KVM_REG_ARM_VENDOR_HYP_BMAP_2
when the impl-cpu property is not used).

Sebastian

Re: [RFC 1/3] target/arm/cpu: Add new CPU property for KVM regs to hide

[Eric Auger]

Hi Sebastian,

On 9/18/25 6:16 PM, Sebastian Ott wrote:
> On Thu, 11 Sep 2025, Eric Auger wrote:
>> New kernels sometimes expose new registers in an unconditionnal
>> manner.  This situation breaks backward migration as qemu notices
>> there are more registers to store on guest than supported in the
>> destination kerenl. This leads to a "failed to load
>> cpu:cpreg_vmstate_array_len" error.
>>
>> A good example is the introduction of KVM_REG_ARM_VENDOR_HYP_BMAP_2
>> pseudo FW register in v6.16 by commit C0000e58c74e (“KVM: arm64:
>> Introduce KVM_REG_ARM_VENDOR_HYP_BMAP_2”). Trying to do backward
>> migration from a host kernel which features the commit to a destination
>> host that doesn't fail.
>>
>> Currently QEMU is not using that feature so ignoring this latter
>> is not a problem. An easy way to fix the migration issue is to teach
>> qemu we don't care about that register and we can simply ignore it,
>> including its state migration.
>>
>> This patch introduces a CPU property, under the form of an array of
>> reg indices which indicates which registers can be ignored.
>>
>> The goal then is to set this property in machine type compats such
>> as:
>> static GlobalProperty arm_virt_kernel_compat_10_1[] = {
>>    /* KVM_REG_ARM_VENDOR_HYP_BMAP_2 */
>>    { TYPE_ARM_CPU, "kvm-hidden-regs", "0x6030000000160003" },
>> }
>
> One thing worth noting - once this series lands:
> https://lore.kernel.org/qemu-devel/20250801074730.28329-1-shameerkolothum@gmail.com/
>
> we might need to add a bit more logic here. Either using the kvm
> interfaces (only ignore KVM_REG_ARM_VENDOR_HYP_BMAP_2 when the register
> value is 0) or qemu knowledge (only ignore KVM_REG_ARM_VENDOR_HYP_BMAP_2
> when the impl-cpu property is not used).

Effectively if we "hide" KVM_REG_ARM_VENDOR_HYP_BMAP_2 on save/restore
we must enforce the reg is not used by userspace.

One way would be to test whether KVM_REG_ARM_VENDOR_HYP_BMAP_2 is hidden
in kvm_arm_target_impl_cpus_supported() and if it is, report false.
However for every new functionality in qemu it does not sound sensible
to check whether new KVM regs being used are anonymously hidden.

Another way could be to fail kvm_set_one_reg/kvm_get_one_reg in case the
register is hidden. That way in Shameer's series, kvm_arch_init_vcpu()
would fail if BMAP_2 is hidden, ie. in our example for all machines
types before 10.2. By the way adding Shameer.

Thanks

Eric


>
> Sebastian

Re: [RFC 1/3] target/arm/cpu: Add new CPU property for KVM regs to hide

[Cornelia Huck]

On Fri, Oct 03 2025, Eric Auger <eric.auger@redhat.com> wrote:

> Hi Sebastian,
>
> On 9/18/25 6:16 PM, Sebastian Ott wrote:
>> On Thu, 11 Sep 2025, Eric Auger wrote:
>>> New kernels sometimes expose new registers in an unconditionnal
>>> manner.  This situation breaks backward migration as qemu notices
>>> there are more registers to store on guest than supported in the
>>> destination kerenl. This leads to a "failed to load
>>> cpu:cpreg_vmstate_array_len" error.
>>>
>>> A good example is the introduction of KVM_REG_ARM_VENDOR_HYP_BMAP_2
>>> pseudo FW register in v6.16 by commit C0000e58c74e (“KVM: arm64:
>>> Introduce KVM_REG_ARM_VENDOR_HYP_BMAP_2”). Trying to do backward
>>> migration from a host kernel which features the commit to a destination
>>> host that doesn't fail.
>>>
>>> Currently QEMU is not using that feature so ignoring this latter
>>> is not a problem. An easy way to fix the migration issue is to teach
>>> qemu we don't care about that register and we can simply ignore it,
>>> including its state migration.
>>>
>>> This patch introduces a CPU property, under the form of an array of
>>> reg indices which indicates which registers can be ignored.
>>>
>>> The goal then is to set this property in machine type compats such
>>> as:
>>> static GlobalProperty arm_virt_kernel_compat_10_1[] = {
>>>    /* KVM_REG_ARM_VENDOR_HYP_BMAP_2 */
>>>    { TYPE_ARM_CPU, "kvm-hidden-regs", "0x6030000000160003" },
>>> }
>>
>> One thing worth noting - once this series lands:
>> https://lore.kernel.org/qemu-devel/20250801074730.28329-1-shameerkolothum@gmail.com/
>>
>> we might need to add a bit more logic here. Either using the kvm
>> interfaces (only ignore KVM_REG_ARM_VENDOR_HYP_BMAP_2 when the register
>> value is 0) or qemu knowledge (only ignore KVM_REG_ARM_VENDOR_HYP_BMAP_2
>> when the impl-cpu property is not used).
>
> Effectively if we "hide" KVM_REG_ARM_VENDOR_HYP_BMAP_2 on save/restore
> we must enforce the reg is not used by userspace.
>
> One way would be to test whether KVM_REG_ARM_VENDOR_HYP_BMAP_2 is hidden
> in kvm_arm_target_impl_cpus_supported() and if it is, report false.
> However for every new functionality in qemu it does not sound sensible
> to check whether new KVM regs being used are anonymously hidden.
>
> Another way could be to fail kvm_set_one_reg/kvm_get_one_reg in case the
> register is hidden. That way in Shameer's series, kvm_arch_init_vcpu()
> would fail if BMAP_2 is hidden, ie. in our example for all machines
> types before 10.2. By the way adding Shameer.

I think tying this to the state of the reg (hidden or not) is less
error-prone (I'd assume we'd have different ways of detecting whether
something is used for future cases, and "is the reg hidden?" would work
in all cases.) We'd need to tie migration to matching machine versions
anyway, I think.

Re: [RFC 1/3] target/arm/cpu: Add new CPU property for KVM regs to hide

[Eric Auger]

On 10/8/25 3:49 PM, Cornelia Huck wrote:
> On Fri, Oct 03 2025, Eric Auger <eric.auger@redhat.com> wrote:
>
>> Hi Sebastian,
>>
>> On 9/18/25 6:16 PM, Sebastian Ott wrote:
>>> On Thu, 11 Sep 2025, Eric Auger wrote:
>>>> New kernels sometimes expose new registers in an unconditionnal
>>>> manner.  This situation breaks backward migration as qemu notices
>>>> there are more registers to store on guest than supported in the
>>>> destination kerenl. This leads to a "failed to load
>>>> cpu:cpreg_vmstate_array_len" error.
>>>>
>>>> A good example is the introduction of KVM_REG_ARM_VENDOR_HYP_BMAP_2
>>>> pseudo FW register in v6.16 by commit C0000e58c74e (“KVM: arm64:
>>>> Introduce KVM_REG_ARM_VENDOR_HYP_BMAP_2”). Trying to do backward
>>>> migration from a host kernel which features the commit to a destination
>>>> host that doesn't fail.
>>>>
>>>> Currently QEMU is not using that feature so ignoring this latter
>>>> is not a problem. An easy way to fix the migration issue is to teach
>>>> qemu we don't care about that register and we can simply ignore it,
>>>> including its state migration.
>>>>
>>>> This patch introduces a CPU property, under the form of an array of
>>>> reg indices which indicates which registers can be ignored.
>>>>
>>>> The goal then is to set this property in machine type compats such
>>>> as:
>>>> static GlobalProperty arm_virt_kernel_compat_10_1[] = {
>>>>    /* KVM_REG_ARM_VENDOR_HYP_BMAP_2 */
>>>>    { TYPE_ARM_CPU, "kvm-hidden-regs", "0x6030000000160003" },
>>>> }
>>> One thing worth noting - once this series lands:
>>> https://lore.kernel.org/qemu-devel/20250801074730.28329-1-shameerkolothum@gmail.com/
>>>
>>> we might need to add a bit more logic here. Either using the kvm
>>> interfaces (only ignore KVM_REG_ARM_VENDOR_HYP_BMAP_2 when the register
>>> value is 0) or qemu knowledge (only ignore KVM_REG_ARM_VENDOR_HYP_BMAP_2
>>> when the impl-cpu property is not used).
>> Effectively if we "hide" KVM_REG_ARM_VENDOR_HYP_BMAP_2 on save/restore
>> we must enforce the reg is not used by userspace.
>>
>> One way would be to test whether KVM_REG_ARM_VENDOR_HYP_BMAP_2 is hidden
>> in kvm_arm_target_impl_cpus_supported() and if it is, report false.
>> However for every new functionality in qemu it does not sound sensible
>> to check whether new KVM regs being used are anonymously hidden.
>>
>> Another way could be to fail kvm_set_one_reg/kvm_get_one_reg in case the
>> register is hidden. That way in Shameer's series, kvm_arch_init_vcpu()
>> would fail if BMAP_2 is hidden, ie. in our example for all machines
>> types before 10.2. By the way adding Shameer.
> I think tying this to the state of the reg (hidden or not) is less
> error-prone (I'd assume we'd have different ways of detecting whether
> something is used for future cases, and "is the reg hidden?" would work
> in all cases.) We'd need to tie migration to matching machine versions
> anyway, I think.
>
I guess you suggest to check the hidden/fake state in

kvm_set_one_reg/kvm_get_one_reg too. One issue is those helpers are arch agnostic. I would need to either introduce a callback in the CPU class to check the actual status or add the props in the parent CPU object. Or introduce a KVM IOTCL to teach KVM a reg shall never be accessed.

Thoughts?

Thanks

Eric

Re: [RFC 1/3] target/arm/cpu: Add new CPU property for KVM regs to hide

[Cornelia Huck]

On Tue, Oct 14 2025, Eric Auger <eric.auger@redhat.com> wrote:

> On 10/8/25 3:49 PM, Cornelia Huck wrote:
>> On Fri, Oct 03 2025, Eric Auger <eric.auger@redhat.com> wrote:
>>
>>> Hi Sebastian,
>>>
>>> On 9/18/25 6:16 PM, Sebastian Ott wrote:
>>>> On Thu, 11 Sep 2025, Eric Auger wrote:
>>>>> New kernels sometimes expose new registers in an unconditionnal
>>>>> manner.  This situation breaks backward migration as qemu notices
>>>>> there are more registers to store on guest than supported in the
>>>>> destination kerenl. This leads to a "failed to load
>>>>> cpu:cpreg_vmstate_array_len" error.
>>>>>
>>>>> A good example is the introduction of KVM_REG_ARM_VENDOR_HYP_BMAP_2
>>>>> pseudo FW register in v6.16 by commit C0000e58c74e (“KVM: arm64:
>>>>> Introduce KVM_REG_ARM_VENDOR_HYP_BMAP_2”). Trying to do backward
>>>>> migration from a host kernel which features the commit to a destination
>>>>> host that doesn't fail.
>>>>>
>>>>> Currently QEMU is not using that feature so ignoring this latter
>>>>> is not a problem. An easy way to fix the migration issue is to teach
>>>>> qemu we don't care about that register and we can simply ignore it,
>>>>> including its state migration.
>>>>>
>>>>> This patch introduces a CPU property, under the form of an array of
>>>>> reg indices which indicates which registers can be ignored.
>>>>>
>>>>> The goal then is to set this property in machine type compats such
>>>>> as:
>>>>> static GlobalProperty arm_virt_kernel_compat_10_1[] = {
>>>>>    /* KVM_REG_ARM_VENDOR_HYP_BMAP_2 */
>>>>>    { TYPE_ARM_CPU, "kvm-hidden-regs", "0x6030000000160003" },
>>>>> }
>>>> One thing worth noting - once this series lands:
>>>> https://lore.kernel.org/qemu-devel/20250801074730.28329-1-shameerkolothum@gmail.com/
>>>>
>>>> we might need to add a bit more logic here. Either using the kvm
>>>> interfaces (only ignore KVM_REG_ARM_VENDOR_HYP_BMAP_2 when the register
>>>> value is 0) or qemu knowledge (only ignore KVM_REG_ARM_VENDOR_HYP_BMAP_2
>>>> when the impl-cpu property is not used).
>>> Effectively if we "hide" KVM_REG_ARM_VENDOR_HYP_BMAP_2 on save/restore
>>> we must enforce the reg is not used by userspace.
>>>
>>> One way would be to test whether KVM_REG_ARM_VENDOR_HYP_BMAP_2 is hidden
>>> in kvm_arm_target_impl_cpus_supported() and if it is, report false.
>>> However for every new functionality in qemu it does not sound sensible
>>> to check whether new KVM regs being used are anonymously hidden.
>>>
>>> Another way could be to fail kvm_set_one_reg/kvm_get_one_reg in case the
>>> register is hidden. That way in Shameer's series, kvm_arch_init_vcpu()
>>> would fail if BMAP_2 is hidden, ie. in our example for all machines
>>> types before 10.2. By the way adding Shameer.
>> I think tying this to the state of the reg (hidden or not) is less
>> error-prone (I'd assume we'd have different ways of detecting whether
>> something is used for future cases, and "is the reg hidden?" would work
>> in all cases.) We'd need to tie migration to matching machine versions
>> anyway, I think.
>>
> I guess you suggest to check the hidden/fake state in
>
> kvm_set_one_reg/kvm_get_one_reg too. One issue is those helpers are arch agnostic. I would need to either introduce a callback in the CPU class to check the actual status or add the props in the parent CPU object. Or introduce a KVM IOTCL to teach KVM a reg shall never be accessed.

Aren't they always called from arch code, though? We could easily wrap
them for arm, I think. (Unless there's some more generic interest in
conditional ONE_REG accesses.)

Re: [RFC 1/3] target/arm/cpu: Add new CPU property for KVM regs to hide

[Eric Auger]

Hi Connie,

On 10/15/25 3:12 PM, Cornelia Huck wrote:
> On Tue, Oct 14 2025, Eric Auger <eric.auger@redhat.com> wrote:
>
>> On 10/8/25 3:49 PM, Cornelia Huck wrote:
>>> On Fri, Oct 03 2025, Eric Auger <eric.auger@redhat.com> wrote:
>>>
>>>> Hi Sebastian,
>>>>
>>>> On 9/18/25 6:16 PM, Sebastian Ott wrote:
>>>>> On Thu, 11 Sep 2025, Eric Auger wrote:
>>>>>> New kernels sometimes expose new registers in an unconditionnal
>>>>>> manner.  This situation breaks backward migration as qemu notices
>>>>>> there are more registers to store on guest than supported in the
>>>>>> destination kerenl. This leads to a "failed to load
>>>>>> cpu:cpreg_vmstate_array_len" error.
>>>>>>
>>>>>> A good example is the introduction of KVM_REG_ARM_VENDOR_HYP_BMAP_2
>>>>>> pseudo FW register in v6.16 by commit C0000e58c74e (“KVM: arm64:
>>>>>> Introduce KVM_REG_ARM_VENDOR_HYP_BMAP_2”). Trying to do backward
>>>>>> migration from a host kernel which features the commit to a destination
>>>>>> host that doesn't fail.
>>>>>>
>>>>>> Currently QEMU is not using that feature so ignoring this latter
>>>>>> is not a problem. An easy way to fix the migration issue is to teach
>>>>>> qemu we don't care about that register and we can simply ignore it,
>>>>>> including its state migration.
>>>>>>
>>>>>> This patch introduces a CPU property, under the form of an array of
>>>>>> reg indices which indicates which registers can be ignored.
>>>>>>
>>>>>> The goal then is to set this property in machine type compats such
>>>>>> as:
>>>>>> static GlobalProperty arm_virt_kernel_compat_10_1[] = {
>>>>>>    /* KVM_REG_ARM_VENDOR_HYP_BMAP_2 */
>>>>>>    { TYPE_ARM_CPU, "kvm-hidden-regs", "0x6030000000160003" },
>>>>>> }
>>>>> One thing worth noting - once this series lands:
>>>>> https://lore.kernel.org/qemu-devel/20250801074730.28329-1-shameerkolothum@gmail.com/
>>>>>
>>>>> we might need to add a bit more logic here. Either using the kvm
>>>>> interfaces (only ignore KVM_REG_ARM_VENDOR_HYP_BMAP_2 when the register
>>>>> value is 0) or qemu knowledge (only ignore KVM_REG_ARM_VENDOR_HYP_BMAP_2
>>>>> when the impl-cpu property is not used).
>>>> Effectively if we "hide" KVM_REG_ARM_VENDOR_HYP_BMAP_2 on save/restore
>>>> we must enforce the reg is not used by userspace.
>>>>
>>>> One way would be to test whether KVM_REG_ARM_VENDOR_HYP_BMAP_2 is hidden
>>>> in kvm_arm_target_impl_cpus_supported() and if it is, report false.
>>>> However for every new functionality in qemu it does not sound sensible
>>>> to check whether new KVM regs being used are anonymously hidden.
>>>>
>>>> Another way could be to fail kvm_set_one_reg/kvm_get_one_reg in case the
>>>> register is hidden. That way in Shameer's series, kvm_arch_init_vcpu()
>>>> would fail if BMAP_2 is hidden, ie. in our example for all machines
>>>> types before 10.2. By the way adding Shameer.
>>> I think tying this to the state of the reg (hidden or not) is less
>>> error-prone (I'd assume we'd have different ways of detecting whether
>>> something is used for future cases, and "is the reg hidden?" would work
>>> in all cases.) We'd need to tie migration to matching machine versions
>>> anyway, I think.
>>>
>> I guess you suggest to check the hidden/fake state in
>>
>> kvm_set_one_reg/kvm_get_one_reg too. One issue is those helpers are arch agnostic. I would need to either introduce a callback in the CPU class to check the actual status or add the props in the parent CPU object. Or introduce a KVM IOTCL to teach KVM a reg shall never be accessed.
> Aren't they always called from arch code, though? We could easily wrap
> them for arm, I think. (Unless there's some more generic interest in
> conditional ONE_REG accesses.)

For the time being I chose to add a new CPU class callback to check
whether the reg is hidden in kvm_set/get_one_reg.
Let's wait for other comments ... ;-)

Thanks

Eric
>

Re: [RFC 1/3] target/arm/cpu: Add new CPU property for KVM regs to hide

[Cornelia Huck]

On Thu, Sep 11 2025, Eric Auger <eric.auger@redhat.com> wrote:

> New kernels sometimes expose new registers in an unconditionnal
>  manner.  This situation breaks backward migration as qemu notices
> there are more registers to store on guest than supported in the
> destination kerenl. This leads to a "failed to load
> cpu:cpreg_vmstate_array_len" error.
>
> A good example is the introduction of KVM_REG_ARM_VENDOR_HYP_BMAP_2
> pseudo FW register in v6.16 by commit C0000e58c74e (“KVM: arm64:
> Introduce KVM_REG_ARM_VENDOR_HYP_BMAP_2”). Trying to do backward
> migration from a host kernel which features the commit to a destination
> host that doesn't fail.
>
> Currently QEMU is not using that feature so ignoring this latter
> is not a problem. An easy way to fix the migration issue is to teach
> qemu we don't care about that register and we can simply ignore it,
> including its state migration.
>
> This patch introduces a CPU property, under the form of an array of
> reg indices which indicates which registers can be ignored.
>
> The goal then is to set this property in machine type compats such
> as:
> static GlobalProperty arm_virt_kernel_compat_10_1[] = {
>     /* KVM_REG_ARM_VENDOR_HYP_BMAP_2 */
>     { TYPE_ARM_CPU, "kvm-hidden-regs", "0x6030000000160003" },
> }
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> ---
>  target/arm/cpu.h        |  4 ++++
>  target/arm/kvm.c        | 36 ++++++++++++++++++++++++++++++++++--
>  target/arm/trace-events |  2 ++
>  3 files changed, 40 insertions(+), 2 deletions(-)
>

(...)

> +/**
> + * kvm_vcpu_compat_hidden_reg:
> + * @cpu: ARMCPU
> + * @regidx: index of the register to check
> + *
> + * Depending on the CPU compat returns true if @regidx must be
> + * ignored during sync & migration

Maybe add some more explanation, either here or at the kvm_hidden_regs[]
def? So that people do not need to look at the introducing commit :)

"This is intended for when we know that we do not use the register to be
ignored, and want to keep compatibility."

> + */
> +static inline bool
> +kvm_vcpu_compat_hidden_reg(ARMCPU *cpu, uint64_t regidx)
> +{
> +    for (int i = 0; i < cpu->nr_kvm_hidden_regs; i++) {
> +        if (cpu->kvm_hidden_regs[i] == regidx) {
> +            trace_kvm_vcpu_compat_hidden_reg(regidx);
> +            return true;
> +        }
> +    }
> +    return false;
> +}

Re: [RFC 1/3] target/arm/cpu: Add new CPU property for KVM regs to hide

[Eric Auger]

On 10/8/25 3:43 PM, Cornelia Huck wrote:
> On Thu, Sep 11 2025, Eric Auger <eric.auger@redhat.com> wrote:
>
>> New kernels sometimes expose new registers in an unconditionnal
>>  manner.  This situation breaks backward migration as qemu notices
>> there are more registers to store on guest than supported in the
>> destination kerenl. This leads to a "failed to load
>> cpu:cpreg_vmstate_array_len" error.
>>
>> A good example is the introduction of KVM_REG_ARM_VENDOR_HYP_BMAP_2
>> pseudo FW register in v6.16 by commit C0000e58c74e (“KVM: arm64:
>> Introduce KVM_REG_ARM_VENDOR_HYP_BMAP_2”). Trying to do backward
>> migration from a host kernel which features the commit to a destination
>> host that doesn't fail.
>>
>> Currently QEMU is not using that feature so ignoring this latter
>> is not a problem. An easy way to fix the migration issue is to teach
>> qemu we don't care about that register and we can simply ignore it,
>> including its state migration.
>>
>> This patch introduces a CPU property, under the form of an array of
>> reg indices which indicates which registers can be ignored.
>>
>> The goal then is to set this property in machine type compats such
>> as:
>> static GlobalProperty arm_virt_kernel_compat_10_1[] = {
>>     /* KVM_REG_ARM_VENDOR_HYP_BMAP_2 */
>>     { TYPE_ARM_CPU, "kvm-hidden-regs", "0x6030000000160003" },
>> }
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>> ---
>>  target/arm/cpu.h        |  4 ++++
>>  target/arm/kvm.c        | 36 ++++++++++++++++++++++++++++++++++--
>>  target/arm/trace-events |  2 ++
>>  3 files changed, 40 insertions(+), 2 deletions(-)
>>
> (...)
>
>> +/**
>> + * kvm_vcpu_compat_hidden_reg:
>> + * @cpu: ARMCPU
>> + * @regidx: index of the register to check
>> + *
>> + * Depending on the CPU compat returns true if @regidx must be
>> + * ignored during sync & migration
> Maybe add some more explanation, either here or at the kvm_hidden_regs[]
> def? So that people do not need to look at the introducing commit :)
>
> "This is intended for when we know that we do not use the register to be
> ignored, and want to keep compatibility."

Added here and also in next patch, in cpu.h

Thanks!

Eric
>
>> + */
>> +static inline bool
>> +kvm_vcpu_compat_hidden_reg(ARMCPU *cpu, uint64_t regidx)
>> +{
>> +    for (int i = 0; i < cpu->nr_kvm_hidden_regs; i++) {
>> +        if (cpu->kvm_hidden_regs[i] == regidx) {
>> +            trace_kvm_vcpu_compat_hidden_reg(regidx);
>> +            return true;
>> +        }
>> +    }
>> +    return false;
>> +}

[RFC 2/3] target/arm/kvm: Add new CPU property for KVM regs to enforce

[Eric Auger]

Newer kernels sometimes revokes exposure of KVM regs to
userspace. This can happen if some registers were unconditionnally
exposed, by mistake, in previous versions and then conditionnally exposed.

Recently this happened with 3 registers: TCR2_EL1, PIRE0_EL1,  PIR_EL1.
The associated kernel commit were:
0fcb4eea5345  KVM: arm64: Hide TCR2_EL1 from userspace when disabled for guests
a68cddbe47ef  KVM: arm64: Hide S1PIE registers from userspace when disabled for guests

Those commits were actual fixes but the cons is that is breaks forward
miration from old to new host kernels as the number of KVM regs exposed
to userspace decreases and this leads to "failed to load cpu:cpreg_vmstate_array_len"
when migrating the CPU state.

This patchs adds a new CPU property, under the form of an array of reg
indices that teach (destination) QEMU that some registers must exist.
If they don't (because KVM does not expose them anymore), qemu fakes them but
does not care of their state.

This is meant to be applied as a cmahine type compat like:
static GlobalProperty arm_virt_kernel_compat_10_1[] = {
    { TYPE_ARM_CPU, "kvm-enforced-regs",
      /* TCR_EL1, PIRE0_EL1, PIR_EL1 */
      "0x603000000013c103, 0x603000000013c512, 0x603000000013c513" },
};

In case we migrate from a source where the KVM regs are not exposed and
where regs are faked to a destination where KVM expose them, this means that
anyway FEAT_TCR2 and S1PIE are not supported by the guest so it should
not matter that dumb values are written on dest.

But obviously introducing fake registers is a last resort solution.

Signed-off-by: Eric Auger <eric.auger@redhat.com>
---
 target/arm/cpu.h        | 11 +++++++
 target/arm/kvm.c        | 63 +++++++++++++++++++++++++++++++++++++++--
 target/arm/trace-events |  4 +++
 3 files changed, 76 insertions(+), 2 deletions(-)

diff --git a/target/arm/cpu.h b/target/arm/cpu.h
index 121b4372b2..ea9045e5ff 100644
--- a/target/arm/cpu.h
+++ b/target/arm/cpu.h
@@ -1035,6 +1035,17 @@ struct ArchCPU {
     uint64_t *kvm_hidden_regs;
     uint32_t nr_kvm_hidden_regs;
 
+    /*
+     * KVM registers whose presence must be enforced
+     * Either they must be exposed to user space by KVM or
+     * they must be faked for migration sake
+     */
+    uint64_t *kvm_enforced_regs;
+    uint32_t nr_kvm_enforced_regs;
+
+    /* registers among those to be enforced that are faked */
+    uint64_t *kvm_fake_regs;
+
     /* Uniprocessor system with MP extensions */
     bool mp_is_up;
 
diff --git a/target/arm/kvm.c b/target/arm/kvm.c
index 67675781f4..f8c36ffa2f 100644
--- a/target/arm/kvm.c
+++ b/target/arm/kvm.c
@@ -489,6 +489,11 @@ static const Property arm_cpu_kvm_compat_hidden_regs_property =
     DEFINE_PROP_ARRAY("kvm-hidden-regs", ARMCPU,
                       nr_kvm_hidden_regs, kvm_hidden_regs, qdev_prop_uint64, uint64_t);
 
+static const Property arm_cpu_kvm_compat_enforced_regs_property =
+    DEFINE_PROP_ARRAY("kvm-enforced-regs", ARMCPU,
+                      nr_kvm_enforced_regs, kvm_enforced_regs,
+                      qdev_prop_uint64, uint64_t);
+
 /* KVM VCPU properties should be prefixed with "kvm-". */
 void kvm_arm_add_vcpu_properties(ARMCPU *cpu)
 {
@@ -512,6 +517,7 @@ void kvm_arm_add_vcpu_properties(ARMCPU *cpu)
                                     "Set off to disable KVM steal time.");
 
     qdev_property_add_static(DEVICE(obj), &arm_cpu_kvm_compat_hidden_regs_property);
+    qdev_property_add_static(DEVICE(obj), &arm_cpu_kvm_compat_enforced_regs_property);
 }
 
 bool kvm_arm_pmu_supported(void)
@@ -772,6 +778,27 @@ static bool kvm_arm_reg_syncs_via_cpreg_list(uint64_t regidx)
     }
 }
 
+/**
+ * kvm_vcpu_compat_fake_reg:
+ * @cpu: ARMCPU
+ * @regidx: index of the register to check
+ *
+ * Depending on the CPU compat returns true if @regidx is a
+ * fake register that does not need any sync, false otherwise
+ */
+static inline bool
+kvm_vcpu_compat_fake_reg(ARMCPU *cpu, uint64_t regidx)
+{
+    for (int i = 0; i < cpu->nr_kvm_enforced_regs; i++) {
+        if (cpu->kvm_fake_regs[i] &&
+            cpu->kvm_enforced_regs[i] == regidx) {
+            trace_kvm_vcpu_compat_fake_reg(regidx);
+            return true;
+        }
+    }
+    return false;
+}
+
 /**
  * kvm_vcpu_compat_hidden_reg:
  * @cpu: ARMCPU
@@ -806,7 +833,8 @@ static int kvm_arm_init_cpreg_list(ARMCPU *cpu)
 {
     struct kvm_reg_list rl;
     struct kvm_reg_list *rlp;
-    int i, ret, arraylen;
+    int i, ret, arraylen, rln;
+    int nr_fake_regs = 0;
     CPUState *cs = CPU(cpu);
 
     rl.n = 0;
@@ -814,12 +842,34 @@ static int kvm_arm_init_cpreg_list(ARMCPU *cpu)
     if (ret != -E2BIG) {
         return ret;
     }
-    rlp = g_malloc(sizeof(struct kvm_reg_list) + rl.n * sizeof(uint64_t));
+    rln = rl.n + cpu->nr_kvm_enforced_regs;
+    rlp = g_malloc(sizeof(struct kvm_reg_list) + rln * sizeof(uint64_t));
     rlp->n = rl.n;
     ret = kvm_vcpu_ioctl(cs, KVM_GET_REG_LIST, rlp);
     if (ret) {
         goto out;
     }
+
+    trace_kvm_arm_init_cpreg_list(rlp->n, cpu->nr_kvm_enforced_regs);
+
+    cpu->kvm_fake_regs = g_new0(uint64_t, cpu->nr_kvm_enforced_regs);
+
+    for (int j = 0; j < cpu->nr_kvm_enforced_regs; j++) {
+        uint64_t v64;
+        int res;
+
+        res = kvm_get_one_reg(cs, cpu->kvm_enforced_regs[j], &v64);
+
+        if (res != -ENOENT) {
+            trace_kvm_arm_init_cpreg_exposed(cpu->kvm_enforced_regs[j], v64, res);
+            continue;
+        }
+        rlp->reg[j + rl.n] = cpu->kvm_enforced_regs[j];
+        cpu->kvm_fake_regs[j] = true;
+        nr_fake_regs++;
+    }
+    rlp->n = rl.n + nr_fake_regs;
+
     /* Sort the list we get back from the kernel, since cpreg_tuples
      * must be in strictly ascending order.
      */
@@ -912,6 +962,10 @@ bool write_kvmstate_to_list(ARMCPU *cpu)
         uint32_t v32;
         int ret;
 
+        if (kvm_vcpu_compat_fake_reg(cpu, regidx)) {
+            continue;
+        }
+
         switch (regidx & KVM_REG_SIZE_MASK) {
         case KVM_REG_SIZE_U32:
             ret = kvm_get_one_reg(cs, regidx, &v32);
@@ -947,6 +1001,10 @@ bool write_list_to_kvmstate(ARMCPU *cpu, int level)
             continue;
         }
 
+        if (kvm_vcpu_compat_fake_reg(cpu, regidx)) {
+            continue;
+        }
+
         switch (regidx & KVM_REG_SIZE_MASK) {
         case KVM_REG_SIZE_U32:
             v32 = cpu->cpreg_values[i];
@@ -1654,6 +1712,7 @@ static void kvm_arch_set_eager_split_size(Object *obj, Visitor *v,
     s->kvm_eager_split_size = value;
 }
 
+
 void kvm_arch_accel_class_init(ObjectClass *oc)
 {
     object_class_property_add(oc, "eager-split-size", "size",
diff --git a/target/arm/trace-events b/target/arm/trace-events
index 1b4ab0c683..57d2b6afd4 100644
--- a/target/arm/trace-events
+++ b/target/arm/trace-events
@@ -15,3 +15,7 @@ arm_gt_update_irq(int timer, int irqstate) "gt_update_irq: timer %d irqstate %d"
 kvm_arm_fixup_msi_route(uint64_t iova, uint64_t gpa) "MSI iova = 0x%"PRIx64" is translated into 0x%"PRIx64
 kvm_arm_init_cpreg_list_arraylen(uint32_t arraylen) "arraylen=%d"
 kvm_vcpu_compat_hidden_reg(uint64_t regidx) "0x%"PRIx64" is hidden"
+kvm_arm_init_cpreg_list(uint32_t kvm_regs, uint32_t fake_regs) "%d regs exposed by KVM, %d enforced regs"
+kvm_vcpu_compat_fake_reg(uint64_t regidx) "0x%"PRIx64" is fake"
+kvm_arm_init_cpreg_exposed(uint64_t regidx, uint64_t val, int ret) "enforced reg 0x%"PRIx64" is already exposed by KVM: value=0x%"PRIx64 " ret=%d: nothing to do"
+
-- 
2.49.0

[RFC 3/3] hw/arm/virt: [DO NOT UPSTREAM] Enforce compatibility with older kernels

[Eric Auger]

This is an example on how to use the new CPU options. This catters to
distributions who want machines to be migratable (forward and backward)
accross different host kernel versions in case KVM registers exposed
to qemu vary accross kernels. This patch is not meant to be upstreamed
as it is really kernel dependent. The goal is to illustrate how this
would be used.

In this example, For 10_1 machines types and older we apply the following
host kernel related compats:

1) Make sure the KVM_REG_ARM_VENDOR_HYP_BMAP_2 exposed from v6.15 onwards
   is ignored/hidden.
2) Make sure TCR_EL1, PIRE0_EL1, PIR_EL1 are always seen by qemu
   although not exposed by KVM. They were unconditionnally exposed before
   v6.13 while from v6.13 they are only exposed if supported by the guest.

This will allow 10_1 machines types and older machines to migrate
forward and backward from old downstream kernels that do not feature
those changes to newer kernels (>= v6.15).

Signed-off-by: Eric Auger <eric.auger@redhat.com>
---
 hw/arm/virt.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 6f01746e74..1cceddcd2a 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -96,6 +96,23 @@ static GlobalProperty arm_virt_compat[] = {
 };
 static const size_t arm_virt_compat_len = G_N_ELEMENTS(arm_virt_compat);
 
+/*
+ * if a 10_1 machine type or older is used:
+ * 1) make sure TCR_EL1, PIRE0_EL1, PIR_EL1 are enforced, even if they are not
+ *    exposed by the kernel
+ * 2) hide KVM_REG_ARM_VENDOR_HYP_BMAP_2
+ */
+static GlobalProperty arm_virt_kernel_compat_10_1[] = {
+    /* KVM_REG_ARM_VENDOR_HYP_BMAP_2 */
+    { TYPE_ARM_CPU, "kvm-hidden-regs", "0x6030000000160003" },
+    /* TCR_EL1, PIRE0_EL1, PIR_EL1 */
+    { TYPE_ARM_CPU, "kvm-enforced-regs",
+      "0x603000000013c103, 0x603000000013c512, 0x603000000013c513" },
+};
+static const size_t arm_virt_kernel_compat_10_1_len =
+    G_N_ELEMENTS(arm_virt_kernel_compat_10_1);
+
+
 /*
  * This cannot be called from the virt_machine_class_init() because
  * TYPE_VIRT_MACHINE is abstract and mc->compat_props g_ptr_array_new()
@@ -3462,6 +3479,8 @@ static void virt_machine_10_1_options(MachineClass *mc)
 {
     virt_machine_10_2_options(mc);
     compat_props_add(mc->compat_props, hw_compat_10_1, hw_compat_10_1_len);
+    compat_props_add(mc->compat_props,
+                     arm_virt_kernel_compat_10_1, arm_virt_kernel_compat_10_1_len);
 }
 DEFINE_VIRT_MACHINE(10, 1)
 
-- 
2.49.0

Re: [RFC 0/3] Mitigation of migration failures accross different host kernels

[Eric Auger]

Hi,

On 9/11/25 3:40 PM, Eric Auger wrote:
> When migrating ARM guests accross same machines with different host
> kernels we are likely to encounter failures such as:
>
> "failed to load cpu:cpreg_vmstate_array_len"
>
> This is due to the fact KVM exposes a different number of registers
> to qemu on source and destination. When trying to migrate a bigger
> register set to a smaller one, qemu cannot save the CPU state.
>
> For example, recently we faced such kind of situations with:
> - unconditionnal exposure of KVM_REG_ARM_VENDOR_HYP_BMAP_2 FW pseudo
>   register from v6.16 onwards. Causes backward migration failure.
> - removal of unconditionnal exposure of TCR2_EL1, PIRE0_EL1,  PIR_EL1
>   from v6.13 onwards. Causes forward migration failure.
>
> More details can be found in individual patches.
>
> This situation is really problematic for distributions which want to
> guarantee forward and backward migration of a given machine type
> between different releases.
>
> This small series tries to address that issue by introducing CPU
> array properties that list the registers to ignore or to fake according
> to the situation. An example is given to illustrate how those props
> could be used to apply compats for machine types supposed to "see" the
> same register set accross various host kernels.
>
> Obviously this is a last resort situation and this situation should be
> avoided as much as possible.

Gentle ping. Any other comments/advices on how to mitigate those kinds
of issue?
I think I will split the series because although it tries to address the
same

"failed to load cpu:cpreg_vmstate_array_len" class of error, hiding or faking KVM registers induce different side effects and risks and it may be better to handle them separately.

I forgot to mention that when registers disappear without notice/KVM knob, one could argue that the easiest way is to backport the fix in older kernels but there will always be a point when a VM running on a non fixed host kernel won't be live migratable to a kernel where the fix was backported, which kills the point of doing live migration I think.

Best Regards

Eric 

>
> Eric Auger (3):
>   target/arm/cpu: Add new CPU property for KVM regs to hide
>   target/arm/kvm: Add new CPU property for KVM regs to enforce
>   hw/arm/virt: [DO NOT UPSTREAM] Enforce compatibility with older
>     kernels
>
>  target/arm/cpu.h        | 15 +++++++
>  hw/arm/virt.c           | 19 ++++++++
>  target/arm/kvm.c        | 99 +++++++++++++++++++++++++++++++++++++++--
>  target/arm/trace-events |  6 +++
>  4 files changed, 135 insertions(+), 4 deletions(-)
>