[PATCH] Documentation: KVM: Document guest-visible compatibility expectations

[PATCH] Documentation: KVM: Document guest-visible compatibility expectations

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 4433 bytes --]

From: David Woodhouse <dwmw@amazon.co.uk>

Document the expectation that KVM maintains guest-visible compatibility
across host kernel upgrades and rollbacks.  Specifically:

 - State saved/restored via KVM ioctls must be sufficient for live
   migration (and live update) between kernel versions.

 - Where a new kernel introduces a guest-visible change, it provides a
   mechanism for userspace to select the previous behaviour.

 - This allows both forward migration (upgrade) and backward migration
   (rollback) of guests.

These expectations have been implicitly required on x86 but were not
explicitly documented. Harmonise the expectations across all of KVM.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 Documentation/virt/kvm/api.rst              | 14 ++++++++++++++
 Documentation/virt/kvm/review-checklist.rst | 20 ++++++++++++++------
 2 files changed, 28 insertions(+), 6 deletions(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index 269970221797..864f3daa7acb 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -97,6 +97,20 @@ Instead, kvm defines extension identifiers and a facility to query
 whether a particular extension identifier is available.  If it is, a
 set of ioctls is available for application use.
 
+KVM will ensure that the state that can be saved and restored via the
+KVM ioctls is sufficient to allow migration of a running guest between
+host kernels while maintaining full compatibility of the guest-visible
+device model.  This includes migration to newer kernels (upgrade) and
+to older kernels (rollback), provided that the older kernel supports
+the set of features exposed to the guest.  Where a new kernel version
+introduces a guest-visible change, it will provide a mechanism (such
+as a capability or a device attribute) that allows userspace to select
+the previous behaviour.  This serves two purposes: guests migrated
+from an older kernel can continue to run with their original
+observable environment, and new guests launched on the newer kernel
+can be configured to match the feature set of the older kernel, so
+that they remain migratable to the older kernel in case of rollback.
+
 
 4. API description
 ==================
diff --git a/Documentation/virt/kvm/review-checklist.rst b/Documentation/virt/kvm/review-checklist.rst
index 053f00c50d66..f0fbe1577a90 100644
--- a/Documentation/virt/kvm/review-checklist.rst
+++ b/Documentation/virt/kvm/review-checklist.rst
@@ -18,22 +18,30 @@ Review checklist for kvm patches
 5.  New features must default to off (userspace should explicitly request them).
     Performance improvements can and should default to on.
 
-6.  New cpu features should be exposed via KVM_GET_SUPPORTED_CPUID2,
+6.  Guest-visible changes must not break migration compatibility.  A guest
+    migrated from an older kernel must be able to run with its original
+    observable environment, and a guest launched on a newer kernel must be
+    configurable to match the older kernel's feature set for rollback.
+    Where a change alters guest-visible behaviour, provide a mechanism
+    (capability, device attribute, etc.) for userspace to select the
+    previous behaviour.
+
+7.  New cpu features should be exposed via KVM_GET_SUPPORTED_CPUID2,
     or its equivalent for non-x86 architectures
 
-7.  The feature should be testable (see below).
+8.  The feature should be testable (see below).
 
-8.  Changes should be vendor neutral when possible.  Changes to common code
+9.  Changes should be vendor neutral when possible.  Changes to common code
     are better than duplicating changes to vendor code.
 
-9.  Similarly, prefer changes to arch independent code than to arch dependent
+10. Similarly, prefer changes to arch independent code than to arch dependent
     code.
 
-10. User/kernel interfaces and guest/host interfaces must be 64-bit clean
+11. User/kernel interfaces and guest/host interfaces must be 64-bit clean
     (all variables and sizes naturally aligned on 64-bit; use specific types
     only - u64 rather than ulong).
 
-11. New guest visible features must either be documented in a hardware manual
+12. New guest visible features must either be documented in a hardware manual
     or be accompanied by documentation.
 
 Testing of KVM code
-- 
2.43.0



[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5069 bytes --]

Re: [PATCH] Documentation: KVM: Document guest-visible compatibility expectations

[Paolo Bonzini]

On 5/11/26 10:57, David Woodhouse wrote:
> From: David Woodhouse <dwmw@amazon.co.uk>
> 
> Document the expectation that KVM maintains guest-visible compatibility
> across host kernel upgrades and rollbacks.  Specifically:
> 
>   - State saved/restored via KVM ioctls must be sufficient for live
>     migration (and live update) between kernel versions.
> 
>   - Where a new kernel introduces a guest-visible change, it provides a
>     mechanism for userspace to select the previous behaviour.
> 
>   - This allows both forward migration (upgrade) and backward migration
>     (rollback) of guests.
> 
> These expectations have been implicitly required on x86 but were not
> explicitly documented. Harmonise the expectations across all of KVM.

One big part of achieving this on x86 is the handling of CPUID.  Despite 
all the mess that KVM_SET_CPUID2 is (and sometimes the underlying 
architecture too, as Jim Mattson would certainly agree), KVM is 
generally able to provide a consistent view of its configuration to the 
guest.  This doesn't quite extend to compatibility across vendors, but 
it does work across processor generations from either Intel or AMD.

I understand that Arm traditionally had much more trouble than x86 with 
vendor-specified behavior that goes beyond the set of architectural 
features, so we may need to tune the expectations.  However, I agree 
with David that this is needed at least as long as the host CPU does not 
change.

Thanks,

Paolo

> Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
> ---
>   Documentation/virt/kvm/api.rst              | 14 ++++++++++++++
>   Documentation/virt/kvm/review-checklist.rst | 20 ++++++++++++++------
>   2 files changed, 28 insertions(+), 6 deletions(-)
> 
> diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
> index 269970221797..864f3daa7acb 100644
> --- a/Documentation/virt/kvm/api.rst
> +++ b/Documentation/virt/kvm/api.rst
> @@ -97,6 +97,20 @@ Instead, kvm defines extension identifiers and a facility to query
>   whether a particular extension identifier is available.  If it is, a
>   set of ioctls is available for application use.
>   
> +KVM will ensure that the state that can be saved and restored via the
> +KVM ioctls is sufficient to allow migration of a running guest between
> +host kernels while maintaining full compatibility of the guest-visible
> +device model.  This includes migration to newer kernels (upgrade) and
> +to older kernels (rollback), provided that the older kernel supports
> +the set of features exposed to the guest.  Where a new kernel version
> +introduces a guest-visible change, it will provide a mechanism (such
> +as a capability or a device attribute) that allows userspace to select
> +the previous behaviour.  This serves two purposes: guests migrated
> +from an older kernel can continue to run with their original
> +observable environment, and new guests launched on the newer kernel
> +can be configured to match the feature set of the older kernel, so
> +that they remain migratable to the older kernel in case of rollback.
> +
>   
>   4. API description
>   ==================
> diff --git a/Documentation/virt/kvm/review-checklist.rst b/Documentation/virt/kvm/review-checklist.rst
> index 053f00c50d66..f0fbe1577a90 100644
> --- a/Documentation/virt/kvm/review-checklist.rst
> +++ b/Documentation/virt/kvm/review-checklist.rst
> @@ -18,22 +18,30 @@ Review checklist for kvm patches
>   5.  New features must default to off (userspace should explicitly request them).
>       Performance improvements can and should default to on.
>   
> -6.  New cpu features should be exposed via KVM_GET_SUPPORTED_CPUID2,
> +6.  Guest-visible changes must not break migration compatibility.  A guest
> +    migrated from an older kernel must be able to run with its original
> +    observable environment, and a guest launched on a newer kernel must be
> +    configurable to match the older kernel's feature set for rollback.
> +    Where a change alters guest-visible behaviour, provide a mechanism
> +    (capability, device attribute, etc.) for userspace to select the
> +    previous behaviour.
> +
> +7.  New cpu features should be exposed via KVM_GET_SUPPORTED_CPUID2,
>       or its equivalent for non-x86 architectures
>   
> -7.  The feature should be testable (see below).
> +8.  The feature should be testable (see below).
>   
> -8.  Changes should be vendor neutral when possible.  Changes to common code
> +9.  Changes should be vendor neutral when possible.  Changes to common code
>       are better than duplicating changes to vendor code.
>   
> -9.  Similarly, prefer changes to arch independent code than to arch dependent
> +10. Similarly, prefer changes to arch independent code than to arch dependent
>       code.
>   
> -10. User/kernel interfaces and guest/host interfaces must be 64-bit clean
> +11. User/kernel interfaces and guest/host interfaces must be 64-bit clean
>       (all variables and sizes naturally aligned on 64-bit; use specific types
>       only - u64 rather than ulong).
>   
> -11. New guest visible features must either be documented in a hardware manual
> +12. New guest visible features must either be documented in a hardware manual
>       or be accompanied by documentation.
>   
>   Testing of KVM code

Re: [PATCH] Documentation: KVM: Document guest-visible compatibility expectations

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 2432 bytes --]

On Mon, 2026-05-11 at 17:14 +0200, Paolo Bonzini wrote:
> On 5/11/26 10:57, David Woodhouse wrote:
> > From: David Woodhouse <dwmw@amazon.co.uk>
> > 
> > Document the expectation that KVM maintains guest-visible compatibility
> > across host kernel upgrades and rollbacks.  Specifically:
> > 
> >   - State saved/restored via KVM ioctls must be sufficient for live
> >     migration (and live update) between kernel versions.
> > 
> >   - Where a new kernel introduces a guest-visible change, it provides a
> >     mechanism for userspace to select the previous behaviour.
> > 
> >   - This allows both forward migration (upgrade) and backward migration
> >     (rollback) of guests.
> > 
> > These expectations have been implicitly required on x86 but were not
> > explicitly documented. Harmonise the expectations across all of KVM.
> 
> One big part of achieving this on x86 is the handling of CPUID.  Despite 
> all the mess that KVM_SET_CPUID2 is (and sometimes the underlying 
> architecture too, as Jim Mattson would certainly agree), KVM is 
> generally able to provide a consistent view of its configuration to the 
> guest.  This doesn't quite extend to compatibility across vendors, but 
> it does work across processor generations from either Intel or AMD.

Right. For x86 this is largely covered by CPUID. If you launch a guest
on a new kernel, using the same CPUID bits as an older kernel, then
your guest will mostly not see anything new. And will be migratable to
that older kernel without issue.

Not *everything* is in CPUID; one recent exception that comes to mind
is the SUPPRESS_EOI_BROADCAST quirk. But on x86 we preserve the
existing behaviour of older kernels — even when that behaviour doesn't
make much sense, as with SUPPRESS_EOI_BROADCAST where older KVM would
*advertise* the feature, but not actually *implement* it. Nevertheless,
that remains the default behaviour of future kernels unless userspace
explicitly opts in to fully enable (or disable) the feature.

But this documentation update isn't even asking for that compatible-by-
default behaviour, even though that is the right thing to do. It's only
asking that it be *possible* to reinstate the old behaviour, for
userspace that *knows* about the change and explicitly wants to go back
to the old way to remain compatible.

And sadly, KVM/arm64 doesn't even meet *that* low bar.


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5069 bytes --]

Re: [PATCH] Documentation: KVM: Document guest-visible compatibility expectations

[Paolo Bonzini]

On 5/11/26 18:38, David Woodhouse wrote:
> Not *everything* is in CPUID; one recent exception that comes to mind
> is the SUPPRESS_EOI_BROADCAST quirk. But on x86 we preserve the
> existing behaviour of older kernels — even when that behaviour doesn't
> make much sense, as with SUPPRESS_EOI_BROADCAST where older KVM would
> *advertise* the feature, but not actually *implement* it. Nevertheless,
> that remains the default behaviour of future kernels unless userspace
> explicitly opts in to fully enable (or disable) the feature.
> 
> But this documentation update isn't even asking for that compatible-by-
> default behaviour, even though that is the right thing to do. It's only
> asking that it be *possible* to reinstate the old behaviour, for
> userspace that *knows* about the change and explicitly wants to go back
> to the old way to remain compatible.

Yep, these are the "quirks"---if it's too early for Arm to commit to 
that, I guess it's fine.

However, independent of this patch which I (obviously) believe is a good 
idea, I'd like to understand how far it is, assuming 1) no quirks 2) 
same CPU host.

By the way, you didn't Cc Marc...

Paolo

Re: [PATCH] Documentation: KVM: Document guest-visible compatibility expectations

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 2352 bytes --]

On Mon, 2026-05-11 at 18:56 +0200, Paolo Bonzini wrote:
> On 5/11/26 18:38, David Woodhouse wrote:
> > Not *everything* is in CPUID; one recent exception that comes to mind
> > is the SUPPRESS_EOI_BROADCAST quirk. But on x86 we preserve the
> > existing behaviour of older kernels — even when that behaviour doesn't
> > make much sense, as with SUPPRESS_EOI_BROADCAST where older KVM would
> > *advertise* the feature, but not actually *implement* it. Nevertheless,
> > that remains the default behaviour of future kernels unless userspace
> > explicitly opts in to fully enable (or disable) the feature.
> > 
> > But this documentation update isn't even asking for that compatible-by-
> > default behaviour, even though that is the right thing to do. It's only
> > asking that it be *possible* to reinstate the old behaviour, for
> > userspace that *knows* about the change and explicitly wants to go back
> > to the old way to remain compatible.
> 
> Yep, these are the "quirks"---if it's too early for Arm to commit to 
> that, I guess it's fine.
> 
> However, independent of this patch which I (obviously) believe is a good 
> idea, I'd like to understand how far it is, assuming 1) no quirks 2) 
> same CPU host.

It generally works out on arm64, although it's obviously a lot more
work than x86 which makes an effort to get this stuff right.

When we upgrade the kernel we do a lot of in-guest testing to find the
stuff that "broke", like cache reporting:
https://lore.kernel.org/all/254ca48a67779ccf9b9f60e2bb5796a305c03f95.camel@infradead.org/
... and the GICD_IIDR thing which I reposted today:
https://lore.kernel.org/all/20260511113558.3325004-2-dwmw2@infradead.org/

Those are the ones I came up against recently because someone had just
*reverted* the offending commits local in a previous kernel upgrade,
and I'm trying to fix it *properly* this time around and not carry the
reverts forward for ever.

And fix the expectations too, of course. Being told that we shouldn't
*expect* to be able to upgrade and roll back the kernel while remaining
compatible is... not OK.

> By the way, you didn't Cc Marc...

Ah crap, I meant to. Thanks for spotting that!

I must have screwed up when I combined and dedeuplicated the
get_maintainer.pl output with the recipients of the IIDR patch series.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5069 bytes --]

Re: [PATCH] Documentation: KVM: Document guest-visible compatibility expectations

[Marc Zyngier]

On Mon, 11 May 2026 17:56:15 +0100,
Paolo Bonzini <pbonzini@redhat.com> wrote:
> 
> On 5/11/26 18:38, David Woodhouse wrote:
> > Not *everything* is in CPUID; one recent exception that comes to mind
> > is the SUPPRESS_EOI_BROADCAST quirk. But on x86 we preserve the
> > existing behaviour of older kernels — even when that behaviour doesn't
> > make much sense, as with SUPPRESS_EOI_BROADCAST where older KVM would
> > *advertise* the feature, but not actually *implement* it. Nevertheless,
> > that remains the default behaviour of future kernels unless userspace
> > explicitly opts in to fully enable (or disable) the feature.
> > 
> > But this documentation update isn't even asking for that compatible-by-
> > default behaviour, even though that is the right thing to do. It's only
> > asking that it be *possible* to reinstate the old behaviour, for
> > userspace that *knows* about the change and explicitly wants to go back
> > to the old way to remain compatible.
> 
> Yep, these are the "quirks"---if it's too early for Arm to commit to
> that, I guess it's fine.

Compatible by default means nothing, because userspace needs to
discover the combined capabilities of the host and KVM. This is not a
"CPU model" architecture.

If userspace is not a total joke, it will read all the ID registers,
and configure what it wants to see, assuming it is a feature that can
be configured (not everything can, because the architecture itself is
not fully backward compatible).

Yes, this is buggy at times, because the combinatorial explosion of
CPU capabilities and supported features makes it pretty hard to test
(and really nobody actually does). But overall, it works, and QEMU is
growing an infrastructure to manage it in a "user friendly" way.

But really, this isn't what David is asking. He's demanding "bug for
bug" compatibility. For that, we have two possible cases:

- this is a behaviour that, while undesirable, is allowed by the
  architecture: fine, we preserve the behaviour and add another way to
  expose the one we really want. it is ugly, but we manage.

- this is a behaviour that is not allowed by the architecture: we fix
  it for good. We do that on every release. Some minor, some much more
  visible. And there is no way we will add this sort of "bring the
  bugs back" type of behaviours. Specially when it is really obvious
  that no SW can make any reasonable use of the defect. We allow
  userspace to keep behaving as before, but the guest will not see a
  non-compliant behaviour.

That being said, there is a way out of that: convince people in charge
of the architecture that the non-compliant KVM behaviour is actually
valuable, and deserves to be tolerated. This has happened before (VHE
only and NV2 only, just to name two recent changes).

Other terrible hacks (such as GICv3's GICD_TYPER.num_LPIs which KVM
doesn't support) were added at the request of cloud vendors that David
might be familiar with, so it isn't like it is a brand new process.

And once it is in the architecture, it becomes a behaviour that is
allowed to be exposed to a guest, for better or worse.

These are the rules we have followed since we started KVM/arm, and I
intend to stick to them.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH] Documentation: KVM: Document guest-visible compatibility expectations

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 4564 bytes --]

On Wed, 2026-05-13 at 09:42 +0100, Marc Zyngier wrote:
> On Mon, 11 May 2026 17:56:15 +0100,
> Paolo Bonzini <pbonzini@redhat.com> wrote:
> > 
> > On 5/11/26 18:38, David Woodhouse wrote:
> > > Not *everything* is in CPUID; one recent exception that comes to mind
> > > is the SUPPRESS_EOI_BROADCAST quirk. But on x86 we preserve the
> > > existing behaviour of older kernels — even when that behaviour doesn't
> > > make much sense, as with SUPPRESS_EOI_BROADCAST where older KVM would
> > > *advertise* the feature, but not actually *implement* it. Nevertheless,
> > > that remains the default behaviour of future kernels unless userspace
> > > explicitly opts in to fully enable (or disable) the feature.
> > > 
> > > But this documentation update isn't even asking for that compatible-by-
> > > default behaviour, even though that is the right thing to do. It's only
> > > asking that it be *possible* to reinstate the old behaviour, for
> > > userspace that *knows* about the change and explicitly wants to go back
> > > to the old way to remain compatible.
> > 
> > Yep, these are the "quirks"---if it's too early for Arm to commit to
> > that, I guess it's fine.
> 
> Compatible by default means nothing, because userspace needs to
> discover the combined capabilities of the host and KVM. This is not a
> "CPU model" architecture.
> 
> If userspace is not a total joke, it will read all the ID registers,
> and configure what it wants to see, assuming it is a feature that can
> be configured (not everything can, because the architecture itself is
> not fully backward compatible).
> 
> Yes, this is buggy at times, because the combinatorial explosion of
> CPU capabilities and supported features makes it pretty hard to test
> (and really nobody actually does). But overall, it works, and QEMU is
> growing an infrastructure to manage it in a "user friendly" way.

Yes, that is precisely what I'm asking for. I'm prepared to deal with
the fact that KVM/Arm64 is not a stable and mature platform like x86
is, and that userspace has to find all the random changes from one
version to the next, and explicitly pin things down to be compatible.

All I'm asking for is that KVM makes it *possible* to pin things down
to the behaviour of previously released Linux/KVM kernels.

> But really, this isn't what David is asking. He's demanding "bug for
> bug" compatibility. For that, we have two possible cases:

No, I am not asking you to meet that bar. I merely observed that x86
does and that it would be nice. But we are a *long* way from that.

> - this is a behaviour that, while undesirable, is allowed by the
>   architecture: fine, we preserve the behaviour and add another way to
>   expose the one we really want. it is ugly, but we manage.
> 
> - this is a behaviour that is not allowed by the architecture: we fix
>   it for good. We do that on every release. Some minor, some much more
>   visible. And there is no way we will add this sort of "bring the
>   bugs back" type of behaviours. Specially when it is really obvious
>   that no SW can make any reasonable use of the defect. We allow
>   userspace to keep behaving as before, but the guest will not see a
>   non-compliant behaviour.
> 
> That being said, there is a way out of that: convince people in charge
> of the architecture that the non-compliant KVM behaviour is actually
> valuable, and deserves to be tolerated. This has happened before (VHE
> only and NV2 only, just to name two recent changes).
> 
> Other terrible hacks (such as GICv3's GICD_TYPER.num_LPIs which KVM
> doesn't support) were added at the request of cloud vendors that David
> might be familiar with, so it isn't like it is a brand new process.
> 
> And once it is in the architecture, it becomes a behaviour that is
> allowed to be exposed to a guest, for better or worse.

Marc, this is complete nonsense and you should know better.

Once a behaviour is present in a released version of Linux/KVM, we
can't just declare it "wrong" and unilaterally impose a change in
guest-visible behaviour on *running* guests as a side-effect of a
kernel upgrade.

The criterion for *KVM* to remain compatible is "once it has been in a
released version of the kernel". Not "once it is in the architecture".

> These are the rules we have followed since we started KVM/arm, and I
> intend to stick to them.

Then KVM/arm is falling far short of the standards we expect of KVM and
of Linux in general.

Please do better.



[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5069 bytes --]

Re: [PATCH] Documentation: KVM: Document guest-visible compatibility expectations

[Paolo Bonzini]

On 5/13/26 11:24, David Woodhouse wrote:
> On Wed, 2026-05-13 at 09:42 +0100, Marc Zyngier wrote:
>> If userspace is not a total joke, it will read all the ID registers,
>> and configure what it wants to see, assuming it is a feature that can
>> be configured (not everything can, because the architecture itself is
>> not fully backward compatible).
>>
>> Yes, this is buggy at times, because the combinatorial explosion of
>> CPU capabilities and supported features makes it pretty hard to test
>> (and really nobody actually does). But overall, it works, and QEMU is
>> growing an infrastructure to manage it in a "user friendly" way.
> 
> Yes, that is precisely what I'm asking for. I'm prepared to deal with
> the fact that KVM/Arm64 is not a stable and mature platform like x86
> is, and that userspace has to find all the random changes from one
> version to the next, and explicitly pin things down to be compatible.
> 
> All I'm asking for is that KVM makes it *possible* to pin things down
> to the behaviour of previously released Linux/KVM kernels.
> 
>> But really, this isn't what David is asking. He's demanding "bug for
>> bug" compatibility. For that, we have two possible cases:
> 
> No, I am not asking you to meet that bar. I merely observed that x86
> does and that it would be nice. But we are a *long* way from that.

x86 doesn't do bug-for-bug compatibility, thankfully - we have quirks 
but only 11 of them, or about one per year since we started adding them. 
  We only add quirks, generally speaking, when 1) we change the way file 
descriptors are initialized, 2) guests in the wild were relying on it, 
or 3) it prevends restoring state saved from an old kernel.  Is there 
anything else?

So you're asking something not really far from this:

>> - this is a behaviour that is not allowed by the architecture: we fix
>>    it for good. We do that on every release. Some minor, some much more
>>    visible. And there is no way we will add this sort of "bring the
>>    bugs back" type of behaviours. Specially when it is really obvious
>>    that no SW can make any reasonable use of the defect. We allow
>>    userspace to keep behaving as before, but the guest will not see a
>>    non-compliant behaviour.

... where for example 
https://lore.kernel.org/kvm/e03f092dfbb7d391a6bf2797ba01e122ba080bcd.camel@infradead.org/ 
is an example of a bug that "no SW can make any reasonable use of".

> Marc, this is complete nonsense and you should know better.
> Once a behaviour is present in a released version of Linux/KVM, we
> can't just declare it "wrong" and unilaterally impose a change in
> guest-visible behaviour on *running* guests as a side-effect of a
> kernel upgrade.
> 
> The criterion for *KVM* to remain compatible is "once it has been in a
> released version of the kernel". Not "once it is in the architecture".

That is *also* obviously nonsense though, isn't it (see example above)? 
The truth is in the middle, "once it is in the architecture" is likely 
too narrow but "once it is in a Linux release" is way too broad.  And 
besides, both miss the point of *configurability* which is the basis of 
it all.

The main difference between x86 and Arm is the default state at 
creation; x86 defaults to a blank slate, mostly; and when we didn't do 
that, we regretted it later (cue the STUFF_FEATURE_MSRS quirk).  It's 
too late to change the behavior for Arm, but I think we can agree that 
patches such as 
https://lore.kernel.org/kvm/20260511113558.3325004-2-dwmw2@infradead.org/ 
("KVM: arm64: vgic: Allow userspace to set IIDR revision 1") are what 
the letter and spirit of this proposal is about.

Marc did not mention having to deal with guests in the wild.  Let's 
ignore it for now because even defining "guests in the wild" is hard; 
and anyway it's not related to the patch that triggered the discussion.

So we have the third case, "restoring state saved from an old kernel". 
If this case arises, I do believe that Arm will have to deal with it and 
introduce quirks or KVM_GET/SET_REG hacks.  Maybe it hasn't happened 
yet, lucky you.

Overall, even if we may disagree about the details, are we really on 
terribly distant grounds, or are we not?

Paolo

Re: [PATCH] Documentation: KVM: Document guest-visible compatibility expectations

[Eric Auger]

Hi,

On 5/13/26 2:43 PM, Paolo Bonzini wrote:
> On 5/13/26 11:24, David Woodhouse wrote:
>> On Wed, 2026-05-13 at 09:42 +0100, Marc Zyngier wrote:
>>> If userspace is not a total joke, it will read all the ID registers,
>>> and configure what it wants to see, assuming it is a feature that can
>>> be configured (not everything can, because the architecture itself is
>>> not fully backward compatible).
>>>
>>> Yes, this is buggy at times, because the combinatorial explosion of
>>> CPU capabilities and supported features makes it pretty hard to test
>>> (and really nobody actually does). But overall, it works, and QEMU is
>>> growing an infrastructure to manage it in a "user friendly" way.
>>
>> Yes, that is precisely what I'm asking for. I'm prepared to deal with
>> the fact that KVM/Arm64 is not a stable and mature platform like x86
>> is, and that userspace has to find all the random changes from one
>> version to the next, and explicitly pin things down to be compatible.
>>
>> All I'm asking for is that KVM makes it *possible* to pin things down
>> to the behaviour of previously released Linux/KVM kernels.
>>
>>> But really, this isn't what David is asking. He's demanding "bug for
>>> bug" compatibility. For that, we have two possible cases:
>>
>> No, I am not asking you to meet that bar. I merely observed that x86
>> does and that it would be nice. But we are a *long* way from that.
>
> x86 doesn't do bug-for-bug compatibility, thankfully - we have quirks
> but only 11 of them, or about one per year since we started adding
> them.  We only add quirks, generally speaking, when 1) we change the
> way file descriptors are initialized, 2) guests in the wild were
> relying on it, or 3) it prevends restoring state saved from an old
> kernel.  Is there anything else?
>
> So you're asking something not really far from this:
>
>>> - this is a behaviour that is not allowed by the architecture: we fix
>>>    it for good. We do that on every release. Some minor, some much more
>>>    visible. And there is no way we will add this sort of "bring the
>>>    bugs back" type of behaviours. Specially when it is really obvious
>>>    that no SW can make any reasonable use of the defect. We allow
>>>    userspace to keep behaving as before, but the guest will not see a
>>>    non-compliant behaviour.
>
> ... where for example
> https://lore.kernel.org/kvm/e03f092dfbb7d391a6bf2797ba01e122ba080bcd.camel@infradead.org/
> is an example of a bug that "no SW can make any reasonable use of".
>
>> Marc, this is complete nonsense and you should know better.
>> Once a behaviour is present in a released version of Linux/KVM, we
>> can't just declare it "wrong" and unilaterally impose a change in
>> guest-visible behaviour on *running* guests as a side-effect of a
>> kernel upgrade.
>>
>> The criterion for *KVM* to remain compatible is "once it has been in a
>> released version of the kernel". Not "once it is in the architecture".
>
> That is *also* obviously nonsense though, isn't it (see example
> above)? The truth is in the middle, "once it is in the architecture"
> is likely too narrow but "once it is in a Linux release" is way too
> broad.  And besides, both miss the point of *configurability* which is
> the basis of it all.
>
> The main difference between x86 and Arm is the default state at
> creation; x86 defaults to a blank slate, mostly; and when we didn't do
> that, we regretted it later (cue the STUFF_FEATURE_MSRS quirk).  It's
> too late to change the behavior for Arm, but I think we can agree that
> patches such as
> https://lore.kernel.org/kvm/20260511113558.3325004-2-dwmw2@infradead.org/
> ("KVM: arm64: vgic: Allow userspace to set IIDR revision 1") are what
> the letter and spirit of this proposal is about.
>
> Marc did not mention having to deal with guests in the wild.  Let's
> ignore it for now because even defining "guests in the wild" is hard;
> and anyway it's not related to the patch that triggered the discussion.
>
> So we have the third case, "restoring state saved from an old kernel".
> If this case arises, I do believe that Arm will have to deal with it
> and introduce quirks or KVM_GET/SET_REG hacks.  Maybe it hasn't
> happened yet, lucky you. 

for info, this qemu series was merged laterly.

[PATCH v10 0/7] Mitigation of "failed to load
cpu:cpreg_vmstate_array_len" migration failures <https://lore.kernel.org/all/20260420140552.104369-1-eric.auger@redhat.com/#r>
https://lore.kernel.org/all/20260420140552.104369-1-eric.auger@redhat.com/#r

It brings an infrastructure to mitigate some migration failures accross different kernel versions.

Also there is [PATCH v4 00/17] kvm/arm: Introduce a customizable aarch64 KVM host model, under review
https://lore.kernel.org/all/20260503073541.790215-1-eric.auger@redhat.com/

This series aims at beeing able to offer the capacity to set writable ID regs on the host passthrough vcpu model.

Thanks

Eric


>
> Overall, even if we may disagree about the details, are we really on
> terribly distant grounds, or are we not?
>
> Paolo
>

Re: [PATCH] Documentation: KVM: Document guest-visible compatibility expectations

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 7241 bytes --]

On Wed, 2026-05-13 at 14:43 +0200, Paolo Bonzini wrote:
> On 5/13/26 11:24, David Woodhouse wrote:
> > On Wed, 2026-05-13 at 09:42 +0100, Marc Zyngier wrote:
> > > If userspace is not a total joke, it will read all the ID registers,
> > > and configure what it wants to see, assuming it is a feature that can
> > > be configured (not everything can, because the architecture itself is
> > > not fully backward compatible).
> > > 
> > > Yes, this is buggy at times, because the combinatorial explosion of
> > > CPU capabilities and supported features makes it pretty hard to test
> > > (and really nobody actually does). But overall, it works, and QEMU is
> > > growing an infrastructure to manage it in a "user friendly" way.
> > 
> > Yes, that is precisely what I'm asking for. I'm prepared to deal with
> > the fact that KVM/Arm64 is not a stable and mature platform like x86
> > is, and that userspace has to find all the random changes from one
> > version to the next, and explicitly pin things down to be compatible.
> > 
> > All I'm asking for is that KVM makes it *possible* to pin things down
> > to the behaviour of previously released Linux/KVM kernels.
> > 
> > > But really, this isn't what David is asking. He's demanding "bug for
> > > bug" compatibility. For that, we have two possible cases:
> > 
> > No, I am not asking you to meet that bar. I merely observed that x86
> > does and that it would be nice. But we are a *long* way from that.
> 
> x86 doesn't do bug-for-bug compatibility, thankfully - we have quirks
> but only 11 of them, or about one per year since we started adding them. 
>   We only add quirks, generally speaking, when 1) we change the way file 
> descriptors are initialized, 2) guests in the wild were relying on it, 
> or 3) it prevends restoring state saved from an old kernel.  Is there
> anything else?
> 
> So you're asking something not really far from this:
> 
> > > - this is a behaviour that is not allowed by the architecture: we fix
> > >    it for good. We do that on every release. Some minor, some much more
> > >    visible. And there is no way we will add this sort of "bring the
> > >    bugs back" type of behaviours. Specially when it is really obvious
> > >    that no SW can make any reasonable use of the defect. We allow
> > >    userspace to keep behaving as before, but the guest will not see a
> > >    non-compliant behaviour.
> 
> ... where for example 
> https://lore.kernel.org/kvm/e03f092dfbb7d391a6bf2797ba01e122ba080bcd.camel@infradead.org/ 
> is an example of a bug that "no SW can make any reasonable use of".

I actually believe that the focus on ICEBP was triggered by some weird
gaming software's anti-DRM mechanism, and that it *did* affect actual
guests in the wild?

But yeah, *fixing* it should not have any adverse effects. That's the
key.

> > Marc, this is complete nonsense and you should know better.
> > Once a behaviour is present in a released version of Linux/KVM, we
> > can't just declare it "wrong" and unilaterally impose a change in
> > guest-visible behaviour on *running* guests as a side-effect of a
> > kernel upgrade.
> > 
> > The criterion for *KVM* to remain compatible is "once it has been in a
> > released version of the kernel". Not "once it is in the architecture".
> 
> That is *also* obviously nonsense though, isn't it (see example above)? 
> The truth is in the middle, "once it is in the architecture" is likely 
> too narrow but "once it is in a Linux release" is way too broad.

How about "once it is in a Linux release and guest visible, and unless
we *know* that changing it in either direction underneath running
guests cannot cause problems".

> And besides, both miss the point of *configurability* which is the basis of 
> it all.

Hm, configurability *is* the point, I thought. I'm not asking for the
*default* to remain compatible. I only ask that a VMM *can* ask KVM for
guest-visible things to remain the same as before.

> The main difference between x86 and Arm is the default state at 
> creation; x86 defaults to a blank slate, mostly; and when we didn't do 
> that, we regretted it later (cue the STUFF_FEATURE_MSRS quirk).  It's
> too late to change the behavior for Arm, but I think we can agree that 
> patches such as 
> https://lore.kernel.org/kvm/20260511113558.3325004-2-dwmw2@infradead.org/ 
> ("KVM: arm64: vgic: Allow userspace to set IIDR revision 1") are what
> the letter and spirit of this proposal is about.

Yes. That *exact* patch.

> Marc did not mention having to deal with guests in the wild.  Let's 
> ignore it for now because even defining "guests in the wild" is hard;
> and anyway it's not related to the patch that triggered the discussion.
>
> So we have the third case, "restoring state saved from an old kernel". 
> If this case arises, I do believe that Arm will have to deal with it and 
> introduce quirks or KVM_GET/SET_REG hacks.  Maybe it hasn't happened 
> yet, lucky you.

We literally have those mechanisms already. That's exactly what the
revision field in the IIDR is used for: 
https://developer.arm.com/documentation/111107/2026-03/External-Registers/GICD-IIDR--Distributor-Implementer-Identification-Register

See commit https://git.kernel.org/torvalds/c/49a1a2c70a7f which adds a
new guest-visible feature in revision 3, but allowed userspace to
restore the old behaviour by setting it to revision 2. (Or at least
intended to; there was a separate bug which stopped it working, which I
already fixed last week.)

All my patch above does, is make it possible to set it to revision 1 as
well. Because https://git.kernel.org/torvalds/c/d53c2c29ae0d previously
changed the behaviour and bumped the default to 2 *without* allowing
userspace to restore the prior behaviour, and we've been carrying a
*revert* of that patch.

So the patch we're arguing about is just making that earlier guest-
visible change optional in precisely the way that is already designed
into KVM, and has been used for the subsequent change.

Why would we *not* accept such a patch? 

It's not like I'm trying to upstream something like
https://david.woodhou.se/0001-Allow-writes-via-newly-readonly-PTE-for-buggy-Ubuntu.patch
... but yes, those *are* the lengths we have to go to sometimes to
ensure that when we upgrade the hosting environment, guests which have
worked for years don't suddenly break — however much they DESERVE to :)

> Overall, even if we may disagree about the details, are we really on 
> terribly distant grounds, or are we not?

I genuinely have no idea.

On one hand, no we are not terribly distant. All the mechanisms to do
this properly already *exist*, and the fix I'm asking for is not much
more than a one-liner to fix up the previous oversight.

But on the other hand, Marc seems terribly insistent that we SHOULD NOT
restore the behaviour that older KVM offered to guests, and we MUST
change it unconditionally underneath running guests, making these
registers writable on upgrade... and reverting them to read-only for
running guests on a rollback.

And there we do have a very different viewpoint.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5069 bytes --]

Re: [PATCH] Documentation: KVM: Document guest-visible compatibility expectations

[Paolo Bonzini]

Il mer 13 mag 2026, 15:57 David Woodhouse <dwmw2@infradead.org> ha scritto:
> > x86 doesn't do bug-for-bug compatibility, thankfully - we have quirks
> > but only 11 of them, or about one per year since we started adding them.
> >   We only add quirks, generally speaking, when 1) we change the way file
> > descriptors are initialized, 2) guests in the wild were relying on it,
> > or 3) it prevends restoring state saved from an old kernel.  Is there
> > anything else?
> >
> > https://lore.kernel.org/kvm/e03f092dfbb7d391a6bf2797ba01e122ba080bcd.camel@infradead.org/
> > is an example of a bug that "no SW can make any reasonable use of".
>
> I actually believe that the focus on ICEBP was triggered by some weird
> gaming software's anti-DRM mechanism, and that it *did* affect actual
> guests in the wild?
>
> But yeah, *fixing* it should not have any adverse effects. That's the
> key.

Yep, so "bug for bug" is not it.

> > That is *also* obviously nonsense though, isn't it (see example above)?
> > The truth is in the middle, "once it is in the architecture" is likely
> > too narrow but "once it is in a Linux release" is way too broad.
>
> How about "once it is in a Linux release and guest visible, and unless
> we *know* that changing it in either direction underneath running
> guests cannot cause problems".
>
> > And besides, both miss the point of *configurability* which is the basis of
> > it all.
>
> Hm, configurability *is* the point, I thought.

Yes, and configurability goes way beyond bugs/quirks, which are to
some extent a red herring. Configurability for example says that "KVM:
arm64: vgic: Allow userspace to set IIDR revision 1" shouldn't be
controversial at all.

> > So we have the third case, "restoring state saved from an old kernel".
> > If this case arises, I do believe that Arm will have to deal with it and
> > introduce quirks or KVM_GET/SET_REG hacks.  Maybe it hasn't happened
> > yet, lucky you.
>
> We literally have those mechanisms already.

I am not talking about guest-visible changes across save/restore here,
but rather about round-trips through userspace. For example, see the
effect of KVM_X2APIC_API_USE_32BIT_IDS on KVM_GET/SET_LAPIC: it
couldn't be made the default, because userspace expects to take old
data returned by KVM_GET_LAPIC and shove it into KVM_SET_LAPIC. Sucks
but can't be avoided.

> See commit https://git.kernel.org/torvalds/c/49a1a2c70a7f which adds a
> new guest-visible feature in revision 3, but allowed userspace to
> restore the old behaviour by setting it to revision 2. All my patch above does, is make it possible to set it to revision 1 as
> well. Because https://git.kernel.org/torvalds/c/d53c2c29ae0d previously
> changed the behaviour and bumped the default to 2 *without* allowing
> userspace to restore the prior behaviour, and we've been carrying a
> *revert* of that patch.
>
> Why would we *not* accept such a patch?

Agreed. Even ignoring your revert, there's no reason why any upgrade
past 49a1a2c70a7f has to be from after d53c2c29ae0d.

> Marc seems terribly insistent that we SHOULD NOT
> restore the behaviour that older KVM offered to guests, and we MUST
> change it unconditionally underneath running guests, making these
> registers writable on upgrade... and reverting them to read-only for
> running guests on a rollback.
>
> And there we do have a very different viewpoint.

That's the design decision I mentioned, of not starting the guest
configuration from a clean slate. I believe it complicates things
because you have to design from the beginning with the ability to
rollback to old versions and to potentially detect conflicts
introduced by the rollback. This is exactly why
KVM_X86_QUIRK_STUFF_FEATURE_MSRS was introduced: "KVM's initialization
of feature MSRs during vCPU creation results in a failed save/restore
of PERF_CAPABILITIES. If userspace configures the VM to _not_ have a
PMU, because KVM initializes the vCPU's PERF_CAPABILITIES, trying to
save/restore the non-zero value will be rejected by the destination."
(https://lkml.org/lkml/2024/8/2/1032)

For Arm, however, it may be too late to change it; if not, I'll
happily watch you argue with Marc about it. But even without that,
this doc patch (and the idea that "Where a new kernel introduces a
guest-visible change, it provides a mechanism for userspace to select
the previous behaviour") should be uncontroversial.

Paolo

Re: [PATCH] Documentation: KVM: Document guest-visible compatibility expectations

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 6837 bytes --]

On Wed, 2026-05-13 at 18:24 +0200, Paolo Bonzini wrote:
> Il mer 13 mag 2026, 15:57 David Woodhouse <dwmw2@infradead.org> ha scritto:
> > > x86 doesn't do bug-for-bug compatibility, thankfully - we have quirks
> > > but only 11 of them, or about one per year since we started adding them.
> > >   We only add quirks, generally speaking, when 1) we change the way file
> > > descriptors are initialized, 2) guests in the wild were relying on it,
> > > or 3) it prevends restoring state saved from an old kernel.  Is there
> > > anything else?
> > > 
> > > https://lore.kernel.org/kvm/e03f092dfbb7d391a6bf2797ba01e122ba080bcd.camel@infradead.org/
> > > is an example of a bug that "no SW can make any reasonable use of".
> > 
> > I actually believe that the focus on ICEBP was triggered by some weird
> > gaming software's anti-DRM mechanism, and that it *did* affect actual
> > guests in the wild?
> > 
> > But yeah, *fixing* it should not have any adverse effects. That's the
> > key.
> 
> Yep, so "bug for bug" is not it.

Of course. I'm not discriminating between 'bugs' and 'features'. In
this context I only care about guest-visible behaviour changes,
whatever the reason.

What I said was:
> > > > Once a behaviour is present in a released version of Linux/KVM, we
> > > > can't just declare it "wrong" and unilaterally impose a change in
> > > > guest-visible behaviour on *running* guests as a side-effect of a
> > > > kernel upgrade.

And yes, you're technically right to challenge that phrasing of it. It
does need the additional caveat of "...unless we are sure that changing
it in either direction underneath running guests cannot cause
problems", as discussed. That's the key for the ICEBP thing.

> > 
> > > And besides, both miss the point of *configurability* which is the basis of
> > > it all.
> > 
> > Hm, configurability *is* the point, I thought.
> 
> Yes, and configurability goes way beyond bugs/quirks, which are to
> some extent a red herring. Configurability for example says that "KVM:
> arm64: vgic: Allow userspace to set IIDR revision 1" shouldn't be
> controversial at all.

Indeed it shouldn't. And yet here we are.

> > > So we have the third case, "restoring state saved from an old kernel".
> > > If this case arises, I do believe that Arm will have to deal with it and
> > > introduce quirks or KVM_GET/SET_REG hacks.  Maybe it hasn't happened
> > > yet, lucky you.
> > 
> > We literally have those mechanisms already.
> 
> I am not talking about guest-visible changes across save/restore here,
> but rather about round-trips through userspace. For example, see the
> effect of KVM_X2APIC_API_USE_32BIT_IDS on KVM_GET/SET_LAPIC: it
> couldn't be made the default, because userspace expects to take old
> data returned by KVM_GET_LAPIC and shove it into KVM_SET_LAPIC. Sucks
> but can't be avoided.

Yes, you're right. And I fully expect and trust x86 to get that right
and not break existing userspace in any way at all.

But honestly, the bar for Arm is so low right now that anything I
physically *can* work around in userspace, I'm prepared to tolerate.

If KVM/arm did the equivalent of just changing the KVM_[SG]ET_LAPIC
data without the KVM_X2APIC_API_USE_32BIT_IDS trick, I wouldn't even
bat an eyelid; I'd just accommodate it and move on.

> > See commit https://git.kernel.org/torvalds/c/49a1a2c70a7f which adds a
> > new guest-visible feature in revision 3, but allowed userspace to
> > restore the old behaviour by setting it to revision 2. All my patch above does, is make it possible to set it to revision 1 as
> > well. Because https://git.kernel.org/torvalds/c/d53c2c29ae0d previously
> > changed the behaviour and bumped the default to 2 *without* allowing
> > userspace to restore the prior behaviour, and we've been carrying a
> > *revert* of that patch.
> > 
> > Why would we *not* accept such a patch?
> 
> Agreed. Even ignoring your revert, there's no reason why any upgrade
> past 49a1a2c70a7f has to be from after d53c2c29ae0d.
> 
> > Marc seems terribly insistent that we SHOULD NOT
> > restore the behaviour that older KVM offered to guests, and we MUST
> > change it unconditionally underneath running guests, making these
> > registers writable on upgrade... and reverting them to read-only for
> > running guests on a rollback.
> > 
> > And there we do have a very different viewpoint.
> 
> That's the design decision I mentioned, of not starting the guest
> configuration from a clean slate. I believe it complicates things
> because you have to design from the beginning with the ability to
> rollback to old versions and to potentially detect conflicts
> introduced by the rollback. This is exactly why
> KVM_X86_QUIRK_STUFF_FEATURE_MSRS was introduced: "KVM's initialization
> of feature MSRs during vCPU creation results in a failed save/restore
> of PERF_CAPABILITIES. If userspace configures the VM to _not_ have a
> PMU, because KVM initializes the vCPU's PERF_CAPABILITIES, trying to
> save/restore the non-zero value will be rejected by the destination."
> (https://lkml.org/lkml/2024/8/2/1032)

No, I don't think this is like that. In that case, IIUC it was at least
*possible* for userspace to manually filter out capabilities and adjust
things. But it kind of sucked if we *made* userspace do that and broke
things for existing userspace, so of *course* x86 did better.

I'm not even *dreaming* about a world where KVM/arm meets that bar.

> For Arm, however, it may be too late to change it; if not, I'll
> happily watch you argue with Marc about it. 

I'm not even going to try. You're right that it's the better option,
and it most certainly *isn't* too late for Arm to choose to be a stable
and mature platform providing continuity to userspace like x86 does.

But we are *so* far from that right now; we're fighting even to have
the *possibility* for userspace to remain compatible — even if
userspace *is* updated to know everything that the latest kernel
changed underneath it.

> But even without that,
> this doc patch (and the idea that "Where a new kernel introduces a
> guest-visible change, it provides a mechanism for userspace to select
> the previous behaviour") should be uncontroversial.

Indeed. And again, if you really want then you can add the caveat
discussed above, "unless you're really sure it won't make *any*
difference to the zoo of possible guests running Linux, Windows,
FreeBSD, or any number of random home-grown or network appliance
operating systems".

Although I didn't think it really needed spelling out in the doc, just
as I didn't think it needed spelling out earlier today (although you
called my sentence nonsense purely because it lacked that obvious
caveat, AFAICT).


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5069 bytes --]