Re: [PATCH] Documentation: KVM: Document guest-visible compatibility expectations

[David Woodhouse <dwmw2@infradead.org>]

[-- Attachment #1: Type: text/plain, Size: 6837 bytes --]

On Wed, 2026-05-13 at 18:24 +0200, Paolo Bonzini wrote:
> Il mer 13 mag 2026, 15:57 David Woodhouse <dwmw2@infradead.org> ha scritto:
> > > x86 doesn't do bug-for-bug compatibility, thankfully - we have quirks
> > > but only 11 of them, or about one per year since we started adding them.
> > >   We only add quirks, generally speaking, when 1) we change the way file
> > > descriptors are initialized, 2) guests in the wild were relying on it,
> > > or 3) it prevends restoring state saved from an old kernel.  Is there
> > > anything else?
> > > 
> > > https://lore.kernel.org/kvm/e03f092dfbb7d391a6bf2797ba01e122ba080bcd.camel@infradead.org/
> > > is an example of a bug that "no SW can make any reasonable use of".
> > 
> > I actually believe that the focus on ICEBP was triggered by some weird
> > gaming software's anti-DRM mechanism, and that it *did* affect actual
> > guests in the wild?
> > 
> > But yeah, *fixing* it should not have any adverse effects. That's the
> > key.
> 
> Yep, so "bug for bug" is not it.

Of course. I'm not discriminating between 'bugs' and 'features'. In
this context I only care about guest-visible behaviour changes,
whatever the reason.

What I said was:
> > > > Once a behaviour is present in a released version of Linux/KVM, we
> > > > can't just declare it "wrong" and unilaterally impose a change in
> > > > guest-visible behaviour on *running* guests as a side-effect of a
> > > > kernel upgrade.

And yes, you're technically right to challenge that phrasing of it. It
does need the additional caveat of "...unless we are sure that changing
it in either direction underneath running guests cannot cause
problems", as discussed. That's the key for the ICEBP thing.

> > 
> > > And besides, both miss the point of *configurability* which is the basis of
> > > it all.
> > 
> > Hm, configurability *is* the point, I thought.
> 
> Yes, and configurability goes way beyond bugs/quirks, which are to
> some extent a red herring. Configurability for example says that "KVM:
> arm64: vgic: Allow userspace to set IIDR revision 1" shouldn't be
> controversial at all.

Indeed it shouldn't. And yet here we are.

> > > So we have the third case, "restoring state saved from an old kernel".
> > > If this case arises, I do believe that Arm will have to deal with it and
> > > introduce quirks or KVM_GET/SET_REG hacks.  Maybe it hasn't happened
> > > yet, lucky you.
> > 
> > We literally have those mechanisms already.
> 
> I am not talking about guest-visible changes across save/restore here,
> but rather about round-trips through userspace. For example, see the
> effect of KVM_X2APIC_API_USE_32BIT_IDS on KVM_GET/SET_LAPIC: it
> couldn't be made the default, because userspace expects to take old
> data returned by KVM_GET_LAPIC and shove it into KVM_SET_LAPIC. Sucks
> but can't be avoided.

Yes, you're right. And I fully expect and trust x86 to get that right
and not break existing userspace in any way at all.

But honestly, the bar for Arm is so low right now that anything I
physically *can* work around in userspace, I'm prepared to tolerate.

If KVM/arm did the equivalent of just changing the KVM_[SG]ET_LAPIC
data without the KVM_X2APIC_API_USE_32BIT_IDS trick, I wouldn't even
bat an eyelid; I'd just accommodate it and move on.

> > See commit https://git.kernel.org/torvalds/c/49a1a2c70a7f which adds a
> > new guest-visible feature in revision 3, but allowed userspace to
> > restore the old behaviour by setting it to revision 2. All my patch above does, is make it possible to set it to revision 1 as
> > well. Because https://git.kernel.org/torvalds/c/d53c2c29ae0d previously
> > changed the behaviour and bumped the default to 2 *without* allowing
> > userspace to restore the prior behaviour, and we've been carrying a
> > *revert* of that patch.
> > 
> > Why would we *not* accept such a patch?
> 
> Agreed. Even ignoring your revert, there's no reason why any upgrade
> past 49a1a2c70a7f has to be from after d53c2c29ae0d.
> 
> > Marc seems terribly insistent that we SHOULD NOT
> > restore the behaviour that older KVM offered to guests, and we MUST
> > change it unconditionally underneath running guests, making these
> > registers writable on upgrade... and reverting them to read-only for
> > running guests on a rollback.
> > 
> > And there we do have a very different viewpoint.
> 
> That's the design decision I mentioned, of not starting the guest
> configuration from a clean slate. I believe it complicates things
> because you have to design from the beginning with the ability to
> rollback to old versions and to potentially detect conflicts
> introduced by the rollback. This is exactly why
> KVM_X86_QUIRK_STUFF_FEATURE_MSRS was introduced: "KVM's initialization
> of feature MSRs during vCPU creation results in a failed save/restore
> of PERF_CAPABILITIES. If userspace configures the VM to _not_ have a
> PMU, because KVM initializes the vCPU's PERF_CAPABILITIES, trying to
> save/restore the non-zero value will be rejected by the destination."
> (https://lkml.org/lkml/2024/8/2/1032)

No, I don't think this is like that. In that case, IIUC it was at least
*possible* for userspace to manually filter out capabilities and adjust
things. But it kind of sucked if we *made* userspace do that and broke
things for existing userspace, so of *course* x86 did better.

I'm not even *dreaming* about a world where KVM/arm meets that bar.

> For Arm, however, it may be too late to change it; if not, I'll
> happily watch you argue with Marc about it. 

I'm not even going to try. You're right that it's the better option,
and it most certainly *isn't* too late for Arm to choose to be a stable
and mature platform providing continuity to userspace like x86 does.

But we are *so* far from that right now; we're fighting even to have
the *possibility* for userspace to remain compatible — even if
userspace *is* updated to know everything that the latest kernel
changed underneath it.

> But even without that,
> this doc patch (and the idea that "Where a new kernel introduces a
> guest-visible change, it provides a mechanism for userspace to select
> the previous behaviour") should be uncontroversial.

Indeed. And again, if you really want then you can add the caveat
discussed above, "unless you're really sure it won't make *any*
difference to the zoo of possible guests running Linux, Windows,
FreeBSD, or any number of random home-grown or network appliance
operating systems".

Although I didn't think it really needed spelling out in the doc, just
as I didn't think it needed spelling out earlier today (although you
called my sentence nonsense purely because it lacked that obvious
caveat, AFAICT).


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5069 bytes --]