Re: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates

[tarunsahu@google.com]

Sean Christopherson <seanjc@google.com> writes:

> On Fri, Aug 21, 2026, Pratyush Yadav wrote:
>> Hi Sean,
>> 
>> On Tue, Aug 18 2026, Sean Christopherson wrote:
>> >> Does this idea of "backwards compatibility" sound acceptable to you, at
>> >> least at a high level?
>> >
>> > No.
>> >
>> > It's probably fine for Google and other large companies that tightly control their
>> > kernels and use cases, and have the resources to juggle the resulting complexity,
>> > e.g. have kernel engineers on staff to track feature and dependencies, coordinate
>> > and plan kernel upgrades, etc.
>> >
>> > It's not acceptable for upstream, where downstream consumers often run a distro
>> > kernel, have much more varied use cases, and don't always have a horde of kernel
>> > engineers on staff to help them thread the needle you describe above.  And if
>> > supporting live update as a general feature for all users of the kernel isn't
>> > being factored into design considerations, then that needs to change, otherwise
>> > this is all dead in the water.
>> >
>> > I also don't see the point.  Maintaining a rigid save/restore ABI is annoying,
>> > but it's not _hard_ (or at least, not _that_ hard), especially if there's a set
>> > of well-documented best known practices that subsystems can follow, e.g. so that
>> > individual subsystems don't need to learn painful lessons first-hand.  I genuinely
>> > believe that maintaining the version hell you describe above would be more costly
>> > in the long run than simply committing to full backwards compatibility within a
>> > given subsystem.  I can imagine that enumerating what subsystems' information is
>> > in the payload will require a different scheme, but for a given subsystem, I don't
>> > see any reason to aim for anything less than full backwards compatibility.
>> 
>> Let's say for argument's sake that we commit for a fully stable
>> backwards compatible ABI. Even then, you have to deal with multiple ABI
>> versions.
>> 
>> Live update's ABI is more complex compared to KVM's save/restore ABI.
>> For the KVM save/restore uAPI, you are largely describing architectural
>> state like CPU registers, etc. These things don't evolve as fast and
>> more or less stay the same.
>
> Right, because nothing meaningful has changed in any architecture in the 20+
> years since KVM has provided save/restore support, whereas guest_memfd looks
> nothing like it did when it was introduced three years ago.
>
>> Live update needs to describe the state of kernel objects. These are
>> more complex
>
> LOL, you might be the first to claim x86 virtualization isn't all that complex.
>
>> and evolve faster.
>
> The speed at which things change doesn't automatically mean we shouldn't strive
> for backwards compatibility.  Yes, providing backwards compatibility requires
> additional care and planning, and over time *might* lead to an ABI that is
> difficult to maintain.  But IMO, that just makes it all the more important to
> get the design right the first time, not that we shouldn't even try because it's
> hard.
>
>> For example, say you merge guest_memfd preservation today. Some time
>> later, someone comes up with a more efficient data structure to track
>> the folios in the file. You _have_ to make a backwards-incompatible ABI
>> change to use this data structure.
>
> Only if those details bleed into the ABI/contract.  I actually have a concrete
> KVM (well, virtualization) example for this.
>
> Intel's VMX architecture disallows direct memory accesses to the VMCS, and instead
> requires software to access the VMCS via dedicated ISA, using architectural encoding
> numbers to reference VMCS fields.  I.e. VMX decouples how data is stored in memory
> (the data structures) from the ABI/contract with software (VMCS field encodings).
>
> This allows Intel to optimize the data structures to be more efficient and performant
> for each microarchitecture based on the features and properties of each uarch, all
> without breaking backwards/forwards compatibility with software.
>
> My favorite esoteric example is AR_BYTES packing.  For Haswell, Intel added an
> optimization in ucode to allow saving/loading segment register state in a single
> uop (IIRC).  The optimization was especially valuable for virtualization as it
> shaved cycles off the VM-Enter/VM-Exit hot paths.  A key piece of the optimization
> was it required the AR_BYTES metadata to be stored in 16 bits, but existing CPUs
> stored AR_BYTES using 32 bits in an "unpacked" format.
>
> Fortunately, because the in-memory representation was decoupled from the contract
> with software, Intel could pack AR_BYTES into 16 bits for Haswell+ and pack/unpack
> the data on VMWRITE/VMREAD, so that the format presented to software remained
> unchanged.
>
> Does VMX's decoupling of the in-memory represntation of a VMCS have downsides?
> Absolutely.  Most notably, it incurs extra complexity (in software and hardware)
> to achieve comparable performance to directly accessible data structures (AMD's
> VMCB and Hyper-V's eVMCS) for nested virtualization.  And I'm sure it has placed
> contraints on Intel's designs, and obviously introduces complexity into the
> overall system by adding a layer of indirection.  But IMO the VMX architecture
> has been a huge win overall for Intel.
>
>> Or say you add a new memory backend (like the HugeTLB patches in
>> flight). That likely will need a different ABI to describe the state of
>> the guest_memfd.
>> 
>> So you will end up with multiple ABI versions that aren't always
>> backwards compatible.
>
> No, you end up with *features* that aren't backwards compatible.  I can't imagine
> anyone will argue that we should never add new features because then we can't
> rollback to an older kernel.
>
> But adding a new feature shouldn't break the existing ABI.  E.g. adding support
> for HugeTLB in guest_memfd shouldn't prevent rolling back to an older kernel when
> the HugeTLB functionality isn't being used.
>
> Using AMD's VMCB and Intel's VMCS as examples, literally every major new AMD/Intel
> uarch extends the VMC{B,S} in some way, but without fail it's always done in a way
> that is backwards compatible with existing software.  I.e. AMD and Intel ship new
> features, but existing software continues to work, and VMs continue to be migratable
> across CPU generations[*], with the obvious restriction that migrating a VM using
> a feature introduced on generation N to a generation N-1 CPU isn't a smart idea.
>
> [*] There are exceptions.  E.g. Intel removed MPX, and so VMs with MPX can't be
>     migrated to newer CPUs.  Migrating between CPUs with different MAXPHYADDR is
>     sketchy (and simply not done by some CSPs) because neither AMD nor Intel
>     virtualizes MAXPHYADDR.  But those exceptions are absolutely Big Deals that
>     undergo significant scrutiny, from all parties involved.
>
>> If you refuse that idea too, then KVM live update will be dead in the
>> water for a different reason. It will be damn near useless because it
>> can't keep up with an evolving subsystem.
>> 
>> Now once you get multiple ABI versions and you can seamlessly go from
>> old to new one, say you have a version that was superseded 5 years ago.
>> It would be completely reasonable to say that this version is old enough
>> and no one should be going from a 5 year old kernel to a modern one.
>
> LOL, Google literally does this.  Granted, the extreme cases only happen for
> stragglers, and I think we do force VMs to bounce through a "middle" kernel in
> those cases, but I doubt Google is the only company that runs frankenkernels
> for an absurd number of years for a variety of reasons.  E.g. 4.4 LTS was officially
> supported for 6 years, and I'll bet dollars to donuts people ran it for much longer
> than that.
>
>> So you deprecate this ABI. Deprecating old unused uAPIs is not unprecedented.
>
> When there are provably no users, or we can convince existing userspace to migrate
> to an alternative.
>
>> I think we are better off formalizing this deprecation period from the
>> get go.
>
> Why?  What does it buy us?  Because all I see is potential abuse and an excuse
> for not spending time getting the designs right.
>
>> A somewhat tangential example is BPF kfuncs. My BPF program that works
>> in kernel X might not work in kernel Y because the kfunc has changed or
>> been removed.
>> 
>> The argument they make in kfuncs.rst is that kfuncs "provide a kernel
>> <-> kernel API, and thus are not bound by any of the strict stability
>> restrictions associated with kernel <-> user UAPIs".
>
> BPF's documentation isn't arguing anything, it's merely reiterating Linux's
> long-standing policy that there is no such thing as a stable kernel ABI (in upstream).
>
>> For LUO as well, this is a kernel -> kernel API.
>
> Stating the obvious, I disagree with this.  As I said before, if this is the
> stance LUO wants to take, then so be it, but my NAK stands.
>
>> Users can also still do a regular kexec or reboot. They just won't get the
>> performance optimization of LUO.
>
> The amount of time, energy, and money poured into minimizing VM downtime on live
> migration suggests the overwhelming majority of LUO's targeted users aren't going
> to take kindly to this stance.
>
>> Regardless of if you agree with the last bit about deprecating old
>> versions, ABIs evolving with the subsystem is a ground reality of live
>> update and it would be foolish to think we can do with only
>> backwards-compatible ABI changes forever.
>
> I never said the ABI is immutable, I said it needs to be backwards/forwards
> compatible.


Thanks Sean, Pratyush. @loganodell has recently sent the RFC[1] on
backward compatibility for liveupdate. This discussion is important
so I propose to move the backward compatibility discussion on [1].

For guest_memfd preservation, I will sent v5 soon with the following
changes:
1. Suggestions from across the patches
2. tentative Plan for future Guest_memfd preservation
3. tentative Plan for VM preservation.
4. Backward compatibility built on top of RFC [1]

What do you think?

[1]: https://lore.kernel.org/all/20260903023452.721732-1-loganodell@google.com/