Re: [PATCH RFCv2 00/20] kvm/arm: Introduce a customizable aarch64 KVM host model

[Kashyap Chamarthy <kchamart@redhat.com>]

On Thu, Dec 19, 2024 at 03:41:56PM +0000, Marc Zyngier wrote:
> On Thu, 19 Dec 2024 15:07:25 +0000,
> Kashyap Chamarthy <kchamart@redhat.com> wrote:
> > 
> > On Thu, Dec 19, 2024 at 12:26:29PM +0000, Marc Zyngier wrote:
> > > On Thu, 19 Dec 2024 11:35:16 +0000,
> > > Kashyap Chamarthy <kchamart@redhat.com> wrote:

[...]

> > > You can't rely on userspace for security, that'd be completely
> > > ludicrous.
> > 
> > As Dan Berrangé points out, it's the bog-standard way QEMU deals with
> > some of the CPU-related issues on x86 today.  See this "important CPU
> > flags"[2] section in the QEMU docs.
> 
> I had a look, and we do things quite differently. For example, the
> spec-ctrl equivalent in implemented in FW and in KVM, and is exposed
> by default if the HW is vulnerable. Userspace could hide that the
> mitigation is there, but that's the extent of the configurability.

Noted.  As Dan says, as long as QEMU can toggle the feature on/off, then
that might be sufficient in the context of migratability.

[...]

> > To reply to your other question on this thread[3] about "which ABI?"  I
> > think Dan is talking about the *guest* ABI: the virtual "chipset" that
> > is exposed to a guest (e.g. PCI(e) topology, ACPI tables, CPU model,
> > etc).  As I understand it, this "guest ABI" should remain predictable,
> > regardless of:
> > 
> >   - whether you're updating KVM, QEMU, or the underlying physical
> >     hardware itself; or
> >   - if the guest is migrated, live or offline
> > 
> > (As you might know, QEMU's "machine types" concept allows to create a
> > stable guest ABI.)
> 
> All of this is under control of QEMU, *except* for the "maximum" of
> the architectural features exposed to the guest. All you can do is
> *downgrade* from there, and only to a limited extent.
> 
> That, in turn has a direct impact on what you call the "CPU model",
> which for the ARM architecture really doesn't exist. All we have is a
> bag of discrete features, with intricate dependencies between them.

I see; thanks for this explanation.  Your last sentence above is the
shortest summary of the CPU features situation on ARM I've ever read so
far. 

So, I infer this from what you're saying (do correct if it's wrong):

  • Currently it is impractical (not feasible?) to pull together a
    minimal-and-usable set of CPU features + their dependencies on ARM
    to come up with a "CPU model" that can work across a reasonable set
    of hardware.

  • If the above is true, then the ability to toggle CPU features on and
    off might become even more important for QEMU — if it wants to be
    able to support live migration across mixed set of hardware on ARM.  

    NB: by "mixed set of hardware", I mean hardware that is *close
    enough* (e.g. among the "Ampere Altra Family" - BTW, this "family"
    seems to be only 2 systems far).  Not arbitrarily mixed.  I did read
    your response in this thread about "who in their right mind" would
    want to migrate from Nvidia "Grace" to "AmpereOne".

      https://lore.kernel.org/linux-arm-kernel/86y10ytpo6.wl-maz@kernel.org/
      — KVM: arm64: Make the exposed feature bits in AA64DFR0_EL1
      writable from userspace

> Even ignoring virtualisation: you can readily find two machines using
> the same CPUs (let's say Neoverse-N1), integrated by the same vendor
> (let's say, Ampere), in SoCs that bear the same name (Altra), and
> realise that they have a different feature set. Fun, isn't it?

Yikes!  I would use a different word, that starts with "m" and ends with
"s" (the resulting word rhymes with the latter) ;-)

    * * *

Related tangent on CPU feature discoverability on ARM:

Speaking of "Neoverse-N1", looking at a system that I have access to,
the `lscpu` output does not say anything about who the integrator is; it
only says:

    ...
    Vendor ID:                ARM
    Model name:             Neoverse-N1
    ...

I realize, `lscpu` displays only whatever the kernel knows.  Nothing in
`dmidecode` either.

Also, it looks like there's no equivalent of a "CPUID" instruction (I
realize it is x86-specific) on ARM.  Although, I came across a Google
Git repo that seems to implement a bespoke, "aarch64_cpuid".  From a
what I see, it seems to fetch the "Main ID Register" (MIDR_EL1) - I
don't know enough about it to understand its implications:

    https://github.com/google/cpu_features/blob/main/src/impl_aarch64_cpuid.c

> That's why I don't see CPU models as a viable thing in terms of ABI.
> They are an approximation of what you could have, but the ABI is
> elsewhere.

Hmm, this is "significant new information" for me.  If CPU models can't
be part of the guest ABI on ARM, then the whole "migratability across
heterogenous hardware" on QEMU requires deeper thinking.

Thanks for this discussion.

-- 
/kashyap