Re: [RFC 21/21] arm/cpu-features: Document custom vcpu model

[Kashyap Chamarthy <kchamart@redhat.com>]

On Fri, Oct 25, 2024 at 12:17:40PM +0200, Eric Auger wrote:
> From: Cornelia Huck <cohuck@redhat.com>
> 
> Add some documentation for the custom model.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> Signed-off-by: Cornelia Huck <cohuck@redhat.com>
> ---
>  docs/system/arm/cpu-features.rst | 55 +++++++++++++++++++++++++++-----
>  1 file changed, 47 insertions(+), 8 deletions(-)
> 
> diff --git a/docs/system/arm/cpu-features.rst b/docs/system/arm/cpu-features.rst
> index a5fb929243..962a2c6c26 100644
> --- a/docs/system/arm/cpu-features.rst
> +++ b/docs/system/arm/cpu-features.rst
> @@ -2,7 +2,10 @@ Arm CPU Features

[...]

> +Using the ``host`` type means the guest is provided all the same CPU
> +features as the host CPU type has.  And, for this reason, the ``host``
> +CPU type should enable all CPU features that the host has by default.
> +
> +In case some features need to be hidden to the guest, ``custom`` model
> +shall be used instead. This is especially useful for migration purpose.
> +
> +The ``custom`` CPU model generally is the better choice if you want more
> +flexibility or stability across different machines or with different kernel
> +versions. 

Does "more flexibility or stability across different machines" also
imply "live migration compatiblity across host CPUs"?

> However, even the ``custom`` CPU model will not allow configuring
> +an arbitrary set of features; the ID registers must describe a subset of the
> +host's features, and all differences to the host's configuration must actually
> +be supported by the kernel to be deconfigured.

[...]

> +The ``custom`` CPU model needs to be configured via individual ID register
> +field properties, for example::
> +
> +  $ qemu-system-aarch64 -M virt -cpu custom,SYSREG_ID_AA64ISAR0_EL1_DP=0x0

If possible, it would be really helpful (and user-friendly) to be able
to specify the CPU feature names as you see under /proc/cpuinfo, and be
able to turn the flags on or off:
    
        -M virt -cpu franken,rndr=on,ts=on,fhm=off

(... instead of specifying long system register IDs that groups together
a bunch of CPU features.  If I understand it correctly, the register
"ID_AA64ISAR0_EL1" maps to a set of visible features listed here:
https://docs.kernel.org/arch/arm64/cpu-feature-registers.html)


Next, I prefix the below by noting that I wrote it before seeing
Cornelia's reply that the name "custom" is not set in stone:
https://lists.nongnu.org/archive/html/qemu-arm/2024-10/msg00987.html.

I wonder if the word "custom" is starting to get overloaded; on x86:

  - Libvirt itself uses the term "custom" this way, to quote its
    documentation[1] for the 'custom' XML attribute:

      custom
    
      In this mode, the 'cpu' element describes the CPU that should be
      presented to the guest. This is the default when no 'mode'
      attribute is specified. This mode makes it so that a persistent
      guest will see the same hardware no matter what host the guest is
      booted on.

  - Some management tools also follow libvirt and use the term "custom"
    to refer to one of two things, (a) a specific named CPU model that
    libvirt and QEMU recognize, e.g. "Cascadelake-Server"; or (b) a
    named CPU model + extra CPU flags, e.g. this is how OpenStack
    uses[2] "custom" to configure CPU models, and flags that can be
    enabled or disabled via "+" or "-":

      [libvirt]
      cpu_mode = custom
      cpu_model = IvyBridge-IBRS
      cpu_model_extra_flags="ss,+vmx,-pcid [...]"

    (Note the "cpu_mode" there: it is referring to the three possible
    modes that libvirt and QEMU support today: 'host-passthrough',
    'host-model', and named CPU models via "custom".)

    The above config translates to this QEMU command-line:

        -cpu IvyBridge-IBRS,ss=on,vmx=on,pcid=off [...]

Now if QEMU introduces "custom", it is likely to create some confusion.
But luckily, as referenced above, it is open to change. :)

    * * *

FWIW, I agree with Dan here[3] that it would cause less future pain if
Arm's named CPU models also decides on a "baseline that matches some
corresponding real world silicon".  I've experienced plenty of such
debugging pain in x86-land from years of troubleshooting live migration
bugs involving CPU model (in)compatibility.  (Often, with help from
DanPB and Jiri Denemark).

[1] https://docs.openstack.org/nova/latest/admin/cpu-models.html#cpu-modes
[2] https://libvirt.org/formatdomain.html#cpu-model-and-topology
[3] https://lists.nongnu.org/archive/html/qemu-arm/2024-10/msg00888.html
    — [RFC 21/21] arm/cpu-features: Document custom vcpu model

[...]

-- 
/kashyap