Re: [RFC 21/21] arm/cpu-features: Document custom vcpu model

[Eric Auger <eric.auger@redhat.com>]

Hi Kashyap,

On 10/28/24 22:17, Kashyap Chamarthy wrote:
> On Fri, Oct 25, 2024 at 12:17:40PM +0200, Eric Auger wrote:
>> From: Cornelia Huck <cohuck@redhat.com>
>>
>> Add some documentation for the custom model.
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>> Signed-off-by: Cornelia Huck <cohuck@redhat.com>
>> ---
>>  docs/system/arm/cpu-features.rst | 55 +++++++++++++++++++++++++++-----
>>  1 file changed, 47 insertions(+), 8 deletions(-)
>>
>> diff --git a/docs/system/arm/cpu-features.rst b/docs/system/arm/cpu-features.rst
>> index a5fb929243..962a2c6c26 100644
>> --- a/docs/system/arm/cpu-features.rst
>> +++ b/docs/system/arm/cpu-features.rst
>> @@ -2,7 +2,10 @@ Arm CPU Features
> [...]
>
>> +Using the ``host`` type means the guest is provided all the same CPU
>> +features as the host CPU type has.  And, for this reason, the ``host``
>> +CPU type should enable all CPU features that the host has by default.
>> +
>> +In case some features need to be hidden to the guest, ``custom`` model
>> +shall be used instead. This is especially useful for migration purpose.
>> +
>> +The ``custom`` CPU model generally is the better choice if you want more
>> +flexibility or stability across different machines or with different kernel
>> +versions. 
> Does "more flexibility or stability across different machines" also
> imply "live migration compatiblity across host CPUs"?
yes that's the goal
>
>> However, even the ``custom`` CPU model will not allow configuring
>> +an arbitrary set of features; the ID registers must describe a subset of the
>> +host's features, and all differences to the host's configuration must actually
>> +be supported by the kernel to be deconfigured.
> [...]
>
>> +The ``custom`` CPU model needs to be configured via individual ID register
>> +field properties, for example::
>> +
>> +  $ qemu-system-aarch64 -M virt -cpu custom,SYSREG_ID_AA64ISAR0_EL1_DP=0x0
> If possible, it would be really helpful (and user-friendly) to be able
> to specify the CPU feature names as you see under /proc/cpuinfo, and be
> able to turn the flags on or off:
>     
>         -M virt -cpu franken,rndr=on,ts=on,fhm=off
>
> (... instead of specifying long system register IDs that groups together
> a bunch of CPU features.  If I understand it correctly, the register
> "ID_AA64ISAR0_EL1" maps to a set of visible features listed here:
> https://docs.kernel.org/arch/arm64/cpu-feature-registers.html)
Not all the writable ID regs are visible through the above technique.
But indeed I think we converged on the idea to use higher level feature
names than ID reg field values.
However we need to study the feasibility and mappings between those high
level features and ID reg field values.
The cons is that we need to describe this mapping manually. Besides
being cumbersome this is also error prone.
>
>
> Next, I prefix the below by noting that I wrote it before seeing
> Cornelia's reply that the name "custom" is not set in stone:
> https://lists.nongnu.org/archive/html/qemu-arm/2024-10/msg00987.html.
>
> I wonder if the word "custom" is starting to get overloaded; on x86:
>
>   - Libvirt itself uses the term "custom" this way, to quote its
>     documentation[1] for the 'custom' XML attribute:
>
>       custom
>     
>       In this mode, the 'cpu' element describes the CPU that should be
>       presented to the guest. This is the default when no 'mode'
>       attribute is specified. This mode makes it so that a persistent
>       guest will see the same hardware no matter what host the guest is
>       booted on.
>
>   - Some management tools also follow libvirt and use the term "custom"
>     to refer to one of two things, (a) a specific named CPU model that
>     libvirt and QEMU recognize, e.g. "Cascadelake-Server"; or (b) a
>     named CPU model + extra CPU flags, e.g. this is how OpenStack
>     uses[2] "custom" to configure CPU models, and flags that can be
>     enabled or disabled via "+" or "-":
>
>       [libvirt]
>       cpu_mode = custom
>       cpu_model = IvyBridge-IBRS
>       cpu_model_extra_flags="ss,+vmx,-pcid [...]"
>
>     (Note the "cpu_mode" there: it is referring to the three possible
>     modes that libvirt and QEMU support today: 'host-passthrough',
>     'host-model', and named CPU models via "custom".)
>
>     The above config translates to this QEMU command-line:
>
>         -cpu IvyBridge-IBRS,ss=on,vmx=on,pcid=off [...]
>
> Now if QEMU introduces "custom", it is likely to create some confusion.
> But luckily, as referenced above, it is open to change. :)
Agreed! Thank you for the references!

Eric
>
>     * * *
>
> FWIW, I agree with Dan here[3] that it would cause less future pain if
> Arm's named CPU models also decides on a "baseline that matches some
> corresponding real world silicon".  I've experienced plenty of such
> debugging pain in x86-land from years of troubleshooting live migration
> bugs involving CPU model (in)compatibility.  (Often, with help from
> DanPB and Jiri Denemark).
>
> [1] https://docs.openstack.org/nova/latest/admin/cpu-models.html#cpu-modes
> [2] https://libvirt.org/formatdomain.html#cpu-model-and-topology
> [3] https://lists.nongnu.org/archive/html/qemu-arm/2024-10/msg00888.html
>     — [RFC 21/21] arm/cpu-features: Document custom vcpu model
>
> [...]
>