Re: [PATCH v16 11/11] docs/s390x/cpu topology: document s390x cpu topology

[Pierre Morel <pmorel@linux.ibm.com>]

On 2/27/23 14:58, Thomas Huth wrote:
> On 22/02/2023 15.21, Pierre Morel wrote:
>> Add some basic examples for the definition of cpu topology
>> in s390x.
>>
>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>> ---
>>   docs/system/s390x/cpu-topology.rst | 378 +++++++++++++++++++++++++++++
>>   docs/system/target-s390x.rst       |   1 +
>>   2 files changed, 379 insertions(+)
>>   create mode 100644 docs/system/s390x/cpu-topology.rst
>>
>> diff --git a/docs/system/s390x/cpu-topology.rst 
>> b/docs/system/s390x/cpu-topology.rst
>> new file mode 100644
>> index 0000000000..d470e28b97
>> --- /dev/null
>> +++ b/docs/system/s390x/cpu-topology.rst
>> @@ -0,0 +1,378 @@
>> +CPU topology on s390x
>> +=====================
>> +
>> +Since QEMU 8.0, CPU topology on s390x provides up to 3 levels of
>> +topology containers: drawers, books, sockets, defining a tree shaped
>> +hierarchy.
>> +
>> +The socket container contains one or more CPU entries consisting
>> +of a bitmap of three dentical CPU attributes:
>
> What do you mean by "dentical" here?

:D i.. dentical

I change it to identical

>
>> +- CPU type
>> +- polarization entitlement
>> +- dedication
>> +
>> +Note also that since 7.2 threads are no longer supported in the 
>> topology
>> +and the ``-smp`` command line argument accepts only ``threads=1``.
>> +
>> +Prerequisites
>> +-------------
>> +
>> +To use CPU topology a Linux QEMU/KVM machine providing the CPU 
>> topology facility
>> +(STFLE bit 11) is required.
>> +
>> +However, since this facility has been enabled by default in an early 
>> version
>> +of QEMU, we use a capability, ``KVM_CAP_S390_CPU_TOPOLOGY``, to 
>> notify KVM
>> +that QEMU supports CPU topology.
>
> This still sounds like a mix of documentation for the user and 
> documentation for developers. The normal user won't know what a KVM 
> capability is and does not need to know that QEMU does this 
> automatically under the hood.
>
> If you still want to mention it, I'd maybe rather phrase it like this:
>
> "To use the CPU topology, you need to run with KVM on a s390x host 
> that uses the Linux kernel v6.0 or newer (which provide the so-called 
> ``KVM_CAP_S390_CPU_TOPOLOGY`` capability that allows QEMU to signal 
> the CPU topology facility via the so-called STFLE bit 11 to the VM)."
>
> ... or something similar?


Yes, OK.


>
> [...]
>> +When a CPU is defined by the ``-device`` command argument, the
>> +tree topology attributes must be all defined or all not defined.
>> +
>> +.. code-block:: bash
>> +
>> +    -device 
>> gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1
>> +
>> +or
>> +
>> +.. code-block:: bash
>> +
>> +    -device gen16b-s390x-cpu,core-id=1,dedication=true
>> +
>> +If none of the tree attributes (drawer, book, sockets), are specified
>> +for the ``-device`` argument, as for all CPUs defined on the ``-smp``
>
> s/defined on/defined with/ ?


OK


>
>> +command argument the topology tree attributes will be set by simply
>> +adding the CPUs to the topology based on the core-id starting with
>> +core-0 at position 0 of socket-0, book-0, drawer-0.
>> +
>> +QEMU will not try to solve collisions and will report an error if the
>> +CPU topology, explicitely or implicitely defined on a ``-device``
>
> explicitly or implicitly


OK


>
>> +argument collides with the definition of a CPU implicitely defined
>
> implicitly

Yes


>
>> +on the ``-smp`` argument.
>> +
>> +When the topology modifier attributes are not defined for the
>> +``-device`` command argument they takes following default values:
>> +
>> +- dedication: ``false``
>> +- entitlement: ``medium``
>> +
>> +
>> +Hot plug
>> +++++++++
>> +
>> +New CPUs can be plugged using the device_add hmp command as in:
>> +
>> +.. code-block:: bash
>> +
>> +  (qemu) device_add gen16b-s390x-cpu,core-id=9
>> +
>> +The same placement of the CPU is derived from the core-id as 
>> described above.
>> +
>> +The topology can of course be fully defined:
>> +
>> +.. code-block:: bash
>> +
>> +    (qemu) device_add 
>> gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1
>> +
>> +
>> +Examples
>> +++++++++
>> +
>> +In the following machine we define 8 sockets with 4 cores each.
>> +
>> +.. code-block:: bash
>> +
>> +  $ qemu-system-s390x -m 2G \
>> +    -cpu gen16b,ctop=on \
>> +    -smp cpus=5,sockets=8,cores=4,maxcpus=32 \
>> +    -device host-s390x-cpu,core-id=14 \
>> +
>> +A new CPUs can be plugged using the device_add hmp command as before:
>> +
>> +.. code-block:: bash
>> +
>> +  (qemu) device_add gen16b-s390x-cpu,core-id=9
>> +
>> +The core-id defines the placement of the core in the topology by
>> +starting with core 0 in socket 0 up to maxcpus.
>> +
>> +In the example above:
>> +
>> +* There are 5 CPUs provided to the guest with the ``-smp`` command line
>> +  They will take the core-ids 0,1,2,3,4
>> +  As we have 4 cores in a socket, we have 4 CPUs provided
>> +  to the guest in socket 0, with core-ids 0,1,2,3.
>> +  The last cpu, with core-id 4, will be on socket 1.
>> +
>> +* the core with ID 14 provided by the ``-device`` command line will
>> +  be placed in socket 3, with core-id 14
>> +
>> +* the core with ID 9 provided by the ``device_add`` qmp command will
>> +  be placed in socket 2, with core-id 9
>> +
>> +
>> +Polarization, entitlement and dedication
>> +----------------------------------------
>> +
>> +Polarization
>> +++++++++++++
>> +
>> +The polarization is an indication given by the ``guest`` to the host
>> +that it is able to make use of CPU provisioning information.
>> +The guest indicates the polarization by using the PTF instruction.
>> +
>> +Polarization is define two models of CPU provisioning: horizontal
>> +and vertical.
>> +
>> +The horizontal polarization is the default model on boot and after
>> +subsystem reset in which the guest considers all vCPUs being having
>> +an equal provisioning of CPUs by the host.
>> +
>> +In the vertical polarization model the guest can make use of the
>> +vCPU entitlement information provided by the host to optimize
>> +kernel thread scheduling.
>> +
>> +A subsystem reset puts all vCPU of the configuration into the
>> +horizontal polarization.
>> +
>> +Entitlement
>> ++++++++++++
>> +
>> +The vertical polarization specifies that guest's vCPU can get
>
> "that *the* guest's vCPU" ?

Yes


>
>> +different real CPU provisions:
>> +
>> +- a vCPU with vertical high entitlement specifies that this
>> +  vCPU gets 100% of the real CPU provisioning.
>> +
>> +- a vCPU with vertical medium entitlement specifies that this
>> +  vCPU shares the real CPU with other vCPUs.
>> +
>> +- a vCPU with vertical low entitlement specifies that this
>> +  vCPU only gets real CPU provisioning when no other vCPUs needs it.
>> +
>> +In the case a vCPU with vertical high entitlement does not use
>> +the real CPU, the unused "slack" can be dispatched to other vCPU
>> +with medium or low entitlement.
>> +
>> +The admin specifies a vCPU as ``dedicated`` when the vCPU is fully 
>> dedicated
>> +to a single real CPU.
>> +
>> +The dedicated bit is an indication of affinity of a vCPU for a real CPU
>> +while the entitlement indicates the sharing or exclusivity of use.
>> +
>> +Defining the topology on command line
>> +-------------------------------------
>> +
>> +The topology can entirely be defined using -device cpu statements,
>> +with the exception of CPU 0 which must be defined with the -smp
>> +argument.
>> +
>> +For example, here we set the position of the cores 1,2,3 to
>> +drawer 1, book 1, socket 2 and cores 0,9 and 14 to drawer 0,
>> +book 0, socket 0 with all horizontal polarization and not dedicated.
>> +The core 4, will be set on its default position on socket 1
>> +(since we have 4 core per socket) and we define it with dedication and
>> +vertical high entitlement.
>> +
>> +.. code-block:: bash
>> +
>> +  $ qemu-system-s390x -m 2G \
>> +    -cpu gen16b,ctop=on \
>> +    -smp cpus=1,sockets=8,cores=4,maxcpus=32 \
>> +    \
>> +    -device 
>> gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1 \
>> +    -device 
>> gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=2 \
>> +    -device 
>> gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=3 \
>> +    \
>> +    -device 
>> gen16b-s390x-cpu,drawer-id=0,book-id=0,socket-id=0,core-id=9 \
>> +    -device 
>> gen16b-s390x-cpu,drawer-id=0,book-id=0,socket-id=0,core-id=14 \
>> +    \
>> +    -device gen16b-s390x-cpu,core-id=4,dedicated=on,polarization=3 \
>> +
>> +QAPI interface for topology
>> +---------------------------
>> +
>> +Let's start QEMU with the following command:
>> +
>> +.. code-block:: bash
>> +
>> + qemu-system-s390x \
>> +    -enable-kvm \
>> +    -cpu z14,ctop=on \
>> +    -smp 1,drawers=3,books=3,sockets=2,cores=2,maxcpus=36 \
>> +    \
>> +    -device z14-s390x-cpu,core-id=19,polarization=3 \
>> +    -device z14-s390x-cpu,core-id=11,polarization=1 \
>> +    -device z14-s390x-cpu,core-id=112,polarization=3 \
>> +   ...
>> +
>> +and see the result when using the QAPI interface.
>> +
>> +addons to query-cpus-fast
>> ++++++++++++++++++++++++++
>> +
>> +The command query-cpus-fast allows the admin to query the topology
>
> I'd just say "allows to query", without "the admin".


OK


>
>> +tree and modifiers for all configured vCPUs.
>> +
>> +.. code-block:: QMP
>> +
>> + { "execute": "query-cpus-fast" }
>> + {
>> +  "return": [
>> +    {
>> +      "dedicated": false,
>> +      "thread-id": 536993,
>> +      "props": {
>> +        "core-id": 0,
>> +        "socket-id": 0,
>> +        "drawer-id": 0,
>> +        "book-id": 0
>> +      },
>> +      "cpu-state": "operating",
>> +      "entitlement": "medium",
>> +      "qom-path": "/machine/unattached/device[0]",
>> +      "cpu-index": 0,
>> +      "target": "s390x"
>> +    },
>> +    {
>> +      "dedicated": false,
>> +      "thread-id": 537003,
>> +      "props": {
>> +        "core-id": 19,
>> +        "socket-id": 1,
>> +        "drawer-id": 0,
>> +        "book-id": 2
>> +      },
>> +      "cpu-state": "operating",
>> +      "entitlement": "high",
>> +      "qom-path": "/machine/peripheral-anon/device[0]",
>> +      "cpu-index": 19,
>> +      "target": "s390x"
>> +    },
>> +    {
>> +      "dedicated": false,
>> +      "thread-id": 537004,
>> +      "props": {
>> +        "core-id": 11,
>> +        "socket-id": 1,
>> +        "drawer-id": 0,
>> +        "book-id": 1
>> +      },
>> +      "cpu-state": "operating",
>> +      "entitlement": "low",
>> +      "qom-path": "/machine/peripheral-anon/device[1]",
>> +      "cpu-index": 11,
>> +      "target": "s390x"
>> +    },
>> +    {
>> +      "dedicated": true,
>> +      "thread-id": 537005,
>> +      "props": {
>> +        "core-id": 112,
>> +        "socket-id": 0,
>> +        "drawer-id": 3,
>> +        "book-id": 2
>> +      },
>> +      "cpu-state": "operating",
>> +      "entitlement": "high",
>> +      "qom-path": "/machine/peripheral-anon/device[2]",
>> +      "cpu-index": 112,
>> +      "target": "s390x"
>> +    }
>> +  ]
>> + }
>> +
>> +
>> +set-cpu-topology
>> +++++++++++++++++
>> +
>> +The command set-cpu-topology allows the admin to modify the topology
>
> "allows to modify"

OK


>
>> +tree or the topology modifiers of a vCPU in the configuration.
>> +
>> +.. code-block:: QMP
>> +
>> + -> { "execute": "set-cpu-topology",
>> +      "arguments": {
>> +         "core-id": 11,
>> +         "socket-id": 0,
>> +         "book-id": 0,
>> +         "drawer-id": 0,
>> +         "entitlement": low,
>> +         "dedicated": false
>> +      }
>> +    }
>> + <- {"return": {}}
>> +
>> +The core-id parameter is the only non optional parameter and every
>> +unspecified parameter keeps its previous value.
>> +
>> +event CPU_POLARIZATION_CHANGE
>> ++++++++++++++++++++++++++++++
>> +
>> +When a guest is requests a modification of the polarization,
>> +QEMU sends a CPU_POLARIZATION_CHANGE event.
>> +
>> +When requesting the change, the guest only specifies horizontal or
>> +vertical polarization.
>> +It is the job of the admin to set the dedication and fine grained 
>> vertical entitlement
>> +in response to this event.
>> +
>> +Note that a vertical polarized dedicated vCPU can only have a high
>> +entitlement, this gives 6 possibilities for vCPU polarization:
>> +
>> +- Horizontal
>> +- Horizontal dedicated
>> +- Vertical low
>> +- Vertical medium
>> +- Vertical high
>> +- Vertical high dedicated
>> +
>> +Example of the event received when the guest issues the CPU instruction
>> +Perform Topology Function PTF(0) to request an horizontal polarization:
>> +
>> +.. code-block:: QMP
>> +
>> + <- { "event": "CPU_POLARIZATION_CHANGE",
>> +      "data": { "polarization": 0 },
>> +      "timestamp": { "seconds": 1401385907, "microseconds": 422329 } }
>
>  Thomas
>
>
Thanks!

Regards,

Pierre