Re: [Qemu-devel] [RFC 00/28] s390x CPU models: exposing features

[Eduardo Habkost <ehabkost@redhat.com>]

(CCing libvirt people)

On Tue, Jun 21, 2016 at 03:02:05PM +0200, David Hildenbrand wrote:
> This is our second attempt to implement CPU models for s390x. We realized
> that we also want to have features exposed via the CPU model. While doing
> that we realized that we want to have a better interface for libvirt.

Before getting into the details, I would like to clarify how the
following could be accomplished using the new commands:

Example:

1) User configures libvirt with:
   <cpu match='exact'>
       <model fallback='forbid'>Westmere</model>
       <feature policy='require' name='aes'/>
   </cpu>
2) libvirt will translate that to:
   "-cpu Westmere,+aes" or "-cpu Westmere,aes=on"
3) libvirt wants to know if "-cpu Westmere,aes=on" is usable in
   the current host, before trying to start the VM.

How exactly would this be done using the new commands?

> 
> Unfortunately, CPU models on s390x are special and we have to take care of:
> - A CPU like z13 looks differently in various environments (under different
>   LPAR versions, under different z/VM versions, under different KVM
>   versions, export regulation) - we have _a lot_ of feature variability.
> - We still have certain features that are not published but might be
>   implemented/introduced in the future. As they are a theoretical part
>   of a CPU already, we have to find a way to model these future changes.
> - We still have certain features that are already published, but not
>   implemented. Implementation might be added in the future in KVM.
> - We heavily rely on KVM to tell us which features it can actually
>   virtualize - user space queries like "STFL(e)" give no guarantees.
> - Certain "subfeatures" were introduced in one run. In practice, they are
>   always around, but in theory, subfeatures could be dropped in the future.
> - Just because two CPU models have the same features doesn't mean they
>   are equal - some internal numbers might be different. E.g. we won't allow
>   running a z13 under a zBC12 just by turning off features.
> - We cannot blindly enable all possible features for a CPU generation,
>   the IBC "Instruction Blocking Control" in KVM will try to block
>   instructions introduced with certain features. So a CPU generation always
>   has some maximum feature set that is guaranteed to work.
> 
> It all boils down to a specific released CPU to have.
> a) A fixed feature set that we expect it to be have on every hypervisor.
> b) A variable part that depends on the hypervisor and that could be
>    extended in the future (adding not yet implemented features) that we
>    always want to enable later on.
> c) A variable part that we want to enable only if requested - nested
>    virtualization ("vsie") and assists are one example.
> 
> But, the fixed feature set is not really what we want to use as a default.
> It is just like a really minimum, stable base.
> 
> So we have
> a) A "stable" CPU model for each released CPU that will never change and
>    maps to the minimum feature set we expect to be around on all
>    hypervisors. e.g. "z13-base" or "z10EC.2-base". These are migration
>    safe.
> b) A "default" CPU model for each released CPU, that can change between
>    QEMU versions and that will always include the features we expect to
>    be around in our currently supported environments and will contain only
>    features we expect to be stable. E.g. nested virtualization will not be
>    contained in these models. These models are not migration safe, e.g
>    "z13" or "z10EC.2". The feature set can differ between QEMU versions.
> c) An internal "maximum" CPU model for each generation that tells us which
>    features were supported as a maximum back when the hardware was
>    released. This will not be exposed
> 
> To not have to replicate all CPU model changes ("new default fetaures") in
> libvirt, to not duplicate the logic about compatibility and the like,
> our approach tries to keep all the QEMU logic in libvirt and provide
> standardized interfaces for libvirt to e.g. baseline, compare. This
> allows libvirt to not have to care about any model names or feature names,
> it can just pass the data from interface to interface and report it to
> the user.
> 
> Also, libvirt might want to know what the "host" model looks like and
> convert a CPU model to a migration safe variant. For this reason, a QMP
> command is added that can create a migration safe variant of a variable
> CPU model, indicating only the delta changes done to a stable model.
> 
> So we have:
> a) "query-cpu-model-expansion" - tell us what the "host" or a migration
>    unsafe model looks like. Either falling back to a stable model or
>    completely exposing all properties. We are interested in stable models.
> b) "query-cpu-model-comparison" - tell us how two CPU models compare,
>     indicating which properties were responsible for the decision.
> c) "query-cpu-model-baseline" - create a new model out of two models,
>     taking a requested level of stability into account.
> 
> As we are aware that e.g. x86 has their own idea of a CPU model and their
> existing implementation in place, but are also looking into to ways to e.g.
> expand the "host" CPU model to a detailed representation, we designed the
> "expansion" interface to also allow that.
> 
> Comments are very welcome, but please always keep the restrictions and
> specialties in mind when suggesting some major design changes.
> 
> The header update will be replaced by a kvm-next header update as soon as
> the VSIE patches are upstream. The major KVM interface changes are already
> part of kvm-next.
> 
> The current state is available on git://github.com/cohuck/qemu on branch
> "cpumodel-s390x".
> 
> David Hildenbrand (26):
>   s390x/cpumodel: "host" and "qemu" as CPU subclasses
>   s390x/cpumodel: expose CPU class properties
>   s390x/cpumodel: generate CPU feature group lists
>   s390x/cpumodel: introduce CPU feature group definitions
>   s390x/cpumodel: register defined CPU models as subclasses
>   s390x/cpumodel: store the CPU model in the CPU instance
>   s390x/cpumodel: expose features and feature groups as properties
>   s390x/cpumodel: let the CPU model handle feature checks
>   s390x/cpumodel: check and apply the CPU model
>   s390x/sclp: factor out preparation of cpu entries
>   s390x/sclp: introduce sclp feature blocks
>   s390x/sclp: indicate sclp features
>   s390x/sclp: propagate the ibc val(lowest and unblocked ibc)
>   s390x/sclp: propagate the mha via sclp
>   s390x/sclp: propagate hmfai
>   update linux headers (CPU model)
>   s390x/kvm: allow runtime-instrumentation for "none" machine
>   s390x/kvm: implement CPU model support
>   s390x/kvm: disable host model for existing compat machines
>   s390x/kvm: let the CPU model control CMM(A)
>   qmp: add QMP interface "query-cpu-model-expansion"
>   qmp: add QMP interface "query-cpu-model-comparison"
>   qmp: add QMP interface "query-cpu-model-baseline"
>   s390x/cpumodel: implement QMP interface "query-cpu-model-expansion"
>   s390x/cpumodel: implement QMP interface "query-cpu-model-comparison"
>   s390x/cpumodel: implement QMP interface "query-cpu-model-baseline"
> 
> Michael Mueller (2):
>   s390x/cpumodel: introduce CPU features
>   s390x/cpumodel: generate CPU feature lists for CPU models
> 
>  Makefile.target                         |    2 +-
>  hw/s390x/s390-virtio-ccw.c              |    5 +
>  hw/s390x/s390-virtio.c                  |    6 +-
>  hw/s390x/sclp.c                         |   35 +-
>  include/hw/s390x/sclp.h                 |   17 +-
>  include/sysemu/arch_init.h              |   10 +
>  linux-headers/asm-s390/kvm.h            |   40 ++
>  qapi-schema.json                        |  184 ++++++
>  qmp-commands.hx                         |   18 +
>  qmp.c                                   |   22 +
>  rules.mak                               |    1 +
>  stubs/Makefile.objs                     |    3 +
>  stubs/arch-query-cpu-model-baseline.c   |   13 +
>  stubs/arch-query-cpu-model-comparison.c |   12 +
>  stubs/arch-query-cpu-model-expansion.c  |   12 +
>  target-s390x/Makefile.objs              |   22 +-
>  target-s390x/cpu-qom.h                  |    5 +
>  target-s390x/cpu.c                      |   35 +-
>  target-s390x/cpu.h                      |    5 +
>  target-s390x/cpu_features.c             |  376 +++++++++++
>  target-s390x/cpu_features.h             |  302 +++++++++
>  target-s390x/cpu_models.c               | 1055 +++++++++++++++++++++++++++++++
>  target-s390x/cpu_models.h               |  113 ++++
>  target-s390x/gen-features.c             |  587 +++++++++++++++++
>  target-s390x/helper.c                   |   29 +-
>  target-s390x/kvm.c                      |  346 +++++++++-
>  target-s390x/machine.c                  |   14 +-
>  27 files changed, 3203 insertions(+), 66 deletions(-)
>  create mode 100644 stubs/arch-query-cpu-model-baseline.c
>  create mode 100644 stubs/arch-query-cpu-model-comparison.c
>  create mode 100644 stubs/arch-query-cpu-model-expansion.c
>  create mode 100644 target-s390x/cpu_features.c
>  create mode 100644 target-s390x/cpu_features.h
>  create mode 100644 target-s390x/cpu_models.c
>  create mode 100644 target-s390x/cpu_models.h
>  create mode 100644 target-s390x/gen-features.c
> 
> -- 
> 2.6.6
> 

-- 
Eduardo