Re: [Qemu-devel] [libvirt] Question about the host-model CPU mode

Re: [Qemu-devel] [libvirt] Question about the host-model CPU mode

[Marc Hartmayer]

On Thu, Oct 12, 2017 at 02:07 PM +0200, Jiri Denemark <jdenemar@redhat.com> wrote:
> On Mon, Oct 09, 2017 at 10:16:48 +0200, Marc Hartmayer wrote:
>> On Thu, Oct 05, 2017 at 02:11 PM +0200, Jiri Denemark <jdenemar@redhat.com> wrote:
>> > But it's going to be a bit complicated because we ask QEMU what the
>> > host CPU is and the interface we used would need to be enhanced to
>> > give us different results for all supported machine types so that we
>> > can select the right one when a domain is started.
>>
>> So how do we deal with this?
>
> I think we need to start discussing this on qemu-devel list. If I
> remember correctly, David Hildenbrand is the original author of the
> query-cpu-model-expansion QMP command.
>
> Please, Cc me so that the thread is more visible to me.
>
> Thanks,
>
> Jirka
>

Hi all,

we recently encountered the problem that the 'host-model' [1] has to be
related to the machine type of a domain. We have following problem:

   Let's assume we've a z13 system with a QEMU 2.9 and we define a
   domain using the default s390-virtio-ccw machine together with the
   host-model CPU mode [1]. The definition will have the machine
   expanded to s390-virtio-ccw-2.9 but retain the host-model CPU mode
   in the domain definition. In a next step we upgrade to QEMU 2.10
   (first version to recognize z14). Everything is still fine, even
   though the machine runs in 2.9 compatibility mode. Finally we
   upgrade to a z14. As a consequence it is not possible to start the
   domain anymore as the machine type doesn't support our CPU host
   model (which is expanded at start time of the domain).

For determining the actual host-model the QMP command
'query-cpu-model-expansion' is used. This is done once per QEMU binary
and the result of it is cached by libvirt. The problem with that is
that libvirt queries with the newest machine type of the QEMU binary
for the host CPU model. But now we realized that we have to consider
the machine type of each domain for the determination of the host CPU
model of a domain.

We could now either probe the host CPU model for each QEMU binary +
machine type combination and for this we've to start a new QEMU
process each time. Or we can add a QMP command/parameter that allows
us to determine the host CPU model(s) with respect to the passed
machine type and/or all supported machine types.

The QMP command query-cpu-model-expansion is currently described in
qapi-schema.json as follows:

<quotation>
##
# @query-cpu-model-expansion:
#
# Expands a given CPU model (or a combination of CPU model + additional options)
# to different granularities, allowing tooling to get an understanding what a
# specific CPU model looks like in QEMU under a certain configuration.
#
# This interface can be used to query the "host" CPU model.
#
# The data returned by this command may be affected by:
#
# * QEMU version: CPU models may look different depending on the QEMU version.
# (Except for CPU models reported as "static" in query-cpu-definitions.)
# * machine-type: CPU model may look different depending on the machine-type.
# (Except for CPU models reported as "static" in query-cpu-definitions.)
# * machine options (including accelerator): in some architectures, CPU models
# may look different depending on machine and accelerator options. (Except for
# CPU models reported as "static" in query-cpu-definitions.)
# * "-cpu" arguments and global properties: arguments to the -cpu option and
# global properties may affect expansion of CPU models. Using
# query-cpu-model-expansion while using these is not advised.
#
# Some architectures may not support all expansion types. s390x supports
# "full" and "static".
#
# Returns: a CpuModelExpansionInfo. Returns an error if expanding CPU models is
# not supported, if the model cannot be expanded, if the model contains
# an unknown CPU definition name, unknown properties or properties
# with a wrong type. Also returns an error if an expansion type is
# not supported.
#
# Since: 2.8.0
##
{ 'command': 'query-cpu-model-expansion',
'data': { 'type': 'CpuModelExpansionType',
'model': 'CpuModelInfo' },
'returns': 'CpuModelExpansionInfo' }

4:46:25 PM
◄
qapi-schema.json
</quotation>

What do you think?

Thank you.

 Marc



[1] https://libvirt.org/formatdomain.html#elementsCPU

--
Beste Grüße / Kind regards
   Marc Hartmayer

IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294

Re: [Qemu-devel] [libvirt] Question about the host-model CPU mode

[David Hildenbrand]

On 20.10.2017 13:09, Marc Hartmayer wrote:
> On Thu, Oct 12, 2017 at 02:07 PM +0200, Jiri Denemark <jdenemar@redhat.com> wrote:
>> On Mon, Oct 09, 2017 at 10:16:48 +0200, Marc Hartmayer wrote:
>>> On Thu, Oct 05, 2017 at 02:11 PM +0200, Jiri Denemark <jdenemar@redhat.com> wrote:
>>>> But it's going to be a bit complicated because we ask QEMU what the
>>>> host CPU is and the interface we used would need to be enhanced to
>>>> give us different results for all supported machine types so that we
>>>> can select the right one when a domain is started.
>>>
>>> So how do we deal with this?
>>
>> I think we need to start discussing this on qemu-devel list. If I
>> remember correctly, David Hildenbrand is the original author of the
>> query-cpu-model-expansion QMP command.
>>
>> Please, Cc me so that the thread is more visible to me.
>>
>> Thanks,
>>
>> Jirka
>>
> 
> Hi all,
> 
> we recently encountered the problem that the 'host-model' [1] has to be
> related to the machine type of a domain. We have following problem:
> 
>    Let's assume we've a z13 system with a QEMU 2.9 and we define a
>    domain using the default s390-virtio-ccw machine together with the
>    host-model CPU mode [1]. The definition will have the machine
>    expanded to s390-virtio-ccw-2.9 but retain the host-model CPU mode
>    in the domain definition. In a next step we upgrade to QEMU 2.10
>    (first version to recognize z14). Everything is still fine, even
>    though the machine runs in 2.9 compatibility mode. Finally we
>    upgrade to a z14. As a consequence it is not possible to start the
>    domain anymore as the machine type doesn't support our CPU host
>    model (which is expanded at start time of the domain).

Yes, the host model may vary depending on QEMU version and to some
degree also on compatibility machines (although I always try to push
people to not introduce any new such stuff).

Quoting for the libvirt documentation: https://libvirt.org/formatdomain.html

"host-model
    The host-model mode is essentially a shortcut to copying host CPU
definition from capabilities XML into domain XML. Since the CPU
definition is copied just before starting a domain, exactly the same XML
can be used on different hosts while still providing the best guest CPU
each host supports."

You're telling me that that copying does not happen, which seems to be
the problematic part about this in my opinion.

So I am not sure if probing anything else (as you noted below) is the
correct thing to do. Copy it and you have a migration_safe and even
static version of the _current_ host CPU.

> 
> For determining the actual host-model the QMP command
> 'query-cpu-model-expansion' is used. This is done once per QEMU binary
> and the result of it is cached by libvirt. The problem with that is
> that libvirt queries with the newest machine type of the QEMU binary
> for the host CPU model. But now we realized that we have to consider
> the machine type of each domain for the determination of the host CPU
> model of a domain.
> 
> We could now either probe the host CPU model for each QEMU binary +
> machine type combination and for this we've to start a new QEMU
> process each time. Or we can add a QMP command/parameter that allows
> us to determine the host CPU model(s) with respect to the passed
> machine type and/or all supported machine types.
> 
> The QMP command query-cpu-model-expansion is currently described in
> qapi-schema.json as follows:
> 
> <quotation>
> ##
> # @query-cpu-model-expansion:
> #
> # Expands a given CPU model (or a combination of CPU model + additional options)
> # to different granularities, allowing tooling to get an understanding what a
> # specific CPU model looks like in QEMU under a certain configuration.
> #
> # This interface can be used to query the "host" CPU model.
> #
> # The data returned by this command may be affected by:
> #
> # * QEMU version: CPU models may look different depending on the QEMU version.
> # (Except for CPU models reported as "static" in query-cpu-definitions.)
> # * machine-type: CPU model may look different depending on the machine-type.
> # (Except for CPU models reported as "static" in query-cpu-definitions.)
> # * machine options (including accelerator): in some architectures, CPU models
> # may look different depending on machine and accelerator options. (Except for
> # CPU models reported as "static" in query-cpu-definitions.)
> # * "-cpu" arguments and global properties: arguments to the -cpu option and
> # global properties may affect expansion of CPU models. Using
> # query-cpu-model-expansion while using these is not advised.
> #
> # Some architectures may not support all expansion types. s390x supports
> # "full" and "static".
> #
> # Returns: a CpuModelExpansionInfo. Returns an error if expanding CPU models is
> # not supported, if the model cannot be expanded, if the model contains
> # an unknown CPU definition name, unknown properties or properties
> # with a wrong type. Also returns an error if an expansion type is
> # not supported.
> #
> # Since: 2.8.0
> ##
> { 'command': 'query-cpu-model-expansion',
> 'data': { 'type': 'CpuModelExpansionType',
> 'model': 'CpuModelInfo' },
> 'returns': 'CpuModelExpansionInfo' }
> 
> 4:46:25 PM
> ◄
> qapi-schema.json
> </quotation>
> 
> What do you think?
> 
> Thank you.
> 
>  Marc
> 
> 
> 
> [1] https://libvirt.org/formatdomain.html#elementsCPU
> 
> --
> Beste Grüße / Kind regards
>    Marc Hartmayer
> 
> IBM Deutschland Research & Development GmbH
> Vorsitzende des Aufsichtsrats: Martina Koederitz
> Geschäftsführung: Dirk Wittkopp
> Sitz der Gesellschaft: Böblingen
> Registergericht: Amtsgericht Stuttgart, HRB 243294
> 


-- 

Thanks,

David

Re: [Qemu-devel] [libvirt] Question about the host-model CPU mode

[Jiri Denemark]

On Fri, Oct 20, 2017 at 13:37:42 +0200, David Hildenbrand wrote:
> On 20.10.2017 13:09, Marc Hartmayer wrote:
> > we recently encountered the problem that the 'host-model' [1] has to be
> > related to the machine type of a domain. We have following problem:
> > 
> >    Let's assume we've a z13 system with a QEMU 2.9 and we define a
> >    domain using the default s390-virtio-ccw machine together with the
> >    host-model CPU mode [1]. The definition will have the machine
> >    expanded to s390-virtio-ccw-2.9 but retain the host-model CPU mode
> >    in the domain definition. In a next step we upgrade to QEMU 2.10
> >    (first version to recognize z14). Everything is still fine, even
> >    though the machine runs in 2.9 compatibility mode. Finally we
> >    upgrade to a z14. As a consequence it is not possible to start the
> >    domain anymore as the machine type doesn't support our CPU host
> >    model (which is expanded at start time of the domain).
> 
> Yes, the host model may vary depending on QEMU version and to some
> degree also on compatibility machines (although I always try to push
> people to not introduce any new such stuff).
> 
> Quoting for the libvirt documentation: https://libvirt.org/formatdomain.html
> 
> "host-model
>     The host-model mode is essentially a shortcut to copying host CPU
> definition from capabilities XML into domain XML. Since the CPU
> definition is copied just before starting a domain, exactly the same XML
> can be used on different hosts while still providing the best guest CPU
> each host supports."
> 
> You're telling me that that copying does not happen, which seems to be
> the problematic part about this in my opinion.
> 
> So I am not sure if probing anything else (as you noted below) is the
> correct thing to do. Copy it and you have a migration_safe and even
> static version of the _current_ host CPU.

The thing is libvirt calls query-cpu-model-expansion to check what the
host CPU is. This 'host-model' CPU is replaced with the probed CPU model
when a domain starts. The problem described by Marc is that the probed
CPU model cannot be used universally with all machine types. So starting
a domain on such host with machine type s390-virtio-ccw-2.10 works, but
a domain with machine type s390-virtio-ccw-2.9 fails to start with the
same probed CPU model.

Jirka

Re: [Qemu-devel] [libvirt] Question about the host-model CPU mode

[David Hildenbrand]

On 20.10.2017 14:50, Jiri Denemark wrote:
> On Fri, Oct 20, 2017 at 13:37:42 +0200, David Hildenbrand wrote:
>> On 20.10.2017 13:09, Marc Hartmayer wrote:
>>> we recently encountered the problem that the 'host-model' [1] has to be
>>> related to the machine type of a domain. We have following problem:
>>>
>>>    Let's assume we've a z13 system with a QEMU 2.9 and we define a
>>>    domain using the default s390-virtio-ccw machine together with the
>>>    host-model CPU mode [1]. The definition will have the machine
>>>    expanded to s390-virtio-ccw-2.9 but retain the host-model CPU mode
>>>    in the domain definition. In a next step we upgrade to QEMU 2.10
>>>    (first version to recognize z14). Everything is still fine, even
>>>    though the machine runs in 2.9 compatibility mode. Finally we
>>>    upgrade to a z14. As a consequence it is not possible to start the
>>>    domain anymore as the machine type doesn't support our CPU host
>>>    model (which is expanded at start time of the domain).
>>
>> Yes, the host model may vary depending on QEMU version and to some
>> degree also on compatibility machines (although I always try to push
>> people to not introduce any new such stuff).
>>
>> Quoting for the libvirt documentation: https://libvirt.org/formatdomain.html
>>
>> "host-model
>>     The host-model mode is essentially a shortcut to copying host CPU
>> definition from capabilities XML into domain XML. Since the CPU
>> definition is copied just before starting a domain, exactly the same XML
>> can be used on different hosts while still providing the best guest CPU
>> each host supports."
>>
>> You're telling me that that copying does not happen, which seems to be
>> the problematic part about this in my opinion.
>>
>> So I am not sure if probing anything else (as you noted below) is the
>> correct thing to do. Copy it and you have a migration_safe and even
>> static version of the _current_ host CPU.
> 
> The thing is libvirt calls query-cpu-model-expansion to check what the
> host CPU is. This 'host-model' CPU is replaced with the probed CPU model
> when a domain starts. The problem described by Marc is that the probed
> CPU model cannot be used universally with all machine types. So starting
> a domain on such host with machine type s390-virtio-ccw-2.10 works, but
> a domain with machine type s390-virtio-ccw-2.9 fails to start with the
> same probed CPU model.
> 

My assumption would be that the CPU model is copied into the XML when
the domain fist starts. This is what the documentation describes.

So when upgrading QEMU, the CPU model in the XML is still the same
(z13), even though something different is now reported in the host info
after upgrading QEMU (z14).

In this case it would continue to work.

The problem is that the CPU model is not copied into the XML doesn't
remain there, right? It is suddenly replaced with a z14 host model.

> Jirka
> 


-- 

Thanks,

David

Re: [Qemu-devel] [libvirt] Question about the host-model CPU mode

[Jiri Denemark]

On Fri, Oct 20, 2017 at 15:04:57 +0200, David Hildenbrand wrote:
> On 20.10.2017 14:50, Jiri Denemark wrote:
> > The thing is libvirt calls query-cpu-model-expansion to check what the
> > host CPU is. This 'host-model' CPU is replaced with the probed CPU model
> > when a domain starts. The problem described by Marc is that the probed
> > CPU model cannot be used universally with all machine types. So starting
> > a domain on such host with machine type s390-virtio-ccw-2.10 works, but
> > a domain with machine type s390-virtio-ccw-2.9 fails to start with the
> > same probed CPU model.
> > 
> 
> My assumption would be that the CPU model is copied into the XML when
> the domain fist starts. This is what the documentation describes.

The CPU model is copied into the XML each time the domain starts.

> So when upgrading QEMU, the CPU model in the XML is still the same
> (z13), even though something different is now reported in the host info
> after upgrading QEMU (z14).
> 
> In this case it would continue to work.
> 
> The problem is that the CPU model is not copied into the XML doesn't
> remain there, right? It is suddenly replaced with a z14 host model.

Even preserving the actual CPU model in the XML from the first start
wouldn't solve the issue. You can always create a new domain with
s390-virtio-ccw-2.9 machine type even on the upgraded host.

Jirka

Re: [Qemu-devel] [libvirt] Question about the host-model CPU mode

[Jiri Denemark]

On Fri, Oct 20, 2017 at 13:09:26 +0200, Marc Hartmayer wrote:
> we recently encountered the problem that the 'host-model' [1] has to be
> related to the machine type of a domain. We have following problem:
> 
>    Let's assume we've a z13 system with a QEMU 2.9 and we define a
>    domain using the default s390-virtio-ccw machine together with the
>    host-model CPU mode [1]. The definition will have the machine
>    expanded to s390-virtio-ccw-2.9 but retain the host-model CPU mode
>    in the domain definition. In a next step we upgrade to QEMU 2.10
>    (first version to recognize z14). Everything is still fine, even
>    though the machine runs in 2.9 compatibility mode. Finally we
>    upgrade to a z14. As a consequence it is not possible to start the
>    domain anymore as the machine type doesn't support our CPU host
>    model (which is expanded at start time of the domain).
> 
> For determining the actual host-model the QMP command
> 'query-cpu-model-expansion' is used. This is done once per QEMU binary
> and the result of it is cached by libvirt. The problem with that is
> that libvirt queries with the newest machine type of the QEMU binary
> for the host CPU model.

No, libvirt probes QEMU with -machine none.

> We could now either probe the host CPU model for each QEMU binary +
> machine type combination and for this we've to start a new QEMU
> process each time.

This is not really a viable solution.

Jirka

Re: [Qemu-devel] [libvirt] Question about the host-model CPU mode

[David Hildenbrand]

> Hi all,
> 
> we recently encountered the problem that the 'host-model' [1] has to be
> related to the machine type of a domain. We have following problem:
> 
>    Let's assume we've a z13 system with a QEMU 2.9 and we define a
>    domain using the default s390-virtio-ccw machine together with the
>    host-model CPU mode [1]. The definition will have the machine
>    expanded to s390-virtio-ccw-2.9 but retain the host-model CPU mode
>    in the domain definition. In a next step we upgrade to QEMU 2.10
>    (first version to recognize z14). Everything is still fine, even
>    though the machine runs in 2.9 compatibility mode. Finally we
>    upgrade to a z14. As a consequence it is not possible to start the
>    domain anymore as the machine type doesn't support our CPU host
>    model (which is expanded at start time of the domain).

Actually, what is the cause of that problem? I assume it is the gs
feature (gs_allowed)?

We should really avoid such things (..._allowed) for CPU model features
in the future and clue all new such stuff to cpumodel_allowed.

-- 

Thanks,

David

Re: [Qemu-devel] [libvirt] Question about the host-model CPU mode

[Christian Borntraeger]

On 10/20/2017 03:16 PM, David Hildenbrand wrote:
> 
>> Hi all,
>>
>> we recently encountered the problem that the 'host-model' [1] has to be
>> related to the machine type of a domain. We have following problem:
>>
>>    Let's assume we've a z13 system with a QEMU 2.9 and we define a
>>    domain using the default s390-virtio-ccw machine together with the
>>    host-model CPU mode [1]. The definition will have the machine
>>    expanded to s390-virtio-ccw-2.9 but retain the host-model CPU mode
>>    in the domain definition. In a next step we upgrade to QEMU 2.10
>>    (first version to recognize z14). Everything is still fine, even
>>    though the machine runs in 2.9 compatibility mode. Finally we
>>    upgrade to a z14. As a consequence it is not possible to start the
>>    domain anymore as the machine type doesn't support our CPU host
>>    model (which is expanded at start time of the domain).
> 
> Actually, what is the cause of that problem? I assume it is the gs
> feature (gs_allowed)?
> 
> We should really avoid such things (..._allowed) for CPU model features
> in the future and clue all new such stuff to cpumodel_allowed.

Yes, starting a guest with
   <os>
    <type arch='s390x' machine='s390-ccw-virtio-2.9'>hvm</type>
  </os>
  <cpu mode='host-model'/>

results in

qemu-system-s390x: Some features requested in the CPU model are not available in the configuration: gs 

Tying it to cpumodel_allowed would not help, migration-wise though.
libvirt would still transform

   <os>
    <type arch='s390x' machine='s390-ccw-virtio-2.9'>hvm</type>
  </os>
  <cpu mode='host-model'/>


into 
-cpu z14-base,aen=on,cmmnt=on,aefsi=on,mepoch=on,msa8=on,msa7=on,msa6=on,msa5=on,msa4=on, \
msa3=on,msa2=on,msa1=on,sthyi=on,edat=on,ri=on,edat2=on,vx=on,ipter=on,vxeh=on,vxpd=on, \
esop=on,iep=on,cte=on,ais=on,gs=on,zpci=on,sea_esop2=on,te=on,cmm=on
                             ^^^^^
because cpu model is certainly there. Now the guest would start but migration would
later fail because what we create now would never have been possible with 2.9.

If libvirt could get information from QEMU depending on the machine version, this would
make it work. But of course this has other issues.

Christian

Re: [Qemu-devel] [libvirt] Question about the host-model CPU mode

[David Hildenbrand]

On 20.10.2017 15:36, Christian Borntraeger wrote:
> 
> 
> On 10/20/2017 03:16 PM, David Hildenbrand wrote:
>>
>>> Hi all,
>>>
>>> we recently encountered the problem that the 'host-model' [1] has to be
>>> related to the machine type of a domain. We have following problem:
>>>
>>>    Let's assume we've a z13 system with a QEMU 2.9 and we define a
>>>    domain using the default s390-virtio-ccw machine together with the
>>>    host-model CPU mode [1]. The definition will have the machine
>>>    expanded to s390-virtio-ccw-2.9 but retain the host-model CPU mode
>>>    in the domain definition. In a next step we upgrade to QEMU 2.10
>>>    (first version to recognize z14). Everything is still fine, even
>>>    though the machine runs in 2.9 compatibility mode. Finally we
>>>    upgrade to a z14. As a consequence it is not possible to start the
>>>    domain anymore as the machine type doesn't support our CPU host
>>>    model (which is expanded at start time of the domain).
>>
>> Actually, what is the cause of that problem? I assume it is the gs
>> feature (gs_allowed)?
>>
>> We should really avoid such things (..._allowed) for CPU model features
>> in the future and clue all new such stuff to cpumodel_allowed.
> 
> Yes, starting a guest with
>    <os>
>     <type arch='s390x' machine='s390-ccw-virtio-2.9'>hvm</type>
>   </os>
>   <cpu mode='host-model'/>
> 
> results in
> 
> qemu-system-s390x: Some features requested in the CPU model are not available in the configuration: gs 
> 
> Tying it to cpumodel_allowed would not help, migration-wise though.
> libvirt would still transform
> 
>    <os>
>     <type arch='s390x' machine='s390-ccw-virtio-2.9'>hvm</type>
>   </os>
>   <cpu mode='host-model'/>

My point was, that the host model would have to be copied and _remain_
there when s390-ccw-virtio was expanded to s390-ccw-virtio-2.9.

So really replacing <cpu mode='host-model'/> by the model z13-base....
This would at least fix this issue. Just like s390-ccw-virtio get's
replaced and remains thats way.

But this might for sure have other problems.

> 
> 
> into 
> -cpu z14-base,aen=on,cmmnt=on,aefsi=on,mepoch=on,msa8=on,msa7=on,msa6=on,msa5=on,msa4=on, \
> msa3=on,msa2=on,msa1=on,sthyi=on,edat=on,ri=on,edat2=on,vx=on,ipter=on,vxeh=on,vxpd=on, \
> esop=on,iep=on,cte=on,ais=on,gs=on,zpci=on,sea_esop2=on,te=on,cmm=on
>                              ^^^^^
> because cpu model is certainly there. Now the guest would start but migration would
> later fail because what we create now would never have been possible with 2.9.

Migration is a totally different story, as tooling has to make sure to
find a CPU model that is compatible over all host models. So
cpumodel_allowed would indeed work. (at least in my theory ;) )

> 
> If libvirt could get information from QEMU depending on the machine version, this would
> make it work. But of course this has other issues.

I am not sure if that is the right thing to do.

The documentation states clearly that the host model is copied. If that
is not runnable, fix the setup.

> 
> Christian
> 


-- 

Thanks,

David

Re: [Qemu-devel] [libvirt] Question about the host-model CPU mode

[Christian Borntraeger]

On 10/20/2017 03:43 PM, David Hildenbrand wrote:
> On 20.10.2017 15:36, Christian Borntraeger wrote:
>>
>>
>> On 10/20/2017 03:16 PM, David Hildenbrand wrote:
>>>
>>>> Hi all,
>>>>
>>>> we recently encountered the problem that the 'host-model' [1] has to be
>>>> related to the machine type of a domain. We have following problem:
>>>>
>>>>    Let's assume we've a z13 system with a QEMU 2.9 and we define a
>>>>    domain using the default s390-virtio-ccw machine together with the
>>>>    host-model CPU mode [1]. The definition will have the machine
>>>>    expanded to s390-virtio-ccw-2.9 but retain the host-model CPU mode
>>>>    in the domain definition. In a next step we upgrade to QEMU 2.10
>>>>    (first version to recognize z14). Everything is still fine, even
>>>>    though the machine runs in 2.9 compatibility mode. Finally we
>>>>    upgrade to a z14. As a consequence it is not possible to start the
>>>>    domain anymore as the machine type doesn't support our CPU host
>>>>    model (which is expanded at start time of the domain).
>>>
>>> Actually, what is the cause of that problem? I assume it is the gs
>>> feature (gs_allowed)?
>>>
>>> We should really avoid such things (..._allowed) for CPU model features
>>> in the future and clue all new such stuff to cpumodel_allowed.
>>
>> Yes, starting a guest with
>>    <os>
>>     <type arch='s390x' machine='s390-ccw-virtio-2.9'>hvm</type>
>>   </os>
>>   <cpu mode='host-model'/>
>>
>> results in
>>
>> qemu-system-s390x: Some features requested in the CPU model are not available in the configuration: gs 
>>
>> Tying it to cpumodel_allowed would not help, migration-wise though.
>> libvirt would still transform
>>
>>    <os>
>>     <type arch='s390x' machine='s390-ccw-virtio-2.9'>hvm</type>
>>   </os>
>>   <cpu mode='host-model'/>
> 
> My point was, that the host model would have to be copied and _remain_
> there when s390-ccw-virtio was expanded to s390-ccw-virtio-2.9.
> 
> So really replacing <cpu mode='host-model'/> by the model z13-base....
> This would at least fix this issue. Just like s390-ccw-virtio get's
> replaced and remains thats way.
> 
> But this might for sure have other problems.

The problem goes much further.
A fresh guest with

    <os>
     <type arch='s390x' machine='s390-ccw-virtio-2.9'>hvm</type>
   </os>
   <cpu mode='host-model'/>

does not start. No migration from an older system is necessary.

Re: [Qemu-devel] [libvirt] Question about the host-model CPU mode

[David Hildenbrand]

On 20.10.2017 15:49, Christian Borntraeger wrote:
> 
> 
> On 10/20/2017 03:43 PM, David Hildenbrand wrote:
>> On 20.10.2017 15:36, Christian Borntraeger wrote:
>>>
>>>
>>> On 10/20/2017 03:16 PM, David Hildenbrand wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> we recently encountered the problem that the 'host-model' [1] has to be
>>>>> related to the machine type of a domain. We have following problem:
>>>>>
>>>>>    Let's assume we've a z13 system with a QEMU 2.9 and we define a
>>>>>    domain using the default s390-virtio-ccw machine together with the
>>>>>    host-model CPU mode [1]. The definition will have the machine
>>>>>    expanded to s390-virtio-ccw-2.9 but retain the host-model CPU mode
>>>>>    in the domain definition. In a next step we upgrade to QEMU 2.10
>>>>>    (first version to recognize z14). Everything is still fine, even
>>>>>    though the machine runs in 2.9 compatibility mode. Finally we
>>>>>    upgrade to a z14. As a consequence it is not possible to start the
>>>>>    domain anymore as the machine type doesn't support our CPU host
>>>>>    model (which is expanded at start time of the domain).
>>>>
>>>> Actually, what is the cause of that problem? I assume it is the gs
>>>> feature (gs_allowed)?
>>>>
>>>> We should really avoid such things (..._allowed) for CPU model features
>>>> in the future and clue all new such stuff to cpumodel_allowed.
>>>
>>> Yes, starting a guest with
>>>    <os>
>>>     <type arch='s390x' machine='s390-ccw-virtio-2.9'>hvm</type>
>>>   </os>
>>>   <cpu mode='host-model'/>
>>>
>>> results in
>>>
>>> qemu-system-s390x: Some features requested in the CPU model are not available in the configuration: gs 
>>>
>>> Tying it to cpumodel_allowed would not help, migration-wise though.
>>> libvirt would still transform
>>>
>>>    <os>
>>>     <type arch='s390x' machine='s390-ccw-virtio-2.9'>hvm</type>
>>>   </os>
>>>   <cpu mode='host-model'/>
>>
>> My point was, that the host model would have to be copied and _remain_
>> there when s390-ccw-virtio was expanded to s390-ccw-virtio-2.9.
>>
>> So really replacing <cpu mode='host-model'/> by the model z13-base....
>> This would at least fix this issue. Just like s390-ccw-virtio get's
>> replaced and remains thats way.
>>
>> But this might for sure have other problems.
> 
> The problem goes much further.
> A fresh guest with
> 
>     <os>
>      <type arch='s390x' machine='s390-ccw-virtio-2.9'>hvm</type>
>    </os>
>    <cpu mode='host-model'/>
> 
> does not start. No migration from an older system is necessary.
> 

Yes, as stated in the documentation "copying host CPU definition from
capabilities XML" this can not work. And it works just as documented.
Not saying this is a nice thing :)

I think we should try to fix gs_allowed (if possible) and avoid
something like that in the future. This would avoid other complexity
involved when suddenly having X host models.

-- 

Thanks,

David

Re: [Qemu-devel] [libvirt] Question about the host-model CPU mode

[Christian Borntraeger]

On 10/20/2017 03:51 PM, David Hildenbrand wrote:
[...]
>> The problem goes much further.
>> A fresh guest with
>>
>>     <os>
>>      <type arch='s390x' machine='s390-ccw-virtio-2.9'>hvm</type>
>>    </os>
>>    <cpu mode='host-model'/>
>>
>> does not start. No migration from an older system is necessary.
>>
> 
> Yes, as stated in the documentation "copying host CPU definition from
> capabilities XML" this can not work. And it works just as documented.
> Not saying this is a nice thing :)
> 
> I think we should try to fix gs_allowed (if possible) and avoid
> something like that in the future. This would avoid other complexity
> involved when suddenly having X host models.

Maybe this one is really a proper fix. It will allow the guest to start
and on migration the cpu model will complain if the target cannot provide gs.
Similar things can happen if - for example - the host kernel lacks some features.


diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c
index 77169bb..97a08fa 100644
--- a/hw/s390x/s390-virtio-ccw.c
+++ b/hw/s390x/s390-virtio-ccw.c
@@ -430,7 +430,6 @@ static void ccw_machine_class_init(ObjectClass *oc, void *data)
     s390mc->ri_allowed = true;
     s390mc->cpu_model_allowed = true;
     s390mc->css_migration_enabled = true;
-    s390mc->gs_allowed = true;
     mc->init = ccw_init;
     mc->reset = s390_machine_reset;
     mc->hot_add_cpu = s390_hot_add_cpu;
@@ -509,12 +508,6 @@ bool cpu_model_allowed(void)
     return get_machine_class()->cpu_model_allowed;
 }
 
-bool gs_allowed(void)
-{
-    /* for "none" machine this results in true */
-    return get_machine_class()->gs_allowed;
-}
-
 static char *machine_get_loadparm(Object *obj, Error **errp)
 {
     S390CcwMachineState *ms = S390_CCW_MACHINE(obj);
@@ -757,7 +750,6 @@ static void ccw_machine_2_9_class_options(MachineClass *mc)
 {
     S390CcwMachineClass *s390mc = S390_MACHINE_CLASS(mc);
 
-    s390mc->gs_allowed = false;
     ccw_machine_2_10_class_options(mc);
     SET_MACHINE_COMPAT(mc, CCW_COMPAT_2_9);
     s390mc->css_migration_enabled = false;
diff --git a/include/hw/s390x/s390-virtio-ccw.h b/include/hw/s390x/s390-virtio-ccw.h
index a9a90c2..1de53b0 100644
--- a/include/hw/s390x/s390-virtio-ccw.h
+++ b/include/hw/s390x/s390-virtio-ccw.h
@@ -40,7 +40,6 @@ typedef struct S390CcwMachineClass {
     bool ri_allowed;
     bool cpu_model_allowed;
     bool css_migration_enabled;
-    bool gs_allowed;
 } S390CcwMachineClass;
 
 /* runtime-instrumentation allowed by the machine */
diff --git a/target/s390x/kvm.c b/target/s390x/kvm.c
index a0d5052..3f13fc2 100644
--- a/target/s390x/kvm.c
+++ b/target/s390x/kvm.c
@@ -362,7 +362,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
             cap_ri = 1;
         }
     }
-    if (gs_allowed()) {
+    if (cpu_model_allowed()) {
         if (kvm_vm_enable_cap(s, KVM_CAP_S390_GS, 0) == 0) {
             cap_gs = 1;
         }

Re: [Qemu-devel] [libvirt] Question about the host-model CPU mode

[David Hildenbrand]

On 20.10.2017 16:02, Christian Borntraeger wrote:
> 
> 
> On 10/20/2017 03:51 PM, David Hildenbrand wrote:
> [...]
>>> The problem goes much further.
>>> A fresh guest with
>>>
>>>     <os>
>>>      <type arch='s390x' machine='s390-ccw-virtio-2.9'>hvm</type>
>>>    </os>
>>>    <cpu mode='host-model'/>
>>>
>>> does not start. No migration from an older system is necessary.
>>>
>>
>> Yes, as stated in the documentation "copying host CPU definition from
>> capabilities XML" this can not work. And it works just as documented.
>> Not saying this is a nice thing :)
>>
>> I think we should try to fix gs_allowed (if possible) and avoid
>> something like that in the future. This would avoid other complexity
>> involved when suddenly having X host models.
> 
> Maybe this one is really a proper fix. It will allow the guest to start
> and on migration the cpu model will complain if the target cannot provide gs.
> Similar things can happen if - for example - the host kernel lacks some features.

Right, just what I had in mind.

> 
> 
> diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c
> index 77169bb..97a08fa 100644
> --- a/hw/s390x/s390-virtio-ccw.c
> +++ b/hw/s390x/s390-virtio-ccw.c
> @@ -430,7 +430,6 @@ static void ccw_machine_class_init(ObjectClass *oc, void *data)
>      s390mc->ri_allowed = true;
>      s390mc->cpu_model_allowed = true;
>      s390mc->css_migration_enabled = true;
> -    s390mc->gs_allowed = true;
>      mc->init = ccw_init;
>      mc->reset = s390_machine_reset;
>      mc->hot_add_cpu = s390_hot_add_cpu;
> @@ -509,12 +508,6 @@ bool cpu_model_allowed(void)
>      return get_machine_class()->cpu_model_allowed;
>  }
>  
> -bool gs_allowed(void)
> -{
> -    /* for "none" machine this results in true */
> -    return get_machine_class()->gs_allowed;
> -}
> -
>  static char *machine_get_loadparm(Object *obj, Error **errp)
>  {
>      S390CcwMachineState *ms = S390_CCW_MACHINE(obj);
> @@ -757,7 +750,6 @@ static void ccw_machine_2_9_class_options(MachineClass *mc)
>  {
>      S390CcwMachineClass *s390mc = S390_MACHINE_CLASS(mc);
>  
> -    s390mc->gs_allowed = false;
>      ccw_machine_2_10_class_options(mc);
>      SET_MACHINE_COMPAT(mc, CCW_COMPAT_2_9);
>      s390mc->css_migration_enabled = false;
> diff --git a/include/hw/s390x/s390-virtio-ccw.h b/include/hw/s390x/s390-virtio-ccw.h
> index a9a90c2..1de53b0 100644
> --- a/include/hw/s390x/s390-virtio-ccw.h
> +++ b/include/hw/s390x/s390-virtio-ccw.h
> @@ -40,7 +40,6 @@ typedef struct S390CcwMachineClass {
>      bool ri_allowed;
>      bool cpu_model_allowed;
>      bool css_migration_enabled;
> -    bool gs_allowed;
>  } S390CcwMachineClass;
>  
>  /* runtime-instrumentation allowed by the machine */
> diff --git a/target/s390x/kvm.c b/target/s390x/kvm.c
> index a0d5052..3f13fc2 100644
> --- a/target/s390x/kvm.c
> +++ b/target/s390x/kvm.c
> @@ -362,7 +362,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
>              cap_ri = 1;
>          }
>      }
> -    if (gs_allowed()) {
> +    if (cpu_model_allowed()) {
>          if (kvm_vm_enable_cap(s, KVM_CAP_S390_GS, 0) == 0) {
>              cap_gs = 1;
>          }
> 

And the last hunk makes sure we keep same handling for machines without
CPU model support and therefore no way to mask support. For all recent
machines, we expect CPU model checks to be in place.

Will have to think about this a bit more. Will you send this as a proper
patch?

-- 

Thanks,

David

Re: [Qemu-devel] [libvirt] Question about the host-model CPU mode

[Christian Borntraeger]

On 10/20/2017 04:06 PM, David Hildenbrand wrote:
> On 20.10.2017 16:02, Christian Borntraeger wrote:
>>
>>
>> On 10/20/2017 03:51 PM, David Hildenbrand wrote:
>> [...]
>>>> The problem goes much further.
>>>> A fresh guest with
>>>>
>>>>     <os>
>>>>      <type arch='s390x' machine='s390-ccw-virtio-2.9'>hvm</type>
>>>>    </os>
>>>>    <cpu mode='host-model'/>
>>>>
>>>> does not start. No migration from an older system is necessary.
>>>>
>>>
>>> Yes, as stated in the documentation "copying host CPU definition from
>>> capabilities XML" this can not work. And it works just as documented.
>>> Not saying this is a nice thing :)
>>>
>>> I think we should try to fix gs_allowed (if possible) and avoid
>>> something like that in the future. This would avoid other complexity
>>> involved when suddenly having X host models.
>>
>> Maybe this one is really a proper fix. It will allow the guest to start
>> and on migration the cpu model will complain if the target cannot provide gs.
>> Similar things can happen if - for example - the host kernel lacks some features.
> 
> Right, just what I had in mind.
> 
>>
>>
>> diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c
>> index 77169bb..97a08fa 100644
>> --- a/hw/s390x/s390-virtio-ccw.c
>> +++ b/hw/s390x/s390-virtio-ccw.c
>> @@ -430,7 +430,6 @@ static void ccw_machine_class_init(ObjectClass *oc, void *data)
>>      s390mc->ri_allowed = true;
>>      s390mc->cpu_model_allowed = true;
>>      s390mc->css_migration_enabled = true;
>> -    s390mc->gs_allowed = true;
>>      mc->init = ccw_init;
>>      mc->reset = s390_machine_reset;
>>      mc->hot_add_cpu = s390_hot_add_cpu;
>> @@ -509,12 +508,6 @@ bool cpu_model_allowed(void)
>>      return get_machine_class()->cpu_model_allowed;
>>  }
>>  
>> -bool gs_allowed(void)
>> -{
>> -    /* for "none" machine this results in true */
>> -    return get_machine_class()->gs_allowed;
>> -}
>> -
>>  static char *machine_get_loadparm(Object *obj, Error **errp)
>>  {
>>      S390CcwMachineState *ms = S390_CCW_MACHINE(obj);
>> @@ -757,7 +750,6 @@ static void ccw_machine_2_9_class_options(MachineClass *mc)
>>  {
>>      S390CcwMachineClass *s390mc = S390_MACHINE_CLASS(mc);
>>  
>> -    s390mc->gs_allowed = false;
>>      ccw_machine_2_10_class_options(mc);
>>      SET_MACHINE_COMPAT(mc, CCW_COMPAT_2_9);
>>      s390mc->css_migration_enabled = false;
>> diff --git a/include/hw/s390x/s390-virtio-ccw.h b/include/hw/s390x/s390-virtio-ccw.h
>> index a9a90c2..1de53b0 100644
>> --- a/include/hw/s390x/s390-virtio-ccw.h
>> +++ b/include/hw/s390x/s390-virtio-ccw.h
>> @@ -40,7 +40,6 @@ typedef struct S390CcwMachineClass {
>>      bool ri_allowed;
>>      bool cpu_model_allowed;
>>      bool css_migration_enabled;
>> -    bool gs_allowed;
>>  } S390CcwMachineClass;
>>  
>>  /* runtime-instrumentation allowed by the machine */
>> diff --git a/target/s390x/kvm.c b/target/s390x/kvm.c
>> index a0d5052..3f13fc2 100644
>> --- a/target/s390x/kvm.c
>> +++ b/target/s390x/kvm.c
>> @@ -362,7 +362,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
>>              cap_ri = 1;
>>          }
>>      }
>> -    if (gs_allowed()) {
>> +    if (cpu_model_allowed()) {
>>          if (kvm_vm_enable_cap(s, KVM_CAP_S390_GS, 0) == 0) {
>>              cap_gs = 1;
>>          }
>>
> 
> And the last hunk makes sure we keep same handling for machines without
> CPU model support and therefore no way to mask support. For all recent
> machines, we expect CPU model checks to be in place.
> 
> Will have to think about this a bit more. Will you send this as a proper
> patch?

After thinking about it :-)

Re: [Qemu-devel] [libvirt] Question about the host-model CPU mode

[Halil Pasic]

On 10/20/2017 04:12 PM, Christian Borntraeger wrote:
> 
> 
> On 10/20/2017 04:06 PM, David Hildenbrand wrote:
>> On 20.10.2017 16:02, Christian Borntraeger wrote:
>>>
>>>
>>> On 10/20/2017 03:51 PM, David Hildenbrand wrote:
>>> [...]
>>>>> The problem goes much further.
>>>>> A fresh guest with
>>>>>
>>>>>     <os>
>>>>>      <type arch='s390x' machine='s390-ccw-virtio-2.9'>hvm</type>
>>>>>    </os>
>>>>>    <cpu mode='host-model'/>
>>>>>
>>>>> does not start. No migration from an older system is necessary.
>>>>>
>>>>
>>>> Yes, as stated in the documentation "copying host CPU definition from
>>>> capabilities XML" this can not work. And it works just as documented.
>>>> Not saying this is a nice thing :)
>>>>
>>>> I think we should try to fix gs_allowed (if possible) and avoid
>>>> something like that in the future. This would avoid other complexity
>>>> involved when suddenly having X host models.
>>>
>>> Maybe this one is really a proper fix. It will allow the guest to start
>>> and on migration the cpu model will complain if the target cannot provide gs.
>>> Similar things can happen if - for example - the host kernel lacks some features.
>>
>> Right, just what I had in mind.
>>
>>>
>>>
>>> diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c
>>> index 77169bb..97a08fa 100644
>>> --- a/hw/s390x/s390-virtio-ccw.c
>>> +++ b/hw/s390x/s390-virtio-ccw.c
>>> @@ -430,7 +430,6 @@ static void ccw_machine_class_init(ObjectClass *oc, void *data)
>>>      s390mc->ri_allowed = true;
>>>      s390mc->cpu_model_allowed = true;
>>>      s390mc->css_migration_enabled = true;
>>> -    s390mc->gs_allowed = true;
>>>      mc->init = ccw_init;
>>>      mc->reset = s390_machine_reset;
>>>      mc->hot_add_cpu = s390_hot_add_cpu;
>>> @@ -509,12 +508,6 @@ bool cpu_model_allowed(void)
>>>      return get_machine_class()->cpu_model_allowed;
>>>  }
>>>  
>>> -bool gs_allowed(void)
>>> -{
>>> -    /* for "none" machine this results in true */
>>> -    return get_machine_class()->gs_allowed;
>>> -}
>>> -
>>>  static char *machine_get_loadparm(Object *obj, Error **errp)
>>>  {
>>>      S390CcwMachineState *ms = S390_CCW_MACHINE(obj);
>>> @@ -757,7 +750,6 @@ static void ccw_machine_2_9_class_options(MachineClass *mc)
>>>  {
>>>      S390CcwMachineClass *s390mc = S390_MACHINE_CLASS(mc);
>>>  
>>> -    s390mc->gs_allowed = false;
>>>      ccw_machine_2_10_class_options(mc);
>>>      SET_MACHINE_COMPAT(mc, CCW_COMPAT_2_9);
>>>      s390mc->css_migration_enabled = false;
>>> diff --git a/include/hw/s390x/s390-virtio-ccw.h b/include/hw/s390x/s390-virtio-ccw.h
>>> index a9a90c2..1de53b0 100644
>>> --- a/include/hw/s390x/s390-virtio-ccw.h
>>> +++ b/include/hw/s390x/s390-virtio-ccw.h
>>> @@ -40,7 +40,6 @@ typedef struct S390CcwMachineClass {
>>>      bool ri_allowed;
>>>      bool cpu_model_allowed;
>>>      bool css_migration_enabled;
>>> -    bool gs_allowed;
>>>  } S390CcwMachineClass;
>>>  
>>>  /* runtime-instrumentation allowed by the machine */
>>> diff --git a/target/s390x/kvm.c b/target/s390x/kvm.c
>>> index a0d5052..3f13fc2 100644
>>> --- a/target/s390x/kvm.c
>>> +++ b/target/s390x/kvm.c
>>> @@ -362,7 +362,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
>>>              cap_ri = 1;
>>>          }
>>>      }
>>> -    if (gs_allowed()) {
>>> +    if (cpu_model_allowed()) {
>>>          if (kvm_vm_enable_cap(s, KVM_CAP_S390_GS, 0) == 0) {
>>>              cap_gs = 1;
>>>          }
>>>
>>
>> And the last hunk makes sure we keep same handling for machines without
>> CPU model support and therefore no way to mask support. For all recent
>> machines, we expect CPU model checks to be in place.
>>
>> Will have to think about this a bit more. Will you send this as a proper
>> patch?
> 
> After thinking about it :-)
> 

I intend to put some brain-power in this too. Probably next week.

My general impression is, that I have a at places different understanding
of how things should work compared to David. Especially when it comes
to this concept of persistent copying, and also an end-user-digestible
definition of what host-model does. (I think this with copying capabilities
from whatever xml which is subject to convoluted caching is a bit too
hard to digest for an end user not involved in the development of qemu
and libvirt).

I had a conversation with Boris a couple of hours ago, and it seems, things
are pretty convoluted.

If I understand the train of thought here (David) it can be summarized like this:
For a certain QEMU binary no aspect of the cpu-models may depend on the
machine type. In particular, compat properties and compat handling is
not alowed to alter cpu-models (whether the available cpu models nor the
capabilities of each of these).

This we would have to make a part of the external interface. That way
one could be sure that the 'cpu capabilities' are machine-type independent
(that is, the same for all the machine types).

Or did I get this completely wrong? (Based on the answer branches my
think).

Halil

Re: [Qemu-devel] [libvirt] Question about the host-model CPU mode

[David Hildenbrand]

> 
> I intend to put some brain-power in this too. Probably next week.
> 
> My general impression is, that I have a at places different understanding
> of how things should work compared to David. Especially when it comes
> to this concept of persistent copying, and also an end-user-digestible
> definition of what host-model does. (I think this with copying capabilities
> from whatever xml which is subject to convoluted caching is a bit too
> hard to digest for an end user not involved in the development of qemu
> and libvirt).

When reading the doc I was assuming that it would be a persistent copy.
But it would only fix part of the issue.

In the end, it specifies quite clearly what is copied. And if that is
not runnable with the selected machine, bad luck. Therefore ...

> 
> I had a conversation with Boris a couple of hours ago, and it seems, things
> are pretty convoluted.
> 
> If I understand the train of thought here (David) it can be summarized like this:
> For a certain QEMU binary no aspect of the cpu-models may depend on the
> machine type. In particular, compat properties and compat handling is
> not alowed to alter cpu-models (whether the available cpu models nor the
> capabilities of each of these).

... I always recommended avoiding such compatibility settings in the
machine. But before we had CPU models it was really all we had. No we
should let go of it.

We might be able to fix this one (gs) and take care of it in the future,
but ...

> 
> This we would have to make a part of the external interface. That way
> one could be sure that the 'cpu capabilities' are machine-type independent
> (that is, the same for all the machine types).

... the problem is once nasty stuff like zPCI comes in place. If we ever
have (other?) machine dependent stuff that toggles CPU features, we can
only try limit the damage.

> 
> Or did I get this completely wrong? (Based on the answer branches my
> think).

Not at all.

> 
> Halil
> 


-- 

Thanks,

David