Re: [Qemu-devel] libvirt<->QEMU interfaces for CPU models

Re: [Qemu-devel] libvirt<->QEMU interfaces for CPU models

[Jiri Denemark]

On Thu, Feb 21, 2013 at 11:58:18 -0300, Eduardo Habkost wrote:
> Hi,
> 
> After a long time trying to figure out the proper modelling inside QEMU,
> I believe the plans are now clearer in QEMU, so it's time to coordinate
> more closely with libvirt to try to make use of the new stuff.
> 
> I tried to enumerate the libvirt requirements and current problems, and
> how we should be able to solve those problems using the X86CPU
> subclasses and properties, on the following wiki page:
> 
> http://wiki.qemu.org/Features/CPUModels#Interfaces.2Frequirements_from_libvirt

> = Ensuring predictable set of guest features =
> 
> Requirement: libvirt needs to ensure all features required on the command-line
> are present and exposed to the guest.
> 
> Current problem: libvirt doesn't use the "enforce" flag so it can't guarantee
> that a given feature will be actually exposed to the guest.
> 
> Solution: use the "enforce" flag on the "-cpu" option.

Definitely, we plan to start using "enforce" flag as soon as we have
better CPU probing interface with QEMU. Since libvirt does not currently
consult CPU specs with QEMU, some configurations in fact rely on QEMU
dropping features it can't provide. Of course, that's bad for several
reasons but we don't want such configurations to suddenly stop working.
We want to first fix the CPU specs libvirt creates so that we know they
will work with "enforce".

>     Limitation: no proper machine-friendly interface to report which features
>     are missing.
> 
>         Workaround: See "querying for host capabilities" below.

I doubt we will be ready to start using "enforce" before the machine
friendly interface is available...


> = Listing CPU models =
> 
> Requirement: libvirt needs to know which CPU models are available to be used
> with the "-cpu" option.
> 
> Current problem: libvirt relies on help output parsing for that.
> 
> Solution: use QMP qom-list-types command.
> 
>     Dependency: X86CPU subclasses.
>     Limitation: needs a live QEMU process for the query.

No problem, we already run QEMU and use several QMP commands to probe
its capabilities. And "qom-list-types" is actually one of them. To get
the list of CPU models, we would just call

    {
        "execute": "qom-list-types",
        "arguments": {
            "implements": "X86CPU"
        }
    }

right? What about other non-x86 architectures? Will we need to use
different class name or is there a generic CPU class that could be used
universally?

> Solution: use QMP query-cpu-definitions command.
> 
>     Limitation: needs a live QEMU process for the query.

IIUC, the result of this command will depend on machine type and we
can't use -M none we currently use for probing, right?

> == Future plans ==
> 
> It would be interesting to get rid of the requirement for a live QEMU process
> (with a complete machine being created) to be already running.

Hmm, so is this complete machine needed even for getting CPU models from
qom-list-types or only for querying exact definitions using
query-cpu-definitions command?

Actually, what is query-cpu-definitions supposed to return? Currently it
seems it's just the CPU model names rather than details about all CPU
models. From the command name, one would expect to get more than just
names.


> = Getting information about CPU models =
> 
> Requirement: libvirt uses the predefined CPU models from QEMU, but it needs to
> be able to query for CPU model details, to find out how it can create a VM that
> matches what was requested by the user.
> 
> Current problem: libvirt has a copy of the CPU model definitions on its
> cpu_map.xml file, and the copy can be out of sync in case CPU models in QEMU
> change. libvirt also assumes that the set of features on each model is always
> the same on all machine-types, which is not true.
> 
>     Challenge: the resulting CPU features depend on lots of factors, including
>     the machine-type.
> 
>         Workaround: start a paused VM and query for the CPU device information
>         after the CPU was created.
> 
>     Solution: start a paused VM with no devices, but with the right
>     machine-type and right CPU model. Use QMP QOM commands to query for CPU
>     flags (especially the properties starting with the "f-" prefix).
> 
>         Dependency: X86CPU feature properties ("f-*" properties).
>         Limitation: requires a live QEMU process with the right machine-type/
>         CPU-model to be started, to make the query.

This would be very useful for ensuring the guest sees the exact same CPU
after it's been migrated or restored from a stored state or a snapshot.
Should we make sure the guest will always see the same CPU even after
shutdown or is it ok if the guest CPU changes a bit on next boot, e.g.,
in case the host kernel was upgraded and is able to provide more
features?

However, probing several CPU definitions for compatibility with
host/kernel/QEMU would be quite inefficient. Although I guess we should
be able to limit doing so only in case a specific API (e.g.,
virConnectCompareCPU or virConnectBaselineCPU) is called, which would be
acceptable, I think.


> = Querying host capabilities =
> 
> Requirement: libvirt needs to know which feature can really be enabled, before
> it tries to start a VM, and before it tries to start a live-migration process.

If we have machine friendly output of -cpu enforce, we may limit this
need to just a few configuration types (the ones effectively requesting
some kind of "host" CPU model). For other cases, libvirt could just try
to start QEMU with the requested CPU definition and report usable error
back rather than trying to pre-check the CPU.

Jirka

Re: [Qemu-devel] libvirt<->QEMU interfaces for CPU models

[Jiri Denemark]

On Thu, Feb 21, 2013 at 11:58:18 -0300, Eduardo Habkost wrote:
> = Querying host capabilities =
> 
> Requirement: libvirt needs to know which feature can really be enabled, before
> it tries to start a VM, and before it tries to start a live-migration process.
> 
> The set of available capabilities depend on:
> 
>   • Host CPU (hardware) capabilities;
>   • Kernel capabilities (reported by GET_SUPPORTED_CPUID);
>   • QEMU capabilities;
>   • Specific configuration options (e.g. in-kernel IRQ chip is required for
>     some features).

Actually, one more thing. Can any of these requirements change while a
host is up and QEMU is not upgraded? I believe, host CPU capabilities
can only change when the host starts. Kernel capabilities are a bit less
clear since I guess they could possibly change when kvm module is
unloaded and loaded back with a different options. QEMU capabilities
should only change when different version is installed. And the specific
configuration options are the most unclear to me. The reason I'm asking
is whether libvirt could run-time cache CPU definitions (including all
model details) in the same way we currently cache QEMU capabilities,
such as availability of specific QMP commands.

Jirka

Re: [Qemu-devel] libvirt<->QEMU interfaces for CPU models

[Eduardo Habkost]

On Fri, Mar 01, 2013 at 02:12:38PM +0100, Jiri Denemark wrote:
> On Thu, Feb 21, 2013 at 11:58:18 -0300, Eduardo Habkost wrote:
> > Hi,
> > 
> > After a long time trying to figure out the proper modelling inside QEMU,
> > I believe the plans are now clearer in QEMU, so it's time to coordinate
> > more closely with libvirt to try to make use of the new stuff.
> > 
> > I tried to enumerate the libvirt requirements and current problems, and
> > how we should be able to solve those problems using the X86CPU
> > subclasses and properties, on the following wiki page:
> > 
> > http://wiki.qemu.org/Features/CPUModels#Interfaces.2Frequirements_from_libvirt
> 
> > = Ensuring predictable set of guest features =
> > 
> > Requirement: libvirt needs to ensure all features required on the command-line
> > are present and exposed to the guest.
> > 
> > Current problem: libvirt doesn't use the "enforce" flag so it can't guarantee
> > that a given feature will be actually exposed to the guest.
> > 
> > Solution: use the "enforce" flag on the "-cpu" option.
> 
> Definitely, we plan to start using "enforce" flag as soon as we have
> better CPU probing interface with QEMU. Since libvirt does not currently
> consult CPU specs with QEMU, some configurations in fact rely on QEMU
> dropping features it can't provide. Of course, that's bad for several
> reasons but we don't want such configurations to suddenly stop working.
> We want to first fix the CPU specs libvirt creates so that we know they
> will work with "enforce".

Also: more important than fixing the CPU definitions from libvirt, is to
ask QEMU for host capabilities and CPU model definitions. The whole
point of this is to solve the CPU model data duplication/synchronization
problems between libvirt and QEMU.

Once you are able to query CPU model definitions on runtime, you don't
even need to make cpu_map.xml agree with QEMU. You can simply ask QEMU
how each model looks like, and remove/add features from the command-line
as necessary, so the resulting VM matches what the user asked for.


> 
> >     Limitation: no proper machine-friendly interface to report which features
> >     are missing.
> > 
> >         Workaround: See "querying for host capabilities" below.
> 
> I doubt we will be ready to start using "enforce" before the machine
> friendly interface is available...

If you query for the "-cpu host" capabilities first and ensure all
features from a CPU model is available, enforce is supposed to not fail.

I understand that a machine-friendly error reporting for "enforce" would
be very useful, but note that if "enforce" fails, it is probably already
too late for libvirt, and that means that what libvirt thinks about host
capabilities and CPU models is already incorrect.


The main problem preventing us from making a machine-friendly interface
is that "enforce" makes QEMU abort immediately, making us lose the main
machine-friendly communication mechanism, that is QMP.

(But I had an idea to solve that, look for "removed-features" below for
a description).


> 
> 
> > = Listing CPU models =
> > 
> > Requirement: libvirt needs to know which CPU models are available to be used
> > with the "-cpu" option.
> > 
> > Current problem: libvirt relies on help output parsing for that.
> > 
> > Solution: use QMP qom-list-types command.
> > 
> >     Dependency: X86CPU subclasses.
> >     Limitation: needs a live QEMU process for the query.
> 
> No problem, we already run QEMU and use several QMP commands to probe
> its capabilities. And "qom-list-types" is actually one of them. To get
> the list of CPU models, we would just call
> 
>     {
>         "execute": "qom-list-types",
>         "arguments": {
>             "implements": "X86CPU"
>         }
>     }
> 
> right? What about other non-x86 architectures? Will we need to use
> different class name or is there a generic CPU class that could be used
> universally?

Actually I don't know much about the QMP command syntax and didn't test
it a lot. But that's basically how I think it will look like. Except
that instead of "X86CPU", the type name is "x86_64-cpu" (on
qemu-system-x86_64) and "i386-cpu" (on qemu-system-i386). Maybe it is
easier to simply use: implements: "cpu", abstract: false }.

We may also end up with different CPU model classes for KVM and TCG,
this is still under discussion.

Another caveat: the CPU model class names will be longer than the names
used on the "-cpu" command-line: something like "<model>-<arch>-cpu" or
"<model>-kvm-<arch>-cpu".


> 
> > Solution: use QMP query-cpu-definitions command.
> > 
> >     Limitation: needs a live QEMU process for the query.
> 
> IIUC, the result of this command will depend on machine type and we
> can't use -M none we currently use for probing, right?

The class data don't depend on machine-types. But the resulting CPU
objects may look different depending on machine type.

In other words: you don't need -M to list CPU models, but you need -M if
you want to know which features are going to be available on each model,
exactly.

> 
> > == Future plans ==
> > 
> > It would be interesting to get rid of the requirement for a live QEMU process
> > (with a complete machine being created) to be already running.
> 
> Hmm, so is this complete machine needed even for getting CPU models from
> qom-list-types or only for querying exact definitions using
> query-cpu-definitions command?

Maybe "complete machine" isn't the right expression, here. What I mean is:
AFAIK, it is not possible to get a QMP monitor without actually having a
machine being created by QEMU (even if it is a machine that will never run).

But answering your question: to query the CPU model list you don't need
the right machine type or anything fancy. "qemu -nodefaults" should
work. But to query the exact CPU definition (the item below), you need
to specify the right machine-type.

> 
> Actually, what is query-cpu-definitions supposed to return? Currently it
> seems it's just the CPU model names rather than details about all CPU
> models. From the command name, one would expect to get more than just
> names.

query-cpu-definitions returns only the CPU model names, and a proposal
to include more detailed CPUID information was rejected, IIRC. I believe
we should deprecate query-cpu-definitions it in favor of the QOM
commands.

> 
> 
> > = Getting information about CPU models =
> > 
> > Requirement: libvirt uses the predefined CPU models from QEMU, but it needs to
> > be able to query for CPU model details, to find out how it can create a VM that
> > matches what was requested by the user.
> > 
> > Current problem: libvirt has a copy of the CPU model definitions on its
> > cpu_map.xml file, and the copy can be out of sync in case CPU models in QEMU
> > change. libvirt also assumes that the set of features on each model is always
> > the same on all machine-types, which is not true.
> > 
> >     Challenge: the resulting CPU features depend on lots of factors, including
> >     the machine-type.
> > 
> >         Workaround: start a paused VM and query for the CPU device information
> >         after the CPU was created.

I just noticed another problem here, but this gave me an idea that would
help solve the "enforce" error reporting problem:

  Problem: "qemu -machine <M> -cpu <model>" will create CPU objects
  where the CPU features are _already_ filtered based on host
  capabilities.

    * Using "enforce" wouldn't solve it, because then QEMU would abort, and
      QMP would be unavailable.

    Solution: we could have a CPU object property like
    "removed-features" that would have the list of features that were
    disabled because they are not supported by the host (and would make
    "enforce" fail).

      * This would solve the problem above and also be a machine-friendly
        way to check for possible "enforce" errors.

      * In other words: instead of "enforce", libvirt could use "check"
        instead of "enforce", and before unpausing the VM (or even starting
        migration), it should first check if the "removed-features" property is
        empty.

Would that work for you?


> > 
> >     Solution: start a paused VM with no devices, but with the right
> >     machine-type and right CPU model. Use QMP QOM commands to query for CPU
> >     flags (especially the properties starting with the "f-" prefix).
> > 
> >         Dependency: X86CPU feature properties ("f-*" properties).
> >         Limitation: requires a live QEMU process with the right machine-type/
> >         CPU-model to be started, to make the query.
> 
> This would be very useful for ensuring the guest sees the exact same CPU
> after it's been migrated or restored from a stored state or a snapshot.
> Should we make sure the guest will always see the same CPU even after
> shutdown or is it ok if the guest CPU changes a bit on next boot, e.g.,
> in case the host kernel was upgraded and is able to provide more
> features?

It is up to you, it depends on what the libvirt API promises. But I always
assumed that the same libvirt XML config would always create an equivalent
machine (with not guest-visible differences), it didn't matter on which host it
is running. Otherwise you would have problems like:

 * A VM being migratable to another host and suddenly it stops being
   migratable just because it was restarted.
 * A VM being migratable from A to B when started on host A,
   but not migratable from B to A if exactly the same VM XML is started
   on host B.


> 
> However, probing several CPU definitions for compatibility with
> host/kernel/QEMU would be quite inefficient. Although I guess we should
> be able to limit doing so only in case a specific API (e.g.,
> virConnectCompareCPU or virConnectBaselineCPU) is called, which would be
> acceptable, I think.

I am not familiar with the libvirt code, but maybe the only cases where
you need to do the probing are:

 * virConnectCompareCPU()
 * virConnectBaselineCPU() (I don't understand completely what it does)
 * Before starting a VM, when using match={minimu,exact,strict}.
 * When "host-model"/VIR_DOMAIN_XML_UPDATE_CPU is used
 * When checking if a given VM is migratable to a specific host (before
   actually starting migration). How is that implemented today?

But I worry that at some point you may need to list detailed information
about every CPU model, and this would require running QEMU once for each
CPU model. Would that be a huge problem, or something we can work with?

I quickly tested how quick QEMU can answer QMP queries:

    $ time sh -c '(echo "{\"execute\":\"qmp_capabilities\"}";echo "{\"execute\":\"qom-list-types\", \"arguments\": {}}";echo "{\"execute\":\"quit\"}"; ) | qemu-system-x86_64 -S -nodefaults -cpu SandyBridge,check -qmp stdio -nographic'
    [...]
    real    0m0.038s
    user    0m0.024s
    sys     0m0.012s

Maybe this is reasonable enough? You probably just need to do that before
starting a VM you never started before (for the cases where you want to choose
the CPU model that would generate the shorter command-line).

Also, note that the resulting CPU features for a given machine-type + CPU pair
is supposed to never ever change on any future QEMU version, so libvirt should
be able to safely cache those values even if the QEMU binary changes.

> 
> 
> > = Querying host capabilities =
> > 
> > Requirement: libvirt needs to know which feature can really be enabled, before
> > it tries to start a VM, and before it tries to start a live-migration process.
> 
> If we have machine friendly output of -cpu enforce, we may limit this
> need to just a few configuration types (the ones effectively requesting
> some kind of "host" CPU model). For other cases, libvirt could just try
> to start QEMU with the requested CPU definition and report usable error
> back rather than trying to pre-check the CPU.

True.


PS: I will be out of office for 3 weeks, starting tomorrow, so I will probably
not be able to continue this discussion after today, until I'm back on
25/March.

-- 
Eduardo

Re: [Qemu-devel] libvirt<->QEMU interfaces for CPU models

[Eduardo Habkost]

On Fri, Mar 01, 2013 at 02:28:37PM +0100, Jiri Denemark wrote:
> On Thu, Feb 21, 2013 at 11:58:18 -0300, Eduardo Habkost wrote:
> > = Querying host capabilities =
> > 
> > Requirement: libvirt needs to know which feature can really be enabled, before
> > it tries to start a VM, and before it tries to start a live-migration process.
> > 
> > The set of available capabilities depend on:
> > 
> >   • Host CPU (hardware) capabilities;
> >   • Kernel capabilities (reported by GET_SUPPORTED_CPUID);
> >   • QEMU capabilities;
> >   • Specific configuration options (e.g. in-kernel IRQ chip is required for
> >     some features).
> 
> Actually, one more thing. Can any of these requirements change while a
> host is up and QEMU is not upgraded? I believe, host CPU capabilities
> can only change when the host starts. Kernel capabilities are a bit less
> clear since I guess they could possibly change when kvm module is
> unloaded and loaded back with a different options. QEMU capabilities
> should only change when different version is installed. And the specific
> configuration options are the most unclear to me. The reason I'm asking
> is whether libvirt could run-time cache CPU definitions (including all
> model details) in the same way we currently cache QEMU capabilities,
> such as availability of specific QMP commands.


That's a good question. Let's check each item so I don't forget any
detail:

> >   • Host CPU (hardware) capabilities;

This shouldn't change without a host reboot.

> >   • Kernel capabilities (reported by GET_SUPPORTED_CPUID);

This may possibly change if the KVM module is unloaded and reloaded with
different options, but... I guess we should simply require libvirtd to
be restarted if any user does that?

> >   • QEMU capabilities;

This shouldn't change as long as the QEMU binary doesn't change.


> >   • Specific configuration options (e.g. in-kernel IRQ chip is required for
> >     some features).

This part seems tricky. Currently kernel-irqchip is probably the only
option that affects which features are available, but what if other QEMU
options affect the set of features too?

I believe the answer is to rely on machine-types. I mean: if a new
option that affects "-cpu host" and the set of available CPU features is
created, there are two options:

  1) Using the default value;
  2) Setting the option explicitly.

1) If using the default value, the default will depend on machine-type, so
libvirt should already consider machine-type as an option that affects
available-CPU-features.

2) If using an explicit value, libvirt should use the explicit value only
after being changed to take into account the fact that the option
affects available-CPU-features.

So, let's add one more item to the list. The set of available
capabilities depend on:

  • Host CPU (hardware) capabilities;
  • Kernel capabilities (reported by GET_SUPPORTED_CPUID);
  • QEMU capabilities;
  • Specific configuration options:
    • kernel-irqchip (currently affects tsc-deadline and x2apic, but may
      affect other features in the future)
    • machine-type (may affect any feature in the future)

-- 
Eduardo

Re: [Qemu-devel] libvirt<->QEMU interfaces for CPU models

[Andreas Färber]

Am 01.03.2013 14:12, schrieb Jiri Denemark:
> On Thu, Feb 21, 2013 at 11:58:18 -0300, Eduardo Habkost wrote:
>> = Listing CPU models =
>>
>> Requirement: libvirt needs to know which CPU models are available to be used
>> with the "-cpu" option.
>>
>> Current problem: libvirt relies on help output parsing for that.

query-cpu-definitions is the QMP command to retrieve values compatible
with -cpu.

And if libvirt is not using it, I really don't understand why the work
of maintaining this crappy interface has been pushed onto us in the
first place? There is no reuse between -cpu ? and QMP implementations so
it's just extra work, there is no communicated or implemented way to
extend the arch_query_cpu_definitions() implementation to become more
usable for command line output implementation (e.g., associating a PVR
value with the model name for ppc) and, while we're at it, it uses
global functions plus a stub rather than a CPUState hook with a no-op
default implementation in qom/cpu.c...

>> Solution: use QMP qom-list-types command.
>>
>>     Dependency: X86CPU subclasses.
>>     Limitation: needs a live QEMU process for the query.
> 
> No problem, we already run QEMU and use several QMP commands to probe
> its capabilities. And "qom-list-types" is actually one of them. To get
> the list of CPU models, we would just call
> 
>     {
>         "execute": "qom-list-types",
>         "arguments": {
>             "implements": "X86CPU"
>         }
>     }
> 
> right?

Not quite, this would return abstract types as well, so you'd need to
add "abstract": "false" or so.
And you need to use the type name, not the struct name as argument, i.e.
"i386-cpu" or "x86_64-cpu". Note: This does not always match the
executable name since QOM names are supposed to be verbose (e.g., ppc64
vs. powerpc64-cpu) and some executables are misnamed (sh4 vs. superh-cpu).

For x86 today this will return one type, either "i386-cpu" or
"x86_64-cpu", that's why I have been pushing to implement model
subclasses. There's still some controversial discussions about how this
relates to KVM and TCG changing values of classes.

> What about other non-x86 architectures?

For some other architectures like arm this will return the full list of
available classes, but in "cortex-a9-arm-cpu" format, which is not
guaranteed to be compatible with -cpu but rather with -device where
already applicable.

> Will we need to use
> different class name or is there a generic CPU class that could be used
> universally?

"cpu" would currently work as well, but the CPU refactorings are
targetting to compile, e.g., arm-cpu and microblaze-cpu types into the
same executable, so "cpu" may lead to undesired results in the future
depending on what assumptions your code makes.

Regards,
Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] libvirt<->QEMU interfaces for CPU models

[Daniel P. Berrange]

On Fri, Mar 01, 2013 at 07:31:46PM +0100, Andreas Färber wrote:
> Am 01.03.2013 14:12, schrieb Jiri Denemark:
> > On Thu, Feb 21, 2013 at 11:58:18 -0300, Eduardo Habkost wrote:
> >> = Listing CPU models =
> >>
> >> Requirement: libvirt needs to know which CPU models are available to be used
> >> with the "-cpu" option.
> >>
> >> Current problem: libvirt relies on help output parsing for that.
> 
> query-cpu-definitions is the QMP command to retrieve values compatible
> with -cpu.

Yep, Jiri is wrong here actually. -cpu parsing is what we used to
do. With latest libvirt + QEMU we do use query-cpu-definitions
and have no "help parsing" code used at all.


Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] libvirt<->QEMU interfaces for CPU models

[Eduardo Habkost]

On Fri, Mar 01, 2013 at 07:31:46PM +0100, Andreas Färber wrote:
> Am 01.03.2013 14:12, schrieb Jiri Denemark:
> > On Thu, Feb 21, 2013 at 11:58:18 -0300, Eduardo Habkost wrote:
> >> = Listing CPU models =
> >>
> >> Requirement: libvirt needs to know which CPU models are available to be used
> >> with the "-cpu" option.
> >>
> >> Current problem: libvirt relies on help output parsing for that.
> 
> query-cpu-definitions is the QMP command to retrieve values compatible
> with -cpu.
> 
> And if libvirt is not using it, I really don't understand why the work
> of maintaining this crappy interface has been pushed onto us in the
> first place? There is no reuse between -cpu ? and QMP implementations so
> it's just extra work, there is no communicated or implemented way to
> extend the arch_query_cpu_definitions() implementation to become more
> usable for command line output implementation (e.g., associating a PVR
> value with the model name for ppc) and, while we're at it, it uses
> global functions plus a stub rather than a CPUState hook with a no-op
> default implementation in qom/cpu.c...

I have the same questions you have.  :-)

But my main complaint about query-cpu-definitions is not about the
implementation: it's that the interface was introduced without taking
into account the requirements of libvirt regarding CPU features. It was
found to be not appropriate for what libvirt needs[1], but somehow it
got applied anyway.

[1] http://article.gmane.org/gmane.comp.emulators.qemu/164772

-- 
Eduardo

Re: [Qemu-devel] libvirt<->QEMU interfaces for CPU models

[Eduardo Habkost]

On Fri, Mar 01, 2013 at 06:34:31PM +0000, Daniel P. Berrange wrote:
> On Fri, Mar 01, 2013 at 07:31:46PM +0100, Andreas Färber wrote:
> > Am 01.03.2013 14:12, schrieb Jiri Denemark:
> > > On Thu, Feb 21, 2013 at 11:58:18 -0300, Eduardo Habkost wrote:
> > >> = Listing CPU models =
> > >>
> > >> Requirement: libvirt needs to know which CPU models are available to be used
> > >> with the "-cpu" option.
> > >>
> > >> Current problem: libvirt relies on help output parsing for that.
> > 
> > query-cpu-definitions is the QMP command to retrieve values compatible
> > with -cpu.
> 
> Yep, Jiri is wrong here actually. -cpu parsing is what we used to
> do. With latest libvirt + QEMU we do use query-cpu-definitions
> and have no "help parsing" code used at all.

I wrote the above, not Jiri. I will fix the wiki page to note that.

-- 
Eduardo

Re: [Qemu-devel] libvirt<->QEMU interfaces for CPU models

[Jiri Denemark]

On Fri, Mar 01, 2013 at 12:02:07 -0300, Eduardo Habkost wrote:
> On Fri, Mar 01, 2013 at 02:12:38PM +0100, Jiri Denemark wrote:
> > Definitely, we plan to start using "enforce" flag as soon as we have
> > better CPU probing interface with QEMU. Since libvirt does not currently
> > consult CPU specs with QEMU, some configurations in fact rely on QEMU
> > dropping features it can't provide. Of course, that's bad for several
> > reasons but we don't want such configurations to suddenly stop working.
> > We want to first fix the CPU specs libvirt creates so that we know they
> > will work with "enforce".
> 
> Also: more important than fixing the CPU definitions from libvirt, is to
> ask QEMU for host capabilities and CPU model definitions. The whole
> point of this is to solve the CPU model data duplication/synchronization
> problems between libvirt and QEMU.
> 
> Once you are able to query CPU model definitions on runtime, you don't
> even need to make cpu_map.xml agree with QEMU. You can simply ask QEMU
> how each model looks like, and remove/add features from the command-line
> as necessary, so the resulting VM matches what the user asked for.

Right, that's what I had in mind. By libvirt providing correct CPU
definitions (probably better called the right -cpu command line option)
I meant that we need to actually probe QEMU rather than making the
definitions on our own from cpu_map.xml and host CPUID.

> > >     Limitation: no proper machine-friendly interface to report which features
> > >     are missing.
> > > 
> > >         Workaround: See "querying for host capabilities" below.
> > 
> > I doubt we will be ready to start using "enforce" before the machine
> > friendly interface is available...
> 
> If you query for the "-cpu host" capabilities first and ensure all
> features from a CPU model is available, enforce is supposed to not fail.
> 
> I understand that a machine-friendly error reporting for "enforce" would
> be very useful, but note that if "enforce" fails, it is probably already
> too late for libvirt, and that means that what libvirt thinks about host
> capabilities and CPU models is already incorrect.

But we still need to know details about each CPU model so that we
can choose the right one. I was trying to say that by the time we have
this probing and can start using enforce, the machine friendly reporting
could be available as well and the limitation will be gone.


> > > == Future plans ==
> > > 
> > > It would be interesting to get rid of the requirement for a live QEMU process
> > > (with a complete machine being created) to be already running.
> > 
> > Hmm, so is this complete machine needed even for getting CPU models from
> > qom-list-types or only for querying exact definitions using
> > query-cpu-definitions command?
> 
> Maybe "complete machine" isn't the right expression, here. What I mean is:
> AFAIK, it is not possible to get a QMP monitor without actually having a
> machine being created by QEMU (even if it is a machine that will never run).

OK, I thought there was something special needed. We already start QEMU
in such a way that we can communicate with it using QMP monitor. So this
is just a question of using the right machine if we need to know the
details about given CPU model.

> > > = Getting information about CPU models =
> > > 
> > > Requirement: libvirt uses the predefined CPU models from QEMU, but it needs to
> > > be able to query for CPU model details, to find out how it can create a VM that
> > > matches what was requested by the user.
> > > 
> > > Current problem: libvirt has a copy of the CPU model definitions on its
> > > cpu_map.xml file, and the copy can be out of sync in case CPU models in QEMU
> > > change. libvirt also assumes that the set of features on each model is always
> > > the same on all machine-types, which is not true.
> > > 
> > >     Challenge: the resulting CPU features depend on lots of factors, including
> > >     the machine-type.
> > > 
> > >         Workaround: start a paused VM and query for the CPU device information
> > >         after the CPU was created.
> 
> I just noticed another problem here, but this gave me an idea that would
> help solve the "enforce" error reporting problem:
> 
>   Problem: "qemu -machine <M> -cpu <model>" will create CPU objects
>   where the CPU features are _already_ filtered based on host
>   capabilities.

Ah, it seems logical now that you mention it :-)

>     * Using "enforce" wouldn't solve it, because then QEMU would abort, and
>       QMP would be unavailable.
> 
>     Solution: we could have a CPU object property like
>     "removed-features" that would have the list of features that were
>     disabled because they are not supported by the host (and would make
>     "enforce" fail).
> 
>       * This would solve the problem above and also be a machine-friendly
>         way to check for possible "enforce" errors.
> 
>       * In other words: instead of "enforce", libvirt could use "check"
>         instead of "enforce", and before unpausing the VM (or even starting
>         migration), it should first check if the "removed-features" property is
>         empty.
> 
> Would that work for you?

Yes, that seems like it could work. In fact, it seems much better than
using enforce and trying to deal with aborted QEMU.


Jirka

Re: [Qemu-devel] libvirt<->QEMU interfaces for CPU models

[Daniel P. Berrange]

On Fri, Mar 01, 2013 at 03:58:18PM -0300, Eduardo Habkost wrote:
> On Fri, Mar 01, 2013 at 07:31:46PM +0100, Andreas Färber wrote:
> > Am 01.03.2013 14:12, schrieb Jiri Denemark:
> > > On Thu, Feb 21, 2013 at 11:58:18 -0300, Eduardo Habkost wrote:
> > >> = Listing CPU models =
> > >>
> > >> Requirement: libvirt needs to know which CPU models are available to be used
> > >> with the "-cpu" option.
> > >>
> > >> Current problem: libvirt relies on help output parsing for that.
> > 
> > query-cpu-definitions is the QMP command to retrieve values compatible
> > with -cpu.
> > 
> > And if libvirt is not using it, I really don't understand why the work
> > of maintaining this crappy interface has been pushed onto us in the
> > first place? There is no reuse between -cpu ? and QMP implementations so
> > it's just extra work, there is no communicated or implemented way to
> > extend the arch_query_cpu_definitions() implementation to become more
> > usable for command line output implementation (e.g., associating a PVR
> > value with the model name for ppc) and, while we're at it, it uses
> > global functions plus a stub rather than a CPUState hook with a no-op
> > default implementation in qom/cpu.c...
> 
> I have the same questions you have.  :-)
> 
> But my main complaint about query-cpu-definitions is not about the
> implementation: it's that the interface was introduced without taking
> into account the requirements of libvirt regarding CPU features. It was
> found to be not appropriate for what libvirt needs[1], but somehow it
> got applied anyway.

The requirement at the time we did this was just to have a straight
adaptation of the stuff libvirt parsed from -help, into QMP format.
The query-cpu-definitions command served that purpose fine. It was
expected that some of this may be sub-optimal in the long term. This
is fine - we can either extend the existing data, or introduce brand
new commands to deal with the problems. The key thing was just to get
100% into the QMP world for capabilities at that time, and then iterate
to improve things.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] libvirt<->QEMU interfaces for CPU models

[Eduardo Habkost]

On Fri, Mar 01, 2013 at 11:56:00PM +0100, Jiri Denemark wrote:
[...]
> > > > = Getting information about CPU models =
> > > > 
> > > > Requirement: libvirt uses the predefined CPU models from QEMU, but it needs to
> > > > be able to query for CPU model details, to find out how it can create a VM that
> > > > matches what was requested by the user.
> > > > 
> > > > Current problem: libvirt has a copy of the CPU model definitions on its
> > > > cpu_map.xml file, and the copy can be out of sync in case CPU models in QEMU
> > > > change. libvirt also assumes that the set of features on each model is always
> > > > the same on all machine-types, which is not true.
> > > > 
> > > >     Challenge: the resulting CPU features depend on lots of factors, including
> > > >     the machine-type.
> > > > 
> > > >         Workaround: start a paused VM and query for the CPU device information
> > > >         after the CPU was created.
> > 
> > I just noticed another problem here, but this gave me an idea that would
> > help solve the "enforce" error reporting problem:
> > 
> >   Problem: "qemu -machine <M> -cpu <model>" will create CPU objects
> >   where the CPU features are _already_ filtered based on host
> >   capabilities.
> 
> Ah, it seems logical now that you mention it :-)
> 
> >     * Using "enforce" wouldn't solve it, because then QEMU would abort, and
> >       QMP would be unavailable.
> > 
> >     Solution: we could have a CPU object property like
> >     "removed-features" that would have the list of features that were
> >     disabled because they are not supported by the host (and would make
> >     "enforce" fail).
> > 
> >       * This would solve the problem above and also be a machine-friendly
> >         way to check for possible "enforce" errors.
> > 
> >       * In other words: instead of "enforce", libvirt could use "check"
> >         instead of "enforce", and before unpausing the VM (or even starting
> >         migration), it should first check if the "removed-features" property is
> >         empty.
> > 
> > Would that work for you?
> 
> Yes, that seems like it could work. In fact, it seems much better than
> using enforce and trying to deal with aborted QEMU.

To make the libvirt logic simpler, we could do this: have a "force" mode
(in addition to check/enforce), that wouldn't filter any CPU feature
based on host capabilities. This way libvirt wouldn't need any
non-trivial logic to combine the "f-*" flags and "removed-features" to
find out the CPU model details.

Then libvirt would need to look only at "f-*" to find out the CPU model
details at probing time (as long as "force" is used at probing time),
and just make sure "removed-features" is empty before starting the VM
(optionally parsing its value or checking the "f-*" property values, to
find out which features are missing exactly).

-- 
Eduardo