[RFC] Vhost-user backends cross-version migration support

[RFC] Vhost-user backends cross-version migration support

[Maxime Coquelin]

Hi,

  Few months ago, Michael reported a problem about migrating VMs relying
on vhost-user between hosts supporting different backend versions:
  - Message-Id: <20161011173526-mutt-send-email-mst@kernel.org>
  - https://lists.gnu.org/archive/html/qemu-devel/2016-10/msg03026.html

  The goal of this thread is to draft a proposal based on the outcomes
of discussions with contributors of the different parties (DPDK/OVS
/libvirt/...).

Problem statement:
==================

  When migrating a VM from one host to another, the interfaces exposed by
QEMU must stay unchanged in order to guarantee a successful migration.
In the case of vhost-user interface, parameters like supported Virtio
feature set, max number of queues, max vring sizes,... must remain
compatible. Indeed, the frontend not being re-initialized, no
renegotiation happens at migration time.

  For example, we have a VM that runs on host A, which has its vhost-user
backend advertising VIRTIO_F_RING_INDIRECT_DESC feature. Since the Guest
also support this feature, it is successfully negotiated, and guest
transmit packets using indirect descriptor tables, that the backend
knows to handle.
At some point, the VM is being migrated to host B, which runs an older
version of the backend not supporting this VIRTIO_F_RING_INDIRECT_DESC
feature. The migration would break, because the Guest still have the
VIRTIO_F_RING_INDIRECT_DESC bit sets, and the virtqueue contains some
decriptors pointing to indirect tables, that backend B doesn't know to
handle.
  This is just an example about Virtio features compatibility, but other
backend implementation details could cause other failures.

  What we need is to be able to query the destination host's backend to
ensure migration is possible. Also, we would need to query this
statically, even before the VM is started, to be sure it could be
migrated elsewhere for any reason.


Solution 1: Libvirt queries DPDK vhost lib: *KO*
================================================

Initial idea was to have the management tool (libvirt) to query DPDK
vhost lib and get key/value pairs and check whether migration is
possible. This solution doesn't work for several reasons:
  1. Vhost lib API provide a way for the application to disable features
at runtime (understand, not at build time). So coming back to previous
example, DPDK v16.11 supports indirect descriptor features, but it could
be disabled by OVS. We had a look at whether this API was mandatory, and
it turns out to be, as TSO feature is supported on DPDK but not in OVS.
So we cannot rely on DPDK only.
  2. Some parameter may be not only DPDK specific, such as the maximum
number of queues for example.


Solution 2: Libvirt queries OVS for vhost backend key/value pairs: *KO*
=======================================================================

Second idea was for OVS to expose its vhost backend implementation
parameters as key/value pairs, for example in the DB or by a dedicated
tool. For example, you could have this kind of information:
  - virtio-features: 0x12045694
  - max-rx-queues: 1024
  - max-rx-ring-size: 512
Doing this, Libvirt has the information to take decision whether
migration is possible or not.
The problem is that Libvirt doesn't know (and want) to interpret these
values (should it be equal/lower/greater/...?), and each time a new
key is introduced in OVS, Libvirt will have to be updated to handle it,
adding an unwanted synchronization constraint between the projects.


Solution 3: Libvirt queries OVS for vhost backend version string: *OK*
======================================================================


  The idea is to have a table of supported versions, associated to
key/value pairs. Libvirt could query the list of supported versions
strings for each hosts, and select the first common one among all hosts.

  Then, libvirt would ask OVS to probe the vhost-user interfaces in the
selected version (compatibility mode). For example host A runs OVS-2.7,
and host B OVS-2.6. Host A's OVS-2.7 has an OVS-2.6 compatibility mode
(e.g. with indirect descriptors disabled), which should be selected at
vhost-user interface probe time.

  Advantage of doing so is that libvirt does not need any update if new
keys are introduced (i.e. it does not need to know how the new keys have
to be handled), all these checks remain in OVS's vhost-user implementation.

  Ideally, we would support per vhost-user interface compatibility mode,
which may have an impact also on DPDK API, as the Virtio feature update
API is global, and not per port.

- Implementation:
-----------------

  Goal here is just to illustrate this proposal, I'm sure you will have
good suggestion to improve it.
  In OVS vhost-user library, we would introduce a new structure, for
example (neither compiled nor tested):

struct vhostuser_compat {
  char *version;
  uint64_t virtio_features;
  uint32_t max_rx_queue_sz;
  uint32_t max_nr_queues;
};

  *version* field is the compatibility version string.
   It could be something like: "upstream.ovs-dpdk.v2.6"
   In case for example Fedora adds some more patches to its
   package that would break migration to upstream version, it could have
   a dedicated compatibility string: "fc26.ovs-dpdk.v2.6".
   In case OVS-v2.7 does not break compatibility with previous OVS-v2.6
   version, then no need to create a new compatibility entry, just keep
   v2.6 one.

  *virtio_features* field is the Virtio features set for a given
   compatibility version. When an OVS tag is to be created, it would be
   associated to a DPDK version. The Virtio features for these version
   would be stored in this field. It would allow to upgrade the DPDK
   package for example from v16.07 to v16.11 without breaking migration.
   In case the distribution wants to benefit from latests Virtio
   features, it would have to create a new entry to ensure migration
   won't be broken.

  *max_rx_queue_sz*
  *max_nr_queues* fields are just here for example, don't think this is
   needed today. I just want to illustrate that we have to anticipate
   other parameters than the Virtio feature set, even if not necessary
   at the moment.

  We create a table with different compatibility versions in OVS
vhost-user lib:

static struct vhostuser_compat vu_compat[] = {
  {
    .version = "upstream.ovs-dpdk.v2.7",
    .virtio_features = 0x12045694,
    .max_rx_queue_sz = 512,
  },
  {
    .version = "upstream.ovs-dpdk.v2.6",
    .virtio_features = 0x10045694,
    .max_rx_queue_sz = 1024,
  },
}

  At some time during installation, or system init, the table would be
parsed, and compatibility version strings would be stored into the OVS
database, or a new tool would be created to list these strings.

  Before launching the VM, libvirt will query the version strings for
each hosts using for example the JSON RPC API of OVS (maybe not the best
solution, looking forward for your comments on this). Libvirt would then
select the first common supported version, and insert this string into
the vhost-user interfaces parameters in the OVS DBs of each host.

  When the vhost-user connection is initiated, OVS would know in which
compatibility mode to init the interface, for example by restricting
the support Virtio features of the interface.

  Do you think this is reasonable? Or maybe you have alternative ideas
that would be best fit to ensure successful migration?

Cheers,
Maxime

Re: [libvirt] [RFC] Vhost-user backends cross-version migration support

[Michal Privoznik]

On 02/01/2017 09:35 AM, Maxime Coquelin wrote:
> Hi,
> 
>  Few months ago, Michael reported a problem about migrating VMs relying
> on vhost-user between hosts supporting different backend versions:
>  - Message-Id: <20161011173526-mutt-send-email-mst@kernel.org>
>  - https://lists.gnu.org/archive/html/qemu-devel/2016-10/msg03026.html
> 
>  The goal of this thread is to draft a proposal based on the outcomes
> of discussions with contributors of the different parties (DPDK/OVS
> /libvirt/...).
> 
> Problem statement:
> ==================
> 
>  When migrating a VM from one host to another, the interfaces exposed by
> QEMU must stay unchanged in order to guarantee a successful migration.
> In the case of vhost-user interface, parameters like supported Virtio
> feature set, max number of queues, max vring sizes,... must remain
> compatible. Indeed, the frontend not being re-initialized, no
> renegotiation happens at migration time.
> 
>  For example, we have a VM that runs on host A, which has its vhost-user
> backend advertising VIRTIO_F_RING_INDIRECT_DESC feature. Since the Guest
> also support this feature, it is successfully negotiated, and guest
> transmit packets using indirect descriptor tables, that the backend
> knows to handle.
> At some point, the VM is being migrated to host B, which runs an older
> version of the backend not supporting this VIRTIO_F_RING_INDIRECT_DESC
> feature. The migration would break, because the Guest still have the
> VIRTIO_F_RING_INDIRECT_DESC bit sets, and the virtqueue contains some
> decriptors pointing to indirect tables, that backend B doesn't know to
> handle.
>  This is just an example about Virtio features compatibility, but other
> backend implementation details could cause other failures.

Exactly. Libvirt can't possibly know which virtio features has guest
negotiated to use. Therefore I don't think this falls into libvirt scope.

> 
>  What we need is to be able to query the destination host's backend to
> ensure migration is possible. Also, we would need to query this
> statically, even before the VM is started, to be sure it could be
> migrated elsewhere for any reason.

Again, if you have more than two hosts, say A-Z, I don't see how libvirt
could know what hosts to asks (where you will migrate your guest), and
what combination of virtio features is okay and which is a deal breaker.

> 
> 
> Solution 1: Libvirt queries DPDK vhost lib: *KO*
> ================================================
> 
> Initial idea was to have the management tool (libvirt) to query DPDK
> vhost lib and get key/value pairs and check whether migration is
> possible. This solution doesn't work for several reasons:
>  1. Vhost lib API provide a way for the application to disable features
> at runtime (understand, not at build time). So coming back to previous
> example, DPDK v16.11 supports indirect descriptor features, but it could
> be disabled by OVS. We had a look at whether this API was mandatory, and
> it turns out to be, as TSO feature is supported on DPDK but not in OVS.
> So we cannot rely on DPDK only.
>  2. Some parameter may be not only DPDK specific, such as the maximum
> number of queues for example.
> 
> 
> Solution 2: Libvirt queries OVS for vhost backend key/value pairs: *KO*
> =======================================================================
> 
> Second idea was for OVS to expose its vhost backend implementation
> parameters as key/value pairs, for example in the DB or by a dedicated
> tool. For example, you could have this kind of information:
>  - virtio-features: 0x12045694
>  - max-rx-queues: 1024
>  - max-rx-ring-size: 512
> Doing this, Libvirt has the information to take decision whether
> migration is possible or not.
> The problem is that Libvirt doesn't know (and want) to interpret these
> values (should it be equal/lower/greater/...?), and each time a new
> key is introduced in OVS, Libvirt will have to be updated to handle it,
> adding an unwanted synchronization constraint between the projects.
> 
> 
> Solution 3: Libvirt queries OVS for vhost backend version string: *OK*
> ======================================================================
> 
> 
>  The idea is to have a table of supported versions, associated to
> key/value pairs. Libvirt could query the list of supported versions
> strings for each hosts, and select the first common one among all hosts.

How does libvirt know what hosts to ask? Libvirt aims on managing a
single host. It has no knowledge of other hosts on the network. That's
task for upper layers like RHEV, OpenStack, etc.

> 
>  Then, libvirt would ask OVS to probe the vhost-user interfaces in the
> selected version (compatibility mode). For example host A runs OVS-2.7,
> and host B OVS-2.6. Host A's OVS-2.7 has an OVS-2.6 compatibility mode
> (e.g. with indirect descriptors disabled), which should be selected at
> vhost-user interface probe time.
> 
>  Advantage of doing so is that libvirt does not need any update if new
> keys are introduced (i.e. it does not need to know how the new keys have
> to be handled), all these checks remain in OVS's vhost-user implementation.

And that's where they should stay. Duplicating code between projects
will inevitably lead to a divergence.

> 
>  Ideally, we would support per vhost-user interface compatibility mode,
> which may have an impact also on DPDK API, as the Virtio feature update
> API is global, and not per port.

In general, I don't think we want any kind of this logic in libvirt. Either:

a) fallback logic should be implemented in qemu (e.g. upon migration it
should detect that the migrated guest uses certain version and thus set
backend to use that version or error out and cancel migration), or

b) libvirt would grew another element/attribute to specify version of
vhost-user backend in use and do nothing more than pass it to qemu. At
the same time, we can provide an API (or extend and existing one, e.g.
virsh domcapabilities) to list all available versions on given host.
Upper layer, which knows what are the possible hosts suitable for
virtualization, can then use this API to ask all the hosts, construct
the matrix, select preferred version and put it into libvirt's domain XML.

But frankly, I don't like b) that much. Lets put the fact this is OVS
aside for a moment. Just pretend this is a generic device in qemu. Would
we do the same magic with it? No! Or lets talk about machine types. You
spawn -M type$((X+1)) guest and then decide to migrate it to a host with
older qemu wich supports just typeX. Well, you get an error. Do we care?
Not at all! It's your responsibility (as user/admin) to upgrade the qemu
so that it supports new machine type. I think the same applies to OVS.

Sorry.

Michal

Re: [libvirt] [RFC] Vhost-user backends cross-version migration support

[Daniel P. Berrange]

On Wed, Feb 01, 2017 at 10:14:54AM +0100, Michal Privoznik wrote:
> On 02/01/2017 09:35 AM, Maxime Coquelin wrote:

> > Solution 3: Libvirt queries OVS for vhost backend version string: *OK*
> > ======================================================================
> > 
> > 
> >  The idea is to have a table of supported versions, associated to
> > key/value pairs. Libvirt could query the list of supported versions
> > strings for each hosts, and select the first common one among all hosts.
> 
> How does libvirt know what hosts to ask? Libvirt aims on managing a
> single host. It has no knowledge of other hosts on the network. That's
> task for upper layers like RHEV, OpenStack, etc.
> 
> > 
> >  Then, libvirt would ask OVS to probe the vhost-user interfaces in the
> > selected version (compatibility mode). For example host A runs OVS-2.7,
> > and host B OVS-2.6. Host A's OVS-2.7 has an OVS-2.6 compatibility mode
> > (e.g. with indirect descriptors disabled), which should be selected at
> > vhost-user interface probe time.
> > 
> >  Advantage of doing so is that libvirt does not need any update if new
> > keys are introduced (i.e. it does not need to know how the new keys have
> > to be handled), all these checks remain in OVS's vhost-user implementation.
> 
> And that's where they should stay. Duplicating code between projects
> will inevitably lead to a divergence.
> 
> > 
> >  Ideally, we would support per vhost-user interface compatibility mode,
> > which may have an impact also on DPDK API, as the Virtio feature update
> > API is global, and not per port.
> 
> In general, I don't think we want any kind of this logic in libvirt. Either:
> 
> a) fallback logic should be implemented in qemu (e.g. upon migration it
> should detect that the migrated guest uses certain version and thus set
> backend to use that version or error out and cancel migration), or
> 
> b) libvirt would grew another element/attribute to specify version of
> vhost-user backend in use and do nothing more than pass it to qemu. At
> the same time, we can provide an API (or extend and existing one, e.g.
> virsh domcapabilities) to list all available versions on given host.
> Upper layer, which knows what are the possible hosts suitable for
> virtualization, can then use this API to ask all the hosts, construct
> the matrix, select preferred version and put it into libvirt's domain XML.
> 
> But frankly, I don't like b) that much. Lets put the fact this is OVS
> aside for a moment. Just pretend this is a generic device in qemu. Would
> we do the same magic with it? No! Or lets talk about machine types. You
> spawn -M type$((X+1)) guest and then decide to migrate it to a host with
> older qemu wich supports just typeX. Well, you get an error. Do we care?
> Not at all! It's your responsibility (as user/admin) to upgrade the qemu
> so that it supports new machine type. I think the same applies to OVS.

With machine types, if the latest machine type is X, libvirt allows
the mgmt app to spawn a guest with mcahine type X-1, so that you can
later migrate the VM to a host with older QEMU.

With the vhost user device, the VM will always be launched with version
Y. There's currently no way to request launching the vhost user device
wtih version Y-1. So even if you set your machine type to X-1 for
compat with older host, vhost user will be stuck at version Y preventing
the migration.

One argument would be to say that the vhost user featureset should be
determined by the machine type version, instead of introducing a new
version. The complexity is that vhost-user is a pretty dumb device
from QEMUs POV - most of the interesting logic & the features that
need to be constrained lie in code outside of QEMU, in whatever
server is connected to the vhost user socket.

So I can see the value in allowing a simple version string to be
associated with the vhost-user NIC.

What I'm unclear about is how we would be able to report capabilities
for a host to enumerate what versions were possible. Libvirt doesn't
interact with any of the 3rd party vhost user servers, so it is probably
out of scope - it'd be upto the higher level mgmt app to talk to DPDK
and figure out what versions it supports. 

That does make me wonder though if libvirt & QEMU need to be involved
at all in any part.

When provisioning a new guest, the mgmt app presumably has to talk to
DPDK to setup the NIC port, so DPDK is ready when QEMU launches and
connects. Surely as part of that NIC port setup, it could directly
tell DPDK which version or featureset to permit on the port ? It is
not obvious why the version string has to be fed in via QEMU.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

Re: [libvirt] [RFC] Vhost-user backends cross-version migration support

[Maxime Coquelin]

On 02/01/2017 10:43 AM, Daniel P. Berrange wrote:
> On Wed, Feb 01, 2017 at 10:14:54AM +0100, Michal Privoznik wrote:
>> On 02/01/2017 09:35 AM, Maxime Coquelin wrote:
>
>>> Solution 3: Libvirt queries OVS for vhost backend version string: *OK*
>>> ======================================================================
>>>
>>>
>>>  The idea is to have a table of supported versions, associated to
>>> key/value pairs. Libvirt could query the list of supported versions
>>> strings for each hosts, and select the first common one among all hosts.
>>
>> How does libvirt know what hosts to ask? Libvirt aims on managing a
>> single host. It has no knowledge of other hosts on the network. That's
>> task for upper layers like RHEV, OpenStack, etc.
>>
>>>
>>>  Then, libvirt would ask OVS to probe the vhost-user interfaces in the
>>> selected version (compatibility mode). For example host A runs OVS-2.7,
>>> and host B OVS-2.6. Host A's OVS-2.7 has an OVS-2.6 compatibility mode
>>> (e.g. with indirect descriptors disabled), which should be selected at
>>> vhost-user interface probe time.
>>>
>>>  Advantage of doing so is that libvirt does not need any update if new
>>> keys are introduced (i.e. it does not need to know how the new keys have
>>> to be handled), all these checks remain in OVS's vhost-user implementation.
>>
>> And that's where they should stay. Duplicating code between projects
>> will inevitably lead to a divergence.
>>
>>>
>>>  Ideally, we would support per vhost-user interface compatibility mode,
>>> which may have an impact also on DPDK API, as the Virtio feature update
>>> API is global, and not per port.
>>
>> In general, I don't think we want any kind of this logic in libvirt. Either:
>>
>> a) fallback logic should be implemented in qemu (e.g. upon migration it
>> should detect that the migrated guest uses certain version and thus set
>> backend to use that version or error out and cancel migration), or
>>
>> b) libvirt would grew another element/attribute to specify version of
>> vhost-user backend in use and do nothing more than pass it to qemu. At
>> the same time, we can provide an API (or extend and existing one, e.g.
>> virsh domcapabilities) to list all available versions on given host.
>> Upper layer, which knows what are the possible hosts suitable for
>> virtualization, can then use this API to ask all the hosts, construct
>> the matrix, select preferred version and put it into libvirt's domain XML.
>>
>> But frankly, I don't like b) that much. Lets put the fact this is OVS
>> aside for a moment. Just pretend this is a generic device in qemu. Would
>> we do the same magic with it? No! Or lets talk about machine types. You
>> spawn -M type$((X+1)) guest and then decide to migrate it to a host with
>> older qemu wich supports just typeX. Well, you get an error. Do we care?
>> Not at all! It's your responsibility (as user/admin) to upgrade the qemu
>> so that it supports new machine type. I think the same applies to OVS.
>
> With machine types, if the latest machine type is X, libvirt allows
> the mgmt app to spawn a guest with mcahine type X-1, so that you can
> later migrate the VM to a host with older QEMU.
>
> With the vhost user device, the VM will always be launched with version
> Y. There's currently no way to request launching the vhost user device
> wtih version Y-1. So even if you set your machine type to X-1 for
> compat with older host, vhost user will be stuck at version Y preventing
> the migration.
>
> One argument would be to say that the vhost user featureset should be
> determined by the machine type version, instead of introducing a new
> version. The complexity is that vhost-user is a pretty dumb device
> from QEMUs POV - most of the interesting logic & the features that
> need to be constrained lie in code outside of QEMU, in whatever
> server is connected to the vhost user socket.
>
> So I can see the value in allowing a simple version string to be
> associated with the vhost-user NIC.
>
> What I'm unclear about is how we would be able to report capabilities
> for a host to enumerate what versions were possible. Libvirt doesn't
> interact with any of the 3rd party vhost user servers, so it is probably
> out of scope - it'd be upto the higher level mgmt app to talk to DPDK
> and figure out what versions it supports.
>
> That does make me wonder though if libvirt & QEMU need to be involved
> at all in any part.

Indeed, if the higher level management tool decides for the VM's machine
type, it is where it should also be done for the vhost-user backend. I
now understand this does not make much sense to have libvirt being
involved in this, all (querying/selecting/setting compat mode) should be
manageable in the upper layer.

I'm not familiar with these layers, so your inputs are really helpful.

>
> When provisioning a new guest, the mgmt app presumably has to talk to
> DPDK to setup the NIC port, so DPDK is ready when QEMU launches and
> connects. Surely as part of that NIC port setup, it could directly
> tell DPDK which version or featureset to permit on the port ? It is
> not obvious why the version string has to be fed in via QEMU.
No it is not, my proposal was that libvirt set the version string in
OVS, QEMU was not involved.

 From these inputs, the idea remains valid, except that libvirt is not
the right place to manage this. Instead, RHEV, Openstack or any other
management tool should handle the compat mode selection.

Do you agree?

Thanks,
Maxime

>
> Regards,
> Daniel
>

Re: [libvirt] [RFC] Vhost-user backends cross-version migration support

[Daniel P. Berrange]

On Wed, Feb 01, 2017 at 12:33:22PM +0100, Maxime Coquelin wrote:
> 
> 
> On 02/01/2017 10:43 AM, Daniel P. Berrange wrote:
> > On Wed, Feb 01, 2017 at 10:14:54AM +0100, Michal Privoznik wrote:
> > > On 02/01/2017 09:35 AM, Maxime Coquelin wrote:
> > 
> > > > Solution 3: Libvirt queries OVS for vhost backend version string: *OK*
> > > > ======================================================================
> > > > 
> > > > 
> > > >  The idea is to have a table of supported versions, associated to
> > > > key/value pairs. Libvirt could query the list of supported versions
> > > > strings for each hosts, and select the first common one among all hosts.
> > > 
> > > How does libvirt know what hosts to ask? Libvirt aims on managing a
> > > single host. It has no knowledge of other hosts on the network. That's
> > > task for upper layers like RHEV, OpenStack, etc.
> > > 
> > > > 
> > > >  Then, libvirt would ask OVS to probe the vhost-user interfaces in the
> > > > selected version (compatibility mode). For example host A runs OVS-2.7,
> > > > and host B OVS-2.6. Host A's OVS-2.7 has an OVS-2.6 compatibility mode
> > > > (e.g. with indirect descriptors disabled), which should be selected at
> > > > vhost-user interface probe time.
> > > > 
> > > >  Advantage of doing so is that libvirt does not need any update if new
> > > > keys are introduced (i.e. it does not need to know how the new keys have
> > > > to be handled), all these checks remain in OVS's vhost-user implementation.
> > > 
> > > And that's where they should stay. Duplicating code between projects
> > > will inevitably lead to a divergence.
> > > 
> > > > 
> > > >  Ideally, we would support per vhost-user interface compatibility mode,
> > > > which may have an impact also on DPDK API, as the Virtio feature update
> > > > API is global, and not per port.
> > > 
> > > In general, I don't think we want any kind of this logic in libvirt. Either:
> > > 
> > > a) fallback logic should be implemented in qemu (e.g. upon migration it
> > > should detect that the migrated guest uses certain version and thus set
> > > backend to use that version or error out and cancel migration), or
> > > 
> > > b) libvirt would grew another element/attribute to specify version of
> > > vhost-user backend in use and do nothing more than pass it to qemu. At
> > > the same time, we can provide an API (or extend and existing one, e.g.
> > > virsh domcapabilities) to list all available versions on given host.
> > > Upper layer, which knows what are the possible hosts suitable for
> > > virtualization, can then use this API to ask all the hosts, construct
> > > the matrix, select preferred version and put it into libvirt's domain XML.
> > > 
> > > But frankly, I don't like b) that much. Lets put the fact this is OVS
> > > aside for a moment. Just pretend this is a generic device in qemu. Would
> > > we do the same magic with it? No! Or lets talk about machine types. You
> > > spawn -M type$((X+1)) guest and then decide to migrate it to a host with
> > > older qemu wich supports just typeX. Well, you get an error. Do we care?
> > > Not at all! It's your responsibility (as user/admin) to upgrade the qemu
> > > so that it supports new machine type. I think the same applies to OVS.
> > 
> > With machine types, if the latest machine type is X, libvirt allows
> > the mgmt app to spawn a guest with mcahine type X-1, so that you can
> > later migrate the VM to a host with older QEMU.
> > 
> > With the vhost user device, the VM will always be launched with version
> > Y. There's currently no way to request launching the vhost user device
> > wtih version Y-1. So even if you set your machine type to X-1 for
> > compat with older host, vhost user will be stuck at version Y preventing
> > the migration.
> > 
> > One argument would be to say that the vhost user featureset should be
> > determined by the machine type version, instead of introducing a new
> > version. The complexity is that vhost-user is a pretty dumb device
> > from QEMUs POV - most of the interesting logic & the features that
> > need to be constrained lie in code outside of QEMU, in whatever
> > server is connected to the vhost user socket.
> > 
> > So I can see the value in allowing a simple version string to be
> > associated with the vhost-user NIC.
> > 
> > What I'm unclear about is how we would be able to report capabilities
> > for a host to enumerate what versions were possible. Libvirt doesn't
> > interact with any of the 3rd party vhost user servers, so it is probably
> > out of scope - it'd be upto the higher level mgmt app to talk to DPDK
> > and figure out what versions it supports.
> > 
> > That does make me wonder though if libvirt & QEMU need to be involved
> > at all in any part.
> 
> Indeed, if the higher level management tool decides for the VM's machine
> type, it is where it should also be done for the vhost-user backend. I
> now understand this does not make much sense to have libvirt being
> involved in this, all (querying/selecting/setting compat mode) should be
> manageable in the upper layer.
> 
> I'm not familiar with these layers, so your inputs are really helpful.
> 
> > 
> > When provisioning a new guest, the mgmt app presumably has to talk to
> > DPDK to setup the NIC port, so DPDK is ready when QEMU launches and
> > connects. Surely as part of that NIC port setup, it could directly
> > tell DPDK which version or featureset to permit on the port ? It is
> > not obvious why the version string has to be fed in via QEMU.
> No it is not, my proposal was that libvirt set the version string in
> OVS, QEMU was not involved.
> 
> From these inputs, the idea remains valid, except that libvirt is not
> the right place to manage this. Instead, RHEV, Openstack or any other
> management tool should handle the compat mode selection.

It depends where / how in OVS it needs to be set. The only stuff libvirt
does with OVS is to run 'add-port' and 'del-port' commands via the ovs
cli tool. We pass through arguments from the port profile stored in the
XML config.

  <interface type='bridge'>
    <source bridge='ovsbr'/>
    <virtualport type='openvswitch'>
      <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
    </virtualport>
  </interface>

eg those things in <parameters/> get passed as cli args to the 'add-port'
command. Soo if add-port needs this new version string, then we'd need
to add the version to the openvswitch virtualport XML.

If the version is provided to OVS in a different command, then it would
probably be outside scope of libvirt.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

Re: [libvirt] [RFC] Vhost-user backends cross-version migration support

[Michael S. Tsirkin]

On Wed, Feb 01, 2017 at 11:41:19AM +0000, Daniel P. Berrange wrote:
> On Wed, Feb 01, 2017 at 12:33:22PM +0100, Maxime Coquelin wrote:
> > 
> > 
> > On 02/01/2017 10:43 AM, Daniel P. Berrange wrote:
> > > On Wed, Feb 01, 2017 at 10:14:54AM +0100, Michal Privoznik wrote:
> > > > On 02/01/2017 09:35 AM, Maxime Coquelin wrote:
> > > 
> > > > > Solution 3: Libvirt queries OVS for vhost backend version string: *OK*
> > > > > ======================================================================
> > > > > 
> > > > > 
> > > > >  The idea is to have a table of supported versions, associated to
> > > > > key/value pairs. Libvirt could query the list of supported versions
> > > > > strings for each hosts, and select the first common one among all hosts.
> > > > 
> > > > How does libvirt know what hosts to ask? Libvirt aims on managing a
> > > > single host. It has no knowledge of other hosts on the network. That's
> > > > task for upper layers like RHEV, OpenStack, etc.
> > > > 
> > > > > 
> > > > >  Then, libvirt would ask OVS to probe the vhost-user interfaces in the
> > > > > selected version (compatibility mode). For example host A runs OVS-2.7,
> > > > > and host B OVS-2.6. Host A's OVS-2.7 has an OVS-2.6 compatibility mode
> > > > > (e.g. with indirect descriptors disabled), which should be selected at
> > > > > vhost-user interface probe time.
> > > > > 
> > > > >  Advantage of doing so is that libvirt does not need any update if new
> > > > > keys are introduced (i.e. it does not need to know how the new keys have
> > > > > to be handled), all these checks remain in OVS's vhost-user implementation.
> > > > 
> > > > And that's where they should stay. Duplicating code between projects
> > > > will inevitably lead to a divergence.
> > > > 
> > > > > 
> > > > >  Ideally, we would support per vhost-user interface compatibility mode,
> > > > > which may have an impact also on DPDK API, as the Virtio feature update
> > > > > API is global, and not per port.
> > > > 
> > > > In general, I don't think we want any kind of this logic in libvirt. Either:
> > > > 
> > > > a) fallback logic should be implemented in qemu (e.g. upon migration it
> > > > should detect that the migrated guest uses certain version and thus set
> > > > backend to use that version or error out and cancel migration), or
> > > > 
> > > > b) libvirt would grew another element/attribute to specify version of
> > > > vhost-user backend in use and do nothing more than pass it to qemu. At
> > > > the same time, we can provide an API (or extend and existing one, e.g.
> > > > virsh domcapabilities) to list all available versions on given host.
> > > > Upper layer, which knows what are the possible hosts suitable for
> > > > virtualization, can then use this API to ask all the hosts, construct
> > > > the matrix, select preferred version and put it into libvirt's domain XML.
> > > > 
> > > > But frankly, I don't like b) that much. Lets put the fact this is OVS
> > > > aside for a moment. Just pretend this is a generic device in qemu. Would
> > > > we do the same magic with it? No! Or lets talk about machine types. You
> > > > spawn -M type$((X+1)) guest and then decide to migrate it to a host with
> > > > older qemu wich supports just typeX. Well, you get an error. Do we care?
> > > > Not at all! It's your responsibility (as user/admin) to upgrade the qemu
> > > > so that it supports new machine type. I think the same applies to OVS.
> > > 
> > > With machine types, if the latest machine type is X, libvirt allows
> > > the mgmt app to spawn a guest with mcahine type X-1, so that you can
> > > later migrate the VM to a host with older QEMU.
> > > 
> > > With the vhost user device, the VM will always be launched with version
> > > Y. There's currently no way to request launching the vhost user device
> > > wtih version Y-1. So even if you set your machine type to X-1 for
> > > compat with older host, vhost user will be stuck at version Y preventing
> > > the migration.
> > > 
> > > One argument would be to say that the vhost user featureset should be
> > > determined by the machine type version, instead of introducing a new
> > > version. The complexity is that vhost-user is a pretty dumb device
> > > from QEMUs POV - most of the interesting logic & the features that
> > > need to be constrained lie in code outside of QEMU, in whatever
> > > server is connected to the vhost user socket.
> > > 
> > > So I can see the value in allowing a simple version string to be
> > > associated with the vhost-user NIC.
> > > 
> > > What I'm unclear about is how we would be able to report capabilities
> > > for a host to enumerate what versions were possible. Libvirt doesn't
> > > interact with any of the 3rd party vhost user servers, so it is probably
> > > out of scope - it'd be upto the higher level mgmt app to talk to DPDK
> > > and figure out what versions it supports.
> > > 
> > > That does make me wonder though if libvirt & QEMU need to be involved
> > > at all in any part.
> > 
> > Indeed, if the higher level management tool decides for the VM's machine
> > type, it is where it should also be done for the vhost-user backend. I
> > now understand this does not make much sense to have libvirt being
> > involved in this, all (querying/selecting/setting compat mode) should be
> > manageable in the upper layer.
> > 
> > I'm not familiar with these layers, so your inputs are really helpful.
> > 
> > > 
> > > When provisioning a new guest, the mgmt app presumably has to talk to
> > > DPDK to setup the NIC port, so DPDK is ready when QEMU launches and
> > > connects. Surely as part of that NIC port setup, it could directly
> > > tell DPDK which version or featureset to permit on the port ? It is
> > > not obvious why the version string has to be fed in via QEMU.
> > No it is not, my proposal was that libvirt set the version string in
> > OVS, QEMU was not involved.
> > 
> > From these inputs, the idea remains valid, except that libvirt is not
> > the right place to manage this. Instead, RHEV, Openstack or any other
> > management tool should handle the compat mode selection.
> 
> It depends where / how in OVS it needs to be set. The only stuff libvirt
> does with OVS is to run 'add-port' and 'del-port' commands via the ovs
> cli tool. We pass through arguments from the port profile stored in the
> XML config.
> 
>   <interface type='bridge'>
>     <source bridge='ovsbr'/>
>     <virtualport type='openvswitch'>
>       <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
>     </virtualport>
>   </interface>
> 
> eg those things in <parameters/> get passed as cli args to the 'add-port'
> command. Soo if add-port needs this new version string, then we'd need
> to add the version to the openvswitch virtualport XML.
> 
> If the version is provided to OVS in a different command, then it would
> probably be outside scope of libvirt.
> 
> Regards,
> Daniel

OK but how does it end up there in the 1st place?
Something like we do for qemu, where there's a query
for "pc" alias, and that is stored, seems called for.

Further, it seems ugly to assume add will fail
if it can not support what is required.
Better to query what is supported I think.

> -- 
> |: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
> |: http://libvirt.org              -o-             http://virt-manager.org :|
> |: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

Re: [libvirt] [RFC] Vhost-user backends cross-version migration support

[Maxime Coquelin]

On 02/01/2017 12:41 PM, Daniel P. Berrange wrote:
> On Wed, Feb 01, 2017 at 12:33:22PM +0100, Maxime Coquelin wrote:
>>
>>
>> On 02/01/2017 10:43 AM, Daniel P. Berrange wrote:
>>> On Wed, Feb 01, 2017 at 10:14:54AM +0100, Michal Privoznik wrote:
>>>> On 02/01/2017 09:35 AM, Maxime Coquelin wrote:
>>>
>>>>> Solution 3: Libvirt queries OVS for vhost backend version string: *OK*
>>>>> ======================================================================
>>>>>
>>>>>
>>>>>  The idea is to have a table of supported versions, associated to
>>>>> key/value pairs. Libvirt could query the list of supported versions
>>>>> strings for each hosts, and select the first common one among all hosts.
>>>>
>>>> How does libvirt know what hosts to ask? Libvirt aims on managing a
>>>> single host. It has no knowledge of other hosts on the network. That's
>>>> task for upper layers like RHEV, OpenStack, etc.
>>>>
>>>>>
>>>>>  Then, libvirt would ask OVS to probe the vhost-user interfaces in the
>>>>> selected version (compatibility mode). For example host A runs OVS-2.7,
>>>>> and host B OVS-2.6. Host A's OVS-2.7 has an OVS-2.6 compatibility mode
>>>>> (e.g. with indirect descriptors disabled), which should be selected at
>>>>> vhost-user interface probe time.
>>>>>
>>>>>  Advantage of doing so is that libvirt does not need any update if new
>>>>> keys are introduced (i.e. it does not need to know how the new keys have
>>>>> to be handled), all these checks remain in OVS's vhost-user implementation.
>>>>
>>>> And that's where they should stay. Duplicating code between projects
>>>> will inevitably lead to a divergence.
>>>>
>>>>>
>>>>>  Ideally, we would support per vhost-user interface compatibility mode,
>>>>> which may have an impact also on DPDK API, as the Virtio feature update
>>>>> API is global, and not per port.
>>>>
>>>> In general, I don't think we want any kind of this logic in libvirt. Either:
>>>>
>>>> a) fallback logic should be implemented in qemu (e.g. upon migration it
>>>> should detect that the migrated guest uses certain version and thus set
>>>> backend to use that version or error out and cancel migration), or
>>>>
>>>> b) libvirt would grew another element/attribute to specify version of
>>>> vhost-user backend in use and do nothing more than pass it to qemu. At
>>>> the same time, we can provide an API (or extend and existing one, e.g.
>>>> virsh domcapabilities) to list all available versions on given host.
>>>> Upper layer, which knows what are the possible hosts suitable for
>>>> virtualization, can then use this API to ask all the hosts, construct
>>>> the matrix, select preferred version and put it into libvirt's domain XML.
>>>>
>>>> But frankly, I don't like b) that much. Lets put the fact this is OVS
>>>> aside for a moment. Just pretend this is a generic device in qemu. Would
>>>> we do the same magic with it? No! Or lets talk about machine types. You
>>>> spawn -M type$((X+1)) guest and then decide to migrate it to a host with
>>>> older qemu wich supports just typeX. Well, you get an error. Do we care?
>>>> Not at all! It's your responsibility (as user/admin) to upgrade the qemu
>>>> so that it supports new machine type. I think the same applies to OVS.
>>>
>>> With machine types, if the latest machine type is X, libvirt allows
>>> the mgmt app to spawn a guest with mcahine type X-1, so that you can
>>> later migrate the VM to a host with older QEMU.
>>>
>>> With the vhost user device, the VM will always be launched with version
>>> Y. There's currently no way to request launching the vhost user device
>>> wtih version Y-1. So even if you set your machine type to X-1 for
>>> compat with older host, vhost user will be stuck at version Y preventing
>>> the migration.
>>>
>>> One argument would be to say that the vhost user featureset should be
>>> determined by the machine type version, instead of introducing a new
>>> version. The complexity is that vhost-user is a pretty dumb device
>>> from QEMUs POV - most of the interesting logic & the features that
>>> need to be constrained lie in code outside of QEMU, in whatever
>>> server is connected to the vhost user socket.
>>>
>>> So I can see the value in allowing a simple version string to be
>>> associated with the vhost-user NIC.
>>>
>>> What I'm unclear about is how we would be able to report capabilities
>>> for a host to enumerate what versions were possible. Libvirt doesn't
>>> interact with any of the 3rd party vhost user servers, so it is probably
>>> out of scope - it'd be upto the higher level mgmt app to talk to DPDK
>>> and figure out what versions it supports.
>>>
>>> That does make me wonder though if libvirt & QEMU need to be involved
>>> at all in any part.
>>
>> Indeed, if the higher level management tool decides for the VM's machine
>> type, it is where it should also be done for the vhost-user backend. I
>> now understand this does not make much sense to have libvirt being
>> involved in this, all (querying/selecting/setting compat mode) should be
>> manageable in the upper layer.
>>
>> I'm not familiar with these layers, so your inputs are really helpful.
>>
>>>
>>> When provisioning a new guest, the mgmt app presumably has to talk to
>>> DPDK to setup the NIC port, so DPDK is ready when QEMU launches and
>>> connects. Surely as part of that NIC port setup, it could directly
>>> tell DPDK which version or featureset to permit on the port ? It is
>>> not obvious why the version string has to be fed in via QEMU.
>> No it is not, my proposal was that libvirt set the version string in
>> OVS, QEMU was not involved.
>>
>> From these inputs, the idea remains valid, except that libvirt is not
>> the right place to manage this. Instead, RHEV, Openstack or any other
>> management tool should handle the compat mode selection.
>
> It depends where / how in OVS it needs to be set. The only stuff libvirt
> does with OVS is to run 'add-port' and 'del-port' commands via the ovs
> cli tool. We pass through arguments from the port profile stored in the
> XML config.
>
>   <interface type='bridge'>
>     <source bridge='ovsbr'/>
>     <virtualport type='openvswitch'>
>       <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
>     </virtualport>
>   </interface>
>
> eg those things in <parameters/> get passed as cli args to the 'add-port'
> command. Soo if add-port needs this new version string, then we'd need
> to add the version to the openvswitch virtualport XML.
>
> If the version is provided to OVS in a different command, then it would
> probably be outside scope of libvirt.

I think it would make sense to be a parameter of the add-port command.
But it would be for vhost-user related add-port command, I didn't find 
where/if this is managed in libvirt XML.

Regards,
Maxime

Re: [libvirt] [RFC] Vhost-user backends cross-version migration support

[Daniel P. Berrange]

On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote:
> 
> 
> On 02/01/2017 12:41 PM, Daniel P. Berrange wrote:
> > 
> > It depends where / how in OVS it needs to be set. The only stuff libvirt
> > does with OVS is to run 'add-port' and 'del-port' commands via the ovs
> > cli tool. We pass through arguments from the port profile stored in the
> > XML config.
> > 
> >   <interface type='bridge'>
> >     <source bridge='ovsbr'/>
> >     <virtualport type='openvswitch'>
> >       <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
> >     </virtualport>
> >   </interface>
> > 
> > eg those things in <parameters/> get passed as cli args to the 'add-port'
> > command. Soo if add-port needs this new version string, then we'd need
> > to add the version to the openvswitch virtualport XML.
> > 
> > If the version is provided to OVS in a different command, then it would
> > probably be outside scope of libvirt.
> 
> I think it would make sense to be a parameter of the add-port command.
> But it would be for vhost-user related add-port command, I didn't find
> where/if this is managed in libvirt XML.

For vhost-user, libvirt does not have any interaction with OVS at
all. If the thing that's using the vhost-user UNIX socket, in turn
connects to OVS, that's outside scope of libvirt. IOW, for vhost-user
OVS it seems like that job is for Nova / os-vif to solve.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

Re: [libvirt] [RFC] Vhost-user backends cross-version migration support

[Michael S. Tsirkin]

On Thu, Feb 02, 2017 at 03:06:21PM +0000, Daniel P. Berrange wrote:
> On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote:
> > 
> > 
> > On 02/01/2017 12:41 PM, Daniel P. Berrange wrote:
> > > 
> > > It depends where / how in OVS it needs to be set. The only stuff libvirt
> > > does with OVS is to run 'add-port' and 'del-port' commands via the ovs
> > > cli tool. We pass through arguments from the port profile stored in the
> > > XML config.
> > > 
> > >   <interface type='bridge'>
> > >     <source bridge='ovsbr'/>
> > >     <virtualport type='openvswitch'>
> > >       <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
> > >     </virtualport>
> > >   </interface>
> > > 
> > > eg those things in <parameters/> get passed as cli args to the 'add-port'
> > > command. Soo if add-port needs this new version string, then we'd need
> > > to add the version to the openvswitch virtualport XML.
> > > 
> > > If the version is provided to OVS in a different command, then it would
> > > probably be outside scope of libvirt.
> > 
> > I think it would make sense to be a parameter of the add-port command.
> > But it would be for vhost-user related add-port command, I didn't find
> > where/if this is managed in libvirt XML.
> 
> For vhost-user, libvirt does not have any interaction with OVS at
> all. If the thing that's using the vhost-user UNIX socket, in turn
> connects to OVS, that's outside scope of libvirt. IOW, for vhost-user
> OVS it seems like that job is for Nova / os-vif to solve.
> 
> Regards,
> Daniel

I don't think they currently understand the issues involved in
cross-version migration though.  This is a complex subject and easy to
get wrong.  It would be significantly better to figure it out at libvirt
level since it already deals with cross-version migration issues.


> -- 
> |: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
> |: http://libvirt.org              -o-             http://virt-manager.org :|
> |: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

Re: [libvirt] [RFC] Vhost-user backends cross-version migration support

[Daniel P. Berrange]

On Thu, Feb 02, 2017 at 06:21:55PM +0200, Michael S. Tsirkin wrote:
> On Thu, Feb 02, 2017 at 03:06:21PM +0000, Daniel P. Berrange wrote:
> > On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote:
> > > 
> > > 
> > > On 02/01/2017 12:41 PM, Daniel P. Berrange wrote:
> > > > 
> > > > It depends where / how in OVS it needs to be set. The only stuff libvirt
> > > > does with OVS is to run 'add-port' and 'del-port' commands via the ovs
> > > > cli tool. We pass through arguments from the port profile stored in the
> > > > XML config.
> > > > 
> > > >   <interface type='bridge'>
> > > >     <source bridge='ovsbr'/>
> > > >     <virtualport type='openvswitch'>
> > > >       <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
> > > >     </virtualport>
> > > >   </interface>
> > > > 
> > > > eg those things in <parameters/> get passed as cli args to the 'add-port'
> > > > command. Soo if add-port needs this new version string, then we'd need
> > > > to add the version to the openvswitch virtualport XML.
> > > > 
> > > > If the version is provided to OVS in a different command, then it would
> > > > probably be outside scope of libvirt.
> > > 
> > > I think it would make sense to be a parameter of the add-port command.
> > > But it would be for vhost-user related add-port command, I didn't find
> > > where/if this is managed in libvirt XML.
> > 
> > For vhost-user, libvirt does not have any interaction with OVS at
> > all. If the thing that's using the vhost-user UNIX socket, in turn
> > connects to OVS, that's outside scope of libvirt. IOW, for vhost-user
> > OVS it seems like that job is for Nova / os-vif to solve.
> 
> I don't think they currently understand the issues involved in
> cross-version migration though.  This is a complex subject and easy to
> get wrong.  It would be significantly better to figure it out at libvirt
> level since it already deals with cross-version migration issues.

Libvirt considers vhost-user to be a blackbox though - it just exposes
a UNIX socket, and whatever is on the other end is completely opaque.
The fact that the other end might plumb the data stream into openvswitch
is not something libvirt should know, as we don't want to end up having
to add custom code to libvirt for every different vhost-user server
impl.

IOW, if the version str can be passed to QEMU, and then onto vhost-user
backend in QEMU, then libvirt can be involved. If the version str has
to be given to openvswitch that's not for libvirt to deall with.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

Re: [libvirt] [RFC] Vhost-user backends cross-version migration support

[Michael S. Tsirkin]

On Thu, Feb 02, 2017 at 05:10:28PM +0000, Daniel P. Berrange wrote:
> On Thu, Feb 02, 2017 at 06:21:55PM +0200, Michael S. Tsirkin wrote:
> > On Thu, Feb 02, 2017 at 03:06:21PM +0000, Daniel P. Berrange wrote:
> > > On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote:
> > > > 
> > > > 
> > > > On 02/01/2017 12:41 PM, Daniel P. Berrange wrote:
> > > > > 
> > > > > It depends where / how in OVS it needs to be set. The only stuff libvirt
> > > > > does with OVS is to run 'add-port' and 'del-port' commands via the ovs
> > > > > cli tool. We pass through arguments from the port profile stored in the
> > > > > XML config.
> > > > > 
> > > > >   <interface type='bridge'>
> > > > >     <source bridge='ovsbr'/>
> > > > >     <virtualport type='openvswitch'>
> > > > >       <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
> > > > >     </virtualport>
> > > > >   </interface>
> > > > > 
> > > > > eg those things in <parameters/> get passed as cli args to the 'add-port'
> > > > > command. Soo if add-port needs this new version string, then we'd need
> > > > > to add the version to the openvswitch virtualport XML.
> > > > > 
> > > > > If the version is provided to OVS in a different command, then it would
> > > > > probably be outside scope of libvirt.
> > > > 
> > > > I think it would make sense to be a parameter of the add-port command.
> > > > But it would be for vhost-user related add-port command, I didn't find
> > > > where/if this is managed in libvirt XML.
> > > 
> > > For vhost-user, libvirt does not have any interaction with OVS at
> > > all. If the thing that's using the vhost-user UNIX socket, in turn
> > > connects to OVS, that's outside scope of libvirt. IOW, for vhost-user
> > > OVS it seems like that job is for Nova / os-vif to solve.
> > 
> > I don't think they currently understand the issues involved in
> > cross-version migration though.  This is a complex subject and easy to
> > get wrong.  It would be significantly better to figure it out at libvirt
> > level since it already deals with cross-version migration issues.
> 
> Libvirt considers vhost-user to be a blackbox though - it just exposes
> a UNIX socket, and whatever is on the other end is completely opaque.
> The fact that the other end might plumb the data stream into openvswitch
> is not something libvirt should know, as we don't want to end up having
> to add custom code to libvirt for every different vhost-user server
> impl.
> 
> IOW, if the version str can be passed to QEMU, and then onto vhost-user
> backend in QEMU, then libvirt can be involved. If the version str has
> to be given to openvswitch that's not for libvirt to deall with.
> 
> Regards,
> Daniel

It's not just about passing it to QEMU. The following is needed:
- need to query version when creating VM/device
- need to query supported versions when migrating VM


> -- 
> |: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
> |: http://libvirt.org              -o-             http://virt-manager.org :|
> |: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

Re: [libvirt] [RFC] Vhost-user backends cross-version migration support

[Daniel P. Berrange]

On Thu, Feb 02, 2017 at 07:20:35PM +0200, Michael S. Tsirkin wrote:
> On Thu, Feb 02, 2017 at 05:10:28PM +0000, Daniel P. Berrange wrote:
> > On Thu, Feb 02, 2017 at 06:21:55PM +0200, Michael S. Tsirkin wrote:
> > > On Thu, Feb 02, 2017 at 03:06:21PM +0000, Daniel P. Berrange wrote:
> > > > On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote:
> > > > > 
> > > > > 
> > > > > On 02/01/2017 12:41 PM, Daniel P. Berrange wrote:
> > > > > > 
> > > > > > It depends where / how in OVS it needs to be set. The only stuff libvirt
> > > > > > does with OVS is to run 'add-port' and 'del-port' commands via the ovs
> > > > > > cli tool. We pass through arguments from the port profile stored in the
> > > > > > XML config.
> > > > > > 
> > > > > >   <interface type='bridge'>
> > > > > >     <source bridge='ovsbr'/>
> > > > > >     <virtualport type='openvswitch'>
> > > > > >       <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
> > > > > >     </virtualport>
> > > > > >   </interface>
> > > > > > 
> > > > > > eg those things in <parameters/> get passed as cli args to the 'add-port'
> > > > > > command. Soo if add-port needs this new version string, then we'd need
> > > > > > to add the version to the openvswitch virtualport XML.
> > > > > > 
> > > > > > If the version is provided to OVS in a different command, then it would
> > > > > > probably be outside scope of libvirt.
> > > > > 
> > > > > I think it would make sense to be a parameter of the add-port command.
> > > > > But it would be for vhost-user related add-port command, I didn't find
> > > > > where/if this is managed in libvirt XML.
> > > > 
> > > > For vhost-user, libvirt does not have any interaction with OVS at
> > > > all. If the thing that's using the vhost-user UNIX socket, in turn
> > > > connects to OVS, that's outside scope of libvirt. IOW, for vhost-user
> > > > OVS it seems like that job is for Nova / os-vif to solve.
> > > 
> > > I don't think they currently understand the issues involved in
> > > cross-version migration though.  This is a complex subject and easy to
> > > get wrong.  It would be significantly better to figure it out at libvirt
> > > level since it already deals with cross-version migration issues.
> > 
> > Libvirt considers vhost-user to be a blackbox though - it just exposes
> > a UNIX socket, and whatever is on the other end is completely opaque.
> > The fact that the other end might plumb the data stream into openvswitch
> > is not something libvirt should know, as we don't want to end up having
> > to add custom code to libvirt for every different vhost-user server
> > impl.
> > 
> > IOW, if the version str can be passed to QEMU, and then onto vhost-user
> > backend in QEMU, then libvirt can be involved. If the version str has
> > to be given to openvswitch that's not for libvirt to deall with.
> > 
> > Regards,
> > Daniel
> 
> It's not just about passing it to QEMU. The following is needed:
> - need to query version when creating VM/device
> - need to query supported versions when migrating VM

Those are both things that nova can do, since it knows what the vhost-user
device in question is connected to, and so can query the versions, and check
versions before triggering migration with libvirt

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

Re: [libvirt] [RFC] Vhost-user backends cross-version migration support

[Michael S. Tsirkin]

On Thu, Feb 02, 2017 at 05:29:08PM +0000, Daniel P. Berrange wrote:
> On Thu, Feb 02, 2017 at 07:20:35PM +0200, Michael S. Tsirkin wrote:
> > On Thu, Feb 02, 2017 at 05:10:28PM +0000, Daniel P. Berrange wrote:
> > > On Thu, Feb 02, 2017 at 06:21:55PM +0200, Michael S. Tsirkin wrote:
> > > > On Thu, Feb 02, 2017 at 03:06:21PM +0000, Daniel P. Berrange wrote:
> > > > > On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote:
> > > > > > 
> > > > > > 
> > > > > > On 02/01/2017 12:41 PM, Daniel P. Berrange wrote:
> > > > > > > 
> > > > > > > It depends where / how in OVS it needs to be set. The only stuff libvirt
> > > > > > > does with OVS is to run 'add-port' and 'del-port' commands via the ovs
> > > > > > > cli tool. We pass through arguments from the port profile stored in the
> > > > > > > XML config.
> > > > > > > 
> > > > > > >   <interface type='bridge'>
> > > > > > >     <source bridge='ovsbr'/>
> > > > > > >     <virtualport type='openvswitch'>
> > > > > > >       <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
> > > > > > >     </virtualport>
> > > > > > >   </interface>
> > > > > > > 
> > > > > > > eg those things in <parameters/> get passed as cli args to the 'add-port'
> > > > > > > command. Soo if add-port needs this new version string, then we'd need
> > > > > > > to add the version to the openvswitch virtualport XML.
> > > > > > > 
> > > > > > > If the version is provided to OVS in a different command, then it would
> > > > > > > probably be outside scope of libvirt.
> > > > > > 
> > > > > > I think it would make sense to be a parameter of the add-port command.
> > > > > > But it would be for vhost-user related add-port command, I didn't find
> > > > > > where/if this is managed in libvirt XML.
> > > > > 
> > > > > For vhost-user, libvirt does not have any interaction with OVS at
> > > > > all. If the thing that's using the vhost-user UNIX socket, in turn
> > > > > connects to OVS, that's outside scope of libvirt. IOW, for vhost-user
> > > > > OVS it seems like that job is for Nova / os-vif to solve.
> > > > 
> > > > I don't think they currently understand the issues involved in
> > > > cross-version migration though.  This is a complex subject and easy to
> > > > get wrong.  It would be significantly better to figure it out at libvirt
> > > > level since it already deals with cross-version migration issues.
> > > 
> > > Libvirt considers vhost-user to be a blackbox though - it just exposes
> > > a UNIX socket, and whatever is on the other end is completely opaque.
> > > The fact that the other end might plumb the data stream into openvswitch
> > > is not something libvirt should know, as we don't want to end up having
> > > to add custom code to libvirt for every different vhost-user server
> > > impl.
> > > 
> > > IOW, if the version str can be passed to QEMU, and then onto vhost-user
> > > backend in QEMU, then libvirt can be involved. If the version str has
> > > to be given to openvswitch that's not for libvirt to deall with.
> > > 
> > > Regards,
> > > Daniel
> > 
> > It's not just about passing it to QEMU. The following is needed:
> > - need to query version when creating VM/device
> > - need to query supported versions when migrating VM
> 
> Those are both things that nova can do, since it knows what the vhost-user
> device in question is connected to, and so can query the versions, and check
> versions before triggering migration with libvirt
> 
> Regards,
> Daniel

It can, but then it will need to query libvirt or source for the version
string since it's in the XML.

> -- 
> |: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
> |: http://libvirt.org              -o-             http://virt-manager.org :|
> |: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

Re: [libvirt] [RFC] Vhost-user backends cross-version migration support

[Daniel P. Berrange]

On Thu, Feb 02, 2017 at 07:31:49PM +0200, Michael S. Tsirkin wrote:
> On Thu, Feb 02, 2017 at 05:29:08PM +0000, Daniel P. Berrange wrote:
> > On Thu, Feb 02, 2017 at 07:20:35PM +0200, Michael S. Tsirkin wrote:
> > > On Thu, Feb 02, 2017 at 05:10:28PM +0000, Daniel P. Berrange wrote:
> > > > On Thu, Feb 02, 2017 at 06:21:55PM +0200, Michael S. Tsirkin wrote:
> > > > > On Thu, Feb 02, 2017 at 03:06:21PM +0000, Daniel P. Berrange wrote:
> > > > > > On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote:
> > > > > > > 
> > > > > > > 
> > > > > > > On 02/01/2017 12:41 PM, Daniel P. Berrange wrote:
> > > > > > > > 
> > > > > > > > It depends where / how in OVS it needs to be set. The only stuff libvirt
> > > > > > > > does with OVS is to run 'add-port' and 'del-port' commands via the ovs
> > > > > > > > cli tool. We pass through arguments from the port profile stored in the
> > > > > > > > XML config.
> > > > > > > > 
> > > > > > > >   <interface type='bridge'>
> > > > > > > >     <source bridge='ovsbr'/>
> > > > > > > >     <virtualport type='openvswitch'>
> > > > > > > >       <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
> > > > > > > >     </virtualport>
> > > > > > > >   </interface>
> > > > > > > > 
> > > > > > > > eg those things in <parameters/> get passed as cli args to the 'add-port'
> > > > > > > > command. Soo if add-port needs this new version string, then we'd need
> > > > > > > > to add the version to the openvswitch virtualport XML.
> > > > > > > > 
> > > > > > > > If the version is provided to OVS in a different command, then it would
> > > > > > > > probably be outside scope of libvirt.
> > > > > > > 
> > > > > > > I think it would make sense to be a parameter of the add-port command.
> > > > > > > But it would be for vhost-user related add-port command, I didn't find
> > > > > > > where/if this is managed in libvirt XML.
> > > > > > 
> > > > > > For vhost-user, libvirt does not have any interaction with OVS at
> > > > > > all. If the thing that's using the vhost-user UNIX socket, in turn
> > > > > > connects to OVS, that's outside scope of libvirt. IOW, for vhost-user
> > > > > > OVS it seems like that job is for Nova / os-vif to solve.
> > > > > 
> > > > > I don't think they currently understand the issues involved in
> > > > > cross-version migration though.  This is a complex subject and easy to
> > > > > get wrong.  It would be significantly better to figure it out at libvirt
> > > > > level since it already deals with cross-version migration issues.
> > > > 
> > > > Libvirt considers vhost-user to be a blackbox though - it just exposes
> > > > a UNIX socket, and whatever is on the other end is completely opaque.
> > > > The fact that the other end might plumb the data stream into openvswitch
> > > > is not something libvirt should know, as we don't want to end up having
> > > > to add custom code to libvirt for every different vhost-user server
> > > > impl.
> > > > 
> > > > IOW, if the version str can be passed to QEMU, and then onto vhost-user
> > > > backend in QEMU, then libvirt can be involved. If the version str has
> > > > to be given to openvswitch that's not for libvirt to deall with.
> > > > 
> > > > Regards,
> > > > Daniel
> > > 
> > > It's not just about passing it to QEMU. The following is needed:
> > > - need to query version when creating VM/device
> > > - need to query supported versions when migrating VM
> > 
> > Those are both things that nova can do, since it knows what the vhost-user
> > device in question is connected to, and so can query the versions, and check
> > versions before triggering migration with libvirt
> 
> It can, but then it will need to query libvirt or source for the version
> string since it's in the XML.

No, it wouldn't be in the XML at all. Nova on the source queries what
vhostuser version it has and what the target host has, and can prevent
the migration if they're incompatible. I dont think libvirt has to be
involved at all for this, as all the info can be obtained by nova/os-vif
from the vhostuser impl it has configured.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

Re: [libvirt] [RFC] Vhost-user backends cross-version migration support

[Michael S. Tsirkin]

On Thu, Feb 02, 2017 at 06:21:55PM +0000, Daniel P. Berrange wrote:
> On Thu, Feb 02, 2017 at 07:31:49PM +0200, Michael S. Tsirkin wrote:
> > On Thu, Feb 02, 2017 at 05:29:08PM +0000, Daniel P. Berrange wrote:
> > > On Thu, Feb 02, 2017 at 07:20:35PM +0200, Michael S. Tsirkin wrote:
> > > > On Thu, Feb 02, 2017 at 05:10:28PM +0000, Daniel P. Berrange wrote:
> > > > > On Thu, Feb 02, 2017 at 06:21:55PM +0200, Michael S. Tsirkin wrote:
> > > > > > On Thu, Feb 02, 2017 at 03:06:21PM +0000, Daniel P. Berrange wrote:
> > > > > > > On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote:
> > > > > > > > 
> > > > > > > > 
> > > > > > > > On 02/01/2017 12:41 PM, Daniel P. Berrange wrote:
> > > > > > > > > 
> > > > > > > > > It depends where / how in OVS it needs to be set. The only stuff libvirt
> > > > > > > > > does with OVS is to run 'add-port' and 'del-port' commands via the ovs
> > > > > > > > > cli tool. We pass through arguments from the port profile stored in the
> > > > > > > > > XML config.
> > > > > > > > > 
> > > > > > > > >   <interface type='bridge'>
> > > > > > > > >     <source bridge='ovsbr'/>
> > > > > > > > >     <virtualport type='openvswitch'>
> > > > > > > > >       <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
> > > > > > > > >     </virtualport>
> > > > > > > > >   </interface>
> > > > > > > > > 
> > > > > > > > > eg those things in <parameters/> get passed as cli args to the 'add-port'
> > > > > > > > > command. Soo if add-port needs this new version string, then we'd need
> > > > > > > > > to add the version to the openvswitch virtualport XML.
> > > > > > > > > 
> > > > > > > > > If the version is provided to OVS in a different command, then it would
> > > > > > > > > probably be outside scope of libvirt.
> > > > > > > > 
> > > > > > > > I think it would make sense to be a parameter of the add-port command.
> > > > > > > > But it would be for vhost-user related add-port command, I didn't find
> > > > > > > > where/if this is managed in libvirt XML.
> > > > > > > 
> > > > > > > For vhost-user, libvirt does not have any interaction with OVS at
> > > > > > > all. If the thing that's using the vhost-user UNIX socket, in turn
> > > > > > > connects to OVS, that's outside scope of libvirt. IOW, for vhost-user
> > > > > > > OVS it seems like that job is for Nova / os-vif to solve.
> > > > > > 
> > > > > > I don't think they currently understand the issues involved in
> > > > > > cross-version migration though.  This is a complex subject and easy to
> > > > > > get wrong.  It would be significantly better to figure it out at libvirt
> > > > > > level since it already deals with cross-version migration issues.
> > > > > 
> > > > > Libvirt considers vhost-user to be a blackbox though - it just exposes
> > > > > a UNIX socket, and whatever is on the other end is completely opaque.
> > > > > The fact that the other end might plumb the data stream into openvswitch
> > > > > is not something libvirt should know, as we don't want to end up having
> > > > > to add custom code to libvirt for every different vhost-user server
> > > > > impl.
> > > > > 
> > > > > IOW, if the version str can be passed to QEMU, and then onto vhost-user
> > > > > backend in QEMU, then libvirt can be involved. If the version str has
> > > > > to be given to openvswitch that's not for libvirt to deall with.
> > > > > 
> > > > > Regards,
> > > > > Daniel
> > > > 
> > > > It's not just about passing it to QEMU. The following is needed:
> > > > - need to query version when creating VM/device
> > > > - need to query supported versions when migrating VM
> > > 
> > > Those are both things that nova can do, since it knows what the vhost-user
> > > device in question is connected to, and so can query the versions, and check
> > > versions before triggering migration with libvirt
> > 
> > It can, but then it will need to query libvirt or source for the version
> > string since it's in the XML.
> 
> No, it wouldn't be in the XML at all. Nova on the source queries what
> vhostuser version it has and what the target host has, and can prevent
> the migration if they're incompatible.

This is not sufficient. Exactly the same as qemu machine type,
this must be preserved from time of install and moved
wherever VM goes.


> I dont think libvirt has to be
> involved at all for this, as all the info can be obtained by nova/os-vif
> from the vhostuser impl it has configured.
> 
> Regards,
> Daniel

Given we are already confused at libvirt level, I find it
highly unlikely upper levels will do the right thing.

Generally I simply don't understand why would we
expose to higher levels something which is fundamentally
not a policy decision.


> -- 
> |: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
> |: http://libvirt.org              -o-             http://virt-manager.org :|
> |: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

Re: [libvirt] [RFC] Vhost-user backends cross-version migration support

[Daniel P. Berrange]

On Thu, Feb 02, 2017 at 08:27:18PM +0200, Michael S. Tsirkin wrote:
> On Thu, Feb 02, 2017 at 06:21:55PM +0000, Daniel P. Berrange wrote:
> > On Thu, Feb 02, 2017 at 07:31:49PM +0200, Michael S. Tsirkin wrote:
> > > On Thu, Feb 02, 2017 at 05:29:08PM +0000, Daniel P. Berrange wrote:
> > > > On Thu, Feb 02, 2017 at 07:20:35PM +0200, Michael S. Tsirkin wrote:
> > > > > On Thu, Feb 02, 2017 at 05:10:28PM +0000, Daniel P. Berrange wrote:
> > > > > > On Thu, Feb 02, 2017 at 06:21:55PM +0200, Michael S. Tsirkin wrote:
> > > > > > > On Thu, Feb 02, 2017 at 03:06:21PM +0000, Daniel P. Berrange wrote:
> > > > > > > > On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote:
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > On 02/01/2017 12:41 PM, Daniel P. Berrange wrote:
> > > > > > > > > > 
> > > > > > > > > > It depends where / how in OVS it needs to be set. The only stuff libvirt
> > > > > > > > > > does with OVS is to run 'add-port' and 'del-port' commands via the ovs
> > > > > > > > > > cli tool. We pass through arguments from the port profile stored in the
> > > > > > > > > > XML config.
> > > > > > > > > > 
> > > > > > > > > >   <interface type='bridge'>
> > > > > > > > > >     <source bridge='ovsbr'/>
> > > > > > > > > >     <virtualport type='openvswitch'>
> > > > > > > > > >       <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
> > > > > > > > > >     </virtualport>
> > > > > > > > > >   </interface>
> > > > > > > > > > 
> > > > > > > > > > eg those things in <parameters/> get passed as cli args to the 'add-port'
> > > > > > > > > > command. Soo if add-port needs this new version string, then we'd need
> > > > > > > > > > to add the version to the openvswitch virtualport XML.
> > > > > > > > > > 
> > > > > > > > > > If the version is provided to OVS in a different command, then it would
> > > > > > > > > > probably be outside scope of libvirt.
> > > > > > > > > 
> > > > > > > > > I think it would make sense to be a parameter of the add-port command.
> > > > > > > > > But it would be for vhost-user related add-port command, I didn't find
> > > > > > > > > where/if this is managed in libvirt XML.
> > > > > > > > 
> > > > > > > > For vhost-user, libvirt does not have any interaction with OVS at
> > > > > > > > all. If the thing that's using the vhost-user UNIX socket, in turn
> > > > > > > > connects to OVS, that's outside scope of libvirt. IOW, for vhost-user
> > > > > > > > OVS it seems like that job is for Nova / os-vif to solve.
> > > > > > > 
> > > > > > > I don't think they currently understand the issues involved in
> > > > > > > cross-version migration though.  This is a complex subject and easy to
> > > > > > > get wrong.  It would be significantly better to figure it out at libvirt
> > > > > > > level since it already deals with cross-version migration issues.
> > > > > > 
> > > > > > Libvirt considers vhost-user to be a blackbox though - it just exposes
> > > > > > a UNIX socket, and whatever is on the other end is completely opaque.
> > > > > > The fact that the other end might plumb the data stream into openvswitch
> > > > > > is not something libvirt should know, as we don't want to end up having
> > > > > > to add custom code to libvirt for every different vhost-user server
> > > > > > impl.
> > > > > > 
> > > > > > IOW, if the version str can be passed to QEMU, and then onto vhost-user
> > > > > > backend in QEMU, then libvirt can be involved. If the version str has
> > > > > > to be given to openvswitch that's not for libvirt to deall with.
> > > > > > 
> > > > > > Regards,
> > > > > > Daniel
> > > > > 
> > > > > It's not just about passing it to QEMU. The following is needed:
> > > > > - need to query version when creating VM/device
> > > > > - need to query supported versions when migrating VM
> > > > 
> > > > Those are both things that nova can do, since it knows what the vhost-user
> > > > device in question is connected to, and so can query the versions, and check
> > > > versions before triggering migration with libvirt
> > > 
> > > It can, but then it will need to query libvirt or source for the version
> > > string since it's in the XML.
> > 
> > No, it wouldn't be in the XML at all. Nova on the source queries what
> > vhostuser version it has and what the target host has, and can prevent
> > the migration if they're incompatible.
> 
> This is not sufficient. Exactly the same as qemu machine type,
> this must be preserved from time of install and moved
> wherever VM goes.

Nova has the ability todo that.

> > I dont think libvirt has to be
> > involved at all for this, as all the info can be obtained by nova/os-vif
> > from the vhostuser impl it has configured.
> 
> Given we are already confused at libvirt level, I find it
> highly unlikely upper levels will do the right thing.

The only confusion is about what must consume the data. I mistakenly
thought it was being proposed that the qemu vhostuser backend was
being given a version string. Now it seems the version info must be
set against the 3rd party component that vhostuser connects to, and
that is outside scope of libvirt. That is entirely managed by Nova
today, so Nova is the right thing to manage this versioning info.


Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

Re: [libvirt] [RFC] Vhost-user backends cross-version migration support

[Maxime Coquelin]

On 02/03/2017 10:27 AM, Daniel P. Berrange wrote:
> On Thu, Feb 02, 2017 at 08:27:18PM +0200, Michael S. Tsirkin wrote:
>> On Thu, Feb 02, 2017 at 06:21:55PM +0000, Daniel P. Berrange wrote:
>>> On Thu, Feb 02, 2017 at 07:31:49PM +0200, Michael S. Tsirkin wrote:
>>>> On Thu, Feb 02, 2017 at 05:29:08PM +0000, Daniel P. Berrange wrote:
>>>>> On Thu, Feb 02, 2017 at 07:20:35PM +0200, Michael S. Tsirkin wrote:
>>>>>> On Thu, Feb 02, 2017 at 05:10:28PM +0000, Daniel P. Berrange wrote:
>>>>>>> On Thu, Feb 02, 2017 at 06:21:55PM +0200, Michael S. Tsirkin wrote:
>>>>>>>> On Thu, Feb 02, 2017 at 03:06:21PM +0000, Daniel P. Berrange wrote:
>>>>>>>>> On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 02/01/2017 12:41 PM, Daniel P. Berrange wrote:
>>>>>>>>>>>
>>>>>>>>>>> It depends where / how in OVS it needs to be set. The only stuff libvirt
>>>>>>>>>>> does with OVS is to run 'add-port' and 'del-port' commands via the ovs
>>>>>>>>>>> cli tool. We pass through arguments from the port profile stored in the
>>>>>>>>>>> XML config.
>>>>>>>>>>>
>>>>>>>>>>>   <interface type='bridge'>
>>>>>>>>>>>     <source bridge='ovsbr'/>
>>>>>>>>>>>     <virtualport type='openvswitch'>
>>>>>>>>>>>       <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
>>>>>>>>>>>     </virtualport>
>>>>>>>>>>>   </interface>
>>>>>>>>>>>
>>>>>>>>>>> eg those things in <parameters/> get passed as cli args to the 'add-port'
>>>>>>>>>>> command. Soo if add-port needs this new version string, then we'd need
>>>>>>>>>>> to add the version to the openvswitch virtualport XML.
>>>>>>>>>>>
>>>>>>>>>>> If the version is provided to OVS in a different command, then it would
>>>>>>>>>>> probably be outside scope of libvirt.
>>>>>>>>>>
>>>>>>>>>> I think it would make sense to be a parameter of the add-port command.
>>>>>>>>>> But it would be for vhost-user related add-port command, I didn't find
>>>>>>>>>> where/if this is managed in libvirt XML.
>>>>>>>>>
>>>>>>>>> For vhost-user, libvirt does not have any interaction with OVS at
>>>>>>>>> all. If the thing that's using the vhost-user UNIX socket, in turn
>>>>>>>>> connects to OVS, that's outside scope of libvirt. IOW, for vhost-user
>>>>>>>>> OVS it seems like that job is for Nova / os-vif to solve.
>>>>>>>>
>>>>>>>> I don't think they currently understand the issues involved in
>>>>>>>> cross-version migration though.  This is a complex subject and easy to
>>>>>>>> get wrong.  It would be significantly better to figure it out at libvirt
>>>>>>>> level since it already deals with cross-version migration issues.
>>>>>>>
>>>>>>> Libvirt considers vhost-user to be a blackbox though - it just exposes
>>>>>>> a UNIX socket, and whatever is on the other end is completely opaque.
>>>>>>> The fact that the other end might plumb the data stream into openvswitch
>>>>>>> is not something libvirt should know, as we don't want to end up having
>>>>>>> to add custom code to libvirt for every different vhost-user server
>>>>>>> impl.
>>>>>>>
>>>>>>> IOW, if the version str can be passed to QEMU, and then onto vhost-user
>>>>>>> backend in QEMU, then libvirt can be involved. If the version str has
>>>>>>> to be given to openvswitch that's not for libvirt to deall with.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Daniel
>>>>>>
>>>>>> It's not just about passing it to QEMU. The following is needed:
>>>>>> - need to query version when creating VM/device
>>>>>> - need to query supported versions when migrating VM
>>>>>
>>>>> Those are both things that nova can do, since it knows what the vhost-user
>>>>> device in question is connected to, and so can query the versions, and check
>>>>> versions before triggering migration with libvirt
>>>>
>>>> It can, but then it will need to query libvirt or source for the version
>>>> string since it's in the XML.
>>>
>>> No, it wouldn't be in the XML at all. Nova on the source queries what
>>> vhostuser version it has and what the target host has, and can prevent
>>> the migration if they're incompatible.
>>
>> This is not sufficient. Exactly the same as qemu machine type,
>> this must be preserved from time of install and moved
>> wherever VM goes.
>
> Nova has the ability todo that.
>
>>> I dont think libvirt has to be
>>> involved at all for this, as all the info can be obtained by nova/os-vif
>>> from the vhostuser impl it has configured.
>>
>> Given we are already confused at libvirt level, I find it
>> highly unlikely upper levels will do the right thing.
>
> The only confusion is about what must consume the data. I mistakenly
> thought it was being proposed that the qemu vhostuser backend was
> being given a version string. Now it seems the version info must be
> set against the 3rd party component that vhostuser connects to, and
> that is outside scope of libvirt. That is entirely managed by Nova
> today, so Nova is the right thing to manage this versioning info.

This is my understanding, with as example using Nova:
1. Before starting the VM, Nova queries source host's OVS and all
    possible destination hosts' OVS to retrieve supported versions
2. Nova chooses the first common version supported by all hosts
3. Nova create the dpdkvhostuser interfaces in OVS with passing the
    selected version as parameter

That should be enough, right? Or maybe I miss something?

Thanks,
Maxime

Re: [libvirt] [RFC] Vhost-user backends cross-version migration support

[Daniel P. Berrange]

On Fri, Feb 03, 2017 at 10:41:12AM +0100, Maxime Coquelin wrote:
> 
> 
> On 02/03/2017 10:27 AM, Daniel P. Berrange wrote:
> > On Thu, Feb 02, 2017 at 08:27:18PM +0200, Michael S. Tsirkin wrote:
> > > On Thu, Feb 02, 2017 at 06:21:55PM +0000, Daniel P. Berrange wrote:
> > > > On Thu, Feb 02, 2017 at 07:31:49PM +0200, Michael S. Tsirkin wrote:
> > > > > On Thu, Feb 02, 2017 at 05:29:08PM +0000, Daniel P. Berrange wrote:
> > > > > > On Thu, Feb 02, 2017 at 07:20:35PM +0200, Michael S. Tsirkin wrote:
> > > > > > > On Thu, Feb 02, 2017 at 05:10:28PM +0000, Daniel P. Berrange wrote:
> > > > > > > > On Thu, Feb 02, 2017 at 06:21:55PM +0200, Michael S. Tsirkin wrote:
> > > > > > > > > On Thu, Feb 02, 2017 at 03:06:21PM +0000, Daniel P. Berrange wrote:
> > > > > > > > > > On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote:
> > > > > > > > > > > 
> > > > > > > > > > > 
> > > > > > > > > > > On 02/01/2017 12:41 PM, Daniel P. Berrange wrote:
> > > > > > > > > > > > 
> > > > > > > > > > > > It depends where / how in OVS it needs to be set. The only stuff libvirt
> > > > > > > > > > > > does with OVS is to run 'add-port' and 'del-port' commands via the ovs
> > > > > > > > > > > > cli tool. We pass through arguments from the port profile stored in the
> > > > > > > > > > > > XML config.
> > > > > > > > > > > > 
> > > > > > > > > > > >   <interface type='bridge'>
> > > > > > > > > > > >     <source bridge='ovsbr'/>
> > > > > > > > > > > >     <virtualport type='openvswitch'>
> > > > > > > > > > > >       <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
> > > > > > > > > > > >     </virtualport>
> > > > > > > > > > > >   </interface>
> > > > > > > > > > > > 
> > > > > > > > > > > > eg those things in <parameters/> get passed as cli args to the 'add-port'
> > > > > > > > > > > > command. Soo if add-port needs this new version string, then we'd need
> > > > > > > > > > > > to add the version to the openvswitch virtualport XML.
> > > > > > > > > > > > 
> > > > > > > > > > > > If the version is provided to OVS in a different command, then it would
> > > > > > > > > > > > probably be outside scope of libvirt.
> > > > > > > > > > > 
> > > > > > > > > > > I think it would make sense to be a parameter of the add-port command.
> > > > > > > > > > > But it would be for vhost-user related add-port command, I didn't find
> > > > > > > > > > > where/if this is managed in libvirt XML.
> > > > > > > > > > 
> > > > > > > > > > For vhost-user, libvirt does not have any interaction with OVS at
> > > > > > > > > > all. If the thing that's using the vhost-user UNIX socket, in turn
> > > > > > > > > > connects to OVS, that's outside scope of libvirt. IOW, for vhost-user
> > > > > > > > > > OVS it seems like that job is for Nova / os-vif to solve.
> > > > > > > > > 
> > > > > > > > > I don't think they currently understand the issues involved in
> > > > > > > > > cross-version migration though.  This is a complex subject and easy to
> > > > > > > > > get wrong.  It would be significantly better to figure it out at libvirt
> > > > > > > > > level since it already deals with cross-version migration issues.
> > > > > > > > 
> > > > > > > > Libvirt considers vhost-user to be a blackbox though - it just exposes
> > > > > > > > a UNIX socket, and whatever is on the other end is completely opaque.
> > > > > > > > The fact that the other end might plumb the data stream into openvswitch
> > > > > > > > is not something libvirt should know, as we don't want to end up having
> > > > > > > > to add custom code to libvirt for every different vhost-user server
> > > > > > > > impl.
> > > > > > > > 
> > > > > > > > IOW, if the version str can be passed to QEMU, and then onto vhost-user
> > > > > > > > backend in QEMU, then libvirt can be involved. If the version str has
> > > > > > > > to be given to openvswitch that's not for libvirt to deall with.
> > > > > > > > 
> > > > > > > > Regards,
> > > > > > > > Daniel
> > > > > > > 
> > > > > > > It's not just about passing it to QEMU. The following is needed:
> > > > > > > - need to query version when creating VM/device
> > > > > > > - need to query supported versions when migrating VM
> > > > > > 
> > > > > > Those are both things that nova can do, since it knows what the vhost-user
> > > > > > device in question is connected to, and so can query the versions, and check
> > > > > > versions before triggering migration with libvirt
> > > > > 
> > > > > It can, but then it will need to query libvirt or source for the version
> > > > > string since it's in the XML.
> > > > 
> > > > No, it wouldn't be in the XML at all. Nova on the source queries what
> > > > vhostuser version it has and what the target host has, and can prevent
> > > > the migration if they're incompatible.
> > > 
> > > This is not sufficient. Exactly the same as qemu machine type,
> > > this must be preserved from time of install and moved
> > > wherever VM goes.
> > 
> > Nova has the ability todo that.
> > 
> > > > I dont think libvirt has to be
> > > > involved at all for this, as all the info can be obtained by nova/os-vif
> > > > from the vhostuser impl it has configured.
> > > 
> > > Given we are already confused at libvirt level, I find it
> > > highly unlikely upper levels will do the right thing.
> > 
> > The only confusion is about what must consume the data. I mistakenly
> > thought it was being proposed that the qemu vhostuser backend was
> > being given a version string. Now it seems the version info must be
> > set against the 3rd party component that vhostuser connects to, and
> > that is outside scope of libvirt. That is entirely managed by Nova
> > today, so Nova is the right thing to manage this versioning info.
> 
> This is my understanding, with as example using Nova:
> 1. Before starting the VM, Nova queries source host's OVS and all
>    possible destination hosts' OVS to retrieve supported versions
> 2. Nova chooses the first common version supported by all hosts
> 3. Nova create the dpdkvhostuser interfaces in OVS with passing the
>    selected version as parameter
> 
> That should be enough, right? Or maybe I miss something?

It isn't realistic to check all possible destination hosts when first
starting QEMU, not least becasue they may not yet exist. New hardware
may be deployed *after* QEMU is first started.

When starting the VM, it should just default to using whatever the
latest version is on the host in question.

There can be an optional host configuration parameter that admins can
set to force use of an older version.

This is how Nova deals with machine type - it defaults to latest, but
you can set hw_machine_type in nova.conf to force an older version if
you need migration compatibility between hosts with varying versions.

At time of migration, Nova need to request the version of the target
host. If this version is not compatible with what the VM is currently
using, then Nova should *not* start the migration, it should raise
an error such that the scheduler retries and picks a different target
host

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

Re: [libvirt] [RFC] Vhost-user backends cross-version migration support

[Maxime Coquelin]

On 02/03/2017 11:11 AM, Daniel P. Berrange wrote:
> On Fri, Feb 03, 2017 at 10:41:12AM +0100, Maxime Coquelin wrote:
>>
>>
>> On 02/03/2017 10:27 AM, Daniel P. Berrange wrote:
>>> On Thu, Feb 02, 2017 at 08:27:18PM +0200, Michael S. Tsirkin wrote:
>>>> On Thu, Feb 02, 2017 at 06:21:55PM +0000, Daniel P. Berrange wrote:
>>>>> On Thu, Feb 02, 2017 at 07:31:49PM +0200, Michael S. Tsirkin wrote:
>>>>>> On Thu, Feb 02, 2017 at 05:29:08PM +0000, Daniel P. Berrange wrote:
>>>>>>> On Thu, Feb 02, 2017 at 07:20:35PM +0200, Michael S. Tsirkin wrote:
>>>>>>>> On Thu, Feb 02, 2017 at 05:10:28PM +0000, Daniel P. Berrange wrote:
>>>>>>>>> On Thu, Feb 02, 2017 at 06:21:55PM +0200, Michael S. Tsirkin wrote:
>>>>>>>>>> On Thu, Feb 02, 2017 at 03:06:21PM +0000, Daniel P. Berrange wrote:
>>>>>>>>>>> On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 02/01/2017 12:41 PM, Daniel P. Berrange wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> It depends where / how in OVS it needs to be set. The only stuff libvirt
>>>>>>>>>>>>> does with OVS is to run 'add-port' and 'del-port' commands via the ovs
>>>>>>>>>>>>> cli tool. We pass through arguments from the port profile stored in the
>>>>>>>>>>>>> XML config.
>>>>>>>>>>>>>
>>>>>>>>>>>>>   <interface type='bridge'>
>>>>>>>>>>>>>     <source bridge='ovsbr'/>
>>>>>>>>>>>>>     <virtualport type='openvswitch'>
>>>>>>>>>>>>>       <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
>>>>>>>>>>>>>     </virtualport>
>>>>>>>>>>>>>   </interface>
>>>>>>>>>>>>>
>>>>>>>>>>>>> eg those things in <parameters/> get passed as cli args to the 'add-port'
>>>>>>>>>>>>> command. Soo if add-port needs this new version string, then we'd need
>>>>>>>>>>>>> to add the version to the openvswitch virtualport XML.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If the version is provided to OVS in a different command, then it would
>>>>>>>>>>>>> probably be outside scope of libvirt.
>>>>>>>>>>>>
>>>>>>>>>>>> I think it would make sense to be a parameter of the add-port command.
>>>>>>>>>>>> But it would be for vhost-user related add-port command, I didn't find
>>>>>>>>>>>> where/if this is managed in libvirt XML.
>>>>>>>>>>>
>>>>>>>>>>> For vhost-user, libvirt does not have any interaction with OVS at
>>>>>>>>>>> all. If the thing that's using the vhost-user UNIX socket, in turn
>>>>>>>>>>> connects to OVS, that's outside scope of libvirt. IOW, for vhost-user
>>>>>>>>>>> OVS it seems like that job is for Nova / os-vif to solve.
>>>>>>>>>>
>>>>>>>>>> I don't think they currently understand the issues involved in
>>>>>>>>>> cross-version migration though.  This is a complex subject and easy to
>>>>>>>>>> get wrong.  It would be significantly better to figure it out at libvirt
>>>>>>>>>> level since it already deals with cross-version migration issues.
>>>>>>>>>
>>>>>>>>> Libvirt considers vhost-user to be a blackbox though - it just exposes
>>>>>>>>> a UNIX socket, and whatever is on the other end is completely opaque.
>>>>>>>>> The fact that the other end might plumb the data stream into openvswitch
>>>>>>>>> is not something libvirt should know, as we don't want to end up having
>>>>>>>>> to add custom code to libvirt for every different vhost-user server
>>>>>>>>> impl.
>>>>>>>>>
>>>>>>>>> IOW, if the version str can be passed to QEMU, and then onto vhost-user
>>>>>>>>> backend in QEMU, then libvirt can be involved. If the version str has
>>>>>>>>> to be given to openvswitch that's not for libvirt to deall with.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Daniel
>>>>>>>>
>>>>>>>> It's not just about passing it to QEMU. The following is needed:
>>>>>>>> - need to query version when creating VM/device
>>>>>>>> - need to query supported versions when migrating VM
>>>>>>>
>>>>>>> Those are both things that nova can do, since it knows what the vhost-user
>>>>>>> device in question is connected to, and so can query the versions, and check
>>>>>>> versions before triggering migration with libvirt
>>>>>>
>>>>>> It can, but then it will need to query libvirt or source for the version
>>>>>> string since it's in the XML.
>>>>>
>>>>> No, it wouldn't be in the XML at all. Nova on the source queries what
>>>>> vhostuser version it has and what the target host has, and can prevent
>>>>> the migration if they're incompatible.
>>>>
>>>> This is not sufficient. Exactly the same as qemu machine type,
>>>> this must be preserved from time of install and moved
>>>> wherever VM goes.
>>>
>>> Nova has the ability todo that.
>>>
>>>>> I dont think libvirt has to be
>>>>> involved at all for this, as all the info can be obtained by nova/os-vif
>>>>> from the vhostuser impl it has configured.
>>>>
>>>> Given we are already confused at libvirt level, I find it
>>>> highly unlikely upper levels will do the right thing.
>>>
>>> The only confusion is about what must consume the data. I mistakenly
>>> thought it was being proposed that the qemu vhostuser backend was
>>> being given a version string. Now it seems the version info must be
>>> set against the 3rd party component that vhostuser connects to, and
>>> that is outside scope of libvirt. That is entirely managed by Nova
>>> today, so Nova is the right thing to manage this versioning info.
>>
>> This is my understanding, with as example using Nova:
>> 1. Before starting the VM, Nova queries source host's OVS and all
>>    possible destination hosts' OVS to retrieve supported versions
>> 2. Nova chooses the first common version supported by all hosts
>> 3. Nova create the dpdkvhostuser interfaces in OVS with passing the
>>    selected version as parameter
>>
>> That should be enough, right? Or maybe I miss something?
>
> It isn't realistic to check all possible destination hosts when first
> starting QEMU, not least becasue they may not yet exist. New hardware
> may be deployed *after* QEMU is first started.
>
> When starting the VM, it should just default to using whatever the
> latest version is on the host in question.
>
> There can be an optional host configuration parameter that admins can
> set to force use of an older version.
>
> This is how Nova deals with machine type - it defaults to latest, but
> you can set hw_machine_type in nova.conf to force an older version if
> you need migration compatibility between hosts with varying versions.

Ok, I see.
I thought it could be automated somehow if a list of available host
is already present.

> At time of migration, Nova need to request the version of the target
> host. If this version is not compatible with what the VM is currently
> using, then Nova should *not* start the migration, it should raise
> an error such that the scheduler retries and picks a different target
> host

Thanks for the clarification.

Maxime

Re: [libvirt] [RFC] Vhost-user backends cross-version migration support

[Laine Stump]

On 02/02/2017 10:06 AM, Daniel P. Berrange wrote:
> On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote:
>>
>> On 02/01/2017 12:41 PM, Daniel P. Berrange wrote:
>>> It depends where / how in OVS it needs to be set. The only stuff libvirt
>>> does with OVS is to run 'add-port' and 'del-port' commands via the ovs
>>> cli tool.

(aside note: the code that exec's ovs-vsctl was written back when there 
was no standardized API for performing such operations. libvirt would 
prefer to not exec external programs though, and I've heard that OVS may 
now have an official API of some sort for doing things like this (maybe 
via netlink or dbus or something?) If that's the case, can someone point 
me in the right direction?)

>>>   We pass through arguments from the port profile stored in the
>>> XML config.
>>>
>>>    <interface type='bridge'>
>>>      <source bridge='ovsbr'/>
>>>      <virtualport type='openvswitch'>
>>>        <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
>>>      </virtualport>
>>>    </interface>
>>>
>>> eg those things in <parameters/> get passed as cli args to the 'add-port'
>>> command. Soo if add-port needs this new version string, then we'd need
>>> to add the version to the openvswitch virtualport XML.
>>>
>>> If the version is provided to OVS in a different command, then it would
>>> probably be outside scope of libvirt.
>> I think it would make sense to be a parameter of the add-port command.
>> But it would be for vhost-user related add-port command, I didn't find
>> where/if this is managed in libvirt XML.
> For vhost-user, libvirt does not have any interaction with OVS at
> all. If the thing that's using the vhost-user UNIX socket, in turn
> connects to OVS, that's outside scope of libvirt. IOW, for vhost-user
> OVS it seems like that job is for Nova / os-vif to solve.

This brings up another tangentially related question I came up against 
last night - qemu now has an option to report the host's MTU to the 
guest for virtio and vhost-user interfaces, and Michal Privoznik 
recently pushed patches to set the MTU sent to the guest via an explicit 
<mtu size='n'/> in libvirt's interface config:

    https://bugzilla.redhat.com/1408701

But it would be much nicer if libvirt could learn the MTU of [that stuff 
at the other end of the unix socket] without requiring intervention in 
libvirt's config. For example, I'm just now testing patches for 
tap-based interfaces (connecting to Linux host bridges or OVS switches) 
that query the current MTU of the bridge and report that to qemu; this 
eliminates the burden of configuring each interface of each guest 
individually (and changing that config in all those places if someone 
ever wants to change the MTU of the bridge).

As Dan says, though, libvirt's only interaction in the case of 
vhost-user is with the unix socket. Is there any way to learn what is 
the appropriate MTU from OVS in these cases? Or must Nova (or ovirt or 
some poor user) set that up in the libvirt config for every single 
interface?

Re: [libvirt] [RFC] Vhost-user backends cross-version migration support

[Michael S. Tsirkin]

On Thu, Feb 02, 2017 at 11:47:57AM -0500, Laine Stump wrote:
> On 02/02/2017 10:06 AM, Daniel P. Berrange wrote:
> > On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote:
> > > 
> > > On 02/01/2017 12:41 PM, Daniel P. Berrange wrote:
> > > > It depends where / how in OVS it needs to be set. The only stuff libvirt
> > > > does with OVS is to run 'add-port' and 'del-port' commands via the ovs
> > > > cli tool.
> 
> (aside note: the code that exec's ovs-vsctl was written back when there was
> no standardized API for performing such operations. libvirt would prefer to
> not exec external programs though, and I've heard that OVS may now have an
> official API of some sort for doing things like this (maybe via netlink or
> dbus or something?) If that's the case, can someone point me in the right
> direction?)
> 
> > > >   We pass through arguments from the port profile stored in the
> > > > XML config.
> > > > 
> > > >    <interface type='bridge'>
> > > >      <source bridge='ovsbr'/>
> > > >      <virtualport type='openvswitch'>
> > > >        <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
> > > >      </virtualport>
> > > >    </interface>
> > > > 
> > > > eg those things in <parameters/> get passed as cli args to the 'add-port'
> > > > command. Soo if add-port needs this new version string, then we'd need
> > > > to add the version to the openvswitch virtualport XML.
> > > > 
> > > > If the version is provided to OVS in a different command, then it would
> > > > probably be outside scope of libvirt.
> > > I think it would make sense to be a parameter of the add-port command.
> > > But it would be for vhost-user related add-port command, I didn't find
> > > where/if this is managed in libvirt XML.
> > For vhost-user, libvirt does not have any interaction with OVS at
> > all. If the thing that's using the vhost-user UNIX socket, in turn
> > connects to OVS, that's outside scope of libvirt. IOW, for vhost-user
> > OVS it seems like that job is for Nova / os-vif to solve.
> 
> This brings up another tangentially related question I came up against last
> night - qemu now has an option to report the host's MTU to the guest for
> virtio and vhost-user interfaces, and Michal Privoznik recently pushed
> patches to set the MTU sent to the guest via an explicit <mtu size='n'/> in
> libvirt's interface config:
> 
>    https://bugzilla.redhat.com/1408701
> 
> But it would be much nicer if libvirt could learn the MTU of [that stuff at
> the other end of the unix socket] without requiring intervention in
> libvirt's config. For example, I'm just now testing patches for tap-based
> interfaces (connecting to Linux host bridges or OVS switches) that query the
> current MTU of the bridge and report that to qemu; this eliminates the
> burden of configuring each interface of each guest individually (and
> changing that config in all those places if someone ever wants to change the
> MTU of the bridge).
> 
> As Dan says, though, libvirt's only interaction in the case of vhost-user is
> with the unix socket. Is there any way to learn what is the appropriate MTU
> from OVS in these cases? Or must Nova (or ovirt or some poor user) set that
> up in the libvirt config for every single interface?

We could add commands for all kind of queries to the vhost-user
protocol.  libvirt would have to learn the vhost-user protocol though.
Interested?

-- 
MST

Re: [libvirt] [RFC] Vhost-user backends cross-version migration support

[Daniel P. Berrange]

On Thu, Feb 02, 2017 at 07:09:24PM +0200, Michael S. Tsirkin wrote:
> On Thu, Feb 02, 2017 at 11:47:57AM -0500, Laine Stump wrote:
> > On 02/02/2017 10:06 AM, Daniel P. Berrange wrote:
> > > On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote:
> > > > 
> > > > On 02/01/2017 12:41 PM, Daniel P. Berrange wrote:
> > > > > It depends where / how in OVS it needs to be set. The only stuff libvirt
> > > > > does with OVS is to run 'add-port' and 'del-port' commands via the ovs
> > > > > cli tool.
> > 
> > (aside note: the code that exec's ovs-vsctl was written back when there was
> > no standardized API for performing such operations. libvirt would prefer to
> > not exec external programs though, and I've heard that OVS may now have an
> > official API of some sort for doing things like this (maybe via netlink or
> > dbus or something?) If that's the case, can someone point me in the right
> > direction?)
> > 
> > > > >   We pass through arguments from the port profile stored in the
> > > > > XML config.
> > > > > 
> > > > >    <interface type='bridge'>
> > > > >      <source bridge='ovsbr'/>
> > > > >      <virtualport type='openvswitch'>
> > > > >        <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
> > > > >      </virtualport>
> > > > >    </interface>
> > > > > 
> > > > > eg those things in <parameters/> get passed as cli args to the 'add-port'
> > > > > command. Soo if add-port needs this new version string, then we'd need
> > > > > to add the version to the openvswitch virtualport XML.
> > > > > 
> > > > > If the version is provided to OVS in a different command, then it would
> > > > > probably be outside scope of libvirt.
> > > > I think it would make sense to be a parameter of the add-port command.
> > > > But it would be for vhost-user related add-port command, I didn't find
> > > > where/if this is managed in libvirt XML.
> > > For vhost-user, libvirt does not have any interaction with OVS at
> > > all. If the thing that's using the vhost-user UNIX socket, in turn
> > > connects to OVS, that's outside scope of libvirt. IOW, for vhost-user
> > > OVS it seems like that job is for Nova / os-vif to solve.
> > 
> > This brings up another tangentially related question I came up against last
> > night - qemu now has an option to report the host's MTU to the guest for
> > virtio and vhost-user interfaces, and Michal Privoznik recently pushed
> > patches to set the MTU sent to the guest via an explicit <mtu size='n'/> in
> > libvirt's interface config:
> > 
> >    https://bugzilla.redhat.com/1408701
> > 
> > But it would be much nicer if libvirt could learn the MTU of [that stuff at
> > the other end of the unix socket] without requiring intervention in
> > libvirt's config. For example, I'm just now testing patches for tap-based
> > interfaces (connecting to Linux host bridges or OVS switches) that query the
> > current MTU of the bridge and report that to qemu; this eliminates the
> > burden of configuring each interface of each guest individually (and
> > changing that config in all those places if someone ever wants to change the
> > MTU of the bridge).
> > 
> > As Dan says, though, libvirt's only interaction in the case of vhost-user is
> > with the unix socket. Is there any way to learn what is the appropriate MTU
> > from OVS in these cases? Or must Nova (or ovirt or some poor user) set that
> > up in the libvirt config for every single interface?
> 
> We could add commands for all kind of queries to the vhost-user
> protocol.  libvirt would have to learn the vhost-user protocol though.
> Interested?

No, I really don't think libvirt should implement the vhost-user protocol

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

Re: [libvirt] [RFC] Vhost-user backends cross-version migration support

[Maxime Coquelin]

On 02/02/2017 06:09 PM, Michael S. Tsirkin wrote:
> On Thu, Feb 02, 2017 at 11:47:57AM -0500, Laine Stump wrote:
>> On 02/02/2017 10:06 AM, Daniel P. Berrange wrote:
>>> On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote:
>>>>
>>>> On 02/01/2017 12:41 PM, Daniel P. Berrange wrote:
>>>>> It depends where / how in OVS it needs to be set. The only stuff libvirt
>>>>> does with OVS is to run 'add-port' and 'del-port' commands via the ovs
>>>>> cli tool.
>>
>> (aside note: the code that exec's ovs-vsctl was written back when there was
>> no standardized API for performing such operations. libvirt would prefer to
>> not exec external programs though, and I've heard that OVS may now have an
>> official API of some sort for doing things like this (maybe via netlink or
>> dbus or something?) If that's the case, can someone point me in the right
>> direction?)
>>
>>>>>   We pass through arguments from the port profile stored in the
>>>>> XML config.
>>>>>
>>>>>    <interface type='bridge'>
>>>>>      <source bridge='ovsbr'/>
>>>>>      <virtualport type='openvswitch'>
>>>>>        <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
>>>>>      </virtualport>
>>>>>    </interface>
>>>>>
>>>>> eg those things in <parameters/> get passed as cli args to the 'add-port'
>>>>> command. Soo if add-port needs this new version string, then we'd need
>>>>> to add the version to the openvswitch virtualport XML.
>>>>>
>>>>> If the version is provided to OVS in a different command, then it would
>>>>> probably be outside scope of libvirt.
>>>> I think it would make sense to be a parameter of the add-port command.
>>>> But it would be for vhost-user related add-port command, I didn't find
>>>> where/if this is managed in libvirt XML.
>>> For vhost-user, libvirt does not have any interaction with OVS at
>>> all. If the thing that's using the vhost-user UNIX socket, in turn
>>> connects to OVS, that's outside scope of libvirt. IOW, for vhost-user
>>> OVS it seems like that job is for Nova / os-vif to solve.
>>
>> This brings up another tangentially related question I came up against last
>> night - qemu now has an option to report the host's MTU to the guest for
>> virtio and vhost-user interfaces, and Michal Privoznik recently pushed
>> patches to set the MTU sent to the guest via an explicit <mtu size='n'/> in
>> libvirt's interface config:
>>
>>    https://bugzilla.redhat.com/1408701
>>
>> But it would be much nicer if libvirt could learn the MTU of [that stuff at
>> the other end of the unix socket] without requiring intervention in
>> libvirt's config. For example, I'm just now testing patches for tap-based
>> interfaces (connecting to Linux host bridges or OVS switches) that query the
>> current MTU of the bridge and report that to qemu; this eliminates the
>> burden of configuring each interface of each guest individually (and
>> changing that config in all those places if someone ever wants to change the
>> MTU of the bridge).
>>
>> As Dan says, though, libvirt's only interaction in the case of vhost-user is
>> with the unix socket. Is there any way to learn what is the appropriate MTU
>> from OVS in these cases? Or must Nova (or ovirt or some poor user) set that
>> up in the libvirt config for every single interface?
>
> We could add commands for all kind of queries to the vhost-user
> protocol.  libvirt would have to learn the vhost-user protocol though.
> Interested?
>

I think it could be possible to query the MTU value from the OVS DB
using its JSON RPC-like API, but this is something I haven't tried.
I guess it would need to resolve the ovs interface from the vhost-user
socket path.

Can people familiar with OVS confirm this is something possible?

Regards,
Maxime

Re: [libvirt] [RFC] Vhost-user backends cross-version migration support

[Michal Privoznik]

On 02/02/2017 06:16 PM, Maxime Coquelin wrote:
> 
> 
> On 02/02/2017 06:09 PM, Michael S. Tsirkin wrote:
>> On Thu, Feb 02, 2017 at 11:47:57AM -0500, Laine Stump wrote:
>>> On 02/02/2017 10:06 AM, Daniel P. Berrange wrote:
>>>> On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote:
>>>>>
>>>>> On 02/01/2017 12:41 PM, Daniel P. Berrange wrote:
>>>>>> It depends where / how in OVS it needs to be set. The only stuff
>>>>>> libvirt
>>>>>> does with OVS is to run 'add-port' and 'del-port' commands via the
>>>>>> ovs
>>>>>> cli tool.
>>>
>>> (aside note: the code that exec's ovs-vsctl was written back when
>>> there was
>>> no standardized API for performing such operations. libvirt would
>>> prefer to
>>> not exec external programs though, and I've heard that OVS may now
>>> have an
>>> official API of some sort for doing things like this (maybe via
>>> netlink or
>>> dbus or something?) If that's the case, can someone point me in the
>>> right
>>> direction?)
>>>
>>>>>>   We pass through arguments from the port profile stored in the
>>>>>> XML config.
>>>>>>
>>>>>>    <interface type='bridge'>
>>>>>>      <source bridge='ovsbr'/>
>>>>>>      <virtualport type='openvswitch'>
>>>>>>        <parameters profileid='menial'
>>>>>> interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
>>>>>>      </virtualport>
>>>>>>    </interface>
>>>>>>
>>>>>> eg those things in <parameters/> get passed as cli args to the
>>>>>> 'add-port'
>>>>>> command. Soo if add-port needs this new version string, then we'd
>>>>>> need
>>>>>> to add the version to the openvswitch virtualport XML.
>>>>>>
>>>>>> If the version is provided to OVS in a different command, then it
>>>>>> would
>>>>>> probably be outside scope of libvirt.
>>>>> I think it would make sense to be a parameter of the add-port command.
>>>>> But it would be for vhost-user related add-port command, I didn't find
>>>>> where/if this is managed in libvirt XML.
>>>> For vhost-user, libvirt does not have any interaction with OVS at
>>>> all. If the thing that's using the vhost-user UNIX socket, in turn
>>>> connects to OVS, that's outside scope of libvirt. IOW, for vhost-user
>>>> OVS it seems like that job is for Nova / os-vif to solve.
>>>
>>> This brings up another tangentially related question I came up
>>> against last
>>> night - qemu now has an option to report the host's MTU to the guest for
>>> virtio and vhost-user interfaces, and Michal Privoznik recently pushed
>>> patches to set the MTU sent to the guest via an explicit <mtu
>>> size='n'/> in
>>> libvirt's interface config:
>>>
>>>    https://bugzilla.redhat.com/1408701
>>>
>>> But it would be much nicer if libvirt could learn the MTU of [that
>>> stuff at
>>> the other end of the unix socket] without requiring intervention in
>>> libvirt's config. For example, I'm just now testing patches for
>>> tap-based
>>> interfaces (connecting to Linux host bridges or OVS switches) that
>>> query the
>>> current MTU of the bridge and report that to qemu; this eliminates the
>>> burden of configuring each interface of each guest individually (and
>>> changing that config in all those places if someone ever wants to
>>> change the
>>> MTU of the bridge).
>>>
>>> As Dan says, though, libvirt's only interaction in the case of
>>> vhost-user is
>>> with the unix socket. Is there any way to learn what is the
>>> appropriate MTU
>>> from OVS in these cases? Or must Nova (or ovirt or some poor user)
>>> set that
>>> up in the libvirt config for every single interface?
>>
>> We could add commands for all kind of queries to the vhost-user
>> protocol.  libvirt would have to learn the vhost-user protocol though.
>> Interested?
>>
> 
> I think it could be possible to query the MTU value from the OVS DB
> using its JSON RPC-like API, but this is something I haven't tried.
> I guess it would need to resolve the ovs interface from the vhost-user
> socket path.
> 
> Can people familiar with OVS confirm this is something possible?

Libvirt does not use OVS' JSON RPC yet (it'll be long way to go anyway).
Therefore, for any OVS interaction it uses ovs-vsctl directly. Therefore:

ovs-vsctl get Interface ovsbr0 mtu

is what Laine is looking for. However, something fishy is happening here:

lisa ~ # ovs-vsctl set Interface ovsbr0 mtu=9000
lisa ~ # ovs-vsctl get Interface ovsbr0 mtu
1500
lisa ~ # ifconfig ovsbr0 mtu 9000
lisa ~ # ovs-vsctl get Interface ovsbr0 mtu
9000

(yes, Lisa Simpson)

But since we just want to query the bridge's MTU and not set it, we
should be safe.

Michal

Re: [libvirt] [RFC] Vhost-user backends cross-version migration support

[Laine Stump]

On 02/03/2017 04:12 AM, Michal Privoznik wrote:
> On 02/02/2017 06:16 PM, Maxime Coquelin wrote:
>>
>> On 02/02/2017 06:09 PM, Michael S. Tsirkin wrote:
>>> On Thu, Feb 02, 2017 at 11:47:57AM -0500, Laine Stump wrote:
>>>> On 02/02/2017 10:06 AM, Daniel P. Berrange wrote:
>>>>> On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote:
>>>>>> On 02/01/2017 12:41 PM, Daniel P. Berrange wrote:
>>>>>>> It depends where / how in OVS it needs to be set. The only stuff
>>>>>>> libvirt
>>>>>>> does with OVS is to run 'add-port' and 'del-port' commands via the
>>>>>>> ovs
>>>>>>> cli tool.
>>>> (aside note: the code that exec's ovs-vsctl was written back when
>>>> there was
>>>> no standardized API for performing such operations. libvirt would
>>>> prefer to
>>>> not exec external programs though, and I've heard that OVS may now
>>>> have an
>>>> official API of some sort for doing things like this (maybe via
>>>> netlink or
>>>> dbus or something?) If that's the case, can someone point me in the
>>>> right
>>>> direction?)
>>>>
>>>>>>>    We pass through arguments from the port profile stored in the
>>>>>>> XML config.
>>>>>>>
>>>>>>>     <interface type='bridge'>
>>>>>>>       <source bridge='ovsbr'/>
>>>>>>>       <virtualport type='openvswitch'>
>>>>>>>         <parameters profileid='menial'
>>>>>>> interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
>>>>>>>       </virtualport>
>>>>>>>     </interface>
>>>>>>>
>>>>>>> eg those things in <parameters/> get passed as cli args to the
>>>>>>> 'add-port'
>>>>>>> command. Soo if add-port needs this new version string, then we'd
>>>>>>> need
>>>>>>> to add the version to the openvswitch virtualport XML.
>>>>>>>
>>>>>>> If the version is provided to OVS in a different command, then it
>>>>>>> would
>>>>>>> probably be outside scope of libvirt.
>>>>>> I think it would make sense to be a parameter of the add-port command.
>>>>>> But it would be for vhost-user related add-port command, I didn't find
>>>>>> where/if this is managed in libvirt XML.
>>>>> For vhost-user, libvirt does not have any interaction with OVS at
>>>>> all. If the thing that's using the vhost-user UNIX socket, in turn
>>>>> connects to OVS, that's outside scope of libvirt. IOW, for vhost-user
>>>>> OVS it seems like that job is for Nova / os-vif to solve.
>>>> This brings up another tangentially related question I came up
>>>> against last
>>>> night - qemu now has an option to report the host's MTU to the guest for
>>>> virtio and vhost-user interfaces, and Michal Privoznik recently pushed
>>>> patches to set the MTU sent to the guest via an explicit <mtu
>>>> size='n'/> in
>>>> libvirt's interface config:
>>>>
>>>>     https://bugzilla.redhat.com/1408701
>>>>
>>>> But it would be much nicer if libvirt could learn the MTU of [that
>>>> stuff at
>>>> the other end of the unix socket] without requiring intervention in
>>>> libvirt's config. For example, I'm just now testing patches for
>>>> tap-based
>>>> interfaces (connecting to Linux host bridges or OVS switches) that
>>>> query the
>>>> current MTU of the bridge and report that to qemu; this eliminates the
>>>> burden of configuring each interface of each guest individually (and
>>>> changing that config in all those places if someone ever wants to
>>>> change the
>>>> MTU of the bridge).
>>>>
>>>> As Dan says, though, libvirt's only interaction in the case of
>>>> vhost-user is
>>>> with the unix socket. Is there any way to learn what is the
>>>> appropriate MTU
>>>> from OVS in these cases? Or must Nova (or ovirt or some poor user)
>>>> set that
>>>> up in the libvirt config for every single interface?
>>> We could add commands for all kind of queries to the vhost-user
>>> protocol.  libvirt would have to learn the vhost-user protocol though.
>>> Interested?
>>>
>> I think it could be possible to query the MTU value from the OVS DB
>> using its JSON RPC-like API, but this is something I haven't tried.
>> I guess it would need to resolve the ovs interface from the vhost-user
>> socket path.
>>
>> Can people familiar with OVS confirm this is something possible?
> Libvirt does not use OVS' JSON RPC yet (it'll be long way to go anyway).
> Therefore, for any OVS interaction it uses ovs-vsctl directly. Therefore:
>
> ovs-vsctl get Interface ovsbr0 mtu
>
> is what Laine is looking for. However, something fishy is happening here:
>
> lisa ~ # ovs-vsctl set Interface ovsbr0 mtu=9000
> lisa ~ # ovs-vsctl get Interface ovsbr0 mtu
> 1500
> lisa ~ # ifconfig ovsbr0 mtu 9000
> lisa ~ # ovs-vsctl get Interface ovsbr0 mtu
> 9000
>
> (yes, Lisa Simpson)
>
> But since we just want to query the bridge's MTU and not set it, we
> should be safe.

Except that (as I just noted in the BZ) libvirt doesn't know the name of 
the bridge in the case of vhost-user. It only knows the name of the 
socket. So maybe in this case the only solution is for Nova (which 
*does* know about the bridge) needs to query the MTU and populate 
libvirt's XML appropriately.

Re: [RFC] Vhost-user backends cross-version migration support

[Maxime Coquelin]

Hi,

On 02/01/2017 09:35 AM, Maxime Coquelin wrote:
> Hi,
>
>  Few months ago, Michael reported a problem about migrating VMs relying
> on vhost-user between hosts supporting different backend versions:
>  - Message-Id: <20161011173526-mutt-send-email-mst@kernel.org>
>  - https://lists.gnu.org/archive/html/qemu-devel/2016-10/msg03026.html
>
>  The goal of this thread is to draft a proposal based on the outcomes
> of discussions with contributors of the different parties (DPDK/OVS
> /libvirt/...).

Thanks the first feedback. It seems to converge that this is Nova's
role, but not Libvirt one to manage these versions from management tool
layer.

This change has has no impact from OVS perspective, same requirements
apply. I am interested on OVS contributors feedback on the below design
proposal.

Especially, I would like to have your opinion on the best way for OVS to
expose its supported versions:
- Static file generated at build time from version table described below
- Entries in the OVS DB
- Dedicated tool listing strings from the version table described below

For selecting the right version of the vhost-user backend, do you agree
it should be done via a new parameter of the ovs-vsctl add-port command
for dpdkvhostuser ports?


> Problem statement:
> ==================
>
>  When migrating a VM from one host to another, the interfaces exposed by
> QEMU must stay unchanged in order to guarantee a successful migration.
> In the case of vhost-user interface, parameters like supported Virtio
> feature set, max number of queues, max vring sizes,... must remain
> compatible. Indeed, the frontend not being re-initialized, no
> renegotiation happens at migration time.
>
>  For example, we have a VM that runs on host A, which has its vhost-user
> backend advertising VIRTIO_F_RING_INDIRECT_DESC feature. Since the Guest
> also support this feature, it is successfully negotiated, and guest
> transmit packets using indirect descriptor tables, that the backend
> knows to handle.
> At some point, the VM is being migrated to host B, which runs an older
> version of the backend not supporting this VIRTIO_F_RING_INDIRECT_DESC
> feature. The migration would break, because the Guest still have the
> VIRTIO_F_RING_INDIRECT_DESC bit sets, and the virtqueue contains some
> decriptors pointing to indirect tables, that backend B doesn't know to
> handle.
>  This is just an example about Virtio features compatibility, but other
> backend implementation details could cause other failures.
>
>  What we need is to be able to query the destination host's backend to
> ensure migration is possible. Also, we would need to query this
> statically, even before the VM is started, to be sure it could be
> migrated elsewhere for any reason.

...

>
> Solution 3: Libvirt queries OVS for vhost backend version string: *OK*
> ======================================================================
>
>
>  The idea is to have a table of supported versions, associated to
> key/value pairs. Libvirt could query the list of supported versions
> strings for each hosts, and select the first common one among all hosts.
>
>  Then, libvirt would ask OVS to probe the vhost-user interfaces in the
> selected version (compatibility mode). For example host A runs OVS-2.7,
> and host B OVS-2.6. Host A's OVS-2.7 has an OVS-2.6 compatibility mode
> (e.g. with indirect descriptors disabled), which should be selected at
> vhost-user interface probe time.
>
>  Advantage of doing so is that libvirt does not need any update if new
> keys are introduced (i.e. it does not need to know how the new keys have
> to be handled), all these checks remain in OVS's vhost-user implementation.
>
>  Ideally, we would support per vhost-user interface compatibility mode,
> which may have an impact also on DPDK API, as the Virtio feature update
> API is global, and not per port.
>
> - Implementation:
> -----------------
>
>  Goal here is just to illustrate this proposal, I'm sure you will have
> good suggestion to improve it.
>  In OVS vhost-user library, we would introduce a new structure, for
> example (neither compiled nor tested):
>
> struct vhostuser_compat {
>  char *version;
>  uint64_t virtio_features;
>  uint32_t max_rx_queue_sz;
>  uint32_t max_nr_queues;
> };
>
>  *version* field is the compatibility version string.
>   It could be something like: "upstream.ovs-dpdk.v2.6"
>   In case for example Fedora adds some more patches to its
>   package that would break migration to upstream version, it could have
>   a dedicated compatibility string: "fc26.ovs-dpdk.v2.6".
>   In case OVS-v2.7 does not break compatibility with previous OVS-v2.6
>   version, then no need to create a new compatibility entry, just keep
>   v2.6 one.
>
>  *virtio_features* field is the Virtio features set for a given
>   compatibility version. When an OVS tag is to be created, it would be
>   associated to a DPDK version. The Virtio features for these version
>   would be stored in this field. It would allow to upgrade the DPDK
>   package for example from v16.07 to v16.11 without breaking migration.
>   In case the distribution wants to benefit from latests Virtio
>   features, it would have to create a new entry to ensure migration
>   won't be broken.
>
>  *max_rx_queue_sz*
>  *max_nr_queues* fields are just here for example, don't think this is
>   needed today. I just want to illustrate that we have to anticipate
>   other parameters than the Virtio feature set, even if not necessary
>   at the moment.
>
>  We create a table with different compatibility versions in OVS
> vhost-user lib:
>
> static struct vhostuser_compat vu_compat[] = {
>  {
>    .version = "upstream.ovs-dpdk.v2.7",
>    .virtio_features = 0x12045694,
>    .max_rx_queue_sz = 512,
>  },
>  {
>    .version = "upstream.ovs-dpdk.v2.6",
>    .virtio_features = 0x10045694,
>    .max_rx_queue_sz = 1024,
>  },
> }
>
>  At some time during installation, or system init, the table would be
> parsed, and compatibility version strings would be stored into the OVS
> database, or a new tool would be created to list these strings.
>
>  Before launching the VM, libvirt will query the version strings for
> each hosts using for example the JSON RPC API of OVS (maybe not the best
> solution, looking forward for your comments on this). Libvirt would then
> select the first common supported version, and insert this string into
> the vhost-user interfaces parameters in the OVS DBs of each host.
>
>  When the vhost-user connection is initiated, OVS would know in which
> compatibility mode to init the interface, for example by restricting
> the support Virtio features of the interface.
>
>  Do you think this is reasonable? Or maybe you have alternative ideas
> that would be best fit to ensure successful migration?

Thanks,
Maxime

Re: [RFC] Vhost-user backends cross-version migration support

[Michael S. Tsirkin]

On Fri, Feb 03, 2017 at 03:11:10PM +0100, Maxime Coquelin wrote:
> Hi,
> 
> On 02/01/2017 09:35 AM, Maxime Coquelin wrote:
> > Hi,
> > 
> >  Few months ago, Michael reported a problem about migrating VMs relying
> > on vhost-user between hosts supporting different backend versions:
> >  - Message-Id: <20161011173526-mutt-send-email-mst-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
> >  - https://lists.gnu.org/archive/html/qemu-devel/2016-10/msg03026.html
> > 
> >  The goal of this thread is to draft a proposal based on the outcomes
> > of discussions with contributors of the different parties (DPDK/OVS
> > /libvirt/...).
> 
> Thanks the first feedback. It seems to converge that this is Nova's
> role, but not Libvirt one to manage these versions from management tool
> layer.


I think the conclusion is not that it should go up the stack.  I think
this will just get broken all the time.  No one understands versions and
stuff. Even QEMU developers get confused and break compatibility once in
a while.

My conclusion is that doing it from OVS side is wrong.  Migration is not
an OVS thing, it's a QEMU thing, and libvirt abstracts QEMU.    People
just want migration to work, ok? It's our job to do it, we do not really
need a "make things work" flag.

If libvirt does not want to use the vhost-user protocol (which sounds
reasonable, it's rather complex) how about qemu providing a small
utility to query the port?  We could output json or whatever.

This can help with MTU as well.

And maybe it will help with nowait support - if someone uses the utility
to dump backend config once, QEMU can later start the device without
feature queries.


> This change has has no impact from OVS perspective, same requirements
> apply. I am interested on OVS contributors feedback on the below design
> proposal.
> 
> Especially, I would like to have your opinion on the best way for OVS to
> expose its supported versions:
> - Static file generated at build time from version table described below
> - Entries in the OVS DB
> - Dedicated tool listing strings from the version table described below
> 
> For selecting the right version of the vhost-user backend, do you agree
> it should be done via a new parameter of the ovs-vsctl add-port command
> for dpdkvhostuser ports?
> 
> 
> > Problem statement:
> > ==================
> > 
> >  When migrating a VM from one host to another, the interfaces exposed by
> > QEMU must stay unchanged in order to guarantee a successful migration.
> > In the case of vhost-user interface, parameters like supported Virtio
> > feature set, max number of queues, max vring sizes,... must remain
> > compatible. Indeed, the frontend not being re-initialized, no
> > renegotiation happens at migration time.
> > 
> >  For example, we have a VM that runs on host A, which has its vhost-user
> > backend advertising VIRTIO_F_RING_INDIRECT_DESC feature. Since the Guest
> > also support this feature, it is successfully negotiated, and guest
> > transmit packets using indirect descriptor tables, that the backend
> > knows to handle.
> > At some point, the VM is being migrated to host B, which runs an older
> > version of the backend not supporting this VIRTIO_F_RING_INDIRECT_DESC
> > feature. The migration would break, because the Guest still have the
> > VIRTIO_F_RING_INDIRECT_DESC bit sets, and the virtqueue contains some
> > decriptors pointing to indirect tables, that backend B doesn't know to
> > handle.
> >  This is just an example about Virtio features compatibility, but other
> > backend implementation details could cause other failures.
> > 
> >  What we need is to be able to query the destination host's backend to
> > ensure migration is possible. Also, we would need to query this
> > statically, even before the VM is started, to be sure it could be
> > migrated elsewhere for any reason.
> 
> ...
> 
> > 
> > Solution 3: Libvirt queries OVS for vhost backend version string: *OK*
> > ======================================================================
> > 
> > 
> >  The idea is to have a table of supported versions, associated to
> > key/value pairs. Libvirt could query the list of supported versions
> > strings for each hosts, and select the first common one among all hosts.
> > 
> >  Then, libvirt would ask OVS to probe the vhost-user interfaces in the
> > selected version (compatibility mode). For example host A runs OVS-2.7,
> > and host B OVS-2.6. Host A's OVS-2.7 has an OVS-2.6 compatibility mode
> > (e.g. with indirect descriptors disabled), which should be selected at
> > vhost-user interface probe time.
> > 
> >  Advantage of doing so is that libvirt does not need any update if new
> > keys are introduced (i.e. it does not need to know how the new keys have
> > to be handled), all these checks remain in OVS's vhost-user implementation.
> > 
> >  Ideally, we would support per vhost-user interface compatibility mode,
> > which may have an impact also on DPDK API, as the Virtio feature update
> > API is global, and not per port.
> > 
> > - Implementation:
> > -----------------
> > 
> >  Goal here is just to illustrate this proposal, I'm sure you will have
> > good suggestion to improve it.
> >  In OVS vhost-user library, we would introduce a new structure, for
> > example (neither compiled nor tested):
> > 
> > struct vhostuser_compat {
> >  char *version;
> >  uint64_t virtio_features;
> >  uint32_t max_rx_queue_sz;
> >  uint32_t max_nr_queues;
> > };
> > 
> >  *version* field is the compatibility version string.
> >   It could be something like: "upstream.ovs-dpdk.v2.6"
> >   In case for example Fedora adds some more patches to its
> >   package that would break migration to upstream version, it could have
> >   a dedicated compatibility string: "fc26.ovs-dpdk.v2.6".
> >   In case OVS-v2.7 does not break compatibility with previous OVS-v2.6
> >   version, then no need to create a new compatibility entry, just keep
> >   v2.6 one.
> > 
> >  *virtio_features* field is the Virtio features set for a given
> >   compatibility version. When an OVS tag is to be created, it would be
> >   associated to a DPDK version. The Virtio features for these version
> >   would be stored in this field. It would allow to upgrade the DPDK
> >   package for example from v16.07 to v16.11 without breaking migration.
> >   In case the distribution wants to benefit from latests Virtio
> >   features, it would have to create a new entry to ensure migration
> >   won't be broken.
> > 
> >  *max_rx_queue_sz*
> >  *max_nr_queues* fields are just here for example, don't think this is
> >   needed today. I just want to illustrate that we have to anticipate
> >   other parameters than the Virtio feature set, even if not necessary
> >   at the moment.
> > 
> >  We create a table with different compatibility versions in OVS
> > vhost-user lib:
> > 
> > static struct vhostuser_compat vu_compat[] = {
> >  {
> >    .version = "upstream.ovs-dpdk.v2.7",
> >    .virtio_features = 0x12045694,
> >    .max_rx_queue_sz = 512,
> >  },
> >  {
> >    .version = "upstream.ovs-dpdk.v2.6",
> >    .virtio_features = 0x10045694,
> >    .max_rx_queue_sz = 1024,
> >  },
> > }
> > 
> >  At some time during installation, or system init, the table would be
> > parsed, and compatibility version strings would be stored into the OVS
> > database, or a new tool would be created to list these strings.
> > 
> >  Before launching the VM, libvirt will query the version strings for
> > each hosts using for example the JSON RPC API of OVS (maybe not the best
> > solution, looking forward for your comments on this). Libvirt would then
> > select the first common supported version, and insert this string into
> > the vhost-user interfaces parameters in the OVS DBs of each host.
> > 
> >  When the vhost-user connection is initiated, OVS would know in which
> > compatibility mode to init the interface, for example by restricting
> > the support Virtio features of the interface.
> > 
> >  Do you think this is reasonable? Or maybe you have alternative ideas
> > that would be best fit to ensure successful migration?
> 
> Thanks,
> Maxime

Re: [RFC] Vhost-user backends cross-version migration support

[Daniel P. Berrange]

On Fri, Feb 03, 2017 at 05:34:07PM +0200, Michael S. Tsirkin wrote:
> On Fri, Feb 03, 2017 at 03:11:10PM +0100, Maxime Coquelin wrote:
> > Hi,
> > 
> > On 02/01/2017 09:35 AM, Maxime Coquelin wrote:
> > > Hi,
> > > 
> > >  Few months ago, Michael reported a problem about migrating VMs relying
> > > on vhost-user between hosts supporting different backend versions:
> > >  - Message-Id: <20161011173526-mutt-send-email-mst@kernel.org>
> > >  - https://lists.gnu.org/archive/html/qemu-devel/2016-10/msg03026.html
> > > 
> > >  The goal of this thread is to draft a proposal based on the outcomes
> > > of discussions with contributors of the different parties (DPDK/OVS
> > > /libvirt/...).
> > 
> > Thanks the first feedback. It seems to converge that this is Nova's
> > role, but not Libvirt one to manage these versions from management tool
> > layer.
> 
> 
> I think the conclusion is not that it should go up the stack.  I think
> this will just get broken all the time.  No one understands versions and
> stuff. Even QEMU developers get confused and break compatibility once in
> a while.

I don't think it is really true, nor is it really specific to migration,
it can hit any time you have a network connection that is re-opened even
without migration. Version compatibility negotiations are an inherant part
of any network protocol that is expected to evolve over time.

> My conclusion is that doing it from OVS side is wrong.  Migration is not
> an OVS thing, it's a QEMU thing, and libvirt abstracts QEMU.    People
> just want migration to work, ok? It's our job to do it, we do not really
> need a "make things work" flag.
> 
> If libvirt does not want to use the vhost-user protocol (which sounds
> reasonable, it's rather complex) how about qemu providing a small
> utility to query the port?  We could output json or whatever.
> 
> This can help with MTU as well.
> 
> And maybe it will help with nowait support - if someone uses the utility
> to dump backend config once, QEMU can later start the device without
> feature queries.

I don't think it is QEMU's job to deal with external components in
this way and don't think QEMU vhost-user should be treated as special.
This general scenario arises any time there is a QEMU backend is talking
to an external service/process.

For example, a virtio-serial talking to a daemon in the host. This daemon
can support different versions of the protocol being spoken, so we have
the same compat issue on migration. Or a traditional serial device, which
the guest is using to talk to an external daemon the host. In a few cases
we might know what the protocol is, but in general the data stream  is
opaque to QEMU.

QEMU should not have need to learn about every single protocol that might
be used to communicate with some external service. This is an unbounded
set of possibilities to deal with, some of which will not even be open
source.

This all needs to be delegated to whatever mgmt app is responsible for
setting up these external services, as it has the right domain knowledge
about the applications being used, to make the policy decisions that are
suitable for its usage scenario.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

Re: [RFC] Vhost-user backends cross-version migration support

[Michael S. Tsirkin]

On Fri, Feb 03, 2017 at 03:54:52PM +0000, Daniel P. Berrange wrote:
> On Fri, Feb 03, 2017 at 05:34:07PM +0200, Michael S. Tsirkin wrote:
> > On Fri, Feb 03, 2017 at 03:11:10PM +0100, Maxime Coquelin wrote:
> > > Hi,
> > > 
> > > On 02/01/2017 09:35 AM, Maxime Coquelin wrote:
> > > > Hi,
> > > > 
> > > >  Few months ago, Michael reported a problem about migrating VMs relying
> > > > on vhost-user between hosts supporting different backend versions:
> > > >  - Message-Id: <20161011173526-mutt-send-email-mst@kernel.org>
> > > >  - https://lists.gnu.org/archive/html/qemu-devel/2016-10/msg03026.html
> > > > 
> > > >  The goal of this thread is to draft a proposal based on the outcomes
> > > > of discussions with contributors of the different parties (DPDK/OVS
> > > > /libvirt/...).
> > > 
> > > Thanks the first feedback. It seems to converge that this is Nova's
> > > role, but not Libvirt one to manage these versions from management tool
> > > layer.
> > 
> > 
> > I think the conclusion is not that it should go up the stack.  I think
> > this will just get broken all the time.  No one understands versions and
> > stuff. Even QEMU developers get confused and break compatibility once in
> > a while.
> 
> I don't think it is really true, nor is it really specific to migration,
> it can hit any time you have a network connection that is re-opened even
> without migration. Version compatibility negotiations are an inherant part
> of any network protocol that is expected to evolve over time.
> 
> > My conclusion is that doing it from OVS side is wrong.  Migration is not
> > an OVS thing, it's a QEMU thing, and libvirt abstracts QEMU.    People
> > just want migration to work, ok? It's our job to do it, we do not really
> > need a "make things work" flag.
> > 
> > If libvirt does not want to use the vhost-user protocol (which sounds
> > reasonable, it's rather complex) how about qemu providing a small
> > utility to query the port?  We could output json or whatever.
> > 
> > This can help with MTU as well.
> > 
> > And maybe it will help with nowait support - if someone uses the utility
> > to dump backend config once, QEMU can later start the device without
> > feature queries.
> 
> I don't think it is QEMU's job to deal with external components in
> this way and don't think QEMU vhost-user should be treated as special.

It's very special because it's the only time we ship parts of
device emulation out to an external service.
So they need to be versioned together with QEMU.


> This general scenario arises any time there is a QEMU backend is talking
> to an external service/process.
> 
> For example, a virtio-serial talking to a daemon in the host. This daemon
> can support different versions of the protocol being spoken, so we have
> the same compat issue on migration. Or a traditional serial device, which
> the guest is using to talk to an external daemon the host. In a few cases
> we might know what the protocol is, but in general the data stream  is
> opaque to QEMU.

Are there any examples of people actually managing this across versions
and getting it right?

AFAIK there are cases where data stream is opaque to QEMU like -cpu host
and in every such case VM is unmigrateable.

OTOH vhost-user is not opaque to QEMU at all, and that is intentional
so we can support migration.

> QEMU should not have need to learn about every single protocol that might
> be used to communicate with some external service. This is an unbounded
> set of possibilities to deal with, some of which will not even be open
> source.
> 
> This all needs to be delegated to whatever mgmt app is responsible for
> setting up these external services, as it has the right domain knowledge
> about the applications being used, to make the policy decisions that are
> suitable for its usage scenario.
> 
> Regards,
> Daniel

I disagree it should not and in fact this is exactly what QEMU often does.
This is how in the past we were mostly able to emulate any missing
backend features in QEMU to make things transparent to guest.
E.g. it can actually support the same virtio net device with userspace and
kernel backends and migrate between them.

> -- 
> |: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
> |: http://libvirt.org              -o-             http://virt-manager.org :|
> |: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

Re: [RFC] Vhost-user backends cross-version migration support

[Maxime Coquelin]

On 02/03/2017 04:34 PM, Michael S. Tsirkin wrote:
> On Fri, Feb 03, 2017 at 03:11:10PM +0100, Maxime Coquelin wrote:
>> Hi,
>>
>> On 02/01/2017 09:35 AM, Maxime Coquelin wrote:
>>> Hi,
>>>
>>>  Few months ago, Michael reported a problem about migrating VMs relying
>>> on vhost-user between hosts supporting different backend versions:
>>>  - Message-Id: <20161011173526-mutt-send-email-mst@kernel.org>
>>>  - https://lists.gnu.org/archive/html/qemu-devel/2016-10/msg03026.html
>>>
>>>  The goal of this thread is to draft a proposal based on the outcomes
>>> of discussions with contributors of the different parties (DPDK/OVS
>>> /libvirt/...).
>>
>> Thanks the first feedback. It seems to converge that this is Nova's
>> role, but not Libvirt one to manage these versions from management tool
>> layer.
>
>
> I think the conclusion is not that it should go up the stack.  I think
> this will just get broken all the time.  No one understands versions and
> stuff. Even QEMU developers get confused and break compatibility once in
> a while.

Is it that difficult for OVS to know with which versions of OVS it is
compatible with, and propose a way to be compatible with older versions?

 From Nova perspective, it just has to collect supported versions
strings, and select the most recent common one.

> My conclusion is that doing it from OVS side is wrong.  Migration is not
> an OVS thing, it's a QEMU thing, and libvirt abstracts QEMU.    People
> just want migration to work, ok? It's our job to do it, we do not really
> need a "make things work" flag.
>
> If libvirt does not want to use the vhost-user protocol (which sounds
> reasonable, it's rather complex) how about qemu providing a small
> utility to query the port?  We could output json or whatever.

Doing this, it would be libvirt role to compare key/value pairs in the 
json output, right? If so, it would mean teaching libvirt to compare
things it knows nothing about, that are likely to evolve over time.

Also, it means the port will have already been created, so I'm not
clear how we would manage compatibility version.

>
> This can help with MTU as well.
>
> And maybe it will help with nowait support - if someone uses the utility
> to dump backend config once, QEMU can later start the device without
> feature queries.

Regards,
Maxime