[Qemu-devel] vhost-user graceful connect/disconnect

[Qemu-devel] vhost-user graceful connect/disconnect

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2215 bytes --]

Hi,
Vhost-user implementations assume the slave is already running before
the master starts.  The slave is required during virtio device
initialization (e.g. feature bit negotiation) and so it is simplest to
assume that the master is already available and will respond immediately
to the VHOST_USER_GET_FEATURES message.

I have thought about how to let master and slave start in any order.
Some approaches involve changes to the VIRTIO specification so that
guest drivers can wait until the vhost-user connection is established.

We can avoid spec changes using PCI hotplug:

1. Introduce a new vhost-user object that manages a connection:

   -chardev ...
   -object vhost-user,id=vhost-user0,chardev=chr0

   Note this object is *not* a NetClient.  It's a resource for managing
   a vhost-user connection and can be used with any device type (net,
   scsi, blk, etc).

   This object tries to establish a connection.  When a connection is
   established it sends vhost-user protocol messages to fetch
   information needed for virtio device initialization (like the number
   of virtqueues supported, features bits, etc).  This information is
   stashed so that vhost_*() calls later on do not require synchronous
   communication with the vhost-user slave.

2. When the vhost-user connection is established the object emits a QMP
   event so management software can hotplug the virtio device:

   VHOST_USER_CONNECTED source=vhost-user0

3. The management software hotplugs the virtio device:

   (qmp) netdev_add vhost-user,id=netdev0,vhost-user=vhost-user0
   (qmp) device_add virtio-net-pci,netdev=netdev0

Advantages of this approach:

 * Does not require spec changes.
 * Can be implemented without vhost-user.c qemu_chr_fe_read/write_all()
   calls that hang QEMU until the slave sends data.
 * Allows slave to set the number of queues, feature bits, etc via the
   vhost-user socket.  It is not necessary to manually specify feature
   bitmasks and other slave-specific information on the QEMU
   command-line.

I haven't thought through disconnection but I imagine the vhost-user
object would emit a VHOST_USER_DISCONNECTED event so the management
software can hot unplug the device.

Thoughts?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] vhost-user graceful connect/disconnect

[Fam Zheng]

On Tue, 12/19 16:21, Stefan Hajnoczi wrote:
> Hi,
> Vhost-user implementations assume the slave is already running before
> the master starts.  The slave is required during virtio device
> initialization (e.g. feature bit negotiation) and so it is simplest to
> assume that the master is already available and will respond immediately
> to the VHOST_USER_GET_FEATURES message.
> 
> I have thought about how to let master and slave start in any order.
> Some approaches involve changes to the VIRTIO specification so that
> guest drivers can wait until the vhost-user connection is established.
> 
> We can avoid spec changes using PCI hotplug:
> 
> 1. Introduce a new vhost-user object that manages a connection:
> 
>    -chardev ...
>    -object vhost-user,id=vhost-user0,chardev=chr0
> 
>    Note this object is *not* a NetClient.  It's a resource for managing
>    a vhost-user connection and can be used with any device type (net,
>    scsi, blk, etc).
> 
>    This object tries to establish a connection.  When a connection is
>    established it sends vhost-user protocol messages to fetch
>    information needed for virtio device initialization (like the number
>    of virtqueues supported, features bits, etc).  This information is
>    stashed so that vhost_*() calls later on do not require synchronous
>    communication with the vhost-user slave.
> 
> 2. When the vhost-user connection is established the object emits a QMP
>    event so management software can hotplug the virtio device:
> 
>    VHOST_USER_CONNECTED source=vhost-user0
> 
> 3. The management software hotplugs the virtio device:
> 
>    (qmp) netdev_add vhost-user,id=netdev0,vhost-user=vhost-user0
>    (qmp) device_add virtio-net-pci,netdev=netdev0

Why cannot the management software start the slave then hot plug chardev +
netdev + virtio-net-pci? That way we don't need the vhost-user connected event.

Fam

Re: [Qemu-devel] vhost-user graceful connect/disconnect

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2593 bytes --]

On Wed, Dec 20, 2017 at 02:45:48PM +0800, Fam Zheng wrote:
> On Tue, 12/19 16:21, Stefan Hajnoczi wrote:
> > Hi,
> > Vhost-user implementations assume the slave is already running before
> > the master starts.  The slave is required during virtio device
> > initialization (e.g. feature bit negotiation) and so it is simplest to
> > assume that the master is already available and will respond immediately
> > to the VHOST_USER_GET_FEATURES message.
> > 
> > I have thought about how to let master and slave start in any order.
> > Some approaches involve changes to the VIRTIO specification so that
> > guest drivers can wait until the vhost-user connection is established.
> > 
> > We can avoid spec changes using PCI hotplug:
> > 
> > 1. Introduce a new vhost-user object that manages a connection:
> > 
> >    -chardev ...
> >    -object vhost-user,id=vhost-user0,chardev=chr0
> > 
> >    Note this object is *not* a NetClient.  It's a resource for managing
> >    a vhost-user connection and can be used with any device type (net,
> >    scsi, blk, etc).
> > 
> >    This object tries to establish a connection.  When a connection is
> >    established it sends vhost-user protocol messages to fetch
> >    information needed for virtio device initialization (like the number
> >    of virtqueues supported, features bits, etc).  This information is
> >    stashed so that vhost_*() calls later on do not require synchronous
> >    communication with the vhost-user slave.
> > 
> > 2. When the vhost-user connection is established the object emits a QMP
> >    event so management software can hotplug the virtio device:
> > 
> >    VHOST_USER_CONNECTED source=vhost-user0
> > 
> > 3. The management software hotplugs the virtio device:
> > 
> >    (qmp) netdev_add vhost-user,id=netdev0,vhost-user=vhost-user0
> >    (qmp) device_add virtio-net-pci,netdev=netdev0
> 
> Why cannot the management software start the slave then hot plug chardev +
> netdev + virtio-net-pci? That way we don't need the vhost-user connected event.

There's an advantage to keeping the vhost-user object in QEMU:

In Step 1 the vhost-user object pre-initializes the vhost-user slave to
avoid synchronous vhost_*() calls during hotplug (e.g.
vhost_set_owner(), vhost_get_features()).  When vhost_dev_init() and
other code gets called it won't be necessary to block anymore.  This
makes the vhost-user master robust against slaves that respond slowly or
are malicious.

It's messy to do pre-init outside QEMU and then pass the necessary state
in.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] vhost-user graceful connect/disconnect

[Marc-André Lureau]

Hi

----- Original Message -----
> Hi,
> Vhost-user implementations assume the slave is already running before
> the master starts.  The slave is required during virtio device
> initialization (e.g. feature bit negotiation) and so it is simplest to
> assume that the master is already available and will respond immediately
> to the VHOST_USER_GET_FEATURES message.
> 
> I have thought about how to let master and slave start in any order.
> Some approaches involve changes to the VIRTIO specification so that
> guest drivers can wait until the vhost-user connection is established.
> 
> We can avoid spec changes using PCI hotplug:
> 
> 1. Introduce a new vhost-user object that manages a connection:
> 
>    -chardev ...
>    -object vhost-user,id=vhost-user0,chardev=chr0
> 
>    Note this object is *not* a NetClient.  It's a resource for managing
>    a vhost-user connection and can be used with any device type (net,
>    scsi, blk, etc).
> 
>    This object tries to establish a connection.  When a connection is
>    established it sends vhost-user protocol messages to fetch
>    information needed for virtio device initialization (like the number
>    of virtqueues supported, features bits, etc).  This information is
>    stashed so that vhost_*() calls later on do not require synchronous
>    communication with the vhost-user slave.

This share similarities with vhost-user-backend I proposed here: https://lists.gnu.org/archive/html/qemu-devel/2016-06/msg01014.html

(instead of spawning a slave, talk to a chardev - both could eventually co-exist)

> 
> 2. When the vhost-user connection is established the object emits a QMP
>    event so management software can hotplug the virtio device:
> 
>    VHOST_USER_CONNECTED source=vhost-user0
> 
> 3. The management software hotplugs the virtio device:
> 
>    (qmp) netdev_add vhost-user,id=netdev0,vhost-user=vhost-user0
>    (qmp) device_add virtio-net-pci,netdev=netdev0
> 
> Advantages of this approach:
> 
>  * Does not require spec changes.
>  * Can be implemented without vhost-user.c qemu_chr_fe_read/write_all()
>    calls that hang QEMU until the slave sends data.
>  * Allows slave to set the number of queues, feature bits, etc via the
>    vhost-user socket.  It is not necessary to manually specify feature
>    bitmasks and other slave-specific information on the QEMU
>    command-line.
> 
> I haven't thought through disconnection but I imagine the vhost-user
> object would emit a VHOST_USER_DISCONNECTED event so the management
> software can hot unplug the device.
> 
> Thoughts?

I am sorry, I can't think through all the potential issues easily. But that sounds interesting enough to start a proof-of-concept.

And we need more tests for vhost-user (including fixing the existing tests!)...

Thanks

Re: [Qemu-devel] vhost-user graceful connect/disconnect

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 3243 bytes --]

On Thu, Dec 21, 2017 at 06:01:29AM -0500, Marc-André Lureau wrote:
> Hi
> 
> ----- Original Message -----
> > Hi,
> > Vhost-user implementations assume the slave is already running before
> > the master starts.  The slave is required during virtio device
> > initialization (e.g. feature bit negotiation) and so it is simplest to
> > assume that the master is already available and will respond immediately
> > to the VHOST_USER_GET_FEATURES message.
> > 
> > I have thought about how to let master and slave start in any order.
> > Some approaches involve changes to the VIRTIO specification so that
> > guest drivers can wait until the vhost-user connection is established.
> > 
> > We can avoid spec changes using PCI hotplug:
> > 
> > 1. Introduce a new vhost-user object that manages a connection:
> > 
> >    -chardev ...
> >    -object vhost-user,id=vhost-user0,chardev=chr0
> > 
> >    Note this object is *not* a NetClient.  It's a resource for managing
> >    a vhost-user connection and can be used with any device type (net,
> >    scsi, blk, etc).
> > 
> >    This object tries to establish a connection.  When a connection is
> >    established it sends vhost-user protocol messages to fetch
> >    information needed for virtio device initialization (like the number
> >    of virtqueues supported, features bits, etc).  This information is
> >    stashed so that vhost_*() calls later on do not require synchronous
> >    communication with the vhost-user slave.
> 
> This share similarities with vhost-user-backend I proposed here: https://lists.gnu.org/archive/html/qemu-devel/2016-06/msg01014.html
> 
> (instead of spawning a slave, talk to a chardev - both could eventually co-exist)

Yes, it's similar.  Thanks for the link!

> > 
> > 2. When the vhost-user connection is established the object emits a QMP
> >    event so management software can hotplug the virtio device:
> > 
> >    VHOST_USER_CONNECTED source=vhost-user0
> > 
> > 3. The management software hotplugs the virtio device:
> > 
> >    (qmp) netdev_add vhost-user,id=netdev0,vhost-user=vhost-user0
> >    (qmp) device_add virtio-net-pci,netdev=netdev0
> > 
> > Advantages of this approach:
> > 
> >  * Does not require spec changes.
> >  * Can be implemented without vhost-user.c qemu_chr_fe_read/write_all()
> >    calls that hang QEMU until the slave sends data.
> >  * Allows slave to set the number of queues, feature bits, etc via the
> >    vhost-user socket.  It is not necessary to manually specify feature
> >    bitmasks and other slave-specific information on the QEMU
> >    command-line.
> > 
> > I haven't thought through disconnection but I imagine the vhost-user
> > object would emit a VHOST_USER_DISCONNECTED event so the management
> > software can hot unplug the device.
> > 
> > Thoughts?
> 
> I am sorry, I can't think through all the potential issues easily. But that sounds interesting enough to start a proof-of-concept.
> 
> And we need more tests for vhost-user (including fixing the existing tests!)...

I'm not going to prototype this yet, I'm working on virtio-vhost-user
first, but eventually I might get back to -object vhost-user(-backend).

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] vhost-user graceful connect/disconnect

[Wei Wang]

On 01/04/2018 06:47 PM, Stefan Hajnoczi wrote:
> On Thu, Dec 21, 2017 at 06:01:29AM -0500, Marc-André Lureau wrote:
>
> I'm not going to prototype this yet, I'm working on virtio-vhost-user
> first, but eventually I might get back to -object vhost-user(-backend).
>

Hi Stefan, are you implementing the guest slave and vhost-pci driver 
(we've posted to the dpdk mailinglist) as well? and do you have an 
estimation when would the prototype be ready?

Best,
Wei

Re: [Qemu-devel] vhost-user graceful connect/disconnect

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1291 bytes --]

On Thu, Jan 04, 2018 at 07:15:38PM +0800, Wei Wang wrote:
> On 01/04/2018 06:47 PM, Stefan Hajnoczi wrote:
> > On Thu, Dec 21, 2017 at 06:01:29AM -0500, Marc-André Lureau wrote:
> > 
> > I'm not going to prototype this yet, I'm working on virtio-vhost-user
> > first, but eventually I might get back to -object vhost-user(-backend).
> > 
> 
> Hi Stefan, are you implementing the guest slave and vhost-pci driver (we've
> posted to the dpdk mailinglist) as well? and do you have an estimation when
> would the prototype be ready?

I'm implementing the "[RFC virtio-dev] vhost-user-slave: add vhost-user
slave device type" device in QEMU and DPDK in order to show how the
ideas we've discussed work.

Here is the VIRTIO spec link again:
https://stefanha.github.io/virtio/vhost-user-slave.html#x1-2830007

It integrates into DPDK's librte_vhost so that existing vhost-user code
works over AF_UNIX and virtio-vhost-user without code duplication or
rewriting the devices.

I hope you'll like the code when it's done.  If not, it still has useful
code and ideas that would be needed to complete the vhost-pci RFC work
like extending the PCI transport in the VIRTIO spec, handling vhost-user
reconnection, etc.

I'm aiming to send an RFC in the next 2 weeks.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] vhost-user graceful connect/disconnect

[Wei Wang]

On 01/05/2018 11:49 PM, Stefan Hajnoczi wrote:
> On Thu, Jan 04, 2018 at 07:15:38PM +0800, Wei Wang wrote:
>> On 01/04/2018 06:47 PM, Stefan Hajnoczi wrote:
>>> On Thu, Dec 21, 2017 at 06:01:29AM -0500, Marc-André Lureau wrote:
>>>
>>> I'm not going to prototype this yet, I'm working on virtio-vhost-user
>>> first, but eventually I might get back to -object vhost-user(-backend).
>>>
>> Hi Stefan, are you implementing the guest slave and vhost-pci driver (we've
>> posted to the dpdk mailinglist) as well? and do you have an estimation when
>> would the prototype be ready?
> I'm implementing the "[RFC virtio-dev] vhost-user-slave: add vhost-user
> slave device type" device in QEMU and DPDK in order to show how the
> ideas we've discussed work.
>
> Here is the VIRTIO spec link again:
> https://stefanha.github.io/virtio/vhost-user-slave.html#x1-2830007

There are four virtqueues documented in the spec, would two suffice? 
Request and Response can be distinguished by VHOST_USER_REPLY_MASK.

>
> It integrates into DPDK's librte_vhost so that existing vhost-user code
> works over AF_UNIX and virtio-vhost-user without code duplication or
> rewriting the devices.
>
> I hope you'll like the code when it's done.  If not, it still has useful
> code and ideas that would be needed to complete the vhost-pci RFC work
> like extending the PCI transport in the VIRTIO spec, handling vhost-user
> reconnection, etc.
>
> I'm aiming to send an RFC in the next 2 weeks.
>

Thanks. There would be at least three Slave handlers I can imagine:
- QEMU Slave handler to send master requests/responses to the guest
- Guest Slave handler
- QEMU Slave handler to send Guest Requests/responses to the master
I'm curious to see the code how could one be implemented so that the 
other other two could reuse.

I think the key issue is that we have a different viewpoint of protocol 
gating and protocol relaying. It is a high-level direction we need to 
align first before we could get into more details. Hope your upcoming 
code can get us a decision. Please also remember to reuse the dpdk code 
that my coworker posted to the dpdk mailinglist wherever possible, it 
may save your time to debug.

Best,
Wei

Re: [Qemu-devel] vhost-user graceful connect/disconnect

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2040 bytes --]

On Mon, Jan 08, 2018 at 07:22:37PM +0800, Wei Wang wrote:
> On 01/05/2018 11:49 PM, Stefan Hajnoczi wrote:
> > On Thu, Jan 04, 2018 at 07:15:38PM +0800, Wei Wang wrote:
> > > On 01/04/2018 06:47 PM, Stefan Hajnoczi wrote:
> > > > On Thu, Dec 21, 2017 at 06:01:29AM -0500, Marc-André Lureau wrote:
> > > > 
> > > > I'm not going to prototype this yet, I'm working on virtio-vhost-user
> > > > first, but eventually I might get back to -object vhost-user(-backend).
> > > > 
> > > Hi Stefan, are you implementing the guest slave and vhost-pci driver (we've
> > > posted to the dpdk mailinglist) as well? and do you have an estimation when
> > > would the prototype be ready?
> > I'm implementing the "[RFC virtio-dev] vhost-user-slave: add vhost-user
> > slave device type" device in QEMU and DPDK in order to show how the
> > ideas we've discussed work.
> > 
> > Here is the VIRTIO spec link again:
> > https://stefanha.github.io/virtio/vhost-user-slave.html#x1-2830007
> 
> There are four virtqueues documented in the spec, would two suffice? Request
> and Response can be distinguished by VHOST_USER_REPLY_MASK.

That's a good idea and I'll do it for the master-to-slave virtqueue.

Unfortunately virtqueues are asymmetric so a single virtqueue cannot
support the slave-to-master message flow (VHOST_USER_SET_SLAVE_REQ_FD):
1. Master puts an empty buffer onto vq.
2. Slave takes buffer and fills in a request.  The buffer is now used.
3. Master pops the used buffer but there is no way to reply back to the
   slave!

We still need 2 virtqueues for the slave-to-master message flow.

> I think the key issue is that we have a different viewpoint of protocol
> gating and protocol relaying. It is a high-level direction we need to align
> first before we could get into more details. Hope your upcoming code can get
> us a decision. Please also remember to reuse the dpdk code that my coworker
> posted to the dpdk mailinglist wherever possible, it may save your time to
> debug.

Thanks!

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] vhost-user graceful connect/disconnect

[Wei Wang]

On 01/09/2018 12:09 AM, Stefan Hajnoczi wrote:
> On Mon, Jan 08, 2018 at 07:22:37PM +0800, Wei Wang wrote:
>> On 01/05/2018 11:49 PM, Stefan Hajnoczi wrote:
>>> On Thu, Jan 04, 2018 at 07:15:38PM +0800, Wei Wang wrote:
>>>> On 01/04/2018 06:47 PM, Stefan Hajnoczi wrote:
>>>>> On Thu, Dec 21, 2017 at 06:01:29AM -0500, Marc-André Lureau wrote:
>>>>>
>>>>> I'm not going to prototype this yet, I'm working on virtio-vhost-user
>>>>> first, but eventually I might get back to -object vhost-user(-backend).
>>>>>
>>>> Hi Stefan, are you implementing the guest slave and vhost-pci driver (we've
>>>> posted to the dpdk mailinglist) as well? and do you have an estimation when
>>>> would the prototype be ready?
>>> I'm implementing the "[RFC virtio-dev] vhost-user-slave: add vhost-user
>>> slave device type" device in QEMU and DPDK in order to show how the
>>> ideas we've discussed work.
>>>
>>> Here is the VIRTIO spec link again:
>>> https://stefanha.github.io/virtio/vhost-user-slave.html#x1-2830007
>> There are four virtqueues documented in the spec, would two suffice? Request
>> and Response can be distinguished by VHOST_USER_REPLY_MASK.
> That's a good idea and I'll do it for the master-to-slave virtqueue.
>
> Unfortunately virtqueues are asymmetric so a single virtqueue cannot
> support the slave-to-master message flow (VHOST_USER_SET_SLAVE_REQ_FD):
> 1. Master puts an empty buffer onto vq.
> 2. Slave takes buffer and fills in a request.  The buffer is now used.
> 3. Master pops the used buffer but there is no way to reply back to the
>     slave!


Essentially, I think two virtqueues should be sufficient for 
guest-to-host and host-to-guest communication.

Host-to-guest virtqueue: used for master request/response messages 
passed to the guest slave (driver). The buffer can be primed by the 
guest driver (like virtio-net rxq).
Guest-to-host virtqueue: used for guest slave request/response messages, 
like virtio-net txq, it doesn't need to prime buffers in advance.


Best,
Wei

Re: [Qemu-devel] vhost-user graceful connect/disconnect

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2700 bytes --]

On Fri, Jan 12, 2018 at 02:53:43PM +0800, Wei Wang wrote:
> On 01/09/2018 12:09 AM, Stefan Hajnoczi wrote:
> > On Mon, Jan 08, 2018 at 07:22:37PM +0800, Wei Wang wrote:
> > > On 01/05/2018 11:49 PM, Stefan Hajnoczi wrote:
> > > > On Thu, Jan 04, 2018 at 07:15:38PM +0800, Wei Wang wrote:
> > > > > On 01/04/2018 06:47 PM, Stefan Hajnoczi wrote:
> > > > > > On Thu, Dec 21, 2017 at 06:01:29AM -0500, Marc-André Lureau wrote:
> > > > > > 
> > > > > > I'm not going to prototype this yet, I'm working on virtio-vhost-user
> > > > > > first, but eventually I might get back to -object vhost-user(-backend).
> > > > > > 
> > > > > Hi Stefan, are you implementing the guest slave and vhost-pci driver (we've
> > > > > posted to the dpdk mailinglist) as well? and do you have an estimation when
> > > > > would the prototype be ready?
> > > > I'm implementing the "[RFC virtio-dev] vhost-user-slave: add vhost-user
> > > > slave device type" device in QEMU and DPDK in order to show how the
> > > > ideas we've discussed work.
> > > > 
> > > > Here is the VIRTIO spec link again:
> > > > https://stefanha.github.io/virtio/vhost-user-slave.html#x1-2830007
> > > There are four virtqueues documented in the spec, would two suffice? Request
> > > and Response can be distinguished by VHOST_USER_REPLY_MASK.
> > That's a good idea and I'll do it for the master-to-slave virtqueue.
> > 
> > Unfortunately virtqueues are asymmetric so a single virtqueue cannot
> > support the slave-to-master message flow (VHOST_USER_SET_SLAVE_REQ_FD):
> > 1. Master puts an empty buffer onto vq.
> > 2. Slave takes buffer and fills in a request.  The buffer is now used.
> > 3. Master pops the used buffer but there is no way to reply back to the
> >     slave!
> 
> 
> Essentially, I think two virtqueues should be sufficient for guest-to-host
> and host-to-guest communication.
> 
> Host-to-guest virtqueue: used for master request/response messages passed to
> the guest slave (driver). The buffer can be primed by the guest driver (like
> virtio-net rxq).
> Guest-to-host virtqueue: used for guest slave request/response messages,
> like virtio-net txq, it doesn't need to prime buffers in advance.

Yes, multiplexing all messages over an rx and tx virtqueue is possible.
QEMU needs to demultiplex them on to the appropriate AF_UNIX fds.

The downside to multiplexing is that it makes per-queue policies
impossible (e.g. per-queue interrupts and affinity, multiqueue, etc).  I
think that's okay because we don't need them for virtio-vhost-user.  And
it's possible to add a feature bit for a new virtqueue layout later if
we hit problems...

I'll give this a try.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] vhost-user graceful connect/disconnect

[Michael S. Tsirkin]

On Tue, Dec 19, 2017 at 04:21:51PM +0000, Stefan Hajnoczi wrote:
> Hi,
> Vhost-user implementations assume the slave is already running before
> the master starts.  The slave is required during virtio device
> initialization (e.g. feature bit negotiation) and so it is simplest to
> assume that the master is already available and will respond immediately
> to the VHOST_USER_GET_FEATURES message.
> 
> I have thought about how to let master and slave start in any order.
> Some approaches involve changes to the VIRTIO specification so that
> guest drivers can wait until the vhost-user connection is established.
> 
> We can avoid spec changes using PCI hotplug:
> 
> 1. Introduce a new vhost-user object that manages a connection:
> 
>    -chardev ...
>    -object vhost-user,id=vhost-user0,chardev=chr0
> 
>    Note this object is *not* a NetClient.  It's a resource for managing
>    a vhost-user connection and can be used with any device type (net,
>    scsi, blk, etc).
> 
>    This object tries to establish a connection.  When a connection is
>    established it sends vhost-user protocol messages to fetch
>    information needed for virtio device initialization (like the number
>    of virtqueues supported, features bits, etc).  This information is
>    stashed so that vhost_*() calls later on do not require synchronous
>    communication with the vhost-user slave.

I'm not sure how important or feasible this last part is.

> 2. When the vhost-user connection is established the object emits a QMP
>    event so management software can hotplug the virtio device:
> 
>    VHOST_USER_CONNECTED source=vhost-user0
> 
> 3. The management software hotplugs the virtio device:
> 
>    (qmp) netdev_add vhost-user,id=netdev0,vhost-user=vhost-user0
>    (qmp) device_add virtio-net-pci,netdev=netdev0
> 
> Advantages of this approach:
> 
>  * Does not require spec changes.
>  * Can be implemented without vhost-user.c qemu_chr_fe_read/write_all()
>    calls that hang QEMU until the slave sends data.
>  * Allows slave to set the number of queues, feature bits, etc via the
>    vhost-user socket.  It is not necessary to manually specify feature
>    bitmasks and other slave-specific information on the QEMU
>    command-line.
> 
> I haven't thought through disconnection but I imagine the vhost-user
> object would emit a VHOST_USER_DISCONNECTED event so the management
> software can hot unplug the device.
> 
> Thoughts?
> 
> Stefan

This sounds like an interesting set of ideas to me.
Something to keep in mind for the future.

Thanks!

-- 
MST

Re: [Qemu-devel] vhost-user graceful connect/disconnect

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2733 bytes --]

On Thu, Jan 04, 2018 at 06:53:07PM +0200, Michael S. Tsirkin wrote:
> On Tue, Dec 19, 2017 at 04:21:51PM +0000, Stefan Hajnoczi wrote:
> > Hi,
> > Vhost-user implementations assume the slave is already running before
> > the master starts.  The slave is required during virtio device
> > initialization (e.g. feature bit negotiation) and so it is simplest to
> > assume that the master is already available and will respond immediately
> > to the VHOST_USER_GET_FEATURES message.
> > 
> > I have thought about how to let master and slave start in any order.
> > Some approaches involve changes to the VIRTIO specification so that
> > guest drivers can wait until the vhost-user connection is established.
> > 
> > We can avoid spec changes using PCI hotplug:
> > 
> > 1. Introduce a new vhost-user object that manages a connection:
> > 
> >    -chardev ...
> >    -object vhost-user,id=vhost-user0,chardev=chr0
> > 
> >    Note this object is *not* a NetClient.  It's a resource for managing
> >    a vhost-user connection and can be used with any device type (net,
> >    scsi, blk, etc).
> > 
> >    This object tries to establish a connection.  When a connection is
> >    established it sends vhost-user protocol messages to fetch
> >    information needed for virtio device initialization (like the number
> >    of virtqueues supported, features bits, etc).  This information is
> >    stashed so that vhost_*() calls later on do not require synchronous
> >    communication with the vhost-user slave.
> 
> I'm not sure how important or feasible this last part is.

The guest should not hang if the vhost-user slave fails to respond in a
timely manner.  This is important to prevent failures cascading if a
vhost-user network switch hangs, for example.

There are a few points where device emulation requires a full vhost-user
message exchange.  Theses are the vulnerable points.

Depending on how virtio emulation is implemented, it may be possible to
perform the vhost-user communication out-of-line with vcpu execution.
That way the vcpu continues to execute while vhost-user communication is
in progress.

Unfortunately the VIRTIO_PCI_STATUS/VIRTIO_PCI_COMMON_STATUS hardware
register writes are points where vhost-user message exchanges need to
happen.  That means the vcpu will hang until vhost-user communication
has finished.

It's not possible to resume guest execution without waiting for the
vhost-user slave because the guest might start using the device before
the vhost-user slave has been fully initialized.

I don't see a way around this problem yet.  If everything was async at
the virtio hardware level then it would be easy to prevent hangs.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]