Re: [Qemu-devel] [snabb-devel] Re: [PATCH v5] vhost-user: add multi queue support

["Michael S. Tsirkin" <mst@redhat.com>]

On Thu, Jul 09, 2015 at 01:29:48AM +0000, Ouyang, Changchun wrote:
> 
> 
> > -----Original Message-----
> > From: snabb-devel@googlegroups.com [mailto:snabb-
> > devel@googlegroups.com] On Behalf Of Maxime Leroy
> > Sent: Thursday, July 9, 2015 6:01 AM
> > To: Michael S. Tsirkin
> > Cc: Ouyang, Changchun; snabb-devel@googlegroups.com; Marcel
> > Apfelbaum; qemu-devel@nongnu.org; Nikolay Nikolaev; Luke Gorrie; Long,
> > Thomas; rkhan@redhat.com
> > Subject: [snabb-devel] Re: [Qemu-devel] [PATCH v5] vhost-user: add multi
> > queue support
> > 
> > Hi Michael,
> > 
> > On Wed, Jul 8, 2015 at 4:29 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> > > On Thu, May 28, 2015 at 09:23:06AM +0800, Ouyang Changchun wrote:
> > >> Based on patch by Nikolay Nikolaev:
> > >> Vhost-user will implement the multi queue support in a similar way to
> > >> what vhost already has - a separate thread for each queue.
> > >> To enable the multi queue functionality - a new command line
> > >> parameter "queues" is introduced for the vhost-user netdev.
> > >>
> > >> Signed-off-by: Nikolay Nikolaev <n.nikolaev@virtualopensystems.com>
> > >> Signed-off-by: Changchun Ouyang <changchun.ouyang@intel.com>
> > >
> > > So testing turned up a significant issue with the protocol extension
> > > in this one.  Specifically, remote has no idea how many queues guest
> > > actually wants to use (it's dynamic, guest changes this at any time).
> > > We need support for enabling and disabling queues dynamically.
> 
> Do you mean we need control queue to negotiate the actual queue number between
> Guest and host? Or something like that

Exactly. In fact vhost user currently gets CTRL_VQ feature flag
which it really shouldn't since dpdk does not handle the
control VQ.

> > >
> > > Given we are past hard freeze, and given no one uses this yet (dpdk
> > > upstream did not merge supporting protocol), I think the best thing to
> > > do is to disable this functionality for 2.4.
> > > I will send a patch to do this shortly.
> > 
> > You are making a wrong statement, we already use multiqueue for vhost-
> > user and we expected to have this support officially integrated in qemu 2.4.
> > 
> > Libvirt 1.2.17 has been released with multiqueue support for vhost-user.
> > (http://libvirt.org/git/?p=libvirt.git;a=commit;h=366c22f2bcf1ddb8253c123f93
> > fd18d1ba9eacd6)
> > It checks against the version of qemu (i.e. 2.4)  to know if multiqueue is
> > supported or not by qemu.
> > (http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=7971723b985b9adc27122a
> > 3503e7ab38ced2b57f;hp=e7f5510ef2d28ca0ae0ed5751b1fd3018130d6c1)
> > 
> > Dynamically enabling/disabling queue between host/guest is a nice feature
> > to have.
> > But it's not mandatory. You can still configure manually guest and host to use
> > the same number of queues.
> 
> Same number of queues on host and guest can work normally, I have validated it with dpdk.

Try to use upstream dpdk without your patches and upstream qemu (with
patches integrated) and you will see some of the issues I am
talking about: crashes in guest and in dpdk.


> > 
> 
> Maybe we could consider still having this in 2.4,

I think that would be a mistake simply because it affects the
guest/host interface.

I think spec compliance is extremely important. Otherwise guest driver
writers need to study individual implementation bugs and work around
them, and that's just too painful.

virtio spec requires the number of queues to be configurable on the fly,
assuming we can't do that at the moment (and to me, it looks like
there is no way to give the necessary info to dpdk)
we simply must not set the multiqueue capability.

Two main things are broken on the QEMU side:
- MQ feature requires CTRL vq. QEMU does not pass CTRL vq info to DPDK
  so of course it just lies to guest that it's supported:
  all info is buffered in QEMU but stays unused.

- MQ feature requires that max number of queues is reported
  to guest. QEMU does not request this info from DPDK so
  of course it has no way to know this number.
  Instead it just assumes DPDK can handle whatever's thrown at it,
  but naturally this can't work in practice so of course it doesn't.

> And have an enhancement patch set to implement dynamically enabling/disabling in 2.5 or 2.4.x
> After extending the vhost-user spec. 
> 
> Thanks
> Changchun

2.5 development is just around the corner. With enough effort we can
have the patches upstream first thing in 2.5 cycle.

Once there, the whole feature can be backported if enough people
are prepared to dedicate resources to maintaining 2.4.x.
To me this sounds like a much better deal for everyone
than releasing a known-broken device and trying to fix it up.

-- 
MST