[Qemu-devel] virtio-blk multiqueue support in qemu.

[Qemu-devel] virtio-blk multiqueue support in qemu.

[Naredula Janardhana Reddy]

[-- Attachment #1: Type: text/plain, Size: 474 bytes --]

HI,
     I am trying to improve the throughput of  virtio-blk for my kernel (
https://github.com/naredula-jana/Jiny-Kernel  )  to make use of the
multi-core using multi-queue. But I found the latest qemu  does not support
multiqueue for virtio-blk, The feature flag does not return multi-queue
support. Do we have any patches to the qemu for enabling multi-queue for
virtio-blk similar to that of virtio-net.  Can someone help some pointers
related to this..



Thanks
Jana

[-- Attachment #2: Type: text/html, Size: 646 bytes --]

Re: [Qemu-devel] virtio-blk multiqueue support in qemu.

[Fam Zheng]

On Wed, 07/01 22:45, Naredula Janardhana Reddy wrote:
> HI,
>      I am trying to improve the throughput of  virtio-blk for my kernel (
> https://github.com/naredula-jana/Jiny-Kernel  )  to make use of the
> multi-core using multi-queue. But I found the latest qemu  does not support
> multiqueue for virtio-blk, The feature flag does not return multi-queue
> support. Do we have any patches to the qemu for enabling multi-queue for
> virtio-blk similar to that of virtio-net.  Can someone help some pointers
> related to this..

There was a series last year adding support for multi-queue but wan't merged to
the code base:

http://comments.gmane.org/gmane.comp.emulators.qemu/289126

Fam

Re: [Qemu-devel] virtio-blk multiqueue support in qemu.

[Michael S. Tsirkin]

On Thu, Jul 02, 2015 at 09:21:21AM +0800, Fam Zheng wrote:
> On Wed, 07/01 22:45, Naredula Janardhana Reddy wrote:
> > HI,
> >      I am trying to improve the throughput of  virtio-blk for my kernel (
> > https://github.com/naredula-jana/Jiny-Kernel  )  to make use of the
> > multi-core using multi-queue. But I found the latest qemu  does not support
> > multiqueue for virtio-blk, The feature flag does not return multi-queue
> > support. Do we have any patches to the qemu for enabling multi-queue for
> > virtio-blk similar to that of virtio-net.  Can someone help some pointers
> > related to this..
> 
> There was a series last year adding support for multi-queue but wan't merged to
> the code base:
> 
> http://comments.gmane.org/gmane.comp.emulators.qemu/289126
> 
> Fam

Looks like it was mostly about the missing GPL license on some files.
Or did I miss some bigger issues?

-- 
MST

Re: [Qemu-devel] virtio-blk multiqueue support in qemu.

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1334 bytes --]

On Wed, Jul 08, 2015 at 05:37:12PM +0300, Michael S. Tsirkin wrote:
> On Thu, Jul 02, 2015 at 09:21:21AM +0800, Fam Zheng wrote:
> > On Wed, 07/01 22:45, Naredula Janardhana Reddy wrote:
> > > HI,
> > >      I am trying to improve the throughput of  virtio-blk for my kernel (
> > > https://github.com/naredula-jana/Jiny-Kernel  )  to make use of the
> > > multi-core using multi-queue. But I found the latest qemu  does not support
> > > multiqueue for virtio-blk, The feature flag does not return multi-queue
> > > support. Do we have any patches to the qemu for enabling multi-queue for
> > > virtio-blk similar to that of virtio-net.  Can someone help some pointers
> > > related to this..
> > 
> > There was a series last year adding support for multi-queue but wan't merged to
> > the code base:
> > 
> > http://comments.gmane.org/gmane.comp.emulators.qemu/289126
> > 
> > Fam
> 
> Looks like it was mostly about the missing GPL license on some files.
> Or did I miss some bigger issues?

I think this led to a lot of optimization and some controversial design
questions about bypassing parts of the QEMU block layer.  So I doubt
that this series can be merged without significant changes.

Will take a look at it again though because I'm also interested in
multiqueue block layer support.

Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] virtio-blk multiqueue support in qemu.

[Michael S. Tsirkin]

On Thu, Jul 09, 2015 at 02:22:48PM +0100, Stefan Hajnoczi wrote:
> On Wed, Jul 08, 2015 at 05:37:12PM +0300, Michael S. Tsirkin wrote:
> > On Thu, Jul 02, 2015 at 09:21:21AM +0800, Fam Zheng wrote:
> > > On Wed, 07/01 22:45, Naredula Janardhana Reddy wrote:
> > > > HI,
> > > >      I am trying to improve the throughput of  virtio-blk for my kernel (
> > > > https://github.com/naredula-jana/Jiny-Kernel  )  to make use of the
> > > > multi-core using multi-queue. But I found the latest qemu  does not support
> > > > multiqueue for virtio-blk, The feature flag does not return multi-queue
> > > > support. Do we have any patches to the qemu for enabling multi-queue for
> > > > virtio-blk similar to that of virtio-net.  Can someone help some pointers
> > > > related to this..
> > > 
> > > There was a series last year adding support for multi-queue but wan't merged to
> > > the code base:
> > > 
> > > http://comments.gmane.org/gmane.comp.emulators.qemu/289126
> > > 
> > > Fam
> > 
> > Looks like it was mostly about the missing GPL license on some files.
> > Or did I miss some bigger issues?
> 
> I reviewed the series again.  The most important part:
> 
> It was not a true multiqueue implementation - all the virtqueues are
> handled from a single thread in QEMU.  The reason for this is that
> QEMU's block layer isn't threadsafe so a lot of work is necessary to
> support true multiqueue.
> 
> Also, the performance numbers quoted cannot be attributed to multiqueue
> due to the number of additional changes the series made.  In the
> meantime, some of these additional changes have been merged as separate
> series.
> 
> On the positive side, we do need something similar to the last 3 patches
> to expose multiqueue to the guest.  But it will require much more work
> in QEMU to achieve true multiqueue.
> 
> Stefan

Might this be useful as an intermediate step, e.g. for testing
guest code?

-- 
MST

Re: [Qemu-devel] virtio-blk multiqueue support in qemu.

[Naredula Janardhana Reddy]

[-- Attachment #1: Type: text/plain, Size: 2350 bytes --]

Yes, It will be usefull for testing the guest code.  Currently virtio-blk
does not returns the multi-queue in feature flag of virtio, so no way
testing the feature in the guest driver.

In the virtio-net,  multi-queue  feature can be enabled  and tested.

Thanks
Jana

On 9 July 2015 at 18:59, Michael S. Tsirkin <mst@redhat.com> wrote:

> On Thu, Jul 09, 2015 at 02:22:48PM +0100, Stefan Hajnoczi wrote:
> > On Wed, Jul 08, 2015 at 05:37:12PM +0300, Michael S. Tsirkin wrote:
> > > On Thu, Jul 02, 2015 at 09:21:21AM +0800, Fam Zheng wrote:
> > > > On Wed, 07/01 22:45, Naredula Janardhana Reddy wrote:
> > > > > HI,
> > > > >      I am trying to improve the throughput of  virtio-blk for my
> kernel (
> > > > > https://github.com/naredula-jana/Jiny-Kernel  )  to make use of
> the
> > > > > multi-core using multi-queue. But I found the latest qemu  does
> not support
> > > > > multiqueue for virtio-blk, The feature flag does not return
> multi-queue
> > > > > support. Do we have any patches to the qemu for enabling
> multi-queue for
> > > > > virtio-blk similar to that of virtio-net.  Can someone help some
> pointers
> > > > > related to this..
> > > >
> > > > There was a series last year adding support for multi-queue but
> wan't merged to
> > > > the code base:
> > > >
> > > > http://comments.gmane.org/gmane.comp.emulators.qemu/289126
> > > >
> > > > Fam
> > >
> > > Looks like it was mostly about the missing GPL license on some files.
> > > Or did I miss some bigger issues?
> >
> > I reviewed the series again.  The most important part:
> >
> > It was not a true multiqueue implementation - all the virtqueues are
> > handled from a single thread in QEMU.  The reason for this is that
> > QEMU's block layer isn't threadsafe so a lot of work is necessary to
> > support true multiqueue.
> >
> > Also, the performance numbers quoted cannot be attributed to multiqueue
> > due to the number of additional changes the series made.  In the
> > meantime, some of these additional changes have been merged as separate
> > series.
> >
> > On the positive side, we do need something similar to the last 3 patches
> > to expose multiqueue to the guest.  But it will require much more work
> > in QEMU to achieve true multiqueue.
> >
> > Stefan
>
> Might this be useful as an intermediate step, e.g. for testing
> guest code?
>
> --
> MST
>

[-- Attachment #2: Type: text/html, Size: 3396 bytes --]

Re: [Qemu-devel] virtio-blk multiqueue support in qemu.

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2328 bytes --]

On Thu, Jul 09, 2015 at 04:29:30PM +0300, Michael S. Tsirkin wrote:
> On Thu, Jul 09, 2015 at 02:22:48PM +0100, Stefan Hajnoczi wrote:
> > On Wed, Jul 08, 2015 at 05:37:12PM +0300, Michael S. Tsirkin wrote:
> > > On Thu, Jul 02, 2015 at 09:21:21AM +0800, Fam Zheng wrote:
> > > > On Wed, 07/01 22:45, Naredula Janardhana Reddy wrote:
> > > > > HI,
> > > > >      I am trying to improve the throughput of  virtio-blk for my kernel (
> > > > > https://github.com/naredula-jana/Jiny-Kernel  )  to make use of the
> > > > > multi-core using multi-queue. But I found the latest qemu  does not support
> > > > > multiqueue for virtio-blk, The feature flag does not return multi-queue
> > > > > support. Do we have any patches to the qemu for enabling multi-queue for
> > > > > virtio-blk similar to that of virtio-net.  Can someone help some pointers
> > > > > related to this..
> > > > 
> > > > There was a series last year adding support for multi-queue but wan't merged to
> > > > the code base:
> > > > 
> > > > http://comments.gmane.org/gmane.comp.emulators.qemu/289126
> > > > 
> > > > Fam
> > > 
> > > Looks like it was mostly about the missing GPL license on some files.
> > > Or did I miss some bigger issues?
> > 
> > I reviewed the series again.  The most important part:
> > 
> > It was not a true multiqueue implementation - all the virtqueues are
> > handled from a single thread in QEMU.  The reason for this is that
> > QEMU's block layer isn't threadsafe so a lot of work is necessary to
> > support true multiqueue.
> > 
> > Also, the performance numbers quoted cannot be attributed to multiqueue
> > due to the number of additional changes the series made.  In the
> > meantime, some of these additional changes have been merged as separate
> > series.
> > 
> > On the positive side, we do need something similar to the last 3 patches
> > to expose multiqueue to the guest.  But it will require much more work
> > in QEMU to achieve true multiqueue.
> > 
> > Stefan
> 
> Might this be useful as an intermediate step, e.g. for testing
> guest code?

Yes.

It will have a performance impact since there are now more virtqueues
and interrupts.  It just won't scale the same way blk-mq does on the
host.

I'll put together patches based on qemu.git/master.

Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] virtio-blk multiqueue support in qemu.

[Paolo Bonzini]

On 09/07/2015 15:29, Michael S. Tsirkin wrote:
> > On the positive side, we do need something similar to the last 3 patches
> > to expose multiqueue to the guest.  But it will require much more work
> > in QEMU to achieve true multiqueue.
> 
> Might this be useful as an intermediate step, e.g. for testing
> guest code?

Yes, this is already how virtio-scsi multiqueue is implemented.  The
speedup is noticeable.  However, for virtio-blk there is a huge
consumption of interrupt vectors in the guest, so it doesn't scale to
multiple disks as well as virtio-scsi.

Re: [Qemu-devel] virtio-blk multiqueue support in qemu.

[Michael S. Tsirkin]

On Mon, Jul 13, 2015 at 12:17:43PM +0200, Paolo Bonzini wrote:
> 
> 
> On 09/07/2015 15:29, Michael S. Tsirkin wrote:
> > > On the positive side, we do need something similar to the last 3 patches
> > > to expose multiqueue to the guest.  But it will require much more work
> > > in QEMU to achieve true multiqueue.
> > 
> > Might this be useful as an intermediate step, e.g. for testing
> > guest code?
> 
> Yes, this is already how virtio-scsi multiqueue is implemented.  The
> speedup is noticeable.  However, for virtio-blk there is a huge
> consumption of interrupt vectors in the guest, so it doesn't scale to
> multiple disks as well as virtio-scsi.

Could you elaborate please?  interrupt vector usage is generally
up to the guest, so whatever the issue is might be fixable by driver
changes alone.

-- 
MST

Re: [Qemu-devel] virtio-blk multiqueue support in qemu.

[Paolo Bonzini]

On 13/07/2015 13:21, Michael S. Tsirkin wrote:
> > Yes, this is already how virtio-scsi multiqueue is implemented.  The
> > speedup is noticeable.  However, for virtio-blk there is a huge
> > consumption of interrupt vectors in the guest, so it doesn't scale to
> > multiple disks as well as virtio-scsi.
>
> Could you elaborate please?  interrupt vector usage is generally
> up to the guest, so whatever the issue is might be fixable by driver
> changes alone.

Yes, but in practice neither Linux nor Windows (AFAIK) are able to share
the same vector across multiple MSIs.

Paolo

Re: [Qemu-devel] virtio-blk multiqueue support in qemu.

[Michael S. Tsirkin]

On Mon, Jul 13, 2015 at 01:24:15PM +0200, Paolo Bonzini wrote:
> 
> 
> On 13/07/2015 13:21, Michael S. Tsirkin wrote:
> > > Yes, this is already how virtio-scsi multiqueue is implemented.  The
> > > speedup is noticeable.  However, for virtio-blk there is a huge
> > > consumption of interrupt vectors in the guest, so it doesn't scale to
> > > multiple disks as well as virtio-scsi.
> >
> > Could you elaborate please?  interrupt vector usage is generally
> > up to the guest, so whatever the issue is might be fixable by driver
> > changes alone.
> 
> Yes, but in practice neither Linux nor Windows (AFAIK) are able to share
> the same vector across multiple MSIs.
> 
> Paolo

AFAIK Linux can do this of you pass in IRQF_SHARED.
We just never saw a need to do this before.
Sharing between devices might also work, I think
Amos Kong made it work under qemu, though this relied
on reading ISR when in MSI mode, which is only allowed
in virtio 1 mode.

virtio spec also lets drivers control the mapping from
VQs to MSI vectors. That's portable across OSes.
That one is actually used, at least the Linux drivers support
a mode where all VQs share a single vector.

-- 
MST