Re: [Qemu-devel] [PATCH v3 0/7] virtio-blk: multiqueue support

[Stefan Hajnoczi <stefanha@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 5757 bytes --]

On Mon, Jun 20, 2016 at 03:29:43PM +0200, Roman Penyaev wrote:
> Hi, Stefan.
> 
> On Mon, Jun 20, 2016 at 12:36 PM, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > On Tue, Jun 07, 2016 at 05:28:24PM +0100, Stefan Hajnoczi wrote:
> >> v3:
> >>  * Drop Patch 1 to batch guest notify for non-dataplane
> >>
> >>    The Linux AIO completion BH and the virtio-blk batch notify BH changed order
> >>    in the AioContext->first_bh list as a side-effect of moving the BH from
> >>    hw/block/dataplane/virtio-blk.c to hw/block/virtio-blk.c.  This caused a
> >>    serious performance regression for both dataplane and non-dataplane.
> >>
> >>    I've decided not to move the BH in this series and work on a separate
> >>    solution for making batch notify generic.
> >>
> >>    The remaining patches have been reordered and cleaned up.
> >>
> >>  * See performance data below.
> >>
> >> v2:
> >>  * Simplify s->rq live migration [Paolo]
> >>  * Use more efficient bitmap ops for batch notification [Paolo]
> >>  * Fix perf regression due to batch notify BH in wrong AioContext [Christian]
> >>
> >> The virtio_blk guest driver has supported multiple virtqueues since Linux 3.17.
> >> This patch series adds multiple virtqueues to QEMU's virtio-blk emulated
> >> device.
> >>
> >> Ming Lei sent patches previously but these were not merged.  This series
> >> implements virtio-blk multiqueue for QEMU from scratch since the codebase has
> >> changed.  Live migration support for s->rq was also missing from the previous
> >> series and has been added.
> >>
> >> It's important to note that QEMU's block layer does not support multiqueue yet.
> >> Therefore virtio-blk device processes all virtqueues in the same AioContext
> >> (IOThread).  Further work is necessary to take advantage of multiqueue support
> >> in QEMU's block layer once it becomes available.
> >>
> >> Performance results:
> >>
> >> Using virtio-blk-pci,num-queues=4 can produce a speed-up but -smp 4
> >> introduces a lot of variance across runs.  No pinning was performed.
> >>
> >> Results show that there is no regression anymore, thanks to dropping the
> >> batch notify BH patch.
> >>
> >> RHEL 7.2 guest on RHEL 7.2 host with 1 vcpu and 1 GB RAM unless otherwise
> >> noted.  The default configuration of the Linux null_blk driver is used as
> >> /dev/vdb.
> >>
> >> $ cat files/fio.job
> >> [global]
> >> filename=/dev/vdb
> >> ioengine=libaio
> >> direct=1
> >> runtime=60
> >> ramp_time=5
> >> gtod_reduce=1
> >>
> >> [job1]
> >> numjobs=4
> >> iodepth=16
> >> rw=randread
> >> bs=4K
> >>
> >> $ ./analyze.py runs/
> >> Name                                   IOPS   Error
> >> unpatched-d6550e9ed2             19269820.2 ± 1.36%
> >> unpatched-dataplane-d6550e9ed2   22351400.4 ± 1.07%
> >> v3-dataplane                     22318511.2 ± 0.77%
> >> v3-no-dataplane                  18936103.8 ± 1.12%
> >> v3-queues-4-no-dataplane         19177021.8 ± 1.45%
> >> v3-smp-4-no-dataplane            25509585.2 ± 29.50%
> >> v3-smp-4-no-dataplane-no-mq      12466177.2 ± 7.88%
> >>
> >> Configuration:
> >> Name                             Patched? Dataplane? SMP? MQ?
> >> unpatched-d6550e9ed2                    N          N    N   N
> >> unpatched-dataplane-d6550e9ed2          N          Y    N   N
> >> v3-dataplane                            Y          Y    N   N
> >> v3-no-dataplane                         Y          N    N   N
> >> v3-queues-4-no-dataplane                Y          N    N   Y
> >> v3-smp-4-no-dataplane                   Y          N    Y   Y
> >> v3-smp-4-no-dataplane-no-mq             Y          N    Y   N
> >>
> >> SMP means -smp 4.
> >> MQ means virtio-blk-pci,num-queues=4.
> >>
> >> Stefan Hajnoczi (7):
> >>   virtio-blk: add VirtIOBlockConf->num_queues
> >>   virtio-blk: multiqueue batch notify
> >>   virtio-blk: tell dataplane which vq to notify
> >>   virtio-blk: associate request with a virtqueue
> >>   virtio-blk: live migrate s->rq with multiqueue
> >>   virtio-blk: dataplane multiqueue support
> >>   virtio-blk: add num-queues device property
> >>
> >>  hw/block/dataplane/virtio-blk.c | 81 +++++++++++++++++++++++++++++------------
> >>  hw/block/dataplane/virtio-blk.h |  2 +-
> >>  hw/block/virtio-blk.c           | 52 +++++++++++++++++++++-----
> >>  include/hw/virtio/virtio-blk.h  |  6 ++-
> >>  4 files changed, 105 insertions(+), 36 deletions(-)
> >
> > Ping?
> 
> I have one minor note regarding the following test:
> 
> "Name                             Patched? Dataplane? SMP? MQ?"
> "v3-queues-4-no-dataplane                Y          N    N   Y"
> 
> If I am not mistaken and understand your test description, it does not
> make a lot sense to use queues when you have VCPUs=1 (i.e. SMP=N),
> because block layer will not create nr_queues > CPUs number.  So what
> I want to say is that in this test even you specify num_queues=4 for
> virtio_blk, only 1 software queue will be created by the guest block
> layer and then it will be mapped to only 1 HW queue, even you have
> requested 4.  On host userspace side you will always receive IO in the
> queue #0.

Good point.

> And I've rebased and tested all my changes on top of your latest v3 set.
> The question is: are you interested in this up-to-date RFC of mine:
> "simple multithreaded MQ implementation for bdrv_raw" which I sent
> couple of weeks ago?  I can resend it by single merged commit once more.

If it's not too much work updating the RFC then I think it's useful.
Others may wish to experiment with your series to gather performance
numbers, especially for comparision when the QEMU block layer becomes mq
capable.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]