[Qemu-devel] [PATCH v3 0/7] virtio-blk: multiqueue support

[Qemu-devel] [PATCH v3 0/7] virtio-blk: multiqueue support

[Stefan Hajnoczi]

v3:
 * Drop Patch 1 to batch guest notify for non-dataplane

   The Linux AIO completion BH and the virtio-blk batch notify BH changed order
   in the AioContext->first_bh list as a side-effect of moving the BH from
   hw/block/dataplane/virtio-blk.c to hw/block/virtio-blk.c.  This caused a
   serious performance regression for both dataplane and non-dataplane.

   I've decided not to move the BH in this series and work on a separate
   solution for making batch notify generic.

   The remaining patches have been reordered and cleaned up.

 * See performance data below.

v2:
 * Simplify s->rq live migration [Paolo]
 * Use more efficient bitmap ops for batch notification [Paolo]
 * Fix perf regression due to batch notify BH in wrong AioContext [Christian]

The virtio_blk guest driver has supported multiple virtqueues since Linux 3.17.
This patch series adds multiple virtqueues to QEMU's virtio-blk emulated
device.

Ming Lei sent patches previously but these were not merged.  This series
implements virtio-blk multiqueue for QEMU from scratch since the codebase has
changed.  Live migration support for s->rq was also missing from the previous
series and has been added.

It's important to note that QEMU's block layer does not support multiqueue yet.
Therefore virtio-blk device processes all virtqueues in the same AioContext
(IOThread).  Further work is necessary to take advantage of multiqueue support
in QEMU's block layer once it becomes available.

Performance results:

Using virtio-blk-pci,num-queues=4 can produce a speed-up but -smp 4
introduces a lot of variance across runs.  No pinning was performed.

Results show that there is no regression anymore, thanks to dropping the
batch notify BH patch.

RHEL 7.2 guest on RHEL 7.2 host with 1 vcpu and 1 GB RAM unless otherwise
noted.  The default configuration of the Linux null_blk driver is used as
/dev/vdb.

$ cat files/fio.job
[global]
filename=/dev/vdb
ioengine=libaio
direct=1
runtime=60
ramp_time=5
gtod_reduce=1

[job1]
numjobs=4
iodepth=16
rw=randread
bs=4K

$ ./analyze.py runs/
Name                                   IOPS   Error
unpatched-d6550e9ed2             19269820.2 ± 1.36%
unpatched-dataplane-d6550e9ed2   22351400.4 ± 1.07%
v3-dataplane                     22318511.2 ± 0.77%
v3-no-dataplane                  18936103.8 ± 1.12%
v3-queues-4-no-dataplane         19177021.8 ± 1.45%
v3-smp-4-no-dataplane            25509585.2 ± 29.50%
v3-smp-4-no-dataplane-no-mq      12466177.2 ± 7.88%

Configuration:
Name                             Patched? Dataplane? SMP? MQ?
unpatched-d6550e9ed2                    N          N    N   N
unpatched-dataplane-d6550e9ed2          N          Y    N   N
v3-dataplane                            Y          Y    N   N
v3-no-dataplane                         Y          N    N   N
v3-queues-4-no-dataplane                Y          N    N   Y
v3-smp-4-no-dataplane                   Y          N    Y   Y
v3-smp-4-no-dataplane-no-mq             Y          N    Y   N

SMP means -smp 4.
MQ means virtio-blk-pci,num-queues=4.

Stefan Hajnoczi (7):
  virtio-blk: add VirtIOBlockConf->num_queues
  virtio-blk: multiqueue batch notify
  virtio-blk: tell dataplane which vq to notify
  virtio-blk: associate request with a virtqueue
  virtio-blk: live migrate s->rq with multiqueue
  virtio-blk: dataplane multiqueue support
  virtio-blk: add num-queues device property

 hw/block/dataplane/virtio-blk.c | 81 +++++++++++++++++++++++++++++------------
 hw/block/dataplane/virtio-blk.h |  2 +-
 hw/block/virtio-blk.c           | 52 +++++++++++++++++++++-----
 include/hw/virtio/virtio-blk.h  |  6 ++-
 4 files changed, 105 insertions(+), 36 deletions(-)

-- 
2.5.5

[Qemu-devel] [PATCH v3 1/7] virtio-blk: add VirtIOBlockConf->num_queues

[Stefan Hajnoczi]

The num_queues field is always 1 for the time being.  A later patch will
make it a configurable device property so that multiqueue can be
enabled.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 hw/block/virtio-blk.c          | 1 +
 include/hw/virtio/virtio-blk.h | 1 +
 2 files changed, 2 insertions(+)

diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
index 284e646..0df7b35 100644
--- a/hw/block/virtio-blk.c
+++ b/hw/block/virtio-blk.c
@@ -888,6 +888,7 @@ static void virtio_blk_device_realize(DeviceState *dev, Error **errp)
     s->rq = NULL;
     s->sector_mask = (s->conf.conf.logical_block_size / BDRV_SECTOR_SIZE) - 1;
 
+    conf->num_queues = 1;
     s->vq = virtio_add_queue(vdev, 128, virtio_blk_handle_output);
     virtio_blk_data_plane_create(vdev, conf, &s->dataplane, &err);
     if (err != NULL) {
diff --git a/include/hw/virtio/virtio-blk.h b/include/hw/virtio/virtio-blk.h
index 8f2b056..9b03b6a 100644
--- a/include/hw/virtio/virtio-blk.h
+++ b/include/hw/virtio/virtio-blk.h
@@ -38,6 +38,7 @@ struct VirtIOBlkConf
     uint32_t scsi;
     uint32_t config_wce;
     uint32_t request_merging;
+    uint16_t num_queues;
 };
 
 struct VirtIOBlockDataPlane;
-- 
2.5.5

[Qemu-devel] [PATCH v3 2/7] virtio-blk: multiqueue batch notify

[Stefan Hajnoczi]

The batch notification BH needs to know which virtqueues to notify when
multiqueue is enabled.  Use a bitmap to track the virtqueues with
pending notifications.

At this point there is only one virtqueue so hard-code virtqueue index
0.  A later patch will switch to real virtqueue indices.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 hw/block/dataplane/virtio-blk.c | 29 +++++++++++++++++++++++------
 1 file changed, 23 insertions(+), 6 deletions(-)

diff --git a/hw/block/dataplane/virtio-blk.c b/hw/block/dataplane/virtio-blk.c
index 3cb97c9..4439c7b 100644
--- a/hw/block/dataplane/virtio-blk.c
+++ b/hw/block/dataplane/virtio-blk.c
@@ -34,8 +34,8 @@ struct VirtIOBlockDataPlane {
 
     VirtIODevice *vdev;
     VirtQueue *vq;                  /* virtqueue vring */
-    EventNotifier *guest_notifier;  /* irq */
     QEMUBH *bh;                     /* bh for guest notification */
+    unsigned long *batch_notify_vqs;
 
     Notifier insert_notifier, remove_notifier;
 
@@ -54,18 +54,34 @@ struct VirtIOBlockDataPlane {
 /* Raise an interrupt to signal guest, if necessary */
 void virtio_blk_data_plane_notify(VirtIOBlockDataPlane *s)
 {
+    set_bit(0, s->batch_notify_vqs);
     qemu_bh_schedule(s->bh);
 }
 
 static void notify_guest_bh(void *opaque)
 {
     VirtIOBlockDataPlane *s = opaque;
+    unsigned nvqs = s->conf->num_queues;
+    unsigned long bitmap[BITS_TO_LONGS(nvqs)];
+    unsigned j;
 
-    if (!virtio_should_notify(s->vdev, s->vq)) {
-        return;
+    memcpy(bitmap, s->batch_notify_vqs, sizeof(bitmap));
+    memset(s->batch_notify_vqs, 0, sizeof(bitmap));
+
+    for (j = 0; j < nvqs; j += BITS_PER_LONG) {
+        unsigned long bits = bitmap[j];
+
+        while (bits != 0) {
+            unsigned i = j + ctzl(bits);
+            VirtQueue *vq = virtio_get_queue(s->vdev, i);
+
+            if (virtio_should_notify(s->vdev, vq)) {
+                event_notifier_set(virtio_queue_get_guest_notifier(vq));
+            }
+
+            bits &= bits - 1; /* clear right-most bit */
+        }
     }
-
-    event_notifier_set(s->guest_notifier);
 }
 
 static void data_plane_set_up_op_blockers(VirtIOBlockDataPlane *s)
@@ -157,6 +173,7 @@ void virtio_blk_data_plane_create(VirtIODevice *vdev, VirtIOBlkConf *conf,
     }
     s->ctx = iothread_get_aio_context(s->iothread);
     s->bh = aio_bh_new(s->ctx, notify_guest_bh, s);
+    s->batch_notify_vqs = bitmap_new(conf->num_queues);
 
     s->insert_notifier.notify = data_plane_blk_insert_notifier;
     s->remove_notifier.notify = data_plane_blk_remove_notifier;
@@ -179,6 +196,7 @@ void virtio_blk_data_plane_destroy(VirtIOBlockDataPlane *s)
     data_plane_remove_op_blockers(s);
     notifier_remove(&s->insert_notifier);
     notifier_remove(&s->remove_notifier);
+    g_free(s->batch_notify_vqs);
     qemu_bh_delete(s->bh);
     object_unref(OBJECT(s->iothread));
     g_free(s);
@@ -217,7 +235,6 @@ void virtio_blk_data_plane_start(VirtIOBlockDataPlane *s)
                 "ensure -enable-kvm is set\n", r);
         goto fail_guest_notifiers;
     }
-    s->guest_notifier = virtio_queue_get_guest_notifier(s->vq);
 
     /* Set up virtqueue notify */
     r = k->set_host_notifier(qbus->parent, 0, true);
-- 
2.5.5

[Qemu-devel] [PATCH v3 3/7] virtio-blk: tell dataplane which vq to notify

[Stefan Hajnoczi]

Let the virtio_blk_data_plane_notify() caller decide which virtqueue to
notify.  This will allow the function to be used with multiqueue.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 hw/block/dataplane/virtio-blk.c | 4 ++--
 hw/block/dataplane/virtio-blk.h | 2 +-
 hw/block/virtio-blk.c           | 2 +-
 3 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/hw/block/dataplane/virtio-blk.c b/hw/block/dataplane/virtio-blk.c
index 4439c7b..221f970 100644
--- a/hw/block/dataplane/virtio-blk.c
+++ b/hw/block/dataplane/virtio-blk.c
@@ -52,9 +52,9 @@ struct VirtIOBlockDataPlane {
 };
 
 /* Raise an interrupt to signal guest, if necessary */
-void virtio_blk_data_plane_notify(VirtIOBlockDataPlane *s)
+void virtio_blk_data_plane_notify(VirtIOBlockDataPlane *s, VirtQueue *vq)
 {
-    set_bit(0, s->batch_notify_vqs);
+    set_bit(virtio_queue_get_id(vq), s->batch_notify_vqs);
     qemu_bh_schedule(s->bh);
 }
 
diff --git a/hw/block/dataplane/virtio-blk.h b/hw/block/dataplane/virtio-blk.h
index 0714c11..b1f0b95 100644
--- a/hw/block/dataplane/virtio-blk.h
+++ b/hw/block/dataplane/virtio-blk.h
@@ -26,6 +26,6 @@ void virtio_blk_data_plane_destroy(VirtIOBlockDataPlane *s);
 void virtio_blk_data_plane_start(VirtIOBlockDataPlane *s);
 void virtio_blk_data_plane_stop(VirtIOBlockDataPlane *s);
 void virtio_blk_data_plane_drain(VirtIOBlockDataPlane *s);
-void virtio_blk_data_plane_notify(VirtIOBlockDataPlane *s);
+void virtio_blk_data_plane_notify(VirtIOBlockDataPlane *s, VirtQueue *vq);
 
 #endif /* HW_DATAPLANE_VIRTIO_BLK_H */
diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
index 0df7b35..fa05e41 100644
--- a/hw/block/virtio-blk.c
+++ b/hw/block/virtio-blk.c
@@ -55,7 +55,7 @@ static void virtio_blk_req_complete(VirtIOBlockReq *req, unsigned char status)
     stb_p(&req->in->status, status);
     virtqueue_push(s->vq, &req->elem, req->in_len);
     if (s->dataplane_started && !s->dataplane_disabled) {
-        virtio_blk_data_plane_notify(s->dataplane);
+        virtio_blk_data_plane_notify(s->dataplane, s->vq);
     } else {
         virtio_notify(vdev, s->vq);
     }
-- 
2.5.5

[Qemu-devel] [PATCH v3 4/7] virtio-blk: associate request with a virtqueue

[Stefan Hajnoczi]

Multiqueue requires that each request knows to which virtqueue it
belongs.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 hw/block/virtio-blk.c          | 20 +++++++++++---------
 include/hw/virtio/virtio-blk.h |  4 +++-
 2 files changed, 14 insertions(+), 10 deletions(-)

diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
index fa05e41..222611c 100644
--- a/hw/block/virtio-blk.c
+++ b/hw/block/virtio-blk.c
@@ -29,9 +29,11 @@
 #include "hw/virtio/virtio-bus.h"
 #include "hw/virtio/virtio-access.h"
 
-void virtio_blk_init_request(VirtIOBlock *s, VirtIOBlockReq *req)
+void virtio_blk_init_request(VirtIOBlock *s, VirtQueue *vq,
+                             VirtIOBlockReq *req)
 {
     req->dev = s;
+    req->vq = vq;
     req->qiov.size = 0;
     req->in_len = 0;
     req->next = NULL;
@@ -53,11 +55,11 @@ static void virtio_blk_req_complete(VirtIOBlockReq *req, unsigned char status)
     trace_virtio_blk_req_complete(req, status);
 
     stb_p(&req->in->status, status);
-    virtqueue_push(s->vq, &req->elem, req->in_len);
+    virtqueue_push(req->vq, &req->elem, req->in_len);
     if (s->dataplane_started && !s->dataplane_disabled) {
-        virtio_blk_data_plane_notify(s->dataplane, s->vq);
+        virtio_blk_data_plane_notify(s->dataplane, req->vq);
     } else {
-        virtio_notify(vdev, s->vq);
+        virtio_notify(vdev, req->vq);
     }
 }
 
@@ -187,12 +189,12 @@ out:
 
 #endif
 
-static VirtIOBlockReq *virtio_blk_get_request(VirtIOBlock *s)
+static VirtIOBlockReq *virtio_blk_get_request(VirtIOBlock *s, VirtQueue *vq)
 {
-    VirtIOBlockReq *req = virtqueue_pop(s->vq, sizeof(VirtIOBlockReq));
+    VirtIOBlockReq *req = virtqueue_pop(vq, sizeof(VirtIOBlockReq));
 
     if (req) {
-        virtio_blk_init_request(s, req);
+        virtio_blk_init_request(s, vq, req);
     }
     return req;
 }
@@ -583,7 +585,7 @@ void virtio_blk_handle_vq(VirtIOBlock *s, VirtQueue *vq)
 
     blk_io_plug(s->blk);
 
-    while ((req = virtio_blk_get_request(s))) {
+    while ((req = virtio_blk_get_request(s, vq))) {
         virtio_blk_handle_request(req, &mrb);
     }
 
@@ -836,7 +838,7 @@ static int virtio_blk_load_device(VirtIODevice *vdev, QEMUFile *f,
     while (qemu_get_sbyte(f)) {
         VirtIOBlockReq *req;
         req = qemu_get_virtqueue_element(f, sizeof(VirtIOBlockReq));
-        virtio_blk_init_request(s, req);
+        virtio_blk_init_request(s, s->vq, req);
         req->next = s->rq;
         s->rq = req;
     }
diff --git a/include/hw/virtio/virtio-blk.h b/include/hw/virtio/virtio-blk.h
index 9b03b6a..a25b344 100644
--- a/include/hw/virtio/virtio-blk.h
+++ b/include/hw/virtio/virtio-blk.h
@@ -63,6 +63,7 @@ typedef struct VirtIOBlockReq {
     VirtQueueElement elem;
     int64_t sector_num;
     VirtIOBlock *dev;
+    VirtQueue *vq;
     struct virtio_blk_inhdr *in;
     struct virtio_blk_outhdr out;
     QEMUIOVector qiov;
@@ -80,7 +81,8 @@ typedef struct MultiReqBuffer {
     bool is_write;
 } MultiReqBuffer;
 
-void virtio_blk_init_request(VirtIOBlock *s, VirtIOBlockReq *req);
+void virtio_blk_init_request(VirtIOBlock *s, VirtQueue *vq,
+                             VirtIOBlockReq *req);
 void virtio_blk_free_request(VirtIOBlockReq *req);
 
 void virtio_blk_handle_request(VirtIOBlockReq *req, MultiReqBuffer *mrb);
-- 
2.5.5

[Qemu-devel] [PATCH v3 5/7] virtio-blk: live migrate s->rq with multiqueue

[Stefan Hajnoczi]

Add a field for the virtqueue index when migrating the s->rq request
list.  The new field is only needed when num_queues > 1.  Existing QEMUs
are unaffected by this change and therefore virtio-blk migration stays
compatible.

Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 hw/block/virtio-blk.c | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
index 222611c..6f122cf 100644
--- a/hw/block/virtio-blk.c
+++ b/hw/block/virtio-blk.c
@@ -813,6 +813,11 @@ static void virtio_blk_save_device(VirtIODevice *vdev, QEMUFile *f)
 
     while (req) {
         qemu_put_sbyte(f, 1);
+
+        if (s->conf.num_queues > 1) {
+            qemu_put_be32(f, virtio_queue_get_id(req->vq));
+        }
+
         qemu_put_virtqueue_element(f, &req->elem);
         req = req->next;
     }
@@ -836,9 +841,22 @@ static int virtio_blk_load_device(VirtIODevice *vdev, QEMUFile *f,
     VirtIOBlock *s = VIRTIO_BLK(vdev);
 
     while (qemu_get_sbyte(f)) {
+        unsigned nvqs = s->conf.num_queues;
+        unsigned vq_idx = 0;
         VirtIOBlockReq *req;
+
+        if (nvqs > 1) {
+            vq_idx = qemu_get_be32(f);
+
+            if (vq_idx >= nvqs) {
+                error_report("Invalid virtqueue index in request list: %#x",
+                             vq_idx);
+                return -EINVAL;
+            }
+        }
+
         req = qemu_get_virtqueue_element(f, sizeof(VirtIOBlockReq));
-        virtio_blk_init_request(s, s->vq, req);
+        virtio_blk_init_request(s, virtio_get_queue(vdev, vq_idx), req);
         req->next = s->rq;
         s->rq = req;
     }
-- 
2.5.5

[Qemu-devel] [PATCH v3 6/7] virtio-blk: dataplane multiqueue support

[Stefan Hajnoczi]

Monitor ioeventfds for all virtqueues in the device's AioContext.  This
is not true multiqueue because requests from all virtqueues are
processed in a single IOThread.  In the future it will be possible to
use multiple IOThreads when the QEMU block layer supports multiqueue.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 hw/block/dataplane/virtio-blk.c | 50 ++++++++++++++++++++++++++++-------------
 1 file changed, 34 insertions(+), 16 deletions(-)

diff --git a/hw/block/dataplane/virtio-blk.c b/hw/block/dataplane/virtio-blk.c
index 221f970..f54e9f6 100644
--- a/hw/block/dataplane/virtio-blk.c
+++ b/hw/block/dataplane/virtio-blk.c
@@ -31,9 +31,7 @@ struct VirtIOBlockDataPlane {
     bool stopping;
 
     VirtIOBlkConf *conf;
-
     VirtIODevice *vdev;
-    VirtQueue *vq;                  /* virtqueue vring */
     QEMUBH *bh;                     /* bh for guest notification */
     unsigned long *batch_notify_vqs;
 
@@ -219,6 +217,8 @@ void virtio_blk_data_plane_start(VirtIOBlockDataPlane *s)
     BusState *qbus = BUS(qdev_get_parent_bus(DEVICE(s->vdev)));
     VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(qbus);
     VirtIOBlock *vblk = VIRTIO_BLK(s->vdev);
+    unsigned i;
+    unsigned nvqs = s->conf->num_queues;
     int r;
 
     if (vblk->dataplane_started || s->starting) {
@@ -226,10 +226,9 @@ void virtio_blk_data_plane_start(VirtIOBlockDataPlane *s)
     }
 
     s->starting = true;
-    s->vq = virtio_get_queue(s->vdev, 0);
 
     /* Set up guest notifier (irq) */
-    r = k->set_guest_notifiers(qbus->parent, 1, true);
+    r = k->set_guest_notifiers(qbus->parent, nvqs, true);
     if (r != 0) {
         fprintf(stderr, "virtio-blk failed to set guest notifier (%d), "
                 "ensure -enable-kvm is set\n", r);
@@ -237,10 +236,15 @@ void virtio_blk_data_plane_start(VirtIOBlockDataPlane *s)
     }
 
     /* Set up virtqueue notify */
-    r = k->set_host_notifier(qbus->parent, 0, true);
-    if (r != 0) {
-        fprintf(stderr, "virtio-blk failed to set host notifier (%d)\n", r);
-        goto fail_host_notifier;
+    for (i = 0; i < nvqs; i++) {
+        r = k->set_host_notifier(qbus->parent, i, true);
+        if (r != 0) {
+            fprintf(stderr, "virtio-blk failed to set host notifier (%d)\n", r);
+            while (i--) {
+                k->set_host_notifier(qbus->parent, i, false);
+            }
+            goto fail_guest_notifiers;
+        }
     }
 
     s->starting = false;
@@ -250,17 +254,23 @@ void virtio_blk_data_plane_start(VirtIOBlockDataPlane *s)
     blk_set_aio_context(s->conf->conf.blk, s->ctx);
 
     /* Kick right away to begin processing requests already in vring */
-    event_notifier_set(virtio_queue_get_host_notifier(s->vq));
+    for (i = 0; i < nvqs; i++) {
+        VirtQueue *vq = virtio_get_queue(s->vdev, i);
+
+        event_notifier_set(virtio_queue_get_host_notifier(vq));
+    }
 
     /* Get this show started by hooking up our callbacks */
     aio_context_acquire(s->ctx);
-    virtio_queue_aio_set_host_notifier_handler(s->vq, s->ctx,
-                                               virtio_blk_data_plane_handle_output);
+    for (i = 0; i < nvqs; i++) {
+        VirtQueue *vq = virtio_get_queue(s->vdev, i);
+
+        virtio_queue_aio_set_host_notifier_handler(vq, s->ctx,
+                virtio_blk_data_plane_handle_output);
+    }
     aio_context_release(s->ctx);
     return;
 
-  fail_host_notifier:
-    k->set_guest_notifiers(qbus->parent, 1, false);
   fail_guest_notifiers:
     vblk->dataplane_disabled = true;
     s->starting = false;
@@ -273,6 +283,8 @@ void virtio_blk_data_plane_stop(VirtIOBlockDataPlane *s)
     BusState *qbus = BUS(qdev_get_parent_bus(DEVICE(s->vdev)));
     VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(qbus);
     VirtIOBlock *vblk = VIRTIO_BLK(s->vdev);
+    unsigned i;
+    unsigned nvqs = s->conf->num_queues;
 
     if (!vblk->dataplane_started || s->stopping) {
         return;
@@ -290,17 +302,23 @@ void virtio_blk_data_plane_stop(VirtIOBlockDataPlane *s)
     aio_context_acquire(s->ctx);
 
     /* Stop notifications for new requests from guest */
-    virtio_queue_aio_set_host_notifier_handler(s->vq, s->ctx, NULL);
+    for (i = 0; i < nvqs; i++) {
+        VirtQueue *vq = virtio_get_queue(s->vdev, i);
+
+        virtio_queue_aio_set_host_notifier_handler(vq, s->ctx, NULL);
+    }
 
     /* Drain and switch bs back to the QEMU main loop */
     blk_set_aio_context(s->conf->conf.blk, qemu_get_aio_context());
 
     aio_context_release(s->ctx);
 
-    k->set_host_notifier(qbus->parent, 0, false);
+    for (i = 0; i < nvqs; i++) {
+        k->set_host_notifier(qbus->parent, i, false);
+    }
 
     /* Clean up guest notifier (irq) */
-    k->set_guest_notifiers(qbus->parent, 1, false);
+    k->set_guest_notifiers(qbus->parent, nvqs, false);
 
     vblk->dataplane_started = false;
     s->stopping = false;
-- 
2.5.5

[Qemu-devel] [PATCH v3 7/7] virtio-blk: add num-queues device property

[Stefan Hajnoczi]

Multiqueue virtio-blk can be enabled as follows:

  qemu -device virtio-blk-pci,num-queues=8

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 hw/block/virtio-blk.c          | 15 +++++++++++++--
 include/hw/virtio/virtio-blk.h |  1 -
 2 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
index 6f122cf..a4d300d 100644
--- a/hw/block/virtio-blk.c
+++ b/hw/block/virtio-blk.c
@@ -710,6 +710,7 @@ static void virtio_blk_update_config(VirtIODevice *vdev, uint8_t *config)
     blkcfg.physical_block_exp = get_physical_block_exp(conf);
     blkcfg.alignment_offset = 0;
     blkcfg.wce = blk_enable_write_cache(s->blk);
+    virtio_stw_p(vdev, &blkcfg.num_queues, s->conf.num_queues);
     memcpy(config, &blkcfg, sizeof(struct virtio_blk_config));
 }
 
@@ -753,6 +754,9 @@ static uint64_t virtio_blk_get_features(VirtIODevice *vdev, uint64_t features,
     if (blk_is_read_only(s->blk)) {
         virtio_add_feature(&features, VIRTIO_BLK_F_RO);
     }
+    if (s->conf.num_queues > 1) {
+        virtio_add_feature(&features, VIRTIO_BLK_F_MQ);
+    }
 
     return features;
 }
@@ -882,6 +886,7 @@ static void virtio_blk_device_realize(DeviceState *dev, Error **errp)
     VirtIOBlkConf *conf = &s->conf;
     Error *err = NULL;
     static int virtio_blk_id;
+    unsigned i;
 
     if (!conf->conf.blk) {
         error_setg(errp, "drive property not set");
@@ -891,6 +896,10 @@ static void virtio_blk_device_realize(DeviceState *dev, Error **errp)
         error_setg(errp, "Device needs media, but drive is empty");
         return;
     }
+    if (!conf->num_queues) {
+        error_setg(errp, "num-queues property must be larger than 0");
+        return;
+    }
 
     blkconf_serial(&conf->conf, &conf->serial);
     s->original_wce = blk_enable_write_cache(conf->conf.blk);
@@ -908,8 +917,9 @@ static void virtio_blk_device_realize(DeviceState *dev, Error **errp)
     s->rq = NULL;
     s->sector_mask = (s->conf.conf.logical_block_size / BDRV_SECTOR_SIZE) - 1;
 
-    conf->num_queues = 1;
-    s->vq = virtio_add_queue(vdev, 128, virtio_blk_handle_output);
+    for (i = 0; i < conf->num_queues; i++) {
+        virtio_add_queue(vdev, 128, virtio_blk_handle_output);
+    }
     virtio_blk_data_plane_create(vdev, conf, &s->dataplane, &err);
     if (err != NULL) {
         error_propagate(errp, err);
@@ -962,6 +972,7 @@ static Property virtio_blk_properties[] = {
 #endif
     DEFINE_PROP_BIT("request-merging", VirtIOBlock, conf.request_merging, 0,
                     true),
+    DEFINE_PROP_UINT16("num-queues", VirtIOBlock, conf.num_queues, 1),
     DEFINE_PROP_END_OF_LIST(),
 };
 
diff --git a/include/hw/virtio/virtio-blk.h b/include/hw/virtio/virtio-blk.h
index a25b344..e9bf463 100644
--- a/include/hw/virtio/virtio-blk.h
+++ b/include/hw/virtio/virtio-blk.h
@@ -47,7 +47,6 @@ struct VirtIOBlockReq;
 typedef struct VirtIOBlock {
     VirtIODevice parent_obj;
     BlockBackend *blk;
-    VirtQueue *vq;
     void *rq;
     QEMUBH *bh;
     VirtIOBlkConf conf;
-- 
2.5.5

Re: [Qemu-devel] [PATCH v3 0/7] virtio-blk: multiqueue support

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 4041 bytes --]

On Tue, Jun 07, 2016 at 05:28:24PM +0100, Stefan Hajnoczi wrote:
> v3:
>  * Drop Patch 1 to batch guest notify for non-dataplane
> 
>    The Linux AIO completion BH and the virtio-blk batch notify BH changed order
>    in the AioContext->first_bh list as a side-effect of moving the BH from
>    hw/block/dataplane/virtio-blk.c to hw/block/virtio-blk.c.  This caused a
>    serious performance regression for both dataplane and non-dataplane.
> 
>    I've decided not to move the BH in this series and work on a separate
>    solution for making batch notify generic.
> 
>    The remaining patches have been reordered and cleaned up.
> 
>  * See performance data below.
> 
> v2:
>  * Simplify s->rq live migration [Paolo]
>  * Use more efficient bitmap ops for batch notification [Paolo]
>  * Fix perf regression due to batch notify BH in wrong AioContext [Christian]
> 
> The virtio_blk guest driver has supported multiple virtqueues since Linux 3.17.
> This patch series adds multiple virtqueues to QEMU's virtio-blk emulated
> device.
> 
> Ming Lei sent patches previously but these were not merged.  This series
> implements virtio-blk multiqueue for QEMU from scratch since the codebase has
> changed.  Live migration support for s->rq was also missing from the previous
> series and has been added.
> 
> It's important to note that QEMU's block layer does not support multiqueue yet.
> Therefore virtio-blk device processes all virtqueues in the same AioContext
> (IOThread).  Further work is necessary to take advantage of multiqueue support
> in QEMU's block layer once it becomes available.
> 
> Performance results:
> 
> Using virtio-blk-pci,num-queues=4 can produce a speed-up but -smp 4
> introduces a lot of variance across runs.  No pinning was performed.
> 
> Results show that there is no regression anymore, thanks to dropping the
> batch notify BH patch.
> 
> RHEL 7.2 guest on RHEL 7.2 host with 1 vcpu and 1 GB RAM unless otherwise
> noted.  The default configuration of the Linux null_blk driver is used as
> /dev/vdb.
> 
> $ cat files/fio.job
> [global]
> filename=/dev/vdb
> ioengine=libaio
> direct=1
> runtime=60
> ramp_time=5
> gtod_reduce=1
> 
> [job1]
> numjobs=4
> iodepth=16
> rw=randread
> bs=4K
> 
> $ ./analyze.py runs/
> Name                                   IOPS   Error
> unpatched-d6550e9ed2             19269820.2 ± 1.36%
> unpatched-dataplane-d6550e9ed2   22351400.4 ± 1.07%
> v3-dataplane                     22318511.2 ± 0.77%
> v3-no-dataplane                  18936103.8 ± 1.12%
> v3-queues-4-no-dataplane         19177021.8 ± 1.45%
> v3-smp-4-no-dataplane            25509585.2 ± 29.50%
> v3-smp-4-no-dataplane-no-mq      12466177.2 ± 7.88%
> 
> Configuration:
> Name                             Patched? Dataplane? SMP? MQ?
> unpatched-d6550e9ed2                    N          N    N   N
> unpatched-dataplane-d6550e9ed2          N          Y    N   N
> v3-dataplane                            Y          Y    N   N
> v3-no-dataplane                         Y          N    N   N
> v3-queues-4-no-dataplane                Y          N    N   Y
> v3-smp-4-no-dataplane                   Y          N    Y   Y
> v3-smp-4-no-dataplane-no-mq             Y          N    Y   N
> 
> SMP means -smp 4.
> MQ means virtio-blk-pci,num-queues=4.
> 
> Stefan Hajnoczi (7):
>   virtio-blk: add VirtIOBlockConf->num_queues
>   virtio-blk: multiqueue batch notify
>   virtio-blk: tell dataplane which vq to notify
>   virtio-blk: associate request with a virtqueue
>   virtio-blk: live migrate s->rq with multiqueue
>   virtio-blk: dataplane multiqueue support
>   virtio-blk: add num-queues device property
> 
>  hw/block/dataplane/virtio-blk.c | 81 +++++++++++++++++++++++++++++------------
>  hw/block/dataplane/virtio-blk.h |  2 +-
>  hw/block/virtio-blk.c           | 52 +++++++++++++++++++++-----
>  include/hw/virtio/virtio-blk.h  |  6 ++-
>  4 files changed, 105 insertions(+), 36 deletions(-)

Ping?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH v3 0/7] virtio-blk: multiqueue support

[Paolo Bonzini]

On 20/06/2016 12:36, Stefan Hajnoczi wrote:
> On Tue, Jun 07, 2016 at 05:28:24PM +0100, Stefan Hajnoczi wrote:
>> v3:
>>  * Drop Patch 1 to batch guest notify for non-dataplane
>>
>>    The Linux AIO completion BH and the virtio-blk batch notify BH changed order
>>    in the AioContext->first_bh list as a side-effect of moving the BH from
>>    hw/block/dataplane/virtio-blk.c to hw/block/virtio-blk.c.  This caused a
>>    serious performance regression for both dataplane and non-dataplane.
>>
>>    I've decided not to move the BH in this series and work on a separate
>>    solution for making batch notify generic.
>>
>>    The remaining patches have been reordered and cleaned up.
>>
>>  * See performance data below.
>>
>> v2:
>>  * Simplify s->rq live migration [Paolo]
>>  * Use more efficient bitmap ops for batch notification [Paolo]
>>  * Fix perf regression due to batch notify BH in wrong AioContext [Christian]
>>
>> The virtio_blk guest driver has supported multiple virtqueues since Linux 3.17.
>> This patch series adds multiple virtqueues to QEMU's virtio-blk emulated
>> device.
>>
>> Ming Lei sent patches previously but these were not merged.  This series
>> implements virtio-blk multiqueue for QEMU from scratch since the codebase has
>> changed.  Live migration support for s->rq was also missing from the previous
>> series and has been added.
>>
>> It's important to note that QEMU's block layer does not support multiqueue yet.
>> Therefore virtio-blk device processes all virtqueues in the same AioContext
>> (IOThread).  Further work is necessary to take advantage of multiqueue support
>> in QEMU's block layer once it becomes available.
>>
>> Performance results:
>>
>> Using virtio-blk-pci,num-queues=4 can produce a speed-up but -smp 4
>> introduces a lot of variance across runs.  No pinning was performed.
>>
>> Results show that there is no regression anymore, thanks to dropping the
>> batch notify BH patch.
>>
>> RHEL 7.2 guest on RHEL 7.2 host with 1 vcpu and 1 GB RAM unless otherwise
>> noted.  The default configuration of the Linux null_blk driver is used as
>> /dev/vdb.
>>
>> $ cat files/fio.job
>> [global]
>> filename=/dev/vdb
>> ioengine=libaio
>> direct=1
>> runtime=60
>> ramp_time=5
>> gtod_reduce=1
>>
>> [job1]
>> numjobs=4
>> iodepth=16
>> rw=randread
>> bs=4K
>>
>> $ ./analyze.py runs/
>> Name                                   IOPS   Error
>> unpatched-d6550e9ed2             19269820.2 ± 1.36%
>> unpatched-dataplane-d6550e9ed2   22351400.4 ± 1.07%
>> v3-dataplane                     22318511.2 ± 0.77%
>> v3-no-dataplane                  18936103.8 ± 1.12%
>> v3-queues-4-no-dataplane         19177021.8 ± 1.45%
>> v3-smp-4-no-dataplane            25509585.2 ± 29.50%
>> v3-smp-4-no-dataplane-no-mq      12466177.2 ± 7.88%
>>
>> Configuration:
>> Name                             Patched? Dataplane? SMP? MQ?
>> unpatched-d6550e9ed2                    N          N    N   N
>> unpatched-dataplane-d6550e9ed2          N          Y    N   N
>> v3-dataplane                            Y          Y    N   N
>> v3-no-dataplane                         Y          N    N   N
>> v3-queues-4-no-dataplane                Y          N    N   Y
>> v3-smp-4-no-dataplane                   Y          N    Y   Y
>> v3-smp-4-no-dataplane-no-mq             Y          N    Y   N
>>
>> SMP means -smp 4.
>> MQ means virtio-blk-pci,num-queues=4.
>>
>> Stefan Hajnoczi (7):
>>   virtio-blk: add VirtIOBlockConf->num_queues
>>   virtio-blk: multiqueue batch notify
>>   virtio-blk: tell dataplane which vq to notify
>>   virtio-blk: associate request with a virtqueue
>>   virtio-blk: live migrate s->rq with multiqueue
>>   virtio-blk: dataplane multiqueue support
>>   virtio-blk: add num-queues device property
>>
>>  hw/block/dataplane/virtio-blk.c | 81 +++++++++++++++++++++++++++++------------
>>  hw/block/dataplane/virtio-blk.h |  2 +-
>>  hw/block/virtio-blk.c           | 52 +++++++++++++++++++++-----
>>  include/hw/virtio/virtio-blk.h  |  6 ++-
>>  4 files changed, 105 insertions(+), 36 deletions(-)
> 
> Ping?
> 

Looks good to me, but I'll never understand who's supposed to apply
virtio-blk patches. :)

Paolo

Re: [Qemu-devel] [PATCH v3 0/7] virtio-blk: multiqueue support

[Roman Penyaev]

Hi, Stefan.

On Mon, Jun 20, 2016 at 12:36 PM, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> On Tue, Jun 07, 2016 at 05:28:24PM +0100, Stefan Hajnoczi wrote:
>> v3:
>>  * Drop Patch 1 to batch guest notify for non-dataplane
>>
>>    The Linux AIO completion BH and the virtio-blk batch notify BH changed order
>>    in the AioContext->first_bh list as a side-effect of moving the BH from
>>    hw/block/dataplane/virtio-blk.c to hw/block/virtio-blk.c.  This caused a
>>    serious performance regression for both dataplane and non-dataplane.
>>
>>    I've decided not to move the BH in this series and work on a separate
>>    solution for making batch notify generic.
>>
>>    The remaining patches have been reordered and cleaned up.
>>
>>  * See performance data below.
>>
>> v2:
>>  * Simplify s->rq live migration [Paolo]
>>  * Use more efficient bitmap ops for batch notification [Paolo]
>>  * Fix perf regression due to batch notify BH in wrong AioContext [Christian]
>>
>> The virtio_blk guest driver has supported multiple virtqueues since Linux 3.17.
>> This patch series adds multiple virtqueues to QEMU's virtio-blk emulated
>> device.
>>
>> Ming Lei sent patches previously but these were not merged.  This series
>> implements virtio-blk multiqueue for QEMU from scratch since the codebase has
>> changed.  Live migration support for s->rq was also missing from the previous
>> series and has been added.
>>
>> It's important to note that QEMU's block layer does not support multiqueue yet.
>> Therefore virtio-blk device processes all virtqueues in the same AioContext
>> (IOThread).  Further work is necessary to take advantage of multiqueue support
>> in QEMU's block layer once it becomes available.
>>
>> Performance results:
>>
>> Using virtio-blk-pci,num-queues=4 can produce a speed-up but -smp 4
>> introduces a lot of variance across runs.  No pinning was performed.
>>
>> Results show that there is no regression anymore, thanks to dropping the
>> batch notify BH patch.
>>
>> RHEL 7.2 guest on RHEL 7.2 host with 1 vcpu and 1 GB RAM unless otherwise
>> noted.  The default configuration of the Linux null_blk driver is used as
>> /dev/vdb.
>>
>> $ cat files/fio.job
>> [global]
>> filename=/dev/vdb
>> ioengine=libaio
>> direct=1
>> runtime=60
>> ramp_time=5
>> gtod_reduce=1
>>
>> [job1]
>> numjobs=4
>> iodepth=16
>> rw=randread
>> bs=4K
>>
>> $ ./analyze.py runs/
>> Name                                   IOPS   Error
>> unpatched-d6550e9ed2             19269820.2 ± 1.36%
>> unpatched-dataplane-d6550e9ed2   22351400.4 ± 1.07%
>> v3-dataplane                     22318511.2 ± 0.77%
>> v3-no-dataplane                  18936103.8 ± 1.12%
>> v3-queues-4-no-dataplane         19177021.8 ± 1.45%
>> v3-smp-4-no-dataplane            25509585.2 ± 29.50%
>> v3-smp-4-no-dataplane-no-mq      12466177.2 ± 7.88%
>>
>> Configuration:
>> Name                             Patched? Dataplane? SMP? MQ?
>> unpatched-d6550e9ed2                    N          N    N   N
>> unpatched-dataplane-d6550e9ed2          N          Y    N   N
>> v3-dataplane                            Y          Y    N   N
>> v3-no-dataplane                         Y          N    N   N
>> v3-queues-4-no-dataplane                Y          N    N   Y
>> v3-smp-4-no-dataplane                   Y          N    Y   Y
>> v3-smp-4-no-dataplane-no-mq             Y          N    Y   N
>>
>> SMP means -smp 4.
>> MQ means virtio-blk-pci,num-queues=4.
>>
>> Stefan Hajnoczi (7):
>>   virtio-blk: add VirtIOBlockConf->num_queues
>>   virtio-blk: multiqueue batch notify
>>   virtio-blk: tell dataplane which vq to notify
>>   virtio-blk: associate request with a virtqueue
>>   virtio-blk: live migrate s->rq with multiqueue
>>   virtio-blk: dataplane multiqueue support
>>   virtio-blk: add num-queues device property
>>
>>  hw/block/dataplane/virtio-blk.c | 81 +++++++++++++++++++++++++++++------------
>>  hw/block/dataplane/virtio-blk.h |  2 +-
>>  hw/block/virtio-blk.c           | 52 +++++++++++++++++++++-----
>>  include/hw/virtio/virtio-blk.h  |  6 ++-
>>  4 files changed, 105 insertions(+), 36 deletions(-)
>
> Ping?

I have one minor note regarding the following test:

"Name                             Patched? Dataplane? SMP? MQ?"
"v3-queues-4-no-dataplane                Y          N    N   Y"

If I am not mistaken and understand your test description, it does not
make a lot sense to use queues when you have VCPUs=1 (i.e. SMP=N),
because block layer will not create nr_queues > CPUs number.  So what
I want to say is that in this test even you specify num_queues=4 for
virtio_blk, only 1 software queue will be created by the guest block
layer and then it will be mapped to only 1 HW queue, even you have
requested 4.  On host userspace side you will always receive IO in the
queue #0.

And I've rebased and tested all my changes on top of your latest v3 set.
The question is: are you interested in this up-to-date RFC of mine:
"simple multithreaded MQ implementation for bdrv_raw" which I sent
couple of weeks ago?  I can resend it by single merged commit once more.

--
Roman

Re: [Qemu-devel] [PATCH v3 0/7] virtio-blk: multiqueue support

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 5757 bytes --]

On Mon, Jun 20, 2016 at 03:29:43PM +0200, Roman Penyaev wrote:
> Hi, Stefan.
> 
> On Mon, Jun 20, 2016 at 12:36 PM, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > On Tue, Jun 07, 2016 at 05:28:24PM +0100, Stefan Hajnoczi wrote:
> >> v3:
> >>  * Drop Patch 1 to batch guest notify for non-dataplane
> >>
> >>    The Linux AIO completion BH and the virtio-blk batch notify BH changed order
> >>    in the AioContext->first_bh list as a side-effect of moving the BH from
> >>    hw/block/dataplane/virtio-blk.c to hw/block/virtio-blk.c.  This caused a
> >>    serious performance regression for both dataplane and non-dataplane.
> >>
> >>    I've decided not to move the BH in this series and work on a separate
> >>    solution for making batch notify generic.
> >>
> >>    The remaining patches have been reordered and cleaned up.
> >>
> >>  * See performance data below.
> >>
> >> v2:
> >>  * Simplify s->rq live migration [Paolo]
> >>  * Use more efficient bitmap ops for batch notification [Paolo]
> >>  * Fix perf regression due to batch notify BH in wrong AioContext [Christian]
> >>
> >> The virtio_blk guest driver has supported multiple virtqueues since Linux 3.17.
> >> This patch series adds multiple virtqueues to QEMU's virtio-blk emulated
> >> device.
> >>
> >> Ming Lei sent patches previously but these were not merged.  This series
> >> implements virtio-blk multiqueue for QEMU from scratch since the codebase has
> >> changed.  Live migration support for s->rq was also missing from the previous
> >> series and has been added.
> >>
> >> It's important to note that QEMU's block layer does not support multiqueue yet.
> >> Therefore virtio-blk device processes all virtqueues in the same AioContext
> >> (IOThread).  Further work is necessary to take advantage of multiqueue support
> >> in QEMU's block layer once it becomes available.
> >>
> >> Performance results:
> >>
> >> Using virtio-blk-pci,num-queues=4 can produce a speed-up but -smp 4
> >> introduces a lot of variance across runs.  No pinning was performed.
> >>
> >> Results show that there is no regression anymore, thanks to dropping the
> >> batch notify BH patch.
> >>
> >> RHEL 7.2 guest on RHEL 7.2 host with 1 vcpu and 1 GB RAM unless otherwise
> >> noted.  The default configuration of the Linux null_blk driver is used as
> >> /dev/vdb.
> >>
> >> $ cat files/fio.job
> >> [global]
> >> filename=/dev/vdb
> >> ioengine=libaio
> >> direct=1
> >> runtime=60
> >> ramp_time=5
> >> gtod_reduce=1
> >>
> >> [job1]
> >> numjobs=4
> >> iodepth=16
> >> rw=randread
> >> bs=4K
> >>
> >> $ ./analyze.py runs/
> >> Name                                   IOPS   Error
> >> unpatched-d6550e9ed2             19269820.2 ± 1.36%
> >> unpatched-dataplane-d6550e9ed2   22351400.4 ± 1.07%
> >> v3-dataplane                     22318511.2 ± 0.77%
> >> v3-no-dataplane                  18936103.8 ± 1.12%
> >> v3-queues-4-no-dataplane         19177021.8 ± 1.45%
> >> v3-smp-4-no-dataplane            25509585.2 ± 29.50%
> >> v3-smp-4-no-dataplane-no-mq      12466177.2 ± 7.88%
> >>
> >> Configuration:
> >> Name                             Patched? Dataplane? SMP? MQ?
> >> unpatched-d6550e9ed2                    N          N    N   N
> >> unpatched-dataplane-d6550e9ed2          N          Y    N   N
> >> v3-dataplane                            Y          Y    N   N
> >> v3-no-dataplane                         Y          N    N   N
> >> v3-queues-4-no-dataplane                Y          N    N   Y
> >> v3-smp-4-no-dataplane                   Y          N    Y   Y
> >> v3-smp-4-no-dataplane-no-mq             Y          N    Y   N
> >>
> >> SMP means -smp 4.
> >> MQ means virtio-blk-pci,num-queues=4.
> >>
> >> Stefan Hajnoczi (7):
> >>   virtio-blk: add VirtIOBlockConf->num_queues
> >>   virtio-blk: multiqueue batch notify
> >>   virtio-blk: tell dataplane which vq to notify
> >>   virtio-blk: associate request with a virtqueue
> >>   virtio-blk: live migrate s->rq with multiqueue
> >>   virtio-blk: dataplane multiqueue support
> >>   virtio-blk: add num-queues device property
> >>
> >>  hw/block/dataplane/virtio-blk.c | 81 +++++++++++++++++++++++++++++------------
> >>  hw/block/dataplane/virtio-blk.h |  2 +-
> >>  hw/block/virtio-blk.c           | 52 +++++++++++++++++++++-----
> >>  include/hw/virtio/virtio-blk.h  |  6 ++-
> >>  4 files changed, 105 insertions(+), 36 deletions(-)
> >
> > Ping?
> 
> I have one minor note regarding the following test:
> 
> "Name                             Patched? Dataplane? SMP? MQ?"
> "v3-queues-4-no-dataplane                Y          N    N   Y"
> 
> If I am not mistaken and understand your test description, it does not
> make a lot sense to use queues when you have VCPUs=1 (i.e. SMP=N),
> because block layer will not create nr_queues > CPUs number.  So what
> I want to say is that in this test even you specify num_queues=4 for
> virtio_blk, only 1 software queue will be created by the guest block
> layer and then it will be mapped to only 1 HW queue, even you have
> requested 4.  On host userspace side you will always receive IO in the
> queue #0.

Good point.

> And I've rebased and tested all my changes on top of your latest v3 set.
> The question is: are you interested in this up-to-date RFC of mine:
> "simple multithreaded MQ implementation for bdrv_raw" which I sent
> couple of weeks ago?  I can resend it by single merged commit once more.

If it's not too much work updating the RFC then I think it's useful.
Others may wish to experiment with your series to gather performance
numbers, especially for comparision when the QEMU block layer becomes mq
capable.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH v3 0/7] virtio-blk: multiqueue support

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 4620 bytes --]

On Mon, Jun 20, 2016 at 01:42:14PM +0200, Paolo Bonzini wrote:
> 
> 
> On 20/06/2016 12:36, Stefan Hajnoczi wrote:
> > On Tue, Jun 07, 2016 at 05:28:24PM +0100, Stefan Hajnoczi wrote:
> >> v3:
> >>  * Drop Patch 1 to batch guest notify for non-dataplane
> >>
> >>    The Linux AIO completion BH and the virtio-blk batch notify BH changed order
> >>    in the AioContext->first_bh list as a side-effect of moving the BH from
> >>    hw/block/dataplane/virtio-blk.c to hw/block/virtio-blk.c.  This caused a
> >>    serious performance regression for both dataplane and non-dataplane.
> >>
> >>    I've decided not to move the BH in this series and work on a separate
> >>    solution for making batch notify generic.
> >>
> >>    The remaining patches have been reordered and cleaned up.
> >>
> >>  * See performance data below.
> >>
> >> v2:
> >>  * Simplify s->rq live migration [Paolo]
> >>  * Use more efficient bitmap ops for batch notification [Paolo]
> >>  * Fix perf regression due to batch notify BH in wrong AioContext [Christian]
> >>
> >> The virtio_blk guest driver has supported multiple virtqueues since Linux 3.17.
> >> This patch series adds multiple virtqueues to QEMU's virtio-blk emulated
> >> device.
> >>
> >> Ming Lei sent patches previously but these were not merged.  This series
> >> implements virtio-blk multiqueue for QEMU from scratch since the codebase has
> >> changed.  Live migration support for s->rq was also missing from the previous
> >> series and has been added.
> >>
> >> It's important to note that QEMU's block layer does not support multiqueue yet.
> >> Therefore virtio-blk device processes all virtqueues in the same AioContext
> >> (IOThread).  Further work is necessary to take advantage of multiqueue support
> >> in QEMU's block layer once it becomes available.
> >>
> >> Performance results:
> >>
> >> Using virtio-blk-pci,num-queues=4 can produce a speed-up but -smp 4
> >> introduces a lot of variance across runs.  No pinning was performed.
> >>
> >> Results show that there is no regression anymore, thanks to dropping the
> >> batch notify BH patch.
> >>
> >> RHEL 7.2 guest on RHEL 7.2 host with 1 vcpu and 1 GB RAM unless otherwise
> >> noted.  The default configuration of the Linux null_blk driver is used as
> >> /dev/vdb.
> >>
> >> $ cat files/fio.job
> >> [global]
> >> filename=/dev/vdb
> >> ioengine=libaio
> >> direct=1
> >> runtime=60
> >> ramp_time=5
> >> gtod_reduce=1
> >>
> >> [job1]
> >> numjobs=4
> >> iodepth=16
> >> rw=randread
> >> bs=4K
> >>
> >> $ ./analyze.py runs/
> >> Name                                   IOPS   Error
> >> unpatched-d6550e9ed2             19269820.2 ± 1.36%
> >> unpatched-dataplane-d6550e9ed2   22351400.4 ± 1.07%
> >> v3-dataplane                     22318511.2 ± 0.77%
> >> v3-no-dataplane                  18936103.8 ± 1.12%
> >> v3-queues-4-no-dataplane         19177021.8 ± 1.45%
> >> v3-smp-4-no-dataplane            25509585.2 ± 29.50%
> >> v3-smp-4-no-dataplane-no-mq      12466177.2 ± 7.88%
> >>
> >> Configuration:
> >> Name                             Patched? Dataplane? SMP? MQ?
> >> unpatched-d6550e9ed2                    N          N    N   N
> >> unpatched-dataplane-d6550e9ed2          N          Y    N   N
> >> v3-dataplane                            Y          Y    N   N
> >> v3-no-dataplane                         Y          N    N   N
> >> v3-queues-4-no-dataplane                Y          N    N   Y
> >> v3-smp-4-no-dataplane                   Y          N    Y   Y
> >> v3-smp-4-no-dataplane-no-mq             Y          N    Y   N
> >>
> >> SMP means -smp 4.
> >> MQ means virtio-blk-pci,num-queues=4.
> >>
> >> Stefan Hajnoczi (7):
> >>   virtio-blk: add VirtIOBlockConf->num_queues
> >>   virtio-blk: multiqueue batch notify
> >>   virtio-blk: tell dataplane which vq to notify
> >>   virtio-blk: associate request with a virtqueue
> >>   virtio-blk: live migrate s->rq with multiqueue
> >>   virtio-blk: dataplane multiqueue support
> >>   virtio-blk: add num-queues device property
> >>
> >>  hw/block/dataplane/virtio-blk.c | 81 +++++++++++++++++++++++++++++------------
> >>  hw/block/dataplane/virtio-blk.h |  2 +-
> >>  hw/block/virtio-blk.c           | 52 +++++++++++++++++++++-----
> >>  include/hw/virtio/virtio-blk.h  |  6 ++-
> >>  4 files changed, 105 insertions(+), 36 deletions(-)
> > 
> > Ping?
> > 
> 
> Looks good to me, but I'll never understand who's supposed to apply
> virtio-blk patches. :)

I'll rebase this series and include it for QEMU 2.7.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]