[PATCH] rbd: add queuing delay

[Christian Brunner <chb@muc.de>]

Hi Yehuda,

while running tests with qemu-io I've been experiencing a lot of
messages when running a large writev request (several hundred MB in
a single call):

10.06.20 22:10:07.337108 b67dcb70 client4136.objecter  pg 3.437e on [0] is laggy: 33
10.06.20 22:10:07.337708 b67dcb70 client4136.objecter  pg 3.2553 on [0] is laggy: 19
[...]

Everything is working fine, though. I think that the large number of
queued requests is the cause for this behaviour and I would propose to
delay futher requests (see attached patch).

What do you think about it?

Another question: I there a way to figure out max_osd through librados?

Christian

---
 block/rbd.c |   13 +++++++++++++
 1 files changed, 13 insertions(+), 0 deletions(-)

diff --git a/block/rbd.c b/block/rbd.c
index 74589cb..241b0c6 100644
--- a/block/rbd.c
+++ b/block/rbd.c
@@ -47,6 +47,14 @@
 
 #define OBJ_MAX_SIZE (1UL << OBJ_DEFAULT_OBJ_ORDER)
 
+/*
+ * For best performance MAX_RADOS_REQS should be at least as large as the 
+ * number of osds. It may be larger, but if to high you may experience lagging
+ *
+ * XXX: automatically set to 2*max_osd ???
+ */
+#define MAX_RADOS_REQS 16
+
 typedef struct RBDAIOCB {
     BlockDriverAIOCB common;
     QEMUBH *bh;
@@ -507,6 +515,11 @@ static BlockDriverAIOCB *rbd_aio_rw_vector(BlockDriverState *bs,
         rcb->segsize = segsize;
         rcb->buf = buf;
 
+        /* delay rados aio requests when the queue is getting to large */
+        while ((segnr - last_segnr + acb->aiocnt) > MAX_RADOS_REQS) {
+		usleep(100);
+	}
+
         if (write) {
             rados_aio_create_completion(rcb, NULL,
                                         (rados_callback_t) rbd_finish_aiocb,
-- 
1.7.0.4