From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53460) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1b7M61-0003S9-6C for qemu-devel@nongnu.org; Mon, 30 May 2016 08:15:02 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1b7M5y-0002qr-BU for qemu-devel@nongnu.org; Mon, 30 May 2016 08:15:00 -0400 Received: from mail-io0-x22c.google.com ([2607:f8b0:4001:c06::22c]:34931) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1b7M5y-0002qn-4X for qemu-devel@nongnu.org; Mon, 30 May 2016 08:14:58 -0400 Received: by mail-io0-x22c.google.com with SMTP id p64so64431115ioi.2 for ; Mon, 30 May 2016 05:14:58 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <1786478881.754331933.1464590439361.JavaMail.zimbra@oxygem.tv> References: <1463787632-7241-1-git-send-email-stefanha@redhat.com> <1464350104-24825-1-git-send-email-roman.penyaev@profitbricks.com> <20160527222710.GS27946@stefanha-x1.localdomain> <1786478881.754331933.1464590439361.JavaMail.zimbra@oxygem.tv> From: Roman Penyaev Date: Mon, 30 May 2016 14:14:38 +0200 Message-ID: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [Qemu-devel] [RFC] virtio-blk: simple multithreaded MQ implementation for bdrv_raw List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Alexandre DERUMIER Cc: Stefan Hajnoczi , qemu-devel , stefanha On Mon, May 30, 2016 at 8:40 AM, Alexandre DERUMIER w= rote: > Hi, > >>>To avoid any locks in qemu backend and not to introduce thread safety >>>into qemu block-layer I open same backend device several times, one >>>device per one MQ. e.g. the following is the stack for a virtio-blk >>>with num-queues=3D2: > > Could it be possible in the future to not open several times the same bac= kend ? You are too fast :) I think nobody will do that in nearest future. > I'm thinking about ceph/librbd, which since last version allow only to op= en once a backend by default > (exclusive-lock, which is a requirement for advanced features like rbd-mi= rroring, fast-diff,....) Consider my patch as a hack for only one reason: make true MQ support for non-expandable file images and/or block devices to get some perf numbers on lockless IO path. If you are who is using block device as a backend and want to squeeze out the IO till last drop from guest MQ bdev to host MQ bdev - feel free to apply. That's the only reason of this work. -- Roman > > Regards, > > Alexandre Derumier > > > ----- Mail original ----- > De: "Stefan Hajnoczi" > =C3=80: "Roman Pen" > Cc: "qemu-devel" , "stefanha" > Envoy=C3=A9: Samedi 28 Mai 2016 00:27:10 > Objet: Re: [Qemu-devel] [RFC] virtio-blk: simple multithreaded MQ impleme= ntation for bdrv_raw > > On Fri, May 27, 2016 at 01:55:04PM +0200, Roman Pen wrote: >> Hello, all. >> >> This is RFC because mostly this patch is a quick attempt to get true >> multithreaded multiqueue support for a block device with native AIO. >> The goal is to squeeze everything possible on lockless IO path from >> MQ block on a guest to MQ block on a host. >> >> To avoid any locks in qemu backend and not to introduce thread safety >> into qemu block-layer I open same backend device several times, one >> device per one MQ. e.g. the following is the stack for a virtio-blk >> with num-queues=3D2: >> >> VirtIOBlock >> / \ >> VirtQueue#0 VirtQueue#1 >> IOThread#0 IOThread#1 >> BH#0 BH#1 >> Backend#0 Backend#1 >> \ / >> /dev/null0 >> >> To group all objects related to one vq new structure is introduced: >> >> typedef struct VirtQueueCtx { >> BlockBackend *blk; >> struct VirtIOBlock *s; >> VirtQueue *vq; >> void *rq; >> QEMUBH *bh; >> QEMUBH *batch_notify_bh; >> IOThread *iothread; >> Notifier insert_notifier; >> Notifier remove_notifier; >> /* Operation blocker on BDS */ >> Error *blocker; >> } VirtQueueCtx; >> >> And VirtIOBlock includes an array of these contexts: >> >> typedef struct VirtIOBlock { >> VirtIODevice parent_obj; >> + VirtQueueCtx mq[VIRTIO_QUEUE_MAX]; >> ... >> >> This patch is based on Stefan's series: "virtio-blk: multiqueue support"= , >> with minor difference: I reverted "virtio-blk: multiqueue batch notify", >> which does not make a lot sense when each VQ is handled by it's own >> iothread. >> >> The qemu configuration stays the same, i.e. put num-queues=3DN and N >> iothreads will be started on demand and N drives will be opened: >> >> qemu -device virtio-blk-pci,num-queues=3D8 >> >> My configuration is the following: >> >> host: >> Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz, >> 8 CPUs, >> /dev/nullb0 as backend with the following parameters: >> $ cat /sys/module/null_blk/parameters/submit_queues >> 8 >> $ cat /sys/module/null_blk/parameters/irqmode >> 1 >> >> guest: >> 8 VCPUs >> >> qemu: >> -object iothread,id=3Dt0 \ >> -drive if=3Dnone,id=3Dd0,file=3D/dev/nullb0,format=3Draw,snapshot=3Doff,= cache=3Dnone,aio=3Dnative \ >> -device virtio-blk-pci,num-queues=3D$N,iothread=3Dt0,drive=3Dd0,disable-= modern=3Doff,disable-legacy=3Don >> >> where $N varies during the tests. >> >> fio: >> [global] >> description=3DEmulation of Storage Server Access Pattern >> bssplit=3D512/20:1k/16:2k/9:4k/12:8k/19:16k/10:32k/8:64k/4 >> fadvise_hint=3D0 >> rw=3Drandrw:2 >> direct=3D1 >> >> ioengine=3Dlibaio >> iodepth=3D64 >> iodepth_batch_submit=3D64 >> iodepth_batch_complete=3D64 >> numjobs=3D8 >> gtod_reduce=3D1 >> group_reporting=3D1 >> >> time_based=3D1 >> runtime=3D30 >> >> [job] >> filename=3D/dev/vda >> >> Results: >> num-queues RD bw WR bw >> ---------- ----- ----- >> >> * with 1 iothread * >> >> 1 thr 1 mq 1225MB/s 1221MB/s >> 1 thr 2 mq 1559MB/s 1553MB/s >> 1 thr 4 mq 1729MB/s 1725MB/s >> 1 thr 8 mq 1660MB/s 1655MB/s >> >> * with N iothreads * >> >> 2 thr 2 mq 1845MB/s 1842MB/s >> 4 thr 4 mq 2187MB/s 2183MB/s >> 8 thr 8 mq 1383MB/s 1378MB/s >> >> Obviously, 8 iothreads + 8 vcpu threads is too much for my machine >> with 8 CPUs, but 4 iothreads show quite good result. > > Cool, thanks for trying this experiment and posting results. > > It's encouraging to see the improvement. Did you use any CPU affinity > settings to co-locate vcpu and iothreads onto host CPUs? > > Stefan