From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:53338) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TajhY-0001U9-By for qemu-devel@nongnu.org; Tue, 20 Nov 2012 04:01:12 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TajhP-0007iG-M9 for qemu-devel@nongnu.org; Tue, 20 Nov 2012 04:01:04 -0500 Received: from mx1.redhat.com ([209.132.183.28]:39697) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TajhP-0007hz-Dn for qemu-devel@nongnu.org; Tue, 20 Nov 2012 04:00:55 -0500 Message-ID: <50AB470F.7050408@redhat.com> Date: Tue, 20 Nov 2012 17:02:07 +0800 From: Asias He MIME-Version: 1.0 References: <1352992746-8767-1-git-send-email-stefanha@redhat.com> In-Reply-To: <1352992746-8767-1-git-send-email-stefanha@redhat.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [PATCH 0/7] virtio: virtio-blk data plane List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Stefan Hajnoczi Cc: Kevin Wolf , Anthony Liguori , "Michael S. Tsirkin" , qemu-devel@nongnu.org, khoa@us.ibm.com, Paolo Bonzini Hello Stefan, On 11/15/2012 11:18 PM, Stefan Hajnoczi wrote: > This series adds the -device virtio-blk-pci,x-data-plane=on property that > enables a high performance I/O codepath. A dedicated thread is used to process > virtio-blk requests outside the global mutex and without going through the QEMU > block layer. > > Khoa Huynh reported an increase from 140,000 IOPS to 600,000 > IOPS for a single VM using virtio-blk-data-plane in July: > > http://comments.gmane.org/gmane.comp.emulators.kvm.devel/94580 > > The virtio-blk-data-plane approach was originally presented at Linux Plumbers > Conference 2010. The following slides contain a brief overview: > > http://linuxplumbersconf.org/2010/ocw/system/presentations/651/original/Optimizing_the_QEMU_Storage_Stack.pdf > > The basic approach is: > 1. Each virtio-blk device has a thread dedicated to handling ioeventfd > signalling when the guest kicks the virtqueue. > 2. Requests are processed without going through the QEMU block layer using > Linux AIO directly. > 3. Completion interrupts are injected via irqfd from the dedicated thread. > > To try it out: > > qemu -drive if=none,id=drive0,cache=none,aio=native,format=raw,file=... > -device virtio-blk-pci,drive=drive0,scsi=off,x-data-plane=on Is this the latest dataplane bits: (git://github.com/stefanha/qemu.git virtio-blk-data-plane) commit 7872075c24fa01c925d4f41faa9d04ce69bf5328 Author: Stefan Hajnoczi Date: Wed Nov 14 15:45:38 2012 +0100 virtio-blk: add x-data-plane=on|off performance feature With this commit on a ramdisk based box, I am seeing about 10K IOPS with x-data-plane on and 90K IOPS with x-data-plane off. Any ideas? Command line I used: IMG=/dev/ram0 x86_64-softmmu/qemu-system-x86_64 \ -drive file=/root/img/sid.img,if=ide \ -drive file=${IMG},if=none,cache=none,aio=native,id=disk1 -device virtio-blk-pci,x-data-plane=off,drive=disk1,scsi=off \ -kernel $KERNEL -append "root=/dev/sdb1 console=tty0" \ -L /tmp/qemu-dataplane/share/qemu/ -nographic -vnc :0 -enable-kvm -m 2048 -smp 4 -cpu qemu64,+x2apic -M pc > > Limitations: > * Only format=raw is supported > * Live migration is not supported > * Block jobs, hot unplug, and other operations fail with -EBUSY > * I/O throttling limits are ignored > * Only Linux hosts are supported due to Linux AIO usage > > The code has reached a stage where I feel it is ready to merge. Users have > been playing with it for some time and want the significant performance boost. > > We are refactoring QEMU to get rid of the global mutex. I believe that > virtio-blk-data-plane can eventually become the default mode of operation. > > Instead of waiting for global mutex removal efforts to finish, I want to use > virtio-blk-data-plane as an example device for AioContext and threaded hw > dispatch refactoring. This means: > > 1. When the block layer can bind to an AioContext and execute I/O outside the > global mutex, virtio-blk-data-plane can use this (and gain image format > support). > > 2. When hw dispatch no longer needs the global mutex we can use hw/virtio.c > again and perhaps run a pool of iothreads instead of dedicated data plane > threads. > > But in the meantime, I have cleaned up the virtio-blk-data-plane code so that > it can be merged as an experimental feature. > > Changes from the RFC v9: > * Add x-data-plane=on|off option and coexist with regular virtio-blk code > * Create thread from BH so it inherits iothread cpusets > * Drain requests on vm_stop() so stopped guest does not access image file > * Add migration blocker > * Add bdrv_in_use() to prevent block jobs and other operations that can interfere > * Drop IOQueue request merging for simplicity > * Drop ioctl interrupt injection and always use irqfd for simplicity > * Major cleanup to split up source files > * Rebase from qemu-kvm.git onto qemu.git > * Address Michael Tsirkin's review comments > > Stefan Hajnoczi (7): > raw-posix: add raw_get_aio_fd() for virtio-blk-data-plane > configure: add CONFIG_VIRTIO_BLK_DATA_PLANE > dataplane: add virtqueue vring code > dataplane: add event loop > dataplane: add Linux AIO request queue > dataplane: add virtio-blk data plane code > virtio-blk: add x-data-plane=on|off performance feature > > block.h | 9 + > block/raw-posix.c | 34 ++++ > configure | 21 +++ > hw/Makefile.objs | 2 +- > hw/dataplane/Makefile.objs | 3 + > hw/dataplane/event-poll.c | 109 ++++++++++++ > hw/dataplane/event-poll.h | 40 +++++ > hw/dataplane/ioq.c | 118 +++++++++++++ > hw/dataplane/ioq.h | 57 +++++++ > hw/dataplane/virtio-blk.c | 414 +++++++++++++++++++++++++++++++++++++++++++++ > hw/dataplane/virtio-blk.h | 41 +++++ > hw/dataplane/vring.c | 321 +++++++++++++++++++++++++++++++++++ > hw/dataplane/vring.h | 54 ++++++ > hw/virtio-blk.c | 59 ++++++- > hw/virtio-blk.h | 1 + > hw/virtio-pci.c | 3 + > trace-events | 9 + > 17 files changed, 1293 insertions(+), 2 deletions(-) > create mode 100644 hw/dataplane/Makefile.objs > create mode 100644 hw/dataplane/event-poll.c > create mode 100644 hw/dataplane/event-poll.h > create mode 100644 hw/dataplane/ioq.c > create mode 100644 hw/dataplane/ioq.h > create mode 100644 hw/dataplane/virtio-blk.c > create mode 100644 hw/dataplane/virtio-blk.h > create mode 100644 hw/dataplane/vring.c > create mode 100644 hw/dataplane/vring.h > -- Asias