Re: [Qemu-devel] [PATCH 0/7] virtio: virtio-blk data plane

[Asias He <asias@redhat.com>]

Hello Stefan,

On 11/15/2012 11:18 PM, Stefan Hajnoczi wrote:
> This series adds the -device virtio-blk-pci,x-data-plane=on property that
> enables a high performance I/O codepath.  A dedicated thread is used to process
> virtio-blk requests outside the global mutex and without going through the QEMU
> block layer.
> 
> Khoa Huynh <khoa@us.ibm.com> reported an increase from 140,000 IOPS to 600,000
> IOPS for a single VM using virtio-blk-data-plane in July:
> 
>   http://comments.gmane.org/gmane.comp.emulators.kvm.devel/94580
> 
> The virtio-blk-data-plane approach was originally presented at Linux Plumbers
> Conference 2010.  The following slides contain a brief overview:
> 
>   http://linuxplumbersconf.org/2010/ocw/system/presentations/651/original/Optimizing_the_QEMU_Storage_Stack.pdf
> 
> The basic approach is:
> 1. Each virtio-blk device has a thread dedicated to handling ioeventfd
>    signalling when the guest kicks the virtqueue.
> 2. Requests are processed without going through the QEMU block layer using
>    Linux AIO directly.
> 3. Completion interrupts are injected via irqfd from the dedicated thread.
> 
> To try it out:
> 
>   qemu -drive if=none,id=drive0,cache=none,aio=native,format=raw,file=...
>        -device virtio-blk-pci,drive=drive0,scsi=off,x-data-plane=on


Is this the latest dataplane bits:
(git://github.com/stefanha/qemu.git virtio-blk-data-plane)

commit 7872075c24fa01c925d4f41faa9d04ce69bf5328
Author: Stefan Hajnoczi <stefanha@redhat.com>
Date:   Wed Nov 14 15:45:38 2012 +0100

    virtio-blk: add x-data-plane=on|off performance feature


With this commit on a ramdisk based box, I am seeing about 10K IOPS with
x-data-plane on and 90K IOPS with x-data-plane off.

Any ideas?

Command line I used:

IMG=/dev/ram0
x86_64-softmmu/qemu-system-x86_64 \
-drive file=/root/img/sid.img,if=ide \
-drive file=${IMG},if=none,cache=none,aio=native,id=disk1 -device
virtio-blk-pci,x-data-plane=off,drive=disk1,scsi=off \
-kernel $KERNEL -append "root=/dev/sdb1 console=tty0" \
-L /tmp/qemu-dataplane/share/qemu/ -nographic -vnc :0 -enable-kvm -m
2048 -smp 4 -cpu qemu64,+x2apic -M pc


> 
> Limitations:
>  * Only format=raw is supported
>  * Live migration is not supported
>  * Block jobs, hot unplug, and other operations fail with -EBUSY
>  * I/O throttling limits are ignored
>  * Only Linux hosts are supported due to Linux AIO usage
> 
> The code has reached a stage where I feel it is ready to merge.  Users have
> been playing with it for some time and want the significant performance boost.
> 
> We are refactoring QEMU to get rid of the global mutex.  I believe that
> virtio-blk-data-plane can eventually become the default mode of operation.
> 
> Instead of waiting for global mutex removal efforts to finish, I want to use
> virtio-blk-data-plane as an example device for AioContext and threaded hw
> dispatch refactoring.  This means:
> 
> 1. When the block layer can bind to an AioContext and execute I/O outside the
>    global mutex, virtio-blk-data-plane can use this (and gain image format
>    support).
> 
> 2. When hw dispatch no longer needs the global mutex we can use hw/virtio.c
>    again and perhaps run a pool of iothreads instead of dedicated data plane
>    threads.
> 
> But in the meantime, I have cleaned up the virtio-blk-data-plane code so that
> it can be merged as an experimental feature.
> 
> Changes from the RFC v9:
>  * Add x-data-plane=on|off option and coexist with regular virtio-blk code
>  * Create thread from BH so it inherits iothread cpusets
>  * Drain requests on vm_stop() so stopped guest does not access image file
>  * Add migration blocker
>  * Add bdrv_in_use() to prevent block jobs and other operations that can interfere
>  * Drop IOQueue request merging for simplicity
>  * Drop ioctl interrupt injection and always use irqfd for simplicity
>  * Major cleanup to split up source files
>  * Rebase from qemu-kvm.git onto qemu.git
>  * Address Michael Tsirkin's review comments
> 
> Stefan Hajnoczi (7):
>   raw-posix: add raw_get_aio_fd() for virtio-blk-data-plane
>   configure: add CONFIG_VIRTIO_BLK_DATA_PLANE
>   dataplane: add virtqueue vring code
>   dataplane: add event loop
>   dataplane: add Linux AIO request queue
>   dataplane: add virtio-blk data plane code
>   virtio-blk: add x-data-plane=on|off performance feature
> 
>  block.h                    |   9 +
>  block/raw-posix.c          |  34 ++++
>  configure                  |  21 +++
>  hw/Makefile.objs           |   2 +-
>  hw/dataplane/Makefile.objs |   3 +
>  hw/dataplane/event-poll.c  | 109 ++++++++++++
>  hw/dataplane/event-poll.h  |  40 +++++
>  hw/dataplane/ioq.c         | 118 +++++++++++++
>  hw/dataplane/ioq.h         |  57 +++++++
>  hw/dataplane/virtio-blk.c  | 414 +++++++++++++++++++++++++++++++++++++++++++++
>  hw/dataplane/virtio-blk.h  |  41 +++++
>  hw/dataplane/vring.c       | 321 +++++++++++++++++++++++++++++++++++
>  hw/dataplane/vring.h       |  54 ++++++
>  hw/virtio-blk.c            |  59 ++++++-
>  hw/virtio-blk.h            |   1 +
>  hw/virtio-pci.c            |   3 +
>  trace-events               |   9 +
>  17 files changed, 1293 insertions(+), 2 deletions(-)
>  create mode 100644 hw/dataplane/Makefile.objs
>  create mode 100644 hw/dataplane/event-poll.c
>  create mode 100644 hw/dataplane/event-poll.h
>  create mode 100644 hw/dataplane/ioq.c
>  create mode 100644 hw/dataplane/ioq.h
>  create mode 100644 hw/dataplane/virtio-blk.c
>  create mode 100644 hw/dataplane/virtio-blk.h
>  create mode 100644 hw/dataplane/vring.c
>  create mode 100644 hw/dataplane/vring.h
> 


-- 
Asias