Re: [Qemu-devel] [PATCH 0/7] virtio: virtio-blk data plane

[Stefan Hajnoczi <stefanha@redhat.com>]

On Wed, Nov 21, 2012 at 01:22:22PM +0800, Asias He wrote:
> On 11/20/2012 08:21 PM, Stefan Hajnoczi wrote:
> > On Tue, Nov 20, 2012 at 10:02 AM, Asias He <asias@redhat.com> wrote:
> >> Hello Stefan,
> >>
> >> On 11/15/2012 11:18 PM, Stefan Hajnoczi wrote:
> >>> This series adds the -device virtio-blk-pci,x-data-plane=on property that
> >>> enables a high performance I/O codepath.  A dedicated thread is used to process
> >>> virtio-blk requests outside the global mutex and without going through the QEMU
> >>> block layer.
> >>>
> >>> Khoa Huynh <khoa@us.ibm.com> reported an increase from 140,000 IOPS to 600,000
> >>> IOPS for a single VM using virtio-blk-data-plane in July:
> >>>
> >>>   http://comments.gmane.org/gmane.comp.emulators.kvm.devel/94580
> >>>
> >>> The virtio-blk-data-plane approach was originally presented at Linux Plumbers
> >>> Conference 2010.  The following slides contain a brief overview:
> >>>
> >>>   http://linuxplumbersconf.org/2010/ocw/system/presentations/651/original/Optimizing_the_QEMU_Storage_Stack.pdf
> >>>
> >>> The basic approach is:
> >>> 1. Each virtio-blk device has a thread dedicated to handling ioeventfd
> >>>    signalling when the guest kicks the virtqueue.
> >>> 2. Requests are processed without going through the QEMU block layer using
> >>>    Linux AIO directly.
> >>> 3. Completion interrupts are injected via irqfd from the dedicated thread.
> >>>
> >>> To try it out:
> >>>
> >>>   qemu -drive if=none,id=drive0,cache=none,aio=native,format=raw,file=...
> >>>        -device virtio-blk-pci,drive=drive0,scsi=off,x-data-plane=on
> >>
> >>
> >> Is this the latest dataplane bits:
> >> (git://github.com/stefanha/qemu.git virtio-blk-data-plane)
> >>
> >> commit 7872075c24fa01c925d4f41faa9d04ce69bf5328
> >> Author: Stefan Hajnoczi <stefanha@redhat.com>
> >> Date:   Wed Nov 14 15:45:38 2012 +0100
> >>
> >>     virtio-blk: add x-data-plane=on|off performance feature
> >>
> >>
> >> With this commit on a ramdisk based box, I am seeing about 10K IOPS with
> >> x-data-plane on and 90K IOPS with x-data-plane off.
> >>
> >> Any ideas?
> >>
> >> Command line I used:
> >>
> >> IMG=/dev/ram0
> >> x86_64-softmmu/qemu-system-x86_64 \
> >> -drive file=/root/img/sid.img,if=ide \
> >> -drive file=${IMG},if=none,cache=none,aio=native,id=disk1 -device
> >> virtio-blk-pci,x-data-plane=off,drive=disk1,scsi=off \
> >> -kernel $KERNEL -append "root=/dev/sdb1 console=tty0" \
> >> -L /tmp/qemu-dataplane/share/qemu/ -nographic -vnc :0 -enable-kvm -m
> >> 2048 -smp 4 -cpu qemu64,+x2apic -M pc
> > 
> > Was just about to send out the latest patch series which addresses
> > review comments, so I have tested the latest code
> > (61b70fef489ce51ecd18d69afb9622c110b9315c).
> > 
> > I was unable to reproduce a ramdisk performance regression on Linux
> > 3.6.6-3.fc18.x86_64 with Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz with
> > 8 GB RAM.
> 
> I am using the latest upstream kernel.
> 
> > The ramdisk is 4 GB and I used your QEMU command-line with a RHEL 6.3 guest.
> > 
> > Summary results:
> > x-data-plane-on: iops=132856 aggrb=1039.1MB/s
> > x-data-plane-off: iops=126236 aggrb=988.40MB/s
> > 
> > virtio-blk-data-plane is ~5% faster in this benchmark.
> > 
> > fio jobfile:
> > [global]
> > filename=/dev/vda
> > blocksize=8k
> > ioengine=libaio
> > direct=1
> > iodepth=8
> > runtime=120
> > time_based=1
> > 
> > [reads]
> > readwrite=randread
> > numjobs=4
> > 
> > Perf top (data-plane-on):
> >   3.71%  [kvm]               [k] kvm_arch_vcpu_ioctl_run
> >   3.27%  [kernel]            [k] memset    <--- ramdisk
> >   2.98%  [kernel]            [k] do_blockdev_direct_IO
> >   2.82%  [kvm_intel]         [k] vmx_vcpu_run
> >   2.66%  [kernel]            [k] _raw_spin_lock_irqsave
> >   2.06%  [kernel]            [k] put_compound_page
> >   2.06%  [kernel]            [k] __get_page_tail
> >   1.83%  [i915]              [k] __gen6_gt_force_wake_mt_get
> >   1.75%  [kernel]            [k] _raw_spin_unlock_irqrestore
> >   1.33%  qemu-system-x86_64  [.] vring_pop <--- virtio-blk-data-plane
> >   1.19%  [kernel]            [k] compound_unlock_irqrestore
> >   1.13%  [kernel]            [k] gup_huge_pmd
> >   1.11%  [kernel]            [k] __audit_syscall_exit
> >   1.07%  [kernel]            [k] put_page_testzero
> >   1.01%  [kernel]            [k] fget
> >   1.01%  [kernel]            [k] do_io_submit
> > 
> > Since the ramdisk (memset and page-related functions) is so prominent
> > in perf top, I also tried a 1-job 8k dd sequential write test on a
> > Samsung 830 Series SSD where virtio-blk-data-plane was 9% faster than
> > virtio-blk.  Optimizing against ramdisk isn't a good idea IMO because
> > it acts very differently from real hardware where the driver relies on
> > mmio, DMA, and interrupts (vs synchronous memcpy/memset).
> 
> For the memset in ramdisk, you can simply patch drivers/block/brd.c to
> do nop instead of memset for testing.
> 
> Yes, if you have fast SSD device （sometimes you need multiple which I
> do not have）, it makes more sense to test on real hardware. However,
> ramdisk test is still useful. It gives rough performance numbers. If A
> and B are both tested against ramdisk. The difference between A and B
> are still useful.

Optimizing the difference between A and B on ramdisk is only guaranteed
to optimize the ramdisk case.  On real hardware the bottleneck might be
elsewhere and we'd be chasing the wrong lead.

I don't think it's a waste of time but I think to stay healthy we need
to focus on real disks and SSDs most of the time.

Stefan