From mboxrd@z Thu Jan  1 00:00:00 1970
From: "Brian J. Murrell" <brian@interlinx.bc.ca>
Subject: 5x slower guest disk performance with virtio disk
Date: Thu, 15 Dec 2011 09:28:55 -0500
Message-ID: <jcd078$n2s$1@dough.gmane.org>
Mime-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
 protocol="application/pgp-signature";
 boundary="------------enig835425252F0BBF5B9F24BF02"
To: kvm@vger.kernel.org
Return-path: <kvm-owner@vger.kernel.org>
Received: from lo.gmane.org ([80.91.229.12]:39773 "EHLO lo.gmane.org"
	rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
	id S1751395Ab1LOOpU (ORCPT <rfc822;kvm@vger.kernel.org>);
	Thu, 15 Dec 2011 09:45:20 -0500
Received: from list by lo.gmane.org with local (Exim 4.69)
	(envelope-from <gcekd-kvm-devel@m.gmane.org>)
	id 1RbCYh-0000x7-6w
	for kvm@vger.kernel.org; Thu, 15 Dec 2011 15:45:19 +0100
Received: from d67-193-214-242.home3.cgocable.net ([67.193.214.242])
        by main.gmane.org with esmtp (Gmexim 0.1 (Debian))
        id 1AlnuQ-0007hv-00
        for <kvm@vger.kernel.org>; Thu, 15 Dec 2011 15:45:19 +0100
Received: from brian by d67-193-214-242.home3.cgocable.net with local (Gmexim 0.1 (Debian))
        id 1AlnuQ-0007hv-00
        for <kvm@vger.kernel.org>; Thu, 15 Dec 2011 15:45:19 +0100
Sender: kvm-owner@vger.kernel.org
List-ID: <kvm.vger.kernel.org>

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--------------enig835425252F0BBF5B9F24BF02
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I have a CentOS 6 host system running a CentOS 6 KVM guest and the
guest seems to get about 5x slower disk throughput than the host:

host $ sudo dd of=3D/dev/datavol/disk1 if=3D/dev/zero bs=3D1M oflag=3Ddir=
ect count=3D50000
50000+0 records in
50000+0 records out
52428800000 bytes (52 GB) copied, 150.36 s, 349 MB/s
host $ ssh guest dd of=3D/dev/vdb if=3D/dev/zero bs=3D1M oflag=3Ddirect c=
ount=3D50000
50000+0 records in
50000+0 records out
52428800000 bytes (52 GB) copied, 731.007 s, 71.7 MB/s

This is actually a vast improvement over earlier tests where it was a
factor of 40x difference.  But even still, I thought virtio disks were
supposed to be much closer to native host speed than 5x.

/dev/datavol/disk1 is a RAIDed LV in an LVM volume group consisting of 5 =
disks:

$ sudo pvs
  PV         VG      Fmt  Attr PSize   PFree =20
  /dev/sda3          lvm2 a-   420.16g 420.16g
  /dev/sdb   datavol lvm2 a-   465.76g 227.66g
  /dev/sdc   datavol lvm2 a-   465.76g 245.76g
  /dev/sdd   datavol lvm2 a-   465.76g 245.76g
  /dev/sde   datavol lvm2 a-   465.76g 245.76g
  /dev/sdf   datavol lvm2 a-   465.76g 245.76g
$ sudo lvdisplay --maps /dev/datavol/disk1
  --- Logical volume ---
  LV Name                /dev/datavol/disk1
  VG Name                datavol
  LV UUID                3gD1N3-ybAU-GzUO-8QBV-b6op-lsK9-GMNm3w
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                500.00 GiB
  Current LE             128000
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1
  =20
  --- Segments ---
  Logical extent 0 to 127999:
    Type		striped
    Stripes		5
    Stripe size		4.00 KiB
    Stripe 0:
      Physical volume	/dev/sdb
      Physical extents	5120 to 30719
    Stripe 1:
      Physical volume	/dev/sdc
      Physical extents	5120 to 30719
    Stripe 2:
      Physical volume	/dev/sdd
      Physical extents	5120 to 30719
    Stripe 3:
      Physical volume	/dev/sde
      Physical extents	5120 to 30719
    Stripe 4:
      Physical volume	/dev/sdf
      Physical extents	5120 to 30719

The KVM command line is as follows:

$ tr '\0' '\n' < /proc/9409/cmdline=20
/usr/libexec/qemu-kvm
-S
-M
rhel6.0.0
-enable-kvm
-m
1024
-smp
1,sockets=3D1,cores=3D1,threads=3D1
-name
guest
-uuid
e2bf97e0-cfb7-444c-abc3-9efe262d8efe
-nodefconfig
-nodefaults
-chardev
socket,id=3Dmonitor,path=3D/var/lib/libvirt/qemu/guest.monitor,server,now=
ait
-mon
chardev=3Dmonitor,mode=3Dcontrol
-rtc
base=3Dutc
-boot
c
-drive
file=3D/var/lib/libvirt/images/cdrom.iso,if=3Dnone,media=3Dcdrom,id=3Ddri=
ve-ide0-1-0,readonly=3Don,format=3Draw
-device
ide-drive,bus=3Dide.1,unit=3D0,drive=3Ddrive-ide0-1-0,id=3Dide0-1-0
-drive
file=3D/var/lib/libvirt/images/guest.qcow2,if=3Dnone,id=3Ddrive-virtio-di=
sk0,boot=3Don,format=3Dqcow2
-device
virtio-blk-pci,bus=3Dpci.0,addr=3D0x4,drive=3Ddrive-virtio-disk0,id=3Dvir=
tio-disk0
-drive
file=3D/dev/datavol/disk1,if=3Dnone,id=3Ddrive-virtio-disk1,format=3Draw
-device
virtio-blk-pci,bus=3Dpci.0,addr=3D0x6,drive=3Ddrive-virtio-disk1,id=3Dvir=
tio-disk1
-netdev
tap,fd=3D23,id=3Dhostnet0
-device
virtio-net-pci,netdev=3Dhostnet0,id=3Dnet0,mac=3D52:54:00:ec:8d:47,bus=3D=
pci.0,addr=3D0x3
-chardev
pty,id=3Dserial0
-device
isa-serial,chardev=3Dserial0
-usb
-vnc
127.0.0.1:2
-vga
cirrus
-device
virtio-balloon-pci,id=3Dballoon0,bus=3Dpci.0,addr=3D0x5

That is started from virt-manager given the following configuration:

<domain type=3D'kvm' id=3D'18'>
  <name>guest</name>
  <uuid>e2bf97e0-cfb7-444c-abc3-9efe262d8efe</uuid>
  <memory>1048576</memory>
  <currentMemory>1048576</currentMemory>
  <vcpu>1</vcpu>
  <os>
    <type arch=3D'x86_64' machine=3D'rhel6.0.0'>hvm</type>
    <boot dev=3D'hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset=3D'utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type=3D'file' device=3D'cdrom'>
      <driver name=3D'qemu' type=3D'raw'/>
      <source file=3D'/var/lib/libvirt/images/cdrom.iso'/>
      <target dev=3D'hdc' bus=3D'ide'/>
      <readonly/>
      <alias name=3D'ide0-1-0'/>
      <address type=3D'drive' controller=3D'0' bus=3D'1' unit=3D'0'/>
    </disk>
    <disk type=3D'file' device=3D'disk'>
      <driver name=3D'qemu' type=3D'qcow2'/>
      <source file=3D'/var/lib/libvirt/images/guest.qcow2'/>
      <target dev=3D'vda' bus=3D'virtio'/>
      <alias name=3D'virtio-disk0'/>
      <address type=3D'pci' domain=3D'0x0000' bus=3D'0x00' slot=3D'0x04' =
function=3D'0x0'/>
    </disk>
    <disk type=3D'block' device=3D'disk'>
      <driver name=3D'qemu' type=3D'raw'/>
      <source dev=3D'/dev/datavol/disk1'/>
      <target dev=3D'vdb' bus=3D'virtio'/>
      <alias name=3D'virtio-disk1'/>
      <address type=3D'pci' domain=3D'0x0000' bus=3D'0x00' slot=3D'0x06' =
function=3D'0x0'/>
    </disk>
    <controller type=3D'ide' index=3D'0'>
      <alias name=3D'ide0'/>
      <address type=3D'pci' domain=3D'0x0000' bus=3D'0x00' slot=3D'0x01' =
function=3D'0x1'/>
    </controller>
    <interface type=3D'network'>
      <mac address=3D'52:54:00:ec:8d:47'/>
      <source network=3D'default'/>
      <target dev=3D'vnet2'/>
      <model type=3D'virtio'/>
      <alias name=3D'net0'/>
      <address type=3D'pci' domain=3D'0x0000' bus=3D'0x00' slot=3D'0x03' =
function=3D'0x0'/>
    </interface>
    <serial type=3D'pty'>
      <source path=3D'/dev/pts/3'/>
      <target port=3D'0'/>
      <alias name=3D'serial0'/>
    </serial>
    <console type=3D'pty' tty=3D'/dev/pts/3'>
      <source path=3D'/dev/pts/3'/>
      <target port=3D'0'/>
      <alias name=3D'serial0'/>
    </console>
    <input type=3D'mouse' bus=3D'ps2'/>
    <graphics type=3D'vnc' port=3D'5902' autoport=3D'yes'/>
    <video>
      <model type=3D'cirrus' vram=3D'9216' heads=3D'1'/>
      <alias name=3D'video0'/>
      <address type=3D'pci' domain=3D'0x0000' bus=3D'0x00' slot=3D'0x02' =
function=3D'0x0'/>
    </video>
    <memballoon model=3D'virtio'>
      <alias name=3D'balloon0'/>
      <address type=3D'pci' domain=3D'0x0000' bus=3D'0x00' slot=3D'0x05' =
function=3D'0x0'/>
    </memballoon>
  </devices>
</domain>

Any ideas why the huge performance difference?

Cheers,
b.


--------------enig835425252F0BBF5B9F24BF02
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk7qBCcACgkQl3EQlGLyuXAZzwCghPL5iI98QKhZOvdG6EfUXl5F
Q/IAn01eGPWPScXNpDI93rtQaY8D0Czr
=ktCq
-----END PGP SIGNATURE-----

--------------enig835425252F0BBF5B9F24BF02--