Re: KVM incompatible with multipath?

[infernix <infernix@infernix.net>]

Whoops, please disregard the previous mail; I hit ctrl-enter while still 
composing, sorry.

Anthony Liguori wrote:
 > You need to create a partition table and setup grub in order to be 
able to use something as -hda.  You don't get that automatically with 
debootstrap.

Although I didn't include that in my mail, I did configure partitions 
and debootstrapped Lenny on part1. But that's not what's important 
anymore, so far I'm unable to reproduce the data corruption. Probably 
some glitch in the matrix.

This post kind of grew into a performance test of virtio block and net 
for Xen 3.2.1 and KVM 87, specifically when using multipath IO to an 
Equallogic iSCSI box.

I added Xen 3.2.1 to the same box, installed a domU and ran some 
performance tests. Afterwards I retried KVM and I didn't experience the 
problems I had before. I boot the same multipathed disk that I used for 
Xen with 2.6.27.25 (+ kvm-87 modules in initrd). You can find the script 
below. It actually boots now and I have no problems whatsoever. Note 
that this kernel I built works for xen domU, Linux native and for KVM guest.

I've ran some bonnie++ tests, see below for the results.

At first I had thought that using iSCSI and multipath on the host, 
whether KVM or Xen, would be the fastest. So I ran a bunch of tests 
against that. Whats interesting in these results is that KVM guest has 
much lower sequential block output than on the host kernel, but much 
better sequential input. The latter is probably due to caching and 
buffers in the KVM host kernel. Setting cache=writeback improves both, 
but still block output is ~75MB/sec slower than on the host. It seems 
that KVM guest write performance is CPU limited. Any advice on how to 
get better write speeds is highly appreciated.


Afterwards, I decided to try multipath+iscsi in the guest. It turns out 
that Xen is the big winner in the end, but with a catch. The highest 
performance was measured when using iscsi+multipath in the Xen domU with 
jumbo frames. Thus the rest of the results with the disk being mapped in 
the host aren't relevant at this point.

Iscsi multipath to an Equallogic box with 3x1gbit, inside a xen domU 
(e.g. domU gets 3 nics, each bridged to one of the nics on dom0):

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
64011  97 196484  49 91937  33 54583  81 160031  33 531.6   0

KVM guest with iscsi multipath, same bridging setup with 3 tap devices 
but *NO* jumbo frames:
------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
48326  72 104770  39 57539  26 42610  79 123867  43 547.5   1


This is pretty good performance for Xen, and at 1500 byte MTU it's 
definitely not bad for KVM either. The catch, however, is that I was 
unable to turn on Jumbo frames in the KVM guest. I've applied the patch 
in https://bugzilla.redhat.com/show_bug.cgi?id=473114 to the guest 
kernel and changed some 4096 byte long buffers in vl.c and net.c to 
16384, but I still couldn't configure the MTU on the guest to anything 
above 1500:

test:~# ifconfig eth3 mtu 1500
test:~# ifconfig eth3 mtu 1501
SIOCSIFMTU: Invalid argument

So my question is, how can I get virtio to play nice and accept a 9000 
byte MTU? I can then finish my performance comparison and perhaps extend 
it with iometer.


Thanks!

Regards, infernix





bonnie++ output with Xen domU (kernel 2.6.27.25) which has the 
multipathed disk configured as root disk xvda1:

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
77918  98 115400  18 56323   8 60034  69 122466   7 422.4   0

Here's bonnie++ output with KVM guest (kernel 2.6.27.25) using virtio 
disk vda1 mapped to /dev/mapper/multipath disk:

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
59461  95 97238  17 60831  12 62181  95 205094  25 656.4   2

Here's host performance with Xen dom0 (kernel 2.6.26-2-amd64 from lenny) 
directly on the /dev/mapper/multipath disk:

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
66519  93 172569  50 85457  35 59671  86 164754  40 451.8   0

And here's native performance with 2.6.27.25 (no xen) directly on the 
/dev/mapper/multipath disk:

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
71796  98 182818  53 85511  29 61484  79 165302  31 668.4   1

The above tests were performed with blockdev --setra 16384 and MTU 1500.


Here's native Linux host with jumbo frames doing direct io on 
/dev/mapper/multipath disk:

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
73616  99 195577  48 96794  27 68845  84 201899  29 630.3   1

Here's KVM guest performance (with jumbo frames in the hosts iscsi 
interfaces), cache=writethrough with the multipath disk as /dev/vda1:

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
60814  96 93222  15 64166  12 58015  94 258557  31 649.3   2

Here's KVM guest performance (with jumbo frames in the hosts iscsi 
interfaces), cache=writeback and bonnie size=2.5*host RAM with the 
multipath disk as /dev/vda1:

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
52627  95 120630  23 100333  22 61271  94 284889  37 464.6   2

Xen domU (kernel 2.6.27.25) which has the multipathed disk configured as 
root disk xvda1 + jumbo frames in dom0:
------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
76316  97 116028  19 58278   9 60066  71 131953   9 282.8   0



KVM guest script:

#!/bin/sh
( sleep 2s; for i in 0 4 5 6; do brctl addif br$i tap$i; ifconfig tap$i 
0.0.0.0 up promisc; ifconfig tap$i mtu 9000; done ) &
/usr/local/kvm/bin/qemu-system-x86_64 -m 2048 -localtime -curses \
-net nic,model=virtio,vlan=0 -net tap,vlan=0,ifname=tap0,script=/bin/true \
-net nic,model=virtio,vlan=4,macaddr=00:16:42:51:34:a0 -net 
tap,vlan=4,ifname=tap4,script=/bin/true \
-net nic,model=virtio,vlan=5,macaddr=00:16:42:51:34:a1 -net 
tap,vlan=5,ifname=tap5,script=/bin/true \
-net nic,model=virtio,vlan=6,macaddr=00:16:42:51:34:a2 -net 
tap,vlan=6,ifname=tap6,script=/bin/true \
-initrd  /boot/initrd.img-2.6.27.25-001.core2 \
-kernel /boot/vmlinuz-2.6.27.25-001.core2 \
-append 'root=/dev/vda1' \
-drive 
file=/dev/mapper/36090a0383049a2ac41a4643f000070c2,if=virtio,boot=on,cache=writeback