Re: KVM incompatible with multipath?

[infernix <infernix@infernix.net>]

Anthony Liguori wrote:
> You need to create a partition table and setup grub in order to be able 
> to use something as -hda.  You don't get that automatically with 
> debootstrap.

Although I didn't include that in my mail, I did configure partitions 
and debootstrapped Lenny on part1. But that's not what's important 
anymore; this post kind of grew into a performance test of virtio block 
and net for Xen 3.2.1 and KVM 87, specifically when using multipath IO 
to an Equallogic iSCSI box.

I made this work as a Xen 3.2.1 domU on the same box and ran some 
performance tests. Afterwards I retried KVM and I didn't experience the 
problems I had before, for reasons unknown.

I boot the same multipathed disk that I used for Xen with 2.6.27.25 (+ 
kvm-87 modules in initrd).




It actually boots now and I have no problems whatsoever. Note that this 
kernel I built works for xen domU, Linux native and for KVM guest.

I've ran some bonnie++ tests, see below for the results.

Whats interesting in these results is that KVM guest has much lower 
sequential block output than on the host kernel, but much better 
sequential input. The latter is probably due to caching and buffers in 
the KVM host kernel. Setting cache=writeback improves both, but still 
block output is ~75MB/sec slower than on the host.

It seems that KVM guest write performance is CPU limited. Any advice on 
how to get better write speeds is highly appreciated.

In any case, so far I'm unable to reproduce the data corruption. 
Probably some glitch in the matrix.





Here's bonnie++ output with Xen domU (kernel 2.6.27.25) which has the 
multipathed 36090a0383049a2ac41a4643f000070c2-part1 configured as root 
disk xvda1:

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
77918  98 115400  18 56323   8 60034  69 122466   7 422.4   0

Here's bonnie++ output with KVM guest (kernel 2.6.27.25) using virtio:

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
59461  95 97238  17 60831  12 62181  95 205094  25 656.4   2

Here's host performance with Xen dom0 (kernel 2.6.26-2-amd64 from lenny):

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
66519  93 172569  50 85457  35 59671  86 164754  40 451.8   0

And here's native performance with 2.6.27.25 (no xen):

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
71796  98 182818  53 85511  29 61484  79 165302  31 668.4   1

The above tests were performed with blockdev --setra 16384 and MTU 1500.


Here's native Linux with jumbo frames:

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
73616  99 195577  48 96794  27 68845  84 201899  29 630.3   1

Here's KVM guest performance (with jumbo frames in the hosts iscsi 
interfaces), cache=writethrough:

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
60814  96 93222  15 64166  12 58015  94 258557  31 649.3   2

Here's KVM guest performance (with jumbo frames in the hosts iscsi 
interfaces), cache=writeback and bonnie size=2.5 times host RAM:

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
52627  95 120630  23 100333  22 61271  94 284889  37 464.6   2


Xen domU with jumbo in dom0:

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
76316  97 116028  19 58278   9 60066  71 131953   9 282.8   0



Now for something different, iscsi multipath inside xen domU (e.g. domU 
gets 3 nics, each bridged to one of the nics on dom0):

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
64011  97 196484  49 91937  33 54583  81 160031  33 531.6   0

KVM guest with iscsi multipath, same bridging setup with 3 tap devices 
but *NO* jumbo frames: