KVM incompatible with multipath?

KVM incompatible with multipath?

[infernix]

Hi,

I've got an Equallogic iSCSI box with 3x1GBit nics on my storage
network. I also have a server with 4 nics of which 3 are connected to
the storage network with 3x1GBit.

On this host, open-iscsi is running and connecting to the SAN over 3 
interfaces. I'm seeing /dev/sdb, sdc and sdd, which are all the same 
harddisk (same WWID).

With multipath I combine them to one disk to get a nice boost in
performance. This works perfectly fine (and very speedy) on the host.
Relevant multipath.conf data:

device {
  vendor "EQLOGIC"
  product "100E-00"
  path_grouping_policy multibus
  getuid_callout "/lib/udev/scsi_id -g -u -s /block/%n"
  features "1 queue_if_no_path"
  path_checker readsector0
  failback immediate
  path_selector "round-robin 0"
  rr_weight uniform
  rr_min_io 128
}

The problem with KVM (version 87) is that KVM cannot make use of the
multipath device in /dev/mapper.

If I boot a debootstrapped Debian Lenny KVM guest and configure it with
-hda /dev/mapper/my-multipathed-iscsi-disk, I cannot mount the root
partition. It'll complain about "Invalid target".

If I use -hda /dev/sdb or sdc or sdd (e.g. any of the multipathed disks,
which again are all the same but just through 3 separate connections),
everything works perfectly fine. Note that the behavior is the same when
using virtio.


I have tried to put LVM on top of the multipath device in the host, and
then run the KVM guest with -hda /dev/volumegroup/logicalvolume-blah but
that leads to serious data corruption. If I mount one of the lvm
partitions on the host, copy data to it, then unmount it and boot the
KVM guest, the data isn't there. If I shutdown the guest and do an
e2fsck on the host, I get lots of missing/unlinked inodes and a lot of
it is moved to /lost+found.


Does anyone have experience with KVM on multipathed disks? I'm really
puzzled here because everything works perfectly fine from the hosts
perspective.

Thanks for any help!

Regards,

infernix

Re: KVM incompatible with multipath?

[Anthony Liguori]

infernix wrote:
> Hi,
>
> I've got an Equallogic iSCSI box with 3x1GBit nics on my storage
> network. I also have a server with 4 nics of which 3 are connected to
> the storage network with 3x1GBit.
>
> On this host, open-iscsi is running and connecting to the SAN over 3 
> interfaces. I'm seeing /dev/sdb, sdc and sdd, which are all the same 
> harddisk (same WWID).
>
> With multipath I combine them to one disk to get a nice boost in
> performance. This works perfectly fine (and very speedy) on the host.
> Relevant multipath.conf data:
>
> device {
>  vendor "EQLOGIC"
>  product "100E-00"
>  path_grouping_policy multibus
>  getuid_callout "/lib/udev/scsi_id -g -u -s /block/%n"
>  features "1 queue_if_no_path"
>  path_checker readsector0
>  failback immediate
>  path_selector "round-robin 0"
>  rr_weight uniform
>  rr_min_io 128
> }
>
> The problem with KVM (version 87) is that KVM cannot make use of the
> multipath device in /dev/mapper.
>
> If I boot a debootstrapped Debian Lenny KVM guest and configure it with
> -hda /dev/mapper/my-multipathed-iscsi-disk, I cannot mount the root
> partition. It'll complain about "Invalid target".

How do you get the disk image onto /dev/mapper/my-multipathed-iscsi-disk?

You need to create a partition table and setup grub in order to be able 
to use something as -hda.  You don't get that automatically with 
debootstrap.

Regards,

Anthony Liguori

Re: KVM incompatible with multipath?

[infernix]

Anthony Liguori wrote:
> You need to create a partition table and setup grub in order to be able 
> to use something as -hda.  You don't get that automatically with 
> debootstrap.

Although I didn't include that in my mail, I did configure partitions 
and debootstrapped Lenny on part1. But that's not what's important 
anymore; this post kind of grew into a performance test of virtio block 
and net for Xen 3.2.1 and KVM 87, specifically when using multipath IO 
to an Equallogic iSCSI box.

I made this work as a Xen 3.2.1 domU on the same box and ran some 
performance tests. Afterwards I retried KVM and I didn't experience the 
problems I had before, for reasons unknown.

I boot the same multipathed disk that I used for Xen with 2.6.27.25 (+ 
kvm-87 modules in initrd).




It actually boots now and I have no problems whatsoever. Note that this 
kernel I built works for xen domU, Linux native and for KVM guest.

I've ran some bonnie++ tests, see below for the results.

Whats interesting in these results is that KVM guest has much lower 
sequential block output than on the host kernel, but much better 
sequential input. The latter is probably due to caching and buffers in 
the KVM host kernel. Setting cache=writeback improves both, but still 
block output is ~75MB/sec slower than on the host.

It seems that KVM guest write performance is CPU limited. Any advice on 
how to get better write speeds is highly appreciated.

In any case, so far I'm unable to reproduce the data corruption. 
Probably some glitch in the matrix.





Here's bonnie++ output with Xen domU (kernel 2.6.27.25) which has the 
multipathed 36090a0383049a2ac41a4643f000070c2-part1 configured as root 
disk xvda1:

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
77918  98 115400  18 56323   8 60034  69 122466   7 422.4   0

Here's bonnie++ output with KVM guest (kernel 2.6.27.25) using virtio:

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
59461  95 97238  17 60831  12 62181  95 205094  25 656.4   2

Here's host performance with Xen dom0 (kernel 2.6.26-2-amd64 from lenny):

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
66519  93 172569  50 85457  35 59671  86 164754  40 451.8   0

And here's native performance with 2.6.27.25 (no xen):

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
71796  98 182818  53 85511  29 61484  79 165302  31 668.4   1

The above tests were performed with blockdev --setra 16384 and MTU 1500.


Here's native Linux with jumbo frames:

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
73616  99 195577  48 96794  27 68845  84 201899  29 630.3   1

Here's KVM guest performance (with jumbo frames in the hosts iscsi 
interfaces), cache=writethrough:

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
60814  96 93222  15 64166  12 58015  94 258557  31 649.3   2

Here's KVM guest performance (with jumbo frames in the hosts iscsi 
interfaces), cache=writeback and bonnie size=2.5 times host RAM:

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
52627  95 120630  23 100333  22 61271  94 284889  37 464.6   2


Xen domU with jumbo in dom0:

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
76316  97 116028  19 58278   9 60066  71 131953   9 282.8   0



Now for something different, iscsi multipath inside xen domU (e.g. domU 
gets 3 nics, each bridged to one of the nics on dom0):

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
64011  97 196484  49 91937  33 54583  81 160031  33 531.6   0

KVM guest with iscsi multipath, same bridging setup with 3 tap devices 
but *NO* jumbo frames:

Re: KVM incompatible with multipath?

[infernix]

Whoops, please disregard the previous mail; I hit ctrl-enter while still 
composing, sorry.

Anthony Liguori wrote:
 > You need to create a partition table and setup grub in order to be 
able to use something as -hda.  You don't get that automatically with 
debootstrap.

Although I didn't include that in my mail, I did configure partitions 
and debootstrapped Lenny on part1. But that's not what's important 
anymore, so far I'm unable to reproduce the data corruption. Probably 
some glitch in the matrix.

This post kind of grew into a performance test of virtio block and net 
for Xen 3.2.1 and KVM 87, specifically when using multipath IO to an 
Equallogic iSCSI box.

I added Xen 3.2.1 to the same box, installed a domU and ran some 
performance tests. Afterwards I retried KVM and I didn't experience the 
problems I had before. I boot the same multipathed disk that I used for 
Xen with 2.6.27.25 (+ kvm-87 modules in initrd). You can find the script 
below. It actually boots now and I have no problems whatsoever. Note 
that this kernel I built works for xen domU, Linux native and for KVM guest.

I've ran some bonnie++ tests, see below for the results.

At first I had thought that using iSCSI and multipath on the host, 
whether KVM or Xen, would be the fastest. So I ran a bunch of tests 
against that. Whats interesting in these results is that KVM guest has 
much lower sequential block output than on the host kernel, but much 
better sequential input. The latter is probably due to caching and 
buffers in the KVM host kernel. Setting cache=writeback improves both, 
but still block output is ~75MB/sec slower than on the host. It seems 
that KVM guest write performance is CPU limited. Any advice on how to 
get better write speeds is highly appreciated.


Afterwards, I decided to try multipath+iscsi in the guest. It turns out 
that Xen is the big winner in the end, but with a catch. The highest 
performance was measured when using iscsi+multipath in the Xen domU with 
jumbo frames. Thus the rest of the results with the disk being mapped in 
the host aren't relevant at this point.

Iscsi multipath to an Equallogic box with 3x1gbit, inside a xen domU 
(e.g. domU gets 3 nics, each bridged to one of the nics on dom0):

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
64011  97 196484  49 91937  33 54583  81 160031  33 531.6   0

KVM guest with iscsi multipath, same bridging setup with 3 tap devices 
but *NO* jumbo frames:
------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
48326  72 104770  39 57539  26 42610  79 123867  43 547.5   1


This is pretty good performance for Xen, and at 1500 byte MTU it's 
definitely not bad for KVM either. The catch, however, is that I was 
unable to turn on Jumbo frames in the KVM guest. I've applied the patch 
in https://bugzilla.redhat.com/show_bug.cgi?id=473114 to the guest 
kernel and changed some 4096 byte long buffers in vl.c and net.c to 
16384, but I still couldn't configure the MTU on the guest to anything 
above 1500:

test:~# ifconfig eth3 mtu 1500
test:~# ifconfig eth3 mtu 1501
SIOCSIFMTU: Invalid argument

So my question is, how can I get virtio to play nice and accept a 9000 
byte MTU? I can then finish my performance comparison and perhaps extend 
it with iometer.


Thanks!

Regards, infernix





bonnie++ output with Xen domU (kernel 2.6.27.25) which has the 
multipathed disk configured as root disk xvda1:

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
77918  98 115400  18 56323   8 60034  69 122466   7 422.4   0

Here's bonnie++ output with KVM guest (kernel 2.6.27.25) using virtio 
disk vda1 mapped to /dev/mapper/multipath disk:

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
59461  95 97238  17 60831  12 62181  95 205094  25 656.4   2

Here's host performance with Xen dom0 (kernel 2.6.26-2-amd64 from lenny) 
directly on the /dev/mapper/multipath disk:

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
66519  93 172569  50 85457  35 59671  86 164754  40 451.8   0

And here's native performance with 2.6.27.25 (no xen) directly on the 
/dev/mapper/multipath disk:

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
71796  98 182818  53 85511  29 61484  79 165302  31 668.4   1

The above tests were performed with blockdev --setra 16384 and MTU 1500.


Here's native Linux host with jumbo frames doing direct io on 
/dev/mapper/multipath disk:

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
73616  99 195577  48 96794  27 68845  84 201899  29 630.3   1

Here's KVM guest performance (with jumbo frames in the hosts iscsi 
interfaces), cache=writethrough with the multipath disk as /dev/vda1:

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
60814  96 93222  15 64166  12 58015  94 258557  31 649.3   2

Here's KVM guest performance (with jumbo frames in the hosts iscsi 
interfaces), cache=writeback and bonnie size=2.5*host RAM with the 
multipath disk as /dev/vda1:

------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
52627  95 120630  23 100333  22 61271  94 284889  37 464.6   2

Xen domU (kernel 2.6.27.25) which has the multipathed disk configured as 
root disk xvda1 + jumbo frames in dom0:
------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
76316  97 116028  19 58278   9 60066  71 131953   9 282.8   0



KVM guest script:

#!/bin/sh
( sleep 2s; for i in 0 4 5 6; do brctl addif br$i tap$i; ifconfig tap$i 
0.0.0.0 up promisc; ifconfig tap$i mtu 9000; done ) &
/usr/local/kvm/bin/qemu-system-x86_64 -m 2048 -localtime -curses \
-net nic,model=virtio,vlan=0 -net tap,vlan=0,ifname=tap0,script=/bin/true \
-net nic,model=virtio,vlan=4,macaddr=00:16:42:51:34:a0 -net 
tap,vlan=4,ifname=tap4,script=/bin/true \
-net nic,model=virtio,vlan=5,macaddr=00:16:42:51:34:a1 -net 
tap,vlan=5,ifname=tap5,script=/bin/true \
-net nic,model=virtio,vlan=6,macaddr=00:16:42:51:34:a2 -net 
tap,vlan=6,ifname=tap6,script=/bin/true \
-initrd  /boot/initrd.img-2.6.27.25-001.core2 \
-kernel /boot/vmlinuz-2.6.27.25-001.core2 \
-append 'root=/dev/vda1' \
-drive 
file=/dev/mapper/36090a0383049a2ac41a4643f000070c2,if=virtio,boot=on,cache=writeback