Issues with KVM

Issues with KVM

[Victor Stinner]

  Hi,

I have a new fast computer to run many virtual machines. Everything 
looks very fast, except the installation of new operating systems in 
KVM. The installation is very fast until it begins to write on disk. It 
looks like it writes slower and slower. I tried Debian, FreeBSD, 
OpenIndiana and OpenBSD: same problem. The FreeBSD installer displays 
the speed: it starts at 780 KB/sec (which it already very slow) to 
finish between 1 and 8 KB/sec.

darksatanic suggested me to use nodatacow mount flag: it is not faster, 
and it looks even slower (fewer wsec/s in iostat output, see below).

The computer is an Intel i7 2600 (4 cores with hyper threading: 8 
threads), 12 GB or RAM, 2 hard drives of 1 TB (Western Digital Caviar 
Blue 1 To 7200 RPM 32 Mo Serial ATA 6Gb/s - WD10EALX). Both disks are 
connected to SATA 6 GB/sec connectors using a P67 chipset. I'm using 
RAID 0 with Linux software (MD) RAID, and I have one unique btrfs 
partition of 2 TB. The host OS is Fedora 15 (Linux kernel 2.6.38).

I'm using hardware virtualisation with KVM. Debian is installed using 
virtio, so it should not be an issue with the hard drive driver of KVM.

I'm watching iostat during the Debian installation. With the default 
mount option, wsec/s starts at 49000 to finish near 42000 (on the MD 
device), %usage is greater than 50% of both disks (near 80% for sda, 
near 60% for sdb). Using nodatacow option, wsec/s starts at 12000 
(%usage > 75%) to finish near 10000 (%usage always > 75%). It is slower, 
right? A sector is 512 bytes. The Debian image size is 40 GB, its type 
is "raw". The system load is greater than 2 (or maybe 3) during the 
installation of the VM, while CPU usage is under 8% and wa% is also low 
(maybe 10% or lower, I don't remember).

bonnie++ output (on the Fedora host, not in a VM):

Version  1.96       ------Sequential Output------ --Sequential Input- 
--Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP 
/sec %CP
ned          24048M   346  98 220839  24 98489  19   245  84 251547  18 
199.2 259
Latency             37256us     326ms     943ms     251ms     197ms 
151ms
Version  1.96       ------Sequential Create------ --------Random 
Create--------
ned                 -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP 
/sec %CP
                  16 11128  34 +++++ +++ 16558  45 14006  40 +++++ +++ 
17226  48
Latency             14997us     663us   11401us    8115us     282us 
10105us
1.96,1.96,ned,1,1311365747,24048M,,346,98,220839,24,98489,19,245,84,251547,18,199.2,259,16,,,,,11128,34,+++++,+++,16558,45,14006,40,+++++,+++,17226,48,37256us,326ms,943ms,251ms,197ms,151ms,14997us,663us,11401us,8115us,282us,10105us

Do you have any idea why the %usage is so high (in iostat), while the 
speed looks so low? The disk speed during the installation is between 5 
MB/sec and 23 MB/sec, whereas the raw speed is greater than 200 MB/sec 
on the host (234 MB/sec according to hdparm -t /dev/md127, 220 MB/sec 
according to bonnie++ on sequential output).

It's difficult to read bonnie++ output: random create is near 14 MB/sec 
if I read correctly. btrfs behaves maybe very badly with a raw image of 
40 GB, especially on RAID 0 with 2 disks?

Should I try other KVM option (e.g. use another type of image)? Try 
btrfs RAID instead of Linux MD RAID? Try to disable some CPU cores? Or 
maybe not using btrfs for KVM images? :-)

Victor

Re: Issues with KVM

[C Anthony Risinger]

On Fri, Jul 22, 2011 at 2:44 PM, Victor Stinner
<victor.stinner@haypocalc.com> wrote:
> =C2=A0Hi,
>
> I have a new fast computer to run many virtual machines. Everything l=
ooks
> very fast, except the installation of new operating systems in KVM. T=
he
> installation is very fast until it begins to write on disk. It looks =
like it
> writes slower and slower. I tried Debian, FreeBSD, OpenIndiana and Op=
enBSD:
> same problem. The FreeBSD installer displays the speed: it starts at =
780
> KB/sec (which it already very slow) to finish between 1 and 8 KB/sec.

) is the host FS btrfs?
) are virtio modules in the initramfs (or kernel probably)?
) are you sure virtio is being used (eg. are the disks called vdX vs sd=
X)?
) is the disk bus set to virtio (virtmanager)?
) is the disk's cache mode set to none [or maybe writeback] (virtmanage=
r)?
) is the disk's storage format set to "raw", should be (virtmanager)?
) is caching enabled on the image? ()

probably need to change the cache mode on the disk, or if the host is
btrfs you need to flag the image with whetever is needed to prevent
continuous COWing.

C Anthony
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Issues with KVM

[C Anthony Risinger]

On Fri, Jul 22, 2011 at 2:59 PM, C Anthony Risinger <anthony@xtfx.me> wrote:
> On Fri, Jul 22, 2011 at 2:44 PM, Victor Stinner
> <victor.stinner@haypocalc.com> wrote:
>
> ) is caching enabled on the image? ()

oops, disregard that ... remainder left over from editing copy/paste :-)

C Anthony

Re: Issues with KVM

[Josef Bacik]

On 07/22/2011 03:44 PM, Victor Stinner wrote:
>  Hi,
> 
> I have a new fast computer to run many virtual machines. Everything
> looks very fast, except the installation of new operating systems in
> KVM. The installation is very fast until it begins to write on disk. It
> looks like it writes slower and slower. I tried Debian, FreeBSD,
> OpenIndiana and OpenBSD: same problem. The FreeBSD installer displays
> the speed: it starts at 780 KB/sec (which it already very slow) to
> finish between 1 and 8 KB/sec.
> 
> darksatanic suggested me to use nodatacow mount flag: it is not faster,
> and it looks even slower (fewer wsec/s in iostat output, see below).
> 
> The computer is an Intel i7 2600 (4 cores with hyper threading: 8
> threads), 12 GB or RAM, 2 hard drives of 1 TB (Western Digital Caviar
> Blue 1 To 7200 RPM 32 Mo Serial ATA 6Gb/s - WD10EALX). Both disks are
> connected to SATA 6 GB/sec connectors using a P67 chipset. I'm using
> RAID 0 with Linux software (MD) RAID, and I have one unique btrfs
> partition of 2 TB. The host OS is Fedora 15 (Linux kernel 2.6.38).
> 
> I'm using hardware virtualisation with KVM. Debian is installed using
> virtio, so it should not be an issue with the hard drive driver of KVM.
> 
> I'm watching iostat during the Debian installation. With the default
> mount option, wsec/s starts at 49000 to finish near 42000 (on the MD
> device), %usage is greater than 50% of both disks (near 80% for sda,
> near 60% for sdb). Using nodatacow option, wsec/s starts at 12000
> (%usage > 75%) to finish near 10000 (%usage always > 75%). It is slower,
> right? A sector is 512 bytes. The Debian image size is 40 GB, its type
> is "raw". The system load is greater than 2 (or maybe 3) during the
> installation of the VM, while CPU usage is under 8% and wa% is also low
> (maybe 10% or lower, I don't remember).
> 
> bonnie++ output (on the Fedora host, not in a VM):
> 
> Version  1.96       ------Sequential Output------ --Sequential Input-
> --Random-
> Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
> --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
> /sec %CP
> ned          24048M   346  98 220839  24 98489  19   245  84 251547  18
> 199.2 259
> Latency             37256us     326ms     943ms     251ms     197ms 151ms
> Version  1.96       ------Sequential Create------ --------Random
> Create--------
> ned                 -Create-- --Read--- -Delete-- -Create-- --Read---
> -Delete--
>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
> /sec %CP
>                  16 11128  34 +++++ +++ 16558  45 14006  40 +++++ +++
> 17226  48
> Latency             14997us     663us   11401us    8115us     282us 10105us
> 1.96,1.96,ned,1,1311365747,24048M,,346,98,220839,24,98489,19,245,84,251547,18,199.2,259,16,,,,,11128,34,+++++,+++,16558,45,14006,40,+++++,+++,17226,48,37256us,326ms,943ms,251ms,197ms,151ms,14997us,663us,11401us,8115us,282us,10105us
> 
> 
> Do you have any idea why the %usage is so high (in iostat), while the
> speed looks so low? The disk speed during the installation is between 5
> MB/sec and 23 MB/sec, whereas the raw speed is greater than 200 MB/sec
> on the host (234 MB/sec according to hdparm -t /dev/md127, 220 MB/sec
> according to bonnie++ on sequential output).
> 
> It's difficult to read bonnie++ output: random create is near 14 MB/sec
> if I read correctly. btrfs behaves maybe very badly with a raw image of
> 40 GB, especially on RAID 0 with 2 disks?
> 
> Should I try other KVM option (e.g. use another type of image)? Try
> btrfs RAID instead of Linux MD RAID? Try to disable some CPU cores? Or
> maybe not using btrfs for KVM images? :-)
> 

Use the kvm option of cache=none for your device.  Granted its still
going to be slow, but it should be a bit faster.  Thanks,

Josef

Re: Issues with KVM

[Morten P.D. Stevens]

On Fri, 22 Jul 2011 21:44:24 +0200, Victor Stinner wrote:
> Should I try other KVM option (e.g. use another type of image)? Try
> btrfs RAID instead of Linux MD RAID? Try to disable some CPU cores? 
> Or
> maybe not using btrfs for KVM images? :-)

Hi,

I would suggest you the following points:

- qemu-img create -f qcow2 -o size=40000000000,preallocation=metadata 
vdisk.img
- disk: cache=none
- controller: virtio

Best regards,

Morten

Re: Issues with KVM

[Victor Stinner]

Hi,

Le 22/07/2011 21:59, C Anthony Risinger a =C3=A9crit :
> ) is the host FS btrfs?
The host (Fedora) FS is btrfs, the VM (Debian) uses ext4 (or maybe ext3=
,=20
I will have to check).
> ) are virtio modules in the initramfs (or kernel probably)?
> ) are you sure virtio is being used (eg. are the disks called vdX vs =
sdX)?
> ) is the disk bus set to virtio (virtmanager)?
I'm not sure. I will have to check.
> ) is the disk's cache mode set to none [or maybe writeback] (virtmana=
ger)?
I don't remember, I will check.

What is the effect of this option? (What is the effect on the host FS=20
(btrfs)?)
> ) is the disk's storage format set to "raw", should be (virtmanager)?
Yes, the image type is "raw".
> ) is caching enabled on the image? ()
What do you mean? Is it possible to enable or disable the cache on a=20
specific file on the host (Fedora, btrfs)? Or you mean set cache to non=
e=20
in the configuration of the VM?
> probably need to change the cache mode on the disk, or if the host is
> btrfs you need to flag the image with whetever is needed to prevent
> continuous COWing.
Can nodatacow option of btrfs have an effect on this issue? Is it=20
possible to set this option for a specific file or directory?

Victor

PS: I have subscribed to the mailing list.

PS2: I'm not at home this week-end, I will check monday.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Issues with KVM

[Victor Stinner]

On 22/07/2011 21:59, C Anthony Risinger wrote:
 > is the disk's cache mode set to none [or maybe writeback]
 > (virtmanager)?

I tested various options: the most important is the cache mode.

cache=none is a little bit better than cache=default (writethrough), but 
there is an huge difference using cache=unsafe. This option is not 
supported by virt-manager, but by kvm. Using cache=unsafe, the 
installation of FreeBSD is ~100x faster:

  - cache=default: write at 1 to 8 KB/sec
  - cache=none: write at 40 KB/sec
  - cache=unsafe: write at 1200 KB/sec

According to agraf__ on IRC (#kvm on FreeNode), the cache mode has the 
following effect:

  - cache=writethrough calls fsync() after every write()
  - cache=none uses O_DIRECT
  - cache=writeback calls fsync() when the guest issues a barrier() 
(don't use O_DIRECT)
  - cache=unsafe doesn't do any fsync() (but don't use O_DIRECT)

I still have to test writeback ;-)

I found another mail thread, of last summer (July 2010), which is 
exactly the same problem that I had: "BTRFS: Unbelievably slow with 
kvm/qemu" on this mailing list. It contains another advice:

"Make sure you build your file system with "mkfs.btrfs -m single -d 
single /dev/whatever". You may well be writing duplicate copies of 
everything."

http://lkml.org/lkml/2010/7/12/5

--

FreeBSD installation in VirtualBox is as fast (or maybe a little bit 
slower) than the installation in kvm using cache=unsafe. I suppose that 
VirtualBox uses something like cache=unsafe or cache=writeback.

Victor

Re: Issues with KVM

[Christoph Hellwig]

On Mon, Jul 25, 2011 at 04:04:49PM +0200, Victor Stinner wrote:
> According to agraf__ on IRC (#kvm on FreeNode), the cache mode has
> the following effect:
> 
>  - cache=writethrough calls fsync() after every write()

It uses O_DSYNC, which an be appromiately described as an fdatasync
after every write.  Not that it makes a different for most workloads on
btrfs.

>  - cache=none uses O_DIRECT
>  - cache=writeback calls fsync() when the guest issues a barrier()
> (don't use O_DIRECT)

Issue a cache flush.  Barriers were an Linux-internal concept that
is now gone.  And there never was a barriere() call.

> FreeBSD installation in VirtualBox is as fast (or maybe a little bit
> slower) than the installation in kvm using cache=unsafe. I suppose
> that VirtualBox uses something like cache=unsafe or cache=writeback.

What filesystem do you use in FreeBSD?  FFS traditionally never issued
cache flushes, so cache=writeback is equal to cache=unsafe for it.  ZFS
can issue cache flushes on Solaris, but I'm not sure if this was ported
correctly to FreeBSD.