[Qemu-devel] extremely low IOPS performance of QCOW2 image format on an SSD RAID1

[Qemu-devel] extremely low IOPS performance of QCOW2 image format on an SSD RAID1

[lihuiba]

[-- Attachment #1: Type: text/plain, Size: 1816 bytes --]

Hi, all


I'm using a qcow2 image stored on a SSD RAID1 (2 x intel S3500), and I'm benchmarking the
system using fio. Although the throughput in VM (with KVM and virtio enabled) is acceptable (67%
of thoughtput in host), the IOPS performance seems is extremely low ---- only 2% of IOPS in host.


I was initially using qemu-1.1.2, and I also tried qemu-1.7.1 for comparison. There was no significant
difference.


In contrast, raw image and LVM perform very well. They usually achieve 90%+ of throughput and
60%+ of IOPS. So the problem must lie in the QCOW2 image format.


And I observed that, when I perform 4KB IOPS benchmark in VM with a QCOW2 image, fio in VM reports
it is reading 9.x MB/s, while iostat in host reports the SSD is being read 150+ MB/s. So QEMU or QCOW2
must have amplified the amount of read by nearly 16 times.


So, how can I fix or tune the performance issue of qcow2?


Thanks!




PS:
1. qemu parameters:
-enable-kvm -cpu qemu64 -rtc base=utc,clock=host,driftfix=none -usb -device usb-tablet -nodefaults -nodefconfig -no-kvm-pit-reinjection -global kvm-pit.lost_tick_policy=discard -machine pc,accel=kvm -vga std -k en-us -smp 8 -m 4096 -boot order=cdn -vnc :1 -drive file=$1,if=none,id=drive_0,cache=none,aio=native -device virtio-blk-pci,drive=drive_0,bus=pci.0,addr=0x5 -drive file=$2,if=none,id=drive_2,cache=none,aio=native -device virtio-blk-pci,drive=drive_2,bus=pci.0,addr=0x7


2. fio parameters for IOPS:
fio --filename=/dev/vdb --direct=1 --ioengine=libaio --iodepth 32 --thread --numjobs=1 --rw=randread --bs=4k --size=100% --runtime=60s --group_reporting --name=test


3. fio parameters for throughput:
fio --filename=/dev/vdb--direct=1 --ioengine=psync --thread --numjobs=3 --rw=randread --bs=1024k --size=100% --runtime=60s --name=randread --group_reporting -name=test




[-- Attachment #2: Type: text/html, Size: 2255 bytes --]

Re: [Qemu-devel] extremely low IOPS performance of QCOW2 image format on an SSD RAID1

[Fam Zheng]

On Mon, 06/23 10:06, lihuiba wrote:
> Hi, all
> 
> 
> I'm using a qcow2 image stored on a SSD RAID1 (2 x intel S3500), and I'm benchmarking the
> system using fio. Although the throughput in VM (with KVM and virtio enabled) is acceptable (67%
> of thoughtput in host), the IOPS performance seems is extremely low ---- only 2% of IOPS in host.
> 
> 
> I was initially using qemu-1.1.2, and I also tried qemu-1.7.1 for comparison. There was no significant
> difference.
> 
> 
> In contrast, raw image and LVM perform very well. They usually achieve 90%+ of throughput and
> 60%+ of IOPS. So the problem must lie in the QCOW2 image format.
> 
> 
> And I observed that, when I perform 4KB IOPS benchmark in VM with a QCOW2 image, fio in VM reports
> it is reading 9.x MB/s, while iostat in host reports the SSD is being read 150+ MB/s. So QEMU or QCOW2
> must have amplified the amount of read by nearly 16 times.
> 
> 
> So, how can I fix or tune the performance issue of qcow2?

Did you prefill the image? Amplification could come from cluster allocation.

Fam

> 
> 
> Thanks!
> 
> 
> 
> 
> PS:
> 1. qemu parameters:
> -enable-kvm -cpu qemu64 -rtc base=utc,clock=host,driftfix=none -usb -device usb-tablet -nodefaults -nodefconfig -no-kvm-pit-reinjection -global kvm-pit.lost_tick_policy=discard -machine pc,accel=kvm -vga std -k en-us -smp 8 -m 4096 -boot order=cdn -vnc :1 -drive file=$1,if=none,id=drive_0,cache=none,aio=native -device virtio-blk-pci,drive=drive_0,bus=pci.0,addr=0x5 -drive file=$2,if=none,id=drive_2,cache=none,aio=native -device virtio-blk-pci,drive=drive_2,bus=pci.0,addr=0x7
> 
> 
> 2. fio parameters for IOPS:
> fio --filename=/dev/vdb --direct=1 --ioengine=libaio --iodepth 32 --thread --numjobs=1 --rw=randread --bs=4k --size=100% --runtime=60s --group_reporting --name=test
> 
> 
> 3. fio parameters for throughput:
> fio --filename=/dev/vdb--direct=1 --ioengine=psync --thread --numjobs=3 --rw=randread --bs=1024k --size=100% --runtime=60s --name=randread --group_reporting -name=test
> 
> 
> 

Re: [Qemu-devel] extremely low IOPS performance of QCOW2 image format on an SSD RAID1

[lihuiba]

[-- Attachment #1: Type: text/plain, Size: 2546 bytes --]

>Did you prefill the image? Amplification could come from cluster allocation.
Yes! 
I forgot to mention that I created the qcow2 image with 'preallocation=metadata', and I have allocated
the data blocks with dd in VM.


Creating image in host:
qemu-img create -f qcow2 -preallocation=metadata test.qcow2 100G


Allocating the blocks in VM:
dd if=/dev/zero of=/dev/vdb bs=1M
where vdb is the target image.







At 2014-06-23 11:01:20, "Fam Zheng" <famz@redhat.com> wrote:
>On Mon, 06/23 10:06, lihuiba wrote:
>> Hi, all
>> 
>> 
>> I'm using a qcow2 image stored on a SSD RAID1 (2 x intel S3500), and I'm benchmarking the
>> system using fio. Although the throughput in VM (with KVM and virtio enabled) is acceptable (67%
>> of thoughtput in host), the IOPS performance seems is extremely low ---- only 2% of IOPS in host.
>> 
>> 
>> I was initially using qemu-1.1.2, and I also tried qemu-1.7.1 for comparison. There was no significant
>> difference.
>> 
>> 
>> In contrast, raw image and LVM perform very well. They usually achieve 90%+ of throughput and
>> 60%+ of IOPS. So the problem must lie in the QCOW2 image format.
>> 
>> 
>> And I observed that, when I perform 4KB IOPS benchmark in VM with a QCOW2 image, fio in VM reports
>> it is reading 9.x MB/s, while iostat in host reports the SSD is being read 150+ MB/s. So QEMU or QCOW2
>> must have amplified the amount of read by nearly 16 times.
>> 
>> 
>> So, how can I fix or tune the performance issue of qcow2?
>
>Did you prefill the image? Amplification could come from cluster allocation.
>
>Fam
>
>> 
>> 
>> Thanks!
>> 
>> 
>> 
>> 
>> PS:
>> 1. qemu parameters:
>> -enable-kvm -cpu qemu64 -rtc base=utc,clock=host,driftfix=none -usb -device usb-tablet -nodefaults -nodefconfig -no-kvm-pit-reinjection -global kvm-pit.lost_tick_policy=discard -machine pc,accel=kvm -vga std -k en-us -smp 8 -m 4096 -boot order=cdn -vnc :1 -drive file=$1,if=none,id=drive_0,cache=none,aio=native -device virtio-blk-pci,drive=drive_0,bus=pci.0,addr=0x5 -drive file=$2,if=none,id=drive_2,cache=none,aio=native -device virtio-blk-pci,drive=drive_2,bus=pci.0,addr=0x7
>> 
>> 
>> 2. fio parameters for IOPS:
>> fio --filename=/dev/vdb --direct=1 --ioengine=libaio --iodepth 32 --thread --numjobs=1 --rw=randread --bs=4k --size=100% --runtime=60s --group_reporting --name=test
>> 
>> 
>> 3. fio parameters for throughput:
>> fio --filename=/dev/vdb--direct=1 --ioengine=psync --thread --numjobs=3 --rw=randread --bs=1024k --size=100% --runtime=60s --name=randread --group_reporting -name=test
>> 
>> 
>> 
>

[-- Attachment #2: Type: text/html, Size: 3189 bytes --]

Re: [Qemu-devel] extremely low IOPS performance of QCOW2 image format on an SSD RAID1

[Fam Zheng]

Cc'ing more qcow2 experts.

On Mon, 06/23 11:14, lihuiba wrote:
> >Did you prefill the image? Amplification could come from cluster allocation.
> Yes! 
> I forgot to mention that I created the qcow2 image with 'preallocation=metadata', and I have allocated
> the data blocks with dd in VM.
> 
> 
> Creating image in host:
> qemu-img create -f qcow2 -preallocation=metadata test.qcow2 100G
> 
> 
> Allocating the blocks in VM:
> dd if=/dev/zero of=/dev/vdb bs=1M
> where vdb is the target image.
> 
> 
> 
> 
> 
> 
> 
> At 2014-06-23 11:01:20, "Fam Zheng" <famz@redhat.com> wrote:
> >On Mon, 06/23 10:06, lihuiba wrote:
> >> Hi, all
> >> 
> >> 
> >> I'm using a qcow2 image stored on a SSD RAID1 (2 x intel S3500), and I'm benchmarking the
> >> system using fio. Although the throughput in VM (with KVM and virtio enabled) is acceptable (67%
> >> of thoughtput in host), the IOPS performance seems is extremely low ---- only 2% of IOPS in host.
> >> 
> >> 
> >> I was initially using qemu-1.1.2, and I also tried qemu-1.7.1 for comparison. There was no significant
> >> difference.
> >> 
> >> 
> >> In contrast, raw image and LVM perform very well. They usually achieve 90%+ of throughput and
> >> 60%+ of IOPS. So the problem must lie in the QCOW2 image format.
> >> 
> >> 
> >> And I observed that, when I perform 4KB IOPS benchmark in VM with a QCOW2 image, fio in VM reports
> >> it is reading 9.x MB/s, while iostat in host reports the SSD is being read 150+ MB/s. So QEMU or QCOW2
> >> must have amplified the amount of read by nearly 16 times.
> >> 
> >> 
> >> So, how can I fix or tune the performance issue of qcow2?
> >
> >Did you prefill the image? Amplification could come from cluster allocation.
> >
> >Fam
> >
> >> 
> >> 
> >> Thanks!
> >> 
> >> 
> >> 
> >> 
> >> PS:
> >> 1. qemu parameters:
> >> -enable-kvm -cpu qemu64 -rtc base=utc,clock=host,driftfix=none -usb -device usb-tablet -nodefaults -nodefconfig -no-kvm-pit-reinjection -global kvm-pit.lost_tick_policy=discard -machine pc,accel=kvm -vga std -k en-us -smp 8 -m 4096 -boot order=cdn -vnc :1 -drive file=$1,if=none,id=drive_0,cache=none,aio=native -device virtio-blk-pci,drive=drive_0,bus=pci.0,addr=0x5 -drive file=$2,if=none,id=drive_2,cache=none,aio=native -device virtio-blk-pci,drive=drive_2,bus=pci.0,addr=0x7
> >> 
> >> 
> >> 2. fio parameters for IOPS:
> >> fio --filename=/dev/vdb --direct=1 --ioengine=libaio --iodepth 32 --thread --numjobs=1 --rw=randread --bs=4k --size=100% --runtime=60s --group_reporting --name=test
> >> 
> >> 
> >> 3. fio parameters for throughput:
> >> fio --filename=/dev/vdb--direct=1 --ioengine=psync --thread --numjobs=3 --rw=randread --bs=1024k --size=100% --runtime=60s --name=randread --group_reporting -name=test
> >> 
> >> 
> >> 
> >

Re: [Qemu-devel] extremely low IOPS performance of QCOW2 image format on an SSD RAID1

[lihuiba]

[-- Attachment #1: Type: text/plain, Size: 3295 bytes --]

I think I have found the reason:
There's a cache in qemu that accelerates the transform of virtual LBA to cluster offset of qcow2 image.
The cache has a fixed size of 16x8192=128k in my configuration, which corresponds to a 8GB (128K*64KB)
mapping size. So when the "working set" of fio exceeds 8GB, the transform wil be degraded to reading
external table, and the performances goes extremely low.








At 2014-06-23 11:22:37, "Fam Zheng" <famz@redhat.com> wrote:
>Cc'ing more qcow2 experts.
>
>On Mon, 06/23 11:14, lihuiba wrote:
>> >Did you prefill the image? Amplification could come from cluster allocation.
>> Yes! 
>> I forgot to mention that I created the qcow2 image with 'preallocation=metadata', and I have allocated
>> the data blocks with dd in VM.
>> 
>> 
>> Creating image in host:
>> qemu-img create -f qcow2 -preallocation=metadata test.qcow2 100G
>> 
>> 
>> Allocating the blocks in VM:
>> dd if=/dev/zero of=/dev/vdb bs=1M
>> where vdb is the target image.
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> At 2014-06-23 11:01:20, "Fam Zheng" <famz@redhat.com> wrote:
>> >On Mon, 06/23 10:06, lihuiba wrote:
>> >> Hi, all
>> >> 
>> >> 
>> >> I'm using a qcow2 image stored on a SSD RAID1 (2 x intel S3500), and I'm benchmarking the
>> >> system using fio. Although the throughput in VM (with KVM and virtio enabled) is acceptable (67%
>> >> of thoughtput in host), the IOPS performance seems is extremely low ---- only 2% of IOPS in host.
>> >> 
>> >> 
>> >> I was initially using qemu-1.1.2, and I also tried qemu-1.7.1 for comparison. There was no significant
>> >> difference.
>> >> 
>> >> 
>> >> In contrast, raw image and LVM perform very well. They usually achieve 90%+ of throughput and
>> >> 60%+ of IOPS. So the problem must lie in the QCOW2 image format.
>> >> 
>> >> 
>> >> And I observed that, when I perform 4KB IOPS benchmark in VM with a QCOW2 image, fio in VM reports
>> >> it is reading 9.x MB/s, while iostat in host reports the SSD is being read 150+ MB/s. So QEMU or QCOW2
>> >> must have amplified the amount of read by nearly 16 times.
>> >> 
>> >> 
>> >> So, how can I fix or tune the performance issue of qcow2?
>> >
>> >Did you prefill the image? Amplification could come from cluster allocation.
>> >
>> >Fam
>> >
>> >> 
>> >> 
>> >> Thanks!
>> >> 
>> >> 
>> >> 
>> >> 
>> >> PS:
>> >> 1. qemu parameters:
>> >> -enable-kvm -cpu qemu64 -rtc base=utc,clock=host,driftfix=none -usb -device usb-tablet -nodefaults -nodefconfig -no-kvm-pit-reinjection -global kvm-pit.lost_tick_policy=discard -machine pc,accel=kvm -vga std -k en-us -smp 8 -m 4096 -boot order=cdn -vnc :1 -drive file=$1,if=none,id=drive_0,cache=none,aio=native -device virtio-blk-pci,drive=drive_0,bus=pci.0,addr=0x5 -drive file=$2,if=none,id=drive_2,cache=none,aio=native -device virtio-blk-pci,drive=drive_2,bus=pci.0,addr=0x7
>> >> 
>> >> 
>> >> 2. fio parameters for IOPS:
>> >> fio --filename=/dev/vdb --direct=1 --ioengine=libaio --iodepth 32 --thread --numjobs=1 --rw=randread --bs=4k --size=100% --runtime=60s --group_reporting --name=test
>> >> 
>> >> 
>> >> 3. fio parameters for throughput:
>> >> fio --filename=/dev/vdb--direct=1 --ioengine=psync --thread --numjobs=3 --rw=randread --bs=1024k --size=100% --runtime=60s --name=randread --group_reporting -name=test
>> >> 
>> >> 
>> >> 
>> >
>

[-- Attachment #2: Type: text/html, Size: 4315 bytes --]

Re: [Qemu-devel] extremely low IOPS performance of QCOW2 image format on an SSD RAID1

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 924 bytes --]

On Mon, Jun 23, 2014 at 11:14:25AM +0800, lihuiba wrote:
> >Did you prefill the image? Amplification could come from cluster allocation.
> Yes! 
> I forgot to mention that I created the qcow2 image with 'preallocation=metadata', and I have allocated
> the data blocks with dd in VM.
> 
> 
> Creating image in host:
> qemu-img create -f qcow2 -preallocation=metadata test.qcow2 100G
> 
> 
> Allocating the blocks in VM:
> dd if=/dev/zero of=/dev/vdb bs=1M
> where vdb is the target image.

If you used dd inside the guest to preallocate the entire image, then
both benchmark read and write requests work similarly to raw image
access (the qcow2 metadata is cached in RAM and metadata updates are
necessary, so accesses are very similar to raw).

Did you use the same file system on the host when comparing qcow2 to
raw image files?

Please post the full QEMU command-line and fio job files.

Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] extremely low IOPS performance of QCOW2 image format on an SSD RAID1

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 742 bytes --]

On Mon, Jun 23, 2014 at 02:20:25PM +0800, lihuiba wrote:
> I think I have found the reason:
> There's a cache in qemu that accelerates the transform of virtual LBA to cluster offset of qcow2 image.
> The cache has a fixed size of 16x8192=128k in my configuration, which corresponds to a 8GB (128K*64KB)
> mapping size. So when the "working set" of fio exceeds 8GB, the transform wil be degraded to reading
> external table, and the performances goes extremely low.

Can you confirm that making L2_CACHE_SIZE much bigger solves the
problem?

You also have the optional of specifying the cluster size when creating
the qcow2 image file.  A larger cluster size reduces the amount of
metadata overhead and therefore increases cache hits.

Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] extremely low IOPS performance of QCOW2 image format on an SSD RAID1

[Kevin Wolf]

Am 24.06.2014 um 09:25 hat lihuiba geschrieben:
> >Can you confirm that making L2_CACHE_SIZE much bigger solves the
> >problem?
> Yes, it is confirmed.
> When I run fio randread with a 7GB or 8GB size, result is close to that of raw image.
> But when the size is increased to 9GB, the result drops dramatically. And I have modified
> qcow2-cache.c to print a log when cache misses. When testing the 9GB size, there are
> lots of cache misses shown.
> 
> >You also have the optional of specifying the cluster size when creating
> >the qcow2 image file.  A larger cluster size reduces the amount of
> >metadata overhead and therefore increases cache hits.
> I didn't find any command-line option to increase the size of the cache, so I increased
> cluster_size to 1MB or 2MB. This worked very well for me.
> 
> BTW
> qemu-img of version 1.7.1 has bug when creating qcow2 image with options preallocation=metadata
> and cluster_size > 64K. It reports:
> 
> 
> # qemu-img create -f qcow2 -o preallocation=metadata,cluster_size=1M asdf.qcow2 100G
> Formatting 'asdf.qcow2', fmt=qcow2 size=107374182400 encryption=off cluster_size=1048576 preallocation='metadata' lazy_refcounts=off 
> qemu-img: block/qcow2-cluster.c:1196: qcow2_alloc_cluster_offset: Assertion `n_start * (1ULL << 9) == offset_into_cluster(s, offset)' failed.

This appears to be fixed in current git master.

Kevin