[Qemu-devel] qcow2: slow internal snapshot creation

[Qemu-devel] qcow2: slow internal snapshot creation

[Dietmar Maurer]

[-- Attachment #1: Type: text/plain, Size: 507 bytes --]

qcow2 snapshot on newly created files are fast:

# qemu-img create -f qcow2 test.img 200G
# time qemu-img snapshot -c snap1 test.img
real       0m0.014s

but if metadata is allocated it gets very slow:

# qemu-img create  -f qcow2 -o "preallocation=metadata" test.img 200G
# time qemu-img snapshot -c snap1 test.img
real       1m20.399s

but reading the metadata is also fast:

# time qemu-img check test.img
real       0m0.371s

So why is creating a new snapshot that slow - any ideas?


[-- Attachment #2: Type: text/html, Size: 3523 bytes --]

Re: [Qemu-devel] qcow2: slow internal snapshot creation

[Alexandre DERUMIER]

performance is also reduced when snapshot exist. (like if they are no preallocated metadatas)


see initial git commit

http://git.qemu.org/?p=qemu.git;a=commit;h=a35e1c177debb01240243bd656caca302410d38c
"qcow2: Metadata preallocation

This introduces a qemu-img create option for qcow2 which allows the metadata to
be preallocated, i.e. clusters are reserved in the refcount table and L1/L2
tables, but no data is written to them. Metadata is quite small, so this
happens in almost no time.

Especially with qcow2 on virtio this helps to gain a bit of performance during
the initial writes. However, as soon as create a snapshot, we're back to the
normal slow speed, obviously. So this isn't the real fix, but kind of a cheat
while we're still having trouble with qcow2 on virtio."



----- Mail original -----

De: "Dietmar Maurer" <dietmar@proxmox.com>
À: qemu-devel@nongnu.org
Cc: "Kevin Wolf (kwolf@redhat.com)" <kwolf@redhat.com>
Envoyé: Vendredi 23 Novembre 2012 08:26:13
Objet: [Qemu-devel] qcow2: slow internal snapshot creation



qcow2 snapshot on newly created files are fast:

# qemu-img create -f qcow2 test.img 200G
# time qemu-img snapshot -c snap1 test.img
real 0m0.014s

but if metadata is allocated it gets very slow:

# qemu-img create -f qcow2 -o "preallocation=metadata" test.img 200G
# time qemu-img snapshot -c snap1 test.img
real 1m20.399s

but reading the metadata is also fast:

# time qemu-img check test.img
real 0m0.371s

So why is creating a new snapshot that slow – any ideas?

Re: [Qemu-devel] qcow2: slow internal snapshot creation

[Kevin Wolf]

Am 23.11.2012 15:03, schrieb Alexandre DERUMIER:
> performance is also reduced when snapshot exist. (like if they are no preallocated metadatas)

Preallocation doesn't matter much these days.

> see initial git commit
> 
> http://git.qemu.org/?p=qemu.git;a=commit;h=a35e1c177debb01240243bd656caca302410d38c
> "qcow2: Metadata preallocation
> 
> This introduces a qemu-img create option for qcow2 which allows the metadata to
> be preallocated, i.e. clusters are reserved in the refcount table and L1/L2
> tables, but no data is written to them. Metadata is quite small, so this
> happens in almost no time.
> 
> Especially with qcow2 on virtio this helps to gain a bit of performance during
> the initial writes. However, as soon as create a snapshot, we're back to the
> normal slow speed, obviously. So this isn't the real fix, but kind of a cheat
> while we're still having trouble with qcow2 on virtio."

This is a commit message from 2009 that already says that this wasn't
the final solution.

Kevin

Re: [Qemu-devel] qcow2: slow internal snapshot creation

[Alexandre DERUMIER]

>>Preallocation doesn't matter much these days.

I have done benchmark through nfs on a netapp san, (with directio),

I got 300 iops without preallocation, and 6000 iops with preallocation ....



----- Mail original -----

De: "Kevin Wolf" <kwolf@redhat.com>
À: "Alexandre DERUMIER" <aderumier@odiso.com>
Cc: "Dietmar Maurer" <dietmar@proxmox.com>, qemu-devel@nongnu.org
Envoyé: Vendredi 23 Novembre 2012 15:09:15
Objet: Re: [Qemu-devel] qcow2: slow internal snapshot creation

Am 23.11.2012 15:03, schrieb Alexandre DERUMIER:
> performance is also reduced when snapshot exist. (like if they are no preallocated metadatas)

Preallocation doesn't matter much these days.

> see initial git commit
>
> http://git.qemu.org/?p=qemu.git;a=commit;h=a35e1c177debb01240243bd656caca302410d38c
> "qcow2: Metadata preallocation
>
> This introduces a qemu-img create option for qcow2 which allows the metadata to
> be preallocated, i.e. clusters are reserved in the refcount table and L1/L2
> tables, but no data is written to them. Metadata is quite small, so this 
> happens in almost no time.
>
> Especially with qcow2 on virtio this helps to gain a bit of performance during
> the initial writes. However, as soon as create a snapshot, we're back to the
> normal slow speed, obviously. So this isn't the real fix, but kind of a cheat
> while we're still having trouble with qcow2 on virtio."

This is a commit message from 2009 that already says that this wasn't
the final solution.

Kevin

Re: [Qemu-devel] qcow2: slow internal snapshot creation

[Kevin Wolf]

Am 23.11.2012 15:17, schrieb Alexandre DERUMIER:
>>> Preallocation doesn't matter much these days.
> 
> I have done benchmark through nfs on a netapp san, (with directio),
> 
> I got 300 iops without preallocation, and 6000 iops with preallocation ....

This doesn't sound right, even for writethrough modes.

Can you post more details about your benchmark setup so that this can be
reproduced? And does it reproduce with local files as well or only using
NFS?

Kevin

Re: [Qemu-devel] qcow2: slow internal snapshot creation

[Kevin Wolf]

Am 23.11.2012 08:26, schrieb Dietmar Maurer:
> qcow2 snapshot on newly created files are fast:
> 
> # qemu-img create -f qcow2 test.img 200G
> # time qemu-img snapshot -c snap1 test.img
> real       0m0.014s
> 
> but if metadata is allocated it gets very slow:
> 
> # qemu-img create  -f qcow2 -o "preallocation=metadata" test.img 200G
> # time qemu-img snapshot -c snap1 test.img
> real       1m20.399s
> 
> but reading the metadata is also fast:
> 
> # time qemu-img check test.img
> real       0m0.371s
> 
> So why is creating a new snapshot that slow – any ideas?

Had a look at this now. The culprit is the unconditional bdrv_flush() in
update_cluster_refcount(). I suspect it's completely unnecessary by now,
but I need to give it a closer review before I send a patch to remove
it. (And in any case error handling is missing there)

Affected are writing compressed images and creating/deleting internal
snapshots, no other code uses this path.

Kevin

Re: [Qemu-devel] qcow2: slow internal snapshot creation

[Alexandre DERUMIER]

>>Can you post more details about your benchmark setup so that this can be 
>>reproduced? And does it reproduce with local files as well or only using 
>>NFS?

Sure, I have redone tests, here the results:
(note that my netapp san have a nvram journal, so directio are as fast than buffered io)

preallocated vs non preallocated have a ratio of 5/1 for local, and 10/1 for nfs.



directio bench
---------------
fio --filename=/dev/vdb --direct=1 -rw=randwrite --bs=4K --size=5G --iodepth=100 --group-reporting --name=file1 --ioengine=libaio

buffered io bench
------------------
fio --filename=/dev/vdb --direct=1 -rw=randwrite --bs=4K --size=5G --iodepth=100 --group-reporting --name=file1 --ioengine=libaio


results
-------
local raw file
---------------
cache=none, directio, 9000 iops
cache=none,  buffered io, 10000 iops

local qcow2 preallocated
--------------------------
cache=none, directio, 8000 iops
cache=none, buffered io, 9000 iops

local qcow2 non preallocated
-----------------------------
cache=none, directio, 1400 iops
cache=none, buffered io, 2000 iops




nfs rawfile (netapp with nvram journal)
----------------------------------------
cache=none, directio, 15000 iops
cache=none,  buffered io, 15000 iops


nfs qcow2 preallocated (netapp with nvram journal)
-------------------
cache=none, directio, 12000 iops
cache=none, buffered io, 12000 iops

nfs qcow2 non preallocated (netapp with nvram journal)
-----------------------------------------------------
cache=none, directio, 1500 iops
cache=none, buffered io, 1500 iops






Alexandre D e rumier

Ingénieur Systèmes et Réseaux


Fixe : 03 20 68 88 85

Fax : 03 20 68 90 88


45 Bvd du Général Leclerc 59100 Roubaix
12 rue Marivaux 75002 Paris


----- Mail original -----

De: "Kevin Wolf" <kwolf@redhat.com>
À: "Alexandre DERUMIER" <aderumier@odiso.com>
Cc: "Dietmar Maurer" <dietmar@proxmox.com>, qemu-devel@nongnu.org
Envoyé: Vendredi 23 Novembre 2012 15:58:09
Objet: Re: [Qemu-devel] qcow2: slow internal snapshot creation

Am 23.11.2012 15:17, schrieb Alexandre DERUMIER:
>>> Preallocation doesn't matter much these days.
>
> I have done benchmark through nfs on a netapp san, (with directio),
>
> I got 300 iops without preallocation, and 6000 iops with preallocation ....

This doesn't sound right, even for writethrough modes.

Can you post more details about your benchmark setup so that this can be
reproduced? And does it reproduce with local files as well or only using
NFS?

Kevin