From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:55796) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TbneU-0005fp-UU for qemu-devel@nongnu.org; Fri, 23 Nov 2012 02:26:20 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TbneT-0003jc-GE for qemu-devel@nongnu.org; Fri, 23 Nov 2012 02:26:18 -0500 Received: from www.maurer-it.com ([213.129.239.114]:42445 helo=proxmox.maurer-it.com) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TbneT-0003jY-5w for qemu-devel@nongnu.org; Fri, 23 Nov 2012 02:26:17 -0500 From: Dietmar Maurer Date: Fri, 23 Nov 2012 07:26:13 +0000 Message-ID: <24E144B8C0207547AD09C467A8259F755782F76D@lisa.maurer-it.com> Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_24E144B8C0207547AD09C467A8259F755782F76Dlisamaureritcom_" MIME-Version: 1.0 Subject: [Qemu-devel] qcow2: slow internal snapshot creation List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "qemu-devel@nongnu.org" Cc: "Kevin Wolf (kwolf@redhat.com)" --_000_24E144B8C0207547AD09C467A8259F755782F76Dlisamaureritcom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable qcow2 snapshot on newly created files are fast: # qemu-img create -f qcow2 test.img 200G # time qemu-img snapshot -c snap1 test.img real 0m0.014s but if metadata is allocated it gets very slow: # qemu-img create -f qcow2 -o "preallocation=3Dmetadata" test.img 200G # time qemu-img snapshot -c snap1 test.img real 1m20.399s but reading the metadata is also fast: # time qemu-img check test.img real 0m0.371s So why is creating a new snapshot that slow - any ideas? --_000_24E144B8C0207547AD09C467A8259F755782F76Dlisamaureritcom_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

qcow2 snapshot on newly created= files are fast:

# qemu-img create -f qcow2 test= .img 200G

# time qemu-img snapshot -c sna= p1 test.img

real &nb= sp; 0m0.014s

but if metadata is allocated it= gets very slow:

# qemu-img create -f qcow= 2 -o "preallocation=3Dmetadata" test.img 200G

# time qemu-img snapshot -c sna= p1 test.img

real &nb= sp; 1m20.399s

but reading the metadata is als= o fast:

# time qemu-img check test.img<= o:p>

real &nb= sp; 0m0.371s

So why is creating a new snapsh= ot that slow – any ideas?

--_000_24E144B8C0207547AD09C467A8259F755782F76Dlisamaureritcom_-- From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:49682) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TbtsI-0003Xe-0b for qemu-devel@nongnu.org; Fri, 23 Nov 2012 09:05:06 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TbtsA-0000xT-IQ for qemu-devel@nongnu.org; Fri, 23 Nov 2012 09:04:57 -0500 Received: from mailpro.odiso.net ([89.248.209.98]:40287) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TbtsA-0000nc-Ca for qemu-devel@nongnu.org; Fri, 23 Nov 2012 09:04:50 -0500 Date: Fri, 23 Nov 2012 15:03:59 +0100 (CET) From: Alexandre DERUMIER Message-ID: In-Reply-To: <24E144B8C0207547AD09C467A8259F755782F76D@lisa.maurer-it.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [Qemu-devel] qcow2: slow internal snapshot creation List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Dietmar Maurer Cc: "Kevin Wolf (kwolf@redhat.com)" , qemu-devel@nongnu.org performance is also reduced when snapshot exist. (like if they are no preal= located metadatas) see initial git commit http://git.qemu.org/?p=3Dqemu.git;a=3Dcommit;h=3Da35e1c177debb01240243bd656= caca302410d38c "qcow2: Metadata preallocation This introduces a qemu-img create option for qcow2 which allows the metadat= a to be preallocated, i.e. clusters are reserved in the refcount table and L1/L2= tables, but no data is written to them. Metadata is quite small, so this happens in almost no time. Especially with qcow2 on virtio this helps to gain a bit of performance dur= ing the initial writes. However, as soon as create a snapshot, we're back to th= e normal slow speed, obviously. So this isn't the real fix, but kind of a che= at while we're still having trouble with qcow2 on virtio." ----- Mail original ----- De: "Dietmar Maurer" =C3=80: qemu-devel@nongnu.org Cc: "Kevin Wolf (kwolf@redhat.com)" Envoy=C3=A9: Vendredi 23 Novembre 2012 08:26:13 Objet: [Qemu-devel] qcow2: slow internal snapshot creation qcow2 snapshot on newly created files are fast: # qemu-img create -f qcow2 test.img 200G # time qemu-img snapshot -c snap1 test.img real 0m0.014s but if metadata is allocated it gets very slow: # qemu-img create -f qcow2 -o "preallocation=3Dmetadata" test.img 200G # time qemu-img snapshot -c snap1 test.img real 1m20.399s but reading the metadata is also fast: # time qemu-img check test.img real 0m0.371s So why is creating a new snapshot that slow =E2=80=93 any ideas? From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:50247) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Tbtwe-0004qI-9O for qemu-devel@nongnu.org; Fri, 23 Nov 2012 09:09:33 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TbtwU-000264-Q8 for qemu-devel@nongnu.org; Fri, 23 Nov 2012 09:09:28 -0500 Received: from mx1.redhat.com ([209.132.183.28]:36046) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TbtwU-00025H-IE for qemu-devel@nongnu.org; Fri, 23 Nov 2012 09:09:18 -0500 Message-ID: <50AF838B.5010703@redhat.com> Date: Fri, 23 Nov 2012 15:09:15 +0100 From: Kevin Wolf MIME-Version: 1.0 References: In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] qcow2: slow internal snapshot creation List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Alexandre DERUMIER Cc: Dietmar Maurer , qemu-devel@nongnu.org Am 23.11.2012 15:03, schrieb Alexandre DERUMIER: > performance is also reduced when snapshot exist. (like if they are no preallocated metadatas) Preallocation doesn't matter much these days. > see initial git commit > > http://git.qemu.org/?p=qemu.git;a=commit;h=a35e1c177debb01240243bd656caca302410d38c > "qcow2: Metadata preallocation > > This introduces a qemu-img create option for qcow2 which allows the metadata to > be preallocated, i.e. clusters are reserved in the refcount table and L1/L2 > tables, but no data is written to them. Metadata is quite small, so this > happens in almost no time. > > Especially with qcow2 on virtio this helps to gain a bit of performance during > the initial writes. However, as soon as create a snapshot, we're back to the > normal slow speed, obviously. So this isn't the real fix, but kind of a cheat > while we're still having trouble with qcow2 on virtio." This is a commit message from 2009 that already says that this wasn't the final solution. Kevin From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:57318) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Tbu4d-0001R8-89 for qemu-devel@nongnu.org; Fri, 23 Nov 2012 09:17:49 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Tbu4V-0004SI-Db for qemu-devel@nongnu.org; Fri, 23 Nov 2012 09:17:43 -0500 Received: from mailpro.odiso.net ([89.248.209.98]:34123) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Tbu4V-0004SD-7I for qemu-devel@nongnu.org; Fri, 23 Nov 2012 09:17:35 -0500 Date: Fri, 23 Nov 2012 15:17:04 +0100 (CET) From: Alexandre DERUMIER Message-ID: <9574d6cc-30fe-4424-b7e7-d363e2abdba9@mailpro> In-Reply-To: <50AF838B.5010703@redhat.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [Qemu-devel] qcow2: slow internal snapshot creation List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Kevin Wolf Cc: Dietmar Maurer , qemu-devel@nongnu.org >>Preallocation doesn't matter much these days. I have done benchmark through nfs on a netapp san, (with directio), I got 300 iops without preallocation, and 6000 iops with preallocation ....= ----- Mail original ----- De: "Kevin Wolf" =C3=80: "Alexandre DERUMIER" Cc: "Dietmar Maurer" , qemu-devel@nongnu.org Envoy=C3=A9: Vendredi 23 Novembre 2012 15:09:15 Objet: Re: [Qemu-devel] qcow2: slow internal snapshot creation Am 23.11.2012 15:03, schrieb Alexandre DERUMIER: > performance is also reduced when snapshot exist. (like if they are no pre= allocated metadatas) Preallocation doesn't matter much these days. > see initial git commit > > http://git.qemu.org/?p=3Dqemu.git;a=3Dcommit;h=3Da35e1c177debb01240243bd6= 56caca302410d38c > "qcow2: Metadata preallocation > > This introduces a qemu-img create option for qcow2 which allows the metad= ata to > be preallocated, i.e. clusters are reserved in the refcount table and L1/= L2 > tables, but no data is written to them. Metadata is quite small, so this = > happens in almost no time. > > Especially with qcow2 on virtio this helps to gain a bit of performance d= uring > the initial writes. However, as soon as create a snapshot, we're back to = the > normal slow speed, obviously. So this isn't the real fix, but kind of a c= heat > while we're still having trouble with qcow2 on virtio." This is a commit message from 2009 that already says that this wasn't the final solution. Kevin From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:59135) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Tbuhv-0000Jv-OI for qemu-devel@nongnu.org; Fri, 23 Nov 2012 09:58:25 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Tbuhq-00081k-2W for qemu-devel@nongnu.org; Fri, 23 Nov 2012 09:58:19 -0500 Received: from mx1.redhat.com ([209.132.183.28]:10428) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Tbuhp-00081c-Qc for qemu-devel@nongnu.org; Fri, 23 Nov 2012 09:58:13 -0500 Message-ID: <50AF8F01.9060008@redhat.com> Date: Fri, 23 Nov 2012 15:58:09 +0100 From: Kevin Wolf MIME-Version: 1.0 References: <9574d6cc-30fe-4424-b7e7-d363e2abdba9@mailpro> In-Reply-To: <9574d6cc-30fe-4424-b7e7-d363e2abdba9@mailpro> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] qcow2: slow internal snapshot creation List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Alexandre DERUMIER Cc: Dietmar Maurer , qemu-devel@nongnu.org Am 23.11.2012 15:17, schrieb Alexandre DERUMIER: >>> Preallocation doesn't matter much these days. > > I have done benchmark through nfs on a netapp san, (with directio), > > I got 300 iops without preallocation, and 6000 iops with preallocation .... This doesn't sound right, even for writethrough modes. Can you post more details about your benchmark setup so that this can be reproduced? And does it reproduce with local files as well or only using NFS? Kevin From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:36124) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TbvE2-0008Iw-Fv for qemu-devel@nongnu.org; Fri, 23 Nov 2012 10:31:34 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TbvDw-0001nj-FM for qemu-devel@nongnu.org; Fri, 23 Nov 2012 10:31:30 -0500 Received: from mx1.redhat.com ([209.132.183.28]:45535) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TbvDw-0001nQ-89 for qemu-devel@nongnu.org; Fri, 23 Nov 2012 10:31:24 -0500 Message-ID: <50AF96C7.3030506@redhat.com> Date: Fri, 23 Nov 2012 16:31:19 +0100 From: Kevin Wolf MIME-Version: 1.0 References: <24E144B8C0207547AD09C467A8259F755782F76D@lisa.maurer-it.com> In-Reply-To: <24E144B8C0207547AD09C467A8259F755782F76D@lisa.maurer-it.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Subject: Re: [Qemu-devel] qcow2: slow internal snapshot creation List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Dietmar Maurer Cc: "qemu-devel@nongnu.org" Am 23.11.2012 08:26, schrieb Dietmar Maurer: > qcow2 snapshot on newly created files are fast: >=20 > # qemu-img create -f qcow2 test.img 200G > # time qemu-img snapshot -c snap1 test.img > real 0m0.014s >=20 > but if metadata is allocated it gets very slow: >=20 > # qemu-img create -f qcow2 -o "preallocation=3Dmetadata" test.img 200G > # time qemu-img snapshot -c snap1 test.img > real 1m20.399s >=20 > but reading the metadata is also fast: >=20 > # time qemu-img check test.img > real 0m0.371s >=20 > So why is creating a new snapshot that slow =96 any ideas? Had a look at this now. The culprit is the unconditional bdrv_flush() in update_cluster_refcount(). I suspect it's completely unnecessary by now, but I need to give it a closer review before I send a patch to remove it. (And in any case error handling is missing there) Affected are writing compressed images and creating/deleting internal snapshots, no other code uses this path. Kevin From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:58500) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TbvZM-0003BE-Oi for qemu-devel@nongnu.org; Fri, 23 Nov 2012 10:53:33 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TbvZL-0008AV-Ew for qemu-devel@nongnu.org; Fri, 23 Nov 2012 10:53:32 -0500 Received: from mailpro.odiso.net ([89.248.209.98]:34385) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TbvZL-0008AR-6L for qemu-devel@nongnu.org; Fri, 23 Nov 2012 10:53:31 -0500 Date: Fri, 23 Nov 2012 16:53:00 +0100 (CET) From: Alexandre DERUMIER Message-ID: In-Reply-To: <50AF8F01.9060008@redhat.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [Qemu-devel] qcow2: slow internal snapshot creation List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Kevin Wolf Cc: Dietmar Maurer , qemu-devel@nongnu.org >>Can you post more details about your benchmark setup so that this can be = >>reproduced? And does it reproduce with local files as well or only using = >>NFS? Sure, I have redone tests, here the results: (note that my netapp san have a nvram journal, so directio are as fast than= buffered io) preallocated vs non preallocated have a ratio of 5/1 for local, and 10/1 fo= r nfs. directio bench --------------- fio --filename=3D/dev/vdb --direct=3D1 -rw=3Drandwrite --bs=3D4K --size=3D5= G --iodepth=3D100 --group-reporting --name=3Dfile1 --ioengine=3Dlibaio buffered io bench ------------------ fio --filename=3D/dev/vdb --direct=3D1 -rw=3Drandwrite --bs=3D4K --size=3D5= G --iodepth=3D100 --group-reporting --name=3Dfile1 --ioengine=3Dlibaio results ------- local raw file --------------- cache=3Dnone, directio, 9000 iops cache=3Dnone, buffered io, 10000 iops local qcow2 preallocated -------------------------- cache=3Dnone, directio, 8000 iops cache=3Dnone, buffered io, 9000 iops local qcow2 non preallocated ----------------------------- cache=3Dnone, directio, 1400 iops cache=3Dnone, buffered io, 2000 iops nfs rawfile (netapp with nvram journal) ---------------------------------------- cache=3Dnone, directio, 15000 iops cache=3Dnone, buffered io, 15000 iops nfs qcow2 preallocated (netapp with nvram journal) ------------------- cache=3Dnone, directio, 12000 iops cache=3Dnone, buffered io, 12000 iops nfs qcow2 non preallocated (netapp with nvram journal) ----------------------------------------------------- cache=3Dnone, directio, 1500 iops cache=3Dnone, buffered io, 1500 iops Alexandre D e rumier Ing=C3=A9nieur Syst=C3=A8mes et R=C3=A9seaux Fixe : 03 20 68 88 85 Fax : 03 20 68 90 88 45 Bvd du G=C3=A9n=C3=A9ral Leclerc 59100 Roubaix 12 rue Marivaux 75002 Paris ----- Mail original ----- De: "Kevin Wolf" =C3=80: "Alexandre DERUMIER" Cc: "Dietmar Maurer" , qemu-devel@nongnu.org Envoy=C3=A9: Vendredi 23 Novembre 2012 15:58:09 Objet: Re: [Qemu-devel] qcow2: slow internal snapshot creation Am 23.11.2012 15:17, schrieb Alexandre DERUMIER: >>> Preallocation doesn't matter much these days. > > I have done benchmark through nfs on a netapp san, (with directio), > > I got 300 iops without preallocation, and 6000 iops with preallocation ..= .. This doesn't sound right, even for writethrough modes. Can you post more details about your benchmark setup so that this can be reproduced? And does it reproduce with local files as well or only using NFS? Kevin