From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:52073) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TqoQz-0006as-MD for qemu-devel@nongnu.org; Thu, 03 Jan 2013 12:18:26 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TqoQw-0004be-UG for qemu-devel@nongnu.org; Thu, 03 Jan 2013 12:18:25 -0500 Received: from nodalink.pck.nerim.net ([62.212.105.220]:56688 helo=paradis.irqsave.net) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TqoQw-0004bE-IY for qemu-devel@nongnu.org; Thu, 03 Jan 2013 12:18:22 -0500 Date: Thu, 3 Jan 2013 18:18:40 +0100 From: =?iso-8859-1?Q?Beno=EEt?= Canet Message-ID: <20130103171840.GA4910@irqsave.net> References: <1357143393-29832-1-git-send-email-benoit@irqsave.net> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline In-Reply-To: <1357143393-29832-1-git-send-email-benoit@irqsave.net> Content-Transfer-Encoding: quoted-printable Subject: Re: [Qemu-devel] [RFC V4 00/30] QCOW2 deduplication List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: qemu-devel@nongnu.org Cc: kwolf@redhat.com, pbonzini@redhat.com, stefanha@redhat.com Hello, I started to write the deduplication metrics code in order to be able to design asynchronous deduplication. I am looking for a way to create a metric allowing deduplication to be pa= used or resumed on a given threshold. Does anyone have a sugestion regarding the metric that could be used for = this ? Best regards Beno=EEt > Le Wednesday 02 Jan 2013 =E0 17:16:03 (+0100), Beno=EEt Canet a =E9crit= : > This patchset is a cleanup of the previous QCOW2 deduplication rfc. >=20 > One can compile and install https://github.com/wernerd/Skein3Fish and u= se the > --enable-skein-dedup configure option in order to use the faster skein = HASH. >=20 > Images must be created with "-o dedup=3D[skein|sha256]" in order to act= ivate the > deduplication in the image. >=20 > Deduplication is now fast enough to be usable. >=20 > v4: Fix and complete qcow2 spec [Stefan] > Hash the hash_algo field in the header extension [Stefan] > Fix qcow2 spec [Eric] > Remove pointer to hash and simplify hash memory management [Stefan] > Rename and move qcow2_read_cluster_data to qcow2.c [Stefan] > Document lock dropping behaviour of the previous function [Stefan] > cleanup qcow2_dedup_read_missing_cluster_data [Stefan] > rename *_offset to *_sect [Stefan] > add a ./configure check for ssl [Stefan] > Replace openssl by gnutls [Stefan] > Implement Skein hashes > Rewrite pretty every qcow2-dedup.c commits after Add > qcow2_dedup_read_missing_and_concatenate to simplify the code > Use 64KB deduplication hash block to reduce allocation flushes > Use 64KB l2 tables to reduce allocation flushes [breaks compatibili= ty] > Use lazy refcounts to avoid qcow2_cache_set_dependency loops result= ings > in frequent caches flushes > Do not create and load dedup RAM structures when bdrs->read_only is= true >=20 > v3: make it work barely > replace kernel red black trees by gtree. >=20 > *** BLURB HERE *** >=20 > Beno=EEt Canet (30): > qcow2: Add deduplication to the qcow2 specification. > qcow2: Add deduplication structures and fields. > qcow2: Add qcow2_dedup_read_missing_and_concatenate > qcow2: Make update_refcount public. > qcow2: Create a way to link to l2 tables when deduplicating. > qcow2: Add qcow2_dedup and related functions > qcow2: Add qcow2_dedup_store_new_hashes. > qcow2: Implement qcow2_compute_cluster_hash. > qcow2: Extract qcow2_dedup_grow_table > qcow2: Add qcow2_dedup_grow_table and use it. > qcow2: create function to load deduplication hashes at startup. > qcow2: Load and save deduplication table header extension. > qcow2: Extract qcow2_do_table_init. > qcow2-cache: Allow to choose table size at creation. > qcow2: Add qcow2_dedup_init and qcow2_dedup_close. > qcow2: Extract qcow2_add_feature and qcow2_remove_feature. > block: Add qemu-img dedup create option. > qcow2: Behave correctly when refcount reach 0 or 2^16. > qcow2: Integrate deduplication in qcow2_co_writev loop. > qcow2: Serialize write requests when deduplication is activated. > qcow2: Add verification of dedup table. > qcow2: Adapt checking of QCOW_OFLAG_COPIED for dedup. > qcow2: Add check_dedup_l2 in order to check l2 of dedup table. > qcow2: Do not overwrite existing entries with QCOW_OFLAG_COPIED. > qcow2: Integrate SKEIN hash algorithm in deduplication. > qcow2: Add lazy refcounts to deduplication to prevent > qcow2_cache_set_dependency loops > qcow2: Use large L2 table for deduplication. > qcow: Set dedup cluster block size to 64KB. > qcow2: init and cleanup deduplication. > qemu-iotests: Filter dedup=3Don/off so existing tests don't break. >=20 > block/Makefile.objs | 1 + > block/qcow2-cache.c | 12 +- > block/qcow2-cluster.c | 116 +++-- > block/qcow2-dedup.c | 1157 ++++++++++++++++++++++++++++++++++= ++++++++ > block/qcow2-refcount.c | 157 ++++-- > block/qcow2.c | 357 +++++++++++-- > block/qcow2.h | 120 ++++- > configure | 55 ++ > docs/specs/qcow2.txt | 100 +++- > include/block/block_int.h | 1 + > tests/qemu-iotests/common.rc | 3 +- > 11 files changed, 1955 insertions(+), 124 deletions(-) > create mode 100644 block/qcow2-dedup.c >=20 > --=20 > 1.7.10.4 >=20