Re: [Qemu-devel] [PATCH v3 00/12] qcow2: Add new overlap check functions

[Max Reitz <mreitz@redhat.com>]

Pïng

On 2015-02-27 at 17:20, Max Reitz wrote:
> Ping
>
> On 2015-02-09 at 14:25, Max Reitz wrote:
>> As has been requested, this series adds new overlap check functions to
>> the qcow2 code. My local branch is called "qcow2-improved-overlap-v1",
>> but I am not so sure whether it is actually an improvement; that is left
>> for you to decide, dear reviewers.
>>
>> See patch 1 for an explanation of why this series exists and what it
>> does. Patch 1 is basically the core of this series, the rest just
>> employs the functions introduced there.
>>
>> In a later patch, we may want to change the meaning of the "constant"
>> overlap checking option to mean the same as "cached", which is
>> everything except for inactive L2 tables. This series does make
>> checking for overlaps with inactive L2 tables at runtime just as cheap
>> as everything else (constant time plus caching), but using these checks
>> means qemu has to read all the snapshot L1 tables when opening a qcow2
>> file. This does not take long, of course, but it does result in a bit of
>> overhead so I did not want to enable it by default.
>>
>> I think just enabling all overlap checks by default after this series
>> should be fine and useful, though.
>>
>>
>> For benchmarks, please see my cover letter for v2:
>> http://lists.nongnu.org/archive/html/qemu-devel/2014-11/msg03430.html
>>
>> I'll just quote the (fixed) tl;dr here:
>>
>> * CPU usage at runtime decreased by 15000 to 27500 percent on
>>    overlap-check-heavy tasks
>> * No additional performance problems at loading time (in theory has the
>>    same runtime complexity as a single overlap check right now; in
>>    practice I could not find any problems)
>> * Decent RAM usage (170 kB for a 1 TB image with 64 kB clusters; 170 MB
>>    for a 1 PB image etc. pp.)
>>
>>
>> v3:
>> - Patch 1:
>>    - s/few memory/little memory/ [Eric]
>>    - s/2014/2015/ [Eric]
>>    - s/of a or a/of or a/ [Eric]
>>    - Use nb_clusters_minus_one instead of nb_clusters, because
>>      nb_clusters == 0 is invalid [Eric]
>>    - Drop cached_windows_index [Stefan]
>>    - Reset current_nb_clusters to 1 when a new range starts in the
>>      algorithm which rebuilds the fragment list from the bitmap
>>      representation
>> - Patch 3: Added a TODO comment that the user should be able to override
>>    the default cache size limit
>> - Patch 8: As the patch "qcow2: Buffer L1 table in snapshot refcount
>>    update" has been dropped, this patch was changed to no longer rely on
>>    it
>> - Patch 11: Replaced the size limit for a snapshot's L1 table by
>>    QCOW_MAX_L1_SIZE (was: INT_MAX / sizeof(uint64_t)) [Eric]
>>
>>
>> git-backport-diff against v2:
>>
>> Key:
>> [----] : patches are identical
>> [####] : number of functional differences between upstream/downstream 
>> patch
>> [down] : patch is downstream-only
>> The flags [FC] indicate (F)unctional and (C)ontextual differences, 
>> respectively
>>
>> 001/12:[0026] [FC] 'qcow2: Add new overlap check functions'
>> 002/12:[----] [--] 'qcow2: Pull up overlap check option evaluation'
>> 003/12:[0001] [FC] 'qcow2: Create metadata list'
>> 004/12:[----] [--] 'qcow2/overlaps: Protect image header'
>> 005/12:[----] [--] 'qcow2/overlaps: Protect refcount table'
>> 006/12:[----] [--] 'qcow2/overlaps: Protect refcount blocks'
>> 007/12:[----] [--] 'qcow2/overlaps: Protect active L1 table'
>> 008/12:[0002] [FC] 'qcow2/overlaps: Protect active L2 tables'
>> 009/12:[----] [--] 'qcow2/overlaps: Protect snapshot table'
>> 010/12:[----] [--] 'qcow2/overlaps: Protect inactive L1 tables'
>> 011/12:[0002] [FC] 'qcow2/overlaps: Protect inactive L2 tables'
>> 012/12:[----] [--] 'qcow2: Use new metadata overlap check function'
>>
>>
>> Max Reitz (12):
>>    qcow2: Add new overlap check functions
>>    qcow2: Pull up overlap check option evaluation
>>    qcow2: Create metadata list
>>    qcow2/overlaps: Protect image header
>>    qcow2/overlaps: Protect refcount table
>>    qcow2/overlaps: Protect refcount blocks
>>    qcow2/overlaps: Protect active L1 table
>>    qcow2/overlaps: Protect active L2 tables
>>    qcow2/overlaps: Protect snapshot table
>>    qcow2/overlaps: Protect inactive L1 tables
>>    qcow2/overlaps: Protect inactive L2 tables
>>    qcow2: Use new metadata overlap check function
>>
>>   block/Makefile.objs    |   3 +-
>>   block/qcow2-cluster.c  |  13 ++
>>   block/qcow2-overlap.c  | 396 
>> +++++++++++++++++++++++++++++++++++++++++++++++++
>>   block/qcow2-refcount.c | 202 ++++++++++---------------
>>   block/qcow2-snapshot.c | 105 ++++++++++++-
>>   block/qcow2.c          | 131 ++++++++++------
>>   block/qcow2.h          |  13 ++
>>   7 files changed, 691 insertions(+), 172 deletions(-)
>>   create mode 100644 block/qcow2-overlap.c