[Qemu-devel] [PATCH] iotests: Test non-self-referential qcow2 refblocks

[Qemu-devel] [PATCH] iotests: Test non-self-referential qcow2 refblocks

[Max Reitz]

It is easy to create only self-referential refblocks, but there are
cases where that is impossible. This adds a test for two of those cases
(combined in a single test case).

Suggested-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
This patch depends on version 4 (or hopefully any later version) of my
"qcow2: Support refcount orders != 4" series.
---
 tests/qemu-iotests/115     | 95 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/115.out |  8 ++++
 tests/qemu-iotests/group   |  1 +
 3 files changed, 104 insertions(+)
 create mode 100755 tests/qemu-iotests/115
 create mode 100644 tests/qemu-iotests/115.out

diff --git a/tests/qemu-iotests/115 b/tests/qemu-iotests/115
new file mode 100755
index 0000000..a6be187
--- /dev/null
+++ b/tests/qemu-iotests/115
@@ -0,0 +1,95 @@
+#!/bin/bash
+#
+# Test case for non-self-referential qcow2 refcount blocks
+#
+# Copyright (C) 2014 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
+owner=mreitz@redhat.com
+
+seq="$(basename $0)"
+echo "QA output created by $seq"
+
+here="$PWD"
+tmp=/tmp/$$
+status=1	# failure is the default!
+
+_cleanup()
+{
+	_cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+_supported_fmt qcow2
+_supported_proto file
+_supported_os Linux
+# This test relies on refcounts being 64 bits wide (which does not work with
+# compat=0.10)
+_unsupported_imgopts 'refcount_bits=\([^6]\|.\([^4]\|$\)\)' 'compat=0.10'
+
+echo
+echo '=== Testing large refcount and L1 table ==='
+echo
+
+# Create an image with an L1 table and a refcount table that each span twice the
+# number of clusters which can be described by a single refblock; therefore, at
+# least two refblocks cannot count their own refcounts because all the clusters
+# they describe are part of the L1 table or refcount table.
+
+# One refblock can describe (with cluster_size=512 and refcount_bits=64)
+# 512/8 = 64 clusters, therefore the L1 table should cover 128 clusters, which
+# equals 128 * (512/8) = 8192 entries (actually, 8192 - 512/8 = 8129 would
+# suffice, but it does not really matter). 8192 L2 tables can in turn describe
+# 8192 * 512/8 = 524,288 clusters which cover a space of 256 MB.
+
+# Since with refcount_bits=64 every refcount block entry is 64 bits wide (just
+# like the L2 table entries), the same calculation applies to the refcount table
+# as well; the difference is that while for the L1 table the guest disk size is
+# concerned, for the refcount table it is the image length that has to be at
+# least 256 MB. We can achieve that by using preallocation=metadata for an image
+# which has a guest disk size of 256 MB.
+
+IMGOPTS="$IMGOPTS,refcount_bits=64,cluster_size=512,preallocation=metadata" \
+    _make_test_img 256M
+
+# We know for sure that the L1 and refcount tables do not overlap with any other
+# structure because the metadata overlap checks would have caught that case.
+
+# Because qemu refuses to open qcow2 files whose L1 table does not cover the
+# whole guest disk size, it is definitely large enough. On the other hand, to
+# test whether the refcount table is large enough, we simply have to verify that
+# indeed all the clusters are allocated, which is done by qemu-img check.
+
+# The final thing we need to test is whether the tables are actually covered by
+# refcount blocks; since all clusters of the tables are referenced, we can use
+# qemu-img check for that purpose, too.
+
+$QEMU_IMG check "$TEST_IMG" | \
+    sed -e 's/^.* = \([0-9]\+\.[0-9]\+% allocated\).*\(clusters\)$/\1 \2/' \
+        -e '/^Image end offset/d'
+
+# (Note that we cannot use _check_test_img because that function filters out the
+# allocation status)
+
+# success, all done
+echo '*** done'
+rm -f $seq.full
+status=0
diff --git a/tests/qemu-iotests/115.out b/tests/qemu-iotests/115.out
new file mode 100644
index 0000000..7b2c5e0
--- /dev/null
+++ b/tests/qemu-iotests/115.out
@@ -0,0 +1,8 @@
+QA output created by 115
+
+=== Testing large refcount and L1 table ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=268435456 preallocation='metadata'
+No errors were found on the image.
+100.00% allocated clusters
+*** done
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index 7689770..52ae888 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -115,3 +115,4 @@
 111 rw auto quick
 112 rw auto
 114 rw auto quick
+115 rw auto
-- 
1.9.3

Re: [Qemu-devel] [PATCH] iotests: Test non-self-referential qcow2 refblocks

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 3665 bytes --]

On 12/05/2014 09:53 AM, Max Reitz wrote:
> It is easy to create only self-referential refblocks, but there are
> cases where that is impossible. This adds a test for two of those cases
> (combined in a single test case).
> 
> Suggested-by: Eric Blake <eblake@redhat.com>
> Signed-off-by: Max Reitz <mreitz@redhat.com>
> ---
> This patch depends on version 4 (or hopefully any later version) of my
> "qcow2: Support refcount orders != 4" series.
> ---
>  tests/qemu-iotests/115     | 95 ++++++++++++++++++++++++++++++++++++++++++++++
>  tests/qemu-iotests/115.out |  8 ++++
>  tests/qemu-iotests/group   |  1 +
>  3 files changed, 104 insertions(+)
>  create mode 100755 tests/qemu-iotests/115
>  create mode 100644 tests/qemu-iotests/115.out

> +
> +# One refblock can describe (with cluster_size=512 and refcount_bits=64)
> +# 512/8 = 64 clusters, therefore the L1 table should cover 128 clusters, which
> +# equals 128 * (512/8) = 8192 entries (actually, 8192 - 512/8 = 8129 would

This math is slightly off; you really only need 127 consecutive clusters
to guarantee that you have an aligned 64 clusters somewhere in the mix.
 Also, 8192 - 512/8 = 8128 (you are missing the +1 L2 entry that is
necessary to ensure the rollover to 128 clusters).  The real minimum of
L2 entries to provoke the situation we are after is thus 127 * (512/8) -
512/8 + 1 = 8065...

> +# suffice, but it does not really matter). 8192 L2 tables can in turn describe

...at any rate, your conclusion that it does not really matter is
correct; and rounding up to the actual power of 2 is both easier to
explain and more likely to happen in practice (I'm not going to go out
of my way to create a 255.9M guest image, after all).

So, whether or not you want to tweak the comment wording, the test
itself is correct.

Reviewed-by: Eric Blake <eblake@redhat.com>

> +# 8192 * 512/8 = 524,288 clusters which cover a space of 256 MB.
> +
> +# Since with refcount_bits=64 every refcount block entry is 64 bits wide (just
> +# like the L2 table entries), the same calculation applies to the refcount table
> +# as well; the difference is that while for the L1 table the guest disk size is
> +# concerned, for the refcount table it is the image length that has to be at
> +# least 256 MB. We can achieve that by using preallocation=metadata for an image
> +# which has a guest disk size of 256 MB.
> +
> +IMGOPTS="$IMGOPTS,refcount_bits=64,cluster_size=512,preallocation=metadata" \
> +    _make_test_img 256M

Wow - that's MUCH faster than my earlier message that omitted
preallocation then used qemu-io as a followup (around 8 seconds on my
machine); but ALSO much more fragmented (the resulting image, per
qemu-img map, of my multi-minute approach had guest data runs of  mainly
0x8000 bytes at a time, with a few 0x7e00 and 0x8200 sprinkled in, while
your preallocation has lots of runs of 0x200 (1 cluster), 0x400 and so
on, a few runs of 0x1000 or larger, but no run larger than 0x1a00).  But
I did confirm with 'qemu-img map -f raw' that the file is non-sparse, so
you indeed covered both L1 and reftables.

[and seeing how fragmented things can be makes me wonder what sort of
performance gains we could get, if any, by writing a qcow2 defragmenter
- but that's a project for another day, if ever]

> +++ b/tests/qemu-iotests/group
> @@ -115,3 +115,4 @@
>  111 rw auto quick
>  112 rw auto
>  114 rw auto quick
> +115 rw auto

8 seconds is still pretty slow, so I agree with not marking this 'quick'.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 539 bytes --]

Re: [Qemu-devel] [PATCH] iotests: Test non-self-referential qcow2 refblocks

[Max Reitz]

On 2014-12-05 at 12:43, Eric Blake wrote:
> On 12/05/2014 09:53 AM, Max Reitz wrote:
>> It is easy to create only self-referential refblocks, but there are
>> cases where that is impossible. This adds a test for two of those cases
>> (combined in a single test case).
>>
>> Suggested-by: Eric Blake <eblake@redhat.com>
>> Signed-off-by: Max Reitz <mreitz@redhat.com>
>> ---
>> This patch depends on version 4 (or hopefully any later version) of my
>> "qcow2: Support refcount orders != 4" series.
>> ---
>>   tests/qemu-iotests/115     | 95 ++++++++++++++++++++++++++++++++++++++++++++++
>>   tests/qemu-iotests/115.out |  8 ++++
>>   tests/qemu-iotests/group   |  1 +
>>   3 files changed, 104 insertions(+)
>>   create mode 100755 tests/qemu-iotests/115
>>   create mode 100644 tests/qemu-iotests/115.out
>> +
>> +# One refblock can describe (with cluster_size=512 and refcount_bits=64)
>> +# 512/8 = 64 clusters, therefore the L1 table should cover 128 clusters, which
>> +# equals 128 * (512/8) = 8192 entries (actually, 8192 - 512/8 = 8129 would
> This math is slightly off; you really only need 127 consecutive clusters
> to guarantee that you have an aligned 64 clusters somewhere in the mix.
>   Also, 8192 - 512/8 = 8128 (you are missing the +1 L2 entry that is
> necessary to ensure the rollover to 128 clusters).  The real minimum of
> L2 entries to provoke the situation we are after is thus 127 * (512/8) -
> 512/8 + 1 = 8065...
>
>> +# suffice, but it does not really matter). 8192 L2 tables can in turn describe
> ...at any rate, your conclusion that it does not really matter is
> correct; and rounding up to the actual power of 2 is both easier to
> explain and more likely to happen in practice (I'm not going to go out
> of my way to create a 255.9M guest image, after all).
>
> So, whether or not you want to tweak the comment wording, the test
> itself is correct.

Once again, I'm super-fast with my responses.

You're right. However, since the wording is not "This is the minimum for 
getting a non-self-referential refblock" but more like "If we do this, 
we are sure to get a non-self-referential refblock", I'll just leave it 
as it is.

> Reviewed-by: Eric Blake <eblake@redhat.com>

Thanks!

Re: [Qemu-devel] [PATCH] iotests: Test non-self-referential qcow2 refblocks

[Max Reitz]

On 2014-12-05 at 11:53, Max Reitz wrote:
> It is easy to create only self-referential refblocks, but there are
> cases where that is impossible. This adds a test for two of those cases
> (combined in a single test case).
>
> Suggested-by: Eric Blake <eblake@redhat.com>
> Signed-off-by: Max Reitz <mreitz@redhat.com>
> ---
> This patch depends on version 4 (or hopefully any later version) of my
> "qcow2: Support refcount orders != 4" series.
> ---
>   tests/qemu-iotests/115     | 95 ++++++++++++++++++++++++++++++++++++++++++++++
>   tests/qemu-iotests/115.out |  8 ++++
>   tests/qemu-iotests/group   |  1 +
>   3 files changed, 104 insertions(+)
>   create mode 100755 tests/qemu-iotests/115
>   create mode 100644 tests/qemu-iotests/115.out

Applied to my block tree:

https://github.com/XanClic/qemu/commits/block