[Qemu-devel] [PATCH 0/4] qemu-img: support compression regardless of cluster size

[Qemu-devel] [PATCH 0/4] qemu-img: support compression regardless of cluster size

[Stefan Hajnoczi]

It was hard to find a short email subject line.  Anyway, the problem is that
qemu-img convert -c fails with the following error message if the input image
length is not a multiple of the output cluster size:

  qemu-img: error while compressing sector 0: Input/output error

Ilkka Tengvall <ilkka.tengvall@cybercom.com> reported the failure.  kwolf and
eblake suggested a fix which this patch series implements.

qemu-img convert -c succeeds with these patches applied.

The final patch adds qemu-iotests case 053 to protect against regressions.

Stefan Hajnoczi (4):
  qcow2: allow sub-cluster compressed write to last cluster
  qcow: allow sub-cluster compressed write to last cluster
  qemu-img: do not zero-pad the compressed write buffer
  qemu-iotests: add 053 unaligned compressed image size test

 block/qcow.c               | 17 +++++++++++--
 block/qcow2.c              | 17 +++++++++++--
 qemu-img.c                 |  8 ++----
 tests/qemu-iotests/053     | 63 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/053.out | 10 ++++++++
 tests/qemu-iotests/group   |  1 +
 6 files changed, 106 insertions(+), 10 deletions(-)
 create mode 100755 tests/qemu-iotests/053
 create mode 100644 tests/qemu-iotests/053.out

-- 
1.8.1.4

[Qemu-devel] [PATCH 1/4] qcow2: allow sub-cluster compressed write to last cluster

[Stefan Hajnoczi]

Compression in qcow2 requires image length to be a multiple of the
cluster size.  Lift this requirement by zero-padding the final cluster
when necessary.  The virtual disk size is still not cluster-aligned, so
the guest cannot access the zero sectors.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 block/qcow2.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/block/qcow2.c b/block/qcow2.c
index 1d18073..555cd75 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -1537,8 +1537,21 @@ static int qcow2_write_compressed(BlockDriverState *bs, int64_t sector_num,
         return 0;
     }
 
-    if (nb_sectors != s->cluster_sectors)
-        return -EINVAL;
+    if (nb_sectors != s->cluster_sectors) {
+        ret = -EINVAL;
+
+        /* Zero-pad last write if image size is not cluster aligned */
+        if (sector_num + nb_sectors == bs->total_sectors &&
+            nb_sectors < s->cluster_sectors) {
+            uint8_t *pad_buf = qemu_blockalign(bs, s->cluster_size);
+            memset(pad_buf, 0, s->cluster_size);
+            memcpy(pad_buf, buf, nb_sectors * BDRV_SECTOR_SIZE);
+            ret = qcow2_write_compressed(bs, sector_num,
+                                         pad_buf, s->cluster_sectors);
+            qemu_vfree(pad_buf);
+        }
+        return ret;
+    }
 
     out_buf = g_malloc(s->cluster_size + (s->cluster_size / 1000) + 128);
 
-- 
1.8.1.4

[Qemu-devel] [PATCH 2/4] qcow: allow sub-cluster compressed write to last cluster

[Stefan Hajnoczi]

Compression in qcow requires image length to be a multiple of the
cluster size.  Lift this requirement by zero-padding the final cluster
when necessary.  The virtual disk size is still not cluster-aligned, so
the guest cannot access the zero sectors.

Note that this is almost identical to the qcow2 version of this code.
qcow2's compression code is drawn from qcow.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 block/qcow.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/block/qcow.c b/block/qcow.c
index 3278e55..e2a64c7 100644
--- a/block/qcow.c
+++ b/block/qcow.c
@@ -787,8 +787,21 @@ static int qcow_write_compressed(BlockDriverState *bs, int64_t sector_num,
     uint8_t *out_buf;
     uint64_t cluster_offset;
 
-    if (nb_sectors != s->cluster_sectors)
-        return -EINVAL;
+    if (nb_sectors != s->cluster_sectors) {
+        ret = -EINVAL;
+
+        /* Zero-pad last write if image size is not cluster aligned */
+        if (sector_num + nb_sectors == bs->total_sectors &&
+            nb_sectors < s->cluster_sectors) {
+            uint8_t *pad_buf = qemu_blockalign(bs, s->cluster_size);
+            memset(pad_buf, 0, s->cluster_size);
+            memcpy(pad_buf, buf, nb_sectors * BDRV_SECTOR_SIZE);
+            ret = qcow_write_compressed(bs, sector_num,
+                                        pad_buf, s->cluster_sectors);
+            qemu_vfree(pad_buf);
+        }
+        return ret;
+    }
 
     out_buf = g_malloc(s->cluster_size + (s->cluster_size / 1000) + 128);
 
-- 
1.8.1.4

[Qemu-devel] [PATCH 3/4] qemu-img: do not zero-pad the compressed write buffer

[Stefan Hajnoczi]

bdrv_write_compressed() does not allow requests that span the end of the
device.  Therefore it is useless to zero-pad the last cluster and
thereby exceed the end of the device.

Let image formats handle zero-padding the final compressed cluster, if
necessary.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 qemu-img.c | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)

diff --git a/qemu-img.c b/qemu-img.c
index 31627b0..cd096a1 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -1425,12 +1425,8 @@ static int img_convert(int argc, char **argv)
             }
             assert (remainder == 0);
 
-            if (n < cluster_sectors) {
-                memset(buf + n * 512, 0, cluster_size - n * 512);
-            }
-            if (!buffer_is_zero(buf, cluster_size)) {
-                ret = bdrv_write_compressed(out_bs, sector_num, buf,
-                                            cluster_sectors);
+            if (!buffer_is_zero(buf, n * BDRV_SECTOR_SIZE)) {
+                ret = bdrv_write_compressed(out_bs, sector_num, buf, n);
                 if (ret != 0) {
                     error_report("error while compressing sector %" PRId64
                                  ": %s", sector_num, strerror(-ret));
-- 
1.8.1.4

[Qemu-devel] [PATCH 4/4] qemu-iotests: add 053 unaligned compressed image size test

[Stefan Hajnoczi]

Test that qemu-img convert -c works when input image length is not a
multiple of the cluster size.

Previously an error message would be produced:

  qemu-img: error while compressing sector 0: Input/output error

Now that qcow2 and qcow handle this case the test passes successfully.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 tests/qemu-iotests/053     | 63 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/053.out | 10 ++++++++
 tests/qemu-iotests/group   |  1 +
 3 files changed, 74 insertions(+)
 create mode 100755 tests/qemu-iotests/053
 create mode 100644 tests/qemu-iotests/053.out

diff --git a/tests/qemu-iotests/053 b/tests/qemu-iotests/053
new file mode 100755
index 0000000..45d846c
--- /dev/null
+++ b/tests/qemu-iotests/053
@@ -0,0 +1,63 @@
+#!/bin/bash
+#
+# Test qemu-img convert when image length is not a multiple of cluster size
+#
+# Copyright (C) 2013 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
+owner=stefanha@redhat.com
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+
+_cleanup()
+{
+	rm -f $TEST_IMG.orig
+	_cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+_supported_fmt qcow2 qcow
+_supported_proto file
+_supported_os Linux
+
+echo
+echo "== Creating single sector image =="
+
+_make_test_img 512
+$QEMU_IO -c "write -P0xa 0 512" $TEST_IMG | _filter_qemu_io
+mv $TEST_IMG $TEST_IMG.orig
+
+echo
+echo "== Converting the image, compressed =="
+
+$QEMU_IMG convert -c -O $IMGFMT $TEST_IMG.orig $TEST_IMG
+_check_test_img
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
+
diff --git a/tests/qemu-iotests/053.out b/tests/qemu-iotests/053.out
new file mode 100644
index 0000000..1c96b6b
--- /dev/null
+++ b/tests/qemu-iotests/053.out
@@ -0,0 +1,10 @@
+QA output created by 053
+
+== Creating single sector image ==
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=512 
+wrote 512/512 bytes at offset 0
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+== Converting the image, compressed ==
+No errors were found on the image.
+*** done
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index 73c5966..2fba9a4 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -58,3 +58,4 @@
 049 rw auto
 050 rw auto backing quick
 052 rw auto backing
+053 rw auto
-- 
1.8.1.4

Re: [Qemu-devel] [PATCH 4/4] qemu-iotests: add 053 unaligned compressed image size test

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1200 bytes --]

On 04/15/2013 09:17 AM, Stefan Hajnoczi wrote:
> Test that qemu-img convert -c works when input image length is not a
> multiple of the cluster size.
> 
> Previously an error message would be produced:
> 
>   qemu-img: error while compressing sector 0: Input/output error
> 
> Now that qcow2 and qcow handle this case the test passes successfully.
> 
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---

> +echo
> +echo "== Creating single sector image =="
> +
> +_make_test_img 512
> +$QEMU_IO -c "write -P0xa 0 512" $TEST_IMG | _filter_qemu_io
> +mv $TEST_IMG $TEST_IMG.orig
> +
> +echo
> +echo "== Converting the image, compressed =="
> +
> +$QEMU_IMG convert -c -O $IMGFMT $TEST_IMG.orig $TEST_IMG
> +_check_test_img
> +
> +# success, all done

Is it worth also testing that qemu-img info on the converted image still
reports that the guest sees a size of 512, to prove that the virtual
size was not expanded as a result of compression tail padding?

With that question answered, you have:
Series: Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 621 bytes --]

Re: [Qemu-devel] [PATCH 4/4] qemu-iotests: add 053 unaligned compressed image size test

[Stefan Hajnoczi]

On Mon, Apr 15, 2013 at 09:38:28AM -0600, Eric Blake wrote:
> On 04/15/2013 09:17 AM, Stefan Hajnoczi wrote:
> > Test that qemu-img convert -c works when input image length is not a
> > multiple of the cluster size.
> > 
> > Previously an error message would be produced:
> > 
> >   qemu-img: error while compressing sector 0: Input/output error
> > 
> > Now that qcow2 and qcow handle this case the test passes successfully.
> > 
> > Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> > ---
> 
> > +echo
> > +echo "== Creating single sector image =="
> > +
> > +_make_test_img 512
> > +$QEMU_IO -c "write -P0xa 0 512" $TEST_IMG | _filter_qemu_io
> > +mv $TEST_IMG $TEST_IMG.orig
> > +
> > +echo
> > +echo "== Converting the image, compressed =="
> > +
> > +$QEMU_IMG convert -c -O $IMGFMT $TEST_IMG.orig $TEST_IMG
> > +_check_test_img
> > +
> > +# success, all done
> 
> Is it worth also testing that qemu-img info on the converted image still
> reports that the guest sees a size of 512, to prove that the virtual
> size was not expanded as a result of compression tail padding?

Good point.  Let's beef up the test case.  We can also read the first
sector to verify to still contains 0xa bytes.

Stefan

Re: [Qemu-devel] [PATCH 4/4] qemu-iotests: add 053 unaligned compressed image size test

[Kevin Wolf]

Am 16.04.2013 um 10:36 hat Stefan Hajnoczi geschrieben:
> On Mon, Apr 15, 2013 at 09:38:28AM -0600, Eric Blake wrote:
> > On 04/15/2013 09:17 AM, Stefan Hajnoczi wrote:
> > > Test that qemu-img convert -c works when input image length is not a
> > > multiple of the cluster size.
> > > 
> > > Previously an error message would be produced:
> > > 
> > >   qemu-img: error while compressing sector 0: Input/output error
> > > 
> > > Now that qcow2 and qcow handle this case the test passes successfully.
> > > 
> > > Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> > > ---
> > 
> > > +echo
> > > +echo "== Creating single sector image =="
> > > +
> > > +_make_test_img 512
> > > +$QEMU_IO -c "write -P0xa 0 512" $TEST_IMG | _filter_qemu_io
> > > +mv $TEST_IMG $TEST_IMG.orig
> > > +
> > > +echo
> > > +echo "== Converting the image, compressed =="
> > > +
> > > +$QEMU_IMG convert -c -O $IMGFMT $TEST_IMG.orig $TEST_IMG
> > > +_check_test_img
> > > +
> > > +# success, all done
> > 
> > Is it worth also testing that qemu-img info on the converted image still
> > reports that the guest sees a size of 512, to prove that the virtual
> > size was not expanded as a result of compression tail padding?
> 
> Good point.  Let's beef up the test case.  We can also read the first
> sector to verify to still contains 0xa bytes.

Maybe just qemu-img compare -s both images? And you can add the info as
well just in case that compare is broken...

Kevin

Re: [Qemu-devel] [PATCH 0/4] qemu-img: support compression regardless of cluster size

[Kevin Wolf]

Am 15.04.2013 um 17:17 hat Stefan Hajnoczi geschrieben:
> It was hard to find a short email subject line.  Anyway, the problem is that
> qemu-img convert -c fails with the following error message if the input image
> length is not a multiple of the output cluster size:
> 
>   qemu-img: error while compressing sector 0: Input/output error
> 
> Ilkka Tengvall <ilkka.tengvall@cybercom.com> reported the failure.  kwolf and
> eblake suggested a fix which this patch series implements.
> 
> qemu-img convert -c succeeds with these patches applied.
> 
> The final patch adds qemu-iotests case 053 to protect against regressions.

Thanks, applied all to the block branch. If you're going to send an
updated test case, I'll replace it in my tree.

Kevin