[Qemu-devel] [PATCH v2 0/4] qemu-img: support compression regardless of cluster size

[Qemu-devel] [PATCH v2 0/4] qemu-img: support compression regardless of cluster size

[Stefan Hajnoczi]

It was hard to find a short email subject line.  Anyway, the problem is that
qemu-img convert -c fails with the following error message if the input image
length is not a multiple of the output cluster size:

  qemu-img: error while compressing sector 0: Input/output error

Ilkka Tengvall <ilkka.tengvall@cybercom.com> reported the failure.  kwolf and
eblake suggested a fix which this patch series implements.

qemu-img convert -c succeeds with these patches applied.

The final patch adds qemu-iotests case 053 to protect against regressions.

v2:
 * Extend test case to check virtual disk size and contents [eblake]
 * Note: only the last commit changed

Stefan Hajnoczi (4):
  qcow2: allow sub-cluster compressed write to last cluster
  qcow: allow sub-cluster compressed write to last cluster
  qemu-img: do not zero-pad the compressed write buffer
  qemu-iotests: add 053 unaligned compressed image size test

 block/qcow.c               | 17 +++++++++--
 block/qcow2.c              | 17 +++++++++--
 qemu-img.c                 |  8 ++---
 tests/qemu-iotests/053     | 73 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/053.out | 17 +++++++++++
 tests/qemu-iotests/group   |  1 +
 6 files changed, 123 insertions(+), 10 deletions(-)
 create mode 100755 tests/qemu-iotests/053
 create mode 100644 tests/qemu-iotests/053.out

-- 
1.8.1.4

[Qemu-devel] [PATCH v2 1/4] qcow2: allow sub-cluster compressed write to last cluster

[Stefan Hajnoczi]

Compression in qcow2 requires image length to be a multiple of the
cluster size.  Lift this requirement by zero-padding the final cluster
when necessary.  The virtual disk size is still not cluster-aligned, so
the guest cannot access the zero sectors.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 block/qcow2.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/block/qcow2.c b/block/qcow2.c
index e8934de..2e346d8 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -1537,8 +1537,21 @@ static int qcow2_write_compressed(BlockDriverState *bs, int64_t sector_num,
         return 0;
     }
 
-    if (nb_sectors != s->cluster_sectors)
-        return -EINVAL;
+    if (nb_sectors != s->cluster_sectors) {
+        ret = -EINVAL;
+
+        /* Zero-pad last write if image size is not cluster aligned */
+        if (sector_num + nb_sectors == bs->total_sectors &&
+            nb_sectors < s->cluster_sectors) {
+            uint8_t *pad_buf = qemu_blockalign(bs, s->cluster_size);
+            memset(pad_buf, 0, s->cluster_size);
+            memcpy(pad_buf, buf, nb_sectors * BDRV_SECTOR_SIZE);
+            ret = qcow2_write_compressed(bs, sector_num,
+                                         pad_buf, s->cluster_sectors);
+            qemu_vfree(pad_buf);
+        }
+        return ret;
+    }
 
     out_buf = g_malloc(s->cluster_size + (s->cluster_size / 1000) + 128);
 
-- 
1.8.1.4

[Qemu-devel] [PATCH v2 2/4] qcow: allow sub-cluster compressed write to last cluster

[Stefan Hajnoczi]

Compression in qcow requires image length to be a multiple of the
cluster size.  Lift this requirement by zero-padding the final cluster
when necessary.  The virtual disk size is still not cluster-aligned, so
the guest cannot access the zero sectors.

Note that this is almost identical to the qcow2 version of this code.
qcow2's compression code is drawn from qcow.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 block/qcow.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/block/qcow.c b/block/qcow.c
index 3278e55..e2a64c7 100644
--- a/block/qcow.c
+++ b/block/qcow.c
@@ -787,8 +787,21 @@ static int qcow_write_compressed(BlockDriverState *bs, int64_t sector_num,
     uint8_t *out_buf;
     uint64_t cluster_offset;
 
-    if (nb_sectors != s->cluster_sectors)
-        return -EINVAL;
+    if (nb_sectors != s->cluster_sectors) {
+        ret = -EINVAL;
+
+        /* Zero-pad last write if image size is not cluster aligned */
+        if (sector_num + nb_sectors == bs->total_sectors &&
+            nb_sectors < s->cluster_sectors) {
+            uint8_t *pad_buf = qemu_blockalign(bs, s->cluster_size);
+            memset(pad_buf, 0, s->cluster_size);
+            memcpy(pad_buf, buf, nb_sectors * BDRV_SECTOR_SIZE);
+            ret = qcow_write_compressed(bs, sector_num,
+                                        pad_buf, s->cluster_sectors);
+            qemu_vfree(pad_buf);
+        }
+        return ret;
+    }
 
     out_buf = g_malloc(s->cluster_size + (s->cluster_size / 1000) + 128);
 
-- 
1.8.1.4

[Qemu-devel] [PATCH v2 3/4] qemu-img: do not zero-pad the compressed write buffer

[Stefan Hajnoczi]

bdrv_write_compressed() does not allow requests that span the end of the
device.  Therefore it is useless to zero-pad the last cluster and
thereby exceed the end of the device.

Let image formats handle zero-padding the final compressed cluster, if
necessary.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 qemu-img.c | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)

diff --git a/qemu-img.c b/qemu-img.c
index 31627b0..cd096a1 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -1425,12 +1425,8 @@ static int img_convert(int argc, char **argv)
             }
             assert (remainder == 0);
 
-            if (n < cluster_sectors) {
-                memset(buf + n * 512, 0, cluster_size - n * 512);
-            }
-            if (!buffer_is_zero(buf, cluster_size)) {
-                ret = bdrv_write_compressed(out_bs, sector_num, buf,
-                                            cluster_sectors);
+            if (!buffer_is_zero(buf, n * BDRV_SECTOR_SIZE)) {
+                ret = bdrv_write_compressed(out_bs, sector_num, buf, n);
                 if (ret != 0) {
                     error_report("error while compressing sector %" PRId64
                                  ": %s", sector_num, strerror(-ret));
-- 
1.8.1.4

[Qemu-devel] [PATCH v2 4/4] qemu-iotests: add 053 unaligned compressed image size test

[Stefan Hajnoczi]

Test that qemu-img convert -c works when input image length is not a
multiple of the cluster size.

Previously an error message would be produced:

  qemu-img: error while compressing sector 0: Input/output error

Now that qcow2 and qcow handle this case the test passes successfully.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 tests/qemu-iotests/053     | 73 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/053.out | 17 +++++++++++
 tests/qemu-iotests/group   |  1 +
 3 files changed, 91 insertions(+)
 create mode 100755 tests/qemu-iotests/053
 create mode 100644 tests/qemu-iotests/053.out

diff --git a/tests/qemu-iotests/053 b/tests/qemu-iotests/053
new file mode 100755
index 0000000..bc56992
--- /dev/null
+++ b/tests/qemu-iotests/053
@@ -0,0 +1,73 @@
+#!/bin/bash
+#
+# Test qemu-img convert when image length is not a multiple of cluster size
+#
+# Copyright (C) 2013 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
+owner=stefanha@redhat.com
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+
+_cleanup()
+{
+	rm -f $TEST_IMG.orig
+	_cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+_supported_fmt qcow2 qcow
+_supported_proto file
+_supported_os Linux
+
+echo
+echo "== Creating single sector image =="
+
+_make_test_img 512
+$QEMU_IO -c "write -P0xa 0 512" $TEST_IMG | _filter_qemu_io
+mv $TEST_IMG $TEST_IMG.orig
+
+echo
+echo "== Converting the image, compressed =="
+
+$QEMU_IMG convert -c -O $IMGFMT $TEST_IMG.orig $TEST_IMG
+_check_test_img
+
+echo
+echo "== Checking compressed image virtual disk size =="
+
+_img_info | grep '^virtual size:'
+
+echo
+echo "== Verifying the compressed image =="
+
+$QEMU_IO -c "read -P0xa 0 512" $TEST_IMG | _filter_qemu_io
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
+
diff --git a/tests/qemu-iotests/053.out b/tests/qemu-iotests/053.out
new file mode 100644
index 0000000..16464e6
--- /dev/null
+++ b/tests/qemu-iotests/053.out
@@ -0,0 +1,17 @@
+QA output created by 053
+
+== Creating single sector image ==
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=512 
+wrote 512/512 bytes at offset 0
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+== Converting the image, compressed ==
+No errors were found on the image.
+
+== Checking compressed image virtual disk size ==
+virtual size: 512 (512 bytes)
+
+== Verifying the compressed image ==
+read 512/512 bytes at offset 0
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+*** done
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index 324bacb..68eabda 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -59,3 +59,4 @@
 050 rw auto backing quick
 051 rw auto
 052 rw auto backing
+053 rw auto
-- 
1.8.1.4

Re: [Qemu-devel] [PATCH v2 0/4] qemu-img: support compression regardless of cluster size

[Kevin Wolf]

Am 16.04.2013 um 11:14 hat Stefan Hajnoczi geschrieben:
> It was hard to find a short email subject line.  Anyway, the problem is that
> qemu-img convert -c fails with the following error message if the input image
> length is not a multiple of the output cluster size:
> 
>   qemu-img: error while compressing sector 0: Input/output error
> 
> Ilkka Tengvall <ilkka.tengvall@cybercom.com> reported the failure.  kwolf and
> eblake suggested a fix which this patch series implements.
> 
> qemu-img convert -c succeeds with these patches applied.
> 
> The final patch adds qemu-iotests case 053 to protect against regressions.
> 
> v2:
>  * Extend test case to check virtual disk size and contents [eblake]
>  * Note: only the last commit changed

Thanks, updated patch 4/4.

Kevin