* [PATCH v2 0/4] qcow2: Implement zstd cluster compression method
@ 2020-03-02 8:21 Denis Plotnikov
2020-03-02 8:21 ` [PATCH v2 1/4] qcow2: introduce compression type feature Denis Plotnikov
` (4 more replies)
0 siblings, 5 replies; 10+ messages in thread
From: Denis Plotnikov @ 2020-03-02 8:21 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, den, vsementsov, qemu-block, armbru, mreitz
v2:
* rework compression type setting [Vladimir]
* squash iotest changes to the compression type introduction patch [Vladimir, Eric]
* fix zstd availability checking in zstd iotest [Vladimir]
* remove unnecessry casting [Eric]
* remove rudundant checks [Eric]
* fix compressed cluster layout in qcow2 spec [Vladimir]
* fix wording [Eric, Vladimir]
* fix compression type filtering in iotests [Eric]
v1:
the initial series
---
zstd comression method is faster than the only available zlib.
The series adds zstd to the methods available for clusters compression.
The implementation is done with respect to the recently added compression
type additional header to the qcow2 specification.
Denis Plotnikov (4):
qcow2: introduce compression type feature
qcow2: rework the cluster compression routine
qcow2: add zstd cluster compression
iotests: 287: add qcow2 compression type test
docs/interop/qcow2.txt | 20 ++++
configure | 29 +++++
qapi/block-core.json | 23 +++-
block/qcow2.h | 18 ++-
include/block/block_int.h | 1 +
block/qcow2-threads.c | 197 ++++++++++++++++++++++++++++---
block/qcow2.c | 116 ++++++++++++++++++
tests/qemu-iotests/031.out | 14 +--
tests/qemu-iotests/036.out | 4 +-
tests/qemu-iotests/049.out | 102 ++++++++--------
tests/qemu-iotests/060.out | 1 +
tests/qemu-iotests/061.out | 34 +++---
tests/qemu-iotests/065 | 20 ++--
tests/qemu-iotests/080 | 2 +-
tests/qemu-iotests/144.out | 4 +-
tests/qemu-iotests/182.out | 2 +-
tests/qemu-iotests/242.out | 5 +
tests/qemu-iotests/255.out | 8 +-
tests/qemu-iotests/287 | 127 ++++++++++++++++++++
tests/qemu-iotests/287.out | 43 +++++++
tests/qemu-iotests/common.filter | 3 +-
tests/qemu-iotests/group | 1 +
22 files changed, 666 insertions(+), 108 deletions(-)
create mode 100755 tests/qemu-iotests/287
create mode 100644 tests/qemu-iotests/287.out
--
2.17.0
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH v2 1/4] qcow2: introduce compression type feature
2020-03-02 8:21 [PATCH v2 0/4] qcow2: Implement zstd cluster compression method Denis Plotnikov
@ 2020-03-02 8:21 ` Denis Plotnikov
2020-03-02 11:24 ` Vladimir Sementsov-Ogievskiy
2020-03-02 8:21 ` [PATCH v2 2/4] qcow2: rework the cluster compression routine Denis Plotnikov
` (3 subsequent siblings)
4 siblings, 1 reply; 10+ messages in thread
From: Denis Plotnikov @ 2020-03-02 8:21 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, den, vsementsov, qemu-block, armbru, mreitz
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for all the tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feture compression type
backing_file_offset += 56 (8 + 48 -> header_change + fature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 144, 182, 242, 255
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
---
qapi/block-core.json | 22 ++++++-
block/qcow2.h | 18 ++++-
include/block/block_int.h | 1 +
block/qcow2.c | 109 +++++++++++++++++++++++++++++++
tests/qemu-iotests/031.out | 14 ++--
tests/qemu-iotests/036.out | 4 +-
tests/qemu-iotests/049.out | 102 ++++++++++++++---------------
tests/qemu-iotests/060.out | 1 +
tests/qemu-iotests/061.out | 34 ++++++----
tests/qemu-iotests/065 | 20 +++---
tests/qemu-iotests/080 | 2 +-
tests/qemu-iotests/144.out | 4 +-
tests/qemu-iotests/182.out | 2 +-
tests/qemu-iotests/242.out | 5 ++
tests/qemu-iotests/255.out | 8 +--
tests/qemu-iotests/common.filter | 3 +-
16 files changed, 255 insertions(+), 94 deletions(-)
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 85e27bb61f..a67eb8bff4 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -78,6 +78,8 @@
#
# @bitmaps: A list of qcow2 bitmap details (since 4.0)
#
+# @compression-type: the image cluster compression method (since 5.0)
+#
# Since: 1.7
##
{ 'struct': 'ImageInfoSpecificQCow2',
@@ -89,7 +91,8 @@
'*corrupt': 'bool',
'refcount-bits': 'int',
'*encrypt': 'ImageInfoSpecificQCow2Encryption',
- '*bitmaps': ['Qcow2BitmapInfo']
+ '*bitmaps': ['Qcow2BitmapInfo'],
+ 'compression-type': 'Qcow2CompressionType'
} }
##
@@ -4392,6 +4395,18 @@
'data': [ 'v2', 'v3' ] }
+##
+# @Qcow2CompressionType:
+#
+# Compression type used in qcow2 image file
+#
+# @zlib: zlib compression, see <http://zlib.net/>
+#
+# Since: 5.0
+##
+{ 'enum': 'Qcow2CompressionType',
+ 'data': [ 'zlib' ] }
+
##
# @BlockdevCreateOptionsQcow2:
#
@@ -4415,6 +4430,8 @@
# allowed values: off, falloc, full, metadata)
# @lazy-refcounts: True if refcounts may be updated lazily (default: off)
# @refcount-bits: Width of reference counts in bits (default: 16)
+# @compression-type: The image cluster compression method
+# (default: zlib, since 5.0)
#
# Since: 2.12
##
@@ -4430,7 +4447,8 @@
'*cluster-size': 'size',
'*preallocation': 'PreallocMode',
'*lazy-refcounts': 'bool',
- '*refcount-bits': 'int' } }
+ '*refcount-bits': 'int',
+ '*compression-type':'Qcow2CompressionType' } }
##
# @BlockdevCreateOptionsQed:
diff --git a/block/qcow2.h b/block/qcow2.h
index 0942126232..6a43495c27 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -146,6 +146,12 @@ typedef struct QCowHeader {
uint32_t refcount_order;
uint32_t header_length;
+
+ /* Additional fields */
+ uint8_t compression_type;
+
+ /* header must be a multiple of 8 */
+ uint8_t padding[7];
} QEMU_PACKED QCowHeader;
typedef struct QEMU_PACKED QCowSnapshotHeader {
@@ -216,13 +222,16 @@ enum {
QCOW2_INCOMPAT_DIRTY_BITNR = 0,
QCOW2_INCOMPAT_CORRUPT_BITNR = 1,
QCOW2_INCOMPAT_DATA_FILE_BITNR = 2,
+ QCOW2_INCOMPAT_COMPRESSION_BITNR= 3,
QCOW2_INCOMPAT_DIRTY = 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
QCOW2_INCOMPAT_CORRUPT = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR,
QCOW2_INCOMPAT_DATA_FILE = 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR,
+ QCOW2_INCOMPAT_COMPRESSION = 1 << QCOW2_INCOMPAT_COMPRESSION_BITNR,
QCOW2_INCOMPAT_MASK = QCOW2_INCOMPAT_DIRTY
| QCOW2_INCOMPAT_CORRUPT
- | QCOW2_INCOMPAT_DATA_FILE,
+ | QCOW2_INCOMPAT_DATA_FILE
+ | QCOW2_INCOMPAT_COMPRESSION,
};
/* Compatible feature bits */
@@ -369,6 +378,13 @@ typedef struct BDRVQcow2State {
bool metadata_preallocation_checked;
bool metadata_preallocation;
+ /*
+ * Compression type used for the image. Default: 0 - ZLIB
+ * The image compression type is set on image creation.
+ * The only way to change the compression type is to convert the image
+ * with the desired compression type set
+ */
+ Qcow2CompressionType compression_type;
} BDRVQcow2State;
typedef struct Qcow2COWRegion {
diff --git a/include/block/block_int.h b/include/block/block_int.h
index 6f9fd5e20e..2c6bb9dc99 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -57,6 +57,7 @@
#define BLOCK_OPT_REFCOUNT_BITS "refcount_bits"
#define BLOCK_OPT_DATA_FILE "data_file"
#define BLOCK_OPT_DATA_FILE_RAW "data_file_raw"
+#define BLOCK_OPT_COMPRESSION_TYPE "compression_type"
#define BLOCK_PROBE_BUF_SIZE 512
diff --git a/block/qcow2.c b/block/qcow2.c
index 3c754f616b..fc5232a5d6 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -1242,6 +1242,50 @@ static int qcow2_update_options(BlockDriverState *bs, QDict *options,
return ret;
}
+static int validate_compression_type(BDRVQcow2State *s, Error **errp)
+{
+ /*
+ * Sanity check
+ * according to qcow2 spec, the compression type is 1-byte field
+ * but in BDRVQcow2State the compression_type is enum sizeof(int)
+ * so, the max compression_type value is 255.
+ */
+ if (s->compression_type > 0xff) {
+ error_setg(errp, "qcow2: compression type value is too big");
+ return -EINVAL;
+ }
+
+ switch (s->compression_type) {
+ case QCOW2_COMPRESSION_TYPE_ZLIB:
+ break;
+
+ default:
+ error_setg(errp, "qcow2: unknown compression type: %u",
+ s->compression_type);
+ return -ENOTSUP;
+ }
+
+ /*
+ * if the compression type differs from QCOW2_COMPRESSION_TYPE_ZLIB
+ * the incompatible feature flag must be set
+ */
+ if (s->compression_type == QCOW2_COMPRESSION_TYPE_ZLIB) {
+ if (s->incompatible_features & QCOW2_INCOMPAT_COMPRESSION) {
+ error_setg(errp, "qcow2: Compression type incompatible feature "
+ "bit must not be set");
+ return -EINVAL;
+ }
+ } else {
+ if (!(s->incompatible_features & QCOW2_INCOMPAT_COMPRESSION)) {
+ error_setg(errp, "qcow2: Compression type incompatible feature "
+ "bit must be set");
+ return -EINVAL;
+ }
+ }
+
+ return 0;
+}
+
/* Called with s->lock held. */
static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
int flags, Error **errp)
@@ -1357,6 +1401,23 @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
s->compatible_features = header.compatible_features;
s->autoclear_features = header.autoclear_features;
+ /*
+ * Handle compression type
+ * Older qcow2 images don't contain the compression type header.
+ * Distinguish them by the header length and use
+ * the only valid (default) compression type in that case
+ */
+ if (header.header_length > offsetof(QCowHeader, compression_type)) {
+ s->compression_type = header.compression_type;
+ } else {
+ s->compression_type = QCOW2_COMPRESSION_TYPE_ZLIB;
+ }
+
+ ret = validate_compression_type(s, errp);
+ if (ret) {
+ goto fail;
+ }
+
if (s->incompatible_features & ~QCOW2_INCOMPAT_MASK) {
void *feature_table = NULL;
qcow2_read_extensions(bs, header.header_length, ext_end,
@@ -2720,6 +2781,12 @@ int qcow2_update_header(BlockDriverState *bs)
total_size = bs->total_sectors * BDRV_SECTOR_SIZE;
refcount_table_clusters = s->refcount_table_size >> (s->cluster_bits - 3);
+ ret = validate_compression_type(s, NULL);
+
+ if (ret) {
+ goto fail;
+ }
+
*header = (QCowHeader) {
/* Version 2 fields */
.magic = cpu_to_be32(QCOW_MAGIC),
@@ -2742,6 +2809,7 @@ int qcow2_update_header(BlockDriverState *bs)
.autoclear_features = cpu_to_be64(s->autoclear_features),
.refcount_order = cpu_to_be32(s->refcount_order),
.header_length = cpu_to_be32(header_length),
+ .compression_type = s->compression_type,
};
/* For older versions, write a shorter header */
@@ -2834,6 +2902,11 @@ int qcow2_update_header(BlockDriverState *bs)
.bit = QCOW2_INCOMPAT_DATA_FILE_BITNR,
.name = "external data file",
},
+ {
+ .type = QCOW2_FEAT_TYPE_INCOMPATIBLE,
+ .bit = QCOW2_INCOMPAT_COMPRESSION_BITNR,
+ .name = "compression type",
+ },
{
.type = QCOW2_FEAT_TYPE_COMPATIBLE,
.bit = QCOW2_COMPAT_LAZY_REFCOUNTS_BITNR,
@@ -3262,6 +3335,7 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
uint64_t* refcount_table;
Error *local_err = NULL;
int ret;
+ uint8_t compression_type = QCOW2_COMPRESSION_TYPE_ZLIB;
assert(create_options->driver == BLOCKDEV_DRIVER_QCOW2);
qcow2_opts = &create_options->u.qcow2;
@@ -3379,6 +3453,27 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
}
}
+ if (qcow2_opts->has_compression_type &&
+ qcow2_opts->compression_type != QCOW2_COMPRESSION_TYPE_ZLIB) {
+
+ ret = -EINVAL;
+
+ if (version < 3) {
+ error_setg(errp, "Compression type is only supported with "
+ "compatibility level 1.1 and above (use version=v3 or "
+ "greater)");
+ goto out;
+ }
+
+ switch (qcow2_opts->compression_type) {
+ default:
+ error_setg(errp, "Unknown compression type");
+ goto out;
+ }
+
+ compression_type = qcow2_opts->compression_type;
+ }
+
/* Create BlockBackend to write to the image */
blk = blk_new(bdrv_get_aio_context(bs),
BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL);
@@ -3401,6 +3496,8 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
.refcount_table_offset = cpu_to_be64(cluster_size),
.refcount_table_clusters = cpu_to_be32(1),
.refcount_order = cpu_to_be32(refcount_order),
+ /* don't deal with endians since compression_type is 1 byte long */
+ .compression_type = compression_type,
.header_length = cpu_to_be32(sizeof(*header)),
};
@@ -3419,6 +3516,10 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
header->autoclear_features |=
cpu_to_be64(QCOW2_AUTOCLEAR_DATA_FILE_RAW);
}
+ if (compression_type != QCOW2_COMPRESSION_TYPE_ZLIB) {
+ header->incompatible_features |=
+ cpu_to_be64(QCOW2_INCOMPAT_COMPRESSION);
+ }
ret = blk_pwrite(blk, 0, header, cluster_size, 0);
g_free(header);
@@ -3602,6 +3703,7 @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt
{ BLOCK_OPT_ENCRYPT, BLOCK_OPT_ENCRYPT_FORMAT },
{ BLOCK_OPT_COMPAT_LEVEL, "version" },
{ BLOCK_OPT_DATA_FILE_RAW, "data-file-raw" },
+ { BLOCK_OPT_COMPRESSION_TYPE, "compression-type" },
{ NULL, NULL },
};
@@ -4859,6 +4961,7 @@ static ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *bs,
.data_file = g_strdup(s->image_data_file),
.has_data_file_raw = has_data_file(bs),
.data_file_raw = data_file_is_raw(bs),
+ .compression_type = s->compression_type,
};
} else {
/* if this assertion fails, this probably means a new version was
@@ -5516,6 +5619,12 @@ static QemuOptsList qcow2_create_opts = {
.help = "Width of a reference count entry in bits",
.def_value_str = "16"
},
+ {
+ .name = BLOCK_OPT_COMPRESSION_TYPE,
+ .type = QEMU_OPT_STRING,
+ .help = "Compression method used for image clusters compression",
+ .def_value_str = "zlib"
+ },
{ /* end of list */ }
}
};
diff --git a/tests/qemu-iotests/031.out b/tests/qemu-iotests/031.out
index d535e407bc..ed51afe9ce 100644
--- a/tests/qemu-iotests/031.out
+++ b/tests/qemu-iotests/031.out
@@ -113,11 +113,11 @@ incompatible_features []
compatible_features []
autoclear_features []
refcount_order 4
-header_length 104
+header_length 112
Header extension:
magic 0x6803f857
-length 192
+length 240
data <binary>
Header extension:
@@ -146,11 +146,11 @@ incompatible_features []
compatible_features []
autoclear_features []
refcount_order 4
-header_length 104
+header_length 112
Header extension:
magic 0x6803f857
-length 192
+length 240
data <binary>
Header extension:
@@ -164,7 +164,7 @@ No errors were found on the image.
magic 0x514649fb
version 3
-backing_file_offset 0x178
+backing_file_offset 0x1b0
backing_file_size 0x17
cluster_bits 16
size 67108864
@@ -179,7 +179,7 @@ incompatible_features []
compatible_features []
autoclear_features []
refcount_order 4
-header_length 104
+header_length 112
Header extension:
magic 0xe2792aca
@@ -188,7 +188,7 @@ data 'host_device'
Header extension:
magic 0x6803f857
-length 192
+length 240
data <binary>
Header extension:
diff --git a/tests/qemu-iotests/036.out b/tests/qemu-iotests/036.out
index 0b52b934e1..fb509f6357 100644
--- a/tests/qemu-iotests/036.out
+++ b/tests/qemu-iotests/036.out
@@ -26,7 +26,7 @@ compatible_features []
autoclear_features [63]
Header extension:
magic 0x6803f857
-length 192
+length 240
data <binary>
@@ -38,7 +38,7 @@ compatible_features []
autoclear_features []
Header extension:
magic 0x6803f857
-length 192
+length 240
data <binary>
*** done
diff --git a/tests/qemu-iotests/049.out b/tests/qemu-iotests/049.out
index affa55b341..a5cfba1756 100644
--- a/tests/qemu-iotests/049.out
+++ b/tests/qemu-iotests/049.out
@@ -4,90 +4,90 @@ QA output created by 049
== 1. Traditional size parameter ==
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024b
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1k
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1K
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1G
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1T
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024.0
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024.0b
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5k
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5K
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5G
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5T
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
== 2. Specifying size via -o ==
qemu-img create -f qcow2 -o size=1024 TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1024b TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1k TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1K TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1M TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1G TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1T TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1024.0 TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1024.0b TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1.5k TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1.5K TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1.5M TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1.5G TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1.5T TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
== 3. Invalid sizes ==
@@ -129,84 +129,84 @@ qemu-img: TEST_DIR/t.qcow2: The image size must be specified only once
== Check correct interpretation of suffixes for cluster size ==
qemu-img create -f qcow2 -o cluster_size=1024 TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=1024b TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=1k TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=1K TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=1M TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1048576 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1048576 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=1024.0 TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=1024.0b TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=0.5k TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=0.5K TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=0.5M TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=524288 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=524288 lazy_refcounts=off refcount_bits=16 compression_type=zlib
== Check compat level option ==
qemu-img create -f qcow2 -o compat=0.10 TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o compat=1.1 TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o compat=0.42 TEST_DIR/t.qcow2 64M
qemu-img: TEST_DIR/t.qcow2: Invalid parameter '0.42'
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.42 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.42 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o compat=foobar TEST_DIR/t.qcow2 64M
qemu-img: TEST_DIR/t.qcow2: Invalid parameter 'foobar'
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=foobar cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=foobar cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
== Check preallocation option ==
qemu-img create -f qcow2 -o preallocation=off TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=off lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=off lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o preallocation=metadata TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=metadata lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=metadata lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o preallocation=1234 TEST_DIR/t.qcow2 64M
qemu-img: TEST_DIR/t.qcow2: Invalid parameter '1234'
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=1234 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=1234 lazy_refcounts=off refcount_bits=16 compression_type=zlib
== Check encryption option ==
qemu-img create -f qcow2 -o encryption=off TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 --object secret,id=sec0,data=123456 -o encryption=on,encrypt.key-secret=sec0 TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=on encrypt.key-secret=sec0 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=on encrypt.key-secret=sec0 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
== Check lazy_refcounts option (only with v3) ==
qemu-img create -f qcow2 -o compat=1.1,lazy_refcounts=off TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o compat=1.1,lazy_refcounts=on TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=on refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=on refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o compat=0.10,lazy_refcounts=off TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o compat=0.10,lazy_refcounts=on TEST_DIR/t.qcow2 64M
qemu-img: TEST_DIR/t.qcow2: Lazy refcounts only supported with compatibility level 1.1 and above (use version=v3 or greater)
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=on refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=on refcount_bits=16 compression_type=zlib
*** done
diff --git a/tests/qemu-iotests/060.out b/tests/qemu-iotests/060.out
index d27692a33c..3e47f537e8 100644
--- a/tests/qemu-iotests/060.out
+++ b/tests/qemu-iotests/060.out
@@ -17,6 +17,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
corrupt: true
diff --git a/tests/qemu-iotests/061.out b/tests/qemu-iotests/061.out
index 8b3091a412..c913f02ad6 100644
--- a/tests/qemu-iotests/061.out
+++ b/tests/qemu-iotests/061.out
@@ -22,11 +22,11 @@ incompatible_features []
compatible_features [0]
autoclear_features []
refcount_order 4
-header_length 104
+header_length 112
Header extension:
magic 0x6803f857
-length 192
+length 240
data <binary>
magic 0x514649fb
@@ -80,11 +80,11 @@ incompatible_features []
compatible_features [0]
autoclear_features []
refcount_order 4
-header_length 104
+header_length 112
Header extension:
magic 0x6803f857
-length 192
+length 240
data <binary>
magic 0x514649fb
@@ -136,11 +136,11 @@ incompatible_features [0]
compatible_features [0]
autoclear_features []
refcount_order 4
-header_length 104
+header_length 112
Header extension:
magic 0x6803f857
-length 192
+length 240
data <binary>
ERROR cluster 5 refcount=0 reference=1
@@ -191,11 +191,11 @@ incompatible_features []
compatible_features [42]
autoclear_features [42]
refcount_order 4
-header_length 104
+header_length 112
Header extension:
magic 0x6803f857
-length 192
+length 240
data <binary>
magic 0x514649fb
@@ -260,11 +260,11 @@ incompatible_features []
compatible_features [0]
autoclear_features []
refcount_order 4
-header_length 104
+header_length 112
Header extension:
magic 0x6803f857
-length 192
+length 240
data <binary>
read 65536/65536 bytes at offset 44040192
@@ -294,11 +294,11 @@ incompatible_features [0]
compatible_features [0]
autoclear_features []
refcount_order 4
-header_length 104
+header_length 112
Header extension:
magic 0x6803f857
-length 192
+length 240
data <binary>
ERROR cluster 5 refcount=0 reference=1
@@ -323,11 +323,11 @@ incompatible_features []
compatible_features []
autoclear_features []
refcount_order 4
-header_length 104
+header_length 112
Header extension:
magic 0x6803f857
-length 192
+length 240
data <binary>
read 131072/131072 bytes at offset 0
@@ -491,6 +491,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
data file: TEST_DIR/t.IMGFMT.data
@@ -511,6 +512,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
data file: foo
@@ -524,6 +526,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
data file raw: false
@@ -538,6 +541,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
data file: TEST_DIR/t.IMGFMT.data
@@ -550,6 +554,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
data file: TEST_DIR/t.IMGFMT.data
@@ -563,6 +568,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
data file: TEST_DIR/t.IMGFMT.data
diff --git a/tests/qemu-iotests/065 b/tests/qemu-iotests/065
index 6426474271..106303b5a5 100755
--- a/tests/qemu-iotests/065
+++ b/tests/qemu-iotests/065
@@ -88,23 +88,25 @@ class TestQMP(TestImageInfoSpecific):
class TestQCow2(TestQemuImgInfo):
'''Testing a qcow2 version 2 image'''
img_options = 'compat=0.10'
- json_compare = { 'compat': '0.10', 'refcount-bits': 16 }
- human_compare = [ 'compat: 0.10', 'refcount bits: 16' ]
+ json_compare = { 'compat': '0.10', 'refcount-bits': 16, 'compression-type': 'zlib' }
+ human_compare = [ 'compat: 0.10', 'compression type: zlib', 'refcount bits: 16' ]
class TestQCow3NotLazy(TestQemuImgInfo):
'''Testing a qcow2 version 3 image with lazy refcounts disabled'''
img_options = 'compat=1.1,lazy_refcounts=off'
json_compare = { 'compat': '1.1', 'lazy-refcounts': False,
- 'refcount-bits': 16, 'corrupt': False }
- human_compare = [ 'compat: 1.1', 'lazy refcounts: false',
+ 'refcount-bits': 16, 'corrupt': False,
+ 'compression-type': 'zlib' }
+ human_compare = [ 'compat: 1.1', 'compression type: zlib', 'lazy refcounts: false',
'refcount bits: 16', 'corrupt: false' ]
class TestQCow3Lazy(TestQemuImgInfo):
'''Testing a qcow2 version 3 image with lazy refcounts enabled'''
img_options = 'compat=1.1,lazy_refcounts=on'
json_compare = { 'compat': '1.1', 'lazy-refcounts': True,
- 'refcount-bits': 16, 'corrupt': False }
- human_compare = [ 'compat: 1.1', 'lazy refcounts: true',
+ 'refcount-bits': 16, 'corrupt': False,
+ 'compression-type': 'zlib' }
+ human_compare = [ 'compat: 1.1', 'compression type: zlib', 'lazy refcounts: true',
'refcount bits: 16', 'corrupt: false' ]
class TestQCow3NotLazyQMP(TestQMP):
@@ -113,7 +115,8 @@ class TestQCow3NotLazyQMP(TestQMP):
img_options = 'compat=1.1,lazy_refcounts=off'
qemu_options = 'lazy-refcounts=on'
compare = { 'compat': '1.1', 'lazy-refcounts': False,
- 'refcount-bits': 16, 'corrupt': False }
+ 'refcount-bits': 16, 'corrupt': False,
+ 'compression-type': 'zlib' }
class TestQCow3LazyQMP(TestQMP):
@@ -122,7 +125,8 @@ class TestQCow3LazyQMP(TestQMP):
img_options = 'compat=1.1,lazy_refcounts=on'
qemu_options = 'lazy-refcounts=off'
compare = { 'compat': '1.1', 'lazy-refcounts': True,
- 'refcount-bits': 16, 'corrupt': False }
+ 'refcount-bits': 16, 'corrupt': False,
+ 'compression-type': 'zlib' }
TestImageInfoSpecific = None
TestQemuImgInfo = None
diff --git a/tests/qemu-iotests/080 b/tests/qemu-iotests/080
index a3d13c414e..7588c63b6c 100755
--- a/tests/qemu-iotests/080
+++ b/tests/qemu-iotests/080
@@ -45,7 +45,7 @@ _supported_os Linux
# - This is generally a test for compat=1.1 images
_unsupported_imgopts 'refcount_bits=1[^0-9]' data_file 'compat=0.10'
-header_size=104
+header_size=112
offset_backing_file_offset=8
offset_backing_file_size=16
diff --git a/tests/qemu-iotests/144.out b/tests/qemu-iotests/144.out
index c7aa2e4820..885a8874a5 100644
--- a/tests/qemu-iotests/144.out
+++ b/tests/qemu-iotests/144.out
@@ -9,7 +9,7 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=536870912
{ 'execute': 'qmp_capabilities' }
{"return": {}}
{ 'execute': 'blockdev-snapshot-sync', 'arguments': { 'device': 'virtio0', 'snapshot-file':'TEST_DIR/tmp.IMGFMT', 'format': 'IMGFMT' } }
-Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
{"return": {}}
=== Performing block-commit on active layer ===
@@ -31,6 +31,6 @@ Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/
=== Performing Live Snapshot 2 ===
{ 'execute': 'blockdev-snapshot-sync', 'arguments': { 'device': 'virtio0', 'snapshot-file':'TEST_DIR/tmp2.IMGFMT', 'format': 'IMGFMT' } }
-Formatting 'TEST_DIR/tmp2.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/tmp2.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
{"return": {}}
*** done
diff --git a/tests/qemu-iotests/182.out b/tests/qemu-iotests/182.out
index a8eea166c3..ae43654d32 100644
--- a/tests/qemu-iotests/182.out
+++ b/tests/qemu-iotests/182.out
@@ -13,7 +13,7 @@ Is another process using the image [TEST_DIR/t.qcow2]?
{'execute': 'blockdev-add', 'arguments': { 'node-name': 'node0', 'driver': 'file', 'filename': 'TEST_DIR/t.IMGFMT', 'locking': 'on' } }
{"return": {}}
{'execute': 'blockdev-snapshot-sync', 'arguments': { 'node-name': 'node0', 'snapshot-file': 'TEST_DIR/t.IMGFMT.overlay', 'snapshot-node-name': 'node1' } }
-Formatting 'TEST_DIR/t.qcow2.overlay', fmt=qcow2 size=197120 backing_file=TEST_DIR/t.qcow2 backing_fmt=file cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2.overlay', fmt=qcow2 size=197120 backing_file=TEST_DIR/t.qcow2 backing_fmt=file cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
{"return": {}}
{'execute': 'blockdev-add', 'arguments': { 'node-name': 'node1', 'driver': 'file', 'filename': 'TEST_DIR/t.IMGFMT', 'locking': 'on' } }
{"return": {}}
diff --git a/tests/qemu-iotests/242.out b/tests/qemu-iotests/242.out
index 7ac8404d11..091b9126ce 100644
--- a/tests/qemu-iotests/242.out
+++ b/tests/qemu-iotests/242.out
@@ -12,6 +12,7 @@ virtual size: 1 MiB (1048576 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
corrupt: false
@@ -32,6 +33,7 @@ virtual size: 1 MiB (1048576 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
bitmaps:
[0]:
@@ -64,6 +66,7 @@ virtual size: 1 MiB (1048576 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
bitmaps:
[0]:
@@ -104,6 +107,7 @@ virtual size: 1 MiB (1048576 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
bitmaps:
[0]:
@@ -153,6 +157,7 @@ virtual size: 1 MiB (1048576 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
bitmaps:
[0]:
diff --git a/tests/qemu-iotests/255.out b/tests/qemu-iotests/255.out
index 348909fdef..a3c99fd62e 100644
--- a/tests/qemu-iotests/255.out
+++ b/tests/qemu-iotests/255.out
@@ -3,9 +3,9 @@ Finishing a commit job with background reads
=== Create backing chain and start VM ===
-Formatting 'TEST_DIR/PID-t.qcow2.mid', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/PID-t.qcow2.mid', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
-Formatting 'TEST_DIR/PID-t.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/PID-t.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
=== Start background read requests ===
@@ -23,9 +23,9 @@ Closing the VM while a job is being cancelled
=== Create images and start VM ===
-Formatting 'TEST_DIR/PID-src.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/PID-src.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
-Formatting 'TEST_DIR/PID-dst.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/PID-dst.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
wrote 1048576/1048576 bytes at offset 0
1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.filter
index 3f8ee3e5f7..279e0bbb0d 100644
--- a/tests/qemu-iotests/common.filter
+++ b/tests/qemu-iotests/common.filter
@@ -152,7 +152,8 @@ _filter_img_create()
-e "s# refcount_bits=[0-9]\\+##g" \
-e "s# key-secret=[a-zA-Z0-9]\\+##g" \
-e "s# iter-time=[0-9]\\+##g" \
- -e "s# force_size=\\(on\\|off\\)##g"
+ -e "s# force_size=\\(on\\|off\\)##g" \
+ -e "s# compression_type=[a-zA-Z0-9]\\+##g"
}
_filter_img_info()
--
2.17.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH v2 2/4] qcow2: rework the cluster compression routine
2020-03-02 8:21 [PATCH v2 0/4] qcow2: Implement zstd cluster compression method Denis Plotnikov
2020-03-02 8:21 ` [PATCH v2 1/4] qcow2: introduce compression type feature Denis Plotnikov
@ 2020-03-02 8:21 ` Denis Plotnikov
2020-03-02 13:02 ` Vladimir Sementsov-Ogievskiy
2020-03-02 8:21 ` [PATCH v2 3/4] qcow2: add zstd cluster compression Denis Plotnikov
` (2 subsequent siblings)
4 siblings, 1 reply; 10+ messages in thread
From: Denis Plotnikov @ 2020-03-02 8:21 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, den, vsementsov, qemu-block, armbru, mreitz
The patch enables processing the image compression type defined
for the image and chooses an appropriate method for image clusters
(de)compression.
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
---
block/qcow2-threads.c | 77 +++++++++++++++++++++++++++++++++++--------
1 file changed, 63 insertions(+), 14 deletions(-)
diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
index 77bb578cdf..9288a4f852 100644
--- a/block/qcow2-threads.c
+++ b/block/qcow2-threads.c
@@ -74,7 +74,9 @@ typedef struct Qcow2CompressData {
} Qcow2CompressData;
/*
- * qcow2_compress()
+ * qcow2_zlib_compress()
+ *
+ * Compress @src_size bytes of data using zlib compression method
*
* @dest - destination buffer, @dest_size bytes
* @src - source buffer, @src_size bytes
@@ -83,8 +85,8 @@ typedef struct Qcow2CompressData {
* -ENOMEM destination buffer is not enough to store compressed data
* -EIO on any other error
*/
-static ssize_t qcow2_compress(void *dest, size_t dest_size,
- const void *src, size_t src_size)
+static ssize_t qcow2_zlib_compress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
{
ssize_t ret;
z_stream strm;
@@ -119,19 +121,19 @@ static ssize_t qcow2_compress(void *dest, size_t dest_size,
}
/*
- * qcow2_decompress()
+ * qcow2_zlib_decompress()
*
* Decompress some data (not more than @src_size bytes) to produce exactly
- * @dest_size bytes.
+ * @dest_size bytes using zlib compression method
*
* @dest - destination buffer, @dest_size bytes
* @src - source buffer, @src_size bytes
*
* Returns: 0 on success
- * -1 on fail
+ * -EIO on failure
*/
-static ssize_t qcow2_decompress(void *dest, size_t dest_size,
- const void *src, size_t src_size)
+static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
{
int ret = 0;
z_stream strm;
@@ -144,7 +146,7 @@ static ssize_t qcow2_decompress(void *dest, size_t dest_size,
ret = inflateInit2(&strm, -12);
if (ret != Z_OK) {
- return -1;
+ return -EIO;
}
ret = inflate(&strm, Z_FINISH);
@@ -154,7 +156,7 @@ static ssize_t qcow2_decompress(void *dest, size_t dest_size,
* @src buffer may be processed partly (because in qcow2 we know size of
* compressed data with precision of one sector)
*/
- ret = -1;
+ ret = -EIO;
}
inflateEnd(&strm);
@@ -189,20 +191,67 @@ qcow2_co_do_compress(BlockDriverState *bs, void *dest, size_t dest_size,
return arg.ret;
}
+/*
+ * qcow2_co_compress()
+ *
+ * Compress @src_size bytes of data using the compression
+ * method defined by the image compression type
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: compressed size on success
+ * a negative error code on failure
+ */
ssize_t coroutine_fn
qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size,
const void *src, size_t src_size)
{
- return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
- qcow2_compress);
+ BDRVQcow2State *s = bs->opaque;
+ Qcow2CompressFunc fn;
+
+ switch (s->compression_type) {
+ case QCOW2_COMPRESSION_TYPE_ZLIB:
+ fn = qcow2_zlib_compress;
+ break;
+
+ default:
+ abort();
+ }
+
+ return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
}
+/*
+ * qcow2_co_decompress()
+ *
+ * Decompress some data (not more than @src_size bytes) to produce exactly
+ * @dest_size bytes using the compression method defined by the image
+ * compression type
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ * a negative error code on failure
+ */
ssize_t coroutine_fn
qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size,
const void *src, size_t src_size)
{
- return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
- qcow2_decompress);
+ BDRVQcow2State *s = bs->opaque;
+ Qcow2CompressFunc fn;
+
+ switch (s->compression_type) {
+ case QCOW2_COMPRESSION_TYPE_ZLIB:
+ fn = qcow2_zlib_decompress;
+ break;
+
+ default:
+ return -ENOTSUP;
+ }
+
+ return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
}
--
2.17.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH v2 3/4] qcow2: add zstd cluster compression
2020-03-02 8:21 [PATCH v2 0/4] qcow2: Implement zstd cluster compression method Denis Plotnikov
2020-03-02 8:21 ` [PATCH v2 1/4] qcow2: introduce compression type feature Denis Plotnikov
2020-03-02 8:21 ` [PATCH v2 2/4] qcow2: rework the cluster compression routine Denis Plotnikov
@ 2020-03-02 8:21 ` Denis Plotnikov
2020-03-02 8:21 ` [PATCH v2 4/4] iotests: 287: add qcow2 compression type test Denis Plotnikov
2020-03-02 8:51 ` [PATCH v2 0/4] qcow2: Implement zstd cluster compression method Vladimir Sementsov-Ogievskiy
4 siblings, 0 replies; 10+ messages in thread
From: Denis Plotnikov @ 2020-03-02 8:21 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, den, vsementsov, qemu-block, armbru, mreitz
zstd significantly reduces cluster compression time.
It provides better compression performance maintaining
the same level of the compression ratio in comparison with
zlib, which, at the moment, is the only compression
method available.
The performance test results:
Test compresses and decompresses qemu qcow2 image with just
installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G
The test was conducted with brd disk to reduce the influence
of disk subsystem to the test results.
The results is given in seconds.
compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
src.img [zlib|zstd]_compressed.img
decompress cmd
time ./qemu-img convert -O qcow2
[zlib|zstd]_compressed.img uncompressed.img
compression decompression
zlib zstd zlib zstd
------------------------------------------------------------
real 65.5 16.3 (-75 %) 1.9 1.6 (-16 %)
user 65.0 15.8 5.3 2.5
sys 3.3 0.2 2.0 2.0
Both ZLIB and ZSTD gave the same compression ratio: 1.57
compressed image size in both cases: 1.4G
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
---
docs/interop/qcow2.txt | 20 +++++++
configure | 29 ++++++++++
qapi/block-core.json | 3 +-
block/qcow2-threads.c | 120 +++++++++++++++++++++++++++++++++++++++++
block/qcow2.c | 7 +++
5 files changed, 178 insertions(+), 1 deletion(-)
diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt
index 5597e24474..9048114445 100644
--- a/docs/interop/qcow2.txt
+++ b/docs/interop/qcow2.txt
@@ -208,6 +208,7 @@ version 2.
Available compression type values:
0: zlib <https://www.zlib.net/>
+ 1: zstd <http://github.com/facebook/zstd>
=== Header padding ===
@@ -575,11 +576,30 @@ Compressed Clusters Descriptor (x = 62 - (cluster_bits - 8)):
Another compressed cluster may map to the tail of the final
sector used by this compressed cluster.
+ The layout of the compressed data depends on the compression
+ type used for the image (see compressed cluster layout).
+
If a cluster is unallocated, read requests shall read the data from the backing
file (except if bit 0 in the Standard Cluster Descriptor is set). If there is
no backing file or the backing file is smaller than the image, they shall read
zeros for all parts that are not covered by the backing file.
+=== Compressed Cluster Layout ===
+
+The compressed cluster data has a layout depending on the compression
+type used for the image, as follows:
+
+Compressed data layout for the available compression types:
+data_space_lenght - data chunk length available to store a compressed cluster.
+(for more details see "Compressed Clusters Descriptor")
+x = data_space_length - 1
+
+ 0: (default) zlib <http://zlib.net/>:
+ Byte 0 - x: the compressed data content
+ all the space provided used for compressed data
+ 1: zstd <http://github.com/facebook/zstd>:
+ Byte 0 - 3: the length of compressed data in bytes
+ 4 - x: the compressed data content
== Snapshots ==
diff --git a/configure b/configure
index 48d6f89d57..4690a7ea9f 100755
--- a/configure
+++ b/configure
@@ -444,6 +444,7 @@ opengl_dmabuf="no"
cpuid_h="no"
avx2_opt=""
zlib="yes"
+zstd=""
capstone=""
lzo=""
snappy=""
@@ -1371,6 +1372,10 @@ for opt do
;;
--disable-lzfse) lzfse="no"
;;
+ --enable-zstd) zstd="yes"
+ ;;
+ --disable-zstd) zstd="no"
+ ;;
--enable-guest-agent) guest_agent="yes"
;;
--disable-guest-agent) guest_agent="no"
@@ -1829,6 +1834,7 @@ disabled with --disable-FEATURE, default is enabled if available:
(for reading bzip2-compressed dmg images)
lzfse support of lzfse compression library
(for reading lzfse-compressed dmg images)
+ zstd support of zstd compression library
seccomp seccomp support
coroutine-pool coroutine freelist (better performance)
glusterfs GlusterFS backend
@@ -2453,6 +2459,25 @@ EOF
fi
fi
+#########################################
+# zstd check
+
+if test "$zstd" != "no" ; then
+ cat > $TMPC << EOF
+#include <zstd.h>
+int main(void) { ZSTD_versionNumber(); return 0; }
+EOF
+ if compile_prog "" "-lzstd" ; then
+ LIBS="$LIBS -lzstd"
+ zstd="yes"
+ else
+ if test "$zstd" = "yes"; then
+ feature_not_found "zstd" "Install libzstd-devel"
+ fi
+ zstd="no"
+ fi
+fi
+
##########################################
# libseccomp check
@@ -6668,6 +6693,7 @@ echo "lzo support $lzo"
echo "snappy support $snappy"
echo "bzip2 support $bzip2"
echo "lzfse support $lzfse"
+echo "zstd support $zstd"
echo "NUMA host support $numa"
echo "libxml2 $libxml2"
echo "tcmalloc support $tcmalloc"
@@ -7559,6 +7585,9 @@ if test "$plugins" = "yes" ; then
"\$ld_exported_symbols_list should have been set to 'yes'."
fi
fi
+if test "$zstd" = "yes" ; then
+ echo "CONFIG_ZSTD=y" >> $config_host_mak
+fi
if test "$tcg_interpreter" = "yes"; then
QEMU_INCLUDES="-iquote \$(SRC_PATH)/tcg/tci $QEMU_INCLUDES"
diff --git a/qapi/block-core.json b/qapi/block-core.json
index a67eb8bff4..84889fb741 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -4401,11 +4401,12 @@
# Compression type used in qcow2 image file
#
# @zlib: zlib compression, see <http://zlib.net/>
+# @zstd: zstd compression, see <http://github.com/facebook/zstd>
#
# Since: 5.0
##
{ 'enum': 'Qcow2CompressionType',
- 'data': [ 'zlib' ] }
+ 'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }
##
# @BlockdevCreateOptionsQcow2:
diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
index 9288a4f852..c52e985525 100644
--- a/block/qcow2-threads.c
+++ b/block/qcow2-threads.c
@@ -28,6 +28,11 @@
#define ZLIB_CONST
#include <zlib.h>
+#ifdef CONFIG_ZSTD
+#include <zstd.h>
+#include <zstd_errors.h>
+#endif
+
#include "qcow2.h"
#include "block/thread-pool.h"
#include "crypto.h"
@@ -164,6 +169,111 @@ static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,
return ret;
}
+#ifdef CONFIG_ZSTD
+
+#define ZSTD_LEN_BUF 4
+
+/*
+ * qcow2_zstd_compress()
+ *
+ * Compress @src_size bytes of data using zstd compression method
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: compressed size on success
+ * -ENOMEM destination buffer is not enough to store compressed data
+ * -EIO on any other error
+ */
+static ssize_t qcow2_zstd_compress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
+{
+ size_t ret;
+
+ /*
+ * steal ZSTD_LEN_BUF bytes in the very beginning of the buffer
+ * to store compressed chunk size
+ */
+ char *d_buf = ((char *) dest) + ZSTD_LEN_BUF;
+
+ /*
+ * sanity check that we can store the compressed data length,
+ * and there is some space left for the compressor buffer
+ */
+ if (dest_size <= ZSTD_LEN_BUF) {
+ return -ENOMEM;
+ }
+
+ dest_size -= ZSTD_LEN_BUF;
+
+ ret = ZSTD_compress(d_buf, dest_size, src, src_size, 5);
+
+ if (ZSTD_isError(ret)) {
+ if (ZSTD_getErrorCode(ret) == ZSTD_error_dstSize_tooSmall) {
+ return -ENOMEM;
+ } else {
+ return -EIO;
+ }
+ }
+
+ /* paranoid sanity check that we can store the compressed size in the first 4 bytes */
+ if (ret > UINT32_MAX) {
+ return -ENOMEM;
+ }
+
+ /* store the compressed chunk size in the very beginning of the buffer */
+ stl_be_p(dest, ret);
+
+ return ret + ZSTD_LEN_BUF;
+}
+
+/*
+ * qcow2_zstd_decompress()
+ *
+ * Decompress some data (not more than @src_size bytes) to produce exactly
+ * @dest_size bytes using zstd compression method
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ * -EIO on any error
+ */
+static ssize_t qcow2_zstd_decompress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
+{
+ /*
+ * zstd decompress wants to know the exact length of the data.
+ * For that purpose, on compression, the length is stored in
+ * the very beginning of the compressed buffer
+ */
+ size_t s_size;
+ const char *s_buf = ((const char *) src) + ZSTD_LEN_BUF;
+
+ /*
+ * sanity check that we can read 4 byte the content length and
+ * and there is some content to decompress
+ */
+ if (src_size <= ZSTD_LEN_BUF) {
+ return -EIO;
+ }
+
+ s_size = ldl_be_p(src);
+
+ /* sanity check that the buffer is big enough to read the content from */
+ if (src_size - ZSTD_LEN_BUF < s_size) {
+ return -EIO;
+ }
+
+ if (ZSTD_isError(
+ ZSTD_decompress(dest, dest_size, s_buf, s_size))) {
+ return -EIO;
+ }
+
+ return 0;
+}
+#endif
+
static int qcow2_compress_pool_func(void *opaque)
{
Qcow2CompressData *data = opaque;
@@ -215,6 +325,11 @@ qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size,
fn = qcow2_zlib_compress;
break;
+#ifdef CONFIG_ZSTD
+ case QCOW2_COMPRESSION_TYPE_ZSTD:
+ fn = qcow2_zstd_compress;
+ break;
+#endif
default:
abort();
}
@@ -247,6 +362,11 @@ qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size,
fn = qcow2_zlib_decompress;
break;
+#ifdef CONFIG_ZSTD
+ case QCOW2_COMPRESSION_TYPE_ZSTD:
+ fn = qcow2_zstd_decompress;
+ break;
+#endif
default:
return -ENOTSUP;
}
diff --git a/block/qcow2.c b/block/qcow2.c
index fc5232a5d6..bb0bad5d56 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -1257,6 +1257,9 @@ static int validate_compression_type(BDRVQcow2State *s, Error **errp)
switch (s->compression_type) {
case QCOW2_COMPRESSION_TYPE_ZLIB:
+#ifdef CONFIG_ZSTD
+ case QCOW2_COMPRESSION_TYPE_ZSTD:
+#endif
break;
default:
@@ -3466,6 +3469,10 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
}
switch (qcow2_opts->compression_type) {
+#ifdef CONFIG_ZSTD
+ case QCOW2_COMPRESSION_TYPE_ZSTD:
+ break;
+#endif
default:
error_setg(errp, "Unknown compression type");
goto out;
--
2.17.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH v2 4/4] iotests: 287: add qcow2 compression type test
2020-03-02 8:21 [PATCH v2 0/4] qcow2: Implement zstd cluster compression method Denis Plotnikov
` (2 preceding siblings ...)
2020-03-02 8:21 ` [PATCH v2 3/4] qcow2: add zstd cluster compression Denis Plotnikov
@ 2020-03-02 8:21 ` Denis Plotnikov
2020-03-02 8:51 ` [PATCH v2 0/4] qcow2: Implement zstd cluster compression method Vladimir Sementsov-Ogievskiy
4 siblings, 0 replies; 10+ messages in thread
From: Denis Plotnikov @ 2020-03-02 8:21 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, den, vsementsov, qemu-block, armbru, mreitz
The test checks fulfilling qcow2 requiriements for the compression
type feature and zstd compression type operability.
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
---
tests/qemu-iotests/287 | 127 +++++++++++++++++++++++++++++++++++++
tests/qemu-iotests/287.out | 43 +++++++++++++
tests/qemu-iotests/group | 1 +
3 files changed, 171 insertions(+)
create mode 100755 tests/qemu-iotests/287
create mode 100644 tests/qemu-iotests/287.out
diff --git a/tests/qemu-iotests/287 b/tests/qemu-iotests/287
new file mode 100755
index 0000000000..39cb665c85
--- /dev/null
+++ b/tests/qemu-iotests/287
@@ -0,0 +1,127 @@
+#!/usr/bin/env bash
+#
+# Test case for an image using zstd compression
+#
+# Copyright (c) 2020 Virtuozzo International GmbH
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
+owner=dplotnikov@virtuozzo.com
+
+seq="$(basename $0)"
+echo "QA output created by $seq"
+
+status=1 # failure is the default!
+
+_cleanup()
+{
+ _cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# standard environment
+. ./common.rc
+. ./common.filter
+
+# This tests qocw2-specific low-level functionality
+_supported_fmt qcow2
+_supported_proto file
+_supported_os Linux
+
+# Check if we can run this test.
+IMGOPTS='compression_type=zstd'
+
+_make_test_img 64M | grep "Invalid parameter 'zstd'" 2>&1 1>/dev/null
+
+ZSTD_SUPPORTED=$?
+
+if (($ZSTD_SUPPORTED==0)); then
+ _notrun "ZSTD is disabled"
+fi
+
+# Test: when compression is zlib the incompatible bit is unset
+echo
+echo "=== Testing compression type incompatible bit setting for zlib ==="
+echo
+
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+
+# Test: when compression differs from zlib the incompatible bit is set
+echo
+echo "=== Testing compression type incompatible bit setting for zstd ==="
+echo
+
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+
+# Test: an image can't be openned if compression type is zlib and
+# incompatible feature compression type is set
+echo
+echo "=== Testing zlib with incompatible bit set ==="
+echo
+
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" set-feature-bit incompatible 3
+# to make sure the bit was actually set
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null
+if (($?==0)); then
+ echo "Error: The image openned successfully. The image must not be openned"
+fi
+
+# Test: an image can't be openned if compression type is NOT zlib and
+# incompatible feature compression type is UNSET
+echo
+echo "=== Testing zstd with incompatible bit unset ==="
+echo
+
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" set-header incompatible_features 0
+# to make sure the bit was actually unset
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null
+if (($?==0)); then
+ echo "Error: The image openned successfully. The image must not be openned"
+fi
+# Test: check compression type values
+echo
+echo "=== Testing compression type values ==="
+echo
+# zlib=0
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+od -j104 -N1 -An -vtu1 "$TEST_IMG"
+
+# zstd=1
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+od -j104 -N1 -An -vtu1 "$TEST_IMG"
+
+# Test: using zstd compression, write to and read from an image
+echo
+echo "=== Testing reading and writing with zstd ==="
+echo
+
+CLUSTER_SIZE=65536
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$QEMU_IO -c "write -c -P 0xAC 65536 64k " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -P 0xAC 65536 65536 " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -v 131070 8 " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -v 65534 8" "$TEST_IMG" | _filter_qemu_io
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
diff --git a/tests/qemu-iotests/287.out b/tests/qemu-iotests/287.out
new file mode 100644
index 0000000000..8e51c3078d
--- /dev/null
+++ b/tests/qemu-iotests/287.out
@@ -0,0 +1,43 @@
+QA output created by 287
+
+=== Testing compression type incompatible bit setting for zlib ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+incompatible_features []
+
+=== Testing compression type incompatible bit setting for zstd ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+incompatible_features [3]
+
+=== Testing zlib with incompatible bit set ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+incompatible_features [3]
+
+=== Testing zstd with incompatible bit unset ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+incompatible_features []
+
+=== Testing compression type values ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+ 0
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+ 1
+
+=== Testing reading and writing with zstd ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+wrote 65536/65536 bytes at offset 65536
+64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 65536/65536 bytes at offset 65536
+64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+0001fffe: ac ac 00 00 00 00 00 00 ........
+read 8/8 bytes at offset 131070
+8 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+0000fffe: 00 00 ac ac ac ac ac ac ........
+read 8/8 bytes at offset 65534
+8 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+*** done
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index 0317667695..5edbadef40 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -293,3 +293,4 @@
283 auto quick
284 rw
286 rw quick
+287 auto quick
--
2.17.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH v2 0/4] qcow2: Implement zstd cluster compression method
2020-03-02 8:21 [PATCH v2 0/4] qcow2: Implement zstd cluster compression method Denis Plotnikov
` (3 preceding siblings ...)
2020-03-02 8:21 ` [PATCH v2 4/4] iotests: 287: add qcow2 compression type test Denis Plotnikov
@ 2020-03-02 8:51 ` Vladimir Sementsov-Ogievskiy
2020-03-02 8:54 ` Denis Plotnikov
4 siblings, 1 reply; 10+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-03-02 8:51 UTC (permalink / raw)
To: Denis Plotnikov, qemu-devel; +Cc: kwolf, den, qemu-block, armbru, mreitz
Doesn't apply to master, as zstd already exists in ./configure :) (for migration)
02.03.2020 11:21, Denis Plotnikov wrote:
> v2:
> * rework compression type setting [Vladimir]
> * squash iotest changes to the compression type introduction patch [Vladimir, Eric]
> * fix zstd availability checking in zstd iotest [Vladimir]
> * remove unnecessry casting [Eric]
> * remove rudundant checks [Eric]
> * fix compressed cluster layout in qcow2 spec [Vladimir]
> * fix wording [Eric, Vladimir]
> * fix compression type filtering in iotests [Eric]
>
> v1:
> the initial series
>
> ---
> zstd comression method is faster than the only available zlib.
> The series adds zstd to the methods available for clusters compression.
>
> The implementation is done with respect to the recently added compression
> type additional header to the qcow2 specification.
>
> Denis Plotnikov (4):
> qcow2: introduce compression type feature
> qcow2: rework the cluster compression routine
> qcow2: add zstd cluster compression
> iotests: 287: add qcow2 compression type test
>
> docs/interop/qcow2.txt | 20 ++++
> configure | 29 +++++
> qapi/block-core.json | 23 +++-
> block/qcow2.h | 18 ++-
> include/block/block_int.h | 1 +
> block/qcow2-threads.c | 197 ++++++++++++++++++++++++++++---
> block/qcow2.c | 116 ++++++++++++++++++
> tests/qemu-iotests/031.out | 14 +--
> tests/qemu-iotests/036.out | 4 +-
> tests/qemu-iotests/049.out | 102 ++++++++--------
> tests/qemu-iotests/060.out | 1 +
> tests/qemu-iotests/061.out | 34 +++---
> tests/qemu-iotests/065 | 20 ++--
> tests/qemu-iotests/080 | 2 +-
> tests/qemu-iotests/144.out | 4 +-
> tests/qemu-iotests/182.out | 2 +-
> tests/qemu-iotests/242.out | 5 +
> tests/qemu-iotests/255.out | 8 +-
> tests/qemu-iotests/287 | 127 ++++++++++++++++++++
> tests/qemu-iotests/287.out | 43 +++++++
> tests/qemu-iotests/common.filter | 3 +-
> tests/qemu-iotests/group | 1 +
> 22 files changed, 666 insertions(+), 108 deletions(-)
> create mode 100755 tests/qemu-iotests/287
> create mode 100644 tests/qemu-iotests/287.out
>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v2 0/4] qcow2: Implement zstd cluster compression method
2020-03-02 8:51 ` [PATCH v2 0/4] qcow2: Implement zstd cluster compression method Vladimir Sementsov-Ogievskiy
@ 2020-03-02 8:54 ` Denis Plotnikov
0 siblings, 0 replies; 10+ messages in thread
From: Denis Plotnikov @ 2020-03-02 8:54 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-devel
Cc: kwolf, den, qemu-block, armbru, mreitz
On 02.03.2020 11:51, Vladimir Sementsov-Ogievskiy wrote:
> Doesn't apply to master, as zstd already exists in ./configure :) (for
> migration)
Mmmmm, will rebase it. Any other comments?
>
>
> 02.03.2020 11:21, Denis Plotnikov wrote:
>> v2:
>> * rework compression type setting [Vladimir]
>> * squash iotest changes to the compression type introduction patch
>> [Vladimir, Eric]
>> * fix zstd availability checking in zstd iotest [Vladimir]
>> * remove unnecessry casting [Eric]
>> * remove rudundant checks [Eric]
>> * fix compressed cluster layout in qcow2 spec [Vladimir]
>> * fix wording [Eric, Vladimir]
>> * fix compression type filtering in iotests [Eric]
>>
>> v1:
>> the initial series
>>
>> ---
>> zstd comression method is faster than the only available zlib.
>> The series adds zstd to the methods available for clusters compression.
>>
>> The implementation is done with respect to the recently added
>> compression
>> type additional header to the qcow2 specification.
>>
>> Denis Plotnikov (4):
>> qcow2: introduce compression type feature
>> qcow2: rework the cluster compression routine
>> qcow2: add zstd cluster compression
>> iotests: 287: add qcow2 compression type test
>>
>> docs/interop/qcow2.txt | 20 ++++
>> configure | 29 +++++
>> qapi/block-core.json | 23 +++-
>> block/qcow2.h | 18 ++-
>> include/block/block_int.h | 1 +
>> block/qcow2-threads.c | 197 ++++++++++++++++++++++++++++---
>> block/qcow2.c | 116 ++++++++++++++++++
>> tests/qemu-iotests/031.out | 14 +--
>> tests/qemu-iotests/036.out | 4 +-
>> tests/qemu-iotests/049.out | 102 ++++++++--------
>> tests/qemu-iotests/060.out | 1 +
>> tests/qemu-iotests/061.out | 34 +++---
>> tests/qemu-iotests/065 | 20 ++--
>> tests/qemu-iotests/080 | 2 +-
>> tests/qemu-iotests/144.out | 4 +-
>> tests/qemu-iotests/182.out | 2 +-
>> tests/qemu-iotests/242.out | 5 +
>> tests/qemu-iotests/255.out | 8 +-
>> tests/qemu-iotests/287 | 127 ++++++++++++++++++++
>> tests/qemu-iotests/287.out | 43 +++++++
>> tests/qemu-iotests/common.filter | 3 +-
>> tests/qemu-iotests/group | 1 +
>> 22 files changed, 666 insertions(+), 108 deletions(-)
>> create mode 100755 tests/qemu-iotests/287
>> create mode 100644 tests/qemu-iotests/287.out
>>
>
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v2 1/4] qcow2: introduce compression type feature
2020-03-02 8:21 ` [PATCH v2 1/4] qcow2: introduce compression type feature Denis Plotnikov
@ 2020-03-02 11:24 ` Vladimir Sementsov-Ogievskiy
2020-03-02 11:29 ` Denis Plotnikov
0 siblings, 1 reply; 10+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-03-02 11:24 UTC (permalink / raw)
To: Denis Plotnikov, qemu-devel; +Cc: kwolf, den, qemu-block, armbru, mreitz
02.03.2020 11:21, Denis Plotnikov wrote:
> The patch adds some preparation parts for incompatible compression type
> feature to qcow2 allowing the use different compression methods for
> image clusters (de)compressing.
>
> It is implied that the compression type is set on the image creation and
> can be changed only later by image conversion, thus compression type
> defines the only compression algorithm used for the image, and thus,
> for all image clusters.
>
> The goal of the feature is to add support of other compression methods
> to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
>
> The default compression is ZLIB. Images created with ZLIB compression type
> are backward compatible with older qemu versions.
>
> Adding of the compression type breaks a number of tests because now the
> compression type is reported on image creation and there are some changes
> in the qcow2 header in size and offsets.
>
> The tests are fixed in the following ways:
> * filter out compression_type for all the tests
> * fix header size, feature table size and backing file offset
> affected tests: 031, 036, 061, 080
> header_size +=8: 1 byte compression type
> 7 bytes padding
> feature_table += 48: incompatible feture compression type
> backing_file_offset += 56 (8 + 48 -> header_change + fature_table_change)
> * add "compression type" for test output matching when it isn't filtered
> affected tests: 049, 060, 061, 065, 144, 182, 242, 255
>
> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
I'm almost OK with this patch. Some notes below and:
Seems, new option should be handled in qcow2_amend_options among other unsupported ones (otherwise qcow2_amend_options aborts).
> ---
> qapi/block-core.json | 22 ++++++-
> block/qcow2.h | 18 ++++-
> include/block/block_int.h | 1 +
> block/qcow2.c | 109 +++++++++++++++++++++++++++++++
> tests/qemu-iotests/031.out | 14 ++--
> tests/qemu-iotests/036.out | 4 +-
> tests/qemu-iotests/049.out | 102 ++++++++++++++---------------
> tests/qemu-iotests/060.out | 1 +
> tests/qemu-iotests/061.out | 34 ++++++----
> tests/qemu-iotests/065 | 20 +++---
> tests/qemu-iotests/080 | 2 +-
> tests/qemu-iotests/144.out | 4 +-
> tests/qemu-iotests/182.out | 2 +-
> tests/qemu-iotests/242.out | 5 ++
> tests/qemu-iotests/255.out | 8 +--
> tests/qemu-iotests/common.filter | 3 +-
> 16 files changed, 255 insertions(+), 94 deletions(-)
>
[..]
> --- a/block/qcow2.h
> +++ b/block/qcow2.h
> @@ -146,6 +146,12 @@ typedef struct QCowHeader {
>
> uint32_t refcount_order;
> uint32_t header_length;
> +
> + /* Additional fields */
> + uint8_t compression_type;
> +
> + /* header must be a multiple of 8 */
> + uint8_t padding[7];
> } QEMU_PACKED QCowHeader;
>
> typedef struct QEMU_PACKED QCowSnapshotHeader {
> @@ -216,13 +222,16 @@ enum {
> QCOW2_INCOMPAT_DIRTY_BITNR = 0,
> QCOW2_INCOMPAT_CORRUPT_BITNR = 1,
> QCOW2_INCOMPAT_DATA_FILE_BITNR = 2,
> + QCOW2_INCOMPAT_COMPRESSION_BITNR= 3,
checkpatch complains. I think, you can just use one space before '=' and don't
care about alignment.
> QCOW2_INCOMPAT_DIRTY = 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
> QCOW2_INCOMPAT_CORRUPT = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR,
> QCOW2_INCOMPAT_DATA_FILE = 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR,
> + QCOW2_INCOMPAT_COMPRESSION = 1 << QCOW2_INCOMPAT_COMPRESSION_BITNR,
>
> QCOW2_INCOMPAT_MASK = QCOW2_INCOMPAT_DIRTY
> | QCOW2_INCOMPAT_CORRUPT
> - | QCOW2_INCOMPAT_DATA_FILE,
> + | QCOW2_INCOMPAT_DATA_FILE
> + | QCOW2_INCOMPAT_COMPRESSION,
> };
>
> /* Compatible feature bits */
> @@ -369,6 +378,13 @@ typedef struct BDRVQcow2State {
>
> bool metadata_preallocation_checked;
> bool metadata_preallocation;
> + /*
> + * Compression type used for the image. Default: 0 - ZLIB
> + * The image compression type is set on image creation.
> + * The only way to change the compression type is to convert the image
> + * with the desired compression type set
> + */
> + Qcow2CompressionType compression_type;
> } BDRVQcow2State;
>
> typedef struct Qcow2COWRegion {
> diff --git a/include/block/block_int.h b/include/block/block_int.h
> index 6f9fd5e20e..2c6bb9dc99 100644
> --- a/include/block/block_int.h
> +++ b/include/block/block_int.h
> @@ -57,6 +57,7 @@
> #define BLOCK_OPT_REFCOUNT_BITS "refcount_bits"
> #define BLOCK_OPT_DATA_FILE "data_file"
> #define BLOCK_OPT_DATA_FILE_RAW "data_file_raw"
> +#define BLOCK_OPT_COMPRESSION_TYPE "compression_type"
>
> #define BLOCK_PROBE_BUF_SIZE 512
>
> diff --git a/block/qcow2.c b/block/qcow2.c
> index 3c754f616b..fc5232a5d6 100644
> --- a/block/qcow2.c
> +++ b/block/qcow2.c
> @@ -1242,6 +1242,50 @@ static int qcow2_update_options(BlockDriverState *bs, QDict *options,
> return ret;
> }
>
> +static int validate_compression_type(BDRVQcow2State *s, Error **errp)
> +{
> + /*
> + * Sanity check
> + * according to qcow2 spec, the compression type is 1-byte field
> + * but in BDRVQcow2State the compression_type is enum sizeof(int)
> + * so, the max compression_type value is 255.
> + */
> + if (s->compression_type > 0xff) {
This code is unreachable, I'd prefer assertion.
> + error_setg(errp, "qcow2: compression type value is too big");
> + return -EINVAL;
> + }
> +
> + switch (s->compression_type) {
> + case QCOW2_COMPRESSION_TYPE_ZLIB:
> + break;
> +
> + default:
> + error_setg(errp, "qcow2: unknown compression type: %u",
> + s->compression_type);
> + return -ENOTSUP;
> + }
> +
> + /*
> + * if the compression type differs from QCOW2_COMPRESSION_TYPE_ZLIB
> + * the incompatible feature flag must be set
> + */
> + if (s->compression_type == QCOW2_COMPRESSION_TYPE_ZLIB) {
> + if (s->incompatible_features & QCOW2_INCOMPAT_COMPRESSION) {
> + error_setg(errp, "qcow2: Compression type incompatible feature "
> + "bit must not be set");
> + return -EINVAL;
> + }
> + } else {
> + if (!(s->incompatible_features & QCOW2_INCOMPAT_COMPRESSION)) {
> + error_setg(errp, "qcow2: Compression type incompatible feature "
> + "bit must be set");
> + return -EINVAL;
> + }
> + }
> +
> + return 0;
> +}
> +
[..]
> @@ -3379,6 +3453,27 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
> }
> }
>
> + if (qcow2_opts->has_compression_type &&
> + qcow2_opts->compression_type != QCOW2_COMPRESSION_TYPE_ZLIB) {
> +
> + ret = -EINVAL;
> +
> + if (version < 3) {
> + error_setg(errp, "Compression type is only supported with "
Hmm. "Non-zlib compression type is only.." would be a bit more honest :)
> + "compatibility level 1.1 and above (use version=v3 or "
> + "greater)");
> + goto out;
> + }
> +
> + switch (qcow2_opts->compression_type) {
> + default:
> + error_setg(errp, "Unknown compression type");
> + goto out;
> + }
> +
> + compression_type = qcow2_opts->compression_type;
> + }
> +
[..]
> --- a/tests/qemu-iotests/065
> +++ b/tests/qemu-iotests/065
> @@ -88,23 +88,25 @@ class TestQMP(TestImageInfoSpecific):
> class TestQCow2(TestQemuImgInfo):
> '''Testing a qcow2 version 2 image'''
> img_options = 'compat=0.10'
> - json_compare = { 'compat': '0.10', 'refcount-bits': 16 }
> - human_compare = [ 'compat: 0.10', 'refcount bits: 16' ]
> + json_compare = { 'compat': '0.10', 'refcount-bits': 16, 'compression-type': 'zlib' }
> + human_compare = [ 'compat: 0.10', 'compression type: zlib', 'refcount bits: 16' ]
over-80 line (and several below).
>
> class TestQCow3NotLazy(TestQemuImgInfo):
> '''Testing a qcow2 version 3 image with lazy refcounts disabled'''
> img_options = 'compat=1.1,lazy_refcounts=off'
> json_compare = { 'compat': '1.1', 'lazy-refcounts': False,
> - 'refcount-bits': 16, 'corrupt': False }
> - human_compare = [ 'compat: 1.1', 'lazy refcounts: false',
> + 'refcount-bits': 16, 'corrupt': False,
> + 'compression-type': 'zlib' }
> + human_compare = [ 'compat: 1.1', 'compression type: zlib', 'lazy refcounts: false',
> 'refcount bits: 16', 'corrupt: false' ]
>
> class TestQCow3Lazy(TestQemuImgInfo):
> '''Testing a qcow2 version 3 image with lazy refcounts enabled'''
> img_options = 'compat=1.1,lazy_refcounts=on'
> json_compare = { 'compat': '1.1', 'lazy-refcounts': True,
> - 'refcount-bits': 16, 'corrupt': False }
> - human_compare = [ 'compat: 1.1', 'lazy refcounts: true',
> + 'refcount-bits': 16, 'corrupt': False,
> + 'compression-type': 'zlib' }
> + human_compare = [ 'compat: 1.1', 'compression type: zlib', 'lazy refcounts: true',
> 'refcount bits: 16', 'corrupt: false' ]
>
[..]
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v2 1/4] qcow2: introduce compression type feature
2020-03-02 11:24 ` Vladimir Sementsov-Ogievskiy
@ 2020-03-02 11:29 ` Denis Plotnikov
0 siblings, 0 replies; 10+ messages in thread
From: Denis Plotnikov @ 2020-03-02 11:29 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-devel
Cc: kwolf, den, qemu-block, armbru, mreitz
On 02.03.2020 14:24, Vladimir Sementsov-Ogievskiy wrote:
> 02.03.2020 11:21, Denis Plotnikov wrote:
>> The patch adds some preparation parts for incompatible compression type
>> feature to qcow2 allowing the use different compression methods for
>> image clusters (de)compressing.
>>
>> It is implied that the compression type is set on the image creation and
>> can be changed only later by image conversion, thus compression type
>> defines the only compression algorithm used for the image, and thus,
>> for all image clusters.
>>
>> The goal of the feature is to add support of other compression methods
>> to qcow2. For example, ZSTD which is more effective on compression
>> than ZLIB.
>>
>> The default compression is ZLIB. Images created with ZLIB compression
>> type
>> are backward compatible with older qemu versions.
>>
>> Adding of the compression type breaks a number of tests because now the
>> compression type is reported on image creation and there are some
>> changes
>> in the qcow2 header in size and offsets.
>>
>> The tests are fixed in the following ways:
>> * filter out compression_type for all the tests
>> * fix header size, feature table size and backing file offset
>> affected tests: 031, 036, 061, 080
>> header_size +=8: 1 byte compression type
>> 7 bytes padding
>> feature_table += 48: incompatible feture compression type
>> backing_file_offset += 56 (8 + 48 -> header_change +
>> fature_table_change)
>> * add "compression type" for test output matching when it isn't
>> filtered
>> affected tests: 049, 060, 061, 065, 144, 182, 242, 255
>>
>> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
>
> I'm almost OK with this patch. Some notes below and:
>
> Seems, new option should be handled in qcow2_amend_options among other
> unsupported ones (otherwise qcow2_amend_options aborts).
>
>> ---
>> qapi/block-core.json | 22 ++++++-
>> block/qcow2.h | 18 ++++-
>> include/block/block_int.h | 1 +
>> block/qcow2.c | 109 +++++++++++++++++++++++++++++++
>> tests/qemu-iotests/031.out | 14 ++--
>> tests/qemu-iotests/036.out | 4 +-
>> tests/qemu-iotests/049.out | 102 ++++++++++++++---------------
>> tests/qemu-iotests/060.out | 1 +
>> tests/qemu-iotests/061.out | 34 ++++++----
>> tests/qemu-iotests/065 | 20 +++---
>> tests/qemu-iotests/080 | 2 +-
>> tests/qemu-iotests/144.out | 4 +-
>> tests/qemu-iotests/182.out | 2 +-
>> tests/qemu-iotests/242.out | 5 ++
>> tests/qemu-iotests/255.out | 8 +--
>> tests/qemu-iotests/common.filter | 3 +-
>> 16 files changed, 255 insertions(+), 94 deletions(-)
>>
>
> [..]
>
>> --- a/block/qcow2.h
>> +++ b/block/qcow2.h
>> @@ -146,6 +146,12 @@ typedef struct QCowHeader {
>> uint32_t refcount_order;
>> uint32_t header_length;
>> +
>> + /* Additional fields */
>> + uint8_t compression_type;
>> +
>> + /* header must be a multiple of 8 */
>> + uint8_t padding[7];
>> } QEMU_PACKED QCowHeader;
>> typedef struct QEMU_PACKED QCowSnapshotHeader {
>> @@ -216,13 +222,16 @@ enum {
>> QCOW2_INCOMPAT_DIRTY_BITNR = 0,
>> QCOW2_INCOMPAT_CORRUPT_BITNR = 1,
>> QCOW2_INCOMPAT_DATA_FILE_BITNR = 2,
>> + QCOW2_INCOMPAT_COMPRESSION_BITNR= 3,
>
> checkpatch complains. I think, you can just use one space before '='
> and don't
> care about alignment.
ok
>
>> QCOW2_INCOMPAT_DIRTY = 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
>> QCOW2_INCOMPAT_CORRUPT = 1 <<
>> QCOW2_INCOMPAT_CORRUPT_BITNR,
>> QCOW2_INCOMPAT_DATA_FILE = 1 <<
>> QCOW2_INCOMPAT_DATA_FILE_BITNR,
>> + QCOW2_INCOMPAT_COMPRESSION = 1 <<
>> QCOW2_INCOMPAT_COMPRESSION_BITNR,
>> QCOW2_INCOMPAT_MASK = QCOW2_INCOMPAT_DIRTY
>> | QCOW2_INCOMPAT_CORRUPT
>> - | QCOW2_INCOMPAT_DATA_FILE,
>> + | QCOW2_INCOMPAT_DATA_FILE
>> + | QCOW2_INCOMPAT_COMPRESSION,
>> };
>> /* Compatible feature bits */
>> @@ -369,6 +378,13 @@ typedef struct BDRVQcow2State {
>> bool metadata_preallocation_checked;
>> bool metadata_preallocation;
>> + /*
>> + * Compression type used for the image. Default: 0 - ZLIB
>> + * The image compression type is set on image creation.
>> + * The only way to change the compression type is to convert the
>> image
>> + * with the desired compression type set
>> + */
>> + Qcow2CompressionType compression_type;
>> } BDRVQcow2State;
>> typedef struct Qcow2COWRegion {
>> diff --git a/include/block/block_int.h b/include/block/block_int.h
>> index 6f9fd5e20e..2c6bb9dc99 100644
>> --- a/include/block/block_int.h
>> +++ b/include/block/block_int.h
>> @@ -57,6 +57,7 @@
>> #define BLOCK_OPT_REFCOUNT_BITS "refcount_bits"
>> #define BLOCK_OPT_DATA_FILE "data_file"
>> #define BLOCK_OPT_DATA_FILE_RAW "data_file_raw"
>> +#define BLOCK_OPT_COMPRESSION_TYPE "compression_type"
>> #define BLOCK_PROBE_BUF_SIZE 512
>> diff --git a/block/qcow2.c b/block/qcow2.c
>> index 3c754f616b..fc5232a5d6 100644
>> --- a/block/qcow2.c
>> +++ b/block/qcow2.c
>> @@ -1242,6 +1242,50 @@ static int
>> qcow2_update_options(BlockDriverState *bs, QDict *options,
>> return ret;
>> }
>> +static int validate_compression_type(BDRVQcow2State *s, Error **errp)
>> +{
>> + /*
>> + * Sanity check
>> + * according to qcow2 spec, the compression type is 1-byte field
>> + * but in BDRVQcow2State the compression_type is enum sizeof(int)
>> + * so, the max compression_type value is 255.
>> + */
>> + if (s->compression_type > 0xff) {
>
> This code is unreachable, I'd prefer assertion.
Hm, it shouldn't be here. I'll remove it.
>
>
>> + error_setg(errp, "qcow2: compression type value is too big");
>> + return -EINVAL;
>> + }
>> +
>> + switch (s->compression_type) {
>> + case QCOW2_COMPRESSION_TYPE_ZLIB:
>> + break;
>> +
>> + default:
>> + error_setg(errp, "qcow2: unknown compression type: %u",
>> + s->compression_type);
>> + return -ENOTSUP;
>> + }
>> +
>> + /*
>> + * if the compression type differs from QCOW2_COMPRESSION_TYPE_ZLIB
>> + * the incompatible feature flag must be set
>> + */
>> + if (s->compression_type == QCOW2_COMPRESSION_TYPE_ZLIB) {
>> + if (s->incompatible_features & QCOW2_INCOMPAT_COMPRESSION) {
>> + error_setg(errp, "qcow2: Compression type incompatible
>> feature "
>> + "bit must not be set");
>> + return -EINVAL;
>> + }
>> + } else {
>> + if (!(s->incompatible_features & QCOW2_INCOMPAT_COMPRESSION)) {
>> + error_setg(errp, "qcow2: Compression type incompatible
>> feature "
>> + "bit must be set");
>> + return -EINVAL;
>> + }
>> + }
>> +
>> + return 0;
>> +}
>> +
>
>
> [..]
>
>> @@ -3379,6 +3453,27 @@ qcow2_co_create(BlockdevCreateOptions
>> *create_options, Error **errp)
>> }
>> }
>> + if (qcow2_opts->has_compression_type &&
>> + qcow2_opts->compression_type != QCOW2_COMPRESSION_TYPE_ZLIB) {
>> +
>> + ret = -EINVAL;
>> +
>> + if (version < 3) {
>> + error_setg(errp, "Compression type is only supported with "
>
> Hmm. "Non-zlib compression type is only.." would be a bit more honest :)
valid. to be changed.
>
>> + "compatibility level 1.1 and above (use
>> version=v3 or "
>> + "greater)");
>> + goto out;
>> + }
>> +
>> + switch (qcow2_opts->compression_type) {
>> + default:
>> + error_setg(errp, "Unknown compression type");
>> + goto out;
>> + }
>> +
>> + compression_type = qcow2_opts->compression_type;
>> + }
>> +
>
>
> [..]
>
>> --- a/tests/qemu-iotests/065
>> +++ b/tests/qemu-iotests/065
>> @@ -88,23 +88,25 @@ class TestQMP(TestImageInfoSpecific):
>> class TestQCow2(TestQemuImgInfo):
>> '''Testing a qcow2 version 2 image'''
>> img_options = 'compat=0.10'
>> - json_compare = { 'compat': '0.10', 'refcount-bits': 16 }
>> - human_compare = [ 'compat: 0.10', 'refcount bits: 16' ]
>> + json_compare = { 'compat': '0.10', 'refcount-bits': 16,
>> 'compression-type': 'zlib' }
>> + human_compare = [ 'compat: 0.10', 'compression type: zlib',
>> 'refcount bits: 16' ]
>
> over-80 line (and several below).
>
>> class TestQCow3NotLazy(TestQemuImgInfo):
>> '''Testing a qcow2 version 3 image with lazy refcounts disabled'''
>> img_options = 'compat=1.1,lazy_refcounts=off'
>> json_compare = { 'compat': '1.1', 'lazy-refcounts': False,
>> - 'refcount-bits': 16, 'corrupt': False }
>> - human_compare = [ 'compat: 1.1', 'lazy refcounts: false',
>> + 'refcount-bits': 16, 'corrupt': False,
>> + 'compression-type': 'zlib' }
>> + human_compare = [ 'compat: 1.1', 'compression type: zlib', 'lazy
>> refcounts: false',
>> 'refcount bits: 16', 'corrupt: false' ]
>> class TestQCow3Lazy(TestQemuImgInfo):
>> '''Testing a qcow2 version 3 image with lazy refcounts enabled'''
>> img_options = 'compat=1.1,lazy_refcounts=on'
>> json_compare = { 'compat': '1.1', 'lazy-refcounts': True,
>> - 'refcount-bits': 16, 'corrupt': False }
>> - human_compare = [ 'compat: 1.1', 'lazy refcounts: true',
>> + 'refcount-bits': 16, 'corrupt': False,
>> + 'compression-type': 'zlib' }
>> + human_compare = [ 'compat: 1.1', 'compression type: zlib', 'lazy
>> refcounts: true',
>> 'refcount bits: 16', 'corrupt: false' ]
>
> [..]
>
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v2 2/4] qcow2: rework the cluster compression routine
2020-03-02 8:21 ` [PATCH v2 2/4] qcow2: rework the cluster compression routine Denis Plotnikov
@ 2020-03-02 13:02 ` Vladimir Sementsov-Ogievskiy
0 siblings, 0 replies; 10+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-03-02 13:02 UTC (permalink / raw)
To: Denis Plotnikov, qemu-devel; +Cc: kwolf, den, qemu-block, armbru, mreitz
02.03.2020 11:21, Denis Plotnikov wrote:
> The patch enables processing the image compression type defined
> for the image and chooses an appropriate method for image clusters
> (de)compression.
>
> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
> ---
> block/qcow2-threads.c | 77 +++++++++++++++++++++++++++++++++++--------
> 1 file changed, 63 insertions(+), 14 deletions(-)
>
> diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
> index 77bb578cdf..9288a4f852 100644
> --- a/block/qcow2-threads.c
> +++ b/block/qcow2-threads.c
> @@ -74,7 +74,9 @@ typedef struct Qcow2CompressData {
> } Qcow2CompressData;
>
> /*
> - * qcow2_compress()
> + * qcow2_zlib_compress()
> + *
> + * Compress @src_size bytes of data using zlib compression method
> *
> * @dest - destination buffer, @dest_size bytes
> * @src - source buffer, @src_size bytes
> @@ -83,8 +85,8 @@ typedef struct Qcow2CompressData {
> * -ENOMEM destination buffer is not enough to store compressed data
> * -EIO on any other error
> */
> -static ssize_t qcow2_compress(void *dest, size_t dest_size,
> - const void *src, size_t src_size)
> +static ssize_t qcow2_zlib_compress(void *dest, size_t dest_size,
> + const void *src, size_t src_size)
> {
> ssize_t ret;
> z_stream strm;
> @@ -119,19 +121,19 @@ static ssize_t qcow2_compress(void *dest, size_t dest_size,
> }
>
> /*
> - * qcow2_decompress()
> + * qcow2_zlib_decompress()
> *
> * Decompress some data (not more than @src_size bytes) to produce exactly
> - * @dest_size bytes.
> + * @dest_size bytes using zlib compression method
> *
> * @dest - destination buffer, @dest_size bytes
> * @src - source buffer, @src_size bytes
> *
> * Returns: 0 on success
> - * -1 on fail
> + * -EIO on failure
> */
> -static ssize_t qcow2_decompress(void *dest, size_t dest_size,
> - const void *src, size_t src_size)
> +static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,
> + const void *src, size_t src_size)
> {
> int ret = 0;
> z_stream strm;
> @@ -144,7 +146,7 @@ static ssize_t qcow2_decompress(void *dest, size_t dest_size,
>
> ret = inflateInit2(&strm, -12);
> if (ret != Z_OK) {
> - return -1;
> + return -EIO;
> }
>
> ret = inflate(&strm, Z_FINISH);
more context:
if ((ret != Z_STREAM_END && ret != Z_BUF_ERROR) || strm.avail_out != 0) {
/*
* We approve Z_BUF_ERROR because we need @dest buffer to be filled, but
> * @src buffer may be processed partly (because in qcow2 we know size of
> * compressed data with precision of one sector)
> */
> - ret = -1;
> + ret = -EIO;
> }
Good thing that you've changed -1 to -EIO, to be more compatible with _compress() function.
But seems, that there is an actual (preexisting) bug here, as we return direct ret of inflate, which is
incompatible with our function defined API. And this mean, that Z_STREAM_END most probably work,
as it is positive. But Z_BUF_ERROR will be treated as error actually.
Hmm, it was broken in far 341926ab83e2b4d in 2018.. By me of course :(
I think, I'll send a separate patch with stable@ in cc.
>
> inflateEnd(&strm);
> @@ -189,20 +191,67 @@ qcow2_co_do_compress(BlockDriverState *bs, void *dest, size_t dest_size,
> return arg.ret;
> }
>
> +/*
> + * qcow2_co_compress()
> + *
> + * Compress @src_size bytes of data using the compression
> + * method defined by the image compression type
> + *
> + * @dest - destination buffer, @dest_size bytes
> + * @src - source buffer, @src_size bytes
> + *
> + * Returns: compressed size on success
> + * a negative error code on failure
> + */
> ssize_t coroutine_fn
> qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size,
> const void *src, size_t src_size)
> {
> - return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
> - qcow2_compress);
> + BDRVQcow2State *s = bs->opaque;
> + Qcow2CompressFunc fn;
> +
> + switch (s->compression_type) {
> + case QCOW2_COMPRESSION_TYPE_ZLIB:
> + fn = qcow2_zlib_compress;
> + break;
> +
> + default:
> + abort();
> + }
> +
> + return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
> }
>
> +/*
> + * qcow2_co_decompress()
> + *
> + * Decompress some data (not more than @src_size bytes) to produce exactly
> + * @dest_size bytes using the compression method defined by the image
> + * compression type
> + *
> + * @dest - destination buffer, @dest_size bytes
> + * @src - source buffer, @src_size bytes
> + *
> + * Returns: 0 on success
> + * a negative error code on failure
> + */
> ssize_t coroutine_fn
> qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size,
> const void *src, size_t src_size)
> {
> - return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
> - qcow2_decompress);
> + BDRVQcow2State *s = bs->opaque;
> + Qcow2CompressFunc fn;
> +
> + switch (s->compression_type) {
> + case QCOW2_COMPRESSION_TYPE_ZLIB:
> + fn = qcow2_zlib_decompress;
> + break;
> +
> + default:
> + return -ENOTSUP;
> + }
> +
> + return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
> }
>
>
>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2020-03-02 13:03 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-03-02 8:21 [PATCH v2 0/4] qcow2: Implement zstd cluster compression method Denis Plotnikov
2020-03-02 8:21 ` [PATCH v2 1/4] qcow2: introduce compression type feature Denis Plotnikov
2020-03-02 11:24 ` Vladimir Sementsov-Ogievskiy
2020-03-02 11:29 ` Denis Plotnikov
2020-03-02 8:21 ` [PATCH v2 2/4] qcow2: rework the cluster compression routine Denis Plotnikov
2020-03-02 13:02 ` Vladimir Sementsov-Ogievskiy
2020-03-02 8:21 ` [PATCH v2 3/4] qcow2: add zstd cluster compression Denis Plotnikov
2020-03-02 8:21 ` [PATCH v2 4/4] iotests: 287: add qcow2 compression type test Denis Plotnikov
2020-03-02 8:51 ` [PATCH v2 0/4] qcow2: Implement zstd cluster compression method Vladimir Sementsov-Ogievskiy
2020-03-02 8:54 ` Denis Plotnikov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).