[Qemu-devel] [PATCH v2 0/2] qcow2: include LUKS payload overhead in qemu-img measure

[Qemu-devel] [PATCH v2 0/2] qcow2: include LUKS payload overhead in qemu-img measure

[Stefan Hajnoczi]

v2:
 * Take into account all "encrypt." image creation options since the LUKS
   payload size varies depending on the cipher [Max]

The LUKS payload has a significant size (>1 MB).  It must be included in the
qemu-img measure calculation so we arrive at the correct minimum image size.

Stefan Hajnoczi (2):
  qcow2: include LUKS payload overhead in qemu-img measure
  iotests: add LUKS payload overhead to 178 qemu-img measure test

 block/qcow2.c                    | 72 +++++++++++++++++++++++++++++++-
 tests/qemu-iotests/178           |  8 ++++
 tests/qemu-iotests/178.out.qcow2 | 24 +++++++++++
 3 files changed, 103 insertions(+), 1 deletion(-)

-- 
2.20.1

[Qemu-devel] [PATCH v2 1/2] qcow2: include LUKS payload overhead in qemu-img measure

[Stefan Hajnoczi]

LUKS encryption reserves clusters for its own payload data.  The size of
this area must be included in the qemu-img measure calculation so that
we arrive at the correct minimum required image size.

(Ab)use the qcrypto_block_create() API to determine the payload
overhead.  We discard the payload data that qcrypto thinks will be
written to the image.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 block/qcow2.c | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 71 insertions(+), 1 deletion(-)

diff --git a/block/qcow2.c b/block/qcow2.c
index 4897abae5e..e6f7685706 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -4231,6 +4231,60 @@ static coroutine_fn int qcow2_co_flush_to_os(BlockDriverState *bs)
     return ret;
 }
 
+static ssize_t qcow2_measure_crypto_hdr_init_func(QCryptoBlock *block,
+        size_t headerlen, void *opaque, Error **errp)
+{
+    size_t *headerlenp = opaque;
+
+    /* Stash away the payload size */
+    *headerlenp = headerlen;
+    return 0;
+}
+
+static ssize_t qcow2_measure_crypto_hdr_write_func(QCryptoBlock *block,
+        size_t offset, const uint8_t *buf, size_t buflen,
+        void *opaque, Error **errp)
+{
+    /* Discard the bytes, we're not actually writing to an image */
+    return buflen;
+}
+
+/* Determine the number of bytes for the LUKS payload */
+static bool qcow2_measure_luks_headerlen(QemuOpts *opts, size_t *len,
+                                         Error **errp)
+{
+    QDict *opts_qdict;
+    QDict *cryptoopts_qdict;
+    QCryptoBlockCreateOptions *cryptoopts;
+    QCryptoBlock *crypto;
+
+    /* Extract "encrypt." options into a qdict */
+    opts_qdict = qemu_opts_to_qdict(opts, NULL);
+    qdict_extract_subqdict(opts_qdict, &cryptoopts_qdict, "encrypt.");
+    qobject_unref(opts_qdict);
+
+    /* Build QCryptoBlockCreateOptions object from qdict */
+    qdict_put_str(cryptoopts_qdict, "format", "luks");
+    cryptoopts = block_crypto_create_opts_init(cryptoopts_qdict, errp);
+    qobject_unref(cryptoopts_qdict);
+    if (!cryptoopts) {
+        return false;
+    }
+
+    /* Fake LUKS creation in order to determine the payload size */
+    crypto = qcrypto_block_create(cryptoopts, "encrypt.",
+                                  qcow2_measure_crypto_hdr_init_func,
+                                  qcow2_measure_crypto_hdr_write_func,
+                                  len, errp);
+    qapi_free_QCryptoBlockCreateOptions(cryptoopts);
+    if (!crypto) {
+        return false;
+    }
+
+    qcrypto_block_free(crypto);
+    return true;
+}
+
 static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs,
                                        Error **errp)
 {
@@ -4240,11 +4294,13 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs,
     uint64_t virtual_size; /* disk size as seen by guest */
     uint64_t refcount_bits;
     uint64_t l2_tables;
+    uint64_t luks_payload_size = 0;
     size_t cluster_size;
     int version;
     char *optstr;
     PreallocMode prealloc;
     bool has_backing_file;
+    bool has_luks;
 
     /* Parse image creation options */
     cluster_size = qcow2_opt_get_cluster_size_del(opts, &local_err);
@@ -4274,6 +4330,20 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs,
     has_backing_file = !!optstr;
     g_free(optstr);
 
+    optstr = qemu_opt_get_del(opts, BLOCK_OPT_ENCRYPT_FORMAT);
+    has_luks = optstr && strcmp(optstr, "luks") == 0;
+    g_free(optstr);
+
+    if (has_luks) {
+        size_t headerlen;
+
+        if (!qcow2_measure_luks_headerlen(opts, &headerlen, &local_err)) {
+            goto err;
+        }
+
+        luks_payload_size = ROUND_UP(headerlen, cluster_size);
+    }
+
     virtual_size = qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0);
     virtual_size = ROUND_UP(virtual_size, cluster_size);
 
@@ -4344,7 +4414,7 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs,
     info = g_new(BlockMeasureInfo, 1);
     info->fully_allocated =
         qcow2_calc_prealloc_size(virtual_size, cluster_size,
-                                 ctz32(refcount_bits));
+                                 ctz32(refcount_bits)) + luks_payload_size;
 
     /* Remove data clusters that are not required.  This overestimates the
      * required size because metadata needed for the fully allocated file is
-- 
2.20.1

[Qemu-devel] [PATCH v2 2/2] iotests: add LUKS payload overhead to 178 qemu-img measure test

[Stefan Hajnoczi]

The previous patch includes the LUKS payload overhead into the qemu-img
measure calculation for qcow2.  Update qemu-iotests 178 to exercise this
new code path.

Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 tests/qemu-iotests/178           |  8 ++++++++
 tests/qemu-iotests/178.out.qcow2 | 24 ++++++++++++++++++++++++
 2 files changed, 32 insertions(+)

diff --git a/tests/qemu-iotests/178 b/tests/qemu-iotests/178
index 3f4b4a4564..23eb017ea1 100755
--- a/tests/qemu-iotests/178
+++ b/tests/qemu-iotests/178
@@ -142,6 +142,14 @@ for ofmt in human json; do
             # The backing file doesn't need to exist :)
             $QEMU_IMG measure --output=$ofmt -o backing_file=x \
                               -f "$fmt" -O "$IMGFMT" "$TEST_IMG"
+
+            echo
+            echo "== $fmt input image and LUKS encryption =="
+            echo
+            $QEMU_IMG measure --output=$ofmt \
+                              --object secret,id=sec0,data=base \
+                              -o encrypt.format=luks,encrypt.key-secret=sec0 \
+                              -f "$fmt" -O "$IMGFMT" "$TEST_IMG"
         fi
 
         echo
diff --git a/tests/qemu-iotests/178.out.qcow2 b/tests/qemu-iotests/178.out.qcow2
index d42d4a4597..55a8dc926f 100644
--- a/tests/qemu-iotests/178.out.qcow2
+++ b/tests/qemu-iotests/178.out.qcow2
@@ -68,6 +68,11 @@ converted image file size in bytes: 458752
 required size: 1074135040
 fully allocated size: 1074135040
 
+== qcow2 input image and LUKS encryption ==
+
+required size: 2686976
+fully allocated size: 1076232192
+
 == qcow2 input image and preallocation (human) ==
 
 required size: 1074135040
@@ -114,6 +119,11 @@ converted image file size in bytes: 524288
 required size: 1074135040
 fully allocated size: 1074135040
 
+== raw input image and LUKS encryption ==
+
+required size: 2686976
+fully allocated size: 1076232192
+
 == raw input image and preallocation (human) ==
 
 required size: 1074135040
@@ -205,6 +215,13 @@ converted image file size in bytes: 458752
     "fully-allocated": 1074135040
 }
 
+== qcow2 input image and LUKS encryption ==
+
+{
+    "required": 2686976,
+    "fully-allocated": 1076232192
+}
+
 == qcow2 input image and preallocation (json) ==
 
 {
@@ -263,6 +280,13 @@ converted image file size in bytes: 524288
     "fully-allocated": 1074135040
 }
 
+== raw input image and LUKS encryption ==
+
+{
+    "required": 2686976,
+    "fully-allocated": 1076232192
+}
+
 == raw input image and preallocation (json) ==
 
 {
-- 
2.20.1

Re: [Qemu-devel] [PATCH v2 1/2] qcow2: include LUKS payload overhead in qemu-img measure

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 658 bytes --]

On 23.01.19 11:33, Stefan Hajnoczi wrote:
> LUKS encryption reserves clusters for its own payload data.  The size of
> this area must be included in the qemu-img measure calculation so that
> we arrive at the correct minimum required image size.
> 
> (Ab)use the qcrypto_block_create() API to determine the payload
> overhead.  We discard the payload data that qcrypto thinks will be
> written to the image.
> 
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
>  block/qcow2.c | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 71 insertions(+), 1 deletion(-)

Reviewed-by: Max Reitz <mreitz@redhat.com>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v2 2/2] iotests: add LUKS payload overhead to 178 qemu-img measure test

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 1608 bytes --]

On 23.01.19 11:33, Stefan Hajnoczi wrote:
> The previous patch includes the LUKS payload overhead into the qemu-img
> measure calculation for qcow2.  Update qemu-iotests 178 to exercise this
> new code path.
> 
> Reviewed-by: Max Reitz <mreitz@redhat.com>
> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
>  tests/qemu-iotests/178           |  8 ++++++++
>  tests/qemu-iotests/178.out.qcow2 | 24 ++++++++++++++++++++++++
>  2 files changed, 32 insertions(+)
> 
> diff --git a/tests/qemu-iotests/178 b/tests/qemu-iotests/178
> index 3f4b4a4564..23eb017ea1 100755
> --- a/tests/qemu-iotests/178
> +++ b/tests/qemu-iotests/178
> @@ -142,6 +142,14 @@ for ofmt in human json; do
>              # The backing file doesn't need to exist :)
>              $QEMU_IMG measure --output=$ofmt -o backing_file=x \
>                                -f "$fmt" -O "$IMGFMT" "$TEST_IMG"
> +
> +            echo
> +            echo "== $fmt input image and LUKS encryption =="
> +            echo
> +            $QEMU_IMG measure --output=$ofmt \
> +                              --object secret,id=sec0,data=base \
> +                              -o encrypt.format=luks,encrypt.key-secret=sec0 \
> +                              -f "$fmt" -O "$IMGFMT" "$TEST_IMG"

With the change in patch 1, it would be possible to set
encrypt.iter-time=10, for example (default for the LUKS tests).  For me,
that decreases the test's runtime from 24 to 15 seconds.

Pretty please? :-)

Max

>          fi
>  
>          echo


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v2 2/2] iotests: add LUKS payload overhead to 178 qemu-img measure test

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1193 bytes --]

On Fri, Feb 15, 2019 at 08:14:59PM +0100, Max Reitz wrote:
> On 23.01.19 11:33, Stefan Hajnoczi wrote:
> > diff --git a/tests/qemu-iotests/178 b/tests/qemu-iotests/178
> > index 3f4b4a4564..23eb017ea1 100755
> > --- a/tests/qemu-iotests/178
> > +++ b/tests/qemu-iotests/178
> > @@ -142,6 +142,14 @@ for ofmt in human json; do
> >              # The backing file doesn't need to exist :)
> >              $QEMU_IMG measure --output=$ofmt -o backing_file=x \
> >                                -f "$fmt" -O "$IMGFMT" "$TEST_IMG"
> > +
> > +            echo
> > +            echo "== $fmt input image and LUKS encryption =="
> > +            echo
> > +            $QEMU_IMG measure --output=$ofmt \
> > +                              --object secret,id=sec0,data=base \
> > +                              -o encrypt.format=luks,encrypt.key-secret=sec0 \
> > +                              -f "$fmt" -O "$IMGFMT" "$TEST_IMG"
> 
> With the change in patch 1, it would be possible to set
> encrypt.iter-time=10, for example (default for the LUKS tests).  For me,
> that decreases the test's runtime from 24 to 15 seconds.
> 
> Pretty please? :-)

Will fix in v3.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]