[PATCH 0/4] Make qemu-img dd more flexible

[PATCH 0/4] Make qemu-img dd more flexible

[Fabian Ebner]

Adds support for reading from stdin and writing to stdout (when raw
format is used), as well as overriding the size of the output and
input image/stream.

Additionally, the options -n for skipping output image creation and -l
for loading a snapshot are made available like for convert.

Alexandre Derumier (1):
  qemu-img: dd: add -n option (skip target volume creation)

Fabian Ebner (1):
  qemu-img: dd: add -l option for loading a snapshot

Wolfgang Bumiller (2):
  qemu-img: dd: add osize and read from/to stdin/stdout
  qemu-img: dd: add isize parameter

 docs/tools/qemu-img.rst |  28 ++++-
 qemu-img-cmds.hx        |   4 +-
 qemu-img.c              | 261 +++++++++++++++++++++++++++++-----------
 3 files changed, 215 insertions(+), 78 deletions(-)

-- 
2.30.2

[PATCH 1/4] qemu-img: dd: add osize and read from/to stdin/stdout

[Fabian Ebner]

From: Wolfgang Bumiller <w.bumiller@proxmox.com>

Neither convert nor dd were previously able to write to or
read from a pipe. Particularly serializing an image file
into a raw stream or vice versa can be useful, but using
`qemu-img convert -f qcow2 -O raw foo.qcow2 /dev/stdout` in
a pipe will fail trying to seek.

While dd and convert have overlapping use cases, `dd` is a
simple read/write loop while convert is much more
sophisticated and has ways to dealing with holes and blocks
of zeroes.
Since these typically can't be detected in pipes via
SEEK_DATA/HOLE or skipped while writing, dd seems to be the
better choice for implementing stdin/stdout streams.

This patch causes "if" and "of" to default to stdin and
stdout respectively, allowing only the "raw" format to be
used in these cases.
Since the input can now be a pipe we have no way of
detecting the size of the output image to create. Since we
also want to support images with a size not matching the
dd command's "bs" parameter (which, together with "count"
could be used to calculate the desired size, and is already
used to limit it), the "osize" option is added to explicitly
override the output file's size.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
[FE: add documentation
     avoid error when osize is larger than input image's size
     fail if both count and osize are specified
     fail if skip is specified when reading from stdin]
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---
 docs/tools/qemu-img.rst |  17 +++-
 qemu-img-cmds.hx        |   4 +-
 qemu-img.c              | 201 ++++++++++++++++++++++++++--------------
 3 files changed, 146 insertions(+), 76 deletions(-)

diff --git a/docs/tools/qemu-img.rst b/docs/tools/qemu-img.rst
index 8885ea11cf..775eaf3097 100644
--- a/docs/tools/qemu-img.rst
+++ b/docs/tools/qemu-img.rst
@@ -220,16 +220,20 @@ Parameters to dd subcommand:
 
 .. option:: if=INPUT
 
-  Sets the input file
+  Sets the input file (defaults to STDIN)
 
 .. option:: of=OUTPUT
 
-  Sets the output file
+  Sets the output file (defaults to STDOUT)
 
 .. option:: skip=BLOCKS
 
   Sets the number of input blocks to skip
 
+.. option:: osize=OUTPUT_SIZE
+
+  Sets the output image's size
+
 Parameters to snapshot subcommand:
 
 .. program:: qemu-img-snapshot
@@ -488,10 +492,10 @@ Command description:
   it doesn't need to be specified separately in this case.
 
 
-.. option:: dd [--image-opts] [-U] [-f FMT] [-O OUTPUT_FMT] [bs=BLOCK_SIZE] [count=BLOCKS] [skip=BLOCKS] if=INPUT of=OUTPUT
+.. option:: dd [--image-opts] [-U] [-f FMT] [-O OUTPUT_FMT] [bs=BLOCK_SIZE] [count=BLOCKS] [skip=BLOCKS] [osize=OUTPUT_SIZE] [if=INPUT] [of=OUTPUT]
 
-  dd copies from *INPUT* file to *OUTPUT* file converting it from
-  *FMT* format to *OUTPUT_FMT* format.
+  dd copies from *INPUT* file (default: STDIN) to *OUTPUT* file (default:
+  STDOUT) converting it from *FMT* format to *OUTPUT_FMT* format.
 
   The data is by default read and written using blocks of 512 bytes but can be
   modified by specifying *BLOCK_SIZE*. If count=\ *BLOCKS* is specified
@@ -499,6 +503,9 @@ Command description:
 
   The size syntax is similar to :manpage:`dd(1)`'s size syntax.
 
+  The output image will be created with size *OUTPUT_SIZE* and at most this many
+  bytes will be copied.
+
 .. option:: info [--object OBJECTDEF] [--image-opts] [-f FMT] [--output=OFMT] [--backing-chain] [-U] FILENAME
 
   Give information about the disk image *FILENAME*. Use it in
diff --git a/qemu-img-cmds.hx b/qemu-img-cmds.hx
index 1b1dab5b17..e4935365c9 100644
--- a/qemu-img-cmds.hx
+++ b/qemu-img-cmds.hx
@@ -58,9 +58,9 @@ SRST
 ERST
 
 DEF("dd", img_dd,
-    "dd [--image-opts] [-U] [-f fmt] [-O output_fmt] [bs=block_size] [count=blocks] [skip=blocks] if=input of=output")
+    "dd [--image-opts] [-U] [-f fmt] [-O output_fmt] [bs=block_size] [count=blocks] [skip=blocks] [osize=output_size] [if=input] [of=output]")
 SRST
-.. option:: dd [--image-opts] [-U] [-f FMT] [-O OUTPUT_FMT] [bs=BLOCK_SIZE] [count=BLOCKS] [skip=BLOCKS] if=INPUT of=OUTPUT
+.. option:: dd [--image-opts] [-U] [-f FMT] [-O OUTPUT_FMT] [bs=BLOCK_SIZE] [count=BLOCKS] [skip=BLOCKS] [osize=OUTPUT_SIZE] [if=INPUT] [of=OUTPUT]
 ERST
 
 DEF("info", img_info,
diff --git a/qemu-img.c b/qemu-img.c
index 6fe2466032..ea488fd190 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -4819,10 +4819,12 @@ static int img_bitmap(int argc, char **argv)
 #define C_IF      04
 #define C_OF      010
 #define C_SKIP    020
+#define C_OSIZE   040
 
 struct DdInfo {
     unsigned int flags;
     int64_t count;
+    int64_t osize;
 };
 
 struct DdIo {
@@ -4898,6 +4900,19 @@ static int img_dd_skip(const char *arg,
     return 0;
 }
 
+static int img_dd_osize(const char *arg,
+                        struct DdIo *in, struct DdIo *out,
+                        struct DdInfo *dd)
+{
+    dd->osize = cvtnum("osize", arg);
+
+    if (dd->osize < 0) {
+        return 1;
+    }
+
+    return 0;
+}
+
 static int img_dd(int argc, char **argv)
 {
     int ret = 0;
@@ -4912,12 +4927,13 @@ static int img_dd(int argc, char **argv)
     int c, i;
     const char *out_fmt = "raw";
     const char *fmt = NULL;
-    int64_t size = 0;
+    int64_t size = 0, readsize = 0;
     int64_t block_count = 0, out_pos, in_pos;
     bool force_share = false;
     struct DdInfo dd = {
         .flags = 0,
         .count = 0,
+        .osize = 0,
     };
     struct DdIo in = {
         .bsz = 512, /* Block size is by default 512 bytes */
@@ -4938,6 +4954,7 @@ static int img_dd(int argc, char **argv)
         { "if", img_dd_if, C_IF },
         { "of", img_dd_of, C_OF },
         { "skip", img_dd_skip, C_SKIP },
+        { "osize", img_dd_osize, C_OSIZE },
         { NULL, NULL, 0 }
     };
     const struct option long_options[] = {
@@ -5013,91 +5030,122 @@ static int img_dd(int argc, char **argv)
         arg = NULL;
     }
 
-    if (!(dd.flags & C_IF && dd.flags & C_OF)) {
-        error_report("Must specify both input and output files");
+    if (!(dd.flags & C_IF) && (!fmt || strcmp(fmt, "raw") != 0)) {
+        error_report("Input format must be raw when reading from stdin");
         ret = -1;
         goto out;
     }
-
-    blk1 = img_open(image_opts, in.filename, fmt, 0, false, false,
-                    force_share);
-
-    if (!blk1) {
-        ret = -1;
-        goto out;
-    }
-
-    drv = bdrv_find_format(out_fmt);
-    if (!drv) {
-        error_report("Unknown file format");
-        ret = -1;
-        goto out;
-    }
-    proto_drv = bdrv_find_protocol(out.filename, true, &local_err);
-
-    if (!proto_drv) {
-        error_report_err(local_err);
+    if (!(dd.flags & C_OF) && strcmp(out_fmt, "raw") != 0) {
+        error_report("Output format must be raw when writing to stdout");
         ret = -1;
         goto out;
     }
-    if (!drv->create_opts) {
-        error_report("Format driver '%s' does not support image creation",
-                     drv->format_name);
+    if (dd.flags & C_OSIZE && dd.flags & C_COUNT) {
+        error_report("Cannot specify both count and osize");
         ret = -1;
         goto out;
     }
-    if (!proto_drv->create_opts) {
-        error_report("Protocol driver '%s' does not support image creation",
-                     proto_drv->format_name);
+    if (!(dd.flags & C_IF) && dd.flags & C_SKIP) {
+        error_report("Cannot specify skip when reading from stdin");
         ret = -1;
         goto out;
     }
-    create_opts = qemu_opts_append(create_opts, drv->create_opts);
-    create_opts = qemu_opts_append(create_opts, proto_drv->create_opts);
 
-    opts = qemu_opts_create(create_opts, NULL, 0, &error_abort);
+    if (dd.flags & C_IF) {
+        blk1 = img_open(image_opts, in.filename, fmt, 0, false, false,
+                        force_share);
 
-    size = blk_getlength(blk1);
-    if (size < 0) {
-        error_report("Failed to get size for '%s'", in.filename);
+        if (!blk1) {
+            ret = -1;
+            goto out;
+        }
+    }
+
+    if (dd.flags & C_IF) {
+        size = blk_getlength(blk1);
+        if (size < 0) {
+            error_report("Failed to get size for '%s'", in.filename);
+            ret = -1;
+            goto out;
+        }
+    } else if (dd.flags & C_OSIZE) {
+        size = dd.osize;
+    } else if (dd.flags & C_COUNT) {
+        size = dd.count * in.bsz;
+    } else {
+        error_report("Output size must be known when reading from stdin");
         ret = -1;
         goto out;
     }
 
-    if (dd.flags & C_COUNT && dd.count <= INT64_MAX / in.bsz &&
-        dd.count * in.bsz < size) {
+    if (!(dd.flags & C_OSIZE) && dd.flags & C_COUNT &&
+        dd.count <= INT64_MAX / in.bsz && dd.count * in.bsz < size) {
         size = dd.count * in.bsz;
     }
 
-    /* Overflow means the specified offset is beyond input image's size */
-    if (dd.flags & C_SKIP && (in.offset > INT64_MAX / in.bsz ||
-                              size < in.bsz * in.offset)) {
-        qemu_opt_set_number(opts, BLOCK_OPT_SIZE, 0, &error_abort);
-    } else {
-        qemu_opt_set_number(opts, BLOCK_OPT_SIZE,
-                            size - in.bsz * in.offset, &error_abort);
-    }
+    if (dd.flags & C_OF) {
+        drv = bdrv_find_format(out_fmt);
+        if (!drv) {
+            error_report("Unknown file format");
+            ret = -1;
+            goto out;
+        }
+        proto_drv = bdrv_find_protocol(out.filename, true, &local_err);
 
-    ret = bdrv_create(drv, out.filename, opts, &local_err);
-    if (ret < 0) {
-        error_reportf_err(local_err,
-                          "%s: error while creating output image: ",
-                          out.filename);
-        ret = -1;
-        goto out;
-    }
+        if (!proto_drv) {
+            error_report_err(local_err);
+            ret = -1;
+            goto out;
+        }
+        if (!drv->create_opts) {
+            error_report("Format driver '%s' does not support image creation",
+                         drv->format_name);
+            ret = -1;
+            goto out;
+        }
+        if (!proto_drv->create_opts) {
+            error_report("Protocol driver '%s' does not support image creation",
+                         proto_drv->format_name);
+            ret = -1;
+            goto out;
+        }
+        create_opts = qemu_opts_append(create_opts, drv->create_opts);
+        create_opts = qemu_opts_append(create_opts, proto_drv->create_opts);
 
-    /* TODO, we can't honour --image-opts for the target,
-     * since it needs to be given in a format compatible
-     * with the bdrv_create() call above which does not
-     * support image-opts style.
-     */
-    blk2 = img_open_file(out.filename, NULL, out_fmt, BDRV_O_RDWR,
-                         false, false, false);
+        opts = qemu_opts_create(create_opts, NULL, 0, &error_abort);
 
-    if (!blk2) {
-        ret = -1;
-        goto out;
+        /* Overflow means the specified offset is beyond input image's size */
+        if (dd.flags & C_OSIZE) {
+            qemu_opt_set_number(opts, BLOCK_OPT_SIZE, dd.osize, &error_abort);
+        } else if (dd.flags & C_SKIP && (in.offset > INT64_MAX / in.bsz ||
+                                         size < in.bsz * in.offset)) {
+            qemu_opt_set_number(opts, BLOCK_OPT_SIZE, 0, &error_abort);
+        } else {
+            qemu_opt_set_number(opts, BLOCK_OPT_SIZE,
+                                size - in.bsz * in.offset, &error_abort);
+        }
+
+        ret = bdrv_create(drv, out.filename, opts, &local_err);
+        if (ret < 0) {
+            error_reportf_err(local_err,
+                              "%s: error while creating output image: ",
+                              out.filename);
+            ret = -1;
+            goto out;
+        }
+
+        /* TODO, we can't honour --image-opts for the target,
+         * since it needs to be given in a format compatible
+         * with the bdrv_create() call above which does not
+         * support image-opts style.
+         */
+        blk2 = img_open_file(out.filename, NULL, out_fmt, BDRV_O_RDWR,
+                             false, false, false);
+
+        if (!blk2) {
+            ret = -1;
+            goto out;
+        }
     }
 
     if (dd.flags & C_SKIP && (in.offset > INT64_MAX / in.bsz ||
@@ -5113,13 +5161,24 @@ static int img_dd(int argc, char **argv)
 
     in.buf = g_new(uint8_t, in.bsz);
 
-    for (out_pos = 0; in_pos < size; block_count++) {
+    if (dd.flags & C_OSIZE && dd.osize < size - in_pos) {
+        readsize = in_pos + dd.osize;
+    } else {
+        readsize = size;
+    }
+    for (out_pos = 0; in_pos < readsize; block_count++) {
         int in_ret, out_ret;
-
-        if (in_pos + in.bsz > size) {
-            in_ret = blk_pread(blk1, in_pos, in.buf, size - in_pos);
+        size_t in_bsz = in_pos + in.bsz > readsize ? readsize - in_pos : in.bsz;
+        if (blk1) {
+            in_ret = blk_pread(blk1, in_pos, in.buf, in_bsz);
         } else {
-            in_ret = blk_pread(blk1, in_pos, in.buf, in.bsz);
+            in_ret = read(STDIN_FILENO, in.buf, in_bsz);
+            if (in_ret == 0) {
+                /* early EOF is considered an error */
+                error_report("Input ended unexpectedly");
+                ret = -1;
+                goto out;
+            }
         }
         if (in_ret < 0) {
             error_report("error while reading from input image file: %s",
@@ -5129,9 +5188,13 @@ static int img_dd(int argc, char **argv)
         }
         in_pos += in_ret;
 
-        out_ret = blk_pwrite(blk2, out_pos, in.buf, in_ret, 0);
+        if (blk2) {
+            out_ret = blk_pwrite(blk2, out_pos, in.buf, in_ret, 0);
+        } else {
+            out_ret = write(STDOUT_FILENO, in.buf, in_ret);
+        }
 
-        if (out_ret < 0) {
+        if (out_ret != in_ret) {
             error_report("error while writing to output image file: %s",
                          strerror(-out_ret));
             ret = -1;
-- 
2.30.2

[PATCH 2/4] qemu-img: dd: add isize parameter

[Fabian Ebner]

From: Wolfgang Bumiller <w.bumiller@proxmox.com>

for writing small images from stdin to bigger ones.

In order to distinguish between an actually unexpected and
an expected end of input.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
[FE: override size earlier
     use flag to detect parameter
     add documenation]
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---
 docs/tools/qemu-img.rst | 10 ++++++++--
 qemu-img-cmds.hx        |  4 ++--
 qemu-img.c              | 24 +++++++++++++++++++++++-
 3 files changed, 33 insertions(+), 5 deletions(-)

diff --git a/docs/tools/qemu-img.rst b/docs/tools/qemu-img.rst
index 775eaf3097..43328fe108 100644
--- a/docs/tools/qemu-img.rst
+++ b/docs/tools/qemu-img.rst
@@ -230,6 +230,10 @@ Parameters to dd subcommand:
 
   Sets the number of input blocks to skip
 
+.. option:: isize=INPUT_SIZE
+
+  Treat the input image or stream as if it had this size
+
 .. option:: osize=OUTPUT_SIZE
 
   Sets the output image's size
@@ -492,7 +496,7 @@ Command description:
   it doesn't need to be specified separately in this case.
 
 
-.. option:: dd [--image-opts] [-U] [-f FMT] [-O OUTPUT_FMT] [bs=BLOCK_SIZE] [count=BLOCKS] [skip=BLOCKS] [osize=OUTPUT_SIZE] [if=INPUT] [of=OUTPUT]
+.. option:: dd [--image-opts] [-U] [-f FMT] [-O OUTPUT_FMT] [bs=BLOCK_SIZE] [count=BLOCKS] [skip=BLOCKS] [isize=INPUT_SIZE] [osize=OUTPUT_SIZE] [if=INPUT] [of=OUTPUT]
 
   dd copies from *INPUT* file (default: STDIN) to *OUTPUT* file (default:
   STDOUT) converting it from *FMT* format to *OUTPUT_FMT* format.
@@ -504,7 +508,9 @@ Command description:
   The size syntax is similar to :manpage:`dd(1)`'s size syntax.
 
   The output image will be created with size *OUTPUT_SIZE* and at most this many
-  bytes will be copied.
+  bytes will be copied. When *INPUT_SIZE* is positive, it overrides the input
+  image's size for the copy operation. When *INPUT_SIZE* is zero and reading
+  from STDIN, do not treat premature end of the input stream as an error.
 
 .. option:: info [--object OBJECTDEF] [--image-opts] [-f FMT] [--output=OFMT] [--backing-chain] [-U] FILENAME
 
diff --git a/qemu-img-cmds.hx b/qemu-img-cmds.hx
index e4935365c9..50993e6c47 100644
--- a/qemu-img-cmds.hx
+++ b/qemu-img-cmds.hx
@@ -58,9 +58,9 @@ SRST
 ERST
 
 DEF("dd", img_dd,
-    "dd [--image-opts] [-U] [-f fmt] [-O output_fmt] [bs=block_size] [count=blocks] [skip=blocks] [osize=output_size] [if=input] [of=output]")
+    "dd [--image-opts] [-U] [-f fmt] [-O output_fmt] [bs=block_size] [count=blocks] [skip=blocks] [isize=input_size] [osize=output_size] [if=input] [of=output]")
 SRST
-.. option:: dd [--image-opts] [-U] [-f FMT] [-O OUTPUT_FMT] [bs=BLOCK_SIZE] [count=BLOCKS] [skip=BLOCKS] [osize=OUTPUT_SIZE] [if=INPUT] [of=OUTPUT]
+.. option:: dd [--image-opts] [-U] [-f FMT] [-O OUTPUT_FMT] [bs=BLOCK_SIZE] [count=BLOCKS] [skip=BLOCKS] [isize=INPUT_SIZE] [osize=OUTPUT_SIZE] [if=INPUT] [of=OUTPUT]
 ERST
 
 DEF("info", img_info,
diff --git a/qemu-img.c b/qemu-img.c
index ea488fd190..630928773d 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -4820,11 +4820,13 @@ static int img_bitmap(int argc, char **argv)
 #define C_OF      010
 #define C_SKIP    020
 #define C_OSIZE   040
+#define C_ISIZE   0100
 
 struct DdInfo {
     unsigned int flags;
     int64_t count;
     int64_t osize;
+    int64_t isize;
 };
 
 struct DdIo {
@@ -4913,6 +4915,19 @@ static int img_dd_osize(const char *arg,
     return 0;
 }
 
+static int img_dd_isize(const char *arg,
+                        struct DdIo *in, struct DdIo *out,
+                        struct DdInfo *dd)
+{
+    dd->isize = cvtnum("isize", arg);
+
+    if (dd->isize < 0) {
+        return 1;
+    }
+
+    return 0;
+}
+
 static int img_dd(int argc, char **argv)
 {
     int ret = 0;
@@ -4934,6 +4949,7 @@ static int img_dd(int argc, char **argv)
         .flags = 0,
         .count = 0,
         .osize = 0,
+        .isize = 0,
     };
     struct DdIo in = {
         .bsz = 512, /* Block size is by default 512 bytes */
@@ -4955,6 +4971,7 @@ static int img_dd(int argc, char **argv)
         { "of", img_dd_of, C_OF },
         { "skip", img_dd_skip, C_SKIP },
         { "osize", img_dd_osize, C_OSIZE },
+        { "isize", img_dd_isize, C_ISIZE },
         { NULL, NULL, 0 }
     };
     const struct option long_options[] = {
@@ -5061,7 +5078,9 @@ static int img_dd(int argc, char **argv)
         }
     }
 
-    if (dd.flags & C_IF) {
+    if (dd.flags & C_ISIZE && dd.isize > 0) {
+        size = dd.isize;
+    } else if (dd.flags & C_IF) {
         size = blk_getlength(blk1);
         if (size < 0) {
             error_report("Failed to get size for '%s'", in.filename);
@@ -5174,6 +5193,9 @@ static int img_dd(int argc, char **argv)
         } else {
             in_ret = read(STDIN_FILENO, in.buf, in_bsz);
             if (in_ret == 0) {
+                if (dd.flags & C_ISIZE && dd.isize == 0) {
+                    goto out;
+                }
                 /* early EOF is considered an error */
                 error_report("Input ended unexpectedly");
                 ret = -1;
-- 
2.30.2

[PATCH 3/4] qemu-img: dd: add -n option (skip target volume creation)

[Fabian Ebner]

From: Alexandre Derumier <aderumier@odiso.com>

Same rationale as in
b2e10493c7 ("add qemu-img convert -n option (skip target volume creation)")

Originally-by: Alexandre Derumier <aderumier@odiso.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
[FE: avoid wrong colon in getopt's optstring
     add documentation + commit message]
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---
 docs/tools/qemu-img.rst |  6 +++++-
 qemu-img-cmds.hx        |  4 ++--
 qemu-img.c              | 23 ++++++++++++++---------
 3 files changed, 21 insertions(+), 12 deletions(-)

diff --git a/docs/tools/qemu-img.rst b/docs/tools/qemu-img.rst
index 43328fe108..9b022d9363 100644
--- a/docs/tools/qemu-img.rst
+++ b/docs/tools/qemu-img.rst
@@ -210,6 +210,10 @@ Parameters to dd subcommand:
 
 .. program:: qemu-img-dd
 
+.. option:: -n
+
+  Skip the creation of the target volume
+
 .. option:: bs=BLOCK_SIZE
 
   Defines the block size
@@ -496,7 +500,7 @@ Command description:
   it doesn't need to be specified separately in this case.
 
 
-.. option:: dd [--image-opts] [-U] [-f FMT] [-O OUTPUT_FMT] [bs=BLOCK_SIZE] [count=BLOCKS] [skip=BLOCKS] [isize=INPUT_SIZE] [osize=OUTPUT_SIZE] [if=INPUT] [of=OUTPUT]
+.. option:: dd [--image-opts] [-U] [-f FMT] [-O OUTPUT_FMT] [-n] [bs=BLOCK_SIZE] [count=BLOCKS] [skip=BLOCKS] [isize=INPUT_SIZE] [osize=OUTPUT_SIZE] [if=INPUT] [of=OUTPUT]
 
   dd copies from *INPUT* file (default: STDIN) to *OUTPUT* file (default:
   STDOUT) converting it from *FMT* format to *OUTPUT_FMT* format.
diff --git a/qemu-img-cmds.hx b/qemu-img-cmds.hx
index 50993e6c47..97e750623f 100644
--- a/qemu-img-cmds.hx
+++ b/qemu-img-cmds.hx
@@ -58,9 +58,9 @@ SRST
 ERST
 
 DEF("dd", img_dd,
-    "dd [--image-opts] [-U] [-f fmt] [-O output_fmt] [bs=block_size] [count=blocks] [skip=blocks] [isize=input_size] [osize=output_size] [if=input] [of=output]")
+    "dd [--image-opts] [-U] [-f fmt] [-O output_fmt] [-n] [bs=block_size] [count=blocks] [skip=blocks] [isize=input_size] [osize=output_size] [if=input] [of=output]")
 SRST
-.. option:: dd [--image-opts] [-U] [-f FMT] [-O OUTPUT_FMT] [bs=BLOCK_SIZE] [count=BLOCKS] [skip=BLOCKS] [isize=INPUT_SIZE] [osize=OUTPUT_SIZE] [if=INPUT] [of=OUTPUT]
+.. option:: dd [--image-opts] [-U] [-f FMT] [-O OUTPUT_FMT] [-n] [bs=BLOCK_SIZE] [count=BLOCKS] [skip=BLOCKS] [isize=INPUT_SIZE] [osize=OUTPUT_SIZE] [if=INPUT] [of=OUTPUT]
 ERST
 
 DEF("info", img_info,
diff --git a/qemu-img.c b/qemu-img.c
index 630928773d..89bf6fd087 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -4944,7 +4944,7 @@ static int img_dd(int argc, char **argv)
     const char *fmt = NULL;
     int64_t size = 0, readsize = 0;
     int64_t block_count = 0, out_pos, in_pos;
-    bool force_share = false;
+    bool force_share = false, skip_create = false;
     struct DdInfo dd = {
         .flags = 0,
         .count = 0,
@@ -4982,7 +4982,7 @@ static int img_dd(int argc, char **argv)
         { 0, 0, 0, 0 }
     };
 
-    while ((c = getopt_long(argc, argv, ":hf:O:U", long_options, NULL))) {
+    while ((c = getopt_long(argc, argv, ":hf:O:Un", long_options, NULL))) {
         if (c == EOF) {
             break;
         }
@@ -5002,6 +5002,9 @@ static int img_dd(int argc, char **argv)
         case 'h':
             help();
             break;
+        case 'n':
+            skip_create = true;
+            break;
         case 'U':
             force_share = true;
             break;
@@ -5144,13 +5147,15 @@ static int img_dd(int argc, char **argv)
                                 size - in.bsz * in.offset, &error_abort);
         }
 
-        ret = bdrv_create(drv, out.filename, opts, &local_err);
-        if (ret < 0) {
-            error_reportf_err(local_err,
-                              "%s: error while creating output image: ",
-                              out.filename);
-            ret = -1;
-            goto out;
+        if (!skip_create) {
+            ret = bdrv_create(drv, out.filename, opts, &local_err);
+            if (ret < 0) {
+                error_reportf_err(local_err,
+                                  "%s: error while creating output image: ",
+                                  out.filename);
+                ret = -1;
+                goto out;
+            }
         }
 
         /* TODO, we can't honour --image-opts for the target,
-- 
2.30.2

[PATCH 4/4] qemu-img: dd: add -l option for loading a snapshot

[Fabian Ebner]

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---
 docs/tools/qemu-img.rst |  7 ++++---
 qemu-img-cmds.hx        |  4 ++--
 qemu-img.c              | 33 +++++++++++++++++++++++++++++++--
 3 files changed, 37 insertions(+), 7 deletions(-)

diff --git a/docs/tools/qemu-img.rst b/docs/tools/qemu-img.rst
index 9b022d9363..b2333d7b04 100644
--- a/docs/tools/qemu-img.rst
+++ b/docs/tools/qemu-img.rst
@@ -500,10 +500,11 @@ Command description:
   it doesn't need to be specified separately in this case.
 
 
-.. option:: dd [--image-opts] [-U] [-f FMT] [-O OUTPUT_FMT] [-n] [bs=BLOCK_SIZE] [count=BLOCKS] [skip=BLOCKS] [isize=INPUT_SIZE] [osize=OUTPUT_SIZE] [if=INPUT] [of=OUTPUT]
+.. option:: dd [--image-opts] [-U] [-f FMT] [-O OUTPUT_FMT] [-n] [-l SNAPSHOT_PARAM] [bs=BLOCK_SIZE] [count=BLOCKS] [skip=BLOCKS] [isize=INPUT_SIZE] [osize=OUTPUT_SIZE] [if=INPUT] [of=OUTPUT]
 
-  dd copies from *INPUT* file (default: STDIN) to *OUTPUT* file (default:
-  STDOUT) converting it from *FMT* format to *OUTPUT_FMT* format.
+  dd copies from *INPUT* file (default: STDIN) or snapshot *SNAPSHOT_PARAM* to
+  *OUTPUT* file (default: STDOUT) converting it from *FMT* format to
+  *OUTPUT_FMT* format.
 
   The data is by default read and written using blocks of 512 bytes but can be
   modified by specifying *BLOCK_SIZE*. If count=\ *BLOCKS* is specified
diff --git a/qemu-img-cmds.hx b/qemu-img-cmds.hx
index 97e750623f..2f527306b0 100644
--- a/qemu-img-cmds.hx
+++ b/qemu-img-cmds.hx
@@ -58,9 +58,9 @@ SRST
 ERST
 
 DEF("dd", img_dd,
-    "dd [--image-opts] [-U] [-f fmt] [-O output_fmt] [-n] [bs=block_size] [count=blocks] [skip=blocks] [isize=input_size] [osize=output_size] [if=input] [of=output]")
+    "dd [--image-opts] [-U] [-f fmt] [-O output_fmt] [-n] [-l snapshot_param] [bs=block_size] [count=blocks] [skip=blocks] [isize=input_size] [osize=output_size] [if=input] [of=output]")
 SRST
-.. option:: dd [--image-opts] [-U] [-f FMT] [-O OUTPUT_FMT] [-n] [bs=BLOCK_SIZE] [count=BLOCKS] [skip=BLOCKS] [isize=INPUT_SIZE] [osize=OUTPUT_SIZE] [if=INPUT] [of=OUTPUT]
+.. option:: dd [--image-opts] [-U] [-f FMT] [-O OUTPUT_FMT] [-n] [-l SNAPSHOT_PARAM] [bs=BLOCK_SIZE] [count=BLOCKS] [skip=BLOCKS] [isize=INPUT_SIZE] [osize=OUTPUT_SIZE] [if=INPUT] [of=OUTPUT]
 ERST
 
 DEF("info", img_info,
diff --git a/qemu-img.c b/qemu-img.c
index 89bf6fd087..28b6430800 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -4936,6 +4936,7 @@ static int img_dd(int argc, char **argv)
     BlockDriver *drv = NULL, *proto_drv = NULL;
     BlockBackend *blk1 = NULL, *blk2 = NULL;
     QemuOpts *opts = NULL;
+    QemuOpts *sn_opts = NULL;
     QemuOptsList *create_opts = NULL;
     Error *local_err = NULL;
     bool image_opts = false;
@@ -4945,6 +4946,7 @@ static int img_dd(int argc, char **argv)
     int64_t size = 0, readsize = 0;
     int64_t block_count = 0, out_pos, in_pos;
     bool force_share = false, skip_create = false;
+    const char *snapshot_name = NULL;
     struct DdInfo dd = {
         .flags = 0,
         .count = 0,
@@ -4982,7 +4984,7 @@ static int img_dd(int argc, char **argv)
         { 0, 0, 0, 0 }
     };
 
-    while ((c = getopt_long(argc, argv, ":hf:O:Un", long_options, NULL))) {
+    while ((c = getopt_long(argc, argv, ":hf:O:l:Un", long_options, NULL))) {
         if (c == EOF) {
             break;
         }
@@ -5005,6 +5007,19 @@ static int img_dd(int argc, char **argv)
         case 'n':
             skip_create = true;
             break;
+        case 'l':
+            if (strstart(optarg, SNAPSHOT_OPT_BASE, NULL)) {
+                sn_opts = qemu_opts_parse_noisily(&internal_snapshot_opts,
+                                                  optarg, false);
+                if (!sn_opts) {
+                    error_report("Failed in parsing snapshot param '%s'",
+                                 optarg);
+                    goto out;
+                }
+            } else {
+                snapshot_name = optarg;
+            }
+            break;
         case 'U':
             force_share = true;
             break;
@@ -5074,11 +5089,24 @@ static int img_dd(int argc, char **argv)
     if (dd.flags & C_IF) {
         blk1 = img_open(image_opts, in.filename, fmt, 0, false, false,
                         force_share);
-
         if (!blk1) {
             ret = -1;
             goto out;
         }
+        if (sn_opts) {
+            bdrv_snapshot_load_tmp(blk_bs(blk1),
+                                   qemu_opt_get(sn_opts, SNAPSHOT_OPT_ID),
+                                   qemu_opt_get(sn_opts, SNAPSHOT_OPT_NAME),
+                                   &local_err);
+        } else if (snapshot_name != NULL) {
+            bdrv_snapshot_load_tmp_by_id_or_name(blk_bs(blk1), snapshot_name,
+                                                 &local_err);
+        }
+        if (local_err) {
+            error_reportf_err(local_err, "Failed to load snapshot: ");
+            ret = -1;
+            goto out;
+        }
     }
 
     if (dd.flags & C_ISIZE && dd.isize > 0) {
@@ -5233,6 +5261,7 @@ static int img_dd(int argc, char **argv)
 out:
     g_free(arg);
     qemu_opts_del(opts);
+    qemu_opts_del(sn_opts);
     qemu_opts_free(create_opts);
     blk_unref(blk1);
     blk_unref(blk2);
-- 
2.30.2

Re: [PATCH 0/4] Make qemu-img dd more flexible

[Eric Blake]

On Thu, Feb 10, 2022 at 02:31:19PM +0100, Fabian Ebner wrote:
> Adds support for reading from stdin and writing to stdout (when raw
> format is used), as well as overriding the size of the output and
> input image/stream.
> 
> Additionally, the options -n for skipping output image creation and -l
> for loading a snapshot are made available like for convert.

Without looking at the series itself, I want to refer back to earlier
times that someone proposed improving 'qemu-img dd':

https://lists.gnu.org/archive/html/qemu-devel/2021-02/msg00636.html
https://lists.gnu.org/archive/html/qemu-devel/2018-08/msg02618.html

As well as the observation that when we originally allowed 'qemu-img
dd' to be added, the end goal was that if 'qemu-img dd' can't operate
as a thin wrapper around 'qemu-img convert', then 'qemu-img convert'
needs to be made more powerful first.  Every time we diverge on what
the two uses can do, rather than keeping dd as a thin wrapper, we add
to our maintenance burden.

Sadly, there is a lot of technical debt in this area ('qemu-img dd
skip= count=' is STILL broken, more than 4 years after I first
proposed a potential patch), where no one has spent the necessary time
to improve the situation.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [PATCH 0/4] Make qemu-img dd more flexible

[Hanna Reitz]

On 11.02.22 17:31, Eric Blake wrote:
> On Thu, Feb 10, 2022 at 02:31:19PM +0100, Fabian Ebner wrote:
>> Adds support for reading from stdin and writing to stdout (when raw
>> format is used), as well as overriding the size of the output and
>> input image/stream.
>>
>> Additionally, the options -n for skipping output image creation and -l
>> for loading a snapshot are made available like for convert.
> Without looking at the series itself, I want to refer back to earlier
> times that someone proposed improving 'qemu-img dd':
>
> https://lists.gnu.org/archive/html/qemu-devel/2021-02/msg00636.html
> https://lists.gnu.org/archive/html/qemu-devel/2018-08/msg02618.html
>
> As well as the observation that when we originally allowed 'qemu-img
> dd' to be added, the end goal was that if 'qemu-img dd' can't operate
> as a thin wrapper around 'qemu-img convert', then 'qemu-img convert'
> needs to be made more powerful first.  Every time we diverge on what
> the two uses can do, rather than keeping dd as a thin wrapper, we add
> to our maintenance burden.
>
> Sadly, there is a lot of technical debt in this area ('qemu-img dd
> skip= count=' is STILL broken, more than 4 years after I first
> proposed a potential patch), where no one has spent the necessary time
> to improve the situation.

Note that by now (in contrast to 2018), we have FUSE disk exports, and I 
even have a script that uses them to let you run dd on any image:

https://gitlab.com/hreitz/qemu-scripts/-/blob/main/qemu-dd.py

Which is nice, because it gives you feature parity with dd, because it 
simply runs dd.

(The main problem with the script is that it lives in that personal repo 
of mine and so nobody but me knows about it.  Suggestions to improve 
that situation are more than welcome.)

Now, the qemu storage daemon does not support loading qcow2 snapshots 
(as far as I’m aware), which is proposed in patch 4 of this series.  But 
I think that just means that it would be nice if the QSD could support that.

Hanna

Re: [PATCH 0/4] Make qemu-img dd more flexible

[Fabian Ebner]

Am 11.02.22 um 17:42 schrieb Hanna Reitz:
> On 11.02.22 17:31, Eric Blake wrote:
>> On Thu, Feb 10, 2022 at 02:31:19PM +0100, Fabian Ebner wrote:
>>> Adds support for reading from stdin and writing to stdout (when raw
>>> format is used), as well as overriding the size of the output and
>>> input image/stream.
>>>
>>> Additionally, the options -n for skipping output image creation and -l
>>> for loading a snapshot are made available like for convert.
>> Without looking at the series itself, I want to refer back to earlier
>> times that someone proposed improving 'qemu-img dd':
>>
>> https://lists.gnu.org/archive/html/qemu-devel/2021-02/msg00636.html
>> https://lists.gnu.org/archive/html/qemu-devel/2018-08/msg02618.html
>>
>> As well as the observation that when we originally allowed 'qemu-img
>> dd' to be added, the end goal was that if 'qemu-img dd' can't operate
>> as a thin wrapper around 'qemu-img convert', then 'qemu-img convert'
>> needs to be made more powerful first.  Every time we diverge on what
>> the two uses can do, rather than keeping dd as a thin wrapper, we add
>> to our maintenance burden.

I'm wondering why it's not actually implemented as a thin wrapper then?
The fact that it isn't is (part of) the reason why dd was chosen, as
mentioned in the first patch:

"While dd and convert have overlapping use cases, `dd` is a
simple read/write loop while convert is much more
sophisticated and has ways to dealing with holes and blocks
of zeroes.
Since these typically can't be detected in pipes via
SEEK_DATA/HOLE or skipped while writing, dd seems to be the
better choice for implementing stdin/stdout streams."

Adding the same feature to convert seems much more involved.

>>
>> Sadly, there is a lot of technical debt in this area ('qemu-img dd
>> skip= count=' is STILL broken, more than 4 years after I first
>> proposed a potential patch), where no one has spent the necessary time
>> to improve the situation.
> 
> Note that by now (in contrast to 2018), we have FUSE disk exports, and I
> even have a script that uses them to let you run dd on any image:
> 
> https://gitlab.com/hreitz/qemu-scripts/-/blob/main/qemu-dd.py
> 
> Which is nice, because it gives you feature parity with dd, because it
> simply runs dd.

Thank you for the suggestion. It's definitely worth considering,
although it does add a bit of complexity and we currently don't have
FUSE support enabled in our builds.

> 
> (The main problem with the script is that it lives in that personal repo
> of mine and so nobody but me knows about it.  Suggestions to improve
> that situation are more than welcome.)
> 
> Now, the qemu storage daemon does not support loading qcow2 snapshots
> (as far as I’m aware), which is proposed in patch 4 of this series.  But
> I think that just means that it would be nice if the QSD could support
> that.

I suppose adding a snapshot-load QMP command would be a natural way to
add it?

> 
> Hanna
> 
>