[Qemu-devel] [PULL 00/22] Block patches

[Qemu-devel] [PULL 00/22] Block patches

[Kevin Wolf]

The following changes since commit 331636431af32ece373f4b1fb7c3ae9d0615e2a6:

  vga: compile cirrus_vga in hwlib (2012-01-25 18:32:59 +0000)

are available in the git repository at:
  git://repo.or.cz/qemu/kevin.git for-anthony

Gregory Farnum (1):
      rbd: wire up snapshot removal and rollback functionality

Li Zhi Hui (2):
      qcow: Return real error code in qcow_open
      qcow: Use bdrv functions to replace file operation

Marcelo Tosatti (4):
      block: add bdrv_find_backing_image
      add QERR_BASE_NOT_FOUND
      block: add support for partial streaming
      docs: describe live block operations

Stefan Hajnoczi (13):
      block: replace unchecked strdup/malloc/calloc with glib
      coroutine: add co_sleep_ns() coroutine sleep function
      block: check bdrv_in_use() before blockdev operations
      block: make copy-on-read a per-request flag
      block: add BlockJob interface for long-running operations
      block: add image streaming block job
      block: rate-limit streaming operations
      qmp: add block_stream command
      qmp: add block_job_set_speed command
      qmp: add block_job_cancel command
      qmp: add query-block-jobs
      blockdev: make image streaming safe across hotplug
      virtio-blk: add virtio_blk_handle_read trace event

Stefan Weil (1):
      block/vdi: Zero unused parts when allocating a new block (fix #919242)

Thomas Higdon (1):
      scsi: Guard against buflen exceeding req->cmd.xfer in scsi_disk_emulate_command

 Makefile.objs           |    2 +
 QMP/qmp-events.txt      |   53 +++++++++
 block.c                 |  119 +++++++++++++++++++--
 block.h                 |    4 +
 block/blkdebug.c        |    4 +-
 block/blkverify.c       |    4 +-
 block/qcow.c            |  104 ++++++++++++-------
 block/rbd.c             |   22 ++++
 block/stream.c          |  269 +++++++++++++++++++++++++++++++++++++++++++++++
 block/vdi.c             |    8 +-
 block_int.h             |   47 ++++++++
 blockdev.c              |  199 ++++++++++++++++++++++++++++++++++-
 docs/live-block-ops.txt |   58 ++++++++++
 hmp-commands.hx         |   41 +++++++
 hmp.c                   |   68 ++++++++++++
 hmp.h                   |    4 +
 hw/scsi-disk.c          |   10 +--
 hw/virtio-blk.c         |    2 +
 monitor.c               |   13 +++
 monitor.h               |    2 +
 qapi-schema.json        |  115 ++++++++++++++++++++
 qemu-coroutine-sleep.c  |   38 +++++++
 qemu-coroutine.h        |    9 ++
 qemu-io.c               |   48 ++++----
 qerror.c                |    8 ++
 qerror.h                |    6 +
 qmp-commands.hx         |   24 ++++
 trace-events            |   13 ++-
 28 files changed, 1206 insertions(+), 88 deletions(-)
 create mode 100644 block/stream.c
 create mode 100644 docs/live-block-ops.txt
 create mode 100644 qemu-coroutine-sleep.c

Re: [Qemu-devel] [PULL 00/22] Block patches

[Anthony Liguori]

On 01/26/2012 10:37 AM, Kevin Wolf wrote:
> The following changes since commit 331636431af32ece373f4b1fb7c3ae9d0615e2a6:
>
>    vga: compile cirrus_vga in hwlib (2012-01-25 18:32:59 +0000)
>
> are available in the git repository at:
>    git://repo.or.cz/qemu/kevin.git for-anthony

Pulled.  Thanks.

Regards,

Anthony Liguori

>
> Gregory Farnum (1):
>        rbd: wire up snapshot removal and rollback functionality
>
> Li Zhi Hui (2):
>        qcow: Return real error code in qcow_open
>        qcow: Use bdrv functions to replace file operation
>
> Marcelo Tosatti (4):
>        block: add bdrv_find_backing_image
>        add QERR_BASE_NOT_FOUND
>        block: add support for partial streaming
>        docs: describe live block operations
>
> Stefan Hajnoczi (13):
>        block: replace unchecked strdup/malloc/calloc with glib
>        coroutine: add co_sleep_ns() coroutine sleep function
>        block: check bdrv_in_use() before blockdev operations
>        block: make copy-on-read a per-request flag
>        block: add BlockJob interface for long-running operations
>        block: add image streaming block job
>        block: rate-limit streaming operations
>        qmp: add block_stream command
>        qmp: add block_job_set_speed command
>        qmp: add block_job_cancel command
>        qmp: add query-block-jobs
>        blockdev: make image streaming safe across hotplug
>        virtio-blk: add virtio_blk_handle_read trace event
>
> Stefan Weil (1):
>        block/vdi: Zero unused parts when allocating a new block (fix #919242)
>
> Thomas Higdon (1):
>        scsi: Guard against buflen exceeding req->cmd.xfer in scsi_disk_emulate_command
>
>   Makefile.objs           |    2 +
>   QMP/qmp-events.txt      |   53 +++++++++
>   block.c                 |  119 +++++++++++++++++++--
>   block.h                 |    4 +
>   block/blkdebug.c        |    4 +-
>   block/blkverify.c       |    4 +-
>   block/qcow.c            |  104 ++++++++++++-------
>   block/rbd.c             |   22 ++++
>   block/stream.c          |  269 +++++++++++++++++++++++++++++++++++++++++++++++
>   block/vdi.c             |    8 +-
>   block_int.h             |   47 ++++++++
>   blockdev.c              |  199 ++++++++++++++++++++++++++++++++++-
>   docs/live-block-ops.txt |   58 ++++++++++
>   hmp-commands.hx         |   41 +++++++
>   hmp.c                   |   68 ++++++++++++
>   hmp.h                   |    4 +
>   hw/scsi-disk.c          |   10 +--
>   hw/virtio-blk.c         |    2 +
>   monitor.c               |   13 +++
>   monitor.h               |    2 +
>   qapi-schema.json        |  115 ++++++++++++++++++++
>   qemu-coroutine-sleep.c  |   38 +++++++
>   qemu-coroutine.h        |    9 ++
>   qemu-io.c               |   48 ++++----
>   qerror.c                |    8 ++
>   qerror.h                |    6 +
>   qmp-commands.hx         |   24 ++++
>   trace-events            |   13 ++-
>   28 files changed, 1206 insertions(+), 88 deletions(-)
>   create mode 100644 block/stream.c
>   create mode 100644 docs/live-block-ops.txt
>   create mode 100644 qemu-coroutine-sleep.c
>
>

[Qemu-devel] [PULL 00/22] Block patches

[Kevin Wolf]

The following changes since commit 1a381811b495651ddfc2b50d7c8cfaaf375816b0:

  Merge remote-tracking branch 'remotes/rth/tags/pull-tcg-s390-20140515' into staging (2014-05-15 18:56:08 +0100)

are available in the git repository at:


  git://repo.or.cz/qemu/kevin.git tags/for-upstream

for you to fetch changes up to 465bee1da82e43f18d10c43cc7566d0284ad13a9:

  block: optimize zero writes with bdrv_write_zeroes (2014-05-19 13:42:27 +0200)

----------------------------------------------------------------
Block patches

----------------------------------------------------------------
Fam Zheng (1):
      qemu-iotests: Fix blkdebug in VM drive in 030

Jeff Cody (2):
      block: vhdx - account for identical header sections
      block: add test for vhdx image created by Disk2VHD

Kevin Wolf (6):
      block: Fix bdrv_is_allocated() for short backing files
      qcow1: Make padding in the header explicit
      qcow1: Check maximum cluster size
      qcow1: Validate L2 table size (CVE-2014-0222)
      qcow1: Validate image size (CVE-2014-0223)
      qcow1: Stricter backing file length check

Markus Armbruster (1):
      qemu-iotests: Fix core dump suppression in test 039

Matthew Booth (4):
      curl: Fix build when curl_multi_socket_action isn't available
      curl: Remove broken parsing of options from url
      curl: Add sslverify option
      curl: Add usage documentation

Max Reitz (4):
      qdict: Add qdict_join()
      check-qdict: Add test for qdict_join()
      block: Allow JSON filenames
      iotests: Add test for the JSON protocol

Mike Day (1):
      Remove g_sequence_lookup from qemu-img help function

Peter Lieven (3):
      util: add qemu_iovec_is_zero
      blockdev: add a function to parse enum ids from strings
      block: optimize zero writes with bdrv_write_zeroes

 block.c                                            |  60 +++++++++-
 block/curl.c                                       |  79 ++++++-------
 block/qapi.c                                       |   1 +
 block/qcow.c                                       |  44 +++++--
 block/vhdx.c                                       |   9 +-
 blockdev.c                                         |  43 +++++++
 hmp.c                                              |   5 +
 include/block/block.h                              |  11 +-
 include/block/block_int.h                          |   1 +
 include/qapi/qmp/qdict.h                           |   3 +
 include/qemu-common.h                              |   1 +
 qapi-schema.json                                   |  52 ++++++---
 qemu-img.c                                         |   7 +-
 qemu-options.hx                                    |  74 ++++++++++++
 qmp-commands.hx                                    |   3 +
 qobject/qdict.c                                    |  32 +++++
 tests/check-qdict.c                                |  87 ++++++++++++++
 tests/qemu-iotests/030                             |   2 +-
 tests/qemu-iotests/039                             |  20 ++--
 tests/qemu-iotests/039.out                         |   3 +
 tests/qemu-iotests/067.out                         |  10 +-
 tests/qemu-iotests/070                             |   7 ++
 tests/qemu-iotests/070.out                         |   7 ++
 tests/qemu-iotests/089                             | 130 +++++++++++++++++++++
 tests/qemu-iotests/089.out                         |  54 +++++++++
 tests/qemu-iotests/092                             |  98 ++++++++++++++++
 tests/qemu-iotests/092.out                         |  38 ++++++
 tests/qemu-iotests/common.filter                   |   1 +
 tests/qemu-iotests/group                           |   2 +
 .../sample_images/test-disk2vhd.vhdx.bz2           | Bin 0 -> 1424 bytes
 util/iov.c                                         |  21 ++++
 31 files changed, 809 insertions(+), 96 deletions(-)
 create mode 100755 tests/qemu-iotests/089
 create mode 100644 tests/qemu-iotests/089.out
 create mode 100755 tests/qemu-iotests/092
 create mode 100644 tests/qemu-iotests/092.out
 create mode 100644 tests/qemu-iotests/sample_images/test-disk2vhd.vhdx.bz2

[Qemu-devel] [PULL 01/22] block: Fix bdrv_is_allocated() for short backing files

[Kevin Wolf]

bdrv_is_allocated() shouldn't return true for sectors that are
unallocated, but after the end of a short backing file, even though
such sectors are (correctly) marked as containing zeros.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
---
 block.c               | 10 ++++++----
 include/block/block.h | 11 +++++++----
 2 files changed, 13 insertions(+), 8 deletions(-)

diff --git a/block.c b/block.c
index c90c71a..65e8191 100644
--- a/block.c
+++ b/block.c
@@ -3864,7 +3864,7 @@ static int64_t coroutine_fn bdrv_co_get_block_status(BlockDriverState *bs,
 
     if (!bs->drv->bdrv_co_get_block_status) {
         *pnum = nb_sectors;
-        ret = BDRV_BLOCK_DATA;
+        ret = BDRV_BLOCK_DATA | BDRV_BLOCK_ALLOCATED;
         if (bs->drv->protocol_name) {
             ret |= BDRV_BLOCK_OFFSET_VALID | (sector_num * BDRV_SECTOR_SIZE);
         }
@@ -3883,6 +3883,10 @@ static int64_t coroutine_fn bdrv_co_get_block_status(BlockDriverState *bs,
                                      *pnum, pnum);
     }
 
+    if (ret & (BDRV_BLOCK_DATA | BDRV_BLOCK_ZERO)) {
+        ret |= BDRV_BLOCK_ALLOCATED;
+    }
+
     if (!(ret & BDRV_BLOCK_DATA) && !(ret & BDRV_BLOCK_ZERO)) {
         if (bdrv_unallocated_blocks_are_zero(bs)) {
             ret |= BDRV_BLOCK_ZERO;
@@ -3959,9 +3963,7 @@ int coroutine_fn bdrv_is_allocated(BlockDriverState *bs, int64_t sector_num,
     if (ret < 0) {
         return ret;
     }
-    return
-        (ret & BDRV_BLOCK_DATA) ||
-        ((ret & BDRV_BLOCK_ZERO) && !bdrv_has_zero_init(bs));
+    return (ret & BDRV_BLOCK_ALLOCATED);
 }
 
 /*
diff --git a/include/block/block.h b/include/block/block.h
index 1b119aa..59be83f 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -120,6 +120,8 @@ typedef enum {
 /* BDRV_BLOCK_DATA: data is read from bs->file or another file
  * BDRV_BLOCK_ZERO: sectors read as zero
  * BDRV_BLOCK_OFFSET_VALID: sector stored in bs->file as raw data
+ * BDRV_BLOCK_ALLOCATED: the content of the block is determined by this
+ *                       layer (as opposed to the backing file)
  * BDRV_BLOCK_RAW: used internally to indicate that the request
  *                 was answered by the raw driver and that one
  *                 should look in bs->file directly.
@@ -141,10 +143,11 @@ typedef enum {
  *  f    t        f       not allocated or unknown offset, read as zero
  *  f    f        f       not allocated or unknown offset, read from backing_hd
  */
-#define BDRV_BLOCK_DATA         1
-#define BDRV_BLOCK_ZERO         2
-#define BDRV_BLOCK_OFFSET_VALID 4
-#define BDRV_BLOCK_RAW          8
+#define BDRV_BLOCK_DATA         0x01
+#define BDRV_BLOCK_ZERO         0x02
+#define BDRV_BLOCK_OFFSET_VALID 0x04
+#define BDRV_BLOCK_RAW          0x08
+#define BDRV_BLOCK_ALLOCATED    0x10
 #define BDRV_BLOCK_OFFSET_MASK  BDRV_SECTOR_MASK
 
 typedef enum {
-- 
1.8.3.1

[Qemu-devel] [PULL 02/22] Remove g_sequence_lookup from qemu-img help function

[Kevin Wolf]

From: Mike Day <ncmike@ncultra.org>

g_sequence_lookup is not supported by glib < 2.28. The usage
of g_sequence_lookup is not essential in this context (it's a
safeguard against duplicate values in the help message).
Removing the call enables the build on all platforms and
does not change the operation of the help function.

Signed-off-by: Mike Day <ncmike@ncultra.org>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 qemu-img.c | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

diff --git a/qemu-img.c b/qemu-img.c
index 04ce02a..1ad899e 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -70,11 +70,8 @@ static void add_format_to_seq(void *opaque, const char *fmt_name)
 {
     GSequence *seq = opaque;
 
-    if (!g_sequence_lookup(seq, (gpointer)fmt_name,
-                           compare_data, NULL)) {
-        g_sequence_insert_sorted(seq, (gpointer)fmt_name,
-                                 compare_data, NULL);
-    }
+    g_sequence_insert_sorted(seq, (gpointer)fmt_name,
+                             compare_data, NULL);
 }
 
 static void QEMU_NORETURN GCC_FMT_ATTR(1, 2) error_exit(const char *fmt, ...)
-- 
1.8.3.1

[Qemu-devel] [PULL 03/22] block: vhdx - account for identical header sections

[Kevin Wolf]

From: Jeff Cody <jcody@redhat.com>

The VHDX spec v1.00 declares that "a header is current if it is the only
valid header or if it is valid and its SequenceNumber field is greater
than the other header’s SequenceNumber field. The parser must only use
data from the current header. If there is no current header, then the
VHDX file is corrupt."

However, the Disk2VHD tool from Microsoft creates a VHDX image file that
has 2 identical headers, including matching checksums and matching
sequence numbers.  Likely, as a shortcut the tool is just writing the
header twice, for the active and inactive headers, during the image
creation.  Technically, this should be considered a corrupt VHDX file
(at least per the 1.00 spec, and that is how we currently treat it).

But in order to accomodate images created with Disk2VHD, we can safely
create an exception for this case.  If we find identical sequence
numbers, then we check the VHDXHeader-sized chunks of each 64KB header
sections (we won't rely just on the crc32c to indicate the headers are
the same).  If they are identical, then we go ahead and use the first
one.

Reported-by: Nerijus Baliūnas <nerijus@users.sourceforge.net>
Signed-off-by: Jeff Cody <jcody@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 block/vhdx.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/block/vhdx.c b/block/vhdx.c
index 509baaf..353c74d 100644
--- a/block/vhdx.c
+++ b/block/vhdx.c
@@ -473,7 +473,14 @@ static void vhdx_parse_header(BlockDriverState *bs, BDRVVHDXState *s,
         } else if (h2_seq > h1_seq) {
             s->curr_header = 1;
         } else {
-            goto fail;
+            /* The Microsoft Disk2VHD tool will create 2 identical
+             * headers, with identical sequence numbers.  If the headers are
+             * identical, don't consider the file corrupt */
+            if (!memcmp(header1, header2, sizeof(VHDXHeader))) {
+                s->curr_header = 0;
+            } else {
+                goto fail;
+            }
         }
     }
 
-- 
1.8.3.1

[Qemu-devel] [PULL 04/22] block: add test for vhdx image created by Disk2VHD

[Kevin Wolf]

From: Jeff Cody <jcody@redhat.com>

This adds a test for VHDX images created by Microsoft's tool, Disk2VHD.

VHDX images created by this tool have 2 identical header sections, with
identical sequence numbers.  This makes sure we detect VHDX images with
identical headers, and do not flag them as corrupt.

Signed-off-by: Jeff Cody <jcody@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 tests/qemu-iotests/070                                  |   7 +++++++
 tests/qemu-iotests/070.out                              |   7 +++++++
 tests/qemu-iotests/sample_images/test-disk2vhd.vhdx.bz2 | Bin 0 -> 1424 bytes
 3 files changed, 14 insertions(+)
 create mode 100644 tests/qemu-iotests/sample_images/test-disk2vhd.vhdx.bz2

diff --git a/tests/qemu-iotests/070 b/tests/qemu-iotests/070
index ce71fa4..ea0dae7 100755
--- a/tests/qemu-iotests/070
+++ b/tests/qemu-iotests/070
@@ -72,6 +72,13 @@ echo "=== Verify open image read-only succeeds after log replay ==="
 $QEMU_IO -r -c "read -pP 0xa5 0 18M" "$TEST_IMG" 2>&1 | _filter_testdir \
                                                       | _filter_qemu_io
 
+_cleanup_test_img
+_use_sample_img test-disk2vhd.vhdx.bz2
+
+echo
+echo "=== Verify image created by Disk2VHD can be opened ==="
+$QEMU_IMG info "$TEST_IMG" 2>&1 | _filter_testdir | _filter_qemu
+
 # success, all done
 echo "*** done"
 rm -f $seq.full
diff --git a/tests/qemu-iotests/070.out b/tests/qemu-iotests/070.out
index 922d62c..15f1fc1 100644
--- a/tests/qemu-iotests/070.out
+++ b/tests/qemu-iotests/070.out
@@ -18,4 +18,11 @@ No errors were found on the image.
 === Verify open image read-only succeeds after log replay ===
 read 18874368/18874368 bytes at offset 0
 18 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+=== Verify image created by Disk2VHD can be opened ===
+image: TEST_DIR/test-disk2vhd.vhdx
+file format: vhdx
+virtual size: 256M (268435456 bytes)
+disk size: 260M
+cluster_size: 2097152
 *** done
diff --git a/tests/qemu-iotests/sample_images/test-disk2vhd.vhdx.bz2 b/tests/qemu-iotests/sample_images/test-disk2vhd.vhdx.bz2
new file mode 100644
index 0000000000000000000000000000000000000000..2891c9a6d62908e0195d3791817284016249aaee
GIT binary patch
literal 1424
zcmZ>Y%CIzaj8qGbG}k{O$Xb_J|NsC0KmUCH-JDzhyZ-zC_y7OrRo{&NR|gc{RqOEm
z=imPqn0Gw*bmiUDGLEd7O!Jz9a-FYyG%@kz4`91;!B*gMCgZ|o{hrI3-5eMf7&2U(
ze2MLfRMjk#3m2F92b;9b3)~wx*_X}#(!AiWP7J&Z7Z*rLWn7%V=F8TQk#Sieg@GYO
z%H#s$1cuB7Co(4-Ibm{f0b4Ia3Io%!MYdbrr~3cDvda022}1^3FSCOb0|sEu@-&&Z
zd{yQz7v?J$7%p6>a$t6L0IPy1Hi78)%CahBQl<Z;+5VngDYs_v^vp6jaD~m;eM-i~
z&Sh$|j1q($7cl#p@Y?z`Wbn)~;$8WKbK(Lv_Z21=7;XxsEqh$#HSI@WE6^;a!cbHI
zu?%7hkTh5`=j@LcDLKD{B^^#a3zjsRcFl8cicgpF_s*b{y^=;gXHGaUxHsk6Z1z!6
zR8jOb5@75|V437mU-yJTMS)R+<!|9h(XdP|@fL<tERWTG&+}1W<Y45=R(WVS*~r-_
zC)tvLS9QX(l#{Jfd<|w?_&irwSR~K$n-GH&gVWNbjzLvv#!ZvhEgL4x<WyO{%dgF0
zhm(_slhXnI%U%l|_%|#KQ59iedZ^gAy6MTT^pA1#JRe_E5V)X}!2L#gS<#~h?Gtqv
zl&`O9bYKze@DSi|;D~C!8#&cpD`XX`!J3mz{!xVw{r4+rTyzx)bS+3;^k;kfgqh(R
zjB+v<0`Hud8Jn$R?UuFBl*37&r7&VvuZxF5ll`XNKxZk5#5RfKr9KH<!3>rSX^b=2
zQf8bIGb{Zh_kHjC@8`d&KfM}#^|g=#lbQEvqpGz5Ed?!#9L1S`SKYn`R8T$Vx#hW1
zCM6{$l_f4NE*^KTTb?Ukoc4OXmY1i>B+sy}&s%2&>@2zH!^iM2gHz2Te8X9{cMVoM
z;{=)JSmb5gwe(qeljqKyZ`#t=IU7>$_6lS~Td$G%SNN0X?aCy3waorU$G7pHk*Ma-
zYE<;TxbKbjzRG&1nl|xWH#-<gzZXBa@XdC5+2yW=XNLk^I~R&OZFSL<+bNOsMeTpb
z%%mvCGRqI%Rym??0~1n(1@CMZGrbn8sA)Z0k$X~Q0^5#teF5#=oBpZXdBt9_ShVJu
z8-w~y8S_&dArUE&X2}KGr!EJkWlT{K;ttAXNYjmGtKHP1b&A_jYaOdaPOr<6nbTE{
zuGQZxQ2p%4>&9b+6?O;aW$X+xPMFR3&d_YRW#T@i-OCo2#W_wCnEbBIs?xo=)HQ$R
z>o>}_dZp{atdD$}H#<@;Max;n<kaU)7XF+%XO&;=U*>$qQfE(*sioE9rtPc(Y*#s7
zpKUfvXG`_DIWbL<qv^M0nZ<#RLK5XFQaOwErZ9+!_<mihd;E~K^vhov_a?RNTBv-^
zx$nd!MiKQrW>eatZPZ^(zWe`GGwaHqI+yFZw>+A3{QKrRzAG9SB77d|ev|%_V)0-`
z-JZ?tMvvIkH2+6tf9dm9+PF?_erw&^`i@ni63vV(0;eV%=E*Bq#jok#6Tz6rbtWUl
zLX<`8`FCKbFRwON)jb?9dDZx@toqN7-~A3R)HZBdvAAbTO@<1S0Fy|{so18Xs4bHB
zS53EGJoP`Mc;n-Ibe}Q4&w=3p0|SQwgF%C@o6`c-m0S(e=_6mW?bJO+Z@mKy2Ux)N
dpW$4UIbn)dbcj%=_t85O{X|r_v(Ec8000NwOCSIM

literal 0
HcmV?d00001

-- 
1.8.3.1

[Qemu-devel] [PULL 05/22] qdict: Add qdict_join()

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

This function joins two QDicts by absorbing one into the other.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Benoit Canet <benoit@irqsave.net>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 include/qapi/qmp/qdict.h |  3 +++
 qobject/qdict.c          | 32 ++++++++++++++++++++++++++++++++
 2 files changed, 35 insertions(+)

diff --git a/include/qapi/qmp/qdict.h b/include/qapi/qmp/qdict.h
index 1ddf97b..d68f4eb 100644
--- a/include/qapi/qmp/qdict.h
+++ b/include/qapi/qmp/qdict.h
@@ -16,6 +16,7 @@
 #include "qapi/qmp/qobject.h"
 #include "qapi/qmp/qlist.h"
 #include "qemu/queue.h"
+#include <stdbool.h>
 #include <stdint.h>
 
 #define QDICT_BUCKET_MAX 512
@@ -70,4 +71,6 @@ void qdict_flatten(QDict *qdict);
 void qdict_extract_subqdict(QDict *src, QDict **dst, const char *start);
 void qdict_array_split(QDict *src, QList **dst);
 
+void qdict_join(QDict *dest, QDict *src, bool overwrite);
+
 #endif /* QDICT_H */
diff --git a/qobject/qdict.c b/qobject/qdict.c
index 42ec4c0..ea239f0 100644
--- a/qobject/qdict.c
+++ b/qobject/qdict.c
@@ -665,3 +665,35 @@ void qdict_array_split(QDict *src, QList **dst)
         qlist_append_obj(*dst, subqobj ?: QOBJECT(subqdict));
     }
 }
+
+/**
+ * qdict_join(): Absorb the src QDict into the dest QDict, that is, move all
+ * elements from src to dest.
+ *
+ * If an element from src has a key already present in dest, it will not be
+ * moved unless overwrite is true.
+ *
+ * If overwrite is true, the conflicting values in dest will be discarded and
+ * replaced by the corresponding values from src.
+ *
+ * Therefore, with overwrite being true, the src QDict will always be empty when
+ * this function returns. If overwrite is false, the src QDict will be empty
+ * iff there were no conflicts.
+ */
+void qdict_join(QDict *dest, QDict *src, bool overwrite)
+{
+    const QDictEntry *entry, *next;
+
+    entry = qdict_first(src);
+    while (entry) {
+        next = qdict_next(src, entry);
+
+        if (overwrite || !qdict_haskey(dest, entry->key)) {
+            qobject_incref(entry->value);
+            qdict_put_obj(dest, entry->key, entry->value);
+            qdict_del(src, entry->key);
+        }
+
+        entry = next;
+    }
+}
-- 
1.8.3.1

[Qemu-devel] [PULL 06/22] check-qdict: Add test for qdict_join()

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

Add some test cases for qdict_join().

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Benoit Canet <benoit@irqsave.net>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 tests/check-qdict.c | 87 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 87 insertions(+)

diff --git a/tests/check-qdict.c b/tests/check-qdict.c
index 2ad0f78..a9296f0 100644
--- a/tests/check-qdict.c
+++ b/tests/check-qdict.c
@@ -444,6 +444,92 @@ static void qdict_array_split_test(void)
     QDECREF(test_dict);
 }
 
+static void qdict_join_test(void)
+{
+    QDict *dict1, *dict2;
+    bool overwrite = false;
+    int i;
+
+    dict1 = qdict_new();
+    dict2 = qdict_new();
+
+
+    /* Test everything once without overwrite and once with */
+    do
+    {
+        /* Test empty dicts */
+        qdict_join(dict1, dict2, overwrite);
+
+        g_assert(qdict_size(dict1) == 0);
+        g_assert(qdict_size(dict2) == 0);
+
+
+        /* First iteration: Test movement */
+        /* Second iteration: Test empty source and non-empty destination */
+        qdict_put(dict2, "foo", qint_from_int(42));
+
+        for (i = 0; i < 2; i++) {
+            qdict_join(dict1, dict2, overwrite);
+
+            g_assert(qdict_size(dict1) == 1);
+            g_assert(qdict_size(dict2) == 0);
+
+            g_assert(qdict_get_int(dict1, "foo") == 42);
+        }
+
+
+        /* Test non-empty source and destination without conflict */
+        qdict_put(dict2, "bar", qint_from_int(23));
+
+        qdict_join(dict1, dict2, overwrite);
+
+        g_assert(qdict_size(dict1) == 2);
+        g_assert(qdict_size(dict2) == 0);
+
+        g_assert(qdict_get_int(dict1, "foo") == 42);
+        g_assert(qdict_get_int(dict1, "bar") == 23);
+
+
+        /* Test conflict */
+        qdict_put(dict2, "foo", qint_from_int(84));
+
+        qdict_join(dict1, dict2, overwrite);
+
+        g_assert(qdict_size(dict1) == 2);
+        g_assert(qdict_size(dict2) == !overwrite);
+
+        g_assert(qdict_get_int(dict1, "foo") == overwrite ? 84 : 42);
+        g_assert(qdict_get_int(dict1, "bar") == 23);
+
+        if (!overwrite) {
+            g_assert(qdict_get_int(dict2, "foo") == 84);
+        }
+
+
+        /* Check the references */
+        g_assert(qdict_get(dict1, "foo")->refcnt == 1);
+        g_assert(qdict_get(dict1, "bar")->refcnt == 1);
+
+        if (!overwrite) {
+            g_assert(qdict_get(dict2, "foo")->refcnt == 1);
+        }
+
+
+        /* Clean up */
+        qdict_del(dict1, "foo");
+        qdict_del(dict1, "bar");
+
+        if (!overwrite) {
+            qdict_del(dict2, "foo");
+        }
+    }
+    while (overwrite ^= true);
+
+
+    QDECREF(dict1);
+    QDECREF(dict2);
+}
+
 /*
  * Errors test-cases
  */
@@ -584,6 +670,7 @@ int main(int argc, char **argv)
     g_test_add_func("/public/iterapi", qdict_iterapi_test);
     g_test_add_func("/public/flatten", qdict_flatten_test);
     g_test_add_func("/public/array_split", qdict_array_split_test);
+    g_test_add_func("/public/join", qdict_join_test);
 
     g_test_add_func("/errors/put_exists", qdict_put_exists_test);
     g_test_add_func("/errors/get_not_exists", qdict_get_not_exists_test);
-- 
1.8.3.1

[Qemu-devel] [PULL 07/22] block: Allow JSON filenames

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

If the filename given to bdrv_open() is prefixed with "json:", parse the
rest as a JSON object and merge the result into the options QDict. If
there are conflicts, the options QDict takes precedence.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 block.c | 41 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)

diff --git a/block.c b/block.c
index 65e8191..415ae3d 100644
--- a/block.c
+++ b/block.c
@@ -1274,6 +1274,33 @@ out:
     g_free(tmp_filename);
 }
 
+static QDict *parse_json_filename(const char *filename, Error **errp)
+{
+    QObject *options_obj;
+    QDict *options;
+    int ret;
+
+    ret = strstart(filename, "json:", &filename);
+    assert(ret);
+
+    options_obj = qobject_from_json(filename);
+    if (!options_obj) {
+        error_setg(errp, "Could not parse the JSON options");
+        return NULL;
+    }
+
+    if (qobject_type(options_obj) != QTYPE_QDICT) {
+        qobject_decref(options_obj);
+        error_setg(errp, "Invalid JSON object given");
+        return NULL;
+    }
+
+    options = qobject_to_qdict(options_obj);
+    qdict_flatten(options);
+
+    return options;
+}
+
 /*
  * Opens a disk image (raw, qcow2, vmdk, ...)
  *
@@ -1337,6 +1364,20 @@ int bdrv_open(BlockDriverState **pbs, const char *filename,
         options = qdict_new();
     }
 
+    if (filename && g_str_has_prefix(filename, "json:")) {
+        QDict *json_options = parse_json_filename(filename, &local_err);
+        if (local_err) {
+            ret = -EINVAL;
+            goto fail;
+        }
+
+        /* Options given in the filename have lower priority than options
+         * specified directly */
+        qdict_join(options, json_options, false);
+        QDECREF(json_options);
+        filename = NULL;
+    }
+
     bs->options = options;
     options = qdict_clone_shallow(options);
 
-- 
1.8.3.1

[Qemu-devel] [PULL 08/22] iotests: Add test for the JSON protocol

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

Add a test for the JSON protocol driver.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 tests/qemu-iotests/089     | 130 +++++++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/089.out |  54 +++++++++++++++++++
 tests/qemu-iotests/group   |   1 +
 3 files changed, 185 insertions(+)
 create mode 100755 tests/qemu-iotests/089
 create mode 100644 tests/qemu-iotests/089.out

diff --git a/tests/qemu-iotests/089 b/tests/qemu-iotests/089
new file mode 100755
index 0000000..ef47601
--- /dev/null
+++ b/tests/qemu-iotests/089
@@ -0,0 +1,130 @@
+#!/bin/bash
+#
+# Test case for support of JSON filenames
+#
+# Copyright (C) 2014 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
+owner=mreitz@redhat.com
+
+seq="$(basename $0)"
+echo "QA output created by $seq"
+
+here="$PWD"
+tmp=/tmp/$$
+status=1	# failure is the default!
+
+_cleanup()
+{
+	_cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+_supported_fmt qcow2
+_supported_proto file
+_supported_os Linux
+
+# Using an image filename containing quotation marks will render the JSON data
+# below invalid. In that case, we have little choice but simply not to run this
+# test.
+case $TEST_IMG in
+    *'"'*)
+        _notrun "image filename may not contain quotation marks"
+        ;;
+esac
+
+IMG_SIZE=64M
+
+# Taken from test 072
+echo
+echo "=== Testing nested image formats ==="
+echo
+
+TEST_IMG="$TEST_IMG.base" _make_test_img $IMG_SIZE
+
+$QEMU_IO -c 'write -P 42 0 512' -c 'write -P 23 512 512' \
+         -c 'write -P 66 1024 512' "$TEST_IMG.base" | _filter_qemu_io
+
+$QEMU_IMG convert -f raw -O $IMGFMT "$TEST_IMG.base" "$TEST_IMG"
+
+$QEMU_IO -c 'read -P 42 0 512' -c 'read -P 23 512 512' \
+         -c 'read -P 66 1024 512' "json:{
+    \"driver\": \"$IMGFMT\",
+    \"file\": {
+        \"driver\": \"$IMGFMT\",
+        \"file\": {
+            \"filename\": \"$TEST_IMG\"
+        }
+    }
+}" | _filter_qemu_io
+
+# This should fail (see test 072)
+$QEMU_IO -c 'read -P 42 0 512' "$TEST_IMG" | _filter_qemu_io
+
+
+# Taken from test 071
+echo
+echo "=== Testing blkdebug ==="
+echo
+
+_make_test_img $IMG_SIZE
+
+$QEMU_IO -c 'write -P 42 0x38000 512' "$TEST_IMG" | _filter_qemu_io
+
+# The "image.filename" part tests whether "a": { "b": "c" } and "a.b": "c" do
+# the same (which they should).
+$QEMU_IO -c 'read -P 42 0x38000 512' "json:{
+    \"driver\": \"$IMGFMT\",
+    \"file\": {
+        \"driver\": \"blkdebug\",
+        \"inject-error\": [{
+            \"event\": \"l2_load\"
+        }],
+        \"image.filename\": \"$TEST_IMG\"
+    }
+}" | _filter_qemu_io
+
+
+echo
+echo "=== Testing qemu-img info output ==="
+echo
+
+$QEMU_IMG info "json:{\"driver\":\"qcow2\",\"file.filename\":\"$TEST_IMG\"}" \
+    | _filter_testdir | _filter_imgfmt
+
+
+echo
+echo "=== Testing option merging ==="
+echo
+
+# Both options given directly and those given in the filename should be used
+$QEMU_IO -c "open -o driver=qcow2 json:{\"file.filename\":\"$TEST_IMG\"}" \
+         -c "info" 2>&1 | _filter_testdir | _filter_imgfmt
+
+# Options given directly should be prioritized over those given in the filename
+$QEMU_IO -c "open -o driver=qcow2 json:{\"driver\":\"raw\",\"file.filename\":\"$TEST_IMG\"}" \
+         -c "info" 2>&1 | _filter_testdir | _filter_imgfmt
+
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
diff --git a/tests/qemu-iotests/089.out b/tests/qemu-iotests/089.out
new file mode 100644
index 0000000..d6cf783
--- /dev/null
+++ b/tests/qemu-iotests/089.out
@@ -0,0 +1,54 @@
+QA output created by 089
+
+=== Testing nested image formats ===
+
+Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=67108864 
+wrote 512/512 bytes at offset 0
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset 512
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset 1024
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 512/512 bytes at offset 0
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 512/512 bytes at offset 512
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 512/512 bytes at offset 1024
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+Pattern verification failed at offset 0, 512 bytes
+read 512/512 bytes at offset 0
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+=== Testing blkdebug ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 
+wrote 512/512 bytes at offset 229376
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read failed: Input/output error
+
+=== Testing qemu-img info output ===
+
+image: TEST_DIR/t.IMGFMT
+file format: IMGFMT
+virtual size: 64M (67108864 bytes)
+disk size: 324K
+cluster_size: 65536
+Format specific information:
+    compat: 1.1
+    lazy refcounts: false
+
+=== Testing option merging ===
+
+format name: IMGFMT
+cluster size: 64 KiB
+vm state offset: 512 MiB
+Format specific information:
+    compat: 1.1
+    lazy refcounts: false
+format name: IMGFMT
+cluster size: 64 KiB
+vm state offset: 512 MiB
+Format specific information:
+    compat: 1.1
+    lazy refcounts: false
+*** done
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index cd3e4d2..2988cfd 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -95,5 +95,6 @@
 086 rw auto quick
 087 rw auto
 088 rw auto
+089 rw auto quick
 090 rw auto quick
 091 rw auto
-- 
1.8.3.1

[Qemu-devel] [PULL 09/22] qemu-iotests: Fix core dump suppression in test 039

[Kevin Wolf]

From: Markus Armbruster <armbru@redhat.com>

The shell script attempts to suppress core dumps like this:

    old_ulimit=$(ulimit -c)
    ulimit -c 0
    $QEMU_IO arg...
    ulimit -c "$old_ulimit"

This breaks the test hard unless the limit was zero to begin with!
ulimit sets both hard and soft limit by default, and (re-)raising the
hard limit requires privileges.  Broken since it was added in commit
dc68afe.

Could be fixed by adding -S to set only the soft limit, but I'm not
sure how portable that is in practice.  Simply do it in a subshell
instead, like this:

    (ulimit -c 0; exec $QEMU_IO arg...)

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 tests/qemu-iotests/039           | 20 ++++++++------------
 tests/qemu-iotests/039.out       |  3 +++
 tests/qemu-iotests/common.filter |  1 +
 3 files changed, 12 insertions(+), 12 deletions(-)

diff --git a/tests/qemu-iotests/039 b/tests/qemu-iotests/039
index b9cbe99..27fe4bd 100755
--- a/tests/qemu-iotests/039
+++ b/tests/qemu-iotests/039
@@ -47,6 +47,11 @@ _supported_os Linux
 _default_cache_mode "writethrough"
 _supported_cache_modes "writethrough"
 
+_no_dump_exec()
+{
+    (ulimit -c 0; exec "$@")
+}
+
 size=128M
 
 echo
@@ -67,10 +72,7 @@ echo "== Creating a dirty image file =="
 IMGOPTS="compat=1.1,lazy_refcounts=on"
 _make_test_img $size
 
-old_ulimit=$(ulimit -c)
-ulimit -c 0 # do not produce a core dump on abort(3)
-$QEMU_IO -c "write -P 0x5a 0 512" -c "abort" "$TEST_IMG" | _filter_qemu_io
-ulimit -c "$old_ulimit"
+_no_dump_exec $QEMU_IO -c "write -P 0x5a 0 512" -c "abort" "$TEST_IMG" 2>&1 | _filter_qemu_io
 
 # The dirty bit must be set
 ./qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
@@ -103,10 +105,7 @@ echo "== Opening a dirty image read/write should repair it =="
 IMGOPTS="compat=1.1,lazy_refcounts=on"
 _make_test_img $size
 
-old_ulimit=$(ulimit -c)
-ulimit -c 0 # do not produce a core dump on abort(3)
-$QEMU_IO -c "write -P 0x5a 0 512" -c "abort" "$TEST_IMG" | _filter_qemu_io
-ulimit -c "$old_ulimit"
+_no_dump_exec $QEMU_IO -c "write -P 0x5a 0 512" -c "abort" "$TEST_IMG" 2>&1 | _filter_qemu_io
 
 # The dirty bit must be set
 ./qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
@@ -122,10 +121,7 @@ echo "== Creating an image file with lazy_refcounts=off =="
 IMGOPTS="compat=1.1,lazy_refcounts=off"
 _make_test_img $size
 
-old_ulimit=$(ulimit -c)
-ulimit -c 0 # do not produce a core dump on abort(3)
-$QEMU_IO -c "write -P 0x5a 0 512" -c "abort" "$TEST_IMG" | _filter_qemu_io
-ulimit -c "$old_ulimit"
+_no_dump_exec $QEMU_IO -c "write -P 0x5a 0 512" -c "abort" "$TEST_IMG" 2>&1 | _filter_qemu_io
 
 # The dirty bit must not be set since lazy_refcounts=off
 ./qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
diff --git a/tests/qemu-iotests/039.out b/tests/qemu-iotests/039.out
index fb31ae0..67e7744 100644
--- a/tests/qemu-iotests/039.out
+++ b/tests/qemu-iotests/039.out
@@ -11,6 +11,7 @@ No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 
 wrote 512/512 bytes at offset 0
 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+./039: Aborted                 ( ulimit -c 0; exec "$@" )
 incompatible_features     0x1
 ERROR cluster 5 refcount=0 reference=1
 ERROR OFLAG_COPIED data cluster: l2_entry=8000000000050000 refcount=0
@@ -42,6 +43,7 @@ read 512/512 bytes at offset 0
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 
 wrote 512/512 bytes at offset 0
 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+./039: Aborted                 ( ulimit -c 0; exec "$@" )
 incompatible_features     0x1
 Repairing cluster 5 refcount=0 reference=1
 wrote 512/512 bytes at offset 0
@@ -52,6 +54,7 @@ incompatible_features     0x0
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 
 wrote 512/512 bytes at offset 0
 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+./039: Aborted                 ( ulimit -c 0; exec "$@" )
 incompatible_features     0x0
 No errors were found on the image.
 
diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.filter
index 776985d..a04df7f 100644
--- a/tests/qemu-iotests/common.filter
+++ b/tests/qemu-iotests/common.filter
@@ -150,6 +150,7 @@ _filter_win32()
 _filter_qemu_io()
 {
     _filter_win32 | sed -e "s/[0-9]* ops\; [0-9/:. sec]* ([0-9/.inf]* [EPTGMKiBbytes]*\/sec and [0-9/.inf]* ops\/sec)/X ops\; XX:XX:XX.X (XXX YYY\/sec and XXX ops\/sec)/" \
+        -e "s/: line [0-9][0-9]*:  *[0-9][0-9]*\( Aborted\)/:\1/" \
         -e "s/qemu-io> //g"
 }
 
-- 
1.8.3.1

[Qemu-devel] [PULL 10/22] qemu-iotests: Fix blkdebug in VM drive in 030

[Kevin Wolf]

From: Fam Zheng <famz@redhat.com>

The test test_stream_pause in this class uses vm.pause_drive, which
requires a blkdebug driver on top of image, otherwise it's no-op and the
test running is undeterministic.

So add it.

Signed-off-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 tests/qemu-iotests/030 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/030 b/tests/qemu-iotests/030
index 8cb61fd..8ce2373 100755
--- a/tests/qemu-iotests/030
+++ b/tests/qemu-iotests/030
@@ -35,7 +35,7 @@ class TestSingleDrive(iotests.QMPTestCase):
         qemu_img('create', '-f', iotests.imgfmt, '-o', 'backing_file=%s' % backing_img, mid_img)
         qemu_img('create', '-f', iotests.imgfmt, '-o', 'backing_file=%s' % mid_img, test_img)
         qemu_io('-c', 'write -P 0x1 0 512', backing_img)
-        self.vm = iotests.VM().add_drive(test_img)
+        self.vm = iotests.VM().add_drive("blkdebug::" + test_img)
         self.vm.launch()
 
     def tearDown(self):
-- 
1.8.3.1

[Qemu-devel] [PULL 11/22] curl: Fix build when curl_multi_socket_action isn't available

[Kevin Wolf]

From: Matthew Booth <mbooth@redhat.com>

Signed-off-by: Matthew Booth <mbooth@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 block/curl.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/block/curl.c b/block/curl.c
index d2f1084..f3c797a 100644
--- a/block/curl.c
+++ b/block/curl.c
@@ -37,6 +37,21 @@
 #if LIBCURL_VERSION_NUM >= 0x071000
 /* The multi interface timer callback was introduced in 7.16.0 */
 #define NEED_CURL_TIMER_CALLBACK
+#define HAVE_SOCKET_ACTION
+#endif
+
+#ifndef HAVE_SOCKET_ACTION
+/* If curl_multi_socket_action isn't available, define it statically here in
+ * terms of curl_multi_socket. Note that ev_bitmask will be ignored, which is
+ * less efficient but still safe. */
+static CURLMcode __curl_multi_socket_action(CURLM *multi_handle,
+                                            curl_socket_t sockfd,
+                                            int ev_bitmask,
+                                            int *running_handles)
+{
+    return curl_multi_socket(multi_handle, sockfd, running_handles);
+}
+#define curl_multi_socket_action __curl_multi_socket_action
 #endif
 
 #define PROTOCOLS (CURLPROTO_HTTP | CURLPROTO_HTTPS | \
-- 
1.8.3.1

[Qemu-devel] [PULL 12/22] curl: Remove broken parsing of options from url

[Kevin Wolf]

From: Matthew Booth <mbooth@redhat.com>

The block layer now supports a generic json syntax for passing option parameters
explicitly, making parsing of options from the url redundant.

Signed-off-by: Matthew Booth <mbooth@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 block/curl.c | 52 ++++++++++------------------------------------------
 1 file changed, 10 insertions(+), 42 deletions(-)

diff --git a/block/curl.c b/block/curl.c
index f3c797a..1b9f2f2 100644
--- a/block/curl.c
+++ b/block/curl.c
@@ -61,12 +61,15 @@ static CURLMcode __curl_multi_socket_action(CURLM *multi_handle,
 #define CURL_NUM_STATES 8
 #define CURL_NUM_ACB    8
 #define SECTOR_SIZE     512
-#define READ_AHEAD_SIZE (256 * 1024)
+#define READ_AHEAD_DEFAULT (256 * 1024)
 
 #define FIND_RET_NONE   0
 #define FIND_RET_OK     1
 #define FIND_RET_WAIT   2
 
+#define CURL_BLOCK_OPT_URL       "url"
+#define CURL_BLOCK_OPT_READAHEAD "readahead"
+
 struct BDRVCURLState;
 
 typedef struct CURLAIOCB {
@@ -411,43 +414,7 @@ static void curl_clean_state(CURLState *s)
 static void curl_parse_filename(const char *filename, QDict *options,
                                 Error **errp)
 {
-
-    #define RA_OPTSTR ":readahead="
-    char *file;
-    char *ra;
-    const char *ra_val;
-    int parse_state = 0;
-
-    file = g_strdup(filename);
-
-    /* Parse a trailing ":readahead=#:" param, if present. */
-    ra = file + strlen(file) - 1;
-    while (ra >= file) {
-        if (parse_state == 0) {
-            if (*ra == ':') {
-                parse_state++;
-            } else {
-                break;
-            }
-        } else if (parse_state == 1) {
-            if (*ra > '9' || *ra < '0') {
-                char *opt_start = ra - strlen(RA_OPTSTR) + 1;
-                if (opt_start > file &&
-                    strncmp(opt_start, RA_OPTSTR, strlen(RA_OPTSTR)) == 0) {
-                    ra_val = ra + 1;
-                    ra -= strlen(RA_OPTSTR) - 1;
-                    *ra = '\0';
-                    qdict_put(options, "readahead", qstring_from_str(ra_val));
-                }
-                break;
-            }
-        }
-        ra--;
-    }
-
-    qdict_put(options, "url", qstring_from_str(file));
-
-    g_free(file);
+    qdict_put(options, CURL_BLOCK_OPT_URL, qstring_from_str(filename));
 }
 
 static QemuOptsList runtime_opts = {
@@ -455,12 +422,12 @@ static QemuOptsList runtime_opts = {
     .head = QTAILQ_HEAD_INITIALIZER(runtime_opts.head),
     .desc = {
         {
-            .name = "url",
+            .name = CURL_BLOCK_OPT_URL,
             .type = QEMU_OPT_STRING,
             .help = "URL to open",
         },
         {
-            .name = "readahead",
+            .name = CURL_BLOCK_OPT_READAHEAD,
             .type = QEMU_OPT_SIZE,
             .help = "Readahead size",
         },
@@ -492,14 +459,15 @@ static int curl_open(BlockDriverState *bs, QDict *options, int flags,
         goto out_noclean;
     }
 
-    s->readahead_size = qemu_opt_get_size(opts, "readahead", READ_AHEAD_SIZE);
+    s->readahead_size = qemu_opt_get_size(opts, CURL_BLOCK_OPT_READAHEAD,
+                                          READ_AHEAD_DEFAULT);
     if ((s->readahead_size & 0x1ff) != 0) {
         error_setg(errp, "HTTP_READAHEAD_SIZE %zd is not a multiple of 512",
                    s->readahead_size);
         goto out_noclean;
     }
 
-    file = qemu_opt_get(opts, "url");
+    file = qemu_opt_get(opts, CURL_BLOCK_OPT_URL);
     if (file == NULL) {
         error_setg(errp, "curl block driver requires an 'url' option");
         goto out_noclean;
-- 
1.8.3.1

[Qemu-devel] [PULL 13/22] curl: Add sslverify option

[Kevin Wolf]

From: Matthew Booth <mbooth@redhat.com>

This allows qemu to use images over https with a self-signed certificate. It
defaults to verifying the certificate.

Signed-off-by: Matthew Booth <mbooth@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 block/curl.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/block/curl.c b/block/curl.c
index 1b9f2f2..f491b0b 100644
--- a/block/curl.c
+++ b/block/curl.c
@@ -23,6 +23,7 @@
  */
 #include "qemu-common.h"
 #include "block/block_int.h"
+#include "qapi/qmp/qbool.h"
 #include <curl/curl.h>
 
 // #define DEBUG
@@ -69,6 +70,7 @@ static CURLMcode __curl_multi_socket_action(CURLM *multi_handle,
 
 #define CURL_BLOCK_OPT_URL       "url"
 #define CURL_BLOCK_OPT_READAHEAD "readahead"
+#define CURL_BLOCK_OPT_SSLVERIFY "sslverify"
 
 struct BDRVCURLState;
 
@@ -106,6 +108,7 @@ typedef struct BDRVCURLState {
     CURLState states[CURL_NUM_STATES];
     char *url;
     size_t readahead_size;
+    bool sslverify;
     bool accept_range;
 } BDRVCURLState;
 
@@ -372,6 +375,8 @@ static CURLState *curl_init_state(BDRVCURLState *s)
             return NULL;
         }
         curl_easy_setopt(state->curl, CURLOPT_URL, s->url);
+        curl_easy_setopt(state->curl, CURLOPT_SSL_VERIFYPEER,
+                         (long) s->sslverify);
         curl_easy_setopt(state->curl, CURLOPT_TIMEOUT, 5);
         curl_easy_setopt(state->curl, CURLOPT_WRITEFUNCTION,
                          (void *)curl_read_cb);
@@ -431,6 +436,11 @@ static QemuOptsList runtime_opts = {
             .type = QEMU_OPT_SIZE,
             .help = "Readahead size",
         },
+        {
+            .name = CURL_BLOCK_OPT_SSLVERIFY,
+            .type = QEMU_OPT_BOOL,
+            .help = "Verify SSL certificate"
+        },
         { /* end of list */ }
     },
 };
@@ -467,6 +477,8 @@ static int curl_open(BlockDriverState *bs, QDict *options, int flags,
         goto out_noclean;
     }
 
+    s->sslverify = qemu_opt_get_bool(opts, CURL_BLOCK_OPT_SSLVERIFY, true);
+
     file = qemu_opt_get(opts, CURL_BLOCK_OPT_URL);
     if (file == NULL) {
         error_setg(errp, "curl block driver requires an 'url' option");
-- 
1.8.3.1

[Qemu-devel] [PULL 14/22] curl: Add usage documentation

[Kevin Wolf]

From: Matthew Booth <mbooth@redhat.com>

Signed-off-by: Matthew Booth <mbooth@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 qemu-options.hx | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 68 insertions(+)

diff --git a/qemu-options.hx b/qemu-options.hx
index 781af14..7587bce 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -2191,6 +2191,74 @@ qemu-system-x86_64 --drive file=gluster://192.0.2.1/testvol/a.img
 @end example
 
 See also @url{http://www.gluster.org}.
+
+@item HTTP/HTTPS/FTP/FTPS/TFTP
+QEMU supports read-only access to files accessed over http(s), ftp(s) and tftp.
+
+Syntax using a single filename:
+@example
+<protocol>://[<username>[:<password>]@@]<host>/<path>
+@end example
+
+where:
+@table @option
+@item protocol
+'http', 'https', 'ftp', 'ftps', or 'tftp'.
+
+@item username
+Optional username for authentication to the remote server.
+
+@item password
+Optional password for authentication to the remote server.
+
+@item host
+Address of the remote server.
+
+@item path
+Path on the remote server, including any query string.
+@end table
+
+The following options are also supported:
+@table @option
+@item url
+The full URL when passing options to the driver explicitly.
+
+@item readahead
+The amount of data to read ahead with each range request to the remote server.
+This value may optionally have the suffix 'T', 'G', 'M', 'K', 'k' or 'b'. If it
+does not have a suffix, it will be assumed to be in bytes. The value must be a
+multiple of 512 bytes. It defaults to 256k.
+
+@item sslverify
+Whether to verify the remote server's certificate when connecting over SSL. It
+can have the value 'on' or 'off'. It defaults to 'on'.
+@end table
+
+Note that when passing options to qemu explicitly, @option{driver} is the value
+of <protocol>.
+
+Example: boot from a remote Fedora 20 live ISO image
+@example
+qemu-system-x86_64 --drive media=cdrom,file=http://dl.fedoraproject.org/pub/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
+
+qemu-system-x86_64 --drive media=cdrom,file.driver=http,file.url=http://dl.fedoraproject.org/pub/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
+@end example
+
+Example: boot from a remote Fedora 20 cloud image using a local overlay for
+writes, copy-on-read, and a readahead of 64k
+@example
+qemu-img create -f qcow2 -o backing_file='json:@{"file.driver":"http",, "file.url":"https://dl.fedoraproject.org/pub/fedora/linux/releases/20/Images/x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2",, "file.readahead":"64k"@}' /tmp/Fedora-x86_64-20-20131211.1-sda.qcow2
+
+qemu-system-x86_64 -drive file=/tmp/Fedora-x86_64-20-20131211.1-sda.qcow2,copy-on-read=on
+@end example
+
+Example: boot from an image stored on a VMware vSphere server with a self-signed
+certificate using a local overlay for writes and a readahead of 64k
+@example
+qemu-img create -f qcow2 -o backing_file='json:@{"file.driver":"https",, "file.url":"https://user:password@@vsphere.example.com/folder/test/test-flat.vmdk?dcPath=Datacenter&dsName=datastore1",, "file.sslverify":"off",, "file.readahead":"64k"@}' /tmp/test.qcow2
+
+qemu-system-x86_64 -drive file=/tmp/test.qcow2
+@end example
 ETEXI
 
 STEXI
-- 
1.8.3.1

[Qemu-devel] [PULL 15/22] qcow1: Make padding in the header explicit

[Kevin Wolf]

We were relying on all compilers inserting the same padding in the
header struct that is used for the on-disk format. Let's not do that.
Mark the struct as packed and insert an explicit padding field for
compatibility.

Cc: qemu-stable@nongnu.org
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Benoit Canet <benoit@irqsave.net>
---
 block/qcow.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/block/qcow.c b/block/qcow.c
index 937dd6d..3684794 100644
--- a/block/qcow.c
+++ b/block/qcow.c
@@ -48,9 +48,10 @@ typedef struct QCowHeader {
     uint64_t size; /* in bytes */
     uint8_t cluster_bits;
     uint8_t l2_bits;
+    uint16_t padding;
     uint32_t crypt_method;
     uint64_t l1_table_offset;
-} QCowHeader;
+} QEMU_PACKED QCowHeader;
 
 #define L2_CACHE_SIZE 16
 
-- 
1.8.3.1

[Qemu-devel] [PULL 16/22] qcow1: Check maximum cluster size

[Kevin Wolf]

Huge values for header.cluster_bits cause unbounded allocations (e.g.
for s->cluster_cache) and crash qemu this way. Less huge values may
survive those allocations, but can cause integer overflows later on.

The only cluster sizes that qemu can create are 4k (for standalone
images) and 512 (for images with backing files), so we can limit it
to 64k.

Cc: qemu-stable@nongnu.org
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Benoit Canet <benoit@irqsave.net>
---
 block/qcow.c               | 10 ++++++--
 tests/qemu-iotests/092     | 63 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/092.out | 13 ++++++++++
 tests/qemu-iotests/group   |  1 +
 4 files changed, 85 insertions(+), 2 deletions(-)
 create mode 100755 tests/qemu-iotests/092
 create mode 100644 tests/qemu-iotests/092.out

diff --git a/block/qcow.c b/block/qcow.c
index 3684794..e60df23 100644
--- a/block/qcow.c
+++ b/block/qcow.c
@@ -128,11 +128,17 @@ static int qcow_open(BlockDriverState *bs, QDict *options, int flags,
         goto fail;
     }
 
-    if (header.size <= 1 || header.cluster_bits < 9) {
-        error_setg(errp, "invalid value in qcow header");
+    if (header.size <= 1) {
+        error_setg(errp, "Image size is too small (must be at least 2 bytes)");
         ret = -EINVAL;
         goto fail;
     }
+    if (header.cluster_bits < 9 || header.cluster_bits > 16) {
+        error_setg(errp, "Cluster size must be between 512 and 64k");
+        ret = -EINVAL;
+        goto fail;
+    }
+
     if (header.crypt_method > QCOW_CRYPT_AES) {
         error_setg(errp, "invalid encryption method in qcow header");
         ret = -EINVAL;
diff --git a/tests/qemu-iotests/092 b/tests/qemu-iotests/092
new file mode 100755
index 0000000..d060e6f
--- /dev/null
+++ b/tests/qemu-iotests/092
@@ -0,0 +1,63 @@
+#!/bin/bash
+#
+# qcow1 format input validation tests
+#
+# Copyright (C) 2014 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
+owner=kwolf@redhat.com
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+
+_cleanup()
+{
+    rm -f $TEST_IMG.snap
+    _cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+_supported_fmt qcow
+_supported_proto generic
+_supported_os Linux
+
+offset_cluster_bits=32
+
+echo
+echo "== Invalid cluster size =="
+_make_test_img 64M
+poke_file "$TEST_IMG" "$offset_cluster_bits" "\xff"
+{ $QEMU_IO -c "read 0 512" $TEST_IMG; } 2>&1 | _filter_qemu_io | _filter_testdir
+poke_file "$TEST_IMG" "$offset_cluster_bits" "\x1f"
+{ $QEMU_IO -c "read 0 512" $TEST_IMG; } 2>&1 | _filter_qemu_io | _filter_testdir
+poke_file "$TEST_IMG" "$offset_cluster_bits" "\x08"
+{ $QEMU_IO -c "read 0 512" $TEST_IMG; } 2>&1 | _filter_qemu_io | _filter_testdir
+poke_file "$TEST_IMG" "$offset_cluster_bits" "\x11"
+{ $QEMU_IO -c "read 0 512" $TEST_IMG; } 2>&1 | _filter_qemu_io | _filter_testdir
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
diff --git a/tests/qemu-iotests/092.out b/tests/qemu-iotests/092.out
new file mode 100644
index 0000000..8bf8158
--- /dev/null
+++ b/tests/qemu-iotests/092.out
@@ -0,0 +1,13 @@
+QA output created by 092
+
+== Invalid cluster size ==
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 
+qemu-io: can't open device TEST_DIR/t.qcow: Cluster size must be between 512 and 64k
+no file open, try 'help open'
+qemu-io: can't open device TEST_DIR/t.qcow: Cluster size must be between 512 and 64k
+no file open, try 'help open'
+qemu-io: can't open device TEST_DIR/t.qcow: Cluster size must be between 512 and 64k
+no file open, try 'help open'
+qemu-io: can't open device TEST_DIR/t.qcow: Cluster size must be between 512 and 64k
+no file open, try 'help open'
+*** done
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index 2988cfd..0f07440 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -98,3 +98,4 @@
 089 rw auto quick
 090 rw auto quick
 091 rw auto
+092 rw auto quick
-- 
1.8.3.1

[Qemu-devel] [PULL 17/22] qcow1: Validate L2 table size (CVE-2014-0222)

[Kevin Wolf]

Too large L2 table sizes cause unbounded allocations. Images actually
created by qemu-img only have 512 byte or 4k L2 tables.

To keep things consistent with cluster sizes, allow ranges between 512
bytes and 64k (in fact, down to 1 entry = 8 bytes is technically
working, but L2 table sizes smaller than a cluster don't make a lot of
sense).

This also means that the number of bytes on the virtual disk that are
described by the same L2 table is limited to at most 8k * 64k or 2^29,
preventively avoiding any integer overflows.

Cc: qemu-stable@nongnu.org
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Benoit Canet <benoit@irqsave.net>
---
 block/qcow.c               |  8 ++++++++
 tests/qemu-iotests/092     | 15 +++++++++++++++
 tests/qemu-iotests/092.out | 11 +++++++++++
 3 files changed, 34 insertions(+)

diff --git a/block/qcow.c b/block/qcow.c
index e60df23..e8038e5 100644
--- a/block/qcow.c
+++ b/block/qcow.c
@@ -139,6 +139,14 @@ static int qcow_open(BlockDriverState *bs, QDict *options, int flags,
         goto fail;
     }
 
+    /* l2_bits specifies number of entries; storing a uint64_t in each entry,
+     * so bytes = num_entries << 3. */
+    if (header.l2_bits < 9 - 3 || header.l2_bits > 16 - 3) {
+        error_setg(errp, "L2 table size must be between 512 and 64k");
+        ret = -EINVAL;
+        goto fail;
+    }
+
     if (header.crypt_method > QCOW_CRYPT_AES) {
         error_setg(errp, "invalid encryption method in qcow header");
         ret = -EINVAL;
diff --git a/tests/qemu-iotests/092 b/tests/qemu-iotests/092
index d060e6f..fb8bacc 100755
--- a/tests/qemu-iotests/092
+++ b/tests/qemu-iotests/092
@@ -44,6 +44,7 @@ _supported_proto generic
 _supported_os Linux
 
 offset_cluster_bits=32
+offset_l2_bits=33
 
 echo
 echo "== Invalid cluster size =="
@@ -57,6 +58,20 @@ poke_file "$TEST_IMG" "$offset_cluster_bits" "\x08"
 poke_file "$TEST_IMG" "$offset_cluster_bits" "\x11"
 { $QEMU_IO -c "read 0 512" $TEST_IMG; } 2>&1 | _filter_qemu_io | _filter_testdir
 
+echo
+echo "== Invalid L2 table size =="
+_make_test_img 64M
+poke_file "$TEST_IMG" "$offset_l2_bits" "\xff"
+{ $QEMU_IO -c "read 0 512" $TEST_IMG; } 2>&1 | _filter_qemu_io | _filter_testdir
+poke_file "$TEST_IMG" "$offset_l2_bits" "\x05"
+{ $QEMU_IO -c "read 0 512" $TEST_IMG; } 2>&1 | _filter_qemu_io | _filter_testdir
+poke_file "$TEST_IMG" "$offset_l2_bits" "\x0e"
+{ $QEMU_IO -c "read 0 512" $TEST_IMG; } 2>&1 | _filter_qemu_io | _filter_testdir
+
+# 1 << 0x1b = 2^31 / L2_CACHE_SIZE
+poke_file "$TEST_IMG" "$offset_l2_bits" "\x1b"
+{ $QEMU_IO -c "read 0 512" $TEST_IMG; } 2>&1 | _filter_qemu_io | _filter_testdir
+
 # success, all done
 echo "*** done"
 rm -f $seq.full
diff --git a/tests/qemu-iotests/092.out b/tests/qemu-iotests/092.out
index 8bf8158..73918b3 100644
--- a/tests/qemu-iotests/092.out
+++ b/tests/qemu-iotests/092.out
@@ -10,4 +10,15 @@ qemu-io: can't open device TEST_DIR/t.qcow: Cluster size must be between 512 and
 no file open, try 'help open'
 qemu-io: can't open device TEST_DIR/t.qcow: Cluster size must be between 512 and 64k
 no file open, try 'help open'
+
+== Invalid L2 table size ==
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 
+qemu-io: can't open device TEST_DIR/t.qcow: L2 table size must be between 512 and 64k
+no file open, try 'help open'
+qemu-io: can't open device TEST_DIR/t.qcow: L2 table size must be between 512 and 64k
+no file open, try 'help open'
+qemu-io: can't open device TEST_DIR/t.qcow: L2 table size must be between 512 and 64k
+no file open, try 'help open'
+qemu-io: can't open device TEST_DIR/t.qcow: L2 table size must be between 512 and 64k
+no file open, try 'help open'
 *** done
-- 
1.8.3.1

[Qemu-devel] [PULL 18/22] qcow1: Validate image size (CVE-2014-0223)

[Kevin Wolf]

A huge image size could cause s->l1_size to overflow. Make sure that
images never require a L1 table larger than what fits in s->l1_size.

This cannot only cause unbounded allocations, but also the allocation of
a too small L1 table, resulting in out-of-bounds array accesses (both
reads and writes).

Cc: qemu-stable@nongnu.org
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 block/qcow.c               | 16 ++++++++++++++--
 tests/qemu-iotests/092     |  9 +++++++++
 tests/qemu-iotests/092.out |  7 +++++++
 3 files changed, 30 insertions(+), 2 deletions(-)

diff --git a/block/qcow.c b/block/qcow.c
index e8038e5..3566c05 100644
--- a/block/qcow.c
+++ b/block/qcow.c
@@ -61,7 +61,7 @@ typedef struct BDRVQcowState {
     int cluster_sectors;
     int l2_bits;
     int l2_size;
-    int l1_size;
+    unsigned int l1_size;
     uint64_t cluster_offset_mask;
     uint64_t l1_table_offset;
     uint64_t *l1_table;
@@ -166,7 +166,19 @@ static int qcow_open(BlockDriverState *bs, QDict *options, int flags,
 
     /* read the level 1 table */
     shift = s->cluster_bits + s->l2_bits;
-    s->l1_size = (header.size + (1LL << shift) - 1) >> shift;
+    if (header.size > UINT64_MAX - (1LL << shift)) {
+        error_setg(errp, "Image too large");
+        ret = -EINVAL;
+        goto fail;
+    } else {
+        uint64_t l1_size = (header.size + (1LL << shift) - 1) >> shift;
+        if (l1_size > INT_MAX / sizeof(uint64_t)) {
+            error_setg(errp, "Image too large");
+            ret = -EINVAL;
+            goto fail;
+        }
+        s->l1_size = l1_size;
+    }
 
     s->l1_table_offset = header.l1_table_offset;
     s->l1_table = g_malloc(s->l1_size * sizeof(uint64_t));
diff --git a/tests/qemu-iotests/092 b/tests/qemu-iotests/092
index fb8bacc..ae6ca76 100755
--- a/tests/qemu-iotests/092
+++ b/tests/qemu-iotests/092
@@ -43,6 +43,7 @@ _supported_fmt qcow
 _supported_proto generic
 _supported_os Linux
 
+offset_size=24
 offset_cluster_bits=32
 offset_l2_bits=33
 
@@ -72,6 +73,14 @@ poke_file "$TEST_IMG" "$offset_l2_bits" "\x0e"
 poke_file "$TEST_IMG" "$offset_l2_bits" "\x1b"
 { $QEMU_IO -c "read 0 512" $TEST_IMG; } 2>&1 | _filter_qemu_io | _filter_testdir
 
+echo
+echo "== Invalid size =="
+_make_test_img 64M
+poke_file "$TEST_IMG" "$offset_size" "\xee\xee\xee\xee\xee\xee\xee\xee"
+{ $QEMU_IO -c "read 0 512" $TEST_IMG; } 2>&1 | _filter_qemu_io | _filter_testdir
+poke_file "$TEST_IMG" "$offset_size" "\x7f\xff\xff\xff\xff\xff\xff\xff"
+{ $QEMU_IO -c "write 0 64M" $TEST_IMG; } 2>&1 | _filter_qemu_io | _filter_testdir
+
 # success, all done
 echo "*** done"
 rm -f $seq.full
diff --git a/tests/qemu-iotests/092.out b/tests/qemu-iotests/092.out
index 73918b3..ac03302 100644
--- a/tests/qemu-iotests/092.out
+++ b/tests/qemu-iotests/092.out
@@ -21,4 +21,11 @@ qemu-io: can't open device TEST_DIR/t.qcow: L2 table size must be between 512 an
 no file open, try 'help open'
 qemu-io: can't open device TEST_DIR/t.qcow: L2 table size must be between 512 and 64k
 no file open, try 'help open'
+
+== Invalid size ==
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 
+qemu-io: can't open device TEST_DIR/t.qcow: Image too large
+no file open, try 'help open'
+qemu-io: can't open device TEST_DIR/t.qcow: Image too large
+no file open, try 'help open'
 *** done
-- 
1.8.3.1

[Qemu-devel] [PULL 19/22] qcow1: Stricter backing file length check

[Kevin Wolf]

Like qcow2 since commit 6d33e8e7, error out on invalid lengths instead
of silently truncating them to 1023.

Also don't rely on bdrv_pread() catching integer overflows that make len
negative, but use unsigned variables in the first place.

Cc: qemu-stable@nongnu.org
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Benoit Canet <benoit@irqsave.net>
---
 block/qcow.c               |  7 +++++--
 tests/qemu-iotests/092     | 11 +++++++++++
 tests/qemu-iotests/092.out |  7 +++++++
 3 files changed, 23 insertions(+), 2 deletions(-)

diff --git a/block/qcow.c b/block/qcow.c
index 3566c05..7fd57d7 100644
--- a/block/qcow.c
+++ b/block/qcow.c
@@ -97,7 +97,8 @@ static int qcow_open(BlockDriverState *bs, QDict *options, int flags,
                      Error **errp)
 {
     BDRVQcowState *s = bs->opaque;
-    int len, i, shift, ret;
+    unsigned int len, i, shift;
+    int ret;
     QCowHeader header;
 
     ret = bdrv_pread(bs->file, 0, &header, sizeof(header));
@@ -202,7 +203,9 @@ static int qcow_open(BlockDriverState *bs, QDict *options, int flags,
     if (header.backing_file_offset != 0) {
         len = header.backing_file_size;
         if (len > 1023) {
-            len = 1023;
+            error_setg(errp, "Backing file name too long");
+            ret = -EINVAL;
+            goto fail;
         }
         ret = bdrv_pread(bs->file, header.backing_file_offset,
                    bs->backing_file, len);
diff --git a/tests/qemu-iotests/092 b/tests/qemu-iotests/092
index ae6ca76..a8c0c9c 100755
--- a/tests/qemu-iotests/092
+++ b/tests/qemu-iotests/092
@@ -43,6 +43,8 @@ _supported_fmt qcow
 _supported_proto generic
 _supported_os Linux
 
+offset_backing_file_offset=8
+offset_backing_file_size=16
 offset_size=24
 offset_cluster_bits=32
 offset_l2_bits=33
@@ -81,6 +83,15 @@ poke_file "$TEST_IMG" "$offset_size" "\xee\xee\xee\xee\xee\xee\xee\xee"
 poke_file "$TEST_IMG" "$offset_size" "\x7f\xff\xff\xff\xff\xff\xff\xff"
 { $QEMU_IO -c "write 0 64M" $TEST_IMG; } 2>&1 | _filter_qemu_io | _filter_testdir
 
+echo
+echo "== Invalid backing file length =="
+_make_test_img 64M
+poke_file "$TEST_IMG" "$offset_backing_file_offset" "\x00\x00\x00\xff"
+poke_file "$TEST_IMG" "$offset_backing_file_size" "\xff\xff\xff\xff"
+{ $QEMU_IO -c "read 0 512" $TEST_IMG; } 2>&1 | _filter_qemu_io | _filter_testdir
+poke_file "$TEST_IMG" "$offset_backing_file_size" "\x7f\xff\xff\xff"
+{ $QEMU_IO -c "read 0 512" $TEST_IMG; } 2>&1 | _filter_qemu_io | _filter_testdir
+
 # success, all done
 echo "*** done"
 rm -f $seq.full
diff --git a/tests/qemu-iotests/092.out b/tests/qemu-iotests/092.out
index ac03302..496d8f0 100644
--- a/tests/qemu-iotests/092.out
+++ b/tests/qemu-iotests/092.out
@@ -28,4 +28,11 @@ qemu-io: can't open device TEST_DIR/t.qcow: Image too large
 no file open, try 'help open'
 qemu-io: can't open device TEST_DIR/t.qcow: Image too large
 no file open, try 'help open'
+
+== Invalid backing file length ==
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 
+qemu-io: can't open device TEST_DIR/t.qcow: Backing file name too long
+no file open, try 'help open'
+qemu-io: can't open device TEST_DIR/t.qcow: Backing file name too long
+no file open, try 'help open'
 *** done
-- 
1.8.3.1

[Qemu-devel] [PULL 20/22] util: add qemu_iovec_is_zero

[Kevin Wolf]

From: Peter Lieven <pl@kamp.de>

Signed-off-by: Peter Lieven <pl@kamp.de>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 include/qemu-common.h |  1 +
 util/iov.c            | 21 +++++++++++++++++++++
 2 files changed, 22 insertions(+)

diff --git a/include/qemu-common.h b/include/qemu-common.h
index 3f3fd60..66ceceb 100644
--- a/include/qemu-common.h
+++ b/include/qemu-common.h
@@ -318,6 +318,7 @@ void qemu_iovec_concat(QEMUIOVector *dst,
 void qemu_iovec_concat_iov(QEMUIOVector *dst,
                            struct iovec *src_iov, unsigned int src_cnt,
                            size_t soffset, size_t sbytes);
+bool qemu_iovec_is_zero(QEMUIOVector *qiov);
 void qemu_iovec_destroy(QEMUIOVector *qiov);
 void qemu_iovec_reset(QEMUIOVector *qiov);
 size_t qemu_iovec_to_buf(QEMUIOVector *qiov, size_t offset,
diff --git a/util/iov.c b/util/iov.c
index 6569b5a..49f8838 100644
--- a/util/iov.c
+++ b/util/iov.c
@@ -335,6 +335,27 @@ void qemu_iovec_concat(QEMUIOVector *dst,
     qemu_iovec_concat_iov(dst, src->iov, src->niov, soffset, sbytes);
 }
 
+/*
+ * Check if the contents of the iovecs are all zero
+ */
+bool qemu_iovec_is_zero(QEMUIOVector *qiov)
+{
+    int i;
+    for (i = 0; i < qiov->niov; i++) {
+        size_t offs = QEMU_ALIGN_DOWN(qiov->iov[i].iov_len, 4 * sizeof(long));
+        uint8_t *ptr = qiov->iov[i].iov_base;
+        if (offs && !buffer_is_zero(qiov->iov[i].iov_base, offs)) {
+            return false;
+        }
+        for (; offs < qiov->iov[i].iov_len; offs++) {
+            if (ptr[offs]) {
+                return false;
+            }
+        }
+    }
+    return true;
+}
+
 void qemu_iovec_destroy(QEMUIOVector *qiov)
 {
     assert(qiov->nalloc != -1);
-- 
1.8.3.1

[Qemu-devel] [PULL 21/22] blockdev: add a function to parse enum ids from strings

[Kevin Wolf]

From: Peter Lieven <pl@kamp.de>

this adds a generic function to recover the enum id of a parameter
given as a string.

Signed-off-by: Peter Lieven <pl@kamp.de>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 blockdev.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/blockdev.c b/blockdev.c
index 7810e9f..78b927f 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -288,6 +288,25 @@ static int parse_block_error_action(const char *buf, bool is_read, Error **errp)
     }
 }
 
+static inline int parse_enum_option(const char *lookup[], const char *buf,
+                                    int max, int def, Error **errp)
+{
+    int i;
+
+    if (!buf) {
+        return def;
+    }
+
+    for (i = 0; i < max; i++) {
+        if (!strcmp(buf, lookup[i])) {
+            return i;
+        }
+    }
+
+    error_setg(errp, "invalid parameter value: %s", buf);
+    return def;
+}
+
 static bool check_throttle_config(ThrottleConfig *cfg, Error **errp)
 {
     if (throttle_conflicting(cfg)) {
-- 
1.8.3.1

[Qemu-devel] [PULL 22/22] block: optimize zero writes with bdrv_write_zeroes

[Kevin Wolf]

From: Peter Lieven <pl@kamp.de>

this patch tries to optimize zero write requests
by automatically using bdrv_write_zeroes if it is
supported by the format.

This significantly speeds up file system initialization and
should speed zero write test used to test backend storage
performance.

I ran the following 2 tests on my internal SSD with a
50G QCOW2 container and on an attached iSCSI storage.

a) mkfs.ext4 -E lazy_itable_init=0,lazy_journal_init=0 /dev/vdX

QCOW2         [off]     [on]     [unmap]
-----
runtime:       14secs    1.1secs  1.1secs
filesize:      937M      18M      18M

iSCSI         [off]     [on]     [unmap]
----
runtime:       9.3s      0.9s     0.9s

b) dd if=/dev/zero of=/dev/vdX bs=1M oflag=direct

QCOW2         [off]     [on]     [unmap]
-----
runtime:       246secs   18secs   18secs
filesize:      51G       192K     192K
throughput:    203M/s    2.3G/s   2.3G/s

iSCSI*        [off]     [on]     [unmap]
----
runtime:       8mins     45secs   33secs
throughput:    106M/s    1.2G/s   1.6G/s
allocated:     100%      100%     0%

* The storage was connected via an 1Gbit interface.
  It seems to internally handle writing zeroes
  via WRITESAME16 very fast.

Signed-off-by: Peter Lieven <pl@kamp.de>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 block.c                    |  9 ++++++++
 block/qapi.c               |  1 +
 blockdev.c                 | 24 +++++++++++++++++++++
 hmp.c                      |  5 +++++
 include/block/block_int.h  |  1 +
 qapi-schema.json           | 52 +++++++++++++++++++++++++++++++++-------------
 qemu-options.hx            |  6 ++++++
 qmp-commands.hx            |  3 +++
 tests/qemu-iotests/067.out | 10 ++++-----
 9 files changed, 91 insertions(+), 20 deletions(-)

diff --git a/block.c b/block.c
index 415ae3d..40c5e1a 100644
--- a/block.c
+++ b/block.c
@@ -3289,6 +3289,15 @@ static int coroutine_fn bdrv_aligned_pwritev(BlockDriverState *bs,
 
     ret = notifier_with_return_list_notify(&bs->before_write_notifiers, req);
 
+    if (!ret && bs->detect_zeroes != BLOCKDEV_DETECT_ZEROES_OPTIONS_OFF &&
+        !(flags & BDRV_REQ_ZERO_WRITE) && drv->bdrv_co_write_zeroes &&
+        qemu_iovec_is_zero(qiov)) {
+        flags |= BDRV_REQ_ZERO_WRITE;
+        if (bs->detect_zeroes == BLOCKDEV_DETECT_ZEROES_OPTIONS_UNMAP) {
+            flags |= BDRV_REQ_MAY_UNMAP;
+        }
+    }
+
     if (ret < 0) {
         /* Do nothing, write notifier decided to fail this request */
     } else if (flags & BDRV_REQ_ZERO_WRITE) {
diff --git a/block/qapi.c b/block/qapi.c
index af11445..75f44f1 100644
--- a/block/qapi.c
+++ b/block/qapi.c
@@ -50,6 +50,7 @@ BlockDeviceInfo *bdrv_block_device_info(BlockDriverState *bs)
     }
 
     info->backing_file_depth = bdrv_get_backing_file_depth(bs);
+    info->detect_zeroes = bs->detect_zeroes;
 
     if (bs->io_limits_enabled) {
         ThrottleConfig cfg;
diff --git a/blockdev.c b/blockdev.c
index 78b927f..1cbcc1c 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -343,6 +343,7 @@ static DriveInfo *blockdev_init(const char *file, QDict *bs_opts,
     QemuOpts *opts;
     const char *id;
     bool has_driver_specific_opts;
+    BlockdevDetectZeroesOptions detect_zeroes;
     BlockDriver *drv = NULL;
 
     /* Check common options by copying from bs_opts to opts, all other options
@@ -471,6 +472,24 @@ static DriveInfo *blockdev_init(const char *file, QDict *bs_opts,
         }
     }
 
+    detect_zeroes =
+        parse_enum_option(BlockdevDetectZeroesOptions_lookup,
+                          qemu_opt_get(opts, "detect-zeroes"),
+                          BLOCKDEV_DETECT_ZEROES_OPTIONS_MAX,
+                          BLOCKDEV_DETECT_ZEROES_OPTIONS_OFF,
+                          &error);
+    if (error) {
+        error_propagate(errp, error);
+        goto early_err;
+    }
+
+    if (detect_zeroes == BLOCKDEV_DETECT_ZEROES_OPTIONS_UNMAP &&
+        !(bdrv_flags & BDRV_O_UNMAP)) {
+        error_setg(errp, "setting detect-zeroes to unmap is not allowed "
+                         "without setting discard operation to unmap");
+        goto early_err;
+    }
+
     /* init */
     dinfo = g_malloc0(sizeof(*dinfo));
     dinfo->id = g_strdup(qemu_opts_id(opts));
@@ -481,6 +500,7 @@ static DriveInfo *blockdev_init(const char *file, QDict *bs_opts,
     }
     dinfo->bdrv->open_flags = snapshot ? BDRV_O_SNAPSHOT : 0;
     dinfo->bdrv->read_only = ro;
+    dinfo->bdrv->detect_zeroes = detect_zeroes;
     dinfo->refcount = 1;
     if (serial != NULL) {
         dinfo->serial = g_strdup(serial);
@@ -2474,6 +2494,10 @@ QemuOptsList qemu_common_drive_opts = {
             .name = "copy-on-read",
             .type = QEMU_OPT_BOOL,
             .help = "copy read data from backing file into image file",
+        },{
+            .name = "detect-zeroes",
+            .type = QEMU_OPT_STRING,
+            .help = "try to optimize zero writes (off, on, unmap)",
         },
         { /* end of list */ }
     },
diff --git a/hmp.c b/hmp.c
index 5c4d612..ff506f3 100644
--- a/hmp.c
+++ b/hmp.c
@@ -341,6 +341,11 @@ void hmp_info_block(Monitor *mon, const QDict *qdict)
                            info->value->inserted->backing_file_depth);
         }
 
+        if (info->value->inserted->detect_zeroes != BLOCKDEV_DETECT_ZEROES_OPTIONS_OFF) {
+            monitor_printf(mon, "    Detect zeroes:    %s\n",
+                           BlockdevDetectZeroesOptions_lookup[info->value->inserted->detect_zeroes]);
+        }
+
         if (info->value->inserted->bps
             || info->value->inserted->bps_rd
             || info->value->inserted->bps_wr
diff --git a/include/block/block_int.h b/include/block/block_int.h
index 9ffcb69..b8cc926 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -364,6 +364,7 @@ struct BlockDriverState {
     BlockJob *job;
 
     QDict *options;
+    BlockdevDetectZeroesOptions detect_zeroes;
 };
 
 int get_tmp_filename(char *filename, int size);
diff --git a/qapi-schema.json b/qapi-schema.json
index 36cb964..70ed7e3 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -942,6 +942,8 @@
 # @encryption_key_missing: true if the backing device is encrypted but an
 #                          valid encryption key is missing
 #
+# @detect_zeroes: detect and optimize zero writes (Since 2.1)
+#
 # @bps: total throughput limit in bytes per second is specified
 #
 # @bps_rd: read throughput limit in bytes per second is specified
@@ -977,6 +979,7 @@
   'data': { 'file': 'str', '*node-name': 'str', 'ro': 'bool', 'drv': 'str',
             '*backing_file': 'str', 'backing_file_depth': 'int',
             'encrypted': 'bool', 'encryption_key_missing': 'bool',
+            'detect_zeroes': 'BlockdevDetectZeroesOptions',
             'bps': 'int', 'bps_rd': 'int', 'bps_wr': 'int',
             'iops': 'int', 'iops_rd': 'int', 'iops_wr': 'int',
             'image': 'ImageInfo',
@@ -4255,6 +4258,22 @@
   'data': [ 'ignore', 'unmap' ] }
 
 ##
+# @BlockdevDetectZeroesOptions
+#
+# Describes the operation mode for the automatic conversion of plain
+# zero writes by the OS to driver specific optimized zero write commands.
+#
+# @off:      Disabled (default)
+# @on:       Enabled
+# @unmap:    Enabled and even try to unmap blocks if possible. This requires
+#            also that @BlockdevDiscardOptions is set to unmap for this device.
+#
+# Since: 2.1
+##
+{ 'enum': 'BlockdevDetectZeroesOptions',
+  'data': [ 'off', 'on', 'unmap' ] }
+
+##
 # @BlockdevAioOptions
 #
 # Selects the AIO backend to handle I/O requests
@@ -4306,20 +4325,22 @@
 # Options that are available for all block devices, independent of the block
 # driver.
 #
-# @driver:      block driver name
-# @id:          #optional id by which the new block device can be referred to.
-#               This is a required option on the top level of blockdev-add, and
-#               currently not allowed on any other level.
-# @node-name:   #optional the name of a block driver state node (Since 2.0)
-# @discard:     #optional discard-related options (default: ignore)
-# @cache:       #optional cache-related options
-# @aio:         #optional AIO backend (default: threads)
-# @rerror:      #optional how to handle read errors on the device
-#               (default: report)
-# @werror:      #optional how to handle write errors on the device
-#               (default: enospc)
-# @read-only:   #optional whether the block device should be read-only
-#               (default: false)
+# @driver:        block driver name
+# @id:            #optional id by which the new block device can be referred to.
+#                 This is a required option on the top level of blockdev-add, and
+#                 currently not allowed on any other level.
+# @node-name:     #optional the name of a block driver state node (Since 2.0)
+# @discard:       #optional discard-related options (default: ignore)
+# @cache:         #optional cache-related options
+# @aio:           #optional AIO backend (default: threads)
+# @rerror:        #optional how to handle read errors on the device
+#                 (default: report)
+# @werror:        #optional how to handle write errors on the device
+#                 (default: enospc)
+# @read-only:     #optional whether the block device should be read-only
+#                 (default: false)
+# @detect-zeroes: #optional detect and optimize zero writes (Since 2.1)
+#                 (default: off)
 #
 # Since: 1.7
 ##
@@ -4332,7 +4353,8 @@
             '*aio': 'BlockdevAioOptions',
             '*rerror': 'BlockdevOnError',
             '*werror': 'BlockdevOnError',
-            '*read-only': 'bool' } }
+            '*read-only': 'bool',
+            '*detect-zeroes': 'BlockdevDetectZeroesOptions' } }
 
 ##
 # @BlockdevOptionsFile
diff --git a/qemu-options.hx b/qemu-options.hx
index 7587bce..c2c0823 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -414,6 +414,7 @@ DEF("drive", HAS_ARG, QEMU_OPTION_drive,
     "       [,serial=s][,addr=A][,rerror=ignore|stop|report]\n"
     "       [,werror=ignore|stop|report|enospc][,id=name][,aio=threads|native]\n"
     "       [,readonly=on|off][,copy-on-read=on|off]\n"
+    "       [,detect-zeroes=on|off|unmap]\n"
     "       [[,bps=b]|[[,bps_rd=r][,bps_wr=w]]]\n"
     "       [[,iops=i]|[[,iops_rd=r][,iops_wr=w]]]\n"
     "       [[,bps_max=bm]|[[,bps_rd_max=rm][,bps_wr_max=wm]]]\n"
@@ -475,6 +476,11 @@ Open drive @option{file} as read-only. Guest write attempts will fail.
 @item copy-on-read=@var{copy-on-read}
 @var{copy-on-read} is "on" or "off" and enables whether to copy read backing
 file sectors into the image file.
+@item detect-zeroes=@var{detect-zeroes}
+@var{detect-zeroes} is "off", "on" or "unmap" and enables the automatic
+conversion of plain zero writes by the OS to driver specific optimized
+zero write commands. You may even choose "unmap" if @var{discard} is set
+to "unmap" to allow a zero write to be converted to an UNMAP operation.
 @end table
 
 By default, the @option{cache=writeback} mode is used. It will report data
diff --git a/qmp-commands.hx b/qmp-commands.hx
index cae890e..2c53c1b 100644
--- a/qmp-commands.hx
+++ b/qmp-commands.hx
@@ -2032,6 +2032,8 @@ Each json-object contain the following:
          - "iops_rd_max":  read I/O operations max (json-int)
          - "iops_wr_max":  write I/O operations max (json-int)
          - "iops_size": I/O size when limiting by iops (json-int)
+         - "detect_zeroes": detect and optimize zero writing (json-string)
+             - Possible values: "off", "on", "unmap"
          - "image": the detail of the image, it is a json-object containing
             the following:
              - "filename": image file name (json-string)
@@ -2108,6 +2110,7 @@ Example:
                "iops_rd_max": 0,
                "iops_wr_max": 0,
                "iops_size": 0,
+               "detect_zeroes": "on",
                "image":{
                   "filename":"disks/test.qcow2",
                   "format":"qcow2",
diff --git a/tests/qemu-iotests/067.out b/tests/qemu-iotests/067.out
index 8d271cc..7e090b9 100644
--- a/tests/qemu-iotests/067.out
+++ b/tests/qemu-iotests/067.out
@@ -6,7 +6,7 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728
 Testing: -drive file=TEST_DIR/t.qcow2,format=qcow2,if=none,id=disk -device virtio-blk-pci,drive=disk,id=virtio0
 QMP_VERSION
 {"return": {}}
-{"return": [{"io-status": "ok", "device": "disk", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "image": {"virtual-size": 134217728, "filename": "TEST_DIR/t.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": SIZE, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "file": "TEST_DIR/t.qcow2", "encryption_key_missing": false}, "type": "unknown"}, {"io-status": "ok", "device": "ide1-cd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "floppy0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "sd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}]}
+{"return": [{"io-status": "ok", "device": "disk", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 134217728, "filename": "TEST_DIR/t.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": SIZE, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "file": "TEST_DIR/t.qcow2", "encryption_key_missing": false}, "type": "unknown"}, {"io-status": "ok", "device": "ide1-cd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "floppy0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "sd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}]}
 {"return": {}}
 {"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "DEVICE_DELETED", "data": {"path": "/machine/peripheral/virtio0/virtio-backend"}}
@@ -24,7 +24,7 @@ QMP_VERSION
 Testing: -drive file=TEST_DIR/t.qcow2,format=qcow2,if=none,id=disk
 QMP_VERSION
 {"return": {}}
-{"return": [{"device": "disk", "locked": false, "removable": true, "inserted": {"iops_rd": 0, "image": {"virtual-size": 134217728, "filename": "TEST_DIR/t.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": SIZE, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "file": "TEST_DIR/t.qcow2", "encryption_key_missing": false}, "tray_open": false, "type": "unknown"}, {"io-status": "ok", "device": "ide1-cd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "floppy0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "sd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}]}
+{"return": [{"device": "disk", "locked": false, "removable": true, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 134217728, "filename": "TEST_DIR/t.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": SIZE, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "file": "TEST_DIR/t.qcow2", "encryption_key_missing": false}, "tray_open": false, "type": "unknown"}, {"io-status": "ok", "device": "ide1-cd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "floppy0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "sd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}]}
 {"return": {}}
 {"return": {}}
 {"return": {}}
@@ -44,7 +44,7 @@ Testing:
 QMP_VERSION
 {"return": {}}
 {"return": "OK\r\n"}
-{"return": [{"io-status": "ok", "device": "ide1-cd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "floppy0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "sd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "disk", "locked": false, "removable": true, "inserted": {"iops_rd": 0, "image": {"virtual-size": 134217728, "filename": "TEST_DIR/t.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": SIZE, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "file": "TEST_DIR/t.qcow2", "encryption_key_missing": false}, "tray_open": false, "type": "unknown"}]}
+{"return": [{"io-status": "ok", "device": "ide1-cd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "floppy0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "sd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "disk", "locked": false, "removable": true, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 134217728, "filename": "TEST_DIR/t.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": SIZE, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "file": "TEST_DIR/t.qcow2", "encryption_key_missing": false}, "tray_open": false, "type": "unknown"}]}
 {"return": {}}
 {"return": {}}
 {"return": {}}
@@ -64,14 +64,14 @@ Testing:
 QMP_VERSION
 {"return": {}}
 {"return": {}}
-{"return": [{"io-status": "ok", "device": "ide1-cd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "floppy0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "sd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "disk", "locked": false, "removable": true, "inserted": {"iops_rd": 0, "image": {"virtual-size": 134217728, "filename": "TEST_DIR/t.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": SIZE, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "file": "TEST_DIR/t.qcow2", "encryption_key_missing": false}, "tray_open": false, "type": "unknown"}]}
+{"return": [{"io-status": "ok", "device": "ide1-cd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "floppy0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "sd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "disk", "locked": false, "removable": true, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 134217728, "filename": "TEST_DIR/t.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": SIZE, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "file": "TEST_DIR/t.qcow2", "encryption_key_missing": false}, "tray_open": false, "type": "unknown"}]}
 {"return": {}}
 {"return": {}}
 {"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "DEVICE_DELETED", "data": {"path": "/machine/peripheral/virtio0/virtio-backend"}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "DEVICE_DELETED", "data": {"device": "virtio0", "path": "/machine/peripheral/virtio0"}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "RESET"}
-{"return": [{"io-status": "ok", "device": "ide1-cd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "floppy0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "sd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"io-status": "ok", "device": "disk", "locked": false, "removable": true, "inserted": {"iops_rd": 0, "image": {"virtual-size": 134217728, "filename": "TEST_DIR/t.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": SIZE, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "file": "TEST_DIR/t.qcow2", "encryption_key_missing": false}, "tray_open": false, "type": "unknown"}]}
+{"return": [{"io-status": "ok", "device": "ide1-cd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "floppy0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"device": "sd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown"}, {"io-status": "ok", "device": "disk", "locked": false, "removable": true, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 134217728, "filename": "TEST_DIR/t.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": SIZE, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "file": "TEST_DIR/t.qcow2", "encryption_key_missing": false}, "tray_open": false, "type": "unknown"}]}
 {"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN"}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "DEVICE_TRAY_MOVED", "data": {"device": "ide1-cd0", "tray-open": true}}
-- 
1.8.3.1

Re: [Qemu-devel] [PULL 00/22] Block patches

[Peter Maydell]

On 19 May 2014 15:22, Kevin Wolf <kwolf@redhat.com> wrote:
> The following changes since commit 1a381811b495651ddfc2b50d7c8cfaaf375816b0:
>
>   Merge remote-tracking branch 'remotes/rth/tags/pull-tcg-s390-20140515' into staging (2014-05-15 18:56:08 +0100)
>
> are available in the git repository at:
>
>
>   git://repo.or.cz/qemu/kevin.git tags/for-upstream
>
> for you to fetch changes up to 465bee1da82e43f18d10c43cc7566d0284ad13a9:
>
>   block: optimize zero writes with bdrv_write_zeroes (2014-05-19 13:42:27 +0200)

Applied, thanks.

-- PMM

[Qemu-devel] [PULL 00/22] Block patches

[Kevin Wolf]

The following changes since commit 10601bef566461efbea81ee344ed3355ce71954a:

  Merge remote-tracking branch 'remotes/mcayland/tags/qemu-sparc-signed' into staging (2014-09-09 15:08:05 +0100)

are available in the git repository at:


  git://repo.or.cz/qemu/kevin.git tags/for-upstream

for you to fetch changes up to 0e4271b711a8ea766d29824c844e268b91ac3ae5:

  qcow2: Add falloc and full preallocation option (2014-09-12 15:43:06 +0200)

----------------------------------------------------------------
Block patches

----------------------------------------------------------------
Benoît Canet (4):
      block: Extract the BlockAcctStats structure
      block: Extract the block accounting code
      block: rename BlockAcctType members to start with BLOCK_ instead of BDRV_
      block: Make the block accounting functions operate on BlockAcctStats

Chrysostomos Nanakos (2):
      block/archipelago: Implement bdrv_truncate()
      qemu-iotests: Run 025 for Archipelago block driver

Hu Tao (5):
      block: round up file size to nearest sector
      block: don't convert file size to sector size
      qapi: introduce PreallocMode and new PreallocModes full and falloc.
      raw-posix: Add falloc and full preallocation option
      qcow2: Add falloc and full preallocation option

Luiz Capitulino (2):
      block: extend BLOCK_IO_ERROR event with nospace indicator
      block: extend BLOCK_IO_ERROR with reason string

Markus Armbruster (5):
      qemu-io: Clean up openfile() after commit 2e40134
      xen_disk: Plug memory leak on error path
      xen: Drop redundant bdrv_close() from pci_piix3_xen_ide_unplug()
      thread-pool: Drop unnecessary includes
      blockdev: Refuse to drive_del something added with blockdev-add

Stefan Hajnoczi (1):
      dataplane: fix virtio_blk_data_plane_create() op blocker error path

Stratos Psomadakis (1):
      iotests: Send the correct fd in socket_scm_helper

Valentin Manea (1):
      IDE: MMIO IDE device control should be little endian

Xiaodong Gong (1):
      Fix improper usage of cpu_to_be32 in vpc

 block.c                                |  60 +++++++++----------
 block/Makefile.objs                    |   1 +
 block/accounting.c                     |  54 +++++++++++++++++
 block/archipelago.c                    |  66 ++++++++++++++++++++-
 block/cow.c                            |   3 +-
 block/gluster.c                        |   9 ++-
 block/iscsi.c                          |   4 +-
 block/nfs.c                            |   3 +-
 block/qapi.c                           |  19 +++---
 block/qcow.c                           |   7 ++-
 block/qcow2.c                          |  82 +++++++++++++++++++++-----
 block/qed.c                            |   3 +-
 block/raw-posix.c                      | 102 +++++++++++++++++++++++++--------
 block/raw-win32.c                      |   6 +-
 block/rbd.c                            |   3 +-
 block/sheepdog.c                       |   3 +-
 block/ssh.c                            |   3 +-
 block/vdi.c                            |   3 +-
 block/vhdx.c                           |   3 +-
 block/vmdk.c                           |   3 +-
 block/vpc.c                            |  11 ++--
 blockdev.c                             |  10 +++-
 dma-helpers.c                          |   2 +-
 hw/block/dataplane/virtio-blk.c        |   4 +-
 hw/block/nvme.c                        |   4 +-
 hw/block/virtio-blk.c                  |  15 +++--
 hw/block/xen_disk.c                    |  39 ++++++-------
 hw/ide/ahci.c                          |   7 ++-
 hw/ide/atapi.c                         |  18 +++---
 hw/ide/core.c                          |  24 ++++----
 hw/ide/macio.c                         |  19 +++---
 hw/ide/mmio.c                          |   4 +-
 hw/ide/piix.c                          |   1 -
 hw/scsi/scsi-disk.c                    |  45 +++++++++------
 include/block/accounting.h             |  57 ++++++++++++++++++
 include/block/block.h                  |  20 +------
 include/block/block_int.h              |   6 +-
 include/block/thread-pool.h            |   6 +-
 include/hw/virtio/virtio-blk.h         |   1 +
 include/sysemu/dma.h                   |   1 +
 qapi/block-core.json                   |  30 +++++++++-
 qemu-doc.texi                          |  17 +++++-
 qemu-img.texi                          |  17 +++++-
 qemu-io.c                              |  36 +++++-------
 tests/qemu-iotests/025                 |   2 +-
 tests/qemu-iotests/049.out             |   2 +-
 tests/qemu-iotests/082.out             |  54 ++++++++---------
 tests/qemu-iotests/104                 |  57 ++++++++++++++++++
 tests/qemu-iotests/104.out             |  12 ++++
 tests/qemu-iotests/common.filter       |  21 +++++++
 tests/qemu-iotests/group               |   1 +
 tests/qemu-iotests/socket_scm_helper.c |   2 +-
 thread-pool.c                          |   1 -
 53 files changed, 704 insertions(+), 279 deletions(-)
 create mode 100644 block/accounting.c
 create mode 100644 include/block/accounting.h
 create mode 100755 tests/qemu-iotests/104
 create mode 100644 tests/qemu-iotests/104.out

Re: [Qemu-devel] [PULL 00/22] Block patches

[Peter Maydell]

On 12 September 2014 08:56, Kevin Wolf <kwolf@redhat.com> wrote:
> The following changes since commit 10601bef566461efbea81ee344ed3355ce71954a:
>
>   Merge remote-tracking branch 'remotes/mcayland/tags/qemu-sparc-signed' into staging (2014-09-09 15:08:05 +0100)
>
> are available in the git repository at:
>
>
>   git://repo.or.cz/qemu/kevin.git tags/for-upstream
>
> for you to fetch changes up to 0e4271b711a8ea766d29824c844e268b91ac3ae5:
>
>   qcow2: Add falloc and full preallocation option (2014-09-12 15:43:06 +0200)
>
> ----------------------------------------------------------------
> Block patches
>
> ----------------------------------------------------------------

Applied, thanks.

-- PMM