[Qemu-devel] [PULL 00/20] Block patches

[Qemu-devel] [PULL 00/20] Block patches

[Kevin Wolf]

The following changes since commit a287916c712b0c57a97cd35c663c5e7ba061bc7e:

  Merge remote branch 'mst/for_anthony' into staging (2010-09-20 13:22:20 -0500)

are available in the git repository at:

  git://repo.or.cz/qemu/kevin.git for-anthony

Bernhard Kohl (2):
      scsi-generic: add missing reset handler
      scsi_bus: fix length and xfer_mode for RESERVE and RELEASE commands

Christoph Hellwig (5):
      use qemu_blockalign consistently
      raw-posix: handle > 512 byte alignment correctly
      virtio-blk: propagate the required alignment
      scsi-disk: propagate the required alignment
      ide: propagate the required alignment

Kevin Wolf (10):
      vvfat: Fix segfault on write to read-only disk
      vvfat: Fix double free for opening the image rw
      vvfat: Use cache=unsafe
      qcow2: Move sync out of write_refcount_block_entries
      qcow2: Move sync out of update_refcount
      qcow2: Move sync out of qcow2_alloc_clusters
      qcow2: Get rid of additional sync on COW
      cutils: qemu_iovec_copy and qemu_iovec_memset
      qcow2: Avoid bounce buffers for AIO read requests
      qcow2: Avoid bounce buffers for AIO write requests

Laurent Vivier (2):
      Improve qemu-nbd performance by 4400 %
      nbd: correctly manage default port

Stefan Hajnoczi (1):
      blkverify: Add block driver for verifying I/O

 Makefile.objs          |    2 +-
 block/blkverify.c      |  382 ++++++++++++++++++++++++++++++++++++++++++++++++
 block/nbd.c            |    2 -
 block/qcow2-cluster.c  |   19 ++-
 block/qcow2-refcount.c |   13 ++-
 block/qcow2-snapshot.c |    2 +
 block/qcow2.c          |  115 +++++++++------
 block/qcow2.h          |    4 +-
 block/raw-posix.c      |   79 ++++++----
 block/vvfat.c          |   26 +++-
 cutils.c               |   50 +++++-
 docs/blkverify.txt     |   69 +++++++++
 hw/ide/core.c          |    4 +-
 hw/scsi-bus.c          |    3 +-
 hw/scsi-disk.c         |   10 +-
 hw/scsi-generic.c      |   21 +++-
 hw/sd.c                |    2 +-
 hw/virtio-blk.c        |    1 +
 nbd.c                  |   25 +++-
 posix-aio-compat.c     |    2 +-
 qemu-common.h          |    3 +
 qemu-io.c              |    2 +-
 qemu-nbd.c             |    8 +-
 23 files changed, 721 insertions(+), 123 deletions(-)
 create mode 100644 block/blkverify.c
 create mode 100644 docs/blkverify.txt

Re: [Qemu-devel] [PULL 00/20] Block patches

[Anthony Liguori]

On 09/21/2010 10:21 AM, Kevin Wolf wrote:
> The following changes since commit a287916c712b0c57a97cd35c663c5e7ba061bc7e:
>
>    Merge remote branch 'mst/for_anthony' into staging (2010-09-20 13:22:20 -0500)
>
> are available in the git repository at:
>
>    git://repo.or.cz/qemu/kevin.git for-anthony
>    

Pulled.  Thanks.

Regards,

Anthony Liguori

> Bernhard Kohl (2):
>        scsi-generic: add missing reset handler
>        scsi_bus: fix length and xfer_mode for RESERVE and RELEASE commands
>
> Christoph Hellwig (5):
>        use qemu_blockalign consistently
>        raw-posix: handle>  512 byte alignment correctly
>        virtio-blk: propagate the required alignment
>        scsi-disk: propagate the required alignment
>        ide: propagate the required alignment
>
> Kevin Wolf (10):
>        vvfat: Fix segfault on write to read-only disk
>        vvfat: Fix double free for opening the image rw
>        vvfat: Use cache=unsafe
>        qcow2: Move sync out of write_refcount_block_entries
>        qcow2: Move sync out of update_refcount
>        qcow2: Move sync out of qcow2_alloc_clusters
>        qcow2: Get rid of additional sync on COW
>        cutils: qemu_iovec_copy and qemu_iovec_memset
>        qcow2: Avoid bounce buffers for AIO read requests
>        qcow2: Avoid bounce buffers for AIO write requests
>
> Laurent Vivier (2):
>        Improve qemu-nbd performance by 4400 %
>        nbd: correctly manage default port
>
> Stefan Hajnoczi (1):
>        blkverify: Add block driver for verifying I/O
>
>   Makefile.objs          |    2 +-
>   block/blkverify.c      |  382 ++++++++++++++++++++++++++++++++++++++++++++++++
>   block/nbd.c            |    2 -
>   block/qcow2-cluster.c  |   19 ++-
>   block/qcow2-refcount.c |   13 ++-
>   block/qcow2-snapshot.c |    2 +
>   block/qcow2.c          |  115 +++++++++------
>   block/qcow2.h          |    4 +-
>   block/raw-posix.c      |   79 ++++++----
>   block/vvfat.c          |   26 +++-
>   cutils.c               |   50 +++++-
>   docs/blkverify.txt     |   69 +++++++++
>   hw/ide/core.c          |    4 +-
>   hw/scsi-bus.c          |    3 +-
>   hw/scsi-disk.c         |   10 +-
>   hw/scsi-generic.c      |   21 +++-
>   hw/sd.c                |    2 +-
>   hw/virtio-blk.c        |    1 +
>   nbd.c                  |   25 +++-
>   posix-aio-compat.c     |    2 +-
>   qemu-common.h          |    3 +
>   qemu-io.c              |    2 +-
>   qemu-nbd.c             |    8 +-
>   23 files changed, 721 insertions(+), 123 deletions(-)
>   create mode 100644 block/blkverify.c
>   create mode 100644 docs/blkverify.txt
>
>
>    

[Qemu-devel] [PULL 00/20] Block patches

[Kevin Wolf]

The following changes since commit 530889ff95659d8fea81eb556e5706387fdddfa7:

  sun4u: don't set up isa_mem_base (2011-09-18 12:00:19 +0000)

are available in the git repository at:
  git://repo.or.cz/qemu/kevin.git for-anthony

Alexander Motin (1):
      AHCI Port Interrupt Enable register cleaning on soft reset

Fam Zheng (1):
      VMDK: fix leak of extent_file

Frediano Ziglio (2):
      posix-aio-compat: Removed unused offset variable
      block: avoid SIGUSR2

Kevin Wolf (1):
      raw-posix: Fix bdrv_flush error return values

Paolo Bonzini (11):
      nbd: support feature negotiation
      nbd: sync API definitions with upstream
      nbd: support NBD_SET_FLAGS ioctl
      scsi-generic: do not disable FUA
      dma-helpers: rename is_write to to_dev
      dma-helpers: allow including from target-independent code
      dma-helpers: rewrite completion/cancellation
      scsi-disk: commonize iovec creation between reads and writes
      scsi-disk: lazily allocate bounce buffer
      scsi: fix sign extension problems
      linux-aio: remove process requests callback

Sage Weil (4):
      rbd: ignore failures when reading from default conf location
      rbd: update comment heading
      rbd: call flush, if available
      rbd: allow escaping in config string

 block/nbd.c        |    4 +-
 block/raw-posix.c  |    9 +++++-
 block/rbd.c        |   83 ++++++++++++++++++++++++++++++++++----------------
 block/vmdk.c       |   14 ++++++--
 cpus.c             |    5 ---
 dma-helpers.c      |   58 +++++++++++++++++++++++------------
 dma.h              |   10 ++++--
 hw/ide/ahci.c      |    8 +++--
 hw/ide/core.c      |    2 +-
 hw/ide/macio.c     |    2 +-
 hw/scsi-bus.c      |   22 ++++---------
 hw/scsi-disk.c     |   84 +++++++++++++++++++++++++++++++---------------------
 hw/scsi-generic.c  |    6 ----
 linux-aio.c        |   11 +------
 nbd.c              |   42 +++++++++++++++++++++----
 nbd.h              |   20 ++++++++++--
 posix-aio-compat.c |   34 +++++++--------------
 qemu-nbd.c         |   13 ++++----
 18 files changed, 254 insertions(+), 173 deletions(-)

Re: [Qemu-devel] [PULL 00/20] Block patches

[Anthony Liguori]

On 09/20/2011 06:11 AM, Kevin Wolf wrote:
> The following changes since commit 530889ff95659d8fea81eb556e5706387fdddfa7:
>
>    sun4u: don't set up isa_mem_base (2011-09-18 12:00:19 +0000)
>
> are available in the git repository at:
>    git://repo.or.cz/qemu/kevin.git for-anthony

Pulled.  Thanks.

Regards,

Anthony Liguori

>
> Alexander Motin (1):
>        AHCI Port Interrupt Enable register cleaning on soft reset
>
> Fam Zheng (1):
>        VMDK: fix leak of extent_file
>
> Frediano Ziglio (2):
>        posix-aio-compat: Removed unused offset variable
>        block: avoid SIGUSR2
>
> Kevin Wolf (1):
>        raw-posix: Fix bdrv_flush error return values
>
> Paolo Bonzini (11):
>        nbd: support feature negotiation
>        nbd: sync API definitions with upstream
>        nbd: support NBD_SET_FLAGS ioctl
>        scsi-generic: do not disable FUA
>        dma-helpers: rename is_write to to_dev
>        dma-helpers: allow including from target-independent code
>        dma-helpers: rewrite completion/cancellation
>        scsi-disk: commonize iovec creation between reads and writes
>        scsi-disk: lazily allocate bounce buffer
>        scsi: fix sign extension problems
>        linux-aio: remove process requests callback
>
> Sage Weil (4):
>        rbd: ignore failures when reading from default conf location
>        rbd: update comment heading
>        rbd: call flush, if available
>        rbd: allow escaping in config string
>
>   block/nbd.c        |    4 +-
>   block/raw-posix.c  |    9 +++++-
>   block/rbd.c        |   83 ++++++++++++++++++++++++++++++++++----------------
>   block/vmdk.c       |   14 ++++++--
>   cpus.c             |    5 ---
>   dma-helpers.c      |   58 +++++++++++++++++++++++------------
>   dma.h              |   10 ++++--
>   hw/ide/ahci.c      |    8 +++--
>   hw/ide/core.c      |    2 +-
>   hw/ide/macio.c     |    2 +-
>   hw/scsi-bus.c      |   22 ++++---------
>   hw/scsi-disk.c     |   84 +++++++++++++++++++++++++++++++---------------------
>   hw/scsi-generic.c  |    6 ----
>   linux-aio.c        |   11 +------
>   nbd.c              |   42 +++++++++++++++++++++----
>   nbd.h              |   20 ++++++++++--
>   posix-aio-compat.c |   34 +++++++--------------
>   qemu-nbd.c         |   13 ++++----
>   18 files changed, 254 insertions(+), 173 deletions(-)
>
>

[Qemu-devel] [PULL 00/20] Block patches

[Kevin Wolf]

The following changes since commit a348f108842fb928563865c9918642900cd0d477:

  Add missing const attributes for MemoryRegionOps (2012-03-11 11:40:15 +0000)

are available in the git repository at:
  git://repo.or.cz/qemu/kevin.git for-anthony

Alex Barcelo (4):
      coroutine: adding sigaltstack method (.c source)
      coroutine: adding configure choose mechanism for coroutine backend
      coroutine: adding configure option for sigaltstack coroutine backend
      test-coroutine: add performance test for nesting

Kevin Wolf (8):
      qcow2: Add some tracing
      qcow2: Add error messages in qcow2_truncate
      qemu-iotests: Mark some tests as quick
      make check: Add qemu-iotests subset
      Add 'make check-block'
      qcow2: Factor out count_cow_clusters
      qcow2: Add qcow2_alloc_clusters_at()
      qcow2: Reduce number of I/O requests

Paolo Bonzini (6):
      Group snapshot: Fix format name for backing file
      use QSIMPLEQ_FOREACH_SAFE when freeing list elements
      qapi: complete implementation of unions
      rename blockdev-group-snapshot-sync
      add mode field to blockdev-snapshot-sync transaction item
      qmp: convert blockdev-snapshot-sync to a wrapper around transactions

Stefan Hajnoczi (2):
      qed: do not evict in-use L2 table cache entries
      block: handle -EBUSY in bdrv_commit_all()

 Makefile.objs               |    4 +
 block.c                     |    8 +-
 block.h                     |    2 +-
 block/qcow2-cache.c         |   18 +++
 block/qcow2-cluster.c       |  279 +++++++++++++++++++++++++----------
 block/qcow2-refcount.c      |   28 ++++
 block/qcow2.c               |   12 ++
 block/qcow2.h               |    3 +
 block/qed-l2-cache.c        |   22 +++-
 blockdev.c                  |  192 +++++++++++--------------
 configure                   |   41 +++++-
 coroutine-sigaltstack.c     |  334 +++++++++++++++++++++++++++++++++++++++++++
 hmp-commands.hx             |    9 +-
 hmp.c                       |    6 +-
 qapi-schema-test.json       |   10 ++
 qapi-schema.json            |   74 +++++++---
 qemu-img.c                  |    2 +-
 qmp-commands.hx             |   62 +++++---
 scripts/qapi-types.py       |    6 +
 scripts/qapi-visit.py       |   31 ++++-
 test-coroutine.c            |   27 ++++
 test-qmp-input-visitor.c    |   18 +++
 test-qmp-output-visitor.c   |   34 +++++
 tests/Makefile              |   12 ++-
 tests/check-block.sh        |   21 +++
 tests/qemu-iotests-quick.sh |   17 +++
 tests/qemu-iotests/group    |   24 ++--
 trace-events                |   25 ++++
 28 files changed, 1054 insertions(+), 267 deletions(-)
 create mode 100644 coroutine-sigaltstack.c
 create mode 100755 tests/check-block.sh
 create mode 100755 tests/qemu-iotests-quick.sh

Re: [Qemu-devel] [PULL 00/20] Block patches

[Anthony Liguori]

On 03/12/2012 10:19 AM, Kevin Wolf wrote:
> The following changes since commit a348f108842fb928563865c9918642900cd0d477:
>
>    Add missing const attributes for MemoryRegionOps (2012-03-11 11:40:15 +0000)
>
> are available in the git repository at:
>    git://repo.or.cz/qemu/kevin.git for-anthony
>
> Alex Barcelo (4):
>        coroutine: adding sigaltstack method (.c source)
>        coroutine: adding configure choose mechanism for coroutine backend
>        coroutine: adding configure option for sigaltstack coroutine backend
>        test-coroutine: add performance test for nesting
>
> Kevin Wolf (8):
>        qcow2: Add some tracing
>        qcow2: Add error messages in qcow2_truncate
>        qemu-iotests: Mark some tests as quick
>        make check: Add qemu-iotests subset
>        Add 'make check-block'
>        qcow2: Factor out count_cow_clusters
>        qcow2: Add qcow2_alloc_clusters_at()
>        qcow2: Reduce number of I/O requests
>
> Paolo Bonzini (6):
>        Group snapshot: Fix format name for backing file
>        use QSIMPLEQ_FOREACH_SAFE when freeing list elements
>        qapi: complete implementation of unions
>        rename blockdev-group-snapshot-sync
>        add mode field to blockdev-snapshot-sync transaction item
>        qmp: convert blockdev-snapshot-sync to a wrapper around transactions
>
> Stefan Hajnoczi (2):
>        qed: do not evict in-use L2 table cache entries
>        block: handle -EBUSY in bdrv_commit_all()


Pulled.  Thanks.

And the make check integration with qemu-iotest is sweet!

Regards,

Anthony Liguori


>
>   Makefile.objs               |    4 +
>   block.c                     |    8 +-
>   block.h                     |    2 +-
>   block/qcow2-cache.c         |   18 +++
>   block/qcow2-cluster.c       |  279 +++++++++++++++++++++++++----------
>   block/qcow2-refcount.c      |   28 ++++
>   block/qcow2.c               |   12 ++
>   block/qcow2.h               |    3 +
>   block/qed-l2-cache.c        |   22 +++-
>   blockdev.c                  |  192 +++++++++++--------------
>   configure                   |   41 +++++-
>   coroutine-sigaltstack.c     |  334 +++++++++++++++++++++++++++++++++++++++++++
>   hmp-commands.hx             |    9 +-
>   hmp.c                       |    6 +-
>   qapi-schema-test.json       |   10 ++
>   qapi-schema.json            |   74 +++++++---
>   qemu-img.c                  |    2 +-
>   qmp-commands.hx             |   62 +++++---
>   scripts/qapi-types.py       |    6 +
>   scripts/qapi-visit.py       |   31 ++++-
>   test-coroutine.c            |   27 ++++
>   test-qmp-input-visitor.c    |   18 +++
>   test-qmp-output-visitor.c   |   34 +++++
>   tests/Makefile              |   12 ++-
>   tests/check-block.sh        |   21 +++
>   tests/qemu-iotests-quick.sh |   17 +++
>   tests/qemu-iotests/group    |   24 ++--
>   trace-events                |   25 ++++
>   28 files changed, 1054 insertions(+), 267 deletions(-)
>   create mode 100644 coroutine-sigaltstack.c
>   create mode 100755 tests/check-block.sh
>   create mode 100755 tests/qemu-iotests-quick.sh
>
>

[Qemu-devel] [PULL 00/20] Block patches

[Kevin Wolf]

The following changes since commit 68c0aa6e02f79f8825c0c5dc4c7ed25d524aaa8b:

  ui/cocoa.m: Fix recent compile breakage (2013-04-21 16:44:26 +0000)

are available in the git repository at:

  git://repo.or.cz/qemu/kevin.git for-anthony

for you to fetch changes up to 7da94ca741e01a80afd65e107cc2cee160d1b2d2:

  qemu-iotests: add 053 unaligned compressed image size test (2013-04-22 11:37:12 +0200)

----------------------------------------------------------------
Kevin Wolf (16):
      qemu-iotests: Fix _filter_qemu
      block: Fail gracefully when using a format driver on protocol level
      block: Add driver-specific options for backing files
      block: Enable filename option
      raw-posix: Use bdrv_open options instead of filename
      raw-win32: Use bdrv_open options instead of filename
      blkdebug: Use bdrv_open options instead of filename
      blkverify: Use bdrv_open options instead of filename
      curl: Use bdrv_open options instead of filename
      gluster: Use bdrv_open options instead of filename
      iscsi: Use bdrv_open options instead of filename
      rbd: Use bdrv_open options instead of filename
      sheepdog: Use bdrv_open options instead of filename
      vvfat: Use bdrv_open options instead of filename
      block: Remove filename parameter from .bdrv_file_open()
      block: Allow overriding backing.file.filename

Stefan Hajnoczi (4):
      qcow2: allow sub-cluster compressed write to last cluster
      qcow: allow sub-cluster compressed write to last cluster
      qemu-img: do not zero-pad the compressed write buffer
      qemu-iotests: add 053 unaligned compressed image size test

 block.c                          |  65 +++++++++--
 block/blkdebug.c                 | 113 +++++++++++++------
 block/blkverify.c                | 113 ++++++++++++++-----
 block/curl.c                     | 152 +++++++++++++++++---------
 block/gluster.c                  |  34 +++++-
 block/iscsi.c                    |  40 ++++++-
 block/mirror.c                   |   2 +-
 block/nbd.c                      |   3 +-
 block/qcow.c                     |  17 ++-
 block/qcow2.c                    |  17 ++-
 block/raw-posix.c                |  70 ++++++++----
 block/raw-win32.c                |  59 ++++++++--
 block/rbd.c                      |  32 +++++-
 block/sheepdog.c                 |  33 +++++-
 block/ssh.c                      |   3 +-
 block/vvfat.c                    | 229 ++++++++++++++++++++++++++++-----------
 include/block/block.h            |   2 +-
 include/block/block_int.h        |   3 +-
 qemu-img.c                       |   8 +-
 tests/qemu-iotests/051           |   7 ++
 tests/qemu-iotests/051.out       |  10 ++
 tests/qemu-iotests/053           |  73 +++++++++++++
 tests/qemu-iotests/053.out       |  17 +++
 tests/qemu-iotests/common.filter |   2 +-
 tests/qemu-iotests/group         |   1 +
 25 files changed, 862 insertions(+), 243 deletions(-)
 create mode 100755 tests/qemu-iotests/053
 create mode 100644 tests/qemu-iotests/053.out

[Qemu-devel] [PULL 00/20] Block patches

[Kevin Wolf]

The following changes since commit ec3f8c9913c1eeab78a02711be7c2a803dfb4d62:

  linux-user: Fix compilation failure (2013-06-27 15:38:35 -0500)

are available in the git repository at:

  git://repo.or.cz/qemu/kevin.git for-anthony

for you to fetch changes up to 721da65c6eba9c053d73744ecaa882b0f7cd634a:

  cmd646: fix build when DEBUG_IDE is enabled. (2013-06-28 15:46:38 +0200)

----------------------------------------------------------------
Dietmar Maurer (1):
      block: add basic backup support to block driver

Fam Zheng (1):
      vmdk: remove wrong calculation of relative path

Kevin Wolf (4):
      raw-posix: Fix /dev/cdrom magic on OS X
      block: Make BlockJobTypes const
      gluster: Return bdrv_has_zero_init = 0
      vpc: Implement .bdrv_has_zero_init

Mark Cave-Ayland (1):
      cmd646: fix build when DEBUG_IDE is enabled.

Peter Lieven (1):
      block: change default of .has_zero_init to 0

Richard W.M. Jones (1):
      block/ssh: Set bdrv_has_zero_init according to the file type.

Stefan Hajnoczi (11):
      notify: add NotiferWithReturn so notifier list can abort
      block: add bdrv_add_before_write_notifier()
      blockdev: drop redundant proto_drv check
      blockdev: use bdrv_getlength() in qmp_drive_mirror()
      block: add drive-backup QMP command
      blockdev: rename BlkTransactionStates to singular
      blockdev: allow BdrvActionOps->commit() to be NULL
      blockdev: add DriveBackup transaction
      blockdev: add Abort transaction
      qemu-iotests: extract wait_until_completed() into iotests.py
      qemu-iotests: add 055 drive-backup test case

 block.c                       |  31 ++--
 block/Makefile.objs           |   1 +
 block/backup.c                | 341 ++++++++++++++++++++++++++++++++++++++++++
 block/commit.c                |   2 +-
 block/cow.c                   |   1 +
 block/gluster.c               |  10 ++
 block/mirror.c                |   2 +-
 block/qcow.c                  |   1 +
 block/qcow2.c                 |   1 +
 block/qed.c                   |   1 +
 block/raw-posix.c             |  11 +-
 block/raw-win32.c             |   7 +-
 block/rbd.c                   |   1 +
 block/sheepdog.c              |   1 +
 block/ssh.c                   |  16 ++
 block/stream.c                |   2 +-
 block/vdi.c                   |   1 +
 block/vmdk.c                  |  44 +-----
 block/vpc.c                   |  25 +++-
 blockdev.c                    | 288 ++++++++++++++++++++++++++---------
 hw/ide/cmd646.c               |   4 +-
 include/block/block.h         |   1 +
 include/block/block_int.h     |  42 +++++-
 include/qemu/notify.h         |  29 ++++
 qapi-schema.json              |  97 +++++++++++-
 qmp-commands.hx               |  46 ++++++
 tests/qemu-iotests/041        |  14 +-
 tests/qemu-iotests/055        | 282 ++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/055.out    |   5 +
 tests/qemu-iotests/group      |   1 +
 tests/qemu-iotests/iotests.py |  15 ++
 trace-events                  |   8 +
 util/notify.c                 |  30 ++++
 33 files changed, 1198 insertions(+), 163 deletions(-)
 create mode 100644 block/backup.c
 create mode 100755 tests/qemu-iotests/055
 create mode 100644 tests/qemu-iotests/055.out

[Qemu-devel] [PULL 00/20] Block patches

[Stefan Hajnoczi]

The following changes since commit 482b61844ae7c6df39df0b48ac90ffbc87bed7d2:

  Merge remote-tracking branch 'remotes/pmaydell/tags/pull-target-arm-20160617' into staging (2016-06-17 16:16:37 +0100)

are available in the git repository at:

  git://github.com/stefanha/qemu.git tags/block-pull-request

for you to fetch changes up to 5ab4b69ce29908b327a91966dc78ea0fd7424075:

  backup: follow AioContext change gracefully (2016-06-20 14:25:41 +0100)

----------------------------------------------------------------

----------------------------------------------------------------

Denis V. Lunev (3):
  block: fixed BdrvTrackedRequest filling in bdrv_co_discard
  block: fix race in bdrv_co_discard with drive-mirror
  block: process before_write_notifiers in bdrv_co_discard

Stefan Hajnoczi (17):
  libqos: use virtio_ids.h for device ID definitions
  libqos: drop duplicated PCI vendor ID definition
  libqos: drop duplicated virtio_config.h definitions
  libqos: drop duplicated virtio_ring.h bit definitions
  libqos: drop duplicated virtio_vring.h structs
  libqos: drop duplicated virtio_blk.h definitions
  libqos: drop duplicated virtio_scsi.h definitions
  libqos: drop duplicated virtio_pci.h definitions
  libqos: add qvirtqueue_cleanup()
  blockjob: move iostatus reset out of block_job_enter()
  blockjob: rename block_job_is_paused()
  blockjob: add pause points
  blockjob: add block_job_get_aio_context()
  block: use safe iteration over AioContext notifiers
  blockjob: add AioContext attached callback
  mirror: follow AioContext change gracefully
  backup: follow AioContext change gracefully

 block.c                    | 46 +++++++++++++++++-----
 block/backup.c             | 22 +++++++----
 block/io.c                 | 12 ++++--
 block/mirror.c             | 43 ++++++++++++++++----
 blockdev.c                 |  1 +
 blockjob.c                 | 97 ++++++++++++++++++++++++++++++++++++++++------
 include/block/block_int.h  |  2 +
 include/block/blockjob.h   | 51 ++++++++++++++++++------
 tests/libqos/virtio-mmio.c | 13 ++++++-
 tests/libqos/virtio-pci.c  | 60 +++++++++++++++++-----------
 tests/libqos/virtio-pci.h  | 17 --------
 tests/libqos/virtio.c      | 48 +++++++++++++----------
 tests/libqos/virtio.h      | 78 ++++++-------------------------------
 tests/virtio-blk-test.c    | 96 ++++++++++++++++++++-------------------------
 tests/virtio-net-test.c    | 12 +++---
 tests/virtio-scsi-test.c   | 55 ++++++++++----------------
 16 files changed, 381 insertions(+), 272 deletions(-)

-- 
2.5.5

Re: [Qemu-devel] [PULL 00/20] Block patches

[Peter Maydell]

On 20 June 2016 at 15:05, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> The following changes since commit 482b61844ae7c6df39df0b48ac90ffbc87bed7d2:
>
>   Merge remote-tracking branch 'remotes/pmaydell/tags/pull-target-arm-20160617' into staging (2016-06-17 16:16:37 +0100)
>
> are available in the git repository at:
>
>   git://github.com/stefanha/qemu.git tags/block-pull-request
>
> for you to fetch changes up to 5ab4b69ce29908b327a91966dc78ea0fd7424075:
>
>   backup: follow AioContext change gracefully (2016-06-20 14:25:41 +0100)
>
> ----------------------------------------------------------------
>
> ----------------------------------------------------------------

Applied, thanks.

-- PMM

[Qemu-devel] [PULL 00/20] Block patches

[Fam Zheng]

The following changes since commit 9879b75873cacc88cdee490f6ab481e8ce766c69:

  Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging (2016-10-28 12:06:41 +0100)

are available in the git repository at:

  git@github.com:famz/qemu tags/for-upstream

for you to fetch changes up to 3fe71223374e71436d4aced8865e50fd36588ff7:

  aio: convert from RFifoLock to QemuRecMutex (2016-10-28 21:50:18 +0800)

----------------------------------------------------------------

Hi Peter,

This is Paolo's RFifoLock removal series.

----------------------------------------------------------------

Fam Zheng (1):
  qed: Implement .bdrv_drain

Paolo Bonzini (19):
  replication: interrupt failover if the main device is closed
  blockjob: introduce .drain callback for jobs
  mirror: use bdrv_drained_begin/bdrv_drained_end
  block: add BDS field to count in-flight requests
  block: change drain to look only at one child at a time
  block: introduce BDRV_POLL_WHILE
  nfs: move nfs_set_events out of the while loops
  nfs: use BDRV_POLL_WHILE
  sheepdog: use BDRV_POLL_WHILE
  aio: introduce qemu_get_current_aio_context
  iothread: detach all block devices before stopping them
  replication: pass BlockDriverState to reopen_backing_file
  block: prepare bdrv_reopen_multiple to release AioContext
  qemu-io: acquire AioContext
  qemu-img: call aio_context_acquire/release around block job
  block: only call aio_poll on the current thread's AioContext
  iothread: release AioContext around aio_poll
  qemu-thread: introduce QemuRecMutex
  aio: convert from RFifoLock to QemuRecMutex

 async.c                         |  29 ++-------
 block.c                         |   6 +-
 block/backup.c                  |  17 +++++
 block/block-backend.c           |  30 ++++++---
 block/commit.c                  |   2 +-
 block/io.c                      | 137 ++++++++++++++++++++++------------------
 block/mirror.c                  |  70 ++++++++++++++------
 block/nfs.c                     |  55 +++++++++-------
 block/qed-table.c               |  16 ++---
 block/qed.c                     |  16 ++++-
 block/replication.c             |  27 +++++---
 block/sheepdog.c                |  67 +++++++++++---------
 blockjob.c                      |  37 ++++++-----
 docs/multiple-iothreads.txt     |  38 ++++++-----
 hw/scsi/virtio-scsi-dataplane.c |   4 +-
 include/block/aio.h             |  24 +++++--
 include/block/block.h           |  31 ++++++++-
 include/block/block_int.h       |  27 ++++++--
 include/block/blockjob.h        |   7 ++
 include/qemu/rfifolock.h        |  54 ----------------
 include/qemu/thread-posix.h     |   6 ++
 include/qemu/thread-win32.h     |  10 +++
 include/qemu/thread.h           |   3 +
 iothread.c                      |  33 +++++++---
 qemu-img.c                      |   6 ++
 qemu-io-cmds.c                  |   6 +-
 stubs/Makefile.objs             |   1 +
 stubs/iothread.c                |   8 +++
 tests/.gitignore                |   1 -
 tests/Makefile.include          |   2 -
 tests/test-aio.c                |  22 ++++---
 tests/test-rfifolock.c          |  91 --------------------------
 util/Makefile.objs              |   1 -
 util/qemu-thread-posix.c        |  14 ++++
 util/qemu-thread-win32.c        |  25 ++++++++
 util/rfifolock.c                |  78 -----------------------
 36 files changed, 520 insertions(+), 481 deletions(-)
 delete mode 100644 include/qemu/rfifolock.h
 create mode 100644 stubs/iothread.c
 delete mode 100644 tests/test-rfifolock.c
 delete mode 100644 util/rfifolock.c

-- 
2.7.4

Re: [Qemu-devel] [PULL 00/20] Block patches

[Peter Maydell]

On 28 October 2016 at 15:49, Fam Zheng <famz@redhat.com> wrote:
> The following changes since commit 9879b75873cacc88cdee490f6ab481e8ce766c69:
>
>   Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging (2016-10-28 12:06:41 +0100)
>
> are available in the git repository at:
>
>   git@github.com:famz/qemu tags/for-upstream
>
> for you to fetch changes up to 3fe71223374e71436d4aced8865e50fd36588ff7:
>
>   aio: convert from RFifoLock to QemuRecMutex (2016-10-28 21:50:18 +0800)
>
> ----------------------------------------------------------------
>
> Hi Peter,
>
> This is Paolo's RFifoLock removal series.

Applied, thanks.

-- PMM

[Qemu-devel] [PULL 00/20] Block patches

[Max Reitz]

The following changes since commit 5ec2eca83dc478ddf24077e02a8b34dd26cd3ff9:

  Merge remote-tracking branch 'remotes/awilliam/tags/vfio-updates-20190613.0' into staging (2019-06-14 09:33:55 +0100)

are available in the Git repository at:

  https://github.com/XanClic/qemu.git tags/pull-block-2019-06-14

for you to fetch changes up to 21c1ce592a144188dfe59b9e156a97da412a59a2:

  iotests: Test qemu-img convert -C --salvage (2019-06-14 15:09:42 +0200)

----------------------------------------------------------------
Block patches:
- Allow blockdev-backup from nodes that are not in qemu's main AIO
  context to newly added nodes
- Add salvaging mode to qemu-img convert
- Minor fixes to tests, documentation, and for less Valgrind annoyance

----------------------------------------------------------------
Andrey Shinkevich (1):
  hw/block/fdc: floppy command FIFO memory initialization

John Snow (6):
  blockdev-backup: don't check aio_context too early
  iotests.py: do not use infinite waits
  QEMUMachine: add events_wait method
  iotests.py: rewrite run_job to be pickier
  iotests: add iotest 256 for testing blockdev-backup across iothread
    contexts
  event_match: always match on None value

Max Reitz (12):
  iotests: Filter 175's allocation information
  iotests: Fix intermittent failure in 219
  qemu-img: Fix options leakage in img_rebase()
  qapi/block-core: Overlays are not snapshots
  blockdev: Overlays are not snapshots
  qemu-img: Move quiet into ImgConvertState
  qemu-img: Add salvaging mode to convert
  blkdebug: Add @iotype error option
  blkdebug: Add "none" event
  blkdebug: Inject errors on .bdrv_co_block_status()
  iotests: Test qemu-img convert --salvage
  iotests: Test qemu-img convert -C --salvage

Vladimir Sementsov-Ogievskiy (1):
  iotests: restrict 254 to support only qcow2

 qapi/block-core.json          |  53 ++++++++---
 block/blkdebug.c              |  60 ++++++++++--
 blockdev.c                    |  14 +--
 hw/block/fdc.c                |   1 +
 qemu-img.c                    | 106 +++++++++++++++------
 python/qemu/__init__.py       |  67 ++++++++++----
 qemu-img-cmds.hx              |   4 +-
 qemu-img.texi                 |   4 +
 tests/qemu-iotests/082        |   1 +
 tests/qemu-iotests/082.out    |   3 +
 tests/qemu-iotests/085.out    |  10 +-
 tests/qemu-iotests/175        |  26 +++++-
 tests/qemu-iotests/175.out    |   8 +-
 tests/qemu-iotests/219        |  13 ++-
 tests/qemu-iotests/251        | 170 ++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/251.out    |  43 +++++++++
 tests/qemu-iotests/254        |   2 +
 tests/qemu-iotests/256        | 122 ++++++++++++++++++++++++
 tests/qemu-iotests/256.out    | 119 ++++++++++++++++++++++++
 tests/qemu-iotests/group      |   2 +
 tests/qemu-iotests/iotests.py |  60 +++++++-----
 21 files changed, 772 insertions(+), 116 deletions(-)
 create mode 100755 tests/qemu-iotests/251
 create mode 100644 tests/qemu-iotests/251.out
 create mode 100755 tests/qemu-iotests/256
 create mode 100644 tests/qemu-iotests/256.out

-- 
2.21.0

[Qemu-devel] [PULL 01/20] blockdev-backup: don't check aio_context too early

[Max Reitz]

From: John Snow <jsnow@redhat.com>

in blockdev_backup_prepare, we check to make sure that the target is
associated with a compatible aio context. However, do_blockdev_backup is
called later and has some logic to move the target to a compatible
aio_context. The transaction version will fail certain commands
needlessly early as a result.

Allow blockdev_backup_prepare to simply call do_blockdev_backup, which
will ultimately decide if the contexts are compatible or not.

Note: the transaction version has always disallowed this operation since
its initial commit bd8baecd (2014), whereas the version of
qmp_blockdev_backup at the time, from commit c29c1dd312f, tried to
enforce the aio_context switch instead. It's not clear, and I can't see
from the mailing list archives at the time, why the two functions take a
different approach. It wasn't until later in efd7556708b (2016) that the
standalone version tried to determine if it could set the context or
not.

Reported-by: aihua liang <aliang@redhat.com>
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1683498
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20190523170643.20794-2-jsnow@redhat.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 blockdev.c | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/blockdev.c b/blockdev.c
index 3f44b891eb..fdafa173cc 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -1876,10 +1876,6 @@ static void blockdev_backup_prepare(BlkActionState *common, Error **errp)
     }
 
     aio_context = bdrv_get_aio_context(bs);
-    if (aio_context != bdrv_get_aio_context(target)) {
-        error_setg(errp, "Backup between two IO threads is not implemented");
-        return;
-    }
     aio_context_acquire(aio_context);
     state->bs = bs;
 
-- 
2.21.0

[Qemu-devel] [PULL 02/20] iotests.py: do not use infinite waits

[Max Reitz]

From: John Snow <jsnow@redhat.com>

Cap waits to 60 seconds so that iotests can fail gracefully if something
goes wrong.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20190523170643.20794-3-jsnow@redhat.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/iotests.py | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index f11482f3dc..6a3703e6ee 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -524,7 +524,7 @@ class VM(qtest.QEMUQtestMachine):
             output_list += [key + '=' + obj[key]]
         return ','.join(output_list)
 
-    def get_qmp_events_filtered(self, wait=True):
+    def get_qmp_events_filtered(self, wait=60.0):
         result = []
         for ev in self.get_qmp_events(wait=wait):
             result.append(filter_qmp_event(ev))
@@ -542,10 +542,10 @@ class VM(qtest.QEMUQtestMachine):
 
     # Returns None on success, and an error string on failure
     def run_job(self, job, auto_finalize=True, auto_dismiss=False,
-                pre_finalize=None):
+                pre_finalize=None, wait=60.0):
         error = None
         while True:
-            for ev in self.get_qmp_events_filtered(wait=True):
+            for ev in self.get_qmp_events_filtered(wait=wait):
                 if ev['event'] == 'JOB_STATUS_CHANGE':
                     status = ev['data']['status']
                     if status == 'aborting':
@@ -650,7 +650,7 @@ class QMPTestCase(unittest.TestCase):
         self.assertEqual(self.vm.flatten_qmp_object(json.loads(json_filename[5:])),
                          self.vm.flatten_qmp_object(reference))
 
-    def cancel_and_wait(self, drive='drive0', force=False, resume=False):
+    def cancel_and_wait(self, drive='drive0', force=False, resume=False, wait=60.0):
         '''Cancel a block job and wait for it to finish, returning the event'''
         result = self.vm.qmp('block-job-cancel', device=drive, force=force)
         self.assert_qmp(result, 'return', {})
@@ -661,7 +661,7 @@ class QMPTestCase(unittest.TestCase):
         cancelled = False
         result = None
         while not cancelled:
-            for event in self.vm.get_qmp_events(wait=True):
+            for event in self.vm.get_qmp_events(wait=wait):
                 if event['event'] == 'BLOCK_JOB_COMPLETED' or \
                    event['event'] == 'BLOCK_JOB_CANCELLED':
                     self.assert_qmp(event, 'data/device', drive)
@@ -674,10 +674,10 @@ class QMPTestCase(unittest.TestCase):
         self.assert_no_active_block_jobs()
         return result
 
-    def wait_until_completed(self, drive='drive0', check_offset=True):
+    def wait_until_completed(self, drive='drive0', check_offset=True, wait=60.0):
         '''Wait for a block job to finish, returning the event'''
         while True:
-            for event in self.vm.get_qmp_events(wait=True):
+            for event in self.vm.get_qmp_events(wait=wait):
                 if event['event'] == 'BLOCK_JOB_COMPLETED':
                     self.assert_qmp(event, 'data/device', drive)
                     self.assert_qmp_absent(event, 'data/error')
-- 
2.21.0

[Qemu-devel] [PULL 03/20] QEMUMachine: add events_wait method

[Max Reitz]

From: John Snow <jsnow@redhat.com>

Instead of event_wait which looks for a single event, add an events_wait
which can look for any number of events simultaneously. However, it
will still only return one at a time, whichever happens first.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20190523170643.20794-4-jsnow@redhat.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 python/qemu/__init__.py | 69 +++++++++++++++++++++++++++++------------
 1 file changed, 49 insertions(+), 20 deletions(-)

diff --git a/python/qemu/__init__.py b/python/qemu/__init__.py
index 81d9657ec0..98ed8a2e28 100644
--- a/python/qemu/__init__.py
+++ b/python/qemu/__init__.py
@@ -402,42 +402,71 @@ class QEMUMachine(object):
         self._qmp.clear_events()
         return events
 
-    def event_wait(self, name, timeout=60.0, match=None):
+    @staticmethod
+    def event_match(event, match=None):
         """
-        Wait for specified timeout on named event in QMP; optionally filter
-        results by match.
+        Check if an event matches optional match criteria.
 
-        The 'match' is checked to be a recursive subset of the 'event'; skips
-        branch processing on match's value None
-           {"foo": {"bar": 1}} matches {"foo": None}
-           {"foo": {"bar": 1}} does not matches {"foo": {"baz": None}}
+        The match criteria takes the form of a matching subdict. The event is
+        checked to be a superset of the subdict, recursively, with matching
+        values whenever those values are not None.
+
+        Examples, with the subdict queries on the left:
+         - None matches any object.
+         - {"foo": None} matches {"foo": {"bar": 1}}
+         - {"foo": {"baz": None}} does not match {"foo": {"bar": 1}}
+         - {"foo": {"baz": 2}} matches {"foo": {"bar": 1, "baz": 2}}
         """
-        def event_match(event, match=None):
-            if match is None:
-                return True
+        if match is None:
+            return True
 
-            for key in match:
-                if key in event:
-                    if isinstance(event[key], dict):
-                        if not event_match(event[key], match[key]):
-                            return False
-                    elif event[key] != match[key]:
+        for key in match:
+            if key in event:
+                if isinstance(event[key], dict):
+                    if not QEMUMachine.event_match(event[key], match[key]):
                         return False
-                else:
+                elif event[key] != match[key]:
                     return False
+            else:
+                return False
+        return True
 
-            return True
+    def event_wait(self, name, timeout=60.0, match=None):
+        """
+        event_wait waits for and returns a named event from QMP with a timeout.
+
+        name: The event to wait for.
+        timeout: QEMUMonitorProtocol.pull_event timeout parameter.
+        match: Optional match criteria. See event_match for details.
+        """
+        return self.events_wait([(name, match)], timeout)
+
+    def events_wait(self, events, timeout=60.0):
+        """
+        events_wait waits for and returns a named event from QMP with a timeout.
+
+        events: a sequence of (name, match_criteria) tuples.
+                The match criteria are optional and may be None.
+                See event_match for details.
+        timeout: QEMUMonitorProtocol.pull_event timeout parameter.
+        """
+        def _match(event):
+            for name, match in events:
+                if (event['event'] == name and
+                    self.event_match(event, match)):
+                    return True
+            return False
 
         # Search cached events
         for event in self._events:
-            if (event['event'] == name) and event_match(event, match):
+            if _match(event):
                 self._events.remove(event)
                 return event
 
         # Poll for new events
         while True:
             event = self._qmp.pull_event(wait=timeout)
-            if (event['event'] == name) and event_match(event, match):
+            if _match(event):
                 return event
             self._events.append(event)
 
-- 
2.21.0

[Qemu-devel] [PULL 04/20] iotests.py: rewrite run_job to be pickier

[Max Reitz]

From: John Snow <jsnow@redhat.com>

Don't pull events out of the queue that don't belong to us;
be choosier so that we can use this method to drive jobs that
were launched by transactions that may have more jobs.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20190523170643.20794-5-jsnow@redhat.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/iotests.py | 48 +++++++++++++++++++++--------------
 1 file changed, 29 insertions(+), 19 deletions(-)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 6a3703e6ee..3ecef5bc90 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -543,27 +543,37 @@ class VM(qtest.QEMUQtestMachine):
     # Returns None on success, and an error string on failure
     def run_job(self, job, auto_finalize=True, auto_dismiss=False,
                 pre_finalize=None, wait=60.0):
+        match_device = {'data': {'device': job}}
+        match_id = {'data': {'id': job}}
+        events = [
+            ('BLOCK_JOB_COMPLETED', match_device),
+            ('BLOCK_JOB_CANCELLED', match_device),
+            ('BLOCK_JOB_ERROR', match_device),
+            ('BLOCK_JOB_READY', match_device),
+            ('BLOCK_JOB_PENDING', match_id),
+            ('JOB_STATUS_CHANGE', match_id)
+        ]
         error = None
         while True:
-            for ev in self.get_qmp_events_filtered(wait=wait):
-                if ev['event'] == 'JOB_STATUS_CHANGE':
-                    status = ev['data']['status']
-                    if status == 'aborting':
-                        result = self.qmp('query-jobs')
-                        for j in result['return']:
-                            if j['id'] == job:
-                                error = j['error']
-                                log('Job failed: %s' % (j['error']))
-                    elif status == 'pending' and not auto_finalize:
-                        if pre_finalize:
-                            pre_finalize()
-                        self.qmp_log('job-finalize', id=job)
-                    elif status == 'concluded' and not auto_dismiss:
-                        self.qmp_log('job-dismiss', id=job)
-                    elif status == 'null':
-                        return error
-                else:
-                    log(ev)
+            ev = filter_qmp_event(self.events_wait(events))
+            if ev['event'] != 'JOB_STATUS_CHANGE':
+                log(ev)
+                continue
+            status = ev['data']['status']
+            if status == 'aborting':
+                result = self.qmp('query-jobs')
+                for j in result['return']:
+                    if j['id'] == job:
+                        error = j['error']
+                        log('Job failed: %s' % (j['error']))
+            elif status == 'pending' and not auto_finalize:
+                if pre_finalize:
+                    pre_finalize()
+                self.qmp_log('job-finalize', id=job)
+            elif status == 'concluded' and not auto_dismiss:
+                self.qmp_log('job-dismiss', id=job)
+            elif status == 'null':
+                return error
 
     def node_info(self, node_name):
         nodes = self.qmp('query-named-block-nodes')
-- 
2.21.0

[Qemu-devel] [PULL 05/20] iotests: add iotest 256 for testing blockdev-backup across iothread contexts

[Max Reitz]

From: John Snow <jsnow@redhat.com>

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20190523170643.20794-6-jsnow@redhat.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
[mreitz: Moved from 250 to 256]
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/256     | 122 +++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/256.out | 119 ++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/group   |   1 +
 3 files changed, 242 insertions(+)
 create mode 100755 tests/qemu-iotests/256
 create mode 100644 tests/qemu-iotests/256.out

diff --git a/tests/qemu-iotests/256 b/tests/qemu-iotests/256
new file mode 100755
index 0000000000..c594a43205
--- /dev/null
+++ b/tests/qemu-iotests/256
@@ -0,0 +1,122 @@
+#!/usr/bin/env python
+#
+# Test incremental/backup across iothread contexts
+#
+# Copyright (c) 2019 John Snow for Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# owner=jsnow@redhat.com
+
+import os
+import iotests
+from iotests import log
+
+iotests.verify_image_format(supported_fmts=['qcow2'])
+size = 64 * 1024 * 1024
+
+with iotests.FilePath('img0') as img0_path, \
+     iotests.FilePath('img1') as img1_path, \
+     iotests.FilePath('img0-full') as img0_full_path, \
+     iotests.FilePath('img1-full') as img1_full_path, \
+     iotests.FilePath('img0-incr') as img0_incr_path, \
+     iotests.FilePath('img1-incr') as img1_incr_path, \
+     iotests.VM() as vm:
+
+    def create_target(filepath, name, size):
+        basename = os.path.basename(filepath)
+        nodename = "file_{}".format(basename)
+        log(vm.command('blockdev-create', job_id='job1',
+                       options={
+                           'driver': 'file',
+                           'filename': filepath,
+                           'size': 0,
+                       }))
+        vm.run_job('job1')
+        log(vm.command('blockdev-add', driver='file',
+                       node_name=nodename, filename=filepath))
+        log(vm.command('blockdev-create', job_id='job2',
+                       options={
+                           'driver': iotests.imgfmt,
+                           'file': nodename,
+                           'size': size,
+                       }))
+        vm.run_job('job2')
+        log(vm.command('blockdev-add', driver=iotests.imgfmt,
+                       node_name=name,
+                       file=nodename))
+
+    log('--- Preparing images & VM ---\n')
+    vm.add_object('iothread,id=iothread0')
+    vm.add_object('iothread,id=iothread1')
+    vm.add_device('virtio-scsi-pci,id=scsi0,iothread=iothread0')
+    vm.add_device('virtio-scsi-pci,id=scsi1,iothread=iothread1')
+    iotests.qemu_img_create('-f', iotests.imgfmt, img0_path, str(size))
+    iotests.qemu_img_create('-f', iotests.imgfmt, img1_path, str(size))
+    vm.add_drive(img0_path, interface='none')
+    vm.add_device('scsi-hd,id=device0,drive=drive0,bus=scsi0.0')
+    vm.add_drive(img1_path, interface='none')
+    vm.add_device('scsi-hd,id=device1,drive=drive1,bus=scsi1.0')
+
+    log('--- Starting VM ---\n')
+    vm.launch()
+
+    log('--- Create Targets & Full Backups ---\n')
+    create_target(img0_full_path, 'img0-full', size)
+    create_target(img1_full_path, 'img1-full', size)
+    ret = vm.qmp_log('transaction', indent=2, actions=[
+        { 'type': 'block-dirty-bitmap-add',
+          'data': { 'node': 'drive0', 'name': 'bitmap0' }},
+        { 'type': 'block-dirty-bitmap-add',
+          'data': { 'node': 'drive1', 'name': 'bitmap1' }},
+        { 'type': 'blockdev-backup',
+          'data': { 'device': 'drive0',
+                    'target': 'img0-full',
+                    'sync': 'full',
+                    'job-id': 'j0' }},
+        { 'type': 'blockdev-backup',
+          'data': { 'device': 'drive1',
+                    'target': 'img1-full',
+                    'sync': 'full',
+                    'job-id': 'j1' }}
+    ])
+    if "error" in ret:
+        raise Exception(ret['error']['desc'])
+    vm.run_job('j0', auto_dismiss=True)
+    vm.run_job('j1', auto_dismiss=True)
+
+    log('\n--- Create Targets & Incremental Backups ---\n')
+    create_target(img0_incr_path, 'img0-incr', size)
+    create_target(img1_incr_path, 'img1-incr', size)
+    ret = vm.qmp_log('transaction', indent=2, actions=[
+        { 'type': 'blockdev-backup',
+          'data': { 'device': 'drive0',
+                    'target': 'img0-incr',
+                    'sync': 'incremental',
+                    'bitmap': 'bitmap0',
+                    'job-id': 'j2' }},
+        { 'type': 'blockdev-backup',
+          'data': { 'device': 'drive1',
+                    'target': 'img1-incr',
+                    'sync': 'incremental',
+                    'bitmap': 'bitmap1',
+                    'job-id': 'j3' }}
+    ])
+    if "error" in ret:
+        raise Exception(ret['error']['desc'])
+    vm.run_job('j2', auto_dismiss=True)
+    vm.run_job('j3', auto_dismiss=True)
+
+    log('\n--- Done ---')
+    vm.shutdown()
diff --git a/tests/qemu-iotests/256.out b/tests/qemu-iotests/256.out
new file mode 100644
index 0000000000..eec38614ec
--- /dev/null
+++ b/tests/qemu-iotests/256.out
@@ -0,0 +1,119 @@
+--- Preparing images & VM ---
+
+--- Starting VM ---
+
+--- Create Targets & Full Backups ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job1"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job2"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job1"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job2"}}
+{"return": {}}
+{}
+{
+  "execute": "transaction",
+  "arguments": {
+    "actions": [
+      {
+        "data": {
+          "name": "bitmap0",
+          "node": "drive0"
+        },
+        "type": "block-dirty-bitmap-add"
+      },
+      {
+        "data": {
+          "name": "bitmap1",
+          "node": "drive1"
+        },
+        "type": "block-dirty-bitmap-add"
+      },
+      {
+        "data": {
+          "device": "drive0",
+          "job-id": "j0",
+          "sync": "full",
+          "target": "img0-full"
+        },
+        "type": "blockdev-backup"
+      },
+      {
+        "data": {
+          "device": "drive1",
+          "job-id": "j1",
+          "sync": "full",
+          "target": "img1-full"
+        },
+        "type": "blockdev-backup"
+      }
+    ]
+  }
+}
+{
+  "return": {}
+}
+{"data": {"device": "j0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "j1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Create Targets & Incremental Backups ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job1"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job2"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job1"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job2"}}
+{"return": {}}
+{}
+{
+  "execute": "transaction",
+  "arguments": {
+    "actions": [
+      {
+        "data": {
+          "bitmap": "bitmap0",
+          "device": "drive0",
+          "job-id": "j2",
+          "sync": "incremental",
+          "target": "img0-incr"
+        },
+        "type": "blockdev-backup"
+      },
+      {
+        "data": {
+          "bitmap": "bitmap1",
+          "device": "drive1",
+          "job-id": "j3",
+          "sync": "incremental",
+          "target": "img1-incr"
+        },
+        "type": "blockdev-backup"
+      }
+    ]
+  }
+}
+{
+  "return": {}
+}
+{"data": {"device": "j2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "j3", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Done ---
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index f3b6d601b2..0842167b7b 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -267,3 +267,4 @@
 253 rw auto quick
 254 rw auto backing quick
 255 rw auto quick
+256 rw auto quick
-- 
2.21.0

[Qemu-devel] [PULL 06/20] event_match: always match on None value

[Max Reitz]

From: John Snow <jsnow@redhat.com>

Before, event_match didn't always recurse if the event value was not a
dictionary, and would instead check for equality immediately.

By delaying equality checking to post-recursion, we can allow leaf
values like "5" to match "None" and take advantage of the generic
None-returns-True clause.

This makes the matching a little more obviously consistent at the
expense of being able to check for explicit None values, which is
probably not that important given what this function is used for.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20190528183857.26167-1-jsnow@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 python/qemu/__init__.py | 24 ++++++++++++++----------
 1 file changed, 14 insertions(+), 10 deletions(-)

diff --git a/python/qemu/__init__.py b/python/qemu/__init__.py
index 98ed8a2e28..dbaf8a5311 100644
--- a/python/qemu/__init__.py
+++ b/python/qemu/__init__.py
@@ -409,27 +409,31 @@ class QEMUMachine(object):
 
         The match criteria takes the form of a matching subdict. The event is
         checked to be a superset of the subdict, recursively, with matching
-        values whenever those values are not None.
+        values whenever the subdict values are not None.
+
+        This has a limitation that you cannot explicitly check for None values.
 
         Examples, with the subdict queries on the left:
          - None matches any object.
          - {"foo": None} matches {"foo": {"bar": 1}}
-         - {"foo": {"baz": None}} does not match {"foo": {"bar": 1}}
-         - {"foo": {"baz": 2}} matches {"foo": {"bar": 1, "baz": 2}}
+         - {"foo": None} matches {"foo": 5}
+         - {"foo": {"abc": None}} does not match {"foo": {"bar": 1}}
+         - {"foo": {"rab": 2}} matches {"foo": {"bar": 1, "rab": 2}}
         """
         if match is None:
             return True
 
-        for key in match:
-            if key in event:
-                if isinstance(event[key], dict):
+        try:
+            for key in match:
+                if key in event:
                     if not QEMUMachine.event_match(event[key], match[key]):
                         return False
-                elif event[key] != match[key]:
+                else:
                     return False
-            else:
-                return False
-        return True
+            return True
+        except TypeError:
+            # either match or event wasn't iterable (not a dict)
+            return match == event
 
     def event_wait(self, name, timeout=60.0, match=None):
         """
-- 
2.21.0

[Qemu-devel] [PULL 07/20] iotests: Filter 175's allocation information

[Max Reitz]

It is possible for an empty file to take up blocks on a filesystem, for
example:

$ qemu-img create -f raw test.img 1G
Formatting 'test.img', fmt=raw size=1073741824
$ mkfs.ext4 -I 128 -q test.img
$ mkdir test-mount
$ sudo mount -o loop test.img test-mount
$ sudo touch test-mount/test-file
$ stat -c 'blocks=%b' test-mount/test-file
blocks=8

These extra blocks (one cluster) are apparently used for metadata,
because they are always there, on top of blocks used for data:

$ sudo dd if=/dev/zero of=test-mount/test-file bs=1M count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00135339 s, 775 MB/s
$ stat -c 'blocks=%b' test-mount/test-file
blocks=2056

Make iotest 175 take this into account.

Reported-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Nir Soffer <nsoffer@redhat.com>
Message-id: 20190516144319.12570-1-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/175     | 26 ++++++++++++++++++++++----
 tests/qemu-iotests/175.out |  8 ++++----
 2 files changed, 26 insertions(+), 8 deletions(-)

diff --git a/tests/qemu-iotests/175 b/tests/qemu-iotests/175
index d0ffc495c2..51e62c8276 100755
--- a/tests/qemu-iotests/175
+++ b/tests/qemu-iotests/175
@@ -28,10 +28,25 @@ status=1	# failure is the default!
 
 _cleanup()
 {
-	_cleanup_test_img
+    _cleanup_test_img
+    rm -f "$TEST_DIR/empty"
 }
 trap "_cleanup; exit \$status" 0 1 2 3 15
 
+# Some file systems sometimes allocate extra blocks independently of
+# the file size.  This function hides the resulting difference in the
+# stat -c '%b' output.
+# Parameter 1: Number of blocks an empty file occupies
+# Parameter 2: Image size in bytes
+_filter_blocks()
+{
+    extra_blocks=$1
+    img_size=$2
+
+    sed -e "s/blocks=$extra_blocks\\(\$\\|[^0-9]\\)/nothing allocated/" \
+        -e "s/blocks=$((extra_blocks + img_size / 512))\\(\$\\|[^0-9]\\)/everything allocated/"
+}
+
 # get standard environment, filters and checks
 . ./common.rc
 . ./common.filter
@@ -40,18 +55,21 @@ _supported_fmt raw
 _supported_proto file
 _supported_os Linux
 
-size=1m
+size=$((1 * 1024 * 1024))
+
+touch "$TEST_DIR/empty"
+extra_blocks=$(stat -c '%b' "$TEST_DIR/empty")
 
 echo
 echo "== creating image with default preallocation =="
 _make_test_img $size | _filter_imgfmt
-stat -c "size=%s, blocks=%b" $TEST_IMG
+stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size
 
 for mode in off full falloc; do
     echo
     echo "== creating image with preallocation $mode =="
     IMGOPTS=preallocation=$mode _make_test_img $size | _filter_imgfmt
-    stat -c "size=%s, blocks=%b" $TEST_IMG
+    stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size
 done
 
 # success, all done
diff --git a/tests/qemu-iotests/175.out b/tests/qemu-iotests/175.out
index 76c02c6a57..6d9a5ed84e 100644
--- a/tests/qemu-iotests/175.out
+++ b/tests/qemu-iotests/175.out
@@ -2,17 +2,17 @@ QA output created by 175
 
 == creating image with default preallocation ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576
-size=1048576, blocks=0
+size=1048576, nothing allocated
 
 == creating image with preallocation off ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=off
-size=1048576, blocks=0
+size=1048576, nothing allocated
 
 == creating image with preallocation full ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=full
-size=1048576, blocks=2048
+size=1048576, everything allocated
 
 == creating image with preallocation falloc ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=falloc
-size=1048576, blocks=2048
+size=1048576, everything allocated
  *** done
-- 
2.21.0

[Qemu-devel] [PULL 08/20] iotests: Fix intermittent failure in 219

[Max Reitz]

In 219, we wait for the job to make progress before we emit its status.
This makes the output reliable.  We do not wait for any more progress if
the job's current-progress already matches its total-progress.

Unfortunately, there is a bug: Right after the job has been started,
it's possible that total-progress is still 0.  In that case, we may skip
the first progress-making step and keep ending up 64 kB short.

To fix that bug, we can simply wait for total-progress to reach 4 MB
(the image size) after starting the job.

Reported-by: Karen Mezick <kmezick@redhat.com>
Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=1686651
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190516161114.27596-1-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
[mreitz: Adjusted commit message as per John's proposal]
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/219 | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/tests/qemu-iotests/219 b/tests/qemu-iotests/219
index c03bbdb294..e0c51662c0 100755
--- a/tests/qemu-iotests/219
+++ b/tests/qemu-iotests/219
@@ -23,6 +23,8 @@ import iotests
 
 iotests.verify_image_format(supported_fmts=['qcow2'])
 
+img_size = 4 * 1024 * 1024
+
 def pause_wait(vm, job_id):
     with iotests.Timeout(3, "Timeout waiting for job to pause"):
         while True:
@@ -62,6 +64,8 @@ def test_pause_resume(vm):
                 iotests.log(vm.qmp('query-jobs'))
 
 def test_job_lifecycle(vm, job, job_args, has_ready=False):
+    global img_size
+
     iotests.log('')
     iotests.log('')
     iotests.log('Starting block job: %s (auto-finalize: %s; auto-dismiss: %s)' %
@@ -84,6 +88,10 @@ def test_job_lifecycle(vm, job, job_args, has_ready=False):
     iotests.log(iotests.filter_qmp_event(vm.event_wait('JOB_STATUS_CHANGE')))
     iotests.log(iotests.filter_qmp_event(vm.event_wait('JOB_STATUS_CHANGE')))
 
+    # Wait for total-progress to stabilize
+    while vm.qmp('query-jobs')['return'][0]['total-progress'] < img_size:
+        pass
+
     # RUNNING state:
     # pause/resume should work, complete/finalize/dismiss should error out
     iotests.log('')
@@ -173,9 +181,8 @@ with iotests.FilePath('disk.img') as disk_path, \
      iotests.FilePath('copy.img') as copy_path, \
      iotests.VM() as vm:
 
-    img_size = '4M'
-    iotests.qemu_img_create('-f', iotests.imgfmt, disk_path, img_size)
-    iotests.qemu_io('-c', 'write 0 %s' % (img_size),
+    iotests.qemu_img_create('-f', iotests.imgfmt, disk_path, str(img_size))
+    iotests.qemu_io('-c', 'write 0 %i' % (img_size),
                     '-f', iotests.imgfmt, disk_path)
 
     iotests.log('Launching VM...')
-- 
2.21.0

[Qemu-devel] [PULL 09/20] hw/block/fdc: floppy command FIFO memory initialization

[Max Reitz]

From: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>

The uninitialized memory allocated for the command FIFO of the
floppy controller during the VM hardware initialization incurs
many unwanted reports by Valgrind when VM state is being saved.
That verbosity hardens a search for the real memory issues when
the iotests run. Particularly, the patch eliminates 20 unnecessary
reports of the Valgrind tool in the iotest #169.

Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Message-id: 1559154027-282547-1-git-send-email-andrey.shinkevich@virtuozzo.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 hw/block/fdc.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/hw/block/fdc.c b/hw/block/fdc.c
index a64378f84b..77af9979de 100644
--- a/hw/block/fdc.c
+++ b/hw/block/fdc.c
@@ -2648,6 +2648,7 @@ static void fdctrl_realize_common(DeviceState *dev, FDCtrl *fdctrl,
 
     FLOPPY_DPRINTF("init controller\n");
     fdctrl->fifo = qemu_memalign(512, FD_SECTOR_LEN);
+    memset(fdctrl->fifo, 0, FD_SECTOR_LEN);
     fdctrl->fifo_size = 512;
     fdctrl->result_timer = timer_new_ns(QEMU_CLOCK_VIRTUAL,
                                              fdctrl_result_timer, fdctrl);
-- 
2.21.0

[Qemu-devel] [PULL 10/20] iotests: restrict 254 to support only qcow2

[Max Reitz]

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Test fails at least for qcow, because of different cluster sizes in
base and top (and therefore different granularities of bitmaps we are
trying to merge).

The test aim is to check block-dirty-bitmap-merge between different
nodes functionality, no needs to check all formats. So, let's just drop
support for anything except qcow2.

Reported-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190605155405.104384-1-vsementsov@virtuozzo.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/254 | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tests/qemu-iotests/254 b/tests/qemu-iotests/254
index 33cb80a512..8edba91c5d 100755
--- a/tests/qemu-iotests/254
+++ b/tests/qemu-iotests/254
@@ -21,6 +21,8 @@
 import iotests
 from iotests import qemu_img_create, file_path, log
 
+iotests.verify_image_format(supported_fmts=['qcow2'])
+
 disk, top = file_path('disk', 'top')
 size = 1024 * 1024
 
-- 
2.21.0

[Qemu-devel] [PULL 11/20] qemu-img: Fix options leakage in img_rebase()

[Max Reitz]

img_rebase() can leak a QDict in two occasions.  Fix it.

Coverity: CID 1401416
Fixes: d16699b64671466b42079c45b89127aeea1ca565
Fixes: 330c72957196e0ae382abcaa97ebf4eb9bc8574f
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190528195338.12376-1-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 qemu-img.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/qemu-img.c b/qemu-img.c
index fd62e3ad5d..da14aea46a 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -3350,6 +3350,7 @@ static int img_rebase(int argc, char **argv)
                                                              out_baseimg,
                                                              &local_err);
             if (local_err) {
+                qobject_unref(options);
                 error_reportf_err(local_err,
                                   "Could not resolve backing filename: ");
                 ret = -1;
@@ -3362,7 +3363,9 @@ static int img_rebase(int argc, char **argv)
              */
             prefix_chain_bs = bdrv_find_backing_image(bs, out_real_path);
             if (prefix_chain_bs) {
+                qobject_unref(options);
                 g_free(out_real_path);
+
                 blk_new_backing = blk_new(qemu_get_aio_context(),
                                           BLK_PERM_CONSISTENT_READ,
                                           BLK_PERM_ALL);
-- 
2.21.0

[Qemu-devel] [PULL 12/20] qapi/block-core: Overlays are not snapshots

[Max Reitz]

A snapshot is something that reflects the state of something at a
certain point in time.  It does not change.

The file our snapshot commands create (or the node they install) is not
a snapshot, as it does change over time.  It is an overlay.  We cannot
do anything about the parameter names, but we can at least adjust the
descriptions to reflect that fact.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20190603202236.1342-2-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 qapi/block-core.json | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index fcd054fcb1..c0ff3a83ef 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -1279,17 +1279,17 @@
 #
 # Either @device or @node-name must be set but not both.
 #
-# @device: the name of the device to generate the snapshot from.
+# @device: the name of the device to take a snapshot of.
 #
 # @node-name: graph node name to generate the snapshot from (Since 2.0)
 #
-# @snapshot-file: the target of the new image. If the file exists, or
-# if it is a device, the snapshot will be created in the existing
-# file/device. Otherwise, a new file will be created.
+# @snapshot-file: the target of the new overlay image. If the file
+# exists, or if it is a device, the overlay will be created in the
+# existing file/device. Otherwise, a new file will be created.
 #
 # @snapshot-node-name: the graph node name of the new image (Since 2.0)
 #
-# @format: the format of the snapshot image, default is 'qcow2'.
+# @format: the format of the overlay image, default is 'qcow2'.
 #
 # @mode: whether and how QEMU should create a new image, default is
 #        'absolute-paths'.
@@ -1302,10 +1302,10 @@
 ##
 # @BlockdevSnapshot:
 #
-# @node: device or node name that will have a snapshot created.
+# @node: device or node name that will have a snapshot taken.
 #
 # @overlay: reference to the existing block device that will become
-#           the overlay of @node, as part of creating the snapshot.
+#           the overlay of @node, as part of taking the snapshot.
 #           It must not have a current backing file (this can be
 #           achieved by passing "backing": null to blockdev-add).
 #
@@ -1443,7 +1443,7 @@
 ##
 # @blockdev-snapshot-sync:
 #
-# Generates a synchronous snapshot of a block device.
+# Takes a synchronous snapshot of a block device.
 #
 # For the arguments, see the documentation of BlockdevSnapshotSync.
 #
@@ -1469,9 +1469,9 @@
 ##
 # @blockdev-snapshot:
 #
-# Generates a snapshot of a block device.
+# Takes a snapshot of a block device.
 #
-# Create a snapshot, by installing 'node' as the backing image of
+# Take a snapshot, by installing 'node' as the backing image of
 # 'overlay'. Additionally, if 'node' is associated with a block
 # device, the block device changes to using 'overlay' as its new active
 # image.
-- 
2.21.0

[Qemu-devel] [PULL 13/20] blockdev: Overlays are not snapshots

[Max Reitz]

There are error messages which refer to an overlay node as the snapshot.
That is wrong, those are two different things.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20190603202236.1342-3-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 blockdev.c                 | 10 +++++-----
 tests/qemu-iotests/085.out | 10 +++++-----
 2 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/blockdev.c b/blockdev.c
index fdafa173cc..b5c0fd3c49 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -1608,13 +1608,13 @@ static void external_snapshot_prepare(BlkActionState *common,
             s->has_snapshot_node_name ? s->snapshot_node_name : NULL;
 
         if (node_name && !snapshot_node_name) {
-            error_setg(errp, "New snapshot node name missing");
+            error_setg(errp, "New overlay node name missing");
             goto out;
         }
 
         if (snapshot_node_name &&
             bdrv_lookup_bs(snapshot_node_name, snapshot_node_name, NULL)) {
-            error_setg(errp, "New snapshot node name already in use");
+            error_setg(errp, "New overlay node name already in use");
             goto out;
         }
 
@@ -1656,7 +1656,7 @@ static void external_snapshot_prepare(BlkActionState *common,
     }
 
     if (bdrv_has_blk(state->new_bs)) {
-        error_setg(errp, "The snapshot is already in use");
+        error_setg(errp, "The overlay is already in use");
         goto out;
     }
 
@@ -1666,12 +1666,12 @@ static void external_snapshot_prepare(BlkActionState *common,
     }
 
     if (state->new_bs->backing != NULL) {
-        error_setg(errp, "The snapshot already has a backing image");
+        error_setg(errp, "The overlay already has a backing image");
         goto out;
     }
 
     if (!state->new_bs->drv->supports_backing) {
-        error_setg(errp, "The snapshot does not support backing images");
+        error_setg(errp, "The overlay does not support backing images");
         goto out;
     }
 
diff --git a/tests/qemu-iotests/085.out b/tests/qemu-iotests/085.out
index 6edf107f55..2a5f256cd3 100644
--- a/tests/qemu-iotests/085.out
+++ b/tests/qemu-iotests/085.out
@@ -64,13 +64,13 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 backing_file=TEST_DIR/
 
 === Invalid command - cannot create a snapshot using a file BDS ===
 
-{"error": {"class": "GenericError", "desc": "The snapshot does not support backing images"}}
+{"error": {"class": "GenericError", "desc": "The overlay does not support backing images"}}
 
 === Invalid command - snapshot node used as active layer ===
 
-{"error": {"class": "GenericError", "desc": "The snapshot is already in use"}}
-{"error": {"class": "GenericError", "desc": "The snapshot is already in use"}}
-{"error": {"class": "GenericError", "desc": "The snapshot is already in use"}}
+{"error": {"class": "GenericError", "desc": "The overlay is already in use"}}
+{"error": {"class": "GenericError", "desc": "The overlay is already in use"}}
+{"error": {"class": "GenericError", "desc": "The overlay is already in use"}}
 
 === Invalid command - snapshot node used as backing hd ===
 
@@ -81,7 +81,7 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 backing_file=TEST_DIR/
 Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=134217728
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 backing_file=TEST_DIR/t.IMGFMT.base
 {"return": {}}
-{"error": {"class": "GenericError", "desc": "The snapshot already has a backing image"}}
+{"error": {"class": "GenericError", "desc": "The overlay already has a backing image"}}
 
 === Invalid command - The node does not exist ===
 
-- 
2.21.0

[Qemu-devel] [PULL 14/20] qemu-img: Move quiet into ImgConvertState

[Max Reitz]

Move img_convert()'s quiet flag into the ImgConvertState so it is
accessible by nested functions.  -q dictates that it suppresses anything
but errors, so if those functions want to emit warnings, they need to
query this flag first.  (There currently are no such warnings, but there
will be as of the next patch.)

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190507203508.18026-2-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 qemu-img.c | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/qemu-img.c b/qemu-img.c
index da14aea46a..e15e617256 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -1581,6 +1581,7 @@ typedef struct ImgConvertState {
     int64_t target_backing_sectors; /* negative if unknown */
     bool wr_in_order;
     bool copy_range;
+    bool quiet;
     int min_sparse;
     int alignment;
     size_t cluster_sectors;
@@ -2012,7 +2013,7 @@ static int img_convert(int argc, char **argv)
     QDict *open_opts = NULL;
     char *options = NULL;
     Error *local_err = NULL;
-    bool writethrough, src_writethrough, quiet = false, image_opts = false,
+    bool writethrough, src_writethrough, image_opts = false,
          skip_create = false, progress = false, tgt_image_opts = false;
     int64_t ret = -EINVAL;
     bool force_share = false;
@@ -2120,7 +2121,7 @@ static int img_convert(int argc, char **argv)
             src_cache = optarg;
             break;
         case 'q':
-            quiet = true;
+            s.quiet = true;
             break;
         case 'n':
             skip_create = true;
@@ -2209,7 +2210,7 @@ static int img_convert(int argc, char **argv)
     }
 
     /* Initialize before goto out */
-    if (quiet) {
+    if (s.quiet) {
         progress = false;
     }
     qemu_progress_init(progress, 1.0);
@@ -2220,7 +2221,7 @@ static int img_convert(int argc, char **argv)
 
     for (bs_i = 0; bs_i < s.src_num; bs_i++) {
         s.src[bs_i] = img_open(image_opts, argv[optind + bs_i],
-                               fmt, src_flags, src_writethrough, quiet,
+                               fmt, src_flags, src_writethrough, s.quiet,
                                force_share);
         if (!s.src[bs_i]) {
             ret = -1;
@@ -2383,7 +2384,7 @@ static int img_convert(int argc, char **argv)
 
     if (skip_create) {
         s.target = img_open(tgt_image_opts, out_filename, out_fmt,
-                            flags, writethrough, quiet, false);
+                            flags, writethrough, s.quiet, false);
     } else {
         /* TODO ultimately we should allow --target-image-opts
          * to be used even when -n is not given.
@@ -2391,7 +2392,7 @@ static int img_convert(int argc, char **argv)
          * to allow filenames in option syntax
          */
         s.target = img_open_file(out_filename, open_opts, out_fmt,
-                                 flags, writethrough, quiet, false);
+                                 flags, writethrough, s.quiet, false);
         open_opts = NULL; /* blk_new_open will have freed it */
     }
     if (!s.target) {
-- 
2.21.0

[Qemu-devel] [PULL 15/20] qemu-img: Add salvaging mode to convert

[Max Reitz]

This adds a salvaging mode (--salvage) to qemu-img convert which ignores
read errors and treats the respective areas as containing only zeroes.
This can be used for instance to at least partially recover the data
from terminally corrupted qcow2 images.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190507203508.18026-3-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 qemu-img.c       | 90 +++++++++++++++++++++++++++++++++++++-----------
 qemu-img-cmds.hx |  4 +--
 qemu-img.texi    |  4 +++
 3 files changed, 75 insertions(+), 23 deletions(-)

diff --git a/qemu-img.c b/qemu-img.c
index e15e617256..158b3a505f 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -69,6 +69,7 @@ enum {
     OPTION_SIZE = 264,
     OPTION_PREALLOCATION = 265,
     OPTION_SHRINK = 266,
+    OPTION_SALVAGE = 267,
 };
 
 typedef enum OutputFormat {
@@ -1581,6 +1582,7 @@ typedef struct ImgConvertState {
     int64_t target_backing_sectors; /* negative if unknown */
     bool wr_in_order;
     bool copy_range;
+    bool salvage;
     bool quiet;
     int min_sparse;
     int alignment;
@@ -1628,25 +1630,44 @@ static int convert_iteration_sectors(ImgConvertState *s, int64_t sector_num)
     }
 
     if (s->sector_next_status <= sector_num) {
-        int64_t count = n * BDRV_SECTOR_SIZE;
+        uint64_t offset = (sector_num - src_cur_offset) * BDRV_SECTOR_SIZE;
+        int64_t count;
 
-        if (s->target_has_backing) {
+        do {
+            count = n * BDRV_SECTOR_SIZE;
+
+            if (s->target_has_backing) {
+                ret = bdrv_block_status(blk_bs(s->src[src_cur]), offset,
+                                        count, &count, NULL, NULL);
+            } else {
+                ret = bdrv_block_status_above(blk_bs(s->src[src_cur]), NULL,
+                                              offset, count, &count, NULL,
+                                              NULL);
+            }
+
+            if (ret < 0) {
+                if (s->salvage) {
+                    if (n == 1) {
+                        if (!s->quiet) {
+                            warn_report("error while reading block status at "
+                                        "offset %" PRIu64 ": %s", offset,
+                                        strerror(-ret));
+                        }
+                        /* Just try to read the data, then */
+                        ret = BDRV_BLOCK_DATA;
+                        count = BDRV_SECTOR_SIZE;
+                    } else {
+                        /* Retry on a shorter range */
+                        n = DIV_ROUND_UP(n, 4);
+                    }
+                } else {
+                    error_report("error while reading block status at offset "
+                                 "%" PRIu64 ": %s", offset, strerror(-ret));
+                    return ret;
+                }
+            }
+        } while (ret < 0);
 
-            ret = bdrv_block_status(blk_bs(s->src[src_cur]),
-                                    (sector_num - src_cur_offset) *
-                                    BDRV_SECTOR_SIZE,
-                                    count, &count, NULL, NULL);
-        } else {
-            ret = bdrv_block_status_above(blk_bs(s->src[src_cur]), NULL,
-                                          (sector_num - src_cur_offset) *
-                                          BDRV_SECTOR_SIZE,
-                                          count, &count, NULL, NULL);
-        }
-        if (ret < 0) {
-            error_report("error while reading block status of sector %" PRId64
-                         ": %s", sector_num, strerror(-ret));
-            return ret;
-        }
         n = DIV_ROUND_UP(count, BDRV_SECTOR_SIZE);
 
         if (ret & BDRV_BLOCK_ZERO) {
@@ -1683,6 +1704,7 @@ static int convert_iteration_sectors(ImgConvertState *s, int64_t sector_num)
 static int coroutine_fn convert_co_read(ImgConvertState *s, int64_t sector_num,
                                         int nb_sectors, uint8_t *buf)
 {
+    uint64_t single_read_until = 0;
     int n, ret;
 
     assert(nb_sectors <= s->buf_sectors);
@@ -1690,6 +1712,7 @@ static int coroutine_fn convert_co_read(ImgConvertState *s, int64_t sector_num,
         BlockBackend *blk;
         int src_cur;
         int64_t bs_sectors, src_cur_offset;
+        uint64_t offset;
 
         /* In the case of compression with multiple source files, we can get a
          * nb_sectors that spreads into the next part. So we must be able to
@@ -1698,13 +1721,29 @@ static int coroutine_fn convert_co_read(ImgConvertState *s, int64_t sector_num,
         blk = s->src[src_cur];
         bs_sectors = s->src_sectors[src_cur];
 
+        offset = (sector_num - src_cur_offset) << BDRV_SECTOR_BITS;
+
         n = MIN(nb_sectors, bs_sectors - (sector_num - src_cur_offset));
+        if (single_read_until > offset) {
+            n = 1;
+        }
 
-        ret = blk_co_pread(
-                blk, (sector_num - src_cur_offset) << BDRV_SECTOR_BITS,
-                n << BDRV_SECTOR_BITS, buf, 0);
+        ret = blk_co_pread(blk, offset, n << BDRV_SECTOR_BITS, buf, 0);
         if (ret < 0) {
-            return ret;
+            if (s->salvage) {
+                if (n > 1) {
+                    single_read_until = offset + (n << BDRV_SECTOR_BITS);
+                    continue;
+                } else {
+                    if (!s->quiet) {
+                        warn_report("error while reading offset %" PRIu64
+                                    ": %s", offset, strerror(-ret));
+                    }
+                    memset(buf, 0, BDRV_SECTOR_SIZE);
+                }
+            } else {
+                return ret;
+            }
         }
 
         sector_num += n;
@@ -2035,6 +2074,7 @@ static int img_convert(int argc, char **argv)
             {"image-opts", no_argument, 0, OPTION_IMAGE_OPTS},
             {"force-share", no_argument, 0, 'U'},
             {"target-image-opts", no_argument, 0, OPTION_TARGET_IMAGE_OPTS},
+            {"salvage", no_argument, 0, OPTION_SALVAGE},
             {0, 0, 0, 0}
         };
         c = getopt_long(argc, argv, ":hf:O:B:Cco:l:S:pt:T:qnm:WU",
@@ -2152,6 +2192,9 @@ static int img_convert(int argc, char **argv)
         case OPTION_IMAGE_OPTS:
             image_opts = true;
             break;
+        case OPTION_SALVAGE:
+            s.salvage = true;
+            break;
         case OPTION_TARGET_IMAGE_OPTS:
             tgt_image_opts = true;
             break;
@@ -2178,6 +2221,11 @@ static int img_convert(int argc, char **argv)
         goto fail_getopt;
     }
 
+    if (s.copy_range && s.salvage) {
+        error_report("Cannot use copy offloading in salvaging mode");
+        goto fail_getopt;
+    }
+
     if (tgt_image_opts && !skip_create) {
         error_report("--target-image-opts requires use of -n flag");
         goto fail_getopt;
diff --git a/qemu-img-cmds.hx b/qemu-img-cmds.hx
index 4b47f7495d..1c93e6d185 100644
--- a/qemu-img-cmds.hx
+++ b/qemu-img-cmds.hx
@@ -44,9 +44,9 @@ STEXI
 ETEXI
 
 DEF("convert", img_convert,
-    "convert [--object objectdef] [--image-opts] [--target-image-opts] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] filename [filename2 [...]] output_filename")
+    "convert [--object objectdef] [--image-opts] [--target-image-opts] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] [--salvage] filename [filename2 [...]] output_filename")
 STEXI
-@item convert [--object @var{objectdef}] [--image-opts] [--target-image-opts] [-U] [-C] [-c] [-p] [-q] [-n] [-f @var{fmt}] [-t @var{cache}] [-T @var{src_cache}] [-O @var{output_fmt}] [-B @var{backing_file}] [-o @var{options}] [-l @var{snapshot_param}] [-S @var{sparse_size}] [-m @var{num_coroutines}] [-W] @var{filename} [@var{filename2} [...]] @var{output_filename}
+@item convert [--object @var{objectdef}] [--image-opts] [--target-image-opts] [-U] [-C] [-c] [-p] [-q] [-n] [-f @var{fmt}] [-t @var{cache}] [-T @var{src_cache}] [-O @var{output_fmt}] [-B @var{backing_file}] [-o @var{options}] [-l @var{snapshot_param}] [-S @var{sparse_size}] [-m @var{num_coroutines}] [-W] [--salvage] @var{filename} [@var{filename2} [...]] @var{output_filename}
 ETEXI
 
 DEF("create", img_create,
diff --git a/qemu-img.texi b/qemu-img.texi
index e8bc0fd7a2..c8e9bba515 100644
--- a/qemu-img.texi
+++ b/qemu-img.texi
@@ -175,6 +175,10 @@ improve performance if the data is remote, such as with NFS or iSCSI backends,
 but will not automatically sparsify zero sectors, and may result in a fully
 allocated target image depending on the host support for getting allocation
 information.
+@item --salvage
+Try to ignore I/O errors when reading.  Unless in quiet mode (@code{-q}), errors
+will still be printed.  Areas that cannot be read from the source will be
+treated as containing only zeroes.
 @end table
 
 Parameters to dd subcommand:
-- 
2.21.0

[Qemu-devel] [PULL 16/20] blkdebug: Add @iotype error option

[Max Reitz]

This new error option allows users of blkdebug to inject errors only on
certain kinds of I/O operations.  Users usually want to make a very
specific operation fail, not just any; but right now they simply hope
that the event that triggers the error injection is followed up with
that very operation.  That may not be true, however, because the block
layer is changing (including blkdebug, which may increase the number of
types of I/O operations on which to inject errors).

The new option's default has been chosen to keep backwards
compatibility.

Note that similar to the internal representation, we could choose to
expose this option as a list of I/O types.  But there is no practical
use for this, because as described above, users usually know exactly
which kind of operation they want to make fail, so there is no need to
specify multiple I/O types at once.  In addition, exposing this option
as a list would require non-trivial changes to qemu_opts_absorb_qdict().

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190507203508.18026-4-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 qapi/block-core.json | 26 +++++++++++++++++++++++
 block/blkdebug.c     | 50 ++++++++++++++++++++++++++++++++++++--------
 2 files changed, 67 insertions(+), 9 deletions(-)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index c0ff3a83ef..34617a2c7a 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -3264,6 +3264,26 @@
             'l1_shrink_write_table', 'l1_shrink_free_l2_clusters',
             'cor_write', 'cluster_alloc_space'] }
 
+##
+# @BlkdebugIOType:
+#
+# Kinds of I/O that blkdebug can inject errors in.
+#
+# @read: .bdrv_co_preadv()
+#
+# @write: .bdrv_co_pwritev()
+#
+# @write-zeroes: .bdrv_co_pwrite_zeroes()
+#
+# @discard: .bdrv_co_pdiscard()
+#
+# @flush: .bdrv_co_flush_to_disk()
+#
+# Since: 4.1
+##
+{ 'enum': 'BlkdebugIOType', 'prefix': 'BLKDEBUG_IO_TYPE',
+  'data': [ 'read', 'write', 'write-zeroes', 'discard', 'flush' ] }
+
 ##
 # @BlkdebugInjectErrorOptions:
 #
@@ -3274,6 +3294,11 @@
 # @state:       the state identifier blkdebug needs to be in to
 #               actually trigger the event; defaults to "any"
 #
+# @iotype:      the type of I/O operations on which this error should
+#               be injected; defaults to "all read, write,
+#               write-zeroes, discard, and flush operations"
+#               (since: 4.1)
+#
 # @errno:       error identifier (errno) to be returned; defaults to
 #               EIO
 #
@@ -3291,6 +3316,7 @@
 { 'struct': 'BlkdebugInjectErrorOptions',
   'data': { 'event': 'BlkdebugEvent',
             '*state': 'int',
+            '*iotype': 'BlkdebugIOType',
             '*errno': 'int',
             '*sector': 'int',
             '*once': 'bool',
diff --git a/block/blkdebug.c b/block/blkdebug.c
index efd9441625..3f3ec11230 100644
--- a/block/blkdebug.c
+++ b/block/blkdebug.c
@@ -75,6 +75,7 @@ typedef struct BlkdebugRule {
     int state;
     union {
         struct {
+            uint64_t iotype_mask;
             int error;
             int immediately;
             int once;
@@ -91,6 +92,9 @@ typedef struct BlkdebugRule {
     QSIMPLEQ_ENTRY(BlkdebugRule) active_next;
 } BlkdebugRule;
 
+QEMU_BUILD_BUG_MSG(BLKDEBUG_IO_TYPE__MAX > 64,
+                   "BlkdebugIOType mask does not fit into an uint64_t");
+
 static QemuOptsList inject_error_opts = {
     .name = "inject-error",
     .head = QTAILQ_HEAD_INITIALIZER(inject_error_opts.head),
@@ -103,6 +107,10 @@ static QemuOptsList inject_error_opts = {
             .name = "state",
             .type = QEMU_OPT_NUMBER,
         },
+        {
+            .name = "iotype",
+            .type = QEMU_OPT_STRING,
+        },
         {
             .name = "errno",
             .type = QEMU_OPT_NUMBER,
@@ -162,6 +170,8 @@ static int add_rule(void *opaque, QemuOpts *opts, Error **errp)
     int event;
     struct BlkdebugRule *rule;
     int64_t sector;
+    BlkdebugIOType iotype;
+    Error *local_error = NULL;
 
     /* Find the right event for the rule */
     event_name = qemu_opt_get(opts, "event");
@@ -192,6 +202,26 @@ static int add_rule(void *opaque, QemuOpts *opts, Error **errp)
         sector = qemu_opt_get_number(opts, "sector", -1);
         rule->options.inject.offset =
             sector == -1 ? -1 : sector * BDRV_SECTOR_SIZE;
+
+        iotype = qapi_enum_parse(&BlkdebugIOType_lookup,
+                                 qemu_opt_get(opts, "iotype"),
+                                 BLKDEBUG_IO_TYPE__MAX, &local_error);
+        if (local_error) {
+            error_propagate(errp, local_error);
+            return -1;
+        }
+        if (iotype != BLKDEBUG_IO_TYPE__MAX) {
+            rule->options.inject.iotype_mask = (1ull << iotype);
+        } else {
+            /* Apply the default */
+            rule->options.inject.iotype_mask =
+                (1ull << BLKDEBUG_IO_TYPE_READ)
+                | (1ull << BLKDEBUG_IO_TYPE_WRITE)
+                | (1ull << BLKDEBUG_IO_TYPE_WRITE_ZEROES)
+                | (1ull << BLKDEBUG_IO_TYPE_DISCARD)
+                | (1ull << BLKDEBUG_IO_TYPE_FLUSH);
+        }
+
         break;
 
     case ACTION_SET_STATE:
@@ -470,7 +500,8 @@ out:
     return ret;
 }
 
-static int rule_check(BlockDriverState *bs, uint64_t offset, uint64_t bytes)
+static int rule_check(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
+                      BlkdebugIOType iotype)
 {
     BDRVBlkdebugState *s = bs->opaque;
     BlkdebugRule *rule = NULL;
@@ -480,9 +511,10 @@ static int rule_check(BlockDriverState *bs, uint64_t offset, uint64_t bytes)
     QSIMPLEQ_FOREACH(rule, &s->active_rules, active_next) {
         uint64_t inject_offset = rule->options.inject.offset;
 
-        if (inject_offset == -1 ||
-            (bytes && inject_offset >= offset &&
-             inject_offset < offset + bytes))
+        if ((inject_offset == -1 ||
+             (bytes && inject_offset >= offset &&
+              inject_offset < offset + bytes)) &&
+            (rule->options.inject.iotype_mask & (1ull << iotype)))
         {
             break;
         }
@@ -521,7 +553,7 @@ blkdebug_co_preadv(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
         assert(bytes <= bs->bl.max_transfer);
     }
 
-    err = rule_check(bs, offset, bytes);
+    err = rule_check(bs, offset, bytes, BLKDEBUG_IO_TYPE_READ);
     if (err) {
         return err;
     }
@@ -542,7 +574,7 @@ blkdebug_co_pwritev(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
         assert(bytes <= bs->bl.max_transfer);
     }
 
-    err = rule_check(bs, offset, bytes);
+    err = rule_check(bs, offset, bytes, BLKDEBUG_IO_TYPE_WRITE);
     if (err) {
         return err;
     }
@@ -552,7 +584,7 @@ blkdebug_co_pwritev(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
 
 static int blkdebug_co_flush(BlockDriverState *bs)
 {
-    int err = rule_check(bs, 0, 0);
+    int err = rule_check(bs, 0, 0, BLKDEBUG_IO_TYPE_FLUSH);
 
     if (err) {
         return err;
@@ -586,7 +618,7 @@ static int coroutine_fn blkdebug_co_pwrite_zeroes(BlockDriverState *bs,
         assert(bytes <= bs->bl.max_pwrite_zeroes);
     }
 
-    err = rule_check(bs, offset, bytes);
+    err = rule_check(bs, offset, bytes, BLKDEBUG_IO_TYPE_WRITE_ZEROES);
     if (err) {
         return err;
     }
@@ -620,7 +652,7 @@ static int coroutine_fn blkdebug_co_pdiscard(BlockDriverState *bs,
         assert(bytes <= bs->bl.max_pdiscard);
     }
 
-    err = rule_check(bs, offset, bytes);
+    err = rule_check(bs, offset, bytes, BLKDEBUG_IO_TYPE_DISCARD);
     if (err) {
         return err;
     }
-- 
2.21.0

[Qemu-devel] [PULL 17/20] blkdebug: Add "none" event

[Max Reitz]

Together with @iotypes and @sector, this can be used to trap e.g. the
first read or write access to a certain sector without having to know
what happens internally in the block layer, i.e. which "real" events
happen right before such an access.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190507203508.18026-5-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 qapi/block-core.json | 4 +++-
 block/blkdebug.c     | 2 ++
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index 34617a2c7a..60f903afa3 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -3244,6 +3244,8 @@
 #
 # @cluster_alloc_space: an allocation of file space for a cluster (since 4.1)
 #
+# @none: triggers once at creation of the blkdebug node (since 4.1)
+#
 # Since: 2.9
 ##
 { 'enum': 'BlkdebugEvent', 'prefix': 'BLKDBG',
@@ -3262,7 +3264,7 @@
             'pwritev_rmw_tail', 'pwritev_rmw_after_tail', 'pwritev',
             'pwritev_zero', 'pwritev_done', 'empty_image_prepare',
             'l1_shrink_write_table', 'l1_shrink_free_l2_clusters',
-            'cor_write', 'cluster_alloc_space'] }
+            'cor_write', 'cluster_alloc_space', 'none'] }
 
 ##
 # @BlkdebugIOType:
diff --git a/block/blkdebug.c b/block/blkdebug.c
index 3f3ec11230..1663ed25af 100644
--- a/block/blkdebug.c
+++ b/block/blkdebug.c
@@ -491,6 +491,8 @@ static int blkdebug_open(BlockDriverState *bs, QDict *options, int flags,
         goto out;
     }
 
+    bdrv_debug_event(bs, BLKDBG_NONE);
+
     ret = 0;
 out:
     if (ret < 0) {
-- 
2.21.0

[Qemu-devel] [PULL 18/20] blkdebug: Inject errors on .bdrv_co_block_status()

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190507203508.18026-6-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 qapi/block-core.json | 5 ++++-
 block/blkdebug.c     | 8 ++++++++
 2 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index 60f903afa3..61124431d8 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -3281,10 +3281,13 @@
 #
 # @flush: .bdrv_co_flush_to_disk()
 #
+# @block-status: .bdrv_co_block_status()
+#
 # Since: 4.1
 ##
 { 'enum': 'BlkdebugIOType', 'prefix': 'BLKDEBUG_IO_TYPE',
-  'data': [ 'read', 'write', 'write-zeroes', 'discard', 'flush' ] }
+  'data': [ 'read', 'write', 'write-zeroes', 'discard', 'flush',
+            'block-status' ] }
 
 ##
 # @BlkdebugInjectErrorOptions:
diff --git a/block/blkdebug.c b/block/blkdebug.c
index 1663ed25af..5ae96c52b0 100644
--- a/block/blkdebug.c
+++ b/block/blkdebug.c
@@ -670,7 +670,15 @@ static int coroutine_fn blkdebug_co_block_status(BlockDriverState *bs,
                                                  int64_t *map,
                                                  BlockDriverState **file)
 {
+    int err;
+
     assert(QEMU_IS_ALIGNED(offset | bytes, bs->bl.request_alignment));
+
+    err = rule_check(bs, offset, bytes, BLKDEBUG_IO_TYPE_BLOCK_STATUS);
+    if (err) {
+        return err;
+    }
+
     return bdrv_co_block_status_from_file(bs, want_zero, offset, bytes,
                                           pnum, map, file);
 }
-- 
2.21.0

[Qemu-devel] [PULL 19/20] iotests: Test qemu-img convert --salvage

[Max Reitz]

This test converts a simple image to another, but blkdebug injects
block_status and read faults at some offsets.  The resulting image
should be the same as the input image, except that sectors that could
not be read have to be 0.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190507203508.18026-7-mreitz@redhat.com
Tested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
[mreitz: Dropped superfluous printf from _filter_offsets, as suggested
         by Vladimir; disable test for VDI and IMGOPTSSYNTAX]
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/251     | 170 +++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/251.out |  43 ++++++++++
 tests/qemu-iotests/group   |   1 +
 3 files changed, 214 insertions(+)
 create mode 100755 tests/qemu-iotests/251
 create mode 100644 tests/qemu-iotests/251.out

diff --git a/tests/qemu-iotests/251 b/tests/qemu-iotests/251
new file mode 100755
index 0000000000..13f85de9cd
--- /dev/null
+++ b/tests/qemu-iotests/251
@@ -0,0 +1,170 @@
+#!/usr/bin/env bash
+#
+# Test qemu-img convert --salvage
+#
+# Copyright (C) 2019 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
+owner=mreitz@redhat.com
+
+seq=$(basename $0)
+echo "QA output created by $seq"
+
+status=1	# failure is the default!
+
+_cleanup()
+{
+    _cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+. ./common.qemu
+
+_supported_fmt generic
+_supported_proto file
+_supported_os Linux
+
+if [ "$IMGOPTSSYNTAX" = "true" ]; then
+    # We use json:{} filenames here, so we cannot work with additional options.
+    _unsupported_fmt $IMGFMT
+else
+    # With VDI, the output is ordered differently.  Just disable it.
+    _unsupported_fmt vdi
+fi
+
+
+TEST_IMG="$TEST_IMG.orig" _make_test_img 64M
+
+$QEMU_IO -c 'write -P 42 0 64M' "$TEST_IMG.orig" | _filter_qemu_io
+
+
+sector_size=512
+
+# Offsets on which to fail block-status.  Keep in ascending order so
+# the indexing done by _filter_offsets will appear in ascending order
+# in the output as well.
+status_fail_offsets="$((16 * 1024 * 1024 + 8192))
+                     $((33 * 1024 * 1024 + 512))"
+
+# Offsets on which to fail reads.  Keep in ascending order for the
+# same reason.
+# The second element is shared with $status_fail_offsets on purpose.
+# Starting with the third element, we test what happens when a
+# continuous range of sectors is inaccessible.
+read_fail_offsets="$((32 * 1024 * 1024 - 65536))
+                   $((33 * 1024 * 1024 + 512))
+                   $(seq $((34 * 1024 * 1024)) $sector_size \
+                         $((34 * 1024 * 1024 + 4096 - $sector_size)))"
+
+
+# blkdebug must be above the format layer so it can intercept all
+# block-status events
+source_img="json:{'driver': 'blkdebug',
+                  'image': {
+                      'driver': '$IMGFMT',
+                      'file': {
+                          'driver': 'file',
+                          'filename': '$TEST_IMG.orig'
+                      }
+                  },
+                  'inject-error': ["
+
+for ofs in $status_fail_offsets
+do
+    source_img+="{ 'event': 'none',
+                   'iotype': 'block-status',
+                   'errno': 5,
+                   'sector': $((ofs / sector_size)) },"
+done
+
+for ofs in $read_fail_offsets
+do
+    source_img+="{ 'event': 'none',
+                   'iotype': 'read',
+                   'errno': 5,
+                   'sector': $((ofs / sector_size)) },"
+done
+
+# Remove the trailing comma and terminate @inject-error and json:{}
+source_img="${source_img%,} ] }"
+
+
+echo
+
+
+_filter_offsets() {
+    filters=
+
+    index=0
+    for ofs in $1
+    do
+        filters+=" -e s/$ofs/status_fail_offset_$index/"
+        index=$((index + 1))
+    done
+
+    index=0
+    for ofs in $2
+    do
+        filters+=" -e s/$ofs/read_fail_offset_$index/"
+        index=$((index + 1))
+    done
+
+    sed $filters
+}
+
+# While determining the number of allocated sectors in the input
+# image, we should see one block status warning per element of
+# $status_fail_offsets.
+#
+# Then, the image is read.  Since the block status is queried in
+# basically the same way, the same warnings as in the previous step
+# should reappear.  Interleaved with those we should see a read
+# warning per element of $read_fail_offsets.
+# Note that $read_fail_offsets and $status_fail_offsets share an
+# element (read_fail_offset_1 == status_fail_offset_1), so
+# "status_fail_offset_1" in the output is the same as
+# "read_fail_offset_1".
+$QEMU_IMG convert --salvage "$source_img" "$TEST_IMG" 2>&1 \
+    | _filter_offsets "$status_fail_offsets" "$read_fail_offsets"
+
+echo
+
+# The offsets where the block status could not be determined should
+# have been treated as containing data and thus should be correct in
+# the output image.
+# The offsets where reading failed altogether should be 0.  Make them
+# 0 in the input image, too, so we can compare both images.
+for ofs in $read_fail_offsets
+do
+    $QEMU_IO -c "write -z $ofs $sector_size" "$TEST_IMG.orig" \
+        | _filter_qemu_io \
+        | _filter_offsets '' "$read_fail_offsets"
+done
+
+echo
+
+# These should be equal now.
+$QEMU_IMG compare "$TEST_IMG.orig" "$TEST_IMG"
+
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
diff --git a/tests/qemu-iotests/251.out b/tests/qemu-iotests/251.out
new file mode 100644
index 0000000000..75b8796aad
--- /dev/null
+++ b/tests/qemu-iotests/251.out
@@ -0,0 +1,43 @@
+QA output created by 251
+Formatting 'TEST_DIR/t.IMGFMT.orig', fmt=IMGFMT size=67108864
+wrote 67108864/67108864 bytes at offset 0
+64 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+qemu-img: warning: error while reading block status at offset status_fail_offset_0: Input/output error
+qemu-img: warning: error while reading block status at offset status_fail_offset_1: Input/output error
+qemu-img: warning: error while reading block status at offset status_fail_offset_0: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_0: Input/output error
+qemu-img: warning: error while reading block status at offset status_fail_offset_1: Input/output error
+qemu-img: warning: error while reading offset status_fail_offset_1: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_2: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_3: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_4: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_5: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_6: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_7: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_8: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_9: Input/output error
+
+wrote 512/512 bytes at offset read_fail_offset_0
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_1
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_2
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_3
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_4
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_5
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_6
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_7
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_8
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_9
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+Images are identical.
+*** done
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index 0842167b7b..b34c8e3c0c 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -263,6 +263,7 @@
 248 rw quick
 249 rw auto quick
 250 rw auto quick
+251 rw auto quick
 252 rw auto backing quick
 253 rw auto quick
 254 rw auto backing quick
-- 
2.21.0

[Qemu-devel] [PULL 20/20] iotests: Test qemu-img convert -C --salvage

[Max Reitz]

We do not support this combination (yet), so this should yield an error
message.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Tested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190507203508.18026-8-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/082     | 1 +
 tests/qemu-iotests/082.out | 3 +++
 2 files changed, 4 insertions(+)

diff --git a/tests/qemu-iotests/082 b/tests/qemu-iotests/082
index bbbdf555dc..3286c2c6db 100755
--- a/tests/qemu-iotests/082
+++ b/tests/qemu-iotests/082
@@ -162,6 +162,7 @@ echo === convert: -C and other options ===
 run_qemu_img convert -C -S 4k -O $IMGFMT "$TEST_IMG" "$TEST_IMG".target
 run_qemu_img convert -C -S 8k -O $IMGFMT "$TEST_IMG" "$TEST_IMG".target
 run_qemu_img convert -C -c -O $IMGFMT "$TEST_IMG" "$TEST_IMG".target
+run_qemu_img convert -C --salvage -O $IMGFMT "$TEST_IMG" "$TEST_IMG".target
 
 echo
 echo === amend: Options specified more than once ===
diff --git a/tests/qemu-iotests/082.out b/tests/qemu-iotests/082.out
index 7e25706813..58de358b38 100644
--- a/tests/qemu-iotests/082.out
+++ b/tests/qemu-iotests/082.out
@@ -567,6 +567,9 @@ qemu-img: Cannot enable copy offloading when -S is used
 Testing: convert -C -c -O qcow2 TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.target
 qemu-img: Cannot enable copy offloading when -c is used
 
+Testing: convert -C --salvage -O qcow2 TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.target
+qemu-img: Cannot use copy offloading in salvaging mode
+
 === amend: Options specified more than once ===
 
 Testing: amend -f foo -f qcow2 -o lazy_refcounts=on TEST_DIR/t.qcow2
-- 
2.21.0

Re: [Qemu-devel] [PULL 00/20] Block patches

[Peter Maydell]

On Fri, 14 Jun 2019 at 14:40, Max Reitz <mreitz@redhat.com> wrote:
>
> The following changes since commit 5ec2eca83dc478ddf24077e02a8b34dd26cd3ff9:
>
>   Merge remote-tracking branch 'remotes/awilliam/tags/vfio-updates-20190613.0' into staging (2019-06-14 09:33:55 +0100)
>
> are available in the Git repository at:
>
>   https://github.com/XanClic/qemu.git tags/pull-block-2019-06-14
>
> for you to fetch changes up to 21c1ce592a144188dfe59b9e156a97da412a59a2:
>
>   iotests: Test qemu-img convert -C --salvage (2019-06-14 15:09:42 +0200)
>
> ----------------------------------------------------------------
> Block patches:
> - Allow blockdev-backup from nodes that are not in qemu's main AIO
>   context to newly added nodes
> - Add salvaging mode to qemu-img convert
> - Minor fixes to tests, documentation, and for less Valgrind annoyance
>

Applied, thanks.

Please update the changelog at https://wiki.qemu.org/ChangeLog/4.1
for any user-visible changes.

-- PMM