[Qemu-devel] [PULL 0/4] Block patches

[Qemu-devel] [PULL 0/4] Block patches

[Stefan Hajnoczi]

The following changes since commit 2b483739791b33c46e6084b51edcf62107058ae1:

  Merge remote-tracking branch 'remotes/pmaydell/tags/pull-target-arm-20170904-2' into staging (2017-09-04 17:21:24 +0100)

are available in the git repository at:

  git://github.com/stefanha/qemu.git tags/block-pull-request

for you to fetch changes up to b461151ff31c7925f271c297e8abed20231ac7d3:

  block: document semantics of bdrv_co_preadv|pwritev (2017-09-05 11:07:02 +0100)

----------------------------------------------------------------

----------------------------------------------------------------

Daniel P. Berrange (1):
  block: document semantics of bdrv_co_preadv|pwritev

Stefan Hajnoczi (3):
  qemu.py: make VM() a context manager
  iotests.py: add FilePath context manager
  qemu-iotests: use context managers for resource cleanup in 194

 include/block/block_int.h     |  31 +++++++++++
 scripts/qemu.py               |  16 +++++-
 tests/qemu-iotests/194        | 117 +++++++++++++++++++++---------------------
 tests/qemu-iotests/iotests.py |  26 ++++++++++
 4 files changed, 130 insertions(+), 60 deletions(-)

-- 
2.13.5

[Qemu-devel] [PULL 1/4] qemu.py: make VM() a context manager

[Stefan Hajnoczi]

There are a number of ways to ensure that the QEMU process is shut down
when the test ends, including atexit.register(), try: finally:, or
unittest.teardown() methods.  All of these require extra code and the
programmer must remember to add vm.shutdown().

A nice solution is context managers:

  with VM(binary) as vm:
      ...
  # vm is guaranteed to be shut down here

Cc: Eduardo Habkost <ehabkost@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Message-id: 20170824072202.26818-2-stefanha@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 scripts/qemu.py | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/scripts/qemu.py b/scripts/qemu.py
index 880e3e8219..4d8ee10943 100644
--- a/scripts/qemu.py
+++ b/scripts/qemu.py
@@ -21,7 +21,14 @@ import qmp.qmp
 
 
 class QEMUMachine(object):
-    '''A QEMU VM'''
+    '''A QEMU VM
+
+    Use this object as a context manager to ensure the QEMU process terminates::
+
+        with VM(binary) as vm:
+            ...
+        # vm is guaranteed to be shut down here
+    '''
 
     def __init__(self, binary, args=[], wrapper=[], name=None, test_dir="/var/tmp",
                  monitor_address=None, socket_scm_helper=None, debug=False):
@@ -40,6 +47,13 @@ class QEMUMachine(object):
         self._socket_scm_helper = socket_scm_helper
         self._debug = debug
 
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        self.shutdown()
+        return False
+
     # This can be used to add an unused monitor instance.
     def add_monitor_telnet(self, ip, port):
         args = 'tcp:%s:%d,server,nowait,telnet' % (ip, port)
-- 
2.13.5

[Qemu-devel] [PULL 2/4] iotests.py: add FilePath context manager

[Stefan Hajnoczi]

The scratch/ (TEST_DIR) directory is not automatically cleaned up after
test execution.  It is the responsibility of tests to remove any files
they create.

A nice way of doing this is to declare files at the beginning of the
test and automatically remove them with a context manager:

  with iotests.FilePath('test.img') as img_path:
      qemu_img(...)
      qemu_io(...)
  # img_path is guaranteed to be deleted here

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: 20170824072202.26818-3-stefanha@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 tests/qemu-iotests/iotests.py | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 7233983f3c..07fa1626a0 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -160,6 +160,32 @@ class Timeout:
     def timeout(self, signum, frame):
         raise Exception(self.errmsg)
 
+
+class FilePath(object):
+    '''An auto-generated filename that cleans itself up.
+
+    Use this context manager to generate filenames and ensure that the file
+    gets deleted::
+
+        with TestFilePath('test.img') as img_path:
+            qemu_img('create', img_path, '1G')
+        # migration_sock_path is automatically deleted
+    '''
+    def __init__(self, name):
+        filename = '{0}-{1}'.format(os.getpid(), name)
+        self.path = os.path.join(test_dir, filename)
+
+    def __enter__(self):
+        return self.path
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        try:
+            os.remove(self.path)
+        except OSError:
+            pass
+        return False
+
+
 class VM(qtest.QEMUQtestMachine):
     '''A QEMU VM'''
 
-- 
2.13.5

[Qemu-devel] [PULL 3/4] qemu-iotests: use context managers for resource cleanup in 194

[Stefan Hajnoczi]

Switch from atexit.register() to a more elegant idiom of declaring
resources in a with statement:

  with FilePath('monitor.sock') as monitor_path,
       VM() as vm:
      ...

The files and VMs will be automatically cleaned up whether the test
passes or fails.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: 20170824072202.26818-4-stefanha@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 tests/qemu-iotests/194 | 117 ++++++++++++++++++++++++-------------------------
 1 file changed, 58 insertions(+), 59 deletions(-)

diff --git a/tests/qemu-iotests/194 b/tests/qemu-iotests/194
index a3e3bad664..6449b9b64a 100755
--- a/tests/qemu-iotests/194
+++ b/tests/qemu-iotests/194
@@ -19,66 +19,65 @@
 #
 # Non-shared storage migration test using NBD server and drive-mirror
 
-import os
-import atexit
 import iotests
 
 iotests.verify_platform(['linux'])
 
-img_size = '1G'
-source_img_path = os.path.join(iotests.test_dir, 'source.img')
-dest_img_path = os.path.join(iotests.test_dir, 'dest.img')
-iotests.qemu_img_pipe('create', '-f', iotests.imgfmt, source_img_path, img_size)
-iotests.qemu_img_pipe('create', '-f', iotests.imgfmt, dest_img_path, img_size)
-
-iotests.log('Launching VMs...')
-migration_sock_path = os.path.join(iotests.test_dir, 'migration.sock')
-nbd_sock_path = os.path.join(iotests.test_dir, 'nbd.sock')
-source_vm = iotests.VM('source').add_drive(source_img_path)
-dest_vm = (iotests.VM('dest').add_drive(dest_img_path)
-                             .add_incoming('unix:{0}'.format(migration_sock_path)))
-source_vm.launch()
-atexit.register(source_vm.shutdown)
-dest_vm.launch()
-atexit.register(dest_vm.shutdown)
-
-iotests.log('Launching NBD server on destination...')
-iotests.log(dest_vm.qmp('nbd-server-start', addr={'type': 'unix', 'data': {'path': nbd_sock_path}}))
-iotests.log(dest_vm.qmp('nbd-server-add', device='drive0', writable=True))
-
-iotests.log('Starting `drive-mirror` on source...')
-iotests.log(source_vm.qmp(
-              'drive-mirror',
-              device='drive0',
-              target='nbd+unix:///drive0?socket={0}'.format(nbd_sock_path),
-              sync='full',
-              format='raw', # always raw, the server handles the format
-              mode='existing',
-              job_id='mirror-job0'))
-
-iotests.log('Waiting for `drive-mirror` to complete...')
-iotests.log(source_vm.event_wait('BLOCK_JOB_READY'),
-            filters=[iotests.filter_qmp_event])
-
-iotests.log('Starting migration...')
-source_vm.qmp('migrate-set-capabilities',
-              capabilities=[{'capability': 'events', 'state': True}])
-dest_vm.qmp('migrate-set-capabilities',
-            capabilities=[{'capability': 'events', 'state': True}])
-iotests.log(source_vm.qmp('migrate', uri='unix:{0}'.format(migration_sock_path)))
-
-while True:
-    event1 = source_vm.event_wait('MIGRATION')
-    iotests.log(event1, filters=[iotests.filter_qmp_event])
-    if event1['data']['status'] in ('completed', 'failed'):
-        iotests.log('Gracefully ending the `drive-mirror` job on source...')
-        iotests.log(source_vm.qmp('block-job-cancel', device='mirror-job0'))
-        break
-
-while True:
-    event2 = source_vm.event_wait('BLOCK_JOB_COMPLETED')
-    iotests.log(event2, filters=[iotests.filter_qmp_event])
-    if event2['event'] == 'BLOCK_JOB_COMPLETED':
-        iotests.log('Stopping the NBD server on destination...')
-        iotests.log(dest_vm.qmp('nbd-server-stop'))
-        break
+with iotests.FilePath('source.img') as source_img_path, \
+     iotests.FilePath('dest.img') as dest_img_path, \
+     iotests.FilePath('migration.sock') as migration_sock_path, \
+     iotests.FilePath('nbd.sock') as nbd_sock_path, \
+     iotests.VM('source') as source_vm, \
+     iotests.VM('dest') as dest_vm:
+
+    img_size = '1G'
+    iotests.qemu_img_pipe('create', '-f', iotests.imgfmt, source_img_path, img_size)
+    iotests.qemu_img_pipe('create', '-f', iotests.imgfmt, dest_img_path, img_size)
+
+    iotests.log('Launching VMs...')
+    (source_vm.add_drive(source_img_path)
+              .launch())
+    (dest_vm.add_drive(dest_img_path)
+            .add_incoming('unix:{0}'.format(migration_sock_path))
+            .launch())
+
+    iotests.log('Launching NBD server on destination...')
+    iotests.log(dest_vm.qmp('nbd-server-start', addr={'type': 'unix', 'data': {'path': nbd_sock_path}}))
+    iotests.log(dest_vm.qmp('nbd-server-add', device='drive0', writable=True))
+
+    iotests.log('Starting `drive-mirror` on source...')
+    iotests.log(source_vm.qmp(
+                  'drive-mirror',
+                  device='drive0',
+                  target='nbd+unix:///drive0?socket={0}'.format(nbd_sock_path),
+                  sync='full',
+                  format='raw', # always raw, the server handles the format
+                  mode='existing',
+                  job_id='mirror-job0'))
+
+    iotests.log('Waiting for `drive-mirror` to complete...')
+    iotests.log(source_vm.event_wait('BLOCK_JOB_READY'),
+                filters=[iotests.filter_qmp_event])
+
+    iotests.log('Starting migration...')
+    source_vm.qmp('migrate-set-capabilities',
+                  capabilities=[{'capability': 'events', 'state': True}])
+    dest_vm.qmp('migrate-set-capabilities',
+                capabilities=[{'capability': 'events', 'state': True}])
+    iotests.log(source_vm.qmp('migrate', uri='unix:{0}'.format(migration_sock_path)))
+
+    while True:
+        event1 = source_vm.event_wait('MIGRATION')
+        iotests.log(event1, filters=[iotests.filter_qmp_event])
+        if event1['data']['status'] in ('completed', 'failed'):
+            iotests.log('Gracefully ending the `drive-mirror` job on source...')
+            iotests.log(source_vm.qmp('block-job-cancel', device='mirror-job0'))
+            break
+
+    while True:
+        event2 = source_vm.event_wait('BLOCK_JOB_COMPLETED')
+        iotests.log(event2, filters=[iotests.filter_qmp_event])
+        if event2['event'] == 'BLOCK_JOB_COMPLETED':
+            iotests.log('Stopping the NBD server on destination...')
+            iotests.log(dest_vm.qmp('nbd-server-stop'))
+            break
-- 
2.13.5

[Qemu-devel] [PULL 4/4] block: document semantics of bdrv_co_preadv|pwritev

[Stefan Hajnoczi]

From: "Daniel P. Berrange" <berrange@redhat.com>

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Message-id: 20170831105456.9558-1-berrange@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 include/block/block_int.h | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/include/block/block_int.h b/include/block/block_int.h
index 7571c0aaaf..7816b43a27 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -146,12 +146,43 @@ struct BlockDriver {
 
     int coroutine_fn (*bdrv_co_readv)(BlockDriverState *bs,
         int64_t sector_num, int nb_sectors, QEMUIOVector *qiov);
+
+    /**
+     * @offset: position in bytes to read at
+     * @bytes: number of bytes to read
+     * @qiov: the buffers to fill with read data
+     * @flags: currently unused, always 0
+     *
+     * @offset and @bytes will be a multiple of 'request_alignment',
+     * but the length of individual @qiov elements does not have to
+     * be a multiple.
+     *
+     * @bytes will always equal the total size of @qiov, and will be
+     * no larger than 'max_transfer'.
+     *
+     * The buffer in @qiov may point directly to guest memory.
+     */
     int coroutine_fn (*bdrv_co_preadv)(BlockDriverState *bs,
         uint64_t offset, uint64_t bytes, QEMUIOVector *qiov, int flags);
     int coroutine_fn (*bdrv_co_writev)(BlockDriverState *bs,
         int64_t sector_num, int nb_sectors, QEMUIOVector *qiov);
     int coroutine_fn (*bdrv_co_writev_flags)(BlockDriverState *bs,
         int64_t sector_num, int nb_sectors, QEMUIOVector *qiov, int flags);
+    /**
+     * @offset: position in bytes to write at
+     * @bytes: number of bytes to write
+     * @qiov: the buffers containing data to write
+     * @flags: zero or more bits allowed by 'supported_write_flags'
+     *
+     * @offset and @bytes will be a multiple of 'request_alignment',
+     * but the length of individual @qiov elements does not have to
+     * be a multiple.
+     *
+     * @bytes will always equal the total size of @qiov, and will be
+     * no larger than 'max_transfer'.
+     *
+     * The buffer in @qiov may point directly to guest memory.
+     */
     int coroutine_fn (*bdrv_co_pwritev)(BlockDriverState *bs,
         uint64_t offset, uint64_t bytes, QEMUIOVector *qiov, int flags);
 
-- 
2.13.5

Re: [Qemu-devel] [PULL 0/4] Block patches

[Peter Maydell]

On 5 September 2017 at 14:29, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> The following changes since commit 2b483739791b33c46e6084b51edcf62107058ae1:
>
>   Merge remote-tracking branch 'remotes/pmaydell/tags/pull-target-arm-20170904-2' into staging (2017-09-04 17:21:24 +0100)
>
> are available in the git repository at:
>
>   git://github.com/stefanha/qemu.git tags/block-pull-request
>
> for you to fetch changes up to b461151ff31c7925f271c297e8abed20231ac7d3:
>
>   block: document semantics of bdrv_co_preadv|pwritev (2017-09-05 11:07:02 +0100)
>
> ----------------------------------------------------------------
>
> ----------------------------------------------------------------
>
> Daniel P. Berrange (1):
>   block: document semantics of bdrv_co_preadv|pwritev
>
> Stefan Hajnoczi (3):
>   qemu.py: make VM() a context manager
>   iotests.py: add FilePath context manager
>   qemu-iotests: use context managers for resource cleanup in 194
>
>  include/block/block_int.h     |  31 +++++++++++
>  scripts/qemu.py               |  16 +++++-
>  tests/qemu-iotests/194        | 117 +++++++++++++++++++++---------------------
>  tests/qemu-iotests/iotests.py |  26 ++++++++++
>  4 files changed, 130 insertions(+), 60 deletions(-)

Applied, thanks.

-- PMM