[Qemu-devel] [PATCH 0/4] Image Streaming API

[Qemu-devel] [PATCH 0/4] Image Streaming API

[Stefan Hajnoczi]

These patches put in place the image streaming QMP/HMP commands and
documentation.  Image streaming itself is not implemented by this patch series
but the HMP/QMP commands that libvirt uses are implemented to return
NotSupported.

The Image Streaming API can be used to copy the contents of a backing file into
the image file while the guest is running.  The API is described on the wiki:
http://wiki.qemu.org/Features/LiveBlockMigration/ImageStreamingAPI

The point of this series is to commit QEMU to the API that we have worked out
with libvirt.  The QED Image Streaming series that I posted earlier provides an
implementation for the QED image format only.  I am currently working on a
generic block layer implementation so that any format with backing file support
can do image streaming.

For reference, the QED-specific implementation lives here:
http://repo.or.cz/w/qemu/stefanha.git/shortlog/refs/heads/stream-command

Stefan Hajnoczi (4):
  qmp: add block_stream command
  qmp: add block_job_set_speed command
  qmp: add block_job_cancel command
  qmp: add query-block-jobs

 blockdev.c      |   55 ++++++++++++++++++
 blockdev.h      |    6 ++
 hmp-commands.hx |   44 ++++++++++++++
 monitor.c       |   19 ++++++
 monitor.h       |    1 +
 qerror.h        |    3 +
 qmp-commands.hx |  172 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 7 files changed, 300 insertions(+), 0 deletions(-)

-- 
1.7.5.4

[Qemu-devel] [PATCH 1/4] qmp: add block_stream command

[Stefan Hajnoczi]

This patch introduces the block_stream HMP/QMP command.  It is currently
unimplemented and returns the 'NotSupported' error.

block_stream
------------

Copy data from a backing file into a block device.

The block streaming operation is performed in the background until the
entire backing file has been copied.  This command returns immediately
once streaming has started.  The status of ongoing block streaming
operations can be checked with query-block-jobs.  The operation can be
stopped before it has completed using the block_job_cancel command.

If a base file is specified then sectors are not copied from that base
file and its backing chain.  When streaming completes the image file
will have the base file as its backing file.  This can be used to stream
a subset of the backing file chain instead of flattening the entire
image.

On successful completion the image file is updated to drop the backing
file.

Arguments:

- device: device name (json-string)
- base:   common backing file (json-string, optional)

Errors:

DeviceInUse:    streaming is already active on this device
DeviceNotFound: device name is invalid
NotSupported:   image streaming is not supported by this device

Events:

On completion the BLOCK_JOB_COMPLETED event is raised with the following
fields:

- type:     job type ("stream" for image streaming, json-string)
- device:   device name (json-string)
- end:      maximum progress value (json-int)
- position: current progress value (json-int)
- speed:    rate limit, bytes per second (json-int)
- error:    error message (json-string, only on error)

The completion event is raised both on success and on failure.  On
success position is equal to end.  On failure position and end can be
used to indicate at which point the operation failed.

On failure the error field contains a human-readable error message.
There are no semantics other than that streaming has failed and clients
should not try to interpret the error string.

Examples:

-> { "execute": "block_stream", "arguments": { "device": "virtio0" } }
<- { "return":  {} }

Signed-off-by: Stefan Hajnoczi <stefanha@linux.vnet.ibm.com>
---
 blockdev.c      |   18 +++++++++++++++
 blockdev.h      |    1 +
 hmp-commands.hx |   14 +++++++++++
 monitor.c       |    3 ++
 monitor.h       |    1 +
 qerror.h        |    3 ++
 qmp-commands.hx |   65 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 7 files changed, 105 insertions(+), 0 deletions(-)

diff --git a/blockdev.c b/blockdev.c
index d272659..208bfc9 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -790,3 +790,21 @@ int do_block_resize(Monitor *mon, const QDict *qdict, QObject **ret_data)
 
     return 0;
 }
+
+int do_block_stream(Monitor *mon, const QDict *params, QObject **ret_data)
+{
+    const char *device = qdict_get_str(params, "device");
+    BlockDriverState *bs;
+
+    bs = bdrv_find(device);
+    if (!bs) {
+        qerror_report(QERR_DEVICE_NOT_FOUND, device);
+        return -1;
+    }
+
+    /* This command is not yet implemented.  The device not found check above
+     * is done so that error ordering will not change when fully implemented.
+     */
+    qerror_report(QERR_NOT_SUPPORTED);
+    return -1;
+}
diff --git a/blockdev.h b/blockdev.h
index 3587786..ad98d37 100644
--- a/blockdev.h
+++ b/blockdev.h
@@ -65,5 +65,6 @@ int do_change_block(Monitor *mon, const char *device,
 int do_drive_del(Monitor *mon, const QDict *qdict, QObject **ret_data);
 int do_snapshot_blkdev(Monitor *mon, const QDict *qdict, QObject **ret_data);
 int do_block_resize(Monitor *mon, const QDict *qdict, QObject **ret_data);
+int do_block_stream(Monitor *mon, const QDict *qdict, QObject **ret_data);
 
 #endif
diff --git a/hmp-commands.hx b/hmp-commands.hx
index 0ccfb28..2a16fd9 100644
--- a/hmp-commands.hx
+++ b/hmp-commands.hx
@@ -70,6 +70,20 @@ but should be used with extreme caution.  Note that this command only
 resizes image files, it can not resize block devices like LVM volumes.
 ETEXI
 
+    {
+        .name       = "block_stream",
+        .args_type  = "device:B,base:s?",
+        .params     = "device [base]",
+        .help       = "copy data from a backing file into a block device",
+        .user_print = monitor_user_noop,
+        .mhandler.cmd_new = do_block_stream,
+    },
+
+STEXI
+@item block_stream
+@findex block_stream
+Copy data from a backing file into a block device.
+ETEXI
 
     {
         .name       = "eject",
diff --git a/monitor.c b/monitor.c
index ada51d0..dc55fca 100644
--- a/monitor.c
+++ b/monitor.c
@@ -468,6 +468,9 @@ void monitor_protocol_event(MonitorEvent event, QObject *data)
         case QEVENT_SPICE_DISCONNECTED:
             event_name = "SPICE_DISCONNECTED";
             break;
+        case QEVENT_BLOCK_JOB_COMPLETED:
+            event_name = "BLOCK_JOB_COMPLETED";
+            break;
         default:
             abort();
             break;
diff --git a/monitor.h b/monitor.h
index 4f2d328..135c927 100644
--- a/monitor.h
+++ b/monitor.h
@@ -35,6 +35,7 @@ typedef enum MonitorEvent {
     QEVENT_SPICE_CONNECTED,
     QEVENT_SPICE_INITIALIZED,
     QEVENT_SPICE_DISCONNECTED,
+    QEVENT_BLOCK_JOB_COMPLETED,
     QEVENT_MAX,
 } MonitorEvent;
 
diff --git a/qerror.h b/qerror.h
index 8058456..eba9238 100644
--- a/qerror.h
+++ b/qerror.h
@@ -139,6 +139,9 @@ QError *qobject_to_qerror(const QObject *obj);
 #define QERR_NO_BUS_FOR_DEVICE \
     "{ 'class': 'NoBusForDevice', 'data': { 'device': %s, 'bus': %s } }"
 
+#define QERR_NOT_SUPPORTED \
+    "{ 'class': 'NotSupported', 'data': {} }"
+
 #define QERR_OPEN_FILE_FAILED \
     "{ 'class': 'OpenFileFailed', 'data': { 'filename': %s } }"
 
diff --git a/qmp-commands.hx b/qmp-commands.hx
index 03f67da..60c9bdf 100644
--- a/qmp-commands.hx
+++ b/qmp-commands.hx
@@ -694,6 +694,71 @@ Example:
 EQMP
 
     {
+        .name       = "block_stream",
+        .args_type  = "device:B,base:s?",
+        .params     = "device [base]",
+        .user_print = monitor_user_noop,
+        .mhandler.cmd_new = do_block_stream,
+    },
+
+SQMP
+block_stream
+------------
+
+Copy data from a backing file into a block device.
+
+The block streaming operation is performed in the background until the entire
+backing file has been copied.  This command returns immediately once streaming
+has started.  The status of ongoing block streaming operations can be checked
+with query-block-jobs.  The operation can be stopped before it has completed
+using the block_job_cancel command.
+
+If a base file is specified then sectors are not copied from that base file and
+its backing chain.  When streaming completes the image file will have the base
+file as its backing file.  This can be used to stream a subset of the backing
+file chain instead of flattening the entire image.
+
+On successful completion the image file is updated to drop the backing file.
+
+Arguments:
+
+- device: device name (json-string)
+- base:   common backing file (json-string, optional)
+
+Errors:
+
+DeviceInUse:    streaming is already active on this device
+DeviceNotFound: device name is invalid
+NotSupported:   image streaming is not supported by this device
+
+Events:
+
+On completion the BLOCK_JOB_COMPLETED event is raised with the following
+fields:
+
+- type:     job type ("stream" for image streaming, json-string)
+- device:   device name (json-string)
+- end:      maximum progress value (json-int)
+- position: current progress value (json-int)
+- speed:    rate limit, bytes per second (json-int)
+- error:    error message (json-string, only on error)
+
+The completion event is raised both on success and on failure.  On
+success position is equal to end.  On failure position and end can be
+used to indicate at which point the operation failed.
+
+On failure the error field contains a human-readable error message.  There are
+no semantics other than that streaming has failed and clients should not try
+to interpret the error string.
+
+Examples:
+
+-> { "execute": "block_stream", "arguments": { "device": "virtio0" } }
+<- { "return":  {} }
+
+EQMP
+
+    {
         .name       = "blockdev-snapshot-sync",
         .args_type  = "device:B,snapshot-file:s?,format:s?",
         .params     = "device [new-image-file] [format]",
-- 
1.7.5.4

Re: [Qemu-devel] [PATCH 1/4] qmp: add block_stream command

[Adam Litke]

Under libvirt, I get the following error when trying to start a block
stream:

qerror: bad call in function 'do_block_stream':
qerror: -> error format '{ 'class': 'NotSupported', 'data': {} }' not found
qerror: call at blockdev.c:808
2011-08-23 11:30:09.974: shutting down

Is this patch missing a part of the qerror definition?

On Tue, 2011-08-23 at 13:58 +0100, Stefan Hajnoczi wrote:
> This patch introduces the block_stream HMP/QMP command.  It is currently
> unimplemented and returns the 'NotSupported' error.
> 
> block_stream
> ------------
> 
> Copy data from a backing file into a block device.
> 
> The block streaming operation is performed in the background until the
> entire backing file has been copied.  This command returns immediately
> once streaming has started.  The status of ongoing block streaming
> operations can be checked with query-block-jobs.  The operation can be
> stopped before it has completed using the block_job_cancel command.
> 
> If a base file is specified then sectors are not copied from that base
> file and its backing chain.  When streaming completes the image file
> will have the base file as its backing file.  This can be used to stream
> a subset of the backing file chain instead of flattening the entire
> image.
> 
> On successful completion the image file is updated to drop the backing
> file.
> 
> Arguments:
> 
> - device: device name (json-string)
> - base:   common backing file (json-string, optional)
> 
> Errors:
> 
> DeviceInUse:    streaming is already active on this device
> DeviceNotFound: device name is invalid
> NotSupported:   image streaming is not supported by this device
> 
> Events:
> 
> On completion the BLOCK_JOB_COMPLETED event is raised with the following
> fields:
> 
> - type:     job type ("stream" for image streaming, json-string)
> - device:   device name (json-string)
> - end:      maximum progress value (json-int)
> - position: current progress value (json-int)
> - speed:    rate limit, bytes per second (json-int)
> - error:    error message (json-string, only on error)
> 
> The completion event is raised both on success and on failure.  On
> success position is equal to end.  On failure position and end can be
> used to indicate at which point the operation failed.
> 
> On failure the error field contains a human-readable error message.
> There are no semantics other than that streaming has failed and clients
> should not try to interpret the error string.
> 
> Examples:
> 
> -> { "execute": "block_stream", "arguments": { "device": "virtio0" } }
> <- { "return":  {} }
> 
> Signed-off-by: Stefan Hajnoczi <stefanha@linux.vnet.ibm.com>
> ---
>  blockdev.c      |   18 +++++++++++++++
>  blockdev.h      |    1 +
>  hmp-commands.hx |   14 +++++++++++
>  monitor.c       |    3 ++
>  monitor.h       |    1 +
>  qerror.h        |    3 ++
>  qmp-commands.hx |   65 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  7 files changed, 105 insertions(+), 0 deletions(-)
> 
> diff --git a/blockdev.c b/blockdev.c
> index d272659..208bfc9 100644
> --- a/blockdev.c
> +++ b/blockdev.c
> @@ -790,3 +790,21 @@ int do_block_resize(Monitor *mon, const QDict *qdict, QObject **ret_data)
> 
>      return 0;
>  }
> +
> +int do_block_stream(Monitor *mon, const QDict *params, QObject **ret_data)
> +{
> +    const char *device = qdict_get_str(params, "device");
> +    BlockDriverState *bs;
> +
> +    bs = bdrv_find(device);
> +    if (!bs) {
> +        qerror_report(QERR_DEVICE_NOT_FOUND, device);
> +        return -1;
> +    }
> +
> +    /* This command is not yet implemented.  The device not found check above
> +     * is done so that error ordering will not change when fully implemented.
> +     */
> +    qerror_report(QERR_NOT_SUPPORTED);
> +    return -1;
> +}
> diff --git a/blockdev.h b/blockdev.h
> index 3587786..ad98d37 100644
> --- a/blockdev.h
> +++ b/blockdev.h
> @@ -65,5 +65,6 @@ int do_change_block(Monitor *mon, const char *device,
>  int do_drive_del(Monitor *mon, const QDict *qdict, QObject **ret_data);
>  int do_snapshot_blkdev(Monitor *mon, const QDict *qdict, QObject **ret_data);
>  int do_block_resize(Monitor *mon, const QDict *qdict, QObject **ret_data);
> +int do_block_stream(Monitor *mon, const QDict *qdict, QObject **ret_data);
> 
>  #endif
> diff --git a/hmp-commands.hx b/hmp-commands.hx
> index 0ccfb28..2a16fd9 100644
> --- a/hmp-commands.hx
> +++ b/hmp-commands.hx
> @@ -70,6 +70,20 @@ but should be used with extreme caution.  Note that this command only
>  resizes image files, it can not resize block devices like LVM volumes.
>  ETEXI
> 
> +    {
> +        .name       = "block_stream",
> +        .args_type  = "device:B,base:s?",
> +        .params     = "device [base]",
> +        .help       = "copy data from a backing file into a block device",
> +        .user_print = monitor_user_noop,
> +        .mhandler.cmd_new = do_block_stream,
> +    },
> +
> +STEXI
> +@item block_stream
> +@findex block_stream
> +Copy data from a backing file into a block device.
> +ETEXI
> 
>      {
>          .name       = "eject",
> diff --git a/monitor.c b/monitor.c
> index ada51d0..dc55fca 100644
> --- a/monitor.c
> +++ b/monitor.c
> @@ -468,6 +468,9 @@ void monitor_protocol_event(MonitorEvent event, QObject *data)
>          case QEVENT_SPICE_DISCONNECTED:
>              event_name = "SPICE_DISCONNECTED";
>              break;
> +        case QEVENT_BLOCK_JOB_COMPLETED:
> +            event_name = "BLOCK_JOB_COMPLETED";
> +            break;
>          default:
>              abort();
>              break;
> diff --git a/monitor.h b/monitor.h
> index 4f2d328..135c927 100644
> --- a/monitor.h
> +++ b/monitor.h
> @@ -35,6 +35,7 @@ typedef enum MonitorEvent {
>      QEVENT_SPICE_CONNECTED,
>      QEVENT_SPICE_INITIALIZED,
>      QEVENT_SPICE_DISCONNECTED,
> +    QEVENT_BLOCK_JOB_COMPLETED,
>      QEVENT_MAX,
>  } MonitorEvent;
> 
> diff --git a/qerror.h b/qerror.h
> index 8058456..eba9238 100644
> --- a/qerror.h
> +++ b/qerror.h
> @@ -139,6 +139,9 @@ QError *qobject_to_qerror(const QObject *obj);
>  #define QERR_NO_BUS_FOR_DEVICE \
>      "{ 'class': 'NoBusForDevice', 'data': { 'device': %s, 'bus': %s } }"
> 
> +#define QERR_NOT_SUPPORTED \
> +    "{ 'class': 'NotSupported', 'data': {} }"
> +
>  #define QERR_OPEN_FILE_FAILED \
>      "{ 'class': 'OpenFileFailed', 'data': { 'filename': %s } }"
> 
> diff --git a/qmp-commands.hx b/qmp-commands.hx
> index 03f67da..60c9bdf 100644
> --- a/qmp-commands.hx
> +++ b/qmp-commands.hx
> @@ -694,6 +694,71 @@ Example:
>  EQMP
> 
>      {
> +        .name       = "block_stream",
> +        .args_type  = "device:B,base:s?",
> +        .params     = "device [base]",
> +        .user_print = monitor_user_noop,
> +        .mhandler.cmd_new = do_block_stream,
> +    },
> +
> +SQMP
> +block_stream
> +------------
> +
> +Copy data from a backing file into a block device.
> +
> +The block streaming operation is performed in the background until the entire
> +backing file has been copied.  This command returns immediately once streaming
> +has started.  The status of ongoing block streaming operations can be checked
> +with query-block-jobs.  The operation can be stopped before it has completed
> +using the block_job_cancel command.
> +
> +If a base file is specified then sectors are not copied from that base file and
> +its backing chain.  When streaming completes the image file will have the base
> +file as its backing file.  This can be used to stream a subset of the backing
> +file chain instead of flattening the entire image.
> +
> +On successful completion the image file is updated to drop the backing file.
> +
> +Arguments:
> +
> +- device: device name (json-string)
> +- base:   common backing file (json-string, optional)
> +
> +Errors:
> +
> +DeviceInUse:    streaming is already active on this device
> +DeviceNotFound: device name is invalid
> +NotSupported:   image streaming is not supported by this device
> +
> +Events:
> +
> +On completion the BLOCK_JOB_COMPLETED event is raised with the following
> +fields:
> +
> +- type:     job type ("stream" for image streaming, json-string)
> +- device:   device name (json-string)
> +- end:      maximum progress value (json-int)
> +- position: current progress value (json-int)
> +- speed:    rate limit, bytes per second (json-int)
> +- error:    error message (json-string, only on error)
> +
> +The completion event is raised both on success and on failure.  On
> +success position is equal to end.  On failure position and end can be
> +used to indicate at which point the operation failed.
> +
> +On failure the error field contains a human-readable error message.  There are
> +no semantics other than that streaming has failed and clients should not try
> +to interpret the error string.
> +
> +Examples:
> +
> +-> { "execute": "block_stream", "arguments": { "device": "virtio0" } }
> +<- { "return":  {} }
> +
> +EQMP
> +
> +    {
>          .name       = "blockdev-snapshot-sync",
>          .args_type  = "device:B,snapshot-file:s?,format:s?",
>          .params     = "device [new-image-file] [format]",

-- 
Thanks,
Adam

Re: [Qemu-devel] [PATCH 1/4] qmp: add block_stream command

[Stefan Hajnoczi]

On Tue, Aug 23, 2011 at 11:33:25AM -0500, Adam Litke wrote:
> Under libvirt, I get the following error when trying to start a block
> stream:
> 
> qerror: bad call in function 'do_block_stream':
> qerror: -> error format '{ 'class': 'NotSupported', 'data': {} }' not found
> qerror: call at blockdev.c:808
> 2011-08-23 11:30:09.974: shutting down
> 
> Is this patch missing a part of the qerror definition?

Thanks Adam.  I need to fix this, will send out a v2.

Stefan

[Qemu-devel] [PATCH 2/4] qmp: add block_job_set_speed command

[Stefan Hajnoczi]

This patch introduces the block_job_set_speed HMP/QMP command.  It is
currently unimplemented and returns the 'NotSupported' error.

block_job_set_speed
-------------------

Set maximum speed for a background block operation.

This command can only be issued when there is an active block job.

Throttling can be disabled by setting the speed to 0.

Arguments:

- device: device name (json-string)
- value:  maximum speed, in bytes per second (json-int)

Errors:
NotSupported:    job type does not support speed setting
DeviceNotActive: streaming is not active on this device

Example:

-> { "execute": "block_job_set_speed",
    "arguments": { "device": "virtio0", "value": 1024 } }

Signed-off-by: Stefan Hajnoczi <stefanha@linux.vnet.ibm.com>
---
 blockdev.c      |   19 +++++++++++++++++++
 blockdev.h      |    2 ++
 hmp-commands.hx |   15 +++++++++++++++
 qmp-commands.hx |   35 +++++++++++++++++++++++++++++++++++
 4 files changed, 71 insertions(+), 0 deletions(-)

diff --git a/blockdev.c b/blockdev.c
index 208bfc9..ee40263 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -808,3 +808,22 @@ int do_block_stream(Monitor *mon, const QDict *params, QObject **ret_data)
     qerror_report(QERR_NOT_SUPPORTED);
     return -1;
 }
+
+int do_block_job_set_speed(Monitor *mon, const QDict *params,
+                           QObject **ret_data)
+{
+    const char *device = qdict_get_str(params, "device");
+    BlockDriverState *bs;
+
+    bs = bdrv_find(device);
+    if (!bs) {
+        qerror_report(QERR_DEVICE_NOT_ACTIVE, device);
+        return -1;
+    }
+
+    /* This command is not yet implemented.  The device not found check above
+     * is done so that error ordering will not change when fully implemented.
+     */
+    qerror_report(QERR_NOT_SUPPORTED);
+    return -1;
+}
diff --git a/blockdev.h b/blockdev.h
index ad98d37..6b48405 100644
--- a/blockdev.h
+++ b/blockdev.h
@@ -66,5 +66,7 @@ int do_drive_del(Monitor *mon, const QDict *qdict, QObject **ret_data);
 int do_snapshot_blkdev(Monitor *mon, const QDict *qdict, QObject **ret_data);
 int do_block_resize(Monitor *mon, const QDict *qdict, QObject **ret_data);
 int do_block_stream(Monitor *mon, const QDict *qdict, QObject **ret_data);
+int do_block_job_set_speed(Monitor *mon, const QDict *qdict,
+                           QObject **ret_data);
 
 #endif
diff --git a/hmp-commands.hx b/hmp-commands.hx
index 2a16fd9..a5fb4d5 100644
--- a/hmp-commands.hx
+++ b/hmp-commands.hx
@@ -86,6 +86,21 @@ Copy data from a backing file into a block device.
 ETEXI
 
     {
+        .name       = "block_job_set_speed",
+        .args_type  = "device:B,value:o",
+        .params     = "device value",
+        .help       = "set maximum speed for a background block operation",
+        .user_print = monitor_user_noop,
+        .mhandler.cmd_new = do_block_job_set_speed,
+    },
+
+STEXI
+@item block_job_set_stream
+@findex block_job_set_stream
+Set maximum speed for a background block operation.
+ETEXI
+
+    {
         .name       = "eject",
         .args_type  = "force:-f,device:B",
         .params     = "[-f] device",
diff --git a/qmp-commands.hx b/qmp-commands.hx
index 60c9bdf..eface05 100644
--- a/qmp-commands.hx
+++ b/qmp-commands.hx
@@ -759,6 +759,41 @@ Examples:
 EQMP
 
     {
+        .name       = "block_job_set_speed",
+        .args_type  = "device:B,value:o",
+        .params     = "device value",
+        .user_print = monitor_user_noop,
+        .mhandler.cmd_new = do_block_job_set_speed,
+    },
+
+SQMP
+
+block_job_set_speed
+-------------------
+
+Set maximum speed for a background block operation.
+
+This command can only be issued when there is an active block job.
+
+Throttling can be disabled by setting the speed to 0.
+
+Arguments:
+
+- device: device name (json-string)
+- value:  maximum speed, in bytes per second (json-int)
+
+Errors:
+NotSupported:    job type does not support speed setting
+DeviceNotActive: streaming is not active on this device
+
+Example:
+
+-> { "execute": "block_job_set_speed",
+    "arguments": { "device": "virtio0", "value": 1024 } }
+
+EQMP
+
+    {
         .name       = "blockdev-snapshot-sync",
         .args_type  = "device:B,snapshot-file:s?,format:s?",
         .params     = "device [new-image-file] [format]",
-- 
1.7.5.4

[Qemu-devel] [PATCH 3/4] qmp: add block_job_cancel command

[Stefan Hajnoczi]

This patch introduces the block_job_cancel HMP/QMP command.  It is
currently unimplemented and returns the 'DeviceNotActive' error.

block_job_cancel
----------------

Stop an active block streaming operation.

This command returns once the active block streaming operation has been
stopped.  It is an error to call this command if no operation is in
progress.

The image file retains its backing file unless the streaming operation
happens to complete just as it is being cancelled.

A new block streaming operation can be started at a later time to finish
copying all data from the backing file.

Arguments:

- device: device name (json-string)

Errors:

DeviceNotActive: streaming is not active on this device
DeviceInUse:     cancellation already in progress

Examples:

-> { "execute": "block_job_cancel", "arguments": { "device": "virtio0" }
}
<- { "return":  {} }

Signed-off-by: Stefan Hajnoczi <stefanha@linux.vnet.ibm.com>
---
 blockdev.c      |    8 ++++++++
 blockdev.h      |    1 +
 hmp-commands.hx |   15 +++++++++++++++
 qmp-commands.hx |   40 ++++++++++++++++++++++++++++++++++++++++
 4 files changed, 64 insertions(+), 0 deletions(-)

diff --git a/blockdev.c b/blockdev.c
index ee40263..036b7eb 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -827,3 +827,11 @@ int do_block_job_set_speed(Monitor *mon, const QDict *params,
     qerror_report(QERR_NOT_SUPPORTED);
     return -1;
 }
+
+int do_block_job_cancel(Monitor *mon, const QDict *params, QObject **ret_data)
+{
+    const char *device = qdict_get_str(params, "device");
+
+    qerror_report(QERR_DEVICE_NOT_ACTIVE, device);
+    return -1;
+}
diff --git a/blockdev.h b/blockdev.h
index 6b48405..a132d36 100644
--- a/blockdev.h
+++ b/blockdev.h
@@ -68,5 +68,6 @@ int do_block_resize(Monitor *mon, const QDict *qdict, QObject **ret_data);
 int do_block_stream(Monitor *mon, const QDict *qdict, QObject **ret_data);
 int do_block_job_set_speed(Monitor *mon, const QDict *qdict,
                            QObject **ret_data);
+int do_block_job_cancel(Monitor *mon, const QDict *qdict, QObject **ret_data);
 
 #endif
diff --git a/hmp-commands.hx b/hmp-commands.hx
index a5fb4d5..ff1a54b 100644
--- a/hmp-commands.hx
+++ b/hmp-commands.hx
@@ -101,6 +101,21 @@ Set maximum speed for a background block operation.
 ETEXI
 
     {
+        .name       = "block_job_cancel",
+        .args_type  = "device:B",
+        .params     = "device",
+        .help       = "stop an active block streaming operation",
+        .user_print = monitor_user_noop,
+        .mhandler.cmd_new = do_block_job_cancel,
+    },
+
+STEXI
+@item block_job_cancel
+@findex block_job_cancel
+Stop an active block streaming operation.
+ETEXI
+
+    {
         .name       = "eject",
         .args_type  = "force:-f,device:B",
         .params     = "[-f] device",
diff --git a/qmp-commands.hx b/qmp-commands.hx
index eface05..de442f7 100644
--- a/qmp-commands.hx
+++ b/qmp-commands.hx
@@ -794,6 +794,46 @@ Example:
 EQMP
 
     {
+        .name       = "block_job_cancel",
+        .args_type  = "device:B",
+        .params     = "device",
+        .user_print = monitor_user_noop,
+        .mhandler.cmd_new = do_block_job_cancel,
+    },
+
+SQMP
+
+block_job_cancel
+----------------
+
+Stop an active block streaming operation.
+
+This command returns once the active block streaming operation has been
+stopped.  It is an error to call this command if no operation is in progress.
+
+The image file retains its backing file unless the streaming operation happens
+to complete just as it is being cancelled.
+
+A new block streaming operation can be started at a later time to finish
+copying all data from the backing file.
+
+Arguments:
+
+- device: device name (json-string)
+
+Errors:
+
+DeviceNotActive: streaming is not active on this device
+DeviceInUse:     cancellation already in progress
+
+Examples:
+
+-> { "execute": "block_job_cancel", "arguments": { "device": "virtio0" } }
+<- { "return":  {} }
+
+EQMP
+
+    {
         .name       = "blockdev-snapshot-sync",
         .args_type  = "device:B,snapshot-file:s?,format:s?",
         .params     = "device [new-image-file] [format]",
-- 
1.7.5.4

[Qemu-devel] [PATCH 4/4] qmp: add query-block-jobs

[Stefan Hajnoczi]

This patch introduces the query-block-jobs HMP/QMP command.  It is
currently unimplemented and returns an empty dict.

query-block-jobs
----------------

Show progress of ongoing block device operations.

Return a json-array of all block device operations.  If no operation is
active then return an empty array.  Each operation is a json-object with
the following data:

- type:     job type ("stream" for image streaming, json-string)
- device:   device name (json-string)
- end:      maximum progress value (json-int)
- position: current progress value (json-int)
- speed:    rate limit, bytes per second (json-int)

Progress can be observed as position increases and it reaches end when
the operation completes.  Position and end have undefined units but can
be used to calculate a percentage indicating the progress that has been
made.

Example:

-> { "execute": "query-block-jobs" }
<- { "return":[
      { "type": "stream", "device": "virtio0",
        "end": 10737418240, "position": 709632,
        "speed": 0 }
   ]
 }

Signed-off-by: Stefan Hajnoczi <stefanha@linux.vnet.ibm.com>
---
 blockdev.c      |   10 ++++++++++
 blockdev.h      |    2 ++
 monitor.c       |   16 ++++++++++++++++
 qmp-commands.hx |   32 ++++++++++++++++++++++++++++++++
 4 files changed, 60 insertions(+), 0 deletions(-)

diff --git a/blockdev.c b/blockdev.c
index 036b7eb..e9098f6 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -835,3 +835,13 @@ int do_block_job_cancel(Monitor *mon, const QDict *params, QObject **ret_data)
     qerror_report(QERR_DEVICE_NOT_ACTIVE, device);
     return -1;
 }
+
+void monitor_print_block_jobs(Monitor *mon, const QObject *data)
+{
+    monitor_printf(mon, "No active jobs\n");
+}
+
+void do_info_block_jobs(Monitor *mon, QObject **ret_data)
+{
+    *ret_data = QOBJECT(qdict_new());
+}
diff --git a/blockdev.h b/blockdev.h
index a132d36..f199d7a 100644
--- a/blockdev.h
+++ b/blockdev.h
@@ -69,5 +69,7 @@ int do_block_stream(Monitor *mon, const QDict *qdict, QObject **ret_data);
 int do_block_job_set_speed(Monitor *mon, const QDict *qdict,
                            QObject **ret_data);
 int do_block_job_cancel(Monitor *mon, const QDict *qdict, QObject **ret_data);
+void monitor_print_block_jobs(Monitor *mon, const QObject *data);
+void do_info_block_jobs(Monitor *mon, QObject **ret_data);
 
 #endif
diff --git a/monitor.c b/monitor.c
index dc55fca..e832d10 100644
--- a/monitor.c
+++ b/monitor.c
@@ -2907,6 +2907,14 @@ static const mon_cmd_t info_cmds[] = {
         .mhandler.info_new = bdrv_info_stats,
     },
     {
+        .name       = "block-jobs",
+        .args_type  = "",
+        .params     = "",
+        .help       = "show progress of ongoing block device operations",
+        .user_print = monitor_print_block_jobs,
+        .mhandler.info_new = do_info_block_jobs,
+    },
+    {
         .name       = "registers",
         .args_type  = "",
         .params     = "",
@@ -3206,6 +3214,14 @@ static const mon_cmd_t qmp_query_cmds[] = {
         .mhandler.info_new = bdrv_info_stats,
     },
     {
+        .name       = "block-jobs",
+        .args_type  = "",
+        .params     = "",
+        .help       = "show progress of ongoing block device operations",
+        .user_print = monitor_print_block_jobs,
+        .mhandler.info_new = do_info_block_jobs,
+    },
+    {
         .name       = "cpus",
         .args_type  = "",
         .params     = "",
diff --git a/qmp-commands.hx b/qmp-commands.hx
index de442f7..ffac014 100644
--- a/qmp-commands.hx
+++ b/qmp-commands.hx
@@ -2006,3 +2006,35 @@ Example:
 
 EQMP
 
+SQMP
+
+query-block-jobs
+----------------
+
+Show progress of ongoing block device operations.
+
+Return a json-array of all block device operations.  If no operation is active
+then return an empty array.  Each operation is a json-object with the following
+data:
+
+- type:     job type ("stream" for image streaming, json-string)
+- device:   device name (json-string)
+- end:      maximum progress value (json-int)
+- position: current progress value (json-int)
+- speed:    rate limit, bytes per second (json-int)
+
+Progress can be observed as position increases and it reaches end when the
+operation completes.  Position and end have undefined units but can be used to
+calculate a percentage indicating the progress that has been made.
+
+Example:
+
+-> { "execute": "query-block-jobs" }
+<- { "return":[
+      { "type": "stream", "device": "virtio0",
+        "end": 10737418240, "position": 709632,
+        "speed": 0 }
+   ]
+ }
+
+EQMP
-- 
1.7.5.4

Re: [Qemu-devel] [PATCH 4/4] qmp: add query-block-jobs

[Adam Litke]

On Tue, 2011-08-23 at 13:58 +0100, Stefan Hajnoczi wrote:
> diff --git a/blockdev.c b/blockdev.c
> index 036b7eb..e9098f6 100644
> --- a/blockdev.c
> +++ b/blockdev.c
> @@ -835,3 +835,13 @@ int do_block_job_cancel(Monitor *mon, const QDict *params, QObject **ret_data)
>      qerror_report(QERR_DEVICE_NOT_ACTIVE, device);
>      return -1;
>  }
> +
> +void monitor_print_block_jobs(Monitor *mon, const QObject *data)
> +{
> +    monitor_printf(mon, "No active jobs\n");
> +}
> +
> +void do_info_block_jobs(Monitor *mon, QObject **ret_data)
> +{
> +    *ret_data = QOBJECT(qdict_new());
> +}

This should return an empty qlist, not a qdict.

-- 
Thanks,
Adam

Re: [Qemu-devel] [PATCH 0/4] Image Streaming API

[Anthony Liguori]

On 08/23/2011 07:58 AM, Stefan Hajnoczi wrote:
> These patches put in place the image streaming QMP/HMP commands and
> documentation.  Image streaming itself is not implemented by this patch series
> but the HMP/QMP commands that libvirt uses are implemented to return
> NotSupported.
>
> The Image Streaming API can be used to copy the contents of a backing file into
> the image file while the guest is running.  The API is described on the wiki:
> http://wiki.qemu.org/Features/LiveBlockMigration/ImageStreamingAPI
>
> The point of this series is to commit QEMU to the API that we have worked out
> with libvirt.  The QED Image Streaming series that I posted earlier provides an
> implementation for the QED image format only.  I am currently working on a
> generic block layer implementation so that any format with backing file support
> can do image streaming.
>
> For reference, the QED-specific implementation lives here:
> http://repo.or.cz/w/qemu/stefanha.git/shortlog/refs/heads/stream-command

Luiz, can you bring this in through your tree with an Ack from Kevin.

Regards,

Anthony Liguori

>
> Stefan Hajnoczi (4):
>    qmp: add block_stream command
>    qmp: add block_job_set_speed command
>    qmp: add block_job_cancel command
>    qmp: add query-block-jobs
>
>   blockdev.c      |   55 ++++++++++++++++++
>   blockdev.h      |    6 ++
>   hmp-commands.hx |   44 ++++++++++++++
>   monitor.c       |   19 ++++++
>   monitor.h       |    1 +
>   qerror.h        |    3 +
>   qmp-commands.hx |  172 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
>   7 files changed, 300 insertions(+), 0 deletions(-)
>

Re: [Qemu-devel] [PATCH 0/4] Image Streaming API

[Luiz Capitulino]

On Mon, 29 Aug 2011 14:16:44 -0500
Anthony Liguori <aliguori@us.ibm.com> wrote:

> On 08/23/2011 07:58 AM, Stefan Hajnoczi wrote:
> > These patches put in place the image streaming QMP/HMP commands and
> > documentation.  Image streaming itself is not implemented by this patch series
> > but the HMP/QMP commands that libvirt uses are implemented to return
> > NotSupported.
> >
> > The Image Streaming API can be used to copy the contents of a backing file into
> > the image file while the guest is running.  The API is described on the wiki:
> > http://wiki.qemu.org/Features/LiveBlockMigration/ImageStreamingAPI
> >
> > The point of this series is to commit QEMU to the API that we have worked out
> > with libvirt.  The QED Image Streaming series that I posted earlier provides an
> > implementation for the QED image format only.  I am currently working on a
> > generic block layer implementation so that any format with backing file support
> > can do image streaming.
> >
> > For reference, the QED-specific implementation lives here:
> > http://repo.or.cz/w/qemu/stefanha.git/shortlog/refs/heads/stream-command
> 
> Luiz, can you bring this in through your tree with an Ack from Kevin.

Sure.

Re: [Qemu-devel] [PATCH 0/4] Image Streaming API

[Zhi Yong Wu]

On Tue, Aug 23, 2011 at 8:58 PM, Stefan Hajnoczi
<stefanha@linux.vnet.ibm.com> wrote:
> These patches put in place the image streaming QMP/HMP commands and
> documentation.  Image streaming itself is not implemented by this patch series
> but the HMP/QMP commands that libvirt uses are implemented to return
> NotSupported.
>
> The Image Streaming API can be used to copy the contents of a backing file into
> the image file while the guest is running.  The API is described on the wiki:
> http://wiki.qemu.org/Features/LiveBlockMigration/ImageStreamingAPI

If query-block-jobs returns one percent value, it will be more
readable and convenient for users.

>
> The point of this series is to commit QEMU to the API that we have worked out
> with libvirt.  The QED Image Streaming series that I posted earlier provides an
> implementation for the QED image format only.  I am currently working on a
> generic block layer implementation so that any format with backing file support
> can do image streaming.
>
> For reference, the QED-specific implementation lives here:
> http://repo.or.cz/w/qemu/stefanha.git/shortlog/refs/heads/stream-command
>
> Stefan Hajnoczi (4):
>  qmp: add block_stream command
>  qmp: add block_job_set_speed command
>  qmp: add block_job_cancel command
>  qmp: add query-block-jobs
>
>  blockdev.c      |   55 ++++++++++++++++++
>  blockdev.h      |    6 ++
>  hmp-commands.hx |   44 ++++++++++++++
>  monitor.c       |   19 ++++++
>  monitor.h       |    1 +
>  qerror.h        |    3 +
>  qmp-commands.hx |  172 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  7 files changed, 300 insertions(+), 0 deletions(-)
>
> --
> 1.7.5.4
>
>
>



-- 
Regards,

Zhi Yong Wu

Re: [Qemu-devel] [PATCH 0/4] Image Streaming API

[Stefan Hajnoczi]

On Tue, Aug 30, 2011 at 4:19 AM, Zhi Yong Wu <zwu.kernel@gmail.com> wrote:
> On Tue, Aug 23, 2011 at 8:58 PM, Stefan Hajnoczi
> <stefanha@linux.vnet.ibm.com> wrote:
>> These patches put in place the image streaming QMP/HMP commands and
>> documentation.  Image streaming itself is not implemented by this patch series
>> but the HMP/QMP commands that libvirt uses are implemented to return
>> NotSupported.
>>
>> The Image Streaming API can be used to copy the contents of a backing file into
>> the image file while the guest is running.  The API is described on the wiki:
>> http://wiki.qemu.org/Features/LiveBlockMigration/ImageStreamingAPI
>
> If query-block-jobs returns one percent value, it will be more
> readable and convenient for users.

QMP is not a human interface.  Although a percentage value is good for
progress bars, the absolute value is also interesting in terms of how
much work is being done.  By providing an end value and a current
value the management tool can display either or both.

Stefan

Re: [Qemu-devel] [PATCH 0/4] Image Streaming API

[Adam Litke]

On Tue, Aug 30, 2011 at 10:28:09AM +0100, Stefan Hajnoczi wrote:
> On Tue, Aug 30, 2011 at 4:19 AM, Zhi Yong Wu <zwu.kernel@gmail.com> wrote:
> > On Tue, Aug 23, 2011 at 8:58 PM, Stefan Hajnoczi
> > <stefanha@linux.vnet.ibm.com> wrote:
> >> These patches put in place the image streaming QMP/HMP commands and
> >> documentation.  Image streaming itself is not implemented by this patch series
> >> but the HMP/QMP commands that libvirt uses are implemented to return
> >> NotSupported.
> >>
> >> The Image Streaming API can be used to copy the contents of a backing file into
> >> the image file while the guest is running.  The API is described on the wiki:
> >> http://wiki.qemu.org/Features/LiveBlockMigration/ImageStreamingAPI
> >
> > If query-block-jobs returns one percent value, it will be more
> > readable and convenient for users.
> 
> QMP is not a human interface.  Although a percentage value is good for
> progress bars, the absolute value is also interesting in terms of how
> much work is being done.  By providing an end value and a current
> value the management tool can display either or both.

I agree.  In QMP, we should report the best, most granular information we have
and allow management tools to present it as they see fit.  For example, a
management tool may wish to present streaming progress as a spinner (rather than
a progress bar).  In that case you would want to see the numbers updated
whenever progress has been made.