* [Qemu-devel] [PATCH 0/2] mirror: Allow detection of zeroes on source sectors @ 2015-06-08 7:06 Fam Zheng 2015-06-08 7:06 ` [Qemu-devel] [PATCH 1/2] qapi: Add "detect-zeroes" option to drive-mirror Fam Zheng 2015-06-08 7:06 ` [Qemu-devel] [PATCH 2/2] iotests: Add test cases for drive-mirror "detect-zeroes" option Fam Zheng 0 siblings, 2 replies; 7+ messages in thread From: Fam Zheng @ 2015-06-08 7:06 UTC (permalink / raw) To: qemu-devel Cc: Kevin Wolf, qemu-block, Jeff Cody, Markus Armbruster, stefanha, pbonzini [These patches go on top of the "block: Mirror discarded sectors" series] Some protocols don't have an easy way to query sparseness, (e.g. block/nfs.c, block/nbd.c), for which block layer always reports block status as "allocated data". This will let mirror job do full provisioning even if data is actually sparse under the hood. With the new "detect-zeroes" option, we can let mirror job detect zeroes before sending the data to target, and use zero write when it is more efficient. Fam Fam Zheng (2): qapi: Add "detect-zeroes" option to drive-mirror iotests: Add test cases for drive-mirror "detect-zeroes" option block/mirror.c | 21 +++++++++++++++------ blockdev.c | 6 +++++- hmp.c | 2 +- include/block/block_int.h | 3 ++- qapi/block-core.json | 4 +++- qmp-commands.hx | 4 +++- tests/qemu-iotests/132 | 26 +++++++++++++++++++++++--- tests/qemu-iotests/132.out | 4 ++-- tests/qemu-iotests/iotests.py | 7 +++++++ 9 files changed, 61 insertions(+), 16 deletions(-) -- 2.4.2 ^ permalink raw reply [flat|nested] 7+ messages in thread
* [Qemu-devel] [PATCH 1/2] qapi: Add "detect-zeroes" option to drive-mirror 2015-06-08 7:06 [Qemu-devel] [PATCH 0/2] mirror: Allow detection of zeroes on source sectors Fam Zheng @ 2015-06-08 7:06 ` Fam Zheng 2015-06-10 15:41 ` Andrey Korolyov 2015-06-08 7:06 ` [Qemu-devel] [PATCH 2/2] iotests: Add test cases for drive-mirror "detect-zeroes" option Fam Zheng 1 sibling, 1 reply; 7+ messages in thread From: Fam Zheng @ 2015-06-08 7:06 UTC (permalink / raw) To: qemu-devel Cc: Kevin Wolf, qemu-block, Jeff Cody, Markus Armbruster, stefanha, pbonzini The new optional flag defaults to true, in which case, mirror job would check the read sectors and use sparse write if they are zero. Otherwise data will be fully copied. Signed-off-by: Fam Zheng <famz@redhat.com> --- block/mirror.c | 21 +++++++++++++++------ blockdev.c | 6 +++++- hmp.c | 2 +- include/block/block_int.h | 3 ++- qapi/block-core.json | 4 +++- qmp-commands.hx | 4 +++- 6 files changed, 29 insertions(+), 11 deletions(-) diff --git a/block/mirror.c b/block/mirror.c index 3c38695..261373c 100644 --- a/block/mirror.c +++ b/block/mirror.c @@ -58,6 +58,7 @@ typedef struct MirrorBlockJob { int sectors_in_flight; int ret; bool unmap; + bool detect_zeroes; } MirrorBlockJob; typedef struct MirrorOp { @@ -153,8 +154,14 @@ static void mirror_read_complete(void *opaque, int ret) mirror_iteration_done(op, ret); return; } - bdrv_aio_writev(s->target, op->sector_num, &op->qiov, op->nb_sectors, - mirror_write_complete, op); + if (s->detect_zeroes && qemu_iovec_is_zero(&op->qiov)) { + bdrv_aio_write_zeroes(s->target, op->sector_num, op->nb_sectors, + s->unmap ? BDRV_REQ_MAY_UNMAP : 0, + mirror_write_complete, op); + } else { + bdrv_aio_writev(s->target, op->sector_num, &op->qiov, op->nb_sectors, + mirror_write_complete, op); + } } static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s) @@ -668,7 +675,7 @@ static void mirror_start_job(BlockDriverState *bs, BlockDriverState *target, int64_t buf_size, BlockdevOnError on_source_error, BlockdevOnError on_target_error, - bool unmap, + bool unmap, bool detect_zeroes, BlockCompletionFunc *cb, void *opaque, Error **errp, const BlockJobDriver *driver, @@ -704,6 +711,7 @@ static void mirror_start_job(BlockDriverState *bs, BlockDriverState *target, s->granularity = granularity; s->buf_size = MAX(buf_size, granularity); s->unmap = unmap; + s->detect_zeroes = detect_zeroes; s->dirty_bitmap = bdrv_create_dirty_bitmap(bs, granularity, NULL, errp); if (!s->dirty_bitmap) { @@ -722,7 +730,7 @@ void mirror_start(BlockDriverState *bs, BlockDriverState *target, int64_t speed, uint32_t granularity, int64_t buf_size, MirrorSyncMode mode, BlockdevOnError on_source_error, BlockdevOnError on_target_error, - bool unmap, + bool unmap, bool detect_zeroes, BlockCompletionFunc *cb, void *opaque, Error **errp) { @@ -737,7 +745,8 @@ void mirror_start(BlockDriverState *bs, BlockDriverState *target, base = mode == MIRROR_SYNC_MODE_TOP ? bs->backing_hd : NULL; mirror_start_job(bs, target, replaces, speed, granularity, buf_size, - on_source_error, on_target_error, unmap, cb, opaque, errp, + on_source_error, on_target_error, unmap, + detect_zeroes, cb, opaque, errp, &mirror_job_driver, is_none_mode, base); } @@ -785,7 +794,7 @@ void commit_active_start(BlockDriverState *bs, BlockDriverState *base, bdrv_ref(base); mirror_start_job(bs, base, NULL, speed, 0, 0, - on_error, on_error, false, cb, opaque, &local_err, + on_error, on_error, false, false, cb, opaque, &local_err, &commit_active_job_driver, false, base); if (local_err) { error_propagate(errp, local_err); diff --git a/blockdev.c b/blockdev.c index 4387e14..d86ec1c 100644 --- a/blockdev.c +++ b/blockdev.c @@ -2633,6 +2633,7 @@ void qmp_drive_mirror(const char *device, const char *target, bool has_on_source_error, BlockdevOnError on_source_error, bool has_on_target_error, BlockdevOnError on_target_error, bool has_unmap, bool unmap, + bool has_detect_zeroes, bool detect_zeroes, Error **errp) { BlockBackend *blk; @@ -2667,6 +2668,9 @@ void qmp_drive_mirror(const char *device, const char *target, if (!has_unmap) { unmap = true; } + if (!has_detect_zeroes) { + detect_zeroes = true; + } if (granularity != 0 && (granularity < 512 || granularity > 1048576 * 64)) { error_set(errp, QERR_INVALID_PARAMETER_VALUE, "granularity", @@ -2806,7 +2810,7 @@ void qmp_drive_mirror(const char *device, const char *target, has_replaces ? replaces : NULL, speed, granularity, buf_size, sync, on_source_error, on_target_error, - unmap, + unmap, detect_zeroes, block_job_cb, bs, &local_err); if (local_err != NULL) { bdrv_unref(target_bs); diff --git a/hmp.c b/hmp.c index 62c53e0..28a597f 100644 --- a/hmp.c +++ b/hmp.c @@ -1056,7 +1056,7 @@ void hmp_drive_mirror(Monitor *mon, const QDict *qdict) false, NULL, false, NULL, full ? MIRROR_SYNC_MODE_FULL : MIRROR_SYNC_MODE_TOP, true, mode, false, 0, false, 0, false, 0, - false, 0, false, 0, false, true, &err); + false, 0, false, 0, false, true, false, true, &err); hmp_handle_error(mon, &err); } diff --git a/include/block/block_int.h b/include/block/block_int.h index 459fe1c..10715a6 100644 --- a/include/block/block_int.h +++ b/include/block/block_int.h @@ -591,6 +591,7 @@ void commit_active_start(BlockDriverState *bs, BlockDriverState *base, * @on_source_error: The action to take upon error reading from the source. * @on_target_error: The action to take upon error writing to the target. * @unmap: Whether to unmap target where source sectors only contain zeroes. + * @detect_zero: Whether to do zero detect of source sectors. * @cb: Completion function for the job. * @opaque: Opaque pointer value passed to @cb. * @errp: Error object. @@ -605,7 +606,7 @@ void mirror_start(BlockDriverState *bs, BlockDriverState *target, int64_t speed, uint32_t granularity, int64_t buf_size, MirrorSyncMode mode, BlockdevOnError on_source_error, BlockdevOnError on_target_error, - bool unmap, + bool unmap, bool detect_zeroes, BlockCompletionFunc *cb, void *opaque, Error **errp); diff --git a/qapi/block-core.json b/qapi/block-core.json index a59063d..2014dc6 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -959,6 +959,8 @@ # target image sectors will be unmapped; otherwise, zeroes will be # written. Both will result in identical contents. # Default is true. (Since 2.4) +# @detect-zeroes: #optional Whether to detect zero sectors in source and use +# zero write on target. Default is true. (Since 2.4) # # Returns: nothing on success # If @device is not a valid block device, DeviceNotFound @@ -972,7 +974,7 @@ '*speed': 'int', '*granularity': 'uint32', '*buf-size': 'int', '*on-source-error': 'BlockdevOnError', '*on-target-error': 'BlockdevOnError', - '*unmap': 'bool' } } + '*unmap': 'bool', '*detect-zeroes': 'bool' } } ## # @BlockDirtyBitmap diff --git a/qmp-commands.hx b/qmp-commands.hx index 63c86fc..cac07fc 100644 --- a/qmp-commands.hx +++ b/qmp-commands.hx @@ -1503,7 +1503,7 @@ EQMP .args_type = "sync:s,device:B,target:s,speed:i?,mode:s?,format:s?," "node-name:s?,replaces:s?," "on-source-error:s?,on-target-error:s?," - "unmap:b?," + "unmap:b?,detect-zeroes:b?" "granularity:i?,buf-size:i?", .mhandler.cmd_new = qmp_marshal_input_drive_mirror, }, @@ -1545,6 +1545,8 @@ Arguments: (BlockdevOnError, default 'report') - "unmap": whether the target sectors should be discarded where source has only zeroes. (json-bool, optional, default true) +- "detect-zeroes": if true, the source sectors that are zeroes will be written as + sparse on target. (json-bool, optional, default true) The default value of the granularity is the image cluster size clamped between 4096 and 65536, if the image format defines one. If the format -- 2.4.2 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [Qemu-devel] [PATCH 1/2] qapi: Add "detect-zeroes" option to drive-mirror 2015-06-08 7:06 ` [Qemu-devel] [PATCH 1/2] qapi: Add "detect-zeroes" option to drive-mirror Fam Zheng @ 2015-06-10 15:41 ` Andrey Korolyov 2015-06-10 15:48 ` Andrey Korolyov 0 siblings, 1 reply; 7+ messages in thread From: Andrey Korolyov @ 2015-06-10 15:41 UTC (permalink / raw) To: Fam Zheng Cc: Kevin Wolf, qemu block, Jeff Cody, qemu-devel@nongnu.org, Markus Armbruster, Stefan Hajnoczi, Paolo Bonzini On Mon, Jun 8, 2015 at 10:06 AM, Fam Zheng <famz@redhat.com> wrote: > The new optional flag defaults to true, in which case, mirror job would > check the read sectors and use sparse write if they are zero. Otherwise > data will be fully copied. > > Signed-off-by: Fam Zheng <famz@redhat.com> > --- > block/mirror.c | 21 +++++++++++++++------ > blockdev.c | 6 +++++- > hmp.c | 2 +- > include/block/block_int.h | 3 ++- > qapi/block-core.json | 4 +++- > qmp-commands.hx | 4 +++- > 6 files changed, 29 insertions(+), 11 deletions(-) > > diff --git a/block/mirror.c b/block/mirror.c > index 3c38695..261373c 100644 > --- a/block/mirror.c > +++ b/block/mirror.c > @@ -58,6 +58,7 @@ typedef struct MirrorBlockJob { > int sectors_in_flight; > int ret; > bool unmap; > + bool detect_zeroes; > } MirrorBlockJob; > > typedef struct MirrorOp { > @@ -153,8 +154,14 @@ static void mirror_read_complete(void *opaque, int ret) > mirror_iteration_done(op, ret); > return; > } > - bdrv_aio_writev(s->target, op->sector_num, &op->qiov, op->nb_sectors, > - mirror_write_complete, op); > + if (s->detect_zeroes && qemu_iovec_is_zero(&op->qiov)) { > + bdrv_aio_write_zeroes(s->target, op->sector_num, op->nb_sectors, > + s->unmap ? BDRV_REQ_MAY_UNMAP : 0, > + mirror_write_complete, op); > + } else { > + bdrv_aio_writev(s->target, op->sector_num, &op->qiov, op->nb_sectors, > + mirror_write_complete, op); > + } > } > > static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s) > @@ -668,7 +675,7 @@ static void mirror_start_job(BlockDriverState *bs, BlockDriverState *target, > int64_t buf_size, > BlockdevOnError on_source_error, > BlockdevOnError on_target_error, > - bool unmap, > + bool unmap, bool detect_zeroes, > BlockCompletionFunc *cb, > void *opaque, Error **errp, > const BlockJobDriver *driver, > @@ -704,6 +711,7 @@ static void mirror_start_job(BlockDriverState *bs, BlockDriverState *target, > s->granularity = granularity; > s->buf_size = MAX(buf_size, granularity); > s->unmap = unmap; > + s->detect_zeroes = detect_zeroes; > > s->dirty_bitmap = bdrv_create_dirty_bitmap(bs, granularity, NULL, errp); > if (!s->dirty_bitmap) { > @@ -722,7 +730,7 @@ void mirror_start(BlockDriverState *bs, BlockDriverState *target, > int64_t speed, uint32_t granularity, int64_t buf_size, > MirrorSyncMode mode, BlockdevOnError on_source_error, > BlockdevOnError on_target_error, > - bool unmap, > + bool unmap, bool detect_zeroes, > BlockCompletionFunc *cb, > void *opaque, Error **errp) > { > @@ -737,7 +745,8 @@ void mirror_start(BlockDriverState *bs, BlockDriverState *target, > base = mode == MIRROR_SYNC_MODE_TOP ? bs->backing_hd : NULL; > mirror_start_job(bs, target, replaces, > speed, granularity, buf_size, > - on_source_error, on_target_error, unmap, cb, opaque, errp, > + on_source_error, on_target_error, unmap, > + detect_zeroes, cb, opaque, errp, > &mirror_job_driver, is_none_mode, base); > } > > @@ -785,7 +794,7 @@ void commit_active_start(BlockDriverState *bs, BlockDriverState *base, > > bdrv_ref(base); > mirror_start_job(bs, base, NULL, speed, 0, 0, > - on_error, on_error, false, cb, opaque, &local_err, > + on_error, on_error, false, false, cb, opaque, &local_err, > &commit_active_job_driver, false, base); > if (local_err) { > error_propagate(errp, local_err); > diff --git a/blockdev.c b/blockdev.c > index 4387e14..d86ec1c 100644 > --- a/blockdev.c > +++ b/blockdev.c > @@ -2633,6 +2633,7 @@ void qmp_drive_mirror(const char *device, const char *target, > bool has_on_source_error, BlockdevOnError on_source_error, > bool has_on_target_error, BlockdevOnError on_target_error, > bool has_unmap, bool unmap, > + bool has_detect_zeroes, bool detect_zeroes, > Error **errp) > { > BlockBackend *blk; > @@ -2667,6 +2668,9 @@ void qmp_drive_mirror(const char *device, const char *target, > if (!has_unmap) { > unmap = true; > } > + if (!has_detect_zeroes) { > + detect_zeroes = true; > + } > > if (granularity != 0 && (granularity < 512 || granularity > 1048576 * 64)) { > error_set(errp, QERR_INVALID_PARAMETER_VALUE, "granularity", > @@ -2806,7 +2810,7 @@ void qmp_drive_mirror(const char *device, const char *target, > has_replaces ? replaces : NULL, > speed, granularity, buf_size, sync, > on_source_error, on_target_error, > - unmap, > + unmap, detect_zeroes, > block_job_cb, bs, &local_err); > if (local_err != NULL) { > bdrv_unref(target_bs); > diff --git a/hmp.c b/hmp.c > index 62c53e0..28a597f 100644 > --- a/hmp.c > +++ b/hmp.c > @@ -1056,7 +1056,7 @@ void hmp_drive_mirror(Monitor *mon, const QDict *qdict) > false, NULL, false, NULL, > full ? MIRROR_SYNC_MODE_FULL : MIRROR_SYNC_MODE_TOP, > true, mode, false, 0, false, 0, false, 0, > - false, 0, false, 0, false, true, &err); > + false, 0, false, 0, false, true, false, true, &err); > hmp_handle_error(mon, &err); > } > > diff --git a/include/block/block_int.h b/include/block/block_int.h > index 459fe1c..10715a6 100644 > --- a/include/block/block_int.h > +++ b/include/block/block_int.h > @@ -591,6 +591,7 @@ void commit_active_start(BlockDriverState *bs, BlockDriverState *base, > * @on_source_error: The action to take upon error reading from the source. > * @on_target_error: The action to take upon error writing to the target. > * @unmap: Whether to unmap target where source sectors only contain zeroes. > + * @detect_zero: Whether to do zero detect of source sectors. > * @cb: Completion function for the job. > * @opaque: Opaque pointer value passed to @cb. > * @errp: Error object. > @@ -605,7 +606,7 @@ void mirror_start(BlockDriverState *bs, BlockDriverState *target, > int64_t speed, uint32_t granularity, int64_t buf_size, > MirrorSyncMode mode, BlockdevOnError on_source_error, > BlockdevOnError on_target_error, > - bool unmap, > + bool unmap, bool detect_zeroes, > BlockCompletionFunc *cb, > void *opaque, Error **errp); > > diff --git a/qapi/block-core.json b/qapi/block-core.json > index a59063d..2014dc6 100644 > --- a/qapi/block-core.json > +++ b/qapi/block-core.json > @@ -959,6 +959,8 @@ > # target image sectors will be unmapped; otherwise, zeroes will be > # written. Both will result in identical contents. > # Default is true. (Since 2.4) > +# @detect-zeroes: #optional Whether to detect zero sectors in source and use > +# zero write on target. Default is true. (Since 2.4) > # > # Returns: nothing on success > # If @device is not a valid block device, DeviceNotFound > @@ -972,7 +974,7 @@ > '*speed': 'int', '*granularity': 'uint32', > '*buf-size': 'int', '*on-source-error': 'BlockdevOnError', > '*on-target-error': 'BlockdevOnError', > - '*unmap': 'bool' } } > + '*unmap': 'bool', '*detect-zeroes': 'bool' } } > > ## > # @BlockDirtyBitmap > diff --git a/qmp-commands.hx b/qmp-commands.hx > index 63c86fc..cac07fc 100644 > --- a/qmp-commands.hx > +++ b/qmp-commands.hx > @@ -1503,7 +1503,7 @@ EQMP > .args_type = "sync:s,device:B,target:s,speed:i?,mode:s?,format:s?," > "node-name:s?,replaces:s?," > "on-source-error:s?,on-target-error:s?," > - "unmap:b?," > + "unmap:b?,detect-zeroes:b?" > "granularity:i?,buf-size:i?", > .mhandler.cmd_new = qmp_marshal_input_drive_mirror, > }, > @@ -1545,6 +1545,8 @@ Arguments: > (BlockdevOnError, default 'report') > - "unmap": whether the target sectors should be discarded where source has only > zeroes. (json-bool, optional, default true) > +- "detect-zeroes": if true, the source sectors that are zeroes will be written as > + sparse on target. (json-bool, optional, default true) > > The default value of the granularity is the image cluster size clamped > between 4096 and 65536, if the image format defines one. If the format > -- > 2.4.2 > > Hi Fam, just checked this one with slight adaptation to the current master on an RBD backend, it looks like despite the source returns zeroes (no reads in stat), the same zeroes are happily allocated at the target by the drivemirror: 2015-06-10 18:33:10.376619 mon.0 [INF] pgmap v6126601: 384 pgs: 384 active+clean; 453 GB data, 1788 GB used, 20511 GB / 22300 GB avail; 5164KB/s wr, 1op/s 2015-06-10 18:33:11.387558 mon.0 [INF] pgmap v6126602: 384 pgs: 384 active+clean; 453 GB data, 1788 GB used, 20511 GB / 22300 GB avail; 14251KB/s wr, 3op/s 2015-06-10 18:33:13.393758 mon.0 [INF] pgmap v6126603: 384 pgs: 384 active+clean; 453 GB data, 1788 GB used, 20511 GB / 22300 GB avail; 12582KB/s wr, 3op/s 2015-06-10 18:33:14.396960 mon.0 [INF] pgmap v6126604: 384 pgs: 384 active+clean; 453 GB data, 1788 GB used, 20511 GB / 22300 GB avail; 2039B/s rd, 11237KB/s wr, 4op/s 2015-06-10 18:33:15.406989 mon.0 [INF] pgmap v6126605: 384 pgs: 384 active+clean; 453 GB data, 1788 GB used, 20511 GB / 22300 GB avail; 3054B/s rd, 14255KB/s wr, 5op/s 2015-06-10 18:33:16.414723 mon.0 [INF] pgmap v6126606: 384 pgs: 384 active+clean; 453 GB data, 1789 GB used, 20511 GB / 22300 GB avail; 14579KB/s wr, 5op/s 2015-06-10 18:33:18.421028 mon.0 [INF] pgmap v6126607: 384 pgs: 384 active+clean; 453 GB data, 1789 GB used, 20511 GB / 22300 GB avail; 13816KB/s wr, 4op/s 2015-06-10 18:33:19.423108 mon.0 [INF] pgmap v6126608: 384 pgs: 384 active+clean; 453 GB data, 1789 GB used, 20511 GB / 22300 GB avail; 9519KB/s wr, 2op/s 2015-06-10 18:33:20.433639 mon.0 [INF] pgmap v6126609: 384 pgs: 384 active+clean; 453 GB data, 1789 GB used, 20510 GB / 22300 GB avail; 10186KB/s wr, 2op/s 2015-06-10 18:33:21.436827 mon.0 [INF] pgmap v6126610: 384 pgs: 384 active+clean; 453 GB data, 1789 GB used, 20510 GB / 22300 GB avail; 15123KB/s wr, 4op/s 2015-06-10 18:33:23.446657 mon.0 [INF] pgmap v6126611: 384 pgs: 384 active+clean; 453 GB data, 1789 GB used, 20510 GB / 22300 GB avail; 2037B/s rd, 12813KB/s wr, 5op/s 2015-06-10 18:33:25.452221 mon.0 [INF] pgmap v6126612: 384 pgs: 384 active+clean; 453 GB data, 1789 GB used, 20510 GB / 22300 GB avail; 1530B/s rd, 8160KB/s wr, 2op/s 2015-06-10 18:33:26.455455 mon.0 [INF] pgmap v6126613: 384 pgs: 384 active+clean; 454 GB data, 1789 GB used, 20510 GB / 22300 GB avail; 12799KB/s wr, 4op/s 2015-06-10 18:33:28.479202 mon.0 [INF] pgmap v6126614: 384 pgs: 384 active+clean; 454 GB data, 1789 GB used, 20510 GB / 22300 GB avail; 12828KB/s wr, 4op/s 2015-06-10 18:33:30.501242 mon.0 [INF] pgmap v6126615: 384 pgs: 384 active+clean; 454 GB data, 1789 GB used, 20510 GB / 22300 GB avail; 5105KB/s wr, 1op/s 2015-06-10 18:33:31.504592 mon.0 [INF] pgmap v6126616: 384 pgs: 384 active+clean; 454 GB data, 1789 GB used, 20510 GB / 22300 GB avail; 14829KB/s wr, 4op/s 2015-06-10 18:33:33.524838 mon.0 [INF] pgmap v6126617: 384 pgs: 384 active+clean; 454 GB data, 1789 GB used, 20510 GB / 22300 GB avail; 2034B/s rd, 16863KB/s wr, 6op/s 2015-06-10 18:33:35.530814 mon.0 [INF] pgmap v6126618: 384 pgs: 384 active+clean; 454 GB data, 1789 GB used, 20510 GB / 22300 GB avail; 1526B/s rd, 6546KB/s wr, 2op/s 2015-06-10 18:33:36.535921 mon.0 [INF] pgmap v6126619: 384 pgs: 384 active+clean; 454 GB data, 1790 GB used, 20510 GB / 22300 GB avail; 13574KB/s wr, 3op/s comparing to the real data inbetween: 2015-06-10 18:30:55.489146 mon.0 [INF] pgmap v6126503: 384 pgs: 384 active+clean; 452 GB data, 1784 GB used, 20515 GB / 22300 GB avail; 6154KB/s rd, 6336KB/s wr, 9op/s 2015-06-10 18:30:56.496681 mon.0 [INF] pgmap v6126504: 384 pgs: 384 active+clean; 452 GB data, 1784 GB used, 20515 GB / 22300 GB avail; 2759KB/s rd, 5449KB/s wr, 9op/s 2015-06-10 18:30:58.506213 mon.0 [INF] pgmap v6126505: 384 pgs: 384 active+clean; 452 GB data, 1784 GB used, 20515 GB / 22300 GB avail; 5489KB/s rd, 9230KB/s wr, 9op/s 2015-06-10 18:31:00.512232 mon.0 [INF] pgmap v6126506: 384 pgs: 384 active+clean; 452 GB data, 1784 GB used, 20515 GB / 22300 GB avail; 6133KB/s rd, 6915KB/s wr, 5op/s 2015-06-10 18:31:01.521438 mon.0 [INF] pgmap v6126507: 384 pgs: 384 active+clean; 452 GB data, 1784 GB used, 20515 GB / 22300 GB avail; 9578KB/s rd, 9744KB/s wr, 7op/s 2015-06-10 18:31:03.526486 mon.0 [INF] pgmap v6126508: 384 pgs: 384 active+clean; 452 GB data, 1784 GB used, 20515 GB / 22300 GB avail; 7283KB/s rd, 8383KB/s wr, 17op/s 2015-06-10 18:31:05.532062 mon.0 [INF] pgmap v6126509: 384 pgs: 384 active+clean; 452 GB data, 1784 GB used, 20515 GB / 22300 GB avail; 3379KB/s rd, 4160KB/s wr, 11op/s 2015-06-10 18:31:06.535260 mon.0 [INF] pgmap v6126510: 384 pgs: 384 active+clean; 452 GB data, 1784 GB used, 20515 GB / 22300 GB avail; 7829KB/s rd, 8062KB/s wr, 11op/s 2015-06-10 18:31:08.548050 mon.0 [INF] pgmap v6126511: 384 pgs: 384 active+clean; 452 GB data, 1784 GB used, 20515 GB / 22300 GB avail; 7815KB/s rd, 6629KB/s wr, 10op/s The RBD backend should recognize discard for sure, so either there is something wrong with the particular driver coupled with the block job or with the discard logic. Sample invocation is given below: '{"execute":"drive-mirror", "arguments": { "device": "drive-virtio-disk0", "target": "rbd:dev-rack2/vm33090-dest:id=qemukvm:key=xxx:auth_supported=cephx\\;none:mon_host=10.6.0.1\\:6789\\;10.6.0.3\\:6789\\;10.6.0.4\\:6789", "mode": "existing", "sync": "full", "detect-zeroes": true, "format": "raw" } }' ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Qemu-devel] [PATCH 1/2] qapi: Add "detect-zeroes" option to drive-mirror 2015-06-10 15:41 ` Andrey Korolyov @ 2015-06-10 15:48 ` Andrey Korolyov 2015-06-10 16:04 ` Alexandre DERUMIER 0 siblings, 1 reply; 7+ messages in thread From: Andrey Korolyov @ 2015-06-10 15:48 UTC (permalink / raw) To: Fam Zheng Cc: Kevin Wolf, qemu block, Jeff Cody, qemu-devel@nongnu.org, Markus Armbruster, Stefan Hajnoczi, Paolo Bonzini > '{"execute":"drive-mirror", "arguments": { "device": > "drive-virtio-disk0", "target": > "rbd:dev-rack2/vm33090-dest:id=qemukvm:key=xxx:auth_supported=cephx\\;none:mon_host=10.6.0.1\\:6789\\;10.6.0.3\\:6789\\;10.6.0.4\\:6789", > "mode": "existing", "sync": "full", "detect-zeroes": true, "format": > "raw" } }' Sorry, forgot to mention - of course I`ve pulled all previous zeroing-related queue, so I haven`t had only the QMP-related fix running on top of the master :) ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Qemu-devel] [PATCH 1/2] qapi: Add "detect-zeroes" option to drive-mirror 2015-06-10 15:48 ` Andrey Korolyov @ 2015-06-10 16:04 ` Alexandre DERUMIER 2015-06-10 16:09 ` Andrey Korolyov 0 siblings, 1 reply; 7+ messages in thread From: Alexandre DERUMIER @ 2015-06-10 16:04 UTC (permalink / raw) To: Andrey Korolyov Cc: Kevin Wolf, Fam Zheng, qemu block, Jeff Cody, qemu-devel, Markus Armbruster, stefanha, pbonzini >>Sorry, forgot to mention - of course I`ve pulled all previous >>zeroing-related queue, so I haven`t had only the QMP-related fix >>running on top of the master :) Hi, I had a discussion about rbd mirroring, detect-zeroes and sparse target, some months ago with paolo http://qemu.11.n7.nabble.com/qemu-drive-mirror-to-rbd-storage-no-sparse-rbd-image-td293642.html Not sure it's related but "Lack of bdrv_co_write_zeroes is why detect-zeroes does not work. Lack of bdrv_get_block_status is why sparse->sparse does not work without detect-zeroes. Paolo " ----- Mail original ----- De: "Andrey Korolyov" <andrey@xdel.ru> À: "Fam Zheng" <famz@redhat.com> Cc: "Kevin Wolf" <kwolf@redhat.com>, "qemu block" <qemu-block@nongnu.org>, "Jeff Cody" <jcody@redhat.com>, "qemu-devel" <qemu-devel@nongnu.org>, "Markus Armbruster" <armbru@redhat.com>, "stefanha" <stefanha@redhat.com>, "pbonzini" <pbonzini@redhat.com> Envoyé: Mercredi 10 Juin 2015 17:48:54 Objet: Re: [Qemu-devel] [PATCH 1/2] qapi: Add "detect-zeroes" option to drive-mirror > '{"execute":"drive-mirror", "arguments": { "device": > "drive-virtio-disk0", "target": > "rbd:dev-rack2/vm33090-dest:id=qemukvm:key=xxx:auth_supported=cephx\\;none:mon_host=10.6.0.1\\:6789\\;10.6.0.3\\:6789\\;10.6.0.4\\:6789", > "mode": "existing", "sync": "full", "detect-zeroes": true, "format": > "raw" } }' Sorry, forgot to mention - of course I`ve pulled all previous zeroing-related queue, so I haven`t had only the QMP-related fix running on top of the master :) ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Qemu-devel] [PATCH 1/2] qapi: Add "detect-zeroes" option to drive-mirror 2015-06-10 16:04 ` Alexandre DERUMIER @ 2015-06-10 16:09 ` Andrey Korolyov 0 siblings, 0 replies; 7+ messages in thread From: Andrey Korolyov @ 2015-06-10 16:09 UTC (permalink / raw) To: Alexandre DERUMIER Cc: Kevin Wolf, Fam Zheng, qemu block, Jeff Cody, qemu-devel, Markus Armbruster, stefanha, pbonzini On Wed, Jun 10, 2015 at 7:04 PM, Alexandre DERUMIER <aderumier@odiso.com> wrote: >>>Sorry, forgot to mention - of course I`ve pulled all previous >>>zeroing-related queue, so I haven`t had only the QMP-related fix >>>running on top of the master :) > > Hi, I had a discussion about rbd mirroring, detect-zeroes and sparse target, some months ago with paolo > > http://qemu.11.n7.nabble.com/qemu-drive-mirror-to-rbd-storage-no-sparse-rbd-image-td293642.html > > Not sure it's related but > > "Lack of bdrv_co_write_zeroes is why detect-zeroes does not work. > > Lack of bdrv_get_block_status is why sparse->sparse does not work > without detect-zeroes. > Yes, I remember this one and the recent queue pull was the reason for me to re-test it, despite the current patch variant would more likely to be updated by comments from Eric, the behavior for non-file backends like rbd can/will stay the same. ^ permalink raw reply [flat|nested] 7+ messages in thread
* [Qemu-devel] [PATCH 2/2] iotests: Add test cases for drive-mirror "detect-zeroes" option 2015-06-08 7:06 [Qemu-devel] [PATCH 0/2] mirror: Allow detection of zeroes on source sectors Fam Zheng 2015-06-08 7:06 ` [Qemu-devel] [PATCH 1/2] qapi: Add "detect-zeroes" option to drive-mirror Fam Zheng @ 2015-06-08 7:06 ` Fam Zheng 1 sibling, 0 replies; 7+ messages in thread From: Fam Zheng @ 2015-06-08 7:06 UTC (permalink / raw) To: qemu-devel Cc: Kevin Wolf, qemu-block, Jeff Cody, Markus Armbruster, stefanha, pbonzini Signed-off-by: Fam Zheng <famz@redhat.com> --- tests/qemu-iotests/132 | 26 +++++++++++++++++++++++--- tests/qemu-iotests/132.out | 4 ++-- tests/qemu-iotests/iotests.py | 7 +++++++ 3 files changed, 32 insertions(+), 5 deletions(-) diff --git a/tests/qemu-iotests/132 b/tests/qemu-iotests/132 index f53ef6e..e506490 100644 --- a/tests/qemu-iotests/132 +++ b/tests/qemu-iotests/132 @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Test mirror with unmap +# Test mirror with unmap and zero source clusters # # Copyright (C) 2015 Red Hat, Inc. # @@ -21,7 +21,7 @@ import time import os import iotests -from iotests import qemu_img, qemu_io +from iotests import qemu_img, qemu_io, qemu_img_map test_img = os.path.join(iotests.test_dir, 'test.img') target_img = os.path.join(iotests.test_dir, 'target.img') @@ -55,5 +55,25 @@ class TestSingleDrive(iotests.QMPTestCase): self.assertTrue(iotests.compare_images(test_img, target_img), 'target image does not match source after mirroring') + def test_detect_zeroes(self): + self.vm.hmp_qemu_io('drive0', 'write -P 0 0 2M') + result = self.vm.qmp('drive-mirror', device='drive0', sync='full', + target=target_img, detect_zeroes=True) + self.assert_qmp(result, 'return', {}) + self.complete_and_wait('drive0') + self.vm.shutdown() + m = qemu_img_map(target_img) + self.assertTrue(m[0]["zero"]) + + def test_no_detect_zeroes(self): + self.vm.hmp_qemu_io('drive0', 'write -P 0 0 2M') + result = self.vm.qmp('drive-mirror', device='drive0', sync='full', + target=target_img, detect_zeroes=False) + self.assert_qmp(result, 'return', {}) + self.complete_and_wait('drive0') + self.vm.shutdown() + m = qemu_img_map(target_img) + self.assertFalse(m[0]["zero"]) + if __name__ == '__main__': - iotests.main(supported_fmts=['raw', 'qcow2']) + iotests.main(supported_fmts=['qcow2']) diff --git a/tests/qemu-iotests/132.out b/tests/qemu-iotests/132.out index ae1213e..8d7e996 100644 --- a/tests/qemu-iotests/132.out +++ b/tests/qemu-iotests/132.out @@ -1,5 +1,5 @@ -. +... ---------------------------------------------------------------------- -Ran 1 tests +Ran 3 tests OK diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index 8615b10..2ddc735 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -27,6 +27,7 @@ sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'scripts', ' import qmp import qtest import struct +import json __all__ = ['imgfmt', 'imgproto', 'test_dir' 'qemu_img', 'qemu_io', 'VM', 'QMPTestCase', 'notrun', 'main'] @@ -58,6 +59,12 @@ def qemu_img_pipe(*args): '''Run qemu-img and return its output''' return subprocess.Popen(qemu_img_args + list(args), stdout=subprocess.PIPE).communicate()[0] +def qemu_img_map(*args): + '''Run qemu-img map and return the result parsed from the json formated + output ''' + output = qemu_img_pipe(*(['map', '--output=json'] + list(args))) + return json.loads(output) + def qemu_io(*args): '''Run qemu-io and return the stdout data''' args = qemu_io_args + list(args) -- 2.4.2 ^ permalink raw reply related [flat|nested] 7+ messages in thread
end of thread, other threads:[~2015-06-10 16:09 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2015-06-08 7:06 [Qemu-devel] [PATCH 0/2] mirror: Allow detection of zeroes on source sectors Fam Zheng 2015-06-08 7:06 ` [Qemu-devel] [PATCH 1/2] qapi: Add "detect-zeroes" option to drive-mirror Fam Zheng 2015-06-10 15:41 ` Andrey Korolyov 2015-06-10 15:48 ` Andrey Korolyov 2015-06-10 16:04 ` Alexandre DERUMIER 2015-06-10 16:09 ` Andrey Korolyov 2015-06-08 7:06 ` [Qemu-devel] [PATCH 2/2] iotests: Add test cases for drive-mirror "detect-zeroes" option Fam Zheng
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).