* [PATCH v3 00/17] migration/snapshot: External snapshot utility
@ 2022-06-16 10:27 nikita.lapshin
2022-06-16 10:27 ` [PATCH v3 01/17] migration: Implemented new parameter stream_content nikita.lapshin
` (16 more replies)
0 siblings, 17 replies; 20+ messages in thread
From: nikita.lapshin @ 2022-06-16 10:27 UTC (permalink / raw)
To: qemu-devel; +Cc: den, andrey.drobyshev, quintela, dgilbert, nikita.lapshin
From: Nikita Lapshin <nikita.lapshin@openvz.org>
Changes v2 -> v3
* Refactored tool code to decrease duplications of migration code.
* Used sequential migration for saving which means that vmstate
will be send first and only after that ram part will be sent.
For this purpose stream-content-list paramter was used.
* Removed aio work with block driver. Should be replaced with
existing in qcow2 format in next versions.
* Removed postcopy percent. Should be added in next versions.
Changes v1 -> v2:
* Fixed CI checks
Changes v0 -> v1:
* Changed command-line format, now use blockdev specification to
define vmstate image.
* Don't deal with image creation in the tool, create externally.
* Better block layer AIO handling in the load path.
* Reduced fragmentation of the image backing file by using 'writtent-slice'
bitmaps in RAM blocks. Zero block write is issued to a never written slice
before the actual memory page write takes place.
* Improved load performance in postcopy by using 'loaded-slice' bitmaps
in RAM blocks.
* Refactored error handling/messages.
* Refactored namings.
This series is a kind of PoC for asynchronous snapshot reverting. It is
about external snapshots only and doesn't involve block devices. Thus, it's
mainly intended to be used with the new 'background-snapshot' migration
capability and otherwise standard QEMU migration mechanism.
The major ideas behind this version were:
* Make it compatible with 'exec:'-style migration - options can be create
some separate tool or integrate into qemu-system.
* Support asynchronous revert stage by using unaltered postcopy logic
at destination. To do this, we should be capable of saving RAM pages
so that any particular page can be directly addressed by it's block ID
and page offset. Possible solutions here seem to be:
use separate index (and storing it somewhere)
create sparse file on host FS and address pages with file offset
use QCOW2 (or other) image container with inherent sparsity support
* Make image file dense on the host FS so we don't depend on
copy/backup tools and how they deal with sparse files. Off course,
there's some performance cost for this choice.
* Try to keep page save latencies small while not degrading migration
bandwidth too much.
This version of snapshot-tool is the first step to integrate tool into
main QEMU. Now tool replace ram hanlers so it can call existing functions
in migration/* part to parse migration stream.
For the storage format, QCOW2 as a container and large (1MB) cluster size seem
to be an optimal choice. Larger cluster is beneficial for performance
particularly
in the case when image preallocation is disabled. Such cluster size does not
result
in too high internal fragmentation level (~10% of space waste in most cases) yet
allows to reduce significantly the number of expensive cluster allocations.
"stream-content-list"
There was no strict guarantee that there is no sections in ram part
rather than ram. So to solve this problem we decided to implement
parameters stream-content-list to provide such guarantee strictly.
This decision also helps with reusing of existed migration code.
You can see it in tool load part where tool disables all handlers except
ram using this parameter. If you have already seen it in previous patches you
can skip first 8 commits.
"sequential migration"
One problem remains unsolved. We need to run two migrations first to
save vmstate and second to save ram. We cannot run migration if VM is in
postmigrate state. But if we want to make snapshot this prohibition is
unnecessary so I changed some parts of migration and softmmu so
sequential migration become permitted. But that is not a solution. May
be new capability should be implementedi for that purpose.
Some of the upgrades were removed for now. This happened because of refactoring
and should be implemented in next versions.
How to use:
**Save:**
* > qemu-img create -f qcow2 -o size=<2_x_ram_size>,cluster_size=1M,
preallocation=off,refcount_bits=8 <image-filename>
* qemu> migrate_set_capability background-snapshot on
* #set SCL to "vmstate" only
* qemu> migrate "exec:qemu-snapshot --save-vmstate
<image-filename>,cache.direct=off,file.aio=threads"
* #set SCL to "ram" only
* qemu> migrate "exec:qemu-snapshot
<image-filename>,cache.direct=off,file.aio=threads"
**Load:**
* Use 'qemu-system-* -incoming defer'
* qemu> migrate_incoming "exec:qemu-snapshot --revert
<image-filename>,cache.direct=on,file.aio=native"
**Load with postcopy:**
* Use 'qemu-system-* -incoming defer'
* qemu> migrate_set_capability postcopy-ram on
* qemu> migrate_incoming "exec:qemu-snapshot --revert --postcopy
<image-filename>,cache.direct=on,file.aio=native"
Nikita Lapshin (17):
migration: Implemented new parameter stream_content
migration: should_skip() implemented
migration: Add vmstate part of migration stream
igration: Add dirty-bitmaps part of migration stream
Add block part of migration stream
migration: Add RAM part of migration stream
migration: analyze-migration script changed
migration: Test for RAM and vmstate parts
migration/snapshot: Introduce qemu-snapshot tool
migration/snapshot: Build changes for qemu-snapshot-tool
migration/qemu-file: Fix qemu_ftell() for non-writable file
migration/snapshot: Move RAM_SAVE_FLAG_xxx defines to migration/ram.h
migration/snapshot: Block layer support in qemu-snapshot
migration/snpashot: Implement API for RAMBlock
migration/snapshot: Save part implement
migration/snapshot: Precopy load implemented
migration/snapshot: Postcopy load implemented
include/qemu-snapshot.h | 94 ++
meson.build | 18 +
migration/meson.build | 4 +-
migration/migration.c | 199 ++-
migration/migration.h | 4 +
migration/qemu-file.c | 3 +-
migration/qemu-snapshot-io.c | 112 ++
migration/qemu-snapshot.c | 1126 +++++++++++++++++
migration/ram.c | 22 +-
migration/ram.h | 16 +
migration/savevm.c | 116 +-
migration/savevm.h | 8 +
qapi/migration.json | 21 +-
qemu-snapshot.c | 540 ++++++++
scripts/analyze-migration.py | 19 +-
.../tests/migrate-ram-stream-content-test | 96 ++
.../tests/migrate-ram-stream-content-test.out | 5 +
17 files changed, 2342 insertions(+), 61 deletions(-)
create mode 100644 include/qemu-snapshot.h
create mode 100644 migration/qemu-snapshot-io.c
create mode 100644 migration/qemu-snapshot.c
create mode 100644 qemu-snapshot.c
create mode 100755 tests/qemu-iotests/tests/migrate-ram-stream-content-test
create mode 100644 tests/qemu-iotests/tests/migrate-ram-stream-content-test.out
--
2.31.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v3 01/17] migration: Implemented new parameter stream_content
2022-06-16 10:27 [PATCH v3 00/17] migration/snapshot: External snapshot utility nikita.lapshin
@ 2022-06-16 10:27 ` nikita.lapshin
2022-06-16 10:27 ` [PATCH v3 02/17] migration: should_skip() implemented nikita.lapshin
` (15 subsequent siblings)
16 siblings, 0 replies; 20+ messages in thread
From: nikita.lapshin @ 2022-06-16 10:27 UTC (permalink / raw)
To: qemu-devel; +Cc: den, andrey.drobyshev, quintela, dgilbert, nikita.lapshin
From: Nikita Lapshin <nikita.lapshin@openvz.org>
This new optional parameter contains inormation about migration
stream parts to be sent (such as RAM, block, bitmap). This looks
better than using capabilities to solve problem of dividing
migration stream.
Signed-off-by: Nikita Lapshin <nikita.lapshin@openvz.org>
---
migration/migration.c | 47 ++++++++++++++++++++++++++++++++++++++++++-
migration/migration.h | 2 ++
qapi/migration.json | 21 ++++++++++++++++---
3 files changed, 66 insertions(+), 4 deletions(-)
diff --git a/migration/migration.c b/migration/migration.c
index 695f0f2900..4adcc87d1d 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -1334,6 +1334,12 @@ void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params,
}
}
+static bool check_stream_parts(strList *stream_content_list)
+{
+ /* To be implemented in ext commits */
+ return true;
+}
+
/*
* Check whether the parameters are valid. Error will be put into errp
* (if provided). Return true if valid, otherwise false.
@@ -1482,7 +1488,12 @@ static bool migrate_params_check(MigrationParameters *params, Error **errp)
return false;
}
- return true;
+ if (params->has_stream_content_list &&
+ !check_stream_parts(params->stream_content_list)) {
+ error_setg(errp, "Invalid parts of stream given for stream-content");
+ }
+
+ return true;
}
static void migrate_params_test_apply(MigrateSetParameters *params,
@@ -1581,6 +1592,11 @@ static void migrate_params_test_apply(MigrateSetParameters *params,
dest->has_block_bitmap_mapping = true;
dest->block_bitmap_mapping = params->block_bitmap_mapping;
}
+
+ if (params->has_stream_content_list) {
+ dest->has_stream_content_list = true;
+ dest->stream_content_list = params->stream_content_list;
+ }
}
static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
@@ -1703,6 +1719,13 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
QAPI_CLONE(BitmapMigrationNodeAliasList,
params->block_bitmap_mapping);
}
+
+ if (params->has_stream_content_list) {
+ qapi_free_strList(s->parameters.stream_content_list);
+ s->parameters.has_stream_content_list = true;
+ s->parameters.stream_content_list =
+ QAPI_CLONE(strList, params->stream_content_list);
+ }
}
void qmp_migrate_set_parameters(MigrateSetParameters *params, Error **errp)
@@ -2620,6 +2643,28 @@ bool migrate_background_snapshot(void)
return s->enabled_capabilities[MIGRATION_CAPABILITY_BACKGROUND_SNAPSHOT];
}
+/* Checks if stream-content parameter has section_name in list */
+bool migrate_find_stream_content(const char *section_name)
+{
+ MigrationState *s;
+
+ s = migrate_get_current();
+
+ if (!s->parameters.has_stream_content_list) {
+ return false;
+ }
+
+ strList *list = s->parameters.stream_content_list;
+
+ for (; list; list = list->next) {
+ if (!strcmp(list->value, section_name)) {
+ return true;
+ }
+ }
+
+ return false;
+}
+
/* migration thread support */
/*
* Something bad happened to the RP stream, mark an error
diff --git a/migration/migration.h b/migration/migration.h
index 2de861df01..411c58e919 100644
--- a/migration/migration.h
+++ b/migration/migration.h
@@ -396,6 +396,8 @@ bool migrate_use_events(void);
bool migrate_postcopy_blocktime(void);
bool migrate_background_snapshot(void);
+bool migrate_find_stream_content(const char *section_name);
+
/* Sending on the return path - generic and then for each message type */
void migrate_send_rp_shut(MigrationIncomingState *mis,
uint32_t value);
diff --git a/qapi/migration.json b/qapi/migration.json
index 18e2610e88..80acf6dbc3 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -760,6 +760,12 @@
# block device name if there is one, and to their node name
# otherwise. (Since 5.2)
#
+# @stream-content-list: Parameter control content of migration stream such as RAM,
+# vmstate, block and dirty-bitmaps. This is optional parameter
+# so migration will work correctly without it.
+# This parameter takes string list as description of content
+# and include that part of migration stream. (Since 7.0)
+#
# Features:
# @unstable: Member @x-checkpoint-delay is experimental.
#
@@ -780,7 +786,8 @@
'xbzrle-cache-size', 'max-postcopy-bandwidth',
'max-cpu-throttle', 'multifd-compression',
'multifd-zlib-level' ,'multifd-zstd-level',
- 'block-bitmap-mapping' ] }
+ 'block-bitmap-mapping',
+ 'stream-content-list' ] }
##
# @MigrateSetParameters:
@@ -925,6 +932,12 @@
# block device name if there is one, and to their node name
# otherwise. (Since 5.2)
#
+# @stream-content-list: Parameter control content of migration stream such as RAM,
+# vmstate, block and dirty-bitmaps. This is optional parameter
+# so migration will work correctly without it.
+# This parameter takes string list as description of content
+# and include that part of migration stream. (Since 7.0)
+#
# Features:
# @unstable: Member @x-checkpoint-delay is experimental.
#
@@ -960,7 +973,8 @@
'*multifd-compression': 'MultiFDCompression',
'*multifd-zlib-level': 'uint8',
'*multifd-zstd-level': 'uint8',
- '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ] } }
+ '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ],
+ '*stream-content-list': [ 'str' ] } }
##
# @migrate-set-parameters:
@@ -1158,7 +1172,8 @@
'*multifd-compression': 'MultiFDCompression',
'*multifd-zlib-level': 'uint8',
'*multifd-zstd-level': 'uint8',
- '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ] } }
+ '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ],
+ '*stream-content-list': [ 'str' ] } }
##
# @query-migrate-parameters:
--
2.31.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3 02/17] migration: should_skip() implemented
2022-06-16 10:27 [PATCH v3 00/17] migration/snapshot: External snapshot utility nikita.lapshin
2022-06-16 10:27 ` [PATCH v3 01/17] migration: Implemented new parameter stream_content nikita.lapshin
@ 2022-06-16 10:27 ` nikita.lapshin
2022-06-16 10:27 ` [PATCH v3 03/17] migration: Add vmstate part of migration stream nikita.lapshin
` (14 subsequent siblings)
16 siblings, 0 replies; 20+ messages in thread
From: nikita.lapshin @ 2022-06-16 10:27 UTC (permalink / raw)
To: qemu-devel; +Cc: den, andrey.drobyshev, quintela, dgilbert, nikita.lapshin
From: Nikita Lapshin <nikita.lapshin@openvz.org>
For next changes it is convenient to make all decisions about
sections skipping in one function.
Signed-off-by: Nikita Lapshin <nikita.lapshin@openvz.org>
---
migration/savevm.c | 54 ++++++++++++++++++++++++----------------------
1 file changed, 28 insertions(+), 26 deletions(-)
diff --git a/migration/savevm.c b/migration/savevm.c
index 02ed94c180..c68f187ef7 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -943,6 +943,15 @@ static int vmstate_save(QEMUFile *f, SaveStateEntry *se,
return vmstate_save_state(f, se->vmsd, se->opaque, vmdesc);
}
+static bool should_skip(SaveStateEntry *se)
+{
+ if (se->ops && se->ops->is_active && !se->ops->is_active(se->opaque)) {
+ return true;
+ }
+
+ return false;
+}
+
/*
* Write the header for device section (QEMU_VM_SECTION START/END/PART/FULL)
*/
@@ -1207,10 +1216,8 @@ void qemu_savevm_state_setup(QEMUFile *f)
if (!se->ops || !se->ops->save_setup) {
continue;
}
- if (se->ops->is_active) {
- if (!se->ops->is_active(se->opaque)) {
- continue;
- }
+ if (should_skip(se)) {
+ continue;
}
save_section_header(f, se, QEMU_VM_SECTION_START);
@@ -1238,10 +1245,8 @@ int qemu_savevm_state_resume_prepare(MigrationState *s)
if (!se->ops || !se->ops->resume_prepare) {
continue;
}
- if (se->ops->is_active) {
- if (!se->ops->is_active(se->opaque)) {
- continue;
- }
+ if (should_skip(se)) {
+ continue;
}
ret = se->ops->resume_prepare(s, se->opaque);
if (ret < 0) {
@@ -1268,8 +1273,7 @@ int qemu_savevm_state_iterate(QEMUFile *f, bool postcopy)
if (!se->ops || !se->ops->save_live_iterate) {
continue;
}
- if (se->ops->is_active &&
- !se->ops->is_active(se->opaque)) {
+ if (should_skip(se)) {
continue;
}
if (se->ops->is_active_iterate &&
@@ -1337,10 +1341,8 @@ void qemu_savevm_state_complete_postcopy(QEMUFile *f)
if (!se->ops || !se->ops->save_live_complete_postcopy) {
continue;
}
- if (se->ops->is_active) {
- if (!se->ops->is_active(se->opaque)) {
- continue;
- }
+ if (should_skip(se)) {
+ continue;
}
trace_savevm_section_start(se->idstr, se->section_id);
/* Section type */
@@ -1374,10 +1376,8 @@ int qemu_savevm_state_complete_precopy_iterable(QEMUFile *f, bool in_postcopy)
continue;
}
- if (se->ops->is_active) {
- if (!se->ops->is_active(se->opaque)) {
- continue;
- }
+ if (should_skip(se)) {
+ continue;
}
trace_savevm_section_start(se->idstr, se->section_id);
@@ -1417,6 +1417,9 @@ int qemu_savevm_state_complete_precopy_non_iterable(QEMUFile *f,
trace_savevm_section_skip(se->idstr, se->section_id);
continue;
}
+ if (should_skip(se)) {
+ continue;
+ }
trace_savevm_section_start(se->idstr, se->section_id);
@@ -1522,10 +1525,8 @@ void qemu_savevm_state_pending(QEMUFile *f, uint64_t threshold_size,
if (!se->ops || !se->ops->save_live_pending) {
continue;
}
- if (se->ops->is_active) {
- if (!se->ops->is_active(se->opaque)) {
- continue;
- }
+ if (should_skip(se)) {
+ continue;
}
se->ops->save_live_pending(f, se->opaque, threshold_size,
res_precopy_only, res_compatible,
@@ -1635,6 +1636,9 @@ int qemu_save_device_state(QEMUFile *f)
if (se->vmsd && !vmstate_save_needed(se->vmsd, se->opaque)) {
continue;
}
+ if (should_skip(se)) {
+ continue;
+ }
save_section_header(f, se, QEMU_VM_SECTION_FULL);
@@ -2542,10 +2546,8 @@ static int qemu_loadvm_state_setup(QEMUFile *f)
if (!se->ops || !se->ops->load_setup) {
continue;
}
- if (se->ops->is_active) {
- if (!se->ops->is_active(se->opaque)) {
- continue;
- }
+ if (should_skip(se)) {
+ continue;
}
ret = se->ops->load_setup(f, se->opaque);
--
2.31.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3 03/17] migration: Add vmstate part of migration stream
2022-06-16 10:27 [PATCH v3 00/17] migration/snapshot: External snapshot utility nikita.lapshin
2022-06-16 10:27 ` [PATCH v3 01/17] migration: Implemented new parameter stream_content nikita.lapshin
2022-06-16 10:27 ` [PATCH v3 02/17] migration: should_skip() implemented nikita.lapshin
@ 2022-06-16 10:27 ` nikita.lapshin
2022-06-16 10:27 ` [PATCH v3 04/17] migration: Add dirty-bitmaps " nikita.lapshin
` (13 subsequent siblings)
16 siblings, 0 replies; 20+ messages in thread
From: nikita.lapshin @ 2022-06-16 10:27 UTC (permalink / raw)
To: qemu-devel; +Cc: den, andrey.drobyshev, quintela, dgilbert, nikita.lapshin
From: Nikita Lapshin <nikita.lapshin@openvz.org>
Now we can disable and enable vmstate part by stream_content parameter.
Signed-off-by: Nikita Lapshin <nikita.lapshin@openvz.org>
---
migration/migration.c | 10 ++++++++--
migration/savevm.c | 15 +++++++++++++++
2 files changed, 23 insertions(+), 2 deletions(-)
diff --git a/migration/migration.c b/migration/migration.c
index 4adcc87d1d..bbf9b6aad1 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -1334,9 +1334,15 @@ void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params,
}
}
-static bool check_stream_parts(strList *stream_content_list)
+static bool check_stream_parts(strList *stream_list)
{
- /* To be implemented in ext commits */
+ for (; stream_list; stream_list = stream_list->next) {
+ if (!strcmp(stream_list->value, "vmstate")) {
+ continue;
+ }
+
+ return false;
+ }
return true;
}
diff --git a/migration/savevm.c b/migration/savevm.c
index c68f187ef7..48603517ba 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -949,6 +949,21 @@ static bool should_skip(SaveStateEntry *se)
return true;
}
+ /*
+ * Assume that any SaveStateEntry with non-null vmsd is
+ * part of vmstate.
+ *
+ *
+ * Vmstate is included by default so firstly check if
+ * stream-content-list is enabled.
+ */
+
+ if (se->vmsd &&
+ migrate_get_current()->parameters.has_stream_content_list &&
+ !migrate_find_stream_content("vmstate")) {
+ return true;
+ }
+
return false;
}
--
2.31.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3 04/17] migration: Add dirty-bitmaps part of migration stream
2022-06-16 10:27 [PATCH v3 00/17] migration/snapshot: External snapshot utility nikita.lapshin
` (2 preceding siblings ...)
2022-06-16 10:27 ` [PATCH v3 03/17] migration: Add vmstate part of migration stream nikita.lapshin
@ 2022-06-16 10:27 ` nikita.lapshin
2022-06-16 10:27 ` [PATCH v3 05/17] migration: Add block " nikita.lapshin
` (12 subsequent siblings)
16 siblings, 0 replies; 20+ messages in thread
From: nikita.lapshin @ 2022-06-16 10:27 UTC (permalink / raw)
To: qemu-devel; +Cc: den, andrey.drobyshev, quintela, dgilbert, nikita.lapshin
From: Nikita Lapshin <nikita.lapshin@openvz.org>
This patch enables and disable dirty-bitmaps in migration stream.
Signed-off-by: Nikita Lapshin <nikita.lapshin@openvz.org>
---
migration/migration.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/migration/migration.c b/migration/migration.c
index bbf9b6aad1..ad789915ce 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -1337,7 +1337,8 @@ void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params,
static bool check_stream_parts(strList *stream_list)
{
for (; stream_list; stream_list = stream_list->next) {
- if (!strcmp(stream_list->value, "vmstate")) {
+ if (!strcmp(stream_list->value, "vmstate") ||
+ !strcmp(stream_list->value, "dirty-bitmaps")) {
continue;
}
@@ -2501,7 +2502,8 @@ bool migrate_dirty_bitmaps(void)
s = migrate_get_current();
- return s->enabled_capabilities[MIGRATION_CAPABILITY_DIRTY_BITMAPS];
+ return s->enabled_capabilities[MIGRATION_CAPABILITY_DIRTY_BITMAPS] ||
+ migrate_find_stream_content("dirty-bitmaps");
}
bool migrate_ignore_shared(void)
--
2.31.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3 05/17] migration: Add block part of migration stream
2022-06-16 10:27 [PATCH v3 00/17] migration/snapshot: External snapshot utility nikita.lapshin
` (3 preceding siblings ...)
2022-06-16 10:27 ` [PATCH v3 04/17] migration: Add dirty-bitmaps " nikita.lapshin
@ 2022-06-16 10:27 ` nikita.lapshin
2022-06-16 10:28 ` [PATCH v3 06/17] migration: Add RAM " nikita.lapshin
` (11 subsequent siblings)
16 siblings, 0 replies; 20+ messages in thread
From: nikita.lapshin @ 2022-06-16 10:27 UTC (permalink / raw)
To: qemu-devel; +Cc: den, andrey.drobyshev, quintela, dgilbert, nikita.lapshin
From: Nikita Lapshin <nikita.lapshin@openvz.org>
This patch enables and disable block in migration stream.
Signed-off-by: Nikita Lapshin <nikita.lapshin@openvz.org>
---
migration/migration.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/migration/migration.c b/migration/migration.c
index ad789915ce..d81f3c6891 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -1338,7 +1338,8 @@ static bool check_stream_parts(strList *stream_list)
{
for (; stream_list; stream_list = stream_list->next) {
if (!strcmp(stream_list->value, "vmstate") ||
- !strcmp(stream_list->value, "dirty-bitmaps")) {
+ !strcmp(stream_list->value, "dirty-bitmaps") ||
+ !strcmp(stream_list->value, "block")) {
continue;
}
@@ -2621,7 +2622,8 @@ bool migrate_use_block(void)
s = migrate_get_current();
- return s->enabled_capabilities[MIGRATION_CAPABILITY_BLOCK];
+ return s->enabled_capabilities[MIGRATION_CAPABILITY_BLOCK] ||
+ migrate_find_stream_content("block");
}
bool migrate_use_return_path(void)
--
2.31.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3 06/17] migration: Add RAM part of migration stream
2022-06-16 10:27 [PATCH v3 00/17] migration/snapshot: External snapshot utility nikita.lapshin
` (4 preceding siblings ...)
2022-06-16 10:27 ` [PATCH v3 05/17] migration: Add block " nikita.lapshin
@ 2022-06-16 10:28 ` nikita.lapshin
2022-06-16 10:28 ` [PATCH v3 07/17] migration: analyze-migration script changed nikita.lapshin
` (10 subsequent siblings)
16 siblings, 0 replies; 20+ messages in thread
From: nikita.lapshin @ 2022-06-16 10:28 UTC (permalink / raw)
To: qemu-devel; +Cc: den, andrey.drobyshev, quintela, dgilbert, nikita.lapshin
From: Nikita Lapshin <nikita.lapshin@openvz.org>
'ram' parameter enable RAM sections in migration stream. If it
isn't specified it will be skipped.
Signed-off-by: Nikita Lapshin <nikita.lapshin@openvz.org>
---
migration/migration.c | 17 ++++++++++++++++-
migration/migration.h | 1 +
migration/ram.c | 6 ++++++
3 files changed, 23 insertions(+), 1 deletion(-)
diff --git a/migration/migration.c b/migration/migration.c
index d81f3c6891..6528b3ad41 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -1339,7 +1339,8 @@ static bool check_stream_parts(strList *stream_list)
for (; stream_list; stream_list = stream_list->next) {
if (!strcmp(stream_list->value, "vmstate") ||
!strcmp(stream_list->value, "dirty-bitmaps") ||
- !strcmp(stream_list->value, "block")) {
+ !strcmp(stream_list->value, "block") ||
+ !strcmp(stream_list->value, "ram")) {
continue;
}
@@ -2653,6 +2654,20 @@ bool migrate_background_snapshot(void)
return s->enabled_capabilities[MIGRATION_CAPABILITY_BACKGROUND_SNAPSHOT];
}
+bool migrate_ram(void)
+{
+ MigrationState *s;
+
+ s = migrate_get_current();
+
+ /*
+ * By default RAM is enabled so if stream-content-list disabled
+ * RAM will be passed.
+ */
+ return !s->parameters.has_stream_content_list ||
+ migrate_find_stream_content("ram");
+}
+
/* Checks if stream-content parameter has section_name in list */
bool migrate_find_stream_content(const char *section_name)
{
diff --git a/migration/migration.h b/migration/migration.h
index 411c58e919..5c43788a2b 100644
--- a/migration/migration.h
+++ b/migration/migration.h
@@ -395,6 +395,7 @@ int migrate_decompress_threads(void);
bool migrate_use_events(void);
bool migrate_postcopy_blocktime(void);
bool migrate_background_snapshot(void);
+bool migrate_ram(void);
bool migrate_find_stream_content(const char *section_name);
diff --git a/migration/ram.c b/migration/ram.c
index 170e522a1f..ddc7abd08a 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -4263,6 +4263,11 @@ static int ram_resume_prepare(MigrationState *s, void *opaque)
return 0;
}
+static bool is_ram_active(void *opaque)
+{
+ return migrate_ram();
+}
+
static SaveVMHandlers savevm_ram_handlers = {
.save_setup = ram_save_setup,
.save_live_iterate = ram_save_iterate,
@@ -4275,6 +4280,7 @@ static SaveVMHandlers savevm_ram_handlers = {
.load_setup = ram_load_setup,
.load_cleanup = ram_load_cleanup,
.resume_prepare = ram_resume_prepare,
+ .is_active = is_ram_active,
};
static void ram_mig_ram_block_resized(RAMBlockNotifier *n, void *host,
--
2.31.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3 07/17] migration: analyze-migration script changed
2022-06-16 10:27 [PATCH v3 00/17] migration/snapshot: External snapshot utility nikita.lapshin
` (5 preceding siblings ...)
2022-06-16 10:28 ` [PATCH v3 06/17] migration: Add RAM " nikita.lapshin
@ 2022-06-16 10:28 ` nikita.lapshin
2022-06-16 10:28 ` [PATCH v3 08/17] migration: Test for RAM and vmstate parts nikita.lapshin
` (9 subsequent siblings)
16 siblings, 0 replies; 20+ messages in thread
From: nikita.lapshin @ 2022-06-16 10:28 UTC (permalink / raw)
To: qemu-devel; +Cc: den, andrey.drobyshev, quintela, dgilbert, nikita.lapshin
From: Nikita Lapshin <nikita.lapshin@openvz.org>
This script is used for RAM capabilities test. But it cannot work
in case of no vm description in migration stream.
So new flag is added to allow work this script with ram-only
migration stream.
Signed-off-by: Nikita Lapshin <nikita.lapshin@openvz.org>
---
scripts/analyze-migration.py | 19 ++++++++++++-------
1 file changed, 12 insertions(+), 7 deletions(-)
diff --git a/scripts/analyze-migration.py b/scripts/analyze-migration.py
index b82a1b0c58..80077a09bc 100755
--- a/scripts/analyze-migration.py
+++ b/scripts/analyze-migration.py
@@ -495,7 +495,7 @@ def __init__(self, filename):
self.filename = filename
self.vmsd_desc = None
- def read(self, desc_only = False, dump_memory = False, write_memory = False):
+ def read(self, ram_only, desc_only = False, dump_memory = False, write_memory = False):
# Read in the whole file
file = MigrationFile(self.filename)
@@ -509,7 +509,8 @@ def read(self, desc_only = False, dump_memory = False, write_memory = False):
if data != self.QEMU_VM_FILE_VERSION:
raise Exception("Invalid version number %d" % data)
- self.load_vmsd_json(file)
+ if not ram_only:
+ self.load_vmsd_json(file)
# Read sections
self.sections = collections.OrderedDict()
@@ -518,7 +519,10 @@ def read(self, desc_only = False, dump_memory = False, write_memory = False):
return
ramargs = {}
- ramargs['page_size'] = self.vmsd_desc['page_size']
+ if ram_only:
+ ramargs['page_size'] = 4096
+ else:
+ ramargs['page_size'] = self.vmsd_desc['page_size']
ramargs['dump_memory'] = dump_memory
ramargs['write_memory'] = write_memory
self.section_classes[('ram',0)][1] = ramargs
@@ -579,6 +583,7 @@ def default(self, o):
parser.add_argument("-m", "--memory", help='dump RAM contents as well', action='store_true')
parser.add_argument("-d", "--dump", help='what to dump ("state" or "desc")', default='state')
parser.add_argument("-x", "--extract", help='extract contents into individual files', action='store_true')
+parser.add_argument("--ram-only", help='parse migration dump containing only RAM', action='store_true')
args = parser.parse_args()
jsonenc = JSONEncoder(indent=4, separators=(',', ': '))
@@ -586,14 +591,14 @@ def default(self, o):
if args.extract:
dump = MigrationDump(args.file)
- dump.read(desc_only = True)
+ dump.read(desc_only = True, ram_only = args.ram_only)
print("desc.json")
f = open("desc.json", "w")
f.truncate()
f.write(jsonenc.encode(dump.vmsd_desc))
f.close()
- dump.read(write_memory = True)
+ dump.read(write_memory = True, ram_only = args.ram_only)
dict = dump.getDict()
print("state.json")
f = open("state.json", "w")
@@ -602,12 +607,12 @@ def default(self, o):
f.close()
elif args.dump == "state":
dump = MigrationDump(args.file)
- dump.read(dump_memory = args.memory)
+ dump.read(dump_memory = args.memory, ram_only = args.ram_only)
dict = dump.getDict()
print(jsonenc.encode(dict))
elif args.dump == "desc":
dump = MigrationDump(args.file)
- dump.read(desc_only = True)
+ dump.read(desc_only = True, ram_only = args.ram_only)
print(jsonenc.encode(dump.vmsd_desc))
else:
raise Exception("Please specify either -x, -d state or -d desc")
--
2.31.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3 08/17] migration: Test for RAM and vmstate parts
2022-06-16 10:27 [PATCH v3 00/17] migration/snapshot: External snapshot utility nikita.lapshin
` (6 preceding siblings ...)
2022-06-16 10:28 ` [PATCH v3 07/17] migration: analyze-migration script changed nikita.lapshin
@ 2022-06-16 10:28 ` nikita.lapshin
2022-06-16 10:28 ` [PATCH v3 09/17] migration/snapshot: Introduce qemu-snapshot tool nikita.lapshin
` (8 subsequent siblings)
16 siblings, 0 replies; 20+ messages in thread
From: nikita.lapshin @ 2022-06-16 10:28 UTC (permalink / raw)
To: qemu-devel; +Cc: den, andrey.drobyshev, quintela, dgilbert, nikita.lapshin
From: Nikita Lapshin <nikita.lapshin@openvz.org>
All other parts works just like existed capabilities. Thus there is
no need to make new tests. Though RAM and vmstate are new so here
is new test for that parts.
Signed-off-by: Nikita Lapshin <nikita.lapshin@openvz.org>
---
.../tests/migrate-ram-stream-content-test | 96 +++++++++++++++++++
.../tests/migrate-ram-stream-content-test.out | 5 +
2 files changed, 101 insertions(+)
create mode 100755 tests/qemu-iotests/tests/migrate-ram-stream-content-test
create mode 100644 tests/qemu-iotests/tests/migrate-ram-stream-content-test.out
diff --git a/tests/qemu-iotests/tests/migrate-ram-stream-content-test b/tests/qemu-iotests/tests/migrate-ram-stream-content-test
new file mode 100755
index 0000000000..2855ca4a64
--- /dev/null
+++ b/tests/qemu-iotests/tests/migrate-ram-stream-content-test
@@ -0,0 +1,96 @@
+#!/usr/bin/env python3
+# group: rw migration
+#
+# Tests for 'no-ram' and 'ram-only' capabilities
+#
+# Copyright (c) 2021 Virtuozzo International GmbH.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+import os
+import json
+import subprocess
+import iotests
+
+img = os.path.join(iotests.test_dir, 'disk.img')
+
+class TestRamCapabilities(iotests.QMPTestCase):
+ def setUp(self):
+ iotests.qemu_img('create', '-f', iotests.imgfmt, img, '10M')
+ self.vm = iotests.VM()
+ self.vm.launch()
+ self.vm.qmp('migrate-set-capabilities', capabilities=[
+ {
+ 'capability': 'events',
+ 'state': True
+ }
+ ])
+
+ def tearDown(self):
+ self.vm.shutdown()
+ os.remove(img)
+
+ def check_ram_only(self, output):
+ str_json = output.decode()
+ json_obj = json.loads(str_json)
+
+ success = False
+ for key in json_obj:
+ self.assertTrue("ram" in key)
+ success = True
+ self.assertTrue(success)
+
+ def run_migration(self, no_ram, tmp_stream):
+ if no_ram:
+ output = self.vm.qmp('migrate-set-parameters',
+ stream_content_list = ['vmstate'])
+ else:
+ self.vm.qmp('migrate-set-parameters',
+ stream_content_list = ['ram'])
+
+ self.vm.qmp('migrate', uri='exec:cat>' + tmp_stream)
+
+ while True:
+ event = self.vm.event_wait('MIGRATION')
+
+ if event['data']['status'] == 'completed':
+ break
+
+
+ def test_no_ram(self):
+ with iotests.FilePath('tmp_stream') as tmp_stream:
+ self.run_migration(True, tmp_stream)
+ output = subprocess.run(
+ ['../../../scripts/analyze-migration.py', '-f', tmp_stream],
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT,
+ check=False).stdout
+
+ self.assertFalse('ram' in output.decode())
+
+ def test_ram_only(self):
+ with iotests.FilePath('tmp_stream') as tmp_stream:
+ self.run_migration(False, tmp_stream)
+ output = subprocess.run(
+ ['../../../scripts/analyze-migration.py', '-f', tmp_stream,
+ '--ram-only'],
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT,
+ check=False).stdout
+
+ self.check_ram_only(output)
+
+if __name__ == '__main__':
+ iotests.main(supported_protocols=['file'])
diff --git a/tests/qemu-iotests/tests/migrate-ram-stream-content-test.out b/tests/qemu-iotests/tests/migrate-ram-stream-content-test.out
new file mode 100644
index 0000000000..fbc63e62f8
--- /dev/null
+++ b/tests/qemu-iotests/tests/migrate-ram-stream-content-test.out
@@ -0,0 +1,5 @@
+..
+----------------------------------------------------------------------
+Ran 2 tests
+
+OK
--
2.31.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3 09/17] migration/snapshot: Introduce qemu-snapshot tool
2022-06-16 10:27 [PATCH v3 00/17] migration/snapshot: External snapshot utility nikita.lapshin
` (7 preceding siblings ...)
2022-06-16 10:28 ` [PATCH v3 08/17] migration: Test for RAM and vmstate parts nikita.lapshin
@ 2022-06-16 10:28 ` nikita.lapshin
2022-06-16 10:28 ` [PATCH v3 10/17] migration/snapshot: Build changes for qemu-snapshot-tool nikita.lapshin
` (7 subsequent siblings)
16 siblings, 0 replies; 20+ messages in thread
From: nikita.lapshin @ 2022-06-16 10:28 UTC (permalink / raw)
To: qemu-devel; +Cc: den, andrey.drobyshev, quintela, dgilbert, nikita.lapshin
From: Nikita Lapshin <nikita.lapshin@openvz.org>
Execution environment, command-line argument parsing, usage/version info etc.
Signed-off-by: Nikita Lapshin <nikita.lapshin@openvz.org>
---
include/qemu-snapshot.h | 65 ++++++
migration/qemu-snapshot.c | 57 +++++
qemu-snapshot.c | 433 ++++++++++++++++++++++++++++++++++++++
3 files changed, 555 insertions(+)
create mode 100644 include/qemu-snapshot.h
create mode 100644 migration/qemu-snapshot.c
create mode 100644 qemu-snapshot.c
diff --git a/include/qemu-snapshot.h b/include/qemu-snapshot.h
new file mode 100644
index 0000000000..8e548e7630
--- /dev/null
+++ b/include/qemu-snapshot.h
@@ -0,0 +1,65 @@
+/*
+ * QEMU External Snapshot Utility
+ *
+ * Copyright Virtuozzo GmbH, 2021
+ *
+ * Authors:
+ * Andrey Gruzdev <andrey.gruzdev@virtuozzo.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later. See the COPYING file in the top-level directory.
+ */
+
+#ifndef QEMU_SNAPSHOT_H
+#define QEMU_SNAPSHOT_H
+
+/* Invalid offset */
+#define INVALID_OFFSET -1
+/* Maximum byte count for qemu_get_buffer_in_place() */
+#define INPLACE_READ_MAX (32768 - 4096)
+
+/* Backing cluster size */
+#define BDRV_CLUSTER_SIZE (1024 * 1024)
+
+/* Minimum supported target page size */
+#define PAGE_SIZE_MIN 4096
+/*
+ * Maximum supported target page size. The limit is caused by using
+ * QEMUFile and qemu_get_buffer_in_place() on migration channel.
+ * IO_BUF_SIZE is currently 32KB.
+ */
+#define PAGE_SIZE_MAX 16384
+/* RAM slice size for snapshot saving */
+#define SLICE_SIZE PAGE_SIZE_MAX
+/* RAM slice size for snapshot revert */
+#define SLICE_SIZE_REVERT (16 * PAGE_SIZE_MAX)
+
+typedef struct StateInfo {
+ int64_t page_size;
+ int64_t page_mask;
+ int page_bits;
+ int64_t slice_size;
+ int64_t slice_mask;
+ int slice_bits;
+} StateInfo;
+
+typedef struct StateSaveCtx {
+ BlockBackend *blk; /* Block backend */
+
+ StateInfo state_parameters; /* Migration state info*/
+} StateSaveCtx;
+
+typedef struct StateLoadCtx {
+ BlockBackend *blk; /* Block backend */
+
+ StateInfo state_parameters; /* Migration state info*/
+} StateLoadCtx;
+
+void ram_init_state(void);
+void ram_destroy_state(void);
+StateSaveCtx *get_save_context(void);
+StateLoadCtx *get_load_context(void);
+int coroutine_fn save_state_main(StateSaveCtx *s);
+int coroutine_fn load_state_main(StateLoadCtx *s);
+
+#endif /* QEMU_SNAPSHOT_H */
diff --git a/migration/qemu-snapshot.c b/migration/qemu-snapshot.c
new file mode 100644
index 0000000000..f7695e75c7
--- /dev/null
+++ b/migration/qemu-snapshot.c
@@ -0,0 +1,57 @@
+/*
+ * QEMU External Snapshot Utility
+ *
+ * Copyright Virtuozzo GmbH, 2021
+ *
+ * Authors:
+ * Andrey Gruzdev <andrey.gruzdev@virtuozzo.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later. See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "sysemu/block-backend.h"
+#include "qemu/coroutine.h"
+#include "qemu/cutils.h"
+#include "qemu/bitmap.h"
+#include "qemu/error-report.h"
+#include "io/channel-buffer.h"
+#include "migration/qemu-file-channel.h"
+#include "migration/qemu-file.h"
+#include "migration/savevm.h"
+#include "migration/ram.h"
+#include "qemu-snapshot.h"
+
+/* RAM transfer context */
+typedef struct RAMCtx {
+ int64_t normal_pages; /* Total number of normal pages */
+} RAMCtx;
+
+static RAMCtx ram_ctx;
+
+int coroutine_fn save_state_main(StateSaveCtx *s)
+{
+ /* TODO: implement */
+ return 0;
+}
+
+int coroutine_fn load_state_main(StateLoadCtx *s)
+{
+ /* TODO: implement */
+ return 0;
+}
+
+/* Initialize snapshot RAM state */
+void ram_init_state(void)
+{
+ RAMCtx *ram = &ram_ctx;
+
+ memset(ram, 0, sizeof(ram_ctx));
+}
+
+/* Destroy snapshot RAM state */
+void ram_destroy_state(void)
+{
+ /* TODO: implement */
+}
diff --git a/qemu-snapshot.c b/qemu-snapshot.c
new file mode 100644
index 0000000000..683f1b265a
--- /dev/null
+++ b/qemu-snapshot.c
@@ -0,0 +1,433 @@
+/*
+ * QEMU External Snapshot Utility
+ *
+ * Copyright Virtuozzo GmbH, 2021
+ *
+ * Authors:
+ * Andrey Gruzdev <andrey.gruzdev@virtuozzo.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later. See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include <getopt.h>
+
+#include "qemu/memalign.h"
+#include "qemu-common.h"
+#include "qemu-version.h"
+#include "qapi/error.h"
+#include "qapi/qmp/qdict.h"
+#include "sysemu/sysemu.h"
+#include "sysemu/block-backend.h"
+#include "qemu/cutils.h"
+#include "qemu/coroutine.h"
+#include "qemu/error-report.h"
+#include "qemu/config-file.h"
+#include "qemu/log.h"
+#include "qemu/option_int.h"
+#include "qemu/main-loop.h"
+#include "trace/control.h"
+#include "io/channel-util.h"
+#include "io/channel-buffer.h"
+#include "migration/qemu-file-channel.h"
+#include "migration/qemu-file.h"
+#include "migration/savevm.h"
+#include "migration/misc.h"
+#include "qemu-snapshot.h"
+
+int64_t page_size;
+int64_t page_mask;
+int page_bits;
+int64_t slice_size;
+int64_t slice_mask;
+int slice_bits;
+
+static QemuOptsList snap_blk_optslist = {
+ .name = "blockdev",
+ .implied_opt_name = "file.filename",
+ .head = QTAILQ_HEAD_INITIALIZER(snap_blk_optslist.head),
+ .desc = {
+ { /*End of the list */ }
+ },
+};
+
+static struct {
+ bool revert; /* Operation is snapshot revert */
+
+ int fd; /* Migration channel fd */
+ int rp_fd; /* Return path fd (for postcopy) */
+
+ const char *blk_optstr; /* Command-line options for vmstate blockdev */
+ QDict *blk_options; /* Blockdev options */
+ int blk_flags; /* Blockdev flags */
+
+ bool postcopy; /* Use postcopy */
+ bool save_vmstate; /* If true tool will get onle vmstate part */
+} params;
+
+static StateSaveCtx state_save_ctx;
+static StateLoadCtx state_load_ctx;
+
+static enum {
+ RUNNING = 0,
+ TERMINATED
+} state;
+
+StateSaveCtx *get_save_context(void)
+{
+ return &state_save_ctx;
+}
+
+StateLoadCtx *get_load_context(void)
+{
+ return &state_load_ctx;
+}
+
+static void init_save_context(void)
+{
+ memset(&state_save_ctx, 0, sizeof(state_save_ctx));
+}
+
+static void destroy_save_context(void)
+{
+ /* TODO: implement */
+}
+
+static void init_load_context(void)
+{
+ memset(&state_load_ctx, 0, sizeof(state_load_ctx));
+}
+
+static void destroy_load_context(void)
+{
+ /* TODO: implement */
+}
+
+static BlockBackend *image_open_opts(const char *optstr, QDict *options,
+ int flags)
+{
+ BlockBackend *blk;
+ Error *local_err = NULL;
+
+ /* Open image and create block backend */
+ blk = blk_new_open(NULL, NULL, options, flags, &local_err);
+ if (!blk) {
+ error_reportf_err(local_err, "Failed to open image '%s': ", optstr);
+ return NULL;
+ }
+
+ blk_set_enable_write_cache(blk, true);
+
+ return blk;
+}
+
+/* Use BH to enter coroutine from the main loop */
+static void enter_co_bh(void *opaque)
+{
+ Coroutine *co = (Coroutine *) opaque;
+ qemu_coroutine_enter(co);
+}
+
+static void coroutine_fn snapshot_save_co(void *opaque)
+{
+ StateSaveCtx *s = get_save_context();
+ int res = -1;
+ init_save_context();
+
+ /* Block backend */
+ s->blk = image_open_opts(params.blk_optstr, params.blk_options,
+ params.blk_flags);
+ if (!s->blk) {
+ goto fail;
+ }
+
+ res = save_state_main(s);
+ if (res) {
+ error_report("Failed to save snapshot: %s", strerror(-res));
+
+fail:
+ destroy_save_context();
+ state = TERMINATED;
+}
+
+static void coroutine_fn snapshot_load_co(void *opaque)
+{
+ StateLoadCtx *s = get_load_context();
+ int res = -1;
+
+ init_load_context();
+
+ /* Block backend */
+ s->blk = image_open_opts(params.blk_optstr, params.blk_options,
+ params.blk_flags);
+ if (!s->blk) {
+ goto fail;
+ }
+
+ res = load_state_main(s);
+ if (res) {
+ error_report("Failed to load snapshot: %s", strerror(-res));
+ }
+
+fail:
+ destroy_load_context();
+ state = TERMINATED;
+}
+
+static void usage(const char *name)
+{
+ printf(
+ "Usage: %s [options] <image-blockspec>\n"
+ "QEMU External Snapshot Utility\n"
+ "\n"
+ "'image-blockspec' is a block device specification for vmstate image\n"
+ "\n"
+ " -h, --help display this help and exit\n"
+ " -V, --version output version information and exit\n"
+ "\n"
+ "Options:\n"
+ " -T, --trace [[enable=]<pattern>][,events=<file>][,file=<file>]\n"
+ " specify tracing options\n"
+ " -r, --revert revert to snapshot\n"
+ " --uri=fd:<fd> specify migration fd\n"
+ " --page-size=<size> specify target page size\n"
+ " --postcopy load ram in postcopy mode\n"
+ "\n"
+ QEMU_HELP_BOTTOM "\n", name);
+}
+
+static void version(const char *name)
+{
+ printf(
+ "%s " QEMU_FULL_VERSION "\n"
+ "Written by Andrey Gruzdev.\n"
+ "\n"
+ QEMU_COPYRIGHT "\n"
+ "This is free software; see the source for copying conditions. There is NO\n"
+ "warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.\n",
+ name);
+}
+
+enum {
+ OPTION_PAGE_SIZE = 256,
+ OPTION_POSTCOPY,
+ OPTION_URI,
+};
+
+static void process_options(int argc, char *argv[], StateInfo *si)
+{
+ static const char *s_opt = "rhVT:";
+ static const struct option l_opt[] = {
+ { "page-size", required_argument, NULL, OPTION_PAGE_SIZE },
+ { "postcopy", no_argument, NULL, OPTION_POSTCOPY },
+ { "uri", required_argument, NULL, OPTION_URI },
+ { "revert", no_argument, NULL, 'r' },
+ { "help", no_argument, NULL, 'h' },
+ { "version", no_argument, NULL, 'V' },
+ { "trace", required_argument, NULL, 'T' },
+ { "save-vmstate", no_argument, NULL, 'S' },
+ { NULL, 0, NULL, 0 }
+ };
+
+ bool has_page_size = false;
+ bool has_uri = false;
+
+ long target_page_size = qemu_real_host_page_size;
+ int uri_fd = -1;
+ bool revert = false;
+ bool postcopy = false;
+ const char *blk_optstr;
+ QemuOpts *blk_opts;
+ QDict *blk_options;
+ int c;
+
+ while ((c = getopt_long(argc, argv, s_opt, l_opt, NULL)) != -1) {
+ switch (c) {
+ case '?':
+ exit(EXIT_FAILURE);
+
+ case 'h':
+ usage(argv[0]);
+ exit(EXIT_SUCCESS);
+
+ case 'V':
+ version(argv[0]);
+ exit(EXIT_SUCCESS);
+
+ case 'T':
+ trace_opt_parse(optarg);
+ break;
+
+ case 'r':
+ if (revert) {
+ error_report("-r and --revert can only be specified once");
+ exit(EXIT_FAILURE);
+ }
+ revert = true;
+
+ break;
+
+ case 'S':
+ params.save_vmstate = true;
+
+ break;
+
+ case OPTION_POSTCOPY:
+ {
+ if (postcopy) {
+ error_report("--postcopy can only be specified once");
+ exit(EXIT_FAILURE);
+ }
+ postcopy = true;
+
+ break;
+ }
+
+ case OPTION_PAGE_SIZE:
+ {
+ const char *r;
+
+ if (has_page_size) {
+ error_report("--page-size can only be specified once");
+ exit(EXIT_FAILURE);
+ }
+ has_page_size = true;
+
+ qemu_strtol(optarg, &r, 0, &target_page_size);
+ if (*r != '\0' ||
+ (target_page_size & (target_page_size - 1)) != 0 ||
+ target_page_size < PAGE_SIZE_MIN ||
+ target_page_size > PAGE_SIZE_MAX) {
+ error_report("Invalid argument to --page-size");
+ exit(EXIT_FAILURE);
+ }
+
+ break;
+ }
+
+ case OPTION_URI:
+ {
+ const char *p;
+
+ if (has_uri) {
+ error_report("--uri can only be specified once");
+ exit(EXIT_FAILURE);
+ }
+ has_uri = true;
+
+ /* Only "--uri=fd:<fd>" is currently supported */
+ if (strstart(optarg, "fd:", &p)) {
+ const char *r;
+ long fd;
+
+ qemu_strtol(p, &r, 10, &fd);
+ if (*r != '\0' || fd <= STDERR_FILENO) {
+ error_report("Invalid FD value");
+ exit(EXIT_FAILURE);
+ }
+
+ uri_fd = qemu_dup_flags(fd, O_CLOEXEC);
+ if (uri_fd < 0) {
+ error_report("Could not dup FD %ld", fd);
+ exit(EXIT_FAILURE);
+ }
+
+ /* Close original fd */
+ close(fd);
+ } else {
+ error_report("Invalid argument to --uri");
+ exit(EXIT_FAILURE);
+ }
+
+ break;
+ }
+
+ default:
+ g_assert_not_reached();
+ }
+ }
+
+ if ((argc - optind) != 1) {
+ error_report("Invalid number of arguments");
+ exit(EXIT_FAILURE);
+ }
+
+ blk_optstr = argv[optind];
+
+ blk_opts = qemu_opts_parse_noisily(&snap_blk_optslist, blk_optstr, true);
+ if (!blk_opts) {
+ exit(EXIT_FAILURE);
+ }
+ blk_options = qemu_opts_to_qdict(blk_opts, NULL);
+ qemu_opts_reset(&snap_blk_optslist);
+
+ /* Enforced block layer options */
+ qdict_put_str(blk_options, "driver", "qcow2");
+ qdict_put_null(blk_options, "backing");
+ qdict_put_str(blk_options, "overlap-check", "none");
+ qdict_put_str(blk_options, "auto-read-only", "off");
+ qdict_put_str(blk_options, "detect-zeroes", "off");
+ qdict_put_str(blk_options, "lazy-refcounts", "on");
+ qdict_put_str(blk_options, "file.auto-read-only", "off");
+ qdict_put_str(blk_options, "file.detect-zeroes", "off");
+
+ params.revert = revert;
+
+ if (uri_fd != -1) {
+ params.fd = params.rp_fd = uri_fd;
+ } else {
+ params.fd = revert ? STDOUT_FILENO : STDIN_FILENO;
+ params.rp_fd = revert ? STDIN_FILENO : -1;
+ }
+ params.blk_optstr = blk_optstr;
+ params.blk_options = blk_options;
+ params.blk_flags = revert ? 0 : BDRV_O_RDWR;
+ params.postcopy = postcopy;
+
+ si->page_size = target_page_size;
+ si->page_mask = ~(target_page_size - 1);
+ si->page_bits = ctz64(target_page_size);
+ si->slice_size = SLICE_SIZE;
+ si->slice_mask = ~(si->slice_size - 1);
+ si->slice_bits = ctz64(si->slice_size);
+}
+
+int main(int argc, char **argv)
+{
+ Coroutine *co;
+ StateInfo state_info;
+
+ os_setup_early_signal_handling();
+ os_setup_signal_handling();
+ error_init(argv[0]);
+ qemu_init_exec_dir(argv[0]);
+ module_call_init(MODULE_INIT_QOM);
+ qemu_init_main_loop(&error_fatal);
+ qemu_init_subsystems();
+
+ migration_object_init();
+ qemu_add_opts(&qemu_trace_opts);
+ process_options(argc, argv, &state_info);
+
+ if (!trace_init_backends()) {
+ exit(EXIT_FAILURE);
+ }
+ trace_init_file();
+ qemu_set_log(LOG_TRACE);
+
+ ram_init_state();
+
+ if (params.revert) {
+ co = qemu_coroutine_create(snapshot_load_co, &state_info);
+ } else {
+ co = qemu_coroutine_create(snapshot_save_co, &state_info);
+ }
+ aio_bh_schedule_oneshot(qemu_get_aio_context(), enter_co_bh, co);
+
+ do {
+ main_loop_wait(false);
+ } while (state != TERMINATED);
+
+ exit(EXIT_SUCCESS);
+}
--
2.31.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3 10/17] migration/snapshot: Build changes for qemu-snapshot-tool
2022-06-16 10:27 [PATCH v3 00/17] migration/snapshot: External snapshot utility nikita.lapshin
` (8 preceding siblings ...)
2022-06-16 10:28 ` [PATCH v3 09/17] migration/snapshot: Introduce qemu-snapshot tool nikita.lapshin
@ 2022-06-16 10:28 ` nikita.lapshin
2022-06-16 10:28 ` [PATCH v3 11/17] migration/qemu-file: Fix qemu_ftell() for non-writable file nikita.lapshin
` (6 subsequent siblings)
16 siblings, 0 replies; 20+ messages in thread
From: nikita.lapshin @ 2022-06-16 10:28 UTC (permalink / raw)
To: qemu-devel; +Cc: den, andrey.drobyshev, quintela, dgilbert, nikita.lapshin
From: Nikita Lapshin <nikita.lapshin@openvz.org>
Tool uses part of migration code thus new lib with stubs for migration
code is added here.
Signed-off-by: Nikita Lapshin <nikita.lapshin@openvz.org>
---
meson.build | 18 ++++++++++++++++++
migration/meson.build | 3 ++-
2 files changed, 20 insertions(+), 1 deletion(-)
diff --git a/meson.build b/meson.build
index 2d6601467f..ba3b9ab4bd 100644
--- a/meson.build
+++ b/meson.build
@@ -3287,6 +3287,16 @@ if xkbcommon.found()
endif
if have_tools
+ tool_inc = include_directories('include/hw/core', 'include')
+ lib_tool = static_library('snapshot-tool',
+ sources: [migration_files],
+ dependencies: arch_deps,
+ include_directories: [tool_inc, target_inc],
+ name_suffix: 'fa',
+ build_by_default: false)
+ snapshot_tool = declare_dependency(link_with: lib_tool,
+ dependencies: [zlib, qom, io])
+
qemu_img = executable('qemu-img', [files('qemu-img.c'), hxdep],
dependencies: [authz, block, crypto, io, qom, qemuutil], install: true)
qemu_io = executable('qemu-io', files('qemu-io.c'),
@@ -3294,6 +3304,14 @@ if have_tools
qemu_nbd = executable('qemu-nbd', files('qemu-nbd.c'),
dependencies: [blockdev, qemuutil, gnutls, selinux],
install: true)
+ qemu_snapshot = executable('qemu-snapshot',
+ files('qemu-snapshot.c'),
+ dependencies: [snapshot_tool] + arch_deps + deps,
+ objects: lib.extract_all_objects(recursive: true),
+ link_depends: [block_syms, qemu_syms],
+ link_args: link_args,
+
+ install: true)
subdir('storage-daemon')
subdir('contrib/rdmacm-mux')
diff --git a/migration/meson.build b/migration/meson.build
index 8b5ca5c047..13498a6db3 100644
--- a/migration/meson.build
+++ b/migration/meson.build
@@ -7,6 +7,8 @@ migration_files = files(
'qemu-file-channel.c',
'qemu-file.c',
'yank_functions.c',
+ 'migration.c',
+ 'qemu-snapshot.c',
)
softmmu_ss.add(migration_files)
@@ -18,7 +20,6 @@ softmmu_ss.add(files(
'exec.c',
'fd.c',
'global_state.c',
- 'migration.c',
'multifd.c',
'multifd-zlib.c',
'postcopy-ram.c',
--
2.31.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3 11/17] migration/qemu-file: Fix qemu_ftell() for non-writable file
2022-06-16 10:27 [PATCH v3 00/17] migration/snapshot: External snapshot utility nikita.lapshin
` (9 preceding siblings ...)
2022-06-16 10:28 ` [PATCH v3 10/17] migration/snapshot: Build changes for qemu-snapshot-tool nikita.lapshin
@ 2022-06-16 10:28 ` nikita.lapshin
2022-06-16 11:20 ` Daniel P. Berrangé
2022-06-16 10:28 ` [PATCH v3 12/17] migration/snapshot: Move RAM_SAVE_FLAG_xxx defines to migration/ram.h nikita.lapshin
` (5 subsequent siblings)
16 siblings, 1 reply; 20+ messages in thread
From: nikita.lapshin @ 2022-06-16 10:28 UTC (permalink / raw)
To: qemu-devel; +Cc: den, andrey.drobyshev, quintela, dgilbert, nikita.lapshin
From: Nikita Lapshin <nikita.lapshin@openvz.org>
qemu_ftell() will return wrong value for non-writable QEMUFile.
This happens due to call qemu_fflush() inside qemu_ftell(), this
function won't flush if file is readable.
Signed-off-by: Nikita Lapshin <nikita.lapshin@openvz.org>
---
migration/qemu-file.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/migration/qemu-file.c b/migration/qemu-file.c
index 1479cddad9..53ccef80ac 100644
--- a/migration/qemu-file.c
+++ b/migration/qemu-file.c
@@ -663,7 +663,8 @@ int64_t qemu_ftell_fast(QEMUFile *f)
int64_t qemu_ftell(QEMUFile *f)
{
qemu_fflush(f);
- return f->pos;
+ /* Consider that qemu_fflush() won't work if file is non-writable */
+ return f->pos + f->buf_index;
}
int qemu_file_rate_limit(QEMUFile *f)
--
2.31.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3 12/17] migration/snapshot: Move RAM_SAVE_FLAG_xxx defines to migration/ram.h
2022-06-16 10:27 [PATCH v3 00/17] migration/snapshot: External snapshot utility nikita.lapshin
` (10 preceding siblings ...)
2022-06-16 10:28 ` [PATCH v3 11/17] migration/qemu-file: Fix qemu_ftell() for non-writable file nikita.lapshin
@ 2022-06-16 10:28 ` nikita.lapshin
2022-06-16 10:28 ` [PATCH v3 13/17] migration/snapshot: Block layer support in qemu-snapshot nikita.lapshin
` (4 subsequent siblings)
16 siblings, 0 replies; 20+ messages in thread
From: nikita.lapshin @ 2022-06-16 10:28 UTC (permalink / raw)
To: qemu-devel; +Cc: den, andrey.drobyshev, quintela, dgilbert, nikita.lapshin
From: Nikita Lapshin <nikita.lapshin@openvz.org>
Move RAM_SAVE_FLAG_xxx defines from migration/ram.c to migration/ram.h
Signed-off-by: Nikita Lapshin <nikita.lapshin@openvz.org>
---
migration/ram.c | 16 ----------------
migration/ram.h | 16 ++++++++++++++++
2 files changed, 16 insertions(+), 16 deletions(-)
diff --git a/migration/ram.c b/migration/ram.c
index ddc7abd08a..da7c7ec0e5 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -66,22 +66,6 @@
/***********************************************************/
/* ram save/restore */
-/* RAM_SAVE_FLAG_ZERO used to be named RAM_SAVE_FLAG_COMPRESS, it
- * worked for pages that where filled with the same char. We switched
- * it to only search for the zero value. And to avoid confusion with
- * RAM_SSAVE_FLAG_COMPRESS_PAGE just rename it.
- */
-
-#define RAM_SAVE_FLAG_FULL 0x01 /* Obsolete, not used anymore */
-#define RAM_SAVE_FLAG_ZERO 0x02
-#define RAM_SAVE_FLAG_MEM_SIZE 0x04
-#define RAM_SAVE_FLAG_PAGE 0x08
-#define RAM_SAVE_FLAG_EOS 0x10
-#define RAM_SAVE_FLAG_CONTINUE 0x20
-#define RAM_SAVE_FLAG_XBZRLE 0x40
-/* 0x80 is reserved in migration.h start with 0x100 next */
-#define RAM_SAVE_FLAG_COMPRESS_PAGE 0x100
-
XBZRLECacheStats xbzrle_counters;
/* struct contains XBZRLE cache and a static page
diff --git a/migration/ram.h b/migration/ram.h
index 2c6dc3675d..9dddfd381a 100644
--- a/migration/ram.h
+++ b/migration/ram.h
@@ -33,6 +33,22 @@
#include "exec/cpu-common.h"
#include "io/channel.h"
+/* RAM_SAVE_FLAG_ZERO used to be named RAM_SAVE_FLAG_COMPRESS, it
+ * worked for pages that where filled with the same char. We switched
+ * it to only search for the zero value. And to avoid confusion with
+ * RAM_SSAVE_FLAG_COMPRESS_PAGE just rename it.
+ */
+
+#define RAM_SAVE_FLAG_FULL 0x01 /* Obsolete, not used anymore */
+#define RAM_SAVE_FLAG_ZERO 0x02
+#define RAM_SAVE_FLAG_MEM_SIZE 0x04
+#define RAM_SAVE_FLAG_PAGE 0x08
+#define RAM_SAVE_FLAG_EOS 0x10
+#define RAM_SAVE_FLAG_CONTINUE 0x20
+#define RAM_SAVE_FLAG_XBZRLE 0x40
+/* 0x80 is reserved in migration.h start with 0x100 next */
+#define RAM_SAVE_FLAG_COMPRESS_PAGE 0x100
+
extern MigrationStats ram_counters;
extern XBZRLECacheStats xbzrle_counters;
extern CompressionStats compression_counters;
--
2.31.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3 13/17] migration/snapshot: Block layer support in qemu-snapshot
2022-06-16 10:27 [PATCH v3 00/17] migration/snapshot: External snapshot utility nikita.lapshin
` (11 preceding siblings ...)
2022-06-16 10:28 ` [PATCH v3 12/17] migration/snapshot: Move RAM_SAVE_FLAG_xxx defines to migration/ram.h nikita.lapshin
@ 2022-06-16 10:28 ` nikita.lapshin
2022-06-16 10:28 ` [PATCH v3 14/17] migration/snpashot: Implement API for RAMBlock nikita.lapshin
` (3 subsequent siblings)
16 siblings, 0 replies; 20+ messages in thread
From: nikita.lapshin @ 2022-06-16 10:28 UTC (permalink / raw)
To: qemu-devel; +Cc: den, andrey.drobyshev, quintela, dgilbert, nikita.lapshin
From: Nikita Lapshin <nikita.lapshin@openvz.org>
This commit enables few functions to simplify block layer work
for qemu-snapshot tool.
Signed-off-by: Nikita Lapshin <nikita.lapshin@openvz.org>
---
include/qemu-snapshot.h | 3 +
migration/meson.build | 1 +
migration/qemu-snapshot-io.c | 112 +++++++++++++++++++++++++++++++++++
3 files changed, 116 insertions(+)
create mode 100644 migration/qemu-snapshot-io.c
diff --git a/include/qemu-snapshot.h b/include/qemu-snapshot.h
index 8e548e7630..be2557f6a0 100644
--- a/include/qemu-snapshot.h
+++ b/include/qemu-snapshot.h
@@ -62,4 +62,7 @@ StateLoadCtx *get_load_context(void);
int coroutine_fn save_state_main(StateSaveCtx *s);
int coroutine_fn load_state_main(StateLoadCtx *s);
+QEMUFile *qemu_fopen_bdrv_vmstate(BlockDriverState *bs, int is_writable);
+void qemu_fsplice(QEMUFile *f_dst, QEMUFile *f_src, size_t size);
+size_t qemu_fsplice_tail(QEMUFile *f_dst, QEMUFile *f_src);
#endif /* QEMU_SNAPSHOT_H */
diff --git a/migration/meson.build b/migration/meson.build
index 13498a6db3..3a04576c30 100644
--- a/migration/meson.build
+++ b/migration/meson.build
@@ -9,6 +9,7 @@ migration_files = files(
'yank_functions.c',
'migration.c',
'qemu-snapshot.c',
+ 'qemu-snapshot-io.c'
)
softmmu_ss.add(migration_files)
diff --git a/migration/qemu-snapshot-io.c b/migration/qemu-snapshot-io.c
new file mode 100644
index 0000000000..904cb92c84
--- /dev/null
+++ b/migration/qemu-snapshot-io.c
@@ -0,0 +1,112 @@
+/*
+ * QEMU External Snapshot Utility
+ *
+ * Copyright Virtuozzo GmbH, 2021
+ *
+ * Authors:
+ * Andrey Gruzdev <andrey.gruzdev@virtuozzo.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later. See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu/coroutine.h"
+#include "sysemu/block-backend.h"
+#include "migration/qemu-file.h"
+#include "qemu-snapshot.h"
+
+static ssize_t bdrv_vmstate_get_buffer(void *opaque, uint8_t *buf, int64_t pos,
+ size_t size, Error **errp)
+{
+ return bdrv_load_vmstate((BlockDriverState *) opaque, buf, pos, size);
+}
+
+static ssize_t bdrv_vmstate_writev_buffer(void *opaque, struct iovec *iov,
+ int iovcnt, int64_t pos, Error **errp)
+{
+ QEMUIOVector qiov;
+ int res;
+
+ qemu_iovec_init_external(&qiov, iov, iovcnt);
+
+ res = bdrv_writev_vmstate((BlockDriverState *) opaque, &qiov, pos);
+ if (res < 0) {
+ return res;
+ }
+
+ return qiov.size;
+}
+
+static int bdrv_vmstate_fclose(void *opaque, Error **errp)
+{
+ return bdrv_flush((BlockDriverState *) opaque);
+}
+
+static const QEMUFileOps bdrv_vmstate_read_ops = {
+ .get_buffer = bdrv_vmstate_get_buffer,
+ .close = bdrv_vmstate_fclose,
+};
+
+static const QEMUFileOps bdrv_vmstate_write_ops = {
+ .writev_buffer = bdrv_vmstate_writev_buffer,
+ .close = bdrv_vmstate_fclose,
+};
+
+/* Create QEMUFile to access vmstate stream on QCOW2 image */
+QEMUFile *qemu_fopen_bdrv_vmstate(BlockDriverState *bs, int is_writable)
+{
+ if (is_writable) {
+ return qemu_fopen_ops(bs, &bdrv_vmstate_write_ops, true);
+ }
+
+ return qemu_fopen_ops(bs, &bdrv_vmstate_read_ops, true);
+}
+
+/* Move number of bytes from the source QEMUFile to destination */
+void qemu_fsplice(QEMUFile *f_dst, QEMUFile *f_src, size_t size)
+{
+ size_t rest = size;
+
+ while (rest) {
+ uint8_t *ptr = NULL;
+ size_t req_size;
+ size_t count;
+
+ req_size = MIN(rest, INPLACE_READ_MAX);
+ count = qemu_peek_buffer(f_src, &ptr, req_size, 0);
+ qemu_file_skip(f_src, count);
+
+ qemu_put_buffer(f_dst, ptr, count);
+ rest -= count;
+ }
+}
+
+/*
+ * Move data from source QEMUFile to destination
+ * until EOF is reached on source.
+ */
+size_t qemu_fsplice_tail(QEMUFile *f_dst, QEMUFile *f_src)
+{
+ bool eof = false;
+ size_t res = 0;
+
+ while (!eof) {
+ const size_t size = INPLACE_READ_MAX;
+ uint8_t *buffer = NULL;
+ size_t count;
+
+ count = qemu_peek_buffer(f_src, &buffer, size, 0);
+ qemu_file_skip(f_src, count);
+
+ /* Reached EOF on source? */
+ if (count != size) {
+ eof = true;
+ }
+
+ qemu_put_buffer(f_dst, buffer, count);
+ res += count;
+ }
+
+ return res;
+}
--
2.31.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3 14/17] migration/snpashot: Implement API for RAMBlock
2022-06-16 10:27 [PATCH v3 00/17] migration/snapshot: External snapshot utility nikita.lapshin
` (12 preceding siblings ...)
2022-06-16 10:28 ` [PATCH v3 13/17] migration/snapshot: Block layer support in qemu-snapshot nikita.lapshin
@ 2022-06-16 10:28 ` nikita.lapshin
2022-06-16 10:28 ` [PATCH v3 15/17] migration/snapshot: Save part implement nikita.lapshin
` (2 subsequent siblings)
16 siblings, 0 replies; 20+ messages in thread
From: nikita.lapshin @ 2022-06-16 10:28 UTC (permalink / raw)
To: qemu-devel; +Cc: den, andrey.drobyshev, quintela, dgilbert, nikita.lapshin
From: Nikita Lapshin <nikita.lapshin@openvz.org>
Implemented RAMBlock is used for managing ram block from VM.
This structure is close to existing RAMBlock in migration but
has few differences.
May be it should be replaced with existing RAMBlock it can lead to
lots of reuses.
Signed-off-by: Nikita Lapshin <nikita.lapshin@openvz.org>
---
migration/qemu-snapshot.c | 180 ++++++++++++++++++++++++++++++++++++++
1 file changed, 180 insertions(+)
diff --git a/migration/qemu-snapshot.c b/migration/qemu-snapshot.c
index f7695e75c7..394c6acb77 100644
--- a/migration/qemu-snapshot.c
+++ b/migration/qemu-snapshot.c
@@ -23,13 +23,193 @@
#include "migration/ram.h"
#include "qemu-snapshot.h"
+/* RAM block */
+/* TODO RAMBlock should be replace with existing struct RAMBlock in ram.c */
+typedef struct RAMBlock {
+ int64_t bdrv_offset; /* Offset on backing storage */
+ int64_t length; /* Length */
+ int64_t nr_pages; /* Page count */
+ int64_t nr_slices; /* Number of slices (for bitmap bookkeeping) */
+ int64_t discard_offset; /* Used for postcopy dicarding of ram blocks */
+
+ unsigned long *bitmap; /* Bitmap of RAM slices */
+
+ /* Link into ram_list */
+ QSIMPLEQ_ENTRY(RAMBlock) next;
+
+ char idstr[256]; /* RAM block id string */
+} RAMBlock;
+
/* RAM transfer context */
typedef struct RAMCtx {
int64_t normal_pages; /* Total number of normal pages */
+
+ /* RAM block list head */
+ QSIMPLEQ_HEAD(, RAMBlock) ram_block_list;
+
} RAMCtx;
static RAMCtx ram_ctx;
+static inline
+bool ram_offset_in_block(RAMBlock *block, int64_t offset)
+{
+ return block && offset < block->length;
+}
+
+static inline
+bool ram_bdrv_offset_in_block(RAMBlock *block, int64_t bdrv_offset)
+{
+ return block && bdrv_offset >= block->bdrv_offset &&
+ bdrv_offset < block->bdrv_offset + block->length;
+}
+
+static inline
+int64_t ram_bdrv_from_block_offset(RAMBlock *block, int64_t offset)
+{
+ if (!ram_offset_in_block(block, offset)) {
+ return INVALID_OFFSET;
+ }
+
+ return block->bdrv_offset + offset;
+}
+
+static inline
+int64_t ram_block_offset_from_bdrv(RAMBlock *block, int64_t bdrv_offset)
+{
+ int64_t offset;
+
+ if (!block) {
+ return INVALID_OFFSET;
+ }
+
+ offset = bdrv_offset - block->bdrv_offset;
+ return offset >= 0 ? offset : INVALID_OFFSET;
+}
+
+static RAMBlock *ram_block_by_idstr(const char *idstr)
+{
+ RAMBlock *block;
+
+ QSIMPLEQ_FOREACH(block, &ram_ctx.ram_block_list, next) {
+ if (!strcmp(idstr, block->idstr)) {
+ return block;
+ }
+ }
+
+ return NULL;
+}
+
+/*
+ * Assume QEMUFile is migration stream and try to get ram block from it.
+ * Also check if this ram block exists.
+ */
+static RAMBlock *ram_block_from_stream(QEMUFile *f, int flags)
+{
+ static RAMBlock *block;
+ char idstr[256];
+
+ if (flags & RAM_SAVE_FLAG_CONTINUE) {
+ if (!block) {
+ error_report("RAM_SAVE_FLAG_CONTINUE outside RAM block");
+ return NULL;
+ }
+
+ return block;
+ }
+
+ if (!qemu_get_counted_string(f, idstr)) {
+ error_report("Failed to get RAM block name");
+ return NULL;
+ }
+
+ block = ram_block_by_idstr(idstr);
+ if (!block) {
+ error_report("Can't find RAM block %s", idstr);
+ return NULL;
+ }
+
+ return block;
+}
+
+static int64_t ram_block_next_bdrv_offset(void)
+{
+ RAMBlock *last_block;
+ int64_t offset;
+
+ last_block = QSIMPLEQ_LAST(&ram_ctx.ram_block_list, RAMBlock, next);
+ if (!last_block) {
+ return 0;
+ }
+
+ offset = last_block->bdrv_offset + last_block->length;
+ return ROUND_UP(offset, BDRV_CLUSTER_SIZE);
+}
+
+static void ram_block_add(const char *idstr, int64_t size)
+{
+ RAMBlock *block;
+
+ block = g_new0(RAMBlock, 1);
+ block->length = size;
+ block->bdrv_offset = ram_block_next_bdrv_offset();
+ strcpy(block->idstr, idstr);
+
+ QSIMPLEQ_INSERT_TAIL(&ram_ctx.ram_block_list, block, next);
+}
+
+/*
+ * Assume that QEMUFile is migration stream and try to get
+ * from f_src ram blocks list. mem_size is a total amount of bytes of whole
+ * ram blocks.
+ */
+static int ram_block_list_from_stream(QEMUFile *f_src, int64_t mem_size)
+{
+ int64_t total_ram_bytes;
+
+ total_ram_bytes = mem_size;
+ while (total_ram_bytes > 0) {
+ char idstr[256];
+ int64_t size;
+
+ if (!qemu_get_counted_string(f_src, idstr)) {
+ error_report("Failed to get RAM block list");
+ return -EINVAL;
+ }
+
+ size = qemu_get_be64(f_src);
+
+ ram_block_add(idstr, size);
+ total_ram_bytes -= size;
+ }
+
+ if (total_ram_bytes != 0) {
+ error_report("Corrupted RAM block list");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+/* Send ram block list using migration rule */
+static int ram_block_list_to_stream(QEMUFile *f_dest)
+{
+ RAMBlock *block;
+ uint64_t total = 0;
+
+ QSIMPLEQ_FOREACH(block, &ram_ctx.ram_block_list, next) {
+ total += block->length;
+ }
+ qemu_put_be64(f_dest, total | RAM_SAVE_FLAG_MEM_SIZE);
+
+ QSIMPLEQ_FOREACH(block, &ram_ctx.ram_block_list, next) {
+ qemu_put_counted_string(f_dest, block->idstr);
+ qemu_put_be64(f_dest, block->length);
+ }
+
+ return qemu_file_get_error(f_dest);
+}
+
int coroutine_fn save_state_main(StateSaveCtx *s)
{
/* TODO: implement */
--
2.31.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3 15/17] migration/snapshot: Save part implement
2022-06-16 10:27 [PATCH v3 00/17] migration/snapshot: External snapshot utility nikita.lapshin
` (13 preceding siblings ...)
2022-06-16 10:28 ` [PATCH v3 14/17] migration/snpashot: Implement API for RAMBlock nikita.lapshin
@ 2022-06-16 10:28 ` nikita.lapshin
2022-06-16 10:28 ` [PATCH v3 16/17] migration/snapshot: Precopy load implemented nikita.lapshin
2022-06-16 10:28 ` [PATCH v3 17/17] migration/snapshot: Postcopy " nikita.lapshin
16 siblings, 0 replies; 20+ messages in thread
From: nikita.lapshin @ 2022-06-16 10:28 UTC (permalink / raw)
To: qemu-devel; +Cc: den, andrey.drobyshev, quintela, dgilbert, nikita.lapshin
From: Nikita Lapshin <nikita.lapshin@openvz.org>
Snapshot save are done in two stages. First tool save vmstate part. It must
be guaranteed that tool will get vmstate part only. This is because tool
won't parse this stream and will just save it non-modified.
Second stage is ram sending. Also it must be guarantee that ram part only will
be passed. Ram will be saved in qcow2 file.
The goal was to avoid duplicate same part of already existed migration code
so in this patch savevm handlers were used. Tool replace existed ram handler
with its own and after that call existed functions. It is work correctly
because algorithm of tool saving is similar to loadvm algorithm. I think
this isn't obvious part so it should be described here.
Signed-off-by: Nikita Lapshin <nikita.lapshin@openvz.org>
---
include/qemu-snapshot.h | 16 +-
migration/qemu-snapshot.c | 323 +++++++++++++++++++++++++++++++++++++-
migration/savevm.c | 22 ++-
migration/savevm.h | 4 +
qemu-snapshot.c | 82 +++++++++-
5 files changed, 435 insertions(+), 12 deletions(-)
diff --git a/include/qemu-snapshot.h b/include/qemu-snapshot.h
index be2557f6a0..a97abd9f25 100644
--- a/include/qemu-snapshot.h
+++ b/include/qemu-snapshot.h
@@ -21,6 +21,7 @@
/* Backing cluster size */
#define BDRV_CLUSTER_SIZE (1024 * 1024)
+#define VMSTATE_SIZE (1024 * 1024)
/* Minimum supported target page size */
#define PAGE_SIZE_MIN 4096
/*
@@ -34,6 +35,8 @@
/* RAM slice size for snapshot revert */
#define SLICE_SIZE_REVERT (16 * PAGE_SIZE_MAX)
+typedef struct QIOChannelBuffer QIOChannelBuffer;
+
typedef struct StateInfo {
int64_t page_size;
int64_t page_mask;
@@ -44,9 +47,17 @@ typedef struct StateInfo {
} StateInfo;
typedef struct StateSaveCtx {
- BlockBackend *blk; /* Block backend */
+ BlockBackend *blk; /* Block backend */
- StateInfo state_parameters; /* Migration state info*/
+ QEMUFile *f_fd; /* QEMUFile for incoming stream */
+ QEMUFile *f_vmstate; /* QEMUFile for vmstate backing */
+
+ QIOChannelBuffer *ioc_vmstate; /* Buffer for vmstate */
+ QIOChannelBuffer *ioc_pages; /* Page coalescing buffer */
+
+ StateInfo state_parameters; /* Migration state info*/
+
+ size_t vmstate_len; /* vmstate len */
} StateSaveCtx;
typedef struct StateLoadCtx {
@@ -60,6 +71,7 @@ void ram_destroy_state(void);
StateSaveCtx *get_save_context(void);
StateLoadCtx *get_load_context(void);
int coroutine_fn save_state_main(StateSaveCtx *s);
+void save_vmstate(StateSaveCtx *s);
int coroutine_fn load_state_main(StateLoadCtx *s);
QEMUFile *qemu_fopen_bdrv_vmstate(BlockDriverState *bs, int is_writable);
diff --git a/migration/qemu-snapshot.c b/migration/qemu-snapshot.c
index 394c6acb77..2c9909fc8e 100644
--- a/migration/qemu-snapshot.c
+++ b/migration/qemu-snapshot.c
@@ -22,6 +22,8 @@
#include "migration/savevm.h"
#include "migration/ram.h"
#include "qemu-snapshot.h"
+#include "migration/savevm.h"
+#include "migration/register.h"
/* RAM block */
/* TODO RAMBlock should be replace with existing struct RAMBlock in ram.c */
@@ -40,6 +42,11 @@ typedef struct RAMBlock {
char idstr[256]; /* RAM block id string */
} RAMBlock;
+typedef struct RAMPage {
+ RAMBlock *block; /* RAM block containing the page */
+ int64_t offset; /* Page offset in RAM block */
+} RAMPage;
+
/* RAM transfer context */
typedef struct RAMCtx {
int64_t normal_pages; /* Total number of normal pages */
@@ -51,6 +58,26 @@ typedef struct RAMCtx {
static RAMCtx ram_ctx;
+static int64_t page_size;
+static int page_bits;
+static int64_t page_mask;
+static int64_t slice_size;
+static int slice_bits;
+static int64_t slice_mask;
+/*
+ * Init sufficient global variables
+ * TODO: These variables should be removed or add to existing global structures
+ */
+static void init_global_var(StateInfo *si)
+{
+ page_size = si->page_size;
+ page_bits = si->page_bits;
+ page_mask = si->page_mask;
+ slice_size = si->slice_size;
+ slice_bits = si->slice_bits;
+ slice_mask = si->slice_mask;
+}
+
static inline
bool ram_offset_in_block(RAMBlock *block, int64_t offset)
{
@@ -158,6 +185,19 @@ static void ram_block_add(const char *idstr, int64_t size)
QSIMPLEQ_INSERT_TAIL(&ram_ctx.ram_block_list, block, next);
}
+static void ram_block_list_init_bitmaps(void)
+{
+ RAMBlock *block;
+
+ QSIMPLEQ_FOREACH(block, &ram_ctx.ram_block_list, next) {
+ block->nr_pages = block->length >> page_bits;
+ block->nr_slices = ROUND_UP(block->length, slice_size) >> slice_bits;
+
+ block->bitmap = bitmap_new(block->nr_slices);
+ bitmap_set(block->bitmap, 0, block->nr_slices);
+ }
+}
+
/*
* Assume that QEMUFile is migration stream and try to get
* from f_src ram blocks list. mem_size is a total amount of bytes of whole
@@ -188,6 +228,9 @@ static int ram_block_list_from_stream(QEMUFile *f_src, int64_t mem_size)
return -EINVAL;
}
+ /* Initialize per-block bitmaps */
+ ram_block_list_init_bitmaps();
+
return 0;
}
@@ -210,12 +253,276 @@ static int ram_block_list_to_stream(QEMUFile *f_dest)
return qemu_file_get_error(f_dest);
}
-int coroutine_fn save_state_main(StateSaveCtx *s)
+static void save_state_check_errors(StateSaveCtx *s, int *res)
{
- /* TODO: implement */
+ /* Check for -EIO which indicates input stream EOF */
+ if (*res == -EIO) {
+ *res = 0;
+ }
+
+ /*
+ * Check for file errors on success. Replace generic -EINVAL
+ * retcode with file error if possible.
+ */
+ if (*res >= 0 || *res == -EINVAL) {
+ int f_res = qemu_file_get_error(s->f_fd);
+
+ f_res = (f_res == -EIO) ? 0 : f_res;
+ if (!f_res) {
+ f_res = qemu_file_get_error(s->f_vmstate);
+ }
+ if (f_res) {
+ *res = f_res;
+ }
+ }
+}
+
+static int ram_save_page(StateSaveCtx *s, RAMPage *page, uint8_t *data)
+{
+ int64_t bdrv_offset;
+ int res = 0;
+
+ bdrv_offset = ram_bdrv_from_block_offset(page->block, page->offset);
+ if (bdrv_offset == INVALID_OFFSET) {
+ error_report("Corrupted RAM page");
+ return -EINVAL;
+ }
+
+ res = blk_pwrite(s->blk, bdrv_offset, data, page_size, 0);
+
+ res = MIN(res, 0);
+ return res;
+}
+
+static int ram_save(QEMUFile *f, void *opaque, int version_id)
+{
+ StateSaveCtx *s = (StateSaveCtx *) opaque;
+ int incompat_flags = RAM_SAVE_FLAG_COMPRESS_PAGE | RAM_SAVE_FLAG_XBZRLE;
+ int flags = 0;
+ int res = 0;
+
+ if (version_id != 4) {
+ error_report("Unsupported version %d for 'ram' handler v4", version_id);
+ return -EINVAL;
+ }
+
+ while (!res && !(flags & RAM_SAVE_FLAG_EOS)) {
+ RAMBlock *block = NULL;
+ int64_t offset;
+
+ offset = qemu_get_be64(f);
+ flags = offset & ~page_mask;
+ offset &= page_mask;
+
+ if (flags & incompat_flags) {
+ error_report("Incompatible RAM page flags 0x%x", flags);
+ res = -EINVAL;
+ break;
+ }
+
+ /* Lookup RAM block for the page */
+ if (flags & (RAM_SAVE_FLAG_ZERO | RAM_SAVE_FLAG_PAGE)) {
+ block = ram_block_from_stream(f, flags);
+ if (!block) {
+ error_report("Failed to get ram block form stream");
+ res = -EINVAL;
+ break;
+ }
+ }
+
+ switch (flags & ~RAM_SAVE_FLAG_CONTINUE) {
+ case RAM_SAVE_FLAG_MEM_SIZE:
+ /* Get RAM block list */
+ if (ram_block_list_from_stream(f, offset) ||
+ ram_block_list_to_stream(s->f_vmstate)) {
+ error_report("Failed to get ram block list");
+ res = -EINVAL;
+ }
+ break;
+
+ case RAM_SAVE_FLAG_ZERO:
+ /* Nothing to do with zero page */
+ qemu_get_byte(f);
+ break;
+
+ case RAM_SAVE_FLAG_PAGE:
+ {
+ RAMPage page = { .block = block, .offset = offset };
+ uint8_t *data;
+ ssize_t count;
+
+ count = qemu_peek_buffer(f, &data, page_size, 0);
+ qemu_file_skip(f, count);
+ if (count != page_size) {
+ /* I/O error */
+ res = -EINVAL;
+ break;
+ }
+
+ res = ram_save_page(s, &page, data);
+
+ /* Update normal page count */
+ ram_ctx.normal_pages++;
+ break;
+ }
+
+ case RAM_SAVE_FLAG_EOS:
+ /* Normal exit */
+ break;
+
+ default:
+ error_report("RAM page with unknown combination of flags 0x%x",
+ flags);
+ res = -EINVAL;
+
+ }
+
+ /* Make additional check for file errors */
+ if (!res) {
+ res = qemu_file_get_error(f);
+ }
+ }
+
+ return res;
+}
+
+static int save_section_config(StateSaveCtx *s)
+{
+ QEMUFile *f = s->f_fd;
+ uint32_t id_len;
+ uint8_t buf[256];
+
+ qemu_put_byte(s->f_vmstate, QEMU_VM_CONFIGURATION);
+
+ id_len = qemu_get_be32(f);
+ qemu_put_be32(s->f_vmstate, id_len);
+
+ if (id_len > 255) {
+ error_report("Corrupted configuration section");
+ return -EINVAL;
+ }
+
+ qemu_get_buffer(f, buf, id_len);
+ qemu_put_buffer(s->f_vmstate, buf, id_len);
+
+ buf[id_len] = '\0';
+ return 0;
+}
+
+static int save_state_header(StateSaveCtx *s)
+{
+ QEMUFile *f = s->f_fd;
+ QEMUFile *f_vmstate = s->f_vmstate;
+ uint32_t v;
+
+ /* Validate qemu magic */
+ v = qemu_get_be32(f);
+ if (v != QEMU_VM_FILE_MAGIC) {
+ error_report("Not a migration stream");
+ return -EINVAL;
+ }
+ qemu_put_be32(f_vmstate, v);
+
+ v = qemu_get_be32(f);
+ if (v == QEMU_VM_FILE_VERSION_COMPAT) {
+ error_report("SaveVM v2 format is obsolete");
+ return -EINVAL;
+ }
+
+ if (v != QEMU_VM_FILE_VERSION) {
+ error_report("Unsupported migration stream version");
+ return -EINVAL;
+ }
+
+ qemu_put_be32(f_vmstate, v);
return 0;
}
+static int save_completion(StateSaveCtx *s)
+{
+ uint8_t *ptr;
+ ptr = s->ioc_vmstate->data;
+ qemu_put_be64(s->f_vmstate, s->vmstate_len);
+ qemu_put_buffer(s->f_vmstate, ptr, s->vmstate_len);
+ return 0;
+}
+
+int coroutine_fn save_state_main(StateSaveCtx *s)
+{
+ QEMUFile *f = s->f_fd;
+ uint8_t section_type;
+ int res = 0;
+ SaveVMHandlers ram_ops;
+
+ init_global_var(&s->state_parameters);
+
+ ram_ops.load_state = ram_save;
+
+ res = qemu_replace_ram_handler((void *) &ram_ops, (void *) s);
+ if (res) {
+ error_report("Error replacing ram handler");
+ return res;
+ }
+
+ /* Deal with migration stream header */
+ res = save_state_header(s);
+ if (res) {
+ /* Check for file errors in case we have -EINVAL */
+ save_state_check_errors(s, &res);
+ return res;
+ }
+
+ while (!res) {
+ /* Read section type token */
+ section_type = qemu_get_byte(f);
+
+ switch (section_type) {
+ case QEMU_VM_CONFIGURATION:
+ res = save_section_config(s);
+ break;
+
+ /* We use already existed function which will call our handler */
+ case QEMU_VM_SECTION_FULL:
+ case QEMU_VM_SECTION_START:
+ res = qemu_loadvm_section_start_full(f, NULL);
+ break;
+
+ case QEMU_VM_SECTION_PART:
+ case QEMU_VM_SECTION_END:
+ res = qemu_loadvm_section_part_end(f, NULL);
+ break;
+
+ case QEMU_VM_EOF:
+ /* End of migration stream */
+ save_completion(s);
+ return res;
+
+ default:
+ error_report("Unknown section type %d", section_type);
+ res = -EINVAL;
+
+ }
+
+ /* Additional check for file errors */
+ save_state_check_errors(s, &res);
+ }
+
+ /* Replace positive retcode with 0 */
+ return MIN(res, 0);
+}
+
+void save_vmstate(StateSaveCtx *s)
+{
+ size_t res;
+
+ qemu_put_be64(s->f_vmstate, 0);
+ res = qemu_fsplice_tail(s->f_vmstate, s->f_fd);
+
+ qemu_fclose(s->f_vmstate);
+ s->f_vmstate = qemu_fopen_bdrv_vmstate(blk_bs(s->blk), 1);
+ qemu_put_be64(s->f_vmstate, res);
+}
+
int coroutine_fn load_state_main(StateLoadCtx *s)
{
/* TODO: implement */
@@ -228,10 +535,20 @@ void ram_init_state(void)
RAMCtx *ram = &ram_ctx;
memset(ram, 0, sizeof(ram_ctx));
+
+ /* Initialize RAM block list head */
+ QSIMPLEQ_INIT(&ram->ram_block_list);
}
/* Destroy snapshot RAM state */
void ram_destroy_state(void)
{
- /* TODO: implement */
+ RAMBlock *block;
+ RAMBlock *next_block;
+
+ /* Free RAM blocks */
+ QSIMPLEQ_FOREACH_SAFE(block, &ram_ctx.ram_block_list, next, next_block) {
+ g_free(block->bitmap);
+ g_free(block);
+ }
}
diff --git a/migration/savevm.c b/migration/savevm.c
index 48603517ba..b722e51163 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -2409,8 +2409,7 @@ static bool check_section_footer(QEMUFile *f, SaveStateEntry *se)
return true;
}
-static int
-qemu_loadvm_section_start_full(QEMUFile *f, MigrationIncomingState *mis)
+int qemu_loadvm_section_start_full(QEMUFile *f, MigrationIncomingState *mis)
{
uint32_t instance_id, version_id, section_id;
SaveStateEntry *se;
@@ -2474,8 +2473,7 @@ qemu_loadvm_section_start_full(QEMUFile *f, MigrationIncomingState *mis)
return 0;
}
-static int
-qemu_loadvm_section_part_end(QEMUFile *f, MigrationIncomingState *mis)
+int qemu_loadvm_section_part_end(QEMUFile *f, MigrationIncomingState *mis)
{
uint32_t section_id;
SaveStateEntry *se;
@@ -2645,6 +2643,22 @@ static bool postcopy_pause_incoming(MigrationIncomingState *mis)
return true;
}
+int qemu_replace_ram_handler(void *ram_ops, void *opaque)
+{
+ /* Find savevm section and change it to tools handler */
+ SaveStateEntry *se = find_se("ram", 0);
+ if (se) {
+ unregister_savevm(NULL, se->idstr, se->opaque);
+ }
+
+ if (register_savevm_live("ram", 0, 4, (SaveVMHandlers *) ram_ops, opaque)) {
+ error_report("Error register snapshot tool ram handlers");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis)
{
uint8_t section_type;
diff --git a/migration/savevm.h b/migration/savevm.h
index 6461342cb4..9abfcd88e5 100644
--- a/migration/savevm.h
+++ b/migration/savevm.h
@@ -67,5 +67,9 @@ int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis);
int qemu_load_device_state(QEMUFile *f);
int qemu_savevm_state_complete_precopy_non_iterable(QEMUFile *f,
bool in_postcopy, bool inactivate_disks);
+int qemu_loadvm_section_start_full(QEMUFile *f, MigrationIncomingState *mis);
+int qemu_loadvm_section_part_end(QEMUFile *f, MigrationIncomingState *mis);
+
+int qemu_replace_ram_handler(void *ram_ops, void *opaque);
#endif
diff --git a/qemu-snapshot.c b/qemu-snapshot.c
index 683f1b265a..172a9596df 100644
--- a/qemu-snapshot.c
+++ b/qemu-snapshot.c
@@ -20,6 +20,7 @@
#include "qapi/qmp/qdict.h"
#include "sysemu/sysemu.h"
#include "sysemu/block-backend.h"
+#include "sysemu/runstate.h"
#include "qemu/cutils.h"
#include "qemu/coroutine.h"
#include "qemu/error-report.h"
@@ -91,7 +92,22 @@ static void init_save_context(void)
static void destroy_save_context(void)
{
- /* TODO: implement */
+ StateSaveCtx *s = get_save_context();
+
+ if (s->f_vmstate) {
+ qemu_fclose(s->f_vmstate);
+ }
+ if (s->blk) {
+ blk_flush(s->blk);
+ blk_unref(s->blk);
+ }
+ if (s->ioc_pages) {
+ object_unref(OBJECT(s->ioc_pages));
+ }
+ if (s->ioc_vmstate) {
+ object_unref(OBJECT(s->ioc_vmstate));
+ }
+
}
static void init_load_context(void)
@@ -132,7 +148,11 @@ static void enter_co_bh(void *opaque)
static void coroutine_fn snapshot_save_co(void *opaque)
{
StateSaveCtx *s = get_save_context();
- int res = -1;
+ QIOChannel *ioc_fd;
+ int res = 0;
+ size_t conf_size = 0;
+ size_t size;
+
init_save_context();
/* Block backend */
@@ -142,9 +162,65 @@ static void coroutine_fn snapshot_save_co(void *opaque)
goto fail;
}
- res = save_state_main(s);
+ /*
+ * Check if we are saving vmstate or ram. We need ram to be separate
+ * from anything else.
+ */
+ if (!params.save_vmstate) {
+ /*
+ * We assume that we have saved vmstate stream.
+ * So now we need to remove some useless parts like header, magic
+ * number and conf section if it was saved.
+ */
+ s->ioc_vmstate = qio_channel_buffer_new(VMSTATE_SIZE);
+ qio_channel_set_name(QIO_CHANNEL(s->ioc_vmstate),
+ "migration-vmstate-buffer");
+
+ s->f_vmstate = qemu_fopen_bdrv_vmstate(blk_bs(s->blk), 0);
+
+ qemu_file_set_blocking(s->f_vmstate, false);
+ size = qemu_get_be64(s->f_vmstate);
+
+ /* Here we will skip magic and version and check these numbers later */
+ qemu_get_be64(s->f_vmstate);
+
+ if (qemu_peek_byte(s->f_vmstate, 0) == QEMU_VM_CONFIGURATION) {
+ qemu_get_byte(s->f_vmstate);
+ /*
+ * 4 bytes are string length and last is string itself.
+ * That's why we need to add 5 to conf_size.
+ */
+ conf_size = qemu_get_be32(s->f_vmstate);
+ qemu_file_skip(s->f_vmstate, conf_size);
+ conf_size += 5;
+ }
+
+ /* 8 is 4+4 bytes - migration magic number and migration header. */
+ s->vmstate_len = size - 8 - conf_size;
+ qemu_get_buffer(s->f_vmstate, s->ioc_vmstate->data, s->vmstate_len);
+ qemu_fclose(s->f_vmstate);
+ }
+
+ s->f_vmstate = qemu_fopen_bdrv_vmstate(blk_bs(s->blk), 1);
+ qemu_file_set_blocking(s->f_vmstate, false);
+
+ /* QEMUFile on migration fd */
+ ioc_fd = qio_channel_new_fd(params.fd, &error_fatal);
+ qio_channel_set_name(QIO_CHANNEL(ioc_fd), "migration-channel-incoming");
+ s->f_fd = qemu_fopen_channel_input(ioc_fd);
+ object_unref(OBJECT(ioc_fd));
+ /* Use non-blocking mode in coroutine */
+ qemu_file_set_blocking(s->f_fd, false);
+
+ s->state_parameters = *((StateInfo *) opaque);
+ if (params.save_vmstate) {
+ save_vmstate(s);
+ } else {
+ res = save_state_main(s);
+ }
if (res) {
error_report("Failed to save snapshot: %s", strerror(-res));
+ }
fail:
destroy_save_context();
--
2.31.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3 16/17] migration/snapshot: Precopy load implemented
2022-06-16 10:27 [PATCH v3 00/17] migration/snapshot: External snapshot utility nikita.lapshin
` (14 preceding siblings ...)
2022-06-16 10:28 ` [PATCH v3 15/17] migration/snapshot: Save part implement nikita.lapshin
@ 2022-06-16 10:28 ` nikita.lapshin
2022-06-16 10:28 ` [PATCH v3 17/17] migration/snapshot: Postcopy " nikita.lapshin
16 siblings, 0 replies; 20+ messages in thread
From: nikita.lapshin @ 2022-06-16 10:28 UTC (permalink / raw)
To: qemu-devel; +Cc: den, andrey.drobyshev, quintela, dgilbert, nikita.lapshin
From: Nikita Lapshin <nikita.lapshin@openvz.org>
Load snapshot from qcow2 file. This part also work only with ram part
and send vmstate part as it was saved previously without parsing.
Here also migration code was reused but now it is savevm part. Tool replace
ram handlers as it did before in tool save part but now it is also needed to
avoid using another handlers. That is where stream-content-list is used
it helps to "disable" all handlers except ram.
Also here slices were used to increase efficiency of reading from disk.
So when tool will need to read a page from block it will read a number
of pages and send them all.
Signed-off-by: Nikita Lapshin <nikita.lapshin@openvz.org>
---
include/qemu-snapshot.h | 4 +-
migration/qemu-snapshot.c | 333 +++++++++++++++++++++++++++++++++++++-
qemu-snapshot.c | 25 ++-
3 files changed, 356 insertions(+), 6 deletions(-)
diff --git a/include/qemu-snapshot.h b/include/qemu-snapshot.h
index a97abd9f25..74885c03bb 100644
--- a/include/qemu-snapshot.h
+++ b/include/qemu-snapshot.h
@@ -62,8 +62,10 @@ typedef struct StateSaveCtx {
typedef struct StateLoadCtx {
BlockBackend *blk; /* Block backend */
+ QEMUFile *f_fd;
+ QEMUFile *f_vmstate;
- StateInfo state_parameters; /* Migration state info*/
+ StateInfo state_parameters;
} StateLoadCtx;
void ram_init_state(void);
diff --git a/migration/qemu-snapshot.c b/migration/qemu-snapshot.c
index 2c9909fc8e..280f5be25c 100644
--- a/migration/qemu-snapshot.c
+++ b/migration/qemu-snapshot.c
@@ -21,9 +21,11 @@
#include "migration/qemu-file.h"
#include "migration/savevm.h"
#include "migration/ram.h"
+#include "migration/migration.h"
#include "qemu-snapshot.h"
#include "migration/savevm.h"
#include "migration/register.h"
+#include "qapi/qapi-types-migration.h"
/* RAM block */
/* TODO RAMBlock should be replace with existing struct RAMBlock in ram.c */
@@ -53,7 +55,8 @@ typedef struct RAMCtx {
/* RAM block list head */
QSIMPLEQ_HEAD(, RAMBlock) ram_block_list;
-
+ RAMPage last_page;
+ RAMBlock *last_sent_block;
} RAMCtx;
static RAMCtx ram_ctx;
@@ -523,12 +526,336 @@ void save_vmstate(StateSaveCtx *s)
qemu_put_be64(s->f_vmstate, res);
}
-int coroutine_fn load_state_main(StateLoadCtx *s)
+static void load_state_check_errors(StateLoadCtx *s, int *res)
+{
+ /*
+ * Check for file errors on success. Replace generic -EINVAL
+ * retcode with file error if possible.
+ */
+ if (*res >= 0 || *res == -EINVAL) {
+ int f_res = qemu_file_get_error(s->f_fd);
+
+ if (!f_res) {
+ f_res = qemu_file_get_error(s->f_vmstate);
+ }
+ if (f_res) {
+ *res = f_res;
+ }
+ }
+}
+
+static int send_conf(StateLoadCtx *s)
+{
+ QEMUFile *f = s->f_vmstate;
+ uint32_t id_len;
+ uint8_t buf[256];
+
+ qemu_put_byte(s->f_fd, QEMU_VM_CONFIGURATION);
+ id_len = qemu_get_be32(f);
+ qemu_put_be32(s->f_fd, id_len);
+
+ if (id_len > 255) {
+ error_report("Corrupted configuration section");
+ return -EINVAL;
+ }
+
+ qemu_get_buffer(f, buf, id_len);
+ qemu_put_buffer(s->f_fd, buf, id_len);
+ buf[id_len] = '\0';
+
+ return 0;
+}
+
+static int send_header(StateLoadCtx *s)
+{
+ QEMUFile *f = s->f_vmstate;
+ uint32_t v;
+
+ /* Validate magic */
+ v = qemu_get_be32(f);
+ if (v != QEMU_VM_FILE_MAGIC) {
+ error_report("Not a valid snapshot");
+ return -EINVAL;
+ }
+ qemu_put_be32(s->f_fd, v);
+
+ v = qemu_get_be32(f);
+ if (v == QEMU_VM_FILE_VERSION_COMPAT) {
+ error_report("SaveVM v2 format is obsolete");
+ return -EINVAL;
+ }
+ qemu_put_be32(s->f_fd, v);
+
+ return 0;
+}
+
+static int load_state_ramlist(StateLoadCtx *s)
{
- /* TODO: implement */
+ uint64_t size = qemu_get_be64(s->f_vmstate);
+ size = size & (~RAM_SAVE_FLAG_MEM_SIZE);
+ return ram_block_list_from_stream(s->f_vmstate, size);
+}
+
+static int send_setup(StateLoadCtx *s)
+{
+ /* We need to enable only ram using parameter stream-content-list */
+ qemu_savevm_state_setup(s->f_fd);
+
+ ram_block_list_init_bitmaps();
+
return 0;
}
+/* No need to parse vmstate part, we just send it using known offset */
+static int send_vmstate(QEMUFile *f_vmstate, QEMUFile *f_dest)
+{
+ /* Send vmstate without last byte because it is QEMU_VM_EOF */
+ size_t len;
+ len = qemu_get_be64(f_vmstate);
+ qemu_fsplice(f_dest, f_vmstate, len);
+
+ return 0;
+}
+
+static int ram_send_setup(QEMUFile *f, void *opaque)
+{
+ int res = ram_block_list_to_stream(f);
+
+ qemu_put_be64(f, RAM_SAVE_FLAG_EOS);
+ qemu_fflush(f);
+
+ return res;
+}
+
+static bool find_next_page(RAMPage *page)
+{
+ RAMCtx *ram = &ram_ctx;
+ RAMBlock *block = ram->last_page.block;
+ int64_t slice = ram->last_page.offset >> slice_bits;
+ bool full_round = false;
+ bool found = false;
+
+ if (!block) {
+restart:
+ block = QSIMPLEQ_FIRST(&ram->ram_block_list);
+ slice = 0;
+ full_round = true;
+ }
+
+ while (!found && block) {
+ slice = find_next_bit(block->bitmap, block->nr_slices, slice);
+ /* Can't find unsent slice in block? */
+ if (slice >= block->nr_slices) {
+ /* Try next block */
+ block = QSIMPLEQ_NEXT(block, next);
+ slice = 0;
+
+ continue;
+ }
+
+ found = true;
+ }
+
+ /*
+ * Re-start from the beginning if couldn't find unsent slice,
+ * but do it only once.
+ */
+ if (!found && !full_round) {
+ goto restart;
+ }
+
+ if (found) {
+ page->block = block;
+ page->offset = slice << slice_bits;
+ }
+
+ return found;
+}
+
+static void get_page_range(RAMPage *page, int64_t *length)
+{
+ int64_t start_slice;
+ int64_t end_slice;
+ int64_t tmp;
+
+ assert(QEMU_IS_ALIGNED(page->offset, slice_size));
+
+ start_slice = page->offset >> slice_bits;
+ end_slice = find_next_zero_bit(page->block->bitmap, page->block->nr_slices,
+ page->offset >> slice_bits);
+
+ tmp = (end_slice - start_slice) << slice_bits;
+ *length = page->block->length - page->offset;
+ *length = MIN(*length, tmp);
+
+ /*
+ * Length is always aligned to slice_size with the exception of case
+ * when it is the last slice in RAM block.
+ */
+ *length = MIN(slice_size, *length);
+}
+
+static inline
+void clear_page_range(RAMPage *page, int64_t length)
+{
+ assert(QEMU_IS_ALIGNED(page->offset, slice_size));
+ assert(length);
+
+ /*
+ * Page offsets are aligned to the slice boundary so we only need
+ * to round up length for the case when we load last slice in the block.
+ */
+ bitmap_clear(page->block->bitmap, page->offset >> slice_bits,
+ ((length - 1) >> slice_bits) + 1);
+}
+
+static int coroutine_fn ram_load_pages(StateLoadCtx *s)
+{
+ RAMCtx *ram = &ram_ctx;
+ RAMPage page;
+ int64_t length;
+ uint8_t *data;
+ int64_t blk_offset, slice_offset, bdrv_offset;
+ ssize_t res;
+ int64_t flags = RAM_SAVE_FLAG_CONTINUE;
+
+ if (!find_next_page(&page)) {
+ return 0;
+ }
+
+ /* Get range of contiguous pages that were not transferred yet */
+ get_page_range(&page, &length);
+ /* Clear range of pages to be queued for I/O */
+ clear_page_range(&page, length);
+
+ /* Used by find_next_page() */
+ ram->last_page.block = page.block;
+ ram->last_page.offset = page.offset + length;
+
+ /* Read found slice from block */
+ bdrv_offset = ram_bdrv_from_block_offset(page.block, page.offset);
+ data = (uint8_t *) qemu_blockalign(blk_bs(s->blk), length);
+ res = blk_pread(s->blk, bdrv_offset, data, length);
+
+ if (res < 0) {
+ error_report("Read from file failed");
+ return res;
+ }
+
+ /* Send slice to destination */
+ slice_offset = 0;
+ while (length > slice_offset) {
+ if (!ram_bdrv_offset_in_block(ram->last_sent_block, bdrv_offset)) {
+ ram->last_sent_block = page.block;
+ flags = 0;
+ }
+
+ blk_offset = ram_block_offset_from_bdrv(ram->last_sent_block,
+ slice_offset + bdrv_offset);
+ qemu_put_be64(s->f_fd, blk_offset | RAM_SAVE_FLAG_PAGE | flags);
+ if (!flags) {
+ qemu_put_counted_string(s->f_fd, page.block->idstr);
+ }
+ qemu_put_buffer(s->f_fd, data + slice_offset, page_size);
+ slice_offset += page_size;
+ qemu_fflush(s->f_fd);
+ }
+ qemu_fflush(s->f_fd);
+ return 1;
+}
+
+static int ram_load_iterate(QEMUFile *f, void *opaque)
+{
+ StateLoadCtx *s = (StateLoadCtx *) opaque;
+ int res;
+
+ res = ram_load_pages(s);
+
+ /* Zero retcode means that there're no more pages to load */
+ if (res >= 0) {
+ res = res ? 0 : 1;
+ }
+
+ /* Send EOS flag before section footer */
+ qemu_put_be64(s->f_fd, RAM_SAVE_FLAG_EOS);
+ qemu_fflush(s->f_fd);
+
+ return res;
+}
+
+static bool is_active_ram(void *opaque)
+{
+ return true;
+}
+
+int coroutine_fn load_state_main(StateLoadCtx *s)
+{
+ int res;
+ MigrationState *ms = migrate_get_current();
+ SaveVMHandlers ram_ops;
+
+ init_global_var(&s->state_parameters);
+
+ strList *list = g_malloc(sizeof(strList));
+ list->value = g_malloc(sizeof("ram"));
+ list->next = NULL;
+ memcpy(list->value, "ram", sizeof("ram"));
+
+ ms->parameters.has_stream_content_list = true;
+ ms->parameters.stream_content_list = list;
+ ms->send_section_footer = true;
+
+ ram_ops.save_setup = ram_send_setup;
+ ram_ops.save_live_iterate = ram_load_iterate;
+ ram_ops.is_active = is_active_ram;
+ ram_ops.is_active_iterate = NULL;
+
+ res = qemu_replace_ram_handler((void *) &ram_ops, (void *) s);
+
+ res = send_header(s);
+ if (res) {
+ goto fail;
+ }
+ if (qemu_peek_byte(s->f_vmstate, 0) == QEMU_VM_CONFIGURATION) {
+ qemu_get_byte(s->f_vmstate);
+ res = send_conf(s);
+ if (res) {
+ goto fail;
+ }
+ }
+
+ res = load_state_ramlist(s);
+ if (res) {
+ goto fail;
+ }
+
+ res = send_setup(s);
+ if (res) {
+ error_report("Send setup failed");
+ goto fail;
+ }
+
+ do {
+ res = qemu_savevm_state_iterate(s->f_fd, false);
+ /* Check for file errors */
+ load_state_check_errors(s, &res);
+ } while (!res);
+
+ if (res < 0) {
+ goto fail;
+ }
+
+
+ send_vmstate(s->f_vmstate, s->f_fd);
+ qemu_put_byte(s->f_fd, QEMU_VM_EOF);
+ qemu_fflush(s->f_fd);
+fail:
+ load_state_check_errors(s, &res);
+
+ /* Replace positive retcode with 0 */
+ return MIN(res, 0);
+}
+
/* Initialize snapshot RAM state */
void ram_init_state(void)
{
diff --git a/qemu-snapshot.c b/qemu-snapshot.c
index 172a9596df..04bda74fb4 100644
--- a/qemu-snapshot.c
+++ b/qemu-snapshot.c
@@ -117,7 +117,14 @@ static void init_load_context(void)
static void destroy_load_context(void)
{
- /* TODO: implement */
+ StateLoadCtx *s = get_load_context();
+
+ if (s->f_vmstate) {
+ qemu_fclose(s->f_vmstate);
+ }
+ if (s->blk) {
+ blk_unref(s->blk);
+ }
}
static BlockBackend *image_open_opts(const char *optstr, QDict *options,
@@ -230,7 +237,8 @@ fail:
static void coroutine_fn snapshot_load_co(void *opaque)
{
StateLoadCtx *s = get_load_context();
- int res = -1;
+ int res;
+ QIOChannel *ioc_fd;
init_load_context();
@@ -241,6 +249,19 @@ static void coroutine_fn snapshot_load_co(void *opaque)
goto fail;
}
+ /* QEMUFile on vmstate */
+ s->f_vmstate = qemu_fopen_bdrv_vmstate(blk_bs(s->blk), 0);
+ qemu_file_set_blocking(s->f_vmstate, false);
+
+ /* QEMUFile on migration fd */
+ ioc_fd = qio_channel_new_fd(params.fd, NULL);
+ qio_channel_set_name(QIO_CHANNEL(ioc_fd), "migration-channel-outgoing");
+ s->f_fd = qemu_fopen_channel_output(ioc_fd);
+ object_unref(OBJECT(ioc_fd));
+ qemu_file_set_blocking(s->f_fd, false);
+
+ s->state_parameters = *((StateInfo *) opaque);
+
res = load_state_main(s);
if (res) {
error_report("Failed to load snapshot: %s", strerror(-res));
--
2.31.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3 17/17] migration/snapshot: Postcopy load implemented
2022-06-16 10:27 [PATCH v3 00/17] migration/snapshot: External snapshot utility nikita.lapshin
` (15 preceding siblings ...)
2022-06-16 10:28 ` [PATCH v3 16/17] migration/snapshot: Precopy load implemented nikita.lapshin
@ 2022-06-16 10:28 ` nikita.lapshin
16 siblings, 0 replies; 20+ messages in thread
From: nikita.lapshin @ 2022-06-16 10:28 UTC (permalink / raw)
To: qemu-devel; +Cc: den, andrey.drobyshev, quintela, dgilbert, nikita.lapshin
From: Nikita Lapshin <nikita.lapshin@openvz.org>
It is a modified load part from previous patch.
Implemented new rp listen thread for snapshot-tool. Also implemented
functions for starting postcopy.
This mode can be turned on by specifying --postcopy flag.
Signed-off-by: Nikita Lapshin <nikita.lapshin@openvz.org>
---
include/qemu-snapshot.h | 12 ++
migration/migration.c | 123 +++++++++++++++++++
migration/migration.h | 1 +
migration/qemu-snapshot.c | 249 +++++++++++++++++++++++++++++++++++++-
migration/savevm.c | 25 ++++
migration/savevm.h | 4 +
qemu-snapshot.c | 10 ++
7 files changed, 422 insertions(+), 2 deletions(-)
diff --git a/include/qemu-snapshot.h b/include/qemu-snapshot.h
index 74885c03bb..b0f235747f 100644
--- a/include/qemu-snapshot.h
+++ b/include/qemu-snapshot.h
@@ -65,6 +65,15 @@ typedef struct StateLoadCtx {
QEMUFile *f_fd;
QEMUFile *f_vmstate;
+ /* Postcopy part */
+ bool postcopy;
+ bool in_postcopy;
+
+ /* Return path part */
+ QemuThread rp_listen_thread;
+ QEMUFile *f_rp_fd;
+ bool has_rp_listen_thread;
+
StateInfo state_parameters;
} StateLoadCtx;
@@ -76,6 +85,9 @@ int coroutine_fn save_state_main(StateSaveCtx *s);
void save_vmstate(StateSaveCtx *s);
int coroutine_fn load_state_main(StateLoadCtx *s);
+int queue_page_request(const char *idstr, uint64_t offset,
+ uint32_t size);
+
QEMUFile *qemu_fopen_bdrv_vmstate(BlockDriverState *bs, int is_writable);
void qemu_fsplice(QEMUFile *f_dst, QEMUFile *f_src, size_t size);
size_t qemu_fsplice_tail(QEMUFile *f_dst, QEMUFile *f_src);
diff --git a/migration/migration.c b/migration/migration.c
index 6528b3ad41..6f82e8ea48 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -61,6 +61,7 @@
#include "sysemu/cpus.h"
#include "yank_functions.h"
#include "sysemu/qtest.h"
+#include "qemu-snapshot.h"
#define MAX_THROTTLE (128 << 20) /* Migration transfer speed throttling */
@@ -3517,6 +3518,128 @@ static MigThrError postcopy_pause(MigrationState *s)
}
}
+/*
+ * Return-path message processing thread for qemu-snapshot tool
+ */
+void *qemu_snapshot_rp_listen_thread(void *opaque)
+{
+ QEMUFile *f = (QEMUFile *) opaque;
+ int res = 0;
+ uint64_t pages = 0;
+
+ while (!res) {
+ uint8_t h_buf[512];
+ const int h_max_len = sizeof(h_buf);
+ int h_type;
+ int h_len;
+ size_t count;
+
+ h_type = qemu_get_be16(f);
+ h_len = qemu_get_be16(f);
+
+ /* Make early check for input errors */
+ res = qemu_file_get_error(f);
+ if (res) {
+ break;
+ }
+
+ /* Check message type */
+ if (h_type >= MIG_RP_MSG_MAX || h_type == MIG_RP_MSG_INVALID) {
+ error_report("RP: received invalid message type %d length %d",
+ h_type, h_len);
+ res = -EINVAL;
+ break;
+ }
+
+ /* Check message length */
+ if (rp_cmd_args[h_type].len != -1 && h_len != rp_cmd_args[h_type].len) {
+ error_report("RP: received %s message len %d expected %ld",
+ rp_cmd_args[h_type].name,
+ h_len, rp_cmd_args[h_type].len);
+ res = -EINVAL;
+ break;
+ } else if (h_len > h_max_len) {
+ error_report("RP: received %s message len %d max_len %d",
+ rp_cmd_args[h_type].name, h_len, h_max_len);
+ res = -EINVAL;
+ break;
+ }
+
+ count = qemu_get_buffer(f, h_buf, h_len);
+ if (count != h_len) {
+ break;
+ }
+
+ switch (h_type) {
+ case MIG_RP_MSG_SHUT:
+ {
+ int shut_error;
+
+ shut_error = be32_to_cpu(*(uint32_t *) h_buf);
+ if (shut_error) {
+ error_report("RP: sibling shutdown, error %d", shut_error);
+ }
+
+ /* Exit processing loop */
+ res = 1;
+ break;
+ }
+
+ case MIG_RP_MSG_REQ_PAGES:
+ case MIG_RP_MSG_REQ_PAGES_ID:
+ {
+ pages++;
+ uint64_t offset;
+ uint32_t size;
+ char *id_str = NULL;
+
+ offset = be64_to_cpu(*(uint64_t *) (h_buf + 0));
+ size = be32_to_cpu(*(uint32_t *) (h_buf + 8));
+
+ if (h_type == MIG_RP_MSG_REQ_PAGES_ID) {
+ int h_parsed_len = rp_cmd_args[MIG_RP_MSG_REQ_PAGES].len;
+
+ if (h_len > h_parsed_len) {
+ int id_len;
+
+ /* RAM block id string */
+ id_len = h_buf[h_parsed_len];
+ id_str = (char *) &h_buf[h_parsed_len + 1];
+ id_str[id_len] = 0;
+
+ h_parsed_len += id_len + 1;
+ }
+
+ if (h_parsed_len != h_len) {
+ error_report("RP: received %s message len %d expected %d",
+ rp_cmd_args[MIG_RP_MSG_REQ_PAGES_ID].name,
+ h_len, h_parsed_len);
+ res = -EINVAL;
+ break;
+ }
+ }
+
+ res = queue_page_request(id_str, offset, size);
+ break;
+ }
+
+ default:
+ error_report("RP: received unexpected message type %d len %d",
+ h_type, h_len);
+ res = -EINVAL;
+ }
+ }
+
+ if (res >= 0) {
+ res = qemu_file_get_error(f);
+ }
+ if (res) {
+ error_report("RP: listen thread exit, error %d", res);
+ }
+
+ return NULL;
+}
+
static MigThrError migration_detect_error(MigrationState *s)
{
int ret;
diff --git a/migration/migration.h b/migration/migration.h
index 5c43788a2b..fd6f8d3083 100644
--- a/migration/migration.h
+++ b/migration/migration.h
@@ -433,4 +433,5 @@ void migration_cancel(const Error *error);
void populate_vfio_info(MigrationInfo *info);
void postcopy_temp_page_reset(PostcopyTmpPage *tmp_page);
+void *qemu_snapshot_rp_listen_thread(void *opaque);
#endif
diff --git a/migration/qemu-snapshot.c b/migration/qemu-snapshot.c
index 280f5be25c..8090c7032d 100644
--- a/migration/qemu-snapshot.c
+++ b/migration/qemu-snapshot.c
@@ -44,6 +44,16 @@ typedef struct RAMBlock {
char idstr[256]; /* RAM block id string */
} RAMBlock;
+/* Page request from destination in postcopy */
+typedef struct RAMPageRequest {
+ RAMBlock *block;
+ int64_t offset;
+ unsigned size;
+
+ /* Link into ram_ctx.page_req */
+ QSIMPLEQ_ENTRY(RAMPageRequest) next;
+} RAMPageRequest;
+
typedef struct RAMPage {
RAMBlock *block; /* RAM block containing the page */
int64_t offset; /* Page offset in RAM block */
@@ -57,6 +67,11 @@ typedef struct RAMCtx {
QSIMPLEQ_HEAD(, RAMBlock) ram_block_list;
RAMPage last_page;
RAMBlock *last_sent_block;
+
+ /* Page request queue for postcopy */
+ QemuMutex page_req_mutex;
+ QSIMPLEQ_HEAD(, RAMPageRequest) page_req;
+ RAMBlock *last_req_block;
} RAMCtx;
static RAMCtx ram_ctx;
@@ -627,6 +642,143 @@ static int ram_send_setup(QEMUFile *f, void *opaque)
return res;
}
+static int send_each_ram_block_discard(QEMUFile *f)
+{
+ RAMBlock *block;
+ int res = 0;
+ uint64_t start;
+ uint64_t len;
+
+ QSIMPLEQ_FOREACH(block, &ram_ctx.ram_block_list, next) {
+ start = block->discard_offset;
+ len = block->length - block->discard_offset;
+ qemu_savevm_send_postcopy_ram_discard(f, block->idstr, 1, &start, &len);
+
+ res = qemu_file_get_error(f);
+ if (res) {
+ break;
+ }
+ }
+
+ return res;
+}
+
+static int prepare_postcopy(StateLoadCtx *s)
+{
+ int res = 0;
+
+ res = qemu_snapshot_postcopy_prepare(s->f_fd, page_size, page_size);
+
+ if (!res) {
+ qemu_thread_create(&s->rp_listen_thread, "rp_thread",
+ qemu_snapshot_rp_listen_thread, s->f_rp_fd,
+ QEMU_THREAD_JOINABLE);
+ s->has_rp_listen_thread = true;
+ }
+
+ return res;
+}
+
+static int start_postcopy(StateLoadCtx *s)
+{
+ QIOChannelBuffer *bioc;
+ QEMUFile *fb;
+ int eof_pos;
+ int res = 0;
+
+ /*
+ * Send RAM discards for each block's unsent part. Without discards,
+ * the userfault_fd code on destination will not trigger page requests
+ * as expected. Also, the UFFDIO_COPY ioctl that is used to place incoming
+ * page in postcopy would give an error if that page has not faulted
+ * with MISSING reason.
+ */
+ res = send_each_ram_block_discard(s->f_fd);
+ if (res) {
+ error_report("here");
+ return res;
+ }
+
+ /*
+ * To perform a switch to postcopy on destination, we need to send
+ * commands and the device state data in the following order:
+ * * MIG_CMD_POSTCOPY_LISTEN
+ * * Non-iterable device state sections
+ * * MIG_CMD_POSTCOPY_RUN
+ *
+ * All this has to be packaged into a single blob using MIG_CMD_PACKAGED
+ * command. While loading the device state we may trigger page transfer
+ * requests and the fd must be free to process those, thus the destination
+ * must read the whole device state off the fd before it starts
+ * processing it. To wrap it up in a package, QEMU buffer channel is used.
+ */
+ bioc = qio_channel_buffer_new(512 * 1024);
+ qio_channel_set_name(QIO_CHANNEL(bioc), "migration-postcopy-buffer");
+ fb = qemu_fopen_channel_output(QIO_CHANNEL(bioc));
+ object_unref(OBJECT(bioc));
+
+ /* MIG_CMD_POSTCOPY_LISTEN command */
+ qemu_savevm_send_postcopy_listen(fb);
+
+ /* The rest of non-iterable device state with an optional vmdesc section */
+ send_vmstate(s->f_vmstate, fb);
+ qemu_fflush(fb);
+
+ /*
+ * vmdesc section may optionally be present at the end of the stream
+ * so we'll try to locate it and truncate the trailer.
+ */
+ eof_pos = bioc->usage - 1;
+
+ for (int offset = (bioc->usage - 11); offset >= 0; offset--) {
+ if (bioc->data[offset] == QEMU_VM_SECTION_FOOTER &&
+ bioc->data[offset + 5] == QEMU_VM_EOF &&
+ bioc->data[offset + 6] == QEMU_VM_VMDESCRIPTION) {
+ uint32_t expected_length = bioc->usage - (offset + 11);
+ uint32_t json_length;
+
+ json_length = be32_to_cpu(*(uint32_t *) &bioc->data[offset + 7]);
+ if (json_length != expected_length) {
+ error_report("Corrupted vmdesc trailer: length %" PRIu32
+ " expected %" PRIu32,
+ json_length, expected_length);
+ res = -EINVAL;
+ goto fail;
+ }
+
+ eof_pos = offset + 5;
+ break;
+ }
+ }
+
+ /*
+ * When switching to postcopy we need to skip QEMU_VM_EOF token which
+ * normally is placed after the last non-iterable device state section
+ * (but before the vmdesc section).
+ *
+ * Skipping QEMU_VM_EOF is required to allow migration process to
+ * continue in postcopy. Vmdesc section also has to be skipped here.
+ */
+ if (eof_pos >= 0 && bioc->data[eof_pos] == QEMU_VM_EOF) {
+ bioc->usage = eof_pos;
+ bioc->offset = eof_pos;
+ }
+
+ /* Finally is the MIG_CMD_POSTCOPY_RUN command */
+ qemu_savevm_send_postcopy_run(fb);
+
+ /* Now send that blob */
+ qemu_savevm_send_packaged(s->f_fd, bioc->data, bioc->usage);
+ qemu_fflush(s->f_fd);
+
+ s->in_postcopy = true;
+fail:
+ qemu_fclose(fb);
+ load_state_check_errors(s, &res);
+
+ return res;
+}
+
static bool find_next_page(RAMPage *page)
{
RAMCtx *ram = &ram_ctx;
@@ -709,6 +861,74 @@ void clear_page_range(RAMPage *page, int64_t length)
((length - 1) >> slice_bits) + 1);
}
+int queue_page_request(const char *idstr, uint64_t offset,
+ uint32_t size)
+{
+ RAMCtx *ram = &ram_ctx;
+ RAMBlock *block;
+ RAMPageRequest *new_entry;
+
+ if (!idstr) {
+ block = ram->last_req_block;
+ if (!block) {
+ error_report("RP-REQ_PAGES: no previous block");
+ return -EINVAL;
+ }
+ } else {
+ block = ram_block_by_idstr(idstr);
+ if (!block) {
+ error_report("RP-REQ_PAGES: cannot find block %s", idstr);
+ return -EINVAL;
+ }
+
+ ram->last_req_block = block;
+ }
+
+ if (!ram_offset_in_block(block, offset)) {
+ error_report("RP-REQ_PAGES: offset 0x%" PRIx64 " out of RAM block %s",
+ offset, idstr);
+ return -EINVAL;
+ }
+
+ new_entry = g_new0(RAMPageRequest, 1);
+ new_entry->block = block;
+ new_entry->offset = offset;
+ new_entry->size = size;
+
+ qemu_mutex_lock(&ram->page_req_mutex);
+ QSIMPLEQ_INSERT_TAIL(&ram->page_req, new_entry, next);
+ qemu_mutex_unlock(&ram->page_req_mutex);
+
+ return 0;
+}
+
+static bool get_queued_page(RAMPage *page) {
+ RAMCtx *ram = &ram_ctx;
+
+ if (QSIMPLEQ_EMPTY_ATOMIC(&ram->page_req)) {
+ return false;
+ }
+
+ QEMU_LOCK_GUARD(&ram->page_req_mutex);
+ if (!QSIMPLEQ_EMPTY(&ram->page_req)) {
+ RAMPageRequest *entry = QSIMPLEQ_FIRST(&ram->page_req);
+ RAMBlock *block = entry->block;
+ int64_t slice = entry->offset >> slice_bits;
+
+ QSIMPLEQ_REMOVE_HEAD(&ram->page_req, next);
+ g_free(entry);
+
+ if (test_bit(slice, block->bitmap)) {
+ page->block = block;
+ page->offset = slice << slice_bits;
+
+ return true;
+ }
+ }
+
+ return false;
+}
+
static int coroutine_fn ram_load_pages(StateLoadCtx *s)
{
RAMCtx *ram = &ram_ctx;
@@ -718,8 +938,9 @@ static int coroutine_fn ram_load_pages(StateLoadCtx *s)
int64_t blk_offset, slice_offset, bdrv_offset;
ssize_t res;
int64_t flags = RAM_SAVE_FLAG_CONTINUE;
+ bool urgent = get_queued_page(&page);
- if (!find_next_page(&page)) {
+ if (!urgent && !find_next_page(&page)) {
return 0;
}
@@ -835,6 +1056,20 @@ int coroutine_fn load_state_main(StateLoadCtx *s)
goto fail;
}
+ if (s->postcopy) {
+ res = prepare_postcopy(s);
+ if (res) {
+ error_report("Prepare postcopy failed");
+ goto fail;
+ }
+ /* TODO: Add condition to start postcopy during the cycle below */
+ res = start_postcopy(s);
+ if (res) {
+ error_report("Postcopy start failed");
+ goto fail;
+ }
+ }
+
do {
res = qemu_savevm_state_iterate(s->f_fd, false);
/* Check for file errors */
@@ -845,8 +1080,11 @@ int coroutine_fn load_state_main(StateLoadCtx *s)
goto fail;
}
+ /* If tool is in posctopy mode then vmstate have been already sent */
+ if (!s->in_postcopy) {
+ send_vmstate(s->f_vmstate, s->f_fd);
+ }
- send_vmstate(s->f_vmstate, s->f_fd);
qemu_put_byte(s->f_fd, QEMU_VM_EOF);
qemu_fflush(s->f_fd);
fail:
@@ -865,6 +1103,10 @@ void ram_init_state(void)
/* Initialize RAM block list head */
QSIMPLEQ_INIT(&ram->ram_block_list);
+
+ /* Initialize postcopy page request queue */
+ qemu_mutex_init(&ram->page_req_mutex);
+ QSIMPLEQ_INIT(&ram->page_req);
}
/* Destroy snapshot RAM state */
@@ -878,4 +1120,7 @@ void ram_destroy_state(void)
g_free(block->bitmap);
g_free(block);
}
+
+ /* Destroy page request mutex */
+ qemu_mutex_destroy(&ram_ctx.page_req_mutex);
}
diff --git a/migration/savevm.c b/migration/savevm.c
index b722e51163..b1320bd813 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -3356,3 +3356,28 @@ void qmp_snapshot_delete(const char *job_id,
job_start(&s->common);
}
+
+/* Do preparation before qemu-snapshot tool start postcopy */
+int qemu_snapshot_postcopy_prepare(QEMUFile *f_dest,
+ uint64_t sps,
+ uint64_t tps)
+{
+ uint64_t tmp[2];
+ int res;
+
+ /* Send POSTCOPY_ADVISE */
+ tmp[0] = cpu_to_be64(sps);
+ tmp[1] = cpu_to_be64(tps);
+ qemu_savevm_command_send(f_dest, MIG_CMD_POSTCOPY_ADVISE, 16, (uint8_t *) tmp);
+
+ /* Open return path on destination */
+ qemu_savevm_command_send(f_dest, MIG_CMD_OPEN_RETURN_PATH, 0, NULL);
+
+ /*
+ * Check for file errors after sending POSTCOPY_ADVISE command
+ * since destination may already have closed input pipe in case
+ * postcopy had not been enabled in advance.
+ */
+ res = qemu_file_get_error(f_dest);
+ return res;
+}
diff --git a/migration/savevm.h b/migration/savevm.h
index 9abfcd88e5..94b6f60496 100644
--- a/migration/savevm.h
+++ b/migration/savevm.h
@@ -72,4 +72,8 @@ int qemu_loadvm_section_part_end(QEMUFile *f, MigrationIncomingState *mis);
int qemu_replace_ram_handler(void *ram_ops, void *opaque);
+int qemu_snapshot_postcopy_prepare(QEMUFile *f_dest,
+ uint64_t sps,
+ uint64_t tps);
+
#endif
diff --git a/qemu-snapshot.c b/qemu-snapshot.c
index 04bda74fb4..893086eb8a 100644
--- a/qemu-snapshot.c
+++ b/qemu-snapshot.c
@@ -239,6 +239,7 @@ static void coroutine_fn snapshot_load_co(void *opaque)
StateLoadCtx *s = get_load_context();
int res;
QIOChannel *ioc_fd;
+ QIOChannel *ioc_rp_fd;
init_load_context();
@@ -260,8 +261,17 @@ static void coroutine_fn snapshot_load_co(void *opaque)
object_unref(OBJECT(ioc_fd));
qemu_file_set_blocking(s->f_fd, false);
+ /* qemufile on return path fd if we are going to use postcopy */
+ if (params.postcopy) {
+ ioc_rp_fd = qio_channel_new_fd(params.rp_fd, NULL);
+ qio_channel_set_name(QIO_CHANNEL(ioc_fd), "migration-channel-rp");
+ s->f_rp_fd = qemu_fopen_channel_input(ioc_rp_fd);
+ object_unref(OBJECT(ioc_rp_fd));
+ }
+
s->state_parameters = *((StateInfo *) opaque);
+ s->postcopy = params.postcopy;
res = load_state_main(s);
if (res) {
error_report("Failed to load snapshot: %s", strerror(-res));
--
2.31.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* Re: [PATCH v3 11/17] migration/qemu-file: Fix qemu_ftell() for non-writable file
2022-06-16 10:28 ` [PATCH v3 11/17] migration/qemu-file: Fix qemu_ftell() for non-writable file nikita.lapshin
@ 2022-06-16 11:20 ` Daniel P. Berrangé
2022-06-16 12:54 ` Nikita
0 siblings, 1 reply; 20+ messages in thread
From: Daniel P. Berrangé @ 2022-06-16 11:20 UTC (permalink / raw)
To: nikita.lapshin; +Cc: qemu-devel, den, andrey.drobyshev, quintela, dgilbert
On Thu, Jun 16, 2022 at 01:28:05PM +0300, nikita.lapshin@openvz.org wrote:
> From: Nikita Lapshin <nikita.lapshin@openvz.org>
>
> qemu_ftell() will return wrong value for non-writable QEMUFile.
> This happens due to call qemu_fflush() inside qemu_ftell(), this
> function won't flush if file is readable.
Well the return value isn't necessarily wrong today - it really
depends what semantics each callers desires.
Can you say what particular caller needs these semantics changed
and the impact on them from current behaviour ?
> Signed-off-by: Nikita Lapshin <nikita.lapshin@openvz.org>
> ---
> migration/qemu-file.c | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/migration/qemu-file.c b/migration/qemu-file.c
> index 1479cddad9..53ccef80ac 100644
> --- a/migration/qemu-file.c
> +++ b/migration/qemu-file.c
> @@ -663,7 +663,8 @@ int64_t qemu_ftell_fast(QEMUFile *f)
> int64_t qemu_ftell(QEMUFile *f)
> {
> qemu_fflush(f);
> - return f->pos;
> + /* Consider that qemu_fflush() won't work if file is non-writable */
> + return f->pos + f->buf_index;
> }
IIUC, this is more or less trying to make 'qemu_ftell' be
equivalent to 'qemu_ftell_fast' semantics in the non-writable
case. But that makes me wonder if whichever calls has problems,
shouldn't be just changed to use qemu_ftell_fast instead ?
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v3 11/17] migration/qemu-file: Fix qemu_ftell() for non-writable file
2022-06-16 11:20 ` Daniel P. Berrangé
@ 2022-06-16 12:54 ` Nikita
0 siblings, 0 replies; 20+ messages in thread
From: Nikita @ 2022-06-16 12:54 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: qemu-devel, den, andrey.drobyshev, quintela, dgilbert
On 6/16/22 2:20 PM, Daniel P. Berrangé wrote:
> On Thu, Jun 16, 2022 at 01:28:05PM +0300, nikita.lapshin@openvz.org wrote:
>> From: Nikita Lapshin <nikita.lapshin@openvz.org>
>>
>> qemu_ftell() will return wrong value for non-writable QEMUFile.
>> This happens due to call qemu_fflush() inside qemu_ftell(), this
>> function won't flush if file is readable.
> Well the return value isn't necessarily wrong today - it really
> depends what semantics each callers desires.
>
> Can you say what particular caller needs these semantics changed
> and the impact on them from current behaviour ?
Sorry, it's my bad. It was used in previous version. But after
refactoring I threw this function out of implementation.
So it is in fact not used now.
>
>> Signed-off-by: Nikita Lapshin <nikita.lapshin@openvz.org>
>> ---
>> migration/qemu-file.c | 3 ++-
>> 1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/migration/qemu-file.c b/migration/qemu-file.c
>> index 1479cddad9..53ccef80ac 100644
>> --- a/migration/qemu-file.c
>> +++ b/migration/qemu-file.c
>> @@ -663,7 +663,8 @@ int64_t qemu_ftell_fast(QEMUFile *f)
>> int64_t qemu_ftell(QEMUFile *f)
>> {
>> qemu_fflush(f);
>> - return f->pos;
>> + /* Consider that qemu_fflush() won't work if file is non-writable */
>> + return f->pos + f->buf_index;
>> }
> IIUC, this is more or less trying to make 'qemu_ftell' be
> equivalent to 'qemu_ftell_fast' semantics in the non-writable
> case. But that makes me wonder if whichever calls has problems,
> shouldn't be just changed to use qemu_ftell_fast instead ?
qemu_ftell_fast() counts iovec length. When we try to read from QEMUFile
using for exmaple qemu_peek_buffer f->buf will be filled with data so we
need to consider buf_index.
>
> With regards,
> Daniel
Thank you for your review!
^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2022-06-16 13:20 UTC | newest]
Thread overview: 20+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-06-16 10:27 [PATCH v3 00/17] migration/snapshot: External snapshot utility nikita.lapshin
2022-06-16 10:27 ` [PATCH v3 01/17] migration: Implemented new parameter stream_content nikita.lapshin
2022-06-16 10:27 ` [PATCH v3 02/17] migration: should_skip() implemented nikita.lapshin
2022-06-16 10:27 ` [PATCH v3 03/17] migration: Add vmstate part of migration stream nikita.lapshin
2022-06-16 10:27 ` [PATCH v3 04/17] migration: Add dirty-bitmaps " nikita.lapshin
2022-06-16 10:27 ` [PATCH v3 05/17] migration: Add block " nikita.lapshin
2022-06-16 10:28 ` [PATCH v3 06/17] migration: Add RAM " nikita.lapshin
2022-06-16 10:28 ` [PATCH v3 07/17] migration: analyze-migration script changed nikita.lapshin
2022-06-16 10:28 ` [PATCH v3 08/17] migration: Test for RAM and vmstate parts nikita.lapshin
2022-06-16 10:28 ` [PATCH v3 09/17] migration/snapshot: Introduce qemu-snapshot tool nikita.lapshin
2022-06-16 10:28 ` [PATCH v3 10/17] migration/snapshot: Build changes for qemu-snapshot-tool nikita.lapshin
2022-06-16 10:28 ` [PATCH v3 11/17] migration/qemu-file: Fix qemu_ftell() for non-writable file nikita.lapshin
2022-06-16 11:20 ` Daniel P. Berrangé
2022-06-16 12:54 ` Nikita
2022-06-16 10:28 ` [PATCH v3 12/17] migration/snapshot: Move RAM_SAVE_FLAG_xxx defines to migration/ram.h nikita.lapshin
2022-06-16 10:28 ` [PATCH v3 13/17] migration/snapshot: Block layer support in qemu-snapshot nikita.lapshin
2022-06-16 10:28 ` [PATCH v3 14/17] migration/snpashot: Implement API for RAMBlock nikita.lapshin
2022-06-16 10:28 ` [PATCH v3 15/17] migration/snapshot: Save part implement nikita.lapshin
2022-06-16 10:28 ` [PATCH v3 16/17] migration/snapshot: Precopy load implemented nikita.lapshin
2022-06-16 10:28 ` [PATCH v3 17/17] migration/snapshot: Postcopy " nikita.lapshin
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).