[Qemu-devel] Debugging Migration

[Qemu-devel] Debugging Migration

[John Snow]

Is there a convenient way of "pausing" or stalling a live migration to
allow methodical testing of race conditions?

I'd like to instrument something along the lines of:

(1) Live migration begins.
(2) migration is artificially halted or paused, but QEMU is allowed to run.
(3) Some additional qtest/QMP commands are received and processed.
(4) migration is allowed to resume.

Does anyone have perhaps even test patches to instrument this sort of
thing, or is it up to detective john to add it if he wants it?

Thanks,
--js

Re: [Qemu-devel] Debugging Migration

[Dr. David Alan Gilbert]

* John Snow (jsnow@redhat.com) wrote:
> Is there a convenient way of "pausing" or stalling a live migration to
> allow methodical testing of race conditions?
> 
> I'd like to instrument something along the lines of:
> 
> (1) Live migration begins.
> (2) migration is artificially halted or paused, but QEMU is allowed to run.
> (3) Some additional qtest/QMP commands are received and processed.
> (4) migration is allowed to resume.
> 
> Does anyone have perhaps even test patches to instrument this sort of
> thing, or is it up to detective john to add it if he wants it?

If you catch it during the iterative stage you can probably just
gdb or ctrl-z the destination and the migration thread should block;
or alternatively migrate to a pipe and similarly ctrl-z what ever is
there.

Mostly I do a few things:
  1) use tracing to follow it
     mostly just stderr tracing, but I've done systemtap scripts
     for some hairy stuff.
  2) Set the migration speed (migrate_set_speed) very very low
  3) Keep the source busy dirtying memory.

Of course that does lead to the question of what fun problem are
you trying to debug?

Dave

> --js
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] Debugging Migration

[Denis V. Lunev]

On 10/06/2015 09:46 PM, John Snow wrote:
> Is there a convenient way of "pausing" or stalling a live migration to
> allow methodical testing of race conditions?
>
> I'd like to instrument something along the lines of:
>
> (1) Live migration begins.
> (2) migration is artificially halted or paused, but QEMU is allowed to run.
> (3) Some additional qtest/QMP commands are received and processed.
> (4) migration is allowed to resume.
>
> Does anyone have perhaps even test patches to instrument this sort of
> thing, or is it up to detective john to add it if he wants it?
>
> Thanks,
> --js
>
we have performed some experiments with migration "test mode".
The idea was to estimate the time to migrate and estimate
expected downtime for distributed resource management.
In this mode the migration was not performed but all stuff around
like dirty bitmap was touched.

There are several patches for this in my queue but they are very
hackish.

Will it help you?

Den

Re: [Qemu-devel] Debugging Migration

[John Snow]

On 10/06/2015 06:40 PM, Denis V. Lunev wrote:
> On 10/06/2015 09:46 PM, John Snow wrote:
>> Is there a convenient way of "pausing" or stalling a live migration to
>> allow methodical testing of race conditions?
>>
>> I'd like to instrument something along the lines of:
>>
>> (1) Live migration begins.
>> (2) migration is artificially halted or paused, but QEMU is allowed to
>> run.
>> (3) Some additional qtest/QMP commands are received and processed.
>> (4) migration is allowed to resume.
>>
>> Does anyone have perhaps even test patches to instrument this sort of
>> thing, or is it up to detective john to add it if he wants it?
>>
>> Thanks,
>> --js
>>
> we have performed some experiments with migration "test mode".
> The idea was to estimate the time to migrate and estimate
> expected downtime for distributed resource management.
> In this mode the migration was not performed but all stuff around
> like dirty bitmap was touched.
> 
> There are several patches for this in my queue but they are very
> hackish.
> 
> Will it help you?
> 
> Den

It certainly won't hurt me to look at them if you'll share :)

--js

[Qemu-devel] [RFC 0/8] QEMUFile-way to gather VM's memory statistics

[Denis V. Lunev]

This code is not ready to be merged. It is sent in the hope that it could
be useful and we could invent better architectural approach for the
feature if possible.

+tch set summary:
- we added a feature that allows to gather VM's memory usage statistics
  with minimum interference with VM’s work (no transferring data thru
  network, probing time ~ downtime, etc.)
- this feature is implemented as a new capability for the migrate command.
  Migration started with this capability turned on will only gather dirty
  memory statistics regardless of given URI.

More details on measurement procedure:
To gather statistics, mechanisms created for live migration and snapshots
are used. This is the easiest approach for getting dirty bitmap of memory
pages.

Main points:
1. A function for bitmap resetting was added. During migration the process
   is bit-wise and so very time-consuming. What is more, it involves memory
   pages copying to the buffer.
2. A new hook hook_ram_sync() for QEMUFile was added. According to the
   previous point, we should pass the number of dirty pages before the
   resetting. As a result of passing control using hook_ram_sync() we can
   add a delay and let the VM to 'dirty' memory
3. New transport was created (QEMUFile implementation). As we want to use
   migration_thread() function, all logic that processes and outputs the
   gathered information should be hidden here.

Algorithm for dirty rate estimation:
1. Start memory logging (ram_save_setup())
2. The first reset of dirty bitmap, pass size of VM memory
3. Sleep на время равное downtime'у
4. Synchronization of bitmap
5. The second reset of dirty bitmap, pass size of dirty VM memory.
6. Compute dirty byte rate, assuming that time between two synchronization
   equals downtime.
7. Fill corresponding fields in MigrationState.
8. After finishing of hook_ram_sync return error code "-42"
9. In migration_thread() error code "-42" is interpreted as successful
   completion and MigrationStatus is changed

Signed-off-by: Denis V. Lunev <den@openvz.org>
Signed-off-by: Igor Redko <redkoi@virtuozzo.com>
Reviewed-by: Anna Melekhova <annam@virtuozzo.com>

Igor Redko (7):
  qemu-file: new hook in qemu-file
  migration: add new capability test-only
  migration: add function for reseting migration bitmap
  migration: add draft of new transport
  migration: implementation of hook_ram_sync
  migration: new migration test mode
  migration: add output of gathered statistics

 include/migration/migration.h |  5 +++
 include/migration/qemu-file.h |  1 +
 migration/Makefile.objs       |  2 +-
 migration/migration.c         | 53 +++++++++++++++++++++--
 migration/qemu-file.c         | 12 ++++++
 migration/ram.c               | 30 ++++++++++++-
 migration/test.c              | 99 +++++++++++++++++++++++++++++++++++++++++++
 qapi-schema.json              |  9 +++-
 qmp-commands.hx               |  1 +
 9 files changed, 204 insertions(+), 8 deletions(-)
 create mode 100644 migration/test.c

[Qemu-devel] [PATCH 1/8] migration: fix expected_downtime

[Denis V. Lunev]

From: Igor Redko <redkoi@virtuozzo.com>

To get this estimation we must divide pending_size by bandwidth
according to description of expected-downtime ("qmp-commands.hx:3246"):
  "expected-downtime": only present while migration is active
              total amount in ms for downtime that was calculated on
              the last bitmap round (json-int)

Previous version was just wrong because dirty_bytes_rate and bandwidth
are measured in Bytes/ms, so dividing first by second we get some
dimensionless quantity.
As it said in description above this value is showed during active
migration phase and recalculated only after transferring all memory
and if this process took more than 1 sec. So maybe just nobody noticed
that bug.

Signed-off-by: Igor Redko <redkoi@virtuozzo.com>
Reviewed-by: Anna Melekhova <annam@virtuozzo.com>
Signed-off-by: Denis V. Lunev <den@openvz.org>
---
 migration/migration.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/migration/migration.c b/migration/migration.c
index b7de9b7..b710a2d 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -1063,7 +1063,7 @@ static void *migration_thread(void *opaque)
             /* if we haven't sent anything, we don't want to recalculate
                10000 is a small enough number for our purposes */
             if (s->dirty_bytes_rate && transferred_bytes > 10000) {
-                s->expected_downtime = s->dirty_bytes_rate / bandwidth;
+                s->expected_downtime = pending_size / bandwidth;
             }
 
             qemu_file_reset_rate_limit(s->file);
-- 
2.1.4

[Qemu-devel] [PATCH 2/8] qemu-file: new hook in qemu-file

[Denis V. Lunev]

From: Igor Redko <redkoi@virtuozzo.com>

This patch adds hook_ram_sync() to QEMUFile abstraction. This hook
can be used for passing information about dirty memory.
An alternative is using existing hook_ram_load(). But this hook is
designed for incoming VM migration, so using it for outcoming VM
migration may complicate understanding of the code. On the other
hand, using existing code decreases volume of the patchset and its
impact.

Signed-off-by: Igor Redko <redkoi@virtuozzo.com>
Reviewed-by: Anna Melekhova <annam@virtuozzo.com>
Signed-off-by: Denis V. Lunev <den@openvz.org>
---
 include/migration/migration.h |  1 +
 include/migration/qemu-file.h |  1 +
 migration/qemu-file.c         | 12 ++++++++++++
 3 files changed, 14 insertions(+)

diff --git a/include/migration/migration.h b/include/migration/migration.h
index 8334621..deb0d21 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -182,6 +182,7 @@ bool migrate_use_events(void);
 void ram_control_before_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_after_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_load_hook(QEMUFile *f, uint64_t flags, void *data);
+void ram_control_sync_hook(QEMUFile *f, uint64_t flags, void *data);
 
 /* Whenever this is found in the data stream, the flags
  * will be passed to ram_control_load_hook in the incoming-migration
diff --git a/include/migration/qemu-file.h b/include/migration/qemu-file.h
index 29a338d..770dd98 100644
--- a/include/migration/qemu-file.h
+++ b/include/migration/qemu-file.h
@@ -105,6 +105,7 @@ typedef struct QEMUFileOps {
     QEMURamHookFunc *before_ram_iterate;
     QEMURamHookFunc *after_ram_iterate;
     QEMURamHookFunc *hook_ram_load;
+    QEMURamHookFunc *hook_ram_sync;
     QEMURamSaveFunc *save_page;
     QEMUFileShutdownFunc *shut_down;
 } QEMUFileOps;
diff --git a/migration/qemu-file.c b/migration/qemu-file.c
index 49addf6..a05d672 100644
--- a/migration/qemu-file.c
+++ b/migration/qemu-file.c
@@ -168,6 +168,18 @@ void ram_control_load_hook(QEMUFile *f, uint64_t flags, void *data)
     }
 }
 
+void ram_control_sync_hook(QEMUFile *f, uint64_t flags, void *data)
+{
+    int ret = 0;
+
+    if (f->ops->hook_ram_sync) {
+        ret = f->ops->hook_ram_sync(f, f->opaque, flags, data);
+        if (ret < 0) {
+            qemu_file_set_error(f, ret);
+        }
+    }
+}
+
 size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset,
                              ram_addr_t offset, size_t size,
                              uint64_t *bytes_sent)
-- 
2.1.4

[Qemu-devel] [PATCH 3/8] migration: add new capability test-only

[Denis V. Lunev]

From: Igor Redko <redkoi@virtuozzo.com>

This patch declares a new migration capability that allows
to distinguish between true migration and the test.

An alternative is using a custom URI but in this case the
following problems should be solved:
1/ QEMUFile abstraction hides the real transport type
2/ We must be ensured that VM will not be stopped due to
“successful migration”

Possible solutions:
1/ Using global variable analogous to
s->enabled_capabilities[MIGRATION_CAPABILITY_TEST_ONLY] that
will be initialized according to the migration type at the
beginning of the migration process
2/ Additional hook, presence of implementation of which allows
to decide whether the migration is true

Signed-off-by: Igor Redko <redkoi@virtuozzo.com>
Reviewed-by: Anna Melekhova <annam@virtuozzo.com>
Signed-off-by: Denis V. Lunev <den@openvz.org>
---
 include/migration/migration.h | 2 ++
 migration/migration.c         | 9 +++++++++
 qapi-schema.json              | 5 ++++-
 qmp-commands.hx               | 1 +
 4 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/include/migration/migration.h b/include/migration/migration.h
index deb0d21..8611750 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -179,6 +179,8 @@ int migrate_compress_threads(void);
 int migrate_decompress_threads(void);
 bool migrate_use_events(void);
 
+bool migrate_is_test(void);
+
 void ram_control_before_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_after_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_load_hook(QEMUFile *f, uint64_t flags, void *data);
diff --git a/migration/migration.c b/migration/migration.c
index b710a2d..e0cad54 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -954,6 +954,15 @@ int migrate_use_xbzrle(void)
     return s->enabled_capabilities[MIGRATION_CAPABILITY_XBZRLE];
 }
 
+bool migrate_is_test(void)
+{
+    MigrationState *s;
+
+    s = migrate_get_current();
+
+    return s->enabled_capabilities[MIGRATION_CAPABILITY_TEST_ONLY];
+}
+
 int64_t migrate_xbzrle_cache_size(void)
 {
     MigrationState *s;
diff --git a/qapi-schema.json b/qapi-schema.json
index 8b0520c..38bf199 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -540,11 +540,14 @@
 # @auto-converge: If enabled, QEMU will automatically throttle down the guest
 #          to speed up convergence of RAM migration. (since 1.6)
 #
+# @test-only: If enabled, QEMU instead of migrating will estimate migration
+#          time basing on given downtime and current dirty page rate
+#
 # Since: 1.2
 ##
 { 'enum': 'MigrationCapability',
   'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks',
-           'compress', 'events'] }
+           'compress', 'events', 'test-only'] }
 
 ##
 # @MigrationCapabilityStatus
diff --git a/qmp-commands.hx b/qmp-commands.hx
index d2ba800..741d088 100644
--- a/qmp-commands.hx
+++ b/qmp-commands.hx
@@ -3456,6 +3456,7 @@ Enable/Disable migration capabilities
 - "auto-converge": throttle down guest to help convergence of migration
 - "zero-blocks": compress zero blocks during block migration
 - "events": generate events for each migration state change
+- "test-only": don't send any data, instead estimate dirty page rate
 
 Arguments:
 
-- 
2.1.4

[Qemu-devel] [PATCH 4/8] migration: add function for reseting migration bitmap

[Denis V. Lunev]

From: Igor Redko <redkoi@virtuozzo.com>

Adds function ram_migration_bitmap_reset(), which resets
migration_bitmap and sends a number of dirtied bytes since the
last call.

During estimation of dirty bytes rate and migration downtime
we must avoid any copying and transferring data, but we also need
to get the number of dirtied bytes and pass it to an estimator.
And even more importantly - we MUST NOT stop the virtual machine
during test. So we only do “begin”, “pending” and “iterate” stages
of migration.

>if ((pending_size && pending_size >= max_size)
>     || (migrate_is_test())) {
>    qemu_savevm_state_iterate(s->file);
>} else {

If we didn't explicitly check the migration for being testing one,
 we would have to make expression
"(pending_size && pending_size >= max_size)" always evaluate as true
during test. For example we could set max_downtime to 0, so that
max_size would be 0. But the check for pending_size remains.
This check was added with commit
https://github.com/qemu/qemu/commit/b22ff1fbed9d7f1f677804cbaa9ee03ca17d0013
If we removed this check, migration might hang at “iterate” stage
in ram_find_and_save_block() (that happens under rare circumstances)
If we left this check as is, VM would stop during test (that would happen
also under rare condition: no pages have been dirtied during test).

Signed-off-by: Igor Redko <redkoi@virtuozzo.com>
Reviewed-by: Anna Melekhova <annam@virtuozzo.com>
Signed-off-by: Denis V. Lunev <den@openvz.org>
---
 migration/migration.c |  3 ++-
 migration/ram.c       | 26 +++++++++++++++++++++++++-
 2 files changed, 27 insertions(+), 2 deletions(-)

diff --git a/migration/migration.c b/migration/migration.c
index e0cad54..d6cb3e2 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -1043,7 +1043,8 @@ static void *migration_thread(void *opaque)
         if (!qemu_file_rate_limit(s->file)) {
             pending_size = qemu_savevm_state_pending(s->file, max_size);
             trace_migrate_pending(pending_size, max_size);
-            if (pending_size && pending_size >= max_size) {
+            if ((pending_size && pending_size >= max_size)
+                || (migrate_is_test())) {
                 qemu_savevm_state_iterate(s->file);
             } else {
                 trace_migration_thread_low_pending(pending_size);
diff --git a/migration/ram.c b/migration/ram.c
index 2d1d0b9..fbf0b7a 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -1120,6 +1120,25 @@ static void ram_migration_cancel(void *opaque)
     migration_end();
 }
 
+static uint64_t ram_migration_bitmap_reset(void)
+{
+    uint64_t dirty_pages_remaining;
+    int64_t ram_bitmap_pages; /* Size of bitmap in pages, including gaps */
+    /* TODO think about more locks?
+     * For now only using for prediction so the only another writer
+     * is migration_bitmap_sync_range()
+     */
+    qemu_mutex_lock(&migration_bitmap_mutex);
+    rcu_read_lock();
+    ram_bitmap_pages = last_ram_offset() >> TARGET_PAGE_BITS;
+    dirty_pages_remaining = migration_dirty_pages;
+    bitmap_zero(migration_bitmap, ram_bitmap_pages);
+    migration_dirty_pages = 0;
+    rcu_read_unlock();
+    qemu_mutex_unlock(&migration_bitmap_mutex);
+    return dirty_pages_remaining;
+}
+
 static void reset_ram_globals(void)
 {
     last_seen_block = NULL;
@@ -1249,6 +1268,10 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
     int64_t t0;
     int pages_sent = 0;
 
+    if (migrate_is_test()) {
+        return ram_migration_bitmap_reset();
+    }
+
     rcu_read_lock();
     if (ram_list.version != last_version) {
         reset_ram_globals();
@@ -1346,13 +1369,14 @@ static uint64_t ram_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)
 
     remaining_size = ram_save_remaining() * TARGET_PAGE_SIZE;
 
-    if (remaining_size < max_size) {
+    if ((remaining_size < max_size) || (migrate_is_test())) {
         qemu_mutex_lock_iothread();
         rcu_read_lock();
         migration_bitmap_sync();
         rcu_read_unlock();
         qemu_mutex_unlock_iothread();
         remaining_size = ram_save_remaining() * TARGET_PAGE_SIZE;
+        ram_control_sync_hook(f, RAM_CONTROL_HOOK, &remaining_size);
     }
     return remaining_size;
 }
-- 
2.1.4

[Qemu-devel] [PATCH 5/8] migration: add draft of new transport

[Denis V. Lunev]

From: Igor Redko <redkoi@virtuozzo.com>

In this patch transport test is added. It can be used to estimate
the possibility of live migration given downtime and bandwidth.
In this patch basic functionality is implemented to meet the
QEMUFile interface requirements.

This transport is write-only. Moreover, it saves only the size of
the transferred data and drops the data itself.

Also, the Makefile modification to link this file included.

Signed-off-by: Igor Redko <redkoi@virtuozzo.com>
Reviewed-by: Anna Melekhova <annam@virtuozzo.com>
Signed-off-by: Denis V. Lunev <den@openvz.org>
---
 include/migration/migration.h |  2 ++
 migration/Makefile.objs       |  2 +-
 migration/test.c              | 66 +++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 69 insertions(+), 1 deletion(-)
 create mode 100644 migration/test.c

diff --git a/include/migration/migration.h b/include/migration/migration.h
index 8611750..555267b 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -108,6 +108,8 @@ void rdma_start_outgoing_migration(void *opaque, const char *host_port, Error **
 
 void rdma_start_incoming_migration(const char *host_port, Error **errp);
 
+void test_start_migration(void *opaque, const char *host_port, Error **errp);
+
 void migrate_fd_error(MigrationState *s);
 
 void migrate_fd_connect(MigrationState *s);
diff --git a/migration/Makefile.objs b/migration/Makefile.objs
index d929e96..2da590b 100644
--- a/migration/Makefile.objs
+++ b/migration/Makefile.objs
@@ -1,4 +1,4 @@
-common-obj-y += migration.o tcp.o
+common-obj-y += migration.o tcp.o test.o
 common-obj-y += vmstate.o
 common-obj-y += qemu-file.o qemu-file-buf.o qemu-file-unix.o qemu-file-stdio.o
 common-obj-y += xbzrle.o
diff --git a/migration/test.c b/migration/test.c
new file mode 100644
index 0000000..8d06988
--- /dev/null
+++ b/migration/test.c
@@ -0,0 +1,66 @@
+#include "qemu-common.h"
+#include "migration/migration.h"
+#include "migration/qemu-file.h"
+#include "exec/cpu-common.h"
+#include "qemu/error-report.h"
+#include "qemu/main-loop.h"
+#include "qemu/sockets.h"
+#include "qemu/bitmap.h"
+#include <stdio.h>
+#include <string.h>
+#include "qmp-commands.h"
+
+typedef struct QEMUFileTest {
+    MigrationState *s;
+    size_t len;
+    QEMUFile *file;
+} QEMUFileTest;
+
+static uint64_t transfered_bytes;
+static uint64_t initial_bytes;
+
+static ssize_t qemu_test_put_buffer(void *opaque, const uint8_t *buf,
+                                    int64_t pos, size_t size)
+{
+    transfered_bytes += size;
+    return size;
+}
+
+static int qemu_test_close(void *opaque)
+{
+    return 0;
+}
+
+static const QEMUFileOps test_write_ops = {
+    .put_buffer         = qemu_test_put_buffer,
+    .close              = qemu_test_close,
+};
+
+static void *qemu_fopen_test(MigrationState *s, const char *mode)
+{
+    QEMUFileTest *t;
+    transfered_bytes = 0;
+    initial_bytes = 0;
+    if (qemu_file_mode_is_not_valid(mode)) {
+        return NULL;
+    }
+
+    t = g_malloc0(sizeof(QEMUFileTest));
+    t->s = s;
+
+    if (mode[0] == 'w') {
+        t->file = qemu_fopen_ops(s, &test_write_ops);
+    } else {
+        return NULL;
+    }
+    qemu_file_set_rate_limit(t->file, -1);
+    return t->file;
+}
+
+void test_start_migration(void *opaque, const char *host_port, Error **errp)
+{
+    MigrationState *s = opaque;
+    s->file = qemu_fopen_test(s, "wb");
+    migrate_fd_connect(s);
+    return;
+}
-- 
2.1.4

[Qemu-devel] [PATCH 6/8] migration: implementation of hook_ram_sync

[Denis V. Lunev]

From: Igor Redko <redkoi@virtuozzo.com>

The key feature of the test transport is receiving information
about dirty memory. The qemu_test_sync_hook() allows to use
the migration infrastructure(code) for this purpose.

All calls of this hook will be from ram_save_pending().

At the first call of this hook we need to save the initial
size of VM memory and put the migration thread to sleep for
decent period (downtime for example). During this period
guest would dirty memory.

The second and the last call.
We make our estimation of dirty bytes rate assuming that time
between two synchronizations of dirty bitmap differs from downtime
negligibly.

An alternative to this approach is receiving information about
size of data “transmitted” through the transport. However, this
way creates large time and memory overheads:
1/Transmitted guest’s memory pages are copied to QEMUFile’s buffer
  (~8 sec per 4GB VM)
2/Dirty memory pages are processed one by one (~60msec per 4GB VM)

Signed-off-by: Igor Redko <redkoi@virtuozzo.com>
Reviewed-by: Anna Melekhova <annam@virtuozzo.com>
Signed-off-by: Denis V. Lunev <den@openvz.org>
---
 migration/migration.c |  8 ++++++++
 migration/test.c      | 36 ++++++++++++++++++++++++++++++++++++
 2 files changed, 44 insertions(+)

diff --git a/migration/migration.c b/migration/migration.c
index d6cb3e2..3182e15 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -1058,6 +1058,14 @@ static void *migration_thread(void *opaque)
                               MIGRATION_STATUS_FAILED);
             break;
         }
+
+        if (migrate_is_test()) {
+            /* since no data is transfered during estimation all
+               all measurements below will be incorrect.
+               as well no need for delays. */
+            continue;
+        }
+
         current_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
         if (current_time >= initial_time + BUFFER_DELAY) {
             uint64_t transferred_bytes = qemu_ftell(s->file) - initial_bytes;
diff --git a/migration/test.c b/migration/test.c
index 8d06988..b4d0761 100644
--- a/migration/test.c
+++ b/migration/test.c
@@ -18,6 +18,7 @@ typedef struct QEMUFileTest {
 
 static uint64_t transfered_bytes;
 static uint64_t initial_bytes;
+static int sync_cnt;
 
 static ssize_t qemu_test_put_buffer(void *opaque, const uint8_t *buf,
                                     int64_t pos, size_t size)
@@ -31,7 +32,41 @@ static int qemu_test_close(void *opaque)
     return 0;
 }
 
+static int qemu_test_sync_hook(QEMUFile *f, void *opaque,
+                                        uint64_t flags, void *data)
+{
+    static uint64_t dirtied_bytes;
+    static uint64_t sleeptime_mcs;
+    int64_t time_delta;
+    uint64_t remaining_bytes = *((uint64_t *) data);
+    MigrationState *s = (MigrationState *) opaque;
+    switch (sync_cnt++) {
+    case 0:
+    /* First call will be from ram_save_begin
+     * so we need to save initial size of VM memory
+     * and sleep for decent period (downtime for example). */
+        sleeptime_mcs = migrate_max_downtime()/1000;
+        initial_bytes = remaining_bytes;
+        usleep(sleeptime_mcs);
+        break;
+    case 1:
+    /* Second and last call.
+     * We assume that time between two synchronizations of
+     * dirty bitmap differs from downtime negligibly and
+     * make our estimation of dirty bytes rate. */
+        dirtied_bytes = remaining_bytes;
+        time_delta = sleeptime_mcs / 1000;
+        s->dirty_bytes_rate = dirtied_bytes * 1000 / time_delta;
+        return -42;
+    default:
+    /* All calls after second are errors */
+        return -1;
+    }
+        return 0;
+}
+
 static const QEMUFileOps test_write_ops = {
+    .hook_ram_sync      = qemu_test_sync_hook,
     .put_buffer         = qemu_test_put_buffer,
     .close              = qemu_test_close,
 };
@@ -41,6 +76,7 @@ static void *qemu_fopen_test(MigrationState *s, const char *mode)
     QEMUFileTest *t;
     transfered_bytes = 0;
     initial_bytes = 0;
+    sync_cnt = 0;
     if (qemu_file_mode_is_not_valid(mode)) {
         return NULL;
     }
-- 
2.1.4

[Qemu-devel] [PATCH 7/8] migration: new migration test mode

[Denis V. Lunev]

From: Igor Redko <redkoi@virtuozzo.com>

In this patch the ability to start a migration with test-only
capability was added. It allows to gather the guest VM’s memory
usage statistics avoiding time and memory overheads and real
data transmission.  New MIGRATION_STATUS_TEST_COMPLETED was
added to distinguish between test migration and true migration
success states.

Signed-off-by: Igor Redko <redkoi@virtuozzo.com>
Reviewed-by: Anna Melekhova <annam@virtuozzo.com>
Signed-off-by: Denis V. Lunev <den@openvz.org>
---
 migration/migration.c | 12 ++++++++++--
 qapi-schema.json      |  4 +++-
 2 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/migration/migration.c b/migration/migration.c
index 3182e15..3470d39 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -790,7 +790,9 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
 
     s = migrate_init(&params);
 
-    if (strstart(uri, "tcp:", &p)) {
+    if (migrate_is_test()) {
+        test_start_migration(s, p, &local_err);
+    } else if (strstart(uri, "tcp:", &p)) {
         tcp_start_outgoing_migration(s, p, &local_err);
 #ifdef CONFIG_RDMA
     } else if (strstart(uri, "rdma:", &p)) {
@@ -1054,8 +1056,14 @@ static void *migration_thread(void *opaque)
         }
 
         if (qemu_file_get_error(s->file)) {
-            migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
+            /*FIXME replace magic number with smth legit*/
+            if (migrate_is_test() && qemu_file_get_error(s->file) == -42) {
+                migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
+                              MIGRATION_STATUS_TEST_COMPLETED);
+            } else {
+                migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
                               MIGRATION_STATUS_FAILED);
+            }
             break;
         }
 
diff --git a/qapi-schema.json b/qapi-schema.json
index 38bf199..e022f9c 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -432,6 +432,8 @@
 #
 # @completed: migration is finished.
 #
+# @test-completed: migration time estimation finished.
+#
 # @failed: some error occurred during migration process.
 #
 # Since: 2.3
@@ -439,7 +441,7 @@
 ##
 { 'enum': 'MigrationStatus',
   'data': [ 'none', 'setup', 'cancelling', 'cancelled',
-            'active', 'completed', 'failed' ] }
+            'active', 'completed', 'test-completed', 'failed' ] }
 
 ##
 # @MigrationInfo
-- 
2.1.4

[Qemu-devel] [PATCH 8/8] migration: add output of gathered statistics

[Denis V. Lunev]

From: Igor Redko <redkoi@virtuozzo.com>

In this patch the output of the test results was added.
If the test was successfully completed, it would be displayed
in info migrate.

Signed-off-by: Igor Redko <redkoi@virtuozzo.com>
Reviewed-by: Anna Melekhova <annam@virtuozzo.com>
Signed-off-by: Denis V. Lunev <den@openvz.org>
---
 migration/migration.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/migration/migration.c b/migration/migration.c
index 3470d39..b1c5162 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -454,6 +454,24 @@ MigrationInfo *qmp_query_migrate(Error **errp)
 
         get_xbzrle_cache_stats(info);
         break;
+    case MIGRATION_STATUS_TEST_COMPLETED:
+        info->has_status = true;
+
+        info->has_ram = true;
+        info->ram = g_malloc0(sizeof(*info->ram));
+        info->ram->total = ram_bytes_total();
+        info->ram->dirty_pages_rate = s->dirty_bytes_rate;
+        info->ram->mbps = s->mbps;
+        info->ram->dirty_sync_count = s->dirty_sync_count;
+
+        if (blk_mig_active()) {
+            info->has_disk = true;
+            info->disk = g_malloc0(sizeof(*info->disk));
+            info->disk->transferred = blk_mig_bytes_transferred();
+            info->disk->remaining = blk_mig_bytes_remaining();
+            info->disk->total = blk_mig_bytes_total();
+        }
+        break;
     case MIGRATION_STATUS_COMPLETED:
         get_xbzrle_cache_stats(info);
 
-- 
2.1.4

Re: [Qemu-devel] Debugging Migration

[Denis V. Lunev]

On 10/07/2015 02:02 AM, John Snow wrote:
>
> On 10/06/2015 06:40 PM, Denis V. Lunev wrote:
>> On 10/06/2015 09:46 PM, John Snow wrote:
>>> Is there a convenient way of "pausing" or stalling a live migration to
>>> allow methodical testing of race conditions?
>>>
>>> I'd like to instrument something along the lines of:
>>>
>>> (1) Live migration begins.
>>> (2) migration is artificially halted or paused, but QEMU is allowed to
>>> run.
>>> (3) Some additional qtest/QMP commands are received and processed.
>>> (4) migration is allowed to resume.
>>>
>>> Does anyone have perhaps even test patches to instrument this sort of
>>> thing, or is it up to detective john to add it if he wants it?
>>>
>>> Thanks,
>>> --js
>>>
>> we have performed some experiments with migration "test mode".
>> The idea was to estimate the time to migrate and estimate
>> expected downtime for distributed resource management.
>> In this mode the migration was not performed but all stuff around
>> like dirty bitmap was touched.
>>
>> There are several patches for this in my queue but they are very
>> hackish.
>>
>> Will it help you?
>>
>> Den
> It certainly won't hurt me to look at them if you'll share :)
>
> --js

sent as
[RFC 0/8]  QEMUFile-way to gather VM's memory statistics

Sorry, I have lost threading accidentally.

Once again, that code is dirty and IMHO does not meet QEMU coding
standards at the moment.

Den

Re: [Qemu-devel] [PATCH 6/8] migration: implementation of hook_ram_sync

[Paolo Bonzini]

On 07/10/2015 08:20, Denis V. Lunev wrote:
> 
> All calls of this hook will be from ram_save_pending().
> 
> At the first call of this hook we need to save the initial
> size of VM memory and put the migration thread to sleep for
> decent period (downtime for example). During this period
> guest would dirty memory.
> 
> The second and the last call.
> We make our estimation of dirty bytes rate assuming that time
> between two synchronizations of dirty bitmap differs from downtime
> negligibly.
> 
> An alternative to this approach is receiving information about
> size of data “transmitted” through the transport.

This would use before_ram_iterate/after_ram_iterate, right?

> However, this
> way creates large time and memory overheads:
> 1/Transmitted guest’s memory pages are copied to QEMUFile’s buffer
>   (~8 sec per 4GB VM)

Note that they are not if you implement writev_buffer.

> 2/Dirty memory pages are processed one by one (~60msec per 4GB VM)

That however improves the accuracy, doesn't it?

Paolo

Re: [Qemu-devel] [PATCH 6/8] migration: implementation of hook_ram_sync

[Paolo Bonzini]

On 07/10/2015 08:20, Denis V. Lunev wrote:
> +        if (migrate_is_test()) {
> +            /* since no data is transfered during estimation all
> +               all measurements below will be incorrect.
> +               as well no need for delays. */
> +            continue;
> +        }

By applying delays, you can also test migration using bandwidth
limitations and try to estimate whether it will converge or not.
Perhaps if you use writev_buffer to implement the test QEMUFile you do
not need this anymore.

Paolo

Re: [Qemu-devel] [PATCH 7/8] migration: new migration test mode

[Dr. David Alan Gilbert]

* Denis V. Lunev (den@openvz.org) wrote:
> From: Igor Redko <redkoi@virtuozzo.com>
> 
> In this patch the ability to start a migration with test-only
> capability was added. It allows to gather the guest VM’s memory
> usage statistics avoiding time and memory overheads and real
> data transmission.  New MIGRATION_STATUS_TEST_COMPLETED was
> added to distinguish between test migration and true migration
> success states.

Why isn't this just a new transport? i.e. I could do this just by doing
a migrate to test:   ?

It seems simpler and avoids some of the special casing?

Dave

> Signed-off-by: Igor Redko <redkoi@virtuozzo.com>
> Reviewed-by: Anna Melekhova <annam@virtuozzo.com>
> Signed-off-by: Denis V. Lunev <den@openvz.org>
> ---
>  migration/migration.c | 12 ++++++++++--
>  qapi-schema.json      |  4 +++-
>  2 files changed, 13 insertions(+), 3 deletions(-)
> 
> diff --git a/migration/migration.c b/migration/migration.c
> index 3182e15..3470d39 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -790,7 +790,9 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
>  
>      s = migrate_init(&params);
>  
> -    if (strstart(uri, "tcp:", &p)) {
> +    if (migrate_is_test()) {
> +        test_start_migration(s, p, &local_err);
> +    } else if (strstart(uri, "tcp:", &p)) {
>          tcp_start_outgoing_migration(s, p, &local_err);
>  #ifdef CONFIG_RDMA
>      } else if (strstart(uri, "rdma:", &p)) {
> @@ -1054,8 +1056,14 @@ static void *migration_thread(void *opaque)
>          }
>  
>          if (qemu_file_get_error(s->file)) {
> -            migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
> +            /*FIXME replace magic number with smth legit*/
> +            if (migrate_is_test() && qemu_file_get_error(s->file) == -42) {
> +                migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
> +                              MIGRATION_STATUS_TEST_COMPLETED);
> +            } else {
> +                migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
>                                MIGRATION_STATUS_FAILED);
> +            }
>              break;
>          }
>  
> diff --git a/qapi-schema.json b/qapi-schema.json
> index 38bf199..e022f9c 100644
> --- a/qapi-schema.json
> +++ b/qapi-schema.json
> @@ -432,6 +432,8 @@
>  #
>  # @completed: migration is finished.
>  #
> +# @test-completed: migration time estimation finished.
> +#
>  # @failed: some error occurred during migration process.
>  #
>  # Since: 2.3
> @@ -439,7 +441,7 @@
>  ##
>  { 'enum': 'MigrationStatus',
>    'data': [ 'none', 'setup', 'cancelling', 'cancelled',
> -            'active', 'completed', 'failed' ] }
> +            'active', 'completed', 'test-completed', 'failed' ] }
>  
>  ##
>  # @MigrationInfo
> -- 
> 2.1.4
> 
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [PATCH 6/8] migration: implementation of hook_ram_sync

[Dr. David Alan Gilbert]

* Denis V. Lunev (den@openvz.org) wrote:
> From: Igor Redko <redkoi@virtuozzo.com>
> 
> The key feature of the test transport is receiving information
> about dirty memory. The qemu_test_sync_hook() allows to use
> the migration infrastructure(code) for this purpose.
> 
> All calls of this hook will be from ram_save_pending().
> 
> At the first call of this hook we need to save the initial
> size of VM memory and put the migration thread to sleep for
> decent period (downtime for example). During this period
> guest would dirty memory.
> 
> The second and the last call.
> We make our estimation of dirty bytes rate assuming that time
> between two synchronizations of dirty bitmap differs from downtime
> negligibly.
> 
> An alternative to this approach is receiving information about
> size of data “transmitted” through the transport. However, this
> way creates large time and memory overheads:
> 1/Transmitted guest’s memory pages are copied to QEMUFile’s buffer
>   (~8 sec per 4GB VM)
> 2/Dirty memory pages are processed one by one (~60msec per 4GB VM)

That's not true for two reasons:
   1) As long as you register a writev_buffer method on the QEMUFile
      RAM Pages get added by using add_to_iovec rather than actually
      copying the data; so all the other stuff does go that way (as
      do the page headers)

   2) If you make it look like the rdma transport and register the 'save_page'
      hook I think the overhead is even smaller.

Dave

> 
> Signed-off-by: Igor Redko <redkoi@virtuozzo.com>
> Reviewed-by: Anna Melekhova <annam@virtuozzo.com>
> Signed-off-by: Denis V. Lunev <den@openvz.org>
> ---
>  migration/migration.c |  8 ++++++++
>  migration/test.c      | 36 ++++++++++++++++++++++++++++++++++++
>  2 files changed, 44 insertions(+)
> 
> diff --git a/migration/migration.c b/migration/migration.c
> index d6cb3e2..3182e15 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -1058,6 +1058,14 @@ static void *migration_thread(void *opaque)
>                                MIGRATION_STATUS_FAILED);
>              break;
>          }
> +
> +        if (migrate_is_test()) {
> +            /* since no data is transfered during estimation all
> +               all measurements below will be incorrect.
> +               as well no need for delays. */
> +            continue;
> +        }
> +
>          current_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
>          if (current_time >= initial_time + BUFFER_DELAY) {
>              uint64_t transferred_bytes = qemu_ftell(s->file) - initial_bytes;
> diff --git a/migration/test.c b/migration/test.c
> index 8d06988..b4d0761 100644
> --- a/migration/test.c
> +++ b/migration/test.c
> @@ -18,6 +18,7 @@ typedef struct QEMUFileTest {
>  
>  static uint64_t transfered_bytes;
>  static uint64_t initial_bytes;
> +static int sync_cnt;
>  
>  static ssize_t qemu_test_put_buffer(void *opaque, const uint8_t *buf,
>                                      int64_t pos, size_t size)
> @@ -31,7 +32,41 @@ static int qemu_test_close(void *opaque)
>      return 0;
>  }
>  
> +static int qemu_test_sync_hook(QEMUFile *f, void *opaque,
> +                                        uint64_t flags, void *data)
> +{
> +    static uint64_t dirtied_bytes;
> +    static uint64_t sleeptime_mcs;
> +    int64_t time_delta;
> +    uint64_t remaining_bytes = *((uint64_t *) data);
> +    MigrationState *s = (MigrationState *) opaque;
> +    switch (sync_cnt++) {
> +    case 0:
> +    /* First call will be from ram_save_begin
> +     * so we need to save initial size of VM memory
> +     * and sleep for decent period (downtime for example). */
> +        sleeptime_mcs = migrate_max_downtime()/1000;
> +        initial_bytes = remaining_bytes;
> +        usleep(sleeptime_mcs);
> +        break;
> +    case 1:
> +    /* Second and last call.
> +     * We assume that time between two synchronizations of
> +     * dirty bitmap differs from downtime negligibly and
> +     * make our estimation of dirty bytes rate. */
> +        dirtied_bytes = remaining_bytes;
> +        time_delta = sleeptime_mcs / 1000;
> +        s->dirty_bytes_rate = dirtied_bytes * 1000 / time_delta;
> +        return -42;
> +    default:
> +    /* All calls after second are errors */
> +        return -1;
> +    }
> +        return 0;
> +}
> +
>  static const QEMUFileOps test_write_ops = {
> +    .hook_ram_sync      = qemu_test_sync_hook,
>      .put_buffer         = qemu_test_put_buffer,
>      .close              = qemu_test_close,
>  };
> @@ -41,6 +76,7 @@ static void *qemu_fopen_test(MigrationState *s, const char *mode)
>      QEMUFileTest *t;
>      transfered_bytes = 0;
>      initial_bytes = 0;
> +    sync_cnt = 0;
>      if (qemu_file_mode_is_not_valid(mode)) {
>          return NULL;
>      }
> -- 
> 2.1.4
> 
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC 0/8] QEMUFile-way to gather VM's memory statistics

[Dr. David Alan Gilbert]

(ccing Juan and Amit in)

* Denis V. Lunev (den@openvz.org) wrote:
> This code is not ready to be merged. It is sent in the hope that it could
> be useful and we could invent better architectural approach for the
> feature if possible.
> 
> +tch set summary:
> - we added a feature that allows to gather VM's memory usage statistics
>   with minimum interference with VM’s work (no transferring data thru
>   network, probing time ~ downtime, etc.)
> - this feature is implemented as a new capability for the migrate command.
>   Migration started with this capability turned on will only gather dirty
>   memory statistics regardless of given URI.
> 
> More details on measurement procedure:
> To gather statistics, mechanisms created for live migration and snapshots
> are used. This is the easiest approach for getting dirty bitmap of memory
> pages.

Hi Denis,
  Thank you; it's something I know we had wondered about doing, so we could
get some idea of how bad a particular application was.
  We'd also been thinking of doing it as a dummy transport.


> Main points:
> 1. A function for bitmap resetting was added. During migration the process
>    is bit-wise and so very time-consuming. What is more, it involves memory
>    pages copying to the buffer.
> 2. A new hook hook_ram_sync() for QEMUFile was added. According to the
>    previous point, we should pass the number of dirty pages before the
>    resetting. As a result of passing control using hook_ram_sync() we can
>    add a delay and let the VM to 'dirty' memory
> 3. New transport was created (QEMUFile implementation). As we want to use
>    migration_thread() function, all logic that processes and outputs the
>    gathered information should be hidden here.
> 
> Algorithm for dirty rate estimation:
> 1. Start memory logging (ram_save_setup())
> 2. The first reset of dirty bitmap, pass size of VM memory
> 3. Sleep на время равное downtime'у

 | translate > 'for a time equal to'

> 4. Synchronization of bitmap
> 5. The second reset of dirty bitmap, pass size of dirty VM memory.
> 6. Compute dirty byte rate, assuming that time between two synchronization
>    equals downtime.
> 7. Fill corresponding fields in MigrationState.
> 8. After finishing of hook_ram_sync return error code "-42"
> 9. In migration_thread() error code "-42" is interpreted as successful
>    completion and MigrationStatus is changed

Yeh I'm not quite sure why you don't just let it fall through the end of migration
and dump these into the normal output of info migrate in the 'completed' output.

Dave

> 
> Signed-off-by: Denis V. Lunev <den@openvz.org>
> Signed-off-by: Igor Redko <redkoi@virtuozzo.com>
> Reviewed-by: Anna Melekhova <annam@virtuozzo.com>
> 
> Igor Redko (7):
>   qemu-file: new hook in qemu-file
>   migration: add new capability test-only
>   migration: add function for reseting migration bitmap
>   migration: add draft of new transport
>   migration: implementation of hook_ram_sync
>   migration: new migration test mode
>   migration: add output of gathered statistics
> 
>  include/migration/migration.h |  5 +++
>  include/migration/qemu-file.h |  1 +
>  migration/Makefile.objs       |  2 +-
>  migration/migration.c         | 53 +++++++++++++++++++++--
>  migration/qemu-file.c         | 12 ++++++
>  migration/ram.c               | 30 ++++++++++++-
>  migration/test.c              | 99 +++++++++++++++++++++++++++++++++++++++++++
>  qapi-schema.json              |  9 +++-
>  qmp-commands.hx               |  1 +
>  9 files changed, 204 insertions(+), 8 deletions(-)
>  create mode 100644 migration/test.c
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [PATCH 3/8] migration: add new capability test-only

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1543 bytes --]

On 10/07/2015 12:20 AM, Denis V. Lunev wrote:
> From: Igor Redko <redkoi@virtuozzo.com>
> 
> This patch declares a new migration capability that allows
> to distinguish between true migration and the test.
> 
> An alternative is using a custom URI but in this case the
> following problems should be solved:
> 1/ QEMUFile abstraction hides the real transport type
> 2/ We must be ensured that VM will not be stopped due to
> “successful migration”
> 
> Possible solutions:
> 1/ Using global variable analogous to
> s->enabled_capabilities[MIGRATION_CAPABILITY_TEST_ONLY] that
> will be initialized according to the migration type at the
> beginning of the migration process
> 2/ Additional hook, presence of implementation of which allows
> to decide whether the migration is true
> 

Do we want to name the capability 'x-test-only', since this series is
experimental, to make it obvious that production code should not rely on it?


> +++ b/qapi-schema.json
> @@ -540,11 +540,14 @@
>  # @auto-converge: If enabled, QEMU will automatically throttle down the guest
>  #          to speed up convergence of RAM migration. (since 1.6)
>  #
> +# @test-only: If enabled, QEMU instead of migrating will estimate migration
> +#          time basing on given downtime and current dirty page rate

If we do decide to add this to mainline, it would need 'since 2.5' (or
whatever version) documentation.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH 7/8] migration: new migration test mode

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1498 bytes --]

On 10/07/2015 07:56 AM, Dr. David Alan Gilbert wrote:
> * Denis V. Lunev (den@openvz.org) wrote:
>> From: Igor Redko <redkoi@virtuozzo.com>
>>
>> In this patch the ability to start a migration with test-only
>> capability was added. It allows to gather the guest VM’s memory
>> usage statistics avoiding time and memory overheads and real
>> data transmission.  New MIGRATION_STATUS_TEST_COMPLETED was
>> added to distinguish between test migration and true migration
>> success states.
> 
> Why isn't this just a new transport? i.e. I could do this just by doing
> a migrate to test:   ?
> 
> It seems simpler and avoids some of the special casing?

Indeed. Even if the new state can only be triggered by using a new
option, the fact that the new state exists in introspection means
clients have to figure out what to do with it; while a new transport
would not require any new states ("running" to completion means that
nothing migrated, but the existing states can then be reused to see that
the test: transport has finished computing its numbers).


>> +++ b/qapi-schema.json
>> @@ -432,6 +432,8 @@
>>  #
>>  # @completed: migration is finished.
>>  #
>> +# @test-completed: migration time estimation finished.
>> +#

If this gets added, in spite of our ideas to use a 'test:' transport
instead of a new state, it would need a 'since 2.5' marker.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH 3/8] migration: add new capability test-only

[Denis V. Lunev]

On 10/07/2015 06:05 PM, Eric Blake wrote:
> On 10/07/2015 12:20 AM, Denis V. Lunev wrote:
>> From: Igor Redko <redkoi@virtuozzo.com>
>>
>> This patch declares a new migration capability that allows
>> to distinguish between true migration and the test.
>>
>> An alternative is using a custom URI but in this case the
>> following problems should be solved:
>> 1/ QEMUFile abstraction hides the real transport type
>> 2/ We must be ensured that VM will not be stopped due to
>> “successful migration”
>>
>> Possible solutions:
>> 1/ Using global variable analogous to
>> s->enabled_capabilities[MIGRATION_CAPABILITY_TEST_ONLY] that
>> will be initialized according to the migration type at the
>> beginning of the migration process
>> 2/ Additional hook, presence of implementation of which allows
>> to decide whether the migration is true
>>
> Do we want to name the capability 'x-test-only', since this series is
> experimental, to make it obvious that production code should not rely on it?
>
actually the answer is 'no'. This feature is designated for production.
May be the name is just wrong or description is bad :) Yep, I know this
for sure...

OK. The idea of this feature is to estimate efforts required to
perform migration and downtime during that migration.
This estimate is necessary to decide whether we are able
to automatically move the VM from one node (loaded) to
different node with less load.

Should we rename 'test' to 'estimate' or something like that
to make the situation clearer?

On the other hand this 'estimate' state would be very useful
for testing that is why I have proposed this code to John.
At least we were able to catch recently fixed hang on migration
cancel (in pipeline already).

I am completely open to any idea.


>> +++ b/qapi-schema.json
>> @@ -540,11 +540,14 @@
>>   # @auto-converge: If enabled, QEMU will automatically throttle down the guest
>>   #          to speed up convergence of RAM migration. (since 1.6)
>>   #
>> +# @test-only: If enabled, QEMU instead of migrating will estimate migration
>> +#          time basing on given downtime and current dirty page rate
> If we do decide to add this to mainline, it would need 'since 2.5' (or
> whatever version) documentation.
>
Sure!

Re: [Qemu-devel] [PATCH 6/8] migration: implementation of hook_ram_sync

[Denis V. Lunev]

On 10/07/2015 12:44 PM, Paolo Bonzini wrote:
>
> On 07/10/2015 08:20, Denis V. Lunev wrote:
>> +        if (migrate_is_test()) {
>> +            /* since no data is transfered during estimation all
>> +               all measurements below will be incorrect.
>> +               as well no need for delays. */
>> +            continue;
>> +        }
> By applying delays, you can also test migration using bandwidth
> limitations and try to estimate whether it will converge or not.
> Perhaps if you use writev_buffer to implement the test QEMUFile you do
> not need this anymore.
>
> Paolo
you see, your proposal would be better for emulation approach.
In this case the 'test' will take much more time and may be
it will be better for testing purpose.

Here we are solving a bit different problem. We are trying to
collect reasonable data to make a prediction for migration
time and downtime for cluster or cloud management where
VMs are hosted on different physical hosts and are migrated
from one host to another using different policies.

In this case we need to make an estimate to check migration
feasibility and it would be nice if we will be able to achieve
this goal in the minimal period of time with a minimal overhead :)))
OK, this sounds a bit trivial :)

On the other hand this code could be used for testing that is
why I have spoken about this set.

Den

Re: [Qemu-devel] [PATCH 6/8] migration: implementation of hook_ram_sync

[Denis V. Lunev]

On 10/07/2015 12:44 PM, Paolo Bonzini wrote:
>
> On 07/10/2015 08:20, Denis V. Lunev wrote:
>> All calls of this hook will be from ram_save_pending().
>>
>> At the first call of this hook we need to save the initial
>> size of VM memory and put the migration thread to sleep for
>> decent period (downtime for example). During this period
>> guest would dirty memory.
>>
>> The second and the last call.
>> We make our estimation of dirty bytes rate assuming that time
>> between two synchronizations of dirty bitmap differs from downtime
>> negligibly.
>>
>> An alternative to this approach is receiving information about
>> size of data “transmitted” through the transport.
> This would use before_ram_iterate/after_ram_iterate, right?
>
>> However, this
>> way creates large time and memory overheads:
>> 1/Transmitted guest’s memory pages are copied to QEMUFile’s buffer
>>    (~8 sec per 4GB VM)
> Note that they are not if you implement writev_buffer.

yep, but we will have to setup iovec entry for each page
but pls see below

>> 2/Dirty memory pages are processed one by one (~60msec per 4GB VM)
> That however improves the accuracy, doesn't it?
>
> Paolo
from the point of estimate we need we need amount of dirtied
page per second as a count as a result thus I do not think
that this will make a difference.

Though the approach proposed by David in the letter below
is much better from the point of overhead and the result
was presented in the original description  as (2) aka ~60 msecs
per 4 GB VM was obtained that way. Sorry that this was not
clearly exposed in the description.

Den

Re: [Qemu-devel] [PATCH 7/8] migration: new migration test mode

[Denis V. Lunev]

On 10/07/2015 06:08 PM, Eric Blake wrote:
> On 10/07/2015 07:56 AM, Dr. David Alan Gilbert wrote:
>> * Denis V. Lunev (den@openvz.org) wrote:
>>> From: Igor Redko <redkoi@virtuozzo.com>
>>>
>>> In this patch the ability to start a migration with test-only
>>> capability was added. It allows to gather the guest VM’s memory
>>> usage statistics avoiding time and memory overheads and real
>>> data transmission.  New MIGRATION_STATUS_TEST_COMPLETED was
>>> added to distinguish between test migration and true migration
>>> success states.
>> Why isn't this just a new transport? i.e. I could do this just by doing
>> a migrate to test:   ?
>>
>> It seems simpler and avoids some of the special casing?
> Indeed. Even if the new state can only be triggered by using a new
> option, the fact that the new state exists in introspection means
> clients have to figure out what to do with it; while a new transport
> would not require any new states ("running" to completion means that
> nothing migrated, but the existing states can then be reused to see that
> the test: transport has finished computing its numbers).
>

we need to expose calculated numbers to the caller somehow.
This could be done in MIGRATION_STATUS_COMPLETED state
even for ordinary migration. Will it be OK?

Den

>>> +++ b/qapi-schema.json
>>> @@ -432,6 +432,8 @@
>>>   #
>>>   # @completed: migration is finished.
>>>   #
>>> +# @test-completed: migration time estimation finished.
>>> +#
> If this gets added, in spite of our ideas to use a 'test:' transport
> instead of a new state, it would need a 'since 2.5' marker.
>

Re: [Qemu-devel] [PATCH 7/8] migration: new migration test mode

[Denis V. Lunev]

On 10/07/2015 04:56 PM, Dr. David Alan Gilbert wrote:
> * Denis V. Lunev (den@openvz.org) wrote:
>> From: Igor Redko <redkoi@virtuozzo.com>
>>
>> In this patch the ability to start a migration with test-only
>> capability was added. It allows to gather the guest VM’s memory
>> usage statistics avoiding time and memory overheads and real
>> data transmission.  New MIGRATION_STATUS_TEST_COMPLETED was
>> added to distinguish between test migration and true migration
>> success states.
> Why isn't this just a new transport? i.e. I could do this just by doing
> a migrate to test:   ?
>
> It seems simpler and avoids some of the special casing?
>
> Dave
we would like to avoid VM pause in the migration_thread when
the process is finished. Thus we should have a capability
for this in the rest of the code.

Though we can setup the capability here or in the suitable
place and check that capability in the migration thread
using transport here as a distinction.

Will it be OK for you?

Den

>> Signed-off-by: Igor Redko <redkoi@virtuozzo.com>
>> Reviewed-by: Anna Melekhova <annam@virtuozzo.com>
>> Signed-off-by: Denis V. Lunev <den@openvz.org>
>> ---
>>   migration/migration.c | 12 ++++++++++--
>>   qapi-schema.json      |  4 +++-
>>   2 files changed, 13 insertions(+), 3 deletions(-)
>>
>> diff --git a/migration/migration.c b/migration/migration.c
>> index 3182e15..3470d39 100644
>> --- a/migration/migration.c
>> +++ b/migration/migration.c
>> @@ -790,7 +790,9 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
>>   
>>       s = migrate_init(&params);
>>   
>> -    if (strstart(uri, "tcp:", &p)) {
>> +    if (migrate_is_test()) {
>> +        test_start_migration(s, p, &local_err);
>> +    } else if (strstart(uri, "tcp:", &p)) {
>>           tcp_start_outgoing_migration(s, p, &local_err);
>>   #ifdef CONFIG_RDMA
>>       } else if (strstart(uri, "rdma:", &p)) {
>> @@ -1054,8 +1056,14 @@ static void *migration_thread(void *opaque)
>>           }
>>   
>>           if (qemu_file_get_error(s->file)) {
>> -            migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
>> +            /*FIXME replace magic number with smth legit*/
>> +            if (migrate_is_test() && qemu_file_get_error(s->file) == -42) {
>> +                migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
>> +                              MIGRATION_STATUS_TEST_COMPLETED);
>> +            } else {
>> +                migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
>>                                 MIGRATION_STATUS_FAILED);
>> +            }
>>               break;
>>           }
>>   
>> diff --git a/qapi-schema.json b/qapi-schema.json
>> index 38bf199..e022f9c 100644
>> --- a/qapi-schema.json
>> +++ b/qapi-schema.json
>> @@ -432,6 +432,8 @@
>>   #
>>   # @completed: migration is finished.
>>   #
>> +# @test-completed: migration time estimation finished.
>> +#
>>   # @failed: some error occurred during migration process.
>>   #
>>   # Since: 2.3
>> @@ -439,7 +441,7 @@
>>   ##
>>   { 'enum': 'MigrationStatus',
>>     'data': [ 'none', 'setup', 'cancelling', 'cancelled',
>> -            'active', 'completed', 'failed' ] }
>> +            'active', 'completed', 'test-completed', 'failed' ] }
>>   
>>   ##
>>   # @MigrationInfo
>> -- 
>> 2.1.4
>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [PATCH 7/8] migration: new migration test mode

[Dr. David Alan Gilbert]

* Denis V. Lunev (den@openvz.org) wrote:
> On 10/07/2015 06:08 PM, Eric Blake wrote:
> >On 10/07/2015 07:56 AM, Dr. David Alan Gilbert wrote:
> >>* Denis V. Lunev (den@openvz.org) wrote:
> >>>From: Igor Redko <redkoi@virtuozzo.com>
> >>>
> >>>In this patch the ability to start a migration with test-only
> >>>capability was added. It allows to gather the guest VM’s memory
> >>>usage statistics avoiding time and memory overheads and real
> >>>data transmission.  New MIGRATION_STATUS_TEST_COMPLETED was
> >>>added to distinguish between test migration and true migration
> >>>success states.
> >>Why isn't this just a new transport? i.e. I could do this just by doing
> >>a migrate to test:   ?
> >>
> >>It seems simpler and avoids some of the special casing?
> >Indeed. Even if the new state can only be triggered by using a new
> >option, the fact that the new state exists in introspection means
> >clients have to figure out what to do with it; while a new transport
> >would not require any new states ("running" to completion means that
> >nothing migrated, but the existing states can then be reused to see that
> >the test: transport has finished computing its numbers).
> >
> 
> we need to expose calculated numbers to the caller somehow.
> This could be done in MIGRATION_STATUS_COMPLETED state
> even for ordinary migration. Will it be OK?

I think so; if you look at the way the information is displayed
in info migrate, a lot of the data is only displayed 'sometimes',
e.g. :
        if (blk_mig_active()) {
            info->has_disk = true;
            info->disk = g_malloc0(sizeof(*info->disk));
            info->disk->transferred = blk_mig_bytes_transferred();
            info->disk->remaining = blk_mig_bytes_remaining();
            info->disk->total = blk_mig_bytes_total();
        }

It seems fair to do the same type of thing if you have an extra
block of values.

Dave

> 
> Den
> 
> >>>+++ b/qapi-schema.json
> >>>@@ -432,6 +432,8 @@
> >>>  #
> >>>  # @completed: migration is finished.
> >>>  #
> >>>+# @test-completed: migration time estimation finished.
> >>>+#
> >If this gets added, in spite of our ideas to use a 'test:' transport
> >instead of a new state, it would need a 'since 2.5' marker.
> >
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [PATCH 7/8] migration: new migration test mode

[Dr. David Alan Gilbert]

* Denis V. Lunev (den@openvz.org) wrote:
> On 10/07/2015 04:56 PM, Dr. David Alan Gilbert wrote:
> >* Denis V. Lunev (den@openvz.org) wrote:
> >>From: Igor Redko <redkoi@virtuozzo.com>
> >>
> >>In this patch the ability to start a migration with test-only
> >>capability was added. It allows to gather the guest VM’s memory
> >>usage statistics avoiding time and memory overheads and real
> >>data transmission.  New MIGRATION_STATUS_TEST_COMPLETED was
> >>added to distinguish between test migration and true migration
> >>success states.
> >Why isn't this just a new transport? i.e. I could do this just by doing
> >a migrate to test:   ?
> >
> >It seems simpler and avoids some of the special casing?
> >
> >Dave
> we would like to avoid VM pause in the migration_thread when
> the process is finished. Thus we should have a capability
> for this in the rest of the code.

Ah OK.

> Though we can setup the capability here or in the suitable
> place and check that capability in the migration thread
> using transport here as a distinction.
> 
> Will it be OK for you?

Yes, that works for me I think.

Dave

> 
> Den
> 
> >>Signed-off-by: Igor Redko <redkoi@virtuozzo.com>
> >>Reviewed-by: Anna Melekhova <annam@virtuozzo.com>
> >>Signed-off-by: Denis V. Lunev <den@openvz.org>
> >>---
> >>  migration/migration.c | 12 ++++++++++--
> >>  qapi-schema.json      |  4 +++-
> >>  2 files changed, 13 insertions(+), 3 deletions(-)
> >>
> >>diff --git a/migration/migration.c b/migration/migration.c
> >>index 3182e15..3470d39 100644
> >>--- a/migration/migration.c
> >>+++ b/migration/migration.c
> >>@@ -790,7 +790,9 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
> >>      s = migrate_init(&params);
> >>-    if (strstart(uri, "tcp:", &p)) {
> >>+    if (migrate_is_test()) {
> >>+        test_start_migration(s, p, &local_err);
> >>+    } else if (strstart(uri, "tcp:", &p)) {
> >>          tcp_start_outgoing_migration(s, p, &local_err);
> >>  #ifdef CONFIG_RDMA
> >>      } else if (strstart(uri, "rdma:", &p)) {
> >>@@ -1054,8 +1056,14 @@ static void *migration_thread(void *opaque)
> >>          }
> >>          if (qemu_file_get_error(s->file)) {
> >>-            migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
> >>+            /*FIXME replace magic number with smth legit*/
> >>+            if (migrate_is_test() && qemu_file_get_error(s->file) == -42) {
> >>+                migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
> >>+                              MIGRATION_STATUS_TEST_COMPLETED);
> >>+            } else {
> >>+                migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
> >>                                MIGRATION_STATUS_FAILED);
> >>+            }
> >>              break;
> >>          }
> >>diff --git a/qapi-schema.json b/qapi-schema.json
> >>index 38bf199..e022f9c 100644
> >>--- a/qapi-schema.json
> >>+++ b/qapi-schema.json
> >>@@ -432,6 +432,8 @@
> >>  #
> >>  # @completed: migration is finished.
> >>  #
> >>+# @test-completed: migration time estimation finished.
> >>+#
> >>  # @failed: some error occurred during migration process.
> >>  #
> >>  # Since: 2.3
> >>@@ -439,7 +441,7 @@
> >>  ##
> >>  { 'enum': 'MigrationStatus',
> >>    'data': [ 'none', 'setup', 'cancelling', 'cancelled',
> >>-            'active', 'completed', 'failed' ] }
> >>+            'active', 'completed', 'test-completed', 'failed' ] }
> >>  ##
> >>  # @MigrationInfo
> >>-- 
> >>2.1.4
> >>
> >>
> >--
> >Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [PATCH 3/8] migration: add new capability test-only

[Dr. David Alan Gilbert]

* Denis V. Lunev (den@openvz.org) wrote:
> On 10/07/2015 06:05 PM, Eric Blake wrote:
> >On 10/07/2015 12:20 AM, Denis V. Lunev wrote:
> >>From: Igor Redko <redkoi@virtuozzo.com>
> >>
> >>This patch declares a new migration capability that allows
> >>to distinguish between true migration and the test.
> >>
> >>An alternative is using a custom URI but in this case the
> >>following problems should be solved:
> >>1/ QEMUFile abstraction hides the real transport type
> >>2/ We must be ensured that VM will not be stopped due to
> >>“successful migration”
> >>
> >>Possible solutions:
> >>1/ Using global variable analogous to
> >>s->enabled_capabilities[MIGRATION_CAPABILITY_TEST_ONLY] that
> >>will be initialized according to the migration type at the
> >>beginning of the migration process
> >>2/ Additional hook, presence of implementation of which allows
> >>to decide whether the migration is true
> >>
> >Do we want to name the capability 'x-test-only', since this series is
> >experimental, to make it obvious that production code should not rely on it?
> >
> actually the answer is 'no'. This feature is designated for production.
> May be the name is just wrong or description is bad :) Yep, I know this
> for sure...
> 
> OK. The idea of this feature is to estimate efforts required to
> perform migration and downtime during that migration.
> This estimate is necessary to decide whether we are able
> to automatically move the VM from one node (loaded) to
> different node with less load.

Yes, this is the use we had also been considering.

> Should we rename 'test' to 'estimate' or something like that
> to make the situation clearer?
> 
> On the other hand this 'estimate' state would be very useful
> for testing that is why I have proposed this code to John.
> At least we were able to catch recently fixed hang on migration
> cancel (in pipeline already).
> 
> I am completely open to any idea.

I can see how 'test' might confuse with other test things; but
I'm not too fussy about the name.  I was thinking 'null' if it
was a transport name.

Dave

> 
> 
> >>+++ b/qapi-schema.json
> >>@@ -540,11 +540,14 @@
> >>  # @auto-converge: If enabled, QEMU will automatically throttle down the guest
> >>  #          to speed up convergence of RAM migration. (since 1.6)
> >>  #
> >>+# @test-only: If enabled, QEMU instead of migrating will estimate migration
> >>+#          time basing on given downtime and current dirty page rate
> >If we do decide to add this to mainline, it would need 'since 2.5' (or
> >whatever version) documentation.
> >
> Sure!
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK