[Qemu-devel] [PATCH 0/2] Add non-arbitrary treshold for migration

[Qemu-devel] [PATCH 0/2] Add non-arbitrary treshold for migration

[Glauber Costa]

Hi.
This is the patch I sent last week (if you missed the introductory
message, it is probably in your spam box), but with fixes
and comments addressed. All user visible quantities are in
seconds now.

[Qemu-devel] [PATCH 1/2] add non-arbitrary migration stop condition

[Glauber Costa]

Currently, we're entering migration's stage 3 when
a treshold of 10 pages remain to be transferred in the system.

This has hurt some users. However, any proposed threshold is
arbitrary by nature, and would only shift the annoyance.

The proposal of this patch is to define a max_downtime variable,
which represents the maximum downtime a migration user is willing
to suffer. Then, based on the bandwidth of last iteration, we
calculate how much data we can transfer in such a window of time.

Whenever we reach that value (or lower), we know is safe to enter
stage3.

This has largely improved the situation for me.
On localhost migrations, where one would expect things to go as
quickly as me running away from the duty of writting software for
windows, a kernel compile was enough to get the migration stuck.

It takes 20 ~ 30 iterations now.

Signed-off-by: Glauber Costa <glommer@redhat.com>
---
 migration.c |   11 +++++++++++
 migration.h |    2 ++
 vl.c        |   19 +++++++++++++++++--
 3 files changed, 30 insertions(+), 2 deletions(-)

diff --git a/migration.c b/migration.c
index 401383c..754c2fa 100644
--- a/migration.c
+++ b/migration.c
@@ -107,6 +107,17 @@ void do_migrate_set_speed(Monitor *mon, const char *value)
     
 }
 
+/* amount of nanoseconds we are willing to wait for migration to be down.
+ * the choice of nanoseconds is because it is the maximum resolution that
+ * get_clock() can achieve. It is an internal measure. All user-visible
+ * units must be in seconds */
+static uint64_t max_downtime = 30000000;
+
+uint64_t migrate_max_downtime(void)
+{
+    return max_downtime;
+}
+
 void do_info_migrate(Monitor *mon)
 {
     MigrationState *s = current_migration;
diff --git a/migration.h b/migration.h
index 696618d..0596f24 100644
--- a/migration.h
+++ b/migration.h
@@ -55,6 +55,8 @@ void do_migrate_cancel(Monitor *mon);
 
 void do_migrate_set_speed(Monitor *mon, const char *value);
 
+uint64_t migrate_max_downtime(void);
+
 void do_info_migrate(Monitor *mon);
 
 int exec_start_incoming_migration(const char *host_port);
diff --git a/vl.c b/vl.c
index 2c1f0e0..1a227bb 100644
--- a/vl.c
+++ b/vl.c
@@ -3197,7 +3197,6 @@ static int ram_save_block(QEMUFile *f)
     return found;
 }
 
-static ram_addr_t ram_save_threshold = 10;
 static uint64_t bytes_transferred = 0;
 
 static ram_addr_t ram_save_remaining(void)
@@ -3231,6 +3230,9 @@ uint64_t ram_bytes_total(void)
 static int ram_save_live(QEMUFile *f, int stage, void *opaque)
 {
     ram_addr_t addr;
+    uint64_t bytes_transferred_last;
+    double bwidth = 0;
+    uint64_t expected_time = 0;
 
     if (cpu_physical_sync_dirty_bitmap(0, last_ram_offset) != 0) {
         qemu_file_set_error(f);
@@ -3250,6 +3252,9 @@ static int ram_save_live(QEMUFile *f, int stage, void *opaque)
         qemu_put_be64(f, last_ram_offset | RAM_SAVE_FLAG_MEM_SIZE);
     }
 
+    bytes_transferred_last = bytes_transferred;
+    bwidth = get_clock();
+
     while (!qemu_file_rate_limit(f)) {
         int ret;
 
@@ -3259,6 +3264,14 @@ static int ram_save_live(QEMUFile *f, int stage, void *opaque)
             break;
     }
 
+    bwidth = get_clock() - bwidth;
+    bwidth = (bytes_transferred - bytes_transferred_last) / bwidth;
+
+    /* if we haven't transferred anything this round, force expected_time to a
+     * a very high value, but without crashing */
+    if (bwidth == 0)
+        bwidth = 0.000001;
+
     /* try transferring iterative blocks of memory */
 
     if (stage == 3) {
@@ -3272,7 +3285,9 @@ static int ram_save_live(QEMUFile *f, int stage, void *opaque)
 
     qemu_put_be64(f, RAM_SAVE_FLAG_EOS);
 
-    return (stage == 2) && (ram_save_remaining() < ram_save_threshold);
+    expected_time = ram_save_remaining() * TARGET_PAGE_SIZE / bwidth;
+
+    return (stage == 2) && (expected_time <= migrate_max_downtime());
 }
 
 static int ram_load_dead(QEMUFile *f, void *opaque)
-- 
1.5.6.6

[Qemu-devel] [PATCH 2/2] set migration max downtime

[Glauber Costa]

provide a monitor command to allow one to set the maximum
downtime he is willing to suffer during migration, in seconds.
"ms", "us", "ns" and "s" are accepted as modifiers.

This parameter will be used by ram_save_live() code to determine
a safe moment to enter stage 3

Signed-off-by: Glauber Costa <glommer@redhat.com>
---
 migration.c |   19 +++++++++++++++++++
 migration.h |    2 ++
 monitor.c   |    2 ++
 3 files changed, 23 insertions(+), 0 deletions(-)

diff --git a/migration.c b/migration.c
index 754c2fa..5c62eee 100644
--- a/migration.c
+++ b/migration.c
@@ -118,6 +118,25 @@ uint64_t migrate_max_downtime(void)
     return max_downtime;
 }
 
+void do_migrate_set_downtime(Monitor *mon, const char *value)
+{
+    char *ptr;
+    double d;
+
+    d = strtod(value, &ptr);
+    if (!strcmp(ptr,"ms")) {
+        d *= 1000000;
+    } else if (!strcmp(ptr,"us")) {
+        d *= 1000;
+    } else if (!strcmp(ptr,"ns")) {
+    } else {
+        /* all else considered to be seconds */
+        d *= 1000000000;
+    }
+
+    max_downtime = (uint64_t)d;
+}
+
 void do_info_migrate(Monitor *mon)
 {
     MigrationState *s = current_migration;
diff --git a/migration.h b/migration.h
index 0596f24..37c7f8e 100644
--- a/migration.h
+++ b/migration.h
@@ -57,6 +57,8 @@ void do_migrate_set_speed(Monitor *mon, const char *value);
 
 uint64_t migrate_max_downtime(void);
 
+void do_migrate_set_downtime(Monitor *mon, const char *value);
+
 void do_info_migrate(Monitor *mon);
 
 int exec_start_incoming_migration(const char *host_port);
diff --git a/monitor.c b/monitor.c
index 0f38c71..55b09c3 100644
--- a/monitor.c
+++ b/monitor.c
@@ -1744,6 +1744,8 @@ static const mon_cmd_t mon_cmds[] = {
       "", "cancel the current VM migration" },
     { "migrate_set_speed", "s", do_migrate_set_speed,
       "value", "set maximum speed (in bytes) for migrations" },
+    { "migrate_set_downtime", "s", do_migrate_set_downtime,
+      "value", "set maximum tolerated downtime (in seconds) for migrations" },
 #if defined(TARGET_I386)
     { "drive_add", "ss", drive_hot_add, "pci_addr=[[<domain>:]<bus>:]<slot>\n"
                                          "[file=file][,if=type][,bus=n]\n"
-- 
1.5.6.6