[Qemu-devel] [PULL 0/6] Migration pull request

[Qemu-devel] [PULL 0/6] Migration pull request

[Juan Quintela]

Hi

This pull requset:
- small optimization by Li
- fixes global state with savevm (reported by John Snow)
- get migration traces for migration state changes even if migration
  events are not enabled
- Change state after migration has finished (reported by Christian)
- add documentation for events

Please, Apply.

PD.  Yes, it had to pass:
   make check
   virt-test
   savevm by hand


The following changes since commit 661725da09f47eb92d356fac10a4cf3b7ad1f61d:

  Merge remote-tracking branch 'remotes/cohuck/tags/s390x-20150714' into staging (2015-07-14 18:50:17 +0100)

are available in the git repository at:

  git://github.com/juanquintela/qemu.git tags/migration/20150715

for you to fetch changes up to 4915a5ad93e6c8ac77e2e5a007be178d50c7b019:

  migration: We also want to store the global state for savevm (2015-07-15 10:51:05 +0200)

----------------------------------------------------------------
migration/next for 20150715

----------------------------------------------------------------
Juan Quintela (5):
      migration: Only change state after migration has finished
      migration: Trace event and migration event are different things
      migration: Write documetation for events capabilites
      migration: Register global state section before loadvm
      migration: We also want to store the global state for savevm

Liang Li (1):
      migration: reduce the count of strlen call

 include/migration/migration.h |  1 +
 migration/migration.c         | 52 ++++++++++++++++++++++++-------------------
 migration/ram.c               | 10 ++++-----
 migration/savevm.c            |  6 +++++
 qmp-commands.hx               |  1 +
 vl.c                          |  2 +-
 6 files changed, 43 insertions(+), 29 deletions(-)

Re: [Qemu-devel] [PULL 0/6] Migration pull request

[Amit Shah]

On (Wed) 15 Jul 2015 [11:04:34], Juan Quintela wrote:
> Hi
> 
> This pull requset:
> - small optimization by Li
> - fixes global state with savevm (reported by John Snow)
> - get migration traces for migration state changes even if migration
>   events are not enabled
> - Change state after migration has finished (reported by Christian)
> - add documentation for events
> 
> Please, Apply.

Peter, can you fix the typos in my name in patches 5 and 6, or do you
want a new pull req?

Thanks,

		Amit

Re: [Qemu-devel] [PULL 0/6] Migration pull request

[Juan Quintela]

Self-Nack

Misspelled Amit name, resending with name fixed.  Sorry.

Juan Quintela <quintela@redhat.com> wrote:
> Hi
>
> This pull requset:
> - small optimization by Li
> - fixes global state with savevm (reported by John Snow)
> - get migration traces for migration state changes even if migration
>   events are not enabled
> - Change state after migration has finished (reported by Christian)
> - add documentation for events
>
> Please, Apply.
>
> PD.  Yes, it had to pass:
>    make check
>    virt-test
>    savevm by hand
>
>
> The following changes since commit 661725da09f47eb92d356fac10a4cf3b7ad1f61d:
>
>   Merge remote-tracking branch 'remotes/cohuck/tags/s390x-20150714' into staging (2015-07-14 18:50:17 +0100)
>
> are available in the git repository at:
>
>   git://github.com/juanquintela/qemu.git tags/migration/20150715
>
> for you to fetch changes up to 4915a5ad93e6c8ac77e2e5a007be178d50c7b019:
>
>   migration: We also want to store the global state for savevm (2015-07-15 10:51:05 +0200)
>
> ----------------------------------------------------------------
> migration/next for 20150715
>
> ----------------------------------------------------------------
> Juan Quintela (5):
>       migration: Only change state after migration has finished
>       migration: Trace event and migration event are different things
>       migration: Write documetation for events capabilites
>       migration: Register global state section before loadvm
>       migration: We also want to store the global state for savevm
>
> Liang Li (1):
>       migration: reduce the count of strlen call
>
>  include/migration/migration.h |  1 +
>  migration/migration.c         | 52 ++++++++++++++++++++++++-------------------
>  migration/ram.c               | 10 ++++-----
>  migration/savevm.c            |  6 +++++
>  qmp-commands.hx               |  1 +
>  vl.c                          |  2 +-
>  6 files changed, 43 insertions(+), 29 deletions(-)

Re: [Qemu-devel] [PULL 0/6] Migration pull request

[Peter Maydell]

On 15 July 2015 at 11:16, Amit Shah <amit.shah@redhat.com> wrote:
> On (Wed) 15 Jul 2015 [11:04:34], Juan Quintela wrote:
>> Hi
>>
>> This pull requset:
>> - small optimization by Li
>> - fixes global state with savevm (reported by John Snow)
>> - get migration traces for migration state changes even if migration
>>   events are not enabled
>> - Change state after migration has finished (reported by Christian)
>> - add documentation for events
>>
>> Please, Apply.
>
> Peter, can you fix the typos in my name in patches 5 and 6, or do you
> want a new pull req?

I've applied Juan's fixed pull. For the record, I always need a
new pullreq -- it's not possible to edit the commits in a pull,
I can only edit if I'm applying patches individually to master.

thanks
-- PMM

Re: [Qemu-devel] [PULL 0/6] Migration pull request

[Juan Quintela]

Peter Maydell <peter.maydell@linaro.org> wrote:
> On 15 July 2015 at 11:16, Amit Shah <amit.shah@redhat.com> wrote:
>> On (Wed) 15 Jul 2015 [11:04:34], Juan Quintela wrote:
>>> Hi
>>>
>>> This pull requset:
>>> - small optimization by Li
>>> - fixes global state with savevm (reported by John Snow)
>>> - get migration traces for migration state changes even if migration
>>>   events are not enabled
>>> - Change state after migration has finished (reported by Christian)
>>> - add documentation for events
>>>
>>> Please, Apply.
>>
>> Peter, can you fix the typos in my name in patches 5 and 6, or do you
>> want a new pull req?
>
> I've applied Juan's fixed pull. For the record, I always need a
> new pullreq -- it's not possible to edit the commits in a pull,
> I can only edit if I'm applying patches individually to master.

I used a different algorithm.

What is easier for Peter?

O:-)

Thanks, Juan.

[Qemu-devel] [PULL 0/6] Migration pull request

[Juan Quintela]

Hi

This get the auto-converge changes for migration, please pull.


The following changes since commit b2312c680084ea18cd55fa7093397cad2224ec14:

  Merge remote-tracking branch 'remotes/amit-migration/tags/for-juan-201509' into staging (2015-09-29 12:41:19 +0100)

are available in the git repository at:

  git://github.com/juanquintela/qemu.git tags/migration/20150930

for you to fetch changes up to dc3256272cf70b2152279b013a8abb16e0f6fe96:

  migration: Disambiguate MAX_THROTTLE (2015-09-30 09:42:04 +0200)

----------------------------------------------------------------
migration/next for 20150930

----------------------------------------------------------------
Jason J. Herne (5):
      cpu: Provide vcpu throttling interface
      migration: Parameters for auto-converge cpu throttling
      migration: Dynamic cpu throttling for auto-converge
      qmp/hmp: Add throttle ratio to query-migrate and info migrate
      migration: Disambiguate MAX_THROTTLE

Juan Quintela (1):
      migration: yet more possible state transitions

 cpus.c                | 78 ++++++++++++++++++++++++++++++++++++++++++++
 hmp.c                 | 21 ++++++++++++
 include/qom/cpu.h     | 42 ++++++++++++++++++++++++
 migration/migration.c | 57 +++++++++++++++++++++++++++++++--
 migration/ram.c       | 89 +++++++++++++++++----------------------------------
 qapi-schema.json      | 40 ++++++++++++++++++++---
 vl.c                  |  1 +
 7 files changed, 263 insertions(+), 65 deletions(-)

[Qemu-devel] [PULL 1/6] migration: yet more possible state transitions

[Juan Quintela]

On destination, we move from INMIGRATE to FINISH_MIGRATE.  Add that to
the list of allowed states.

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
---
 vl.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/vl.c b/vl.c
index e211f6a..8d1846c 100644
--- a/vl.c
+++ b/vl.c
@@ -580,6 +580,7 @@ static const RunStateTransition runstate_transitions_def[] = {
     { RUN_STATE_INMIGRATE, RUN_STATE_SUSPENDED },
     { RUN_STATE_INMIGRATE, RUN_STATE_WATCHDOG },
     { RUN_STATE_INMIGRATE, RUN_STATE_GUEST_PANICKED },
+    { RUN_STATE_INMIGRATE, RUN_STATE_FINISH_MIGRATE },

     { RUN_STATE_INTERNAL_ERROR, RUN_STATE_PAUSED },
     { RUN_STATE_INTERNAL_ERROR, RUN_STATE_FINISH_MIGRATE },
-- 
2.4.3

[Qemu-devel] [PULL 2/6] cpu: Provide vcpu throttling interface

[Juan Quintela]

From: "Jason J. Herne" <jjherne@linux.vnet.ibm.com>

Provide a method to throttle guest cpu execution. CPUState is augmented with
timeout controls and throttle start/stop functions. To throttle the guest cpu
the caller simply has to call the throttle set function and provide a percentage
of throttle time.

Signed-off-by: Jason J. Herne <jjherne@linux.vnet.ibm.com>
Reviewed-by: Matthew Rosato <mjrosato@linux.vnet.ibm.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
---
 cpus.c            | 78 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 include/qom/cpu.h | 42 ++++++++++++++++++++++++++++++
 2 files changed, 120 insertions(+)

diff --git a/cpus.c b/cpus.c
index dddd056..d44c0ed 100644
--- a/cpus.c
+++ b/cpus.c
@@ -69,6 +69,14 @@ static CPUState *next_cpu;
 int64_t max_delay;
 int64_t max_advance;

+/* vcpu throttling controls */
+static QEMUTimer *throttle_timer;
+static unsigned int throttle_percentage;
+
+#define CPU_THROTTLE_PCT_MIN 1
+#define CPU_THROTTLE_PCT_MAX 99
+#define CPU_THROTTLE_TIMESLICE_NS 10000000
+
 bool cpu_is_stopped(CPUState *cpu)
 {
     return cpu->stopped || !runstate_is_running();
@@ -505,10 +513,80 @@ static const VMStateDescription vmstate_timers = {
     }
 };

+static void cpu_throttle_thread(void *opaque)
+{
+    CPUState *cpu = opaque;
+    double pct;
+    double throttle_ratio;
+    long sleeptime_ns;
+
+    if (!cpu_throttle_get_percentage()) {
+        return;
+    }
+
+    pct = (double)cpu_throttle_get_percentage()/100;
+    throttle_ratio = pct / (1 - pct);
+    sleeptime_ns = (long)(throttle_ratio * CPU_THROTTLE_TIMESLICE_NS);
+
+    qemu_mutex_unlock_iothread();
+    atomic_set(&cpu->throttle_thread_scheduled, 0);
+    g_usleep(sleeptime_ns / 1000); /* Convert ns to us for usleep call */
+    qemu_mutex_lock_iothread();
+}
+
+static void cpu_throttle_timer_tick(void *opaque)
+{
+    CPUState *cpu;
+    double pct;
+
+    /* Stop the timer if needed */
+    if (!cpu_throttle_get_percentage()) {
+        return;
+    }
+    CPU_FOREACH(cpu) {
+        if (!atomic_xchg(&cpu->throttle_thread_scheduled, 1)) {
+            async_run_on_cpu(cpu, cpu_throttle_thread, cpu);
+        }
+    }
+
+    pct = (double)cpu_throttle_get_percentage()/100;
+    timer_mod(throttle_timer, qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL_RT) +
+                                   CPU_THROTTLE_TIMESLICE_NS / (1-pct));
+}
+
+void cpu_throttle_set(int new_throttle_pct)
+{
+    /* Ensure throttle percentage is within valid range */
+    new_throttle_pct = MIN(new_throttle_pct, CPU_THROTTLE_PCT_MAX);
+    new_throttle_pct = MAX(new_throttle_pct, CPU_THROTTLE_PCT_MIN);
+
+    atomic_set(&throttle_percentage, new_throttle_pct);
+
+    timer_mod(throttle_timer, qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL_RT) +
+                                       CPU_THROTTLE_TIMESLICE_NS);
+}
+
+void cpu_throttle_stop(void)
+{
+    atomic_set(&throttle_percentage, 0);
+}
+
+bool cpu_throttle_active(void)
+{
+    return (cpu_throttle_get_percentage() != 0);
+}
+
+int cpu_throttle_get_percentage(void)
+{
+    return atomic_read(&throttle_percentage);
+}
+
 void cpu_ticks_init(void)
 {
     seqlock_init(&timers_state.vm_clock_seqlock, NULL);
     vmstate_register(NULL, 0, &vmstate_timers, &timers_state);
+    throttle_timer = timer_new_ns(QEMU_CLOCK_VIRTUAL_RT,
+                                           cpu_throttle_timer_tick, NULL);
 }

 void configure_icount(QemuOpts *opts, Error **errp)
diff --git a/include/qom/cpu.h b/include/qom/cpu.h
index 302673d..9405554 100644
--- a/include/qom/cpu.h
+++ b/include/qom/cpu.h
@@ -321,6 +321,11 @@ struct CPUState {
     uint32_t can_do_io;
     int32_t exception_index; /* used by m68k TCG */

+    /* Used to keep track of an outstanding cpu throttle thread for migration
+     * autoconverge
+     */
+    bool throttle_thread_scheduled;
+
     /* Note that this is accessed at the start of every TB via a negative
        offset from AREG0.  Leave this field at the end so as to make the
        (absolute value) offset as small as possible.  This reduces code
@@ -565,6 +570,43 @@ CPUState *qemu_get_cpu(int index);
  */
 bool cpu_exists(int64_t id);

+/**
+ * cpu_throttle_set:
+ * @new_throttle_pct: Percent of sleep time. Valid range is 1 to 99.
+ *
+ * Throttles all vcpus by forcing them to sleep for the given percentage of
+ * time. A throttle_percentage of 25 corresponds to a 75% duty cycle roughly.
+ * (example: 10ms sleep for every 30ms awake).
+ *
+ * cpu_throttle_set can be called as needed to adjust new_throttle_pct.
+ * Once the throttling starts, it will remain in effect until cpu_throttle_stop
+ * is called.
+ */
+void cpu_throttle_set(int new_throttle_pct);
+
+/**
+ * cpu_throttle_stop:
+ *
+ * Stops the vcpu throttling started by cpu_throttle_set.
+ */
+void cpu_throttle_stop(void);
+
+/**
+ * cpu_throttle_active:
+ *
+ * Returns: %true if the vcpus are currently being throttled, %false otherwise.
+ */
+bool cpu_throttle_active(void);
+
+/**
+ * cpu_throttle_get_percentage:
+ *
+ * Returns the vcpu throttle percentage. See cpu_throttle_set for details.
+ *
+ * Returns: The throttle percentage in range 1 to 99.
+ */
+int cpu_throttle_get_percentage(void);
+
 #ifndef CONFIG_USER_ONLY

 typedef void (*CPUInterruptHandler)(CPUState *, int);
-- 
2.4.3

[Qemu-devel] [PULL 3/6] migration: Parameters for auto-converge cpu throttling

[Juan Quintela]

From: "Jason J. Herne" <jjherne@linux.vnet.ibm.com>

Add migration parameters to allow the user to adjust the parameters
that control cpu throttling when auto-converge is in effect. The added
parameters are as follows:

x-cpu-throttle-initial : Initial percantage of time guest cpus are throttled
when migration auto-converge is activated.

x-cpu-throttle-increment: throttle percantage increase each time
auto-converge detects that migration is not making progress.

Signed-off-by: Jason J. Herne <jjherne@linux.vnet.ibm.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
---
 hmp.c                 | 16 ++++++++++++++++
 migration/migration.c | 46 +++++++++++++++++++++++++++++++++++++++++++++-
 qapi-schema.json      | 33 ++++++++++++++++++++++++++++++---
 3 files changed, 91 insertions(+), 4 deletions(-)

diff --git a/hmp.c b/hmp.c
index 3f807b7..48ce372 100644
--- a/hmp.c
+++ b/hmp.c
@@ -272,6 +272,12 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)
         monitor_printf(mon, " %s: %" PRId64,
             MigrationParameter_lookup[MIGRATION_PARAMETER_DECOMPRESS_THREADS],
             params->decompress_threads);
+        monitor_printf(mon, " %s: %" PRId64,
+            MigrationParameter_lookup[MIGRATION_PARAMETER_X_CPU_THROTTLE_INITIAL],
+            params->x_cpu_throttle_initial);
+        monitor_printf(mon, " %s: %" PRId64,
+            MigrationParameter_lookup[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT],
+            params->x_cpu_throttle_increment);
         monitor_printf(mon, "\n");
     }

@@ -1221,6 +1227,8 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
     bool has_compress_level = false;
     bool has_compress_threads = false;
     bool has_decompress_threads = false;
+    bool has_x_cpu_throttle_initial = false;
+    bool has_x_cpu_throttle_increment = false;
     int i;

     for (i = 0; i < MIGRATION_PARAMETER_MAX; i++) {
@@ -1235,10 +1243,18 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
             case MIGRATION_PARAMETER_DECOMPRESS_THREADS:
                 has_decompress_threads = true;
                 break;
+            case MIGRATION_PARAMETER_X_CPU_THROTTLE_INITIAL:
+                has_x_cpu_throttle_initial = true;
+                break;
+            case MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT:
+                has_x_cpu_throttle_increment = true;
+                break;
             }
             qmp_migrate_set_parameters(has_compress_level, value,
                                        has_compress_threads, value,
                                        has_decompress_threads, value,
+                                       has_x_cpu_throttle_initial, value,
+                                       has_x_cpu_throttle_increment, value,
                                        &err);
             break;
         }
diff --git a/migration/migration.c b/migration/migration.c
index e48dd13..8a1af3b 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -44,6 +44,9 @@
 #define DEFAULT_MIGRATE_DECOMPRESS_THREAD_COUNT 2
 /*0: means nocompress, 1: best speed, ... 9: best compress ratio */
 #define DEFAULT_MIGRATE_COMPRESS_LEVEL 1
+/* Define default autoconverge cpu throttle migration parameters */
+#define DEFAULT_MIGRATE_X_CPU_THROTTLE_INITIAL 20
+#define DEFAULT_MIGRATE_X_CPU_THROTTLE_INCREMENT 10

 /* Migration XBZRLE default cache size */
 #define DEFAULT_MIGRATE_CACHE_SIZE (64 * 1024 * 1024)
@@ -71,6 +74,10 @@ MigrationState *migrate_get_current(void)
                 DEFAULT_MIGRATE_COMPRESS_THREAD_COUNT,
         .parameters[MIGRATION_PARAMETER_DECOMPRESS_THREADS] =
                 DEFAULT_MIGRATE_DECOMPRESS_THREAD_COUNT,
+        .parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INITIAL] =
+                DEFAULT_MIGRATE_X_CPU_THROTTLE_INITIAL,
+        .parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT] =
+                DEFAULT_MIGRATE_X_CPU_THROTTLE_INCREMENT,
     };

     return &current_migration;
@@ -372,6 +379,10 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)
             s->parameters[MIGRATION_PARAMETER_COMPRESS_THREADS];
     params->decompress_threads =
             s->parameters[MIGRATION_PARAMETER_DECOMPRESS_THREADS];
+    params->x_cpu_throttle_initial =
+            s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INITIAL];
+    params->x_cpu_throttle_increment =
+            s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT];

     return params;
 }
@@ -494,7 +505,11 @@ void qmp_migrate_set_parameters(bool has_compress_level,
                                 bool has_compress_threads,
                                 int64_t compress_threads,
                                 bool has_decompress_threads,
-                                int64_t decompress_threads, Error **errp)
+                                int64_t decompress_threads,
+                                bool has_x_cpu_throttle_initial,
+                                int64_t x_cpu_throttle_initial,
+                                bool has_x_cpu_throttle_increment,
+                                int64_t x_cpu_throttle_increment, Error **errp)
 {
     MigrationState *s = migrate_get_current();

@@ -517,6 +532,18 @@ void qmp_migrate_set_parameters(bool has_compress_level,
                    "is invalid, it should be in the range of 1 to 255");
         return;
     }
+    if (has_x_cpu_throttle_initial &&
+            (x_cpu_throttle_initial < 1 || x_cpu_throttle_initial > 99)) {
+        error_setg(errp, QERR_INVALID_PARAMETER_VALUE,
+                   "x_cpu_throttle_initial",
+                   "an integer in the range of 1 to 99");
+    }
+    if (has_x_cpu_throttle_increment &&
+            (x_cpu_throttle_increment < 1 || x_cpu_throttle_increment > 99)) {
+        error_setg(errp, QERR_INVALID_PARAMETER_VALUE,
+                   "x_cpu_throttle_increment",
+                   "an integer in the range of 1 to 99");
+    }

     if (has_compress_level) {
         s->parameters[MIGRATION_PARAMETER_COMPRESS_LEVEL] = compress_level;
@@ -528,6 +555,15 @@ void qmp_migrate_set_parameters(bool has_compress_level,
         s->parameters[MIGRATION_PARAMETER_DECOMPRESS_THREADS] =
                                                     decompress_threads;
     }
+    if (has_x_cpu_throttle_initial) {
+        s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INITIAL] =
+                                                    x_cpu_throttle_initial;
+    }
+
+    if (has_x_cpu_throttle_increment) {
+        s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT] =
+                                                    x_cpu_throttle_increment;
+    }
 }

 /* shared migration helpers */
@@ -643,6 +679,10 @@ static MigrationState *migrate_init(const MigrationParams *params)
             s->parameters[MIGRATION_PARAMETER_COMPRESS_THREADS];
     int decompress_thread_count =
             s->parameters[MIGRATION_PARAMETER_DECOMPRESS_THREADS];
+    int x_cpu_throttle_initial =
+            s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INITIAL];
+    int x_cpu_throttle_increment =
+            s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT];

     memcpy(enabled_capabilities, s->enabled_capabilities,
            sizeof(enabled_capabilities));
@@ -658,6 +698,10 @@ static MigrationState *migrate_init(const MigrationParams *params)
                compress_thread_count;
     s->parameters[MIGRATION_PARAMETER_DECOMPRESS_THREADS] =
                decompress_thread_count;
+    s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INITIAL] =
+                x_cpu_throttle_initial;
+    s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT] =
+                x_cpu_throttle_increment;
     s->bandwidth_limit = bandwidth_limit;
     migrate_set_state(s, MIGRATION_STATUS_NONE, MIGRATION_STATUS_SETUP);

diff --git a/qapi-schema.json b/qapi-schema.json
index 582a817..646c0fa 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -596,10 +596,18 @@
 #          compression, so set the decompress-threads to the number about 1/4
 #          of compress-threads is adequate.
 #
+# @x-cpu-throttle-initial: Initial percentage of time guest cpus are throttled
+#                          when migration auto-converge is activated. The
+#                          default value is 20. (Since 2.5)
+#
+# @x-cpu-throttle-increment: throttle percentage increase each time
+#                            auto-converge detects that migration is not making
+#                            progress. The default value is 10. (Since 2.5)
 # Since: 2.4
 ##
 { 'enum': 'MigrationParameter',
-  'data': ['compress-level', 'compress-threads', 'decompress-threads'] }
+  'data': ['compress-level', 'compress-threads', 'decompress-threads',
+           'x-cpu-throttle-initial', 'x-cpu-throttle-increment'] }

 #
 # @migrate-set-parameters
@@ -612,12 +620,21 @@
 #
 # @decompress-threads: decompression thread count
 #
+# @x-cpu-throttle-initial: Initial percentage of time guest cpus are throttled
+#                          when migration auto-converge is activated. The
+#                          default value is 20. (Since 2.5)
+#
+# @x-cpu-throttle-increment: throttle percentage increase each time
+#                            auto-converge detects that migration is not making
+#                            progress. The default value is 10. (Since 2.5)
 # Since: 2.4
 ##
 { 'command': 'migrate-set-parameters',
   'data': { '*compress-level': 'int',
             '*compress-threads': 'int',
-            '*decompress-threads': 'int'} }
+            '*decompress-threads': 'int',
+            '*x-cpu-throttle-initial': 'int',
+            '*x-cpu-throttle-increment': 'int'} }

 #
 # @MigrationParameters
@@ -628,12 +645,22 @@
 #
 # @decompress-threads: decompression thread count
 #
+# @x-cpu-throttle-initial: Initial percentage of time guest cpus are throttled
+#                          when migration auto-converge is activated. The
+#                          default value is 20. (Since 2.5)
+#
+# @x-cpu-throttle-increment: throttle percentage increase each time
+#                            auto-converge detects that migration is not making
+#                            progress. The default value is 10. (Since 2.5)
+#
 # Since: 2.4
 ##
 { 'struct': 'MigrationParameters',
   'data': { 'compress-level': 'int',
             'compress-threads': 'int',
-            'decompress-threads': 'int'} }
+            'decompress-threads': 'int',
+            'x-cpu-throttle-initial': 'int',
+            'x-cpu-throttle-increment': 'int'} }
 ##
 # @query-migrate-parameters
 #
-- 
2.4.3

[Qemu-devel] [PULL 4/6] migration: Dynamic cpu throttling for auto-converge

[Juan Quintela]

From: "Jason J. Herne" <jjherne@linux.vnet.ibm.com>

Remove traditional auto-converge static 30ms throttling code and replace it
with a dynamic throttling algorithm.

Additionally, be more aggressive when deciding when to start throttling.
Previously we waited until four unproductive memory passes. Now we begin
throttling after only two unproductive memory passes. Four seemed quite
arbitrary and only waiting for two passes allows us to complete the migration
faster.

Signed-off-by: Jason J. Herne <jjherne@linux.vnet.ibm.com>
Reviewed-by: Matthew Rosato <mjrosato@linux.vnet.ibm.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
---
 migration/migration.c |  4 +++
 migration/ram.c       | 89 +++++++++++++++++----------------------------------
 2 files changed, 34 insertions(+), 59 deletions(-)

diff --git a/migration/migration.c b/migration/migration.c
index 8a1af3b..e829231 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -29,6 +29,7 @@
 #include "trace.h"
 #include "qapi/util.h"
 #include "qapi-event.h"
+#include "qom/cpu.h"

 #define MAX_THROTTLE  (32 << 20)      /* Migration speed throttling */

@@ -1070,6 +1071,9 @@ static void *migration_thread(void *opaque)
         }
     }

+    /* If we enabled cpu throttling for auto-converge, turn it off. */
+    cpu_throttle_stop();
+
     qemu_mutex_lock_iothread();
     if (s->state == MIGRATION_STATUS_COMPLETED) {
         int64_t end_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
diff --git a/migration/ram.c b/migration/ram.c
index 5187637..2d1d0b9 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -47,9 +47,7 @@
     do { } while (0)
 #endif

-static bool mig_throttle_on;
 static int dirty_rate_high_cnt;
-static void check_guest_throttling(void);

 static uint64_t bitmap_sync_count;

@@ -407,6 +405,29 @@ static size_t save_page_header(QEMUFile *f, RAMBlock *block, ram_addr_t offset)
     return size;
 }

+/* Reduce amount of guest cpu execution to hopefully slow down memory writes.
+ * If guest dirty memory rate is reduced below the rate at which we can
+ * transfer pages to the destination then we should be able to complete
+ * migration. Some workloads dirty memory way too fast and will not effectively
+ * converge, even with auto-converge.
+ */
+static void mig_throttle_guest_down(void)
+{
+    MigrationState *s = migrate_get_current();
+    uint64_t pct_initial =
+            s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INITIAL];
+    uint64_t pct_icrement =
+            s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT];
+
+    /* We have not started throttling yet. Let's start it. */
+    if (!cpu_throttle_active()) {
+        cpu_throttle_set(pct_initial);
+    } else {
+        /* Throttling already on, just increase the rate */
+        cpu_throttle_set(cpu_throttle_get_percentage() + pct_icrement);
+    }
+}
+
 /* Update the xbzrle cache to reflect a page that's been sent as all 0.
  * The important thing is that a stale (not-yet-0'd) page be replaced
  * by the new data.
@@ -599,21 +620,21 @@ static void migration_bitmap_sync(void)
             /* The following detection logic can be refined later. For now:
                Check to see if the dirtied bytes is 50% more than the approx.
                amount of bytes that just got transferred since the last time we
-               were in this routine. If that happens >N times (for now N==4)
-               we turn on the throttle down logic */
+               were in this routine. If that happens twice, start or increase
+               throttling */
             bytes_xfer_now = ram_bytes_transferred();
+
             if (s->dirty_pages_rate &&
                (num_dirty_pages_period * TARGET_PAGE_SIZE >
                    (bytes_xfer_now - bytes_xfer_prev)/2) &&
-               (dirty_rate_high_cnt++ > 4)) {
+               (dirty_rate_high_cnt++ >= 2)) {
                     trace_migration_throttle();
-                    mig_throttle_on = true;
                     dirty_rate_high_cnt = 0;
+                    mig_throttle_guest_down();
              }
              bytes_xfer_prev = bytes_xfer_now;
-        } else {
-             mig_throttle_on = false;
         }
+
         if (migrate_use_xbzrle()) {
             if (iterations_prev != acct_info.iterations) {
                 acct_info.xbzrle_cache_miss_rate =
@@ -1146,7 +1167,6 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
     RAMBlock *block;
     int64_t ram_bitmap_pages; /* Size of bitmap in pages, including gaps */

-    mig_throttle_on = false;
     dirty_rate_high_cnt = 0;
     bitmap_sync_count = 0;
     migration_bitmap_sync_init();
@@ -1251,7 +1271,7 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
         }
         pages_sent += pages;
         acct_info.iterations++;
-        check_guest_throttling();
+
         /* we want to check in the 1st loop, just in case it was the 1st time
            and we had to sync the dirty bitmap.
            qemu_get_clock_ns() is a bit expensive, so we only check each some
@@ -1664,52 +1684,3 @@ void ram_mig_init(void)
     qemu_mutex_init(&XBZRLE.lock);
     register_savevm_live(NULL, "ram", 0, 4, &savevm_ram_handlers, NULL);
 }
-/* Stub function that's gets run on the vcpu when its brought out of the
-   VM to run inside qemu via async_run_on_cpu()*/
-
-static void mig_sleep_cpu(void *opq)
-{
-    qemu_mutex_unlock_iothread();
-    g_usleep(30*1000);
-    qemu_mutex_lock_iothread();
-}
-
-/* To reduce the dirty rate explicitly disallow the VCPUs from spending
-   much time in the VM. The migration thread will try to catchup.
-   Workload will experience a performance drop.
-*/
-static void mig_throttle_guest_down(void)
-{
-    CPUState *cpu;
-
-    qemu_mutex_lock_iothread();
-    CPU_FOREACH(cpu) {
-        async_run_on_cpu(cpu, mig_sleep_cpu, NULL);
-    }
-    qemu_mutex_unlock_iothread();
-}
-
-static void check_guest_throttling(void)
-{
-    static int64_t t0;
-    int64_t        t1;
-
-    if (!mig_throttle_on) {
-        return;
-    }
-
-    if (!t0)  {
-        t0 = qemu_clock_get_ns(QEMU_CLOCK_REALTIME);
-        return;
-    }
-
-    t1 = qemu_clock_get_ns(QEMU_CLOCK_REALTIME);
-
-    /* If it has been more than 40 ms since the last time the guest
-     * was throttled then do it again.
-     */
-    if (40 < (t1-t0)/1000000) {
-        mig_throttle_guest_down();
-        t0 = t1;
-    }
-}
-- 
2.4.3

[Qemu-devel] [PULL 5/6] qmp/hmp: Add throttle ratio to query-migrate and info migrate

[Juan Quintela]

From: "Jason J. Herne" <jjherne@linux.vnet.ibm.com>

Report throttle percentage in info migrate and query-migrate responses when
cpu throttling is active.

Signed-off-by: Jason J. Herne <jjherne@linux.vnet.ibm.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
---
 hmp.c                 | 5 +++++
 migration/migration.c | 5 +++++
 qapi-schema.json      | 7 ++++++-
 3 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/hmp.c b/hmp.c
index 48ce372..5048eee 100644
--- a/hmp.c
+++ b/hmp.c
@@ -232,6 +232,11 @@ void hmp_info_migrate(Monitor *mon, const QDict *qdict)
                        info->xbzrle_cache->overflow);
     }

+    if (info->has_x_cpu_throttle_percentage) {
+        monitor_printf(mon, "cpu throttle percentage: %" PRIu64 "\n",
+                       info->x_cpu_throttle_percentage);
+    }
+
     qapi_free_MigrationInfo(info);
     qapi_free_MigrationCapabilityStatusList(caps);
 }
diff --git a/migration/migration.c b/migration/migration.c
index e829231..c7472ed 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -447,6 +447,11 @@ MigrationInfo *qmp_query_migrate(Error **errp)
             info->disk->total = blk_mig_bytes_total();
         }

+        if (cpu_throttle_active()) {
+            info->has_x_cpu_throttle_percentage = true;
+            info->x_cpu_throttle_percentage = cpu_throttle_get_percentage();
+        }
+
         get_xbzrle_cache_stats(info);
         break;
     case MIGRATION_STATUS_COMPLETED:
diff --git a/qapi-schema.json b/qapi-schema.json
index 646c0fa..8b0520c 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -480,6 +480,10 @@
 #        may be expensive, but do not actually occur during the iterative
 #        migration rounds themselves. (since 1.6)
 #
+# @x-cpu-throttle-percentage: #optional percentage of time guest cpus are being
+#       throttled during auto-converge. This is only present when auto-converge
+#       has started throttling guest cpus. (Since 2.5)
+#
 # Since: 0.14.0
 ##
 { 'struct': 'MigrationInfo',
@@ -489,7 +493,8 @@
            '*total-time': 'int',
            '*expected-downtime': 'int',
            '*downtime': 'int',
-           '*setup-time': 'int'} }
+           '*setup-time': 'int',
+           '*x-cpu-throttle-percentage': 'int'} }

 ##
 # @query-migrate
-- 
2.4.3

[Qemu-devel] [PULL 6/6] migration: Disambiguate MAX_THROTTLE

[Juan Quintela]

From: "Jason J. Herne" <jjherne@linux.vnet.ibm.com>

Migration has a define for MAX_THROTTLE. Update comment to clarify that this is
used for throttling transfer speed. Hopefully this will prevent it from being
confused with a guest cpu throttling entity.

Signed-off-by: Jason J. Herne <jjherne@linux.vnet.ibm.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
---
 migration/migration.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/migration/migration.c b/migration/migration.c
index c7472ed..b7de9b7 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -31,7 +31,7 @@
 #include "qapi-event.h"
 #include "qom/cpu.h"

-#define MAX_THROTTLE  (32 << 20)      /* Migration speed throttling */
+#define MAX_THROTTLE  (32 << 20)      /* Migration transfer speed throttling */

 /* Amount of time to allocate to each "chunk" of bandwidth-throttled
  * data. */
-- 
2.4.3

Re: [Qemu-devel] [PULL 0/6] Migration pull request

[Peter Maydell]

On 30 September 2015 at 09:56, Juan Quintela <quintela@redhat.com> wrote:
> Hi
>
> This get the auto-converge changes for migration, please pull.
>
>
> The following changes since commit b2312c680084ea18cd55fa7093397cad2224ec14:
>
>   Merge remote-tracking branch 'remotes/amit-migration/tags/for-juan-201509' into staging (2015-09-29 12:41:19 +0100)
>
> are available in the git repository at:
>
>   git://github.com/juanquintela/qemu.git tags/migration/20150930
>
> for you to fetch changes up to dc3256272cf70b2152279b013a8abb16e0f6fe96:
>
>   migration: Disambiguate MAX_THROTTLE (2015-09-30 09:42:04 +0200)
>
> ----------------------------------------------------------------
> migration/next for 20150930
>
> ----------------------------------------------------------------

Applied, thanks.

-- PMM

[Qemu-devel] [PULL 0/6] Migration pull request

[Juan Quintela]

Hi

- fix for mutex initialzation (dave)
- different page sizes fixes (dave)
- better text (dave)
- Migration 32bits fixes (myself)

Please, apply.

The following changes since commit ed6c64489ef11d9ac5fb4b4c89d455a4f1ae8083:

  target-arm: Update PC before calling gen_helper_check_breakpoints() (2015-11-12 16:16:02 +0000)

are available in the git repository at:

  git://github.com/juanquintela/qemu.git tags/migration/20151112

for you to fetch changes up to 389775d1f67b2c8f44f9473b1e5363735972e389:

  migration_init: Fix lock initialisation/make it explicit (2015-11-12 17:55:27 +0100)

----------------------------------------------------------------
migration/next for 20151112

----------------------------------------------------------------
Dr. David Alan Gilbert (4):
      Finish non-postcopiable iterative devices before package
      Postcopy: Fix TP!=HP zero case
      migrate-start-postcopy: Improve text
      migration_init: Fix lock initialisation/make it explicit

Juan Quintela (2):
      migration: print ram_addr_t as RAM_ADDR_FMT not %zx
      migration: Make 32bit linux compile with RDMA

 hmp-commands.hx         |  4 +++-
 include/sysemu/sysemu.h |  2 +-
 migration/migration.c   | 63 ++++++++++++++++++++++++-------------------------
 migration/ram.c         |  7 +++---
 migration/rdma.c        |  2 +-
 migration/savevm.c      | 10 ++++++--
 qapi-schema.json        |  4 +++-
 7 files changed, 51 insertions(+), 41 deletions(-)

Re: [Qemu-devel] [PULL 0/6] Migration pull request

[Peter Maydell]

On 12 November 2015 at 17:20, Juan Quintela <quintela@redhat.com> wrote:
> Hi
>
> - fix for mutex initialzation (dave)
> - different page sizes fixes (dave)
> - better text (dave)
> - Migration 32bits fixes (myself)
>
> Please, apply.
>
> The following changes since commit ed6c64489ef11d9ac5fb4b4c89d455a4f1ae8083:
>
>   target-arm: Update PC before calling gen_helper_check_breakpoints() (2015-11-12 16:16:02 +0000)
>
> are available in the git repository at:
>
>   git://github.com/juanquintela/qemu.git tags/migration/20151112
>
> for you to fetch changes up to 389775d1f67b2c8f44f9473b1e5363735972e389:
>
>   migration_init: Fix lock initialisation/make it explicit (2015-11-12 17:55:27 +0100)
>
> ----------------------------------------------------------------
> migration/next for 20151112
>
> ----------------------------------------------------------------

Applied, thanks.

-- PMM