[Qemu-devel] [PATCH 00/34] migration thread and queue

[Qemu-devel] [PATCH 00/34] migration thread and queue

[Juan Quintela]

Hi

This is my queue for migration-thread and patches associated.  This
integrates review comments & code for Paolo.  This is the subset from
both approachs that we agreed with. rest of patches need more review
and are not here.

Migrating and idle guest with upstwream:

(qemu) info migrate
capabilities: xbzrle: off
Migration status: completed
total time: 34251 milliseconds
downtime: 492 milliseconds
transferred ram: 762458 kbytes
remaining ram: 0 kbytes
total ram: 14688768 kbytes
duplicate: 3492606 pages
normal: 189762 pages
normal bytes: 759048 kbytes

with this series of patches.

(qemu) info migrate
capabilities: xbzrle: off
Migration status: completed
total time: 30712 milliseconds
downtime: 29 milliseconds
transferred ram: 738857 kbytes
remaining ram: 0 kbytes
total ram: 14688768 kbytes
duplicate: 3503423 pages
normal: 176671 pages
normal bytes: 706684 kbytes

Notice the big difference in downtime.  And that is also seen inside
the guest a program that just do an idle loop seeing how "long" it
takes to wait for 10ms.

with upstream:

[root@d1 ~]# ./timer
delay of 452 ms
delay of 114 ms
delay of 136 ms
delay of 135 ms
delay of 136 ms
delay of 131 ms
delay of 134 ms

with this series of patches, wait never takes 100ms, nothing is printed.

Please review.

Thanks, Juan.


The following changes since commit 914606d26e654d4c01bd5186f4d05e3fd445e219:

  Merge remote-tracking branch 'stefanha/trivial-patches' into staging (2012-12-18 15:41:43 -0600)

are available in the git repository at:


  git://repo.or.cz/qemu/quintela.git thread.next

for you to fetch changes up to 065bebbacf54dcad48aad42112417dbb44451499:

  migration: merge QEMUFileBuffered into MigrationState (2012-12-19 13:21:29 +0100)

----------------------------------------------------------------
Juan Quintela (25):
      migration: include qemu-file.h
      migration-fd: remove duplicate include
      buffered_file: Move from using a timer to use a thread
      migration: make qemu_fopen_ops_buffered() return void
      migration: move migration thread init code to migrate_fd_put_ready
      migration: make writes blocking
      migration: remove unfreeze logic
      migration: just lock migrate_fd_put_ready
      buffered_file: Unfold the trick to restart generating migration data
      buffered_file: don't flush on put buffer
      buffered_file: unfold buffered_append in buffered_put_buffer
      savevm: New save live migration method: pending
      migration: move buffered_file.c code into migration.c
      migration: add XFER_LIMIT_RATIO
      migration: move migration_fd_put_ready()
      migration: Inline qemu_fopen_ops_buffered into migrate_fd_connect
      migration: move migration notifier
      ram: rename last_block to last_seen_block
      ram: Add last_sent_block
      memory: introduce memory_region_test_and_clear_dirty
      ram: Use memory_region_test_and_clear_dirty
      ram: optimize migration bitmap walking
      ram: account the amount of transferred ram better
      ram: refactor ram_save_block() return value
      migration: merge QEMUFileBuffered into MigrationState

Paolo Bonzini (7):
      migration: fix migration_bitmap leak
      buffered_file: do not send more than s->bytes_xfer bytes per tick
      migration: remove double call to migrate_fd_close
      exec: change ramlist from MRU order to a 1-item cache
      exec: change RAM list to a TAILQ
      exec: sort the memory from biggest to smallest
      migration: fix qemu_get_fd for BufferedFile

Umesh Deshpande (2):
      add a version number to ram_list
      protect the ramlist with a separate mutex

 Makefile.objs           |   2 +-
 arch_init.c             | 245 +++++++++++++++---------------
 block-migration.c       |  49 ++----
 buffered_file.c         | 269 ---------------------------------
 buffered_file.h         |  22 ---
 cpu-all.h               |  15 +-
 dump.c                  |   8 +-
 exec.c                  | 128 +++++++++++-----
 memory.c                |  16 ++
 memory.h                |  16 ++
 memory_mapping.c        |   4 +-
 migration-exec.c        |   3 +-
 migration-fd.c          |   4 +-
 migration-tcp.c         |   3 +-
 migration-unix.c        |   3 +-
 migration.c             | 390 ++++++++++++++++++++++++++++++++++++------------
 migration.h             |  13 +-
 qemu-file.h             |   5 -
 savevm.c                |  24 ++-
 sysemu.h                |   1 +
 target-i386/arch_dump.c |   2 +-
 vmstate.h               |   1 +
 22 files changed, 599 insertions(+), 624 deletions(-)
 delete mode 100644 buffered_file.c
 delete mode 100644 buffered_file.h

[Qemu-devel] [PATCH 01/34] migration: fix migration_bitmap leak

[Juan Quintela]

From: Paolo Bonzini <pbonzini@redhat.com>

Cc: qemu-stable@nongnu.org
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 arch_init.c | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/arch_init.c b/arch_init.c
index 83dcc53..0d7844c 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -535,9 +535,13 @@ static void sort_ram_list(void)

 static void migration_end(void)
 {
-    memory_global_dirty_log_stop();
+    if (migration_bitmap) {
+        memory_global_dirty_log_stop();
+        g_free(migration_bitmap);
+        migration_bitmap = NULL;
+    }

-    if (migrate_use_xbzrle()) {
+    if (XBZRLE.cache) {
         cache_fini(XBZRLE.cache);
         g_free(XBZRLE.cache);
         g_free(XBZRLE.encoded_buf);
@@ -689,13 +693,10 @@ static int ram_save_complete(QEMUFile *f, void *opaque)
         }
         bytes_transferred += bytes_sent;
     }
-    memory_global_dirty_log_stop();
+    migration_end();

     qemu_put_be64(f, RAM_SAVE_FLAG_EOS);

-    g_free(migration_bitmap);
-    migration_bitmap = NULL;
-
     return 0;
 }

-- 
1.7.11.7

[Qemu-devel] [PATCH 02/34] buffered_file: do not send more than s->bytes_xfer bytes per tick

[Juan Quintela]

From: Paolo Bonzini <pbonzini@redhat.com>

Sending more was possible if the buffer was large.

Cc: qemu-stable@nongnu.org
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 buffered_file.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/buffered_file.c b/buffered_file.c
index bd0f61d..9a8bd04 100644
--- a/buffered_file.c
+++ b/buffered_file.c
@@ -66,9 +66,9 @@ static ssize_t buffered_flush(QEMUFileBuffered *s)
     DPRINTF("flushing %zu byte(s) of data\n", s->buffer_size);

     while (s->bytes_xfer < s->xfer_limit && offset < s->buffer_size) {
-
+        size_t to_send = MIN(s->buffer_size - offset, s->xfer_limit - s->bytes_xfer);
         ret = migrate_fd_put_buffer(s->migration_state, s->buffer + offset,
-                                    s->buffer_size - offset);
+                                    to_send);
         if (ret == -EAGAIN) {
             DPRINTF("backend not ready, freezing\n");
             ret = 0;
-- 
1.7.11.7

[Qemu-devel] [PATCH 03/34] migration: remove double call to migrate_fd_close

[Juan Quintela]

From: Paolo Bonzini <pbonzini@redhat.com>

The call in buffered_close is enough, because buffered_close is called
already by migrate_fd_cleanup.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/migration.c b/migration.c
index 73ce170..bde4956 100644
--- a/migration.c
+++ b/migration.c
@@ -272,7 +272,7 @@ static int migrate_fd_cleanup(MigrationState *s)
         s->file = NULL;
     }

-    migrate_fd_close(s);
+    assert(s->fd == -1);
     return ret;
 }

-- 
1.7.11.7

[Qemu-devel] [PATCH 04/34] migration: include qemu-file.h

[Juan Quintela]

They don't use/know anything about buffered-file.

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration-exec.c | 2 +-
 migration-fd.c   | 2 +-
 migration-tcp.c  | 2 +-
 migration-unix.c | 2 +-
 4 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/migration-exec.c b/migration-exec.c
index 2b6fcb4..f449a22 100644
--- a/migration-exec.c
+++ b/migration-exec.c
@@ -19,7 +19,7 @@
 #include "qemu_socket.h"
 #include "migration.h"
 #include "qemu-char.h"
-#include "buffered_file.h"
+#include "qemu-file.h"
 #include "block.h"
 #include <sys/types.h>
 #include <sys/wait.h>
diff --git a/migration-fd.c b/migration-fd.c
index 5fe28e0..c501c4b 100644
--- a/migration-fd.c
+++ b/migration-fd.c
@@ -18,7 +18,7 @@
 #include "migration.h"
 #include "monitor.h"
 #include "qemu-char.h"
-#include "buffered_file.h"
+#include "qemu-file.h"
 #include "block.h"
 #include "qemu_socket.h"

diff --git a/migration-tcp.c b/migration-tcp.c
index 5e855fe..1683158 100644
--- a/migration-tcp.c
+++ b/migration-tcp.c
@@ -17,7 +17,7 @@
 #include "qemu_socket.h"
 #include "migration.h"
 #include "qemu-char.h"
-#include "buffered_file.h"
+#include "qemu-file.h"
 #include "block.h"

 //#define DEBUG_MIGRATION_TCP
diff --git a/migration-unix.c b/migration-unix.c
index dba72b4..da00f2f 100644
--- a/migration-unix.c
+++ b/migration-unix.c
@@ -17,7 +17,7 @@
 #include "qemu_socket.h"
 #include "migration.h"
 #include "qemu-char.h"
-#include "buffered_file.h"
+#include "qemu-file.h"
 #include "block.h"

 //#define DEBUG_MIGRATION_UNIX
-- 
1.7.11.7

[Qemu-devel] [PATCH 05/34] migration-fd: remove duplicate include

[Juan Quintela]

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration-fd.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/migration-fd.c b/migration-fd.c
index c501c4b..b8d16ad 100644
--- a/migration-fd.c
+++ b/migration-fd.c
@@ -20,7 +20,6 @@
 #include "qemu-char.h"
 #include "qemu-file.h"
 #include "block.h"
-#include "qemu_socket.h"

 //#define DEBUG_MIGRATION_FD

-- 
1.7.11.7

[Qemu-devel] [PATCH 06/34] exec: change ramlist from MRU order to a 1-item cache

[Juan Quintela]

From: Paolo Bonzini <pbonzini@redhat.com>

Most of the time, only 2 items will be active (from/to for a string operation,
or code/data).  But TCG guests likely won't have gigabytes of memory, so
this actually goes down to 1 item.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 arch_init.c |  1 +
 cpu-all.h   |  1 +
 exec.c      | 49 ++++++++++++++++++++++++++++---------------------
 3 files changed, 30 insertions(+), 21 deletions(-)

diff --git a/arch_init.c b/arch_init.c
index 0d7844c..b03b1d4 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -48,6 +48,7 @@
 #include "qemu/page_cache.h"
 #include "qmp-commands.h"
 #include "trace.h"
+#include "cpu-all.h"

 #ifdef DEBUG_ARCH_INIT
 #define DPRINTF(fmt, ...) \
diff --git a/cpu-all.h b/cpu-all.h
index d6b2b19..973b504 100644
--- a/cpu-all.h
+++ b/cpu-all.h
@@ -495,6 +495,7 @@ typedef struct RAMBlock {

 typedef struct RAMList {
     uint8_t *phys_dirty;
+    RAMBlock *mru_block;
     QLIST_HEAD(, RAMBlock) blocks;
 } RAMList;
 extern RAMList ram_list;
diff --git a/exec.c b/exec.c
index 4c1246a..a3dbe2f 100644
--- a/exec.c
+++ b/exec.c
@@ -42,6 +42,7 @@
 #include "xen-mapcache.h"
 #include "trace.h"
 #endif
+#include "cpu-all.h"

 #include "cputlb.h"
 #include "translate-all.h"
@@ -1042,6 +1043,7 @@ ram_addr_t qemu_ram_alloc_from_ptr(ram_addr_t size, void *host,
     new_block->length = size;

     QLIST_INSERT_HEAD(&ram_list.blocks, new_block, next);
+    ram_list.mru_block = NULL;

     ram_list.phys_dirty = g_realloc(ram_list.phys_dirty,
                                        last_ram_offset() >> TARGET_PAGE_BITS);
@@ -1070,6 +1072,7 @@ void qemu_ram_free_from_ptr(ram_addr_t addr)
     QLIST_FOREACH(block, &ram_list.blocks, next) {
         if (addr == block->offset) {
             QLIST_REMOVE(block, next);
+            ram_list.mru_block = NULL;
             g_free(block);
             return;
         }
@@ -1083,6 +1086,7 @@ void qemu_ram_free(ram_addr_t addr)
     QLIST_FOREACH(block, &ram_list.blocks, next) {
         if (addr == block->offset) {
             QLIST_REMOVE(block, next);
+            ram_list.mru_block = NULL;
             if (block->flags & RAM_PREALLOC_MASK) {
                 ;
             } else if (mem_path) {
@@ -1188,37 +1192,40 @@ void *qemu_get_ram_ptr(ram_addr_t addr)
 {
     RAMBlock *block;

+    block = ram_list.mru_block;
+    if (block && addr - block->offset < block->length) {
+        goto found;
+    }
     QLIST_FOREACH(block, &ram_list.blocks, next) {
         if (addr - block->offset < block->length) {
-            /* Move this entry to to start of the list.  */
-            if (block != QLIST_FIRST(&ram_list.blocks)) {
-                QLIST_REMOVE(block, next);
-                QLIST_INSERT_HEAD(&ram_list.blocks, block, next);
-            }
-            if (xen_enabled()) {
-                /* We need to check if the requested address is in the RAM
-                 * because we don't want to map the entire memory in QEMU.
-                 * In that case just map until the end of the page.
-                 */
-                if (block->offset == 0) {
-                    return xen_map_cache(addr, 0, 0);
-                } else if (block->host == NULL) {
-                    block->host =
-                        xen_map_cache(block->offset, block->length, 1);
-                }
-            }
-            return block->host + (addr - block->offset);
+            goto found;
         }
     }

     fprintf(stderr, "Bad ram offset %" PRIx64 "\n", (uint64_t)addr);
     abort();

-    return NULL;
+found:
+    ram_list.mru_block = block;
+    if (xen_enabled()) {
+        /* We need to check if the requested address is in the RAM
+         * because we don't want to map the entire memory in QEMU.
+         * In that case just map until the end of the page.
+         */
+        if (block->offset == 0) {
+            return xen_map_cache(addr, 0, 0);
+        } else if (block->host == NULL) {
+            block->host =
+                xen_map_cache(block->offset, block->length, 1);
+        }
+    }
+    return block->host + (addr - block->offset);
 }

-/* Return a host pointer to ram allocated with qemu_ram_alloc.
- * Same as qemu_get_ram_ptr but avoid reordering ramblocks.
+/* Return a host pointer to ram allocated with qemu_ram_alloc.  Same as
+ * qemu_get_ram_ptr but do not touch ram_list.mru_block.
+ *
+ * ??? Is this still necessary?
  */
 static void *qemu_safe_ram_ptr(ram_addr_t addr)
 {
-- 
1.7.11.7

[Qemu-devel] [PATCH 07/34] exec: change RAM list to a TAILQ

[Juan Quintela]

From: Paolo Bonzini <pbonzini@redhat.com>

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 arch_init.c             | 24 ++++++++++++------------
 cpu-all.h               |  4 ++--
 dump.c                  |  8 ++++----
 exec.c                  | 34 +++++++++++++++++-----------------
 memory_mapping.c        |  4 ++--
 target-i386/arch_dump.c |  2 +-
 6 files changed, 38 insertions(+), 38 deletions(-)

diff --git a/arch_init.c b/arch_init.c
index b03b1d4..3c1aa00 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -382,7 +382,7 @@ static void migration_bitmap_sync(void)
     trace_migration_bitmap_sync_start();
     memory_global_sync_dirty_bitmap(get_system_memory());

-    QLIST_FOREACH(block, &ram_list.blocks, next) {
+    QTAILQ_FOREACH(block, &ram_list.blocks, next) {
         for (addr = 0; addr < block->length; addr += TARGET_PAGE_SIZE) {
             if (memory_region_get_dirty(block->mr, addr, TARGET_PAGE_SIZE,
                                         DIRTY_MEMORY_MIGRATION)) {
@@ -424,7 +424,7 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
     ram_addr_t current_addr;

     if (!block)
-        block = QLIST_FIRST(&ram_list.blocks);
+        block = QTAILQ_FIRST(&ram_list.blocks);

     do {
         mr = block->mr;
@@ -465,9 +465,9 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
         offset += TARGET_PAGE_SIZE;
         if (offset >= block->length) {
             offset = 0;
-            block = QLIST_NEXT(block, next);
+            block = QTAILQ_NEXT(block, next);
             if (!block)
-                block = QLIST_FIRST(&ram_list.blocks);
+                block = QTAILQ_FIRST(&ram_list.blocks);
         }
     } while (block != last_block || offset != last_offset);

@@ -499,7 +499,7 @@ uint64_t ram_bytes_total(void)
     RAMBlock *block;
     uint64_t total = 0;

-    QLIST_FOREACH(block, &ram_list.blocks, next)
+    QTAILQ_FOREACH(block, &ram_list.blocks, next)
         total += block->length;

     return total;
@@ -518,18 +518,18 @@ static void sort_ram_list(void)
     RAMBlock *block, *nblock, **blocks;
     int n;
     n = 0;
-    QLIST_FOREACH(block, &ram_list.blocks, next) {
+    QTAILQ_FOREACH(block, &ram_list.blocks, next) {
         ++n;
     }
     blocks = g_malloc(n * sizeof *blocks);
     n = 0;
-    QLIST_FOREACH_SAFE(block, &ram_list.blocks, next, nblock) {
+    QTAILQ_FOREACH_SAFE(block, &ram_list.blocks, next, nblock) {
         blocks[n++] = block;
-        QLIST_REMOVE(block, next);
+        QTAILQ_REMOVE(&ram_list.blocks, block, next);
     }
     qsort(blocks, n, sizeof *blocks, block_compar);
     while (--n >= 0) {
-        QLIST_INSERT_HEAD(&ram_list.blocks, blocks[n], next);
+        QTAILQ_INSERT_HEAD(&ram_list.blocks, blocks[n], next);
     }
     g_free(blocks);
 }
@@ -597,7 +597,7 @@ static int ram_save_setup(QEMUFile *f, void *opaque)

     qemu_put_be64(f, ram_bytes_total() | RAM_SAVE_FLAG_MEM_SIZE);

-    QLIST_FOREACH(block, &ram_list.blocks, next) {
+    QTAILQ_FOREACH(block, &ram_list.blocks, next) {
         qemu_put_byte(f, strlen(block->idstr));
         qemu_put_buffer(f, (uint8_t *)block->idstr, strlen(block->idstr));
         qemu_put_be64(f, block->length);
@@ -763,7 +763,7 @@ static inline void *host_from_stream_offset(QEMUFile *f,
     qemu_get_buffer(f, (uint8_t *)id, len);
     id[len] = 0;

-    QLIST_FOREACH(block, &ram_list.blocks, next) {
+    QTAILQ_FOREACH(block, &ram_list.blocks, next) {
         if (!strncmp(id, block->idstr, sizeof(id)))
             return memory_region_get_ram_ptr(block->mr) + offset;
     }
@@ -807,7 +807,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
                     id[len] = 0;
                     length = qemu_get_be64(f);

-                    QLIST_FOREACH(block, &ram_list.blocks, next) {
+                    QTAILQ_FOREACH(block, &ram_list.blocks, next) {
                         if (!strncmp(id, block->idstr, sizeof(id))) {
                             if (block->length != length) {
                                 ret =  -EINVAL;
diff --git a/cpu-all.h b/cpu-all.h
index 973b504..cd61320 100644
--- a/cpu-all.h
+++ b/cpu-all.h
@@ -487,7 +487,7 @@ typedef struct RAMBlock {
     ram_addr_t length;
     uint32_t flags;
     char idstr[256];
-    QLIST_ENTRY(RAMBlock) next;
+    QTAILQ_ENTRY(RAMBlock) next;
 #if defined(__linux__) && !defined(TARGET_S390X)
     int fd;
 #endif
@@ -496,7 +496,7 @@ typedef struct RAMBlock {
 typedef struct RAMList {
     uint8_t *phys_dirty;
     RAMBlock *mru_block;
-    QLIST_HEAD(, RAMBlock) blocks;
+    QTAILQ_HEAD(, RAMBlock) blocks;
 } RAMList;
 extern RAMList ram_list;

diff --git a/dump.c b/dump.c
index 5640c2c..b088cb4 100644
--- a/dump.c
+++ b/dump.c
@@ -427,7 +427,7 @@ static hwaddr get_offset(hwaddr phys_addr,
         }
     }

-    QLIST_FOREACH(block, &ram_list.blocks, next) {
+    QTAILQ_FOREACH(block, &ram_list.blocks, next) {
         if (s->has_filter) {
             if (block->offset >= s->begin + s->length ||
                 block->offset + block->length <= s->begin) {
@@ -594,7 +594,7 @@ static int dump_completed(DumpState *s)
 static int get_next_block(DumpState *s, RAMBlock *block)
 {
     while (1) {
-        block = QLIST_NEXT(block, next);
+        block = QTAILQ_NEXT(block, next);
         if (!block) {
             /* no more block */
             return 1;
@@ -670,11 +670,11 @@ static ram_addr_t get_start_block(DumpState *s)
     RAMBlock *block;

     if (!s->has_filter) {
-        s->block = QLIST_FIRST(&ram_list.blocks);
+        s->block = QTAILQ_FIRST(&ram_list.blocks);
         return 0;
     }

-    QLIST_FOREACH(block, &ram_list.blocks, next) {
+    QTAILQ_FOREACH(block, &ram_list.blocks, next) {
         if (block->offset >= s->begin + s->length ||
             block->offset + block->length <= s->begin) {
             /* This block is out of the range */
diff --git a/exec.c b/exec.c
index a3dbe2f..13c894d 100644
--- a/exec.c
+++ b/exec.c
@@ -56,7 +56,7 @@
 int phys_ram_fd;
 static int in_migration;

-RAMList ram_list = { .blocks = QLIST_HEAD_INITIALIZER(ram_list.blocks) };
+RAMList ram_list = { .blocks = QTAILQ_HEAD_INITIALIZER(ram_list.blocks) };

 static MemoryRegion *system_memory;
 static MemoryRegion *system_io;
@@ -901,15 +901,15 @@ static ram_addr_t find_ram_offset(ram_addr_t size)
     RAMBlock *block, *next_block;
     ram_addr_t offset = RAM_ADDR_MAX, mingap = RAM_ADDR_MAX;

-    if (QLIST_EMPTY(&ram_list.blocks))
+    if (QTAILQ_EMPTY(&ram_list.blocks))
         return 0;

-    QLIST_FOREACH(block, &ram_list.blocks, next) {
+    QTAILQ_FOREACH(block, &ram_list.blocks, next) {
         ram_addr_t end, next = RAM_ADDR_MAX;

         end = block->offset + block->length;

-        QLIST_FOREACH(next_block, &ram_list.blocks, next) {
+        QTAILQ_FOREACH(next_block, &ram_list.blocks, next) {
             if (next_block->offset >= end) {
                 next = MIN(next, next_block->offset);
             }
@@ -934,7 +934,7 @@ ram_addr_t last_ram_offset(void)
     RAMBlock *block;
     ram_addr_t last = 0;

-    QLIST_FOREACH(block, &ram_list.blocks, next)
+    QTAILQ_FOREACH(block, &ram_list.blocks, next)
         last = MAX(last, block->offset + block->length);

     return last;
@@ -963,7 +963,7 @@ void qemu_ram_set_idstr(ram_addr_t addr, const char *name, DeviceState *dev)
     RAMBlock *new_block, *block;

     new_block = NULL;
-    QLIST_FOREACH(block, &ram_list.blocks, next) {
+    QTAILQ_FOREACH(block, &ram_list.blocks, next) {
         if (block->offset == addr) {
             new_block = block;
             break;
@@ -981,7 +981,7 @@ void qemu_ram_set_idstr(ram_addr_t addr, const char *name, DeviceState *dev)
     }
     pstrcat(new_block->idstr, sizeof(new_block->idstr), name);

-    QLIST_FOREACH(block, &ram_list.blocks, next) {
+    QTAILQ_FOREACH(block, &ram_list.blocks, next) {
         if (block != new_block && !strcmp(block->idstr, new_block->idstr)) {
             fprintf(stderr, "RAMBlock \"%s\" already registered, abort!\n",
                     new_block->idstr);
@@ -1042,7 +1042,7 @@ ram_addr_t qemu_ram_alloc_from_ptr(ram_addr_t size, void *host,
     }
     new_block->length = size;

-    QLIST_INSERT_HEAD(&ram_list.blocks, new_block, next);
+    QTAILQ_INSERT_HEAD(&ram_list.blocks, new_block, next);
     ram_list.mru_block = NULL;

     ram_list.phys_dirty = g_realloc(ram_list.phys_dirty,
@@ -1069,9 +1069,9 @@ void qemu_ram_free_from_ptr(ram_addr_t addr)
 {
     RAMBlock *block;

-    QLIST_FOREACH(block, &ram_list.blocks, next) {
+    QTAILQ_FOREACH(block, &ram_list.blocks, next) {
         if (addr == block->offset) {
-            QLIST_REMOVE(block, next);
+            QTAILQ_REMOVE(&ram_list.blocks, block, next);
             ram_list.mru_block = NULL;
             g_free(block);
             return;
@@ -1083,9 +1083,9 @@ void qemu_ram_free(ram_addr_t addr)
 {
     RAMBlock *block;

-    QLIST_FOREACH(block, &ram_list.blocks, next) {
+    QTAILQ_FOREACH(block, &ram_list.blocks, next) {
         if (addr == block->offset) {
-            QLIST_REMOVE(block, next);
+            QTAILQ_REMOVE(&ram_list.blocks, block, next);
             ram_list.mru_block = NULL;
             if (block->flags & RAM_PREALLOC_MASK) {
                 ;
@@ -1126,7 +1126,7 @@ void qemu_ram_remap(ram_addr_t addr, ram_addr_t length)
     int flags;
     void *area, *vaddr;

-    QLIST_FOREACH(block, &ram_list.blocks, next) {
+    QTAILQ_FOREACH(block, &ram_list.blocks, next) {
         offset = addr - block->offset;
         if (offset < block->length) {
             vaddr = block->host + offset;
@@ -1196,7 +1196,7 @@ void *qemu_get_ram_ptr(ram_addr_t addr)
     if (block && addr - block->offset < block->length) {
         goto found;
     }
-    QLIST_FOREACH(block, &ram_list.blocks, next) {
+    QTAILQ_FOREACH(block, &ram_list.blocks, next) {
         if (addr - block->offset < block->length) {
             goto found;
         }
@@ -1231,7 +1231,7 @@ static void *qemu_safe_ram_ptr(ram_addr_t addr)
 {
     RAMBlock *block;

-    QLIST_FOREACH(block, &ram_list.blocks, next) {
+    QTAILQ_FOREACH(block, &ram_list.blocks, next) {
         if (addr - block->offset < block->length) {
             if (xen_enabled()) {
                 /* We need to check if the requested address is in the RAM
@@ -1267,7 +1267,7 @@ static void *qemu_ram_ptr_length(ram_addr_t addr, ram_addr_t *size)
     } else {
         RAMBlock *block;

-        QLIST_FOREACH(block, &ram_list.blocks, next) {
+        QTAILQ_FOREACH(block, &ram_list.blocks, next) {
             if (addr - block->offset < block->length) {
                 if (addr - block->offset + *size > block->length)
                     *size = block->length - addr + block->offset;
@@ -1295,7 +1295,7 @@ int qemu_ram_addr_from_host(void *ptr, ram_addr_t *ram_addr)
         return 0;
     }

-    QLIST_FOREACH(block, &ram_list.blocks, next) {
+    QTAILQ_FOREACH(block, &ram_list.blocks, next) {
         /* This case append when the block is not mapped. */
         if (block->host == NULL) {
             continue;
diff --git a/memory_mapping.c b/memory_mapping.c
index a82e190..42aa98a 100644
--- a/memory_mapping.c
+++ b/memory_mapping.c
@@ -200,7 +200,7 @@ int qemu_get_guest_memory_mapping(MemoryMappingList *list)
      * If the guest doesn't use paging, the virtual address is equal to physical
      * address.
      */
-    QLIST_FOREACH(block, &ram_list.blocks, next) {
+    QTAILQ_FOREACH(block, &ram_list.blocks, next) {
         offset = block->offset;
         length = block->length;
         create_new_memory_mapping(list, offset, offset, length);
@@ -213,7 +213,7 @@ void qemu_get_guest_simple_memory_mapping(MemoryMappingList *list)
 {
     RAMBlock *block;

-    QLIST_FOREACH(block, &ram_list.blocks, next) {
+    QTAILQ_FOREACH(block, &ram_list.blocks, next) {
         create_new_memory_mapping(list, block->offset, 0, block->length);
     }
 }
diff --git a/target-i386/arch_dump.c b/target-i386/arch_dump.c
index 4240278..c88f21f 100644
--- a/target-i386/arch_dump.c
+++ b/target-i386/arch_dump.c
@@ -403,7 +403,7 @@ int cpu_get_dump_info(ArchDumpInfo *info)
     } else {
         info->d_class = ELFCLASS32;

-        QLIST_FOREACH(block, &ram_list.blocks, next) {
+        QTAILQ_FOREACH(block, &ram_list.blocks, next) {
             if (block->offset + block->length > UINT_MAX) {
                 /* The memory size is greater than 4G */
                 info->d_class = ELFCLASS64;
-- 
1.7.11.7

[Qemu-devel] [PATCH 08/34] exec: sort the memory from biggest to smallest

[Juan Quintela]

From: Paolo Bonzini <pbonzini@redhat.com>

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 arch_init.c | 30 ------------------------------
 exec.c      | 14 ++++++++++++--
 2 files changed, 12 insertions(+), 32 deletions(-)

diff --git a/arch_init.c b/arch_init.c
index 3c1aa00..8b5980f 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -505,35 +505,6 @@ uint64_t ram_bytes_total(void)
     return total;
 }

-static int block_compar(const void *a, const void *b)
-{
-    RAMBlock * const *ablock = a;
-    RAMBlock * const *bblock = b;
-
-    return strcmp((*ablock)->idstr, (*bblock)->idstr);
-}
-
-static void sort_ram_list(void)
-{
-    RAMBlock *block, *nblock, **blocks;
-    int n;
-    n = 0;
-    QTAILQ_FOREACH(block, &ram_list.blocks, next) {
-        ++n;
-    }
-    blocks = g_malloc(n * sizeof *blocks);
-    n = 0;
-    QTAILQ_FOREACH_SAFE(block, &ram_list.blocks, next, nblock) {
-        blocks[n++] = block;
-        QTAILQ_REMOVE(&ram_list.blocks, block, next);
-    }
-    qsort(blocks, n, sizeof *blocks, block_compar);
-    while (--n >= 0) {
-        QTAILQ_INSERT_HEAD(&ram_list.blocks, blocks[n], next);
-    }
-    g_free(blocks);
-}
-
 static void migration_end(void)
 {
     if (migration_bitmap) {
@@ -562,7 +533,6 @@ static void reset_ram_globals(void)
 {
     last_block = NULL;
     last_offset = 0;
-    sort_ram_list();
 }

 #define MAX_WAIT 50 /* ms, half buffered_file limit */
diff --git a/exec.c b/exec.c
index 13c894d..5f501d4 100644
--- a/exec.c
+++ b/exec.c
@@ -1006,7 +1006,7 @@ static int memory_try_enable_merging(void *addr, size_t len)
 ram_addr_t qemu_ram_alloc_from_ptr(ram_addr_t size, void *host,
                                    MemoryRegion *mr)
 {
-    RAMBlock *new_block;
+    RAMBlock *block, *new_block;

     size = TARGET_PAGE_ALIGN(size);
     new_block = g_malloc0(sizeof(*new_block));
@@ -1042,7 +1042,17 @@ ram_addr_t qemu_ram_alloc_from_ptr(ram_addr_t size, void *host,
     }
     new_block->length = size;

-    QTAILQ_INSERT_HEAD(&ram_list.blocks, new_block, next);
+    /* Keep the list sorted from biggest to smallest block.  */
+    QTAILQ_FOREACH(block, &ram_list.blocks, next) {
+        if (block->length < new_block->length) {
+            break;
+        }
+    }
+    if (block) {
+        QTAILQ_INSERT_BEFORE(block, new_block, next);
+    } else {
+        QTAILQ_INSERT_TAIL(&ram_list.blocks, new_block, next);
+    }
     ram_list.mru_block = NULL;

     ram_list.phys_dirty = g_realloc(ram_list.phys_dirty,
-- 
1.7.11.7

[Qemu-devel] [PATCH 09/34] add a version number to ram_list

[Juan Quintela]

From: Umesh Deshpande <udeshpan@redhat.com>

This will be used to detect if last_block might have become invalid
across different calls to ram_save_live.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Umesh Deshpande <udeshpan@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
---
 arch_init.c | 7 ++++++-
 cpu-all.h   | 1 +
 exec.c      | 4 ++++
 3 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/arch_init.c b/arch_init.c
index 8b5980f..136d0e8 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -336,6 +336,7 @@ static RAMBlock *last_block;
 static ram_addr_t last_offset;
 static unsigned long *migration_bitmap;
 static uint64_t migration_dirty_pages;
+static uint32_t last_version;

 static inline bool migration_bitmap_test_and_reset_dirty(MemoryRegion *mr,
                                                          ram_addr_t offset)
@@ -406,7 +407,6 @@ static void migration_bitmap_sync(void)
     }
 }

-
 /*
  * ram_save_block: Writes a page of memory to the stream f
  *
@@ -533,6 +533,7 @@ static void reset_ram_globals(void)
 {
     last_block = NULL;
     last_offset = 0;
+    last_version = ram_list.version;
 }

 #define MAX_WAIT 50 /* ms, half buffered_file limit */
@@ -587,6 +588,10 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
     uint64_t expected_downtime;
     MigrationState *s = migrate_get_current();

+    if (ram_list.version != last_version) {
+        reset_ram_globals();
+    }
+
     bytes_transferred_last = bytes_transferred;
     bwidth = qemu_get_clock_ns(rt_clock);

diff --git a/cpu-all.h b/cpu-all.h
index cd61320..c69d602 100644
--- a/cpu-all.h
+++ b/cpu-all.h
@@ -497,6 +497,7 @@ typedef struct RAMList {
     uint8_t *phys_dirty;
     RAMBlock *mru_block;
     QTAILQ_HEAD(, RAMBlock) blocks;
+    uint32_t version;
 } RAMList;
 extern RAMList ram_list;

diff --git a/exec.c b/exec.c
index 5f501d4..4474d60 100644
--- a/exec.c
+++ b/exec.c
@@ -1055,6 +1055,8 @@ ram_addr_t qemu_ram_alloc_from_ptr(ram_addr_t size, void *host,
     }
     ram_list.mru_block = NULL;

+    ram_list.version++;
+
     ram_list.phys_dirty = g_realloc(ram_list.phys_dirty,
                                        last_ram_offset() >> TARGET_PAGE_BITS);
     memset(ram_list.phys_dirty + (new_block->offset >> TARGET_PAGE_BITS),
@@ -1083,6 +1085,7 @@ void qemu_ram_free_from_ptr(ram_addr_t addr)
         if (addr == block->offset) {
             QTAILQ_REMOVE(&ram_list.blocks, block, next);
             ram_list.mru_block = NULL;
+            ram_list.version++;
             g_free(block);
             return;
         }
@@ -1097,6 +1100,7 @@ void qemu_ram_free(ram_addr_t addr)
         if (addr == block->offset) {
             QTAILQ_REMOVE(&ram_list.blocks, block, next);
             ram_list.mru_block = NULL;
+            ram_list.version++;
             if (block->flags & RAM_PREALLOC_MASK) {
                 ;
             } else if (mem_path) {
-- 
1.7.11.7

[Qemu-devel] [PATCH 10/34] protect the ramlist with a separate mutex

[Juan Quintela]

From: Umesh Deshpande <udeshpan@redhat.com>

Add the new mutex that protects shared state between ram_save_live
and the iothread.  If the iothread mutex has to be taken together
with the ramlist mutex, the iothread shall always be _outside_.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Umesh Deshpande <udeshpan@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
---
 arch_init.c |  9 ++++++++-
 cpu-all.h   |  9 +++++++++
 exec.c      | 29 +++++++++++++++++++++++++++--
 3 files changed, 44 insertions(+), 3 deletions(-)

diff --git a/arch_init.c b/arch_init.c
index 136d0e8..9cee58a 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -528,7 +528,6 @@ static void ram_migration_cancel(void *opaque)
     migration_end();
 }

-
 static void reset_ram_globals(void)
 {
     last_block = NULL;
@@ -547,6 +546,7 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
     bitmap_set(migration_bitmap, 0, ram_pages);
     migration_dirty_pages = ram_pages;

+    qemu_mutex_lock_ramlist();
     bytes_transferred = 0;
     reset_ram_globals();

@@ -574,6 +574,7 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
         qemu_put_be64(f, block->length);
     }

+    qemu_mutex_unlock_ramlist();
     qemu_put_be64(f, RAM_SAVE_FLAG_EOS);

     return 0;
@@ -588,6 +589,8 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
     uint64_t expected_downtime;
     MigrationState *s = migrate_get_current();

+    qemu_mutex_lock_ramlist();
+
     if (ram_list.version != last_version) {
         reset_ram_globals();
     }
@@ -636,6 +639,7 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
         bwidth = 0.000001;
     }

+    qemu_mutex_unlock_ramlist();
     qemu_put_be64(f, RAM_SAVE_FLAG_EOS);

     expected_downtime = ram_save_remaining() * TARGET_PAGE_SIZE / bwidth;
@@ -656,6 +660,8 @@ static int ram_save_complete(QEMUFile *f, void *opaque)
 {
     migration_bitmap_sync();

+    qemu_mutex_lock_ramlist();
+
     /* try transferring iterative blocks of memory */

     /* flush all remaining blocks regardless of rate limiting */
@@ -671,6 +677,7 @@ static int ram_save_complete(QEMUFile *f, void *opaque)
     }
     migration_end();

+    qemu_mutex_unlock_ramlist();
     qemu_put_be64(f, RAM_SAVE_FLAG_EOS);

     return 0;
diff --git a/cpu-all.h b/cpu-all.h
index c69d602..db79894 100644
--- a/cpu-all.h
+++ b/cpu-all.h
@@ -22,6 +22,7 @@
 #include "qemu-common.h"
 #include "qemu-tls.h"
 #include "cpu-common.h"
+#include "qemu-thread.h"

 /* some important defines:
  *
@@ -487,6 +488,9 @@ typedef struct RAMBlock {
     ram_addr_t length;
     uint32_t flags;
     char idstr[256];
+    /* Reads can take either the iothread or the ramlist lock.
+     * Writes must take both locks.
+     */
     QTAILQ_ENTRY(RAMBlock) next;
 #if defined(__linux__) && !defined(TARGET_S390X)
     int fd;
@@ -494,8 +498,11 @@ typedef struct RAMBlock {
 } RAMBlock;

 typedef struct RAMList {
+    QemuMutex mutex;
+    /* Protected by the iothread lock.  */
     uint8_t *phys_dirty;
     RAMBlock *mru_block;
+    /* Protected by the ramlist lock.  */
     QTAILQ_HEAD(, RAMBlock) blocks;
     uint32_t version;
 } RAMList;
@@ -516,6 +523,8 @@ extern int mem_prealloc;

 void dump_exec_info(FILE *f, fprintf_function cpu_fprintf);
 ram_addr_t last_ram_offset(void);
+void qemu_mutex_lock_ramlist(void);
+void qemu_mutex_unlock_ramlist(void);
 #endif /* !CONFIG_USER_ONLY */

 int cpu_memory_rw_debug(CPUArchState *env, target_ulong addr,
diff --git a/exec.c b/exec.c
index 4474d60..a43659b 100644
--- a/exec.c
+++ b/exec.c
@@ -212,6 +212,7 @@ bool memory_region_is_unassigned(MemoryRegion *mr)
 void cpu_exec_init_all(void)
 {
 #if !defined(CONFIG_USER_ONLY)
+    qemu_mutex_init(&ram_list.mutex);
     memory_map_init();
     io_mem_init();
 #endif
@@ -800,6 +801,16 @@ void qemu_flush_coalesced_mmio_buffer(void)
         kvm_flush_coalesced_mmio_buffer();
 }

+void qemu_mutex_lock_ramlist(void)
+{
+    qemu_mutex_lock(&ram_list.mutex);
+}
+
+void qemu_mutex_unlock_ramlist(void)
+{
+    qemu_mutex_unlock(&ram_list.mutex);
+}
+
 #if defined(__linux__) && !defined(TARGET_S390X)

 #include <sys/vfs.h>
@@ -981,6 +992,8 @@ void qemu_ram_set_idstr(ram_addr_t addr, const char *name, DeviceState *dev)
     }
     pstrcat(new_block->idstr, sizeof(new_block->idstr), name);

+    /* This assumes the iothread lock is taken here too.  */
+    qemu_mutex_lock_ramlist();
     QTAILQ_FOREACH(block, &ram_list.blocks, next) {
         if (block != new_block && !strcmp(block->idstr, new_block->idstr)) {
             fprintf(stderr, "RAMBlock \"%s\" already registered, abort!\n",
@@ -988,6 +1001,7 @@ void qemu_ram_set_idstr(ram_addr_t addr, const char *name, DeviceState *dev)
             abort();
         }
     }
+    qemu_mutex_unlock_ramlist();
 }

 static int memory_try_enable_merging(void *addr, size_t len)
@@ -1011,6 +1025,8 @@ ram_addr_t qemu_ram_alloc_from_ptr(ram_addr_t size, void *host,
     size = TARGET_PAGE_ALIGN(size);
     new_block = g_malloc0(sizeof(*new_block));

+    /* This assumes the iothread lock is taken here too.  */
+    qemu_mutex_lock_ramlist();
     new_block->mr = mr;
     new_block->offset = find_ram_offset(size);
     if (host) {
@@ -1056,6 +1072,7 @@ ram_addr_t qemu_ram_alloc_from_ptr(ram_addr_t size, void *host,
     ram_list.mru_block = NULL;

     ram_list.version++;
+    qemu_mutex_unlock_ramlist();

     ram_list.phys_dirty = g_realloc(ram_list.phys_dirty,
                                        last_ram_offset() >> TARGET_PAGE_BITS);
@@ -1081,21 +1098,26 @@ void qemu_ram_free_from_ptr(ram_addr_t addr)
 {
     RAMBlock *block;

+    /* This assumes the iothread lock is taken here too.  */
+    qemu_mutex_lock_ramlist();
     QTAILQ_FOREACH(block, &ram_list.blocks, next) {
         if (addr == block->offset) {
             QTAILQ_REMOVE(&ram_list.blocks, block, next);
             ram_list.mru_block = NULL;
             ram_list.version++;
             g_free(block);
-            return;
+            break;
         }
     }
+    qemu_mutex_unlock_ramlist();
 }

 void qemu_ram_free(ram_addr_t addr)
 {
     RAMBlock *block;

+    /* This assumes the iothread lock is taken here too.  */
+    qemu_mutex_lock_ramlist();
     QTAILQ_FOREACH(block, &ram_list.blocks, next) {
         if (addr == block->offset) {
             QTAILQ_REMOVE(&ram_list.blocks, block, next);
@@ -1126,9 +1148,10 @@ void qemu_ram_free(ram_addr_t addr)
 #endif
             }
             g_free(block);
-            return;
+            break;
         }
     }
+    qemu_mutex_unlock_ramlist();

 }

@@ -1206,6 +1229,7 @@ void *qemu_get_ram_ptr(ram_addr_t addr)
 {
     RAMBlock *block;

+    /* The list is protected by the iothread lock here.  */
     block = ram_list.mru_block;
     if (block && addr - block->offset < block->length) {
         goto found;
@@ -1245,6 +1269,7 @@ static void *qemu_safe_ram_ptr(ram_addr_t addr)
 {
     RAMBlock *block;

+    /* The list is protected by the iothread lock here.  */
     QTAILQ_FOREACH(block, &ram_list.blocks, next) {
         if (addr - block->offset < block->length) {
             if (xen_enabled()) {
-- 
1.7.11.7

[Qemu-devel] [PATCH 11/34] buffered_file: Move from using a timer to use a thread

[Juan Quintela]

We still protect everything except the wait with the iothread lock.
But we moved from a timer to a thread.  Steps one by one.

We also need to detect when we have finished with a variable "complete".

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 buffered_file.c | 58 +++++++++++++++++++++++++++++++++++----------------------
 1 file changed, 36 insertions(+), 22 deletions(-)

diff --git a/buffered_file.c b/buffered_file.c
index 9a8bd04..2750cc3 100644
--- a/buffered_file.c
+++ b/buffered_file.c
@@ -18,6 +18,7 @@
 #include "qemu-timer.h"
 #include "qemu-char.h"
 #include "buffered_file.h"
+#include "qemu-thread.h"

 //#define DEBUG_BUFFERED_FILE

@@ -31,7 +32,8 @@ typedef struct QEMUFileBuffered
     uint8_t *buffer;
     size_t buffer_size;
     size_t buffer_capacity;
-    QEMUTimer *timer;
+    QemuThread thread;
+    bool complete;
 } QEMUFileBuffered;

 #ifdef DEBUG_BUFFERED_FILE
@@ -160,11 +162,8 @@ static int buffered_close(void *opaque)
     if (ret >= 0) {
         ret = ret2;
     }
-    qemu_del_timer(s->timer);
-    qemu_free_timer(s->timer);
-    g_free(s->buffer);
-    g_free(s);
-
+    ret = migrate_fd_close(s->migration_state);
+    s->complete = true;
     return ret;
 }

@@ -222,23 +221,38 @@ static int64_t buffered_get_rate_limit(void *opaque)
     return s->xfer_limit;
 }

-static void buffered_rate_tick(void *opaque)
+/* 10ms  xfer_limit is the limit that we should write each 10ms */
+#define BUFFER_DELAY 100
+
+static void *buffered_file_thread(void *opaque)
 {
     QEMUFileBuffered *s = opaque;
+    int64_t expire_time = qemu_get_clock_ms(rt_clock) + BUFFER_DELAY;

-    if (qemu_file_get_error(s->file)) {
-        buffered_close(s);
-        return;
-    }
-
-    qemu_mod_timer(s->timer, qemu_get_clock_ms(rt_clock) + 100);
-
-    if (s->freeze_output)
-        return;
-
-    s->bytes_xfer = 0;
+    while (true) {
+        int64_t current_time = qemu_get_clock_ms(rt_clock);

-    buffered_put_buffer(s, NULL, 0, 0);
+        if (s->complete) {
+            break;
+        }
+        if (s->freeze_output) {
+            continue;
+        }
+        if (current_time >= expire_time) {
+            s->bytes_xfer = 0;
+            expire_time = current_time + BUFFER_DELAY;
+        }
+        if (s->bytes_xfer >= s->xfer_limit) {
+            /* usleep expects microseconds */
+            g_usleep((expire_time - current_time)*1000);
+        }
+        qemu_mutex_lock_iothread();
+        buffered_put_buffer(s, NULL, 0, 0);
+        qemu_mutex_unlock_iothread();
+    }
+    g_free(s->buffer);
+    g_free(s);
+    return NULL;
 }

 static const QEMUFileOps buffered_file_ops = {
@@ -258,12 +272,12 @@ QEMUFile *qemu_fopen_ops_buffered(MigrationState *migration_state)

     s->migration_state = migration_state;
     s->xfer_limit = migration_state->bandwidth_limit / 10;
+    s->complete = false;

     s->file = qemu_fopen_ops(s, &buffered_file_ops);

-    s->timer = qemu_new_timer_ms(rt_clock, buffered_rate_tick, s);
-
-    qemu_mod_timer(s->timer, qemu_get_clock_ms(rt_clock) + 100);
+    qemu_thread_create(&s->thread, buffered_file_thread, s,
+                       QEMU_THREAD_DETACHED);

     return s->file;
 }
-- 
1.7.11.7

[Qemu-devel] [PATCH 12/34] migration: make qemu_fopen_ops_buffered() return void

[Juan Quintela]

We want the file assignment to happen before the thread is created to
avoid locking, so we just do it before creating the thread.

Signed-off-by: Juan Quintela <quintela@redhat.com>

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
---
 buffered_file.c | 13 ++++++-------
 buffered_file.h |  2 +-
 migration.c     |  2 +-
 migration.h     |  1 +
 4 files changed, 9 insertions(+), 9 deletions(-)

diff --git a/buffered_file.c b/buffered_file.c
index 2750cc3..18da74b 100644
--- a/buffered_file.c
+++ b/buffered_file.c
@@ -33,7 +33,6 @@ typedef struct QEMUFileBuffered
     size_t buffer_size;
     size_t buffer_capacity;
     QemuThread thread;
-    bool complete;
 } QEMUFileBuffered;

 #ifdef DEBUG_BUFFERED_FILE
@@ -163,7 +162,7 @@ static int buffered_close(void *opaque)
         ret = ret2;
     }
     ret = migrate_fd_close(s->migration_state);
-    s->complete = true;
+    s->migration_state->complete = true;
     return ret;
 }

@@ -232,7 +231,7 @@ static void *buffered_file_thread(void *opaque)
     while (true) {
         int64_t current_time = qemu_get_clock_ms(rt_clock);

-        if (s->complete) {
+        if (s->migration_state->complete) {
             break;
         }
         if (s->freeze_output) {
@@ -264,7 +263,7 @@ static const QEMUFileOps buffered_file_ops = {
     .set_rate_limit = buffered_set_rate_limit,
 };

-QEMUFile *qemu_fopen_ops_buffered(MigrationState *migration_state)
+void qemu_fopen_ops_buffered(MigrationState *migration_state)
 {
     QEMUFileBuffered *s;

@@ -272,12 +271,12 @@ QEMUFile *qemu_fopen_ops_buffered(MigrationState *migration_state)

     s->migration_state = migration_state;
     s->xfer_limit = migration_state->bandwidth_limit / 10;
-    s->complete = false;
+    s->migration_state->complete = false;

     s->file = qemu_fopen_ops(s, &buffered_file_ops);

+    migration_state->file = s->file;
+
     qemu_thread_create(&s->thread, buffered_file_thread, s,
                        QEMU_THREAD_DETACHED);
-
-    return s->file;
 }
diff --git a/buffered_file.h b/buffered_file.h
index ef010fe..8a246fd 100644
--- a/buffered_file.h
+++ b/buffered_file.h
@@ -17,6 +17,6 @@
 #include "hw/hw.h"
 #include "migration.h"

-QEMUFile *qemu_fopen_ops_buffered(MigrationState *migration_state);
+void qemu_fopen_ops_buffered(MigrationState *migration_state);

 #endif
diff --git a/migration.c b/migration.c
index bde4956..f32fe3e 100644
--- a/migration.c
+++ b/migration.c
@@ -448,7 +448,7 @@ void migrate_fd_connect(MigrationState *s)
     int ret;

     s->state = MIG_STATE_ACTIVE;
-    s->file = qemu_fopen_ops_buffered(s);
+    qemu_fopen_ops_buffered(s);

     DPRINTF("beginning savevm\n");
     ret = qemu_savevm_state_begin(s->file, &s->params);
diff --git a/migration.h b/migration.h
index c3a23cc..b66fd60 100644
--- a/migration.h
+++ b/migration.h
@@ -45,6 +45,7 @@ struct MigrationState
     int64_t dirty_pages_rate;
     bool enabled_capabilities[MIGRATION_CAPABILITY_MAX];
     int64_t xbzrle_cache_size;
+    bool complete;
 };

 void process_incoming_migration(QEMUFile *f);
-- 
1.7.11.7

[Qemu-devel] [PATCH 13/34] migration: move migration thread init code to migrate_fd_put_ready

[Juan Quintela]

This way everything related with migration is run on the migration
thread and no locking is needed.

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration.c | 29 +++++++++++++++++------------
 migration.h |  1 +
 2 files changed, 18 insertions(+), 12 deletions(-)

diff --git a/migration.c b/migration.c
index f32fe3e..90dbe8e 100644
--- a/migration.c
+++ b/migration.c
@@ -20,6 +20,7 @@
 #include "sysemu.h"
 #include "block.h"
 #include "qemu_socket.h"
+#include "qemu-thread.h"
 #include "block-migration.h"
 #include "qmp-commands.h"

@@ -339,6 +340,16 @@ void migrate_fd_put_ready(MigrationState *s)
         DPRINTF("put_ready returning because of non-active state\n");
         return;
     }
+    if (s->first_time) {
+        s->first_time = false;
+        DPRINTF("beginning savevm\n");
+        ret = qemu_savevm_state_begin(s->file, &s->params);
+        if (ret < 0) {
+            DPRINTF("failed, %d\n", ret);
+            migrate_fd_error(s);
+            return;
+        }
+    }

     DPRINTF("iterate\n");
     ret = qemu_savevm_state_iterate(s->file);
@@ -351,7 +362,11 @@ void migrate_fd_put_ready(MigrationState *s)
         DPRINTF("done iterating\n");
         start_time = qemu_get_clock_ms(rt_clock);
         qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
-        vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
+        if (old_vm_running) {
+            vm_stop(RUN_STATE_FINISH_MIGRATE);
+        } else {
+            vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
+        }

         if (qemu_savevm_state_complete(s->file) < 0) {
             migrate_fd_error(s);
@@ -445,19 +460,9 @@ bool migration_has_failed(MigrationState *s)

 void migrate_fd_connect(MigrationState *s)
 {
-    int ret;
-
     s->state = MIG_STATE_ACTIVE;
+    s->first_time = true;
     qemu_fopen_ops_buffered(s);
-
-    DPRINTF("beginning savevm\n");
-    ret = qemu_savevm_state_begin(s->file, &s->params);
-    if (ret < 0) {
-        DPRINTF("failed, %d\n", ret);
-        migrate_fd_error(s);
-        return;
-    }
-    migrate_fd_put_ready(s);
 }

 static MigrationState *migrate_init(const MigrationParams *params)
diff --git a/migration.h b/migration.h
index b66fd60..8f2ff8a 100644
--- a/migration.h
+++ b/migration.h
@@ -46,6 +46,7 @@ struct MigrationState
     bool enabled_capabilities[MIGRATION_CAPABILITY_MAX];
     int64_t xbzrle_cache_size;
     bool complete;
+    bool first_time;
 };

 void process_incoming_migration(QEMUFile *f);
-- 
1.7.11.7

[Qemu-devel] [PATCH 14/34] migration: make writes blocking

[Juan Quintela]

Move all the writes to the migration_thread, and make writings
blocking.  Notice that are still using the iothread for everything
that we do.

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration-exec.c |  1 -
 migration-fd.c   |  1 -
 migration-tcp.c  |  1 +
 migration-unix.c |  1 +
 migration.c      | 17 -----------------
 qemu-file.h      |  5 -----
 savevm.c         |  5 -----
 7 files changed, 2 insertions(+), 29 deletions(-)

diff --git a/migration-exec.c b/migration-exec.c
index f449a22..0997a24 100644
--- a/migration-exec.c
+++ b/migration-exec.c
@@ -70,7 +70,6 @@ void exec_start_outgoing_migration(MigrationState *s, const char *command, Error

     s->fd = fileno(f);
     assert(s->fd != -1);
-    socket_set_nonblock(s->fd);

     s->opaque = qemu_popen(f, "w");

diff --git a/migration-fd.c b/migration-fd.c
index b8d16ad..77aef6d 100644
--- a/migration-fd.c
+++ b/migration-fd.c
@@ -76,7 +76,6 @@ void fd_start_outgoing_migration(MigrationState *s, const char *fdname, Error **
         return;
     }

-    fcntl(s->fd, F_SETFL, O_NONBLOCK);
     s->get_error = fd_errno;
     s->write = fd_write;
     s->close = fd_close;
diff --git a/migration-tcp.c b/migration-tcp.c
index 1683158..5808857 100644
--- a/migration-tcp.c
+++ b/migration-tcp.c
@@ -61,6 +61,7 @@ static void tcp_wait_for_connect(int fd, void *opaque)
     } else {
         DPRINTF("migrate connect success\n");
         s->fd = fd;
+        socket_set_block(s->fd);
         migrate_fd_connect(s);
     }
 }
diff --git a/migration-unix.c b/migration-unix.c
index da00f2f..81a8176 100644
--- a/migration-unix.c
+++ b/migration-unix.c
@@ -61,6 +61,7 @@ static void unix_wait_for_connect(int fd, void *opaque)
     } else {
         DPRINTF("migrate connect success\n");
         s->fd = fd;
+        socket_set_block(s->fd);
         migrate_fd_connect(s);
     }
 }
diff --git a/migration.c b/migration.c
index 90dbe8e..5aa3bca 100644
--- a/migration.c
+++ b/migration.c
@@ -297,18 +297,6 @@ static void migrate_fd_completed(MigrationState *s)
     notifier_list_notify(&migration_state_notifiers, s);
 }

-static void migrate_fd_put_notify(void *opaque)
-{
-    MigrationState *s = opaque;
-    int ret;
-
-    qemu_set_fd_handler2(s->fd, NULL, NULL, NULL, NULL);
-    ret = qemu_file_put_notify(s->file);
-    if (ret) {
-        migrate_fd_error(s);
-    }
-}
-
 ssize_t migrate_fd_put_buffer(MigrationState *s, const void *data,
                               size_t size)
 {
@@ -325,10 +313,6 @@ ssize_t migrate_fd_put_buffer(MigrationState *s, const void *data,
     if (ret == -1)
         ret = -(s->get_error(s));

-    if (ret == -EAGAIN) {
-        qemu_set_fd_handler2(s->fd, NULL, NULL, migrate_fd_put_notify, s);
-    }
-
     return ret;
 }

@@ -425,7 +409,6 @@ int migrate_fd_close(MigrationState *s)
 {
     int rc = 0;
     if (s->fd != -1) {
-        qemu_set_fd_handler2(s->fd, NULL, NULL, NULL, NULL);
         rc = s->close(s);
         s->fd = -1;
     }
diff --git a/qemu-file.h b/qemu-file.h
index d64bdbb..68deefb 100644
--- a/qemu-file.h
+++ b/qemu-file.h
@@ -113,11 +113,6 @@ int64_t qemu_file_set_rate_limit(QEMUFile *f, int64_t new_rate);
 int64_t qemu_file_get_rate_limit(QEMUFile *f);
 int qemu_file_get_error(QEMUFile *f);

-/* Try to send any outstanding data.  This function is useful when output is
- * halted due to rate limiting or EAGAIN errors occur as it can be used to
- * resume output. */
-int qemu_file_put_notify(QEMUFile *f);
-
 static inline void qemu_put_be64s(QEMUFile *f, const uint64_t *pv)
 {
     qemu_put_be64(f, *pv);
diff --git a/savevm.c b/savevm.c
index 5d04d59..c4ee899 100644
--- a/savevm.c
+++ b/savevm.c
@@ -556,11 +556,6 @@ int qemu_fclose(QEMUFile *f)
     return ret;
 }

-int qemu_file_put_notify(QEMUFile *f)
-{
-    return f->ops->put_buffer(f->opaque, NULL, 0, 0);
-}
-
 void qemu_put_buffer(QEMUFile *f, const uint8_t *buf, int size)
 {
     int l;
-- 
1.7.11.7

[Qemu-devel] [PATCH 15/34] migration: remove unfreeze logic

[Juan Quintela]

Now that we have a thread, and blocking writes, we don't need it.

Signed-off-by: Juan Quintela <quintela@redhat.com>

Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
---
 buffered_file.c | 24 +-----------------------
 migration.c     | 23 -----------------------
 migration.h     |  1 -
 3 files changed, 1 insertion(+), 47 deletions(-)

diff --git a/buffered_file.c b/buffered_file.c
index 18da74b..31f31f4 100644
--- a/buffered_file.c
+++ b/buffered_file.c
@@ -26,7 +26,6 @@ typedef struct QEMUFileBuffered
 {
     MigrationState *migration_state;
     QEMUFile *file;
-    int freeze_output;
     size_t bytes_xfer;
     size_t xfer_limit;
     uint8_t *buffer;
@@ -70,13 +69,6 @@ static ssize_t buffered_flush(QEMUFileBuffered *s)
         size_t to_send = MIN(s->buffer_size - offset, s->xfer_limit - s->bytes_xfer);
         ret = migrate_fd_put_buffer(s->migration_state, s->buffer + offset,
                                     to_send);
-        if (ret == -EAGAIN) {
-            DPRINTF("backend not ready, freezing\n");
-            ret = 0;
-            s->freeze_output = 1;
-            break;
-        }
-
         if (ret <= 0) {
             DPRINTF("error flushing data, %zd\n", ret);
             break;
@@ -110,9 +102,6 @@ static int buffered_put_buffer(void *opaque, const uint8_t *buf, int64_t pos, in
         return error;
     }

-    DPRINTF("unfreezing output\n");
-    s->freeze_output = 0;
-
     if (size > 0) {
         DPRINTF("buffering %d bytes\n", size - offset);
         buffered_append(s, buf, size);
@@ -126,7 +115,7 @@ static int buffered_put_buffer(void *opaque, const uint8_t *buf, int64_t pos, in

     if (pos == 0 && size == 0) {
         DPRINTF("file is ready\n");
-        if (!s->freeze_output && s->bytes_xfer < s->xfer_limit) {
+        if (s->bytes_xfer < s->xfer_limit) {
             DPRINTF("notifying client\n");
             migrate_fd_put_ready(s->migration_state);
         }
@@ -149,12 +138,6 @@ static int buffered_close(void *opaque)
         if (ret < 0) {
             break;
         }
-        if (s->freeze_output) {
-            ret = migrate_fd_wait_for_unfreeze(s->migration_state);
-            if (ret < 0) {
-                break;
-            }
-        }
     }

     ret2 = migrate_fd_close(s->migration_state);
@@ -188,8 +171,6 @@ static int buffered_rate_limit(void *opaque)
     if (ret) {
         return ret;
     }
-    if (s->freeze_output)
-        return 1;

     if (s->bytes_xfer > s->xfer_limit)
         return 1;
@@ -234,9 +215,6 @@ static void *buffered_file_thread(void *opaque)
         if (s->migration_state->complete) {
             break;
         }
-        if (s->freeze_output) {
-            continue;
-        }
         if (current_time >= expire_time) {
             s->bytes_xfer = 0;
             expire_time = current_time + BUFFER_DELAY;
diff --git a/migration.c b/migration.c
index 5aa3bca..ae0d7bd 100644
--- a/migration.c
+++ b/migration.c
@@ -382,29 +382,6 @@ static void migrate_fd_cancel(MigrationState *s)
     migrate_fd_cleanup(s);
 }

-int migrate_fd_wait_for_unfreeze(MigrationState *s)
-{
-    int ret;
-
-    DPRINTF("wait for unfreeze\n");
-    if (s->state != MIG_STATE_ACTIVE)
-        return -EINVAL;
-
-    do {
-        fd_set wfds;
-
-        FD_ZERO(&wfds);
-        FD_SET(s->fd, &wfds);
-
-        ret = select(s->fd + 1, NULL, &wfds, NULL, NULL);
-    } while (ret == -1 && (s->get_error(s)) == EINTR);
-
-    if (ret == -1) {
-        return -s->get_error(s);
-    }
-    return 0;
-}
-
 int migrate_fd_close(MigrationState *s)
 {
     int rc = 0;
diff --git a/migration.h b/migration.h
index 8f2ff8a..6760d7f 100644
--- a/migration.h
+++ b/migration.h
@@ -82,7 +82,6 @@ void migrate_fd_connect(MigrationState *s);
 ssize_t migrate_fd_put_buffer(MigrationState *s, const void *data,
                               size_t size);
 void migrate_fd_put_ready(MigrationState *s);
-int migrate_fd_wait_for_unfreeze(MigrationState *s);
 int migrate_fd_close(MigrationState *s);

 void add_migration_state_change_notifier(Notifier *notify);
-- 
1.7.11.7

[Qemu-devel] [PATCH 16/34] migration: just lock migrate_fd_put_ready

[Juan Quintela]

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 buffered_file.c | 2 --
 migration.c     | 5 +++++
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/buffered_file.c b/buffered_file.c
index 31f31f4..5cd477a 100644
--- a/buffered_file.c
+++ b/buffered_file.c
@@ -223,9 +223,7 @@ static void *buffered_file_thread(void *opaque)
             /* usleep expects microseconds */
             g_usleep((expire_time - current_time)*1000);
         }
-        qemu_mutex_lock_iothread();
         buffered_put_buffer(s, NULL, 0, 0);
-        qemu_mutex_unlock_iothread();
     }
     g_free(s->buffer);
     g_free(s);
diff --git a/migration.c b/migration.c
index ae0d7bd..d6cb320 100644
--- a/migration.c
+++ b/migration.c
@@ -320,8 +320,10 @@ void migrate_fd_put_ready(MigrationState *s)
 {
     int ret;

+    qemu_mutex_lock_iothread();
     if (s->state != MIG_STATE_ACTIVE) {
         DPRINTF("put_ready returning because of non-active state\n");
+        qemu_mutex_unlock_iothread();
         return;
     }
     if (s->first_time) {
@@ -331,6 +333,7 @@ void migrate_fd_put_ready(MigrationState *s)
         if (ret < 0) {
             DPRINTF("failed, %d\n", ret);
             migrate_fd_error(s);
+            qemu_mutex_unlock_iothread();
             return;
         }
     }
@@ -366,6 +369,8 @@ void migrate_fd_put_ready(MigrationState *s)
             }
         }
     }
+    qemu_mutex_unlock_iothread();
+
 }

 static void migrate_fd_cancel(MigrationState *s)
-- 
1.7.11.7

[Qemu-devel] [PATCH 17/34] buffered_file: Unfold the trick to restart generating migration data

[Juan Quintela]

This was needed before due to the way that the callbacks worked.

Signed-off-by: Juan Quintela <quintela@redhat.com>

Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
---
 buffered_file.c | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)

diff --git a/buffered_file.c b/buffered_file.c
index 5cd477a..f4f5110 100644
--- a/buffered_file.c
+++ b/buffered_file.c
@@ -113,14 +113,6 @@ static int buffered_put_buffer(void *opaque, const uint8_t *buf, int64_t pos, in
         return error;
     }

-    if (pos == 0 && size == 0) {
-        DPRINTF("file is ready\n");
-        if (s->bytes_xfer < s->xfer_limit) {
-            DPRINTF("notifying client\n");
-            migrate_fd_put_ready(s->migration_state);
-        }
-    }
-
     return size;
 }

@@ -223,8 +215,17 @@ static void *buffered_file_thread(void *opaque)
             /* usleep expects microseconds */
             g_usleep((expire_time - current_time)*1000);
         }
-        buffered_put_buffer(s, NULL, 0, 0);
+        if (buffered_flush(s) < 0) {
+            break;
+        }
+
+        DPRINTF("file is ready\n");
+        if (s->bytes_xfer < s->xfer_limit) {
+            DPRINTF("notifying client\n");
+            migrate_fd_put_ready(s->migration_state);
+        }
     }
+
     g_free(s->buffer);
     g_free(s);
     return NULL;
-- 
1.7.11.7

[Qemu-devel] [PATCH 18/34] buffered_file: don't flush on put buffer

[Juan Quintela]

We call buffered_put_buffer with iothread held, and buffered_flush() does
synchronous writes.  We only want to do the synchronous writes outside.

Signed-off-by: Juan Quintela <quintela@redhat.com>

Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
---
 buffered_file.c | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/buffered_file.c b/buffered_file.c
index f4f5110..2c3820c 100644
--- a/buffered_file.c
+++ b/buffered_file.c
@@ -107,12 +107,6 @@ static int buffered_put_buffer(void *opaque, const uint8_t *buf, int64_t pos, in
         buffered_append(s, buf, size);
     }

-    error = buffered_flush(s);
-    if (error < 0) {
-        DPRINTF("buffered flush error. bailing: %s\n", strerror(-error));
-        return error;
-    }
-
     return size;
 }

-- 
1.7.11.7

[Qemu-devel] [PATCH 19/34] buffered_file: unfold buffered_append in buffered_put_buffer

[Juan Quintela]

It was the only user, and now buffered_put_buffer just do the append

Signed-off-by: Juan Quintela <quintela@redhat.com>

Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
---
 buffered_file.c | 33 ++++++++++++++-------------------
 1 file changed, 14 insertions(+), 19 deletions(-)

diff --git a/buffered_file.c b/buffered_file.c
index 2c3820c..11efd8f 100644
--- a/buffered_file.c
+++ b/buffered_file.c
@@ -42,22 +42,6 @@ typedef struct QEMUFileBuffered
     do { } while (0)
 #endif

-static void buffered_append(QEMUFileBuffered *s,
-                            const uint8_t *buf, size_t size)
-{
-    if (size > (s->buffer_capacity - s->buffer_size)) {
-        DPRINTF("increasing buffer capacity from %zu by %zu\n",
-                s->buffer_capacity, size + 1024);
-
-        s->buffer_capacity += size + 1024;
-
-        s->buffer = g_realloc(s->buffer, s->buffer_capacity);
-    }
-
-    memcpy(s->buffer + s->buffer_size, buf, size);
-    s->buffer_size += size;
-}
-
 static ssize_t buffered_flush(QEMUFileBuffered *s)
 {
     size_t offset = 0;
@@ -102,11 +86,22 @@ static int buffered_put_buffer(void *opaque, const uint8_t *buf, int64_t pos, in
         return error;
     }

-    if (size > 0) {
-        DPRINTF("buffering %d bytes\n", size - offset);
-        buffered_append(s, buf, size);
+    if (size <= 0) {
+        return size;
     }

+    if (size > (s->buffer_capacity - s->buffer_size)) {
+        DPRINTF("increasing buffer capacity from %zu by %zu\n",
+                s->buffer_capacity, size + 1024);
+
+        s->buffer_capacity += size + 1024;
+
+        s->buffer = g_realloc(s->buffer, s->buffer_capacity);
+    }
+
+    memcpy(s->buffer + s->buffer_size, buf, size);
+    s->buffer_size += size;
+
     return size;
 }

-- 
1.7.11.7

[Qemu-devel] [PATCH 20/34] savevm: New save live migration method: pending

[Juan Quintela]

Code just now does (simplified for clarity)

    if (qemu_savevm_state_iterate(s->file) == 1) {
       vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
       qemu_savevm_state_complete(s->file);
    }

Problem here is that qemu_savevm_state_iterate() returns 1 when it
knows that remaining memory to sent takes less than max downtime.

But this means that we could end spending 2x max_downtime, one
downtime in qemu_savevm_iterate, and the other in
qemu_savevm_state_complete.

Changed code to:

    pending_size = qemu_savevm_state_pending(s->file, max_size);
    DPRINTF("pending size %lu max %lu\n", pending_size, max_size);
    if (pending_size >= max_size) {
        ret = qemu_savevm_state_iterate(s->file);
     } else {
        vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
        qemu_savevm_state_complete(s->file);
     }

So what we do is: at current network speed, we calculate the maximum
number of bytes we can sent: max_size.

Then we ask every save_live section how much they have pending.  If
they are less than max_size, we move to complete phase, otherwise we
do an iterate one.

This makes things much simpler, because now individual sections don't
have to caluclate the bandwidth (it was implossible to do right from
there).

Signed-off-by: Juan Quintela <quintela@redhat.com>

Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch_init.c       | 48 ++++++++++++++++++------------------------------
 block-migration.c | 49 ++++++++++---------------------------------------
 buffered_file.c   | 25 ++++++++++++++++++-------
 migration.c       | 22 +++++++++++++++-------
 migration.h       |  2 +-
 savevm.c          | 19 +++++++++++++++++++
 sysemu.h          |  1 +
 vmstate.h         |  1 +
 8 files changed, 83 insertions(+), 84 deletions(-)

diff --git a/arch_init.c b/arch_init.c
index 9cee58a..f092ea2 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -582,12 +582,9 @@ static int ram_save_setup(QEMUFile *f, void *opaque)

 static int ram_save_iterate(QEMUFile *f, void *opaque)
 {
-    uint64_t bytes_transferred_last;
-    double bwidth = 0;
     int ret;
     int i;
-    uint64_t expected_downtime;
-    MigrationState *s = migrate_get_current();
+    int64_t t0;

     qemu_mutex_lock_ramlist();

@@ -595,9 +592,7 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
         reset_ram_globals();
     }

-    bytes_transferred_last = bytes_transferred;
-    bwidth = qemu_get_clock_ns(rt_clock);
-
+    t0 = qemu_get_clock_ns(rt_clock);
     i = 0;
     while ((ret = qemu_file_rate_limit(f)) == 0) {
         int bytes_sent;
@@ -615,7 +610,7 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
            iterations
         */
         if ((i & 63) == 0) {
-            uint64_t t1 = (qemu_get_clock_ns(rt_clock) - bwidth) / 1000000;
+            uint64_t t1 = (qemu_get_clock_ns(rt_clock) - t0) / 1000000;
             if (t1 > MAX_WAIT) {
                 DPRINTF("big wait: %" PRIu64 " milliseconds, %d iterations\n",
                         t1, i);
@@ -629,31 +624,10 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
         return ret;
     }

-    bwidth = qemu_get_clock_ns(rt_clock) - bwidth;
-    bwidth = (bytes_transferred - bytes_transferred_last) / bwidth;
-
-    /* if we haven't transferred anything this round, force
-     * expected_downtime to a very high value, but without
-     * crashing */
-    if (bwidth == 0) {
-        bwidth = 0.000001;
-    }
-
     qemu_mutex_unlock_ramlist();
     qemu_put_be64(f, RAM_SAVE_FLAG_EOS);

-    expected_downtime = ram_save_remaining() * TARGET_PAGE_SIZE / bwidth;
-    DPRINTF("ram_save_live: expected(%" PRIu64 ") <= max(" PRIu64 ")?\n",
-            expected_downtime, migrate_max_downtime());
-
-    if (expected_downtime <= migrate_max_downtime()) {
-        migration_bitmap_sync();
-        expected_downtime = ram_save_remaining() * TARGET_PAGE_SIZE / bwidth;
-        s->expected_downtime = expected_downtime / 1000000; /* ns -> ms */
-
-        return expected_downtime <= migrate_max_downtime();
-    }
-    return 0;
+    return i;
 }

 static int ram_save_complete(QEMUFile *f, void *opaque)
@@ -683,6 +657,19 @@ static int ram_save_complete(QEMUFile *f, void *opaque)
     return 0;
 }

+static uint64_t ram_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)
+{
+    uint64_t remaining_size;
+
+    remaining_size = ram_save_remaining() * TARGET_PAGE_SIZE;
+
+    if (remaining_size < max_size) {
+        migration_bitmap_sync();
+        remaining_size = ram_save_remaining() * TARGET_PAGE_SIZE;
+    }
+    return remaining_size;
+}
+
 static int load_xbzrle(QEMUFile *f, ram_addr_t addr, void *host)
 {
     int ret, rc = 0;
@@ -869,6 +856,7 @@ SaveVMHandlers savevm_ram_handlers = {
     .save_live_setup = ram_save_setup,
     .save_live_iterate = ram_save_iterate,
     .save_live_complete = ram_save_complete,
+    .save_live_pending = ram_save_pending,
     .load_state = ram_load,
     .cancel = ram_migration_cancel,
 };
diff --git a/block-migration.c b/block-migration.c
index 71b9601..5db01fe 100644
--- a/block-migration.c
+++ b/block-migration.c
@@ -77,9 +77,7 @@ typedef struct BlkMigState {
     int64_t total_sector_sum;
     int prev_progress;
     int bulk_completed;
-    long double total_time;
     long double prev_time_offset;
-    int reads;
 } BlkMigState;

 static BlkMigState block_mig_state;
@@ -132,12 +130,6 @@ uint64_t blk_mig_bytes_total(void)
     return sum << BDRV_SECTOR_BITS;
 }

-static inline long double compute_read_bwidth(void)
-{
-    assert(block_mig_state.total_time != 0);
-    return (block_mig_state.reads / block_mig_state.total_time) * BLOCK_SIZE;
-}
-
 static int bmds_aio_inflight(BlkMigDevState *bmds, int64_t sector)
 {
     int64_t chunk = sector / (int64_t)BDRV_SECTORS_PER_DIRTY_CHUNK;
@@ -191,8 +183,6 @@ static void blk_mig_read_cb(void *opaque, int ret)

     blk->ret = ret;

-    block_mig_state.reads++;
-    block_mig_state.total_time += (curr_time - block_mig_state.prev_time_offset);
     block_mig_state.prev_time_offset = curr_time;

     QSIMPLEQ_INSERT_TAIL(&block_mig_state.blk_list, blk, entry);
@@ -310,8 +300,6 @@ static void init_blk_migration(QEMUFile *f)
     block_mig_state.total_sector_sum = 0;
     block_mig_state.prev_progress = -1;
     block_mig_state.bulk_completed = 0;
-    block_mig_state.total_time = 0;
-    block_mig_state.reads = 0;

     bdrv_iterate(init_blk_migration_it, NULL);
 }
@@ -493,32 +481,6 @@ static int64_t get_remaining_dirty(void)
     return dirty * BLOCK_SIZE;
 }

-static int is_stage2_completed(void)
-{
-    int64_t remaining_dirty;
-    long double bwidth;
-
-    if (block_mig_state.bulk_completed == 1) {
-
-        remaining_dirty = get_remaining_dirty();
-        if (remaining_dirty == 0) {
-            return 1;
-        }
-
-        bwidth = compute_read_bwidth();
-
-        if ((remaining_dirty / bwidth) <=
-            migrate_max_downtime()) {
-            /* finish stage2 because we think that we can finish remaining work
-               below max_downtime */
-
-            return 1;
-        }
-    }
-
-    return 0;
-}
-
 static void blk_mig_cleanup(void)
 {
     BlkMigDevState *bmds;
@@ -619,7 +581,7 @@ static int block_save_iterate(QEMUFile *f, void *opaque)

     qemu_put_be64(f, BLK_MIG_FLAG_EOS);

-    return is_stage2_completed();
+    return 0;
 }

 static int block_save_complete(QEMUFile *f, void *opaque)
@@ -659,6 +621,14 @@ static int block_save_complete(QEMUFile *f, void *opaque)
     return 0;
 }

+static uint64_t block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)
+{
+
+    DPRINTF("Enter save live pending  %ld\n", get_remaining_dirty());
+
+    return get_remaining_dirty();
+}
+
 static int block_load(QEMUFile *f, void *opaque, int version_id)
 {
     static int banner_printed;
@@ -755,6 +725,7 @@ SaveVMHandlers savevm_block_handlers = {
     .save_live_setup = block_save_setup,
     .save_live_iterate = block_save_iterate,
     .save_live_complete = block_save_complete,
+    .save_live_pending = block_save_pending,
     .load_state = block_load,
     .cancel = block_migration_cancel,
     .is_active = block_is_active,
diff --git a/buffered_file.c b/buffered_file.c
index 11efd8f..dda9db8 100644
--- a/buffered_file.c
+++ b/buffered_file.c
@@ -182,13 +182,15 @@ static int64_t buffered_get_rate_limit(void *opaque)
     return s->xfer_limit;
 }

-/* 10ms  xfer_limit is the limit that we should write each 10ms */
+/* 100ms  xfer_limit is the limit that we should write each 100ms */
 #define BUFFER_DELAY 100

 static void *buffered_file_thread(void *opaque)
 {
     QEMUFileBuffered *s = opaque;
-    int64_t expire_time = qemu_get_clock_ms(rt_clock) + BUFFER_DELAY;
+    int64_t initial_time = qemu_get_clock_ms(rt_clock);
+    int64_t max_size = 0;
+    bool last_round = false;

     while (true) {
         int64_t current_time = qemu_get_clock_ms(rt_clock);
@@ -196,13 +198,22 @@ static void *buffered_file_thread(void *opaque)
         if (s->migration_state->complete) {
             break;
         }
-        if (current_time >= expire_time) {
+        if (current_time >= initial_time + BUFFER_DELAY) {
+            uint64_t transferred_bytes = s->bytes_xfer;
+            uint64_t time_spent = current_time - initial_time;
+            double bandwidth = transferred_bytes / time_spent;
+            max_size = bandwidth * migrate_max_downtime() / 1000000;
+
+            DPRINTF("transferred %" PRIu64 " time_spent %" PRIu64
+                    " bandwidth %g max_size %" PRId64 "\n",
+                    transferred_bytes, time_spent, bandwidth, max_size);
+
             s->bytes_xfer = 0;
-            expire_time = current_time + BUFFER_DELAY;
+            initial_time = current_time;
         }
-        if (s->bytes_xfer >= s->xfer_limit) {
+        if (!last_round && (s->bytes_xfer >= s->xfer_limit)) {
             /* usleep expects microseconds */
-            g_usleep((expire_time - current_time)*1000);
+            g_usleep((initial_time + BUFFER_DELAY - current_time)*1000);
         }
         if (buffered_flush(s) < 0) {
             break;
@@ -211,7 +222,7 @@ static void *buffered_file_thread(void *opaque)
         DPRINTF("file is ready\n");
         if (s->bytes_xfer < s->xfer_limit) {
             DPRINTF("notifying client\n");
-            migrate_fd_put_ready(s->migration_state);
+            last_round = migrate_fd_put_ready(s->migration_state, max_size);
         }
     }

diff --git a/migration.c b/migration.c
index d6cb320..fa34e19 100644
--- a/migration.c
+++ b/migration.c
@@ -316,15 +316,17 @@ ssize_t migrate_fd_put_buffer(MigrationState *s, const void *data,
     return ret;
 }

-void migrate_fd_put_ready(MigrationState *s)
+bool migrate_fd_put_ready(MigrationState *s, uint64_t max_size)
 {
     int ret;
+    uint64_t pending_size;
+    bool last_round = false;

     qemu_mutex_lock_iothread();
     if (s->state != MIG_STATE_ACTIVE) {
         DPRINTF("put_ready returning because of non-active state\n");
         qemu_mutex_unlock_iothread();
-        return;
+        return false;
     }
     if (s->first_time) {
         s->first_time = false;
@@ -334,15 +336,19 @@ void migrate_fd_put_ready(MigrationState *s)
             DPRINTF("failed, %d\n", ret);
             migrate_fd_error(s);
             qemu_mutex_unlock_iothread();
-            return;
+            return false;
         }
     }

     DPRINTF("iterate\n");
-    ret = qemu_savevm_state_iterate(s->file);
-    if (ret < 0) {
-        migrate_fd_error(s);
-    } else if (ret == 1) {
+    pending_size = qemu_savevm_state_pending(s->file, max_size);
+    DPRINTF("pending size %lu max %lu\n", pending_size, max_size);
+    if (pending_size >= max_size) {
+        ret = qemu_savevm_state_iterate(s->file);
+        if (ret < 0) {
+            migrate_fd_error(s);
+        }
+    } else {
         int old_vm_running = runstate_is_running();
         int64_t start_time, end_time;

@@ -368,9 +374,11 @@ void migrate_fd_put_ready(MigrationState *s)
                 vm_start();
             }
         }
+        last_round = true;
     }
     qemu_mutex_unlock_iothread();

+    return last_round;
 }

 static void migrate_fd_cancel(MigrationState *s)
diff --git a/migration.h b/migration.h
index 6760d7f..92f658d 100644
--- a/migration.h
+++ b/migration.h
@@ -81,7 +81,7 @@ void migrate_fd_connect(MigrationState *s);

 ssize_t migrate_fd_put_buffer(MigrationState *s, const void *data,
                               size_t size);
-void migrate_fd_put_ready(MigrationState *s);
+bool migrate_fd_put_ready(MigrationState *s, uint64_t max_size);
 int migrate_fd_close(MigrationState *s);

 void add_migration_state_change_notifier(Notifier *notify);
diff --git a/savevm.c b/savevm.c
index c4ee899..f6e91cf 100644
--- a/savevm.c
+++ b/savevm.c
@@ -1754,6 +1754,25 @@ int qemu_savevm_state_complete(QEMUFile *f)
     return qemu_file_get_error(f);
 }

+uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size)
+{
+    SaveStateEntry *se;
+    uint64_t ret = 0;
+
+    QTAILQ_FOREACH(se, &savevm_handlers, entry) {
+        if (!se->ops || !se->ops->save_live_pending) {
+            continue;
+        }
+        if (se->ops && se->ops->is_active) {
+            if (!se->ops->is_active(se->opaque)) {
+                continue;
+            }
+        }
+        ret += se->ops->save_live_pending(f, se->opaque, max_size);
+    }
+    return ret;
+}
+
 void qemu_savevm_state_cancel(QEMUFile *f)
 {
     SaveStateEntry *se;
diff --git a/sysemu.h b/sysemu.h
index 1b6add2..7832c69 100644
--- a/sysemu.h
+++ b/sysemu.h
@@ -78,6 +78,7 @@ int qemu_savevm_state_begin(QEMUFile *f,
 int qemu_savevm_state_iterate(QEMUFile *f);
 int qemu_savevm_state_complete(QEMUFile *f);
 void qemu_savevm_state_cancel(QEMUFile *f);
+uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size);
 int qemu_loadvm_state(QEMUFile *f);

 /* SLIRP */
diff --git a/vmstate.h b/vmstate.h
index 623af0a..f27276c 100644
--- a/vmstate.h
+++ b/vmstate.h
@@ -35,6 +35,7 @@ typedef struct SaveVMHandlers {
     int (*save_live_setup)(QEMUFile *f, void *opaque);
     int (*save_live_iterate)(QEMUFile *f, void *opaque);
     int (*save_live_complete)(QEMUFile *f, void *opaque);
+    uint64_t (*save_live_pending)(QEMUFile *f, void *opaque, uint64_t max_size);
     void (*cancel)(void *opaque);
     LoadStateHandler *load_state;
     bool (*is_active)(void *opaque);
-- 
1.7.11.7

[Qemu-devel] [PATCH 21/34] migration: move buffered_file.c code into migration.c

[Juan Quintela]

This only moves the code (also from buffered_file.h to migration.h).
Fix whitespace until checkpatch is happy.

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 Makefile.objs   |   2 +-
 buffered_file.c | 259 --------------------------------------------------------
 buffered_file.h |  22 -----
 migration.c     | 233 +++++++++++++++++++++++++++++++++++++++++++++++++-
 migration.h     |   1 +
 5 files changed, 234 insertions(+), 283 deletions(-)
 delete mode 100644 buffered_file.c
 delete mode 100644 buffered_file.h

diff --git a/Makefile.objs b/Makefile.objs
index 3c7abca..f0309ac 100644
--- a/Makefile.objs
+++ b/Makefile.objs
@@ -79,7 +79,7 @@ extra-obj-$(CONFIG_LINUX) += fsdev/

 common-obj-y += tcg-runtime.o host-utils.o main-loop.o
 common-obj-y += input.o
-common-obj-y += buffered_file.o migration.o migration-tcp.o
+common-obj-y += migration.o migration-tcp.o
 common-obj-y += qemu-char.o #aio.o
 common-obj-y += block-migration.o iohandler.o
 common-obj-y += bitmap.o bitops.o
diff --git a/buffered_file.c b/buffered_file.c
deleted file mode 100644
index dda9db8..0000000
--- a/buffered_file.c
+++ /dev/null
@@ -1,259 +0,0 @@
-/*
- * QEMU buffered QEMUFile
- *
- * Copyright IBM, Corp. 2008
- *
- * Authors:
- *  Anthony Liguori   <aliguori@us.ibm.com>
- *
- * This work is licensed under the terms of the GNU GPL, version 2.  See
- * the COPYING file in the top-level directory.
- *
- * Contributions after 2012-01-13 are licensed under the terms of the
- * GNU GPL, version 2 or (at your option) any later version.
- */
-
-#include "qemu-common.h"
-#include "hw/hw.h"
-#include "qemu-timer.h"
-#include "qemu-char.h"
-#include "buffered_file.h"
-#include "qemu-thread.h"
-
-//#define DEBUG_BUFFERED_FILE
-
-typedef struct QEMUFileBuffered
-{
-    MigrationState *migration_state;
-    QEMUFile *file;
-    size_t bytes_xfer;
-    size_t xfer_limit;
-    uint8_t *buffer;
-    size_t buffer_size;
-    size_t buffer_capacity;
-    QemuThread thread;
-} QEMUFileBuffered;
-
-#ifdef DEBUG_BUFFERED_FILE
-#define DPRINTF(fmt, ...) \
-    do { printf("buffered-file: " fmt, ## __VA_ARGS__); } while (0)
-#else
-#define DPRINTF(fmt, ...) \
-    do { } while (0)
-#endif
-
-static ssize_t buffered_flush(QEMUFileBuffered *s)
-{
-    size_t offset = 0;
-    ssize_t ret = 0;
-
-    DPRINTF("flushing %zu byte(s) of data\n", s->buffer_size);
-
-    while (s->bytes_xfer < s->xfer_limit && offset < s->buffer_size) {
-        size_t to_send = MIN(s->buffer_size - offset, s->xfer_limit - s->bytes_xfer);
-        ret = migrate_fd_put_buffer(s->migration_state, s->buffer + offset,
-                                    to_send);
-        if (ret <= 0) {
-            DPRINTF("error flushing data, %zd\n", ret);
-            break;
-        } else {
-            DPRINTF("flushed %zd byte(s)\n", ret);
-            offset += ret;
-            s->bytes_xfer += ret;
-        }
-    }
-
-    DPRINTF("flushed %zu of %zu byte(s)\n", offset, s->buffer_size);
-    memmove(s->buffer, s->buffer + offset, s->buffer_size - offset);
-    s->buffer_size -= offset;
-
-    if (ret < 0) {
-        return ret;
-    }
-    return offset;
-}
-
-static int buffered_put_buffer(void *opaque, const uint8_t *buf, int64_t pos, int size)
-{
-    QEMUFileBuffered *s = opaque;
-    ssize_t error;
-
-    DPRINTF("putting %d bytes at %" PRId64 "\n", size, pos);
-
-    error = qemu_file_get_error(s->file);
-    if (error) {
-        DPRINTF("flush when error, bailing: %s\n", strerror(-error));
-        return error;
-    }
-
-    if (size <= 0) {
-        return size;
-    }
-
-    if (size > (s->buffer_capacity - s->buffer_size)) {
-        DPRINTF("increasing buffer capacity from %zu by %zu\n",
-                s->buffer_capacity, size + 1024);
-
-        s->buffer_capacity += size + 1024;
-
-        s->buffer = g_realloc(s->buffer, s->buffer_capacity);
-    }
-
-    memcpy(s->buffer + s->buffer_size, buf, size);
-    s->buffer_size += size;
-
-    return size;
-}
-
-static int buffered_close(void *opaque)
-{
-    QEMUFileBuffered *s = opaque;
-    ssize_t ret = 0;
-    int ret2;
-
-    DPRINTF("closing\n");
-
-    s->xfer_limit = INT_MAX;
-    while (!qemu_file_get_error(s->file) && s->buffer_size) {
-        ret = buffered_flush(s);
-        if (ret < 0) {
-            break;
-        }
-    }
-
-    ret2 = migrate_fd_close(s->migration_state);
-    if (ret >= 0) {
-        ret = ret2;
-    }
-    ret = migrate_fd_close(s->migration_state);
-    s->migration_state->complete = true;
-    return ret;
-}
-
-/*
- * The meaning of the return values is:
- *   0: We can continue sending
- *   1: Time to stop
- *   negative: There has been an error
- */
-static int buffered_get_fd(void *opaque)
-{
-    QEMUFileBuffered *s = opaque;
-
-    return qemu_get_fd(s->file);
-}
-
-static int buffered_rate_limit(void *opaque)
-{
-    QEMUFileBuffered *s = opaque;
-    int ret;
-
-    ret = qemu_file_get_error(s->file);
-    if (ret) {
-        return ret;
-    }
-
-    if (s->bytes_xfer > s->xfer_limit)
-        return 1;
-
-    return 0;
-}
-
-static int64_t buffered_set_rate_limit(void *opaque, int64_t new_rate)
-{
-    QEMUFileBuffered *s = opaque;
-    if (qemu_file_get_error(s->file)) {
-        goto out;
-    }
-    if (new_rate > SIZE_MAX) {
-        new_rate = SIZE_MAX;
-    }
-
-    s->xfer_limit = new_rate / 10;
-    
-out:
-    return s->xfer_limit;
-}
-
-static int64_t buffered_get_rate_limit(void *opaque)
-{
-    QEMUFileBuffered *s = opaque;
-  
-    return s->xfer_limit;
-}
-
-/* 100ms  xfer_limit is the limit that we should write each 100ms */
-#define BUFFER_DELAY 100
-
-static void *buffered_file_thread(void *opaque)
-{
-    QEMUFileBuffered *s = opaque;
-    int64_t initial_time = qemu_get_clock_ms(rt_clock);
-    int64_t max_size = 0;
-    bool last_round = false;
-
-    while (true) {
-        int64_t current_time = qemu_get_clock_ms(rt_clock);
-
-        if (s->migration_state->complete) {
-            break;
-        }
-        if (current_time >= initial_time + BUFFER_DELAY) {
-            uint64_t transferred_bytes = s->bytes_xfer;
-            uint64_t time_spent = current_time - initial_time;
-            double bandwidth = transferred_bytes / time_spent;
-            max_size = bandwidth * migrate_max_downtime() / 1000000;
-
-            DPRINTF("transferred %" PRIu64 " time_spent %" PRIu64
-                    " bandwidth %g max_size %" PRId64 "\n",
-                    transferred_bytes, time_spent, bandwidth, max_size);
-
-            s->bytes_xfer = 0;
-            initial_time = current_time;
-        }
-        if (!last_round && (s->bytes_xfer >= s->xfer_limit)) {
-            /* usleep expects microseconds */
-            g_usleep((initial_time + BUFFER_DELAY - current_time)*1000);
-        }
-        if (buffered_flush(s) < 0) {
-            break;
-        }
-
-        DPRINTF("file is ready\n");
-        if (s->bytes_xfer < s->xfer_limit) {
-            DPRINTF("notifying client\n");
-            last_round = migrate_fd_put_ready(s->migration_state, max_size);
-        }
-    }
-
-    g_free(s->buffer);
-    g_free(s);
-    return NULL;
-}
-
-static const QEMUFileOps buffered_file_ops = {
-    .get_fd =         buffered_get_fd,
-    .put_buffer =     buffered_put_buffer,
-    .close =          buffered_close,
-    .rate_limit =     buffered_rate_limit,
-    .get_rate_limit = buffered_get_rate_limit,
-    .set_rate_limit = buffered_set_rate_limit,
-};
-
-void qemu_fopen_ops_buffered(MigrationState *migration_state)
-{
-    QEMUFileBuffered *s;
-
-    s = g_malloc0(sizeof(*s));
-
-    s->migration_state = migration_state;
-    s->xfer_limit = migration_state->bandwidth_limit / 10;
-    s->migration_state->complete = false;
-
-    s->file = qemu_fopen_ops(s, &buffered_file_ops);
-
-    migration_state->file = s->file;
-
-    qemu_thread_create(&s->thread, buffered_file_thread, s,
-                       QEMU_THREAD_DETACHED);
-}
diff --git a/buffered_file.h b/buffered_file.h
deleted file mode 100644
index 8a246fd..0000000
--- a/buffered_file.h
+++ /dev/null
@@ -1,22 +0,0 @@
-/*
- * QEMU buffered QEMUFile
- *
- * Copyright IBM, Corp. 2008
- *
- * Authors:
- *  Anthony Liguori   <aliguori@us.ibm.com>
- *
- * This work is licensed under the terms of the GNU GPL, version 2.  See
- * the COPYING file in the top-level directory.
- *
- */
-
-#ifndef QEMU_BUFFERED_FILE_H
-#define QEMU_BUFFERED_FILE_H
-
-#include "hw/hw.h"
-#include "migration.h"
-
-void qemu_fopen_ops_buffered(MigrationState *migration_state);
-
-#endif
diff --git a/migration.c b/migration.c
index fa34e19..26bb85e 100644
--- a/migration.c
+++ b/migration.c
@@ -16,7 +16,7 @@
 #include "qemu-common.h"
 #include "migration.h"
 #include "monitor.h"
-#include "buffered_file.h"
+#include "qemu-file.h"
 #include "sysemu.h"
 #include "block.h"
 #include "qemu_socket.h"
@@ -587,3 +587,234 @@ int64_t migrate_xbzrle_cache_size(void)

     return s->xbzrle_cache_size;
 }
+
+/* migration thread support */
+
+typedef struct QEMUFileBuffered {
+    MigrationState *migration_state;
+    QEMUFile *file;
+    size_t bytes_xfer;
+    size_t xfer_limit;
+    uint8_t *buffer;
+    size_t buffer_size;
+    size_t buffer_capacity;
+    QemuThread thread;
+} QEMUFileBuffered;
+
+static ssize_t buffered_flush(QEMUFileBuffered *s)
+{
+    size_t offset = 0;
+    ssize_t ret = 0;
+
+    DPRINTF("flushing %zu byte(s) of data\n", s->buffer_size);
+
+    while (s->bytes_xfer < s->xfer_limit && offset < s->buffer_size) {
+        size_t to_send = MIN(s->buffer_size - offset, s->xfer_limit - s->bytes_xfer);
+        ret = migrate_fd_put_buffer(s->migration_state, s->buffer + offset,
+                                    to_send);
+        if (ret <= 0) {
+            DPRINTF("error flushing data, %zd\n", ret);
+            break;
+        } else {
+            DPRINTF("flushed %zd byte(s)\n", ret);
+            offset += ret;
+            s->bytes_xfer += ret;
+        }
+    }
+
+    DPRINTF("flushed %zu of %zu byte(s)\n", offset, s->buffer_size);
+    memmove(s->buffer, s->buffer + offset, s->buffer_size - offset);
+    s->buffer_size -= offset;
+
+    if (ret < 0) {
+        return ret;
+    }
+    return offset;
+}
+
+static int buffered_put_buffer(void *opaque, const uint8_t *buf,
+                               int64_t pos, int size)
+{
+    QEMUFileBuffered *s = opaque;
+    ssize_t error;
+
+    DPRINTF("putting %d bytes at %" PRId64 "\n", size, pos);
+
+    error = qemu_file_get_error(s->file);
+    if (error) {
+        DPRINTF("flush when error, bailing: %s\n", strerror(-error));
+        return error;
+    }
+
+    if (size <= 0) {
+        return size;
+    }
+
+    if (size > (s->buffer_capacity - s->buffer_size)) {
+        DPRINTF("increasing buffer capacity from %zu by %zu\n",
+                s->buffer_capacity, size + 1024);
+
+        s->buffer_capacity += size + 1024;
+
+        s->buffer = g_realloc(s->buffer, s->buffer_capacity);
+    }
+
+    memcpy(s->buffer + s->buffer_size, buf, size);
+    s->buffer_size += size;
+
+    return size;
+}
+
+static int buffered_close(void *opaque)
+{
+    QEMUFileBuffered *s = opaque;
+    ssize_t ret = 0;
+    int ret2;
+
+    DPRINTF("closing\n");
+
+    s->xfer_limit = INT_MAX;
+    while (!qemu_file_get_error(s->file) && s->buffer_size) {
+        ret = buffered_flush(s);
+        if (ret < 0) {
+            break;
+        }
+    }
+
+    ret2 = migrate_fd_close(s->migration_state);
+    if (ret >= 0) {
+        ret = ret2;
+    }
+    ret = migrate_fd_close(s->migration_state);
+    s->migration_state->complete = true;
+    return ret;
+}
+
+static int buffered_get_fd(void *opaque)
+{
+    QEMUFileBuffered *s = opaque;
+
+    return qemu_get_fd(s->file);
+}
+
+/*
+ * The meaning of the return values is:
+ *   0: We can continue sending
+ *   1: Time to stop
+ *   negative: There has been an error
+ */
+static int buffered_rate_limit(void *opaque)
+{
+    QEMUFileBuffered *s = opaque;
+    int ret;
+
+    ret = qemu_file_get_error(s->file);
+    if (ret) {
+        return ret;
+    }
+
+    if (s->bytes_xfer > s->xfer_limit) {
+        return 1;
+    }
+
+    return 0;
+}
+
+static int64_t buffered_set_rate_limit(void *opaque, int64_t new_rate)
+{
+    QEMUFileBuffered *s = opaque;
+    if (qemu_file_get_error(s->file)) {
+        goto out;
+    }
+    if (new_rate > SIZE_MAX) {
+        new_rate = SIZE_MAX;
+    }
+
+    s->xfer_limit = new_rate / 10;
+
+out:
+    return s->xfer_limit;
+}
+
+static int64_t buffered_get_rate_limit(void *opaque)
+{
+    QEMUFileBuffered *s = opaque;
+
+    return s->xfer_limit;
+}
+
+/* 100ms  xfer_limit is the limit that we should write each 100ms */
+#define BUFFER_DELAY 100
+
+static void *buffered_file_thread(void *opaque)
+{
+    QEMUFileBuffered *s = opaque;
+    int64_t initial_time = qemu_get_clock_ms(rt_clock);
+    int64_t max_size = 0;
+    bool last_round = false;
+
+    while (true) {
+        int64_t current_time = qemu_get_clock_ms(rt_clock);
+
+        if (s->migration_state->complete) {
+            break;
+        }
+        if (current_time >= initial_time + BUFFER_DELAY) {
+            uint64_t transferred_bytes = s->bytes_xfer;
+            uint64_t time_spent = current_time - initial_time;
+            double bandwidth = transferred_bytes / time_spent;
+            max_size = bandwidth * migrate_max_downtime() / 1000000;
+
+            DPRINTF("transferred %" PRIu64 " time_spent %" PRIu64
+                    " bandwidth %g max_size %" PRId64 "\n",
+                    transferred_bytes, time_spent, bandwidth, max_size);
+
+            s->bytes_xfer = 0;
+            initial_time = current_time;
+        }
+        if (!last_round && (s->bytes_xfer >= s->xfer_limit)) {
+            /* usleep expects microseconds */
+            g_usleep((initial_time + BUFFER_DELAY - current_time)*1000);
+        }
+        if (buffered_flush(s) < 0) {
+            break;
+        }
+
+        DPRINTF("file is ready\n");
+        if (s->bytes_xfer < s->xfer_limit) {
+            DPRINTF("notifying client\n");
+            last_round = migrate_fd_put_ready(s->migration_state, max_size);
+        }
+    }
+
+    g_free(s->buffer);
+    g_free(s);
+    return NULL;
+}
+
+static const QEMUFileOps buffered_file_ops = {
+    .get_fd =         buffered_get_fd,
+    .put_buffer =     buffered_put_buffer,
+    .close =          buffered_close,
+    .rate_limit =     buffered_rate_limit,
+    .get_rate_limit = buffered_get_rate_limit,
+    .set_rate_limit = buffered_set_rate_limit,
+};
+
+void qemu_fopen_ops_buffered(MigrationState *migration_state)
+{
+    QEMUFileBuffered *s;
+
+    s = g_malloc0(sizeof(*s));
+
+    s->migration_state = migration_state;
+    s->xfer_limit = migration_state->bandwidth_limit / 10;
+    s->migration_state->complete = false;
+
+    s->file = qemu_fopen_ops(s, &buffered_file_ops);
+
+    migration_state->file = s->file;
+
+    qemu_thread_create(&s->thread, buffered_file_thread, s,
+                       QEMU_THREAD_DETACHED);
+}
diff --git a/migration.h b/migration.h
index 92f658d..31ed7af 100644
--- a/migration.h
+++ b/migration.h
@@ -129,4 +129,5 @@ int64_t migrate_xbzrle_cache_size(void);

 int64_t xbzrle_cache_resize(int64_t new_size);

+void qemu_fopen_ops_buffered(MigrationState *migration_state);
 #endif
-- 
1.7.11.7

[Qemu-devel] [PATCH 22/34] migration: add XFER_LIMIT_RATIO

[Juan Quintela]

The "magic" divisions by 10 are there because of the value of BUFFER_DELAY.
Introduce a constant to explain them better.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/migration.c b/migration.c
index 26bb85e..a233215 100644
--- a/migration.c
+++ b/migration.c
@@ -44,6 +44,11 @@ enum {

 #define MAX_THROTTLE  (32 << 20)      /* Migration speed throttling */

+/* Amount of time to allocate to each "chunk" of bandwidth-throttled
+ * data. */
+#define BUFFER_DELAY     100
+#define XFER_LIMIT_RATIO (1000 / BUFFER_DELAY)
+
 /* Migration XBZRLE default cache size */
 #define DEFAULT_MIGRATE_CACHE_SIZE (64 * 1024 * 1024)

@@ -743,9 +748,6 @@ static int64_t buffered_get_rate_limit(void *opaque)
     return s->xfer_limit;
 }

-/* 100ms  xfer_limit is the limit that we should write each 100ms */
-#define BUFFER_DELAY 100
-
 static void *buffered_file_thread(void *opaque)
 {
     QEMUFileBuffered *s = opaque;
@@ -808,7 +810,7 @@ void qemu_fopen_ops_buffered(MigrationState *migration_state)
     s = g_malloc0(sizeof(*s));

     s->migration_state = migration_state;
-    s->xfer_limit = migration_state->bandwidth_limit / 10;
+    s->xfer_limit = s->migration_state->bandwidth_limit / XFER_LIMIT_RATIO;
     s->migration_state->complete = false;

     s->file = qemu_fopen_ops(s, &buffered_file_ops);
-- 
1.7.11.7

[Qemu-devel] [PATCH 23/34] migration: move migration_fd_put_ready()

[Juan Quintela]

Put it near its use and un-export it.

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration.c | 130 ++++++++++++++++++++++++++++++------------------------------
 migration.h |   1 -
 2 files changed, 65 insertions(+), 66 deletions(-)

diff --git a/migration.c b/migration.c
index a233215..7a27987 100644
--- a/migration.c
+++ b/migration.c
@@ -321,71 +321,6 @@ ssize_t migrate_fd_put_buffer(MigrationState *s, const void *data,
     return ret;
 }

-bool migrate_fd_put_ready(MigrationState *s, uint64_t max_size)
-{
-    int ret;
-    uint64_t pending_size;
-    bool last_round = false;
-
-    qemu_mutex_lock_iothread();
-    if (s->state != MIG_STATE_ACTIVE) {
-        DPRINTF("put_ready returning because of non-active state\n");
-        qemu_mutex_unlock_iothread();
-        return false;
-    }
-    if (s->first_time) {
-        s->first_time = false;
-        DPRINTF("beginning savevm\n");
-        ret = qemu_savevm_state_begin(s->file, &s->params);
-        if (ret < 0) {
-            DPRINTF("failed, %d\n", ret);
-            migrate_fd_error(s);
-            qemu_mutex_unlock_iothread();
-            return false;
-        }
-    }
-
-    DPRINTF("iterate\n");
-    pending_size = qemu_savevm_state_pending(s->file, max_size);
-    DPRINTF("pending size %lu max %lu\n", pending_size, max_size);
-    if (pending_size >= max_size) {
-        ret = qemu_savevm_state_iterate(s->file);
-        if (ret < 0) {
-            migrate_fd_error(s);
-        }
-    } else {
-        int old_vm_running = runstate_is_running();
-        int64_t start_time, end_time;
-
-        DPRINTF("done iterating\n");
-        start_time = qemu_get_clock_ms(rt_clock);
-        qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
-        if (old_vm_running) {
-            vm_stop(RUN_STATE_FINISH_MIGRATE);
-        } else {
-            vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
-        }
-
-        if (qemu_savevm_state_complete(s->file) < 0) {
-            migrate_fd_error(s);
-        } else {
-            migrate_fd_completed(s);
-        }
-        end_time = qemu_get_clock_ms(rt_clock);
-        s->total_time = end_time - s->total_time;
-        s->downtime = end_time - start_time;
-        if (s->state != MIG_STATE_COMPLETED) {
-            if (old_vm_running) {
-                vm_start();
-            }
-        }
-        last_round = true;
-    }
-    qemu_mutex_unlock_iothread();
-
-    return last_round;
-}
-
 static void migrate_fd_cancel(MigrationState *s)
 {
     if (s->state != MIG_STATE_ACTIVE)
@@ -748,6 +683,71 @@ static int64_t buffered_get_rate_limit(void *opaque)
     return s->xfer_limit;
 }

+static bool migrate_fd_put_ready(MigrationState *s, uint64_t max_size)
+{
+    int ret;
+    uint64_t pending_size;
+    bool last_round = false;
+
+    qemu_mutex_lock_iothread();
+    if (s->state != MIG_STATE_ACTIVE) {
+        DPRINTF("put_ready returning because of non-active state\n");
+        qemu_mutex_unlock_iothread();
+        return false;
+    }
+    if (s->first_time) {
+        s->first_time = false;
+        DPRINTF("beginning savevm\n");
+        ret = qemu_savevm_state_begin(s->file, &s->params);
+        if (ret < 0) {
+            DPRINTF("failed, %d\n", ret);
+            migrate_fd_error(s);
+            qemu_mutex_unlock_iothread();
+            return false;
+        }
+    }
+
+    DPRINTF("iterate\n");
+    pending_size = qemu_savevm_state_pending(s->file, max_size);
+    DPRINTF("pending size %lu max %lu\n", pending_size, max_size);
+    if (pending_size >= max_size) {
+        ret = qemu_savevm_state_iterate(s->file);
+        if (ret < 0) {
+            migrate_fd_error(s);
+        }
+    } else {
+        int old_vm_running = runstate_is_running();
+        int64_t start_time, end_time;
+
+        DPRINTF("done iterating\n");
+        start_time = qemu_get_clock_ms(rt_clock);
+        qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
+        if (old_vm_running) {
+            vm_stop(RUN_STATE_FINISH_MIGRATE);
+        } else {
+            vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
+        }
+
+        if (qemu_savevm_state_complete(s->file) < 0) {
+            migrate_fd_error(s);
+        } else {
+            migrate_fd_completed(s);
+        }
+        end_time = qemu_get_clock_ms(rt_clock);
+        s->total_time = end_time - s->total_time;
+        s->downtime = end_time - start_time;
+        if (s->state != MIG_STATE_COMPLETED) {
+            if (old_vm_running) {
+                vm_start();
+            }
+        }
+        last_round = true;
+    }
+    qemu_mutex_unlock_iothread();
+
+    return last_round;
+}
+
 static void *buffered_file_thread(void *opaque)
 {
     QEMUFileBuffered *s = opaque;
diff --git a/migration.h b/migration.h
index 31ed7af..705a331 100644
--- a/migration.h
+++ b/migration.h
@@ -81,7 +81,6 @@ void migrate_fd_connect(MigrationState *s);

 ssize_t migrate_fd_put_buffer(MigrationState *s, const void *data,
                               size_t size);
-bool migrate_fd_put_ready(MigrationState *s, uint64_t max_size);
 int migrate_fd_close(MigrationState *s);

 void add_migration_state_change_notifier(Notifier *notify);
-- 
1.7.11.7

[Qemu-devel] [PATCH 24/34] migration: Inline qemu_fopen_ops_buffered into migrate_fd_connect

[Juan Quintela]

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration.c | 11 +++--------
 migration.h |  2 --
 2 files changed, 3 insertions(+), 10 deletions(-)

diff --git a/migration.c b/migration.c
index 7a27987..5a4ad74 100644
--- a/migration.c
+++ b/migration.c
@@ -371,13 +371,6 @@ bool migration_has_failed(MigrationState *s)
             s->state == MIG_STATE_ERROR);
 }

-void migrate_fd_connect(MigrationState *s)
-{
-    s->state = MIG_STATE_ACTIVE;
-    s->first_time = true;
-    qemu_fopen_ops_buffered(s);
-}
-
 static MigrationState *migrate_init(const MigrationParams *params)
 {
     MigrationState *s = migrate_get_current();
@@ -803,10 +796,12 @@ static const QEMUFileOps buffered_file_ops = {
     .set_rate_limit = buffered_set_rate_limit,
 };

-void qemu_fopen_ops_buffered(MigrationState *migration_state)
+void migrate_fd_connect(MigrationState *migration_state)
 {
     QEMUFileBuffered *s;

+    migration_state->state = MIG_STATE_ACTIVE;
+    migration_state->first_time = true;
     s = g_malloc0(sizeof(*s));

     s->migration_state = migration_state;
diff --git a/migration.h b/migration.h
index 705a331..542197e 100644
--- a/migration.h
+++ b/migration.h
@@ -127,6 +127,4 @@ int migrate_use_xbzrle(void);
 int64_t migrate_xbzrle_cache_size(void);

 int64_t xbzrle_cache_resize(int64_t new_size);
-
-void qemu_fopen_ops_buffered(MigrationState *migration_state);
 #endif
-- 
1.7.11.7

[Qemu-devel] [PATCH 25/34] migration: move migration notifier

[Juan Quintela]

At this point, it is waranteed that state is ACTIVE.  Old position
didn't assured hat.

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/migration.c b/migration.c
index 5a4ad74..cabf17b 100644
--- a/migration.c
+++ b/migration.c
@@ -455,8 +455,6 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
         error_propagate(errp, local_err);
         return;
     }
-
-    notifier_list_notify(&migration_state_notifiers, s);
 }

 void qmp_migrate_cancel(Error **errp)
@@ -814,4 +812,5 @@ void migrate_fd_connect(MigrationState *migration_state)

     qemu_thread_create(&s->thread, buffered_file_thread, s,
                        QEMU_THREAD_DETACHED);
+    notifier_list_notify(&migration_state_notifiers, s);
 }
-- 
1.7.11.7

[Qemu-devel] [PATCH 26/34] ram: rename last_block to last_seen_block

[Juan Quintela]

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 arch_init.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)

diff --git a/arch_init.c b/arch_init.c
index f092ea2..34a2f4b 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -332,7 +332,10 @@ static int save_xbzrle_page(QEMUFile *f, uint8_t *current_data,
     return bytes_sent;
 }

-static RAMBlock *last_block;
+
+/* This is the last block that we have visited serching for dirty pages
+ */
+static RAMBlock *last_seen_block;
 static ram_addr_t last_offset;
 static unsigned long *migration_bitmap;
 static uint64_t migration_dirty_pages;
@@ -417,7 +420,7 @@ static void migration_bitmap_sync(void)

 static int ram_save_block(QEMUFile *f, bool last_stage)
 {
-    RAMBlock *block = last_block;
+    RAMBlock *block = last_seen_block;
     ram_addr_t offset = last_offset;
     int bytes_sent = -1;
     MemoryRegion *mr;
@@ -430,7 +433,8 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
         mr = block->mr;
         if (migration_bitmap_test_and_reset_dirty(mr, offset)) {
             uint8_t *p;
-            int cont = (block == last_block) ? RAM_SAVE_FLAG_CONTINUE : 0;
+            int cont = (block == last_seen_block) ?
+                RAM_SAVE_FLAG_CONTINUE : 0;

             p = memory_region_get_ram_ptr(mr) + offset;

@@ -469,9 +473,9 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
             if (!block)
                 block = QTAILQ_FIRST(&ram_list.blocks);
         }
-    } while (block != last_block || offset != last_offset);
+    } while (block != last_seen_block || offset != last_offset);

-    last_block = block;
+    last_seen_block = block;
     last_offset = offset;

     return bytes_sent;
@@ -530,7 +534,7 @@ static void ram_migration_cancel(void *opaque)

 static void reset_ram_globals(void)
 {
-    last_block = NULL;
+    last_seen_block = NULL;
     last_offset = 0;
     last_version = ram_list.version;
 }
-- 
1.7.11.7

[Qemu-devel] [PATCH 27/34] ram: Add last_sent_block

[Juan Quintela]

This is the last block from where we have sent data.

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 arch_init.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/arch_init.c b/arch_init.c
index 34a2f4b..af57cdb 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -336,6 +336,8 @@ static int save_xbzrle_page(QEMUFile *f, uint8_t *current_data,
 /* This is the last block that we have visited serching for dirty pages
  */
 static RAMBlock *last_seen_block;
+/* This is the last block from where we have sent data */
+static RAMBlock *last_sent_block;
 static ram_addr_t last_offset;
 static unsigned long *migration_bitmap;
 static uint64_t migration_dirty_pages;
@@ -433,7 +435,7 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
         mr = block->mr;
         if (migration_bitmap_test_and_reset_dirty(mr, offset)) {
             uint8_t *p;
-            int cont = (block == last_seen_block) ?
+            int cont = (block == last_sent_block) ?
                 RAM_SAVE_FLAG_CONTINUE : 0;

             p = memory_region_get_ram_ptr(mr) + offset;
@@ -462,6 +464,7 @@ static int ram_save_block(QEMUFile *f, bool last_stage)

             /* if page is unmodified, continue to the next */
             if (bytes_sent != 0) {
+                last_sent_block = block;
                 break;
             }
         }
@@ -535,6 +538,7 @@ static void ram_migration_cancel(void *opaque)
 static void reset_ram_globals(void)
 {
     last_seen_block = NULL;
+    last_sent_block = NULL;
     last_offset = 0;
     last_version = ram_list.version;
 }
-- 
1.7.11.7

[Qemu-devel] [PATCH 28/34] memory: introduce memory_region_test_and_clear_dirty

[Juan Quintela]

This function avoids having to do two calls, one to test the dirty bit, and
other to reset it.

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 memory.c | 16 ++++++++++++++++
 memory.h | 16 ++++++++++++++++
 2 files changed, 32 insertions(+)

diff --git a/memory.c b/memory.c
index 7419853..c72a5e2 100644
--- a/memory.c
+++ b/memory.c
@@ -1081,6 +1081,22 @@ void memory_region_set_dirty(MemoryRegion *mr, hwaddr addr,
     return cpu_physical_memory_set_dirty_range(mr->ram_addr + addr, size, -1);
 }

+bool memory_region_test_and_clear_dirty(MemoryRegion *mr, hwaddr addr,
+                                        hwaddr size, unsigned client)
+{
+    bool ret;
+    assert(mr->terminates);
+    ret = cpu_physical_memory_get_dirty(mr->ram_addr + addr, size,
+                                        1 << client);
+    if (ret) {
+        cpu_physical_memory_reset_dirty(mr->ram_addr + addr,
+                                        mr->ram_addr + addr + size,
+                                        1 << client);
+    }
+    return ret;
+}
+
+
 void memory_region_sync_dirty_bitmap(MemoryRegion *mr)
 {
     AddressSpace *as;
diff --git a/memory.h b/memory.h
index 9462bfd..bc63a87 100644
--- a/memory.h
+++ b/memory.h
@@ -454,6 +454,22 @@ void memory_region_set_dirty(MemoryRegion *mr, hwaddr addr,
                              hwaddr size);

 /**
+ * memory_region_test_and_clear_dirty: Check whether a range of bytes is dirty
+ *                                     for a specified client. It clears them.
+ *
+ * Checks whether a range of bytes has been written to since the last
+ * call to memory_region_reset_dirty() with the same @client.  Dirty logging
+ * must be enabled.
+ *
+ * @mr: the memory region being queried.
+ * @addr: the address (relative to the start of the region) being queried.
+ * @size: the size of the range being queried.
+ * @client: the user of the logging information; %DIRTY_MEMORY_MIGRATION or
+ *          %DIRTY_MEMORY_VGA.
+ */
+bool memory_region_test_and_clear_dirty(MemoryRegion *mr, hwaddr addr,
+                                        hwaddr size, unsigned client);
+/**
  * memory_region_sync_dirty_bitmap: Synchronize a region's dirty bitmap with
  *                                  any external TLBs (e.g. kvm)
  *
-- 
1.7.11.7

[Qemu-devel] [PATCH 29/34] ram: Use memory_region_test_and_clear_dirty

[Juan Quintela]

This avoids having to do two walks over the dirty bitmap, once reading
the dirty bits, and anthoer cleaning them.

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 arch_init.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/arch_init.c b/arch_init.c
index af57cdb..562f721 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -390,13 +390,12 @@ static void migration_bitmap_sync(void)

     QTAILQ_FOREACH(block, &ram_list.blocks, next) {
         for (addr = 0; addr < block->length; addr += TARGET_PAGE_SIZE) {
-            if (memory_region_get_dirty(block->mr, addr, TARGET_PAGE_SIZE,
-                                        DIRTY_MEMORY_MIGRATION)) {
+            if (memory_region_test_and_clear_dirty(block->mr,
+                                                   addr, TARGET_PAGE_SIZE,
+                                                   DIRTY_MEMORY_MIGRATION)) {
                 migration_bitmap_set_dirty(block->mr, addr);
             }
         }
-        memory_region_reset_dirty(block->mr, 0, block->length,
-                                  DIRTY_MEMORY_MIGRATION);
     }
     trace_migration_bitmap_sync_end(migration_dirty_pages
                                     - num_dirty_pages_init);
-- 
1.7.11.7

[Qemu-devel] [PATCH 30/34] ram: optimize migration bitmap walking

[Juan Quintela]

Instead of testing each page individually, we search what is the next
dirty page with a bitmap operation.  We have to reorganize the code to
move from a "for" loop, to a while(dirty) loop.

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 arch_init.c | 45 ++++++++++++++++++++++++++-------------------
 1 file changed, 26 insertions(+), 19 deletions(-)

diff --git a/arch_init.c b/arch_init.c
index 562f721..978fe37 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -343,18 +343,21 @@ static unsigned long *migration_bitmap;
 static uint64_t migration_dirty_pages;
 static uint32_t last_version;

-static inline bool migration_bitmap_test_and_reset_dirty(MemoryRegion *mr,
-                                                         ram_addr_t offset)
+static inline
+ram_addr_t migration_bitmap_find_and_reset_dirty(MemoryRegion *mr,
+                                                 ram_addr_t start)
 {
-    bool ret;
-    int nr = (mr->ram_addr + offset) >> TARGET_PAGE_BITS;
+    unsigned long base = mr->ram_addr >> TARGET_PAGE_BITS;
+    unsigned long nr = base + (start >> TARGET_PAGE_BITS);
+    unsigned long size = base + (int128_get64(mr->size) >> TARGET_PAGE_BITS);

-    ret = test_and_clear_bit(nr, migration_bitmap);
+    unsigned long next = find_next_bit(migration_bitmap, size, nr);

-    if (ret) {
+    if (next < size) {
+        clear_bit(next, migration_bitmap);
         migration_dirty_pages--;
     }
-    return ret;
+    return (next - base) << TARGET_PAGE_BITS;
 }

 static inline bool migration_bitmap_set_dirty(MemoryRegion *mr,
@@ -423,6 +426,7 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
 {
     RAMBlock *block = last_seen_block;
     ram_addr_t offset = last_offset;
+    bool complete_round = false;
     int bytes_sent = -1;
     MemoryRegion *mr;
     ram_addr_t current_addr;
@@ -430,9 +434,21 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
     if (!block)
         block = QTAILQ_FIRST(&ram_list.blocks);

-    do {
+    while (true) {
         mr = block->mr;
-        if (migration_bitmap_test_and_reset_dirty(mr, offset)) {
+        offset = migration_bitmap_find_and_reset_dirty(mr, offset);
+        if (complete_round && block == last_seen_block &&
+            offset >= last_offset) {
+            break;
+        }
+        if (offset >= block->length) {
+            offset = 0;
+            block = QTAILQ_NEXT(block, next);
+            if (!block) {
+                block = QTAILQ_FIRST(&ram_list.blocks);
+                complete_round = true;
+            }
+        } else {
             uint8_t *p;
             int cont = (block == last_sent_block) ?
                 RAM_SAVE_FLAG_CONTINUE : 0;
@@ -467,16 +483,7 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
                 break;
             }
         }
-
-        offset += TARGET_PAGE_SIZE;
-        if (offset >= block->length) {
-            offset = 0;
-            block = QTAILQ_NEXT(block, next);
-            if (!block)
-                block = QTAILQ_FIRST(&ram_list.blocks);
-        }
-    } while (block != last_seen_block || offset != last_offset);
-
+    }
     last_seen_block = block;
     last_offset = offset;

-- 
1.7.11.7

[Qemu-devel] [PATCH 31/34] ram: account the amount of transferred ram better

[Juan Quintela]

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 arch_init.c | 34 ++++++++++++++++++++--------------
 1 file changed, 20 insertions(+), 14 deletions(-)

diff --git a/arch_init.c b/arch_init.c
index 978fe37..43a5b9e 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -265,16 +265,21 @@ uint64_t xbzrle_mig_pages_overflow(void)
     return acct_info.xbzrle_overflows;
 }

-static void save_block_hdr(QEMUFile *f, RAMBlock *block, ram_addr_t offset,
-        int cont, int flag)
+static size_t save_block_hdr(QEMUFile *f, RAMBlock *block, ram_addr_t offset,
+                             int cont, int flag)
 {
-        qemu_put_be64(f, offset | cont | flag);
-        if (!cont) {
-                qemu_put_byte(f, strlen(block->idstr));
-                qemu_put_buffer(f, (uint8_t *)block->idstr,
-                                strlen(block->idstr));
-        }
+    size_t size;
+
+    qemu_put_be64(f, offset | cont | flag);
+    size = 8;

+    if (!cont) {
+        qemu_put_byte(f, strlen(block->idstr));
+        qemu_put_buffer(f, (uint8_t *)block->idstr,
+                        strlen(block->idstr));
+        size += 1 + strlen(block->idstr);
+    }
+    return size;
 }

 #define ENCODING_FLAG_XBZRLE 0x1
@@ -321,11 +326,11 @@ static int save_xbzrle_page(QEMUFile *f, uint8_t *current_data,
     }

     /* Send XBZRLE based compressed page */
-    save_block_hdr(f, block, offset, cont, RAM_SAVE_FLAG_XBZRLE);
+    bytes_sent = save_block_hdr(f, block, offset, cont, RAM_SAVE_FLAG_XBZRLE);
     qemu_put_byte(f, ENCODING_FLAG_XBZRLE);
     qemu_put_be16(f, encoded_len);
     qemu_put_buffer(f, XBZRLE.encoded_buf, encoded_len);
-    bytes_sent = encoded_len + 1 + 2;
+    bytes_sent += encoded_len + 1 + 2;
     acct_info.xbzrle_pages++;
     acct_info.xbzrle_bytes += bytes_sent;

@@ -457,9 +462,10 @@ static int ram_save_block(QEMUFile *f, bool last_stage)

             if (is_dup_page(p)) {
                 acct_info.dup_pages++;
-                save_block_hdr(f, block, offset, cont, RAM_SAVE_FLAG_COMPRESS);
+                bytes_sent = save_block_hdr(f, block, offset, cont,
+                                            RAM_SAVE_FLAG_COMPRESS);
                 qemu_put_byte(f, *p);
-                bytes_sent = 1;
+                bytes_sent += 1;
             } else if (migrate_use_xbzrle()) {
                 current_addr = block->offset + offset;
                 bytes_sent = save_xbzrle_page(f, p, current_addr, block,
@@ -471,9 +477,9 @@ static int ram_save_block(QEMUFile *f, bool last_stage)

             /* either we didn't send yet (we may have had XBZRLE overflow) */
             if (bytes_sent == -1) {
-                save_block_hdr(f, block, offset, cont, RAM_SAVE_FLAG_PAGE);
+                bytes_sent = save_block_hdr(f, block, offset, cont, RAM_SAVE_FLAG_PAGE);
                 qemu_put_buffer(f, p, TARGET_PAGE_SIZE);
-                bytes_sent = TARGET_PAGE_SIZE;
+                bytes_sent += TARGET_PAGE_SIZE;
                 acct_info.norm_pages++;
             }

-- 
1.7.11.7

[Qemu-devel] [PATCH 32/34] ram: refactor ram_save_block() return value

[Juan Quintela]

It could only return 0 if we only found dirty xbzrle pages that hadn't
changed (i.e. they were written with the same content).  We don't care
about that case, it is the same than nothing dirty.

So now the return of the function is how much have it written, nothing
else. Adjust callers.

And we also made ram_save_iterate() return the number of transferred
bytes, not the number of transferred pages.

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 arch_init.c | 25 +++++++++++++++----------
 1 file changed, 15 insertions(+), 10 deletions(-)

diff --git a/arch_init.c b/arch_init.c
index 43a5b9e..588561b 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -422,9 +422,8 @@ static void migration_bitmap_sync(void)
 /*
  * ram_save_block: Writes a page of memory to the stream f
  *
- * Returns:  0: if the page hasn't changed
- *          -1: if there are no more dirty pages
- *           n: the amount of bytes written in other case
+ * Returns:  The number of bytes written.
+ *           0 means no dirty pages
  */

 static int ram_save_block(QEMUFile *f, bool last_stage)
@@ -432,7 +431,7 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
     RAMBlock *block = last_seen_block;
     ram_addr_t offset = last_offset;
     bool complete_round = false;
-    int bytes_sent = -1;
+    int bytes_sent = 0;
     MemoryRegion *mr;
     ram_addr_t current_addr;

@@ -460,6 +459,8 @@ static int ram_save_block(QEMUFile *f, bool last_stage)

             p = memory_region_get_ram_ptr(mr) + offset;

+            /* In doubt sent page as normal */
+            bytes_sent = -1;
             if (is_dup_page(p)) {
                 acct_info.dup_pages++;
                 bytes_sent = save_block_hdr(f, block, offset, cont,
@@ -475,7 +476,7 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
                 }
             }

-            /* either we didn't send yet (we may have had XBZRLE overflow) */
+            /* XBZRLE overflow or normal page */
             if (bytes_sent == -1) {
                 bytes_sent = save_block_hdr(f, block, offset, cont, RAM_SAVE_FLAG_PAGE);
                 qemu_put_buffer(f, p, TARGET_PAGE_SIZE);
@@ -484,7 +485,7 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
             }

             /* if page is unmodified, continue to the next */
-            if (bytes_sent != 0) {
+            if (bytes_sent > 0) {
                 last_sent_block = block;
                 break;
             }
@@ -605,6 +606,7 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
     int ret;
     int i;
     int64_t t0;
+    int total_sent = 0;

     qemu_mutex_lock_ramlist();

@@ -619,10 +621,10 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)

         bytes_sent = ram_save_block(f, false);
         /* no more blocks to sent */
-        if (bytes_sent < 0) {
+        if (bytes_sent == 0) {
             break;
         }
-        bytes_transferred += bytes_sent;
+        total_sent += bytes_sent;
         acct_info.iterations++;
         /* we want to check in the 1st loop, just in case it was the 1st time
            and we had to sync the dirty bitmap.
@@ -641,13 +643,16 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
     }

     if (ret < 0) {
+        bytes_transferred += total_sent;
         return ret;
     }

     qemu_mutex_unlock_ramlist();
     qemu_put_be64(f, RAM_SAVE_FLAG_EOS);
+    total_sent += 8;
+    bytes_transferred += total_sent;

-    return i;
+    return total_sent;
 }

 static int ram_save_complete(QEMUFile *f, void *opaque)
@@ -664,7 +669,7 @@ static int ram_save_complete(QEMUFile *f, void *opaque)

         bytes_sent = ram_save_block(f, true);
         /* no more blocks to sent */
-        if (bytes_sent < 0) {
+        if (bytes_sent == 0) {
             break;
         }
         bytes_transferred += bytes_sent;
-- 
1.7.11.7

[Qemu-devel] [PATCH 33/34] migration: fix qemu_get_fd for BufferedFile

[Juan Quintela]

From: Paolo Bonzini <pbonzini@redhat.com>

Not really used, but nice to have it correct. :)

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/migration.c b/migration.c
index cabf17b..ca39564 100644
--- a/migration.c
+++ b/migration.c
@@ -625,7 +625,7 @@ static int buffered_get_fd(void *opaque)
 {
     QEMUFileBuffered *s = opaque;

-    return qemu_get_fd(s->file);
+    return s->migration_state->fd;
 }

 /*
-- 
1.7.11.7

[Qemu-devel] [PATCH 34/34] migration: merge QEMUFileBuffered into MigrationState

[Juan Quintela]

Avoid splitting the state of outgoing migration, more or less arbitrarily,
between two data structures.  QEMUFileBuffered anyway is used only during
migration.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration.c | 63 ++++++++++++++++++++++++-------------------------------------
 migration.h |  8 ++++++++
 2 files changed, 33 insertions(+), 38 deletions(-)

diff --git a/migration.c b/migration.c
index ca39564..1d7ffb2 100644
--- a/migration.c
+++ b/migration.c
@@ -521,18 +521,8 @@ int64_t migrate_xbzrle_cache_size(void)

 /* migration thread support */

-typedef struct QEMUFileBuffered {
-    MigrationState *migration_state;
-    QEMUFile *file;
-    size_t bytes_xfer;
-    size_t xfer_limit;
-    uint8_t *buffer;
-    size_t buffer_size;
-    size_t buffer_capacity;
-    QemuThread thread;
-} QEMUFileBuffered;
-
-static ssize_t buffered_flush(QEMUFileBuffered *s)
+
+static ssize_t buffered_flush(MigrationState *s)
 {
     size_t offset = 0;
     ssize_t ret = 0;
@@ -541,8 +531,7 @@ static ssize_t buffered_flush(QEMUFileBuffered *s)

     while (s->bytes_xfer < s->xfer_limit && offset < s->buffer_size) {
         size_t to_send = MIN(s->buffer_size - offset, s->xfer_limit - s->bytes_xfer);
-        ret = migrate_fd_put_buffer(s->migration_state, s->buffer + offset,
-                                    to_send);
+        ret = migrate_fd_put_buffer(s, s->buffer + offset, to_send);
         if (ret <= 0) {
             DPRINTF("error flushing data, %zd\n", ret);
             break;
@@ -566,7 +555,7 @@ static ssize_t buffered_flush(QEMUFileBuffered *s)
 static int buffered_put_buffer(void *opaque, const uint8_t *buf,
                                int64_t pos, int size)
 {
-    QEMUFileBuffered *s = opaque;
+    MigrationState *s = opaque;
     ssize_t error;

     DPRINTF("putting %d bytes at %" PRId64 "\n", size, pos);
@@ -598,7 +587,7 @@ static int buffered_put_buffer(void *opaque, const uint8_t *buf,

 static int buffered_close(void *opaque)
 {
-    QEMUFileBuffered *s = opaque;
+    MigrationState *s = opaque;
     ssize_t ret = 0;
     int ret2;

@@ -612,20 +601,20 @@ static int buffered_close(void *opaque)
         }
     }

-    ret2 = migrate_fd_close(s->migration_state);
+    ret2 = migrate_fd_close(s);
     if (ret >= 0) {
         ret = ret2;
     }
-    ret = migrate_fd_close(s->migration_state);
-    s->migration_state->complete = true;
+    ret = migrate_fd_close(s);
+    s->complete = true;
     return ret;
 }

 static int buffered_get_fd(void *opaque)
 {
-    QEMUFileBuffered *s = opaque;
+    MigrationState *s = opaque;

-    return s->migration_state->fd;
+    return s->fd;
 }

 /*
@@ -636,7 +625,7 @@ static int buffered_get_fd(void *opaque)
  */
 static int buffered_rate_limit(void *opaque)
 {
-    QEMUFileBuffered *s = opaque;
+    MigrationState *s = opaque;
     int ret;

     ret = qemu_file_get_error(s->file);
@@ -653,7 +642,7 @@ static int buffered_rate_limit(void *opaque)

 static int64_t buffered_set_rate_limit(void *opaque, int64_t new_rate)
 {
-    QEMUFileBuffered *s = opaque;
+    MigrationState *s = opaque;
     if (qemu_file_get_error(s->file)) {
         goto out;
     }
@@ -669,7 +658,7 @@ out:

 static int64_t buffered_get_rate_limit(void *opaque)
 {
-    QEMUFileBuffered *s = opaque;
+    MigrationState *s = opaque;

     return s->xfer_limit;
 }
@@ -741,7 +730,7 @@ static bool migrate_fd_put_ready(MigrationState *s, uint64_t max_size)

 static void *buffered_file_thread(void *opaque)
 {
-    QEMUFileBuffered *s = opaque;
+    MigrationState *s = opaque;
     int64_t initial_time = qemu_get_clock_ms(rt_clock);
     int64_t max_size = 0;
     bool last_round = false;
@@ -749,7 +738,7 @@ static void *buffered_file_thread(void *opaque)
     while (true) {
         int64_t current_time = qemu_get_clock_ms(rt_clock);

-        if (s->migration_state->complete) {
+        if (s->complete) {
             break;
         }
         if (current_time >= initial_time + BUFFER_DELAY) {
@@ -776,12 +765,11 @@ static void *buffered_file_thread(void *opaque)
         DPRINTF("file is ready\n");
         if (s->bytes_xfer < s->xfer_limit) {
             DPRINTF("notifying client\n");
-            last_round = migrate_fd_put_ready(s->migration_state, max_size);
+            last_round = migrate_fd_put_ready(s, max_size);
         }
     }

     g_free(s->buffer);
-    g_free(s);
     return NULL;
 }

@@ -794,22 +782,21 @@ static const QEMUFileOps buffered_file_ops = {
     .set_rate_limit = buffered_set_rate_limit,
 };

-void migrate_fd_connect(MigrationState *migration_state)
+void migrate_fd_connect(MigrationState *s)
 {
-    QEMUFileBuffered *s;
+    s->state = MIG_STATE_ACTIVE;
+    s->bytes_xfer = 0;
+    s->buffer = NULL;
+    s->buffer_size = 0;
+    s->buffer_capacity = 0;

-    migration_state->state = MIG_STATE_ACTIVE;
-    migration_state->first_time = true;
-    s = g_malloc0(sizeof(*s));
+    s->first_time = true;

-    s->migration_state = migration_state;
-    s->xfer_limit = s->migration_state->bandwidth_limit / XFER_LIMIT_RATIO;
-    s->migration_state->complete = false;
+    s->xfer_limit = s->bandwidth_limit / XFER_LIMIT_RATIO;
+    s->complete = false;

     s->file = qemu_fopen_ops(s, &buffered_file_ops);

-    migration_state->file = s->file;
-
     qemu_thread_create(&s->thread, buffered_file_thread, s,
                        QEMU_THREAD_DETACHED);
     notifier_list_notify(&migration_state_notifiers, s);
diff --git a/migration.h b/migration.h
index 542197e..1b344d5 100644
--- a/migration.h
+++ b/migration.h
@@ -16,6 +16,7 @@

 #include "qdict.h"
 #include "qemu-common.h"
+#include "qemu-thread.h"
 #include "notify.h"
 #include "error.h"
 #include "vmstate.h"
@@ -31,6 +32,13 @@ typedef struct MigrationState MigrationState;
 struct MigrationState
 {
     int64_t bandwidth_limit;
+    size_t bytes_xfer;
+    size_t xfer_limit;
+    uint8_t *buffer;
+    size_t buffer_size;
+    size_t buffer_capacity;
+    QemuThread thread;
+
     QEMUFile *file;
     int fd;
     int state;
-- 
1.7.11.7

Re: [Qemu-devel] [PATCH 00/34] migration thread and queue

[Paolo Bonzini]

Il 19/12/2012 13:33, Juan Quintela ha scritto:
> Hi
> 
> This is my queue for migration-thread and patches associated.  This
> integrates review comments & code for Paolo.  This is the subset from
> both approachs that we agreed with. rest of patches need more review
> and are not here.
> 
> Migrating and idle guest with upstwream:
> 
> (qemu) info migrate
> capabilities: xbzrle: off
> Migration status: completed
> total time: 34251 milliseconds
> downtime: 492 milliseconds
> transferred ram: 762458 kbytes
> remaining ram: 0 kbytes
> total ram: 14688768 kbytes
> duplicate: 3492606 pages
> normal: 189762 pages
> normal bytes: 759048 kbytes
> 
> with this series of patches.
> 
> (qemu) info migrate
> capabilities: xbzrle: off
> Migration status: completed
> total time: 30712 milliseconds
> downtime: 29 milliseconds
> transferred ram: 738857 kbytes
> remaining ram: 0 kbytes
> total ram: 14688768 kbytes
> duplicate: 3503423 pages
> normal: 176671 pages
> normal bytes: 706684 kbytes
> 
> Notice the big difference in downtime.  And that is also seen inside
> the guest a program that just do an idle loop seeing how "long" it
> takes to wait for 10ms.
> 
> with upstream:
> 
> [root@d1 ~]# ./timer
> delay of 452 ms
> delay of 114 ms
> delay of 136 ms
> delay of 135 ms
> delay of 136 ms
> delay of 131 ms
> delay of 134 ms
> 
> with this series of patches, wait never takes 100ms, nothing is printed.
> 
> Please review.
> 
> Thanks, Juan.
> 
> 
> The following changes since commit 914606d26e654d4c01bd5186f4d05e3fd445e219:
> 
>   Merge remote-tracking branch 'stefanha/trivial-patches' into staging (2012-12-18 15:41:43 -0600)
> 
> are available in the git repository at:
> 
> 
>   git://repo.or.cz/qemu/quintela.git thread.next
> 
> for you to fetch changes up to 065bebbacf54dcad48aad42112417dbb44451499:
> 
>   migration: merge QEMUFileBuffered into MigrationState (2012-12-19 13:21:29 +0100)
> 
> ----------------------------------------------------------------
> Juan Quintela (25):
>       migration: include qemu-file.h
>       migration-fd: remove duplicate include
>       buffered_file: Move from using a timer to use a thread
>       migration: make qemu_fopen_ops_buffered() return void
>       migration: move migration thread init code to migrate_fd_put_ready
>       migration: make writes blocking
>       migration: remove unfreeze logic
>       migration: just lock migrate_fd_put_ready
>       buffered_file: Unfold the trick to restart generating migration data
>       buffered_file: don't flush on put buffer
>       buffered_file: unfold buffered_append in buffered_put_buffer
>       savevm: New save live migration method: pending
>       migration: move buffered_file.c code into migration.c
>       migration: add XFER_LIMIT_RATIO
>       migration: move migration_fd_put_ready()
>       migration: Inline qemu_fopen_ops_buffered into migrate_fd_connect
>       migration: move migration notifier
>       ram: rename last_block to last_seen_block
>       ram: Add last_sent_block
>       memory: introduce memory_region_test_and_clear_dirty
>       ram: Use memory_region_test_and_clear_dirty
>       ram: optimize migration bitmap walking
>       ram: account the amount of transferred ram better
>       ram: refactor ram_save_block() return value
>       migration: merge QEMUFileBuffered into MigrationState
> 
> Paolo Bonzini (7):
>       migration: fix migration_bitmap leak
>       buffered_file: do not send more than s->bytes_xfer bytes per tick
>       migration: remove double call to migrate_fd_close
>       exec: change ramlist from MRU order to a 1-item cache
>       exec: change RAM list to a TAILQ
>       exec: sort the memory from biggest to smallest
>       migration: fix qemu_get_fd for BufferedFile
> 
> Umesh Deshpande (2):
>       add a version number to ram_list
>       protect the ramlist with a separate mutex
> 
>  Makefile.objs           |   2 +-
>  arch_init.c             | 245 +++++++++++++++---------------
>  block-migration.c       |  49 ++----
>  buffered_file.c         | 269 ---------------------------------
>  buffered_file.h         |  22 ---
>  cpu-all.h               |  15 +-
>  dump.c                  |   8 +-
>  exec.c                  | 128 +++++++++++-----
>  memory.c                |  16 ++
>  memory.h                |  16 ++
>  memory_mapping.c        |   4 +-
>  migration-exec.c        |   3 +-
>  migration-fd.c          |   4 +-
>  migration-tcp.c         |   3 +-
>  migration-unix.c        |   3 +-
>  migration.c             | 390 ++++++++++++++++++++++++++++++++++++------------
>  migration.h             |  13 +-
>  qemu-file.h             |   5 -
>  savevm.c                |  24 ++-
>  sysemu.h                |   1 +
>  target-i386/arch_dump.c |   2 +-
>  vmstate.h               |   1 +
>  22 files changed, 599 insertions(+), 624 deletions(-)
>  delete mode 100644 buffered_file.c
>  delete mode 100644 buffered_file.h
> 
> 

Yay.

Acked-by: Paolo Bonzini <pbonzini@redhat.com>

Paolo

Re: [Qemu-devel] [PATCH 29/34] ram: Use memory_region_test_and_clear_dirty

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 299 bytes --]

On 12/19/2012 05:33 AM, Juan Quintela wrote:
> This avoids having to do two walks over the dirty bitmap, once reading
> the dirty bits, and anthoer cleaning them.

s/anthoer/another/

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: [Qemu-devel] [PATCH 32/34] ram: refactor ram_save_block() return value

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1241 bytes --]

On 12/19/2012 05:33 AM, Juan Quintela wrote:
> It could only return 0 if we only found dirty xbzrle pages that hadn't
> changed (i.e. they were written with the same content).  We don't care
> about that case, it is the same than nothing dirty.

s/than/as/

> 
> So now the return of the function is how much have it written, nothing

s/have/has/

> else. Adjust callers.
> 
> And we also made ram_save_iterate() return the number of transferred
> bytes, not the number of transferred pages.
> 
> Signed-off-by: Juan Quintela <quintela@redhat.com>
> ---
>  arch_init.c | 25 +++++++++++++++----------
>  1 file changed, 15 insertions(+), 10 deletions(-)
> 

>  static int ram_save_block(QEMUFile *f, bool last_stage)
> @@ -432,7 +431,7 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
>      RAMBlock *block = last_seen_block;
>      ram_addr_t offset = last_offset;
>      bool complete_round = false;
> -    int bytes_sent = -1;
> +    int bytes_sent = 0;

Are we guaranteed that we will never send more than 2G bytes in one call
to this function, or should this be changed to int64_t?

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]