* [Qemu-devel] [PATCH 00/35] Migration thread (20121211)
@ 2012-12-11 12:46 Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 01/35] split MRU ram list Juan Quintela
` (35 more replies)
0 siblings, 36 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:46 UTC (permalink / raw)
To: qemu-devel
Hi
Changes:
- we are back to full patch series
- we reuse save_iterate() code in complete phase
- I didn't try to change the block-migration code, it should continue
working, but has none of the optimizations.
- we split the complete phase into:
* we write some ram into the buffer
* we flush the buffer
instead of doing it in a big push
- we change the iterate phase to tell the subsystem how much free space
is in the buffer. We should be able now to use smaller buffers.
- We "could" change easily to move back to the iterate phase if the
complete phase is taking too long, is just the case of having to put
a timing test into the new loop. I didn't put it, because my
testing showed that now we are much, much better and being inside
the downtime target.
- Last patch is just to printing where the timing is spent in the
complete phase, that would be removed when I ask for a pull.
- this branch includes the patches that I sent for request sooner today.
The following changes since commit 77db8657048f233edf21e1a9ebdc30a367fbdc36:
migration: Fix madvise breakage if host and guest have different page sizes (2012-12-11 12:45:56 +0100)
are available in the git repository at:
ssh://repo.or.cz/srv/git/qemu/quintela.git thread-20121211
for you to fetch changes up to 8c109644beeb1e9d228e0b3b2fb65adad80c3b96:
Please test & comment & review, Juan.
[20121029 version]
After discussing with Anthony and Paolo, this is the minimal migration
thread support that we can do for 1.3.
- fixed all the comments (thanks eric, paolo and orit).
- buffered_file.c remains until 1.4.
- testing for vinod showed very nice results, that is why the bitmap
handling optimizations remain.
Note: Writes has become blocking, and I have to change the "remove"
the feature now in qemu-sockets.c. Checked that migration was the
only user of that feature. If new users appear, they just need to add
the socket_set_nonblock() by hand.
Please, review.
Thanks, Juan.
Juan Quintela (32):
buffered_file: Move from using a timer to use a thread
migration: make qemu_fopen_ops_buffered() return void
migration: stop all cpus correctly
migration: make writes blocking
migration: remove unfreeze logic
migration: take finer locking
buffered_file: Unfold the trick to restart generating migration data
buffered_file: don't flush on put buffer
buffered_file: unfold buffered_append in buffered_put_buffer
savevm: New save live migration method: pending
migration: include qemu-file.h
migration-fd: remove duplicate include
migration: move buffered_file.c code into migration.c
migration: move migration_fd_put_ready()
migration: Inline qemu_fopen_ops_buffered into migrate_fd_connect
migration: move migration notifier
migration: move begining stage to the migration thread
migration: Add buffered_flush error handling
migration: move exit condition to migration thread
migration: unfold rest of migrate_fd_put_ready() into thread
ram: rename last_block to last_seen_block
ram: Add last_sent_block
memory: introduce memory_region_test_and_clear_dirty
ram: Use memory_region_test_and_clear_dirty
migration: Only go to the iterate stage if there is anything to send
ram: optimize migration bitmap walking
ram: account the amount of transferred ram better
ram: refactor ram_save_block() return value
ram: add free_space parameter to save_live functions
ram: remove xbrle last_stage optimization
ram: reuse ram_save_iterate() for the complete stage
migration: print times for end phase
Paolo Bonzini (1):
split MRU ram list
Umesh Deshpande (2):
add a version number to ram_list
protect the ramlist with a separate mutex
Makefile.objs | 2 +-
arch_init.c | 241 +++++++++++++++----------------
block-migration.c | 51 ++-----
block.c | 6 +
buffered_file.c | 269 -----------------------------------
buffered_file.h | 22 ---
cpu-all.h | 13 +-
cpus.c | 17 +++
exec.c | 44 +++++-
memory.c | 16 +++
memory.h | 16 +++
migration-exec.c | 3 +-
migration-fd.c | 4 +-
migration-tcp.c | 3 +-
migration-unix.c | 3 +-
migration.c | 419 +++++++++++++++++++++++++++++++++++++++++-------------
migration.h | 4 +-
qemu-file.h | 5 -
savevm.c | 47 ++++--
sysemu.h | 3 +-
vmstate.h | 3 +-
21 files changed, 608 insertions(+), 583 deletions(-)
delete mode 100644 buffered_file.c
delete mode 100644 buffered_file.h
--
1.7.11.7
^ permalink raw reply [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 01/35] split MRU ram list
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
@ 2012-12-11 12:46 ` Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 02/35] add a version number to ram_list Juan Quintela
` (34 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:46 UTC (permalink / raw)
To: qemu-devel; +Cc: Paolo Bonzini
From: Paolo Bonzini <pbonzini@redhat.com>
Outside the execution threads the normal, non-MRU-ized order of
the RAM blocks should always be enough. So manage two separate
lists, which will have separate locking rules.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
---
arch_init.c | 1 +
cpu-all.h | 4 +++-
exec.c | 17 ++++++++++++-----
3 files changed, 16 insertions(+), 6 deletions(-)
diff --git a/arch_init.c b/arch_init.c
index 83dcc53..e794332 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -48,6 +48,7 @@
#include "qemu/page_cache.h"
#include "qmp-commands.h"
#include "trace.h"
+#include "cpu-all.h"
#ifdef DEBUG_ARCH_INIT
#define DPRINTF(fmt, ...) \
diff --git a/cpu-all.h b/cpu-all.h
index d6b2b19..070271c 100644
--- a/cpu-all.h
+++ b/cpu-all.h
@@ -486,8 +486,9 @@ typedef struct RAMBlock {
ram_addr_t offset;
ram_addr_t length;
uint32_t flags;
- char idstr[256];
+ QLIST_ENTRY(RAMBlock) next_mru;
QLIST_ENTRY(RAMBlock) next;
+ char idstr[256];
#if defined(__linux__) && !defined(TARGET_S390X)
int fd;
#endif
@@ -495,6 +496,7 @@ typedef struct RAMBlock {
typedef struct RAMList {
uint8_t *phys_dirty;
+ QLIST_HEAD(, RAMBlock) blocks_mru;
QLIST_HEAD(, RAMBlock) blocks;
} RAMList;
extern RAMList ram_list;
diff --git a/exec.c b/exec.c
index 0594b07..acbc30f 100644
--- a/exec.c
+++ b/exec.c
@@ -57,6 +57,7 @@
#include "xen-mapcache.h"
#include "trace.h"
#endif
+#include "cpu-all.h"
#include "cputlb.h"
@@ -97,7 +98,10 @@ static uint8_t *code_gen_ptr;
int phys_ram_fd;
static int in_migration;
-RAMList ram_list = { .blocks = QLIST_HEAD_INITIALIZER(ram_list.blocks) };
+RAMList ram_list = {
+ .blocks = QLIST_HEAD_INITIALIZER(ram_list.blocks),
+ .blocks_mru = QLIST_HEAD_INITIALIZER(ram_list.blocks_mru)
+};
static MemoryRegion *system_memory;
static MemoryRegion *system_io;
@@ -2585,6 +2589,7 @@ ram_addr_t qemu_ram_alloc_from_ptr(ram_addr_t size, void *host,
new_block->length = size;
QLIST_INSERT_HEAD(&ram_list.blocks, new_block, next);
+ QLIST_INSERT_HEAD(&ram_list.blocks_mru, new_block, next_mru);
ram_list.phys_dirty = g_realloc(ram_list.phys_dirty,
last_ram_offset() >> TARGET_PAGE_BITS);
@@ -2613,6 +2618,7 @@ void qemu_ram_free_from_ptr(ram_addr_t addr)
QLIST_FOREACH(block, &ram_list.blocks, next) {
if (addr == block->offset) {
QLIST_REMOVE(block, next);
+ QLIST_REMOVE(block, next_mru);
g_free(block);
return;
}
@@ -2626,6 +2632,7 @@ void qemu_ram_free(ram_addr_t addr)
QLIST_FOREACH(block, &ram_list.blocks, next) {
if (addr == block->offset) {
QLIST_REMOVE(block, next);
+ QLIST_REMOVE(block, next_mru);
if (block->flags & RAM_PREALLOC_MASK) {
;
} else if (mem_path) {
@@ -2731,12 +2738,12 @@ void *qemu_get_ram_ptr(ram_addr_t addr)
{
RAMBlock *block;
- QLIST_FOREACH(block, &ram_list.blocks, next) {
+ QLIST_FOREACH(block, &ram_list.blocks_mru, next_mru) {
if (addr - block->offset < block->length) {
/* Move this entry to to start of the list. */
if (block != QLIST_FIRST(&ram_list.blocks)) {
- QLIST_REMOVE(block, next);
- QLIST_INSERT_HEAD(&ram_list.blocks, block, next);
+ QLIST_REMOVE(block, next_mru);
+ QLIST_INSERT_HEAD(&ram_list.blocks_mru, block, next_mru);
}
if (xen_enabled()) {
/* We need to check if the requested address is in the RAM
@@ -2831,7 +2838,7 @@ int qemu_ram_addr_from_host(void *ptr, ram_addr_t *ram_addr)
return 0;
}
- QLIST_FOREACH(block, &ram_list.blocks, next) {
+ QLIST_FOREACH(block, &ram_list.blocks_mru, next_mru) {
/* This case append when the block is not mapped. */
if (block->host == NULL) {
continue;
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 02/35] add a version number to ram_list
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 01/35] split MRU ram list Juan Quintela
@ 2012-12-11 12:46 ` Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 03/35] protect the ramlist with a separate mutex Juan Quintela
` (33 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:46 UTC (permalink / raw)
To: qemu-devel; +Cc: Paolo Bonzini, Umesh Deshpande
From: Umesh Deshpande <udeshpan@redhat.com>
This will be used to detect if last_block might have become invalid
across different calls to ram_save_live.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Umesh Deshpande <udeshpan@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
---
arch_init.c | 7 ++++++-
cpu-all.h | 1 +
exec.c | 4 ++++
3 files changed, 11 insertions(+), 1 deletion(-)
diff --git a/arch_init.c b/arch_init.c
index e794332..edcc6d6 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -336,6 +336,7 @@ static RAMBlock *last_block;
static ram_addr_t last_offset;
static unsigned long *migration_bitmap;
static uint64_t migration_dirty_pages;
+static uint32_t last_version;
static inline bool migration_bitmap_test_and_reset_dirty(MemoryRegion *mr,
ram_addr_t offset)
@@ -406,7 +407,6 @@ static void migration_bitmap_sync(void)
}
}
-
/*
* ram_save_block: Writes a page of memory to the stream f
*
@@ -558,6 +558,7 @@ static void reset_ram_globals(void)
{
last_block = NULL;
last_offset = 0;
+ last_version = ram_list.version;
sort_ram_list();
}
@@ -613,6 +614,10 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
uint64_t expected_downtime;
MigrationState *s = migrate_get_current();
+ if (ram_list.version != last_version) {
+ reset_ram_globals();
+ }
+
bytes_transferred_last = bytes_transferred;
bwidth = qemu_get_clock_ns(rt_clock);
diff --git a/cpu-all.h b/cpu-all.h
index 070271c..1ebf17c 100644
--- a/cpu-all.h
+++ b/cpu-all.h
@@ -496,6 +496,7 @@ typedef struct RAMBlock {
typedef struct RAMList {
uint8_t *phys_dirty;
+ uint32_t version;
QLIST_HEAD(, RAMBlock) blocks_mru;
QLIST_HEAD(, RAMBlock) blocks;
} RAMList;
diff --git a/exec.c b/exec.c
index acbc30f..a412513 100644
--- a/exec.c
+++ b/exec.c
@@ -2591,6 +2591,8 @@ ram_addr_t qemu_ram_alloc_from_ptr(ram_addr_t size, void *host,
QLIST_INSERT_HEAD(&ram_list.blocks, new_block, next);
QLIST_INSERT_HEAD(&ram_list.blocks_mru, new_block, next_mru);
+ ram_list.version++;
+
ram_list.phys_dirty = g_realloc(ram_list.phys_dirty,
last_ram_offset() >> TARGET_PAGE_BITS);
memset(ram_list.phys_dirty + (new_block->offset >> TARGET_PAGE_BITS),
@@ -2619,6 +2621,7 @@ void qemu_ram_free_from_ptr(ram_addr_t addr)
if (addr == block->offset) {
QLIST_REMOVE(block, next);
QLIST_REMOVE(block, next_mru);
+ ram_list.version++;
g_free(block);
return;
}
@@ -2633,6 +2636,7 @@ void qemu_ram_free(ram_addr_t addr)
if (addr == block->offset) {
QLIST_REMOVE(block, next);
QLIST_REMOVE(block, next_mru);
+ ram_list.version++;
if (block->flags & RAM_PREALLOC_MASK) {
;
} else if (mem_path) {
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 03/35] protect the ramlist with a separate mutex
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 01/35] split MRU ram list Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 02/35] add a version number to ram_list Juan Quintela
@ 2012-12-11 12:46 ` Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 04/35] buffered_file: Move from using a timer to use a thread Juan Quintela
` (32 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:46 UTC (permalink / raw)
To: qemu-devel; +Cc: Paolo Bonzini, Umesh Deshpande
From: Umesh Deshpande <udeshpan@redhat.com>
Add the new mutex that protects shared state between ram_save_live
and the iothread. If the iothread mutex has to be taken together
with the ramlist mutex, the iothread shall always be _outside_.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Umesh Deshpande <udeshpan@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
---
arch_init.c | 9 ++++++++-
cpu-all.h | 8 ++++++++
exec.c | 23 +++++++++++++++++++++--
3 files changed, 37 insertions(+), 3 deletions(-)
diff --git a/arch_init.c b/arch_init.c
index edcc6d6..9d1109b 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -553,7 +553,6 @@ static void ram_migration_cancel(void *opaque)
migration_end();
}
-
static void reset_ram_globals(void)
{
last_block = NULL;
@@ -573,6 +572,7 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
bitmap_set(migration_bitmap, 0, ram_pages);
migration_dirty_pages = ram_pages;
+ qemu_mutex_lock_ramlist();
bytes_transferred = 0;
reset_ram_globals();
@@ -600,6 +600,7 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
qemu_put_be64(f, block->length);
}
+ qemu_mutex_unlock_ramlist();
qemu_put_be64(f, RAM_SAVE_FLAG_EOS);
return 0;
@@ -614,6 +615,8 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
uint64_t expected_downtime;
MigrationState *s = migrate_get_current();
+ qemu_mutex_lock_ramlist();
+
if (ram_list.version != last_version) {
reset_ram_globals();
}
@@ -662,6 +665,7 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
bwidth = 0.000001;
}
+ qemu_mutex_unlock_ramlist();
qemu_put_be64(f, RAM_SAVE_FLAG_EOS);
expected_downtime = ram_save_remaining() * TARGET_PAGE_SIZE / bwidth;
@@ -682,6 +686,8 @@ static int ram_save_complete(QEMUFile *f, void *opaque)
{
migration_bitmap_sync();
+ qemu_mutex_lock_ramlist();
+
/* try transferring iterative blocks of memory */
/* flush all remaining blocks regardless of rate limiting */
@@ -697,6 +703,7 @@ static int ram_save_complete(QEMUFile *f, void *opaque)
}
memory_global_dirty_log_stop();
+ qemu_mutex_unlock_ramlist();
qemu_put_be64(f, RAM_SAVE_FLAG_EOS);
g_free(migration_bitmap);
diff --git a/cpu-all.h b/cpu-all.h
index 1ebf17c..13c393b 100644
--- a/cpu-all.h
+++ b/cpu-all.h
@@ -22,6 +22,7 @@
#include "qemu-common.h"
#include "qemu-tls.h"
#include "cpu-common.h"
+#include "qemu-thread.h"
/* some important defines:
*
@@ -486,7 +487,9 @@ typedef struct RAMBlock {
ram_addr_t offset;
ram_addr_t length;
uint32_t flags;
+ /* Protected by the iothread lock. */
QLIST_ENTRY(RAMBlock) next_mru;
+ /* Protected by the ramlist lock. */
QLIST_ENTRY(RAMBlock) next;
char idstr[256];
#if defined(__linux__) && !defined(TARGET_S390X)
@@ -495,9 +498,12 @@ typedef struct RAMBlock {
} RAMBlock;
typedef struct RAMList {
+ QemuMutex mutex;
+ /* Protected by the iothread lock. */
uint8_t *phys_dirty;
uint32_t version;
QLIST_HEAD(, RAMBlock) blocks_mru;
+ /* Protected by the ramlist lock. */
QLIST_HEAD(, RAMBlock) blocks;
} RAMList;
extern RAMList ram_list;
@@ -517,6 +523,8 @@ extern int mem_prealloc;
void dump_exec_info(FILE *f, fprintf_function cpu_fprintf);
ram_addr_t last_ram_offset(void);
+void qemu_mutex_lock_ramlist(void);
+void qemu_mutex_unlock_ramlist(void);
#endif /* !CONFIG_USER_ONLY */
int cpu_memory_rw_debug(CPUArchState *env, target_ulong addr,
diff --git a/exec.c b/exec.c
index a412513..f299908 100644
--- a/exec.c
+++ b/exec.c
@@ -652,6 +652,7 @@ bool tcg_enabled(void)
void cpu_exec_init_all(void)
{
#if !defined(CONFIG_USER_ONLY)
+ qemu_mutex_init(&ram_list.mutex);
memory_map_init();
io_mem_init();
#endif
@@ -2346,6 +2347,16 @@ void qemu_flush_coalesced_mmio_buffer(void)
kvm_flush_coalesced_mmio_buffer();
}
+void qemu_mutex_lock_ramlist(void)
+{
+ qemu_mutex_lock(&ram_list.mutex);
+}
+
+void qemu_mutex_unlock_ramlist(void)
+{
+ qemu_mutex_unlock(&ram_list.mutex);
+}
+
#if defined(__linux__) && !defined(TARGET_S390X)
#include <sys/vfs.h>
@@ -2527,6 +2538,7 @@ void qemu_ram_set_idstr(ram_addr_t addr, const char *name, DeviceState *dev)
}
pstrcat(new_block->idstr, sizeof(new_block->idstr), name);
+ qemu_mutex_lock_ramlist();
QLIST_FOREACH(block, &ram_list.blocks, next) {
if (block != new_block && !strcmp(block->idstr, new_block->idstr)) {
fprintf(stderr, "RAMBlock \"%s\" already registered, abort!\n",
@@ -2534,6 +2546,7 @@ void qemu_ram_set_idstr(ram_addr_t addr, const char *name, DeviceState *dev)
abort();
}
}
+ qemu_mutex_unlock_ramlist();
}
static int memory_try_enable_merging(void *addr, size_t len)
@@ -2557,6 +2570,7 @@ ram_addr_t qemu_ram_alloc_from_ptr(ram_addr_t size, void *host,
size = TARGET_PAGE_ALIGN(size);
new_block = g_malloc0(sizeof(*new_block));
+ qemu_mutex_lock_ramlist();
new_block->mr = mr;
new_block->offset = find_ram_offset(size);
if (host) {
@@ -2592,6 +2606,7 @@ ram_addr_t qemu_ram_alloc_from_ptr(ram_addr_t size, void *host,
QLIST_INSERT_HEAD(&ram_list.blocks_mru, new_block, next_mru);
ram_list.version++;
+ qemu_mutex_unlock_ramlist();
ram_list.phys_dirty = g_realloc(ram_list.phys_dirty,
last_ram_offset() >> TARGET_PAGE_BITS);
@@ -2617,21 +2632,24 @@ void qemu_ram_free_from_ptr(ram_addr_t addr)
{
RAMBlock *block;
+ qemu_mutex_lock_ramlist();
QLIST_FOREACH(block, &ram_list.blocks, next) {
if (addr == block->offset) {
QLIST_REMOVE(block, next);
QLIST_REMOVE(block, next_mru);
ram_list.version++;
g_free(block);
- return;
+ break;
}
}
+ qemu_mutex_unlock_ramlist();
}
void qemu_ram_free(ram_addr_t addr)
{
RAMBlock *block;
+ qemu_mutex_lock_ramlist();
QLIST_FOREACH(block, &ram_list.blocks, next) {
if (addr == block->offset) {
QLIST_REMOVE(block, next);
@@ -2662,9 +2680,10 @@ void qemu_ram_free(ram_addr_t addr)
#endif
}
g_free(block);
- return;
+ break;
}
}
+ qemu_mutex_unlock_ramlist();
}
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 04/35] buffered_file: Move from using a timer to use a thread
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (2 preceding siblings ...)
2012-12-11 12:46 ` [Qemu-devel] [PATCH 03/35] protect the ramlist with a separate mutex Juan Quintela
@ 2012-12-11 12:46 ` Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 05/35] migration: make qemu_fopen_ops_buffered() return void Juan Quintela
` (31 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:46 UTC (permalink / raw)
To: qemu-devel
We still protect everything except the wait with the iothread lock.
But we moved from a timer to a thread. Steps one by one.
We also need to detect when we have finished with a variable "complete".
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
buffered_file.c | 58 +++++++++++++++++++++++++++++++++++----------------------
1 file changed, 36 insertions(+), 22 deletions(-)
diff --git a/buffered_file.c b/buffered_file.c
index bd0f61d..418d22a 100644
--- a/buffered_file.c
+++ b/buffered_file.c
@@ -18,6 +18,7 @@
#include "qemu-timer.h"
#include "qemu-char.h"
#include "buffered_file.h"
+#include "qemu-thread.h"
//#define DEBUG_BUFFERED_FILE
@@ -31,7 +32,8 @@ typedef struct QEMUFileBuffered
uint8_t *buffer;
size_t buffer_size;
size_t buffer_capacity;
- QEMUTimer *timer;
+ QemuThread thread;
+ bool complete;
} QEMUFileBuffered;
#ifdef DEBUG_BUFFERED_FILE
@@ -160,11 +162,8 @@ static int buffered_close(void *opaque)
if (ret >= 0) {
ret = ret2;
}
- qemu_del_timer(s->timer);
- qemu_free_timer(s->timer);
- g_free(s->buffer);
- g_free(s);
-
+ ret = migrate_fd_close(s->migration_state);
+ s->complete = true;
return ret;
}
@@ -222,23 +221,38 @@ static int64_t buffered_get_rate_limit(void *opaque)
return s->xfer_limit;
}
-static void buffered_rate_tick(void *opaque)
+/* 10ms xfer_limit is the limit that we should write each 10ms */
+#define BUFFER_DELAY 100
+
+static void *buffered_file_thread(void *opaque)
{
QEMUFileBuffered *s = opaque;
+ int64_t expire_time = qemu_get_clock_ms(rt_clock) + BUFFER_DELAY;
- if (qemu_file_get_error(s->file)) {
- buffered_close(s);
- return;
- }
-
- qemu_mod_timer(s->timer, qemu_get_clock_ms(rt_clock) + 100);
-
- if (s->freeze_output)
- return;
-
- s->bytes_xfer = 0;
+ while (true) {
+ int64_t current_time = qemu_get_clock_ms(rt_clock);
- buffered_put_buffer(s, NULL, 0, 0);
+ if (s->complete) {
+ break;
+ }
+ if (s->freeze_output) {
+ continue;
+ }
+ if (current_time >= expire_time) {
+ s->bytes_xfer = 0;
+ expire_time = current_time + BUFFER_DELAY;
+ }
+ if (s->bytes_xfer >= s->xfer_limit) {
+ /* usleep expects microseconds */
+ usleep((expire_time - current_time)*1000);
+ }
+ qemu_mutex_lock_iothread();
+ buffered_put_buffer(s, NULL, 0, 0);
+ qemu_mutex_unlock_iothread();
+ }
+ g_free(s->buffer);
+ g_free(s);
+ return NULL;
}
static const QEMUFileOps buffered_file_ops = {
@@ -258,12 +272,12 @@ QEMUFile *qemu_fopen_ops_buffered(MigrationState *migration_state)
s->migration_state = migration_state;
s->xfer_limit = migration_state->bandwidth_limit / 10;
+ s->complete = false;
s->file = qemu_fopen_ops(s, &buffered_file_ops);
- s->timer = qemu_new_timer_ms(rt_clock, buffered_rate_tick, s);
-
- qemu_mod_timer(s->timer, qemu_get_clock_ms(rt_clock) + 100);
+ qemu_thread_create(&s->thread, buffered_file_thread, s,
+ QEMU_THREAD_DETACHED);
return s->file;
}
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 05/35] migration: make qemu_fopen_ops_buffered() return void
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (3 preceding siblings ...)
2012-12-11 12:46 ` [Qemu-devel] [PATCH 04/35] buffered_file: Move from using a timer to use a thread Juan Quintela
@ 2012-12-11 12:46 ` Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 06/35] migration: stop all cpus correctly Juan Quintela
` (30 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:46 UTC (permalink / raw)
To: qemu-devel
We want the file assignment to happen before the thread is created to
avoid locking, so we just do it before creating the thread.
Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
---
buffered_file.c | 13 ++++++-------
buffered_file.h | 2 +-
migration.c | 2 +-
migration.h | 1 +
4 files changed, 9 insertions(+), 9 deletions(-)
diff --git a/buffered_file.c b/buffered_file.c
index 418d22a..27610b4 100644
--- a/buffered_file.c
+++ b/buffered_file.c
@@ -33,7 +33,6 @@ typedef struct QEMUFileBuffered
size_t buffer_size;
size_t buffer_capacity;
QemuThread thread;
- bool complete;
} QEMUFileBuffered;
#ifdef DEBUG_BUFFERED_FILE
@@ -163,7 +162,7 @@ static int buffered_close(void *opaque)
ret = ret2;
}
ret = migrate_fd_close(s->migration_state);
- s->complete = true;
+ s->migration_state->complete = true;
return ret;
}
@@ -232,7 +231,7 @@ static void *buffered_file_thread(void *opaque)
while (true) {
int64_t current_time = qemu_get_clock_ms(rt_clock);
- if (s->complete) {
+ if (s->migration_state->complete) {
break;
}
if (s->freeze_output) {
@@ -264,7 +263,7 @@ static const QEMUFileOps buffered_file_ops = {
.set_rate_limit = buffered_set_rate_limit,
};
-QEMUFile *qemu_fopen_ops_buffered(MigrationState *migration_state)
+void qemu_fopen_ops_buffered(MigrationState *migration_state)
{
QEMUFileBuffered *s;
@@ -272,12 +271,12 @@ QEMUFile *qemu_fopen_ops_buffered(MigrationState *migration_state)
s->migration_state = migration_state;
s->xfer_limit = migration_state->bandwidth_limit / 10;
- s->complete = false;
+ s->migration_state->complete = false;
s->file = qemu_fopen_ops(s, &buffered_file_ops);
+ migration_state->file = s->file;
+
qemu_thread_create(&s->thread, buffered_file_thread, s,
QEMU_THREAD_DETACHED);
-
- return s->file;
}
diff --git a/buffered_file.h b/buffered_file.h
index ef010fe..8a246fd 100644
--- a/buffered_file.h
+++ b/buffered_file.h
@@ -17,6 +17,6 @@
#include "hw/hw.h"
#include "migration.h"
-QEMUFile *qemu_fopen_ops_buffered(MigrationState *migration_state);
+void qemu_fopen_ops_buffered(MigrationState *migration_state);
#endif
diff --git a/migration.c b/migration.c
index 73ce170..d4b8406 100644
--- a/migration.c
+++ b/migration.c
@@ -448,7 +448,7 @@ void migrate_fd_connect(MigrationState *s)
int ret;
s->state = MIG_STATE_ACTIVE;
- s->file = qemu_fopen_ops_buffered(s);
+ qemu_fopen_ops_buffered(s);
DPRINTF("beginning savevm\n");
ret = qemu_savevm_state_begin(s->file, &s->params);
diff --git a/migration.h b/migration.h
index c3a23cc..b66fd60 100644
--- a/migration.h
+++ b/migration.h
@@ -45,6 +45,7 @@ struct MigrationState
int64_t dirty_pages_rate;
bool enabled_capabilities[MIGRATION_CAPABILITY_MAX];
int64_t xbzrle_cache_size;
+ bool complete;
};
void process_incoming_migration(QEMUFile *f);
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 06/35] migration: stop all cpus correctly
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (4 preceding siblings ...)
2012-12-11 12:46 ` [Qemu-devel] [PATCH 05/35] migration: make qemu_fopen_ops_buffered() return void Juan Quintela
@ 2012-12-11 12:46 ` Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 07/35] migration: make writes blocking Juan Quintela
` (29 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:46 UTC (permalink / raw)
To: qemu-devel
You can only stop all cpus from the iothread or an vcpu. As we want
to do it from the migration_thread, we need to do this dance with the
botton handlers.
This patch is a request for ideas. I can move this function to cpus.c, but
wondered if there is an easy way of doing this?
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
migration.c | 29 +++++++++++++++++------------
1 file changed, 17 insertions(+), 12 deletions(-)
diff --git a/migration.c b/migration.c
index d4b8406..9c34627 100644
--- a/migration.c
+++ b/migration.c
@@ -20,6 +20,7 @@
#include "sysemu.h"
#include "block.h"
#include "qemu_socket.h"
+#include "qemu-thread.h"
#include "block-migration.h"
#include "qmp-commands.h"
@@ -334,11 +335,22 @@ ssize_t migrate_fd_put_buffer(MigrationState *s, const void *data,
void migrate_fd_put_ready(MigrationState *s)
{
int ret;
+ static bool first_time = true;
if (s->state != MIG_STATE_ACTIVE) {
DPRINTF("put_ready returning because of non-active state\n");
return;
}
+ if (first_time) {
+ first_time = false;
+ DPRINTF("beginning savevm\n");
+ ret = qemu_savevm_state_begin(s->file, &s->params);
+ if (ret < 0) {
+ DPRINTF("failed, %d\n", ret);
+ migrate_fd_error(s);
+ return;
+ }
+ }
DPRINTF("iterate\n");
ret = qemu_savevm_state_iterate(s->file);
@@ -351,7 +363,11 @@ void migrate_fd_put_ready(MigrationState *s)
DPRINTF("done iterating\n");
start_time = qemu_get_clock_ms(rt_clock);
qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
- vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
+ if (old_vm_running) {
+ vm_stop(RUN_STATE_FINISH_MIGRATE);
+ } else {
+ vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
+ }
if (qemu_savevm_state_complete(s->file) < 0) {
migrate_fd_error(s);
@@ -445,19 +461,8 @@ bool migration_has_failed(MigrationState *s)
void migrate_fd_connect(MigrationState *s)
{
- int ret;
-
s->state = MIG_STATE_ACTIVE;
qemu_fopen_ops_buffered(s);
-
- DPRINTF("beginning savevm\n");
- ret = qemu_savevm_state_begin(s->file, &s->params);
- if (ret < 0) {
- DPRINTF("failed, %d\n", ret);
- migrate_fd_error(s);
- return;
- }
- migrate_fd_put_ready(s);
}
static MigrationState *migrate_init(const MigrationParams *params)
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 07/35] migration: make writes blocking
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (5 preceding siblings ...)
2012-12-11 12:46 ` [Qemu-devel] [PATCH 06/35] migration: stop all cpus correctly Juan Quintela
@ 2012-12-11 12:46 ` Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 08/35] migration: remove unfreeze logic Juan Quintela
` (28 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:46 UTC (permalink / raw)
To: qemu-devel
Move all the writes to the migration_thread, and make writings
blocking. Notice that are still using the iothread for everything
that we do.
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
migration-exec.c | 1 -
migration-fd.c | 1 -
migration-tcp.c | 1 +
migration-unix.c | 1 +
migration.c | 17 -----------------
qemu-file.h | 5 -----
savevm.c | 5 -----
7 files changed, 2 insertions(+), 29 deletions(-)
diff --git a/migration-exec.c b/migration-exec.c
index 2b6fcb4..df76606 100644
--- a/migration-exec.c
+++ b/migration-exec.c
@@ -70,7 +70,6 @@ void exec_start_outgoing_migration(MigrationState *s, const char *command, Error
s->fd = fileno(f);
assert(s->fd != -1);
- socket_set_nonblock(s->fd);
s->opaque = qemu_popen(f, "w");
diff --git a/migration-fd.c b/migration-fd.c
index 5fe28e0..a161338 100644
--- a/migration-fd.c
+++ b/migration-fd.c
@@ -77,7 +77,6 @@ void fd_start_outgoing_migration(MigrationState *s, const char *fdname, Error **
return;
}
- fcntl(s->fd, F_SETFL, O_NONBLOCK);
s->get_error = fd_errno;
s->write = fd_write;
s->close = fd_close;
diff --git a/migration-tcp.c b/migration-tcp.c
index 5e855fe..c0256ec 100644
--- a/migration-tcp.c
+++ b/migration-tcp.c
@@ -61,6 +61,7 @@ static void tcp_wait_for_connect(int fd, void *opaque)
} else {
DPRINTF("migrate connect success\n");
s->fd = fd;
+ socket_set_block(s->fd);
migrate_fd_connect(s);
}
}
diff --git a/migration-unix.c b/migration-unix.c
index dba72b4..536cec8 100644
--- a/migration-unix.c
+++ b/migration-unix.c
@@ -61,6 +61,7 @@ static void unix_wait_for_connect(int fd, void *opaque)
} else {
DPRINTF("migrate connect success\n");
s->fd = fd;
+ socket_set_block(s->fd);
migrate_fd_connect(s);
}
}
diff --git a/migration.c b/migration.c
index 9c34627..17ee872 100644
--- a/migration.c
+++ b/migration.c
@@ -297,18 +297,6 @@ static void migrate_fd_completed(MigrationState *s)
notifier_list_notify(&migration_state_notifiers, s);
}
-static void migrate_fd_put_notify(void *opaque)
-{
- MigrationState *s = opaque;
- int ret;
-
- qemu_set_fd_handler2(s->fd, NULL, NULL, NULL, NULL);
- ret = qemu_file_put_notify(s->file);
- if (ret) {
- migrate_fd_error(s);
- }
-}
-
ssize_t migrate_fd_put_buffer(MigrationState *s, const void *data,
size_t size)
{
@@ -325,10 +313,6 @@ ssize_t migrate_fd_put_buffer(MigrationState *s, const void *data,
if (ret == -1)
ret = -(s->get_error(s));
- if (ret == -EAGAIN) {
- qemu_set_fd_handler2(s->fd, NULL, NULL, migrate_fd_put_notify, s);
- }
-
return ret;
}
@@ -426,7 +410,6 @@ int migrate_fd_close(MigrationState *s)
{
int rc = 0;
if (s->fd != -1) {
- qemu_set_fd_handler2(s->fd, NULL, NULL, NULL, NULL);
rc = s->close(s);
s->fd = -1;
}
diff --git a/qemu-file.h b/qemu-file.h
index d64bdbb..68deefb 100644
--- a/qemu-file.h
+++ b/qemu-file.h
@@ -113,11 +113,6 @@ int64_t qemu_file_set_rate_limit(QEMUFile *f, int64_t new_rate);
int64_t qemu_file_get_rate_limit(QEMUFile *f);
int qemu_file_get_error(QEMUFile *f);
-/* Try to send any outstanding data. This function is useful when output is
- * halted due to rate limiting or EAGAIN errors occur as it can be used to
- * resume output. */
-int qemu_file_put_notify(QEMUFile *f);
-
static inline void qemu_put_be64s(QEMUFile *f, const uint64_t *pv)
{
qemu_put_be64(f, *pv);
diff --git a/savevm.c b/savevm.c
index 5d04d59..c4ee899 100644
--- a/savevm.c
+++ b/savevm.c
@@ -556,11 +556,6 @@ int qemu_fclose(QEMUFile *f)
return ret;
}
-int qemu_file_put_notify(QEMUFile *f)
-{
- return f->ops->put_buffer(f->opaque, NULL, 0, 0);
-}
-
void qemu_put_buffer(QEMUFile *f, const uint8_t *buf, int size)
{
int l;
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 08/35] migration: remove unfreeze logic
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (6 preceding siblings ...)
2012-12-11 12:46 ` [Qemu-devel] [PATCH 07/35] migration: make writes blocking Juan Quintela
@ 2012-12-11 12:46 ` Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 09/35] migration: take finer locking Juan Quintela
` (27 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:46 UTC (permalink / raw)
To: qemu-devel
Now that we have a thread, and blocking writes, we don't need it.
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
buffered_file.c | 24 +-----------------------
migration.c | 23 -----------------------
migration.h | 1 -
3 files changed, 1 insertion(+), 47 deletions(-)
diff --git a/buffered_file.c b/buffered_file.c
index 27610b4..234b968 100644
--- a/buffered_file.c
+++ b/buffered_file.c
@@ -26,7 +26,6 @@ typedef struct QEMUFileBuffered
{
MigrationState *migration_state;
QEMUFile *file;
- int freeze_output;
size_t bytes_xfer;
size_t xfer_limit;
uint8_t *buffer;
@@ -70,13 +69,6 @@ static ssize_t buffered_flush(QEMUFileBuffered *s)
ret = migrate_fd_put_buffer(s->migration_state, s->buffer + offset,
s->buffer_size - offset);
- if (ret == -EAGAIN) {
- DPRINTF("backend not ready, freezing\n");
- ret = 0;
- s->freeze_output = 1;
- break;
- }
-
if (ret <= 0) {
DPRINTF("error flushing data, %zd\n", ret);
break;
@@ -110,9 +102,6 @@ static int buffered_put_buffer(void *opaque, const uint8_t *buf, int64_t pos, in
return error;
}
- DPRINTF("unfreezing output\n");
- s->freeze_output = 0;
-
if (size > 0) {
DPRINTF("buffering %d bytes\n", size - offset);
buffered_append(s, buf, size);
@@ -126,7 +115,7 @@ static int buffered_put_buffer(void *opaque, const uint8_t *buf, int64_t pos, in
if (pos == 0 && size == 0) {
DPRINTF("file is ready\n");
- if (!s->freeze_output && s->bytes_xfer < s->xfer_limit) {
+ if (s->bytes_xfer < s->xfer_limit) {
DPRINTF("notifying client\n");
migrate_fd_put_ready(s->migration_state);
}
@@ -149,12 +138,6 @@ static int buffered_close(void *opaque)
if (ret < 0) {
break;
}
- if (s->freeze_output) {
- ret = migrate_fd_wait_for_unfreeze(s->migration_state);
- if (ret < 0) {
- break;
- }
- }
}
ret2 = migrate_fd_close(s->migration_state);
@@ -188,8 +171,6 @@ static int buffered_rate_limit(void *opaque)
if (ret) {
return ret;
}
- if (s->freeze_output)
- return 1;
if (s->bytes_xfer > s->xfer_limit)
return 1;
@@ -234,9 +215,6 @@ static void *buffered_file_thread(void *opaque)
if (s->migration_state->complete) {
break;
}
- if (s->freeze_output) {
- continue;
- }
if (current_time >= expire_time) {
s->bytes_xfer = 0;
expire_time = current_time + BUFFER_DELAY;
diff --git a/migration.c b/migration.c
index 17ee872..693f329 100644
--- a/migration.c
+++ b/migration.c
@@ -383,29 +383,6 @@ static void migrate_fd_cancel(MigrationState *s)
migrate_fd_cleanup(s);
}
-int migrate_fd_wait_for_unfreeze(MigrationState *s)
-{
- int ret;
-
- DPRINTF("wait for unfreeze\n");
- if (s->state != MIG_STATE_ACTIVE)
- return -EINVAL;
-
- do {
- fd_set wfds;
-
- FD_ZERO(&wfds);
- FD_SET(s->fd, &wfds);
-
- ret = select(s->fd + 1, NULL, &wfds, NULL, NULL);
- } while (ret == -1 && (s->get_error(s)) == EINTR);
-
- if (ret == -1) {
- return -s->get_error(s);
- }
- return 0;
-}
-
int migrate_fd_close(MigrationState *s)
{
int rc = 0;
diff --git a/migration.h b/migration.h
index b66fd60..5ff68eb 100644
--- a/migration.h
+++ b/migration.h
@@ -81,7 +81,6 @@ void migrate_fd_connect(MigrationState *s);
ssize_t migrate_fd_put_buffer(MigrationState *s, const void *data,
size_t size);
void migrate_fd_put_ready(MigrationState *s);
-int migrate_fd_wait_for_unfreeze(MigrationState *s);
int migrate_fd_close(MigrationState *s);
void add_migration_state_change_notifier(Notifier *notify);
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 09/35] migration: take finer locking
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (7 preceding siblings ...)
2012-12-11 12:46 ` [Qemu-devel] [PATCH 08/35] migration: remove unfreeze logic Juan Quintela
@ 2012-12-11 12:46 ` Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 10/35] buffered_file: Unfold the trick to restart generating migration data Juan Quintela
` (26 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:46 UTC (permalink / raw)
To: qemu-devel
Instead of locking the whole migration_thread inside loop, just lock
migration_fd_put_notify, that is what interacts with the rest of the
world.
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
buffered_file.c | 2 --
migration.c | 5 +++++
2 files changed, 5 insertions(+), 2 deletions(-)
diff --git a/buffered_file.c b/buffered_file.c
index 234b968..d68eb66 100644
--- a/buffered_file.c
+++ b/buffered_file.c
@@ -223,9 +223,7 @@ static void *buffered_file_thread(void *opaque)
/* usleep expects microseconds */
usleep((expire_time - current_time)*1000);
}
- qemu_mutex_lock_iothread();
buffered_put_buffer(s, NULL, 0, 0);
- qemu_mutex_unlock_iothread();
}
g_free(s->buffer);
g_free(s);
diff --git a/migration.c b/migration.c
index 693f329..df5ab16 100644
--- a/migration.c
+++ b/migration.c
@@ -321,8 +321,10 @@ void migrate_fd_put_ready(MigrationState *s)
int ret;
static bool first_time = true;
+ qemu_mutex_lock_iothread();
if (s->state != MIG_STATE_ACTIVE) {
DPRINTF("put_ready returning because of non-active state\n");
+ qemu_mutex_unlock_iothread();
return;
}
if (first_time) {
@@ -332,6 +334,7 @@ void migrate_fd_put_ready(MigrationState *s)
if (ret < 0) {
DPRINTF("failed, %d\n", ret);
migrate_fd_error(s);
+ qemu_mutex_unlock_iothread();
return;
}
}
@@ -367,6 +370,8 @@ void migrate_fd_put_ready(MigrationState *s)
}
}
}
+ qemu_mutex_unlock_iothread();
+
}
static void migrate_fd_cancel(MigrationState *s)
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 10/35] buffered_file: Unfold the trick to restart generating migration data
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (8 preceding siblings ...)
2012-12-11 12:46 ` [Qemu-devel] [PATCH 09/35] migration: take finer locking Juan Quintela
@ 2012-12-11 12:46 ` Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 11/35] buffered_file: don't flush on put buffer Juan Quintela
` (25 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:46 UTC (permalink / raw)
To: qemu-devel
This was needed before due to the way that the callbacks worked.
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
buffered_file.c | 19 ++++++++++---------
1 file changed, 10 insertions(+), 9 deletions(-)
diff --git a/buffered_file.c b/buffered_file.c
index d68eb66..9ef30b3 100644
--- a/buffered_file.c
+++ b/buffered_file.c
@@ -113,14 +113,6 @@ static int buffered_put_buffer(void *opaque, const uint8_t *buf, int64_t pos, in
return error;
}
- if (pos == 0 && size == 0) {
- DPRINTF("file is ready\n");
- if (s->bytes_xfer < s->xfer_limit) {
- DPRINTF("notifying client\n");
- migrate_fd_put_ready(s->migration_state);
- }
- }
-
return size;
}
@@ -223,8 +215,17 @@ static void *buffered_file_thread(void *opaque)
/* usleep expects microseconds */
usleep((expire_time - current_time)*1000);
}
- buffered_put_buffer(s, NULL, 0, 0);
+ if (buffered_flush(s) < 0) {
+ break;
+ }
+
+ DPRINTF("file is ready\n");
+ if (s->bytes_xfer < s->xfer_limit) {
+ DPRINTF("notifying client\n");
+ migrate_fd_put_ready(s->migration_state);
+ }
}
+
g_free(s->buffer);
g_free(s);
return NULL;
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 11/35] buffered_file: don't flush on put buffer
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (9 preceding siblings ...)
2012-12-11 12:46 ` [Qemu-devel] [PATCH 10/35] buffered_file: Unfold the trick to restart generating migration data Juan Quintela
@ 2012-12-11 12:46 ` Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 12/35] buffered_file: unfold buffered_append in buffered_put_buffer Juan Quintela
` (24 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:46 UTC (permalink / raw)
To: qemu-devel
We call buffered_put_buffer with iothread held, and buffered_flush() does
synchronous writes. We only want to do the synchronous writes outside.
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
buffered_file.c | 6 ------
1 file changed, 6 deletions(-)
diff --git a/buffered_file.c b/buffered_file.c
index 9ef30b3..85772e1 100644
--- a/buffered_file.c
+++ b/buffered_file.c
@@ -107,12 +107,6 @@ static int buffered_put_buffer(void *opaque, const uint8_t *buf, int64_t pos, in
buffered_append(s, buf, size);
}
- error = buffered_flush(s);
- if (error < 0) {
- DPRINTF("buffered flush error. bailing: %s\n", strerror(-error));
- return error;
- }
-
return size;
}
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 12/35] buffered_file: unfold buffered_append in buffered_put_buffer
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (10 preceding siblings ...)
2012-12-11 12:46 ` [Qemu-devel] [PATCH 11/35] buffered_file: don't flush on put buffer Juan Quintela
@ 2012-12-11 12:46 ` Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 13/35] savevm: New save live migration method: pending Juan Quintela
` (23 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:46 UTC (permalink / raw)
To: qemu-devel
It was the only user, and now buffered_put_buffer just do the append
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
buffered_file.c | 33 ++++++++++++++-------------------
1 file changed, 14 insertions(+), 19 deletions(-)
diff --git a/buffered_file.c b/buffered_file.c
index 85772e1..0b9c316 100644
--- a/buffered_file.c
+++ b/buffered_file.c
@@ -42,22 +42,6 @@ typedef struct QEMUFileBuffered
do { } while (0)
#endif
-static void buffered_append(QEMUFileBuffered *s,
- const uint8_t *buf, size_t size)
-{
- if (size > (s->buffer_capacity - s->buffer_size)) {
- DPRINTF("increasing buffer capacity from %zu by %zu\n",
- s->buffer_capacity, size + 1024);
-
- s->buffer_capacity += size + 1024;
-
- s->buffer = g_realloc(s->buffer, s->buffer_capacity);
- }
-
- memcpy(s->buffer + s->buffer_size, buf, size);
- s->buffer_size += size;
-}
-
static ssize_t buffered_flush(QEMUFileBuffered *s)
{
size_t offset = 0;
@@ -102,11 +86,22 @@ static int buffered_put_buffer(void *opaque, const uint8_t *buf, int64_t pos, in
return error;
}
- if (size > 0) {
- DPRINTF("buffering %d bytes\n", size - offset);
- buffered_append(s, buf, size);
+ if (size <= 0) {
+ return size;
}
+ if (size > (s->buffer_capacity - s->buffer_size)) {
+ DPRINTF("increasing buffer capacity from %zu by %zu\n",
+ s->buffer_capacity, size + 1024);
+
+ s->buffer_capacity += size + 1024;
+
+ s->buffer = g_realloc(s->buffer, s->buffer_capacity);
+ }
+
+ memcpy(s->buffer + s->buffer_size, buf, size);
+ s->buffer_size += size;
+
return size;
}
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 13/35] savevm: New save live migration method: pending
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (11 preceding siblings ...)
2012-12-11 12:46 ` [Qemu-devel] [PATCH 12/35] buffered_file: unfold buffered_append in buffered_put_buffer Juan Quintela
@ 2012-12-11 12:46 ` Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 14/35] migration: include qemu-file.h Juan Quintela
` (22 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:46 UTC (permalink / raw)
To: qemu-devel
Code just now does (simplified for clarity)
if (qemu_savevm_state_iterate(s->file) == 1) {
vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
qemu_savevm_state_complete(s->file);
}
Problem here is that qemu_savevm_state_iterate() returns 1 when it
knows that remaining memory to sent takes less than max downtime.
But this means that we could end spending 2x max_downtime, one
downtime in qemu_savevm_iterate, and the other in
qemu_savevm_state_complete.
Changed code to:
pending_size = qemu_savevm_state_pending(s->file, max_size);
DPRINTF("pending size %lu max %lu\n", pending_size, max_size);
if (pending_size >= max_size) {
ret = qemu_savevm_state_iterate(s->file);
} else {
vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
qemu_savevm_state_complete(s->file);
}
So what we do is: at current network speed, we calculate the maximum
number of bytes we can sent: max_size.
Then we ask every save_live section how much they have pending. If
they are less than max_size, we move to complete phase, otherwise we
do an iterate one.
This makes things much simpler, because now individual sections don't
have to caluclate the bandwidth (it was implossible to do right from
there).
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
arch_init.c | 48 ++++++++++++++++++------------------------------
block-migration.c | 49 ++++++++++---------------------------------------
buffered_file.c | 25 ++++++++++++++++++-------
migration.c | 22 +++++++++++++++-------
migration.h | 2 +-
savevm.c | 19 +++++++++++++++++++
sysemu.h | 1 +
vmstate.h | 1 +
8 files changed, 83 insertions(+), 84 deletions(-)
diff --git a/arch_init.c b/arch_init.c
index 9d1109b..4379108 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -608,12 +608,9 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
static int ram_save_iterate(QEMUFile *f, void *opaque)
{
- uint64_t bytes_transferred_last;
- double bwidth = 0;
int ret;
int i;
- uint64_t expected_downtime;
- MigrationState *s = migrate_get_current();
+ int64_t t0;
qemu_mutex_lock_ramlist();
@@ -621,9 +618,7 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
reset_ram_globals();
}
- bytes_transferred_last = bytes_transferred;
- bwidth = qemu_get_clock_ns(rt_clock);
-
+ t0 = qemu_get_clock_ns(rt_clock);
i = 0;
while ((ret = qemu_file_rate_limit(f)) == 0) {
int bytes_sent;
@@ -641,7 +636,7 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
iterations
*/
if ((i & 63) == 0) {
- uint64_t t1 = (qemu_get_clock_ns(rt_clock) - bwidth) / 1000000;
+ uint64_t t1 = (qemu_get_clock_ns(rt_clock) - t0) / 1000000;
if (t1 > MAX_WAIT) {
DPRINTF("big wait: %" PRIu64 " milliseconds, %d iterations\n",
t1, i);
@@ -655,31 +650,10 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
return ret;
}
- bwidth = qemu_get_clock_ns(rt_clock) - bwidth;
- bwidth = (bytes_transferred - bytes_transferred_last) / bwidth;
-
- /* if we haven't transferred anything this round, force
- * expected_downtime to a very high value, but without
- * crashing */
- if (bwidth == 0) {
- bwidth = 0.000001;
- }
-
qemu_mutex_unlock_ramlist();
qemu_put_be64(f, RAM_SAVE_FLAG_EOS);
- expected_downtime = ram_save_remaining() * TARGET_PAGE_SIZE / bwidth;
- DPRINTF("ram_save_live: expected(%" PRIu64 ") <= max(" PRIu64 ")?\n",
- expected_downtime, migrate_max_downtime());
-
- if (expected_downtime <= migrate_max_downtime()) {
- migration_bitmap_sync();
- expected_downtime = ram_save_remaining() * TARGET_PAGE_SIZE / bwidth;
- s->expected_downtime = expected_downtime / 1000000; /* ns -> ms */
-
- return expected_downtime <= migrate_max_downtime();
- }
- return 0;
+ return i;
}
static int ram_save_complete(QEMUFile *f, void *opaque)
@@ -712,6 +686,19 @@ static int ram_save_complete(QEMUFile *f, void *opaque)
return 0;
}
+static uint64_t ram_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)
+{
+ uint64_t remaining_size;
+
+ remaining_size = ram_save_remaining() * TARGET_PAGE_SIZE;
+
+ if (remaining_size < max_size) {
+ migration_bitmap_sync();
+ remaining_size = ram_save_remaining() * TARGET_PAGE_SIZE;
+ }
+ return remaining_size;
+}
+
static int load_xbzrle(QEMUFile *f, ram_addr_t addr, void *host)
{
int ret, rc = 0;
@@ -898,6 +885,7 @@ SaveVMHandlers savevm_ram_handlers = {
.save_live_setup = ram_save_setup,
.save_live_iterate = ram_save_iterate,
.save_live_complete = ram_save_complete,
+ .save_live_pending = ram_save_pending,
.load_state = ram_load,
.cancel = ram_migration_cancel,
};
diff --git a/block-migration.c b/block-migration.c
index 71b9601..5db01fe 100644
--- a/block-migration.c
+++ b/block-migration.c
@@ -77,9 +77,7 @@ typedef struct BlkMigState {
int64_t total_sector_sum;
int prev_progress;
int bulk_completed;
- long double total_time;
long double prev_time_offset;
- int reads;
} BlkMigState;
static BlkMigState block_mig_state;
@@ -132,12 +130,6 @@ uint64_t blk_mig_bytes_total(void)
return sum << BDRV_SECTOR_BITS;
}
-static inline long double compute_read_bwidth(void)
-{
- assert(block_mig_state.total_time != 0);
- return (block_mig_state.reads / block_mig_state.total_time) * BLOCK_SIZE;
-}
-
static int bmds_aio_inflight(BlkMigDevState *bmds, int64_t sector)
{
int64_t chunk = sector / (int64_t)BDRV_SECTORS_PER_DIRTY_CHUNK;
@@ -191,8 +183,6 @@ static void blk_mig_read_cb(void *opaque, int ret)
blk->ret = ret;
- block_mig_state.reads++;
- block_mig_state.total_time += (curr_time - block_mig_state.prev_time_offset);
block_mig_state.prev_time_offset = curr_time;
QSIMPLEQ_INSERT_TAIL(&block_mig_state.blk_list, blk, entry);
@@ -310,8 +300,6 @@ static void init_blk_migration(QEMUFile *f)
block_mig_state.total_sector_sum = 0;
block_mig_state.prev_progress = -1;
block_mig_state.bulk_completed = 0;
- block_mig_state.total_time = 0;
- block_mig_state.reads = 0;
bdrv_iterate(init_blk_migration_it, NULL);
}
@@ -493,32 +481,6 @@ static int64_t get_remaining_dirty(void)
return dirty * BLOCK_SIZE;
}
-static int is_stage2_completed(void)
-{
- int64_t remaining_dirty;
- long double bwidth;
-
- if (block_mig_state.bulk_completed == 1) {
-
- remaining_dirty = get_remaining_dirty();
- if (remaining_dirty == 0) {
- return 1;
- }
-
- bwidth = compute_read_bwidth();
-
- if ((remaining_dirty / bwidth) <=
- migrate_max_downtime()) {
- /* finish stage2 because we think that we can finish remaining work
- below max_downtime */
-
- return 1;
- }
- }
-
- return 0;
-}
-
static void blk_mig_cleanup(void)
{
BlkMigDevState *bmds;
@@ -619,7 +581,7 @@ static int block_save_iterate(QEMUFile *f, void *opaque)
qemu_put_be64(f, BLK_MIG_FLAG_EOS);
- return is_stage2_completed();
+ return 0;
}
static int block_save_complete(QEMUFile *f, void *opaque)
@@ -659,6 +621,14 @@ static int block_save_complete(QEMUFile *f, void *opaque)
return 0;
}
+static uint64_t block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)
+{
+
+ DPRINTF("Enter save live pending %ld\n", get_remaining_dirty());
+
+ return get_remaining_dirty();
+}
+
static int block_load(QEMUFile *f, void *opaque, int version_id)
{
static int banner_printed;
@@ -755,6 +725,7 @@ SaveVMHandlers savevm_block_handlers = {
.save_live_setup = block_save_setup,
.save_live_iterate = block_save_iterate,
.save_live_complete = block_save_complete,
+ .save_live_pending = block_save_pending,
.load_state = block_load,
.cancel = block_migration_cancel,
.is_active = block_is_active,
diff --git a/buffered_file.c b/buffered_file.c
index 0b9c316..a6d4bd8 100644
--- a/buffered_file.c
+++ b/buffered_file.c
@@ -182,13 +182,15 @@ static int64_t buffered_get_rate_limit(void *opaque)
return s->xfer_limit;
}
-/* 10ms xfer_limit is the limit that we should write each 10ms */
+/* 100ms xfer_limit is the limit that we should write each 100ms */
#define BUFFER_DELAY 100
static void *buffered_file_thread(void *opaque)
{
QEMUFileBuffered *s = opaque;
- int64_t expire_time = qemu_get_clock_ms(rt_clock) + BUFFER_DELAY;
+ int64_t initial_time = qemu_get_clock_ms(rt_clock);
+ int64_t max_size = 0;
+ bool last_round = false;
while (true) {
int64_t current_time = qemu_get_clock_ms(rt_clock);
@@ -196,13 +198,22 @@ static void *buffered_file_thread(void *opaque)
if (s->migration_state->complete) {
break;
}
- if (current_time >= expire_time) {
+ if (current_time >= initial_time + BUFFER_DELAY) {
+ uint64_t transferred_bytes = s->bytes_xfer;
+ uint64_t time_spent = current_time - initial_time;
+ double bandwidth = transferred_bytes / time_spent;
+ max_size = bandwidth * migrate_max_downtime() / 1000000;
+
+ DPRINTF("transferred %" PRIu64 " time_spent %" PRIu64
+ " bandwidth %g max_size %" PRId64 "\n",
+ transferred_bytes, time_spent, bandwidth, max_size);
+
s->bytes_xfer = 0;
- expire_time = current_time + BUFFER_DELAY;
+ initial_time = current_time;
}
- if (s->bytes_xfer >= s->xfer_limit) {
+ if (!last_round && (s->bytes_xfer >= s->xfer_limit)) {
/* usleep expects microseconds */
- usleep((expire_time - current_time)*1000);
+ usleep((initial_time + BUFFER_DELAY - current_time)*1000);
}
if (buffered_flush(s) < 0) {
break;
@@ -211,7 +222,7 @@ static void *buffered_file_thread(void *opaque)
DPRINTF("file is ready\n");
if (s->bytes_xfer < s->xfer_limit) {
DPRINTF("notifying client\n");
- migrate_fd_put_ready(s->migration_state);
+ last_round = migrate_fd_put_ready(s->migration_state, max_size);
}
}
diff --git a/migration.c b/migration.c
index df5ab16..d658022 100644
--- a/migration.c
+++ b/migration.c
@@ -316,16 +316,18 @@ ssize_t migrate_fd_put_buffer(MigrationState *s, const void *data,
return ret;
}
-void migrate_fd_put_ready(MigrationState *s)
+bool migrate_fd_put_ready(MigrationState *s, uint64_t max_size)
{
int ret;
static bool first_time = true;
+ uint64_t pending_size;
+ bool last_round = false;
qemu_mutex_lock_iothread();
if (s->state != MIG_STATE_ACTIVE) {
DPRINTF("put_ready returning because of non-active state\n");
qemu_mutex_unlock_iothread();
- return;
+ return false;
}
if (first_time) {
first_time = false;
@@ -335,15 +337,19 @@ void migrate_fd_put_ready(MigrationState *s)
DPRINTF("failed, %d\n", ret);
migrate_fd_error(s);
qemu_mutex_unlock_iothread();
- return;
+ return false;
}
}
DPRINTF("iterate\n");
- ret = qemu_savevm_state_iterate(s->file);
- if (ret < 0) {
- migrate_fd_error(s);
- } else if (ret == 1) {
+ pending_size = qemu_savevm_state_pending(s->file, max_size);
+ DPRINTF("pending size %lu max %lu\n", pending_size, max_size);
+ if (pending_size >= max_size) {
+ ret = qemu_savevm_state_iterate(s->file);
+ if (ret < 0) {
+ migrate_fd_error(s);
+ }
+ } else {
int old_vm_running = runstate_is_running();
int64_t start_time, end_time;
@@ -369,9 +375,11 @@ void migrate_fd_put_ready(MigrationState *s)
vm_start();
}
}
+ last_round = true;
}
qemu_mutex_unlock_iothread();
+ return last_round;
}
static void migrate_fd_cancel(MigrationState *s)
diff --git a/migration.h b/migration.h
index 5ff68eb..72e638b 100644
--- a/migration.h
+++ b/migration.h
@@ -80,7 +80,7 @@ void migrate_fd_connect(MigrationState *s);
ssize_t migrate_fd_put_buffer(MigrationState *s, const void *data,
size_t size);
-void migrate_fd_put_ready(MigrationState *s);
+bool migrate_fd_put_ready(MigrationState *s, uint64_t max_size);
int migrate_fd_close(MigrationState *s);
void add_migration_state_change_notifier(Notifier *notify);
diff --git a/savevm.c b/savevm.c
index c4ee899..f6e91cf 100644
--- a/savevm.c
+++ b/savevm.c
@@ -1754,6 +1754,25 @@ int qemu_savevm_state_complete(QEMUFile *f)
return qemu_file_get_error(f);
}
+uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size)
+{
+ SaveStateEntry *se;
+ uint64_t ret = 0;
+
+ QTAILQ_FOREACH(se, &savevm_handlers, entry) {
+ if (!se->ops || !se->ops->save_live_pending) {
+ continue;
+ }
+ if (se->ops && se->ops->is_active) {
+ if (!se->ops->is_active(se->opaque)) {
+ continue;
+ }
+ }
+ ret += se->ops->save_live_pending(f, se->opaque, max_size);
+ }
+ return ret;
+}
+
void qemu_savevm_state_cancel(QEMUFile *f)
{
SaveStateEntry *se;
diff --git a/sysemu.h b/sysemu.h
index 1b6add2..7832c69 100644
--- a/sysemu.h
+++ b/sysemu.h
@@ -78,6 +78,7 @@ int qemu_savevm_state_begin(QEMUFile *f,
int qemu_savevm_state_iterate(QEMUFile *f);
int qemu_savevm_state_complete(QEMUFile *f);
void qemu_savevm_state_cancel(QEMUFile *f);
+uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size);
int qemu_loadvm_state(QEMUFile *f);
/* SLIRP */
diff --git a/vmstate.h b/vmstate.h
index 623af0a..f27276c 100644
--- a/vmstate.h
+++ b/vmstate.h
@@ -35,6 +35,7 @@ typedef struct SaveVMHandlers {
int (*save_live_setup)(QEMUFile *f, void *opaque);
int (*save_live_iterate)(QEMUFile *f, void *opaque);
int (*save_live_complete)(QEMUFile *f, void *opaque);
+ uint64_t (*save_live_pending)(QEMUFile *f, void *opaque, uint64_t max_size);
void (*cancel)(void *opaque);
LoadStateHandler *load_state;
bool (*is_active)(void *opaque);
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 14/35] migration: include qemu-file.h
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (12 preceding siblings ...)
2012-12-11 12:46 ` [Qemu-devel] [PATCH 13/35] savevm: New save live migration method: pending Juan Quintela
@ 2012-12-11 12:46 ` Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 15/35] migration-fd: remove duplicate include Juan Quintela
` (21 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:46 UTC (permalink / raw)
To: qemu-devel
They don't use/know anything about buffered-file.
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
migration-exec.c | 2 +-
migration-fd.c | 2 +-
migration-tcp.c | 2 +-
migration-unix.c | 2 +-
4 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/migration-exec.c b/migration-exec.c
index df76606..0997a24 100644
--- a/migration-exec.c
+++ b/migration-exec.c
@@ -19,7 +19,7 @@
#include "qemu_socket.h"
#include "migration.h"
#include "qemu-char.h"
-#include "buffered_file.h"
+#include "qemu-file.h"
#include "block.h"
#include <sys/types.h>
#include <sys/wait.h>
diff --git a/migration-fd.c b/migration-fd.c
index a161338..487b17e 100644
--- a/migration-fd.c
+++ b/migration-fd.c
@@ -18,7 +18,7 @@
#include "migration.h"
#include "monitor.h"
#include "qemu-char.h"
-#include "buffered_file.h"
+#include "qemu-file.h"
#include "block.h"
#include "qemu_socket.h"
diff --git a/migration-tcp.c b/migration-tcp.c
index c0256ec..5808857 100644
--- a/migration-tcp.c
+++ b/migration-tcp.c
@@ -17,7 +17,7 @@
#include "qemu_socket.h"
#include "migration.h"
#include "qemu-char.h"
-#include "buffered_file.h"
+#include "qemu-file.h"
#include "block.h"
//#define DEBUG_MIGRATION_TCP
diff --git a/migration-unix.c b/migration-unix.c
index 536cec8..81a8176 100644
--- a/migration-unix.c
+++ b/migration-unix.c
@@ -17,7 +17,7 @@
#include "qemu_socket.h"
#include "migration.h"
#include "qemu-char.h"
-#include "buffered_file.h"
+#include "qemu-file.h"
#include "block.h"
//#define DEBUG_MIGRATION_UNIX
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 15/35] migration-fd: remove duplicate include
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (13 preceding siblings ...)
2012-12-11 12:46 ` [Qemu-devel] [PATCH 14/35] migration: include qemu-file.h Juan Quintela
@ 2012-12-11 12:46 ` Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 16/35] migration: move buffered_file.c code into migration.c Juan Quintela
` (20 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:46 UTC (permalink / raw)
To: qemu-devel
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
migration-fd.c | 1 -
1 file changed, 1 deletion(-)
diff --git a/migration-fd.c b/migration-fd.c
index 487b17e..77aef6d 100644
--- a/migration-fd.c
+++ b/migration-fd.c
@@ -20,7 +20,6 @@
#include "qemu-char.h"
#include "qemu-file.h"
#include "block.h"
-#include "qemu_socket.h"
//#define DEBUG_MIGRATION_FD
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 16/35] migration: move buffered_file.c code into migration.c
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (14 preceding siblings ...)
2012-12-11 12:46 ` [Qemu-devel] [PATCH 15/35] migration-fd: remove duplicate include Juan Quintela
@ 2012-12-11 12:46 ` Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 17/35] migration: move migration_fd_put_ready() Juan Quintela
` (19 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:46 UTC (permalink / raw)
To: qemu-devel
This only moves the code (also from buffered_file.h to migration.h).
Fix whitespace until checkpatch is happy.
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
Makefile.objs | 2 +-
buffered_file.c | 259 --------------------------------------------------------
buffered_file.h | 22 -----
migration.c | 233 +++++++++++++++++++++++++++++++++++++++++++++++++-
migration.h | 1 +
5 files changed, 234 insertions(+), 283 deletions(-)
delete mode 100644 buffered_file.c
delete mode 100644 buffered_file.h
diff --git a/Makefile.objs b/Makefile.objs
index 3c7abca..f0309ac 100644
--- a/Makefile.objs
+++ b/Makefile.objs
@@ -79,7 +79,7 @@ extra-obj-$(CONFIG_LINUX) += fsdev/
common-obj-y += tcg-runtime.o host-utils.o main-loop.o
common-obj-y += input.o
-common-obj-y += buffered_file.o migration.o migration-tcp.o
+common-obj-y += migration.o migration-tcp.o
common-obj-y += qemu-char.o #aio.o
common-obj-y += block-migration.o iohandler.o
common-obj-y += bitmap.o bitops.o
diff --git a/buffered_file.c b/buffered_file.c
deleted file mode 100644
index a6d4bd8..0000000
--- a/buffered_file.c
+++ /dev/null
@@ -1,259 +0,0 @@
-/*
- * QEMU buffered QEMUFile
- *
- * Copyright IBM, Corp. 2008
- *
- * Authors:
- * Anthony Liguori <aliguori@us.ibm.com>
- *
- * This work is licensed under the terms of the GNU GPL, version 2. See
- * the COPYING file in the top-level directory.
- *
- * Contributions after 2012-01-13 are licensed under the terms of the
- * GNU GPL, version 2 or (at your option) any later version.
- */
-
-#include "qemu-common.h"
-#include "hw/hw.h"
-#include "qemu-timer.h"
-#include "qemu-char.h"
-#include "buffered_file.h"
-#include "qemu-thread.h"
-
-//#define DEBUG_BUFFERED_FILE
-
-typedef struct QEMUFileBuffered
-{
- MigrationState *migration_state;
- QEMUFile *file;
- size_t bytes_xfer;
- size_t xfer_limit;
- uint8_t *buffer;
- size_t buffer_size;
- size_t buffer_capacity;
- QemuThread thread;
-} QEMUFileBuffered;
-
-#ifdef DEBUG_BUFFERED_FILE
-#define DPRINTF(fmt, ...) \
- do { printf("buffered-file: " fmt, ## __VA_ARGS__); } while (0)
-#else
-#define DPRINTF(fmt, ...) \
- do { } while (0)
-#endif
-
-static ssize_t buffered_flush(QEMUFileBuffered *s)
-{
- size_t offset = 0;
- ssize_t ret = 0;
-
- DPRINTF("flushing %zu byte(s) of data\n", s->buffer_size);
-
- while (s->bytes_xfer < s->xfer_limit && offset < s->buffer_size) {
-
- ret = migrate_fd_put_buffer(s->migration_state, s->buffer + offset,
- s->buffer_size - offset);
- if (ret <= 0) {
- DPRINTF("error flushing data, %zd\n", ret);
- break;
- } else {
- DPRINTF("flushed %zd byte(s)\n", ret);
- offset += ret;
- s->bytes_xfer += ret;
- }
- }
-
- DPRINTF("flushed %zu of %zu byte(s)\n", offset, s->buffer_size);
- memmove(s->buffer, s->buffer + offset, s->buffer_size - offset);
- s->buffer_size -= offset;
-
- if (ret < 0) {
- return ret;
- }
- return offset;
-}
-
-static int buffered_put_buffer(void *opaque, const uint8_t *buf, int64_t pos, int size)
-{
- QEMUFileBuffered *s = opaque;
- ssize_t error;
-
- DPRINTF("putting %d bytes at %" PRId64 "\n", size, pos);
-
- error = qemu_file_get_error(s->file);
- if (error) {
- DPRINTF("flush when error, bailing: %s\n", strerror(-error));
- return error;
- }
-
- if (size <= 0) {
- return size;
- }
-
- if (size > (s->buffer_capacity - s->buffer_size)) {
- DPRINTF("increasing buffer capacity from %zu by %zu\n",
- s->buffer_capacity, size + 1024);
-
- s->buffer_capacity += size + 1024;
-
- s->buffer = g_realloc(s->buffer, s->buffer_capacity);
- }
-
- memcpy(s->buffer + s->buffer_size, buf, size);
- s->buffer_size += size;
-
- return size;
-}
-
-static int buffered_close(void *opaque)
-{
- QEMUFileBuffered *s = opaque;
- ssize_t ret = 0;
- int ret2;
-
- DPRINTF("closing\n");
-
- s->xfer_limit = INT_MAX;
- while (!qemu_file_get_error(s->file) && s->buffer_size) {
- ret = buffered_flush(s);
- if (ret < 0) {
- break;
- }
- }
-
- ret2 = migrate_fd_close(s->migration_state);
- if (ret >= 0) {
- ret = ret2;
- }
- ret = migrate_fd_close(s->migration_state);
- s->migration_state->complete = true;
- return ret;
-}
-
-/*
- * The meaning of the return values is:
- * 0: We can continue sending
- * 1: Time to stop
- * negative: There has been an error
- */
-static int buffered_get_fd(void *opaque)
-{
- QEMUFileBuffered *s = opaque;
-
- return qemu_get_fd(s->file);
-}
-
-static int buffered_rate_limit(void *opaque)
-{
- QEMUFileBuffered *s = opaque;
- int ret;
-
- ret = qemu_file_get_error(s->file);
- if (ret) {
- return ret;
- }
-
- if (s->bytes_xfer > s->xfer_limit)
- return 1;
-
- return 0;
-}
-
-static int64_t buffered_set_rate_limit(void *opaque, int64_t new_rate)
-{
- QEMUFileBuffered *s = opaque;
- if (qemu_file_get_error(s->file)) {
- goto out;
- }
- if (new_rate > SIZE_MAX) {
- new_rate = SIZE_MAX;
- }
-
- s->xfer_limit = new_rate / 10;
-
-out:
- return s->xfer_limit;
-}
-
-static int64_t buffered_get_rate_limit(void *opaque)
-{
- QEMUFileBuffered *s = opaque;
-
- return s->xfer_limit;
-}
-
-/* 100ms xfer_limit is the limit that we should write each 100ms */
-#define BUFFER_DELAY 100
-
-static void *buffered_file_thread(void *opaque)
-{
- QEMUFileBuffered *s = opaque;
- int64_t initial_time = qemu_get_clock_ms(rt_clock);
- int64_t max_size = 0;
- bool last_round = false;
-
- while (true) {
- int64_t current_time = qemu_get_clock_ms(rt_clock);
-
- if (s->migration_state->complete) {
- break;
- }
- if (current_time >= initial_time + BUFFER_DELAY) {
- uint64_t transferred_bytes = s->bytes_xfer;
- uint64_t time_spent = current_time - initial_time;
- double bandwidth = transferred_bytes / time_spent;
- max_size = bandwidth * migrate_max_downtime() / 1000000;
-
- DPRINTF("transferred %" PRIu64 " time_spent %" PRIu64
- " bandwidth %g max_size %" PRId64 "\n",
- transferred_bytes, time_spent, bandwidth, max_size);
-
- s->bytes_xfer = 0;
- initial_time = current_time;
- }
- if (!last_round && (s->bytes_xfer >= s->xfer_limit)) {
- /* usleep expects microseconds */
- usleep((initial_time + BUFFER_DELAY - current_time)*1000);
- }
- if (buffered_flush(s) < 0) {
- break;
- }
-
- DPRINTF("file is ready\n");
- if (s->bytes_xfer < s->xfer_limit) {
- DPRINTF("notifying client\n");
- last_round = migrate_fd_put_ready(s->migration_state, max_size);
- }
- }
-
- g_free(s->buffer);
- g_free(s);
- return NULL;
-}
-
-static const QEMUFileOps buffered_file_ops = {
- .get_fd = buffered_get_fd,
- .put_buffer = buffered_put_buffer,
- .close = buffered_close,
- .rate_limit = buffered_rate_limit,
- .get_rate_limit = buffered_get_rate_limit,
- .set_rate_limit = buffered_set_rate_limit,
-};
-
-void qemu_fopen_ops_buffered(MigrationState *migration_state)
-{
- QEMUFileBuffered *s;
-
- s = g_malloc0(sizeof(*s));
-
- s->migration_state = migration_state;
- s->xfer_limit = migration_state->bandwidth_limit / 10;
- s->migration_state->complete = false;
-
- s->file = qemu_fopen_ops(s, &buffered_file_ops);
-
- migration_state->file = s->file;
-
- qemu_thread_create(&s->thread, buffered_file_thread, s,
- QEMU_THREAD_DETACHED);
-}
diff --git a/buffered_file.h b/buffered_file.h
deleted file mode 100644
index 8a246fd..0000000
--- a/buffered_file.h
+++ /dev/null
@@ -1,22 +0,0 @@
-/*
- * QEMU buffered QEMUFile
- *
- * Copyright IBM, Corp. 2008
- *
- * Authors:
- * Anthony Liguori <aliguori@us.ibm.com>
- *
- * This work is licensed under the terms of the GNU GPL, version 2. See
- * the COPYING file in the top-level directory.
- *
- */
-
-#ifndef QEMU_BUFFERED_FILE_H
-#define QEMU_BUFFERED_FILE_H
-
-#include "hw/hw.h"
-#include "migration.h"
-
-void qemu_fopen_ops_buffered(MigrationState *migration_state);
-
-#endif
diff --git a/migration.c b/migration.c
index d658022..6b33e83 100644
--- a/migration.c
+++ b/migration.c
@@ -16,7 +16,7 @@
#include "qemu-common.h"
#include "migration.h"
#include "monitor.h"
-#include "buffered_file.h"
+#include "qemu-file.h"
#include "sysemu.h"
#include "block.h"
#include "qemu_socket.h"
@@ -587,3 +587,234 @@ int64_t migrate_xbzrle_cache_size(void)
return s->xbzrle_cache_size;
}
+
+/* migration thread support */
+
+typedef struct QEMUFileBuffered {
+ MigrationState *migration_state;
+ QEMUFile *file;
+ size_t bytes_xfer;
+ size_t xfer_limit;
+ uint8_t *buffer;
+ size_t buffer_size;
+ size_t buffer_capacity;
+ QemuThread thread;
+} QEMUFileBuffered;
+
+static ssize_t buffered_flush(QEMUFileBuffered *s)
+{
+ size_t offset = 0;
+ ssize_t ret = 0;
+
+ DPRINTF("flushing %zu byte(s) of data\n", s->buffer_size);
+
+ while (s->bytes_xfer < s->xfer_limit && offset < s->buffer_size) {
+
+ ret = migrate_fd_put_buffer(s->migration_state, s->buffer + offset,
+ s->buffer_size - offset);
+ if (ret <= 0) {
+ DPRINTF("error flushing data, %zd\n", ret);
+ break;
+ } else {
+ DPRINTF("flushed %zd byte(s)\n", ret);
+ offset += ret;
+ s->bytes_xfer += ret;
+ }
+ }
+
+ DPRINTF("flushed %zu of %zu byte(s)\n", offset, s->buffer_size);
+ memmove(s->buffer, s->buffer + offset, s->buffer_size - offset);
+ s->buffer_size -= offset;
+
+ if (ret < 0) {
+ return ret;
+ }
+ return offset;
+}
+
+static int buffered_put_buffer(void *opaque, const uint8_t *buf,
+ int64_t pos, int size)
+{
+ QEMUFileBuffered *s = opaque;
+ ssize_t error;
+
+ DPRINTF("putting %d bytes at %" PRId64 "\n", size, pos);
+
+ error = qemu_file_get_error(s->file);
+ if (error) {
+ DPRINTF("flush when error, bailing: %s\n", strerror(-error));
+ return error;
+ }
+
+ if (size <= 0) {
+ return size;
+ }
+
+ if (size > (s->buffer_capacity - s->buffer_size)) {
+ DPRINTF("increasing buffer capacity from %zu by %zu\n",
+ s->buffer_capacity, size + 1024);
+
+ s->buffer_capacity += size + 1024;
+
+ s->buffer = g_realloc(s->buffer, s->buffer_capacity);
+ }
+
+ memcpy(s->buffer + s->buffer_size, buf, size);
+ s->buffer_size += size;
+
+ return size;
+}
+
+static int buffered_close(void *opaque)
+{
+ QEMUFileBuffered *s = opaque;
+ ssize_t ret = 0;
+ int ret2;
+
+ DPRINTF("closing\n");
+
+ s->xfer_limit = INT_MAX;
+ while (!qemu_file_get_error(s->file) && s->buffer_size) {
+ ret = buffered_flush(s);
+ if (ret < 0) {
+ break;
+ }
+ }
+
+ ret2 = migrate_fd_close(s->migration_state);
+ if (ret >= 0) {
+ ret = ret2;
+ }
+ ret = migrate_fd_close(s->migration_state);
+ s->migration_state->complete = true;
+ return ret;
+}
+
+static int buffered_get_fd(void *opaque)
+{
+ QEMUFileBuffered *s = opaque;
+
+ return qemu_get_fd(s->file);
+}
+
+/*
+ * The meaning of the return values is:
+ * 0: We can continue sending
+ * 1: Time to stop
+ * negative: There has been an error
+ */
+static int buffered_rate_limit(void *opaque)
+{
+ QEMUFileBuffered *s = opaque;
+ int ret;
+
+ ret = qemu_file_get_error(s->file);
+ if (ret) {
+ return ret;
+ }
+
+ if (s->bytes_xfer > s->xfer_limit) {
+ return 1;
+ }
+
+ return 0;
+}
+
+static int64_t buffered_set_rate_limit(void *opaque, int64_t new_rate)
+{
+ QEMUFileBuffered *s = opaque;
+ if (qemu_file_get_error(s->file)) {
+ goto out;
+ }
+ if (new_rate > SIZE_MAX) {
+ new_rate = SIZE_MAX;
+ }
+
+ s->xfer_limit = new_rate / 10;
+
+out:
+ return s->xfer_limit;
+}
+
+static int64_t buffered_get_rate_limit(void *opaque)
+{
+ QEMUFileBuffered *s = opaque;
+
+ return s->xfer_limit;
+}
+
+/* 100ms xfer_limit is the limit that we should write each 100ms */
+#define BUFFER_DELAY 100
+
+static void *buffered_file_thread(void *opaque)
+{
+ QEMUFileBuffered *s = opaque;
+ int64_t initial_time = qemu_get_clock_ms(rt_clock);
+ int64_t max_size = 0;
+ bool last_round = false;
+
+ while (true) {
+ int64_t current_time = qemu_get_clock_ms(rt_clock);
+
+ if (s->migration_state->complete) {
+ break;
+ }
+ if (current_time >= initial_time + BUFFER_DELAY) {
+ uint64_t transferred_bytes = s->bytes_xfer;
+ uint64_t time_spent = current_time - initial_time;
+ double bandwidth = transferred_bytes / time_spent;
+ max_size = bandwidth * migrate_max_downtime() / 1000000;
+
+ DPRINTF("transferred %" PRIu64 " time_spent %" PRIu64
+ " bandwidth %g max_size %" PRId64 "\n",
+ transferred_bytes, time_spent, bandwidth, max_size);
+
+ s->bytes_xfer = 0;
+ initial_time = current_time;
+ }
+ if (!last_round && (s->bytes_xfer >= s->xfer_limit)) {
+ /* usleep expects microseconds */
+ usleep((initial_time + BUFFER_DELAY - current_time)*1000);
+ }
+ if (buffered_flush(s) < 0) {
+ break;
+ }
+
+ DPRINTF("file is ready\n");
+ if (s->bytes_xfer < s->xfer_limit) {
+ DPRINTF("notifying client\n");
+ last_round = migrate_fd_put_ready(s->migration_state, max_size);
+ }
+ }
+
+ g_free(s->buffer);
+ g_free(s);
+ return NULL;
+}
+
+static const QEMUFileOps buffered_file_ops = {
+ .get_fd = buffered_get_fd,
+ .put_buffer = buffered_put_buffer,
+ .close = buffered_close,
+ .rate_limit = buffered_rate_limit,
+ .get_rate_limit = buffered_get_rate_limit,
+ .set_rate_limit = buffered_set_rate_limit,
+};
+
+void qemu_fopen_ops_buffered(MigrationState *migration_state)
+{
+ QEMUFileBuffered *s;
+
+ s = g_malloc0(sizeof(*s));
+
+ s->migration_state = migration_state;
+ s->xfer_limit = migration_state->bandwidth_limit / 10;
+ s->migration_state->complete = false;
+
+ s->file = qemu_fopen_ops(s, &buffered_file_ops);
+
+ migration_state->file = s->file;
+
+ qemu_thread_create(&s->thread, buffered_file_thread, s,
+ QEMU_THREAD_DETACHED);
+}
diff --git a/migration.h b/migration.h
index 72e638b..1b7ca4b 100644
--- a/migration.h
+++ b/migration.h
@@ -128,4 +128,5 @@ int64_t migrate_xbzrle_cache_size(void);
int64_t xbzrle_cache_resize(int64_t new_size);
+void qemu_fopen_ops_buffered(MigrationState *migration_state);
#endif
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 17/35] migration: move migration_fd_put_ready()
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (15 preceding siblings ...)
2012-12-11 12:46 ` [Qemu-devel] [PATCH 16/35] migration: move buffered_file.c code into migration.c Juan Quintela
@ 2012-12-11 12:46 ` Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 18/35] migration: Inline qemu_fopen_ops_buffered into migrate_fd_connect Juan Quintela
` (18 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:46 UTC (permalink / raw)
To: qemu-devel
Put it near its use and un-export it.
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
migration.c | 132 ++++++++++++++++++++++++++++++------------------------------
migration.h | 1 -
2 files changed, 66 insertions(+), 67 deletions(-)
diff --git a/migration.c b/migration.c
index 6b33e83..34ba6ab 100644
--- a/migration.c
+++ b/migration.c
@@ -316,72 +316,6 @@ ssize_t migrate_fd_put_buffer(MigrationState *s, const void *data,
return ret;
}
-bool migrate_fd_put_ready(MigrationState *s, uint64_t max_size)
-{
- int ret;
- static bool first_time = true;
- uint64_t pending_size;
- bool last_round = false;
-
- qemu_mutex_lock_iothread();
- if (s->state != MIG_STATE_ACTIVE) {
- DPRINTF("put_ready returning because of non-active state\n");
- qemu_mutex_unlock_iothread();
- return false;
- }
- if (first_time) {
- first_time = false;
- DPRINTF("beginning savevm\n");
- ret = qemu_savevm_state_begin(s->file, &s->params);
- if (ret < 0) {
- DPRINTF("failed, %d\n", ret);
- migrate_fd_error(s);
- qemu_mutex_unlock_iothread();
- return false;
- }
- }
-
- DPRINTF("iterate\n");
- pending_size = qemu_savevm_state_pending(s->file, max_size);
- DPRINTF("pending size %lu max %lu\n", pending_size, max_size);
- if (pending_size >= max_size) {
- ret = qemu_savevm_state_iterate(s->file);
- if (ret < 0) {
- migrate_fd_error(s);
- }
- } else {
- int old_vm_running = runstate_is_running();
- int64_t start_time, end_time;
-
- DPRINTF("done iterating\n");
- start_time = qemu_get_clock_ms(rt_clock);
- qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
- if (old_vm_running) {
- vm_stop(RUN_STATE_FINISH_MIGRATE);
- } else {
- vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
- }
-
- if (qemu_savevm_state_complete(s->file) < 0) {
- migrate_fd_error(s);
- } else {
- migrate_fd_completed(s);
- }
- end_time = qemu_get_clock_ms(rt_clock);
- s->total_time = end_time - s->total_time;
- s->downtime = end_time - start_time;
- if (s->state != MIG_STATE_COMPLETED) {
- if (old_vm_running) {
- vm_start();
- }
- }
- last_round = true;
- }
- qemu_mutex_unlock_iothread();
-
- return last_round;
-}
-
static void migrate_fd_cancel(MigrationState *s)
{
if (s->state != MIG_STATE_ACTIVE)
@@ -743,6 +677,72 @@ static int64_t buffered_get_rate_limit(void *opaque)
return s->xfer_limit;
}
+static bool migrate_fd_put_ready(MigrationState *s, uint64_t max_size)
+{
+ int ret;
+ static bool first_time = true;
+ uint64_t pending_size;
+ bool last_round = false;
+
+ qemu_mutex_lock_iothread();
+ if (s->state != MIG_STATE_ACTIVE) {
+ DPRINTF("put_ready returning because of non-active state\n");
+ qemu_mutex_unlock_iothread();
+ return false;
+ }
+ if (first_time) {
+ first_time = false;
+ DPRINTF("beginning savevm\n");
+ ret = qemu_savevm_state_begin(s->file, &s->params);
+ if (ret < 0) {
+ DPRINTF("failed, %d\n", ret);
+ migrate_fd_error(s);
+ qemu_mutex_unlock_iothread();
+ return false;
+ }
+ }
+
+ DPRINTF("iterate\n");
+ pending_size = qemu_savevm_state_pending(s->file, max_size);
+ DPRINTF("pending size %lu max %lu\n", pending_size, max_size);
+ if (pending_size >= max_size) {
+ ret = qemu_savevm_state_iterate(s->file);
+ if (ret < 0) {
+ migrate_fd_error(s);
+ }
+ } else {
+ int old_vm_running = runstate_is_running();
+ int64_t start_time, end_time;
+
+ DPRINTF("done iterating\n");
+ start_time = qemu_get_clock_ms(rt_clock);
+ qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
+ if (old_vm_running) {
+ vm_stop(RUN_STATE_FINISH_MIGRATE);
+ } else {
+ vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
+ }
+
+ if (qemu_savevm_state_complete(s->file) < 0) {
+ migrate_fd_error(s);
+ } else {
+ migrate_fd_completed(s);
+ }
+ end_time = qemu_get_clock_ms(rt_clock);
+ s->total_time = end_time - s->total_time;
+ s->downtime = end_time - start_time;
+ if (s->state != MIG_STATE_COMPLETED) {
+ if (old_vm_running) {
+ vm_start();
+ }
+ }
+ last_round = true;
+ }
+ qemu_mutex_unlock_iothread();
+
+ return last_round;
+}
+
/* 100ms xfer_limit is the limit that we should write each 100ms */
#define BUFFER_DELAY 100
diff --git a/migration.h b/migration.h
index 1b7ca4b..e95d37a 100644
--- a/migration.h
+++ b/migration.h
@@ -80,7 +80,6 @@ void migrate_fd_connect(MigrationState *s);
ssize_t migrate_fd_put_buffer(MigrationState *s, const void *data,
size_t size);
-bool migrate_fd_put_ready(MigrationState *s, uint64_t max_size);
int migrate_fd_close(MigrationState *s);
void add_migration_state_change_notifier(Notifier *notify);
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 18/35] migration: Inline qemu_fopen_ops_buffered into migrate_fd_connect
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (16 preceding siblings ...)
2012-12-11 12:46 ` [Qemu-devel] [PATCH 17/35] migration: move migration_fd_put_ready() Juan Quintela
@ 2012-12-11 12:46 ` Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 19/35] migration: move migration notifier Juan Quintela
` (17 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:46 UTC (permalink / raw)
To: qemu-devel
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
migration.c | 9 ++-------
migration.h | 2 --
2 files changed, 2 insertions(+), 9 deletions(-)
diff --git a/migration.c b/migration.c
index 34ba6ab..371ff0c 100644
--- a/migration.c
+++ b/migration.c
@@ -366,12 +366,6 @@ bool migration_has_failed(MigrationState *s)
s->state == MIG_STATE_ERROR);
}
-void migrate_fd_connect(MigrationState *s)
-{
- s->state = MIG_STATE_ACTIVE;
- qemu_fopen_ops_buffered(s);
-}
-
static MigrationState *migrate_init(const MigrationParams *params)
{
MigrationState *s = migrate_get_current();
@@ -801,10 +795,11 @@ static const QEMUFileOps buffered_file_ops = {
.set_rate_limit = buffered_set_rate_limit,
};
-void qemu_fopen_ops_buffered(MigrationState *migration_state)
+void migrate_fd_connect(MigrationState *migration_state)
{
QEMUFileBuffered *s;
+ migration_state->state = MIG_STATE_ACTIVE;
s = g_malloc0(sizeof(*s));
s->migration_state = migration_state;
diff --git a/migration.h b/migration.h
index e95d37a..63da641 100644
--- a/migration.h
+++ b/migration.h
@@ -126,6 +126,4 @@ int migrate_use_xbzrle(void);
int64_t migrate_xbzrle_cache_size(void);
int64_t xbzrle_cache_resize(int64_t new_size);
-
-void qemu_fopen_ops_buffered(MigrationState *migration_state);
#endif
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 19/35] migration: move migration notifier
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (17 preceding siblings ...)
2012-12-11 12:46 ` [Qemu-devel] [PATCH 18/35] migration: Inline qemu_fopen_ops_buffered into migrate_fd_connect Juan Quintela
@ 2012-12-11 12:46 ` Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 20/35] migration: move begining stage to the migration thread Juan Quintela
` (16 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:46 UTC (permalink / raw)
To: qemu-devel
At this point, it is waranteed that state is ACTIVE. Old position
didn't assured hat.
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
migration.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/migration.c b/migration.c
index 371ff0c..913f3bc 100644
--- a/migration.c
+++ b/migration.c
@@ -450,8 +450,6 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
error_propagate(errp, local_err);
return;
}
-
- notifier_list_notify(&migration_state_notifiers, s);
}
void qmp_migrate_cancel(Error **errp)
@@ -812,4 +810,5 @@ void migrate_fd_connect(MigrationState *migration_state)
qemu_thread_create(&s->thread, buffered_file_thread, s,
QEMU_THREAD_DETACHED);
+ notifier_list_notify(&migration_state_notifiers, s);
}
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 20/35] migration: move begining stage to the migration thread
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (18 preceding siblings ...)
2012-12-11 12:46 ` [Qemu-devel] [PATCH 19/35] migration: move migration notifier Juan Quintela
@ 2012-12-11 12:46 ` Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 21/35] migration: Add buffered_flush error handling Juan Quintela
` (15 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:46 UTC (permalink / raw)
To: qemu-devel
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
migration.c | 30 +++++++++++++++++-------------
1 file changed, 17 insertions(+), 13 deletions(-)
diff --git a/migration.c b/migration.c
index 913f3bc..336941b 100644
--- a/migration.c
+++ b/migration.c
@@ -672,7 +672,6 @@ static int64_t buffered_get_rate_limit(void *opaque)
static bool migrate_fd_put_ready(MigrationState *s, uint64_t max_size)
{
int ret;
- static bool first_time = true;
uint64_t pending_size;
bool last_round = false;
@@ -682,17 +681,6 @@ static bool migrate_fd_put_ready(MigrationState *s, uint64_t max_size)
qemu_mutex_unlock_iothread();
return false;
}
- if (first_time) {
- first_time = false;
- DPRINTF("beginning savevm\n");
- ret = qemu_savevm_state_begin(s->file, &s->params);
- if (ret < 0) {
- DPRINTF("failed, %d\n", ret);
- migrate_fd_error(s);
- qemu_mutex_unlock_iothread();
- return false;
- }
- }
DPRINTF("iterate\n");
pending_size = qemu_savevm_state_pending(s->file, max_size);
@@ -741,9 +729,21 @@ static bool migrate_fd_put_ready(MigrationState *s, uint64_t max_size)
static void *buffered_file_thread(void *opaque)
{
QEMUFileBuffered *s = opaque;
+ MigrationState *m = s->migration_state;
int64_t initial_time = qemu_get_clock_ms(rt_clock);
int64_t max_size = 0;
bool last_round = false;
+ int ret;
+
+ qemu_mutex_lock_iothread();
+ DPRINTF("beginning savevm\n");
+ ret = qemu_savevm_state_begin(m->file, &m->params);
+ if (ret < 0) {
+ DPRINTF("failed, %d\n", ret);
+ qemu_mutex_unlock_iothread();
+ goto out;
+ }
+ qemu_mutex_unlock_iothread();
while (true) {
int64_t current_time = qemu_get_clock_ms(rt_clock);
@@ -775,10 +775,14 @@ static void *buffered_file_thread(void *opaque)
DPRINTF("file is ready\n");
if (s->bytes_xfer < s->xfer_limit) {
DPRINTF("notifying client\n");
- last_round = migrate_fd_put_ready(s->migration_state, max_size);
+ last_round = migrate_fd_put_ready(m, max_size);
}
}
+out:
+ if (ret < 0) {
+ migrate_fd_error(m);
+ }
g_free(s->buffer);
g_free(s);
return NULL;
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 21/35] migration: Add buffered_flush error handling
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (19 preceding siblings ...)
2012-12-11 12:46 ` [Qemu-devel] [PATCH 20/35] migration: move begining stage to the migration thread Juan Quintela
@ 2012-12-11 12:46 ` Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 22/35] migration: move exit condition to migration thread Juan Quintela
` (14 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:46 UTC (permalink / raw)
To: qemu-devel
Now that we have error handling we can do proper handling of
buffered_flush().
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
migration.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/migration.c b/migration.c
index 336941b..031302b 100644
--- a/migration.c
+++ b/migration.c
@@ -768,7 +768,8 @@ static void *buffered_file_thread(void *opaque)
/* usleep expects microseconds */
usleep((initial_time + BUFFER_DELAY - current_time)*1000);
}
- if (buffered_flush(s) < 0) {
+ ret = buffered_flush(s);
+ if (ret < 0) {
break;
}
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 22/35] migration: move exit condition to migration thread
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (20 preceding siblings ...)
2012-12-11 12:46 ` [Qemu-devel] [PATCH 21/35] migration: Add buffered_flush error handling Juan Quintela
@ 2012-12-11 12:46 ` Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 23/35] migration: unfold rest of migrate_fd_put_ready() into thread Juan Quintela
` (13 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:46 UTC (permalink / raw)
To: qemu-devel
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
migration.c | 17 ++++++++++-------
1 file changed, 10 insertions(+), 7 deletions(-)
diff --git a/migration.c b/migration.c
index 031302b..5c2f413 100644
--- a/migration.c
+++ b/migration.c
@@ -676,12 +676,6 @@ static bool migrate_fd_put_ready(MigrationState *s, uint64_t max_size)
bool last_round = false;
qemu_mutex_lock_iothread();
- if (s->state != MIG_STATE_ACTIVE) {
- DPRINTF("put_ready returning because of non-active state\n");
- qemu_mutex_unlock_iothread();
- return false;
- }
-
DPRINTF("iterate\n");
pending_size = qemu_savevm_state_pending(s->file, max_size);
DPRINTF("pending size %lu max %lu\n", pending_size, max_size);
@@ -748,9 +742,18 @@ static void *buffered_file_thread(void *opaque)
while (true) {
int64_t current_time = qemu_get_clock_ms(rt_clock);
- if (s->migration_state->complete) {
+ qemu_mutex_lock_iothread();
+ if (m->state != MIG_STATE_ACTIVE) {
+ DPRINTF("put_ready returning because of non-active state\n");
+ qemu_mutex_unlock_iothread();
break;
}
+ if (m->complete) {
+ qemu_mutex_unlock_iothread();
+ break;
+ }
+ qemu_mutex_unlock_iothread();
+
if (current_time >= initial_time + BUFFER_DELAY) {
uint64_t transferred_bytes = s->bytes_xfer;
uint64_t time_spent = current_time - initial_time;
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 23/35] migration: unfold rest of migrate_fd_put_ready() into thread
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (21 preceding siblings ...)
2012-12-11 12:46 ` [Qemu-devel] [PATCH 22/35] migration: move exit condition to migration thread Juan Quintela
@ 2012-12-11 12:46 ` Juan Quintela
2012-12-11 12:47 ` [Qemu-devel] [PATCH 24/35] ram: rename last_block to last_seen_block Juan Quintela
` (12 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:46 UTC (permalink / raw)
To: qemu-devel
This will allow us finer control in next patches.
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
migration.c | 95 ++++++++++++++++++++++++++-----------------------------------
1 file changed, 41 insertions(+), 54 deletions(-)
diff --git a/migration.c b/migration.c
index 5c2f413..7a4a6f0 100644
--- a/migration.c
+++ b/migration.c
@@ -669,54 +669,6 @@ static int64_t buffered_get_rate_limit(void *opaque)
return s->xfer_limit;
}
-static bool migrate_fd_put_ready(MigrationState *s, uint64_t max_size)
-{
- int ret;
- uint64_t pending_size;
- bool last_round = false;
-
- qemu_mutex_lock_iothread();
- DPRINTF("iterate\n");
- pending_size = qemu_savevm_state_pending(s->file, max_size);
- DPRINTF("pending size %lu max %lu\n", pending_size, max_size);
- if (pending_size >= max_size) {
- ret = qemu_savevm_state_iterate(s->file);
- if (ret < 0) {
- migrate_fd_error(s);
- }
- } else {
- int old_vm_running = runstate_is_running();
- int64_t start_time, end_time;
-
- DPRINTF("done iterating\n");
- start_time = qemu_get_clock_ms(rt_clock);
- qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
- if (old_vm_running) {
- vm_stop(RUN_STATE_FINISH_MIGRATE);
- } else {
- vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
- }
-
- if (qemu_savevm_state_complete(s->file) < 0) {
- migrate_fd_error(s);
- } else {
- migrate_fd_completed(s);
- }
- end_time = qemu_get_clock_ms(rt_clock);
- s->total_time = end_time - s->total_time;
- s->downtime = end_time - start_time;
- if (s->state != MIG_STATE_COMPLETED) {
- if (old_vm_running) {
- vm_start();
- }
- }
- last_round = true;
- }
- qemu_mutex_unlock_iothread();
-
- return last_round;
-}
-
/* 100ms xfer_limit is the limit that we should write each 100ms */
#define BUFFER_DELAY 100
@@ -741,6 +693,7 @@ static void *buffered_file_thread(void *opaque)
while (true) {
int64_t current_time = qemu_get_clock_ms(rt_clock);
+ uint64_t pending_size;
qemu_mutex_lock_iothread();
if (m->state != MIG_STATE_ACTIVE) {
@@ -752,6 +705,46 @@ static void *buffered_file_thread(void *opaque)
qemu_mutex_unlock_iothread();
break;
}
+ if (s->bytes_xfer < s->xfer_limit) {
+ DPRINTF("iterate\n");
+ pending_size = qemu_savevm_state_pending(m->file, max_size);
+ DPRINTF("pending size %lu max %lu\n", pending_size, max_size);
+ if (pending_size >= max_size) {
+ ret = qemu_savevm_state_iterate(m->file);
+ if (ret < 0) {
+ qemu_mutex_unlock_iothread();
+ break;
+ }
+ } else {
+ int old_vm_running = runstate_is_running();
+ int64_t start_time, end_time;
+
+ DPRINTF("done iterating\n");
+ start_time = qemu_get_clock_ms(rt_clock);
+ qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
+ if (old_vm_running) {
+ vm_stop(RUN_STATE_FINISH_MIGRATE);
+ } else {
+ vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
+ }
+ ret = qemu_savevm_state_complete(m->file);
+ if (ret < 0) {
+ qemu_mutex_unlock_iothread();
+ break;
+ } else {
+ migrate_fd_completed(m);
+ }
+ end_time = qemu_get_clock_ms(rt_clock);
+ m->total_time = end_time - m->total_time;
+ m->downtime = end_time - start_time;
+ if (m->state != MIG_STATE_COMPLETED) {
+ if (old_vm_running) {
+ vm_start();
+ }
+ }
+ last_round = true;
+ }
+ }
qemu_mutex_unlock_iothread();
if (current_time >= initial_time + BUFFER_DELAY) {
@@ -775,12 +768,6 @@ static void *buffered_file_thread(void *opaque)
if (ret < 0) {
break;
}
-
- DPRINTF("file is ready\n");
- if (s->bytes_xfer < s->xfer_limit) {
- DPRINTF("notifying client\n");
- last_round = migrate_fd_put_ready(m, max_size);
- }
}
out:
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 24/35] ram: rename last_block to last_seen_block
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (22 preceding siblings ...)
2012-12-11 12:46 ` [Qemu-devel] [PATCH 23/35] migration: unfold rest of migrate_fd_put_ready() into thread Juan Quintela
@ 2012-12-11 12:47 ` Juan Quintela
2012-12-11 12:47 ` [Qemu-devel] [PATCH 25/35] ram: Add last_sent_block Juan Quintela
` (11 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:47 UTC (permalink / raw)
To: qemu-devel
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
arch_init.c | 16 ++++++++++------
1 file changed, 10 insertions(+), 6 deletions(-)
diff --git a/arch_init.c b/arch_init.c
index 4379108..73bcbcc 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -332,7 +332,10 @@ static int save_xbzrle_page(QEMUFile *f, uint8_t *current_data,
return bytes_sent;
}
-static RAMBlock *last_block;
+
+/* This is the last block that we have visited serching for dirty pages
+ */
+static RAMBlock *last_seen_block;
static ram_addr_t last_offset;
static unsigned long *migration_bitmap;
static uint64_t migration_dirty_pages;
@@ -417,7 +420,7 @@ static void migration_bitmap_sync(void)
static int ram_save_block(QEMUFile *f, bool last_stage)
{
- RAMBlock *block = last_block;
+ RAMBlock *block = last_seen_block;
ram_addr_t offset = last_offset;
int bytes_sent = -1;
MemoryRegion *mr;
@@ -430,7 +433,8 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
mr = block->mr;
if (migration_bitmap_test_and_reset_dirty(mr, offset)) {
uint8_t *p;
- int cont = (block == last_block) ? RAM_SAVE_FLAG_CONTINUE : 0;
+ int cont = (block == last_seen_block) ?
+ RAM_SAVE_FLAG_CONTINUE : 0;
p = memory_region_get_ram_ptr(mr) + offset;
@@ -469,9 +473,9 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
if (!block)
block = QLIST_FIRST(&ram_list.blocks);
}
- } while (block != last_block || offset != last_offset);
+ } while (block != last_seen_block || offset != last_offset);
- last_block = block;
+ last_seen_block = block;
last_offset = offset;
return bytes_sent;
@@ -555,7 +559,7 @@ static void ram_migration_cancel(void *opaque)
static void reset_ram_globals(void)
{
- last_block = NULL;
+ last_seen_block = NULL;
last_offset = 0;
last_version = ram_list.version;
sort_ram_list();
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 25/35] ram: Add last_sent_block
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (23 preceding siblings ...)
2012-12-11 12:47 ` [Qemu-devel] [PATCH 24/35] ram: rename last_block to last_seen_block Juan Quintela
@ 2012-12-11 12:47 ` Juan Quintela
2012-12-11 12:47 ` [Qemu-devel] [PATCH 26/35] memory: introduce memory_region_test_and_clear_dirty Juan Quintela
` (10 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:47 UTC (permalink / raw)
To: qemu-devel; +Cc: Orit Wasserman
This is the last block from where we have sent data.
Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
arch_init.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/arch_init.c b/arch_init.c
index 73bcbcc..2d566d6 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -336,6 +336,8 @@ static int save_xbzrle_page(QEMUFile *f, uint8_t *current_data,
/* This is the last block that we have visited serching for dirty pages
*/
static RAMBlock *last_seen_block;
+/* This is the last block from where we have sent data */
+static RAMBlock *last_sent_block;
static ram_addr_t last_offset;
static unsigned long *migration_bitmap;
static uint64_t migration_dirty_pages;
@@ -433,7 +435,7 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
mr = block->mr;
if (migration_bitmap_test_and_reset_dirty(mr, offset)) {
uint8_t *p;
- int cont = (block == last_seen_block) ?
+ int cont = (block == last_sent_block) ?
RAM_SAVE_FLAG_CONTINUE : 0;
p = memory_region_get_ram_ptr(mr) + offset;
@@ -462,6 +464,7 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
/* if page is unmodified, continue to the next */
if (bytes_sent != 0) {
+ last_sent_block = block;
break;
}
}
@@ -560,6 +563,7 @@ static void ram_migration_cancel(void *opaque)
static void reset_ram_globals(void)
{
last_seen_block = NULL;
+ last_sent_block = NULL;
last_offset = 0;
last_version = ram_list.version;
sort_ram_list();
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 26/35] memory: introduce memory_region_test_and_clear_dirty
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (24 preceding siblings ...)
2012-12-11 12:47 ` [Qemu-devel] [PATCH 25/35] ram: Add last_sent_block Juan Quintela
@ 2012-12-11 12:47 ` Juan Quintela
2012-12-11 12:47 ` [Qemu-devel] [PATCH 27/35] ram: Use memory_region_test_and_clear_dirty Juan Quintela
` (9 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:47 UTC (permalink / raw)
To: qemu-devel
This function avoids having to do two calls, one to test the dirty bit, and
other to reset it.
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
memory.c | 16 ++++++++++++++++
memory.h | 16 ++++++++++++++++
2 files changed, 32 insertions(+)
diff --git a/memory.c b/memory.c
index 7419853..c72a5e2 100644
--- a/memory.c
+++ b/memory.c
@@ -1081,6 +1081,22 @@ void memory_region_set_dirty(MemoryRegion *mr, hwaddr addr,
return cpu_physical_memory_set_dirty_range(mr->ram_addr + addr, size, -1);
}
+bool memory_region_test_and_clear_dirty(MemoryRegion *mr, hwaddr addr,
+ hwaddr size, unsigned client)
+{
+ bool ret;
+ assert(mr->terminates);
+ ret = cpu_physical_memory_get_dirty(mr->ram_addr + addr, size,
+ 1 << client);
+ if (ret) {
+ cpu_physical_memory_reset_dirty(mr->ram_addr + addr,
+ mr->ram_addr + addr + size,
+ 1 << client);
+ }
+ return ret;
+}
+
+
void memory_region_sync_dirty_bitmap(MemoryRegion *mr)
{
AddressSpace *as;
diff --git a/memory.h b/memory.h
index 9462bfd..bc63a87 100644
--- a/memory.h
+++ b/memory.h
@@ -454,6 +454,22 @@ void memory_region_set_dirty(MemoryRegion *mr, hwaddr addr,
hwaddr size);
/**
+ * memory_region_test_and_clear_dirty: Check whether a range of bytes is dirty
+ * for a specified client. It clears them.
+ *
+ * Checks whether a range of bytes has been written to since the last
+ * call to memory_region_reset_dirty() with the same @client. Dirty logging
+ * must be enabled.
+ *
+ * @mr: the memory region being queried.
+ * @addr: the address (relative to the start of the region) being queried.
+ * @size: the size of the range being queried.
+ * @client: the user of the logging information; %DIRTY_MEMORY_MIGRATION or
+ * %DIRTY_MEMORY_VGA.
+ */
+bool memory_region_test_and_clear_dirty(MemoryRegion *mr, hwaddr addr,
+ hwaddr size, unsigned client);
+/**
* memory_region_sync_dirty_bitmap: Synchronize a region's dirty bitmap with
* any external TLBs (e.g. kvm)
*
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 27/35] ram: Use memory_region_test_and_clear_dirty
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (25 preceding siblings ...)
2012-12-11 12:47 ` [Qemu-devel] [PATCH 26/35] memory: introduce memory_region_test_and_clear_dirty Juan Quintela
@ 2012-12-11 12:47 ` Juan Quintela
2012-12-11 12:47 ` [Qemu-devel] [PATCH 28/35] migration: Only go to the iterate stage if there is anything to send Juan Quintela
` (8 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:47 UTC (permalink / raw)
To: qemu-devel
This avoids having to do two walks over the dirty bitmap, once reading
the dirty bits, and anthoer cleaning them.
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
arch_init.c | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/arch_init.c b/arch_init.c
index 2d566d6..b204602 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -390,13 +390,12 @@ static void migration_bitmap_sync(void)
QLIST_FOREACH(block, &ram_list.blocks, next) {
for (addr = 0; addr < block->length; addr += TARGET_PAGE_SIZE) {
- if (memory_region_get_dirty(block->mr, addr, TARGET_PAGE_SIZE,
- DIRTY_MEMORY_MIGRATION)) {
+ if (memory_region_test_and_clear_dirty(block->mr,
+ addr, TARGET_PAGE_SIZE,
+ DIRTY_MEMORY_MIGRATION)) {
migration_bitmap_set_dirty(block->mr, addr);
}
}
- memory_region_reset_dirty(block->mr, 0, block->length,
- DIRTY_MEMORY_MIGRATION);
}
trace_migration_bitmap_sync_end(migration_dirty_pages
- num_dirty_pages_init);
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 28/35] migration: Only go to the iterate stage if there is anything to send
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (26 preceding siblings ...)
2012-12-11 12:47 ` [Qemu-devel] [PATCH 27/35] ram: Use memory_region_test_and_clear_dirty Juan Quintela
@ 2012-12-11 12:47 ` Juan Quintela
2012-12-11 12:47 ` [Qemu-devel] [PATCH 29/35] ram: optimize migration bitmap walking Juan Quintela
` (7 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:47 UTC (permalink / raw)
To: qemu-devel; +Cc: Orit Wasserman
Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
migration.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/migration.c b/migration.c
index 7a4a6f0..216356d 100644
--- a/migration.c
+++ b/migration.c
@@ -709,7 +709,7 @@ static void *buffered_file_thread(void *opaque)
DPRINTF("iterate\n");
pending_size = qemu_savevm_state_pending(m->file, max_size);
DPRINTF("pending size %lu max %lu\n", pending_size, max_size);
- if (pending_size >= max_size) {
+ if (pending_size && pending_size >= max_size) {
ret = qemu_savevm_state_iterate(m->file);
if (ret < 0) {
qemu_mutex_unlock_iothread();
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 29/35] ram: optimize migration bitmap walking
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (27 preceding siblings ...)
2012-12-11 12:47 ` [Qemu-devel] [PATCH 28/35] migration: Only go to the iterate stage if there is anything to send Juan Quintela
@ 2012-12-11 12:47 ` Juan Quintela
2012-12-11 12:47 ` [Qemu-devel] [PATCH 30/35] ram: account the amount of transferred ram better Juan Quintela
` (6 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:47 UTC (permalink / raw)
To: qemu-devel
Instead of testing each page individually, we search what is the next
dirty page with a bitmap operation. We have to reorganize the code to
move from a "for" loop, to a while(dirty) loop.
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
arch_init.c | 45 ++++++++++++++++++++++++++-------------------
1 file changed, 26 insertions(+), 19 deletions(-)
diff --git a/arch_init.c b/arch_init.c
index b204602..6c12a7d 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -343,18 +343,21 @@ static unsigned long *migration_bitmap;
static uint64_t migration_dirty_pages;
static uint32_t last_version;
-static inline bool migration_bitmap_test_and_reset_dirty(MemoryRegion *mr,
- ram_addr_t offset)
+static inline
+ram_addr_t migration_bitmap_find_and_reset_dirty(MemoryRegion *mr,
+ ram_addr_t start)
{
- bool ret;
- int nr = (mr->ram_addr + offset) >> TARGET_PAGE_BITS;
+ unsigned long base = mr->ram_addr >> TARGET_PAGE_BITS;
+ unsigned long nr = base + (start >> TARGET_PAGE_BITS);
+ unsigned long size = base + (int128_get64(mr->size) >> TARGET_PAGE_BITS);
- ret = test_and_clear_bit(nr, migration_bitmap);
+ unsigned long next = find_next_bit(migration_bitmap, size, nr);
- if (ret) {
+ if (next < size) {
+ clear_bit(next, migration_bitmap);
migration_dirty_pages--;
}
- return ret;
+ return (next - base) << TARGET_PAGE_BITS;
}
static inline bool migration_bitmap_set_dirty(MemoryRegion *mr,
@@ -423,6 +426,7 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
{
RAMBlock *block = last_seen_block;
ram_addr_t offset = last_offset;
+ bool complete_round = false;
int bytes_sent = -1;
MemoryRegion *mr;
ram_addr_t current_addr;
@@ -430,9 +434,21 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
if (!block)
block = QLIST_FIRST(&ram_list.blocks);
- do {
+ while(true) {
mr = block->mr;
- if (migration_bitmap_test_and_reset_dirty(mr, offset)) {
+ offset = migration_bitmap_find_and_reset_dirty(mr, offset);
+ if (complete_round && block == last_seen_block &&
+ offset >= last_offset) {
+ break;
+ }
+ if (offset >= block->length) {
+ offset = 0;
+ block = QLIST_NEXT(block, next);
+ if (!block) {
+ block = QLIST_FIRST(&ram_list.blocks);
+ complete_round = true;
+ }
+ } else {
uint8_t *p;
int cont = (block == last_sent_block) ?
RAM_SAVE_FLAG_CONTINUE : 0;
@@ -467,16 +483,7 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
break;
}
}
-
- offset += TARGET_PAGE_SIZE;
- if (offset >= block->length) {
- offset = 0;
- block = QLIST_NEXT(block, next);
- if (!block)
- block = QLIST_FIRST(&ram_list.blocks);
- }
- } while (block != last_seen_block || offset != last_offset);
-
+ }
last_seen_block = block;
last_offset = offset;
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 30/35] ram: account the amount of transferred ram better
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (28 preceding siblings ...)
2012-12-11 12:47 ` [Qemu-devel] [PATCH 29/35] ram: optimize migration bitmap walking Juan Quintela
@ 2012-12-11 12:47 ` Juan Quintela
2012-12-11 12:47 ` [Qemu-devel] [PATCH 31/35] ram: refactor ram_save_block() return value Juan Quintela
` (5 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:47 UTC (permalink / raw)
To: qemu-devel
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
arch_init.c | 34 ++++++++++++++++++++--------------
1 file changed, 20 insertions(+), 14 deletions(-)
diff --git a/arch_init.c b/arch_init.c
index 6c12a7d..3e82588 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -265,16 +265,21 @@ uint64_t xbzrle_mig_pages_overflow(void)
return acct_info.xbzrle_overflows;
}
-static void save_block_hdr(QEMUFile *f, RAMBlock *block, ram_addr_t offset,
- int cont, int flag)
+static size_t save_block_hdr(QEMUFile *f, RAMBlock *block, ram_addr_t offset,
+ int cont, int flag)
{
- qemu_put_be64(f, offset | cont | flag);
- if (!cont) {
- qemu_put_byte(f, strlen(block->idstr));
- qemu_put_buffer(f, (uint8_t *)block->idstr,
- strlen(block->idstr));
- }
+ size_t size;
+
+ qemu_put_be64(f, offset | cont | flag);
+ size = 8;
+ if (!cont) {
+ qemu_put_byte(f, strlen(block->idstr));
+ qemu_put_buffer(f, (uint8_t *)block->idstr,
+ strlen(block->idstr));
+ size += 1 + strlen(block->idstr);
+ }
+ return size;
}
#define ENCODING_FLAG_XBZRLE 0x1
@@ -321,11 +326,11 @@ static int save_xbzrle_page(QEMUFile *f, uint8_t *current_data,
}
/* Send XBZRLE based compressed page */
- save_block_hdr(f, block, offset, cont, RAM_SAVE_FLAG_XBZRLE);
+ bytes_sent = save_block_hdr(f, block, offset, cont, RAM_SAVE_FLAG_XBZRLE);
qemu_put_byte(f, ENCODING_FLAG_XBZRLE);
qemu_put_be16(f, encoded_len);
qemu_put_buffer(f, XBZRLE.encoded_buf, encoded_len);
- bytes_sent = encoded_len + 1 + 2;
+ bytes_sent += encoded_len + 1 + 2;
acct_info.xbzrle_pages++;
acct_info.xbzrle_bytes += bytes_sent;
@@ -457,9 +462,10 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
if (is_dup_page(p)) {
acct_info.dup_pages++;
- save_block_hdr(f, block, offset, cont, RAM_SAVE_FLAG_COMPRESS);
+ bytes_sent = save_block_hdr(f, block, offset, cont,
+ RAM_SAVE_FLAG_COMPRESS);
qemu_put_byte(f, *p);
- bytes_sent = 1;
+ bytes_sent += 1;
} else if (migrate_use_xbzrle()) {
current_addr = block->offset + offset;
bytes_sent = save_xbzrle_page(f, p, current_addr, block,
@@ -471,9 +477,9 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
/* either we didn't send yet (we may have had XBZRLE overflow) */
if (bytes_sent == -1) {
- save_block_hdr(f, block, offset, cont, RAM_SAVE_FLAG_PAGE);
+ bytes_sent = save_block_hdr(f, block, offset, cont, RAM_SAVE_FLAG_PAGE);
qemu_put_buffer(f, p, TARGET_PAGE_SIZE);
- bytes_sent = TARGET_PAGE_SIZE;
+ bytes_sent += TARGET_PAGE_SIZE;
acct_info.norm_pages++;
}
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 31/35] ram: refactor ram_save_block() return value
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (29 preceding siblings ...)
2012-12-11 12:47 ` [Qemu-devel] [PATCH 30/35] ram: account the amount of transferred ram better Juan Quintela
@ 2012-12-11 12:47 ` Juan Quintela
2012-12-11 12:47 ` [Qemu-devel] [PATCH 32/35] ram: add free_space parameter to save_live functions Juan Quintela
` (4 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:47 UTC (permalink / raw)
To: qemu-devel
It could only return 0 if we only found dirty xbzrle pages that hadn't
changed (i.e. they were written with the same content). We don't care
about that case, it is the same than nothing dirty.
So now the return of the function is how much have it written, nothing
else. Adjust callers.
And we also made ram_save_iterate() return the number of transferred
bytes, not the number of transferred pages.
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
arch_init.c | 25 +++++++++++++++----------
1 file changed, 15 insertions(+), 10 deletions(-)
diff --git a/arch_init.c b/arch_init.c
index 3e82588..723529a 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -422,9 +422,8 @@ static void migration_bitmap_sync(void)
/*
* ram_save_block: Writes a page of memory to the stream f
*
- * Returns: 0: if the page hasn't changed
- * -1: if there are no more dirty pages
- * n: the amount of bytes written in other case
+ * Returns: The number of bytes written.
+ * 0 means no dirty pages
*/
static int ram_save_block(QEMUFile *f, bool last_stage)
@@ -432,7 +431,7 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
RAMBlock *block = last_seen_block;
ram_addr_t offset = last_offset;
bool complete_round = false;
- int bytes_sent = -1;
+ int bytes_sent = 0;
MemoryRegion *mr;
ram_addr_t current_addr;
@@ -460,6 +459,8 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
p = memory_region_get_ram_ptr(mr) + offset;
+ /* In doubt sent page as normal */
+ bytes_sent = -1;
if (is_dup_page(p)) {
acct_info.dup_pages++;
bytes_sent = save_block_hdr(f, block, offset, cont,
@@ -475,7 +476,7 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
}
}
- /* either we didn't send yet (we may have had XBZRLE overflow) */
+ /* XBZRLE overflow or normal page */
if (bytes_sent == -1) {
bytes_sent = save_block_hdr(f, block, offset, cont, RAM_SAVE_FLAG_PAGE);
qemu_put_buffer(f, p, TARGET_PAGE_SIZE);
@@ -484,7 +485,7 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
}
/* if page is unmodified, continue to the next */
- if (bytes_sent != 0) {
+ if (bytes_sent > 0) {
last_sent_block = block;
break;
}
@@ -631,6 +632,7 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
int ret;
int i;
int64_t t0;
+ int total_sent = 0;
qemu_mutex_lock_ramlist();
@@ -645,10 +647,10 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
bytes_sent = ram_save_block(f, false);
/* no more blocks to sent */
- if (bytes_sent < 0) {
+ if (bytes_sent == 0) {
break;
}
- bytes_transferred += bytes_sent;
+ total_sent += bytes_sent;
acct_info.iterations++;
/* we want to check in the 1st loop, just in case it was the 1st time
and we had to sync the dirty bitmap.
@@ -667,13 +669,16 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
}
if (ret < 0) {
+ bytes_transferred += total_sent;
return ret;
}
qemu_mutex_unlock_ramlist();
qemu_put_be64(f, RAM_SAVE_FLAG_EOS);
+ total_sent += 8;
+ bytes_transferred += total_sent;
- return i;
+ return total_sent;
}
static int ram_save_complete(QEMUFile *f, void *opaque)
@@ -690,7 +695,7 @@ static int ram_save_complete(QEMUFile *f, void *opaque)
bytes_sent = ram_save_block(f, true);
/* no more blocks to sent */
- if (bytes_sent < 0) {
+ if (bytes_sent == 0) {
break;
}
bytes_transferred += bytes_sent;
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 32/35] ram: add free_space parameter to save_live functions
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (30 preceding siblings ...)
2012-12-11 12:47 ` [Qemu-devel] [PATCH 31/35] ram: refactor ram_save_block() return value Juan Quintela
@ 2012-12-11 12:47 ` Juan Quintela
2012-12-11 12:47 ` [Qemu-devel] [PATCH 33/35] ram: remove xbrle last_stage optimization Juan Quintela
` (3 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:47 UTC (permalink / raw)
To: qemu-devel
As we really know how much space we have free in the buffers, we can
send that information instead of guessing how much we can sent each time.
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
arch_init.c | 20 +++++++++-----------
block-migration.c | 2 +-
migration.c | 3 ++-
savevm.c | 10 +++++++---
sysemu.h | 2 +-
vmstate.h | 2 +-
6 files changed, 21 insertions(+), 18 deletions(-)
diff --git a/arch_init.c b/arch_init.c
index 723529a..f5f9324 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -627,9 +627,12 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
return 0;
}
-static int ram_save_iterate(QEMUFile *f, void *opaque)
+/* Maximum size for a transmited page
+ header + len + idstr + page size */
+#define MAX_PAGE_SIZE ( 8 + 1 + 256 + TARGET_PAGE_SIZE)
+
+static int ram_save_iterate(QEMUFile *f, void *opaque, uint64_t free_space)
{
- int ret;
int i;
int64_t t0;
int total_sent = 0;
@@ -642,15 +645,15 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
t0 = qemu_get_clock_ns(rt_clock);
i = 0;
- while ((ret = qemu_file_rate_limit(f)) == 0) {
- int bytes_sent;
-
- bytes_sent = ram_save_block(f, false);
+ /* We need space for at least one page and end of section marker */
+ while (free_space > MAX_PAGE_SIZE + 8) {
+ int bytes_sent = ram_save_block(f, false);
/* no more blocks to sent */
if (bytes_sent == 0) {
break;
}
total_sent += bytes_sent;
+ free_space -= bytes_sent;
acct_info.iterations++;
/* we want to check in the 1st loop, just in case it was the 1st time
and we had to sync the dirty bitmap.
@@ -668,11 +671,6 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
i++;
}
- if (ret < 0) {
- bytes_transferred += total_sent;
- return ret;
- }
-
qemu_mutex_unlock_ramlist();
qemu_put_be64(f, RAM_SAVE_FLAG_EOS);
total_sent += 8;
diff --git a/block-migration.c b/block-migration.c
index 5db01fe..b11d7e0 100644
--- a/block-migration.c
+++ b/block-migration.c
@@ -535,7 +535,7 @@ static int block_save_setup(QEMUFile *f, void *opaque)
return 0;
}
-static int block_save_iterate(QEMUFile *f, void *opaque)
+static int block_save_iterate(QEMUFile *f, void *opaque, uint64_t free_space)
{
int ret;
diff --git a/migration.c b/migration.c
index 216356d..d59f7b4 100644
--- a/migration.c
+++ b/migration.c
@@ -694,6 +694,7 @@ static void *buffered_file_thread(void *opaque)
while (true) {
int64_t current_time = qemu_get_clock_ms(rt_clock);
uint64_t pending_size;
+ size_t free_space = s->buffer_capacity - s->buffer_size;
qemu_mutex_lock_iothread();
if (m->state != MIG_STATE_ACTIVE) {
@@ -710,7 +711,7 @@ static void *buffered_file_thread(void *opaque)
pending_size = qemu_savevm_state_pending(m->file, max_size);
DPRINTF("pending size %lu max %lu\n", pending_size, max_size);
if (pending_size && pending_size >= max_size) {
- ret = qemu_savevm_state_iterate(m->file);
+ ret = qemu_savevm_state_iterate(m->file, free_space);
if (ret < 0) {
qemu_mutex_unlock_iothread();
break;
diff --git a/savevm.c b/savevm.c
index f6e91cf..4b7715a 100644
--- a/savevm.c
+++ b/savevm.c
@@ -1655,10 +1655,11 @@ int qemu_savevm_state_begin(QEMUFile *f,
* 0 : We haven't finished, caller have to go again
* 1 : We have finished, we can go to complete phase
*/
-int qemu_savevm_state_iterate(QEMUFile *f)
+int qemu_savevm_state_iterate(QEMUFile *f, uint64_t free_space)
{
SaveStateEntry *se;
int ret = 1;
+ size_t remaining_space = free_space;
QTAILQ_FOREACH(se, &savevm_handlers, entry) {
if (!se->ops || !se->ops->save_live_iterate) {
@@ -1675,9 +1676,11 @@ int qemu_savevm_state_iterate(QEMUFile *f)
trace_savevm_section_start();
/* Section type */
qemu_put_byte(f, QEMU_VM_SECTION_PART);
+ remaining_space -= 1;
qemu_put_be32(f, se->section_id);
+ remaining_space -= 4;
- ret = se->ops->save_live_iterate(f, se->opaque);
+ ret = se->ops->save_live_iterate(f, se->opaque, remaining_space);
trace_savevm_section_end(se->section_id);
if (ret <= 0) {
@@ -1687,6 +1690,7 @@ int qemu_savevm_state_iterate(QEMUFile *f)
synchronized over and over again. */
break;
}
+ remaining_space -= ret;
}
if (ret != 0) {
return ret;
@@ -1802,7 +1806,7 @@ static int qemu_savevm_state(QEMUFile *f)
goto out;
do {
- ret = qemu_savevm_state_iterate(f);
+ ret = qemu_savevm_state_iterate(f, SIZE_MAX);
if (ret < 0)
goto out;
} while (ret == 0);
diff --git a/sysemu.h b/sysemu.h
index 7832c69..7fc5124 100644
--- a/sysemu.h
+++ b/sysemu.h
@@ -75,7 +75,7 @@ void qemu_announce_self(void);
bool qemu_savevm_state_blocked(Error **errp);
int qemu_savevm_state_begin(QEMUFile *f,
const MigrationParams *params);
-int qemu_savevm_state_iterate(QEMUFile *f);
+int qemu_savevm_state_iterate(QEMUFile *f, uint64_t free_space);
int qemu_savevm_state_complete(QEMUFile *f);
void qemu_savevm_state_cancel(QEMUFile *f);
uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size);
diff --git a/vmstate.h b/vmstate.h
index f27276c..0b55cf4 100644
--- a/vmstate.h
+++ b/vmstate.h
@@ -33,7 +33,7 @@ typedef struct SaveVMHandlers {
void (*set_params)(const MigrationParams *params, void * opaque);
SaveStateHandler *save_state;
int (*save_live_setup)(QEMUFile *f, void *opaque);
- int (*save_live_iterate)(QEMUFile *f, void *opaque);
+ int (*save_live_iterate)(QEMUFile *f, void *opaque, uint64_t free_space);
int (*save_live_complete)(QEMUFile *f, void *opaque);
uint64_t (*save_live_pending)(QEMUFile *f, void *opaque, uint64_t max_size);
void (*cancel)(void *opaque);
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 33/35] ram: remove xbrle last_stage optimization
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (31 preceding siblings ...)
2012-12-11 12:47 ` [Qemu-devel] [PATCH 32/35] ram: add free_space parameter to save_live functions Juan Quintela
@ 2012-12-11 12:47 ` Juan Quintela
2012-12-11 12:47 ` [Qemu-devel] [PATCH 34/35] ram: reuse ram_save_iterate() for the complete stage Juan Quintela
` (2 subsequent siblings)
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:47 UTC (permalink / raw)
To: qemu-devel
We need to remove it to be able to return from complete to iterative
phases of migration.
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
arch_init.c | 24 +++++++++---------------
1 file changed, 9 insertions(+), 15 deletions(-)
diff --git a/arch_init.c b/arch_init.c
index f5f9324..6a4ee24 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -286,16 +286,14 @@ static size_t save_block_hdr(QEMUFile *f, RAMBlock *block, ram_addr_t offset,
static int save_xbzrle_page(QEMUFile *f, uint8_t *current_data,
ram_addr_t current_addr, RAMBlock *block,
- ram_addr_t offset, int cont, bool last_stage)
+ ram_addr_t offset, int cont)
{
int encoded_len = 0, bytes_sent = -1;
uint8_t *prev_cached_page;
if (!cache_is_cached(XBZRLE.cache, current_addr)) {
- if (!last_stage) {
- cache_insert(XBZRLE.cache, current_addr,
- g_memdup(current_data, TARGET_PAGE_SIZE));
- }
+ cache_insert(XBZRLE.cache, current_addr,
+ g_memdup(current_data, TARGET_PAGE_SIZE));
acct_info.xbzrle_cache_miss++;
return -1;
}
@@ -321,9 +319,7 @@ static int save_xbzrle_page(QEMUFile *f, uint8_t *current_data,
}
/* we need to update the data in the cache, in order to get the same data */
- if (!last_stage) {
- memcpy(prev_cached_page, XBZRLE.current_buf, TARGET_PAGE_SIZE);
- }
+ memcpy(prev_cached_page, XBZRLE.current_buf, TARGET_PAGE_SIZE);
/* Send XBZRLE based compressed page */
bytes_sent = save_block_hdr(f, block, offset, cont, RAM_SAVE_FLAG_XBZRLE);
@@ -426,7 +422,7 @@ static void migration_bitmap_sync(void)
* 0 means no dirty pages
*/
-static int ram_save_block(QEMUFile *f, bool last_stage)
+static int ram_save_block(QEMUFile *f)
{
RAMBlock *block = last_seen_block;
ram_addr_t offset = last_offset;
@@ -470,10 +466,8 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
} else if (migrate_use_xbzrle()) {
current_addr = block->offset + offset;
bytes_sent = save_xbzrle_page(f, p, current_addr, block,
- offset, cont, last_stage);
- if (!last_stage) {
- p = get_cached_data(XBZRLE.cache, current_addr);
- }
+ offset, cont);
+ p = get_cached_data(XBZRLE.cache, current_addr);
}
/* XBZRLE overflow or normal page */
@@ -647,7 +641,7 @@ static int ram_save_iterate(QEMUFile *f, void *opaque, uint64_t free_space)
i = 0;
/* We need space for at least one page and end of section marker */
while (free_space > MAX_PAGE_SIZE + 8) {
- int bytes_sent = ram_save_block(f, false);
+ int bytes_sent = ram_save_block(f);
/* no more blocks to sent */
if (bytes_sent == 0) {
break;
@@ -691,7 +685,7 @@ static int ram_save_complete(QEMUFile *f, void *opaque)
while (true) {
int bytes_sent;
- bytes_sent = ram_save_block(f, true);
+ bytes_sent = ram_save_block(f);
/* no more blocks to sent */
if (bytes_sent == 0) {
break;
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 34/35] ram: reuse ram_save_iterate() for the complete stage
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (32 preceding siblings ...)
2012-12-11 12:47 ` [Qemu-devel] [PATCH 33/35] ram: remove xbrle last_stage optimization Juan Quintela
@ 2012-12-11 12:47 ` Juan Quintela
2012-12-11 12:47 ` [Qemu-devel] [PATCH 35/35] migration: print times for end phase Juan Quintela
2012-12-11 14:22 ` [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Paolo Bonzini
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:47 UTC (permalink / raw)
To: qemu-devel
This means that we only have one memory loop for the iterate and
complete phase.
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
arch_init.c | 17 -----------------
migration.c | 12 ++++++++++++
2 files changed, 12 insertions(+), 17 deletions(-)
diff --git a/arch_init.c b/arch_init.c
index 6a4ee24..87aecf8 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -675,25 +675,8 @@ static int ram_save_iterate(QEMUFile *f, void *opaque, uint64_t free_space)
static int ram_save_complete(QEMUFile *f, void *opaque)
{
- migration_bitmap_sync();
-
qemu_mutex_lock_ramlist();
-
- /* try transferring iterative blocks of memory */
-
- /* flush all remaining blocks regardless of rate limiting */
- while (true) {
- int bytes_sent;
-
- bytes_sent = ram_save_block(f);
- /* no more blocks to sent */
- if (bytes_sent == 0) {
- break;
- }
- bytes_transferred += bytes_sent;
- }
memory_global_dirty_log_stop();
-
qemu_mutex_unlock_ramlist();
qemu_put_be64(f, RAM_SAVE_FLAG_EOS);
diff --git a/migration.c b/migration.c
index d59f7b4..dd6a401 100644
--- a/migration.c
+++ b/migration.c
@@ -728,6 +728,18 @@ static void *buffered_file_thread(void *opaque)
} else {
vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
}
+
+ /* 8 is the size of an end of section mark, so empty section */
+ while ((ret = qemu_savevm_state_iterate(m->file, free_space))
+ > 8) {
+ ret = buffered_flush(s);
+ if (ret < 0) {
+ qemu_mutex_unlock_iothread();
+ break;
+ }
+ free_space = s->buffer_capacity - s->buffer_size;
+ }
+
ret = qemu_savevm_state_complete(m->file);
if (ret < 0) {
qemu_mutex_unlock_iothread();
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [Qemu-devel] [PATCH 35/35] migration: print times for end phase
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (33 preceding siblings ...)
2012-12-11 12:47 ` [Qemu-devel] [PATCH 34/35] ram: reuse ram_save_iterate() for the complete stage Juan Quintela
@ 2012-12-11 12:47 ` Juan Quintela
2012-12-11 14:22 ` [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Paolo Bonzini
35 siblings, 0 replies; 37+ messages in thread
From: Juan Quintela @ 2012-12-11 12:47 UTC (permalink / raw)
To: qemu-devel
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
block.c | 6 ++++++
cpus.c | 17 +++++++++++++++++
migration.c | 12 ++++++++++++
savevm.c | 13 +++++++++++++
4 files changed, 48 insertions(+)
diff --git a/block.c b/block.c
index c05875f..2fa0827 100644
--- a/block.c
+++ b/block.c
@@ -2680,9 +2680,15 @@ int bdrv_get_flags(BlockDriverState *bs)
void bdrv_flush_all(void)
{
BlockDriverState *bs;
+ int64_t start_time, end_time;
+
+ start_time = qemu_get_clock_ms(rt_clock);
QTAILQ_FOREACH(bs, &bdrv_states, list) {
bdrv_flush(bs);
+ end_time = qemu_get_clock_ms(rt_clock);
+ printf("time flush device %s: %ld\n", bs->filename,
+ end_time - start_time);
}
}
diff --git a/cpus.c b/cpus.c
index d9c332f..a920a06 100644
--- a/cpus.c
+++ b/cpus.c
@@ -437,14 +437,31 @@ bool cpu_is_stopped(CPUState *cpu)
static void do_vm_stop(RunState state)
{
+ int64_t start_time, end_time;
+
if (runstate_is_running()) {
+ start_time = qemu_get_clock_ms(rt_clock);
cpu_disable_ticks();
+ end_time = qemu_get_clock_ms(rt_clock);
+ printf("time cpu_disable_ticks %ld\n", end_time - start_time);
pause_all_vcpus();
+ end_time = qemu_get_clock_ms(rt_clock);
+ printf("time pause_all_vcpus %ld\n", end_time - start_time);
runstate_set(state);
+ end_time = qemu_get_clock_ms(rt_clock);
+ printf("time runstate_set %ld\n", end_time - start_time);
vm_state_notify(0, state);
+ end_time = qemu_get_clock_ms(rt_clock);
+ printf("time vmstate_notify %ld\n", end_time - start_time);
bdrv_drain_all();
+ end_time = qemu_get_clock_ms(rt_clock);
+ printf("time bdrv_drain_all %ld\n", end_time - start_time);
bdrv_flush_all();
+ end_time = qemu_get_clock_ms(rt_clock);
+ printf("time bdrv_flush_all %ld\n", end_time - start_time);
monitor_protocol_event(QEVENT_STOP, NULL);
+ end_time = qemu_get_clock_ms(rt_clock);
+ printf("time monitor_protocol_event %ld\n", end_time - start_time);
}
}
diff --git a/migration.c b/migration.c
index dd6a401..fc0548c 100644
--- a/migration.c
+++ b/migration.c
@@ -723,12 +723,17 @@ static void *buffered_file_thread(void *opaque)
DPRINTF("done iterating\n");
start_time = qemu_get_clock_ms(rt_clock);
qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
+ end_time = qemu_get_clock_ms(rt_clock);
+ printf("wakeup_request %ld\n", end_time - start_time);
if (old_vm_running) {
vm_stop(RUN_STATE_FINISH_MIGRATE);
} else {
vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
}
+ end_time = qemu_get_clock_ms(rt_clock);
+ printf("vm_stop %ld\n", end_time - start_time);
+
/* 8 is the size of an end of section mark, so empty section */
while ((ret = qemu_savevm_state_iterate(m->file, free_space))
> 8) {
@@ -739,15 +744,20 @@ static void *buffered_file_thread(void *opaque)
}
free_space = s->buffer_capacity - s->buffer_size;
}
+ end_time = qemu_get_clock_ms(rt_clock);
+ printf("iterate phase %ld\n", end_time - start_time);
ret = qemu_savevm_state_complete(m->file);
if (ret < 0) {
qemu_mutex_unlock_iothread();
break;
} else {
+ end_time = qemu_get_clock_ms(rt_clock);
+ printf("complete without error 3a %ld\n", end_time - start_time);
migrate_fd_completed(m);
}
end_time = qemu_get_clock_ms(rt_clock);
+ printf("completed %ld\n", end_time - start_time);
m->total_time = end_time - m->total_time;
m->downtime = end_time - start_time;
if (m->state != MIG_STATE_COMPLETED) {
@@ -755,6 +765,8 @@ static void *buffered_file_thread(void *opaque)
vm_start();
}
}
+ end_time = qemu_get_clock_ms(rt_clock);
+ printf("end completed stage %ld\n", end_time - start_time);
last_round = true;
}
}
diff --git a/savevm.c b/savevm.c
index 4b7715a..c964b66 100644
--- a/savevm.c
+++ b/savevm.c
@@ -1706,9 +1706,14 @@ int qemu_savevm_state_complete(QEMUFile *f)
{
SaveStateEntry *se;
int ret;
+ int64_t t1;
+ int64_t t0 = qemu_get_clock_ms(rt_clock);
cpu_synchronize_all_states();
+ t1 = qemu_get_clock_ms(rt_clock);
+ printf("synchronize_all_states %ld\n", t1 - t0);
+ t0 = t1;
QTAILQ_FOREACH(se, &savevm_handlers, entry) {
if (!se->ops || !se->ops->save_live_complete) {
continue;
@@ -1729,6 +1734,11 @@ int qemu_savevm_state_complete(QEMUFile *f)
return ret;
}
}
+ t1 = qemu_get_clock_ms(rt_clock);
+
+ printf("migrate save live complete %ld\n", t1 - t0);
+
+ t0 = t1;
QTAILQ_FOREACH(se, &savevm_handlers, entry) {
int len;
@@ -1753,6 +1763,9 @@ int qemu_savevm_state_complete(QEMUFile *f)
trace_savevm_section_end(se->section_id);
}
+ t1 = qemu_get_clock_ms(rt_clock);
+
+ printf("migrate rest devices %ld\n", t1 - t0);
qemu_put_byte(f, QEMU_VM_EOF);
return qemu_file_get_error(f);
--
1.7.11.7
^ permalink raw reply related [flat|nested] 37+ messages in thread
* Re: [Qemu-devel] [PATCH 00/35] Migration thread (20121211)
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
` (34 preceding siblings ...)
2012-12-11 12:47 ` [Qemu-devel] [PATCH 35/35] migration: print times for end phase Juan Quintela
@ 2012-12-11 14:22 ` Paolo Bonzini
35 siblings, 0 replies; 37+ messages in thread
From: Paolo Bonzini @ 2012-12-11 14:22 UTC (permalink / raw)
To: Juan Quintela; +Cc: qemu-devel
Il 11/12/2012 13:46, Juan Quintela ha scritto:
> Please test & comment & review, Juan.
Please, not this way.
This hardly takes into account all of the comments I sent to this series.
For example on 21 September I wrote about patch 4 that "usleep is broken
under IIRC Solaris (it uses signals and conflicts with qemu-timer.c).
Please use g_usleep instead." It still doesn't do.
At the beginning of November I went through another review of this code.
I found another small problem in patch 4 (it calls migrate_fd_close
twice). I commented that patch 6 has a stale commit message, and
incorrectly uses a global. It still does all of these things.
At the time, I also found that ratelimiting is currently broken
(including in the 1.3 release), and you haven't reviewed my patches to
fix that, nor included them in this series.
You are also ignoring other people's comments too. Peter mentioned on
21 October "Maybe worth adding a comment explaining why we have the two
lists, to save people having to ferret around in the revision history
later?". Nope.
But this is all fine. The worst part is that you *knew* I had worked on
the migration thread code, fixing all the above and more, and redoing
the second half of your series in a way that actually removed the memory
copies and serialized memory contents outside the BQL (no doubt there
are other cleanups that may be in these series and not in mine; I grant
that). Vinod had tested the branch successfully and Orit is already
using it as a base for further improvements.
I was going to ask you today about how to move on with the migration
thread work. Instead I found out that we duplicated effort on fixing
the conflicts, and the review comments have fallen on the floor once more.
So, I'm not going to review this series. I pushed my version of the
patches to github on the migration-thread-20121211 branch. Please
consider at least reviewing it.
Paolo
^ permalink raw reply [flat|nested] 37+ messages in thread
end of thread, other threads:[~2012-12-11 14:22 UTC | newest]
Thread overview: 37+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-12-11 12:46 [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 01/35] split MRU ram list Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 02/35] add a version number to ram_list Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 03/35] protect the ramlist with a separate mutex Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 04/35] buffered_file: Move from using a timer to use a thread Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 05/35] migration: make qemu_fopen_ops_buffered() return void Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 06/35] migration: stop all cpus correctly Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 07/35] migration: make writes blocking Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 08/35] migration: remove unfreeze logic Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 09/35] migration: take finer locking Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 10/35] buffered_file: Unfold the trick to restart generating migration data Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 11/35] buffered_file: don't flush on put buffer Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 12/35] buffered_file: unfold buffered_append in buffered_put_buffer Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 13/35] savevm: New save live migration method: pending Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 14/35] migration: include qemu-file.h Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 15/35] migration-fd: remove duplicate include Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 16/35] migration: move buffered_file.c code into migration.c Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 17/35] migration: move migration_fd_put_ready() Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 18/35] migration: Inline qemu_fopen_ops_buffered into migrate_fd_connect Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 19/35] migration: move migration notifier Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 20/35] migration: move begining stage to the migration thread Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 21/35] migration: Add buffered_flush error handling Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 22/35] migration: move exit condition to migration thread Juan Quintela
2012-12-11 12:46 ` [Qemu-devel] [PATCH 23/35] migration: unfold rest of migrate_fd_put_ready() into thread Juan Quintela
2012-12-11 12:47 ` [Qemu-devel] [PATCH 24/35] ram: rename last_block to last_seen_block Juan Quintela
2012-12-11 12:47 ` [Qemu-devel] [PATCH 25/35] ram: Add last_sent_block Juan Quintela
2012-12-11 12:47 ` [Qemu-devel] [PATCH 26/35] memory: introduce memory_region_test_and_clear_dirty Juan Quintela
2012-12-11 12:47 ` [Qemu-devel] [PATCH 27/35] ram: Use memory_region_test_and_clear_dirty Juan Quintela
2012-12-11 12:47 ` [Qemu-devel] [PATCH 28/35] migration: Only go to the iterate stage if there is anything to send Juan Quintela
2012-12-11 12:47 ` [Qemu-devel] [PATCH 29/35] ram: optimize migration bitmap walking Juan Quintela
2012-12-11 12:47 ` [Qemu-devel] [PATCH 30/35] ram: account the amount of transferred ram better Juan Quintela
2012-12-11 12:47 ` [Qemu-devel] [PATCH 31/35] ram: refactor ram_save_block() return value Juan Quintela
2012-12-11 12:47 ` [Qemu-devel] [PATCH 32/35] ram: add free_space parameter to save_live functions Juan Quintela
2012-12-11 12:47 ` [Qemu-devel] [PATCH 33/35] ram: remove xbrle last_stage optimization Juan Quintela
2012-12-11 12:47 ` [Qemu-devel] [PATCH 34/35] ram: reuse ram_save_iterate() for the complete stage Juan Quintela
2012-12-11 12:47 ` [Qemu-devel] [PATCH 35/35] migration: print times for end phase Juan Quintela
2012-12-11 14:22 ` [Qemu-devel] [PATCH 00/35] Migration thread (20121211) Paolo Bonzini
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).