From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46066) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a7Ju2-000864-HP for qemu-devel@nongnu.org; Fri, 11 Dec 2015 04:22:15 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1a7Jty-0001AT-6t for qemu-devel@nongnu.org; Fri, 11 Dec 2015 04:22:14 -0500 Received: from mx1.redhat.com ([209.132.183.28]:45936) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a7Jtx-0001AB-QE for qemu-devel@nongnu.org; Fri, 11 Dec 2015 04:22:10 -0500 Date: Fri, 11 Dec 2015 09:22:00 +0000 From: "Dr. David Alan Gilbert" Message-ID: <20151211092159.GB2987@work-vm> References: <1448357149-17572-1-git-send-email-zhang.zhanghailiang@huawei.com> <1448357149-17572-24-git-send-email-zhang.zhanghailiang@huawei.com> <20151210183457.GJ2570@work-vm> <566A8146.6010700@huawei.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <566A8146.6010700@huawei.com> Subject: Re: [Qemu-devel] [PATCH COLO-Frame v11 23/39] COLO: Implement failover work for Primary VM List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Hailiang Zhang Cc: lizhijian@cn.fujitsu.com, quintela@redhat.com, yunhong.jiang@intel.com, eddie.dong@intel.com, peter.huangpeng@huawei.com, qemu-devel@nongnu.org, arei.gonglei@huawei.com, stefanha@redhat.com, amit.shah@redhat.com, hongyang.yang@easystack.cn * Hailiang Zhang (zhang.zhanghailiang@huawei.com) wrote: > On 2015/12/11 2:34, Dr. David Alan Gilbert wrote: > >* zhanghailiang (zhang.zhanghailiang@huawei.com) wrote: > >>For PVM, if there is failover request from users. > >>The colo thread will exit the loop while the failover BH does the > >>cleanup work and resumes VM. > >> > >>Signed-off-by: zhanghailiang > >>Signed-off-by: Li Zhijian > >>--- > >>v11: > >>- Don't call migration_end() in primary_vm_do_failover(), > >> The cleanup work will be done in migration_thread(). > >>- Remove vm_start() in primary_vm_do_failover() which also been done > >> in migraiton_thread() > >>v10: > >>- Call migration_end() in primary_vm_do_failover() > >>--- > >> include/migration/colo.h | 3 +++ > >> include/migration/failover.h | 1 + > >> migration/colo-failover.c | 7 +++++- > >> migration/colo.c | 56 ++++++++++++++++++++++++++++++++++++++++++-- > >> 4 files changed, 64 insertions(+), 3 deletions(-) > >> > >>diff --git a/include/migration/colo.h b/include/migration/colo.h > >>index ba27719..0b02e95 100644 > >>--- a/include/migration/colo.h > >>+++ b/include/migration/colo.h > >>@@ -32,4 +32,7 @@ void *colo_process_incoming_thread(void *opaque); > >> bool migration_incoming_in_colo_state(void); > >> > >> COLOMode get_colo_mode(void); > >>+ > >>+/* failover */ > >>+void colo_do_failover(MigrationState *s); > >> #endif > >>diff --git a/include/migration/failover.h b/include/migration/failover.h > >>index 882c625..fba3931 100644 > >>--- a/include/migration/failover.h > >>+++ b/include/migration/failover.h > >>@@ -26,5 +26,6 @@ void failover_init_state(void); > >> int failover_set_state(int old_state, int new_state); > >> int failover_get_state(void); > >> void failover_request_active(Error **errp); > >>+bool failover_request_is_active(void); > >> > >> #endif > >>diff --git a/migration/colo-failover.c b/migration/colo-failover.c > >>index 1b1be24..0c525da 100644 > >>--- a/migration/colo-failover.c > >>+++ b/migration/colo-failover.c > >>@@ -32,7 +32,7 @@ static void colo_failover_bh(void *opaque) > >> error_report("Unkown error for failover, old_state=%d", old_state); > >> return; > >> } > >>- /*TODO: Do failover work */ > >>+ colo_do_failover(NULL); > >> } > >> > >> void failover_request_active(Error **errp) > >>@@ -67,6 +67,11 @@ int failover_get_state(void) > >> return atomic_read(&failover_state); > >> } > >> > >>+bool failover_request_is_active(void) > >>+{ > >>+ return ((failover_get_state() != FAILOVER_STATUS_NONE)); > >>+} > >>+ > >> void qmp_x_colo_lost_heartbeat(Error **errp) > >> { > >> if (get_colo_mode() == COLO_MODE_UNKNOWN) { > >>diff --git a/migration/colo.c b/migration/colo.c > >>index cedfc63..7a42fc6 100644 > >>--- a/migration/colo.c > >>+++ b/migration/colo.c > >>@@ -41,6 +41,42 @@ bool migration_incoming_in_colo_state(void) > >> return mis && (mis->state == MIGRATION_STATUS_COLO); > >> } > >> > >>+static bool colo_runstate_is_stopped(void) > >>+{ > >>+ return runstate_check(RUN_STATE_COLO) || !runstate_is_running(); > >>+} > >>+ > >>+static void primary_vm_do_failover(void) > >>+{ > >>+ MigrationState *s = migrate_get_current(); > >>+ int old_state; > >>+ > >>+ if (s->state != MIGRATION_STATUS_FAILED) { > >>+ migrate_set_state(&s->state, MIGRATION_STATUS_COLO, > >>+ MIGRATION_STATUS_COMPLETED); > >>+ } > > > >That's a little odd; it will only move to completed if the current > >state is MIGRATION_STATUS_COLO, but you only do it if the state isn't > >FAILED. You could remove the if and just call migrate_set_state > >like that and it would be safe as long as you really only expect > >to have to deal with MIGRATION_STATUS_COLO. > > > > Yes, you are right, i will remove the judgement, and set the state directly. > > >>+ old_state = failover_set_state(FAILOVER_STATUS_HANDLING, > >>+ FAILOVER_STATUS_COMPLETED); > >>+ if (old_state != FAILOVER_STATUS_HANDLING) { > >>+ error_report("Serious error while do failover for Primary VM," > >>+ "old_state: %d", old_state); > > > >It's generally better to state the reason rather than say it's 'serious'; > >so something like: > > 'Incorrect state (%d) while doing failover for Primary VM' > >tells you more, and looks less scary! > > > > Ha, OK, i will fix it. > > >>+ return; > >>+ } > >>+} > >>+ > >>+void colo_do_failover(MigrationState *s) > >>+{ > >>+ /* Make sure vm stopped while failover */ > >>+ if (!colo_runstate_is_stopped()) { > >>+ vm_stop_force_state(RUN_STATE_COLO); > >>+ } > > > >That feels like a race? couldn't we be just at the end > >of taking a checkpoint and about to restart when you do > >the if, so it reads that it's currently stopped but > >then it restarts it by the time you have a chance to > >do anything? > > Do you mean, after we stopped VM in failover() but before done the failover work, > the migration (checkpoint) thread may starts VM just in the middle time ? > The colo_do_failover() is in the context of BH, it holds > the __lock__, so checkpoint thread has no chance to execute vm_start(). What happens if the failover code was executed just before the checkpoint thread executed the _lock_, when the failover code finishes what happens? (I'm not sure this is a problem - I just thought I should check). > >I see in patch 13 (Save PVM....) you have: > > qemu_mutex_lock_iothread(); > > vm_start(); > > qemu_mutex_unlock_iothread(); > > > >So maybe if that code is executed just before the > >failover, then it would stop at the _lock_, we would > >run here but then as soon as we finish wouldn't it vm_start > >there? > > > > But it seems that, we don't need to stop VM to do the failover work. > I'm not so sure, i will investigate if we can do it without stopping VM. Stopping it seems safer. > > >>+ if (get_colo_mode() == COLO_MODE_PRIMARY) { > >>+ primary_vm_do_failover(); > >>+ } > >>+} > >>+ > >> /* colo checkpoint control helper */ > >> static int colo_ctl_put(QEMUFile *f, uint32_t cmd, uint64_t value) > >> { > >>@@ -122,9 +158,22 @@ static int colo_do_checkpoint_transaction(MigrationState *s, > >> } > >> > >> qemu_mutex_lock_iothread(); > >>+ if (failover_request_is_active()) { > >>+ qemu_mutex_unlock_iothread(); > >>+ ret = -1; > >>+ goto out; > >>+ } > >> vm_stop_force_state(RUN_STATE_COLO); > >> qemu_mutex_unlock_iothread(); > >> trace_colo_vm_state_change("run", "stop"); > >>+ /* > >>+ * failover request bh could be called after > >>+ * vm_stop_force_state so we check failover_request_is_active() again. > >>+ */ > >>+ if (failover_request_is_active()) { > >>+ ret = -1; > >>+ goto out; > >>+ } > > > >I'm confused about why the check is needed specifically here; > >for example can't that happen at any point where we're ousite of the > >iothread lock? e.g. couldn't we set failover just a couple of > >lines lower, lets say just after the s->params.blk= 0 ? > > > > Yes, you are right, failover could happen at any place where we are not > holding iothread lock. We should checkpoint the failover status after every > important steps. I'll add more check in next version ... > > > >> /* Disable block migration */ > >> s->params.blk = 0; > >>@@ -221,6 +270,11 @@ static void colo_process_checkpoint(MigrationState *s) > >> trace_colo_vm_state_change("stop", "run"); > >> > >> while (s->state == MIGRATION_STATUS_COLO) { > >>+ if (failover_request_is_active()) { > >>+ error_report("failover request"); > >>+ goto out; > >>+ } > >>+ > >> current_time = qemu_clock_get_ms(QEMU_CLOCK_HOST); > >> if (current_time - checkpoint_time < > >> s->parameters[MIGRATION_PARAMETER_CHECKPOINT_DELAY]) { > >>@@ -242,8 +296,6 @@ out: > >> if (ret < 0) { > >> error_report("%s: %s", __func__, strerror(-ret)); > >> } > >>- migrate_set_state(&s->state, MIGRATION_STATUS_COLO, > >>- MIGRATION_STATUS_COMPLETED); > > > >I had to think about this; but yes, I guess the only way out is via the failover > >which does the completed above. > > > > Yes, thanks. > > Hailiang > > >Dave > > > >> qsb_free(buffer); > >> buffer = NULL; > >>-- > >>1.8.3.1 > >> > >> > >-- > >Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > > > >. > > > > -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK