From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34121) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Uop13-0007b2-RR for qemu-devel@nongnu.org; Tue, 18 Jun 2013 02:03:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Uop12-0002Sz-IP for qemu-devel@nongnu.org; Tue, 18 Jun 2013 02:03:41 -0400 Received: from e23smtp06.au.ibm.com ([202.81.31.148]:58312) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Uop0z-0002Lx-FT for qemu-devel@nongnu.org; Tue, 18 Jun 2013 02:03:40 -0400 Received: from /spool/local by e23smtp06.au.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 18 Jun 2013 15:56:25 +1000 Received: from d23relay03.au.ibm.com (d23relay03.au.ibm.com [9.190.235.21]) by d23dlp03.au.ibm.com (Postfix) with ESMTP id 394FC3578045 for ; Tue, 18 Jun 2013 16:03:25 +1000 (EST) Received: from d23av03.au.ibm.com (d23av03.au.ibm.com [9.190.234.97]) by d23relay03.au.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id r5I63Gq74522366 for ; Tue, 18 Jun 2013 16:03:16 +1000 Received: from d23av03.au.ibm.com (loopback [127.0.0.1]) by d23av03.au.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id r5I63OZL022913 for ; Tue, 18 Jun 2013 16:03:24 +1000 Message-ID: <51BFF7F0.6090908@linux.vnet.ibm.com> Date: Tue, 18 Jun 2013 14:02:24 +0800 From: Lei Li MIME-Version: 1.0 References: <1371397053-4503-1-git-send-email-lilei@linux.vnet.ibm.com> <87hagyhwok.fsf@codemonkey.ws> In-Reply-To: <87hagyhwok.fsf@codemonkey.ws> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [PATCH 0/7 RFC] Localhost live migration List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Anthony Liguori Cc: lagarcia@br.ibm.com, Paolo Bonzini , Andrea Arcangeli , qemu-devel@nongnu.org, quintela@redhat.com On 06/17/2013 12:01 AM, Anthony Liguori wrote: > Lei Li writes: > >> Hi, >> >> This patch series tries to add localhost live migration support >> to Qemu. It is just a draft version which is far from completing >> and I am still working on, send it out to have your suggestions >> and make sure it is the way should be headed. Your comments are >> very appreciated! > Thanks for posting Lei! > > For the benefit of the other reviewers, the goal here is to allow "live > upgrade" of a running QEMU instance. The work flow would look like > this: > > 1) Guests are running QEMU release 1.6.1 > 2) Admin installs QEMU release 1.6.2 via RPM or deb > 3) Admin does localhost migration with page flipping to use new version > of QEMU. > > Page flipping is used in order to avoid requiring that there is enough > free memory to fit an additional copy of the largest guest which is the > requirement today with localhost migration. > >> When doing localhost live migration, the host memory will balloon >> up during the period, might consume double memories for some time. >> So we want to add a new live migration mechanism localhost live >> migration. >> >> The plan is: >> >> 1) Add new command to do localhost migration. >> >> The qmp interface introduced like: >> >> { 'command': 'localhost-migrate', 'data': {'uri': 'str'} } >> >> 2) Use different mechanism than current live migration. >> >> The very basic work flow like: >> >> qemu on the source (the source and destination are both on localhost) >> | >> V >> Stop VM >> | >> V >> Create threads >> | >> V >> Page flipping through vmspice >> | >> V >> MADV_DONTNEED the ram pages which are already flipped > vmsplice() is a weird system call. > > You would think that SPLICE_F_GIFT would imply MADV_DONTNEED but it > doesn't appear today. So to start with, we're doing it explicitly. I see, thanks for your explanation! > > We're planning on also looking at the kernel side of this as today the > combination of vmsplice() to a pipe plus vmsplice() from a pipe() will > not actually flip the page but instead copy it. But I think this is a > fixable problem. > > It would also be handy to flip large pages when THP is active but that's > future work. > > Regards, > > Anthony Liguori > >> | >> V >> Migration completes >> >> As stopping VM first, we expect/resume the page flipping through vmspice >> is fast enough to meet *live migration (low downtime). >> >> Notes: >> Currently the work flow is not exactly the same as description >> above. For the first step, the work flow we plan to implement is: >> stop VM and copy ram pages via unix domain socket, MADV_DONTNEED >> ram pages that already copied. After that, will replace to vmsplice >> mechanism instead of copying pages. >> >> TODO: >> - Working on the completing of ram copy and MADV_DONTNEED part. >> - Introduce a mechanism to exchange a PIPE via SCM_RIGHTS. >> - benchmark/evaluation. >> >> Lei Li (7): >> migration: export MIG_STATE_xxx flags >> savevm: export qemu_save_device_state() >> arch_init: add ram_madvise_free() >> migration-local: implementation of outgoing part >> migration-local: impelementation of incoming part >> migration-local: add option to command line for local incoming >> hmp:add hmp_localhost_migration interface >> >> >> Makefile.objs | 1 + >> arch_init.c | 13 ++++ >> hmp-commands.hx | 17 +++++ >> hmp.c | 13 ++++ >> hmp.h | 1 + >> include/migration/migration.h | 32 +++++++++ >> include/sysemu/sysemu.h | 1 + >> migration-local.c | 155 +++++++++++++++++++++++++++++++++++++++++ >> migration-unix.c | 60 ++++++++++++++++ >> migration.c | 8 -- >> qapi-schema.json | 14 ++++ >> qemu-options.hx | 9 +++ >> qmp-commands.hx | 22 ++++++ >> savevm.c | 2 +- >> vl.c | 14 ++++ >> 15 files changed, 353 insertions(+), 9 deletions(-) >> create mode 100644 migration-local.c > -- Lei