From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:39784) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZviRO-0006ss-Ai for qemu-devel@nongnu.org; Mon, 09 Nov 2015 04:08:48 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ZviRK-00006Z-VX for qemu-devel@nongnu.org; Mon, 09 Nov 2015 04:08:42 -0500 Received: from mx1.redhat.com ([209.132.183.28]:34437) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZviRK-00006V-Np for qemu-devel@nongnu.org; Mon, 09 Nov 2015 04:08:38 -0500 Date: Mon, 9 Nov 2015 09:08:33 +0000 From: "Dr. David Alan Gilbert" Message-ID: <20151109090832.GA2513@work-vm> References: <1446747083-18205-1-git-send-email-dgilbert@redhat.com> <20151106034846.GC29481@in.ibm.com> <20151106090952.GA2459@work-vm> <20151106122222.GF2459@work-vm> <20151106134341.GG2459@work-vm> <20151106153314.GA14232@in.ibm.com> <20151106154810.GC4267@work-vm> <20151109032116.GB14232@in.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151109032116.GB14232@in.ibm.com> Subject: Re: [Qemu-devel] [PATCH v9 00/56] Postcopy implementation List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Bharata B Rao Cc: aarcange@redhat.com, yamahata@private.email.ne.jp, quintela@redhat.com, liang.z.li@intel.com, "qemu-devel@nongnu.org" , luis@cs.umu.se, "amit.shah@redhat.com" , Paolo Bonzini , David Gibson * Bharata B Rao (bharata@linux.vnet.ibm.com) wrote: > On Fri, Nov 06, 2015 at 03:48:11PM +0000, Dr. David Alan Gilbert wrote: > > * Bharata B Rao (bharata@linux.vnet.ibm.com) wrote: > > > > > > Where we have iterable, but non-postcopiable devices (e.g. htab > > > > or block migration), complete them before forming the 'package' > > > > but with the CPUs stopped. This stops them filling up the package. > > > > > > That helps and the migration suceeds now when I switch to postcopy > > > immediately after starting the migration. > > > > Excellent. > > > > > However after postcopy migration, when I attempt to start an incoming > > > instance again to migrate the guest back, I see this failure: > > > > > > qemu-system-ppc64: cannot set up guest memory 'ppc_spapr.ram': Cannot allocate memory > > > > > > The same doesn't happen with normal migration. > > > > Huh that's fun; that's the original source guest that's running out of RAM? > > It's original QEMU should be gone by that point. > > Yes, the original source QEMU is gone, but there is not enough memory left > in the host to start another incoming QEMU instance because... Ah, so this is ping-pong on one host? > At the beginning > ----------------- > $ grep -i mem /proc/meminfo > MemTotal: 132816832 kB > MemFree: 128781632 kB > MemAvailable: 131668224 kB > > After starting the guest (-m 64G,slots=32,maxmem=128G) > ------------------------ > $ grep -i mem /proc/meminfo > MemTotal: 132816832 kB > MemFree: 124866880 kB > MemAvailable: 127753728 kB > > After starting the destination instance (incoming) > ------------------------------------------------- > $ grep -i mem /proc/meminfo > MemTotal: 132816832 kB > MemFree: 122514880 kB > MemAvailable: 125401920 kB > > After postcopy migration completes > ---------------------------------- > $ grep -i mem /proc/meminfo > MemTotal: 132816832 kB > MemFree: 55150592 kB > MemAvailable: 58037888 kB > > After terminating the source instance > ------------------------------------- > $ grep -i mem /proc/meminfo > MemTotal: 132816832 kB > MemFree: 59432448 kB > MemAvailable: 62319872 kB > > So as you can see, postcopy migration will result in guest claiming > its entire RAM memory from host. This doesn't happen during normal migration. I'll try and see if I can replicate this. Can you : 1) show me the command line you're using 2) Show me /proc/pid/smaps for the destination qemu 3) Turn on the trace postcopy_place_page_zero the theory is that most of your pages should end up as zero pages and not be allocated. Dave > The above results are with the additional patch you sent in this thread > and I was switching to postcopy migration immediately after starting the > migration. Without this patch, when I delay the switching to postcopy > migration a bit, I see the free memory as below when postcopy migration > completes. > > $ grep -i mem /proc/meminfo > MemTotal: 132816832 kB > MemFree: 85018176 kB > MemAvailable: 87937024 kB > > Regards, > Bharata. > -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK