Re: [Qemu-devel] [PATCH v9 00/56] Postcopy implementation

[Bharata B Rao <bharata@linux.vnet.ibm.com>]

On Fri, Nov 06, 2015 at 03:48:11PM +0000, Dr. David Alan Gilbert wrote:
> * Bharata B Rao (bharata@linux.vnet.ibm.com) wrote:
> 
> > > Where we have iterable, but non-postcopiable devices (e.g. htab
> > > or block migration), complete them before forming the 'package'
> > > but with the CPUs stopped.  This stops them filling up the package.
> > 
> > That helps and the migration suceeds now when I switch to postcopy
> > immediately after starting the migration.
> 
> Excellent.
> 
> > However after postcopy migration, when I attempt to start an incoming
> > instance again to migrate the guest back, I see this failure:
> > 
> > qemu-system-ppc64: cannot set up guest memory 'ppc_spapr.ram': Cannot allocate memory
> > 
> > The same doesn't happen with normal migration.
> 
> Huh that's fun; that's the original source guest that's running out of RAM?
> It's original QEMU should be gone by that point.

Yes, the original source QEMU is gone, but there is not enough memory left
in the host to start another incoming QEMU instance because...

At the beginning
-----------------
$ grep -i mem /proc/meminfo 
MemTotal:       132816832 kB
MemFree:        128781632 kB
MemAvailable:   131668224 kB

After starting the guest (-m 64G,slots=32,maxmem=128G)
------------------------
$ grep -i mem /proc/meminfo 
MemTotal:       132816832 kB
MemFree:        124866880 kB
MemAvailable:   127753728 kB

After starting the destination instance (incoming)
-------------------------------------------------
$ grep -i mem /proc/meminfo 
MemTotal:       132816832 kB
MemFree:        122514880 kB
MemAvailable:   125401920 kB

After postcopy migration completes
----------------------------------
$ grep -i mem /proc/meminfo 
MemTotal:       132816832 kB
MemFree:        55150592 kB
MemAvailable:   58037888 kB

After terminating the source instance
-------------------------------------
$ grep -i mem /proc/meminfo 
MemTotal:       132816832 kB
MemFree:        59432448 kB
MemAvailable:   62319872 kB

So as you can see, postcopy migration will result in guest claiming
its entire RAM memory from host. This doesn't happen during normal migration.

The above results are with the additional patch you sent in this thread
and I was switching to postcopy migration immediately after starting the
migration. Without this patch, when I delay the switching to postcopy
migration a bit, I see the free memory as below when postcopy migration
completes.

$ grep -i mem /proc/meminfo 
MemTotal:       132816832 kB
MemFree:        85018176 kB
MemAvailable:   87937024 kB

Regards,
Bharata.