From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Dr. David Alan Gilbert" Subject: Re: [RFC Design Doc]Speed up live migration by skipping free pages Date: Thu, 24 Mar 2016 10:23:55 +0000 Message-ID: <20160324102354.GB2230@work-vm> References: <1458632629-4649-1-git-send-email-liang.z.li@intel.com> <20160322190530.GI2216@work-vm> <20160324012424.GB14956@linux-gk3p> <20160324090004.GA2230@work-vm> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: Wei Yang , "qemu-devel@nongnu.org" , "kvm@vger.kernel.org" , "linux-kernel@vger.kenel.org" , "pbonzini@redhat.com" , "rth@twiddle.net" , "ehabkost@redhat.com" , "mst@redhat.com" , "amit.shah@redhat.com" , "quintela@redhat.com" , "mohan_parthasarathy@hpe.com" , "jitendra.kolhe@hpe.com" , "simhan@hpe.com" , "rkagan@virtuozzo.com" , "riel@redhat.com" To: "Li, Liang Z" Return-path: Received: from mx1.redhat.com ([209.132.183.28]:60478 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755318AbcCXKYB (ORCPT ); Thu, 24 Mar 2016 06:24:01 -0400 Content-Disposition: inline In-Reply-To: Sender: kvm-owner@vger.kernel.org List-ID: * Li, Liang Z (liang.z.li@intel.com) wrote: > > * Wei Yang (richard.weiyang@huawei.com) wrote: > > > On Wed, Mar 23, 2016 at 06:48:22AM +0000, Li, Liang Z wrote: > > > [...] > > > >> > 8. Pseudo code > > > >> > Dirty page logging should be enabled before getting the free= page > > > >> > information from guest, this is important because during the > > > >> > process of getting free pages, some free pages may be used a= nd > > > >> > written by the guest, dirty page logging can trace these pag= es. > > > >> > The pseudo code=A0is like below: > > > >> > > > > >> > ----------------------------------------------- > > > >> > MigrationState *s =3D migrate_get_current(); > > > >> > ... > > > >> > > > > >> > memory_global_dirty_log_start(); > > > >> > > > > >> > if (get_guest_mem_info(&info)) { > > > >> > while (!get_free_page_bmap(free_page_bitmap, > > > >> > drop_page_cache) > > > >> && > > > >> > s->state !=3D MIGRATION_STATUS_CANCELLING) { > > > >> > usleep(1000) // sleep for 1 ms > > > >> > } > > > >> > > > > >> > tighten_free_page_bmap =3D > > > >> tighten_guest_free_pages(free_page_bitmap); > > > >> > filter_out_guest_free_pages(tighten_free_page_bmap); > > > >> > } > > > >> > > > >> Given the typical speed of networks; it wouldn't do too much h= arm > > > >> to start sending assuming all pages are dirty and then when th= e > > > >> guest finally gets around to finishing the bitmap then update,= so > > > >> it's asynchronous - and then if the guest never responds we do= n't really > > care. > > > > > > > >Indeed, thanks! > > > > > > > > > > This is interesting. By doing so, the threshold I mentioned in an= other > > > mail is not necessary, since we can do it in parallel. > >=20 > > Actually I just realised it's a little more complex; we can't sync = the dirty > > bitmap again from the guest until after we've received the guests '= free' > > bitmap; that's because we wouldn't know if a 'dirty' page reflected= that a > > page declared as 'free' had now been reused - so there is still an = ordering > > there. > >=20 > > Dave >=20 > Not very complex, we can implement like this: >=20 > 1. Set all the bits in the migration_bitmap_rcu->bmap to 1 > 2. Clear all the bits in ram_list. dirty_memory[DIRTY_MEMORY_MIGRATIO= N] > 3. Send the get_free_page_bitmap request > 4. Start to send pages to destination and check if the free_page_bitm= ap is ready > if (is_ready) { > filter out the free pages from migration_bitmap_rcu->bmap; > migration_bitmap_sync(); > }=20 > continue until live migration complete.=20 > =20 >=20 > Is that right? The order I'm trying to understand is something like: a) Send the get_free_page_bitmap request b) Start sending pages c) Reach the end of memory [ is_ready is false - guest hasn't made free map yet ] d) normal migration_bitmap_sync() at end of first pass e) Carry on sending dirty pages f) is_ready is true f.1) filter out free pages? f.2) migration_bitmap_sync() It's f.1 I'm worried about. If the guest started generating the free bitmap before (d), then a page marked as 'free' in f.1=20 might have become dirty before (d) and so (f.2) doesn't set the dirty again, and so we can't filter out pages in f.1. Dave >=20 > Liang > >=20 > > > > > > >Liang -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK