From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Zhai, Edwin" Subject: Re: slow live magration / xc_restore on xen4 pvops Date: Thu, 03 Jun 2010 16:58:26 +0800 Message-ID: <4C076EB2.9030108@intel.com> References: <20100603010418.GB2028@kremvax.cs.ubc.ca> <20100603064545.GB52378@zanzibar.kublai.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------050307000604070608040606" Return-path: In-Reply-To: <20100603064545.GB52378@zanzibar.kublai.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: keir.fraser@eu.citrix.com, andreas.olsowski@uni.leuphana.de, xen-devel@lists.xensource.com, Ian.Jackson@eu.citrix.com, edwin.zhai@intel.com, jeremy@goop.org List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format. --------------050307000604070608040606 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit I assume this is PV domU rather than HVM, right? 1. we need check if super page is the culprit by SP_check1.patch. 2. if this can fix this problem, we need further check where the extra costs comes: the speculative algorithm, or the super page population hypercall by SP_check2.patch If SP_check2.patch works, the culprit is the new allocation hypercall(so guest creation also suffer); Else, the speculative algorithm. Does it make sense? Thanks, edwin Brendan Cully wrote: > On Thursday, 03 June 2010 at 06:47, Keir Fraser wrote: > >> On 03/06/2010 02:04, "Brendan Cully" wrote: >> >> >>> I've done a bit of profiling of the restore code and observed the >>> slowness here too. It looks to me like it's probably related to >>> superpage changes. The big hit appears to be at the front of the >>> restore process during calls to allocate_mfn_list, under the >>> normal_page case. It looks like we're calling >>> xc_domain_memory_populate_physmap once per page here, instead of >>> batching the allocation? I haven't had time to investigate further >>> today, but I think this is the culprit. >>> >> Ccing Edwin Zhai. He wrote the superpage logic for domain restore. >> > > Here's some data on the slowdown going from 2.6.18 to pvops dom0: > > I wrapped the call to allocate_mfn_list in uncanonicalize_pagetable > to measure the time to do the allocation. > > kernel, min call time, max call time > 2.6.18, 4 us, 72 us > pvops, 202 us, 10696 us (!) > > It looks like pvops is dramatically slower to perform the > xc_domain_memory_populate_physmap call! > > I'll attach the patch and raw data below. > -- best rgds, edwin --------------050307000604070608040606 Content-Type: text/plain; name="SP_check1.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="SP_check1.patch" diff -r 4ab68bf4c37e tools/libxc/xc_domain_restore.c --- a/tools/libxc/xc_domain_restore.c Thu Jun 03 07:30:54 2010 +0100 +++ b/tools/libxc/xc_domain_restore.c Thu Jun 03 16:30:30 2010 +0800 @@ -1392,6 +1392,8 @@ int xc_domain_restore(xc_interface *xch, if ( hvm ) superpages = 1; + superpages = 0; + if ( read_exact(io_fd, &dinfo->p2m_size, sizeof(unsigned long)) ) { PERROR("read: p2m_size"); --------------050307000604070608040606 Content-Type: text/plain; name="SP_check2.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="SP_check2.patch" diff -r 4ab68bf4c37e tools/libxc/xc_domain_restore.c --- a/tools/libxc/xc_domain_restore.c Thu Jun 03 07:30:54 2010 +0100 +++ b/tools/libxc/xc_domain_restore.c Thu Jun 03 16:48:38 2010 +0800 @@ -248,6 +248,7 @@ static int allocate_mfn_list(xc_interfac if ( super_page_populated(xch, ctx, pfn) ) goto normal_page; +#if 0 pfn &= ~(SUPERPAGE_NR_PFNS - 1); mfn = pfn; @@ -263,6 +264,7 @@ static int allocate_mfn_list(xc_interfac DPRINTF("No 2M page available for pfn 0x%lx, fall back to 4K page.\n", pfn); ctx->no_superpage_mem = 1; +#endif normal_page: if ( !batch_buf ) --------------050307000604070608040606 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --------------050307000604070608040606--