From mboxrd@z Thu Jan 1 00:00:00 1970 From: David Vrabel Subject: Re: [PATCHv11 4/4] gnttab: use per-VCPU maptrack free lists Date: Fri, 5 Jun 2015 16:55:24 +0100 Message-ID: <5571C66C.5010403@citrix.com> References: <1433262381-8965-1-git-send-email-david.vrabel@citrix.com> <1433262381-8965-5-git-send-email-david.vrabel@citrix.com> <5571D3860200007800081756@mail.emea.novell.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1Z0tyG-0000k6-8N for xen-devel@lists.xenproject.org; Fri, 05 Jun 2015 15:55:48 +0000 In-Reply-To: <5571D3860200007800081756@mail.emea.novell.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Jan Beulich , Tim Deegan Cc: xen-devel , Malcolm Crossley , Keir Fraser , Ian Campbell List-Id: xen-devel@lists.xenproject.org On 05/06/15 15:51, Jan Beulich wrote: >>>> On 02.06.15 at 18:26, wrote: >> Performance analysis of aggregate network throughput with many VMs >> shows that performance is signficantly limited by contention on the >> maptrack lock when obtaining/releasing maptrack handles from the free >> list. >> >> Instead of a single free list use a per-VCPU list. This avoids any >> contention when obtaining a handle. Handles must be released back to >> their original list and since this may occur on a different VCPU there >> is some contention on the destination VCPU's free list tail pointer >> (but this is much better than a per-domain lock). >> >> Increase the default maximum number of maptrack frames by 4 times >> because: a) struct grant_mapping is now 16 bytes (instead of 8); and >> b) a guest may not evenly distribute all the grant map operations >> across the VCPUs (meaning some VCPUs need more maptrack entries than >> others). >> >> Signed-off-by: Malcolm Crossley >> Signed-off-by: David Vrabel >> Acked-by: Tim Deegan > > What version was that ack given for? v9 >> + /* >> + * max_maptrack_frames is per domain so each VCPU gets a share of >> + * the maximum, but allow at least one frame per VCPU. >> + */ >> + if ( v->maptrack_frames >> + && v->maptrack_frames >= max_maptrack_frames / v->domain->max_vcpus ) >> + return -1; > > So with e.g. max_maptrack_frames being 256 and ->max_vcpus > being 129 you'd potentially allow each vCPU to only have exactly > one page despite there being 127 more to use. There's a limit to how many wacky combinations we can support with a single default limit. With the standard defaults and 129 VCPUs: Before 131072 entries (256 * 4096 / 8) After 231168 entries (1024 / 129 * 129 * 4096 / 16) 1792 entries per vcpu. David