From mboxrd@z Thu Jan 1 00:00:00 1970 From: George Dunlap Subject: Re: [PATCHv2 3/3] p2m: convert grant table rwlock to percpu rwlock Date: Wed, 25 Nov 2015 12:00:22 +0000 Message-ID: <5655A2D6.6020106@citrix.com> References: <1448035423-24242-1-git-send-email-malcolm.crossley@citrix.com> <1448035423-24242-4-git-send-email-malcolm.crossley@citrix.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Received: from mail6.bemta14.messagelabs.com ([193.109.254.103]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1a1YkO-000535-VN for xen-devel@lists.xenproject.org; Wed, 25 Nov 2015 12:00:29 +0000 In-Reply-To: <1448035423-24242-4-git-send-email-malcolm.crossley@citrix.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Malcolm Crossley , JBeulich@suse.com, ian.campbell@citrix.com, andrew.cooper3@citrix.com, Marcos.Matsunaga@oracle.com, keir@xen.org, konrad.wilk@oracle.com, george.dunlap@eu.citrix.com Cc: xen-devel@lists.xenproject.org, stefano.stabellini@citrix.com List-Id: xen-devel@lists.xenproject.org On 20/11/15 16:03, Malcolm Crossley wrote: > The per domain p2m read lock suffers from significant contention when > performance multi-queue block or network IO due to the parallel > grant map/unmaps/copies occuring on the DomU's p2m. > > On multi-socket systems, the contention results in the locked compare swap > operation failing frequently which results in a tight loop of retries of the > compare swap operation. As the coherency fabric can only support a specific > rate of compare swap operations for a particular data location then taking > the read lock itself becomes a bottleneck for p2m operations. > > Percpu rwlock p2m performance with the same configuration is approximately > 64 gbit/s vs the 48 gbit/s with grant table percpu rwlocks only. > > Oprofile was used to determine the initial overhead of the read-write locks > and to confirm the overhead was dramatically reduced by the percpu rwlocks. > > Note: altp2m users will not achieve a gain if they take an altp2m read lock > simultaneously with the main p2m lock. > > Signed-off-by: Malcolm Crossley Looks good to me. I'll save the reviewed-by until the question about the calling convention has been discussed. -George