From mboxrd@z Thu Jan  1 00:00:00 1970
From: Andrew Cooper <andrew.cooper3@citrix.com>
Subject: Re: [PATCH 2/2] grant_table: convert grant table rwlock
 to percpu rwlock
Date: Tue, 17 Nov 2015 17:30:59 +0000
Message-ID: <564B6453.6050008@citrix.com>
References: <1446573502-8019-1-git-send-email-malcolm.crossley@citrix.com>
	<1446573502-8019-2-git-send-email-malcolm.crossley@citrix.com>
	<564B6C1A02000078000B603C@prv-mh.provo.novell.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Return-path: <xen-devel-bounces@lists.xen.org>
Received: from mail6.bemta14.messagelabs.com ([193.109.254.103])
	by lists.xen.org with esmtp (Exim 4.72)
	(envelope-from <prvs=756b7c393=Andrew.Cooper3@citrix.com>)
	id 1Zyk5w-0002Dq-0K
	for xen-devel@lists.xenproject.org; Tue, 17 Nov 2015 17:31:05 +0000
In-Reply-To: <564B6C1A02000078000B603C@prv-mh.provo.novell.com>
List-Unsubscribe: <http://lists.xen.org/cgi-bin/mailman/options/xen-devel>,
	<mailto:xen-devel-request@lists.xen.org?subject=unsubscribe>
List-Post: <mailto:xen-devel@lists.xen.org>
List-Help: <mailto:xen-devel-request@lists.xen.org?subject=help>
List-Subscribe: <http://lists.xen.org/cgi-bin/mailman/listinfo/xen-devel>,
	<mailto:xen-devel-request@lists.xen.org?subject=subscribe>
Sender: xen-devel-bounces@lists.xen.org
Errors-To: xen-devel-bounces@lists.xen.org
To: Jan Beulich <JBeulich@suse.com>, Malcolm Crossley <malcolm.crossley@citrix.com>
Cc: xen-devel@lists.xenproject.org, keir@xen.org, stefano.stabellini@citrix.com, ian.campbell@citrix.com
List-Id: xen-devel@lists.xenproject.org

On 17/11/15 17:04, Jan Beulich wrote:
>>>> On 03.11.15 at 18:58, <malcolm.crossley@citrix.com> wrote:
>> --- a/xen/common/grant_table.c
>> +++ b/xen/common/grant_table.c
>> @@ -178,6 +178,10 @@ struct active_grant_entry {
>>  #define _active_entry(t, e) \
>>      ((t)->active[(e)/ACGNT_PER_PAGE][(e)%ACGNT_PER_PAGE])
>>  
>> +bool_t grant_rwlock_barrier;
>> +
>> +DEFINE_PER_CPU(rwlock_t *, grant_rwlock);
> Shouldn't these be per grant table? And wouldn't doing so eliminate
> the main limitation of the per-CPU rwlocks?

The grant rwlock is per grant table.

The entire point of this series is to reduce the cmpxchg storm which
happens when many pcpus attempt to grap the same domains grant read lock.

As identified in the commit message, reducing the cmpxchg pressure on
the cache coherency fabric increases intra-vm network through from
10Gbps to 50Gbps when running iperf between two 16-vcpu guests.

Or in other words, 80% of cpu time is wasted with waiting on an atomic
read/modify/write operation against a remote hot cache line.

~Andrew