From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: [RFC PATCH 2/2] gnttab: refactor locking for better scalability Date: Tue, 12 Nov 2013 08:07:03 +0000 Message-ID: References: <20131112071857.GA11872@u109add4315675089e695.ant.amazon.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Received: from mail6.bemta4.messagelabs.com ([85.158.143.247]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1Vg90D-00060j-CI for xen-devel@lists.xenproject.org; Tue, 12 Nov 2013 08:07:13 +0000 Received: by mail-wg0-f53.google.com with SMTP id b13so2111864wgh.32 for ; Tue, 12 Nov 2013 00:07:11 -0800 (PST) In-Reply-To: <20131112071857.GA11872@u109add4315675089e695.ant.amazon.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Matt Wilson Cc: Felipe Franciosi , Anthony Liguori , Andrew Cooper , David Vrabel , Jan Beulich , xen-devel@lists.xenproject.org, Matt Wilson , Roger Pau =?ISO-8859-1?B?TW9ubuk=?= List-Id: xen-devel@lists.xenproject.org On 12/11/2013 07:18, "Matt Wilson" wrote: >> Is there any concern about writer starvation here? I know our spinlocks >> aren't 'fair' but our rwlocks are guaranteed to starve out writers if there >> is a steady continuous stream of readers. Perhaps we should write-bias our >> rwlock, or at least make that an option. We could get fancier but would >> probably hurt performance. > > Yes, I'm a little concerned about writer starvation. But so far even > in the presence of very frequent readers it seems like the infrequent > writers are able to get the lock when they need to. However, I've not > tested the iommu=strict path yet. I'm thinking that in that case > there's just going to be frequent writers, so there's less risk of > readers starving writers. For what it's worth, when mapcount() gets in > the picture with persistent grants, I'd expect to see some pretty > significant performance degradation for map/unmap operations. This was > also observed in [1] under different circumstances. The average case isn't the only concern here, but also the worst case, which could maybe tie up a CPU for unbounded time. Could a malicious guest set up such a workload? I'm just thinking we don't want to end up with a DoS XSA on this down the line. :) > But right now I'm more curious about cache line bouncing between all > the readers. I've not done any study of inter-arrival times for > typical workloads (much less some more extreme workloads like we've > been testing), but lock profiling of grant table operations when a > spinlock was used showed some pretty long hold times, which should > translate fairly well to decent rwlock performance. I'm by no means an > expert in this area so I'm eager to hear the thoughts of others. In the read-heavy case the only improvement would be with the old Linux-style biglock (spinlock per CPU; writers must take all spinlocks), or working out a lock-free scheme for readers (perhaps making use of RCU). -- Keir > I should also mention that some of the improvement I mentioned from > 3,000 MB/s to 3,600 MB/s was due to avoiding the m2p override spinlock > in dom0.