From mboxrd@z Thu Jan 1 00:00:00 1970 From: Matt Wilson Subject: [RFC PATCH 0/2] gnttab: refactor locking for better scalability Date: Mon, 11 Nov 2013 18:03:10 -0800 Message-ID: <1384221792-3456-1-git-send-email-msw@linux.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1Vg3KK-0003k9-0Y for xen-devel@lists.xenproject.org; Tue, 12 Nov 2013 02:03:36 +0000 Received: by mail-pd0-f178.google.com with SMTP id p10so1497897pdj.37 for ; Mon, 11 Nov 2013 18:03:32 -0800 (PST) List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: xen-devel@lists.xenproject.org Cc: Felipe Franciosi , Anthony Liguori , Andrew Cooper , David Vrabel , Jan Beulich , Keir Fraser , Matt Wilson , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= List-Id: xen-devel@lists.xenproject.org From: Matt Wilson As discussed in the Xen Developer Summit Storage Performance BoF, there is a lot of room for improvement in grant table locking. Anthony and I have been working on refactoring the locking over the past few weeks. The performance improvement is considerable and I'd like to hear from others if this approach is fundamentally wrong for some reason. The previous single spinlock per grant table is split into multiple locks. The heavily modified components of the grant table (the maptrack state and the active entries) are now protected by their own spinlocks. The remaining elements of the grant table are read-mostly, so I modified the main grant table lock to be a rwlock to improve concurrency. On the performance improvement: Without persistent grants, a domU with 24 VBDs plummbed to local HDDs in a streaming 2M write workload achieved 1,400 MB/sec before this change. Performance more than doubles with this patch, reaching 3,000 MB/sec before tuning and 3,600 MB/sec after adjusting event channel vCPU bindings. I included the previously posted patch to __gnttab_unmap_common() in the series since it makes a bit more sense in this context, and the follow on refactor patch is on top of it. DISCLAIMER: I ported this patch from a different Xen version earlier today, and I've only compile tested so far. In the original state we've pushed a lot of concurrent I/O through dom0 and haven't seen any stability issues. Matt Wilson (2): gnttab: lock the local grant table earlier in __gnttab_unmap_common() gnttab: refactor locking for better scalability docs/misc/grant-tables.txt | 56 +++++++- xen/arch/x86/mm.c | 4 +- xen/common/grant_table.c | 308 ++++++++++++++++++++++++++--------------- xen/include/xen/grant_table.h | 9 +- 4 files changed, 261 insertions(+), 116 deletions(-) -- 1.7.9.5