From mboxrd@z Thu Jan 1 00:00:00 1970 From: Bob Peterson Date: Wed, 18 Apr 2018 09:58:36 -0700 Subject: [Cluster-devel] [GFS2 PATCH v1 0/2] Improve throughput through rgrp sharing Message-ID: <20180418165838.8342-1-rpeterso@redhat.com> List-Id: To: cluster-devel.redhat.com MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit This is a preliminary patch set, but the results are promising. I've been testing it pretty hard in a variety of circumstances, and it seems to be fairly solid, so I thought I'd get it out here for people to review. On 17 January 2018, I posted an experimental patch set meant to improve intra-node resource group sharing, titled "GFS2: Rework rgrp glock congestion functions for intra-node". It improved rgrp contention by simply distributing contentious processes to use different rgrps. In RHEL6 we used "try locks" which basically accomplished the same thing. Steve Whitehouse suggested a better approach: to actually share the same rgrps within a node. This patch set implements Steve's suggestion. The first patch introduces a new glock locking mode called EXSH, meaning exclusively shared within one node. To all other nodes (and to DLM) the glock looks and acts like it is held EX. But to the node that has it locked, it may be shared among processes like an SH lock. The second patch adds hooks to the rgrp code to use the new glock locking mode. A new rwsem, rd_sem, ensures exclusive use of the rgrp when it is needed. Whenever an rgrp is added to a transaction, the rwsem is taken and it is queued to the transaction. When the transaction is ended, every rwsem for all rgrps queued to that transaction are unlocked. Preliminary performance testing using iozone looks very promising. With 16 simultaneous writers, GFS2 performs 6 times faster with the patch. Even with 4 writers, overall performance is doubled: 7.5 kernel Patched kernel -------------- -------------- Children see throughput for 1 initial writers 525062.81 kB/s 527972.50 kB/s Parent sees throughput for 1 initial writers 525049.74 kB/s 527971.69 kB/s Children see throughput for 2 initial writers 612600.62 kB/s 603398.75 kB/s Parent sees throughput for 2 initial writers 600944.08 kB/s 603140.65 kB/s Children see throughput for 4 initial writers 596730.64 kB/s 694901.31 kB/s Parent sees throughput for 4 initial writers 232777.32 kB/s 472287.19 kB/s Children see throughput for 6 initial writers 574034.05 kB/s 739531.62 kB/s Parent sees throughput for 6 initial writers 160751.73 kB/s 515363.98 kB/s Children see throughput for 8 initial writers 644463.33 kB/s 727810.48 kB/s Parent sees throughput for 8 initial writers 155939.49 kB/s 559100.85 kB/s Children see throughput for 10 initial writers 613880.30 kB/s 736029.91 kB/s Parent sees throughput for 10 initial writers 174366.86 kB/s 663429.43 kB/s Children see throughput for 12 initial writers 610206.54 kB/s 744490.04 kB/s Parent sees throughput for 12 initial writers 150910.72 kB/s 682414.33 kB/s Children see throughput for 14 initial writers 625055.97 kB/s 804518.57 kB/s Parent sees throughput for 14 initial writers 129122.67 kB/s 781340.39 kB/s Children see throughput for 16 initial writers 627972.96 kB/s 794149.06 kB/s Parent sees throughput for 16 initial writers 124565.02 kB/s 764981.28 kB/s There are still some fairness/parallelism issues. It's not perfect. But when multiple processes are sharing the same resource, I'm not sure how much better we can go without separating them to their own rgrps. The statistics indicate this is well worth pursuing. --- Bob Peterson (2): GFS2: Introduce EXSH (exclusively shared on one node) GFS2: Take advantage of new EXSH glock mode for rgrps fs/gfs2/bmap.c | 2 +- fs/gfs2/dir.c | 2 +- fs/gfs2/glock.c | 12 +++++++- fs/gfs2/glock.h | 16 +++++++--- fs/gfs2/glops.c | 3 +- fs/gfs2/incore.h | 12 +++++--- fs/gfs2/inode.c | 4 +-- fs/gfs2/lock_dlm.c | 5 +++- fs/gfs2/rgrp.c | 84 ++++++++++++++++++++++++++++++++++++++++++++++++---- fs/gfs2/rgrp.h | 5 ++++ fs/gfs2/super.c | 2 +- fs/gfs2/trace_gfs2.h | 2 ++ fs/gfs2/trans.c | 16 ++++++++++ fs/gfs2/xattr.c | 6 ++-- 14 files changed, 147 insertions(+), 24 deletions(-) -- 2.14.3