[Cluster-devel] [GFS2 PATCH 0/4] Patches to reduce GFS2 fragmentation

[Bob Peterson <rpeterso@redhat.com>]

Hi,

On October 8, I posted a GFS2 patch that greatly reduced inter-node
contention for resource group glocks. The patch was called:
"GFS2: Set of distributed preferences for rgrps". It implemented a
new scheme whereby each node in a cluster tries to "keep to itself"
for allocations. This is not unlike GFS1, which has a different scheme.

Although the patch sped up GFS2 performance in general, it also caused
more file fragmentation, because each node tended to focus on a smaller
subset of resource groups.

Here are run times and file fragmentation extent counts for my favorite
customer application, using a STOCK RHEL7 kernel (no patches):

Run times:
Run 1 time: 2hr 40min 33sec
Run 2 time: 2hr 39min 52sec
Run 3 time: 2hr 39min 31sec
Run 4 time: 2hr 33min 57sec
Run 5 time: 2hr 41min 6sec

Total file extents (File fragmentation):
EXTENT COUNT FOR OUTPUT FILES =  744708
EXTENT COUNT FOR OUTPUT FILES =  749868
EXTENT COUNT FOR OUTPUT FILES =  721862
EXTENT COUNT FOR OUTPUT FILES =  635301
EXTENT COUNT FOR OUTPUT FILES =  689263

The times are bad and the fragmentation level is also bad. If I add
just the first patch, "GFS2: Set of distributed preferences for rgrps"
you can see that the performance improves, but the fragmentation is
worse (I only did three iterations this time):

Run times:
Run 1 time: 2hr 2min 47sec
Run 2 time: 2hr 8min 37sec
Run 3 time: 2hr 10min 0sec

Total file extents (File fragmentation):
EXTENT COUNT FOR OUTPUT FILES =  1011217
EXTENT COUNT FOR OUTPUT FILES =  1025973
EXTENT COUNT FOR OUTPUT FILES =  1070163

So the patch improved performance by 25 percent, but file fragmentation is
30 percent worse. Some of this is undoubtedly due to the SAN array buffering,
hiding our multitude of sins. But not every customer will have this quality
of SAN. So it's important to reduce the fragmentation as well, so that some
people are helped while others are hurt by the patch.

Toward this end, I devised three relatively simple patches that greatly
reduce file fragmentation. With all four patches, the numbers are as follows:

Run times:
Run 1 time: 2hr 5min 46sec
Run 2 time: 2hr 10min 15sec
Run 3 time: 2hr 8min 4sec
Run 4 time: 2hr 9min 27sec
Run 5 time: 2hr 6min 15sec

Total file extents (File fragmentation):
EXTENT COUNT FOR OUTPUT FILES =  330276
EXTENT COUNT FOR OUTPUT FILES =  358939
EXTENT COUNT FOR OUTPUT FILES =  375374
EXTENT COUNT FOR OUTPUT FILES =  383071
EXTENT COUNT FOR OUTPUT FILES =  369269

As you can see, with this combination of four patches, the run times are
good as well as the file fragmentation levels. The file fragmentation is
about twice as good as the stock kernel, and significantly better (almost
three times better) than with the first patch alone.

This patch set includes all four patches.

Bob Peterson (4):
  GFS2: Set of distributed preferences for rgrps
  GFS2: Make block reservations more persistent
  GFS2: Only increase rs_sizehint
  GFS2: If we use up our block reservation, request more next time

 fs/gfs2/file.c       | 10 ++------
 fs/gfs2/incore.h     |  2 ++
 fs/gfs2/lock_dlm.c   |  2 ++
 fs/gfs2/ops_fstype.c |  1 +
 fs/gfs2/rgrp.c       | 69 ++++++++++++++++++++++++++++++++++++++++++++++++----
 5 files changed, 71 insertions(+), 13 deletions(-)

-- 
1.9.3