[Cluster-devel] [PATCH 0/2] GFS2: inplace_reserve performance improvements

[Mark Syms <mark.syms@citrix.com>]

While testing GFS2 as a storage repository for virtual machines we
discovered a number of scenarios where the performance was being
pathologically poor.

The scenarios are simplfied to the following -

  * On a single host in the cluster grow a number of files to a
    significant proportion of the filesystems LUN size, exceeding the
    hosts preferred resource group allocation. This can be replicated
    by using fio and writing to 20 different files with a script like

[test-files]
directory=gfs2/a:gfs2/b:gfs2/c:gfs2/d:gfs2/e:gfs2/f:gfs2/g:gfs2/h:gfs2/i:gfs2/j:gfs2/k:gfs2/l:gfs2/m:gfs2/n:gfs2/o:gfs2/p:gfs2/q:gfs2/r:gfs2/s:gfs2/t
nrfiles=1
size=20G
bs=512k
rw=write
buffered=0
ioengine=libaio
fallocate=none
numjobs=20

    After starting off at network wire speed this will rapidly degrade
    with the fio process reporting large sys time.

    This was diagnosed to all the processes contending on the glock in
    gfs2_inplace_reserve having all selected the same resource
    group. Patch 1 addresses this with an optional module parameter
    which enables behaviour to "randomly" skip a selected resource
    group in the first two passes in gfs_inplace_reserve in order to
    spread the processes out.

    Worth noting that this would probably also be addressed if the
    comment in Documentation/gfs2-glocks.txt about eventually making
    glock EX locally shared was made to happen. However, this looks
    like it would require quite a bit of coordination and design so
    this stop-gap helps in the meantime.

  * With two or more hosts growing files at high data rates the
    throughput drops to a small proportion of the maximum storage
    I/O. This is the several VMs all writing to the filesystem
    scenario. Sometimes this test would run through clean at 80-90% of
    storage wire speed but at other times the performance would drop
    on one or more hosts to a small number of KiB/s.

    This was diagnosed to the different hosts repeatedly bouncing
    resource group glocks between them as different hosts selected
    the same resource group (having exhausted the preferred groups).

    Patch 2 addresses this by -
      * adding a hold delay to the resource group glock if there are
        local waiters, following the pattern already in place for
        inodes, this should also provide more data for
        gfs2_rgrp_congested to work on.
      * remembering when we were last asked to demote the lock on a
        resource group
      * in the first two passes in gfs2_inplace_reserve avoiding
        resource groups where we have been asked to demote the glock
        within the last second

Mark Syms (1):
  GFS2: Avoid recently demoted rgrps.

Tim Smith (1):
  Add some randomisation to the GFS2 resource group allocator

 fs/gfs2/glock.c      |  7 +++++--
 fs/gfs2/incore.h     |  2 ++
 fs/gfs2/main.c       |  1 +
 fs/gfs2/rgrp.c       | 49 +++++++++++++++++++++++++++++++++++++++++++++----
 fs/gfs2/trace_gfs2.h | 12 +++++++++---
 5 files changed, 62 insertions(+), 9 deletions(-)

-- 
1.8.3.1