[Cluster-devel] [GFS2 PATCH 0/3] GFS2: Reduce contention on gfs2_log_lock

[Bob Peterson <rpeterso@redhat.com>]

Simultaneous writes from a single node were very slow due to
contention for the gfs2_log_lock. This lock is used to protect the
journaled data and metadata buffer lists, and the list headers in the
superblock that hold pointers to them. The log_lock is used whenever
a metadata block is added to a transaction (which happens a lot).
The problem is, the vast majority of times this call is made (e.g.
to function gfs2_trans_add_meta) the metadata buffer is already in
the log (in the superblock's list), and therefore, the locking is not
needed, and causes unnecessary contention. Since it's a spin_lock,
this consumes lots of CPU time. The more simultaneous writers, the
more CPUs are contending for the spin_lock. In other words, they
wait for permission to add a buffer to the journal, when in fact,
most of the time, the buffer is already there.

This is a series of patches designed to reduce contention on the
gfs2_log_lock. It works by not locking the log lock unless it
determines the metadata block is not already in the transaction.
It can safely do that by checking the value in the bd while the
buffer_head is locked.

IMPORTANT: This patch hinges on a lock inversion: I switched the
order of the gfs2_log_lock and the ail lock throughout gfs2.
Without doing that, you can't provide adequate protection of the
lists in the superblock without holding the gfs2_log_lock for long
periods of time. There were other places where the log_lock was
locked unnecessarily, and I've tried to eliminate them.

Also note that this patch tries to separate the log_lock, which
protects the sd_log_le_buf (and similar) and its constituent
bd_list members, and the sd_ail_lock which protects the active
items lists.

Also, prior to this patch, the allocation of new gfs2_bufdata items
was protected by the gfs2_log_lock. Now it's entirely protected by
lock_buffer in a new function, lock_bh_get_bd.

I consider these experimental patches for now; so not the final
solution. Testing is still ongoing, but positive. I'm just looking
for feedback at this point.

Signed-off-by: Bob Peterson <rpeterso@redhat.com>
---
Bob Peterson (3):
  GFS2: Switch tr_touched to flag in transaction
  GFS2: Inline function meta_lo_add
  GFS2: Reduce contention on gfs2_log_lock

 fs/gfs2/aops.c    |  23 ++++--------
 fs/gfs2/glops.c   |   4 +--
 fs/gfs2/incore.h  |  10 ++++--
 fs/gfs2/log.c     |  12 +++----
 fs/gfs2/lops.c    |   3 +-
 fs/gfs2/meta_io.c |  17 +++++----
 fs/gfs2/trans.c   | 103 ++++++++++++++++++++++++------------------------------
 7 files changed, 80 insertions(+), 92 deletions(-)

-- 
2.9.3