[Cluster-devel] [GFS2 PATCH] GFS2: Change minimum reservation size to 64 blocks

[Bob Peterson <rpeterso@redhat.com>]

Hi,

I've been working on the problems of both slow performance and excessive
file fragmentation, especially with regard to the RHEL7 kernel. I've
created some patches (still forthcoming) to improve these things. For a
long time, I thought the fragmentation issue was related to GFS2 not
using its reservations efficiently. I wrote a patch that revises the
calculations in function gfs2_write_calc_reserv, but it didn't seem to
help. I also thought it might be releasing the reservations too early,
but that turned out not to be the case. I've done extensive performance
testing and determined that, even under the best circumstances (IOW by
minimizing both inter and intra-node resource group contention), file
fragmentation can be improved significantly by doubling the minimum
block reservation size from 32 to 64 blocks. To test this, I did
several two+ hour tests (real-life application) that uses lots of
files in GFS2. After the test is complete, I totaled all the file
extents (fragments) using filefrag.

Here are the results of five two-hour runs performed with new, revised
calculations, and today's 32-block reservations:

EXTENT COUNT FOR OUTPUT FILES =  444352
EXTENT COUNT FOR OUTPUT FILES =  468425
EXTENT COUNT FOR OUTPUT FILES =  472464
EXTENT COUNT FOR OUTPUT FILES =  482224
EXTENT COUNT FOR OUTPUT FILES =  481174

Here are the results of five runs performed with the old calculations
and 64-block reservations:

EXTENT COUNT FOR OUTPUT FILES =  365743
EXTENT COUNT FOR OUTPUT FILES =  398628
EXTENT COUNT FOR OUTPUT FILES =  404614
EXTENT COUNT FOR OUTPUT FILES =  401355
EXTENT COUNT FOR OUTPUT FILES =  384599

As you can see, by doubling the minimum reservation size, the files
have about 20 percent fewer extents.

This patch, therefore, doubles the minimum block reservations.

Incidentally, if we don't take measures to minimize resource group
contention (both inter and intra-node) the results are much worse.
Here is the same test, done on a "stock" RHEL7 kernel:

EXTENT COUNT FOR OUTPUT FILES =  826314
EXTENT COUNT FOR OUTPUT FILES =  791406
EXTENT COUNT FOR OUTPUT FILES =  735760

Patch description:

The minimum block reservation size was 32 blocks. This patch doubles
it to 64 blocks.

Regards,

Bob Peterson
Red Hat File Systems

Signed-off-by: Bob Peterson <rpeterso@redhat.com> 
---
diff --git a/fs/gfs2/rgrp.h b/fs/gfs2/rgrp.h
index 5d8f085..e2058a7 100644
--- a/fs/gfs2/rgrp.h
+++ b/fs/gfs2/rgrp.h
@@ -14,11 +14,9 @@
 #include <linux/uaccess.h>
 
 /* Since each block in the file system is represented by two bits in the
- * bitmap, one 64-bit word in the bitmap will represent 32 blocks.
- * By reserving 32 blocks at a time, we can optimize / shortcut how we search
- * through the bitmaps by looking a word at a time.
+ * bitmap, each 64-bit word in the bitmap represents 32 blocks.
  */
-#define RGRP_RSRV_MINBYTES 8
+#define RGRP_RSRV_MINBYTES 16 /* Reserve 64 blocks */
 #define RGRP_RSRV_MINBLKS ((u32)(RGRP_RSRV_MINBYTES * GFS2_NBBY))
 
 struct gfs2_rgrpd;