[Cluster-devel] [GFS2 PATCH] GFS2: Change minimum reservation size to 64 blocks

[Cluster-devel] [GFS2 PATCH] GFS2: Change minimum reservation size to 64 blocks

[Bob Peterson]

Hi,

I've been working on the problems of both slow performance and excessive
file fragmentation, especially with regard to the RHEL7 kernel. I've
created some patches (still forthcoming) to improve these things. For a
long time, I thought the fragmentation issue was related to GFS2 not
using its reservations efficiently. I wrote a patch that revises the
calculations in function gfs2_write_calc_reserv, but it didn't seem to
help. I also thought it might be releasing the reservations too early,
but that turned out not to be the case. I've done extensive performance
testing and determined that, even under the best circumstances (IOW by
minimizing both inter and intra-node resource group contention), file
fragmentation can be improved significantly by doubling the minimum
block reservation size from 32 to 64 blocks. To test this, I did
several two+ hour tests (real-life application) that uses lots of
files in GFS2. After the test is complete, I totaled all the file
extents (fragments) using filefrag.

Here are the results of five two-hour runs performed with new, revised
calculations, and today's 32-block reservations:

EXTENT COUNT FOR OUTPUT FILES =  444352
EXTENT COUNT FOR OUTPUT FILES =  468425
EXTENT COUNT FOR OUTPUT FILES =  472464
EXTENT COUNT FOR OUTPUT FILES =  482224
EXTENT COUNT FOR OUTPUT FILES =  481174

Here are the results of five runs performed with the old calculations
and 64-block reservations:

EXTENT COUNT FOR OUTPUT FILES =  365743
EXTENT COUNT FOR OUTPUT FILES =  398628
EXTENT COUNT FOR OUTPUT FILES =  404614
EXTENT COUNT FOR OUTPUT FILES =  401355
EXTENT COUNT FOR OUTPUT FILES =  384599

As you can see, by doubling the minimum reservation size, the files
have about 20 percent fewer extents.

This patch, therefore, doubles the minimum block reservations.

Incidentally, if we don't take measures to minimize resource group
contention (both inter and intra-node) the results are much worse.
Here is the same test, done on a "stock" RHEL7 kernel:

EXTENT COUNT FOR OUTPUT FILES =  826314
EXTENT COUNT FOR OUTPUT FILES =  791406
EXTENT COUNT FOR OUTPUT FILES =  735760

Patch description:

The minimum block reservation size was 32 blocks. This patch doubles
it to 64 blocks.

Regards,

Bob Peterson
Red Hat File Systems

Signed-off-by: Bob Peterson <rpeterso@redhat.com> 
---
diff --git a/fs/gfs2/rgrp.h b/fs/gfs2/rgrp.h
index 5d8f085..e2058a7 100644
--- a/fs/gfs2/rgrp.h
+++ b/fs/gfs2/rgrp.h
@@ -14,11 +14,9 @@
 #include <linux/uaccess.h>
 
 /* Since each block in the file system is represented by two bits in the
- * bitmap, one 64-bit word in the bitmap will represent 32 blocks.
- * By reserving 32 blocks at a time, we can optimize / shortcut how we search
- * through the bitmaps by looking a word at a time.
+ * bitmap, each 64-bit word in the bitmap represents 32 blocks.
  */
-#define RGRP_RSRV_MINBYTES 8
+#define RGRP_RSRV_MINBYTES 16 /* Reserve 64 blocks */
 #define RGRP_RSRV_MINBLKS ((u32)(RGRP_RSRV_MINBYTES * GFS2_NBBY))
 
 struct gfs2_rgrpd;

[Cluster-devel] [GFS2 PATCH] GFS2: Change minimum reservation size to 64 blocks

[Steven Whitehouse]

Hi,

On 04/10/14 15:33, Bob Peterson wrote:
> Hi,
>
> I've been working on the problems of both slow performance and excessive
> file fragmentation, especially with regard to the RHEL7 kernel. I've
> created some patches (still forthcoming) to improve these things. For a
> long time, I thought the fragmentation issue was related to GFS2 not
> using its reservations efficiently. I wrote a patch that revises the
> calculations in function gfs2_write_calc_reserv, but it didn't seem to
> help. I also thought it might be releasing the reservations too early,
> but that turned out not to be the case. I've done extensive performance
> testing and determined that, even under the best circumstances (IOW by
> minimizing both inter and intra-node resource group contention), file
> fragmentation can be improved significantly by doubling the minimum
> block reservation size from 32 to 64 blocks. To test this, I did
> several two+ hour tests (real-life application) that uses lots of
> files in GFS2. After the test is complete, I totaled all the file
> extents (fragments) using filefrag.
>
> Here are the results of five two-hour runs performed with new, revised
> calculations, and today's 32-block reservations:
>
> EXTENT COUNT FOR OUTPUT FILES =  444352
> EXTENT COUNT FOR OUTPUT FILES =  468425
> EXTENT COUNT FOR OUTPUT FILES =  472464
> EXTENT COUNT FOR OUTPUT FILES =  482224
> EXTENT COUNT FOR OUTPUT FILES =  481174
>
> Here are the results of five runs performed with the old calculations
> and 64-block reservations:
>
> EXTENT COUNT FOR OUTPUT FILES =  365743
> EXTENT COUNT FOR OUTPUT FILES =  398628
> EXTENT COUNT FOR OUTPUT FILES =  404614
> EXTENT COUNT FOR OUTPUT FILES =  401355
> EXTENT COUNT FOR OUTPUT FILES =  384599
>
> As you can see, by doubling the minimum reservation size, the files
> have about 20 percent fewer extents.
>
> This patch, therefore, doubles the minimum block reservations.
>
> Incidentally, if we don't take measures to minimize resource group
> contention (both inter and intra-node) the results are much worse.
> Here is the same test, done on a "stock" RHEL7 kernel:
>
> EXTENT COUNT FOR OUTPUT FILES =  826314
> EXTENT COUNT FOR OUTPUT FILES =  791406
> EXTENT COUNT FOR OUTPUT FILES =  735760
>
> Patch description:
>
> The minimum block reservation size was 32 blocks. This patch doubles
> it to 64 blocks.

Whether that makes sense to do or not depends upon the distribution of 
the file sizes in the workload, and also the way in which the files are 
written. I don't think there is any guarantee that a different workload 
would not produce a different result here. Just out of interest, what is 
the distribution of file sizes in this workload?

I'd much prefer to see an algorithm that is adaptive, rather than simply 
bumping up the default here. We do need to be able to cope with cases 
where the files are much smaller, and without adaptive sizing, we might 
land up creating small holes in the allocations, which would then cause 
problems for later allocations,

Steve.

> Regards,
>
> Bob Peterson
> Red Hat File Systems
>
> Signed-off-by: Bob Peterson <rpeterso@redhat.com>
> ---
> diff --git a/fs/gfs2/rgrp.h b/fs/gfs2/rgrp.h
> index 5d8f085..e2058a7 100644
> --- a/fs/gfs2/rgrp.h
> +++ b/fs/gfs2/rgrp.h
> @@ -14,11 +14,9 @@
>   #include <linux/uaccess.h>
>   
>   /* Since each block in the file system is represented by two bits in the
> - * bitmap, one 64-bit word in the bitmap will represent 32 blocks.
> - * By reserving 32 blocks at a time, we can optimize / shortcut how we search
> - * through the bitmaps by looking a word at a time.
> + * bitmap, each 64-bit word in the bitmap represents 32 blocks.
>    */
> -#define RGRP_RSRV_MINBYTES 8
> +#define RGRP_RSRV_MINBYTES 16 /* Reserve 64 blocks */
>   #define RGRP_RSRV_MINBLKS ((u32)(RGRP_RSRV_MINBYTES * GFS2_NBBY))
>   
>   struct gfs2_rgrpd;