[Cluster-devel] [fsck.gfs2 v2 PATCH 00/40] fsck.gfs2: memory reduction patches

[Bob Peterson <rpeterso@redhat.com>]

On 15 April, I posted a set of 30 patches for fsck.gfs2 to reduce its
memory requirements. Since that time, I've been testing and fixing
problems. Here is version 2.
---
Right now, fsck.gfs2 is a memory hog with regard to very large file
systems. A file system of 250TB would require an absurd about of memory.
That's because it uses memory for (1) rgrp bitmaps, (2) its own internal
blockmap, (3) a directory tree, and (4) a tree of ALL inodes in the
file system, for reconciling link counts.

This series of 40 patches is designed to reduce the memory footprint
of fsck.gfs2. It does so by almost entirely eliminating the inode
tree in favor of two much smaller bitmaps. In order to do that, I
rearranged the order in which things are done.

In order to save on memory, some things will be slower, so this will
cause a performance hit to fsck.gfs2. However, I'm confident that
we can fix them and make it fast again. The slowness is caused because
most bitmap manipulation is now done on the bitmaps in the resource
groups, which means the code needs to search out the rgrp for these
operations. This can be optimized to pass in the rgrp, since it will
be known in most cases (for example, pass1 operates on one rgrp at a
time, and chances are quite good that the rgrp needed will be that one.)
These further optimizations will be done in later patches.

I'm still testing these changes against my collection of gfs2 and gfs1
metadata, but the results are good so far. So there may be changes.

Signed-off-by: Bob Peterson <rpeterso@redhat.com>
---
Bob Peterson (40):
  fsck.gfs2: Move pass5 to immediately follow pass1
  fsck.gfs2: Convert block_type to bitmap_type after pass1 and 5
  fsck.gfs2: Change bitmap_type variables to int
  fsck.gfs2: Use di_entries to determine if lost+found was created
  fsck.gfs2: pass1b shouldn't complain about non-bitmap blocks
  fsck.gfs2: Change all fsck_blockmap_set to fsck_bitmap_set
  fsck.gfs2: Move set_ip_blockmap to pass1
  fsck.gfs2: Remove unneeded parameter instree from set_ip_blockmap
  fsck.gfs2: Move leaf repair to pass2
  fsck.gfs2: Eliminate astate code
  fsck.gfs2: Move reprocess code to pass1
  fsck.gfs2: Separate out functions that may only be done after pass1
  fsck.gfs2: Divest check_metatree from fsck_blockmap_set
  fsck.gfs2: eliminate fsck_blockmap_set from check_eattr_entries
  fsck.gfs2: Move blockmap stuff to pass1.c
  fsck: make pass1 call bitmap reconciliation AKA pass5
  fsck.gfs2: make blockmap global variable only to pass1
  fsck.gfs2: Add wrapper function pass1_check_metatree
  fsck.gfs2: pass counted_links into fix_link_count in pass4
  fsck.gfs2: refactor pass4 function scan_inode_list
  fsck.gfs2: More refactoring of pass4 function scan_inode_list
  fsck.gfs2: Fix white space problems
  fsck.gfs2: move link count info for directories to directory tree
  fsck.gfs2: Use bitmaps instead of linked list for inodes w/nlink == 1
  fsck.gfs2: Refactor check_n_fix_bitmap to make it more readable
  fsck.gfs2: adjust rgrp inode count when fixing bitmap
  fsck.gfs2: blocks cannot be UNLINKED in pass1b or after that
  fsck.gfs2: Add error checks to get_next_leaf
  fsck.gfs2: re-add a non-allocating repair_leaf to pass1
  libgfs2: Allocate new GFS1 metadata as type 3, not type 1
  fsck.gfs2: Undo partially done metadata records
  fsck.gfs2: Eliminate redundant code in _fsck_bitmap_set
  fsck.gfs2: Fix inode counting bug
  fsck.gfs2: Adjust bitmap for lost+found after adding to dirtree
  GFS2: Add initialization checks for GFS1 used metadata
  fsck.gfs2: Use BLKST constants to make pass5 more clear
  fsck.gfs2: Fix GFS1 "used meta" accounting bug
  fsck.gfs2: pass1b is too noisy wrt gfs1 non-dinode metadata
  fsck.gfs2: Fix rgrp dinode accounting bug
  fsck.gfs2: Fix rgrp accounting in check_n_fix_bitmap

 gfs2/fsck/Makefile.am         |   1 +
 gfs2/fsck/afterpass1_common.c | 319 ++++++++++++++++
 gfs2/fsck/afterpass1_common.h |  31 ++
 gfs2/fsck/fsck.h              |  26 +-
 gfs2/fsck/initialize.c        |  63 +++-
 gfs2/fsck/inode_hash.c        |   3 +-
 gfs2/fsck/link.c              | 139 ++++++-
 gfs2/fsck/link.h              |   6 +-
 gfs2/fsck/lost_n_found.c      |  17 +-
 gfs2/fsck/main.c              |  38 +-
 gfs2/fsck/metawalk.c          | 847 ++++++++----------------------------------
 gfs2/fsck/metawalk.h          |  59 +--
 gfs2/fsck/pass1.c             | 479 ++++++++++++++++++++++--
 gfs2/fsck/pass1b.c            | 148 +++++---
 gfs2/fsck/pass2.c             | 276 ++++++++++----
 gfs2/fsck/pass3.c             |  52 +--
 gfs2/fsck/pass4.c             | 339 ++++++++++-------
 gfs2/fsck/pass5.c             |  60 +--
 gfs2/fsck/util.c              | 171 +++------
 gfs2/fsck/util.h              |  70 ++--
 gfs2/libgfs2/fs_ops.c         |  33 +-
 21 files changed, 1849 insertions(+), 1328 deletions(-)
 create mode 100644 gfs2/fsck/afterpass1_common.c
 create mode 100644 gfs2/fsck/afterpass1_common.h

-- 
2.5.5