[Cluster-devel] [PATCH 00/44] fsck.gfs2: Support for checking gfs1 file systems

[Cluster-devel] [PATCH 00/44] fsck.gfs2: Support for checking gfs1 file systems

[Bob Peterson]

Hi,

For a long time now I've been working on a series of patches that
allow fsck.gfs2 to analyze and fix GFS (GFS1) file systems as well
as GFS2 file systems.  There are several reasons to do this:

1. There is no "gfs_fsck" in upstream, RHEL6, Fedora or newer, which
   means that anyone who has an existing GFS1 file system who
   upgrades their cluster to newer software will not be able to
   check their file system before porting it.  In other words,
   when gfs2_convert runs to convert a file system from GFS to GFS2
   it instructs the user to fsck their file system before the
   conversion.  Right now the fsck is impossible unless the gfs_fsck
   is done prior to the software upgrade.
2. The fsck.gfs2 tool has been debugged, maintained and tested much
   more thoroughly than gfs_fsck, so it has made many major advances:
   it makes better decisions and may be able to recover more data
   than gfs_fsck can.
3. The fsck.gfs2 tool is _much_ faster than gfs_fsck.  In some
   extreme cases the speed is orders of magnitude faster.  Some
   fscks that would take gfs_fsck 5 days to run now complete in
   about an hour's time.

In getting fsck.gfs2 to operate on GFS file systems, I've run
various prototypes through a series of GFS (and GFS2) metadata in
order to test how it behaves.  After the tests, I've analyzed the
output to determine if it's making "sane" decisions about the file
system.  In picking through all this output, I've found and fixed
many problems with today's fsck.gfs2.  My collection of GFS metadata
exposed many problems I would never find in my GFS2 metadata
collection, despite it being bigger. (My collection includes
about 36 GFS1 metadata sets and 84 GFS2 metadata sets.)

I've created a set of these 44 patches to address these problems
and add the capability to fix GFS1:

01/44 fsck.gfs2: Make functions consistently use sdp rather than sbp
02/44 fsck.gfs2: Change "if(" to "if ("
03/44 libgfs1: Add a centralized gfs1 variable to superblock variables
04/44 libgfs2: Make check_sb and read_sb operate on gfs1 file systems
05/44 libgfs2: add generic risize function, move gfs1 structures to libgfs2
06/44 fsck.gfs2: Check for blocks wrongly inside resource groups
07/44 fsck.gfs2: Rename function check_leaf to check_ealeaf_block
08/44 fsck.gfs2: eliminate vestigial buffer_head variable in check_leaf
09/44 fsck.gfs2: Rename the nlink functions to make them more intuitive
10/44 fsck.gfs2: Keep di_nlink in sync when adding links for lost+found

11/44 fsck.gfs2: directory entry count was only 16 bits in check_entries
12/44 fsck.gfs2: get rid of triple negative logic
13/44 dirent_repair needs to mark the buffer as modified
14/44 fsck.gfs2: Ask to reclaim unlinked meta on a per-rgrp basis only
15/44 fsck.gfs2: Factor out function to add ".." entry when linking to lost+found
16/44 libgfs2: Use __FUNCTION__ rather than __FILE__ for debug messages
17/44 fsck.gfs2: Don't stop invalidating blocks if an invalid one is found
18/44 fsck.gfs2: Find and clear duplicate references that are leaf blocks
19/44 fsck.gfs2: Move function check_num_ptrs from metawalk.c to pass1.c
20/44 fsck.gfs2: Add duplicate reference processing for leaf blocks

21/44 fsck.gfs2: split function check_leaf_blks to make it more understandable
22/44 fsck.gfs2: Shorten output
23/44 fsck.gfs2: Make output messages more sensible
24/44 fsck.gfs2 pass2: Refactor function set_dotdor_dir
25/44 fsck.gfs2 pass2: When deleting an inode, delete its extended attributes
26/44 fsck.gfs2 pass2: Only delete metadata if "bad" (not "invalid") inode
27/44 fsck.gfs2 pass3: Refactor mark_and_return_parent
28/44 fsck.gfs2: misc cosmetic changes
29/44 fsck.gfs2: check_leaf_blks: Don't use old_leaf if it was a duplicate
30/44 fsck.gfs2: Add find_remove_dup and free_block_if_notdup

31/44 fsck.gfs2: don't free previous rgrp list when trying to repair rgrps
32/44 libgfs2: eliminate gfs1_readi in favor of gfs2_readi
33/44 libgfs2: when adding a new GFS1 block, mark buffers modified
34/44 libgfs2: when mapping gfs1 dinode blocks, use dinode buffer
35/44 libgfs2: move block_map functions to fsck.gfs2
36/44 libgfs2: eliminate gfs1_rindex_read in favor of rindex_read
37/44 libgfs2: combine ri_update and gfs1_ri_update
38/44 libgfs2: combine gfs_inode_read and gfs_inode_get
39/44 libgfs2: move gfs1 functions from edit to libgfs2
40/44 gfs2_edit savemeta: save_inode_data backwards for gfs1

41/44 libgfs2: expand libgfs2's capabilities to operate on gfs1
42/44 fsck.gfs2: Combine block and char device inode types
43/44 fsck.gfs2: four-step duplicate elimination process
44/44 fsck.gfs2: Add ability to check gfs1 file systems

The patches will be posted shortly.  Testing is ongoing.
If you have GFS1 or GFS2 file systems and would like to contribute
metadata for testing, please use "gfs2_edit savemeta", gzip the
output, and send me a link where I can download it.

Regards,

Bob Peterson
Red Hat File Systems

[Cluster-devel] [PATCH 00/44] fsck.gfs2: Support for checking gfs1 file systems

[Andrew Price]

Hi Bob,

Doing an apply-and-build test on this patchset found a few trivial issues:

Firstly, "git am" found these whitespace issues:

  Applying: fsck.gfs2: split function check_leaf_blks to make it
  /home/andy/src/gfs2-utils/.git/rebase-apply/patch:16: trailing whitespace.
  static int warn_and_patch(struct gfs2_inode *ip, uint64_t *leaf_no,

  Applying: libgfs2: move gfs1 functions from edit to libgfs2
  /home/andy/src/gfs2-utils/.git/rebase-apply/patch:89: new blank line 
at EOF.
  +

And then, when building with gcc 4.6, the patchset introduces the 
following warnings:

  In file included from eattr.c:9:0:
  fsck.h: In function ?is_dir?:
  fsck.h:145:2: warning: implicit declaration of function ?S_ISDIR? 
[-Wimplicit-function-declaration]

  In file included from fs_recovery.c:12:0:
  fsck.h: In function ?is_dir?:
  fsck.h:145:2: warning: implicit declaration of function ?S_ISDIR? 
[-Wimplicit-function-declaration]

  initialize.c: In function ?reconstruct_single_journal?:
  initialize.c:1286:2: warning: implicit declaration of function ?time? 
[-Wimplicit-function-declaration]
  initialize.c: In function ?initialize?:
  initialize.c:630:5: warning: ?ret? may be used uninitialized in this 
function [-Wuninitialized]
  initialize.c:617:7: note: ?ret? was declared here

  In file included from inode_hash.c:11:0:
  fsck.h: In function ?is_dir?:
  fsck.h:145:2: warning: implicit declaration of function ?S_ISDIR? 
[-Wimplicit-function-declaration]

  In file included from link.c:12:0:
  fsck.h: In function ?is_dir?:
  fsck.h:145:2: warning: implicit declaration of function ?S_ISDIR? 
[-Wimplicit-function-declaration]

  In file included from main.c:18:0:
  fsck.h: In function ?is_dir?:
  fsck.h:145:2: warning: implicit declaration of function ?S_ISDIR? 
[-Wimplicit-function-declaration]

  metawalk.c: In function ?check_leaf_blks?:
  metawalk.c:622:6: warning: ?okay_to_fix? may be used uninitialized in 
this function [-Wuninitialized]
  metawalk.c:477:6: note: ?okay_to_fix? was declared here

  pass1b.c: In function ?handle_dup_blk?:
  pass1b.c:543:26: warning: variable ?ref_in_invalid_inode? set but not 
used [-Wunused-but-set-variable]

  pass1.c: In function ?pass1?:
  pass1.c:1567:11: warning: variable ?offset? set but not used 
[-Wunused-but-set-variable]
  pass1.c:1566:11: warning: variable ?blk_count? set but not used 
[-Wunused-but-set-variable]

  In file included from pass1c.c:11:0:
  fsck.h: In function ?is_dir?:
  fsck.h:145:2: warning: implicit declaration of function ?S_ISDIR? 
[-Wimplicit-function-declaration]

  In file included from pass3.c:13:0:
  fsck.h: In function ?is_dir?:
  fsck.h:145:2: warning: implicit declaration of function ?S_ISDIR? 
[-Wimplicit-function-declaration]

  In file included from pass4.c:10:0:
  fsck.h: In function ?is_dir?:
  fsck.h:145:2: warning: implicit declaration of function ?S_ISDIR? 
[-Wimplicit-function-declaration]

  In file included from pass5.c:11:0:
  fsck.h: In function ?is_dir?:
  fsck.h:145:2: warning: implicit declaration of function ?S_ISDIR? 
[-Wimplicit-function-declaration]

  In file included from rgrepair.c:15:0:
  fsck.h: In function ?is_dir?:
  fsck.h:145:2: warning: implicit declaration of function ?S_ISDIR? 
[-Wimplicit-function-declaration]

  In file included from fs_bits.h:5:0,
                   from util.c:14:
  fsck.h: In function ?is_dir?:
  fsck.h:145:2: warning: implicit declaration of function ?S_ISDIR? 
[-Wimplicit-function-declaration]

Also I noticed that the subject lines in your patch emails were wrapped 
as this caused git am to chop the ends off the commit shortlogs.

Otherwise this looks like a really promising patchset, nice job :)

Andy

[Cluster-devel] [PATCH 00/44] fsck.gfs2: Support for checking gfs1 file systems

[Bob Peterson]

----- Original Message -----
| Hi Bob,
| 
| Doing an apply-and-build test on this patchset found a few trivial
| issues:
| 
| Firstly, "git am" found these whitespace issues:
| 
| Applying: fsck.gfs2: split function check_leaf_blks to make it
| /home/andy/src/gfs2-utils/.git/rebase-apply/patch:16: trailing
| whitespace.
| static int warn_and_patch(struct gfs2_inode *ip, uint64_t *leaf_no,
| 
| Applying: libgfs2: move gfs1 functions from edit to libgfs2
| /home/andy/src/gfs2-utils/.git/rebase-apply/patch:89: new blank line
| at EOF.
| +
| 
| And then, when building with gcc 4.6, the patchset introduces the
| following warnings:

Hi,

Good catches.  I'll clean these up.

Regards,

Bob Peterson
Red Hat File Systems