Re: XFS corruption help; xfs_repair isn't working

[Dave Chinner <david@fromorbit.com>]

On Tue, Nov 29, 2022 at 08:49:27PM +0000, Chris Boot wrote:
> Hi all,
> 
> Sorry, I'm mailing here as a last resort before declaring this filesystem
> done for. Following a string of unclean reboots and a dying hard disk I have
> this filesystem in a very poor state that xfs_repair can't make any progress
> on.
> 
> It has been mounted on kernel 5.18.14-1~bpo11+1 (from Debian
> bullseye-backports). Most of the repairs were done using xfsprogs 5.10.0-4
> (from Debian bullseye stable), though I did also try with 6.0.0-1 (from
> Debian bookworm/testing re-built myself).
> 
> I've attached the full log from xfs_repair, but the summary is it all starts
> with multiple instances of this in Phase 3:
> 
> Metadata CRC error detected at 0x5609236ce178, xfs_dir3_block block
> 0xe101f32f8/0x1000
> bad directory block magic # 0x1859dc06 in block 0 for directory inode
> 64426557977
> bad bestfree table in block 0 in directory inode 64426557977: repairing
> table

I think that the problem is that we are trying to repair garbage
without completely reinitialising the directory block header. We
don't bother checking the incoming directory block for sanity after
the CRC fails, and then we only warn that it has a bad magic number.

We then go a process it as though it is a directory block,
essentially trusting that the directory block header is actually
sane. Which it clearly isn't because the magic number in the dir
block has been trashed.

We then rescan parts of the directory block and rewrite parts of the
block header, but the next time we re-scan the block we find that
there are still bad parts in the header/directory block. Then we
rewrite the magic number to make it look like a directory block,
and when repair is finished it goes to write the recovered directory
block to disk and it fails the verifier check - it's still a corrupt
directory block because it's still full of garbage that doesn't pass
muster.

From a recovery persepective, I think that if we get a bad CRC and
an unrecognisable magic number, we have no idea what the block is
meant to contain - we cannot trust it to contain directory
information, so we should just trash the block rather than try to
rebuild it. If it was a valid directory block, this will result in
the files it pointed to being moved to lost+found so no data is
actually lost.

If it wasn't a dir block at all, then simply trashing the data fork
of the inode and not touching the contents of the block at all is
right thing to do. Modifying something that may be cross-linked
before we've resolved all the cross-linked extents is a bad thing to
be doing, so if we cannot recognise the block as a directory block,
we shouldn't try to recover it as a directory block at all....

Darrick, what are your thoughts on this?

> As it is the filesystem can be mounted and most data appears accessible, but
> several directories are corrupt and can't be read or removed; the kernel
> reports metadata corruption and CRC errors and returns EUCLEAN.
> 
> Ideally I'd like to remove the corrupt directories, recover as much of
> what's left as possible, and make the filesystem usable again (it's an
> rsnapshot destination) - but I'll take what I can.

Yup, it's only a small number of directory inodes, so we might be
able to do this with some manual xfs_db magic. I think all we'd
need to do is rewrite specific parts of the dir block header and
repair should then do the rest...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com