Re: XFS File system in trouble

Dave Chinner
2015-08-18T02:14:19Z

On Fri, Aug 14, 2015 at 06:12:10PM -0500, Leslie Rhorer wrote:
>
> OK, try http://fletchergeek.com/images/md0.metadump.gz. It's only
> about 18M.

There's a very specific corruption that has occurred here. There are
7 inodes in a row (i.e. same 4k block) with the same corruption
signature. Only the first inode has the version number corrupted, so
that is the only one that is being picked up as corrupt on
allocation, however.

The pattern is that there are certain fields that have wacky values.
The timestamps, the uid/gid, the projid_lo/hi, dmstate
(completely unused field), the extent size and the generation
number.

The generation number is particularly interesting, because every
inode in a chunk is stamped with the same number when the inode
chunk is allocated on disk. The majority of the inodes in the chunk
(which have no corruption) have a value of 0x93dc8d4. One of the
corrupted inodes has a value of 0x7b4bada6, which if valid means the
inode has been allocated and freed almost 2 billion times.

That doesn't seem reasonable, especially as the other corrupted
inodes also having generation cycle deltas of between 50 million and
1.2 billion I'm verymuch doubting these numbers are correct as it
implies that these 7 inodes have run through almost 3.7 billion
inode alloc/free cycles just by themselves.

What's really interesting is this pattern shows up in most of the
corrupted inodes:

600: 494e0000 02020000 00bdf700 00aa4ae8 00000000 005a4a00 00054800 00a94806
^^^^ ^^^^ ^^^^ ^^^^ ^^^^
620: 011cc1e5 3c3aa24c 1ca3ba8c 00a94800 004bf6a2 3868ff92 00000000 00000000
^^^^ ^^^^
640: 00000000 00000000 0087f500 00000000 00000002 00000000 004b0000 093880d4
^^^^ ^^^^ ^^^^

All the highlighted bytes are ones that I can confirm are corrupt.
They are all the middle 2 bytes of a 4 byte word, and they are all
random garbage. The last four (of 7) corrupted inodes have the same corrupted
bytes. The bytes have different values, but they all have, at
minimum, the above bytes corrupted. The initial inode that was
corrupted (with version = 3) has more corrupted bytes than the
others, but the corruption follows a very similar pattern and is
almost identical in inodes 2 and 3.

Patterns of corruption like this don't come from software. All of
the corruptions are in the first 64 byte cacheline of the inodes,
and all have a very similar pattern of corruption and the corrupted
byte values all appear to be random. Given that you initially said
this:

| I found the problem with md5sum (and probably nfs, as
| well). One of the memory modules in the server was bad. The
| problem with XFS persists. Every time tar tried to create the
| directory:

That's what caused the corruption on disk. XFS has validated the
buffer while it was hot in the CPU cache, and when submitted to the
hardware to DMA it to disk it first has to be written back to
memory. That inode buffer page happened to span the bad memory
location and so the inodes were corrupted on their way to disk by
the bad memory.

Not a software bug, but a clear demonstration of why we consider
metadata CRCs very important. On a v5 filesystem, this type of
metadata corruption will show up as a CRC failure, and hence we'll
know straight away that the likely cause is a hardware issue....

And to close the loop, I have confirmed that Roger's patch fixes
repair - it detects the bad inode and fixes it (tested against
xfsprogs 4.2.0-rc1).

Cheers,

Dave.
--
Dave Chinner
david@fromorbit.com