Re: "Internal error xfs_attr3_leaf_write_verify at line 216", "directory flags set on non-directory inode" and other errors

[Dave Chinner <david@fromorbit.com>]

On Wed, Jul 08, 2015 at 12:28:19PM +0200, Rasmus Borup Hansen
wrote:
> > On 07 Jul 2015, at 02:19, Dave Chinner <david@fromorbit.com>
> > wrote:
> > 
> > On Mon, Jul 06, 2015 at 01:08:52PM +0200, Rasmus Borup Hansen
> > wrote:
> >> I've made a metadump and I'm running another xfs_repair, but
> >> given that the first metadump is 132 GB, will you still be
> >> interested in looking at the dumps?
> > 
> > That's significantly larger than my monthly download quota.  How
> > big is it once you compress it?
> 
> One metadump is 25 GB when compressed with xz -9. The server the
> files currently reside on is not very fast, so I've only
> compressed one of them so far.
> 
> I used the strings command on the metadump files and discovered
> that they contain fragments of files that we really don't want to
> leave our IT systems. However, if you think it's worth the effort,
> I could set up a virtual machine with the metadump files and give
> you access with your SSH public key. But then you'll have to tell
> me which tools you'll need for investigating the files.

<sigh>

I only want to look at one inode with xfs_db....

I don't think ssh access is going be very useful. To isolate the
actual cause of the verifier error I usually run an instrumented
kernel, and to verify that I've fixed the problem I'll need to run
custom kernels and/or xfsprogs binaries. So I really need a local fs
image to do this.

I also like to run a kernel I trust (i.e. one I've built myself) on
a machine I trust not to have storage or hardware issues when doing
diagnostics on broken filesystem images.

Of course, nobody can learn about how to find problems like this
when the triage is hidden away in private....

> Output from ls when listing "lost+found":
> 
> $ ls -laF /backup/lost+found/
> ls: /backup/lost+found/11539619467: Structure needs cleaning

This is the inode I need to look at. Can you grep for this inode in
the xfs_repair output and post it? (grab a couple of lines of
context around each match, too...)

> total 4
> drwxr-xr-x 2 root root     32 Jun 30 07:43 ./
> drwxr-xr-x 5 root root     74 Jul  2 12:55 ../
> -rw-rw-rw- 1 tsj  intomics  0 Jun 23 16:11 11539619467

It's a zero length file, so I'll be interested to know what
attributes it has on it. Can you check that the inode number of
that file is 11539619467 (ls -i will tell you that) and then post
the output of this command:

# xfs_db -r -c "inode 11539619467" -c p /dev/mapper/backup01-data

Then I can give you the commands for walking and dumping the
full attribute tree.

> [503166.562498] XFS (dm-0): Corruption detected. Unmount and run xfs_repair
> [503166.589297] XFS (dm-0): metadata I/O error: block 0x157e84da0 ("xfs_trans_read_buf_map") error 117 numblks 8

Also, post the output of:

# xfs_db -r -c "convert daddr 0x157e84da0 fsb"  /dev/mapper/backup01-data
<fsb>
# xfs_db -r -c "fsb <fsb>" -c p -c "type attr" -c p /dev/mapper/backup01-data

Which will give me the contents of the bad block, both in raw format
and as processed by the attribute leaf format parser in xfs_db.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs