Re: File system corruption

[Dave Chinner <david@fromorbit.com>]

On Wed, Oct 24, 2012 at 04:19:28PM -0500, Wayne Walker wrote:
> On 10/12/2012 07:14 PM, Dave Chinner wrote:
> <snip>
> >And SB/AGF 3 and 4 are ok, too. So, the filesystem headers just
> >beyond the 2TB offset are zero. That tends to point to a block
> >device problem, as an offset of 2TB is where a 32 bit sector count
> >will overflow (i.e. 2^32). Next step is to run blktrace/blkparse
> >on the cp workload that generates the error to see if anything
> >actually writes to the 2TB offset region, and if so, where it
> >comes from. Probably best to compress the resultant blkparse
> >output file - it might be quite large but the text will compress
> >well. Cheers, Dave.
> 
> Dave,
> 
> Thank you for your help.
> 
> 10 MB .gz file at http://rx-7.bybent.com/blktrace.sde1.out.gz
> 
> What I can see seems to have most of the writes are around 2^31.

Which is 2^31 sectors of just above 1TB. i.e. writing the data into
AG #1. The writes stop a short way into #AG1. The filesystem does
not issue any writes to the AG#2 headers, only a single read. IOWs,
the filesystem is not overwriting it's own metadata during the
workload, so that implies a problem at a lower storage layer....

IOWs, I can't see the filesystem doing anything wrong here.

FWIW, can you pattern the block device around the 2TB offset and
re-run the test, and see if the sb/agf 2 are zeroed via xfs_db after
the failure occurs? i.e. do something like:

# xfs_io -F -f -c "pwrite 2047g 2g" -c fsync /dev/sde1

the mkfs, run xfs_db to dump sb 2 and agf 2, then run the test and
dump sb 2/agf 2 again after the test? use the same xfs-db scripts as
the previous time (i.e. including the drop caches commands) if you
can.

Cheers,

Dave.

-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs