Re: Sudden File System Corruption

[Stan Hoeppner <stan@hardwarefreak.com>]

On 12/4/2013 8:55 PM, Mike Dacre wrote:
...
> I have a 16 2TB drive RAID6 array powered by an LSI 9240-4i.  It has an XFS.

It's a 9260-4i, not a 9240, a huge difference.  I went digging through
your dmesg output because I knew the 9240 doesn't support RAID6.  A few
questions.  What is the LSI RAID configuration?

1.  Level -- confirm RAID6
2.  Strip size?  (eg 512KB)
3.  Stripe size? (eg 7168KB, 14*256)
4.  BBU module?
5.  Is write cache enabled?

What is the XFS geometry?

5.  xfs_info /dev/sda

A combination of these these being wrong could very well be part of your
problems.

...
> IO errors when any requests were made.  This happened while it was being

I didn't see any IO errors in your dmesg output.  None.

> accessed by  5 different users, one was doing a very large rm operation (rm
> *sh on thousands on files in a directory).  Also, about 30 minutes before
> we had connected the globus connect endpoint to allow easy file transfers
> to SDSC.

With delaylog enabled, which I believe it is in RHEL/CentOS 6, a single
big rm shouldn't kill the disks.  But with the combination of other
workloads it seems you may have been seeking the disks to death.

...
> In the end, I successfully repaired the filesystem with `xfs_repair -L
> /dev/sda1`.  However, I am nervous that some files may have been corrupted.

I'm sure your users will let you know.  I'd definitely have a look in
the directory that was targeted by the big rm operation which apparently
didn't finish when XFS shutdown.

> Do any of you have any idea what could have caused this problem?

Yes.  A few things.  The first is this, and it's a big one:

Dec  4 18:15:28 fruster kernel: io scheduler noop registered
Dec  4 18:15:28 fruster kernel: io scheduler anticipatory registered
Dec  4 18:15:28 fruster kernel: io scheduler deadline registered
Dec  4 18:15:28 fruster kernel: io scheduler cfq registered (default)

http://xfs.org/index.php/XFS_FAQ#Q:_I_want_to_tune_my_XFS_filesystems_for_.3Csomething.3E

"As of kernel 3.2.12, the default i/o scheduler, CFQ, will defeat much
of the parallelization in XFS."

*Never* use the CFQ elevator with XFS, and never with a high performance
storage system.  In fact, IMHO, never use CFQ period.  It was horrible
even before 3.2.12.  It is certain that CFQ is playing a big part in
your 120s timeouts, though it may not be solely responsible for your IO
bottleneck.  Switch to deadline or noop immediately, deadline if LSI
write cache is disabled, noop if it is enabled.  Execute this manually
now, and add it to a startup script and verify it is being set at
startup, as it's not permanent:

echo deadline > /sys/block/sda/queue/scheduler

This one simple command line may help pretty dramatically, immediately,
assuming your hardware array parameters aren't horribly wrong for your
workloads, and your XFS alignment correctly matches the hardware geometry.

-- 
Stan





_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs