Odd RAID failure

Odd RAID failure

[Lelsie Rhorer]

I've been discussing this problem over on the linux-raid list, and one of
the participants suggested I post to this list to see if someone here has
any suggestions.

I'm having a severe problem whose root cause I cannot determine.  I have a
RAID 6 array managed by mdadm running on Debian "Lenny" with a 3.2GHz AMD
Athlon 64 x 2 processor and 4G of RAM.  The kernel is 2.6.26-1-amd64.  There
are ten 1 Terabyte SATA drives, unpartitioned, fully allocated to the
/dev/md0 device. The drive are served by 3 Silicon Image SATA port
multipliers and a Silicon Image 4 port eSATA controller.  The /dev/md0
device is also unpartitioned, and all 8T of active space is formatted as a
single Reiserfs file system.  The entire volume is mounted to /RAID.
Various directories on the volume are shared using both NFS and SAMBA.

Performance of the RAID system is very good.  The array can read and write
at over 450 Mbps, and I don't know if the limit is the array itself or the
network, but since the performance is more than adequate I really am not
concerned which is the case.

The issue is the entire array will occasionally pause completely for about
40 seconds when a file is created.  This does not always happen, but the
situation is easily reproducible.  The frequency at which the symptom occurs
seems to be related to the transfer load on the array.  If no other
transfers are in process, then the failure seems somewhat more rare, perhaps
accompanying less than 1 file creation in 10..  During heavy file transfer
activity, sometimes the system halts with every other file creation.
Although I have observed many dozens of these events, I have never once
observed it to happen except when a file creation occurs. 
Reading and writing existing files never triggers the event, although any
read or write occurring during the event is halted for the duration. 
(There is one cron jog which runs every half-hour that creates a tiny file;
this is the most common failure vector.)  There are other drives formatted
with other file systems on the machine, but the issue has never been seen on
any of the other drives.  When the array runs its regularly scheduled health
check, the problem is much worse.  Not only does it lock up with almost
every single file creation, but the lock-up time is much longer - sometimes
in excess of 2 minutes.

Transfers via Linux based utilities (ftp, NFS, cp, mv, rsync, etc) all
recover after the event, but SAMBA based transfers frequently fail, both
reads and writes.  According to iostat, the writes to all 10 drives and to
the array halt abruptly and completely, and reads from 5 of the drives stop
completely, but 5 of the drives still show a dribble of reads happening.  It
is always the same 5 drives.

Re: Odd RAID failure

[Tejun Heo]

Lelsie Rhorer wrote:
...
> The issue is the entire array will occasionally pause completely for about
> 40 seconds when a file is created.  This does not always happen, but the
> situation is easily reproducible.

Does the kernel whine about anything when this happens?

-- 
tejun

Re: Odd RAID failure

[Greg Freemyer]

On Thu, Apr 9, 2009 at 2:13 PM, Tejun Heo <tj@kernel.org> wrote:
> Lelsie Rhorer wrote:
> ...
>> The issue is the entire array will occasionally pause completely for about
>> 40 seconds when a file is created.  This does not always happen, but the
>> situation is easily reproducible.
>
> Does the kernel whine about anything when this happens?

Tejun,

There is a long thread on linux-raid about this issue:

http://markmail.org/message/a32fsnsaa5mmpthw

Per the thread, Leslie is not seeing anything in the logs.

Greg
-- 
Greg Freemyer
Head of EDD Tape Extraction and Processing team
Litigation Triage Solutions Specialist
http://www.linkedin.com/in/gregfreemyer
First 99 Days Litigation White Paper -
http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf

The Norcross Group
The Intersection of Evidence & Technology
http://www.norcrossgroup.com

Re: Odd RAID failure

[Tejun Heo]

Hello,

Greg Freemyer wrote:
> On Thu, Apr 9, 2009 at 2:13 PM, Tejun Heo <tj@kernel.org> wrote:
>> Lelsie Rhorer wrote:
>> ...
>>> The issue is the entire array will occasionally pause completely for about
>>> 40 seconds when a file is created.  This does not always happen, but the
>>> situation is easily reproducible.
>> Does the kernel whine about anything when this happens?
> 
> Tejun,
> 
> There is a long thread on linux-raid about this issue:
> 
> http://markmail.org/message/a32fsnsaa5mmpthw
> 
> Per the thread, Leslie is not seeing anything in the logs.

That was a long thread.  Heh... yeah, at the first look, it looks like
more of VM / FS interaction problem than low level driver problem.
Using blktrace/blkparse to track IO insertion, issue and completion
latencies would help to whether the problem is above the block layer
or below it.

If you can bear with slower operation, keeping issuing sync repeatedly
(ie. something like while true; do sync; sleep 1; done) and see how
the behavior changes might shed some light on what's going on too.
Also, disabling journal and seeing how the behavior changes could be
revealing too.

Thanks.

-- 
tejun

RE: Odd RAID failure

[Leslie Rhorer]

> > Per the thread, Leslie is not seeing anything in the logs.
> 
> That was a long thread.  Heh... yeah, at the first look, it looks like
> more of VM / FS interaction problem than low level driver problem.

Either that, or a bug in Reiserfs, would be my guess.

> Using blktrace/blkparse to track IO insertion, issue and completion
> latencies would help to whether the problem is above the block layer
> or below it.

I'll look into this, too.  I may need some guidance.

 
> If you can bear with slower operation, keeping issuing sync repeatedly
> (ie. something like while true; do sync; sleep 1; done) and see how
> the behavior changes might shed some light on what's going on too.

I wouldn't expect it to have any diagnostic value, really.  The sync command
locks up just like every other disk I/O when the event occurs.

> Also, disabling journal and seeing how the behavior changes could be
> revealing too.

Well, first of all, I would not expect this to have much effect (one never
knows, of course), because the event can transpire when there are no writes
at all going on and no dirty blocks, although once again one never knows.
Secondly, how does one disable the journal in Reiserfs?

RE: Odd RAID failure

[Leslie Rhorer]

> > If you can bear with slower operation, keeping issuing sync repeatedly
> > (ie. something like while true; do sync; sleep 1; done) and see how
> > the behavior changes might shed some light on what's going on too.
> 
> I wouldn't expect it to have any diagnostic value, really.  The sync
> command
> locks up just like every other disk I/O when the event occurs.

I tried this and it doesn't seem to have made a difference.  Neither does
using noop as the scheduler.

Re: Odd RAID failure

[Tejun Heo]

Leslie Rhorer wrote:
>>> If you can bear with slower operation, keeping issuing sync repeatedly
>>> (ie. something like while true; do sync; sleep 1; done) and see how
>>> the behavior changes might shed some light on what's going on too.
>> I wouldn't expect it to have any diagnostic value, really.  The sync
>> command
>> locks up just like every other disk I/O when the event occurs.
> 
> I tried this and it doesn't seem to have made a difference.  Neither does
> using noop as the scheduler.

Hmm... yeah, I was mostly thinking about ext3 for this and the next
suggestion, which often has large latency when flushing its journal to
disk.  I suppose the next stop is trying another filesystem?

Thanks.

-- 
tejun

RE: Odd RAID failure

[Leslie Rhorer]

> >>> If you can bear with slower operation, keeping issuing sync repeatedly
> >>> (ie. something like while true; do sync; sleep 1; done) and see how
> >>> the behavior changes might shed some light on what's going on too.
> >> I wouldn't expect it to have any diagnostic value, really.  The sync
> >> command
> >> locks up just like every other disk I/O when the event occurs.
> >
> > I tried this and it doesn't seem to have made a difference.  Neither
> does
> > using noop as the scheduler.
> 
> Hmm... yeah, I was mostly thinking about ext3 for this and the next
> suggestion, which often has large latency when flushing its journal to
> disk.  I suppose the next stop is trying another filesystem?
Well, I thought of that, of course.  With over 6T of data, that means taking
the array down (mostly) for more than 3 days.  'Not a propect I cherish,
especially since it measn most of the data will then only exost on the
backup system, and over a period of 3 days there are a lot of opportunities
to lose the backup system , especially since it is JBOD.  Indeed, the last
time I re-formatted the RAID array, the backup system did fail.  The
motherboard shorted and took the power supply with it.  Fortunately, I was
able to recover the backup system and only lost a few files.