Interesting article

Interesting article

[Maurice Hilarius]

I read this today:
http://blogs.zdnet.com/storage/?p=162

Would anyone who knows enough about this care to comment?

Thanks in advance for any thoughts..


-- 
With our best regards,

//Maurice W. Hilarius         Telephone: 01-780-456-9771/
/Hard Data Ltd.                FAX:          01-780-456-9772/
/11060 - 166 Avenue         email:maurice@harddata.com/
/Edmonton, AB, Canada/
/     T5X 1Y3/
/

Re: Interesting article

[Chris Worley]

On Wed, Jan 14, 2009 at 1:02 PM, Maurice Hilarius <maurice@harddata.com> wrote:
> I read this today:
> http://blogs.zdnet.com/storage/?p=162
>
> Would anyone who knows enough about this care to comment?

We need disks continuously scanned in spare cycles, and offline the
drives (or remap the sectors) as soon as an error is found.

Also, when rebuilding an array, don't stop due to a read failure.
mark the sector as bad and complete the rebuild.

Waiting for a drive to fail has been too late to recover the array, in
my experience.

Chris
>
> Thanks in advance for any thoughts..
>
>
> --
> With our best regards,
>
> //Maurice W. Hilarius         Telephone: 01-780-456-9771/
> /Hard Data Ltd.                FAX:          01-780-456-9772/
> /11060 - 166 Avenue         email:maurice@harddata.com/
> /Edmonton, AB, Canada/
> /     T5X 1Y3/
> /
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

RE: Interesting article

[Guy Watkins]

} -----Original Message-----
} From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
} owner@vger.kernel.org] On Behalf Of Maurice Hilarius
} Sent: Wednesday, January 14, 2009 3:03 PM
} To: linux-raid@vger.kernel.org
} Subject: Interesting article
} 
} I read this today:
} http://blogs.zdnet.com/storage/?p=162
} 
} Would anyone who knows enough about this care to comment?
} 
} Thanks in advance for any thoughts..
} 
} 
} --
} With our best regards,
} 
} //Maurice W. Hilarius         Telephone: 01-780-456-9771/
} /Hard Data Ltd.                FAX:          01-780-456-9772/
} /11060 - 166 Avenue         email:maurice@harddata.com/
} /Edmonton, AB, Canada/
} /     T5X 1Y3/

I have seen RAID5 arrays loose data because of a bad block during a rebuild
since the late 1990's.  Even on big hardware RAID systems.  I now use 3 or 4
way mirrors or RAID6.  RAID 5 is too risky for me.

Back in the 1990's we mirrored some RAID5 data, but only about 5% of it.
The really important stuff.  And that has saved the day at least once.

Daily read tests are needed, but they don't give any guarantees!

I disagree with the statement "So RAID 6 will give you no more protection
than RAID 5 does now".  Not true at all, RAID5 has always had this problem
and can't recover.  RAID6 can, even with many bad blocks.  As long as you
don't have 2 bad blocks on the same block stripe.  However, if a RAID6 has a
double disk failure and you try to recover, you now have a problem if a
block is bad on another disk, and the chances of that are good with really
big disks.

Guy

Re: Interesting article

[Jon Hardcastle]

I have read about this previously and what i turned up then was that it is true, a failed drive can cause others to fail.. particularly if the drives are old and haven't been regularly made to sweat!

My setup is 2x320GB mirrored 6x500GB RAID 5 with 1 off them as a spare (for now) - all SW raid.. and with an LVM ontop.

I do checks 6 days a week on each of the drives using smartctl and monitor the results - I rotate through the long and short checks so that i dont have 6 drives all doing long checks on the same day! (really effects performance when streaming video from it!)

I also do a raid 'scrub' (check) once a week and if there is an error i do a repair.

Also, when i 'grow' one of the LV's i do a e2fsck -cc on it to 'flush' out any bad sectors by reading AND writing.

This has actually done just this and one of my drives is now showing 800 reallocated sectors! (it is going back)

As a result of this i now plan to do monthly individual drive checks by dismantling the array and running bad blocks on the individual drives.

The point of all this is to flush out any bad sectors before they get chance to be a problem.

I also do a nightly copy of the files I absolutely can not do without (Photos et al) to the mirrored 320GB drives.... (as well as 6monthly DVD's)

Hope this is of interest to someone!

-----------------------
N: Jon Hardcastle
E: Jon@eHardcastle.com
'..Be fearful when others are greedy, and be greedy when others are fearful.'
-----------------------


--- On Wed, 14/1/09, Maurice Hilarius <maurice@harddata.com> wrote:

> From: Maurice Hilarius <maurice@harddata.com>
> Subject: Interesting article
> To: linux-raid@vger.kernel.org
> Date: Wednesday, 14 January, 2009, 8:02 PM
> I read this today:
> http://blogs.zdnet.com/storage/?p=162
> 
> Would anyone who knows enough about this care to comment?
> 
> Thanks in advance for any thoughts..
> 
> 
> -- 
> With our best regards,
> 
> //Maurice W. Hilarius         Telephone: 01-780-456-9771/
> /Hard Data Ltd.                FAX:         
> 01-780-456-9772/
> /11060 - 166 Avenue         email:maurice@harddata.com/
> /Edmonton, AB, Canada/
> /     T5X 1Y3/
> /
> 
> --
> To unsubscribe from this list: send the line
> "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at 
> http://vger.kernel.org/majordomo-info.html

Re: Interesting article

[David Greaves]

Chris Worley wrote:
> On Wed, Jan 14, 2009 at 1:02 PM, Maurice Hilarius <maurice@harddata.com> wrote:
>> I read this today:
>> http://blogs.zdnet.com/storage/?p=162
>>
>> Would anyone who knows enough about this care to comment?
> 
> We need disks continuously scanned in spare cycles, and offline the
> drives (or remap the sectors) as soon as an error is found.
> 
> Also, when rebuilding an array, don't stop due to a read failure.
> mark the sector as bad and complete the rebuild.
> 
> Waiting for a drive to fail has been too late to recover the array, in
> my experience.

Agreed - and I think another huge gap is that when you lose a block you really
should be able to find out what file was affected. Doable on ext3 but not
(AFAIK) xfs.

Though that is a filesystem issue, not a RAID issue.

David

Re: RAID5 recoverability - Was: Interesting article

[Matti Aarnio]

On Wed, Jan 14, 2009 at 01:02:38PM -0700, Maurice Hilarius wrote:
> 
> I read this today:
> http://blogs.zdnet.com/storage/?p=162
> 
> Would anyone who knows enough about this care to comment?
> 
> Thanks in advance for any thoughts..

Simplistic recovery strategy is indeed to fault entire disk upon read-fail,
and then sync everything from other disks to it.   Linux does this at least
on RAID-1, my RAID-5 systems are controllers with internal software to handle
the recovery.

Smarter approach is to use RAID5(or 1) recovery from other disks on given
block, _and_ write the failed block immediately.  It is surprising how often
this makes the problem to go away!

Disk would hard-fault and drop out of array only when that fixup write, or
subsequent verifying read fails.

Even such soft-fault should raise alarm -- "Disk 5 has soft-faulted 200
times in past hour."

(Somebody has patented all of this, no doubt...)


Enhancing Linux MD to do things like I outlined above would be beneficial.


Adding periodic self-activated low-IO-priority read-scanner on array logics
would also do a world of good on array reliabilities.  But then some people
want to keep their arrays sleeping, and start them only occasionally.
Perhaps it would be better to have explicite  mdadm  option to do such scan,
and recommend adding it to crontab.

> -- 
> With our best regards,
> 
> //Maurice W. Hilarius         Telephone: 01-780-456-9771/
> /Hard Data Ltd.               FAX:       01-780-456-9772/
> /11060 - 166 Avenue           email:maurice@harddata.com/
> /Edmonton, AB, Canada/
> /     T5X 1Y3/

  /Matti Aarnio

Re: Interesting article

[Michal Soltys]

David Greaves wrote:
> 
> Agreed - and I think another huge gap is that when you lose a block you really
> should be able to find out what file was affected. Doable on ext3 but not
> (AFAIK) xfs.
> 

It's doable on xfs as well.

http://oss.sgi.com/archives/xfs/2008-10/msg01448.html <- two methods in
that thread.

On related note - if there's a mismatch on some stripe under linux's md 
- is there any way to pinpoint which stripe exactly (or what blocks 
correspond to fixed/checked stripe) ?

ITOW - is there anything besides mismatch_cnt ?

Re: Interesting article

[David Greaves]

Michal Soltys wrote:
> David Greaves wrote:
>>
>> Agreed - and I think another huge gap is that when you lose a block
>> you really
>> should be able to find out what file was affected. Doable on ext3 but not
>> (AFAIK) xfs.
>>
> 
> It's doable on xfs as well.
> 
> http://oss.sgi.com/archives/xfs/2008-10/msg01448.html <- two methods in
> that thread.
> 
> On related note - if there's a mismatch on some stripe under linux's md
> - is there any way to pinpoint which stripe exactly (or what blocks
> correspond to fixed/checked stripe) ?
> 
> ITOW - is there anything besides mismatch_cnt ?

Thanks, I'll take a look at that - first glance says that it's not exactly
end-user material!

David


-- 
"Don't worry, you'll be fine; I saw it work in a cartoon once..."

Re: Interesting article

[Peter Rabbitson]

Michal Soltys wrote:
 > On related note - if there's a mismatch on some stripe under linux's md
> - is there any way to pinpoint which stripe exactly (or what blocks
> correspond to fixed/checked stripe) ?
> 
> ITOW - is there anything besides mismatch_cnt ?

There was this thread[1] I started some time ago. It was acknowledged
that indeed more info could be supplied[2], but afterwards it never went
anywhere :(

[1] http://www.spinics.net/lists/raid/msg19093.html
[2] http://www.spinics.net/lists/raid/msg19141.html