Best practice for setting up "ordinary" RAID

Best practice for setting up "ordinary" RAID

[Guillaume Betous]

Hi all,

Now my RAID is back thanks to all your precious advices, I want it to
be as reliable as possible, knowing that I own "ordinary" hadrware :
- standard PC (Atom based)
- standard hard drives (some WD Red, but also Green, and Seagate
without ERC capabilities)
- standard need : NAS service, which aims at reliabily serve files for
the whole local network (Gigabit).

So, here is my current setup :
- The RAID contains a spare disk
- I set timeouts to 30" at every boot
- The default SCT ERC value is 7" on my two Red disks
- I cron a check every week

What other advice could be done *before* the failure happens ?

Thanks to all,

gUI

-- 
Pour la santé de votre ordinateur, préférez les logiciels libres.
Lire son mail : http://www.mozilla-europe.org/fr/products/thunderbird/
Browser le web : http://www.mozilla-europe.org/fr/products/firefox/
Suite bureautique : http://www.libreoffice.org/download/
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Best practice for setting up "ordinary" RAID

[Mikael Abrahamsson]

On Fri, 11 Oct 2013, Guillaume Betous wrote:

> Hi all,
>
> Now my RAID is back thanks to all your precious advices, I want it to
> be as reliable as possible, knowing that I own "ordinary" hadrware :
> - standard PC (Atom based)
> - standard hard drives (some WD Red, but also Green, and Seagate
> without ERC capabilities)
> - standard need : NAS service, which aims at reliabily serve files for
> the whole local network (Gigabit).
>
> So, here is my current setup :
> - The RAID contains a spare disk
> - I set timeouts to 30" at every boot

I would recommend setting this to 180 seconds.

> - The default SCT ERC value is 7" on my two Red disks
> - I cron a check every week
>
> What other advice could be done *before* the failure happens ?

RAID6, preferrably with a spare.

Also save mdadm --examine and smartctl -a from all drives at each start 
(or at least after every time you've done a change to the array), so in 
case you get catastrophic superblock failure you can match roles and 
serial numbers of each drive.

I would also recommend having a recent enough kernel so you can do 
in-place replacement. I believe this is 3.3 and later.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: Best practice for setting up "ordinary" RAID

[Guillaume Betous]

> I would recommend setting this to 180 seconds.

Yes, and that's what I've done ! Not 30 seconds, but 3 minutes, you're right.
Sorry :)

> RAID6, preferrably with a spare.

I'm not comfortable with RAID 6 setup. What is the minimal number of
hard drives for having a RAID 6 + spare ?

> Also save mdadm --examine and smartctl -a from all drives at each start (or
> at least after every time you've done a change to the array)

Very good. I'll had this.

> I would also recommend having a recent enough kernel so you can do in-place
> replacement. I believe this is 3.3 and later.

Checked : 3.7, so that's OK.

gUI

-- 
Pour la santé de votre ordinateur, préférez les logiciels libres.
Lire son mail : http://www.mozilla-europe.org/fr/products/thunderbird/
Browser le web : http://www.mozilla-europe.org/fr/products/firefox/
Suite bureautique : http://www.libreoffice.org/download/
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Best practice for setting up "ordinary" RAID

[Mikael Abrahamsson]

On Fri, 11 Oct 2013, Guillaume Betous wrote:

>> I would recommend setting this to 180 seconds.
>
> Yes, and that's what I've done ! Not 30 seconds, but 3 minutes, you're right.
> Sorry :)
>
>> RAID6, preferrably with a spare.
>
> I'm not comfortable with RAID 6 setup. What is the minimal number of
> hard drives for having a RAID 6 + spare ?

Well, I guess that would be 4, but that doesn't make much sense, then you 
could rather have just 3 drives in raid1.

But basically it's usually beneficial to have a RAID6 array over a 
RAID5+spare. I can't think of any "home use" usage patterns where RAID6 
would perform a lot worse than RAID5. You get the benefit of allowing two 
drives to fail or a single drive failure plus read errors to be handled 
without problems.

I have completely stopped using RAID5 it's just not worth it when using 
large non-enterprise drives with 10^-14 read error rates.

What is it about RAID6 that you are not comfortable with?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: Best practice for setting up "ordinary" RAID

[Guillaume Betous]

> What is it about RAID6 that you are not comfortable with?

I've never use it :)

Here the deal : I have 5 drives on my NAS, I can't have more in my
tower (1 drive for the system, 5 drives for the NAS). From Adam's
message, it should be enough for setting up a good RAID 6.

For now, I have a RAID 5 with a spare, meaning that I have 3x(drive
size) as available size. I can have 2 hard drive failures.
If I setup a RAID 6, if I understand well, I'll also have 3x(drive
size) as available size, I will also can have 2 drive failures.

As I read that algorithm on the RAID 6 are way more complex, and
rebuild time is longer, as I have only a Atom D510 as CPU, I consider
that RAID 5 is a better option.

But if you can explain that RAID 6 have pros I haven't seen, I'll
surely swith to RAID 6 (always want to learn :) ).

Thanks,

gUI

-- 
Pour la santé de votre ordinateur, préférez les logiciels libres.
Lire son mail : http://www.mozilla-europe.org/fr/products/thunderbird/
Browser le web : http://www.mozilla-europe.org/fr/products/firefox/
Suite bureautique : http://www.libreoffice.org/download/
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Best practice for setting up "ordinary" RAID

[Mikael Abrahamsson]

On Fri, 11 Oct 2013, Guillaume Betous wrote:

> As I read that algorithm on the RAID 6 are way more complex, and rebuild 
> time is longer, as I have only a Atom D510 as CPU, I consider that RAID 
> 5 is a better option.

It's an old myth that RAID6 uses a lot of CPU. It doesn't, not even 
considering you have an Atom D510.

> But if you can explain that RAID 6 have pros I haven't seen, I'll
> surely swith to RAID 6 (always want to learn :) ).

RAID6 will help you with drive failure plus random read error. RAID5 
doesn't do that.

My recommendation to you is to convert your RAID5+spare to RAID6.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: Best practice for setting up "ordinary" RAID

[Guillaume Betous]

> RAID6 will help you with drive failure plus random read error. RAID5 doesn't
> do that.

Yes, very good point. Next step : switch to RAID 6 :)

gUI

-- 
Pour la santé de votre ordinateur, préférez les logiciels libres.
Lire son mail : http://www.mozilla-europe.org/fr/products/thunderbird/
Browser le web : http://www.mozilla-europe.org/fr/products/firefox/
Suite bureautique : http://www.libreoffice.org/download/
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Best practice for setting up "ordinary" RAID

[Robin Hill]

[-- Attachment #1: Type: text/plain, Size: 1763 bytes --]

On Fri Oct 11, 2013 at 09:14:24AM +0200, Guillaume Betous wrote:

> > What is it about RAID6 that you are not comfortable with?
> 
> I've never use it :)
> 
> Here the deal : I have 5 drives on my NAS, I can't have more in my
> tower (1 drive for the system, 5 drives for the NAS). From Adam's
> message, it should be enough for setting up a good RAID 6.
> 
> For now, I have a RAID 5 with a spare, meaning that I have 3x(drive
> size) as available size. I can have 2 hard drive failures.
> If I setup a RAID 6, if I understand well, I'll also have 3x(drive
> size) as available size, I will also can have 2 drive failures.
> 
RAID 5 + hot spare can only handle 2 drive failures if the second one
occurs after the rebuild onto the hot spare has completed. Any read
error or drive failure during the rebuild and you're in trouble. With
RAID 6 you can handle 2 drive failures at any time, and after a single
drive failure it can still handle read errors without needing to fail
the array.

Rebuilds will put a lot of load on the array, so errors are not
uncommon, especially with modern large hard drives (where a rebuild can
easily take over 24 hours).

> As I read that algorithm on the RAID 6 are way more complex, and
> rebuild time is longer, as I have only a Atom D510 as CPU, I consider
> that RAID 5 is a better option.
> 
They are more complex, but not enough so to trouble any modern CPU - you
might have issues on a 386/486, but I doubt you'll see any difference on
an Atom.

Cheers,
    Robin
-- 
     ___        
    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Best practice for setting up "ordinary" RAID

[Guillaume Betous]

> Any read
> error or drive failure during the rebuild and you're in trouble.

That was exactly what happened to me ! A rebuild is of course a very
big load and another failure becomes more probable.

> With RAID 6 you can handle 2 drive failures at any time

Understood. I'll definitely switch to RAID 6 :)

> They are more complex, but not enough so to trouble any modern CPU - you
> might have issues on a 386/486, but I doubt you'll see any difference on
> an Atom.

Ok. Maybe I should update French version of Wikipedia RAID article :)
I'll test by myself.
For now (RAID 5), on rebuild, it's about 30MB/s. On reading, it's
about 90MB/s (direct access to /dev/md127, not read from FS).

gUI

-- 
Pour la santé de votre ordinateur, préférez les logiciels libres.
Lire son mail : http://www.mozilla-europe.org/fr/products/thunderbird/
Browser le web : http://www.mozilla-europe.org/fr/products/firefox/
Suite bureautique : http://www.libreoffice.org/download/
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Best practice for setting up "ordinary" RAID

[David Brown]

On 11/10/13 10:57, Guillaume Betous wrote:
>> Any read
>> error or drive failure during the rebuild and you're in trouble.
> 
> That was exactly what happened to me ! A rebuild is of course a very
> big load and another failure becomes more probable.
> 
>> With RAID 6 you can handle 2 drive failures at any time
> 
> Understood. I'll definitely switch to RAID 6 :)
> 
>> They are more complex, but not enough so to trouble any modern CPU - you
>> might have issues on a 386/486, but I doubt you'll see any difference on
>> an Atom.

On modern CPUs, the raid6 calculations run fast - and on servers you
often have at least one CPU core doing nothing anyway.

The only slow part of raid6 calculations is when you have to read data
when there are two missing disks.  But reading slowly with raid6 beats
no data at all with raid5!

Partial stripe writes (RMW writes) can be faster with raid5 than with
raid6, although I gather this has been under improvement recently.
(I've lost track of the status of these changes.)  Usually, it is a
small price to pay.

> 
> Ok. Maybe I should update French version of Wikipedia RAID article :)
> I'll test by myself.
> For now (RAID 5), on rebuild, it's about 30MB/s. On reading, it's
> about 90MB/s (direct access to /dev/md127, not read from FS).
> 
> gUI
> 

Re: Best practice for setting up "ordinary" RAID

[Adam Goryachev]

On 11/10/13 17:52, Guillaume Betous wrote:
>> RAID6, preferrably with a spare.
> I'm not comfortable with RAID 6 setup. What is the minimal number of
> hard drives for having a RAID 6 + spare ?
>

Check wikipedia, but minimum of 4 drives for RAID6, consider 2 data
drives plus two redundant drives, allowing you to lose any two drives
(slightly better than RAID10 on 4 drives).

In direct answer, RAID6 + hot spare is a minimum of 5 drives.

Regards,
Adam


-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

Re: Best practice for setting up "ordinary" RAID

[David Brown]

On 11/10/13 09:05, Adam Goryachev wrote:
> On 11/10/13 17:52, Guillaume Betous wrote:
>>> RAID6, preferrably with a spare.
>> I'm not comfortable with RAID 6 setup. What is the minimal number of
>> hard drives for having a RAID 6 + spare ?
>>
> 
> Check wikipedia, but minimum of 4 drives for RAID6, consider 2 data
> drives plus two redundant drives, allowing you to lose any two drives
> (slightly better than RAID10 on 4 drives).
> 
> In direct answer, RAID6 + hot spare is a minimum of 5 drives.
> 

That's just the /sensible/ answer.  The minimum is actually 4 drives -
one data, two redundant, and one spare (though obviously there are
better ways to use your drives).  And with mdadm, you can tell it that
three of these drives are missing.  While such things might seem a bit
silly, it is sometimes convenient to get your raid up and running with
the setup you want even if you haven't physically got the drives in
place yet - you can add them in later.

Re: Best practice for setting up "ordinary" RAID

[Guillaume Betous]

For a personnal use, the issue is often in the number of drive you can
have with your tower/motherboard, more than really the price of hard
drives themselves.

gUI

2013/10/11 David Brown <david.brown@hesbynett.no>:
> On 11/10/13 09:05, Adam Goryachev wrote:
>> On 11/10/13 17:52, Guillaume Betous wrote:
>>>> RAID6, preferrably with a spare.
>>> I'm not comfortable with RAID 6 setup. What is the minimal number of
>>> hard drives for having a RAID 6 + spare ?
>>>
>>
>> Check wikipedia, but minimum of 4 drives for RAID6, consider 2 data
>> drives plus two redundant drives, allowing you to lose any two drives
>> (slightly better than RAID10 on 4 drives).
>>
>> In direct answer, RAID6 + hot spare is a minimum of 5 drives.
>>
>
> That's just the /sensible/ answer.  The minimum is actually 4 drives -
> one data, two redundant, and one spare (though obviously there are
> better ways to use your drives).  And with mdadm, you can tell it that
> three of these drives are missing.  While such things might seem a bit
> silly, it is sometimes convenient to get your raid up and running with
> the setup you want even if you haven't physically got the drives in
> place yet - you can add them in later.
>
>



-- 
Pour la santé de votre ordinateur, préférez les logiciels libres.
Lire son mail : http://www.mozilla-europe.org/fr/products/thunderbird/
Browser le web : http://www.mozilla-europe.org/fr/products/firefox/
Suite bureautique : http://www.libreoffice.org/download/
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Best practice for setting up "ordinary" RAID

[David Brown]

On 11/10/13 11:53, Guillaume Betous wrote:
> For a personnal use, the issue is often in the number of drive you can
> have with your tower/motherboard, more than really the price of hard
> drives themselves.

That's absolutely true.  And for personal use, RAID6 + hot spare is
normally overkill - RAID6 is fine by itself.  You can always add a cold
spare if you need it, such as when you need to replace a drive.
Realistically, the chances of losing data due to disk failure from three
disks are vastly lower than other causes of data loss - user error,
power problems, theft, tripping over the server, cat peeing on the
server, etc.

It is better to use RAID6 than RAID5 + hot spare (this is true in almost
all circumstances, except perhaps when you want to maximise RMW partial
write speeds or to share the hot spare amongst several arrays).  But for
a home NAS with 5 drives, a 3 + 2 raid6 is ideal.

mvh.,

David


> 
> gUI
> 
> 2013/10/11 David Brown <david.brown@hesbynett.no>:
>> On 11/10/13 09:05, Adam Goryachev wrote:
>>> On 11/10/13 17:52, Guillaume Betous wrote:
>>>>> RAID6, preferrably with a spare.
>>>> I'm not comfortable with RAID 6 setup. What is the minimal number of
>>>> hard drives for having a RAID 6 + spare ?
>>>>
>>>
>>> Check wikipedia, but minimum of 4 drives for RAID6, consider 2 data
>>> drives plus two redundant drives, allowing you to lose any two drives
>>> (slightly better than RAID10 on 4 drives).
>>>
>>> In direct answer, RAID6 + hot spare is a minimum of 5 drives.
>>>
>>
>> That's just the /sensible/ answer.  The minimum is actually 4 drives -
>> one data, two redundant, and one spare (though obviously there are
>> better ways to use your drives).  And with mdadm, you can tell it that
>> three of these drives are missing.  While such things might seem a bit
>> silly, it is sometimes convenient to get your raid up and running with
>> the setup you want even if you haven't physically got the drives in
>> place yet - you can add them in later.
>>
>