number of global spares?

number of global spares?

[Dan Stromberg]

I've been working on a RAID setup with dual RAID controllers and
three expansion boxes - 48 disks in all, including data, parity and
global spares.

We originally purchased the equipment expecting to get 16 terabytes of
usable space.

Now that it's "all set up", we're really seeing more like 14 or 15
terabytes, depending on how you do the calculation.

Please be sure to use a fixed-pitch font when viewing the tables found
below.  BTW, if people weren't so terrified of HTML, I could just make a
nice HTML table for easy reading without silly font requirements...

Anyway, what we have right now is:

global spares: 0,16,32,48

Raidset	Disks used	                  Data:parity ratio
0	      1,2,3,4,5,6,7,8,9,10          9:1
1	      11,17,18,19,20,21,22,23,24,25	9:1
2	      26,27,33,34,35,36,37,38,39,40	9:1
3	      41,42,43,49,50,51,52,53,54,55	9:1
4	      56,57,58,59                   3:1


And the vendor is suggesting that we move to something like:

global spares: 0

Raidset	Disks used	                  Data:parity ratio
0	      1,2,3,4,5,6,7,8,9,10          9:1
1	      11,17,18,19,20,21,22,23,24,25	9:1
2	      26,27,33,34,35,36,37,38,39,40	9:1
3	      41,42,43,49,50,51,52,53,54,55	9:1
4	      56,57,58,59,16,32,48          3:1

...or...:

global spares: 0,16

Raidset	Disks used	                  Data:parity ratio
0	      1,2,3,4,5,6,7,8,9,10          9:1
1	      11,17,18,19,20,21,22,23,24,25	9:1
2	      26,27,33,34,35,36,37,38,39,40	9:1
3	      41,42,43,49,50,51,52,53,54,55	9:1
4	      56,57,58,59,32,48             3:1


Does anyone have any comments on:

1) The sanity of these 10 disk RAID 5's?

2) The degree of loss of reliability incurred by moving 3 disks from
global spare to data?

3) The degree of loss of reliability incurred by moving 2 disks from
global spare to data?


To answer these questions, you probably need to know how the storage is to
be used.  This single, large filesystem will be used by a variety of
researchers and students from around The University of California, Irvine,
but was purchased primarily by the Earth System Science part of the
Physical Sciences department, which in turn will primarily be storing many
approximately 100 megabyte files which comprise time series related to
climatology simulations.

They don't feel that the storage has to be blazing fast, and 100% uptime
isn't paramount, however they very much do not want to lose their data.

The filesystem will not be backed up - we simply don't have anything large
enough to back it up -to-, so if the some part of the storage solution
goes kerflooey, we're totally...  er...  out of luck, and they'll probably
be looking at me (the primary sysadmin on the storage configuration),
wondering why their data is gone.

Thanks!

Re: number of global spares?

[Mark Hahn]

> I've been working on a RAID setup with dual RAID controllers and
> three expansion boxes - 48 disks in all, including data, parity and
> global spares.

the first question you should ask is whether you're actually
winning by using HW raid.  yes, you already paid for it, but
SW raid offers some noticably better flexibility.

> Please be sure to use a fixed-pitch font when viewing the tables found
> below.  BTW, if people weren't so terrified of HTML, I could just make a
> nice HTML table for easy reading without silly font requirements...

it's not a matter of terror - many people still prefer ascii email.
(naturally, we also use fixed-pitch fonts for this.)

> global spares: 0,16,32,48
> 
> Raidset	Disks used	                  Data:parity ratio
> 0	      1,2,3,4,5,6,7,8,9,10          9:1
> 1	      11,17,18,19,20,21,22,23,24,25	9:1
> 2	      26,27,33,34,35,36,37,38,39,40	9:1
> 3	      41,42,43,49,50,51,52,53,54,55	9:1
> 4	      56,57,58,59                   3:1

why the magic numbers?  (5 raidsets, 9:1, etc)
you have 48 disks and "dual RAID controllers" (one channel each?) 
in 3 boxes, but what are your actual constraints?

also, if you have dual controllers, can you truely have global spares?
that is, a controller can use a spare disk that it's not connected to?

9:1 is nothing to be scared of, though it means that to do a full-stripe
write, you'll need quite large blocks.  I'd be tempted to use raid6
rather than 5+spares, though.

> And the vendor is suggesting that we move to something like:
> 
> global spares: 0
> 
> Raidset	Disks used	                  Data:parity ratio
> 0	      1,2,3,4,5,6,7,8,9,10          9:1
> 1	      11,17,18,19,20,21,22,23,24,25	9:1
> 2	      26,27,33,34,35,36,37,38,39,40	9:1
> 3	      41,42,43,49,50,51,52,53,54,55	9:1
> 4	      56,57,58,59,16,32,48          3:1

well, it just means that if you get a failure, you'll run in degraded
mode for a while, which is a window of vulnerability.

> ...or...:
> 
> global spares: 0,16
> 
> Raidset	Disks used	                  Data:parity ratio
> 0	      1,2,3,4,5,6,7,8,9,10          9:1
> 1	      11,17,18,19,20,21,22,23,24,25	9:1
> 2	      26,27,33,34,35,36,37,38,39,40	9:1
> 3	      41,42,43,49,50,51,52,53,54,55	9:1
> 4	      56,57,58,59,32,48             3:1

2 spares seems OK to me, assuming a reasonable failure rate (>2 years
aggregate mtbf)

> Does anyone have any comments on:
> 
> 1) The sanity of these 10 disk RAID 5's?

if you're not worried about write performance, then sure.
I had an 18x raid5 for a while, but decided it was too hostile
to writes (iirc, a whole-stripe write was > 1MB)

> 2) The degree of loss of reliability incurred by moving 3 disks from
> global spare to data?

spares do not increase reliability, they reduce the window of 
vulnerability when you do have a "partial lack of reliability"...

> 3) The degree of loss of reliability incurred by moving 2 disks from
> global spare to data?

MTBF/ndisks hasn't changed here, at least for a particular raidset.
the chance of simultaneous failures of 2+ disks in multiple raidsets
seems pretty small...

> They don't feel that the storage has to be blazing fast, and 100% uptime
> isn't paramount, however they very much do not want to lose their data.

but their "very much" doesn't extend to two-site mirroring, eh?
there are unfortunate phenomena that can lead to bad behavior in 
a server like this, even when you do the due-dilligence (mtbf calcs,
spares, etc).  for instance, in the event of r5 failure, the spare
will trigger a rebuild, which can stress the surviving disks enough 
to cause further failures.  oops!  I guess that's the main reason 
I like raid6 better.

> The filesystem will not be backed up - we simply don't have anything large
> enough to back it up -to-, so if the some part of the storage solution
> goes kerflooey, we're totally...  er...  out of luck, and they'll probably
> be looking at me (the primary sysadmin on the storage configuration),
> wondering why their data is gone.

this is a sticky subject, to be sure.  I tell people not to think about 
backups, or if they do, to think more in terms of mirroring.  perhaps that 
reflects scars I bear from dealing with finicky/flakey/frustrating tape
systems.  one good thing for you is that you say the files are fairly small,
so you *could* spew them onto something like DVD's.  I'd treat that as 
an archive, not a backup, and not abandon normal raid5-6 practices.

Re: number of global spares?

[Neil Brown]

On Friday August 26, strombrg@dcs.nac.uci.edu wrote:
> 
> I've been working on a RAID setup with dual RAID controllers and
> three expansion boxes - 48 disks in all, including data, parity and
> global spares.

If there are 48 drives, why do your drive-numbers go up to 59?
Confusing but not important.

Presumably these are 360G drives (or there abouts) and you are hoping
to use about 42 for data and the remaining 6 for redundancy.

I feel this a bit tight but could be workable.

If you were using Linux-soft-raid, I would probably suggest 3 16-drive raid6
arrays, possibly making 1 a 15 drive raid6 so there is one global spare.
However I gather you are using hardware RAID - do the controllers
support RAID6 ??
> 
> Please be sure to use a fixed-pitch font when viewing the tables found
> below.  BTW, if people weren't so terrified of HTML, I could just make a
> nice HTML table for easy reading without silly font requirements...

You mean some mail readers use variable-width-fonts to display
text/plain?  How broken!

> 
> 
> Does anyone have any comments on:
> 
> 1) The sanity of these 10 disk RAID 5's?

It depends on the drives.
If you are using you-only-get-what-you-pay-for-IDE-drives, then I
would say it is insane.
If you are using you-pay-for-the-quality SCSI drives, then you should
be fairly safe.

> 
> 2) The degree of loss of reliability incurred by moving 3 disks from
> global spare to data?

That depends a bit on your warranty arrangements on the drives.  If
it's next-day-replacement (Really, truly) then it is probably OK.  If
it is 'send us the bad drive and we'll see what we can do', then I
would suggest thinking again.

> 
> 3) The degree of loss of reliability incurred by moving 2 disks from
> global spare to data?
> 

Should be safe enough.

However I don't understand why you have 4 9+1 arrays, and 1 X+1 for
varying X.  Maybe 9+1 is the largest the controller will give you, so
you have to go to 5 arrays.
In that case, 4 8+1 arrays, 1 9+1 array, and 2 global spares would
seem a more sensible arrangement.


NeilBrown

RE: number of global spares?

[Guy]

> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
> owner@vger.kernel.org] On Behalf Of Dan Stromberg
> Sent: Friday, August 26, 2005 3:01 PM
> To: Linux RAID
> Cc: strombrg@dcs.nac.uci.edu
> Subject: number of global spares?
> 
> 
> I've been working on a RAID setup with dual RAID controllers and
> three expansion boxes - 48 disks in all, including data, parity and
> global spares.
> 
> We originally purchased the equipment expecting to get 16 terabytes of
> usable space.
> 
> Now that it's "all set up", we're really seeing more like 14 or 15
> terabytes, depending on how you do the calculation.
> 
> Please be sure to use a fixed-pitch font when viewing the tables found
> below.  BTW, if people weren't so terrified of HTML, I could just make a
> nice HTML table for easy reading without silly font requirements...
> 
> Anyway, what we have right now is:
> 
> global spares: 0,16,32,48
> 
> Raidset	Disks used	                  Data:parity ratio
> 0	      1,2,3,4,5,6,7,8,9,10          9:1
> 1	      11,17,18,19,20,21,22,23,24,25	9:1
> 2	      26,27,33,34,35,36,37,38,39,40	9:1
> 3	      41,42,43,49,50,51,52,53,54,55	9:1
> 4	      56,57,58,59                   3:1
> 
> 
> And the vendor is suggesting that we move to something like:
> 
> global spares: 0
> 
> Raidset	Disks used	                  Data:parity ratio
> 0	      1,2,3,4,5,6,7,8,9,10          9:1
> 1	      11,17,18,19,20,21,22,23,24,25	9:1
> 2	      26,27,33,34,35,36,37,38,39,40	9:1
> 3	      41,42,43,49,50,51,52,53,54,55	9:1
> 4	      56,57,58,59,16,32,48          3:1
> 
> ...or...:
> 
> global spares: 0,16
> 
> Raidset	Disks used	                  Data:parity ratio
> 0	      1,2,3,4,5,6,7,8,9,10          9:1
> 1	      11,17,18,19,20,21,22,23,24,25	9:1
> 2	      26,27,33,34,35,36,37,38,39,40	9:1
> 3	      41,42,43,49,50,51,52,53,54,55	9:1
> 4	      56,57,58,59,32,48             3:1
> 
> 
> Does anyone have any comments on:
> 
> 1) The sanity of these 10 disk RAID 5's?
> 
> 2) The degree of loss of reliability incurred by moving 3 disks from
> global spare to data?
> 
> 3) The degree of loss of reliability incurred by moving 2 disks from
> global spare to data?
> 
> 
> To answer these questions, you probably need to know how the storage is to
> be used.  This single, large filesystem will be used by a variety of
> researchers and students from around The University of California, Irvine,
> but was purchased primarily by the Earth System Science part of the
> Physical Sciences department, which in turn will primarily be storing many
> approximately 100 megabyte files which comprise time series related to
> climatology simulations.
> 
> They don't feel that the storage has to be blazing fast, and 100% uptime
> isn't paramount, however they very much do not want to lose their data.
> 
> The filesystem will not be backed up - we simply don't have anything large
> enough to back it up -to-, so if the some part of the storage solution
> goes kerflooey, we're totally...  er...  out of luck, and they'll probably
> be looking at me (the primary sysadmin on the storage configuration),
> wondering why their data is gone.

RAID5, 6 or 1 is not data backup!  It is hardware redundancy!!
Data loss or corruption can still occur with a RAID solution.  RAID won't
help if someone fat fingers a "rm" command.
Corruption of the filesystem can also cause major data loss, without a
failed disk.

If the data was lost, what would it cost to re-create it?
Enough to buy a backup system?

Just my 3 cents!

Guy

> 
> Thanks!
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: number of global spares?

[Daniel Pittman]

"Guy" <bugzilla@watkins-home.com> writes:

[...]

>> I've been working on a RAID setup with dual RAID controllers and
>> three expansion boxes - 48 disks in all, including data, parity and
>> global spares.

[...]

>> They don't feel that the storage has to be blazing fast, and 100% uptime
>> isn't paramount, however they very much do not want to lose their data.
>> 
>> The filesystem will not be backed up - we simply don't have anything large
>> enough to back it up -to-, so if the some part of the storage solution
>> goes kerflooey, we're totally...  er...  out of luck, and they'll probably
>> be looking at me (the primary sysadmin on the storage configuration),
>> wondering why their data is gone.
>
> RAID5, 6 or 1 is not data backup!  It is hardware redundancy!!
> Data loss or corruption can still occur with a RAID solution.  RAID won't
> help if someone fat fingers a "rm" command.
> Corruption of the filesystem can also cause major data loss, without a
> failed disk.
>
> If the data was lost, what would it cost to re-create it?
> Enough to buy a backup system?

I absolutely agree with this.  When - and it is when, not if - the
content of this filesystem goes away, you will be rightly blamed for it.

Invest the few thousand dollars in a good high capacity tape drive and
pay someone to change the tapes.  This will be worth it when the system
finally does fail in some nasty, unpredictable way!

        Daniel

Re: number of global spares?

[Dan Stromberg]

On Sat, 2005-08-27 at 15:00 +1000, Daniel Pittman wrote:
> "Guy" <bugzilla@watkins-home.com> writes:
> 
> [...]
> 
> >> I've been working on a RAID setup with dual RAID controllers and
> >> three expansion boxes - 48 disks in all, including data, parity and
> >> global spares.
> 
> [...]
> 
> >> They don't feel that the storage has to be blazing fast, and 100% uptime
> >> isn't paramount, however they very much do not want to lose their data.
> >> 
> >> The filesystem will not be backed up - we simply don't have anything large
> >> enough to back it up -to-, so if the some part of the storage solution
> >> goes kerflooey, we're totally...  er...  out of luck, and they'll probably
> >> be looking at me (the primary sysadmin on the storage configuration),
> >> wondering why their data is gone.
> >
> > RAID5, 6 or 1 is not data backup!  It is hardware redundancy!!
> > Data loss or corruption can still occur with a RAID solution.  RAID won't
> > help if someone fat fingers a "rm" command.
> > Corruption of the filesystem can also cause major data loss, without a
> > failed disk.
> >
> > If the data was lost, what would it cost to re-create it?
> > Enough to buy a backup system?
> 
> I absolutely agree with this.  When - and it is when, not if - the
> content of this filesystem goes away, you will be rightly blamed for it.
> 
> Invest the few thousand dollars in a good high capacity tape drive and
> pay someone to change the tapes.  This will be worth it when the system
> finally does fail in some nasty, unpredictable way!

I was on paternity leave when the solution was selected, but the guy
with the grant money has been disinterested in backups from the
beginning.

The policy is going to be "your homedir will be backed up.  Your files
under /data will not, unless you back them up yourself."

My job is to work within that restriction, and possibly advise for
backups, but nothing more.  The purchasing decision is not mine.

RE: number of global spares?

[Dan Stromberg]

On Fri, 2005-08-26 at 19:21 -0400, Guy wrote:
> 
> > -----Original Message-----
> > From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
> > owner@vger.kernel.org] On Behalf Of Dan Stromberg
> > Sent: Friday, August 26, 2005 3:01 PM
> > To: Linux RAID
> > Cc: strombrg@dcs.nac.uci.edu
> > Subject: number of global spares?
> > 
> > 
> > I've been working on a RAID setup with dual RAID controllers and
> > three expansion boxes - 48 disks in all, including data, parity and
> > global spares.
> > 
> > We originally purchased the equipment expecting to get 16 terabytes of
> > usable space.
> > 
> > Now that it's "all set up", we're really seeing more like 14 or 15
> > terabytes, depending on how you do the calculation.
> > 
> > Please be sure to use a fixed-pitch font when viewing the tables found
> > below.  BTW, if people weren't so terrified of HTML, I could just make a
> > nice HTML table for easy reading without silly font requirements...
> > 
> > Anyway, what we have right now is:
> > 
> > global spares: 0,16,32,48
> > 
> > Raidset	Disks used	                  Data:parity ratio
> > 0	      1,2,3,4,5,6,7,8,9,10          9:1
> > 1	      11,17,18,19,20,21,22,23,24,25	9:1
> > 2	      26,27,33,34,35,36,37,38,39,40	9:1
> > 3	      41,42,43,49,50,51,52,53,54,55	9:1
> > 4	      56,57,58,59                   3:1
> > 
> > 
> > And the vendor is suggesting that we move to something like:
> > 
> > global spares: 0
> > 
> > Raidset	Disks used	                  Data:parity ratio
> > 0	      1,2,3,4,5,6,7,8,9,10          9:1
> > 1	      11,17,18,19,20,21,22,23,24,25	9:1
> > 2	      26,27,33,34,35,36,37,38,39,40	9:1
> > 3	      41,42,43,49,50,51,52,53,54,55	9:1
> > 4	      56,57,58,59,16,32,48          3:1
> > 
> > ...or...:
> > 
> > global spares: 0,16
> > 
> > Raidset	Disks used	                  Data:parity ratio
> > 0	      1,2,3,4,5,6,7,8,9,10          9:1
> > 1	      11,17,18,19,20,21,22,23,24,25	9:1
> > 2	      26,27,33,34,35,36,37,38,39,40	9:1
> > 3	      41,42,43,49,50,51,52,53,54,55	9:1
> > 4	      56,57,58,59,32,48             3:1
> > 
> > 
> > Does anyone have any comments on:
> > 
> > 1) The sanity of these 10 disk RAID 5's?
> > 
> > 2) The degree of loss of reliability incurred by moving 3 disks from
> > global spare to data?
> > 
> > 3) The degree of loss of reliability incurred by moving 2 disks from
> > global spare to data?
> > 
> > 
> > To answer these questions, you probably need to know how the storage is to
> > be used.  This single, large filesystem will be used by a variety of
> > researchers and students from around The University of California, Irvine,
> > but was purchased primarily by the Earth System Science part of the
> > Physical Sciences department, which in turn will primarily be storing many
> > approximately 100 megabyte files which comprise time series related to
> > climatology simulations.
> > 
> > They don't feel that the storage has to be blazing fast, and 100% uptime
> > isn't paramount, however they very much do not want to lose their data.
> > 
> > The filesystem will not be backed up - we simply don't have anything large
> > enough to back it up -to-, so if the some part of the storage solution
> > goes kerflooey, we're totally...  er...  out of luck, and they'll probably
> > be looking at me (the primary sysadmin on the storage configuration),
> > wondering why their data is gone.
> 
> RAID5, 6 or 1 is not data backup!  It is hardware redundancy!!
> Data loss or corruption can still occur with a RAID solution.  RAID won't
> help if someone fat fingers a "rm" command.
> Corruption of the filesystem can also cause major data loss, without a
> failed disk.

Yeah.  I know.

> If the data was lost, what would it cost to re-create it?
> Enough to buy a backup system?

The data generated by the primary purpose for the hardware, will
primarily be time series from models, that are hoped to match empirical
data obtained from other sources.

IE, it would take months or years to regenerate, but it could be
regenerated.

And people who have info under /data (this RAID rig) that they cannot
lose, are advised to back it up somewhere themselves.

Re: number of global spares?

[Dan Stromberg]

On Sat, 2005-08-27 at 08:56 +1000, Neil Brown wrote:
> On Friday August 26, strombrg@dcs.nac.uci.edu wrote:
> > 
> > I've been working on a RAID setup with dual RAID controllers and
> > three expansion boxes - 48 disks in all, including data, parity and
> > global spares.
> 
> If there are 48 drives, why do your drive-numbers go up to 59?
> Confusing but not important.

Agreed.  The hardware likes to do that.

> Presumably these are 360G drives (or there abouts) and you are hoping
> to use about 42 for data and the remaining 6 for redundancy.

Around 380G.

> I feel this a bit tight but could be workable.
> 
> If you were using Linux-soft-raid, I would probably suggest 3 16-drive raid6
> arrays, possibly making 1 a 15 drive raid6 so there is one global spare.
> However I gather you are using hardware RAID - do the controllers
> support RAID6 ??

I'd feel better about RAID 6, but the hardware doesn't support it.

> > Please be sure to use a fixed-pitch font when viewing the tables found
> > below.  BTW, if people weren't so terrified of HTML, I could just make a
> > nice HTML table for easy reading without silly font requirements...
> 
> You mean some mail readers use variable-width-fonts to display
> text/plain?  How broken!

Heh.

> > 
> > 
> > Does anyone have any comments on:
> > 
> > 1) The sanity of these 10 disk RAID 5's?
> 
> It depends on the drives.
> If you are using you-only-get-what-you-pay-for-IDE-drives, then I
> would say it is insane.
> If you are using you-pay-for-the-quality SCSI drives, then you should
> be fairly safe.

They're SATA's.

> > 
> > 2) The degree of loss of reliability incurred by moving 3 disks from
> > global spare to data?
> 
> That depends a bit on your warranty arrangements on the drives.  If
> it's next-day-replacement (Really, truly) then it is probably OK.  If
> it is 'send us the bad drive and we'll see what we can do', then I
> would suggest thinking again.

This'll be the next thing I ask the vendor.

> > 
> > 3) The degree of loss of reliability incurred by moving 2 disks from
> > global spare to data?
> > 
> 
> Should be safe enough.
> 
> However I don't understand why you have 4 9+1 arrays, and 1 X+1 for
> varying X.  Maybe 9+1 is the largest the controller will give you, so
> you have to go to 5 arrays.
> In that case, 4 8+1 arrays, 1 9+1 array, and 2 global spares would
> seem a more sensible arrangement.

I'm going to pitch this (or maybe 3 8+1's) to the owner of the equipment
and the vendor.

It was the vendor that suggested the 9+1's.

Re: number of global spares?

[Dan Stromberg]

On Fri, 2005-08-26 at 18:21 -0400, Mark Hahn wrote:
> > I've been working on a RAID setup with dual RAID controllers and
> > three expansion boxes - 48 disks in all, including data, parity and
> > global spares.
> 
> the first question you should ask is whether you're actually
> winning by using HW raid.  yes, you already paid for it, but
> SW raid offers some noticably better flexibility.

Nod.  As well as better performance in many cases.

> > Please be sure to use a fixed-pitch font when viewing the tables found
> > below.  BTW, if people weren't so terrified of HTML, I could just make a
> > nice HTML table for easy reading without silly font requirements...
> 
> it's not a matter of terror - many people still prefer ascii email.
> (naturally, we also use fixed-pitch fonts for this.)

Er, IMO it makes *ix folk look like hide bound traditionalists, which is
unfortunate, because *ix is a more capable vehicle for OS evolution.
HTML clearly has a great deal more expressive power - why fight it?

Even mutt can do a decent job of HTML rendering...

> > global spares: 0,16,32,48
> > 
> > Raidset	Disks used	                  Data:parity ratio
> > 0	      1,2,3,4,5,6,7,8,9,10          9:1
> > 1	      11,17,18,19,20,21,22,23,24,25	9:1
> > 2	      26,27,33,34,35,36,37,38,39,40	9:1
> > 3	      41,42,43,49,50,51,52,53,54,55	9:1
> > 4	      56,57,58,59                   3:1
> 
> why the magic numbers?  (5 raidsets, 9:1, etc)
> you have 48 disks and "dual RAID controllers" (one channel each?) 
> in 3 boxes, but what are your actual constraints?

The vendor suggested 9+1's and 4 global spares.  Yes, one channel for
each RAID controller.  I'm going to pitch something like 4 8+1's though.

> also, if you have dual controllers, can you truely have global spares?
> that is, a controller can use a spare disk that it's not connected to?

So I hear.  Right now, we have one global spare per shelf, but the
vendor is advising we decrease the number of global spares.

> 9:1 is nothing to be scared of, though it means that to do a full-stripe
> write, you'll need quite large blocks.  I'd be tempted to use raid6
> rather than 5+spares, though.

I'd feel better about RAID 6, but...

> > And the vendor is suggesting that we move to something like:
> > 
> > global spares: 0
> > 
> > Raidset	Disks used	                  Data:parity ratio
> > 0	      1,2,3,4,5,6,7,8,9,10          9:1
> > 1	      11,17,18,19,20,21,22,23,24,25	9:1
> > 2	      26,27,33,34,35,36,37,38,39,40	9:1
> > 3	      41,42,43,49,50,51,52,53,54,55	9:1
> > 4	      56,57,58,59,16,32,48          3:1
> 
> well, it just means that if you get a failure, you'll run in degraded
> mode for a while, which is a window of vulnerability.
> 
> > ...or...:
> > 
> > global spares: 0,16
> > 
> > Raidset	Disks used	                  Data:parity ratio
> > 0	      1,2,3,4,5,6,7,8,9,10          9:1
> > 1	      11,17,18,19,20,21,22,23,24,25	9:1
> > 2	      26,27,33,34,35,36,37,38,39,40	9:1
> > 3	      41,42,43,49,50,51,52,53,54,55	9:1
> > 4	      56,57,58,59,32,48             3:1
> 
> 2 spares seems OK to me, assuming a reasonable failure rate (>2 years
> aggregate mtbf)
> 
> > Does anyone have any comments on:
> > 
> > 1) The sanity of these 10 disk RAID 5's?
> 
> if you're not worried about write performance, then sure.
> I had an 18x raid5 for a while, but decided it was too hostile
> to writes (iirc, a whole-stripe write was > 1MB)
> 
> > 2) The degree of loss of reliability incurred by moving 3 disks from
> > global spare to data?
> 
> spares do not increase reliability, they reduce the window of 
> vulnerability when you do have a "partial lack of reliability"...
> 
> > 3) The degree of loss of reliability incurred by moving 2 disks from
> > global spare to data?
> 
> MTBF/ndisks hasn't changed here, at least for a particular raidset.
> the chance of simultaneous failures of 2+ disks in multiple raidsets
> seems pretty small...
> 
> > They don't feel that the storage has to be blazing fast, and 100% uptime
> > isn't paramount, however they very much do not want to lose their data.
> 
> but their "very much" doesn't extend to two-site mirroring, eh?

Correct, at least not mirroring provided by their admin.  :)  They -are-
advising that individual users back up what needs to be backed up
though.

> there are unfortunate phenomena that can lead to bad behavior in 
> a server like this, even when you do the due-dilligence (mtbf calcs,
> spares, etc).  for instance, in the event of r5 failure, the spare
> will trigger a rebuild, which can stress the surviving disks enough 
> to cause further failures.  oops!  I guess that's the main reason 
> I like raid6 better.

Nod!  Especially if the RAID solution doesn't know to scan
unused/infrequently used blocks for problems periodically.

> > The filesystem will not be backed up - we simply don't have anything large
> > enough to back it up -to-, so if the some part of the storage solution
> > goes kerflooey, we're totally...  er...  out of luck, and they'll probably
> > be looking at me (the primary sysadmin on the storage configuration),
> > wondering why their data is gone.
> 
> this is a sticky subject, to be sure.  I tell people not to think about 
> backups, or if they do, to think more in terms of mirroring.  perhaps that 
> reflects scars I bear from dealing with finicky/flakey/frustrating tape
> systems.  one good thing for you is that you say the files are fairly small,
> so you *could* spew them onto something like DVD's.  I'd treat that as 
> an archive, not a backup, and not abandon normal raid5-6 practices.

Heh.  Saw an amusing ad with John Clease in it about the frustrations of
tape backup relative to disk to disk backup.

Re: number of global spares?

[Mark Hahn]

> > > Please be sure to use a fixed-pitch font when viewing the tables found
> > > below.  BTW, if people weren't so terrified of HTML, I could just make a
> > > nice HTML table for easy reading without silly font requirements...
> > 
> > it's not a matter of terror - many people still prefer ascii email.
> > (naturally, we also use fixed-pitch fonts for this.)
> 
> Er, IMO it makes *ix folk look like hide bound traditionalists, which is
> unfortunate, because *ix is a more capable vehicle for OS evolution.
> HTML clearly has a great deal more expressive power - why fight it?

people have different value functions - I don't find the expressiveness
of html to be worthwhile enough, for instance.  language has, after all,
proven itself pretty well over the millenia.

> So I hear.  Right now, we have one global spare per shelf, but the
> vendor is advising we decrease the number of global spares.

find out what your controller can do.  if it can actually use a spare
that it's not connected to (doubtful), then you could certainly get 
away with a single hot-spare.  you should also look up the MTBF specs 
of the disks, or make a guess (are they in a well-cooled environment?)
about how often you expect failures.  it may be that having a single
cold spare in a cabinet is good enough.  after all, when a disk goes,
you'll be in degraded mode for at least as long as it takes to rebuild
the raid, plus any latency for making the spare available...

can you go *past* 9+1?  if you're already assuming read-mostly,
or else slow writes (partial stripe RMW cycles), then going 
to a pair of 22+1+spare sounds perfectly plausible to me.
yes, it's not typically done, but not for inherent reasons...

Re: number of global spares?

[Bill Davidsen]

Neil Brown wrote:

>On Friday August 26, strombrg@dcs.nac.uci.edu wrote:
>  
>
>>I've been working on a RAID setup with dual RAID controllers and
>>three expansion boxes - 48 disks in all, including data, parity and
>>global spares.
>>    
>>
>
>If there are 48 drives, why do your drive-numbers go up to 59?
>Confusing but not important.
>
>Presumably these are 360G drives (or there abouts) and you are hoping
>to use about 42 for data and the remaining 6 for redundancy.
>
>I feel this a bit tight but could be workable.
>
>If you were using Linux-soft-raid, I would probably suggest 3 16-drive raid6
>arrays, possibly making 1 a 15 drive raid6 so there is one global spare.
>However I gather you are using hardware RAID - do the controllers
>support RAID6 ??
>  
>
>>Please be sure to use a fixed-pitch font when viewing the tables found
>>below.  BTW, if people weren't so terrified of HTML, I could just make a
>>nice HTML table for easy reading without silly font requirements...
>>    
>>
>
>You mean some mail readers use variable-width-fonts to display
>text/plain?  How broken!
>  
>
Some clients actually do what the user asks in 'preferences' which may 
not be fixed fonts...

More to the point, is there a nice widely accepted calculation for 
reliability vs. MTBF? I started to develop one but stopped because it 
isn't really much use without the S.D. of the MTBF. If drive MTBF is 
equal, the better drive (safer) is one with more S.D. rather than a 
clustering of failures which somewhat increases the chance of a failure 
during rebuild.

The better the quality control the higher the risk of failure clusters, 
unfortunately.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: number of global spares?

[Bill Davidsen]

Dan Stromberg wrote:

>On Sat, 2005-08-27 at 08:56 +1000, Neil Brown wrote:
>  
>
>>On Friday August 26, strombrg@dcs.nac.uci.edu wrote:
>>    
>>
>>>I've been working on a RAID setup with dual RAID controllers and
>>>three expansion boxes - 48 disks in all, including data, parity and
>>>global spares.
>>>      
>>>
>>If there are 48 drives, why do your drive-numbers go up to 59?
>>Confusing but not important.
>>    
>>
>
>Agreed.  The hardware likes to do that.
>
>  
>
>>Presumably these are 360G drives (or there abouts) and you are hoping
>>to use about 42 for data and the remaining 6 for redundancy.
>>    
>>
>
>Around 380G.
>
>  
>
>>I feel this a bit tight but could be workable.
>>
>>If you were using Linux-soft-raid, I would probably suggest 3 16-drive raid6
>>arrays, possibly making 1 a 15 drive raid6 so there is one global spare.
>>However I gather you are using hardware RAID - do the controllers
>>support RAID6 ??
>>    
>>
>
>I'd feel better about RAID 6, but the hardware doesn't support it.
>
>  
>
>>>Please be sure to use a fixed-pitch font when viewing the tables found
>>>below.  BTW, if people weren't so terrified of HTML, I could just make a
>>>nice HTML table for easy reading without silly font requirements...
>>>      
>>>
>>You mean some mail readers use variable-width-fonts to display
>>text/plain?  How broken!
>>    
>>
>
>Heh.
>
>  
>
>>>Does anyone have any comments on:
>>>
>>>1) The sanity of these 10 disk RAID 5's?
>>>      
>>>
>>It depends on the drives.
>>If you are using you-only-get-what-you-pay-for-IDE-drives, then I
>>would say it is insane.
>>If you are using you-pay-for-the-quality SCSI drives, then you should
>>be fairly safe.
>>    
>>
>
>They're SATA's.
>
>  
>
>>>2) The degree of loss of reliability incurred by moving 3 disks from
>>>global spare to data?
>>>      
>>>
>>That depends a bit on your warranty arrangements on the drives.  If
>>it's next-day-replacement (Really, truly) then it is probably OK.  If
>>it is 'send us the bad drive and we'll see what we can do', then I
>>would suggest thinking again.
>>    
>>
>
>This'll be the next thing I ask the vendor.
>
As someone who suports multi-TB machines, I can suggest that you really 
want to have spares on-site rather than depend on the vendor. Not that 
the vendor isn't trying, but shippers go on strike, snow, mud and rain 
can close roads, the vendor's website can be hacked, etc. Oh, and 
earthquakes.

Having drives on-site lets you sleep better, and can be explained to 
management more easily than RAID-5 vs. RAID-6.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: number of global spares?

[Bill Davidsen]

Dan Stromberg wrote:

>On Sat, 2005-08-27 at 15:00 +1000, Daniel Pittman wrote:
>  
>
>>"Guy" <bugzilla@watkins-home.com> writes:
>>
>>[...]
>>
>>    
>>
>>>>I've been working on a RAID setup with dual RAID controllers and
>>>>three expansion boxes - 48 disks in all, including data, parity and
>>>>global spares.
>>>>        
>>>>
>>[...]
>>
>>    
>>
>>>>They don't feel that the storage has to be blazing fast, and 100% uptime
>>>>isn't paramount, however they very much do not want to lose their data.
>>>>
>>>>The filesystem will not be backed up - we simply don't have anything large
>>>>enough to back it up -to-, so if the some part of the storage solution
>>>>goes kerflooey, we're totally...  er...  out of luck, and they'll probably
>>>>be looking at me (the primary sysadmin on the storage configuration),
>>>>wondering why their data is gone.
>>>>        
>>>>
>>>RAID5, 6 or 1 is not data backup!  It is hardware redundancy!!
>>>Data loss or corruption can still occur with a RAID solution.  RAID won't
>>>help if someone fat fingers a "rm" command.
>>>Corruption of the filesystem can also cause major data loss, without a
>>>failed disk.
>>>
>>>If the data was lost, what would it cost to re-create it?
>>>Enough to buy a backup system?
>>>      
>>>
>>I absolutely agree with this.  When - and it is when, not if - the
>>content of this filesystem goes away, you will be rightly blamed for it.
>>
>>Invest the few thousand dollars in a good high capacity tape drive and
>>pay someone to change the tapes.  This will be worth it when the system
>>finally does fail in some nasty, unpredictable way!
>>    
>>
>
>I was on paternity leave when the solution was selected, but the guy
>with the grant money has been disinterested in backups from the
>beginning.
>
>The policy is going to be "your homedir will be backed up.  Your files
>under /data will not, unless you back them up yourself."
>
>My job is to work within that restriction, and possibly advise for
>backups, but nothing more.  The purchasing decision is not mine.
>
Clearly you have some input into it. I would suggest that you at least 
go on record (paper trail CYA) on the need for backup. If you can't get 
incrementals on whatever does the /home directories, at least you could 
suggest a DVD burner and regular backups. That is a tiny bump on the 
hardware budget, and small storage requirement. You could backup as many 
datasets as will fit on one DVD every day, oldest unsaved first. Then 
when it fails you will be the hero ;-)

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: number of global spares?

[Dan Stromberg]

On Wed, 2005-09-28 at 15:47 -0400, Bill Davidsen wrote:
> Dan Stromberg wrote:
> 
> >On Sat, 2005-08-27 at 15:00 +1000, Daniel Pittman wrote:
> >  
> >
> >>"Guy" <bugzilla@watkins-home.com> writes:
> >>
> >>[...]
> >>
> >>    
> >>
> >>>>I've been working on a RAID setup with dual RAID controllers and
> >>>>three expansion boxes - 48 disks in all, including data, parity and
> >>>>global spares.
> >>>>        
> >>>>
> >>[...]
> >>
> >>    
> >>
> >>>>They don't feel that the storage has to be blazing fast, and 100% uptime
> >>>>isn't paramount, however they very much do not want to lose their data.
> >>>>
> >>>>The filesystem will not be backed up - we simply don't have anything large
> >>>>enough to back it up -to-, so if the some part of the storage solution
> >>>>goes kerflooey, we're totally...  er...  out of luck, and they'll probably
> >>>>be looking at me (the primary sysadmin on the storage configuration),
> >>>>wondering why their data is gone.
> >>>>        
> >>>>
> >>>RAID5, 6 or 1 is not data backup!  It is hardware redundancy!!
> >>>Data loss or corruption can still occur with a RAID solution.  RAID won't
> >>>help if someone fat fingers a "rm" command.
> >>>Corruption of the filesystem can also cause major data loss, without a
> >>>failed disk.
> >>>
> >>>If the data was lost, what would it cost to re-create it?
> >>>Enough to buy a backup system?
> >>>      
> >>>
> >>I absolutely agree with this.  When - and it is when, not if - the
> >>content of this filesystem goes away, you will be rightly blamed for it.
> >>
> >>Invest the few thousand dollars in a good high capacity tape drive and
> >>pay someone to change the tapes.  This will be worth it when the system
> >>finally does fail in some nasty, unpredictable way!
> >>    
> >>
> >
> >I was on paternity leave when the solution was selected, but the guy
> >with the grant money has been disinterested in backups from the
> >beginning.
> >
> >The policy is going to be "your homedir will be backed up.  Your files
> >under /data will not, unless you back them up yourself."
> >
> >My job is to work within that restriction, and possibly advise for
> >backups, but nothing more.  The purchasing decision is not mine.
> >
> Clearly you have some input into it. I would suggest that you at least 
> go on record (paper trail CYA) on the need for backup. If you can't get 
> incrementals on whatever does the /home directories, at least you could 
> suggest a DVD burner and regular backups. That is a tiny bump on the 
> hardware budget, and small storage requirement. You could backup as many 
> datasets as will fit on one DVD every day, oldest unsaved first. Then 
> when it fails you will be the hero ;-)

We actually have decent backups of /home.

It's /data that won't be backed up, unless the users make their own
arrangements.

I kind of like the idea of doing incrementals to DVD or something,
augmented by md5 or sha-* signatures, but I don't think it's going to
happen.

I've already discussed this with my management, and with the PI.

Thanks though.  Your advice is helpful.