RE: First RAID Setup

RE: First RAID Setup

[Callahan, Tom]

You "should" have a designated spare for RAID-5.

Not sure why you have 3 disks for each RAID1, RAID1 is mirror, and unless
the third drive is a spare, it is not needed.

Thanks,
Tom Callahan

-----Original Message-----
From: linux-raid-owner@vger.kernel.org
[mailto:linux-raid-owner@vger.kernel.org]On Behalf Of Andargor The Wise
Sent: Thursday, December 15, 2005 2:10 PM
To: linux-raid@vger.kernel.org
Subject: First RAID Setup


I admit it. I'm a RAID virgin.

However, after a disastrous failure of the sole drive
I wasn't backing up, I decided to go RAID-5 under
Slack 10.2 (first time ever with RAID-5).

The config:

Asus P5GL-MX (ICH6) mobo w/1 GB RAM, 4 x SATA ports
P4 3.0G/1M
3 x WD2000JS 200.0 GB SATA drives

First, a question: the BIOS on this machine seems to
list the SATA ports as "third/fourth IDE
master/slave". Further, the documentation seems to say
that SATA 1/2 are "master" and SATA 3/4 are "slave"
(black and red connectors, respectively).

My understanding is that SATA drives are each on
separate buses. Is this because the BIOS offers a
P-ATA emulation mode for SATA and it makes it "easier"
to understand for novices to show them that way?

I ask because people have said that it is not a good
idea to have both IDE masters and slaves on the same
bus as part of a RAID-5 array. I know SATA is
different, but will using three of the SATA ports on
this mobo be OK?

Second, after reading the excellent advice in this
list, I decided that booting from RAID-5 might not be
a good idea. So this is what I've been thinking:

Each disk partitioned alike:
	1	30MB 
	2	8GB (to allow for memory upgrades later)
	5	rest_of_disk

mds:
	md0	raid1 sda1 sdb1 sdc1
	md1	raid1 sda2 sdb2 sdc2
	md2	raid5 sda5 sdb5 sdc5

	md0	/boot
	md1	swap
	md2	/

Does this look OK? What should the stripe and chunk
sizes be, considering I'll be going with reiserfs?
Typical usage: development machine, some DB apps with
medium load, read-only mostly, not many writes. Very
few large files (such as multimedia).

Or should I set up separate RAID-5's for /usr and /var
as well?

Lastly, can I install directly to this configuration,
or should I install on a separate disk and move things
into the array?

Andargor


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: First RAID Setup

[Andargor The Wise]

The RAID1 partitions are to make sure:

1) The machine is able to boot even if a disk is lost
(/boot).
2) The machine isn't brought down if a disk is lost
(swap)

I thought about a spare drive, but I don't need high
availability. I'm satisfied with being able to recover
my data.

Andargor


--- "Callahan, Tom" <CallahanT@tessco.com> wrote:

> You "should" have a designated spare for RAID-5.
> 
> Not sure why you have 3 disks for each RAID1, RAID1
> is mirror, and unless
> the third drive is a spare, it is not needed.
> 
> Thanks,
> Tom Callahan
> 
> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org
> [mailto:linux-raid-owner@vger.kernel.org]On Behalf
> Of Andargor The Wise
> Sent: Thursday, December 15, 2005 2:10 PM
> To: linux-raid@vger.kernel.org
> Subject: First RAID Setup
> 
> 
> I admit it. I'm a RAID virgin.
> 
> However, after a disastrous failure of the sole
> drive
> I wasn't backing up, I decided to go RAID-5 under
> Slack 10.2 (first time ever with RAID-5).
> 
> The config:
> 
> Asus P5GL-MX (ICH6) mobo w/1 GB RAM, 4 x SATA ports
> P4 3.0G/1M
> 3 x WD2000JS 200.0 GB SATA drives
> 
> First, a question: the BIOS on this machine seems to
> list the SATA ports as "third/fourth IDE
> master/slave". Further, the documentation seems to
> say
> that SATA 1/2 are "master" and SATA 3/4 are "slave"
> (black and red connectors, respectively).
> 
> My understanding is that SATA drives are each on
> separate buses. Is this because the BIOS offers a
> P-ATA emulation mode for SATA and it makes it
> "easier"
> to understand for novices to show them that way?
> 
> I ask because people have said that it is not a good
> idea to have both IDE masters and slaves on the same
> bus as part of a RAID-5 array. I know SATA is
> different, but will using three of the SATA ports on
> this mobo be OK?
> 
> Second, after reading the excellent advice in this
> list, I decided that booting from RAID-5 might not
> be
> a good idea. So this is what I've been thinking:
> 
> Each disk partitioned alike:
> 	1	30MB 
> 	2	8GB (to allow for memory upgrades later)
> 	5	rest_of_disk
> 
> mds:
> 	md0	raid1 sda1 sdb1 sdc1
> 	md1	raid1 sda2 sdb2 sdc2
> 	md2	raid5 sda5 sdb5 sdc5
> 
> 	md0	/boot
> 	md1	swap
> 	md2	/
> 
> Does this look OK? What should the stripe and chunk
> sizes be, considering I'll be going with reiserfs?
> Typical usage: development machine, some DB apps
> with
> medium load, read-only mostly, not many writes. Very
> few large files (such as multimedia).
> 
> Or should I set up separate RAID-5's for /usr and
> /var
> as well?
> 
> Lastly, can I install directly to this
> configuration,
> or should I install on a separate disk and move
> things
> into the array?
> 
> Andargor
> 
> 
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam
> protection around 
> http://mail.yahoo.com 
> -
> To unsubscribe from this list: send the line
> "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at 
> http://vger.kernel.org/majordomo-info.html
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: First RAID Setup

[Bill Davidsen]

Callahan, Tom wrote:

>You "should" have a designated spare for RAID-5.
>
>Not sure why you have 3 disks for each RAID1, RAID1 is mirror, and unless
>the third drive is a spare, it is not needed.
>
>Thanks,
>Tom Callahan
>  
>
Why would you have a hot spare? In RAID-1 if one drive failed you would 
copy the full set of data to the "spare" so you know what to copy, 
another mirror can help your read performance under heavy load. You get 
the same protection as a hot spare, it's online if you need it, it 
covers the case where two drives fail... what's not to like? ;-)

If a third drive is justified, I see no reason to keep it as a spare 
rather than a live mirror in RAID-1.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: First RAID Setup

[Andrew Burgess]

>Since I'll be recreating the array anyway, I might as
>well split /, /home, and /var into three RAID5's.

Consider LVM2 which allows you to change the sizes of those
three partitions.

RE: First RAID Setup

[Callahan, Tom]

You forgot to </delurk>

:) Happy Holidays

Tom Callahan

-----Original Message-----
From: Tobias Hofmann [mailto:tobias.hofmann@medien.uni-weimar.de]
Sent: Thursday, December 15, 2005 4:32 PM
To: Brad Campbell
Cc: Callahan, Tom; 'Andargor The Wise'; linux-raid@vger.kernel.org
Subject: Re: First RAID Setup


<delurk>

On 15.12.2005 21:46, Brad Campbell wrote:
> Callahan, Tom wrote:
> 
>> It is always wise to build in a spare however, that being said about all
>> raid levels. In your configuration, if a disk fails in your RAID5, your
>> array will go down. RAID5 is usually 3+ disks, with a mirror. So you 
>> should
>> have 3 disks at minimum, and then a 4th as a spare.
> 
> /me wonders in the days of reliable RAID-6 why we use RAID-5 + spare?

Me too. ;) So, with holidays ahead, two questions (as I might tackle 
that soon and have not found it mentioned):

- How would one "switch" from the latter to the former? Is there 
something like "grow_to_RAID_6"?
- Does RAID6 have disadvantages wrt write speed?

TIA for any comments,

greets, tobi... :)

RE: First RAID Setup

[Callahan, Tom]

Sorry, I'm programmed for HA lately. Your plan sounds good then, I wish you
the best of luck.

Tom Callahan

-----Original Message-----
From: Andargor The Wise [mailto:andargor@yahoo.com]
Sent: Thursday, December 15, 2005 3:22 PM
To: Callahan, Tom; linux-raid@vger.kernel.org
Subject: RE: First RAID Setup




--- "Callahan, Tom" <CallahanT@tessco.com> wrote:

> I understand the reason for the RAID1 devices..... I
> was asking why you have
> 3 devices in the RAID1 setup? RAID1 is a mirrored
> configuration, requiring
> only 2 disks for operation.

Right. Like I indicated, I reviewed this list for
suggested configurations and this is what I came up
with as a result.

I guess you do only need two disks, since you'll be
able to recover anyway, but it seemed a simpler config
with identical disk partitions and only a small amount
of space wasted on one disk.

> It is always wise to build in a spare however, that
> being said about all
> raid levels. In your configuration, if a disk fails
> in your RAID5, your
> array will go down. RAID5 is usually 3+ disks, with
> a mirror. So you should
> have 3 disks at minimum, and then a 4th as a spare.

But if I don't mind the machine coming down, I don't
think I need a spare? I just want to be able to rip
out the bad drive, slap in a new one, rebuild, and be
back in business with all my data. I don't need HA.

(snip)
> Another gotcha, it's usually better to use entire
> disks, if you can afford
> to, in an MD array. This alleviates growing pains of
> having to manually
> repartition if you want to grow an exisiting
> filesystem. This may not make
> much sense now, but once you have to do it, you'll
> smack your forehead in
> grief.

Yes, I can see that, you instead grow by slapping in
extra disks and then resizing the array. Hmm. I'll
have to think about that.

> 
> Thanks,
> Tom Callahan


Thanks for the advice, lots to mull over. I've got
time, I'm still ddrescue'ing my crashed drive... :)

Andargor



> 
> -----Original Message-----
> From: Andargor The Wise [mailto:andargor@yahoo.com]
> Sent: Thursday, December 15, 2005 2:45 PM
> To: Callahan, Tom; linux-raid@vger.kernel.org
> Subject: RE: First RAID Setup
> 
> 
> The RAID1 partitions are to make sure:
> 
> 1) The machine is able to boot even if a disk is
> lost
> (/boot).
> 2) The machine isn't brought down if a disk is lost
> (swap)
> 
> I thought about a spare drive, but I don't need high
> availability. I'm satisfied with being able to
> recover
> my data.
> 
> Andargor
> 
> 
> --- "Callahan, Tom" <CallahanT@tessco.com> wrote:
> 
> > You "should" have a designated spare for RAID-5.
> > 
> > Not sure why you have 3 disks for each RAID1,
> RAID1
> > is mirror, and unless
> > the third drive is a spare, it is not needed.
> > 
> > Thanks,
> > Tom Callahan
> > 
> > -----Original Message-----
> > From: linux-raid-owner@vger.kernel.org
> > [mailto:linux-raid-owner@vger.kernel.org]On Behalf
> > Of Andargor The Wise
> > Sent: Thursday, December 15, 2005 2:10 PM
> > To: linux-raid@vger.kernel.org
> > Subject: First RAID Setup
> > 
> > 
> > I admit it. I'm a RAID virgin.
> > 
> > However, after a disastrous failure of the sole
> > drive
> > I wasn't backing up, I decided to go RAID-5 under
> > Slack 10.2 (first time ever with RAID-5).
> > 
> > The config:
> > 
> > Asus P5GL-MX (ICH6) mobo w/1 GB RAM, 4 x SATA
> ports
> > P4 3.0G/1M
> > 3 x WD2000JS 200.0 GB SATA drives
> > 
> > First, a question: the BIOS on this machine seems
> to
> > list the SATA ports as "third/fourth IDE
> > master/slave". Further, the documentation seems to
> > say
> > that SATA 1/2 are "master" and SATA 3/4 are
> "slave"
> > (black and red connectors, respectively).
> > 
> > My understanding is that SATA drives are each on
> > separate buses. Is this because the BIOS offers a
> > P-ATA emulation mode for SATA and it makes it
> > "easier"
> > to understand for novices to show them that way?
> > 
> > I ask because people have said that it is not a
> good
> > idea to have both IDE masters and slaves on the
> same
> > bus as part of a RAID-5 array. I know SATA is
> > different, but will using three of the SATA ports
> on
> > this mobo be OK?
> > 
> > Second, after reading the excellent advice in this
> > list, I decided that booting from RAID-5 might not
> > be
> > a good idea. So this is what I've been thinking:
> > 
> > Each disk partitioned alike:
> > 	1	30MB 
> > 	2	8GB (to allow for memory upgrades later)
> > 	5	rest_of_disk
> > 
> > mds:
> > 	md0	raid1 sda1 sdb1 sdc1
> > 	md1	raid1 sda2 sdb2 sdc2
> > 	md2	raid5 sda5 sdb5 sdc5
> > 
> > 	md0	/boot
> > 	md1	swap
> > 	md2	/
> > 
> > Does this look OK? What should the stripe and
> chunk
> > sizes be, considering I'll be going with reiserfs?
> > Typical usage: development machine, some DB apps
> > with
> > medium load, read-only mostly, not many writes.
> Very
> > few large files (such as multimedia).
> > 
> > Or should I set up separate RAID-5's for /usr and
> > /var
> > as well?
> > 
> > Lastly, can I install directly to this
> > configuration,
> > or should I install on a separate disk and move
> > things
> > into the array?
> > 
> > Andargor
> > 
> > 
> > __________________________________________________
> > Do You Yahoo!?
> > Tired of spam?  Yahoo! Mail has the best spam
> > protection around 
> > http://mail.yahoo.com 
> > -
> > To unsubscribe from this list: send the line
> > "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at 
> > http://vger.kernel.org/majordomo-info.html
> > 
> 
> 
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam
> protection around 
> http://mail.yahoo.com 
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

RE: First RAID Setup

[Andargor The Wise]

Ok, I got things up and running, hopefully. A question
about LILO, however.

This is my raidtab:

raiddev /dev/md0
        raid-level 1
        nr-raid-disks 3
        nr-spare-disks 0
        chunk-size 32
        persistent-superblock 1
        device /dev/sda1
        raid-disk 0
        device /dev/sdb1
        raid-disk 1
        device /dev/sdc1
        raid-disk 2

raiddev /dev/md1
        raid-level 1
        nr-raid-disks 3
        nr-spare-disks 0
        chunk-size 32
        persistent-superblock 1
        device /dev/sda2
        raid-disk 0
        device /dev/sdb2
        raid-disk 1
        device /dev/sdc2
        raid-disk 2

raiddev /dev/md2
        raid-level 5
        nr-raid-disks 3
        nr-spare-disks 0
        parity-algorithm left-symmetric
        chunk-size 32
        persistent-superblock 1
        device /dev/sda5
        raid-disk 0
        device /dev/sdb5
        raid-disk 1
        device /dev/sdc5
        raid-disk 2

And my lilo.conf:

boot=/dev/md0
raid-extra-boot=/dev/sda,/dev/sdb,/dev/sdc
map=/boot/System.map
install=/boot/boot.b
message=/boot/boot_message.txt
prompt
compact
lba32
timeout=30
default=Linux

change-rules
  reset
vga = normal

image = /boot/bzImage-2.6.14.4
  root = /dev/md2
  append="idebus=66"
  label = Linux
  read-only

image = /boot/vmlinuz
  root = /dev/md2
  append="idebus=66"
  label = slack_orig
  read-only

When I run LILO, I get:

Warning: COMPACT may conflict with LBA32 on some
systems
Added Linux *
Added slack_orig
The boot record of  /dev/md0  has been updated.
The boot record of  /dev/sda  has been updated.
Warning: /dev/sdb is not on the first disk
The boot record of  /dev/sdb  has been updated.
Warning: /dev/sdc is not on the first disk
The boot record of  /dev/sdc  has been updated.

Are the warnings normal?

Andargor



--- "Callahan, Tom" <CallahanT@tessco.com> wrote:

> Sorry, I'm programmed for HA lately. Your plan
> sounds good then, I wish you
> the best of luck.
> 
> Tom Callahan
> 
> -----Original Message-----
> From: Andargor The Wise [mailto:andargor@yahoo.com]
> Sent: Thursday, December 15, 2005 3:22 PM
> To: Callahan, Tom; linux-raid@vger.kernel.org
> Subject: RE: First RAID Setup
> 
> 
> 
> 
> --- "Callahan, Tom" <CallahanT@tessco.com> wrote:
> 
> > I understand the reason for the RAID1 devices.....
> I
> > was asking why you have
> > 3 devices in the RAID1 setup? RAID1 is a mirrored
> > configuration, requiring
> > only 2 disks for operation.
> 
> Right. Like I indicated, I reviewed this list for
> suggested configurations and this is what I came up
> with as a result.
> 
> I guess you do only need two disks, since you'll be
> able to recover anyway, but it seemed a simpler
> config
> with identical disk partitions and only a small
> amount
> of space wasted on one disk.
> 
> > It is always wise to build in a spare however,
> that
> > being said about all
> > raid levels. In your configuration, if a disk
> fails
> > in your RAID5, your
> > array will go down. RAID5 is usually 3+ disks,
> with
> > a mirror. So you should
> > have 3 disks at minimum, and then a 4th as a
> spare.
> 
> But if I don't mind the machine coming down, I don't
> think I need a spare? I just want to be able to rip
> out the bad drive, slap in a new one, rebuild, and
> be
> back in business with all my data. I don't need HA.
> 
> (snip)
> > Another gotcha, it's usually better to use entire
> > disks, if you can afford
> > to, in an MD array. This alleviates growing pains
> of
> > having to manually
> > repartition if you want to grow an exisiting
> > filesystem. This may not make
> > much sense now, but once you have to do it, you'll
> > smack your forehead in
> > grief.
> 
> Yes, I can see that, you instead grow by slapping in
> extra disks and then resizing the array. Hmm. I'll
> have to think about that.
> 
> > 
> > Thanks,
> > Tom Callahan
> 
> 
> Thanks for the advice, lots to mull over. I've got
> time, I'm still ddrescue'ing my crashed drive... :)
> 
> Andargor
> 
> 
> 
> > 
> > -----Original Message-----
> > From: Andargor The Wise
> [mailto:andargor@yahoo.com]
> > Sent: Thursday, December 15, 2005 2:45 PM
> > To: Callahan, Tom; linux-raid@vger.kernel.org
> > Subject: RE: First RAID Setup
> > 
> > 
> > The RAID1 partitions are to make sure:
> > 
> > 1) The machine is able to boot even if a disk is
> > lost
> > (/boot).
> > 2) The machine isn't brought down if a disk is
> lost
> > (swap)
> > 
> > I thought about a spare drive, but I don't need
> high
> > availability. I'm satisfied with being able to
> > recover
> > my data.
> > 
> > Andargor
> > 
> > 
> > --- "Callahan, Tom" <CallahanT@tessco.com> wrote:
> > 
> > > You "should" have a designated spare for RAID-5.
> > > 
> > > Not sure why you have 3 disks for each RAID1,
> > RAID1
> > > is mirror, and unless
> > > the third drive is a spare, it is not needed.
> > > 
> > > Thanks,
> > > Tom Callahan
> > > 
> > > -----Original Message-----
> > > From: linux-raid-owner@vger.kernel.org
> > > [mailto:linux-raid-owner@vger.kernel.org]On
> Behalf
> > > Of Andargor The Wise
> > > Sent: Thursday, December 15, 2005 2:10 PM
> > > To: linux-raid@vger.kernel.org
> > > Subject: First RAID Setup
> > > 
> > > 
> > > I admit it. I'm a RAID virgin.
> > > 
> > > However, after a disastrous failure of the sole
> > > drive
> > > I wasn't backing up, I decided to go RAID-5
> under
> > > Slack 10.2 (first time ever with RAID-5).
> > > 
> > > The config:
> > > 
> > > Asus P5GL-MX (ICH6) mobo w/1 GB RAM, 4 x SATA
> > ports
> > > P4 3.0G/1M
> > > 3 x WD2000JS 200.0 GB SATA drives
> > > 
> > > First, a question: the BIOS on this machine
> seems
> > to
> > > list the SATA ports as "third/fourth IDE
> > > master/slave". Further, the documentation seems
> to
> > > say
> > > that SATA 1/2 are "master" and SATA 3/4 are
> > "slave"
> > > (black and red connectors, respectively).
> > > 
> > > My understanding is that SATA drives are each on
> > > separate buses. Is this because the BIOS offers
> a
> > > P-ATA emulation mode for SATA and it makes it
> > > "easier"
> > > to understand for novices to show them that way?
> > > 
> > > I ask because people have said that it is not a
> > good
> > > idea to have both IDE masters and slaves on the
> > same
> > > bus as part of a RAID-5 array. I know SATA is
> > > different, but will using three of the SATA
> ports
> > on
> > > this mobo be OK?
> > > 
> > > Second, after reading the excellent advice in
> this
> > > list, I decided that booting from RAID-5 might
> not
> > > be
> > > a good idea. So this is what I've been thinking:
> > > 
> > > Each disk partitioned alike:
> > > 	1	30MB 
> > > 	2	8GB (to allow for memory upgrades later)
> > > 	5	rest_of_disk
> > > 
> > > mds:
> > > 	md0	raid1 sda1 sdb1 sdc1
> > > 	md1	raid1 sda2 sdb2 sdc2
> > > 	md2	raid5 sda5 sdb5 sdc5
> > > 
> > > 	md0	/boot
> > > 	md1	swap
> > > 	md2	/
> > > 
> > > Does this look OK? What should the stripe and
> > chunk
> > > sizes be, considering I'll be going with
> reiserfs?
> > > Typical usage: development machine, some DB apps
> > > with
> > > medium load, read-only mostly, not many writes.
> 
=== message truncated ===


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

RE: First RAID Setup

[Andargor The Wise]

Another question related to my first post in the
thread.

I'm currently locally rsync'ing the data I pulled off
my damaged disk to the RAID-5 array from one ssh
session. However, I notice a pause if I 'ls' on
another session while a large file is being rsync'ed
(until it is finished).

Is this normal? Does the fact that the Asus P5GL-MX
has 4 SATA ports, but they are marked "master/slave"
have anything to do with these pauses? I thought 

I would think that I have plenty of horsepower (P4
3.0G), memory and I/O bandwidth to avoid this...

I am using Slack 10.2, kernel 2.6.14.4 with AHCI
enabled and SATA in native mode on the three disks.

Andargor


--- Andargor The Wise <andargor@yahoo.com> wrote:

> Ok, I got things up and running, hopefully. A
> question
> about LILO, however.
> 
> This is my raidtab:
> 
> raiddev /dev/md0
>         raid-level 1
>         nr-raid-disks 3
>         nr-spare-disks 0
>         chunk-size 32
>         persistent-superblock 1
>         device /dev/sda1
>         raid-disk 0
>         device /dev/sdb1
>         raid-disk 1
>         device /dev/sdc1
>         raid-disk 2
> 
> raiddev /dev/md1
>         raid-level 1
>         nr-raid-disks 3
>         nr-spare-disks 0
>         chunk-size 32
>         persistent-superblock 1
>         device /dev/sda2
>         raid-disk 0
>         device /dev/sdb2
>         raid-disk 1
>         device /dev/sdc2
>         raid-disk 2
> 
> raiddev /dev/md2
>         raid-level 5
>         nr-raid-disks 3
>         nr-spare-disks 0
>         parity-algorithm left-symmetric
>         chunk-size 32
>         persistent-superblock 1
>         device /dev/sda5
>         raid-disk 0
>         device /dev/sdb5
>         raid-disk 1
>         device /dev/sdc5
>         raid-disk 2
> 
> And my lilo.conf:
> 
> boot=/dev/md0
> raid-extra-boot=/dev/sda,/dev/sdb,/dev/sdc
> map=/boot/System.map
> install=/boot/boot.b
> message=/boot/boot_message.txt
> prompt
> compact
> lba32
> timeout=30
> default=Linux
> 
> change-rules
>   reset
> vga = normal
> 
> image = /boot/bzImage-2.6.14.4
>   root = /dev/md2
>   append="idebus=66"
>   label = Linux
>   read-only
> 
> image = /boot/vmlinuz
>   root = /dev/md2
>   append="idebus=66"
>   label = slack_orig
>   read-only
> 
> When I run LILO, I get:
> 
> Warning: COMPACT may conflict with LBA32 on some
> systems
> Added Linux *
> Added slack_orig
> The boot record of  /dev/md0  has been updated.
> The boot record of  /dev/sda  has been updated.
> Warning: /dev/sdb is not on the first disk
> The boot record of  /dev/sdb  has been updated.
> Warning: /dev/sdc is not on the first disk
> The boot record of  /dev/sdc  has been updated.
> 
> Are the warnings normal?
> 
> Andargor
> 
> 
> 
> --- "Callahan, Tom" <CallahanT@tessco.com> wrote:
> 
> > Sorry, I'm programmed for HA lately. Your plan
> > sounds good then, I wish you
> > the best of luck.
> > 
> > Tom Callahan
> > 
> > -----Original Message-----
> > From: Andargor The Wise
> [mailto:andargor@yahoo.com]
> > Sent: Thursday, December 15, 2005 3:22 PM
> > To: Callahan, Tom; linux-raid@vger.kernel.org
> > Subject: RE: First RAID Setup
> > 
> > 
> > 
> > 
> > --- "Callahan, Tom" <CallahanT@tessco.com> wrote:
> > 
> > > I understand the reason for the RAID1
> devices.....
> > I
> > > was asking why you have
> > > 3 devices in the RAID1 setup? RAID1 is a
> mirrored
> > > configuration, requiring
> > > only 2 disks for operation.
> > 
> > Right. Like I indicated, I reviewed this list for
> > suggested configurations and this is what I came
> up
> > with as a result.
> > 
> > I guess you do only need two disks, since you'll
> be
> > able to recover anyway, but it seemed a simpler
> > config
> > with identical disk partitions and only a small
> > amount
> > of space wasted on one disk.
> > 
> > > It is always wise to build in a spare however,
> > that
> > > being said about all
> > > raid levels. In your configuration, if a disk
> > fails
> > > in your RAID5, your
> > > array will go down. RAID5 is usually 3+ disks,
> > with
> > > a mirror. So you should
> > > have 3 disks at minimum, and then a 4th as a
> > spare.
> > 
> > But if I don't mind the machine coming down, I
> don't
> > think I need a spare? I just want to be able to
> rip
> > out the bad drive, slap in a new one, rebuild, and
> > be
> > back in business with all my data. I don't need
> HA.
> > 
> > (snip)
> > > Another gotcha, it's usually better to use
> entire
> > > disks, if you can afford
> > > to, in an MD array. This alleviates growing
> pains
> > of
> > > having to manually
> > > repartition if you want to grow an exisiting
> > > filesystem. This may not make
> > > much sense now, but once you have to do it,
> you'll
> > > smack your forehead in
> > > grief.
> > 
> > Yes, I can see that, you instead grow by slapping
> in
> > extra disks and then resizing the array. Hmm. I'll
> > have to think about that.
> > 
> > > 
> > > Thanks,
> > > Tom Callahan
> > 
> > 
> > Thanks for the advice, lots to mull over. I've got
> > time, I'm still ddrescue'ing my crashed drive...
> :)
> > 
> > Andargor
> > 
> > 
> > 
> > > 
> > > -----Original Message-----
> > > From: Andargor The Wise
> > [mailto:andargor@yahoo.com]
> > > Sent: Thursday, December 15, 2005 2:45 PM
> > > To: Callahan, Tom; linux-raid@vger.kernel.org
> > > Subject: RE: First RAID Setup
> > > 
> > > 
> > > The RAID1 partitions are to make sure:
> > > 
> 
=== message truncated ===


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

RE: First RAID Setup

[Andargor The Wise]

Yet another thing, someone has suggested that I should
increase the chunk size for my RAID5 from 32 to either
64 or 128.

Is it worth it, considering that the system doesn't
normally run on a heavy load? Mail for a few users,
some read-only database applications, website, etc.
Mostly a development machine.

Would this alleviate the "pauses" during large file
transfers/copies that I have indicated in my previous
post?

I'm asking because backing up ~176 GB, reconfiguring
the RAID, and restoring it properly so the machine
boots (the RAID5 is /) is quite a PITA.

Andargor


--- Andargor The Wise <andargor@yahoo.com> wrote:

> Another question related to my first post in the
> thread.
> 
> I'm currently locally rsync'ing the data I pulled
> off
> my damaged disk to the RAID-5 array from one ssh
> session. However, I notice a pause if I 'ls' on
> another session while a large file is being rsync'ed
> (until it is finished).
> 
> Is this normal? Does the fact that the Asus P5GL-MX
> has 4 SATA ports, but they are marked "master/slave"
> have anything to do with these pauses? I thought 
> 
> I would think that I have plenty of horsepower (P4
> 3.0G), memory and I/O bandwidth to avoid this...
> 
> I am using Slack 10.2, kernel 2.6.14.4 with AHCI
> enabled and SATA in native mode on the three disks.
> 
> Andargor
> 
> 
> --- Andargor The Wise <andargor@yahoo.com> wrote:
> 
> > Ok, I got things up and running, hopefully. A
> > question
> > about LILO, however.
> > 
> > This is my raidtab:
> > 
> > raiddev /dev/md0
> >         raid-level 1
> >         nr-raid-disks 3
> >         nr-spare-disks 0
> >         chunk-size 32
> >         persistent-superblock 1
> >         device /dev/sda1
> >         raid-disk 0
> >         device /dev/sdb1
> >         raid-disk 1
> >         device /dev/sdc1
> >         raid-disk 2
> > 
> > raiddev /dev/md1
> >         raid-level 1
> >         nr-raid-disks 3
> >         nr-spare-disks 0
> >         chunk-size 32
> >         persistent-superblock 1
> >         device /dev/sda2
> >         raid-disk 0
> >         device /dev/sdb2
> >         raid-disk 1
> >         device /dev/sdc2
> >         raid-disk 2
> > 
> > raiddev /dev/md2
> >         raid-level 5
> >         nr-raid-disks 3
> >         nr-spare-disks 0
> >         parity-algorithm left-symmetric
> >         chunk-size 32
> >         persistent-superblock 1
> >         device /dev/sda5
> >         raid-disk 0
> >         device /dev/sdb5
> >         raid-disk 1
> >         device /dev/sdc5
> >         raid-disk 2
> > 
> > And my lilo.conf:
> > 
> > boot=/dev/md0
> > raid-extra-boot=/dev/sda,/dev/sdb,/dev/sdc
> > map=/boot/System.map
> > install=/boot/boot.b
> > message=/boot/boot_message.txt
> > prompt
> > compact
> > lba32
> > timeout=30
> > default=Linux
> > 
> > change-rules
> >   reset
> > vga = normal
> > 
> > image = /boot/bzImage-2.6.14.4
> >   root = /dev/md2
> >   append="idebus=66"
> >   label = Linux
> >   read-only
> > 
> > image = /boot/vmlinuz
> >   root = /dev/md2
> >   append="idebus=66"
> >   label = slack_orig
> >   read-only
> > 
> > When I run LILO, I get:
> > 
> > Warning: COMPACT may conflict with LBA32 on some
> > systems
> > Added Linux *
> > Added slack_orig
> > The boot record of  /dev/md0  has been updated.
> > The boot record of  /dev/sda  has been updated.
> > Warning: /dev/sdb is not on the first disk
> > The boot record of  /dev/sdb  has been updated.
> > Warning: /dev/sdc is not on the first disk
> > The boot record of  /dev/sdc  has been updated.
> > 
> > Are the warnings normal?
> > 
> > Andargor
> > 
> > 
> > 
> > --- "Callahan, Tom" <CallahanT@tessco.com> wrote:
> > 
> > > Sorry, I'm programmed for HA lately. Your plan
> > > sounds good then, I wish you
> > > the best of luck.
> > > 
> > > Tom Callahan
> > > 
> > > -----Original Message-----
> > > From: Andargor The Wise
> > [mailto:andargor@yahoo.com]
> > > Sent: Thursday, December 15, 2005 3:22 PM
> > > To: Callahan, Tom; linux-raid@vger.kernel.org
> > > Subject: RE: First RAID Setup
> > > 
> > > 
> > > 
> > > 
> > > --- "Callahan, Tom" <CallahanT@tessco.com>
> wrote:
> > > 
> > > > I understand the reason for the RAID1
> > devices.....
> > > I
> > > > was asking why you have
> > > > 3 devices in the RAID1 setup? RAID1 is a
> > mirrored
> > > > configuration, requiring
> > > > only 2 disks for operation.
> > > 
> > > Right. Like I indicated, I reviewed this list
> for
> > > suggested configurations and this is what I came
> > up
> > > with as a result.
> > > 
> > > I guess you do only need two disks, since you'll
> > be
> > > able to recover anyway, but it seemed a simpler
> > > config
> > > with identical disk partitions and only a small
> > > amount
> > > of space wasted on one disk.
> > > 
> > > > It is always wise to build in a spare however,
> > > that
> > > > being said about all
> > > > raid levels. In your configuration, if a disk
> > > fails
> > > > in your RAID5, your
> > > > array will go down. RAID5 is usually 3+ disks,
> > > with
> > > > a mirror. So you should
> > > > have 3 disks at minimum, and then a 4th as a
> > > spare.
> > > 
> > > But if I don't mind the machine coming down, I
> > don't
> > > think I need a spare? I just want to be able to
> > rip
> > > out the bad drive, slap in a new one, rebuild,
> and
> > > be
> > > back in business with all my data. I don't need
> > HA.
> > > 
> > > (snip)
> > > > Another gotcha, it's usually better to use
> > entire
> > > > disks, if you can afford
> > > > to, in an MD array. This alleviates growing
> > pains
> > > of
> > > > having to manually
> > > > repartition if you want to grow an exisiting
> > > > filesystem. This may not make
> > > > much sense now, but once you have to do it,
> > you'll
> > > > smack your forehead in
> > > > grief.
> > > 
> > > Yes, I can see that, you instead grow by
> slapping
> 
=== message truncated ===


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: First RAID Setup

[Bill Davidsen]

Andargor The Wise wrote:

>Yet another thing, someone has suggested that I should
>increase the chunk size for my RAID5 from 32 to either
>64 or 128.
>
>Is it worth it, considering that the system doesn't
>normally run on a heavy load? Mail for a few users,
>some read-only database applications, website, etc.
>Mostly a development machine.
>  
>

Can't think of a case where it's not worth having better performance... 
I should write a WP on stripe size, and what happens as you change it 
with given loads.

>Would this alleviate the "pauses" during large file
>transfers/copies that I have indicated in my previous
>post?
>
>I'm asking because backing up ~176 GB, reconfiguring
>the RAID, and restoring it properly so the machine
>boots (the RAID5 is /) is quite a PITA.
>

You may have some special case, but I would never put data that large in 
/ just as a system admin issue. It makes backups and restores, as well 
as upgrades quite unpleasant. I guess you may have noticed that by now 
;-) Good luck!

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: First RAID Setup

[Andargor The Wise]

--- Bill Davidsen <davidsen@tmr.com> wrote:

> Andargor The Wise wrote:
> 
> >Yet another thing, someone has suggested that I
> should
> >increase the chunk size for my RAID5 from 32 to
> either
> >64 or 128.
> >
> >Is it worth it, considering that the system doesn't
> >normally run on a heavy load? Mail for a few users,
> >some read-only database applications, website, etc.
> >Mostly a development machine.
> >  
> >
> Can't think of a case where it's not worth having
> better performance... 
> I should write a WP on stripe size, and what happens
> as you change it 
> with given loads.
>

If anyone's interested, I ran some benchmarks:

http://www.andargor.com/raid5.html

I'll be spending the time to go to 128k chunks. :)

I was considering raidreconf, and tested it on a
scratch partition. It worked ok, but it was slow (1.17
MB/sec overall). I'll rsync or find -xdev | cpio
-admpv instead to mirror the existing stuff elsewhere
before blowing away the array.


> >Would this alleviate the "pauses" during large file
> >transfers/copies that I have indicated in my
> previous
> >post?
> >
> >I'm asking because backing up ~176 GB,
> reconfiguring
> >the RAID, and restoring it properly so the machine
> >boots (the RAID5 is /) is quite a PITA.
> >
> 
> You may have some special case, but I would never
> put data that large in 
> / just as a system admin issue. It makes backups and
> restores, as well 
> as upgrades quite unpleasant. I guess you may have
> noticed that by now 
> ;-) Good luck!

Concerning the "pause" issue, I believe my problem was
not really a problem at all. The priority isn't on
intreactive traffic (CONFIG_PREEMPT=n), so it was just
a perception of a pause during "ls". If I had real
"pauses" in my IO, I would have seen dismal benchmark
results.

Since I'll be recreating the array anyway, I might as
well split /, /home, and /var into three RAID5's.

Andargor




__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

RE: First RAID Setup

[Callahan, Tom]

I understand the reason for the RAID1 devices..... I was asking why you have
3 devices in the RAID1 setup? RAID1 is a mirrored configuration, requiring
only 2 disks for operation.

It is always wise to build in a spare however, that being said about all
raid levels. In your configuration, if a disk fails in your RAID5, your
array will go down. RAID5 is usually 3+ disks, with a mirror. So you should
have 3 disks at minimum, and then a 4th as a spare.

The MD modules/subsystem will then automagically bring in that spare disk if
any of the existing 3 in your RAID5 setup fail.

It is wise to think through your layout prior to building, and I commend you
for that. You may also want to review/experiment with the MD subsystem. For
instance, There is a neat --grow mode that is not mentioned in many vendor
man pages that can allow you to grow an MD device as needed.

Another gotcha, it's usually better to use entire disks, if you can afford
to, in an MD array. This alleviates growing pains of having to manually
repartition if you want to grow an exisiting filesystem. This may not make
much sense now, but once you have to do it, you'll smack your forehead in
grief.

Thanks,
Tom Callahan

-----Original Message-----
From: Andargor The Wise [mailto:andargor@yahoo.com]
Sent: Thursday, December 15, 2005 2:45 PM
To: Callahan, Tom; linux-raid@vger.kernel.org
Subject: RE: First RAID Setup


The RAID1 partitions are to make sure:

1) The machine is able to boot even if a disk is lost
(/boot).
2) The machine isn't brought down if a disk is lost
(swap)

I thought about a spare drive, but I don't need high
availability. I'm satisfied with being able to recover
my data.

Andargor


--- "Callahan, Tom" <CallahanT@tessco.com> wrote:

> You "should" have a designated spare for RAID-5.
> 
> Not sure why you have 3 disks for each RAID1, RAID1
> is mirror, and unless
> the third drive is a spare, it is not needed.
> 
> Thanks,
> Tom Callahan
> 
> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org
> [mailto:linux-raid-owner@vger.kernel.org]On Behalf
> Of Andargor The Wise
> Sent: Thursday, December 15, 2005 2:10 PM
> To: linux-raid@vger.kernel.org
> Subject: First RAID Setup
> 
> 
> I admit it. I'm a RAID virgin.
> 
> However, after a disastrous failure of the sole
> drive
> I wasn't backing up, I decided to go RAID-5 under
> Slack 10.2 (first time ever with RAID-5).
> 
> The config:
> 
> Asus P5GL-MX (ICH6) mobo w/1 GB RAM, 4 x SATA ports
> P4 3.0G/1M
> 3 x WD2000JS 200.0 GB SATA drives
> 
> First, a question: the BIOS on this machine seems to
> list the SATA ports as "third/fourth IDE
> master/slave". Further, the documentation seems to
> say
> that SATA 1/2 are "master" and SATA 3/4 are "slave"
> (black and red connectors, respectively).
> 
> My understanding is that SATA drives are each on
> separate buses. Is this because the BIOS offers a
> P-ATA emulation mode for SATA and it makes it
> "easier"
> to understand for novices to show them that way?
> 
> I ask because people have said that it is not a good
> idea to have both IDE masters and slaves on the same
> bus as part of a RAID-5 array. I know SATA is
> different, but will using three of the SATA ports on
> this mobo be OK?
> 
> Second, after reading the excellent advice in this
> list, I decided that booting from RAID-5 might not
> be
> a good idea. So this is what I've been thinking:
> 
> Each disk partitioned alike:
> 	1	30MB 
> 	2	8GB (to allow for memory upgrades later)
> 	5	rest_of_disk
> 
> mds:
> 	md0	raid1 sda1 sdb1 sdc1
> 	md1	raid1 sda2 sdb2 sdc2
> 	md2	raid5 sda5 sdb5 sdc5
> 
> 	md0	/boot
> 	md1	swap
> 	md2	/
> 
> Does this look OK? What should the stripe and chunk
> sizes be, considering I'll be going with reiserfs?
> Typical usage: development machine, some DB apps
> with
> medium load, read-only mostly, not many writes. Very
> few large files (such as multimedia).
> 
> Or should I set up separate RAID-5's for /usr and
> /var
> as well?
> 
> Lastly, can I install directly to this
> configuration,
> or should I install on a separate disk and move
> things
> into the array?
> 
> Andargor
> 
> 
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam
> protection around 
> http://mail.yahoo.com 
> -
> To unsubscribe from this list: send the line
> "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at 
> http://vger.kernel.org/majordomo-info.html
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

RE: First RAID Setup

[Andargor The Wise]

--- "Callahan, Tom" <CallahanT@tessco.com> wrote:

> I understand the reason for the RAID1 devices..... I
> was asking why you have
> 3 devices in the RAID1 setup? RAID1 is a mirrored
> configuration, requiring
> only 2 disks for operation.

Right. Like I indicated, I reviewed this list for
suggested configurations and this is what I came up
with as a result.

I guess you do only need two disks, since you'll be
able to recover anyway, but it seemed a simpler config
with identical disk partitions and only a small amount
of space wasted on one disk.

> It is always wise to build in a spare however, that
> being said about all
> raid levels. In your configuration, if a disk fails
> in your RAID5, your
> array will go down. RAID5 is usually 3+ disks, with
> a mirror. So you should
> have 3 disks at minimum, and then a 4th as a spare.

But if I don't mind the machine coming down, I don't
think I need a spare? I just want to be able to rip
out the bad drive, slap in a new one, rebuild, and be
back in business with all my data. I don't need HA.

(snip)
> Another gotcha, it's usually better to use entire
> disks, if you can afford
> to, in an MD array. This alleviates growing pains of
> having to manually
> repartition if you want to grow an exisiting
> filesystem. This may not make
> much sense now, but once you have to do it, you'll
> smack your forehead in
> grief.

Yes, I can see that, you instead grow by slapping in
extra disks and then resizing the array. Hmm. I'll
have to think about that.

> 
> Thanks,
> Tom Callahan


Thanks for the advice, lots to mull over. I've got
time, I'm still ddrescue'ing my crashed drive... :)

Andargor



> 
> -----Original Message-----
> From: Andargor The Wise [mailto:andargor@yahoo.com]
> Sent: Thursday, December 15, 2005 2:45 PM
> To: Callahan, Tom; linux-raid@vger.kernel.org
> Subject: RE: First RAID Setup
> 
> 
> The RAID1 partitions are to make sure:
> 
> 1) The machine is able to boot even if a disk is
> lost
> (/boot).
> 2) The machine isn't brought down if a disk is lost
> (swap)
> 
> I thought about a spare drive, but I don't need high
> availability. I'm satisfied with being able to
> recover
> my data.
> 
> Andargor
> 
> 
> --- "Callahan, Tom" <CallahanT@tessco.com> wrote:
> 
> > You "should" have a designated spare for RAID-5.
> > 
> > Not sure why you have 3 disks for each RAID1,
> RAID1
> > is mirror, and unless
> > the third drive is a spare, it is not needed.
> > 
> > Thanks,
> > Tom Callahan
> > 
> > -----Original Message-----
> > From: linux-raid-owner@vger.kernel.org
> > [mailto:linux-raid-owner@vger.kernel.org]On Behalf
> > Of Andargor The Wise
> > Sent: Thursday, December 15, 2005 2:10 PM
> > To: linux-raid@vger.kernel.org
> > Subject: First RAID Setup
> > 
> > 
> > I admit it. I'm a RAID virgin.
> > 
> > However, after a disastrous failure of the sole
> > drive
> > I wasn't backing up, I decided to go RAID-5 under
> > Slack 10.2 (first time ever with RAID-5).
> > 
> > The config:
> > 
> > Asus P5GL-MX (ICH6) mobo w/1 GB RAM, 4 x SATA
> ports
> > P4 3.0G/1M
> > 3 x WD2000JS 200.0 GB SATA drives
> > 
> > First, a question: the BIOS on this machine seems
> to
> > list the SATA ports as "third/fourth IDE
> > master/slave". Further, the documentation seems to
> > say
> > that SATA 1/2 are "master" and SATA 3/4 are
> "slave"
> > (black and red connectors, respectively).
> > 
> > My understanding is that SATA drives are each on
> > separate buses. Is this because the BIOS offers a
> > P-ATA emulation mode for SATA and it makes it
> > "easier"
> > to understand for novices to show them that way?
> > 
> > I ask because people have said that it is not a
> good
> > idea to have both IDE masters and slaves on the
> same
> > bus as part of a RAID-5 array. I know SATA is
> > different, but will using three of the SATA ports
> on
> > this mobo be OK?
> > 
> > Second, after reading the excellent advice in this
> > list, I decided that booting from RAID-5 might not
> > be
> > a good idea. So this is what I've been thinking:
> > 
> > Each disk partitioned alike:
> > 	1	30MB 
> > 	2	8GB (to allow for memory upgrades later)
> > 	5	rest_of_disk
> > 
> > mds:
> > 	md0	raid1 sda1 sdb1 sdc1
> > 	md1	raid1 sda2 sdb2 sdc2
> > 	md2	raid5 sda5 sdb5 sdc5
> > 
> > 	md0	/boot
> > 	md1	swap
> > 	md2	/
> > 
> > Does this look OK? What should the stripe and
> chunk
> > sizes be, considering I'll be going with reiserfs?
> > Typical usage: development machine, some DB apps
> > with
> > medium load, read-only mostly, not many writes.
> Very
> > few large files (such as multimedia).
> > 
> > Or should I set up separate RAID-5's for /usr and
> > /var
> > as well?
> > 
> > Lastly, can I install directly to this
> > configuration,
> > or should I install on a separate disk and move
> > things
> > into the array?
> > 
> > Andargor
> > 
> > 
> > __________________________________________________
> > Do You Yahoo!?
> > Tired of spam?  Yahoo! Mail has the best spam
> > protection around 
> > http://mail.yahoo.com 
> > -
> > To unsubscribe from this list: send the line
> > "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at 
> > http://vger.kernel.org/majordomo-info.html
> > 
> 
> 
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam
> protection around 
> http://mail.yahoo.com 
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: First RAID Setup

[Brad Campbell]

Callahan, Tom wrote:

> It is always wise to build in a spare however, that being said about all
> raid levels. In your configuration, if a disk fails in your RAID5, your
> array will go down. RAID5 is usually 3+ disks, with a mirror. So you should
> have 3 disks at minimum, and then a 4th as a spare.
> 

/me wonders in the days of reliable RAID-6 why we use RAID-5 + spare?

RAID-6 has saved me twice now from dual drive failures on a 15 disk array.
It's schweeeeeeeeeett

Regards,
Brad
-- 
"Human beings, who are almost unique in having the ability
to learn from the experience of others, are also remarkable
for their apparent disinclination to do so." -- Douglas Adams

Re: First RAID Setup

[Tobias Hofmann]

<delurk>

On 15.12.2005 21:46, Brad Campbell wrote:
> Callahan, Tom wrote:
> 
>> It is always wise to build in a spare however, that being said about all
>> raid levels. In your configuration, if a disk fails in your RAID5, your
>> array will go down. RAID5 is usually 3+ disks, with a mirror. So you 
>> should
>> have 3 disks at minimum, and then a 4th as a spare.
> 
> /me wonders in the days of reliable RAID-6 why we use RAID-5 + spare?

Me too. ;) So, with holidays ahead, two questions (as I might tackle 
that soon and have not found it mentioned):

- How would one "switch" from the latter to the former? Is there 
something like "grow_to_RAID_6"?
- Does RAID6 have disadvantages wrt write speed?

TIA for any comments,

greets, tobi... :)

Re: First RAID Setup

[Max Waterman]

Tobias Hofmann wrote:
> <delurk>
> 
> On 15.12.2005 21:46, Brad Campbell wrote:
> 
>> Callahan, Tom wrote:
>>
>>> It is always wise to build in a spare however, that being said about all
>>> raid levels. In your configuration, if a disk fails in your RAID5, your
>>> array will go down. RAID5 is usually 3+ disks, with a mirror. So you 
>>> should
>>> have 3 disks at minimum, and then a 4th as a spare.
>>
>>
>> /me wonders in the days of reliable RAID-6 why we use RAID-5 + spare?

Just so I am clear on this : while RAID5 consumes a disk's worth of 
space for parity, resulting in n-1 disk's worth of space available for 
storage, RAID6 consumes two disk's worth of space for two parities, 
resulting in n-2 disk's worth of space available for storage.

Is that correct?

> 
> 
> Me too. ;) So, with holidays ahead, two questions (as I might tackle 
> that soon and have not found it mentioned):

I would guess one reason might be that you only have 3 disks - ok 
(minimum) for RAID5, but not for RAID6 (minimum is 4?)?

> 
> - How would one "switch" from the latter to the former? Is there 
> something like "grow_to_RAID_6"?

I'd like to know this too. RAID6 sounds ideal for my array (8 disks - 
currently 6 RAID5 + 2 spares; one is currently at the cleaners) - I 
could even have an extra disk's worth of space, I think.

> - Does RAID6 have disadvantages wrt write speed?

...and what about rebuilding speed? Does it have to read and process 
twice as much parity data?.

Perhaps it is faster for reading (data can be spread across another disk 
[which would be empty in the RAID5+spare config]).

Max.

> 
> TIA for any comments,
> 
> greets, tobi... :)
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: First RAID Setup

[Tobias Hofmann]

On 16.12.2005 02:51, Max Waterman wrote:
> Tobias Hofmann wrote:
> 
>> <delurk>
>>
>> On 15.12.2005 21:46, Brad Campbell wrote:

[...]

>>> /me wonders in the days of reliable RAID-6 why we use RAID-5 + spare?
> 
> Just so I am clear on this : while RAID5 consumes a disk's worth of 
> space for parity, resulting in n-1 disk's worth of space available for 
> storage, RAID6 consumes two disk's worth of space for two parities, 
> resulting in n-2 disk's worth of space available for storage.
> 
> Is that correct?

imho yes.

>> Me too. ;) So, with holidays ahead, two questions (as I might tackle 
>> that soon and have not found it mentioned):
> 
> I would guess one reason might be that you only have 3 disks - ok 
> (minimum) for RAID5, but not for RAID6 (minimum is 4?)?

I would, in the long run, probably prefer to shell out the money for 
another disk (RAID6, then) instead of being bitten of a bad block 
showing up while resyncing a RAID5, leading to all the effects mentioned 
on this list before. I have been there already with a commercial EIDE/FC 
raid enclosure - ugly. Of course, this risk can be reduced once the 
infamous passthrough patch for enabling SMART capabilities for SATA 
disks finally makes it into mainline kernel, which seems to be the case 
with 2.6.15, and one can have a daily check for bad blocks...

greets, tobi... :)

Re: First RAID Setup

[Neil Brown]

On Thursday December 15, tobias.hofmann@medien.uni-weimar.de wrote:
> <delurk>
> 
> On 15.12.2005 21:46, Brad Campbell wrote:
> > Callahan, Tom wrote:
> > 
> >> It is always wise to build in a spare however, that being said about all
> >> raid levels. In your configuration, if a disk fails in your RAID5, your
> >> array will go down. RAID5 is usually 3+ disks, with a mirror. So you 
> >> should
> >> have 3 disks at minimum, and then a 4th as a spare.
> > 
> > /me wonders in the days of reliable RAID-6 why we use RAID-5 + spare?
> 
> Me too. ;) So, with holidays ahead, two questions (as I might tackle 
> that soon and have not found it mentioned):
> 
> - How would one "switch" from the latter to the former? Is there 
> something like "grow_to_RAID_6"?

No... at least not yet....

> - Does RAID6 have disadvantages wrt write speed?

Probably.  I haven't done any measurements myself, but from a
theoretical standpoint, you would expect raid6 to impose more CPU load
(though that may not be noticeable) and as raid6 need to see the whole
stripe to update the P and Q blocks (it cannot do a subtract old, add
new update) there could well be more IO happening, particularly on
large arrays (>5 devices).

Of course, whether this caused noticeable reduction in throughput and
latency would be very workload-dependant.

NeilBrown

Re: First RAID Setup

[Tobias Hofmann]

On 16.12.2005 02:53, Neil Brown wrote:
> On Thursday December 15, tobias.hofmann@medien.uni-weimar.de wrote:
> 
>><delurk>
>>
>>On 15.12.2005 21:46, Brad Campbell wrote:
>>
>>>Callahan, Tom wrote:
>>>
>>>
>>>>It is always wise to build in a spare however, that being said about all
>>>>raid levels. In your configuration, if a disk fails in your RAID5, your
>>>>array will go down. RAID5 is usually 3+ disks, with a mirror. So you 
>>>>should
>>>>have 3 disks at minimum, and then a 4th as a spare.
>>>
>>>/me wonders in the days of reliable RAID-6 why we use RAID-5 + spare?
>>
>>Me too. ;) So, with holidays ahead, two questions (as I might tackle 
>>that soon and have not found it mentioned):
>>
>>- How would one "switch" from the latter to the former? Is there 
>>something like "grow_to_RAID_6"?
> 
> 
> No... at least not yet....

Hm. :) Around here, it is pretty much Christmas time, with all the 
"wishing" going on - hint, hint? ;)

>>- Does RAID6 have disadvantages wrt write speed?
> 
> Probably.  I haven't done any measurements myself, but from a
> theoretical standpoint, you would expect raid6 to impose more CPU load
> (though that may not be noticeable) and as raid6 need to see the whole
> stripe to update the P and Q blocks (it cannot do a subtract old, add
> new update) there could well be more IO happening, particularly on
> large arrays (>5 devices).

That,s about the line of thinking I was considering. I,ll see, if I 
tackle this over the holidays, to do some tests...

> Of course, whether this caused noticeable reduction in throughput and
> latency would be very workload-dependant.

My scenarios here @home will probably not be very relevant to what one 
would like to know in a production environment - whatever that is... :)

Thanks for the input, Neil,

greets, tobi... :)

Re: First RAID Setup

[Neil Brown]

On Friday December 16, tobias.hofmann@medien.uni-weimar.de wrote:
> >>
> >>- How would one "switch" from the latter to the former? Is there 
> >>something like "grow_to_RAID_6"?
> > 
> > 
> > No... at least not yet....
> 
> Hm. :) Around here, it is pretty much Christmas time, with all the 
> "wishing" going on - hint, hint? ;)
> 

Growing a raid5 to a raid6 with the same amount of data is only really
achievable if we use a modified raid6 layout with the Q syndrome all
on the one drive (a bit like raid4 which has parity all on the one
drive).
This gives you the better redundancy, but you have a hot-spot for
writes and don't distribute the reads any better.

This sort of conversion may be available in a couple of months.


Rearranging the data blocks is impractical as you need a temporary
storage:
    copy to blocks somewhere else and be sure they are safe
    there.  Then copy them back and be sure they are safe in the new
    place. 
Without this it is not possible to be sure to survive a crash.

Converting a raid5 to a raid6 with 2 extra drives (thus having more
space as well as more redundancy) is a possibility as the extra space
can be effectively used to keep to old copy of a stripe safe while
writing the new copy.

This sort of conversion is planned, but will take somewhat longer.

NeilBrown

Re: First RAID Setup

[Max Waterman]

Neil Brown wrote:
>
> Converting a raid5 to a raid6 with 2 extra drives (thus having more
> space as well as more redundancy)

Yeah! That's me :D

> is a possibility as the extra space
> can be effectively used to keep to old copy of a stripe safe while
> writing the new copy.
> 
> This sort of conversion is planned, but will take somewhat longer.

Great!

I'm not in a hurry (though perhaps I should be :|).

Max.

Re: First RAID Setup

[Gordon Henderson]

On Fri, 16 Dec 2005, Neil Brown wrote:

> > - Does RAID6 have disadvantages wrt write speed?
>
> Probably.  I haven't done any measurements myself, but from a
> theoretical standpoint, you would expect raid6 to impose more CPU load
> (though that may not be noticeable) and as raid6 need to see the whole
> stripe to update the P and Q blocks (it cannot do a subtract old, add
> new update) there could well be more IO happening, particularly on
> large arrays (>5 devices).
>
> Of course, whether this caused noticeable reduction in throughput and
> latency would be very workload-dependant.

I now have several servers using RAID-6 in a production environment - so
far so good. For the most part, they are all connected to 100Mb networks,
and they are file-servers (rather than serving to a local application) and
throughput from the servers to the clients hasn't been a problem. I have
no problems saturating the Ethernet with file traffic in experiments I've
performed (mainly running Bonnie via NFS!)

Running Bonnie locally, I can get over 100MB/sec on some of these boses,
depending on the disks and processor, so it might just be able to cope
with Gb Ethernet, but if I were building something with those speeds
in-mind, then I probably wouldn't be using comodity hardware...

There is something that says that if you have a batch of disks which are
likely to fail then theres a certian probability that they'll all fail
round about the same time, so having an array with a cold spare might help
that, however that might be more hassle than it's worth, who knows!

Bonnie (and hdparm) are fairly crude when it comes to benchmarks, but they
give a fair idea of what to expect when streaming data - here is output of
a Bonnie run over a 6-disk RAID-6 array (SCSI) on a Dull 2850 dual Xeon
jobbie:

Version 1.02b       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
black            2G           74811  34 35960  16           79723  17 461.7   1
black,2G,,,74811,34,35960,16,,,79723,17,461.7,1,,,,,,,,,,,,,

I haven't tried to do anything clever (not sure I could with this!) All
drives are on the same SCSI bus (I did ask for the split backplane option
when I ordered it, but Dull said it couldn't be done, even though they
advertise it)

md6 : active raid6 sdf6[5] sde6[4] sdd6[3] sdc6[2] sdb6[1] sda6[0]
      394523648 blocks level 6, 64k chunk, algorithm 2 [6/6] [UUUUUU]

I was geting over 200MB/sec in & out of it when I configured it as a
RAID-0 when testing it.

And to compare, on a crappy old twin xeon/500MHz box with 8 external SCSI
drives:

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
bertha           1G           38659  69 18568  50           44925  73 562.2   8
bertha,1G,,,38659,69,18568,50,,,44925,73,562.2,8,,,,,,,,,,,,,

md0 : active raid6 sdl1[11] sdk1[8] sdj1[5] sdi1[2] sdh1[10] sdg1[7]
sdf1[4] sde1[1] sdd1[9] sdc1[6] sdb1[3] sda1[0]
      179203840 blocks level 6, 64k chunk, algorithm 2 [12/12] [UUUUUUUUUUUU]

There are split 4+4 over 2 SCSI buses, but the server only has one PCI
bus, so it all ends up going down the same path anyway.

So I'd say that with RAID-6 you can easilly produce something that will be
fully capable of saturating a 100Mb network, with capacity left over, but
you'll have to work a bit harder to saturate a Gb network.

Gordon

Re: First RAID Setup

[Bill Davidsen]

Brad Campbell wrote:

> Callahan, Tom wrote:
>
>> It is always wise to build in a spare however, that being said about all
>> raid levels. In your configuration, if a disk fails in your RAID5, your
>> array will go down. RAID5 is usually 3+ disks, with a mirror. So you 
>> should
>> have 3 disks at minimum, and then a 4th as a spare.
>>
>
> /me wonders in the days of reliable RAID-6 why we use RAID-5 + spare?
>
> RAID-6 has saved me twice now from dual drive failures on a 15 disk 
> array.
> It's schweeeeeeeeeett 


It's also a lot more overhead... RAID-5 needs to update just one parity 
block beyond the data written. As I understand the Q sum in RAID-6, and 
watching disk access rates, each write requires the entire stripe to be 
read, then P and Q calculated, then written. You can do the P with a 
read+write, but since you have to read the entire stripe for Q, you save 
a read by recalculating the P from data.

Did I say that right, Neil?

If you are seeing dual drive failures, I suspect your hardware has 
problems. We run multiple 3 and 6 TB databases, and over a dozen 1 TB 
data caching servers, all using a lot of small fast disk, and I haven't 
seen a real dual drive failure in  about 8 years.

We did see some cases which looked like dual failures, it turned out to 
be a firmware limitation, controller not waiting for the bus to settle 
after a real failure, and thinking the next i/o had failed (or similar, 
in any case a false fail on the transaction after the real fail). If you 
run two PATA drives on the same cable in master/slave, it's at least 
possible that this could happen with consumer grade hardware as well. 
Just a thought, dual failures are VERY unlikely unless one triggers the 
other in some way, like failing the bus or cabinet power supply.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: First RAID Setup

[Mattias Wadenstein]

On Thu, 22 Dec 2005, Bill Davidsen wrote:

> If you are seeing dual drive failures, I suspect your hardware has problems. 
> We run multiple 3 and 6 TB databases, and over a dozen 1 TB data caching 
> servers, all using a lot of small fast disk, and I haven't seen a real dual 
> drive failure in  about 8 years.
>
> We did see some cases which looked like dual failures, it turned out to be a 
> firmware limitation, controller not waiting for the bus to settle after a 
> real failure, and thinking the next i/o had failed (or similar, in any case a 
> false fail on the transaction after the real fail). If you run two PATA 
> drives on the same cable in master/slave, it's at least possible that this 
> could happen with consumer grade hardware as well. Just a thought, dual 
> failures are VERY unlikely unless one triggers the other in some way, like 
> failing the bus or cabinet power supply.

Not really, it depends on how lucky you are with your disks. We've had 
real dual-drive failures in a system with hot spares, where the second 
drive failed during resync.

Now, we have gotten the manufacturer to replace those with another model, 
but some of these problems don't occur until a year of two into production 
use (I haven't seen quite this bad, but a high replacement rate ramping up 
after a while).

Choosing high-end disks usually helps, but raid6 is really great in that 
you always have redundancy, even when replacing a failed or failing drive. 
If you have a large raidset with a fairly heavy load the resync time can 
easily extend into days, if not weeks. With raid5, during that entire 
period, if one more drive fails you're screwed.

Btw, in our practical usage, we haven't seen that big a difference 
between raid5 and rad6, but I guess that depends on your usage pattern.

/Mattias Wadenstein

First RAID Setup

[Andargor The Wise]

I admit it. I'm a RAID virgin.

However, after a disastrous failure of the sole drive
I wasn't backing up, I decided to go RAID-5 under
Slack 10.2 (first time ever with RAID-5).

The config:

Asus P5GL-MX (ICH6) mobo w/1 GB RAM, 4 x SATA ports
P4 3.0G/1M
3 x WD2000JS 200.0 GB SATA drives

First, a question: the BIOS on this machine seems to
list the SATA ports as "third/fourth IDE
master/slave". Further, the documentation seems to say
that SATA 1/2 are "master" and SATA 3/4 are "slave"
(black and red connectors, respectively).

My understanding is that SATA drives are each on
separate buses. Is this because the BIOS offers a
P-ATA emulation mode for SATA and it makes it "easier"
to understand for novices to show them that way?

I ask because people have said that it is not a good
idea to have both IDE masters and slaves on the same
bus as part of a RAID-5 array. I know SATA is
different, but will using three of the SATA ports on
this mobo be OK?

Second, after reading the excellent advice in this
list, I decided that booting from RAID-5 might not be
a good idea. So this is what I've been thinking:

Each disk partitioned alike:
	1	30MB 
	2	8GB (to allow for memory upgrades later)
	5	rest_of_disk

mds:
	md0	raid1 sda1 sdb1 sdc1
	md1	raid1 sda2 sdb2 sdc2
	md2	raid5 sda5 sdb5 sdc5

	md0	/boot
	md1	swap
	md2	/

Does this look OK? What should the stripe and chunk
sizes be, considering I'll be going with reiserfs?
Typical usage: development machine, some DB apps with
medium load, read-only mostly, not many writes. Very
few large files (such as multimedia).

Or should I set up separate RAID-5's for /usr and /var
as well?

Lastly, can I install directly to this configuration,
or should I install on a separate disk and move things
into the array?

Andargor


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com