Questions about VERY large software raid config

Questions about VERY large software raid config

[Rechenberg, Andrew]

Hello,

I am currently testing switching from Dell PERC hardware RAID cards to
Linux software RAID and I wanted to see if anyone out there has anything
this large or larger and can give me some tips.

Basically I'm going to do a bunch of RAID1 mirrors (56 drives), RAID)
those in sets of 14 and then stripe those 4 into one big array.  We want
to get the most spindles possible servicing disk requests and I wanted
to know if anyone has tried such a setup.

I have test hardware with 20 drives and it appears to behave nicely and
I'm getting much better throughput from software RAID than the Dell PERC
hardware RAID cards, but I wanted to make sure that this solution would
scale up to 56.  Looking at dmesg (MAX_MD_DEVS=256, MD_SB_DISKS=27) and
looking at the /dev directory (up to 128 SCSI disks) it looks like it
should, but I wanted to get some first hand experience, if there is any
:)

Thanks in advance.

Regards,
Andy.

Andrew Rechenberg
Infrastructure Team, Sherman Financial Group

Re: Questions about VERY large software raid config

[Mike Dresser]

On Fri, 21 Feb 2003, Rechenberg, Andrew wrote:

> Basically I'm going to do a bunch of RAID1 mirrors (56 drives), RAID)
> those in sets of 14 and then stripe those 4 into one big array.  We want
> to get the most spindles possible servicing disk requests and I wanted
> to know if anyone has tried such a setup.

How big are your drives?  There's apparently a 2 TiB limit on raid sizes,
there's even someone who has been asking if 12 x 160 or 13 x 160 is
ok(sorry, I don't have an answer for said person)

Mike

Re: Questions about VERY large software raid config

[Scott Mcdermott]

Rechenberg, Andrew on Fri 21/02 10:42 -0500:
> Basically I'm going to do a bunch of RAID1 mirrors (56 drives), RAID)
> those in sets of 14 and then stripe those 4 into one big array.

I'm curious, why aren't you doing a RAID0 stripe of 28 RAID1s instead?
It sounds like you are doing a RAID0 stripe of 4 RAID0 stripes of 14
RAID1s.  Why the extra step, it can only slow you down???

RE: Questions about VERY large software raid config

[Rechenberg, Andrew]

Isn't it 2TB per RAID device (i.e. Linux block device) or 2TB in total?


I have 28 36GB 15K drives and 28 18GB 15K drives.

36GB * 28 = 1008GB
18GB * 28 = 504GB

Either way, that's below the 2TB limit.  However, since I'm mirroring,
it will effectively be ~750GB.  

But I would like to know if there is a 2TB *total* limit for ALL RAID
devices (good information to know in advance :) ).

I'm going to be running LVM on top of the resulting md device for
snapshots as well.  I have a Red Hat kernel (2.4.18-24) so I assume that
the vfs-lock patch is already integrated since Red Hat offers LVM as an
option in 7.3.  I've tested snapshots on my small setup and it appears
to work just fine with ext3 on LVM on software RAID.

Thanks for the help,
Andy.  

-----Original Message-----
From: Mike Dresser [mailto:mdresser_l@windsormachine.com] 
Sent: Friday, February 21, 2003 10:50 AM
To: Rechenberg, Andrew
Cc: linux-raid@vger.kernel.org
Subject: Re: Questions about VERY large software raid config


On Fri, 21 Feb 2003, Rechenberg, Andrew wrote:

> Basically I'm going to do a bunch of RAID1 mirrors (56 drives), RAID)
> those in sets of 14 and then stripe those 4 into one big array.  We
want
> to get the most spindles possible servicing disk requests and I wanted
> to know if anyone has tried such a setup.

How big are your drives?  There's apparently a 2 TiB limit on raid
sizes,
there's even someone who has been asking if 12 x 160 or 13 x 160 is
ok(sorry, I don't have an answer for said person)

Mike

RE: Questions about VERY large software raid config

[Mike Dresser]

On Fri, 21 Feb 2003, Rechenberg, Andrew wrote:

> Isn't it 2TB per RAID device (i.e. Linux block device) or 2TB in total?
>
>
> I have 28 36GB 15K drives and 28 18GB 15K drives.

Sorry, I got the impression you were making one device of 56/2 = 28
drives in raid0
It should be per raid device, yes.

Re: Questions about VERY large software raid config

[Ross Vandegrift]

On Fri, Feb 21, 2003 at 11:03:27AM -0500, Rechenberg, Andrew wrote:
> Isn't it 2TB per RAID device (i.e. Linux block device) or 2TB in total?

Linux has a 2TB block device limit.  There's a patch floating around to
increase that, but I don't know very much more than that.  Someone
around, or google, should know more.

-- 
Ross Vandegrift
ross@willow.seitz.com

A Pope has a Water Cannon.                               It is a Water Cannon.
He fires Holy-Water from it.                        It is a Holy-Water Cannon.
He Blesses it.                                 It is a Holy Holy-Water Cannon.
He Blesses the Hell out of it.          It is a Wholly Holy Holy-Water Cannon.
He has it pierced.                It is a Holey Wholly Holy Holy-Water Cannon.
He makes it official.       It is a Canon Holey Wholly Holy Holy-Water Cannon.
Batman and Robin arrive.                                       He shoots them.

RE: Questions about VERY large software raid config

[C. R. Oldham]

> I have test hardware with 20 drives and it appears to behave
> nicely and I'm getting much better throughput from software
> RAID than the Dell PERC hardware

Have you tracked your CPU load?  That is one thing that has kept me from
going back to software RAID.


-- 
C. R. Oldham
Director of Technology
NCA CASI

RE: Questions about VERY large software raid config

[Rechenberg, Andrew]

According to sar and Orca, over the past 4 months the box is about 95%
idle most of the time so I don't think CPU load will be an issue.  The
box is a quad Xeon 1.4GHz with 8GB RAM.

-----Original Message-----
From: C. R. Oldham [mailto:cro@ncacasi.org] 
Sent: Friday, February 21, 2003 11:35 AM
To: Rechenberg, Andrew; Mike Dresser
Cc: linux-raid@vger.kernel.org
Subject: RE: Questions about VERY large software raid config


> I have test hardware with 20 drives and it appears to behave
> nicely and I'm getting much better throughput from software
> RAID than the Dell PERC hardware

Have you tracked your CPU load?  That is one thing that has kept me from
going back to software RAID.


-- 
C. R. Oldham
Director of Technology
NCA CASI