From mboxrd@z Thu Jan 1 00:00:00 1970 From: Raz Subject: How to configure 36 disks ? Date: Mon, 23 Mar 2009 15:59:47 +0200 Message-ID: <5d96567b0903230659t734677a3pb4fd77cccb54008b@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Return-path: Sender: owner-linux-aio@kvack.org To: linux-fsdevel@vger.kernel.org, Linux RAID Mailing List , linux-xfs@oss.sgi.com, linux-aio@kvack.org, "linux-ide@vger.kernel.org" List-Id: linux-ide@vger.kernel.org Hello I need to configure 3xDAS'es, each with 12 disks. All three DAS'es are connected to a single machine. I have the following requirements (in this order of importance) from the storage: 1. redundancy. having two disks failing in one raid5 breaks the entire raid. when you have 30TB storage it is a disaster. 2. performance. My code eliminates Linux raid5/6 write penalty. I managed to do by manipulating xfs and patching linux raid5 a bit. 3. modularity ( a "grow" and it will be nice to have "shrink" ) file system and volume must be able to grow. shrinking is possible by unifying multiple file systems under unionfs or aufs. 4. Utilize storage size. I assume each disk is 1TB. Solution #1 raid0 DAS1: raid5: D,D,D,D,D,D | raid5: D,D,D,D,D,D | | DAS2: raid5: D,D,D,D,D,D | xfs raid5: D,D,D,D,D,D | | DAS3: raid5: D,D,D,D,D,D | raid5: D,D,D,D,D,D | 1. redundancy. no. if a single raid fails, 30 TB fails. 2. performance. good. 3. modularity. no. raid0 does not grow. 4. Size. 30TB. Solution #2 raid0 DAS1: raid6: D,D,D,D,D,D | raid6: D,D,D,D,D,D | | DAS2: raid6: D,D,D,D,D,D | xfs. raid6: D,D,D,D,D,D | | DAS3: raid6: D,D,D,D,D,D | raid6: D,D,D,D,D,D | 1. redundancy. fair. less likely three disks will break in a single raid. 2. performance. good. 3. modularity. no. raid0 does not grow. 4. size. 24 TB Solution #3 unionfs/aufs DAS1: raid5: D,D,D,D,D,D xfs | raid5: D,D,D,D,D,D xfs | | DAS2: raid5: D,D,D,D,D,D xfs | raid5: D,D,D,D,D,D xfs | | DAS3: raid5: D,D,D,D,D,D xfs | raid5: D,D,D,D,D,D xfs | 1. redundancy. fair. if a single raid fails, only this raid fails. 2. performance. fair. unionfs is not mainline and does not support write balancing. aufs is not mature enough. 3. modularity. yes. grow and shrinks. 4. Size. 30TB. Solution #4 xfs over Linux LVM DAS1: raid6: D,D,D,D,D,D | raid6: D,D,D,D,D,D | | DAS2: raid6: D,D,D,D,D,D | raid6: D,D,D,D,D,D | | DAS3: raid6: D,D,D,D,D,D | raid6: D,D,D,D,D,D | 1. redundancy. fair. less likely three disks will break in a single raid 2. performance. bad. 3. modularity. yes. grows 4. Size 24TB Any other ideas ? -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to majordomo@kvack.org. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: aart@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 From: Emmanuel Florac Subject: Re: How to configure 36 disks ? Date: Mon, 23 Mar 2009 15:12:57 +0100 Message-ID: <20090323151257.1c7c0a8e@harpe.intellique.com> References: <5d96567b0903230659t734677a3pb4fd77cccb54008b@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <5d96567b0903230659t734677a3pb4fd77cccb54008b@mail.gmail.com> Sender: owner-linux-aio@kvack.org To: Raz Cc: linux-fsdevel@vger.kernel.org, Linux RAID Mailing List , linux-xfs@oss.sgi.com, linux-aio@kvack.org, "linux-ide@vger.kernel.org" List-Id: linux-ide@vger.kernel.org Le Mon, 23 Mar 2009 15:59:47 +0200 Raz =E9crivait: > 1. redundancy. fair. less likely three disks will break in a single > raid 2. performance. bad. > 3. modularity. yes. grows > 4. Size 24TB I don't understand why you'd have a bad performance here. I usually notice a very slight difference between LVM and linux-raid raid0 in all practical benchmarks. Did you use LVM striping capability? You may have to adjust readahead (using blockdev --setra) on each device /dev/dm-X and /dev/md-X) to achieve optimal performance. --=20 ---------------------------------------- Emmanuel Florac | Intellique ---------------------------------------- -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to majordomo@kvack.org. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: aart@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 From: Bill Davidsen Subject: Re: How to configure 36 disks ? Date: Mon, 23 Mar 2009 11:35:08 -0400 Message-ID: <49C7AC2C.9090307@tmr.com> References: <5d96567b0903230659t734677a3pb4fd77cccb54008b@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <5d96567b0903230659t734677a3pb4fd77cccb54008b@mail.gmail.com> Sender: linux-raid-owner@vger.kernel.org To: Raz Cc: linux-fsdevel@vger.kernel.org, Linux RAID Mailing List , linux-xfs@oss.sgi.com, linux-aio@kvack.org, "linux-ide@vger.kernel.org" List-Id: linux-ide@vger.kernel.org Raz wrote: > Hello > I need to configure 3xDAS'es, each with 12 disks. > All three DAS'es are connected to a single machine. > I have the following requirements (in this order of importance) > from the storage: > > 1. redundancy. > having two disks failing in one raid5 breaks the entire raid. when > you have 30TB storage > it is a disaster. > > 2. performance. > My code eliminates Linux raid5/6 write penalty. I managed to do by > manipulating xfs and patching linux raid5 a bit. > > 3. modularity ( a "grow" and it will be nice to have "shrink" ) > file system and volume must be able to grow. shrinking is possible > by unifying multiple file systems > under unionfs or aufs. > > 4. Utilize storage size. > > I assume each disk is 1TB. > > ___ snip ___ > Any other ideas ? Yes, you have the whole solution rotated 90 degrees. Consider your original solution #2 below... You have no redundancy if one whole DAS box fails, which is certainly a possible failure mode. If you put the RAID0 horizontally, two arrays size six in each DAS, then RAID6 vertically, if one DAS fails completely you still have a functioning system, and the failure results for individual drives remains about the same, while the rebuild time will be longer. Solution #2 raid0 DAS1: raid6: D,D,D,D,D,D | raid6: D,D,D,D,D,D | | DAS2: raid6: D,D,D,D,D,D | xfs. raid6: D,D,D,D,D,D | | DAS3: raid6: D,D,D,D,D,D | raid6: D,D,D,D,D,D | In addition, you can expand this configuration by adding more DAS units. This addresses several of your goals. In practice, just to get faster rebuild as the array gets larger, I suspect you would find it was worth making the horizontal arrays RAID5 instead of RAID0, just to minimize time to full performance. -- bill davidsen CTO TMR Associates, Inc "You are disgraced professional losers. And by the way, give us our money back." - Representative Earl Pomeroy, Democrat of North Dakota on the A.I.G. executives who were paid bonuses after a federal bailout. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jon Hardcastle Subject: Re: How to configure 36 disks ? Date: Mon, 23 Mar 2009 09:02:14 -0700 (PDT) Message-ID: <174852.24274.qm@web51309.mail.re2.yahoo.com> References: <49C7AC2C.9090307@tmr.com> Reply-To: Jon@eHardcastle.com Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <49C7AC2C.9090307@tmr.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: xfs-bounces@oss.sgi.com Errors-To: xfs-bounces@oss.sgi.com To: Raz , Bill Davidsen Cc: linux-fsdevel@vger.kernel.org, Linux RAID Mailing List , "linux-ide@vger.kernel.org" , linux-xfs@oss.sgi.com, linux-aio@kvack.org List-Id: linux-ide@vger.kernel.org I'd like to understand how you even go attaching that many devices to a system.. I am 'comparatively' new to this.. and have a 6 raid5 system.. not enterprise.. and i have slammed into case/power/sat slot issues already. What sort of hardware must one use to grow to a 36 array system! ----------------------- N: Jon Hardcastle E: Jon@eHardcastle.com '..Be fearful when others are greedy, and be greedy when others are fearful.' ----------------------- --- On Mon, 23/3/09, Bill Davidsen wrote: > From: Bill Davidsen > Subject: Re: How to configure 36 disks ? > To: "Raz" > Cc: linux-fsdevel@vger.kernel.org, "Linux RAID Mailing List" , linux-xfs@oss.sgi.com, linux-aio@kvack.org, "linux-ide@vger.kernel.org" > Date: Monday, 23 March, 2009, 3:35 PM > Raz wrote: > > Hello > > I need to configure 3xDAS'es, each with 12 disks. > > All three DAS'es are connected to a single > machine. > > I have the following requirements (in this order of > importance) > > from the storage: > > > > 1. redundancy. > > having two disks failing in one raid5 breaks the > entire raid. when > > you have 30TB storage > > it is a disaster. > > > > 2. performance. > > My code eliminates Linux raid5/6 write penalty. I > managed to do by > > manipulating xfs and patching linux raid5 a bit. > > > > 3. modularity ( a "grow" and it will be nice > to have "shrink" ) > > file system and volume must be able to grow. > shrinking is possible > > by unifying multiple file systems > > under unionfs or aufs. > > > > 4. Utilize storage size. > > > > I assume each disk is 1TB. > > > > > ___ snip ___ > > > Any other ideas ? > > Yes, you have the whole solution rotated 90 degrees. > Consider your original solution #2 below... You have no > redundancy if one whole DAS box fails, which is certainly a > possible failure mode. If you put the RAID0 horizontally, > two arrays size six in each DAS, then RAID6 vertically, if > one DAS fails completely you still have a functioning > system, and the failure results for individual drives > remains about the same, while the rebuild time will be > longer. > > Solution #2 > raid0 > DAS1: raid6: D,D,D,D,D,D | > raid6: D,D,D,D,D,D | > | > DAS2: raid6: D,D,D,D,D,D | xfs. > raid6: D,D,D,D,D,D | > | > DAS3: raid6: D,D,D,D,D,D | > raid6: D,D,D,D,D,D | > > > In addition, you can expand this configuration by adding > more DAS units. This addresses several of your goals. > > In practice, just to get faster rebuild as the array gets > larger, I suspect you would find it was worth making the > horizontal arrays RAID5 instead of RAID0, just to minimize > time to full performance. > > -- bill davidsen > CTO TMR Associates, Inc > > "You are disgraced professional losers. And by the > way, give us our money back." > - Representative Earl Pomeroy, Democrat of North Dakota > on the A.I.G. executives who were paid bonuses after a > federal bailout. > > > -- > To unsubscribe from this list: send the line > "unsubscribe linux-raid" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at > http://vger.kernel.org/majordomo-info.html _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mark Lord Subject: Re: How to configure 36 disks ? Date: Mon, 23 Mar 2009 12:22:08 -0400 Message-ID: <49C7B730.7030506@rtr.ca> References: <174852.24274.qm@web51309.mail.re2.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <174852.24274.qm@web51309.mail.re2.yahoo.com> Sender: owner-linux-aio@kvack.org To: Jon@eHardcastle.com Cc: Raz , Bill Davidsen , linux-fsdevel@vger.kernel.org, Linux RAID Mailing List , linux-xfs@oss.sgi.com, linux-aio@kvack.org, "linux-ide@vger.kernel.org" List-Id: linux-ide@vger.kernel.org Jon Hardcastle wrote: > I'd like to understand how you even go attaching that many devices to a system.. I am 'comparatively' new to this.. and have a 6 raid5 system.. not enterprise.. and i have slammed into case/power/sat slot issues already. What sort of hardware must one use to grow to a 36 array system! .. Here's one way: -- four onboard SATA ports -- plus four add-in PCI/PCIX/PCIe 8-port Marvell SATA cards. -- and a honkin' huge PSU! :) -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to majordomo@kvack.org. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: aart@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 From: Christopher Smith Subject: Re: How to configure 36 disks ? Date: Mon, 23 Mar 2009 09:23:20 -0700 Message-ID: <49C7B778.7000604@xman.org> References: <174852.24274.qm@web51309.mail.re2.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <174852.24274.qm@web51309.mail.re2.yahoo.com> Sender: owner-linux-aio@kvack.org To: Jon@eHardcastle.com Cc: Raz , Bill Davidsen , linux-fsdevel@vger.kernel.org, Linux RAID Mailing List , linux-xfs@oss.sgi.com, linux-aio@kvack.org, "linux-ide@vger.kernel.org" List-Id: linux-ide@vger.kernel.org Jon Hardcastle wrote: > I'd like to understand how you even go attaching that many devices to a system.. I am 'comparatively' new to this.. and have a 6 raid5 system.. not enterprise.. and i have slammed into case/power/sat slot issues already. What sort of hardware must one use to grow to a 36 array system! You can always use this approach: http://www.youtube.com/watch?v=96dWOEa4Djs --Chris P.S.: Sorry, couldn't resist. -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to majordomo@kvack.org. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: aart@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 From: Raz Subject: Re: How to configure 36 disks ? Date: Mon, 23 Mar 2009 18:28:07 +0200 Message-ID: <5d96567b0903230928p794e4140y9263f4095beb5b52@mail.gmail.com> References: <174852.24274.qm@web51309.mail.re2.yahoo.com> <49C7B778.7000604@xman.org> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <49C7B778.7000604@xman.org> Sender: owner-linux-aio@kvack.org To: Christopher Smith Cc: Jon@ehardcastle.com, Bill Davidsen , linux-fsdevel@vger.kernel.org, Linux RAID Mailing List , linux-xfs@oss.sgi.com, linux-aio@kvack.org, "linux-ide@vger.kernel.org" List-Id: linux-ide@vger.kernel.org On Mon, Mar 23, 2009 at 6:23 PM, Christopher Smith wrote: > Jon Hardcastle wrote: >> >> I'd like to understand how you even go attaching that many devices to a >> system.. I am 'comparatively' new to this.. and have a 6 raid5 system.. not >> enterprise.. and i have slammed into case/power/sat slot issues already. >> What sort of hardware must one use to grow to a 36 array system! > > You can always use this approach: > > http://www.youtube.com/watch?v=96dWOEa4Djs > > --Chris > > P.S.: Sorry, couldn't resist. > > me too. http://blogs.sun.com/observatory/entry/don_t_shout_at_your -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to majordomo@kvack.org. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: aart@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 From: Greg Freemyer Subject: Re: How to configure 36 disks ? Date: Mon, 23 Mar 2009 12:45:23 -0400 Message-ID: <87f94c370903230945k66a804a0iefbb76dd9c93fc16@mail.gmail.com> References: <49C7AC2C.9090307@tmr.com> <174852.24274.qm@web51309.mail.re2.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <174852.24274.qm@web51309.mail.re2.yahoo.com> Sender: owner-linux-aio@kvack.org To: Jon@ehardcastle.com Cc: Raz , Bill Davidsen , linux-fsdevel@vger.kernel.org, Linux RAID Mailing List , linux-xfs@oss.sgi.com, linux-aio@kvack.org, "linux-ide@vger.kernel.org" List-Id: linux-ide@vger.kernel.org On Mon, Mar 23, 2009 at 12:02 PM, Jon Hardcastle wrote: > > I'd like to understand how you even go attaching that many devices to a s= ystem.. I am 'comparatively' new to this.. and have a 6 raid5 system.. not = enterprise.. and i have slammed into case/power/sat slot issues already. Wh= at sort of hardware must one use to grow to a 36 array system! For the drives you could get a few big enclosures: http://www.pc-pitstop.com/sata_enclosures/sat122urd.asp http://www.pc-pitstop.com/sata_enclosures/scsas16rm.asp I don't think these offer PMP (port multiplexer) support. I would look for ones that did if you're serious about doing something like this. If you used PMP with 4x drives per sata port, you would only need 9 sata ports in the PC to control the 36 drives. Not sure what the limits of PMP is, but 4x seems reasonable to me. Greg --=20 Greg Freemyer Head of EDD Tape Extraction and Processing team Litigation Triage Solutions Specialist http://www.linkedin.com/in/gregfreemyer First 99 Days Litigation White Paper - http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf The Norcross Group The Intersection of Evidence & Technology http://www.norcrossgroup.com -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to majordomo@kvack.org. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: aart@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mikael Abrahamsson Subject: Re: How to configure 36 disks ? Date: Mon, 23 Mar 2009 19:32:28 +0100 (CET) Message-ID: References: <49C7AC2C.9090307@tmr.com> <174852.24274.qm@web51309.mail.re2.yahoo.com> <87f94c370903230945k66a804a0iefbb76dd9c93fc16@mail.gmail.com> Mime-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Return-path: In-Reply-To: <87f94c370903230945k66a804a0iefbb76dd9c93fc16@mail.gmail.com> Sender: owner-linux-aio@kvack.org To: Greg Freemyer Cc: Jon@ehardcastle.com, Raz , Bill Davidsen , linux-fsdevel@vger.kernel.org, Linux RAID Mailing List , linux-xfs@oss.sgi.com, linux-aio@kvack.org, "linux-ide@vger.kernel.org" List-Id: linux-ide@vger.kernel.org On Mon, 23 Mar 2009, Greg Freemyer wrote: > http://www.pc-pitstop.com/sata_enclosures/sat122urd.asp > http://www.pc-pitstop.com/sata_enclosures/scsas16rm.asp I would like to add this if you're on a budget: > sata ports in the PC to control the 36 drives. Not sure what the limits > of PMP is, but 4x seems reasonable to me. The PMPs I've seen take 5 drives each: -- Mikael Abrahamsson email: swmike@swm.pp.se -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to majordomo@kvack.org. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: aart@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 From: Goswin von Brederlow Subject: Re: How to configure 36 disks ? Date: Tue, 24 Mar 2009 20:38:35 +0100 Message-ID: <87hc1iemo4.fsf@frosties.localdomain> References: <174852.24274.qm@web51309.mail.re2.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: In-Reply-To: <174852.24274.qm@web51309.mail.re2.yahoo.com> (Jon Hardcastle's message of "Mon, 23 Mar 2009 09:02:14 -0700 (PDT)") Sender: owner-linux-aio@kvack.org To: Jon@eHardcastle.com Cc: Raz , Bill Davidsen , linux-fsdevel@vger.kernel.org, Linux RAID Mailing List , linux-xfs@oss.sgi.com, linux-aio@kvack.org, "linux-ide@vger.kernel.org" List-Id: linux-ide@vger.kernel.org Jon Hardcastle writes: > I'd like to understand how you even go attaching that many devices to a system.. I am 'comparatively' new to this.. and have a 6 raid5 system.. not enterprise.. and i have slammed into case/power/sat slot issues already. What sort of hardware must one use to grow to a 36 array system! Well, lets see. Put 3 dual channel SAS controler in the box giving you 6 external SAS connectors. Buy 6 48x disk enclosures and connect them. Configure them all as JBOD and you get your 288 disks. MfG Goswin -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to majordomo@kvack.org. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: aart@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 From: Drew Subject: Re: How to configure 36 disks ? Date: Wed, 25 Mar 2009 05:14:42 -0700 Message-ID: References: <174852.24274.qm@web51309.mail.re2.yahoo.com> <87hc1iemo4.fsf@frosties.localdomain> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <87hc1iemo4.fsf@frosties.localdomain> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: xfs-bounces@oss.sgi.com Errors-To: xfs-bounces@oss.sgi.com To: Goswin von Brederlow Cc: Jon@ehardcastle.com, Linux RAID Mailing List , linux-aio@kvack.org, linux-xfs@oss.sgi.com, "linux-ide@vger.kernel.org" , Raz , linux-fsdevel@vger.kernel.org, Bill Davidsen List-Id: linux-ide@vger.kernel.org >> I'd like to understand how you even go attaching that many devices to a system.. I am 'comparatively' >> new to this.. and have a 6 raid5 system.. not enterprise.. and i have slammed into case/power/sat slot >> issues already. What sort of hardware must one use to grow to a 36 array system! > > Well, lets see. > > Put 3 dual channel SAS controler in the box giving you 6 external SAS > connectors. Buy 6 48x disk enclosures and connect them. Configure them > all as JBOD and you get your 288 disks. And after you've mortgaged your house, your future, and your first born ... :-P -- Drew "Nothing in life is to be feared. It is only to be understood." --Marie Curie _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs