From mboxrd@z Thu Jan 1 00:00:00 1970 From: Raz Subject: How to configure 36 disks ? Date: Mon, 23 Mar 2009 15:59:47 +0200 Message-ID: <5d96567b0903230659t734677a3pb4fd77cccb54008b@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Return-path: Sender: owner-linux-aio@kvack.org To: linux-fsdevel@vger.kernel.org, Linux RAID Mailing List , linux-xfs@oss.sgi.com, linux-aio@kvack.org, "linux-ide@vger.kernel.org" List-Id: linux-ide@vger.kernel.org Hello I need to configure 3xDAS'es, each with 12 disks. All three DAS'es are connected to a single machine. I have the following requirements (in this order of importance) from the storage: 1. redundancy. having two disks failing in one raid5 breaks the entire raid. when you have 30TB storage it is a disaster. 2. performance. My code eliminates Linux raid5/6 write penalty. I managed to do by manipulating xfs and patching linux raid5 a bit. 3. modularity ( a "grow" and it will be nice to have "shrink" ) file system and volume must be able to grow. shrinking is possible by unifying multiple file systems under unionfs or aufs. 4. Utilize storage size. I assume each disk is 1TB. Solution #1 raid0 DAS1: raid5: D,D,D,D,D,D | raid5: D,D,D,D,D,D | | DAS2: raid5: D,D,D,D,D,D | xfs raid5: D,D,D,D,D,D | | DAS3: raid5: D,D,D,D,D,D | raid5: D,D,D,D,D,D | 1. redundancy. no. if a single raid fails, 30 TB fails. 2. performance. good. 3. modularity. no. raid0 does not grow. 4. Size. 30TB. Solution #2 raid0 DAS1: raid6: D,D,D,D,D,D | raid6: D,D,D,D,D,D | | DAS2: raid6: D,D,D,D,D,D | xfs. raid6: D,D,D,D,D,D | | DAS3: raid6: D,D,D,D,D,D | raid6: D,D,D,D,D,D | 1. redundancy. fair. less likely three disks will break in a single raid. 2. performance. good. 3. modularity. no. raid0 does not grow. 4. size. 24 TB Solution #3 unionfs/aufs DAS1: raid5: D,D,D,D,D,D xfs | raid5: D,D,D,D,D,D xfs | | DAS2: raid5: D,D,D,D,D,D xfs | raid5: D,D,D,D,D,D xfs | | DAS3: raid5: D,D,D,D,D,D xfs | raid5: D,D,D,D,D,D xfs | 1. redundancy. fair. if a single raid fails, only this raid fails. 2. performance. fair. unionfs is not mainline and does not support write balancing. aufs is not mature enough. 3. modularity. yes. grow and shrinks. 4. Size. 30TB. Solution #4 xfs over Linux LVM DAS1: raid6: D,D,D,D,D,D | raid6: D,D,D,D,D,D | | DAS2: raid6: D,D,D,D,D,D | raid6: D,D,D,D,D,D | | DAS3: raid6: D,D,D,D,D,D | raid6: D,D,D,D,D,D | 1. redundancy. fair. less likely three disks will break in a single raid 2. performance. bad. 3. modularity. yes. grows 4. Size 24TB Any other ideas ? -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to majordomo@kvack.org. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: aart@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 From: Emmanuel Florac Subject: Re: How to configure 36 disks ? Date: Mon, 23 Mar 2009 15:12:57 +0100 Message-ID: <20090323151257.1c7c0a8e@harpe.intellique.com> References: <5d96567b0903230659t734677a3pb4fd77cccb54008b@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <5d96567b0903230659t734677a3pb4fd77cccb54008b@mail.gmail.com> Sender: owner-linux-aio@kvack.org To: Raz Cc: linux-fsdevel@vger.kernel.org, Linux RAID Mailing List , linux-xfs@oss.sgi.com, linux-aio@kvack.org, "linux-ide@vger.kernel.org" List-Id: linux-ide@vger.kernel.org Le Mon, 23 Mar 2009 15:59:47 +0200 Raz =E9crivait: > 1. redundancy. fair. less likely three disks will break in a single > raid 2. performance. bad. > 3. modularity. yes. grows > 4. Size 24TB I don't understand why you'd have a bad performance here. I usually notice a very slight difference between LVM and linux-raid raid0 in all practical benchmarks. Did you use LVM striping capability? You may have to adjust readahead (using blockdev --setra) on each device /dev/dm-X and /dev/md-X) to achieve optimal performance. --=20 ---------------------------------------- Emmanuel Florac | Intellique ---------------------------------------- -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to majordomo@kvack.org. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: aart@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 From: Bill Davidsen Subject: Re: How to configure 36 disks ? Date: Mon, 23 Mar 2009 11:35:08 -0400 Message-ID: <49C7AC2C.9090307@tmr.com> References: <5d96567b0903230659t734677a3pb4fd77cccb54008b@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <5d96567b0903230659t734677a3pb4fd77cccb54008b@mail.gmail.com> Sender: linux-raid-owner@vger.kernel.org To: Raz Cc: linux-fsdevel@vger.kernel.org, Linux RAID Mailing List , linux-xfs@oss.sgi.com, linux-aio@kvack.org, "linux-ide@vger.kernel.org" List-Id: linux-ide@vger.kernel.org Raz wrote: > Hello > I need to configure 3xDAS'es, each with 12 disks. > All three DAS'es are connected to a single machine. > I have the following requirements (in this order of importance) > from the storage: > > 1. redundancy. > having two disks failing in one raid5 breaks the entire raid. when > you have 30TB storage > it is a disaster. > > 2. performance. > My code eliminates Linux raid5/6 write penalty. I managed to do by > manipulating xfs and patching linux raid5 a bit. > > 3. modularity ( a "grow" and it will be nice to have "shrink" ) > file system and volume must be able to grow. shrinking is possible > by unifying multiple file systems > under unionfs or aufs. > > 4. Utilize storage size. > > I assume each disk is 1TB. > > ___ snip ___ > Any other ideas ? Yes, you have the whole solution rotated 90 degrees. Consider your original solution #2 below... You have no redundancy if one whole DAS box fails, which is certainly a possible failure mode. If you put the RAID0 horizontally, two arrays size six in each DAS, then RAID6 vertically, if one DAS fails completely you still have a functioning system, and the failure results for individual drives remains about the same, while the rebuild time will be longer. Solution #2 raid0 DAS1: raid6: D,D,D,D,D,D | raid6: D,D,D,D,D,D | | DAS2: raid6: D,D,D,D,D,D | xfs. raid6: D,D,D,D,D,D | | DAS3: raid6: D,D,D,D,D,D | raid6: D,D,D,D,D,D | In addition, you can expand this configuration by adding more DAS units. This addresses several of your goals. In practice, just to get faster rebuild as the array gets larger, I suspect you would find it was worth making the horizontal arrays RAID5 instead of RAID0, just to minimize time to full performance. -- bill davidsen CTO TMR Associates, Inc "You are disgraced professional losers. And by the way, give us our money back." - Representative Earl Pomeroy, Democrat of North Dakota on the A.I.G. executives who were paid bonuses after a federal bailout. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jon Hardcastle Subject: Re: How to configure 36 disks ? Date: Mon, 23 Mar 2009 09:02:14 -0700 (PDT) Message-ID: <174852.24274.qm@web51309.mail.re2.yahoo.com> References: <49C7AC2C.9090307@tmr.com> Reply-To: Jon@eHardcastle.com Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <49C7AC2C.9090307@tmr.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: xfs-bounces@oss.sgi.com Errors-To: xfs-bounces@oss.sgi.com To: Raz , Bill Davidsen Cc: linux-fsdevel@vger.kernel.org, Linux RAID Mailing List , "linux-ide@vger.kernel.org" , linux-xfs@oss.sgi.com, linux-aio@kvack.org List-Id: linux-ide@vger.kernel.org I'd like to understand how you even go attaching that many devices to a system.. I am 'comparatively' new to this.. and have a 6 raid5 system.. not enterprise.. and i have slammed into case/power/sat slot issues already. What sort of hardware must one use to grow to a 36 array system! ----------------------- N: Jon Hardcastle E: Jon@eHardcastle.com '..Be fearful when others are greedy, and be greedy when others are fearful.' ----------------------- --- On Mon, 23/3/09, Bill Davidsen wrote: > From: Bill Davidsen > Subject: Re: How to configure 36 disks ? > To: "Raz" > Cc: linux-fsdevel@vger.kernel.org, "Linux RAID Mailing List" , linux-xfs@oss.sgi.com, linux-aio@kvack.org, "linux-ide@vger.kernel.org" > Date: Monday, 23 March, 2009, 3:35 PM > Raz wrote: > > Hello > > I need to configure 3xDAS'es, each with 12 disks. > > All three DAS'es are connected to a single > machine. > > I have the following requirements (in this order of > importance) > > from the storage: > > > > 1. redundancy. > > having two disks failing in one raid5 breaks the > entire raid. when > > you have 30TB storage > > it is a disaster. > > > > 2. performance. > > My code eliminates Linux raid5/6 write penalty. I > managed to do by > > manipulating xfs and patching linux raid5 a bit. > > > > 3. modularity ( a "grow" and it will be nice > to have "shrink" ) > > file system and volume must be able to grow. > shrinking is possible > > by unifying multiple file systems > > under unionfs or aufs. > > > > 4. Utilize storage size. > > > > I assume each disk is 1TB. > > > > > ___ snip ___ > > > Any other ideas ? > > Yes, you have the whole solution rotated 90 degrees. > Consider your original solution #2 below... You have no > redundancy if one whole DAS box fails, which is certainly a > possible failure mode. If you put the RAID0 horizontally, > two arrays size six in each DAS, then RAID6 vertically, if > one DAS fails completely you still have a functioning > system, and the failure results for individual drives > remains about the same, while the rebuild time will be > longer. > > Solution #2 > raid0 > DAS1: raid6: D,D,D,D,D,D | > raid6: D,D,D,D,D,D | > | > DAS2: raid6: D,D,D,D,D,D | xfs. > raid6: D,D,D,D,D,D | > | > DAS3: raid6: D,D,D,D,D,D | > raid6: D,D,D,D,D,D | > > > In addition, you can expand this configuration by adding > more DAS units. This addresses several of your goals. > > In practice, just to get faster rebuild as the array gets > larger, I suspect you would find it was worth making the > horizontal arrays RAID5 instead of RAID0, just to minimize > time to full performance. > > -- bill davidsen > CTO TMR Associates, Inc > > "You are disgraced professional losers. And by the > way, give us our money back." > - Representative Earl Pomeroy, Democrat of North Dakota > on the A.I.G. executives who were paid bonuses after a > federal bailout. > > > -- > To unsubscribe from this list: send the line > "unsubscribe linux-raid" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at > http://vger.kernel.org/majordomo-info.html _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mark Lord Subject: Re: How to configure 36 disks ? Date: Mon, 23 Mar 2009 12:22:08 -0400 Message-ID: <49C7B730.7030506@rtr.ca> References: <174852.24274.qm@web51309.mail.re2.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <174852.24274.qm@web51309.mail.re2.yahoo.com> Sender: owner-linux-aio@kvack.org To: Jon@eHardcastle.com Cc: Raz , Bill Davidsen , linux-fsdevel@vger.kernel.org, Linux RAID Mailing List , linux-xfs@oss.sgi.com, linux-aio@kvack.org, "linux-ide@vger.kernel.org" List-Id: linux-ide@vger.kernel.org Jon Hardcastle wrote: > I'd like to understand how you even go attaching that many devices to a system.. I am 'comparatively' new to this.. and have a 6 raid5 system.. not enterprise.. and i have slammed into case/power/sat slot issues already. What sort of hardware must one use to grow to a 36 array system! .. Here's one way: -- four onboard SATA ports -- plus four add-in PCI/PCIX/PCIe 8-port Marvell SATA cards. -- and a honkin' huge PSU! :) -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to majordomo@kvack.org. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: aart@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 From: Christopher Smith Subject: Re: How to configure 36 disks ? Date: Mon, 23 Mar 2009 09:23:20 -0700 Message-ID: <49C7B778.7000604@xman.org> References: <174852.24274.qm@web51309.mail.re2.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <174852.24274.qm@web51309.mail.re2.yahoo.com> Sender: owner-linux-aio@kvack.org To: Jon@eHardcastle.com Cc: Raz , Bill Davidsen , linux-fsdevel@vger.kernel.org, Linux RAID Mailing List , linux-xfs@oss.sgi.com, linux-aio@kvack.org, "linux-ide@vger.kernel.org" List-Id: linux-ide@vger.kernel.org Jon Hardcastle wrote: > I'd like to understand how you even go attaching that many devices to a system.. I am 'comparatively' new to this.. and have a 6 raid5 system.. not enterprise.. and i have slammed into case/power/sat slot issues already. What sort of hardware must one use to grow to a 36 array system! You can always use this approach: http://www.youtube.com/watch?v=96dWOEa4Djs --Chris P.S.: Sorry, couldn't resist. -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to majordomo@kvack.org. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: aart@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 From: Raz Subject: Re: How to configure 36 disks ? Date: Mon, 23 Mar 2009 18:28:07 +0200 Message-ID: <5d96567b0903230928p794e4140y9263f4095beb5b52@mail.gmail.com> References: <174852.24274.qm@web51309.mail.re2.yahoo.com> <49C7B778.7000604@xman.org> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <49C7B778.7000604@xman.org> Sender: owner-linux-aio@kvack.org To: Christopher Smith Cc: Jon@ehardcastle.com, Bill Davidsen , linux-fsdevel@vger.kernel.org, Linux RAID Mailing List , linux-xfs@oss.sgi.com, linux-aio@kvack.org, "linux-ide@vger.kernel.org" List-Id: linux-ide@vger.kernel.org On Mon, Mar 23, 2009 at 6:23 PM, Christopher Smith wrote: > Jon Hardcastle wrote: >> >> I'd like to understand how you even go attaching that many devices to a >> system.. I am 'comparatively' new to this.. and have a 6 raid5 system.. not >> enterprise.. and i have slammed into case/power/sat slot issues already. >> What sort of hardware must one use to grow to a 36 array system! > > You can always use this approach: > > http://www.youtube.com/watch?v=96dWOEa4Djs > > --Chris > > P.S.: Sorry, couldn't resist. > > me too. http://blogs.sun.com/observatory/entry/don_t_shout_at_your -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to majordomo@kvack.org. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: aart@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 From: Greg Freemyer Subject: Re: How to configure 36 disks ? Date: Mon, 23 Mar 2009 12:45:23 -0400 Message-ID: <87f94c370903230945k66a804a0iefbb76dd9c93fc16@mail.gmail.com> References: <49C7AC2C.9090307@tmr.com> <174852.24274.qm@web51309.mail.re2.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <174852.24274.qm@web51309.mail.re2.yahoo.com> Sender: owner-linux-aio@kvack.org To: Jon@ehardcastle.com Cc: Raz , Bill Davidsen , linux-fsdevel@vger.kernel.org, Linux RAID Mailing List , linux-xfs@oss.sgi.com, linux-aio@kvack.org, "linux-ide@vger.kernel.org" List-Id: linux-ide@vger.kernel.org On Mon, Mar 23, 2009 at 12:02 PM, Jon Hardcastle wrote: > > I'd like to understand how you even go attaching that many devices to a s= ystem.. I am 'comparatively' new to this.. and have a 6 raid5 system.. not = enterprise.. and i have slammed into case/power/sat slot issues already. Wh= at sort of hardware must one use to grow to a 36 array system! For the drives you could get a few big enclosures: http://www.pc-pitstop.com/sata_enclosures/sat122urd.asp http://www.pc-pitstop.com/sata_enclosures/scsas16rm.asp I don't think these offer PMP (port multiplexer) support. I would look for ones that did if you're serious about doing something like this. If you used PMP with 4x drives per sata port, you would only need 9 sata ports in the PC to control the 36 drives. Not sure what the limits of PMP is, but 4x seems reasonable to me. Greg --=20 Greg Freemyer Head of EDD Tape Extraction and Processing team Litigation Triage Solutions Specialist http://www.linkedin.com/in/gregfreemyer First 99 Days Litigation White Paper - http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf The Norcross Group The Intersection of Evidence & Technology http://www.norcrossgroup.com -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to majordomo@kvack.org. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: aart@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mikael Abrahamsson Subject: Re: How to configure 36 disks ? Date: Mon, 23 Mar 2009 19:32:28 +0100 (CET) Message-ID: References: <49C7AC2C.9090307@tmr.com> <174852.24274.qm@web51309.mail.re2.yahoo.com> <87f94c370903230945k66a804a0iefbb76dd9c93fc16@mail.gmail.com> Mime-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Return-path: In-Reply-To: <87f94c370903230945k66a804a0iefbb76dd9c93fc16@mail.gmail.com> Sender: owner-linux-aio@kvack.org To: Greg Freemyer Cc: Jon@ehardcastle.com, Raz , Bill Davidsen , linux-fsdevel@vger.kernel.org, Linux RAID Mailing List , linux-xfs@oss.sgi.com, linux-aio@kvack.org, "linux-ide@vger.kernel.org" List-Id: linux-ide@vger.kernel.org On Mon, 23 Mar 2009, Greg Freemyer wrote: > http://www.pc-pitstop.com/sata_enclosures/sat122urd.asp > http://www.pc-pitstop.com/sata_enclosures/scsas16rm.asp I would like to add this if you're on a budget: > sata ports in the PC to control the 36 drives. Not sure what the limits > of PMP is, but 4x seems reasonable to me. The PMPs I've seen take 5 drives each: -- Mikael Abrahamsson email: swmike@swm.pp.se -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to majordomo@kvack.org. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: aart@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 From: Goswin von Brederlow Subject: Re: How to configure 36 disks ? Date: Tue, 24 Mar 2009 20:38:35 +0100 Message-ID: <87hc1iemo4.fsf@frosties.localdomain> References: <174852.24274.qm@web51309.mail.re2.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: In-Reply-To: <174852.24274.qm@web51309.mail.re2.yahoo.com> (Jon Hardcastle's message of "Mon, 23 Mar 2009 09:02:14 -0700 (PDT)") Sender: owner-linux-aio@kvack.org To: Jon@eHardcastle.com Cc: Raz , Bill Davidsen , linux-fsdevel@vger.kernel.org, Linux RAID Mailing List , linux-xfs@oss.sgi.com, linux-aio@kvack.org, "linux-ide@vger.kernel.org" List-Id: linux-ide@vger.kernel.org Jon Hardcastle writes: > I'd like to understand how you even go attaching that many devices to a system.. I am 'comparatively' new to this.. and have a 6 raid5 system.. not enterprise.. and i have slammed into case/power/sat slot issues already. What sort of hardware must one use to grow to a 36 array system! Well, lets see. Put 3 dual channel SAS controler in the box giving you 6 external SAS connectors. Buy 6 48x disk enclosures and connect them. Configure them all as JBOD and you get your 288 disks. MfG Goswin -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to majordomo@kvack.org. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: aart@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 From: Drew Subject: Re: How to configure 36 disks ? Date: Wed, 25 Mar 2009 05:14:42 -0700 Message-ID: References: <174852.24274.qm@web51309.mail.re2.yahoo.com> <87hc1iemo4.fsf@frosties.localdomain> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <87hc1iemo4.fsf@frosties.localdomain> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: xfs-bounces@oss.sgi.com Errors-To: xfs-bounces@oss.sgi.com To: Goswin von Brederlow Cc: Jon@ehardcastle.com, Linux RAID Mailing List , linux-aio@kvack.org, linux-xfs@oss.sgi.com, "linux-ide@vger.kernel.org" , Raz , linux-fsdevel@vger.kernel.org, Bill Davidsen List-Id: linux-ide@vger.kernel.org >> I'd like to understand how you even go attaching that many devices to a system.. I am 'comparatively' >> new to this.. and have a 6 raid5 system.. not enterprise.. and i have slammed into case/power/sat slot >> issues already. What sort of hardware must one use to grow to a 36 array system! > > Well, lets see. > > Put 3 dual channel SAS controler in the box giving you 6 external SAS > connectors. Buy 6 48x disk enclosures and connect them. Configure them > all as JBOD and you get your 288 disks. And after you've mortgaged your house, your future, and your first born ... :-P -- Drew "Nothing in life is to be feared. It is only to be understood." --Marie Curie _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs From mboxrd@z Thu Jan 1 00:00:00 1970 From: Raz Subject: Re: How to configure 36 disks ? Date: Mon, 23 Mar 2009 18:32:46 +0200 Message-ID: <5d96567b0903230932j5bf11d88yee16be70759ef73b@mail.gmail.com> References: <5d96567b0903230659t734677a3pb4fd77cccb54008b@mail.gmail.com> <20090323151257.1c7c0a8e@harpe.intellique.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: QUOTED-PRINTABLE Return-path: In-Reply-To: <20090323151257.1c7c0a8e@harpe.intellique.com> Sender: linux-raid-owner@vger.kernel.org To: Emmanuel Florac Cc: Linux RAID Mailing List List-Id: linux-raid.ids On Mon, Mar 23, 2009 at 4:12 PM, Emmanuel Florac wrote: > Le Mon, 23 Mar 2009 15:59:47 +0200 > Raz =E9crivait: > >> 1. redundancy. fair. less likely three disks will break in a single >> raid 2. performance. bad. >> 3. modularity. yes. grows >> 4. Size 24TB > > I don't understand why you'd have a bad performance here. I usually > notice a very slight difference between LVM and linux-raid raid0 in a= ll > practical benchmarks. Did you use LVM striping capability? You may > have to adjust readahead (using blockdev --setra) on each > device /dev/dm-X and /dev/md-X) to achieve optimal performance. Problem i have with performance is that once you *add* to a active volume a new PE, you lose striping in the sense that you cannot tell where the IO is directed to ( which disk). My implementation relies on this fact. If you know I am wrong please correct me. > > -- > ---------------------------------------- > Emmanuel Florac =A0 =A0 | =A0 Intellique > ---------------------------------------- > > -- To unsubscribe from this list: send the line "unsubscribe linux-raid" i= n the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html From mboxrd@z Thu Jan 1 00:00:00 1970 From: Raz Subject: Re: How to configure 36 disks ? Date: Mon, 23 Mar 2009 18:52:51 +0200 Message-ID: <5d96567b0903230952h1929e208se550eff25eac864a@mail.gmail.com> References: <49C7AC2C.9090307@tmr.com> <174852.24274.qm@web51309.mail.re2.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: QUOTED-PRINTABLE Return-path: In-Reply-To: <174852.24274.qm@web51309.mail.re2.yahoo.com> Sender: linux-raid-owner@vger.kernel.org To: Jon@ehardcastle.com Cc: Linux RAID Mailing List List-Id: linux-raid.ids On Mon, Mar 23, 2009 at 6:02 PM, Jon Hardcastle wrote: > > I'd like to understand how you even go attaching that many devices to= a system.. I am 'comparatively' new to this.. and have a 6 raid5 syste= m.. not enterprise.. and i have slammed into case/power/sat slot issues= already. What sort of hardware must one use to grow to a 36 array syst= em! well, i am using a DAS in a JBOD mode. Cards are LSI logic Fiber channe= l. > ----------------------- > N: Jon Hardcastle > E: Jon@eHardcastle.com > '..Be fearful when others are greedy, and be greedy when others are f= earful.' > ----------------------- > > > --- On Mon, 23/3/09, Bill Davidsen wrote: > >> From: Bill Davidsen >> Subject: Re: How to configure 36 disks ? >> To: "Raz" >> Cc: linux-fsdevel@vger.kernel.org, "Linux RAID Mailing List" , linux-xfs@oss.sgi.com, linux-aio@kvack.org, "lin= ux-ide@vger.kernel.org" >> Date: Monday, 23 March, 2009, 3:35 PM >> Raz wrote: >> > Hello >> > I need to configure 3xDAS'es, each with 12 disks. >> > All three DAS'es are connected to a single >> machine. >> > I have the following requirements (in this order of >> importance) >> > from the storage: >> > >> > 1. redundancy. >> > =A0 =A0having two disks failing in one raid5 breaks the >> entire raid. when >> > you have 30TB storage >> > =A0 =A0it is a disaster. >> > >> > 2. performance. >> > =A0 =A0My code eliminates Linux raid5/6 write penalty. I >> managed to do by >> > =A0 =A0manipulating xfs and patching linux raid5 a bit. >> > >> > 3. modularity ( a "grow" and it will be nice >> to have "shrink" ) >> > =A0 =A0file system and volume must be able to grow. >> shrinking is possible >> > by unifying multiple file systems >> > =A0 =A0under unionfs or aufs. >> > >> > 4. Utilize storage size. >> > >> > I assume each disk is 1TB. >> > >> > >> ___ snip ___ >> >> > Any other ideas ? >> >> Yes, you have the whole solution rotated 90 degrees. >> Consider your original solution #2 below... You have no >> redundancy if one whole DAS box fails, which is certainly a >> possible failure mode. If you put the RAID0 horizontally, >> two arrays size six in each DAS, then RAID6 vertically, if >> one DAS fails completely you still have a functioning >> system, and the failure results for individual drives >> remains about the same, while the rebuild time will be >> longer. >> >> Solution #2 >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0raid0 >> DAS1: raid6: D,D,D,D,D,D =A0 =A0 | >> =A0 =A0 =A0 raid6: D,D,D,D,D,D =A0 =A0 | >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 | >> DAS2: raid6: D,D,D,D,D,D =A0 =A0 | xfs. >> =A0 =A0 =A0 raid6: D,D,D,D,D,D =A0 =A0 | >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0| >> DAS3: raid6: D,D,D,D,D,D =A0 =A0 | >> =A0 =A0 =A0 raid6: D,D,D,D,D,D =A0 =A0 | >> >> >> In addition, you can expand this configuration by adding >> more DAS units. This addresses several of your goals. >> >> In practice, just to get faster rebuild as the array gets >> larger, I suspect you would find it was worth making the >> horizontal arrays RAID5 instead of RAID0, just to minimize >> time to full performance. >> >> -- bill davidsen >> =A0CTO TMR Associates, Inc >> >> "You are disgraced professional losers. And by the >> way, give us our money back." >> =A0 =A0- Representative Earl Pomeroy, =A0Democrat of North Dakota >> on the A.I.G. executives who were paid bonuses =A0after a >> federal bailout. >> >> >> -- >> To unsubscribe from this list: send the line >> "unsubscribe linux-raid" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at >> http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe linux-raid" i= n the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html From mboxrd@z Thu Jan 1 00:00:00 1970 From: Richard Scobie Subject: Re: How to configure 36 disks ? Date: Tue, 24 Mar 2009 07:33:49 +1300 Message-ID: <49C7D60D.6090901@sauce.co.nz> References: <174852.24274.qm@web51309.mail.re2.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <174852.24274.qm@web51309.mail.re2.yahoo.com> Sender: linux-raid-owner@vger.kernel.org To: Jon@eHardcastle.com Cc: Linux RAID Mailing List List-Id: linux-raid.ids Jon Hardcastle wrote: > I'd like to understand how you even go attaching that many devices to a system.. I am 'comparatively' new to this.. and have a 6 raid5 system.. not enterprise.. and i have slammed into case/power/sat slot issues already. What sort of hardware must one use to grow to a 36 array system! A reasonably economical option is a SAS controller like the LSISAS3442E-R or LSISAS3801E in conjunction with enclosures like these that can be daisychained together. http://www.aicipc.com/ProductList.aspx?ref=RSC and http://www.aicipc.com/ProductList.aspx?ref=XJ Regards, Richard From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by oss.sgi.com (8.14.3/8.14.3/SuSE Linux 0.8) with ESMTP id n2NE0RxA159505 for ; Mon, 23 Mar 2009 09:00:47 -0500 Received: from mail-bw0-f226.google.com (localhost [127.0.0.1]) by cuda.sgi.com (Spam Firewall) with ESMTP id 04AE71B87AB for ; Mon, 23 Mar 2009 06:59:48 -0700 (PDT) Received: from mail-bw0-f226.google.com (mail-bw0-f226.google.com [209.85.218.226]) by cuda.sgi.com with ESMTP id 5aZZ6QbWonKkSFrp for ; Mon, 23 Mar 2009 06:59:48 -0700 (PDT) Received: by bwz26 with SMTP id 26so1776595bwz.20 for ; Mon, 23 Mar 2009 06:59:47 -0700 (PDT) MIME-Version: 1.0 Date: Mon, 23 Mar 2009 15:59:47 +0200 Message-ID: <5d96567b0903230659t734677a3pb4fd77cccb54008b@mail.gmail.com> Subject: How to configure 36 disks ? From: Raz List-Id: XFS Filesystem from SGI List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: xfs-bounces@oss.sgi.com Errors-To: xfs-bounces@oss.sgi.com To: linux-fsdevel@vger.kernel.org, Linux RAID Mailing List , linux-xfs@oss.sgi.com, linux-aio@kvack.org, "linux-ide@vger.kernel.org" Hello I need to configure 3xDAS'es, each with 12 disks. All three DAS'es are connected to a single machine. I have the following requirements (in this order of importance) from the storage: 1. redundancy. having two disks failing in one raid5 breaks the entire raid. when you have 30TB storage it is a disaster. 2. performance. My code eliminates Linux raid5/6 write penalty. I managed to do by manipulating xfs and patching linux raid5 a bit. 3. modularity ( a "grow" and it will be nice to have "shrink" ) file system and volume must be able to grow. shrinking is possible by unifying multiple file systems under unionfs or aufs. 4. Utilize storage size. I assume each disk is 1TB. Solution #1 raid0 DAS1: raid5: D,D,D,D,D,D | raid5: D,D,D,D,D,D | | DAS2: raid5: D,D,D,D,D,D | xfs raid5: D,D,D,D,D,D | | DAS3: raid5: D,D,D,D,D,D | raid5: D,D,D,D,D,D | 1. redundancy. no. if a single raid fails, 30 TB fails. 2. performance. good. 3. modularity. no. raid0 does not grow. 4. Size. 30TB. Solution #2 raid0 DAS1: raid6: D,D,D,D,D,D | raid6: D,D,D,D,D,D | | DAS2: raid6: D,D,D,D,D,D | xfs. raid6: D,D,D,D,D,D | | DAS3: raid6: D,D,D,D,D,D | raid6: D,D,D,D,D,D | 1. redundancy. fair. less likely three disks will break in a single raid. 2. performance. good. 3. modularity. no. raid0 does not grow. 4. size. 24 TB Solution #3 unionfs/aufs DAS1: raid5: D,D,D,D,D,D xfs | raid5: D,D,D,D,D,D xfs | | DAS2: raid5: D,D,D,D,D,D xfs | raid5: D,D,D,D,D,D xfs | | DAS3: raid5: D,D,D,D,D,D xfs | raid5: D,D,D,D,D,D xfs | 1. redundancy. fair. if a single raid fails, only this raid fails. 2. performance. fair. unionfs is not mainline and does not support write balancing. aufs is not mature enough. 3. modularity. yes. grow and shrinks. 4. Size. 30TB. Solution #4 xfs over Linux LVM DAS1: raid6: D,D,D,D,D,D | raid6: D,D,D,D,D,D | | DAS2: raid6: D,D,D,D,D,D | raid6: D,D,D,D,D,D | | DAS3: raid6: D,D,D,D,D,D | raid6: D,D,D,D,D,D | 1. redundancy. fair. less likely three disks will break in a single raid 2. performance. bad. 3. modularity. yes. grows 4. Size 24TB Any other ideas ? _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by oss.sgi.com (8.14.3/8.14.3/SuSE Linux 0.8) with ESMTP id n2NFa1vo166268 for ; Mon, 23 Mar 2009 10:36:22 -0500 Received: from partygirl.tmr.com (localhost [127.0.0.1]) by cuda.sgi.com (Spam Firewall) with ESMTP id 991CE13BEEAD for ; Mon, 23 Mar 2009 08:35:27 -0700 (PDT) Received: from partygirl.tmr.com (mail.tmr.com [64.65.253.246]) by cuda.sgi.com with ESMTP id ohkuR7nCDbmT4D2G for ; Mon, 23 Mar 2009 08:35:27 -0700 (PDT) Message-ID: <49C7AC2C.9090307@tmr.com> Date: Mon, 23 Mar 2009 11:35:08 -0400 From: Bill Davidsen MIME-Version: 1.0 Subject: Re: How to configure 36 disks ? References: <5d96567b0903230659t734677a3pb4fd77cccb54008b@mail.gmail.com> In-Reply-To: <5d96567b0903230659t734677a3pb4fd77cccb54008b@mail.gmail.com> List-Id: XFS Filesystem from SGI List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: xfs-bounces@oss.sgi.com Errors-To: xfs-bounces@oss.sgi.com To: Raz Cc: linux-fsdevel@vger.kernel.org, Linux RAID Mailing List , "linux-ide@vger.kernel.org" , linux-xfs@oss.sgi.com, linux-aio@kvack.org Raz wrote: > Hello > I need to configure 3xDAS'es, each with 12 disks. > All three DAS'es are connected to a single machine. > I have the following requirements (in this order of importance) > from the storage: > > 1. redundancy. > having two disks failing in one raid5 breaks the entire raid. when > you have 30TB storage > it is a disaster. > > 2. performance. > My code eliminates Linux raid5/6 write penalty. I managed to do by > manipulating xfs and patching linux raid5 a bit. > > 3. modularity ( a "grow" and it will be nice to have "shrink" ) > file system and volume must be able to grow. shrinking is possible > by unifying multiple file systems > under unionfs or aufs. > > 4. Utilize storage size. > > I assume each disk is 1TB. > > ___ snip ___ > Any other ideas ? Yes, you have the whole solution rotated 90 degrees. Consider your original solution #2 below... You have no redundancy if one whole DAS box fails, which is certainly a possible failure mode. If you put the RAID0 horizontally, two arrays size six in each DAS, then RAID6 vertically, if one DAS fails completely you still have a functioning system, and the failure results for individual drives remains about the same, while the rebuild time will be longer. Solution #2 raid0 DAS1: raid6: D,D,D,D,D,D | raid6: D,D,D,D,D,D | | DAS2: raid6: D,D,D,D,D,D | xfs. raid6: D,D,D,D,D,D | | DAS3: raid6: D,D,D,D,D,D | raid6: D,D,D,D,D,D | In addition, you can expand this configuration by adding more DAS units. This addresses several of your goals. In practice, just to get faster rebuild as the array gets larger, I suspect you would find it was worth making the horizontal arrays RAID5 instead of RAID0, just to minimize time to full performance. -- bill davidsen CTO TMR Associates, Inc "You are disgraced professional losers. And by the way, give us our money back." - Representative Earl Pomeroy, Democrat of North Dakota on the A.I.G. executives who were paid bonuses after a federal bailout. _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by oss.sgi.com (8.14.3/8.14.3/SuSE Linux 0.8) with ESMTP id n2NGMnmR169500 for ; Mon, 23 Mar 2009 11:23:10 -0500 Received: from mail.rtr.ca (localhost [127.0.0.1]) by cuda.sgi.com (Spam Firewall) with ESMTP id 075C41C62FAE for ; Mon, 23 Mar 2009 09:22:10 -0700 (PDT) Received: from mail.rtr.ca (rtr.ca [76.10.145.34]) by cuda.sgi.com with ESMTP id fwAZb287VIkYkLdW for ; Mon, 23 Mar 2009 09:22:10 -0700 (PDT) Message-ID: <49C7B730.7030506@rtr.ca> Date: Mon, 23 Mar 2009 12:22:08 -0400 From: Mark Lord MIME-Version: 1.0 Subject: Re: How to configure 36 disks ? References: <174852.24274.qm@web51309.mail.re2.yahoo.com> In-Reply-To: <174852.24274.qm@web51309.mail.re2.yahoo.com> List-Id: XFS Filesystem from SGI List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: xfs-bounces@oss.sgi.com Errors-To: xfs-bounces@oss.sgi.com To: Jon@eHardcastle.com Cc: linux-aio@kvack.org, Linux RAID Mailing List , linux-xfs@oss.sgi.com, "linux-ide@vger.kernel.org" , Raz , linux-fsdevel@vger.kernel.org, Bill Davidsen Jon Hardcastle wrote: > I'd like to understand how you even go attaching that many devices to a system.. I am 'comparatively' new to this.. and have a 6 raid5 system.. not enterprise.. and i have slammed into case/power/sat slot issues already. What sort of hardware must one use to grow to a 36 array system! .. Here's one way: -- four onboard SATA ports -- plus four add-in PCI/PCIX/PCIe 8-port Marvell SATA cards. -- and a honkin' huge PSU! :) _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by oss.sgi.com (8.14.3/8.14.3/SuSE Linux 0.8) with ESMTP id n2NGNgGv169559 for ; Mon, 23 Mar 2009 11:23:58 -0500 Received: from postman.lauml.com (localhost [127.0.0.1]) by cuda.sgi.com (Spam Firewall) with ESMTP id 363881C62FCA for ; Mon, 23 Mar 2009 09:23:24 -0700 (PDT) Received: from postman.lauml.com (postman.lauml.com [205.134.240.66]) by cuda.sgi.com with ESMTP id aRWSXwGswmKXQgDq for ; Mon, 23 Mar 2009 09:23:24 -0700 (PDT) Message-ID: <49C7B778.7000604@xman.org> Date: Mon, 23 Mar 2009 09:23:20 -0700 From: Christopher Smith MIME-Version: 1.0 Subject: Re: How to configure 36 disks ? References: <174852.24274.qm@web51309.mail.re2.yahoo.com> In-Reply-To: <174852.24274.qm@web51309.mail.re2.yahoo.com> List-Id: XFS Filesystem from SGI List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: xfs-bounces@oss.sgi.com Errors-To: xfs-bounces@oss.sgi.com To: Jon@eHardcastle.com Cc: linux-aio@kvack.org, Linux RAID Mailing List , linux-xfs@oss.sgi.com, "linux-ide@vger.kernel.org" , Raz , linux-fsdevel@vger.kernel.org, Bill Davidsen Jon Hardcastle wrote: > I'd like to understand how you even go attaching that many devices to a system.. I am 'comparatively' new to this.. and have a 6 raid5 system.. not enterprise.. and i have slammed into case/power/sat slot issues already. What sort of hardware must one use to grow to a 36 array system! You can always use this approach: http://www.youtube.com/watch?v=96dWOEa4Djs --Chris P.S.: Sorry, couldn't resist. _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by oss.sgi.com (8.14.3/8.14.3/SuSE Linux 0.8) with ESMTP id n2NGSRhh169970 for ; Mon, 23 Mar 2009 11:28:47 -0500 Received: from mail-bw0-f226.google.com (localhost [127.0.0.1]) by cuda.sgi.com (Spam Firewall) with ESMTP id AC39213BF72A for ; Mon, 23 Mar 2009 09:28:14 -0700 (PDT) Received: from mail-bw0-f226.google.com (mail-bw0-f226.google.com [209.85.218.226]) by cuda.sgi.com with ESMTP id bs03lWLoVoEY6zMI for ; Mon, 23 Mar 2009 09:28:14 -0700 (PDT) Received: by bwz26 with SMTP id 26so1842719bwz.20 for ; Mon, 23 Mar 2009 09:28:08 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <49C7B778.7000604@xman.org> References: <174852.24274.qm@web51309.mail.re2.yahoo.com> <49C7B778.7000604@xman.org> Date: Mon, 23 Mar 2009 18:28:07 +0200 Message-ID: <5d96567b0903230928p794e4140y9263f4095beb5b52@mail.gmail.com> Subject: Re: How to configure 36 disks ? From: Raz List-Id: XFS Filesystem from SGI List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: xfs-bounces@oss.sgi.com Errors-To: xfs-bounces@oss.sgi.com To: Christopher Smith Cc: Jon@ehardcastle.com, Linux RAID Mailing List , linux-aio@kvack.org, linux-xfs@oss.sgi.com, "linux-ide@vger.kernel.org" , linux-fsdevel@vger.kernel.org, Bill Davidsen On Mon, Mar 23, 2009 at 6:23 PM, Christopher Smith wrote: > Jon Hardcastle wrote: >> >> I'd like to understand how you even go attaching that many devices to a >> system.. I am 'comparatively' new to this.. and have a 6 raid5 system.. not >> enterprise.. and i have slammed into case/power/sat slot issues already. >> What sort of hardware must one use to grow to a 36 array system! > > You can always use this approach: > > http://www.youtube.com/watch?v=96dWOEa4Djs > > --Chris > > P.S.: Sorry, couldn't resist. > > me too. http://blogs.sun.com/observatory/entry/don_t_shout_at_your _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by oss.sgi.com (8.14.3/8.14.3/SuSE Linux 0.8) with ESMTP id n2NGjfqU170969 for ; Mon, 23 Mar 2009 11:46:02 -0500 Received: from qw-out-1920.google.com (localhost [127.0.0.1]) by cuda.sgi.com (Spam Firewall) with ESMTP id 1C78D1C63466 for ; Mon, 23 Mar 2009 09:45:23 -0700 (PDT) Received: from qw-out-1920.google.com (qw-out-1920.google.com [74.125.92.146]) by cuda.sgi.com with ESMTP id yl1lVcxBCkgRYSH1 for ; Mon, 23 Mar 2009 09:45:23 -0700 (PDT) Received: by qw-out-1920.google.com with SMTP id 9so1119514qwj.32 for ; Mon, 23 Mar 2009 09:45:23 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <174852.24274.qm@web51309.mail.re2.yahoo.com> References: <49C7AC2C.9090307@tmr.com> <174852.24274.qm@web51309.mail.re2.yahoo.com> Date: Mon, 23 Mar 2009 12:45:23 -0400 Message-ID: <87f94c370903230945k66a804a0iefbb76dd9c93fc16@mail.gmail.com> Subject: Re: How to configure 36 disks ? From: Greg Freemyer List-Id: XFS Filesystem from SGI List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: xfs-bounces@oss.sgi.com Errors-To: xfs-bounces@oss.sgi.com To: Jon@ehardcastle.com Cc: linux-aio@kvack.org, Linux RAID Mailing List , linux-xfs@oss.sgi.com, "linux-ide@vger.kernel.org" , Raz , linux-fsdevel@vger.kernel.org, Bill Davidsen On Mon, Mar 23, 2009 at 12:02 PM, Jon Hardcastle wrote: > > I'd like to understand how you even go attaching that many devices to a system.. I am 'comparatively' new to this.. and have a 6 raid5 system.. not enterprise.. and i have slammed into case/power/sat slot issues already. What sort of hardware must one use to grow to a 36 array system! For the drives you could get a few big enclosures: http://www.pc-pitstop.com/sata_enclosures/sat122urd.asp http://www.pc-pitstop.com/sata_enclosures/scsas16rm.asp I don't think these offer PMP (port multiplexer) support. I would look for ones that did if you're serious about doing something like this. If you used PMP with 4x drives per sata port, you would only need 9 sata ports in the PC to control the 36 drives. Not sure what the limits of PMP is, but 4x seems reasonable to me. Greg -- Greg Freemyer Head of EDD Tape Extraction and Processing team Litigation Triage Solutions Specialist http://www.linkedin.com/in/gregfreemyer First 99 Days Litigation White Paper - http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf The Norcross Group The Intersection of Evidence & Technology http://www.norcrossgroup.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by oss.sgi.com (8.14.3/8.14.3/SuSE Linux 0.8) with ESMTP id n2NIX9K3177619 for ; Mon, 23 Mar 2009 13:33:31 -0500 Received: from uplift.swm.pp.se (localhost [127.0.0.1]) by cuda.sgi.com (Spam Firewall) with ESMTP id E7B8D13C02E1 for ; Mon, 23 Mar 2009 11:32:36 -0700 (PDT) Received: from uplift.swm.pp.se (swm.pp.se [212.247.200.143]) by cuda.sgi.com with ESMTP id NKoFNBXdxeYuyU61 for ; Mon, 23 Mar 2009 11:32:36 -0700 (PDT) Date: Mon, 23 Mar 2009 19:32:28 +0100 (CET) From: Mikael Abrahamsson Subject: Re: How to configure 36 disks ? In-Reply-To: <87f94c370903230945k66a804a0iefbb76dd9c93fc16@mail.gmail.com> Message-ID: References: <49C7AC2C.9090307@tmr.com> <174852.24274.qm@web51309.mail.re2.yahoo.com> <87f94c370903230945k66a804a0iefbb76dd9c93fc16@mail.gmail.com> MIME-Version: 1.0 List-Id: XFS Filesystem from SGI List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: xfs-bounces@oss.sgi.com Errors-To: xfs-bounces@oss.sgi.com To: Greg Freemyer Cc: Jon@ehardcastle.com, Linux RAID Mailing List , linux-aio@kvack.org, linux-xfs@oss.sgi.com, "linux-ide@vger.kernel.org" , Raz , linux-fsdevel@vger.kernel.org, Bill Davidsen On Mon, 23 Mar 2009, Greg Freemyer wrote: > http://www.pc-pitstop.com/sata_enclosures/sat122urd.asp > http://www.pc-pitstop.com/sata_enclosures/scsas16rm.asp I would like to add this if you're on a budget: > sata ports in the PC to control the 36 drives. Not sure what the limits > of PMP is, but 4x seems reasonable to me. The PMPs I've seen take 5 drives each: -- Mikael Abrahamsson email: swmike@swm.pp.se _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by oss.sgi.com (8.14.3/8.14.3/SuSE Linux 0.8) with ESMTP id n2OLoAlO047868 for ; Tue, 24 Mar 2009 16:50:20 -0500 Received: from fmmailgate02.web.de (localhost [127.0.0.1]) by cuda.sgi.com (Spam Firewall) with ESMTP id 81CF21C69A6F for ; Tue, 24 Mar 2009 14:49:20 -0700 (PDT) Received: from fmmailgate02.web.de (fmmailgate02.web.de [217.72.192.227]) by cuda.sgi.com with ESMTP id iRJf3NESOmraa8wW for ; Tue, 24 Mar 2009 14:49:20 -0700 (PDT) From: Goswin von Brederlow Subject: Re: How to configure 36 disks ? References: <174852.24274.qm@web51309.mail.re2.yahoo.com> Date: Tue, 24 Mar 2009 20:38:35 +0100 In-Reply-To: <174852.24274.qm@web51309.mail.re2.yahoo.com> (Jon Hardcastle's message of "Mon, 23 Mar 2009 09:02:14 -0700 (PDT)") Message-ID: <87hc1iemo4.fsf@frosties.localdomain> MIME-Version: 1.0 List-Id: XFS Filesystem from SGI List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: xfs-bounces@oss.sgi.com Errors-To: xfs-bounces@oss.sgi.com To: Jon@eHardcastle.com Cc: linux-aio@kvack.org, Linux RAID Mailing List , linux-xfs@oss.sgi.com, "linux-ide@vger.kernel.org" , Raz , linux-fsdevel@vger.kernel.org, Bill Davidsen Jon Hardcastle writes: > I'd like to understand how you even go attaching that many devices to a system.. I am 'comparatively' new to this.. and have a 6 raid5 system.. not enterprise.. and i have slammed into case/power/sat slot issues already. What sort of hardware must one use to grow to a 36 array system! Well, lets see. Put 3 dual channel SAS controler in the box giving you 6 external SAS connectors. Buy 6 48x disk enclosures and connect them. Configure them all as JBOD and you get your 288 disks. MfG Goswin _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by oss.sgi.com (8.14.3/8.14.3/SuSE Linux 0.8) with ESMTP id n2R8VYnu224613 for ; Fri, 27 Mar 2009 03:31:49 -0500 Received: from mailsrv1.zmi.at (localhost [127.0.0.1]) by cuda.sgi.com (Spam Firewall) with ESMTP id 7943B1C752DF for ; Fri, 27 Mar 2009 01:31:06 -0700 (PDT) Received: from mailsrv1.zmi.at (mailsrv1.zmi.at [212.69.162.198]) by cuda.sgi.com with ESMTP id ml1SleM1SjyHsQxE for ; Fri, 27 Mar 2009 01:31:06 -0700 (PDT) Received: from mailsrv2.i.zmi.at (unknown [81.217.54.243]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "mailsrv2.i.zmi.at", Issuer "power4u.zmi.at" (not verified)) by mailsrv1.zmi.at (Postfix) with ESMTP id 9AF944C29 for ; Fri, 27 Mar 2009 09:31:04 +0100 (CET) Received: from saturn.localnet (saturn.i.zmi.at [10.0.0.2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mailsrv2.i.zmi.at (Postfix) with ESMTPSA id 9491640DC04 for ; Fri, 27 Mar 2009 09:31:04 +0100 (CET) Subject: Re: How to configure 36 disks ? Content-Disposition: inline From: Michael Monnerie Date: Fri, 27 Mar 2009 09:31:03 +0100 MIME-Version: 1.0 Message-Id: <200903270931.03964@zmi.at> List-Id: XFS Filesystem from SGI List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Sender: xfs-bounces@oss.sgi.com Errors-To: xfs-bounces@oss.sgi.com To: xfs@oss.sgi.com (I sent this mail on Tue 23:40 CET, but it seems it didn't arrive on the = list. At least I didn't receive it. So I send again) On Montag 23 M=E4rz 2009 Raz wrote: > 1. redundancy. > 2. performance. First, you should have bought enclosures with included RAID controllers, = then you'd be finished already. But then, I guess you wanted to save the money, but at least go get some = real hardware RAID controller. Look at Areca ( http://www.areca.com.tw ) = if you use Linux or Windows, the 1680 series SAS controllers can cope = with SAS/SATA drives, up to 128 drives per controller. You can put a = cache up to 4GB on the controller, and a battery backup module that = protects your data in case of power outages. Those controllers are = blazing fast, have good admin tools, send you an e-mail in case of = problems, and they used it in the video that Chris mentioned: http://blogs.sun.com/observatory/entry/don_t_shout_at_your In the end you see they used an Areca 16port, plus an Adaptec 8port (I = guess because they only had the Areca in 16port available). Then configure a RAID-60 or whatever you like. mfg zmi -- = // Michael Monnerie, Ing.BSc ----- http://it-management.at // Tel: 0660 / 415 65 31 .network.your.ideas. // PGP Key: "curl -s http://zmi.at/zmi.asc | gpg --import" // Fingerprint: AC19 F9D5 36ED CD8A EF38 500E CE14 91F7 1C12 09B4 // Keyserver: wwwkeys.eu.pgp.net Key-ID: 1C1209B4 _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs