Re: Triple parity and beyond

[NeilBrown <neilb@suse.de>]

[-- Attachment #1: Type: text/plain, Size: 2225 bytes --]

On Fri, 22 Nov 2013 21:34:41 -0800 John Williams <jwilliams4200@gmail.com>
wrote:

> On Fri, Nov 22, 2013 at 9:04 PM, NeilBrown <neilb@suse.de> wrote:
> 
> > I guess with that many drives you could hit PCI bus throughput limits.
> >
> > A 16-lane PCIe 4.0 could just about give 100MB/s to each of 16 devices.  So
> > you would really need top-end hardware to keep all of 16 drives busy in a
> > recovery.
> > So yes: rebuilding a drive in a 16-drive RAID6+ would be slower than in e.g.
> > a 20 drive RAID10.
> 
> Not really. A single 8x PCIe 2.0 card has 8 x 500MB/s = 4000MB/s of
> potential bandwidth. That would be 250MB/s per drive for 16 drives.
> 
> But quite a few people running software RAID with many drives have
> multiple PCIe cards. For example, in one machine I have three IBM
> M1015 cards (which I got for $75/ea) that are 8x PCIe 2.0. That comes
> to 3 x 500MB/s x 8 = 12GB/s of IO bandwidth.
> 
> Also, your math is wrong. PCIe 3.0 is 985 MB/s per lane. If we assume
> PCIe 4.0 would double that, we would have 1970MB/s per lane. So one
> lane of the hypothetical PCIe 4.0 would have enough IO bandwidth to
> give about 120MB/s to each of 16 drives. A single 8x PCIe 4.0 card
> would have 8 times that capability which is more than 15GB/s.

It wasn't my math, it was my reading :-(
16-lane PCIe 4.0 is 31 GB/sec so 2GB/sec per drive.  I was reading the
"1-lane" number...

> 
> Even a single 8x PCIe 3.0 card has potentially over 7GB/s of bandwidth.
> 
> Bottom line is that IO bandwidth is not a problem for a system with
> prudently chosen hardware.
> 
> More likely is that you would be CPU limited (rather than bus limited)
> in a high-parity rebuild where more than one drive failed. But even
> that is not likely to be too bad, since Andrea's single-threaded
> recovery code can recover two drives at nearly 1GB/s on one of my
> machines. I think the code could probably be threaded to achieve a
> multiple of that running on multiple cores.

Indeed.  It seems likely that with modern hardware, the  linear write speed
would be the limiting factor for spinning-rust drives.
For SSDs the limit might end up being somewhere else ...

Thanks,
NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]