Maximum theoretical RAID-0 Speed

[AndyLiebman@aol.com]

I'm wondering if anyone on this list can shed some light on a question that 
pertains to the maximum theoretical read speed for the RAIDS on my Linux box, 
and whether I have reached it. My guess is, there are about 2 people in the 
world who possibly understand this. Linus Torvolds, perhaps. And maybe somebody 
else. But I'll give this list a try. I've met some pretty sharp people here. 

Here's the scenario I have been testing. 

I have a single Xeon 3.06 processor set to use Hyperthreading, 2 GB of RAM on 
a SuperMicro Motherboard. The motherboard has 4 PCI "bus segments" with a 
total of six expansion slots. There are two PCI-X 133 Mhz slots (each associated 
with its own PCI bus segment). There is one PCI-X 100 Mhz slot (on ITS own 
segment) and  three PCI-32bit 33/66 Mhz slots (all sharing the same bus segment). 
Each of the PCI-X 133 Mhz slots also has one of the built-in GigE ports on it 
(and I put all my other Intel GigE ports on these two bus segments -- 
sometimes I have up to 6 ports in total on my machine). So I leave the 133 Mhz slots 
out of the RAIDS. 

I have 16 or 24 SATA drive bays in my enclosures. 

My basic design is to make Hardware RAID-5 arrays with 3ware 9000 cards and 
Serial ATA drives. Then I make a Software RAID-0 stripe on top of the Hardware 
RAID-5. Sometimes I work with 8-channel 3ware cards, sometimes with 12-channel 
cards. So far, I have always put the cards (they're 66Mhz cards) in a 
combination of the 3 PCI 33/66 Mhz slots and the one PCI-X 100 Mhz slot. 

So, as I said above,  that means I don't have any drives connected to the two 
PCI-X 133 slots (or to the segments they correspond to) because that would 
slow down the bus speed for those segments and presumably hurt my network 
performance. 

When I make a single 8-drive array and test it with Bonnie++, I get a write 
speed of about 75 MB/sec and a read speed of about 300 MB/sec. It's the same 
whether I put the 3ware card and drives on the PCI 33/66 slots or in the PCI-X 
100Mhz slot (or in a PCI-X 133 slot for that matter, which I haven't done 
except once for a test). 

When I make a single 12-drive array and do the same test, I get a write speed 
of about 90 MB/sec, and a read speed of about 375 MB/sec. So, 12-drive arrays 
are faster than 8 drive arrays. Sensible. 

When I put a software stripe on top of two 8 drive arrays, I get a write 
speed of about 100 MB/sec, and a read speed of about 475-500 MB/sec. So striping 
two 8-drive arrays gives a significant boost in read performance. Almost double 
the performance of a single 8-drive array. 

When I put a software stripe on top of three 8 drive arrays, the write speed 
goes up to about 150 MB/sec, but the read speed drops a bit from the maximum 
-- I get about 450 MB/sec. One explanation for the lower read speed may be that 
I have two 8 channel cards on the same PCI bus segment, and one 8 channel 
card on its own segment. Maybe there's an imbalance in bandwidth to the cards. 

When I put a software stripe on top of two 12-drive arrays, the read and 
write speeds is about the same as I get with two 8-drive arrays. So, there's no 
advantage in striping 8-drive arrays versus 12-drive arrays -- even though the 
12-drive arrays on their own perform better than the 8-drive arrays on THEIR 
own. 

The key point is, I get the best performance (at least as measured by 
Bonnie++) striping two arrays as opposed to striping three arrays. 

By the way, my measurements have been taken with the 2.6.6 kernel -- and I've 
tested each scenario at least 3 or 4 times and averaged the results. 
Preliminary testing with the 2.6.9 kernel shows about 50 percent higher write speeds, 
and a slight drop (like 3 or 4 percent) in read speeds. 

My question is, do you think I have reached some sort bandwidth limit with a 
read speed of around 500 MB/sec? Could it be that the CPU/RAM/PCI-X buses just 
can't handle any more data? Or might I be missing some tricks? 

Would having a second CPU or more RAM make any difference (I don't believe 
so, but I'm no expert on this). Would switching to the new Intel 800 Mhz 
frontside bus help (my current CPUs are 533Mhz)? Would if make a difference if I put 
ALL of my GigE ports on a single PCI-X 133 bus, thus freeing up a third PCI 
bus segment for a 3ware card (allowing me to put three 8-drive arrays each on 
its own bus segment)? 

I also understand that the new Xeons coming out now have 64-bit extensions 
and run the 64-bit versions of Linux, just as the AMD Opterons do. Would that 
make a big difference? Would Opterons make a big difference. 

I have played around a lot with the "blockdev --setra" settings. 3ware 
recommends a readahead of 16384 to get the best performance with their cards. And at 
least with Bonnie++, and the hard drives that I am using, I have found that 
to be true. 

I have also played around with the readahead settings for the Linux Software 
RAID-0 array. The default readahead seems to be 1024 per drive. So, for a two 
drive array, the default gets set to 2048. For three drives the default is 
3072. The default, indeed, gives me the best write speed as measured by Bonnie++. 
However, for my particular application, I get much better real world 
performance with a higher readahead. (An illustration of the dangers of tweaking your 
system to get the best results on benchmark tests.) 

This is obviously a very complex problem, and many many factors can influence 
performance. It WOULD be good to have some sense for the relationship between 
all the various bottlenecks and variables. 

Looking forward to some thoughtful answers. 

Regards, 
Andy Liebman