southbridge/sata controller performance?

southbridge/sata controller performance?

[Matt Garman]

When using Linux software RAID, I was thinking that, from a
performance perspective, the southbridge/SATA controller is probably
one of the most important components (assuming a reasonable CPU).
Is this a correct generalization?

If it is, has anyone done a study or benchmarked SATA controller
performance?  Particularly with consumer-grade hardware?

I haven't been able to find much info about this on the web; the
Tech Report seems to consistently benchmark SATA performance:

    AMD SB600: http://techreport.com/articles.x/13832/5
    AMD SB700: http://techreport.com/articles.x/14261/10
    AMD SB750: http://techreport.com/articles.x/13628/9
    Intel ICH10: http://techreport.com/articles.x/15653/9
    nVidia GeForce 8300: http://techreport.com/articles.x/14993/9

My (over) generalization of the above is that SB600 is to be
avoided, and performance ranking is Intel/nVidia over AMD.

But this is just one set of benchmarks and obviously they used
Windows and therefore Windows drivers, so my simplification really
isn't fair.

Thoughts?

Matt

Re: southbridge/sata controller performance?

[Roger Heflin]

Matt Garman wrote:
> When using Linux software RAID, I was thinking that, from a
> performance perspective, the southbridge/SATA controller is probably
> one of the most important components (assuming a reasonable CPU).
> Is this a correct generalization?
> 
> If it is, has anyone done a study or benchmarked SATA controller
> performance?  Particularly with consumer-grade hardware?
> 
> I haven't been able to find much info about this on the web; the
> Tech Report seems to consistently benchmark SATA performance:
> 
>     AMD SB600: http://techreport.com/articles.x/13832/5
>     AMD SB700: http://techreport.com/articles.x/14261/10
>     AMD SB750: http://techreport.com/articles.x/13628/9
>     Intel ICH10: http://techreport.com/articles.x/15653/9
>     nVidia GeForce 8300: http://techreport.com/articles.x/14993/9

In general those benchmarks are mostly useless for Raid.

The biggest different for RAID is what happens when multiple channels 
are used.   On full speed streaming read/write almost every (even the 
worst PCI controllers on a PCI bus) sata controller is close to equal 
when you only have one drive, once you use 2 drives or more things 
change.   If the given controller setup can actually run several 
drives at close to full single drive speed performance will be good, 
if it cannot things are going to get much slower.

One test I do is a single dd from one disk and watch the io speed, and 
then I add dd's and watch what happens.   I have 4 identical disks, 2 
are on a ICH7, 2 are on a pci bus based MB controller, any single of 
these disks will do about 75MB/second no matter were it is, using 2 on 
the ich7 gets about 140-150MB/second, using the 2 on the pci bus based 
MB controller gets 98MB/second.   If all of the controllers 
techreports tested were equal on this test, then it might be worth it 
to look at the benchmarks techreport is using, but the simple dd 
benchmark is likely more important, and I am pretty sure that someone 
could use a controller (on a bandwidth limited bus) that would do good 
on the above techreport benchmark, but fail horribly when several 
disks with a high io speed were being used and work badly for raid.

Re: southbridge/sata controller performance?

[Justin Piszcz]

On Sat, 3 Jan 2009, Roger Heflin wrote:

> Matt Garman wrote:
>> When using Linux software RAID, I was thinking that, from a
>> performance perspective, the southbridge/SATA controller is probably
>> one of the most important components (assuming a reasonable CPU).
>> Is this a correct generalization?
>> 
>> If it is, has anyone done a study or benchmarked SATA controller
>> performance?  Particularly with consumer-grade hardware?
>> 
>> I haven't been able to find much info about this on the web; the
>> Tech Report seems to consistently benchmark SATA performance:
>>
>>     AMD SB600: http://techreport.com/articles.x/13832/5
>>     AMD SB700: http://techreport.com/articles.x/14261/10
>>     AMD SB750: http://techreport.com/articles.x/13628/9
>>     Intel ICH10: http://techreport.com/articles.x/15653/9
>>     nVidia GeForce 8300: http://techreport.com/articles.x/14993/9
>
> In general those benchmarks are mostly useless for Raid.
>
> The biggest different for RAID is what happens when multiple channels are 
> used.   On full speed streaming read/write almost every (even the worst PCI 
> controllers on a PCI bus) sata controller is close to equal when you only 
> have one drive, once you use 2 drives or more things change.   If the given 
> controller setup can actually run several drives at close to full single 
> drive speed performance will be good, if it cannot things are going to get 
> much slower.
>
> One test I do is a single dd from one disk and watch the io speed, and then I 
> add dd's and watch what happens.   I have 4 identical disks, 2 are on a ICH7, 
> 2 are on a pci bus based MB controller, any single of these disks will do 
> about 75MB/second no matter were it is, using 2 on the ich7 gets about 
> 140-150MB/second, using the 2 on the pci bus based MB controller gets 
> 98MB/second.   If all of the controllers techreports tested were equal on 
> this test, then it might be worth it to look at the benchmarks techreport is 
> using, but the simple dd benchmark is likely more important, and I am pretty 
> sure that someone could use a controller (on a bandwidth limited bus) that 
> would do good on the above techreport benchmark, but fail horribly when 
> several disks with a high io speed were being used and work badly for raid.
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

This has been discussed before, on the southbridge it is usually good/you 
will get the maximum bandwidth for each of the 6 sata ports.  After that, 
you need to use PCI-e lanes from the northbridge.  Using PCI-e x1 slots 
that come off the southbridge can degrade performance of the 6 SATA ports 
or the speed coming off the drives connected to the x1 slots will be slow.

What are you trying to accomplish?

As Roger pointed out, doing a dd is a good way to test, from each disk, 
simultaneously, on an old Intel P965 board I was able to achieve 
1.0-1.1Gbyte/sec doing that with 12 Velociraptors and 1.0Gbyte/sec reads 
on the XFS filesystem when dd (reading) large data on the volume.  Approx 
500-600MiB/s from the southbridge, the other 400MiB/s from the 
northbridge.

Justin.

Re: southbridge/sata controller performance?

[Matt Garman]

On Sun, Jan 04, 2009 at 04:55:18AM -0500, Justin Piszcz wrote:
>> The biggest different for RAID is what happens when multiple
>> channels are used.   On full speed streaming read/write almost
>> every (even the worst PCI controllers on a PCI bus) sata
>> controller is close to equal when you only have one drive, once
>> you use 2 drives or more things change.   If the given controller
>> setup can actually run several drives at close to full single
>> drive speed performance will be good, if it cannot things are
>> going to get much slower.

So, in general, the ideal is to be able to simultaneously read/write
from multiple disks at the same speed as a single drive.

> This has been discussed before, on the southbridge it is usually
> good/you will get the maximum bandwidth for each of the 6 sata
> ports.  After that, you need to use PCI-e lanes from the
> northbridge.  Using PCI-e x1 slots that come off the southbridge
> can degrade performance of the 6 SATA ports or the speed coming
> off the drives connected to the x1 slots will be slow.

Are you talking about using additional SATA controllers (i.e. via
add-on cards)?  Or simply talking about the interconnect between the
southbridge and the northbridge?

I think you're talking about the former, i.e. the SATA controller
integrated in the southbridge generally ought to be fine, but if you
start adding additional controllers that hang off the south bridge,
their could be competition for bandwidth to the northbridge...
right?  (And wouldn't the nvidia chips have an edge here, since they
have everything combined into one chip?)

Makes intuitive sense anyway; but in my case I'm really just curious
about the SATA controller integrated into the southbridge; not
concerned with additional SATA controllers.

> What are you trying to accomplish?

Trying to determine what motherboard would be ideal for a home NAS
box AND have the lowest power consumption... the AMD solutions seem
to win on the power consumption front, but I'm not sure about the
performance.

> As Roger pointed out, doing a dd is a good way to test, from each
> disk, simultaneously, on an old Intel P965 board I was able to
> achieve 1.0-1.1Gbyte/sec doing that with 12 Velociraptors and
> 1.0Gbyte/sec reads on the XFS filesystem when dd (reading) large
> data on the volume.  Approx 500-600MiB/s from the southbridge, the
> other 400MiB/s from the northbridge.

Is the "parallel dd test" valid if I do a raw read off the device,
e.g. "dd if=/dev/sda of=/dev/null"?  All my drives are already in an
md array, so I can't access them individually at the filesystem
level.

Thanks for the feedback and info!

Matt

Re: southbridge/sata controller performance?

[Roger Heflin]

Matt Garman wrote:

> 
> Are you talking about using additional SATA controllers (i.e. via
> add-on cards)?  Or simply talking about the interconnect between the
> southbridge and the northbridge?
> 

Don't count on it, it depends on the specific setup of the MB.

> I think you're talking about the former, i.e. the SATA controller
> integrated in the southbridge generally ought to be fine, but if you
> start adding additional controllers that hang off the south bridge,
> their could be competition for bandwidth to the northbridge...
> right?  (And wouldn't the nvidia chips have an edge here, since they
> have everything combined into one chip?)

Again, it depends on how much bandwidth was allocated internally in 
the chip just because it is one chip does not mean that anyone 
actually allocated enough resources to the given part of the chip, or 
even has enough feed to the chip to support them all.   The Seagate 
1.5's will stream quite a bit higher numbers than previous disks, so 
if one had several of them they could overload the allocated 
bandwidth.   Things build-into a given MB aren't always connected at 
full speed often some of them are actually pci bus parts and suffer 
from slow access, I would expect there to be similar tradeoffs even in 
single chip cases.

> 
> Makes intuitive sense anyway; but in my case I'm really just curious
> about the SATA controller integrated into the southbridge; not
> concerned with additional SATA controllers.
> 

The build-in intel ich series controllers vary from version to version 
on exactly how fast that they are in this test, the newer ones are 
faster than the older ones.

> 
> Is the "parallel dd test" valid if I do a raw read off the device,
> e.g. "dd if=/dev/sda of=/dev/null"?  All my drives are already in an
> md array, so I can't access them individually at the filesystem
> level.
> 
> Thanks for the feedback and info!
> 
> Matt
> 
> 
Yes, that is valid, so long as the md device is fairly quiet at the 
time of the test.  Since you already have a machine, please post the 
mb type, and number/type of sata ports and the 1-disk, 2-disk, 
3-disk,... numbers that you get.

I am looking at getting a new AMD solution in the next month or so and 
would like also know which scales reasonably.

Re: southbridge/sata controller performance?

[Justin Piszcz]

On Sun, 4 Jan 2009, Roger Heflin wrote:

> Matt Garman wrote:
>
>> 
>> Are you talking about using additional SATA controllers (i.e. via
>> add-on cards)?  Or simply talking about the interconnect between the
>> southbridge and the northbridge?
>> 
>
> Don't count on it, it depends on the specific setup of the MB.
>
>> I think you're talking about the former, i.e. the SATA controller
>> integrated in the southbridge generally ought to be fine, but if you
>> start adding additional controllers that hang off the south bridge,
>> their could be competition for bandwidth to the northbridge...
>> right?  (And wouldn't the nvidia chips have an edge here, since they
>> have everything combined into one chip?)
>
> Again, it depends on how much bandwidth was allocated internally in the chip 
> just because it is one chip does not mean that anyone actually allocated 
> enough resources to the given part of the chip, or even has enough feed to 
> the chip to support them all.   The Seagate 1.5's will stream quite a bit 
> higher numbers than previous disks, so if one had several of them they could 
> overload the allocated bandwidth.   Things build-into a given MB aren't 
> always connected at full speed often some of them are actually pci bus parts 
> and suffer from slow access, I would expect there to be similar tradeoffs 
> even in single chip cases.
Yup that is why you need to check the schematic for the motherboard before
you purchase as to which PCI-e are "shared" vs. "switched" vs. dedicated

>
>> 
>> Makes intuitive sense anyway; but in my case I'm really just curious
>> about the SATA controller integrated into the southbridge; not
>> concerned with additional SATA controllers.
>> 
>
> The build-in intel ich series controllers vary from version to version on 
> exactly how fast that they are in this test, the newer ones are faster than 
> the older ones.
Agree, the Northbridge <-> CPU generally increases in speed each new chipset,
965 -> X38 -> X48 etc..

>
>> 
>> Is the "parallel dd test" valid if I do a raw read off the device,
>> e.g. "dd if=/dev/sda of=/dev/null"?  All my drives are already in an
>> md array, so I can't access them individually at the filesystem
>> level.
>> 
>> Thanks for the feedback and info!
>> 
>> Matt

Re: southbridge/sata controller performance?

[Matt Garman]

[-- Attachment #1: Type: text/plain, Size: 4306 bytes --]

On Sun, Jan 04, 2009 at 03:02:37PM -0600, Roger Heflin wrote:
> Yes, that is valid, so long as the md device is fairly quiet at
> the time of the test.  Since you already have a machine, please
> post the mb type, and number/type of sata ports and the 1-disk,
> 2-disk, 3-disk,... numbers that you get.

I attached a little Perl script I hacked up for running many
instances of dd in parallel or individually.  It takes one or more
block devices on the command line, and issues parallel dd reads for
each one.  (Note my block size and count params are enormous; I made
these huge to avoid caching effects of having 4 GB of RAM.)

I have two MD arrays in my system:

$ cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md1 : active raid5 sdb1[0] sdd1[3] sdc1[2] sde1[1]
      2930279808 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
            
md0 : active raid5 sdf1[0] sdi1[3] sdh1[2] sdg1[1]
      2197715712 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
            
unused devices: <none>


This is on Ubuntu server 8.04.  Hardware is an Intel DQ35JO
motherboard (Q35/ICH9DO chipsets) with 4 GB RAM and an E5200 CPU.

    - md1 has four WD RE2 GP 1 TB hard drives (the 5400 RPM ones).
      These are drives are connected to two 2-port PCIe SATA add-on
      cards with the Silicon Image 3132 controller.

    - md0 has four WD 750 GB drives (neither "RE2" version nor
      GreenPower).  These drives are connected directly to the
      on-board SATA ports (i.e. ICH9DO).

For md1 (RE2 GP drives on SiI 3132), it looks like I get about 80
MB/s when running dd on one drive:

$ sudo ./sata_performance.pl /dev/sd[b]
/dev/sdb: 13107200000 bytes (13 GB) copied, 163.757 s, 80.0 MB/s

With all four drives, it only drops to about 75 MB/s per drive:

$ sudo ./sata_performance.pl /dev/sd[bcde]
/dev/sdb: 13107200000 bytes (13 GB) copied, 171.647 s, 76.4 MB/s
/dev/sde: 13107200000 bytes (13 GB) copied, 172.052 s, 76.2 MB/s
/dev/sdd: 13107200000 bytes (13 GB) copied, 172.737 s, 75.9 MB/s
/dev/sdc: 13107200000 bytes (13 GB) copied, 172.777 s, 75.9 MB/s

For md0 (4x750 GB drives), performance is a little better; these
drives are definitely faster (7200 rpm vs 5400 rpm), and IIRC, the
SiI 3132 chipsets are known to be lousy performers.

Anyway, single drive performance is just shy of 100 MB/s:

$ sudo ./sata_performance.pl /dev/sd[f]
/dev/sdf: 13107200000 bytes (13 GB) copied, 133.099 s, 98.5 MB/s

And I lose virtually nothing when running all four disks at once:

$ sudo ./sata_performance.pl /dev/sd[fghi]
/dev/sdi: 13107200000 bytes (13 GB) copied, 130.632 s, 100 MB/s
/dev/sdf: 13107200000 bytes (13 GB) copied, 133.077 s, 98.5 MB/s
/dev/sdh: 13107200000 bytes (13 GB) copied, 133.411 s, 98.2 MB/s
/dev/sdg: 13107200000 bytes (13 GB) copied, 138.481 s, 94.6 MB/s

Read performance only drops a bit when reading from all eight
drives:

$ sudo ./sata_performance.pl /dev/sd[bcdefghi]
/dev/sdi: 13107200000 bytes (13 GB) copied, 133.086 s, 98.5 MB/s
/dev/sdf: 13107200000 bytes (13 GB) copied, 135.59 s, 96.7 MB/s
/dev/sdh: 13107200000 bytes (13 GB) copied, 135.86 s, 96.5 MB/s
/dev/sdg: 13107200000 bytes (13 GB) copied, 140.215 s, 93.5 MB/s
/dev/sdb: 13107200000 bytes (13 GB) copied, 182.402 s, 71.9 MB/s
/dev/sdc: 13107200000 bytes (13 GB) copied, 183.234 s, 71.5 MB/s
/dev/sdd: 13107200000 bytes (13 GB) copied, 189.025 s, 69.3 MB/s
/dev/sde: 13107200000 bytes (13 GB) copied, 189.517 s, 69.2 MB/s


Note: in all the above test runs, I actually ran them all three
times to catch any variance; all runs were consistent.

> I am looking at getting a new AMD solution in the next month or so
> and would like also know which scales reasonably.

The one I've got my eye on is the GA-MA74GM-S2.  It's got six
onboard SATA ports (provided by the SB700 southbridge).
SilentPCReview reviewed this board[1] and found it to have extremely
low power consumption when paired with a 4050e CPU.  On that site
and other hardware enthusiast sites, there are a lot of people using
it for home NAS/fileserver boxes.  I thinking about picking one up
to replace this Intel board and CPU; if I do, I'll report back the
results.

[1] http://www.silentpcreview.com/article859-page1.html

Thanks again!  Hope someone finds this info useful.

Matt


[-- Attachment #2: sata_performance.pl --]
[-- Type: text/x-perl, Size: 867 bytes --]

#!/usr/bin/perl

# License: public domain

use strict;

my $dd="/bin/dd";
my $bs="128k";
my $count="100000";

my @pids;
for my $dev (@ARGV)
{
    my $pid = fork();
    if ($pid == 0) # child
    {
        my $cmd = "$dd if=$dev of=/dev/null bs=$bs count=$count 2>&1";
        my @output = `$cmd`;
        my $n_lines = scalar(@output);
        my $rc = $? >> 8;
        if ($rc != 0 || $n_lines != 3)
        {
            print STDERR "ERROR: $dev: return code=$rc, n_lines=$n_lines:\n";
            foreach my $line (@output)
            {
                print STDERR "  " . $line;
            }
        }
        else
        {
            my $line = $output[2];
            chomp($line);
            print "$dev: $line\n";
        }
        exit(0);
    }
    else # parent
    {
        push(@pids, $pid);
    }
}

for my $pid (@pids)
{
    waitpid($pid, 0);
}

Re: southbridge/sata controller performance?

[Keld Jørn Simonsen]

What about CPU usage?

What happens when you put it into one or more raids, what are then your
combinde performance? Did you experiment with different raid types?

best regards
keld

On Sun, Jan 04, 2009 at 09:27:28PM -0600, Matt Garman wrote:
> On Sun, Jan 04, 2009 at 03:02:37PM -0600, Roger Heflin wrote:
> > Yes, that is valid, so long as the md device is fairly quiet at
> > the time of the test.  Since you already have a machine, please
> > post the mb type, and number/type of sata ports and the 1-disk,
> > 2-disk, 3-disk,... numbers that you get.
> 
> I attached a little Perl script I hacked up for running many
> instances of dd in parallel or individually.  It takes one or more
> block devices on the command line, and issues parallel dd reads for
> each one.  (Note my block size and count params are enormous; I made
> these huge to avoid caching effects of having 4 GB of RAM.)
> 
> I have two MD arrays in my system:
> 
> $ cat /proc/mdstat 
> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
> md1 : active raid5 sdb1[0] sdd1[3] sdc1[2] sde1[1]
>       2930279808 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
>             
> md0 : active raid5 sdf1[0] sdi1[3] sdh1[2] sdg1[1]
>       2197715712 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
>             
> unused devices: <none>
> 
> 
> This is on Ubuntu server 8.04.  Hardware is an Intel DQ35JO
> motherboard (Q35/ICH9DO chipsets) with 4 GB RAM and an E5200 CPU.
> 
>     - md1 has four WD RE2 GP 1 TB hard drives (the 5400 RPM ones).
>       These are drives are connected to two 2-port PCIe SATA add-on
>       cards with the Silicon Image 3132 controller.
> 
>     - md0 has four WD 750 GB drives (neither "RE2" version nor
>       GreenPower).  These drives are connected directly to the
>       on-board SATA ports (i.e. ICH9DO).
> 
> For md1 (RE2 GP drives on SiI 3132), it looks like I get about 80
> MB/s when running dd on one drive:
> 
> $ sudo ./sata_performance.pl /dev/sd[b]
> /dev/sdb: 13107200000 bytes (13 GB) copied, 163.757 s, 80.0 MB/s
> 
> With all four drives, it only drops to about 75 MB/s per drive:
> 
> $ sudo ./sata_performance.pl /dev/sd[bcde]
> /dev/sdb: 13107200000 bytes (13 GB) copied, 171.647 s, 76.4 MB/s
> /dev/sde: 13107200000 bytes (13 GB) copied, 172.052 s, 76.2 MB/s
> /dev/sdd: 13107200000 bytes (13 GB) copied, 172.737 s, 75.9 MB/s
> /dev/sdc: 13107200000 bytes (13 GB) copied, 172.777 s, 75.9 MB/s
> 
> For md0 (4x750 GB drives), performance is a little better; these
> drives are definitely faster (7200 rpm vs 5400 rpm), and IIRC, the
> SiI 3132 chipsets are known to be lousy performers.
> 
> Anyway, single drive performance is just shy of 100 MB/s:
> 
> $ sudo ./sata_performance.pl /dev/sd[f]
> /dev/sdf: 13107200000 bytes (13 GB) copied, 133.099 s, 98.5 MB/s
> 
> And I lose virtually nothing when running all four disks at once:
> 
> $ sudo ./sata_performance.pl /dev/sd[fghi]
> /dev/sdi: 13107200000 bytes (13 GB) copied, 130.632 s, 100 MB/s
> /dev/sdf: 13107200000 bytes (13 GB) copied, 133.077 s, 98.5 MB/s
> /dev/sdh: 13107200000 bytes (13 GB) copied, 133.411 s, 98.2 MB/s
> /dev/sdg: 13107200000 bytes (13 GB) copied, 138.481 s, 94.6 MB/s
> 
> Read performance only drops a bit when reading from all eight
> drives:
> 
> $ sudo ./sata_performance.pl /dev/sd[bcdefghi]
> /dev/sdi: 13107200000 bytes (13 GB) copied, 133.086 s, 98.5 MB/s
> /dev/sdf: 13107200000 bytes (13 GB) copied, 135.59 s, 96.7 MB/s
> /dev/sdh: 13107200000 bytes (13 GB) copied, 135.86 s, 96.5 MB/s
> /dev/sdg: 13107200000 bytes (13 GB) copied, 140.215 s, 93.5 MB/s
> /dev/sdb: 13107200000 bytes (13 GB) copied, 182.402 s, 71.9 MB/s
> /dev/sdc: 13107200000 bytes (13 GB) copied, 183.234 s, 71.5 MB/s
> /dev/sdd: 13107200000 bytes (13 GB) copied, 189.025 s, 69.3 MB/s
> /dev/sde: 13107200000 bytes (13 GB) copied, 189.517 s, 69.2 MB/s
> 
> 
> Note: in all the above test runs, I actually ran them all three
> times to catch any variance; all runs were consistent.
> 
> > I am looking at getting a new AMD solution in the next month or so
> > and would like also know which scales reasonably.
> 
> The one I've got my eye on is the GA-MA74GM-S2.  It's got six
> onboard SATA ports (provided by the SB700 southbridge).
> SilentPCReview reviewed this board[1] and found it to have extremely
> low power consumption when paired with a 4050e CPU.  On that site
> and other hardware enthusiast sites, there are a lot of people using
> it for home NAS/fileserver boxes.  I thinking about picking one up
> to replace this Intel board and CPU; if I do, I'll report back the
> results.
> 
> [1] http://www.silentpcreview.com/article859-page1.html
> 
> Thanks again!  Hope someone finds this info useful.
> 
> Matt
> 

> #!/usr/bin/perl
> 
> # License: public domain
> 
> use strict;
> 
> my $dd="/bin/dd";
> my $bs="128k";
> my $count="100000";
> 
> my @pids;
> for my $dev (@ARGV)
> {
>     my $pid = fork();
>     if ($pid == 0) # child
>     {
>         my $cmd = "$dd if=$dev of=/dev/null bs=$bs count=$count 2>&1";
>         my @output = `$cmd`;
>         my $n_lines = scalar(@output);
>         my $rc = $? >> 8;
>         if ($rc != 0 || $n_lines != 3)
>         {
>             print STDERR "ERROR: $dev: return code=$rc, n_lines=$n_lines:\n";
>             foreach my $line (@output)
>             {
>                 print STDERR "  " . $line;
>             }
>         }
>         else
>         {
>             my $line = $output[2];
>             chomp($line);
>             print "$dev: $line\n";
>         }
>         exit(0);
>     }
>     else # parent
>     {
>         push(@pids, $pid);
>     }
> }
> 
> for my $pid (@pids)
> {
>     waitpid($pid, 0);
> }

Re: southbridge/sata controller performance?

[Matt Garman]

On Mon, Jan 05, 2009 at 08:08:07AM +0100, Keld J?rn Simonsen wrote:
> What about CPU usage?

I didn't watch that while running the test, but I can certainly
re-run them.  Is watching top the best way measure CPU usage, or is
there a better/more precise (and scriptable) way to capture CPU
usage for a process or group of processes?

> What happens when you put it into one or more raids, what are then
> your combinde performance? Did you experiment with different raid
> types?

I was only doing read tests at the raw sata device level, so the
actual raid shouldn't have made a difference.  Although as an
afterthought I realized could run the script at the md-device level.

Plus, I have semi-valuable data on these raids, and don't want to go
messing with them at this point!  :)

Re: southbridge/sata controller performance?

[Keld J?rn Simonsen]

On Mon, Jan 05, 2009 at 08:21:20AM -0600, Matt Garman wrote:
> On Mon, Jan 05, 2009 at 08:08:07AM +0100, Keld J?rn Simonsen wrote:
> > What about CPU usage?
> 
> I didn't watch that while running the test, but I can certainly
> re-run them.  Is watching top the best way measure CPU usage, or is
> there a better/more precise (and scriptable) way to capture CPU
> usage for a process or group of processes?

I would  use iostat on a reasonably otherwise quiet system.

> > What happens when you put it into one or more raids, what are then
> > your combinde performance? Did you experiment with different raid
> > types?
> 
> I was only doing read tests at the raw sata device level, so the
> actual raid shouldn't have made a difference.  Although as an
> afterthought I realized could run the script at the md-device level.
> 
> Plus, I have semi-valuable data on these raids, and don't want to go
> messing with them at this point!  :)

I see. It is just that your choice of raid could affect the performance
and thus the need for southbridge bandwidth. Anyway it would be
interesting to see how much you could get out of your system with a live
fs.

best regards
keld

Re: southbridge/sata controller performance?

[Matt Garman]

[-- Attachment #1: Type: text/plain, Size: 5346 bytes --]

On Mon, Jan 05, 2009 at 05:11:47PM +0100, Keld J?rn Simonsen wrote:
> On Mon, Jan 05, 2009 at 08:21:20AM -0600, Matt Garman wrote:
> > On Mon, Jan 05, 2009 at 08:08:07AM +0100, Keld J?rn Simonsen wrote:
> > > What about CPU usage?
> > 
> > I didn't watch that while running the test, but I can certainly
> > re-run them.  Is watching top the best way measure CPU usage, or
> > is there a better/more precise (and scriptable) way to capture
> > CPU usage for a process or group of processes?
> 
> I would  use iostat on a reasonably otherwise quiet system.

OK, see attached script.  It kicks off iostat with a 1 second
repeating interval in the background, runs either dd or bonnie, then
averages the iostat results.  This scheme should give a reasonable
approximation of system load during such a test.

(Side note: it would be nice to have a shell program like "time",
that returns the CPU usage for the process that follows the command.
E.g. "cputime <my program>", and when <my program> finishes you get
output similar to the iostat CPU info.)

> > > What happens when you put it into one or more raids, what are
> > > then your combinde performance? Did you experiment with
> > > different raid types?
> > 
> > I was only doing read tests at the raw sata device level, so the
> > actual raid shouldn't have made a difference.  Although as an
> > afterthought I realized could run the script at the md-device level.
> > 
> > Plus, I have semi-valuable data on these raids, and don't want to go
> > messing with them at this point!  :)
> 
> I see. It is just that your choice of raid could affect the
> performance and thus the need for southbridge bandwidth. Anyway it
> would be interesting to see how much you could get out of your
> system with a live fs.

The script does the dd test as before, but can now also do single or
parallel bonnie++ runs.

If I didn't mention this, the machine specs are: Intel DQ35JO
motherboard (Q35 northbridge, ICH9DO southbridge), E5200 CPU, 4 GB
RAM.  md0 is four WD 7200rpm 750 GB drives on the built-in SATA
controller (ICH9); md1 is four WD RE2 5400rpm 1 TB drives on two
2-port SATA PCIe cards.  Both filesystems are XFS (created with
ubuntu 8.04 server's defaults).

I'll be swapping out this hardware for the AMD solution soon: AMD
740G northbridge, SB700 southbridge/SATA controller, 4 GB RAM and
4850e CPU.  I'll be dropping the 4x750 array and using only the
4x1TB array on the native SATA controller.  (And FWIW, in the more
distant future, I'll probably upgrade this to a six disk raid-6
scheme using linux md.)

Let me know if my script can be enhanced or upgraded in any way...
or if I overlooked anything.

EDIT: just before sending I checked the linux-raid FAQ, and it
mentions the tiobench (Threaded I/O tester) program.  Has anyone
played with this?  I'm guessing it's more sophisticated than my
little hack job.  <shrug>

Here's the results from my md1 array:

$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md1 : active raid5 sdb1[0] sdd1[3] sdc1[2] sde1[1]
      2930279808 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

md0 : active raid5 sdf1[0] sdi1[3] sdh1[2] sdg1[1]
      2197715712 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

$ sudo ~/python/sata_performance.py -d /dev/sd[bcde]
Running dd benchmark(s)...
dd for /dev/sdd: 13107200000 bytes (13 GB) copied, 173.099 s, 75.7 MB/s
dd for /dev/sde: 13107200000 bytes (13 GB) copied, 172.1 s, 76.2 MB/s
dd for /dev/sdb: 13107200000 bytes (13 GB) copied, 171.922 s, 76.2 MB/s
dd for /dev/sdc: 13107200000 bytes (13 GB) copied, 172.903 s, 75.8 MB/s
ios avg: user=0.17, nice=0.00, sys=29.55, iowait=60.26, steal=0.00, idle=10.02

$ ~/python/sata_performance.py -b 1
Running bonnie++ benchmark(s)...
bonnie instance 0 results:
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.93c       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
septictank       8G  1038  94 100356  29 48513  15  2245  98 112360  17 283.5   5
Latency             14914us    1649ms     645ms   18961us   75485us   70498us
Version 1.93c       ------Sequential Create------ --------Random Create--------
septictank          -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  1945  14 +++++ +++  7122  48  5330  46 +++++ +++  7046  49
Latency             50418us      66us   94561us   74528us      29us   27207us
1.93c,1.93c,septictank,1,1231867303,8G,,1038,94,100356,29,48513,15,2245,98,112360,17,283.5,5,16,,,,,1945,14,+++++,+++,7122,48,5330,46,+++++,+++,7046,49,14914us,1649ms,645ms,18961us,75485us,70498us,50418us,66us,94561us,74528us,29us,27207us


ios avg: user=0.42, nice=0.00, sys=16.02, iowait=15.99, steal=0.00, idle=67.57



[-- Attachment #2: sata_performance.py --]
[-- Type: text/x-python, Size: 7188 bytes --]

#!/usr/bin/python
# vim:textwidth=0 nowrap

# Simple program that attempts to determine SATA controller/southbridge
# performance.
#
#   - Performs parallel block device reads using dd.  For example, does raw
#     read performance decrease when reading from multiple drives?
#   - Can also spawn concurrent bonnie++ instances.
#   - Records continuous iostat output while running dd or bonnie, and averages
#     the results to give an approximate CPU load measurement.
#
# Developed using Python 2.5.2 on Linux.
#
# Author: Matt Garman <matthew.garman@gmail.com>
#
# License: Public Domain

import re, os, sys, getopt, subprocess, signal, tempfile

# run-time configuration/global parameters
rtcfg = {
    'iostat'     : '/usr/bin/iostat',
    'ios_opts'   : [ '-c', '-m', '1' ],
    'dd'         : '/bin/dd',
    'dd_bs'      : '128k',
    'dd_count'   : '100000',
    'bonnie'     : '/usr/local/sbin/bonnie++',
}


##############################################################################
def usage():
    print """
sata_performance.py - (attempt to) assess SATA controller/southbridge 
                      performance
Usage: %s -b <n>
       %s -d <devices>

  OPTIONS:
      -b    run bonnie++ benchmarks
            <n> is number of parallel instances of bonnie++ to run
      -d    run dd read benchmarks
            <devices> is a list of block devices from which we'll read
      -h    show this help

    """ % (sys.argv[0], sys.argv[0])


##############################################################################
def main():
    global rtcfg
    run_dd = False
    n_bonnie_instances = 0
    optstr = "dhb:"
    try:
        opts, args = getopt.getopt(sys.argv[1:], optstr)
    except getopt.GetoptError, err:
        print str(err)
        usage()
        sys.exit(2)
    for o, a in opts:
        if o == '-b': n_bonnie_instances = int(a)
        if o == '-d': run_dd = True
        if o == '-h':
            usage()
            sys.exit()

    # basic input/parameter checking
    #if n_bonnie_instances > 0 and run_dd:
    #    print >>sys.stderr, 'ERROR: use -b OR -d, but not both (use -h for help)'
    #    sys.exit()

    # first kick off iostat
    ios_cmd  = [ rtcfg['iostat'] ] + rtcfg['ios_opts']
    #print "ios_cmd = " + str(ios_cmd)
    ios_proc = subprocess.Popen(ios_cmd, shell=False, \
            stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

    if n_bonnie_instances > 0:
        run_bonnie_benchmarks(n_bonnie_instances)

    if run_dd:
        devices = list()
        if len(args) < 1:
            print >>sys.stderr, 'ERROR: no devices specified, use -h for help'
            os.kill(ios_proc.pid, signal.SIGTERM)
            sys.exit(1)
        else:
            for dev in args[0:]: devices.append(dev)
            run_dd_benchmarks(devices)

    # iostat: kill, get output, parse output
    #ios_proc.terminate() # need version 2.6
    os.kill(ios_proc.pid, signal.SIGTERM)
    ios_stdout, ios_stderr = ios_proc.communicate()
    parse_iostat_output(ios_stdout)
    sys.exit()


##############################################################################
def parse_iostat_output(iosdata):
    user = list()
    nice = list()
    system = list()
    iowait = list()
    steal = list()
    idle = list()
    is_first = True
    #print "iosdata = \"\n" + iosdata + "\n\""
    lines = iosdata.split('\n')
    for line in lines:
        line = line.strip()
        #print "line=\"" + line + "\""
        if len(line) == 0: continue
        if re.compile('^\d').match(line):
            # skip the first iostat report, it contains stats since
            # system boot
            if is_first:
                is_first = False
                continue
            #avg-cpu:  %user   %nice %system %iowait  %steal   %idle
            #           0.00    0.00    0.48    0.00    0.00   99.52
            fields = line.split()
            #print "fields = " + str(fields)
            if len(fields) == 6:
                user.append(fields[0])
                nice.append(fields[1])
                system.append(fields[2])
                iowait.append(fields[3])
                steal.append(fields[4])
                idle.append(fields[5])
            else:
                print >>sys.stderr, "parse error: line=\"" + line + "\""

    print "ios avg: user=%3.2lf, nice=%3.2lf, sys=%3.2lf, iowait=%3.2lf, steal=%3.2lf, idle=%3.2lf" % \
        (avg(user), avg(nice), avg(system), avg(iowait), avg(steal), avg(idle))


##############################################################################
def avg(numbers):
    avg = 0
    if len(numbers):
        sum = 0.0
        for n in numbers:
            sum += float(n)
        avg = sum/len(numbers)
    return avg


##############################################################################
def run_bonnie_benchmarks(n_instances):
    global rtcfg
    print "Running bonnie++ benchmark(s)..."
    # kick off n bonnie++ process(es)
    processes = dict()
    dirs = list()
    for n in range(n_instances):
        dir = tempfile.mkdtemp(dir=os.getcwd())
        dirs.append(dir)
        cmd = [ rtcfg['bonnie'], '-d', dir ]
        processes[n] = subprocess.Popen(cmd, shell=False, \
                stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

    # wait for dd jobs to finish
    for n in processes.keys():
        proc = processes[n] # convenience variable
        proc.wait()
        proc_stdout, proc_stderr = proc.communicate()
        if proc.returncode != 0:
            print >>sys.stderr, 'ERROR: bonnie instance ' + str(n) + ' return value = ' + str(proc.returncode)
            print >>sys.stderr, '       result="' + proc_stdout + '"'
        else:
            print 'bonnie instance ' + str(n) + ' results:'
            print proc_stdout + '\n'

    # cleanup temp directories
    for dir in dirs:
        os.rmdir(dir)


##############################################################################
def run_dd_benchmarks(devices):
    global rtcfg
    print "Running dd benchmark(s)..."
    # kick of dd process for each device specified on commandline
    processes = dict()
    for dev in devices:
        cmd = [ rtcfg['dd'], 'if='+dev, 'of=/dev/null', \
              'bs='+rtcfg['dd_bs'], 'count='+rtcfg['dd_count'] ]
        processes[dev] = subprocess.Popen(cmd, shell=False, \
                stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

    # wait for dd jobs to finish
    for dev in processes.keys():
        proc = processes[dev] # convenience variable
        proc.wait()
        proc_stdout, proc_stderr = proc.communicate()
        if proc.returncode != 0:
            print >>sys.stderr, 'ERROR: dd for ' + dev + ' return value = ' + str(proc.returncode)
        else:
            lines = proc_stdout.split('\n')
            if len(lines) != 4:
                print >>sys.stderr, 'ERROR: dd for ' + dev + ' output="' + proc_stdout + '"'
            else:
                print 'dd for ' + dev + ': ' + lines[2]


##############################################################################
# program entry point
##############################################################################
if '__main__' == __name__:
    main()

Re: southbridge/sata controller performance?

[Bill Davidsen]

Matt Garman wrote:
> On Sun, Jan 04, 2009 at 03:02:37PM -0600, Roger Heflin wrote:
>   
>> Yes, that is valid, so long as the md device is fairly quiet at
>> the time of the test.  Since you already have a machine, please
>> post the mb type, and number/type of sata ports and the 1-disk,
>> 2-disk, 3-disk,... numbers that you get.
>>     
>
> I attached a little Perl script I hacked up for running many
> instances of dd in parallel or individually.  It takes one or more
> block devices on the command line, and issues parallel dd reads for
> each one.  (Note my block size and count params are enormous; I made
> these huge to avoid caching effects of having 4 GB of RAM.)
>
> I have two MD arrays in my system:
>
> $ cat /proc/mdstat 
> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
> md1 : active raid5 sdb1[0] sdd1[3] sdc1[2] sde1[1]
>       2930279808 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
>             
> md0 : active raid5 sdf1[0] sdi1[3] sdh1[2] sdg1[1]
>       2197715712 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
>             
> unused devices: <none>
>
>
> This is on Ubuntu server 8.04.  Hardware is an Intel DQ35JO
> motherboard (Q35/ICH9DO chipsets) with 4 GB RAM and an E5200 CPU.
>
>     - md1 has four WD RE2 GP 1 TB hard drives (the 5400 RPM ones).
>       These are drives are connected to two 2-port PCIe SATA add-on
>       cards with the Silicon Image 3132 controller.
>
>     - md0 has four WD 750 GB drives (neither "RE2" version nor
>       GreenPower).  These drives are connected directly to the
>       on-board SATA ports (i.e. ICH9DO).
>
> For md1 (RE2 GP drives on SiI 3132), it looks like I get about 80
> MB/s when running dd on one drive:
>
> $ sudo ./sata_performance.pl /dev/sd[b]
> /dev/sdb: 13107200000 bytes (13 GB) copied, 163.757 s, 80.0 MB/s
>
> With all four drives, it only drops to about 75 MB/s per drive:
>
> $ sudo ./sata_performance.pl /dev/sd[bcde]
> /dev/sdb: 13107200000 bytes (13 GB) copied, 171.647 s, 76.4 MB/s
> /dev/sde: 13107200000 bytes (13 GB) copied, 172.052 s, 76.2 MB/s
> /dev/sdd: 13107200000 bytes (13 GB) copied, 172.737 s, 75.9 MB/s
> /dev/sdc: 13107200000 bytes (13 GB) copied, 172.777 s, 75.9 MB/s
>
> For md0 (4x750 GB drives), performance is a little better; these
> drives are definitely faster (7200 rpm vs 5400 rpm), and IIRC, the
> SiI 3132 chipsets are known to be lousy performers.
>
> Anyway, single drive performance is just shy of 100 MB/s:
>
> $ sudo ./sata_performance.pl /dev/sd[f]
> /dev/sdf: 13107200000 bytes (13 GB) copied, 133.099 s, 98.5 MB/s
>
> And I lose virtually nothing when running all four disks at once:
>
> $ sudo ./sata_performance.pl /dev/sd[fghi]
> /dev/sdi: 13107200000 bytes (13 GB) copied, 130.632 s, 100 MB/s
> /dev/sdf: 13107200000 bytes (13 GB) copied, 133.077 s, 98.5 MB/s
> /dev/sdh: 13107200000 bytes (13 GB) copied, 133.411 s, 98.2 MB/s
> /dev/sdg: 13107200000 bytes (13 GB) copied, 138.481 s, 94.6 MB/s
>
> Read performance only drops a bit when reading from all eight
> drives:
>
> $ sudo ./sata_performance.pl /dev/sd[bcdefghi]
> /dev/sdi: 13107200000 bytes (13 GB) copied, 133.086 s, 98.5 MB/s
> /dev/sdf: 13107200000 bytes (13 GB) copied, 135.59 s, 96.7 MB/s
> /dev/sdh: 13107200000 bytes (13 GB) copied, 135.86 s, 96.5 MB/s
> /dev/sdg: 13107200000 bytes (13 GB) copied, 140.215 s, 93.5 MB/s
> /dev/sdb: 13107200000 bytes (13 GB) copied, 182.402 s, 71.9 MB/s
> /dev/sdc: 13107200000 bytes (13 GB) copied, 183.234 s, 71.5 MB/s
> /dev/sdd: 13107200000 bytes (13 GB) copied, 189.025 s, 69.3 MB/s
> /dev/sde: 13107200000 bytes (13 GB) copied, 189.517 s, 69.2 MB/s
>
>
> Note: in all the above test runs, I actually ran them all three
> times to catch any variance; all runs were consistent.
>   

There are several things left out here. First, you didn't say you 
dropped caches with
  echo 1 >/proc/sys/vm/drop_caches
before you start. And running dd as a test, using the "iflag=direct" 
will allow you to isolate the effect of the system cache on i/o performance.

-- 
Bill Davidsen <davidsen@tmr.com>
  "Woe unto the statesman who makes war without a reason that will still
  be valid when the war is over..." Otto von Bismark 

Re: southbridge/sata controller performance?

[Justin Piszcz]

On Sun, 4 Jan 2009, Matt Garman wrote:

> On Sun, Jan 04, 2009 at 04:55:18AM -0500, Justin Piszcz wrote:
>>> The biggest different for RAID is what happens when multiple
>>> channels are used.   On full speed streaming read/write almost
>>> every (even the worst PCI controllers on a PCI bus) sata
>>> controller is close to equal when you only have one drive, once
>>> you use 2 drives or more things change.   If the given controller
>>> setup can actually run several drives at close to full single
>>> drive speed performance will be good, if it cannot things are
>>> going to get much slower.
>
> So, in general, the ideal is to be able to simultaneously read/write
> from multiple disks at the same speed as a single drive.
>
>> This has been discussed before, on the southbridge it is usually
>> good/you will get the maximum bandwidth for each of the 6 sata
>> ports.  After that, you need to use PCI-e lanes from the
>> northbridge.  Using PCI-e x1 slots that come off the southbridge
>> can degrade performance of the 6 SATA ports or the speed coming
>> off the drives connected to the x1 slots will be slow.
>
> Are you talking about using additional SATA controllers (i.e. via
> add-on cards)?  Or simply talking about the interconnect between the
> southbridge and the northbridge?
Interconnect between northbridge and southbridge.  When all 6 channels on
mobo are in use, speed from PCI-e x1 is ~80MiB/s, when you stop that access
from all 6 ports, I get ~120MiB/s.

>
> I think you're talking about the former, i.e. the SATA controller
> integrated in the southbridge generally ought to be fine, but if you
> start adding additional controllers that hang off the south bridge,
> their could be competition for bandwidth to the northbridge...
> right?  (And wouldn't the nvidia chips have an edge here, since they
> have everything combined into one chip?)
>
> Makes intuitive sense anyway; but in my case I'm really just curious
> about the SATA controller integrated into the southbridge; not
> concerned with additional SATA controllers.
>
>> What are you trying to accomplish?
>
> Trying to determine what motherboard would be ideal for a home NAS
> box AND have the lowest power consumption... the AMD solutions seem
> to win on the power consumption front, but I'm not sure about the
> performance.
How fast do you need? Gigabit is only ~100MiB/s.  Are you buying a 10Gbps
card?

>
>> As Roger pointed out, doing a dd is a good way to test, from each
>> disk, simultaneously, on an old Intel P965 board I was able to
>> achieve 1.0-1.1Gbyte/sec doing that with 12 Velociraptors and
>> 1.0Gbyte/sec reads on the XFS filesystem when dd (reading) large
>> data on the volume.  Approx 500-600MiB/s from the southbridge, the
>> other 400MiB/s from the northbridge.
>
> Is the "parallel dd test" valid if I do a raw read off the device,
> e.g. "dd if=/dev/sda of=/dev/null"?  All my drives are already in an
> md array, so I can't access them individually at the filesystem
> level.
Yes.  You do not need to access them at the filesystem level.  Both RAW
and on the filesystem, my benchmarks were the same when reading from 10 disks
raw or reading in a large file with dd using XFS as the filesystem.

Justin.

Re: southbridge/sata controller performance?

[Keld Jørn Simonsen]

On Sun, Jan 04, 2009 at 04:32:27PM -0500, Justin Piszcz wrote:
> 
> 
> On Sun, 4 Jan 2009, Matt Garman wrote:
> 
> >On Sun, Jan 04, 2009 at 04:55:18AM -0500, Justin Piszcz wrote:
> >
> >>What are you trying to accomplish?
> >
> >Trying to determine what motherboard would be ideal for a home NAS
> >box AND have the lowest power consumption... the AMD solutions seem
> >to win on the power consumption front, but I'm not sure about the
> >performance.

> How fast do you need? Gigabit is only ~100MiB/s.  Are you buying a 10Gbps
> card?

My impression is that using on-mobo sata controllers gives you adequate
bandwidth. SATA-controllers with 20 Gbit/s - or 2,5 GB/s bidirectional
speeds are more than adequate for say 4 disks of 80 - 120 MB/s speed.
And anyway, if you run in a multiprocess environment the random access
read or write speed per disk is normally only about half of the
sequential speed.

I have a mobo with 2 SATA controllers with 4 ports each, with my GB
disks, it can generate max 700 MB/s which is much less than the 2,5 GB/s
that the southbridge can deliver.

Using 1 Gbit/s ethernet connections may easily become a bottleneck. 

We do have a bottleneck section on our wiki:
http://linux-raid.osdl.org/index.php/Performance#Bottlenecks

> >>As Roger pointed out, doing a dd is a good way to test, from each
> >>disk, simultaneously, on an old Intel P965 board I was able to
> >>achieve 1.0-1.1Gbyte/sec doing that with 12 Velociraptors and
> >>1.0Gbyte/sec reads on the XFS filesystem when dd (reading) large
> >>data on the volume.  Approx 500-600MiB/s from the southbridge, the
> >>other 400MiB/s from the northbridge.
> >
> >Is the "parallel dd test" valid if I do a raw read off the device,
> >e.g. "dd if=/dev/sda of=/dev/null"?  All my drives are already in an
> >md array, so I can't access them individually at the filesystem
> >level.
> Yes.  You do not need to access them at the filesystem level.  Both RAW
> and on the filesystem, my benchmarks were the same when reading from 10 
> disks
> raw or reading in a large file with dd using XFS as the filesystem.

My impression is different, it does matter for certain raid types in a parallel dd test
whether you run it off the raw devices or off a file system. At least if you dd
different files in parallel off the same device.

best regards
keld