Linux RAID subsystem development
 help / color / mirror / Atom feed
From: Justin Piszcz <jpiszcz@lucidpixels.com>
To: Jan Wagner <jwagner@kurp.hut.fi>
Cc: linux-raid@vger.kernel.org
Subject: Re: single RAID slower than aggregate multi-RAID?
Date: Fri, 15 Aug 2008 15:35:42 -0400 (EDT)	[thread overview]
Message-ID: <alpine.DEB.1.10.0808151528090.25378@p34.internal.lan> (raw)
In-Reply-To: <alpine.DEB.1.10.0808151607560.5400@kurp.hut.fi>



On Fri, 15 Aug 2008, Jan Wagner wrote:

> Hi,
>
> at work we have been experimenting with SATA-II port multipliers and SATA 
> controllers (FIS-based). For some reason the RAID-0 performance is relatively 
> awful.
>
> We have one 4xeSATA controller card ADSA3GPX8-4E, twelve SpinPoint F1 disks 
> that handle ~110MB/sec, and four port Addonics port multiplier boxes. Ubuntu, 
> kernel 2.6.24-19-generic x86_64, 2 x AMD2212 Opteron on an Asus L1N64-SLI WS.
>
> We did two sequential writing tests. One test with all 12 disks behind the 
> controller placed into one mdadm RAID-0, 512k..2048k chunksize. We get ~475 
> MB/s.
>
> The second test used four "mdadm --create"'s to reassign the 12 drives into 
> four RAID-0 with 3 disks in each. All four raids were simultaneously written 
> to. We get ~250 MB/s/raid and 1000 MB/s aggregate.
>
> So the controller can handle at least 1000 MB/s.
>
> My question is why is the single RAID setup more than half slower? Any ideas?
Because you are using port-multipliers, they do not allocate full 
bandwidth to each HDD simultaneously.  You will get a maximum of 3gbps per 
each port, but if you are reading from 12 ports into a single 3gbps port?

1-
2--
3---
4-----
5-------
6------------->
7------------->
8-------
9-----
10---
11--
12-

Fewer drives means less requests to handle:

1--->
2--->--> port1 
3--->

With that you should be able to max out the sata link without a problem 
and at the same time, not bombard the link with too many requests.

From: http://www.serialata.org/portmultiplier.asp

Here is the answer to your question:
While it is possible to connect up to 15 drives to each SATA PM port via a 
port multiplier, drive connectivity is practically limited to the maximum 
available bandwidth on the 3Gb/s link. Sustained I/O rates from the drives 
are kept to within the 3Gb/s host port connection limit for maximum 
efficiency and performance.

> Is there some kernel per-device bandwidth throttling?
Answer is, if you want speed, don't use port multipliers.

On my old P965 board I have optimized where to put the drives and the 
speeds I get with Linux SW RAID5 with 10 velociraptors are as folows:

About the same for write:
$ dd if=/dev/zero of=bigfile.1 bs=1M count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 20.4056 s, 526 MB/s

$ dd if=bigfile.1 of=/dev/null bs=1M count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 10.2841 s, 1.0 GB/s

Justin.


  reply	other threads:[~2008-08-15 19:35 UTC|newest]

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2008-08-15 13:55 single RAID slower than aggregate multi-RAID? Jan Wagner
2008-08-15 19:35 ` Justin Piszcz [this message]
2008-08-15 21:53   ` Richard Scobie
2008-08-15 22:02 ` Neil Brown
2008-08-18 10:48   ` Jan Wagner
2008-08-18 11:54     ` Neil Brown
2008-08-18 12:27       ` Jan Wagner
2008-08-20 12:24         ` Jan Wagner
2008-08-21  7:34           ` Peter Grandi
2008-08-26  8:27             ` Jan Wagner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=alpine.DEB.1.10.0808151528090.25378@p34.internal.lan \
    --to=jpiszcz@lucidpixels.com \
    --cc=jwagner@kurp.hut.fi \
    --cc=linux-raid@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox