Linux RAID subsystem development
 help / color / mirror / Atom feed
* single RAID slower than aggregate multi-RAID?
@ 2008-08-15 13:55 Jan Wagner
  2008-08-15 19:35 ` Justin Piszcz
  2008-08-15 22:02 ` Neil Brown
  0 siblings, 2 replies; 10+ messages in thread
From: Jan Wagner @ 2008-08-15 13:55 UTC (permalink / raw)
  To: linux-raid

Hi,

at work we have been experimenting with SATA-II port multipliers and SATA 
controllers (FIS-based). For some reason the RAID-0 performance is 
relatively awful.

We have one 4xeSATA controller card ADSA3GPX8-4E, twelve SpinPoint F1 
disks that handle ~110MB/sec, and four port Addonics port multiplier 
boxes. Ubuntu, kernel 2.6.24-19-generic x86_64, 2 x AMD2212 Opteron on an 
Asus L1N64-SLI WS.

We did two sequential writing tests. One test with all 12 disks 
behind the controller placed into one mdadm RAID-0, 512k..2048k 
chunksize. We get ~475 MB/s.

The second test used four "mdadm --create"'s to reassign the 12 drives 
into four RAID-0 with 3 disks in each. All four raids were simultaneously 
written to. We get ~250 MB/s/raid and 1000 MB/s aggregate.

So the controller can handle at least 1000 MB/s.

My question is why is the single RAID setup more than half slower? Any 
ideas?

Is there some kernel per-device bandwidth throttling?

I already tried: O_DIRECT, posix_fadvise(), madvise(), pdflush tuning, 
different test programs, using a raw /dev/md{0,1,2,3} vs testing 
the xfs filesystem. None of them get the single RAID performance over 475 
MB/s. CPU load isn't a likely problem either. To check I wrote a threaded 
program[1] that has threads writing in-sequence into the same output file. 
All 4 cores loaded very evenly and are >=60% idle. But the rate is still 
475 MB/s. Puzzling and frustrating :-)

Is there still something that a desperate person could try to tune to get 
this lousy mdadm raid throughput improved?? :-)

Thanks in advance,

  - Jan

[1] http://www.metsahovi.fi/~jwagner/tmp/wr-nexgen-threaded.c


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: single RAID slower than aggregate multi-RAID?
  2008-08-15 13:55 single RAID slower than aggregate multi-RAID? Jan Wagner
@ 2008-08-15 19:35 ` Justin Piszcz
  2008-08-15 21:53   ` Richard Scobie
  2008-08-15 22:02 ` Neil Brown
  1 sibling, 1 reply; 10+ messages in thread
From: Justin Piszcz @ 2008-08-15 19:35 UTC (permalink / raw)
  To: Jan Wagner; +Cc: linux-raid



On Fri, 15 Aug 2008, Jan Wagner wrote:

> Hi,
>
> at work we have been experimenting with SATA-II port multipliers and SATA 
> controllers (FIS-based). For some reason the RAID-0 performance is relatively 
> awful.
>
> We have one 4xeSATA controller card ADSA3GPX8-4E, twelve SpinPoint F1 disks 
> that handle ~110MB/sec, and four port Addonics port multiplier boxes. Ubuntu, 
> kernel 2.6.24-19-generic x86_64, 2 x AMD2212 Opteron on an Asus L1N64-SLI WS.
>
> We did two sequential writing tests. One test with all 12 disks behind the 
> controller placed into one mdadm RAID-0, 512k..2048k chunksize. We get ~475 
> MB/s.
>
> The second test used four "mdadm --create"'s to reassign the 12 drives into 
> four RAID-0 with 3 disks in each. All four raids were simultaneously written 
> to. We get ~250 MB/s/raid and 1000 MB/s aggregate.
>
> So the controller can handle at least 1000 MB/s.
>
> My question is why is the single RAID setup more than half slower? Any ideas?
Because you are using port-multipliers, they do not allocate full 
bandwidth to each HDD simultaneously.  You will get a maximum of 3gbps per 
each port, but if you are reading from 12 ports into a single 3gbps port?

1-
2--
3---
4-----
5-------
6------------->
7------------->
8-------
9-----
10---
11--
12-

Fewer drives means less requests to handle:

1--->
2--->--> port1 
3--->

With that you should be able to max out the sata link without a problem 
and at the same time, not bombard the link with too many requests.

From: http://www.serialata.org/portmultiplier.asp

Here is the answer to your question:
While it is possible to connect up to 15 drives to each SATA PM port via a 
port multiplier, drive connectivity is practically limited to the maximum 
available bandwidth on the 3Gb/s link. Sustained I/O rates from the drives 
are kept to within the 3Gb/s host port connection limit for maximum 
efficiency and performance.

> Is there some kernel per-device bandwidth throttling?
Answer is, if you want speed, don't use port multipliers.

On my old P965 board I have optimized where to put the drives and the 
speeds I get with Linux SW RAID5 with 10 velociraptors are as folows:

About the same for write:
$ dd if=/dev/zero of=bigfile.1 bs=1M count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 20.4056 s, 526 MB/s

$ dd if=bigfile.1 of=/dev/null bs=1M count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 10.2841 s, 1.0 GB/s

Justin.


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: single RAID slower than aggregate multi-RAID?
  2008-08-15 19:35 ` Justin Piszcz
@ 2008-08-15 21:53   ` Richard Scobie
  0 siblings, 0 replies; 10+ messages in thread
From: Richard Scobie @ 2008-08-15 21:53 UTC (permalink / raw)
  To: Justin Piszcz; +Cc: Jan Wagner, linux-raid

Justin Piszcz wrote:

> Because you are using port-multipliers, they do not allocate full 
> bandwidth to each HDD simultaneously.  You will get a maximum of 3gbps 
> per each port, but if you are reading from 12 ports into a single 3gbps 
> port?

Jan's setup is 3 drives on each port multiplier, which are each attached 
to a port on the controller, so should be just on the limit of the 3Gb/s 
link between controller and multiplier.

According to this, with the same controller and 12 portmultiplied 
Seagate drives on a Mac, in one single RAID 0, you should see just under 
700MB/s peak, so there may be room for improvement yet.

http://www.amug.org/amug-web/html/amug/reviews/articles/addonics/adsa3gpx8-4em/

Regards,

Richard

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: single RAID slower than aggregate multi-RAID?
  2008-08-15 13:55 single RAID slower than aggregate multi-RAID? Jan Wagner
  2008-08-15 19:35 ` Justin Piszcz
@ 2008-08-15 22:02 ` Neil Brown
  2008-08-18 10:48   ` Jan Wagner
  1 sibling, 1 reply; 10+ messages in thread
From: Neil Brown @ 2008-08-15 22:02 UTC (permalink / raw)
  To: Jan Wagner; +Cc: linux-raid

On Friday August 15, jwagner@kurp.hut.fi wrote:
> Hi,
> 
> at work we have been experimenting with SATA-II port multipliers and SATA 
> controllers (FIS-based). For some reason the RAID-0 performance is 
> relatively awful.
> 
> We have one 4xeSATA controller card ADSA3GPX8-4E, twelve SpinPoint F1 
> disks that handle ~110MB/sec, and four port Addonics port multiplier 
> boxes. Ubuntu, kernel 2.6.24-19-generic x86_64, 2 x AMD2212 Opteron on an 
> Asus L1N64-SLI WS.
> 
> We did two sequential writing tests. One test with all 12 disks 
> behind the controller placed into one mdadm RAID-0, 512k..2048k 
> chunksize. We get ~475 MB/s.
> 
> The second test used four "mdadm --create"'s to reassign the 12 drives 
> into four RAID-0 with 3 disks in each. All four raids were simultaneously 
> written to. We get ~250 MB/s/raid and 1000 MB/s aggregate.
> 
> So the controller can handle at least 1000 MB/s.
> 
> My question is why is the single RAID setup more than half slower? Any 
> ideas?
> 
> Is there some kernel per-device bandwidth throttling?

Yes, but it shouldn't be affecting you.
The kernel only allows 40% (/proc/sys/vm/dirty_ratio) of memory to be
dirty.  Once you have more than that, writes will be throttled to
avoid dirtying all of memory.  Writes to devices which are slower will
be allowed a smaller share of the 40%.
You should only need about 12Meg of dirty memory to keep your RAID0
busy, and I doubt that is even 1% of your memory.

md/raid0 has very little overhead.  It just encourages the filesystem
to send requests that are aligned with the chunks, and then sends each
request on to the target drive without any further intervention.   So
the queue for each device should be kept full, and the individual
devices should be going at full speed.

So to make sure I'm not misunderstanding you description, could you
run the two tests again: 12disk raid0 and 4*3disk raid0,  and for
each report
  mdadm -D   of each array
  output of  
     time dd if=/dev/zero of=/dev/mdXX bs=1024k
  on all arrays in parallel
Using a chunksize of 1024.

While doing this, watch the values of "Dirty" and "Writeback" in
/proc/meminfo and see how high they get.  And report MemTotal just to
get a complete picture.

Thanks,
NeilBrown

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: single RAID slower than aggregate multi-RAID?
  2008-08-15 22:02 ` Neil Brown
@ 2008-08-18 10:48   ` Jan Wagner
  2008-08-18 11:54     ` Neil Brown
  0 siblings, 1 reply; 10+ messages in thread
From: Jan Wagner @ 2008-08-18 10:48 UTC (permalink / raw)
  To: Neil Brown; +Cc: linux-raid

Hello Neil and all,

first off thanks for your interest.

On Sat, 16 Aug 2008, Neil Brown wrote:
> You should only need about 12Meg of dirty memory to keep your RAID0
> busy, and I doubt that is even 1% of your memory.

Strange. With the 12disk raid we get as much as 200MB dirty and throughput 
is slow. With the 4*3disk raid's we see just around 12MB and throughput 
is good. We have 4GB RAM.

> md/raid0 has very little overhead.  It just encourages the filesystem
> to send requests that are aligned with the chunks, and then sends each
> request on to the target drive without any further intervention.   So
> the queue for each device should be kept full, and the individual
> devices should be going at full speed.

Okay. We have been using power-of-2 raw write sizes, and in the case of 
using a filesystem, it was XFS with settings in mkfs.xfs that tell it 
about the chunk size and device count.

> So to make sure I'm not misunderstanding you description, could you
> run the two tests again: 12disk raid0 and 4*3disk raid0,  and for
> each report
>  mdadm -D   of each array
>  output of
>     time dd if=/dev/zero of=/dev/mdXX bs=1024k
>  on all arrays in parallel
> Using a chunksize of 1024.

One of our eSATA<=>SATA conversion cables got lost, so I couldn't right 
now do your test on the 4*3disk raid0 in my first posting. Instead I ran 
with 12disk and 3*4disk raid0.

http://www.metsahovi.fi/~jwagner/10g-tests/pmp/linux-raid-1.txt

473 MB/s with single raid, high 'dirty', 692 MB/s with triple raid and low 
'dirty.

Any ideas?

TIA,
  - Jan

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: single RAID slower than aggregate multi-RAID?
  2008-08-18 10:48   ` Jan Wagner
@ 2008-08-18 11:54     ` Neil Brown
  2008-08-18 12:27       ` Jan Wagner
  0 siblings, 1 reply; 10+ messages in thread
From: Neil Brown @ 2008-08-18 11:54 UTC (permalink / raw)
  To: Jan Wagner; +Cc: linux-raid

On Monday August 18, jwagner@kurp.hut.fi wrote:
> Hello Neil and all,
> 
> first off thanks for your interest.
> 
> On Sat, 16 Aug 2008, Neil Brown wrote:
> > You should only need about 12Meg of dirty memory to keep your RAID0
> > busy, and I doubt that is even 1% of your memory.
> 
> Strange. With the 12disk raid we get as much as 200MB dirty and throughput 
> is slow. With the 4*3disk raid's we see just around 12MB and throughput 
> is good. We have 4GB RAM.

The differences in Dirty and Writeback numbers are interesting.  I'm
not sure what they signify yet.


in:
> 
> http://www.metsahovi.fi/~jwagner/10g-tests/pmp/linux-raid-1.txt
> 

you appear to be doing the write tests to the three small arrays in
series, one after the other.
You need to do them in parallel to find the aggregate bandwidth of the
system.
i.e.  run

sudo /usr/bin/time dd if=/dev/zero of=/dev/md0 bs=1024k count=48000 & 
sudo /usr/bin/time dd if=/dev/zero of=/dev/md1 bs=1024k count=48000 &
sudo /usr/bin/time dd if=/dev/zero of=/dev/md2 bs=1024k count=48000 &
 
all at once in the background.  What does that produce?

NeilBrown

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: single RAID slower than aggregate multi-RAID?
  2008-08-18 11:54     ` Neil Brown
@ 2008-08-18 12:27       ` Jan Wagner
  2008-08-20 12:24         ` Jan Wagner
  0 siblings, 1 reply; 10+ messages in thread
From: Jan Wagner @ 2008-08-18 12:27 UTC (permalink / raw)
  To: Neil Brown; +Cc: linux-raid

Hi Neil,

On Mon, 18 Aug 2008, Neil Brown wrote:
>> http://www.metsahovi.fi/~jwagner/10g-tests/pmp/linux-raid-1.txt
>
> you appear to be doing the write tests to the three small arrays in
> series, one after the other.
> You need to do them in parallel to find the aggregate bandwidth of the
> system.
> i.e.  run
>
> sudo /usr/bin/time dd if=/dev/zero of=/dev/md0 bs=1024k count=48000 &
> sudo /usr/bin/time dd if=/dev/zero of=/dev/md1 bs=1024k count=48000 &
> sudo /usr/bin/time dd if=/dev/zero of=/dev/md2 bs=1024k count=48000 &
>
> all at once in the background.  What does that produce?

Sorry, I forgot to mention this in the .txt file: all the three tests 
were in fact running in parallel, just on different xterm terminals. The 
.txt is a copypaste from the three terminals.

When I used '&' backgrounding the results were the same.


But let's redo it, for completeness:

oper@abidal:~$ sudo /usr/bin/time dd if=/dev/zero of=/dev/md0 bs=1024k count=48000 &
[1] 13283
oper@abidal:~$ sudo /usr/bin/time dd if=/dev/zero of=/dev/md1 bs=1024k count=48000 &
[2] 13286
oper@abidal:~$ sudo /usr/bin/time dd if=/dev/zero of=/dev/md2 bs=1024k count=48000 &
[3] 13288

48000+0 records in
48000+0 records out
50331648000 bytes (50 GB) copied, 218.485 s, 230 MB/s
0.07user 147.77system 3:38.50elapsed 67%CPU (0avgtext+0avgdata 0maxresident)k
48inputs+98304008outputs (1major+520minor)pagefaults 0swaps
48000+0 records in
48000+0 records out
50331648000 bytes (50 GB) copied, 218.206 s, 231 MB/s
0.05user 145.51system 3:38.20elapsed 66%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+98304008outputs (0major+521minor)pagefaults 0swaps
48000+0 records in
48000+0 records out
50331648000 bytes (50 GB) copied, 218.895 s, 230 MB/s
0.06user 137.92system 3:38.89elapsed 63%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+98304008outputs (0major+520minor)pagefaults 0swaps

Kind regards,
  - Jan

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: single RAID slower than aggregate multi-RAID?
  2008-08-18 12:27       ` Jan Wagner
@ 2008-08-20 12:24         ` Jan Wagner
  2008-08-21  7:34           ` Peter Grandi
  0 siblings, 1 reply; 10+ messages in thread
From: Jan Wagner @ 2008-08-20 12:24 UTC (permalink / raw)
  To: Neil Brown; +Cc: linux-raid

Hi again,

just thought I should re-post and add the 12disk raid0 stats here as well:

On Mon, 18 Aug 2008, Jan Wagner wrote:
> When I used '&' backgrounding the results were the same.
>
> But let's redo it, for completeness, three 4disk raid0 in parallel:
>
> oper@abidal:~$ sudo /usr/bin/time dd if=/dev/zero of=/dev/md0 bs=1024k 
> count=48000 &
> [1] 13283
> oper@abidal:~$ sudo /usr/bin/time dd if=/dev/zero of=/dev/md1 bs=1024k 
> count=48000 &
> [2] 13286
> oper@abidal:~$ sudo /usr/bin/time dd if=/dev/zero of=/dev/md2 bs=1024k 
> count=48000 &
> [3] 13288
>
> 48000+0 records in
> 48000+0 records out
> 50331648000 bytes (50 GB) copied, 218.485 s, 230 MB/s
> 0.07user 147.77system 3:38.50elapsed 67%CPU (0avgtext+0avgdata 0maxresident)k
> 48inputs+98304008outputs (1major+520minor)pagefaults 0swaps
> 48000+0 records in
> 48000+0 records out
> 50331648000 bytes (50 GB) copied, 218.206 s, 231 MB/s
> 0.05user 145.51system 3:38.20elapsed 66%CPU (0avgtext+0avgdata 0maxresident)k
> 0inputs+98304008outputs (0major+521minor)pagefaults 0swaps
> 48000+0 records in
> 48000+0 records out
> 50331648000 bytes (50 GB) copied, 218.895 s, 230 MB/s
> 0.06user 137.92system 3:38.89elapsed 63%CPU (0avgtext+0avgdata 0maxresident)k
> 0inputs+98304008outputs (0major+520minor)pagefaults 0swaps

And a single 12disk raid0:

oper@abidal:~$ sudo /usr/bin/time dd if=/dev/zero of=/dev/md0 bs=1024k
48000+0 records in
48000+0 records out
50331648000 bytes (50 GB) copied, 106.334 s, 473 MB/s
0.10user 90.41system 1:46.38elapsed 85%CPU (0avgtext+0avgdata 
0maxresident)k
88inputs+98304000outputs (2major+526minor)pagefaults 0swaps

The Dirty/Writeback values were at 
http://www.metsahovi.fi/~jwagner/10g-tests/pmp/linux-raid-1.txt

For the 3 x 4disk raid0s the values were ~390MB Writeback and ~14MB 
Dirty. Aggretate write rate 690MB/sec.

For the 1 x 12disk raid0 just ~14MB Writeback and ~190MB Dirty. Write 
rate 473 MB/sec.

Didn't mention before but the kernel is the Ubuntu 2.6.24-19-generic SMP, 
and it is on a 4 core system. I also booted with 'nosmp', but all the 
write rates were approximately the same.

Do you have any more thoughts or possible tests?

Perhaps I should read up on a kernel profiling howto...?

Kind regards,
  - Jan

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: single RAID slower than aggregate multi-RAID?
  2008-08-20 12:24         ` Jan Wagner
@ 2008-08-21  7:34           ` Peter Grandi
  2008-08-26  8:27             ` Jan Wagner
  0 siblings, 1 reply; 10+ messages in thread
From: Peter Grandi @ 2008-08-21  7:34 UTC (permalink / raw)
  To: Linux RAID

> [ ... ] For the 3 x 4disk raid0s the values were ~390MB
> Writeback and ~14MB Dirty. Aggretate write rate 690MB/sec.

> For the 1 x 12disk raid0 just ~14MB Writeback and ~190MB
> Dirty. Write rate 473 MB/sec. [ ... ]

That coincides with my own experience. But note that it is a very
special case, where optimum speed is reached only if requests hit
the member block devices in exactly the "right" way. In most
workloads latency also is a big part, so the ability to issue long
sequential streams of back-to-back requests is less important.

Anyhow my impression is that the recent 2.6 linux block IO
subsystem has some rather misguided behaviours (but then so much
of the Linux kernel contains several "brilliant" ideas of random
busybodies, most notably in paging, swapping and memory mangement),
and MD does not interact well with them.

As previously mentioned, on the read side the amount of data read
in a single transaction seems to be exactly the same as the block
device read ahead, which means that very large read-ahead values
give (much) better performance than smaller values, which would
not be the case if the block layer was streaming.

As to the write side, my current guess is that the effect above is
due to similar non streaming behaviour, probably dependent on the
unwise "plugging" idea that one of the usual busybodies has added
to the block layer. I have tried to disable that but there seem to
be deeper problems. I suspect also request reordering in the
elevator algorithms; using 'noop' as the elevator sometimes helps.

Overall the current Linux block layer instead of being based on
streaming seems to be based on batching of requests, and this
interacts pooly with MD, as the batch sizes and their passing on
timing might not be those that can best keep several MD member
block devices busy.

To see the detailed effects of all these layers of "brilliant"
policies, IO rates on individual devices should be looked at, and
I use something like this command line (with a very tall terminal
window).

  watch -n2 iostat -d -m /dev/mdXXX /dev/sdNNN /dev/sdMMM ... 1 2

My experience is that often the traffic is not quite evenly
balanced across the member drives, and when it is, the rate
sometimes is well below the one at which the member device
can operate.

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: single RAID slower than aggregate multi-RAID?
  2008-08-21  7:34           ` Peter Grandi
@ 2008-08-26  8:27             ` Jan Wagner
  0 siblings, 0 replies; 10+ messages in thread
From: Jan Wagner @ 2008-08-26  8:27 UTC (permalink / raw)
  To: Peter Grandi; +Cc: Linux RAID


On Thu, 21 Aug 2008, Peter Grandi wrote:
>> [ ... ] For the 3 x 4disk raid0s the values were ~390MB
>> Writeback and ~14MB Dirty. Aggretate write rate 690MB/sec.
>
>> For the 1 x 12disk raid0 just ~14MB Writeback and ~190MB
>> Dirty. Write rate 473 MB/sec. [ ... ]

There was an old Microsoft research article from 2004/2005:
http://research.microsoft.com/barc/Sequential_IO/

Linear scaling with more drives. Versus kernel 2.6.22/.24 on our test 
machine where the rate saturates already with ~5 drives in raid0. I'm not 
familiar enough with md source to know what to fix.

> That coincides with my own experience. But note that it is a very
> special case, where optimum speed is reached only if requests hit
> the member block devices in exactly the "right" way. In most
> workloads latency also is a big part, so the ability to issue long
> sequential streams of back-to-back requests is less important.

Data streaming is quite commonplace. The kernel should be able to handle 
it efficiently.

Getting requests hit the "right" way should be deterministic. Achievable 
with correct settings. I have already tried a few tunings, to no avail. 
Specifically:

QT=64
NCQD=8
for drv in $@; do
  echo $QT > /sys/block/${drv}/queue/iosched/quantum
  cat /sys/block/${drv}/queue/max_hw_sectors_kb > /sys/block/${drv}/queue/max_sectors_kb
  blockdev --setra 16384 /dev/${drv}
  echo "noop" > /sys/block/${drv}/queue/scheduler
  echo "${NCQD}" > /sys/block/${drv}/device/queue_depth
  echo "$(($NCQD * 2))" > /sys/block/${drv}/device/nr_requests
done

On average this doubles the sequential read performance. But not with 
sequential write.

Set "nop" elevator on the md slaves. No difference.

Additionally tried telling the kernel that sequential I/O is expected 
(madvise, posix_fadvise) and trying to bypass some caching w O_DIRECT, 
even tested mmap() and msync(), and tuned /proc/sys/vm/* settings. Always 
doing raw I/O to /dev/mdX.

But the kernel or md or pdflush do not "get" any of the obvious hints that 
they could dispatch requests optimally.

About elevators, MD itself has no "input side" elevator setting AFAICT.

Perhaps this is the problem (?)

> Overall the current Linux block layer instead of being based on
> streaming seems to be based on batching of requests, and this
> interacts pooly with MD, as the batch sizes and their passing on
> timing might not be those that can best keep several MD member
> block devices busy.

Sounds plausible.

> To see the detailed effects of all these layers of "brilliant"
> policies, IO rates on individual devices should be looked at, and
> I use something like this command line (with a very tall terminal
> window).
>
>  watch -n2 iostat -d -m /dev/mdXXX /dev/sdNNN /dev/sdMMM ... 1 2

Hmm, I tried it. With a single 4-disk raid0 2048k chunk, all disks behind 
same PMP:

$ grep MemTotal /proc/meminfo
MemTotal:      4044384 kB

screen1$ dd if=/dev/zero of=/dev/md0 bs=2048k
screen2$ watch -n2 iostat -d -m /dev/sd{b,c,d,e} 1 2
Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sdb             865.00         0.00        45.80          0         45
sdc             881.00         0.00        45.95          0         45
sdd             881.00         0.00        45.87          0         45
sde             898.00         0.00        45.55          0         45

With the 12-disk raid0, with linear scaling expected to do 12*46MB/s 
or 552MB/s:

screen1$ dd if=/dev/zero of=/dev/md0 bs=2048k
screen2$ watch -n2 iostat -d -m /dev/sd{b,c,d,e,f,g,h,i,j,k,l,m} 1 2
Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sdb             473.00         0.00        17.71          0         17
sdc             513.00         0.00        18.88          0         18
sdd             495.00         0.00        19.41          0         19
sde             520.00         0.00        19.12          0         19
sdf             536.00         0.00        19.37          0         19
sdg             479.00         0.00        19.39          0         19
sdh             489.00         0.00        19.10          0         19
sdi             440.00         0.00        19.43          0         19
sdj             468.00         0.00        18.68          0         18
sdk             428.00         0.00        17.40          0         17
sdl             404.00         0.00        17.58          0         17
sdm             464.00         0.00        17.14          0         17
screen1$ # 42960158720 bytes (43 GB) copied, 210.794 s, 204 MB/s

So the rates are quite evenly distributed within maybe +-1.5 MB/s from the 
mean.

> My experience is that often the traffic is not quite evenly
> balanced across the member drives, and when it is, the rate
> sometimes is well below the one at which the member device
> can operate.

Perhaps what I see here is yet another problem.

Now, things get curious and fast with O_DIRECT and a block size up to half 
of the available 4GB memory. Without O_DIRECT the rates are always quite 
low.

screen1$ dd if=/dev/zero of=/dev/md0 bs=16M oflag=direct
screen2$ watch -n2 iostat -d -m /dev/sd{b,c,d,e,f,g,h,i,j,k,l,m} 1 2
Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sdb              56.00         0.00        28.00          0         28
...
sdm              52.00         0.00        26.00          0         26
screen1$ # 14612955136 bytes (15 GB) copied, 44.7189 s, 327 MB/s
screen1$ sudo dd if=/dev/md0 of=/dev/null bs=16M iflag=direct
screen1$ # 14092861440 bytes (14 GB) copied, 31.681 s, 445 MB/s
------------------------------------------------------------------
screen1$ dd if=/dev/zero of=/dev/md0 bs=64M oflag=direct
screen2$ watch -n2 iostat -d -m /dev/sd{b,c,d,e,f,g,h,i,j,k,l,m} 1 2
Device:           tps     MB_read/s    MB_wrtn/s    MB_read     MB_wrtn
sdb             70.00          0.00        35.00          0          35
...
sdm             65.00          0.00        34.50          0          34
screen1$ # 12683575296 bytes (13 GB) copied, 28.4022 s, 447 MB/s
screen1$ sudo dd if=/dev/md0 of=/dev/null bs=64M iflag=direct
screen1$ # 79054241792 bytes (79 GB) copied, 128.795 s, 614 MB/s
------------------------------------------------------------------
screen1$ dd if=/dev/zero of=/dev/md0 bs=512M oflag=direct
screen2$ watch -n2 iostat -d -m /dev/sd{b,c,d,e,f,g,h,i,j,k,l,m} 1 2
sdb              25.00         0.00        42.00          0         42
...
sdm              23.00         0.00        42.00          0         42
screen1$ # 25769803776 bytes (26 GB) copied, 53.3836 s, 483 MB/
screen1$ sudo dd if=/dev/md0 of=/dev/null bs=512M iflag=direct
screen1$ # 23622320128 bytes (24 GB) copied, 36.4848 s, 647 MB/s
------------------------------------------------------------------
screen1$ sudo dd if=/dev/zero of=/dev/md0 bs=1024M oflag=direct
screen2$ watch -n2 iostat -d -m /dev/sd{b,c,d,e,f,g,h,i,j,k,l,m} 1 2
sdb              17.00         0.00        24.50          0         24
...
sde              14.00         0.00        19.50          0         19
sdf              17.00         0.00        24.50          0         24
...
sdk              14.00         0.00        11.50          0         11
sdl              15.00         0.00        26.50          0         26
sdm              15.00         0.00        26.50          0         26
screen1$ # the rates jump around between 20MB/s, 60MB/s
screen1$ # 20401094656 bytes (20 GB) copied, 41.9148 s, 487 MB/s
screen1$ sudo dd if=/dev/md0 of=/dev/null bs=1024M iflag=direct
screen1$ # 23622320128 bytes (24 GB) copied, 36.4848 s, 647 MB/s
------------------------------------------------------------------
screen1$ dd if=/dev/zero of=/dev/md0 bs=2048M oflag=direct
screen2$ watch -n2 iostat -d -m /dev/sd{b,c,d,e,f,g,h,i,j,k,l,m} 1 2
Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sdb              72.00         0.00        53.00          0         53
...
sdm              74.00         0.00        52.00          0         52
screen1$ # the above happens in bursts and there are pauses with 0MB/s
screen1$ # 19327315968 bytes (19 GB) copied, 41.8337 s, 462 MB/s
screen1$ sudo dd if=/dev/md0 of=/dev/null bs=2048M iflag=direct
screen1$ # 27917234176 bytes (28 GB) copied, 46.4147 s, 601 MB/s

I guess large writes "force" something in the vm...

  - Jan

^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2008-08-26  8:27 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2008-08-15 13:55 single RAID slower than aggregate multi-RAID? Jan Wagner
2008-08-15 19:35 ` Justin Piszcz
2008-08-15 21:53   ` Richard Scobie
2008-08-15 22:02 ` Neil Brown
2008-08-18 10:48   ` Jan Wagner
2008-08-18 11:54     ` Neil Brown
2008-08-18 12:27       ` Jan Wagner
2008-08-20 12:24         ` Jan Wagner
2008-08-21  7:34           ` Peter Grandi
2008-08-26  8:27             ` Jan Wagner

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox