* single RAID slower than aggregate multi-RAID?
@ 2008-08-15 13:55 Jan Wagner
2008-08-15 19:35 ` Justin Piszcz
2008-08-15 22:02 ` Neil Brown
0 siblings, 2 replies; 10+ messages in thread
From: Jan Wagner @ 2008-08-15 13:55 UTC (permalink / raw)
To: linux-raid
Hi,
at work we have been experimenting with SATA-II port multipliers and SATA
controllers (FIS-based). For some reason the RAID-0 performance is
relatively awful.
We have one 4xeSATA controller card ADSA3GPX8-4E, twelve SpinPoint F1
disks that handle ~110MB/sec, and four port Addonics port multiplier
boxes. Ubuntu, kernel 2.6.24-19-generic x86_64, 2 x AMD2212 Opteron on an
Asus L1N64-SLI WS.
We did two sequential writing tests. One test with all 12 disks
behind the controller placed into one mdadm RAID-0, 512k..2048k
chunksize. We get ~475 MB/s.
The second test used four "mdadm --create"'s to reassign the 12 drives
into four RAID-0 with 3 disks in each. All four raids were simultaneously
written to. We get ~250 MB/s/raid and 1000 MB/s aggregate.
So the controller can handle at least 1000 MB/s.
My question is why is the single RAID setup more than half slower? Any
ideas?
Is there some kernel per-device bandwidth throttling?
I already tried: O_DIRECT, posix_fadvise(), madvise(), pdflush tuning,
different test programs, using a raw /dev/md{0,1,2,3} vs testing
the xfs filesystem. None of them get the single RAID performance over 475
MB/s. CPU load isn't a likely problem either. To check I wrote a threaded
program[1] that has threads writing in-sequence into the same output file.
All 4 cores loaded very evenly and are >=60% idle. But the rate is still
475 MB/s. Puzzling and frustrating :-)
Is there still something that a desperate person could try to tune to get
this lousy mdadm raid throughput improved?? :-)
Thanks in advance,
- Jan
[1] http://www.metsahovi.fi/~jwagner/tmp/wr-nexgen-threaded.c
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: single RAID slower than aggregate multi-RAID?
2008-08-15 13:55 single RAID slower than aggregate multi-RAID? Jan Wagner
@ 2008-08-15 19:35 ` Justin Piszcz
2008-08-15 21:53 ` Richard Scobie
2008-08-15 22:02 ` Neil Brown
1 sibling, 1 reply; 10+ messages in thread
From: Justin Piszcz @ 2008-08-15 19:35 UTC (permalink / raw)
To: Jan Wagner; +Cc: linux-raid
On Fri, 15 Aug 2008, Jan Wagner wrote:
> Hi,
>
> at work we have been experimenting with SATA-II port multipliers and SATA
> controllers (FIS-based). For some reason the RAID-0 performance is relatively
> awful.
>
> We have one 4xeSATA controller card ADSA3GPX8-4E, twelve SpinPoint F1 disks
> that handle ~110MB/sec, and four port Addonics port multiplier boxes. Ubuntu,
> kernel 2.6.24-19-generic x86_64, 2 x AMD2212 Opteron on an Asus L1N64-SLI WS.
>
> We did two sequential writing tests. One test with all 12 disks behind the
> controller placed into one mdadm RAID-0, 512k..2048k chunksize. We get ~475
> MB/s.
>
> The second test used four "mdadm --create"'s to reassign the 12 drives into
> four RAID-0 with 3 disks in each. All four raids were simultaneously written
> to. We get ~250 MB/s/raid and 1000 MB/s aggregate.
>
> So the controller can handle at least 1000 MB/s.
>
> My question is why is the single RAID setup more than half slower? Any ideas?
Because you are using port-multipliers, they do not allocate full
bandwidth to each HDD simultaneously. You will get a maximum of 3gbps per
each port, but if you are reading from 12 ports into a single 3gbps port?
1-
2--
3---
4-----
5-------
6------------->
7------------->
8-------
9-----
10---
11--
12-
Fewer drives means less requests to handle:
1--->
2--->--> port1
3--->
With that you should be able to max out the sata link without a problem
and at the same time, not bombard the link with too many requests.
From: http://www.serialata.org/portmultiplier.asp
Here is the answer to your question:
While it is possible to connect up to 15 drives to each SATA PM port via a
port multiplier, drive connectivity is practically limited to the maximum
available bandwidth on the 3Gb/s link. Sustained I/O rates from the drives
are kept to within the 3Gb/s host port connection limit for maximum
efficiency and performance.
> Is there some kernel per-device bandwidth throttling?
Answer is, if you want speed, don't use port multipliers.
On my old P965 board I have optimized where to put the drives and the
speeds I get with Linux SW RAID5 with 10 velociraptors are as folows:
About the same for write:
$ dd if=/dev/zero of=bigfile.1 bs=1M count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 20.4056 s, 526 MB/s
$ dd if=bigfile.1 of=/dev/null bs=1M count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 10.2841 s, 1.0 GB/s
Justin.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: single RAID slower than aggregate multi-RAID?
2008-08-15 19:35 ` Justin Piszcz
@ 2008-08-15 21:53 ` Richard Scobie
0 siblings, 0 replies; 10+ messages in thread
From: Richard Scobie @ 2008-08-15 21:53 UTC (permalink / raw)
To: Justin Piszcz; +Cc: Jan Wagner, linux-raid
Justin Piszcz wrote:
> Because you are using port-multipliers, they do not allocate full
> bandwidth to each HDD simultaneously. You will get a maximum of 3gbps
> per each port, but if you are reading from 12 ports into a single 3gbps
> port?
Jan's setup is 3 drives on each port multiplier, which are each attached
to a port on the controller, so should be just on the limit of the 3Gb/s
link between controller and multiplier.
According to this, with the same controller and 12 portmultiplied
Seagate drives on a Mac, in one single RAID 0, you should see just under
700MB/s peak, so there may be room for improvement yet.
http://www.amug.org/amug-web/html/amug/reviews/articles/addonics/adsa3gpx8-4em/
Regards,
Richard
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: single RAID slower than aggregate multi-RAID?
2008-08-15 13:55 single RAID slower than aggregate multi-RAID? Jan Wagner
2008-08-15 19:35 ` Justin Piszcz
@ 2008-08-15 22:02 ` Neil Brown
2008-08-18 10:48 ` Jan Wagner
1 sibling, 1 reply; 10+ messages in thread
From: Neil Brown @ 2008-08-15 22:02 UTC (permalink / raw)
To: Jan Wagner; +Cc: linux-raid
On Friday August 15, jwagner@kurp.hut.fi wrote:
> Hi,
>
> at work we have been experimenting with SATA-II port multipliers and SATA
> controllers (FIS-based). For some reason the RAID-0 performance is
> relatively awful.
>
> We have one 4xeSATA controller card ADSA3GPX8-4E, twelve SpinPoint F1
> disks that handle ~110MB/sec, and four port Addonics port multiplier
> boxes. Ubuntu, kernel 2.6.24-19-generic x86_64, 2 x AMD2212 Opteron on an
> Asus L1N64-SLI WS.
>
> We did two sequential writing tests. One test with all 12 disks
> behind the controller placed into one mdadm RAID-0, 512k..2048k
> chunksize. We get ~475 MB/s.
>
> The second test used four "mdadm --create"'s to reassign the 12 drives
> into four RAID-0 with 3 disks in each. All four raids were simultaneously
> written to. We get ~250 MB/s/raid and 1000 MB/s aggregate.
>
> So the controller can handle at least 1000 MB/s.
>
> My question is why is the single RAID setup more than half slower? Any
> ideas?
>
> Is there some kernel per-device bandwidth throttling?
Yes, but it shouldn't be affecting you.
The kernel only allows 40% (/proc/sys/vm/dirty_ratio) of memory to be
dirty. Once you have more than that, writes will be throttled to
avoid dirtying all of memory. Writes to devices which are slower will
be allowed a smaller share of the 40%.
You should only need about 12Meg of dirty memory to keep your RAID0
busy, and I doubt that is even 1% of your memory.
md/raid0 has very little overhead. It just encourages the filesystem
to send requests that are aligned with the chunks, and then sends each
request on to the target drive without any further intervention. So
the queue for each device should be kept full, and the individual
devices should be going at full speed.
So to make sure I'm not misunderstanding you description, could you
run the two tests again: 12disk raid0 and 4*3disk raid0, and for
each report
mdadm -D of each array
output of
time dd if=/dev/zero of=/dev/mdXX bs=1024k
on all arrays in parallel
Using a chunksize of 1024.
While doing this, watch the values of "Dirty" and "Writeback" in
/proc/meminfo and see how high they get. And report MemTotal just to
get a complete picture.
Thanks,
NeilBrown
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: single RAID slower than aggregate multi-RAID?
2008-08-15 22:02 ` Neil Brown
@ 2008-08-18 10:48 ` Jan Wagner
2008-08-18 11:54 ` Neil Brown
0 siblings, 1 reply; 10+ messages in thread
From: Jan Wagner @ 2008-08-18 10:48 UTC (permalink / raw)
To: Neil Brown; +Cc: linux-raid
Hello Neil and all,
first off thanks for your interest.
On Sat, 16 Aug 2008, Neil Brown wrote:
> You should only need about 12Meg of dirty memory to keep your RAID0
> busy, and I doubt that is even 1% of your memory.
Strange. With the 12disk raid we get as much as 200MB dirty and throughput
is slow. With the 4*3disk raid's we see just around 12MB and throughput
is good. We have 4GB RAM.
> md/raid0 has very little overhead. It just encourages the filesystem
> to send requests that are aligned with the chunks, and then sends each
> request on to the target drive without any further intervention. So
> the queue for each device should be kept full, and the individual
> devices should be going at full speed.
Okay. We have been using power-of-2 raw write sizes, and in the case of
using a filesystem, it was XFS with settings in mkfs.xfs that tell it
about the chunk size and device count.
> So to make sure I'm not misunderstanding you description, could you
> run the two tests again: 12disk raid0 and 4*3disk raid0, and for
> each report
> mdadm -D of each array
> output of
> time dd if=/dev/zero of=/dev/mdXX bs=1024k
> on all arrays in parallel
> Using a chunksize of 1024.
One of our eSATA<=>SATA conversion cables got lost, so I couldn't right
now do your test on the 4*3disk raid0 in my first posting. Instead I ran
with 12disk and 3*4disk raid0.
http://www.metsahovi.fi/~jwagner/10g-tests/pmp/linux-raid-1.txt
473 MB/s with single raid, high 'dirty', 692 MB/s with triple raid and low
'dirty.
Any ideas?
TIA,
- Jan
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: single RAID slower than aggregate multi-RAID?
2008-08-18 10:48 ` Jan Wagner
@ 2008-08-18 11:54 ` Neil Brown
2008-08-18 12:27 ` Jan Wagner
0 siblings, 1 reply; 10+ messages in thread
From: Neil Brown @ 2008-08-18 11:54 UTC (permalink / raw)
To: Jan Wagner; +Cc: linux-raid
On Monday August 18, jwagner@kurp.hut.fi wrote:
> Hello Neil and all,
>
> first off thanks for your interest.
>
> On Sat, 16 Aug 2008, Neil Brown wrote:
> > You should only need about 12Meg of dirty memory to keep your RAID0
> > busy, and I doubt that is even 1% of your memory.
>
> Strange. With the 12disk raid we get as much as 200MB dirty and throughput
> is slow. With the 4*3disk raid's we see just around 12MB and throughput
> is good. We have 4GB RAM.
The differences in Dirty and Writeback numbers are interesting. I'm
not sure what they signify yet.
in:
>
> http://www.metsahovi.fi/~jwagner/10g-tests/pmp/linux-raid-1.txt
>
you appear to be doing the write tests to the three small arrays in
series, one after the other.
You need to do them in parallel to find the aggregate bandwidth of the
system.
i.e. run
sudo /usr/bin/time dd if=/dev/zero of=/dev/md0 bs=1024k count=48000 &
sudo /usr/bin/time dd if=/dev/zero of=/dev/md1 bs=1024k count=48000 &
sudo /usr/bin/time dd if=/dev/zero of=/dev/md2 bs=1024k count=48000 &
all at once in the background. What does that produce?
NeilBrown
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: single RAID slower than aggregate multi-RAID?
2008-08-18 11:54 ` Neil Brown
@ 2008-08-18 12:27 ` Jan Wagner
2008-08-20 12:24 ` Jan Wagner
0 siblings, 1 reply; 10+ messages in thread
From: Jan Wagner @ 2008-08-18 12:27 UTC (permalink / raw)
To: Neil Brown; +Cc: linux-raid
Hi Neil,
On Mon, 18 Aug 2008, Neil Brown wrote:
>> http://www.metsahovi.fi/~jwagner/10g-tests/pmp/linux-raid-1.txt
>
> you appear to be doing the write tests to the three small arrays in
> series, one after the other.
> You need to do them in parallel to find the aggregate bandwidth of the
> system.
> i.e. run
>
> sudo /usr/bin/time dd if=/dev/zero of=/dev/md0 bs=1024k count=48000 &
> sudo /usr/bin/time dd if=/dev/zero of=/dev/md1 bs=1024k count=48000 &
> sudo /usr/bin/time dd if=/dev/zero of=/dev/md2 bs=1024k count=48000 &
>
> all at once in the background. What does that produce?
Sorry, I forgot to mention this in the .txt file: all the three tests
were in fact running in parallel, just on different xterm terminals. The
.txt is a copypaste from the three terminals.
When I used '&' backgrounding the results were the same.
But let's redo it, for completeness:
oper@abidal:~$ sudo /usr/bin/time dd if=/dev/zero of=/dev/md0 bs=1024k count=48000 &
[1] 13283
oper@abidal:~$ sudo /usr/bin/time dd if=/dev/zero of=/dev/md1 bs=1024k count=48000 &
[2] 13286
oper@abidal:~$ sudo /usr/bin/time dd if=/dev/zero of=/dev/md2 bs=1024k count=48000 &
[3] 13288
48000+0 records in
48000+0 records out
50331648000 bytes (50 GB) copied, 218.485 s, 230 MB/s
0.07user 147.77system 3:38.50elapsed 67%CPU (0avgtext+0avgdata 0maxresident)k
48inputs+98304008outputs (1major+520minor)pagefaults 0swaps
48000+0 records in
48000+0 records out
50331648000 bytes (50 GB) copied, 218.206 s, 231 MB/s
0.05user 145.51system 3:38.20elapsed 66%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+98304008outputs (0major+521minor)pagefaults 0swaps
48000+0 records in
48000+0 records out
50331648000 bytes (50 GB) copied, 218.895 s, 230 MB/s
0.06user 137.92system 3:38.89elapsed 63%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+98304008outputs (0major+520minor)pagefaults 0swaps
Kind regards,
- Jan
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: single RAID slower than aggregate multi-RAID?
2008-08-18 12:27 ` Jan Wagner
@ 2008-08-20 12:24 ` Jan Wagner
2008-08-21 7:34 ` Peter Grandi
0 siblings, 1 reply; 10+ messages in thread
From: Jan Wagner @ 2008-08-20 12:24 UTC (permalink / raw)
To: Neil Brown; +Cc: linux-raid
Hi again,
just thought I should re-post and add the 12disk raid0 stats here as well:
On Mon, 18 Aug 2008, Jan Wagner wrote:
> When I used '&' backgrounding the results were the same.
>
> But let's redo it, for completeness, three 4disk raid0 in parallel:
>
> oper@abidal:~$ sudo /usr/bin/time dd if=/dev/zero of=/dev/md0 bs=1024k
> count=48000 &
> [1] 13283
> oper@abidal:~$ sudo /usr/bin/time dd if=/dev/zero of=/dev/md1 bs=1024k
> count=48000 &
> [2] 13286
> oper@abidal:~$ sudo /usr/bin/time dd if=/dev/zero of=/dev/md2 bs=1024k
> count=48000 &
> [3] 13288
>
> 48000+0 records in
> 48000+0 records out
> 50331648000 bytes (50 GB) copied, 218.485 s, 230 MB/s
> 0.07user 147.77system 3:38.50elapsed 67%CPU (0avgtext+0avgdata 0maxresident)k
> 48inputs+98304008outputs (1major+520minor)pagefaults 0swaps
> 48000+0 records in
> 48000+0 records out
> 50331648000 bytes (50 GB) copied, 218.206 s, 231 MB/s
> 0.05user 145.51system 3:38.20elapsed 66%CPU (0avgtext+0avgdata 0maxresident)k
> 0inputs+98304008outputs (0major+521minor)pagefaults 0swaps
> 48000+0 records in
> 48000+0 records out
> 50331648000 bytes (50 GB) copied, 218.895 s, 230 MB/s
> 0.06user 137.92system 3:38.89elapsed 63%CPU (0avgtext+0avgdata 0maxresident)k
> 0inputs+98304008outputs (0major+520minor)pagefaults 0swaps
And a single 12disk raid0:
oper@abidal:~$ sudo /usr/bin/time dd if=/dev/zero of=/dev/md0 bs=1024k
48000+0 records in
48000+0 records out
50331648000 bytes (50 GB) copied, 106.334 s, 473 MB/s
0.10user 90.41system 1:46.38elapsed 85%CPU (0avgtext+0avgdata
0maxresident)k
88inputs+98304000outputs (2major+526minor)pagefaults 0swaps
The Dirty/Writeback values were at
http://www.metsahovi.fi/~jwagner/10g-tests/pmp/linux-raid-1.txt
For the 3 x 4disk raid0s the values were ~390MB Writeback and ~14MB
Dirty. Aggretate write rate 690MB/sec.
For the 1 x 12disk raid0 just ~14MB Writeback and ~190MB Dirty. Write
rate 473 MB/sec.
Didn't mention before but the kernel is the Ubuntu 2.6.24-19-generic SMP,
and it is on a 4 core system. I also booted with 'nosmp', but all the
write rates were approximately the same.
Do you have any more thoughts or possible tests?
Perhaps I should read up on a kernel profiling howto...?
Kind regards,
- Jan
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: single RAID slower than aggregate multi-RAID?
2008-08-20 12:24 ` Jan Wagner
@ 2008-08-21 7:34 ` Peter Grandi
2008-08-26 8:27 ` Jan Wagner
0 siblings, 1 reply; 10+ messages in thread
From: Peter Grandi @ 2008-08-21 7:34 UTC (permalink / raw)
To: Linux RAID
> [ ... ] For the 3 x 4disk raid0s the values were ~390MB
> Writeback and ~14MB Dirty. Aggretate write rate 690MB/sec.
> For the 1 x 12disk raid0 just ~14MB Writeback and ~190MB
> Dirty. Write rate 473 MB/sec. [ ... ]
That coincides with my own experience. But note that it is a very
special case, where optimum speed is reached only if requests hit
the member block devices in exactly the "right" way. In most
workloads latency also is a big part, so the ability to issue long
sequential streams of back-to-back requests is less important.
Anyhow my impression is that the recent 2.6 linux block IO
subsystem has some rather misguided behaviours (but then so much
of the Linux kernel contains several "brilliant" ideas of random
busybodies, most notably in paging, swapping and memory mangement),
and MD does not interact well with them.
As previously mentioned, on the read side the amount of data read
in a single transaction seems to be exactly the same as the block
device read ahead, which means that very large read-ahead values
give (much) better performance than smaller values, which would
not be the case if the block layer was streaming.
As to the write side, my current guess is that the effect above is
due to similar non streaming behaviour, probably dependent on the
unwise "plugging" idea that one of the usual busybodies has added
to the block layer. I have tried to disable that but there seem to
be deeper problems. I suspect also request reordering in the
elevator algorithms; using 'noop' as the elevator sometimes helps.
Overall the current Linux block layer instead of being based on
streaming seems to be based on batching of requests, and this
interacts pooly with MD, as the batch sizes and their passing on
timing might not be those that can best keep several MD member
block devices busy.
To see the detailed effects of all these layers of "brilliant"
policies, IO rates on individual devices should be looked at, and
I use something like this command line (with a very tall terminal
window).
watch -n2 iostat -d -m /dev/mdXXX /dev/sdNNN /dev/sdMMM ... 1 2
My experience is that often the traffic is not quite evenly
balanced across the member drives, and when it is, the rate
sometimes is well below the one at which the member device
can operate.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: single RAID slower than aggregate multi-RAID?
2008-08-21 7:34 ` Peter Grandi
@ 2008-08-26 8:27 ` Jan Wagner
0 siblings, 0 replies; 10+ messages in thread
From: Jan Wagner @ 2008-08-26 8:27 UTC (permalink / raw)
To: Peter Grandi; +Cc: Linux RAID
On Thu, 21 Aug 2008, Peter Grandi wrote:
>> [ ... ] For the 3 x 4disk raid0s the values were ~390MB
>> Writeback and ~14MB Dirty. Aggretate write rate 690MB/sec.
>
>> For the 1 x 12disk raid0 just ~14MB Writeback and ~190MB
>> Dirty. Write rate 473 MB/sec. [ ... ]
There was an old Microsoft research article from 2004/2005:
http://research.microsoft.com/barc/Sequential_IO/
Linear scaling with more drives. Versus kernel 2.6.22/.24 on our test
machine where the rate saturates already with ~5 drives in raid0. I'm not
familiar enough with md source to know what to fix.
> That coincides with my own experience. But note that it is a very
> special case, where optimum speed is reached only if requests hit
> the member block devices in exactly the "right" way. In most
> workloads latency also is a big part, so the ability to issue long
> sequential streams of back-to-back requests is less important.
Data streaming is quite commonplace. The kernel should be able to handle
it efficiently.
Getting requests hit the "right" way should be deterministic. Achievable
with correct settings. I have already tried a few tunings, to no avail.
Specifically:
QT=64
NCQD=8
for drv in $@; do
echo $QT > /sys/block/${drv}/queue/iosched/quantum
cat /sys/block/${drv}/queue/max_hw_sectors_kb > /sys/block/${drv}/queue/max_sectors_kb
blockdev --setra 16384 /dev/${drv}
echo "noop" > /sys/block/${drv}/queue/scheduler
echo "${NCQD}" > /sys/block/${drv}/device/queue_depth
echo "$(($NCQD * 2))" > /sys/block/${drv}/device/nr_requests
done
On average this doubles the sequential read performance. But not with
sequential write.
Set "nop" elevator on the md slaves. No difference.
Additionally tried telling the kernel that sequential I/O is expected
(madvise, posix_fadvise) and trying to bypass some caching w O_DIRECT,
even tested mmap() and msync(), and tuned /proc/sys/vm/* settings. Always
doing raw I/O to /dev/mdX.
But the kernel or md or pdflush do not "get" any of the obvious hints that
they could dispatch requests optimally.
About elevators, MD itself has no "input side" elevator setting AFAICT.
Perhaps this is the problem (?)
> Overall the current Linux block layer instead of being based on
> streaming seems to be based on batching of requests, and this
> interacts pooly with MD, as the batch sizes and their passing on
> timing might not be those that can best keep several MD member
> block devices busy.
Sounds plausible.
> To see the detailed effects of all these layers of "brilliant"
> policies, IO rates on individual devices should be looked at, and
> I use something like this command line (with a very tall terminal
> window).
>
> watch -n2 iostat -d -m /dev/mdXXX /dev/sdNNN /dev/sdMMM ... 1 2
Hmm, I tried it. With a single 4-disk raid0 2048k chunk, all disks behind
same PMP:
$ grep MemTotal /proc/meminfo
MemTotal: 4044384 kB
screen1$ dd if=/dev/zero of=/dev/md0 bs=2048k
screen2$ watch -n2 iostat -d -m /dev/sd{b,c,d,e} 1 2
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sdb 865.00 0.00 45.80 0 45
sdc 881.00 0.00 45.95 0 45
sdd 881.00 0.00 45.87 0 45
sde 898.00 0.00 45.55 0 45
With the 12-disk raid0, with linear scaling expected to do 12*46MB/s
or 552MB/s:
screen1$ dd if=/dev/zero of=/dev/md0 bs=2048k
screen2$ watch -n2 iostat -d -m /dev/sd{b,c,d,e,f,g,h,i,j,k,l,m} 1 2
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sdb 473.00 0.00 17.71 0 17
sdc 513.00 0.00 18.88 0 18
sdd 495.00 0.00 19.41 0 19
sde 520.00 0.00 19.12 0 19
sdf 536.00 0.00 19.37 0 19
sdg 479.00 0.00 19.39 0 19
sdh 489.00 0.00 19.10 0 19
sdi 440.00 0.00 19.43 0 19
sdj 468.00 0.00 18.68 0 18
sdk 428.00 0.00 17.40 0 17
sdl 404.00 0.00 17.58 0 17
sdm 464.00 0.00 17.14 0 17
screen1$ # 42960158720 bytes (43 GB) copied, 210.794 s, 204 MB/s
So the rates are quite evenly distributed within maybe +-1.5 MB/s from the
mean.
> My experience is that often the traffic is not quite evenly
> balanced across the member drives, and when it is, the rate
> sometimes is well below the one at which the member device
> can operate.
Perhaps what I see here is yet another problem.
Now, things get curious and fast with O_DIRECT and a block size up to half
of the available 4GB memory. Without O_DIRECT the rates are always quite
low.
screen1$ dd if=/dev/zero of=/dev/md0 bs=16M oflag=direct
screen2$ watch -n2 iostat -d -m /dev/sd{b,c,d,e,f,g,h,i,j,k,l,m} 1 2
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sdb 56.00 0.00 28.00 0 28
...
sdm 52.00 0.00 26.00 0 26
screen1$ # 14612955136 bytes (15 GB) copied, 44.7189 s, 327 MB/s
screen1$ sudo dd if=/dev/md0 of=/dev/null bs=16M iflag=direct
screen1$ # 14092861440 bytes (14 GB) copied, 31.681 s, 445 MB/s
------------------------------------------------------------------
screen1$ dd if=/dev/zero of=/dev/md0 bs=64M oflag=direct
screen2$ watch -n2 iostat -d -m /dev/sd{b,c,d,e,f,g,h,i,j,k,l,m} 1 2
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sdb 70.00 0.00 35.00 0 35
...
sdm 65.00 0.00 34.50 0 34
screen1$ # 12683575296 bytes (13 GB) copied, 28.4022 s, 447 MB/s
screen1$ sudo dd if=/dev/md0 of=/dev/null bs=64M iflag=direct
screen1$ # 79054241792 bytes (79 GB) copied, 128.795 s, 614 MB/s
------------------------------------------------------------------
screen1$ dd if=/dev/zero of=/dev/md0 bs=512M oflag=direct
screen2$ watch -n2 iostat -d -m /dev/sd{b,c,d,e,f,g,h,i,j,k,l,m} 1 2
sdb 25.00 0.00 42.00 0 42
...
sdm 23.00 0.00 42.00 0 42
screen1$ # 25769803776 bytes (26 GB) copied, 53.3836 s, 483 MB/
screen1$ sudo dd if=/dev/md0 of=/dev/null bs=512M iflag=direct
screen1$ # 23622320128 bytes (24 GB) copied, 36.4848 s, 647 MB/s
------------------------------------------------------------------
screen1$ sudo dd if=/dev/zero of=/dev/md0 bs=1024M oflag=direct
screen2$ watch -n2 iostat -d -m /dev/sd{b,c,d,e,f,g,h,i,j,k,l,m} 1 2
sdb 17.00 0.00 24.50 0 24
...
sde 14.00 0.00 19.50 0 19
sdf 17.00 0.00 24.50 0 24
...
sdk 14.00 0.00 11.50 0 11
sdl 15.00 0.00 26.50 0 26
sdm 15.00 0.00 26.50 0 26
screen1$ # the rates jump around between 20MB/s, 60MB/s
screen1$ # 20401094656 bytes (20 GB) copied, 41.9148 s, 487 MB/s
screen1$ sudo dd if=/dev/md0 of=/dev/null bs=1024M iflag=direct
screen1$ # 23622320128 bytes (24 GB) copied, 36.4848 s, 647 MB/s
------------------------------------------------------------------
screen1$ dd if=/dev/zero of=/dev/md0 bs=2048M oflag=direct
screen2$ watch -n2 iostat -d -m /dev/sd{b,c,d,e,f,g,h,i,j,k,l,m} 1 2
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sdb 72.00 0.00 53.00 0 53
...
sdm 74.00 0.00 52.00 0 52
screen1$ # the above happens in bursts and there are pauses with 0MB/s
screen1$ # 19327315968 bytes (19 GB) copied, 41.8337 s, 462 MB/s
screen1$ sudo dd if=/dev/md0 of=/dev/null bs=2048M iflag=direct
screen1$ # 27917234176 bytes (28 GB) copied, 46.4147 s, 601 MB/s
I guess large writes "force" something in the vm...
- Jan
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2008-08-26 8:27 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2008-08-15 13:55 single RAID slower than aggregate multi-RAID? Jan Wagner
2008-08-15 19:35 ` Justin Piszcz
2008-08-15 21:53 ` Richard Scobie
2008-08-15 22:02 ` Neil Brown
2008-08-18 10:48 ` Jan Wagner
2008-08-18 11:54 ` Neil Brown
2008-08-18 12:27 ` Jan Wagner
2008-08-20 12:24 ` Jan Wagner
2008-08-21 7:34 ` Peter Grandi
2008-08-26 8:27 ` Jan Wagner
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox