* single RAID slower than aggregate multi-RAID?
@ 2008-08-15 13:55 Jan Wagner
2008-08-15 19:35 ` Justin Piszcz
2008-08-15 22:02 ` Neil Brown
0 siblings, 2 replies; 10+ messages in thread
From: Jan Wagner @ 2008-08-15 13:55 UTC (permalink / raw)
To: linux-raid
Hi,
at work we have been experimenting with SATA-II port multipliers and SATA
controllers (FIS-based). For some reason the RAID-0 performance is
relatively awful.
We have one 4xeSATA controller card ADSA3GPX8-4E, twelve SpinPoint F1
disks that handle ~110MB/sec, and four port Addonics port multiplier
boxes. Ubuntu, kernel 2.6.24-19-generic x86_64, 2 x AMD2212 Opteron on an
Asus L1N64-SLI WS.
We did two sequential writing tests. One test with all 12 disks
behind the controller placed into one mdadm RAID-0, 512k..2048k
chunksize. We get ~475 MB/s.
The second test used four "mdadm --create"'s to reassign the 12 drives
into four RAID-0 with 3 disks in each. All four raids were simultaneously
written to. We get ~250 MB/s/raid and 1000 MB/s aggregate.
So the controller can handle at least 1000 MB/s.
My question is why is the single RAID setup more than half slower? Any
ideas?
Is there some kernel per-device bandwidth throttling?
I already tried: O_DIRECT, posix_fadvise(), madvise(), pdflush tuning,
different test programs, using a raw /dev/md{0,1,2,3} vs testing
the xfs filesystem. None of them get the single RAID performance over 475
MB/s. CPU load isn't a likely problem either. To check I wrote a threaded
program[1] that has threads writing in-sequence into the same output file.
All 4 cores loaded very evenly and are >=60% idle. But the rate is still
475 MB/s. Puzzling and frustrating :-)
Is there still something that a desperate person could try to tune to get
this lousy mdadm raid throughput improved?? :-)
Thanks in advance,
- Jan
[1] http://www.metsahovi.fi/~jwagner/tmp/wr-nexgen-threaded.c
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: single RAID slower than aggregate multi-RAID? 2008-08-15 13:55 single RAID slower than aggregate multi-RAID? Jan Wagner @ 2008-08-15 19:35 ` Justin Piszcz 2008-08-15 21:53 ` Richard Scobie 2008-08-15 22:02 ` Neil Brown 1 sibling, 1 reply; 10+ messages in thread From: Justin Piszcz @ 2008-08-15 19:35 UTC (permalink / raw) To: Jan Wagner; +Cc: linux-raid On Fri, 15 Aug 2008, Jan Wagner wrote: > Hi, > > at work we have been experimenting with SATA-II port multipliers and SATA > controllers (FIS-based). For some reason the RAID-0 performance is relatively > awful. > > We have one 4xeSATA controller card ADSA3GPX8-4E, twelve SpinPoint F1 disks > that handle ~110MB/sec, and four port Addonics port multiplier boxes. Ubuntu, > kernel 2.6.24-19-generic x86_64, 2 x AMD2212 Opteron on an Asus L1N64-SLI WS. > > We did two sequential writing tests. One test with all 12 disks behind the > controller placed into one mdadm RAID-0, 512k..2048k chunksize. We get ~475 > MB/s. > > The second test used four "mdadm --create"'s to reassign the 12 drives into > four RAID-0 with 3 disks in each. All four raids were simultaneously written > to. We get ~250 MB/s/raid and 1000 MB/s aggregate. > > So the controller can handle at least 1000 MB/s. > > My question is why is the single RAID setup more than half slower? Any ideas? Because you are using port-multipliers, they do not allocate full bandwidth to each HDD simultaneously. You will get a maximum of 3gbps per each port, but if you are reading from 12 ports into a single 3gbps port? 1- 2-- 3--- 4----- 5------- 6-------------> 7-------------> 8------- 9----- 10--- 11-- 12- Fewer drives means less requests to handle: 1---> 2--->--> port1 3---> With that you should be able to max out the sata link without a problem and at the same time, not bombard the link with too many requests. From: http://www.serialata.org/portmultiplier.asp Here is the answer to your question: While it is possible to connect up to 15 drives to each SATA PM port via a port multiplier, drive connectivity is practically limited to the maximum available bandwidth on the 3Gb/s link. Sustained I/O rates from the drives are kept to within the 3Gb/s host port connection limit for maximum efficiency and performance. > Is there some kernel per-device bandwidth throttling? Answer is, if you want speed, don't use port multipliers. On my old P965 board I have optimized where to put the drives and the speeds I get with Linux SW RAID5 with 10 velociraptors are as folows: About the same for write: $ dd if=/dev/zero of=bigfile.1 bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 20.4056 s, 526 MB/s $ dd if=bigfile.1 of=/dev/null bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 10.2841 s, 1.0 GB/s Justin. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: single RAID slower than aggregate multi-RAID? 2008-08-15 19:35 ` Justin Piszcz @ 2008-08-15 21:53 ` Richard Scobie 0 siblings, 0 replies; 10+ messages in thread From: Richard Scobie @ 2008-08-15 21:53 UTC (permalink / raw) To: Justin Piszcz; +Cc: Jan Wagner, linux-raid Justin Piszcz wrote: > Because you are using port-multipliers, they do not allocate full > bandwidth to each HDD simultaneously. You will get a maximum of 3gbps > per each port, but if you are reading from 12 ports into a single 3gbps > port? Jan's setup is 3 drives on each port multiplier, which are each attached to a port on the controller, so should be just on the limit of the 3Gb/s link between controller and multiplier. According to this, with the same controller and 12 portmultiplied Seagate drives on a Mac, in one single RAID 0, you should see just under 700MB/s peak, so there may be room for improvement yet. http://www.amug.org/amug-web/html/amug/reviews/articles/addonics/adsa3gpx8-4em/ Regards, Richard ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: single RAID slower than aggregate multi-RAID? 2008-08-15 13:55 single RAID slower than aggregate multi-RAID? Jan Wagner 2008-08-15 19:35 ` Justin Piszcz @ 2008-08-15 22:02 ` Neil Brown 2008-08-18 10:48 ` Jan Wagner 1 sibling, 1 reply; 10+ messages in thread From: Neil Brown @ 2008-08-15 22:02 UTC (permalink / raw) To: Jan Wagner; +Cc: linux-raid On Friday August 15, jwagner@kurp.hut.fi wrote: > Hi, > > at work we have been experimenting with SATA-II port multipliers and SATA > controllers (FIS-based). For some reason the RAID-0 performance is > relatively awful. > > We have one 4xeSATA controller card ADSA3GPX8-4E, twelve SpinPoint F1 > disks that handle ~110MB/sec, and four port Addonics port multiplier > boxes. Ubuntu, kernel 2.6.24-19-generic x86_64, 2 x AMD2212 Opteron on an > Asus L1N64-SLI WS. > > We did two sequential writing tests. One test with all 12 disks > behind the controller placed into one mdadm RAID-0, 512k..2048k > chunksize. We get ~475 MB/s. > > The second test used four "mdadm --create"'s to reassign the 12 drives > into four RAID-0 with 3 disks in each. All four raids were simultaneously > written to. We get ~250 MB/s/raid and 1000 MB/s aggregate. > > So the controller can handle at least 1000 MB/s. > > My question is why is the single RAID setup more than half slower? Any > ideas? > > Is there some kernel per-device bandwidth throttling? Yes, but it shouldn't be affecting you. The kernel only allows 40% (/proc/sys/vm/dirty_ratio) of memory to be dirty. Once you have more than that, writes will be throttled to avoid dirtying all of memory. Writes to devices which are slower will be allowed a smaller share of the 40%. You should only need about 12Meg of dirty memory to keep your RAID0 busy, and I doubt that is even 1% of your memory. md/raid0 has very little overhead. It just encourages the filesystem to send requests that are aligned with the chunks, and then sends each request on to the target drive without any further intervention. So the queue for each device should be kept full, and the individual devices should be going at full speed. So to make sure I'm not misunderstanding you description, could you run the two tests again: 12disk raid0 and 4*3disk raid0, and for each report mdadm -D of each array output of time dd if=/dev/zero of=/dev/mdXX bs=1024k on all arrays in parallel Using a chunksize of 1024. While doing this, watch the values of "Dirty" and "Writeback" in /proc/meminfo and see how high they get. And report MemTotal just to get a complete picture. Thanks, NeilBrown ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: single RAID slower than aggregate multi-RAID? 2008-08-15 22:02 ` Neil Brown @ 2008-08-18 10:48 ` Jan Wagner 2008-08-18 11:54 ` Neil Brown 0 siblings, 1 reply; 10+ messages in thread From: Jan Wagner @ 2008-08-18 10:48 UTC (permalink / raw) To: Neil Brown; +Cc: linux-raid Hello Neil and all, first off thanks for your interest. On Sat, 16 Aug 2008, Neil Brown wrote: > You should only need about 12Meg of dirty memory to keep your RAID0 > busy, and I doubt that is even 1% of your memory. Strange. With the 12disk raid we get as much as 200MB dirty and throughput is slow. With the 4*3disk raid's we see just around 12MB and throughput is good. We have 4GB RAM. > md/raid0 has very little overhead. It just encourages the filesystem > to send requests that are aligned with the chunks, and then sends each > request on to the target drive without any further intervention. So > the queue for each device should be kept full, and the individual > devices should be going at full speed. Okay. We have been using power-of-2 raw write sizes, and in the case of using a filesystem, it was XFS with settings in mkfs.xfs that tell it about the chunk size and device count. > So to make sure I'm not misunderstanding you description, could you > run the two tests again: 12disk raid0 and 4*3disk raid0, and for > each report > mdadm -D of each array > output of > time dd if=/dev/zero of=/dev/mdXX bs=1024k > on all arrays in parallel > Using a chunksize of 1024. One of our eSATA<=>SATA conversion cables got lost, so I couldn't right now do your test on the 4*3disk raid0 in my first posting. Instead I ran with 12disk and 3*4disk raid0. http://www.metsahovi.fi/~jwagner/10g-tests/pmp/linux-raid-1.txt 473 MB/s with single raid, high 'dirty', 692 MB/s with triple raid and low 'dirty. Any ideas? TIA, - Jan ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: single RAID slower than aggregate multi-RAID? 2008-08-18 10:48 ` Jan Wagner @ 2008-08-18 11:54 ` Neil Brown 2008-08-18 12:27 ` Jan Wagner 0 siblings, 1 reply; 10+ messages in thread From: Neil Brown @ 2008-08-18 11:54 UTC (permalink / raw) To: Jan Wagner; +Cc: linux-raid On Monday August 18, jwagner@kurp.hut.fi wrote: > Hello Neil and all, > > first off thanks for your interest. > > On Sat, 16 Aug 2008, Neil Brown wrote: > > You should only need about 12Meg of dirty memory to keep your RAID0 > > busy, and I doubt that is even 1% of your memory. > > Strange. With the 12disk raid we get as much as 200MB dirty and throughput > is slow. With the 4*3disk raid's we see just around 12MB and throughput > is good. We have 4GB RAM. The differences in Dirty and Writeback numbers are interesting. I'm not sure what they signify yet. in: > > http://www.metsahovi.fi/~jwagner/10g-tests/pmp/linux-raid-1.txt > you appear to be doing the write tests to the three small arrays in series, one after the other. You need to do them in parallel to find the aggregate bandwidth of the system. i.e. run sudo /usr/bin/time dd if=/dev/zero of=/dev/md0 bs=1024k count=48000 & sudo /usr/bin/time dd if=/dev/zero of=/dev/md1 bs=1024k count=48000 & sudo /usr/bin/time dd if=/dev/zero of=/dev/md2 bs=1024k count=48000 & all at once in the background. What does that produce? NeilBrown ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: single RAID slower than aggregate multi-RAID? 2008-08-18 11:54 ` Neil Brown @ 2008-08-18 12:27 ` Jan Wagner 2008-08-20 12:24 ` Jan Wagner 0 siblings, 1 reply; 10+ messages in thread From: Jan Wagner @ 2008-08-18 12:27 UTC (permalink / raw) To: Neil Brown; +Cc: linux-raid Hi Neil, On Mon, 18 Aug 2008, Neil Brown wrote: >> http://www.metsahovi.fi/~jwagner/10g-tests/pmp/linux-raid-1.txt > > you appear to be doing the write tests to the three small arrays in > series, one after the other. > You need to do them in parallel to find the aggregate bandwidth of the > system. > i.e. run > > sudo /usr/bin/time dd if=/dev/zero of=/dev/md0 bs=1024k count=48000 & > sudo /usr/bin/time dd if=/dev/zero of=/dev/md1 bs=1024k count=48000 & > sudo /usr/bin/time dd if=/dev/zero of=/dev/md2 bs=1024k count=48000 & > > all at once in the background. What does that produce? Sorry, I forgot to mention this in the .txt file: all the three tests were in fact running in parallel, just on different xterm terminals. The .txt is a copypaste from the three terminals. When I used '&' backgrounding the results were the same. But let's redo it, for completeness: oper@abidal:~$ sudo /usr/bin/time dd if=/dev/zero of=/dev/md0 bs=1024k count=48000 & [1] 13283 oper@abidal:~$ sudo /usr/bin/time dd if=/dev/zero of=/dev/md1 bs=1024k count=48000 & [2] 13286 oper@abidal:~$ sudo /usr/bin/time dd if=/dev/zero of=/dev/md2 bs=1024k count=48000 & [3] 13288 48000+0 records in 48000+0 records out 50331648000 bytes (50 GB) copied, 218.485 s, 230 MB/s 0.07user 147.77system 3:38.50elapsed 67%CPU (0avgtext+0avgdata 0maxresident)k 48inputs+98304008outputs (1major+520minor)pagefaults 0swaps 48000+0 records in 48000+0 records out 50331648000 bytes (50 GB) copied, 218.206 s, 231 MB/s 0.05user 145.51system 3:38.20elapsed 66%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+98304008outputs (0major+521minor)pagefaults 0swaps 48000+0 records in 48000+0 records out 50331648000 bytes (50 GB) copied, 218.895 s, 230 MB/s 0.06user 137.92system 3:38.89elapsed 63%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+98304008outputs (0major+520minor)pagefaults 0swaps Kind regards, - Jan ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: single RAID slower than aggregate multi-RAID? 2008-08-18 12:27 ` Jan Wagner @ 2008-08-20 12:24 ` Jan Wagner 2008-08-21 7:34 ` Peter Grandi 0 siblings, 1 reply; 10+ messages in thread From: Jan Wagner @ 2008-08-20 12:24 UTC (permalink / raw) To: Neil Brown; +Cc: linux-raid Hi again, just thought I should re-post and add the 12disk raid0 stats here as well: On Mon, 18 Aug 2008, Jan Wagner wrote: > When I used '&' backgrounding the results were the same. > > But let's redo it, for completeness, three 4disk raid0 in parallel: > > oper@abidal:~$ sudo /usr/bin/time dd if=/dev/zero of=/dev/md0 bs=1024k > count=48000 & > [1] 13283 > oper@abidal:~$ sudo /usr/bin/time dd if=/dev/zero of=/dev/md1 bs=1024k > count=48000 & > [2] 13286 > oper@abidal:~$ sudo /usr/bin/time dd if=/dev/zero of=/dev/md2 bs=1024k > count=48000 & > [3] 13288 > > 48000+0 records in > 48000+0 records out > 50331648000 bytes (50 GB) copied, 218.485 s, 230 MB/s > 0.07user 147.77system 3:38.50elapsed 67%CPU (0avgtext+0avgdata 0maxresident)k > 48inputs+98304008outputs (1major+520minor)pagefaults 0swaps > 48000+0 records in > 48000+0 records out > 50331648000 bytes (50 GB) copied, 218.206 s, 231 MB/s > 0.05user 145.51system 3:38.20elapsed 66%CPU (0avgtext+0avgdata 0maxresident)k > 0inputs+98304008outputs (0major+521minor)pagefaults 0swaps > 48000+0 records in > 48000+0 records out > 50331648000 bytes (50 GB) copied, 218.895 s, 230 MB/s > 0.06user 137.92system 3:38.89elapsed 63%CPU (0avgtext+0avgdata 0maxresident)k > 0inputs+98304008outputs (0major+520minor)pagefaults 0swaps And a single 12disk raid0: oper@abidal:~$ sudo /usr/bin/time dd if=/dev/zero of=/dev/md0 bs=1024k 48000+0 records in 48000+0 records out 50331648000 bytes (50 GB) copied, 106.334 s, 473 MB/s 0.10user 90.41system 1:46.38elapsed 85%CPU (0avgtext+0avgdata 0maxresident)k 88inputs+98304000outputs (2major+526minor)pagefaults 0swaps The Dirty/Writeback values were at http://www.metsahovi.fi/~jwagner/10g-tests/pmp/linux-raid-1.txt For the 3 x 4disk raid0s the values were ~390MB Writeback and ~14MB Dirty. Aggretate write rate 690MB/sec. For the 1 x 12disk raid0 just ~14MB Writeback and ~190MB Dirty. Write rate 473 MB/sec. Didn't mention before but the kernel is the Ubuntu 2.6.24-19-generic SMP, and it is on a 4 core system. I also booted with 'nosmp', but all the write rates were approximately the same. Do you have any more thoughts or possible tests? Perhaps I should read up on a kernel profiling howto...? Kind regards, - Jan ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: single RAID slower than aggregate multi-RAID? 2008-08-20 12:24 ` Jan Wagner @ 2008-08-21 7:34 ` Peter Grandi 2008-08-26 8:27 ` Jan Wagner 0 siblings, 1 reply; 10+ messages in thread From: Peter Grandi @ 2008-08-21 7:34 UTC (permalink / raw) To: Linux RAID > [ ... ] For the 3 x 4disk raid0s the values were ~390MB > Writeback and ~14MB Dirty. Aggretate write rate 690MB/sec. > For the 1 x 12disk raid0 just ~14MB Writeback and ~190MB > Dirty. Write rate 473 MB/sec. [ ... ] That coincides with my own experience. But note that it is a very special case, where optimum speed is reached only if requests hit the member block devices in exactly the "right" way. In most workloads latency also is a big part, so the ability to issue long sequential streams of back-to-back requests is less important. Anyhow my impression is that the recent 2.6 linux block IO subsystem has some rather misguided behaviours (but then so much of the Linux kernel contains several "brilliant" ideas of random busybodies, most notably in paging, swapping and memory mangement), and MD does not interact well with them. As previously mentioned, on the read side the amount of data read in a single transaction seems to be exactly the same as the block device read ahead, which means that very large read-ahead values give (much) better performance than smaller values, which would not be the case if the block layer was streaming. As to the write side, my current guess is that the effect above is due to similar non streaming behaviour, probably dependent on the unwise "plugging" idea that one of the usual busybodies has added to the block layer. I have tried to disable that but there seem to be deeper problems. I suspect also request reordering in the elevator algorithms; using 'noop' as the elevator sometimes helps. Overall the current Linux block layer instead of being based on streaming seems to be based on batching of requests, and this interacts pooly with MD, as the batch sizes and their passing on timing might not be those that can best keep several MD member block devices busy. To see the detailed effects of all these layers of "brilliant" policies, IO rates on individual devices should be looked at, and I use something like this command line (with a very tall terminal window). watch -n2 iostat -d -m /dev/mdXXX /dev/sdNNN /dev/sdMMM ... 1 2 My experience is that often the traffic is not quite evenly balanced across the member drives, and when it is, the rate sometimes is well below the one at which the member device can operate. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: single RAID slower than aggregate multi-RAID? 2008-08-21 7:34 ` Peter Grandi @ 2008-08-26 8:27 ` Jan Wagner 0 siblings, 0 replies; 10+ messages in thread From: Jan Wagner @ 2008-08-26 8:27 UTC (permalink / raw) To: Peter Grandi; +Cc: Linux RAID On Thu, 21 Aug 2008, Peter Grandi wrote: >> [ ... ] For the 3 x 4disk raid0s the values were ~390MB >> Writeback and ~14MB Dirty. Aggretate write rate 690MB/sec. > >> For the 1 x 12disk raid0 just ~14MB Writeback and ~190MB >> Dirty. Write rate 473 MB/sec. [ ... ] There was an old Microsoft research article from 2004/2005: http://research.microsoft.com/barc/Sequential_IO/ Linear scaling with more drives. Versus kernel 2.6.22/.24 on our test machine where the rate saturates already with ~5 drives in raid0. I'm not familiar enough with md source to know what to fix. > That coincides with my own experience. But note that it is a very > special case, where optimum speed is reached only if requests hit > the member block devices in exactly the "right" way. In most > workloads latency also is a big part, so the ability to issue long > sequential streams of back-to-back requests is less important. Data streaming is quite commonplace. The kernel should be able to handle it efficiently. Getting requests hit the "right" way should be deterministic. Achievable with correct settings. I have already tried a few tunings, to no avail. Specifically: QT=64 NCQD=8 for drv in $@; do echo $QT > /sys/block/${drv}/queue/iosched/quantum cat /sys/block/${drv}/queue/max_hw_sectors_kb > /sys/block/${drv}/queue/max_sectors_kb blockdev --setra 16384 /dev/${drv} echo "noop" > /sys/block/${drv}/queue/scheduler echo "${NCQD}" > /sys/block/${drv}/device/queue_depth echo "$(($NCQD * 2))" > /sys/block/${drv}/device/nr_requests done On average this doubles the sequential read performance. But not with sequential write. Set "nop" elevator on the md slaves. No difference. Additionally tried telling the kernel that sequential I/O is expected (madvise, posix_fadvise) and trying to bypass some caching w O_DIRECT, even tested mmap() and msync(), and tuned /proc/sys/vm/* settings. Always doing raw I/O to /dev/mdX. But the kernel or md or pdflush do not "get" any of the obvious hints that they could dispatch requests optimally. About elevators, MD itself has no "input side" elevator setting AFAICT. Perhaps this is the problem (?) > Overall the current Linux block layer instead of being based on > streaming seems to be based on batching of requests, and this > interacts pooly with MD, as the batch sizes and their passing on > timing might not be those that can best keep several MD member > block devices busy. Sounds plausible. > To see the detailed effects of all these layers of "brilliant" > policies, IO rates on individual devices should be looked at, and > I use something like this command line (with a very tall terminal > window). > > watch -n2 iostat -d -m /dev/mdXXX /dev/sdNNN /dev/sdMMM ... 1 2 Hmm, I tried it. With a single 4-disk raid0 2048k chunk, all disks behind same PMP: $ grep MemTotal /proc/meminfo MemTotal: 4044384 kB screen1$ dd if=/dev/zero of=/dev/md0 bs=2048k screen2$ watch -n2 iostat -d -m /dev/sd{b,c,d,e} 1 2 Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sdb 865.00 0.00 45.80 0 45 sdc 881.00 0.00 45.95 0 45 sdd 881.00 0.00 45.87 0 45 sde 898.00 0.00 45.55 0 45 With the 12-disk raid0, with linear scaling expected to do 12*46MB/s or 552MB/s: screen1$ dd if=/dev/zero of=/dev/md0 bs=2048k screen2$ watch -n2 iostat -d -m /dev/sd{b,c,d,e,f,g,h,i,j,k,l,m} 1 2 Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sdb 473.00 0.00 17.71 0 17 sdc 513.00 0.00 18.88 0 18 sdd 495.00 0.00 19.41 0 19 sde 520.00 0.00 19.12 0 19 sdf 536.00 0.00 19.37 0 19 sdg 479.00 0.00 19.39 0 19 sdh 489.00 0.00 19.10 0 19 sdi 440.00 0.00 19.43 0 19 sdj 468.00 0.00 18.68 0 18 sdk 428.00 0.00 17.40 0 17 sdl 404.00 0.00 17.58 0 17 sdm 464.00 0.00 17.14 0 17 screen1$ # 42960158720 bytes (43 GB) copied, 210.794 s, 204 MB/s So the rates are quite evenly distributed within maybe +-1.5 MB/s from the mean. > My experience is that often the traffic is not quite evenly > balanced across the member drives, and when it is, the rate > sometimes is well below the one at which the member device > can operate. Perhaps what I see here is yet another problem. Now, things get curious and fast with O_DIRECT and a block size up to half of the available 4GB memory. Without O_DIRECT the rates are always quite low. screen1$ dd if=/dev/zero of=/dev/md0 bs=16M oflag=direct screen2$ watch -n2 iostat -d -m /dev/sd{b,c,d,e,f,g,h,i,j,k,l,m} 1 2 Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sdb 56.00 0.00 28.00 0 28 ... sdm 52.00 0.00 26.00 0 26 screen1$ # 14612955136 bytes (15 GB) copied, 44.7189 s, 327 MB/s screen1$ sudo dd if=/dev/md0 of=/dev/null bs=16M iflag=direct screen1$ # 14092861440 bytes (14 GB) copied, 31.681 s, 445 MB/s ------------------------------------------------------------------ screen1$ dd if=/dev/zero of=/dev/md0 bs=64M oflag=direct screen2$ watch -n2 iostat -d -m /dev/sd{b,c,d,e,f,g,h,i,j,k,l,m} 1 2 Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sdb 70.00 0.00 35.00 0 35 ... sdm 65.00 0.00 34.50 0 34 screen1$ # 12683575296 bytes (13 GB) copied, 28.4022 s, 447 MB/s screen1$ sudo dd if=/dev/md0 of=/dev/null bs=64M iflag=direct screen1$ # 79054241792 bytes (79 GB) copied, 128.795 s, 614 MB/s ------------------------------------------------------------------ screen1$ dd if=/dev/zero of=/dev/md0 bs=512M oflag=direct screen2$ watch -n2 iostat -d -m /dev/sd{b,c,d,e,f,g,h,i,j,k,l,m} 1 2 sdb 25.00 0.00 42.00 0 42 ... sdm 23.00 0.00 42.00 0 42 screen1$ # 25769803776 bytes (26 GB) copied, 53.3836 s, 483 MB/ screen1$ sudo dd if=/dev/md0 of=/dev/null bs=512M iflag=direct screen1$ # 23622320128 bytes (24 GB) copied, 36.4848 s, 647 MB/s ------------------------------------------------------------------ screen1$ sudo dd if=/dev/zero of=/dev/md0 bs=1024M oflag=direct screen2$ watch -n2 iostat -d -m /dev/sd{b,c,d,e,f,g,h,i,j,k,l,m} 1 2 sdb 17.00 0.00 24.50 0 24 ... sde 14.00 0.00 19.50 0 19 sdf 17.00 0.00 24.50 0 24 ... sdk 14.00 0.00 11.50 0 11 sdl 15.00 0.00 26.50 0 26 sdm 15.00 0.00 26.50 0 26 screen1$ # the rates jump around between 20MB/s, 60MB/s screen1$ # 20401094656 bytes (20 GB) copied, 41.9148 s, 487 MB/s screen1$ sudo dd if=/dev/md0 of=/dev/null bs=1024M iflag=direct screen1$ # 23622320128 bytes (24 GB) copied, 36.4848 s, 647 MB/s ------------------------------------------------------------------ screen1$ dd if=/dev/zero of=/dev/md0 bs=2048M oflag=direct screen2$ watch -n2 iostat -d -m /dev/sd{b,c,d,e,f,g,h,i,j,k,l,m} 1 2 Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sdb 72.00 0.00 53.00 0 53 ... sdm 74.00 0.00 52.00 0 52 screen1$ # the above happens in bursts and there are pauses with 0MB/s screen1$ # 19327315968 bytes (19 GB) copied, 41.8337 s, 462 MB/s screen1$ sudo dd if=/dev/md0 of=/dev/null bs=2048M iflag=direct screen1$ # 27917234176 bytes (28 GB) copied, 46.4147 s, 601 MB/s I guess large writes "force" something in the vm... - Jan ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2008-08-26 8:27 UTC | newest] Thread overview: 10+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2008-08-15 13:55 single RAID slower than aggregate multi-RAID? Jan Wagner 2008-08-15 19:35 ` Justin Piszcz 2008-08-15 21:53 ` Richard Scobie 2008-08-15 22:02 ` Neil Brown 2008-08-18 10:48 ` Jan Wagner 2008-08-18 11:54 ` Neil Brown 2008-08-18 12:27 ` Jan Wagner 2008-08-20 12:24 ` Jan Wagner 2008-08-21 7:34 ` Peter Grandi 2008-08-26 8:27 ` Jan Wagner
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox