Re: RAID 5 write performance advice

[Ming Zhang <mingz@ele.uri.edu>]

On Thu, 2005-08-25 at 18:38 +0200, Mirko Benz wrote:
> Hello,
> 
> We intend to export a lvm/md volume via iSCSI or SRP using InfiniBand to 
> remote clients. There is no local file system processing on the storage 
> platform. The clients may have a variety of file systems including ext3, 
> GFS.
> 
> Single disk write performance is: 58,5 MB/s. With large sequential write 
> operations I would expect something like 90% of n-1 * 
> single_disk_performance if stripe write can be utilized. So roughly 400 
> MB/s – which the HW RAID devices achieve.

change to RAID0 and test to see if u controller will be a bottleneck.



> 
> RAID setup:
> Personalities : [raid0] [raid5]
> md0 : active raid5 sdi[7] sdh[6] sdg[5] sdf[4] sde[3] sdd[2] sdc[1] sdb[0]
> 1094035712 blocks level 5, 64k chunk, algorithm 2 [8/8] [UUUUUUUU]
> 
> We have assigned the deadline scheduler to every disk in the RAID. The 
> default scheduler gives much lower results.
> 
> *** dd TEST ***
> 
> time dd if=/dev/zero of=/dev/md0 bs=1M
> 5329911808 bytes transferred in 28,086199 seconds (189769779 bytes/sec)
> 
> iostat 5 output:
> avg-cpu: %user %nice %sys %iowait %idle
> 0,10 0,00 87,80 7,30 4,80
> 
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> hda 0,00 0,00 0,00 0 0
> sda 0,00 0,00 0,00 0 0
> sdb 1976,10 1576,10 53150,60 7912 266816
> sdc 2072,31 1478,88 53150,60 7424 266816
> sdd 2034,06 1525,10 53150,60 7656 266816
> sde 1988,05 1439,04 53147,41 7224 266800
> sdf 1975,10 1499,60 53147,41 7528 266800
> sdg 1383,07 1485,26 53145,82 7456 266792
> sdh 1562,55 1311,55 53145,82 6584 266792
> sdi 1586,85 1295,62 53145,82 6504 266792
> sdj 0,00 0,00 0,00 0 0
> sdk 0,00 0,00 0,00 0 0
> sdl 0,00 0,00 0,00 0 0
> sdm 0,00 0,00 0,00 0 0
> sdn 0,00 0,00 0,00 0 0
> md0 46515,54 0,00 372124,30 0 1868064
> 
> Comments: Large write should not see any read operations. But there are 
> some???

i always saw those small number reads and i feel it is reasonable since
u stripe is 7 * 64KB


> 
> 
> *** disktest ***
> 
> disktest -w -PT -T30 -h1 -K8 -B512k -ID /dev/md0
> 
> | 2005/08/25-17:27:04 | STAT | 4072 | v1.1.12 | /dev/md0 | Write 
> throughput: 160152507.7B/s (152.73MB/s), IOPS 305.7/s.
> | 2005/08/25-17:27:05 | STAT | 4072 | v1.1.12 | /dev/md0 | Write 
> throughput: 160694272.0B/s (153.25MB/s), IOPS 306.6/s.
> | 2005/08/25-17:27:06 | STAT | 4072 | v1.1.12 | /dev/md0 | Write 
> throughput: 160339606.6B/s (152.91MB/s), IOPS 305.8/s.

so here 152/7 = 21, large than what u sdc sdd got.


> 
> iostat 5 output:
> avg-cpu: %user %nice %sys %iowait %idle
> 38,96 0,00 50,25 5,29 5,49
> 
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> hda 0,00 0,00 0,00 0 0
> sda 1,20 0,00 11,18 0 56
> sdb 986,43 0,00 39702,99 0 198912
> sdc 922,75 0,00 39728,54 0 199040
> sdd 895,81 0,00 39728,54 0 199040
> sde 880,84 0,00 39728,54 0 199040
> sdf 839,92 0,00 39728,54 0 199040
> sdg 842,91 0,00 39728,54 0 199040
> sdh 1557,49 0,00 79431,54 0 397952
> sdi 2246,71 0,00 104411,98 0 523104
> sdj 0,00 0,00 0,00 0 0
> sdk 0,00 0,00 0,00 0 0
> sdl 0,00 0,00 0,00 0 0
> sdm 0,00 0,00 0,00 0 0
> sdn 0,00 0,00 0,00 0 0
> md0 1550,70 0,00 317574,45 0 1591048
> 
> Comments:
> Zero read requests – as it should be. But the write requests are not 
> proportional. sdh and sdi have significantly more requests???

yes, interesting.


> The write requests to the disks of the RAID should be 1/7 higher than to 
> the md device.
> But there are significantly more write operations.
> 
> All these operations are to the raw device. Setting up a ext3 fs we get 
> around 127 MB/s with dd.
> 
> Any idea?
> 
> --Mirko
> 

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html