Re: O_DIRECT to md raid 6 is slow

[Miquel van Smoorenburg <mikevs@xs4all.net>]

On 16-08-12 1:05 PM, Stan Hoeppner wrote:
> On 8/15/2012 6:07 PM, Miquel van Smoorenburg wrote:
>> Ehrm no. If you modify, say, a 4K block on a RAID5 array, you just have
>> to read that 4K block, and the corresponding 4K block on the
>> parity drive, recalculate parity, and write back 4K of data and 4K
>> of parity. (read|read) modify (write|write). You do not have to
>> do I/O in chunksize, ehm, chunks, and you do not have to rmw all disks.
>
> See:  http://www.spinics.net/lists/xfs/msg12627.html
>
> Dave usually knows what he's talking about, and I didn't see Neil nor
> anyone else correcting him on his description of md RMW behavior.

Well he's wrong, or you're interpreting it incorrectly.

I did a simple test:

* created a 1G partition on 3 seperate disks
* created a md raid5 array with 512K chunksize:
   mdadm -C /dev/md0 -l 5 -c $((1024*512)) -n 3 /dev/sdb1 /dev/sdc1 
/dev/sdd1
* ran disk monitoring using 'iostat -k 5 /dev/sdb1 /dev/sdc1 /dev/sdd1'
* wrote a single 4K block:
   dd if=/dev/zero bs=4K count=1 oflag=direct seek=30 of=/dev/md0

Output from iostat over the period in which the 4K write was done. Look 
at kB read and kB written:

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sdb1              0.60         0.00         1.60          0          8
sdc1              0.60         0.80         0.80          4          4
sdd1              0.60         0.00         1.60          0          8

As you can see, a single 4K read, and a few writes. You see a few blocks 
more written that you'd expect because the superblock is updated too.

Mike.