Re: Odd IO traffic during raid5 reshape

[NeilBrown <neilb@suse.de>]

[-- Attachment #1: Type: text/plain, Size: 2396 bytes --]

On Thu, 18 Apr 2013 20:48:22 +0200 Goswin von Brederlow <goswin-v-b@web.de>
wrote:

> Hi,
> 
> I'm currently upgrading a NAS system with new disks. Since I'm
> changing the filesystem type and due to a lack of enough SATA ports I
> have to do add one new disk at a time, copy data, shrink the old
> filesystem, remove an old disks and repeat. I've started with a 2 disk
> raid5, copied data, freed a 3rd SATA slot and added the 3rd new disk.
> 
> Now I'm reshaping the new raid5 from 2 disks to 3 disks:
> 
> md0 : active raid5 sdd1[3] sdc1[2] sda1[0]
>       3907015168 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
>       [==>..................]  reshape = 14.0% (547848840/3907015168) finish=1355.4min speed=41302K/sec
> 
> so far everything works fine. But the speed is rather low and the IO
> traffic is higher than I think it should be:
> 
> Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
> sda             604.33     81706.00     40904.73    4902360    2454284
> md0               0.00         0.00         0.00          0          0
> sdc             440.78     81839.40     40542.07    4910364    2432524
> sdd             509.72         0.00     40817.67          0    2449060
> 
> To reshape the kernel needs to read 1 data block from sda, 1 data
> block from sdc, compute the XOR of both blocks and write 2 data blocks
> + parity block back to the 3 disks. The kernel read 160MB/s, add
> 80MB/s parity and it should write 240MB/s (or 80MB/s per disk).
> Instead it only writes 120MB/s (40MB/s per disk), only half of what I
> expect.
> 
> So what is going on there? Is the kernel reading both data and parity
> blocks and verifying them?

The kernel is reading data and parity.  Maybe it doesn't need to, but unless
your chunks are very big (10s of megatabyes?) reading takes about as long as
seeking over, so it is unlikely to affect total time.

Reshape simple is not a fast operation, nowhere near as fast as resync.
It needs to
  - read a few stripes
  - seek backward to where that data now belong
  - write the data as slightly fewer stripes
  - update the metadata to record where the data now is.
  - repeat

So there is lots of seeking.  md/raid5 tries to avoid unnecessary seeking,
but quite a bit of it is necessary.

It looks to me like it is performing quite well.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]