Re: Reshape speed

[Roger Heflin <rogerheflin@gmail.com>]

Iain Rauch wrote:
> Hi,
> 
> I've recently assembled a new RAID 6.
> 
> When I assembled it, I had 3 disks and one missing.
> 
> I then added two disks.
> 
> The first disk was used to recover the originally missing disk.
> 
> The second disk is being used to grow the array.
> 
> The recovery went at about 45MB/s but the reshape is only around 11MB/s.
> 
> CPU usage is as follows:
>  12.8   1398    root     [md1_raid5]
>  2.0    7996    root     [md1_reshape]
> 
> Memory usage is as follows:
> Type                    Percent Capacity    Free        Used        Size
> Physical Memory         18%                 2.97 GB     661.03 MB   3.62 GB
> - Kernel + applications 6%                              218.21 MB
> - Buffers               0%                              7.67 MB
> - Cached                12%                             435.14 MB
> Disk Swap               0%                  0.00 KB     0.00 KB     0.00 KB
> Disk Swap               0%                  0.00 KB     0.00 KB     0.00 KB
> 
> #uname -a
> Linux edna 2.6.24-etchnhalf.1-amd64 #1 SMP Tue Dec 2 17:21:26 UTC 2008
> x86_64 GNU/Linux
> 
> Why is the reshape so much slower? Where's the bottleneck?

Reshapes are expensive.

Rebuilds only have to read the other n disks and calculate parity and 
put it on the replaced disk.

Reshapes have to move data around to make a n+1 disk array from a n 
disk array, which means *lots* of reading and writing on all of the 
other n disks to make it look like it was originally built as a n+1 
disk array.     It is very likely that it will have to read every bit 
of the data, calculate parity and then rewrite it back into the 
correct new location for the entire array, so that adds an entire new 
write step for all of the disks, and on top of that adds a seek from 
the read location to the new write location onto the disk too.