raid5 to raid6 - reshape very slow

raid5 to raid6 - reshape very slow

[Jonathan Molyneux]

Hi Everyone,

I'm currently reshaping my raid5 array into a raid6 array (by adding a 
spare and growing).

Unfortunately I didn't research the nuances before doing so and missed 
the opportunity to --layout=preserve.

As a result the array is rebuilding by use of a backup-file, at a rather 
slow pace.

md1 : active raid6 sde1[0] sdb1[6] sdh1[5] sdc1[4] sdg1[3] sdd1[2] sdf1[1]
       7325679680 blocks super 0.91 level 6, 64k chunk, algorithm 18 
[7/6] [UUUUUU_]
       [=========>...........]  reshape = 47.0% (689413888/1465135936) 
finish=4126.9min speed=3132K/sec

I have tried the following so far:

* read ahead tweaking
* stripe_cache_size tweaking (not a limiting factor - mdadm seems to 
tune this appropriately)
* running the backup file from dev/shm (only for testing purposes)

The reshape does not seem to be IO or CPU bound, it just seems to be 
limited by the sync's and meta data updates to each disk.

Is it possible to increase the buffer/stripes that mdadm process in a 
batch to speed up the process (~41m/2 -> 1G) ?

Any thoughts would be appreciated (even to confirm there is nothing to 
be done).

Re: raid5 to raid6 - reshape very slow

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1595 bytes --]

On Tue, 29 May 2012 10:33:52 +1000 Jonathan Molyneux
<jonathan@infinitedepth.com.au> wrote:

> Hi Everyone,
> 
> I'm currently reshaping my raid5 array into a raid6 array (by adding a 
> spare and growing).
> 
> Unfortunately I didn't research the nuances before doing so and missed 
> the opportunity to --layout=preserve.
> 
> As a result the array is rebuilding by use of a backup-file, at a rather 
> slow pace.
> 
> md1 : active raid6 sde1[0] sdb1[6] sdh1[5] sdc1[4] sdg1[3] sdd1[2] sdf1[1]
>        7325679680 blocks super 0.91 level 6, 64k chunk, algorithm 18 
> [7/6] [UUUUUU_]
>        [=========>...........]  reshape = 47.0% (689413888/1465135936) 
> finish=4126.9min speed=3132K/sec
> 
> I have tried the following so far:
> 
> * read ahead tweaking
> * stripe_cache_size tweaking (not a limiting factor - mdadm seems to 
> tune this appropriately)
> * running the backup file from dev/shm (only for testing purposes)
> 
> The reshape does not seem to be IO or CPU bound, it just seems to be 
> limited by the sync's and meta data updates to each disk.
> 
> Is it possible to increase the buffer/stripes that mdadm process in a 
> batch to speed up the process (~41m/2 -> 1G) ?
> 
> Any thoughts would be appreciated (even to confirm there is nothing to 
> be done).

The only thing to do is to wait.  This is very much a seek-bound operation
and there is little room for making it go faster.

In a few months we might see the need for the backup file disappear, which
will improve things somewhat, but it will still be slowish.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: raid5 to raid6 - reshape very slow

[Stan Hoeppner]

On 5/28/2012 7:56 PM, NeilBrown wrote:
> On Tue, 29 May 2012 10:33:52 +1000 Jonathan Molyneux
> <jonathan@infinitedepth.com.au> wrote:

>> md1 : active raid6 sde1[0] sdb1[6] sdh1[5] sdc1[4] sdg1[3] sdd1[2] sdf1[1]
>>        7325679680 blocks super 0.91 level 6, 64k chunk, algorithm 18 
>> [7/6] [UUUUUU_]
>>        [=========>...........]  reshape = 47.0% (689413888/1465135936) 
>> finish=4126.9min speed=3132K/sec

> The only thing to do is to wait.  This is very much a seek-bound operation
> and there is little room for making it go faster.

With SSDs you'd see the throughput of this operation increase by 2
orders of magnitude (100x) or more, as SSDs have application level seek
latency of ~50-80 microseconds, over 100x lower than rotational drives
which are in the 10-30 millisecond range depending on spindle speed.

Of course, cost/GB for the same total storage is 2 to 10 times higher
with SSD.  Though interestingly the price of SSDs continues to fall,
whereas the rotational drive manufacturers are currently keeping drive
prices artificially high, between 50-100% higher than last year
depending on drive model, in an effort to recoup apparent financial
losses caused by the flooding in Thailand.

-- 
Stan

Re: raid5 to raid6 - reshape very slow

[jonathan]

I suspected the only move was to wait it out.

Thanks for the feedback Stan & Neil.

On 29/05/2012 5:21 PM, Stan Hoeppner wrote:
> On 5/28/2012 7:56 PM, NeilBrown wrote:
>> On Tue, 29 May 2012 10:33:52 +1000 Jonathan Molyneux
>> <jonathan@infinitedepth.com.au>  wrote:
>>> md1 : active raid6 sde1[0] sdb1[6] sdh1[5] sdc1[4] sdg1[3] sdd1[2] sdf1[1]
>>>         7325679680 blocks super 0.91 level 6, 64k chunk, algorithm 18
>>> [7/6] [UUUUUU_]
>>>         [=========>...........]  reshape = 47.0% (689413888/1465135936)
>>> finish=4126.9min speed=3132K/sec
>> The only thing to do is to wait.  This is very much a seek-bound operation
>> and there is little room for making it go faster.
> With SSDs you'd see the throughput of this operation increase by 2
> orders of magnitude (100x) or more, as SSDs have application level seek
> latency of ~50-80 microseconds, over 100x lower than rotational drives
> which are in the 10-30 millisecond range depending on spindle speed.
>
> Of course, cost/GB for the same total storage is 2 to 10 times higher
> with SSD.  Though interestingly the price of SSDs continues to fall,
> whereas the rotational drive manufacturers are currently keeping drive
> prices artificially high, between 50-100% higher than last year
> depending on drive model, in an effort to recoup apparent financial
> losses caused by the flooding in Thailand.
>

Re: raid5 to raid6 - reshape very slow

[Alexander Kühn]

You can check your actual io utilization using "iostat -d 2 -kx".
If thats too low you may try to raise the min_sync_speed somewhere in  
/sys/block/md* but it can't do magic.
If you raise it too much all other io to your raid will suffer delays.

Zitat von jonathan@infinitedepth.com.au:

> I suspected the only move was to wait it out.
>
> Thanks for the feedback Stan & Neil.
>
> On 29/05/2012 5:21 PM, Stan Hoeppner wrote:
>> On 5/28/2012 7:56 PM, NeilBrown wrote:
>>> On Tue, 29 May 2012 10:33:52 +1000 Jonathan Molyneux
>>> <jonathan@infinitedepth.com.au>  wrote:
>>>> md1 : active raid6 sde1[0] sdb1[6] sdh1[5] sdc1[4] sdg1[3] sdd1[2] sdf1[1]
>>>>        7325679680 blocks super 0.91 level 6, 64k chunk, algorithm 18
>>>> [7/6] [UUUUUU_]
>>>>        [=========>...........]  reshape = 47.0% (689413888/1465135936)
>>>> finish=4126.9min speed=3132K/sec
>>> The only thing to do is to wait.  This is very much a seek-bound operation
>>> and there is little room for making it go faster.
>> With SSDs you'd see the throughput of this operation increase by 2
>> orders of magnitude (100x) or more, as SSDs have application level seek
>> latency of ~50-80 microseconds, over 100x lower than rotational drives
>> which are in the 10-30 millisecond range depending on spindle speed.
>>
>> Of course, cost/GB for the same total storage is 2 to 10 times higher
>> with SSD.  Though interestingly the price of SSDs continues to fall,
>> whereas the rotational drive manufacturers are currently keeping drive
>> prices artificially high, between 50-100% higher than last year
>> depending on drive model, in an effort to recoup apparent financial
>> losses caused by the flooding in Thailand.
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html