Cancel reshape?

Cancel reshape?

[John Robinson]

I've just started reshaping a RAID-5 to a RAID-6. I had meant to 
increase the number of drives too, so avoiding using the backup file all 
the way through. Unfortunately I forgot to specify that, so it's going 
rather slowly. I said:

mdadm /dev/md1 --grow --level=6 --backup-file=/some/where/else

where md1 had 3 active and 2 spare devices. It's now reshaping itself to 
RAID-6 with 4 active and 1 spare.

What I meant was

mdadm /dev/md1 --grow --level=6 --raid-disks=5 --backup-file=...

Can I cancel reshape that's currently running, and using the backup file 
that's actually on another (RAID-1) array on three of the same drives, 
and running at ~7MB/s, and try again?

Cheers,

John.

Re: Cancel reshape?

[NeilBrown]

On Wed, 12 Jan 2011 04:19:30 +0000 John Robinson
<john.robinson@anonymous.org.uk> wrote:

> I've just started reshaping a RAID-5 to a RAID-6. I had meant to 
> increase the number of drives too, so avoiding using the backup file all 
> the way through. Unfortunately I forgot to specify that, so it's going 
> rather slowly. I said:
> 
> mdadm /dev/md1 --grow --level=6 --backup-file=/some/where/else
> 
> where md1 had 3 active and 2 spare devices. It's now reshaping itself to 
> RAID-6 with 4 active and 1 spare.
> 
> What I meant was
> 
> mdadm /dev/md1 --grow --level=6 --raid-disks=5 --backup-file=...
> 
> Can I cancel reshape that's currently running, and using the backup file 
> that's actually on another (RAID-1) array on three of the same drives, 
> and running at ~7MB/s, and try again?
> 
Sorry, but no.  This isn't supported yet.
It would require modification to the kernel to be able to run a reshape in
reverse.

NeilBrown

Re: Cancel reshape?

[John Robinson]

On 12/01/2011 04:19, John Robinson wrote:
[...]
> Can I cancel reshape that's currently running, and using the backup file
> that's actually on another (RAID-1) array on three of the same drives,
> and running at ~7MB/s, and try again?

Blast. I found this patch to add a revert-reshape option from archives 
of this list - http://www.spinics.net/lists/raid/msg30961.html - but 
it's not included in mdadm 3.1.4 and I'm running from a rescue CD so I 
can't rebuild mdadm with it. I think I might just have to wait a while.

Perhaps I can stop the reshape and restart it with the backup file 
elsewhere. Hmm, I've an SSD I haven't installed yet... Maybe time to 
give it a wee workout :-)

Cheers,

John.

Re: Cancel reshape?

[NeilBrown]

On Wed, 12 Jan 2011 04:49:35 +0000 John Robinson
<john.robinson@anonymous.org.uk> wrote:

> On 12/01/2011 04:19, John Robinson wrote:
> [...]
> > Can I cancel reshape that's currently running, and using the backup file
> > that's actually on another (RAID-1) array on three of the same drives,
> > and running at ~7MB/s, and try again?
> 
> Blast. I found this patch to add a revert-reshape option from archives 
> of this list - http://www.spinics.net/lists/raid/msg30961.html - but 
> it's not included in mdadm 3.1.4 and I'm running from a rescue CD so I 
> can't rebuild mdadm with it. I think I might just have to wait a while.
> 

That patch only works to revert a reshape that is increasing the number of
data-devices in the array.

> Perhaps I can stop the reshape and restart it with the backup file 
> elsewhere. Hmm, I've an SSD I haven't installed yet... Maybe time to 
> give it a wee workout :-)

That should work.

 mdadm --stop ...
 copy file
 mdadm --assemble ...... --backup-file=/new/place

NeilBrown



> 
> Cheers,
> 
> John.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Speed up reshape? (was Re: Cancel reshape?)

[John Robinson]

On 12/01/2011 05:32, NeilBrown wrote:
> On Wed, 12 Jan 2011 04:49:35 +0000 John Robinson
> <john.robinson@anonymous.org.uk>  wrote:
[...]
>> Perhaps I can stop the reshape and restart it with the backup file
>> elsewhere. Hmm, I've an SSD I haven't installed yet... Maybe time to
>> give it a wee workout :-)
>
> That should work.
>
>   mdadm --stop ...
>   copy file
>   mdadm --assemble ...... --backup-file=/new/place

Yup. Not much of an improvement, though; the reshape is now running at 
~10MB/s instead of ~6.5MB/s, so ~27 hours to go instead of ~40 hours.

iostat tells me the original 3 discs are doing ~120 transactions per 
second, reading ~20MB/s and writing ~10MB/s, while the one new disc is 
doing ~60 tps, reading nothing (obviously) and writing ~10MB/s and the 
backup (SSD) is doing ~45 tps and writing ~20MB/s. ~120tps sounds about 
right for 120 seeks, 60 reads and 60 writes, and would explain the 
speed, but otherwise this looks odd to me - why are we reading ~60MB/s 
in total and writing ~40MB/s? I'd have expected to be reading less than 
we're writing. And why are we apparently reading and writing in blocks 
of under 512KB per device, when the backup file is 32MB? Is there 
something I can tweak to run the reshape in bigger chunks? 
sync_speed_min is already 200000...

Currently running SystemRescueCD 2.0.0, 32-bit kernel 2.6.35 and mdadm 
3.1.4. CPU usage is 2% system, 4% iowait.

Cheers,

John.

Re: Speed up reshape? (was Re: Cancel reshape?)

[NeilBrown]

On Wed, 12 Jan 2011 05:53:40 +0000 John Robinson
<john.robinson@anonymous.org.uk> wrote:

> On 12/01/2011 05:32, NeilBrown wrote:
> > On Wed, 12 Jan 2011 04:49:35 +0000 John Robinson
> > <john.robinson@anonymous.org.uk>  wrote:
> [...]
> >> Perhaps I can stop the reshape and restart it with the backup file
> >> elsewhere. Hmm, I've an SSD I haven't installed yet... Maybe time to
> >> give it a wee workout :-)
> >
> > That should work.
> >
> >   mdadm --stop ...
> >   copy file
> >   mdadm --assemble ...... --backup-file=/new/place
> 
> Yup. Not much of an improvement, though; the reshape is now running at 
> ~10MB/s instead of ~6.5MB/s, so ~27 hours to go instead of ~40 hours.
> 
> iostat tells me the original 3 discs are doing ~120 transactions per 
> second, reading ~20MB/s and writing ~10MB/s, while the one new disc is 
> doing ~60 tps, reading nothing (obviously) and writing ~10MB/s and the 
> backup (SSD) is doing ~45 tps and writing ~20MB/s. ~120tps sounds about 
> right for 120 seeks, 60 reads and 60 writes, and would explain the 
> speed, but otherwise this looks odd to me - why are we reading ~60MB/s 
> in total and writing ~40MB/s? I'd have expected to be reading less than 
> we're writing. And why are we apparently reading and writing in blocks 
> of under 512KB per device, when the backup file is 32MB? Is there 
> something I can tweak to run the reshape in bigger chunks? 
> sync_speed_min is already 200000...

1/ everything is read twice, once to back it up, once to relocate it.
  That is unfortunate, but awkward to avoid.

2/ try increasing the stripe_cache_size - it might help.

NeilBrown


> 
> Currently running SystemRescueCD 2.0.0, 32-bit kernel 2.6.35 and mdadm 
> 3.1.4. CPU usage is 2% system, 4% iowait.
> 
> Cheers,
> 
> John.

Re: Speed up reshape? (was Re: Cancel reshape?)

[John Robinson]

On 12/01/2011 06:02, NeilBrown wrote:
> On Wed, 12 Jan 2011 05:53:40 +0000 John Robinson
[...]
> 1/ everything is read twice, once to back it up, once to relocate it.
>    That is unfortunate, but awkward to avoid.

OK... but why only one stripe at a time? I eventually worked out that 
the disc transactions (shown by iostat) were of 256KB at a time which is 
the chunk size of the array, and the ~120 transactions per second would 
relate to the spin speed of the discs, 7200rpm being 120 revolutions per 
second. Even at the slow end of the discs, there's at least 512KB of 
data per revolution, so slurping up at least a couple of revolutions' 
worth before rewriting it might save waiting for quite so many disc 
revolutions to finish.

I may test this by doing another in-place reshape, changing the chunk 
size to 512KB, which I expect will run at twice the speed of the reshape 
on 256KB chunks.

Also, would there be any way to allow the second read to come from cache 
rather than from the media surface again? Is this a side-effect of using 
O_DIRECT or something? Could O_DIRECT be used only for writing?

> 2/ try increasing the stripe_cache_size - it might help.

mdadm had already set it to 1071, I upped it to 8192, then 32768, but it 
didn't make any difference at all, and stripe_cache_active showed 8 or 
even 0 when I checked. I imagine this is because the in-place reshape 
needs to wait for writes to be sync'ed all the time (or written with 
O_DIRECT or whatever) so there's little or no scope for cacheing stripes 
outside the reshape process itself.

I'm now reshaping again, from 4 to 5 discs. That's whizzing along at a 
fairly healthy 35MB/s, which according to iostat breaks down as about 
50MB/s read from 4 drives and 35MB/s written to 5 drives and is as much 
as I could reasonably expect from these discs. stripe_cache_active shows 
~25000, CPU usage is 12% system, 14% iowait. I expect the speed to slow 
down a bit towards the end of the reshape, where the slowest of the 
drives can only manage 60MB/s streaming and the seeks between reading 
and writing are longer (further across the disc surface).

Anyway, quite apart from all my whingeing about the speed of the 
in-place reshape, it's all working perfectly which is after all the most 
important thing, so many thanks for all your hard work!

Cheers,

John.

Re: Speed up reshape? (was Re: Cancel reshape?)

[John Robinson]

On 13/01/2011 13:44, John Robinson wrote:
[...]
> I may test this by doing another in-place reshape, changing the chunk
> size to 512KB, which I expect will run at twice the speed of the reshape
> on 256KB chunks.

Well, I got that wrong. Currently doing ~9MB/s, though the backup file 
is writing faster (~27MB/s). Another 30 hours reshaping. Oh well it 
should be worth it in the end.

Cheers,

John.