migration of raid 5 to raid 6 and disk of 2TB to 4TB

migration of raid 5 to raid 6 and disk of 2TB to 4TB

[bobzer]

Hi,

I'm looking for advice to not mess up my migration. because
unfortunately i miss drive
I wanna migrate all my data from a raid 5 of 4x2TB (actually 3 because
i'm degraded right now) to a raid 6 of 4x4TB
so exactly i got :
- raid 5, should be 4 disk of 2T but got problem and so right now it's
just 2x2T and a 2Tb disk image in a 4T disk (the raid crashed and is
not start right now but is clean)
- 2 disk of 4TB
- the raid 5 use lvm2 on top of mdadm

I want a raid 6 so i thought i could create a raid 6 degraded with the
2x4TB and after copy the data to this new raid and after add the third
4TB and finally add the 2  2TB as a raid 0 to get a 4TB
I know it's an ideal solution to get 3x4TB + 2x2TB to do my raid but
it give me time to invest.

So my question is :
- is it a really bad idea ?
- do you have a better idea? solution ?
- maybe i can migrate instead of copy a raid to another ?

thank you very much
Math

Re: migration of raid 5 to raid 6 and disk of 2TB to 4TB

[Wols Lists]

On 25/07/16 22:14, bobzer wrote:
> Hi,
> 
> I'm looking for advice to not mess up my migration. because
> unfortunately i miss drive
> I wanna migrate all my data from a raid 5 of 4x2TB (actually 3 because
> i'm degraded right now) to a raid 6 of 4x4TB
> so exactly i got :
> - raid 5, should be 4 disk of 2T but got problem and so right now it's
> just 2x2T and a 2Tb disk image in a 4T disk (the raid crashed and is
> not start right now but is clean)
> - 2 disk of 4TB
> - the raid 5 use lvm2 on top of mdadm
> 
> I want a raid 6 so i thought i could create a raid 6 degraded with the
> 2x4TB and after copy the data to this new raid and after add the third
> 4TB and finally add the 2  2TB as a raid 0 to get a 4TB
> I know it's an ideal solution to get 3x4TB + 2x2TB to do my raid but
> it give me time to invest.
> 
> So my question is :
> - is it a really bad idea ?
> - do you have a better idea? solution ?
> - maybe i can migrate instead of copy a raid to another ?

Yup. You should be able to migrate.

Add in one of the 4TB drives as 4TB. It'll only use 2TB because it
matches the other drives, but it'll get your raid 5 back fully redundant.

Now replace the two 2TB drives with the 4TB drives. Use the --replace
option to mdadm - it's fairly new.

Then I think you'll have to fail the 2TB image in order to add the last
4TB drive in - maybe you won't - others might be able to help here ...
or you might be able to just remove the 2TB image and convert it to a
full raid 5 on 3 x 4TB disks - that's the best option if you can because
at no point are you surrendering a working raid and risking a failure.

(The thing to remember, is that raid will take the smallest drive, and
use that amount of space on all the drives. If you replace all the 2TBs
with 4TBs, suddenly the raid can use the full 4TB.)

Now you're using half your raid-space for your array, you should be able
to convert to raid 6, then expand the array to use the new space. I'm
not sure exactly how all this works, but having replaced all 2TB drives
with 4TB drives, I'm sure mdadm can claim the expanded space.

JUST MAKE SURE YOUR NEW DRIVES ARE ENTERPRISE RAID DRIVES! What are
they? WD Reds? Seagate NAS? If they don't support SCT/ERC or whatever
it's called, then your array will be a time bomb!

> thank you very much
> Math

Cheers,
Wol

Re: migration of raid 5 to raid 6 and disk of 2TB to 4TB

[Adam Goryachev]

On 26/07/16 07:14, bobzer wrote:
> Hi,
>
> I'm looking for advice to not mess up my migration. because
> unfortunately i miss drive
> I wanna migrate all my data from a raid 5 of 4x2TB (actually 3 because
> i'm degraded right now) to a raid 6 of 4x4TB
> so exactly i got :
> - raid 5, should be 4 disk of 2T but got problem and so right now it's
> just 2x2T and a 2Tb disk image in a 4T disk (the raid crashed and is
> not start right now but is clean)
> - 2 disk of 4TB
> - the raid 5 use lvm2 on top of mdadm
>
> I want a raid 6 so i thought i could create a raid 6 degraded with the
> 2x4TB and after copy the data to this new raid and after add the third
> 4TB and finally add the 2  2TB as a raid 0 to get a 4TB
> I know it's an ideal solution to get 3x4TB + 2x2TB to do my raid but
> it give me time to invest.
>
> So my question is :
> - is it a really bad idea ?
> - do you have a better idea? solution ?
> - maybe i can migrate instead of copy a raid to another ?
>
> thank you very much
> Math
My head hurts, I don't understand what you currently have, but I think 
you have:
RAID5 with three members, sda1(2TB), sdb1(2TB) and an 2TB image stored 
on sdc1(4TB)
Two spare 4TB drives sdd1 and sde1.

I would do a replace from the image on sdc1 to sde1
Do a replace from sda1 to sdc1
Do a replace from sdb1 to sdd1
Create a RAID0 array on sda1 + sdb1 (linear or stripe, guess it doesn't 
matter much either way, though if performance matters, then think about 
this as it will throw out the expected performance of your array).
Do a add of the RAID0 to the RAID5 (it will add as a spare)
Do a grow of the RAID5 to RAID6
Do a grow --size=max to increase capacity to the smallest drive (don't 
do the grow earlier, the RAID0 might be slightly smaller).

The above means at no point is your RAID5 degraded, and you end up with 
RAID6.
It will probably take some time to do all that, I'm not sure if the 2nd 
and 3rd steps can be done in parallel or not...

PS, you should take a copy of all your RAID information and drive member 
details (lsdrv script) before commencing so that if something goes 
wrong, you have the best chance of recovery.
!!!Also, definitely ensure you have SCT/ERC configured properly on both 
the old and new drives!!!

Regards,
Adam
-- 
Adam Goryachev Website Managers www.websitemanagers.com.au

Re: migration of raid 5 to raid 6 and disk of 2TB to 4TB

[Peter Grandi]

[ ... ]
> migrate all my data from a raid 5 of 4x2TB (actually 3 because
> i'm degraded right now) to a raid 6 of 4x4TB so exactly i got :
> - raid 5, should be 4 disk of 2T but got problem and so right
>   now it's just 2x2T and a 2Tb disk image in a 4T disk (the
>   raid crashed and is not start right now but is clean)
> - 2 disk of 4TB
> - the raid 5 use lvm2 on top of mdadm
> I want a raid 6 [ ... ]

In general an in-place migration is a very dangerous operation
because it stresses existing hardware a lot plus it uses code
that is rarely used and is quite complex. Given that your
situation is already compromised.

Plus your goal does not make a lot of sense: from a RAID5 of 4
drives to a RAID6 of 4 drives. Very strange.

Also I note that you currently don't have 4 drives of 4TB, but
3, so there you get your wish 2 drives of 2TB as the fourth
member as in «2 2TB as a raid 0»

The good thing about your plan is to use the larger drives to
make a *copy* of your data, so you don't quite do an in-place
migration.

So, the questions really are how much data you have on your
existing 4x (degraded) 2TB RAID5, which will be at most 6TB, and
how many drives you can connect *at the same time*. You seem
sure to be able to connect all 5 drives: the 3x 4TB and the 2x
2TB. I really hope none of them is USB.

So in total you have 2x 2TB drives and 3x 4TB drives, of which
currently the 2TB drive are full and 1x 4TB drive is half full,
thus you have 6TB of data without redundancy and 10TB of free
space. In a very ideal world you would get an extra 4TB disk,
but you seem unable to do so...

Also, given that you are are seriously considering all of this,
I must assume that you are a very experienced RAID and storage
guru knowing all the little details that matter to success, and
to recovery in case of problems arising.

Let's call A and B the 2x 2TB disks, C the 4TB disk with the 2TB
disk image, and D and E the empty 4TB disks.

The least scary option might be:

* Split into two partitions D and E.
* Block copy the image on C to D1 and E2.
* Re-partition C also in two.
* Block copy B to C1.
* Block copy A to E2.
* Now we have greatly increased redundancy, as we have the 3x
  2TB data slices on two separate sets, A, C1, D1 and D2, B,
  E2.
* Add E1 as a spare to the A, C1, D1 RAID5 set, and start it so
  you end up with a full RAID5 set on A, C1, D1, E1 after
  resync end.
* Now that the first RAID5 set is not degraded, you can erase
  the copies on D2, B, E2, and create a second RAID5 set on
  B, C2, D2, E2, which will be empty.

A slight improvement to your scheme:

* Add a second 2TB disk image to C.
* Add the second 2TB disk image to the existing RAID5 and wait
  until resync to end, creating a full RAID5 set; having two
  members on the same disk is far from ideal, but better than
  nothing, and temporary.
* Make D and E as a degraded RAID6 or RAID10.
* Copy the data to the newly created RAID6 or RAID10.
* Reset A, B, C.
* Make A and B into a RAID0.
* Add A+B and C to the RAID6/RAID10 as spares and wait for the
  resync to end.

A scheme that relies on in-place conversion from RAID5 to RAID6:

* Copy the image on C to E.
* Copy B to D.
* Copy A to C.
* Create an empty RAID0 of A+B.
* Add A+B as spare to to C, D, E, and wait for the resync to
  end, creating a full RAID5.
* In-place expand the RAID5 to 4TB members, that should take
  no time.
* In-place convert the RAID5 to RAID6.

But all these seem to me at the limit of plausibility.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: migration of raid 5 to raid 6 and disk of 2TB to 4TB

[Wols Lists]

On 26/07/16 19:33, Peter Grandi wrote:
> [ ... ]
>> migrate all my data from a raid 5 of 4x2TB (actually 3 because
>> i'm degraded right now) to a raid 6 of 4x4TB so exactly i got :
>> - raid 5, should be 4 disk of 2T but got problem and so right
>>   now it's just 2x2T and a 2Tb disk image in a 4T disk (the
>>   raid crashed and is not start right now but is clean)
>> - 2 disk of 4TB
>> - the raid 5 use lvm2 on top of mdadm
>> I want a raid 6 [ ... ]
> 
> In general an in-place migration is a very dangerous operation
> because it stresses existing hardware a lot plus it uses code
> that is rarely used and is quite complex. Given that your
> situation is already compromised.

But copying everything off will stress it just as much, surely? The
alternatives imho are worse ...
> 
> Plus your goal does not make a lot of sense: from a RAID5 of 4
> drives to a RAID6 of 4 drives. Very strange.
> 
From a 6TB raid5 to an 8TB raid6? What's strange about that?

> Also I note that you currently don't have 4 drives of 4TB, but
> 3, so there you get your wish 2 drives of 2TB as the fourth
> member as in «2 2TB as a raid 0»
> 
> The good thing about your plan is to use the larger drives to
> make a *copy* of your data, so you don't quite do an in-place
> migration.
> 
So beg or borrow a 2TB (or larger) drive, provided you have enough slots
for 6 drives. Borrow or buy (they're cheap enough) an add-in SATA card
if you need to.

Now he can add the second 4TB drive to complete his raid5.

Add in the third 4TB to replace the 2TB partition and free up the partly
used 4TB drive.

Use the now-free first 4TB drive to replace one of the 2TB drives. Shame
he can't reshape the resulting array into a 3 x 4TB raid5 (at least, I
don't think he can).

Use the borrowed drive to replace the last 2TB drive.

Combine the 2 2TB drives into a 4TB raid0 array, and use that to replace
the borrowed drive.

We now have a 4 x 4TB raid5 array which he can convert to raid6.

If the worst comes to the worst and he can't get that spare drive, he
could always fail the last 2TB drive (taking it back to degraded raid5
:-( then combine the 2 x 2TB and add them back to get a fully working
array again.

Unfortunately, I can't think of any way to get to the planned final
config without either borrowing an extra drive, or having (at some
point) a degraded array.

Cheers,
Wol

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: migration of raid 5 to raid 6 and disk of 2TB to 4TB

[bobzer]

Thank you very much for your help

I realized that there's much better idea than my first idea....

I can plug all my drives the sata plug are not a problem :-)
I use around 4.5TB on my current raid 5
I will continue to think on this problem and in the meanwhile try to
find another disk

i didn't think about :
> > In general an in-place migration is a very dangerous operation
> > because it stresses existing hardware a lot plus it uses code
> > that is rarely used and is quite complex. Given that your
> > situation is already compromised.
>
> But copying everything off will stress it just as much, surely? The
> alternatives imho are worse ...

at first i thought about using rsync to do my copy (anyway i don't have a GUI)
at second i thought that an in-place migration would be a nice and
safe operation
now i don't what to think ?
the 3  4TB disk are new but the 2  2TB are 4 year old and consumer
grade (the 4TB are seagate NAS HDD)
so i'm asking myself should I do a copy (with rsync or another tool)
or an in-place migration ? which one is the safest ?

thanks again
Cheers,
Math

Re: migration of raid 5 to raid 6 and disk of 2TB to 4TB

[Peter Grandi]

>> In general an in-place migration is a very dangerous operation
>> because it stresses existing hardware a lot plus it uses code
>> that is rarely used and is quite complex.

> But copying everything off will stress it just as much,
> surely? The alternatives imho are worse ...

* The very different access patterns of a block-by-block copy to
  new fresh disks and the migration-in-place having to rewrite by
  moving around the contents of all involved disks.

* That in a copy there is always the original to fall back on, a
  migration-in-place that fails can be fatal.

I am giving for granted here that one way or another at least one
copy of all the data has to be done; and block-by-block device
sequential copy is probably the best, followed by file-by-file
sequential copy, followed by in-place migration.

Note: RAID5 in-place migration does not mean that existing blocks
stay where they are, it means that (nearly) all blocks get moved,
but inside the existing device set.

So probably the best combination here, because the O.P. does not
currently have any redundancy, is to first ensure there is some
kind of copy of the data and/or redundancy, on the way to getting
a new storage layout. Because *any* whole-content data operation
stresses the storage system.

Re: migration of raid 5 to raid 6 and disk of 2TB to 4TB

[Peter Grandi]

[ ... ]

> I can plug all my drives the sata plug are not a problem :-) I
> use around 4.5TB on my current raid 5

That opens up some more options, if it can be cut down to less
than 4TB, which may be possible e.g. by compression (unless most
files are already compressed), for example by doing a full
backup of your existing content as a '.tar.lzo' to one of the
new 4TB drives, which would give you backup, and one from which
after you create a new RAID set, a source to repopulate your
data.

Also, if the data is just 4.5TB, why not go for just a 3x RAID5
with the 3x 4TB drives, which a usable capacity of 8TB? A degree
of redundancy of 1-in-3 is not bad. It would have less IOPS
though.

Another option I did not mention would be to split the 3x 4TB
drives in two, and then create a 3x RAID5 set on one half with a
4TB usable capacity, and a 5x RAID6 set on the other half plus
the 2x 2TB drives, with a 6TB usable capacity.

> at first i thought about using rsync to do my copy

The 'rsync' is good, I use that especially for updates, but
something like this is often better for an initial full copy
because it parallelizes a bit reading and writing:

  tar -C $FROM -c --one -b 512 -f - . \
    | tar -C $TO -x --preserve-p -b 512 -f -

The alternative of a 'dd' block-by-block physical copy of the
existing members onto suitably sized partitions on the new
drives, which would be faster and safer; but a 'rsync'/'tar'
file-by-file logical copy has the advantage of defragmenting the
destination, at the price of much slower and riskier seek-based
reading of the source.

> in-place migration would be a nice and safe operation now i
> don't what to think ? [ ... ]

Most in-place migrations succeed, even if very very slowly.

The problem happens in the rare but not so uncommon cases where
they fail: if you have a full backup it's not a big problem, if
you don't then it gets rather "challenging".

Re: migration of raid 5 to raid 6 and disk of 2TB to 4TB

[Anthony Youngman]

On 27/07/16 03:16, bobzer wrote:
> Thank you very much for your help
>
> I realized that there's much better idea than my first idea....
>
> I can plug all my drives the sata plug are not a problem :-)
> I use around 4.5TB on my current raid 5
> I will continue to think on this problem and in the meanwhile try to
> find another disk
>
> i didn't think about :
>>> In general an in-place migration is a very dangerous operation
>>> because it stresses existing hardware a lot plus it uses code
>>> that is rarely used and is quite complex. Given that your
>>> situation is already compromised.
>> But copying everything off will stress it just as much, surely? The
>> alternatives imho are worse ...
> at first i thought about using rsync to do my copy (anyway i don't have a GUI)
> at second i thought that an in-place migration would be a nice and
> safe operation
> now i don't what to think ?
> the 3  4TB disk are new but the 2  2TB are 4 year old and consumer
> grade (the 4TB are seagate NAS HDD)
> so i'm asking myself should I do a copy (with rsync or another tool)
> or an in-place migration ? which one is the safest ?
>
>
Get another 4TB drive?

Firstly, desktop grade is dangerous. You should never have been using 
these disks in a raid5 (is that why you lost the other two drives?). 
Have you still got those drives? If so, try wiping them with "dd 
if=/dev/random of=/dev/dud-drive" (I've said /dev/random rather than 
/dev/zero because I'm worried the drive might try and do something 
clever). This may get those drives back and usable, although I wouldn't 
trust them. IFF that works, you now have your 2 x  2TB array back. 
(It'll hopefully fix any dud sectors by overwriting them.)

You can now create a 3 x 4TB raid-5 and copy your data across. If it 
doesn't work, you can see why you need to get that next 4TB drive :-) 
And once you've got your fourth drive, you have your raid6.

The big problem you have is that your data is over 4TB, and won't fit on 
one drive. So there is no way, whether you do an in-place migration, or 
a copy-to-new-raid, that you can transfer the data without losing raid 
protection somewhere along the line unless you get more disk space.

With that extra 4TB drive, personally I'd migrate in place - as I said 
add in the three new 4TB drives, convert the two 2TB drives to a 4TB 
raid0 so I can free up the 4TB drive with the 2TB image, then use that 
freed drive to replace the raid0.

Cheers,
Wol

Re: migration of raid 5 to raid 6 and disk of 2TB to 4TB

[Wols Lists]

On 27/07/16 14:22, Peter Grandi wrote:
> Also, if the data is just 4.5TB, why not go for just a 3x RAID5
> with the 3x 4TB drives, which a usable capacity of 8TB? A degree
> of redundancy of 1-in-3 is not bad. It would have less IOPS
> though.

The problem is that one of the three new 4TB drives is currently part of
the old - partly failed - array. So without buying a fourth 4TB drive,
that option isn't on the table.

Cheers,
Wol

Re: migration of raid 5 to raid 6 and disk of 2TB to 4TB

[Peter Grandi]

>>> On Wed, 27 Jul 2016 14:44:03 +0100, Wols Lists <antlists@youngman.org.uk> said:

> On 27/07/16 14:22, Peter Grandi wrote:
>> Also, if the data is just 4.5TB, why not go for just a 3x
>> RAID5 with the 3x 4TB drives, which a usable capacity of 8TB?
>> A degree of redundancy of 1-in-3 is not bad. It would have
>> less IOPS though.

> The problem is that one of the three new 4TB drives is
> currently part of the old - partly failed - array. So without
> buying a fourth 4TB drive, that option isn't on the table.

A and B are the 2TB drives, C, D and E the 4TB drives.

* Copy the 4.5TB if they can be compressed to drive E (it does
  not neet to be partitioned).
* Setup drives C and D as a degraded (3-1)x RAID5.
* Create a RAID0 of the 2x 2TB drives.
* Add the RAID0 as the 3rd member to the new RAID5.
* Copy the 4.5TB from drive E to the new RAID5.
* Wait for sync to finish.
* Stop the RAID5 set.
* Copy by 'dd' the RAID0 set's content to drive E.
* Stop the RAID0 set and '--zero' A and B.
* Start the RAID5.
* Reuse A and B for something else.

Re: migration of raid 5 to raid 6 and disk of 2TB to 4TB

[Peter Grandi]

BTW as to in general this question:

> which one is the safest ?

That would be the first option I presented, where the 3x 4TB
disks get split in two, and the existing 3 2TB slices get copied
onto them, as I wrote:

> The least scary option might be:
> 
> * Split into two partitions D and E.
> * Block copy the image on C to D1 and E2.
> * Re-partition C also in two.
> * Block copy B to C1.
> * Block copy A to E2.
> * Now we have greatly increased redundancy, as we have the 3x
>   2TB data slices on two separate sets, A, C1, D1 and D2, B,
>   E2.

That is about as good as it gets as to safety with the limited
number of drives in play.  BTW note that it is really important
not to start the second copy, because it has the same set and
member UUIDs as the other one. But the O.P. must be a very
experienced RAID specialist, or he would not be trying to do
subtle device restructuring like this.

> * Add E1 as a spare to the A, C1, D1 RAID5 set, and start it so
>   you end up with a full RAID5 set on A, C1, D1, E1 after
>   resync end.

This resync is less dangerous because we have a full (if
degraded) copy of the data on D2, B, E2.

With an extra 4TB drive, even a temporary one, to hold an
offline backup (if the 4.5TB can be compressed to fit), more
dangerous options become less scary.

> * Now that the first RAID5 set is not degraded, you can erase
>   the copies on D2, B, E2, and create a second RAID5 set on
>   B, C2, D2, E2, which will be empty.

Re: migration of raid 5 to raid 6 and disk of 2TB to 4TB

[bobzer]

thank you very much for all your help

I manage to borrow 2  3TB drive
So i was thinking of dd copie my image to one, use the other to
reconstruct my raid
so at this point i got my first raid5 clean
then i prepare my raid6 degraded with the the 3  4TB and do a rsync

another solution could be to move the image disk to one 3TB
create the raid6 degraded with the 3 4TB
move the image in the raid 6
replace the 2 2TB with the 2 3TB (with a DD copy ?)
reuse the 2 2TB to have a clean raid 6
rsync from the degraded raid5 to the clean raid6

I prefere the first solution because the raid 5 is clean and so it
should be safer but i worrying about the stress of rebuild the raid

i like the second solution because i don't need to rebuild the raid 5
and so i don't take the risk that my 2T disk died
I mean, they should be ok but i prefere to just have to read on them
than to write for the rebuild (last time it's the stress of the
rebuild which destroy my raid...)
the interest is that there if one 2TB died it more likely to happend
when is in the raid 6 not when i read from them to copy in the 3T, and
when my data are in 3tb they should be safe

there is plenty more solution available to me now that i got these 2
drives except that i have to give then back as soon as possible

anyway i still don't know which scenario is the best

but thanks for your help

Re: migration of raid 5 to raid 6 and disk of 2TB to 4TB

[Wols Lists]

On 06/08/16 00:32, bobzer wrote:
> thank you very much for all your help
> 
> I manage to borrow 2  3TB drive
> So i was thinking of dd copie my image to one, use the other to
> reconstruct my raid
> so at this point i got my first raid5 clean
> then i prepare my raid6 degraded with the the 3  4TB and do a rsync
> 
> another solution could be to move the image disk to one 3TB
> create the raid6 degraded with the 3 4TB
> move the image in the raid 6
> replace the 2 2TB with the 2 3TB (with a DD copy ?)
> reuse the 2 2TB to have a clean raid 6
> rsync from the degraded raid5 to the clean raid6
> 
> I prefere the first solution because the raid 5 is clean and so it
> should be safer but i worrying about the stress of rebuild the raid

Okay. So rebuilding a raid could tip it over the edge.

But you now have four spare drives. So the obvious solution is

create a new raid6 with 2 x 4TB and 2 x 3TB drives.
Copy from the old raid to the new one - the reads won't stress the old
raid anywhere near as much as a rebuild.
Replace one of the 3TB disks with the 4TB that's currently in the old raid.
Raid0 the 2 x 2TB disks from the old raid, and replace the second 3TB disk.

At no point are you rebuilding a degraded raid, and when your data is at
its greatest risk (no backup, raid reshaping) you have a raid6 so you
could survive a disk failure. Oh - and always use mdadm --replace so the
raid always has spare disks ...

If you want to do it in place, ddrescue the two 2TB drives on to the two
3TB drives, so again you can raid0 the 2 x 2TB together, but I think
that's still taking a risk.

Cheers,
Wol