Expanding array with multiple devices

Expanding array with multiple devices

[Oliver Schinagl]

Hello list,

I've looked online a bit, but didn't really find the answer to the 
following question.

While expanding an existing raid is as simple as mdadm --add, followed 
by an mdadm --grow, How would you go about this when expanding it with 
multiple devices.

A little bit of background. I have 8 2TB disks, 4 in a current raid5 
array, which will be the donor disks en 4 'new' disk going to form the 
basis of a new array. I want to change chunk size and re-format to ext4 
to a) optimize the new array according to the new parameters 
(stride,stripe-width) and (possibly) a new ondisk ext4 disk format (new 
features added since last time). The old array was 4 disk raid5, the new 
array will be 8 disk raid6.

So far, I have created a new raid6 array with the 4 'new' disks + 2 
'missing' for the parity. I have done it this way, so that I would have 
enough space to copy (and verify) files. having done a 4 disk raid6 
without any missing devices, there would simply have not been enough 
space. True I could have used 3 data disks, 1 parity and 1 missing.

Anyway, my plan next, is to expand the array. I can think of the 
following options.

a) Remove 1 disk at a time from the old array, and add it to the new 
array, first 2 disks would be parity, after that --growing the last two. 
Big disadvantage is of course that there will be almost a week or two of 
re-syncing going on.

b) Would be to somehow add all 'extra' disks at once, and have it rsync 
it all in one go.

Any idea's how best to expand all this?

Oliver

Re: Expanding array with multiple devices

[Mikael Abrahamsson]

On Thu, 28 Feb 2013, Oliver Schinagl wrote:

> A little bit of background. I have 8 2TB disks, 4 in a current raid5 array, 
> which will be the donor disks en 4 'new' disk going to form the basis of a 
> new array. I want to change chunk size and re-format to ext4 to a) optimize 
> the new array according to the new parameters (stride,stripe-width) and 
> (possibly) a new ondisk ext4 disk format (new features added since last 
> time). The old array was 4 disk raid5, the new array will be 8 disk raid6.
>
> Any idea's how best to expand all this?

You didn't mention what kernel and mdadm version you're running.

Anyhow, what I would do is the following:

Create RAID6 with 3 data drives, 1 parity and 1 missing. Do whatever you 
need to create the fs etc, and copy the files. When you're satisfied with 
the result (checksumming the files etc), destroy the old raid5, add all 
the drives to the new raid6, and tell it to --grow to a total of 8 drives. 
What will probably happen is that as soon as you add the 4 drives it'll 
start to resync onto one of them to restore state to a non-degraded raid6.

I would personally let this complete so you actually have a fully 
functional non-degraded raid6 with 5 drives and 3 spares that you then 
grow in one go to 8 drives and 0 spares. After this is done you can grow 
the fs and hopefully everything should be well.

Personally, I use LVM on top of md, but in your case this wouldn't help 
because you want to mkfs the filesystem.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: Expanding array with multiple devices

[Oliver Schinagl]

On 03/01/13 07:52, Mikael Abrahamsson wrote:
> On Thu, 28 Feb 2013, Oliver Schinagl wrote:
>
>> A little bit of background. I have 8 2TB disks, 4 in a current raid5
>> array, which will be the donor disks en 4 'new' disk going to form the
>> basis of a new array. I want to change chunk size and re-format to
>> ext4 to a) optimize the new array according to the new parameters
>> (stride,stripe-width) and (possibly) a new ondisk ext4 disk format
>> (new features added since last time). The old array was 4 disk raid5,
>> the new array will be 8 disk raid6.
>>
>> Any idea's how best to expand all this?
>
> You didn't mention what kernel and mdadm version you're running.
Would it mattered? It's 3.7 kernel with a reasonable late mdadm.
>
> Anyhow, what I would do is the following:
>
> Create RAID6 with 3 data drives, 1 parity and 1 missing. Do whatever you
> need to create the fs etc, and copy the files. When you're satisfied
> with the result (checksumming the files etc), destroy the old raid5, add
> all the drives to the new raid6, and tell it to --grow to a total of 8
> drives. What will probably happen is that as soon as you add the 4
> drives it'll start to resync onto one of them to restore state to a
> non-degraded raid6.
>
> I would personally let this complete so you actually have a fully
> functional non-degraded raid6 with 5 drives and 3 spares that you then
> grow in one go to 8 drives and 0 spares. After this is done you can grow
> the fs and hopefully everything should be well.
Since I had a 4 disk, 2 missing array already with the data on it, I 
failed 1 disk on the raid5, and used that as first parity disk on the 
raid6. Then destroyed the raid5 and added the rest. 2x 10 hours later 
the parity was fully restored and now i'm gonna wait 24-36 hrs for the 
drive to be expanded. I guess there's no easy and to do it all in one 
go. I guess I could have done as you said, 3 disks 1 parity (same size) 
then fail 1 disk in raid 5 and add it to the raid6 and grow it with 3. 
This worked though, so thanks!

oliver
>
> Personally, I use LVM on top of md, but in your case this wouldn't help
> because you want to mkfs the filesystem.
>

Re: Expanding array with multiple devices

[Roy Sigurd Karlsbakk]

> While expanding an existing raid is as simple as mdadm --add, followed
> by an mdadm --grow, How would you go about this when expanding it with
> multiple devices.

Given a RAID-5 on /dev/sd[bcde], I'd do

# mdadm --add /dev/md0 /dev/sd[fghi]
# mdadm --grow --level=6 --raid-devices=8

No need to create a new RAID and then move the data.

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
roy@karlsbakk.net
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Expanding array with multiple devices

[Phil Turmel]

On 03/02/2013 09:07 AM, Roy Sigurd Karlsbakk wrote:
>> While expanding an existing raid is as simple as mdadm --add, followed
>> by an mdadm --grow, How would you go about this when expanding it with
>> multiple devices.
> 
> Given a RAID-5 on /dev/sd[bcde], I'd do
> 
> # mdadm --add /dev/md0 /dev/sd[fghi]
> # mdadm --grow --level=6 --raid-devices=8
> 
> No need to create a new RAID and then move the data.

You missed the bit where he states he needs to use mkfs to switch
filesystem features.

Phil

Re: Expanding array with multiple devices

[Roy Sigurd Karlsbakk]

> On 03/02/2013 09:07 AM, Roy Sigurd Karlsbakk wrote:
> >> While expanding an existing raid is as simple as mdadm --add,
> >> followed
> >> by an mdadm --grow, How would you go about this when expanding it
> >> with
> >> multiple devices.
> >
> > Given a RAID-5 on /dev/sd[bcde], I'd do
> >
> > # mdadm --add /dev/md0 /dev/sd[fghi]
> > # mdadm --grow --level=6 --raid-devices=8
> >
> > No need to create a new RAID and then move the data.
> 
> You missed the bit where he states he needs to use mkfs to switch
> filesystem features.

Oh, sorry. Then why not start off with an identical RAID-5, move the data, and then expand as above? No dangling "missing" drives, just plain and simple.

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
roy@karlsbakk.net
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Expanding array with multiple devices

[Phil Turmel]

On 03/02/2013 12:15 PM, Roy Sigurd Karlsbakk wrote:
>> On 03/02/2013 09:07 AM, Roy Sigurd Karlsbakk wrote:
>>>> While expanding an existing raid is as simple as mdadm --add, 
>>>> followed by an mdadm --grow, How would you go about this when
>>>> expanding it with multiple devices.
>>>
>>> Given a RAID-5 on /dev/sd[bcde], I'd do
>>>
>>> # mdadm --add /dev/md0 /dev/sd[fghi]
>>> # mdadm --grow --level=6 --raid-devices=8
>>> 
>>> No need to create a new RAID and then move the data.
>> 
>> You missed the bit where he states he needs to use mkfs to switch 
>> filesystem features.
> 
> Oh, sorry. Then why not start off with an identical RAID-5, move the
> data, and then expand as above? No dangling "missing" drives, just
> plain and simple.

I agree.  And the file copy to the raid5 would almost certainly run
faster than any raid6 with "missing".  Then the reshape is one operation
to both expand the drive count and switch to raid6.

Phil

Re: Expanding array with multiple devices

[Mikael Abrahamsson]

On Sat, 2 Mar 2013, Phil Turmel wrote:

> I agree.  And the file copy to the raid5 would almost certainly run 
> faster than any raid6 with "missing".  Then the reshape is one operation 
> to both expand the drive count and switch to raid6.

If he's running 2.6.32 that option isn't available (if I remember 
correctly). Since he didn't disclose what mdadm and kernel version he was 
running, I suggested the safest path that works on most kernels and mdadm 
versions.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: Expanding array with multiple devices

[Roy Sigurd Karlsbakk]

> > I agree. And the file copy to the raid5 would almost certainly run
> > faster than any raid6 with "missing". Then the reshape is one
> > operation
> > to both expand the drive count and switch to raid6.
> 
> If he's running 2.6.32 that option isn't available (if I remember
> correctly). Since he didn't disclose what mdadm and kernel version he
> was running, I suggested the safest path that works on most kernels and
> mdadm versions.

You're right. If he's on Ubuntu Lucid, that should be solvable with an upgrade to Precise with "do-release-upgrade", which has been stable for some time.

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
roy@karlsbakk.net
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Expanding array with multiple devices

[Mikael Abrahamsson]

On Sat, 2 Mar 2013, Roy Sigurd Karlsbakk wrote:

> You're right. If he's on Ubuntu Lucid, that should be solvable with an 
> upgrade to Precise with "do-release-upgrade", which has been stable for 
> some time.

If he is on Debian, they're still on 2.6.32 for their stable release.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se