Expanding array by changing disks (one by one)

Expanding array by changing disks (one by one)

[Laurent CARON]

Hello,

We are in the process of increasing the size our RAID Arrays as our 
storage needs increase.

I've got 2 solutions for this:

- Copy the data over a new array and replace the disks

- Replace each disk (one after the other(after resync)) of the existing 
array with a bigger one.

Start:
    - Array is ok
- Remove 1 disk
    - Array is degraded
- Add a bigger disk
    - Resync
- Remove another disk
    - Array is degraded
- Add a bigger disk
    - Resync
.....

Would this be the 'state of the art' way ?

Will the filesystem cope with it?

Is my mind completely broken?

Thanks for your help.

Laurent

Re: Expanding array by changing disks (one by one)

[Andy Smith]

[-- Attachment #1: Type: text/plain, Size: 390 bytes --]

On Thu, Apr 14, 2005 at 12:11:20PM +0200, Laurent CARON wrote:
> Hello,
> 
> We are in the process of increasing the size our RAID Arrays as our 
> storage needs increase.

[...]

> - Replace each disk (one after the other(after resync)) of the existing 
> array with a bigger one.

I'd also like to know (but only out of interest) if this will work
specifically with a RAID5.

[-- Attachment #2: Type: application/pgp-signature, Size: 187 bytes --]

Re: Expanding array by changing disks (one by one)

[Gordon Henderson]

On Thu, 14 Apr 2005, Laurent CARON wrote:

> Hello,
>
> We are in the process of increasing the size our RAID Arrays as our
> storage needs increase.
>
> I've got 2 solutions for this:
>
> - Copy the data over a new array and replace the disks

Do this! You know it makes sense. If nothing else, it'll make sure you
have an archive of the data. Do verify the copy before you replace the
disks too.

> - Replace each disk (one after the other(after resync)) of the existing
> array with a bigger one.
>
> Start:
>     - Array is ok
> - Remove 1 disk
>     - Array is degraded
> - Add a bigger disk
>     - Resync
> - Remove another disk
>     - Array is degraded
> - Add a bigger disk
>     - Resync
> .....
>
> Would this be the 'state of the art' way ?

You would have to guarantee that each disk had zero surface defects before
you started this. Use badblocks (or dd) to do a surface scan of each disk.
One bad block and you'll be in the 2-disk failure mode which will likely
cause loss of all data.

> Will the filesystem cope with it?

Firstly, will the RAID cope with it, and my understanding is that it might
not - initially. When you put in a new, bigger disk, it will only use the
same number of blocks as the disk you are replacing it had.

So after doing all the disks, the array size won't have changed!

So next you'd need to look at the grow the array mechanism, and as I
understand it, this is supported in the 2.6 kernel and mdadm, but the man
page is short on examples, but I'm sure someone here might fill in the
details.

Then you'd have to grow the filesystem under it.

> Is my mind completely broken?

Well - not really, but IMO, doing it the individual disk swapping way is
just too prone to errors and failure. I'd backup the data, replace the
drives and restore it.

And even if you do go down this route, I'd backup my data first anyway!
(And then, since you have a backup, you have nothing to lose doing it the
multiple disk swapping way, so write back and let us know if it worked :)

Good luck!

Gordon

Re: Expanding array by changing disks (one by one)

[David Greaves]

Laurent CARON wrote:

> Hello,
>
> We are in the process of increasing the size our RAID Arrays as our 
> storage needs increase.

Hi,
I'm no expert but I'll bounce some thoughts back at you...

>
> I've got 2 solutions for this:
>
> - Copy the data over a new array and replace the disks

pros:
sane and minimal risk of data loss since you don't delete data until the 
new one is up.
quickest solution (in manpower, elapsed time and system outages)

cons:
doesn't test Neils grow mode in mdadm (without which none of this would 
be possible)

> - Replace each disk (one after the other(after resync)) of the 
> existing array with a bigger one.

and the corollorary of sane would be... <joke>

>
> Start:
>    - Array is ok
> - Remove 1 disk
>    - Array is degraded
> - Add a bigger disk
>    - Resync
> - Remove another disk
>    - Array is degraded
> - Add a bigger disk
>    - Resync
> .....
>
This seems like it would work with mdadm's shiny new(?) grow command.
It looks from the man page like you use the grow command *after* 
changing out all the disks; yes?
Shiny and new means you need a reliable backup.
And this is likely to be slow - 1 resync per disk - and needs a system 
shutdown/outage for each disk swapped (assuming not hot-swap-able)
If you do have hot-swap then this seems like quite a nice option (no 
outages)

Mad speculative question ...
Would doing this work:
- add new big disk
- mdadm --remove a disk
- dd if=<removed disk> of=<new disk>
- mdadm -add new disk?
- remove small disk

if so it'd be a lot quicker than a resync each time...

> Would this be the 'state of the art' way ?

Have you seen EVMS? It has the ability to add devices to a raid5 array.
I'd class that as state of the art since you aren't 'throwing away' the 
smaller disks.

> Will the filesystem cope with it?

yes assuming it has a growfs.

>
> Is my mind completely broken?

not yet - let us know after you try it...

David

Re: Expanding array by changing disks (one by one)

[Mike Hardy]

Gordon Henderson wrote:
> On Thu, 14 Apr 2005, Laurent CARON wrote:
>>
>>- Copy the data over a new array and replace the disks
> 
> 
> Do this! You know it makes sense. If nothing else, it'll make sure you
> have an archive of the data. Do verify the copy before you replace the
> disks too.

Definitely make a backup, and once you do that the rest is just testing
anyway, isn't it?

>>- Replace each disk (one after the other(after resync)) of the existing
>>array with a bigger one.
> 
> You would have to guarantee that each disk had zero surface defects before
> you started this. Use badblocks (or dd) to do a surface scan of each disk.
> One bad block and you'll be in the 2-disk failure mode which will likely
> cause loss of all data.

Alternatively, you could do a round of SMART long tests via smartctl.

If you have one bad block you will be in 2-disk failure mode, but this
will not result in loss of all data. It will result in loss of whichever
file had the bad block in it. Even then, there are now tools (namely a
script I've posted a couple times) that can calculate the contents of a
given parity block or disk block just from the disk contents (while the
array is offline) so its possible to manually correct single block
failures as long as the rest of the blocks in the stripe are okay.

That solves the most usual scenario, albeit via a process that is pretty
ugly. Should make you feel safer if you feel like experimenting though.

The real takeaway is that making a backup, verifying it, and then doing
your maintenance is going to be the fastest and safest way to do this,
even if the disk-by-disk swap / mdadm grow way works

-Mike