Adding larger disks to RAID5

Adding larger disks to RAID5

[Christian]

Hello,

I have 8x250gig hard drives in a RAID5 configuration.  What I'm trying
to do is add more storage to the raid by replacing the 250's with
larger disks one-by-one over time.  I just replaced the first disk
with a 750, and the raid resync'd, but now I'm unable to grow it to
the additional space.  The raid is just using the first 250 gigs and
nothing more on the drive.   The command I issued in attempt to grow
it was 'mdadm --grow /dev/md0 -z max', but nothing happens.  Is what
I'm trying to do possible?  I'm running kernel 2.6.21 w/ raid5 expansion
enabled.

Thanks,
Christian

SCSI device sda: 1465149168 512-byte hdwr sectors (750156 MB)
SCSI device sdb: 488397168 512-byte hdwr sectors (250059 MB)
SCSI device sdc: 488397168 512-byte hdwr sectors (250059 MB)
SCSI device sdd: 488397168 512-byte hdwr sectors (250059 MB)
SCSI device sde: 488397168 512-byte hdwr sectors (250059 MB)
SCSI device sdf: 488397168 512-byte hdwr sectors (250059 MB)
SCSI device sdg: 488397168 512-byte hdwr sectors (250059 MB)
SCSI device sdh: 488397168 512-byte hdwr sectors (250059 MB)
SCSI device sdi: 488397168 512-byte hdwr sectors (250059 MB)


Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sda[0] sdb[1] sdh[7] sdg[6] sdf[5] sde[4] sdd[3] sdc[2]
      1709389248 blocks level 5, 64k chunk, algorithm 2 [8/8] [UUUUUUUU]


/dev/md0:
        Version : 00.90.03
  Creation Time : Fri May 18 01:49:46 2007
     Raid Level : raid5
     Array Size : 1709389248 (1630.20 GiB 1750.41 GB)
  Used Dev Size : 244198464 (232.89 GiB 250.06 GB)
   Raid Devices : 8
  Total Devices : 8
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Mon May 21 13:27:34 2007
          State : clean
 Active Devices : 8
Working Devices : 8
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : 905a278f:a088d952:fab6e626:2309d439
         Events : 0.94

    Number   Major   Minor   RaidDevice State
       0       8        0        0      active sync   /dev/sda
       1       8       16        1      active sync   /dev/sdb
       2       8       32        2      active sync   /dev/sdc
       3       8       48        3      active sync   /dev/sdd
       4       8       64        4      active sync   /dev/sde
       5       8       80        5      active sync   /dev/sdf
       6       8       96        6      active sync   /dev/sdg
       7       8      112        7      active sync   /dev/sdh

Re: Adding larger disks to RAID5

[Neil Brown]

On Monday May 21, cjs@dhp.com wrote:
> Hello,
> 
> I have 8x250gig hard drives in a RAID5 configuration.  What I'm trying
> to do is add more storage to the raid by replacing the 250's with
> larger disks one-by-one over time.  I just replaced the first disk
> with a 750, and the raid resync'd, but now I'm unable to grow it to
> the additional space.  The raid is just using the first 250 gigs and
> nothing more on the drive.   The command I issued in attempt to grow
> it was 'mdadm --grow /dev/md0 -z max', but nothing happens.  Is what
> I'm trying to do possible?  I'm running kernel 2.6.21 w/ raid5 expansion
> enabled.

No, what you are trying to do is not possible.
A raid5 uses equal amounts of space on each device.  So if the devices
are of different sizes, it will use an amount equivalent to the
smallest device from every device.

If you replace all the devices with 750G drives, then you could grow
the array (-z max) to use all the space, not before.

What you could do is partition the new drives in a 250G partition and
a 500G partition.  Add the 250 to the current array.  Create a new
raid5 with the 500G.
With only one large drive, you could make a degraded 2-device raid5
which would store 500G, but would not have any data redundancy.
When you get a second 750G, add that and you will have redundancy.
Then as you replace more drives and get more 500G partitions you can
add them to the array growing it as you go until you ultimately have 2
arrays, one with 8 250G partitions, and one with 8 500G partitions.

That seems the best fit for your needs as you describe them.

If possible, it would be best to get all the drives up front and just
have one large array.

NeilBrown

Re: Adding larger disks to RAID5

[Bill Rugolsky Jr.]

Neil,

What seems desirable to me is a way to take a new (larger) spare drive and
add it to a RAID1 for a particular RAID 4/5/6 component, and then when
it's sync'd, replace the now redundant small drive with another larger
drive.  Wash, rinse, repeat.  This way the array is never degraded.
Though I imagine that this particular arrangement doesn't have the
benefit of the stripe rewrite when encountering a latent error on the
drive that is being migrated.  [Presumably the failing addresses could
be cycled through the check from userland though, by doing a read above
the stacked RAID.]

One could start a RAID 4/5/6 array over a degraded RAID1 for each
component, (i.e., a degraded RAID1).

I haven't been following the metadata changes closely.  Is it possible
to do this with external MD metadata?  It can also be done with
device-mapper, but dm-mirror is very immature compared to MD RAID1.

Comments?

Regards,

	Bill Rugolsky

Re: Adding larger disks to RAID5

[Neil Brown]

On Monday May 21, brugolsky@telemetry-investments.com wrote:
> Neil,
> 
> What seems desirable to me is a way to take a new (larger) spare drive and
> add it to a RAID1 for a particular RAID 4/5/6 component, and then when
> it's sync'd, replace the now redundant small drive with another larger
> drive.  Wash, rinse, repeat.  This way the array is never degraded.
> Though I imagine that this particular arrangement doesn't have the
> benefit of the stripe rewrite when encountering a latent error on the
> drive that is being migrated.  [Presumably the failing addresses could
> be cycled through the check from userland though, by doing a read above
> the stacked RAID.]
> 
> One could start a RAID 4/5/6 array over a degraded RAID1 for each
> component, (i.e., a degraded RAID1).
> 
> I haven't been following the metadata changes closely.  Is it possible
> to do this with external MD metadata?  It can also be done with
> device-mapper, but dm-mirror is very immature compared to MD RAID1.
> 
> Comments?

This doesn't really have anything to do with the metadata used - it is
primarily an implementation issue (though you would need to be careful
picking up the pieces after a crash).

If we could freeze an array (so that all writes block), then we could
do much of what you suggest:
  - freeze the array
  - remove the target device
  - create a raid1 of the target and the new
  - re-add the raid1
  - unfreeze the array.

The issue of dealing with read errors on the target device is much
more awkward to deal with.  The approach that seems right to me is:

  - create a raid1 variant which does a passive resync:  When the
    next-needed block is read or written, write it to the second
    device and advance the "next-needed" pointer.
  - Get this raid1 to simply return read errors (which might be OK
    already) so that a read-error won't be fatal.  But a read request
    that be behind the "next-needed" pointer gets served from the
    second device if the first does fail.
  - Implement a 'check-one-disk' operation on raid5 (and others) so
    that instead of reading all devices, it just reads all through
    one.  If this one is really a raid1-variant, doing that read will
    effect a resync on the raid1, and any read error will be handled
    correctly.

So it is all quite possible, and I agree that it could be valuable.
It just needs someone to do it, and work out all the fine details.

Anyone want to try some coding ????

NeilBrown

Re: Adding larger disks to RAID5

[Bill Rugolsky Jr.]

> On Monday May 21, brugolsky@telemetry-investments.com wrote:
> > One could start a RAID 4/5/6 array over a degraded RAID1 for each
> > component, (i.e., a degraded RAID1).
 
That was supposed to say "a degraded RAID5+1".

On Tue, May 22, 2007 at 10:40:15AM +1000, Neil Brown wrote:
> This doesn't really have anything to do with the metadata used - it is
> primarily an implementation issue (though you would need to be careful
> picking up the pieces after a crash).
 
Right.  Doing it all automatically would be great, but involves some
kernel work.

What I was suggesting is that one administratively stop an array
built conventionally, and manually start a degraded RAID1 for each
component.  It seems that is possible using "--build".  Then one can
manually assemble the RAID5+1 from those RAID1s, and proceed with the
disk-by-disk replacement while the array is online.  Looks as if that
ought to be possible.  I'll have to give it a try, as soon as I have an
afternoon to tinker [say, ... mid-summer :-/].

Many thanks, Neil.

	-Bill