Resize Raid5 devices

Resize Raid5 devices

[Ralf Müller]

Problem: I would like to grow the member devices of an existing raid5  
without
a 40 hour rebuild.

What happend up to here:
Two or three years before I created a raid5 with 5 300GB disks. A year  
before
I replaced 3 of these disks by 1.5TB models. At this time I decided to
partition these disks to have a 1.2TB partition at the beginning
and a 300GB partition at the end. The 300GB partitions replaced the  
300GB
disks, the 1.2 TB partitions became member of a new raid5 with a  
version 1.2
superblocks. I choose version 1.2 superblocks because they are at the  
beginning
of the disk so I thought I could resize the partition later without  
need to
rebuild the array.

Later has been an hour before ;)

After I added a forth 1.5TB disk to the array yesterday and reshaped the
former 3 disk raid5 to a 4 disk one, I wiped out the 300GB disk raid5,  
set one
of the 1.2TB partitions faulty, removed it from its raid, removed the  
300GB
partition and resized the former 1.2TB partition (in place) to 1.5TB.  
Now I
added this partition to its raid.

It has been recognized as a former member of this raid - so far so  
good - but
for whatever reason the raid subsystem decided to start a complete  
recovery.
So here my question:

What went wrong and what do I have to do to avoid a full recovery for  
the
next disk?

I do not feel well with a degraded raid5 in a 10 hour rebuild ...

Regards
Ralf

PS: no the system is not the fastest - it is optimized for size not for
speed.

DatenGrab:/media # uname -a
Linux DatenGrab 2.6.25.20-0.4-default #1 SMP 2009-06-01 09:57:12 +0200  
i686 i686 i386 GNU/Linux
DatenGrab:/media # mdadm --version
mdadm - v2.6.4 - 19th October 2007
DatenGrab:/media # cat /etc/SuSE-release
openSUSE 11.0 (i586)
VERSION = 11.0

DatenGrab:/media # cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid1]
md3 : active raid5 sdf1[3] sdi1[0] sdj1[4] sdh1[1]
       3457099008 blocks super 1.2 level 5, 256k chunk, algorithm 2  
[4/4] [UUUU]
       bitmap: 0/275 pages [0KB], 2048KB chunk

DatenGrab:/media # mdadm -f /dev/md3 /dev/sdf1
mdadm: set /dev/sdf1 faulty in /dev/md3

DatenGrab:/media # mdadm -r /dev/md3 /dev/sdf1
mdadm: hot removed /dev/sdf1

DatenGrab:/media # fdisk /dev/sdf
Command (m for help): p

Disk /dev/sdf: 1500.3 GB, 1500301910016 bytes
255 heads, 63 sectors/track, 182401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x26583dd9

    Device Boot      Start         End      Blocks   Id  System
/dev/sdf1               1      143463  1152366516   fd  Linux raid  
autodetect
/dev/sdf2          143489      182401   312568672+  fd  Linux raid  
autodetect

Command (m for help): d
Partition number (1-4): 2

Command (m for help): d
Selected partition 1

Command (m for help): n
[...]
Command (m for help): p

Disk /dev/sdf: 1500.3 GB, 1500301910016 bytes
255 heads, 63 sectors/track, 182401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x26583dd9

    Device Boot      Start         End      Blocks   Id  System
/dev/sdf1               1      182386  1465015513+  fd  Linux raid  
autodetect

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

DatenGrab:/media # mdadm -a /dev/md3 /dev/sdf1
mdadm: re-added /dev/sdf1

DatenGrab:/media # cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid1]
md3 : active raid5 sdf1[3] sdi1[0] sdj1[4] sdh1[1]
       3457099008 blocks super 1.2 level 5, 256k chunk, algorithm 2  
[4/3] [UUU_]
       [>....................]  recovery =  0.0% (83184/1152366336)  
finish=461.6min speed=41592K/sec
       bitmap: 2/275 pages [8KB], 2048KB chunk

Kernel log:
Jun 17 15:34:46 DatenGrab kernel: raid5: Disk failure on sdf1,  
disabling device.
  Operation continuing on 3 devices
Jun 17 15:34:55 DatenGrab kernel: RAID5 conf printout:
Jun 17 15:34:55 DatenGrab kernel:  --- rd:4 wd:3
Jun 17 15:34:55 DatenGrab kernel:  disk 0, o:1, dev:sdi1
Jun 17 15:34:55 DatenGrab kernel:  disk 1, o:1, dev:sdh1
Jun 17 15:34:55 DatenGrab kernel:  disk 2, o:1, dev:sdj1
Jun 17 15:34:55 DatenGrab kernel:  disk 3, o:0, dev:sdf1
Jun 17 15:34:55 DatenGrab kernel: RAID5 conf printout:
Jun 17 15:34:55 DatenGrab kernel:  --- rd:4 wd:3
Jun 17 15:34:55 DatenGrab kernel:  disk 0, o:1, dev:sdi1
Jun 17 15:34:55 DatenGrab kernel:  disk 1, o:1, dev:sdh1Jun 17  
15:34:55 DatenGrab kernel:  disk 2, o:1, dev:sdj1
Jun 17 15:34:55 DatenGrab kernel: md: unbind<sdf1>
Jun 17 15:34:55 DatenGrab kernel: md: export_rdev(sdf1)
Jun 17 15:37:54 DatenGrab kernel: sd 6:0:0:0: [sdf] 2930277168 512- 
byte hardware sectors (1500302 MB)
Jun 17 15:37:54 DatenGrab kernel: sd 6:0:0:0: [sdf] Write Protect is off
Jun 17 15:37:54 DatenGrab kernel: sd 6:0:0:0: [sdf] Mode Sense: 00 3a  
00 00Jun 17 15:37:54 DatenGrab kernel: sd 6:0:0:0: [sdf] Write cache:  
enabled, read c
ache: enabled, doesn't support DPO or FUA
Jun 17 15:37:54 DatenGrab kernel:  sdf: sdf1
Jun 17 15:37:56 DatenGrab kernel: sd 6:0:0:0: [sdf] 2930277168 512- 
byte hardware sectors (1500302 MB)
Jun 17 15:37:56 DatenGrab kernel: sd 6:0:0:0: [sdf] Write Protect is off
Jun 17 15:37:56 DatenGrab kernel: sd 6:0:0:0: [sdf] Mode Sense: 00 3a  
00 00
Jun 17 15:37:56 DatenGrab kernel: sd 6:0:0:0: [sdf] Write cache:  
enabled, read c
ache: enabled, doesn't support DPO or FUA
Jun 17 15:37:56 DatenGrab kernel:  sdf: sdf1

Jun 17 15:38:15 DatenGrab kernel: md: bind<sdf1>
Jun 17 15:38:15 DatenGrab kernel: RAID5 conf printout:
Jun 17 15:38:15 DatenGrab kernel:  --- rd:4 wd:3
Jun 17 15:38:15 DatenGrab kernel:  disk 0, o:1, dev:sdi1
Jun 17 15:38:15 DatenGrab kernel:  disk 1, o:1, dev:sdh1
Jun 17 15:38:15 DatenGrab kernel:  disk 2, o:1, dev:sdj1
Jun 17 15:38:15 DatenGrab kernel:  disk 3, o:1, dev:sdf1
Jun 17 15:38:15 DatenGrab kernel: md: recovery of RAID array md3
Jun 17 15:38:15 DatenGrab kernel: md: minimum _guaranteed_  speed:  
1000 KB/sec/disk.
Jun 17 15:38:15 DatenGrab kernel: md: using maximum available idle IO  
bandwidth (but not more than 200000 KB/sec) for recovery.
Jun 17 15:38:15 DatenGrab kernel: md: using 128k window, over a total  
of 1152366

-- 
Van Roy's Law: -------------------------------------------------------
        An unbreakable toy is useful for breaking other toys.

Re: Resize Raid5 devices

[Goswin von Brederlow]

Ralf Müller <ralf@bj-ig.de> writes:

> After I added a forth 1.5TB disk to the array yesterday and reshaped the
> former 3 disk raid5 to a 4 disk one, I wiped out the 300GB disk raid5,
> set one
> of the 1.2TB partitions faulty, removed it from its raid, removed the
> 300GB
> partition and resized the former 1.2TB partition (in place) to
> 1.5TB. Now I
> added this partition to its raid.
>
> It has been recognized as a former member of this raid - so far so
> good - but
> for whatever reason the raid subsystem decided to start a complete
> recovery.
> So here my question:
>
> What went wrong and what do I have to do to avoid a full recovery for
> the
> next disk?

I guess something wrote to your raid5 while the one disk was
removed. When you added it back the event counter would differ and the
disk needs to be resynced completly. A bitmap would help limiting this
to the parts that have changed. Add an internal bitmap before you
remove the next disk.

Alternatively stop the raid completly, resize all the partitions and
start the raid again. No need to fail/readd each disk in turn.

> I do not feel well with a degraded raid5 in a 10 hour rebuild ...
>
> Regards
> Ralf

MfG
        Goswin
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Resize Raid5 devices

[Ralf Müller]

Am 18.06.2009 um 10:27 schrieb Goswin von Brederlow:
> Ralf Müller <ralf@bj-ig.de> writes:
>
>> After I added a forth 1.5TB disk to the array yesterday and  
>> reshaped the
>> former 3 disk raid5 to a 4 disk one, I wiped out the 300GB disk  
>> raid5,
>> set one of the 1.2TB partitions faulty, removed it from its raid,
>> removed the 300GB partition and resized the former 1.2TB partition
>> (in place) to 1.5TB. Now I added this partition to its raid.
>>
>> It has been recognized as a former member of this raid - so far so
>> good - but for whatever reason the raid subsystem decided to start
>> a complete recovery.
>> So here my question:
>>
>> What went wrong and what do I have to do to avoid a full recovery for
>> the next disk?
>
> I guess something wrote to your raid5 while the one disk was
> removed.

Thats possible - after re-add, the bitmap showed 2/275 pages as unclean.

> When you added it back the event counter would differ and the
> disk needs to be resynced completly. A bitmap would help limiting this
> to the parts that have changed. Add an internal bitmap before you
> remove the next disk.

There is an internal write intent bitmap at the array:
DatenGrab:/media # cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid1]
md3 : active raid5 sdf1[3] sdi1[0] sdj1[4] sdh1[1]
      3457099008 blocks super 1.2 level 5, 256k chunk, algorithm 2  
[4/4] [UUUU]
      bitmap: 0/275 pages [0KB], 2048KB chunk

Do you have an idea why this bitmap has been ignored?

> Alternatively stop the raid completly, resize all the partitions and
> start the raid again. No need to fail/readd each disk in turn.

good idea ...

Thanks so far
Ralf

-- 
Van Roy's Law: -------------------------------------------------------
        An unbreakable toy is useful for breaking other toys.



--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Resize Raid5 devices

[Goswin von Brederlow]

Ralf Müller <ralf@bj-ig.de> writes:

> Am 18.06.2009 um 10:27 schrieb Goswin von Brederlow:
>> Ralf Müller <ralf@bj-ig.de> writes:
>>
>>> After I added a forth 1.5TB disk to the array yesterday and  
>>> reshaped the
>>> former 3 disk raid5 to a 4 disk one, I wiped out the 300GB disk  
>>> raid5,
>>> set one of the 1.2TB partitions faulty, removed it from its raid,
>>> removed the 300GB partition and resized the former 1.2TB partition
>>> (in place) to 1.5TB. Now I added this partition to its raid.
>>>
>>> It has been recognized as a former member of this raid - so far so
>>> good - but for whatever reason the raid subsystem decided to start
>>> a complete recovery.
>>> So here my question:
>>>
>>> What went wrong and what do I have to do to avoid a full recovery for
>>> the next disk?
>>
>> I guess something wrote to your raid5 while the one disk was
>> removed.
>
> Thats possible - after re-add, the bitmap showed 2/275 pages as unclean.
>
>> When you added it back the event counter would differ and the
>> disk needs to be resynced completly. A bitmap would help limiting this
>> to the parts that have changed. Add an internal bitmap before you
>> remove the next disk.
>
> There is an internal write intent bitmap at the array:
> DatenGrab:/media # cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4] [raid1]
> md3 : active raid5 sdf1[3] sdi1[0] sdj1[4] sdh1[1]
>       3457099008 blocks super 1.2 level 5, 256k chunk, algorithm 2  
> [4/4] [UUUU]
>       bitmap: 0/275 pages [0KB], 2048KB chunk
>
> Do you have an idea why this bitmap has been ignored?

No idea.

MfG
        Goswin
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Resize Raid5 devices

[Ralf Müller]

Am 19.06.2009 um 03:58 schrieb Neil Brown:
>> There is an internal write intent bitmap at the array:
>> DatenGrab:/media # cat /proc/mdstat
>> Personalities : [raid6] [raid5] [raid4] [raid1]
>> md3 : active raid5 sdf1[3] sdi1[0] sdj1[4] sdh1[1]
>>      3457099008 blocks super 1.2 level 5, 256k chunk, algorithm 2
>> [4/4] [UUUU]
>>      bitmap: 0/275 pages [0KB], 2048KB chunk
>>
>> Do you have an idea why this bitmap has been ignored?
>
> Probably a kernel bug.  There have been a couple of fixes relating to
> this since 2.6.25.  Hopefully it is all working in 2.6.30...
>
> The bug that you hit was probably the one fixed by
>  commit a0da84f35b25875870270d16b6eccda4884d61a7
> which is in 2.6.27.

Mh ... ok. So it may be a good idea to update to OpenSuSE 11.1 before
growing the other devices. Will do that at the weekend.

>>> Alternatively stop the raid completly, resize all the partitions and
>>> start the raid again. No need to fail/readd each disk in turn.
>>
>> good idea ...
>
> And when you re-assemble the array, use --update=devicesize.  That
> will ensure that md sees all of the new space on the devices.

This hint may have saved me a lot of time.

Thanks a lot
Ralf

-- 
Van Roy's Law: -------------------------------------------------------
        An unbreakable toy is useful for breaking other toys.

Re: Resize Raid5 devices

[Ralf Müller]

Am 19.06.2009 um 03:58 schrieb Neil Brown:
>>> When you added it back the event counter would differ and the
>>> disk needs to be resynced completly. A bitmap would help limiting  
>>> this
>>> to the parts that have changed. Add an internal bitmap before you
>>> remove the next disk.
>>
>> There is an internal write intent bitmap at the array:
>> DatenGrab:/media # cat /proc/mdstat
>> Personalities : [raid6] [raid5] [raid4] [raid1]
>> md3 : active raid5 sdf1[3] sdi1[0] sdj1[4] sdh1[1]
>>      3457099008 blocks super 1.2 level 5, 256k chunk, algorithm 2
>> [4/4] [UUUU]
>>      bitmap: 0/275 pages [0KB], 2048KB chunk
>>
>> Do you have an idea why this bitmap has been ignored?
>
> Probably a kernel bug.  There have been a couple of fixes relating to
> this since 2.6.25.  Hopefully it is all working in 2.6.30...
>
> The bug that you hit was probably the one fixed by
>  commit a0da84f35b25875870270d16b6eccda4884d61a7
> which is in 2.6.27.

Just for your information: I updated to OpenSuSE 11.1
(kernel 2.6.27.23-0.1) and tried to remove, grow, re-add the other
devices. For the first disk it worked - re-add took nearly no time
because of the bitmap. For the second disk - it did not work: complete
resync triggered. No idea why. It has been the same sequence of
commands on the same kind of device. There were no filesystem write
operations on the raid while updating.

I will wait for the resync to complete, offline the array, grow the
last device and reassemble with "--update=devicesize". Would there
have been a way to update the recorded device-size of the member
devices online? If the fail/grow/re-add cycle would have worked, I
still would have superblocks with the old size - right?

Regards
Ralf

-- 
Van Roy's Law: -------------------------------------------------------
        An unbreakable toy is useful for breaking other toys.

Re: Resize Raid5 devices

[NeilBrown]

On Sun, June 21, 2009 12:21 am, Ralf Müller wrote:
>
> Am 19.06.2009 um 03:58 schrieb Neil Brown:
>>>> When you added it back the event counter would differ and the
>>>> disk needs to be resynced completly. A bitmap would help limiting
>>>> this
>>>> to the parts that have changed. Add an internal bitmap before you
>>>> remove the next disk.
>>>
>>> There is an internal write intent bitmap at the array:
>>> DatenGrab:/media # cat /proc/mdstat
>>> Personalities : [raid6] [raid5] [raid4] [raid1]
>>> md3 : active raid5 sdf1[3] sdi1[0] sdj1[4] sdh1[1]
>>>      3457099008 blocks super 1.2 level 5, 256k chunk, algorithm 2
>>> [4/4] [UUUU]
>>>      bitmap: 0/275 pages [0KB], 2048KB chunk
>>>
>>> Do you have an idea why this bitmap has been ignored?
>>
>> Probably a kernel bug.  There have been a couple of fixes relating to
>> this since 2.6.25.  Hopefully it is all working in 2.6.30...
>>
>> The bug that you hit was probably the one fixed by
>>  commit a0da84f35b25875870270d16b6eccda4884d61a7
>> which is in 2.6.27.
>
> Just for your information: I updated to OpenSuSE 11.1
> (kernel 2.6.27.23-0.1) and tried to remove, grow, re-add the other
> devices. For the first disk it worked - re-add took nearly no time
> because of the bitmap. For the second disk - it did not work: complete
> resync triggered. No idea why. It has been the same sequence of
> commands on the same kind of device. There were no filesystem write
> operations on the raid while updating.

Thanks for keeping us informed.
The was another bug related to this that was fixed much more recently -
in 2.6.30 I think.
commit d0a4bb492772ce5c4bdfba3744a99ed6f6fb238f

I think it was a lot harder to hit, but maybe you hit it.

>
> I will wait for the resync to complete, offline the array, grow the
> last device and reassemble with "--update=devicesize". Would there
> have been a way to update the recorded device-size of the member
> devices online? If the fail/grow/re-add cycle would have worked, I
> still would have superblocks with the old size - right?

With a recent kernel, you can write "0" (or some other size
in sectors) to /sys/block/mdX/md/dev-YYY/size, and it will update the
device size online.  That functionality appeared in 2.6.26 so
OpenSUSE 11.1 should have it.

Yes - after fail/grow/re-add, the size recored in the metadata is
unchanged.  You need to increase the size on each device as described
above or with --update=devicesize during assemble, then update the
size of the array using "mdadm --grow --size=max /dev/mdX".

NeilBrown


--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Resize Raid5 devices

[Ralf Müller]

Am 20.06.2009 um 23:26 schrieb NeilBrown:

> On Sun, June 21, 2009 12:21 am, Ralf Müller wrote:
>> Am 19.06.2009 um 03:58 schrieb Neil Brown:
>>>> Do you have an idea why this bitmap has been ignored?
>>>
>>> Probably a kernel bug.  There have been a couple of fixes relating  
>>> to
>>> this since 2.6.25.  Hopefully it is all working in 2.6.30...
>>>
>>> The bug that you hit was probably the one fixed by
>>> commit a0da84f35b25875870270d16b6eccda4884d61a7
>>> which is in 2.6.27.
>>
>> Just for your information: I updated to OpenSuSE 11.1
>> (kernel 2.6.27.23-0.1) and tried to remove, grow, re-add the other
>> devices. For the first disk it worked - re-add took nearly no time
>> because of the bitmap. For the second disk - it did not work:  
>> complete
>> resync triggered. No idea why. It has been the same sequence of
>> commands on the same kind of device. There were no filesystem write
>> operations on the raid while updating.
>
> Thanks for keeping us informed.
> The was another bug related to this that was fixed much more  
> recently -
> in 2.6.30 I think.
> commit d0a4bb492772ce5c4bdfba3744a99ed6f6fb238f
>
> I think it was a lot harder to hit, but maybe you hit it.

Ok - when you said it was a lot harder to hit I thought "Hey - hit two
bugs in a row was a great deal, hit a seldom bug a second time ... this
will not happen" and I tried the 4th disk with fail/grow/re-add too.

I hit this bug again ... so it seems having a bitmap at these arrays
saved 10 out of 40 hours for me. Not that impressive ;)

The array does not have a lot of write load (actually write operations
happen in _multi_ GB chunks one to four times a year).

>> Would there have been a way to update the recorded device-size of
>> the member devices online? If the fail/grow/re-add cycle would
>> have worked, I still would have superblocks with the old
>> size - right?
>
> With a recent kernel, you can write "0" (or some other size
> in sectors) to /sys/block/mdX/md/dev-YYY/size, and it will update the
> device size online.  That functionality appeared in 2.6.26 so
> OpenSUSE 11.1 should have it.

It does. And as far as I can say it works. The new device size is
reported in /sys/.../dev-YYY/size and with mdadm -E. So I expect
these values to actually be written to the superblocks.

Regards
Ralf

-- 
Van Roy's Law: -------------------------------------------------------
        An unbreakable toy is useful for breaking other toys.



--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html