Using mdadm --grow to resize a RAID1

Using mdadm --grow to resize a RAID1

[Benjammin2068]

hey all,

Quick question...

I've replaced and resync'd my root drives /dev/sda and /dev/sdb with 1TB drives up from the 250GB drives I started with.

/boot and swap are fine as is, but I'd like the root mount "/" to use the max available. (as anyone would)

My partition setup looks like:


> Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
> 255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
> Units = sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 4096 bytes
> I/O size (minimum/optimal): 4096 bytes / 4096 bytes
> Disk identifier: 0x0007868a
>
>    Device Boot      Start         End      Blocks   Id  System
> /dev/sda1            2048     8390655     4194304   fd  Linux raid autodetect
> /dev/sda2   *     8390656     9414655      512000   fd  Linux raid autodetect
> /dev/sda3         9414656  1953525167   972055256   fd  Linux raid autodetect
>
> Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
> 255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
> Units = sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 4096 bytes
> I/O size (minimum/optimal): 4096 bytes / 4096 bytes
> Disk identifier: 0x000d707c
>
>    Device Boot      Start         End      Blocks   Id  System
> /dev/sdb1            2048     8390655     4194304   fd  Linux raid autodetect
> /dev/sdb2   *     8390656     9414655      512000   fd  Linux raid autodetect
> /dev/sdb3         9414656  1953525167   972055256   fd  Linux raid autodetect
>
>

Currently, the array I want to resize has:

> [root@quantum ~]# mdadm --detail /dev/md1
> /dev/md1:
>         Version : 1.1
>   Creation Time : Sat Jul 30 15:21:45 2011
>      Raid Level : raid1
>      Array Size : 239490048 (228.40 GiB 245.24 GB)
>   Used Dev Size : 239490048 (228.40 GiB 245.24 GB)
>    Raid Devices : 2
>   Total Devices : 2
>     Persistence : Superblock is persistent
>
>     Update Time : Mon Jan  2 04:17:51 2017
>           State : clean
>  Active Devices : 2
> Working Devices : 2
>  Failed Devices : 0
>   Spare Devices : 0
>
>            Name : localhost.localdomain:1
>            UUID : 8d487d89:63690945:85c66cf4:e9885093
>          Events : 161820
>
>     Number   Major   Minor   RaidDevice State
>        2       8       19        0      active sync   /dev/sdb3
>        4       8        3        1      active sync   /dev/sda3

When I reference https://raid.wiki.kernel.org/index.php/Growing

>  mdadm --grow /dev/mdX --bitmap none
>  mdadm --grow /dev/mdX --size max
>  mdadm --wait /dev/mdX
>  mdadm --grow /dev/mdX --bitmap internal


I get stuck at the --size max.

it seems the correct syntax is "--size=" but that "max" is not supported as an argument.

little help?


Thanks,

 -Ben

Re: Using mdadm --grow to resize a RAID1

[Wols Lists]

On 02/01/17 10:18, Benjammin2068 wrote:
> I get stuck at the --size max.
> 
> it seems the correct syntax is "--size=" but that "max" is not supported as an argument.
> 
> little help?

Off the top of my head - try with no argument. iirc it defaults to max,
you don't need to tell it.

Cheers,
Wol

Re: Using mdadm --grow to resize a RAID1

[Benjammin2068]

On 01/02/2017 06:19 AM, Wols Lists wrote:
> On 02/01/17 10:18, Benjammin2068 wrote:
>> I get stuck at the --size max.
>>
>> it seems the correct syntax is "--size=" but that "max" is not supported as an argument.
>>
>> little help?
> Off the top of my head - try with no argument. iirc it defaults to max,
> you don't need to tell it.
>

Did that.


> [root@quantum ~]# mdadm --grow /dev/md1
> mdadm: no changes to --grow

> [root@quantum ~]# mdadm --grow /dev/md1 --size
> mdadm: option '--size' requires an argument
> Usage: mdadm --help
>   for help

> [root@quantum ~]# mdadm --grow /dev/md1 --size max
> mdadm: component size of /dev/md1 unchanged at 239490048K

> [root@quantum ~]# mdadm --grow /dev/md1 --size=max
> mdadm: component size of /dev/md1 unchanged at 239490048K

Re: Using mdadm --grow to resize a RAID1

[Jack Wang]

2017-01-02 16:55 GMT+01:00 Benjammin2068 <benjammin2068@gmail.com>:
>
>
> On 01/02/2017 06:19 AM, Wols Lists wrote:
>> On 02/01/17 10:18, Benjammin2068 wrote:
>>> I get stuck at the --size max.
>>>
>>> it seems the correct syntax is "--size=" but that "max" is not supported as an argument.
>>>
>>> little help?
>> Off the top of my head - try with no argument. iirc it defaults to max,
>> you don't need to tell it.
>>
>
> Did that.
>
>
>> [root@quantum ~]# mdadm --grow /dev/md1
>> mdadm: no changes to --grow
>
>> [root@quantum ~]# mdadm --grow /dev/md1 --size
>> mdadm: option '--size' requires an argument
>> Usage: mdadm --help
>>   for help
>
>> [root@quantum ~]# mdadm --grow /dev/md1 --size max
>> mdadm: component size of /dev/md1 unchanged at 239490048K
>
>> [root@quantum ~]# mdadm --grow /dev/md1 --size=max
>> mdadm: component size of /dev/md1 unchanged at 239490048K

So we need someone who has edit permission to update the the wiki page
in https://raid.wiki.kernel.org/index.php/Growing

Re: Using mdadm --grow to resize a RAID1

[Wols Lists]

On 02/01/17 16:14, Jack Wang wrote:
> 2017-01-02 16:55 GMT+01:00 Benjammin2068 <benjammin2068@gmail.com>:
>>
>>
>> On 01/02/2017 06:19 AM, Wols Lists wrote:
>>> On 02/01/17 10:18, Benjammin2068 wrote:
>>>> I get stuck at the --size max.
>>>>
>>>> it seems the correct syntax is "--size=" but that "max" is not supported as an argument.
>>>>
>>>> little help?
>>> Off the top of my head - try with no argument. iirc it defaults to max,
>>> you don't need to tell it.
>>>
>>
>> Did that.
>>
>>
>>> [root@quantum ~]# mdadm --grow /dev/md1
>>> mdadm: no changes to --grow
>>
>>> [root@quantum ~]# mdadm --grow /dev/md1 --size
>>> mdadm: option '--size' requires an argument
>>> Usage: mdadm --help
>>>   for help
>>
>>> [root@quantum ~]# mdadm --grow /dev/md1 --size max
>>> mdadm: component size of /dev/md1 unchanged at 239490048K
>>
>>> [root@quantum ~]# mdadm --grow /dev/md1 --size=max
>>> mdadm: component size of /dev/md1 unchanged at 239490048K
> 
> So we need someone who has edit permission to update the the wiki page
> in https://raid.wiki.kernel.org/index.php/Growing
> 
I've just checked the wiki page - it's not one of the ones I've yet
updated. I will be only too happy to do so, as soon as we can work out
(or someone tells us) the correct syntax to do so.

I must admit, using "man mdadm", it looks like Benjammin is doing the
right thing, so something's not quite right here ...

Cheers,
Wol

Re: Using mdadm --grow to resize a RAID1

[Roman Mamedov]

On Mon, 2 Jan 2017 09:55:34 -0600
Benjammin2068 <benjammin2068@gmail.com> wrote:

> 
> 
> On 01/02/2017 06:19 AM, Wols Lists wrote:
> > On 02/01/17 10:18, Benjammin2068 wrote:
> >> I get stuck at the --size max.
> >>
> >> it seems the correct syntax is "--size=" but that "max" is not supported as an argument.

First you say it's unsupported as an argument, but then actually 

> > [root@quantum ~]# mdadm --grow /dev/md1 --size max
> > mdadm: component size of /dev/md1 unchanged at 239490048K
> 
> > [root@quantum ~]# mdadm --grow /dev/md1 --size=max
> > mdadm: component size of /dev/md1 unchanged at 239490048K

Unsupported as an argument looks like that:

> mdadm --grow /dev/md1 --size=blah
> mdadm: invalid size: blah

In your case mdadm appears to think there is nowhere to grow the array.

Check with "blockdev --getsz /dev/sda3" and sdb3 that they actually do have a
proper size (the result will be in 512-byte sectors). 

If not, maybe your kernel didn't re-read the partition table after resizing
partitions (if that was what you did)?

If the blockdevice size is wrong, do a "blockdev --rereadpt /dev/sda" and sdb,
or just rebooting (as in some cases rereading will fail with a "device busy"
error).

-- 
With respect,
Roman

Re: Using mdadm --grow to resize a RAID1

[Benjammin2068]

On 01/02/2017 10:24 AM, Roman Mamedov wrote:
> On Mon, 2 Jan 2017 09:55:34 -0600
> Benjammin2068 <benjammin2068@gmail.com> wrote:
>
>>
>> On 01/02/2017 06:19 AM, Wols Lists wrote:
>>> On 02/01/17 10:18, Benjammin2068 wrote:
>>>> I get stuck at the --size max.
>>>>
>>>> it seems the correct syntax is "--size=" but that "max" is not supported as an argument.
> First you say it's unsupported as an argument, but then actually 
>
>>> [root@quantum ~]# mdadm --grow /dev/md1 --size max
>>> mdadm: component size of /dev/md1 unchanged at 239490048K
>>> [root@quantum ~]# mdadm --grow /dev/md1 --size=max
>>> mdadm: component size of /dev/md1 unchanged at 239490048K
> Unsupported as an argument looks like that:
>
>> mdadm --grow /dev/md1 --size=blah
>> mdadm: invalid size: blah

Sorry bout that -- when I said  "unsupported" - it was from the combined results that there is no mention of it in the man page even though the Wiki mentions it and the results are ambiguous in actual use.


> In your case mdadm appears to think there is nowhere to grow the array.
>
> Check with "blockdev --getsz /dev/sda3" and sdb3 that they actually do have a
> proper size (the result will be in 512-byte sectors). 
>
> If not, maybe your kernel didn't re-read the partition table after resizing
> partitions (if that was what you did)?
>
> If the blockdevice size is wrong, do a "blockdev --rereadpt /dev/sda" and sdb,
> or just rebooting (as in some cases rereading will fail with a "device busy"
> error).
>
Yep -- sizes are off and --rereadpt returns "device busy".

Be right back....

 -Ben

Re: Using mdadm --grow to resize a RAID1

[Ben Kamen]

On Mon, Jan 2, 2017 at 10:38 AM, Benjammin2068 <benjammin2068@gmail.com> wrote:
>
>
> Yep -- sizes are off and --rereadpt returns "device busy".
>
> Be right back....
>


ok. back.

the block sizes are the same...

There's still no bitmap active (from my turning it off previously)

So I issued the command again to expand the size... and my system has
been acting weird... my access to /proc/mdstat gets stalled if I'm
watching it..

The grow command "mdadm --grow /dev/md1 --size=max" seems to be
"blocking". I don't get back a prompt to enter the --wait command (and
the final bitmap command)

I can't even CTRL-C out of it. I had to reboot once already and the
filesystem comes back up as it was. But "reboot" even gets stuck. I
have to eventually reset.

Sounds like I have to do this from single user mode or an boot disk/CD?

 -Ben

Re: Using mdadm --grow to resize a RAID1

[Wols Lists]

On 02/01/17 17:59, Ben Kamen wrote:
> Sounds like I have to do this from single user mode or an boot disk/CD?

Important snippet of information we should have asked and I don't
remember seeing ... what kernel and mdadm are you using?

Cheers,
Wol

Re: Using mdadm --grow to resize a RAID1

[Ben Kamen]

On Mon, Jan 2, 2017 at 12:17 PM, Wols Lists <antlists@youngman.org.uk> wrote:
> On 02/01/17 17:59, Ben Kamen wrote:
>> Sounds like I have to do this from single user mode or an boot disk/CD?
>
> Important snippet of information we should have asked and I don't
> remember seeing ... what kernel and mdadm are you using?


My bad...

kernel is 2.6.32-642.11.1.el6.centos.plus.x86_64
mdadm.x86_64 is 3.3.4-1.el6_8.5

 -ben

Re: Using mdadm --grow to resize a RAID1

[Wols Lists]

On 02/01/17 18:38, Ben Kamen wrote:
> On Mon, Jan 2, 2017 at 12:17 PM, Wols Lists <antlists@youngman.org.uk> wrote:
>> On 02/01/17 17:59, Ben Kamen wrote:
>>> Sounds like I have to do this from single user mode or an boot disk/CD?
>>
>> Important snippet of information we should have asked and I don't
>> remember seeing ... what kernel and mdadm are you using?
> 
> 
> My bad...
> 
> kernel is 2.6.32-642.11.1.el6.centos.plus.x86_64
> mdadm.x86_64 is 3.3.4-1.el6_8.5
> 
OUCH!

I don't know how far the kernel and mdadm have come since then, and what
has or has not been backported, but it sounds like you want a modern
rescue CD.

I'd be inclined to get something like SystemRescueCD
https://www.system-rescue-cd.org/SystemRescueCd_Homepage and see if the
mdadm and kernel on that work any better. It wouldn't surprise me if
they do.

Cheers,
Wol

Re: Using mdadm --grow to resize a RAID1

[Ben Kamen]

On Mon, Jan 2, 2017 at 12:52 PM, Wols Lists <antlists@youngman.org.uk> wrote:
>
> OUCH!
>
> I don't know how far the kernel and mdadm have come since then, and what
> has or has not been backported, but it sounds like you want a modern
> rescue CD.
>
> I'd be inclined to get something like SystemRescueCD
> https://www.system-rescue-cd.org/SystemRescueCd_Homepage and see if the
> mdadm and kernel on that work any better. It wouldn't surprise me if
> they do.


LoL... yea... Long term stability over bleeding-nose "now what broke?!?"

This server has been plodding away for (I think) over 6yrs now.

I'll download and check it out. I had a feeling I'd need to go
single-user or boot from an external source.

I'm doing the grow on that pesky raid5 (now raid6) I mentioned a while
back. Turned out it looks to be the old samsung drives that were
causing me problems with that JBOD controller. I upgraded the RAID6 to
2TB drives while I was in the midst of getting rid of the 40,000+hr
old 1TB samsungs.

So I'm doing the grow on that from 3tb to 6tb as we speak from single
user mode and that's chugging along fine.

I'll report back later..

Thanks a bunch of the insight.

 -Ben

Re: Using mdadm --grow to resize a RAID1

[Ben Kamen]

Hey all --  just thought I'd report back.

Single User mode was enough to take care of doing the 'mdadm --grow'
and I just did everything else there too (resize2fs).

It was the block size kernel sync (or lack thereof) and then just
being up and running with alll those processes... hahaha.

Thanks everyone for the quick help!

 -Ben