RAID6 grow unexpected results

RAID6 grow unexpected results

[Liwei]

Hi list,
    I tried growing a RAID6 array by executing the mdadm --add and
mdadm --grow --raid-devices commands. Usually that should run fine and
I can continue with pvresize.

    This time however, the actual usable raid array size did not
change even though reshape begins. In my confusion, I decided to
reboot. That didn't solve the problem (and apparently the reshape was
interrupted, but mdadm shows the grown array as clean?)

    I then executed mdadm --grow --size=max, which seemed to have
increased the array size, but apparently by too much. Now I'm getting
"attempt to access beyond end of device" errors for each of the
component drives. A new reshape operation has also begun, which I
expect would fail as the array size is now too big.

    What did I do wrong and how can I remedy this?

Liwei

Re: RAID6 grow unexpected results

[Neil Brown]

[-- Attachment #1: Type: text/plain, Size: 1833 bytes --]

Liwei <xieliwei@gmail.com> writes:

> Hi list,
>     I tried growing a RAID6 array by executing the mdadm --add and
> mdadm --grow --raid-devices commands. Usually that should run fine and
> I can continue with pvresize.

That is incorrect.  You cannot run "pvresize" until the reshape
completes.

>
>     This time however, the actual usable raid array size did not
> change even though reshape begins. In my confusion, I decided to
> reboot. That didn't solve the problem (and apparently the reshape was
> interrupted, but mdadm shows the grown array as clean?)

Clean is expected.  It will be in an intermediate state, part has
reshaped to extra devices, part is still the original.

>
>     I then executed mdadm --grow --size=max, which seemed to have
> increased the array size, but apparently by too much. Now I'm getting
> "attempt to access beyond end of device" errors for each of the
> component drives. A new reshape operation has also begun, which I
> expect would fail as the array size is now too big.

"--size" isn't the same as "--array-size".
"--size" is the size of the component devices to use.

The "attempt to access beyond end of device" errors are unexpected but
presumably related o the "--size=max".  Maybe the 'max' chosen was too
large.  That suggests a bug somewhere.
You probably want to reduce that size back to where it was.

>
>     What did I do wrong and how can I remedy this?

Could we start with "mdadm --examine" status of all devices, "mdadm
--detail" of the array, and all kernel messages that could possibly
relate to the array since you first started this exercise?

NeilBrown

>
> Liwei
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: RAID6 grow unexpected results

[Liwei]

On Fri, 23 Oct 2015 at 06:37 Neil Brown <nfbrown@novell.com> wrote:
>
> Liwei <xieliwei@gmail.com> writes:
>
> > Hi list,
> >     I tried growing a RAID6 array by executing the mdadm --add and
> > mdadm --grow --raid-devices commands. Usually that should run fine and
> > I can continue with pvresize.
>
> That is incorrect.  You cannot run "pvresize" until the reshape
> completes.


Ah, my bad. It's been a while since I did that. So if I had been
patient, all would have been well?

>
>
> >
> >     This time however, the actual usable raid array size did not
> > change even though reshape begins. In my confusion, I decided to
> > reboot. That didn't solve the problem (and apparently the reshape was
> > interrupted, but mdadm shows the grown array as clean?)
>
> Clean is expected.  It will be in an intermediate state, part has
> reshaped to extra devices, part is still the original.
>
> >
> >     I then executed mdadm --grow --size=max, which seemed to have
> > increased the array size, but apparently by too much. Now I'm getting
> > "attempt to access beyond end of device" errors for each of the
> > component drives. A new reshape operation has also begun, which I
> > expect would fail as the array size is now too big.
>
> "--size" isn't the same as "--array-size".
> "--size" is the size of the component devices to use.
>
> The "attempt to access beyond end of device" errors are unexpected but
> presumably related o the "--size=max".  Maybe the 'max' chosen was too
> large.  That suggests a bug somewhere.
> You probably want to reduce that size back to where it was.


I just realised I got the wrong command. I used --array-size=max
instead. A reboot seemed to have reverted the change.

>
>
> >
> >     What did I do wrong and how can I remedy this?
>
> Could we start with "mdadm --examine" status of all devices, "mdadm
> --detail" of the array, and all kernel messages that could possibly
> relate to the array since you first started this exercise?


Eventually it was a foul up on my part. Things seem to be back to
normal after a reboot, and the reshape is in progress. Apologies for
the confusion!

Liwei

>
>
> NeilBrown
>
> >
> > Liwei
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID6 grow unexpected results

[Neil Brown]

[-- Attachment #1: Type: text/plain, Size: 2292 bytes --]

Liwei <xieliwei@gmail.com> writes:

> On Fri, 23 Oct 2015 at 06:37 Neil Brown <nfbrown@novell.com> wrote:
>>
>> Liwei <xieliwei@gmail.com> writes:
>>
>> > Hi list,
>> >     I tried growing a RAID6 array by executing the mdadm --add and
>> > mdadm --grow --raid-devices commands. Usually that should run fine and
>> > I can continue with pvresize.
>>
>> That is incorrect.  You cannot run "pvresize" until the reshape
>> completes.
>
>
> Ah, my bad. It's been a while since I did that. So if I had been
> patient, all would have been well?
>
>>
>>
>> >
>> >     This time however, the actual usable raid array size did not
>> > change even though reshape begins. In my confusion, I decided to
>> > reboot. That didn't solve the problem (and apparently the reshape was
>> > interrupted, but mdadm shows the grown array as clean?)
>>
>> Clean is expected.  It will be in an intermediate state, part has
>> reshaped to extra devices, part is still the original.
>>
>> >
>> >     I then executed mdadm --grow --size=max, which seemed to have
>> > increased the array size, but apparently by too much. Now I'm getting
>> > "attempt to access beyond end of device" errors for each of the
>> > component drives. A new reshape operation has also begun, which I
>> > expect would fail as the array size is now too big.
>>
>> "--size" isn't the same as "--array-size".
>> "--size" is the size of the component devices to use.
>>
>> The "attempt to access beyond end of device" errors are unexpected but
>> presumably related o the "--size=max".  Maybe the 'max' chosen was too
>> large.  That suggests a bug somewhere.
>> You probably want to reduce that size back to where it was.
>
>
> I just realised I got the wrong command. I used --array-size=max
> instead. A reboot seemed to have reverted the change.
>
>>
>>
>> >
>> >     What did I do wrong and how can I remedy this?
>>
>> Could we start with "mdadm --examine" status of all devices, "mdadm
>> --detail" of the array, and all kernel messages that could possibly
>> relate to the array since you first started this exercise?
>
>
> Eventually it was a foul up on my part. Things seem to be back to
> normal after a reboot, and the reshape is in progress. Apologies for
> the confusion!

Good to know that thing are working - thanks.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]