mdadm: recovering from an aborted reshape op - boot messages

mdadm: recovering from an aborted reshape op - boot messages

[Gavin Flower]

Hi Neil,

I did not notice this before (note: I have poor eyesight, so unless I explicitly look, I may not notice things!). but just before Fedora drops to the shell on a reboot I saw these messages (hand transcribed, so might have the odd transcription error):

/dev/md1: The filing system size (according to the superblock) is 76799952 blocks
The physical size of the device is 76799616
Either the superblock or the partition table is likely to be corrupt!

/dev/md1: UNEXPECTED INCONSISTENCY: RUN fsck manually
(i.e. without -a or -p options)

Note that original size according mdadm was not a multiple of 512KB, so I reshaped it to be the largest multiple or 512KB less than the original size.  So my second attempt to reshape, using the 512 chunk size, started okay.

Advice appreciated.


Thanks,
Gavin
--
All Adults share the Responsibility
to help Raise Today's Children,
for they are Tomorrow's Society!


      

Re: mdadm: recovering from an aborted reshape op - boot messages

[NeilBrown]

On Mon, 14 Feb 2011 14:47:48 -0800 (PST) Gavin Flower <gavinflower@yahoo.com>
wrote:

> Hi Neil,
> 
> I did not notice this before (note: I have poor eyesight, so unless I explicitly look, I may not notice things!). but just before Fedora drops to the shell on a reboot I saw these messages (hand transcribed, so might have the odd transcription error):
> 
> /dev/md1: The filing system size (according to the superblock) is 76799952 blocks
> The physical size of the device is 76799616
> Either the superblock or the partition table is likely to be corrupt!
> 
> /dev/md1: UNEXPECTED INCONSISTENCY: RUN fsck manually
> (i.e. without -a or -p options)
> 
> Note that original size according mdadm was not a multiple of 512KB, so I reshaped it to be the largest multiple or 512KB less than the original size.  So my second attempt to reshape, using the 512 chunk size, started okay.
> 
> Advice appreciated.

Hmmm....

Firstly, the -A and -E output you sent are inconsistent.

The "-A" output reports:

mdadm:/dev/md1 has an active reshape - checking if critical section needs to be restored

For 0.90 metadata (which you are using), that can only be reported if the
minor number is at least 91.  i.e. it has been temporarily set to 0.91.

However the "-E" output show that all devices are "0.90.00", not 0.91.

So those devices cannot possibly produce that -A output.

The devices appear to have all completely transitioned to 512K chunksize....

And the -D output seems to show that the array is fine and working properly.

Secondly, as you say you reshaped the array to make it slightly smaller so it
would be a multiple of 512K.  This is obviously needed to change the chunk
size.

But before you did that - did you resize the filesystem to be only that big?
I suspect not.  So the filesystem thinks that it is bigger than the device.
I don't know how best to fix that.

You could try running 'resize2fs" now (was it ext3? I don't remember).  Or
maybe an 'fsck -f' might fix it.

It might be safest to ask on ext3-users@redhat.com.  Report that you shrunk
your array before shrinking the filesystem and ask what the best remedial
strategy is.

NeilBrown

Re: mdadm: recovering from an aborted reshape op - boot messages

[Gavin Flower]

Hi Neil,

Comments interspersed..

--- On Tue, 15/2/11, NeilBrown <neilb@suse.de> wrote:

> From: NeilBrown <neilb@suse.de>
> Subject: Re: mdadm: recovering from an aborted reshape op - boot messages
> To: "Gavin Flower" <gavinflower@yahoo.com>
> Cc: linux-raid@vger.kernel.org
> Date: Tuesday, 15 February, 2011, 12:55
> On Mon, 14 Feb 2011 14:47:48 -0800
> (PST) Gavin Flower <gavinflower@yahoo.com>
> wrote:
> 
> > Hi Neil,
> > 
> > I did not notice this before (note: I have poor
> eyesight, so unless I explicitly look, I may not notice
> things!). but just before Fedora drops to the shell on a
> reboot I saw these messages (hand transcribed, so might have
> the odd transcription error):
> > 
> > /dev/md1: The filing system size (according to the
> superblock) is 76799952 blocks
> > The physical size of the device is 76799616
> > Either the superblock or the partition table is likely
> to be corrupt!
> > 
> > /dev/md1: UNEXPECTED INCONSISTENCY: RUN fsck manually
> > (i.e. without -a or -p options)
> > 
> > Note that original size according mdadm was not a
> multiple of 512KB, so I reshaped it to be the largest
> multiple or 512KB less than the original size.  So my
> second attempt to reshape, using the 512 chunk size, started
> okay.
> > 
> > Advice appreciated.
> 
> Hmmm....
> 
> Firstly, the -A and -E output you sent are inconsistent.

I can not explain the inconsistency.

However, they were both done on the same machine ('saturn').

No software updates were done on 'saturn' since before the reshaping.

The -A output was the process that took over an hour.

> The "-A" output reports:
> 
> mdadm:/dev/md1 has an active reshape - checking if critical
> section needs to be restored
> 
> For 0.90 metadata (which you are using), that can only be
> reported if the
> minor number is at least 91.  i.e. it has been
> temporarily set to 0.91.
> 
> However the "-E" output show that all devices are
> "0.90.00", not 0.91.

I grepped strings /sbin/mdadm for '.9', and found both '0.90' and '0.91' - for what it is worth.

ls on /sbin/mdadm gives the size of 362296 bytes and the date 5 Aug 2010.
version is v3.1.2 - 10th March 2010

> 
> So those devices cannot possibly produce that -A output.

The output was sent directly to the USB stick, so there are no transcription errors.  So as far as I can tell, these devices did produce the output. They are the only devices I have accessed using RAID many months.  There are only the 5 hard disks on 'saturn'.

Is there anything I can do to track down this anomaly?

> 
> The devices appear to have all completely transitioned to
> 512K chunksize....
> 
> And the -D output seems to show that the array is fine and
> working properly.
> 
> Secondly, as you say you reshaped the array to make it
> slightly smaller so it
> would be a multiple of 512K.  This is obviously needed
> to change the chunk
> size.

I used the –size=
 option of mdadm
> 
> But before you did that - did you resize the filesystem to
> be only that big?

No, and there is no mention in man mdadm to do so, that I could see.

> I suspect not.  So the filesystem thinks that it is
> bigger than the device.
> I don't know how best to fix that.

I would have thought mdadm would have done that as part of the process – as surely the size of the filesystem could not be reduced in advance of the reshaping.

Perhaps, I have overlooked the obvious?

> 
> You could try running 'resize2fs" now (was it ext3? I don't
> remember).  Or
> maybe an 'fsck -f' might fix it.
> 
> It might be safest to ask on ext3-users@redhat.com. 
> Report that you shrunk
> your array before shrinking the filesystem and ask what the
> best remedial
> strategy is.
> 
> NeilBrown
> 
> 

I will look into your other suggestions about recovery.

If there is anything further I can do, to provide useful diagnostics, please let me now.


Thanks,
Gavin


      
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mdadm: recovering from an aborted reshape op - boot messages

[NeilBrown]

On Mon, 14 Feb 2011 17:11:49 -0800 (PST) Gavin Flower <gavinflower@yahoo.com>
wrote:

> Hi Neil,
> 
> Comments interspersed..
> 
> --- On Tue, 15/2/11, NeilBrown <neilb@suse.de> wrote:
> 
> > From: NeilBrown <neilb@suse.de>
> > Subject: Re: mdadm: recovering from an aborted reshape op - boot messages
> > To: "Gavin Flower" <gavinflower@yahoo.com>
> > Cc: linux-raid@vger.kernel.org
> > Date: Tuesday, 15 February, 2011, 12:55
> > On Mon, 14 Feb 2011 14:47:48 -0800
> > (PST) Gavin Flower <gavinflower@yahoo.com>
> > wrote:
> > 
> > > Hi Neil,
> > > 
> > > I did not notice this before (note: I have poor
> > eyesight, so unless I explicitly look, I may not notice
> > things!). but just before Fedora drops to the shell on a
> > reboot I saw these messages (hand transcribed, so might have
> > the odd transcription error):
> > > 
> > > /dev/md1: The filing system size (according to the
> > superblock) is 76799952 blocks
> > > The physical size of the device is 76799616
> > > Either the superblock or the partition table is likely
> > to be corrupt!
> > > 
> > > /dev/md1: UNEXPECTED INCONSISTENCY: RUN fsck manually
> > > (i.e. without -a or -p options)
> > > 
> > > Note that original size according mdadm was not a
> > multiple of 512KB, so I reshaped it to be the largest
> > multiple or 512KB less than the original size.  So my
> > second attempt to reshape, using the 512 chunk size, started
> > okay.
> > > 
> > > Advice appreciated.
> > 
> > Hmmm....
> > 
> > Firstly, the -A and -E output you sent are inconsistent.
> 
> I can not explain the inconsistency.
> 
> However, they were both done on the same machine ('saturn').
> 
> No software updates were done on 'saturn' since before the reshaping.
> 
> The -A output was the process that took over an hour.
                                     ^^^^^^^^^^^^^^^^^^^

That is the piece of information that I was missing.
The array was assembled, reshape continued. Reshape completed.
Then you looked at the -E output after that.

So it all makes sense.  The reshape has completed.


> > Secondly, as you say you reshaped the array to make it
> > slightly smaller so it
> > would be a multiple of 512K.  This is obviously needed
> > to change the chunk
> > size.
> 
> I used the –size=
>  option of mdadm
> > 
> > But before you did that - did you resize the filesystem to
> > be only that big?
> 
> No, and there is no mention in man mdadm to do so, that I could see.
> 
> > I suspect not.  So the filesystem thinks that it is
> > bigger than the device.
> > I don't know how best to fix that.
> 
> I would have thought mdadm would have done that as part of the process – as surely the size of the filesystem could not be reduced in advance of the reshaping.
> 
> Perhaps, I have overlooked the obvious?

mdadm isn't not able to make the filesystem smaller, nor is it able to tell
if the filesystem is small enough that it won't be a problem.

I thought it was "obvious" that you would shrink the filesystem before
shrinking the device that contains it, but clearly it is not. I have updated
the man page - see patch below.

It would be really nice of the kernel could have some sort of inter-lock so
the device could ask the filesystem is shrinkage is OK, but we don't have
that currently.

NeilBrown



commit c26d78fe040673b019e1c2e4a5507969d2869765
Author: NeilBrown <neilb@suse.de>
Date:   Tue Feb 15 12:40:21 2011 +1100

    mdadm.man add encouragement to shrink filesystem before shrinking array.
    
    Before resizing an array with --size or --array-size, then filesystem
    should be resized.  mdadm cannot do this so the user should.
    
    Reported-by: Gavin Flower <gavinflower@yahoo.com>
    Signed-off-by: NeilBrown <neilb@suse.de>

diff --git a/mdadm.8.in b/mdadm.8.in
index fafb305..b2e6c05 100644
--- a/mdadm.8.in
+++ b/mdadm.8.in
@@ -427,12 +427,24 @@ The size can be given as
 .B max
 which means to choose the largest size that fits on all current drives.
 
+Before reducing the size of the array (with
+.BR "\-\-grow \-\-size=" )
+you should make sure that space isn't needed.  If the device holds a
+filesystem, you would need to resize the filesystem to use less space.
+
+After reducing the array size you should check that the data stored in
+the device is still available.  If the device holds a filesystem, then
+an 'fsck' of the filesystem is a minimum requirement.  If there are
+problems the array can be made bigger again with no loss with another
+.B "\-\-grow \-\-size="
+command.
+
 This value can not be used with
 .B CONTAINER
 metadata such as DDF and IMSM.
 
 .TP
-.BR \-Z ", " \-\-array-size=
+.BR \-Z ", " \-\-array\-size=
 This is only meaningful with
 .B \-\-grow
 and its effect is not persistent: when the array is stopped and
@@ -446,6 +458,17 @@ but setting the size with
 is, it is required that the array size is reduced as appropriate
 before the number of devices in the array is reduced.
 
+Before reducing the size of the array you should make sure that space
+isn't needed.  If the device holds a filesystem, you would need to
+resize the filesystem to use less space.
+
+After reducing the array size you should check that the data stored in
+the device is still available.  If the device holds a filesystem, then
+an 'fsck' of the filesystem is a minimum requirement.  If there are
+problems the array can be made bigger again with no loss with another
+.B "\-\-grow \-\-array\-size="
+command.
+
 A suffix of 'M' or 'G' can be given to indicate Megabytes or
 Gigabytes respectively.
 A value of

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mdadm: recovering from an aborted reshape op - boot messages

[Gavin Flower]

Hiel,

Comments interspersed.

--- On Tue, 15/2/11, NeilBrown <neilb@suse.de> wrote:

> From: NeilBrown <neilb@suse.de>
> Subject: Re: mdadm: recovering from an aborted reshape op - boot messages
> To: "Gavin Flower" <gavinflower@yahoo.com>
> Cc: linux-raid@vger.kernel.org
> Date: Tuesday, 15 February, 2011, 14:41
> On Mon, 14 Feb 2011 17:11:49 -0800
> (PST) Gavin Flower <gavinflower@yahoo.com>
> wrote:
> 
> > Hi Neil,
> > 
> > Comments interspersed..
> > 
> > --- On Tue, 15/2/11, NeilBrown <neilb@suse.de>
> wrote:
> > 
> > > From: NeilBrown <neilb@suse.de>
> > > Subject: Re: mdadm: recovering from an aborted
> reshape op - boot messages
> > > To: "Gavin Flower" <gavinflower@yahoo.com>
> > > Cc: linux-raid@vger.kernel.org
> > > Date: Tuesday, 15 February, 2011, 12:55
> > > On Mon, 14 Feb 2011 14:47:48 -0800
> > > (PST) Gavin Flower <gavinflower@yahoo.com>
> > > wrote:
> > > 
> > > > Hi Neil,
> > > > 
> > > > I did not notice this before (note: I have
> poor
> > > eyesight, so unless I explicitly look, I may not
> notice
> > > things!). but just before Fedora drops to the
> shell on a
> > > reboot I saw these messages (hand transcribed, so
> might have
> > > the odd transcription error):
> > > > 
> > > > /dev/md1: The filing system size (according
> to the
> > > superblock) is 76799952 blocks
> > > > The physical size of the device is 76799616
> > > > Either the superblock or the partition table
> is likely
> > > to be corrupt!
> > > > 
> > > > /dev/md1: UNEXPECTED INCONSISTENCY: RUN fsck
> manually
> > > > (i.e. without -a or -p options)
> > > > 
> > > > Note that original size according mdadm was
> not a
> > > multiple of 512KB, so I reshaped it to be the
> largest
> > > multiple or 512KB less than the original
> size.  So my
> > > second attempt to reshape, using the 512 chunk
> size, started
> > > okay.
> > > > 
> > > > Advice appreciated.
> > > 
> > > Hmmm....
> > > 
> > > Firstly, the -A and -E output you sent are
> inconsistent.
> > 
> > I can not explain the inconsistency.
> > 
> > However, they were both done on the same machine
> ('saturn').
> > 
> > No software updates were done on 'saturn' since before
> the reshaping.
> > 
> > The -A output was the process that took over an hour.
>                
>                
>      ^^^^^^^^^^^^^^^^^^^
> 
> That is the piece of information that I was missing.
> The array was assembled, reshape continued. Reshape
> completed.
> Then you looked at the -E output after that.
> 
> So it all makes sense.  The reshape has completed.
> 
> 
> > > Secondly, as you say you reshaped the array to
> make it
> > > slightly smaller so it
> > > would be a multiple of 512K.  This is
> obviously needed
> > > to change the chunk
> > > size.
> > 
> > I used the –size=
> >  option of mdadm
> > > 
> > > But before you did that - did you resize the
> filesystem to
> > > be only that big?
> > 
> > No, and there is no mention in man mdadm to do so,
> that I could see.
> > 
> > > I suspect not.  So the filesystem thinks
> that it is
> > > bigger than the device.
> > > I don't know how best to fix that.
> > 
> > I would have thought mdadm would have done that as
> part of the process – as surely the size of the filesystem
> could not be reduced in advance of the reshaping.
> > 

One obvious thing that I had overlooked, is that the RAID array _CONTAINS_ the filing system and _NOT_ vice versa- something I knew, but didn't recall when I wrote the above!

> > Perhaps, I have overlooked the obvious?
> 
> mdadm isn't not able to make the filesystem smaller, nor is
> it able to tell
> if the filesystem is small enough that it won't be a
> problem.
> 
> I thought it was "obvious" that you would shrink the
> filesystem before
> shrinking the device that contains it, but clearly it is
> not. I have updated
> the man page - see patch below.
> 
> It would be really nice of the kernel could have some sort
> of inter-lock so
> the device could ask the filesystem is shrinkage is OK, but
> we don't have
> that currently.
> 
> NeilBrown
> 
> 
> 
> commit c26d78fe040673b019e1c2e4a5507969d2869765
> Author: NeilBrown <neilb@suse.de>
> Date:   Tue Feb 15 12:40:21 2011 +1100
> 
>     mdadm.man add encouragement to shrink
> filesystem before shrinking array.
>     
>     Before resizing an array with --size or
> --array-size, then filesystem
>     should be resized.  mdadm cannot do this
> so the user should.
>     
>     Reported-by: Gavin Flower <gavinflower@yahoo.com>
>     Signed-off-by: NeilBrown <neilb@suse.de>
> 
> diff --git a/mdadm.8.in b/mdadm.8.in
> index fafb305..b2e6c05 100644
> --- a/mdadm.8.in
> +++ b/mdadm.8.in
> @@ -427,12 +427,24 @@ The size can be given as
>  .B max
>  which means to choose the largest size that fits on all
> current drives.
>  
> +Before reducing the size of the array (with
> +.BR "\-\-grow \-\-size=" )
> +you should make sure that space isn't needed.  If the
> device holds a
> +filesystem, you would need to resize the filesystem to use
> less space.
> +
> +After reducing the array size you should check that the
> data stored in
> +the device is still available.  If the device holds a
> filesystem, then
> +an 'fsck' of the filesystem is a minimum
> requirement.  If there are
> +problems the array can be made bigger again with no loss
> with another
> +.B "\-\-grow \-\-size="
> +command.
> +
>  This value can not be used with
>  .B CONTAINER
>  metadata such as DDF and IMSM.
>  
>  .TP
> -.BR \-Z ", " \-\-array-size=
> +.BR \-Z ", " \-\-array\-size=
>  This is only meaningful with
>  .B \-\-grow
>  and its effect is not persistent: when the array is
> stopped and
> @@ -446,6 +458,17 @@ but setting the size with
>  is, it is required that the array size is reduced as
> appropriate
>  before the number of devices in the array is reduced.
>  
> +Before reducing the size of the array you should make sure
> that space
> +isn't needed.  If the device holds a filesystem, you
> would need to
> +resize the filesystem to use less space.
> +
> +After reducing the array size you should check that the
> data stored in
> +the device is still available.  If the device holds a
> filesystem, then
> +an 'fsck' of the filesystem is a minimum
> requirement.  If there are
> +problems the array can be made bigger again with no loss
> with another
> +.B "\-\-grow \-\-array\-size="
> +command.
> +
>  A suffix of 'M' or 'G' can be given to indicate Megabytes
> or
>  Gigabytes respectively.
>  A value of
> 
> 

<Chuckle> I never expected to have my name on a commit to Linux!

output from: fsck.ext4 -f -n /dev/md1

e2fsck 1.41.12 (17-May-2010)
The filesystem size (according to the superblock) is 76799952 blocks
The physical size of the device is 76799616 blocks
Either the superblock or the partition table is likely to be corrupt!
Abort? no

Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences:  -9626 -(9728--9752) +(405344--405369)
Fix? no


I suspect that it might be simple to recover.  Are you able to advise on this, or should I run further diagnostics, or ask elsewhere.


Thanks,
Gavin

/dev/md1: ********** WARNING: Filesystem still has errors **********

/dev/md1: 1693644/19202048 files (0.3% non-contiguous), 54273929/76799952 blocks


      
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mdadm: recovering from an aborted reshape op - boot messages

[NeilBrown]

On Mon, 14 Feb 2011 18:04:01 -0800 (PST) Gavin Flower <gavinflower@yahoo.com>
wrote:

> output from: fsck.ext4 -f -n /dev/md1
> 
> e2fsck 1.41.12 (17-May-2010)
> The filesystem size (according to the superblock) is 76799952 blocks
> The physical size of the device is 76799616 blocks
> Either the superblock or the partition table is likely to be corrupt!
> Abort? no
> 
> Pass 1: Checking inodes, blocks, and sizes
> Pass 2: Checking directory structure
> Pass 3: Checking directory connectivity
> Pass 4: Checking reference counts
> Pass 5: Checking group summary information
> Block bitmap differences:  -9626 -(9728--9752) +(405344--405369)
> Fix? no
> 
> 
> I suspect that it might be simple to recover.  Are you able to advise on this, or should I run further diagnostics, or ask elsewhere.
> 

No I am not.

e2fsck has a very good reputation so if it seems to claim that it can fix
things then it almost certainly can.

Did you try asking on ext3-users@redhat.com ??

NeilBrown