Re: mdadm: recovering from an aborted reshape op - boot messages

[NeilBrown <neilb@suse.de>]

On Mon, 14 Feb 2011 17:11:49 -0800 (PST) Gavin Flower <gavinflower@yahoo.com>
wrote:

> Hi Neil,
> 
> Comments interspersed..
> 
> --- On Tue, 15/2/11, NeilBrown <neilb@suse.de> wrote:
> 
> > From: NeilBrown <neilb@suse.de>
> > Subject: Re: mdadm: recovering from an aborted reshape op - boot messages
> > To: "Gavin Flower" <gavinflower@yahoo.com>
> > Cc: linux-raid@vger.kernel.org
> > Date: Tuesday, 15 February, 2011, 12:55
> > On Mon, 14 Feb 2011 14:47:48 -0800
> > (PST) Gavin Flower <gavinflower@yahoo.com>
> > wrote:
> > 
> > > Hi Neil,
> > > 
> > > I did not notice this before (note: I have poor
> > eyesight, so unless I explicitly look, I may not notice
> > things!). but just before Fedora drops to the shell on a
> > reboot I saw these messages (hand transcribed, so might have
> > the odd transcription error):
> > > 
> > > /dev/md1: The filing system size (according to the
> > superblock) is 76799952 blocks
> > > The physical size of the device is 76799616
> > > Either the superblock or the partition table is likely
> > to be corrupt!
> > > 
> > > /dev/md1: UNEXPECTED INCONSISTENCY: RUN fsck manually
> > > (i.e. without -a or -p options)
> > > 
> > > Note that original size according mdadm was not a
> > multiple of 512KB, so I reshaped it to be the largest
> > multiple or 512KB less than the original size.  So my
> > second attempt to reshape, using the 512 chunk size, started
> > okay.
> > > 
> > > Advice appreciated.
> > 
> > Hmmm....
> > 
> > Firstly, the -A and -E output you sent are inconsistent.
> 
> I can not explain the inconsistency.
> 
> However, they were both done on the same machine ('saturn').
> 
> No software updates were done on 'saturn' since before the reshaping.
> 
> The -A output was the process that took over an hour.
                                     ^^^^^^^^^^^^^^^^^^^

That is the piece of information that I was missing.
The array was assembled, reshape continued. Reshape completed.
Then you looked at the -E output after that.

So it all makes sense.  The reshape has completed.


> > Secondly, as you say you reshaped the array to make it
> > slightly smaller so it
> > would be a multiple of 512K.  This is obviously needed
> > to change the chunk
> > size.
> 
> I used the –size=
>  option of mdadm
> > 
> > But before you did that - did you resize the filesystem to
> > be only that big?
> 
> No, and there is no mention in man mdadm to do so, that I could see.
> 
> > I suspect not.  So the filesystem thinks that it is
> > bigger than the device.
> > I don't know how best to fix that.
> 
> I would have thought mdadm would have done that as part of the process – as surely the size of the filesystem could not be reduced in advance of the reshaping.
> 
> Perhaps, I have overlooked the obvious?

mdadm isn't not able to make the filesystem smaller, nor is it able to tell
if the filesystem is small enough that it won't be a problem.

I thought it was "obvious" that you would shrink the filesystem before
shrinking the device that contains it, but clearly it is not. I have updated
the man page - see patch below.

It would be really nice of the kernel could have some sort of inter-lock so
the device could ask the filesystem is shrinkage is OK, but we don't have
that currently.

NeilBrown



commit c26d78fe040673b019e1c2e4a5507969d2869765
Author: NeilBrown <neilb@suse.de>
Date:   Tue Feb 15 12:40:21 2011 +1100

    mdadm.man add encouragement to shrink filesystem before shrinking array.
    
    Before resizing an array with --size or --array-size, then filesystem
    should be resized.  mdadm cannot do this so the user should.
    
    Reported-by: Gavin Flower <gavinflower@yahoo.com>
    Signed-off-by: NeilBrown <neilb@suse.de>

diff --git a/mdadm.8.in b/mdadm.8.in
index fafb305..b2e6c05 100644
--- a/mdadm.8.in
+++ b/mdadm.8.in
@@ -427,12 +427,24 @@ The size can be given as
 .B max
 which means to choose the largest size that fits on all current drives.
 
+Before reducing the size of the array (with
+.BR "\-\-grow \-\-size=" )
+you should make sure that space isn't needed.  If the device holds a
+filesystem, you would need to resize the filesystem to use less space.
+
+After reducing the array size you should check that the data stored in
+the device is still available.  If the device holds a filesystem, then
+an 'fsck' of the filesystem is a minimum requirement.  If there are
+problems the array can be made bigger again with no loss with another
+.B "\-\-grow \-\-size="
+command.
+
 This value can not be used with
 .B CONTAINER
 metadata such as DDF and IMSM.
 
 .TP
-.BR \-Z ", " \-\-array-size=
+.BR \-Z ", " \-\-array\-size=
 This is only meaningful with
 .B \-\-grow
 and its effect is not persistent: when the array is stopped and
@@ -446,6 +458,17 @@ but setting the size with
 is, it is required that the array size is reduced as appropriate
 before the number of devices in the array is reduced.
 
+Before reducing the size of the array you should make sure that space
+isn't needed.  If the device holds a filesystem, you would need to
+resize the filesystem to use less space.
+
+After reducing the array size you should check that the data stored in
+the device is still available.  If the device holds a filesystem, then
+an 'fsck' of the filesystem is a minimum requirement.  If there are
+problems the array can be made bigger again with no loss with another
+.B "\-\-grow \-\-array\-size="
+command.
+
 A suffix of 'M' or 'G' can be given to indicate Megabytes or
 Gigabytes respectively.
 A value of

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html