Shrinking number of devices on a RAID-10 (near 2) array

Shrinking number of devices on a RAID-10 (near 2) array

[Andy Smith]

Hi,

I am aware that for a very long time it was not possible to change
the number of devices in an mdadm RAID-10. But then recently I am
sure I saw threads here alluding to this being possible. e.g.:

    http://marc.info/?l=linux-raid&m=140768923829685&w=2

I have a 6 device RAID-10 near=2 array that I would like to shrink
down to 4 devices.

I have compiled mdadm 3.3.2 and am using kernel 3.16.0.

$ sudo mdadm --detail /dev/md2
/dev/md2:
        Version : 0.90
  Creation Time : Sun Jun  4 08:18:58 2006
     Raid Level : raid10
     Array Size : 471859200 (450.00 GiB 483.18 GB)
  Used Dev Size : 309363520 (295.03 GiB 316.79 GB)
   Raid Devices : 6
  Total Devices : 6
Preferred Minor : 2
    Persistence : Superblock is persistent

    Update Time : Sat Aug 23 15:23:38 2014
          State : active 
 Active Devices : 6
Working Devices : 6
 Failed Devices : 0
  Spare Devices : 0

         Layout : near=2
     Chunk Size : 64K

           UUID : 3905b303:ca604b72:be5949c4:ab051b7a
         Events : 0.312149991

    Number   Major   Minor   RaidDevice State
       0       8       51        0      active sync   /dev/sdd3
       1       8       67        1      active sync   /dev/sde3
       2       8       83        2      active sync   /dev/sdf3
       3       8       19        3      active sync   /dev/sdb3
       4       8       35        4      active sync   /dev/sdc3
       5       8        3        5      active sync   /dev/sda3

$ sudo ./mdadm --grow -n4 /dev/md2 --backup-file /var/tmp/mdadm.backup
mdadm: Cannot set new_data_offset: RAID10 reshape not
       supported on this kernel

Is that not yet possible, then?

(Each device is 320GB so it should all fit with only four of them)

Cheers,
Andy

Re: Shrinking number of devices on a RAID-10 (near 2) array

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2578 bytes --]

On Sat, 23 Aug 2014 16:31:10 +0000 Andy Smith <andy@strugglers.net> wrote:

> Hi,
> 
> I am aware that for a very long time it was not possible to change
> the number of devices in an mdadm RAID-10. But then recently I am
> sure I saw threads here alluding to this being possible. e.g.:
> 
>     http://marc.info/?l=linux-raid&m=140768923829685&w=2
> 
> I have a 6 device RAID-10 near=2 array that I would like to shrink
> down to 4 devices.
> 
> I have compiled mdadm 3.3.2 and am using kernel 3.16.0.
> 
> $ sudo mdadm --detail /dev/md2
> /dev/md2:
>         Version : 0.90
>   Creation Time : Sun Jun  4 08:18:58 2006
>      Raid Level : raid10
>      Array Size : 471859200 (450.00 GiB 483.18 GB)
>   Used Dev Size : 309363520 (295.03 GiB 316.79 GB)
>    Raid Devices : 6
>   Total Devices : 6
> Preferred Minor : 2
>     Persistence : Superblock is persistent
> 
>     Update Time : Sat Aug 23 15:23:38 2014
>           State : active 
>  Active Devices : 6
> Working Devices : 6
>  Failed Devices : 0
>   Spare Devices : 0
> 
>          Layout : near=2
>      Chunk Size : 64K
> 
>            UUID : 3905b303:ca604b72:be5949c4:ab051b7a
>          Events : 0.312149991
> 
>     Number   Major   Minor   RaidDevice State
>        0       8       51        0      active sync   /dev/sdd3
>        1       8       67        1      active sync   /dev/sde3
>        2       8       83        2      active sync   /dev/sdf3
>        3       8       19        3      active sync   /dev/sdb3
>        4       8       35        4      active sync   /dev/sdc3
>        5       8        3        5      active sync   /dev/sda3
> 
> $ sudo ./mdadm --grow -n4 /dev/md2 --backup-file /var/tmp/mdadm.backup
> mdadm: Cannot set new_data_offset: RAID10 reshape not
>        supported on this kernel
> 
> Is that not yet possible, then?
> 
> (Each device is 320GB so it should all fit with only four of them)

raid10 reshape is only supported for v1.x metadata, and you have 0.90.
This is because it needs to move the data_offset and 0.90 cannot do that.

I see you have already adjusted the --array-size correctly - good.

You can achieve your result by:

 1/ stop the array and assemble with --update=metadata.  This will convert the
    array to v1.0
 2/ "mdadm /dev/md2 --grow --disks=4"  That should perform the reshape.
    You don't need a --backup-file with RAID10 ... or with other arrays if you
    have recent tools and enough space before or after.

I guess the error message could be better....

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

RE: Shrinking number of devices on a RAID-10 (near 2) array

[Craig Curtin]

Neil,

I assume you would need to shrink the filesystem first before this ? or does MDADM nicely do that as part of the process also ?

Craig

-----Original Message-----
From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-owner@vger.kernel.org] On Behalf Of NeilBrown
Sent: Sunday, 24 August 2014 1:10 PM
To: Andy Smith
Cc: linux-raid@vger.kernel.org
Subject: Re: Shrinking number of devices on a RAID-10 (near 2) array

On Sat, 23 Aug 2014 16:31:10 +0000 Andy Smith <andy@strugglers.net> wrote:

> Hi,
>
> I am aware that for a very long time it was not possible to change the
> number of devices in an mdadm RAID-10. But then recently I am sure I
> saw threads here alluding to this being possible. e.g.:
>
>     http://marc.info/?l=linux-raid&m=140768923829685&w=2
>
> I have a 6 device RAID-10 near=2 array that I would like to shrink
> down to 4 devices.
>
> I have compiled mdadm 3.3.2 and am using kernel 3.16.0.
>
> $ sudo mdadm --detail /dev/md2
> /dev/md2:
>         Version : 0.90
>   Creation Time : Sun Jun  4 08:18:58 2006
>      Raid Level : raid10
>      Array Size : 471859200 (450.00 GiB 483.18 GB)
>   Used Dev Size : 309363520 (295.03 GiB 316.79 GB)
>    Raid Devices : 6
>   Total Devices : 6
> Preferred Minor : 2
>     Persistence : Superblock is persistent
>
>     Update Time : Sat Aug 23 15:23:38 2014
>           State : active
>  Active Devices : 6
> Working Devices : 6
>  Failed Devices : 0
>   Spare Devices : 0
>
>          Layout : near=2
>      Chunk Size : 64K
>
>            UUID : 3905b303:ca604b72:be5949c4:ab051b7a
>          Events : 0.312149991
>
>     Number   Major   Minor   RaidDevice State
>        0       8       51        0      active sync   /dev/sdd3
>        1       8       67        1      active sync   /dev/sde3
>        2       8       83        2      active sync   /dev/sdf3
>        3       8       19        3      active sync   /dev/sdb3
>        4       8       35        4      active sync   /dev/sdc3
>        5       8        3        5      active sync   /dev/sda3
>
> $ sudo ./mdadm --grow -n4 /dev/md2 --backup-file /var/tmp/mdadm.backup
> mdadm: Cannot set new_data_offset: RAID10 reshape not
>        supported on this kernel
>
> Is that not yet possible, then?
>
> (Each device is 320GB so it should all fit with only four of them)

raid10 reshape is only supported for v1.x metadata, and you have 0.90.
This is because it needs to move the data_offset and 0.90 cannot do that.

I see you have already adjusted the --array-size correctly - good.

You can achieve your result by:

 1/ stop the array and assemble with --update=metadata.  This will convert the
    array to v1.0
 2/ "mdadm /dev/md2 --grow --disks=4"  That should perform the reshape.
    You don't need a --backup-file with RAID10 ... or with other arrays if you
    have recent tools and enough space before or after.

I guess the error message could be better....

NeilBrown

Disclaimer

CONFIDENTIAL

This message contains confidential information and is intended only for the intended recipients. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.

Re: Shrinking number of devices on a RAID-10 (near 2) array

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 4992 bytes --]

On Sun, 24 Aug 2014 06:28:25 +0000 Craig Curtin <craigc@prosis.com.au> wrote:

> Neil,
> 
> I assume you would need to shrink the filesystem first before this ? or does MDADM nicely do that as part of the process also ?

mdadm is not a jack-of-all-trades.  It won't touch your filesystem.

A careful look at Andy's "mdadm --detail" shows:

> >      Array Size : 471859200 (450.00 GiB 483.18 GB)
> >   Used Dev Size : 309363520 (295.03 GiB 316.79 GB)

This is a near=2 6 drive RAID10, so Array Size should be 3 times Used Dev
Size.
The fact that it isn't shows that he has already explicitly set the
--array-size to something appropriate.   I'm sure he reduced the size of the
filesystem first as the man page says:

              Before  reducing the size of the array you should make sure that
              space isn't needed.  If the device holds a filesystem, you would
              need to resize the filesystem to use less space.

If you try to resize an array to fewer drives and you haven't adjusted the
--array-size already, mdadm will tell you what --array-size you need.  This
forces you into a two (at least) step process to minimize the chance of doing
something silly.
The idea is that you "--grow --array-size" first, then use "fsck" to make
sure the filesystem is still happy.  Then use "--grow --raid-disks" to
reshape the array without changing the size.
If you change the array size and get it wrong, you can always put it back.
But once you start the reshape happening, you will have lost any data beyond
"array-size".

NeilBrown


> 
> Craig
> 
> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-owner@vger.kernel.org] On Behalf Of NeilBrown
> Sent: Sunday, 24 August 2014 1:10 PM
> To: Andy Smith
> Cc: linux-raid@vger.kernel.org
> Subject: Re: Shrinking number of devices on a RAID-10 (near 2) array
> 
> On Sat, 23 Aug 2014 16:31:10 +0000 Andy Smith <andy@strugglers.net> wrote:
> 
> > Hi,
> >
> > I am aware that for a very long time it was not possible to change the
> > number of devices in an mdadm RAID-10. But then recently I am sure I
> > saw threads here alluding to this being possible. e.g.:
> >
> >     http://marc.info/?l=linux-raid&m=140768923829685&w=2
> >
> > I have a 6 device RAID-10 near=2 array that I would like to shrink
> > down to 4 devices.
> >
> > I have compiled mdadm 3.3.2 and am using kernel 3.16.0.
> >
> > $ sudo mdadm --detail /dev/md2
> > /dev/md2:
> >         Version : 0.90
> >   Creation Time : Sun Jun  4 08:18:58 2006
> >      Raid Level : raid10
> >      Array Size : 471859200 (450.00 GiB 483.18 GB)
> >   Used Dev Size : 309363520 (295.03 GiB 316.79 GB)
> >    Raid Devices : 6
> >   Total Devices : 6
> > Preferred Minor : 2
> >     Persistence : Superblock is persistent
> >
> >     Update Time : Sat Aug 23 15:23:38 2014
> >           State : active
> >  Active Devices : 6
> > Working Devices : 6
> >  Failed Devices : 0
> >   Spare Devices : 0
> >
> >          Layout : near=2
> >      Chunk Size : 64K
> >
> >            UUID : 3905b303:ca604b72:be5949c4:ab051b7a
> >          Events : 0.312149991
> >
> >     Number   Major   Minor   RaidDevice State
> >        0       8       51        0      active sync   /dev/sdd3
> >        1       8       67        1      active sync   /dev/sde3
> >        2       8       83        2      active sync   /dev/sdf3
> >        3       8       19        3      active sync   /dev/sdb3
> >        4       8       35        4      active sync   /dev/sdc3
> >        5       8        3        5      active sync   /dev/sda3
> >
> > $ sudo ./mdadm --grow -n4 /dev/md2 --backup-file /var/tmp/mdadm.backup
> > mdadm: Cannot set new_data_offset: RAID10 reshape not
> >        supported on this kernel
> >
> > Is that not yet possible, then?
> >
> > (Each device is 320GB so it should all fit with only four of them)
> 
> raid10 reshape is only supported for v1.x metadata, and you have 0.90.
> This is because it needs to move the data_offset and 0.90 cannot do that.
> 
> I see you have already adjusted the --array-size correctly - good.
> 
> You can achieve your result by:
> 
>  1/ stop the array and assemble with --update=metadata.  This will convert the
>     array to v1.0
>  2/ "mdadm /dev/md2 --grow --disks=4"  That should perform the reshape.
>     You don't need a --backup-file with RAID10 ... or with other arrays if you
>     have recent tools and enough space before or after.
> 
> I guess the error message could be better....
> 
> NeilBrown
> 
> Disclaimer
> 
> CONFIDENTIAL
> 
> This message contains confidential information and is intended only for the intended recipients. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Shrinking number of devices on a RAID-10 (near 2) array

[Andy Smith]

On Sun, Aug 24, 2014 at 04:45:07PM +1000, NeilBrown wrote:
> > >      Array Size : 471859200 (450.00 GiB 483.18 GB)
> > >   Used Dev Size : 309363520 (295.03 GiB 316.79 GB)
> 
> This is a near=2 6 drive RAID10, so Array Size should be 3 times Used Dev
> Size.
> The fact that it isn't shows that he has already explicitly set the
> --array-size to something appropriate.   I'm sure he reduced the size of the
> filesystem first as the man page says:

Yup, it's an LVM PV so I did pvresize and mdadm --array-size first
to some safe lower value, before I realised that mdadm would have
told me the exact --array-size I needed anyway.

I should have mentioned that to reduce confusion, but it didn't
seem pertinent to where I got stuck.

Cheers,
Andy

Re: Shrinking number of devices on a RAID-10 (near 2) array

[Andy Smith]

Hi Neil,

Thanks for the quick response!

On Sun, Aug 24, 2014 at 01:09:39PM +1000, NeilBrown wrote:
> raid10 reshape is only supported for v1.x metadata, and you have 0.90.
> This is because it needs to move the data_offset and 0.90 cannot do that.
> 
> I see you have already adjusted the --array-size correctly - good.
> 
> You can achieve your result by:
> 
>  1/ stop the array and assemble with --update=metadata.  This will convert the
>     array to v1.0

I've now done this, so:

$ sudo mdadm --detail /dev/md2
/dev/md2:
        Version : 1.0
  Creation Time : Sun Jun  4 08:18:59 2006
     Raid Level : raid10
     Array Size : 618726528 (590.06 GiB 633.58 GB)
  Used Dev Size : 309363264 (295.03 GiB 316.79 GB)
   Raid Devices : 6
  Total Devices : 6
    Persistence : Superblock is persistent

    Update Time : Sun Aug 24 14:29:15 2014
          State : active 
 Active Devices : 6
Working Devices : 6
 Failed Devices : 0
  Spare Devices : 0

         Layout : near=2
     Chunk Size : 64K

           Name : 2
           UUID : 3905b303:ca604b72:be5949c4:ab051b7a
         Events : 1

    Number   Major   Minor   RaidDevice State
       0       8       51        0      active sync   /dev/sdd3
       1       8       67        1      active sync   /dev/sde3
       2       8       83        2      active sync   /dev/sdf3
       3       8       19        3      active sync   /dev/sdb3
       4       8       35        4      active sync   /dev/sdc3
       5       8        3        5      active sync   /dev/sda3

I noticed something: The assemble with update re-set the array size
to 928089792 (885.10 GiB 950.36 GB).

(the above --detail is after all steps below)

>  2/ "mdadm /dev/md2 --grow --disks=4"  That should perform the reshape.

I assume that should have been --raid-devices; --disks seems to be a
syntax error. :)

Firstly it told me the array was too big, as expected.

$ sudo ./mdadm /dev/md2 --grow --raid-devices=4
mdadm: this change will reduce the size of the array.
       use --grow --array-size first to truncate array.
       e.g. mdadm --grow /dev/md2 --array-size 618726528
unfreeze

Okay.

$ sudo ./mdadm --grow /dev/md2 --array-size 618726528

But:

$ sudo ./mdadm /dev/md2 --grow --raid-devices=4
mdadm: Cannot set new_data_offset: RAID10 reshape not
       supported on this kernel

> I guess the error message could be better....

Will be happy to send the trivial patch once we get to the bottom of
this!

Cheers,
Andy

Re: Shrinking number of devices on a RAID-10 (near 2) array

[Andy Smith]

On Sun, Aug 24, 2014 at 02:39:44PM +0000, Andy Smith wrote:
> $ sudo ./mdadm /dev/md2 --grow --raid-devices=4
> mdadm: Cannot set new_data_offset: RAID10 reshape not
>        supported on this kernel

Grow.c from latest git:

2397                         err = sysfs_set_num(sra, sd, "new_offset",
2398                                             new_data_offset);

For the first device in the for loop (sda3), the above line 2397 is
returning -1.

2399                 }
2400                 if (err < 0) {
2401                         if (errno == E2BIG && data_offset != INVALID_SECTORS) {
2402                                 pr_err("data-offset is too big for %s\n",
2403                                        dn);
2404                                 goto release;
2405                         }
2406                         if (sd == sra->devs &&
2407                             (errno == ENOENT || errno == E2BIG))
2408                                 /* Early kernel, no 'new_offset' file,
2409                                  * or kernel doesn't like us.
2410                                  * For RAID5/6 this is not fatal
2411                                  */
2412                                 return 1;

It reaches here line 2412 with errno == E2BIG.
/sys/block/md2/md/dev-sda3/new_offset exists.

From strace:

open("/sys/block/md2/md/dev-sda3/new_offset", O_WRONLY) = 4
write(4, "128", 3)                      = -1 E2BIG (Argument list too long)
close(4)                                = 0

Not sure where to debug next.

Cheers,
Andy

-- 
> The optimum programming team size is 1.
Has Jurassic Park taught us nothing?
 — pfilandr
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Shrinking number of devices on a RAID-10 (near 2) array

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2150 bytes --]

On Mon, 25 Aug 2014 10:32:34 +0000 Andy Smith <andy@strugglers.net> wrote:

> On Sun, Aug 24, 2014 at 02:39:44PM +0000, Andy Smith wrote:
> > $ sudo ./mdadm /dev/md2 --grow --raid-devices=4
> > mdadm: Cannot set new_data_offset: RAID10 reshape not
> >        supported on this kernel
> 
> Grow.c from latest git:
> 
> 2397                         err = sysfs_set_num(sra, sd, "new_offset",
> 2398                                             new_data_offset);
> 
> For the first device in the for loop (sda3), the above line 2397 is
> returning -1.
> 
> 2399                 }
> 2400                 if (err < 0) {
> 2401                         if (errno == E2BIG && data_offset != INVALID_SECTORS) {
> 2402                                 pr_err("data-offset is too big for %s\n",
> 2403                                        dn);
> 2404                                 goto release;
> 2405                         }
> 2406                         if (sd == sra->devs &&
> 2407                             (errno == ENOENT || errno == E2BIG))
> 2408                                 /* Early kernel, no 'new_offset' file,
> 2409                                  * or kernel doesn't like us.
> 2410                                  * For RAID5/6 this is not fatal
> 2411                                  */
> 2412                                 return 1;
> 
> It reaches here line 2412 with errno == E2BIG.
> /sys/block/md2/md/dev-sda3/new_offset exists.
> 
> >From strace:
> 
> open("/sys/block/md2/md/dev-sda3/new_offset", O_WRONLY) = 4
> write(4, "128", 3)                      = -1 E2BIG (Argument list too long)
> close(4)                                = 0
> 
> Not sure where to debug next.

That comes from
	else if (new_offset > rdev->data_offset) {
		/* must not push array size beyond rdev_sectors */
		if (new_offset - rdev->data_offset
		    + mddev->dev_sectors > rdev->sectors)
				return -E2BIG;

in drivers/md/md.c in the kernel...

What does "mdadm --examine" on one of the devices report now that it is v1.0?

Particularly interested in the "Unused Space :" line.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Shrinking number of devices on a RAID-10 (near 2) array

[Andy Smith]

Hi Neil,

On Mon, Aug 25, 2014 at 09:26:14PM +1000, NeilBrown wrote:
> What does "mdadm --examine" on one of the devices report now that it is v1.0?
> 
> Particularly interested in the "Unused Space :" line.

/dev/sda3:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x0
     Array UUID : 3905b303:ca604b72:be5949c4:ab051b7a
           Name : 2
  Creation Time : Sun Jun  4 08:18:59 2006
     Raid Level : raid10
   Raid Devices : 6

 Avail Dev Size : 618726528 (295.03 GiB 316.79 GB)
     Array Size : 928089792 (885.10 GiB 950.36 GB)
   Super Offset : 618727392 sectors
   Unused Space : before=0 sectors, after=864 sectors
          State : active
    Device UUID : e30176be:81a57e84:1f2aa206:9515150f

    Update Time : Sun Aug 24 14:29:15 2014
       Checksum : bd599c72 - correct
         Events : 1

         Layout : near=2
     Chunk Size : 64K

   Device Role : Active device 5
   Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)

All other devices report same (apart from UUID/checksum/role).

Cheers,
Andy

Re: Shrinking number of devices on a RAID-10 (near 2) array

[Andy Smith]

Hi Neil,

Any ideas on the below? Not in a huge rush but at some point I might
have to just re-create the array and copy data back.

Cheers,
Andy

On Mon, Aug 25, 2014 at 11:34:03AM +0000, Andy Smith wrote:
> Hi Neil,
> 
> On Mon, Aug 25, 2014 at 09:26:14PM +1000, NeilBrown wrote:
> > What does "mdadm --examine" on one of the devices report now that it is v1.0?
> > 
> > Particularly interested in the "Unused Space :" line.
> 
> /dev/sda3:
>           Magic : a92b4efc
>         Version : 1.0
>     Feature Map : 0x0
>      Array UUID : 3905b303:ca604b72:be5949c4:ab051b7a
>            Name : 2
>   Creation Time : Sun Jun  4 08:18:59 2006
>      Raid Level : raid10
>    Raid Devices : 6
> 
>  Avail Dev Size : 618726528 (295.03 GiB 316.79 GB)
>      Array Size : 928089792 (885.10 GiB 950.36 GB)
>    Super Offset : 618727392 sectors
>    Unused Space : before=0 sectors, after=864 sectors
>           State : active
>     Device UUID : e30176be:81a57e84:1f2aa206:9515150f
> 
>     Update Time : Sun Aug 24 14:29:15 2014
>        Checksum : bd599c72 - correct
>          Events : 1
> 
>          Layout : near=2
>      Chunk Size : 64K
> 
>    Device Role : Active device 5
>    Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
> 
> All other devices report same (apart from UUID/checksum/role).

Re: Shrinking number of devices on a RAID-10 (near 2) array

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2229 bytes --]

On Mon, 25 Aug 2014 11:34:03 +0000 Andy Smith <andy@strugglers.net> wrote:

> Hi Neil,
> 
> On Mon, Aug 25, 2014 at 09:26:14PM +1000, NeilBrown wrote:
> > What does "mdadm --examine" on one of the devices report now that it is v1.0?
> > 
> > Particularly interested in the "Unused Space :" line.
> 
> /dev/sda3:
>           Magic : a92b4efc
>         Version : 1.0
>     Feature Map : 0x0
>      Array UUID : 3905b303:ca604b72:be5949c4:ab051b7a
>            Name : 2
>   Creation Time : Sun Jun  4 08:18:59 2006
>      Raid Level : raid10
>    Raid Devices : 6
> 
>  Avail Dev Size : 618726528 (295.03 GiB 316.79 GB)
>      Array Size : 928089792 (885.10 GiB 950.36 GB)
>    Super Offset : 618727392 sectors
>    Unused Space : before=0 sectors, after=864 sectors
>           State : active
>     Device UUID : e30176be:81a57e84:1f2aa206:9515150f
> 
>     Update Time : Sun Aug 24 14:29:15 2014
>        Checksum : bd599c72 - correct
>          Events : 1
> 
>          Layout : near=2
>      Chunk Size : 64K
> 
>    Device Role : Active device 5
>    Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
> 
> All other devices report same (apart from UUID/checksum/role).

'after=864' should be plenty.

mdadm writes '128' to 'new_offset'
This is bigger than 'data_offset' (which must be zero, or it would have been
reported), mddev->dev_sectors is 618726528 while rdev->sectors is 618727392
(difference is 864), so 
		if (new_offset - rdev->data_offset
		    + mddev->dev_sectors > rdev->sectors)
cannot return 'true'.  Yet it seems that it does.

Ahh.. somewhere else returns -E2BIG:

	if (mddev->pers && mddev->persistent &&
	    !super_types[mddev->major_version]
	    .allow_new_offset(rdev, new_offset))
		return -E2BIG;

But super_1_allow_new_offset always succeeds if new_offset >
rdev->data_offset, so that cannot be the problem.


I know.  rdev->sectors isn't "Super Offset", it is only "Avail Dev Size".
You need to
  echo 618727392 > /sys/block/md2/md/dev-sda3/size
and same for the other devices.

I looks like I need to get mdadm to be more clever when it gets -E2BIG.
Can you check that the above works?

Thanks,
NeilBrown


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Shrinking number of devices on a RAID-10 (near 2) array

[Andy Smith]

Hi Neil,

On Fri, Aug 29, 2014 at 01:53:12PM +1000, NeilBrown wrote:
> You need to
>   echo 618727392 > /sys/block/md2/md/dev-sda3/size
> and same for the other devices.
> 
> I looks like I need to get mdadm to be more clever when it gets -E2BIG.
> Can you check that the above works?

Unfortunately it seems I cannot change that size:

# echo 618727392 > /sys/block/md2/md/dev-sda3/size
# cat /sys/block/md2/md/dev-sda3/size
309363264

Cheers,
Andy

Re: Shrinking number of devices on a RAID-10 (near 2) array

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 707 bytes --]

On Fri, 29 Aug 2014 04:02:31 +0000 Andy Smith <andy@strugglers.net> wrote:

> Hi Neil,
> 
> On Fri, Aug 29, 2014 at 01:53:12PM +1000, NeilBrown wrote:
> > You need to
> >   echo 618727392 > /sys/block/md2/md/dev-sda3/size
> > and same for the other devices.
> > 
> > I looks like I need to get mdadm to be more clever when it gets -E2BIG.
> > Can you check that the above works?
> 
> Unfortunately it seems I cannot change that size:
> 
> # echo 618727392 > /sys/block/md2/md/dev-sda3/size
> # cat /sys/block/md2/md/dev-sda3/size
> 309363264
> 

Of course 'size' is in kilobytes, not sectors (which moron did that !??!?!)

echo 309363696 > /sys/block/md2/md/dev-sda3/size

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Shrinking number of devices on a RAID-10 (near 2) array

[Andy Smith]

On Fri, Aug 29, 2014 at 02:18:12PM +1000, NeilBrown wrote:
> Of course 'size' is in kilobytes, not sectors (which moron did that !??!?!)
> 
> echo 309363696 > /sys/block/md2/md/dev-sda3/size

Sorry, still no joy:

# echo 309363696 > /sys/block/md2/md/dev-sda3/size
# cat /sys/block/md2/md/dev-sda3/size
309363264

Re: Shrinking number of devices on a RAID-10 (near 2) array

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 667 bytes --]

On Fri, 29 Aug 2014 04:26:18 +0000 Andy Smith <andy@strugglers.net> wrote:

> On Fri, Aug 29, 2014 at 02:18:12PM +1000, NeilBrown wrote:
> > Of course 'size' is in kilobytes, not sectors (which moron did that !??!?!)
> > 
> > echo 309363696 > /sys/block/md2/md/dev-sda3/size
> 
> Sorry, still no joy:
> 
> # echo 309363696 > /sys/block/md2/md/dev-sda3/size
> # cat /sys/block/md2/md/dev-sda3/size
> 309363264

Try use "/bin/echo" so you get an error message.
Also try smaller numbers.  Can you write anything larger than what is already
there?  Increasing by 64 (match the 128 sectors that mdadm wants for
new_offset) should  be enough.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Shrinking number of devices on a RAID-10 (near 2) array

[Andy Smith]

Hi Neil,

On Fri, Aug 29, 2014 at 02:35:28PM +1000, NeilBrown wrote:
> On Fri, 29 Aug 2014 04:26:18 +0000 Andy Smith <andy@strugglers.net> wrote:
> > Sorry, still no joy:
> > 
> > # echo 309363696 > /sys/block/md2/md/dev-sda3/size
> > # cat /sys/block/md2/md/dev-sda3/size
> > 309363264
> 
> Try use "/bin/echo" so you get an error message.
> Also try smaller numbers.  Can you write anything larger than what is already
> there?  Increasing by 64 (match the 128 sectors that mdadm wants for
> new_offset) should  be enough.

I can't even increase it by 64 from what it is now:

# /bin/echo 309363328 > /sys/block/md2/md/dev-sda3/size
# cat /sys/block/md2/md/dev-sda3/size
309363264

In fact I can't increase it by 1, and /bin/echo doesn't say why.

Cheers,
Andy

Re: Shrinking number of devices on a RAID-10 (near 2) array

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2370 bytes --]

On Fri, 29 Aug 2014 04:42:26 +0000 Andy Smith <andy@strugglers.net> wrote:

> Hi Neil,
> 
> On Fri, Aug 29, 2014 at 02:35:28PM +1000, NeilBrown wrote:
> > On Fri, 29 Aug 2014 04:26:18 +0000 Andy Smith <andy@strugglers.net> wrote:
> > > Sorry, still no joy:
> > > 
> > > # echo 309363696 > /sys/block/md2/md/dev-sda3/size
> > > # cat /sys/block/md2/md/dev-sda3/size
> > > 309363264
> > 
> > Try use "/bin/echo" so you get an error message.
> > Also try smaller numbers.  Can you write anything larger than what is already
> > there?  Increasing by 64 (match the 128 sectors that mdadm wants for
> > new_offset) should  be enough.
> 
> I can't even increase it by 64 from what it is now:
> 
> # /bin/echo 309363328 > /sys/block/md2/md/dev-sda3/size
> # cat /sys/block/md2/md/dev-sda3/size
> 309363264
> 
> In fact I can't increase it by 1, and /bin/echo doesn't say why.
> 

Hmm.... I've obviously been assuming something that was wrong.

So I did some experiments myself....

If you apply the following patch to mdadm, stop the array, and then assemble
with --update=devicesize, you should then be able to reshape the array to
fewer devices.

This isn't a final solution, I need to think about the whole thing more
carefully.  But it will work for you.

Thanks for your patience.

NeilBrown

diff --git a/super1.c b/super1.c
index 727a08a110e0..efa1042ab47a 100644
--- a/super1.c
+++ b/super1.c
@@ -1251,6 +1251,17 @@ static int update_super1(struct supertype *st, struct mdinfo *info,
 			misc->device_size - __le64_to_cpu(sb->data_offset));
 		printf("Size is %llu\n", (unsigned long long)
 		       __le64_to_cpu(sb->data_size));
+	} else if (strcmp(update, "devicesize") == 0 &&
+		   !(__le32_to_cpu(sb->feature_map)&MD_FEATURE_BITMAP_OFFSET) &&
+		   __le64_to_cpu(sb->super_offset) >
+		   __le64_to_cpu(sb->data_offset)) {
+		/* set data_size to device size less data_offset */
+		printf("Size was %llu\n", (unsigned long long)
+		       __le64_to_cpu(sb->data_size));
+		sb->data_size = __cpu_to_le64(
+			__le64_to_cpu(sb->super_offset) - __le64_to_cpu(sb->data_offset));
+		printf("Size is %llu\n", (unsigned long long)
+		       __le64_to_cpu(sb->data_size));
 	} else if (strcmp(update, "revert-reshape") == 0) {
 		rv = -2;
 		if (!(sb->feature_map & __cpu_to_le32(MD_FEATURE_RESHAPE_ACTIVE)))

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Shrinking number of devices on a RAID-10 (near 2) array

[Andy Smith]

Hi Neil,

On Fri, Aug 29, 2014 at 04:04:43PM +1000, NeilBrown wrote:
> On Fri, 29 Aug 2014 04:42:26 +0000 Andy Smith <andy@strugglers.net> wrote:
> > In fact I can't increase it by 1, and /bin/echo doesn't say why.
> > 
> 
> Hmm.... I've obviously been assuming something that was wrong.
> 
> So I did some experiments myself....
> 
> If you apply the following patch to mdadm, stop the array, and then assemble
> with --update=devicesize, you should then be able to reshape the array to
> fewer devices.

Thanks, I think we are getting somewhere:

# ./mdadm --stop /dev/md2
mdadm: stopped /dev/md2
# ./mdadm --assemble --update=devicesize /dev/md2
Size was 618726528
Size is 618727392
Size was 618726528
Size is 618727392
Size was 618726528
Size is 618727392
Size was 618726528
Size is 618727392
Size was 618726528
Size is 618727392
Size was 618726528
Size is 618727392
mdadm: /dev/md2 has been started with 6 drives.
# logout
$ head -5 /proc/mdstat
Personalities : [raid1] [raid10]
md2 : active raid10 sdd3[0] sda3[5] sdc3[4] sdb3[3] sdf3[2] sde3[1]
      928089792 blocks super 1.0 64K chunks 2 near-copies [6/6] [UUUUUU]

$ sudo ./mdadm /dev/md2 --grow --raid-devices=4
mdadm: this change will reduce the size of the array.
       use --grow --array-size first to truncate array.
       e.g. mdadm --grow /dev/md2 --array-size 618726528
unfreeze
$ sudo ./mdadm --grow /dev/md2 --array-size 618726528
$ sudo ./mdadm /dev/md2 --grow --raid-devices=4
$ head -5 /proc/mdstat
Personalities : [raid1] [raid10]
md2 : active raid10 sdd3[0] sda3[5] sdc3[4] sdb3[3] sdf3[2] sde3[1]
      618726528 blocks super 1.0 64K chunks 2 near-copies [4/4] [UUUU]
      [>....................]  reshape =  0.0% (211136/618726528) finish=244.1min speed=42227K/sec

> Thanks for your patience.

No problem, glad we got to the bottom of it!

Cheers,
Andy

[PATCH 1/1] Grow: Report when grow needs metadata update

[Andy Smith]

Report when the array's metadata needs updating instead of just
reporting the generic "kernel too old" message.

Signed-off-by: Andy Smith <andy@strugglers.net>
---
 Grow.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/Grow.c b/Grow.c
index a9c8589..76bb35a 100644
--- a/Grow.c
+++ b/Grow.c
@@ -2255,7 +2255,9 @@ static int set_new_data_offset(struct mdinfo *sra, struct supertype *st,
 		if (info2.space_before == 0 &&
 		    info2.space_after == 0) {
 			/* Metadata doesn't support data_offset changes */
-			return 1;
+			pr_err("%s: Metadata version doesn't support"
+			       " data_offset changes\n", devname);
+			goto release;
 		}
 		if (before > info2.space_before)
 			before = info2.space_before;
-- 
1.7.10.4

Re: [PATCH 1/1] Grow: Report when grow needs metadata update

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 919 bytes --]

On Fri, 29 Aug 2014 20:47:12 +0000 Andy Smith <andy@strugglers.net> wrote:

> Report when the array's metadata needs updating instead of just
> reporting the generic "kernel too old" message.
> 
> Signed-off-by: Andy Smith <andy@strugglers.net>
> ---
>  Grow.c |    4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/Grow.c b/Grow.c
> index a9c8589..76bb35a 100644
> --- a/Grow.c
> +++ b/Grow.c
> @@ -2255,7 +2255,9 @@ static int set_new_data_offset(struct mdinfo *sra, struct supertype *st,
>  		if (info2.space_before == 0 &&
>  		    info2.space_after == 0) {
>  			/* Metadata doesn't support data_offset changes */
> -			return 1;
> +			pr_err("%s: Metadata version doesn't support"
> +			       " data_offset changes\n", devname);
> +			goto release;
>  		}
>  		if (before > info2.space_before)
>  			before = info2.space_before;

Applied, thanks.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]