mdadm 2.6, linux 2.6.17, failed grow/reshape

mdadm 2.6, linux 2.6.17, failed grow/reshape

[Thomas Langås]

I sent this message directly to Neil Brown first, but then I read
his homepage and found out I should have sent it here, so here goes.

I've got 5 250GB drives, 4 of which I used to create an
RAID5-md-device, after all that was done (striping and all that) I
added the last drive.  Problem is that about 30% into the process of
reshaping, my computer died, and when I've got things up on a new
system now, I can get it to start where it left off.

It says that the device (md0) is in delta 4->5, but I'm unable to get
it to restart reshaping.  Are all the data I had on the raid-device
lost?  Or is it possible to make it restart where it left off?  It
clearly knows how far it has come along, since I get some stats about
that.


In advance, thanks!


-- 
Thomas

mdadm 2.6, linux 2.6.17, failed grow/reshape

[Thomas Langås]

On 2/20/07, Neil Brown <neilb@suse.de> wrote:
> > It says that the device (md0) is in delta 4->5, but I'm unable to get
> > it to restart reshaping.  Are all the data I had on the raid-device
> > lost?  Or is it possible to make it restart where it left off?  It
> > clearly knows how far it has come along, since I get some stats about
> > that.
> Hmm... it should "just work", and it my testing it does.

What should "kick off" the rebuild/reshape-process once it has been
interrupted?

> The reshape process regularly checkpoints its progress so it knows
> where to start off from, and when you assemble the array again, it
> should restart the reshape.

Ok, no matter if it's a different system assembling the drives?  The
kernel is not running the md-device (if there is one?) from
mdadm-2.6-source, it's running the one inside 2.6.17.

> Can you post kernel log messages showing md0 being assembled?  Also
> the output of "mdadm -E" on a couple of drives might be helpful.

Running assemble:
# ./mdadm --assemble --scan
mdadm: failed to RUN_ARRAY /dev/.static/dev/md0: Invalid argument

Ouput from syslog:
Feb 20 20:20:09 ubuntu kernel: [17197083.928000] md: md0 stopped.
Feb 20 20:20:09 ubuntu kernel: [17197084.012000] md: bind<sdd1>
Feb 20 20:20:09 ubuntu kernel: [17197084.012000] md: bind<sde1>
Feb 20 20:20:09 ubuntu kernel: [17197084.012000] md: bind<sdf1>
Feb 20 20:20:09 ubuntu kernel: [17197084.012000] md: bind<sdg1>
Feb 20 20:20:09 ubuntu kernel: [17197084.016000] md: bind<sdc1>

Details:
# ./mdadm --detail /dev/md0
/dev/md0:
        Version : 00.91.03
  Creation Time : Sun Feb 18 01:19:36 2007
     Raid Level : raid5
  Used Dev Size : 244195904 (232.88 GiB 250.06 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Sun Feb 18 14:18:48 2007
          State : active, Not Started
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

  Delta Devices : 1, (4->5)

           UUID : e100e921:53bc1ec5:0ff90348:cdad8788
         Events : 0.71599

    Number   Major   Minor   RaidDevice State
       0       8       33        0      active sync   /dev/sdc1
       1       8       49        1      active sync   /dev/sdd1
       2       8       65        2      active sync   /dev/sde1
       3       8       81        3      active sync   /dev/sdf1
       4       8       97        4      active sync   /dev/sdg1

# ./mdadm -E /dev/sdc1
/dev/sdc1:
          Magic : a92b4efc
        Version : 00.91.03
           UUID : e100e921:53bc1ec5:0ff90348:cdad8788
  Creation Time : Sun Feb 18 01:19:36 2007
     Raid Level : raid5
  Used Dev Size : 244195904 (232.88 GiB 250.06 GB)
     Array Size : 976783616 (931.53 GiB 1000.23 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 0

  Reshape pos'n : 383536128 (365.77 GiB 392.74 GB)
  Delta Devices : 1 (4->5)

    Update Time : Sun Feb 18 14:18:48 2007
          State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 8389d7b3 - correct
         Events : 0.71599

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     0       8       33        0      active sync   /dev/sdc1

   0     0       8       33        0      active sync   /dev/sdc1
   1     1       8       49        1      active sync   /dev/sdd1
   2     2       8       65        2      active sync   /dev/sde1
   3     3       8       81        3      active sync   /dev/sdf1
   4     4       8      129        4      active sync
root@ubuntu:/home/ubuntu/mdadm-2.6# ./mdadm -E /dev/sdd1
/dev/sdd1:
          Magic : a92b4efc
        Version : 00.91.03
           UUID : e100e921:53bc1ec5:0ff90348:cdad8788
  Creation Time : Sun Feb 18 01:19:36 2007
     Raid Level : raid5
  Used Dev Size : 244195904 (232.88 GiB 250.06 GB)
     Array Size : 976783616 (931.53 GiB 1000.23 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 0

  Reshape pos'n : 383536128 (365.77 GiB 392.74 GB)
  Delta Devices : 1 (4->5)

    Update Time : Sun Feb 18 14:18:48 2007
          State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 8389d7c5 - correct
         Events : 0.71599

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     1       8       49        1      active sync   /dev/sdd1

   0     0       8       33        0      active sync   /dev/sdc1
   1     1       8       49        1      active sync   /dev/sdd1
   2     2       8       65        2      active sync   /dev/sde1
   3     3       8       81        3      active sync   /dev/sdf1
   4     4       8      129        4      active sync
root@ubuntu:/home/ubuntu/mdadm-2.6# ./mdadm -E /dev/sde1
/dev/sde1:
          Magic : a92b4efc
        Version : 00.91.03
           UUID : e100e921:53bc1ec5:0ff90348:cdad8788
  Creation Time : Sun Feb 18 01:19:36 2007
     Raid Level : raid5
  Used Dev Size : 244195904 (232.88 GiB 250.06 GB)
     Array Size : 976783616 (931.53 GiB 1000.23 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 0

  Reshape pos'n : 383536128 (365.77 GiB 392.74 GB)
  Delta Devices : 1 (4->5)

    Update Time : Sun Feb 18 14:18:48 2007
          State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 8389d7d7 - correct
         Events : 0.71599

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     2       8       65        2      active sync   /dev/sde1

   0     0       8       33        0      active sync   /dev/sdc1
   1     1       8       49        1      active sync   /dev/sdd1
   2     2       8       65        2      active sync   /dev/sde1
   3     3       8       81        3      active sync   /dev/sdf1
   4     4       8      129        4      active sync
root@ubuntu:/home/ubuntu/mdadm-2.6# ./mdadm -E /dev/sdf1
/dev/sdf1:
          Magic : a92b4efc
        Version : 00.91.03
           UUID : e100e921:53bc1ec5:0ff90348:cdad8788
  Creation Time : Sun Feb 18 01:19:36 2007
     Raid Level : raid5
  Used Dev Size : 244195904 (232.88 GiB 250.06 GB)
     Array Size : 976783616 (931.53 GiB 1000.23 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 0

  Reshape pos'n : 383536128 (365.77 GiB 392.74 GB)
  Delta Devices : 1 (4->5)

    Update Time : Sun Feb 18 14:18:48 2007
          State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 8389d7e9 - correct
         Events : 0.71599

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     3       8       81        3      active sync   /dev/sdf1

   0     0       8       33        0      active sync   /dev/sdc1
   1     1       8       49        1      active sync   /dev/sdd1
   2     2       8       65        2      active sync   /dev/sde1
   3     3       8       81        3      active sync   /dev/sdf1
   4     4       8      129        4      active sync


If you have any more I need to run, please let me know and I'll run
the commands :)

--
Thomas

Re: mdadm 2.6, linux 2.6.17, failed grow/reshape

[Neil Brown]

On Tuesday February 20, thomas.langaas@gmail.com wrote:
> On 2/20/07, Neil Brown <neilb@suse.de> wrote:
> > > It says that the device (md0) is in delta 4->5, but I'm unable to get
> > > it to restart reshaping.  Are all the data I had on the raid-device
> > > lost?  Or is it possible to make it restart where it left off?  It
> > > clearly knows how far it has come along, since I get some stats about
> > > that.
> > Hmm... it should "just work", and it my testing it does.
> 
> What should "kick off" the rebuild/reshape-process once it has been
> interrupted?

When the array is started, it notices the reshape is incomplete and
kick-starts the reshape.

> 
> > The reshape process regularly checkpoints its progress so it knows
> > where to start off from, and when you assemble the array again, it
> > should restart the reshape.
> 
> Ok, no matter if it's a different system assembling the drives?  The
> kernel is not running the md-device (if there is one?) from
> mdadm-2.6-source, it's running the one inside 2.6.17.

No, it doesn't matter if it is a different system, as long as the
kernel is new enough to understand reshaping (and i think 2.6.17 is
the earliest that understands it properly).

> 
> > Can you post kernel log messages showing md0 being assembled?  Also
> > the output of "mdadm -E" on a couple of drives might be helpful.
> 
> Running assemble:
> # ./mdadm --assemble --scan
> mdadm: failed to RUN_ARRAY /dev/.static/dev/md0: Invalid argument
> 
> Ouput from syslog:
> Feb 20 20:20:09 ubuntu kernel: [17197083.928000] md: md0 stopped.
> Feb 20 20:20:09 ubuntu kernel: [17197084.012000] md: bind<sdd1>
> Feb 20 20:20:09 ubuntu kernel: [17197084.012000] md: bind<sde1>
> Feb 20 20:20:09 ubuntu kernel: [17197084.012000] md: bind<sdf1>
> Feb 20 20:20:09 ubuntu kernel: [17197084.012000] md: bind<sdg1>
> Feb 20 20:20:09 ubuntu kernel: [17197084.016000] md: bind<sdc1>

Is that all....
Could it be that the kernel you are now using was not compiled with
  CONFIG_MD_RAID5_RESHAPE=y
that would explain an error with no message.  In fact it is the only
thing that would explain the messages you got.  So please try with a
kernel that has RAID5_RESHAPE enabled.

Thanks,
NeilBrown

Re: mdadm 2.6, linux 2.6.17, failed grow/reshape

[Thomas Langås]

On 2/20/07, Neil Brown <neilb@suse.de> wrote:
> Is that all....
> Could it be that the kernel you are now using was not compiled with
>   CONFIG_MD_RAID5_RESHAPE=y
> that would explain an error with no message.  In fact it is the only
> thing that would explain the messages you got.  So please try with a
> kernel that has RAID5_RESHAPE enabled.

Which is correct, it seems, when looking at the config for the kernel:
# CONFIG_MD_RAID5_RESHAPE is not set

I need to figure out how to recompile a kernel and make my own bootcd
(the machine is in a state which only accepts boot cd now).

-- 
Thomas

Re: mdadm 2.6, linux 2.6.17, failed grow/reshape

[Thomas Langås]

On 2/20/07, Neil Brown <neilb@suse.de> wrote:
> Is that all....
> Could it be that the kernel you are now using was not compiled with
>   CONFIG_MD_RAID5_RESHAPE=y
> that would explain an error with no message.  In fact it is the only
> thing that would explain the messages you got.  So please try with a
> kernel that has RAID5_RESHAPE enabled.

Ok, booted with Knoppix-CD which had RAID5_RESHAPE, did --examine --scan,
added that line to mdadm.conf and ran --assemble --scan.  It's currently
continuing reshaping.

Thanks a lot for helping out!   Great piece of software :)

-- 
Thomas