Re: change from raid 5 to raid 6 interrupted

Re: change from raid 5 to raid 6 interrupted

[Raphael Müller]

[-- Attachment #1: Type: text/plain, Size: 645 bytes --]

Hi

last week i tried to grow my raid 5 with 4 disks to a raid 6 with 5
disks. accidentialy i dropped the powersource and the resync process
stopped.

i could start the nas with a rescue stick, but i still wasnt' able to
complete the resync. there are some output from different commands
(examine and assemble tries)

i'm very confused, because there stands active sync, but raid 6 and i
think since it was very at the beginning, there was no data changed on
the 4 raid 5 disks.

what should i do? and is it possible to rebuild this array? luckily i
have somewhere around 90% of the data backed up on another nas.

thank you in advance

Raphael

[-- Attachment #2: examine_output.log --]
[-- Type: text/x-log, Size: 5675 bytes --]

/dev/sda2:
          Magic : a92b4efc
        Version : 0.91.00
           UUID : aeda8268:b9ac1fcc:244c7d09:ca70a1d5
  Creation Time : Wed Mar  9 17:01:18 2011
     Raid Level : raid6
  Used Dev Size : 1953263488 (1862.78 GiB 2000.14 GB)
     Array Size : 5859790464 (5588.33 GiB 6000.43 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 1

  Reshape pos'n : 8134656 (7.76 GiB 8.33 GB)
     New Layout : left-symmetric

    Update Time : Fri Apr 15 19:57:05 2011
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 1
  Spare Devices : 0
       Checksum : 10f62459 - correct
         Events : 3481

         Layout : left-symmetric-6
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     0       8       18        0      active sync   /dev/sdb2

   0     0       8       18        0      active sync   /dev/sdb2
   1     1       8       34        1      active sync   /dev/sdc2
   2     2       8       50        2      active sync   /dev/sdd2
   3     3       8       66        3      active sync   /dev/sde2
   4     4       0        0        4      active sync
/dev/sdb2:
          Magic : a92b4efc
        Version : 0.91.00
           UUID : aeda8268:b9ac1fcc:244c7d09:ca70a1d5
  Creation Time : Wed Mar  9 17:01:18 2011
     Raid Level : raid6
  Used Dev Size : 1953263488 (1862.78 GiB 2000.14 GB)
     Array Size : 5859790464 (5588.33 GiB 6000.43 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 1

  Reshape pos'n : 8134656 (7.76 GiB 8.33 GB)
     New Layout : left-symmetric

    Update Time : Fri Apr 15 19:57:05 2011
          State : active
 Active Devices : 4
Working Devices : 4
 Failed Devices : 1
  Spare Devices : 0
       Checksum : 10f6246d - correct
         Events : 3481

         Layout : left-symmetric-6
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     1       8       34        1      active sync   /dev/sdc2

   0     0       8       18        0      active sync   /dev/sdb2
   1     1       8       34        1      active sync   /dev/sdc2
   2     2       8       50        2      active sync   /dev/sdd2
   3     3       8       66        3      active sync   /dev/sde2
   4     4       0        0        4      faulty removed
/dev/sdc2:
          Magic : a92b4efc
        Version : 0.91.00
           UUID : aeda8268:b9ac1fcc:244c7d09:ca70a1d5
  Creation Time : Wed Mar  9 17:01:18 2011
     Raid Level : raid6
  Used Dev Size : 1953263488 (1862.78 GiB 2000.14 GB)
     Array Size : 5859790464 (5588.33 GiB 6000.43 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 1

  Reshape pos'n : 8134656 (7.76 GiB 8.33 GB)
     New Layout : left-symmetric

    Update Time : Fri Apr 15 19:57:05 2011
          State : active
 Active Devices : 4
Working Devices : 4
 Failed Devices : 1
  Spare Devices : 0
       Checksum : 10f6247f - correct
         Events : 3481

         Layout : left-symmetric-6
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     2       8       50        2      active sync   /dev/sdd2

   0     0       8       18        0      active sync   /dev/sdb2
   1     1       8       34        1      active sync   /dev/sdc2
   2     2       8       50        2      active sync   /dev/sdd2
   3     3       8       66        3      active sync   /dev/sde2
   4     4       0        0        4      faulty removed
/dev/sdd2:
          Magic : a92b4efc
        Version : 0.91.00
           UUID : aeda8268:b9ac1fcc:244c7d09:ca70a1d5
  Creation Time : Wed Mar  9 17:01:18 2011
     Raid Level : raid6
  Used Dev Size : 1953263488 (1862.78 GiB 2000.14 GB)
     Array Size : 5859790464 (5588.33 GiB 6000.43 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 1

  Reshape pos'n : 8134656 (7.76 GiB 8.33 GB)
     New Layout : left-symmetric

    Update Time : Fri Apr 15 19:57:05 2011
          State : active
 Active Devices : 4
Working Devices : 4
 Failed Devices : 1
  Spare Devices : 0
       Checksum : 10f62491 - correct
         Events : 3481

         Layout : left-symmetric-6
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     3       8       66        3      active sync   /dev/sde2

   0     0       8       18        0      active sync   /dev/sdb2
   1     1       8       34        1      active sync   /dev/sdc2
   2     2       8       50        2      active sync   /dev/sdd2
   3     3       8       66        3      active sync   /dev/sde2
   4     4       0        0        4      faulty removed
/dev/sde2:
          Magic : a92b4efc
        Version : 0.91.00
           UUID : aeda8268:b9ac1fcc:244c7d09:ca70a1d5
  Creation Time : Wed Mar  9 17:01:18 2011
     Raid Level : raid6
  Used Dev Size : 1953263488 (1862.78 GiB 2000.14 GB)
     Array Size : 5859790464 (5588.33 GiB 6000.43 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 1

  Reshape pos'n : 8134656 (7.76 GiB 8.33 GB)
     New Layout : left-symmetric

    Update Time : Fri Apr 15 19:56:49 2011
          State : active
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 10f6242d - correct
         Events : 3480

         Layout : left-symmetric-6
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     4       8        2        4      active   /dev/sda2

   0     0       8       18        0      active sync   /dev/sdb2
   1     1       8       34        1      active sync   /dev/sdc2
   2     2       8       50        2      active sync   /dev/sdd2
   3     3       8       66        3      active sync   /dev/sde2
   4     4       8        2        4      active   /dev/sda2

[-- Attachment #3: assemble_output.log --]
[-- Type: text/x-log, Size: 2098 bytes --]

mdadm: looking for devices for /dev/md1
mdadm: cannot open device /dev/sdg1: Device or resource busy
mdadm: /dev/sdg1 has wrong uuid.
mdadm: cannot open device /dev/sdg: Device or resource busy
mdadm: /dev/sdg has wrong uuid.
mdadm: no recogniseable superblock on /dev/md/0_0
mdadm: /dev/md/0_0 has wrong uuid.
mdadm: cannot open device /dev/sdf1: Device or resource busy
mdadm: /dev/sdf1 has wrong uuid.
mdadm: cannot open device /dev/sdf: Device or resource busy
mdadm: /dev/sdf has wrong uuid.
mdadm: cannot open device /dev/sde1: Device or resource busy
mdadm: /dev/sde1 has wrong uuid.
mdadm: cannot open device /dev/sde: Device or resource busy
mdadm: /dev/sde has wrong uuid.
mdadm: cannot open device /dev/sdd1: Device or resource busy
mdadm: /dev/sdd1 has wrong uuid.
mdadm: cannot open device /dev/sdd: Device or resource busy
mdadm: /dev/sdd has wrong uuid.
mdadm: cannot open device /dev/sdc1: Device or resource busy
mdadm: /dev/sdc1 has wrong uuid.
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: /dev/sdc has wrong uuid.
mdadm: cannot open device /dev/sdb1: Device or resource busy
mdadm: /dev/sdb1 has wrong uuid.
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: /dev/sdb has wrong uuid.
mdadm: cannot open device /dev/sda1: Device or resource busy
mdadm: /dev/sda1 has wrong uuid.
mdadm: cannot open device /dev/sda: Device or resource busy
mdadm: /dev/sda has wrong uuid.
mdadm: cannot open device /dev/loop0: Device or resource busy
mdadm: /dev/loop0 has wrong uuid.
mdadm: /dev/sde2 is identified as a member of /dev/md1, slot 4.
mdadm: /dev/sdd2 is identified as a member of /dev/md1, slot 3.
mdadm: /dev/sdc2 is identified as a member of /dev/md1, slot 2.
mdadm: /dev/sdb2 is identified as a member of /dev/md1, slot 1.
mdadm: /dev/sda2 is identified as a member of /dev/md1, slot 0.
mdadm:/dev/md1 has an active reshape - checking if critical section needs to be restored
mdadm: Failed to find backup of critical section
mdadm: Failed to restore critical section for reshape, sorry.
      Possibly you needed to specify the --backup-file

Re: change from raid 5 to raid 6 interrupted

[John Robinson]

On 22/04/2011 15:49, Raphael Müller wrote:
> last week i tried to grow my raid 5 with 4 disks to a raid 6 with 5
> disks. accidentialy i dropped the powersource and the resync process
> stopped.
>
> i could start the nas with a rescue stick, but i still wasnt' able to
> complete the resync. there are some output from different commands
> (examine and assemble tries)
>
> i'm very confused, because there stands active sync, but raid 6 and i
> think since it was very at the beginning, there was no data changed on
> the 4 raid 5 disks.
>
> what should i do? and is it possible to rebuild this array? luckily i
> have somewhere around 90% of the data backed up on another nas.

The end of the assemble output has the answer:
> mdadm:/dev/md1 has an active reshape - checking if critical section needs to be restored
> mdadm: Failed to find backup of critical section
> mdadm: Failed to restore critical section for reshape, sorry.
>       Possibly you needed to specify the --backup-file

You need to manually re-issue the assemble command, and specify the 
--backup-file you gave when you started the --grow. That should get the 
array assembled and the reshape running again.

Cheers,

John.

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: change from raid 5 to raid 6 interrupted

[John Robinson]

(Please always remember to cc the list when replying; cc readded)

On 22/04/2011 16:09, Raphael Müller wrote:
> Ok i forgot to say, that the backup file is also gone... (my mistake)
>
> is this backup file the only option i got?

Without the backup file you've almost certainly lost some data. I 
suspect there is a way of restarting the array and reshape without the 
backup file, but I don't know what it is - you'll have to wait for a 
proper guru to get back to you with an answer.

Cheers,

John.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: change from raid 5 to raid 6 interrupted

[NeilBrown]

On Fri, 22 Apr 2011 16:23:19 +0100 John Robinson
<john.robinson@anonymous.org.uk> wrote:

> (Please always remember to cc the list when replying; cc readded)
> 
> On 22/04/2011 16:09, Raphael Müller wrote:
> > Ok i forgot to say, that the backup file is also gone... (my mistake)
> >
> > is this backup file the only option i got?
> 
> Without the backup file you've almost certainly lost some data. I 
> suspect there is a way of restarting the array and reshape without the 
> backup file, but I don't know what it is - you'll have to wait for a 
> proper guru to get back to you with an answer.
> 

With mdadm 3.2.1 there is a new flag "--invalid-backup".  It tells mdadm that
you know that the backup file is invalid, but you want it to continue anyway
(and risk having some corrupted data).

You still need to give a backup file, but it can be a newly created empty
file.

So:
  # install mdadm 3.2.1
  > /root/my-backup
  mdadm -A /dev/mdthing --backup-file=/root/my-backup  \
  --invalid-backup /dev/device1 /dev/device2 ....

should get you going again.

NeilBrown
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: change from raid 5 to raid 6 interrupted

[Raphael Müller]

Ok thank you =D now it works again. but at very slow speed (~4000kb/s)
normally this array brings nearly 70Mb/s.

I tried different performance related stuff, like stripe_cache_size,
readaheadbuffer but nothing could bring better performance.

And i noticed that the newly added drive now is listed as spare
rebuilding. is this normal?

thanks

Raphael

2011/4/23 NeilBrown <neilb@suse.de>:
> On Fri, 22 Apr 2011 16:23:19 +0100 John Robinson
> <john.robinson@anonymous.org.uk> wrote:
>
>> (Please always remember to cc the list when replying; cc readded)
>>
>> On 22/04/2011 16:09, Raphael Müller wrote:
>> > Ok i forgot to say, that the backup file is also gone... (my mistake)
>> >
>> > is this backup file the only option i got?
>>
>> Without the backup file you've almost certainly lost some data. I
>> suspect there is a way of restarting the array and reshape without the
>> backup file, but I don't know what it is - you'll have to wait for a
>> proper guru to get back to you with an answer.
>>
>
> With mdadm 3.2.1 there is a new flag "--invalid-backup".  It tells mdadm that
> you know that the backup file is invalid, but you want it to continue anyway
> (and risk having some corrupted data).
>
> You still need to give a backup file, but it can be a newly created empty
> file.
>
> So:
>  # install mdadm 3.2.1
>  > /root/my-backup
>  mdadm -A /dev/mdthing --backup-file=/root/my-backup  \
>  --invalid-backup /dev/device1 /dev/device2 ....
>
> should get you going again.
>
> NeilBrown
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html