Update: mdadm 3.1.1 / 2.6.32 - trouble reducing active devices in 13TB RAID6

Update: mdadm 3.1.1 / 2.6.32 - trouble reducing active devices in 13TB RAID6

[Brett King]

Hi All,
Just an update for anyone following this; it's now working as expected
on mdadm v3.1.2 & kernel 2.6.34-rc3.

/dev/md0:
        Version : 0.91
  Creation Time : Fri Apr  4 21:17:09 2008
     Raid Level : raid6
     Array Size : 9767624960 (9315.13 GiB 10002.05 GB)
  Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
   Raid Devices : 12
  Total Devices : 13
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Sun Apr  4 17:14:25 2010
          State : clean, recovering
 Active Devices : 13
Working Devices : 13
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

 Reshape Status : 14% complete
  Delta Devices : -1, (12->11)

           UUID : 251f6c00:b15f5541:4eb0bc47:eeccb517
         Events : 0.6084463

    Number   Major   Minor   RaidDevice State
       0     259    557056        0      active sync   /dev/sdq
       1     259    294912        1      active sync   /dev/sdr
       2     259    819200        2      active sync   /dev/sds
       3     259    786432        3      active sync   /dev/sdd
       4     259    688128        4      active sync   /dev/sdu
       5     259    425984        5      active sync   /dev/sdv
       6     259    851968        6      active sync   /dev/sdl
       7     259    196608        7      active sync   /dev/sdm
       8     259    720896        8      active sync   /dev/sdn
       9     259    458752        9      active sync   /dev/sdo
      10     259        0       10      active sync   /dev/sda
      11     259    524288       11      active sync   /dev/sdb

      12     259    262144       12      active sync   /dev/sdc

md0 : active raid6 sdq[0] sdc[12] sdb[11] sda[10] sdo[9] sdn[8] sdm[7]
sdl[6] sdv[5] sdu[4] sdd[3] sds[2] sdr[1]
      9767624960 blocks super 0.91 level 6, 64k chunk, algorithm 2
[12/12] [UUUUUUUUUUUU]
      [==>..................]  reshape = 14.7% (143969472/976762496)
finish=1871.0min speed=7417K/sec

Cheers,
Brett.

Re: Update: mdadm 3.1.1 / 2.6.32 - trouble reducing active devices in 13TB RAID6

[Brett King]

Another update: things didn't quite go as planned .. the reshape
didn't quite finish - it got to 99.9% and stopped there:

md0 : active raid6 sdq[0] sdc[12] sdb[11] sda[10] sdo[9] sdn[8] sdm[7]
sdl[6] sdv[5] sdu[4] sdd[3] sds[2] sdr[1]
      9767624960 blocks super 0.91 level 6, 64k chunk, algorithm 2
[12/12] [UUUUUUUUUUUU]
      [===================>.]  reshape = 99.9% (976761856/976762496)
finish=0.0min speed=4K/sec

I left it overnight hence the low speed average. A stop and restart
using the backup-file also didn't end well:

array:/tmp/mdadm-3.1.2 # ./mdadm -S /dev/md0
mdadm: stopped /dev/md0
array:/tmp/mdadm-3.1.2 # ./mdadm -A /dev/md0 --backup-file /export/md0-backup
mdadm: Failed to restore critical section for reshape, sorry.

(the backup-file isn't on the array being reshaped).

Looks like the code's almost there but not quite .. there are 640
remaining blocks.

array:/tmp/mdadm-3.1.2 # uname -a
Linux array 2.6.34-rc3-7-default #1 SMP 2010-03-31 17:12:53 +0200
x86_64 x86_64 x86_64 GNU/Linux
array:/tmp/mdadm-3.1.2 # ./mdadm -V
mdadm - v3.1.2 - 10th March 2010
array:/tmp/mdadm-3.1.2 #

Any ideas ?

Cheers,
Brett.

Re: Update: mdadm 3.1.1 / 2.6.32 - trouble reducing active devices in 13TB RAID6

[Michael Evans]

On Mon, Apr 5, 2010 at 4:10 PM, Brett King <king.br@gmail.com> wrote:
> Another update: things didn't quite go as planned .. the reshape
> didn't quite finish - it got to 99.9% and stopped there:
>
> md0 : active raid6 sdq[0] sdc[12] sdb[11] sda[10] sdo[9] sdn[8] sdm[7]
> sdl[6] sdv[5] sdu[4] sdd[3] sds[2] sdr[1]
>      9767624960 blocks super 0.91 level 6, 64k chunk, algorithm 2
> [12/12] [UUUUUUUUUUUU]
>      [===================>.]  reshape = 99.9% (976761856/976762496)
> finish=0.0min speed=4K/sec
>
> I left it overnight hence the low speed average. A stop and restart
> using the backup-file also didn't end well:
>
> array:/tmp/mdadm-3.1.2 # ./mdadm -S /dev/md0
> mdadm: stopped /dev/md0
> array:/tmp/mdadm-3.1.2 # ./mdadm -A /dev/md0 --backup-file /export/md0-backup
> mdadm: Failed to restore critical section for reshape, sorry.
>
> (the backup-file isn't on the array being reshaped).
>
> Looks like the code's almost there but not quite .. there are 640
> remaining blocks.
>
> array:/tmp/mdadm-3.1.2 # uname -a
> Linux array 2.6.34-rc3-7-default #1 SMP 2010-03-31 17:12:53 +0200
> x86_64 x86_64 x86_64 GNU/Linux
> array:/tmp/mdadm-3.1.2 # ./mdadm -V
> mdadm - v3.1.2 - 10th March 2010
> array:/tmp/mdadm-3.1.2 #
>
> Any ideas ?
>
> Cheers,
> Brett.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Anything useful/interesting in the dmesg output?
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Update: mdadm 3.1.1 / 2.6.32 - trouble reducing active devices in 13TB RAID6

[Brett King]

I've actually since lost the dmesg output across a reboot .. trying to
restart the array now with the same backup-file gives the below in
dmesg:

[ 2009.365158] md: md0 stopped.