RAID 5 reshape stalled at 77.5% - next steps??

RAID 5 reshape stalled at 77.5% - next steps??

[George Rapp]

Hello linux-raid team. I have a reshape operation that is stuck and
refuses to respond to commands. I'm wondering what my options are to
safely get it moving again.

Background: I added two new partitions to a RAID 5 array, using a
backup-file on a
separate device:

# mdadm --add /dev/md4 /dev/sdb4 /dev/sdd4
mdadm: added /dev/sdb4
mdadm: added /dev/sdd4

# mdadm --grow --raid-devices=10
--backup-file=/home/gwr/c/md4_backup__2017-01-25 /dev/md4
mdadm: Need to backup 32256K of critical section..

# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
[...]
md4 : active raid5 sdd4[13](R) sdb4[12] sdg4[10] sdi4[8] sdl4[9]
sdf4[1] sdj4[7] sdh4[2] sde4[0] sdk4[11]
13454923776 blocks super 1.1 level 5, 512k chunk, algorithm 2 [10/9]
[UUUUUUUUU_]
[>....................] reshape = 0.8% (16715456/1922131968)
finish=965.4min speed=32892K/sec

The reshape proceeded normally until it hit 77.5%, where it has been
stuck for the last couple of days:

# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md4 : active raid5 sdd4[13](R) sdb4[12] sdg4[10](F) sdi4[8] sdl4[9]
sdf4[1] sdj4[7] sdh4[2] sde4[0] sdk4[11]

13454923776 blocks super 1.1 level 5, 512k chunk, algorithm 2 [10/9]
[UUUU_UUUU_]
[===============>.....] reshape = 77.5% (1490403328/1922131968)
finish=2544246.9min speed=2K/sec

The backup file was last accessed at about the time I started the reshape:
-rw-------. 1 root root  33034240 Jan 25 11:52 md4_backup__2017-01-25

I tried to idle the RAID reshape, but the "echo" command just hung:

# cd /sys/block/md4/md
# echo idle > sync_action

I can get some data from the files in this directory, though:

# cat reshape_direction
forwards
# cat reshape_position
26825379840

I tried to pull mdadm data about this array to add to this post, but that
command also hung:

# mdadm --misc --examine /dev/md4

The server CPU load is pegged, with md4_raid5 as the top CPU hog.

What are my safe alternatives here? Can I safely reboot without corrupting
the reshape? How can I get the reshape unstuck?

-- 
George Rapp  (Pataskala, OH) Home: george.rapp -- at -- gmail.com
LinkedIn profile: https://www.linkedin.com/in/georgerapp
Phone: +1 740 936 RAPP (740 936 7277)

Re: RAID 5 reshape stalled at 77.5% - next steps??

[Roman Mamedov]

On Sat, 28 Jan 2017 18:01:30 -0500
George Rapp <george.rapp@gmail.com> wrote:

> The reshape proceeded normally until it hit 77.5%, where it has been
> stuck for the last couple of days:
> 
> # cat /proc/mdstat
> Personalities : [raid1] [raid6] [raid5] [raid4]
> md4 : active raid5 sdd4[13](R) sdb4[12] sdg4[10](F) sdi4[8] sdl4[9]
> sdf4[1] sdj4[7] sdh4[2] sde4[0] sdk4[11]
> 
> 13454923776 blocks super 1.1 level 5, 512k chunk, algorithm 2 [10/9]
> [UUUU_UUUU_]
> [===============>.....] reshape = 77.5% (1490403328/1922131968)
> finish=2544246.9min speed=2K/sec

It shows you have a failed device (sdg4) but you don't mention anything about
that? Post your mdadm --detail /dev/md4, and what do you have in dmesg.

-- 
With respect,
Roman

Re: RAID 5 reshape stalled at 77.5% - next steps??

[George Rapp]

On Sat, Jan 28, 2017 at 6:15 PM, Roman Mamedov <rm@romanrm.net> wrote:
> On Sat, 28 Jan 2017 18:01:30 -0500
> George Rapp <george.rapp@gmail.com> wrote:
>
>> The reshape proceeded normally until it hit 77.5%, where it has been
>> stuck for the last couple of days:
>>
>> # cat /proc/mdstat
>> Personalities : [raid1] [raid6] [raid5] [raid4]
>> md4 : active raid5 sdd4[13](R) sdb4[12] sdg4[10](F) sdi4[8] sdl4[9]
>> sdf4[1] sdj4[7] sdh4[2] sde4[0] sdk4[11]
>>
>> 13454923776 blocks super 1.1 level 5, 512k chunk, algorithm 2 [10/9]
>> [UUUU_UUUU_]
>> [===============>.....] reshape = 77.5% (1490403328/1922131968)
>> finish=2544246.9min speed=2K/sec
>
> It shows you have a failed device (sdg4) but you don't mention anything about
> that? Post your mdadm --detail /dev/md4, and what do you have in dmesg.

Roman -

Good catch. I didn't notice that.

# mdadm --detail /dev/md4
/dev/md4:
        Version : 1.1
  Creation Time : Thu Feb 17 14:54:06 2011
     Raid Level : raid5
     Array Size : 13454923776 (12831.62 GiB 13777.84 GB)
  Used Dev Size : 1922131968 (1833.09 GiB 1968.26 GB)
   Raid Devices : 10
  Total Devices : 10
    Persistence : Superblock is persistent

    Update Time : Thu Jan 26 08:06:56 2017
          State : active, FAILED, reshaping
 Active Devices : 8
Working Devices : 9
 Failed Devices : 1
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 512K

 Reshape Status : 77% complete
  Delta Devices : 2, (8->10)

           Name : localhost.localdomain:4
           UUID : 359d41dc:a2e506e3:5e802a49:a84ef89c
         Events : 3957775

    Number   Major   Minor   RaidDevice State
       0       8       68        0      active sync   /dev/sde4
       1       8       84        1      active sync   /dev/sdf4
       2       8      116        2      active sync   /dev/sdh4
       9       8      180        3      active sync   /dev/sdl4
      10       8      100        4      faulty   /dev/sdg4
      13       8       52        4      spare rebuilding   /dev/sdd4
      11       8      164        5      active sync   /dev/sdk4
       8       8      132        6      active sync   /dev/sdi4
       7       8      148        7      active sync   /dev/sdj4
      12       8       20        8      active sync   /dev/sdb4
      18       0        0       18      removed

Relevant dmesg output:

[128702.154193] md: super_written gets error=-5
[128702.154197] md/raid:md4: Disk failure on sdg4, disabling device.
                md/raid:md4: Operation continuing on 9 devices.
[128702.154205] md: super_written gets error=-5
[128702.254561] mvsas 0000:03:00.0: Phy2 : No sig fis
[128703.151620] md: md4: reshape interrupted.
[128706.343757] sas: sas_form_port: phy2 belongs to port2 already(1)!

Attempting to re-add /dev/sdg4 to the array fails on a busy device:

# mdadm --manage /dev/md4 --re-add /dev/sdg4
mdadm: Cannot open /dev/sdg4: Device or resource busy

To free up /dev/sdg4, I tried to stop the array. Not surprisingly,
this command hung as well:

# mdadm --stop /dev/md4


-- 
George Rapp  (Pataskala, OH) Home: george.rapp -- at -- gmail.com
LinkedIn profile: https://www.linkedin.com/in/georgerapp
Phone: +1 740 936 RAPP (740 936 7277)

Re: RAID 5 reshape stalled at 77.5% - next steps??

[Roman Mamedov]

On Sat, 28 Jan 2017 18:29:32 -0500
George Rapp <george.rapp@gmail.com> wrote:

> Attempting to re-add /dev/sdg4 to the array fails on a busy device:
> 
> # mdadm --manage /dev/md4 --re-add /dev/sdg4
> mdadm: Cannot open /dev/sdg4: Device or resource busy

You need to remove it first

  mdadm --remove /dev/md4 /dev/sdg4

or 

  mdadm --remove /dev/md4 faulty

But honestly I am not sure if simply removing and re-adding will bring your
reshape back to its working order at this point.

Also you should figure out why did it fail in the first place. Check
SMART, check dmesg further back rather than a few lines only. Maybe the disk
needs a replacement, not just a blind re-add.

-- 
With respect,
Roman

Re: RAID 5 reshape stalled at 77.5% - next steps??

[George Rapp]

On Sat, Jan 28, 2017 at 6:33 PM, Roman Mamedov <rm@romanrm.net> wrote:
> On Sat, 28 Jan 2017 18:29:32 -0500
> George Rapp <george.rapp@gmail.com> wrote:
>
>> Attempting to re-add /dev/sdg4 to the array fails on a busy device:
>>
>> # mdadm --manage /dev/md4 --re-add /dev/sdg4
>> mdadm: Cannot open /dev/sdg4: Device or resource busy
>
> You need to remove it first
>
>   mdadm --remove /dev/md4 /dev/sdg4
>
> or
>
>   mdadm --remove /dev/md4 faulty
>
> But honestly I am not sure if simply removing and re-adding will bring your
> reshape back to its working order at this point.
>
> Also you should figure out why did it fail in the first place. Check
> SMART, check dmesg further back rather than a few lines only. Maybe the disk
> needs a replacement, not just a blind re-add.

Perhaps not surprisingly, the --remove command also hung.

/dev/sdg4 apparently suffered an uncorrectable read error. Entire
dmesg output (2172 lines) is at
https://app.box.com/s/7brp7c53a51zw4ez5to0m12oc5hxeq92 for your
reference.

Since none of the mdadm commands will respond, I'm thinking we need to
reboot the machine at this point to do any more diagnostics.

Thanks for your quick replies!

-- 
George Rapp  (Pataskala, OH) Home: george.rapp -- at -- gmail.com
LinkedIn profile: https://www.linkedin.com/in/georgerapp
Phone: +1 740 936 RAPP (740 936 7277)