(R) in mdstat output, clean && degraded

(R) in mdstat output, clean && degraded

[Jared Mauch]

I’ve been searching high and low the past few days and have been unable to diagnose what this (R) is in my raid1 mdstat output indicates.

It seems something is ‘stuck’ somehow as I’m not sure how the array is both clean and degraded at the same time.

Some insights are welcome.

kernel 4.0.5-300.fc22 (fedora 22)

/proc/mdstat

md127 : active raid1 sdg1[2](R) sdd1[3]
      976630464 blocks super 1.2 [2/1] [U_]
      bitmap: 8/8 pages [32KB], 65536KB chunk


# mdadm -D /dev/md127 ; mdadm -E /dev/sdg1 ; mdadm -E /dev/sdd1
/dev/md127:
        Version : 1.2
  Creation Time : Sat Jan 24 10:22:05 2015
     Raid Level : raid1
     Array Size : 976630464 (931.39 GiB 1000.07 GB)
  Used Dev Size : 976630464 (931.39 GiB 1000.07 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Tue Jun 23 14:53:50 2015
          State : clean, degraded 
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           Name : jail-lnx:ssd-array
           UUID : a6277db4:da27d506:916a2c7a:d144aed6
         Events : 9594760

    Number   Major   Minor   RaidDevice State
       2       8       97        0      active sync   /dev/sdg1
       3       8       49        0      active sync   /dev/sdd1
       2       0        0        2      removed
/dev/sdg1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x11
     Array UUID : a6277db4:da27d506:916a2c7a:d144aed6
           Name : jail-lnx:ssd-array
  Creation Time : Sat Jan 24 10:22:05 2015
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 1953261038 (931.39 GiB 1000.07 GB)
     Array Size : 976630464 (931.39 GiB 1000.07 GB)
  Used Dev Size : 1953260928 (931.39 GiB 1000.07 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=110 sectors
          State : clean
    Device UUID : d5fd7437:1fd04c64:a9327851:b22e8008

Internal Bitmap : 8 sectors from superblock
    Update Time : Tue Jun 23 14:53:50 2015
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : fa563013 - correct
         Events : 9594760


   Device Role : Replacement device 0
   Array State : R. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : a6277db4:da27d506:916a2c7a:d144aed6
           Name : jail-lnx:ssd-array
  Creation Time : Sat Jan 24 10:22:05 2015
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 1953261038 (931.39 GiB 1000.07 GB)
     Array Size : 976630464 (931.39 GiB 1000.07 GB)
  Used Dev Size : 1953260928 (931.39 GiB 1000.07 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=110 sectors
          State : clean
    Device UUID : fa65731c:d8c703be:bbf05cfe:c89740f2

Internal Bitmap : 8 sectors from superblock
    Update Time : Tue Jun 23 14:53:50 2015
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : cfb0b70c - correct
         Events : 9594760


   Device Role : Active device 0
   Array State : R. ('A' == active, '.' == missing, 'R' == replacing)

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: (R) in mdstat output, clean && degraded

[Phil Turmel]

Hi Jared,

On 06/23/2015 02:56 PM, Jared Mauch wrote:
> I’ve been searching high and low the past few days and have been unable to diagnose what this (R) is in my raid1 mdstat output indicates.

Replacement device.

> It seems something is ‘stuck’ somehow as I’m not sure how the array is both clean and degraded at the same time.

Role '0', currently sdd1, is being replaced with sdg1.  Both appear to
be working as expected and are therefore individually 'clean'.  Role '1'
is missing, and therefore the array as a whole is degraded.

Looks to me like you lost a device and should have used --add to put a
new device into service, but you used --replace instead.  --replace is
for use when a functioning device shows signs of failing and is to be
replaced 'on-the-fly'.

[trim /]

>    Device Role : Replacement device 0
>    Array State : R. ('A' == active, '.' == missing, 'R' == replacing)

However, the array doesn't seem to be making progress on the replacement
(didn't even start ?), so this pathological situation might have
triggered a bug in the resync code.

Neil?

Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: (R) in mdstat output, clean && degraded

[Jared Mauch]

> On Jun 23, 2015, at 4:12 PM, Phil Turmel <philip@turmel.org> wrote:
> 
> Hi Jared,
> 
> On 06/23/2015 02:56 PM, Jared Mauch wrote:
>> I’ve been searching high and low the past few days and have been unable to diagnose what this (R) is in my raid1 mdstat output indicates.
> 
> Replacement device.

Yeah, I inferred that from the device role below, but I also wanted 
something that google would index well for others to confirm.

>> It seems something is ‘stuck’ somehow as I’m not sure how the array is both clean and degraded at the same time.
> 
> Role '0', currently sdd1, is being replaced with sdg1.  Both appear to
> be working as expected and are therefore individually 'clean'.  Role '1'
> is missing, and therefore the array as a whole is degraded.
> 
> Looks to me like you lost a device and should have used --add to put a
> new device into service, but you used --replace instead.  --replace is
> for use when a functioning device shows signs of failing and is to be
> replaced 'on-the-fly’.

I’ve tried doing a few things including repair to see what it will do
but when it completes it’s in the same end-state.


> [trim /]
> 
>>   Device Role : Replacement device 0
>>   Array State : R. ('A' == active, '.' == missing, 'R' == replacing)
> 
> However, the array doesn't seem to be making progress on the replacement
> (didn't even start ?), so this pathological situation might have
> triggered a bug in the resync code.

Let me know what debugging/details might be interesting to collect.  

Based on the Events# being in-sync, I am going to assume that the drives
may be operating in a protected-type mode even if the output doesn’t
concur.

I can say this state seems to survive reboots.

- Jared--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: (R) in mdstat output, clean && degraded

[Phil Turmel]

On 06/23/2015 05:21 PM, Jared Mauch wrote:
> Let me know what debugging/details might be interesting to collect.  

A good sequence of events (from bash history perhaps) with corresponding
log messages is always helpful.  Your initial report was pretty good.

> Based on the Events# being in-sync, I am going to assume that the drives
> may be operating in a protected-type mode even if the output doesn’t
> concur.

I think that's unlikely.  In replace, one drive responds for sectors
below the progress mark, and the other drive responds for sectors above it.

> I can say this state seems to survive reboots.

That suggests you give a vanilla (or near-vanilla) kernel a try with one
of the various liveCDs that have them.  I personally find the one from
sysrescuecd.org to be most convenient, but I'm a gentoo guy :-)

Otherwise, consider adding a third drive to see if it will rebuild and
break the deadlock.

I should have to say, but I hope you have a backup :-)

Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: (R) in mdstat output, clean && degraded

[Jared Mauch]

> On Jun 23, 2015, at 6:09 PM, Phil Turmel <philip@turmel.org> wrote:
> 
> On 06/23/2015 05:21 PM, Jared Mauch wrote:
>> Let me know what debugging/details might be interesting to collect.  
> 
> A good sequence of events (from bash history perhaps) with corresponding
> log messages is always helpful.  Your initial report was pretty good.

I looked, this seems to have aged out.  Plus, bash.. meh.. tcsh :)

>> Based on the Events# being in-sync, I am going to assume that the drives
>> may be operating in a protected-type mode even if the output doesn’t
>> concur.
> 
> I think that's unlikely.  In replace, one drive responds for sectors
> below the progress mark, and the other drive responds for sectors above it.

>> I can say this state seems to survive reboots.
> 
> That suggests you give a vanilla (or near-vanilla) kernel a try with one
> of the various liveCDs that have them.  I personally find the one from
> sysrescuecd.org to be most convenient, but I'm a gentoo guy :-)

I used to run SLS back in the day before everyone got all crazy with
the distributions.

> Otherwise, consider adding a third drive to see if it will rebuild and
> break the deadlock.

I’m currently out of physical slots for drives so this poses a challenge,
plus the machine is 250 mi away, but I do have console/IPMI.

> I should have to say, but I hope you have a backup :-)

There is a qcow file on the mirrored SSDs which seems to be ok as I was 
able to SCP it off to another host once I found space to toss a 800gb
file.  I may even be mailing you from said qcow.

it seems others were able to perhaps recover from not quite exact situations
by stopping and forcing a reassemble, but as copying 800gb would take a few
hours again, I’m nervous to perform such an action.  e.g.:
http://ubuntuforums.org/showthread.php?t=1707416

I guess this means when I visit the machine next week I should take the 
large external drive array and sas controller card I have here at home
as a checked bag.

	- Jared--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: (R) in mdstat output, clean && degraded

[NeilBrown]

On Tue, 23 Jun 2015 14:56:09 -0400 Jared Mauch <jared@puck.nether.net>
wrote:

> I’ve been searching high and low the past few days and have been unable to diagnose what this (R) is in my raid1 mdstat output indicates.
> 
> It seems something is ‘stuck’ somehow as I’m not sure how the array is both clean and degraded at the same time.
> 
> Some insights are welcome.
> 
> kernel 4.0.5-300.fc22 (fedora 22)
> 
> /proc/mdstat
> 
> md127 : active raid1 sdg1[2](R) sdd1[3]
>       976630464 blocks super 1.2 [2/1] [U_]
>       bitmap: 8/8 pages [32KB], 65536KB chunk

Hmm.....

It isn't at all clear to me how you could get into this state, but I
think I can describe the state the array is in.

The array is degraded, but the one working device has been "replaced"
almost completely.
The data has all been copied from sdd1 to sdg1, but sdg1 hasn't been
marked 'faulty' yet.  Normally when the 'replace' finishes, the
original gets marked 'faulty' as the new device is being marked
'in-sync'.
Once it is faulty it is removed from the array.

Some how, your replacement device got marked 'in-sync' but the original
didn't get marked 'faulty'.

Currently I believe that all writes are going to both devices, and all
reads are being served by the replacement: sdg1.
You could verify this by looking at io stats (e.g. /proc/diskstats,
though the meanings of the columns aren't obvious...)

You should be able to turn this into a fully functional RAID1 array by:

 mdadm /dev/md127 --fail /dev/sdd1 
 mdadm /dev/md127 --remove /dev/sdd1
 mdadm /dev/md127 --re-add /dev/sdd1

When you fail sdd1, sdg1 will change from being a 'replacement' to being
a regular member.
When you --re-add /dev/sdd1 you benefit from the fact that raid1
doesn't really care which device is in which slot (unlike raid5).
So re-adding something marked for slot 0 into slot 1 is perfectly
acceptable.
As the bitmap is present and uptodate, the recovery will be very fast.

I would recommend doing some basic checks for data consistency after
removing sdd1 and before re-adding it.  I might be wrong about
something and sdg1 might contain complete garbage - it never hurts to
check :-)

NeilBrown



> 
> 
> # mdadm -D /dev/md127 ; mdadm -E /dev/sdg1 ; mdadm -E /dev/sdd1
> /dev/md127:
>         Version : 1.2
>   Creation Time : Sat Jan 24 10:22:05 2015
>      Raid Level : raid1
>      Array Size : 976630464 (931.39 GiB 1000.07 GB)
>   Used Dev Size : 976630464 (931.39 GiB 1000.07 GB)
>    Raid Devices : 2
>   Total Devices : 2
>     Persistence : Superblock is persistent
> 
>   Intent Bitmap : Internal
> 
>     Update Time : Tue Jun 23 14:53:50 2015
>           State : clean, degraded 
>  Active Devices : 2
> Working Devices : 2
>  Failed Devices : 0
>   Spare Devices : 0
> 
>            Name : jail-lnx:ssd-array
>            UUID : a6277db4:da27d506:916a2c7a:d144aed6
>          Events : 9594760
> 
>     Number   Major   Minor   RaidDevice State
>        2       8       97        0      active sync   /dev/sdg1
>        3       8       49        0      active sync   /dev/sdd1
>        2       0        0        2      removed
> /dev/sdg1:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x11
>      Array UUID : a6277db4:da27d506:916a2c7a:d144aed6
>            Name : jail-lnx:ssd-array
>   Creation Time : Sat Jan 24 10:22:05 2015
>      Raid Level : raid1
>    Raid Devices : 2
> 
>  Avail Dev Size : 1953261038 (931.39 GiB 1000.07 GB)
>      Array Size : 976630464 (931.39 GiB 1000.07 GB)
>   Used Dev Size : 1953260928 (931.39 GiB 1000.07 GB)
>     Data Offset : 262144 sectors
>    Super Offset : 8 sectors
>    Unused Space : before=262056 sectors, after=110 sectors
>           State : clean
>     Device UUID : d5fd7437:1fd04c64:a9327851:b22e8008
> 
> Internal Bitmap : 8 sectors from superblock
>     Update Time : Tue Jun 23 14:53:50 2015
>   Bad Block Log : 512 entries available at offset 72 sectors
>        Checksum : fa563013 - correct
>          Events : 9594760
> 
> 
>    Device Role : Replacement device 0
>    Array State : R. ('A' == active, '.' == missing, 'R' == replacing)
> /dev/sdd1:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x1
>      Array UUID : a6277db4:da27d506:916a2c7a:d144aed6
>            Name : jail-lnx:ssd-array
>   Creation Time : Sat Jan 24 10:22:05 2015
>      Raid Level : raid1
>    Raid Devices : 2
> 
>  Avail Dev Size : 1953261038 (931.39 GiB 1000.07 GB)
>      Array Size : 976630464 (931.39 GiB 1000.07 GB)
>   Used Dev Size : 1953260928 (931.39 GiB 1000.07 GB)
>     Data Offset : 262144 sectors
>    Super Offset : 8 sectors
>    Unused Space : before=262056 sectors, after=110 sectors
>           State : clean
>     Device UUID : fa65731c:d8c703be:bbf05cfe:c89740f2
> 
> Internal Bitmap : 8 sectors from superblock
>     Update Time : Tue Jun 23 14:53:50 2015
>   Bad Block Log : 512 entries available at offset 72 sectors
>        Checksum : cfb0b70c - correct
>          Events : 9594760
> 
> 
>    Device Role : Active device 0
>    Array State : R. ('A' == active, '.' == missing, 'R' == replacing)
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: (R) in mdstat output, clean && degraded

[Jared Mauch]

> On Jun 24, 2015, at 6:52 PM, NeilBrown <neilb@suse.com> wrote:
> 
> You should be able to turn this into a fully functional RAID1 array by:
> 
> mdadm /dev/md127 --fail /dev/sdd1 
> mdadm /dev/md127 --remove /dev/sdd1
> mdadm /dev/md127 --re-add /dev/sdd1
> 
> When you fail sdd1, sdg1 will change from being a 'replacement' to being
> a regular member.
> When you --re-add /dev/sdd1 you benefit from the fact that raid1
> doesn't really care which device is in which slot (unlike raid5).
> So re-adding something marked for slot 0 into slot 1 is perfectly
> acceptable.
> As the bitmap is present and uptodate, the recovery will be very fast.
> 
> I would recommend doing some basic checks for data consistency after
> removing sdd1 and before re-adding it.  I might be wrong about
> something and sdg1 might contain complete garbage - it never hurts to
> check :-)

So this actually doesn’t work:

# mdadm /dev/md127 --fail /dev/sdd1
mdadm: set device faulty failed for /dev/sdd1:  Device or resource busy
# mdadm /dev/md127 --fail /dev/sdg1
mdadm: set device faulty failed for /dev/sdg1:  Device or resource busy

so I may have to physically remove sdd1 to recover this it seems, or stop
and reassemble is my guess.

- Jared

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html