degraded raid array with bad blocks

degraded raid array with bad blocks

[Fabian Fischer]

[-- Attachment #1: Type: text/plain, Size: 2517 bytes --]

Hi,
today I had some problems with my mdadm raid5 (4disks). Firstly I try to
explaine what happened and what the result is:

One disk in my array has some bad blocks. After some hardware-changes
one of the intact disks was thrown out of the array due to a faulty
sata-cable.
I shut down the server and replaced the cable.
After booting, the removed disk wasn't re added to the array (maybe
because of different event count). --re-add doesn't work.
So I used --add.

Because of the bad blocks on one of the remaining disks, the rebuild
stops when reaching the first bad block. The re added disk is declared
as spare, 2 disks active and the disk with bad blocks as faulty.

/dev/md127:
        Version : 1.2
  Creation Time : Tue Apr 19 08:51:32 2011
     Raid Level : raid5
     Array Size : 5860538880 (5589.05 GiB 6001.19 GB)
  Used Dev Size : 1953512960 (1863.02 GiB 2000.40 GB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Thu Jul 16 19:02:09 2015
          State : clean, FAILED
 Active Devices : 2
Working Devices : 3
 Failed Devices : 1
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 512K

           Name : FiFa-Server:0
           UUID : 839fb405:d0b1f13a:5a55ee42:fc8a2061
         Events : 107223

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8       80        1      active sync   /dev/sdf
       5       8       32        2      active sync   /dev/sdc
       6       0        0        6      removed

       4       8       96        -      faulty   /dev/sdg
       6       8       64        -      spare   /dev/sde


In my opinion there a 3 possibilities to get the array back working. I
am not sure whether both possibilities really exist and which one is the
most promising.
	- Using the 'spare'-disk as active disk. The data on the disk
	  should be still there.
	- Ignoring the bad blocks and loose information stored in this
	  blocks
	- force start the array without the 'spare' disk and copy the
	  data to backup-storage, or does the bad block will cause the
	  array to fail when reaching a bad block?

In the attachment you can find the output of --examine.
In can not explain why 3 disk have a Bad Block Log. According to
smart-values only sdg has Reallocated_Sector_Ct >0
Another thing I can't explain is why sdg (which is the disk with known
bad blocks) has a lower event count.


I hope I can get some great ideas how to fix my array.

Fabian


[-- Attachment #2: examine.txt --]
[-- Type: text/plain, Size: 3741 bytes --]

/dev/sdc:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 839fb405:d0b1f13a:5a55ee42:fc8a2061
           Name : FiFa-Server:0
  Creation Time : Tue Apr 19 08:51:32 2011
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3907027120 (1863.02 GiB 2000.40 GB)
     Array Size : 5860538880 (5589.05 GiB 6001.19 GB)
  Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1960 sectors, after=1200 sectors
          State : clean
    Device UUID : 97ccd551:8820c0e3:4ab3d67d:908a2fd9

    Update Time : Thu Jul 16 19:02:09 2015
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 9d018eea - correct
         Events : 107223

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : .AA. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x8
     Array UUID : 839fb405:d0b1f13a:5a55ee42:fc8a2061
           Name : FiFa-Server:0
  Creation Time : Tue Apr 19 08:51:32 2011
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3907027120 (1863.02 GiB 2000.40 GB)
     Array Size : 5860538880 (5589.05 GiB 6001.19 GB)
  Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1960 sectors, after=1200 sectors
          State : clean
    Device UUID : 6db7566d:3c709370:12634f66:f6bfd4f6

    Update Time : Thu Jul 16 19:02:09 2015
  Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
       Checksum : 4b73aeab - correct
         Events : 107223

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : spare
   Array State : .AA. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 839fb405:d0b1f13a:5a55ee42:fc8a2061
           Name : FiFa-Server:0
  Creation Time : Tue Apr 19 08:51:32 2011
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3907027120 (1863.02 GiB 2000.40 GB)
     Array Size : 5860538880 (5589.05 GiB 6001.19 GB)
  Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=1200 sectors
          State : clean
    Device UUID : 27df429f:c9661838:3a8024e4:d55055e8

    Update Time : Thu Jul 16 19:02:09 2015
       Checksum : b4327aa4 - correct
         Events : 107223

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : .AA. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x8
     Array UUID : 839fb405:d0b1f13a:5a55ee42:fc8a2061
           Name : FiFa-Server:0
  Creation Time : Tue Apr 19 08:51:32 2011
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3907027120 (1863.02 GiB 2000.40 GB)
     Array Size : 5860538880 (5589.05 GiB 6001.19 GB)
  Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1960 sectors, after=1200 sectors
          State : clean
    Device UUID : b6786dff:c6be0236:10d8f95f:63afad93

    Update Time : Thu Jul 16 19:02:03 2015
  Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
       Checksum : 397b22c8 - correct
         Events : 107192

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)

Re: degraded raid array with bad blocks

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 1208 bytes --]

On Thu, 16 Jul 2015 20:14:21 +0200
Fabian Fischer <raid@fabianfischer.org> wrote:

> After booting, the removed disk wasn't re added to the array (maybe
> because of different event count). --re-add doesn't work.
> So I used --add.

As to why --re-add didn't work, I *just* had the same situation, maybe you
needed to do 'mdadm --remove /dev/md127 faulty' first.

> Because of the bad blocks on one of the remaining disks, the rebuild
> stops when reaching the first bad block. The re added disk is declared
> as spare, 2 disks active and the disk with bad blocks as faulty.

One course of action is to use dd_rescue to clone the disk with bad blocks to
a new clean disk (skipping the bad blocks as you go -- you will lose some
data), then assemble the array with the new disk in place of the cloned one and
proceed with trying to rebuild. This  time it will not have bad blocks, but
will have just zeroes at those locations, so rebuild should complete
successfully. After the rebuild completes you should fsck the filesystem and
check file checksums (if you saved them), to figure out where the damage
actually landed, and restore those files from backup.

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: degraded raid array with bad blocks

[NeilBrown]

On Thu, 16 Jul 2015 20:14:21 +0200 Fabian Fischer
<raid@fabianfischer.org> wrote:

> Hi,
> today I had some problems with my mdadm raid5 (4disks). Firstly I try to
> explaine what happened and what the result is:
> 
> One disk in my array has some bad blocks. After some hardware-changes
> one of the intact disks was thrown out of the array due to a faulty
> sata-cable.
> I shut down the server and replaced the cable.
> After booting, the removed disk wasn't re added to the array (maybe
> because of different event count). --re-add doesn't work.
> So I used --add.

You need a bitmap configured for --re-add to be useful.
(It is generally a good idea anyway).

> 
> Because of the bad blocks on one of the remaining disks, the rebuild
> stops when reaching the first bad block. The re added disk is declared
> as spare, 2 disks active and the disk with bad blocks as faulty.

That shouldn't happen.
Your devices all have a bad block log present, so when rebuilding hits
a bad block it should just record that as a bad block on the recovering
disk and keep going.  That is the whole point of the bad block log.

What kernel are you running?  And what version of mdadm?


> 
> /dev/md127:
>         Version : 1.2
>   Creation Time : Tue Apr 19 08:51:32 2011
>      Raid Level : raid5
>      Array Size : 5860538880 (5589.05 GiB 6001.19 GB)
>   Used Dev Size : 1953512960 (1863.02 GiB 2000.40 GB)
>    Raid Devices : 4
>   Total Devices : 4
>     Persistence : Superblock is persistent
> 
>     Update Time : Thu Jul 16 19:02:09 2015
>           State : clean, FAILED
>  Active Devices : 2
> Working Devices : 3
>  Failed Devices : 1
>   Spare Devices : 1
> 
>          Layout : left-symmetric
>      Chunk Size : 512K
> 
>            Name : FiFa-Server:0
>            UUID : 839fb405:d0b1f13a:5a55ee42:fc8a2061
>          Events : 107223
> 
>     Number   Major   Minor   RaidDevice State
>        0       0        0        0      removed
>        1       8       80        1      active sync   /dev/sdf
>        5       8       32        2      active sync   /dev/sdc
>        6       0        0        6      removed
> 
>        4       8       96        -      faulty   /dev/sdg
>        6       8       64        -      spare   /dev/sde
> 
> 
> In my opinion there a 3 possibilities to get the array back working. I
> am not sure whether both possibilities really exist and which one is the
> most promising.
> 	- Using the 'spare'-disk as active disk. The data on the disk
> 	  should be still there.
> 	- Ignoring the bad blocks and loose information stored in this
> 	  blocks
> 	- force start the array without the 'spare' disk and copy the
> 	  data to backup-storage, or does the bad block will cause the
> 	  array to fail when reaching a bad block?

If you have somewhere to store backed up data, and if you can assemble
the array with "--assemble --force", then taking that approach and
copying all the data to somewhere else is the safest option.

To do anything else would require a clear understanding of the history
of the array.  Maybe re-creating the array using the 3 "best" devices
would help, but you would want to be really sure what you were doing.

The data in the record bad blocks is probably lost already anyway -
hopefully there is nothing critical there.

Given the update times on the superblocks are very close the array is
probably quite consistent.

I think "--assemble --force" listing the three devices was "Device
Role: Active device ..." should work and give you a degraded array.
Then 'fsck' that and copy data off.
Then maybe recreate the array from scratch.

NeilBrown



> 
> In the attachment you can find the output of --examine.
> In can not explain why 3 disk have a Bad Block Log. According to
> smart-values only sdg has Reallocated_Sector_Ct >0
> Another thing I can't explain is why sdg (which is the disk with known
> bad blocks) has a lower event count.
> 
> 
> I hope I can get some great ideas how to fix my array.
> 
> Fabian
>