Corrupted Cisco NSS4000 raid

Corrupted Cisco NSS4000 raid

[Jan Hejl]

[-- Attachment #1.1: Type: text/plain, Size: 630 bytes --]

Hello everyone,

right after finish rebuilding faulty, but still usable, RAID array on 
Cisco NSS4000 NAS box, array stopped working completly.
I took all 4 drives (1TB each) out and connected to my linux machine to 
take a look what happened. You can find raid-status (mdadm --examine 
/dev/sd[bcde]) attached.

After reading this page 
https://raid.wiki.kernel.org/index.php/RAID_Recovery I'm considering to 
try recreate array, as soon as dd ends with image backup of these 
drives, but before I screw something I would to ask you RAID wizards for 
help. Was anyone solving similiar issue?

Thanks for help
Jan

[-- Attachment #1.2: raid-status --]
[-- Type: text/plain, Size: 3246 bytes --]

/dev/sdb:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : c5e32635:d1ed305c:c7804bbe:d0dd962d
           Name : 0
  Creation Time : Mon Feb 11 16:58:03 2013
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 1953524896 (931.51 GiB 1000.20 GB)
     Array Size : 2930287296 (2794.54 GiB 3000.61 GB)
  Used Dev Size : 1953524864 (931.51 GiB 1000.20 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : bbd560dd:768ade9f:dd15ef88:53d2dd62

    Update Time : Wed Jan 14 17:24:39 2015
       Checksum : 1e255442 - correct
         Events : 1190

         Layout : left-symmetric
     Chunk Size : 32K

   Device Role : Active device 3
   Array State : A..A ('A' == active, '.' == missing)
/dev/sdc:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : c5e32635:d1ed305c:c7804bbe:d0dd962d
           Name : 0
  Creation Time : Mon Feb 11 16:58:03 2013
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 1953524896 (931.51 GiB 1000.20 GB)
     Array Size : 2930287296 (2794.54 GiB 3000.61 GB)
  Used Dev Size : 1953524864 (931.51 GiB 1000.20 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 0c7f1826:b250a6da:fcdeef79:bb3845ad

    Update Time : Wed Jan 14 17:24:39 2015
       Checksum : dd0cf356 - correct
         Events : 1190

         Layout : left-symmetric
     Chunk Size : 32K

   Device Role : spare
   Array State : A..A ('A' == active, '.' == missing)
/dev/sdd:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : c5e32635:d1ed305c:c7804bbe:d0dd962d
           Name : 0
  Creation Time : Mon Feb 11 16:58:03 2013
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 1953524896 (931.51 GiB 1000.20 GB)
     Array Size : 2930287296 (2794.54 GiB 3000.61 GB)
  Used Dev Size : 1953524864 (931.51 GiB 1000.20 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : ba398ff8:329c83c3:4e9a58be:5e3ea83e

    Update Time : Wed Jan 14 17:24:39 2015
       Checksum : 6e2cba76 - correct
         Events : 1190

         Layout : left-symmetric
     Chunk Size : 32K

   Device Role : Active device 0
   Array State : A..A ('A' == active, '.' == missing)
/dev/sde:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : c5e32635:d1ed305c:c7804bbe:d0dd962d
           Name : 0
  Creation Time : Mon Feb 11 16:58:03 2013
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 1953524896 (931.51 GiB 1000.20 GB)
     Array Size : 2930287296 (2794.54 GiB 3000.61 GB)
  Used Dev Size : 1953524864 (931.51 GiB 1000.20 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 358b8c70:0307f5df:5aeabe91:ba88e079

    Update Time : Wed Jan 14 16:57:26 2015
       Checksum : 113d0d6d - correct
         Events : 1186

         Layout : left-symmetric
     Chunk Size : 32K

   Device Role : Active device 1
   Array State : AA.A ('A' == active, '.' == missing)

[-- Attachment #2: Elektronicky podpis S/MIME --]
[-- Type: application/pkcs7-signature, Size: 4559 bytes --]

Re: Corrupted Cisco NSS4000 raid

[Valentijn Sessink]

Hello Jan,

On 17-01-15 16:23, Jan Hejl wrote:
> right after finish rebuilding faulty, but still usable, RAID array on
> Cisco NSS4000 NAS box, array stopped working completly.

I'd go for
mdadm --assemble --force --run /dev/md666 $sdb $sdc $sdd
... because sdb, sdc and sdd seem to have the same number of "events" -
and sde has a lower number. However, sdc is a "spare" according to the
mdadm information, so that may not work, in which case you could try to
assemble sdb, sdd and sde and then later on try to recreate the array
with sdc in it.

sdb..d are variables in my "assemble" command, because I'd NEVER do this
on a live disk. Please see my posting about what I did to "clone" the
disks with dmsetup: http://valentijn.sessink.nl/?p=557 (I'm sure there
are other ways - having a disk image with "dd" would also help).

For dd to fill unreadable sectors of the source disks with zeroes,
you'll need a few options, otherwise dd will stop, or, even worse, just
"ignore" unreadable sectors like this: 01234xxx89  -> becomes 0123489 at
the target disk. What you want is 0123400089 (unreadable, but not
truncated). "conv=sync,noerror" is, IIRC, the right option but please
check this (I didn't)

Using 3 out of 4 disks will not help you work around read errors on the
disks themselves, so you may want to use smartctl first to see what the
disk statuses are. Using /dev/sde in the array could work, but as it has
a lower "events" count, you can expect to lose some data.

I woudn't go for 4 out of 4 disks, because that will start rebuilding
the array as soon as you have it read/write and you don't want to change
data on any one of the disks.

If "assemble" doesn't work, you could use "create". Please note that
different versions of mdadm have different ideas about "Data Offset",
and only the newest mdadm can handle a user defined data offset, with
the "--data-offset" option. So your best bet for now is using
assemble+force, although I have seen situations where mdadm will just
not assemble the array due to one or more disks having different
information than mdadm expects.

I hope this helps. Good luck.

Best regards,

Valentijn