Help with assembling a stopped array

Linux RAID subsystem development
 help / color / mirror / Atom feed

* Help with assembling a stopped array
@ 2016-07-13  1:23 Vegard Haugland
  2016-07-13 15:29 ` Phil Turmel
  0 siblings, 1 reply; 6+ messages in thread
From: Vegard Haugland @ 2016-07-13  1:23 UTC (permalink / raw)
  To: linux-raid

Hi all,

I have a raid6 setup (called /dev/md4) with 10 harddrives, ranged from
sda to sdj.

One of my drives fell out recently, /dev/sde3/ (not entirely sure how
this happened), but I readded it as /dev/sdk3, and the array started
rebuilding. This went on for several hours, and it hit 60% completed
around 8 hours ago.

However, a few hours after that, I lost both /dev/sdg3 and /dev/sdh3 -
and the array stopped completely. My OS was also running on that
array, so my entire system freezed forcing me to reboot.

I've now booted into busybox and md tells me that it cannot reassemble
the array as it can only find 7 out of 10 good drives..

I then read the instructions on
https://raid.wiki.kernel.org/index.php/RAID_Recovery, I started using
mdadm --examine and found that 8 out of 10 drives has identical event
counters. The two that deviates from this is /dev/sdg3 and /dev/sdh3.
I'm not entirely sure if the disk symbolic names are still the same,
so I further assume that the 8/10 drives that are still good are the
ones that were running OK before any of this started happening.

As such, I try forcing a reassemble using mdadm -A /dev/md4 --force.
When I now run mdadm -D /md4, it lists one device as spare (even
though my array does not use any spares) - and I guess this is the
main reason the array doesn't start. Can anyone confirm this, or am i
missing something?

Also, if I try setting the spare as failed by running mdadm -f
/dev/md4 /dev/sde3, I get "no such device" (as recalled from human
memory). Removing it works (mdadm -r /dev/md4 /dev/sde3), but every
time I reassemble, it gets the "spare" tag. I also tried reassemble
with force, but the same happened..

The initrd on this computer does not have networking unfortunately, so
I cannot attach any output from any logs.

If you agree with my summary so far, what should be my next action? Is
there anything left to try before running mdadm create (with level 6
and 2 missing drives)?

Best regards,
Vegard Haugland

-- 
Vegard Haugland <vegard@haugland.at>
Mob. +47 976 68 262
Web: www.v3gard.com | foto.haugland.at

If you want to send me encrypted email, my public GPG key is located
at http://v3gard.com/files/gpg.pub

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Help with assembling a stopped array
  2016-07-13  1:23 Help with assembling a stopped array Vegard Haugland
@ 2016-07-13 15:29 ` Phil Turmel
  2016-07-13 19:11   ` Vegard Haugland
  0 siblings, 1 reply; 6+ messages in thread
From: Phil Turmel @ 2016-07-13 15:29 UTC (permalink / raw)
  To: Vegard Haugland, linux-raid

Hello Vegard,

On 07/12/2016 09:23 PM, Vegard Haugland wrote:
> Hi all,
> 
> I have a raid6 setup (called /dev/md4) with 10 harddrives, ranged from
> sda to sdj.

You'll have to supply more detail.  Uncut mdadm -E /dev/sdXY for each
member device in its current state.  smartctl -iA -l scterc /dev/sdX for
each member device's drive.

> One of my drives fell out recently, /dev/sde3/ (not entirely sure how
> this happened), but I readded it as /dev/sdk3, and the array started
> rebuilding. This went on for several hours, and it hit 60% completed
> around 8 hours ago.
> 
> However, a few hours after that, I lost both /dev/sdg3 and /dev/sdh3 -
> and the array stopped completely. My OS was also running on that
> array, so my entire system freezed forcing me to reboot.

You would not believe how often we encounter reports like yours where
more member devices fail while trying to rebuild/resync/re-add after a
first failure.  There's some reading assignments for you at the at end
of this mail that you *must* read and understand or this array will blow
up again.

> I've now booted into busybox and md tells me that it cannot reassemble
> the array as it can only find 7 out of 10 good drives..
> 
> I then read the instructions on
> https://raid.wiki.kernel.org/index.php/RAID_Recovery, I started using
> mdadm --examine and found that 8 out of 10 drives has identical event
> counters. The two that deviates from this is /dev/sdg3 and /dev/sdh3.
> I'm not entirely sure if the disk symbolic names are still the same,
> so I further assume that the 8/10 drives that are still good are the
> ones that were running OK before any of this started happening.
> 
> As such, I try forcing a reassemble using mdadm -A /dev/md4 --force.
> When I now run mdadm -D /md4, it lists one device as spare (even
> though my array does not use any spares) - and I guess this is the
> main reason the array doesn't start. Can anyone confirm this, or am i
> missing something?

When you used --assemble --force, did you spell out exactly which
devices to use?

> Also, if I try setting the spare as failed by running mdadm -f
> /dev/md4 /dev/sde3, I get "no such device" (as recalled from human
> memory). Removing it works (mdadm -r /dev/md4 /dev/sde3), but every
> time I reassemble, it gets the "spare" tag. I also tried reassemble
> with force, but the same happened..
> 
> The initrd on this computer does not have networking unfortunately, so
> I cannot attach any output from any logs.

Boot from a modern rescueCD or USB drive that lets you save the
requested information and output on the console.  Include any dmesg
content that involves these devices.  When running mdadm, include the -v
option so your console has more info for us.

> If you agree with my summary so far, what should be my next action? Is
> there anything left to try before running mdadm create (with level 6
> and 2 missing drives)?

Absolutely do not use --create!

mdadm --assemble --force is the correct tool, but you *must* resolve the
underlying reason your devices are failing.

Phil

Readings for timeout mismatch issues:  (whole threads if possible)

http://marc.info/?l=linux-raid&m=139050322510249&w=2
http://marc.info/?l=linux-raid&m=135863964624202&w=2
http://marc.info/?l=linux-raid&m=135811522817345&w=1
http://marc.info/?l=linux-raid&m=133761065622164&w=2
http://marc.info/?l=linux-raid&m=132477199207506
http://marc.info/?l=linux-raid&m=133665797115876&w=2
http://marc.info/?l=linux-raid&m=142487508806844&w=3
http://marc.info/?l=linux-raid&m=144535576302583&w=2


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Help with assembling a stopped array
  2016-07-13 15:29 ` Phil Turmel
@ 2016-07-13 19:11   ` Vegard Haugland
  2016-07-13 19:57     ` Phil Turmel
  0 siblings, 1 reply; 6+ messages in thread
From: Vegard Haugland @ 2016-07-13 19:11 UTC (permalink / raw)
  To: Phil Turmel; +Cc: linux-raid

Hello Phil.

Thanks for taking your time to answer my post. I really appreciate it!

> You'll have to supply more detail.  Uncut mdadm -E /dev/sdXY for each
> member device in its current state.  smartctl -iA -l scterc /dev/sdX for
> each member device's drive.

Sure. I wrote that post at 03:23 AM local time somewhat panicked, so I
didn't think of booting a live distro to easily get the logs you
require.

Output from $ for x in /dev/sd?3; do mdadm --examine $x >> output_examine; done;

http://paste.debian.net/780968/

Output from $ for x in /dev/sd?3; do echo $x >> output_smartctl;
smartctl -iA -l scterc $x >> output_smartctl; done;

http://paste.debian.net/780967/

Note that since I booted into a live CD, the device names do no longer
match what I wrote in my original post.

> You would not believe how often we encounter reports like yours where
> more member devices fail while trying to rebuild/resync/re-add after a
> first failure.  There's some reading assignments for you at the at end
> of this mail that you *must* read and understand or this array will blow
> up again.

Of the links you posted, only the last one appeared relevant to my
problem. If your intention was that I read these to increase my
knowledge of md in general to avoid issues like mine from happening in
the future, I'm happy to oblige. There were a lot of technical terms
and details in those threads that I don't yet fully grasp, so please
bear with me while I take my time to fully digest this. Do you
recommend I read these in the suggested order?

>
> When you used --assemble --force, did you spell out exactly which
> devices to use?
>

Yes. I spelled out all the good drives and omitted the faulty ones.
The result of this was that one of the drives showed up as spare.
Also, the event counter on all drives specified incremented by one
except the spare one. The current event counters are, as shown while
booting the live CD (again, note that the device names differ from my
original post):

/dev/sda3         Events : 32680547
/dev/sdb3         Events : 32680549
/dev/sdc3         Events : 32679348
/dev/sdd3         Events : 32677935
/dev/sde3         Events : 32680549
/dev/sdg3         Events : 32680549
/dev/sdh3         Events : 32680549
/dev/sdi3         Events : 32680549
/dev/sdj3         Events : 32680549
/dev/sdk3         Events : 32680549

The "spare" (which isn't a spare) is /dev/sda3. I've attempted forcing
assembly twice with no prevail. As such, the event counter has
incremented by 2 on the good drives compared to /dev/sda. The faulty
drives are /dev/sdc3 and /dev/sdd3 as shown by the event counter.

> Boot from a modern rescueCD or USB drive that lets you save the
> requested information and output on the console.  Include any dmesg
> content that involves these devices.  When running mdadm, include the -v
> option so your console has more info for us.

Noted. Will do that from now on.

> Absolutely do not use --create!

Noted!

Again, thanks for helping out.

Vegard

-- 
Vegard Haugland <vegard@haugland.at>
Mob. +47 976 68 262
Web: www.v3gard.com | foto.haugland.at

If you want to send me encrypted email, my public GPG key is located
at http://v3gard.com/files/gpg.pub

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Help with assembling a stopped array
  2016-07-13 19:11   ` Vegard Haugland
@ 2016-07-13 19:57     ` Phil Turmel
  2016-07-15 17:02       ` Vegard Haugland
  0 siblings, 1 reply; 6+ messages in thread
From: Phil Turmel @ 2016-07-13 19:57 UTC (permalink / raw)
  To: Vegard Haugland; +Cc: linux-raid

Hi Vegard,

On 07/13/2016 03:11 PM, Vegard Haugland wrote:
> Hello Phil.
> 
> Thanks for taking your time to answer my post. I really appreciate it!
> 
>> You'll have to supply more detail.  Uncut mdadm -E /dev/sdXY for each
>> member device in its current state.  smartctl -iA -l scterc /dev/sdX for
>> each member device's drive.
> 
> Sure. I wrote that post at 03:23 AM local time somewhat panicked, so I
> didn't think of booting a live distro to easily get the logs you
> require.
> 
> Output from $ for x in /dev/sd?3; do mdadm --examine $x >> output_examine; done;
> 
> http://paste.debian.net/780968/

The device that's popping up as a spare every time really does have the
spare marker (role #10).  That's the drive that was partially re-added
when the array crashed again.  Just leave it out of your next attempts.

> Output from $ for x in /dev/sd?3; do echo $x >> output_smartctl;
> smartctl -iA -l scterc $x >> output_smartctl; done;
> 
> http://paste.debian.net/780967/

The key is "SCT Error Recovery Control command not supported" on most of
your drives.  In fact, upgrading model 7200.11 1T drives to your 2T
Barracudas is precisely the set of models that first bit me in the ass.
I mention that somewhere in the threads I pointed at.

The key is that you have desktop drives in a raid array, and they don't
handle read errors in a way friendly to the linux kernel -- they take
too long.  The one 7200.11 drive can be told to timeout quicker (7.0
seconds is typical for spinning disks).  The kernel will have to be told
to wait extra long for all of the others.  2-3 minutes.  Details in the
reading assignments.  The precise sequence of events that breaks MD raid
is described in the sixth:

http://marc.info/?l=linux-raid&m=133665797115876&w=2

> Note that since I booted into a live CD, the device names do no longer
> match what I wrote in my original post.

Yes, device names are not guaranteed to remain constant in linux.  MD
stores metadata in a superblock that includes array role (position) and
layout details so the name is superfluous once set up.  Getting the
roles wrong with changed device names is a key reason --create is so
dangerous.

After you get your array assembled again, I suggest you run the lsdrv[1]
script to document which drive serial numbers correspond to which array
roles.  Also consider using the --update=metadata at some point (not
now) to get away from the v0.90 metadata.  It is unreliable when used on
partitions that extend to the end of their parent device.

>> You would not believe how often we encounter reports like yours where
>> more member devices fail while trying to rebuild/resync/re-add after a
>> first failure.  There's some reading assignments for you at the at end
>> of this mail that you *must* read and understand or this array will blow
>> up again.
> 
> Of the links you posted, only the last one appeared relevant to my
> problem. If your intention was that I read these to increase my
> knowledge of md in general to avoid issues like mine from happening in
> the future, I'm happy to oblige. There were a lot of technical terms
> and details in those threads that I don't yet fully grasp, so please
> bear with me while I take my time to fully digest this. Do you
> recommend I read these in the suggested order?

Yes.  Your problem is "timeout mismatch", which is use of drives with
extended error recovery times in any linux software raid array. (I
understand hardware raid also struggles with this, but I don't know the
details.)

For now, use at every boot:

for x in /sys/block/*/device/timeout ; do echo 180 > $x ; done

You can stop doing that when you've replaced all of your desktop drives.

> Yes. I spelled out all the good drives and omitted the faulty ones.

This time, omit the drive that shows up as "spare".  Use all nine
others.  You really want nine, so the redundancy in your array can
reconstruct when it hits the UREs you obviously have.  See "Current
Pending Sector" != 0 in your smartctl reports.

After it assembles the nine, issue "mdadm --run /dev/md4" if it didn't
start.  Then "echo check >>/sys/block/md4/md/sync_action".

Wait for that to finish.  Then add the spare back to the array.

Also, in the future, paste drive reports and console output inline in
your mails, with word wrapping disabled.  That puts the details in the
archives for future googlers to find.  (This list allows ~ 100k per mail.)

Phil

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Help with assembling a stopped array
  2016-07-13 19:57     ` Phil Turmel
@ 2016-07-15 17:02       ` Vegard Haugland
  2016-07-15 21:44         ` Phil Turmel
  0 siblings, 1 reply; 6+ messages in thread
From: Vegard Haugland @ 2016-07-15 17:02 UTC (permalink / raw)
  To: Phil Turmel; +Cc: linux-raid

Hi Phil

Here's an update to my problem:.

> The device that's popping up as a spare every time really does have the
> spare marker (role #10).  That's the drive that was partially re-added
> when the array crashed again.  Just leave it out of your next attempts.

OK. Done.

> The key is "SCT Error Recovery Control command not supported" on most of
> your drives.  In fact, upgrading model 7200.11 1T drives to your 2T
> Barracudas is precisely the set of models that first bit me in the ass.
> I mention that somewhere in the threads I pointed at.
>
> The key is that you have desktop drives in a raid array, and they don't
> handle read errors in a way friendly to the linux kernel -- they take
> too long.  The one 7200.11 drive can be told to timeout quicker (7.0
> seconds is typical for spinning disks).  The kernel will have to be told
> to wait extra long for all of the others.  2-3 minutes.  Details in the
> reading assignments.  The precise sequence of events that breaks MD raid
> is described in the sixth:
>
> http://marc.info/?l=linux-raid&m=133665797115876&w=2

Thanks for the explanation. I'll look into this more to get a better
comprehension of this.

>
> After you get your array assembled again, I suggest you run the lsdrv[1]
> script to document which drive serial numbers correspond to which array
> roles.  Also consider using the --update=metadata at some point (not
> now) to get away from the v0.90 metadata.  It is unreliable when used on
> partitions that extend to the end of their parent device.
>

Adding this to the todo list. Thanks.

> Your problem is "timeout mismatch", which is use of drives with
> extended error recovery times in any linux software raid array. (I
> understand hardware raid also struggles with this, but I don't know the
> details.)
>
> For now, use at every boot:
>
> for x in /sys/block/*/device/timeout ; do echo 180 > $x ; done

Done.

> This time, omit the drive that shows up as "spare".  Use all nine
> others.  You really want nine, so the redundancy in your array can
> reconstruct when it hits the UREs you obviously have.  See "Current
> Pending Sector" != 0 in your smartctl reports.
>
> After it assembles the nine, issue "mdadm --run /dev/md4" if it didn't
> start.  Then "echo check >>/sys/block/md4/md/sync_action".
>
> Wait for that to finish.  Then add the spare back to the array.

OK. Here's whats been happening for the past two days. I can this
command to rebuild the array

# mdadm -A /dev/md4 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3 /dev/sde3
/dev/sdg3 /dev/sdh3 /dev/sdi3 /dev/sdj3 --force
mdadm: forcing event count in /dev/sdc3(4) from 32679348 upto 32680549
mdadm: forcing event count in /dev/sdd3(3) from 32677935 upto 32680549
mdadm: clearing FAULTY flag for device 3 in /dev/md4 for /dev/sdd3
mdadm: clearing FAULTY flag for device 2 in /dev/md4 for /dev/sdc3
mdadm: Marking array /dev/md4 as 'clean'
mdadm: /dev/md4 has been started with 8 drives (out of 10) and 1 spare.

As the output mentioned, the array started and I got access to the
data again. Yay! In order to start the rebuild, I ran "echo check
>>/sys/block/md4/md/sync_action".

The array just finished the rebuild, but not with the results I hoped
for. Here's the output from mdadm -D /dev/md4

# mdadm -D -v /dev/md4
/dev/md4:
        Version : 0.90
  Creation Time : Tue Jun 15 06:37:57 2010
     Raid Level : raid6
     Array Size : 11719545344 (11176.63 GiB 12000.81 GB)
  Used Dev Size : 1464943168 (1397.08 GiB 1500.10 GB)
   Raid Devices : 10
  Total Devices : 9
Preferred Minor : 4
    Persistence : Superblock is persistent

    Update Time : Fri Jul 15 14:36:51 2016
          State : clean, FAILED
 Active Devices : 7
Working Devices : 8
 Failed Devices : 1
  Spare Devices : 1

         Layout : left-symmetric-6
     Chunk Size : 64K

           UUID : c807562b:f8dbc4e5:457de8e9:4807bcad
         Events : 0.32687312

    Number   Major   Minor   RaidDevice State
       0       8       19        0      active sync   /dev/sdb3
       1       8       99        1      active sync   /dev/sdg3
       4       0        0        4      removed
       3       8       51        3      active sync   /dev/sdd3
       8       0        0        8      removed
       5       8      115        5      active sync   /dev/sdh3
       6       8       67        6      active sync   /dev/sde3
       7       8      131        7      active sync   /dev/sdi3
       8       8      147        8      active sync   /dev/sdj3
      18       0        0       18      removed

      10       8        3        -      spare   /dev/sda3
      11       8       35        -      faulty   /dev/sdc3

Here's the output from /proc/mdstat during the rebuild (which just completed).

root@debian:~# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md4 : active raid6 sdb3[0] sda3[10] sdj3[8] sdi3[7] sde3[6] sdh3[5]
sdc3[4] sdd3[3] sdg3[1]
      11719545344 blocks level 6, 64k chunk, algorithm 18 [10/8] [UU_UUUUUU_]
      [=================>...]  recovery = 86.1%
(1262406304/1464943168) finish=178.0min speed=18957K/sec

The current output from /proc/mdstat is the following:

Personalities : [raid1] [raid6] [raid5] [raid4]
md4 : active raid6 sdb3[0] sda3[10](S) sdj3[8] sdi3[7] sde3[6] sdh3[5]
sdc3[11](F) sdd3[3] sdg3[1]
      11719545344 blocks level 6, 64k chunk, algorithm 18 [10/7] [UU_U_UUUU_]

Looks like /dev/sdc is faulty. I ran smartctl on that disk, and the
output is as follows. Note that this error was logged just 4 hours ago
(mentioned in the below log at 24702 hours).

# root@debian:~# smartctl -a /dev/sdc
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-3.16.0-4-amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.14 (AF)
Device Model:     ST1500DM003-9YN16G
Serial Number:    W1E2VMF3
LU WWN Device Id: 5 000c50 05e0ec925
Firmware Version: CC4B
User Capacity:    1,500,301,910,016 bytes [1.50 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Fri Jul 15 18:56:16 2016 UTC

==> WARNING: A firmware update for this drive may be available,
see the following Seagate web pages:
http://knowledge.seagate.com/articles/en_US/FAQ/207931en
http://knowledge.seagate.com/articles/en_US/FAQ/223651en

SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (  600) seconds.
Offline data collection
capabilities: (0x73) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: (   1) minutes.
Extended self-test routine
recommended polling time: ( 185) minutes.
Conveyance self-test routine
recommended polling time: (   2) minutes.
SCT capabilities:       (0x3085) SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE
UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   109   099   006    Pre-fail
Always       -       45495242
  3 Spin_Up_Time            0x0003   096   095   000    Pre-fail
Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age
Always       -       31
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail
Always       -       0
  7 Seek_Error_Rate         0x000f   087   060   030    Pre-fail
Always       -       653305433
  9 Power_On_Hours          0x0032   072   072   000    Old_age
Always       -       24706
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail
Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age
Always       -       28
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age
Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age
Always       -       0
187 Reported_Uncorrect      0x0032   091   091   000    Old_age
Always       -       9
188 Command_Timeout         0x0032   100   099   000    Old_age
Always       -       1 1 2
189 High_Fly_Writes         0x003a   050   050   000    Old_age
Always       -       50
190 Airflow_Temperature_Cel 0x0022   065   052   045    Old_age
Always       -       35 (Min/Max 33/40)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age
Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age
Always       -       19
193 Load_Cycle_Count        0x0032   095   095   000    Old_age
Always       -       10077
194 Temperature_Celsius     0x0022   035   048   000    Old_age
Always       -       35 (0 17 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age
Always       -       80
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age
Offline      -       80
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age
Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age
Offline      -       24656h+37m+41.577s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age
Offline      -       2232552343992
242 Total_LBAs_Read         0x0000   100   253   000    Old_age
Offline      -       144336591164585

SMART Error Log Version: 1
ATA Error Count: 2
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 2 occurred at disk power-on lifetime: 24702 hours (1029 days + 6 hours)
  When the command that caused the error occurred, the device was
active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 a0 ff ff ff ef 00   1d+19:17:33.086  READ DMA EXT
  25 00 18 ff ff ff ef 00   1d+19:17:32.528  READ DMA EXT
  25 00 f8 ff ff ff ef 00   1d+19:17:32.167  READ DMA EXT
  27 00 00 00 00 00 e0 00   1d+19:17:32.166  READ NATIVE MAX ADDRESS
EXT [OBS-ACS-3]
  ec 00 00 00 00 00 a0 00   1d+19:17:32.158  IDENTIFY DEVICE

Error 1 occurred at disk power-on lifetime: 24702 hours (1029 days + 6 hours)
  When the command that caused the error occurred, the device was
active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 f0 ff ff ff ef 00   1d+19:17:29.011  READ DMA EXT
  25 00 18 ff ff ff ef 00   1d+19:17:29.010  READ DMA EXT
  25 00 f8 ff ff ff ef 00   1d+19:17:28.988  READ DMA EXT
  25 00 f8 ff ff ff ef 00   1d+19:17:21.227  READ DMA EXT
  25 00 f8 ff ff ff ef 00   1d+19:17:21.195  READ DMA EXT

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining
LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     24622         -
# 2  Short offline       Completed without error       00%     24598         -
# 3  Short offline       Completed without error       00%     24574         -
# 4  Extended offline    Completed without error       00%     24556         -
# 5  Short offline       Completed without error       00%     24550         -
# 6  Short offline       Completed without error       00%     24526         -
# 7  Short offline       Completed without error       00%     24502         -
# 8  Short offline       Completed without error       00%     24478         -
# 9  Short offline       Completed without error       00%     24454         -
#10  Short offline       Completed without error       00%     24430         -
#11  Short offline       Completed without error       00%     24406         -
#12  Extended offline    Completed without error       00%     24388         -
#13  Short offline       Completed without error       00%     24382         -
#14  Short offline       Completed without error       00%     24358         -
#15  Short offline       Completed without error       00%     24334         -
#16  Short offline       Completed without error       00%     24310         -
#17  Short offline       Completed without error       00%     24286         -
#18  Short offline       Completed without error       00%     24262         -
#19  Short offline       Completed without error       00%     24238         -
#20  Extended offline    Completed without error       00%     24220         -
#21  Short offline       Completed without error       00%     24214         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

----

For some reason, the other faulty disk (not the one mentioned below,
or the one that initially showed up as spare) also shows up as spare
now (like the good one did earlier). Issuing "echo check >>
/sys/block/md4/md/sync_action" does not make any attempts to rebuild
the array. Should I use mdadm --manage --add to re-add it before I
replace the faulty disk with a new one?

>
> Also, in the future, paste drive reports and console output inline in
> your mails, with word wrapping disabled.  That puts the details in the
> archives for future googlers to find.  (This list allows ~ 100k per mail.)
>

OK. I did that this time instead of using pastebins.

Vegard

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Help with assembling a stopped array
  2016-07-15 17:02       ` Vegard Haugland
@ 2016-07-15 21:44         ` Phil Turmel
  0 siblings, 0 replies; 6+ messages in thread
From: Phil Turmel @ 2016-07-15 21:44 UTC (permalink / raw)
  To: Vegard Haugland; +Cc: linux-raid

Hi Vegard,

On 07/15/2016 01:02 PM, Vegard Haugland wrote:
>> This time, omit the drive that shows up as "spare".  Use all nine
>> others.  You really want nine, so the redundancy in your array can
>> reconstruct when it hits the UREs you obviously have.  See "Current
>> Pending Sector" != 0 in your smartctl reports.
>>
>> After it assembles the nine, issue "mdadm --run /dev/md4" if it didn't
>> start.  Then "echo check >>/sys/block/md4/md/sync_action".
>>
>> Wait for that to finish.  Then add the spare back to the array.
> 
> OK. Here's whats been happening for the past two days. I can this
> command to rebuild the array
> 
> # mdadm -A /dev/md4 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3 /dev/sde3
> /dev/sdg3 /dev/sdh3 /dev/sdi3 /dev/sdj3 --force
> mdadm: forcing event count in /dev/sdc3(4) from 32679348 upto 32680549
> mdadm: forcing event count in /dev/sdd3(3) from 32677935 upto 32680549
> mdadm: clearing FAULTY flag for device 3 in /dev/md4 for /dev/sdd3
> mdadm: clearing FAULTY flag for device 2 in /dev/md4 for /dev/sdc3
> mdadm: Marking array /dev/md4 as 'clean'
> mdadm: /dev/md4 has been started with 8 drives (out of 10) and 1 spare.

Well, that sucks.  I expected 9 drives out of 10 and no spare.  You are
somewhat screwed.

> As the output mentioned, the array started and I got access to the
> data again. Yay! In order to start the rebuild, I ran "echo check
>>> /sys/block/md4/md/sync_action".
> 
> The array just finished the rebuild, but not with the results I hoped
> for. Here's the output from mdadm -D /dev/md4

You can't successfully check or rebuild without redundancy if any device
has an Unrecoverable Read Error.  Which you know you have because
Pending is greater than zero on sdc (maybe others, I didn't check).

> For some reason, the other faulty disk (not the one mentioned below,
> or the one that initially showed up as spare) also shows up as spare
> now (like the good one did earlier). Issuing "echo check >>
> /sys/block/md4/md/sync_action" does not make any attempts to rebuild
> the array. Should I use mdadm --manage --add to re-add it before I
> replace the faulty disk with a new one?

It is possible your mdadm is too old to fully --force assembly in this
case, or it is a side effect of old v0.90 metadata.  With v1.x+ and
bitmaps, re-add is very fast and unlikely to hit other UREs.  Too late
for you.

Anyways, you can only use the eight non-spare drives.  Since at least
one has a URE, you will need to use ddrescue to complete copy those
drives onto new drives (or your "spares" with their superblocks erased).
 Then assemble the array (8 members) with the new drives and the non-URE
old drives.  Then add spares one at a time, waiting for rebuild to
finish for each.

If you have very critical data you want to make sure you retrieve,
assemble the eight non-spare drives one more time before making copies,
*don't* do the check, then mount and copy those critical files.  Might
hit a URE anyways, but is the best way to get a quick backup.

Once you start copying devices, be very careful that you keep copies and
roles straight, so that you don't try to assemble with two devices with
the same role number.

You *will* lose some data where the UREs are.  You'll need to fsck to
fix the corruption, if it happens to be in a file or folder.

Phil

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2016-07-15 21:44 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-07-13  1:23 Help with assembling a stopped array Vegard Haugland
2016-07-13 15:29 ` Phil Turmel
2016-07-13 19:11   ` Vegard Haugland
2016-07-13 19:57     ` Phil Turmel
2016-07-15 17:02       ` Vegard Haugland
2016-07-15 21:44         ` Phil Turmel

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox