* raid5 messed up
@ 2017-09-01 20:15 Thomas C. Bishop
2017-09-01 22:47 ` Anthony Youngman
` (2 more replies)
0 siblings, 3 replies; 11+ messages in thread
From: Thomas C. Bishop @ 2017-09-01 20:15 UTC (permalink / raw)
To: linux-raid
I messed up my raid5 array . I know a two of HDs are "failure
prediction" and one is out.. seagate is shipping me replacements.
This is my backup server so there's the actual copy of data but I'd
prefer to recover the array because other data scripts/tools have crept
into it ... mostly junk but would like to verify.
Here's out put as recommended at
https://raid.wiki.kernel.org/index.php/Linux_Raid
Thanks in advance for any assistance,
TOm
**************************************
cat-mdstat.txt
Personalities : [raid6] [raid5] [raid4]
md127 : inactive sdd1[1] sdh1[5] sdc1[8] sdg1[9](S) sdb1[0] sdf1[3]
23442105036 blocks super 1.0
unused devices: <none>
**************************************
lsdrv.log
PCI [mpt3sas]
├scsi 0:0:0:0 SEAGATE ST4000NM0023
│└sdb 3.64t [8:16] Empty/Unknown
│ └sdb1 3.64t [8:17] Empty/Unknown
│ └md127 0.00k [9:127] MD v1.0 raid5 (7) inactive, 128k Chunk, None
(None) None {None}
│ Empty/Unknown
├scsi 0:0:1:0 SEAGATE ST4000NM0023
│└sdc 3.64t [8:32] Empty/Unknown
│ └sdc1 3.64t [8:33] Empty/Unknown
│ └md127 0.00k [9:127] MD v1.0 raid5 (7) inactive, 128k Chunk, None
(None) None {None}
│ Empty/Unknown
├scsi 0:0:2:0 SEAGATE ST4000NM0023
│└sdd 3.64t [8:48] Empty/Unknown
│ └sdd1 3.64t [8:49] Empty/Unknown
│ └md127 0.00k [9:127] MD v1.0 raid5 (7) inactive, 128k Chunk, None
(None) None {None}
│ Empty/Unknown
├scsi 0:0:3:0 SEAGATE ST4000NM0023
│└sde 0.00k [8:64] Empty/Unknown
├scsi 0:0:4:0 SEAGATE ST4000NM0023
│└sdf 3.64t [8:80] Empty/Unknown
│ └sdf1 3.64t [8:81] Empty/Unknown
│ └md127 0.00k [9:127] MD v1.0 raid5 (7) inactive, 128k Chunk, None
(None) None {None}
│ Empty/Unknown
├scsi 0:0:5:0 SEAGATE ST4000NM0023
│└sdg 3.64t [8:96] Empty/Unknown
│ └sdg1 3.64t [8:97] Empty/Unknown
│ └md127 0.00k [9:127] MD v1.0 raid5 (7) inactive, 128k Chunk, None
(None) None {None}
│ Empty/Unknown
├scsi 0:0:6:0 SEAGATE ST4000NM0023
│└sdh 3.64t [8:112] Empty/Unknown
│ └sdh1 3.64t [8:113] Empty/Unknown
│ └md127 0.00k [9:127] MD v1.0 raid5 (7) inactive, 128k Chunk, None
(None) None {None}
│ Empty/Unknown
├scsi 0:0:7:0 SEAGATE ST4000NM0023
│└sdi 3.64t [8:128] Empty/Unknown
│ └sdi1 3.64t [8:129] Empty/Unknown
└scsi 0:x:x:x [Empty]
PCI [pata_atiixp]
├scsi 1:x:x:x [Empty]
└scsi 2:x:x:x [Empty]
PCI [ahci]
├scsi 3:0:0:0 PIONEER DVD-RW DVR-219L {KEQC279436WL}
│└sr0 1.00g [11:0] Empty/Unknown
├scsi 4:x:x:x [Empty]
├scsi 5:0:0:0 ATA INTEL SSDSC2CW12
│└sda 111.79g [8:0] Empty/Unknown
│ ├sda1 20.00g [8:1] Empty/Unknown
│ │└Mounted as /dev/sda1 @ /
│ └sda2 91.79g [8:2] Empty/Unknown
│ └Mounted as /dev/sda2 @ /ssd
└scsi 6:x:x:x [Empty]
**************************************
mdadm-detail.txt
/dev/md127:
Version : 1.0
Creation Time : Fri Aug 28 10:59:49 2015
Raid Level : raid5
Used Dev Size : 18446744073709551615
Raid Devices : 7
Total Devices : 6
Persistence : Superblock is persistent
Update Time : Thu Aug 31 00:59:15 2017
State : active, FAILED, Not Started
Active Devices : 5
Working Devices : 6
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 128K
Consistency Policy : unknown
Name : any:raid5
UUID : 2a235c2d:1ac674d3:7fd8bd23:1ff7e37b
Events : 20211
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 49 1 active sync /dev/sdd1
- 0 0 2 removed
3 8 81 3 active sync /dev/sdf1
8 8 33 4 active sync /dev/sdc1
5 8 113 5 active sync /dev/sdh1
- 0 0 6 removed
9 8 97 - spare /dev/sdg1
**************************************
mdadm-examine.txt
/dev/sdb1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x1
Array UUID : 2a235c2d:1ac674d3:7fd8bd23:1ff7e37b
Name : any:raid5
Creation Time : Fri Aug 28 10:59:49 2015
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7814033128 (3726.02 GiB 4000.78 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=472 sectors
State : clean
Device UUID : fdd6f8fc:316b273c:78ae65ed:9b779577
Internal Bitmap : -24 sectors from superblock
Update Time : Thu Aug 31 00:59:15 2017
Bad Block Log : 512 entries available at offset -8 sectors
Checksum : ab3cc85c - correct
Events : 20211
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 0
Array State : AA.AAA. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x1
Array UUID : 2a235c2d:1ac674d3:7fd8bd23:1ff7e37b
Name : any:raid5
Creation Time : Fri Aug 28 10:59:49 2015
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7814036816 (3726.02 GiB 4000.79 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814037080 sectors
Unused Space : before=0 sectors, after=4160 sectors
State : clean
Device UUID : 584fb131:a049ae6c:0ac2150d:d3a66665
Internal Bitmap : -24 sectors from superblock
Update Time : Thu Aug 31 00:59:15 2017
Bad Block Log : 512 entries available at offset -8 sectors
Checksum : 1dfd7ec7 - correct
Events : 20211
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 4
Array State : AA.AAA. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x1
Array UUID : 2a235c2d:1ac674d3:7fd8bd23:1ff7e37b
Name : any:raid5
Creation Time : Fri Aug 28 10:59:49 2015
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7814036816 (3726.02 GiB 4000.79 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814037080 sectors
Unused Space : before=0 sectors, after=4160 sectors
State : clean
Device UUID : e5105c14:b165df48:a1c06442:dfb7b075
Internal Bitmap : -24 sectors from superblock
Update Time : Thu Aug 31 00:59:15 2017
Bad Block Log : 512 entries available at offset -8 sectors
Checksum : 22726c01 - correct
Events : 20211
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 1
Array State : AA.AAA. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x1
Array UUID : 2a235c2d:1ac674d3:7fd8bd23:1ff7e37b
Name : any:raid5
Creation Time : Fri Aug 28 10:59:49 2015
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7814033128 (3726.02 GiB 4000.78 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=472 sectors
State : clean
Device UUID : c0cbff05:4b24998c:4f1d290b:26cc9c6a
Internal Bitmap : -24 sectors from superblock
Update Time : Thu Aug 31 00:59:15 2017
Bad Block Log : 512 entries available at offset -8 sectors
Checksum : 1580399d - correct
Events : 20211
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 3
Array State : AA.AAA. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x9
Array UUID : 2a235c2d:1ac674d3:7fd8bd23:1ff7e37b
Name : any:raid5
Creation Time : Fri Aug 28 10:59:49 2015
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7814033368 (3726.02 GiB 4000.79 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=472 sectors
State : clean
Device UUID : 7896d45b:b7037e5d:e30ea8fc:d3f0503c
Internal Bitmap : -24 sectors from superblock
Update Time : Thu Aug 31 00:59:15 2017
Bad Block Log : 512 entries available at offset -8 sectors - bad
blocks present.
Checksum : ff6cfb00 - correct
Events : 20211
Layout : left-symmetric
Chunk Size : 128K
Device Role : spare
Array State : AA.AAA. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdh1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x1
Array UUID : 2a235c2d:1ac674d3:7fd8bd23:1ff7e37b
Name : any:raid5
Creation Time : Fri Aug 28 10:59:49 2015
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7814036816 (3726.02 GiB 4000.79 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814037080 sectors
Unused Space : before=0 sectors, after=4160 sectors
State : clean
Device UUID : 11314133:cb254486:61591214:7e382352
Internal Bitmap : -24 sectors from superblock
Update Time : Thu Aug 31 00:59:15 2017
Bad Block Log : 512 entries available at offset -8 sectors
Checksum : 2cdc65aa - correct
Events : 20211
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 5
Array State : AA.AAA. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdi1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x9
Array UUID : 2a235c2d:1ac674d3:7fd8bd23:1ff7e37b
Name : any:raid5
Creation Time : Fri Aug 28 10:59:49 2015
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7814036816 (3726.02 GiB 4000.79 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814037080 sectors
Unused Space : before=0 sectors, after=4160 sectors
State : clean
Device UUID : b89e3aad:88e3459e:6a1131a0:2136e318
Internal Bitmap : -24 sectors from superblock
Update Time : Thu Aug 31 00:56:04 2017
Bad Block Log : 512 entries available at offset -8 sectors - bad
blocks present.
Checksum : 11c2a904 - correct
Events : 20187
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 6
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
**************************************
smartctl.log
**** /dev/sdb ***
smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.4.79-19-default] (SUSE RPM)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: SEAGATE
Product: ST4000NM0023
Revision: 0004
Compliance: SPC-4
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Logical block size: 512 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000c5007408192f
Serial number: Z1Z4NYQ40000C45018BF
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Fri Sep 1 14:00:46 2017 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK
**** /dev/sdc ***
smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.4.79-19-default] (SUSE RPM)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: SEAGATE
Product: ST4000NM0023
Revision: 0004
Compliance: SPC-4
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Logical block size: 512 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000c50074219767
Serial number: Z1Z2JCK6000094175KML
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Fri Sep 1 14:00:46 2017 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK
**** /dev/sdd ***
smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.4.79-19-default] (SUSE RPM)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: SEAGATE
Product: ST4000NM0023
Revision: 0004
Compliance: SPC-4
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Logical block size: 512 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000c500560462f3
Serial number: Z1Z0APPP0000931601QK
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Fri Sep 1 14:00:46 2017 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK
**** /dev/sde ***
smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.4.79-19-default] (SUSE RPM)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: SEAGATE
Product: ST4000NM0023
Revision: 0004
Compliance: SPC-4
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000c50056055c0b
Serial number: Z1Z0AXBZ0000931612Q8
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Fri Sep 1 14:00:46 2017 CDT
device is NOT READY (e.g. spun down, busy)
A mandatory SMART command failed: exiting. To continue, add one or more
'-T permissive' options.
**** /dev/sdf ***
smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.4.79-19-default] (SUSE RPM)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: SEAGATE
Product: ST4000NM0023
Revision: 0004
Compliance: SPC-4
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Logical block size: 512 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000c5005d7d83bf
Serial number: Z1Z3754K0000C42685HC
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Fri Sep 1 14:00:46 2017 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK
**** /dev/sdg ***
smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.4.79-19-default] (SUSE RPM)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: SEAGATE
Product: ST4000NM0023
Revision: 0006
Compliance: SPC-4
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Logical block size: 512 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000c5008d70f983
Serial number: Z1Z8L3760000C5407BQU
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Fri Sep 1 14:00:47 2017 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK
**** /dev/sdh ***
smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.4.79-19-default] (SUSE RPM)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: SEAGATE
Product: ST4000NM0023
Revision: 0004
Compliance: SPC-4
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Logical block size: 512 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000c500560559bb
Serial number: Z1Z0AXFK0000931612EY
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Fri Sep 1 14:00:47 2017 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Health Status: FAILURE PREDICTION THRESHOLD EXCEEDED: ascq=0x5
[asc=5d, ascq=5]
**** /dev/sdi ***
smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.4.79-19-default] (SUSE RPM)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: SEAGATE
Product: ST4000NM0023
Revision: 0004
Compliance: SPC-4
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Logical block size: 512 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000c50056055ddb
Serial number: Z1Z0AXA20000931723JE
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Fri Sep 1 14:00:47 2017 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Health Status: FAILURE PREDICTION THRESHOLD EXCEEDED: ascq=0x5
[asc=5d, ascq=5]
gua admin3/RAID5#
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: raid5 messed up
2017-09-01 20:15 raid5 messed up Thomas C. Bishop
@ 2017-09-01 22:47 ` Anthony Youngman
2017-09-02 0:24 ` Andreas Klauer
2017-09-05 3:55 ` Phil Turmel
2 siblings, 0 replies; 11+ messages in thread
From: Anthony Youngman @ 2017-09-01 22:47 UTC (permalink / raw)
To: bishop, linux-raid
On 01/09/17 21:15, Thomas C. Bishop wrote:
> I messed up my raid5 array . I know a two of HDs are "failure
> prediction" and one is out.. seagate is shipping me replacements.
>
> This is my backup server so there's the actual copy of data but I'd
> prefer to recover the array because other data scripts/tools have crept
> into it ... mostly junk but would like to verify.
>
> Here's out put as recommended at
> https://raid.wiki.kernel.org/index.php/Linux_Raid
>
> Thanks in advance for any assistance,
>
> TOm
Okay, one failed drive, so it's not looking bad on that front. sdi1
seems to be the broken one.
Seagates - no mention of the model, or whether SCT/ERC is supported. Are
they Seagate NAS drives? Read the timeout mismatch / why you shouldn't
use desktop drives page on the wiki. Could that be the problem?
It's looking good in that nearly all the event counts are identical.
Seagate are sending a replacement? My immediate reaction is to wait
until it arrives, ddrescue sdi onto it, and then re-assemble the array.
It'll probably reject the new sdi because of the event mismatch, but it
might just work fine. If it does reject it, then you can do a --re-add
which because you've got a bitmap, should bring everything back hunky-dory.
Make sure ddrescue generates a log! If ddrescue can't copy the disk
completely, get back here with the contents of that log and we'll see if
we can mark the failed sectors as "bad" on the copy. That way, you can
safely re-add the drive knowing that a scrub will fall over the bad
sectors and re-create them correctly.
My gut feeling is that this should be a simple recovery, though if
you've actually got a spare drive on the array, you should have gone for
raid-6.
Cheers,
Wol
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: raid5 messed up
2017-09-01 20:15 raid5 messed up Thomas C. Bishop
2017-09-01 22:47 ` Anthony Youngman
@ 2017-09-02 0:24 ` Andreas Klauer
2017-09-06 23:00 ` Thomas C. Bishop
2017-09-05 3:55 ` Phil Turmel
2 siblings, 1 reply; 11+ messages in thread
From: Andreas Klauer @ 2017-09-02 0:24 UTC (permalink / raw)
To: Thomas C. Bishop; +Cc: linux-raid
On Fri, Sep 01, 2017 at 03:15:41PM -0500, Thomas C. Bishop wrote:
> I messed up my raid5 array .
That's an understatement...
> I know a two of HDs are "failure
> prediction" and one is out..
RAID 5 with three failed drives, chances of survival are very low.
You should never let things get this far. Timeouts? Doesn't matter!
You either have no disk monitoring at all or never acted on it.
ddrescue the broken drives to new ones first.
Then always use overlays for recovery experiments.
https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file
Experiments for example could be:
*) --assemble --force
*) --assemble --update=force-no-bbl
*) --create --metadata=1.0 --chunk=128 with one 'missing' drive
Again, use overlays for everything.
> Bad Block Log : 512 entries available at offset -8 sectors - bad
> blocks present.
You have bbl entries on more than one drive, use --examine-badblocks
to see if they are identical. You have to clear those or md will
either not work at all or always give read errors even after replacing
the drives. Bad block list issues were previously discussed on the list,
you might find it when searching for "no-bbl".
> === START OF READ SMART DATA SECTION ===
> SMART Health Status: OK
Never trust this unconditionally. It's a false friend.
Always look at the detailed output with reallocated etc. sectors.
Run selftests regularly, detect disk errors early, replace drives immediately.
Good luck
Andreas Klauer
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: raid5 messed up
2017-09-01 20:15 raid5 messed up Thomas C. Bishop
2017-09-01 22:47 ` Anthony Youngman
2017-09-02 0:24 ` Andreas Klauer
@ 2017-09-05 3:55 ` Phil Turmel
2017-09-06 23:47 ` Thomas C. Bishop
2 siblings, 1 reply; 11+ messages in thread
From: Phil Turmel @ 2017-09-05 3:55 UTC (permalink / raw)
To: bishop; +Cc: linux-raid
On 09/01/2017 04:15 PM, Thomas C. Bishop wrote:
> I messed up my raid5 array . I know a two of HDs are "failure
> prediction" and one is out.. seagate is shipping me replacements.
>
> This is my backup server so there's the actual copy of data but I'd
> prefer to recover the array because other data scripts/tools have crept
> into it ... mostly junk but would like to verify.
>
> Here's out put as recommended at
> https://raid.wiki.kernel.org/index.php/Linux_Raid
>
> Thanks in advance for any assistance,
There's a lot of missing data. lsdrv must have reported not finding
tools that it needs for some of it. Please add them and run it again.
Other stuff seems to have been trimmed. Don't do that. Also, use
"smartctl -iA -l scterc /dev/sdXn" for the smartctl reports.
Please resubmit.
Phil
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: raid5 messed up
2017-09-02 0:24 ` Andreas Klauer
@ 2017-09-06 23:00 ` Thomas C. Bishop
0 siblings, 0 replies; 11+ messages in thread
From: Thomas C. Bishop @ 2017-09-06 23:00 UTC (permalink / raw)
To: Andreas Klauer, Thomas C. Bishop; +Cc: linux-raid
note the smartctl report is
"failure prediction"
on two HDs not failed.
I used the scterc option (see Phil's recommend so the smart is cut-off. I 'll check his full ist of options.
smartctl -iA -l scterc /dev/sdXn
Tom
On 09/01/2017 07:24 PM, Andreas Klauer wrote:
> On Fri, Sep 01, 2017 at 03:15:41PM -0500, Thomas C. Bishop wrote:
>> I messed up my raid5 array .
> That's an understatement...
>
>> I know a two of HDs are "failure
>> prediction" and one is out..
> RAID 5 with three failed drives, chances of survival are very low.
> You should never let things get this far. Timeouts? Doesn't matter!
> You either have no disk monitoring at all or never acted on it.
>
> ddrescue the broken drives to new ones first.
> Then always use overlays for recovery experiments.
>
> https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file
>
> Experiments for example could be:
>
> *) --assemble --force
> *) --assemble --update=force-no-bbl
> *) --create --metadata=1.0 --chunk=128 with one 'missing' drive
>
> Again, use overlays for everything.
>
>> Bad Block Log : 512 entries available at offset -8 sectors - bad
>> blocks present.
> You have bbl entries on more than one drive, use --examine-badblocks
> to see if they are identical. You have to clear those or md will
> either not work at all or always give read errors even after replacing
> the drives. Bad block list issues were previously discussed on the list,
> you might find it when searching for "no-bbl".
>
>> === START OF READ SMART DATA SECTION ===
>> SMART Health Status: OK
> Never trust this unconditionally. It's a false friend.
> Always look at the detailed output with reallocated etc. sectors.
> Run selftests regularly, detect disk errors early, replace drives immediately.
>
> Good luck
> Andreas Klauer
--
***********************************
Thomas C. Bishop
Hazel Stewart Garner Associate Professor
Chemistry & Physics
Tel: 318-257-5209
Fax: 318-257-3823
www.latech.edu/~bishop
***********************************
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: raid5 messed up
2017-09-05 3:55 ` Phil Turmel
@ 2017-09-06 23:47 ` Thomas C. Bishop
2017-09-07 0:17 ` Wols Lists
0 siblings, 1 reply; 11+ messages in thread
From: Thomas C. Bishop @ 2017-09-06 23:47 UTC (permalink / raw)
To: bishop, linux-raid
Thanks ya'll for assistance.
Here's complete report as suggested at
https://raid.wiki.kernel.org/index.php/Asking_for_help#lsdrv
but using Phils recommended smartctl command line
smartctl -iA -l scterc /dev/sdXn
The wiki recommends
--xall is rather verbose - for a shorter report you can use "-H -i -l
scterc" instead
Here's the tcsh script followed by the log
setenv LOG RAID5-info.log
date > $LOG
echo " cat /etc/mdadm.conf " >> $LOG
cat /etc/mdadm.conf >> $LOG
echo "************** " >> $LOG
echo " mdadm --detail /dev/md127 " >> $LOG
mdadm --detail /dev/md127 >> $LOG
echo "************** " >> $LOG
echo " lsdrv " >> $LOG
lsdrv/lsdrv/lsdrv >> $LOG
echo "************** " >> $LOG
echo "smartclt -iA -l scterc /dev/sd[b-z] " >> $LOG
foreach i (/dev/sd[b-z] )
echo " **** $i *** " >> $LOG
smartctl -iA -l scterc $i >> $LOG
end
echo "************** " >> $LOG
echo " mdadm --examine /dev/sd[b-z] " >> $LOG
foreach i (/dev/sd[b-z] )
echo " **** $i *** " >> $LOG
mdadm --examine $i >> $LOG
echo " **** ${i}1 *** " >> $LOG
mdadm --examine ${i}1 >> $LOG
end
echo "************** " >> $LOG
*************************************************************************
Wed Sep 6 18:42:15 CDT 2017
cat /etc/mdadm.conf
DEVICE containers partitions
ARRAY /dev/md/raid5 UUID=2a235c2d:1ac674d3:7fd8bd23:1ff7e37b
**************
mdadm --detail /dev/md127
/dev/md127:
Version : 1.0
Creation Time : Fri Aug 28 10:59:49 2015
Raid Level : raid5
Used Dev Size : 18446744073709551615
Raid Devices : 7
Total Devices : 6
Persistence : Superblock is persistent
Update Time : Thu Aug 31 00:59:15 2017
State : active, FAILED, Not Started
Active Devices : 5
Working Devices : 6
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 128K
Consistency Policy : unknown
Name : any:raid5
UUID : 2a235c2d:1ac674d3:7fd8bd23:1ff7e37b
Events : 20211
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 49 1 active sync /dev/sdd1
- 0 0 2 removed
3 8 81 3 active sync /dev/sdf1
8 8 33 4 active sync /dev/sdc1
5 8 113 5 active sync /dev/sdh1
- 0 0 6 removed
9 8 97 - spare /dev/sdg1
**************
lsdrv
PCI [mpt3sas] 03:00.0 Serial Attached SCSI controller: LSI Logic /
Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
├scsi 0:0:0:0 SEAGATE ST4000NM0023 {Z1Z4NYQ40000C45018BF}
│└sdb 3.64t [8:16] Partitioned (gpt)
│ └sdb1 3.64t [8:17] MD raid5 (0/7) (w/ sdc1,sdd1,sdf1,sdg1,sdh1)
in_sync 'any:raid5' {2a235c2d-1ac6-74d3-7fd8-bd231ff7e37b}
│ └md127 0.00k [9:127] MD v1.0 raid5 (7) inactive, 128k Chunk, None
(None) None {2a235c2d:1ac674d3:7fd8bd23:1ff7e37b}
│ Empty/Unknown
├scsi 0:0:1:0 SEAGATE ST4000NM0023 {Z1Z2JCK6000094175KML}
│└sdc 3.64t [8:32] Partitioned (gpt)
│ └sdc1 3.64t [8:33] MD raid5 (4/7) (w/ sdb1,sdd1,sdf1,sdg1,sdh1)
in_sync 'any:raid5' {2a235c2d-1ac6-74d3-7fd8-bd231ff7e37b}
│ └md127 0.00k [9:127] MD v1.0 raid5 (7) inactive, 128k Chunk, None
(None) None {2a235c2d:1ac674d3:7fd8bd23:1ff7e37b}
│ Empty/Unknown
├scsi 0:0:2:0 SEAGATE ST4000NM0023 {Z1Z0APPP0000931601QK}
│└sdd 3.64t [8:48] Partitioned (gpt)
│ └sdd1 3.64t [8:49] MD raid5 (1/7) (w/ sdb1,sdc1,sdf1,sdg1,sdh1)
in_sync 'any:raid5' {2a235c2d-1ac6-74d3-7fd8-bd231ff7e37b}
│ └md127 0.00k [9:127] MD v1.0 raid5 (7) inactive, 128k Chunk, None
(None) None {2a235c2d:1ac674d3:7fd8bd23:1ff7e37b}
│ Empty/Unknown
├scsi 0:0:3:0 SEAGATE ST4000NM0023 {Z1Z0AXBZ0000931612Q8}
│└sde 0.00k [8:64] Empty/Unknown
├scsi 0:0:4:0 SEAGATE ST4000NM0023 {Z1Z3754K0000C42685HC}
│└sdf 3.64t [8:80] Partitioned (gpt)
│ └sdf1 3.64t [8:81] MD raid5 (3/7) (w/ sdb1,sdc1,sdd1,sdg1,sdh1)
in_sync 'any:raid5' {2a235c2d-1ac6-74d3-7fd8-bd231ff7e37b}
│ └md127 0.00k [9:127] MD v1.0 raid5 (7) inactive, 128k Chunk, None
(None) None {2a235c2d:1ac674d3:7fd8bd23:1ff7e37b}
│ Empty/Unknown
├scsi 0:0:5:0 SEAGATE ST4000NM0023 {Z1Z8L3760000C5407BQU}
│└sdg 3.64t [8:96] Partitioned (gpt)
│ └sdg1 3.64t [8:97] MD raid5 (none/7) (w/ sdb1,sdc1,sdd1,sdf1,sdh1)
spare 'any:raid5' {2a235c2d-1ac6-74d3-7fd8-bd231ff7e37b}
│ └md127 0.00k [9:127] MD v1.0 raid5 (7) inactive, 128k Chunk, None
(None) None {2a235c2d:1ac674d3:7fd8bd23:1ff7e37b}
│ Empty/Unknown
├scsi 0:0:6:0 SEAGATE ST4000NM0023 {Z1Z0AXFK0000931612EY}
│└sdh 3.64t [8:112] Partitioned (gpt)
│ └sdh1 3.64t [8:113] MD raid5 (5/7) (w/ sdb1,sdc1,sdd1,sdf1,sdg1)
in_sync 'any:raid5' {2a235c2d-1ac6-74d3-7fd8-bd231ff7e37b}
│ └md127 0.00k [9:127] MD v1.0 raid5 (7) inactive, 128k Chunk, None
(None) None {2a235c2d:1ac674d3:7fd8bd23:1ff7e37b}
│ Empty/Unknown
├scsi 0:0:7:0 SEAGATE ST4000NM0023 {Z1Z0AXA20000931723JE}
│└sdi 3.64t [8:128] Partitioned (gpt)
└scsi 0:x:x:x [Empty]
PCI [pata_atiixp] 00:14.1 IDE interface: Advanced Micro Devices, Inc.
[AMD/ATI] SB7x0/SB8x0/SB9x0 IDE Controller
├scsi 1:x:x:x [Empty]
└scsi 2:x:x:x [Empty]
PCI [ahci] 00:11.0 SATA controller: Advanced Micro Devices, Inc.
[AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [IDE mode]
├scsi 3:0:0:0 PIONEER DVD-RW DVR-219L {KEQC279436WL}
│└sr0 1.00g [11:0] Empty/Unknown
├scsi 4:x:x:x [Empty]
├scsi 5:0:0:0 ATA INTEL SSDSC2CW12 {CVCV2026046T120BGN}
│└sda 111.79g [8:0] Partitioned (dos)
│ ├sda1 20.00g [8:1] Partitioned (dos)
{d8eb06cd-b610-4595-87d1-690981b490c4}
│ │└Mounted as /dev/sda1 @ /
│ └sda2 91.79g [8:2] ext4 {03e3169d-9333-499a-96d6-c795f1d2ec64}
│ └Mounted as /dev/sda2 @ /ssd
└scsi 6:x:x:x [Empty]
**************
smartclt -iA -l scterc /dev/sd[b-z]
**** /dev/sdb ***
smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.4.79-19-default] (SUSE RPM)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: SEAGATE
Product: ST4000NM0023
Revision: 0004
Compliance: SPC-4
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Logical block size: 512 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000c5007408192f
Serial number: Z1Z4NYQ40000C45018BF
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Wed Sep 6 18:43:57 2017 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
=== START OF READ SMART DATA SECTION ===
Current Drive Temperature: 40 C
Drive Trip Temperature: 60 C
Manufactured in week 18 of year 2015
Specified cycle count over device lifetime: 10000
Accumulated start-stop cycles: 189
Specified load-unload count over device lifetime: 300000
Accumulated load-unload cycles: 923
Elements in grown defect list: 0
Vendor (Seagate) cache information
Blocks sent to initiator = 6011066
Blocks received from initiator = 169713
Blocks read from cache and sent to initiator = 129432
Number of read and write commands whose size <= segment size = 11047
Number of read and write commands whose size > segment size = 0
Vendor (Seagate/Hitachi) factory information
number of hours powered up = 18383.32
number of minutes until next internal SMART test = 27
**** /dev/sdc ***
smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.4.79-19-default] (SUSE RPM)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: SEAGATE
Product: ST4000NM0023
Revision: 0004
Compliance: SPC-4
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Logical block size: 512 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000c50074219767
Serial number: Z1Z2JCK6000094175KML
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Wed Sep 6 18:43:57 2017 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
=== START OF READ SMART DATA SECTION ===
Current Drive Temperature: 42 C
Drive Trip Temperature: 60 C
Manufactured in week 35 of year 2015
Specified cycle count over device lifetime: 10000
Accumulated start-stop cycles: 146
Specified load-unload count over device lifetime: 300000
Accumulated load-unload cycles: 822
Elements in grown defect list: 0
Vendor (Seagate) cache information
Blocks sent to initiator = 6053619
Blocks received from initiator = 156131
Blocks read from cache and sent to initiator = 121766
Number of read and write commands whose size <= segment size = 10997
Number of read and write commands whose size > segment size = 0
Vendor (Seagate/Hitachi) factory information
number of hours powered up = 16761.75
number of minutes until next internal SMART test = 27
**** /dev/sdd ***
smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.4.79-19-default] (SUSE RPM)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: SEAGATE
Product: ST4000NM0023
Revision: 0004
Compliance: SPC-4
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Logical block size: 512 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000c500560462f3
Serial number: Z1Z0APPP0000931601QK
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Wed Sep 6 18:43:58 2017 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
=== START OF READ SMART DATA SECTION ===
Current Drive Temperature: 42 C
Drive Trip Temperature: 60 C
Manufactured in week 07 of year 2013
Specified cycle count over device lifetime: 10000
Accumulated start-stop cycles: 461
Specified load-unload count over device lifetime: 300000
Accumulated load-unload cycles: 34770
Elements in grown defect list: 0
Vendor (Seagate) cache information
Blocks sent to initiator = 6069527
Blocks received from initiator = 159843
Blocks read from cache and sent to initiator = 121997
Number of read and write commands whose size <= segment size = 11035
Number of read and write commands whose size > segment size = 0
Vendor (Seagate/Hitachi) factory information
number of hours powered up = 36609.20
number of minutes until next internal SMART test = 12
**** /dev/sde ***
smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.4.79-19-default] (SUSE RPM)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: SEAGATE
Product: ST4000NM0023
Revision: 0004
Compliance: SPC-4
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000c50056055c0b
Serial number: Z1Z0AXBZ0000931612Q8
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Wed Sep 6 18:43:59 2017 CDT
device is NOT READY (e.g. spun down, busy)
A mandatory SMART command failed: exiting. To continue, add one or more
'-T permissive' options.
**** /dev/sdf ***
smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.4.79-19-default] (SUSE RPM)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: SEAGATE
Product: ST4000NM0023
Revision: 0004
Compliance: SPC-4
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Logical block size: 512 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000c5005d7d83bf
Serial number: Z1Z3754K0000C42685HC
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Wed Sep 6 18:43:59 2017 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
=== START OF READ SMART DATA SECTION ===
Current Drive Temperature: 42 C
Drive Trip Temperature: 60 C
Manufactured in week 41 of year 2014
Specified cycle count over device lifetime: 10000
Accumulated start-stop cycles: 172
Specified load-unload count over device lifetime: 300000
Accumulated load-unload cycles: 976
Elements in grown defect list: 0
Vendor (Seagate) cache information
Blocks sent to initiator = 5924501
Blocks received from initiator = 148739
Blocks read from cache and sent to initiator = 117618
Number of read and write commands whose size <= segment size = 10744
Number of read and write commands whose size > segment size = 0
Vendor (Seagate/Hitachi) factory information
number of hours powered up = 20110.87
number of minutes until next internal SMART test = 27
**** /dev/sdg ***
smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.4.79-19-default] (SUSE RPM)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: SEAGATE
Product: ST4000NM0023
Revision: 0006
Compliance: SPC-4
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Logical block size: 512 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000c5008d70f983
Serial number: Z1Z8L3760000C5407BQU
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Wed Sep 6 18:43:59 2017 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
=== START OF READ SMART DATA SECTION ===
Current Drive Temperature: 41 C
Drive Trip Temperature: 60 C
Manufactured in week 30 of year 2016
Specified cycle count over device lifetime: 10000
Accumulated start-stop cycles: 188
Specified load-unload count over device lifetime: 300000
Accumulated load-unload cycles: 13607
Elements in grown defect list: 0
Vendor (Seagate) cache information
Blocks sent to initiator = 62016
Blocks received from initiator = 0
Blocks read from cache and sent to initiator = 171909
Number of read and write commands whose size <= segment size = 72
Number of read and write commands whose size > segment size = 0
Vendor (Seagate/Hitachi) factory information
number of hours powered up = 7598.68
number of minutes until next internal SMART test = 10
**** /dev/sdh ***
smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.4.79-19-default] (SUSE RPM)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: SEAGATE
Product: ST4000NM0023
Revision: 0004
Compliance: SPC-4
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Logical block size: 512 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000c500560559bb
Serial number: Z1Z0AXFK0000931612EY
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Wed Sep 6 18:44:00 2017 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
=== START OF READ SMART DATA SECTION ===
Current Drive Temperature: 38 C
Drive Trip Temperature: 60 C
Manufactured in week 07 of year 2013
Specified cycle count over device lifetime: 10000
Accumulated start-stop cycles: 383
Specified load-unload count over device lifetime: 300000
Accumulated load-unload cycles: 34000
Elements in grown defect list: 4393
Vendor (Seagate) cache information
Blocks sent to initiator = 5970316
Blocks received from initiator = 175474
Blocks read from cache and sent to initiator = 122528
Number of read and write commands whose size <= segment size = 11025
Number of read and write commands whose size > segment size = 0
Vendor (Seagate/Hitachi) factory information
number of hours powered up = 36470.55
number of minutes until next internal SMART test = 9
**** /dev/sdi ***
smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.4.79-19-default] (SUSE RPM)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: SEAGATE
Product: ST4000NM0023
Revision: 0004
Compliance: SPC-4
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Logical block size: 512 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000c50056055ddb
Serial number: Z1Z0AXA20000931723JE
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Wed Sep 6 18:44:01 2017 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
=== START OF READ SMART DATA SECTION ===
Current Drive Temperature: 36 C
Drive Trip Temperature: 60 C
Manufactured in week 07 of year 2013
Specified cycle count over device lifetime: 10000
Accumulated start-stop cycles: 499
Specified load-unload count over device lifetime: 300000
Accumulated load-unload cycles: 27883
Elements in grown defect list: 1271
Vendor (Seagate) cache information
Blocks sent to initiator = 3324560
Blocks received from initiator = 14542227
Blocks read from cache and sent to initiator = 132487
Number of read and write commands whose size <= segment size = 12734
Number of read and write commands whose size > segment size = 0
Vendor (Seagate/Hitachi) factory information
number of hours powered up = 36476.02
number of minutes until next internal SMART test = 40
**************
mdadm --examine /dev/sd[b-z]
**** /dev/sdb ***
/dev/sdb:
MBR Magic : aa55
Partition[3] : 1 sectors at 1 (type ee)
**** /dev/sdb1 ***
/dev/sdb1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x1
Array UUID : 2a235c2d:1ac674d3:7fd8bd23:1ff7e37b
Name : any:raid5
Creation Time : Fri Aug 28 10:59:49 2015
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7814033128 (3726.02 GiB 4000.78 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=472 sectors
State : clean
Device UUID : fdd6f8fc:316b273c:78ae65ed:9b779577
Internal Bitmap : -24 sectors from superblock
Update Time : Thu Aug 31 00:59:15 2017
Bad Block Log : 512 entries available at offset -8 sectors
Checksum : ab3cc85c - correct
Events : 20211
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 0
Array State : AA.AAA. ('A' == active, '.' == missing, 'R' == replacing)
**** /dev/sdc ***
/dev/sdc:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
**** /dev/sdc1 ***
/dev/sdc1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x1
Array UUID : 2a235c2d:1ac674d3:7fd8bd23:1ff7e37b
Name : any:raid5
Creation Time : Fri Aug 28 10:59:49 2015
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7814036816 (3726.02 GiB 4000.79 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814037080 sectors
Unused Space : before=0 sectors, after=4160 sectors
State : clean
Device UUID : 584fb131:a049ae6c:0ac2150d:d3a66665
Internal Bitmap : -24 sectors from superblock
Update Time : Thu Aug 31 00:59:15 2017
Bad Block Log : 512 entries available at offset -8 sectors
Checksum : 1dfd7ec7 - correct
Events : 20211
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 4
Array State : AA.AAA. ('A' == active, '.' == missing, 'R' == replacing)
**** /dev/sdd ***
/dev/sdd:
MBR Magic : aa55
Partition[3] : 1 sectors at 1 (type ee)
**** /dev/sdd1 ***
/dev/sdd1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x1
Array UUID : 2a235c2d:1ac674d3:7fd8bd23:1ff7e37b
Name : any:raid5
Creation Time : Fri Aug 28 10:59:49 2015
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7814036816 (3726.02 GiB 4000.79 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814037080 sectors
Unused Space : before=0 sectors, after=4160 sectors
State : clean
Device UUID : e5105c14:b165df48:a1c06442:dfb7b075
Internal Bitmap : -24 sectors from superblock
Update Time : Thu Aug 31 00:59:15 2017
Bad Block Log : 512 entries available at offset -8 sectors
Checksum : 22726c01 - correct
Events : 20211
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 1
Array State : AA.AAA. ('A' == active, '.' == missing, 'R' == replacing)
**** /dev/sde ***
**** /dev/sde1 ***
**** /dev/sdf ***
/dev/sdf:
MBR Magic : aa55
Partition[3] : 1 sectors at 1 (type ee)
**** /dev/sdf1 ***
/dev/sdf1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x1
Array UUID : 2a235c2d:1ac674d3:7fd8bd23:1ff7e37b
Name : any:raid5
Creation Time : Fri Aug 28 10:59:49 2015
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7814033128 (3726.02 GiB 4000.78 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=472 sectors
State : clean
Device UUID : c0cbff05:4b24998c:4f1d290b:26cc9c6a
Internal Bitmap : -24 sectors from superblock
Update Time : Thu Aug 31 00:59:15 2017
Bad Block Log : 512 entries available at offset -8 sectors
Checksum : 1580399d - correct
Events : 20211
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 3
Array State : AA.AAA. ('A' == active, '.' == missing, 'R' == replacing)
**** /dev/sdg ***
/dev/sdg:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
**** /dev/sdg1 ***
/dev/sdg1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x9
Array UUID : 2a235c2d:1ac674d3:7fd8bd23:1ff7e37b
Name : any:raid5
Creation Time : Fri Aug 28 10:59:49 2015
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7814033368 (3726.02 GiB 4000.79 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=472 sectors
State : clean
Device UUID : 7896d45b:b7037e5d:e30ea8fc:d3f0503c
Internal Bitmap : -24 sectors from superblock
Update Time : Thu Aug 31 00:59:15 2017
Bad Block Log : 512 entries available at offset -8 sectors - bad
blocks present.
Checksum : ff6cfb00 - correct
Events : 20211
Layout : left-symmetric
Chunk Size : 128K
Device Role : spare
Array State : AA.AAA. ('A' == active, '.' == missing, 'R' == replacing)
**** /dev/sdh ***
/dev/sdh:
MBR Magic : aa55
Partition[3] : 1 sectors at 1 (type ee)
**** /dev/sdh1 ***
/dev/sdh1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x1
Array UUID : 2a235c2d:1ac674d3:7fd8bd23:1ff7e37b
Name : any:raid5
Creation Time : Fri Aug 28 10:59:49 2015
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7814036816 (3726.02 GiB 4000.79 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814037080 sectors
Unused Space : before=0 sectors, after=4160 sectors
State : clean
Device UUID : 11314133:cb254486:61591214:7e382352
Internal Bitmap : -24 sectors from superblock
Update Time : Thu Aug 31 00:59:15 2017
Bad Block Log : 512 entries available at offset -8 sectors
Checksum : 2cdc65aa - correct
Events : 20211
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 5
Array State : AA.AAA. ('A' == active, '.' == missing, 'R' == replacing)
**** /dev/sdi ***
/dev/sdi:
MBR Magic : aa55
Partition[3] : 1 sectors at 1 (type ee)
**** /dev/sdi1 ***
/dev/sdi1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x9
Array UUID : 2a235c2d:1ac674d3:7fd8bd23:1ff7e37b
Name : any:raid5
Creation Time : Fri Aug 28 10:59:49 2015
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7814036816 (3726.02 GiB 4000.79 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814037080 sectors
Unused Space : before=0 sectors, after=4160 sectors
State : clean
Device UUID : b89e3aad:88e3459e:6a1131a0:2136e318
Internal Bitmap : -24 sectors from superblock
Update Time : Thu Aug 31 00:56:04 2017
Bad Block Log : 512 entries available at offset -8 sectors - bad
blocks present.
Checksum : 11c2a904 - correct
Events : 20187
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 6
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
**************
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: raid5 messed up
2017-09-06 23:47 ` Thomas C. Bishop
@ 2017-09-07 0:17 ` Wols Lists
2017-09-07 13:33 ` Thomas C. Bishop
0 siblings, 1 reply; 11+ messages in thread
From: Wols Lists @ 2017-09-07 0:17 UTC (permalink / raw)
To: bishop, linux-raid
On 07/09/17 00:47, Thomas C. Bishop wrote:
> === START OF INFORMATION SECTION ===
> Vendor: SEAGATE
> Product: ST4000NM0023
Can't see any mention of ERC in the smart output, but a web search tells
me this is a Constellation, which I believe does support ERC?
Cheers,
Wol
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: raid5 messed up
2017-09-07 0:17 ` Wols Lists
@ 2017-09-07 13:33 ` Thomas C. Bishop
2017-09-07 15:29 ` Wols Lists
0 siblings, 1 reply; 11+ messages in thread
From: Thomas C. Bishop @ 2017-09-07 13:33 UTC (permalink / raw)
To: Wols Lists, bishop, linux-raid
I see what you mean... seems it would be simple enough to find this
spec. but not clear if supported or not.
Seagate claims this is a "best fit applications" drive for High-Capacity
RAID storage but never lists ERC as feature.
http://www.seagate.com/files/www-content/partners/my%20spp%20dashboard/learn/en-us/docs/storage-solutions-guide-jul-2013-ssg1351-13-1307us.pdf
pg 30 of the brochure.
elsewhere@seagate I read ERC is a subset of the smart control commands
which are supported on this drive so one _might_ think it's supported.
FYI: smartctl --xall doesn't provide an answer either.
closest it comes is
SMART support is: Available - device has SMART capability.
Tom
On 09/06/2017 07:17 PM, Wols Lists wrote:
> On 07/09/17 00:47, Thomas C. Bishop wrote:
>> === START OF INFORMATION SECTION ===
>> Vendor: SEAGATE
>> Product: ST4000NM0023
> Can't see any mention of ERC in the smart output, but a web search tells
> me this is a Constellation, which I believe does support ERC?
>
> Cheers,
> Wol
--
***********************************
Thomas C. Bishop
Hazel Stewart Garner Associate Professor
Chemistry & Physics
Tel: 318-257-5209
Fax: 318-257-3823
www.latech.edu/~bishop
***********************************
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: raid5 messed up
2017-09-07 13:33 ` Thomas C. Bishop
@ 2017-09-07 15:29 ` Wols Lists
2017-09-07 15:40 ` Thomas C. Bishop
0 siblings, 1 reply; 11+ messages in thread
From: Wols Lists @ 2017-09-07 15:29 UTC (permalink / raw)
To: bishop, linux-raid
On 07/09/17 14:33, Thomas C. Bishop wrote:
> I see what you mean... seems it would be simple enough to find this
> spec. but not clear if supported or not.
> Seagate claims this is a "best fit applications" drive for High-Capacity
> RAID storage but never lists ERC as feature.
> http://www.seagate.com/files/www-content/partners/my%20spp%20dashboard/learn/en-us/docs/storage-solutions-guide-jul-2013-ssg1351-13-1307us.pdf
>
> pg 30 of the brochure.
>
> elsewhere@seagate I read ERC is a subset of the smart control commands
> which are supported on this drive so one _might_ think it's supported.
>
> FYI: smartctl --xall doesn't provide an answer either.
> closest it comes is
> SMART support is: Available - device has SMART capability.
My Barracudas explicitly say SMART is available (disabled by default on
power-up :-(, and ERC is not available. Yours mentions neither ERC, nor
the error timeout, so something's weird somewhere ... quite possibly the
drive can do it, but it's badly documented and the smartctl authors
don't know the magic incantation ... :-)
Or, like the Barracudas have a long timeout hard encoded, possibly the
Constellations have a short timeout hard encoded.
Cheers,
Wol
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: raid5 messed up
2017-09-07 15:29 ` Wols Lists
@ 2017-09-07 15:40 ` Thomas C. Bishop
2017-09-08 20:10 ` Weedy
0 siblings, 1 reply; 11+ messages in thread
From: Thomas C. Bishop @ 2017-09-07 15:40 UTC (permalink / raw)
To: Wols Lists, bishop, linux-raid
I have servers configured w/ HW controlled raid and have had virtually
NO problems w/ those. Both my backup machines are SW raid... I've had to
replace multiple drives on the SW configured raid. The drives are either
SAME MODEL or same Seagate drive family in all cases and one server is
actually the same SuperMicro model as one of the desktops.
I had attributed this to just a hotter running environment.. the backup
machines are desktop workstations w/ NVIDIA graphics cards that run
pretty hot, but I'm rethinking this now.
Any chance SW raid is running the HDs harder/hotter than the HW raid?
All machines run 24-7-365 so power cycling is not the issue and the
server room is not necessarily cooler than the office/desktop environment.
Tom
On 09/07/2017 10:29 AM, Wols Lists wrote:
> On 07/09/17 14:33, Thomas C. Bishop wrote:
>> I see what you mean... seems it would be simple enough to find this
>> spec. but not clear if supported or not.
>> Seagate claims this is a "best fit applications" drive for High-Capacity
>> RAID storage but never lists ERC as feature.
>> http://www.seagate.com/files/www-content/partners/my%20spp%20dashboard/learn/en-us/docs/storage-solutions-guide-jul-2013-ssg1351-13-1307us.pdf
>>
>> pg 30 of the brochure.
>>
>> elsewhere@seagate I read ERC is a subset of the smart control commands
>> which are supported on this drive so one _might_ think it's supported.
>>
>> FYI: smartctl --xall doesn't provide an answer either.
>> closest it comes is
>> SMART support is: Available - device has SMART capability.
> My Barracudas explicitly say SMART is available (disabled by default on
> power-up :-(, and ERC is not available. Yours mentions neither ERC, nor
> the error timeout, so something's weird somewhere ... quite possibly the
> drive can do it, but it's badly documented and the smartctl authors
> don't know the magic incantation ... :-)
>
> Or, like the Barracudas have a long timeout hard encoded, possibly the
> Constellations have a short timeout hard encoded.
>
> Cheers,
> Wol
>
--
***********************************
Thomas C. Bishop
Hazel Stewart Garner Associate Professor
Chemistry & Physics
Tel: 318-257-5209
Fax: 318-257-3823
www.latech.edu/~bishop
***********************************
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: raid5 messed up
2017-09-07 15:40 ` Thomas C. Bishop
@ 2017-09-08 20:10 ` Weedy
0 siblings, 0 replies; 11+ messages in thread
From: Weedy @ 2017-09-08 20:10 UTC (permalink / raw)
To: bishop, Wols Lists, linux-raid
On 07/09/17 11:40 AM, Thomas C. Bishop wrote:
> I have servers configured w/ HW controlled raid and have had virtually
> NO problems w/ those. Both my backup machines are SW raid... I've had to
> replace multiple drives on the SW configured raid. The drives are either
> SAME MODEL or same Seagate drive family in all cases and one server is
> actually the same SuperMicro model as one of the desktops.
>
> I had attributed this to just a hotter running environment.. the backup
> machines are desktop workstations w/ NVIDIA graphics cards that run
> pretty hot, but I'm rethinking this now.
>
> Any chance SW raid is running the HDs harder/hotter than the HW raid?
> All machines run 24-7-365 so power cycling is not the issue and the
> server room is not necessarily cooler than the office/desktop environment.
>
> Tom
I would argue software raid is going to run your drives harder then a
battery backed raid card.
The cards DRAM buffer will probably shift a large majority of writes to
full stripe writes. Vs. if you do anything with files smaller then
stripe basically EVERYTHING is going to be a read-modify-write on md raid5.
All that said, is it going to be enough of a workload delta to see
lifetime differences? That's going to depend on your workload. I have
quite an old array and my drives seem to not care so... YMMV.
# for drive in sda sdb sdc sdd sde sdf sdg sdh; do smartctl --all
/dev/$drive|grep Power_On_Hours; done
9 Power_On_Hours 0x0032 027 027 000 Old_age Always
- 64114
9 Power_On_Hours 0x0032 035 035 000 Old_age Always
- 57735
## the raid5 ##
9 Power_On_Hours 0x0032 090 090 000 Old_age Always
- 49785
9 Power_On_Hours 0x0032 022 022 000 Old_age Always
- 57543
9 Power_On_Hours 0x0032 084 084 000 Old_age Always
- 80950
9 Power_On_Hours 0x0032 022 022 000 Old_age Always
- 57364
9 Power_On_Hours 0x0032 098 098 000 Old_age Always
- 1078
9 Power_On_Hours 0x0032 098 098 000 Old_age Always
- 1079
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2017-09-08 20:10 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-09-01 20:15 raid5 messed up Thomas C. Bishop
2017-09-01 22:47 ` Anthony Youngman
2017-09-02 0:24 ` Andreas Klauer
2017-09-06 23:00 ` Thomas C. Bishop
2017-09-05 3:55 ` Phil Turmel
2017-09-06 23:47 ` Thomas C. Bishop
2017-09-07 0:17 ` Wols Lists
2017-09-07 13:33 ` Thomas C. Bishop
2017-09-07 15:29 ` Wols Lists
2017-09-07 15:40 ` Thomas C. Bishop
2017-09-08 20:10 ` Weedy
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).