* What is the correct way of making image copy of a member disk in md array?
From: Ram Ramesh @ 2014-11-09 1:44 UTC (permalink / raw)
To: Linux Raid
I have a situation with one of my disks in /dev/md0. It does not play
well in initial hardware detection after bootup. It works very reliably
once it is detected. So I like to keep its image ready should it
actually fail. If I add a spare to md0, this flaky disk gets kicked out
immediately and I do not want that to happen.
Following is a possible solution
1. Add the new disk as spare (call it sdX).
2. echo want_replacement on the flaky drive (call it sdY)
3. When the copy completes and flaky drive becomes spare repeat the
process in the opposite direction
4. Remove the new disk from md0 when it becomes spare.
Specifically, my questions are
1. Is it possible to -re-add sdX after step 4?
2. Is there a recommended/better solution for making image other than
the one given above? If not,
3. Can I skip steps 3 & 4 and simply re-add the flaky drive after step2
completes ? I mean mdadm --remove sdX and --re-add sdY.
4. If above step works, will there be a problem when I reboot with both
sdX and sdY present?
5. If a reboot is a no-no, is there another method available to keep
the new disk (sdX) plugged in without being part of md0? I mean not
even as a spare.
The reason for all this jumping through the hoops is that I want to keep
only one spare across multiple md devices across multiple machines. I
want it to be ready for the most likely job this spare will be needed
(based on my hunch).
Kernel: Linux xxxxx 3.14-0.bpo.2-amd64 #1 SMP Debian 3.14.13-2~bpo70+1
(2014-07-31) x86_64 GNU/Linux
mdadm: mdadm - v3.2.5 - 18th May 2012
Ramesh
^ permalink raw reply
* Re: Raid5 drive fail during grow and no backup
From: Phil Turmel @ 2014-11-08 3:36 UTC (permalink / raw)
To: P. Gautschi; +Cc: Vince, linux-raid
In-Reply-To: <545CEDFB.6060806@gautschi.net>
On 11/07/2014 11:06 AM, P. Gautschi wrote:
> > This is a problem you haven't solved yet, I think. The raid array
> should have fixed this bad sector for you without kicking the drive out.
> The scenario is common with "green" drives and/or consumer-grade drives
> in general.
> > ...
> > Then you can set up your array to properly correct bad sectors, and
> set your system to look for bad sectors on
> > a regular basis.
>
> What is the behavior of mdadm when a disk reports a read error?
> - reconstruct the data, deliver it to the fs and otherwise ignore it?
> - set the disk to fail?
> - reconstruct the data, rewrite the failed data and continue with any
> action?
> - rewrite the failed data and reread it (bypassing the cache on the HD)?
Option 3. Reconstruct and rewrite.
However, if the device with the bad sector is trying to recover longer
than the linux low level driver's timeout, bad things^TM happen.
Specifically, the driver resets the SATA (or SCSI) connection and
attempts to reconnect. During this brief time, it will not accept
further I/O, so the write back of the reconstructed data fails. Then
the device has experienced a *write* error, so MD fails the drive. This
is the out-of-the-box behavior of consumer-grade drives in raid arrays.
> Do read operation always read the parity too in order to detect problems
> early
> before a sector on a other disks fails?
No.
> Can the behavior be configured in any way? I found no documentation
> regarding this.
The administrator must schedule "check" scrubs of the array to look for
bad sectors, or wait for them to be found naturally. Such scrubs will
also find inconsistent parity and report it. A "repair" scrub can then
fix the broken parity.
I understand that some distros include a cron job for this purpose.
I've always rolled my own.
Phil
^ permalink raw reply
* Re: --no-degraded does not work
From: NeilBrown @ 2014-11-07 23:22 UTC (permalink / raw)
To: P. Gautschi; +Cc: linux-raid
In-Reply-To: <545CF69B.6040307@gautschi.net>
[-- Attachment #1: Type: text/plain, Size: 2225 bytes --]
On Fri, 07 Nov 2014 17:43:07 +0100 "P. Gautschi" <linuxlist@gautschi.net>
wrote:
> As far as I understand the documentation --assemble --no-degraded should not start a degraded array.
> However on my system (kubuntu 14.10)
>
> # mdadm --assemble --no-degraded /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1
> mdadm: /dev/md0 has been started with 4 drives (out of 5).
>
> # mdadm --detail /dev/md0
> /dev/md0:
> Version : 1.2
> Creation Time : Tue Nov 4 15:26:46 2014
> Raid Level : raid5
> Array Size : 599469328 (571.70 GiB 613.86 GB)
> Used Dev Size : 149867332 (142.92 GiB 153.46 GB)
> Raid Devices : 5
> Total Devices : 4
> Persistence : Superblock is persistent
>
> Intent Bitmap : Internal
>
> Update Time : Fri Nov 7 17:22:53 2014
> State : clean, degraded
> Active Devices : 4
> Working Devices : 4
> Failed Devices : 0
> Spare Devices : 0
>
> Layout : left-symmetric
> Chunk Size : 4K
>
> Name : 0
> UUID : c7465b19:c149b2d1:5b4d88ce:8c6ce432
> Events : 642
>
> Number Major Minor RaidDevice State
> 0 0 0 0 removed
> 1 8 33 1 active sync /dev/sdc1
> 2 8 49 2 active sync /dev/sdd1
> 3 8 65 3 active sync /dev/sde1
> 5 8 81 4 active sync /dev/sdf1
>
> the array IS started when removing one disk, stopping it, reconnecting the disk and then assemble the array.
> Is this the supposed behavior?
Yes, that is the correct behaviour, though I admit that it is slightly
unintuitive.
--no-degraded will cause mdadm to refuse to assemble an array which is more
degraded than it was last time it was active.
So if you have an optimal array, stop it, then try to assemble with some
devices missing, then --no-degraded will cause that to fail.
If the array is already degraded, then there doesn't seem much point in
stopping it from assembling.
Do you have a particular goal, or were you just making sure you understood?
Thanks,
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* --no-degraded does not work
From: P. Gautschi @ 2014-11-07 16:43 UTC (permalink / raw)
To: linux-raid
As far as I understand the documentation --assemble --no-degraded should not start a degraded array.
However on my system (kubuntu 14.10)
# mdadm --assemble --no-degraded /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1
mdadm: /dev/md0 has been started with 4 drives (out of 5).
# mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Tue Nov 4 15:26:46 2014
Raid Level : raid5
Array Size : 599469328 (571.70 GiB 613.86 GB)
Used Dev Size : 149867332 (142.92 GiB 153.46 GB)
Raid Devices : 5
Total Devices : 4
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Fri Nov 7 17:22:53 2014
State : clean, degraded
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 4K
Name : 0
UUID : c7465b19:c149b2d1:5b4d88ce:8c6ce432
Events : 642
Number Major Minor RaidDevice State
0 0 0 0 removed
1 8 33 1 active sync /dev/sdc1
2 8 49 2 active sync /dev/sdd1
3 8 65 3 active sync /dev/sde1
5 8 81 4 active sync /dev/sdf1
the array IS started when removing one disk, stopping it, reconnecting the disk and then assemble the array.
Is this the supposed behavior?
Patrick
^ permalink raw reply
* Re: Raid5 drive fail during grow and no backup
From: P. Gautschi @ 2014-11-07 16:07 UTC (permalink / raw)
To: Vince, linux-raid
In-Reply-To: <545CCAEA.6070204@turmel.org>
> Interesting. I use the WD Red drives, too, and recommend them. Your drive's smartctl report is clean as far as wear & tear is concerned. That suggests a hardware problem elsewhere in your system. Bad cable, perhaps, or a failing power supply.
To me, it looks like a power problem: 50 out of 177 power cycles are power losses:
> 12 Power_Cycle_Count -O--CK 100 100 000 - 177
> 192 Power-Off_Retract_Count -O--CK 200 200 000 - 50
I had something like this in my system due to a bad SATA power split cable.
Patrick
^ permalink raw reply
* Re: Raid5 drive fail during grow and no backup
From: P. Gautschi @ 2014-11-07 16:06 UTC (permalink / raw)
To: Phil Turmel; +Cc: Vince, linux-raid
In-Reply-To: <5458FC2A.1050308@turmel.org>
> This is a problem you haven't solved yet, I think. The raid array should have fixed this bad sector for you without kicking the drive out. The scenario is common with "green" drives and/or consumer-grade drives in general.
> ...
> Then you can set up your array to properly correct bad sectors, and set your system to look for bad sectors on
> a regular basis.
What is the behavior of mdadm when a disk reports a read error?
- reconstruct the data, deliver it to the fs and otherwise ignore it?
- set the disk to fail?
- reconstruct the data, rewrite the failed data and continue with any action?
- rewrite the failed data and reread it (bypassing the cache on the HD)?
Do read operation always read the parity too in order to detect problems early
before a sector on a other disks fails?
Can the behavior be configured in any way? I found no documentation regarding this.
Patrick
^ permalink raw reply
* Re: RAID 6 (containing LUKS dm-crypt) recovery help.
From: Peter Grandi @ 2014-11-07 14:22 UTC (permalink / raw)
To: Linux RAID
In-Reply-To: <545CAFA8.2030602@retard.io>
[ ... ]
>> But assuming the above is somewhat reliable there is an
>> "interesting" situation: in "21488638704 blocks" the number
>> 21,488,638,704 is not a whole multiple of 9:
>>
>> $ factor 21488638704
>> 21488638704: 2 2 2 2 3 13 1801 19121
[ ... ]
>> If you went to the lengths to write 'dd' expressions, you
>> might as well have saved the output of '--examine'. Perhaps
>> you did, but if you did not attach that output to your
>> request for help it would be rather "stunning".
[ ... ]
>> Both numbers don't match. They are *slightly* different. In
>> particular it is rather strange that the "Used Dev Size" is
>> different. How is that possible? Have the disks shrunk a
>> little in the meantime? :-)
> Yes, I saved the -E/--examine information, "just in case". :-)
But without sending it with your request for help: rather
"stunning".
> [ ... ] some disks have a 272 offset, while most others have a
> 2048 offset, [ ... ] Did older mdadm builds on 12.04 LTS ever
> use offsets of 272, rather than 2048?
Given what '--examine' reports exactly where this came from does
not matter a lot. What matters is this note (in a well written
page!):
https://raid.wiki.kernel.org/index.php/RAID_Recovery
«Recreating an array
When an array is created, superblocks are written to the drive
and according to the defaults of mdadm, a certain area of the
drive is now considered "data area".
The data areas (that might or might not be correct) are not
written to, *provided* the array is created in degraded mode;
that is with a 'missing' device.
If the wrong superblock version is chosen, wrong data offset
(internal default value which has changed over time in mdadm),
chunk size (also value that has changed over time), then the
data area will not match what was previously on the drives.
The md superblock might have overwritten part of your data.
Use with caution!»
You can try to recreate the MD set superblocks specifying the
right data offset and member size for each member, as given by
the '--examine' outputs. Google knows how...
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Raid5 drive fail during grow and no backup
From: Phil Turmel @ 2014-11-07 13:36 UTC (permalink / raw)
To: Vince, linux-raid
In-Reply-To: <loom.20141106T175806-337@post.gmane.org>
On 11/06/2014 12:12 PM, Vince wrote:
> Hi Phil,
>
>> This is a problem you haven't solved yet, I think. The raid array
>> should have fixed this bad sector for you without kicking the drive out.
>> The scenario is common with "green" drives and/or consumer-grade
>> drives in general.
>
> i investigated some time and now i am a bit confused.
>
> All my 5 WD Red drives have ERC enables (7sec)
> The kernel timeout is set to 30sec (/sys/block/sdb/device/timeout)
> on all devices.
>
> Unfortunately i haven't any backup of the dmesg output, but i can remeber i
> got something like:
> "failed command: READ FPDMA QUEUED status: { DRDY ERR } error: { UNC }".
> This shows up several times until it ends with showing me the sector which
> causes the problem.
>
> My raid is still up, but as you mentioned i would like some kind of self
> repair if a sector is unreadable instead throwing the disk out of the array.
>
> Here is the samrtctl of one drive that fails.
>
> Do you have any idea if i missing some settings etc?
Interesting. I use the WD Red drives, too, and recommend them. Your
drive's smartctl report is clean as far as wear & tear is concerned.
That suggests a hardware problem elsewhere in your system. Bad cable,
perhaps, or a failing power supply.
Beyond that, I can only recommend regular "check" scrubs, with "repair"
scrubs only when mismatches are discovered.
Phil
^ permalink raw reply
* Re: RAID 6 (containing LUKS dm-crypt) recovery help.
From: xar @ 2014-11-07 11:40 UTC (permalink / raw)
To: Peter Grandi, Linux RAID
In-Reply-To: <21596.40429.244605.398712@tree.ty.sabi.co.uk>
On 11/7/2014 5:24 AM, Peter Grandi wrote:
>> [ ... ] The server experienced some sort of hardware event
>> that resulted in a mandatory restart of the server.
> Details would be helpful: because if some problem happens the
> standard advice is "reload from backups". If you want to
> shortcut that to mostly-recovery context matters to figuring out
> how and how safely.
>
>> [ ... ] completed the restart, the array looked like this,
>> "all spares":
>> md6 :
> What happened to the other MD sets on the same server, if any?
> Any damage? Because if those suffered no damage, there is the
> possibility that the disk rack backplane holding the members of
> 'md6' got damaged, or the specific host adapter; and that the MD
> set content is entirely undamaged and the funny stuff being read
> is a transmission problem.
>
>> inactive sdl1[7](S) sdh1[13](S) sdg1[14](S) sdk1[11](S)
>> sdj1[10](S) sdi1[6](S) sdd1[2](S) sdf1[8](S) sdb1[12](S)
>> sde1[3](S) sdc1[15](S) 21488638704 blocks super 1.2
> "Clever" people hide details as possible, and go to such lengths
> as to actually remove vital information as for example what
> literally follows "super 1.2" here. Because actual quotes are
> too "insipid" and paraphrases are more "challenging":
>
>> The mdadm array has the following characteristics: RAID level:
>> 6 Chunk size: 256k Version: 1.2 Number of devices: 11
> How do you know? Is this part of your records or from actual
> output of 'mdadm --examine'?
>
> But assuming the above is somewhat reliable there is an
> "interesting" situation: in "21488638704 blocks" the number
> 21,488,638,704 is not a whole multiple of 9:
>
> $ factor 21488638704
> 21488638704: 2 2 2 2 3 13 1801 19121
>
>> All attempts to assemble the array continued to result in the "all
>> spare" condition (output above). Thinking that the metadata had been
>> corrupted somehow,
> Apparently without ever trying 'mdadm --detail /dev/md6' or
> 'mdadm --examine /dev/sd...' as per:
>
> https://raid.wiki.kernel.org/index.php/RAID_Recovery
>
>> I set out to recreate the array.
> Quite "brave":
>
> https://raid.wiki.kernel.org/index.php/RAID_Recovery
> «Restore array by recreating (after multiple device failure)
> Recreating should be considered a *last* resort, only to be
> used when everything else fails.
> People getting this wrong is one of the primary reasons people
> lose data. It is very commonly used way too early in the fault
> finding process. You have been warned!»
>
>> The following is the dev_number fields from the metadata,
>> before I attempted to recreate the array: for i in /dev/sd?1;
>> do echo -n $i '' ; dd 2> /dev/null if=$i bs=1 count=4
>> skip=4256 | od -D | head -n1; done: I used the following to
>> extract the index position of each device on a device I
>> suspected wasn't corrupted (for the record, they all returned
>> the same data): [ ... ]
> It is very "astute" indeed to use 'dd' instead of 'mdadm
> --examine'. For example it "encourages" people who might want
> to help to spend some extra time checking your offsets, that
> "teaches" them.
>
> [ ... ]
>> Number Major Minor RaidDevice State
>> 12 8 17 0 active sync /dev/sdb1
>> 3 8 65 1 active sync /dev/sde1
>> 2 8 49 2 active sync /dev/sdd1
>> 8 8 81 3 active sync /dev/sdf1
>> 6 8 129 4 active sync /dev/sdi1
>> 7 8 177 5 active sync /dev/sdl1
>> 6 0 0 6 removed
>> 10 8 145 7 active sync /dev/sdj1
>> 11 8 161 8 active sync /dev/sdk1
>> 13 8 113 9 active sync /dev/sdh1
>> 14 8 97 10 active sync /dev/sdg1
>> The dev_numbers and index position information in conjunction
>> with the historic data (directly above) seemed to indicate
>> that the proper recreation order and command would be the
>> following:
>> mdadm --create /dev/md6 --assume-clean --level=6
>> --raid-devices=11 --metadata=1.2 --chunk=256 /dev/sdb1
>> /dev/sde1 /dev/sdd1 /dev/sdf1 /dev/sdi1 /dev/sdl1 /dev/sdc1
>> /dev/sdj1 /dev/sdk1 /dev/sdh1 /dev/sdg1
> The main consequence of the above is that the original MD member
> metadata blocks are no longer available unless something like
> this has been done:
>
> https://raid.wiki.kernel.org/index.php/RAID_Recovery
> «Preserving RAID superblock information
> One of the most useful things to do first, when trying to
> recover a broken RAID array, is to preserve the information
> reported in the RAID superblocks on each device at the time
> the array went down (and before you start trying to recreate
> the array). Something like
> mdadm --examine /dev/sd[bcdefghijklmn]1 >> raid.status»
>
> If you went to the lengths to write 'dd' expressions, you might
> as well have saved the output of '--examine'. Perhaps you did,
> but if you did not attach that output to your request for help
> it would be rather "stunning".
>
> [ ... ]
>
>> Is the "mdadm --create" operation that I issued, incorrect?
>> Have I done anything in error?
> There is something strange: what you report being the output of
> '--detail' from July:
>
> Array Size : 17581609728 (16767.13 GiB 18003.57 GB)
> Used Dev Size : 1953512192 (1863.01 GiB 2000.40 GB)
>
> and the output of '--detail' for the re-created:
>
> Array Size : 17580439296 (16766.01 GiB 18002.37 GB)
> Used Dev Size : 1953382144 (1862.89 GiB 2000.26 GB)
>
> Both numbers don't match. They are *slightly* different. In
> particular it is rather strange that the "Used Dev Size" is
> different. How is that possible? Have the disks shrunk a little
> in the meantime? :-)
>
> It is intriguing that the difference between 1953512192 and
> 1953382144 is 1024*127KiB or 1024*254 sectors.
>
> Also I have noticed that the MD set is composed of disk of 3
> different models (ST2000DL003-9VT1, ST2000DM001-1CH1,
> ST32000542AS)...
>
>> Is my data gone? Any and all insight are extremly welcomed and
>> appreciated.
> Whether your data is gone depends on what kind of hardware issue
> you have had, and to the consequence of the "brave" '--create'
> above. But also how the MD set was setup, e.g. with members of
> slightly different sizes. The inconsistencies in the reported
> numbers are "confusing".
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
Hello Peter,
Thank you very much for your thorough and responsive reply. I will do my
best to clarify where possible.
> [ ... ] The server experienced some sort of hardware event
> that resulted in a mandatory restart of the server.
> Details would be helpful: because if some problem happens the
> standard advice is "reload from backups". If you want to
> shortcut that to mostly-recovery context matters to figuring out
> how and how safely.
Regarding the nature of the hardware event, unfortunately details are in
short supply: the server became unresponsive over the console when
attempting to connect via SSH, prompting a restart of the server. I
don't believe there was evidence of a power drop or loss. No server or
kernel logs are available for review.
> [ ... ] completed the restart, the array looked like this,
> "all spares":
>> md6 :
> What happened to the other MD sets on the same server, if any?
> Any damage? Because if those suffered no damage, there is the
> possibility that the disk rack backplane holding the members of
> 'md6' got damaged, or the specific host adapter; and that the MD
> set content is entirely undamaged and the funny stuff being read
> is a transmission problem.
"md6" is the only MD set on the server, so name as it is has a
raid-level 6. Sorry for any confusion.
> The mdadm array has the following characteristics: RAID level:
> 6 Chunk size: 256k Version: 1.2 Number of devices: 11
> How do you know? Is this part of your records or from actual
> output of 'mdadm --examine'?
> All attempts to assemble the array continued to result in the "all
> spare" condition (output above). Thinking that the metadata had been
> corrupted somehow,
> Apparently without ever trying 'mdadm --detail /dev/md6' or
> 'mdadm --examine /dev/sd...' as per:
> If you went to the lengths to write 'dd' expressions, you might
> as well have saved the output of '--examine'. Perhaps you did,
> but if you did not attach that output to your request for help
> it would be rather "stunning".
Yes, I saved the -E/--examine information, "just in case". :-)
Before performing a re-create of the array, I did, in fact, print the
contents (-E, --examine) of the metadata stored on each device:
# cat mdadm.e.bak
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 6958450b:e4dfa2f3:259ff733:f343a584
Name : server:6 (local to host server)
Creation Time : Sat Apr 23 06:22:23 2011
Raid Level : raid6
Raid Devices : 11
Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
Array Size : 17581609728 (16767.13 GiB 18003.57 GB)
Used Dev Size : 3907024384 (1863.01 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : active
Device UUID : 12a56302:5b436263:1b841be2:fccd07ed
Update Time : Fri Nov 7 00:37:26 2014
Checksum : d7063845 - correct
Events : 667126
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 0
Array State : A.A.AA.AAA. ('A' == active, '.' == missing)
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 6958450b:e4dfa2f3:259ff733:f343a584
Name : server:6 (local to host server)
Creation Time : Sat Apr 23 06:22:23 2011
Raid Level : raid6
Raid Devices : 11
Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
Array Size : 17581609728 (16767.13 GiB 18003.57 GB)
Used Dev Size : 3907024384 (1863.01 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 0416e499:16488db2:5473119d:1a0c8141
Update Time : Sun Nov 2 12:24:42 2014
Checksum : cd22e98b - correct
Events : 667122
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 6
Array State : A.A.AAAAAA. ('A' == active, '.' == missing)
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 6958450b:e4dfa2f3:259ff733:f343a584
Name : server:6 (local to host server)
Creation Time : Sat Apr 23 06:22:23 2011
Raid Level : raid6
Raid Devices : 11
Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
Array Size : 17581609728 (16767.13 GiB 18003.57 GB)
Used Dev Size : 3907024384 (1863.01 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : active
Device UUID : 56f35811:d62afc50:a893a3af:10f01367
Update Time : Fri Nov 7 00:37:26 2014
Checksum : 1b299f9b - correct
Events : 667126
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 2
Array State : A.A.AA.AAA. ('A' == active, '.' == missing)
/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 6958450b:e4dfa2f3:259ff733:f343a584
Name : server:6 (local to host server)
Creation Time : Sat Apr 23 06:22:23 2011
Raid Level : raid6
Raid Devices : 11
Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
Array Size : 17581609728 (16767.13 GiB 18003.57 GB)
Used Dev Size : 3907024384 (1863.01 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 63f4d908:16f38b7f:ebd9a1d7:0f186e56
Update Time : Sun Nov 2 10:23:32 2014
Checksum : 5896c904 - correct
Events : 667118
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 1
Array State : AAAAAAAAAA. ('A' == active, '.' == missing)
/dev/sdf1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 6958450b:e4dfa2f3:259ff733:f343a584
Name : server:6 (local to host server)
Creation Time : Sat Apr 23 06:22:23 2011
Raid Level : raid6
Raid Devices : 11
Avail Dev Size : 3907026672 (1863.02 GiB 2000.40 GB)
Array Size : 17581609728 (16767.13 GiB 18003.57 GB)
Used Dev Size : 3907024384 (1863.01 GiB 2000.40 GB)
Data Offset : 272 sectors
Super Offset : 8 sectors
State : clean
Device UUID : ee4ac68b:2152463c:b0d72a12:4da24489
Update Time : Sun Nov 2 10:23:32 2014
Checksum : 59d06a2 - correct
Events : 667118
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 3
Array State : AAAAAAAAAA. ('A' == active, '.' == missing)
/dev/sdg1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 6958450b:e4dfa2f3:259ff733:f343a584
Name : server:6 (local to host server)
Creation Time : Sat Apr 23 06:22:23 2011
Raid Level : raid6
Raid Devices : 11
Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
Array Size : 17581609728 (16767.13 GiB 18003.57 GB)
Used Dev Size : 3907024384 (1863.01 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 72ee0230:51b42c7a:3327c930:302be14e
Update Time : Sun Nov 2 08:35:01 2014
Checksum : cbfacb4a - correct
Events : 667100
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 10
Array State : .AAAAAAAAAA ('A' == active, '.' == missing)
/dev/sdh1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 6958450b:e4dfa2f3:259ff733:f343a584
Name : server:6 (local to host server)
Creation Time : Sat Apr 23 06:22:23 2011
Raid Level : raid6
Raid Devices : 11
Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
Array Size : 17581609728 (16767.13 GiB 18003.57 GB)
Used Dev Size : 3907024384 (1863.01 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : active
Device UUID : 429cfff7:ecadc967:40f73261:bef9656e
Update Time : Fri Nov 7 00:37:26 2014
Checksum : d17f38ee - correct
Events : 667126
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 9
Array State : A.A.AA.AAA. ('A' == active, '.' == missing)
/dev/sdi1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 6958450b:e4dfa2f3:259ff733:f343a584
Name : server:6 (local to host server)
Creation Time : Sat Apr 23 06:22:23 2011
Raid Level : raid6
Raid Devices : 11
Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
Array Size : 17581609728 (16767.13 GiB 18003.57 GB)
Used Dev Size : 3907024384 (1863.01 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : active
Device UUID : 6dea792a:f1117c0c:ac16951c:a8b61783
Update Time : Fri Nov 7 00:37:26 2014
Checksum : 78bfc76c - correct
Events : 667126
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 4
Array State : A.A.AA.AAA. ('A' == active, '.' == missing)
/dev/sdj1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 6958450b:e4dfa2f3:259ff733:f343a584
Name : server:6 (local to host server)
Creation Time : Sat Apr 23 06:22:23 2011
Raid Level : raid6
Raid Devices : 11
Avail Dev Size : 3907026672 (1863.02 GiB 2000.40 GB)
Array Size : 17581609728 (16767.13 GiB 18003.57 GB)
Used Dev Size : 3907024384 (1863.01 GiB 2000.40 GB)
Data Offset : 272 sectors
Super Offset : 8 sectors
State : active
Device UUID : 4b37d852:2236e8e6:15c52c77:4214f7de
Update Time : Fri Nov 7 00:37:26 2014
Checksum : 32014484 - correct
Events : 667126
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 7
Array State : A.A.AA.AAA. ('A' == active, '.' == missing)
/dev/sdk1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 6958450b:e4dfa2f3:259ff733:f343a584
Name : server:6 (local to host server)
Creation Time : Sat Apr 23 06:22:23 2011
Raid Level : raid6
Raid Devices : 11
Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
Array Size : 17581609728 (16767.13 GiB 18003.57 GB)
Used Dev Size : 3907024384 (1863.01 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : active
Device UUID : aa149905:9cd207c4:4bb4c244:3f502348
Update Time : Fri Nov 7 00:37:26 2014
Checksum : f8a3e98f - correct
Events : 667126
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 8
Array State : A.A.AA.AAA. ('A' == active, '.' == missing)
/dev/sdl1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 6958450b:e4dfa2f3:259ff733:f343a584
Name : server:6 (local to host server)
Creation Time : Sat Apr 23 06:22:23 2011
Raid Level : raid6
Raid Devices : 11
Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
Array Size : 17581609728 (16767.13 GiB 18003.57 GB)
Used Dev Size : 3907024384 (1863.01 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : active
Device UUID : 59a2393b:27209cc2:1f6fa576:5ed6e2a7
Update Time : Fri Nov 7 00:37:26 2014
Checksum : be7b7d99 - correct
Events : 667126
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 5
Array State : A.A.AA.AAA. ('A' == active, '.' == missing)
> There is something strange: what you report being the output of
> '--detail' from July:
>
> Array Size : 17581609728 (16767.13 GiB 18003.57 GB)
> Used Dev Size : 1953512192 (1863.01 GiB 2000.40 GB)
>
> and the output of '--detail' for the re-created:
>
> Array Size : 17580439296 (16766.01 GiB 18002.37 GB)
> Used Dev Size : 1953382144 (1862.89 GiB 2000.26 GB)
>
> Both numbers don't match. They are*slightly* different. In
> particular it is rather strange that the "Used Dev Size" is
> different. How is that possible? Have the disks shrunk a little
> in the meantime?
Peter, that is an excellent observation! Indeed, the above -E/--examine
data confirms that some disks have a 272 offset, while most others have
a 2048 offset, for example:
/dev/sdj1:
Data Offset : 272 sectors
Super Offset : 8 sectors
/dev/sdk1:
Data Offset: 2048 sectors
Super Offset: 8 sectors
The current breakdown:
# grep "272 sectors" mdadm.e.bak | wc -l
2
# grep "2048 sectors" mdadm.e.bak | wc -l
9
Therefore, based on the backup -E/--examine data, two out of the 11
total disks have an offset of 272, while the remaining nine are using 2048.
Could this explain the discrepancy you observed?
For the record, every disk is GUID GPT partitioned, with the same sector
size for all partitions. All partitions are identical in sector size,
regardless of the Seagate HDD disk model.
Here is a sample of the partition data:
# parted /dev/sdb unit s print | grep -A1 Number
Number Start End Size File system Name Flags
1 2048s 3907028991s 3907026944s primary raid
# parted /dev/sdc unit s print | grep -A1 Number
Number Start End Size File system Name Flags
1 2048s 3907028991s 3907026944s primary raid
# parted /dev/sdd unit s print | grep -A1 Number
Number Start End Size File system Name Flags
1 2048s 3907028991s 3907026944s primary raid
# parted /dev/sde unit s print | grep -A1 Number
Number Start End Size File system Name Flags
1 2048s 3907028991s 3907026944s primary raid
# parted /dev/sdf unit s print | grep -A1 Number
Number Start End Size File system Name Flags
1 2048s 3907028991s 3907026944s primary raid
# parted /dev/sdg unit s print | grep -A1 Number
Number Start End Size File system Name Flags
1 2048s 3907028991s 3907026944s primary raid
# parted /dev/sdh unit s print | grep -A1 Number
Number Start End Size File system Name Flags
1 2048s 3907028991s 3907026944s ntfs primary raid
# parted /dev/sdi unit s print | grep -A1 Number
Number Start End Size File system Name Flags
1 2048s 3907028991s 3907026944s primary raid
# parted /dev/sdj unit s print | grep -A1 Number
Number Start End Size File system Name Flags
1 2048s 3907028991s 3907026944s primary raid
# parted /dev/sdk unit s print | grep -A1 Number
Number Start End Size File system Name Flags
1 2048s 3907028991s 3907026944s primary raid
# parted /dev/sdl unit s print | grep -A1 Number
Number Start End Size File system Name Flags
1 2048s 3907028991s 3907026944s primary raid
My only explanation is that the cause of this offset discrepancy may
have something to do with the age of the array. The array had an
original creation time of year 2011.
This server was originally running Ubuntu 10.04 LTS (I believe) before
being eventually upgraded to 12.04 LTS--although the server has been
running healthy on 12.04 LTS for several years without issue(s).
If memory serves, the older version of mdadm that shipped with 10.04 LTS
did a myriad of things differently regarding the location of the
superblock(s), offset(s), etc. but I can't say for sure. Did older mdadm
builds on 12.04 LTS ever use offsets of 272, rather than 2048?
Perhaps Neil could comment? :-)
I do hope that supplying the -E/--examine information will be useful to
you all. What's the next step?
Thank you for all your efforts and for your keen eyes.
-xar
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: RAID 6 (containing LUKS dm-crypt) recovery help.
From: Peter Grandi @ 2014-11-07 10:24 UTC (permalink / raw)
To: Linux RAID
In-Reply-To: <545C5C9D.9000309@retard.io>
> [ ... ] The server experienced some sort of hardware event
> that resulted in a mandatory restart of the server.
Details would be helpful: because if some problem happens the
standard advice is "reload from backups". If you want to
shortcut that to mostly-recovery context matters to figuring out
how and how safely.
> [ ... ] completed the restart, the array looked like this,
> "all spares":
> md6 :
What happened to the other MD sets on the same server, if any?
Any damage? Because if those suffered no damage, there is the
possibility that the disk rack backplane holding the members of
'md6' got damaged, or the specific host adapter; and that the MD
set content is entirely undamaged and the funny stuff being read
is a transmission problem.
> inactive sdl1[7](S) sdh1[13](S) sdg1[14](S) sdk1[11](S)
> sdj1[10](S) sdi1[6](S) sdd1[2](S) sdf1[8](S) sdb1[12](S)
> sde1[3](S) sdc1[15](S) 21488638704 blocks super 1.2
"Clever" people hide details as possible, and go to such lengths
as to actually remove vital information as for example what
literally follows "super 1.2" here. Because actual quotes are
too "insipid" and paraphrases are more "challenging":
> The mdadm array has the following characteristics: RAID level:
> 6 Chunk size: 256k Version: 1.2 Number of devices: 11
How do you know? Is this part of your records or from actual
output of 'mdadm --examine'?
But assuming the above is somewhat reliable there is an
"interesting" situation: in "21488638704 blocks" the number
21,488,638,704 is not a whole multiple of 9:
$ factor 21488638704
21488638704: 2 2 2 2 3 13 1801 19121
> All attempts to assemble the array continued to result in the "all
> spare" condition (output above). Thinking that the metadata had been
> corrupted somehow,
Apparently without ever trying 'mdadm --detail /dev/md6' or
'mdadm --examine /dev/sd...' as per:
https://raid.wiki.kernel.org/index.php/RAID_Recovery
> I set out to recreate the array.
Quite "brave":
https://raid.wiki.kernel.org/index.php/RAID_Recovery
«Restore array by recreating (after multiple device failure)
Recreating should be considered a *last* resort, only to be
used when everything else fails.
People getting this wrong is one of the primary reasons people
lose data. It is very commonly used way too early in the fault
finding process. You have been warned!»
> The following is the dev_number fields from the metadata,
> before I attempted to recreate the array: for i in /dev/sd?1;
> do echo -n $i '' ; dd 2> /dev/null if=$i bs=1 count=4
> skip=4256 | od -D | head -n1; done: I used the following to
> extract the index position of each device on a device I
> suspected wasn't corrupted (for the record, they all returned
> the same data): [ ... ]
It is very "astute" indeed to use 'dd' instead of 'mdadm
--examine'. For example it "encourages" people who might want
to help to spend some extra time checking your offsets, that
"teaches" them.
[ ... ]
> Number Major Minor RaidDevice State
> 12 8 17 0 active sync /dev/sdb1
> 3 8 65 1 active sync /dev/sde1
> 2 8 49 2 active sync /dev/sdd1
> 8 8 81 3 active sync /dev/sdf1
> 6 8 129 4 active sync /dev/sdi1
> 7 8 177 5 active sync /dev/sdl1
> 6 0 0 6 removed
> 10 8 145 7 active sync /dev/sdj1
> 11 8 161 8 active sync /dev/sdk1
> 13 8 113 9 active sync /dev/sdh1
> 14 8 97 10 active sync /dev/sdg1
> The dev_numbers and index position information in conjunction
> with the historic data (directly above) seemed to indicate
> that the proper recreation order and command would be the
> following:
> mdadm --create /dev/md6 --assume-clean --level=6
> --raid-devices=11 --metadata=1.2 --chunk=256 /dev/sdb1
> /dev/sde1 /dev/sdd1 /dev/sdf1 /dev/sdi1 /dev/sdl1 /dev/sdc1
> /dev/sdj1 /dev/sdk1 /dev/sdh1 /dev/sdg1
The main consequence of the above is that the original MD member
metadata blocks are no longer available unless something like
this has been done:
https://raid.wiki.kernel.org/index.php/RAID_Recovery
«Preserving RAID superblock information
One of the most useful things to do first, when trying to
recover a broken RAID array, is to preserve the information
reported in the RAID superblocks on each device at the time
the array went down (and before you start trying to recreate
the array). Something like
mdadm --examine /dev/sd[bcdefghijklmn]1 >> raid.status»
If you went to the lengths to write 'dd' expressions, you might
as well have saved the output of '--examine'. Perhaps you did,
but if you did not attach that output to your request for help
it would be rather "stunning".
[ ... ]
> Is the "mdadm --create" operation that I issued, incorrect?
> Have I done anything in error?
There is something strange: what you report being the output of
'--detail' from July:
Array Size : 17581609728 (16767.13 GiB 18003.57 GB)
Used Dev Size : 1953512192 (1863.01 GiB 2000.40 GB)
and the output of '--detail' for the re-created:
Array Size : 17580439296 (16766.01 GiB 18002.37 GB)
Used Dev Size : 1953382144 (1862.89 GiB 2000.26 GB)
Both numbers don't match. They are *slightly* different. In
particular it is rather strange that the "Used Dev Size" is
different. How is that possible? Have the disks shrunk a little
in the meantime? :-)
It is intriguing that the difference between 1953512192 and
1953382144 is 1024*127KiB or 1024*254 sectors.
Also I have noticed that the MD set is composed of disk of 3
different models (ST2000DL003-9VT1, ST2000DM001-1CH1,
ST32000542AS)...
> Is my data gone? Any and all insight are extremly welcomed and
> appreciated.
Whether your data is gone depends on what kind of hardware issue
you have had, and to the consequence of the "brave" '--create'
above. But also how the MD set was setup, e.g. with members of
slightly different sizes. The inconsistencies in the reported
numbers are "confusing".
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* color box, display box, corrugated box, color card, blister card, color sleeve, hang tag, label
From: Jinghao Printing - CHINA @ 2014-11-07 6:24 UTC (permalink / raw)
Hi, this is David Wu from Shanghai, China.
We are a printing company, we can print color box, corrugated box,
label, hang tag etc.
Please let me know if you need these.
I will send you the website then.
Best regards,
David Wu
^ permalink raw reply
* RAID 6 (containing LUKS dm-crypt) recovery help.
From: xar @ 2014-11-07 5:46 UTC (permalink / raw)
To: linux-raid
Greetings,
I have a RAID 6 (which contains a LUKS container) that I'm hoping to get
some help/insight in recovering. The server experienced some sort of
hardware event that resulted in a mandatory restart of the server.
For the record, after the server completed the restart, the array looked
like this, "all spares":
md6 : inactive sdl1[7](S) sdh1[13](S) sdg1[14](S) sdk1[11](S)
sdj1[10](S) sdi1[6](S) sdd1[2](S) sdf1[8](S) sdb1[12](S) sde1[3](S)
sdc1[15](S) 21488638704 blocks super 1.2
The server in question is Ubuntu ("Precise") 12.04.5 LTS with mdadm
version 3.2.5-1ubuntu0 installed.
The mdadm array has the following characteristics:
RAID level: 6
Chunk size: 256k
Version: 1.2
Number of devices: 11
All attempts to assemble the array continued to result in the "all
spare" condition (output above). Thinking that the metadata had been
corrupted somehow, I set out to recreate the array.
The following is the dev_number fields from the metadata, before I
attempted to recreate the array:
# for i in /dev/sd?1; do echo -n $i '' ; dd 2> /dev/null if=$i bs=1
count=4 skip=4256 | od -D | head -n1; done:
/dev/sdb1 0000000 12
/dev/sdc1 0000000 15
/dev/sdd1 0000000 2
/dev/sde1 0000000 3
/dev/sdf1 0000000 8
/dev/sdg1 0000000 14
/dev/sdh1 0000000 13
/dev/sdi1 0000000 6
/dev/sdj1 0000000 10
/dev/sdk1 0000000 11
/dev/sdl1 0000000 7
I used the following to extract the index position of each device on a
device I suspected wasn't corrupted (for the record, they all returned
the same data):
# dd 2> /dev/null if=/dev/sdc1 bs=2 count=6 skip=2176 | od -d
0000000 65534 65534 2 65534 65534 65534 4 5
0000020 65534 65534 7 8 0 9 65534
0000036
As you can see, there's already a visible mismatch between the
dev_number and the listed indexes. For instance, /dev/sdc1 returned a
device number of 15, but there's not a 15th position in the
corresponding list.
I pulled from log history, the last known working "layout", circa July
of this year:
# mdadm -D /dev/md6
/dev/md6:
Version : 1.2
Creation Time : Sat Apr 23 06:22:23 2011
Raid Level : raid6
Array Size : 17581609728 (16767.13 GiB 18003.57 GB)
Used Dev Size : 1953512192 (1863.01 GiB 2000.40 GB)
Raid Devices : 11
Total Devices : 10
Persistence : Superblock is persistent
Update Time : Sat Jun 21 21:13:45 2014
State : clean, degraded
Active Devices : 10
Working Devices : 10
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 256K
Name : server:6 (local to host server)
UUID : 6958450b:e4dfa2f3:259ff733:f343a584
Events : 390345
Number Major Minor RaidDevice State
12 8 17 0 active sync /dev/sdb1
3 8 65 1 active sync /dev/sde1
2 8 49 2 active sync /dev/sdd1
8 8 81 3 active sync /dev/sdf1
6 8 129 4 active sync /dev/sdi1
7 8 177 5 active sync /dev/sdl1
6 0 0 6 removed
10 8 145 7 active sync /dev/sdj1
11 8 161 8 active sync /dev/sdk1
13 8 113 9 active sync /dev/sdh1
14 8 97 10 active sync /dev/sdg1
The dev_numbers and index position information in conjunction with the
historic data (directly above) seemed to indicate that the proper
recreation order and command would be the following:
# mdadm --create /dev/md6 --assume-clean --level=6 --raid-devices=11
--metadata=1.2 --chunk=256 /dev/sdb1 /dev/sde1 /dev/sdd1 /dev/sdf1
/dev/sdi1 /dev/sdl1 /dev/sdc1 /dev/sdj1 /dev/sdk1 /dev/sdh1 /dev/sdg1
I ran this above command.
Here is the current output of "lsdrv" which I observed several people
found to be relevant on this list, from creator 'pturmel' on github:
PCI [ahci] 00:11.0 SATA controller: Advanced Micro Devices, Inc.
[AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] (rev 40)
âscsi 0:0:0:0 ATA ST31500541AS {6XW03WTD}
ââsda 1.36t [8:0] Partitioned (dos)
â âsda1 109.79m [8:1] ext4 {07f99e8c-95d2-483d-9850-05f04820c3f6}
â ââMounted as /dev/sda1 @ /boot
â âsda2 2.01g [8:2] swap {d137430d-815a-4c45-a394-9bece3aa7136}
â âsda3 7.01g [8:3] ext4 {8db73200-8d9d-4991-9802-b13f1550a9d9}
â ââMounted as /dev/disk/by-uuid/8db73200-8d9d-4991-9802-b13f1550a9d9 @ /
â âsda4 1.36t [8:4] Empty/Unknown
â âdm-0 1.36t [252:0] xfs {db7ddb53-080c-45ba-ab4d-e45d35eb451c}
â âMounted as /dev/mapper/enc @ /encrypted
âscsi 1:0:0:0 ATA ST2000DL003-9VT1 {5YD4VZLV}
ââsdb 1.82t [8:16] Partitioned (gpt)
â âsdb1 1.82t [8:17] MD raid6 (11) inactive 'server:6'
{c88f3b40-c7d4-a33a-5007-ab8ef784c0c2}
âscsi 2:0:0:0 ATA ST2000DM001-1CH1 {Z1E8GNFQ}
ââsdc 1.82t [8:32] Partitioned (gpt)
â âsdc1 1.82t [8:33] MD raid6 (11) inactive 'server:6'
{c88f3b40-c7d4-a33a-5007-ab8ef784c0c2}
âscsi 3:0:0:0 ATA ST2000DL003-9VT1 {5YD2PZM3}
ââsdd 1.82t [8:48] Partitioned (gpt)
â âsdd1 1.82t [8:49] MD raid6 (11) inactive 'server:6'
{c88f3b40-c7d4-a33a-5007-ab8ef784c0c2}
âscsi 4:0:0:0 ATA ST2000DL003-9VT1 {5YD2J0XD}
ââsde 1.82t [8:64] Partitioned (gpt)
â âsde1 1.82t [8:65] MD raid6 (11) inactive 'server:6'
{c88f3b40-c7d4-a33a-5007-ab8ef784c0c2}
âscsi 5:0:0:0 ATA ST2000DL003-9VT1 {5YD3XE9M}
âsdf 1.82t [8:80] Partitioned (gpt)
âsdf1 1.82t [8:81] MD raid6 (11) inactive 'server:6'
{c88f3b40-c7d4-a33a-5007-ab8ef784c0c2}
PCI [sata_sil24] 04:04.0 RAID bus controller: Silicon Image, Inc. SiI
3124 PCI-X Serial ATA Controller (rev 02)
âscsi 6:0:0:0 ATA ST2000DL003-9VT1 {5YD6JW2L}
ââsdg 1.82t [8:96] Partitioned (gpt)
â âsdg1 1.82t [8:97] MD raid6 (11) inactive 'server:6'
{65daae65-118b-896a-6205-0f2c4dacb4de}
âscsi 7:0:0:0 ATA ST2000DL003-9VT1 {6YD05E5Y}
ââsdh 1.82t [8:112] Partitioned (gpt)
â âsdh1 1.82t [8:113] MD raid6 (11) inactive 'server:6'
{c88f3b40-c7d4-a33a-5007-ab8ef784c0c2}
âscsi 8:x:x:x [Empty]
âscsi 9:x:x:x [Empty]
PCI [sata_sil24] 06:04.0 RAID bus controller: Silicon Image, Inc. SiI
3124 PCI-X Serial ATA Controller (rev 02)
âscsi 10:0:0:0 ATA ST32000542AS {5XW1PVCZ}
ââsdi 1.82t [8:128] Partitioned (gpt)
â âsdi1 1.82t [8:129] MD raid6 (11) inactive 'server:6'
{c88f3b40-c7d4-a33a-5007-ab8ef784c0c2}
âscsi 11:0:0:0 ATA ST2000DL003-9VT1 {5YD2SND2}
ââsdj 1.82t [8:144] Partitioned (gpt)
â âsdj1 1.82t [8:145] MD raid6 (11) inactive 'server:6'
{c88f3b40-c7d4-a33a-5007-ab8ef784c0c2}
âscsi 12:0:0:0 ATA ST2000DL003-9VT1 {5YD4JTZP}
ââsdk 1.82t [8:160] Partitioned (gpt)
â âsdk1 1.82t [8:161] MD raid6 (11) inactive 'server:6'
{c88f3b40-c7d4-a33a-5007-ab8ef784c0c2}
âscsi 13:0:0:0 ATA ST32000542AS {5XW1KAEA}
âsdl 1.82t [8:176] Partitioned (gpt)
âsdl1 1.82t [8:177] MD raid6 (11) inactive 'server:6'
{c88f3b40-c7d4-a33a-5007-ab8ef784c0c2}
Other Block Devices
âloop0 0.00k [7:0] Empty/Unknown
âloop1 0.00k [7:1] Empty/Unknown
âloop2 0.00k [7:2] Empty/Unknown
âloop3 0.00k [7:3] Empty/Unknown
âloop4 0.00k [7:4] Empty/Unknown
âloop5 0.00k [7:5] Empty/Unknown
âloop6 0.00k [7:6] Empty/Unknown
âloop7 0.00k [7:7] Empty/Unknown
âram0 64.00m [1:0] Empty/Unknown
âram1 64.00m [1:1] Empty/Unknown
âram2 64.00m [1:2] Empty/Unknown
âram3 64.00m [1:3] Empty/Unknown
âram4 64.00m [1:4] Empty/Unknown
âram5 64.00m [1:5] Empty/Unknown
âram6 64.00m [1:6] Empty/Unknown
âram7 64.00m [1:7] Empty/Unknown
âram8 64.00m [1:8] Empty/Unknown
âram9 64.00m [1:9] Empty/Unknown
âram10 64.00m [1:10] Empty/Unknown
âram11 64.00m [1:11] Empty/Unknown
âram12 64.00m [1:12] Empty/Unknown
âram13 64.00m [1:13] Empty/Unknown
âram14 64.00m [1:14] Empty/Unknown
âram15 64.00m [1:15] Empty/Unknown
Following the recreation, the array now looks like this:
# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
md6 : active raid6 sdh1[9] sdk1[8] sdj1[7] sdc1[6] sdl1[5] sdi1[4]
sdf1[3] sdd1[2] sde1[1] sdb1[0]
17580439296 blocks super 1.2 level 6, 256k chunk, algorithm 2
[11/10] [UUUUUUUUUUU]
# mdadm -D /dev/md6
/dev/md6:
Version : 1.2
Creation Time : Fri Nov 7 05:40:16 2014
Raid Level : raid6
Array Size : 17580439296 (16766.01 GiB 18002.37 GB)
Used Dev Size : 1953382144 (1862.89 GiB 2000.26 GB)
Raid Devices : 11
Total Devices : 11
Persistence : Superblock is persistent
Update Time : Fri Nov 7 05:40:16 2014
State : clean
Active Devices : 11
Working Devices : 11
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 256K
Name : server:6 (local to host server)
UUID : b306872f:5ef902a8:76f5e233:f220f4d4
Events : 0
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 65 1 active sync /dev/sde1
2 8 49 2 active sync /dev/sdd1
3 8 81 3 active sync /dev/sdf1
4 8 129 4 active sync /dev/sdi1
5 8 177 5 active sync /dev/sdl1
6 8 33 6 active sync /dev/sdc1
7 8 145 7 active sync /dev/sdj1
8 8 161 8 active sync /dev/sdk1
9 8 113 9 active sync /dev/sdh1
10 8 97 10 active sync /dev/sdg1
This array should contain a LUKS container, however, it's missing. If I
hexdump the first 20 lines, the LUKS header is completely missing:
# cryptsetup luksOpen /dev/md6 luks
Device /dev/md6 is not a valid LUKS device.
# hexdump -C /dev/md6 | head -n16
00000000 0b 37 89 e0 66 96 7a d4 6c 5b 57 09 a5 8d 6a c5
|.7..f.z.l[W...j.|
00000010 a7 65 20 6e f0 db 74 db 03 d8 e9 2b 39 05 37 a4 |.e
n..t....+9.7.|
00000020 cb 25 d7 7b fd cf b5 b4 12 ad e2 24 24 de 66 42
|.%.{.......$$.fB|
00000030 61 a2 1b ea 8b 5c 04 38 7e 5e 61 11 3d ba 99 35
|a....\.8~^a.=..5|
00000040 b7 e9 e6 76 72 18 d2 d5 bd cd 1b ed 59 15 fb 83
|...vr.......Y...|
00000050 bc 57 94 85 31 c1 3e af 51 f1 25 50 db 57 d3 cd
|.W..1.>.Q.%P.W..|
00000060 69 d5 31 23 df 01 ef 03 e3 92 66 c6 1f 38 3f 57
|i.1#......f..8?W|
00000070 67 20 38 8c c2 ec 25 dc 59 42 b4 5d 9d 9e c1 79 |g
8...%.YB.]...y|
00000080 4a f5 e1 ad f8 08 16 d5 37 3f f6 83 62 f2 6f f5
|J.......7?..b.o.|
00000090 53 95 4f 69 ce 7c ba 4c 86 ef a1 1c 04 d7 b3 17
|S.Oi.|.L........|
000000a0 cd ea 5f 25 56 a4 0d 6f 64 e9 51 b5 71 b3 18 7f
|.._%V..od.Q.q...|
000000b0 46 e7 8b ab 08 ae f5 ed 65 0d 8f 3e 8b 03 25 5c
|F.......e..>..%\|
000000c0 bb 50 dc e6 31 33 4a 88 8e 22 20 72 f0 11 71 d0 |.P..13J.."
r..q.|
000000d0 59 c7 9d 20 f8 e2 f0 f8 75 5f ea 4a 57 d7 d7 9e |Y..
....u_.JW...|
000000e0 c8 05 85 9d d7 cf c9 ab 53 de 11 6f bf d4 e3 b2
|........S..o....|
000000f0 f6 5e 1e 46 5c 16 ae 46 a3 b5 9b f4 b9 ff ca 0c
|.^.F\..F........|
Is the "mdadm --create" operation that I issued, incorrect? Have I done
anything in error?
Unfortunately, I do not have a backup of the LUKS header, as I've
personally never encountered a situation like this, nor was I privy to
the knowledge that LUKS headers should be backed up at all.
If relevant, I did observe that /dev/sdb1 and /dev/sdg1 were showing
"Offline_Uncorrectable" errors via SMART, but nothing that I could
imagine would have contributed to this current predicament.
Is my data gone? Any and all insight are extremly welcomed and appreciated.
Warm regards,
-xar
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Raid5 drive fail during grow and no backup
From: Vince @ 2014-11-06 17:12 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <loom.20141105T200200-434@post.gmane.org>
Hi Phil,
> This is a problem you haven't solved yet, I think. The raid array
> should have fixed this bad sector for you without kicking the drive out.
> The scenario is common with "green" drives and/or consumer-grade
> drives in general.
i investigated some time and now i am a bit confused.
All my 5 WD Red drives have ERC enables (7sec)
The kernel timeout is set to 30sec (/sys/block/sdb/device/timeout)
on all devices.
Unfortunately i haven't any backup of the dmesg output, but i can remeber i
got something like:
"failed command: READ FPDMA QUEUED status: { DRDY ERR } error: { UNC }".
This shows up several times until it ends with showing me the sector which
causes the problem.
My raid is still up, but as you mentioned i would like some kind of self
repair if a sector is unreadable instead throwing the disk out of the array.
Here is the samrtctl of one drive that fails.
Do you have any idea if i missing some settings etc?
smartctl -x /dev/sdd
smartctl 5.41 2011-06-09 r3365 [x86_64-linux-3.2.0-4-amd64] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red (AF)
Device Model: WDC WD30EFRX-68AX9N0
Serial Number: WD-WMC1T2041480
LU WWN Device Id: 5 0014ee 058d836a6
Firmware Version: 80.00A80
User Capacity: 3.000.592.982.016 bytes [3,00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: 9
ATA Standard is: Exact ATA specification draft version not indicated
Local Time is: Thu Nov 6 18:10:11 2014 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (42000) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 255) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x70bd) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-K 200 200 051 - 1625
3 Spin_Up_Time POS--K 176 173 021 - 6175
4 Start_Stop_Count -O--CK 099 099 000 - 1290
5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0
7 Seek_Error_Rate -OSR-K 200 200 000 - 0
9 Power_On_Hours -O--CK 084 084 000 - 11786
10 Spin_Retry_Count -O--CK 100 100 000 - 0
11 Calibration_Retry_Count -O--CK 100 100 000 - 0
12 Power_Cycle_Count -O--CK 100 100 000 - 177
192 Power-Off_Retract_Count -O--CK 200 200 000 - 50
193 Load_Cycle_Count -O--CK 200 200 000 - 1239
194 Temperature_Celsius -O---K 121 102 000 - 29
196 Reallocated_Event_Count -O--CK 200 200 000 - 0
197 Current_Pending_Sector -O--CK 200 200 000 - 0
198 Offline_Uncorrectable ----CK 200 200 000 - 0
199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 0
200 Multi_Zone_Error_Rate ---R-- 200 200 000 - 0
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
GP/S Log at address 0x00 has 1 sectors [Log Directory]
SMART Log at address 0x01 has 1 sectors [Summary SMART error log]
SMART Log at address 0x02 has 5 sectors [Comprehensive SMART error log]
GP Log at address 0x03 has 6 sectors [Ext. Comprehensive SMART error log]
SMART Log at address 0x06 has 1 sectors [SMART self-test log]
GP Log at address 0x07 has 1 sectors [Extended self-test log]
SMART Log at address 0x09 has 1 sectors [Selective self-test log]
GP Log at address 0x10 has 1 sectors [NCQ Command Error]
GP Log at address 0x11 has 1 sectors [SATA Phy Event Counters]
GP Log at address 0x21 has 1 sectors [Write stream error log]
GP Log at address 0x22 has 1 sectors [Read stream error log]
GP/S Log at address 0x80 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x81 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x82 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x83 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x84 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x85 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x86 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x87 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x88 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x89 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x8a has 16 sectors [Host vendor specific log]
GP/S Log at address 0x8b has 16 sectors [Host vendor specific log]
GP/S Log at address 0x8c has 16 sectors [Host vendor specific log]
GP/S Log at address 0x8d has 16 sectors [Host vendor specific log]
GP/S Log at address 0x8e has 16 sectors [Host vendor specific log]
GP/S Log at address 0x8f has 16 sectors [Host vendor specific log]
GP/S Log at address 0x90 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x91 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x92 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x93 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x94 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x95 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x96 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x97 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x98 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x99 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x9a has 16 sectors [Host vendor specific log]
GP/S Log at address 0x9b has 16 sectors [Host vendor specific log]
GP/S Log at address 0x9c has 16 sectors [Host vendor specific log]
GP/S Log at address 0x9d has 16 sectors [Host vendor specific log]
GP/S Log at address 0x9e has 16 sectors [Host vendor specific log]
GP/S Log at address 0x9f has 16 sectors [Host vendor specific log]
GP/S Log at address 0xa0 has 16 sectors [Device vendor specific log]
GP/S Log at address 0xa1 has 16 sectors [Device vendor specific log]
GP/S Log at address 0xa2 has 16 sectors [Device vendor specific log]
GP/S Log at address 0xa3 has 16 sectors [Device vendor specific log]
GP/S Log at address 0xa4 has 16 sectors [Device vendor specific log]
GP/S Log at address 0xa5 has 16 sectors [Device vendor specific log]
GP/S Log at address 0xa6 has 16 sectors [Device vendor specific log]
GP/S Log at address 0xa7 has 16 sectors [Device vendor specific log]
GP/S Log at address 0xa8 has 1 sectors [Device vendor specific log]
GP/S Log at address 0xa9 has 1 sectors [Device vendor specific log]
GP/S Log at address 0xaa has 1 sectors [Device vendor specific log]
GP/S Log at address 0xab has 1 sectors [Device vendor specific log]
GP/S Log at address 0xac has 1 sectors [Device vendor specific log]
GP/S Log at address 0xad has 1 sectors [Device vendor specific log]
GP/S Log at address 0xae has 1 sectors [Device vendor specific log]
GP/S Log at address 0xaf has 1 sectors [Device vendor specific log]
GP/S Log at address 0xb0 has 1 sectors [Device vendor specific log]
GP/S Log at address 0xb1 has 1 sectors [Device vendor specific log]
GP/S Log at address 0xb2 has 1 sectors [Device vendor specific log]
GP/S Log at address 0xb3 has 1 sectors [Device vendor specific log]
GP/S Log at address 0xb4 has 1 sectors [Device vendor specific log]
GP/S Log at address 0xb5 has 1 sectors [Device vendor specific log]
GP/S Log at address 0xb6 has 1 sectors [Device vendor specific log]
GP/S Log at address 0xb7 has 1 sectors [Device vendor specific log]
GP/S Log at address 0xbd has 1 sectors [Device vendor specific log]
GP/S Log at address 0xc0 has 1 sectors [Device vendor specific log]
GP Log at address 0xc1 has 93 sectors [Device vendor specific log]
GP/S Log at address 0xe0 has 1 sectors [SCT Command/Status]
GP/S Log at address 0xe1 has 1 sectors [SCT Data Transfer]
SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
Device Error Count: 6807 (device log contains only the most recent 24 errors)
CR = Command Register
FEATR = Features Register
COUNT = Count (was: Sector Count) Register
LBA_48 = Upper bytes of LBA High/Mid/Low Registers ] ATA-8
LH = LBA High (was: Cylinder High) Register ] LBA
LM = LBA Mid (was: Cylinder Low) Register ] Register
LL = LBA Low (was: Sector Number) Register ]
DV = Device (was: Device/Head) Register
DC = Device Control Register
ER = Error register
ST = Status register
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 6807 [14] occurred at disk power-on lifetime: 11653 hours (485 days +
13 hours)
When the command that caused the error occurred, the device was active or
idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 61 00 02 00 00 00 00 00 00 a0 00 Device Fault; Error: ABRT
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
ef 00 10 00 02 00 00 00 00 00 00 a0 08 16:03:04.798 SET FEATURES
[Reserved for Serial ATA]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 16:03:04.798 IDENTIFY DEVICE
ef 00 03 00 46 00 00 00 00 00 00 a0 08 16:03:04.798 SET FEATURES [Set
transfer mode]
ef 00 10 00 02 00 00 00 00 00 00 a0 08 16:03:04.797 SET FEATURES
[Reserved for Serial ATA]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 16:03:04.797 IDENTIFY DEVICE
Error 6806 [13] occurred at disk power-on lifetime: 11653 hours (485 days +
13 hours)
When the command that caused the error occurred, the device was active or
idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 61 00 46 00 00 00 00 00 00 a0 00 Device Fault; Error: ABRT
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
ef 00 03 00 46 00 00 00 00 00 00 a0 08 16:03:04.798 SET FEATURES [Set
transfer mode]
ef 00 10 00 02 00 00 00 00 00 00 a0 08 16:03:04.797 SET FEATURES
[Reserved for Serial ATA]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 16:03:04.797 IDENTIFY DEVICE
35 00 00 04 00 00 01 13 7f 4c 00 e0 08 16:03:04.797 WRITE DMA EXT
ef 00 10 00 02 00 00 00 00 00 00 a0 08 16:03:04.797 SET FEATURES
[Reserved for Serial ATA]
Error 6805 [12] occurred at disk power-on lifetime: 11653 hours (485 days +
13 hours)
When the command that caused the error occurred, the device was active or
idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 61 00 02 00 00 00 00 00 00 a0 00 Device Fault; Error: ABRT
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
ef 00 10 00 02 00 00 00 00 00 00 a0 08 16:03:04.797 SET FEATURES
[Reserved for Serial ATA]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 16:03:04.797 IDENTIFY DEVICE
35 00 00 04 00 00 01 13 7f 4c 00 e0 08 16:03:04.797 WRITE DMA EXT
ef 00 10 00 02 00 00 00 00 00 00 a0 08 16:03:04.797 SET FEATURES
[Reserved for Serial ATA]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 16:03:04.796 IDENTIFY DEVICE
Error 6804 [11] occurred at disk power-on lifetime: 11653 hours (485 days +
13 hours)
When the command that caused the error occurred, the device was active or
idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 61 04 00 00 01 13 7f 4c 00 e0 00 Device Fault; Error: ABRT 1024
sectors at LBA = 0x1137f4c00 = 4622076928
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
35 00 00 04 00 00 01 13 7f 4c 00 e0 08 16:03:04.797 WRITE DMA EXT
ef 00 10 00 02 00 00 00 00 00 00 a0 08 16:03:04.797 SET FEATURES
[Reserved for Serial ATA]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 16:03:04.796 IDENTIFY DEVICE
ef 00 03 00 46 00 00 00 00 00 00 a0 08 16:03:04.796 SET FEATURES [Set
transfer mode]
ef 00 10 00 02 00 00 00 00 00 00 a0 08 16:03:04.796 SET FEATURES
[Reserved for Serial ATA]
Error 6803 [10] occurred at disk power-on lifetime: 11653 hours (485 days +
13 hours)
When the command that caused the error occurred, the device was active or
idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 61 00 02 00 00 00 00 00 00 a0 00 Device Fault; Error: ABRT
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
ef 00 10 00 02 00 00 00 00 00 00 a0 08 16:03:04.797 SET FEATURES
[Reserved for Serial ATA]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 16:03:04.796 IDENTIFY DEVICE
ef 00 03 00 46 00 00 00 00 00 00 a0 08 16:03:04.796 SET FEATURES [Set
transfer mode]
ef 00 10 00 02 00 00 00 00 00 00 a0 08 16:03:04.796 SET FEATURES
[Reserved for Serial ATA]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 16:03:04.795 IDENTIFY DEVICE
Error 6802 [9] occurred at disk power-on lifetime: 11653 hours (485 days +
13 hours)
When the command that caused the error occurred, the device was active or
idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 61 00 46 00 00 00 00 00 00 a0 00 Device Fault; Error: ABRT
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
ef 00 03 00 46 00 00 00 00 00 00 a0 08 16:03:04.796 SET FEATURES [Set
transfer mode]
ef 00 10 00 02 00 00 00 00 00 00 a0 08 16:03:04.796 SET FEATURES
[Reserved for Serial ATA]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 16:03:04.795 IDENTIFY DEVICE
35 00 00 04 00 00 01 13 7f 4c 00 e0 08 16:03:04.795 WRITE DMA EXT
ef 00 10 00 02 00 00 00 00 00 00 a0 08 16:03:04.795 SET FEATURES
[Reserved for Serial ATA]
Error 6801 [8] occurred at disk power-on lifetime: 11653 hours (485 days +
13 hours)
When the command that caused the error occurred, the device was active or
idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 61 00 02 00 00 00 00 00 00 a0 00 Device Fault; Error: ABRT
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
ef 00 10 00 02 00 00 00 00 00 00 a0 08 16:03:04.796 SET FEATURES
[Reserved for Serial ATA]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 16:03:04.795 IDENTIFY DEVICE
35 00 00 04 00 00 01 13 7f 4c 00 e0 08 16:03:04.795 WRITE DMA EXT
ef 00 10 00 02 00 00 00 00 00 00 a0 08 16:03:04.795 SET FEATURES
[Reserved for Serial ATA]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 16:03:04.795 IDENTIFY DEVICE
Error 6800 [7] occurred at disk power-on lifetime: 11653 hours (485 days +
13 hours)
When the command that caused the error occurred, the device was active or
idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 61 04 00 00 01 13 7f 4c 00 e0 00 Device Fault; Error: ABRT 1024
sectors at LBA = 0x1137f4c00 = 4622076928
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
35 00 00 04 00 00 01 13 7f 4c 00 e0 08 16:03:04.795 WRITE DMA EXT
ef 00 10 00 02 00 00 00 00 00 00 a0 08 16:03:04.795 SET FEATURES
[Reserved for Serial ATA]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 16:03:04.795 IDENTIFY DEVICE
ef 00 03 00 46 00 00 00 00 00 00 a0 08 16:03:04.794 SET FEATURES [Set
transfer mode]
ef 00 10 00 02 00 00 00 00 00 00 a0 08 16:03:04.794 SET FEATURES
[Reserved for Serial ATA]
SMART Extended Self-test Log Version: 1 (1 sectors)
Num Test_Description Status Remaining LifeTime(hours)
LBA_of_first_error
# 1 Short offline Completed without error 00% 11770 -
# 2 Short offline Completed without error 00% 11757 -
# 3 Short offline Completed without error 00% 11746 -
# 4 Extended offline Completed without error 00% 11729 -
# 5 Extended offline Completed without error 00% 11698 -
# 6 Extended offline Aborted by host 90% 11677 -
# 7 Short offline Completed without error 00% 11654 -
# 8 Short offline Completed without error 00% 11565 -
# 9 Short offline Completed: read failure 70% 11557 9
#10 Short offline Completed: read failure 70% 11557 9
#11 Extended offline Completed: read failure 90% 11556 9
#12 Short offline Completed without error 00% 1367 -
#13 Short offline Completed without error 00% 1346 -
#14 Extended offline Completed without error 00% 1327 -
#15 Short offline Completed without error 00% 1295 -
#16 Short offline Completed without error 00% 1271 -
#17 Short offline Completed without error 00% 1247 -
#18 Short offline Completed without error 00% 1223 -
3 of 3 failed self-tests are outdated by newer successful extended offline
self-test # 4
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Status Version: 3
SCT Version (vendor specific): 258 (0x0102)
SCT Support Level: 1
Device State: Active (0)
Current Temperature: 29 Celsius
Power Cycle Min/Max Temperature: 22/35 Celsius
Lifetime Min/Max Temperature: 15/49 Celsius
Under/Over Temperature Limit Count: 0/0
SCT Temperature History Version: 2
Temperature Sampling Period: 1 minute
Temperature Logging Interval: 1 minute
Min/Max recommended Temperature: 0/60 Celsius
Min/Max Temperature Limit: -41/85 Celsius
Temperature History Size (Index): 478 (158)
Index Estimated Time Temperature Celsius
159 2014-11-06 10:13 29 **********
... ..( 20 skipped). .. **********
180 2014-11-06 10:34 29 **********
181 2014-11-06 10:35 31 ************
... ..(128 skipped). .. ************
310 2014-11-06 12:44 31 ************
311 2014-11-06 12:45 30 ***********
... ..( 70 skipped). .. ***********
382 2014-11-06 13:56 30 ***********
383 2014-11-06 13:57 31 ************
... ..( 82 skipped). .. ************
466 2014-11-06 15:20 31 ************
467 2014-11-06 15:21 30 ***********
... ..( 32 skipped). .. ***********
22 2014-11-06 15:54 30 ***********
23 2014-11-06 15:55 29 **********
... ..( 49 skipped). .. **********
73 2014-11-06 16:45 29 **********
74 2014-11-06 16:46 30 ***********
... ..( 2 skipped). .. ***********
77 2014-11-06 16:49 30 ***********
78 2014-11-06 16:50 29 **********
... ..( 23 skipped). .. **********
102 2014-11-06 17:14 29 **********
103 2014-11-06 17:15 30 ***********
... ..( 9 skipped). .. ***********
113 2014-11-06 17:25 30 ***********
114 2014-11-06 17:26 29 **********
... ..( 43 skipped). .. **********
158 2014-11-06 18:10 29 **********
SCT Error Recovery Control:
Read: 70 (7,0 seconds)
Write: 70 (7,0 seconds)
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 2 0 Command failed due to ICRC error
0x0002 2 0 R_ERR response for data FIS
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0005 2 0 R_ERR response for non-data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
0x0008 2 0 Device-to-host non-data FIS retries
0x0009 2 15 Transition from drive PhyRdy to drive PhyNRdy
0x000a 2 15 Device-to-host register FISes sent due to a COMRESET
0x000b 2 0 CRC errors within host-to-device FIS
0x000f 2 0 R_ERR response for host-to-device data FIS, CRC
0x0012 2 0 R_ERR response for host-to-device non-data FIS, CRC
0x8000 4 363301 Vendor specific
^ permalink raw reply
* Re: Split RAID: Proposal for archival RAID using incremental batch checksum
From: Anshuman Aggarwal @ 2014-11-06 2:29 UTC (permalink / raw)
To: Piergiorgio Sartor; +Cc: Ethan Wilson, Mdadm
In-Reply-To: <20141101125513.GB2078@lazy.lzy>
On 1 November 2014 18:25, Piergiorgio Sartor
<piergiorgio.sartor@nexgo.de> wrote:
> On Fri, Oct 31, 2014 at 04:35:11PM +0530, Anshuman Aggarwal wrote:
>> Hi pg,
>> With MD raid striping all the writes not only does it keep ALL disks
>> spinning to read/write the current content, it also leads to
>> catastrophic data loss in case the rebuild/disk failure exceeds the
>> number of parity disks.
>
> Hi Anshuman,
>
> yes but do you have hard evidence that
> this is a common RAID-6 problem?
> Considering that we have now bad block list,
> write intent bitmap and proactive replacement,
> it does not seem to me really the main issue,
> having a triple fail in RAID-6.
> Considering that there are available libraries
> for more that 2 parities, I think the multiple
> failure case is quite a rarity.
> Furthermore, I suspect there are other type
> of catastrophic situation (lighting, for example)
> that can destroy an array completely.
I have most definitely lost data when a drive fails and during
reconstruction another drive fails (remember the array has been
chugging away all drives active for 2-3 years). At this point I'm dead
scared of losing another one to avoid catastrophic. If I dont' go out
and buy a replacement right away i'm on borrowed time for my whole
array. For home use this is not fun.
>
>> But more importantly, I find myself setting up multiple RAID levels
>> (at least RAID6 and now thinking of more) just to make sure that MD
>> raid will recover my data and not lose the whole cluster if an
>> additional disk fails above the number of parity!!! The biggest
>> advantage of the scheme that I have outlined is that with a single
>> check sum I am mostly assure of a failed disk restoration and worst
>> case only the media (movies/music) on the failing disk are lost not on
>> the whole cluster.
>
> Each disk will have its own filesystem?
> If this is not the case, you cannot say
> if a single disk failure will lose only
> some files.
Indeed, each device will indeed be an independent block device and
file system. Joined together by some union FS if the user so requires
but that's not in scope for this discussion.
>
>> Also in my experience about disks and usage, while what you are saying
>> was true a while ago when storage capacity had not hit multiple TBs.
>> Now if I am buying 3-4 TB disks they are likely to last a while
>> especially since the incremental % growth in sizes seem to be slowing
>> down.
>
> As wrote above, you can safely replace
> disks before they fail, without compromising
> the array.
Same point above. For home use, I might be away or not have time to
give the array the TLC (tender loving care ;) it needs which is the
only shortcoming of MD really...its hard on the disks and has
potential of compromising the whole array (giving super fast R/W
performance in return for sure)
>
> bye,
>
> pg
>
>> Regards,
>> Anshuman
>>
>> On 30 October 2014 22:55, Piergiorgio Sartor
>> <piergiorgio.sartor@nexgo.de> wrote:
>> > On Thu, Oct 30, 2014 at 08:27:27PM +0530, Anshuman Aggarwal wrote:
>> >> What you are suggesting will work for delaying writing the checksum
>> >> (but still making 2 disks work non stop and lead to failure, cost
>> >> etc).
>> >
>> > Hi Anshuman,
>> >
>> > I'm a bit missing the point here.
>> >
>> > In my experience, with my storage systems, I change
>> > disks because they're too small, way long before they
>> > are too old (way long before they fail).
>> > That's why I end up with a collection of small HDDs.
>> > which, in turn, I recycled in some custom storage
>> > system (using disks of different size, like explained
>> > in one of the links posted before).
>> >
>> > Honestly, the only reason to spin down the disks, still
>> > in my experience, is for reducing power consumption.
>> > And this can be done with a RAID-6 without problems
>> > and in a extremely flexible way.
>> >
>> > So, the bottom line, still in my experience, is that
>> > this you're describing seems quite a nice situation.
>> >
>> > Or, I did not understood what you're proposing.
>> >
>> > Thanks,
>> >
>> > bye,
>> >
>> > pg
>> >
>> >> I am proposing N independent disks which are rarely accessed. When
>> >> parity has to be written to the remaining 1,2 ...X disks ...it is
>> >> batched up (bcache is feasible) and written out once in a while
>> >> depending on how much write is happening. N-1 disks stay spun down and
>> >> only X disks wake up periodically to get checksum written to (this
>> >> would be tweaked by the user based on how up to date he needs the
>> >> parity to be (tolerance of rebuilding parity in case of crash) and vs
>> >> disk access for each parity write)
>> >>
>> >> It can't be done using any RAID6 because RAID5/6 will stripe all the
>> >> data across the devices making any read access wake up all the
>> >> devices. Ditto for writing to parity on every write to a single disk.
>> >>
>> >> The architecture being proposed is a lazy write to manage parity for
>> >> individual disks which won't suffer from RAID catastrophic data loss
>> >> and concurrent disk.
>> >>
>> >>
>> >>
>> >>
>> >> On 30 October 2014 00:57, Ethan Wilson <ethan.wilson@shiftmail.org> wrote:
>> >> > On 29/10/2014 10:25, Anshuman Aggarwal wrote:
>> >> >>
>> >> >> Right on most counts but please see comments below.
>> >> >>
>> >> >> On 29 October 2014 14:35, NeilBrown <neilb@suse.de> wrote:
>> >> >>>
>> >> >>> Just to be sure I understand, you would have N + X devices. Each of the
>> >> >>> N
>> >> >>> devices contains an independent filesystem and could be accessed directly
>> >> >>> if
>> >> >>> needed. Each of the X devices contains some codes so that if at most X
>> >> >>> devices in total died, you would still be able to recover all of the
>> >> >>> data.
>> >> >>> If more than X devices failed, you would still get complete data from the
>> >> >>> working devices.
>> >> >>>
>> >> >>> Every update would only write to the particular N device on which it is
>> >> >>> relevant, and all of the X devices. So N needs to be quite a bit bigger
>> >> >>> than X for the spin-down to be really worth it.
>> >> >>>
>> >> >>> Am I right so far?
>> >> >>
>> >> >> Perfectly right so far. I typically have a N to X ratio of 4 (4
>> >> >> devices to 1 data) so spin down is totally worth it for data
>> >> >> protection but more on that below.
>> >> >>
>> >> >>> For some reason the writes to X are delayed... I don't really understand
>> >> >>> that part.
>> >> >>
>> >> >> This delay is basically designed around archival devices which are
>> >> >> rarely read from and even more rarely written to. By delaying writes
>> >> >> on 2 criteria ( designated cache buffer filling up or preset time
>> >> >> duration from last write expiring) we can significantly reduce the
>> >> >> writes on the parity device. This assumes that we are ok to lose a
>> >> >> movie or two in case the parity disk is not totally up to date but are
>> >> >> more interested in device longevity.
>> >> >>
>> >> >>> Sounds like multi-parity RAID6 with no parity rotation and
>> >> >>> chunksize == devicesize
>> >> >>
>> >> >> RAID6 would present us with a joint device and currently only allows
>> >> >> writes to that directly, yes? Any writes will be striped.
>> >> >
>> >> >
>> >> > I am not totally sure I understand your design, but it seems to me that the
>> >> > following solution could work for you:
>> >> >
>> >> > MD raid-6, maybe multi-parity (multi-parity not implemented yet in MD yet,
>> >> > but just do a periodic scrub and 2 parities can be fine. Wake-up is not so
>> >> > expensive that you can't scrub)
>> >> >
>> >> > Over that you put a raid1 of 2 x 4TB disks as a bcache cache device (those
>> >> > two will never spin-down) in writeback mode with writeback_running=off .
>> >> > This will prevent writes to backend and leave the backend array spun down.
>> >> > When bcache is almost full (poll dirty_data), switch to writeback_running=on
>> >> > and writethrough: it will wake up the backend raid6 array and flush all
>> >> > dirty data. You can then then revert to writeback and writeback_running=off.
>> >> > After this you can spin-down the backend array again.
>> >> >
>> >> > You also get read caching for free, which helps the backend array to stay
>> >> > spun down as much as possible.
>> >> >
>> >> > Maybe you can modify bcache slightly so to implement an automatic switching
>> >> > between the modes as described above, instead of polling the state from
>> >> > outside.
>> >> >
>> >> > Would that work, or you are asking something different?
>> >> >
>> >> > EW
>> >> >
>> >> > --
>> >> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> >> > the body of a message to majordomo@vger.kernel.org
>> >> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>> >> --
>> >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> >> the body of a message to majordomo@vger.kernel.org
>> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
>> >
>> > --
>> >
>> > piergiorgio
>
> --
>
> piergiorgio
^ permalink raw reply
* Re: Split RAID: Proposal for archival RAID using incremental batch checksum
From: Anshuman Aggarwal @ 2014-11-06 2:24 UTC (permalink / raw)
To: NeilBrown; +Cc: Mdadm
In-Reply-To: <20141103165217.3bfd3d3e@notabene.brown>
Pls see below
On 3 November 2014 11:22, NeilBrown <neilb@suse.de> wrote:
> On Thu, 30 Oct 2014 20:30:40 +0530 Anshuman Aggarwal
> <anshuman.aggarwal@gmail.com> wrote:
>
>> Would chunksize==disksize work? Wouldn't that lead to the entire
>> parity be invalidated for any write to any of the disks (assuming md
>> operates at a chunk level)...also please see my reply below
>
> Operating at a chunk level would be a very poor design choice. md/raid5
> operates in units of 1 page (4K).
>
>
>>
>> On 29 October 2014 14:55, Anshuman Aggarwal <anshuman.aggarwal@gmail.com> wrote:
>> > Right on most counts but please see comments below.
>> >
>> > On 29 October 2014 14:35, NeilBrown <neilb@suse.de> wrote:
>> >> Just to be sure I understand, you would have N + X devices. Each of the N
>> >> devices contains an independent filesystem and could be accessed directly if
>> >> needed. Each of the X devices contains some codes so that if at most X
>> >> devices in total died, you would still be able to recover all of the data.
>> >> If more than X devices failed, you would still get complete data from the
>> >> working devices.
>> >>
>> >> Every update would only write to the particular N device on which it is
>> >> relevant, and all of the X devices. So N needs to be quite a bit bigger
>> >> than X for the spin-down to be really worth it.
>> >>
>> >> Am I right so far?
>> >
>> > Perfectly right so far. I typically have a N to X ratio of 4 (4
>> > devices to 1 data) so spin down is totally worth it for data
>> > protection but more on that below.
>> >
>> >>
>> >> For some reason the writes to X are delayed... I don't really understand
>> >> that part.
>> >
>> > This delay is basically designed around archival devices which are
>> > rarely read from and even more rarely written to. By delaying writes
>> > on 2 criteria ( designated cache buffer filling up or preset time
>> > duration from last write expiring) we can significantly reduce the
>> > writes on the parity device. This assumes that we are ok to lose a
>> > movie or two in case the parity disk is not totally up to date but are
>> > more interested in device longevity.
>> >
>> >>
>> >> Sounds like multi-parity RAID6 with no parity rotation and
>> >> chunksize == devicesize
>> > RAID6 would present us with a joint device and currently only allows
>> > writes to that directly, yes? Any writes will be striped.
>
> If the chunksize equals the device size, then you need a very large write for
> it to be striped.
>
>> > In any case would md raid allow the underlying device to be written to
>> > directly? Also how would it know that the device has been written to
>> > and hence parity has to be updated? What about the superblock which
>> > the FS would not know about?
>
> No, you wouldn't write to the underlying device. You would carefully
> partition the RAID5 so each partition aligns exactly with an underlying
> device. Then write to the partition.
This is what I'm unclear about. Even with non rotating parity on RAID
5/6 is it possible to create md partitions such that the writes are
effectively not striped (within each partition) and that each
partition on the md device ends up writing only to that one device?
How is this managed? My understanding is that raid5/6 will stripe any
data blocks across all the devices making all of them spin up for each
read and write.
>
>> >
>> > Also except for the delayed checksum writing part which would be
>> > significant if one of the objectives is to reduce the amount of
>> > writes. Can we delay that in the code currently for RAID6? I
>> > understand the objective of RAID6 is to ensure data recovery and we
>> > are looking at a compromise in this case.
>
> "simple matter of programming"
> Of course there would be a limit to how much data can be buffered in memory
> before it has to be flushed out.
> If you are mostly storing movies, then they are probably too large to
> buffer. Why not just write them out straight away?
Well, yeah if the buffer gets filled (such as by a movie) the parity
will get written pretty much write away (the main data drive gets
written to immediately anyways). The delay is to prevent parity drive
spin ups due to a small updates on any one of the drives in the array.
Maybe a small temp file created by a software etc.
>
> NeilBrown
>
>
>
>> >
>> > If feasible, this can be an enhancement to MD RAID as well where N
>> > devices are presented instead of a single joint device in case of
>> > raid6 (maybe the multi part device can be individual disks?)
>> >
>> > It will certainly solve my problem of where to store the metadata. I
>> > was currently hoping to just store it as a configuration file to be
>> > read by the initramfs since in this case worst case scenario the
>> > checksum goes out of sync and is rebuilt from scratch.
>> >
>> >>
>> >> I wouldn't use device-mapper myself, but you are unlikely to get an entirely
>> >> impartial opinion from me on that topic.
>> >
>> > I haven't hacked around the kernel internals much so far so will have
>> > to dig out that history. I will welcome any particular links/mail
>> > threads I should look at for guidance (with both yours and opposing
>> > points of view)
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply
* Re: Raid5 drive fail during grow and no backup
From: Vince @ 2014-11-05 19:03 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <5458FC2A.1050308@turmel.org>
Hi phil,
I own the WD Red standart drives.
Thx for your advice on fixin it.
I'll investigate some time into solving the problem.
Think it will happen again, as for now it happends twice ;)
^ permalink raw reply
* Re: Trying to get POLICY working
From: NeilBrown @ 2014-11-05 5:28 UTC (permalink / raw)
To: Caspar Smit; +Cc: linux-raid
In-Reply-To: <CADvHm=bE0zcLbDhcdQFfQG8in1Udb7gWcvR-rb9JeUoFaV-v5w@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 4286 bytes --]
On Mon, 3 Nov 2014 10:43:29 +0100 Caspar Smit <c.smit@truebit.nl> wrote:
> Hi Neil,
>
> Actually BOTH your answers were correct, thank you for that.
>
> 1) Your hunge was correct as my disk contained a partition table (in
> my case an msdos label) and was not added with the error in my first
> mail:
>
> mdadm: no RAID superblock on /dev/sdd.
>
> mdadm -E /dev/sdd shows:
>
> /dev/sdd:
> MBR Magic : aa55
>
> So it finds 'something' but clearly unusable to mdadm.
>
> Wiping the partition table and trying again resulted in a different
> error message:
>
> mdadm: no recognisable superblock on /dev/sdd.
>
> Which is better but still the disk was not added to the array.
>
> 2) To make it work i also needed the domain=default in the POLICY setting.
>
> It still gave me the:
>
> mdadm: no recognisable superblock on /dev/sdd.
>
> But now the disk got added to the array and started rebuilding.
>
> Note: ONLY setting the domain=default in POLICY without clearing the
> partition table results in:
> mdadm: no RAID superblock on /dev/sdd. and the disk will not be added
> so BOTH measures were needed.
Thanks for testing and reported.... the patch I posted before (included more
completely below) should allow "domain=default" to be enough.
>
> Note2: I didn't need the spare-group directive so I think
> domain=default is a special case were all disks and arrays are placed
> in the same domain.
"spare-group" is really only for "legacy" support. If a domain is defined
for disks, the array made up of those disks inherits the domain.
>
>
> Furthermore i found out something which i think should not happen
> (bug?) or maybe i am wrong:
>
> With a working clean array:
>
> # more /proc/mdstat
> Personalities : [raid6] [raid5] [raid4]
> md0 : active raid5 sdd[3] sdc[1] sdb[0]
> 203776 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
>
> # mdadm --fail /dev/md0 /dev/sdd
> mdadm: set /dev/sdd faulty in /dev/md0
>
> # mdadm --remove /dev/md0 /dev/sdd
> mdadm: hot removed /dev/sdd from /dev/md0
>
> # mdadm --incremental /dev/sdd
> mdadm: failed to add /dev/sdd to /dev/md/0: Invalid argument.
>
> So when it actually finds a device with an MD superblock it doesn't
> add it, is this expected behavior as the disk was failed (so probably
> not a good idea to add it back) or is this a bug?
Presumably "action=force-spare" was still active when you tried this?
In that case it is a bug (I think). It should clean-out the device and add
it as a spare...
I just tested with mdadm from my 'git', and it works as expected.
When action=force-spare I get
mdadm: /dev/loop2 attached to /dev/md0 which is already active.
When I have "action=re-add" I get:
mdadm: can only add /dev/loop2 to /dev/md0 as a spare, and force-spare is not set.
mdadm: failed to add /dev/loop2 to existing array /dev/md0: Invalid argument.
Maybe you need a newer mdadm ...
Thanks,
NeilBrown
From: NeilBrown <neilb@suse.de>
Date: Wed, 5 Nov 2014 16:21:42 +1100
Subject: [PATCH] Incremental: don't be distracted by partition table when
calling try_spare.
Currently a partition table on a device makes "mdadm -I" think
the array has a particular metadata type and so will only
add it to an array of that (partition table) type .. which doesn't
make any sense.
So tell guess_super to only look for 'array' metadata.
Reported-by: Caspar Smit <c.smit@truebit.nl>
Signed-off-by: NeilBrown <neilb@suse.de>
diff --git a/Incremental.c b/Incremental.c
index c9372587f518..13b68bc0adea 100644
--- a/Incremental.c
+++ b/Incremental.c
@@ -196,13 +196,13 @@ int Incremental(struct mddev_dev *devlist, struct context *c,
policy = disk_policy(&dinfo);
have_target = policy_check_path(&dinfo, &target_array);
- if (st == NULL && (st = guess_super(dfd)) == NULL) {
+ if (st == NULL && (st = guess_super_type(dfd, guess_array)) == NULL) {
if (c->verbose >= 0)
pr_err("no recognisable superblock on %s.\n",
devname);
rv = try_spare(devname, &dfd, policy,
have_target ? &target_array : NULL,
- st, c->verbose);
+ NULL, c->verbose);
goto out;
}
st->ignore_hw_compat = 1;
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply related
* Re: mdadm named array (metadata 1.2) and udev when assembling an array in initramfs
From: Stephen Kent @ 2014-11-04 18:17 UTC (permalink / raw)
To: Michael Tokarev; +Cc: linux-raid
In-Reply-To: <5458970B.3050305@msgid.tls.msk.ru>
On Tue, Nov 4, 2014 at 1:06 AM, Michael Tokarev <mjt@tls.msk.ru> wrote:
>
> 01.11.2014 21:53, Stephen Kent wrote:
> > I have set up a machine to use an md array as the root device. I have
> > created the array with metadata type 1.2, name "root," on host
> > "optimus" (so the full metadata name is "optimus:root").
> >
> > I'm using Gentoo Linux (kernel 3.16.5-gentoo), and previously I had no
> > initramfs so I wrote a short one to assemble the array. My initramfs
> > sets the hostname and runs "mdadm -As," which correctly auto-assembles
> > the root device array on /dev/md/root (even without an
> > /etc/mdadm.conf).
> >
> > Once the initramfs script is finished, init starts which then starts
> > udev. Now, at this point the array is already assembled, but the
> > /dev/md/* name is lost (and so is /run/mdadm/map) since those files
> > were on the initramfs' tempfs root.
>
> This is your problem. You should not lose neither /dev nor /run
> when switching root. Usual thing to do is to mount tmpfs /dev and
> tmpfs /run in initramfs and do mount --move /dev /root/dev (and
> the same for /run) before switching root.
>
> Thanks,
>
> /mjt
Hm, interesting. I didn't realize a mount point could be moved, that's
good to know.
My initramfs is a short custom shell script that I wrote, basically
all it does is assemble the array and mount the root filesystem. I'm
not mounting a tmpfs and/or udev on /run and /dev, so unfortunately
there aren't any mount points to move to the target root mountpoint at
the end of init. /run and /dev are only mounted on my machine after
leaving the initramfs.
Fortunately, NeilBrown's patch works for me which solves the problem
and allows mdadm to work correctly with a minimal initramfs.
Stephen
^ permalink raw reply
* Re: Raid5 drive fail during grow and no backup
From: Phil Turmel @ 2014-11-04 16:17 UTC (permalink / raw)
To: Vince, linux-raid
In-Reply-To: <loom.20141103T151703-83@post.gmane.org>
Hi Vince,
On 11/03/2014 09:45 AM, Vince wrote:
> Phil Turmel <philip <at> turmel.org> writes:
[trim /]
>> You haven't (yet) lost your array. It's just degraded. You should
>> investigate why the one drive was kicked out of the array instead of
>> being rewritten properly (green drives?). In the meantime, assembly
>> with --force should give you access to the data to grab anything
>> critically important.
[trim /]
> Hi Phil,
>
> thx for your reply.
> Already have the raid clean and up.
Very good to hear you haven't lost your data.
> My drive was kicked due to read errors (bad sectors).
> I fixed the bad sectors with hdparm --write-sector $bad_sector /dev/sdx
This is a problem you haven't solved yet, I think. The raid array
should have fixed this bad sector for you without kicking the drive out.
The scenario is common with "green" drives and/or consumer-grade
drives in general.
If you want to be sure your array is safe for the future, you should
search this list's archives for "timeout mismatch", "scterc", and/or
"URE". Then you can set up your array to properly correct bad sectors,
and set your system to look for bad sectors on a regular basis.
Phil
^ permalink raw reply
* Re: "bitmap file is out of date, doing full recovery"
From: Alexander Lyakas @ 2014-11-04 9:17 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20141029101950.5528b347@notabene.brown>
Hi Neil,
thank you for your comments.
On Wed, Oct 29, 2014 at 1:19 AM, NeilBrown <neilb@suse.de> wrote:
> On Thu, 23 Oct 2014 19:04:48 +0300 Alexander Lyakas <alex.bolshoy@gmail.com>
> wrote:
>
>> Hi Neil,
>> I found at least one way of this happening. The problem is that in
>> md_update_sb() we allow to decrease the event count:
>>
>> /* If this is just a dirty<->clean transition, and the array is clean
>> * and 'events' is odd, we can roll back to the previous clean state */
>> if (nospares
>> && (mddev->in_sync && mddev->recovery_cp == MaxSector)
>> && mddev->can_decrease_events
>> && mddev->events != 1) {
>> mddev->events--;
>> mddev->can_decrease_events = 0;
>>
>> Then we call bitmap_update_sb(). If we crash after we update (the
>> first or all of) bitmap superblocks, then after reboot, we will see
>> that bitmap event count is less than MD superblock event count. Then
>> we decide to do full resync.
>>
>> This can be easily reproduced by hacking bitmap_update_sb() to call
>> BUG(), after it calls write_page() in case event count was decreased.
>>
>> Why we are decreasing the event count??? Can we always increase it?
>> u64 is a lot to increase...
>
> The reason for decreasing the event count is so that we don't need to update
> the event count on spares - they can be left spun down.
> We for simple clean/dirty transitions with increment for clean->dirty and
> decrement for dirty->clean. But we should only use this optimisation when
> everything is simple.
> We really shouldn't do this when the array is degraded.
> Do this fix your problem?
>
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index 2c73fcb82593..98fd97b10e13 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -2244,6 +2244,7 @@ repeat:
> * and 'events' is odd, we can roll back to the previous clean state */
> if (nospares
> && (mddev->in_sync && mddev->recovery_cp == MaxSector)
> + && mddev->degraded == 0
> && mddev->can_decrease_events
> && mddev->events != 1) {
> mddev->events--;
>
>
No, unfortunately, this doesn't fix the problem. In my case, the array
is never degraded. Both drives are present and operational, then the
box crashes, and after reboot the bitmap event counter is lower than
we expect. Again, this is easily reproduced by hacking
bitmap_update_sb() as I mentioned earlier.
In my case array does not have spares. (There is some other system on
top, which monitors the array, and, if needed, adds a spare from a
"global" spare pool). Is this ok in this case to always increment the
event count?
Thanks,
Alex.
>>
>> Some other doubt that I have is that bitmap_unplug() and
>> bitmap_daemon_work() call write_page() on page index=0. This page
>> contains both the superblock and also some dirty bits (could not we
>> waste 4KB on bitmap superblock???). I am not sure, but I wonder
>> whether this call can race with md_update_sb (which explicitly calls
>> bitmap_update_sb), and somehow write the outdated superblock, after
>> bitmap_update_sb has completed writing it.
>>
>
> storage.sb_page is exactly the same as storage.filemap[0]
> So once an update has happened, the "outdated superblock" doesn't exist
> anywhere to be written out from.
>
>> Yet another suspect is when loading the bitmap we basically load it
>> from the first up-to-date drive. Maybe we should have scanned all the
>> bitmap superblocks, and selected one that has the higher event count
>> (although as we saw "higher" does not necessarily mean "more
>> up-to-date").
>>
>> Anyways, back to decrementing the event count. Do you see any issue
>> with not doing this and always incrementing?
>>
>> Thanks,
>> Alex.
>>
>
> Thanks,
> NeilBrown
^ permalink raw reply
* Re: mdadm named array (metadata 1.2) and udev when assembling an array in initramfs
From: Michael Tokarev @ 2014-11-04 9:06 UTC (permalink / raw)
To: Stephen Kent, linux-raid
In-Reply-To: <CAC45b7=-dZX-kxDZY3KBEPZhaTuUhTcqNjTL6y8X71q9oDjiwQ@mail.gmail.com>
01.11.2014 21:53, Stephen Kent wrote:
> I have set up a machine to use an md array as the root device. I have
> created the array with metadata type 1.2, name "root," on host
> "optimus" (so the full metadata name is "optimus:root").
>
> I'm using Gentoo Linux (kernel 3.16.5-gentoo), and previously I had no
> initramfs so I wrote a short one to assemble the array. My initramfs
> sets the hostname and runs "mdadm -As," which correctly auto-assembles
> the root device array on /dev/md/root (even without an
> /etc/mdadm.conf).
>
> Once the initramfs script is finished, init starts which then starts
> udev. Now, at this point the array is already assembled, but the
> /dev/md/* name is lost (and so is /run/mdadm/map) since those files
> were on the initramfs' tempfs root.
This is your problem. You should not lose neither /dev nor /run
when switching root. Usual thing to do is to mount tmpfs /dev and
tmpfs /run in initramfs and do mount --move /dev /root/dev (and
the same for /run) before switching root.
Thanks,
/mjt
> udev creates a /dev/mdXXX node
> (ex. /dev/md127) and then runs a rule to try to figure out the
> /dev/md/* name, which should be /dev/md/root. The default udev rule
> calls mdadm --detail --export /dev/md127, which should print
> MD_DEVNAME=root for udev, but instead it prints
> MD_DEVNAME=optimus:root, and the symlink in /dev/md gets created as
> /dev/md/optimus:root.
>
> I think this is a bug in mdadm, but I'm not sure how to dig into it. I
> have made sure that my main OS init sets the hostname before starting
> udev, and I have tried with and without "HOMEHOST optimus" in
> /etc/mdadm.conf.
>
> The workaround I am using right now is to define the array name
> manually in /etc/mdadm.conf like this:
>
> ARRAY /dev/md/root UUID=782a671c:e646f2d5:848baad0:5668ee84
>
> This works by avoiding automatic name detection and using the defined
> name (/dev/md/root), but it really seems that mdadm should know how to
> determine MD_DEVNAME correctly for an already-started array if the
> hostname (or homehost) is correctly set. Is this a bug in mdadm? If
> so, is this the correct place to report it or is there a bug tracking
> system I should use? I am using mdadm v3.3.1.
>
> (Please cc me on replies as I am not currently subscribed to the
> linux-raid list.)
>
> Thanks,
>
> Stephen
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply
* Maintenance Checklist (sanity check request)
From: Barrett Lewis @ 2014-11-03 22:45 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
I am trying to put together a thorough(ish) checklist of things that I
can run through every few months to be sure everything is all good
with my array. I'm interested in any suggestions about what I have as
well as anything else I should include for catching typical problems
of hobby level users with consumer grade hardware and just the kinds
of things you typically ask people who come on the list with crises.
1. Which disks are part of the array? Is everything active?
$ cat /proc/mdstat
2. # for x in /dev/sd? ; do echo "Checking $x" ; smartctl -x $x |
grep "SCT Error Recovery Control " ; done
a. Which disks support ERC?
b. using "# smartctl -x /dev/sdX" on the devices supporing ERC, is ERC enabled?
c. if supported but Disabled, enable ERC with "# smartctl -l
scterc,70,70 /dev/sdX"
d. Which disks to not support ERC?
e. "$ for x in /sys/block/sd*/device/timeout ; do echo $x $(< $x) ;
done" What are the timeouts set to for each drive from d?
f. if any drive which does not support ERC has a default timeout, set
it to 180 with "# echo 180 >/sys/block/sdX/device/timeout"
g. Are any disks which support ERC using timeouts greater than the
default? (they should not be)
3. # for x in /dev/sd? ; do echo "Checking $x" ; smartctl -x $x | grep
-e "Reallocated_Sector_Ct" -e "Current_Pending_Sector" ; done
Note which drives have nonzero reallocations or pending sectors. Any
drive with double digits of reallocations should be replaced.
4. Run a repair to resync the array "# echo repair >
/sys/block/md0/md/sync_action"
a. did it complete without crashing?
b. how long did it take?
c. what is the value of /sys/block/md0/md/mismatch_cnt after the
repair was performed?
5. Scan the filesystem.
sudo service nfs-kernel-server stop
sudo umount /dev/md0 (if this fails, "sudo lsof | grep vault", then
"sudo kill -KILL [pid]" of any processes using it and try again)
a. sudo e2fsck -y /dev/md0 is it clean?
b. sudo e2fsck -fy /dev/md0 any other errors on a full scan?
Thanks for your suggestions
^ permalink raw reply
* Re: [PATCH] mdadm: fix --detail for cases where device count > max_devices/2
From: NeilBrown @ 2014-11-03 22:33 UTC (permalink / raw)
To: Or Sagi; +Cc: linux-raid
In-Reply-To: <CCCA52F1-B747-4762-8D07-12B543996470@reduxio.com>
[-- Attachment #1: Type: text/plain, Size: 2533 bytes --]
On Fri, 31 Oct 2014 11:10:33 +0200 Or Sagi <ors@reduxio.com> wrote:
> mdadm misreports (less devices then there are, and as a result decides the array is degraded) in cases where there are > max_devices/2 devices.
>
> This seems to fix it.
>
> ---
> Detail.c | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/Detail.c b/Detail.c
> index c4fcad9..62e5867 100644
> --- a/Detail.c
> +++ b/Detail.c
> @@ -72,7 +72,7 @@ int Detail(char *dev, struct context *c)
> int failed = 0;
> struct supertype *st;
> char *subarray = NULL;
> - int max_disks = MD_SB_DISKS; /* just a default */
> + int max_disks = MD_SB_DISKS * 2; /* just a default */
> struct mdinfo *info = NULL;
> struct mdinfo *sra;
> struct mdinfo *subdev;
> @@ -124,7 +124,7 @@ int Detail(char *dev, struct context *c)
> rv = 0;
>
> if (st)
> - max_disks = st->max_devs;
> + max_disks = st->max_devs * 2;
>
> if (subarray) {
> /* This is a subarray of some container.
Thanks for the report, but this patch is too simplistic.
Setting "max_disks" to twice the maximum is clearly wrong.
Then it should be called twice_max_disks.
The problem is that the 'disks' array has two entries for each 'raid_disk',
the primary disk and a possible replacement. So we need to check that all
references to the size of the array, or indices to it, are treated properly.
I think this patch should fix it. Can you test please?
Thanks,
NeilBrown
diff --git a/Detail.c b/Detail.c
index c4fcad9620ba..dd72eded995d 100644
--- a/Detail.c
+++ b/Detail.c
@@ -295,8 +295,8 @@ int Detail(char *dev, struct context *c)
goto out;
}
- disks = xmalloc(max_disks * sizeof(mdu_disk_info_t));
- for (d = 0; d < max_disks; d++) {
+ disks = xmalloc(max_disks * 2 * sizeof(mdu_disk_info_t));
+ for (d = 0; d < max_disks * 2; d++) {
disks[d].state = (1<<MD_DISK_REMOVED);
disks[d].major = disks[d].minor = 0;
disks[d].number = disks[d].raid_disk = d;
@@ -327,7 +327,7 @@ int Detail(char *dev, struct context *c)
else if (disk.raid_disk >= 0 && disk.raid_disk < array.raid_disks
&& disks[disk.raid_disk*2+1].state == (1<<MD_DISK_REMOVED))
disks[disk.raid_disk*2+1] = disk;
- else if (next < max_disks)
+ else if (next < max_disks*2)
disks[next++] = disk;
}
@@ -602,7 +602,7 @@ This is pretty boring
}
free(info);
- for (d= 0; d < max_disks; d++) {
+ for (d= 0; d < max_disks * 2; d++) {
char *dv;
mdu_disk_info_t disk = disks[d];
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply related
* Re: Fake RAID 10 on ASRock Z97 Extreme3 with Slackware 14.1
From: NeilBrown @ 2014-11-03 21:57 UTC (permalink / raw)
To: gornea; +Cc: linux-raid
In-Reply-To: <14976849ec4.ace7bf4939116.3019157828391505014@zoho.com>
[-- Attachment #1: Type: text/plain, Size: 2157 bytes --]
On Mon, 03 Nov 2014 17:47:48 +0100 gornea <gornea@zoho.com> wrote:
>
> Hello,
>
> I hope I'm on the right email list. I'm facing a problem during Slackware 14.1 installation. I have a ASRock motherboard (Z97 Extreme3) with Intel Rapid Storage Technology 13. I have 4 identical new 4-TBy WD hard drives. I configured the option ROM for RAID 10. When booting with Slackware 14.1 CD md finds the array but it always mounts it read only. I tried to work on it to make writable but I was not successful. I have tried to boot with the latest Ubuntu Server CD and it works well. I have also tried to delete the RAID 10 in the option ROM and then create it with mdadm after booting with Slackware 14.1 and I always produces a read only volume. I get no errors or warnings and in general everything seems fine except that I can not write on it (for example I can not create partitions). The md version with Slackware 14.1 boot disk is 3.2.6 and the kernel is 3.10.17. So I'm starting to wonder if actually there is a know problem with that md/kernel? Or maybe there is a problem with the way I use it, but frankly I run out of ideas how to proceed further. This is the output of /proc/mdstat
>
> Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
> md126 : active (read-only) raid10 sda[3] sdb[2] sdc[1] sdd[0]
> 7814032384 blocks super external:/md127/0 64K chunks 2 near-copies [4/4] [UUUU]
>
> md127 : inactive sda[3](S) sdb[2](S) sdc[1](S) sdd[0](S)
> 9040 blocks super external:imsm
>
> unused devices: <none>
>
> Thanks a lot for any suggestion about how to proceed! I would like to install the fake RAID if it is not impossible with Slackware 14.1. I know that eventually it is possible to install a software RAID but I would like to understand why the Intel fake RAID is not working with Slackware 14.1.
>
More likely a problem with Slackware than with the kernel/mdadm version.
It should a lot like "mdmon" isn't running.
If you simple run
mdmon md127 &
you will probably get write access.
Mdadm should do this for you, or tell systemd to, depending on configuration.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* Re: Split RAID: Proposal for archival RAID using incremental batch checksum
From: Piergiorgio Sartor @ 2014-11-03 18:04 UTC (permalink / raw)
To: NeilBrown; +Cc: Anshuman Aggarwal, linux-raid
In-Reply-To: <20141103165217.3bfd3d3e@notabene.brown>
On Mon, Nov 03, 2014 at 04:52:17PM +1100, NeilBrown wrote:
[...]
> "simple matter of programming"
> Of course there would be a limit to how much data can be buffered in memory
> before it has to be flushed out.
> If you are mostly storing movies, then they are probably too large to
> buffer. Why not just write them out straight away?
One scenario I can envision is the following.
You've a bunch of HDDs in RAID-5/6, which are
almost always in standby (spin down).
Together, you've 2 SSDs in RAID-10.
All the write (and read, if possible) operations
are done towards the SSDs.
When the SSD RAID is X% full, the RAID-5/6 is
activated and the data *moved* (maybe copied, with
proper cache policy) there.
In case of reading (a large file), the RAID-5/6 is
activated, the file copied to the SSD RAID, and,
when finished, the HDDs put in standby again.
Of course, this is *not* a block device protocol,
it is a filesystem one.
It is the FS that must handle the caching, because
only the FS can know the file size, for example.
bye,
--
piergiorgio
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox