* What is the correct way of making image copy of a member disk in md array?
From: Ram Ramesh @ 2014-11-09 1:44 UTC (permalink / raw)
To: Linux Raid
I have a situation with one of my disks in /dev/md0. It does not play
well in initial hardware detection after bootup. It works very reliably
once it is detected. So I like to keep its image ready should it
actually fail. If I add a spare to md0, this flaky disk gets kicked out
immediately and I do not want that to happen.
Following is a possible solution
1. Add the new disk as spare (call it sdX).
2. echo want_replacement on the flaky drive (call it sdY)
3. When the copy completes and flaky drive becomes spare repeat the
process in the opposite direction
4. Remove the new disk from md0 when it becomes spare.
Specifically, my questions are
1. Is it possible to -re-add sdX after step 4?
2. Is there a recommended/better solution for making image other than
the one given above? If not,
3. Can I skip steps 3 & 4 and simply re-add the flaky drive after step2
completes ? I mean mdadm --remove sdX and --re-add sdY.
4. If above step works, will there be a problem when I reboot with both
sdX and sdY present?
5. If a reboot is a no-no, is there another method available to keep
the new disk (sdX) plugged in without being part of md0? I mean not
even as a spare.
The reason for all this jumping through the hoops is that I want to keep
only one spare across multiple md devices across multiple machines. I
want it to be ready for the most likely job this spare will be needed
(based on my hunch).
Kernel: Linux xxxxx 3.14-0.bpo.2-amd64 #1 SMP Debian 3.14.13-2~bpo70+1
(2014-07-31) x86_64 GNU/Linux
mdadm: mdadm - v3.2.5 - 18th May 2012
Ramesh
^ permalink raw reply
* md with shared disks
From: Anton Ekermans @ 2014-11-09 8:30 UTC (permalink / raw)
To: linux-raid
Good day raiders,
I have a question on md that I cannot find (up to date) answer to.
We use SuperMicro server with 16 shared disks on a shared backplane
between two motherboards, running up to date CentOS7.
If I create an array on one node, the other node can detect it. I put
GFS2 on top of the array so both system can share the filesystem, but I
want to know if md raid is safe to be used in this way with possibly 2
active/active nodes changing the metadata at the same time. I've
disabled raid-check cron job on one node so they don't both resync the
drives weekly, but I suspect there's a lot more to it than that.
If it's not possible, then alternatively some advice on strategy to have
a large active/active shared disk/filesystem would also be welcome.
Best regards
Untitled Document
Anton Ekermans
Technical/R&D
E-mail: antone true co za
Tel: 042 293 4168 Fax: 042 293 1851
Web: www.true.co.za <http://www.true.co.za>
^ permalink raw reply
* Re: --no-degraded does not work
From: P. Gautschi @ 2014-11-10 3:13 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20141108102247.687b8863@notabene.brown>
> Do you have a particular goal, or were you just making sure you understood?
Both. My goal is to keep my data a safe as possible in case of a 2 and more disk failure
for non permanent failures. During experiments I has problems with a power cable
and I was not able to recover from this situation. (3 Disk ok, 2 that failed after
each other)
I therefor would not like to start the array when not all disk are ok.
I created a script that checks in readonly mode first.
On 2014-11-08 00:22, NeilBrown wrote:
> On Fri, 07 Nov 2014 17:43:07 +0100 "P. Gautschi"<linuxlist@gautschi.net>
> wrote:
>
>> As far as I understand the documentation --assemble --no-degraded should not start a degraded array.
>> However on my system (kubuntu 14.10)
>>
>> # mdadm --assemble --no-degraded /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1
>> mdadm: /dev/md0 has been started with 4 drives (out of 5).
>>
>> # mdadm --detail /dev/md0
>> /dev/md0:
>> Version : 1.2
>> Creation Time : Tue Nov 4 15:26:46 2014
>> Raid Level : raid5
>> Array Size : 599469328 (571.70 GiB 613.86 GB)
>> Used Dev Size : 149867332 (142.92 GiB 153.46 GB)
>> Raid Devices : 5
>> Total Devices : 4
>> Persistence : Superblock is persistent
>>
>> Intent Bitmap : Internal
>>
>> Update Time : Fri Nov 7 17:22:53 2014
>> State : clean, degraded
>> Active Devices : 4
>> Working Devices : 4
>> Failed Devices : 0
>> Spare Devices : 0
>>
>> Layout : left-symmetric
>> Chunk Size : 4K
>>
>> Name : 0
>> UUID : c7465b19:c149b2d1:5b4d88ce:8c6ce432
>> Events : 642
>>
>> Number Major Minor RaidDevice State
>> 0 0 0 0 removed
>> 1 8 33 1 active sync /dev/sdc1
>> 2 8 49 2 active sync /dev/sdd1
>> 3 8 65 3 active sync /dev/sde1
>> 5 8 81 4 active sync /dev/sdf1
>>
>> the array IS started when removing one disk, stopping it, reconnecting the disk and then assemble the array.
>> Is this the supposed behavior?
>
> Yes, that is the correct behaviour, though I admit that it is slightly
> unintuitive.
>
> --no-degraded will cause mdadm to refuse to assemble an array which is more
> degraded than it was last time it was active.
>
> So if you have an optimal array, stop it, then try to assemble with some
> devices missing, then --no-degraded will cause that to fail.
>
> If the array is already degraded, then there doesn't seem much point in
> stopping it from assembling.
>
> Do you have a particular goal, or were you just making sure you understood?
>
> Thanks,
> NeilBrown
^ permalink raw reply
* Re: Raid5 drive fail during grow and no backup
From: Jason Keltz @ 2014-11-10 3:20 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
In-Reply-To: <545D8FBA.9090701@turmel.org>
On 07/11/2014 10:36 PM, Phil Turmel wrote:
> On 11/07/2014 11:06 AM, P. Gautschi wrote:
>> > This is a problem you haven't solved yet, I think. The raid array
>> should have fixed this bad sector for you without kicking the drive out.
>> The scenario is common with "green" drives and/or consumer-grade drives
>> in general.
>> > ...
>> > Then you can set up your array to properly correct bad sectors, and
>> set your system to look for bad sectors on
>> > a regular basis.
>>
>> What is the behavior of mdadm when a disk reports a read error?
>> - reconstruct the data, deliver it to the fs and otherwise ignore it?
>> - set the disk to fail?
>> - reconstruct the data, rewrite the failed data and continue with any
>> action?
>> - rewrite the failed data and reread it (bypassing the cache on the HD)?
>
> Option 3. Reconstruct and rewrite.
>
> However, if the device with the bad sector is trying to recover longer
> than the linux low level driver's timeout, bad things^TM happen.
> Specifically, the driver resets the SATA (or SCSI) connection and
> attempts to reconnect. During this brief time, it will not accept
> further I/O, so the write back of the reconstructed data fails. Then
> the device has experienced a *write* error, so MD fails the drive.
> This is the out-of-the-box behavior of consumer-grade drives in raid
> arrays.
Hi Phil,
Sorry to interject..
Since I'm in the midst of setting up a 22 disk RAID 10 with 2 TB WD
black (desktop) drives, I wanted to be clear that I understand this
particular scenerio that you bring up. Should a drive enter a deep
error recovery, would I be correct that the worst that should happen
would be a hang for the users during this recovery time, and, if the
driver does reset the SATA connection (as it likely would do), then a
potential removal of the disk from the array, but not the destruction of
the array? If I had a spare disk, it would be used for a potential
rebuild, but I could test the original disk and re-add it back to the
pool at another time.
Any feedback would be helpful.
Thanks!
Jason.
^ permalink raw reply
* A recommendation from Support
From: support2341 @ 2014-11-10 5:54 UTC (permalink / raw)
To: rwpremiumpainters, good_mom2003, linux-raid, bennukoop, mari0z
Hello,
Hey Friend,
As you know, I’m always on the lookout for
real ways to make decent money.
Not just a measly $20 or $30 a day; I mean real money.
$20,000 or $30,000 A DAY?!?! Now THAT is…
DAMN GOOD REAL MONEY!
His 6 figure bank accounts now are a pretty good
indicator that Trevor Haynes has figured it out.
Check this out - http://tinyurl.com/ko7bljk
The Millionaire Money Machine has nothing to do with
eBooks, websites, T-shirts, MLM, gambling, writing,
flipping sites or any of that.
THIS isn’t about making ten buck here or there.
It’s about making TENS OF THOUSANDS DAILY.
You’ll meet one Millionaire Money Machine user who
banked over $30K overnight…
While he was asleep!
Take a look - http://tinyurl.com/ko7bljk
For a VERY LIMITED number, Haynes is giving away his
Millionaire Money Machine. No strings, No BS.
But DO NOT WAIT on this!
Check it out here:
- http://tinyurl.com/ko7bljk
NOTE: If your email account doesn't automatically turn the URL above into a link,
you can copy and paste it into your browser.
============================================================================
If the link is broken please use this: http://nikkanares.mmmachine.cpa.clicksure.com
============================================================================
Click below to view the webpage that your friend has recommended:
Support has used the Tell-a-Friend form to send you this link.
We look forward to your visit!
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: md with shared disks
From: Ethan Wilson @ 2014-11-10 16:40 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <545F2630.8090307@true.co.za>
On 09/11/2014 09:30, Anton Ekermans wrote:
> Good day raiders,
> I have a question on md that I cannot find (up to date) answer to.
> We use SuperMicro server with 16 shared disks on a shared backplane
> between two motherboards, running up to date CentOS7.
> If I create an array on one node, the other node can detect it. I put
> GFS2 on top of the array so both system can share the filesystem, but
> I want to know if md raid is safe to be used in this way with possibly
> 2 active/active nodes changing the metadata at the same time. I've
> disabled raid-check cron job on one node so they don't both resync the
> drives weekly, but I suspect there's a lot more to it than that.
>
> If it's not possible, then alternatively some advice on strategy to
> have a large active/active shared disk/filesystem would also be welcome.
Not possible, as far as I know: MD does not reload / exchange metadata
information with other MD peers. MD thinks it is the only user of those
disks.
If you attempt to share the arrays and then one head fails one disk and
starts reconstruction onto another disk, while the other head thinks the
array is all right, havoc will arise certainly.
Even without this worst-case scenario, data probably will be still lost
because the two MDs are not cache coherent, so writes on one head will
not invalidate the kernel cache for the same region on the other head,
and this is bad because reads performed on the other head will not see
the changes just written if such area was cached in the kernel.
GFS actually will attempt to invalidate such cache but I am not sure to
what extent: if you use raid5/6 probably it is not enough because the
stripe-cache will hold stale data in a way that GFS probably does not
know about (does not go away even with echo 3 > /proc/sys/vm/drop_caches
). Maybe raid0/1/10 can be safer... anybody knows if cache dropping
works well there?
But the problem of consistent vision of disk failures and raid
reconstruction seems harder to overcome.
You can do an active/passive configuration, shutting down MD on one head
and starting it on the other head.
Another option is the crossed-active or whatever it is called: some
arrays are active on one head node, other arrays on the other head node,
so to share the computational and bandwidth burden.
If other people have better ideas I am all ears.
Regards
EW
^ permalink raw reply
* Disks never stop spinning
From: P. Gautschi @ 2014-11-10 18:44 UTC (permalink / raw)
To: linux-raid
[-- Attachment #1: Type: text/plain, Size: 762 bytes --]
I've got an array (on kubuntu 14.10) with 5 WD-RED drives and would like
them to stop rotating when there is no access for 10 minutes.
I used hdparm -S 120 for all of them but they keep on running.
I stopped the arrays but this does not help either. The problem might therefor
be unrelated to mdadm. However stopping idle disks on notebooks and desktops
too should be common and I would assume that the general mechanisms are
working. Also for NAS stopping disks should be widely used.
Any ideas why the disks never enter standby or sleep state or how to solve
the problem?
Patrick
PS: I found a hint mentioning SMART tests running in the background could cause
this problem. As far as I understand the output of smartctl, no tests are running
on my disks.
[-- Attachment #2: hdinfo.txt --]
[-- Type: text/plain, Size: 4000 bytes --]
/dev/sdf:
Model=WDC WD60EFRX-68MYMN1, FwRev=82.00A82, SerialNo=WD-WX11D4415118
Config={ HardSect NotMFM HdSw>15uSec SpinMotCtl Fixed DTR>5Mbs FmtGapReq }
RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=0
BuffType=unknown, BuffSize=unknown, MaxMultSect=16, MultSect=16
CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=11721045168
IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
PIO modes: pio0 pio3 pio4
DMA modes: mdma0 mdma1 mdma2
UDMA modes: udma0 udma1 udma2 udma3 udma4 udma5 *udma6
AdvancedPM=no WriteCache=enabled
Drive conforms to: Reserved: ATA/ATAPI-1,2,3,4,5,6,7
* signifies the current active mode
ATA device, with non-removable media
Model Number: WDC WD60EFRX-68MYMN1
Serial Number: WD-WX11D4415118
Firmware Revision: 82.00A82
Transport: Serial, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0
Standards:
Used: unknown (minor revision code 0x001f)
Supported: 9 8 7 6 5
Likely used: 9
Configuration:
Logical max current
cylinders 16383 16383
heads 16 16
sectors/track 63 63
--
CHS current addressable sectors: 16514064
LBA user addressable sectors: 268435455
LBA48 user addressable sectors:11721045168
Logical Sector size: 512 bytes
Physical Sector size: 4096 bytes
Logical Sector-0 offset: 0 bytes
device size with M = 1024*1024: 5723166 MBytes
device size with M = 1000*1000: 6001175 MBytes (6001 GB)
cache/buffer size = unknown
Nominal Media Rotation Rate: 5700
Capabilities:
LBA, IORDY(can be disabled)
Queue depth: 32
Standby timer values: spec'd by Standard, with device specific minimum
R/W multiple sector transfer: Max = 16 Current = 16
DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
Cycle time: min=120ns recommended=120ns
PIO: pio0 pio1 pio2 pio3 pio4
Cycle time: no flow control=120ns IORDY flow control=120ns
Commands/features:
Enabled Supported:
* SMART feature set
Security Mode feature set
* Power Management feature set
* Write cache
* Look-ahead
* Host Protected Area feature set
* WRITE_BUFFER command
* READ_BUFFER command
* NOP cmd
* DOWNLOAD_MICROCODE
Power-Up In Standby feature set
* SET_FEATURES required to spinup after power up
SET_MAX security extension
* 48-bit Address feature set
* Device Configuration Overlay feature set
* Mandatory FLUSH_CACHE
* FLUSH_CACHE_EXT
* SMART error logging
* SMART self-test
Media Card Pass-Through
* General Purpose Logging feature set
* 64-bit World wide name
* URG for READ_STREAM[_DMA]_EXT
* URG for WRITE_STREAM[_DMA]_EXT
* IDLE_IMMEDIATE with UNLOAD
* WRITE_UNCORRECTABLE_EXT command
* {READ,WRITE}_DMA_EXT_GPL commands
* Segmented DOWNLOAD_MICROCODE
* Gen1 signaling speed (1.5Gb/s)
* Gen2 signaling speed (3.0Gb/s)
* Gen3 signaling speed (6.0Gb/s)
* Native Command Queueing (NCQ)
* Host-initiated interface power management
* Phy event counters
* Idle-Unload when NCQ is active
* NCQ priority information
* READ_LOG_DMA_EXT equivalent to READ_LOG_EXT
* DMA Setup Auto-Activate optimization
* Device-initiated interface power management
* Software settings preservation
* SMART Command Transport (SCT) feature set
* SCT Write Same (AC2)
* SCT Error Recovery Control (AC3)
* SCT Features Control (AC4)
* SCT Data Tables (AC5)
unknown 206[12] (vendor specific)
unknown 206[13] (vendor specific)
* DOWNLOAD MICROCODE DMA command
* WRITE BUFFER DMA command
* READ BUFFER DMA command
Security:
Master password revision code = 65534
supported
not enabled
not locked
frozen
not expired: security count
supported: enhanced erase
Logical Unit WWN Device Identifier: 50014ee0ae896953
NAA : 5
IEEE OUI : 0014ee
Unique ID : 0ae896953
Checksum: correct
[-- Attachment #3: smart.txt --]
[-- Type: text/plain, Size: 10015 bytes --]
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.16.0-24-generic] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: WDC WD60EFRX-68MYMN1
Serial Number: WD-WX11D4415118
LU WWN Device Id: 5 0014ee 0ae896953
Firmware Version: 82.00A82
User Capacity: 6'001'175'126'016 bytes [6.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5700 rpm
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Nov 10 18:20:06 2014 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
ATA Security is: Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 5204) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 705) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x303d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-K 200 200 051 - 0
3 Spin_Up_Time POS--K 201 201 021 - 8925
4 Start_Stop_Count -O--CK 100 100 000 - 11
5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0
7 Seek_Error_Rate -OSR-K 200 200 000 - 0
9 Power_On_Hours -O--CK 100 100 000 - 16
10 Spin_Retry_Count -O--CK 100 253 000 - 0
11 Calibration_Retry_Count -O--CK 100 253 000 - 0
12 Power_Cycle_Count -O--CK 100 100 000 - 6
192 Power-Off_Retract_Count -O--CK 200 200 000 - 3
193 Load_Cycle_Count -O--CK 200 200 000 - 11
194 Temperature_Celsius -O---K 121 113 000 - 31
196 Reallocated_Event_Count -O--CK 200 200 000 - 0
197 Current_Pending_Sector -O--CK 200 200 000 - 0
198 Offline_Uncorrectable ----CK 100 253 000 - 0
199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 0
200 Multi_Zone_Error_Rate ---R-- 100 253 000 - 0
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 5 Comprehensive SMART error log
0x03 GPL R/O 6 Ext. Comprehensive SMART error log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x09 SL R/W 1 Selective self-test log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters
0x21 GPL R/O 1 Write stream error log
0x22 GPL R/O 1 Read stream error log
0x30 GPL,SL R/O 9 IDENTIFY DEVICE data log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xa0-0xa7 GPL,SL VS 16 Device vendor specific log
0xa8-0xb6 GPL,SL VS 1 Device vendor specific log
0xb7 GPL,SL VS 40 Device vendor specific log
0xbd GPL,SL VS 1 Device vendor specific log
0xc0 GPL,SL VS 1 Device vendor specific log
0xc1 GPL VS 93 Device vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer
SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
No Errors Logged
SMART Error Log Version: 1
No Errors Logged
SMART Extended Self-test Log Version: 1 (1 sectors)
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Status Version: 3
SCT Version (vendor specific): 258 (0x0102)
SCT Support Level: 1
Device State: Active (0)
Current Temperature: 31 Celsius
Power Cycle Min/Max Temperature: 28/31 Celsius
Lifetime Min/Max Temperature: 20/39 Celsius
Under/Over Temperature Limit Count: 0/0
SCT Temperature History Version: 2
Temperature Sampling Period: 1 minute
Temperature Logging Interval: 1 minute
Min/Max recommended Temperature: 0/60 Celsius
Min/Max Temperature Limit: -41/85 Celsius
Temperature History Size (Index): 478 (50)
Index Estimated Time Temperature Celsius
51 2014-11-10 10:23 35 ****************
... ..(125 skipped). .. ****************
177 2014-11-10 12:29 35 ****************
178 2014-11-10 12:30 34 ***************
... ..(247 skipped). .. ***************
426 2014-11-10 16:38 34 ***************
427 2014-11-10 16:39 ? -
428 2014-11-10 16:40 34 ***************
... ..( 2 skipped). .. ***************
431 2014-11-10 16:43 34 ***************
432 2014-11-10 16:44 33 **************
... ..( 6 skipped). .. **************
439 2014-11-10 16:51 33 **************
440 2014-11-10 16:52 ? -
441 2014-11-10 16:53 20 *
... ..( 2 skipped). .. *
444 2014-11-10 16:56 20 *
445 2014-11-10 16:57 21 **
446 2014-11-10 16:58 ? -
447 2014-11-10 16:59 21 **
448 2014-11-10 17:00 22 ***
449 2014-11-10 17:01 22 ***
450 2014-11-10 17:02 23 ****
451 2014-11-10 17:03 23 ****
452 2014-11-10 17:04 23 ****
453 2014-11-10 17:05 24 *****
454 2014-11-10 17:06 24 *****
455 2014-11-10 17:07 24 *****
456 2014-11-10 17:08 ? -
457 2014-11-10 17:09 25 ******
458 2014-11-10 17:10 25 ******
459 2014-11-10 17:11 25 ******
460 2014-11-10 17:12 26 *******
... ..( 4 skipped). .. *******
465 2014-11-10 17:17 26 *******
466 2014-11-10 17:18 27 ********
... ..( 6 skipped). .. ********
473 2014-11-10 17:25 27 ********
474 2014-11-10 17:26 ? -
475 2014-11-10 17:27 28 *********
... ..( 3 skipped). .. *********
1 2014-11-10 17:31 28 *********
2 2014-11-10 17:32 29 **********
... ..( 3 skipped). .. **********
6 2014-11-10 17:36 29 **********
7 2014-11-10 17:37 30 ***********
... ..( 3 skipped). .. ***********
11 2014-11-10 17:41 30 ***********
12 2014-11-10 17:42 31 ************
... ..( 37 skipped). .. ************
50 2014-11-10 18:20 31 ************
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
Device Statistics (GP Log 0x04) not supported
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 2 0 Command failed due to ICRC error
0x0002 2 0 R_ERR response for data FIS
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0005 2 0 R_ERR response for non-data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
0x0008 2 0 Device-to-host non-data FIS retries
0x0009 2 2 Transition from drive PhyRdy to drive PhyNRdy
0x000a 2 3 Device-to-host register FISes sent due to a COMRESET
0x000b 2 0 CRC errors within host-to-device FIS
0x000f 2 0 R_ERR response for host-to-device data FIS, CRC
0x0012 2 0 R_ERR response for host-to-device non-data FIS, CRC
0x8000 4 3231 Vendor specific
^ permalink raw reply
* Re: md with shared disks
From: Stan Hoeppner @ 2014-11-10 22:14 UTC (permalink / raw)
To: Anton Ekermans, linux-raid
In-Reply-To: <545F2630.8090307@true.co.za>
On 11/09/2014 02:30 AM, Anton Ekermans wrote:
> Good day raiders,
> I have a question on md that I cannot find (up to date) answer to.
> We use SuperMicro server with 16 shared disks on a shared backplane
> between two motherboards, running up to date CentOS7.
> If I create an array on one node, the other node can detect it. I put
> GFS2 on top of the array so both system can share the filesystem, but I
> want to know if md raid is safe to be used in this way with possibly 2
> active/active nodes changing the metadata at the same time. I've
> disabled raid-check cron job on one node so they don't both resync the
> drives weekly, but I suspect there's a lot more to it than that.
>
> If it's not possible, then alternatively some advice on strategy to have
> a large active/active shared disk/filesystem would also be welcome.
It's not possible to do what you mention as md is not cluster aware. It
will break, badly. What most people do in such cases in create two md
arrays, one controlled by each host, and mirror them with DRBD, then put
OCFS/GFS atop DRBD. You lose half your capacity doing this, but it's
the only way to do it and have all disks active. Of course you lose
half your bandwidth as well. This is a high availability solution, not
high performance.
You bought this hardware to do something. And that something wasn't
simply making two hosts in one box use all the disks in the box. What
is the workload you plan to run on this hardware? The workload dictates
the needed hardware architecture, not the other way around. If you want
high availability this hardware will work using the stack architecture
above, and work well. If you need high performance shared filesystem
access between both nodes you need an external SAS/FC RAID array and a
cluster FS. In either case you're using a cluster FS which means high
file throughput but low metadata throughgput.
If it's high performance you need, an option is to submit patches to
make md cluster aware. Another is the LSI clustering RAID controller
kit for internal drives. Don't know anything about it other than it is
available and apparently works with RHEL and SUSE. Seems suitable for
what you express as your need.
http://www.lsi.com/products/shared-das/pages/syncro-cs-9271-8i.aspx#tab/tab2
Cheers,
Stan
^ permalink raw reply
* homehost and 0.90 format
From: Boylan, Ross @ 2014-11-11 3:14 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
I am trying to figure out which of my RAID devices is considered home. I haven't switched machines, but I have several operating systems, with different hostnames, on the machine. 0.90 format metadata.
Even 2 RAID devices created on the same OS don't seem to have any UUID in common. i thought that the end of the UUID's would be the same. For example, inhttp://marc.info/?l=linux-raid&m=123999237229707 Neil Brown says
> Note that 0.90 metadata does contain homehost information to some
> extent. When homehost is set, the last few bytes of the uuid is set
> from a hash of the homehost name. That makes it possible to test if a
> 0.90 array was created for 'this' host, but not to find out what host
> it was created for.
mdadm.conf has
HOMEHOST <system>
The 2 UUIDS are
8691ad98:af955756:ae5efbfa:dfd20ce6
6d89d3b5:19a0acce:319e1a9c:58606385
Thanks,
Ross Boylan
Details:
root@wheezy4:~# mdadm --detail /dev/md7
/dev/md7:
Version : 0.90
Creation Time : Fri Oct 3 17:20:13 2014
Raid Level : raid1
Array Size : 48828352 (46.57 GiB 50.00 GB)
Used Dev Size : 48828352 (46.57 GiB 50.00 GB)
Raid Devices : 1
Total Devices : 1
Preferred Minor : 7
Persistence : Superblock is persistent
Update Time : Tue Nov 4 17:48:40 2014
State : clean
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0
UUID : 8691ad98:af955756:ae5efbfa:dfd20ce6
Events : 0.1
Number Major Minor RaidDevice State
0 8 20 0 active sync /dev/sdb4
root@wheezy4:~# mdadm --detail /dev/md8
/dev/md8:
Version : 0.90
Creation Time : Tue Sep 30 13:27:56 2014
Raid Level : raid1
Array Size : 970702784 (925.73 GiB 994.00 GB)
Used Dev Size : 970702784 (925.73 GiB 994.00 GB)
Raid Devices : 2
Total Devices : 1
Preferred Minor : 8
Persistence : Superblock is persistent
Update Time : Mon Nov 10 18:16:11 2014
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0
UUID : 6d89d3b5:19a0acce:319e1a9c:58606385
Events : 0.790792
Number Major Minor RaidDevice State
0 8 19 0 active sync /dev/sdb3
1 0 0 1 removed
^ permalink raw reply
* Re: RAID 6 (containing LUKS dm-crypt) recovery help.
From: Peter Grandi @ 2014-11-12 11:01 UTC (permalink / raw)
To: Linux RAID
In-Reply-To: <21596.54723.487775.652906@tree.ty.sabi.co.uk>
> You can try to recreate the MD set superblocks specifying the
> right data offset and member size for each member, as given by
> the '--examine' outputs. Google knows how...
Just in case, this describes a very similar case:
http://www.spinics.net/lists/raid/msg39353.html
^ permalink raw reply
* failed RAID 5 array
From: DeadManMoving @ 2014-11-12 15:58 UTC (permalink / raw)
To: linux-raid; +Cc: DeadManMoving
Hi list,
I have a failed RAID 5 array, composed of 4 x 2TB drives without hot
spare. On the fail array, it looks like there is one drive out of sync
(the one with a lower Events counts) and another drive with a missing or
corrupted superblock (dmesg is reporting "does not have a valid v1.2
superblock, not importing!" and i have a : Checksum : 5608a55a -
expected 4108a55a).
All drives seems good though, the problem was probably triggered by a a
broken communication between the external eSATA expansion card and
external drive enclosure (card, cable or backplane in the enclosure i
guess...).
I am now in the process of making exact copies of the drives with dd to
other drives.
I have an idea on how to try to get my data back but i would be happy if
someone could help/validate with the steps i intent to follow to get
there.
On that array, i have ~ 85% of data which is already backed up somewhere
else, ~ 10% of data for which i do not care much, but there is ~ 5% of
data that is really important to me and for which i do not have other
copies around :|
So, here are the steps i intend to follow once the dd process is gonna
be over :
- take the drives which i made the dd on it
- try to create a new array on the two good drives, plus the one with
the superblock problem, by respecting the order (according to data i
have gathered from the drives), with the correct chunk size, with a
command like this :
# mdadm --create --assume-clean --level=5 --chunk=512
--raid-devices=4 /dev/md127 /dev/sdd /dev/sde /dev/sdb missing
- if the array is coming up nicely, will try to validate if fs is good
on it :
# fsck.ext4 -n /dev/md127
- if all is still fine, mount the array read-only and backup all i need
as fast as possible!
- then i guess i could add the drive (/dev/sdc) back to the array :
# mdadm --add /dev/md127 /dev/sdc
Can anyone tell me if those steps make sense? Does i miss something
obvious? Does i have any chance to recover my data with that procedure?
I would like to avoid trials and errors since it take 24 hours to make a
full copy of a drive with dd (4 days for the four drives).
Here is the output of mdadm --examine of my four drives :
/dev/sdb:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : d707f577:a9e572d5:e5d5f10c:b232f15a
Name : abc:xyz (local to host abc)
Creation Time : Fri Aug 9 21:55:47 2013
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3907027120 (1863.02 GiB 2000.40 GB)
Array Size : 5860538880 (5589.05 GiB 6001.19 GB)
Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=1200 sectors
State : clean
Device UUID : 2b438b47:db326d4a:0ae82357:1b88590d
Update Time : Mon Nov 10 15:48:17 2014
Checksum : ebfcf43 - correct
Events : 9370
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAA. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc:
Magic : a92b4efc
Version : 1.2
Feature Map : 0xa
Array UUID : d707f577:a9e572d5:e5d5f10c:b232f15a
Name : abx:xyz (local to host abc)
Creation Time : Fri Aug 9 21:55:47 2013
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3907027120 (1863.02 GiB 2000.40 GB)
Array Size : 5860538880 (5589.05 GiB 6001.19 GB)
Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Recovery Offset : 0 sectors
Unused Space : before=1960 sectors, after=1200 sectors
State : active
Device UUID : 011e3cbb:42c0ac0a:d6815904:2150169a
Update Time : Mon Nov 10 15:44:07 2014
Bad Block Log : 512 entries available at offset 72 sectors - bad
blocks present.
Checksum : 7ca998a5 - correct
Events : 9358
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : d707f577:a9e572d5:e5d5f10c:b232f15a
Name : abc:xyz (local to host abc)
Creation Time : Fri Aug 9 21:55:47 2013
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3907027120 (1863.02 GiB 2000.40 GB)
Array Size : 5860538880 (5589.05 GiB 6001.19 GB)
Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=1200 sectors
State : clean
Device UUID : 67ffc02b:c8a013a7:3f17dc65:d1040e05
Update Time : Mon Nov 10 15:48:17 2014
Checksum : 5608a55a - expected 4108a55a
Events : 9370
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAA. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : d707f577:a9e572d5:e5d5f10c:b232f15a
Name : abc:xyz (local to host abc)
Creation Time : Fri Aug 9 21:55:47 2013
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3907027120 (1863.02 GiB 2000.40 GB)
Array Size : 5860538880 (5589.05 GiB 6001.19 GB)
Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=1200 sectors
State : clean
Device UUID : 7b37a749:f1e575d1:50eea3c4:2083b9be
Update Time : Mon Nov 10 15:48:17 2014
Checksum : b6c477f4 - correct
Events : 9370
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAA. ('A' == active, '.' == missing, 'R' == replacing)
Thanks and regards,
Tony
^ permalink raw reply
* A recommendation from Mitchell Miller
From: mitchellmiller @ 2014-11-13 5:50 UTC (permalink / raw)
To: rwpremiumpainters, good_mom2003, linux-raid, bennukoop, mari0z
Hello,
Subject: YOUR Millionaire Money Machine is Here!
Hey Friend,
As you know, I’m always on the lookout for
real ways to make decent money.
Not just a measly $20 or $30 a day; I mean real money.
$20,000 or $30,000 A DAY?!?! Now THAT is…
DAMN GOOD REAL MONEY!
His 6 figure bank accounts now are a pretty good
indicator that Trevor Haynes has figured it out.
Check this out - http://crossfires.mmmachine.cpa.clicksure.com
The Millionaire Money Machine has nothing to do with
eBooks, websites, T-shirts, MLM, gambling, writing,
flipping sites or any of that.
THIS isn’t about making ten buck here or there.
It’s about making TENS OF THOUSANDS DAILY.
You’ll meet one Millionaire Money Machine user who
banked over $30K overnight…
While he was asleep!
Take a look - http://crossfires.mmmachine.cpa.clicksure.com
For a VERY LIMITED number, Haynes is giving away his
Millionaire Money Machine. No strings, No BS.
But DO NOT WAIT on this!
Check it out here:
- http://crossfires.mmmachine.cpa.clicksure.com
NOTE: If your email account doesn't automatically turn the URL above into a link,
you can copy and paste it into your browser.
============================================================================
If the link is broken please use this: http://crossfires.mmmachine.cpa.clicksure.com
============================================================================
Click below to view the webpage that your friend has recommended:
Mitchell Miller has used the Tell-a-Friend form to send you this link.
We look forward to your visit!
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: md with shared disks
From: Anton Ekermans @ 2014-11-13 13:14 UTC (permalink / raw)
To: Stan Hoeppner, linux-raid
In-Reply-To: <546138B5.7020101@hardwarefreak.com>
Thank you very much for your clear response.
The purpose of this hardware is to primarily host ample VM storage for
the 2 nodes itself and 3 other i7 PC/servers.
The HA was hoped to be achieved as active/active with both nodes sharing
the same disks and non-cluster servers(i7) having multi-path to these
two nodes. This is advertised as HA active/active in storage software
such as Nexenta using RSF-1. However upon closer inspection, their
active/active means both nodes share some data and the other can take
over. So for me, in essence it is "active/passive + passive/active" and
not truly "active/active". We will try to config this way to get quasi
active/active for best performance with kind-of high-availability. Seems
the shared disks is not the problem, but combining them on a cluster is.
Thank you again
Best regards
Untitled Document
Anton Ekermans
> It's not possible to do what you mention as md is not cluster aware. It
> will break, badly. What most people do in such cases in create two md
> arrays, one controlled by each host, and mirror them with DRBD, then put
> OCFS/GFS atop DRBD. You lose half your capacity doing this, but it's
> the only way to do it and have all disks active. Of course you lose
> half your bandwidth as well. This is a high availability solution, not
> high performance.
>
> You bought this hardware to do something. And that something wasn't
> simply making two hosts in one box use all the disks in the box. What
> is the workload you plan to run on this hardware? The workload dictates
> the needed hardware architecture, not the other way around. If you want
> high availability this hardware will work using the stack architecture
> above, and work well. If you need high performance shared filesystem
> access between both nodes you need an external SAS/FC RAID array and a
> cluster FS. In either case you're using a cluster FS which means high
> file throughput but low metadata throughgput.
>
> If it's high performance you need, an option is to submit patches to
> make md cluster aware. Another is the LSI clustering RAID controller
> kit for internal drives. Don't know anything about it other than it is
> available and apparently works with RHEL and SUSE. Seems suitable for
> what you express as your need.
>
> http://www.lsi.com/products/shared-das/pages/syncro-cs-9271-8i.aspx#tab/tab2
>
>
> Cheers,
> Stan
^ permalink raw reply
* Re: RAID1 removing failed disk returns EBUSY
From: Joe Lawrence @ 2014-11-13 14:05 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid, Bill Kuzeja
In-Reply-To: <20141029133604.59f9549a@jlaw-desktop.mno.stratus.com>
On Wed, 29 Oct 2014 13:36:04 -0400
Joe Lawrence <joe.lawrence@stratus.com> wrote:
> On Wed, 29 Oct 2014 08:41:13 +1100
> NeilBrown <neilb@suse.de> wrote:
>
> > On Mon, 27 Oct 2014 16:27:48 -0400 Joe Lawrence <joe.lawrence@stratus.com>
> > wrote:
> >
> > > Hi Neil,
> > >
> > > We've encountered changes in MD and mdadm that have broken our automated
> > > disk removal script. In the past, we've been able to run the following
> > > after a RAID1 disk component removal:
> > >
> > > % echo fail > /sys/block/md3/md/dev-sdr5/state
> > > % echo remove > /sys/block/md3/md/dev-sdr5/state
> > >
> > > However, the latest RHEL6.6 code drop has rebased to sufficiently recent
> > > MD kernel and mdadm changes, in which the previous commands occasionally
> > > fail like so:
> > >
> > > * MD array is usually resyncing or checking
> > > * Component disk /dev/sdr removed via HBA sysfs PCI removal
> > > * Following UDEV rule fires:
> > >
> > > SUBSYSTEM=="block", ACTION=="remove", ENV{ID_PATH}=="?*", \
> > > RUN+="/sbin/mdadm -If $name --path $env{ID_PATH}"
> > >
> > > % mdadm --detail /dev/md3
> > > /dev/md3:
> > > Version : 1.1
> > > Creation Time : Tue Oct 14 17:31:59 2014
> > > Raid Level : raid1
> > > Array Size : 25149440 (23.98 GiB 25.75 GB)
> > > Used Dev Size : 25149440 (23.98 GiB 25.75 GB)
> > > Raid Devices : 2
> > > Total Devices : 2
> > > Persistence : Superblock is persistent
> > >
> > > Intent Bitmap : Internal
> > >
> > > Update Time : Wed Oct 15 14:22:34 2014
> > > State : active, degraded
> > > Active Devices : 1
> > > Working Devices : 1
> > > Failed Devices : 1
> > > Spare Devices : 0
> > >
> > > Name : localhost.localdomain:3
> > > UUID : 40ed68ee:ba41d4cd:28c361ed:be7470b8
> > > Events : 142
> > >
> > > Number Major Minor RaidDevice State
> > > 0 65 21 0 faulty
> > > 1 65 5 1 active sync /dev/sdj5
> > >
> > > All attempts to remove this device fail:
> > >
> > > % echo remove > /sys/block/md3/md/dev-sdr5/state
> > > -bash: echo: write error: Device or resource busy
> > >
> > > This can be traced to state_store():
> > >
> > > } else if (cmd_match(buf, "remove")) {
> > > if (rdev->raid_disk >= 0)
> > > err = -EBUSY;
> > >
> > > After much debugging and systemtapping, I think I've figured out that the
> > > sysfs scripting may fail after the following combination of changes:
> > >
> > > mdadm 8af530b07fce "Enhance incremental removal."
> > > kernel 30b8feb730f9 "md/raid5: avoid deadlock when raid5 array has unack
> > > badblocks during md_stop_writes"
> > >
> > > With these two changes:
> > >
> > > 1 - On the user side, mdadm is trying to set the array_state to read-auto
> > > on incremental removal (as invoked by UDEV rule).
> > >
> > > 2 - Kernel side, md_set_readonly() will set the MD_RECOVERY_FROZEN flag,
> > > wake up the mddev->thread and if there is a sync_thread, it will set
> > > MD_RECOVERY_INTR and then wait until the sync_thread is set to NULL.
> > >
> > > When md_check_recovery() gets a chance to run as part of the
> > > raid1d() mddev->thread, it may or may not ever get to
> > > an invocation of remove_and_add_spares(), for there are but *many*
> > > conditional early exits along the way -- for example, if
> > > MD_RECOVERY_FROZEN is set, the following condition will bounce out of
> > > the routine:
> > >
> > > if (!test_and_clear_bit(MD_RECOVERY_NEEDED, &mddev->recovery) ||
> > > test_bit(MD_RECOVERY_FROZEN, &mddev->recovery))
> > > goto unlock;
> > >
> > > the next time around, MD_RECOVERY_NEEDED will have been cleared, so
> > > all future tests will return 0 and the negation will always take the
> > > early exit path.
> > >
> > > Back in md_set_readonly(), it may notice that the MD is still in use,
> > > so it clears the MD_RECOVERY_FROZEN and then returns -EBUSY, without
> > > setting mddev->ro. But the damage has been done as conditions have
> > > been set such that md_check_recovery() will never call
> > > remove_and_add_spares().
> > >
> > > This would also explain why an "idle" sync_action clears the wedge: it
> > > sets MD_RECOVERY_NEEDED allowing md_check_recovery() to continue executing
> > > to remove_and_add_spares().
> > >
> > > As far as I can tell, this is what is happening to prevent the "remove"
> > > write to /sys/block/md3/md/dev-sdr5/state from succeeding. There are
> > > certainly a lot of little bit-states between disk removal, UDEV mdadm, and
> > > various MD kernel threads, so apologies if I missed an important
> > > transition.
> > >
> > > Would you consider writing "idle" to the MD array sync_action file as a
> > > safe and reasonable intermediate workaround step for our script?
> > >
> > > And of course, any suggestions to whether this is intended behavior (ie,
> > > the removed component disk is failed, but stuck in the array)?
> > >
> > > This is fairly easy for us to reproduce with multiple MD arrays per disk
> > > (one per partition) and interrupting a raid check on all of them
> > > (especially when they are delayed waiting for the first to finish) by
> > > removing the component disk via sysfs PCI removal. We can provide
> > > additional debug or testing if required.
> > >
> >
> > Hi Joe,
> > thanks for the details analysis!!
> >
> > I think the correct fix would be that MD_RECOVERY_NEEDED should be set after
> > clearing MD_RECOVERY_FROZEN, like the patch below.
> > Can you confirm that it works for you?
> >
> > Writing 'idle' should in general be safe, so that could be used as an interim.
> >
> > Thanks,
> > NeilBrown
> >
> > diff --git a/drivers/md/md.c b/drivers/md/md.c
> > index c03d87b6890a..2c73fcb82593 100644
> > --- a/drivers/md/md.c
> > +++ b/drivers/md/md.c
> > @@ -5261,6 +5261,7 @@ static int md_set_readonly(struct mddev *mddev, struct block_device *bdev)
> > printk("md: %s still in use.\n",mdname(mddev));
> > if (did_freeze) {
> > clear_bit(MD_RECOVERY_FROZEN, &mddev->recovery);
> > + set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
> > md_wakeup_thread(mddev->thread);
> > }
> > err = -EBUSY;
> > @@ -5275,6 +5276,8 @@ static int md_set_readonly(struct mddev *mddev, struct block_device *bdev)
> > mddev->ro = 1;
> > set_disk_ro(mddev->gendisk, 1);
> > clear_bit(MD_RECOVERY_FROZEN, &mddev->recovery);
> > + set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
> > + md_wakeup_thread(mddev->thread);
> > sysfs_notify_dirent_safe(mddev->sysfs_state);
> > err = 0;
> > }
> > @@ -5318,6 +5321,7 @@ static int do_md_stop(struct mddev *mddev, int mode,
> > mutex_unlock(&mddev->open_mutex);
> > if (did_freeze) {
> > clear_bit(MD_RECOVERY_FROZEN, &mddev->recovery);
> > + set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
> > md_wakeup_thread(mddev->thread);
> > }
> > return -EBUSY;
>
> Hi Neil,
>
> In my tests, the UDEV "mdadm -If" invocation fails *and* removes the
> pulled disk from the MD array. This is okay for our intentions, but I
> wanted to make sure that it's okay to skip any failed-but-not-removed
> state.
>
> Tested-by: Joe Lawrence <joe.lawrence@stratus.com>
>
> and should this have a
>
> Fixes: 30b8feb730f9 ("md/raid5: avoid deadlock when raid5 array has unack badblocks during md_stop_writes")
>
> tag to mark for stable?
Hi Neil,
Would you like me to write up a proper patch, or is this one in the queue?
Thanks,
-- Joe
^ permalink raw reply
* Re: Disks never stop spinning
From: Peter Grandi @ 2014-11-13 14:05 UTC (permalink / raw)
To: Linux RAID
In-Reply-To: <54610788.6060103@gautschi.net>
> I've got an array (on kubuntu 14.10) with 5 WD-RED drives and
> would like them to stop rotating when there is no access for
> 10 minutes. I used hdparm -S 120
That works here for 6 disks on which I have created various
types of MD sets for holding data or for testing.
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid10] [linear] [multipath] [raid0] [raid1]
md1 : active raid10 sde3[6] sdd3[4] sdb3[0] sdc3[5]
486538976 blocks super 1.0 16K chunks 2 near-copies [4/4] [UUUU]
md4 : active raid6 sde4[3] sdb4[0] sdg4[7] sdd4[2] sdf4[6] sdc4[1]
973077760 blocks super 1.0 level 6, 64k chunk, algorithm 2 [6/6] [UUUUUU]
md2 : active raid5 sde2[2] sdg2[5] sdf2[3] sdc2[0] sdd2[1]
486538752 blocks super 1.0 level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]
md0 : active raid10 sde1[3] sdb1[0] sdg1[5] sdf1[4] sdc1[1] sdd1[2]
364904208 blocks super 1.0 16K chunks 2 near-copies [6/6] [UUUUUU]
unused devices: <none>
# hddtemp /dev/sd[bcdef]
/dev/sdb: SAMSUNG HD103UJ: drive is sleeping
/dev/sdc: WDC WD10EZEX-22RKKA0: drive is sleeping
/dev/sdd: ST1000DM003-9YN162: drive is sleeping
/dev/sde: SAMSUNG HD103SJ: drive is sleeping
/dev/sdf: WDC WD10EZEX-22RKKA0: drive is sleeping
> for all of them but they keep on running. [ ... ]
That means something else is waking up those disks. To verify
this use something like:
sudo iostat -dkxz 1
or use something like:
sudo sysctl vm/block_dump=1; sleep 120; sudo sysctl vm/block_dump=0
and then look at the debug log to see which inodes get hit.
^ permalink raw reply
* Re: md with shared disks
From: Stan Hoeppner @ 2014-11-13 20:56 UTC (permalink / raw)
To: Anton Ekermans, linux-raid
In-Reply-To: <5464AEA8.3010106@true.co.za>
With DRBD and GFS2 it is true active/active at the block level. You
just lose half your disk capacity due to the host-to-host mirroring.
Whether your upper layers are active/active is another story. E.g.
getting NFS server/client to do seamless automatic path failover is
still a shaky proposition AIUI.
You mention multipath. If you plan to use iSCSI multipath for the i7
servers you need to make sure each LUN you export has the same WWID on
both cluster nodes.
Stan
On 11/13/2014 07:14 AM, Anton Ekermans wrote:
> Thank you very much for your clear response.
> The purpose of this hardware is to primarily host ample VM storage for
> the 2 nodes itself and 3 other i7 PC/servers.
> The HA was hoped to be achieved as active/active with both nodes sharing
> the same disks and non-cluster servers(i7) having multi-path to these
> two nodes. This is advertised as HA active/active in storage software
> such as Nexenta using RSF-1. However upon closer inspection, their
> active/active means both nodes share some data and the other can take
> over. So for me, in essence it is "active/passive + passive/active" and
> not truly "active/active". We will try to config this way to get quasi
> active/active for best performance with kind-of high-availability. Seems
> the shared disks is not the problem, but combining them on a cluster is.
>
> Thank you again
>
> Best regards
> Untitled Document
>
> Anton Ekermans
>
>> It's not possible to do what you mention as md is not cluster aware. It
>> will break, badly. What most people do in such cases in create two md
>> arrays, one controlled by each host, and mirror them with DRBD, then put
>> OCFS/GFS atop DRBD. You lose half your capacity doing this, but it's
>> the only way to do it and have all disks active. Of course you lose
>> half your bandwidth as well. This is a high availability solution, not
>> high performance.
>>
>> You bought this hardware to do something. And that something wasn't
>> simply making two hosts in one box use all the disks in the box. What
>> is the workload you plan to run on this hardware? The workload dictates
>> the needed hardware architecture, not the other way around. If you want
>> high availability this hardware will work using the stack architecture
>> above, and work well. If you need high performance shared filesystem
>> access between both nodes you need an external SAS/FC RAID array and a
>> cluster FS. In either case you're using a cluster FS which means high
>> file throughput but low metadata throughgput.
>>
>> If it's high performance you need, an option is to submit patches to
>> make md cluster aware. Another is the LSI clustering RAID controller
>> kit for internal drives. Don't know anything about it other than it is
>> available and apparently works with RHEL and SUSE. Seems suitable for
>> what you express as your need.
>>
>> http://www.lsi.com/products/shared-das/pages/syncro-cs-9271-8i.aspx#tab/tab2
>>
>>
>>
>> Cheers,
>> Stan
>
^ permalink raw reply
* Re: md with shared disks
From: Ethan Wilson @ 2014-11-13 22:53 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <54651B05.1050104@hardwarefreak.com>
On 13/11/2014 21:56, Stan Hoeppner wrote:
> With DRBD and GFS2 it is true active/active at the block level. You
> just lose half your disk capacity due to the host-to-host mirroring.
Sorry but I don't share your definition of active/active.
Would you say that a raid1 is an active/active thing?
Doubling the number of disks and repeating the operation on both sides
is not active/active in the sense that people usually want.
Active/active commonly means that you have twice the performance of
active/passive.
In this sense DRBD not only is an active/passive but it is even way
below the performances of an active/passive because it has to transmit
the data to the peer in addition to write to the disks, and this takes
CPU time for memcpy and interrupts, introduces latency, requires
additional hardware (= fast networking dedicated to DRBD). An
active/passive with shared disks is hence "twice" (very roughly) faster
than DRBD at the same price spent on the head nodes. An active/active
with shared disks is hence 4 times (again very roughly) faster than
DRBD, at the same price for the head nodes.
In addition to this with DRBD you have to buy twice the number of disks,
which is also an additional expense. Marginally though, because a
shared-disk infrastructure is way more expensive than a direct-attached
one, but it has to be planned like that in advance, and not retrofitted
like you propose.
His current infrastructure cannot be easily converted to DRBD without
major losses: if he attempts to do so he will have almost double the
costs of a basic DRBD shared-nothing direct-attached infrastructure or
exactly double the cost of a shared-disk infrastructure, intended as
cost per TB of data. Unfortunately, after this he will still have half
the performances of an active/passive shared-disk clustered-MD solution.
^ permalink raw reply
* Re: failed RAID 5 array
From: Phil Turmel @ 2014-11-13 22:56 UTC (permalink / raw)
To: DeadManMoving, linux-raid
In-Reply-To: <1415807882.4241.36.camel@lappy.neofreak.org>
On 11/12/2014 10:58 AM, DeadManMoving wrote:
> Hi list,
>
> I have a failed RAID 5 array, composed of 4 x 2TB drives without hot
> spare. On the fail array, it looks like there is one drive out of sync
> (the one with a lower Events counts) and another drive with a missing or
> corrupted superblock (dmesg is reporting "does not have a valid v1.2
> superblock, not importing!" and i have a : Checksum : 5608a55a -
> expected 4108a55a).
>
> All drives seems good though, the problem was probably triggered by a a
> broken communication between the external eSATA expansion card and
> external drive enclosure (card, cable or backplane in the enclosure i
> guess...).
>
> I am now in the process of making exact copies of the drives with dd to
> other drives.
>
> I have an idea on how to try to get my data back but i would be happy if
> someone could help/validate with the steps i intent to follow to get
> there.
--create is almost always a bad idea.
Just use "mdadm -vv --assemble --force /dev/mdX /dev/sd[abcd]"
One drive will be left behind (the bad superblock), but the stale one
will be revived and you'll be able to start.
If that doesn't work, show the output of the above command. Do NOT do
an mdadm --create.
Phil
^ permalink raw reply
* Business co-operation
From: Emry Holmes @ 2014-11-14 0:00 UTC (permalink / raw)
Dear Sir,
I represent Group of investors in Middle east and they are
seeking`means of expanding business interest abroad such as Real
Estate/Construction, oil industry, Banking, Stock speculation, Film
Industries, Mining,Transportation, agriculture and those any other
Lucrative business.
If you have a solid background and idea of making good profit in any
business, that is capable of generating 10% annual return on
investment(AROI) Joint Venture, please write me for possible business
co-operation, as we are ready to fund any business that will interest
us.
I am available to discuss with you, and answer any questions, based on
the investment. I look forward to discussing this opportunity further
in details with you.
Your Sincerely
Emry Holmes
^ permalink raw reply
* Re: md with shared disks
From: Stan Hoeppner @ 2014-11-14 0:07 UTC (permalink / raw)
To: Ethan Wilson, linux-raid
In-Reply-To: <54653661.80605@shiftmail.org>
On 11/13/2014 04:53 PM, Ethan Wilson wrote:
> On 13/11/2014 21:56, Stan Hoeppner wrote:
>> With DRBD and GFS2 it is true active/active at the block level. You
>> just lose half your disk capacity due to the host-to-host mirroring.
>
> Sorry but I don't share your definition of active/active.
>
> Would you say that a raid1 is an active/active thing?
>
> Doubling the number of disks and repeating the operation on both sides
> is not active/active in the sense that people usually want.
>
> Active/active commonly means that you have twice the performance of
> active/passive.
>
> In this sense DRBD not only is an active/passive but it is even way
> below the performances of an active/passive because it has to transmit
> the data to the peer in addition to write to the disks, and this takes
> CPU time for memcpy and interrupts, introduces latency, requires
> additional hardware (= fast networking dedicated to DRBD). An
> active/passive with shared disks is hence "twice" (very roughly) faster
> than DRBD at the same price spent on the head nodes. An active/active
> with shared disks is hence 4 times (again very roughly) faster than
> DRBD, at the same price for the head nodes.
>
> In addition to this with DRBD you have to buy twice the number of disks,
> which is also an additional expense. Marginally though, because a
> shared-disk infrastructure is way more expensive than a direct-attached
> one, but it has to be planned like that in advance, and not retrofitted
> like you propose.
>
> His current infrastructure cannot be easily converted to DRBD without
> major losses: if he attempts to do so he will have almost double the
> costs of a basic DRBD shared-nothing direct-attached infrastructure or
> exactly double the cost of a shared-disk infrastructure, intended as
> cost per TB of data. Unfortunately, after this he will still have half
> the performances of an active/passive shared-disk clustered-MD solution.
He doesn't have an infrastructure yet. He's attempting to build one but
purchased the wrong gear for his requirements. I presented him with
options to do it the right way, and to salvage what he has already
purchased. The DRBD active/active option is the latter. The SAN option
was the former. You seem to have misunderstood my comments.
Cheers,
Stan
^ permalink raw reply
* Re: Disks never stop spinning
From: Patrick Gautschi @ 2014-11-14 3:05 UTC (permalink / raw)
To: Peter Grandi; +Cc: Linux RAID
In-Reply-To: <21604.47814.651762.246947@tree.ty.sabi.co.uk>
[-- Attachment #1: Type: text/plain, Size: 583 bytes --]
> # hddtemp /dev/sd[bcdef]
> /dev/sdb: SAMSUNG HD103UJ: drive is sleeping
> /dev/sdc: WDC WD10EZEX-22RKKA0: drive is sleeping
> /dev/sdd: ST1000DM003-9YN162: drive is sleeping
> /dev/sde: SAMSUNG HD103SJ: drive is sleeping
> /dev/sdf: WDC WD10EZEX-22RKKA0: drive is sleeping
Different disks; looks like desktop types.
> That means something else is waking up those disks. To verify
> this use something like:
FS are not mounted and arrays stopped. There is nothing that wakes them up.
If a switch them off manually with hdparm -Y they stay off.
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4239 bytes --]
^ permalink raw reply
* Re: Disks never stop spinning
From: Brad Campbell @ 2014-11-14 4:11 UTC (permalink / raw)
To: Patrick Gautschi, Peter Grandi; +Cc: Linux RAID
In-Reply-To: <54657187.6020306@gautschi.net>
On 14/11/14 11:05, Patrick Gautschi wrote:
>> # hddtemp /dev/sd[bcdef]
>> /dev/sdb: SAMSUNG HD103UJ: drive is sleeping
>> /dev/sdc: WDC WD10EZEX-22RKKA0: drive is sleeping
>> /dev/sdd: ST1000DM003-9YN162: drive is sleeping
>> /dev/sde: SAMSUNG HD103SJ: drive is sleeping
>> /dev/sdf: WDC WD10EZEX-22RKKA0: drive is sleeping
>
> Different disks; looks like desktop types.
>
>> That means something else is waking up those disks. To verify
>> this use something like:
>
> FS are not mounted and arrays stopped. There is nothing that wakes them up.
> If a switch them off manually with hdparm -Y they stay off.
>
Are you polling them with smartmontools? I have a unit here where I had
to add the "standby" :
DEVICESCAN -s (L/../../7/07|S/../../[1-6]/00) -n standby,q -m root -M
exec /usr/share/smartmontools/smartd-runner
.. clause in smartd.conf. This ensures smartd does not wake the disk
every 15 minutes for it's poll, and only polls if the drive is already
spinning.
Failing that, I'd put it to sleep and it'd wake up on the next 15
minutes smart poll.
Regards,
Brad
^ permalink raw reply
* Re: failed RAID 5 array
From: DeadManMoving @ 2014-11-14 13:19 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid, DeadManMoving
In-Reply-To: <54653735.90007@turmel.org>
Hi Phil,
Thank you so much to have taken the time to write back to me.
I already tried --assemble --force, indeed and, that did not work. I
guess it can work if you have a single drive which is out of sync but in
my case, it is a mix of a drive with a problematic superblock (dmesg =
does not have a valid v1.2 superblock, not importing!) plus a drive
which is out of sync (dmesg = kicking non-fresh sdx from array!).
Here is the output of --assemble --force with double verbose :
# mdadm -vv --assemble
--force /dev/md127 /dev/sdf /dev/sdg /dev/sdh /dev/sdi
mdadm: looking for devices for /dev/md127
mdadm: /dev/sdf is busy - skipping
mdadm: /dev/sdh is busy - skipping
mdadm: /dev/sdi is busy - skipping
mdadm: Merging with already-assembled /dev/md/xyz
mdadm: /dev/sdi is identified as a member of /dev/md/xyz, slot 2.
mdadm: /dev/sdh is identified as a member of /dev/md/xyz, slot 3.
mdadm: /dev/sdf is identified as a member of /dev/md/xyz, slot 1.
mdadm: /dev/sdg is identified as a member of /dev/md/xyz, slot 0.
mdadm: /dev/sdf is already in /dev/md/xyz as 1
mdadm: /dev/sdi is already in /dev/md/xyz as 2
mdadm: /dev/sdh is already in /dev/md/xyz as 3
mdadm: failed to add /dev/sdg to /dev/md/xyz: Invalid argument
mdadm: failed to RUN_ARRAY /dev/md/xyz: Input/output error
If i stop the array (which was autostarted) and retry, similar output :
# mdadm -S /dev/md127
mdadm: stopped /dev/md127
# mdadm -vv --assemble
--force /dev/md127 /dev/sdf /dev/sdg /dev/sdh /dev/sdi
mdadm: looking for devices for /dev/md127
mdadm: /dev/sdf is identified as a member of /dev/md127, slot 1.
mdadm: /dev/sdg is identified as a member of /dev/md127, slot 0.
mdadm: /dev/sdh is identified as a member of /dev/md127, slot 3.
mdadm: /dev/sdi is identified as a member of /dev/md127, slot 2.
mdadm: added /dev/sdf to /dev/md127 as 1
mdadm: added /dev/sdi to /dev/md127 as 2
mdadm: added /dev/sdh to /dev/md127 as 3 (possibly out of date)
mdadm: failed to add /dev/sdg to /dev/md127: Invalid argument
mdadm: failed to RUN_ARRAY /dev/md127: Input/output error
Here is the relevant dmesg output :
[173174.307703] sdf: unknown partition table
[173174.308374] sdg: unknown partition table
[173174.308811] md: bind<sdf>
[173174.309385] sdh: unknown partition table
[173174.309552] md: bind<sdi>
[173174.310411] sdi: unknown partition table
[173174.310573] md: bind<sdh>
[173174.311299] sdi: unknown partition table
[173174.311449] md: invalid superblock checksum on sdg
[173174.311450] md: sdg does not have a valid v1.2 superblock, not
importing!
[173174.311460] md: md_import_device returned -22
[173174.311482] md: kicking non-fresh sdh from array!
[173174.311498] md: unbind<sdh>
[173174.311909] sdh: unknown partition table
[173174.338007] md: export_rdev(sdh)
[173174.338651] md/raid:md127: device sdi operational as raid disk 2
[173174.338652] md/raid:md127: device sdf operational as raid disk 1
[173174.338868] md/raid:md127: allocated 0kB
[173174.338880] md/raid:md127: not enough operational devices (2/4
failed)
[173174.338886] RAID conf printout:
[173174.338887] --- level:5 rd:4 wd:2
[173174.338887] disk 1, o:1, dev:sdf
[173174.338888] disk 2, o:1, dev:sdi
[173174.339013] md/raid:md127: failed to run raid set.
[173174.339014] md: pers->run() failed ...
Thanks again,
Tony
On Thu, 2014-11-13 at 17:56 -0500, Phil Turmel wrote:
> On 11/12/2014 10:58 AM, DeadManMoving wrote:
> > Hi list,
> >
> > I have a failed RAID 5 array, composed of 4 x 2TB drives without hot
> > spare. On the fail array, it looks like there is one drive out of sync
> > (the one with a lower Events counts) and another drive with a missing or
> > corrupted superblock (dmesg is reporting "does not have a valid v1.2
> > superblock, not importing!" and i have a : Checksum : 5608a55a -
> > expected 4108a55a).
> >
> > All drives seems good though, the problem was probably triggered by a a
> > broken communication between the external eSATA expansion card and
> > external drive enclosure (card, cable or backplane in the enclosure i
> > guess...).
> >
> > I am now in the process of making exact copies of the drives with dd to
> > other drives.
> >
> > I have an idea on how to try to get my data back but i would be happy if
> > someone could help/validate with the steps i intent to follow to get
> > there.
>
> --create is almost always a bad idea.
>
> Just use "mdadm -vv --assemble --force /dev/mdX /dev/sd[abcd]"
>
> One drive will be left behind (the bad superblock), but the stale one
> will be revived and you'll be able to start.
>
> If that doesn't work, show the output of the above command. Do NOT do
> an mdadm --create.
>
> Phil
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: failed RAID 5 array
From: Phil Turmel @ 2014-11-14 13:42 UTC (permalink / raw)
To: DeadManMoving; +Cc: linux-raid
In-Reply-To: <1415971164.4241.38.camel@lappy.neofreak.org>
On 11/14/2014 08:19 AM, DeadManMoving wrote:
> Hi Phil,
>
> Thank you so much to have taken the time to write back to me.
>
> I already tried --assemble --force, indeed and, that did not work. I
> guess it can work if you have a single drive which is out of sync but in
> my case, it is a mix of a drive with a problematic superblock (dmesg =
> does not have a valid v1.2 superblock, not importing!) plus a drive
> which is out of sync (dmesg = kicking non-fresh sdx from array!).
>
> Here is the output of --assemble --force with double verbose :
>
>
> # mdadm -vv --assemble
> --force /dev/md127 /dev/sdf /dev/sdg /dev/sdh /dev/sdi
> mdadm: looking for devices for /dev/md127
> mdadm: /dev/sdf is busy - skipping
> mdadm: /dev/sdh is busy - skipping
> mdadm: /dev/sdi is busy - skipping
> mdadm: Merging with already-assembled /dev/md/xyz
> mdadm: /dev/sdi is identified as a member of /dev/md/xyz, slot 2.
> mdadm: /dev/sdh is identified as a member of /dev/md/xyz, slot 3.
> mdadm: /dev/sdf is identified as a member of /dev/md/xyz, slot 1.
> mdadm: /dev/sdg is identified as a member of /dev/md/xyz, slot 0.
> mdadm: /dev/sdf is already in /dev/md/xyz as 1
> mdadm: /dev/sdi is already in /dev/md/xyz as 2
> mdadm: /dev/sdh is already in /dev/md/xyz as 3
> mdadm: failed to add /dev/sdg to /dev/md/xyz: Invalid argument
> mdadm: failed to RUN_ARRAY /dev/md/xyz: Input/output error
>
>
> If i stop the array (which was autostarted) and retry, similar output :
>
>
> # mdadm -S /dev/md127
> mdadm: stopped /dev/md127
> # mdadm -vv --assemble
> --force /dev/md127 /dev/sdf /dev/sdg /dev/sdh /dev/sdi
> mdadm: looking for devices for /dev/md127
> mdadm: /dev/sdf is identified as a member of /dev/md127, slot 1.
> mdadm: /dev/sdg is identified as a member of /dev/md127, slot 0.
> mdadm: /dev/sdh is identified as a member of /dev/md127, slot 3.
> mdadm: /dev/sdi is identified as a member of /dev/md127, slot 2.
> mdadm: added /dev/sdf to /dev/md127 as 1
> mdadm: added /dev/sdi to /dev/md127 as 2
> mdadm: added /dev/sdh to /dev/md127 as 3 (possibly out of date)
> mdadm: failed to add /dev/sdg to /dev/md127: Invalid argument
> mdadm: failed to RUN_ARRAY /dev/md127: Input/output error
>
>
> Here is the relevant dmesg output :
>
> [173174.307703] sdf: unknown partition table
> [173174.308374] sdg: unknown partition table
> [173174.308811] md: bind<sdf>
> [173174.309385] sdh: unknown partition table
> [173174.309552] md: bind<sdi>
> [173174.310411] sdi: unknown partition table
> [173174.310573] md: bind<sdh>
> [173174.311299] sdi: unknown partition table
> [173174.311449] md: invalid superblock checksum on sdg
> [173174.311450] md: sdg does not have a valid v1.2 superblock, not
> importing!
> [173174.311460] md: md_import_device returned -22
> [173174.311482] md: kicking non-fresh sdh from array!
> [173174.311498] md: unbind<sdh>
> [173174.311909] sdh: unknown partition table
> [173174.338007] md: export_rdev(sdh)
> [173174.338651] md/raid:md127: device sdi operational as raid disk 2
> [173174.338652] md/raid:md127: device sdf operational as raid disk 1
> [173174.338868] md/raid:md127: allocated 0kB
> [173174.338880] md/raid:md127: not enough operational devices (2/4
> failed)
> [173174.338886] RAID conf printout:
> [173174.338887] --- level:5 rd:4 wd:2
> [173174.338887] disk 1, o:1, dev:sdf
> [173174.338888] disk 2, o:1, dev:sdi
> [173174.339013] md/raid:md127: failed to run raid set.
> [173174.339014] md: pers->run() failed ...
Hmmm. Should have worked. Please show kernel version and mdadm
version. There have been bugs fixed in this area in the past couple years.
Also try "mdadm --assemble --force /dev/mdX /dev/sd[fhi]", leaving out
the bad disk.
If it still doesn't work, use alternate boot media, like systemrescuecd,
to get a current kernel and mdadm combination and try again. If that
works, get your critical backups before you do anything else.
Then you can reboot back to your normal kernel and it should assemble
degraded.
Phil
^ permalink raw reply
* Re: failed RAID 5 array
From: DeadManMoving @ 2014-11-14 14:08 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid, DeadManMoving
In-Reply-To: <546606D4.5070406@turmel.org>
Hi Phil,
Unfortunately, that does not work :
# mdadm --assemble --force /dev/md127 /dev/sd[fhi]
mdadm: /dev/md127 assembled from 2 drives - not enough to start the
array.
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md127 : inactive sdf[1](S) sdh[4](S) sdi[2](S)
5860540680 blocks super 1.2
unused devices: <none>
# mdadm -D /dev/md127
/dev/md127:
Version : 1.2
Raid Level : raid0
Total Devices : 3
Persistence : Superblock is persistent
State : inactive
Name : abc:xyz (local to host abc)
UUID : d707f577:a9e572d5:e5d5f10c:b232f15a
Events : 9370
Number Major Minor RaidDevice
- 8 80 - /dev/sdf
- 8 112 - /dev/sdh
- 8 128 - /dev/sdi
I don't think that booting with an alternate boot media will help me out
as kernel and mdadm software are quite recent :
# uname -r
3.14.14-gentoo
# mdadm -V
mdadm - v3.3.1 - 5th June 2014
Thanks again,
Tony
On Fri, 2014-11-14 at 08:42 -0500, Phil Turmel wrote:
> On 11/14/2014 08:19 AM, DeadManMoving wrote:
> > Hi Phil,
> >
> > Thank you so much to have taken the time to write back to me.
> >
> > I already tried --assemble --force, indeed and, that did not work. I
> > guess it can work if you have a single drive which is out of sync but in
> > my case, it is a mix of a drive with a problematic superblock (dmesg =
> > does not have a valid v1.2 superblock, not importing!) plus a drive
> > which is out of sync (dmesg = kicking non-fresh sdx from array!).
> >
> > Here is the output of --assemble --force with double verbose :
> >
> >
> > # mdadm -vv --assemble
> > --force /dev/md127 /dev/sdf /dev/sdg /dev/sdh /dev/sdi
> > mdadm: looking for devices for /dev/md127
> > mdadm: /dev/sdf is busy - skipping
> > mdadm: /dev/sdh is busy - skipping
> > mdadm: /dev/sdi is busy - skipping
> > mdadm: Merging with already-assembled /dev/md/xyz
> > mdadm: /dev/sdi is identified as a member of /dev/md/xyz, slot 2.
> > mdadm: /dev/sdh is identified as a member of /dev/md/xyz, slot 3.
> > mdadm: /dev/sdf is identified as a member of /dev/md/xyz, slot 1.
> > mdadm: /dev/sdg is identified as a member of /dev/md/xyz, slot 0.
> > mdadm: /dev/sdf is already in /dev/md/xyz as 1
> > mdadm: /dev/sdi is already in /dev/md/xyz as 2
> > mdadm: /dev/sdh is already in /dev/md/xyz as 3
> > mdadm: failed to add /dev/sdg to /dev/md/xyz: Invalid argument
> > mdadm: failed to RUN_ARRAY /dev/md/xyz: Input/output error
> >
> >
> > If i stop the array (which was autostarted) and retry, similar output :
> >
> >
> > # mdadm -S /dev/md127
> > mdadm: stopped /dev/md127
> > # mdadm -vv --assemble
> > --force /dev/md127 /dev/sdf /dev/sdg /dev/sdh /dev/sdi
> > mdadm: looking for devices for /dev/md127
> > mdadm: /dev/sdf is identified as a member of /dev/md127, slot 1.
> > mdadm: /dev/sdg is identified as a member of /dev/md127, slot 0.
> > mdadm: /dev/sdh is identified as a member of /dev/md127, slot 3.
> > mdadm: /dev/sdi is identified as a member of /dev/md127, slot 2.
> > mdadm: added /dev/sdf to /dev/md127 as 1
> > mdadm: added /dev/sdi to /dev/md127 as 2
> > mdadm: added /dev/sdh to /dev/md127 as 3 (possibly out of date)
> > mdadm: failed to add /dev/sdg to /dev/md127: Invalid argument
> > mdadm: failed to RUN_ARRAY /dev/md127: Input/output error
> >
> >
> > Here is the relevant dmesg output :
> >
> > [173174.307703] sdf: unknown partition table
> > [173174.308374] sdg: unknown partition table
> > [173174.308811] md: bind<sdf>
> > [173174.309385] sdh: unknown partition table
> > [173174.309552] md: bind<sdi>
> > [173174.310411] sdi: unknown partition table
> > [173174.310573] md: bind<sdh>
> > [173174.311299] sdi: unknown partition table
> > [173174.311449] md: invalid superblock checksum on sdg
> > [173174.311450] md: sdg does not have a valid v1.2 superblock, not
> > importing!
> > [173174.311460] md: md_import_device returned -22
> > [173174.311482] md: kicking non-fresh sdh from array!
> > [173174.311498] md: unbind<sdh>
> > [173174.311909] sdh: unknown partition table
> > [173174.338007] md: export_rdev(sdh)
> > [173174.338651] md/raid:md127: device sdi operational as raid disk 2
> > [173174.338652] md/raid:md127: device sdf operational as raid disk 1
> > [173174.338868] md/raid:md127: allocated 0kB
> > [173174.338880] md/raid:md127: not enough operational devices (2/4
> > failed)
> > [173174.338886] RAID conf printout:
> > [173174.338887] --- level:5 rd:4 wd:2
> > [173174.338887] disk 1, o:1, dev:sdf
> > [173174.338888] disk 2, o:1, dev:sdi
> > [173174.339013] md/raid:md127: failed to run raid set.
> > [173174.339014] md: pers->run() failed ...
>
> Hmmm. Should have worked. Please show kernel version and mdadm
> version. There have been bugs fixed in this area in the past couple years.
>
> Also try "mdadm --assemble --force /dev/mdX /dev/sd[fhi]", leaving out
> the bad disk.
>
> If it still doesn't work, use alternate boot media, like systemrescuecd,
> to get a current kernel and mdadm combination and try again. If that
> works, get your critical backups before you do anything else.
>
> Then you can reboot back to your normal kernel and it should assemble
> degraded.
>
> Phil
>
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox