* Re: RAID1 removing failed disk returns EBUSY
From: Joe Lawrence @ 2015-01-23 15:11 UTC (permalink / raw)
To: Xiao Ni; +Cc: NeilBrown, linux-raid, Bill Kuzeja
In-Reply-To: <1063248306.12205209.1421738206544.JavaMail.zimbra@redhat.com>
On Tue, 20 Jan 2015 02:16:46 -0500
Xiao Ni <xni@redhat.com> wrote:
> Joe
>
> Thanks for the explanation. So echo "idle" to sync_action is a workaround
> without the patch.
>
> It looks like the patch is not enough to fix the problem.
> Do you have a try with the new patch? Is the problem still exist in
> your environment?
>
> If your environment have no problem, can you give me the version number? I'll
> have a try with the same version too.
Hi Xiao,
Bill and I did some more testing yesterday and I think we've figured
out the confusion. Running a 3.18+ kernel and an upstream mdadm, it
was the udev invocation of "mdadm -If <dev>" that was automatically
removing the device for us.
If we ran with an older mdadm and got the MD wedged in the faulty
condition, then nothing we echoed into the sysfs state file ('idle'
'fail' or 'remove') would change anything. I think this agrees with
your testing report.
So two things:
1 - Did you make / make install the latest mdadm and see it try to run
mdadm -If on the removed disk? (You could also try manually running
it.)
2 - I think the sysfs interface to the removed disks is still broken in
cases where (1) doesn't occur.
Thanks,
-- Joe
^ permalink raw reply
* All drive in Raid 5 are in 'spare' mode
From: Dush @ 2015-01-22 23:23 UTC (permalink / raw)
To: linux-raid
Hi,
I have a RAID 5 composed by 4x 500Go hdd but for some days, it's 'inactive'.
I'm not raid expert and I prefer asking before doing an unrecoverable mistake...
Is it possible to fix this raid (md126)?
Is it possible to recover data on it?
Do I have a disk to change or it's "just" a desynchronization between disks?
Thanks for your help
# mdadm --version
mdadm - v3.3.2 - 21st August 2014
Linux Kernel: 3.18.2-2
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md127 : active raid5 sdd2[2] sde2[1] sdc2[0] sdb2[3]
16530624 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
md126 : inactive sde3[1](S) sdd3[4](S) sdc3[0](S) sdb3[3](S)
1929888000 blocks
unused devices: <none>
# mdadm --examine /dev/sd[a-z]3
mdadm: No md superblock detected on /dev/sda3.
/dev/sdb3:
Magic : a92b4efc
Version : 0.90.00
UUID : 3327f442:a00b59b2:1397f3c2:236c0edf
Creation Time : Tue Jan 27 13:03:52 2009
Raid Level : raid5
Used Dev Size : 482472000 (460.12 GiB 494.05 GB)
Array Size : 1447416000 (1380.36 GiB 1482.15 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 126
Update Time : Wed Jan 21 20:55:48 2015
State : active
Active Devices : 3
Working Devices : 4
Failed Devices : 1
Spare Devices : 1
Checksum : 6e656c69 - correct
Events : 49656
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 3 8 19 3 active sync /dev/sdb3
0 0 8 35 0 active sync /dev/sdc3
1 1 8 67 1 active sync /dev/sde3
2 2 0 0 2 faulty removed
3 3 8 19 3 active sync /dev/sdb3
4 4 8 51 4 spare /dev/sdd3
/dev/sdc3:
Magic : a92b4efc
Version : 0.90.00
UUID : 3327f442:a00b59b2:1397f3c2:236c0edf
Creation Time : Tue Jan 27 13:03:52 2009
Raid Level : raid5
Used Dev Size : 482472000 (460.12 GiB 494.05 GB)
Array Size : 1447416000 (1380.36 GiB 1482.15 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 126
Update Time : Wed Jan 21 23:34:52 2015
State : clean
Active Devices : 2
Working Devices : 3
Failed Devices : 2
Spare Devices : 1
Checksum : 6e6653d5 - correct
Events : 49666
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 0 8 35 0 active sync /dev/sdc3
0 0 8 35 0 active sync /dev/sdc3
1 1 8 67 1 active sync /dev/sde3
2 2 0 0 2 faulty removed
3 3 0 0 3 faulty removed
4 4 8 51 4 spare /dev/sdd3
/dev/sdd3:
Magic : a92b4efc
Version : 0.90.00
UUID : 3327f442:a00b59b2:1397f3c2:236c0edf
Creation Time : Tue Jan 27 13:03:52 2009
Raid Level : raid5
Used Dev Size : 482472000 (460.12 GiB 494.05 GB)
Array Size : 1447416000 (1380.36 GiB 1482.15 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 126
Update Time : Wed Jan 21 23:34:52 2015
State : clean
Active Devices : 2
Working Devices : 3
Failed Devices : 2
Spare Devices : 1
Checksum : 6e6653e7 - correct
Events : 49666
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 4 8 51 4 spare /dev/sdd3
0 0 8 35 0 active sync /dev/sdc3
1 1 8 67 1 active sync /dev/sde3
2 2 0 0 2 faulty removed
3 3 0 0 3 faulty removed
4 4 8 51 4 spare /dev/sdd3
/dev/sde3:
Magic : a92b4efc
Version : 0.90.00
UUID : 3327f442:a00b59b2:1397f3c2:236c0edf
Creation Time : Tue Jan 27 13:03:52 2009
Raid Level : raid5
Used Dev Size : 482472000 (460.12 GiB 494.05 GB)
Array Size : 1447416000 (1380.36 GiB 1482.15 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 126
Update Time : Wed Jan 21 23:34:52 2015
State : clean
Active Devices : 2
Working Devices : 3
Failed Devices : 2
Spare Devices : 1
Checksum : 6e6653f7 - correct
Events : 49666
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 1 8 67 1 active sync /dev/sde3
0 0 8 35 0 active sync /dev/sdc3
1 1 8 67 1 active sync /dev/sde3
2 2 0 0 2 faulty removed
3 3 0 0 3 faulty removed
4 4 8 51 4 spare /dev/sdd3
^ permalink raw reply
* Re: mdadm RAID6 "active" with spares and failed disks; need help
From: Valentijn @ 2015-01-22 9:47 UTC (permalink / raw)
To: Matt Callaghan, Wols Lists, linux-raid
In-Reply-To: <BLU436-SMTP1430CA88C11B0E46F15BB3F81480@phx.gbl>
Hi Matt,
As long as your data is still somewhere on these disks, all is not -
necessarily - lost. You could still try using dumpe2fs (and later
e2fsck) and/or dumpe2fs with different superblocks. And even if you
cannot find your file system by any means, you could try to use the
"foremost" utility to scrape off images, documents and the like from
these disks.
So I still don't think all is lost. However, I do think that will cost
more time. You may want to dedicate a spare machine to this task,
because of the resources.
I see that your mdadm says this, somewhere along your odyssee:
mdadm: /dev/sdk1 appears to contain an ext2fs file system
size=1695282944K mtime=Tue Apr 12 11:10:24 1977
... which could mean (I'm not sure, I'm just guessing) that due to the
internal bitmap, your fs has been overwritten.
Your new array in fact said:
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Jan 7 09:46:44 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : c7603819 - correct
Events : 0
... as far as I understand, this means that 8 blocks from the
superblock, some - whatever size - sectors were occupied by the Internal
Bitmap, which, in turn, would mean your filesystem superblock has been
overwritten.
The good news is: there is more than one superblock.
BTW, didn't you have the right raid drive ordering from the original
disks? You did have output of "mdadm --examine" after the array broke
down, didn't you? So your "create" statement is, by definition, correct
if a new "--examine" output shows the same output - hence the filesystem
is correct if the latter is the case?
So please try if "dumpe2fs -h -o superblock=32768" does anything. Or
98304, 163840, 229376. Dumpe2fs just dumps the fs header, nothing more.
If dumpe2fs doesn't do anything (but complain that it "Couldn't find
valid filesystem superblock"), then you could still try if "foremost"
finds anything. It's not that hard to use, you simply dedicate some
storage to it and tell it to scrape your array. It *will* find things
and it's up to you to see if
1) documents, images and the like are all 64K or 128K or less - and/or
contain large blocks of rubbish. This probably means you have the wrong
array config, because foremost in this case only finds single "chunks"
with correct data - if a file is longer, it doesn't find it and/or spews
out random data from other images
2) documents, images etcetera are OK. This means your array is OK. You
then can use foremost to scrape off everything (it may take weeks but it
could work), or simply try to find where the filesystem superblock hangs
out (if the array is in good order, the fs superblock must be somewhere,
right?)
Please, please try to do as little as possible on the real disks. Use
dmsetup to create snapshots. Copy the disks. Use hardware that is in
good state - you don't want to loose your data that you just found back
because the memory is flakey, do you? ;-)
I hope it's going to work.
Best regards,
Valentijn
On 01/21/15 01:34, Matt Callaghan wrote:
> I tried again with the --bitmap=none, clearly that was a miss on my part.
> However, still even with that correction, and attempting across varying
> combinations of "drive ordering", the filesystem appears corrupt.
^ permalink raw reply
* Re: [dm-devel] livelock on MD RAID5
From: Mikulas Patocka @ 2015-01-22 5:00 UTC (permalink / raw)
To: device-mapper development; +Cc: NeilBrown, linux-raid
In-Reply-To: <alpine.LRH.2.02.1501192207260.20870@file01.intranet.prod.int.rdu2.redhat.com>
On Mon, 19 Jan 2015, Mikulas Patocka wrote:
> Hi Neil
>
> I saw your patch "md/raid5: avoid livelock caused by non-aligned writes".
> That patch is only for RAID 6, but we are seeing similar livelock on RAID5
> (mdX_raid5 is constantly looping in
> raid5d/handle_active_stripes/handle_stripe/ops_run_io and some submitted
> bios get stuck). The livelock happens when we issue I/O and simulate disk
> failure.
>
> Do you think that the same issue could happen on RAID5 too?
>
> Do you think that the patch "md/raid5: fetch_block must fetch all the
> blocks handle_stripe_dirtying wants." could fix it?
>
> Mikulas
So, we verified that this RAID5 livelock was introduced in patch
67f455486d2ea20b2d94d6adf5b9b783d079e321. A patch
a40687ff73a5b14909d6aa522f7d778b158911c5 fixes a livelock in RAID6, but a
similar livelock was introduced in RAID5, that is still unfixed upstream.
Mikulas
^ permalink raw reply
* Re: kernel refuses devices mdadm -E accepts
From: Wesley W. Terpstra @ 2015-01-21 12:37 UTC (permalink / raw)
To: Valentijn Sessink; +Cc: linux-raid
In-Reply-To: <FCB084A5-816F-4954-971E-1C69C707F213@blub.net>
[-- Attachment #1: Type: text/plain, Size: 956 bytes --]
On Wed, Jan 21, 2015 at 12:28 AM, Valentijn Sessink <valentyn@blub.net> wrote:
> Make a snapshot using dmsetup.
Thanks for the tip. I did not know the blockdev --setro command.
I've attached the output of smartctl -A in smartctl.log, mdadm -E in
mdadm.log, and mdstat in mdstat.log.
The mdstat needs some explanation, perhaps
sd[a-d] are 3TB hard disks
sd[e-f] are SSDs
md121, md122, md125 : swap disks
md123, md124: uefi, boot
md126: PV with LV root
md125: bcache cache disk
what is missing is a raid5 on /dev/sd[a-d]4 which is the bcache backing device
Try: madam -A -o --run --force --freeze-reshape /dev/md120 /dev/sd[a-d]4
It failed the same as before. Logs in mdadm-a.log and dmesg.log
The system was in the middle of a reshape to move the data-offset from
5120 to 8192. No disks were added or removed. All disks are healthy. I
believe the system was rebooted mid-reshape.
Could this be a bug? That madam won't continue a data-offset reshape?
[-- Attachment #2: dmesg.log --]
[-- Type: application/octet-stream, Size: 669 bytes --]
[ 5.520447] md: sdc4 does not have a valid v1.2 superblock, not importing!
[ 5.520457] md: md_import_device returned -22
[ 5.536177] md: sdb4 does not have a valid v1.2 superblock, not importing!
[ 5.536203] md: md_import_device returned -22
[ 5.536328] md: sda4 does not have a valid v1.2 superblock, not importing!
[ 5.536337] md: md_import_device returned -22
[ 5.536441] md: sdd4 does not have a valid v1.2 superblock, not importing!
[ 5.536449] md: md_import_device returned -22
[ 5.544142] md: sdc4 does not have a valid v1.2 superblock, not importing!
[ 5.544155] md: md_import_device returned -22
[ 5.544176] md: md120 stopped.
[-- Attachment #3: mdadm-a.log --]
[-- Type: application/octet-stream, Size: 308 bytes --]
mdadm: failed to add /dev/sdc4 to /dev/md128: Invalid argument
mdadm: failed to add /dev/sdb4 to /dev/md128: Invalid argument
mdadm: failed to add /dev/sda4 to /dev/md128: Invalid argument
mdadm: failed to add /dev/sdd4 to /dev/md128: Invalid argument
mdadm: failed to RUN_ARRAY /dev/md128: Invalid argument
[-- Attachment #4: mdadm.log --]
[-- Type: application/octet-stream, Size: 3746 bytes --]
/dev/sda4:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 4f83a72f:c52aa18b:59d48798:d706483b
Name : pumpkin:backing (local to host pumpkin)
Creation Time : Sat Nov 30 13:48:44 2013
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 5842897920 (2786.11 GiB 2991.56 GB)
Array Size : 8764342272 (8358.33 GiB 8974.69 GB)
Used Dev Size : 5842894848 (2786.11 GiB 2991.56 GB)
Data Offset : 8192 sectors
Super Offset : 8 sectors
Unused Space : before=8104 sectors, after=0 sectors
State : clean
Device UUID : 65262ece:4cc1be5b:51627b0b:89b74d71
Update Time : Mon Jan 19 20:53:07 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 5558443c - correct
Events : 795592
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb4:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 4f83a72f:c52aa18b:59d48798:d706483b
Name : pumpkin:backing (local to host pumpkin)
Creation Time : Sat Nov 30 13:48:44 2013
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 5842897920 (2786.11 GiB 2991.56 GB)
Array Size : 8764342272 (8358.33 GiB 8974.69 GB)
Used Dev Size : 5842894848 (2786.11 GiB 2991.56 GB)
Data Offset : 8192 sectors
Super Offset : 8 sectors
Unused Space : before=8104 sectors, after=0 sectors
State : clean
Device UUID : 62b10c04:5207bada:02ca566e:3df26814
Update Time : Mon Jan 19 20:53:07 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 1028b7a3 - correct
Events : 795592
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc4:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 4f83a72f:c52aa18b:59d48798:d706483b
Name : pumpkin:backing (local to host pumpkin)
Creation Time : Sat Nov 30 13:48:44 2013
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 5842897920 (2786.11 GiB 2991.56 GB)
Array Size : 8764342272 (8358.33 GiB 8974.69 GB)
Used Dev Size : 5842894848 (2786.11 GiB 2991.56 GB)
Data Offset : 8192 sectors
Super Offset : 8 sectors
Unused Space : before=8112 sectors, after=0 sectors
State : clean
Device UUID : 86bfac3b:c4c39510:1b75d729:8c8b8e9f
Update Time : Mon Jan 19 20:53:07 2015
Checksum : c442c656 - correct
Events : 795592
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd4:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 4f83a72f:c52aa18b:59d48798:d706483b
Name : pumpkin:backing (local to host pumpkin)
Creation Time : Sat Nov 30 13:48:44 2013
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 5842897920 (2786.11 GiB 2991.56 GB)
Array Size : 8764342272 (8358.33 GiB 8974.69 GB)
Used Dev Size : 5842894848 (2786.11 GiB 2991.56 GB)
Data Offset : 8192 sectors
Super Offset : 8 sectors
Unused Space : before=8112 sectors, after=0 sectors
State : clean
Device UUID : 68fa16c1:47652504:c49b4591:9399790e
Update Time : Mon Jan 19 20:53:07 2015
Checksum : 1395d76b - correct
Events : 795592
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
[-- Attachment #5: mdstat.log --]
[-- Type: application/octet-stream, Size: 686 bytes --]
Personalities : [raid1]
md121 : active raid1 sdb3[1] sda3[0]
8388608 blocks super 1.2 [2/2] [UU]
md122 : active raid1 sdc3[0] sdd3[1]
8388608 blocks super 1.2 [2/2] [UU]
md123 : active raid1 sdc1[2] sdb1[1] sda1[0] sdd1[3]
204800 blocks super 1.0 [4/4] [UUUU]
md124 : active raid1 sdc2[2] sda2[0] sdb2[1] sdd2[3]
204800 blocks super 1.2 [4/4] [UUUU]
md125 : active raid1 sde3[0] sdf3[1]
83886080 blocks super 1.2 [2/2] [UU]
md126 : active raid1 sde2[0] sdf2[1]
20975616 blocks super 1.2 [2/2] [UU]
md127 : active raid1 sde1[1] sdf1[0]
8388608 blocks super 1.2 [2/2] [UU]
unused devices: <none>
[-- Attachment #6: smartctl.log --]
[-- Type: application/octet-stream, Size: 7661 bytes --]
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-3.17.7] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 146 145 021 Pre-fail Always - 9658
4 Start_Stop_Count 0x0032 056 056 000 Old_age Always - 44077
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 076 076 000 Old_age Always - 17882
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 101
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 51
193 Load_Cycle_Count 0x0032 158 158 000 Old_age Always - 127271
194 Temperature_Celsius 0x0022 121 086 000 Old_age Always - 31
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-3.17.7] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 148 145 021 Pre-fail Always - 9591
4 Start_Stop_Count 0x0032 055 055 000 Old_age Always - 45623
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 076 076 000 Old_age Always - 18209
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 97
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 49
193 Load_Cycle_Count 0x0032 161 161 000 Old_age Always - 117768
194 Temperature_Celsius 0x0022 120 088 000 Old_age Always - 32
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-3.17.7] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 176 173 021 Pre-fail Always - 6183
4 Start_Stop_Count 0x0032 095 095 000 Old_age Always - 5666
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 082 082 000 Old_age Always - 13432
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 65
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 38
193 Load_Cycle_Count 0x0032 001 001 000 Old_age Always - 702407
194 Temperature_Celsius 0x0022 121 094 000 Old_age Always - 29
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-3.17.7] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 150 147 021 Pre-fail Always - 9491
4 Start_Stop_Count 0x0032 094 094 000 Old_age Always - 6520
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 094 094 000 Old_age Always - 4958
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 120
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 63
193 Load_Cycle_Count 0x0032 197 197 000 Old_age Always - 9745
194 Temperature_Celsius 0x0022 121 097 000 Old_age Always - 31
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
^ permalink raw reply
* Expanding partitions -> missing md on kernel load
From: Scott Tablett @ 2015-01-21 4:40 UTC (permalink / raw)
To: linux-raid
Migrated two drive RAID 1 array (1.2 metadata) to larger drives (larger
gpt partitions)
Followed steps on https://raid.wiki.kernel.org/index.php/Growing
Before using mdadm --grow, the system reliably booted for months,
automatically discovering, mounting md.
Noticed one of next two steps involved fsck. I understand fsck should
be performed on unmounted filesytem. So, changed to runlevel 1 with
init 1 and unmount md.
Ran mdadm --grow --size=max on unmounted md. Command exited normally.
Ran fsck and resize2fs. Commands exited normally. Mounted md and
verified file system contents.
Did same steps for another md and restarted services with CTL+D. Noticed
resync taking place using Disk Utility. After resync completed, NFS
service froze, so decided to test reboot.
The two Expanded md (/var and /home) are no longer automatically
assembled by kernel. The md can be assembled by using mdadm --assemble.
Is there a way to get the two Expanded md to auto assemble again?
Tried using mdadm --create with one drive, but could not mount md (error
message). Guessing it needs formatting after --create?
^ permalink raw reply
* Re: mdadm RAID6 "active" with spares and failed disks; need help
From: Matt Callaghan @ 2015-01-21 0:34 UTC (permalink / raw)
To: Wols Lists, linux-raid
In-Reply-To: <54B3F7ED.1000809@youngman.org.uk>
Thanks Wols and Valentijn for your input.
I tried again with the --bitmap=none, clearly that was a miss on my part.
However, still even with that correction, and attempting across varying
combinations of "drive ordering", the filesystem appears corrupt.
I think I have to accept entire data loss here. :(
~Matt
-------- Original Message --------
From: Wols Lists <antlists@youngman.org.uk>
Sent: 1/12/2015, 11:35:57 AM
To: linux-raid@vger.kernel.org
Cc:
Subject: Re: mdadm RAID6 "active" with spares and failed disks; need help
> On 11/01/15 23:22, Valentijn Sessink wrote:
Also, I would not have dared to run all these statements on "live" (or
dead, for that matter ;-) disks.
I'm no expert either, but looking at the blog, I'm worried he might have
trashed the array with almost the first thing he did :-(
"add" says it will add a drive as spare if it didn't originally belong
to the array. If it adds a spare to a degraded array, the array will
immediately start to repair itself.
OOPS!!! As it sounds like exactly this could have happened - mdadm
didn't recognise the disk as it added it.
Just speculating, but unfortunately this seems quite likely :-(
Cheers,
Wol
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* RE: BUG in "RAID5: batch adjacent full stripe write" - commit 1b8035b0a84a69b3148bbfa1df1350501d0b0869
From: Sushma Gurram @ 2015-01-21 0:05 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
In-Reply-To: <B9FF37F8E2D7894A9493A5314D0816FC24226CEA@SACMBXIP01.sdcorp.global.sandisk.com>
Apologies for resending multiple times. Not sure why the first email didn't end up on the mailing list, but the second one did!!! To keep the whole email context, resending it.
Hi Neil/Shaohua,
The following commit sometimes crashes due to an assert in the function ops_run_biodrain().
http://git.neil.brown.name/?p=md.git;a=commit;h=1b8035b0a84a69b3148bbfa1df1350501d0b0869
The assert which causes the crash is:
BUG_ON(dev->written);
Here's the call trace for the crash.
2580 <4>[53646.100012] [<ffffffffa09e1f80>] __raid_run_ops+0x330/0x470 [raid456]
2581 <4>[53646.100024] [<ffffffffa09e6334>] handle_stripe+0x654/0xda0 [raid456]
2582 <4>[53646.100037] [<ffffffffa09e6b10>] handle_active_stripes+0x90/0x150 [raid456]
2583 <4>[53646.100049] [<ffffffffa09e6c58>] raid5_do_work+0x88/0x120 [raid456]
2584 <4>[53646.100061] [<ffffffff81082812>] kthread_worker_fn+0xd2/0x190
2585 <4>[53646.100069] [<ffffffff81082966>] kthread+0x96/0xa0
2586 <4>[53646.100076] [<ffffffff81469ee4>] kernel_thread_helper+0x4/0x10
2587 <0>[53646.100082] Code: 89 c0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 be 97 05 00 00 48 c7 c7 08 b8 9e a0 4c 89 04 24 e8 08 fb 67 e0 4c 8b 04 24 e9 ba fe ff ff <0f> 0b eb fe 48 8b 4f 38 48 c7 c2 a0 b5 9e a0 48 c7 c6 57 c8 9e
2588 <1>[53646.100118] RIP [<ffffffffa09e0c51>] ops_run_biodrain+0x261/0x2a0 [raid456]
2589 <4>[53646.100125] RSP <ffff8803dd39fb90>
Reproduction steps:
1. Create a 3 drive RAID5 with chunk=32K
2. Run 4K sequential writes using fio
Root cause:
1. Typical code flow when an IO is submitted:
make_request() -> add_stripe_bio() -> stripe_add_to_batch_list()
This puts the stripe in the handle_list OR batch_list.
All non-batched stripes are added to handle_list.
If adjacent stripes can be batched together, the first stripe in the batch is added to handle_list and the remaining stripes are added to batch_list with the batch_head being the first stripe in the batch.
As a result, there will be fewer stripes processed by handle_active_stripes().
Note that only full stripe writes are batched.
2. Stripe processing path:
raid5d()/raid5_do_work() -> handle_active_stripes() -> handle_stripe()
In handle_active_stripes(), stripes are pulled from the handle_list and added to a local array (struct stripe_head *batch[MAX_STRIPE_BATCH]), before calling handle_stripe()
Ideally, we would not expect non-head stripes in the batch_list to be processed by handle_stripe(). However, in the following scenario handle_stripe() operates on non-head batched stripe.
Let’s assume there are two adjacent stripes s1 and s2 on a 3 drive RAID5.
Following is the sequence in which IO happens:
1. make_request() for s1:Disk1 => add s1 to handle_list, batch_head=NULL
2. make_request() for s2:Disk1 => add s2 to handle_list, batch_head=NULL
3. make_request() for s1:Disk2 => add s1 to handle_list, batch_head=NULL
4. make_request() for s2:Disk2 => s1 and s2 can be batched (full stripes) and added to batch_list. Set batch_head=s1 for both stripes. Add only head stripe i.e. s1 to handle_list
If both stripes are processed (in handle_active_stripes) after Step 4, we would have no issues.
However, if handle_active_stripes() is called after Step 2, it’ll remove stripe s2 from handle_list and adds it to the local array. It later calls handle_stripe() for all stripes in the local array.
In handle_stripe(), STRIPE_BATCH_READY flag is cleared which would prevent future IOs to be batched with the active stripe.
However, note that there is a tiny window between handle_active_stripes() pulling the request from handle_list and handle_stripe() clearing STRIPE_BATCH_READY.
It’s in this window that stripe s2 is added to batch_list (with batch_head=s1) and at the same time goes through stripe processing state machine.
During stripe s1 processing, stripe s2 is again incorrectly processed in ops_run_biodrain (because s2 is in its batch_list) and thus causes the kernel assert in ops_run_biodrain().
From the code, it appears that the following check in clear_batch_ready() called from handle_stripe() should handle the above case:
static int clear_batch_ready(struct stripe_head *sh) { ...
/*
* this stripe could be added to a batch list before we check
* BATCH_READY, skips it
*/
if (sh->batch_head != sh) {
spin_unlock(&sh->stripe_lock);
return 1;
}
...
}
However, by the time clear_batch_ready() is called on stripe s2, STRIPE_BATCH_READY bit would have been cleared due to stripe s1's invocation of clear_batch_ready() due to the following code.
static int clear_batch_ready(struct stripe_head *sh) { ....
list_for_each_entry(tmp, &sh->batch_list, batch_list)
clear_bit(STRIPE_BATCH_READY, &tmp->state); ...
}
Hence when clear_batch_ready() is called for stripe s2, it returns from the first "if" check which tests STRIPE_BATCH_READY, even before getting to (sh->head != sh) check. Hence, handle_stripe() incorrectly continues with stripe s2.
Potential fix:
I moved the clear_batch_ready() from handle_stripe() to handle_active_stripes() before adding to the local array as follows:
static int handle_active_stripes(struct r5conf *conf, int group,
struct r5worker *worker,
struct list_head *temp_inactive_list) { ...
while (batch_size < MAX_STRIPE_BATCH &&
(sh = __get_priority_stripe(conf, group)) != NULL) {
if (!clear_batch_ready(sh))
batch[batch_size++] = sh;
}
...
}
This would ensure that the STRIPE_BATCH_READY is cleared before adding it batch[] array in handle_active_stripes().
This prevents new IO requests to be batched with the stripe being processed in handle_active_stripes().
Please let me know if the above fix seems correct.
Thanks,
Sushma
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
^ permalink raw reply
* RE: BUG in "RAID5: batch adjacent full stripe write" - commit 1b8035b0a84a69b3148bbfa1df1350501d0b0869
From: Sushma Gurram @ 2015-01-20 23:57 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
Sorry, the call trace in the previous email is incorrect. It should have been this:
<0>[ 1483.429759] invalid opcode: 0000 [#1] SMP
<4>[ 1483.429761] CPU 1
<4>[ 1483.429762] Modules linked in: raid456(F) async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx raid0(F) raid1(F) joydev ip6t_REJECT ip6t_LOG nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw xt_NOTRACK ipt_REJECT xt_tcpudp xt_pkttype ipt_LOG xt_limit xt_state iptable_raw ip6table_mangle nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip6table_filter ip6_tables scst_vdisk(PFN) iscsi_scst(FN) scst(PFN) scst_mods(FN) dlm(FN) configfs sctp(F) mptctl mptbase ipmi_si ipmi_devintf ipmi_msghandler iptable_filter ip_tables x_tables af_packet cpufreq_conservative cpufreq_userspace cpufreq_powersave pcc_cpufreq mperf binfmt_misc iomemory_vsl(PFN) dm_mod tg3 ptp sr_mod hpwdt cdrom pps_core hpilo sg ipv6 ipv6_lib serio_raw button container acpi_power_meter rtc_cmos ttm drm_kms_helper drm i2c_algo_bit sysimgblt sysfillrect i2c_core syscopyarea btrfs zlib_deflate crc32c libcrc32c sd_mod crc_t10dif usbhid hid uhci_hcd ahci libahci libata ehci_hcd processor thermal_sys hwmon scsi_dh_rdac scsi_dh_hp_sw scsi_dh_emc scsi_dh_alua scsi_dh scsi_mod asix usbnet usbcore usb_common mii [last unloaded: loop]
<4>[ 1483.429819] Supported: No, Proprietary and Unsupported modules are loaded
<4>[ 1483.429821]
<4>[ 1483.429823] Pid: 22628, comm: raid5auxd0_1 Tainted: PF N 3.0.101-0.15.1.6651.0.PTF-default #1 HP ProLiant ML310e Gen8
<4>[ 1483.429828] RIP: 0010:[<ffffffffa0921c51>] [<ffffffffa0921c51>] ops_run_biodrain+0x261/0x2a0 [raid456]
<4>[ 1483.429834] RSP: 0018:ffff88014f6ffb90 EFLAGS: 00010086
<4>[ 1483.429836] RAX: 00000000000000c0 RBX: 0000000000000000 RCX: ffff8801482bf188
<4>[ 1483.429838] RDX: 0000000000000025 RSI: 00000000000001e0 RDI: ffff8801482bf130
<4>[ 1483.429840] RBP: ffff8801482bf130 R08: 0000000000000000 R09: 0000000000000002
<4>[ 1483.429842] R10: ffff8803efc55000 R11: 0000000000000002 R12: ffff8801482bf0c8
<4>[ 1483.429844] R13: ffff8803efc556a0 R14: ffff8801482bf0c8 R15: ffff8801482bf110
<4>[ 1483.429846] FS: 0000000000000000(0000) GS:ffff88040ae20000(0000) knlGS:0000000000000000
<4>[ 1483.429848] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
<4>[ 1483.429850] CR2: 00007f66e5c0d000 CR3: 00000003bf3f5000 CR4: 00000000001407e0
<4>[ 1483.429852] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>[ 1483.429854] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<4>[ 1483.429857] Process raid5auxd0_1 (pid: 22628, threadinfo ffff88014f6fe000, task ffff88034a1fc4c0)
<0>[ 1483.429859] Stack:
<4>[ 1483.429860] 0000000000000000 00000000000001e0 0000000000000210 00000000000000c0
<4>[ 1483.429863] ffffffffa0921240 ffff8801483bf980 ffff8801482bf0c8 ffff8801483bfb68
<4>[ 1483.429866] 000000014090de00 0000000000000000 ffff8801482bf0c8 ffff8801482bf0c8
<0>[ 1483.429869] Call Trace:
<4>[ 1483.429888] [<ffffffffa0922f80>] __raid_run_ops+0x330/0x470 [raid456]
<4>[ 1483.429894] [<ffffffffa0927364>] handle_stripe+0x654/0xda0 [raid456]
<4>[ 1483.429902] [<ffffffffa0927b40>] handle_active_stripes+0x90/0x150 [raid456]
<4>[ 1483.429909] [<ffffffffa0927c88>] raid5_do_work+0x88/0x120 [raid456]
<4>[ 1483.429916] [<ffffffff81082812>] kthread_worker_fn+0xd2/0x190
<4>[ 1483.429920] [<ffffffff81082966>] kthread+0x96/0xa0
<4>[ 1483.429924] [<ffffffff81469ee4>] kernel_thread_helper+0x4/0x10
<0>[ 1483.429927] Code: 89 c0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 be 97 05 00 00 48 c7 c7 08 c8 92 a0 4c 89 04 24 e8 08 eb 73 e0 4c 8b 04 24 e9 ba fe ff ff <0f> 0b eb fe 48 8b 4f 38 48 c7 c2 a0 c5 92 a0 48 c7 c6 87 d8 92
<1>[ 1483.429945] RIP [<ffffffffa0921c51>] ops_run_biodrain+0x261/0x2a0 [raid456]
<4>[ 1483.429949] RSP <ffff88014f6ffb90>
Thanks,
Sushma
-----Original Message-----
From: Sushma Gurram
Sent: Tuesday, January 20, 2015 3:35 PM
To: linux-raid@vger.kernel.org
Subject: BUG in "RAID5: batch adjacent full stripe write" - commit 1b8035b0a84a69b3148bbfa1df1350501d0b0869
Hi Neil/Shaohua,
The following commit sometimes crashes due to an assert in the function ops_run_biodrain().
http://git.neil.brown.name/?p=md.git;a=commit;h=1b8035b0a84a69b3148bbfa1df1350501d0b0869
The assert which causes the crash is:
BUG_ON(dev->written);
Here's the call trace for the crash.
2351 <0>[ 864.952605] Call Trace:
2352 <4>[ 864.952616] [<ffffffff81263919>] flex_array_free+0x9/0x20
2353 <4>[ 864.952622] [<ffffffffa09c5de6>] raid5_free_percpu+0x56/0xc0 [raid456]
2354 <4>[ 864.952644] [<ffffffffa09c5eba>] free_conf+0x6a/0x90 [raid456]
2355 <4>[ 864.952651] [<ffffffffa09c75ff>] stop+0x3f/0x70 [raid456]
2356 <4>[ 864.952661] [<ffffffff8136746f>] md_stop+0x2f/0x80
2357 <4>[ 864.952667] [<ffffffff8136efa6>] do_md_stop+0x86/0x470
2358 <4>[ 864.952672] [<ffffffff81371f26>] md_ioctl+0xba6/0xc90
2359 <4>[ 864.952678] [<ffffffff8122f390>] blkdev_ioctl+0x2a0/0x710
2360 <4>[ 864.952686] [<ffffffff8118dbf5>] block_ioctl+0x35/0x40
2361 <4>[ 864.952694] [<ffffffff8116c01b>] do_vfs_ioctl+0x8b/0x3b0
2362 <4>[ 864.952698] [<ffffffff8116c3e1>] sys_ioctl+0xa1/0xb0
2363 <4>[ 864.952705] [<ffffffff81468d92>] system_call_fastpath+0x16/0x1b
2364 <4>[ 864.952712] [<00007fbef8190fe7>] 0x7fbef8190fe6
Reproduction steps:
1. Create a 3 drive RAID5 with chunk=32K 2. Run 4K sequential writes using fio
Root cause:
1. Typical code flow when an IO is submitted:
make_request() -> add_stripe_bio() -> stripe_add_to_batch_list()
This puts the stripe in the handle_list OR batch_list.
All non-batched stripes are added to handle_list.
If adjacent stripes can be batched together, the first stripe in the batch is added to handle_list and the remaining stripes are added to batch_list with the batch_head being the first stripe in the batch.
As a result, there will be fewer stripes processed by handle_active_stripes().
Note that only full stripe writes are batched.
2. Stripe processing path:
raid5d()/raid5_do_work() -> handle_active_stripes() -> handle_stripe()
In handle_active_stripes(), stripes are pulled from the handle_list and added to a local array (struct stripe_head *batch[MAX_STRIPE_BATCH]), before calling handle_stripe()
Ideally, we would not expect non-head stripes in the batch_list to be processed by handle_stripe(). However, in the following scenario handle_stripe() operates on non-head batched stripe.
Let’s assume there are two adjacent stripes s1 and s2 on a 3 drive RAID5.
Following is the sequence in which IO happens:
1. make_request() for s1:Disk1 => add s1 to handle_list, batch_head=NULL
2. make_request() for s2:Disk1 => add s2 to handle_list, batch_head=NULL
3. make_request() for s1:Disk2 => add s1 to handle_list, batch_head=NULL
4. make_request() for s2:Disk2 => s1 and s2 can be batched (full stripes) and added to batch_list. Set batch_head=s1 for both stripes. Add only head stripe i.e. s1 to handle_list
If both stripes are processed (in handle_active_stripes) after Step 4, we would have no issues.
However, if handle_active_stripes() is called after Step 2, it’ll remove stripe s2 from handle_list and adds it to the local array. It later calls handle_stripe() for all stripes in the local array.
In handle_stripe(), STRIPE_BATCH_READY flag is cleared which would prevent future IOs to be batched with the active stripe.
However, note that there is a tiny window between handle_active_stripes() pulling the request from handle_list and handle_stripe() clearing STRIPE_BATCH_READY.
It’s in this window that stripe s2 is added to batch_list (with batch_head=s1) and at the same time goes through stripe processing state machine.
During stripe s1 processing, stripe s2 is again incorrectly processed in ops_run_biodrain (because s2 is in its batch_list) and thus causes the kernel assert in ops_run_biodrain().
From the code, it appears that the following check in clear_batch_ready() called from handle_stripe() should handle the above case:
static int clear_batch_ready(struct stripe_head *sh) { ...
/*
* this stripe could be added to a batch list before we check
* BATCH_READY, skips it
*/
if (sh->batch_head != sh) {
spin_unlock(&sh->stripe_lock);
return 1;
}
...
}
However, by the time clear_batch_ready() is called on stripe s2, STRIPE_BATCH_READY bit would have been cleared due to stripe s1's invocation of clear_batch_ready() due to the following code.
static int clear_batch_ready(struct stripe_head *sh) { ....
list_for_each_entry(tmp, &sh->batch_list, batch_list)
clear_bit(STRIPE_BATCH_READY, &tmp->state); ...
}
Hence when clear_batch_ready() is called for stripe s2, it returns from the first "if" check which tests STRIPE_BATCH_READY, even before getting to (sh->head != sh) check. Hence, handle_stripe() incorrectly continues with stripe s2.
Potential fix:
I moved the clear_batch_ready() from handle_stripe() to handle_active_stripes() before adding to the local array as follows:
static int handle_active_stripes(struct r5conf *conf, int group,
struct r5worker *worker,
struct list_head *temp_inactive_list) { ...
while (batch_size < MAX_STRIPE_BATCH &&
(sh = __get_priority_stripe(conf, group)) != NULL) {
if (!clear_batch_ready(sh))
batch[batch_size++] = sh;
}
...
}
This would ensure that the STRIPE_BATCH_READY is cleared before adding it batch[] array in handle_active_stripes().
This prevents new IO requests to be batched with the stripe being processed in handle_active_stripes().
Please let me know if the above fix seems correct.
Thanks,
Sushma
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
^ permalink raw reply
* BUG in "RAID5: batch adjacent full stripe write" - commit 1b8035b0a84a69b3148bbfa1df1350501d0b0869
From: Sushma Gurram @ 2015-01-20 23:34 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
Hi Neil/Shaohua,
The following commit sometimes crashes due to an assert in the function ops_run_biodrain().
http://git.neil.brown.name/?p=md.git;a=commit;h=1b8035b0a84a69b3148bbfa1df1350501d0b0869
The assert which causes the crash is:
BUG_ON(dev->written);
Here's the call trace for the crash.
2351 <0>[ 864.952605] Call Trace:
2352 <4>[ 864.952616] [<ffffffff81263919>] flex_array_free+0x9/0x20
2353 <4>[ 864.952622] [<ffffffffa09c5de6>] raid5_free_percpu+0x56/0xc0 [raid456]
2354 <4>[ 864.952644] [<ffffffffa09c5eba>] free_conf+0x6a/0x90 [raid456]
2355 <4>[ 864.952651] [<ffffffffa09c75ff>] stop+0x3f/0x70 [raid456]
2356 <4>[ 864.952661] [<ffffffff8136746f>] md_stop+0x2f/0x80
2357 <4>[ 864.952667] [<ffffffff8136efa6>] do_md_stop+0x86/0x470
2358 <4>[ 864.952672] [<ffffffff81371f26>] md_ioctl+0xba6/0xc90
2359 <4>[ 864.952678] [<ffffffff8122f390>] blkdev_ioctl+0x2a0/0x710
2360 <4>[ 864.952686] [<ffffffff8118dbf5>] block_ioctl+0x35/0x40
2361 <4>[ 864.952694] [<ffffffff8116c01b>] do_vfs_ioctl+0x8b/0x3b0
2362 <4>[ 864.952698] [<ffffffff8116c3e1>] sys_ioctl+0xa1/0xb0
2363 <4>[ 864.952705] [<ffffffff81468d92>] system_call_fastpath+0x16/0x1b
2364 <4>[ 864.952712] [<00007fbef8190fe7>] 0x7fbef8190fe6
Reproduction steps:
1. Create a 3 drive RAID5 with chunk=32K
2. Run 4K sequential writes using fio
Root cause:
1. Typical code flow when an IO is submitted:
make_request() -> add_stripe_bio() -> stripe_add_to_batch_list()
This puts the stripe in the handle_list OR batch_list.
All non-batched stripes are added to handle_list.
If adjacent stripes can be batched together, the first stripe in the batch is added to handle_list and the remaining stripes are added to batch_list with the batch_head being the first stripe in the batch.
As a result, there will be fewer stripes processed by handle_active_stripes().
Note that only full stripe writes are batched.
2. Stripe processing path:
raid5d()/raid5_do_work() -> handle_active_stripes() -> handle_stripe()
In handle_active_stripes(), stripes are pulled from the handle_list and added to a local array (struct stripe_head *batch[MAX_STRIPE_BATCH]), before calling handle_stripe()
Ideally, we would not expect non-head stripes in the batch_list to be processed by handle_stripe(). However, in the following scenario handle_stripe() operates on non-head batched stripe.
Let’s assume there are two adjacent stripes s1 and s2 on a 3 drive RAID5.
Following is the sequence in which IO happens:
1. make_request() for s1:Disk1 => add s1 to handle_list, batch_head=NULL
2. make_request() for s2:Disk1 => add s2 to handle_list, batch_head=NULL
3. make_request() for s1:Disk2 => add s1 to handle_list, batch_head=NULL
4. make_request() for s2:Disk2 => s1 and s2 can be batched (full stripes) and added to batch_list. Set batch_head=s1 for both stripes. Add only head stripe i.e. s1 to handle_list
If both stripes are processed (in handle_active_stripes) after Step 4, we would have no issues.
However, if handle_active_stripes() is called after Step 2, it’ll remove stripe s2 from handle_list and adds it to the local array. It later calls handle_stripe() for all stripes in the local array.
In handle_stripe(), STRIPE_BATCH_READY flag is cleared which would prevent future IOs to be batched with the active stripe.
However, note that there is a tiny window between handle_active_stripes() pulling the request from handle_list and handle_stripe() clearing STRIPE_BATCH_READY.
It’s in this window that stripe s2 is added to batch_list (with batch_head=s1) and at the same time goes through stripe processing state machine.
During stripe s1 processing, stripe s2 is again incorrectly processed in ops_run_biodrain (because s2 is in its batch_list) and thus causes the kernel assert in ops_run_biodrain().
From the code, it appears that the following check in clear_batch_ready() called from handle_stripe() should handle the above case:
static int clear_batch_ready(struct stripe_head *sh)
{
...
/*
* this stripe could be added to a batch list before we check
* BATCH_READY, skips it
*/
if (sh->batch_head != sh) {
spin_unlock(&sh->stripe_lock);
return 1;
}
...
}
However, by the time clear_batch_ready() is called on stripe s2, STRIPE_BATCH_READY bit would have been cleared due to stripe s1's invocation of clear_batch_ready() due to the following code.
static int clear_batch_ready(struct stripe_head *sh)
{
....
list_for_each_entry(tmp, &sh->batch_list, batch_list)
clear_bit(STRIPE_BATCH_READY, &tmp->state);
...
}
Hence when clear_batch_ready() is called for stripe s2, it returns from the first "if" check which tests STRIPE_BATCH_READY, even before getting to (sh->head != sh) check. Hence, handle_stripe() incorrectly continues with stripe s2.
Potential fix:
I moved the clear_batch_ready() from handle_stripe() to handle_active_stripes() before adding to the local array as follows:
static int handle_active_stripes(struct r5conf *conf, int group,
struct r5worker *worker,
struct list_head *temp_inactive_list)
{
...
while (batch_size < MAX_STRIPE_BATCH &&
(sh = __get_priority_stripe(conf, group)) != NULL) {
if (!clear_batch_ready(sh))
batch[batch_size++] = sh;
}
...
}
This would ensure that the STRIPE_BATCH_READY is cleared before adding it batch[] array in handle_active_stripes().
This prevents new IO requests to be batched with the stripe being processed in handle_active_stripes().
Please let me know if the above fix seems correct.
Thanks,
Sushma
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
^ permalink raw reply
* Re: kernel refuses devices mdadm -E accepts
From: Wesley W. Terpstra @ 2015-01-20 18:43 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
In-Reply-To: <54BE9F2A.9090102@turmel.org>
An email race condition. ;-)
On Tue, Jan 20, 2015 at 7:32 PM, Phil Turmel <philip@turmel.org> wrote:
> Woops!
> I'd still try a liveCD with 3.18.x or 3.19.
Right. I am considering ordering one of these new 8TB hard disks so I
can raw copy the raid partitions before I try anything drastic.
Unfortunately even 8TB won't fit 4x3TB of data. :-/ I am fairly
certain my data is still good since all disks have a clean bill of
health and other partitions on them worked with bonnie. So, I figure
my biggest risk right now, is doing something else stupid.
The system remains bootable as the uefi+boot+root filesystems were on
RAID1 x4 arrays and not the data RAID5 array. I'm hesitant to take
action until I find away to store the raw partitions, but installing
3.19 and booting that from my existing root partition sounds harmless
enough.
The system does not have a CD-ROM and has insufficient SATA connectors
to attach one. I could try booting via a USB drive, though.
^ permalink raw reply
* Re: kernel refuses devices mdadm -E accepts
From: Wesley W. Terpstra @ 2015-01-20 18:32 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
In-Reply-To: <54BE9E47.9030706@turmel.org>
On Tue, Jan 20, 2015 at 7:28 PM, Phil Turmel <philip@turmel.org> wrote:
>> Kernel version 3.17.8 and mdadm version 3.3.2.
> There have been many bug fixes to mdadm since that kernel was retired.
Really? I will give that a shot.
> Show us the output of that if it doesn't work (it should resume your
> reshape).
I will try that.
It was totally stupid reshape, I was changing the "data-offset" to
align it. Really unnecessary, and really slow, so I feel like an idiot
for starting it.
>> Thanks for any help. If I lose this array, I am going to face a lot of grief...
> I shouldn't have to say this, but RAID is for availability, not for data
> security. You still need a backup system for any important data.
Sure. The really vital (and small) stuff I have backed up elsewhere,
but the sheer volume of raw video my girlfriend records is not
economically feasible to backup beyond raid+btrfs snapshots.
^ permalink raw reply
* RE: RAID 6 recovery issue
From: Graham Mitchell @ 2015-01-20 18:32 UTC (permalink / raw)
To: 'Roman Mamedov'; +Cc: 'linux-raid'
In-Reply-To: <20150120215202.27bf1c8f@natsu>
> to add some mystery and suspense to the situation? :)
Sorry, didn't want to make it too easy... :) It's been a long couple of
days...
Dmesg says nothing of interest at all
[root@file00bert log]# dmesg | grep sde
sde: sde1
md: bind<sde1>
/var/log/messages says something more, but not too much
[root@file00bert log]# cat messages-20150118 | grep sde
Jan 17 20:29:19 file00bert kernel: md: export_rdev(sde1)
Jan 17 20:29:19 file00bert kernel: md: bind<sde1>
Jan 17 22:35:24 file00bert kernel: md: unbind<sde1>
Jan 17 22:35:24 file00bert kernel: md: export_rdev(sde1)
Jan 17 23:03:05 file00bert kernel: sde: sde1
Jan 17 23:03:05 file00bert kernel: md: bind<sde1>
Not quite sure why md decided to unbind sde1...
Around that point in the messages file, I see this...
Jan 17 20:24:42 file00bert kernel: sdp:
Jan 17 20:27:57 file00bert kernel: sds:
Jan 17 20:29:19 file00bert kernel: md: export_rdev(sde1)
Jan 17 20:29:19 file00bert kernel: md: bind<sde1>
Jan 17 20:29:19 file00bert kernel: md: recovery of RAID array md0
Jan 17 20:29:19 file00bert kernel: md: minimum _guaranteed_ speed: 1000
KB/sec/disk.
Jan 17 20:29:19 file00bert kernel: md: using maximum available idle IO
bandwidth (but not more than 200000 KB/sec) for recovery.
Jan 17 20:29:19 file00bert kernel: md: using 128k window, over a total of
488383488k.
Jan 17 20:30:34 file00bert kernel: sd 0:0:14:0: [sdo] Synchronizing SCSI
cache
Jan 17 20:30:34 file00bert kernel: sd 0:0:14:0: [sdo] Unhandled error code
Jan 17 20:30:34 file00bert kernel: sd 0:0:14:0: [sdo] Result:
hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Jan 17 20:30:34 file00bert kernel: sd 0:0:14:0: [sdo] CDB: Read(10): 28 00
00 5f 42 3f 00 00 e8 00
Jan 17 20:30:34 file00bert kernel: __ratelimit: 182 callbacks suppressed
Jan 17 20:30:34 file00bert kernel: md/raid:md0: Disk failure on sdo1,
disabling device.
Jan 17 20:30:34 file00bert kernel: md/raid:md0: Operation continuing on 14
devices.
Jan 17 20:30:34 file00bert kernel: md/raid:md0: read error not correctable
(sector 6242896 on sdo1).
Jan 17 20:30:34 file00bert kernel: md/raid:md0: read error not correctable
(sector 6242904 on sdo1).
Jan 17 20:30:34 file00bert kernel: md/raid:md0: read error not correctable
(sector 6242912 on sdo1).
Jan 17 20:30:34 file00bert kernel: md/raid:md0: read error not correctable
(sector 6242920 on sdo1).
Jan 17 20:30:34 file00bert kernel: md/raid:md0: read error not correctable
(sector 6242928 on sdo1).
Jan 17 20:30:34 file00bert kernel: md/raid:md0: read error not correctable
(sector 6242936 on sdo1).
Jan 17 20:30:34 file00bert kernel: md/raid:md0: read error not correctable
(sector 6242944 on sdo1).
Jan 17 20:30:34 file00bert kernel: md/raid:md0: read error not correctable
(sector 6242952 on sdo1).
Jan 17 20:30:34 file00bert kernel: md/raid:md0: read error not correctable
(sector 6242960 on sdo1).
Jan 17 20:30:34 file00bert kernel: md/raid:md0: read error not correctable
(sector 6242968 on sdo1).
Jan 17 20:30:34 file00bert kernel: md/raid:md0: read error not correctable
(sector 6242976 on sdo1).
Jan 17 20:30:34 file00bert kernel: md/raid:md0: read error not correctable
(sector 6242984 on sdo1).
Jan 17 20:30:34 file00bert kernel: md/raid:md0: read error not correctable
(sector 6242992 on sdo1).
Jan 17 20:30:34 file00bert kernel: md/raid:md0: read error not correctable
(sector 6243000 on sdo1).
Jan 17 20:30:34 file00bert kernel: md/raid:md0: read error not correctable
(sector 6243008 on sdo1).
Jan 17 20:30:34 file00bert kernel: md/raid:md0: read error not correctable
(sector 6243016 on sdo1).
Jan 17 20:30:34 file00bert kernel: md/raid:md0: read error not correctable
(sector 6243024 on sdo1).
Jan 17 20:30:34 file00bert kernel: md/raid:md0: read error not correctable
(sector 6243032 on sdo1).
Jan 17 20:30:34 file00bert kernel: md/raid:md0: read error not correctable
(sector 6243040 on sdo1).
Jan 17 20:30:34 file00bert kernel: sd 0:0:14:0: [sdo] Unhandled error code
Jan 17 20:30:34 file00bert kernel: sd 0:0:14:0: [sdo] Result:
hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Jan 17 20:30:34 file00bert kernel: sd 0:0:14:0: [sdo] CDB: Read(10): 28 00
00 5f 43 27 00 00 18 00
Jan 17 20:30:34 file00bert kernel: md/raid:md0: read error not correctable
(sector 6243048 on sdo1).
Jan 17 20:30:34 file00bert kernel: md/raid:md0: read error not correctable
(sector 6243056 on sdo1).
Jan 17 20:30:34 file00bert kernel: md/raid:md0: read error not correctable
(sector 6243064 on sdo1).
Jan 17 20:30:34 file00bert kernel: sd 0:0:14:0: [sdo] Unhandled error code
Jan 17 20:30:34 file00bert kernel: sd 0:0:14:0: [sdo] Result:
hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Jan 17 20:30:34 file00bert kernel: sd 0:0:14:0: [sdo] CDB: Read(10): 28 00
00 5f 43 3f 00 01 00 00
Jan 17 20:30:34 file00bert kernel: md/raid:md0: read error not correctable
(sector 6243072 on sdo1).
Jan 17 20:30:34 file00bert kernel: md/raid:md0: read error not correctable
(sector 6243080 on sdo1).
Jan 17 20:30:34 file00bert kernel: md/raid:md0: read error not correctable
(sector 6243088 on sdo1).
Jan 17 20:30:34 file00bert kernel: md/raid:md0: read error not correctable
(sector 6243096 on sdo1).
Jan 17 20:30:34 file00bert kernel: md/raid:md0: read error not correctable
(sector 6243104 on sdo1).
It goes on for a bit, then I get this...
Jan 17 20:30:34 file00bert kernel: md/raid:md0: read error not correctable
(sector 6242808 on sdo1).
Jan 17 20:30:34 file00bert kernel: sd 0:0:14:0: [sdo] Result:
hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Jan 17 20:30:34 file00bert kernel: mpt2sas0: removing handle(0x0018),
sas_addr(0x5001517e3b0af0ae)
Jan 17 20:30:34 file00bert kernel: md: md0: recovery done.
Jan 17 20:30:34 file00bert kernel: md: unbind<sdo1>
Jan 17 20:30:34 file00bert kernel: md: export_rdev(sdo1)
Jan 17 20:30:36 file00bert kernel: scsi 0:0:178:0: Direct-Access ATA
ST3500630AS K PQ: 0 ANSI: 6
Jan 17 20:30:36 file00bert kernel: scsi 0:0:178:0: SATA: handle(0x0018),
sas_addr(0x5001517e3b0af0ae), phy(14), device_name(0x0000000000000000)
Jan 17 20:30:36 file00bert kernel: scsi 0:0:178:0: SATA:
enclosure_logical_id(0x5001517e3b0af0bf), slot(14)
Jan 17 20:30:36 file00bert kernel: scsi 0:0:178:0: atapi(n), ncq(y),
asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Jan 17 20:30:36 file00bert kernel: scsi 0:0:178:0: qdepth(32), tagged(1),
simple(1), ordered(0), scsi_level(7), cmd_que(1)
Jan 17 20:30:36 file00bert kernel: sd 0:0:178:0: Attached scsi generic sg14
type 0
Jan 17 20:30:36 file00bert kernel: sd 0:0:178:0: [sdo] 976773168 512-byte
logical blocks: (500 GB/465 GiB)
Jan 17 20:30:36 file00bert kernel: sd 0:0:178:0: [sdo] Write Protect is off
Jan 17 20:30:36 file00bert kernel: sd 0:0:178:0: [sdo] Write cache: enabled,
read cache: enabled, supports DPO and FUA
Jan 17 20:30:36 file00bert kernel: sdo: sdo1
Jan 17 20:30:36 file00bert kernel: sd 0:0:178:0: [sdo] Attached SCSI disk
Jan 17 20:32:13 file00bert kernel: md: requested-resync of RAID array md0
Jan 17 20:32:13 file00bert kernel: md: minimum _guaranteed_ speed: 1000
KB/sec/disk.
Jan 17 20:32:13 file00bert kernel: md: using maximum available idle IO
bandwidth (but not more than 200000 KB/sec) for requested-resync.
Jan 17 20:32:13 file00bert kernel: md: using 128k window, over a total of
488383488k.
Jan 17 20:32:13 file00bert kernel: md: md0: requested-resync done.
Jan 17 20:38:27 file00bert kernel: [drm] nouveau 0000:05:03.0: Load detected
on output A
Jan 17 21:13:33 file00bert kernel: [drm] nouveau 0000:05:03.0: Setting dpms
mode 3 on vga encoder (output 0)
Jan 17 21:24:43 file00bert kernel: [drm] nouveau 0000:05:03.0: Setting dpms
mode 0 on vga encoder (output 0)
Jan 17 21:54:49 file00bert kernel: [drm] nouveau 0000:05:03.0: Setting dpms
mode 3 on vga encoder (output 0)
Jan 17 22:24:43 file00bert kernel: [drm] nouveau 0000:05:03.0: Setting dpms
mode 0 on vga encoder (output 0)
Jan 17 22:35:24 file00bert kernel: md: unbind<sde1>
Jan 17 22:35:24 file00bert kernel: md: export_rdev(sde1)
Jan 17 22:40:00 file00bert kernel: EXT4-fs error (device md0):
__ext4_get_inode_loc: unable to read inode block - inode=1041409,
block=1066401824
Jan 17 22:40:01 file00bert kernel: Buffer I/O error on device md0, logical
block 0
Jan 17 22:40:01 file00bert kernel: lost page write due to I/O error on md0
Jan 17 22:40:05 file00bert kernel: Aborting journal on device md0-8.
Jan 17 22:40:05 file00bert kernel: Buffer I/O error on device md0, logical
block 793280512
Jan 17 22:40:05 file00bert kernel: lost page write due to I/O error on md0
Jan 17 22:40:05 file00bert kernel: JBD2: I/O error detected when updating
journal superblock for md0-8.
Jan 17 22:40:17 file00bert kernel: EXT4-fs error (device md0):
ext4_journal_start_sb: Detected aborted journal
Jan 17 22:40:17 file00bert kernel: EXT4-fs (md0): Remounting filesystem
read-only
Jan 17 22:40:17 file00bert kernel: EXT4-fs error (device md0):
__ext4_get_inode_loc: unable to read inode block - inode=384003,
block=393216032
-----Original Message-----
From: Roman Mamedov [mailto:rm@romanrm.net]
Sent: Tuesday, January 20, 2015 11:52 AM
To: Graham Mitchell
Cc: linux-raid
Subject: Re: RAID 6 recovery issue
On Tue, 20 Jan 2015 11:46:45 -0500
"Graham Mitchell" <gmitch@woodlea.com> wrote:
> but for some reason it is now marked as a spare.
^^^^^^^^^^^^^^^
And you are not looking into 'dmesg' to find out why on purpose, to add some
mystery and suspense to the situation? :)
--
With respect,
Roman
^ permalink raw reply
* Re: kernel refuses devices mdadm -E accepts
From: Phil Turmel @ 2015-01-20 18:32 UTC (permalink / raw)
To: Wesley W. Terpstra, linux-raid
In-Reply-To: <54BE9E47.9030706@turmel.org>
Woops!
On 01/20/2015 01:28 PM, Phil Turmel wrote:
> Hi Wesley,
>
> On 01/19/2015 06:01 PM, Wesley W. Terpstra wrote:
>
>> I was in the middle of a reshape of this 4-disk raid5 when something
>> rebooted the computer. The system seems otherwise fine, and I suspect
>> someone in the house.
>>
>> What is the correct next step? Should I try --run ? I would obviously
>> prefer not to lose the data on this array. I expect that the reshape
>> was NOT complete, so just recreating the array will probably corrupt
>> its contents.
>>
>> Kernel version 3.17.8 and mdadm version 3.3.2.
>
> There have been many bug fixes to mdadm since that kernel was retired.
> You should temporarily boot a current liveCD (my favorite is
> systemrescuecd) and do "mdadm -Afv /dev/mdX /dev/sdX ..."
I misread that kernel version. It's relatively current. Hmmm.
I'd still try a liveCD with 3.18.x or 3.19.
> Show us the output of that if it doesn't work (it should resume your
> reshape).
This still holds. :-)
Phil
^ permalink raw reply
* Re: kernel refuses devices mdadm -E accepts
From: Phil Turmel @ 2015-01-20 18:28 UTC (permalink / raw)
To: Wesley W. Terpstra, linux-raid
In-Reply-To: <CAA-O0XjoD3fQaMpYYmy4kpBSxD+K9TKX_QGr9mcNVuye43XWHg@mail.gmail.com>
Hi Wesley,
On 01/19/2015 06:01 PM, Wesley W. Terpstra wrote:
> I was in the middle of a reshape of this 4-disk raid5 when something
> rebooted the computer. The system seems otherwise fine, and I suspect
> someone in the house.
>
> What is the correct next step? Should I try --run ? I would obviously
> prefer not to lose the data on this array. I expect that the reshape
> was NOT complete, so just recreating the array will probably corrupt
> its contents.
>
> Kernel version 3.17.8 and mdadm version 3.3.2.
There have been many bug fixes to mdadm since that kernel was retired.
You should temporarily boot a current liveCD (my favorite is
systemrescuecd) and do "mdadm -Afv /dev/mdX /dev/sdX ..."
Show us the output of that if it doesn't work (it should resume your
reshape).
When it is done reshaping/recovering, consider upgrading your kernel.
> Thanks for any help. If I lose this array, I am going to face a lot of grief...
I shouldn't have to say this, but RAID is for availability, not for data
security. You still need a backup system for any important data.
Phil
^ permalink raw reply
* Re: RAID 6 recovery issue
From: Roman Mamedov @ 2015-01-20 16:52 UTC (permalink / raw)
To: Graham Mitchell; +Cc: linux-raid
In-Reply-To: <00b101d034d0$ad7dd050$087970f0$@woodlea.com>
On Tue, 20 Jan 2015 11:46:45 -0500
"Graham Mitchell" <gmitch@woodlea.com> wrote:
> but for some reason it is now marked as a spare.
^^^^^^^^^^^^^^^
And you are not looking into 'dmesg' to find out why on purpose, to add some
mystery and suspense to the situation? :)
--
With respect,
Roman
^ permalink raw reply
* RAID 6 recovery issue
From: Graham Mitchell @ 2015-01-20 16:46 UTC (permalink / raw)
To: linux-raid
I've been having a heck of a time sending this - apologies if anyone sees
this email more than once (I've not see it hit the lists either of the 2
previous times I've sent it).
I’m having an issue with one of my RAID-6 arrays. For some reason, the email
wasn’t set up, so I never found out I had a couple of bad drives in the
array until last night.
Originally, when I looked at the output of /proc/mdstat, it showed that the
array was running with 15 out of the 17 drives still running.
[gmitch@file00bert ~]$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 sde1[19] sdi1[16] sdh1[12] sdf1[4] sdr1[18] sdg1[5](F)
sdj1[7] sdo1[22] sdt1[14] sdd1[13] sdl1[0](F) sda1[20] sdb1[1] sdk1[21]
sdn1[10] sdc1[2] sdm1[15] sdq1[17]
7325752320 blocks super 1.2 level 6, 512k chunk, algorithm 2 [17/15]
[_UUUU_UUUUUUUUUUU]
[>....................] recovery = 0.4% (2421508/488383488)
finish=180.7min speed=44805K/sec
As you can see, device 19 (sde1) is showing as a normal member of the array.
My original plan was to partition off 500GB from one of the 1TB drives I
have spare in the server, add one partition to the array. Once that had been
done, I was going to carve off 500GB from the other drive, and let the
array rebuild with that.
I created the partition on one of the drives and was going to add it to the
array, but stopped when I saw that the array was in recovery (I started up
‘watch /proc/mdstat’ in another window).
I went to have dinner, and came back, and found that the array was now very
unhappy, and cat /proc/mdstat showed
[root@file00bert ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 sde1[19](S) sdi1[16] sdh1[12] sdf1[4] sdr1[18] sdg1[5](F)
sdj1[7] sdt1[14] sdd1[13] sdl1[0](F) sda1[20] sdb1[1] sdk1[21] sdn1[10]
sdc1[2] sdm1[15] sdq1[17]
7325752320 blocks super 1.2 level 6, 512k chunk, algorithm 2 [17/14]
[_UUUU_UUUUUUUUUU_]
With device 19 having gone from a live drive to a spare. I’ve done an
examine of all the drives, and the event counts look to be reasonable
[root@file00bert ~]# mdadm -E /dev/sd[a-z]1 | egrep 'Event|/dev'
/dev/sda1:
Events : 1452687
/dev/sdb1:
Events : 1452687
/dev/sdc1:
Events : 1452687
/dev/sdd1:
Events : 1452687
/dev/sde1:
Events : 1452687
/dev/sdf1:
Events : 1452687
/dev/sdh1:
Events : 1452687
/dev/sdi1:
Events : 1452687
/dev/sdj1:
Events : 1452687
/dev/sdk1:
Events : 1452687
/dev/sdm1:
Events : 1452687
/dev/sdn1:
Events : 1452687
/dev/sdo1:
Events : 1452661
/dev/sdq1:
Events : 1452687
/dev/sdr1:
Events : 1452687
/dev/sdt1:
Events : 1452687
/dev/sdw1:
Events : 1431553
/dev/sdx1:
Events : 1431964
[root@file00bert ~]#
All of the events look to be within acceptable limits (are they?) and device
19 (sde1) has the same event count as most of the drives, but for some
reason it is now marked as a spare. I’ve not stopped the array yet, but I’ve
not written anything to it either. I’m not sure if taking the array down
then restarting it with a –force is the right course of action. My googling
isn’t showing a conclusive answer, so I thought I should seek some advice
before I went and did something that wrecked the array.
What should my next steps to recover the array be? I think all I need to do
is somehow to get device 19 (sde1) back believing that it's a real member of
the array, rather than a spare? Or should I be kicking it out, and getting
things running with sdo1?
[root@file00bert ~]# uname -a
Linux file00bert.woodlea.org.uk 2.6.32-358.2.1.el6.x86_64 #1 SMP Wed Mar 13
00:26:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
[root@file00bert ~]# mdadm --version
mdadm - v3.2.5 - 18th May 2012
Thanks.
Graham
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH RESEND] md:Remove no longer needed fix me comment in dm-region-hash.c for removing the structure pointer, *rh from the definiton of dm_region
From: Nick Krause @ 2015-01-20 13:17 UTC (permalink / raw)
To: nick, agk, linux-raid, dm-devel, linux-kernel@vger.kernel.org,
snitzer
In-Reply-To: <20150120114032.GC17599@agk-dp.fab.redhat.com>
On Tue, Jan 20, 2015 at 6:40 AM, Alasdair G Kergon <agk@redhat.com> wrote:
> On Tue, Jan 20, 2015 at 06:36:10AM -0500, nick wrote:
>> Unless we use a union or something similar there seems to be no easy
>> way without rewriting a lot of an already function driver file.
>
> The comment does not restrict itself to 'easy ways'.
>
> Alasdair
>
>
Then I suggest that I look into this form. Sorry about wasting your time.
Nick
^ permalink raw reply
* [PATCH] IMSM: Clear migration record on disks more often
From: Pawel Baldysiak @ 2015-01-20 12:52 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, pawel.baldysiak, artur.paszkiewicz
Migration record is not always cleared after successful migration. This can
block another reshape from being started. Migration will not be continued via
systemd service due to error in verifying reshape position. This patch added
clearing migration record when disk is added to container, and after successful
migration.
Signed-off-by: Pawel Baldysiak <pawel.baldysiak@intel.com>
---
super-intel.c | 25 +++++++++++++++++++++++++
1 file changed, 25 insertions(+)
diff --git a/super-intel.c b/super-intel.c
index 4c53019..2406991 100644
--- a/super-intel.c
+++ b/super-intel.c
@@ -5055,6 +5055,14 @@ static int add_to_super_imsm(struct supertype *st, mdu_disk_info_t *dk,
}
get_dev_size(fd, NULL, &size);
+ /* clear migr_rec when adding disk to container */
+ memset(super->migr_rec_buf, 0, MIGR_REC_BUF_SIZE);
+ if (lseek64(fd, size - 512, SEEK_SET) >= 0) {
+ if (write(fd, super->migr_rec_buf,
+ MIGR_REC_BUF_SIZE) != MIGR_REC_BUF_SIZE)
+ perror("Write migr_rec failed");
+ }
+
size /= 512;
serialcpy(dd->disk.serial, dd->serial);
set_total_blocks(&dd->disk, size);
@@ -10648,6 +10656,23 @@ static int imsm_manage_reshape(
}
+ /* clear migr_rec on disks after successful migration */
+ struct dl *d;
+
+ memset(super->migr_rec_buf, 0, MIGR_REC_BUF_SIZE);
+ for (d = super->disks; d; d = d->next) {
+ if (d->index < 0 || is_failed(&d->disk))
+ continue;
+ unsigned long long dsize;
+
+ get_dev_size(d->fd, NULL, &dsize);
+ if (lseek64(d->fd, dsize - 512, SEEK_SET) >= 0) {
+ if (write(d->fd, super->migr_rec_buf,
+ MIGR_REC_BUF_SIZE) != MIGR_REC_BUF_SIZE)
+ perror("Write migr_rec failed");
+ }
+ }
+
/* return '1' if done */
ret_val = 1;
abort:
^ permalink raw reply related
* Re: [dm-devel] [PATCH RESEND] md:Remove no longer needed fix me comment in dm-region-hash.c for removing the structure pointer, *rh from the definiton of dm_region
From: Alasdair G Kergon @ 2015-01-20 11:40 UTC (permalink / raw)
To: nick; +Cc: agk, linux-raid, dm-devel, linux-kernel, snitzer
In-Reply-To: <54BE3DAA.5080900@gmail.com>
On Tue, Jan 20, 2015 at 06:36:10AM -0500, nick wrote:
> Unless we use a union or something similar there seems to be no easy
> way without rewriting a lot of an already function driver file.
The comment does not restrict itself to 'easy ways'.
Alasdair
^ permalink raw reply
* Re: [PATCH RESEND] md:Remove no longer needed fix me comment in dm-region-hash.c for removing the structure pointer, *rh from the definiton of dm_region
From: nick @ 2015-01-20 11:36 UTC (permalink / raw)
To: agk, linux-raid, dm-devel, linux-kernel, snitzer
In-Reply-To: <20150120111951.GB17599@agk-dp.fab.redhat.com>
Alasdair,
Unless we use a union or something similar there seems to be no easy
way without rewriting a lot of an already function driver file.
Cheers,
Nick
On 2015-01-20 06:19 AM, Alasdair G Kergon wrote:
> On Mon, Jan 19, 2015 at 10:16:21PM -0500, Nicholas Krause wrote:
>> Removes the no longer needed fix me comment questioning if we need to remove
>> the structure pointer,rh as defined as a structure of type dsm_region_hash
>> from the definition of the structure, dm_region. Furthermore this pointer is
>> needed in order for various functions to run and build successfully in the file,
>> dm-region-hash.c.
>
> You've misunderstood the comment!
> Is there a better alternative to having every dm_region struct store its own copy of
> this pointer?
>
> Alasdair
>
^ permalink raw reply
* Re: [dm-devel] [PATCH RESEND] md:Remove no longer needed fix me comment in dm-region-hash.c for removing the structure pointer, *rh from the definiton of dm_region
From: Alasdair G Kergon @ 2015-01-20 11:19 UTC (permalink / raw)
To: Nicholas Krause; +Cc: agk, linux-raid, dm-devel, linux-kernel, snitzer
In-Reply-To: <1421723781-21984-1-git-send-email-xerofoify@gmail.com>
On Mon, Jan 19, 2015 at 10:16:21PM -0500, Nicholas Krause wrote:
> Removes the no longer needed fix me comment questioning if we need to remove
> the structure pointer,rh as defined as a structure of type dsm_region_hash
> from the definition of the structure, dm_region. Furthermore this pointer is
> needed in order for various functions to run and build successfully in the file,
> dm-region-hash.c.
You've misunderstood the comment!
Is there a better alternative to having every dm_region struct store its own copy of
this pointer?
Alasdair
^ permalink raw reply
* Is it possible to use bad_blocks to simulate write_error and activate hot-replace?
From: Apple Yin @ 2015-01-20 8:25 UTC (permalink / raw)
To: linux-raid
I want to test hot-replace when a write error is recorded in a
bad-block log. Here is my steps:
1. create raid5 with 3 disks (sdb sdc sdd), and 1 spare (sde)
2. echo 2097152 512 > /sys/block/md192/md/dev-sdb/bad_blocks /* set 1
badblock from 1G of sdb*/
3. dd if=/dev/zero of=/dev/md192 bs=1M count=1024 seek=2048 /* from 2G of md */
dd if=/dev/zero of=/dev/md192 bs=1M count=1024 seek=2000 /* near to 2G */
No write error happen, sdb is not marked as want_replacement and not
start the rebuild. Anyone help ?
^ permalink raw reply
* Re: RAID1 removing failed disk returns EBUSY
From: Xiao Ni @ 2015-01-20 7:16 UTC (permalink / raw)
To: Joe Lawrence; +Cc: NeilBrown, linux-raid, Bill Kuzeja
In-Reply-To: <20150119125650.22a75dd3@jlaw-desktop.mno.stratus.com>
----- Original Message -----
> From: "Joe Lawrence" <joe.lawrence@stratus.com>
> To: "Xiao Ni" <xni@redhat.com>
> Cc: "NeilBrown" <neilb@suse.de>, linux-raid@vger.kernel.org, "Bill Kuzeja" <william.kuzeja@stratus.com>
> Sent: Tuesday, January 20, 2015 1:56:50 AM
> Subject: Re: RAID1 removing failed disk returns EBUSY
>
> On Sun, 18 Jan 2015 21:33:50 -0500
> Xiao Ni <xni@redhat.com> wrote:
>
> >
> >
> > ----- Original Message -----
> > > From: "Joe Lawrence" <joe.lawrence@stratus.com>
> > > To: "Xiao Ni" <xni@redhat.com>
> > > Cc: "NeilBrown" <neilb@suse.de>, linux-raid@vger.kernel.org, "Bill
> > > Kuzeja" <william.kuzeja@stratus.com>
> > > Sent: Friday, January 16, 2015 11:10:31 PM
> > > Subject: Re: RAID1 removing failed disk returns EBUSY
> > >
> > > On Fri, 16 Jan 2015 00:20:12 -0500
> > > Xiao Ni <xni@redhat.com> wrote:
> > > >
> > > > Hi Joe
> > > >
> > > > Thanks for reminding me. I didn't do that. Now it can remove
> > > > successfully after writing
> > > > "idle" to sync_action.
> > > >
> > > > I thought wrongly that the patch referenced in this mail is fixed
> > > > for
> > > > the problem.
> > >
> > > So it sounds like even with 3.18 and a new mdadm, this bug still
> > > persists?
> > >
> > > -- Joe
> > >
> > > --
> >
> > Hi Joe
> >
> > I'm a little confused now. Does the patch
> > 45eaf45dfa4850df16bc2e8e7903d89021137f40 from linux-stable
> > resolve the problem?
> >
> > My environment is:
> >
> > [root@dhcp-12-133 mdadm]# mdadm --version
> > mdadm - v3.3.2-18-g93d3bd3 - 18th December 2014 (this is the newest
> > upstream)
> > [root@dhcp-12-133 mdadm]# uname -r
> > 3.18.2
> >
> >
> > My steps are:
> >
> > [root@dhcp-12-133 mdadm]# lsblk
> > sdb 8:16 0 931.5G 0 disk
> > └─sdb1 8:17 0 5G 0 part
> > sdc 8:32 0 186.3G 0 disk
> > sdd 8:48 0 931.5G 0 disk
> > └─sdd1 8:49 0 5G 0 part
> > [root@dhcp-12-133 mdadm]# mdadm -CR /dev/md0 -l1 -n2 /dev/sdb1 /dev/sdd1
> > --assume-clean
> > mdadm: Note: this array has metadata at the start and
> > may not be suitable as a boot device. If you plan to
> > store '/boot' on this device please ensure that
> > your boot-loader understands md/v1.x metadata, or use
> > --metadata=0.90
> > mdadm: Defaulting to version 1.2 metadata
> > mdadm: array /dev/md0 started.
> >
> > Then I unplug the disk.
> >
> > [root@dhcp-12-133 mdadm]# lsblk
> > sdc 8:32 0 186.3G 0 disk
> > sdd 8:48 0 931.5G 0 disk
> > └─sdd1 8:49 0 5G 0 part
> > └─md0 9:0 0 5G 0 raid1
> > [root@dhcp-12-133 mdadm]# echo faulty > /sys/block/md0/md/dev-sdb1/state
> > [root@dhcp-12-133 mdadm]# echo remove > /sys/block/md0/md/dev-sdb1/state
> > -bash: echo: write error: Device or resource busy
> > [root@dhcp-12-133 mdadm]# echo idle > /sys/block/md0/md/sync_action
> > [root@dhcp-12-133 mdadm]# echo remove > /sys/block/md0/md/dev-sdb1/state
> >
> >
> > Now after I set idle to sync_action, it can be removed as you said in
> > the mail.
> > It's a good workaround. Is this OK?
> >
> > Best Regards
> > Xiao
>
> Hi Xiao,
>
> According to my notes, the "idle" sync_action was always a viable
> workaround, with or with this change.
>
> Neil's patch should have made it possible to issue only a
> "faulty" and "remove" to remove the RAID component.
>
> I don't have an exact version, but it appears that my mdadm version was
> an upstream git from Oct 27-th timeframe.
>
> -- Joe
>
Joe
Thanks for the explanation. So echo "idle" to sync_action is a workaround
without the patch.
It looks like the patch is not enough to fix the problem.
Do you have a try with the new patch? Is the problem still exist in
your environment?
If your environment have no problem, can you give me the version number? I'll
have a try with the same version too.
Best Regards
Xiao
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* livelock on MD RAID5
From: Mikulas Patocka @ 2015-01-20 3:24 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid, dm-devel
Hi Neil
I saw your patch "md/raid5: avoid livelock caused by non-aligned writes".
That patch is only for RAID 6, but we are seeing similar livelock on RAID5
(mdX_raid5 is constantly looping in
raid5d/handle_active_stripes/handle_stripe/ops_run_io and some submitted
bios get stuck). The livelock happens when we issue I/O and simulate disk
failure.
Do you think that the same issue could happen on RAID5 too?
Do you think that the patch "md/raid5: fetch_block must fetch all the
blocks handle_stripe_dirtying wants." could fix it?
Mikulas
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox