* Re: dm: fix parameter to blk_delay_queue()
From: Mike Snitzer @ 2016-07-15 13:59 UTC (permalink / raw)
To: Tahsin Erdogan
Cc: Alasdair Kergon, Jens Axboe, dm-devel, Shaohua Li, linux-raid,
linux-kernel
In-Reply-To: <1468589228-20722-1-git-send-email-tahsin@google.com>
On Fri, Jul 15 2016 at 9:27am -0400,
Tahsin Erdogan <tahsin@google.com> wrote:
> Second parameter to blk_delay_queue() must be in msec units not jiffies.
>
> Signed-off-by: Tahsin Erdogan <tahsin@google.com>
This needs to be rebased against linux-next (or linux-dm.git's
'for-next') because the code in question has been moved out to dm-rq.c
But I'll gladly take care of it.
Your change seems fine, and in fact my intent was always to have it be
10 msec, see commit: d548b34b062b60
Curious that the initial "HZ / 10" (from Jens' commit 7eaceaccab) wasn't
expressed in terms of msecs either.
Anyway, thanks for the fix!
Mike
> ---
> drivers/md/dm.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/md/dm.c b/drivers/md/dm.c
> index 1b2f96205361..17c63265a205 100644
> --- a/drivers/md/dm.c
> +++ b/drivers/md/dm.c
> @@ -2175,7 +2175,7 @@ static void dm_request_fn(struct request_queue *q)
> md_in_flight(md) && rq->bio && rq->bio->bi_vcnt == 1 &&
> md->last_rq_pos == pos && md->last_rq_rw == rq_data_dir(rq)) ||
> (ti->type->busy && ti->type->busy(ti))) {
> - blk_delay_queue(q, HZ / 100);
> + blk_delay_queue(q, 10);
> return;
> }
>
> --
> 2.8.0.rc3.226.g39d4020
>
^ permalink raw reply
* Re: dm: fix parameter to blk_delay_queue()
From: Tahsin Erdogan @ 2016-07-15 14:18 UTC (permalink / raw)
To: Mike Snitzer
Cc: Alasdair Kergon, Jens Axboe, dm-devel, Shaohua Li, linux-raid,
linux-kernel
In-Reply-To: <20160715135905.GA22026@redhat.com>
> This needs to be rebased against linux-next (or linux-dm.git's
> 'for-next') because the code in question has been moved out to dm-rq.c
>
> But I'll gladly take care of it.
Thanks Mike!
^ permalink raw reply
* Re: Help with assembling a stopped array
From: Vegard Haugland @ 2016-07-15 17:02 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
In-Reply-To: <69b8afa7-bfeb-e85a-aba6-b6013b2302aa@turmel.org>
Hi Phil
Here's an update to my problem:.
> The device that's popping up as a spare every time really does have the
> spare marker (role #10). That's the drive that was partially re-added
> when the array crashed again. Just leave it out of your next attempts.
OK. Done.
> The key is "SCT Error Recovery Control command not supported" on most of
> your drives. In fact, upgrading model 7200.11 1T drives to your 2T
> Barracudas is precisely the set of models that first bit me in the ass.
> I mention that somewhere in the threads I pointed at.
>
> The key is that you have desktop drives in a raid array, and they don't
> handle read errors in a way friendly to the linux kernel -- they take
> too long. The one 7200.11 drive can be told to timeout quicker (7.0
> seconds is typical for spinning disks). The kernel will have to be told
> to wait extra long for all of the others. 2-3 minutes. Details in the
> reading assignments. The precise sequence of events that breaks MD raid
> is described in the sixth:
>
> http://marc.info/?l=linux-raid&m=133665797115876&w=2
Thanks for the explanation. I'll look into this more to get a better
comprehension of this.
>
> After you get your array assembled again, I suggest you run the lsdrv[1]
> script to document which drive serial numbers correspond to which array
> roles. Also consider using the --update=metadata at some point (not
> now) to get away from the v0.90 metadata. It is unreliable when used on
> partitions that extend to the end of their parent device.
>
Adding this to the todo list. Thanks.
> Your problem is "timeout mismatch", which is use of drives with
> extended error recovery times in any linux software raid array. (I
> understand hardware raid also struggles with this, but I don't know the
> details.)
>
> For now, use at every boot:
>
> for x in /sys/block/*/device/timeout ; do echo 180 > $x ; done
Done.
> This time, omit the drive that shows up as "spare". Use all nine
> others. You really want nine, so the redundancy in your array can
> reconstruct when it hits the UREs you obviously have. See "Current
> Pending Sector" != 0 in your smartctl reports.
>
> After it assembles the nine, issue "mdadm --run /dev/md4" if it didn't
> start. Then "echo check >>/sys/block/md4/md/sync_action".
>
> Wait for that to finish. Then add the spare back to the array.
OK. Here's whats been happening for the past two days. I can this
command to rebuild the array
# mdadm -A /dev/md4 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3 /dev/sde3
/dev/sdg3 /dev/sdh3 /dev/sdi3 /dev/sdj3 --force
mdadm: forcing event count in /dev/sdc3(4) from 32679348 upto 32680549
mdadm: forcing event count in /dev/sdd3(3) from 32677935 upto 32680549
mdadm: clearing FAULTY flag for device 3 in /dev/md4 for /dev/sdd3
mdadm: clearing FAULTY flag for device 2 in /dev/md4 for /dev/sdc3
mdadm: Marking array /dev/md4 as 'clean'
mdadm: /dev/md4 has been started with 8 drives (out of 10) and 1 spare.
As the output mentioned, the array started and I got access to the
data again. Yay! In order to start the rebuild, I ran "echo check
>>/sys/block/md4/md/sync_action".
The array just finished the rebuild, but not with the results I hoped
for. Here's the output from mdadm -D /dev/md4
# mdadm -D -v /dev/md4
/dev/md4:
Version : 0.90
Creation Time : Tue Jun 15 06:37:57 2010
Raid Level : raid6
Array Size : 11719545344 (11176.63 GiB 12000.81 GB)
Used Dev Size : 1464943168 (1397.08 GiB 1500.10 GB)
Raid Devices : 10
Total Devices : 9
Preferred Minor : 4
Persistence : Superblock is persistent
Update Time : Fri Jul 15 14:36:51 2016
State : clean, FAILED
Active Devices : 7
Working Devices : 8
Failed Devices : 1
Spare Devices : 1
Layout : left-symmetric-6
Chunk Size : 64K
UUID : c807562b:f8dbc4e5:457de8e9:4807bcad
Events : 0.32687312
Number Major Minor RaidDevice State
0 8 19 0 active sync /dev/sdb3
1 8 99 1 active sync /dev/sdg3
4 0 0 4 removed
3 8 51 3 active sync /dev/sdd3
8 0 0 8 removed
5 8 115 5 active sync /dev/sdh3
6 8 67 6 active sync /dev/sde3
7 8 131 7 active sync /dev/sdi3
8 8 147 8 active sync /dev/sdj3
18 0 0 18 removed
10 8 3 - spare /dev/sda3
11 8 35 - faulty /dev/sdc3
Here's the output from /proc/mdstat during the rebuild (which just completed).
root@debian:~# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md4 : active raid6 sdb3[0] sda3[10] sdj3[8] sdi3[7] sde3[6] sdh3[5]
sdc3[4] sdd3[3] sdg3[1]
11719545344 blocks level 6, 64k chunk, algorithm 18 [10/8] [UU_UUUUUU_]
[=================>...] recovery = 86.1%
(1262406304/1464943168) finish=178.0min speed=18957K/sec
The current output from /proc/mdstat is the following:
Personalities : [raid1] [raid6] [raid5] [raid4]
md4 : active raid6 sdb3[0] sda3[10](S) sdj3[8] sdi3[7] sde3[6] sdh3[5]
sdc3[11](F) sdd3[3] sdg3[1]
11719545344 blocks level 6, 64k chunk, algorithm 18 [10/7] [UU_U_UUUU_]
Looks like /dev/sdc is faulty. I ran smartctl on that disk, and the
output is as follows. Note that this error was logged just 4 hours ago
(mentioned in the below log at 24702 hours).
# root@debian:~# smartctl -a /dev/sdc
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-3.16.0-4-amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.14 (AF)
Device Model: ST1500DM003-9YN16G
Serial Number: W1E2VMF3
LU WWN Device Id: 5 000c50 05e0ec925
Firmware Version: CC4B
User Capacity: 1,500,301,910,016 bytes [1.50 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Fri Jul 15 18:56:16 2016 UTC
==> WARNING: A firmware update for this drive may be available,
see the following Seagate web pages:
http://knowledge.seagate.com/articles/en_US/FAQ/207931en
http://knowledge.seagate.com/articles/en_US/FAQ/223651en
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 600) seconds.
Offline data collection
capabilities: (0x73) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 185) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x3085) SCT Status supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 109 099 006 Pre-fail
Always - 45495242
3 Spin_Up_Time 0x0003 096 095 000 Pre-fail
Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age
Always - 31
5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail
Always - 0
7 Seek_Error_Rate 0x000f 087 060 030 Pre-fail
Always - 653305433
9 Power_On_Hours 0x0032 072 072 000 Old_age
Always - 24706
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail
Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age
Always - 28
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age
Always - 0
184 End-to-End_Error 0x0032 100 100 099 Old_age
Always - 0
187 Reported_Uncorrect 0x0032 091 091 000 Old_age
Always - 9
188 Command_Timeout 0x0032 100 099 000 Old_age
Always - 1 1 2
189 High_Fly_Writes 0x003a 050 050 000 Old_age
Always - 50
190 Airflow_Temperature_Cel 0x0022 065 052 045 Old_age
Always - 35 (Min/Max 33/40)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age
Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age
Always - 19
193 Load_Cycle_Count 0x0032 095 095 000 Old_age
Always - 10077
194 Temperature_Celsius 0x0022 035 048 000 Old_age
Always - 35 (0 17 0 0 0)
197 Current_Pending_Sector 0x0012 100 100 000 Old_age
Always - 80
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age
Offline - 80
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age
Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age
Offline - 24656h+37m+41.577s
241 Total_LBAs_Written 0x0000 100 253 000 Old_age
Offline - 2232552343992
242 Total_LBAs_Read 0x0000 100 253 000 Old_age
Offline - 144336591164585
SMART Error Log Version: 1
ATA Error Count: 2
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 2 occurred at disk power-on lifetime: 24702 hours (1029 days + 6 hours)
When the command that caused the error occurred, the device was
active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
25 00 a0 ff ff ff ef 00 1d+19:17:33.086 READ DMA EXT
25 00 18 ff ff ff ef 00 1d+19:17:32.528 READ DMA EXT
25 00 f8 ff ff ff ef 00 1d+19:17:32.167 READ DMA EXT
27 00 00 00 00 00 e0 00 1d+19:17:32.166 READ NATIVE MAX ADDRESS
EXT [OBS-ACS-3]
ec 00 00 00 00 00 a0 00 1d+19:17:32.158 IDENTIFY DEVICE
Error 1 occurred at disk power-on lifetime: 24702 hours (1029 days + 6 hours)
When the command that caused the error occurred, the device was
active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
25 00 f0 ff ff ff ef 00 1d+19:17:29.011 READ DMA EXT
25 00 18 ff ff ff ef 00 1d+19:17:29.010 READ DMA EXT
25 00 f8 ff ff ff ef 00 1d+19:17:28.988 READ DMA EXT
25 00 f8 ff ff ff ef 00 1d+19:17:21.227 READ DMA EXT
25 00 f8 ff ff ff ef 00 1d+19:17:21.195 READ DMA EXT
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining
LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 24622 -
# 2 Short offline Completed without error 00% 24598 -
# 3 Short offline Completed without error 00% 24574 -
# 4 Extended offline Completed without error 00% 24556 -
# 5 Short offline Completed without error 00% 24550 -
# 6 Short offline Completed without error 00% 24526 -
# 7 Short offline Completed without error 00% 24502 -
# 8 Short offline Completed without error 00% 24478 -
# 9 Short offline Completed without error 00% 24454 -
#10 Short offline Completed without error 00% 24430 -
#11 Short offline Completed without error 00% 24406 -
#12 Extended offline Completed without error 00% 24388 -
#13 Short offline Completed without error 00% 24382 -
#14 Short offline Completed without error 00% 24358 -
#15 Short offline Completed without error 00% 24334 -
#16 Short offline Completed without error 00% 24310 -
#17 Short offline Completed without error 00% 24286 -
#18 Short offline Completed without error 00% 24262 -
#19 Short offline Completed without error 00% 24238 -
#20 Extended offline Completed without error 00% 24220 -
#21 Short offline Completed without error 00% 24214 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
----
For some reason, the other faulty disk (not the one mentioned below,
or the one that initially showed up as spare) also shows up as spare
now (like the good one did earlier). Issuing "echo check >>
/sys/block/md4/md/sync_action" does not make any attempts to rebuild
the array. Should I use mdadm --manage --add to re-add it before I
replace the faulty disk with a new one?
>
> Also, in the future, paste drive reports and console output inline in
> your mails, with word wrapping disabled. That puts the details in the
> archives for future googlers to find. (This list allows ~ 100k per mail.)
>
OK. I did that this time instead of using pastebins.
Vegard
^ permalink raw reply
* Re: Help with assembling a stopped array
From: Phil Turmel @ 2016-07-15 21:44 UTC (permalink / raw)
To: Vegard Haugland; +Cc: linux-raid
In-Reply-To: <CANz1=THkLw6tEYxPhEwb2WTfhfuRXu2E1--tAPGV2hmSSLpyLQ@mail.gmail.com>
Hi Vegard,
On 07/15/2016 01:02 PM, Vegard Haugland wrote:
>> This time, omit the drive that shows up as "spare". Use all nine
>> others. You really want nine, so the redundancy in your array can
>> reconstruct when it hits the UREs you obviously have. See "Current
>> Pending Sector" != 0 in your smartctl reports.
>>
>> After it assembles the nine, issue "mdadm --run /dev/md4" if it didn't
>> start. Then "echo check >>/sys/block/md4/md/sync_action".
>>
>> Wait for that to finish. Then add the spare back to the array.
>
> OK. Here's whats been happening for the past two days. I can this
> command to rebuild the array
>
> # mdadm -A /dev/md4 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3 /dev/sde3
> /dev/sdg3 /dev/sdh3 /dev/sdi3 /dev/sdj3 --force
> mdadm: forcing event count in /dev/sdc3(4) from 32679348 upto 32680549
> mdadm: forcing event count in /dev/sdd3(3) from 32677935 upto 32680549
> mdadm: clearing FAULTY flag for device 3 in /dev/md4 for /dev/sdd3
> mdadm: clearing FAULTY flag for device 2 in /dev/md4 for /dev/sdc3
> mdadm: Marking array /dev/md4 as 'clean'
> mdadm: /dev/md4 has been started with 8 drives (out of 10) and 1 spare.
Well, that sucks. I expected 9 drives out of 10 and no spare. You are
somewhat screwed.
> As the output mentioned, the array started and I got access to the
> data again. Yay! In order to start the rebuild, I ran "echo check
>>> /sys/block/md4/md/sync_action".
>
> The array just finished the rebuild, but not with the results I hoped
> for. Here's the output from mdadm -D /dev/md4
You can't successfully check or rebuild without redundancy if any device
has an Unrecoverable Read Error. Which you know you have because
Pending is greater than zero on sdc (maybe others, I didn't check).
> For some reason, the other faulty disk (not the one mentioned below,
> or the one that initially showed up as spare) also shows up as spare
> now (like the good one did earlier). Issuing "echo check >>
> /sys/block/md4/md/sync_action" does not make any attempts to rebuild
> the array. Should I use mdadm --manage --add to re-add it before I
> replace the faulty disk with a new one?
It is possible your mdadm is too old to fully --force assembly in this
case, or it is a side effect of old v0.90 metadata. With v1.x+ and
bitmaps, re-add is very fast and unlikely to hit other UREs. Too late
for you.
Anyways, you can only use the eight non-spare drives. Since at least
one has a URE, you will need to use ddrescue to complete copy those
drives onto new drives (or your "spares" with their superblocks erased).
Then assemble the array (8 members) with the new drives and the non-URE
old drives. Then add spares one at a time, waiting for rebuild to
finish for each.
If you have very critical data you want to make sure you retrieve,
assemble the eight non-spare drives one more time before making copies,
*don't* do the check, then mount and copy those critical files. Might
hit a URE anyways, but is the best way to get a quick backup.
Once you start copying devices, be very careful that you keep copies and
roles straight, so that you don't try to assemble with two devices with
the same role number.
You *will* lose some data where the UREs are. You'll need to fsck to
fix the corruption, if it happens to be in a file or folder.
Phil
^ permalink raw reply
* Re: Recover filenames from failed RAID0
From: Michel Dubois @ 2016-07-17 19:10 UTC (permalink / raw)
To: linux-raid
Cc: Lars Ellenberg, Jens Axboe, linux-block, Keith Busch,
Martin K. Petersen, Peter Zijlstra, Jiri Kosina, Ming Lei,
Kirill A. Shutemov, NeilBrown, linux-kernel, Takashi Iwai,
linux-bcache, Zheng Liu, Kent Overstreet, dm-devel, Shaohua Li,
Ingo Molnar, Alasdair Kergon, Roland Kammerer, Mike Snitzer
In-Reply-To: <CAAVC+N5m74MyDqZamHgy71vRMhcEY8w_OkiG-GMQKU=JG-QkxA@mail.gmail.com>
Dear linux-raid mailing list,
I have a RAID0 array of four 3TB disks that failed on the "third" disk.
I am aware of the non-redundancy of RAID0 but I would like to recover
the filenames from that RAID0. If I could recover some data it would
be a bonus.
Below you'll find the outputs of the following commands
mdadm --examine /dev/sd[abcd]1
fdisk -l
where sda1, sdb1, sdc1 and sdd1 should be the 4 RAID devices.
What could be my next step?
I thank you for your time
Michel Dubois
======================
mdadm --examine /dev/sd[abcd]1
/dev/sda1:
Magic : a92b4efc
Version : 00.90.00
UUID : 7d247a6e:7b5d46c8:f52d9c89:db304b21
Creation Time : Mon Apr 23 19:55:36 2012
Raid Level : raid1
Used Dev Size : 20980800 (20.01 GiB 21.48 GB)
Array Size : 20980800 (20.01 GiB 21.48 GB)
Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Update Time : Mon Jun 27 21:12:23 2016
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 1
Spare Devices : 0
Checksum : 1a57db60 - correct
Events : 164275
Number Major Minor RaidDevice State
this 0 8 1 0 active sync /dev/sda1
0 0 8 1 0 active sync /dev/sda1
1 1 8 17 1 active sync /dev/sdb1
2 2 0 0 2 faulty removed
3 3 8 33 3 active sync /dev/sdc1
/dev/sdb1:
Magic : a92b4efc
Version : 00.90.00
UUID : 7d247a6e:7b5d46c8:f52d9c89:db304b21
Creation Time : Mon Apr 23 19:55:36 2012
Raid Level : raid1
Used Dev Size : 20980800 (20.01 GiB 21.48 GB)
Array Size : 20980800 (20.01 GiB 21.48 GB)
Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Update Time : Mon Jun 27 21:12:23 2016
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 1
Spare Devices : 0
Checksum : 1a57db72 - correct
Events : 164275
Number Major Minor RaidDevice State
this 1 8 17 1 active sync /dev/sdb1
0 0 8 1 0 active sync /dev/sda1
1 1 8 17 1 active sync /dev/sdb1
2 2 0 0 2 faulty removed
3 3 8 33 3 active sync /dev/sdc1
/dev/sdc1:
Magic : a92b4efc
Version : 00.90.00
UUID : 7d247a6e:7b5d46c8:f52d9c89:db304b21
Creation Time : Mon Apr 23 19:55:36 2012
Raid Level : raid1
Used Dev Size : 20980800 (20.01 GiB 21.48 GB)
Array Size : 20980800 (20.01 GiB 21.48 GB)
Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Update Time : Mon Jun 27 21:12:23 2016
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 1
Spare Devices : 0
Checksum : 1a57db86 - correct
Events : 164275
Number Major Minor RaidDevice State
this 3 8 33 3 active sync /dev/sdc1
0 0 8 1 0 active sync /dev/sda1
1 1 8 17 1 active sync /dev/sdb1
2 2 0 0 2 faulty removed
3 3 8 33 3 active sync /dev/sdc1
======================
fdisk -l
WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util
fdisk doesn't support GPT. Use GNU Parted.
Disk /dev/sda: 3000.5 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x03afffbe
Device Boot Start End Blocks Id System
/dev/sda1 1 267350 2147483647+ ee EFI GPT
WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util
fdisk doesn't support GPT. Use GNU Parted.
Disk /dev/sdb: 3000.5 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x142a889c
Device Boot Start End Blocks Id System
/dev/sdb1 1 267350 2147483647+ ee EFI GPT
WARNING: GPT (GUID Partition Table) detected on '/dev/sdc'! The util
fdisk doesn't support GPT. Use GNU Parted.
Disk /dev/sdc: 3000.5 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x3daebd50
Device Boot Start End Blocks Id System
/dev/sdc1 1 267350 2147483647+ ee EFI GPT
Disk /dev/md0: 21.4 GB, 21484339200 bytes
2 heads, 4 sectors/track, 5245200 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk identifier: 0x00000000
Disk /dev/md0 doesn't contain a valid partition table
^ permalink raw reply
* Cannot start array on disk
From: Bhatia Amit @ 2016-07-17 19:59 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
In-Reply-To: <1124605843.526681.1468785592654.JavaMail.yahoo.ref@mail.yahoo.com>
Hi
I have a WD Live duo setup with two 3TB drives in RAID1. After a recent power surge etc, I could not access the enclosure data. I removed the drives out from the enclosure and connected them via esata cable to a Linux laptop. One of the drives had physical noise and failed to connect at all. The other drive was able to connect and showed up as /dev/sdc. Following some online search, I issued the following commands to get some results, but am unable to mount the drive to a linux laptop via esata. Questions:
* Ideally both sdc3 and sdc4 should have shown RAID=RAID1. If sdc4 is showing RAID=linear, is it because the other drive went totally bad, and software decided to reconfigure this drive from RAID1 to linear ?
* Given the one of the two drives is totally dead, is it still possible to reconfigure, mount and recover data from the single drive?
* I tried scanning the drive via r-linux and it does show files exist, but does not show any folder information. So data is there on the drive. Is there a way to recreate the setup with just this single drive, so that I can extract files with folder information?
* Looking at an online "derrick" script, the script seems to call mdadm create with the "missing" flag. Is that something I should be doing to be able to assemble and recover data from the drive ?
"mdadm --create $rootfsRaid --verbose --metadata=0.9 --raid-devices=2 --level=raid1 --run $diskRoot1 missing"
Thanks
Amit
Parted shows missing filesystem for sdc4 :
$ sudo parted -l
Model: ATA WDC WD30EZRX-00D (scsi)
Disk /dev/sdc: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Number Start End Size File system Name Flags
3 15.7MB 528MB 513MB primary raid
1 528MB 2576MB 2048MB ext3 primary raid
2 2576MB 4624MB 2048MB ext3 primary raid
4 4624MB 3001GB 2996GB primary raid
Mdstat shows only sdc3 as active, but sdc4 as inactive:
$ sudo cat /proc/mdstat
Personalities : [linear] [raid1]
md126 : active (auto-read-only) raid1 sdc3[2]
500724 blocks super 1.0 [2/1] [_U]
md127 : inactive sdc4[0](S)
2925750264 blocks super 1.0
unused devices: <none>
Trying to assemble and scan only shows /dev/sdc3 as active:
$ sudo mdadm --stop /dev/md12[567]
mdadm: stopped /dev/md126
mdadm: stopped /dev/md127
$ sudo cat /proc/mdstat
Personalities : [linear] [raid1]
unused devices: <none>
$ sudo mdadm --assemble --scan
mdadm: /dev/md/MyBookLiveDuo:3 assembled from 1 drive - not enough to start the array.
mdadm: /dev/md/MyBookLiveDuo:2 has been started with 1 drive (out of 2).
mdadm: /dev/md/MyBookLiveDuo:3 assembled from 1 drive - not enough to start the array.
$ sudo cat /proc/mdstat
Personalities : [linear] [raid1]
md127 : active raid1 sdc3[2]
500724 blocks super 1.0 [2/1] [_U]
unused devices: <none>
Output from examine sdc3 is below, which mdstat says is active. It says RAID level = RAID1.
$ sudo mdadm --examine /dev/sdc3
/dev/sdc3:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x0
Array UUID : 7c040c5e:9c30ac6d:e534a129:20457e22
Name : MyBookLiveDuo:2
Creation Time : Wed Dec 31 19:01:40 1969
Raid Level : raid1
Raid Devices : 2
Avail Dev Size : 1001448 (489.07 MiB 512.74 MB)
Array Size : 500724 (489.07 MiB 512.74 MB)
Super Offset : 1001456 sectors
State : clean
Device UUID : 1d9fe3e3:d5ac7387:d9ededba:88ca24a5
Update Time : Sun Jul 3 11:53:31 2016
Checksum : 31589560 - correct
Events : 101
Device Role : Active device 1
Array State : .A ('A' == active, '.' == missing)
$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md126 : active (auto-read-only) raid1 sdc3[2]
500724 blocks super 1.0 [2/1] [_U]
md127 : inactive sdc4[0](S)
2925750264 blocks super 1.0
unused devices: <none>
But, output from examine sdc4 is below, which says RAID level = linear.
$ sudo mdadm --examine /dev/sdc4
/dev/sdc4:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x0
Array UUID : 374e689e:3bfd050c:ab0b0dce:2d50f5fd
Name : MyBookLiveDuo:3
Creation Time : Mon Sep 16 14:53:47 2013
Raid Level : linear
Raid Devices : 2
Avail Dev Size : 5851500528 (2790.21 GiB 2995.97 GB)
Used Dev Size : 0
Super Offset : 5851500528 sectors
State : clean
Device UUID : 9096f74b:0a8f2b61:93347be3:6d3b6c1b
Update Time : Mon Sep 16 14:53:47 2013
Checksum : 77aa5963 - correct
Events : 0
Rounding : 0K
Device Role : Active device 0
Array State : AA ('A' == active, '.' == missing)
And partitions output is:
$ cat /proc/partitions
major minor #blocks name
8 0 1953514584 sda
8 1 102400 sda1
8 2 1953411072 sda2
8 16 1953514584 sdb
8 17 248832 sdb1
8 18 1 sdb2
8 21 1953263616 sdb5
252 0 1953261568 dm-0
252 1 1919635456 dm-1
252 2 33488896 dm-2
8 32 2930266584 sdc
8 33 1999872 sdc1
8 34 1999872 sdc2
8 35 500736 sdc3
8 36 2925750272 sdc4
9 126 500724 md126
^ permalink raw reply
* Re: Recover filenames from failed RAID0
From: Stewart Ives @ 2016-07-17 22:56 UTC (permalink / raw)
To: Michel Dubois
Cc: Mike Snitzer, Peter Zijlstra, Ming Lei, NeilBrown, Keith Busch,
dm-devel, Alasdair Kergon, Roland Kammerer, Zheng Liu,
Takashi Iwai, Ingo Molnar, Shaohua Li, Kent Overstreet,
linux-block, linux-bcache, Jens Axboe, linux-raid,
Martin K. Petersen, Jiri Kosina, linux-kernel, Lars Ellenberg,
Kirill A. Shutemov
In-Reply-To: <CAAVC+N78o675MpkJN=VO_zZhh7ePer1zU-ePLJi4eNH04ih3mw@mail.gmail.com>
[-- Attachment #1.1: Type: text/plain, Size: 6358 bytes --]
Michael,
I'll preface my reply with the statement that I am far from an expert at
this but I can read and understand the descriptions of the different RAID
levels and it seems to me with a RAID0 you are SOL if you lose a device in
the array. Just by the very nature of the RAID0 configuration there is
absolutely NO redundancy. The only reason anyone would configure such a
system is for SPEED and the only data that should be permitted on a RAID0
array is temp or working data that is recoverable by other means in the
event of a failure. I know of many Videographers that use a SSD RAID0 array
for working on their current project but they also copy that array out
about every hour for backup.
I pose only one question to you. Did you have a backup?
Good luck.
-stew
>>
>> Stewart M. Ives
>> SofTEC USA
>> 1717 Bridge St
>> New Cumberland, PA 17070 USA
>>
>> Tel: 717-910-4600
>> Fax: 888-371-6022
>> Skype: softecusa-ivessm
>> EMail: ivessm@softecusa.com
>> WebSite: www.softecusa.com
>>
On Sun, Jul 17, 2016 at 3:10 PM, Michel Dubois <michel.dubois.mtl@gmail.com>
wrote:
> Dear linux-raid mailing list,
>
> I have a RAID0 array of four 3TB disks that failed on the "third" disk.
>
> I am aware of the non-redundancy of RAID0 but I would like to recover
> the filenames from that RAID0. If I could recover some data it would
> be a bonus.
>
> Below you'll find the outputs of the following commands
> mdadm --examine /dev/sd[abcd]1
> fdisk -l
>
> where sda1, sdb1, sdc1 and sdd1 should be the 4 RAID devices.
>
> What could be my next step?
>
> I thank you for your time
>
> Michel Dubois
>
> ======================
> mdadm --examine /dev/sd[abcd]1
> /dev/sda1:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : 7d247a6e:7b5d46c8:f52d9c89:db304b21
> Creation Time : Mon Apr 23 19:55:36 2012
> Raid Level : raid1
> Used Dev Size : 20980800 (20.01 GiB 21.48 GB)
> Array Size : 20980800 (20.01 GiB 21.48 GB)
> Raid Devices : 4
> Total Devices : 3
> Preferred Minor : 0
>
> Update Time : Mon Jun 27 21:12:23 2016
> State : clean
> Active Devices : 3
> Working Devices : 3
> Failed Devices : 1
> Spare Devices : 0
> Checksum : 1a57db60 - correct
> Events : 164275
>
>
> Number Major Minor RaidDevice State
> this 0 8 1 0 active sync /dev/sda1
>
> 0 0 8 1 0 active sync /dev/sda1
> 1 1 8 17 1 active sync /dev/sdb1
> 2 2 0 0 2 faulty removed
> 3 3 8 33 3 active sync /dev/sdc1
> /dev/sdb1:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : 7d247a6e:7b5d46c8:f52d9c89:db304b21
> Creation Time : Mon Apr 23 19:55:36 2012
> Raid Level : raid1
> Used Dev Size : 20980800 (20.01 GiB 21.48 GB)
> Array Size : 20980800 (20.01 GiB 21.48 GB)
> Raid Devices : 4
> Total Devices : 3
> Preferred Minor : 0
>
> Update Time : Mon Jun 27 21:12:23 2016
> State : clean
> Active Devices : 3
> Working Devices : 3
> Failed Devices : 1
> Spare Devices : 0
> Checksum : 1a57db72 - correct
> Events : 164275
>
>
> Number Major Minor RaidDevice State
> this 1 8 17 1 active sync /dev/sdb1
>
> 0 0 8 1 0 active sync /dev/sda1
> 1 1 8 17 1 active sync /dev/sdb1
> 2 2 0 0 2 faulty removed
> 3 3 8 33 3 active sync /dev/sdc1
> /dev/sdc1:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : 7d247a6e:7b5d46c8:f52d9c89:db304b21
> Creation Time : Mon Apr 23 19:55:36 2012
> Raid Level : raid1
> Used Dev Size : 20980800 (20.01 GiB 21.48 GB)
> Array Size : 20980800 (20.01 GiB 21.48 GB)
> Raid Devices : 4
> Total Devices : 3
> Preferred Minor : 0
>
> Update Time : Mon Jun 27 21:12:23 2016
> State : clean
> Active Devices : 3
> Working Devices : 3
> Failed Devices : 1
> Spare Devices : 0
> Checksum : 1a57db86 - correct
> Events : 164275
>
>
> Number Major Minor RaidDevice State
> this 3 8 33 3 active sync /dev/sdc1
>
> 0 0 8 1 0 active sync /dev/sda1
> 1 1 8 17 1 active sync /dev/sdb1
> 2 2 0 0 2 faulty removed
> 3 3 8 33 3 active sync /dev/sdc1
>
> ======================
> fdisk -l
>
> WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util
> fdisk doesn't support GPT. Use GNU Parted.
>
>
> Disk /dev/sda: 3000.5 GB, 3000592982016 bytes
> 255 heads, 63 sectors/track, 364801 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x03afffbe
>
> Device Boot Start End Blocks Id System
> /dev/sda1 1 267350 2147483647+ ee EFI GPT
>
> WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util
> fdisk doesn't support GPT. Use GNU Parted.
>
>
> Disk /dev/sdb: 3000.5 GB, 3000592982016 bytes
> 255 heads, 63 sectors/track, 364801 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x142a889c
>
> Device Boot Start End Blocks Id System
> /dev/sdb1 1 267350 2147483647+ ee EFI GPT
>
> WARNING: GPT (GUID Partition Table) detected on '/dev/sdc'! The util
> fdisk doesn't support GPT. Use GNU Parted.
>
>
> Disk /dev/sdc: 3000.5 GB, 3000592982016 bytes
> 255 heads, 63 sectors/track, 364801 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x3daebd50
>
> Device Boot Start End Blocks Id System
> /dev/sdc1 1 267350 2147483647+ ee EFI GPT
>
> Disk /dev/md0: 21.4 GB, 21484339200 bytes
> 2 heads, 4 sectors/track, 5245200 cylinders
> Units = cylinders of 8 * 512 = 4096 bytes
> Disk identifier: 0x00000000
>
> Disk /dev/md0 doesn't contain a valid partition table
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
[-- Attachment #1.2: Type: text/html, Size: 8863 bytes --]
[-- Attachment #2: Type: text/plain, Size: 0 bytes --]
^ permalink raw reply
* Re: Page Allocation Failures/OOM with dm-crypt on software RAID10 (Intel Rapid Storage) with check/repair/sync
From: Matthias Dahl @ 2016-07-18 7:24 UTC (permalink / raw)
To: linux-raid
Cc: Tetsuo Handa, Michal Hocko, Mike Snitzer, linux-mm, dm-devel,
linux-kernel
In-Reply-To: <005574d77d3f5dbc2643044a1e2468dc@mail.ud19.udmedia.de>
Hello again...
So I spent all weekend doing further tests, since this issue is
really bugging me for obvious reasons.
I thought it would be beneficial if I created a bug report that
summarized and centralized everything in one place rather than
having everything spread across several lists and posts.
Here the bug report I created:
https://bugzilla.kernel.org/show_bug.cgi?id=135481
If anyone has any suggestions, ideas or wants me to do further tests,
please just let me know. There is not much more I can do at this point
without further help/guidance.
Thanks,
Matthias
--
Dipl.-Inf. (FH) Matthias Dahl | Software Engineer | binary-island.eu
services: custom software [desktop, mobile, web], server administration
^ permalink raw reply
* a hard lockup in md raid5 sequential write (v4.7-rc7)
From: Coly Li @ 2016-07-18 8:55 UTC (permalink / raw)
To: linux-raid
Hi,
These days I observe a hard lockup in md raid5. This issue can be easily
reproduced in kernel v4.7-rc7 (up to commit:
47ef4ad2684d380dd6d596140fb79395115c3950) by this fio job file:
[global]
direct=1
thread=1
[job]
filename=/dev/md0
blocksize=8m
rw=write
name=raid5
lockmem=1
numjobs=40
write_bw_log=example
group_reporting=1
norandommap=1
log_avg_msec=0
runtime=600.0
iodepth=64
write_lat_log=example
Where md0 is a raid5 target assembled by 3 Memblaze (PBlaze3) PCIe SSDs.
This test runs on a dual 10-core processors Dell T7910 machine.
From the crash dump, dmesg of the panic by nmi watchdog timeout is,
[ 2330.544036] NMI watchdog: Watchdog detected hard LOCKUP on cpu
18.dModules linked in: raid456 async_raid6_recov async_memcpy libcrc32c
async_pq async_xor async_tx joydev st memdisk(O) memcon(O) af_packet
iscsi_ibft iscsi_boot_sysfs msr snd_hda_codec_hdmi intel_rapl sb_edac
raid1 edac_core x86_pkg_temp_thermal intel_powerclamp coretemp raid0
md_mod snd_hda_codec_realtek snd_hda_codec_generic kvm_intel kvm
snd_hda_intel irqbypass snd_hda_codec crct10dif_pclmul snd_hda_core
crc32_pclmul ghash_clmulni_intel snd_hwdep dm_mod aesni_intel aes_x86_64
snd_pcm mei_wdt e1000e igb iTCO_wdt lrw dcdbas iTCO_vendor_support
snd_timer gf128mul mei_me dell_smm_hwmon glue_helper serio_raw
ablk_helper cryptd snd lpc_ich pcspkr ptp i2c_i801 mei mptctl dca
mfd_core pps_core soundcore mptbase shpchp fjes tpm_tis tpm btrfs xor
raid6_pq hid_generic usbhid crc32c_intel nouveau video mxm_wmi
i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt xhci_pci
fb_sys_fops ehci_pci xhci_hcd ehci_hcd sr_mod ttm cd
[ 2330.544036] CPU: 18 PID: 30308 Comm: kworker/u42:4 Tainted: G
O 4.7.0-rc7-vanilla #1
[ 2330.544036] Hardware name: Dell Inc. Precision Tower 7910/0215PR,
BIOS A07 04/14/2015
[ 2330.544036] Workqueue: raid5wq raid5_do_work [raid456]
[ 2330.544036] 0000000000000000 ffff88103f405bb0 ffffffff813a6eea
0000000000000000
[ 2330.544036] 0000000000000000 ffff88103f405bc8 ffffffff8113c3e8
ffff8808dc7d8800
[ 2330.544036] ffff88103f405c00 ffffffff81180f8c 0000000000000001
ffff88103f40a440
[ 2330.544036] Call Trace:
[ 2330.544036] <NMI> [<ffffffff813a6eea>] dump_stack+0x63/0x89
[ 2330.544036] [<ffffffff8113c3e8>] watchdog_overflow_callback+0xc8/0xf0
[ 2330.544036] [<ffffffff81180f8c>] __perf_event_overflow+0x7c/0x1b0
[ 2330.544036] [<ffffffff8118b644>] perf_event_overflow+0x14/0x20
[ 2330.544036] [<ffffffff8100bf57>] intel_pmu_handle_irq+0x1c7/0x460
[ 2330.544036] [<ffffffff810053ad>] perf_event_nmi_handler+0x2d/0x50
[ 2330.544036] [<ffffffff810312e1>] nmi_handle+0x61/0x140
[ 2330.544036] [<ffffffff81031888>] default_do_nmi+0x48/0x130
[ 2330.544036] [<ffffffff81031a5b>] do_nmi+0xeb/0x160
[ 2330.544036] [<ffffffff816e5c71>] end_repeat_nmi+0x1a/0x1e
[ 2330.544036] [<ffffffff810cbcc7>] ?
native_queued_spin_lock_slowpath+0x117/0x1a0
[ 2330.544036] [<ffffffff810cbcc7>] ?
native_queued_spin_lock_slowpath+0x117/0x1a0
[ 2330.544036] [<ffffffff810cbcc7>] ?
native_queued_spin_lock_slowpath+0x117/0x1a0
[ 2330.544036] <<EOE>> [<ffffffff81193bbf>]
queued_spin_lock_slowpath+0xb/0xf
[ 2330.544036] [<ffffffff816e31ff>] _raw_spin_lock_irq+0x2f/0x40
[ 2330.544036] [<ffffffffa084c5d8>]
handle_active_stripes.isra.51+0x378/0x4f0 [raid456]
[ 2330.544036] [<ffffffffa083f1a6>] ?
raid5_wakeup_stripe_thread+0x96/0x1b0 [raid456]
[ 2330.544036] [<ffffffffa084cf1d>] raid5_do_work+0x8d/0x120 [raid456]
[ 2330.544036] [<ffffffff8109b5bb>] process_one_work+0x14b/0x450
[ 2330.544036] [<ffffffff8109b9eb>] worker_thread+0x12b/0x490
[ 2330.544036] [<ffffffff8109b8c0>] ? process_one_work+0x450/0x450
[ 2330.544036] [<ffffffff810a1599>] kthread+0xc9/0xe0
[ 2330.544036] [<ffffffff816e3a9f>] ret_from_fork+0x1f/0x40
[ 2330.544036] [<ffffffff810a14d0>] ? kthread_create_on_node+0x180/0x180
[ 2330.544036] Kernel panic - not syncing: Hard LOCKUP
[ 2330.544036] CPU: 18 PID: 30308 Comm: kworker/u42:4 Tainted: G
O 4.7.0-rc7-vanilla #1
[ 2330.544036] Hardware name: Dell Inc. Precision Tower 7910/0215PR,
BIOS A07 04/14/2015
[ 2330.544036] Workqueue: raid5wq raid5_do_work [raid456]
[ 2330.544036] 0000000000000000 ffff88103f405b28 ffffffff813a6eea
ffffffff81a45241
[ 2330.544036] 0000000000000000 ffff88103f405ba0 ffffffff81193642
0000000000000010
[ 2330.544036] ffff88103f405bb0 ffff88103f405b50 0000000000000086
ffffffff81a2a2e2
[ 2330.544036] Call Trace:
[ 2330.544036] <NMI> [<ffffffff813a6eea>] dump_stack+0x63/0x89
[ 2330.544036] [<ffffffff81193642>] panic+0xd2/0x223
[ 2330.544036] [<ffffffff810823af>] nmi_panic+0x3f/0x40
[ 2330.544036] [<ffffffff8113c401>] watchdog_overflow_callback+0xe1/0xf0
[ 2330.544036] [<ffffffff81180f8c>] __perf_event_overflow+0x7c/0x1b0
[ 2330.544036] [<ffffffff8118b644>] perf_event_overflow+0x14/0x20
[ 2330.544036] [<ffffffff8100bf57>] intel_pmu_handle_irq+0x1c7/0x460
[ 2330.544036] [<ffffffff810053ad>] perf_event_nmi_handler+0x2d/0x50
[ 2330.544036] [<ffffffff810312e1>] nmi_handle+0x61/0x140
[ 2330.544036] [<ffffffff81031888>] default_do_nmi+0x48/0x130
[ 2330.544036] [<ffffffff81031a5b>] do_nmi+0xeb/0x160
[ 2330.544036] [<ffffffff816e5c71>] end_repeat_nmi+0x1a/0x1e
[ 2330.544036] [<ffffffff810cbcc7>] ?
native_queued_spin_lock_slowpath+0x117/0x1a0
[ 2330.544036] [<ffffffff810cbcc7>] ?
native_queued_spin_lock_slowpath+0x117/0x1a0
[ 2330.544036] [<ffffffff810cbcc7>] ?
native_queued_spin_lock_slowpath+0x117/0x1a0
[ 2330.544036] <<EOE>> [<ffffffff81193bbf>]
queued_spin_lock_slowpath+0xb/0xf
[ 2330.544036] [<ffffffff816e31ff>] _raw_spin_lock_irq+0x2f/0x40
[ 2330.544036] [<ffffffffa084c5d8>]
handle_active_stripes.isra.51+0x378/0x4f0 [raid456]
[ 2330.544036] [<ffffffffa083f1a6>] ?
raid5_wakeup_stripe_thread+0x96/0x1b0 [raid456]
[ 2330.544036] [<ffffffffa084cf1d>] raid5_do_work+0x8d/0x120 [raid456]
[ 2330.544036] [<ffffffff8109b5bb>] process_one_work+0x14b/0x450
[ 2330.544036] [<ffffffff8109b9eb>] worker_thread+0x12b/0x490
[ 2330.544036] [<ffffffff8109b8c0>] ? process_one_work+0x450/0x450
[ 2330.544036] [<ffffffff810a1599>] kthread+0xc9/0xe0
[ 2330.544036] [<ffffffff816e3a9f>] ret_from_fork+0x1f/0x40
[ 2330.544036] [<ffffffff810a14d0>] ? kthread_create_on_node+0x180/0x180
The crash dump file is quite big (124MB), I need to find a method to
share, if anyone of you wants it, please let me know.
IMHO, this hard lockup seems related to bitmap allocation, because it
can be easily reproduced on a new-created md raid5 target, with 40+
processes doing big size (8MB+) writing.
Thanks.
Coly
^ permalink raw reply
* How does md gurantee not miss to free an active stripe_head when md stops?
From: Vaughan @ 2016-07-18 9:18 UTC (permalink / raw)
To: linux-raid
Hi all,
I'm using v3.10 md code for develop. Recently I encounter a problem where an
read IO usually returned from physical disk after md has been stopped.
I reviewed the code and find when md stops, it unregister raid5d and call
shrink_stripes() to free only the *inactive* stripes.
Why is it sure that there is no active stripes linking in handle_list?
I know before stop, it uses O_EXCL open the md, but that won't stop others
open it and send IO to it.
In my case, an OOPS usually happens like below:
I keep calling mdadm -stop to stop a md, but lsof shows it's opened by
systemd-udevd, so "still is inuse".
30s later, udev reports timeout and be kicked with SIGKILL.
systemd-udevd: worker [19335]/devices/virtual/block/md41 timeout; kill it.
Then md stop process is able to continue and go passed the free_conf(). But
there is an active_stripe left.
kernel:
shrink_stripes:conf(ffff880004affc00)->md(ffff8802d95b4000,md41)active_strip
es=1 <== this is my debug print.
kernel: md41: detected capacity change from 3409128980480 to 0
mdadm: stopped /dev/md41
After md is stopped, an read IO from underlying returned and OOPS.
[190830.867371] md: unbind<dm-64>
[190830.876345] md: export_rdev(dm-64)
[190831.201619] BUG: unable to handle kernel
[190831.202875] paging request at 0000000000002050
[190831.204101] IP: [<ffffffffa089a349>]
raid5_end_read_request+0xf9/0xdc0[raid456]
I found this returned bio is caused by a user read page, which is caused by
a fput to kill_bdev.
PID: 21345 TASK: ffff8803e5a916c0 CPU: 1 COMMAND: "mdadm"
#0 [ffff88016f777b88] __schedule at ffffffff815f513d
#1 [ffff88016f777bf0] io_schedule at ffffffff815f599d
#2 [ffff88016f777c08] sleep_on_page at ffffffff81155f1e
#3 [ffff88016f777c18] __wait_on_bit_lock at ffffffff815f38ab
#4 [ffff88016f777c58] __lock_page at ffffffff81156038
#5 [ffff88016f777cb0] truncate_inode_pages_range at ffffffff8116645e
#6 [ffff88016f777e00] truncate_inode_pages at ffffffff811664b5
#7 [ffff88016f777e10] kill_bdev at ffffffff811ffaef
#8 [ffff88016f777e28] __blkdev_put at ffffffff81201124
#9 [ffff88016f777e68] blkdev_put at ffffffff81201bae
#10 [ffff88016f777e98] blkdev_close at ffffffff81201d55
#11 [ffff88016f777ea8] __fput at ffffffff811c81b9
#12 [ffff88016f777ef0] ____fput at ffffffff811c847e
#13 [ffff88016f777f00] task_work_run at ffffffff81093b37
#14 [ffff88016f777f30] do_notify_resume at ffffffff81013b0c
#15 [ffff88016f777f50] int_signal at ffffffff8160049d
Anyone can answer my question? Thanks.
Vaughan
^ permalink raw reply
* Re: Recover filenames from failed RAID0
From: keld @ 2016-07-18 9:37 UTC (permalink / raw)
To: Michel Dubois
Cc: linux-raid, Lars Ellenberg, Jens Axboe, linux-block, Keith Busch,
Martin K. Petersen, Peter Zijlstra, Jiri Kosina, Ming Lei,
Kirill A. Shutemov, NeilBrown, linux-kernel, Takashi Iwai,
linux-bcache, Zheng Liu, Kent Overstreet, dm-devel, Shaohua Li,
Ingo Molnar, Alasdair Kergon, Roland Kammerer, Mike Snitzer
In-Reply-To: <CAAVC+N78o675MpkJN=VO_zZhh7ePer1zU-ePLJi4eNH04ih3mw@mail.gmail.com>
Hi
Which file system did you use?
I once wrote some code to get files out of an ext3 filesystem,
http://www.open-std.org/keld/readme-salvage.html
Maybe you can make some corrections for it to work
as I remember the code, then if you hit a directory, it will salvage the file names and the files
of that directory.
Best regards
keld
On Sun, Jul 17, 2016 at 03:10:03PM -0400, Michel Dubois wrote:
> Dear linux-raid mailing list,
>
> I have a RAID0 array of four 3TB disks that failed on the "third" disk.
>
> I am aware of the non-redundancy of RAID0 but I would like to recover
> the filenames from that RAID0. If I could recover some data it would
> be a bonus.
>
> Below you'll find the outputs of the following commands
> mdadm --examine /dev/sd[abcd]1
> fdisk -l
>
> where sda1, sdb1, sdc1 and sdd1 should be the 4 RAID devices.
>
> What could be my next step?
>
> I thank you for your time
>
> Michel Dubois
>
> ======================
> mdadm --examine /dev/sd[abcd]1
> /dev/sda1:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : 7d247a6e:7b5d46c8:f52d9c89:db304b21
> Creation Time : Mon Apr 23 19:55:36 2012
> Raid Level : raid1
> Used Dev Size : 20980800 (20.01 GiB 21.48 GB)
> Array Size : 20980800 (20.01 GiB 21.48 GB)
> Raid Devices : 4
> Total Devices : 3
> Preferred Minor : 0
>
> Update Time : Mon Jun 27 21:12:23 2016
> State : clean
> Active Devices : 3
> Working Devices : 3
> Failed Devices : 1
> Spare Devices : 0
> Checksum : 1a57db60 - correct
> Events : 164275
>
>
> Number Major Minor RaidDevice State
> this 0 8 1 0 active sync /dev/sda1
>
> 0 0 8 1 0 active sync /dev/sda1
> 1 1 8 17 1 active sync /dev/sdb1
> 2 2 0 0 2 faulty removed
> 3 3 8 33 3 active sync /dev/sdc1
> /dev/sdb1:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : 7d247a6e:7b5d46c8:f52d9c89:db304b21
> Creation Time : Mon Apr 23 19:55:36 2012
> Raid Level : raid1
> Used Dev Size : 20980800 (20.01 GiB 21.48 GB)
> Array Size : 20980800 (20.01 GiB 21.48 GB)
> Raid Devices : 4
> Total Devices : 3
> Preferred Minor : 0
>
> Update Time : Mon Jun 27 21:12:23 2016
> State : clean
> Active Devices : 3
> Working Devices : 3
> Failed Devices : 1
> Spare Devices : 0
> Checksum : 1a57db72 - correct
> Events : 164275
>
>
> Number Major Minor RaidDevice State
> this 1 8 17 1 active sync /dev/sdb1
>
> 0 0 8 1 0 active sync /dev/sda1
> 1 1 8 17 1 active sync /dev/sdb1
> 2 2 0 0 2 faulty removed
> 3 3 8 33 3 active sync /dev/sdc1
> /dev/sdc1:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : 7d247a6e:7b5d46c8:f52d9c89:db304b21
> Creation Time : Mon Apr 23 19:55:36 2012
> Raid Level : raid1
> Used Dev Size : 20980800 (20.01 GiB 21.48 GB)
> Array Size : 20980800 (20.01 GiB 21.48 GB)
> Raid Devices : 4
> Total Devices : 3
> Preferred Minor : 0
>
> Update Time : Mon Jun 27 21:12:23 2016
> State : clean
> Active Devices : 3
> Working Devices : 3
> Failed Devices : 1
> Spare Devices : 0
> Checksum : 1a57db86 - correct
> Events : 164275
>
>
> Number Major Minor RaidDevice State
> this 3 8 33 3 active sync /dev/sdc1
>
> 0 0 8 1 0 active sync /dev/sda1
> 1 1 8 17 1 active sync /dev/sdb1
> 2 2 0 0 2 faulty removed
> 3 3 8 33 3 active sync /dev/sdc1
>
> ======================
> fdisk -l
>
> WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util
> fdisk doesn't support GPT. Use GNU Parted.
>
>
> Disk /dev/sda: 3000.5 GB, 3000592982016 bytes
> 255 heads, 63 sectors/track, 364801 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x03afffbe
>
> Device Boot Start End Blocks Id System
> /dev/sda1 1 267350 2147483647+ ee EFI GPT
>
> WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util
> fdisk doesn't support GPT. Use GNU Parted.
>
>
> Disk /dev/sdb: 3000.5 GB, 3000592982016 bytes
> 255 heads, 63 sectors/track, 364801 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x142a889c
>
> Device Boot Start End Blocks Id System
> /dev/sdb1 1 267350 2147483647+ ee EFI GPT
>
> WARNING: GPT (GUID Partition Table) detected on '/dev/sdc'! The util
> fdisk doesn't support GPT. Use GNU Parted.
>
>
> Disk /dev/sdc: 3000.5 GB, 3000592982016 bytes
> 255 heads, 63 sectors/track, 364801 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x3daebd50
>
> Device Boot Start End Blocks Id System
> /dev/sdc1 1 267350 2147483647+ ee EFI GPT
>
> Disk /dev/md0: 21.4 GB, 21484339200 bytes
> 2 heads, 4 sectors/track, 5245200 cylinders
> Units = cylinders of 8 * 512 = 4096 bytes
> Disk identifier: 0x00000000
>
> Disk /dev/md0 doesn't contain a valid partition table
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Raid1 rcu stall while I/O stress test
From: Chien Lee @ 2016-07-18 9:58 UTC (permalink / raw)
To: linux-raid, NeilBrown, shli, owner-linux-raid
Hi all,
Recently raid1 happen rcu stall when I do I/O stress test. The raid1
is not in resync/rebuild/reshape/degraded status at the same time.
The Kernel version is 3.19.8 and here is the call trace in the kernel log.
<3>[427882.025091] INFO: rcu_sched self-detected stall on CPU { 2}
(t=60000 jiffies g=23546697 c=23546696 q=316936)
<6>[427882.035342] Task dump for CPU 2:
<6>[427882.038694] kworker/u8:2 R running task 0 438 2 0x00000008
<6>[427882.045989] Workqueue: dm-thin do_worker [dm_thin_pool]
<4>[427882.051388] 0000000000000082 ffff88017fd03db8 ffffffff8107279c
0000000000000002
<4>[427882.059108] ffffffff81bde180 ffff88017fd03dd8 ffffffff8107280a
ffff88017fd03e58
<4>[427882.066836] 0000000000000003 ffff88017fd03e08 ffffffff8109bee9
ffff88017fd12040
<4>[427882.074568] Call Trace:
<4>[427882.077137] <IRQ> [<ffffffff8107279c>] sched_show_task+0xac/0xe0
<4>[427882.083510] [<ffffffff8107280a>] dump_cpu_task+0x3a/0x50
<4>[427882.089034] [<ffffffff8109bee9>] rcu_dump_cpu_stacks+0x89/0xe0
<4>[427882.095078] [<ffffffff8109d563>] rcu_check_callbacks+0x313/0x550
<4>[427882.101291] [<ffffffff810a02e2>] update_process_times+0x32/0x60
<4>[427882.107419] [<ffffffff810ae115>] tick_sched_handle+0x35/0x50
<4>[427882.113285] [<ffffffff810ae27f>] tick_sched_timer+0x3f/0x70
<4>[427882.119068] [<ffffffff810a0b0d>] __run_hrtimer+0x3d/0xc0
<4>[427882.124591] [<ffffffff810a1276>] hrtimer_interrupt+0xe6/0x230
<4>[427882.130549] [<ffffffff81057a1d>] ? __do_softirq+0x15d/0x220
<4>[427882.136333] [<ffffffff810326c4>] local_apic_timer_interrupt+0x34/0x70
<4>[427882.142982] [<ffffffff8103342c>] smp_apic_timer_interrupt+0x3c/0x60
<4>[427882.149458] [<ffffffff818bb06a>] apic_timer_interrupt+0x6a/0x70
<4>[427882.155586] <EOI> [<ffffffff810eacd7>] ? mempool_alloc+0x47/0x130
<4>[427882.162048] [<ffffffff81337097>] ? bio_clone_bioset+0x77/0x300
<4>[427882.168089] [<ffffffff810eacd7>] ? mempool_alloc+0x47/0x130
<4>[427882.173870] [<ffffffff816334da>] bio_clone_mddev+0x1a/0x30
<4>[427882.179566] [<ffffffff816176f8>] make_request+0x608/0xe90
<4>[427882.185173] [<ffffffff8133b915>] ?
generic_make_request_checks+0x125/0x2f0
<4>[427882.192254] [<ffffffff816362ae>] md_make_request+0x7e/0x260
<4>[427882.198037] [<ffffffff81645759>] ? dm_put_live_table+0x9/0x10
<4>[427882.203991] [<ffffffff81645ef9>] ? dm_request+0x99/0x110
<4>[427882.209511] [<ffffffff8133bb7e>] generic_make_request+0x9e/0xf0
<4>[427882.215652] [<ffffffffa0186bfd>] issue+0x3d/0xb0 [dm_thin_pool]
<4>[427882.221786] [<ffffffffa0186c97>] remap_and_issue+0x27/0x40 [dm_thin_pool]
<4>[427882.228787] [<ffffffffa0187378>]
inc_remap_and_issue_cell+0xb8/0xd0 [dm_thin_pool]
<4>[427882.236566] [<ffffffffa0186c97>] ? remap_and_issue+0x27/0x40
[dm_thin_pool]
<4>[427882.243740] [<ffffffffa01874bf>]
process_prepared_mapping+0x12f/0x140 [dm_thin_pool]
<4>[427882.251694] [<ffffffffa01875e5>] schedule_zero+0x115/0x160 [dm_thin_pool]
<4>[427882.258694] [<ffffffffa0188788>] process_cell+0x618/0x620 [dm_thin_pool]
<4>[427882.265603] [<ffffffff810eacd7>] ? mempool_alloc+0x47/0x130
<4>[427882.271388] [<ffffffff8137054a>] ? sort+0x13a/0x200
<4>[427882.276477] [<ffffffff813703e0>] ? u32_swap+0x10/0x10
<4>[427882.281743] [<ffffffffa0185860>] ?
dm_thin_volume_is_full+0x30/0x30 [dm_thin_pool]
<4>[427882.289527] [<ffffffffa0189f9f>] do_worker+0x2af/0x820 [dm_thin_pool]
<4>[427882.296198] [<ffffffff81066021>] ? pwq_activate_first_delayed+0x11/0x20
<4>[427882.303035] [<ffffffff81069403>] process_one_work+0x103/0x2f0
<4>[427882.309015] [<ffffffff81069a17>] worker_thread+0x117/0x390
<4>[427882.314729] [<ffffffff81069900>] ? rescuer_thread+0x2e0/0x2e0
<4>[427882.320683] [<ffffffff8106d85e>] kthread+0xde/0xf0
<4>[427882.325684] [<ffffffff8106d780>] ? kthreadd+0x150/0x150
<4>[427882.331120] [<ffffffff818ba1c8>] ret_from_fork+0x58/0x90
<4>[427882.336640] [<ffffffff8106d780>] ? kthreadd+0x150/0x150
Anyone can give me some advice or help.
Thanks,
--
Chien Lee
^ permalink raw reply
* Re: raid5/6: general protection fault in async_copy_data
From: Joey Liao @ 2016-07-18 10:47 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <CAHvN=in6MFvx7=BN1asctsA=EMPZthQBHUfx5TwY5UmUsmcMFw@mail.gmail.com>
Hi all,
Does anyone have idea about this call trace in linux 3.19.8...???
2016-07-04 16:07 GMT+08:00 Joey Liao <joeyliao@qnap.com>:
> Hi,
>
> We meet a general protection fault issue when doing the I/O stress
> test. (Please refer the following call trace.)
>
> The linux version what we use is 3.19.8.
>
> Besides, it will not only happen in raid5 but also raid6, and our raid
> is normal and clean (not in resync or rebuild or degraded state) in
> this case.
> The general protection fault logs just show-up suddenly during the I/O
> stress test, and no other logs else are before it.
>
> It seems that it is an old issue that someone has been meet in linux 3.8.13.
> http://comments.gmane.org/gmane.linux.raid/48737
>
> However, it did not come to an exact conclusion then.
> Does anyone have any idea about this situation?
>
>
> <4>[ 8415.258965] general protection fault: 0000 [#1] SMP
> <4>[ 8415.263946] Modules linked in: vfio_pci vfio_iommu_type1 vfio
> vringh virtio_scsi virtio_pci virtio_mmio virtio_console
> virtio_balloon virtio_rng virtio_blk virtio_net virtio_ring virtio
> vhost_net vhost tun macvtap macvlan kvm_intel kvm fbdisk(O) xt_mark
> ipt_MASQUERADE iptable_nat nf_nat_masquerade_ipv4 nf_nat_ipv4 nf_nat
> ppp_deflate bsd_comp ppp_mppe ppp_async ppp_generic slhc iscsi_tcp(O)
> libiscsi_tcp(O) libiscsi(O) scsi_transport_iscsi(O) btusb bluetooth
> bonding bridge stp ipv6 uvcvideo videobuf2_vmalloc videobuf2_memops
> videobuf2_core snd_usb_caiaq snd_usb_audio snd_usbmidi_lib
> snd_seq_midi snd_rawmidi fnotify(PO) udf isofs iTCO_wdt psnap llc
> ufsd(PO) jnl(O) pl2303 usbserial intel_ips drbd(O) flashcache(O)
> dm_thin_pool dm_bio_prison dm_persistent_data hal_netlink(O) coretemp
> r8152 usbnet mii igb e1000e(O) mpt3sas mpt2sas scsi_transport_sas
> raid_class uas usb_storage xhci_pci xhci_hcd usblp uhci_hcd ehci_pci
> ehci_hcd [last unloaded: fbdisk]
> <4>[ 8415.348888] CPU: 0 PID: 4611 Comm: md1_raid5 Tainted: P U
> O 3.19.8 #1
> <4>[ 8415.356137] Hardware name: Default string Default string/SKYBAY,
> BIOS QX80AR20 06/07/2016
> <4>[ 8415.364249] task: ffff880847616210 ti: ffff880831950000 task.ti:
> ffff880831950000
> <4>[ 8415.371667] RIP: 0010:[<ffffffff813c4446>] [<ffffffff813c4446>]
> memcpy+0x6/0x110
> <4>[ 8415.379126] RSP: 0000:ffff880831953990 EFLAGS: 00210206
> <4>[ 8415.384398] RAX: ffff880832e87000 RBX: ffff880831953a18 RCX:
> 0000000000001000
> <4>[ 8415.391475] RDX: 0000000000001000 RSI: 0845080000067000 RDI:
> ffff880832e87000
> <4>[ 8415.398553] RBP: ffff8808319539c8 R08: 0000000000001000 R09:
> ffff880831953a18
> <4>[ 8415.405660] R10: 0000000000000001 R11: 0000000000000001 R12:
> ffffea0020cba1c0
> <4>[ 8415.412750] R13: 0000000000000000 R14: 002100000000000e R15:
> 0000000000067000
> <4>[ 8415.419829] FS: 0000000000000000(0000)
> GS:ffff88086dc00000(0000) knlGS:0000000000000000
> <4>[ 8415.427852] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> <4>[ 8415.433550] CR2: 0000000008120480 CR3: 0000000001c7d000 CR4:
> 00000000003407f0
> <4>[ 8415.440648] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> <4>[ 8415.447748] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> 0000000000000400
> <4>[ 8415.454823] Stack:
> <4>[ 8415.456849] ffffffff8138408f 0000000000001000 0000000000080000
> 0000000000080000
> <4>[ 8415.464283] 0000000000000000 0000000000000000 0000000000001000
> ffff880831953a78
> <4>[ 8415.471720] ffffffff8169911e ffff880831953a18 ffffffff8138a5b7
> ffff880831953a18
> <4>[ 8415.479144] Call Trace:
> <4>[ 8415.481578] [<ffffffff8138408f>] ? async_memcpy+0x9f/0x100
> <4>[ 8415.487122] [<ffffffff8169911e>] async_copy_data+0x12e/0x260
> <4>[ 8415.492824] [<ffffffff8138a5b7>] ? bio_attempt_front_merge+0xb7/0xf0
> <4>[ 8415.499214] [<ffffffff81699bb6>] raid_run_ops+0x876/0x1190
> <4>[ 8415.504741] [<ffffffff8138c5b3>] ? generic_make_request+0xa3/0xf0
> <4>[ 8415.510872] [<ffffffff8169f716>] ? ops_run_io+0x36/0x820
> <4>[ 8415.516233] [<ffffffff81085763>] ? __wake_up+0x53/0x70
> <4>[ 8415.521417] [<ffffffff816a0e7b>] handle_stripe+0xa8b/0x1b50
> <4>[ 8415.527031] [<ffffffff81085206>] ? __wake_up_common+0x16/0x90
> <4>[ 8415.532816] [<ffffffff816a22da>] handle_active_stripes+0x39a/0x450
> <4>[ 8415.539035] [<ffffffff816a2849>] raid5d+0x399/0x630
> <4>[ 8415.543964] [<ffffffff816ac4ed>] md_thread+0x7d/0x130
> <4>[ 8415.549064] [<ffffffff81085370>] ? woken_wake_function+0x20/0x20
> <4>[ 8415.555107] [<ffffffff816ac470>] ? errors_store+0x70/0x70
> <4>[ 8415.560552] [<ffffffff8106a263>] kthread+0xe3/0xf0
> <4>[ 8415.565411] [<ffffffff8106a180>] ? kthreadd+0x160/0x160
> <4>[ 8415.570682] [<ffffffff8192d048>] ret_from_fork+0x58/0x90
> <4>[ 8415.576044] [<ffffffff8106a180>] ? kthreadd+0x160/0x160
> <4>[ 8415.581315] Code: 24 4c 8b 64 24 08 c9 c3 e8 68 f9 ff ff 41 80
> 7c 24 05 00 75 d3 eb e4 90 90 90 90 90 90 90 90 90 90 90 90 90 90 48
> 89 f8 48 89 d1 <f3> a4 c3 03 83 e2 07 f3 48 a5 89 d1 f3 a4 c3 20 4c 8b
> 06 4c 8b
> <1>[ 8415.601660] RIP [<ffffffff813c4446>] memcpy+0x6/0x110
> <4>[ 8415.606788] RSP <ffff880831953990>
> <4>[ 8415.612291] ---[ end trace 962cfd98d43b82fa ]---
> <4>[ 8415.620528] ------------[ cut here ]------------
^ permalink raw reply
* [PATCH] Remove: container should wait for an array to release a drive
From: Tomasz Majchrzak @ 2016-07-18 12:11 UTC (permalink / raw)
To: linux-raid
Cc: Jes.Sorensen, aleksey.obitotskiy, pawel.baldysiak,
artur.paszkiewicz
A 'faulty' drive is being removed from a container after it has been
released by an array, however there is a race there. The drive is
released asynchronously by a monitor but sometimes it doesn't happen
before container checks it. It results in a container refusing to remove
a drive as it still seems to be a part of some array.
It seems 'ping_monitor' could be a solution here to assure monitor has
had a chance to process the events, however it doesn't resolve the
problem - sometimes an array has to request a release of the drive few
times (as the array is busy) and single 'ping_monitor' call is not
sufficient. As there is no way to query monitor progress, it forces us
to retry a check several times before an error is returned.
Signed-off-by: Tomasz Majchrzak <tomasz.majchrzak@intel.com>
---
Manage.c | 38 +++++++++++++++++++++++++-------------
1 file changed, 25 insertions(+), 13 deletions(-)
diff --git a/Manage.c b/Manage.c
index e2e88b8..7f8eb88 100644
--- a/Manage.c
+++ b/Manage.c
@@ -1125,19 +1125,31 @@ int Manage_remove(struct supertype *tst, int fd, struct mddev_dev *dv,
*/
if (rdev == 0)
ret = -1;
- else
- ret = sysfs_unique_holder(devnm, rdev);
- if (ret == 0) {
- pr_err("%s is not a member, cannot remove.\n",
- dv->devname);
- close(lfd);
- return -1;
- }
- if (ret >= 2) {
- pr_err("%s is still in use, cannot remove.\n",
- dv->devname);
- close(lfd);
- return -1;
+ else {
+ /* The drive has already been set to 'faulty', however monitor might
+ * not have had time to process it and the drive might still have
+ * an entry in the 'holders' directory. Try a few times to avoid
+ * a false error */
+ int count = 20;
+ do {
+ ret = sysfs_unique_holder(devnm, rdev);
+ if (ret < 2)
+ break;
+ usleep(100000);
+ } while (--count > 0);
+
+ if (ret == 0) {
+ pr_err("%s is not a member, cannot remove.\n",
+ dv->devname);
+ close(lfd);
+ return -1;
+ }
+ if (ret >= 2) {
+ pr_err("%s is still in use, cannot remove.\n",
+ dv->devname);
+ close(lfd);
+ return -1;
+ }
}
}
/* FIXME check that it is a current member */
--
1.8.3.1
^ permalink raw reply related
* superblock help
From: Bryan Hepworth @ 2016-07-18 13:33 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
Hi
I'm trying to salvage some data from disk from a raid I didn't assemble. This is the only disk that has information on. It gets picked up ok as a linux ext4 raid partition in gparted. I was expecting to see raid 5 over four disks, sadly not - the other disks have gpt partitions but report as being unallocated in gparted and gdisk.
Any suggestions as to what to try next in finding a superblock?
Thanks
Bryan
<i>Filesystem volume name: inscribe_data1
Last mounted on: <not available>
Filesystem UUID: 95c359b0-66bf-4394-89da-4d7866a92475
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags: signed_directory_hash
Default mount options: (none)
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 122101760
Block count: 488378368
Reserved block count: 24418918
Free blocks: 480664949
Free inodes: 122101749
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 907
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 8192
Inode blocks per group: 512
Flex block group size: 16
Filesystem created: Thu Aug 28 13:57:16 2014
Last mount time: n/a
Last write time: Thu Aug 28 14:01:42 2014
Mount count: 0
Maximum mount count: 33
Last checked: Thu Aug 28 13:57:16 2014
Check interval: 15552000 (6 months)
Next check after: Tue Feb 24 12:57:16 2015
Lifetime writes: 29 GB
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Required extra isize: 28
Desired extra isize: 28
Journal inode: 8
Default directory hash: half_md4
Directory Hash Seed: 8c787fea-f93e-453d-ac45-d49a339628ab
Journal backup: inode blocks</i>
<i>dumpe2fs 1.41.12 (17-May-2010)
Journal superblock magic number invalid!</i>
<i>Unable to read the contents of this file system!
Because of this some operations may be unavailable.
The cause might be a missing software package.
The following list of software packages is required for ext4 file system support: e2fsprogs v1.41+.</i>
[root@igmimager sbin]# ./gdisk /dev/sdn
GPT fdisk (gdisk) version 0.8.10
Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present
Found valid GPT with protective MBR; using GPT.
Command (? for help): ?
b back up GPT data to a file
c change a partition's name
d delete a partition
i show detailed information on a partition
l list known partition types
n add a new partition
o create a new empty GUID partition table (GPT)
p print the partition table
q quit without saving changes
r recovery and transformation options (experts only)
s sort partitions
t change a partition's type code
v verify disk
w write table to disk and exit
x extra functionality (experts only)
? print this menu
Command (? for help): p
Disk /dev/sdn: 3907029168 sectors, 1.8 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): E6755AAC-5295-4417-BA21-E1016486CBDB
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 3907029134
Partitions will be aligned on 2048-sector boundaries
Total free space is 2157 sectors (1.1 MiB)
Number Start (sector) End (sector) Size Code Name
1 2048 3907028991 1.8 TiB FD00
Command (? for help): i
Using 1
Partition GUID code: A19D880F-05FC-4D3B-A006-743F0F84911E (Linux RAID)
Partition unique GUID: 911F59F3-AB98-4A40-9F0B-B191B5C5010E
First sector: 2048 (at 1024.0 KiB)
Last sector: 3907028991 (at 1.8 TiB)
Partition size: 3907026944 sectors (1.8 TiB)
Attribute flags: 0000000000000000
Partition name: ''
Command (? for help):
^ permalink raw reply
* Re: [PATCH] Remove: container should wait for an array to release a drive
From: John Stoffel @ 2016-07-18 13:37 UTC (permalink / raw)
To: Tomasz Majchrzak
Cc: linux-raid, Jes.Sorensen, aleksey.obitotskiy, pawel.baldysiak,
artur.paszkiewicz
In-Reply-To: <1468843890-30425-1-git-send-email-tomasz.majchrzak@intel.com>
>>>>> "Tomasz" == Tomasz Majchrzak <tomasz.majchrzak@intel.com> writes:
Tomasz> A 'faulty' drive is being removed from a container after it
Tomasz> has been released by an array, however there is a race
Tomasz> there. The drive is released asynchronously by a monitor but
Tomasz> sometimes it doesn't happen before container checks it. It
Tomasz> results in a container refusing to remove a drive as it still
Tomasz> seems to be a part of some array.
Tomasz> It seems 'ping_monitor' could be a solution here to assure
Tomasz> monitor has had a chance to process the events, however it
Tomasz> doesn't resolve the problem - sometimes an array has to
Tomasz> request a release of the drive few times (as the array is
Tomasz> busy) and single 'ping_monitor' call is not sufficient. As
Tomasz> there is no way to query monitor progress, it forces us to
Tomasz> retry a check several times before an error is returned.
Tomasz> Signed-off-by: Tomasz Majchrzak <tomasz.majchrzak@intel.com>
Tomasz> ---
Tomasz> Manage.c | 38 +++++++++++++++++++++++++-------------
Tomasz> 1 file changed, 25 insertions(+), 13 deletions(-)
Tomasz> diff --git a/Manage.c b/Manage.c
Tomasz> index e2e88b8..7f8eb88 100644
Tomasz> --- a/Manage.c
Tomasz> +++ b/Manage.c
Tomasz> @@ -1125,19 +1125,31 @@ int Manage_remove(struct supertype *tst, int fd, struct mddev_dev *dv,
Tomasz> */
Tomasz> if (rdev == 0)
Tomasz> ret = -1;
Tomasz> - else
Tomasz> - ret = sysfs_unique_holder(devnm, rdev);
Tomasz> - if (ret == 0) {
Tomasz> - pr_err("%s is not a member, cannot remove.\n",
Tomasz> - dv->devname);
Tomasz> - close(lfd);
Tomasz> - return -1;
Tomasz> - }
Tomasz> - if (ret >= 2) {
Tomasz> - pr_err("%s is still in use, cannot remove.\n",
Tomasz> - dv->devname);
Tomasz> - close(lfd);
Tomasz> - return -1;
Tomasz> + else {
Tomasz> + /* The drive has already been set to 'faulty', however monitor might
Tomasz> + * not have had time to process it and the drive might still have
Tomasz> + * an entry in the 'holders' directory. Try a few times to avoid
Tomasz> + * a false error */
Tomasz> + int count = 20;
Tomasz> + do {
Tomasz> + ret = sysfs_unique_holder(devnm, rdev);
Tomasz> + if (ret < 2)
Tomasz> + break;
Tomasz> + usleep(100000);
Really, you're sleeping 10 seconds without telling the user? That
seems to be a bit obnoxious. Logging something here would be good.
^ permalink raw reply
* Re: [PATCH] Remove: container should wait for an array to release a drive
From: Jes Sorensen @ 2016-07-18 20:55 UTC (permalink / raw)
To: John Stoffel
Cc: Tomasz Majchrzak, linux-raid, aleksey.obitotskiy, pawel.baldysiak,
artur.paszkiewicz
In-Reply-To: <22412.56226.478064.375583@quad.stoffel.home>
"John Stoffel" <john@stoffel.org> writes:
>>>>>> "Tomasz" == Tomasz Majchrzak <tomasz.majchrzak@intel.com> writes:
>
> Tomasz> A 'faulty' drive is being removed from a container after it
> Tomasz> has been released by an array, however there is a race
> Tomasz> there. The drive is released asynchronously by a monitor but
> Tomasz> sometimes it doesn't happen before container checks it. It
> Tomasz> results in a container refusing to remove a drive as it still
> Tomasz> seems to be a part of some array.
>
> Tomasz> It seems 'ping_monitor' could be a solution here to assure
> Tomasz> monitor has had a chance to process the events, however it
> Tomasz> doesn't resolve the problem - sometimes an array has to
> Tomasz> request a release of the drive few times (as the array is
> Tomasz> busy) and single 'ping_monitor' call is not sufficient. As
> Tomasz> there is no way to query monitor progress, it forces us to
> Tomasz> retry a check several times before an error is returned.
>
> Tomasz> Signed-off-by: Tomasz Majchrzak <tomasz.majchrzak@intel.com>
> Tomasz> ---
> Tomasz> Manage.c | 38 +++++++++++++++++++++++++-------------
> Tomasz> 1 file changed, 25 insertions(+), 13 deletions(-)
>
> Tomasz> diff --git a/Manage.c b/Manage.c
> Tomasz> index e2e88b8..7f8eb88 100644
> Tomasz> --- a/Manage.c
> Tomasz> +++ b/Manage.c
> Tomasz> @@ -1125,19 +1125,31 @@ int Manage_remove(struct supertype *tst, int fd, struct mddev_dev *dv,
> Tomasz> */
> Tomasz> if (rdev == 0)
> Tomasz> ret = -1;
> Tomasz> - else
> Tomasz> - ret = sysfs_unique_holder(devnm, rdev);
> Tomasz> - if (ret == 0) {
> Tomasz> - pr_err("%s is not a member, cannot remove.\n",
> Tomasz> - dv->devname);
> Tomasz> - close(lfd);
> Tomasz> - return -1;
> Tomasz> - }
> Tomasz> - if (ret >= 2) {
> Tomasz> - pr_err("%s is still in use, cannot remove.\n",
> Tomasz> - dv->devname);
> Tomasz> - close(lfd);
> Tomasz> - return -1;
> Tomasz> + else {
> Tomasz> + /* The drive has already been set to 'faulty', however monitor might
> Tomasz> + * not have had time to process it and the drive might still have
> Tomasz> + * an entry in the 'holders' directory. Try a few times to avoid
> Tomasz> + * a false error */
> Tomasz> + int count = 20;
> Tomasz> + do {
> Tomasz> + ret = sysfs_unique_holder(devnm, rdev);
> Tomasz> + if (ret < 2)
> Tomasz> + break;
> Tomasz> + usleep(100000);
>
> Really, you're sleeping 10 seconds without telling the user? That
> seems to be a bit obnoxious. Logging something here would be good.
Hi,
Sorry just back from vacation and just started attacking the mountain of
email.
I agree with John here, please add some logging message. Also is 10
seconds really needed? It seems an awful lot per iteration.
Cheers,
Jes
^ permalink raw reply
* Re: [PATCH] Remove: container should wait for an array to release a drive
From: Tomasz Majchrzak @ 2016-07-19 7:23 UTC (permalink / raw)
To: Jes Sorensen
Cc: John Stoffel, linux-raid, aleksey.obitotskiy, pawel.baldysiak,
artur.paszkiewicz
In-Reply-To: <wrfjzipey8cg.fsf@redhat.com>
On Mon, Jul 18, 2016 at 04:55:27PM -0400, Jes Sorensen wrote:
> "John Stoffel" <john@stoffel.org> writes:
> >>>>>> "Tomasz" == Tomasz Majchrzak <tomasz.majchrzak@intel.com> writes:
> >
> > Tomasz> A 'faulty' drive is being removed from a container after it
> > Tomasz> has been released by an array, however there is a race
> > Tomasz> there. The drive is released asynchronously by a monitor but
> > Tomasz> sometimes it doesn't happen before container checks it. It
> > Tomasz> results in a container refusing to remove a drive as it still
> > Tomasz> seems to be a part of some array.
> >
> > Tomasz> It seems 'ping_monitor' could be a solution here to assure
> > Tomasz> monitor has had a chance to process the events, however it
> > Tomasz> doesn't resolve the problem - sometimes an array has to
> > Tomasz> request a release of the drive few times (as the array is
> > Tomasz> busy) and single 'ping_monitor' call is not sufficient. As
> > Tomasz> there is no way to query monitor progress, it forces us to
> > Tomasz> retry a check several times before an error is returned.
> >
> > Tomasz> Signed-off-by: Tomasz Majchrzak <tomasz.majchrzak@intel.com>
> > Tomasz> ---
> > Tomasz> Manage.c | 38 +++++++++++++++++++++++++-------------
> > Tomasz> 1 file changed, 25 insertions(+), 13 deletions(-)
> >
> > Tomasz> diff --git a/Manage.c b/Manage.c
> > Tomasz> index e2e88b8..7f8eb88 100644
> > Tomasz> --- a/Manage.c
> > Tomasz> +++ b/Manage.c
> > Tomasz> @@ -1125,19 +1125,31 @@ int Manage_remove(struct supertype *tst, int fd, struct mddev_dev *dv,
> > Tomasz> */
> > Tomasz> if (rdev == 0)
> > Tomasz> ret = -1;
> > Tomasz> - else
> > Tomasz> - ret = sysfs_unique_holder(devnm, rdev);
> > Tomasz> - if (ret == 0) {
> > Tomasz> - pr_err("%s is not a member, cannot remove.\n",
> > Tomasz> - dv->devname);
> > Tomasz> - close(lfd);
> > Tomasz> - return -1;
> > Tomasz> - }
> > Tomasz> - if (ret >= 2) {
> > Tomasz> - pr_err("%s is still in use, cannot remove.\n",
> > Tomasz> - dv->devname);
> > Tomasz> - close(lfd);
> > Tomasz> - return -1;
> > Tomasz> + else {
> > Tomasz> + /* The drive has already been set to 'faulty', however monitor might
> > Tomasz> + * not have had time to process it and the drive might still have
> > Tomasz> + * an entry in the 'holders' directory. Try a few times to avoid
> > Tomasz> + * a false error */
> > Tomasz> + int count = 20;
> > Tomasz> + do {
> > Tomasz> + ret = sysfs_unique_holder(devnm, rdev);
> > Tomasz> + if (ret < 2)
> > Tomasz> + break;
> > Tomasz> + usleep(100000);
> >
> > Really, you're sleeping 10 seconds without telling the user? That
> > seems to be a bit obnoxious. Logging something here would be good.
>
> Hi,
>
> Sorry just back from vacation and just started attacking the mountain of
> email.
>
> I agree with John here, please add some logging message. Also is 10
> seconds really needed? It seems an awful lot per iteration.
>
> Cheers,
> Jes
Well, actually it's 20 iteration 100ms each so up to 2 seconds. I have never
seen it taking more than 3 iterations, however I don't have a full knowledge how
long it can take md module to release an array. I just added 2 seconds as a
precaution, better wait a bit longer than leave an array in inconsistent state.
Is it fine?
Regards,
Tomek
^ permalink raw reply
* Re: [PATCH v2 1/1] block: fix blk_queue_split() resource exhaustion
From: Lars Ellenberg @ 2016-07-19 9:00 UTC (permalink / raw)
To: Mike Snitzer
Cc: Eric Wheeler, NeilBrown, Jens Axboe, linux-block,
Martin K. Petersen, Peter Zijlstra, Jiri Kosina, Ming Lei,
Kirill A. Shutemov, linux-kernel, linux-raid, Takashi Iwai,
linux-bcache, Zheng Liu, Kent Overstreet, Keith Busch, dm-devel,
Shaohua Li, Ingo Molnar, Alasdair Kergon, Roland Kammerer
In-Reply-To: <20160713023232.GA6034@redhat.com>
On Tue, Jul 12, 2016 at 10:32:33PM -0400, Mike Snitzer wrote:
> On Tue, Jul 12 2016 at 10:18pm -0400,
> Eric Wheeler <bcache@lists.ewheeler.net> wrote:
>
> > On Tue, 12 Jul 2016, NeilBrown wrote:
> >
> > > On Tue, Jul 12 2016, Lars Ellenberg wrote:
> > > ....
> > > >
> > > > Instead, I suggest to distinguish between recursive calls to
> > > > generic_make_request(), and pushing back the remainder part in
> > > > blk_queue_split(), by pointing current->bio_lists to a
> > > > struct recursion_to_iteration_bio_lists {
> > > > struct bio_list recursion;
> > > > struct bio_list queue;
> > > > }
> > > >
> > > > By providing each q->make_request_fn() with an empty "recursion"
> > > > bio_list, then merging any recursively submitted bios to the
> > > > head of the "queue" list, we can make the recursion-to-iteration
> > > > logic in generic_make_request() process deepest level bios first,
> > > > and "sibling" bios of the same level in "natural" order.
> > > >
> > > > Signed-off-by: Lars Ellenberg <lars.ellenberg@linbit.com>
> > > > Signed-off-by: Roland Kammerer <roland.kammerer@linbit.com>
> > >
> > > Reviewed-by: NeilBrown <neilb@suse.com>
> > >
> > > Thanks again for doing this - I think this is a very significant
> > > improvement and could allow other simplifications.
> >
> > Thank you Lars for all of this work!
> >
> > It seems like there have been many 4.3+ blockdev stacking issues and this
> > will certainly address some of those (maybe all of them?). (I think we
> > hit this while trying drbd in 4.4 so we dropped back to 4.1 without
> > issue.) It would be great to hear 4.4.y stable pick this up if
> > compatible.
> >
> >
> > Do you believe that this patch would solve any of the proposals by others
> > since 4.3 related to bio splitting/large bios? I've been collecting a
> > list, none of which appear have landed yet as of 4.7-rc7 (but correct me
> > if I'm wrong):
> >
> > A. [PATCH v2] block: make sure big bio is splitted into at most 256 bvecs
> > by Ming Lei: https://patchwork.kernel.org/patch/9169483/
That's an independend issue.
> > B. block: don't make BLK_DEF_MAX_SECTORS too big
> > by Shaohua Li: http://www.spinics.net/lists/linux-bcache/msg03525.html
Yet an other independend issue.
> > C. [1/3] block: flush queued bios when process blocks to avoid deadlock
> > by Mikulas Patocka: https://patchwork.kernel.org/patch/9204125/
> > (was https://patchwork.kernel.org/patch/7398411/)
As it stands now,
this is yet an other issue, but related.
From the link above:
| ** Here is the dm-snapshot deadlock that was observed:
|
| 1) Process A sends one-page read bio to the dm-snapshot target. The bio
| spans snapshot chunk boundary and so it is split to two bios by device
| mapper.
|
| 2) Device mapper creates the first sub-bio and sends it to the snapshot
| driver.
|
| 3) The function snapshot_map calls track_chunk (that allocates a
| structure
| dm_snap_tracked_chunk and adds it to tracked_chunk_hash) and then remaps
| the bio to the underlying device and exits with DM_MAPIO_REMAPPED.
|
| 4) The remapped bio is submitted with generic_make_request, but it isn't
| issued - it is added to current->bio_list instead.
|
| 5) Meanwhile, process B (dm's kcopyd) executes pending_complete for the
| chunk affected be the first remapped bio, it takes down_write(&s->lock)
| and then loops in __check_for_conflicting_io, waiting for
| dm_snap_tracked_chunk created in step 3) to be released.
|
| 6) Process A continues, it creates a second sub-bio for the rest of the
| original bio.
Aha.
Here is the relation.
If "A" had only ever processed "just the chunk it can handle now",
and "pushed back" the rest of the incoming bio,
it could rely on all deeper level bios to have been submitted already.
But this does not look like it easily fits into the current DM model.
| 7) snapshot_map is called for this new bio, it waits on
| down_write(&s->lock) that is held by Process B (in step 5).
There is an other suggestion:
Use down_trylock (or down_timeout),
and if it fails, push back the currently to-be-processed bio.
We can introduce a new bio helper for that.
Kind of what blk_queue_split() does with my patch applied.
Or even better, ignore the down_trylock suggestion,
simply not iterate over all pieces first,
but process one piece, and return back the the
iteration in generic_make_request.
A bit of conflict here may be that DM has all its own
split and clone and queue magic, and wants to process
"all of the bio" before returning back to generic_make_request().
To change that, __split_and_process_bio() and all its helpers
would need to learn to "push back" (pieces of) the bio they are
currently working on, and not push back via "DM_ENDIO_REQUEUE",
but by bio_list_add_head(¤t->bio_lists->queue, piece_to_be_done_later).
Then, after they processed each piece,
*return* all the way up to the top-level generic_make_request(),
where the recursion-to-iteration logic would then
make sure that all deeper level bios, submitted via
recursive calls to generic_make_request() will be processed, before the
next, pushed back, piece of the "original incoming" bio.
And *not* do their own iteration over all pieces first.
Probably not as easy as dropping the while loop,
using bio_advance, and pushing that "advanced" bio back to
current->...queue?
static void __split_and_process_bio(struct mapped_device *md,
struct dm_table *map, struct bio *bio)
...
ci.bio = bio;
ci.sector_count = bio_sectors(bio);
while (ci.sector_count && !error)
error = __split_and_process_non_flush(&ci);
...
error = __split_and_process_non_flush(&ci);
if (ci.sector_count)
bio_advance()
bio_list_add_head(¤t->bio_lists->queue, )
...
Something like that, maybe?
Just a thought.
> > D. dm-crypt: Fix error with too large bios
> > by Mikulas Patocka: https://patchwork.kernel.org/patch/9138595/
> >
> > The A,B,D are known to fix large bio issues when stacking dm+bcache
> > (though the B,D are trivial and probably necessary even with your patch).
> >
> > Patch C was mentioned earlier in this thread by Mike Snitzer and you
> > commented briefly that his patch might solve the issue; given that, and in
> > the interest of minimizing duplicate effort, which of the following best
> > describes the situation?
> >
> > 1. Your patch could supersede Mikulas's patch; they address the same
> > issue.
> >
> > 2. Mikulas's patch addresses different issues such and both patches
> > should be applied.
> >
> > 3. There is overlap between both your patch and Mikulas's such that both
> > #1,#2 are true and effort to solve this has been duplicated.
> >
> >
> > If #3, then what might be done to resolve the overlap?
>
> Mikulas confirmed to me that he believes Lars' v2 patch will fix the
> dm-snapshot problem, which is being tracked with this BZ:
> https://bugzilla.kernel.org/show_bug.cgi?id=119841
>
> We'll see how testing goes (currently underway).
^ permalink raw reply
* Re: Cannot start array on disk
From: Phil Turmel @ 2016-07-19 9:12 UTC (permalink / raw)
To: Bhatia Amit, linux-raid@vger.kernel.org
In-Reply-To: <1124605843.526681.1468785592654.JavaMail.yahoo@mail.yahoo.com>
On 07/17/2016 03:59 PM, Bhatia Amit wrote:
> Hi
>
> I have a WD Live duo setup with two 3TB drives in RAID1. After a
> recent power surge etc, I could not access the enclosure data. I
> removed the drives out from the enclosure and connected them via
> esata cable to a Linux laptop. One of the drives had physical noise
> and failed to connect at all. The other drive was able to connect and
> showed up as /dev/sdc. Following some online search, I issued the
> following commands to get some results, but am unable to mount the
> drive to a linux laptop via esata. Questions:
>
> * Ideally both sdc3 and sdc4 should have shown RAID=RAID1. If sdc4 is
> showing RAID=linear, is it because the other drive went totally bad,
> and software decided to reconfigure this drive from RAID1 to linear
> ?
No way to know. I would be inclined to believe that that you didn't
actually have raid1 set for those partitions. Do you recall the
available space in that device while running?
> * Given the one of the two drives is totally dead, is it still
> possible to reconfigure, mount and recover data from the single
> drive?
If truly linear, no, not past the half-way mark. The device is
identified as role 0, so it would be the beginning of the filesystem in
a linear array. If it really was raid1 with v0.9 or v1.0 metadata, you
can mount /dev/sdc4 directly to access your data. That it doesn't show
a detected filesystem suggests that it wasn't raid1.
Even worse, that partition shows a creation date and update date both in
2013, with no events. Suggesting that the device hasn't actually been
running properly in the array since its initial creation.
> * I tried scanning the drive via r-linux and it does show files
> exist, but does not show any folder information. So data is there on
> the drive. Is there a way to recreate the setup with just this single
> drive, so that I can extract files with folder information?
With v1.0 metadata, you should be able to fsck /dev/sdc4 and then mount.
Kinda dangerous though -- I suggest you use an overlay for that. I
suspect that sdc4 hasn't been running in your array for the past three
years.
> * Looking at an online "derrick" script, the script seems to call
> mdadm create with the "missing" flag. Is that something I should be
> doing to be able to assemble and recover data from the drive ?
> "mdadm --create $rootfsRaid --verbose --metadata=0.9 --raid-devices=2
> --level=raid1 --run $diskRoot1 missing"
No.
Using --create is practically never the right solution, unless you've
done operations that destroy the right solution. Metadata v0.9 is
almost certainly wrong.
If you have syslogs on your other array that show the boot details from
the last successful boot *before* it died, that might provide the
details needed to figure if anything recent is on sdc4.
Phil
^ permalink raw reply
* Re: [PATCH RFC] md/raid1: fix deadlock between freeze_array() and wait_barrier().
From: Alexander Lyakas @ 2016-07-19 9:20 UTC (permalink / raw)
To: NeilBrown; +Cc: 马建朋, linux-raid, Jes Sorensen, Shaohua Li
In-Reply-To: <87lh13dd1a.fsf@notabene.neil.brown.name>
Hello Neil,
Thank you for your response.
On Fri, Jul 15, 2016 at 2:18 AM, NeilBrown <neilb@suse.com> wrote:
> On Thu, Jul 14 2016, Alexander Lyakas wrote:
>
>>> diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
>>> index 32e282f4c83c..c528102b80b6 100644
>>> --- a/drivers/md/raid10.c
>>> +++ b/drivers/md/raid10.c
>>> @@ -1288,7 +1288,7 @@ read_again:
>>> sectors_handled;
>>> goto read_again;
>>> } else
>>> - generic_make_request(read_bio);
>>> + bio_list_add_head(¤t->bio_list, read_bio);
>>> return;
>>> }
>>
>> Unfortunately, this patch doesn't work. It is super elegant, and seems
>> like it really should work. But the problem is that the "rdev", to
>> which we want to send the READ bio, might also be a remapping device
>> (dm-linear, for example). This device will create its own remapped-bio
>> and will call generic_make_request(), which will stick the bio onto
>> current->bio_list TAIL:(:(:( So we are back at square one. This patch
>> would work if *all* the remapping drivers in the stack were doing
>> bio_list_add_head() instead of generic_make_request() :(:(:(
>>
>> It seems the real fix should be that generic_make_request() would use
>> bio_list_add_head(), but as pointed in
>> http://www.spinics.net/lists/raid/msg52756.html, there are some
>> concerns about changing the order of remapped bios.
>>
>
> While those concerns are valid, they are about hypothetical performance
> issues rather than observed deadlock issues. So I wouldn't be too
> worried about them.
I am thinking of a hypothetical driver that splits say a 12Kb WRITE
into 3x4kb WRITEs, and submits them in a proper order, hoping they
will get to the disk in the same order, and the disk will work
sequentially. But now we are deliberately hindering this. But I see
that people much smarter than me are in this discussion, so I will
leave it to them:)
> However I think you said that you didn't want to touch core code at all
> (maybe I misunderstood) so that wouldn't help you anyway.
Yes, this is correct. Recompiling the kernel is a bit of a pain for
us. We were smart enough to configure the md_mod as loadable module,
so at least now I can patch MD code easily:)
>
> One option would be to punt the request requests to raidXd:
>
> diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
> index 40b35be34f8d..f795e27b2124 100644
> --- a/drivers/md/raid1.c
> +++ b/drivers/md/raid1.c
> @@ -1229,7 +1229,7 @@ read_again:
> sectors_handled;
> goto read_again;
> } else
> - generic_make_request(read_bio);
> + reschedule_retry(r1_bio);
> return;
> }
>
> diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
> index 32e282f4c83c..eec38443075b 100644
> --- a/drivers/md/raid10.c
> +++ b/drivers/md/raid10.c
> @@ -1288,7 +1288,7 @@ read_again:
> sectors_handled;
> goto read_again;
> } else
> - generic_make_request(read_bio);
> + reschedule_retry(r10_bio);
> return;
> }
This is more or less what my rudimentary patch is doing, except it is
doing it only when we really need to wait for the barrier.
>
>
> That might hurt performance, you would need to measure.
> The other approach would be to revert the patch that caused the problem.
> e.g.
> diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
> index 40b35be34f8d..062bb86e5fd8 100644
> --- a/drivers/md/raid1.c
> +++ b/drivers/md/raid1.c
> @@ -884,7 +884,8 @@ static bool need_to_wait_for_sync(struct r1conf *conf, struct bio *bio)
> wait = false;
> else
> wait = true;
> - }
> + } else if (conf->barrier)
> + wait = true;
>
> return wait;
> }
>
>
I am not sure how this patch helps. You added another condition, and
now READs will also wait in some cases. But still if array_frozen is
set, everybody will wait unconditionally, which is the root cause for
the deadlock I think.
I see that there will be no magic solution for this problem:(
>
>>
>>>
>>>
>>>>
>>>> Since this issue is a real deadlock we are hitting in a long-term 3.18
>>>> kernel, is there any chance for cc-stable fix? Currently we applied
>>>> the rudimentary fix I posted. It basically switches context for
>>>> problematic RAID1 READs, and runs them from a different context. With
>>>> this fix we don't see the deadlock anymore.
>>>>
>>>> Also, can you please comment on another concern I expressed:
>>>> freeze_array() is now not reentrant. Meaning that if two threads call
>>>> it in parallel (and it could happen for the same MD), the first thread
>>>> calling unfreeze_array will mess up things for the second thread.
>>>
>>> Yes, that is a regression. This should be enough to fix it. Do you
>>> agree?
>>>
>>> Thanks,
>>> NeilBrown
>>>
>>>
>>> diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
>>> index 40b35be34f8d..5ad25c7d7453 100644
>>> --- a/drivers/md/raid1.c
>>> +++ b/drivers/md/raid1.c
>>> @@ -984,7 +984,7 @@ static void freeze_array(struct r1conf *conf, int extra)
>>> * we continue.
>>> */
>>> spin_lock_irq(&conf->resync_lock);
>>> - conf->array_frozen = 1;
>>> + conf->array_frozen += 1;
>>> wait_event_lock_irq_cmd(conf->wait_barrier,
>>> conf->nr_pending == conf->nr_queued+extra,
>>> conf->resync_lock,
>>> @@ -995,7 +995,7 @@ static void unfreeze_array(struct r1conf *conf)
>>> {
>>> /* reverse the effect of the freeze */
>>> spin_lock_irq(&conf->resync_lock);
>>> - conf->array_frozen = 0;
>>> + conf->array_frozen -= 1;
>>> wake_up(&conf->wait_barrier);
>>> spin_unlock_irq(&conf->resync_lock);
>>> }
>> I partially agree. The fix that you suggest makes proper accounting of
>> whether the array is considered frozen or not.
>> But the problem is that even with this fix, both threads will think
>> that they "own" the array. And both will do things, like writing data
>> or so, which might interfere. The proper fix would ensure that only
>> one thread "owns" the array, and the second thread waits until the
>> first calls unfreeze_array(), and then the second thread becomes
>> "owner" of the array. And of course while there are threads that want
>> to "freeze" the array, other threads that want to do raise_barrier
>> etc, should wait.
>
> Is that really a problem?
> A call to "freeze_array()" doesn't mean "I want to own the array", but
> rather "No regular IO should be happening now".
>
> Most callers of freeze_array():
> raid1_add_disk(), raid1_remove_disk(), stop(), raid1_reshape()
> are called with the "mddev_lock()" mutex held, so they cannot interfere
> with each other.
>
> handle_read_error() is called with one pending request, which will block
> any call on freeze_array(mddev, 0); - handle_read_error() itself calls
> freeze_array(mddev, 1) so it gets access.
> So these are already locked against the first four (as lock that
> ->array_frozen doesn't get corrupted).
>
> That just leaves raid1_quiesce().
> That is mostly called under mddev_lock() so it won't interfere with the
> others.
> The one exception is where md_do_sync() calls
> mddev->pers->quiesce(mddev, 1);
> mddev->pers->quiesce(mddev, 0);
>
> As this doesn't "claim" the array, but just needs to ensure all pending
> IO completes, I don't think there is a problem.
>
> So it seems to me that your concerns are not actually a problem.
> Did I miss something?
I think you are correct. The only exception is mddev_suspend/resume,
which can be called from another module. But for my case, this is not
happening.
Thanks,
Alex.
>
>>
>> I am sorry for giving two negative responses in one email:)
>
> Better a negative response than no response :-)
>
> NeilBrown
^ permalink raw reply
* Re: [PATCH] Remove: container should wait for an array to release a drive
From: John Stoffel @ 2016-07-19 13:15 UTC (permalink / raw)
To: Tomasz Majchrzak
Cc: Jes Sorensen, John Stoffel, linux-raid, aleksey.obitotskiy,
pawel.baldysiak, artur.paszkiewicz
In-Reply-To: <20160719072338.GA17053@proton.igk.intel.com>
>>>>> "Tomasz" == Tomasz Majchrzak <tomasz.majchrzak@intel.com> writes:
Tomasz> On Mon, Jul 18, 2016 at 04:55:27PM -0400, Jes Sorensen wrote:
>> "John Stoffel" <john@stoffel.org> writes:
>> >>>>>> "Tomasz" == Tomasz Majchrzak <tomasz.majchrzak@intel.com> writes:
>> >
>> > Tomasz> A 'faulty' drive is being removed from a container after it
>> > Tomasz> has been released by an array, however there is a race
>> > Tomasz> there. The drive is released asynchronously by a monitor but
>> > Tomasz> sometimes it doesn't happen before container checks it. It
>> > Tomasz> results in a container refusing to remove a drive as it still
>> > Tomasz> seems to be a part of some array.
>> >
>> > Tomasz> It seems 'ping_monitor' could be a solution here to assure
>> > Tomasz> monitor has had a chance to process the events, however it
>> > Tomasz> doesn't resolve the problem - sometimes an array has to
>> > Tomasz> request a release of the drive few times (as the array is
>> > Tomasz> busy) and single 'ping_monitor' call is not sufficient. As
>> > Tomasz> there is no way to query monitor progress, it forces us to
>> > Tomasz> retry a check several times before an error is returned.
>> >
>> > Tomasz> Signed-off-by: Tomasz Majchrzak <tomasz.majchrzak@intel.com>
>> > Tomasz> ---
>> > Tomasz> Manage.c | 38 +++++++++++++++++++++++++-------------
>> > Tomasz> 1 file changed, 25 insertions(+), 13 deletions(-)
>> >
>> > Tomasz> diff --git a/Manage.c b/Manage.c
>> > Tomasz> index e2e88b8..7f8eb88 100644
>> > Tomasz> --- a/Manage.c
>> > Tomasz> +++ b/Manage.c
>> > Tomasz> @@ -1125,19 +1125,31 @@ int Manage_remove(struct supertype *tst, int fd, struct mddev_dev *dv,
>> > Tomasz> */
>> > Tomasz> if (rdev == 0)
>> > Tomasz> ret = -1;
>> > Tomasz> - else
>> > Tomasz> - ret = sysfs_unique_holder(devnm, rdev);
>> > Tomasz> - if (ret == 0) {
>> > Tomasz> - pr_err("%s is not a member, cannot remove.\n",
>> > Tomasz> - dv->devname);
>> > Tomasz> - close(lfd);
>> > Tomasz> - return -1;
>> > Tomasz> - }
>> > Tomasz> - if (ret >= 2) {
>> > Tomasz> - pr_err("%s is still in use, cannot remove.\n",
>> > Tomasz> - dv->devname);
>> > Tomasz> - close(lfd);
>> > Tomasz> - return -1;
>> > Tomasz> + else {
>> > Tomasz> + /* The drive has already been set to 'faulty', however monitor might
>> > Tomasz> + * not have had time to process it and the drive might still have
>> > Tomasz> + * an entry in the 'holders' directory. Try a few times to avoid
>> > Tomasz> + * a false error */
>> > Tomasz> + int count = 20;
>> > Tomasz> + do {
>> > Tomasz> + ret = sysfs_unique_holder(devnm, rdev);
>> > Tomasz> + if (ret < 2)
>> > Tomasz> + break;
>> > Tomasz> + usleep(100000);
>> >
>> > Really, you're sleeping 10 seconds without telling the user? That
>> > seems to be a bit obnoxious. Logging something here would be good.
>>
>> Hi,
>>
>> Sorry just back from vacation and just started attacking the mountain of
>> email.
>>
>> I agree with John here, please add some logging message. Also is 10
>> seconds really needed? It seems an awful lot per iteration.
>>
>> Cheers,
>> Jes
Tomasz> Well, actually it's 20 iteration 100ms each so up to 2
Tomasz> seconds. I have never seen it taking more than 3 iterations,
Tomasz> however I don't have a full knowledge how long it can take md
Tomasz> module to release an array. I just added 2 seconds as a
Tomasz> precaution, better wait a bit longer than leave an array in
Tomasz> inconsistent state. Is it fine?
Then maybe instead of the magic number 100000, you put in a define
which says the expected sleep time, or maybe even just a commment? I
can never keep the usleep number units straight in my head anyway.
:-/
^ permalink raw reply
* Re: [PATCH] Remove: container should wait for an array to release a drive
From: Jes Sorensen @ 2016-07-19 14:27 UTC (permalink / raw)
To: John Stoffel
Cc: Tomasz Majchrzak, linux-raid, aleksey.obitotskiy, pawel.baldysiak,
artur.paszkiewicz
In-Reply-To: <22414.10215.472325.518194@quad.stoffel.home>
"John Stoffel" <john@stoffel.org> writes:
>>>>>> "Tomasz" == Tomasz Majchrzak <tomasz.majchrzak@intel.com> writes:
>
> Tomasz> On Mon, Jul 18, 2016 at 04:55:27PM -0400, Jes Sorensen wrote:
>>> "John Stoffel" <john@stoffel.org> writes:
>>> > Really, you're sleeping 10 seconds without telling the user? That
>>> > seems to be a bit obnoxious. Logging something here would be good.
>>>
>>> Hi,
>>>
>>> Sorry just back from vacation and just started attacking the mountain of
>>> email.
>>>
>>> I agree with John here, please add some logging message. Also is 10
>>> seconds really needed? It seems an awful lot per iteration.
>>>
>>> Cheers,
>>> Jes
>
> Tomasz> Well, actually it's 20 iteration 100ms each so up to 2
> Tomasz> seconds. I have never seen it taking more than 3 iterations,
> Tomasz> however I don't have a full knowledge how long it can take md
> Tomasz> module to release an array. I just added 2 seconds as a
> Tomasz> precaution, better wait a bit longer than leave an array in
> Tomasz> inconsistent state. Is it fine?
>
> Then maybe instead of the magic number 100000, you put in a define
> which says the expected sleep time, or maybe even just a commment? I
> can never keep the usleep number units straight in my head anyway.
> :-/
Heh, made the same mistake here - I blame the jetlag and the crazy
weather here in NYC :)
If you can add a comment explaining it, that will satisfy my concerns.
Thanks,
Jes
^ permalink raw reply
* Recovering RAID Volumes from 6 Disks
From: Amit Biswas @ 2016-07-19 16:29 UTC (permalink / raw)
To: linux-raid
Greetings!
Backup server was acting up and the issue was the drives (all of them)
:( Could use some guidance or verdict.
It has a total of 6 drives: sda,b,c,d,e,f. From the superblock info
(attached), there is a raid 1, and a raid 10 volume. Problem is all
the disks are part of both raid volumes (according to superblock).
I am currently booted into an ubuntu live disk shell.
Raid 1 : sde2(0) sdb2(1) sdc2(s) sdd2(s) sdf2(s)
- > 2 devices
- > contains the boot partition. I was able to mount this.
Raid 0: sdc3(2) sdd3(3) sde3(4) sdf3(5)
- > 6 Devices
- > currently inactive
Again, attached are the superblocks, dmseg output.
[Thu Jul 7 15:54:07 2016] Buffer I/O error on dev sdb3, logical block
1, async page read
/dev/sdb2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 356f81a0:69057dd3:8f19639c:26b37454
Name : Vlab-backup:0
Creation Time : Wed May 28 20:44:31 2014
Raid Level : raid1
Raid Devices : 2
Avail Dev Size : 976384 (476.83 MiB 499.91 MB)
Array Size : 488128 (476.77 MiB 499.84 MB)
Used Dev Size : 976256 (476.77 MiB 499.84 MB)
Data Offset : 512 sectors
Super Offset : 8 sectors
State : clean
Device UUID : ab7e7ad4:0bcf824d:3976a240:063016cf
Update Time : Wed Jul 6 23:02:21 2016
Checksum : ead06f71 - correct
Events : 142
Device Role : Active device 1
Array State : AA ('A' == active, '.' == missing)
/dev/sdc2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 356f81a0:69057dd3:8f19639c:26b37454
Name : Vlab-backup:0
Creation Time : Wed May 28 20:44:31 2014
Raid Level : raid1
Raid Devices : 2
Avail Dev Size : 976384 (476.83 MiB 499.91 MB)
Array Size : 488128 (476.77 MiB 499.84 MB)
Used Dev Size : 976256 (476.77 MiB 499.84 MB)
Data Offset : 512 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 86665b3f:664bd08b:65195b2a:bd8345b1
Update Time : Wed Jul 6 23:02:21 2016
Checksum : 5fe6ca8b - correct
Events : 142
Device Role : spare
Array State : AA ('A' == active, '.' == missing)
/dev/sdd2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 356f81a0:69057dd3:8f19639c:26b37454
Name : Vlab-backup:0
Creation Time : Wed May 28 20:44:31 2014
Raid Level : raid1
Raid Devices : 2
Avail Dev Size : 976384 (476.83 MiB 499.91 MB)
Array Size : 488128 (476.77 MiB 499.84 MB)
Used Dev Size : 976256 (476.77 MiB 499.84 MB)
Data Offset : 512 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 5ad0350e:5f2b9fad:c8cac4fa:aa5524ef
Update Time : Wed Jul 6 23:02:21 2016
Checksum : 5ed897aa - correct
Events : 142
Device Role : spare
Array State : AA ('A' == active, '.' == missing)
/dev/sde2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 356f81a0:69057dd3:8f19639c:26b37454
Name : Vlab-backup:0
Creation Time : Wed May 28 20:44:31 2014
Raid Level : raid1
Raid Devices : 2
Avail Dev Size : 976384 (476.83 MiB 499.91 MB)
Array Size : 488128 (476.77 MiB 499.84 MB)
Used Dev Size : 976256 (476.77 MiB 499.84 MB)
Data Offset : 512 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 3e6b7c97:3feb9132:235a5a6e:836e4f35
Update Time : Wed Jul 6 23:02:21 2016
Checksum : 26d29aa2 - correct
Events : 142
Device Role : Active device 0
Array State : AA ('A' == active, '.' == missing)
/dev/sdf2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 356f81a0:69057dd3:8f19639c:26b37454
Name : Vlab-backup:0
Creation Time : Wed May 28 20:44:31 2014
Raid Level : raid1
Raid Devices : 2
Avail Dev Size : 976384 (476.83 MiB 499.91 MB)
Array Size : 488128 (476.77 MiB 499.84 MB)
Used Dev Size : 976256 (476.77 MiB 499.84 MB)
Data Offset : 512 sectors
Super Offset : 8 sectors
State : clean
Device UUID : b0b62b5e:56fbc4c5:735250c4:029d3ee2
Update Time : Wed Jul 6 23:02:21 2016
Checksum : 839a1cfc - correct
Events : 142
Device Role : spare
Array State : AA ('A' == active, '.' == missing)
/dev/sdc3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 3d825ac3:a2e5a336:554fa8d8:542297a3
Name : Vlab-backup:1
Creation Time : Wed May 28 20:44:56 2014
Raid Level : raid10
Raid Devices : 6
Avail Dev Size : 1952280576 (930.92 GiB 999.57 GB)
Array Size : 2928419328 (2792.76 GiB 2998.70 GB)
Used Dev Size : 1952279552 (930.92 GiB 999.57 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 0b23c18b:3202f588:9b7b5636:49f80c36
Update Time : Wed Jul 6 22:29:24 2016
Checksum : 34d1544e - correct
Events : 9149455
Layout : near=2
Chunk Size : 512K
Device Role : Active device 2
Array State : .AAAAA ('A' == active, '.' == missing)
/dev/sdd3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 3d825ac3:a2e5a336:554fa8d8:542297a3
Name : Vlab-backup:1
Creation Time : Wed May 28 20:44:56 2014
Raid Level : raid10
Raid Devices : 6
Avail Dev Size : 1952280576 (930.92 GiB 999.57 GB)
Array Size : 2928419328 (2792.76 GiB 2998.70 GB)
Used Dev Size : 1952279552 (930.92 GiB 999.57 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 2fd7193f:502b657f:708dc952:7a0a7309
Update Time : Wed Jul 6 22:29:24 2016
Checksum : ce735592 - correct
Events : 9149455
Layout : near=2
Chunk Size : 512K
Device Role : Active device 3
Array State : .AAAAA ('A' == active, '.' == missing)
/dev/sde3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 3d825ac3:a2e5a336:554fa8d8:542297a3
Name : Vlab-backup:1
Creation Time : Wed May 28 20:44:56 2014
Raid Level : raid10
Raid Devices : 6
Avail Dev Size : 1952280576 (930.92 GiB 999.57 GB)
Array Size : 2928419328 (2792.76 GiB 2998.70 GB)
Used Dev Size : 1952279552 (930.92 GiB 999.57 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 43d64bab:059237da:63729452:2ce3beda
Update Time : Wed Jul 6 22:29:24 2016
Checksum : 668e7903 - correct
Events : 9149455
Layout : near=2
Chunk Size : 512K
Device Role : Active device 4
Array State : .AAAAA ('A' == active, '.' == missing)
/dev/sdf3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 3d825ac3:a2e5a336:554fa8d8:542297a3
Name : Vlab-backup:1
Creation Time : Wed May 28 20:44:56 2014
Raid Level : raid10
Raid Devices : 6
Avail Dev Size : 1952280576 (930.92 GiB 999.57 GB)
Array Size : 2928419328 (2792.76 GiB 2998.70 GB)
Used Dev Size : 1952279552 (930.92 GiB 999.57 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : c243087b:605b0938:f6b1e13b:11cf83ad
Update Time : Wed Jul 6 22:29:24 2016
Checksum : 50334dad - correct
Events : 9149455
Layout : near=2
Chunk Size : 512K
Device Role : Active device 5
Array State : .AAAAA ('A' == active, '.' == missing)
[Thu Jul 7 15:53:41 2016] ata3: SATA link up 3.0 Gbps (SStatus 123
SControl 300)
[Thu Jul 7 15:53:41 2016] ata3.00: ATA-8: ST1000DM003-1CH162, CC46,
max UDMA/133
[Thu Jul 7 15:53:41 2016] ata3.00: 1953525168 sectors, multi 0: LBA48
NCQ (depth 31/32), AA
[Thu Jul 7 15:53:41 2016] ata3.00: configured for UDMA/133
[Thu Jul 7 15:53:41 2016] scsi 2:0:0:0: Direct-Access ATA
ST1000DM003-1CH1 CC46 PQ: 0 ANSI: 5
[Thu Jul 7 15:53:41 2016] sd 2:0:0:0: [sda] 1953525168 512-byte
logical blocks: (1.00 TB/931 GiB)
[Thu Jul 7 15:53:41 2016] sd 2:0:0:0: [sda] 4096-byte physical blocks
[Thu Jul 7 15:53:41 2016] sd 2:0:0:0: Attached scsi generic sg2 type 0
[Thu Jul 7 15:53:41 2016] sd 2:0:0:0: [sda] Write Protect is off
[Thu Jul 7 15:53:41 2016] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
[Thu Jul 7 15:53:41 2016] sd 2:0:0:0: [sda] Write cache: enabled,
read cache: enabled, doesn't support DPO or FUA
[Thu Jul 7 15:53:41 2016] scsi 3:0:0:0: Direct-Access ATA
ST1000DM003-9YN1 CC9C PQ: 0 ANSI: 5
[Thu Jul 7 15:53:41 2016] sd 3:0:0:0: [sdb] 1953525168 512-byte
logical blocks: (1.00 TB/931 GiB)
[Thu Jul 7 15:53:41 2016] sd 3:0:0:0: [sdb] 4096-byte physical blocks
[Thu Jul 7 15:53:41 2016] sd 3:0:0:0: Attached scsi generic sg3 type 0
[Thu Jul 7 15:53:41 2016] sd 3:0:0:0: [sdb] Write Protect is off
[Thu Jul 7 15:53:41 2016] sd 3:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[Thu Jul 7 15:53:41 2016] sd 3:0:0:0: [sdb] Write cache: enabled,
read cache: enabled, doesn't support DPO or FUA
[Thu Jul 7 15:53:41 2016] scsi 4:0:0:0: Direct-Access ATA
ST1000NM0011 SN03 PQ: 0 ANSI: 5
[Thu Jul 7 15:53:41 2016] sd 4:0:0:0: [sdc] 1953525168 512-byte
logical blocks: (1.00 TB/931 GiB)
[Thu Jul 7 15:53:41 2016] sd 4:0:0:0: Attached scsi generic sg4 type 0
[Thu Jul 7 15:53:41 2016] sd 4:0:0:0: [sdc] Write Protect is off
[Thu Jul 7 15:53:41 2016] sd 4:0:0:0: [sdc] Mode Sense: 00 3a 00 00
[Thu Jul 7 15:53:41 2016] sd 4:0:0:0: [sdc] Write cache: enabled,
read cache: enabled, doesn't support DPO or FUA
[Thu Jul 7 15:53:41 2016] scsi 5:0:0:0: Direct-Access ATA
ST1000NM0011 SN03 PQ: 0 ANSI: 5
[Thu Jul 7 15:53:41 2016] sd 5:0:0:0: [sdd] 1953525168 512-byte
logical blocks: (1.00 TB/931 GiB)
[Thu Jul 7 15:53:41 2016] sd 5:0:0:0: Attached scsi generic sg5 type 0
[Thu Jul 7 15:53:41 2016] sd 5:0:0:0: [sdd] Write Protect is off
[Thu Jul 7 15:53:41 2016] sd 5:0:0:0: [sdd] Mode Sense: 00 3a 00 00
[Thu Jul 7 15:53:41 2016] sd 5:0:0:0: [sdd] Write cache: enabled,
read cache: enabled, doesn't support DPO or FUA
[Thu Jul 7 15:53:41 2016] scsi 6:0:0:0: Direct-Access ATA
ST1000DM003-1CH1 CC46 PQ: 0 ANSI: 5
[Thu Jul 7 15:53:41 2016] sd 6:0:0:0: [sde] 1953525168 512-byte
logical blocks: (1.00 TB/931 GiB)
[Thu Jul 7 15:53:41 2016] sd 6:0:0:0: [sde] 4096-byte physical blocks
[Thu Jul 7 15:53:41 2016] sd 6:0:0:0: Attached scsi generic sg6 type 0
[Thu Jul 7 15:53:41 2016] sd 6:0:0:0: [sde] Write Protect is off
[Thu Jul 7 15:53:41 2016] sd 6:0:0:0: [sde] Mode Sense: 00 3a 00 00
[Thu Jul 7 15:53:41 2016] sd 6:0:0:0: [sde] Write cache: enabled,
read cache: enabled, doesn't support DPO or FUA
[Thu Jul 7 15:53:41 2016] scsi 7:0:0:0: Direct-Access ATA
ST1000DM003-9YN1 CC9C PQ: 0 ANSI: 5
[Thu Jul 7 15:53:41 2016] sd 7:0:0:0: [sdf] 1953525168 512-byte
logical blocks: (1.00 TB/931 GiB)
[Thu Jul 7 15:53:41 2016] sd 7:0:0:0: [sdf] 4096-byte physical blocks
[Thu Jul 7 15:53:41 2016] sd 7:0:0:0: Attached scsi generic sg7 type 0
[Thu Jul 7 15:53:41 2016] sd 7:0:0:0: [sdf] Write Protect is off
[Thu Jul 7 15:53:41 2016] sd 7:0:0:0: [sdf] Mode Sense: 00 3a 00 00
[Thu Jul 7 15:53:41 2016] sd 7:0:0:0: [sdf] Write cache: enabled,
read cache: enabled, doesn't support DPO or FUA
[Thu Jul 7 15:53:41 2016] random: nonblocking pool is initialized
[Thu Jul 7 15:53:41 2016] sdd: sdd1 sdd2 sdd3
[Thu Jul 7 15:53:41 2016] sd 5:0:0:0: [sdd] Attached SCSI disk
[Thu Jul 7 15:53:41 2016] sdc: sdc1 sdc2 sdc3
[Thu Jul 7 15:53:41 2016] sd 4:0:0:0: [sdc] Attached SCSI disk
[Thu Jul 7 15:53:41 2016] sdb: sdb1 sdb2 sdb3
[Thu Jul 7 15:53:41 2016] sd 3:0:0:0: [sdb] Attached SCSI disk
[Thu Jul 7 15:53:41 2016] sdf: sdf1 sdf2 sdf3
[Thu Jul 7 15:53:41 2016] sd 7:0:0:0: [sdf] Attached SCSI disk
[Thu Jul 7 15:53:41 2016] sde: sde1 sde2 sde3
[Thu Jul 7 15:53:41 2016] sd 6:0:0:0: [sde] Attached SCSI disk
[Thu Jul 7 15:54:02 2016] ata3.00: qc timeout (cmd 0x47)
[Thu Jul 7 15:54:02 2016] ata3.00: READ LOG DMA EXT failed, trying unqueued
[Thu Jul 7 15:54:02 2016] ata3: failed to read log page 10h (errno=-5)
[Thu Jul 7 15:54:02 2016] ata3.00: exception Emask 0x1 SAct
0x10000000 SErr 0x0 action 0x6 frozen
[Thu Jul 7 15:54:02 2016] ata3.00: irq_stat 0x40000008
[Thu Jul 7 15:54:02 2016] ata3.00: failed command: READ FPDMA QUEUED
[Thu Jul 7 15:54:02 2016] ata3.00: cmd
60/08:e0:a8:6d:70/00:00:74:00:00/40 tag 28 ncq 4096 in
[Thu Jul 7 15:54:02 2016] res
40/00:e0:a8:6d:70/00:00:74:00:00/40 Emask 0x1 (device error)
[Thu Jul 7 15:54:02 2016] ata3.00: status: { DRDY }
[Thu Jul 7 15:54:02 2016] ata3: hard resetting link
[Thu Jul 7 15:54:05 2016] ata3: SATA link up 3.0 Gbps (SStatus 123
SControl 300)
[Thu Jul 7 15:54:05 2016] ata3.00: configured for UDMA/133
[Thu Jul 7 15:54:05 2016] ata3: EH complete
[Thu Jul 7 15:54:06 2016] ata4.00: exception Emask 0x0 SAct 0x8 SErr
0x0 action 0x0
[Thu Jul 7 15:54:06 2016] ata4.00: irq_stat 0x40000008
[Thu Jul 7 15:54:06 2016] ata4.00: failed command: READ FPDMA QUEUED
[Thu Jul 7 15:54:06 2016] ata4.00: cmd
60/08:18:08:f8:0e/00:00:00:00:00/40 tag 3 ncq 4096 in
[Thu Jul 7 15:54:06 2016] res
41/40:08:09:f8:0e/00:00:00:00:00/00 Emask 0x409 (media error) <F>
[Thu Jul 7 15:54:06 2016] ata4.00: status: { DRDY ERR }
[Thu Jul 7 15:54:06 2016] ata4.00: error: { UNC }
[Thu Jul 7 15:54:06 2016] ata4.00: configured for UDMA/133
[Thu Jul 7 15:54:06 2016] sd 3:0:0:0: [sdb] tag#3 FAILED Result:
hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Thu Jul 7 15:54:06 2016] sd 3:0:0:0: [sdb] tag#3 Sense Key : Medium
Error [current] [descriptor]
[Thu Jul 7 15:54:06 2016] sd 3:0:0:0: [sdb] tag#3 Add. Sense:
Unrecovered read error - auto reallocate failed
[Thu Jul 7 15:54:06 2016] sd 3:0:0:0: [sdb] tag#3 CDB: Read(10) 28 00
00 0e f8 08 00 00 08 00
[Thu Jul 7 15:54:06 2016] blk_update_request: I/O error, dev sdb, sector 981001
[Thu Jul 7 15:54:06 2016] ata4: EH complete
[Thu Jul 7 15:54:07 2016] ata4.00: exception Emask 0x0 SAct 0x400
SErr 0x0 action 0x0
[Thu Jul 7 15:54:07 2016] ata4.00: irq_stat 0x40000008
[Thu Jul 7 15:54:07 2016] ata4.00: failed command: READ FPDMA QUEUED
[Thu Jul 7 15:54:07 2016] ata4.00: cmd
60/08:50:08:f8:0e/00:00:00:00:00/40 tag 10 ncq 4096 in
[Thu Jul 7 15:54:07 2016] res
41/40:08:09:f8:0e/00:00:00:00:00/00 Emask 0x409 (media error) <F>
[Thu Jul 7 15:54:07 2016] ata4.00: status: { DRDY ERR }
[Thu Jul 7 15:54:07 2016] ata4.00: error: { UNC }
[Thu Jul 7 15:54:07 2016] ata4.00: configured for UDMA/133
[Thu Jul 7 15:54:07 2016] sd 3:0:0:0: [sdb] tag#10 FAILED Result:
hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Thu Jul 7 15:54:07 2016] sd 3:0:0:0: [sdb] tag#10 Sense Key : Medium
Error [current] [descriptor]
[Thu Jul 7 15:54:07 2016] sd 3:0:0:0: [sdb] tag#10 Add. Sense:
Unrecovered read error - auto reallocate failed
[Thu Jul 7 15:54:07 2016] sd 3:0:0:0: [sdb] tag#10 CDB: Read(10) 28
00 00 0e f8 08 00 00 08 00
[Thu Jul 7 15:54:07 2016] blk_update_request: I/O error, dev sdb, sector 981001
[Thu Jul 7 15:54:07 2016] Buffer I/O error on dev sdb3, logical block
1, async page read
[Thu Jul 7 15:54:07 2016] ata4: EH complete
[Thu Jul 7 15:54:07 2016] ata4.00: exception Emask 0x0 SAct 0x2 SErr
0x0 action 0x0
[Thu Jul 7 15:54:07 2016] ata4.00: irq_stat 0x40000008
[Thu Jul 7 15:54:07 2016] ata4.00: failed command: READ FPDMA QUEUED
[Thu Jul 7 15:54:07 2016] ata4.00: cmd
60/08:08:08:f8:0e/00:00:00:00:00/40 tag 1 ncq 4096 in
[Thu Jul 7 15:54:07 2016] res
41/40:08:09:f8:0e/00:00:00:00:00/00 Emask 0x409 (media error) <F>
[Thu Jul 7 15:54:07 2016] ata4.00: status: { DRDY ERR }
[Thu Jul 7 15:54:07 2016] ata4.00: error: { UNC }
[Thu Jul 7 15:54:07 2016] ata4.00: configured for UDMA/133
[Thu Jul 7 15:54:07 2016] sd 3:0:0:0: [sdb] tag#1 FAILED Result:
hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Thu Jul 7 15:54:07 2016] sd 3:0:0:0: [sdb] tag#1 Sense Key : Medium
Error [current] [descriptor]
[Thu Jul 7 15:54:07 2016] sd 3:0:0:0: [sdb] tag#1 Add. Sense:
Unrecovered read error - auto reallocate failed
[Thu Jul 7 15:54:07 2016] sd 3:0:0:0: [sdb] tag#1 CDB: Read(10) 28 00
00 0e f8 08 00 00 08 00
[Thu Jul 7 15:54:07 2016] blk_update_request: I/O error, dev sdb, sector 981001
[Thu Jul 7 15:54:07 2016] Buffer I/O error on dev sdb3, logical block
1, async page read
[Thu Jul 7 15:54:07 2016] ata4: EH complete
[Thu Jul 7 15:54:08 2016] ata4.00: exception Emask 0x0 SAct 0x20 SErr
0x0 action 0x0
[Thu Jul 7 15:54:08 2016] ata4.00: irq_stat 0x40000008
[Thu Jul 7 15:54:08 2016] ata4.00: failed command: READ FPDMA QUEUED
[Thu Jul 7 15:54:08 2016] ata4.00: cmd
60/08:28:08:f8:0e/00:00:00:00:00/40 tag 5 ncq 4096 in
[Thu Jul 7 15:54:08 2016] res
41/40:08:09:f8:0e/00:00:00:00:00/00 Emask 0x409 (media error) <F>
[Thu Jul 7 15:54:08 2016] ata4.00: status: { DRDY ERR }
[Thu Jul 7 15:54:08 2016] ata4.00: error: { UNC }
[Thu Jul 7 15:54:08 2016] ata4.00: configured for UDMA/133
[Thu Jul 7 15:54:08 2016] sd 3:0:0:0: [sdb] tag#5 FAILED Result:
hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Thu Jul 7 15:54:08 2016] sd 3:0:0:0: [sdb] tag#5 Sense Key : Medium
Error [current] [descriptor]
[Thu Jul 7 15:54:08 2016] sd 3:0:0:0: [sdb] tag#5 Add. Sense:
Unrecovered read error - auto reallocate failed
[Thu Jul 7 15:54:08 2016] sd 3:0:0:0: [sdb] tag#5 CDB: Read(10) 28 00
00 0e f8 08 00 00 08 00
[Thu Jul 7 15:54:08 2016] blk_update_request: I/O error, dev sdb, sector 981001
[Thu Jul 7 15:54:08 2016] Buffer I/O error on dev sdb3, logical block
1, async page read
[Thu Jul 7 15:54:08 2016] ata4: EH complete
[Thu Jul 7 15:54:09 2016] ata4.00: exception Emask 0x0 SAct 0x4000000
SErr 0x0 action 0x0
[Thu Jul 7 15:54:09 2016] ata4.00: irq_stat 0x40000008
[Thu Jul 7 15:54:09 2016] ata4.00: failed command: READ FPDMA QUEUED
[Thu Jul 7 15:54:09 2016] ata4.00: cmd
60/08:d0:08:f8:0e/00:00:00:00:00/40 tag 26 ncq 4096 in
[Thu Jul 7 15:54:09 2016] res
41/40:08:09:f8:0e/00:00:00:00:00/00 Emask 0x409 (media error) <F>
[Thu Jul 7 15:54:09 2016] ata4.00: status: { DRDY ERR }
[Thu Jul 7 15:54:09 2016] ata4.00: error: { UNC }
[Thu Jul 7 15:54:09 2016] ata4.00: configured for UDMA/133
[Thu Jul 7 15:54:09 2016] sd 3:0:0:0: [sdb] tag#26 FAILED Result:
hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Thu Jul 7 15:54:09 2016] sd 3:0:0:0: [sdb] tag#26 Sense Key : Medium
Error [current] [descriptor]
[Thu Jul 7 15:54:09 2016] sd 3:0:0:0: [sdb] tag#26 Add. Sense:
Unrecovered read error - auto reallocate failed
[Thu Jul 7 15:54:09 2016] sd 3:0:0:0: [sdb] tag#26 CDB: Read(10) 28
00 00 0e f8 08 00 00 08 00
[Thu Jul 7 15:54:09 2016] blk_update_request: I/O error, dev sdb, sector 981001
[Thu Jul 7 15:54:09 2016] Buffer I/O error on dev sdb3, logical block
1, async page read
[Thu Jul 7 15:54:09 2016] ata4: EH complete
[Thu Jul 7 15:54:09 2016] ata4.00: exception Emask 0x0 SAct
0x40000000 SErr 0x0 action 0x0
Much appreciated,
Amit Biswas
^ permalink raw reply
* Re: [PATCH] md: use seconds granularity for error logging
From: Shaohua Li @ 2016-07-19 17:59 UTC (permalink / raw)
To: Arnd Bergmann
Cc: linux-raid, Hannes Reinecke, y2038, NeilBrown, linux-kernel,
Jens Axboe, Guoqing Jiang, Goldwyn Rodrigues
In-Reply-To: <20160617153347.380722-1-arnd@arndb.de>
On Fri, Jun 17, 2016 at 05:33:10PM +0200, Arnd Bergmann wrote:
> The md code stores the exact time of the last error in the
> last_read_error variable using a timespec structure. It only
> ever uses the seconds portion of that though, so we can
> use a scalar for it.
>
> There won't be an overflow in 2038 here, because it already
> used monotonic time and 32-bit is enough for that, but I've
> decided to use time64_t for consistency in the conversion.
>
> Signed-off-by: Arnd Bergmann <arnd@arndb.de>
Applied, thanks!
_______________________________________________
Y2038 mailing list
Y2038@lists.linaro.org
https://lists.linaro.org/mailman/listinfo/y2038
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox