linux-raid.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* RAID6 rebuild stuck
@ 2012-10-01 11:29 Brian Candler
  2012-10-02  1:50 ` Chris Murphy
  2012-10-02  5:56 ` NeilBrown
  0 siblings, 2 replies; 8+ messages in thread
From: Brian Candler @ 2012-10-01 11:29 UTC (permalink / raw)
  To: linux-raid

Platform: ubuntu 10.10, kernel 2.6.35-30-server (yes I know it's old -
update to 12.04 is scheduled)

RAID6 array md1 was running as degraded with two failed disks, sdg and sdk. 
I did a dd write of zeros across both disks, which was successful, so
re-introduced them into the array.  I also enabled scterc, which hadn't been
enabled before.

However some hours into the rebuild, sdg failed again. Now the RAID rebuild
has stuck at the following point:

  cat /proc/mdstat
  ...
  md1 : active raid6 sdk[6] sdg[5](F) sdm[0] sdf[2] sdl[1]
        8790786048 blocks super 1.0 level 6, 16384k chunk, algorithm 2 [5/3] [UUU__]
        [================>....]  recovery = 82.1% (2408361728/2930262016) finish=2976485.1min speed=2K/sec

If I repeat cat /proc/mdstat, even after several minutes, the values do not
increment.

The last ~400 lines of dmesg are below. Key items are:

22440448.503388 - sde added to md0 (this array is now fine)
22440458.737931 - sdg added to md1
22440460.941412 - sdk added to md1
22542378.442924 - sdg started to fail, corrected read errors
22581985.862037 - sdg kicked out, uncorrectable errors
22581986.111914 - "recovery done"!!

So mdstat says that recovery is ongoing, when clearly it isn't. I'm not sure
what I should do next.  Should I just hot-swap sdg, or should I reboot to
clear the stuck /proc/mdstat status?

Thanks,

Brian.

[22383576.611801] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22383576.611817] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22383576.611830] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22383576.611843] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22383576.611856] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22383576.611867] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22383576.612049] sd 6:0:6:0: [sdg] Unhandled sense code
[22383576.612059] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[22383576.612068] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] 
[22383576.612079] Info fld=0x5cde847d
[22383576.612084] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
[22383576.612093] sd 6:0:6:0: [sdg] CDB: Read(10): 28 00 5c de 84 00 00 04 00 00
[22383576.612112] end_request: I/O error, dev sdg, sector 1558086656
[22440448.503388] md: bind<sde>
[22440448.531727] RAID conf printout:
[22440448.531735]  --- level:6 rd:5 wd:4
[22440448.531742]  disk 0, o:1, dev:sda
[22440448.531748]  disk 1, o:1, dev:sdh
[22440448.531752]  disk 2, o:1, dev:sdb
[22440448.531756]  disk 3, o:1, dev:sdc
[22440448.531760]  disk 4, o:1, dev:sde
[22440448.531936] md: recovery of RAID array md0
[22440448.531945] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[22440448.531952] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
[22440448.532030] md: using 128k window, over a total of 2930262016 blocks.
[22440458.737931] md: bind<sdg>
[22440458.787997] RAID conf printout:
[22440458.788006]  --- level:6 rd:5 wd:3
[22440458.788012]  disk 0, o:1, dev:sdm
[22440458.788017]  disk 1, o:1, dev:sdl
[22440458.788022]  disk 2, o:1, dev:sdf
[22440458.788026]  disk 3, o:1, dev:sdg
[22440458.788226] md: recovery of RAID array md1
[22440458.788237] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[22440458.788243] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
[22440458.788301] md: using 128k window, over a total of 2930262016 blocks.
[22440460.941412] md: bind<sdk>
[22510058.255395] md: md0: recovery done.
[22510058.283847] RAID conf printout:
[22510058.283857]  --- level:6 rd:5 wd:5
[22510058.283864]  disk 0, o:1, dev:sda
[22510058.283870]  disk 1, o:1, dev:sdh
[22510058.283874]  disk 2, o:1, dev:sdb
[22510058.283879]  disk 3, o:1, dev:sdc
[22510058.283883]  disk 4, o:1, dev:sde
[22522476.742626] md: md1: recovery done.
[22522476.850179] RAID conf printout:
[22522476.850185]  --- level:6 rd:5 wd:4
[22522476.850188]  disk 0, o:1, dev:sdm
[22522476.850190]  disk 1, o:1, dev:sdl
[22522476.850192]  disk 2, o:1, dev:sdf
[22522476.850193]  disk 3, o:1, dev:sdg
[22522476.858669] RAID conf printout:
[22522476.858679]  --- level:6 rd:5 wd:4
[22522476.858686]  disk 0, o:1, dev:sdm
[22522476.858691]  disk 1, o:1, dev:sdl
[22522476.858695]  disk 2, o:1, dev:sdf
[22522476.858700]  disk 3, o:1, dev:sdg
[22522476.858704]  disk 4, o:1, dev:sdk
[22522476.858909] md: recovery of RAID array md1
[22522476.858917] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[22522476.858924] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
[22522476.858999] md: using 128k window, over a total of 2930262016 blocks.
[22542378.442924] sd 6:0:6:0: [sdg] Unhandled sense code
[22542378.442935] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[22542378.442945] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] 
[22542378.442956] Info fld=0x5cb49cab
[22542378.442960] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
[22542378.442969] sd 6:0:6:0: [sdg] CDB: Read(10): 28 00 5c b4 9c 58 00 00 a8 00
[22542378.442988] end_request: I/O error, dev sdg, sector 1555340376
[22542384.592322] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542384.592336] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542384.592346] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542384.592355] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542384.592364] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542384.592373] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542384.592382] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542384.592391] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542384.592521] sd 6:0:6:0: [sdg] Unhandled sense code
[22542384.592533] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[22542384.592543] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] 
[22542384.592554] Info fld=0x5cb4a619
[22542384.592558] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
[22542384.592568] sd 6:0:6:0: [sdg] CDB: Read(10): 28 00 5c b4 a6 00 00 01 00 00
[22542384.592586] end_request: I/O error, dev sdg, sector 1555342848
[22542385.224348] md/raid:md1: read error corrected (8 sectors at 1555340376 on sdg)
[22542385.224359] md/raid:md1: read error corrected (8 sectors at 1555340384 on sdg)
[22542385.224367] md/raid:md1: read error corrected (8 sectors at 1555340392 on sdg)
[22542385.224374] md/raid:md1: read error corrected (8 sectors at 1555340400 on sdg)
[22542385.224381] md/raid:md1: read error corrected (8 sectors at 1555340408 on sdg)
[22542385.224387] md/raid:md1: read error corrected (8 sectors at 1555340416 on sdg)
[22542385.224394] md/raid:md1: read error corrected (8 sectors at 1555340424 on sdg)
[22542385.224401] md/raid:md1: read error corrected (8 sectors at 1555340432 on sdg)
[22542385.224407] md/raid:md1: read error corrected (8 sectors at 1555340440 on sdg)
[22542385.224414] md/raid:md1: read error corrected (8 sectors at 1555340448 on sdg)
[22542395.724610] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542395.724623] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542395.724633] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542395.724642] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542395.724651] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542395.724660] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542395.724670] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542395.724679] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542395.724690] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542395.724848] sd 6:0:6:0: [sdg] Unhandled sense code
[22542395.724859] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[22542395.724868] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] 
[22542395.724880] Info fld=0x5cb564f6
[22542395.724884] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
[22542395.724894] sd 6:0:6:0: [sdg] CDB: Read(10): 28 00 5c b5 64 40 00 00 c0 00
[22542395.724912] end_request: I/O error, dev sdg, sector 1555391552
[22542397.690128] raid5_end_read_request: 43 callbacks suppressed
[22542397.690134] md/raid:md1: read error corrected (8 sectors at 1555391552 on sdg)
[22542397.690138] md/raid:md1: read error corrected (8 sectors at 1555391560 on sdg)
[22542397.690141] md/raid:md1: read error corrected (8 sectors at 1555391568 on sdg)
[22542397.690143] md/raid:md1: read error corrected (8 sectors at 1555391576 on sdg)
[22542397.690145] md/raid:md1: read error corrected (8 sectors at 1555391584 on sdg)
[22542397.690148] md/raid:md1: read error corrected (8 sectors at 1555391592 on sdg)
[22542397.690150] md/raid:md1: read error corrected (8 sectors at 1555391600 on sdg)
[22542397.690153] md/raid:md1: read error corrected (8 sectors at 1555391608 on sdg)
[22542397.690155] md/raid:md1: read error corrected (8 sectors at 1555391616 on sdg)
[22542397.690158] md/raid:md1: read error corrected (8 sectors at 1555391624 on sdg)
[22542437.870581] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542437.870594] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542437.870604] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542437.870615] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542437.870624] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542437.870633] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542437.870642] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542437.870652] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542437.870661] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542437.870669] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542437.870752] sd 6:0:6:0: [sdg] Unhandled sense code
[22542437.870762] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[22542437.870771] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] 
[22542437.870782] Info fld=0x5cde8e01
[22542437.870787] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
[22542437.870796] sd 6:0:6:0: [sdg] CDB: Read(10): 28 00 5c de 8e 00 00 00 28 00
[22542437.870814] end_request: I/O error, dev sdg, sector 1558089216
[22542443.036738] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542443.036752] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542443.036763] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542443.036772] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542443.036781] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542443.036790] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542443.036799] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542443.036807] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542443.036815] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542443.036953] sd 6:0:6:0: [sdg] Unhandled sense code
[22542443.036964] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[22542443.036973] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] 
[22542443.036985] Info fld=0x5cde8e32
[22542443.036989] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
[22542443.036999] sd 6:0:6:0: [sdg] CDB: Read(10): 28 00 5c de 8e 30 00 00 d0 00
[22542443.037017] end_request: I/O error, dev sdg, sector 1558089264
[22542449.577886] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542449.577899] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542449.577909] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542449.577918] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542449.577927] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542449.577937] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542449.577944] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542449.577953] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542449.577962] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22542449.578102] sd 6:0:6:0: [sdg] Unhandled sense code
[22542449.578113] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[22542449.578122] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] 
[22542449.578133] Info fld=0x5cde8fa0
[22542449.578138] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
[22542449.578147] sd 6:0:6:0: [sdg] CDB: Read(10): 28 00 5c de 8f 00 00 01 00 00
[22542449.578165] end_request: I/O error, dev sdg, sector 1558089472
[22542451.795604] raid5_end_read_request: 14 callbacks suppressed
[22542451.795614] md/raid:md1: read error corrected (8 sectors at 1558089216 on sdg)
[22542451.795621] md/raid:md1: read error corrected (8 sectors at 1558089224 on sdg)
[22542451.795628] md/raid:md1: read error corrected (8 sectors at 1558089232 on sdg)
[22542451.795635] md/raid:md1: read error corrected (8 sectors at 1558089240 on sdg)
[22542451.795641] md/raid:md1: read error corrected (8 sectors at 1558089248 on sdg)
[22542452.292777] md/raid:md1: read error corrected (8 sectors at 1558089264 on sdg)
[22542452.292791] md/raid:md1: read error corrected (8 sectors at 1558089272 on sdg)
[22542452.292794] md/raid:md1: read error corrected (8 sectors at 1558089280 on sdg)
[22542452.292797] md/raid:md1: read error corrected (8 sectors at 1558089288 on sdg)
[22542452.292799] md/raid:md1: read error corrected (8 sectors at 1558089296 on sdg)
[22543017.531581] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22543017.531596] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22543017.531607] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22543017.531616] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22543017.531669] sd 6:0:6:0: [sdg] Unhandled sense code
[22543017.531679] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[22543017.531689] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] 
[22543017.531700] Info fld=0x5f53cf52
[22543017.531704] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
[22543017.531714] sd 6:0:6:0: [sdg] CDB: Read(10): 28 00 5f 53 cf 38 00 00 c8 00
[22543017.531732] end_request: I/O error, dev sdg, sector 1599328056
[22543019.313666] raid5_end_read_request: 53 callbacks suppressed
[22543019.313676] md/raid:md1: read error corrected (8 sectors at 1599328056 on sdg)
[22543019.313690] md/raid:md1: read error corrected (8 sectors at 1599328064 on sdg)
[22543019.313697] md/raid:md1: read error corrected (8 sectors at 1599328072 on sdg)
[22543019.313704] md/raid:md1: read error corrected (8 sectors at 1599328080 on sdg)
[22543019.313710] md/raid:md1: read error corrected (8 sectors at 1599328088 on sdg)
[22543019.313717] md/raid:md1: read error corrected (8 sectors at 1599328096 on sdg)
[22543019.313723] md/raid:md1: read error corrected (8 sectors at 1599328104 on sdg)
[22543019.313730] md/raid:md1: read error corrected (8 sectors at 1599328112 on sdg)
[22543019.313736] md/raid:md1: read error corrected (8 sectors at 1599328120 on sdg)
[22543019.313742] md/raid:md1: read error corrected (8 sectors at 1599328128 on sdg)
[22562645.245880] ADDRCONF(NETDEV_UP): eth5: link is not ready
[22562647.040118] ixgbe: eth5 NIC Link is Up 10 Gbps, Flow Control: RX/TX
[22562647.045381] ADDRCONF(NETDEV_CHANGE): eth5: link becomes ready
[22562657.900033] eth5: no IPv6 routers present
[22575844.106752] sd 6:0:6:0: [sdg] Unhandled sense code
[22575844.106759] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[22575844.106763] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] [descriptor]
[22575844.106767] Descriptor sense data with sense descriptors (in hex):
[22575844.106769]         72 03 11 00 00 00 00 0c 00 0a 80 00 00 00 00 01 
[22575844.106776]         00 fa a3 6f 
[22575844.106779] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
[22575844.106783] sd 6:0:6:0: [sdg] CDB: Read(16): 88 00 00 00 00 01 00 fa a3 00 00 00 01 00 00 00
[22575844.106792] end_request: I/O error, dev sdg, sector 4311393024
[22575845.334051] raid5_end_read_request: 15 callbacks suppressed
[22575845.334056] md/raid:md1: read error corrected (8 sectors at 4311393024 on sdg)
[22575845.334065] md/raid:md1: read error corrected (8 sectors at 4311393032 on sdg)
[22575845.334068] md/raid:md1: read error corrected (8 sectors at 4311393040 on sdg)
[22575845.477470] md/raid:md1: read error corrected (8 sectors at 4311393048 on sdg)
[22575845.477481] md/raid:md1: read error corrected (8 sectors at 4311393056 on sdg)
[22575845.477484] md/raid:md1: read error corrected (8 sectors at 4311393064 on sdg)
[22575845.477487] md/raid:md1: read error corrected (8 sectors at 4311393072 on sdg)
[22575845.477490] md/raid:md1: read error corrected (8 sectors at 4311393080 on sdg)
[22575845.477499] md/raid:md1: read error corrected (8 sectors at 4311393088 on sdg)
[22575845.477503] md/raid:md1: read error corrected (8 sectors at 4311393096 on sdg)
[22575854.105793] sd 6:0:6:0: [sdg] Unhandled sense code
[22575854.105799] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[22575854.105803] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] [descriptor]
[22575854.105808] Descriptor sense data with sense descriptors (in hex):
[22575854.105810]         72 03 11 00 00 00 00 0c 00 0a 80 00 00 00 00 01 
[22575854.105816]         00 fa e1 27 
[22575854.105819] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
[22575854.105823] sd 6:0:6:0: [sdg] CDB: Read(16): 88 00 00 00 00 01 00 fa e1 00 00 00 01 00 00 00
[22575854.105832] end_request: I/O error, dev sdg, sector 4311408896
[22575855.115627] raid5_end_read_request: 22 callbacks suppressed
[22575855.115632] md/raid:md1: read error corrected (8 sectors at 4311408896 on sdg)
[22575855.115643] md/raid:md1: read error corrected (8 sectors at 4311408904 on sdg)
[22575855.590725] md/raid:md1: read error corrected (8 sectors at 4311408912 on sdg)
[22575855.590735] md/raid:md1: read error corrected (8 sectors at 4311408920 on sdg)
[22575855.590738] md/raid:md1: read error corrected (8 sectors at 4311408928 on sdg)
[22575855.590742] md/raid:md1: read error corrected (8 sectors at 4311408936 on sdg)
[22575855.591565] md/raid:md1: read error corrected (8 sectors at 4311408944 on sdg)
[22575855.591572] md/raid:md1: read error corrected (8 sectors at 4311408952 on sdg)
[22575855.591576] md/raid:md1: read error corrected (8 sectors at 4311408960 on sdg)
[22575855.591584] md/raid:md1: read error corrected (8 sectors at 4311408968 on sdg)
[22578251.067614] sd 6:0:6:0: [sdg] Unhandled sense code
[22578251.067623] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[22578251.067632] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] [descriptor]
[22578251.067643] Descriptor sense data with sense descriptors (in hex):
[22578251.067648]         72 03 11 00 00 00 00 0c 00 0a 80 00 00 00 00 01 
[22578251.067667]         0c 88 26 c5 
[22578251.067676] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
[22578251.067685] sd 6:0:6:0: [sdg] CDB: Read(16): 88 00 00 00 00 01 0c 88 26 00 00 00 01 00 00 00
[22578251.067708] end_request: I/O error, dev sdg, sector 4505216512
[22578254.167280] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22578254.167289] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22578254.167292] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22578254.167296] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22578254.167301] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22578254.167305] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22578254.167314] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22578254.167320] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22578254.167443] sd 6:0:6:0: [sdg] Unhandled sense code
[22578254.167450] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[22578254.167455] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] [descriptor]
[22578254.167459] Descriptor sense data with sense descriptors (in hex):
[22578254.167461]         72 03 11 00 00 00 00 0c 00 0a 80 00 00 00 00 01 
[22578254.167468]         0c 88 2e 30 
[22578254.167471] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
[22578254.167476] sd 6:0:6:0: [sdg] CDB: Read(16): 88 00 00 00 00 01 0c 88 2e 00 00 00 01 00 00 00
[22578254.167485] end_request: I/O error, dev sdg, sector 4505218560
[22578254.984897] raid5_end_read_request: 22 callbacks suppressed
[22578254.984902] md/raid:md1: read error corrected (8 sectors at 4505216512 on sdg)
[22578254.984908] md/raid:md1: read error corrected (8 sectors at 4505216520 on sdg)
[22578254.984911] md/raid:md1: read error corrected (8 sectors at 4505216528 on sdg)
[22578254.984914] md/raid:md1: read error corrected (8 sectors at 4505216536 on sdg)
[22578254.984916] md/raid:md1: read error corrected (8 sectors at 4505216544 on sdg)
[22578254.984918] md/raid:md1: read error corrected (8 sectors at 4505216552 on sdg)
[22578254.984921] md/raid:md1: read error corrected (8 sectors at 4505216560 on sdg)
[22578254.984923] md/raid:md1: read error corrected (8 sectors at 4505216568 on sdg)
[22578254.984926] md/raid:md1: read error corrected (8 sectors at 4505216576 on sdg)
[22578254.984928] md/raid:md1: read error corrected (8 sectors at 4505216584 on sdg)
[22578260.319089] raid5_end_read_request: 22 callbacks suppressed
[22578260.319095] md/raid:md1: read error corrected (8 sectors at 4505218560 on sdg)
[22578260.319102] md/raid:md1: read error corrected (8 sectors at 4505218568 on sdg)
[22578260.319105] md/raid:md1: read error corrected (8 sectors at 4505218576 on sdg)
[22578260.319108] md/raid:md1: read error corrected (8 sectors at 4505218584 on sdg)
[22578260.319110] md/raid:md1: read error corrected (8 sectors at 4505218592 on sdg)
[22578260.319113] md/raid:md1: read error corrected (8 sectors at 4505218600 on sdg)
[22578260.319115] md/raid:md1: read error corrected (8 sectors at 4505218608 on sdg)
[22578260.319117] md/raid:md1: read error corrected (8 sectors at 4505218616 on sdg)
[22578260.319119] md/raid:md1: read error corrected (8 sectors at 4505218624 on sdg)
[22578260.319122] md/raid:md1: read error corrected (8 sectors at 4505218632 on sdg)
[22581681.874078] sd 6:0:6:0: [sdg] Unhandled sense code
[22581681.874086] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[22581681.874090] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] [descriptor]
[22581681.874095] Descriptor sense data with sense descriptors (in hex):
[22581681.874098]         72 03 11 00 00 00 00 0c 00 0a 80 00 00 00 00 01 
[22581681.874104]         1d fa 42 03 
[22581681.874107] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
[22581681.874112] sd 6:0:6:0: [sdg] CDB: Read(16): 88 00 00 00 00 01 1d fa 42 00 00 00 01 00 00 00
[22581681.874121] end_request: I/O error, dev sdg, sector 4797907456
[22581686.823444] raid5_end_read_request: 22 callbacks suppressed
[22581686.823451] md/raid:md1: read error corrected (8 sectors at 4797907456 on sdg)
[22581687.072047] md/raid:md1: read error corrected (8 sectors at 4797907464 on sdg)
[22581687.072059] md/raid:md1: read error corrected (8 sectors at 4797907472 on sdg)
[22581687.072062] md/raid:md1: read error corrected (8 sectors at 4797907480 on sdg)
[22581687.072065] md/raid:md1: read error corrected (8 sectors at 4797907488 on sdg)
[22581687.072973] md/raid:md1: read error corrected (8 sectors at 4797907496 on sdg)
[22581687.072980] md/raid:md1: read error corrected (8 sectors at 4797907504 on sdg)
[22581687.072982] md/raid:md1: read error corrected (8 sectors at 4797907512 on sdg)
[22581687.072990] md/raid:md1: read error corrected (8 sectors at 4797907520 on sdg)
[22581687.072994] md/raid:md1: read error corrected (8 sectors at 4797907528 on sdg)
[22581698.030939] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22581698.030947] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22581698.030951] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22581698.030954] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22581698.030958] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22581698.030995] sd 6:0:6:0: [sdg] Unhandled sense code
[22581698.031000] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[22581698.031005] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] [descriptor]
[22581698.031010] Descriptor sense data with sense descriptors (in hex):
[22581698.031012]         72 03 11 00 00 00 00 0c 00 0a 80 00 00 00 00 01 
[22581698.031018]         1d fa b7 00 
[22581698.031021] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
[22581698.031026] sd 6:0:6:0: [sdg] CDB: Read(16): 88 00 00 00 00 01 1d fa b7 00 00 00 00 c8 00 00
[22581698.031035] end_request: I/O error, dev sdg, sector 4797937408
[22581701.434353] raid5_end_read_request: 22 callbacks suppressed
[22581701.434357] md/raid:md1: read error corrected (8 sectors at 4797937408 on sdg)
[22581701.434367] md/raid:md1: read error corrected (8 sectors at 4797937416 on sdg)
[22581701.435141] md/raid:md1: read error corrected (8 sectors at 4797937424 on sdg)
[22581701.435148] md/raid:md1: read error corrected (8 sectors at 4797937432 on sdg)
[22581701.435151] md/raid:md1: read error corrected (8 sectors at 4797937440 on sdg)
[22581701.435160] md/raid:md1: read error corrected (8 sectors at 4797937448 on sdg)
[22581701.435164] md/raid:md1: read error corrected (8 sectors at 4797937456 on sdg)
[22581701.435172] md/raid:md1: read error corrected (8 sectors at 4797937464 on sdg)
[22581701.435175] md/raid:md1: read error corrected (8 sectors at 4797937472 on sdg)
[22581701.435183] md/raid:md1: read error corrected (8 sectors at 4797937480 on sdg)
[22581981.870472] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22581981.870482] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22581981.870487] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22581981.870493] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22581981.870498] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22581981.870605] sd 6:0:6:0: [sdg] Unhandled sense code
[22581981.870613] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[22581981.870618] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] [descriptor]
[22581981.870623] Descriptor sense data with sense descriptors (in hex):
[22581981.870625]         72 03 11 00 00 00 00 0c 00 0a 80 00 00 00 00 01 
[22581981.870632]         1f 19 45 58 
[22581981.870635] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
[22581981.870639] sd 6:0:6:0: [sdg] CDB: Read(16): 88 00 00 00 00 01 1f 19 45 00 00 00 01 00 00 00
[22581981.870648] end_request: I/O error, dev sdg, sector 4816717056
[22581985.861765] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22581985.861773] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22581985.861778] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22581985.861782] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22581985.861786] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22581985.861791] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22581985.861799] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[22581985.861911] sd 6:0:6:0: [sdg] Unhandled sense code
[22581985.861919] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[22581985.861923] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] [descriptor]
[22581985.861929] Descriptor sense data with sense descriptors (in hex):
[22581985.861930]         72 03 11 00 00 00 00 0c 00 0a 80 00 00 00 00 01 
[22581985.861937]         1f 19 45 59 
[22581985.861939] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
[22581985.861944] sd 6:0:6:0: [sdg] CDB: Read(16): 88 00 00 00 00 01 1f 19 45 00 00 00 01 00 00 00
[22581985.861953] end_request: I/O error, dev sdg, sector 4816717056
[22581985.862031] raid5_end_read_request: 15 callbacks suppressed
[22581985.862033] md/raid:md1: read error NOT corrected!! (sector 4816717056 on sdg).
[22581985.862037] md/raid:md1: Disk failure on sdg, disabling device.
[22581985.862038] <1>md/raid:md1: Operation continuing on 3 devices.
[22581985.862185] md/raid:md1: read error not correctable (sector 4816717064 on sdg).
[22581985.862188] md/raid:md1: read error not correctable (sector 4816717072 on sdg).
[22581985.862191] md/raid:md1: read error not correctable (sector 4816717080 on sdg).
[22581985.862194] md/raid:md1: read error not correctable (sector 4816717088 on sdg).
[22581985.862196] md/raid:md1: read error not correctable (sector 4816717096 on sdg).
[22581985.862198] md/raid:md1: read error not correctable (sector 4816717104 on sdg).
[22581985.862200] md/raid:md1: read error not correctable (sector 4816717112 on sdg).
[22581985.862202] md/raid:md1: read error not correctable (sector 4816717120 on sdg).
[22581985.862205] md/raid:md1: read error not correctable (sector 4816717128 on sdg).
[22581986.111914] md: md1: recovery done.

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: RAID6 rebuild stuck
  2012-10-01 11:29 RAID6 rebuild stuck Brian Candler
@ 2012-10-02  1:50 ` Chris Murphy
  2012-10-02  5:56 ` NeilBrown
  1 sibling, 0 replies; 8+ messages in thread
From: Chris Murphy @ 2012-10-02  1:50 UTC (permalink / raw)
  To: Linux RAID


On Oct 1, 2012, at 5:29 AM, Brian Candler wrote:

> So mdstat says that recovery is ongoing, when clearly it isn't. I'm not sure
> what I should do next.  Should I just hot-swap sdg, or should I reboot to
> clear the stuck /proc/mdstat status?

It might be superfluous, but looks like you should be able to to stop the resync/repair by using:

echo idle  > md/sync_action

In any case you need to fail the drive, if it hasn't already been rejected again, and remove the drive from the array.

mdadm /dev/md1 --fail /dev/sdg
mdadm /dev/md1 --remove /dev/sdg

Replace the drive - i.e. physically remove the bad one, and replace it with a good one. You'll need to find out what its designation is to add it:

mdadm /dev/md1 --add /dev/sdX

You should start getting a resync right away.

Chris Murphy


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: RAID6 rebuild stuck
  2012-10-01 11:29 RAID6 rebuild stuck Brian Candler
  2012-10-02  1:50 ` Chris Murphy
@ 2012-10-02  5:56 ` NeilBrown
  2012-10-02 15:56   ` Brian Candler
  1 sibling, 1 reply; 8+ messages in thread
From: NeilBrown @ 2012-10-02  5:56 UTC (permalink / raw)
  To: Brian Candler; +Cc: linux-raid

[-- Attachment #1: Type: text/plain, Size: 32670 bytes --]

On Mon, 1 Oct 2012 12:29:46 +0100 Brian Candler <B.Candler@pobox.com> wrote:

> Platform: ubuntu 10.10, kernel 2.6.35-30-server (yes I know it's old -
> update to 12.04 is scheduled)
> 
> RAID6 array md1 was running as degraded with two failed disks, sdg and sdk. 
> I did a dd write of zeros across both disks, which was successful, so
> re-introduced them into the array.  I also enabled scterc, which hadn't been
> enabled before.

Lesson: always do a read test as well as a write test!

> 
> However some hours into the rebuild, sdg failed again. Now the RAID rebuild
> has stuck at the following point:
> 
>   cat /proc/mdstat
>   ...
>   md1 : active raid6 sdk[6] sdg[5](F) sdm[0] sdf[2] sdl[1]
>         8790786048 blocks super 1.0 level 6, 16384k chunk, algorithm 2 [5/3] [UUU__]
>         [================>....]  recovery = 82.1% (2408361728/2930262016) finish=2976485.1min speed=2K/sec
> 
> If I repeat cat /proc/mdstat, even after several minutes, the values do not
> increment.
> 
> The last ~400 lines of dmesg are below. Key items are:
> 
> 22440448.503388 - sde added to md0 (this array is now fine)
> 22440458.737931 - sdg added to md1
> 22440460.941412 - sdk added to md1
> 22542378.442924 - sdg started to fail, corrected read errors
> 22581985.862037 - sdg kicked out, uncorrectable errors
> 22581986.111914 - "recovery done"!!
> 
> So mdstat says that recovery is ongoing, when clearly it isn't. I'm not sure
> what I should do next.  Should I just hot-swap sdg, or should I reboot to
> clear the stuck /proc/mdstat status?

Looks like a bug. md_do_sync() is still waiting for all the submitted sync
requests to complete.  This suggests some sort of accounting problem, but I
cannot easily see it.

A reboot is likely to be the only fix.

NeilBrown




> 
> Thanks,
> 
> Brian.
> 
> [22383576.611801] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22383576.611817] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22383576.611830] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22383576.611843] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22383576.611856] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22383576.611867] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22383576.612049] sd 6:0:6:0: [sdg] Unhandled sense code
> [22383576.612059] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [22383576.612068] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] 
> [22383576.612079] Info fld=0x5cde847d
> [22383576.612084] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
> [22383576.612093] sd 6:0:6:0: [sdg] CDB: Read(10): 28 00 5c de 84 00 00 04 00 00
> [22383576.612112] end_request: I/O error, dev sdg, sector 1558086656
> [22440448.503388] md: bind<sde>
> [22440448.531727] RAID conf printout:
> [22440448.531735]  --- level:6 rd:5 wd:4
> [22440448.531742]  disk 0, o:1, dev:sda
> [22440448.531748]  disk 1, o:1, dev:sdh
> [22440448.531752]  disk 2, o:1, dev:sdb
> [22440448.531756]  disk 3, o:1, dev:sdc
> [22440448.531760]  disk 4, o:1, dev:sde
> [22440448.531936] md: recovery of RAID array md0
> [22440448.531945] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
> [22440448.531952] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
> [22440448.532030] md: using 128k window, over a total of 2930262016 blocks.
> [22440458.737931] md: bind<sdg>
> [22440458.787997] RAID conf printout:
> [22440458.788006]  --- level:6 rd:5 wd:3
> [22440458.788012]  disk 0, o:1, dev:sdm
> [22440458.788017]  disk 1, o:1, dev:sdl
> [22440458.788022]  disk 2, o:1, dev:sdf
> [22440458.788026]  disk 3, o:1, dev:sdg
> [22440458.788226] md: recovery of RAID array md1
> [22440458.788237] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
> [22440458.788243] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
> [22440458.788301] md: using 128k window, over a total of 2930262016 blocks.
> [22440460.941412] md: bind<sdk>
> [22510058.255395] md: md0: recovery done.
> [22510058.283847] RAID conf printout:
> [22510058.283857]  --- level:6 rd:5 wd:5
> [22510058.283864]  disk 0, o:1, dev:sda
> [22510058.283870]  disk 1, o:1, dev:sdh
> [22510058.283874]  disk 2, o:1, dev:sdb
> [22510058.283879]  disk 3, o:1, dev:sdc
> [22510058.283883]  disk 4, o:1, dev:sde
> [22522476.742626] md: md1: recovery done.
> [22522476.850179] RAID conf printout:
> [22522476.850185]  --- level:6 rd:5 wd:4
> [22522476.850188]  disk 0, o:1, dev:sdm
> [22522476.850190]  disk 1, o:1, dev:sdl
> [22522476.850192]  disk 2, o:1, dev:sdf
> [22522476.850193]  disk 3, o:1, dev:sdg
> [22522476.858669] RAID conf printout:
> [22522476.858679]  --- level:6 rd:5 wd:4
> [22522476.858686]  disk 0, o:1, dev:sdm
> [22522476.858691]  disk 1, o:1, dev:sdl
> [22522476.858695]  disk 2, o:1, dev:sdf
> [22522476.858700]  disk 3, o:1, dev:sdg
> [22522476.858704]  disk 4, o:1, dev:sdk
> [22522476.858909] md: recovery of RAID array md1
> [22522476.858917] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
> [22522476.858924] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
> [22522476.858999] md: using 128k window, over a total of 2930262016 blocks.
> [22542378.442924] sd 6:0:6:0: [sdg] Unhandled sense code
> [22542378.442935] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [22542378.442945] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] 
> [22542378.442956] Info fld=0x5cb49cab
> [22542378.442960] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
> [22542378.442969] sd 6:0:6:0: [sdg] CDB: Read(10): 28 00 5c b4 9c 58 00 00 a8 00
> [22542378.442988] end_request: I/O error, dev sdg, sector 1555340376
> [22542384.592322] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542384.592336] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542384.592346] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542384.592355] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542384.592364] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542384.592373] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542384.592382] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542384.592391] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542384.592521] sd 6:0:6:0: [sdg] Unhandled sense code
> [22542384.592533] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [22542384.592543] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] 
> [22542384.592554] Info fld=0x5cb4a619
> [22542384.592558] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
> [22542384.592568] sd 6:0:6:0: [sdg] CDB: Read(10): 28 00 5c b4 a6 00 00 01 00 00
> [22542384.592586] end_request: I/O error, dev sdg, sector 1555342848
> [22542385.224348] md/raid:md1: read error corrected (8 sectors at 1555340376 on sdg)
> [22542385.224359] md/raid:md1: read error corrected (8 sectors at 1555340384 on sdg)
> [22542385.224367] md/raid:md1: read error corrected (8 sectors at 1555340392 on sdg)
> [22542385.224374] md/raid:md1: read error corrected (8 sectors at 1555340400 on sdg)
> [22542385.224381] md/raid:md1: read error corrected (8 sectors at 1555340408 on sdg)
> [22542385.224387] md/raid:md1: read error corrected (8 sectors at 1555340416 on sdg)
> [22542385.224394] md/raid:md1: read error corrected (8 sectors at 1555340424 on sdg)
> [22542385.224401] md/raid:md1: read error corrected (8 sectors at 1555340432 on sdg)
> [22542385.224407] md/raid:md1: read error corrected (8 sectors at 1555340440 on sdg)
> [22542385.224414] md/raid:md1: read error corrected (8 sectors at 1555340448 on sdg)
> [22542395.724610] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542395.724623] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542395.724633] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542395.724642] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542395.724651] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542395.724660] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542395.724670] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542395.724679] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542395.724690] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542395.724848] sd 6:0:6:0: [sdg] Unhandled sense code
> [22542395.724859] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [22542395.724868] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] 
> [22542395.724880] Info fld=0x5cb564f6
> [22542395.724884] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
> [22542395.724894] sd 6:0:6:0: [sdg] CDB: Read(10): 28 00 5c b5 64 40 00 00 c0 00
> [22542395.724912] end_request: I/O error, dev sdg, sector 1555391552
> [22542397.690128] raid5_end_read_request: 43 callbacks suppressed
> [22542397.690134] md/raid:md1: read error corrected (8 sectors at 1555391552 on sdg)
> [22542397.690138] md/raid:md1: read error corrected (8 sectors at 1555391560 on sdg)
> [22542397.690141] md/raid:md1: read error corrected (8 sectors at 1555391568 on sdg)
> [22542397.690143] md/raid:md1: read error corrected (8 sectors at 1555391576 on sdg)
> [22542397.690145] md/raid:md1: read error corrected (8 sectors at 1555391584 on sdg)
> [22542397.690148] md/raid:md1: read error corrected (8 sectors at 1555391592 on sdg)
> [22542397.690150] md/raid:md1: read error corrected (8 sectors at 1555391600 on sdg)
> [22542397.690153] md/raid:md1: read error corrected (8 sectors at 1555391608 on sdg)
> [22542397.690155] md/raid:md1: read error corrected (8 sectors at 1555391616 on sdg)
> [22542397.690158] md/raid:md1: read error corrected (8 sectors at 1555391624 on sdg)
> [22542437.870581] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542437.870594] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542437.870604] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542437.870615] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542437.870624] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542437.870633] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542437.870642] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542437.870652] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542437.870661] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542437.870669] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542437.870752] sd 6:0:6:0: [sdg] Unhandled sense code
> [22542437.870762] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [22542437.870771] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] 
> [22542437.870782] Info fld=0x5cde8e01
> [22542437.870787] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
> [22542437.870796] sd 6:0:6:0: [sdg] CDB: Read(10): 28 00 5c de 8e 00 00 00 28 00
> [22542437.870814] end_request: I/O error, dev sdg, sector 1558089216
> [22542443.036738] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542443.036752] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542443.036763] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542443.036772] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542443.036781] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542443.036790] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542443.036799] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542443.036807] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542443.036815] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542443.036953] sd 6:0:6:0: [sdg] Unhandled sense code
> [22542443.036964] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [22542443.036973] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] 
> [22542443.036985] Info fld=0x5cde8e32
> [22542443.036989] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
> [22542443.036999] sd 6:0:6:0: [sdg] CDB: Read(10): 28 00 5c de 8e 30 00 00 d0 00
> [22542443.037017] end_request: I/O error, dev sdg, sector 1558089264
> [22542449.577886] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542449.577899] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542449.577909] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542449.577918] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542449.577927] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542449.577937] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542449.577944] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542449.577953] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542449.577962] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22542449.578102] sd 6:0:6:0: [sdg] Unhandled sense code
> [22542449.578113] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [22542449.578122] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] 
> [22542449.578133] Info fld=0x5cde8fa0
> [22542449.578138] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
> [22542449.578147] sd 6:0:6:0: [sdg] CDB: Read(10): 28 00 5c de 8f 00 00 01 00 00
> [22542449.578165] end_request: I/O error, dev sdg, sector 1558089472
> [22542451.795604] raid5_end_read_request: 14 callbacks suppressed
> [22542451.795614] md/raid:md1: read error corrected (8 sectors at 1558089216 on sdg)
> [22542451.795621] md/raid:md1: read error corrected (8 sectors at 1558089224 on sdg)
> [22542451.795628] md/raid:md1: read error corrected (8 sectors at 1558089232 on sdg)
> [22542451.795635] md/raid:md1: read error corrected (8 sectors at 1558089240 on sdg)
> [22542451.795641] md/raid:md1: read error corrected (8 sectors at 1558089248 on sdg)
> [22542452.292777] md/raid:md1: read error corrected (8 sectors at 1558089264 on sdg)
> [22542452.292791] md/raid:md1: read error corrected (8 sectors at 1558089272 on sdg)
> [22542452.292794] md/raid:md1: read error corrected (8 sectors at 1558089280 on sdg)
> [22542452.292797] md/raid:md1: read error corrected (8 sectors at 1558089288 on sdg)
> [22542452.292799] md/raid:md1: read error corrected (8 sectors at 1558089296 on sdg)
> [22543017.531581] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22543017.531596] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22543017.531607] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22543017.531616] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22543017.531669] sd 6:0:6:0: [sdg] Unhandled sense code
> [22543017.531679] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [22543017.531689] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] 
> [22543017.531700] Info fld=0x5f53cf52
> [22543017.531704] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
> [22543017.531714] sd 6:0:6:0: [sdg] CDB: Read(10): 28 00 5f 53 cf 38 00 00 c8 00
> [22543017.531732] end_request: I/O error, dev sdg, sector 1599328056
> [22543019.313666] raid5_end_read_request: 53 callbacks suppressed
> [22543019.313676] md/raid:md1: read error corrected (8 sectors at 1599328056 on sdg)
> [22543019.313690] md/raid:md1: read error corrected (8 sectors at 1599328064 on sdg)
> [22543019.313697] md/raid:md1: read error corrected (8 sectors at 1599328072 on sdg)
> [22543019.313704] md/raid:md1: read error corrected (8 sectors at 1599328080 on sdg)
> [22543019.313710] md/raid:md1: read error corrected (8 sectors at 1599328088 on sdg)
> [22543019.313717] md/raid:md1: read error corrected (8 sectors at 1599328096 on sdg)
> [22543019.313723] md/raid:md1: read error corrected (8 sectors at 1599328104 on sdg)
> [22543019.313730] md/raid:md1: read error corrected (8 sectors at 1599328112 on sdg)
> [22543019.313736] md/raid:md1: read error corrected (8 sectors at 1599328120 on sdg)
> [22543019.313742] md/raid:md1: read error corrected (8 sectors at 1599328128 on sdg)
> [22562645.245880] ADDRCONF(NETDEV_UP): eth5: link is not ready
> [22562647.040118] ixgbe: eth5 NIC Link is Up 10 Gbps, Flow Control: RX/TX
> [22562647.045381] ADDRCONF(NETDEV_CHANGE): eth5: link becomes ready
> [22562657.900033] eth5: no IPv6 routers present
> [22575844.106752] sd 6:0:6:0: [sdg] Unhandled sense code
> [22575844.106759] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [22575844.106763] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] [descriptor]
> [22575844.106767] Descriptor sense data with sense descriptors (in hex):
> [22575844.106769]         72 03 11 00 00 00 00 0c 00 0a 80 00 00 00 00 01 
> [22575844.106776]         00 fa a3 6f 
> [22575844.106779] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
> [22575844.106783] sd 6:0:6:0: [sdg] CDB: Read(16): 88 00 00 00 00 01 00 fa a3 00 00 00 01 00 00 00
> [22575844.106792] end_request: I/O error, dev sdg, sector 4311393024
> [22575845.334051] raid5_end_read_request: 15 callbacks suppressed
> [22575845.334056] md/raid:md1: read error corrected (8 sectors at 4311393024 on sdg)
> [22575845.334065] md/raid:md1: read error corrected (8 sectors at 4311393032 on sdg)
> [22575845.334068] md/raid:md1: read error corrected (8 sectors at 4311393040 on sdg)
> [22575845.477470] md/raid:md1: read error corrected (8 sectors at 4311393048 on sdg)
> [22575845.477481] md/raid:md1: read error corrected (8 sectors at 4311393056 on sdg)
> [22575845.477484] md/raid:md1: read error corrected (8 sectors at 4311393064 on sdg)
> [22575845.477487] md/raid:md1: read error corrected (8 sectors at 4311393072 on sdg)
> [22575845.477490] md/raid:md1: read error corrected (8 sectors at 4311393080 on sdg)
> [22575845.477499] md/raid:md1: read error corrected (8 sectors at 4311393088 on sdg)
> [22575845.477503] md/raid:md1: read error corrected (8 sectors at 4311393096 on sdg)
> [22575854.105793] sd 6:0:6:0: [sdg] Unhandled sense code
> [22575854.105799] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [22575854.105803] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] [descriptor]
> [22575854.105808] Descriptor sense data with sense descriptors (in hex):
> [22575854.105810]         72 03 11 00 00 00 00 0c 00 0a 80 00 00 00 00 01 
> [22575854.105816]         00 fa e1 27 
> [22575854.105819] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
> [22575854.105823] sd 6:0:6:0: [sdg] CDB: Read(16): 88 00 00 00 00 01 00 fa e1 00 00 00 01 00 00 00
> [22575854.105832] end_request: I/O error, dev sdg, sector 4311408896
> [22575855.115627] raid5_end_read_request: 22 callbacks suppressed
> [22575855.115632] md/raid:md1: read error corrected (8 sectors at 4311408896 on sdg)
> [22575855.115643] md/raid:md1: read error corrected (8 sectors at 4311408904 on sdg)
> [22575855.590725] md/raid:md1: read error corrected (8 sectors at 4311408912 on sdg)
> [22575855.590735] md/raid:md1: read error corrected (8 sectors at 4311408920 on sdg)
> [22575855.590738] md/raid:md1: read error corrected (8 sectors at 4311408928 on sdg)
> [22575855.590742] md/raid:md1: read error corrected (8 sectors at 4311408936 on sdg)
> [22575855.591565] md/raid:md1: read error corrected (8 sectors at 4311408944 on sdg)
> [22575855.591572] md/raid:md1: read error corrected (8 sectors at 4311408952 on sdg)
> [22575855.591576] md/raid:md1: read error corrected (8 sectors at 4311408960 on sdg)
> [22575855.591584] md/raid:md1: read error corrected (8 sectors at 4311408968 on sdg)
> [22578251.067614] sd 6:0:6:0: [sdg] Unhandled sense code
> [22578251.067623] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [22578251.067632] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] [descriptor]
> [22578251.067643] Descriptor sense data with sense descriptors (in hex):
> [22578251.067648]         72 03 11 00 00 00 00 0c 00 0a 80 00 00 00 00 01 
> [22578251.067667]         0c 88 26 c5 
> [22578251.067676] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
> [22578251.067685] sd 6:0:6:0: [sdg] CDB: Read(16): 88 00 00 00 00 01 0c 88 26 00 00 00 01 00 00 00
> [22578251.067708] end_request: I/O error, dev sdg, sector 4505216512
> [22578254.167280] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22578254.167289] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22578254.167292] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22578254.167296] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22578254.167301] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22578254.167305] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22578254.167314] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22578254.167320] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22578254.167443] sd 6:0:6:0: [sdg] Unhandled sense code
> [22578254.167450] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [22578254.167455] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] [descriptor]
> [22578254.167459] Descriptor sense data with sense descriptors (in hex):
> [22578254.167461]         72 03 11 00 00 00 00 0c 00 0a 80 00 00 00 00 01 
> [22578254.167468]         0c 88 2e 30 
> [22578254.167471] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
> [22578254.167476] sd 6:0:6:0: [sdg] CDB: Read(16): 88 00 00 00 00 01 0c 88 2e 00 00 00 01 00 00 00
> [22578254.167485] end_request: I/O error, dev sdg, sector 4505218560
> [22578254.984897] raid5_end_read_request: 22 callbacks suppressed
> [22578254.984902] md/raid:md1: read error corrected (8 sectors at 4505216512 on sdg)
> [22578254.984908] md/raid:md1: read error corrected (8 sectors at 4505216520 on sdg)
> [22578254.984911] md/raid:md1: read error corrected (8 sectors at 4505216528 on sdg)
> [22578254.984914] md/raid:md1: read error corrected (8 sectors at 4505216536 on sdg)
> [22578254.984916] md/raid:md1: read error corrected (8 sectors at 4505216544 on sdg)
> [22578254.984918] md/raid:md1: read error corrected (8 sectors at 4505216552 on sdg)
> [22578254.984921] md/raid:md1: read error corrected (8 sectors at 4505216560 on sdg)
> [22578254.984923] md/raid:md1: read error corrected (8 sectors at 4505216568 on sdg)
> [22578254.984926] md/raid:md1: read error corrected (8 sectors at 4505216576 on sdg)
> [22578254.984928] md/raid:md1: read error corrected (8 sectors at 4505216584 on sdg)
> [22578260.319089] raid5_end_read_request: 22 callbacks suppressed
> [22578260.319095] md/raid:md1: read error corrected (8 sectors at 4505218560 on sdg)
> [22578260.319102] md/raid:md1: read error corrected (8 sectors at 4505218568 on sdg)
> [22578260.319105] md/raid:md1: read error corrected (8 sectors at 4505218576 on sdg)
> [22578260.319108] md/raid:md1: read error corrected (8 sectors at 4505218584 on sdg)
> [22578260.319110] md/raid:md1: read error corrected (8 sectors at 4505218592 on sdg)
> [22578260.319113] md/raid:md1: read error corrected (8 sectors at 4505218600 on sdg)
> [22578260.319115] md/raid:md1: read error corrected (8 sectors at 4505218608 on sdg)
> [22578260.319117] md/raid:md1: read error corrected (8 sectors at 4505218616 on sdg)
> [22578260.319119] md/raid:md1: read error corrected (8 sectors at 4505218624 on sdg)
> [22578260.319122] md/raid:md1: read error corrected (8 sectors at 4505218632 on sdg)
> [22581681.874078] sd 6:0:6:0: [sdg] Unhandled sense code
> [22581681.874086] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [22581681.874090] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] [descriptor]
> [22581681.874095] Descriptor sense data with sense descriptors (in hex):
> [22581681.874098]         72 03 11 00 00 00 00 0c 00 0a 80 00 00 00 00 01 
> [22581681.874104]         1d fa 42 03 
> [22581681.874107] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
> [22581681.874112] sd 6:0:6:0: [sdg] CDB: Read(16): 88 00 00 00 00 01 1d fa 42 00 00 00 01 00 00 00
> [22581681.874121] end_request: I/O error, dev sdg, sector 4797907456
> [22581686.823444] raid5_end_read_request: 22 callbacks suppressed
> [22581686.823451] md/raid:md1: read error corrected (8 sectors at 4797907456 on sdg)
> [22581687.072047] md/raid:md1: read error corrected (8 sectors at 4797907464 on sdg)
> [22581687.072059] md/raid:md1: read error corrected (8 sectors at 4797907472 on sdg)
> [22581687.072062] md/raid:md1: read error corrected (8 sectors at 4797907480 on sdg)
> [22581687.072065] md/raid:md1: read error corrected (8 sectors at 4797907488 on sdg)
> [22581687.072973] md/raid:md1: read error corrected (8 sectors at 4797907496 on sdg)
> [22581687.072980] md/raid:md1: read error corrected (8 sectors at 4797907504 on sdg)
> [22581687.072982] md/raid:md1: read error corrected (8 sectors at 4797907512 on sdg)
> [22581687.072990] md/raid:md1: read error corrected (8 sectors at 4797907520 on sdg)
> [22581687.072994] md/raid:md1: read error corrected (8 sectors at 4797907528 on sdg)
> [22581698.030939] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22581698.030947] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22581698.030951] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22581698.030954] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22581698.030958] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22581698.030995] sd 6:0:6:0: [sdg] Unhandled sense code
> [22581698.031000] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [22581698.031005] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] [descriptor]
> [22581698.031010] Descriptor sense data with sense descriptors (in hex):
> [22581698.031012]         72 03 11 00 00 00 00 0c 00 0a 80 00 00 00 00 01 
> [22581698.031018]         1d fa b7 00 
> [22581698.031021] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
> [22581698.031026] sd 6:0:6:0: [sdg] CDB: Read(16): 88 00 00 00 00 01 1d fa b7 00 00 00 00 c8 00 00
> [22581698.031035] end_request: I/O error, dev sdg, sector 4797937408
> [22581701.434353] raid5_end_read_request: 22 callbacks suppressed
> [22581701.434357] md/raid:md1: read error corrected (8 sectors at 4797937408 on sdg)
> [22581701.434367] md/raid:md1: read error corrected (8 sectors at 4797937416 on sdg)
> [22581701.435141] md/raid:md1: read error corrected (8 sectors at 4797937424 on sdg)
> [22581701.435148] md/raid:md1: read error corrected (8 sectors at 4797937432 on sdg)
> [22581701.435151] md/raid:md1: read error corrected (8 sectors at 4797937440 on sdg)
> [22581701.435160] md/raid:md1: read error corrected (8 sectors at 4797937448 on sdg)
> [22581701.435164] md/raid:md1: read error corrected (8 sectors at 4797937456 on sdg)
> [22581701.435172] md/raid:md1: read error corrected (8 sectors at 4797937464 on sdg)
> [22581701.435175] md/raid:md1: read error corrected (8 sectors at 4797937472 on sdg)
> [22581701.435183] md/raid:md1: read error corrected (8 sectors at 4797937480 on sdg)
> [22581981.870472] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22581981.870482] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22581981.870487] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22581981.870493] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22581981.870498] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22581981.870605] sd 6:0:6:0: [sdg] Unhandled sense code
> [22581981.870613] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [22581981.870618] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] [descriptor]
> [22581981.870623] Descriptor sense data with sense descriptors (in hex):
> [22581981.870625]         72 03 11 00 00 00 00 0c 00 0a 80 00 00 00 00 01 
> [22581981.870632]         1f 19 45 58 
> [22581981.870635] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
> [22581981.870639] sd 6:0:6:0: [sdg] CDB: Read(16): 88 00 00 00 00 01 1f 19 45 00 00 00 01 00 00 00
> [22581981.870648] end_request: I/O error, dev sdg, sector 4816717056
> [22581985.861765] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22581985.861773] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22581985.861778] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22581985.861782] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22581985.861786] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22581985.861791] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22581985.861799] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [22581985.861911] sd 6:0:6:0: [sdg] Unhandled sense code
> [22581985.861919] sd 6:0:6:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [22581985.861923] sd 6:0:6:0: [sdg] Sense Key : Medium Error [current] [descriptor]
> [22581985.861929] Descriptor sense data with sense descriptors (in hex):
> [22581985.861930]         72 03 11 00 00 00 00 0c 00 0a 80 00 00 00 00 01 
> [22581985.861937]         1f 19 45 59 
> [22581985.861939] sd 6:0:6:0: [sdg] Add. Sense: Unrecovered read error
> [22581985.861944] sd 6:0:6:0: [sdg] CDB: Read(16): 88 00 00 00 00 01 1f 19 45 00 00 00 01 00 00 00
> [22581985.861953] end_request: I/O error, dev sdg, sector 4816717056
> [22581985.862031] raid5_end_read_request: 15 callbacks suppressed
> [22581985.862033] md/raid:md1: read error NOT corrected!! (sector 4816717056 on sdg).
> [22581985.862037] md/raid:md1: Disk failure on sdg, disabling device.
> [22581985.862038] <1>md/raid:md1: Operation continuing on 3 devices.
> [22581985.862185] md/raid:md1: read error not correctable (sector 4816717064 on sdg).
> [22581985.862188] md/raid:md1: read error not correctable (sector 4816717072 on sdg).
> [22581985.862191] md/raid:md1: read error not correctable (sector 4816717080 on sdg).
> [22581985.862194] md/raid:md1: read error not correctable (sector 4816717088 on sdg).
> [22581985.862196] md/raid:md1: read error not correctable (sector 4816717096 on sdg).
> [22581985.862198] md/raid:md1: read error not correctable (sector 4816717104 on sdg).
> [22581985.862200] md/raid:md1: read error not correctable (sector 4816717112 on sdg).
> [22581985.862202] md/raid:md1: read error not correctable (sector 4816717120 on sdg).
> [22581985.862205] md/raid:md1: read error not correctable (sector 4816717128 on sdg).
> [22581986.111914] md: md1: recovery done.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: RAID6 rebuild stuck
  2012-10-02  5:56 ` NeilBrown
@ 2012-10-02 15:56   ` Brian Candler
  2012-10-02 19:51     ` John Robinson
  0 siblings, 1 reply; 8+ messages in thread
From: Brian Candler @ 2012-10-02 15:56 UTC (permalink / raw)
  To: NeilBrown; +Cc: linux-raid

On Tue, Oct 02, 2012 at 03:56:42PM +1000, NeilBrown wrote:
> Lesson: always do a read test as well as a write test!

And there was me thinking that drives read back data after writing it -
clearly they do not :-)

> Looks like a bug. md_do_sync() is still waiting for all the submitted sync
> requests to complete.  This suggests some sort of accounting problem, but I
> cannot easily see it.
> 
> A reboot is likely to be the only fix.

Thank you. We'll take it out of service and upgrade to 12.04 at the same
time.

Cheers,

Brian.

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: RAID6 rebuild stuck
  2012-10-02 15:56   ` Brian Candler
@ 2012-10-02 19:51     ` John Robinson
  2012-10-02 20:29       ` NeilBrown
  0 siblings, 1 reply; 8+ messages in thread
From: John Robinson @ 2012-10-02 19:51 UTC (permalink / raw)
  To: NeilBrown; +Cc: linux-raid

On 02/10/2012 16:56, Brian Candler wrote:
> On Tue, Oct 02, 2012 at 03:56:42PM +1000, NeilBrown wrote:
>> Lesson: always do a read test as well as a write test!
>
> And there was me thinking that drives read back data after writing it -
> clearly they do not :-)

Would it be worth adding a re-read onto the end of the usual read 
failure reconstruct-write cycle?

Cheers,

John.


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: RAID6 rebuild stuck
  2012-10-02 19:51     ` John Robinson
@ 2012-10-02 20:29       ` NeilBrown
  2012-10-02 20:35         ` John Robinson
  0 siblings, 1 reply; 8+ messages in thread
From: NeilBrown @ 2012-10-02 20:29 UTC (permalink / raw)
  To: John Robinson; +Cc: linux-raid

[-- Attachment #1: Type: text/plain, Size: 668 bytes --]

On Tue, 02 Oct 2012 20:51:31 +0100 John Robinson
<john.robinson@anonymous.org.uk> wrote:

> On 02/10/2012 16:56, Brian Candler wrote:
> > On Tue, Oct 02, 2012 at 03:56:42PM +1000, NeilBrown wrote:
> >> Lesson: always do a read test as well as a write test!
> >
> > And there was me thinking that drives read back data after writing it -
> > clearly they do not :-)
> 
> Would it be worth adding a re-read onto the end of the usual read 
> failure reconstruct-write cycle?

What?  Read twice?

We already read after the write when attempting to fix a failure.
I suspect it doesn't do much good though as the data is in cache in the drive.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: RAID6 rebuild stuck
  2012-10-02 20:29       ` NeilBrown
@ 2012-10-02 20:35         ` John Robinson
  2012-10-03  1:21           ` NeilBrown
  0 siblings, 1 reply; 8+ messages in thread
From: John Robinson @ 2012-10-02 20:35 UTC (permalink / raw)
  To: NeilBrown; +Cc: linux-raid

On 02/10/2012 21:29, NeilBrown wrote:
> On Tue, 02 Oct 2012 20:51:31 +0100 John Robinson
> <john.robinson@anonymous.org.uk> wrote:
>
>> On 02/10/2012 16:56, Brian Candler wrote:
>>> On Tue, Oct 02, 2012 at 03:56:42PM +1000, NeilBrown wrote:
>>>> Lesson: always do a read test as well as a write test!
>>>
>>> And there was me thinking that drives read back data after writing it -
>>> clearly they do not :-)
>>
>> Would it be worth adding a re-read onto the end of the usual read
>> failure reconstruct-write cycle?
>
> What?  Read twice?
>
> We already read after the write when attempting to fix a failure.

I didn't realise you already did that.

> I suspect it doesn't do much good though as the data is in cache in the drive.

No, maybe not. There isn't an option you can add to ask for an uncached 
read? But I dare say you'd have thought of that already if there was :-)

Cheers,

John.


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: RAID6 rebuild stuck
  2012-10-02 20:35         ` John Robinson
@ 2012-10-03  1:21           ` NeilBrown
  0 siblings, 0 replies; 8+ messages in thread
From: NeilBrown @ 2012-10-03  1:21 UTC (permalink / raw)
  To: John Robinson; +Cc: linux-raid

[-- Attachment #1: Type: text/plain, Size: 1640 bytes --]

On Tue, 02 Oct 2012 21:35:05 +0100 John Robinson
<john.robinson@anonymous.org.uk> wrote:

> On 02/10/2012 21:29, NeilBrown wrote:
> > On Tue, 02 Oct 2012 20:51:31 +0100 John Robinson
> > <john.robinson@anonymous.org.uk> wrote:
> >
> >> On 02/10/2012 16:56, Brian Candler wrote:
> >>> On Tue, Oct 02, 2012 at 03:56:42PM +1000, NeilBrown wrote:
> >>>> Lesson: always do a read test as well as a write test!
> >>>
> >>> And there was me thinking that drives read back data after writing it -
> >>> clearly they do not :-)
> >>
> >> Would it be worth adding a re-read onto the end of the usual read
> >> failure reconstruct-write cycle?
> >
> > What?  Read twice?
> >
> > We already read after the write when attempting to fix a failure.
> 
> I didn't realise you already did that.
> 
> > I suspect it doesn't do much good though as the data is in cache in the drive.
> 
> No, maybe not. There isn't an option you can add to ask for an uncached 
> read? But I dare say you'd have thought of that already if there was :-)

Maybe setting REQ_FUA would work.  Or maybe it would cause crashes and random
data corruption.  I suspect most people think of REQ_FUA as being associated
with WRITEs.  Maybe I'm to cynical, but this would be extremely hard to test,
and I don't feel up to the code review that would be required to provide
sufficient confidence :-(

NeilBrown


> 
> Cheers,
> 
> John.
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2012-10-03  1:21 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-10-01 11:29 RAID6 rebuild stuck Brian Candler
2012-10-02  1:50 ` Chris Murphy
2012-10-02  5:56 ` NeilBrown
2012-10-02 15:56   ` Brian Candler
2012-10-02 19:51     ` John Robinson
2012-10-02 20:29       ` NeilBrown
2012-10-02 20:35         ` John Robinson
2012-10-03  1:21           ` NeilBrown

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).