From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751408AbcCVUpO (ORCPT ); Tue, 22 Mar 2016 16:45:14 -0400 Received: from e37.co.us.ibm.com ([32.97.110.158]:44542 "EHLO e37.co.us.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750696AbcCVUpJ convert rfc822-to-8bit (ORCPT ); Tue, 22 Mar 2016 16:45:09 -0400 X-IBM-Helo: d03dlp02.boulder.ibm.com X-IBM-MailFrom: paulmck@linux.vnet.ibm.com X-IBM-RcptTo: linux-kernel@vger.kernel.org Date: Tue, 22 Mar 2016 13:45:10 -0700 From: "Paul E. McKenney" To: Bart Van Assche Cc: linux-kernel@vger.kernel.org Subject: Re: RCU stall Message-ID: <20160322204510.GS4287@linux.vnet.ibm.com> Reply-To: paulmck@linux.vnet.ibm.com References: <56F1A8F2.9000905@sandisk.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: 8BIT In-Reply-To: <56F1A8F2.9000905@sandisk.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 16032220-0025-0000-0000-0000280A754C Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Mar 22, 2016 at 01:20:02PM -0700, Bart Van Assche wrote: > Hello Paul, > > Maybe you remember that we met briefly during the LinuxCon VIP > reception in the Chihuly Garden in Seattle (August 2015)? > > Yesterday I tried to verify a change in the ib_srpt driver on my > setup by letting the ib_srp and ib_srpt drivers communicate with > each other on the same system using the loopback functionality > provided by Mellanox ConnectX-3 HCA's. Instead of running into any > SRP related issues I hit something that looks like an RCU stall. I > can reproduce this stall with the v4.4.6 and v4.5-rc7 kernels (I > have not yet tried any other kernels). Can you have a look at the > call traces in the attached netconsole output and check whether or > not this is related to the RCU implementation in the Linux kernel? > The test I ran is as follows: > * Load and configure the ib_srpt target driver such that it exports > a RAM disk (brd) as a SCSI device. > * Load and configure the ib_srp initiator driver (which is a SCSI > initiator driver). > * Start the srp_daemon. This daemon makes the ib_srp initiator > driver log in to the ib_srpt target driver. > * Activate multipath on top of the SCSI devices created by ib_srp. > * Make fio send a workload with a high queue depth to the ib_srp > initiator driver. > * Restart the SCSI target stack (ib_srpt + the kernel modules used > by that driver). > > I can post this information on a public mailing list if you would > prefer this. No need, I added LKML on CC. Depending on what you find, you might want to add more people or more lists. You are getting a soft lockup as well as an RCU CPU stall warning, so it looks like something is taking a very long time in blk_done_softirq(). You have multiple occurrences at different times, so it looks to be a long time as opposed to an infinite time. Are you perhaps doing something that would make a huge amount of work for blk_done_softirq()? See Documentation/RCU/stallwarn.txt in the kernel source tree for more info on how to debug this sort of thing. Thanx, Paul > Thanks, > > Bart. > [10416.888346] device-mapper: multipath: Failing path 8:112. > [10416.894423] device-mapper: multipath: Failing path 8:96. > [10416.906746] device-mapper: multipath: Failing path 8:240. > [10416.907224] device-mapper: multipath: Failing path 66:80. > [10416.915126] device-mapper: multipath: Failing path 65:192. > [10416.915479] device-mapper: multipath: Failing path 66:160. > [10416.990626] device-mapper: multipath: Failing path 67:144. > [10417.961194] device-mapper: multipath: Failing path 66:48. > [10417.968416] device-mapper: multipath: Failing path 67:128. > [10417.976784] device-mapper: multipath: Failing path 67:64. > [10418.996938] device-mapper: multipath: Failing path 8:128. > [10418.997469] device-mapper: multipath: Failing path 65:80. > [10419.004872] device-mapper: multipath: Failing path 65:128. > [10419.006014] device-mapper: multipath: Failing path 8:176. > [10419.007623] device-mapper: multipath: Failing path 67:96. > [10419.011766] device-mapper: multipath: Failing path 65:64. > [10420.054544] scsi_io_completion: 22 callbacks suppressed > [10420.054632] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10420.054704] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] > [10420.054771] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10420.054847] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10420.054917] blk_update_request: 22 callbacks suppressed > [10420.054989] blk_update_request: I/O error, dev sdj, sector 1952320 > [10420.055114] device-mapper: multipath: Failing path 8:144. > [10420.060874] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10420.060927] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] > [10420.060965] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10420.061010] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 00 02 c0 00 00 40 00 > [10420.061051] blk_update_request: I/O error, dev sdz, sector 704 > [10420.061118] device-mapper: multipath: Failing path 65:144. > [10420.067122] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10420.067173] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] > [10420.067224] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10420.067289] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10420.067351] blk_update_request: I/O error, dev sdk, sector 1952320 > [10420.067445] device-mapper: multipath: Failing path 8:160. > [10420.067825] sd 25:0:0:1: [sdaa] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10420.067894] sd 25:0:0:1: [sdaa] tag#0 Sense Key : Not Ready [current] > [10420.067954] sd 25:0:0:1: [sdaa] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10420.068017] sd 25:0:0:1: [sdaa] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10420.068075] blk_update_request: I/O error, dev sdaa, sector 1952320 > [10420.068165] device-mapper: multipath: Failing path 65:160. > [10420.073375] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10420.073456] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] > [10420.073520] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10420.073582] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 00 01 10 00 00 60 00 > [10420.073641] blk_update_request: I/O error, dev sdad, sector 272 > [10420.073738] device-mapper: multipath: Failing path 65:208. > [10420.075521] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10420.075642] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] > [10420.075748] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10420.075855] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 02 60 00 00 60 00 > [10420.075955] blk_update_request: I/O error, dev sdo, sector 608 > [10420.076204] device-mapper: multipath: Failing path 8:224. > [10420.081491] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10420.081597] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] > [10420.081685] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10420.081778] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 1d ca 70 00 00 60 00 > [10420.081850] blk_update_request: I/O error, dev sdai, sector 1952368 > [10420.081939] device-mapper: multipath: Failing path 66:32. > [10420.157841] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10420.157922] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] > [10420.157982] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10420.158023] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10420.158059] blk_update_request: I/O error, dev sdbk, sector 1952320 > [10420.158124] device-mapper: multipath: Failing path 67:224. > [10420.184232] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10420.184336] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] > [10420.184417] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10420.184505] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10420.184588] blk_update_request: I/O error, dev sdas, sector 1952320 > [10420.184713] device-mapper: multipath: Failing path 66:192. > [10420.190423] sd 32:0:0:1: [sdar] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10420.190528] sd 32:0:0:1: [sdar] tag#0 Sense Key : Not Ready [current] > [10420.190711] sd 32:0:0:1: [sdar] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10420.190805] sd 32:0:0:1: [sdar] tag#0 CDB: Read(10) 28 00 00 00 02 c0 00 00 40 00 > [10420.190921] blk_update_request: I/O error, dev sdar, sector 704 > [10420.191092] device-mapper: multipath: Failing path 66:176. > [10421.152191] device-mapper: multipath: Failing path 65:176. > [10421.152905] device-mapper: multipath: Failing path 66:208. > [10421.158332] device-mapper: multipath: Failing path 66:64. > [10421.169109] device-mapper: multipath: Failing path 68:16. > [10421.169450] device-mapper: multipath: Failing path 67:192. > [10421.178113] device-mapper: multipath: Failing path 68:0. > [10422.226466] device-mapper: multipath: Failing path 8:112. > [10422.233709] device-mapper: multipath: Failing path 8:96. > [10422.247912] device-mapper: multipath: Failing path 8:240. > [10422.248844] device-mapper: multipath: Failing path 66:80. > [10422.255501] device-mapper: multipath: Failing path 65:192. > [10422.261653] device-mapper: multipath: Failing path 66:160. > [10422.324643] device-mapper: multipath: Failing path 67:144. > [10423.297527] device-mapper: multipath: Failing path 66:48. > [10423.307128] device-mapper: multipath: Failing path 67:128. > [10423.314250] device-mapper: multipath: Failing path 67:64. > [10424.336170] device-mapper: multipath: Failing path 8:128. > [10424.336732] device-mapper: multipath: Failing path 65:80. > [10424.345244] device-mapper: multipath: Failing path 65:128. > [10424.345717] device-mapper: multipath: Failing path 65:64. > [10424.346791] device-mapper: multipath: Failing path 8:176. > [10424.354214] device-mapper: multipath: Failing path 67:96. > [10425.390756] scsi_io_completion: 22 callbacks suppressed > [10425.390886] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10425.390997] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] > [10425.391099] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10425.391214] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10425.391321] blk_update_request: 22 callbacks suppressed > [10425.391415] blk_update_request: I/O error, dev sdj, sector 1952320 > [10425.391564] device-mapper: multipath: Failing path 8:144. > [10425.398476] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10425.398564] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] > [10425.398632] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10425.398703] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 00 02 c0 00 00 40 00 > [10425.398767] blk_update_request: I/O error, dev sdz, sector 704 > [10425.398866] device-mapper: multipath: Failing path 65:144. > [10425.400705] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10425.400779] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] > [10425.400845] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10425.400911] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10425.400969] blk_update_request: I/O error, dev sdk, sector 1952320 > [10425.401137] device-mapper: multipath: Failing path 8:160. > [10425.401616] sd 25:0:0:1: [sdaa] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10425.401684] sd 25:0:0:1: [sdaa] tag#0 Sense Key : Not Ready [current] > [10425.401743] sd 25:0:0:1: [sdaa] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10425.401807] sd 25:0:0:1: [sdaa] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10425.401866] blk_update_request: I/O error, dev sdaa, sector 1952320 > [10425.401967] device-mapper: multipath: Failing path 65:160. > [10425.404794] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10425.404879] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] > [10425.404952] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10425.405029] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 00 00 78 00 00 08 00 > [10425.405103] blk_update_request: I/O error, dev sdai, sector 120 > [10425.405273] device-mapper: multipath: Failing path 66:32. > [10425.409526] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10425.409609] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] > [10425.409674] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10425.409741] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 02 c0 00 00 40 00 > [10425.409806] blk_update_request: I/O error, dev sdo, sector 704 > [10425.409971] device-mapper: multipath: Failing path 8:224. > [10425.411359] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10425.411471] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] > [10425.411547] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10425.411614] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 00 03 00 00 00 e8 00 > [10425.411680] blk_update_request: I/O error, dev sdas, sector 768 > [10425.411770] device-mapper: multipath: Failing path 66:192. > [10425.417081] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10425.417156] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] > [10425.417217] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10425.417283] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 1d cb b0 00 00 50 00 > [10425.417351] blk_update_request: I/O error, dev sdad, sector 1952688 > [10425.417512] device-mapper: multipath: Failing path 65:208. > [10425.483947] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10425.484013] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] > [10425.484076] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10425.484127] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10425.484177] blk_update_request: I/O error, dev sdbk, sector 1952320 > [10425.484245] device-mapper: multipath: Failing path 67:224. > [10425.497810] sd 32:0:0:1: [sdar] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10425.497866] sd 32:0:0:1: [sdar] tag#0 Sense Key : Not Ready [current] > [10425.497910] sd 32:0:0:1: [sdar] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10425.497957] sd 32:0:0:1: [sdar] tag#0 CDB: Read(10) 28 00 00 1d ca 18 00 00 08 00 > [10425.497999] blk_update_request: I/O error, dev sdar, sector 1952280 > [10425.498065] device-mapper: multipath: Failing path 66:176. > [10426.456675] device-mapper: multipath: Failing path 65:176. > [10426.457202] device-mapper: multipath: Failing path 66:208. > [10426.462722] device-mapper: multipath: Failing path 66:64. > [10426.472875] device-mapper: multipath: Failing path 68:16. > [10426.473257] device-mapper: multipath: Failing path 67:192. > [10426.477439] device-mapper: multipath: Failing path 68:0. > [10427.509583] device-mapper: multipath: Failing path 8:112. > [10427.516675] device-mapper: multipath: Failing path 8:96. > [10427.529858] device-mapper: multipath: Failing path 8:240. > [10427.530437] device-mapper: multipath: Failing path 66:80. > [10427.537627] device-mapper: multipath: Failing path 65:192. > [10427.544651] device-mapper: multipath: Failing path 66:160. > [10427.616573] device-mapper: multipath: Failing path 67:144. > [10428.580910] device-mapper: multipath: Failing path 66:48. > [10428.589846] device-mapper: multipath: Failing path 67:128. > [10428.601369] device-mapper: multipath: Failing path 67:64. > [10429.602027] device-mapper: multipath: Failing path 8:128. > [10429.607145] device-mapper: multipath: Failing path 65:80. > [10429.609131] device-mapper: multipath: Failing path 65:128. > [10429.619473] device-mapper: multipath: Failing path 8:176. > [10429.712106] device-mapper: multipath: Failing path 67:96. > [10430.675345] scsi_io_completion: 21 callbacks suppressed > [10430.675424] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10430.675493] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] > [10430.675563] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10430.675645] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10430.675711] blk_update_request: 21 callbacks suppressed > [10430.675777] blk_update_request: I/O error, dev sdj, sector 1952320 > [10430.675880] device-mapper: multipath: Failing path 8:144. > [10430.677763] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10430.677884] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] > [10430.677980] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10430.678092] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 00 08 00 00 00 08 00 > [10430.678189] blk_update_request: I/O error, dev sdz, sector 2048 > [10430.678408] device-mapper: multipath: Failing path 65:144. > [10430.680621] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10430.680774] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] > [10430.680903] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10430.681010] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10430.681132] blk_update_request: I/O error, dev sdk, sector 1952320 > [10430.681357] device-mapper: multipath: Failing path 8:160. > [10430.686148] sd 25:0:0:1: [sdaa] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10430.686267] sd 25:0:0:1: [sdaa] tag#0 Sense Key : Not Ready [current] > [10430.686364] sd 25:0:0:1: [sdaa] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10430.686468] sd 25:0:0:1: [sdaa] tag#0 CDB: Read(10) 28 00 00 00 00 20 00 00 18 00 > [10430.686557] blk_update_request: I/O error, dev sdaa, sector 32 > [10430.687034] device-mapper: multipath: Failing path 65:160. > [10430.687938] sd 20:0:0:2: [sdu] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10430.688015] sd 20:0:0:2: [sdu] tag#0 Sense Key : Not Ready [current] > [10430.688077] sd 20:0:0:2: [sdu] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10430.688137] sd 20:0:0:2: [sdu] tag#0 CDB: Read(10) 28 00 00 00 02 60 00 00 60 00 > [10430.688193] blk_update_request: I/O error, dev sdu, sector 608 > [10430.688319] device-mapper: multipath: Failing path 65:64. > [10430.690959] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10430.691031] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] > [10430.691091] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10430.691151] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 03 00 00 00 b8 00 > [10430.691211] blk_update_request: I/O error, dev sdo, sector 768 > [10430.691397] device-mapper: multipath: Failing path 8:224. > [10430.694975] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10430.695052] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] > [10430.695115] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10430.695182] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 1d ca 00 00 00 18 00 > [10430.695240] blk_update_request: I/O error, dev sdad, sector 1952256 > [10430.695480] device-mapper: multipath: Failing path 65:208. > [10430.776287] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10430.776347] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] > [10430.776388] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10430.776430] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 1d cd b8 00 00 08 00 > [10430.776471] blk_update_request: I/O error, dev sdai, sector 1953208 > [10430.776537] device-mapper: multipath: Failing path 66:32. > [10430.779410] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10430.779467] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] > [10430.779511] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10430.779562] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d ca 18 00 00 08 00 > [10430.779606] blk_update_request: I/O error, dev sdas, sector 1952280 > [10430.779712] device-mapper: multipath: Failing path 66:192. > [10430.782147] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10430.782219] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] > [10430.782279] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10430.782342] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10430.782402] blk_update_request: I/O error, dev sdbk, sector 1952320 > [10430.782491] device-mapper: multipath: Failing path 67:224. > [10430.784545] device-mapper: multipath: Failing path 66:176. > [10431.767557] device-mapper: multipath: Failing path 65:176. > [10431.767921] device-mapper: multipath: Failing path 66:208. > [10431.773168] device-mapper: multipath: Failing path 66:64. > [10431.793497] device-mapper: multipath: Failing path 68:16. > [10431.794007] device-mapper: multipath: Failing path 67:192. > [10431.798234] device-mapper: multipath: Failing path 68:0. > [10432.827559] device-mapper: multipath: Failing path 8:112. > [10432.834048] device-mapper: multipath: Failing path 8:96. > [10432.847571] device-mapper: multipath: Failing path 8:240. > [10432.848142] device-mapper: multipath: Failing path 66:80. > [10432.854439] device-mapper: multipath: Failing path 65:192. > [10432.859782] device-mapper: multipath: Failing path 66:160. > [10432.925961] device-mapper: multipath: Failing path 67:144. > [10433.916020] device-mapper: multipath: Failing path 66:48. > [10433.923574] device-mapper: multipath: Failing path 67:128. > [10433.931521] device-mapper: multipath: Failing path 67:64. > [10434.944355] device-mapper: multipath: Failing path 8:128. > [10434.944780] device-mapper: multipath: Failing path 65:80. > [10434.946445] device-mapper: multipath: Failing path 67:96. > [10434.953279] device-mapper: multipath: Failing path 65:128. > [10434.961368] device-mapper: multipath: Failing path 8:176. > [10435.997587] scsi_io_completion: 22 callbacks suppressed > [10435.997666] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10435.997735] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] > [10435.997800] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10435.997868] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10435.997932] blk_update_request: 22 callbacks suppressed > [10435.997992] blk_update_request: I/O error, dev sdj, sector 1952320 > [10435.998093] device-mapper: multipath: Failing path 8:144. > [10435.998848] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10435.998940] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] > [10435.999015] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10435.999136] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10435.999212] blk_update_request: I/O error, dev sdz, sector 1952320 > [10435.999358] device-mapper: multipath: Failing path 65:144. > [10436.003810] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10436.003924] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] > [10436.004034] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10436.004182] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10436.004300] blk_update_request: I/O error, dev sdk, sector 1952320 > [10436.004486] device-mapper: multipath: Failing path 8:160. > [10436.006466] sd 25:0:0:1: [sdaa] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10436.006572] sd 25:0:0:1: [sdaa] tag#0 Sense Key : Not Ready [current] > [10436.006677] sd 25:0:0:1: [sdaa] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10436.006835] sd 25:0:0:1: [sdaa] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10436.006943] blk_update_request: I/O error, dev sdaa, sector 1952320 > [10436.007132] device-mapper: multipath: Failing path 65:160. > [10436.009417] sd 20:0:0:2: [sdu] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10436.009500] sd 20:0:0:2: [sdu] tag#0 Sense Key : Not Ready [current] > [10436.009563] sd 20:0:0:2: [sdu] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10436.009626] sd 20:0:0:2: [sdu] tag#0 CDB: Read(10) 28 00 00 00 02 b8 00 00 48 00 > [10436.009686] blk_update_request: I/O error, dev sdu, sector 696 > [10436.009832] device-mapper: multipath: Failing path 65:64. > [10436.013058] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10436.013131] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] > [10436.013213] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10436.013311] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 00 00 20 00 00 18 00 > [10436.013405] blk_update_request: I/O error, dev sdad, sector 32 > [10436.013575] device-mapper: multipath: Failing path 65:208. > [10436.016633] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10436.016704] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] > [10436.016760] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10436.016820] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 02 c0 00 00 40 00 > [10436.016876] blk_update_request: I/O error, dev sdo, sector 704 > [10436.017086] device-mapper: multipath: Failing path 8:224. > [10436.022985] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10436.023062] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] > [10436.023132] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10436.023201] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d ca 70 00 00 60 00 > [10436.023264] blk_update_request: I/O error, dev sdas, sector 1952368 > [10436.023405] device-mapper: multipath: Failing path 66:192. > [10436.024977] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10436.025082] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] > [10436.025178] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10436.025271] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 1d cb 20 00 00 30 00 > [10436.025328] blk_update_request: I/O error, dev sdai, sector 1952544 > [10436.025472] device-mapper: multipath: Failing path 66:32. > [10436.119145] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10436.119237] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] > [10436.119305] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10436.119380] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10436.119486] blk_update_request: I/O error, dev sdbk, sector 1952320 > [10436.119638] device-mapper: multipath: Failing path 67:224. > [10436.133091] device-mapper: multipath: Failing path 66:176. > [10437.082788] device-mapper: multipath: Failing path 65:176. > [10437.083248] device-mapper: multipath: Failing path 66:208. > [10437.083653] device-mapper: multipath: Failing path 68:16. > [10437.083928] device-mapper: multipath: Failing path 67:192. > [10437.087703] device-mapper: multipath: Failing path 68:0. > [10437.088448] device-mapper: multipath: Failing path 66:64. > [10438.129087] device-mapper: multipath: Failing path 8:112. > [10438.138623] device-mapper: multipath: Failing path 8:96. > [10438.151534] device-mapper: multipath: Failing path 8:240. > [10438.152048] device-mapper: multipath: Failing path 66:80. > [10438.158935] device-mapper: multipath: Failing path 65:192. > [10438.164948] device-mapper: multipath: Failing path 66:160. > [10438.232625] device-mapper: multipath: Failing path 67:144. > [10439.204539] device-mapper: multipath: Failing path 66:48. > [10439.212592] device-mapper: multipath: Failing path 67:128. > [10439.220192] device-mapper: multipath: Failing path 67:64. > [10440.234002] device-mapper: multipath: Failing path 8:128. > [10440.234585] device-mapper: multipath: Failing path 65:80. > [10440.237500] device-mapper: multipath: Failing path 67:96. > [10440.241884] device-mapper: multipath: Failing path 8:176. > [10440.241930] device-mapper: multipath: Failing path 65:128. > [10441.293227] scsi_io_completion: 22 callbacks suppressed > [10441.293301] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10441.293360] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] > [10441.293420] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10441.293673] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10441.293733] blk_update_request: 22 callbacks suppressed > [10441.293785] blk_update_request: I/O error, dev sdj, sector 1952320 > [10441.293916] device-mapper: multipath: Failing path 8:144. > [10441.294596] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10441.294703] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] > [10441.294799] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10441.294939] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10441.295032] blk_update_request: I/O error, dev sdz, sector 1952320 > [10441.295203] device-mapper: multipath: Failing path 65:144. > [10441.297186] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10441.297273] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] > [10441.297341] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10441.297412] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10441.297518] blk_update_request: I/O error, dev sdk, sector 1952320 > [10441.297744] device-mapper: multipath: Failing path 8:160. > [10441.300255] sd 25:0:0:1: [sdaa] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10441.300340] sd 25:0:0:1: [sdaa] tag#0 Sense Key : Not Ready [current] > [10441.300419] sd 25:0:0:1: [sdaa] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10441.300535] sd 25:0:0:1: [sdaa] tag#0 CDB: Read(10) 28 00 00 00 00 38 00 00 08 00 > [10441.300626] blk_update_request: I/O error, dev sdaa, sector 56 > [10441.300861] device-mapper: multipath: Failing path 65:160. > [10441.302907] sd 20:0:0:2: [sdu] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10441.302972] sd 20:0:0:2: [sdu] tag#0 Sense Key : Not Ready [current] > [10441.303045] sd 20:0:0:2: [sdu] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10441.303132] sd 20:0:0:2: [sdu] tag#0 CDB: Read(10) 28 00 00 00 02 70 00 00 60 00 > [10441.303215] blk_update_request: I/O error, dev sdu, sector 624 > [10441.303399] device-mapper: multipath: Failing path 65:64. > [10441.306683] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10441.306792] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] > [10441.306862] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10441.306937] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 02 60 00 00 60 00 > [10441.307000] blk_update_request: I/O error, dev sdo, sector 608 > [10441.307201] device-mapper: multipath: Failing path 8:224. > [10441.307621] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10441.307685] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] > [10441.307739] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10441.307799] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 00 02 60 00 00 60 00 > [10441.307855] blk_update_request: I/O error, dev sdad, sector 608 > [10441.307939] device-mapper: multipath: Failing path 65:208. > [10441.317400] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10441.317503] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] > [10441.317591] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10441.317649] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 1d cb 20 00 00 30 00 > [10441.317704] blk_update_request: I/O error, dev sdai, sector 1952544 > [10441.317802] device-mapper: multipath: Failing path 66:32. > [10441.387950] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10441.388012] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] > [10441.388060] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10441.388108] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10441.388156] blk_update_request: I/O error, dev sdbk, sector 1952320 > [10441.388227] device-mapper: multipath: Failing path 67:224. > [10441.396210] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10441.396260] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] > [10441.396308] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10441.396377] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10441.396436] blk_update_request: I/O error, dev sdas, sector 1952320 > [10441.396525] device-mapper: multipath: Failing path 66:192. > [10441.404557] device-mapper: multipath: Failing path 66:176. > [10442.380325] device-mapper: multipath: Failing path 65:176. > [10442.380917] device-mapper: multipath: Failing path 66:208. > [10442.385656] device-mapper: multipath: Failing path 66:64. > [10442.398393] device-mapper: multipath: Failing path 68:16. > [10442.398714] device-mapper: multipath: Failing path 67:192. > [10442.403546] device-mapper: multipath: Failing path 68:0. > [10443.438787] device-mapper: multipath: Failing path 8:112. > [10443.445929] device-mapper: multipath: Failing path 8:96. > [10443.452757] device-mapper: multipath: Failing path 67:144. > [10443.456211] device-mapper: multipath: Failing path 8:240. > [10443.457007] device-mapper: multipath: Failing path 66:80. > [10443.465293] device-mapper: multipath: Failing path 65:192. > [10443.470696] device-mapper: multipath: Failing path 66:160. > [10444.503706] device-mapper: multipath: Failing path 66:48. > [10444.510208] device-mapper: multipath: Failing path 67:128. > [10444.517049] device-mapper: multipath: Failing path 67:64. > [10445.536571] device-mapper: multipath: Failing path 8:128. > [10445.537057] device-mapper: multipath: Failing path 65:80. > [10445.544179] device-mapper: multipath: Failing path 65:128. > [10445.546885] device-mapper: multipath: Failing path 8:176. > [10445.551266] device-mapper: multipath: Failing path 67:96. > [10446.585026] scsi_io_completion: 22 callbacks suppressed > [10446.585106] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10446.585175] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] > [10446.585238] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10446.585305] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10446.585368] blk_update_request: 22 callbacks suppressed > [10446.585427] blk_update_request: I/O error, dev sdj, sector 1952320 > [10446.585532] device-mapper: multipath: Failing path 8:144. > [10446.587225] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10446.587328] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] > [10446.587409] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10446.587544] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 00 00 18 00 00 08 00 > [10446.587626] blk_update_request: I/O error, dev sdz, sector 24 > [10446.588770] device-mapper: multipath: Failing path 65:144. > [10446.596518] sd 20:0:0:2: [sdu] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10446.596664] sd 20:0:0:2: [sdu] tag#0 Sense Key : Not Ready [current] > [10446.596783] sd 20:0:0:2: [sdu] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10446.596934] sd 20:0:0:2: [sdu] tag#0 CDB: Read(10) 28 00 00 00 02 60 00 00 60 00 > [10446.597056] blk_update_request: I/O error, dev sdu, sector 608 > [10446.597236] device-mapper: multipath: Failing path 65:64. > [10446.612060] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10446.612156] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] > [10446.612221] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10446.612292] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10446.612357] blk_update_request: I/O error, dev sdk, sector 1952320 > [10446.612575] device-mapper: multipath: Failing path 8:160. > [10446.613185] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10446.613324] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] > [10446.613442] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10446.613576] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10446.613694] blk_update_request: I/O error, dev sdad, sector 1952320 > [10446.613897] device-mapper: multipath: Failing path 65:208. > [10446.618945] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10446.619076] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] > [10446.619186] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10446.619329] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 00 00 20 00 00 18 00 > [10446.619458] blk_update_request: I/O error, dev sdai, sector 32 > [10446.619717] device-mapper: multipath: Failing path 66:32. > [10446.623887] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10446.624020] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] > [10446.624136] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10446.624269] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 02 60 00 00 60 00 > [10446.624374] blk_update_request: I/O error, dev sdo, sector 608 > [10446.624660] device-mapper: multipath: Failing path 8:224. > [10446.630687] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10446.630831] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] > [10446.630956] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10446.631081] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d ca 70 00 00 90 00 > [10446.631193] blk_update_request: I/O error, dev sdas, sector 1952368 > [10446.631370] device-mapper: multipath: Failing path 66:192. > [10446.701475] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10446.701555] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] > [10446.701616] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10446.701679] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10446.701738] blk_update_request: I/O error, dev sdbk, sector 1952320 > [10446.701834] device-mapper: multipath: Failing path 67:224. > [10446.722225] sd 32:0:0:1: [sdar] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10446.722330] sd 32:0:0:1: [sdar] tag#0 Sense Key : Not Ready [current] > [10446.722422] sd 32:0:0:1: [sdar] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10446.722506] sd 32:0:0:1: [sdar] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10446.722604] blk_update_request: I/O error, dev sdar, sector 1952320 > [10446.722745] device-mapper: multipath: Failing path 66:176. > [10447.703907] device-mapper: multipath: Failing path 65:160. > [10447.704418] device-mapper: multipath: Failing path 66:64. > [10447.711405] device-mapper: multipath: Failing path 65:176. > [10447.717415] device-mapper: multipath: Failing path 66:208. > [10447.736685] device-mapper: multipath: Failing path 68:16. > [10447.737169] device-mapper: multipath: Failing path 67:192. > [10447.742386] device-mapper: multipath: Failing path 68:0. > [10448.746828] device-mapper: multipath: Failing path 8:112. > [10448.752272] device-mapper: multipath: Failing path 8:96. > [10448.777442] device-mapper: multipath: Failing path 8:240. > [10448.778049] device-mapper: multipath: Failing path 66:80. > [10448.783876] device-mapper: multipath: Failing path 65:192. > [10448.784283] device-mapper: multipath: Failing path 66:160. > [10448.846609] device-mapper: multipath: Failing path 67:144. > [10449.839649] device-mapper: multipath: Failing path 66:48. > [10449.846890] device-mapper: multipath: Failing path 67:128. > [10449.855407] device-mapper: multipath: Failing path 67:64. > [10450.875424] device-mapper: multipath: Failing path 8:128. > [10450.875881] device-mapper: multipath: Failing path 65:80. > [10450.876903] device-mapper: multipath: Failing path 67:96. > [10450.883995] device-mapper: multipath: Failing path 65:128. > [10450.886613] device-mapper: multipath: Failing path 8:176. > [10451.926827] scsi_io_completion: 22 callbacks suppressed > [10451.926909] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10451.926974] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] > [10451.927034] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10451.927099] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10451.927160] blk_update_request: 22 callbacks suppressed > [10451.927219] blk_update_request: I/O error, dev sdj, sector 1952320 > [10451.927337] device-mapper: multipath: Failing path 8:144. > [10451.930152] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10451.930249] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] > [10451.930332] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10451.930451] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 00 00 78 00 00 08 00 > [10451.930536] blk_update_request: I/O error, dev sdz, sector 120 > [10451.930684] device-mapper: multipath: Failing path 65:144. > [10451.935021] sd 20:0:0:2: [sdu] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10451.935127] sd 20:0:0:2: [sdu] tag#0 Sense Key : Not Ready [current] > [10451.935186] sd 20:0:0:2: [sdu] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10451.935253] sd 20:0:0:2: [sdu] tag#0 CDB: Read(10) 28 00 00 00 02 60 00 00 60 00 > [10451.935309] blk_update_request: I/O error, dev sdu, sector 608 > [10451.935450] device-mapper: multipath: Failing path 65:64. > [10451.937343] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10451.937431] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] > [10451.937506] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10451.937625] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10451.937705] blk_update_request: I/O error, dev sdk, sector 1952320 > [10451.937857] device-mapper: multipath: Failing path 8:160. > [10451.944256] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10451.944355] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] > [10451.944434] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10451.944502] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 00 00 10 00 00 08 00 > [10451.944560] blk_update_request: I/O error, dev sdad, sector 16 > [10451.944650] device-mapper: multipath: Failing path 65:208. > [10451.948096] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10451.948166] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] > [10451.948221] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10451.948282] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 03 60 00 00 60 00 > [10451.948338] blk_update_request: I/O error, dev sdo, sector 864 > [10451.948431] device-mapper: multipath: Failing path 8:224. > [10451.950866] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10451.950938] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] > [10451.950992] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10451.951048] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 1d c9 80 00 00 80 00 > [10451.951102] blk_update_request: I/O error, dev sdai, sector 1952128 > [10451.951194] device-mapper: multipath: Failing path 66:32. > [10452.032024] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10452.032084] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] > [10452.032122] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10452.032163] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10452.032199] blk_update_request: I/O error, dev sdbk, sector 1952320 > [10452.032299] device-mapper: multipath: Failing path 67:224. > [10452.038388] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10452.038449] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] > [10452.038500] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10452.038552] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10452.038596] blk_update_request: I/O error, dev sdas, sector 1952320 > [10452.038721] device-mapper: multipath: Failing path 66:192. > [10452.045632] sd 32:0:0:1: [sdar] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10452.045688] sd 32:0:0:1: [sdar] tag#0 Sense Key : Not Ready [current] > [10452.045733] sd 32:0:0:1: [sdar] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10452.045779] sd 32:0:0:1: [sdar] tag#0 CDB: Read(10) 28 00 00 00 02 c0 00 00 40 00 > [10452.045822] blk_update_request: I/O error, dev sdar, sector 704 > [10452.045927] device-mapper: multipath: Failing path 66:176. > [10453.023588] device-mapper: multipath: Failing path 65:160. > [10453.024299] device-mapper: multipath: Failing path 66:64. > [10453.030250] device-mapper: multipath: Failing path 65:176. > [10453.036030] device-mapper: multipath: Failing path 66:208. > [10453.082442] device-mapper: multipath: Failing path 68:16. > [10453.082943] device-mapper: multipath: Failing path 67:192. > [10453.088366] device-mapper: multipath: Failing path 68:0. > [10454.101354] device-mapper: multipath: Failing path 8:112. > [10454.109728] device-mapper: multipath: Failing path 8:96. > [10454.122180] device-mapper: multipath: Failing path 8:240. > [10454.123095] device-mapper: multipath: Failing path 66:80. > [10454.133511] device-mapper: multipath: Failing path 65:192. > [10454.218507] device-mapper: multipath: Failing path 67:144. > [10454.225052] device-mapper: multipath: Failing path 66:160. > [10455.184119] device-mapper: multipath: Failing path 66:48. > [10455.192576] device-mapper: multipath: Failing path 67:128. > [10455.199838] device-mapper: multipath: Failing path 67:64. > [10456.219579] device-mapper: multipath: Failing path 8:128. > [10456.220649] device-mapper: multipath: Failing path 65:80. > [10456.224363] device-mapper: multipath: Failing path 8:176. > [10456.228154] device-mapper: multipath: Failing path 65:128. > [10456.236087] device-mapper: multipath: Failing path 67:96. > [10457.265228] scsi_io_completion: 22 callbacks suppressed > [10457.265309] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10457.265376] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] > [10457.265441] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10457.265510] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10457.265573] blk_update_request: 22 callbacks suppressed > [10457.265631] blk_update_request: I/O error, dev sdj, sector 1952320 > [10457.265727] device-mapper: multipath: Failing path 8:144. > [10457.266608] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10457.266722] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] > [10457.266805] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10457.266929] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 1d c1 e8 00 00 08 00 > [10457.267015] blk_update_request: I/O error, dev sdz, sector 1950184 > [10457.267159] device-mapper: multipath: Failing path 65:144. > [10457.273904] sd 20:0:0:2: [sdu] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10457.274003] sd 20:0:0:2: [sdu] tag#0 Sense Key : Not Ready [current] > [10457.274078] sd 20:0:0:2: [sdu] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10457.274161] sd 20:0:0:2: [sdu] tag#0 CDB: Read(10) 28 00 00 00 02 88 00 00 60 00 > [10457.274236] blk_update_request: I/O error, dev sdu, sector 648 > [10457.274357] device-mapper: multipath: Failing path 65:64. > [10457.275488] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10457.275583] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] > [10457.275670] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10457.275796] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10457.275874] blk_update_request: I/O error, dev sdk, sector 1952320 > [10457.276184] device-mapper: multipath: Failing path 8:160. > [10457.276668] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10457.276766] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] > [10457.276841] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10457.276925] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10457.277005] blk_update_request: I/O error, dev sdad, sector 1952320 > [10457.277106] device-mapper: multipath: Failing path 65:208. > [10457.285381] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10457.285455] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] > [10457.285512] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10457.285573] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 02 f0 00 00 10 00 > [10457.285817] blk_update_request: I/O error, dev sdo, sector 752 > [10457.285944] device-mapper: multipath: Failing path 8:224. > [10457.286392] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10457.286477] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] > [10457.286552] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10457.286632] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 00 02 f0 00 00 10 00 > [10457.286707] blk_update_request: I/O error, dev sdai, sector 752 > [10457.286828] device-mapper: multipath: Failing path 66:32. > [10457.294129] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10457.294220] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] > [10457.294295] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10457.294364] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d cb b0 00 00 50 00 > [10457.294424] blk_update_request: I/O error, dev sdas, sector 1952688 > [10457.294523] device-mapper: multipath: Failing path 66:192. > [10457.372572] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10457.372641] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] > [10457.372688] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10457.372737] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10457.372783] blk_update_request: I/O error, dev sdbk, sector 1952320 > [10457.372852] device-mapper: multipath: Failing path 67:224. > [10457.379743] sd 32:0:0:1: [sdar] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10457.379833] sd 32:0:0:1: [sdar] tag#0 Sense Key : Not Ready [current] > [10457.379889] sd 32:0:0:1: [sdar] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10457.379955] sd 32:0:0:1: [sdar] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10457.380011] blk_update_request: I/O error, dev sdar, sector 1952320 > [10457.380163] device-mapper: multipath: Failing path 66:176. > [10458.355184] device-mapper: multipath: Failing path 65:160. > [10458.355600] device-mapper: multipath: Failing path 66:64. > [10458.362158] device-mapper: multipath: Failing path 65:176. > [10458.367380] device-mapper: multipath: Failing path 66:208. > [10458.388284] device-mapper: multipath: Failing path 68:16. > [10458.389179] device-mapper: multipath: Failing path 67:192. > [10458.395063] device-mapper: multipath: Failing path 68:0. > [10459.421645] device-mapper: multipath: Failing path 8:112. > [10459.427728] device-mapper: multipath: Failing path 8:96. > [10459.433866] device-mapper: multipath: Failing path 67:144. > [10459.444039] device-mapper: multipath: Failing path 8:240. > [10459.444657] device-mapper: multipath: Failing path 66:80. > [10459.453974] device-mapper: multipath: Failing path 65:192. > [10459.454422] device-mapper: multipath: Failing path 66:160. > [10460.482933] device-mapper: multipath: Failing path 66:48. > [10460.488513] device-mapper: multipath: Failing path 67:128. > [10460.495239] device-mapper: multipath: Failing path 67:64. > [10461.513588] device-mapper: multipath: Failing path 8:128. > [10461.514034] device-mapper: multipath: Failing path 65:80. > [10461.521182] device-mapper: multipath: Failing path 65:128. > [10461.522550] device-mapper: multipath: Failing path 8:176. > [10461.528881] device-mapper: multipath: Failing path 67:96. > [10462.564110] scsi_io_completion: 22 callbacks suppressed > [10462.564202] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10462.564288] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] > [10462.564368] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10462.564466] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10462.564557] blk_update_request: 22 callbacks suppressed > [10462.564640] blk_update_request: I/O error, dev sdj, sector 1952320 > [10462.564763] device-mapper: multipath: Failing path 8:144. > [10462.566732] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10462.566834] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] > [10462.566929] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10462.567055] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 00 00 18 00 00 08 00 > [10462.567137] blk_update_request: I/O error, dev sdz, sector 24 > [10462.567301] device-mapper: multipath: Failing path 65:144. > [10462.573104] sd 20:0:0:2: [sdu] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10462.573192] sd 20:0:0:2: [sdu] tag#0 Sense Key : Not Ready [current] > [10462.573263] sd 20:0:0:2: [sdu] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10462.573332] sd 20:0:0:2: [sdu] tag#0 CDB: Read(10) 28 00 00 00 02 60 00 00 60 00 > [10462.573390] blk_update_request: I/O error, dev sdu, sector 608 > [10462.573467] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10462.573472] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] > [10462.573477] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10462.573481] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10462.573485] blk_update_request: I/O error, dev sdk, sector 1952320 > [10462.573547] device-mapper: multipath: Failing path 8:160. > [10462.574255] device-mapper: multipath: Failing path 65:64. > [10462.574825] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10462.574884] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] > [10462.574938] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10462.574989] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 00 08 00 00 00 08 00 > [10462.575044] blk_update_request: I/O error, dev sdad, sector 2048 > [10462.575121] device-mapper: multipath: Failing path 65:208. > [10462.581259] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10462.581316] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] > [10462.581365] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10462.581413] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 03 00 00 00 b0 00 > [10462.581469] blk_update_request: I/O error, dev sdo, sector 768 > [10462.581594] device-mapper: multipath: Failing path 8:224. > [10462.583891] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10462.583954] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] > [10462.584001] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10462.584053] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 1d c9 60 00 00 60 00 > [10462.584103] blk_update_request: I/O error, dev sdai, sector 1952096 > [10462.584219] device-mapper: multipath: Failing path 66:32. > [10462.686906] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10462.686982] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] > [10462.687040] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10462.687097] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10462.687151] blk_update_request: I/O error, dev sdbk, sector 1952320 > [10462.687234] device-mapper: multipath: Failing path 67:224. > [10462.692048] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10462.692112] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] > [10462.692159] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10462.692210] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10462.692259] blk_update_request: I/O error, dev sdas, sector 1952320 > [10462.692330] device-mapper: multipath: Failing path 66:192. > [10462.699023] sd 32:0:0:1: [sdar] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10462.699089] sd 32:0:0:1: [sdar] tag#0 Sense Key : Not Ready [current] > [10462.699139] sd 32:0:0:1: [sdar] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10462.699190] sd 32:0:0:1: [sdar] tag#0 CDB: Read(10) 28 00 00 00 03 00 00 00 60 00 > [10462.699246] blk_update_request: I/O error, dev sdar, sector 768 > [10462.699413] device-mapper: multipath: Failing path 66:176. > [10463.689681] device-mapper: multipath: Failing path 65:160. > [10463.690254] device-mapper: multipath: Failing path 66:64. > [10463.697821] device-mapper: multipath: Failing path 65:176. > [10463.705424] device-mapper: multipath: Failing path 66:208. > [10463.726975] device-mapper: multipath: Failing path 68:16. > [10463.727373] device-mapper: multipath: Failing path 67:192. > [10463.736754] device-mapper: multipath: Failing path 68:0. > [10464.765376] device-mapper: multipath: Failing path 8:112. > [10464.772388] device-mapper: multipath: Failing path 8:96. > [10464.784249] device-mapper: multipath: Failing path 8:240. > [10464.784747] device-mapper: multipath: Failing path 66:80. > [10464.792111] device-mapper: multipath: Failing path 65:192. > [10464.798740] device-mapper: multipath: Failing path 66:160. > [10464.852449] device-mapper: multipath: Failing path 67:144. > [10465.837027] device-mapper: multipath: Failing path 66:48. > [10465.840983] device-mapper: multipath: Failing path 67:128. > [10465.847920] device-mapper: multipath: Failing path 67:64. > [10466.866019] device-mapper: multipath: Failing path 8:128. > [10466.866759] device-mapper: multipath: Failing path 65:80. > [10466.872622] device-mapper: multipath: Failing path 8:176. > [10466.874797] device-mapper: multipath: Failing path 65:128. > [10466.882759] device-mapper: multipath: Failing path 67:96. > [10467.910924] scsi_io_completion: 22 callbacks suppressed > [10467.910999] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10467.911058] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] > [10467.911116] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10467.911178] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10467.911235] blk_update_request: 22 callbacks suppressed > [10467.911287] blk_update_request: I/O error, dev sdj, sector 1952320 > [10467.911374] device-mapper: multipath: Failing path 8:144. > [10467.912763] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10467.912851] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] > [10467.912921] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10467.912996] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 00 08 00 00 00 08 00 > [10467.913102] blk_update_request: I/O error, dev sdz, sector 2048 > [10467.913339] device-mapper: multipath: Failing path 65:144. > [10467.919680] sd 20:0:0:2: [sdu] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10467.919777] sd 20:0:0:2: [sdu] tag#0 Sense Key : Not Ready [current] > [10467.919873] sd 20:0:0:2: [sdu] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10467.919942] sd 20:0:0:2: [sdu] tag#0 CDB: Read(10) 28 00 00 00 02 60 00 00 60 00 > [10467.920021] blk_update_request: I/O error, dev sdu, sector 608 > [10467.920234] device-mapper: multipath: Failing path 65:64. > [10467.920949] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10467.921039] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] > [10467.921110] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10467.921225] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10467.921310] blk_update_request: I/O error, dev sdk, sector 1952320 > [10467.921462] device-mapper: multipath: Failing path 8:160. > [10467.925848] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10467.925922] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] > [10467.925987] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10467.926058] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 00 00 20 00 00 18 00 > [10467.926266] blk_update_request: I/O error, dev sdad, sector 32 > [10467.926765] device-mapper: multipath: Failing path 65:208. > [10467.931080] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10467.931152] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] > [10467.931207] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10467.931262] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 03 00 00 00 88 00 > [10467.931316] blk_update_request: I/O error, dev sdo, sector 768 > [10467.931409] device-mapper: multipath: Failing path 8:224. > [10467.935526] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10467.935597] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] > [10467.935657] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10467.935717] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 1d ca 00 00 00 18 00 > [10467.935772] blk_update_request: I/O error, dev sdai, sector 1952256 > [10467.935916] device-mapper: multipath: Failing path 66:32. > [10468.010601] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10468.010674] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] > [10468.010723] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10468.010775] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10468.010823] blk_update_request: I/O error, dev sdbk, sector 1952320 > [10468.010900] device-mapper: multipath: Failing path 67:224. > [10468.017011] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10468.017091] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] > [10468.017156] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10468.017224] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d ca 68 00 00 08 00 > [10468.017293] blk_update_request: I/O error, dev sdas, sector 1952360 > [10468.017447] device-mapper: multipath: Failing path 66:192. > [10468.018534] sd 32:0:0:1: [sdar] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10468.018613] sd 32:0:0:1: [sdar] tag#0 Sense Key : Not Ready [current] > [10468.018679] sd 32:0:0:1: [sdar] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10468.018750] sd 32:0:0:1: [sdar] tag#0 CDB: Read(10) 28 00 00 1d ca 18 00 00 08 00 > [10468.018815] blk_update_request: I/O error, dev sdar, sector 1952280 > [10468.018916] device-mapper: multipath: Failing path 66:176. > [10469.006483] device-mapper: multipath: Failing path 65:160. > [10469.006965] device-mapper: multipath: Failing path 66:64. > [10469.012283] device-mapper: multipath: Failing path 65:176. > [10469.016967] device-mapper: multipath: Failing path 66:208. > [10469.047129] device-mapper: multipath: Failing path 68:16. > [10469.047451] device-mapper: multipath: Failing path 67:192. > [10469.052379] device-mapper: multipath: Failing path 68:0. > [10470.072818] device-mapper: multipath: Failing path 8:112. > [10470.079865] device-mapper: multipath: Failing path 8:96. > [10470.101784] device-mapper: multipath: Failing path 8:240. > [10470.102397] device-mapper: multipath: Failing path 66:80. > [10470.111733] device-mapper: multipath: Failing path 65:192. > [10470.112277] device-mapper: multipath: Failing path 66:160. > [10470.169999] device-mapper: multipath: Failing path 67:144. > [10471.157267] device-mapper: multipath: Failing path 66:48. > [10471.166331] device-mapper: multipath: Failing path 67:128. > [10471.172806] device-mapper: multipath: Failing path 67:64. > [10472.191121] device-mapper: multipath: Failing path 8:128. > [10472.191627] device-mapper: multipath: Failing path 65:80. > [10472.192259] device-mapper: multipath: Failing path 8:176. > [10472.198851] device-mapper: multipath: Failing path 65:128. > [10472.204424] device-mapper: multipath: Failing path 67:96. > [10473.252723] scsi_io_completion: 22 callbacks suppressed > [10473.252798] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10473.252860] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] > [10473.252918] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10473.252981] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10473.253045] blk_update_request: 22 callbacks suppressed > [10473.253099] blk_update_request: I/O error, dev sdj, sector 1952320 > [10473.253236] device-mapper: multipath: Failing path 8:144. > [10473.254825] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10473.254913] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] > [10473.254982] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10473.255051] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 1d c1 e8 00 00 08 00 > [10473.255118] blk_update_request: I/O error, dev sdz, sector 1950184 > [10473.255219] device-mapper: multipath: Failing path 65:144. > [10473.262826] sd 20:0:0:2: [sdu] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10473.262921] sd 20:0:0:2: [sdu] tag#0 Sense Key : Not Ready [current] > [10473.262984] sd 20:0:0:2: [sdu] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10473.263052] sd 20:0:0:2: [sdu] tag#0 CDB: Read(10) 28 00 00 00 02 60 00 00 60 00 > [10473.263123] blk_update_request: I/O error, dev sdu, sector 608 > [10473.263268] device-mapper: multipath: Failing path 65:64. > [10473.264610] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10473.264682] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] > [10473.264744] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10473.264807] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10473.264861] blk_update_request: I/O error, dev sdk, sector 1952320 > [10473.265012] device-mapper: multipath: Failing path 8:160. > [10473.265457] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10473.265532] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] > [10473.265589] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10473.265648] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10473.265704] blk_update_request: I/O error, dev sdad, sector 1952320 > [10473.265790] device-mapper: multipath: Failing path 65:208. > [10473.272881] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10473.272984] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] > [10473.273069] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10473.273166] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 00 02 00 00 00 60 00 > [10473.273255] blk_update_request: I/O error, dev sdai, sector 512 > [10473.273479] device-mapper: multipath: Failing path 66:32. > [10473.273980] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10473.274079] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] > [10473.274168] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10473.274272] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 02 00 00 00 60 00 > [10473.274362] blk_update_request: I/O error, dev sdo, sector 512 > [10473.274532] device-mapper: multipath: Failing path 8:224. > [10473.286394] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10473.286470] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] > [10473.286533] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10473.286597] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d cb b0 00 00 50 00 > [10473.286658] blk_update_request: I/O error, dev sdas, sector 1952688 > [10473.287437] device-mapper: multipath: Failing path 66:192. > [10473.363750] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10473.363816] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] > [10473.363861] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10473.363908] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10473.363957] blk_update_request: I/O error, dev sdbk, sector 1952320 > [10473.364066] device-mapper: multipath: Failing path 67:224. > [10473.365091] sd 32:0:0:1: [sdar] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10473.365138] sd 32:0:0:1: [sdar] tag#0 Sense Key : Not Ready [current] > [10473.365175] sd 32:0:0:1: [sdar] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10473.365215] sd 32:0:0:1: [sdar] tag#0 CDB: Read(10) 28 00 00 1d ca 28 00 00 08 00 > [10473.365269] blk_update_request: I/O error, dev sdar, sector 1952296 > [10473.365347] device-mapper: multipath: Failing path 66:176. > [10474.346552] device-mapper: multipath: Failing path 65:160. > [10474.347094] device-mapper: multipath: Failing path 66:64. > [10474.354623] device-mapper: multipath: Failing path 65:176. > [10474.362549] device-mapper: multipath: Failing path 66:208. > [10474.363551] device-mapper: multipath: Failing path 68:16. > [10474.364351] device-mapper: multipath: Failing path 67:192. > [10474.369349] device-mapper: multipath: Failing path 68:0. > [10475.402882] device-mapper: multipath: Failing path 8:112. > [10475.409928] device-mapper: multipath: Failing path 8:96. > [10475.422390] device-mapper: multipath: Failing path 8:240. > [10475.423089] device-mapper: multipath: Failing path 66:80. > [10475.431476] device-mapper: multipath: Failing path 65:192. > [10475.437756] device-mapper: multipath: Failing path 66:160. > [10475.497531] device-mapper: multipath: Failing path 67:144. > [10476.475289] device-mapper: multipath: Failing path 66:48. > [10476.481673] device-mapper: multipath: Failing path 67:128. > [10476.487874] device-mapper: multipath: Failing path 67:64. > [10477.508134] device-mapper: multipath: Failing path 8:128. > [10477.508759] device-mapper: multipath: Failing path 65:80. > [10477.509813] device-mapper: multipath: Failing path 67:96. > [10477.514796] device-mapper: multipath: Failing path 8:176. > [10477.515948] device-mapper: multipath: Failing path 65:128. > [10478.553204] scsi_io_completion: 22 callbacks suppressed > [10478.553279] sd 12:0:0:2: [sdj] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10478.553345] sd 12:0:0:2: [sdj] tag#0 Sense Key : Not Ready [current] > [10478.553413] sd 12:0:0:2: [sdj] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10478.553482] sd 12:0:0:2: [sdj] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10478.553554] blk_update_request: 22 callbacks suppressed > [10478.553611] blk_update_request: I/O error, dev sdj, sector 1952320 > [10478.553731] device-mapper: multipath: Failing path 8:144. > [10478.554502] sd 18:0:0:2: [sdz] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10478.554579] sd 18:0:0:2: [sdz] tag#0 Sense Key : Not Ready [current] > [10478.554637] sd 18:0:0:2: [sdz] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10478.554700] sd 18:0:0:2: [sdz] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10478.554757] blk_update_request: I/O error, dev sdz, sector 1952320 > [10478.555100] device-mapper: multipath: Failing path 65:144. > [10478.560256] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10478.560329] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] > [10478.560385] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10478.560449] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10478.560503] blk_update_request: I/O error, dev sdk, sector 1952320 > [10478.560729] device-mapper: multipath: Failing path 8:160. > [10478.561184] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10478.561303] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] > [10478.561404] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10478.561504] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10478.561610] blk_update_request: I/O error, dev sdad, sector 1952320 > [10478.561831] device-mapper: multipath: Failing path 65:208. > [10478.562889] sd 20:0:0:2: [sdu] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10478.562993] sd 20:0:0:2: [sdu] tag#0 Sense Key : Not Ready [current] > [10478.563053] sd 20:0:0:2: [sdu] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10478.563116] sd 20:0:0:2: [sdu] tag#0 CDB: Read(10) 28 00 00 00 02 68 00 00 60 00 > [10478.563174] blk_update_request: I/O error, dev sdu, sector 616 > [10478.563274] device-mapper: multipath: Failing path 65:64. > [10478.563686] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10478.563771] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] > [10478.563848] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10478.563931] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 00 00 18 00 00 08 00 > [10478.564008] blk_update_request: I/O error, dev sdas, sector 24 > [10478.564132] device-mapper: multipath: Failing path 66:192. > [10478.570509] sd 32:0:0:1: [sdar] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10478.570583] sd 32:0:0:1: [sdar] tag#0 Sense Key : Not Ready [current] > [10478.570643] sd 32:0:0:1: [sdar] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10478.570705] sd 32:0:0:1: [sdar] tag#0 CDB: Read(10) 28 00 00 00 03 00 00 01 00 00 > [10478.570762] blk_update_request: I/O error, dev sdar, sector 768 > [10478.570853] device-mapper: multipath: Failing path 66:176. > [10478.571267] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10478.571333] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] > [10478.571400] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10478.571463] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 00 03 00 00 01 00 00 > [10478.571522] blk_update_request: I/O error, dev sdai, sector 768 > [10478.571624] device-mapper: multipath: Failing path 66:32. > [10478.577878] sd 16:0:0:1: [sdo] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10478.577956] sd 16:0:0:1: [sdo] tag#0 Sense Key : Not Ready [current] > [10478.578016] sd 16:0:0:1: [sdo] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10478.578078] sd 16:0:0:1: [sdo] tag#0 CDB: Read(10) 28 00 00 00 10 00 00 00 08 00 > [10478.578136] blk_update_request: I/O error, dev sdo, sector 4096 > [10478.578222] device-mapper: multipath: Failing path 8:224. > [10478.646314] sd 41:0:0:2: [sdbk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10478.646387] sd 41:0:0:2: [sdbk] tag#0 Sense Key : Not Ready [current] > [10478.646439] sd 41:0:0:2: [sdbk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10478.646491] sd 41:0:0:2: [sdbk] tag#0 CDB: Read(10) 28 00 00 1d cc f0 00 00 08 00 > [10478.646541] blk_update_request: I/O error, dev sdbk, sector 1953008 > [10478.646616] device-mapper: multipath: Failing path 67:224. > [10479.627635] device-mapper: multipath: Failing path 65:160. > [10479.628207] device-mapper: multipath: Failing path 66:64. > [10479.636021] device-mapper: multipath: Failing path 65:176. > [10479.643095] device-mapper: multipath: Failing path 66:208. > [10479.677320] device-mapper: multipath: Failing path 68:16. > [10479.677851] device-mapper: multipath: Failing path 67:192. > [10479.683638] device-mapper: multipath: Failing path 68:0. > [10480.677063] device-mapper: multipath: Failing path 8:112. > [10480.681958] device-mapper: multipath: Failing path 8:96. > [10480.683898] device-mapper: multipath: Failing path 8:240. > [10480.684200] device-mapper: multipath: Failing path 66:80. > [10480.685455] device-mapper: multipath: Failing path 67:144. > [10480.689543] device-mapper: multipath: Failing path 65:192. > [10480.694580] device-mapper: multipath: Failing path 66:160. > [10481.719022] device-mapper: multipath: Failing path 66:48. > [10481.729178] device-mapper: multipath: Failing path 67:128. > [10481.734260] device-mapper: multipath: Failing path 67:64. > [10482.756839] device-mapper: multipath: Failing path 8:128. > [10482.763827] device-mapper: multipath: Failing path 65:80. > [10482.773300] device-mapper: multipath: Failing path 65:128. > [10482.780948] device-mapper: multipath: Failing path 8:176. > [10483.833855] scsi_io_completion: 21 callbacks suppressed > [10483.833924] sd 24:0:0:1: [sdad] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10483.833996] sd 24:0:0:1: [sdad] tag#0 Sense Key : Not Ready [current] > [10483.834059] sd 24:0:0:1: [sdad] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10483.834123] sd 24:0:0:1: [sdad] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10483.834192] blk_update_request: 21 callbacks suppressed > [10483.834252] blk_update_request: I/O error, dev sdad, sector 1952320 > [10483.834482] device-mapper: multipath: Failing path 65:208. > [10483.835438] sd 31:0:0:1: [sdas] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10483.835506] sd 31:0:0:1: [sdas] tag#0 Sense Key : Not Ready [current] > [10483.835566] sd 31:0:0:1: [sdas] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10483.835631] sd 31:0:0:1: [sdas] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10483.835696] blk_update_request: I/O error, dev sdas, sector 1952320 > [10483.835964] device-mapper: multipath: Failing path 66:192. > [10483.842954] sd 26:0:0:1: [sdai] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10483.843039] sd 26:0:0:1: [sdai] tag#0 Sense Key : Not Ready [current] > [10483.843118] sd 26:0:0:1: [sdai] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10483.843199] sd 26:0:0:1: [sdai] tag#0 CDB: Read(10) 28 00 00 00 03 00 00 00 60 00 > [10483.843271] blk_update_request: I/O error, dev sdai, sector 768 > [10483.843552] device-mapper: multipath: Failing path 66:32. > [10484.844848] sd 14:0:0:1: [sdk] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10484.844932] sd 14:0:0:1: [sdk] tag#0 Sense Key : Not Ready [current] > [10484.845002] sd 14:0:0:1: [sdk] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10484.845081] sd 14:0:0:1: [sdk] tag#0 CDB: Read(10) 28 00 00 1d ca 40 00 00 08 00 > [10484.845152] blk_update_request: I/O error, dev sdk, sector 1952320 > [10484.845737] device-mapper: multipath: Failing path 8:160. > [10509.568010] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kdmwork-254:2:358] > [10509.568088] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg > [10509.575101] irq event stamp: 15233868 > [10509.575158] hardirqs last enabled at (15233867): [] _raw_spin_unlock_irq+0x27/0x40 > [10509.575265] hardirqs last disabled at (15233868): [] apic_timer_interrupt+0x84/0x90 > [10509.575370] softirqs last enabled at (15233850): [] __do_softirq+0x1cb/0x230 > [10509.575836] softirqs last disabled at (15233743): [] irq_exit+0xa8/0xb0 > [10509.575960] CPU: 3 PID: 358 Comm: kdmwork-254:2 Tainted: G O L 4.4.6-debug+ #2 > [10509.576023] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10509.576086] task: ffff88040120af80 ti: ffff880400fc8000 task.ti: ffff880400fc8000 > [10509.576149] RIP: 0010:[] [] _raw_spin_unlock_irq+0x2f/0x40 > [10509.576263] RSP: 0018:ffff880400fcbcd8 EFLAGS: 00000246 > [10509.576321] RAX: ffff88040120af80 RBX: ffff8800694542c0 RCX: 0000000000000001 > [10509.576383] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88040120af80 > [10509.576440] RBP: ffff880400fcbce0 R08: ffff8800694543c0 R09: 0000000000000000 > [10509.576498] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88040caf2290 > [10509.576557] R13: ffff880069453a00 R14: ffff8803b724ab80 R15: 0000000000000000 > [10509.576611] FS: 0000000000000000(0000) GS:ffff88046f260000(0000) knlGS:0000000000000000 > [10509.576667] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10509.576720] CR2: 00007f8aae007000 CR3: 0000000001a0a000 CR4: 00000000001406e0 > [10509.576772] Stack: > [10509.576821] ffff880455f0dcd0 ffff880400fcbd48 ffffffff813dbe28 ffff88040120af80 > [10509.577057] ffff88040caf25d8 ffff8804203cefa8 ffff880455f0dd50 0000000000000046 > [10509.577310] ffff880455f0dd00 ffff880069453a00 ffff880069453a00 0000000000000282 > [10509.577552] Call Trace: > [10509.577606] [] scsi_request_fn+0x118/0x600 > [10509.577666] [] __blk_run_queue+0x2e/0x40 > [10509.577723] [] __elv_add_request+0x75/0x1f0 > [10509.577781] [] blk_insert_cloned_request+0x101/0x190 > [10509.577845] [] map_request+0x18e/0x210 [dm_mod] > [10509.577909] [] map_tio_request+0x1d/0x40 [dm_mod] > [10509.577969] [] kthread_worker_fn+0x7d/0x1a0 > [10509.578028] [] ? __init_kthread_worker+0x60/0x60 > [10509.578085] [] kthread+0xea/0x100 > [10509.578142] [] ? kthread_create_on_node+0x1f0/0x1f0 > [10509.578199] [] ret_from_fork+0x3f/0x70 > [10509.578255] [] ? kthread_create_on_node+0x1f0/0x1f0 > [10509.578310] Code: 00 00 48 89 e5 53 48 89 fb 48 8b 55 08 48 8d 7f 18 e8 26 cf b2 ff 48 89 df e8 8e 00 b3 ff e8 59 a8 b2 ff fb 65 ff 0d a1 2b a9 7e <5b> 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 > [10537.567212] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kdmwork-254:2:358] > [10537.567287] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg > [10537.573758] irq event stamp: 30645910 > [10537.573810] hardirqs last enabled at (30645909): [] _raw_spin_unlock_irq+0x27/0x40 > [10537.573918] hardirqs last disabled at (30645910): [] apic_timer_interrupt+0x84/0x90 > [10537.574020] softirqs last enabled at (30645860): [] __do_softirq+0x1cb/0x230 > [10537.574122] softirqs last disabled at (30645755): [] irq_exit+0xa8/0xb0 > [10537.574223] CPU: 3 PID: 358 Comm: kdmwork-254:2 Tainted: G O L 4.4.6-debug+ #2 > [10537.574279] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10537.574335] task: ffff88040120af80 ti: ffff880400fc8000 task.ti: ffff880400fc8000 > [10537.574390] RIP: 0010:[] [] _raw_spin_unlock_irq+0x2f/0x40 > [10537.574492] RSP: 0018:ffff880400fcbcd8 EFLAGS: 00000246 > [10537.574543] RAX: ffff88040120af80 RBX: ffff880444673440 RCX: 0000000000000001 > [10537.574594] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88040120af80 > [10537.574645] RBP: ffff880400fcbce0 R08: ffff880444673540 R09: 0000000000000000 > [10537.575221] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8804426a4520 > [10537.575274] R13: ffff880444672b80 R14: ffff88033fd9a358 R15: 0000000000000000 > [10537.575327] FS: 0000000000000000(0000) GS:ffff88046f260000(0000) knlGS:0000000000000000 > [10537.575382] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10537.575435] CR2: 00007f8aae007000 CR3: 0000000001a0a000 CR4: 00000000001406e0 > [10537.575487] Stack: > [10537.575536] ffff88040161d388 ffff880400fcbd48 ffffffff813dbe28 ffff88040120af80 > [10537.575783] ffff8804426a4868 ffff880415a49c20 ffff88040161d408 0000000000000046 > [10537.576029] ffff88040161d3b8 ffff880444672b80 ffff880444672b80 0000000000000282 > [10537.576270] Call Trace: > [10537.576324] [] scsi_request_fn+0x118/0x600 > [10537.576384] [] __blk_run_queue+0x2e/0x40 > [10537.576441] [] __elv_add_request+0x75/0x1f0 > [10537.576498] [] blk_insert_cloned_request+0x101/0x190 > [10537.576561] [] map_request+0x18e/0x210 [dm_mod] > [10537.576624] [] map_tio_request+0x1d/0x40 [dm_mod] > [10537.576685] [] kthread_worker_fn+0x7d/0x1a0 > [10537.576742] [] ? __init_kthread_worker+0x60/0x60 > [10537.576798] [] kthread+0xea/0x100 > [10537.576855] [] ? kthread_create_on_node+0x1f0/0x1f0 > [10537.576911] [] ret_from_fork+0x3f/0x70 > [10537.576967] [] ? kthread_create_on_node+0x1f0/0x1f0 > [10537.577022] Code: 00 00 48 89 e5 53 48 89 fb 48 8b 55 08 48 8d 7f 18 e8 26 cf b2 ff 48 89 df e8 8e 00 b3 ff e8 59 a8 b2 ff fb 65 ff 0d a1 2b a9 7e <5b> 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 > [10546.926940] INFO: rcu_sched self-detected stall on CPU > [10546.927120] 3-...: (6497 ticks this GP) idle=fb9/140000000000002/0 softirq=2044956/2045037 fqs=5414 > [10546.927219] (t=6500 jiffies g=219289 c=219288 q=7233211) > [10546.927401] Task dump for CPU 3: > [10546.927486] kdmwork-254:2 R running task 0 358 2 0x00000008 > [10546.927736] 0000000000000002 ffff88046f263848 ffffffff8108195f 0000000000000003 > [10546.928161] ffffffff81a40a40 ffff88046f263860 ffffffff81084742 0000000000000004 > [10546.928578] ffff88046f263890 ffffffff810b5e79 ffff88046f275400 ffffffff81a40a40 > [10546.928989] Call Trace: > [10546.929076] [] sched_show_task+0xbf/0x150 > [10546.929253] [] dump_cpu_task+0x32/0x40 > [10546.929346] [] rcu_dump_cpu_stacks+0x89/0xe0 > [10546.929438] [] rcu_check_callbacks+0x439/0x730 > [10546.929529] [] ? cpuacct_account_field+0xd9/0x1b0 > [10546.929621] [] ? cpuacct_charge+0x1a0/0x1a0 > [10546.929715] [] update_process_times+0x34/0x60 > [10546.929807] [] tick_sched_handle.isra.18+0x20/0x50 > [10546.929899] [] tick_sched_timer+0x38/0x70 > [10546.929991] [] ? tick_sched_do_timer+0x30/0x30 > [10546.930083] [] __hrtimer_run_queues+0xa5/0x1c0 > [10546.930176] [] hrtimer_interrupt+0xa6/0x1b0 > [10546.930267] [] ? trace_hardirqs_off+0xd/0x10 > [10546.930360] [] smp_trace_apic_timer_interrupt+0x63/0x90 > [10546.930451] [] smp_apic_timer_interrupt+0x9/0x10 > [10546.930544] [] apic_timer_interrupt+0x89/0x90 > [10546.930641] [] ? _raw_spin_unlock_irqrestore+0x33/0x50 > [10546.930732] [] __slab_free+0xc6/0x270 > [10546.930835] [] ? call_rcu_sched+0x12/0x20 > [10546.930929] [] ? put_object+0x28/0x40 > [10546.931023] [] ? debug_check_no_locks_freed+0xb1/0x140 > [10546.931117] [] ? kiocb_free+0x32/0x40 > [10546.931209] [] kmem_cache_free+0x159/0x160 > [10546.931301] [] kiocb_free+0x32/0x40 > [10546.931395] [] aio_complete+0x1e5/0x3c0 > [10546.931488] [] dio_complete+0x75/0x1d0 > [10546.931582] [] dio_bio_end_aio+0x7a/0x130 > [10546.931675] [] bio_endio+0x3a/0x60 > [10546.931769] [] blk_update_request+0x7c/0x2a0 > [10546.931867] [] end_clone_bio+0x41/0x70 [dm_mod] > [10546.931961] [] bio_endio+0x3a/0x60 > [10546.932052] [] blk_update_request+0x7c/0x2a0 > [10546.932147] [] scsi_end_request+0x2e/0x1d0 > [10546.932238] [] scsi_io_completion+0xb4/0x610 > [10546.932333] [] scsi_finish_command+0xca/0x120 > [10546.932423] [] scsi_softirq_done+0x120/0x140 > [10546.932516] [] blk_done_softirq+0x72/0x90 > [10546.932607] [] __do_softirq+0x10f/0x230 > [10546.932700] [] irq_exit+0xa8/0xb0 > [10546.932792] [] do_IRQ+0x65/0x110 > [10546.932884] [] common_interrupt+0x89/0x89 > [10546.932973] [] ? _raw_spin_unlock_irq+0x2f/0x40 > [10546.933164] [] __multipath_map.isra.16+0x145/0x260 [dm_multipath] > [10546.933265] [] multipath_map+0x12/0x20 [dm_multipath] > [10546.933365] [] map_request+0x43/0x210 [dm_mod] > [10546.933464] [] map_tio_request+0x1d/0x40 [dm_mod] > [10546.933561] [] kthread_worker_fn+0x7d/0x1a0 > [10546.933653] [] ? __init_kthread_worker+0x60/0x60 > [10546.933747] [] kthread+0xea/0x100 > [10546.933841] [] ? kthread_create_on_node+0x1f0/0x1f0 > [10546.933934] [] ret_from_fork+0x3f/0x70 > [10546.934033] [] ? kthread_create_on_node+0x1f0/0x1f0 > [10573.566190] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kdmwork-254:2:358] > [10573.566286] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg > [10573.575752] irq event stamp: 50535115 > [10573.575821] hardirqs last enabled at (50535114): [] _raw_spin_unlock_irqrestore+0x31/0x50 > [10573.575959] hardirqs last disabled at (50535115): [] apic_timer_interrupt+0x84/0x90 > [10573.576115] softirqs last enabled at (50535044): [] __do_softirq+0x1cb/0x230 > [10573.576259] softirqs last disabled at (50535061): [] irq_exit+0xa8/0xb0 > [10573.576410] CPU: 3 PID: 358 Comm: kdmwork-254:2 Tainted: G O L 4.4.6-debug+ #2 > [10573.576491] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10573.576571] task: ffff88040120af80 ti: ffff880400fc8000 task.ti: ffff880400fc8000 > [10573.576650] RIP: 0010:[] [] _raw_spin_unlock_irqrestore+0x33/0x50 > [10573.576797] RSP: 0018:ffff88046f263b88 EFLAGS: 00000296 > [10573.576868] RAX: ffff88040120af80 RBX: 0000000000000296 RCX: 0000000000000000 > [10573.576938] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88040120af80 > [10573.577007] RBP: ffff88046f263b98 R08: ffff88046f263b08 R09: 0000000000000000 > [10573.577078] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880461584e00 > [10573.577162] R13: ffffea0009ec0400 R14: ffff880461584e00 R15: ffff880461488d80 > [10573.577240] FS: 0000000000000000(0000) GS:ffff88046f260000(0000) knlGS:0000000000000000 > [10573.577323] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10573.577399] CR2: 00007f8aae007000 CR3: 0000000001a0a000 CR4: 00000000001406e0 > [10573.577475] Stack: > [10573.577546] ffff880461584e00 0000000000000001 ffff88046f263c48 ffffffff811640d6 > [10573.577881] ffff88046f263bd8 ffff88040120af80 ffff880266ffaa80 ffff88027b011540 > [10573.578207] ffff88027b011540 ffff000109ec0400 0000000000000296 ffff88046f263bf0 > [10573.578538] Call Trace: > [10573.578608] > [10573.578690] [] __slab_free+0xc6/0x270 > [10573.578860] [] ? call_rcu_sched+0x12/0x20 > [10573.578947] [] ? put_object+0x28/0x40 > [10573.579029] [] ? debug_check_no_locks_freed+0xb1/0x140 > [10573.579114] [] ? dio_complete+0x84/0x1d0 > [10573.579203] [] kmem_cache_free+0x159/0x160 > [10573.579289] [] dio_complete+0x84/0x1d0 > [10573.579372] [] dio_bio_end_aio+0x7a/0x130 > [10573.579453] [] bio_endio+0x3a/0x60 > [10573.579534] [] blk_update_request+0x7c/0x2a0 > [10573.579622] [] end_clone_bio+0x41/0x70 [dm_mod] > [10573.579704] [] bio_endio+0x3a/0x60 > [10573.579784] [] blk_update_request+0x7c/0x2a0 > [10573.579866] [] scsi_end_request+0x2e/0x1d0 > [10573.579946] [] scsi_io_completion+0xb4/0x610 > [10573.580030] [] scsi_finish_command+0xca/0x120 > [10573.580118] [] scsi_softirq_done+0x120/0x140 > [10573.580210] [] blk_done_softirq+0x72/0x90 > [10573.580296] [] __do_softirq+0x10f/0x230 > [10573.580374] [] irq_exit+0xa8/0xb0 > [10573.580453] [] do_IRQ+0x65/0x110 > [10573.580535] [] common_interrupt+0x89/0x89 > [10573.580614] > [10573.580695] [] ? _raw_spin_unlock_irq+0x2f/0x40 > [10573.580853] [] __multipath_map.isra.16+0x145/0x260 [dm_multipath] > [10573.580942] [] multipath_map+0x12/0x20 [dm_multipath] > [10573.581032] [] map_request+0x43/0x210 [dm_mod] > [10573.581122] [] map_tio_request+0x1d/0x40 [dm_mod] > [10573.581205] [] kthread_worker_fn+0x7d/0x1a0 > [10573.581284] [] ? __init_kthread_worker+0x60/0x60 > [10573.581367] [] kthread+0xea/0x100 > [10573.581448] [] ? kthread_create_on_node+0x1f0/0x1f0 > [10573.581530] [] ret_from_fork+0x3f/0x70 > [10573.581609] [] ? kthread_create_on_node+0x1f0/0x1f0 > [10573.581691] Code: fc 48 83 c7 18 48 8b 55 08 53 48 89 f3 be 01 00 00 00 e8 e1 ce b2 ff 4c 89 e7 e8 49 00 b3 ff f6 c7 02 74 13 e8 0f a8 b2 ff 53 9d <65> ff 0d 56 2b a9 7e 5b 41 5c 5d c3 53 9d e8 6a 79 b2 ff eb eb > [10601.565455] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kdmwork-254:2:358] > [10601.565532] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg > [10601.572008] irq event stamp: 65965438 > [10601.572063] hardirqs last enabled at (65965437): [] restore_regs_and_iret+0x0/0x1d > [10601.572175] hardirqs last disabled at (65965438): [] apic_timer_interrupt+0x84/0x90 > [10601.572985] softirqs last enabled at (65965386): [] __do_softirq+0x1cb/0x230 > [10601.573093] softirqs last disabled at (65965185): [] irq_exit+0xa8/0xb0 > [10601.573195] CPU: 3 PID: 358 Comm: kdmwork-254:2 Tainted: G O L 4.4.6-debug+ #2 > [10601.573250] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10601.573304] task: ffff88040120af80 ti: ffff880400fc8000 task.ti: ffff880400fc8000 > [10601.573357] RIP: 0010:[] [] _raw_spin_unlock_irq+0x2f/0x40 > [10601.573457] RSP: 0018:ffff880400fcbcd8 EFLAGS: 00000246 > [10601.573507] RAX: ffff88040120af80 RBX: ffff8800694542c0 RCX: 0000000000000001 > [10601.573557] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88040120af80 > [10601.573609] RBP: ffff880400fcbce0 R08: ffff8800694543c0 R09: 0000000000000000 > [10601.573670] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88040caf2290 > [10601.573724] R13: ffff880069453a00 R14: ffff88020d55f230 R15: 0000000000000000 > [10601.573779] FS: 0000000000000000(0000) GS:ffff88046f260000(0000) knlGS:0000000000000000 > [10601.573835] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10601.573888] CR2: 00007f8aae007000 CR3: 0000000001a0a000 CR4: 00000000001406e0 > [10601.573941] Stack: > [10601.573990] ffff880455f0dcd0 ffff880400fcbd48 ffffffff813dbe28 ffff88040120af80 > [10601.574230] ffff88040caf25d8 ffff8804203cefa8 ffff880455f0dd50 0000000000000046 > [10601.574464] ffff880455f0dd00 ffff880069453a00 ffff880069453a00 0000000000000282 > [10601.574694] Call Trace: > [10601.574746] [] scsi_request_fn+0x118/0x600 > [10601.574804] [] __blk_run_queue+0x2e/0x40 > [10601.575744] [] __elv_add_request+0x75/0x1f0 > [10601.575810] [] blk_insert_cloned_request+0x101/0x190 > [10601.575876] [] map_request+0x18e/0x210 [dm_mod] > [10601.575941] [] map_tio_request+0x1d/0x40 [dm_mod] > [10601.576004] [] kthread_worker_fn+0x7d/0x1a0 > [10601.576063] [] ? __init_kthread_worker+0x60/0x60 > [10601.576122] [] kthread+0xea/0x100 > [10601.576181] [] ? kthread_create_on_node+0x1f0/0x1f0 > [10601.576240] [] ret_from_fork+0x3f/0x70 > [10601.576299] [] ? kthread_create_on_node+0x1f0/0x1f0 > [10601.576356] Code: 00 00 48 89 e5 53 48 89 fb 48 8b 55 08 48 8d 7f 18 e8 26 cf b2 ff 48 89 df e8 8e 00 b3 ff e8 59 a8 b2 ff fb 65 ff 0d a1 2b a9 7e <5b> 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 > [10629.564604] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kdmwork-254:2:358] > [10629.564704] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg > [10629.574344] irq event stamp: 81405685 > [10629.574414] hardirqs last enabled at (81405684): [] _raw_spin_unlock_irqrestore+0x31/0x50 > [10629.574557] hardirqs last disabled at (81405685): [] apic_timer_interrupt+0x84/0x90 > [10629.574713] softirqs last enabled at (81405466): [] __do_softirq+0x1cb/0x230 > [10629.574858] softirqs last disabled at (81405547): [] irq_exit+0xa8/0xb0 > [10629.574999] CPU: 3 PID: 358 Comm: kdmwork-254:2 Tainted: G O L 4.4.6-debug+ #2 > [10629.575074] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10629.575157] task: ffff88040120af80 ti: ffff880400fc8000 task.ti: ffff880400fc8000 > [10629.575235] RIP: 0010:[] [] _raw_spin_unlock_irqrestore+0x33/0x50 > [10629.575379] RSP: 0018:ffff88046f263b08 EFLAGS: 00000292 > [10629.575455] RAX: ffff88040120af80 RBX: 0000000000000292 RCX: 0000000000000000 > [10629.575529] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88040120af80 > [10629.575603] RBP: ffff88046f263b18 R08: ffff88046f263a88 R09: 0000000000000000 > [10629.575675] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880461585340 > [10629.575748] R13: ffffea000827d500 R14: ffff880461585340 R15: ffff880461489440 > [10629.575820] FS: 0000000000000000(0000) GS:ffff88046f260000(0000) knlGS:0000000000000000 > [10629.575909] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10629.575984] CR2: 00007f8aae007000 CR3: 0000000001a0a000 CR4: 00000000001406e0 > [10629.576062] Stack: > [10629.576134] ffff880461585340 0000000000000001 ffff88046f263bc8 ffffffff811640d6 > [10629.576472] ffff88046f263b58 ffff88040120af80 ffff880168a1fcd8 ffff880209f548c0 > [10629.576808] ffff880209f548c0 ffff00010827d500 0000000000000292 ffff88046f263b70 > [10629.577140] Call Trace: > [10629.577210] > [10629.577282] [] __slab_free+0xc6/0x270 > [10629.577430] [] ? call_rcu_sched+0x12/0x20 > [10629.577512] [] ? put_object+0x28/0x40 > [10629.577610] [] ? debug_check_no_locks_freed+0xb1/0x140 > [10629.577697] [] ? kiocb_free+0x32/0x40 > [10629.577781] [] kmem_cache_free+0x159/0x160 > [10629.577870] [] kiocb_free+0x32/0x40 > [10629.577953] [] aio_complete+0x1e5/0x3c0 > [10629.578036] [] dio_complete+0x75/0x1d0 > [10629.578118] [] dio_bio_end_aio+0x7a/0x130 > [10629.578200] [] bio_endio+0x3a/0x60 > [10629.578283] [] blk_update_request+0x7c/0x2a0 > [10629.578371] [] end_clone_bio+0x41/0x70 [dm_mod] > [10629.578465] [] bio_endio+0x3a/0x60 > [10629.578544] [] blk_update_request+0x7c/0x2a0 > [10629.578629] [] scsi_end_request+0x2e/0x1d0 > [10629.578710] [] scsi_io_completion+0xb4/0x610 > [10629.578791] [] scsi_finish_command+0xca/0x120 > [10629.578872] [] scsi_softirq_done+0x120/0x140 > [10629.578954] [] blk_done_softirq+0x72/0x90 > [10629.579034] [] __do_softirq+0x10f/0x230 > [10629.579117] [] irq_exit+0xa8/0xb0 > [10629.579197] [] do_IRQ+0x65/0x110 > [10629.579280] [] common_interrupt+0x89/0x89 > [10629.579357] > [10629.579428] [] ? _raw_spin_unlock_irq+0x2f/0x40 > [10629.579590] [] __multipath_map.isra.16+0x145/0x260 [dm_multipath] > [10629.579680] [] multipath_map+0x12/0x20 [dm_multipath] > [10629.579771] [] map_request+0x43/0x210 [dm_mod] > [10629.579860] [] map_tio_request+0x1d/0x40 [dm_mod] > [10629.579946] [] kthread_worker_fn+0x7d/0x1a0 > [10629.580032] [] ? __init_kthread_worker+0x60/0x60 > [10629.580114] [] kthread+0xea/0x100 > [10629.580195] [] ? kthread_create_on_node+0x1f0/0x1f0 > [10629.580276] [] ret_from_fork+0x3f/0x70 > [10629.580358] [] ? kthread_create_on_node+0x1f0/0x1f0 > [10629.580438] Code: fc 48 83 c7 18 48 8b 55 08 53 48 89 f3 be 01 00 00 00 e8 e1 ce b2 ff 4c 89 e7 e8 49 00 b3 ff f6 c7 02 74 13 e8 0f a8 b2 ff 53 9d <65> ff 0d 56 2b a9 7e 5b 41 5c 5d c3 53 9d e8 6a 79 b2 ff eb eb > [10657.563840] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kdmwork-254:2:358] > [10657.563911] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg > [10657.570523] irq event stamp: 96839370 > [10657.570577] hardirqs last enabled at (96839369): [] restore_regs_and_iret+0x0/0x1d > [10657.570688] hardirqs last disabled at (96839370): [] apic_timer_interrupt+0x84/0x90 > [10657.570792] softirqs last enabled at (96839350): [] __do_softirq+0x1cb/0x230 > [10657.570900] softirqs last disabled at (96839343): [] irq_exit+0xa8/0xb0 > [10657.571007] CPU: 3 PID: 358 Comm: kdmwork-254:2 Tainted: G O L 4.4.6-debug+ #2 > [10657.571066] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10657.571124] task: ffff88040120af80 ti: ffff880400fc8000 task.ti: ffff880400fc8000 > [10657.571180] RIP: 0010:[] [] _raw_spin_unlock_irq+0x2f/0x40 > [10657.571277] RSP: 0018:ffff880400fcbd50 EFLAGS: 00000246 > [10657.571328] RAX: ffff88040120af80 RBX: ffff8803fa3a6928 RCX: 0000000000000001 > [10657.571378] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88040120af80 > [10657.571427] RBP: ffff880400fcbd58 R08: ffff880408393a80 R09: 0000000000000000 > [10657.571485] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000001000 > [10657.571536] R13: ffff8802b3be3c70 R14: ffff88042dbaa358 R15: ffff8803fa3a6900 > [10657.571587] FS: 0000000000000000(0000) GS:ffff88046f260000(0000) knlGS:0000000000000000 > [10657.571641] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10657.571691] CR2: 00007f8aae007000 CR3: 0000000001a0a000 CR4: 00000000001406e0 > [10657.571742] Stack: > [10657.571791] ffff8803fa3a6928 ffff880400fcbdb0 ffffffffa010f815 ffff880407cf26c8 > [10657.572021] ffff880408393a80 0000000000000000 0000000000000000 ffff8802b3be3c28 > [10657.572251] ffff880037cf2610 ffff88042d0fd668 ffffc900030d5040 ffff8802b3be3c48 > [10657.572485] Call Trace: > [10657.572540] [] __multipath_map.isra.16+0x145/0x260 [dm_multipath] > [10657.572613] [] multipath_map+0x12/0x20 [dm_multipath] > [10657.572685] [] map_request+0x43/0x210 [dm_mod] > [10657.572753] [] map_tio_request+0x1d/0x40 [dm_mod] > [10657.572818] [] kthread_worker_fn+0x7d/0x1a0 > [10657.572883] [] ? __init_kthread_worker+0x60/0x60 > [10657.572943] [] kthread+0xea/0x100 > [10657.573005] [] ? kthread_create_on_node+0x1f0/0x1f0 > [10657.573072] [] ret_from_fork+0x3f/0x70 > [10657.573132] [] ? kthread_create_on_node+0x1f0/0x1f0 > [10657.573191] Code: 00 00 48 89 e5 53 48 89 fb 48 8b 55 08 48 8d 7f 18 e8 26 cf b2 ff 48 89 df e8 8e 00 b3 ff e8 59 a8 b2 ff fb 65 ff 0d a1 2b a9 7e <5b> 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 > [10693.562792] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [fio:7931] > [10693.562873] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg > [10693.569186] irq event stamp: 30263533 > [10693.569241] hardirqs last enabled at (30263532): [] _raw_spin_unlock_irqrestore+0x31/0x50 > [10693.569355] hardirqs last disabled at (30263533): [] apic_timer_interrupt+0x84/0x90 > [10693.570408] softirqs last enabled at (7684746): [] __do_softirq+0x1cb/0x230 > [10693.570522] softirqs last disabled at (7684923): [] irq_exit+0xa8/0xb0 > [10693.570633] CPU: 3 PID: 7931 Comm: fio Tainted: G O L 4.4.6-debug+ #2 > [10693.570693] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10693.570755] task: ffff880064ae2f80 ti: ffff8803f7578000 task.ti: ffff8803f7578000 > [10693.570816] RIP: 0010:[] [] _raw_spin_unlock_irqrestore+0x33/0x50 > [10693.570932] RSP: 0018:ffff88046f263d78 EFLAGS: 00000292 > [10693.570989] RAX: ffff880064ae2f80 RBX: 0000000000000292 RCX: 0000000000000000 > [10693.571047] RDX: 0000000000000003 RSI: ffff880064ae3718 RDI: ffff880064ae2f80 > [10693.571105] RBP: ffff88046f263d88 R08: ffff88046f263cf8 R09: 0000000000000000 > [10693.571165] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10693.571223] R13: ffffea00069c1e00 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10693.571280] FS: 00007f8a98e2f700(0000) GS:ffff88046f260000(0000) knlGS:0000000000000000 > [10693.571339] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10693.571397] CR2: 00007f8aae007000 CR3: 00000003f9db8000 CR4: 00000000001406e0 > [10693.571456] Stack: > [10693.571512] ffff88017fc01c00 0000000000000001 ffff88046f263e38 ffffffff811640d6 > [10693.571777] ffff88046f263dc8 ffffffff81001017 0000000100300030 ffff8801a707b7c8 > [10693.572047] ffff8801a707b7c8 ffff000164ae3718 0000000000000292 ffff880064ae3718 > [10693.572314] Call Trace: > [10693.572372] > [10693.572434] [] __slab_free+0xc6/0x270 > [10693.572557] [] ? trace_hardirqs_on_thunk+0x17/0x19 > [10693.572622] [] ? kmem_cache_free+0x9d/0x160 > [10693.572690] [] ? free_object_rcu+0x89/0x90 > [10693.572754] [] ? free_object_rcu+0x89/0x90 > [10693.572820] [] kmem_cache_free+0x159/0x160 > [10693.572884] [] ? memory_failure_work_func+0xb0/0xb0 > [10693.572948] [] free_object_rcu+0x89/0x90 > [10693.573013] [] rcu_process_callbacks+0x2b0/0x5f0 > [10693.574184] [] ? rcu_process_callbacks+0x26f/0x5f0 > [10693.574252] [] __do_softirq+0x10f/0x230 > [10693.574315] [] irq_exit+0xa8/0xb0 > [10693.574379] [] smp_trace_apic_timer_interrupt+0x68/0x90 > [10693.574443] [] smp_apic_timer_interrupt+0x9/0x10 > [10693.574509] [] apic_timer_interrupt+0x89/0x90 > [10693.574573] > [10693.574635] [] ? blk_flush_plug_list+0x224/0x240 > [10693.574759] [] blk_finish_plug+0x27/0x40 > [10693.574824] [] do_io_submit+0x3c6/0x770 > [10693.574887] [] ? do_io_submit+0x193/0x770 > [10693.574951] [] SyS_io_submit+0xb/0x10 > [10693.575014] [] entry_SYSCALL_64_fastpath+0x16/0x6f > [10693.575077] Code: fc 48 83 c7 18 48 8b 55 08 53 48 89 f3 be 01 00 00 00 e8 e1 ce b2 ff 4c 89 e7 e8 49 00 b3 ff f6 c7 02 74 13 e8 0f a8 b2 ff 53 9d <65> ff 0d 56 2b a9 7e 5b 41 5c 5d c3 53 9d e8 6a 79 b2 ff eb eb > [10700.274209] sd 37:0:0:2: [sdbc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [10700.274285] sd 37:0:0:2: [sdbc] tag#0 Sense Key : Not Ready [current] > [10700.274341] sd 37:0:0:2: [sdbc] tag#0 Add. Sense: Logical unit not accessible, target port in standby state > [10700.274400] sd 37:0:0:2: [sdbc] tag#0 CDB: Read(10) 28 00 00 00 00 00 00 00 08 00 > [10700.274455] blk_update_request: I/O error, dev sdbc, sector 0 > [10700.274586] device-mapper: multipath: Failing path 67:96. > [10725.561889] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [fio:7937] > [10725.561895] NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [fio:7930] > [10725.561907] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE > [10725.561907] NMI watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [disk11_0:32753] > [10725.561909] nf_nat_masquerade_ipv4 > [10725.561910] Modules linked in: > [10725.561910] iptable_nat > [10725.561911] ib_srpt(O-) > [10725.561912] nf_nat_ipv4 > [10725.561913] scst_vdisk(O) > [10725.561914] nf_nat > [10725.561914] libcrc32c > [10725.561915] nf_conntrack_ipv4 > [10725.561916] scst(O) > [10725.561917] nf_defrag_ipv4 > [10725.561917] dlm > [10725.561920] xt_conntrack > [10725.561920] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kdmwork-254:2:358] > [10725.561922] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [fio:7934] > [10725.561923] Modules linked in: > [10725.561924] Modules linked in: > [10725.561924] ib_srpt(O-) > [10725.561925] ib_srpt(O-) > [10725.561926] scst_vdisk(O) > [10725.561926] scst_vdisk(O) > [10725.561927] libcrc32c > [10725.561928] libcrc32c > [10725.561928] scst(O) > [10725.561931] scst(O) > [10725.561931] NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [fio:7935] > [10725.561932] dlm > [10725.561933] dlm > [10725.561933] netconsole > [10725.561934] netconsole > [10725.561934] configfs > [10725.561935] configfs > [10725.561936] Modules linked in: > [10725.561937] dm_queue_length > [10725.561938] ib_srpt(O-) > [10725.561939] brd > [10725.561939] scst_vdisk(O) > [10725.561940] nf_conntrack > [10725.561943] netconsole > [10725.561943] NMI watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [fio:7940] > [10725.561944] ipt_REJECT > [10725.561945] libcrc32c > [10725.561946] configfs > [10725.561947] xt_CHECKSUM > [10725.561948] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [fio:7938] > [10725.561949] Modules linked in: > [10725.561950] Modules linked in: > [10725.561950] ib_srpt(O-) > [10725.561951] dm_queue_length > [10725.561952] scst_vdisk(O) > [10725.561953] dm_queue_length > [10725.561954] xt_tcpudp > [10725.561954] ib_srpt(O-) > [10725.561955] libcrc32c > [10725.561956] scst_vdisk(O) > [10725.561957] scst(O) > [10725.561957] libcrc32c > [10725.561958] scst(O) > [10725.561959] dlm > [10725.561960] dlm > [10725.561961] brd > [10725.561961] netconsole > [10725.561962] tun > [10725.561963] netconsole > [10725.561963] scst(O) > [10725.561964] ebtable_filter > [10725.561965] xt_CHECKSUM > [10725.561966] dlm > [10725.561966] configfs > [10725.561967] iptable_mangle > [10725.561968] netconsole > [10725.561969] configfs > [10725.561969] configfs > [10725.561970] dm_queue_length > [10725.561971] dm_queue_length > [10725.561971] ipt_MASQUERADE > [10725.561972] dm_queue_length > [10725.561973] brd > [10725.561974] brd > [10725.561974] brd > [10725.561975] brd > [10725.561976] iptable_mangle > [10725.561976] xt_CHECKSUM > [10725.561977] ipt_MASQUERADE > [10725.561978] xt_CHECKSUM > [10725.561979] xt_CHECKSUM > [10725.561980] nf_nat_masquerade_ipv4 > [10725.561980] nf_nat_masquerade_ipv4 > [10725.561981] iptable_mangle > [10725.561982] iptable_nat > [10725.561982] iptable_mangle > [10725.561983] ebtables > [10725.561984] nf_nat_ipv4 > [10725.561985] ipt_MASQUERADE > [10725.561986] iptable_nat > [10725.561986] ip6table_filter > [10725.561987] nf_nat_ipv4 > [10725.561988] iptable_mangle > [10725.561989] ip6_tables > [10725.561990] xt_CHECKSUM > [10725.561991] ipt_MASQUERADE > [10725.561991] iptable_filter > [10725.561992] nf_nat_masquerade_ipv4 > [10725.561993] nf_nat > [10725.561993] ipt_MASQUERADE > [10725.561994] iptable_mangle > [10725.561995] nf_nat_masquerade_ipv4 > [10725.561996] ip_tables > [10725.561996] nf_nat > [10725.561997] nf_nat_masquerade_ipv4 > [10725.561998] x_tables > [10725.561999] ipt_MASQUERADE > [10725.561999] iptable_nat > [10725.562000] iptable_nat > [10725.562001] nf_nat_ipv4 > [10725.562001] af_packet > [10725.562002] nf_conntrack_ipv4 > [10725.562003] nf_nat_ipv4 > [10725.562004] nf_defrag_ipv4 > [10725.562004] iptable_nat > [10725.562005] nf_conntrack_ipv4 > [10725.562007] nf_nat_ipv4 > [10725.562008] nf_defrag_ipv4 > [10725.562009] nf_nat_masquerade_ipv4 > [10725.562010] xt_conntrack > [10725.562011] xt_conntrack > [10725.562011] bridge > [10725.562012] nf_conntrack > [10725.562013] iptable_nat > [10725.562014] nf_nat > [10725.562014] stp > [10725.562015] nf_conntrack_ipv4 > [10725.562016] nf_nat > [10725.562016] nf_nat > [10725.562017] nf_conntrack > [10725.562018] nf_conntrack_ipv4 > [10725.562019] ipt_REJECT > [10725.562019] nf_defrag_ipv4 > [10725.562020] xt_tcpudp > [10725.562021] ipt_REJECT > [10725.562021] nf_conntrack_ipv4 > [10725.562022] llc > [10725.562023] xt_conntrack > [10725.562024] tun > [10725.562024] nf_conntrack > [10725.562025] ebtable_filter > [10725.562026] nf_nat_ipv4 > [10725.562026] iscsi_ibft > [10725.562027] ebtables > [10725.562028] nf_nat > [10725.562029] nf_defrag_ipv4 > [10725.562029] ip6table_filter > [10725.562030] ipt_REJECT > [10725.562031] ip6_tables > [10725.562031] xt_tcpudp > [10725.562032] iptable_filter > [10725.562033] tun > [10725.562033] iscsi_boot_sysfs > [10725.562034] ebtable_filter > [10725.562035] ip_tables > [10725.562036] xt_tcpudp > [10725.562036] nf_conntrack_ipv4 > [10725.562037] xt_conntrack > [10725.562038] ebtables > [10725.562039] x_tables > [10725.562039] nf_conntrack > [10725.562040] ib_srp > [10725.562041] ipt_REJECT > [10725.562042] tun > [10725.562042] af_packet > [10725.562043] ip6table_filter > [10725.562044] xt_tcpudp > [10725.562045] scsi_transport_srp > [10725.562045] nf_defrag_ipv4 > [10725.562046] ib_ipoib > [10725.562047] xt_conntrack > [10725.562048] ip6_tables > [10725.562048] bridge > [10725.562049] iptable_filter > [10725.562050] rdma_ucm > [10725.562051] tun > [10725.562051] ib_ucm > [10725.562052] stp > [10725.562053] ebtable_filter > [10725.562054] llc > [10725.562054] ip_tables > [10725.562055] iscsi_ibft > [10725.562056] x_tables > [10725.562056] iscsi_boot_sysfs > [10725.562057] ebtables > [10725.562058] ib_srp > [10725.562058] af_packet > [10725.562059] scsi_transport_srp > [10725.562060] bridge > [10725.562061] ib_ipoib > [10725.562061] ebtable_filter > [10725.562062] rdma_ucm > [10725.562063] stp > [10725.562063] ib_ucm > [10725.562064] llc > [10725.562064] nf_conntrack > [10725.562065] nf_defrag_ipv4 > [10725.562066] ip6table_filter > [10725.562067] iscsi_ibft > [10725.562067] ib_uverbs > [10725.562068] iscsi_boot_sysfs > [10725.562069] ib_uverbs > [10725.562069] xt_conntrack > [10725.562070] msr > [10725.562071] ip6_tables > [10725.562072] ebtables > [10725.562073] ipt_REJECT > [10725.562074] ip6table_filter > [10725.562074] ib_umad > [10725.562075] iptable_filter > [10725.562076] nf_conntrack > [10725.562076] xt_tcpudp > [10725.562077] ipt_REJECT > [10725.562078] tun > [10725.562079] rdma_cm > [10725.562079] ib_srp > [10725.562080] msr > [10725.562081] scsi_transport_srp > [10725.562081] ib_umad > [10725.562082] ib_ipoib > [10725.562083] rdma_cm > [10725.562083] ebtable_filter > [10725.562084] xt_tcpudp > [10725.562085] ib_cm > [10725.562087] ebtables > [10725.562087] ip_tables > [10725.562088] iw_cm > [10725.562089] x_tables > [10725.562089] ib_cm > [10725.562090] ip6_tables > [10725.562091] af_packet > [10725.562092] tun > [10725.562092] ip6table_filter > [10725.562093] mlx4_ib > [10725.562093] rdma_ucm > [10725.562094] ib_sa > [10725.562095] iw_cm > [10725.562096] ib_ucm > [10725.562096] ib_mad > [10725.562097] iptable_filter > [10725.562098] ebtable_filter > [10725.562099] bridge > [10725.562099] ip6_tables > [10725.562100] ebtables > [10725.562101] iptable_filter > [10725.562101] ip6table_filter > [10725.562102] ip_tables > [10725.562103] ip6_tables > [10725.562103] ib_uverbs > [10725.562104] mlx4_ib > [10725.562105] x_tables > [10725.562106] iptable_filter > [10725.562106] af_packet > [10725.562107] ip_tables > [10725.562107] bridge > [10725.562108] x_tables > [10725.562109] stp > [10725.562109] ib_sa > [10725.562110] msr > [10725.562111] stp > [10725.562112] af_packet > [10725.562112] llc > [10725.562113] bridge > [10725.562114] iscsi_ibft > [10725.562115] ip_tables > [10725.562115] ib_umad > [10725.562116] ib_core > [10725.562116] rdma_cm > [10725.562117] ib_addr > [10725.562118] ib_mad > [10725.562119] x86_pkg_temp_thermal > [10725.562120] x_tables > [10725.562120] intel_powerclamp > [10725.562121] ib_cm > [10725.562121] coretemp > [10725.562122] stp > [10725.562123] llc > [10725.562123] kvm_intel > [10725.562124] iw_cm > [10725.562125] kvm > [10725.562125] mlx4_ib > [10725.562126] irqbypass > [10725.562127] ib_core > [10725.562127] crct10dif_pclmul > [10725.562128] ib_sa > [10725.562129] crc32_pclmul > [10725.562129] ib_mad > [10725.562130] crc32c_intel > [10725.562131] ib_core > [10725.562132] af_packet > [10725.562132] iscsi_ibft > [10725.562133] ib_addr > [10725.562134] ib_addr > [10725.562134] aesni_intel > [10725.562135] x86_pkg_temp_thermal > [10725.562135] tg3 > [10725.562136] intel_powerclamp > [10725.562137] iTCO_wdt > [10725.562137] x86_pkg_temp_thermal > [10725.562138] coretemp > [10725.562139] aes_x86_64 > [10725.562140] iscsi_boot_sysfs > [10725.562140] bridge > [10725.562141] lrw > [10725.562141] kvm_intel > [10725.562142] ipmi_ssif > [10725.562143] kvm > [10725.562143] ipmi_devintf > [10725.562144] irqbypass > [10725.562145] iscsi_boot_sysfs > [10725.562145] llc > [10725.562146] ib_srp > [10725.562147] ib_srp > [10725.562147] ptp > [10725.562148] crct10dif_pclmul > [10725.562148] iTCO_vendor_support > [10725.562149] crc32_pclmul > [10725.562150] gf128mul > [10725.562150] crc32c_intel > [10725.562151] stp > [10725.562152] scsi_transport_srp > [10725.562153] scsi_transport_srp > [10725.562153] iscsi_ibft > [10725.562154] ib_ipoib > [10725.562155] intel_powerclamp > [10725.562156] ib_ipoib > [10725.562156] iscsi_boot_sysfs > [10725.562157] rdma_ucm > [10725.562158] ib_srp > [10725.562158] rdma_ucm > [10725.562159] scsi_transport_srp > [10725.562160] glue_helper > [10725.562161] ib_ucm > [10725.562161] ib_ipoib > [10725.562162] ib_ucm > [10725.562163] coretemp > [10725.562164] mlx4_core > [10725.562164] llc > [10725.562165] ib_uverbs > [10725.562166] kvm_intel > [10725.562167] aesni_intel > [10725.562167] dcdbas > [10725.562168] tg3 > [10725.562168] ablk_helper > [10725.562169] iTCO_wdt > [10725.562169] pps_core > [10725.562171] aes_x86_64 > [10725.562172] kvm > [10725.562173] lrw > [10725.562173] cryptd > [10725.562174] msr > [10725.562175] rdma_ucm > [10725.562176] libphy > [10725.562176] irqbypass > [10725.562177] ib_uverbs > [10725.562178] ib_ucm > [10725.562178] msr > [10725.562179] ib_uverbs > [10725.562180] ib_umad > [10725.562181] crct10dif_pclmul > [10725.562181] ib_umad > [10725.562182] crc32_pclmul > [10725.562183] ipmi_ssif > [10725.562183] fjes > [10725.562184] ipmi_devintf > [10725.562185] 8250_fintek > [10725.562185] ptp > [10725.562186] sb_edac > [10725.562186] iTCO_vendor_support > [10725.562187] pcspkr > [10725.562188] rdma_cm > [10725.562189] crc32c_intel > [10725.562189] rdma_cm > [10725.562190] iscsi_ibft > [10725.562191] aesni_intel > [10725.562192] ib_cm > [10725.562192] msr > [10725.562193] ib_cm > [10725.562194] ib_umad > [10725.562194] iw_cm > [10725.562195] rdma_cm > [10725.562196] mlx4_ib > [10725.562196] iscsi_boot_sysfs > [10725.562197] iw_cm > [10725.562198] tg3 > [10725.562199] gf128mul > [10725.562199] ipmi_si > [10725.562200] ib_sa > [10725.562201] mlx4_ib > [10725.562202] ib_srp > [10725.562202] edac_core > [10725.562203] glue_helper > [10725.562204] ipmi_msghandler > [10725.562204] mlx4_core > [10725.562205] acpi_power_meter > [10725.562206] ib_cm > [10725.562206] ib_mad > [10725.562207] iw_cm > [10725.562208] scsi_transport_srp > [10725.562208] tpm_tis > [10725.562209] dcdbas > [10725.562210] iTCO_wdt > [10725.562211] ib_sa > [10725.562211] aes_x86_64 > [10725.562212] ib_mad > [10725.562213] ib_ipoib > [10725.562214] lrw > [10725.562214] ib_core > [10725.562215] tpm > [10725.562216] ablk_helper > [10725.562216] mlx4_ib > [10725.562217] ib_core > [10725.562218] ib_sa > [10725.562219] ib_addr > [10725.562219] pps_core > [10725.562220] lpc_ich > [10725.562221] rdma_ucm > [10725.562221] ib_mad > [10725.562222] ib_addr > [10725.562223] ib_core > [10725.562223] x86_pkg_temp_thermal > [10725.562224] cryptd > [10725.562225] ipmi_ssif > [10725.562226] libphy > [10725.562226] button > [10725.562227] fjes > [10725.562228] intel_powerclamp > [10725.562228] ib_addr > [10725.562229] coretemp > [10725.562230] x86_pkg_temp_thermal > [10725.562231] shpchp > [10725.562231] x86_pkg_temp_thermal > [10725.562232] ipmi_devintf > [10725.562233] intel_powerclamp > [10725.562234] ib_ucm > [10725.562234] mei_me > [10725.562235] 8250_fintek > [10725.562236] ptp > [10725.562237] kvm_intel > [10725.562238] iTCO_vendor_support > [10725.562239] coretemp > [10725.562239] intel_powerclamp > [10725.562240] mei > [10725.562241] sb_edac > [10725.562241] mfd_core > [10725.562242] pcspkr > [10725.562243] kvm_intel > [10725.562244] ipmi_si > [10725.562244] wmi > [10725.562245] edac_core > [10725.562245] dm_multipath > [10725.562246] ipmi_msghandler > [10725.562247] dm_mod > [10725.562247] acpi_power_meter > [10725.562248] hid_generic > [10725.562248] tpm_tis > [10725.562249] usbhid > [10725.562249] tpm > [10725.562250] mgag200 > [10725.562252] lpc_ich > [10725.562253] ib_uverbs > [10725.562253] kvm > [10725.562254] coretemp > [10725.562255] kvm > [10725.562255] kvm_intel > [10725.562256] gf128mul > [10725.562257] irqbypass > [10725.562258] glue_helper > [10725.562258] crct10dif_pclmul > [10725.562259] button > [10725.562260] i2c_algo_bit > [10725.562260] shpchp > [10725.562261] crc32_pclmul > [10725.562262] kvm > [10725.562262] irqbypass > [10725.562263] irqbypass > [10725.562264] crct10dif_pclmul > [10725.562264] crc32c_intel > [10725.562265] msr > [10725.562266] aesni_intel > [10725.562267] ib_umad > [10725.562268] tg3 > [10725.562268] mei_me > [10725.562269] iTCO_wdt > [10725.562270] mei > [10725.562270] drm_kms_helper > [10725.562271] mfd_core > [10725.562271] syscopyarea > [10725.562272] wmi > [10725.562273] sysfillrect > [10725.562274] rdma_cm > [10725.562274] mlx4_core > [10725.562275] ib_cm > [10725.562276] aes_x86_64 > [10725.562276] iw_cm > [10725.562277] dm_multipath > [10725.562278] lrw > [10725.562279] dcdbas > [10725.562279] dm_mod > [10725.562280] sysimgblt > [10725.562281] crc32_pclmul > [10725.562282] crct10dif_pclmul > [10725.562282] ipmi_ssif > [10725.562283] crc32_pclmul > [10725.562284] crc32c_intel > [10725.562285] ablk_helper > [10725.562285] aesni_intel > [10725.562286] crc32c_intel > [10725.562287] tg3 > [10725.562287] ipmi_devintf > [10725.562288] hid_generic > [10725.562289] fb_sys_fops > [10725.562289] usbhid > [10725.562290] pps_core > [10725.562291] mgag200 > [10725.562291] ptp > [10725.562292] i2c_algo_bit > [10725.562293] ttm > [10725.562293] aesni_intel > [10725.562294] iTCO_wdt > [10725.562295] tg3 > [10725.562295] aes_x86_64 > [10725.562296] drm > [10725.562297] drm_kms_helper > [10725.562297] ehci_pci > [10725.562298] syscopyarea > [10725.562299] sr_mod > [10725.562299] iTCO_vendor_support > [10725.562300] cryptd > [10725.562301] cdrom > [10725.562301] gf128mul > [10725.562302] mlx4_ib > [10725.562303] ehci_hcd > [10725.562304] sysfillrect > [10725.562304] ib_sa > [10725.562305] iTCO_wdt > [10725.562306] lrw > [10725.562307] sysimgblt > [10725.562307] usbcore > [10725.562308] ipmi_ssif > [10725.562309] aes_x86_64 > [10725.562309] ipmi_devintf > [10725.562310] lrw > [10725.562311] ptp > [10725.562311] libphy > [10725.562312] ib_mad > [10725.562313] iTCO_vendor_support > [10725.562313] ipmi_ssif > [10725.562314] gf128mul > [10725.562315] ipmi_devintf > [10725.562315] glue_helper > [10725.562316] ptp > [10725.562317] glue_helper > [10725.562318] iTCO_vendor_support > [10725.562318] mlx4_core > [10725.562319] gf128mul > [10725.562319] dcdbas > [10725.562320] glue_helper > [10725.562321] ablk_helper > [10725.562321] mlx4_core > [10725.562322] pps_core > [10725.562322] dcdbas > [10725.562323] cryptd > [10725.562325] fb_sys_fops > [10725.562326] fjes > [10725.562326] mlx4_core > [10725.562327] ttm > [10725.562328] usb_common > [10725.562328] drm > [10725.562329] sg > [10725.562329] ehci_pci > [10725.562329] > [10725.562331] 8250_fintek > [10725.562332] dcdbas > [10725.562332] irq event stamp: 11245005 > [10725.562333] libphy > [10725.562334] sb_edac > [10725.562335] ib_core > [10725.562335] fjes > [10725.562336] ablk_helper > [10725.562337] 8250_fintek > [10725.562337] pps_core > [10725.562338] sr_mod > [10725.562339] ablk_helper > [10725.562340] cryptd > [10725.562340] pcspkr > [10725.562341] cdrom > [10725.562342] hardirqs last enabled at (11245004): > [10725.562343] ehci_hcd > [10725.562343] sb_edac > [10725.562344] libphy > [10725.562350] pcspkr > [10725.562350] [] restore_regs_and_iret+0x0/0x1d > [10725.562351] usbcore > [10725.562352] hardirqs last disabled at (11245005): > [10725.562354] usb_common > [10725.562354] [] apic_timer_interrupt+0x84/0x90 > [10725.562355] sg > [10725.562356] ipmi_si > [10725.562357] softirqs last enabled at (8566716): > [10725.562357] > [10725.562358] fjes > [10725.562359] ipmi_si > [10725.562359] 8250_fintek > [10725.562360] edac_core > [10725.562361] sb_edac > [10725.562361] ipmi_msghandler > [10725.562362] ib_addr > [10725.562363] pps_core > [10725.562363] pcspkr > [10725.562364] acpi_power_meter > [10725.562367] ipmi_si > [10725.562367] [] __do_softirq+0x1cb/0x230 > [10725.562368] cryptd > [10725.562369] edac_core > [10725.562370] softirqs last disabled at (8566949): > [10725.562370] irq event stamp: 122720091 > [10725.562372] [] irq_exit+0xa8/0xb0 > [10725.562373] hardirqs last enabled at (122720090): > [10725.562375] x86_pkg_temp_thermal > [10725.562376] CPU: 7 PID: 7934 Comm: fio Tainted: G O L 4.4.6-debug+ #2 > [10725.562378] [] _raw_spin_unlock_irqrestore+0x31/0x50 > [10725.562379] ipmi_msghandler > [10725.562380] hardirqs last disabled at (122720091): > [10725.562381] libphy > [10725.562381] edac_core > [10725.562382] tpm_tis > [10725.562383] intel_powerclamp > [10725.562383] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10725.562385] tpm > [10725.562385] task: ffff8804426faf80 ti: ffff8803f7598000 task.ti: ffff8803f7598000 > [10725.562387] [] apic_timer_interrupt+0x84/0x90 > [10725.562389] RIP: 0010:[] > [10725.562391] softirqs last enabled at (119833606): > [10725.562391] [] _raw_spin_unlock_irqrestore+0x33/0x50 > [10725.562392] lpc_ich > [10725.562393] ipmi_msghandler > [10725.562394] button > [10725.562394] fjes > [10725.562396] acpi_power_meter > [10725.562396] [] __do_softirq+0x1cb/0x230 > [10725.562398] RSP: 0018:ffff88046f2e3d78 EFLAGS: 00000292 > [10725.562399] softirqs last disabled at (119835011): > [10725.562402] tpm_tis tpm > [10725.562402] [] irq_exit+0xa8/0xb0 > [10725.562404] 8250_fintek > [10725.562404] CPU: 1 PID: 358 Comm: kdmwork-254:2 Tainted: G O L 4.4.6-debug+ #2 > [10725.562405] shpchp > [10725.562406] acpi_power_meter > [10725.562406] mei_me > [10725.562407] tpm_tis > [10725.562407] mei > [10725.562408] tpm > [10725.562409] sb_edac > [10725.562410] lpc_ich > [10725.562411] pcspkr > [10725.562412] button > [10725.562412] RAX: ffff8804426faf80 RBX: 0000000000000292 RCX: 0000000000000000 > [10725.562414] ipmi_si > [10725.562414] RDX: 0000000000000003 RSI: ffff8804426fb718 RDI: ffff8804426faf80 > [10725.562415] edac_core > [10725.562417] shpchp > [10725.562417] RBP: ffff88046f2e3d88 R08: ffff88046f2e3cf8 R09: 0000000000000000 > [10725.562418] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10725.562419] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10725.562420] mfd_core > [10725.562421] lpc_ich > [10725.562421] wmi > [10725.562422] coretemp > [10725.562423] ipmi_msghandler > [10725.562424] R13: ffffea0009ac0400 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10725.562425] task: ffff88040120af80 ti: ffff880400fc8000 task.ti: ffff880400fc8000 > [10725.562426] FS: 00007f8a9be35700(0000) GS:ffff88046f2e0000(0000) knlGS:0000000000000000 > [10725.562428] RIP: 0010:[] > [10725.562429] mei_me > [10725.562429] dm_multipath > [10725.562430] button > [10725.562431] dm_mod > [10725.562432] kvm_intel > [10725.562433] [] _raw_spin_unlock_irqrestore+0x33/0x50 > [10725.562434] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10725.562435] RSP: 0018:ffff88046f223d78 EFLAGS: 00000292 > [10725.562436] kvm > [10725.562437] mei > [10725.562437] hid_generic > [10725.562438] shpchp > [10725.562439] usbhid > [10725.562439] mei_me > [10725.562440] mgag200 > [10725.562441] irqbypass > [10725.562441] RAX: ffff88040120af80 RBX: 0000000000000292 RCX: 0000000000000000 > [10725.562442] CR2: 00007ffeaad51d1c CR3: 00000003f9db8000 CR4: 00000000001406e0 > [10725.562443] mei > [10725.562444] i2c_algo_bit > [10725.562445] mfd_core > [10725.562446] mfd_core > [10725.562446] drm_kms_helper > [10725.562447] wmi > [10725.562448] syscopyarea > [10725.562449] RDX: 0000000000000003 RSI: ffff88040120b718 RDI: ffff88040120af80 > [10725.562449] Stack: > [10725.562450] RBP: ffff88046f223d88 R08: ffff88046f223cf8 R09: 0000000000000000 > [10725.562452] ffff88017fc01c00 > [10725.562452] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10725.562453] 0000000000000001 > [10725.562453] R13: ffffea000bac6e00 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10725.562455] ffff88046f2e3e38 > [10725.562455] FS: 0000000000000000(0000) GS:ffff88046f220000(0000) knlGS:0000000000000000 > [10725.562457] ffffffff811640d6 > [10725.562457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10725.562458] > [10725.562459] CR2: 00007ffadf28aa08 CR3: 00000003f9db8000 CR4: 00000000001406e0 > [10725.562460] acpi_power_meter > [10725.562461] wmi > [10725.562462] crct10dif_pclmul > [10725.562463] tpm_tis > [10725.562463] ffff88046f2e3dc8 > [10725.562463] Stack: > [10725.562464] 0000000000000246 > [10725.562465] ffff88017fc01c00 > [10725.562466] ffffffff811859b3 > [10725.562466] 0000000000000001 > [10725.562467] ffff88026b010aa0 > [10725.562467] ffff88046f223e38 > [10725.562467] > [10725.562469] crc32_pclmul > [10725.562470] tpm > [10725.562470] sysfillrect > [10725.562471] dm_multipath > [10725.562472] lpc_ich > [10725.562472] crc32c_intel > [10725.562473] dm_mod > [10725.562474] button > [10725.562475] ffff88026b010aa0 > [10725.562476] shpchp > [10725.562478] dm_multipath dm_mod > [10725.562478] mei_me > [10725.562479] aesni_intel > [10725.562480] ffff0001426fb718 > [10725.562480] ffffffff811640d6 > [10725.562481] 0000000000000292 > [10725.562482] sysimgblt > [10725.562482] hid_generic > [10725.562483] fb_sys_fops > [10725.562483] usbhid > [10725.562484] ttm > [10725.562485] mgag200 > [10725.562485] drm > [10725.562486] i2c_algo_bit > [10725.562486] ehci_pci > [10725.562487] hid_generic > [10725.562488] mei > [10725.562489] sr_mod > [10725.562489] > [10725.562490] ffff8804426fb718 > [10725.562491] cdrom > [10725.562492] drm_kms_helper > [10725.562492] ehci_hcd > [10725.562493] syscopyarea > [10725.562493] usbcore > [10725.562494] sysfillrect > [10725.562495] usb_common > [10725.562495] sysimgblt > [10725.562496] sg > [10725.562497] mfd_core > [10725.562497] fb_sys_fops > [10725.562497] > [10725.562499] ttm > [10725.562499] irq event stamp: 11251127 > [10725.562500] drm > [10725.562501] hardirqs last enabled at (11251126): > [10725.562501] ehci_pci > [10725.562505] wmi > [10725.562505] [] _raw_spin_unlock_irqrestore+0x31/0x50 > [10725.562506] > [10725.562507] ffff88046f223dc8 > [10725.562507] Call Trace: > [10725.562508] usbhid > [10725.562509] sr_mod > [10725.562510] hardirqs last disabled at (11251127): > [10725.562511] tg3 > [10725.562512] 0000000000000246 > [10725.562512] > [10725.562513] 0000000100300030 > [10725.562514] dm_multipath > [10725.562516] cdrom > [10725.562516] [] apic_timer_interrupt+0x84/0x90 > [10725.562520] ehci_hcd > [10725.562520] [] __slab_free+0xc6/0x270 > [10725.562524] ffff8802eb1bf238 > [10725.562524] [] ? file_free_rcu+0x23/0x30 > [10725.562525] > [10725.562526] usbcore > [10725.562527] softirqs last enabled at (8589138): > [10725.562529] ffff8802eb1bf238 > [10725.562530] [] ? kmem_cache_free+0x9d/0x160 > [10725.562533] ffff00010120b718 > [10725.562533] [] ? free_object_rcu+0x89/0x90 > [10725.562536] 0000000000000292 > [10725.562536] [] __do_softirq+0x1cb/0x230 > [10725.562537] usb_common > [10725.562540] softirqs last disabled at (8589495): > [10725.562540] [] ? free_object_rcu+0x89/0x90 > [10725.562541] dm_mod > [10725.562544] mgag200 > [10725.562544] [] kmem_cache_free+0x159/0x160 > [10725.562545] ffff88040120b718 > [10725.562547] i2c_algo_bit drm_kms_helper > [10725.562547] > [10725.562549] [] ? memory_failure_work_func+0xb0/0xb0 > [10725.562551] iTCO_wdt > [10725.562552] sg > [10725.562552] [] irq_exit+0xa8/0xb0 > [10725.562553] > [10725.562556] aes_x86_64 > [10725.562556] CPU: 2 PID: 7938 Comm: fio Tainted: G O L 4.4.6-debug+ #2 > [10725.562557] hid_generic > [10725.562558] syscopyarea > [10725.562559] lrw > [10725.562559] irq event stamp: 11005237 > [10725.562560] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10725.562561] sysfillrect > [10725.562562] ipmi_ssif > [10725.562563] sysimgblt > [10725.562564] usbhid > [10725.562564] fb_sys_fops > [10725.562565] Call Trace: > [10725.562567] [] free_object_rcu+0x89/0x90 > [10725.562569] hardirqs last enabled at (11005236): > [10725.562569] task: ffff880400708000 ti: ffff8803f7af8000 task.ti: ffff8803f7af8000 > [10725.562571] [] _raw_spin_unlock_irqrestore+0x31/0x50 > [10725.562573] RIP: 0010:[] > [10725.562574] hardirqs last disabled at (11005237): > [10725.562575] [] _raw_spin_unlock_irqrestore+0x33/0x50 > [10725.562576] [] apic_timer_interrupt+0x84/0x90 > [10725.562578] mgag200 > [10725.562581] ipmi_devintf > [10725.562581] [] rcu_process_callbacks+0x2b0/0x5f0 > [10725.562583] > [10725.562584] ttm > [10725.562584] RSP: 0018:ffff88046f243d78 EFLAGS: 00000292 > [10725.562586] [] ? rcu_process_callbacks+0x26f/0x5f0 > [10725.562589] drm > [10725.562589] [] __do_softirq+0x10f/0x230 > [10725.562591] [] __slab_free+0xc6/0x270 > [10725.562592] ptp > [10725.562593] i2c_algo_bit > [10725.562594] RAX: ffff880400708000 RBX: 0000000000000292 RCX: 0000000000000000 > [10725.562595] softirqs last enabled at (8447496): > [10725.562596] ehci_pci > [10725.562598] drm_kms_helper > [10725.562598] [] ? kmem_cache_free+0x9d/0x160 > [10725.562600] [] irq_exit+0xa8/0xb0 > [10725.562602] [] ? free_object_rcu+0x89/0x90 > [10725.562605] [] smp_trace_apic_timer_interrupt+0x68/0x90 > [10725.562607] [] ? free_object_rcu+0x89/0x90 > [10725.562609] [] smp_apic_timer_interrupt+0x9/0x10 > [10725.562611] [] kmem_cache_free+0x159/0x160 > [10725.562613] [] apic_timer_interrupt+0x89/0x90 > [10725.562614] [] __do_softirq+0x1cb/0x230 > [10725.562615] RDX: 0000000000000003 RSI: ffff880400708798 RDI: ffff880400708000 > [10725.562617] softirqs last disabled at (8447895): > [10725.562617] RBP: ffff88046f243d88 R08: ffff88046f243cf8 R09: 0000000000000000 > [10725.562619] [] irq_exit+0xa8/0xb0 > [10725.562621] syscopyarea > [10725.562623] iTCO_vendor_support > [10725.562623] CPU: 8 PID: 7940 Comm: fio Tainted: G O L 4.4.6-debug+ #2 > [10725.562624] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10725.562625] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10725.562626] R13: ffffea000beaca00 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10725.562628] [] ? memory_failure_work_func+0xb0/0xb0 > [10725.562631] > [10725.562631] [] free_object_rcu+0x89/0x90 > [10725.562633] FS: 00007f8a99e31700(0000) GS:ffff88046f240000(0000) knlGS:0000000000000000 > [10725.562634] task: ffff8803fabddf00 ti: ffff8803f753c000 task.ti: ffff8803f753c000 > [10725.562635] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10725.562637] RIP: 0010:[] > [10725.562637] CR2: 00007f8aae007000 CR3: 00000003f9db8000 CR4: 00000000001406e0 > [10725.562639] [] _raw_spin_unlock_irqrestore+0x33/0x50 > [10725.562640] gf128mul > [10725.562641] sr_mod > [10725.562642] RSP: 0018:ffff88046f303d78 EFLAGS: 00000292 > [10725.562642] Stack: > [10725.562644] sysfillrect > [10725.562645] [] rcu_process_callbacks+0x2b0/0x5f0 > [10725.562648] [] ? blk_flush_plug_list+0x224/0x240 > [10725.562649] ffff88017fc01c00 > [10725.562649] RAX: ffff8803fabddf00 RBX: 0000000000000292 RCX: 0000000000000000 > [10725.562650] 0000000000000001 > [10725.562651] sysimgblt > [10725.562652] ffff88046f243e38 > [10725.562653] RDX: 0000000000000003 RSI: ffff8803fabde698 RDI: ffff8803fabddf00 > [10725.562654] ffffffff811640d6 > [10725.562654] RBP: ffff88046f303d88 R08: ffff88046f303cf8 R09: 0000000000000000 > [10725.562657] [] blk_queue_bio+0x1f1/0x300 > [10725.562658] cdrom > [10725.562658] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10725.562659] > [10725.562660] R13: ffffea0009b52400 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10725.562662] ffff88046f243dc8 > [10725.562662] FS: 00007f8a96e2b700(0000) GS:ffff88046f300000(0000) knlGS:0000000000000000 > [10725.562664] glue_helper > [10725.562664] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10725.562665] ffff88017fc07600 > [10725.562666] CR2: 00007fe88affadd0 CR3: 00000003f9db8000 CR4: 00000000001406e0 > [10725.562669] fb_sys_fops > [10725.562669] [] generic_make_request+0xd2/0x290 > [10725.562671] [] ? rcu_process_callbacks+0x26f/0x5f0 > [10725.562673] [] submit_bio+0x62/0x140 > [10725.562674] [] __do_softirq+0x10f/0x230 > [10725.562676] [] ? bio_set_pages_dirty+0x49/0x60 > [10725.562679] mlx4_core > [10725.562679] [] irq_exit+0xa8/0xb0 > [10725.562682] [] do_blockdev_direct_IO+0x2b68/0x34a0 > [10725.562684] [] smp_trace_apic_timer_interrupt+0x68/0x90 > [10725.562685] ehci_hcd > [10725.562686] 0000000100300030 > [10725.562687] usbcore > [10725.562689] ffff8802fab2b520 > [10725.562689] [] ? I_BDEV+0x10/0x10 > [10725.562690] [] smp_apic_timer_interrupt+0x9/0x10 > [10725.562692] [] __blockdev_direct_IO+0x3e/0x40 > [10725.562694] [] apic_timer_interrupt+0x89/0x90 > [10725.562694] > [10725.562695] Stack: > [10725.562696] dcdbas > [10725.562698] ttm > [10725.562699] usb_common > [10725.562699] drm > [10725.562700] ablk_helper > [10725.562701] ehci_pci > [10725.562702] sg > [10725.562702] sr_mod > [10725.562703] pps_core > [10725.562704] ffff88017fc01c00 > [10725.562704] > [10725.562707] > [10725.562707] [] blkdev_direct_IO+0x53/0x80 > [10725.562708] ffff8802fab2b520 > [10725.562709] 0000000000000001 > [10725.562710] ffff000100708798 > [10725.562711] cryptd > [10725.562714] cdrom > [10725.562714] [] generic_file_read_iter+0x48c/0x5c0 > [10725.562715] [] ? _raw_spin_unlock_irq+0x2f/0x40 > [10725.562717] ffff88046f303e38 > [10725.562717] 0000000000000292 > [10725.562718] ffffffff811640d6 > [10725.562721] libphy > [10725.562721] [] ? __lock_acquire+0x3be/0x1ab0 > [10725.562725] [] __multipath_map.isra.16+0x145/0x260 [dm_multipath] > [10725.562727] [] ? blkdev_write_iter+0x100/0x100 > [10725.562729] [] multipath_map+0x12/0x20 [dm_multipath] > [10725.562731] [] blkdev_read_iter+0x30/0x40 > [10725.562736] [] map_request+0x43/0x210 [dm_mod] > [10725.562739] [] aio_run_iocb+0x122/0x2a0 > [10725.562739] > [10725.562741] ffff880400708798 > [10725.562741] ehci_hcd > [10725.562742] ffff88046f303dc8 > [10725.562742] > [10725.562744] usbcore > [10725.562745] [] do_io_submit+0x2f6/0x770 > [10725.562749] [] map_tio_request+0x1d/0x40 [dm_mod] > [10725.562751] [] ? do_io_submit+0x193/0x770 > [10725.562754] [] kthread_worker_fn+0x7d/0x1a0 > [10725.562755] [] SyS_io_submit+0xb/0x10 > [10725.562756] Call Trace: > [10725.562757] 0000000000000246 > [10725.562758] usb_common > [10725.562759] ffffffff811859b3 > [10725.562759] irq event stamp: 10703453 > [10725.562761] ffff88026d492a80 > [10725.562762] > [10725.562764] fjes > [10725.562764] [] entry_SYSCALL_64_fastpath+0x16/0x6f > [10725.562766] [] ? __init_kthread_worker+0x60/0x60 > [10725.562767] Code: > [10725.562769] hardirqs last enabled at (10703452): > [10725.562769] [] kthread+0xea/0x100 > [10725.562773] sg > [10725.562773] [] __slab_free+0xc6/0x270 > [10725.562777] [] _raw_spin_unlock_irqrestore+0x31/0x50 > [10725.562777] > [10725.562779] 8250_fintek > [10725.562781] fc > [10725.562781] [] ? kthread_create_on_node+0x1f0/0x1f0 > [10725.562783] hardirqs last disabled at (10703453): > [10725.562784] ffff88026d492a80 > [10725.562785] 48 > [10725.562786] [] ret_from_fork+0x3f/0x70 > [10725.562788] 83 > [10725.562788] [] ? kthread_create_on_node+0x1f0/0x1f0 > [10725.562789] c7 > [10725.562792] sb_edac > [10725.562792] [] apic_timer_interrupt+0x84/0x90 > [10725.562792] > [10725.562794] Code: > [10725.562796] 18 > [10725.562796] [] ? kmem_cache_free+0x9d/0x160 > [10725.562797] ffff0001fabde698 > [10725.562798] 48 > [10725.562801] 0000000000000292 > [10725.562801] [] ? free_object_rcu+0x89/0x90 > [10725.562802] irq event stamp: 9765723 > [10725.562803] pcspkr > [10725.562806] ffff8803fabde698 > [10725.562806] [] ? free_object_rcu+0x89/0x90 > [10725.562807] ipmi_si > [10725.562807] > [10725.562809] [] kmem_cache_free+0x159/0x160 > [10725.562810] Call Trace: > [10725.562812] [] ? memory_failure_work_func+0xb0/0xb0 > [10725.562815] > [10725.562815] [] free_object_rcu+0x89/0x90 > [10725.562817] [] __slab_free+0xc6/0x270 > [10725.562818] 8b > [10725.562819] hardirqs last enabled at (9765722): > [10725.562820] edac_core > [10725.562820] 55 > [10725.562821] fc > [10725.562823] ipmi_msghandler > [10725.562824] [] rcu_process_callbacks+0x2b0/0x5f0 > [10725.562827] acpi_power_meter > [10725.562827] [] ? file_free_rcu+0x23/0x30 > [10725.562828] 08 > [10725.562829] 48 > [10725.562830] 53 > [10725.562830] 83 > [10725.562831] 48 > [10725.562834] c7 > [10725.562834] [] _raw_spin_unlock_irqrestore+0x31/0x50 > [10725.562835] 89 > [10725.562836] 18 > [10725.562837] tpm_tis > [10725.562838] hardirqs last disabled at (9765723): > [10725.562839] [] ? rcu_process_callbacks+0x26f/0x5f0 > [10725.562841] [] ? kmem_cache_free+0x9d/0x160 > [10725.562842] 48 > [10725.562843] f3 > [10725.562843] 8b > [10725.562846] be > [10725.562846] [] ? free_object_rcu+0x89/0x90 > [10725.562848] [] __do_softirq+0x10f/0x230 > [10725.562850] [] irq_exit+0xa8/0xb0 > [10725.562852] [] apic_timer_interrupt+0x84/0x90 > [10725.562853] softirqs last enabled at (8080444): > [10725.562856] softirqs last enabled at (8596058): > [10725.562856] [] __do_softirq+0x1cb/0x230 > [10725.562858] [] __do_softirq+0x1cb/0x230 > [10725.562859] softirqs last disabled at (8080721): > [10725.562860] softirqs last disabled at (8596459): > [10725.562862] tpm > [10725.562862] [] irq_exit+0xa8/0xb0 > [10725.562863] [] irq_exit+0xa8/0xb0 > [10725.562866] CPU: 6 PID: 7930 Comm: fio Tainted: G O L 4.4.6-debug+ #2 > [10725.562868] CPU: 4 PID: 7935 Comm: fio Tainted: G O L 4.4.6-debug+ #2 > [10725.562869] 55 > [10725.562869] 01 > [10725.562870] 08 > [10725.562871] 00 > [10725.562871] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10725.562872] 53 > [10725.562873] lpc_ich > [10725.562874] 48 > [10725.562874] task: ffff8803fbdd8000 ti: ffff8803f7524000 task.ti: ffff8803f7524000 > [10725.562875] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10725.562878] RIP: 0010:[] > [10725.562878] task: ffff880064ae4740 ti: ffff8803f031c000 task.ti: ffff8803f031c000 > [10725.562880] [] _raw_spin_unlock_irqrestore+0x33/0x50 > [10725.562882] 89 > [10725.562882] [] smp_trace_apic_timer_interrupt+0x68/0x90 > [10725.562884] 00 > [10725.562884] RSP: 0018:ffff88046f283d78 EFLAGS: 00000292 > [10725.562886] RIP: 0010:[] > [10725.562887] 00 > [10725.562887] f3 > [10725.562888] e8 > [10725.562888] be > [10725.562889] e1 > [10725.562889] 01 > [10725.562890] ce > [10725.562891] RAX: ffff8803fbdd8000 RBX: 0000000000000292 RCX: 0000000000000000 > [10725.562892] 00 > [10725.562892] b2 > [10725.562893] 00 > [10725.562894] button > [10725.562895] RDX: 0000000000000003 RSI: ffff8803fbdd8798 RDI: ffff8803fbdd8000 > [10725.562896] 00 > [10725.562896] RBP: ffff88046f283d88 R08: ffff88046f283cf8 R09: 0000000000000000 > [10725.562899] [] ? free_object_rcu+0x89/0x90 > [10725.562900] [] smp_apic_timer_interrupt+0x9/0x10 > [10725.562903] [] kmem_cache_free+0x159/0x160 > [10725.562905] shpchp > [10725.562905] [] apic_timer_interrupt+0x89/0x90 > [10725.562906] e8 > [10725.562908] ff > [10725.562908] [] _raw_spin_unlock_irqrestore+0x33/0x50 > [10725.562910] > [10725.562911] RSP: 0018:ffff88046f2c3d78 EFLAGS: 00000292 > [10725.562912] e1 > [10725.562912] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10725.562914] RAX: ffff880064ae4740 RBX: 0000000000000292 RCX: 0000000000000000 > [10725.562915] 4c > [10725.562915] ce > [10725.562916] 89 > [10725.562916] b2 > [10725.562917] e7 > [10725.562918] ff > [10725.562920] e8 > [10725.562920] [] ? memory_failure_work_func+0xb0/0xb0 > [10725.562922] [] ? _raw_spin_unlock_irq+0x2f/0x40 > [10725.562923] R13: ffffea000e1e9e00 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10725.562925] RDX: 0000000000000003 RSI: ffff880064ae4ed8 RDI: ffff880064ae4740 > [10725.562926] 4c > [10725.562927] 49 > [10725.562928] 89 > [10725.562928] FS: 00007f8a9b634700(0000) GS:ffff88046f280000(0000) knlGS:0000000000000000 > [10725.562932] mei_me > [10725.562932] [] get_request+0x447/0x8e0 > [10725.562933] e7 > [10725.562934] 00 > [10725.562934] e8 > [10725.562936] b3 > [10725.562937] [] ? blk_cleanup_queue+0x270/0x270 > [10725.562939] [] free_object_rcu+0x89/0x90 > [10725.562941] [] ? __delay+0x1/0x10 > [10725.562943] [] rcu_process_callbacks+0x2b0/0x5f0 > [10725.562947] [] ? wake_atomic_t_function+0x60/0x60 > [10725.562948] [] ? rcu_process_callbacks+0x26f/0x5f0 > [10725.562951] [] blk_queue_bio+0x111/0x300 > [10725.562952] [] __do_softirq+0x10f/0x230 > [10725.562955] [] generic_make_request+0xd2/0x290 > [10725.562956] RBP: ffff88046f2c3d88 R08: ffff88046f2c3cf8 R09: 0000000000000000 > [10725.562957] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10725.562959] mei > [10725.562959] CR2: 00007f8aae007000 CR3: 00000003f9db8000 CR4: 00000000001406e0 > [10725.562960] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10725.562961] Stack: > [10725.562963] [] submit_bio+0x62/0x140 > [10725.562965] mfd_core > [10725.562965] R13: ffffea0009dcea00 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10725.562966] ff > [10725.562967] 49 > [10725.562967] f6 > [10725.562968] 00 > [10725.562970] wmi > [10725.562970] [] irq_exit+0xa8/0xb0 > [10725.562972] [] ? bio_set_pages_dirty+0x49/0x60 > [10725.562973] ffff88017fc01c00 > [10725.562974] b3 > [10725.562976] dm_multipath > [10725.562976] FS: 00007f8a9862e700(0000) GS:ffff88046f2c0000(0000) knlGS:0000000000000000 > [10725.562979] 0000000000000001 > [10725.562980] [] do_blockdev_direct_IO+0x2b68/0x34a0 > [10725.562981] c7 > [10725.562983] ff > [10725.562983] [] smp_trace_apic_timer_interrupt+0x68/0x90 > [10725.562984] [] ? I_BDEV+0x10/0x10 > [10725.562986] [] smp_apic_timer_interrupt+0x9/0x10 > [10725.562988] [] __blockdev_direct_IO+0x3e/0x40 > [10725.562989] ffff88046f283e38 > [10725.562990] dm_mod > [10725.562990] f6 > [10725.562991] 02 > [10725.562991] c7 > [10725.562992] 74 > [10725.562993] 02 > [10725.562993] 13 > [10725.562994] ffffffff811640d6 > [10725.562995] e8 > [10725.562997] 74 > [10725.562997] [] blkdev_direct_IO+0x53/0x80 > [10725.562999] [] apic_timer_interrupt+0x89/0x90 > [10725.563001] [] generic_file_read_iter+0x48c/0x5c0 > [10725.563004] > [10725.563004] [] ? __lock_acquire+0x3be/0x1ab0 > [10725.563006] [] ? _raw_spin_unlock_irq+0x2f/0x40 > [10725.563007] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10725.563007] > [10725.563009] CR2: 00005633132c3fe8 CR3: 00000003f9db8000 CR4: 00000000001406e0 > [10725.563012] [] get_request+0x447/0x8e0 > [10725.563013] 13 > [10725.563013] 0f > [10725.563014] ffff88046f283dc8 > [10725.563015] a8 > [10725.563015] 0000000000000246 > [10725.563016] hid_generic > [10725.563016] Stack: > [10725.563018] ffffffff811859b3 > [10725.563018] e8 > [10725.563019] b2 > [10725.563019] 0f > [10725.563020] ff > [10725.563021] usbhid > [10725.563022] ffff880387a78d48 > [10725.563022] mgag200 > [10725.563023] 53 > [10725.563023] a8 > [10725.563024] 9d > [10725.563025] b2 > [10725.563026] <65> > [10725.563027] i2c_algo_bit > [10725.563029] ffff88017fc01c00 > [10725.563029] [] ? blk_cleanup_queue+0x270/0x270 > [10725.563030] ff > [10725.563031] ff > [10725.563031] 53 > [10725.563033] 0d > [10725.563033] [] ? blkdev_write_iter+0x100/0x100 > [10725.563034] [] ? __delay+0x1/0x10 > [10725.563036] [] blkdev_read_iter+0x30/0x40 > [10725.563039] [] ? wake_atomic_t_function+0x60/0x60 > [10725.563041] [] aio_run_iocb+0x122/0x2a0 > [10725.563043] [] blk_queue_bio+0x111/0x300 > [10725.563045] [] do_io_submit+0x2f6/0x770 > [10725.563047] [] generic_make_request+0xd2/0x290 > [10725.563049] [] ? do_io_submit+0x193/0x770 > [10725.563051] [] submit_bio+0x62/0x140 > [10725.563054] 0000000000000001 > [10725.563054] [] SyS_io_submit+0xb/0x10 > [10725.563056] [] ? bio_set_pages_dirty+0x49/0x60 > [10725.563056] > [10725.563057] 56 > [10725.563058] ffff880387a78d48 > [10725.563060] ffff88046f2c3e38 > [10725.563060] [] entry_SYSCALL_64_fastpath+0x16/0x6f > [10725.563062] [] do_blockdev_direct_IO+0x2b68/0x34a0 > [10725.563063] Code: > [10725.563064] ffffffff811640d6 > [10725.563065] drm_kms_helper > [10725.563067] fc > [10725.563067] [] ? I_BDEV+0x10/0x10 > [10725.563068] 48 > [10725.563069] syscopyarea > [10725.563069] > [10725.563070] 2b > [10725.563070] 9d > [10725.563071] a9 > [10725.563073] <65> > [10725.563073] [] __blockdev_direct_IO+0x3e/0x40 > [10725.563075] 83 > [10725.563075] ffff88046f2c3dc8 > [10725.563076] sysfillrect > [10725.563077] ff > [10725.563077] 7e > [10725.563078] 0d > [10725.563079] ffff0001fbdd8798 > [10725.563079] 56 > [10725.563081] 0000000000000292 > [10725.563081] [] blkdev_direct_IO+0x53/0x80 > [10725.563084] c7 > [10725.563084] [] generic_file_read_iter+0x48c/0x5c0 > [10725.563087] 18 > [10725.563087] [] ? __lock_acquire+0x3be/0x1ab0 > [10725.563089] 48 > [10725.563089] [] ? blkdev_write_iter+0x100/0x100 > [10725.563091] 8b > [10725.563091] [] blkdev_read_iter+0x30/0x40 > [10725.563093] 5b > [10725.563094] sysimgblt > [10725.563096] 0000000000000246 0000000100300030 > [10725.563096] 41 > [10725.563097] 2b > [10725.563098] 5c > [10725.563098] a9 > [10725.563099] 5d > [10725.563101] ffff8802773ab7c8 > [10725.563101] [] aio_run_iocb+0x122/0x2a0 > [10725.563102] > [10725.563103] fb_sys_fops > [10725.563104] 55 > [10725.563105] ffff8802773ab7c8 > [10725.563105] ttm > [10725.563106] ffff000164ae4ed8 > [10725.563107] ffff8803fbdd8798 > [10725.563108] 08 > [10725.563109] 0000000000000292 > [10725.563109] drm > [10725.563109] > [10725.563111] ffff880064ae4ed8 > [10725.563111] Call Trace: > [10725.563112] ehci_pci > [10725.563113] > [10725.563115] 53 > [10725.563115] [] do_io_submit+0x2f6/0x770 > [10725.563117] sr_mod > [10725.563117] [] ? do_io_submit+0x193/0x770 > [10725.563118] > [10725.563119] c3 > [10725.563120] Call Trace: > [10725.563121] cdrom > [10725.563121] 53 > [10725.563122] 7e > [10725.563125] 9d > [10725.563125] [] __slab_free+0xc6/0x270 > [10725.563126] 5b > [10725.563126] e8 > [10725.563127] 41 > [10725.563127] 6a > [10725.563128] 5c > [10725.563128] 79 > [10725.563131] 5d > [10725.563131] [] ? file_free_rcu+0x23/0x30 > [10725.563133] [] SyS_io_submit+0xb/0x10 > [10725.563135] [] ? kmem_cache_free+0x9d/0x160 > [10725.563137] [] entry_SYSCALL_64_fastpath+0x16/0x6f > [10725.563138] 48 > [10725.563139] c3 > [10725.563140] b2 > [10725.563140] 53 > [10725.563141] ff > [10725.563141] 89 > [10725.563142] Code: > [10725.563143] f3 > [10725.563145] fc > [10725.563145] [] ? free_object_rcu+0x89/0x90 > [10725.563147] ehci_hcd > [10725.563148] 48 > [10725.563148] > [10725.563149] 9d > [10725.563149] eb > [10725.563150] e8 > [10725.563151] eb > [10725.563151] 6a > [10725.563151] > [10725.563154] 79 > [10725.563154] [] ? free_object_rcu+0x89/0x90 > [10725.563155] usbcore > [10725.563156] 83 > [10725.563156] be > [10725.563157] c7 > [10725.563158] b2 > [10725.563161] usb_common > [10725.563161] [] __slab_free+0xc6/0x270 > [10725.563162] 18 > [10725.563163] sg > [10725.563165] ff > [10725.563165] [] kmem_cache_free+0x159/0x160 > [10725.563166] 48 > [10725.563167] 01 > [10725.563167] 8b > [10725.563168] 00 > [10725.563168] 55 > [10725.563169] 00 > [10725.563172] 08 > [10725.563172] [] ? kmem_cache_free+0x9d/0x160 > [10725.563175] [] ? memory_failure_work_func+0xb0/0xb0 > [10725.563177] [] ? free_object_rcu+0x89/0x90 > [10725.563178] > [10725.563179] 00 > [10725.563180] 53 > [10725.563181] e8 > [10725.563181] irq event stamp: 11920625 > [10725.563182] eb > [10725.563183] hardirqs last enabled at (11920624): > [10725.563184] 48 > [10725.563186] e1 > [10725.563186] [] ? free_object_rcu+0x89/0x90 > [10725.563187] eb > [10725.563188] 89 > [10725.563189] ce > [10725.563189] > [10725.563190] b2 > [10725.563190] f3 > [10725.563191] ff > [10725.563192] be > [10725.563192] 4c > [10725.563193] 01 > [10725.563193] 89 > [10725.563194] 00 > [10725.563194] e7 > [10725.563195] 00 > [10725.563195] e8 > [10725.563196] 00 > [10725.563197] 49 > [10725.563197] e8 > [10725.563198] 00 > [10725.563198] e1 > [10725.563199] b3 > [10725.563201] ce > [10725.563201] [] kmem_cache_free+0x159/0x160 > [10725.563204] b2 > [10725.563204] [] ? memory_failure_work_func+0xb0/0xb0 > [10725.563207] [] _raw_spin_unlock_irqrestore+0x31/0x50 > [10725.563210] [] free_object_rcu+0x89/0x90 > [10725.563212] [] free_object_rcu+0x89/0x90 > [10725.563214] [] rcu_process_callbacks+0x2b0/0x5f0 > [10725.563215] ff > [10725.563216] ff > [10725.563217] f6 > [10725.563219] hardirqs last disabled at (11920625): > [10725.563219] [] rcu_process_callbacks+0x2b0/0x5f0 > [10725.563220] c7 > [10725.563222] 4c > [10725.563222] [] ? rcu_process_callbacks+0x26f/0x5f0 > [10725.563224] [] ? rcu_process_callbacks+0x26f/0x5f0 > [10725.563226] [] __do_softirq+0x10f/0x230 > [10725.563227] 02 > [10725.563227] 89 > [10725.563228] 74 > [10725.563230] e7 > [10725.563230] [] apic_timer_interrupt+0x84/0x90 > [10725.563232] [] irq_exit+0xa8/0xb0 > [10725.563233] e8 > [10725.563235] 13 > [10725.563235] [] smp_trace_apic_timer_interrupt+0x68/0x90 > [10725.563237] softirqs last enabled at (10133236): > [10725.563237] [] smp_apic_timer_interrupt+0x9/0x10 > [10725.563240] e8 > [10725.563240] [] apic_timer_interrupt+0x89/0x90 > [10725.563241] 49 > [10725.563243] 0f > [10725.563243] [] __do_softirq+0x10f/0x230 > [10725.563244] 00 > [10725.563245] a8 > [10725.563245] b3 > [10725.563246] > [10725.563247] ff > [10725.563247] b2 > [10725.563249] f6 > [10725.563249] [] ? _raw_spin_unlock_irq+0x2f/0x40 > [10725.563251] c7 > [10725.563251] [] irq_exit+0xa8/0xb0 > [10725.563280] [] scst_check_blocked_dev+0x77/0x1c0 [scst] > [10725.563281] 02 > [10725.563281] ff > [10725.563282] 74 > [10725.563283] 53 > [10725.563283] 13 > [10725.563284] 9d > [10725.563284] e8 > [10725.563287] <65> > [10725.563287] [] get_request+0x447/0x8e0 > [10725.563289] [] smp_trace_apic_timer_interrupt+0x68/0x90 > [10725.563291] softirqs last disabled at (10133267): > [10725.563292] [] smp_apic_timer_interrupt+0x9/0x10 > [10725.563293] [] irq_exit+0xa8/0xb0 > [10725.563296] 0f > [10725.563296] [] ? blk_cleanup_queue+0x270/0x270 > [10725.563298] [] apic_timer_interrupt+0x89/0x90 > [10725.563299] ff > [10725.563300] a8 > [10725.563302] 0d > [10725.563302] [] ? __delay+0x1/0x10 > [10725.563305] > [10725.563305] [] ? wake_atomic_t_function+0x60/0x60 > [10725.563307] CPU: 11 PID: 32753 Comm: disk11_0 Tainted: G O L 4.4.6-debug+ #2 > [10725.563308] 56 > [10725.563309] b2 > [10725.563311] 2b > [10725.563311] [] blk_queue_bio+0x111/0x300 > [10725.563312] [] ? _raw_spin_unlock_irq+0x2f/0x40 > [10725.563315] [] generic_make_request+0xd2/0x290 > [10725.563317] [] get_request+0x447/0x8e0 > [10725.563318] ff > [10725.563319] a9 > [10725.563319] 53 > [10725.563320] 7e > [10725.563322] 9d > [10725.563322] [] submit_bio+0x62/0x140 > [10725.563323] <65> > [10725.563323] 5b > [10725.563324] ff > [10725.563324] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10725.563326] [] ? bio_set_pages_dirty+0x49/0x60 > [10725.563327] 41 > [10725.563328] 0d > [10725.563328] 5c > [10725.563329] 56 > [10725.563329] 5d > [10725.563330] 2b > [10725.563331] c3 > [10725.563331] task: ffff880439375f00 ti: ffff880460d30000 task.ti: ffff880460d30000 > [10725.563332] 53 > [10725.563333] a9 > [10725.563335] 9d > [10725.563335] [] ? blk_cleanup_queue+0x270/0x270 > [10725.563337] RIP: 0010:[] > [10725.563339] 7e > [10725.563339] [] _raw_spin_unlock_irqrestore+0x33/0x50 > [10725.563341] [] ? __delay+0x1/0x10 > [10725.563342] RSP: 0018:ffff88046f363d68 EFLAGS: 00000292 > [10725.563344] [] do_blockdev_direct_IO+0x2b68/0x34a0 > [10725.563345] 5b > [10725.563348] e8 > [10725.563348] [] ? wake_atomic_t_function+0x60/0x60 > [10725.563349] [] ? I_BDEV+0x10/0x10 > [10725.563350] 41 > [10725.563351] 6a > [10725.563353] 5c > [10725.563353] [] __blockdev_direct_IO+0x3e/0x40 > [10725.563355] 5d > [10725.563355] [] blkdev_direct_IO+0x53/0x80 > [10725.563357] c3 > [10725.563359] 79 > [10725.563359] [] blk_queue_bio+0x111/0x300 > [10725.563361] [] generic_file_read_iter+0x48c/0x5c0 > [10725.563364] [] generic_make_request+0xd2/0x290 > [10725.563365] RAX: ffff880439375f00 RBX: 0000000000000292 RCX: 0000000000000000 > [10725.563367] [] ? __lock_acquire+0x3be/0x1ab0 > [10725.563368] 53 > [10725.563368] b2 > [10725.563369] 9d > [10725.563369] ff > [10725.563370] e8 > [10725.563371] eb > [10725.563373] 6a > [10725.563373] [] ? blkdev_write_iter+0x100/0x100 > [10725.563374] eb > [10725.563374] 79 > [10725.563375] > [10725.563377] b2 > [10725.563377] [] blkdev_read_iter+0x30/0x40 > [10725.563379] [] submit_bio+0x62/0x140 > [10725.563381] [] aio_run_iocb+0x122/0x2a0 > [10725.563382] RDX: 0000000000000003 RSI: ffff880439376698 RDI: ffff880439375f00 > [10725.563384] [] ? bio_set_pages_dirty+0x49/0x60 > [10725.563386] [] do_io_submit+0x2f6/0x770 > [10725.563388] [] do_blockdev_direct_IO+0x2b68/0x34a0 > [10725.563390] [] ? do_io_submit+0x193/0x770 > [10725.563391] RBP: ffff88046f363d78 R08: ffff88046f363ce8 R09: 0000000000000000 > [10725.563393] [] ? I_BDEV+0x10/0x10 > [10725.563394] [] SyS_io_submit+0xb/0x10 > [10725.563396] [] __blockdev_direct_IO+0x3e/0x40 > [10725.563399] ff > [10725.563399] [] entry_SYSCALL_64_fastpath+0x16/0x6f > [10725.563400] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10725.563401] eb > [10725.563403] Code: > [10725.563403] [] blkdev_direct_IO+0x53/0x80 > [10725.563405] fc > [10725.563405] R13: ffffea000f204e00 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10725.563408] [] generic_file_read_iter+0x48c/0x5c0 > [10725.563409] FS: 0000000000000000(0000) GS:ffff88046f360000(0000) knlGS:0000000000000000 > [10725.563410] 48 > [10725.563411] eb > [10725.563413] 83 > [10725.563413] [] ? __lock_acquire+0x3be/0x1ab0 > [10725.563415] c7 > [10725.563415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10725.563417] [] ? blkdev_write_iter+0x100/0x100 > [10725.563419] 18 > [10725.563419] [] blkdev_read_iter+0x30/0x40 > [10725.563420] 48 > [10725.563421] > [10725.563422] CR2: 00005633130dd1a8 CR3: 0000000401468000 CR4: 00000000001406e0 > [10725.563424] 8b > [10725.563424] [] aio_run_iocb+0x122/0x2a0 > [10725.563425] 55 > [10725.563425] Stack: > [10725.563427] 08 > [10725.563427] ffff88017fc01c00 > [10725.563429] 53 > [10725.563429] [] do_io_submit+0x2f6/0x770 > [10725.563430] 48 > [10725.563431] 0000000000000001 > [10725.563433] 89 > [10725.563433] [] ? do_io_submit+0x193/0x770 > [10725.563434] f3 > [10725.563435] ffff88046f363e30 > [10725.563437] be > [10725.563437] [] SyS_io_submit+0xb/0x10 > [10725.563438] 01 > [10725.563440] ffffffff811640d6 > [10725.563440] [] entry_SYSCALL_64_fastpath+0x16/0x6f > [10725.563441] 00 > [10725.563441] > [10725.563442] 00 > [10725.563443] Code: > [10725.563444] 00 > [10725.563445] fc > [10725.563445] ffff88046f363db8 > [10725.563446] e8 > [10725.563447] 48 > [10725.563447] e1 > [10725.563448] ffff88016d2e0000 > [10725.563449] 83 > [10725.563449] ce > [10725.563450] c7 > [10725.563451] b2 > [10725.563451] 0000000000000000 > [10725.563452] ff > [10725.563452] 18 > [10725.563453] 4c > [10725.563454] 48 > [10725.563454] 89 > [10725.563455] 8b > [10725.563456] ffff8803c813ef90 > [10725.563456] e7 > [10725.563457] 55 > [10725.563457] e8 > [10725.563458] 08 > [10725.563459] 49 > [10725.563459] 53 > [10725.563460] 00 > [10725.563460] 48 > [10725.563461] > [10725.563462] b3 > [10725.563462] ffff8803c813ef90 > [10725.563463] ff > [10725.563464] 89 > [10725.563464] f6 > [10725.563465] f3 > [10725.563466] c7 > [10725.563466] 000000018157940f > [10725.563467] be > [10725.563467] 0000000000000292 > [10725.563468] 02 > [10725.563469] 01 > [10725.563469] 74 > [10725.563470] ffff880439376698 > [10725.563470] 13 > [10725.563471] > [10725.563472] 00 > [10725.563472] e8 > [10725.563472] Call Trace: > [10725.563473] 0f > [10725.563474] 00 > [10725.563475] a8 > [10725.563475] > [10725.563478] b2 > [10725.563478] [] __slab_free+0xc6/0x270 > [10725.563479] ff > [10725.563480] 00 > [10725.563482] 53 > [10725.563482] [] ? kmem_cache_free+0x9d/0x160 > [10725.563485] 9d > [10725.563485] [] ? free_object_rcu+0x89/0x90 > [10725.563488] e8 > [10725.563488] [] ? free_object_rcu+0x89/0x90 > [10725.563489] e1 > [10725.563490] <65> > [10725.563492] ce > [10725.563492] [] kmem_cache_free+0x159/0x160 > [10725.563494] b2 > [10725.563494] ff > [10725.563497] ff > [10725.563497] [] ? memory_failure_work_func+0xb0/0xb0 > [10725.563498] 4c > [10725.563500] 0d > [10725.563500] [] free_object_rcu+0x89/0x90 > [10725.563503] 56 > [10725.563503] [] rcu_process_callbacks+0x2b0/0x5f0 > [10725.563505] 89 > [10725.563505] [] ? rcu_process_callbacks+0x26f/0x5f0 > [10725.563506] 2b > [10725.563508] e7 > [10725.563508] [] __do_softirq+0x10f/0x230 > [10725.563510] a9 > [10725.563510] [] irq_exit+0xa8/0xb0 > [10725.563511] 7e > [10725.563512] e8 > [10725.563512] 5b > [10725.563515] 49 > [10725.563515] [] do_IRQ+0x65/0x110 > [10725.563518] 41 > [10725.563518] [] ? mempool_free_slab+0x12/0x20 > [10725.563519] 5c > [10725.563521] 00 > [10725.563521] [] common_interrupt+0x89/0x89 > [10725.563523] b3 > [10725.563523] > [10725.563524] 5d > [10725.563526] ff > [10725.563526] [] ? mempool_free_slab+0x12/0x20 > [10725.563527] f6 > [10725.563530] c3 > [10725.563530] [] ? put_object+0x7/0x40 > [10725.563531] c7 > [10725.563533] 53 > [10725.563533] [] ? __delete_object+0x38/0x70 > [10725.563536] 9d > [10725.563536] [] delete_object_full+0x18/0x20 > [10725.563537] e8 > [10725.563540] 02 > [10725.563540] [] kmemleak_free+0x31/0x80 > [10725.563541] 74 > [10725.563542] 6a > [10725.563544] 13 > [10725.563544] [] kmem_cache_free+0x7a/0x160 > [10725.563545] 79 > [10725.563546] e8 > [10725.563548] b2 > [10725.563548] [] mempool_free_slab+0x12/0x20 > [10725.563550] 0f > [10725.563552] ff > [10725.563552] [] mempool_free+0x26/0x80 > [10725.563553] eb > [10725.563555] a8 > [10725.563555] [] bvec_free+0x31/0x40 > [10725.563556] eb > [10725.563556] b2 > [10725.563557] > [10725.563559] ff > [10725.563559] [] bio_free+0x34/0x60 > [10725.563561] 53 > [10725.563561] [] bio_put+0x1e/0x30 > [10725.563568] 9d > [10725.563568] [] blockio_endio+0x2d/0xe0 [scst_vdisk] > [10725.563570] <65> > [10725.563570] [] bio_endio+0x3a/0x60 > [10725.563573] ff > [10725.563573] [] brd_make_request+0xe8/0x1ae [brd] > [10725.563576] 0d > [10725.563576] [] ? generic_make_request_checks+0x10c/0x630 > [10725.563579] 56 > [10725.563579] [] generic_make_request+0xd2/0x290 > [10725.563581] 2b > [10725.563581] [] submit_bio+0x62/0x140 > [10725.563587] a9 7e > [10725.563587] [] blockio_exec_rw+0x553/0x670 [scst_vdisk] > [10725.563593] 5b 41 5c > [10725.563593] [] ? blockio_exec_write_verify+0x50/0x50 [scst_vdisk] > [10725.563598] 5d > [10725.563598] [] blockio_exec_read+0xd/0x20 [scst_vdisk] > [10725.563602] c3 > [10725.563602] [] vdev_do_job+0xb3/0x310 [scst_vdisk] > [10725.563607] 53 9d > [10725.563607] [] blockio_exec+0x74/0x180 [scst_vdisk] > [10725.563610] e8 > [10725.563610] [] ? trace_hardirqs_on_caller+0x121/0x1b0 > [10725.563632] 6a > [10725.563632] [] ? sgv_get_obj+0x72/0x1b0 [scst] > [10725.563649] 79 b2 ff > [10725.563649] [] scst_do_real_exec+0x94/0x3d0 [scst] > [10725.563664] eb > [10725.563664] [] ? scst_is_not_reservation_holder+0x13/0x60 [scst] > [10725.563679] eb > [10725.563679] [] ? __scst_check_local_events+0x19f/0x370 [scst] > [10725.563679] > [10725.563694] [] scst_exec_check_blocking+0xcc/0x340 [scst] > [10725.563709] [] scst_exec_check_sn+0x12d/0x2c0 [scst] > [10725.563711] [] ? finish_task_switch+0xa6/0x220 > [10725.563713] [] ? finish_task_switch+0x67/0x220 > [10725.563728] [] scst_process_active_cmd+0x3e2/0x760 [scst] > [10725.563730] [] ? trace_hardirqs_on+0xd/0x10 > [10725.563744] [] scst_do_job_active+0xe1/0x160 [scst] > [10725.563759] [] scst_cmd_thread+0x132/0x290 [scst] > [10725.563762] [] ? wake_atomic_t_function+0x60/0x60 > [10725.563776] [] ? scst_do_job_active+0x160/0x160 [scst] > [10725.563779] [] kthread+0xea/0x100 > [10725.563782] [] ? kthread_create_on_node+0x1f0/0x1f0 > [10725.563784] [] ret_from_fork+0x3f/0x70 > [10725.563785] [] ? kthread_create_on_node+0x1f0/0x1f0 > [10725.563814] Code: fc 48 83 c7 18 48 8b 55 08 53 48 89 f3 be 01 00 00 00 e8 e1 ce b2 ff 4c 89 e7 e8 49 00 b3 ff f6 c7 02 74 13 e8 0f a8 b2 ff 53 9d <65> ff 0d 56 2b a9 7e 5b 41 5c 5d c3 53 9d e8 6a 79 b2 ff eb eb > [10726.319704] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg > [10726.380711] irq event stamp: 16370765 > [10726.381076] hardirqs last enabled at (16370764): [] _raw_spin_unlock_irqrestore+0x31/0x50 > [10726.382350] hardirqs last disabled at (16370765): [] apic_timer_interrupt+0x84/0x90 > [10726.383582] softirqs last enabled at (13717672): [] __do_softirq+0x1cb/0x230 > [10726.384329] softirqs last disabled at (13717773): [] irq_exit+0xa8/0xb0 > [10726.384996] CPU: 0 PID: 7937 Comm: fio Tainted: G O L 4.4.6-debug+ #2 > [10726.385483] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10726.385685] task: ffff88040070df00 ti: ffff8803f1678000 task.ti: ffff8803f1678000 > [10726.386036] RIP: 0010:[] [] _raw_spin_unlock_irqrestore+0x33/0x50 > [10726.386575] RSP: 0018:ffff88046f203d78 EFLAGS: 00000292 > [10726.387228] RAX: ffff88040070df00 RBX: 0000000000000292 RCX: 0000000000000000 > [10726.387689] RDX: 0000000000000003 RSI: ffff88040070e698 RDI: ffff88040070df00 > [10726.388315] RBP: ffff88046f203d88 R08: ffff88046f203cf8 R09: 0000000000000000 > [10726.388558] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10726.389279] R13: ffffea000bef5800 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10726.389728] FS: 00007f8a9a632700(0000) GS:ffff88046f200000(0000) knlGS:0000000000000000 > [10726.390012] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10726.390373] CR2: 00007f8aae007000 CR3: 00000003f9db8000 CR4: 00000000001406f0 > [10726.390881] Stack: > [10726.391294] ffff88017fc01c00 0000000000000001 ffff88046f203e38 ffffffff811640d6 > [10726.393205] ffff88046f203dc8 0000000000000246 ffffffff811859b3 ffff8802fbd64d08 > [10726.394507] ffff8802fbd64d08 ffff00010070e698 0000000000000292 ffff88040070e698 > [10726.397125] Call Trace: > [10726.397494] > [10726.397750] [] __slab_free+0xc6/0x270 > [10726.398542] [] ? file_free_rcu+0x23/0x30 > [10726.398908] [] ? kmem_cache_free+0x9d/0x160 > [10726.399384] [] ? free_object_rcu+0x89/0x90 > [10726.399674] [] ? free_object_rcu+0x89/0x90 > [10726.399952] [] kmem_cache_free+0x159/0x160 > [10726.400402] [] ? memory_failure_work_func+0xb0/0xb0 > [10726.401001] [] free_object_rcu+0x89/0x90 > [10726.401417] [] rcu_process_callbacks+0x2b0/0x5f0 > [10726.402363] [] ? rcu_process_callbacks+0x26f/0x5f0 > [10726.402673] [] __do_softirq+0x10f/0x230 > [10726.403097] [] irq_exit+0xa8/0xb0 > [10726.403658] [] smp_trace_apic_timer_interrupt+0x68/0x90 > [10726.403861] [] smp_apic_timer_interrupt+0x9/0x10 > [10726.404301] [] apic_timer_interrupt+0x89/0x90 > [10726.404600] > [10726.405112] [] ? _raw_spin_unlock_irq+0x2f/0x40 > [10726.405768] [] get_request+0x447/0x8e0 > [10726.405980] [] ? blk_cleanup_queue+0x270/0x270 > [10726.407030] [] ? __delay+0x1/0x10 > [10726.407348] [] ? wake_atomic_t_function+0x60/0x60 > [10726.407859] [] blk_queue_bio+0x111/0x300 > [10726.408537] [] generic_make_request+0xd2/0x290 > [10726.409417] [] submit_bio+0x62/0x140 > [10726.409897] [] ? bio_set_pages_dirty+0x49/0x60 > [10726.410140] [] do_blockdev_direct_IO+0x2b68/0x34a0 > [10726.410347] [] ? I_BDEV+0x10/0x10 > [10726.410703] [] __blockdev_direct_IO+0x3e/0x40 > [10726.411311] [] blkdev_direct_IO+0x53/0x80 > [10726.412019] [] generic_file_read_iter+0x48c/0x5c0 > [10726.412290] [] ? __lock_acquire+0x3be/0x1ab0 > [10726.412748] [] ? blkdev_write_iter+0x100/0x100 > [10726.413086] [] blkdev_read_iter+0x30/0x40 > [10726.413349] [] aio_run_iocb+0x122/0x2a0 > [10726.413782] [] do_io_submit+0x2f6/0x770 > [10726.414050] [] ? do_io_submit+0x193/0x770 > [10726.414463] [] SyS_io_submit+0xb/0x10 > [10726.414818] [] entry_SYSCALL_64_fastpath+0x16/0x6f > [10726.415555] Code: fc 48 83 c7 18 48 8b 55 08 53 48 89 f3 be 01 00 00 00 e8 e1 ce b2 ff 4c 89 e7 e8 49 00 b3 ff f6 c7 02 74 13 e8 0f a8 b2 ff 53 9d <65> ff 0d 56 2b a9 7e 5b 41 5c 5d c3 53 9d e8 6a 79 b2 ff eb eb > [10753.561101] NMI watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [disk11_0:32753] > [10753.561110] NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [fio:7935] > [10753.561118] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs > [10753.561119] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [fio:7938] > [10753.561120] dm_queue_length > [10753.561121] Modules linked in: > [10753.561121] brd > [10753.561122] ib_srpt(O-) > [10753.561123] xt_CHECKSUM > [10753.561125] scst_vdisk(O) > [10753.561126] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kdmwork-254:2:358] > [10753.561127] libcrc32c > [10753.561128] iptable_mangle > [10753.561128] scst(O) > [10753.561129] ipt_MASQUERADE > [10753.561130] dlm > [10753.561130] Modules linked in: > [10753.561131] netconsole > [10753.561132] ib_srpt(O-) > [10753.561133] nf_nat_masquerade_ipv4 > [10753.561133] configfs > [10753.561134] iptable_nat > [10753.561135] scst_vdisk(O) > [10753.561136] nf_nat_ipv4 > [10753.561136] libcrc32c > [10753.561137] nf_nat > [10753.561138] scst(O) > [10753.561138] nf_conntrack_ipv4 > [10753.561141] dlm > [10753.561141] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [fio:7937] > [10753.561142] nf_defrag_ipv4 > [10753.561143] netconsole > [10753.561143] dm_queue_length > [10753.561144] Modules linked in: > [10753.561145] xt_conntrack > [10753.561146] ib_srpt(O-) > [10753.561147] brd > [10753.561147] scst_vdisk(O) > [10753.561148] nf_conntrack > [10753.561149] libcrc32c > [10753.561150] xt_CHECKSUM > [10753.561150] configfs > [10753.561151] ipt_REJECT > [10753.561152] scst(O) > [10753.561152] xt_tcpudp > [10753.561153] dlm > [10753.561154] tun > [10753.561155] netconsole > [10753.561155] dm_queue_length > [10753.561156] ebtable_filter > [10753.561157] brd > [10753.561157] ebtables > [10753.561158] iptable_mangle > [10753.561159] ip6table_filter > [10753.561160] configfs > [10753.561160] ipt_MASQUERADE > [10753.561161] dm_queue_length > [10753.561162] ip6_tables > [10753.561163] nf_nat_masquerade_ipv4 > [10753.561163] iptable_filter > [10753.561164] xt_CHECKSUM > [10753.561165] ip_tables > [10753.561165] brd > [10753.561166] iptable_nat > [10753.561167] iptable_mangle > [10753.561168] nf_nat_ipv4 > [10753.561168] xt_CHECKSUM > [10753.561169] ipt_MASQUERADE > [10753.561170] iptable_mangle > [10753.561171] x_tables > [10753.561171] ipt_MASQUERADE > [10753.561172] af_packet > [10753.561173] nf_nat_masquerade_ipv4 > [10753.561174] nf_nat_masquerade_ipv4 > [10753.561174] nf_nat > [10753.561175] iptable_nat > [10753.561176] iptable_nat > [10753.561177] bridge > [10753.561177] nf_nat_ipv4 > [10753.561178] nf_nat_ipv4 > [10753.561179] nf_nat > [10753.561179] nf_nat > [10753.561180] stp > [10753.561181] nf_conntrack_ipv4 > [10753.561181] llc > [10753.561182] nf_defrag_ipv4 > [10753.561183] nf_conntrack_ipv4 > [10753.561184] iscsi_ibft > [10753.561184] nf_defrag_ipv4 > [10753.561185] iscsi_boot_sysfs > [10753.561186] nf_conntrack_ipv4 > [10753.561187] xt_conntrack > [10753.561187] xt_conntrack > [10753.561188] nf_defrag_ipv4 > [10753.561189] ib_srp > [10753.561190] nf_conntrack > [10753.561190] xt_conntrack > [10753.561191] scsi_transport_srp > [10753.561192] nf_conntrack > [10753.561192] ipt_REJECT > [10753.561193] ib_ipoib > [10753.561194] nf_conntrack > [10753.561195] rdma_ucm > [10753.561195] xt_tcpudp > [10753.561196] ipt_REJECT > [10753.561197] ib_ucm > [10753.561197] tun > [10753.561198] ipt_REJECT > [10753.561199] xt_tcpudp > [10753.561199] ebtable_filter > [10753.561200] xt_tcpudp > [10753.561201] ib_uverbs > [10753.561201] tun > [10753.561202] ebtables > [10753.561203] ebtable_filter > [10753.561204] ip6table_filter > [10753.561204] ebtables > [10753.561205] tun > [10753.561206] msr > [10753.561206] ip6_tables > [10753.561207] ebtable_filter > [10753.561208] iptable_filter > [10753.561209] ebtables > [10753.561209] ib_umad > [10753.561210] ip_tables > [10753.561211] rdma_cm > [10753.561211] ip6table_filter > [10753.561212] ib_cm > [10753.561213] ip6_tables > [10753.561213] x_tables > [10753.561214] iptable_filter > [10753.561215] iw_cm > [10753.561215] af_packet > [10753.561216] mlx4_ib > [10753.561217] bridge > [10753.561218] ip6table_filter > [10753.561218] stp > [10753.561220] ip_tables > [10753.561221] ip6_tables > [10753.561222] ib_sa > [10753.561222] iptable_filter > [10753.561223] llc > [10753.561224] ib_mad > [10753.561224] ip_tables > [10753.561225] x_tables > [10753.561226] iscsi_ibft > [10753.561226] x_tables > [10753.561227] iscsi_boot_sysfs > [10753.561228] af_packet > [10753.561229] af_packet > [10753.561229] ib_srp > [10753.561230] bridge > [10753.561231] ib_core > [10753.561231] scsi_transport_srp > [10753.561232] ib_addr > [10753.561233] stp > [10753.561234] x86_pkg_temp_thermal > [10753.561234] llc > [10753.561235] ib_ipoib > [10753.561236] iscsi_ibft > [10753.561237] bridge > [10753.561237] iscsi_boot_sysfs > [10753.561238] intel_powerclamp > [10753.561239] ib_srp > [10753.561239] stp > [10753.561240] rdma_ucm > [10753.561241] llc > [10753.561241] coretemp > [10753.561242] ib_ucm > [10753.561243] kvm_intel > [10753.561243] iscsi_ibft > [10753.561244] ib_uverbs > [10753.561245] scsi_transport_srp > [10753.561246] msr > [10753.561246] iscsi_boot_sysfs > [10753.561247] ib_umad > [10753.561248] ib_srp > [10753.561249] ib_ipoib > [10753.561249] kvm > [10753.561250] rdma_ucm > [10753.561251] rdma_cm > [10753.561251] irqbypass > [10753.561252] ib_cm > [10753.561253] crct10dif_pclmul > [10753.561253] scsi_transport_srp > [10753.561254] crc32_pclmul > [10753.561255] iw_cm > [10753.561255] ib_ipoib > [10753.561256] crc32c_intel > [10753.561257] rdma_ucm > [10753.561258] mlx4_ib > [10753.561258] aesni_intel > [10753.561259] ib_sa > [10753.561260] tg3 > [10753.561260] ib_mad > [10753.561261] iTCO_wdt > [10753.561262] ib_ucm > [10753.561263] ib_core > [10753.561263] aes_x86_64 > [10753.561264] ib_uverbs > [10753.561265] lrw > [10753.561265] ib_ucm > [10753.561266] ib_addr > [10753.561267] ipmi_ssif > [10753.561268] x86_pkg_temp_thermal > [10753.561268] ipmi_devintf > [10753.561269] ib_uverbs > [10753.561270] ptp > [10753.561270] msr > [10753.561271] iTCO_vendor_support > [10753.561272] msr > [10753.561272] intel_powerclamp > [10753.561273] gf128mul > [10753.561274] coretemp > [10753.561275] ib_umad > [10753.561275] kvm_intel > [10753.561276] glue_helper > [10753.561277] rdma_cm > [10753.561277] mlx4_core > [10753.561278] ib_umad > [10753.561279] kvm > [10753.561279] dcdbas > [10753.561280] ib_cm > [10753.561281] ablk_helper > [10753.561281] irqbypass > [10753.561282] iw_cm > [10753.561283] rdma_cm > [10753.561283] pps_core > [10753.561284] crct10dif_pclmul > [10753.561285] cryptd > [10753.561285] crc32_pclmul > [10753.561286] mlx4_ib > [10753.561287] libphy > [10753.561287] ib_sa > [10753.561288] crc32c_intel > [10753.561289] fjes > [10753.561290] ib_mad > [10753.561290] aesni_intel > [10753.561291] ib_core > [10753.561292] 8250_fintek > [10753.561292] tg3 > [10753.561293] sb_edac > [10753.561294] ib_addr > [10753.561294] pcspkr > [10753.561295] iTCO_wdt > [10753.561296] ib_cm > [10753.561296] aes_x86_64 > [10753.561297] iw_cm > [10753.561298] lrw > [10753.561298] ipmi_si > [10753.561299] ipmi_ssif > [10753.561300] x86_pkg_temp_thermal > [10753.561301] ipmi_devintf > [10753.561301] mlx4_ib > [10753.561302] edac_core > [10753.561303] ptp > [10753.561303] ipmi_msghandler > [10753.561304] intel_powerclamp > [10753.561305] acpi_power_meter > [10753.561306] coretemp > [10753.561306] tpm_tis > [10753.561307] kvm_intel > [10753.561308] tpm > [10753.561308] ib_sa > [10753.561309] kvm > [10753.561310] ib_mad > [10753.561310] lpc_ich > [10753.561312] ib_core > [10753.561313] button > [10753.561313] irqbypass > [10753.561314] ib_addr > [10753.561315] iTCO_vendor_support > [10753.561315] shpchp > [10753.561316] gf128mul > [10753.561317] mei_me > [10753.561318] x86_pkg_temp_thermal > [10753.561318] glue_helper > [10753.561319] intel_powerclamp > [10753.561320] mei > [10753.561320] mlx4_core > [10753.561321] mfd_core > [10753.561322] coretemp > [10753.561322] wmi > [10753.561323] dcdbas > [10753.561324] kvm_intel > [10753.561324] ablk_helper > [10753.561325] kvm > [10753.561326] dm_multipath > [10753.561327] crct10dif_pclmul > [10753.561327] dm_mod > [10753.561328] pps_core > [10753.561329] hid_generic > [10753.561329] cryptd > [10753.561330] usbhid > [10753.561331] libphy > [10753.561331] mgag200 > [10753.561332] crc32_pclmul > [10753.561333] i2c_algo_bit > [10753.561334] crc32c_intel > [10753.561334] fjes > [10753.561335] irqbypass > [10753.561336] aesni_intel > [10753.561336] drm_kms_helper > [10753.561337] 8250_fintek > [10753.561338] syscopyarea > [10753.561338] sb_edac > [10753.561339] sysfillrect > [10753.561340] pcspkr > [10753.561340] tg3 > [10753.561341] crct10dif_pclmul > [10753.561342] ipmi_si > [10753.561343] iTCO_wdt > [10753.561343] edac_core > [10753.561344] crc32_pclmul > [10753.561345] ipmi_msghandler > [10753.561346] sysimgblt > [10753.561346] crc32c_intel > [10753.561347] fb_sys_fops > [10753.561348] aes_x86_64 > [10753.561348] ttm > [10753.561349] acpi_power_meter > [10753.561350] drm > [10753.561350] tpm_tis > [10753.561351] aesni_intel > [10753.561352] ehci_pci > [10753.561352] tpm > [10753.561353] lrw > [10753.561354] tg3 > [10753.561354] sr_mod > [10753.561355] lpc_ich > [10753.561356] cdrom > [10753.561356] iTCO_wdt > [10753.561357] button > [10753.561358] ehci_hcd > [10753.561359] ipmi_ssif > [10753.561359] aes_x86_64 > [10753.561360] shpchp > [10753.561361] usbcore > [10753.561361] ipmi_devintf > [10753.561362] usb_common > [10753.561363] lrw > [10753.561364] sg > [10753.561364] mei_me > [10753.561365] ipmi_ssif > [10753.561365] > [10753.561366] ptp > [10753.561367] irq event stamp: 10969989 > [10753.561368] mei > [10753.561369] hardirqs last enabled at (10969988): > [10753.561374] mfd_core > [10753.561375] [] _raw_spin_unlock_irqrestore+0x31/0x50 > [10753.561376] ipmi_devintf > [10753.561377] wmi > [10753.561377] iTCO_vendor_support > [10753.561378] hardirqs last disabled at (10969989): > [10753.561380] gf128mul > [10753.561381] [] apic_timer_interrupt+0x84/0x90 > [10753.561382] dm_multipath > [10753.561383] softirqs last enabled at (8596058): > [10753.561385] dm_mod > [10753.561386] [] __do_softirq+0x1cb/0x230 > [10753.561387] ptp > [10753.561388] glue_helper > [10753.561388] softirqs last disabled at (8596459): > [10753.561390] mlx4_core > [10753.561390] [] irq_exit+0xa8/0xb0 > [10753.561391] hid_generic > [10753.561392] dcdbas > [10753.561392] usbhid > [10753.561394] ablk_helper > [10753.561395] CPU: 4 PID: 7935 Comm: fio Tainted: G O L 4.4.6-debug+ #2 > [10753.561396] mgag200 > [10753.561397] pps_core > [10753.561397] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10753.561398] iTCO_vendor_support > [10753.561399] i2c_algo_bit > [10753.561400] cryptd > [10753.561400] task: ffff8803fbdd8000 ti: ffff8803f7524000 task.ti: ffff8803f7524000 > [10753.561401] libphy > [10753.561402] gf128mul > [10753.561403] drm_kms_helper > [10753.561404] RIP: 0010:[] > [10753.561405] fjes > [10753.561406] glue_helper > [10753.561406] [] _raw_spin_unlock_irqrestore+0x33/0x50 > [10753.561408] syscopyarea > [10753.561408] RSP: 0018:ffff88046f283d78 EFLAGS: 00000292 > [10753.561410] 8250_fintek > [10753.561410] RAX: ffff8803fbdd8000 RBX: 0000000000000292 RCX: 0000000000000000 > [10753.561412] sb_edac > [10753.561412] sysfillrect > [10753.561414] mlx4_core > [10753.561414] RDX: 0000000000000003 RSI: ffff8803fbdd8798 RDI: ffff8803fbdd8000 > [10753.561415] dcdbas > [10753.561416] RBP: ffff88046f283d88 R08: ffff88046f283cf8 R09: 0000000000000000 > [10753.561418] ablk_helper > [10753.561418] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10753.561419] pcspkr > [10753.561420] sysimgblt > [10753.561421] ipmi_si > [10753.561421] fb_sys_fops > [10753.561422] pps_core > [10753.561423] ttm > [10753.561424] edac_core > [10753.561424] R13: ffffea000c5d8000 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10753.561425] drm > [10753.561426] ipmi_msghandler > [10753.561426] ehci_pci > [10753.561428] acpi_power_meter > [10753.561428] FS: 00007f8a9b634700(0000) GS:ffff88046f280000(0000) knlGS:0000000000000000 > [10753.561430] sr_mod > [10753.561430] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10753.561431] cdrom > [10753.561432] cryptd > [10753.561432] CR2: 00007f8aae007000 CR3: 00000003f9db8000 CR4: 00000000001406e0 > [10753.561434] tpm_tis > [10753.561434] ehci_hcd > [10753.561435] tpm > [10753.561436] usbcore > [10753.561436] Stack: > [10753.561437] libphy > [10753.561438] usb_common > [10753.561438] ffff88017fc01c00 > [10753.561439] sg > [10753.561440] lpc_ich > [10753.561440] 0000000000000001 > [10753.561441] > [10753.561442] ffff88046f283e38 > [10753.561442] irq event stamp: 19364635 > [10753.561443] fjes > [10753.561444] ffffffff811640d6 > [10753.561445] button > [10753.561445] > [10753.561446] hardirqs last enabled at (19364634): > [10753.561447] ffff88046f283dc8 > [10753.561448] 8250_fintek > [10753.561448] shpchp > [10753.561452] ffff880319820078 > [10753.561452] [] restore_regs_and_iret+0x0/0x1d > [10753.561453] sb_edac > [10753.561454] hardirqs last disabled at (19364635): > [10753.561456] ffff88017fc1ba80 > [10753.561456] [] apic_timer_interrupt+0x84/0x90 > [10753.561457] ffff880317607cd8 > [10753.561458] softirqs last enabled at (13717672): > [10753.561458] > [10753.561459] pcspkr > [10753.561460] mei_me > [10753.561462] ffff880317607cd8 > [10753.561462] [] __do_softirq+0x1cb/0x230 > [10753.561464] ffff0001fbdd8798 > [10753.561464] ipmi_si > [10753.561465] 0000000000000292 > [10753.561466] softirqs last disabled at (13717773): > [10753.561467] ffff8803fbdd8798 > [10753.561468] [] irq_exit+0xa8/0xb0 > [10753.561468] > [10753.561469] mei > [10753.561470] edac_core > [10753.561470] Call Trace: > [10753.561472] CPU: 0 PID: 7937 Comm: fio Tainted: G O L 4.4.6-debug+ #2 > [10753.561473] > [10753.561474] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10753.561477] [] __slab_free+0xc6/0x270 > [10753.561480] mfd_core > [10753.561480] [] ? kmem_cache_free+0x9d/0x160 > [10753.561482] task: ffff88040070df00 ti: ffff8803f1678000 task.ti: ffff8803f1678000 > [10753.561485] [] ? free_object_rcu+0x89/0x90 > [10753.561487] RIP: 0010:[] > [10753.561489] wmi > [10753.561489] [] ? free_object_rcu+0x89/0x90 > [10753.561492] dm_multipath > [10753.561492] [] kmem_cache_free+0x159/0x160 > [10753.561494] [] _raw_spin_unlock_irqrestore+0x33/0x50 > [10753.561496] [] ? memory_failure_work_func+0xb0/0xb0 > [10753.561499] dm_mod > [10753.561499] [] free_object_rcu+0x89/0x90 > [10753.561500] RSP: 0018:ffff88046f203d78 EFLAGS: 00000292 > [10753.561503] [] rcu_process_callbacks+0x2b0/0x5f0 > [10753.561504] RAX: ffff88040070df00 RBX: 0000000000000292 RCX: 0000000000000000 > [10753.561506] ipmi_msghandler > [10753.561506] [] ? rcu_process_callbacks+0x26f/0x5f0 > [10753.561507] RDX: 0000000000000003 RSI: ffff88040070e698 RDI: ffff88040070df00 > [10753.561509] acpi_power_meter > [10753.561510] [] __do_softirq+0x10f/0x230 > [10753.561511] tpm_tis > [10753.561511] RBP: ffff88046f203d88 R08: ffff88046f203cf8 R09: 0000000000000000 > [10753.561514] hid_generic > [10753.561514] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10753.561516] usbhid > [10753.561516] R13: ffffea0007054c00 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10753.561518] tpm > [10753.561518] [] irq_exit+0xa8/0xb0 > [10753.561520] FS: 00007f8a9a632700(0000) GS:ffff88046f200000(0000) knlGS:0000000000000000 > [10753.561521] mgag200 > [10753.561522] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10753.561524] i2c_algo_bit > [10753.561525] [] smp_trace_apic_timer_interrupt+0x68/0x90 > [10753.561526] CR2: 00007f8aae007000 CR3: 00000003f9db8000 CR4: 00000000001406f0 > [10753.561528] drm_kms_helper > [10753.561528] [] smp_apic_timer_interrupt+0x9/0x10 > [10753.561530] syscopyarea > [10753.561531] [] apic_timer_interrupt+0x89/0x90 > [10753.561531] Stack: > [10753.561532] lpc_ich > [10753.561533] > [10753.561535] sysfillrect > [10753.561535] [] ? _raw_spin_unlock_irq+0x2f/0x40 > [10753.561536] button > [10753.561539] ffff88017fc01c00 > [10753.561539] [] get_request+0x447/0x8e0 > [10753.561542] sysimgblt > [10753.561542] [] ? blk_cleanup_queue+0x270/0x270 > [10753.561543] fb_sys_fops > [10753.561545] 0000000000000001 > [10753.561546] [] ? __delay+0x1/0x10 > [10753.561547] ffff88046f203e38 > [10753.561551] ttm > [10753.561551] [] ? wake_atomic_t_function+0x60/0x60 > [10753.561552] ffffffff811640d6 > [10753.561554] drm > [10753.561554] [] blk_queue_bio+0x111/0x300 > [10753.561555] > [10753.561557] [] generic_make_request+0xd2/0x290 > [10753.561558] shpchp > [10753.561560] ffff88046f203dc8 > [10753.561560] [] submit_bio+0x62/0x140 > [10753.561562] ehci_pci > [10753.561562] mei_me > [10753.561565] 0000000000000001 > [10753.561565] [] ? bio_set_pages_dirty+0x49/0x60 > [10753.561569] 0000000000000001 > [10753.561569] [] do_blockdev_direct_IO+0x2b68/0x34a0 > [10753.561570] sr_mod > [10753.561572] mei > [10753.561572] [] ? I_BDEV+0x10/0x10 > [10753.561573] cdrom > [10753.561574] ffff8801c1537a30 > [10753.561576] mfd_core > [10753.561576] [] __blockdev_direct_IO+0x3e/0x40 > [10753.561579] wmi > [10753.561579] [] blkdev_direct_IO+0x53/0x80 > [10753.561580] dm_multipath > [10753.561580] > [10753.561584] [] generic_file_read_iter+0x48c/0x5c0 > [10753.561585] ffff8801c1537a30 > [10753.561587] dm_mod > [10753.561588] [] ? __lock_acquire+0x3be/0x1ab0 > [10753.561590] ffff00010070e698 > [10753.561590] [] ? blkdev_write_iter+0x100/0x100 > [10753.561592] 0000000000000292 > [10753.561592] [] blkdev_read_iter+0x30/0x40 > [10753.561595] ehci_hcd > [10753.561596] [] aio_run_iocb+0x122/0x2a0 > [10753.561597] ffff88040070e698 > [10753.561599] usbcore > [10753.561599] [] do_io_submit+0x2f6/0x770 > [10753.561599] > [10753.561601] [] ? do_io_submit+0x193/0x770 > [10753.561602] hid_generic > [10753.561603] usb_common > [10753.561605] usbhid > [10753.561606] sg > [10753.561606] Call Trace: > [10753.561606] > [10753.561608] [] SyS_io_submit+0xb/0x10 > [10753.561610] > [10753.561610] irq event stamp: 14391137 > [10753.561613] [] __slab_free+0xc6/0x270 > [10753.561615] [] entry_SYSCALL_64_fastpath+0x16/0x6f > [10753.561618] mgag200 > [10753.561618] [] ? rcu_irq_exit+0x41/0x50 > [10753.561619] i2c_algo_bit > [10753.561621] Code: > [10753.561621] [] ? kmem_cache_free+0x9d/0x160 > [10753.561623] hardirqs last enabled at (14391136): > [10753.561625] fc > [10753.561625] [] _raw_spin_unlock_irqrestore+0x31/0x50 > [10753.561628] [] ? free_object_rcu+0x89/0x90 > [10753.561629] drm_kms_helper > [10753.561630] hardirqs last disabled at (14391137): > [10753.561632] 48 > [10753.561632] [] apic_timer_interrupt+0x84/0x90 > [10753.561635] [] ? free_object_rcu+0x89/0x90 > [10753.561636] syscopyarea > [10753.561638] 83 > [10753.561638] [] kmem_cache_free+0x159/0x160 > [10753.561639] c7 > [10753.561640] sysfillrect > [10753.561643] 18 > [10753.561643] [] ? memory_failure_work_func+0xb0/0xb0 > [10753.561645] sysimgblt > [10753.561646] [] free_object_rcu+0x89/0x90 > [10753.561648] 48 > [10753.561648] [] rcu_process_callbacks+0x2b0/0x5f0 > [10753.561651] softirqs last enabled at (8589138): > [10753.561651] [] ? rcu_process_callbacks+0x26f/0x5f0 > [10753.561653] [] __do_softirq+0x1cb/0x230 > [10753.561654] [] __do_softirq+0x10f/0x230 > [10753.561656] softirqs last disabled at (8589495): > [10753.561657] [] irq_exit+0xa8/0xb0 > [10753.561659] fb_sys_fops > [10753.561659] [] smp_trace_apic_timer_interrupt+0x68/0x90 > [10753.561661] [] irq_exit+0xa8/0xb0 > [10753.561663] [] smp_apic_timer_interrupt+0x9/0x10 > [10753.561665] 8b > [10753.561665] CPU: 2 PID: 7938 Comm: fio Tainted: G O L 4.4.6-debug+ #2 > [10753.561666] 55 > [10753.561667] ttm > [10753.561667] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10753.561669] [] apic_timer_interrupt+0x89/0x90 > [10753.561670] 08 > [10753.561671] drm > [10753.561672] 53 > [10753.561672] ehci_pci > [10753.561673] 48 > [10753.561674] sr_mod > [10753.561675] task: ffff880400708000 ti: ffff8803f7af8000 task.ti: ffff8803f7af8000 > [10753.561676] 89 > [10753.561677] > [10753.561677] f3 > [10753.561679] cdrom > [10753.561679] [] ? _raw_spin_unlock_irq+0x2f/0x40 > [10753.561680] be > [10753.561683] ehci_hcd > [10753.561683] [] get_request+0x447/0x8e0 > [10753.561684] 01 > [10753.561685] usbcore > [10753.561686] RIP: 0010:[] > [10753.561688] 00 > [10753.561688] [] _raw_spin_unlock_irqrestore+0x33/0x50 > [10753.561689] 00 > [10753.561690] RSP: 0018:ffff88046f243d78 EFLAGS: 00000292 > [10753.561691] 00 > [10753.561691] usb_common > [10753.561693] e8 > [10753.561694] [] ? blk_cleanup_queue+0x270/0x270 > [10753.561696] e1 > [10753.561696] [] ? __delay+0x1/0x10 > [10753.561697] sg > [10753.561698] ce > [10753.561698] > [10753.561702] [] ? wake_atomic_t_function+0x60/0x60 > [10753.561703] b2 > [10753.561703] irq event stamp: 126134521 > [10753.561705] [] blk_queue_bio+0x111/0x300 > [10753.561706] ff > [10753.561707] hardirqs last enabled at (126134520): > [10753.561709] 4c > [10753.561709] [] generic_make_request+0xd2/0x290 > [10753.561712] [] _raw_spin_unlock_irqrestore+0x31/0x50 > [10753.561714] 89 > [10753.561714] RAX: ffff880400708000 RBX: 0000000000000292 RCX: 0000000000000000 > [10753.561716] e7 > [10753.561716] RDX: 0000000000000003 RSI: ffff880400708798 RDI: ffff880400708000 > [10753.561718] e8 > [10753.561718] [] submit_bio+0x62/0x140 > [10753.561719] 49 > [10753.561721] hardirqs last disabled at (126134521): > [10753.561722] [] ? bio_set_pages_dirty+0x49/0x60 > [10753.561723] RBP: ffff88046f243d88 R08: ffff88046f243cf8 R09: 0000000000000000 > [10753.561725] 00 > [10753.561726] [] do_blockdev_direct_IO+0x2b68/0x34a0 > [10753.561728] b3 > [10753.561728] [] apic_timer_interrupt+0x84/0x90 > [10753.561729] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10753.561731] [] ? I_BDEV+0x10/0x10 > [10753.561732] ff > [10753.561732] R13: ffffea0008065000 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10753.561734] f6 > [10753.561734] FS: 00007f8a99e31700(0000) GS:ffff88046f240000(0000) knlGS:0000000000000000 > [10753.561736] [] __blockdev_direct_IO+0x3e/0x40 > [10753.561738] c7 > [10753.561739] [] blkdev_direct_IO+0x53/0x80 > [10753.561742] 02 > [10753.561742] [] generic_file_read_iter+0x48c/0x5c0 > [10753.561744] 74 > [10753.561744] [] ? __lock_acquire+0x3be/0x1ab0 > [10753.561746] 13 > [10753.561747] softirqs last enabled at (119833606): > [10753.561748] [] ? blkdev_write_iter+0x100/0x100 > [10753.561750] e8 > [10753.561750] [] blkdev_read_iter+0x30/0x40 > [10753.561751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10753.561754] 0f > [10753.561754] [] aio_run_iocb+0x122/0x2a0 > [10753.561756] [] __do_softirq+0x1cb/0x230 > [10753.561758] [] do_io_submit+0x2f6/0x770 > [10753.561760] a8 > [10753.561760] [] ? do_io_submit+0x193/0x770 > [10753.561762] b2 > [10753.561762] CR2: 00007f8aae007000 CR3: 00000003f9db8000 CR4: 00000000001406e0 > [10753.561764] ff > [10753.561764] [] SyS_io_submit+0xb/0x10 > [10753.561766] 53 > [10753.561766] softirqs last disabled at (119835011): > [10753.561766] Stack: > [10753.561768] [] irq_exit+0xa8/0xb0 > [10753.561770] ffff88017fc01c00 > [10753.561771] CPU: 1 PID: 358 Comm: kdmwork-254:2 Tainted: G O L 4.4.6-debug+ #2 > [10753.561772] [] entry_SYSCALL_64_fastpath+0x16/0x6f > [10753.561774] 9d > [10753.561774] 0000000000000001 > [10753.561775] <65> > [10753.561776] Code: > [10753.561777] ff > [10753.561777] ffff88046f243e38 > [10753.561778] fc > [10753.561779] 0d > [10753.561780] ffffffff811640d6 > [10753.561780] 48 > [10753.561781] 56 > [10753.561782] 83 > [10753.561782] > [10753.561783] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10753.561784] 2b > [10753.561784] c7 > [10753.561786] a9 > [10753.561786] task: ffff88040120af80 ti: ffff880400fc8000 task.ti: ffff880400fc8000 > [10753.561787] 7e > [10753.561788] 18 > [10753.561788] 5b > [10753.561789] 48 > [10753.561790] 41 > [10753.561790] 8b > [10753.561791] 5c > [10753.561792] 55 > [10753.561793] RIP: 0010:[] > [10753.561794] 08 > [10753.561794] 5d > [10753.561795] 53 > [10753.561795] c3 > [10753.561796] 48 > [10753.561797] 53 > [10753.561799] 89 f3 > [10753.561799] 9d > [10753.561801] be > [10753.561801] [] _raw_spin_unlock_irqrestore+0x33/0x50 > [10753.561803] e8 > [10753.561803] ffff88046f243dc8 > [10753.561804] 6a > [10753.561804] 0000000000000000 > [10753.561805] 79 > [10753.561806] 01 > [10753.561806] ffff880400708000 > [10753.561807] b2 > [10753.561808] ffff880201943278 > [10753.561809] 00 > [10753.561809] RSP: 0018:ffff88046f223d78 EFLAGS: 00000292 > [10753.561811] ff > [10753.561811] RAX: ffff88040120af80 RBX: 0000000000000292 RCX: 0000000000000000 > [10753.561812] eb > [10753.561813] 00 > [10753.561814] eb > [10753.561814] RDX: 0000000000000003 RSI: ffff88040120b718 RDI: ffff88040120af80 > [10753.561814] > [10753.561815] RBP: ffff88046f223d88 R08: ffff88046f223cf8 R09: 0000000000000000 > [10753.561816] > [10753.561817] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10753.561818] 00 > [10753.561819] ffff880201943278 > [10753.561820] e8 > [10753.561820] R13: ffffea00060e8600 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10753.561821] ffff000100708798 > [10753.561823] e1 > [10753.561823] FS: 0000000000000000(0000) GS:ffff88046f220000(0000) knlGS:0000000000000000 > [10753.561824] ce > [10753.561824] 0000000000000292 > [10753.561826] b2 > [10753.561826] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10753.561827] ff > [10753.561828] ffff880400708798 > [10753.561829] 4c > [10753.561829] CR2: 00007ffadf28aa08 CR3: 0000000001a0a000 CR4: 00000000001406e0 > [10753.561830] 89 > [10753.561830] Stack: > [10753.561831] e7 > [10753.561832] ffff88017fc01c00 > [10753.561833] e8 > [10753.561833] 0000000000000001 > [10753.561834] 49 > [10753.561835] ffff88046f223e38 > [10753.561835] 00 > [10753.561836] ffffffff811640d6 > [10753.561836] > [10753.561837] b3 > [10753.561837] > [10753.561838] ff > [10753.561839] ffff88046f223dc8 > [10753.561840] f6 > [10753.561840] 0000000000000246 > [10753.561841] c7 > [10753.561842] ffffffff811859b3 > [10753.561842] 02 > [10753.561843] Call Trace: > [10753.561844] 74 > [10753.561844] > [10753.561845] 13 > [10753.561846] ffff880183a1d258 > [10753.561846] e8 > [10753.561846] > [10753.561848] 0f > [10753.561848] ffff880183a1d258 > [10753.561851] a8 > [10753.561851] [] __slab_free+0xc6/0x270 > [10753.561854] b2 > [10753.561854] [] ? memory_failure_work_func+0xb0/0xb0 > [10753.561857] ff > [10753.561857] [] ? kmem_cache_free+0x9d/0x160 > [10753.561860] 53 > [10753.561860] [] ? free_object_rcu+0x89/0x90 > [10753.561861] ffff00010120b718 > [10753.561862] 9d > [10753.561862] 0000000000000292 > [10753.561863] <65> > [10753.561864] ffff88040120b718 > [10753.561864] ff > [10753.561864] > [10753.561866] 0d > [10753.561866] Call Trace: > [10753.561868] 56 > [10753.561868] [] ? free_object_rcu+0x89/0x90 > [10753.561871] 2b > [10753.561871] [] kmem_cache_free+0x159/0x160 > [10753.561874] a9 > [10753.561874] [] ? memory_failure_work_func+0xb0/0xb0 > [10753.561875] > [10753.561878] 7e > [10753.561878] [] free_object_rcu+0x89/0x90 > [10753.561881] [] __slab_free+0xc6/0x270 > [10753.561883] 5b > [10753.561883] [] rcu_process_callbacks+0x2b0/0x5f0 > [10753.561886] [] ? file_free_rcu+0x23/0x30 > [10753.561888] 41 > [10753.561888] [] ? rcu_process_callbacks+0x26f/0x5f0 > [10753.561890] [] ? kmem_cache_free+0x9d/0x160 > [10753.561892] 5c > [10753.561893] [] __do_softirq+0x10f/0x230 > [10753.561894] 5d > [10753.561895] [] irq_exit+0xa8/0xb0 > [10753.561897] c3 > [10753.561897] [] smp_trace_apic_timer_interrupt+0x68/0x90 > [10753.561900] 53 > [10753.561900] [] ? free_object_rcu+0x89/0x90 > [10753.561902] [] smp_apic_timer_interrupt+0x9/0x10 > [10753.561904] 9d > [10753.561904] [] apic_timer_interrupt+0x89/0x90 > [10753.561906] [] ? free_object_rcu+0x89/0x90 > [10753.561909] e8 > [10753.561909] [] kmem_cache_free+0x159/0x160 > [10753.561910] > [10753.561914] 6a > [10753.561914] [] ? memory_failure_work_func+0xb0/0xb0 > [10753.561916] 79 > [10753.561916] [] free_object_rcu+0x89/0x90 > [10753.561919] b2 > [10753.561919] [] rcu_process_callbacks+0x2b0/0x5f0 > [10753.561921] [] ? _raw_spin_unlock_irq+0x2f/0x40 > [10753.561923] ff > [10753.561923] [] ? rcu_process_callbacks+0x26f/0x5f0 > [10753.561926] eb > [10753.561926] [] get_request+0x447/0x8e0 > [10753.561928] eb > [10753.561928] [] __do_softirq+0x10f/0x230 > [10753.561928] > [10753.561930] [] irq_exit+0xa8/0xb0 > [10753.561932] [] ? blk_cleanup_queue+0x270/0x270 > [10753.561934] [] smp_trace_apic_timer_interrupt+0x68/0x90 > [10753.561935] [] ? __delay+0x1/0x10 > [10753.561937] [] smp_apic_timer_interrupt+0x9/0x10 > [10753.561940] [] ? wake_atomic_t_function+0x60/0x60 > [10753.561941] [] apic_timer_interrupt+0x89/0x90 > [10753.561943] [] blk_queue_bio+0x111/0x300 > [10753.561946] > [10753.561946] [] generic_make_request+0xd2/0x290 > [10753.561948] [] ? _raw_spin_unlock_irq+0x2f/0x40 > [10753.561950] [] submit_bio+0x62/0x140 > [10753.561953] [] __multipath_map.isra.16+0x145/0x260 [dm_multipath] > [10753.561955] [] ? bio_set_pages_dirty+0x49/0x60 > [10753.561957] [] multipath_map+0x12/0x20 [dm_multipath] > [10753.561960] [] do_blockdev_direct_IO+0x2b68/0x34a0 > [10753.561965] [] map_request+0x43/0x210 [dm_mod] > [10753.561966] [] ? I_BDEV+0x10/0x10 > [10753.561971] [] map_tio_request+0x1d/0x40 [dm_mod] > [10753.561973] [] __blockdev_direct_IO+0x3e/0x40 > [10753.561975] [] kthread_worker_fn+0x7d/0x1a0 > [10753.561977] [] blkdev_direct_IO+0x53/0x80 > [10753.561978] [] ? __init_kthread_worker+0x60/0x60 > [10753.561981] [] generic_file_read_iter+0x48c/0x5c0 > [10753.561983] [] kthread+0xea/0x100 > [10753.561985] [] ? __lock_acquire+0x3be/0x1ab0 > [10753.561986] [] ? kthread_create_on_node+0x1f0/0x1f0 > [10753.561988] [] ? blkdev_write_iter+0x100/0x100 > [10753.561990] [] ret_from_fork+0x3f/0x70 > [10753.561991] [] blkdev_read_iter+0x30/0x40 > [10753.561993] [] ? kthread_create_on_node+0x1f0/0x1f0 > [10753.561995] [] aio_run_iocb+0x122/0x2a0 > [10753.561998] Code: > [10753.561998] [] do_io_submit+0x2f6/0x770 > [10753.562000] fc > [10753.562000] [] ? do_io_submit+0x193/0x770 > [10753.562003] 48 83 > [10753.562003] [] SyS_io_submit+0xb/0x10 > [10753.562006] c7 > [10753.562006] [] entry_SYSCALL_64_fastpath+0x16/0x6f > [10753.562007] 18 48 > [10753.562008] Code: > [10753.562009] 8b > [10753.562009] fc > [10753.562010] 55 > [10753.562011] 48 > [10753.562012] 08 > [10753.562013] 83 > [10753.562013] 53 > [10753.562014] c7 > [10753.562014] 48 > [10753.562015] 18 > [10753.562016] 89 > [10753.562016] 48 > [10753.562017] f3 > [10753.562017] 8b > [10753.562018] be > [10753.562019] 55 > [10753.562019] 01 > [10753.562020] 08 > [10753.562020] 00 > [10753.562021] 53 > [10753.562022] 00 > [10753.562022] 48 > [10753.562023] 00 > [10753.562023] 89 > [10753.562024] e8 > [10753.562025] f3 > [10753.562025] e1 > [10753.562026] be > [10753.562026] ce > [10753.562027] 01 > [10753.562028] b2 > [10753.562028] 00 > [10753.562029] ff > [10753.562029] 00 > [10753.562030] 4c > [10753.562030] 00 > [10753.562031] 89 > [10753.562032] e8 > [10753.562032] e7 > [10753.562033] e1 > [10753.562033] e8 > [10753.562034] ce > [10753.562034] 49 > [10753.562035] b2 > [10753.562036] 00 > [10753.562036] ff > [10753.562037] b3 > [10753.562037] 4c > [10753.562038] ff > [10753.562038] 89 > [10753.562039] f6 > [10753.562040] e7 > [10753.562040] c7 > [10753.562041] e8 > [10753.562041] 02 > [10753.562042] 49 > [10753.562042] 74 > [10753.562043] 00 > [10753.562044] 13 > [10753.562044] b3 > [10753.562045] e8 > [10753.562045] ff > [10753.562046] 0f > [10753.562047] f6 > [10753.562047] a8 > [10753.562048] c7 > [10753.562048] b2 > [10753.562049] 02 > [10753.562049] ff > [10753.562050] 74 > [10753.562051] 53 > [10753.562051] 13 > [10753.562052] 9d > [10753.562052] e8 > [10753.562053] <65> > [10753.562054] 0f > [10753.562054] ff > [10753.562055] a8 > [10753.562055] 0d > [10753.562056] b2 > [10753.562056] 56 > [10753.562057] ff > [10753.562058] 2b > [10753.562058] 53 > [10753.562059] a9 > [10753.562059] 9d > [10753.562060] 7e > [10753.562061] <65> > [10753.562061] 5b > [10753.562062] ff > [10753.562062] 41 > [10753.562063] 0d > [10753.562064] 5c > [10753.562064] 56 > [10753.562065] 5d > [10753.562065] 2b > [10753.562066] c3 > [10753.562066] a9 > [10753.562067] 53 > [10753.562068] 7e > [10753.562068] 9d > [10753.562069] 5b > [10753.562069] e8 > [10753.562070] 41 > [10753.562071] 6a > [10753.562071] 5c > [10753.562072] 79 > [10753.562072] 5d > [10753.562073] b2 > [10753.562073] c3 > [10753.562074] ff > [10753.562075] 53 > [10753.562075] eb > [10753.562076] 9d > [10753.562076] eb > [10753.562077] e8 > [10753.562077] > [10753.562080] 6a 79 b2 ff eb eb > [10753.718766] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg > [10753.737483] irq event stamp: 13918891 > [10753.737667] hardirqs last enabled at (13918890): [] _raw_spin_unlock_irqrestore+0x31/0x50 > [10753.738020] hardirqs last disabled at (13918891): [] apic_timer_interrupt+0x84/0x90 > [10753.738364] softirqs last enabled at (10133236): [] scst_check_blocked_dev+0x77/0x1c0 [scst] > [10753.738765] softirqs last disabled at (10133267): [] irq_exit+0xa8/0xb0 > [10753.739063] CPU: 11 PID: 32753 Comm: disk11_0 Tainted: G O L 4.4.6-debug+ #2 > [10753.739256] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10753.739473] task: ffff880439375f00 ti: ffff880460d30000 task.ti: ffff880460d30000 > [10753.739599] RIP: 0010:[] [] _raw_spin_unlock_irqrestore+0x33/0x50 > [10753.739990] RSP: 0018:ffff88046f363d68 EFLAGS: 00000292 > [10753.740149] RAX: ffff880439375f00 RBX: 0000000000000292 RCX: 0000000000000000 > [10753.740321] RDX: 0000000000000003 RSI: ffff880439376698 RDI: ffff880439375f00 > [10753.740439] RBP: ffff88046f363d78 R08: ffff88046f363ce8 R09: 0000000000000000 > [10753.740664] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10753.740772] R13: ffffea000e725600 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10753.740991] FS: 0000000000000000(0000) GS:ffff88046f360000(0000) knlGS:0000000000000000 > [10753.741189] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10753.741321] CR2: 00005633130dd1a8 CR3: 0000000401468000 CR4: 00000000001406e0 > [10753.741465] Stack: > [10753.741613] ffff88017fc01c00 0000000000000001 ffff88046f363e30 ffffffff811640d6 > [10753.742334] ffff88046f363db8 ffffffff8157940f 0000000100300001 ffff88039c95bfc0 > [10753.743130] ffff88039c95bfc0 0000000100000282 0000000000000292 ffff880439376698 > [10753.743870] Call Trace: > [10753.744031] > [10753.744151] [] __slab_free+0xc6/0x270 > [10753.744464] [] ? retint_kernel+0x10/0x10 > [10753.744618] [] ? kmem_cache_free+0x9d/0x160 > [10753.744803] [] ? free_object_rcu+0x89/0x90 > [10753.744935] [] ? free_object_rcu+0x89/0x90 > [10753.745075] [] kmem_cache_free+0x159/0x160 > [10753.745209] [] ? memory_failure_work_func+0xb0/0xb0 > [10753.745387] [] free_object_rcu+0x89/0x90 > [10753.745585] [] rcu_process_callbacks+0x2b0/0x5f0 > [10753.745769] [] ? rcu_process_callbacks+0x26f/0x5f0 > [10753.745920] [] __do_softirq+0x10f/0x230 > [10753.746063] [] irq_exit+0xa8/0xb0 > [10753.746262] [] do_IRQ+0x65/0x110 > [10753.746437] [] ? mempool_free_slab+0x12/0x20 > [10753.746653] [] common_interrupt+0x89/0x89 > [10753.746803] > [10753.746973] [] ? mempool_free_slab+0x12/0x20 > [10753.747278] [] ? put_object+0x7/0x40 > [10753.747453] [] ? __delete_object+0x38/0x70 > [10753.747650] [] delete_object_full+0x18/0x20 > [10753.747797] [] kmemleak_free+0x31/0x80 > [10753.747987] [] kmem_cache_free+0x7a/0x160 > [10753.748137] [] mempool_free_slab+0x12/0x20 > [10753.748284] [] mempool_free+0x26/0x80 > [10753.748423] [] bvec_free+0x31/0x40 > [10753.748551] [] bio_free+0x34/0x60 > [10753.748737] [] bio_put+0x1e/0x30 > [10753.748865] [] blockio_endio+0x2d/0xe0 [scst_vdisk] > [10753.749111] [] bio_endio+0x3a/0x60 > [10753.749289] [] brd_make_request+0xe8/0x1ae [brd] > [10753.749431] [] ? generic_make_request_checks+0x10c/0x630 > [10753.749628] [] generic_make_request+0xd2/0x290 > [10753.749762] [] submit_bio+0x62/0x140 > [10753.749942] [] blockio_exec_rw+0x553/0x670 [scst_vdisk] > [10753.750123] [] ? blockio_exec_write_verify+0x50/0x50 [scst_vdisk] > [10753.750325] [] blockio_exec_read+0xd/0x20 [scst_vdisk] > [10753.750472] [] vdev_do_job+0xb3/0x310 [scst_vdisk] > [10753.750631] [] blockio_exec+0x74/0x180 [scst_vdisk] > [10753.750759] [] ? trace_hardirqs_on_caller+0x121/0x1b0 > [10753.750930] [] ? sgv_get_obj+0x72/0x1b0 [scst] > [10753.751195] [] scst_do_real_exec+0x94/0x3d0 [scst] > [10753.751372] [] ? scst_is_not_reservation_holder+0x13/0x60 [scst] > [10753.751602] [] ? __scst_check_local_events+0x19f/0x370 [scst] > [10753.751802] [] scst_exec_check_blocking+0xcc/0x340 [scst] > [10753.751983] [] scst_exec_check_sn+0x12d/0x2c0 [scst] > [10753.752136] [] ? finish_task_switch+0xa6/0x220 > [10753.752307] [] ? finish_task_switch+0x67/0x220 > [10753.752548] [] scst_process_active_cmd+0x3e2/0x760 [scst] > [10753.752696] [] ? trace_hardirqs_on+0xd/0x10 > [10753.752875] [] scst_do_job_active+0xe1/0x160 [scst] > [10753.753142] [] scst_cmd_thread+0x132/0x290 [scst] > [10753.753311] [] ? wake_atomic_t_function+0x60/0x60 > [10753.753524] [] ? scst_do_job_active+0x160/0x160 [scst] > [10753.753681] [] kthread+0xea/0x100 > [10753.753836] [] ? kthread_create_on_node+0x1f0/0x1f0 > [10753.753992] [] ret_from_fork+0x3f/0x70 > [10753.754165] [] ? kthread_create_on_node+0x1f0/0x1f0 > [10753.754315] Code: fc 48 83 c7 18 48 8b 55 08 53 48 89 f3 be 01 00 00 00 e8 e1 ce b2 ff 4c 89 e7 e8 49 00 b3 ff f6 c7 02 74 13 e8 0f a8 b2 ff 53 9d <65> ff 0d 56 2b a9 7e 5b 41 5c 5d c3 53 9d e8 6a 79 b2 ff eb eb > [10764.112230] scsi host50: SRP abort called > [10764.112455] scsi host50: Sending SRP abort for tag 0x0 > [10764.116228] [32754]: scst: scst_rx_mgmt_fn:7350:TM fn ABORT_TASK/0 (mcmd ffff88006892e788, initiator fe80:0000:0000:0000:e41d:2d03:000a:85b1, target fe80:0000:0000:0000:e41d:2d03:000a:6d51) > [10764.116401] [32754]: scst_rx_mgmt_fn:7364:sess=ffff88041067f400, tag_set 1, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv ffff880454310548 > [10764.116635] [32754]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892e788 to active mgmt cmd list > [10764.117014] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892e788 from active cmd list > [10764.117333] [31019]: scst_abort_task:6762:Aborting task (cmd ffff8804543140a8, sn -20, set 1, tag 0, queue_type 1) > [10764.117488] [31019]: scst: scst_abort_cmd:5849:Aborting cmd ffff8804543140a8 (tag 0, op READ(10)) > [10764.117908] [31019]: scst_abort_cmd:5934:cmd ffff8804543140a8 (tag 0) is being executed > [10764.118220] [31019]: scst: scst_abort_cmd:5970:cmd ffff8804543140a8 (tag 0, sn 4294967276) being executed/xmitted (state EXEC_WAIT, op READ(10), proc time 63 sec., timeout 60 sec.), deferring ABORT (cmd_done_wait_count 1, cmd_finish_wait_count 1, internal 0, mcmd fn 0 (mcmd ffff88006892e788), initiator fe80:0000:0000:0000:e41d:2d03:000a:85b1, target fe80:0000:0000:0000:e41d:2d03:000a:6d51) > [10764.118408] [31019]: scst: scst_set_mcmd_next_state:6011:cmd_done_wait_count(1) not 0, preparing to wait > [10765.270812] INFO: rcu_sched self-detected stall on CPU > [10765.271293] 4-...: (6499 ticks this GP) idle=ca3/140000000000002/0 softirq=1007789/1007789 fqs=1001 > [10765.271442] (t=6500 jiffies g=219292 c=219291 q=13711541) > [10765.271809] Task dump for CPU 0: > [10765.272032] fio R running task 0 7937 7744 0x00000008 > [10765.272691] ffff8803f167b898 ffff8804536bd140 0000000000000000 0000000000000000 > [10765.273812] 0000000000000000 0000000000000000 ffff88040070df00 0000000000000001 > [10765.274809] 0000000000000000 0000000000000001 ffff88040070df00 ffffffffffffff10 > [10765.275910] Call Trace: > [10765.276149] [] ? _raw_spin_unlock_irq+0x2f/0x40 > [10765.276362] [] get_request+0x447/0x8e0 > [10765.276514] [] ? blk_cleanup_queue+0x270/0x270 > [10765.276828] [] ? __delay+0x1/0x10 > [10765.277080] [] ? wake_atomic_t_function+0x60/0x60 > [10765.277216] [] blk_queue_bio+0x111/0x300 > [10765.277479] [] generic_make_request+0xd2/0x290 > [10765.277651] [] submit_bio+0x62/0x140 > [10765.277899] [] ? bio_set_pages_dirty+0x49/0x60 > [10765.278145] [] do_blockdev_direct_IO+0x2b68/0x34a0 > [10765.278401] [] ? I_BDEV+0x10/0x10 > [10765.278610] [] __blockdev_direct_IO+0x3e/0x40 > [10765.278777] [] blkdev_direct_IO+0x53/0x80 > [10765.278941] [] generic_file_read_iter+0x48c/0x5c0 > [10765.279138] [] ? __lock_acquire+0x3be/0x1ab0 > [10765.279261] [] ? blkdev_write_iter+0x100/0x100 > [10765.279486] [] blkdev_read_iter+0x30/0x40 > [10765.279661] [] aio_run_iocb+0x122/0x2a0 > [10765.279829] [] do_io_submit+0x2f6/0x770 > [10765.280114] [] ? do_io_submit+0x193/0x770 > [10765.280307] [] SyS_io_submit+0xb/0x10 > [10765.280564] [] entry_SYSCALL_64_fastpath+0x16/0x6f > [10765.280766] INFO: rcu_sched self-detected stall on CPU > [10765.280770] 0-...: (6412 ticks this GP) idle=911/140000000000002/0 softirq=1235409/1235409 fqs=1001 > [10765.280773] (t=6500 jiffies g=219292 c=219291 q=13711939) > [10765.280775] INFO: rcu_sched self-detected stall on CPU > [10765.280779] 2-...: (6500 ticks this GP) idle=203/140000000000002/0 softirq=1498700/1498700 fqs=1002 > [10765.280781] (t=6500 jiffies g=219292 c=219291 q=13711939) > [10765.281492] Task dump for CPU 2: > [10765.281636] fio R running task 0 7938 7744 0x00000008 > [10765.281940] ffff8803f7afb898 ffff8804536bd140 0000000000000000 0000000000000000 > [10765.282551] 0000000000000000 0000000000000000 ffff880400708000 0000000000000001 > [10765.283093] 0000000000000000 0000000000000001 ffff880400708000 ffffffffffffff10 > [10765.283618] Call Trace: > [10765.283746] [] ? _raw_spin_unlock_irq+0x2f/0x40 > [10765.283849] [] get_request+0x447/0x8e0 > [10765.283977] [] ? blk_cleanup_queue+0x270/0x270 > [10765.284102] [] ? __delay+0x1/0x10 > [10765.284238] [] ? wake_atomic_t_function+0x60/0x60 > [10765.284368] [] blk_queue_bio+0x111/0x300 > [10765.284486] [] generic_make_request+0xd2/0x290 > [10765.284616] [] submit_bio+0x62/0x140 > [10765.284722] [] ? bio_set_pages_dirty+0x49/0x60 > [10765.284842] [] do_blockdev_direct_IO+0x2b68/0x34a0 > [10765.284985] [] ? I_BDEV+0x10/0x10 > [10765.285085] [] __blockdev_direct_IO+0x3e/0x40 > [10765.285202] [] blkdev_direct_IO+0x53/0x80 > [10765.285313] [] generic_file_read_iter+0x48c/0x5c0 > [10765.285428] [] ? __lock_acquire+0x3be/0x1ab0 > [10765.285558] [] ? blkdev_write_iter+0x100/0x100 > [10765.285688] [] blkdev_read_iter+0x30/0x40 > [10765.285812] [] aio_run_iocb+0x122/0x2a0 > [10765.285932] [] do_io_submit+0x2f6/0x770 > [10765.286048] [] ? do_io_submit+0x193/0x770 > [10765.286193] [] SyS_io_submit+0xb/0x10 > [10765.286317] [] entry_SYSCALL_64_fastpath+0x16/0x6f > [10765.286415] Task dump for CPU 4: > [10765.286525] fio R running task 0 7935 7744 0x00000008 > [10765.286964] 0000000000001e40 ffff88046f283ab8 ffffffff8108195f 0000000000000004 > [10765.287498] ffffffff81a40a40 ffff88046f283ad0 ffffffff81084742 0000000000000005 > [10765.288065] ffff88046f283b00 ffffffff810b5e79 ffff88046f295400 ffffffff81a40a40 > [10765.288674] Call Trace: > [10765.288821] [] sched_show_task+0xbf/0x150 > [10765.289152] [] dump_cpu_task+0x32/0x40 > [10765.289292] [] rcu_dump_cpu_stacks+0x89/0xe0 > [10765.289432] [] rcu_check_callbacks+0x439/0x730 > [10765.289588] [] update_process_times+0x34/0x60 > [10765.289730] [] tick_sched_handle.isra.18+0x20/0x50 > [10765.289896] [] tick_sched_timer+0x38/0x70 > [10765.290036] [] ? tick_sched_do_timer+0x30/0x30 > [10765.290173] [] __hrtimer_run_queues+0xa5/0x1c0 > [10765.290390] [] hrtimer_interrupt+0xa6/0x1b0 > [10765.290529] [] ? trace_hardirqs_off+0xd/0x10 > [10765.290631] [] smp_trace_apic_timer_interrupt+0x63/0x90 > [10765.290744] [] smp_apic_timer_interrupt+0x9/0x10 > [10765.290855] [] apic_timer_interrupt+0x89/0x90 > [10765.290982] [] ? _raw_spin_unlock_irqrestore+0x33/0x50 > [10765.291096] [] __slab_free+0xc6/0x270 > [10765.291208] [] ? kmem_cache_free+0x9d/0x160 > [10765.291322] [] ? free_object_rcu+0x89/0x90 > [10765.291449] [] ? free_object_rcu+0x89/0x90 > [10765.291561] [] kmem_cache_free+0x159/0x160 > [10765.291687] [] ? memory_failure_work_func+0xb0/0xb0 > [10765.291802] [] free_object_rcu+0x89/0x90 > [10765.291915] [] rcu_process_callbacks+0x2b0/0x5f0 > [10765.292040] [] ? rcu_process_callbacks+0x26f/0x5f0 > [10765.292204] [] __do_softirq+0x10f/0x230 > [10765.292330] [] irq_exit+0xa8/0xb0 > [10765.292492] [] smp_trace_apic_timer_interrupt+0x68/0x90 > [10765.292615] [] smp_apic_timer_interrupt+0x9/0x10 > [10765.292751] [] apic_timer_interrupt+0x89/0x90 > [10765.292871] [] ? _raw_spin_unlock_irq+0x2f/0x40 > [10765.293139] [] get_request+0x447/0x8e0 > [10765.293259] [] ? blk_cleanup_queue+0x270/0x270 > [10765.293392] [] ? __delay+0x1/0x10 > [10765.293538] [] ? wake_atomic_t_function+0x60/0x60 > [10765.293670] [] blk_queue_bio+0x111/0x300 > [10765.293790] [] generic_make_request+0xd2/0x290 > [10765.293910] [] submit_bio+0x62/0x140 > [10765.294041] [] ? bio_set_pages_dirty+0x49/0x60 > [10765.294151] [] do_blockdev_direct_IO+0x2b68/0x34a0 > [10765.294295] [] ? I_BDEV+0x10/0x10 > [10765.294446] [] __blockdev_direct_IO+0x3e/0x40 > [10765.294613] [] blkdev_direct_IO+0x53/0x80 > [10765.294747] [] generic_file_read_iter+0x48c/0x5c0 > [10765.294914] [] ? __lock_acquire+0x3be/0x1ab0 > [10765.295062] [] ? blkdev_write_iter+0x100/0x100 > [10765.295183] [] blkdev_read_iter+0x30/0x40 > [10765.295317] [] aio_run_iocb+0x122/0x2a0 > [10765.295451] [] do_io_submit+0x2f6/0x770 > [10765.295576] [] ? do_io_submit+0x193/0x770 > [10765.295698] [] SyS_io_submit+0xb/0x10 > [10765.295830] [] entry_SYSCALL_64_fastpath+0x16/0x6f > [10765.295963] Task dump for CPU 0: > [10765.296100] fio R running task 0 7937 7744 0x00000008 > [10765.296734] ffff8803f167b898 ffff8804536bd140 0000000000000000 0000000000000000 > [10765.297665] 0000000000000000 0000000000000000 ffff88040070df00 0000000000000001 > [10765.298695] 0000000000000000 0000000000000001 ffff88040070df00 ffffffffffffff10 > [10765.299591] Call Trace: > [10765.299782] [] ? _raw_spin_unlock_irq+0x2f/0x40 > [10765.299953] [] get_request+0x447/0x8e0 > [10765.300118] [] ? blk_cleanup_queue+0x270/0x270 > [10765.300363] [] ? __delay+0x1/0x10 > [10765.300506] [] ? wake_atomic_t_function+0x60/0x60 > [10765.300650] [] blk_queue_bio+0x111/0x300 > [10765.300887] [] generic_make_request+0xd2/0x290 > [10765.301094] [] submit_bio+0x62/0x140 > [10765.301393] [] ? bio_set_pages_dirty+0x49/0x60 > [10765.301577] [] do_blockdev_direct_IO+0x2b68/0x34a0 > [10765.301817] [] ? I_BDEV+0x10/0x10 > [10765.301951] [] __blockdev_direct_IO+0x3e/0x40 > [10765.302143] [] blkdev_direct_IO+0x53/0x80 > [10765.302359] [] generic_file_read_iter+0x48c/0x5c0 > [10765.302512] [] ? __lock_acquire+0x3be/0x1ab0 > [10765.302664] [] ? blkdev_write_iter+0x100/0x100 > [10765.302848] [] blkdev_read_iter+0x30/0x40 > [10765.302981] [] aio_run_iocb+0x122/0x2a0 > [10765.303128] [] do_io_submit+0x2f6/0x770 > [10765.303255] [] ? do_io_submit+0x193/0x770 > [10765.303427] [] SyS_io_submit+0xb/0x10 > [10765.303656] [] entry_SYSCALL_64_fastpath+0x16/0x6f > [10765.303787] Task dump for CPU 2: > [10765.303918] fio R running task 0 7938 7744 0x00000008 > [10765.304354] 0000000000001e40 ffff88046f243ab8 ffffffff8108195f 0000000000000002 > [10765.305171] ffffffff81a40a40 ffff88046f243ad0 ffffffff81084742 0000000000000003 > [10765.306025] ffff88046f243b00 ffffffff810b5e79 ffff88046f255400 ffffffff81a40a40 > [10765.306874] Call Trace: > [10765.307035] [] sched_show_task+0xbf/0x150 > [10765.307436] [] dump_cpu_task+0x32/0x40 > [10765.307729] [] rcu_dump_cpu_stacks+0x89/0xe0 > [10765.308054] [] rcu_check_callbacks+0x439/0x730 > [10765.308207] [] update_process_times+0x34/0x60 > [10765.308381] [] tick_sched_handle.isra.18+0x20/0x50 > [10765.308539] [] tick_sched_timer+0x38/0x70 > [10765.308816] [] ? tick_sched_do_timer+0x30/0x30 > [10765.309055] [] __hrtimer_run_queues+0xa5/0x1c0 > [10765.309246] [] hrtimer_interrupt+0xa6/0x1b0 > [10765.309401] [] ? trace_hardirqs_off+0xd/0x10 > [10765.309643] [] smp_trace_apic_timer_interrupt+0x63/0x90 > [10765.309759] [] smp_apic_timer_interrupt+0x9/0x10 > [10765.309909] [] apic_timer_interrupt+0x89/0x90 > [10765.310054] [] ? _raw_spin_unlock_irqrestore+0x33/0x50 > [10765.310226] [] __slab_free+0xc6/0x270 > [10765.310381] [] ? memory_failure_work_func+0xb0/0xb0 > [10765.310522] [] ? kmem_cache_free+0x9d/0x160 > [10765.310667] [] ? free_object_rcu+0x89/0x90 > [10765.310860] [] ? free_object_rcu+0x89/0x90 > [10765.311075] [] kmem_cache_free+0x159/0x160 > [10765.311285] [] ? memory_failure_work_func+0xb0/0xb0 > [10765.311467] [] free_object_rcu+0x89/0x90 > [10765.311660] [] rcu_process_callbacks+0x2b0/0x5f0 > [10765.311842] [] ? rcu_process_callbacks+0x26f/0x5f0 > [10765.311991] [] __do_softirq+0x10f/0x230 > [10765.312170] [] irq_exit+0xa8/0xb0 > [10765.312381] [] smp_trace_apic_timer_interrupt+0x68/0x90 > [10765.312502] [] smp_apic_timer_interrupt+0x9/0x10 > [10765.312632] [] apic_timer_interrupt+0x89/0x90 > [10765.312826] [] ? _raw_spin_unlock_irq+0x2f/0x40 > [10765.313135] [] get_request+0x447/0x8e0 > [10765.313266] [] ? blk_cleanup_queue+0x270/0x270 > [10765.313424] [] ? __delay+0x1/0x10 > [10765.313607] [] ? wake_atomic_t_function+0x60/0x60 > [10765.313738] [] blk_queue_bio+0x111/0x300 > [10765.313954] [] generic_make_request+0xd2/0x290 > [10765.314094] [] submit_bio+0x62/0x140 > [10765.314275] [] ? bio_set_pages_dirty+0x49/0x60 > [10765.314388] [] do_blockdev_direct_IO+0x2b68/0x34a0 > [10765.314516] [] ? I_BDEV+0x10/0x10 > [10765.314738] [] __blockdev_direct_IO+0x3e/0x40 > [10765.314917] [] blkdev_direct_IO+0x53/0x80 > [10765.315116] [] generic_file_read_iter+0x48c/0x5c0 > [10765.315283] [] ? __lock_acquire+0x3be/0x1ab0 > [10765.315415] [] ? blkdev_write_iter+0x100/0x100 > [10765.315566] [] blkdev_read_iter+0x30/0x40 > [10765.315707] [] aio_run_iocb+0x122/0x2a0 > [10765.315868] [] do_io_submit+0x2f6/0x770 > [10765.316010] [] ? do_io_submit+0x193/0x770 > [10765.316198] [] SyS_io_submit+0xb/0x10 > [10765.316322] [] entry_SYSCALL_64_fastpath+0x16/0x6f > [10765.316530] Task dump for CPU 4: > [10765.316737] fio R running task 0 7935 7744 0x00000008 > [10765.317203] 0000000000000001 00007f8a58031000 0000000000001000 000000000000001e > [10765.317947] ffff8803f7527de8 ffff8803fbdd8000 0000000000000246 ffff8803f7437300 > [10765.318778] 000000000000001e 0000000000000246 ffff8803f7527e38 ffff8803fbdd8000 > [10765.319525] Call Trace: > [10765.319669] [] ? do_io_submit+0x2f6/0x770 > [10765.319853] [] ? do_io_submit+0x193/0x770 > [10765.319993] [] ? SyS_io_submit+0xb/0x10 > [10765.320150] [] ? entry_SYSCALL_64_fastpath+0x16/0x6f > [10765.320304] Task dump for CPU 0: > [10765.320555] fio R running task 0 7937 7744 0x00000008 > [10765.321211] 0000000000001e40 ffff88046f203ab8 ffffffff8108195f 0000000000000000 > [10765.322400] ffffffff81a40a40 ffff88046f203ad0 ffffffff81084742 0000000000000001 > [10765.323303] ffff88046f203b00 ffffffff810b5e79 ffff88046f215400 ffffffff81a40a40 > [10765.324130] Call Trace: > [10765.324330] [] sched_show_task+0xbf/0x150 > [10765.324936] [] dump_cpu_task+0x32/0x40 > [10765.325164] [] rcu_dump_cpu_stacks+0x89/0xe0 > [10765.325309] [] rcu_check_callbacks+0x439/0x730 > [10765.325476] [] update_process_times+0x34/0x60 > [10765.325659] [] tick_sched_handle.isra.18+0x20/0x50 > [10765.325980] [] tick_sched_timer+0x38/0x70 > [10765.326220] [] ? tick_sched_do_timer+0x30/0x30 > [10765.326459] [] __hrtimer_run_queues+0xa5/0x1c0 > [10765.326699] [] hrtimer_interrupt+0xa6/0x1b0 > [10765.326895] [] ? trace_hardirqs_off+0xd/0x10 > [10765.327086] [] smp_trace_apic_timer_interrupt+0x63/0x90 > [10765.327224] [] smp_apic_timer_interrupt+0x9/0x10 > [10765.327407] [] apic_timer_interrupt+0x89/0x90 > [10765.327622] [] ? _raw_spin_unlock_irqrestore+0x33/0x50 > [10765.327760] [] __slab_free+0xc6/0x270 > [10765.327916] [] ? file_free_rcu+0x23/0x30 > [10765.328133] [] ? kmem_cache_free+0x9d/0x160 > [10765.328434] [] ? free_object_rcu+0x89/0x90 > [10765.328835] [] ? free_object_rcu+0x89/0x90 > [10765.329043] [] kmem_cache_free+0x159/0x160 > [10765.329243] [] ? memory_failure_work_func+0xb0/0xb0 > [10765.329487] [] free_object_rcu+0x89/0x90 > [10765.329649] [] rcu_process_callbacks+0x2b0/0x5f0 > [10765.329865] [] ? rcu_process_callbacks+0x26f/0x5f0 > [10765.330097] [] __do_softirq+0x10f/0x230 > [10765.330272] [] irq_exit+0xa8/0xb0 > [10765.330456] [] smp_trace_apic_timer_interrupt+0x68/0x90 > [10765.330696] [] smp_apic_timer_interrupt+0x9/0x10 > [10765.330836] [] apic_timer_interrupt+0x89/0x90 > [10765.330993] [] ? _raw_spin_unlock_irq+0x2f/0x40 > [10765.331359] [] get_request+0x447/0x8e0 > [10765.331554] [] ? blk_cleanup_queue+0x270/0x270 > [10765.331757] [] ? __delay+0x1/0x10 > [10765.332129] [] ? wake_atomic_t_function+0x60/0x60 > [10765.332320] [] blk_queue_bio+0x111/0x300 > [10765.332594] [] generic_make_request+0xd2/0x290 > [10765.332749] [] submit_bio+0x62/0x140 > [10765.333071] [] ? bio_set_pages_dirty+0x49/0x60 > [10765.333244] [] do_blockdev_direct_IO+0x2b68/0x34a0 > [10765.333426] [] ? I_BDEV+0x10/0x10 > [10765.333644] [] __blockdev_direct_IO+0x3e/0x40 > [10765.333810] [] blkdev_direct_IO+0x53/0x80 > [10765.334020] [] generic_file_read_iter+0x48c/0x5c0 > [10765.334203] [] ? __lock_acquire+0x3be/0x1ab0 > [10765.334348] [] ? blkdev_write_iter+0x100/0x100 > [10765.334474] [] blkdev_read_iter+0x30/0x40 > [10765.334656] [] aio_run_iocb+0x122/0x2a0 > [10765.334972] [] do_io_submit+0x2f6/0x770 > [10765.335179] [] ? do_io_submit+0x193/0x770 > [10765.335430] [] SyS_io_submit+0xb/0x10 > [10765.335757] [] entry_SYSCALL_64_fastpath+0x16/0x6f > [10765.335928] Task dump for CPU 2: > [10765.336319] fio R running task 0 7938 7744 0x00000008 > [10765.337078] ffff8803f7afb898 ffff8804536bd140 0000000000000000 0000000000000000 > [10765.338168] 0000000000000000 0000000000000000 ffff880400708000 0000000000000001 > [10765.339181] 0000000000000000 0000000000000001 ffff880400708000 ffffffffffffff10 > [10765.340163] Call Trace: > [10765.340325] [] ? _raw_spin_unlock_irq+0x2f/0x40 > [10765.340515] [] get_request+0x447/0x8e0 > [10765.340721] [] ? blk_cleanup_queue+0x270/0x270 > [10765.340843] [] ? __delay+0x1/0x10 > [10765.341044] [] ? wake_atomic_t_function+0x60/0x60 > [10765.341210] [] blk_queue_bio+0x111/0x300 > [10765.341364] [] generic_make_request+0xd2/0x290 > [10765.341552] [] submit_bio+0x62/0x140 > [10765.341776] [] ? bio_set_pages_dirty+0x49/0x60 > [10765.341929] [] do_blockdev_direct_IO+0x2b68/0x34a0 > [10765.342070] [] ? I_BDEV+0x10/0x10 > [10765.342378] [] __blockdev_direct_IO+0x3e/0x40 > [10765.342599] [] blkdev_direct_IO+0x53/0x80 > [10765.342798] [] generic_file_read_iter+0x48c/0x5c0 > [10765.343017] [] ? __lock_acquire+0x3be/0x1ab0 > [10765.343241] [] ? blkdev_write_iter+0x100/0x100 > [10765.343401] [] blkdev_read_iter+0x30/0x40 > [10765.343562] [] aio_run_iocb+0x122/0x2a0 > [10765.343696] [] do_io_submit+0x2f6/0x770 > [10765.343873] [] ? do_io_submit+0x193/0x770 > [10765.344044] [] SyS_io_submit+0xb/0x10 > [10765.344231] [] entry_SYSCALL_64_fastpath+0x16/0x6f > [10765.344385] Task dump for CPU 4: > [10765.344560] fio R running task 0 7935 7744 0x00000008 > [10765.345208] 0000000000000001 00007f8a58031000 0000000000001000 000000000000001e > [10765.345981] ffff8803f7527de8 ffff8803fbdd8000 0000000000000246 ffff8803f7437300 > [10765.347051] 000000000000001e 0000000000000246 ffff8803f7527e38 ffff8803fbdd8000 > [10765.347947] Call Trace: > [10765.348145] [] ? do_io_submit+0x2f6/0x770 > [10765.348313] [] ? do_io_submit+0x193/0x770 > [10765.348448] [] ? SyS_io_submit+0xb/0x10 > [10765.348880] [] ? entry_SYSCALL_64_fastpath+0x16/0x6f > [10777.682813] systemd-journald[8985]: File /var/log/journal/6d2d7eb462015b615a423a1056c6183d/system.journal corrupted or uncleanly shut down, renaming and replacing. > [10778.058082] systemd-coredump[8980]: Detected coredump of the journal daemon itself, diverted to /var/lib/systemd/coredump/core.systemd-journal.0.fd2308d1f77847cf9c32b20535a1d348.499.1458676877000000.xz. > [10781.560307] NMI watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [disk11_0:32753] > [10781.560440] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg > [10781.573088] irq event stamp: 20161671 > [10781.573201] hardirqs last enabled at (20161670): [] _raw_spin_unlock_irqrestore+0x31/0x50 > [10781.573436] hardirqs last disabled at (20161671): [] apic_timer_interrupt+0x84/0x90 > [10781.573698] softirqs last enabled at (10133236): [] scst_check_blocked_dev+0x77/0x1c0 [scst] > [10781.573951] softirqs last disabled at (10133267): [] irq_exit+0xa8/0xb0 > [10781.574196] CPU: 11 PID: 32753 Comm: disk11_0 Tainted: G O L 4.4.6-debug+ #2 > [10781.574316] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10781.574425] task: ffff880439375f00 ti: ffff880460d30000 task.ti: ffff880460d30000 > [10781.574531] RIP: 0010:[] [] _raw_spin_unlock_irqrestore+0x33/0x50 > [10781.574742] RSP: 0018:ffff88046f363d68 EFLAGS: 00000292 > [10781.574854] RAX: ffff880439375f00 RBX: 0000000000000292 RCX: 0000000000000000 > [10781.574969] RDX: 0000000000000003 RSI: ffff880439376698 RDI: ffff880439375f00 > [10781.575105] RBP: ffff88046f363d78 R08: ffff88046f363ce8 R09: 0000000000000000 > [10781.575209] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10781.575310] R13: ffffea0001036200 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10781.575429] FS: 0000000000000000(0000) GS:ffff88046f360000(0000) knlGS:0000000000000000 > [10781.575542] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10781.575676] CR2: 00005633130dd1a8 CR3: 0000000001a0a000 CR4: 00000000001406e0 > [10781.575786] Stack: > [10781.575896] ffff88017fc01c00 0000000000000001 ffff88046f363e30 ffffffff811640d6 > [10781.576422] ffff88046f363db8 ffffffff811640c0 0000000100300001 ffff880040d8b7c8 > [10781.576944] ffff880040d8b7c8 0000000113d14e00 0000000000000292 ffff880439376698 > [10781.577475] Call Trace: > [10781.577585] > [10781.577690] [] __slab_free+0xc6/0x270 > [10781.577918] [] ? __slab_free+0xb0/0x270 > [10781.578050] [] ? kmem_cache_free+0x9d/0x160 > [10781.578176] [] ? free_object_rcu+0x89/0x90 > [10781.578323] [] ? free_object_rcu+0x89/0x90 > [10781.578447] [] kmem_cache_free+0x159/0x160 > [10781.578584] [] ? memory_failure_work_func+0xb0/0xb0 > [10781.578697] [] free_object_rcu+0x89/0x90 > [10781.578805] [] rcu_process_callbacks+0x2b0/0x5f0 > [10781.578927] [] ? rcu_process_callbacks+0x26f/0x5f0 > [10781.579038] [] __do_softirq+0x10f/0x230 > [10781.579164] [] irq_exit+0xa8/0xb0 > [10781.579276] [] do_IRQ+0x65/0x110 > [10781.579407] [] ? mempool_free_slab+0x12/0x20 > [10781.579518] [] common_interrupt+0x89/0x89 > [10781.579629] > [10781.579746] [] ? mempool_free_slab+0x12/0x20 > [10781.579977] [] ? put_object+0x7/0x40 > [10781.580090] [] ? __delete_object+0x38/0x70 > [10781.580212] [] delete_object_full+0x18/0x20 > [10781.580328] [] kmemleak_free+0x31/0x80 > [10781.580430] [] kmem_cache_free+0x7a/0x160 > [10781.580553] [] mempool_free_slab+0x12/0x20 > [10781.580672] [] mempool_free+0x26/0x80 > [10781.580798] [] bvec_free+0x31/0x40 > [10781.580916] [] bio_free+0x34/0x60 > [10781.581037] [] bio_put+0x1e/0x30 > [10781.581162] [] blockio_endio+0x2d/0xe0 [scst_vdisk] > [10781.581287] [] bio_endio+0x3a/0x60 > [10781.581404] [] brd_make_request+0xe8/0x1ae [brd] > [10781.581521] [] ? generic_make_request_checks+0x10c/0x630 > [10781.581631] [] generic_make_request+0xd2/0x290 > [10781.581746] [] submit_bio+0x62/0x140 > [10781.581871] [] blockio_exec_rw+0x553/0x670 [scst_vdisk] > [10781.581992] [] ? blockio_exec_write_verify+0x50/0x50 [scst_vdisk] > [10781.582106] [] blockio_exec_read+0xd/0x20 [scst_vdisk] > [10781.582260] [] vdev_do_job+0xb3/0x310 [scst_vdisk] > [10781.582398] [] blockio_exec+0x74/0x180 [scst_vdisk] > [10781.582514] [] ? trace_hardirqs_on_caller+0x121/0x1b0 > [10781.582652] [] ? sgv_get_obj+0x72/0x1b0 [scst] > [10781.582804] [] scst_do_real_exec+0x94/0x3d0 [scst] > [10781.582952] [] ? scst_is_not_reservation_holder+0x13/0x60 [scst] > [10781.583109] [] ? __scst_check_local_events+0x19f/0x370 [scst] > [10781.583245] [] scst_exec_check_blocking+0xcc/0x340 [scst] > [10781.583387] [] scst_exec_check_sn+0x12d/0x2c0 [scst] > [10781.583520] [] ? finish_task_switch+0xa6/0x220 > [10781.583646] [] ? finish_task_switch+0x67/0x220 > [10781.583777] [] scst_process_active_cmd+0x3e2/0x760 [scst] > [10781.583920] [] ? trace_hardirqs_on+0xd/0x10 > [10781.584054] [] scst_do_job_active+0xe1/0x160 [scst] > [10781.584193] [] scst_cmd_thread+0x132/0x290 [scst] > [10781.584316] [] ? wake_atomic_t_function+0x60/0x60 > [10781.584458] [] ? scst_do_job_active+0x160/0x160 [scst] > [10781.584590] [] kthread+0xea/0x100 > [10781.584706] [] ? kthread_create_on_node+0x1f0/0x1f0 > [10781.584827] [] ret_from_fork+0x3f/0x70 > [10781.584948] [] ? kthread_create_on_node+0x1f0/0x1f0 > [10781.585086] Code: fc 48 83 c7 18 48 8b 55 08 53 48 89 f3 be 01 00 00 00 e8 e1 ce b2 ff 4c 89 e7 e8 49 00 b3 ff f6 c7 02 74 13 e8 0f a8 b2 ff 53 9d <65> ff 0d 56 2b a9 7e 5b 41 5c 5d c3 53 9d e8 6a 79 b2 ff eb eb > [10788.304454] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:85b2-611 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.304852] scsi host60: ib_srp: DREQ received - connection closed > [10788.305727] ib_srpt: RDMA t 5 for idx 5 failed with status 5. If this has not been triggered by a cable pull, please check the involved IB HCA's and cables. > [10788.305926] ib_srpt: sending response for idx 5 failed with status 5 > [10788.306061] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:85b2-611. > [10788.328967] [560]: scst_unregister_session:7901:Unregistering session ffff880416bf0000 (wait 0) > [10788.329005] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:85b2-617 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.329770] [560]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892d810) > [10788.329839] scsi host60: ib_srp: DREQ received - connection closed > [10788.330132] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:85b2-623 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.330698] [574]: scst_unregister_session:7901:Unregistering session ffff880416bf0e80 (wait 0) > [10788.330921] [560]: scst_rx_mgmt_fn:7364:sess=ffff880416bf0000, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.331187] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:85b2-617. > [10788.331547] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:85b2-629 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.332133] [574]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892e5d0) > [10788.332434] [560]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892d810 to active mgmt cmd list > [10788.332490] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:6d52-610 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.332907] [574]: scst_rx_mgmt_fn:7364:sess=ffff880416bf0e80, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.333332] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892d810 from active cmd list > [10788.333516] [586]: scst_unregister_session:7901:Unregistering session ffff880416bf1d00 (wait 0) > [10788.333707] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf0000 (mcmd ffff88006892d810) > [10788.333977] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:6d52-616 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.334379] [574]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892e5d0 to active mgmt cmd list > [10788.334553] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.334780] [586]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892d9c8) > [10788.335366] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.335483] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:6d52-621 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.335955] [586]: scst_rx_mgmt_fn:7364:sess=ffff880416bf1d00, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.336287] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892d810) finished, status 0 > [10788.336554] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:6d52-627 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.337339] [586]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892d9c8 to active mgmt cmd list > [10788.337480] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892e5d0 from active cmd list > [10788.337707] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:6d51-613 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.337926] [603]: scst_unregister_session:7901:Unregistering session ffff880416bf4880 (wait 0) > [10788.338031] [554]: scst_unregister_session:7901:Unregistering session ffff880400faba00 (wait 0) > [10788.338437] [603]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892d810) > [10788.339050] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:6d51-619 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.339367] [570]: scst_unregister_session:7901:Unregistering session ffff880416bf6cc0 (wait 0) > [10788.339774] [555]: scst: scst_cmd_thread:5499:Processing thread disk134_0 finished > [10788.339973] [570]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892e418) > [10788.340435] [603]: scst_rx_mgmt_fn:7364:sess=ffff880416bf4880, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.340624] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf0e80 (mcmd ffff88006892e5d0) > [10788.340759] [554]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892db80) > [10788.340994] [603]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892d810 to active mgmt cmd list > [10788.341354] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.341536] [554]: scst_rx_mgmt_fn:7364:sess=ffff880400faba00, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.342115] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.342324] [593]: scst_unregister_session:7901:Unregistering session ffff880416bf4fc0 (wait 0) > [10788.342613] [554]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892db80 to active mgmt cmd list > [10788.342920] [593]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892dd38) > [10788.343926] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892e5d0) finished, status 0 > [10788.344411] [593]: scst_rx_mgmt_fn:7364:sess=ffff880416bf4fc0, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.344785] [570]: scst_rx_mgmt_fn:7364:sess=ffff880416bf6cc0, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.345000] [593]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892dd38 to active mgmt cmd list > [10788.345266] [570]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892e418 to active mgmt cmd list > [10788.345403] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892d9c8 from active cmd list > [10788.345667] [566]: scst_unregister_session:7901:Unregistering session ffff880416bf0740 (wait 0) > [10788.345840] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf1d00 (mcmd ffff88006892d9c8) > [10788.346315] [566]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892e5d0) > [10788.346665] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.346879] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.347005] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892d9c8) finished, status 0 > [10788.347253] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892d810 from active cmd list > [10788.347462] [566]: scst_rx_mgmt_fn:7364:sess=ffff880416bf0740, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.347810] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf4880 (mcmd ffff88006892d810) > [10788.347858] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:6d51-624 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.348180] [566]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892e5d0 to active mgmt cmd list > [10788.348448] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.348697] [579]: scst_unregister_session:7901:Unregistering session ffff880416bf15c0 (wait 0) > [10788.349339] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.349487] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892d810) finished, status 0 > [10788.349699] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:6d51-630 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.349946] [579]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892d9c8) > [10788.350246] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892db80 from active cmd list > [10788.350560] [579]: scst_rx_mgmt_fn:7364:sess=ffff880416bf15c0, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.350823] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880400faba00 (mcmd ffff88006892db80) > [10788.350953] scsi host58: ib_srp: DREQ received - connection closed > [10788.351102] [579]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892d9c8 to active mgmt cmd list > [10788.351139] scsi host58: ib_srp: DREQ received - connection closed > [10788.351454] scsi host58: ib_srp: DREQ received - connection closed > [10788.351938] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.352233] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.352645] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892db80) finished, status 0 > [10788.352944] [588]: scst_unregister_session:7901:Unregistering session ffff880416bf5700 (wait 0) > [10788.353527] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892e418 from active cmd list > [10788.353540] scsi host58: ib_srp: DREQ received - connection closed > [10788.353933] scsi host61: ib_srp: DREQ received - connection closed > [10788.354074] scsi host61: ib_srp: DREQ received - connection closed > [10788.354416] scsi host61: ib_srp: DREQ received - connection closed > [10788.354762] [588]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892db80) > [10788.354868] scsi host61: ib_srp: DREQ received - connection closed > [10788.355077] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:6d52-610. > [10788.355101] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:6d52-616. > [10788.355141] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:6d52-621. > [10788.357117] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:6d52-627. > [10788.357120] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf6cc0 (mcmd ffff88006892e418) > [10788.357183] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.357185] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.357190] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892e418) finished, status 0 > [10788.357202] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892dd38 from active cmd list > [10788.357276] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf4fc0 (mcmd ffff88006892dd38) > [10788.357337] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.357339] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.357343] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892dd38) finished, status 0 > [10788.357353] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892e5d0 from active cmd list > [10788.357390] [588]: scst_rx_mgmt_fn:7364:sess=ffff880416bf5700, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.357393] [588]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892db80 to active mgmt cmd list > [10788.357445] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf0740 (mcmd ffff88006892e5d0) > [10788.357502] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.357504] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.357508] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892e5d0) finished, status 0 > [10788.357517] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892d9c8 from active cmd list > [10788.357748] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf15c0 (mcmd ffff88006892d9c8) > [10788.357806] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.357808] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.357811] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892d9c8) finished, status 0 > [10788.357821] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892db80 from active cmd list > [10788.357901] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf5700 (mcmd ffff88006892db80) > [10788.357958] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.357960] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.357964] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892db80) finished, status 0 > [10788.357980] scsi host60: ib_srp: DREQ received - connection closed > [10788.358287] scsi host60: ib_srp: DREQ received - connection closed > [10788.359468] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:85b2-623. > [10788.359491] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:85b2-629. > [10788.360123] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:85b1-612 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.360815] scsi host59: ib_srp: DREQ received - connection closed > [10788.361642] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:85b1-618 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.361991] [564]: scst_unregister_session:7901:Unregistering session ffff880416bf7400 (wait 0) > [10788.362038] [564]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892db80) > [10788.362042] [564]: scst_rx_mgmt_fn:7364:sess=ffff880416bf7400, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.362045] [564]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892db80 to active mgmt cmd list > [10788.362094] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:85b1-612. > [10788.362118] scsi host59: ib_srp: DREQ received - connection closed > [10788.362616] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892db80 from active cmd list > [10788.362838] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf7400 (mcmd ffff88006892db80) > [10788.362899] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.362901] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.362906] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892db80) finished, status 0 > [10788.364202] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:85b1-618. > [10788.370985] [576]: scst_unregister_session:7901:Unregistering session ffff880416bf6580 (wait 0) > [10788.371047] [576]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892db80) > [10788.371050] [576]: scst_rx_mgmt_fn:7364:sess=ffff880416bf6580, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.371053] [576]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892db80 to active mgmt cmd list > [10788.371169] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:85b1-625 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.371592] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892db80 from active cmd list > [10788.371644] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf6580 (mcmd ffff88006892db80) > [10788.371705] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.371707] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.371711] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892db80) finished, status 0 > [10788.386343] [601]: scst_unregister_session:7901:Unregistering session ffff880416bf2b80 (wait 0) > [10788.386502] [601]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892db80) > [10788.386505] [601]: scst_rx_mgmt_fn:7364:sess=ffff880416bf2b80, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.386508] [601]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892db80 to active mgmt cmd list > [10788.386522] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892db80 from active cmd list > [10788.386590] scsi host59: ib_srp: DREQ received - connection closed > [10788.386724] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf2b80 (mcmd ffff88006892db80) > [10788.386785] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.386787] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.386791] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892db80) finished, status 0 > [10788.387386] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:85b1-625. > [10788.396186] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:6d51-613. > [10788.396511] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:6d51-619. > [10788.396758] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:6d51-624. > [10788.397245] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:6d51-630. > [10788.397967] ib_srpt: Closing channel fe80:0000:0000:0000:e41d:2d03:000a:85b1-631 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.399003] scsi host59: ib_srp: DREQ received - connection closed > [10788.399155] [590]: scst_unregister_session:7901:Unregistering session ffff880416bf2440 (wait 0) > [10788.399349] [590]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892db80) > [10788.399353] [590]: scst_rx_mgmt_fn:7364:sess=ffff880416bf2440, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.399356] [590]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892db80 to active mgmt cmd list > [10788.399369] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892db80 from active cmd list > [10788.399437] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf2440 (mcmd ffff88006892db80) > [10788.399501] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.399504] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.399508] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892db80) finished, status 0 > [10788.409181] ib_srpt: Closing channel fe80:0000:0000:0000:0002:c903:00a0:2dc2-657 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.409978] [606]: scst_unregister_session:7901:Unregistering session ffff880416bf32c0 (wait 0) > [10788.410171] [606]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892db80) > [10788.410174] [606]: scst_rx_mgmt_fn:7364:sess=ffff880416bf32c0, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.410177] [606]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892db80 to active mgmt cmd list > [10788.410190] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892db80 from active cmd list > [10788.410296] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf32c0 (mcmd ffff88006892db80) > [10788.410359] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.410361] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.410365] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892db80) finished, status 0 > [10788.418821] ib_srpt: Closing channel fe80:0000:0000:0000:0002:c903:00a0:2dc1-658 because target fe80:0000:0000:0000:e41d:2d03:000a:85b1 has been disabled > [10788.419326] [2408]: scst_unregister_session:7901:Unregistering session ffff8803f8f17400 (wait 0) > [10788.419798] [2408]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892db80) > [10788.419801] [2408]: scst_rx_mgmt_fn:7364:sess=ffff8803f8f17400, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.419804] [2408]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892db80 to active mgmt cmd list > [10788.419815] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892db80 from active cmd list > [10788.419981] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff8803f8f17400 (mcmd ffff88006892db80) > [10788.420042] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.420045] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.420049] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892db80) finished, status 0 > [10788.427954] [2410]: scst_unregister_session:7901:Unregistering session ffff8803f8f10740 (wait 0) > [10788.428188] [2410]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892db80) > [10788.428192] [2410]: scst_rx_mgmt_fn:7364:sess=ffff8803f8f10740, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.428195] [2410]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892db80 to active mgmt cmd list > [10788.428208] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892db80 from active cmd list > [10788.428513] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff8803f8f10740 (mcmd ffff88006892db80) > [10788.428576] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.428579] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.428583] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892db80) finished, status 0 > [10788.479546] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0 > [10788.479914] [573]: scst: scst_cmd_thread:5499:Processing thread disk138_0 finished > [10788.484823] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0 > [10788.486170] [585]: scst: scst_cmd_thread:5499:Processing thread disk142_0 finished > [10788.490385] ib_srpt: Received CM TimeWait exit for ch fe80:0000:0000:0000:e41d:2d03:000a:85b1-618. > [10788.490785] scsi host59: ib_srp: connection closed > [10788.490788] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:0002:c903:00a0:2dc2-657. > [10788.490850] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:0002:c903:00a0:2dc1-658. > [10788.490894] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:e41d:2d03:000a:85b1-631. > [10788.493602] scsi host59: ib_srp: connection closed > [10788.493990] ib_srpt: Received CM TimeWait exit for ch fe80:0000:0000:0000:e41d:2d03:000a:85b1-612. > [10788.520887] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0 > [10788.521524] ib_srpt: Received CM TimeWait exit for ch fe80:0000:0000:0000:e41d:2d03:000a:85b1-625. > [10788.522148] scsi host59: ib_srp: connection closed > [10788.526290] [602]: scst: scst_cmd_thread:5499:Processing thread disk147_0 finished > [10788.532024] [553]: scst: scst_cmd_thread:5499:Processing thread disk133_0 finished > [10788.547718] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0 > [10788.548481] [569]: scst: scst_cmd_thread:5499:Processing thread disk137_0 finished > [10788.551854] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0 > [10788.600121] scsi host59: ib_srp: connection closed > [10788.620114] ib_srpt: Received CM TimeWait exit for ch fe80:0000:0000:0000:e41d:2d03:000a:85b1-631. > [10788.670189] [581]: scst_unregister_session:7901:Unregistering session ffff880416bf5e40 (wait 0) > [10788.671892] [581]: scst_rx_mgmt_fn:7354:TM fn UNREG_SESS_TM/10 (mcmd ffff88006892db80) > [10788.673417] [581]: scst_rx_mgmt_fn:7364:sess=ffff880416bf5e40, tag_set 0, tag 0, lun_set 0, lun=0, cmd_sn_set 0, cmd_sn 0, priv (null) > [10788.673860] [581]: scst_post_rx_mgmt_cmd:7277:Adding mgmt cmd ffff88006892db80 to active mgmt cmd list > [10788.674072] [31019]: scst_tm_thread:7162:Deleting mgmt cmd ffff88006892db80 from active cmd list > [10788.674441] [592]: scst: scst_cmd_thread:5499:Processing thread disk145_0 finished > [10788.675123] [31019]: scst_abort_all_nexus_loss_sess:6647:Nexus loss or UNREG SESS for sess ffff880416bf5e40 (mcmd ffff88006892db80) > [10788.676436] [31019]: scst_call_dev_task_mgmt_fn_done:5801:Calling dev handler vdisk_blockio task_mgmt_fn_done(fn=10) > [10788.676676] [31019]: scst_call_dev_task_mgmt_fn_done:5804:Dev handler vdisk_blockio task_mgmt_fn_done() returned > [10788.676914] [31019]: scst_mgmt_cmd_send_done:7016:TM fn 10 (mcmd ffff88006892db80) finished, status 0 > [10788.698994] [565]: scst: scst_cmd_thread:5499:Processing thread disk136_0 finished > [10788.703046] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0 > [10788.703357] [578]: scst: scst_cmd_thread:5499:Processing thread disk140_0 finished > [10788.708079] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0 > [10788.708352] [587]: scst: scst_cmd_thread:5499:Processing thread disk143_0 finished > [10788.712911] [563]: scst: scst_cmd_thread:5499:Processing thread disk135_0 finished > [10788.716807] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0 > [10790.350077] scsi host60: ib_srp: connection closed > [10790.370069] scsi host60: ib_srp: connection closed > [10790.400086] ib_srpt: Received CM TimeWait exit for ch fe80:0000:0000:0000:e41d:2d03:000a:6d52-621. > [10790.400090] scsi host60: ib_srp: connection closed > [10790.400103] scsi host60: ib_srp: connection closed > [10790.400685] scsi host61: ib_srp: connection closed > [10790.400701] scsi host61: ib_srp: connection closed > [10790.400715] scsi host61: ib_srp: connection closed > [10790.400729] scsi host61: ib_srp: connection closed > [10790.402597] scsi host58: ib_srp: connection closed > [10790.403206] scsi host58: ib_srp: connection closed > [10790.403613] scsi host58: ib_srp: connection closed > [10790.404086] scsi host58: ib_srp: connection closed > [10790.440067] ib_srpt: Received CM TimeWait exit for ch fe80:0000:0000:0000:e41d:2d03:000a:6d51-630. > [10790.540711] ib_srpt: Received CM TimeWait exit for ch fe80:0000:0000:0000:0002:c903:00a0:2dc1-658. > [10790.541056] ib_srpt: Received CM TimeWait exit for ch fe80:0000:0000:0000:0002:c903:00a0:2dc2-657. > [10793.419992] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:85b1: waiting for session unregistration ... > [10793.421559] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:6d52-621: state disconnected; 0 commands in progress > [10793.422461] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:6d51-630: state disconnected; 0 commands in progress > [10793.422928] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:85b1-618: state disconnected; 0 commands in progress > [10793.424120] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:85b1-625: state disconnected; 0 commands in progress > [10793.424912] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:85b1-631: state disconnected; 0 commands in progress > [10793.425992] ib_srpt: fe80:0000:0000:0000:0002:c903:00a0:2dc2-657: state disconnected; 0 commands in progress > [10793.427069] ib_srpt: fe80:0000:0000:0000:0002:c903:00a0:2dc1-658: state disconnected; 0 commands in progress > [10798.310258] scsi host60: ib_srp: Sending CM DREQ failed > [10798.310968] scsi host60: ib_srp: Sending CM DREQ failed > [10798.311772] scsi host60: ib_srp: Sending CM DREQ failed > [10798.312243] scsi host60: ib_srp: Sending CM DREQ failed > [10798.369850] scsi host58: ib_srp: Sending CM DREQ failed > [10798.369851] scsi host61: ib_srp: Sending CM DREQ failed > [10798.369854] scsi host61: ib_srp: Sending CM DREQ failed > [10798.369856] scsi host61: ib_srp: Sending CM DREQ failed > [10798.369858] scsi host61: ib_srp: Sending CM DREQ failed > [10798.371422] scsi host58: ib_srp: Sending CM DREQ failed > [10798.372037] scsi host58: ib_srp: Sending CM DREQ failed > [10798.372303] scsi host58: ib_srp: Sending CM DREQ failed > [10798.389849] scsi host59: ib_srp: Sending CM DREQ failed > [10798.390128] scsi host59: ib_srp: Sending CM DREQ failed > [10798.390336] scsi host59: ib_srp: Sending CM DREQ failed > [10798.390551] scsi host59: ib_srp: Sending CM DREQ failed > [10798.429858] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:85b1: waiting for session unregistration ... > [10798.430095] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:6d52-621: state disconnected; 0 commands in progress > [10798.430326] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:6d51-630: state disconnected; 0 commands in progress > [10798.430632] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:85b1-618: state disconnected; 0 commands in progress > [10798.431060] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:85b1-625: state disconnected; 0 commands in progress > [10798.431522] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:85b1-631: state disconnected; 0 commands in progress > [10798.432378] ib_srpt: fe80:0000:0000:0000:0002:c903:00a0:2dc2-657: state disconnected; 0 commands in progress > [10798.432607] ib_srpt: fe80:0000:0000:0000:0002:c903:00a0:2dc1-658: state disconnected; 0 commands in progress > [10798.593380] scsi host60: ib_srp: REJ received > [10798.594137] scsi host60: REJ reason 0x8 > [10798.595188] scsi host60: reconnect attempt 1 failed (-104) > [10798.609908] scsi host61: ib_srp: REJ received > [10798.610291] scsi host61: REJ reason 0x8 > [10798.610828] scsi host61: reconnect attempt 1 failed (-104) > [10798.612989] scsi host58: ib_srp: REJ received > [10798.613366] scsi host58: REJ reason 0x8 > [10798.613752] scsi host58: reconnect attempt 1 failed (-104) > [10798.629912] [575]: scst: scst_cmd_thread:5499:Processing thread disk139_0 finished > [10798.633770] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0 > [10798.634435] [600]: scst: scst_cmd_thread:5499:Processing thread disk146_0 finished > [10798.661741] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0 > [10798.661942] [589]: scst: scst_cmd_thread:5499:Processing thread disk144_0 finished > [10798.665387] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0 > [10798.665829] [604]: scst: scst_cmd_thread:5499:Processing thread disk148_0 finished > [10798.677166] [2407]: scst: scst_cmd_thread:5499:Processing thread disk169_0 finished > [10798.680754] [2409]: scst: scst_cmd_thread:5499:Processing thread disk170_0 finished > [10798.683539] scsi host59: ib_srp: REJ received > [10798.683665] scsi host59: REJ reason 0x8 > [10798.683743] [31020]: scst_free_all_UA:12829:Clearing UA for tgt_dev LUN 0 > [10798.684732] scsi host59: reconnect attempt 1 failed (-104) > [10803.350085] fast_io_fail_tmo expired for SRP port-60:1 / host60. > [10803.354685] sd 60:0:0:0: rejecting I/O to offline device > [10803.358943] sd 60:0:0:0: [sdbx] killing request > [10803.359803] sd 60:0:0:0: rejecting I/O to offline device > [10803.360324] sd 60:0:0:0: [sdbx] killing request > [10803.361332] sd 60:0:0:0: rejecting I/O to offline device > [10803.362291] sd 60:0:0:0: rejecting I/O to offline device > [10803.362295] device-mapper: multipath: Failing path 68:176. > [10803.363197] sd 60:0:0:0: rejecting I/O to offline device > [10803.363526] sd 60:0:0:0: rejecting I/O to offline device > [10803.363812] sd 60:0:0:0: rejecting I/O to offline device > [10803.363983] sd 60:0:0:0: rejecting I/O to offline device > [10803.364277] sd 60:0:0:0: rejecting I/O to offline device > [10803.364472] sd 60:0:0:0: [sdbx] FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK > [10803.364762] sd 60:0:0:0: [sdbx] CDB: Read(10) 28 00 00 00 1a 58 00 00 08 00 > [10803.365071] blk_update_request: I/O error, dev sdbx, sector 6744 > [10803.365345] sd 60:0:0:0: [sdbx] FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK > [10803.365650] sd 60:0:0:0: [sdbx] CDB: Read(10) 28 00 00 00 5c 38 00 00 08 00 > [10803.365915] blk_update_request: I/O error, dev sdbx, sector 23608 > [10803.399704] fast_io_fail_tmo expired for SRP port-59:1 / host59. > [10803.399708] fast_io_fail_tmo expired for SRP port-58:1 / host58. > [10803.400381] sd 59:0:0:0: rejecting I/O to offline device > [10803.400514] fast_io_fail_tmo expired for SRP port-61:1 / host61. > [10803.401006] sd 59:0:0:0: rejecting I/O to offline device > [10803.401008] device-mapper: multipath: Failing path 68:192. > [10803.401448] sd 59:0:0:0: rejecting I/O to offline device > [10803.401660] sd 59:0:0:0: rejecting I/O to offline device > [10803.401808] sd 59:0:0:0: rejecting I/O to offline device > [10803.401999] sd 59:0:0:0: rejecting I/O to offline device > [10803.402200] sd 59:0:0:0: rejecting I/O to offline device > [10803.402387] sd 59:0:0:0: rejecting I/O to offline device > [10803.402586] sd 59:0:0:0: rejecting I/O to offline device > [10803.429706] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:85b1: waiting for session unregistration ... > [10803.430138] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:6d52-621: state disconnected; 0 commands in progress > [10803.904798] device-mapper: multipath: Failing path 68:160. > [10803.909315] device-mapper: multipath: Failing path 68:208. > [10808.429596] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:85b1: waiting for session unregistration ... > [10808.430002] ib_srpt: fe80:0000:0000:0000:e41d:2d03:000a:6d52-621: state disconnected; 0 commands in progress > [10808.609582] scsi host60: ib_srp: Sending CM DREQ failed > [10808.609770] scsi host60: ib_srp: Sending CM DREQ failed > [10808.609938] scsi host60: ib_srp: Sending CM DREQ failed > [10808.610378] scsi host60: ib_srp: Sending CM DREQ failed > [10808.639569] scsi host58: ib_srp: Sending CM DREQ failed > [10808.639765] scsi host58: ib_srp: Sending CM DREQ failed > [10808.640049] scsi host58: ib_srp: Sending CM DREQ failed > [10808.640217] scsi host58: ib_srp: Sending CM DREQ failed > [10808.660821] scsi host61: ib_srp: Sending CM DREQ failed > [10808.661050] scsi host61: ib_srp: Sending CM DREQ failed > [10808.661794] scsi host61: ib_srp: Sending CM DREQ failed > [10808.661961] scsi host61: ib_srp: Sending CM DREQ failed > [10808.690042] [580]: scst: scst_cmd_thread:5499:Processing thread disk141_0 finished > [10808.693028] [8149]: scst_suspend_activity:965:suspend_count 0 > [10808.693136] [8149]: scst: scst_suspend_activity:990:Waiting for 10 active commands to complete... > [10808.709633] scsi host59: ib_srp: Sending CM DREQ failed > [10808.709729] scsi host59: ib_srp: Sending CM DREQ failed > [10808.709807] scsi host59: ib_srp: Sending CM DREQ failed > [10808.709892] scsi host59: ib_srp: Sending CM DREQ failed > [10808.730162] scsi host60: ib_srp: REJ received > [10808.730248] scsi host60: REJ reason 0x8 > [10808.730374] scsi host60: reconnect attempt 2 failed (-104) > [10808.755642] scsi host61: ib_srp: REJ received > [10808.755727] scsi host61: REJ reason 0x8 > [10808.755874] scsi host61: reconnect attempt 2 failed (-104) > [10808.757781] scsi host58: ib_srp: REJ received > [10808.757861] scsi host58: REJ reason 0x8 > [10808.757991] scsi host58: reconnect attempt 2 failed (-104) > [10808.789571] scsi host59: ib_srp: REJ received > [10808.789652] scsi host59: REJ reason 0x8 > [10808.789734] scsi host59: reconnect attempt 2 failed (-104) > [10809.402402] [550]: scst_translate_lun:4957:FLAG SUSPENDED set, skipping > [10809.559516] NMI watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [disk11_0:32753] > [10809.559635] Modules linked in: ib_srpt(O-) scst_vdisk(O) libcrc32c scst(O) dlm netconsole configfs dm_queue_length brd xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet bridge stp llc iscsi_ibft iscsi_boot_sysfs ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs msr ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel aesni_intel tg3 iTCO_wdt aes_x86_64 lrw ipmi_ssif ipmi_devintf ptp iTCO_vendor_support gf128mul glue_helper mlx4_core dcdbas ablk_helper pps_core cryptd libphy fjes 8250_fintek sb_edac pcspkr ipmi_si edac_core ipmi_msghandler acpi_power_meter tpm_tis tpm lpc_ich button shpchp mei_me mei mfd_core wmi dm_multipath dm_mod hid_generic usbhid mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ehci_pci sr_mod cdrom ehci_hcd usbcore usb_common sg > [10809.570166] irq event stamp: 29993531 > [10809.570263] hardirqs last enabled at (29993530): [] _raw_spin_unlock_irqrestore+0x31/0x50 > [10809.570452] hardirqs last disabled at (29993531): [] apic_timer_interrupt+0x84/0x90 > [10809.570640] softirqs last enabled at (10133236): [] scst_check_blocked_dev+0x77/0x1c0 [scst] > [10809.570851] softirqs last disabled at (10133267): [] irq_exit+0xa8/0xb0 > [10809.571039] CPU: 11 PID: 32753 Comm: disk11_0 Tainted: G O L 4.4.6-debug+ #2 > [10809.571140] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014 > [10809.571239] task: ffff880439375f00 ti: ffff880460d30000 task.ti: ffff880460d30000 > [10809.571338] RIP: 0010:[] [] _raw_spin_unlock_irqrestore+0x33/0x50 > [10809.571530] RSP: 0018:ffff88046f363d68 EFLAGS: 00000292 > [10809.571626] RAX: ffff880439375f00 RBX: 0000000000000292 RCX: 0000000000000000 > [10809.571722] RDX: 0000000000000003 RSI: ffff880439376698 RDI: ffff880439375f00 > [10809.571818] RBP: ffff88046f363d78 R08: ffff88046f363ce8 R09: 0000000000000000 > [10809.571914] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88017fc01c00 > [10809.572010] R13: ffffea0008392800 R14: ffff88017fc01c00 R15: ffff88017fc1ba80 > [10809.572106] FS: 0000000000000000(0000) GS:ffff88046f360000(0000) knlGS:0000000000000000 > [10809.572204] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [10809.572298] CR2: 00005633130dd1a8 CR3: 0000000001a0a000 CR4: 00000000001406e0 > [10809.572392] Stack: > [10809.572488] ffff88017fc01c00 0000000000000001 ffff88046f363e30 ffffffff811640d6 > [10809.572943] ffff88046f363db8 ffffffff8157940f 0000000100300001 ffff88020e4a3fc0 > [10809.573396] ffff88020e4a3fc0 0000000100000282 0000000000000292 ffff880439376698 > [10809.573845] Call Trace: > [10809.573938] > [10809.574034] [] __slab_free+0xc6/0x270 > [10809.574236] [] ? retint_kernel+0x10/0x10 > [10809.574338] [] ? kmem_cache_free+0x9d/0x160 > [10809.574441] [] ? free_object_rcu+0x89/0x90 > [10809.574543] [] ? free_object_rcu+0x89/0x90 > [10809.574644] [] kmem_cache_free+0x159/0x160 > [10809.574746] [] ? memory_failure_work_func+0xb0/0xb0 > [10809.574847] [] free_object_rcu+0x89/0x90 > [10809.574948] [] rcu_process_callbacks+0x2b0/0x5f0 > [10809.575049] [] ? rcu_process_callbacks+0x26f/0x5f0 > [10809.575149] [] __do_softirq+0x10f/0x230 > [10809.575248] [] irq_exit+0xa8/0xb0 > [10809.575349] [] do_IRQ+0x65/0x110 > [10809.575450] [] ? mempool_free_slab+0x12/0x20 > [10809.575549] [] common_interrupt+0x89/0x89 > [10809.575650] > [10809.575748] [] ? mempool_free_slab+0x12/0x20 > [10809.575949] [] ? put_object+0x7/0x40 > [10809.576051] [] ? __delete_object+0x38/0x70 > [10809.576153] [] delete_object_full+0x18/0x20 > [10809.576254] [] kmemleak_free+0x31/0x80 > [10809.576355] [] kmem_cache_free+0x7a/0x160 > [10809.576456] [] mempool_free_slab+0x12/0x20 > [10809.576558] [] mempool_free+0x26/0x80 > [10809.576659] [] bvec_free+0x31/0x40 > [10809.576758] [] bio_free+0x34/0x60 > [10809.576858] [] bio_put+0x1e/0x30 > [10809.576962] [] blockio_endio+0x2d/0xe0 [scst_vdisk] > [10809.577067] [] bio_endio+0x3a/0x60 > [10809.577168] [] brd_make_request+0xe8/0x1ae [brd] > [10809.577273] [] ? generic_make_request_checks+0x10c/0x630 > [10809.577378] [] generic_make_request+0xd2/0x290 > [10809.577482] [] submit_bio+0x62/0x140 > [10809.577587] [] blockio_exec_rw+0x553/0x670 [scst_vdisk] > [10809.577697] [] ? blockio_exec_write_verify+0x50/0x50 [scst_vdisk] > [10809.577807] [] blockio_exec_read+0xd/0x20 [scst_vdisk] > [10809.577915] [] vdev_do_job+0xb3/0x310 [scst_vdisk] > [10809.578024] [] blockio_exec+0x74/0x180 [scst_vdisk] > [10809.578130] [] ? trace_hardirqs_on_caller+0x121/0x1b0 > [10809.578253] [] ? sgv_get_obj+0x72/0x1b0 [scst] > [10809.578374] [] scst_do_real_exec+0x94/0x3d0 [scst] > [10809.578500] [] ? scst_is_not_reservation_holder+0x13/0x60 [scst] > [10809.578629] [] ? __scst_check_local_events+0x19f/0x370 [scst] > [10809.578750] [] scst_exec_check_blocking+0xcc/0x340 [scst] > [10809.578876] [] scst_exec_check_sn+0x12d/0x2c0 [scst] > [10809.578982] [] ? finish_task_switch+0xa6/0x220 > [10809.579084] [] ? finish_task_switch+0x67/0x220 > [10809.579202] [] scst_process_active_cmd+0x3e2/0x760 [scst] > [10809.579319] [] ? trace_hardirqs_on+0xd/0x10 > [10809.579435] [] scst_do_job_active+0xe1/0x160 [scst] > [10809.579560] [] scst_cmd_thread+0x132/0x290 [scst] > [10809.579680] [] ? wake_atomic_t_function+0x60/0x60 > [10809.579797] [] ? scst_do_job_active+0x160/0x160 [scst] > [10809.579906] [] kthread+0xea/0x100 > [10809.580009] [] ? kthread_create_on_node+0x1f0/0x1f0 > [10809.580114] [] ret_from_fork+0x3f/0x70 > [10809.580219] [] ? kthread_create_on_node+0x1f0/0x1f0 > [10809.580321] Code: fc 48 83 c7 18 48 8b 55 08 53 48 89 f3 be 01 00 00 00 e8 e1 ce b2 ff 4c 89 e7 e8 49 00 b3 ff f6 c7 02 74 13 e8 0f a8 b2 ff 53 9d <65> ff 0d 56 2b a9 7e 5b 41 5c 5d c3 53 9d e8 6a 79 b2 ff eb eb