* iSCSI bug
@ 2015-03-05 15:56 Edgaras Lukoševičius
2015-03-05 19:22 ` Nicholas A. Bellinger
0 siblings, 1 reply; 3+ messages in thread
From: Edgaras Lukoševičius @ 2015-03-05 15:56 UTC (permalink / raw)
To: linux-scsi@vger.kernel.org
Hello,
we have a problem. iscsi connection is dropping constantly. After enabling iscsi debug following message popped up after iscsi connection got lost:
# Kernel
3.18.6-1.el7.elrepo.x86_64
Message from syslogd@storage-1 at Mar 5 15:55:12 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [iscsi_trx:11958]
Message from syslogd@storage-1 at Mar 5 15:55:40 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_trx:11958]
Message from syslogd@storage-1 at Mar 5 15:56:08 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_trx:11958]
Message from syslogd@storage-1 at Mar 5 15:56:48 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_trx:11958]
P.S.
Kernel 3.19 acts same I just don’t have debug messages for it. --
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: iSCSI bug
2015-03-05 15:56 iSCSI bug Edgaras Lukoševičius
@ 2015-03-05 19:22 ` Nicholas A. Bellinger
2015-03-18 4:57 ` Govindarajan
0 siblings, 1 reply; 3+ messages in thread
From: Nicholas A. Bellinger @ 2015-03-05 19:22 UTC (permalink / raw)
To: Edgaras Lukoševičius; +Cc: linux-scsi@vger.kernel.org, target-devel
Hi Edgaras,
Adding target-devel CC.
On Thu, 2015-03-05 at 17:56 +0200, Edgaras Lukoševičius wrote:
> Hello,
>
> we have a problem. iscsi connection is dropping constantly. After enabling iscsi debug following message popped up after iscsi connection got lost:
>
> # Kernel
> 3.18.6-1.el7.elrepo.x86_64
>
> Message from syslogd@storage-1 at Mar 5 15:55:12 ...
> kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [iscsi_trx:11958]
>
> Message from syslogd@storage-1 at Mar 5 15:55:40 ...
> kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_trx:11958]
>
> Message from syslogd@storage-1 at Mar 5 15:56:08 ...
> kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_trx:11958]
>
> Message from syslogd@storage-1 at Mar 5 15:56:48 ...
> kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [iscsi_trx:11958]
>
>
> P.S.
>
> Kernel 3.19 acts same I just don’t have debug messages for it. --
Please send along the dmesg output of your system so we can have a look
at the specific soft lockup stack back-traces.
Thanks,
--nab
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: iSCSI bug
2015-03-05 19:22 ` Nicholas A. Bellinger
@ 2015-03-18 4:57 ` Govindarajan
0 siblings, 0 replies; 3+ messages in thread
From: Govindarajan @ 2015-03-18 4:57 UTC (permalink / raw)
To: linux-scsi; +Cc: target-devel
I see the same problem. Would this help?
Mar 17 21:13:00 storage1 kernel: [119260.960821] TARGET_CORE[iSCSI]:
Detected NON_EXISTENT_LUN Access for 0x00000090
Mar 17 21:13:00 storage1 kernel: [119260.963752] ABORT_TASK: Sending
TMR_TASK_DOES_NOT_EXIST for ref_tag: 18170949
Mar 17 21:13:00 storage1 kernel: [119260.963759] ABORT_TASK: Sending
TMR_TASK_DOES_NOT_EXIST for ref_tag: 18170948
Mar 17 21:13:00 storage1 kernel: [119260.963761] ABORT_TASK: Sending
TMR_TASK_DOES_NOT_EXIST for ref_tag: 18170950
Mar 17 21:13:04 storage1 kernel: [119265.171082] Unexpected ret: -32 send
data 48
Mar 17 21:13:06 storage1 kernel: [119267.172218] TARGET_CORE[iSCSI]:
Detected NON_EXISTENT_LUN Access for 0x0000000a
Mar 17 21:13:06 storage1 kernel: [119267.175296] ABORT_TASK: Sending
TMR_TASK_DOES_NOT_EXIST for ref_tag: 182076996
Mar 17 21:13:06 storage1 kernel: [119267.175303] ABORT_TASK: Sending
TMR_TASK_DOES_NOT_EXIST for ref_tag: 182076994
Mar 17 21:13:06 storage1 kernel: [119267.175306] ABORT_TASK: Sending
TMR_TASK_DOES_NOT_EXIST for ref_tag: 182076997
Mar 17 21:13:07 storage1 kernel: [119267.796469] iSCSI Login timeout on
Network Portal 10.12.9.249:3260
Mar 17 21:13:17 storage1 kernel: [119278.498259] NMI watchdog: BUG: soft
lockup - CPU#1 stuck for 22s! [iscsi_trx:5637]
Mar 17 21:13:17 storage1 kernel: [119278.501397] Modules linked in:
ib_srpt(E) tcm_qla2xxx(E) tcm_loop(E) tcm_fc(E) iscsi_target_mod(E)
target_core_user(E) uio(E) target_core_pscsi(E) target_core_file(E)
target_core_iblock(E) target_core_mod(E) ib_cm(E) ib_sa(E) ib_mad(E)
ib_core(E) ib_addr(E) qla2xxx(E) libfc(E) scsi_transport_fc(E) configfs(E)
nfsd(E) auth_rpcgss(E) nfs_acl(E) nfs(E) lockd(E) grace(E) sunrpc(E)
fscache(E) gpio_ich(E) amdkfd(E) amd_iommu_v2(E) coretemp(E) kvm_intel(E)
kvm(E) radeon(E) serio_raw(E) 8021q(E) joydev(E) garp(E) ttm(E) mrp(E)
stp(E) llc(E) drm_kms_helper(E) drm(E) lpc_ich(E) i7core_edac(E)
edac_core(E) i5500_temp(E) ioatdma(E) 8250_fintek(E) mac_hid(E) shpchp(E)
bonding(E) btrfs(E) raid456(E) async_raid6_recov(E) async_memcpy(E)
bcache(E) async_pq(E) async_xor(E) raid10(E) async_tx(E) xor(E) raid6_pq(E)
raid1(E) hid_generic(E) raid0(E) multipath(E) igb(E) mpt2sas(E) ixgbe(E)
linear(E) raid_class(E) pata_acpi(E) i2c_algo_bit(E) dca(E) usbhid(E)
scsi_transport_sas(E) hid(E) ahci(E) pata_jmicron(E) psmouse(E) mdio(E)
ptp(E) libahci(E) pps_core(E) [last unloaded: target_core_mod]
Mar 17 21:13:17 storage1 kernel: [119278.501476] CPU: 1 PID: 5637 Comm:
iscsi_trx Tainted: G E 3.19.0-031900-generic #201502091451
Mar 17 21:13:17 storage1 kernel: [119278.501479] Hardware name: Supermicro
X8DTN/X8DTN, BIOS 2.1c 10/28/2011
Mar 17 21:13:17 storage1 kernel: [119278.501481] task: ffff8808f86c2740 ti:
ffff8808e7234000 task.ti: ffff8808e7234000
Mar 17 21:13:17 storage1 kernel: [119278.501484] RIP: 0010:
[<ffffffffc0982e35>] [<ffffffffc0982e35>]
iscsit_remove_cmd_from_immediate_queue+0x25/0x120 [iscsi_target_mod]
Mar 17 21:13:17 storage1 kernel: [119278.501501] RSP: 0018:ffff8808e7237ce8
EFLAGS: 00000246
Mar 17 21:13:17 storage1 kernel: [119278.501503] RAX: 0000000096da96da RBX:
0000000000000001 RCX: 00000000f9a9b4a6
Mar 17 21:13:17 storage1 kernel: [119278.501505] RDX: 00000000000096da RSI:
ffff881081bbc800 RDI: ffff881081bbcbe8
Mar 17 21:13:17 storage1 kernel: [119278.501507] RBP: ffff8808e7237d08 R08:
ffff880f45d68410 R09: 0000000000000101
Mar 17 21:13:17 storage1 kernel: [119278.501509] R10: 0000000000000001 R11:
000000000000001d R12: ffffffffffffff10
Mar 17 21:13:17 storage1 kernel: [119278.501511] R13: ffff8808e7237cd8 R14:
ffff880f45d68350 R15: 00000000f9ac4872
Mar 17 21:13:17 storage1 kernel: [119278.501513] FS: 0000000000000000(0000)
GS:ffff88091bc20000(0000) knlGS:0000000000000000
Mar 17 21:13:17 storage1 kernel: [119278.501516] CS: 0010 DS: 0000 ES: 0000
CR0: 000000008005003b
Mar 17 21:13:17 storage1 kernel: [119278.501518] CR2: 00000000012bb100 CR3:
0000000001c15000 CR4: 00000000000007e0
Mar 17 21:13:17 storage1 kernel: [119278.501520] Stack:
Mar 17 21:13:17 storage1 kernel: [119278.501521] ffff880f45d68140
ffff881081bbc800 0000000000000001 ffff880f45d68350
Mar 17 21:13:17 storage1 kernel: [119278.501525] ffff8808e7237d38
ffffffffc0984230 ffffffffc098d30b ffff880f45d68140
Mar 17 21:13:17 storage1 kernel: [119278.501528] 0000000000000001
0000000000000001 ffff8808e7237d78 ffffffffc098433e
Mar 17 21:13:17 storage1 kernel: [119278.501531] Call Trace:
Mar 17 21:13:17 storage1 kernel: [119278.501545] [<ffffffffc0984230>]
__iscsit_free_cmd+0x50/0xa0 [iscsi_target_mod]
Mar 17 21:13:17 storage1 kernel: [119278.501557] [<ffffffffc098d30b>] ?
iscsit_close_connection+0x3ab/0x640 [iscsi_target_mod]
Mar 17 21:13:17 storage1 kernel: [119278.501568] [<ffffffffc098433e>]
iscsit_free_cmd+0xbe/0x160 [iscsi_target_mod]
Mar 17 21:13:17 storage1 kernel: [119278.501580] [<ffffffffc098d324>]
iscsit_close_connection+0x3c4/0x640 [iscsi_target_mod]
Mar 17 21:13:17 storage1 kernel: [119278.501590] [<ffffffffc0978a63>]
iscsit_take_action_for_connection_exit+0x83/0x110 [iscsi_target_mod]
Mar 17 21:13:17 storage1 kernel: [119278.501601] [<ffffffffc098c3ee>]
iscsi_target_rx_thread+0x22e/0x320 [iscsi_target_mod]
Mar 17 21:13:17 storage1 kernel: [119278.501613] [<ffffffffc098c1c0>] ?
iscsi_target_tx_thread+0x220/0x220 [iscsi_target_mod]
Mar 17 21:13:17 storage1 kernel: [119278.501619] [<ffffffff81095d59>]
kthread+0xc9/0xe0
Mar 17 21:13:17 storage1 kernel: [119278.501624] [<ffffffff81095c90>] ?
flush_kthread_worker+0x90/0x90
Mar 17 21:13:17 storage1 kernel: [119278.501629] [<ffffffff817d1e7c>]
ret_from_fork+0x7c/0xb0
Mar 17 21:13:17 storage1 kernel: [119278.501632] [<ffffffff81095c90>] ?
flush_kthread_worker+0x90/0x90
Mar 17 21:13:17 storage1 kernel: [119278.501634] Code: 84 00 00 00 00 00 66
66 66 66 90 55 48 89 e5 41 56 4c 8d b6 e8 03 00 00 41 55 49 89 f5 41 54 49
89 fc 4c 89 f7 53 e8 6b e7 e4 c0 <41> 8b 84 24 d8 00 00 00 85 c0 0f 84 ab 00
00 00 49 8b 85 50 04
Mar 17 21:13:45 storage1 kernel: [119306.513810] NMI watchdog: BUG: soft
lockup - CPU#1 stuck for 22s! [iscsi_trx:5637]
Mar 17 21:13:45 storage1 kernel: [119306.516944] Modules linked in:
ib_srpt(E) tcm_qla2xxx(E) tcm_loop(E) tcm_fc(E) iscsi_target_mod(E)
target_core_user(E) uio(E) target_core_pscsi(E) target_core_file(E)
target_core_iblock(E) target_core_mod(E) ib_cm(E) ib_sa(E) ib_mad(E)
ib_core(E) ib_addr(E) qla2xxx(E) libfc(E) scsi_transport_fc(E) configfs(E)
nfsd(E) auth_rpcgss(E) nfs_acl(E) nfs(E) lockd(E) grace(E) sunrpc(E)
fscache(E) gpio_ich(E) amdkfd(E) amd_iommu_v2(E) coretemp(E) kvm_intel(E)
kvm(E) radeon(E) serio_raw(E) 8021q(E) joydev(E) garp(E) ttm(E) mrp(E)
stp(E) llc(E) drm_kms_helper(E) drm(E) lpc_ich(E) i7core_edac(E)
edac_core(E) i5500_temp(E) ioatdma(E) 8250_fintek(E) mac_hid(E) shpchp(E)
bonding(E) btrfs(E) raid456(E) async_raid6_recov(E) async_memcpy(E)
bcache(E) async_pq(E) async_xor(E) raid10(E) async_tx(E) xor(E) raid6_pq(E)
raid1(E) hid_generic(E) raid0(E) multipath(E) igb(E) mpt2sas(E) ixgbe(E)
linear(E) raid_class(E) pata_acpi(E) i2c_algo_bit(E) dca(E) usbhid(E)
scsi_transport_sas(E) hid(E) ahci(E) pata_jmicron(E) psmouse(E) mdio(E)
ptp(E) libahci(E) pps_core(E) [last unloaded: target_core_mod]
Mar 17 21:13:45 storage1 kernel: [119306.517024] CPU: 1 PID: 5637 Comm:
iscsi_trx Tainted: G EL 3.19.0-031900-generic #201502091451
Mar 17 21:13:45 storage1 kernel: [119306.517026] Hardware name: Supermicro
X8DTN/X8DTN, BIOS 2.1c 10/28/2011
Mar 17 21:13:45 storage1 kernel: [119306.517029] task: ffff8808f86c2740 ti:
ffff8808e7234000 task.ti: ffff8808e7234000
Mar 17 21:13:45 storage1 kernel: [119306.517031] RIP: 0010:
[<ffffffff817d1612>] [<ffffffff817d1612>] _raw_spin_unlock_bh+0x12/0x40
Mar 17 21:13:45 storage1 kernel: [119306.517040] RSP: 0018:ffff8808e7237cd8
EFLAGS: 00000282
Mar 17 21:13:45 storage1 kernel: [119306.517042] RAX: 0000000000000000 RBX:
0000000000000042 RCX: ffff880f45d68340
Mar 17 21:13:45 storage1 kernel: [119306.517044] RDX: 0000000000009a48 RSI:
ffff881081bbc800 RDI: ffff881081bbcbf0
Mar 17 21:13:45 storage1 kernel: [119306.517046] RBP: ffff8808e7237cd8 R08:
ffff880f45d68410 R09: 0000000000000101
Mar 17 21:13:45 storage1 kernel: [119306.517048] R10: 0000000000000042 R11:
0000000000000000 R12: 000000000000f8ec
Mar 17 21:13:45 storage1 kernel: [119306.517050] R13: ffff8808e7237cc8 R14:
ffff880f45d68410 R15: 0000000000000101
Mar 17 21:13:45 storage1 kernel: [119306.517053] FS: 0000000000000000(0000)
GS:ffff88091bc20000(0000) knlGS:0000000000000000
Mar 17 21:13:45 storage1 kernel: [119306.517055] CS: 0010 DS: 0000 ES: 0000
CR0: 000000008005003b
Mar 17 21:13:45 storage1 kernel: [119306.517057] CR2: 00000000012bb100 CR3:
0000000001c15000 CR4: 00000000000007e0
Mar 17 21:13:45 storage1 kernel: [119306.517059] Stack:
Mar 17 21:13:45 storage1 kernel: [119306.517060] ffff8808e7237d08
ffffffffc0983018 ffff880f45d68140 ffff881081bbc800
Mar 17 21:13:45 storage1 kernel: [119306.517064] 0000000000000001
ffff880f45d68350 ffff8808e7237d38 ffffffffc098423b
Mar 17 21:13:45 storage1 kernel: [119306.517067] ffffffffc09841e0
ffff880f45d68140 0000000000000001 0000000000000001
Mar 17 21:13:45 storage1 kernel: [119306.517070] Call Trace:
Mar 17 21:13:45 storage1 kernel: [119306.517086] [<ffffffffc0983018>]
iscsit_remove_cmd_from_response_queue+0xe8/0x120 [iscsi_target_mod]
Mar 17 21:13:45 storage1 kernel: [119306.517098] [<ffffffffc098423b>]
__iscsit_free_cmd+0x5b/0xa0 [iscsi_target_mod]
Mar 17 21:13:45 storage1 kernel: [119306.517110] [<ffffffffc09841e0>] ?
iscsit_free_queue_reqs_for_conn+0x150/0x150 [iscsi_target_mod]
Mar 17 21:13:45 storage1 kernel: [119306.517121] [<ffffffffc098430b>]
iscsit_free_cmd+0x8b/0x160 [iscsi_target_mod]
Mar 17 21:13:45 storage1 kernel: [119306.517133] [<ffffffffc098d324>]
iscsit_close_connection+0x3c4/0x640 [iscsi_target_mod]
Mar 17 21:13:45 storage1 kernel: [119306.517142] [<ffffffffc0978a63>]
iscsit_take_action_for_connection_exit+0x83/0x110 [iscsi_target_mod]
Mar 17 21:13:45 storage1 kernel: [119306.517154] [<ffffffffc098c3ee>]
iscsi_target_rx_thread+0x22e/0x320 [iscsi_target_mod]
Mar 17 21:13:45 storage1 kernel: [119306.517166] [<ffffffffc098c1c0>] ?
iscsi_target_tx_thread+0x220/0x220 [iscsi_target_mod]
Mar 17 21:13:45 storage1 kernel: [119306.517170] [<ffffffff81095d59>]
kthread+0xc9/0xe0
Mar 17 21:13:45 storage1 kernel: [119306.517174] [<ffffffff81095c90>] ?
flush_kthread_worker+0x90/0x90
Mar 17 21:13:45 storage1 kernel: [119306.517178] [<ffffffff817d1e7c>]
ret_from_fork+0x7c/0xb0
Mar 17 21:13:45 storage1 kernel: [119306.517182] [<ffffffff81095c90>] ?
flush_kthread_worker+0x90/0x90
Mar 17 21:13:45 storage1 kernel: [119306.517184] Code: 0f b7 0f 66 39 ca 75
f1 5d c3 66 66 66 90 66 66 90 eb d5 66 0f 1f 44 00 00 66 66 66 66 90 55 48
89 e5 66 66 66 66 90 66 83 07 02 <48> 8b 7d 08 be 00 02 00 00 e8 20 99 8a ff
5d c3 66 0f 1f 44 00
Mar 17 21:13:52 storage1 kernel: [119313.337597] INFO: rcu_sched self-
detected stall on CPU { 1} (t=15000 jiffies g=3864139 c=3864138 q=0)
Mar 17 21:13:52 storage1 kernel: [119313.340805] Task dump for CPU 1:
Mar 17 21:13:52 storage1 kernel: [119313.340808] iscsi_trx R running
task 0 5637 2 0x0000000c
Mar 17 21:13:52 storage1 kernel: [119313.340812] ffffffff81c54d00
ffff88091bc23d18 ffffffff810a3364 0000000000000001
Mar 17 21:13:52 storage1 kernel: [119313.340815] ffffffff81c54d00
ffff88091bc23d38 ffffffff810a5fdf ffff88091bc23d78
Mar 17 21:13:52 storage1 kernel: [119313.340818] ffffffff81c54e00
ffff88091bc23d68 ffffffff810d686b ffffffff81c54d00
Mar 17 21:13:52 storage1 kernel: [119313.340821] Call Trace:
Mar 17 21:13:52 storage1 kernel: [119313.340824] <IRQ>
[<ffffffff810a3364>] sched_show_task+0xb4/0x120
Mar 17 21:13:52 storage1 kernel: [119313.340834] [<ffffffff810a5fdf>]
dump_cpu_task+0x3f/0x50
Mar 17 21:13:52 storage1 kernel: [119313.340839] [<ffffffff810d686b>]
rcu_dump_cpu_stacks+0x8b/0xc0
Mar 17 21:13:52 storage1 kernel: [119313.340843] [<ffffffff810d8447>]
print_cpu_stall+0xd7/0x140
Mar 17 21:13:52 storage1 kernel: [119313.340848] [<ffffffff810d8e65>]
check_cpu_stall.isra.55+0xa5/0xb0
Mar 17 21:13:52 storage1 kernel: [119313.340852] [<ffffffff810d8ea8>]
__rcu_pending+0x38/0x170
Mar 17 21:13:52 storage1 kernel: [119313.340856] [<ffffffff810da9dd>]
rcu_check_callbacks+0x9d/0x190
Mar 17 21:13:52 storage1 kernel: [119313.340860] [<ffffffff810dfd08>]
update_process_times+0x38/0x70
Mar 17 21:13:52 storage1 kernel: [119313.340866] [<ffffffff810ef9f3>]
tick_sched_handle.isra.12+0x33/0x70
Mar 17 21:13:52 storage1 kernel: [119313.340869] [<ffffffff810efb2b>]
tick_sched_timer+0x4b/0x80
Mar 17 21:13:52 storage1 kernel: [119313.340873] [<ffffffff810e07c6>]
__run_hrtimer+0x76/0x230
Mar 17 21:13:52 storage1 kernel: [119313.340877] [<ffffffff810efae0>] ?
tick_nohz_handler+0xb0/0xb0
Mar 17 21:13:52 storage1 kernel: [119313.340881] [<ffffffff810e0fcf>]
hrtimer_interrupt+0xff/0x260
Mar 17 21:13:52 storage1 kernel: [119313.340887] [<ffffffff8104c339>]
local_apic_timer_interrupt+0x39/0x60
Mar 17 21:13:52 storage1 kernel: [119313.340892] [<ffffffff817d4f45>]
smp_apic_timer_interrupt+0x45/0x60
Mar 17 21:13:52 storage1 kernel: [119313.340895] [<ffffffff817d2fbd>]
apic_timer_interrupt+0x6d/0x80
Mar 17 21:13:52 storage1 kernel: [119313.340897] <EOI>
[<ffffffffc09833a6>] ? iscsit_free_r2ts_from_list+0x16/0x80
[iscsi_target_mod]
Mar 17 21:13:52 storage1 kernel: [119313.340921] [<ffffffffc098426d>]
__iscsit_free_cmd+0x8d/0xa0 [iscsi_target_mod]
Mar 17 21:13:52 storage1 kernel: [119313.340939] [<ffffffffc0b9323a>] ?
target_put_sess_cmd+0x4a/0xd0 [target_core_mod]
Mar 17 21:13:52 storage1 kernel: [119313.340950] [<ffffffffc098430b>]
iscsit_free_cmd+0x8b/0x160 [iscsi_target_mod]
Mar 17 21:13:52 storage1 kernel: [119313.340962] [<ffffffffc098d324>]
iscsit_close_connection+0x3c4/0x640 [iscsi_target_mod]
Mar 17 21:13:52 storage1 kernel: [119313.340972] [<ffffffffc0978a63>]
iscsit_take_action_for_connection_exit+0x83/0x110 [iscsi_target_mod]
Mar 17 21:13:52 storage1 kernel: [119313.340983] [<ffffffffc098c3ee>]
iscsi_target_rx_thread+0x22e/0x320 [iscsi_target_mod]
Mar 17 21:13:52 storage1 kernel: [119313.340995] [<ffffffffc098c1c0>] ?
iscsi_target_tx_thread+0x220/0x220 [iscsi_target_mod]
Mar 17 21:13:52 storage1 kernel: [119313.340999] [<ffffffff81095d59>]
kthread+0xc9/0xe0
Mar 17 21:13:52 storage1 kernel: [119313.341002] [<ffffffff81095c90>] ?
flush_kthread_worker+0x90/0x90
Mar 17 21:13:52 storage1 kernel: [119313.341006] [<ffffffff817d1e7c>]
ret_from_fork+0x7c/0xb0
Mar 17 21:13:52 storage1 kernel: [119313.341010] [<ffffffff81095c90>] ?
flush_kthread_worker+0x90/0x90
Mar 17 21:14:17 storage1 kernel: [119338.531583] NMI watchdog: BUG: soft
lockup - CPU#1 stuck for 22s! [iscsi_trx:5637]
Mar 17 21:14:17 storage1 kernel: [119338.534777] Modules linked in:
ib_srpt(E) tcm_qla2xxx(E) tcm_loop(E) tcm_fc(E) iscsi_target_mod(E)
target_core_user(E) uio(E) target_core_pscsi(E) target_core_file(E)
target_core_iblock(E) target_core_mod(E) ib_cm(E) ib_sa(E) ib_mad(E)
ib_core(E) ib_addr(E) qla2xxx(E) libfc(E) scsi_transport_fc(E) configfs(E)
nfsd(E) auth_rpcgss(E) nfs_acl(E) nfs(E) lockd(E) grace(E) sunrpc(E)
fscache(E) gpio_ich(E) amdkfd(E) amd_iommu_v2(E) coretemp(E) kvm_intel(E)
kvm(E) radeon(E) serio_raw(E) 8021q(E) joydev(E) garp(E) ttm(E) mrp(E)
stp(E) llc(E) drm_kms_helper(E) drm(E) lpc_ich(E) i7core_edac(E)
edac_core(E) i5500_temp(E) ioatdma(E) 8250_fintek(E) mac_hid(E) shpchp(E)
bonding(E) btrfs(E) raid456(E) async_raid6_recov(E) async_memcpy(E)
bcache(E) async_pq(E) async_xor(E) raid10(E) async_tx(E) xor(E) raid6_pq(E)
raid1(E) hid_generic(E) raid0(E) multipath(E) igb(E) mpt2sas(E) ixgbe(E)
linear(E) raid_class(E) pata_acpi(E) i2c_algo_bit(E) dca(E) usbhid(E)
scsi_transport_sas(E) hid(E) ahci(E) pata_jmicron(E) psmouse(E) mdio(E)
ptp(E) libahci(E) pps_core(E) [last unloaded: target_core_mod]
Mar 17 21:14:17 storage1 kernel: [119338.534843] CPU: 1 PID: 5637 Comm:
iscsi_trx Tainted: G EL 3.19.0-031900-generic #201502091451
Mar 17 21:14:17 storage1 kernel: [119338.534845] Hardware name: Supermicro
X8DTN/X8DTN, BIOS 2.1c 10/28/2011
Mar 17 21:14:17 storage1 kernel: [119338.534848] task: ffff8808f86c2740 ti:
ffff8808e7234000 task.ti: ffff8808e7234000
Mar 17 21:14:17 storage1 kernel: [119338.534850] RIP: 0010:
[<ffffffff817d15c7>] [<ffffffff817d15c7>] _raw_spin_lock_bh+0x27/0x60
Mar 17 21:14:17 storage1 kernel: [119338.534856] RSP: 0018:ffff8808e7237ce8
EFLAGS: 00000246
Mar 17 21:14:17 storage1 kernel: [119338.534858] RAX: 000000008f2e8f2e RBX:
ffff880f45d68410 RCX: 00000000eab4386c
Mar 17 21:14:17 storage1 kernel: [119338.534860] RDX: 0000000000008f2e RSI:
0000000000000001 RDI: ffff880f45d68224
Mar 17 21:14:17 storage1 kernel: [119338.534862] RBP: ffff8808e7237ce8 R08:
ffff880f45d68410 R09: 0000000000000101
Mar 17 21:14:17 storage1 kernel: [119338.534864] R10: 0000000000000001 R11:
000000000000001d R12: 0000000000000001
Mar 17 21:14:17 storage1 kernel: [119338.534866] R13: 000000000000001d R14:
0000000000000001 R15: ffff8808e7237cc8
Mar 17 21:14:17 storage1 kernel: [119338.534868] FS: 0000000000000000(0000)
GS:ffff88091bc20000(0000) knlGS:0000000000000000
Mar 17 21:14:17 storage1 kernel: [119338.534871] CS: 0010 DS: 0000 ES: 0000
CR0: 000000008005003b
Mar 17 21:14:17 storage1 kernel: [119338.534873] CR2: 00000000012bb100 CR3:
0000000001c15000 CR4: 00000000000007e0
Mar 17 21:14:17 storage1 kernel: [119338.534874] Stack:
Mar 17 21:14:17 storage1 kernel: [119338.534876] ffff8808e7237d08
ffffffffc09792a7 ffff880f45d68140 ffff881081bbc800
Mar 17 21:14:17 storage1 kernel: [119338.534879] ffff8808e7237d38
ffffffffc0984265 00ff0000000000ff ffff880f45d68140
Mar 17 21:14:17 storage1 kernel: [119338.534882] 0000000000000001
0000000000000001 ffff8808e7237d78 ffffffffc098433e
Mar 17 21:14:17 storage1 kernel: [119338.534886] Call Trace:
Mar 17 21:14:17 storage1 kernel: [119338.534898] [<ffffffffc09792a7>]
iscsit_stop_dataout_timer+0x27/0x90 [iscsi_target_mod]
Mar 17 21:14:17 storage1 kernel: [119338.534910] [<ffffffffc0984265>]
__iscsit_free_cmd+0x85/0xa0 [iscsi_target_mod]
Mar 17 21:14:17 storage1 kernel: [119338.534921] [<ffffffffc098433e>]
iscsit_free_cmd+0xbe/0x160 [iscsi_target_mod]
Mar 17 21:14:17 storage1 kernel: [119338.534931] [<ffffffffc0977643>] ?
iscsit_increment_maxcmdsn+0x13/0x90 [iscsi_target_mod]
Mar 17 21:14:17 storage1 kernel: [119338.534942] [<ffffffffc098d324>]
iscsit_close_connection+0x3c4/0x640 [iscsi_target_mod]
Mar 17 21:14:17 storage1 kernel: [119338.534952] [<ffffffffc0978a63>]
iscsit_take_action_for_connection_exit+0x83/0x110 [iscsi_target_mod]
Mar 17 21:14:17 storage1 kernel: [119338.534963] [<ffffffffc098c3ee>]
iscsi_target_rx_thread+0x22e/0x320 [iscsi_target_mod]
Mar 17 21:14:17 storage1 kernel: [119338.534975] [<ffffffffc098c1c0>] ?
iscsi_target_tx_thread+0x220/0x220 [iscsi_target_mod]
Mar 17 21:14:17 storage1 kernel: [119338.534979] [<ffffffff81095d59>]
kthread+0xc9/0xe0
Mar 17 21:14:17 storage1 kernel: [119338.534983] [<ffffffff81095c90>] ?
flush_kthread_worker+0x90/0x90
Mar 17 21:14:17 storage1 kernel: [119338.534987] [<ffffffff817d1e7c>]
ret_from_fork+0x7c/0xb0
Mar 17 21:14:17 storage1 kernel: [119338.534990] [<ffffffff81095c90>] ?
flush_kthread_worker+0x90/0x90
Mar 17 21:14:17 storage1 kernel: [119338.534992] Code: 00 00 00 00 66 66 66
66 90 55 65 81 05 ef a2 83 7e 00 02 00 00 48 89 e5 b8 00 00 02 00 f0 0f c1
07 89 c2 c1 ea 10 66 39 c2 75 02 <5d> c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 eb
11 66 2e 0f 1f 84 00
Mar 17 21:14:45 storage1 kernel: [119366.547135] NMI watchdog: BUG: soft
lockup - CPU#1 stuck for 22s! [iscsi_trx:5637]
Mar 17 21:14:45 storage1 kernel: [119366.550324] Modules linked in:
ib_srpt(E) tcm_qla2xxx(E) tcm_loop(E) tcm_fc(E) iscsi_target_mod(E)
target_core_user(E) uio(E) target_core_pscsi(E) target_core_file(E)
target_core_iblock(E) target_core_mod(E) ib_cm(E) ib_sa(E) ib_mad(E)
ib_core(E) ib_addr(E) qla2xxx(E) libfc(E) scsi_transport_fc(E) configfs(E)
nfsd(E) auth_rpcgss(E) nfs_acl(E) nfs(E) lockd(E) grace(E) sunrpc(E)
fscache(E) gpio_ich(E) amdkfd(E) amd_iommu_v2(E) coretemp(E) kvm_intel(E)
kvm(E) radeon(E) serio_raw(E) 8021q(E) joydev(E) garp(E) ttm(E) mrp(E)
stp(E) llc(E) drm_kms_helper(E) drm(E) lpc_ich(E) i7core_edac(E)
edac_core(E) i5500_temp(E) ioatdma(E) 8250_fintek(E) mac_hid(E) shpchp(E)
bonding(E) btrfs(E) raid456(E) async_raid6_recov(E) async_memcpy(E)
bcache(E) async_pq(E) async_xor(E) raid10(E) async_tx(E) xor(E) raid6_pq(E)
raid1(E) hid_generic(E) raid0(E) multipath(E) igb(E) mpt2sas(E) ixgbe(E)
linear(E) raid_class(E) pata_acpi(E) i2c_algo_bit(E) dca(E) usbhid(E)
scsi_transport_sas(E) hid(E) ahci(E) pata_jmicron(E) psmouse(E) mdio(E)
ptp(E) libahci(E) pps_core(E) [last unloaded: target_core_mod]
Mar 17 21:14:45 storage1 kernel: [119366.550388] CPU: 1 PID: 5637 Comm:
iscsi_trx Tainted: G EL 3.19.0-031900-generic #201502091451
Mar 17 21:14:45 storage1 kernel: [119366.550390] Hardware name: Supermicro
X8DTN/X8DTN, BIOS 2.1c 10/28/2011
Mar 17 21:14:45 storage1 kernel: [119366.550393] task: ffff8808f86c2740 ti:
ffff8808e7234000 task.ti: ffff8808e7234000
Mar 17 21:14:45 storage1 kernel: [119366.550395] RIP: 0010:
[<ffffffff8107af81>] [<ffffffff8107af81>] __local_bh_enable_ip+0x41/0x90
Mar 17 21:14:45 storage1 kernel: [119366.550401] RSP: 0018:ffff8808e7237cb8
EFLAGS: 00000246
Mar 17 21:14:45 storage1 kernel: [119366.550403] RAX: 0000000000000000 RBX:
00000000e3bd91ab RCX: 00000000e3b8754e
Mar 17 21:14:45 storage1 kernel: [119366.550405] RDX: 000000000000158a RSI:
0000000000000200 RDI: ffffffffc0982ef8
Mar 17 21:14:45 storage1 kernel: [119366.550407] RBP: ffff8808e7237cc8 R08:
ffff880f45d68410 R09: 0000000000000101
Mar 17 21:14:45 storage1 kernel: [119366.550409] R10: 0000000000000042 R11:
0000000000000000 R12: ffff880f45d68410
Mar 17 21:14:45 storage1 kernel: [119366.550411] R13: 0000000000000101 R14:
0000000000000001 R15: dead000000200200
Mar 17 21:14:45 storage1 kernel: [119366.550414] FS: 0000000000000000(0000)
GS:ffff88091bc20000(0000) knlGS:0000000000000000
Mar 17 21:14:45 storage1 kernel: [119366.550416] CS: 0010 DS: 0000 ES: 0000
CR0: 000000008005003b
Mar 17 21:14:45 storage1 kernel: [119366.550418] CR2: 00000000012bb100 CR3:
0000000001c15000 CR4: 00000000000007e0
Mar 17 21:14:45 storage1 kernel: [119366.550420] Stack:
Mar 17 21:14:45 storage1 kernel: [119366.550421] 0000000000000000
ffff880f45d68140 ffff8808e7237cd8 ffffffff817d1620
Mar 17 21:14:45 storage1 kernel: [119366.550424] ffff8808e7237d08
ffffffffc0982ef8 ffff880f45d68140 ffff881081bbc800
Mar 17 21:14:45 storage1 kernel: [119366.550427] 0000000000000001
ffff880f45d68350 ffff8808e7237d38 ffffffffc0984230
Mar 17 21:14:45 storage1 kernel: [119366.550431] Call Trace:
Mar 17 21:14:45 storage1 kernel: [119366.550436] [<ffffffff817d1620>]
_raw_spin_unlock_bh+0x20/0x40
Mar 17 21:14:45 storage1 kernel: [119366.550449] [<ffffffffc0982ef8>]
iscsit_remove_cmd_from_immediate_queue+0xe8/0x120 [iscsi_target_mod]
Mar 17 21:14:45 storage1 kernel: [119366.550460] [<ffffffffc0984230>]
__iscsit_free_cmd+0x50/0xa0 [iscsi_target_mod]
Mar 17 21:14:45 storage1 kernel: [119366.550471] [<ffffffffc098433e>]
iscsit_free_cmd+0xbe/0x160 [iscsi_target_mod]
Mar 17 21:14:45 storage1 kernel: [119366.550483] [<ffffffffc098d324>]
iscsit_close_connection+0x3c4/0x640 [iscsi_target_mod]
Mar 17 21:14:45 storage1 kernel: [119366.550493] [<ffffffffc0978a63>]
iscsit_take_action_for_connection_exit+0x83/0x110 [iscsi_target_mod]
Mar 17 21:14:45 storage1 kernel: [119366.550504] [<ffffffffc098c3ee>]
iscsi_target_rx_thread+0x22e/0x320 [iscsi_target_mod]
Mar 17 21:14:45 storage1 kernel: [119366.550516] [<ffffffffc098c1c0>] ?
iscsi_target_tx_thread+0x220/0x220 [iscsi_target_mod]
Mar 17 21:14:45 storage1 kernel: [119366.550520] [<ffffffff81095d59>]
kthread+0xc9/0xe0
Mar 17 21:14:45 storage1 kernel: [119366.550524] [<ffffffff81095c90>] ?
flush_kthread_worker+0x90/0x90
Mar 17 21:14:45 storage1 kernel: [119366.550528] [<ffffffff817d1e7c>]
ret_from_fork+0x7c/0xb0
Mar 17 21:14:45 storage1 kernel: [119366.550531] [<ffffffff81095c90>] ?
flush_kthread_worker+0x90/0x90
Mar 17 21:14:45 storage1 kernel: [119366.550533] Code: 7e a9 00 00 0f 00 74
32 80 3d 25 48 c7 00 01 75 37 83 eb 01 f7 db 65 01 1d 2d 09 f9 7e 65 8b 05
26 09 f9 7e a9 00 ff 1f 00 74 37 <65> ff 0d 18 09 f9 7e 48 83 c4 08 5b 5d c3
90 9c 58 66 66 90 66
Mar 17 21:15:13 storage1 kernel: [119394.562687] NMI watchdog: BUG: soft
lockup - CPU#1 stuck for 22s! [iscsi_trx:5637]
Mar 17 21:15:13 storage1 kernel: [119394.565872] Modules linked in:
ib_srpt(E) tcm_qla2xxx(E) tcm_loop(E) tcm_fc(E) iscsi_target_mod(E)
target_core_user(E) uio(E) target_core_pscsi(E) target_core_file(E)
target_core_iblock(E) target_core_mod(E) ib_cm(E) ib_sa(E) ib_mad(E)
ib_core(E) ib_addr(E) qla2xxx(E) libfc(E) scsi_transport_fc(E) configfs(E)
nfsd(E) auth_rpcgss(E) nfs_acl(E) nfs(E) lockd(E) grace(E) sunrpc(E)
fscache(E) gpio_ich(E) amdkfd(E) amd_iommu_v2(E) coretemp(E) kvm_intel(E)
kvm(E) radeon(E) serio_raw(E) 8021q(E) joydev(E) garp(E) ttm(E) mrp(E)
stp(E) llc(E) drm_kms_helper(E) drm(E) lpc_ich(E) i7core_edac(E)
edac_core(E) i5500_temp(E) ioatdma(E) 8250_fintek(E) mac_hid(E) shpchp(E)
bonding(E) btrfs(E) raid456(E) async_raid6_recov(E) async_memcpy(E)
bcache(E) async_pq(E) async_xor(E) raid10(E) async_tx(E) xor(E) raid6_pq(E)
raid1(E) hid_generic(E) raid0(E) multipath(E) igb(E) mpt2sas(E) ixgbe(E)
linear(E) raid_class(E) pata_acpi(E) i2c_algo_bit(E) dca(E) usbhid(E)
scsi_transport_sas(E) hid(E) ahci(E) pata_jmicron(E) psmouse(E) mdio(E)
ptp(E) libahci(E) pps_core(E) [last unloaded: target_core_mod]
Mar 17 21:15:13 storage1 kernel: [119394.565937] CPU: 1 PID: 5637 Comm:
iscsi_trx Tainted: G EL 3.19.0-031900-generic #201502091451
Mar 17 21:15:13 storage1 kernel: [119394.565939] Hardware name: Supermicro
X8DTN/X8DTN, BIOS 2.1c 10/28/2011
Mar 17 21:15:13 storage1 kernel: [119394.565941] task: ffff8808f86c2740 ti:
ffff8808e7234000 task.ti: ffff8808e7234000
Mar 17 21:15:13 storage1 kernel: [119394.565943] RIP: 0010:
[<ffffffff8107af50>] [<ffffffff8107af50>] __local_bh_enable_ip+0x10/0x90
Mar 17 21:15:13 storage1 kernel: [119394.565949] RSP: 0018:ffff8808e7237cc8
EFLAGS: 00000292
Mar 17 21:15:13 storage1 kernel: [119394.565951] RAX: 0000000000000000 RBX:
00000000000091b8 RCX: ffff880f45d68340
Mar 17 21:15:13 storage1 kernel: [119394.565953] RDX: 0000000000002dcc RSI:
0000000000000200 RDI: ffffffffc09792e9
Mar 17 21:15:13 storage1 kernel: [119394.565955] RBP: ffff8808e7237cd8 R08:
ffff880f45d68410 R09: 0000000000000101
Mar 17 21:15:13 storage1 kernel: [119394.565957] R10: 0000000000000000 R11:
000000000000001d R12: ffff880f45d68410
Mar 17 21:15:13 storage1 kernel: [119394.565959] R13: 0000000000000101 R14:
0000000000000000 R15: 000000000000001d
Mar 17 21:15:13 storage1 kernel: [119394.565962] FS: 0000000000000000(0000)
GS:ffff88091bc20000(0000) knlGS:0000000000000000
Mar 17 21:15:13 storage1 kernel: [119394.565964] CS: 0010 DS: 0000 ES: 0000
CR0: 000000008005003b
Mar 17 21:15:13 storage1 kernel: [119394.565966] CR2: 00000000012bb100 CR3:
0000000001c15000 CR4: 00000000000007e0
Mar 17 21:15:13 storage1 kernel: [119394.565967] Stack:
Mar 17 21:15:13 storage1 kernel: [119394.565969] ffff8808e7237cd8
ffff880f45d68224 ffff8808e7237ce8 ffffffff817d1620
Mar 17 21:15:13 storage1 kernel: [119394.565972] ffff8808e7237d08
ffffffffc09792e9 ffff880f45d68140 ffff881081bbc800
Mar 17 21:15:13 storage1 kernel: [119394.565975] ffff8808e7237d38
ffffffffc0984265 00000000c7d0c7d0 ffff880f45d68140
Mar 17 21:15:13 storage1 kernel: [119394.565979] Call Trace:
Mar 17 21:15:13 storage1 kernel: [119394.565984] [<ffffffff817d1620>]
_raw_spin_unlock_bh+0x20/0x40
Mar 17 21:15:13 storage1 kernel: [119394.565995] [<ffffffffc09792e9>]
iscsit_stop_dataout_timer+0x69/0x90 [iscsi_target_mod]
Mar 17 21:15:13 storage1 kernel: [119394.566007] [<ffffffffc0984265>]
__iscsit_free_cmd+0x85/0xa0 [iscsi_target_mod]
Mar 17 21:15:13 storage1 kernel: [119394.566018] [<ffffffffc098430b>]
iscsit_free_cmd+0x8b/0x160 [iscsi_target_mod]
Mar 17 21:15:13 storage1 kernel: [119394.566030] [<ffffffffc098d324>]
iscsit_close_connection+0x3c4/0x640 [iscsi_target_mod]
Mar 17 21:15:13 storage1 kernel: [119394.566040] [<ffffffffc0978a63>]
iscsit_take_action_for_connection_exit+0x83/0x110 [iscsi_target_mod]
Mar 17 21:15:13 storage1 kernel: [119394.566051] [<ffffffffc098c3ee>]
iscsi_target_rx_thread+0x22e/0x320 [iscsi_target_mod]
Mar 17 21:15:13 storage1 kernel: [119394.566063] [<ffffffffc098c1c0>] ?
iscsi_target_tx_thread+0x220/0x220 [iscsi_target_mod]
Mar 17 21:15:13 storage1 kernel: [119394.566067] [<ffffffff81095d59>]
kthread+0xc9/0xe0
Mar 17 21:15:13 storage1 kernel: [119394.566071] [<ffffffff81095c90>] ?
flush_kthread_worker+0x90/0x90
Mar 17 21:15:13 storage1 kernel: [119394.566075] [<ffffffff817d1e7c>]
ret_from_fork+0x7c/0xb0
Mar 17 21:15:13 storage1 kernel: [119394.566078] [<ffffffff81095c90>] ?
flush_kthread_worker+0x90/0x90
Mar 17 21:15:13 storage1 kernel: [119394.566080] Code: 89 df 57 9d 66 66 90
66 90 48 83 c4 08 5b 5d c3 0f 1f 00 e8 23 8d 75 00 eb e5 90 66 66 66 66 90
55 48 89 e5 53 89 f3 48 83 ec 08 <65> 8b 05 49 09 f9 7e a9 00 00 0f 00 74 32
80 3d 25 48 c7 00 01
Mar 17 21:15:41 storage1 kernel: [119422.578240] NMI watchdog: BUG: soft
lockup - CPU#1 stuck for 22s! [iscsi_trx:5637]
Mar 17 21:15:41 storage1 kernel: [119422.581431] Modules linked in:
ib_srpt(E) tcm_qla2xxx(E) tcm_loop(E) tcm_fc(E) iscsi_target_mod(E)
target_core_user(E) uio(E) target_core_pscsi(E) target_core_file(E)
target_core_iblock(E) target_core_mod(E) ib_cm(E) ib_sa(E) ib_mad(E)
ib_core(E) ib_addr(E) qla2xxx(E) libfc(E) scsi_transport_fc(E) configfs(E)
nfsd(E) auth_rpcgss(E) nfs_acl(E) nfs(E) lockd(E) grace(E) sunrpc(E)
fscache(E) gpio_ich(E) amdkfd(E) amd_iommu_v2(E) coretemp(E) kvm_intel(E)
kvm(E) radeon(E) serio_raw(E) 8021q(E) joydev(E) garp(E) ttm(E) mrp(E)
stp(E) llc(E) drm_kms_helper(E) drm(E) lpc_ich(E) i7core_edac(E)
edac_core(E) i5500_temp(E) ioatdma(E) 8250_fintek(E) mac_hid(E) shpchp(E)
bonding(E) btrfs(E) raid456(E) async_raid6_recov(E) async_memcpy(E)
bcache(E) async_pq(E) async_xor(E) raid10(E) async_tx(E) xor(E) raid6_pq(E)
raid1(E) hid_generic(E) raid0(E) multipath(E) igb(E) mpt2sas(E) ixgbe(E)
linear(E) raid_class(E) pata_acpi(E) i2c_algo_bit(E) dca(E) usbhid(E)
scsi_transport_sas(E) hid(E) ahci(E) pata_jmicron(E) psmouse(E) mdio(E)
ptp(E) libahci(E) pps_core(E) [last unloaded: target_core_mod]
Mar 17 21:15:41 storage1 kernel: [119422.581495] CPU: 1 PID: 5637 Comm:
iscsi_trx Tainted: G EL 3.19.0-031900-generic #201502091451
Mar 17 21:15:41 storage1 kernel: [119422.581497] Hardware name: Supermicro
X8DTN/X8DTN, BIOS 2.1c 10/28/2011
Mar 17 21:15:41 storage1 kernel: [119422.581499] task: ffff8808f86c2740 ti:
ffff8808e7234000 task.ti: ffff8808e7234000
Mar 17 21:15:41 storage1 kernel: [119422.581501] RIP: 0010:
[<ffffffff817d15bd>] [<ffffffff817d15bd>] _raw_spin_lock_bh+0x1d/0x60
Mar 17 21:15:41 storage1 kernel: [119422.581507] RSP: 0018:ffff8808e7237ce8
EFLAGS: 00000286
Mar 17 21:15:41 storage1 kernel: [119422.581509] RAX: 00000000c828c828 RBX:
0000000070000000 RCX: ffff880f45d68340
Mar 17 21:15:41 storage1 kernel: [119422.581511] RDX: 0000000000000001 RSI:
0000000000000001 RDI: ffff880f45d68224
Mar 17 21:15:41 storage1 kernel: [119422.581513] RBP: ffff8808e7237ce8 R08:
ffff880f45d68410 R09: 0000000000000101
Mar 17 21:15:41 storage1 kernel: [119422.581515] R10: 0000000000000000 R11:
0000000000000001 R12: 0000000000000286
Mar 17 21:15:41 storage1 kernel: [119422.581517] R13: ffff8808e7237cd8 R14:
0000000000003b1e R15: 0000000000000000
Mar 17 21:15:41 storage1 kernel: [119422.581519] FS: 0000000000000000(0000)
GS:ffff88091bc20000(0000) knlGS:0000000000000000
Mar 17 21:15:41 storage1 kernel: [119422.581521] CS: 0010 DS: 0000 ES: 0000
CR0: 000000008005003b
Mar 17 21:15:41 storage1 kernel: [119422.581523] CR2: 00000000012bb100 CR3:
0000000001c15000 CR4: 00000000000007e0
Mar 17 21:15:41 storage1 kernel: [119422.581525] Stack:
Mar 17 21:15:41 storage1 kernel: [119422.581527] ffff8808e7237d08
ffffffffc09792a7 ffff880f45d68140 ffff881081bbc800
Mar 17 21:15:41 storage1 kernel: [119422.581530] ffff8808e7237d38
ffffffffc0984265 000000003cae3cae ffff880f45d68140
Mar 17 21:15:41 storage1 kernel: [119422.581533] 0000000000000001
0000000000000001 ffff8808e7237d78 ffffffffc098430b
Mar 17 21:15:41 storage1 kernel: [119422.581536] Call Trace:
Mar 17 21:15:41 storage1 kernel: [119422.581548] [<ffffffffc09792a7>]
iscsit_stop_dataout_timer+0x27/0x90 [iscsi_target_mod]
Mar 17 21:15:41 storage1 kernel: [119422.581560] [<ffffffffc0984265>]
__iscsit_free_cmd+0x85/0xa0 [iscsi_target_mod]
Mar 17 21:15:41 storage1 kernel: [119422.581571] [<ffffffffc098430b>]
iscsit_free_cmd+0x8b/0x160 [iscsi_target_mod]
Mar 17 21:15:41 storage1 kernel: [119422.581583] [<ffffffffc098d324>]
iscsit_close_connection+0x3c4/0x640 [iscsi_target_mod]
Mar 17 21:15:41 storage1 kernel: [119422.581593] [<ffffffffc0978a63>]
iscsit_take_action_for_connection_exit+0x83/0x110 [iscsi_target_mod]
Mar 17 21:15:41 storage1 kernel: [119422.581604] [<ffffffffc098c3ee>]
iscsi_target_rx_thread+0x22e/0x320 [iscsi_target_mod]
Mar 17 21:15:41 storage1 kernel: [119422.581616] [<ffffffffc098c1c0>] ?
iscsi_target_tx_thread+0x220/0x220 [iscsi_target_mod]
Mar 17 21:15:41 storage1 kernel: [119422.581620] [<ffffffff81095d59>]
kthread+0xc9/0xe0
Mar 17 21:15:41 storage1 kernel: [119422.581624] [<ffffffff81095c90>] ?
flush_kthread_worker+0x90/0x90
Mar 17 21:15:41 storage1 kernel: [119422.581628] [<ffffffff817d1e7c>]
ret_from_fork+0x7c/0xb0
Mar 17 21:15:41 storage1 kernel: [119422.581631] [<ffffffff81095c90>] ?
flush_kthread_worker+0x90/0x90
Mar 17 21:15:41 storage1 kernel: [119422.581633] Code: e0 8e ff 5d c3 66 0f
1f 84 00 00 00 00 00 66 66 66 66 90 55 65 81 05 ef a2 83 7e 00 02 00 00 48
89 e5 b8 00 00 02 00 f0 0f c1 07 <89> c2 c1 ea 10 66 39 c2 75 02 5d c3 83 e2
fe 0f b7 f2 b8 00 80
Mar 17 21:16:09 storage1 kernel: [119450.593796] NMI watchdog: BUG: soft
lockup - CPU#1 stuck for 23s! [iscsi_trx:5637]
Mar 17 21:16:09 storage1 kernel: [119450.596986] Modules linked in:
ib_srpt(E) tcm_qla2xxx(E) tcm_loop(E) tcm_fc(E) iscsi_target_mod(E)
target_core_user(E) uio(E) target_core_pscsi(E) target_core_file(E)
target_core_iblock(E) target_core_mod(E) ib_cm(E) ib_sa(E) ib_mad(E)
ib_core(E) ib_addr(E) qla2xxx(E) libfc(E) scsi_transport_fc(E) configfs(E)
nfsd(E) auth_rpcgss(E) nfs_acl(E) nfs(E) lockd(E) grace(E) sunrpc(E)
fscache(E) gpio_ich(E) amdkfd(E) amd_iommu_v2(E) coretemp(E) kvm_intel(E)
kvm(E) radeon(E) serio_raw(E) 8021q(E) joydev(E) garp(E) ttm(E) mrp(E)
stp(E) llc(E) drm_kms_helper(E) drm(E) lpc_ich(E) i7core_edac(E)
edac_core(E) i5500_temp(E) ioatdma(E) 8250_fintek(E) mac_hid(E) shpchp(E)
bonding(E) btrfs(E) raid456(E) async_raid6_recov(E) async_memcpy(E)
bcache(E) async_pq(E) async_xor(E) raid10(E) async_tx(E) xor(E) raid6_pq(E)
raid1(E) hid_generic(E) raid0(E) multipath(E) igb(E) mpt2sas(E) ixgbe(E)
linear(E) raid_class(E) pata_acpi(E) i2c_algo_bit(E) dca(E) usbhid(E)
scsi_transport_sas(E) hid(E) ahci(E) pata_jmicron(E) psmouse(E) mdio(E)
ptp(E) libahci(E) pps_core(E) [last unloaded: target_core_mod]
Mar 17 21:16:09 storage1 kernel: [119450.597050] CPU: 1 PID: 5637 Comm:
iscsi_trx Tainted: G EL 3.19.0-031900-generic #201502091451
Mar 17 21:16:09 storage1 kernel: [119450.597052] Hardware name: Supermicro
X8DTN/X8DTN, BIOS 2.1c 10/28/2011
Mar 17 21:16:09 storage1 kernel: [119450.597054] task: ffff8808f86c2740 ti:
ffff8808e7234000 task.ti: ffff8808e7234000
Mar 17 21:16:09 storage1 kernel: [119450.597056] RIP: 0010:
[<ffffffff8107af67>] [<ffffffff8107af67>] __local_bh_enable_ip+0x27/0x90
Mar 17 21:16:09 storage1 kernel: [119450.597061] RSP: 0018:ffff8808e7237d58
EFLAGS: 00000202
Mar 17 21:16:09 storage1 kernel: [119450.597063] RAX: 0000000000000246 RBX:
ffffffff817d1620 RCX: ffff880f45d68340
Mar 17 21:16:09 storage1 kernel: [119450.597065] RDX: ffff880f45d68340 RSI:
0000000000000200 RDI: ffffffffc098d30b
Mar 17 21:16:09 storage1 kernel: [119450.597067] RBP: ffff8808e7237d68 R08:
ffff880f45d68410 R09: 0000000000000101
Mar 17 21:16:09 storage1 kernel: [119450.597069] R10: 0000000000000001 R11:
dead000000200200 R12: ffff880f45d68140
Mar 17 21:16:09 storage1 kernel: [119450.597071] R13: 0000000000000000 R14:
0000000000000246 R15: ffff8808e7237cc8
Mar 17 21:16:09 storage1 kernel: [119450.597074] FS: 0000000000000000(0000)
GS:ffff88091bc20000(0000) knlGS:0000000000000000
Mar 17 21:16:09 storage1 kernel: [119450.597076] CS: 0010 DS: 0000 ES: 0000
CR0: 000000008005003b
Mar 17 21:16:09 storage1 kernel: [119450.597078] CR2: 00000000012bb100 CR3:
0000000001c15000 CR4: 00000000000007e0
Mar 17 21:16:09 storage1 kernel: [119450.597080] Stack:
Mar 17 21:16:09 storage1 kernel: [119450.597081] ffff8808e66b8800
ffff881081bbc800 ffff8808e7237d78 ffffffff817d1620
Mar 17 21:16:09 storage1 kernel: [119450.597085] ffff8808e7237dd8
ffffffffc098d30b ffff8808e7237de8 ffff8808e66b8800
Mar 17 21:16:09 storage1 kernel: [119450.597088] ffff881081bbcbe0
ffff881081bbcc40 ffff8808f86c2740 ffff881081bbc800
Mar 17 21:16:09 storage1 kernel: [119450.597091] Call Trace:
Mar 17 21:16:09 storage1 kernel: [119450.597096] [<ffffffff817d1620>]
_raw_spin_unlock_bh+0x20/0x40
Mar 17 21:16:09 storage1 kernel: [119450.597108] [<ffffffffc098d30b>]
iscsit_close_connection+0x3ab/0x640 [iscsi_target_mod]
Mar 17 21:16:09 storage1 kernel: [119450.597118] [<ffffffffc0978a63>]
iscsit_take_action_for_connection_exit+0x83/0x110 [iscsi_target_mod]
Mar 17 21:16:09 storage1 kernel: [119450.597130] [<ffffffffc098c3ee>]
iscsi_target_rx_thread+0x22e/0x320 [iscsi_target_mod]
Mar 17 21:16:09 storage1 kernel: [119450.597142] [<ffffffffc098c1c0>] ?
iscsi_target_tx_thread+0x220/0x220 [iscsi_target_mod]
Mar 17 21:16:09 storage1 kernel: [119450.597146] [<ffffffff81095d59>]
kthread+0xc9/0xe0
Mar 17 21:16:09 storage1 kernel: [119450.597149] [<ffffffff81095c90>] ?
flush_kthread_worker+0x90/0x90
Mar 17 21:16:09 storage1 kernel: [119450.597153] [<ffffffff817d1e7c>]
ret_from_fork+0x7c/0xb0
Mar 17 21:16:09 storage1 kernel: [119450.597157] [<ffffffff81095c90>] ?
flush_kthread_worker+0x90/0x90
Mar 17 21:16:09 storage1 kernel: [119450.597159] Code: 00 eb e5 90 66 66 66
66 90 55 48 89 e5 53 89 f3 48 83 ec 08 65 8b 05 49 09 f9 7e a9 00 00 0f 00
74 32 80 3d 25 48 c7 00 01 75 37 <83> eb 01 f7 db 65 01 1d 2d 09 f9 7e 65 8b
05 26 09 f9 7e a9 00
Mar 17 21:16:26 storage1 kernel: [119466.911040] INFO: task
bcache_writebac:425 blocked for more than 120 seconds.
Mar 17 21:16:26 storage1 kernel: [119466.914309] Tainted: G
EL 3.19.0-031900-generic #201502091451
Mar 17 21:16:26 storage1 kernel: [119466.917723] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 17 21:16:26 storage1 kernel: [119466.921191] bcache_writebac D
ffff8800ba5dfc38 0 425 2 0x00000000
Mar 17 21:16:26 storage1 kernel: [119466.921194] ffff8800ba5dfc38
ffff88093fffbf00 ffff8800ba5dffd8 0000000000013e40
Mar 17 21:16:26 storage1 kernel: [119466.921197] ffff88121147db00
ffffffff81c1c500 ffff880914cff5c0 ffffffff8101358e
Mar 17 21:16:26 storage1 kernel: [119466.921199] ffff88090d830bb0
7fffffffffffffff ffff880914cff5c0 0000000000000002
Mar 17 21:16:26 storage1 kernel: [119466.921202] Call Trace:
Mar 17 21:16:26 storage1 kernel: [119466.921209] [<ffffffff8101358e>] ?
__switch_to+0xbe/0x5b0
Mar 17 21:16:26 storage1 kernel: [119466.921212] [<ffffffff817cdd29>]
schedule+0x29/0x70
Mar 17 21:16:26 storage1 kernel: [119466.921215] [<ffffffff817d0ab5>]
schedule_timeout+0x1b5/0x210
Mar 17 21:16:26 storage1 kernel: [119466.921218] [<ffffffff817cfea7>]
__down_common+0xe0/0x134
Mar 17 21:16:26 storage1 kernel: [119466.921221] [<ffffffff811ccd00>] ?
get_task_policy+0x20/0x60
Mar 17 21:16:26 storage1 kernel: [119466.921224] [<ffffffff817cff6e>]
__down+0x1d/0x1f
Mar 17 21:16:26 storage1 kernel: [119466.921227] [<ffffffff810bdba1>]
down+0x41/0x50
Mar 17 21:16:26 storage1 kernel: [119466.921243] [<ffffffffc0332744>]
read_dirty+0x1f4/0x440 [bcache]
Mar 17 21:16:26 storage1 kernel: [119466.921253] [<ffffffffc0331cc0>] ?
bch_crc64+0x50/0x50 [bcache]
Mar 17 21:16:26 storage1 kernel: [119466.921262] [<ffffffffc0332c2c>]
bch_writeback_thread+0x15c/0x200 [bcache]
Mar 17 21:16:26 storage1 kernel: [119466.921272] [<ffffffffc0332ad0>] ?
write_dirty+0xc0/0xc0 [bcache]
Mar 17 21:16:26 storage1 kernel: [119466.921274] [<ffffffff81095d59>]
kthread+0xc9/0xe0
Mar 17 21:16:26 storage1 kernel: [119466.921277] [<ffffffff81095c90>] ?
flush_kthread_worker+0x90/0x90
Mar 17 21:16:26 storage1 kernel: [119466.921280] [<ffffffff817d1e7c>]
ret_from_fork+0x7c/0xb0
Mar 17 21:16:26 storage1 kernel: [119466.921282] [<ffffffff81095c90>] ?
flush_kthread_worker+0x90/0x90
Mar 17 21:16:26 storage1 kernel: [119466.921288] INFO: task jbd2/dm-1-8:856
blocked for more than 120 seconds.
Mar 17 21:16:26 storage1 kernel: [119466.924848] Tainted: G
EL 3.19.0-031900-generic #201502091451
Mar 17 21:16:26 storage1 kernel: [119466.928460] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 17 21:16:26 storage1 kernel: [119466.932112] jbd2/dm-1-8 D
ffff88120f917b58 0 856 2 0x00000000
Mar 17 21:16:26 storage1 kernel: [119466.932114] ffff88120f917b58
ffff88120f917b28 ffff88120f917fd8 0000000000013e40
Mar 17 21:16:26 storage1 kernel: [119466.932123] ffff88121147db00
ffff8809157c3ae0 ffff88121171e220 ffff88120f917b58
Mar 17 21:16:26 storage1 kernel: [119466.932125] ffff88123fc74738
ffff88121171e220 ffffffff817ce750 ffff88120f917c10
Mar 17 21:16:26 storage1 kernel: [119466.932127] Call Trace:
Mar 17 21:16:26 storage1 kernel: [119466.932131] [<ffffffff817ce750>] ?
bit_wait_timeout+0x80/0x80
Mar 17 21:16:26 storage1 kernel: [119466.932133] [<ffffffff817cdd29>]
schedule+0x29/0x70
Mar 17 21:16:26 storage1 kernel: [119466.932135] [<ffffffff817cddfe>]
io_schedule+0x8e/0xd0
Mar 17 21:16:26 storage1 kernel: [119466.932138] [<ffffffff817ce77c>]
bit_wait_io+0x2c/0x50
Mar 17 21:16:26 storage1 kernel: [119466.932146] [<ffffffff817ce4e5>]
__wait_on_bit+0x65/0x90
Mar 17 21:16:26 storage1 kernel: [119466.932150] [<ffffffff817ce750>] ?
bit_wait_timeout+0x80/0x80
Mar 17 21:16:26 storage1 kernel: [119466.932152] [<ffffffff817ce63c>]
out_of_line_wait_on_bit+0x7c/0x90
Mar 17 21:16:26 storage1 kernel: [119466.932155] [<ffffffff810b75b0>] ?
wake_atomic_t_function+0x40/0x40
Mar 17 21:16:26 storage1 kernel: [119466.932163] [<ffffffff81227686>]
__wait_on_buffer+0x36/0x40
Mar 17 21:16:26 storage1 kernel: [119466.932167] [<ffffffff812c3e05>]
jbd2_journal_commit_transaction+0x1285/0x1440
Mar 17 21:16:26 storage1 kernel: [119466.932171] [<ffffffff812c7923>]
kjournald2+0xb3/0x230
Mar 17 21:16:26 storage1 kernel: [119466.932174] [<ffffffff810b7530>] ?
prepare_to_wait_event+0x100/0x100
Mar 17 21:16:26 storage1 kernel: [119466.932176] [<ffffffff812c7870>] ?
commit_timeout+0x10/0x10
Mar 17 21:16:26 storage1 kernel: [119466.932179] [<ffffffff81095d59>]
kthread+0xc9/0xe0
Mar 17 21:16:26 storage1 kernel: [119466.932181] [<ffffffff81095c90>] ?
flush_kthread_worker+0x90/0x90
Mar 17 21:16:26 storage1 kernel: [119466.932184] [<ffffffff817d1e7c>]
ret_from_fork+0x7c/0xb0
Mar 17 21:16:26 storage1 kernel: [119466.932186] [<ffffffff81095c90>] ?
flush_kthread_worker+0x90/0x90
Mar 17 21:16:26 storage1 kernel: [119466.932189] INFO: task jbd2/dm-2-8:1322
blocked for more than 120 seconds.
Mar 17 21:16:26 storage1 kernel: [119466.935823] Tainted: G
EL 3.19.0-031900-generic #201502091451
Mar 17 21:16:26 storage1 kernel: [119466.939532] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 17 21:16:26 storage1 kernel: [119466.943336] jbd2/dm-2-8 D
ffff88120f8efb58 0 1322 2 0x00000000
Mar 17 21:16:26 storage1 kernel: [119466.943339] ffff88120f8efb58
ffff88120f8efb28 ffff88120f8effd8 0000000000013e40
Mar 17 21:16:26 storage1 kernel: [119466.943341] ffff88121147db00
ffff8809157c2740 ffff88120f235850 ffff88120f8efb58
Mar 17 21:16:26 storage1 kernel: [119466.943344] ffff88123fc34738
ffff88120f235850 ffffffff817ce750 ffff88120f8efc10
Mar 17 21:16:26 storage1 kernel: [119466.943351] Call Trace:
Mar 17 21:16:26 storage1 kernel: [119466.943354] [<ffffffff817ce750>] ?
bit_wait_timeout+0x80/0x80
Mar 17 21:16:26 storage1 kernel: [119466.943357] [<ffffffff817cdd29>]
schedule+0x29/0x70
Mar 17 21:16:26 storage1 kernel: [119466.943359] [<ffffffff817cddfe>]
io_schedule+0x8e/0xd0
Mar 17 21:16:26 storage1 kernel: [119466.943361] [<ffffffff817ce77c>]
bit_wait_io+0x2c/0x50
Mar 17 21:16:26 storage1 kernel: [119466.943363] [<ffffffff817ce4e5>]
__wait_on_bit+0x65/0x90
Mar 17 21:16:26 storage1 kernel: [119466.943366] [<ffffffff817ce750>] ?
bit_wait_timeout+0x80/0x80
Mar 17 21:16:26 storage1 kernel: [119466.943375] [<ffffffff817ce63c>]
out_of_line_wait_on_bit+0x7c/0x90
Mar 17 21:16:26 storage1 kernel: [119466.943379] [<ffffffff810b75b0>] ?
wake_atomic_t_function+0x40/0x40
Mar 17 21:16:26 storage1 kernel: [119466.943382] [<ffffffff81227686>]
__wait_on_buffer+0x36/0x40
Mar 17 21:16:26 storage1 kernel: [119466.943384] [<ffffffff812c3e05>]
jbd2_journal_commit_transaction+0x1285/0x1440
Mar 17 21:16:26 storage1 kernel: [119466.943390] [<ffffffff812c7923>]
kjournald2+0xb3/0x230
Mar 17 21:16:26 storage1 kernel: [119466.943394] [<ffffffff810b7530>] ?
prepare_to_wait_event+0x100/0x100
Mar 17 21:16:26 storage1 kernel: [119466.943397] [<ffffffff812c7870>] ?
commit_timeout+0x10/0x10
Mar 17 21:16:26 storage1 kernel: [119466.943399] [<ffffffff81095d59>]
kthread+0xc9/0xe0
Mar 17 21:16:26 storage1 kernel: [119466.943402] [<ffffffff81095c90>] ?
flush_kthread_worker+0x90/0x90
Mar 17 21:16:26 storage1 kernel: [119466.943404] [<ffffffff817d1e7c>]
ret_from_fork+0x7c/0xb0
Mar 17 21:16:26 storage1 kernel: [119466.943407] [<ffffffff81095c90>] ?
flush_kthread_worker+0x90/0x90
Nicholas A. Bellinger wrote:
> Hi Edgaras,
>
> Adding target-devel CC.
>
> On Thu, 2015-03-05 at 17:56 +0200, Edgaras Lukoševičius wrote:
>> Hello,
>>
>> we have a problem. iscsi connection is dropping constantly. After
>> enabling iscsi debug following message popped up after iscsi connection
>> got lost:
>>
>> # Kernel
>> 3.18.6-1.el7.elrepo.x86_64
>>
>> Message from syslogd@storage-1 at Mar 5 15:55:12 ...
>> kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s!
>> [iscsi_trx:11958]
>>
>> Message from syslogd@storage-1 at Mar 5 15:55:40 ...
>> kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s!
>> [iscsi_trx:11958]
>>
>> Message from syslogd@storage-1 at Mar 5 15:56:08 ...
>> kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s!
>> [iscsi_trx:11958]
>>
>> Message from syslogd@storage-1 at Mar 5 15:56:48 ...
>> kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s!
>> [iscsi_trx:11958]
>>
>>
>> P.S.
>>
>> Kernel 3.19 acts same I just don’t have debug messages for it. --
>
> Please send along the dmesg output of your system so we can have a look
> at the specific soft lockup stack back-traces.
>
> Thanks,
>
> --nab
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2015-03-18 5:40 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-03-05 15:56 iSCSI bug Edgaras Lukoševičius
2015-03-05 19:22 ` Nicholas A. Bellinger
2015-03-18 4:57 ` Govindarajan
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).