From: swise@opengridcomputing.com (Steve Wise)
Subject: nvme/rdma initiator stuck on reboot
Date: Wed, 17 Aug 2016 09:33:48 -0500 [thread overview]
Message-ID: <073301d1f894$5ddb81d0$19928570$@opengridcomputing.com> (raw)
In-Reply-To: <2202d08c-2b4c-3bd9-6340-d630b8e2f8b5@grimberg.me>
> > Hey Sagi,
> >
> > Here is another issue I'm seeing doing reboot testing. The test does
this:
> >
> > 1) connect 10 ram devices over iw_cxgb4
> > 2) reboot the target node
> > 3) the initiator goes into recovery/reconnect mode
> > 4) reboot the inititator at this point.
> >
> > The initiator gets stuck doing this continually and the system never
reboots:
> >
> > [ 596.411842] nvme nvme1: Failed reconnect attempt, requeueing...
> > [ 596.907865] nvme nvme9: rdma_resolve_addr wait failed (-104).
> > [ 596.914461] nvme nvme9: Failed reconnect attempt, requeueing...
> > [ 597.939935] nvme nvme10: rdma_resolve_addr wait failed (-104).
> > [ 597.946625] nvme nvme10: Failed reconnect attempt, requeueing...
> > [ 598.963995] nvme nvme2: rdma_resolve_addr wait failed (-110).
> > [ 598.971968] nvme nvme2: Failed reconnect attempt, requeueing...
> > [ 602.036135] nvme nvme3: rdma_resolve_addr wait failed (-104).
> > [ 602.043797] nvme nvme3: Failed reconnect attempt, requeueing...
> > [ 603.060171] nvme nvme4: rdma_resolve_addr wait failed (-104).
> > [ 603.068153] nvme nvme4: Failed reconnect attempt, requeueing...
> > [ 604.084223] nvme nvme5: rdma_resolve_addr wait failed (-104).
> > [ 604.092191] nvme nvme5: Failed reconnect attempt, requeueing...
> > [ 605.108294] nvme nvme6: rdma_resolve_addr wait failed (-104).
> > [ 605.116251] nvme nvme6: Failed reconnect attempt, requeueing...
> >
> > Debugging now...
>
> Hmm...
>
> Does this reproduce also when you simply delete all the
> controllers (via sysfs)?
>
Yes.
> Do you see the hung task watchdog? can you share the
> threads state? (echo t > /proc/sysrq-trigger)
The deleting thread is stuck:
Aug 17 07:21:32 stevo1 kernel: [ 2213.279798] INFO: task bash:4066 blocked
for more than 120 seconds.
Aug 17 07:21:32 stevo1 kernel: [ 2213.279802] Tainted: G E
4.8.0-rc2-nvmf-4.8-rc-rebased-rc2+ #15
Aug 17 07:21:32 stevo1 kernel: [ 2213.279803] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 17 07:21:32 stevo1 kernel: [ 2213.279811] bash D
ffff881034547ac8 0 4066 4056 0x10000080
Aug 17 07:21:32 stevo1 kernel: [ 2213.279819] ffff881034547ac8
ffff881033f04140 ffff88102bc80000 ffff881032b66250
Aug 17 07:21:32 stevo1 kernel: [ 2213.279824] ffff880fefe64230
ffff88103e88aa50 ffff881034544008 ffff881034547c98
Aug 17 07:21:32 stevo1 kernel: [ 2213.279828] ffff881034547c08
ffff88102bc80000 7fffffffffffffff ffff881034547b18
Aug 17 07:21:32 stevo1 kernel: [ 2213.279829] Call Trace:
Aug 17 07:21:32 stevo1 kernel: [ 2213.279842] [<ffffffff816ddde0>]
schedule+0x40/0xb0
Aug 17 07:21:32 stevo1 kernel: [ 2213.279850] [<ffffffff810b2de4>] ?
try_to_wake_up+0x54/0x240
Aug 17 07:21:32 stevo1 kernel: [ 2213.279856] [<ffffffff816e0a8d>]
schedule_timeout+0x2ad/0x410
Aug 17 07:21:32 stevo1 kernel: [ 2213.279861] [<ffffffff810b3025>] ?
wake_up_process+0x15/0x20
Aug 17 07:21:32 stevo1 kernel: [ 2213.279868] [<ffffffff8109d940>] ?
wake_up_worker+0x30/0x40
Aug 17 07:21:32 stevo1 kernel: [ 2213.279873] [<ffffffff8109fdbc>] ?
insert_work+0x6c/0xb0
Aug 17 07:21:32 stevo1 kernel: [ 2213.279878] [<ffffffff816deefe>]
wait_for_completion+0xde/0x110
Aug 17 07:21:32 stevo1 kernel: [ 2213.279882] [<ffffffff810b2fd0>] ?
try_to_wake_up+0x240/0x240
Aug 17 07:21:32 stevo1 kernel: [ 2213.279887] [<ffffffff810a0e0b>]
flush_work+0x2b/0x40
Aug 17 07:21:32 stevo1 kernel: [ 2213.279892] [<ffffffff8109dcc0>] ?
worker_detach_from_pool+0xa0/0xa0
Aug 17 07:21:32 stevo1 kernel: [ 2213.279902] [<ffffffffa064e46a>] ?
__nvme_rdma_del_ctrl+0x4a/0x60 [nvme_rdma]
Aug 17 07:21:32 stevo1 kernel: [ 2213.279908] [<ffffffffa064e4b8>]
nvme_rdma_del_ctrl+0x38/0x50 [nvme_rdma]
Aug 17 07:21:32 stevo1 kernel: [ 2213.279918] [<ffffffffa02703d4>]
nvme_sysfs_delete+0x34/0x50 [nvme_core]
Aug 17 07:21:32 stevo1 kernel: [ 2213.279923] [<ffffffff814884a0>]
dev_attr_store+0x20/0x30
Aug 17 07:21:32 stevo1 kernel: [ 2213.279931] [<ffffffff812ae29e>]
sysfs_kf_write+0x3e/0x40
Aug 17 07:21:32 stevo1 kernel: [ 2213.279935] [<ffffffff812add13>]
kernfs_fop_write+0x113/0x1b0
Aug 17 07:21:32 stevo1 kernel: [ 2213.279942] [<ffffffff8115a588>] ?
trace_buffer_unlock_commit_regs+0x68/0x80
Aug 17 07:21:32 stevo1 kernel: [ 2213.279949] [<ffffffff8122cfe4>]
__vfs_write+0x34/0x120
Aug 17 07:21:32 stevo1 kernel: [ 2213.279957] [<ffffffff81003725>] ?
trace_event_raw_event_sys_enter+0xb5/0x130
Aug 17 07:21:32 stevo1 kernel: [ 2213.279963] [<ffffffff810d1d06>] ?
percpu_down_read+0x16/0x60
Aug 17 07:21:32 stevo1 kernel: [ 2213.279967] [<ffffffff8122d191>]
vfs_write+0xc1/0x130
Aug 17 07:21:32 stevo1 kernel: [ 2213.279974] [<ffffffff81249633>] ?
__fdget+0x13/0x20
Aug 17 07:21:32 stevo1 kernel: [ 2213.279978] [<ffffffff8122d306>]
SyS_write+0x56/0xc0
Aug 17 07:21:32 stevo1 kernel: [ 2213.279983] [<ffffffff81003e7d>]
do_syscall_64+0x7d/0x230
Aug 17 07:21:32 stevo1 kernel: [ 2213.279991] [<ffffffff8106f057>] ?
do_page_fault+0x37/0x90
Aug 17 07:21:32 stevo1 kernel: [ 2213.279997] [<ffffffff816e1921>]
entry_SYSCALL64_slow_path+0x25/0x25
Nvme_rdma_wq is doing nothing:
Aug 17 07:20:08 stevo1 kernel: [ 2130.461614] nvme_rdma_wq S
ffff880fedd7bd58 0 4607 2 0x10000080
Aug 17 07:20:08 stevo1 kernel: [ 2130.469572] ffff880fedd7bd58
ffff8810381a15c0 ffff88102bdb5700 ffff881032b66250
Aug 17 07:20:08 stevo1 kernel: [ 2130.477972] ffffffff81038959
ffff880fedd7bd48 ffff880fedd78008 ffff8810206dde00
Aug 17 07:20:08 stevo1 kernel: [ 2130.486357] ffff88102bdb5700
ffffffff810a1730 ffff8810381abed8 ffff880fedd7bda8
Aug 17 07:20:08 stevo1 kernel: [ 2130.494755] Call Trace:
Aug 17 07:20:08 stevo1 kernel: [ 2130.498088] [<ffffffff81038959>] ?
sched_clock+0x9/0x10
Aug 17 07:20:08 stevo1 kernel: [ 2130.504287] [<ffffffff810a1730>] ?
process_one_work+0x4d0/0x4d0
Aug 17 07:20:08 stevo1 kernel: [ 2130.511168] [<ffffffff816ddde0>]
schedule+0x40/0xb0
Aug 17 07:20:08 stevo1 kernel: [ 2130.517002] [<ffffffff810b1064>] ?
set_user_nice+0x114/0x1b0
Aug 17 07:20:08 stevo1 kernel: [ 2130.523623] [<ffffffff810a1a22>]
rescuer_thread+0x2f2/0x380
Aug 17 07:20:08 stevo1 kernel: [ 2130.530146] [<ffffffff816dd920>] ?
__schedule+0x1f0/0x5b0
Aug 17 07:20:08 stevo1 kernel: [ 2130.536501] [<ffffffff810cb8a6>] ?
__wake_up_common+0x56/0x90
Aug 17 07:20:08 stevo1 kernel: [ 2130.543173] [<ffffffff810a1730>] ?
process_one_work+0x4d0/0x4d0
Aug 17 07:20:08 stevo1 kernel: [ 2130.550013] [<ffffffff816ddde0>] ?
schedule+0x40/0xb0
Aug 17 07:20:08 stevo1 kernel: [ 2130.555971] [<ffffffff810a1730>] ?
process_one_work+0x4d0/0x4d0
Aug 17 07:20:08 stevo1 kernel: [ 2130.562800] [<ffffffff810a6bbc>]
kthread+0xcc/0xf0
Aug 17 07:20:08 stevo1 kernel: [ 2130.568485] [<ffffffff810b157e>] ?
schedule_tail+0x1e/0xc0
Aug 17 07:20:08 stevo1 kernel: [ 2130.574871] [<ffffffff816e1a7f>]
ret_from_fork+0x1f/0x40
Aug 17 07:20:08 stevo1 kernel: [ 2130.581073] [<ffffffff810a6af0>] ?
kthread_freezable_should_stop+0x70/0x70
Here is another thread in the delete controller sysfs path:
Aug 17 07:20:05 stevo1 kernel: [ 2127.184581] bash D
ffff881034547ac8 0 4066 4056 0x10000080
Aug 17 07:20:05 stevo1 kernel: [ 2127.192445] ffff881034547ac8
ffff881033f04140 ffff88102bc80000 ffff881032b66250
Aug 17 07:20:05 stevo1 kernel: [ 2127.200730] ffff880fefe64230
ffff88103e88aa50 ffff881034544008 ffff881034547c98
Aug 17 07:20:05 stevo1 kernel: [ 2127.209022] ffff881034547c08
ffff88102bc80000 7fffffffffffffff ffff881034547b18
Aug 17 07:20:05 stevo1 kernel: [ 2127.217278] Call Trace:
Aug 17 07:20:05 stevo1 kernel: [ 2127.220523] [<ffffffff816ddde0>]
schedule+0x40/0xb0
Aug 17 07:20:05 stevo1 kernel: [ 2127.226244] [<ffffffff810b2de4>] ?
try_to_wake_up+0x54/0x240
Aug 17 07:20:05 stevo1 kernel: [ 2127.232753] [<ffffffff816e0a8d>]
schedule_timeout+0x2ad/0x410
Aug 17 07:20:05 stevo1 kernel: [ 2127.239353] [<ffffffff810b3025>] ?
wake_up_process+0x15/0x20
Aug 17 07:20:05 stevo1 kernel: [ 2127.245871] [<ffffffff8109d940>] ?
wake_up_worker+0x30/0x40
Aug 17 07:20:05 stevo1 kernel: [ 2127.252285] [<ffffffff8109fdbc>] ?
insert_work+0x6c/0xb0
Aug 17 07:20:05 stevo1 kernel: [ 2127.258432] [<ffffffff816deefe>]
wait_for_completion+0xde/0x110
Aug 17 07:20:05 stevo1 kernel: [ 2127.263351] nvme nvme4: rdma_resolve_addr
wait failed (-110).
Aug 17 07:20:05 stevo1 kernel: [ 2127.263355] nvme nvme4: Failed reconnect
attempt, requeueing...
Aug 17 07:20:05 stevo1 kernel: [ 2127.278353] [<ffffffff810b2fd0>] ?
try_to_wake_up+0x240/0x240
Aug 17 07:20:05 stevo1 kernel: [ 2127.284912] [<ffffffff810a0e0b>]
flush_work+0x2b/0x40
Aug 17 07:20:05 stevo1 kernel: [ 2127.290771] [<ffffffff8109dcc0>] ?
worker_detach_from_pool+0xa0/0xa0
Aug 17 07:20:05 stevo1 kernel: [ 2127.297970] [<ffffffffa064e46a>] ?
__nvme_rdma_del_ctrl+0x4a/0x60 [nvme_rdma]
Aug 17 07:20:05 stevo1 kernel: [ 2127.305945] [<ffffffffa064e4b8>]
nvme_rdma_del_ctrl+0x38/0x50 [nvme_rdma]
Aug 17 07:20:05 stevo1 kernel: [ 2127.313577] [<ffffffffa02703d4>]
nvme_sysfs_delete+0x34/0x50 [nvme_core]
Aug 17 07:20:05 stevo1 kernel: [ 2127.321111] [<ffffffff814884a0>]
dev_attr_store+0x20/0x30
Aug 17 07:20:05 stevo1 kernel: [ 2127.327343] [<ffffffff812ae29e>]
sysfs_kf_write+0x3e/0x40
Aug 17 07:20:05 stevo1 kernel: [ 2127.333551] [<ffffffff812add13>]
kernfs_fop_write+0x113/0x1b0
Aug 17 07:20:05 stevo1 kernel: [ 2127.340127] [<ffffffff8115a588>] ?
trace_buffer_unlock_commit_regs+0x68/0x80
Aug 17 07:20:05 stevo1 kernel: [ 2127.348017] [<ffffffff8122cfe4>]
__vfs_write+0x34/0x120
Aug 17 07:20:05 stevo1 kernel: [ 2127.354083] [<ffffffff81003725>] ?
trace_event_raw_event_sys_enter+0xb5/0x130
Aug 17 07:20:05 stevo1 kernel: [ 2127.362060] [<ffffffff810d1d06>] ?
percpu_down_read+0x16/0x60
Aug 17 07:20:05 stevo1 kernel: [ 2127.368667] [<ffffffff8122d191>]
vfs_write+0xc1/0x130
Aug 17 07:20:05 stevo1 kernel: [ 2127.374559] [<ffffffff81249633>] ?
__fdget+0x13/0x20
Aug 17 07:20:05 stevo1 kernel: [ 2127.380367] [<ffffffff8122d306>]
SyS_write+0x56/0xc0
Aug 17 07:20:05 stevo1 kernel: [ 2127.386201] [<ffffffff81003e7d>]
do_syscall_64+0x7d/0x230
Aug 17 07:20:05 stevo1 kernel: [ 2127.392485] [<ffffffff8106f057>] ?
do_page_fault+0x37/0x90
Aug 17 07:20:05 stevo1 kernel: [ 2127.398866] [<ffffffff816e1921>]
entry_SYSCALL64_slow_path+0x25/0x25
Here is the reconnect workq thread stuck trying to connect:
Aug 17 07:19:45 stevo1 kernel: [ 2107.446780] Workqueue: nvme_rdma_wq
nvme_rdma_reconnect_ctrl_work [nvme_rdma]
Aug 17 07:19:45 stevo1 kernel: [ 2107.454764] ffff881033a57aa8
ffff88103815c140 ffff8810371dab80 ffff881032b66250
Aug 17 07:19:45 stevo1 kernel: [ 2107.463091] 00000001001b93e9
0000000000000001 ffff881033a54008 ffff88103ee500c0
Aug 17 07:19:45 stevo1 kernel: [ 2107.471431] 00000001001b93e9
0000000000000000 ffff88103ee500c0 ffff881033a57af8
Aug 17 07:19:45 stevo1 kernel: [ 2107.479805] Call Trace:
Aug 17 07:19:45 stevo1 kernel: [ 2107.483076] [<ffffffff816ddde0>]
schedule+0x40/0xb0
Aug 17 07:19:45 stevo1 kernel: [ 2107.488892] [<ffffffff810f4a80>] ?
internal_add_timer+0x70/0x80
Aug 17 07:19:45 stevo1 kernel: [ 2107.495765] [<ffffffff816e09fb>]
schedule_timeout+0x21b/0x410
Aug 17 07:19:45 stevo1 kernel: [ 2107.502464] [<ffffffff810f4c10>] ?
del_timer+0x70/0x70
Aug 17 07:19:45 stevo1 kernel: [ 2107.508547] [<ffffffffa031da9f>] ?
rdma_resolve_ip+0x1cf/0x1e0 [ib_core]
Aug 17 07:19:45 stevo1 kernel: [ 2107.516183] [<ffffffffa0736fe0>] ?
cma_work_handler+0xa0/0xa0 [rdma_cm]
Aug 17 07:19:45 stevo1 kernel: [ 2107.523721] [<ffffffff816deb6f>]
wait_for_completion_interruptible_timeout+0xef/0x130
Aug 17 07:19:45 stevo1 kernel: [ 2107.532472] [<ffffffff810b2fd0>] ?
try_to_wake_up+0x240/0x240
Aug 17 07:19:45 stevo1 kernel: [ 2107.539152] [<ffffffffa064f5c2>]
nvme_rdma_init_queue+0xe2/0x180 [nvme_rdma]
Aug 17 07:19:45 stevo1 kernel: [ 2107.547132] [<ffffffffa064f9bc>]
nvme_rdma_reconnect_ctrl_work+0x9c/0x1f0 [nvme_rdma]
Aug 17 07:19:45 stevo1 kernel: [ 2107.555908] [<ffffffff810a13e3>]
process_one_work+0x183/0x4d0
Aug 17 07:19:45 stevo1 kernel: [ 2107.562592] [<ffffffff816dd920>] ?
__schedule+0x1f0/0x5b0
Aug 17 07:19:45 stevo1 kernel: [ 2107.568938] [<ffffffff816ddde0>] ?
schedule+0x40/0xb0
Aug 17 07:19:45 stevo1 kernel: [ 2107.574927] [<ffffffff810a207d>]
worker_thread+0x16d/0x530
Aug 17 07:19:45 stevo1 kernel: [ 2107.581346] [<ffffffff810a1f10>] ?
maybe_create_worker+0x120/0x120
Aug 17 07:19:45 stevo1 kernel: [ 2107.588474] [<ffffffff816dd920>] ?
__schedule+0x1f0/0x5b0
Aug 17 07:19:45 stevo1 kernel: [ 2107.594813] [<ffffffff810cb8a6>] ?
__wake_up_common+0x56/0x90
Aug 17 07:19:45 stevo1 kernel: [ 2107.601488] [<ffffffff810a1f10>] ?
maybe_create_worker+0x120/0x120
Aug 17 07:19:45 stevo1 kernel: [ 2107.608625] [<ffffffff816ddde0>] ?
schedule+0x40/0xb0
Aug 17 07:19:45 stevo1 kernel: [ 2107.614626] [<ffffffff810a1f10>] ?
maybe_create_worker+0x120/0x120
Aug 17 07:19:45 stevo1 kernel: [ 2107.621745] [<ffffffff810a6bbc>]
kthread+0xcc/0xf0
Aug 17 07:19:45 stevo1 kernel: [ 2107.627464] [<ffffffff810b157e>] ?
schedule_tail+0x1e/0xc0
Aug 17 07:19:45 stevo1 kernel: [ 2107.633886] [<ffffffff816e1a7f>]
ret_from_fork+0x1f/0x40
Aug 17 07:19:45 stevo1 kernel: [ 2107.640121] [<ffffffff810a6af0>] ?
kthread_freezable_should_stop+0x70/0x70
Here is controller delete workq thread:
Aug 17 07:19:42 stevo1 kernel: [ 2104.515910] Workqueue: nvme_rdma_wq
nvme_rdma_del_ctrl_work [nvme_rdma]
Aug 17 07:19:42 stevo1 kernel: [ 2104.523373] ffff881033f13888
ffff88103817ab80 ffff881033f04140 ffff881032b66250
Aug 17 07:19:42 stevo1 kernel: [ 2104.531744] 00000001001c1a60
0000000000000005 ffff881033f10008 ffff88103ef500c0
Aug 17 07:19:42 stevo1 kernel: [ 2104.540116] 00000001001c1a60
0000000000000000 ffff88103ef500c0 ffff881033f138d8
Aug 17 07:19:42 stevo1 kernel: [ 2104.548501] Call Trace:
Aug 17 07:19:42 stevo1 kernel: [ 2104.551777] [<ffffffff816ddde0>]
schedule+0x40/0xb0
Aug 17 07:19:42 stevo1 kernel: [ 2104.557593] [<ffffffff810f4a80>] ?
internal_add_timer+0x70/0x80
Aug 17 07:19:42 stevo1 kernel: [ 2104.564445] [<ffffffff816e09fb>]
schedule_timeout+0x21b/0x410
Aug 17 07:19:42 stevo1 kernel: [ 2104.571117] [<ffffffff810f4c10>] ?
del_timer+0x70/0x70
Aug 17 07:19:42 stevo1 kernel: [ 2104.577191] [<ffffffff810fe67c>] ?
ktime_get+0x4c/0xc0
Aug 17 07:19:42 stevo1 kernel: [ 2104.583248] [<ffffffff816dd6c4>]
io_schedule_timeout+0xa4/0x110
Aug 17 07:19:42 stevo1 kernel: [ 2104.590083] [<ffffffff816df013>]
wait_for_completion_io_timeout+0xe3/0x120
Aug 17 07:19:42 stevo1 kernel: [ 2104.597943] [<ffffffff810b2fd0>] ?
try_to_wake_up+0x240/0x240
Aug 17 07:19:42 stevo1 kernel: [ 2104.604613] [<ffffffff81332a4d>] ?
blk_execute_rq_nowait+0x7d/0x160
Aug 17 07:19:42 stevo1 kernel: [ 2104.611810] [<ffffffff81332bbc>]
blk_execute_rq+0x8c/0x110
Aug 17 07:19:42 stevo1 kernel: [ 2104.618221] [<ffffffffa02715c5>]
__nvme_submit_sync_cmd+0x85/0xe0 [nvme_core]
Aug 17 07:19:42 stevo1 kernel: [ 2104.626293] [<ffffffffa0672526>]
nvmf_reg_write32+0x66/0xb0 [nvme_fabrics]
Aug 17 07:19:42 stevo1 kernel: [ 2104.634086] [<ffffffffa027095e>]
nvme_shutdown_ctrl+0x4e/0xe0 [nvme_core]
Aug 17 07:19:42 stevo1 kernel: [ 2104.641796] [<ffffffffa064f161>]
nvme_rdma_shutdown_ctrl+0xa1/0xe0 [nvme_rdma]
Aug 17 07:19:42 stevo1 kernel: [ 2104.649940] [<ffffffffa064f21c>]
__nvme_rdma_remove_ctrl+0x7c/0x80 [nvme_rdma]
Aug 17 07:19:42 stevo1 kernel: [ 2104.658093] [<ffffffffa064f25a>]
nvme_rdma_del_ctrl_work+0x1a/0x20 [nvme_rdma]
Aug 17 07:19:42 stevo1 kernel: [ 2104.666248] [<ffffffff810a13e3>]
process_one_work+0x183/0x4d0
Aug 17 07:19:42 stevo1 kernel: [ 2104.672936] [<ffffffff816dd920>] ?
__schedule+0x1f0/0x5b0
Aug 17 07:19:42 stevo1 kernel: [ 2104.679269] [<ffffffff8109daf2>] ?
move_linked_works+0x62/0x90
Aug 17 07:19:42 stevo1 kernel: [ 2104.686032] [<ffffffff810a2204>]
worker_thread+0x2f4/0x530
Aug 17 07:19:42 stevo1 kernel: [ 2104.692441] [<ffffffff810a1f10>] ?
maybe_create_worker+0x120/0x120
Aug 17 07:19:42 stevo1 kernel: [ 2104.699540] [<ffffffff816dd920>] ?
__schedule+0x1f0/0x5b0
Aug 17 07:19:42 stevo1 kernel: [ 2104.705867] [<ffffffff810cb8a6>] ?
__wake_up_common+0x56/0x90
Aug 17 07:19:42 stevo1 kernel: [ 2104.712534] [<ffffffff810a1f10>] ?
maybe_create_worker+0x120/0x120
Aug 17 07:19:42 stevo1 kernel: [ 2104.719257] nvme nvme4: rdma_resolve_addr
wait failed (-104).
Aug 17 07:19:42 stevo1 kernel: [ 2104.719261] nvme nvme4: Failed reconnect
attempt, requeueing...
Aug 17 07:19:42 stevo1 kernel: [ 2104.733085] [<ffffffff816ddde0>] ?
schedule+0x40/0xb0
Aug 17 07:19:42 stevo1 kernel: [ 2104.739063] [<ffffffff810a1f10>] ?
maybe_create_worker+0x120/0x120
Aug 17 07:19:42 stevo1 kernel: [ 2104.746163] [<ffffffff810a6bbc>]
kthread+0xcc/0xf0
Aug 17 07:19:42 stevo1 kernel: [ 2104.751878] [<ffffffff810b157e>] ?
schedule_tail+0x1e/0xc0
next prev parent reply other threads:[~2016-08-17 14:33 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-08-16 19:40 nvme/rdma initiator stuck on reboot Steve Wise
2016-08-17 10:23 ` Sagi Grimberg
2016-08-17 14:33 ` Steve Wise [this message]
2016-08-17 14:46 ` Sagi Grimberg
2016-08-17 15:13 ` Steve Wise
2016-08-18 7:01 ` Sagi Grimberg
2016-08-18 13:59 ` Steve Wise
2016-08-18 14:47 ` Steve Wise
2016-08-18 15:21 ` 'Christoph Hellwig'
2016-08-18 17:59 ` Steve Wise
2016-08-18 18:50 ` Steve Wise
2016-08-18 19:11 ` Steve Wise
2016-08-19 8:58 ` Sagi Grimberg
2016-08-19 14:22 ` Steve Wise
[not found] ` <008001d1fa25$0c960fb0$25c22f10$@opengridcomputing.com>
2016-08-19 14:24 ` Steve Wise
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='073301d1f894$5ddb81d0$19928570$@opengridcomputing.com' \
--to=swise@opengridcomputing.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.