From: sagi@grimberg.me (Sagi Grimberg)
Subject: nvme/rdma initiator stuck on reboot
Date: Wed, 17 Aug 2016 13:23:05 +0300 [thread overview]
Message-ID: <2202d08c-2b4c-3bd9-6340-d630b8e2f8b5@grimberg.me> (raw)
In-Reply-To: <043901d1f7f5$fb5f73c0$f21e5b40$@opengridcomputing.com>
> Hey Sagi,
>
> Here is another issue I'm seeing doing reboot testing. The test does this:
>
> 1) connect 10 ram devices over iw_cxgb4
> 2) reboot the target node
> 3) the initiator goes into recovery/reconnect mode
> 4) reboot the inititator at this point.
>
> The initiator gets stuck doing this continually and the system never reboots:
>
> [ 596.411842] nvme nvme1: Failed reconnect attempt, requeueing...
> [ 596.907865] nvme nvme9: rdma_resolve_addr wait failed (-104).
> [ 596.914461] nvme nvme9: Failed reconnect attempt, requeueing...
> [ 597.939935] nvme nvme10: rdma_resolve_addr wait failed (-104).
> [ 597.946625] nvme nvme10: Failed reconnect attempt, requeueing...
> [ 598.963995] nvme nvme2: rdma_resolve_addr wait failed (-110).
> [ 598.971968] nvme nvme2: Failed reconnect attempt, requeueing...
> [ 602.036135] nvme nvme3: rdma_resolve_addr wait failed (-104).
> [ 602.043797] nvme nvme3: Failed reconnect attempt, requeueing...
> [ 603.060171] nvme nvme4: rdma_resolve_addr wait failed (-104).
> [ 603.068153] nvme nvme4: Failed reconnect attempt, requeueing...
> [ 604.084223] nvme nvme5: rdma_resolve_addr wait failed (-104).
> [ 604.092191] nvme nvme5: Failed reconnect attempt, requeueing...
> [ 605.108294] nvme nvme6: rdma_resolve_addr wait failed (-104).
> [ 605.116251] nvme nvme6: Failed reconnect attempt, requeueing...
>
> Debugging now...
Hmm...
Does this reproduce also when you simply delete all the
controllers (via sysfs)?
Do you see the hung task watchdog? can you share the
threads state? (echo t > /proc/sysrq-trigger)
next prev parent reply other threads:[~2016-08-17 10:23 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-08-16 19:40 nvme/rdma initiator stuck on reboot Steve Wise
2016-08-17 10:23 ` Sagi Grimberg [this message]
2016-08-17 14:33 ` Steve Wise
2016-08-17 14:46 ` Sagi Grimberg
2016-08-17 15:13 ` Steve Wise
2016-08-18 7:01 ` Sagi Grimberg
2016-08-18 13:59 ` Steve Wise
2016-08-18 14:47 ` Steve Wise
2016-08-18 15:21 ` 'Christoph Hellwig'
2016-08-18 17:59 ` Steve Wise
2016-08-18 18:50 ` Steve Wise
2016-08-18 19:11 ` Steve Wise
2016-08-19 8:58 ` Sagi Grimberg
2016-08-19 14:22 ` Steve Wise
[not found] ` <008001d1fa25$0c960fb0$25c22f10$@opengridcomputing.com>
2016-08-19 14:24 ` Steve Wise
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=2202d08c-2b4c-3bd9-6340-d630b8e2f8b5@grimberg.me \
--to=sagi@grimberg.me \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).