From: jthumshirn@suse.de (Johannes Thumshirn)
Subject: I/O Errors due to keepalive timeouts with NVMf RDMA
Date: Mon, 10 Jul 2017 13:50:03 +0200 [thread overview]
Message-ID: <20170710115003.GH5105@linux-x5ow.site> (raw)
In-Reply-To: <ebc9dccc-ce68-f5f3-3e24-372b2cd7f44b@grimberg.me>
On Mon, Jul 10, 2017@02:41:28PM +0300, Sagi Grimberg wrote:
> >Host:
> >[353698.784927] nvme nvme0: creating 44 I/O queues.
> >[353699.572467] nvme nvme0: new ctrl: NQN
> >"nqn.2014-08.org.nvmexpress:NVMf:uuid:c36f2c23-354d-416c-95de-f2b8ec353a82",
> >addr 1.1.1.2:4420
> >[353960.804750] nvme nvme0: SEND for CQE 0xffff88011c0cca58 failed with status
> >transport retry counter exceeded (12)
>
> Exhausted retries, wow... That is really strange...
>
> Host sent the keep-alive and it never made it to the host, the HCA
> retried for 7+ times and gave up.
>
> Are you running with a switch? which one? is the switch experience
> higher ingress?
This (unfortunately) was the OmniPath setup as I only was a guest on the IB
installation and the other team needed it back. Anyways I did see this on IB
as well (regardless of SLE12-SP3 and v4.12 final). The switch is an Intel Edge
100 OmniPath switch.
[...]
>
> And why aren't you able to reconnect?
>
> Something smells mis-configured here...
I am it just takes ages:
[354235.064586] nvme nvme0: Failed reconnect attempt 27
[354235.076054] nvme nvme0: Reconnecting in 10 seconds...
[354245.117100] nvme nvme0: rdma_resolve_addr wait failed (-104).
[354245.144574] nvme nvme0: Failed reconnect attempt 28
[354245.156097] nvme nvme0: Reconnecting in 10 seconds...
[354255.244008] nvme nvme0: creating 44 I/O queues.
[354255.877529] nvme nvme0: Successfully reconnected
[354255.900579] nvme0n1: detected capacity change from -67526893324191744 to
68719476736
--
Johannes Thumshirn Storage
jthumshirn at suse.de +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850
next prev parent reply other threads:[~2017-07-10 11:50 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-07-07 9:48 I/O Errors due to keepalive timeouts with NVMf RDMA Johannes Thumshirn
2017-07-08 18:14 ` Max Gurtovoy
2017-07-10 7:59 ` Johannes Thumshirn
2017-07-10 7:06 ` Sagi Grimberg
2017-07-10 7:17 ` Hannes Reinecke
2017-07-10 8:46 ` Max Gurtovoy
2017-07-10 9:10 ` Johannes Thumshirn
2017-07-10 10:13 ` Sagi Grimberg
2017-07-10 10:20 ` Johannes Thumshirn
2017-07-10 11:04 ` Sagi Grimberg
2017-07-10 11:33 ` Johannes Thumshirn
2017-07-10 11:41 ` Sagi Grimberg
2017-07-10 11:50 ` Johannes Thumshirn [this message]
2017-07-10 12:04 ` Sagi Grimberg
2017-07-11 8:52 ` Johannes Thumshirn
2017-07-11 9:19 ` Sagi Grimberg
2017-07-11 9:21 ` Johannes Thumshirn
2017-07-14 11:25 ` Johannes Thumshirn
2017-08-15 22:46 ` Guilherme G. Piccoli
2017-08-16 8:16 ` Christoph Hellwig
2017-08-16 16:19 ` Guilherme G. Piccoli
2017-08-28 10:15 ` Guan Junxiong
2017-07-10 8:59 ` Jack Wang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20170710115003.GH5105@linux-x5ow.site \
--to=jthumshirn@suse.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.