Linux-NVME Archive on lore.kernel.org
 help / color / mirror / Atom feed
From: Yi Zhang <yi.zhang@redhat.com>
To: Chao Leng <lengchao@huawei.com>,
	sagi@grimberg.me, Ming Lei <ming.lei@redhat.com>
Cc: kbusch@kernel.org, axboe@fb.com, hch@lst.de,
	linux-nvme@lists.infradead.org
Subject: Re: [PATCH v2 0/5] avoid race for time out
Date: Wed, 28 Oct 2020 19:36:06 +0800	[thread overview]
Message-ID: <aba13cb9-a3d4-490b-4d1d-1deb60eafff6@redhat.com> (raw)
In-Reply-To: <20201022021452.10971-1-lengchao@huawei.com>

Hello

This series fixed the WARNING issue I reported [1], but now the nvme/012 
[2] will be hang there and never finished, here is the log[3]
[1]
https://lore.kernel.org/linux-nvme/1934331639.3314730.1602152202454.JavaMail.zimbra@redhat.com/

[2]
[root@hpe-xw9400-02 blktests]# nvme_trtype=tcp ./check nvme/012
nvme/012 (run mkfs and data verification fio job on NVMeOF block 
device-backed ns)
     runtime  1199.651s  ...

[3]
[  120.550409] run blktests nvme/012 at 2020-10-28 06:50:11
[  121.138234] loop: module loaded
[  121.170869] nvmet: adding nsid 1 to subsystem blktests-subsystem-1
[  121.215930] nvmet_tcp: enabling port 0 (127.0.0.1:4420)
[  121.288229] nvmet: creating controller 1 for subsystem 
blktests-subsystem-1 for NQN 
nqn.2014-08.org.nvmexpress:uuid:ffe2b140e76a45649005853f3b871859.
[  121.302597] nvme nvme0: creating 12 I/O queues.
[  121.308361] nvme nvme0: mapped 12/0/0 default/read/poll queues.
[  121.320030] nvme nvme0: new ctrl: NQN "blktests-subsystem-1", addr 
127.0.0.1:4420
[  123.278903] XFS (nvme0n1): Mounting V5 Filesystem
[  123.291608] XFS (nvme0n1): Ending clean mount
[  123.297321] xfs filesystem being mounted at /mnt/blktests supports 
timestamps until 2038 (0x7fffffff)
[  183.872118] nvme nvme0: queue 1: timeout request 0x6c type 4
[  183.877792] nvme nvme0: starting error recovery
[  183.882376] nvme nvme0: queue 8: timeout request 0x11 type 4
[  183.888149] nvme nvme0: queue 8: timeout request 0x12 type 4
[  183.893805] nvme nvme0: queue 8: timeout request 0x13 type 4
[  183.899469] nvme nvme0: queue 8: timeout request 0x14 type 4
[  183.905130] nvme nvme0: queue 8: timeout request 0x15 type 4
[  183.910792] nvme nvme0: queue 8: timeout request 0x16 type 4
[  183.916453] nvme nvme0: queue 8: timeout request 0x17 type 4
[  183.922114] nvme nvme0: queue 8: timeout request 0x18 type 4
[  183.927777] nvme nvme0: queue 8: timeout request 0x19 type 4
[  183.933450] nvme nvme0: queue 8: timeout request 0x1a type 4
[  183.939110] nvme nvme0: queue 8: timeout request 0x1b type 4
[  183.944771] nvme nvme0: queue 8: timeout request 0x1c type 4
[  183.950431] nvme nvme0: queue 8: timeout request 0x1d type 4
[  183.956095] nvme nvme0: queue 8: timeout request 0x1e type 4
[  183.961755] nvme nvme0: queue 8: timeout request 0x1f type 4
[  183.967414] nvme nvme0: queue 8: timeout request 0x20 type 4
[  183.973218] block nvme0n1: no usable path - requeuing I/O
[  183.978623] block nvme0n1: no usable path - requeuing I/O
[  183.982492] nvme nvme0: Reconnecting in 10 seconds...
[  183.984022] block nvme0n1: no usable path - requeuing I/O
[  183.994476] block nvme0n1: no usable path - requeuing I/O
[  183.999870] block nvme0n1: no usable path - requeuing I/O
[  184.005264] block nvme0n1: no usable path - requeuing I/O
[  184.010669] block nvme0n1: no usable path - requeuing I/O
[  184.016080] block nvme0n1: no usable path - requeuing I/O
[  184.021463] block nvme0n1: no usable path - requeuing I/O
[  184.026858] block nvme0n1: no usable path - requeuing I/O
[  209.472647] nvmet: ctrl 2 keep-alive timer (15 seconds) expired!
[  209.478662] nvmet: ctrl 2 fatal error occurred!
[  213.568765] nvmet: ctrl 1 keep-alive timer (15 seconds) expired!
[  213.574782] nvmet: ctrl 1 fatal error occurred!
[  238.064572] nvmet: creating controller 2 for subsystem 
blktests-subsystem-1 for NQN 
nqn.2014-08.org.nvmexpress:uuid:ffe2b140e76a45649005853f3b871859.
[  256.577658] nvme nvme0: queue 0: timeout request 0x0 type 4
[  256.583333] nvme nvme0: Connect command failed, error wo/DNR bit: 881
[  256.589806] nvme nvme0: failed to connect queue: 0 ret=881
[  256.595326] nvme nvme0: Failed reconnect attempt 1
[  256.600119] nvme nvme0: Reconnecting in 10 seconds...
[  266.818455] nvmet: creating controller 1 for subsystem 
blktests-subsystem-1 for NQN 
nqn.2014-08.org.nvmexpress:uuid:ffe2b140e76a45649005853f3b871859.
[  266.832356] nvme_ns_head_submit_bio: 30 callbacks suppressed
[  266.832362] block nvme0n1: no usable path - requeuing I/O
[  266.843443] block nvme0n1: no usable path - requeuing I/O
[  266.848848] block nvme0n1: no usable path - requeuing I/O
[  266.854244] block nvme0n1: no usable path - requeuing I/O
[  266.859663] block nvme0n1: no usable path - requeuing I/O
[  266.865059] block nvme0n1: no usable path - requeuing I/O
[  266.870454] block nvme0n1: no usable path - requeuing I/O
[  266.875845] block nvme0n1: no usable path - requeuing I/O
[  266.881234] block nvme0n1: no usable path - requeuing I/O
[  266.886632] block nvme0n1: no usable path - requeuing I/O
[  266.892237] nvme nvme0: creating 12 I/O queues.
[  266.903744] nvme nvme0: mapped 12/0/0 default/read/poll queues.
[  266.911929] nvme nvme0: Successfully reconnected (2 attempt)
[  327.747177] nvme nvme0: queue 2: timeout request 0x1e type 4
[  327.752883] nvme nvme0: starting error recovery
[  327.757450] nvme nvme0: queue 4: timeout request 0x63 type 4
[  327.763511] nvme_ns_head_submit_bio: 14 callbacks suppressed
[  327.763520] block nvme0n1: no usable path - requeuing I/O
[  327.774614] block nvme0n1: no usable path - requeuing I/O
[  327.780053] block nvme0n1: no usable path - requeuing I/O
[  327.785450] block nvme0n1: no usable path - requeuing I/O
[  327.790876] block nvme0n1: no usable path - requeuing I/O
[  327.796316] block nvme0n1: no usable path - requeuing I/O
[  327.801727] block nvme0n1: no usable path - requeuing I/O
[  327.807231] block nvme0n1: no usable path - requeuing I/O
[  327.812627] block nvme0n1: no usable path - requeuing I/O
[  327.818025] block nvme0n1: no usable path - requeuing I/O
[  353.859745] nvmet: ctrl 1 keep-alive timer (15 seconds) expired!
[  353.865761] nvmet: ctrl 1 fatal error occurred!


On 10/22/20 10:14 AM, Chao Leng wrote:
> First avoid race between time out and tear down for rdma and tcp.
> Second avoid repeated request completion in time out for rdma and tcp.
>
> V2:
> 	- add avoiding repeated request completion in time out
>
> Chao Leng (3):
>    nvme-core: introduce sync io queues
>    nvme-rdma: avoid race between time out and tear down
>    nvme-tcp: avoid race between time out and tear down
>
> Sagi Grimberg (2):
>    nvme-rdma: avoid repeated request completion
>    nvme-tcp: avoid repeated request completion
>
>   drivers/nvme/host/core.c |  8 ++++++--
>   drivers/nvme/host/nvme.h |  1 +
>   drivers/nvme/host/rdma.c | 14 +++-----------
>   drivers/nvme/host/tcp.c  | 16 ++++------------
>   4 files changed, 14 insertions(+), 25 deletions(-)
>


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

  reply	other threads:[~2020-10-28 11:36 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-10-22  2:14 [PATCH v2 0/5] avoid race for time out Chao Leng
2020-10-28 11:36 ` Yi Zhang [this message]
2020-10-28 13:25   ` Ming Lei
2020-10-29  6:13   ` Chao Leng
2020-10-29 21:00   ` Sagi Grimberg
2020-10-29 21:01     ` Sagi Grimberg
2020-10-30  0:04     ` Yi Zhang
2020-10-30  1:00       ` Ming Lei

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=aba13cb9-a3d4-490b-4d1d-1deb60eafff6@redhat.com \
    --to=yi.zhang@redhat.com \
    --cc=axboe@fb.com \
    --cc=hch@lst.de \
    --cc=kbusch@kernel.org \
    --cc=lengchao@huawei.com \
    --cc=linux-nvme@lists.infradead.org \
    --cc=ming.lei@redhat.com \
    --cc=sagi@grimberg.me \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox