public inbox for linux-nvme@lists.infradead.org
 help / color / mirror / Atom feed
From: Hannes Reinecke <hare@suse.de>
To: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>, Keith Busch <kbusch@kernel.org>,
	linux-nvme@lists.infradead.org, Hannes Reinecke <hare@suse.de>
Subject: [PATCH 1/3] nvme-tcp: spurious I/O timeout under high load
Date: Thu, 19 May 2022 08:26:15 +0200	[thread overview]
Message-ID: <20220519062617.39715-2-hare@suse.de> (raw)
In-Reply-To: <20220519062617.39715-1-hare@suse.de>

When running on slow links requests might take some time
for be processed, and as we always allow to queue requests
timeout may trigger when the requests are still queued.
Eg sending 128M requests over 30 queues over a 1GigE link
will inevitably timeout before the last request could be sent.
So reset the timeout if the request is still being queued
or if it's in the process of being sent.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/nvme/host/tcp.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index bb67538d241b..ede76a0719a0 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -2332,6 +2332,13 @@ nvme_tcp_timeout(struct request *rq, bool reserved)
 		"queue %d: timeout request %#x type %d\n",
 		nvme_tcp_queue_id(req->queue), rq->tag, pdu->hdr.type);
 
+	if (!list_empty(&req->entry) || req->queue->request == req) {
+		dev_warn(ctrl->device,
+			 "queue %d: queue stall, resetting timeout\n",
+			 nvme_tcp_queue_id(req->queue));
+		return BLK_EH_RESET_TIMER;
+	}
+
 	if (ctrl->state != NVME_CTRL_LIVE) {
 		/*
 		 * If we are resetting, connecting or deleting we should
-- 
2.29.2



  reply	other threads:[~2022-05-19  6:26 UTC|newest]

Thread overview: 24+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-05-19  6:26 [PATCH 0/3] nvme-tcp: queue stalls under high load Hannes Reinecke
2022-05-19  6:26 ` Hannes Reinecke [this message]
2022-05-20  9:05   ` [PATCH 1/3] nvme-tcp: spurious I/O timeout " Sagi Grimberg
2022-05-23  8:42     ` Hannes Reinecke
2022-05-23 13:36       ` Sagi Grimberg
2022-05-23 14:01         ` Hannes Reinecke
2022-05-23 15:05           ` Sagi Grimberg
2022-05-23 16:07             ` Hannes Reinecke
2022-05-24  7:57               ` Sagi Grimberg
2022-05-24  8:08                 ` Hannes Reinecke
2022-05-24  8:53                   ` Sagi Grimberg
2022-05-24  9:34                     ` Hannes Reinecke
2022-05-24  9:58                       ` Sagi Grimberg
2022-05-19  6:26 ` [PATCH 2/3] nvme-tcp: Check for write space before queueing requests Hannes Reinecke
2022-05-20  9:17   ` Sagi Grimberg
2022-05-20 10:05     ` Hannes Reinecke
2022-05-21 20:01       ` Sagi Grimberg
2022-05-19  6:26 ` [PATCH 3/3] nvme-tcp: send quota for nvme_tcp_send_all() Hannes Reinecke
2022-05-20  9:19   ` Sagi Grimberg
2022-05-20  9:59     ` Hannes Reinecke
2022-05-21 20:02       ` Sagi Grimberg
2022-05-20  9:20 ` [PATCH 0/3] nvme-tcp: queue stalls under high load Sagi Grimberg
2022-05-20 10:01   ` Hannes Reinecke
2022-05-21 20:03     ` Sagi Grimberg

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20220519062617.39715-2-hare@suse.de \
    --to=hare@suse.de \
    --cc=hch@lst.de \
    --cc=kbusch@kernel.org \
    --cc=linux-nvme@lists.infradead.org \
    --cc=sagi@grimberg.me \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox