All of lore.kernel.org
 help / color / mirror / Atom feed
From: Hannes Reinecke <hare@kernel.org>
To: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>, Keith Busch <kbusch@kernel.org>,
	linux-nvme@lists.infradead.org, Hannes Reinecke <hare@suse.de>,
	Hannes Reinecke <hare@kernel.org>
Subject: [PATCH 2/8] nvme-tcp: io_work stall debugging
Date: Tue, 16 Jul 2024 09:36:10 +0200	[thread overview]
Message-ID: <20240716073616.84417-3-hare@kernel.org> (raw)
In-Reply-To: <20240716073616.84417-1-hare@kernel.org>

From: Hannes Reinecke <hare@suse.de>

Add a debug message when the io workqueue exceeds the latency target
given by 'deadline'.

Signed-off-by: Hannes Reinecke <hare@kernel.org>
---
 drivers/nvme/host/tcp.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)

diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 3cf9a9abb0e0..7876bf7d2fac 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -1282,13 +1282,14 @@ static void nvme_tcp_io_work(struct work_struct *w)
 {
 	struct nvme_tcp_queue *queue =
 		container_of(w, struct nvme_tcp_queue, io_work);
-	u64 start = ktime_to_us(ktime_get());
+	u64 start = ktime_to_us(ktime_get()), overrun;
 	u64 tx_deadline = start + deadline;
+	bool pending = false;
 
 	do {
-		bool pending = false;
 		int result;
 
+		pending = false;
 		if (mutex_trylock(&queue->send_mutex)) {
 			result = nvme_tcp_try_send(queue);
 			mutex_unlock(&queue->send_mutex);
@@ -1304,12 +1305,23 @@ static void nvme_tcp_io_work(struct work_struct *w)
 		else if (unlikely(result < 0))
 			return;
 
-		if (!pending || !queue->rd_enabled)
+		if (!queue->rd_enabled)
 			return;
+		if (!pending)
+			goto check;
 
 	} while (ktime_to_us(ktime_get()) < tx_deadline); /* quota is exhausted */
 
-	queue_work_on(queue->io_cpu, nvme_tcp_wq, &queue->io_work);
+check:
+	overrun = ktime_to_us(ktime_get()) - start;
+	if (overrun > 10 * deadline) {
+		dev_dbg(queue->ctrl->ctrl.device,
+			"queue %d: stall (%llu msecs)%s%s\n",
+			nvme_tcp_queue_id(queue), div_u64(overrun, 1000),
+			list_empty(&queue->send_list) ? " empty" : "", queue->request ? " pending" : "");
+	}
+	if (pending)
+		queue_work_on(queue->io_cpu, nvme_tcp_wq, &queue->io_work);
 }
 
 static void nvme_tcp_free_crypto(struct nvme_tcp_queue *queue)
-- 
2.35.3



  parent reply	other threads:[~2024-07-16  7:36 UTC|newest]

Thread overview: 24+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2024-07-16  7:36 [PATCHv3 0/8] nvme-tcp: improve scalability Hannes Reinecke
2024-07-16  7:36 ` [PATCH 1/8] nvme-tcp: switch TX deadline to microseconds and make it configurable Hannes Reinecke
2024-07-17 21:03   ` Sagi Grimberg
2024-07-18  6:30     ` Hannes Reinecke
2024-07-16  7:36 ` Hannes Reinecke [this message]
2024-07-17 21:05   ` [PATCH 2/8] nvme-tcp: io_work stall debugging Sagi Grimberg
2024-07-16  7:36 ` [PATCH 3/8] nvme-tcp: re-init request list entries Hannes Reinecke
2024-07-17 21:23   ` Sagi Grimberg
2024-07-16  7:36 ` [PATCH 4/8] nvme-tcp: improve stall debugging Hannes Reinecke
2024-07-17 21:11   ` Sagi Grimberg
2024-07-16  7:36 ` [PATCH 5/8] nvme-tcp: debugfs entries for latency statistics Hannes Reinecke
2024-07-17 21:14   ` Sagi Grimberg
2024-07-16  7:36 ` [PATCH 6/8] nvme-tcp: reduce callback lock contention Hannes Reinecke
2024-07-17 21:19   ` Sagi Grimberg
2024-07-18  6:42     ` Hannes Reinecke
2024-07-21 11:46       ` Sagi Grimberg
2024-07-16  7:36 ` [PATCH 7/8] nvme-tcp: check for SOCK_NOSPACE before sending Hannes Reinecke
2024-07-17 21:19   ` Sagi Grimberg
2024-07-16  7:36 ` [PATCH 8/8] nvme-tcp: align I/O cpu with blk-mq mapping Hannes Reinecke
2024-07-17 21:34   ` Sagi Grimberg
2024-08-13 19:36     ` Sagi Grimberg
2024-07-17 21:01 ` [PATCHv3 0/8] nvme-tcp: improve scalability Sagi Grimberg
2024-07-18  6:20   ` Hannes Reinecke
2024-07-21 12:05     ` Sagi Grimberg

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20240716073616.84417-3-hare@kernel.org \
    --to=hare@kernel.org \
    --cc=hare@suse.de \
    --cc=hch@lst.de \
    --cc=kbusch@kernel.org \
    --cc=linux-nvme@lists.infradead.org \
    --cc=sagi@grimberg.me \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.