* [RESEND RFC PATCH v2] nvme-pci: add adaptive interrupt polling
@ 2026-08-06 3:10 Fengnan Chang
2026-08-10 20:55 ` Keith Busch
0 siblings, 1 reply; 3+ messages in thread
From: Fengnan Chang @ 2026-08-06 3:10 UTC (permalink / raw)
To: kbusch, axboe, hch, sagi, linux-nvme
Cc: linux-kernel, Fengnan Chang, Guzebing
The idea behind this approach is: Let each I/O queue switch itself between
interrupt and poll mode based on its own recent completion rate.
This version is still in the testing phase, and there are still some issues
with the code implementation. I releasing it now to see if the approach
is generally acceptable. If the approach looks good, I’ll continue to
refine it and conduct more extensive testing. The main implementation
logic is in `nvme_adaptive_sample` and `nvme_adaptive_irq_poll`; you should
focus on reviewing the implementation of these two functions.
Compared to the previous version, this represents a much smaller
performance regression while offering greater benefits.
In high-IOPS scenarios, relying on interrupts to handle I/O operations
can limit performance. This issue becomes particularly pronounced in
multi-disk environments, where performance is constrained by the CPU's
interrupt-handling capacity.
Each Solidigm SB5PH27X038T device used for testing can deliver about 3.2M
4 KiB random-read IOPS. Four of them should be good for about 12.8M IOPS,
but interrupt-driven completion tops out at 5.59M, only about 44% of that.
Polling gets rid of that cost, but polling every queue all the time burns
CPU and hurts the sparse or bursty queues that interrupts handle just fine.
So instead of a global switch, let each queue make the call on its own,
from how fast it has been completing lately, and re-check often enough that
the decision tracks the workload rather than a fixed tunable.
Each queue runs a small loop with three stages. First it samples its
completion rate while still on interrupts. Only if that rate is high
enough to fill a small batch inside a bounded latency window does it mask
its own IRQ and start draining the CQ from a high-resolution timer, with
each wait sized to collect roughly one batch. It keeps polling as long as
it keeps up with that rate; the moment it stalls or slows down it turns the
IRQ back on and backs off, waiting longer the further behind it fell.
A queue that doesn't benefit drops back quickly and only gets retried once
in a while, so polling stays on the queues that are actually
interrupt-bound and everything else keeps running on the untouched IRQ
path.
Measured with 4 KiB random reads on Solidigm SB5PH27X038T, adaptive on
versus off:
QD32 QD64 QD128
one device, one job +18.44% +24.35% +26.38%
four devices, eight jobs +83.76% +99.02% +96.26%
The four-device eight-job aggregate goes from 5.59M IOPS (44% of the 12.8M
ceiling) to 10.89M IOPS (85%). Tail latency improves too: QD64 p99 drops
from 1073 us to 498 us (-54%) and p99.9 from 1909 us to 741 us (-61%).
Interrupts per I/O drop from about 0.91 to 0.08 (~12x fewer) in the
four-device case, and from 0.96 to 0.03 (~33x fewer) on a single high-QD
queue.
Link:https://lore.kernel.org/linux-nvme/d9210bcdf73fbe1ac8b6ec132865609a3ed68688.ff265e95.1296.491e.89f9.8ae888a03346@bytedance.com/T/#mea881a7898c85b73992f568864001913cb456d59
Signed-off-by: Guzebing <guzebing@bytedance.com>
Signed-off-by: Fengnan Chang <changfengnan@bytedance.com>
---
drivers/nvme/host/Kconfig | 1 +
drivers/nvme/host/pci.c | 468 ++++++++++++++++++++++++++++++++++++--
2 files changed, 452 insertions(+), 17 deletions(-)
diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
index 31974c7dd20c9..22164b901da85 100644
--- a/drivers/nvme/host/Kconfig
+++ b/drivers/nvme/host/Kconfig
@@ -5,6 +5,7 @@ config NVME_CORE
config BLK_DEV_NVME
tristate "NVM Express block device"
depends on PCI && BLOCK
+ select IRQ_POLL
select NVME_CORE
help
The NVM Express driver is for solid state drives directly
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 69932d640b537..18559dd16b752 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -10,9 +10,13 @@
#include <linux/blk-mq-dma.h>
#include <linux/blk-integrity.h>
#include <linux/dmi.h>
+#include <linux/hrtimer.h>
#include <linux/init.h>
#include <linux/interrupt.h>
#include <linux/io.h>
+#include <linux/irq_poll.h>
+#include <linux/jump_label.h>
+#include <linux/ktime.h>
#include <linux/kstrtox.h>
#include <linux/memremap.h>
#include <linux/mm.h>
@@ -82,6 +86,57 @@ struct quirk_entry {
static int use_threaded_interrupts;
module_param(use_threaded_interrupts, int, 0444);
+static DEFINE_STATIC_KEY_FALSE(nvme_adaptive_irq_polling_key);
+static bool use_adaptive_irq_polling;
+
+static int nvme_adaptive_irq_polling_set(const char *val,
+ const struct kernel_param *kp)
+{
+ int ret = param_set_bool(val, kp);
+
+ if (ret)
+ return ret;
+ if (use_adaptive_irq_polling)
+ static_branch_enable(&nvme_adaptive_irq_polling_key);
+ else
+ static_branch_disable(&nvme_adaptive_irq_polling_key);
+ return 0;
+}
+
+static const struct kernel_param_ops nvme_adaptive_irq_polling_ops = {
+ .set = nvme_adaptive_irq_polling_set,
+ .get = param_get_bool,
+};
+
+module_param_cb(use_adaptive_irq_polling, &nvme_adaptive_irq_polling_ops,
+ &use_adaptive_irq_polling, 0644);
+MODULE_PARM_DESC(use_adaptive_irq_polling,
+ "enable adaptive polling on non-threaded MSI-X I/O queues");
+
+/*
+ * Adaptive IRQ polling flips a busy interrupt-driven queue over to a
+ * timer-based poll path and back, based only on how fast that queue is
+ * completing. Each queue goes through three stages:
+ *
+ * 1. Sample (still in IRQ mode): count completions over
+ * NVME_ADAPTIVE_SAMPLE_CQES interrupts and work out the average gap
+ * between them. If the queue is too slow to fill a batch within
+ * NVME_ADAPTIVE_MAX_DELAY_NS, or already fast enough to batch on its own,
+ * leave it alone on the normal IRQ path.
+ * 2. Poll: mask the queue's IRQ and drain the CQ from an hrtimer, arming
+ * each wait for NVME_ADAPTIVE_TARGET_BATCH gaps (but never less than 2 us
+ * or more than NVME_ADAPTIVE_MAX_DELAY_NS). Keep polling as long as the
+ * queue keeps up, up to NVME_ADAPTIVE_EPISODE_CQES completions.
+ * 3. Back off: once a queue falls behind, go back to IRQ mode and skip the
+ * next deficit * NVME_ADAPTIVE_BACKOFF_MULT completions before sampling
+ * it again. Queues that don't benefit get retried only now and then.
+ */
+#define NVME_ADAPTIVE_TARGET_BATCH 5U
+#define NVME_ADAPTIVE_SAMPLE_CQES 256U
+#define NVME_ADAPTIVE_MAX_DELAY_NS (10U * NSEC_PER_USEC)
+#define NVME_ADAPTIVE_EPISODE_CQES (32U * NVME_ADAPTIVE_SAMPLE_CQES)
+#define NVME_ADAPTIVE_BACKOFF_MULT 20U
+
static bool use_cmb_sqes = true;
module_param(use_cmb_sqes, bool, 0444);
MODULE_PARM_DESC(use_cmb_sqes, "use controller's memory buffer for I/O SQes");
@@ -358,6 +413,25 @@ static inline struct nvme_dev *to_nvme_dev(struct nvme_ctrl *ctrl)
return container_of(ctrl, struct nvme_dev, ctrl);
}
+/*
+ * Per-queue adaptive polling state. This sits outside struct nvme_queue on
+ * purpose, so the completion path's layout doesn't change when the feature is
+ * built in but not used. @lock covers every field below; it's separate from
+ * the legacy polling lock because the two paths never touch the same queue at
+ * the same time.
+ */
+struct nvme_adaptive_poll {
+ struct hrtimer timer; /* fires the next poll drain */
+ struct irq_poll iopoll; /* softirq context for the drain */
+ struct nvme_queue *nvmeq;
+ spinlock_t lock;
+ u64 start_ns; /* when the current sample/episode started */
+ u64 retry_completions; /* IRQ completions to skip before sampling again */
+ u32 interval_ns; /* average gap between completions, last sample */
+ u32 completions; /* completions seen so far this sample/episode */
+ int irq;
+};
+
/*
* An NVM Express queue. Each device has at least two (one for admin
* commands and one for I/O commands).
@@ -367,7 +441,8 @@ struct nvme_queue {
struct nvme_descriptor_pools descriptor_pools;
spinlock_t sq_lock;
void *sq_cmds;
- /* only used for poll queues: */
+ struct nvme_adaptive_poll *adaptive;
+ /* Only used for poll queues. */
spinlock_t cq_poll_lock ____cacheline_aligned_in_smp;
struct nvme_completion *cqes;
dma_addr_t sq_dma_addr;
@@ -386,6 +461,8 @@ struct nvme_queue {
#define NVMEQ_SQ_CMB 1
#define NVMEQ_DELETE_ERROR 2
#define NVMEQ_POLLED 3
+#define NVMEQ_ADAPTIVE_POLLING 4
+#define NVMEQ_ADAPTIVE_STALE_IRQ 5
__le32 *dbbuf_sq_db;
__le32 *dbbuf_cq_db;
__le32 *dbbuf_sq_ei;
@@ -1606,13 +1683,12 @@ static inline void nvme_update_cq_head(struct nvme_queue *nvmeq)
}
}
-static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,
- struct io_comp_batch *iob)
+static inline unsigned int nvme_poll_cq(struct nvme_queue *nvmeq,
+ struct io_comp_batch *iob)
{
- bool found = false;
+ unsigned int found = 0;
while (nvme_cqe_pending(nvmeq)) {
- found = true;
/*
* load-load control dependency between phase and the rest of
* the cqe requires a full read memory barrier
@@ -1620,6 +1696,7 @@ static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,
dma_rmb();
nvme_handle_cqe(nvmeq, iob, nvmeq->cq_head);
nvme_update_cq_head(nvmeq);
+ found++;
}
if (found)
@@ -1627,6 +1704,258 @@ static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,
return found;
}
+/* Keep the normal completion loop branch-free. */
+static unsigned int nvme_poll_cq_bounded(struct nvme_queue *nvmeq,
+ struct io_comp_batch *iob,
+ unsigned int limit)
+{
+ unsigned int found = 0;
+
+ while (found < limit && nvme_cqe_pending(nvmeq)) {
+ dma_rmb();
+ nvme_handle_cqe(nvmeq, iob, nvmeq->cq_head);
+ nvme_update_cq_head(nvmeq);
+ found++;
+ }
+ if (found)
+ nvme_ring_cq_doorbell(nvmeq);
+ return found;
+}
+
+static irqreturn_t nvme_irq_check(int irq, void *data)
+{
+ struct nvme_queue *nvmeq = data;
+
+ if (nvme_cqe_pending(nvmeq))
+ return IRQ_WAKE_THREAD;
+ return IRQ_NONE;
+}
+
+static bool nvme_adaptive_enabled(struct nvme_queue *nvmeq)
+{
+ return READ_ONCE(use_adaptive_irq_polling) &&
+ test_bit(NVMEQ_ENABLED, &nvmeq->flags);
+}
+
+/*
+ * Stop polling and turn the queue's IRQ back on. @elapsed is how long the
+ * episode ran after it started falling behind, or 0 if it ended cleanly.
+ * The bigger @elapsed is, the more completions we missed, and the longer we
+ * wait before sampling this queue again, so a queue that polling doesn't
+ * help is left alone most of the time.
+ */
+static void nvme_adaptive_poll_end(struct nvme_queue *nvmeq, u64 elapsed)
+{
+ struct nvme_adaptive_poll *adaptive = nvmeq->adaptive;
+ u64 deficit = 0;
+
+ if (elapsed) {
+ deficit = div64_u64(elapsed - 1, adaptive->interval_ns) + 1;
+ deficit -= min_t(u64, deficit, adaptive->completions);
+ }
+ adaptive->retry_completions =
+ deficit > U64_MAX / NVME_ADAPTIVE_BACKOFF_MULT ? U64_MAX :
+ deficit * NVME_ADAPTIVE_BACKOFF_MULT;
+ adaptive->start_ns = 0;
+ adaptive->completions = 0;
+ set_bit(NVMEQ_ADAPTIVE_STALE_IRQ, &nvmeq->flags);
+ clear_bit(NVMEQ_ADAPTIVE_POLLING, &nvmeq->flags);
+ enable_irq(adaptive->irq);
+}
+
+/*
+ * Set the poll timer to fire one batch from now: how long the sampled
+ * rate needs to produce NVME_ADAPTIVE_TARGET_BATCH completions.
+ */
+static void nvme_adaptive_arm(struct nvme_adaptive_poll *adaptive, u64 now)
+{
+ u64 delay = clamp_t(u64,
+ (u64)adaptive->interval_ns * NVME_ADAPTIVE_TARGET_BATCH,
+ 2U * NSEC_PER_USEC, NVME_ADAPTIVE_MAX_DELAY_NS);
+
+ hrtimer_start(&adaptive->timer,
+ ns_to_ktime(now + delay), HRTIMER_MODE_ABS_PINNED_HARD);
+}
+
+static enum hrtimer_restart nvme_adaptive_poll_timer(struct hrtimer *timer)
+{
+ struct nvme_adaptive_poll *adaptive = container_of(timer,
+ struct nvme_adaptive_poll, timer);
+ struct nvme_queue *nvmeq = adaptive->nvmeq;
+
+ if (test_bit(NVMEQ_ADAPTIVE_POLLING, &nvmeq->flags))
+ irq_poll_sched(&adaptive->iopoll);
+ return HRTIMER_NORESTART;
+}
+
+/*
+ * The poll drain, run from softirq when the timer fires. Reap some CQEs, then
+ * pick one of three things: stop if we hit the episode cap, wait again if the
+ * queue is keeping up, or go back to IRQ mode if it went idle or slowed down.
+ */
+static int nvme_adaptive_irq_poll(struct irq_poll *iop, int budget)
+{
+ struct nvme_adaptive_poll *adaptive = container_of(iop,
+ struct nvme_adaptive_poll, iopoll);
+ struct nvme_queue *nvmeq = adaptive->nvmeq;
+ unsigned int completions, limit;
+ unsigned long flags;
+ bool on_schedule;
+ u64 elapsed, now;
+ DEFINE_IO_COMP_BATCH(iob);
+
+ spin_lock_irqsave(&adaptive->lock, flags);
+ if (unlikely(!test_bit(NVMEQ_ADAPTIVE_POLLING, &nvmeq->flags))) {
+ completions = 0;
+ irq_poll_complete(iop);
+ goto out;
+ }
+
+ limit = min_t(unsigned int,
+ budget - !nvme_adaptive_enabled(nvmeq),
+ NVME_ADAPTIVE_EPISODE_CQES - adaptive->completions);
+ completions = nvme_poll_cq_bounded(nvmeq, &iob, limit);
+ adaptive->completions += completions;
+ if (!rq_list_empty(&iob.req_list))
+ nvme_pci_complete_batch(&iob);
+
+ if (completions >= budget)
+ goto out;
+ irq_poll_complete(iop);
+
+ if (!nvme_adaptive_enabled(nvmeq)) {
+ nvme_adaptive_poll_end(nvmeq, 0);
+ goto out;
+ }
+
+ /*
+ * Only keep polling if the queue is still hitting the sampled rate.
+ * MAX_DELAY leaves room for one empty wait, so a queue that's still
+ * completing keeps polling; one that stalled or slowed down goes back
+ * to IRQ mode and backs off.
+ */
+ now = ktime_get_ns();
+ elapsed = now - adaptive->start_ns;
+ on_schedule = elapsed <= (u64)adaptive->completions *
+ adaptive->interval_ns + NVME_ADAPTIVE_MAX_DELAY_NS;
+ if (adaptive->completions >= NVME_ADAPTIVE_EPISODE_CQES)
+ nvme_adaptive_poll_end(nvmeq, on_schedule ? 0 : elapsed);
+ else if (on_schedule)
+ nvme_adaptive_arm(adaptive, now);
+ else
+ nvme_adaptive_poll_end(nvmeq, elapsed);
+out:
+ spin_unlock_irqrestore(&adaptive->lock, flags);
+ return completions;
+}
+
+/*
+ * Called from the IRQ handler after a reap that found something. If we're
+ * still in backoff, just count it down. Otherwise time how long
+ * NVME_ADAPTIVE_SAMPLE_CQES completions take to get the average gap between
+ * them. If that looks worth polling (see the filter below) mask the IRQ and
+ * switch to poll mode; if not, leave the queue on interrupts.
+ */
+static void nvme_adaptive_sample(struct nvme_queue *nvmeq,
+ unsigned int completions)
+{
+ struct nvme_adaptive_poll *adaptive = nvmeq->adaptive;
+ unsigned int sample;
+ unsigned long flags;
+ u64 delta, interval, now;
+
+ if (adaptive->retry_completions) {
+ if (adaptive->retry_completions != U64_MAX)
+ adaptive->retry_completions -= min_t(u64, completions,
+ adaptive->retry_completions);
+ return;
+ }
+ if (!adaptive->start_ns) {
+ adaptive->start_ns = ktime_get_ns();
+ return;
+ }
+ adaptive->completions += completions;
+ if (adaptive->completions < NVME_ADAPTIVE_SAMPLE_CQES)
+ return;
+
+ now = ktime_get_ns();
+ delta = now - adaptive->start_ns;
+ sample = adaptive->completions;
+ adaptive->start_ns = now;
+ adaptive->completions = 0;
+ if (!delta || delta > (u64)NVME_ADAPTIVE_SAMPLE_CQES *
+ NVME_ADAPTIVE_MAX_DELAY_NS)
+ return;
+ /*
+ * Is this rate worth polling? The 100/99 factor trims 1% off the gap so
+ * a queue sitting right on the threshold isn't pulled in. Skip it if
+ * it's too slow to fill a batch within MAX_DELAY, and also skip it if
+ * it's already fast enough to batch by itself. The hardware's own
+ * coalescing already handles that case, so leave it on interrupts.
+ */
+ interval = div64_u64(delta * 100, sample * 99);
+ if (!interval || interval > NVME_ADAPTIVE_MAX_DELAY_NS ||
+ (delta > NSEC_PER_MSEC &&
+ interval <= NVME_ADAPTIVE_MAX_DELAY_NS /
+ NVME_ADAPTIVE_TARGET_BATCH))
+ return;
+
+ spin_lock_irqsave(&adaptive->lock, flags);
+ if (nvme_adaptive_enabled(nvmeq)) {
+ adaptive->interval_ns = interval;
+ set_bit(NVMEQ_ADAPTIVE_POLLING, &nvmeq->flags);
+ disable_irq_nosync(adaptive->irq);
+ nvme_adaptive_arm(adaptive, now);
+ }
+
+ spin_unlock_irqrestore(&adaptive->lock, flags);
+}
+
+static irqreturn_t nvme_irq(int irq, void *data);
+
+/*
+ * IRQ handler for queues that may switch to adaptive polling. It reaps the CQ
+ * like the normal handler, then feeds the count to the sampler, which may flip
+ * the queue into poll mode. An empty CQ right after an IRQ we masked ourselves
+ * (NVMEQ_ADAPTIVE_STALE_IRQ) is still ours to ack.
+ */
+static noinline irqreturn_t nvme_irq_adaptive_enabled(int irq, void *data)
+{
+ struct nvme_queue *nvmeq = data;
+ unsigned int completions;
+ DEFINE_IO_COMP_BATCH(iob);
+
+ completions = nvme_poll_cq(nvmeq, &iob);
+ if (!completions)
+ return test_and_clear_bit(NVMEQ_ADAPTIVE_STALE_IRQ,
+ &nvmeq->flags) ? IRQ_HANDLED : IRQ_NONE;
+ if (!rq_list_empty(&iob.req_list))
+ nvme_pci_complete_batch(&iob);
+ nvme_adaptive_sample(nvmeq, completions);
+ return IRQ_HANDLED;
+}
+
+static __always_inline bool nvme_adaptive_armed(void)
+{
+ return static_branch_unlikely(&nvme_adaptive_irq_polling_key) &&
+ READ_ONCE(use_adaptive_irq_polling);
+}
+
+static irqreturn_t nvme_irq_adaptive(int irq, void *data)
+{
+ struct nvme_queue *nvmeq = data;
+ irqreturn_t ret;
+
+ if (nvme_adaptive_armed())
+ return nvme_irq_adaptive_enabled(irq, data);
+
+ ret = nvme_irq(irq, data);
+ if (ret == IRQ_NONE &&
+ test_and_clear_bit(NVMEQ_ADAPTIVE_STALE_IRQ, &nvmeq->flags))
+ return IRQ_HANDLED;
+ return ret;
+}
+
static irqreturn_t nvme_irq(int irq, void *data)
{
struct nvme_queue *nvmeq = data;
@@ -1640,13 +1969,20 @@ static irqreturn_t nvme_irq(int irq, void *data)
return IRQ_NONE;
}
-static irqreturn_t nvme_irq_check(int irq, void *data)
+static unsigned long nvme_adaptive_lock(struct nvme_queue *nvmeq)
{
- struct nvme_queue *nvmeq = data;
+ unsigned long flags = 0;
- if (nvme_cqe_pending(nvmeq))
- return IRQ_WAKE_THREAD;
- return IRQ_NONE;
+ if (nvmeq->adaptive)
+ spin_lock_irqsave(&nvmeq->adaptive->lock, flags);
+ return flags;
+}
+
+static void nvme_adaptive_unlock(struct nvme_queue *nvmeq,
+ unsigned long flags)
+{
+ if (nvmeq->adaptive)
+ spin_unlock_irqrestore(&nvmeq->adaptive->lock, flags);
}
/*
@@ -1656,15 +1992,18 @@ static irqreturn_t nvme_irq_check(int irq, void *data)
static void nvme_poll_irqdisable(struct nvme_queue *nvmeq)
{
struct pci_dev *pdev = to_pci_dev(nvmeq->dev->dev);
+ unsigned long flags;
int irq;
WARN_ON_ONCE(test_bit(NVMEQ_POLLED, &nvmeq->flags));
irq = pci_irq_vector(pdev, nvmeq->cq_vector);
disable_irq(irq);
+ flags = nvme_adaptive_lock(nvmeq);
spin_lock(&nvmeq->cq_poll_lock);
nvme_poll_cq(nvmeq, NULL);
spin_unlock(&nvmeq->cq_poll_lock);
+ nvme_adaptive_unlock(nvmeq, flags);
enable_irq(irq);
}
@@ -2017,8 +2356,7 @@ static void nvme_free_queue(struct nvme_queue *nvmeq)
dma_free_coherent(nvmeq->dev->dev, CQ_SIZE(nvmeq),
(void *)nvmeq->cqes, nvmeq->cq_dma_addr);
if (!nvmeq->sq_cmds)
- return;
-
+ goto free_adaptive;
if (test_and_clear_bit(NVMEQ_SQ_CMB, &nvmeq->flags)) {
pci_free_p2pmem(to_pci_dev(nvmeq->dev->dev),
nvmeq->sq_cmds, SQ_SIZE(nvmeq));
@@ -2026,6 +2364,9 @@ static void nvme_free_queue(struct nvme_queue *nvmeq)
dma_free_coherent(nvmeq->dev->dev, SQ_SIZE(nvmeq),
nvmeq->sq_cmds, nvmeq->sq_dma_addr);
}
+free_adaptive:
+ kfree(nvmeq->adaptive);
+ nvmeq->adaptive = NULL;
}
static void nvme_free_queues(struct nvme_dev *dev, int lowest)
@@ -2038,9 +2379,42 @@ static void nvme_free_queues(struct nvme_dev *dev, int lowest)
}
}
+static int nvme_adaptive_suspend(struct nvme_queue *nvmeq)
+{
+ struct nvme_adaptive_poll *adaptive = nvmeq->adaptive;
+ unsigned long flags;
+ int irq;
+
+ if (!adaptive || adaptive->irq < 0)
+ return -1;
+ irq = adaptive->irq;
+ synchronize_irq(irq);
+ irq_poll_disable(&adaptive->iopoll);
+ spin_lock_irqsave(&adaptive->lock, flags);
+ spin_unlock_irqrestore(&adaptive->lock, flags);
+ hrtimer_cancel(&adaptive->timer);
+ spin_lock_irqsave(&adaptive->lock, flags);
+ if (test_and_clear_bit(NVMEQ_ADAPTIVE_POLLING, &nvmeq->flags)) {
+ set_bit(NVMEQ_ADAPTIVE_STALE_IRQ, &nvmeq->flags);
+ enable_irq(irq);
+ }
+ spin_unlock_irqrestore(&adaptive->lock, flags);
+ return irq;
+}
+
+static void nvme_adaptive_suspend_done(struct nvme_queue *nvmeq, int irq)
+{
+ if (irq < 0)
+ return;
+ nvmeq->adaptive->irq = -1;
+ irq_poll_enable(&nvmeq->adaptive->iopoll);
+}
+
static void nvme_suspend_queue(struct nvme_dev *dev, unsigned int qid)
{
struct nvme_queue *nvmeq = &dev->queues[qid];
+ struct pci_dev *pdev = to_pci_dev(dev->dev);
+ int irq;
if (!test_and_clear_bit(NVMEQ_ENABLED, &nvmeq->flags))
return;
@@ -2051,8 +2425,11 @@ static void nvme_suspend_queue(struct nvme_dev *dev, unsigned int qid)
nvmeq->dev->online_queues--;
if (!nvmeq->qid && nvmeq->dev->ctrl.admin_q)
nvme_quiesce_admin_queue(&nvmeq->dev->ctrl);
- if (!test_and_clear_bit(NVMEQ_POLLED, &nvmeq->flags))
- pci_free_irq(to_pci_dev(dev->dev), nvmeq->cq_vector, nvmeq);
+ if (!test_and_clear_bit(NVMEQ_POLLED, &nvmeq->flags)) {
+ irq = nvme_adaptive_suspend(nvmeq);
+ pci_free_irq(pdev, nvmeq->cq_vector, nvmeq);
+ nvme_adaptive_suspend_done(nvmeq, irq);
+ }
}
static void nvme_suspend_io_queues(struct nvme_dev *dev)
@@ -2166,18 +2543,74 @@ static int nvme_alloc_queue(struct nvme_dev *dev, int qid, int depth)
return -ENOMEM;
}
+static bool nvme_adaptive_init(struct nvme_queue *nvmeq)
+{
+ struct nvme_adaptive_poll *adaptive = nvmeq->adaptive;
+ int irq = pci_irq_vector(to_pci_dev(nvmeq->dev->dev),
+ nvmeq->cq_vector);
+
+ if (irq < 0)
+ return false;
+ if (!adaptive) {
+ adaptive = kzalloc_node(sizeof(*adaptive), GFP_KERNEL,
+ dev_to_node(nvmeq->dev->dev));
+ if (!adaptive)
+ return false;
+ adaptive->nvmeq = nvmeq;
+ spin_lock_init(&adaptive->lock);
+ hrtimer_setup(&adaptive->timer, nvme_adaptive_poll_timer,
+ CLOCK_MONOTONIC, HRTIMER_MODE_ABS_PINNED_HARD);
+ irq_poll_init(&adaptive->iopoll, 64, nvme_adaptive_irq_poll);
+ adaptive->irq = irq;
+ WRITE_ONCE(nvmeq->adaptive, adaptive);
+ return true;
+ }
+ adaptive->irq = irq;
+ return true;
+}
+
static int queue_request_irq(struct nvme_queue *nvmeq)
{
struct pci_dev *pdev = to_pci_dev(nvmeq->dev->dev);
int nr = nvmeq->dev->ctrl.instance;
+ bool adaptive_queue;
+ int ret;
if (use_threaded_interrupts) {
return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq_check,
nvme_irq, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
- } else {
- return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq,
- NULL, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
}
+ /*
+ * Decide once, at setup, whether this queue can ever poll adaptively.
+ * nvme_irq_adaptive() is only installed when the adaptive state is
+ * allocated and armed here, so its fast path never has to re-check
+ * these static queue properties on every completion IRQ.
+ */
+ adaptive_queue = nvmeq->qid && nvmeq->dev->num_vecs > 1 &&
+ pdev->msix_enabled &&
+ nvmeq->q_depth >= NVME_ADAPTIVE_TARGET_BATCH;
+ if (adaptive_queue)
+ adaptive_queue = nvme_adaptive_init(nvmeq);
+ ret = pci_request_irq(pdev, nvmeq->cq_vector,
+ adaptive_queue ? nvme_irq_adaptive : nvme_irq,
+ NULL, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
+ if (ret && nvmeq->adaptive)
+ nvmeq->adaptive->irq = -1;
+ return ret;
+}
+
+static void nvme_adaptive_reset(struct nvme_queue *nvmeq)
+{
+ struct nvme_adaptive_poll *adaptive = nvmeq->adaptive;
+
+ clear_bit(NVMEQ_ADAPTIVE_POLLING, &nvmeq->flags);
+ clear_bit(NVMEQ_ADAPTIVE_STALE_IRQ, &nvmeq->flags);
+ if (!adaptive)
+ return;
+ adaptive->start_ns = 0;
+ adaptive->completions = 0;
+ adaptive->retry_completions = 0;
+ adaptive->irq = -1;
}
static void nvme_init_queue(struct nvme_queue *nvmeq, u16 qid)
@@ -2188,6 +2621,7 @@ static void nvme_init_queue(struct nvme_queue *nvmeq, u16 qid)
nvmeq->last_sq_tail = 0;
nvmeq->cq_head = 0;
nvmeq->cq_phase = 1;
+ nvme_adaptive_reset(nvmeq);
nvmeq->q_db = &dev->dbs[qid * 2 * dev->db_stride];
memset((void *)nvmeq->cqes, 0, CQ_SIZE(nvmeq));
nvme_dbbuf_init(dev, nvmeq, qid);
--
2.39.5 (Apple Git-154)
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [RESEND RFC PATCH v2] nvme-pci: add adaptive interrupt polling
2026-08-06 3:10 [RESEND RFC PATCH v2] nvme-pci: add adaptive interrupt polling Fengnan Chang
@ 2026-08-10 20:55 ` Keith Busch
2026-08-11 2:32 ` changfengnan
0 siblings, 1 reply; 3+ messages in thread
From: Keith Busch @ 2026-08-10 20:55 UTC (permalink / raw)
To: Fengnan Chang; +Cc: axboe, hch, sagi, linux-nvme, linux-kernel, Guzebing
On Thu, Aug 06, 2026 at 11:10:58AM +0800, Fengnan Chang wrote:
> The idea behind this approach is: Let each I/O queue switch itself between
> interrupt and poll mode based on its own recent completion rate.
>
> This version is still in the testing phase, and there are still some issues
> with the code implementation. I releasing it now to see if the approach
> is generally acceptable. If the approach looks good, I´ll continue to
> refine it and conduct more extensive testing. The main implementation
> logic is in `nvme_adaptive_sample` and `nvme_adaptive_irq_poll`; you should
> focus on reviewing the implementation of these two functions.
Can we subscribe to the dynamic interrupt moderation (dim) library? I
know it's generally used in conjuction with a hardware interrupt
coalescing feature, but we can just do pure software with it too. The
library provides a hill climb to adapt the policy at run time.
This is a quick PoC I put together. I haven't tested on fast devices, so
I'm not sure if I've dialed in the profiles, but it's start of what I
had in mind.
---
diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
index 31974c7dd20c9..9fad1c7b1f678 100644
--- a/drivers/nvme/host/Kconfig
+++ b/drivers/nvme/host/Kconfig
@@ -6,6 +6,7 @@ config BLK_DEV_NVME
tristate "NVM Express block device"
depends on PCI && BLOCK
select NVME_CORE
+ select DIMLIB
help
The NVM Express driver is for solid state drives directly
connected to the PCI or PCI Express bus. If you know you
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 8438c904ec496..ff0c8d74fae08 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -9,6 +9,8 @@
#include <linux/blkdev.h>
#include <linux/blk-mq-dma.h>
#include <linux/blk-integrity.h>
+#include <linux/delay.h>
+#include <linux/dim.h>
#include <linux/dmi.h>
#include <linux/init.h>
#include <linux/interrupt.h>
@@ -82,6 +84,45 @@ struct quirk_entry {
static int use_threaded_interrupts;
module_param(use_threaded_interrupts, int, 0444);
+static unsigned int irq_poll_thresh = 7;
+module_param(irq_poll_thresh, uint, 0644);
+MODULE_PARM_DESC(irq_poll_thresh,
+ "outstanding depth after which a queue switches to threaded polling");
+
+static unsigned int irq_poll_spin = 7;
+module_param(irq_poll_spin, uint, 0644);
+MODULE_PARM_DESC(irq_poll_spin,
+ "poll iterations to spin (cpu_relax) before sleeping in the poll loop (zero based)");
+
+static unsigned int irq_poll_sleep_us = 20;
+static unsigned int irq_poll_idle_us = 80;
+static unsigned int irq_poll_idle_rounds = DIV_ROUND_UP(80, 20);
+
+static int irq_poll_us_set(const char *val, const struct kernel_param *kp)
+{
+ int ret = param_set_uint(val, kp);
+
+ if (ret)
+ return ret;
+
+ irq_poll_idle_rounds = irq_poll_sleep_us ?
+ DIV_ROUND_UP(irq_poll_idle_us, irq_poll_sleep_us) : 1;
+ return 0;
+}
+
+static const struct kernel_param_ops irq_poll_us_ops = {
+ .set = irq_poll_us_set,
+ .get = param_get_uint,
+};
+
+module_param_cb(irq_poll_sleep_us, &irq_poll_us_ops, &irq_poll_sleep_us, 0644);
+MODULE_PARM_DESC(irq_poll_sleep_us,
+ "microseconds to sleep between poll bursts (<=10 busy-delays, does not yield)");
+
+module_param_cb(irq_poll_idle_us, &irq_poll_us_ops, &irq_poll_idle_us, 0644);
+MODULE_PARM_DESC(irq_poll_idle_us,
+ "microseconds to wait for stragglers before handing a queue back to the IRQ path");
+
static bool use_cmb_sqes = true;
module_param(use_cmb_sqes, bool, 0444);
MODULE_PARM_DESC(use_cmb_sqes, "use controller's memory buffer for I/O SQes");
@@ -381,11 +422,21 @@ struct nvme_queue {
u16 qid;
u8 cq_phase;
u8 sqes;
+ /* Adaptive polling knobs, seeded from the irq_poll_* module params. */
+ unsigned int poll_thresh;
+ unsigned int poll_sleep_us;
+ unsigned int poll_idle_rounds;
+ unsigned int poll_budget;
+ /* Adaptive interrupt moderation (DIM) sampling state. */
+ struct dim dim;
+ u16 dim_events; /* cumulative interrupts (BIT_GAP-safe) */
+ u32 dim_comps; /* cumulative completions */
unsigned long flags;
#define NVMEQ_ENABLED 0
#define NVMEQ_SQ_CMB 1
#define NVMEQ_DELETE_ERROR 2
#define NVMEQ_POLLED 3
+#define NVMEQ_POLLING 4
__le32 *dbbuf_sq_db;
__le32 *dbbuf_cq_db;
__le32 *dbbuf_sq_ei;
@@ -1606,13 +1657,13 @@ static inline void nvme_update_cq_head(struct nvme_queue *nvmeq)
}
}
-static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,
- struct io_comp_batch *iob)
+static inline int __nvme_poll_cq(struct nvme_queue *nvmeq,
+ struct io_comp_batch *iob, int budget)
{
- bool found = false;
+ int found = 0;
while (nvme_cqe_pending(nvmeq)) {
- found = true;
+ found++;
/*
* load-load control dependency between phase and the rest of
* the cqe requires a full read memory barrier
@@ -1620,6 +1671,8 @@ static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,
dma_rmb();
nvme_handle_cqe(nvmeq, iob, nvmeq->cq_head);
nvme_update_cq_head(nvmeq);
+ if (budget && found == budget)
+ break;
}
if (found)
@@ -1627,26 +1680,253 @@ static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,
return found;
}
+/* budget == 0 means reap the whole queue. */
+static int nvme_poll_cq(struct nvme_queue *nvmeq, int budget)
+{
+ DEFINE_IO_COMP_BATCH(iob);
+ int found = __nvme_poll_cq(nvmeq, &iob, budget);
+
+ if (found && !rq_list_empty(&iob.req_list))
+ nvme_pci_complete_batch(&iob);
+ return found;
+}
+
+static inline unsigned int nvmeq_outstanding(struct nvme_queue *nvmeq)
+{
+ u16 sq_tail = READ_ONCE(nvmeq->last_sq_tail);
+ u16 cq_head = nvmeq->cq_head;
+
+ if (sq_tail >= cq_head)
+ return sq_tail - cq_head;
+ return nvmeq->q_depth - cq_head + sq_tail;
+}
+
+static inline bool nvmeq_cq_continue(struct nvme_queue *nvmeq)
+{
+ return nvmeq_outstanding(nvmeq) > nvmeq->poll_thresh;
+}
+
+static inline bool nvmeq_cq_idle(struct nvme_queue *nvmeq)
+{
+ return nvmeq->cq_head == READ_ONCE(nvmeq->last_sq_tail);
+}
+
+/*
+ * Interrupt-moderation profiles, ordered from least moderation (index 0: enter
+ * polling late, short sleep, brief linger -> lowest latency, most interrupts)
+ * to most moderation (enter early, long sleep, long linger -> fewest
+ * interrupts). The hill-climb walks this table to keep completions-per-
+ * interrupt high, which is what bounds the interrupt rate.
+ */
+static const struct nvme_poll_prof {
+ u16 thresh;
+ u16 sleep_us;
+ u16 idle_rounds;
+ u16 budget; /* hard-IRQ initial poll cap; 0 == reap all */
+} nvme_poll_prof[] = {
+ { 31, 0, 1, 0 }, /* drain inline: lowest latency, most interrupts */
+ { 15, 10, 2, 16 },
+ { 7, 20, 4, 8 }, /* start profile; matches irq_poll_* defaults */
+ { 3, 40, 8, 4 },
+ { 1, 80, 16, 2 }, /* reap a little, hand the bulk to the thread */
+};
+
+#define NVME_DIM_START_PROFILE 2
+
+static void nvme_apply_profile(struct nvme_queue *nvmeq)
+{
+ const struct nvme_poll_prof *p = &nvme_poll_prof[nvmeq->dim.profile_ix];
+
+ nvmeq->poll_thresh = p->thresh;
+ nvmeq->poll_sleep_us = p->sleep_us;
+ nvmeq->poll_idle_rounds = p->idle_rounds;
+ nvmeq->poll_budget = p->budget;
+}
+
+/* Mirror of rdma_dim_step() bounded to our profile table. */
+static int nvme_dim_step(struct dim *dim)
+{
+ if (dim->tune_state == DIM_GOING_RIGHT) {
+ if (dim->profile_ix == ARRAY_SIZE(nvme_poll_prof) - 1)
+ return DIM_ON_EDGE;
+ dim->profile_ix++;
+ dim->steps_right++;
+ }
+ if (dim->tune_state == DIM_GOING_LEFT) {
+ if (dim->profile_ix == 0)
+ return DIM_ON_EDGE;
+ dim->profile_ix--;
+ dim->steps_left++;
+ }
+ return DIM_STEPPED;
+}
+
+/* Mirror of rdma_dim_stats_compare(): completion rate first, then batching. */
+static int nvme_dim_stats_compare(struct dim_stats *curr, struct dim_stats *prev)
+{
+ if (!prev->cpms)
+ return DIM_STATS_SAME;
+
+ if (IS_SIGNIFICANT_DIFF(curr->cpms, prev->cpms))
+ return curr->cpms > prev->cpms ? DIM_STATS_BETTER :
+ DIM_STATS_WORSE;
+
+ if (IS_SIGNIFICANT_DIFF(curr->cpe_ratio, prev->cpe_ratio))
+ return curr->cpe_ratio > prev->cpe_ratio ? DIM_STATS_BETTER :
+ DIM_STATS_WORSE;
+
+ return DIM_STATS_SAME;
+}
+
+/* Mirror of rdma_dim_decision(); returns true if the profile changed. */
+static bool nvme_dim_decision(struct dim_stats *curr, struct dim *dim)
+{
+ int prev_ix = dim->profile_ix;
+ int stats_res;
+
+ stats_res = nvme_dim_stats_compare(curr, &dim->prev_stats);
+ switch (stats_res) {
+ case DIM_STATS_SAME:
+ if (curr->cpe_ratio <= 50 * prev_ix)
+ dim->profile_ix = 0;
+ break;
+ case DIM_STATS_WORSE:
+ dim_turn(dim);
+ fallthrough;
+ case DIM_STATS_BETTER:
+ if (nvme_dim_step(dim) == DIM_ON_EDGE)
+ dim_turn(dim);
+ break;
+ }
+
+ dim->prev_stats = *curr;
+ return dim->profile_ix != prev_ix;
+}
+
+/*
+ * Sample the completion/interrupt counters and, once per DIM_NEVENTS
+ * interrupts, compute the load stats. Called only from the interrupt owner
+ * (never concurrently with the poll thread), so the counters have a single
+ * writer. ktime_get() is taken only at a window boundary, not per interrupt.
+ *
+ * On each completed window the hill-climb picks a profile and, if it changed,
+ * applies it to nvmeq->poll_*.
+ */
+static void nvme_dim(struct nvme_queue *nvmeq)
+{
+ struct dim *dim = &nvmeq->dim;
+ struct dim_sample end;
+ struct dim_stats stats;
+
+ if (dim->state == DIM_START_MEASURE) {
+ dim_update_sample_with_comps(nvmeq->dim_events, 0, 0,
+ nvmeq->dim_comps, &dim->start_sample);
+ dim->state = DIM_MEASURE_IN_PROGRESS;
+ return;
+ }
+
+ /* Cheap gate: only recompute once a full window of events accrues. */
+ if ((u16)(nvmeq->dim_events - dim->start_sample.event_ctr) < DIM_NEVENTS)
+ return;
+
+ dim_update_sample_with_comps(nvmeq->dim_events, 0, 0, nvmeq->dim_comps,
+ &end);
+ if (dim_calc_stats(&dim->start_sample, &end, &stats)) {
+ if (nvme_dim_decision(&stats, dim))
+ nvme_apply_profile(nvmeq);
+ trace_nvme_dim(nvmeq->qid, stats.cpms, stats.epms,
+ stats.cpe_ratio, dim->profile_ix);
+ }
+ dim->start_sample = end;
+}
+
static irqreturn_t nvme_irq(int irq, void *data)
{
struct nvme_queue *nvmeq = data;
- DEFINE_IO_COMP_BATCH(iob);
+ struct nvme_dev *dev = nvmeq->dev;
+ struct pci_dev *pdev = to_pci_dev(dev->dev);
+ irqreturn_t ret = IRQ_NONE;
+ unsigned int idle = 0;
- if (nvme_poll_cq(nvmeq, &iob)) {
- if (!rq_list_empty(&iob.req_list))
- nvme_pci_complete_batch(&iob);
- return IRQ_HANDLED;
+ for (;;) {
+ bool worked = false;
+ unsigned int i;
+
+ for (i = 0; i <= irq_poll_spin; i++) {
+ int n = nvme_poll_cq(nvmeq, 0);
+
+ if (n) {
+ nvmeq->dim_comps += n;
+ ret = IRQ_HANDLED;
+ worked = true;
+ idle = 0;
+ }
+
+ if (nvmeq_cq_idle(nvmeq))
+ goto done;
+ else if (need_resched())
+ cond_resched();
+ else
+ cpu_relax();
+ }
+
+ if (worked || ++idle < nvmeq->poll_idle_rounds) {
+ fsleep(nvmeq->poll_sleep_us);
+ continue;
+ }
+done:
+ clear_bit(NVMEQ_POLLING, &nvmeq->flags);
+ if (!nvme_cqe_pending(nvmeq) ||
+ test_and_set_bit(NVMEQ_POLLING, &nvmeq->flags))
+ break;
+
+ if (need_resched())
+ cond_resched();
+ else
+ cpu_relax();
+ idle = 0;
}
- return IRQ_NONE;
+
+ if (pdev->msi_enabled)
+ writel(BIT(nvmeq->cq_vector), dev->bar + NVME_REG_INTMC);
+ return ret;
}
static irqreturn_t nvme_irq_check(int irq, void *data)
{
struct nvme_queue *nvmeq = data;
+ int found;
+
+ nvmeq->dim_events++;
- if (nvme_cqe_pending(nvmeq))
+ if (test_and_set_bit(NVMEQ_POLLING, &nvmeq->flags))
+ return IRQ_HANDLED;
+
+ found = nvme_poll_cq(nvmeq, nvmeq->poll_budget);
+ nvmeq->dim_comps += found;
+ nvme_dim(nvmeq);
+
+ if (!found) {
+ clear_bit(NVMEQ_POLLING, &nvmeq->flags);
+ return IRQ_NONE;
+ }
+
+ /*
+ * Hand off to the poll thread when the queue is still deep, or when the
+ * budgeted initial poll left CQEs behind: their interrupt is already
+ * spent, so releasing here would strand them.
+ */
+ if (nvme_cqe_pending(nvmeq) || nvmeq_cq_continue(nvmeq)) {
+ struct nvme_dev *dev = nvmeq->dev;
+ struct pci_dev *pdev = to_pci_dev(dev->dev);
+
+ if (pdev->msi_enabled)
+ writel(BIT(nvmeq->cq_vector), dev->bar + NVME_REG_INTMS);
return IRQ_WAKE_THREAD;
- return IRQ_NONE;
+ }
+
+ clear_bit(NVMEQ_POLLING, &nvmeq->flags);
+ return IRQ_HANDLED;
}
/*
@@ -1663,7 +1943,7 @@ static void nvme_poll_irqdisable(struct nvme_queue *nvmeq)
irq = pci_irq_vector(pdev, nvmeq->cq_vector);
disable_irq(irq);
spin_lock(&nvmeq->cq_poll_lock);
- nvme_poll_cq(nvmeq, NULL);
+ __nvme_poll_cq(nvmeq, NULL, 0);
spin_unlock(&nvmeq->cq_poll_lock);
enable_irq(irq);
}
@@ -1671,14 +1951,14 @@ static void nvme_poll_irqdisable(struct nvme_queue *nvmeq)
static int nvme_poll(struct blk_mq_hw_ctx *hctx, struct io_comp_batch *iob)
{
struct nvme_queue *nvmeq = hctx->driver_data;
- bool found;
+ int found;
if (!test_bit(NVMEQ_POLLED, &nvmeq->flags) ||
!nvme_cqe_pending(nvmeq))
return 0;
spin_lock(&nvmeq->cq_poll_lock);
- found = nvme_poll_cq(nvmeq, iob);
+ found = __nvme_poll_cq(nvmeq, iob, 0);
spin_unlock(&nvmeq->cq_poll_lock);
return found;
@@ -2075,7 +2355,7 @@ static void nvme_reap_pending_cqes(struct nvme_dev *dev)
for (i = dev->ctrl.queue_count - 1; i > 0; i--) {
spin_lock(&dev->queues[i].cq_poll_lock);
- nvme_poll_cq(&dev->queues[i], NULL);
+ __nvme_poll_cq(&dev->queues[i], NULL, 0);
spin_unlock(&dev->queues[i].cq_poll_lock);
}
}
@@ -2171,13 +2451,8 @@ static int queue_request_irq(struct nvme_queue *nvmeq)
struct pci_dev *pdev = to_pci_dev(nvmeq->dev->dev);
int nr = nvmeq->dev->ctrl.instance;
- if (use_threaded_interrupts) {
- return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq_check,
- nvme_irq, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
- } else {
- return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq,
- NULL, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
- }
+ return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq_check,
+ nvme_irq, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
}
static void nvme_init_queue(struct nvme_queue *nvmeq, u16 qid)
@@ -2188,6 +2463,17 @@ static void nvme_init_queue(struct nvme_queue *nvmeq, u16 qid)
nvmeq->last_sq_tail = 0;
nvmeq->cq_head = 0;
nvmeq->cq_phase = 1;
+ nvmeq->poll_thresh = irq_poll_thresh;
+ nvmeq->poll_sleep_us = irq_poll_sleep_us;
+ nvmeq->poll_idle_rounds = irq_poll_idle_rounds;
+ nvmeq->poll_budget = nvme_poll_prof[NVME_DIM_START_PROFILE].budget;
+ memset(&nvmeq->dim, 0, sizeof(nvmeq->dim));
+ nvmeq->dim.priv = nvmeq;
+ nvmeq->dim.profile_ix = NVME_DIM_START_PROFILE;
+ nvmeq->dim.tune_state = DIM_GOING_RIGHT;
+ nvmeq->dim_events = 0;
+ nvmeq->dim_comps = 0;
+ clear_bit(NVMEQ_POLLING, &nvmeq->flags);
nvmeq->q_db = &dev->dbs[qid * 2 * dev->db_stride];
memset((void *)nvmeq->cqes, 0, CQ_SIZE(nvmeq));
nvme_dbbuf_init(dev, nvmeq, qid);
diff --git a/drivers/nvme/host/trace.c b/drivers/nvme/host/trace.c
index ad25ad1e40412..4bdd5abc60880 100644
--- a/drivers/nvme/host/trace.c
+++ b/drivers/nvme/host/trace.c
@@ -498,3 +498,4 @@ const char *nvme_trace_disk_name(struct trace_seq *p, char *name)
}
EXPORT_TRACEPOINT_SYMBOL_GPL(nvme_sq);
+EXPORT_TRACEPOINT_SYMBOL_GPL(nvme_dim);
diff --git a/drivers/nvme/host/trace.h b/drivers/nvme/host/trace.h
index 4fb5922ffdac5..b7099b3026079 100644
--- a/drivers/nvme/host/trace.h
+++ b/drivers/nvme/host/trace.h
@@ -161,6 +161,29 @@ TRACE_EVENT(nvme_sq,
)
);
+TRACE_EVENT(nvme_dim,
+ TP_PROTO(u16 qid, int cpms, int epms, int cpe_ratio, u8 profile),
+ TP_ARGS(qid, cpms, epms, cpe_ratio, profile),
+ TP_STRUCT__entry(
+ __field(u16, qid)
+ __field(int, cpms)
+ __field(int, epms)
+ __field(int, cpe_ratio)
+ __field(u8, profile)
+ ),
+ TP_fast_assign(
+ __entry->qid = qid;
+ __entry->cpms = cpms;
+ __entry->epms = epms;
+ __entry->cpe_ratio = cpe_ratio;
+ __entry->profile = profile;
+ ),
+ TP_printk("qid=%u cpms=%d epms=%d cpe_ratio=%d profile=%u",
+ __entry->qid, __entry->cpms, __entry->epms,
+ __entry->cpe_ratio, __entry->profile
+ )
+);
+
#endif /* _TRACE_NVME_H */
#undef TRACE_INCLUDE_PATH
--
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [RESEND RFC PATCH v2] nvme-pci: add adaptive interrupt polling
2026-08-10 20:55 ` Keith Busch
@ 2026-08-11 2:32 ` changfengnan
0 siblings, 0 replies; 3+ messages in thread
From: changfengnan @ 2026-08-11 2:32 UTC (permalink / raw)
To: Keith Busch; +Cc: axboe, hch, sagi, linux-nvme, linux-kernel, Guzebing
> From: "Keith Busch"<kbusch@kernel.org>
> Date: Tue, Aug 11, 2026, 04:55
> Subject: Re: [RESEND RFC PATCH v2] nvme-pci: add adaptive interrupt polling
> To: "Fengnan Chang"<changfengnan@bytedance.com>
> Cc: <axboe@kernel.dk>, <hch@lst.de>, <sagi@grimberg.me>, <linux-nvme@lists.infradead.org>, <linux-kernel@vger.kernel.org>, "Guzebing"<guzebing@bytedance.com>
> On Thu, Aug 06, 2026 at 11:10:58AM +0800, Fengnan Chang wrote:
> > The idea behind this approach is: Let each I/O queue switch itself between
> > interrupt and poll mode based on its own recent completion rate.
> >
> > This version is still in the testing phase, and there are still some issues
> > with the code implementation. I releasing it now to see if the approach
> > is generally acceptable. If the approach looks good, I´ll continue to
> > refine it and conduct more extensive testing. The main implementation
> > logic is in `nvme_adaptive_sample` and `nvme_adaptive_irq_poll`; you should
> > focus on reviewing the implementation of these two functions.
>
> Can we subscribe to the dynamic interrupt moderation (dim) library? I
> know it's generally used in conjuction with a hardware interrupt
> coalescing feature, but we can just do pure software with it too. The
> library provides a hill climb to adapt the policy at run time.
>
> This is a quick PoC I put together. I haven't tested on fast devices, so
> I'm not sure if I've dialed in the profiles, but it's start of what I
> had in mind.
The code looks much cleaner when using dim, I'll see if I can replace the
sample-and-poll logic I wrote myself with the dim library.
I ran a quick test on the POC patch and didn't see any performance
improvements; in fact, there were quite a few regressions. Maybe some
parameters need to be adjusted.
>
> ---
> diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
> index 31974c7dd20c9..9fad1c7b1f678 100644
> --- a/drivers/nvme/host/Kconfig
> +++ b/drivers/nvme/host/Kconfig
> @@ -6,6 +6,7 @@ config BLK_DEV_NVME
> tristate "NVM Express block device"
> depends on PCI && BLOCK
> select NVME_CORE
> + select DIMLIB
> help
> The NVM Express driver is for solid state drives directly
> connected to the PCI or PCI Express bus. If you know you
> diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
> index 8438c904ec496..ff0c8d74fae08 100644
> --- a/drivers/nvme/host/pci.c
> +++ b/drivers/nvme/host/pci.c
> @@ -9,6 +9,8 @@
> #include <linux/blkdev.h>
> #include <linux/blk-mq-dma.h>
> #include <linux/blk-integrity.h>
> +#include <linux/delay.h>
> +#include <linux/dim.h>
> #include <linux/dmi.h>
> #include <linux/init.h>
> #include <linux/interrupt.h>
> @@ -82,6 +84,45 @@ struct quirk_entry {
> static int use_threaded_interrupts;
> module_param(use_threaded_interrupts, int, 0444);
>
> +static unsigned int irq_poll_thresh = 7;
> +module_param(irq_poll_thresh, uint, 0644);
> +MODULE_PARM_DESC(irq_poll_thresh,
> + "outstanding depth after which a queue switches to threaded polling");
> +
> +static unsigned int irq_poll_spin = 7;
> +module_param(irq_poll_spin, uint, 0644);
> +MODULE_PARM_DESC(irq_poll_spin,
> + "poll iterations to spin (cpu_relax) before sleeping in the poll loop (zero based)");
> +
> +static unsigned int irq_poll_sleep_us = 20;
> +static unsigned int irq_poll_idle_us = 80;
> +static unsigned int irq_poll_idle_rounds = DIV_ROUND_UP(80, 20);
> +
> +static int irq_poll_us_set(const char *val, const struct kernel_param *kp)
> +{
> + int ret = param_set_uint(val, kp);
> +
> + if (ret)
> + return ret;
> +
> + irq_poll_idle_rounds = irq_poll_sleep_us ?
> + DIV_ROUND_UP(irq_poll_idle_us, irq_poll_sleep_us) : 1;
> + return 0;
> +}
> +
> +static const struct kernel_param_ops irq_poll_us_ops = {
> + .set = irq_poll_us_set,
> + .get = param_get_uint,
> +};
> +
> +module_param_cb(irq_poll_sleep_us, &irq_poll_us_ops, &irq_poll_sleep_us, 0644);
> +MODULE_PARM_DESC(irq_poll_sleep_us,
> + "microseconds to sleep between poll bursts (<=10 busy-delays, does not yield)");
> +
> +module_param_cb(irq_poll_idle_us, &irq_poll_us_ops, &irq_poll_idle_us, 0644);
> +MODULE_PARM_DESC(irq_poll_idle_us,
> + "microseconds to wait for stragglers before handing a queue back to the IRQ path");
> +
> static bool use_cmb_sqes = true;
> module_param(use_cmb_sqes, bool, 0444);
> MODULE_PARM_DESC(use_cmb_sqes, "use controller's memory buffer for I/O SQes");
> @@ -381,11 +422,21 @@ struct nvme_queue {
> u16 qid;
> u8 cq_phase;
> u8 sqes;
> + /* Adaptive polling knobs, seeded from the irq_poll_* module params. */
> + unsigned int poll_thresh;
> + unsigned int poll_sleep_us;
> + unsigned int poll_idle_rounds;
> + unsigned int poll_budget;
> + /* Adaptive interrupt moderation (DIM) sampling state. */
> + struct dim dim;
> + u16 dim_events; /* cumulative interrupts (BIT_GAP-safe) */
> + u32 dim_comps; /* cumulative completions */
> unsigned long flags;
> #define NVMEQ_ENABLED 0
> #define NVMEQ_SQ_CMB 1
> #define NVMEQ_DELETE_ERROR 2
> #define NVMEQ_POLLED 3
> +#define NVMEQ_POLLING 4
> __le32 *dbbuf_sq_db;
> __le32 *dbbuf_cq_db;
> __le32 *dbbuf_sq_ei;
> @@ -1606,13 +1657,13 @@ static inline void nvme_update_cq_head(struct nvme_queue *nvmeq)
> }
> }
>
> -static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,
> - struct io_comp_batch *iob)
> +static inline int __nvme_poll_cq(struct nvme_queue *nvmeq,
> + struct io_comp_batch *iob, int budget)
> {
> - bool found = false;
> + int found = 0;
>
> while (nvme_cqe_pending(nvmeq)) {
> - found = true;
> + found++;
> /*
> * load-load control dependency between phase and the rest of
> * the cqe requires a full read memory barrier
> @@ -1620,6 +1671,8 @@ static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,
> dma_rmb();
> nvme_handle_cqe(nvmeq, iob, nvmeq->cq_head);
> nvme_update_cq_head(nvmeq);
> + if (budget && found == budget)
> + break;
> }
>
> if (found)
> @@ -1627,26 +1680,253 @@ static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,
> return found;
> }
>
> +/* budget == 0 means reap the whole queue. */
> +static int nvme_poll_cq(struct nvme_queue *nvmeq, int budget)
> +{
> + DEFINE_IO_COMP_BATCH(iob);
> + int found = __nvme_poll_cq(nvmeq, &iob, budget);
> +
> + if (found && !rq_list_empty(&iob.req_list))
> + nvme_pci_complete_batch(&iob);
> + return found;
> +}
> +
> +static inline unsigned int nvmeq_outstanding(struct nvme_queue *nvmeq)
> +{
> + u16 sq_tail = READ_ONCE(nvmeq->last_sq_tail);
> + u16 cq_head = nvmeq->cq_head;
> +
> + if (sq_tail >= cq_head)
> + return sq_tail - cq_head;
> + return nvmeq->q_depth - cq_head + sq_tail;
> +}
> +
> +static inline bool nvmeq_cq_continue(struct nvme_queue *nvmeq)
> +{
> + return nvmeq_outstanding(nvmeq) > nvmeq->poll_thresh;
> +}
> +
> +static inline bool nvmeq_cq_idle(struct nvme_queue *nvmeq)
> +{
> + return nvmeq->cq_head == READ_ONCE(nvmeq->last_sq_tail);
> +}
> +
> +/*
> + * Interrupt-moderation profiles, ordered from least moderation (index 0: enter
> + * polling late, short sleep, brief linger -> lowest latency, most interrupts)
> + * to most moderation (enter early, long sleep, long linger -> fewest
> + * interrupts). The hill-climb walks this table to keep completions-per-
> + * interrupt high, which is what bounds the interrupt rate.
> + */
> +static const struct nvme_poll_prof {
> + u16 thresh;
> + u16 sleep_us;
> + u16 idle_rounds;
> + u16 budget; /* hard-IRQ initial poll cap; 0 == reap all */
> +} nvme_poll_prof[] = {
> + { 31, 0, 1, 0 }, /* drain inline: lowest latency, most interrupts */
> + { 15, 10, 2, 16 },
> + { 7, 20, 4, 8 }, /* start profile; matches irq_poll_* defaults */
> + { 3, 40, 8, 4 },
> + { 1, 80, 16, 2 }, /* reap a little, hand the bulk to the thread */
> +};
> +
> +#define NVME_DIM_START_PROFILE 2
> +
> +static void nvme_apply_profile(struct nvme_queue *nvmeq)
> +{
> + const struct nvme_poll_prof *p = &nvme_poll_prof[nvmeq->dim.profile_ix];
> +
> + nvmeq->poll_thresh = p->thresh;
> + nvmeq->poll_sleep_us = p->sleep_us;
> + nvmeq->poll_idle_rounds = p->idle_rounds;
> + nvmeq->poll_budget = p->budget;
> +}
> +
> +/* Mirror of rdma_dim_step() bounded to our profile table. */
> +static int nvme_dim_step(struct dim *dim)
> +{
> + if (dim->tune_state == DIM_GOING_RIGHT) {
> + if (dim->profile_ix == ARRAY_SIZE(nvme_poll_prof) - 1)
> + return DIM_ON_EDGE;
> + dim->profile_ix++;
> + dim->steps_right++;
> + }
> + if (dim->tune_state == DIM_GOING_LEFT) {
> + if (dim->profile_ix == 0)
> + return DIM_ON_EDGE;
> + dim->profile_ix--;
> + dim->steps_left++;
> + }
> + return DIM_STEPPED;
> +}
> +
> +/* Mirror of rdma_dim_stats_compare(): completion rate first, then batching. */
> +static int nvme_dim_stats_compare(struct dim_stats *curr, struct dim_stats *prev)
> +{
> + if (!prev->cpms)
> + return DIM_STATS_SAME;
> +
> + if (IS_SIGNIFICANT_DIFF(curr->cpms, prev->cpms))
> + return curr->cpms > prev->cpms ? DIM_STATS_BETTER :
> + DIM_STATS_WORSE;
> +
> + if (IS_SIGNIFICANT_DIFF(curr->cpe_ratio, prev->cpe_ratio))
> + return curr->cpe_ratio > prev->cpe_ratio ? DIM_STATS_BETTER :
> + DIM_STATS_WORSE;
> +
> + return DIM_STATS_SAME;
> +}
> +
> +/* Mirror of rdma_dim_decision(); returns true if the profile changed. */
> +static bool nvme_dim_decision(struct dim_stats *curr, struct dim *dim)
> +{
> + int prev_ix = dim->profile_ix;
> + int stats_res;
> +
> + stats_res = nvme_dim_stats_compare(curr, &dim->prev_stats);
> + switch (stats_res) {
> + case DIM_STATS_SAME:
> + if (curr->cpe_ratio <= 50 * prev_ix)
> + dim->profile_ix = 0;
> + break;
> + case DIM_STATS_WORSE:
> + dim_turn(dim);
> + fallthrough;
> + case DIM_STATS_BETTER:
> + if (nvme_dim_step(dim) == DIM_ON_EDGE)
> + dim_turn(dim);
> + break;
> + }
> +
> + dim->prev_stats = *curr;
> + return dim->profile_ix != prev_ix;
> +}
> +
> +/*
> + * Sample the completion/interrupt counters and, once per DIM_NEVENTS
> + * interrupts, compute the load stats. Called only from the interrupt owner
> + * (never concurrently with the poll thread), so the counters have a single
> + * writer. ktime_get() is taken only at a window boundary, not per interrupt.
> + *
> + * On each completed window the hill-climb picks a profile and, if it changed,
> + * applies it to nvmeq->poll_*.
> + */
> +static void nvme_dim(struct nvme_queue *nvmeq)
> +{
> + struct dim *dim = &nvmeq->dim;
> + struct dim_sample end;
> + struct dim_stats stats;
> +
> + if (dim->state == DIM_START_MEASURE) {
> + dim_update_sample_with_comps(nvmeq->dim_events, 0, 0,
> + nvmeq->dim_comps, &dim->start_sample);
> + dim->state = DIM_MEASURE_IN_PROGRESS;
> + return;
> + }
> +
> + /* Cheap gate: only recompute once a full window of events accrues. */
> + if ((u16)(nvmeq->dim_events - dim->start_sample.event_ctr) < DIM_NEVENTS)
> + return;
> +
> + dim_update_sample_with_comps(nvmeq->dim_events, 0, 0, nvmeq->dim_comps,
> + &end);
> + if (dim_calc_stats(&dim->start_sample, &end, &stats)) {
> + if (nvme_dim_decision(&stats, dim))
> + nvme_apply_profile(nvmeq);
> + trace_nvme_dim(nvmeq->qid, stats.cpms, stats.epms,
> + stats.cpe_ratio, dim->profile_ix);
> + }
> + dim->start_sample = end;
> +}
> +
> static irqreturn_t nvme_irq(int irq, void *data)
> {
> struct nvme_queue *nvmeq = data;
> - DEFINE_IO_COMP_BATCH(iob);
> + struct nvme_dev *dev = nvmeq->dev;
> + struct pci_dev *pdev = to_pci_dev(dev->dev);
> + irqreturn_t ret = IRQ_NONE;
> + unsigned int idle = 0;
>
> - if (nvme_poll_cq(nvmeq, &iob)) {
> - if (!rq_list_empty(&iob.req_list))
> - nvme_pci_complete_batch(&iob);
> - return IRQ_HANDLED;
> + for (;;) {
> + bool worked = false;
> + unsigned int i;
> +
> + for (i = 0; i <= irq_poll_spin; i++) {
> + int n = nvme_poll_cq(nvmeq, 0);
> +
> + if (n) {
> + nvmeq->dim_comps += n;
> + ret = IRQ_HANDLED;
> + worked = true;
> + idle = 0;
> + }
> +
> + if (nvmeq_cq_idle(nvmeq))
> + goto done;
> + else if (need_resched())
> + cond_resched();
> + else
> + cpu_relax();
> + }
> +
> + if (worked || ++idle < nvmeq->poll_idle_rounds) {
> + fsleep(nvmeq->poll_sleep_us);
> + continue;
> + }
> +done:
> + clear_bit(NVMEQ_POLLING, &nvmeq->flags);
> + if (!nvme_cqe_pending(nvmeq) ||
> + test_and_set_bit(NVMEQ_POLLING, &nvmeq->flags))
> + break;
> +
> + if (need_resched())
> + cond_resched();
> + else
> + cpu_relax();
> + idle = 0;
> }
> - return IRQ_NONE;
> +
> + if (pdev->msi_enabled)
> + writel(BIT(nvmeq->cq_vector), dev->bar + NVME_REG_INTMC);
> + return ret;
> }
>
> static irqreturn_t nvme_irq_check(int irq, void *data)
> {
> struct nvme_queue *nvmeq = data;
> + int found;
> +
> + nvmeq->dim_events++;
>
> - if (nvme_cqe_pending(nvmeq))
> + if (test_and_set_bit(NVMEQ_POLLING, &nvmeq->flags))
> + return IRQ_HANDLED;
> +
> + found = nvme_poll_cq(nvmeq, nvmeq->poll_budget);
> + nvmeq->dim_comps += found;
> + nvme_dim(nvmeq);
> +
> + if (!found) {
> + clear_bit(NVMEQ_POLLING, &nvmeq->flags);
> + return IRQ_NONE;
> + }
> +
> + /*
> + * Hand off to the poll thread when the queue is still deep, or when the
> + * budgeted initial poll left CQEs behind: their interrupt is already
> + * spent, so releasing here would strand them.
> + */
> + if (nvme_cqe_pending(nvmeq) || nvmeq_cq_continue(nvmeq)) {
> + struct nvme_dev *dev = nvmeq->dev;
> + struct pci_dev *pdev = to_pci_dev(dev->dev);
> +
> + if (pdev->msi_enabled)
> + writel(BIT(nvmeq->cq_vector), dev->bar + NVME_REG_INTMS);
> return IRQ_WAKE_THREAD;
> - return IRQ_NONE;
> + }
> +
> + clear_bit(NVMEQ_POLLING, &nvmeq->flags);
> + return IRQ_HANDLED;
> }
>
> /*
> @@ -1663,7 +1943,7 @@ static void nvme_poll_irqdisable(struct nvme_queue *nvmeq)
> irq = pci_irq_vector(pdev, nvmeq->cq_vector);
> disable_irq(irq);
> spin_lock(&nvmeq->cq_poll_lock);
> - nvme_poll_cq(nvmeq, NULL);
> + __nvme_poll_cq(nvmeq, NULL, 0);
> spin_unlock(&nvmeq->cq_poll_lock);
> enable_irq(irq);
> }
> @@ -1671,14 +1951,14 @@ static void nvme_poll_irqdisable(struct nvme_queue *nvmeq)
> static int nvme_poll(struct blk_mq_hw_ctx *hctx, struct io_comp_batch *iob)
> {
> struct nvme_queue *nvmeq = hctx->driver_data;
> - bool found;
> + int found;
>
> if (!test_bit(NVMEQ_POLLED, &nvmeq->flags) ||
> !nvme_cqe_pending(nvmeq))
> return 0;
>
> spin_lock(&nvmeq->cq_poll_lock);
> - found = nvme_poll_cq(nvmeq, iob);
> + found = __nvme_poll_cq(nvmeq, iob, 0);
> spin_unlock(&nvmeq->cq_poll_lock);
>
> return found;
> @@ -2075,7 +2355,7 @@ static void nvme_reap_pending_cqes(struct nvme_dev *dev)
>
> for (i = dev->ctrl.queue_count - 1; i > 0; i--) {
> spin_lock(&dev->queues[i].cq_poll_lock);
> - nvme_poll_cq(&dev->queues[i], NULL);
> + __nvme_poll_cq(&dev->queues[i], NULL, 0);
> spin_unlock(&dev->queues[i].cq_poll_lock);
> }
> }
> @@ -2171,13 +2451,8 @@ static int queue_request_irq(struct nvme_queue *nvmeq)
> struct pci_dev *pdev = to_pci_dev(nvmeq->dev->dev);
> int nr = nvmeq->dev->ctrl.instance;
>
> - if (use_threaded_interrupts) {
> - return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq_check,
> - nvme_irq, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
> - } else {
> - return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq,
> - NULL, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
> - }
> + return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq_check,
> + nvme_irq, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
> }
>
> static void nvme_init_queue(struct nvme_queue *nvmeq, u16 qid)
> @@ -2188,6 +2463,17 @@ static void nvme_init_queue(struct nvme_queue *nvmeq, u16 qid)
> nvmeq->last_sq_tail = 0;
> nvmeq->cq_head = 0;
> nvmeq->cq_phase = 1;
> + nvmeq->poll_thresh = irq_poll_thresh;
> + nvmeq->poll_sleep_us = irq_poll_sleep_us;
> + nvmeq->poll_idle_rounds = irq_poll_idle_rounds;
> + nvmeq->poll_budget = nvme_poll_prof[NVME_DIM_START_PROFILE].budget;
> + memset(&nvmeq->dim, 0, sizeof(nvmeq->dim));
> + nvmeq->dim.priv = nvmeq;
> + nvmeq->dim.profile_ix = NVME_DIM_START_PROFILE;
> + nvmeq->dim.tune_state = DIM_GOING_RIGHT;
> + nvmeq->dim_events = 0;
> + nvmeq->dim_comps = 0;
> + clear_bit(NVMEQ_POLLING, &nvmeq->flags);
> nvmeq->q_db = &dev->dbs[qid * 2 * dev->db_stride];
> memset((void *)nvmeq->cqes, 0, CQ_SIZE(nvmeq));
> nvme_dbbuf_init(dev, nvmeq, qid);
> diff --git a/drivers/nvme/host/trace.c b/drivers/nvme/host/trace.c
> index ad25ad1e40412..4bdd5abc60880 100644
> --- a/drivers/nvme/host/trace.c
> +++ b/drivers/nvme/host/trace.c
> @@ -498,3 +498,4 @@ const char *nvme_trace_disk_name(struct trace_seq *p, char *name)
> }
>
> EXPORT_TRACEPOINT_SYMBOL_GPL(nvme_sq);
> +EXPORT_TRACEPOINT_SYMBOL_GPL(nvme_dim);
> diff --git a/drivers/nvme/host/trace.h b/drivers/nvme/host/trace.h
> index 4fb5922ffdac5..b7099b3026079 100644
> --- a/drivers/nvme/host/trace.h
> +++ b/drivers/nvme/host/trace.h
> @@ -161,6 +161,29 @@ TRACE_EVENT(nvme_sq,
> )
> );
>
> +TRACE_EVENT(nvme_dim,
> + TP_PROTO(u16 qid, int cpms, int epms, int cpe_ratio, u8 profile),
> + TP_ARGS(qid, cpms, epms, cpe_ratio, profile),
> + TP_STRUCT__entry(
> + __field(u16, qid)
> + __field(int, cpms)
> + __field(int, epms)
> + __field(int, cpe_ratio)
> + __field(u8, profile)
> + ),
> + TP_fast_assign(
> + __entry->qid = qid;
> + __entry->cpms = cpms;
> + __entry->epms = epms;
> + __entry->cpe_ratio = cpe_ratio;
> + __entry->profile = profile;
> + ),
> + TP_printk("qid=%u cpms=%d epms=%d cpe_ratio=%d profile=%u",
> + __entry->qid, __entry->cpms, __entry->epms,
> + __entry->cpe_ratio, __entry->profile
> + )
> +);
> +
> #endif /* _TRACE_NVME_H */
>
> #undef TRACE_INCLUDE_PATH
> --
>
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2026-08-11 2:32 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-08-06 3:10 [RESEND RFC PATCH v2] nvme-pci: add adaptive interrupt polling Fengnan Chang
2026-08-10 20:55 ` Keith Busch
2026-08-11 2:32 ` changfengnan
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.