[PATCH v1 0/1] nvme-pci: adaptive interrupt coalescing

[PATCH v1 0/1] nvme-pci: adaptive interrupt coalescing

[Jun Zeng]

Background
==========

Nowadays, with the high performance PCIe 5.0 NVMe drives been used
more increasingly and widely, it's important to maintain and fully
realize NVMe drive's performance at the system level in
scenarios where multiple NVMe drives are used simultaneously.

Problem
=======

With PCIe 5.0 NVMe drives, the interrupt count can reach very high
levels in burst when running IOPS workloads (e.g., 4k/8k randread)
with multiple queues across many NVMe drives. These high interrupts
generated instantaneously may put pressure on current CPU handling
mechanism.

Specifically, IOPS with small IO (e.g., 4k/8k randread) does not
meet expectations when running with multiple queues among many NVMe
drives (e.g., 6-8 NVMe drives per socket), even with each queue and
job assigned to separate CPU cores.

The IOPS for small IO can be significantly improved when enabling
interrupt coalescing in this case. However, in some cases (e.g., big
IO for bandwidth or latency-sensitive tests with small IO and small
queue depth), interrupt coalescing has side effects.

Solution
========

Monitor the IOPS and average queue depth of inflight IO periodically
on each NVMe drive and enable/disable interrupt coalescing
automatically without user intervention. This feature is enabled by
default and can be disabled/enabled through debugfs during runtime.

Test Results
============

1. IOPS improved from < 20M to 55M for 4k randread with fio job=8,
   qd=128 across 16 PCIe Gen5 NVMe drives on Intel Xeon GNR CPUs.
   - Latency not affected for 4k randread with job=1/2/4/8,
     qd=1/2/4/8/16/32
   - Bandwidth not affected for 32k/64k/128k read/write with
     job=1/2/4/8, qd=128

2. Tested with several mainstream PCIe Gen5 SSD vendors and Intel
   CPUs (including AMD) with improved 4K random read IOPS.

Notes
=====

1. Interrupt coalescing is a per-drive feature, not per-queue
   feature. Each queue has a separate interrupt, but coalescing is
   controlled globally per drive. We introduce thresholds for
   IO_DELTA, AVG_QUEUE_DEPTH, and TOTAL_PENDING_IO to judge
   workload status.

2. Tunable parameters exposed via debugfs at
   /sys/kernel/debug/nvme_pci/:
   - disable_adaptive_interrupt_coalescing: 0=enabled, 1=disabled
   - high_io_delta: IO count threshold per cycle (default: 45000)
   - avg_qd_threshold: average queue depth threshold (default: 32)
   - total_inflight_io_threshold: total inflight IO threshold
     (default: 256)

3. The default threshold values are derived from empirical testing
   on various PCIe Gen5 NVMe drives.

Jun Zeng (1):
  nvme-pci: adaptive interrupt coalescing

 drivers/nvme/host/pci.c | 540 +++++++++++++++++++++++++++++++++++++++-
 1 file changed, 539 insertions(+), 1 deletion(-)


base-commit: 5d6919055dec134de3c40167a490f33c74c12581
-- 
2.43.0

[PATCH v1 1/1] nvme-pci: adaptive interrupt coalescing

[Jun Zeng]

Add adaptive interrupt coalescing to improve IOPS for high-throughput
workloads. Monitor IO pressure periodically and enable/disable coalescing
automatically based on IOPS, queue depth, and inflight IO thresholds.
This feature is enabled by default, can be controlled through debugfs
variable during runtime.

This significantly improves 4k random read IOPS on multi-disk Gen5 NVMe
configurations without affecting latency-sensitive workloads (i.e.,
double the total IOPS with 16 NVMe disks).

Co-developed-by: Liang Fang <liang.a.fang@intel.com>
Signed-off-by: Liang Fang <liang.a.fang@intel.com>
Signed-off-by: Jun Zeng <jun1.zeng@intel.com>
---
 drivers/nvme/host/pci.c | 540 +++++++++++++++++++++++++++++++++++++++-
 1 file changed, 539 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 9fd04cd7c5cb..68e010af1e0d 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -9,6 +9,7 @@
 #include <linux/blkdev.h>
 #include <linux/blk-mq-dma.h>
 #include <linux/blk-integrity.h>
+#include <linux/debugfs.h>
 #include <linux/dmi.h>
 #include <linux/init.h>
 #include <linux/interrupt.h>
@@ -19,6 +20,7 @@
 #include <linux/module.h>
 #include <linux/mutex.h>
 #include <linux/nodemask.h>
+#include <linux/notifier.h>
 #include <linux/once.h>
 #include <linux/pci.h>
 #include <linux/suspend.h>
@@ -334,6 +336,20 @@ struct nvme_dev {
 	unsigned int nr_allocated_queues;
 	unsigned int nr_write_queues;
 	unsigned int nr_poll_queues;
+
+	/* adaptive interrupt coalescing support: */
+	bool in_coalescing;
+	bool need_reset_coalesce;
+	bool irq_coalesce_supported;
+	bool coalesce_work_stop;
+	u32 coalesce_set_streak;
+	u32 coalesce_clear_streak;
+	unsigned long *active_q_bitmap;
+	u64 prev_coal_check_time;
+	struct delayed_work coalesce_work;
+	struct notifier_block coalesce_notifier;
+	bool coalesce_notifier_registered;
+
 	struct nvme_descriptor_pools descriptor_pools[];
 };
 
@@ -391,6 +407,10 @@ struct nvme_queue {
 	__le32 *dbbuf_sq_ei;
 	__le32 *dbbuf_cq_ei;
 	struct completion delete_done;
+	/* to count complete io number of this queue */
+	atomic_long_t io_cnt;
+	/* Previous io_cnt for delta calculation */
+	unsigned long prev_io_cnt;
 };
 
 /* bits for iod->flags */
@@ -1402,6 +1422,397 @@ static blk_status_t nvme_prep_rq(struct request *req)
 	return ret;
 }
 
+/* definition and logic for adaptive interrupt coalescing */
+#define NVME_ADAPTIVE_COALESCING_ENABLE		0x0001
+#define NVME_ADAPTIVE_COALESCING_DISABLE	0x0002
+#define COAL_CHECK_INTERVAL_MS			100
+#define COAL_CHECK_IO_DELTA_THRESH		45000 /* 45k io num, in 100 ms */
+#define COAL_CHECK_STREAK_THRESH		5
+#define COAL_CHECK_AVG_INFLT_IO_THRESH		32
+#define COAL_CHECK_INFLT_IO_THRESH		256
+#define COAL_ACTIVE_Q_SAMPLE_INTERVAL		512
+#define COALESCE_SET_VALUE			0x10a
+#define COALESCE_CLEAR_VALUE			0x000
+
+static BLOCKING_NOTIFIER_HEAD(nvme_adaptive_coalesce_notifier_list);
+
+/* enable adaptive interrupt coalescing by default */
+static int adaptive_coalescing_disabled;
+static u32 high_io_delta = COAL_CHECK_IO_DELTA_THRESH;
+static u32 avg_qd_threshold = COAL_CHECK_AVG_INFLT_IO_THRESH;
+static u32 total_inflight_io_threshold = COAL_CHECK_INFLT_IO_THRESH;
+
+static int nvme_set_irq_coalesce(struct nvme_ctrl *ctrl, u32 v)
+{
+	return nvme_set_features(ctrl, NVME_FEAT_IRQ_COALESCE, v, NULL, 0, NULL);
+}
+
+static int nvme_get_irq_coalesce(struct nvme_ctrl *ctrl, u32 *v)
+{
+	return nvme_get_features(ctrl, NVME_FEAT_IRQ_COALESCE, 0, NULL, 0, v);
+}
+
+static bool nvme_check_irq_coalesce_supported(struct nvme_ctrl *ctrl)
+{
+	u32 value;
+
+	if (nvme_get_irq_coalesce(ctrl, &value)) {
+		dev_info(ctrl->device, "interrupt coalescing not supported\n");
+		return false;
+	}
+	return true;
+}
+
+static int disable_adaptive_coalescing_show(void *data, u64 *val)
+{
+	*val = adaptive_coalescing_disabled;
+	return 0;
+}
+
+static int disable_adaptive_coalescing_store(void *data, u64 val)
+{
+	int old_val, new_val;
+
+	if (val != 0 && val != 1)
+		return -EINVAL;
+
+	new_val = (int)val;
+	old_val = xchg(&adaptive_coalescing_disabled, new_val);
+
+	/* If old_val == new_val, no change needed */
+	if (old_val == new_val)
+		return 0;
+
+	if (new_val)
+		blocking_notifier_call_chain(&nvme_adaptive_coalesce_notifier_list,
+					     NVME_ADAPTIVE_COALESCING_DISABLE, NULL);
+	else
+		blocking_notifier_call_chain(&nvme_adaptive_coalesce_notifier_list,
+					     NVME_ADAPTIVE_COALESCING_ENABLE, NULL);
+
+	return 0;
+}
+DEFINE_DEBUGFS_ATTRIBUTE(disable_adaptive_coalescing_fops,
+			 disable_adaptive_coalescing_show,
+			 disable_adaptive_coalescing_store,
+			 "%llu\n");
+
+static int nvme_coalesce_work_cpu(struct nvme_dev *dev)
+{
+	int node = dev_to_node(dev->dev);
+	int cpu;
+
+	if (node != NUMA_NO_NODE && node_online(node)) {
+		cpu = cpumask_last(cpumask_of_node(node));
+		if (cpu < nr_cpu_ids && cpu_online(cpu))
+			return cpu;
+	}
+
+	cpu = cpumask_last(cpu_online_mask);
+	if (cpu < nr_cpu_ids && cpu_online(cpu))
+		return cpu;
+
+	return WORK_CPU_UNBOUND;
+}
+
+/*
+ * Check if coalescing should be enabled based on IO pressure.
+ * Returns true if high IO pressure is detected.
+ */
+static bool nvme_should_enable_coalesce(u64 io_delta, u64 time_interval,
+					u32 avg_inflight_io, u32 busy_queues)
+{
+	/* Check if IOPS exceeds threshold (adjusted for actual time interval) */
+	if (io_delta * COAL_CHECK_INTERVAL_MS * NSEC_PER_MSEC <=
+	    high_io_delta * time_interval)
+		return false;
+
+	/*
+	 * Enable coalescing when average number of inflight IOs on active queues
+	 * is larger enough to indicate a throughput-oriented workload.
+	 */
+	if (avg_inflight_io > avg_qd_threshold)
+		return true;
+
+	/*
+	 * Even when each active queue is relatively shallow, still enable
+	 * coalescing if the aggregate inflight IO across busy queues is high.
+	 */
+	return busy_queues * avg_inflight_io > total_inflight_io_threshold;
+}
+
+/* Update coalesce streak counters based on current IO pressure */
+static void nvme_update_coalesce_streak(struct nvme_dev *dev, bool should_enable)
+{
+	if (should_enable) {
+		dev->coalesce_set_streak++;
+		dev->coalesce_clear_streak = 0;
+	} else {
+		dev->coalesce_clear_streak++;
+		dev->coalesce_set_streak = 0;
+	}
+}
+
+/* Apply coalesce state change if streak threshold is reached */
+static void nvme_apply_coalesce_change(struct nvme_dev *dev)
+{
+	if (!dev->in_coalescing) {
+		if (dev->coalesce_set_streak <= COAL_CHECK_STREAK_THRESH)
+			return;
+
+		if (nvme_set_irq_coalesce(&dev->ctrl, COALESCE_SET_VALUE)) {
+			dev_warn_ratelimited(dev->ctrl.device,
+					     "set irq coalesce with value:%d failed.\n",
+					     COALESCE_SET_VALUE);
+			return;
+		}
+
+		dev->in_coalescing = true;
+		dev->coalesce_set_streak = 0;
+		dev->coalesce_clear_streak = 0;
+		return;
+	}
+
+	if (dev->coalesce_clear_streak <= COAL_CHECK_STREAK_THRESH)
+		return;
+
+	if (nvme_set_irq_coalesce(&dev->ctrl, COALESCE_CLEAR_VALUE)) {
+		dev_warn_ratelimited(dev->ctrl.device,
+				     "clear irq coalesce with value:%d failed.\n",
+				     COALESCE_CLEAR_VALUE);
+		return;
+	}
+
+	dev->in_coalescing = false;
+	dev->coalesce_set_streak = 0;
+	dev->coalesce_clear_streak = 0;
+}
+
+/* Collect IO statistics from all queues */
+static void nvme_collect_io_stats(struct nvme_dev *dev, u64 *io_delta,
+				  u32 *avg_inflight_io, u32 *busy_queues)
+{
+	struct nvme_queue *nvmeq;
+	u64 total_inflight_io = 0;
+	unsigned int i;
+	u32 queues_counted = 0;
+
+	*io_delta = 0;
+
+	/*
+	 * Count active queues during iteration rather than using bitmap_weight()
+	 * upfront. The bitmap can be modified concurrently by nvme_mark_queue_active()
+	 * from other CPUs, so we count the actual queues processed to ensure the
+	 * divisor matches the accumulated total_inflight_io.
+	 */
+	for_each_set_bit(i, dev->active_q_bitmap, dev->nr_allocated_queues) {
+		u32 sq_tail;
+		u32 cq_head;
+		u32 q_depth;
+
+		nvmeq = &dev->queues[i];
+		/*
+		 * Read cq_head before sq_tail in source order for a conservative
+		 * estimate. If we read sq_tail first and completions happen between
+		 * reads, cq_head could advance past the observed sq_tail, causing
+		 * the inflight calculation to produce an artificially high value.
+		 *
+		 * Note: READ_ONCE() only prevents compiler reordering; weakly ordered
+		 * architectures may still reorder at hardware level. This is acceptable
+		 * for approximate statistics - occasional inflated values don't affect
+		 * adaptive coalescing decisions significantly.
+		 */
+		cq_head = READ_ONCE(nvmeq->cq_head);
+		sq_tail = READ_ONCE(nvmeq->sq_tail);
+		q_depth = READ_ONCE(nvmeq->q_depth);
+		/* Skip uninitialized queue to avoid divide-by-zero */
+		if (!q_depth)
+			continue;
+		/* Use unsigned arithmetic to handle ring buffer wrap correctly */
+		total_inflight_io += (sq_tail + q_depth - cq_head) % q_depth;
+		queues_counted++;
+	}
+
+	*busy_queues = queues_counted;
+	if (queues_counted)
+		*avg_inflight_io = div_u64(total_inflight_io, queues_counted);
+	else
+		*avg_inflight_io = 0;
+
+	/*
+	 * Calculate io_delta and advance each queue's baseline every cycle,
+	 * including cycles where no queue crossed an active sampling boundary.
+	 * This keeps the completion delta aligned with the measurement interval.
+	 *
+	 * We must calculate delta in native word size first (unsigned long)
+	 * before accumulating to 64-bit. On 32-bit platforms, if a queue's
+	 * 32-bit counter wraps from 0xFFFFFFFF to 0, calculating delta in
+	 * 32-bit unsigned arithmetic handles wrap correctly (e.g., 5 - 0xFFFFFFF0
+	 * = 21). Accumulating absolute 64-bit values would cause huge errors.
+	 */
+	for (i = dev->nr_allocated_queues - 1; i > 0; i--) {
+		unsigned long curr_cnt, queue_delta;
+
+		nvmeq = &dev->queues[i];
+		curr_cnt = atomic_long_read(&nvmeq->io_cnt);
+		/* Unsigned subtraction handles wrap correctly */
+		queue_delta = curr_cnt - nvmeq->prev_io_cnt;
+		nvmeq->prev_io_cnt = curr_cnt;
+		*io_delta += queue_delta;
+	}
+
+	/*
+	 * Use atomic clear_bit to avoid mixing atomic and non-atomic ops
+	 * on the same memory, which could lose updates and trigger KCSAN.
+	 */
+	for (i = 0; i < dev->nr_allocated_queues; i++)
+		clear_bit(i, dev->active_q_bitmap);
+}
+
+static void nvme_adjust_coalesce(struct nvme_dev *dev)
+{
+	u64 now = ktime_get_ns();
+	u64 time_interval;
+	u64 io_delta;
+	u32 avg_inflight_io;
+	u32 busy_queues;
+	bool should_enable;
+
+	if (READ_ONCE(adaptive_coalescing_disabled)) {
+		if (nvme_ctrl_state(&dev->ctrl) != NVME_CTRL_LIVE)
+			return;
+
+		/* Force clear coalescing when disabled */
+		dev->coalesce_set_streak = 0;
+		dev->coalesce_clear_streak += COAL_CHECK_STREAK_THRESH + 1;
+		dev_info(dev->ctrl.device, "clearing coalesce due to disable\n");
+		nvme_apply_coalesce_change(dev);
+		return;
+	}
+
+	/* If ctrlr state no LIVE, then skip this round */
+	if (nvme_ctrl_state(&dev->ctrl) != NVME_CTRL_LIVE)
+		return;
+
+	/*
+	 * Pairs with smp_wmb() in nvme_reset_work() to ensure we see
+	 * need_reset_coalesce=true if we see state=LIVE after reset.
+	 */
+	smp_rmb();
+
+	/* Check if controller been reset - coalescing config may be lost */
+	if (dev->need_reset_coalesce) {
+		unsigned int j;
+
+		/*
+		 * Use atomic clear_bit to avoid data race with concurrent
+		 * set_bit() from nvme_mark_queue_active() on other CPUs.
+		 */
+		for (j = 0; j < dev->nr_allocated_queues; j++)
+			clear_bit(j, dev->active_q_bitmap);
+
+		/*
+		 * Sync each queue's prev_io_cnt to current io_cnt, because queues
+		 * may not be reallocated and io_cnt retains lifetime values.
+		 * Otherwise io_delta would be erroneously huge on next cycle.
+		 */
+		for (j = dev->nr_allocated_queues - 1; j > 0; j--)
+			dev->queues[j].prev_io_cnt = atomic_long_read(&dev->queues[j].io_cnt);
+
+		/*
+		 * Clear hardware coalescing state to ensure software and hardware
+		 * states are synchronized after controller reset.
+		 */
+		if (dev->in_coalescing)
+			nvme_set_irq_coalesce(&dev->ctrl, COALESCE_CLEAR_VALUE);
+
+		dev->in_coalescing = false;
+		dev->coalesce_set_streak = 0;
+		dev->coalesce_clear_streak = 0;
+		dev->need_reset_coalesce = false;
+		dev->prev_coal_check_time = now;
+		return;
+	}
+
+	nvme_collect_io_stats(dev, &io_delta, &avg_inflight_io, &busy_queues);
+	time_interval = now - dev->prev_coal_check_time;
+	dev->prev_coal_check_time = now;
+
+	should_enable = nvme_should_enable_coalesce(io_delta, time_interval,
+						    avg_inflight_io, busy_queues);
+	nvme_update_coalesce_streak(dev, should_enable);
+	nvme_apply_coalesce_change(dev);
+}
+
+static void nvme_adjust_coalesce_work(struct work_struct *work)
+{
+	struct nvme_dev *dev = container_of(work, struct nvme_dev, coalesce_work.work);
+	/*
+	 * Schedule the delayed work on the last CPU of the NUMA node the disk
+	 * belongs to.
+	 */
+	int cpu = nvme_coalesce_work_cpu(dev);
+
+	/* Check stop flag before doing any work or rescheduling */
+	if (READ_ONCE(dev->coalesce_work_stop))
+		return;
+
+	if (READ_ONCE(adaptive_coalescing_disabled) && !dev->in_coalescing)
+		return;
+
+	nvme_adjust_coalesce(dev);
+
+	/* Check again before rescheduling to avoid use-after-free */
+	if (!READ_ONCE(dev->coalesce_work_stop))
+		schedule_delayed_work_on(cpu, &dev->coalesce_work,
+					 msecs_to_jiffies(COAL_CHECK_INTERVAL_MS));
+}
+
+static void nvme_mark_queue_active(struct nvme_queue *nvmeq, u16 sq_tail)
+{
+	if (READ_ONCE(adaptive_coalescing_disabled))
+		return;
+
+	/* ignore admin op */
+	if (nvmeq->qid == 0)
+		return;
+
+	/*
+	 * Sample at COAL_ACTIVE_Q_SAMPLE_INTERVAL boundaries and whenever the
+	 * submission queue wraps. This ensures queues shallower than the sample
+	 * interval can still be recognized as active.
+	 */
+	if ((sq_tail & (COAL_ACTIVE_Q_SAMPLE_INTERVAL - 1)) == 0)
+		set_bit(nvmeq->qid, nvmeq->dev->active_q_bitmap);
+}
+
+static int nvme_pci_coalescing_notify(struct notifier_block *nb,
+				      unsigned long action, void *data)
+{
+	struct nvme_dev *dev =
+		container_of(nb, struct nvme_dev, coalesce_notifier);
+	/*
+	 * Schedule the delayed work on the last CPU of the NUMA node the disk
+	 * belongs to.
+	 */
+	int cpu = nvme_coalesce_work_cpu(dev);
+
+	switch (action) {
+	case NVME_ADAPTIVE_COALESCING_ENABLE:
+		/* Use cached value to avoid blocking admin command in notifier */
+		if (dev->irq_coalesce_supported)
+			schedule_delayed_work_on(cpu, &dev->coalesce_work,
+						 msecs_to_jiffies(COAL_CHECK_INTERVAL_MS));
+		break;
+	case NVME_ADAPTIVE_COALESCING_DISABLE:
+		/* no need to do things now, the coalesce_work will exit automatically */
+		break;
+	default:
+		break;
+	}
+	return NOTIFY_OK;
+}
+
 static blk_status_t nvme_queue_rq(struct blk_mq_hw_ctx *hctx,
 			 const struct blk_mq_queue_data *bd)
 {
@@ -1424,8 +1835,10 @@ static blk_status_t nvme_queue_rq(struct blk_mq_hw_ctx *hctx,
 	ret = nvme_prep_rq(req);
 	if (unlikely(ret))
 		return ret;
+
 	spin_lock(&nvmeq->sq_lock);
 	nvme_sq_copy_cmd(nvmeq, &iod->cmd);
+	nvme_mark_queue_active(nvmeq, nvmeq->sq_tail);
 	nvme_write_sq_db(nvmeq, bd->last);
 	spin_unlock(&nvmeq->sq_lock);
 	return BLK_STS_OK;
@@ -1443,6 +1856,7 @@ static void nvme_submit_cmds(struct nvme_queue *nvmeq, struct rq_list *rqlist)
 		struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
 
 		nvme_sq_copy_cmd(nvmeq, &iod->cmd);
+		nvme_mark_queue_active(nvmeq, nvmeq->sq_tail);
 	}
 	nvme_write_sq_db(nvmeq, true);
 	spin_unlock(&nvmeq->sq_lock);
@@ -1555,6 +1969,8 @@ static inline void nvme_handle_cqe(struct nvme_queue *nvmeq,
 		return;
 	}
 
+	atomic_long_inc(&nvmeq->io_cnt);
+
 	trace_nvme_sq(req, cqe->sq_head, nvmeq->sq_tail);
 	if (!nvme_try_complete_req(req, cqe->status, cqe->result) &&
 	    !blk_mq_add_to_batch(req, iob,
@@ -2124,6 +2540,8 @@ static int nvme_alloc_queue(struct nvme_dev *dev, int qid, int depth)
 	nvmeq->cq_phase = 1;
 	nvmeq->q_db = &dev->dbs[qid * 2 * dev->db_stride];
 	nvmeq->qid = qid;
+	atomic_long_set(&nvmeq->io_cnt, 0);
+	nvmeq->prev_io_cnt = 0;
 	dev->ctrl.queue_count++;
 
 	return 0;
@@ -3342,8 +3760,18 @@ static void nvme_pci_free_ctrl(struct nvme_ctrl *ctrl)
 {
 	struct nvme_dev *dev = to_nvme_dev(ctrl);
 
+	/* Signal work to stop before canceling to prevent rearming */
+	WRITE_ONCE(dev->coalesce_work_stop, true);
+	if (dev->coalesce_notifier_registered) {
+		blocking_notifier_chain_unregister(
+			&nvme_adaptive_coalesce_notifier_list,
+			&dev->coalesce_notifier);
+		dev->coalesce_notifier_registered = false;
+	}
+	cancel_delayed_work_sync(&dev->coalesce_work);
 	nvme_free_tagset(dev);
 	put_device(dev->dev);
+	bitmap_free(dev->active_q_bitmap);
 	kfree(dev->queues);
 	kfree(dev);
 }
@@ -3429,6 +3857,18 @@ static void nvme_reset_work(struct work_struct *work)
 		nvme_free_tagset(dev);
 	}
 
+	/*
+	 * Set need_reset_coalesce before changing to LIVE state to avoid
+	 * race with nvme_adjust_coalesce_work() which could run with stale
+	 * bitmap and prev_io_cnt values causing erroneous IOPS calculation.
+	 */
+	dev->need_reset_coalesce = true;
+	/*
+	 * Ensure need_reset_coalesce is visible before the LIVE state change.
+	 * Pairs with smp_rmb() in nvme_adjust_coalesce().
+	 */
+	smp_wmb();
+
 	/*
 	 * If only admin queue live, keep it to do further investigation or
 	 * recovery.
@@ -3436,6 +3876,7 @@ static void nvme_reset_work(struct work_struct *work)
 	if (!nvme_change_ctrl_state(&dev->ctrl, NVME_CTRL_LIVE)) {
 		dev_warn(dev->ctrl.device,
 			"failed to mark controller live state\n");
+		dev->need_reset_coalesce = false;
 		result = -ENODEV;
 		goto out;
 	}
@@ -3650,8 +4091,18 @@ static struct nvme_dev *nvme_pci_alloc_dev(struct pci_dev *pdev,
 	if (!dev)
 		return ERR_PTR(-ENOMEM);
 	INIT_WORK(&dev->ctrl.reset_work, nvme_reset_work);
+	INIT_DELAYED_WORK(&dev->coalesce_work, nvme_adjust_coalesce_work);
+	dev->coalesce_notifier.notifier_call = nvme_pci_coalescing_notify;
+	dev->coalesce_notifier_registered = false;
 	mutex_init(&dev->shutdown_lock);
 
+	dev->in_coalescing = false;
+	dev->need_reset_coalesce = false;
+	dev->coalesce_work_stop = false;
+	dev->coalesce_set_streak = 0;
+	dev->coalesce_clear_streak = 0;
+	dev->prev_coal_check_time = ktime_get_ns();
+
 	dev->nr_write_queues = write_queues;
 	dev->nr_poll_queues = poll_queues;
 	dev->nr_allocated_queues = nvme_max_io_queues(dev) + 1;
@@ -3660,6 +4111,12 @@ static struct nvme_dev *nvme_pci_alloc_dev(struct pci_dev *pdev,
 	if (!dev->queues)
 		goto out_free_dev;
 
+	dev->active_q_bitmap = bitmap_zalloc_node(dev->nr_allocated_queues, GFP_KERNEL, node);
+	if (!dev->active_q_bitmap) {
+		ret = -ENOMEM;
+		goto out_free_queues;
+	}
+
 	dev->dev = get_device(&pdev->dev);
 
 	quirks |= check_vendor_combination_bug(pdev);
@@ -3679,6 +4136,7 @@ static struct nvme_dev *nvme_pci_alloc_dev(struct pci_dev *pdev,
 		quirks |= qentry->enabled_quirks;
 		quirks &= ~qentry->disabled_quirks;
 	}
+
 	ret = nvme_init_ctrl(&dev->ctrl, &pdev->dev, &nvme_pci_ctrl_ops,
 			     quirks);
 	if (ret)
@@ -3704,6 +4162,8 @@ static struct nvme_dev *nvme_pci_alloc_dev(struct pci_dev *pdev,
 
 out_put_device:
 	put_device(dev->dev);
+	kfree(dev->active_q_bitmap);
+out_free_queues:
 	kfree(dev->queues);
 out_free_dev:
 	kfree(dev);
@@ -3790,14 +4250,39 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 		goto out_disable;
 	}
 
+	result = blocking_notifier_chain_register(
+			&nvme_adaptive_coalesce_notifier_list,
+			&dev->coalesce_notifier);
+	if (result)
+		goto out_disable;
+	dev->coalesce_notifier_registered = true;
+
 	pci_set_drvdata(pdev, dev);
 
 	nvme_start_ctrl(&dev->ctrl);
 	nvme_put_ctrl(&dev->ctrl);
 	flush_work(&dev->ctrl.scan_work);
+	/* Cache irq coalesce support to avoid admin cmd in notifier callback */
+	dev->irq_coalesce_supported = nvme_check_irq_coalesce_supported(&dev->ctrl);
+	if (dev->irq_coalesce_supported && !adaptive_coalescing_disabled) {
+		/*
+		 * Schedule the delayed work on the last CPU of the
+		 * NUMA node this NVMe disk belongs to.
+		 */
+		schedule_delayed_work_on(nvme_coalesce_work_cpu(dev),
+					 &dev->coalesce_work,
+					 msecs_to_jiffies(COAL_CHECK_INTERVAL_MS));
+	}
 	return 0;
 
 out_disable:
+	if (dev->coalesce_notifier_registered) {
+		blocking_notifier_chain_unregister(
+			&nvme_adaptive_coalesce_notifier_list,
+			&dev->coalesce_notifier);
+		dev->coalesce_notifier_registered = false;
+	}
+	cancel_delayed_work_sync(&dev->coalesce_work);
 	nvme_change_ctrl_state(&dev->ctrl, NVME_CTRL_DELETING);
 	nvme_dev_disable(dev, true);
 	nvme_free_host_mem(dev);
@@ -3853,14 +4338,42 @@ static void nvme_remove(struct pci_dev *pdev)
 {
 	struct nvme_dev *dev = pci_get_drvdata(pdev);
 
+	/*
+	 * Set DELETING state first to fail pending admin commands quickly,
+	 * preventing 60-second timeout stalls during abrupt removal.
+	 */
 	nvme_change_ctrl_state(&dev->ctrl, NVME_CTRL_DELETING);
+
+	/*
+	 * Signal work to stop and unregister notifier early to prevent
+	 * use-after-free: work could rearm itself, or debugfs write could
+	 * trigger notifier callback which submits admin commands after
+	 * admin queue is destroyed.
+	 */
+	WRITE_ONCE(dev->coalesce_work_stop, true);
+	if (dev->coalesce_notifier_registered) {
+		blocking_notifier_chain_unregister(
+			&nvme_adaptive_coalesce_notifier_list,
+			&dev->coalesce_notifier);
+		dev->coalesce_notifier_registered = false;
+	}
 	pci_set_drvdata(pdev, NULL);
 
+	/*
+	 * Disable device BEFORE cancel_delayed_work_sync() to abort any
+	 * pending admin commands. This prevents 60-second timeout stalls
+	 * during surprise removal: if coalesce_work is blocked waiting
+	 * for nvme_set_irq_coalesce() admin command on unresponsive
+	 * hardware, nvme_dev_disable() will abort it immediately.
+	 */
 	if (!pci_device_is_present(pdev)) {
 		nvme_change_ctrl_state(&dev->ctrl, NVME_CTRL_DEAD);
 		nvme_dev_disable(dev, true);
 	}
 
+	/* Now safe to sync - any pending admin commands are aborted */
+	cancel_delayed_work_sync(&dev->coalesce_work);
+
 	flush_work(&dev->ctrl.reset_work);
 	nvme_stop_ctrl(&dev->ctrl);
 	nvme_remove_namespaces(&dev->ctrl);
@@ -4273,14 +4786,37 @@ static struct pci_driver nvme_driver = {
 	.err_handler	= &nvme_err_handler,
 };
 
+static struct dentry *nvme_dbg_root;
+
 static int __init nvme_init(void)
 {
+	int ret;
+
+	/*
+	 * debugfs is not essential for driver functionality.
+	 * Don't check return values - debugfs API handles errors gracefully.
+	 */
+	nvme_dbg_root = debugfs_create_dir("nvme_pci", NULL);
+	debugfs_create_file("disable_adaptive_interrupt_coalescing",
+			    0644, nvme_dbg_root, NULL,
+			    &disable_adaptive_coalescing_fops);
+	debugfs_create_u32("high_io_delta", 0644, nvme_dbg_root,
+			   &high_io_delta);
+	debugfs_create_u32("avg_qd_threshold", 0644, nvme_dbg_root,
+			   &avg_qd_threshold);
+	debugfs_create_u32("total_inflight_io_threshold", 0644,
+			   nvme_dbg_root, &total_inflight_io_threshold);
+
 	BUILD_BUG_ON(sizeof(struct nvme_create_cq) != 64);
 	BUILD_BUG_ON(sizeof(struct nvme_create_sq) != 64);
 	BUILD_BUG_ON(sizeof(struct nvme_delete_queue) != 64);
 	BUILD_BUG_ON(IRQ_AFFINITY_MAX_SETS < 2);
 
-	return pci_register_driver(&nvme_driver);
+	ret = pci_register_driver(&nvme_driver);
+	if (ret)
+		debugfs_remove_recursive(nvme_dbg_root);
+
+	return ret;
 }
 
 static void __exit nvme_exit(void)
@@ -4288,6 +4824,8 @@ static void __exit nvme_exit(void)
 	kfree(nvme_pci_quirk_list);
 	pci_unregister_driver(&nvme_driver);
 	flush_workqueue(nvme_wq);
+
+	debugfs_remove_recursive(nvme_dbg_root);
 }
 
 MODULE_AUTHOR("Matthew Wilcox <willy@linux.intel.com>");
-- 
2.43.0

Re: [PATCH v1 1/1] nvme-pci: adaptive interrupt coalescing

[Keith Busch]

On Wed, Jul 15, 2026 at 03:57:03PM +0800, Jun Zeng wrote:
> Add adaptive interrupt coalescing to improve IOPS for high-throughput
> workloads. Monitor IO pressure periodically and enable/disable coalescing
> automatically based on IOPS, queue depth, and inflight IO thresholds.
> This feature is enabled by default, can be controlled through debugfs
> variable during runtime.

Can't you do all this from user space? You'd maybe need the blk-mq
debugfs to get access to the individual hctx dispatch numbers, but maybe
that's okay?

The atomic_long_inc where you placed it is especially harmful to polled
queues.

RE: [PATCH v1 1/1] nvme-pci: adaptive interrupt coalescing

[Zeng, Jun1]

Hi Busch, 
        Thanks for your reply. The main reason for implementing this in kernel space is the lack of a unified, deployable component in user space to support this feature. You see that there are sheer number of OS distributions. This will require separate adaptations for each, usability might be compromised, and-most importantly-ongoing maintenance and updates would be difficult. Meanwhile, nvme driver in kernel space is providing these unified capabilities for different kinds of usage.
        For the point of using atomic_long_inc in cqe polling we can discuss to do it in a proper way.

-----Original Message-----
From: Keith Busch <kbusch@kernel.org> 
Sent: Friday, July 17, 2026 4:53 AM
To: Zeng, Jun1 <jun1.zeng@intel.com>
Cc: axboe@kernel.dk; hch@lst.de; sagi@grimberg.me; linux-nvme@lists.infradead.org; linux-kernel@vger.kernel.org; Shevchenko, Andriy <andriy.shevchenko@intel.com>; Cao, Gang <gang.cao@intel.com>; Jin, Jun I <jun.i.jin@intel.com>; Hu, Yong <yong.hu@intel.com>; Fang, Liang A <liang.a.fang@intel.com>
Subject: Re: [PATCH v1 1/1] nvme-pci: adaptive interrupt coalescing

On Wed, Jul 15, 2026 at 03:57:03PM +0800, Jun Zeng wrote:
> Add adaptive interrupt coalescing to improve IOPS for high-throughput 
> workloads. Monitor IO pressure periodically and enable/disable 
> coalescing automatically based on IOPS, queue depth, and inflight IO thresholds.
> This feature is enabled by default, can be controlled through debugfs 
> variable during runtime.

Can't you do all this from user space? You'd maybe need the blk-mq debugfs to get access to the individual hctx dispatch numbers, but maybe that's okay?

The atomic_long_inc where you placed it is especially harmful to polled queues.

Re: [PATCH v1 1/1] nvme-pci: adaptive interrupt coalescing

[Shevchenko, Andriy]

On Fri, Jul 17, 2026 at 04:10:08AM +0300, Zeng, Jun1 wrote:
> Hi Busch, 
> Thanks for your reply. The main reason for implementing this in kernel space
> is the lack of a unified, deployable component in user space to support this
> feature. You see that there are sheer number of OS distributions. This will
> require separate adaptations for each, usability might be compromised,
> and-most importantly-ongoing maintenance and updates would be difficult.
> Meanwhile, nvme driver in kernel space is providing these unified
> capabilities for different kinds of usage.  For the point of using
> atomic_long_inc in cqe polling we can discuss to do it in a proper way.

You need to stop top-posting!

> -----Original Message-----
> From: Keith Busch <kbusch@kernel.org> 
> Sent: Friday, July 17, 2026 4:53 AM
> To: Zeng, Jun1 <jun1.zeng@intel.com>
> Cc: axboe@kernel.dk; hch@lst.de; sagi@grimberg.me; linux-nvme@lists.infradead.org; linux-kernel@vger.kernel.org; Shevchenko, Andriy <andriy.shevchenko@intel.com>; Cao, Gang <gang.cao@intel.com>; Jin, Jun I <jun.i.jin@intel.com>; Hu, Yong <yong.hu@intel.com>; Fang, Liang A <liang.a.fang@intel.com>
> Subject: Re: [PATCH v1 1/1] nvme-pci: adaptive interrupt coalescing
> 
> On Wed, Jul 15, 2026 at 03:57:03PM +0800, Jun Zeng wrote:
> > Add adaptive interrupt coalescing to improve IOPS for high-throughput 
> > workloads. Monitor IO pressure periodically and enable/disable 
> > coalescing automatically based on IOPS, queue depth, and inflight IO thresholds.
> > This feature is enabled by default, can be controlled through debugfs 
> > variable during runtime.
> 
> Can't you do all this from user space? You'd maybe need the blk-mq debugfs to get access to the individual hctx dispatch numbers, but maybe that's okay?
> 
> The atomic_long_inc where you placed it is especially harmful to polled queues.

-- 
With Best Regards,
Andy Shevchenko

Re: [PATCH v1 0/1] nvme-pci: adaptive interrupt coalescing

[Bart Van Assche]

On 7/15/26 12:57 AM, Jun Zeng wrote:
> The IOPS for small IO can be significantly improved when enabling
> interrupt coalescing in this case. However, in some cases (e.g., big
> IO for bandwidth or latency-sensitive tests with small IO and small
> queue depth), interrupt coalescing has side effects.
> 
> Solution
> ========
> 
> Monitor the IOPS and average queue depth of inflight IO periodically
> on each NVMe drive and enable/disable interrupt coalescing
> automatically without user intervention. This feature is enabled by
> default and can be disabled/enabled through debugfs during runtime.

An algorithm like the above can be implemented more efficiently in NVMe
controller hardware than in software, isn't it?

Thanks,

Bart.

RE: [PATCH v1 0/1] nvme-pci: adaptive interrupt coalescing

[Zeng, Jun1]

> >On 7/15/26 12:57 AM, Jun Zeng wrote:
> > The IOPS for small IO can be significantly improved when enabling 
> > interrupt coalescing in this case. However, in some cases (e.g., big 
> > IO for bandwidth or latency-sensitive tests with small IO and small 
> > queue depth), interrupt coalescing has side effects.
> > 
> > Solution
> > ========
> > 
> > Monitor the IOPS and average queue depth of inflight IO periodically 
> > on each NVMe drive and enable/disable interrupt coalescing 
> > automatically without user intervention. This feature is enabled by 
> > default and can be disabled/enabled through debugfs during runtime.

> An algorithm like the above can be implemented more efficiently in NVMe controller hardware than in software, isn't it?

Theoretically, this can be implemented in the NVMe controller's firmware, but we have no control over the strategies and decisions of NVMe vendors. We are simply making the best possible optimizations at the software level.

BR,
Jun

Re: [PATCH v1 0/1] nvme-pci: adaptive interrupt coalescing

[Keith Busch]

On Mon, Jul 20, 2026 at 05:24:12AM +0000, Zeng, Jun1 wrote:
> > >On 7/15/26 12:57 AM, Jun Zeng wrote:
> > > The IOPS for small IO can be significantly improved when enabling 
> > > interrupt coalescing in this case. However, in some cases (e.g., big 
> > > IO for bandwidth or latency-sensitive tests with small IO and small 
> > > queue depth), interrupt coalescing has side effects.
> > > 
> > > Solution
> > > ========
> > > 
> > > Monitor the IOPS and average queue depth of inflight IO periodically 
> > > on each NVMe drive and enable/disable interrupt coalescing 
> > > automatically without user intervention. This feature is enabled by 
> > > default and can be disabled/enabled through debugfs during runtime.
> 
> > An algorithm like the above can be implemented more efficiently in NVMe controller hardware than in software, isn't it?
> 
> Theoretically, this can be implemented in the NVMe controller's
> firmware, but we have no control over the strategies and decisions of
> NVMe vendors. We are simply making the best possible optimizations at
> the software level.

But the coalescing feature as it is is too coarse. There is/was some
proposal going through the nvme technical workgroup to improve it. I
really haven't had any time to participate in the standards work to say
if it's any better, but that's probably the forum to improve such things
across vendors.

I'm not sure what concern you have about distro support with respect to
just controlling this from user space. AFAIK, everything usable has
sysfs, and you can't stop passthrough ioctls, so everything you need
should be there.

Re: [PATCH v1 0/1] nvme-pci: adaptive interrupt coalescing

[Christoph Hellwig]

On Fri, Jul 17, 2026 at 10:06:24AM -0700, Bart Van Assche wrote:
>> Monitor the IOPS and average queue depth of inflight IO periodically
>> on each NVMe drive and enable/disable interrupt coalescing
>> automatically without user intervention. This feature is enabled by
>> default and can be disabled/enabled through debugfs during runtime.
>
> An algorithm like the above can be implemented more efficiently in NVMe
> controller hardware than in software, isn't it?

There is a lot of discussion on interrupt coalescing in the NVMe
technical working group right now.  I would suggest everyone working
for a member company (and that should be everyone one the cc list
here) to check that out before rushing anything.

Re: [PATCH v1 1/1] nvme-pci: adaptive interrupt coalescing

[Christoph Hellwig]

On Thu, Jul 16, 2026 at 02:52:41PM -0600, Keith Busch wrote:
> On Wed, Jul 15, 2026 at 03:57:03PM +0800, Jun Zeng wrote:
> > Add adaptive interrupt coalescing to improve IOPS for high-throughput
> > workloads. Monitor IO pressure periodically and enable/disable coalescing
> > automatically based on IOPS, queue depth, and inflight IO thresholds.
> > This feature is enabled by default, can be controlled through debugfs
> > variable during runtime.
> 
> Can't you do all this from user space? You'd maybe need the blk-mq
> debugfs to get access to the individual hctx dispatch numbers, but maybe
> that's okay?

Having these kinds of interfaces where we rely on a userspace daemon
to be in sync with the kernel implementation are a mess, and really
lock us out of future improvements in this area.

> The atomic_long_inc where you placed it is especially harmful to polled
> queues.

We really should be able to do this without any counters, as blk-mq keeps
more than enough statistics.  And of course for polled queues nothing
related to interrupt coalescing is actually need.

And we should take a page from the networking playbook and look into
adaptively switching to polling under high load instead of just
mitigating interrupts.  An important part of that is to move the CQ
reaping from irq context to thread context.

But modulo all these caveats adaptive interrupt coalescing / 
moderation is something we absolute need right now and even more so
going foward.

Note that a few month ago there also was as an interesting patchset
that does global interrupt moderation at the IRQ controller level.
I don't remember what happened to it and can't find a link to it,
but adding the interrupt maintainer in case he remembers.

Re: [PATCH v1 1/1] nvme-pci: adaptive interrupt coalescing

[Keith Busch]

On Mon, Jul 20, 2026 at 05:05:39PM +0200, Christoph Hellwig wrote:
> On Thu, Jul 16, 2026 at 02:52:41PM -0600, Keith Busch wrote:
> > On Wed, Jul 15, 2026 at 03:57:03PM +0800, Jun Zeng wrote:
> > > Add adaptive interrupt coalescing to improve IOPS for high-throughput
> > > workloads. Monitor IO pressure periodically and enable/disable coalescing
> > > automatically based on IOPS, queue depth, and inflight IO thresholds.
> > > This feature is enabled by default, can be controlled through debugfs
> > > variable during runtime.
> > 
> > Can't you do all this from user space? You'd maybe need the blk-mq
> > debugfs to get access to the individual hctx dispatch numbers, but maybe
> > that's okay?
> 
> Having these kinds of interfaces where we rely on a userspace daemon
> to be in sync with the kernel implementation are a mess, and really
> lock us out of future improvements in this area.

We can't stop someone from doing this today, though.
 
> > The atomic_long_inc where you placed it is especially harmful to polled
> > queues.
> 
> We really should be able to do this without any counters, as blk-mq keeps
> more than enough statistics.  And of course for polled queues nothing
> related to interrupt coalescing is actually need.
> 
> And we should take a page from the networking playbook and look into
> adaptively switching to polling under high load instead of just
> mitigating interrupts.  An important part of that is to move the CQ
> reaping from irq context to thread context.

Totally. I had a proposal from I think 7 years ago to converge nvme's
threaded IRQ handling: reap the first X entries from hard-irq context,
and if there's more, return IRQ_WAKE_THREAD to poll the rest and future
completions with that queue's interrupt masked (this is one scenario
where NVMe's MSI masking is better than MSI-x, but it's not a big deal).
I can't find the patch though, so I guess it was in the gap that
infradead lost.

Anyway, I think combine that concept with the isolcpus proposal and you
can have full control over which CPUs can dispatch new IO without being
interrupted from CPUs reaping completions. Then we shouldn't need the
spec's coalescing feature.

> But modulo all these caveats adaptive interrupt coalescing / 
> moderation is something we absolute need right now and even more so
> going foward.
> 
> Note that a few month ago there also was as an interesting patchset
> that does global interrupt moderation at the IRQ controller level.
> I don't remember what happened to it and can't find a link to it,
> but adding the interrupt maintainer in case he remembers.

Re: [PATCH v1 1/1] nvme-pci: adaptive interrupt coalescing

[Fengnan Chang]

Hi all,

I'd like to add some information. A few months ago, when I was
testing Zeng's solution, I encountered some issues, primarily that
enabling or disabling interrupt aggregation on certain disks could
cause I/O latency of several tens of milliseconds. 

Inspired by the design of network NAPI, I came up with a similar idea
and implemented a patch, which performed well in testing.
If you're interested in this approach, I can continue to improve it.
The reason I didn't move forward with this approach is that our
initial goal was to solve the issues encountered with QEMU, but
disabling interrupts causes problems in VFIO+QEMU environments, and
this approach only works in physical machine scenarios. 

The patch estimates how busy a queue is from the submission queue tail
and the SQ head reported in completion entries.  When the number of
pending commands crosses a high watermark, the hard IRQ handles the
current completions, masks the queue's interrupt and wakes the IRQ
thread.  The IRQ thread then polls for more completions.  It switches
back to interrupts when the pending count falls below a low watermark,
or when a bounded number of polling loops is reached.  Using separate
high and low watermarks avoids switching back and forth too often.

This is only enabled when the controller has separate MSI-X vectors, so
masking one queue does not block completions from other queues.

I tested it with four Solidigm NVMe devices using 4K random reads.  fio
used regular io_uring, without hipri or SQ polling.  These are the
median results from three runs on a 7.2-rc1 based kernel:

                         baseline    patched
  IOPS                   7.61M       11.41M  (+50%)
  mean latency           267 us      177 us  (-34%)
  hard IRQs per I/O      0.52        0.21    (-60%)

Without the patch, three of the four devices were limited to about
1.49M IOPS each.  With the patch, all four devices reached around
2.8M-2.9M IOPS.

The patch is included below. 

Thanks,
Fengnan

From 1c84b5afc906e780e1dbd2ee186251f9de2716b5 Mon Sep 17 00:00:00 2001
From: Fengnan Chang <changfengnan@bytedance.com>
Date: Tue, 21 Jul 2026 12:31:46 +0800
Subject: [PATCH] nvme-pci: adaptively poll busy queues from threaded
 interrupts

On systems with multiple fast NVMe devices, hard IRQ completion
processing can become CPU-bound and cap 4K random-read throughput
before the devices are saturated.

Add adaptive polling for busy interrupt-driven queues. Track the SQ head
reported by completions while the CQE is still owned by the host. After
the hard IRQ reaps pending CQEs, wake the IRQ thread and mask the queue
interrupt when the number of SQ entries not yet consumed reaches a high
watermark. Poll in the IRQ thread until the queue drops below a low
watermark or a bounded loop count is reached, then reenable the
interrupt.

Only enable adaptive polling when the controller has independent
vectors, as masking a shared single vector could suppress unrelated
queues. Expose the enable switch, watermarks, loop bound, and sleep
interval as module parameters.

Signed-off-by: Fengnan Chang <changfengnan@bytedance.com>
---
 drivers/nvme/host/pci.c | 212 +++++++++++++++++++++++++++++++++++++---
 1 file changed, 198 insertions(+), 14 deletions(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 69932d640b537..be3a3ea0196f5 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -9,6 +9,7 @@
 #include <linux/blkdev.h>
 #include <linux/blk-mq-dma.h>
 #include <linux/blk-integrity.h>
+#include <linux/delay.h>
 #include <linux/dmi.h>
 #include <linux/init.h>
 #include <linux/interrupt.h>
@@ -79,8 +80,136 @@ struct quirk_entry {
 	u32 disabled_quirks;
 };
 
-static int use_threaded_interrupts;
-module_param(use_threaded_interrupts, int, 0444);
+static int use_adaptive_polling;
+module_param(use_adaptive_polling, int, 0444);
+MODULE_PARM_DESC(use_adaptive_polling,
+		 "use adaptive completion polling on busy interrupt queues");
+
+static unsigned int adaptive_poll_start_pending = 10;
+static unsigned int adaptive_poll_stop_pending = 3;
+static unsigned int adaptive_poll_max_loops = 100;
+static unsigned int adaptive_poll_sleep_min_us = 20;
+static unsigned int adaptive_poll_sleep_max_us = 30;
+
+static int nvme_adaptive_poll_start_pending_set(const char *val, const struct kernel_param *kp)
+{
+	unsigned int pending;
+	int ret;
+
+	ret = kstrtouint(val, 10, &pending);
+	if (ret)
+		return ret;
+	if (!pending || pending < READ_ONCE(adaptive_poll_stop_pending))
+		return -EINVAL;
+
+	WRITE_ONCE(*(unsigned int *)kp->arg, pending);
+	return 0;
+}
+
+static const struct kernel_param_ops nvme_adaptive_poll_start_pending_ops = {
+	.set = nvme_adaptive_poll_start_pending_set,
+	.get = param_get_uint,
+};
+
+module_param_cb(adaptive_poll_start_pending, &nvme_adaptive_poll_start_pending_ops,
+		&adaptive_poll_start_pending, 0644);
+MODULE_PARM_DESC(adaptive_poll_start_pending, "SQ pending high watermark for adaptive polling");
+
+static int nvme_adaptive_poll_stop_pending_set(const char *val, const struct kernel_param *kp)
+{
+	unsigned int pending;
+	int ret;
+
+	ret = kstrtouint(val, 10, &pending);
+	if (ret)
+		return ret;
+	if (!pending || pending > READ_ONCE(adaptive_poll_start_pending))
+		return -EINVAL;
+
+	WRITE_ONCE(*(unsigned int *)kp->arg, pending);
+	return 0;
+}
+
+static const struct kernel_param_ops nvme_adaptive_poll_stop_pending_ops = {
+	.set = nvme_adaptive_poll_stop_pending_set,
+	.get = param_get_uint,
+};
+
+module_param_cb(adaptive_poll_stop_pending, &nvme_adaptive_poll_stop_pending_ops,
+		&adaptive_poll_stop_pending, 0644);
+MODULE_PARM_DESC(adaptive_poll_stop_pending, "SQ pending low watermark for adaptive polling");
+
+static int nvme_adaptive_poll_max_loops_set(const char *val, const struct kernel_param *kp)
+{
+	unsigned int loops;
+	int ret;
+
+	ret = kstrtouint(val, 10, &loops);
+	if (ret)
+		return ret;
+	if (!loops)
+		return -EINVAL;
+
+	WRITE_ONCE(*(unsigned int *)kp->arg, loops);
+	return 0;
+}
+
+static const struct kernel_param_ops nvme_adaptive_poll_max_loops_ops = {
+	.set = nvme_adaptive_poll_max_loops_set,
+	.get = param_get_uint,
+};
+
+module_param_cb(adaptive_poll_max_loops, &nvme_adaptive_poll_max_loops_ops,
+		&adaptive_poll_max_loops, 0644);
+MODULE_PARM_DESC(adaptive_poll_max_loops, "Maximum adaptive polling loops");
+
+static int nvme_adaptive_poll_sleep_min_us_set(const char *val, const struct kernel_param *kp)
+{
+	unsigned int usecs;
+	int ret;
+
+	ret = kstrtouint(val, 10, &usecs);
+	if (ret)
+		return ret;
+	if (!usecs || usecs > READ_ONCE(adaptive_poll_sleep_max_us))
+		return -EINVAL;
+
+	WRITE_ONCE(*(unsigned int *)kp->arg, usecs);
+	return 0;
+}
+
+static const struct kernel_param_ops nvme_adaptive_poll_sleep_min_us_ops = {
+	.set = nvme_adaptive_poll_sleep_min_us_set,
+	.get = param_get_uint,
+};
+
+module_param_cb(adaptive_poll_sleep_min_us, &nvme_adaptive_poll_sleep_min_us_ops,
+		&adaptive_poll_sleep_min_us, 0644);
+MODULE_PARM_DESC(adaptive_poll_sleep_min_us, "Minimum adaptive polling delay in microseconds");
+
+static int nvme_adaptive_poll_sleep_max_us_set(const char *val, const struct kernel_param *kp)
+{
+	unsigned int usecs;
+	int ret;
+
+	ret = kstrtouint(val, 10, &usecs);
+	if (ret)
+		return ret;
+	if (!usecs || usecs < READ_ONCE(adaptive_poll_sleep_min_us))
+		return -EINVAL;
+
+	WRITE_ONCE(*(unsigned int *)kp->arg, usecs);
+	return 0;
+}
+
+static const struct kernel_param_ops nvme_adaptive_poll_sleep_max_us_ops = {
+	.set = nvme_adaptive_poll_sleep_max_us_set,
+	.get = param_get_uint,
+};
+
+module_param_cb(adaptive_poll_sleep_max_us, &nvme_adaptive_poll_sleep_max_us_ops,
+		&adaptive_poll_sleep_max_us, 0644);
+MODULE_PARM_DESC(adaptive_poll_sleep_max_us, "Maximum adaptive polling delay in microseconds");
 
 static bool use_cmb_sqes = true;
 module_param(use_cmb_sqes, bool, 0444);
@@ -377,6 +506,7 @@ struct nvme_queue {
 	u16 cq_vector;
 	u16 sq_tail;
 	u16 last_sq_tail;
+	u16 sq_head;
 	u16 cq_head;
 	u16 qid;
 	u8 cq_phase;
@@ -1607,7 +1737,7 @@ static inline void nvme_update_cq_head(struct nvme_queue *nvmeq)
 }
 
 static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,
-			        struct io_comp_batch *iob)
+				struct io_comp_batch *iob, u16 *sq_head)
 {
 	bool found = false;
 
@@ -1618,6 +1748,8 @@ static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,
 		 * the cqe requires a full read memory barrier
 		 */
 		dma_rmb();
+		if (sq_head)
+			*sq_head = le16_to_cpu(nvmeq->cqes[nvmeq->cq_head].sq_head);
 		nvme_handle_cqe(nvmeq, iob, nvmeq->cq_head);
 		nvme_update_cq_head(nvmeq);
 	}
@@ -1631,8 +1763,11 @@ static irqreturn_t nvme_irq(int irq, void *data)
 {
 	struct nvme_queue *nvmeq = data;
 	DEFINE_IO_COMP_BATCH(iob);
+	u16 sq_head;
+
+	if (nvme_poll_cq(nvmeq, &iob, &sq_head)) {
+		WRITE_ONCE(nvmeq->sq_head, sq_head);
 
-	if (nvme_poll_cq(nvmeq, &iob)) {
 		if (!rq_list_empty(&iob.req_list))
 			nvme_pci_complete_batch(&iob);
 		return IRQ_HANDLED;
@@ -1640,13 +1775,56 @@ static irqreturn_t nvme_irq(int irq, void *data)
 	return IRQ_NONE;
 }
 
-static irqreturn_t nvme_irq_check(int irq, void *data)
+static unsigned int nvme_sq_pending(struct nvme_queue *nvmeq)
+{
+	int pending = READ_ONCE(nvmeq->sq_tail) - READ_ONCE(nvmeq->sq_head);
+
+	if (pending < 0)
+		pending += nvmeq->q_depth;
+	return pending;
+}
+
+static bool nvme_adaptive_poll_should_start(struct nvme_queue *nvmeq)
+{
+	return nvme_sq_pending(nvmeq) >= READ_ONCE(adaptive_poll_start_pending);
+}
+
+static bool nvme_adaptive_poll_should_stop(struct nvme_queue *nvmeq)
+{
+	return nvme_sq_pending(nvmeq) < READ_ONCE(adaptive_poll_stop_pending);
+}
+
+static irqreturn_t nvme_adaptive_poll(int irq, void *data)
 {
 	struct nvme_queue *nvmeq = data;
+	unsigned int loops = 0;
+	unsigned int max_loops = READ_ONCE(adaptive_poll_max_loops);
+	unsigned int sleep_min = READ_ONCE(adaptive_poll_sleep_min_us);
+	unsigned int sleep_max = READ_ONCE(adaptive_poll_sleep_max_us);
 
-	if (nvme_cqe_pending(nvmeq))
-		return IRQ_WAKE_THREAD;
-	return IRQ_NONE;
+	do {
+		if (nvme_cqe_pending(nvmeq))
+			nvme_irq(irq, data);
+		if (nvme_adaptive_poll_should_stop(nvmeq))
+			break;
+		usleep_range(sleep_min, sleep_max);
+	} while (++loops < max_loops);
+
+	enable_irq(irq);
+	return IRQ_HANDLED;
+}
+
+static irqreturn_t nvme_irq_adaptive_poll(int irq, void *data)
+{
+	struct nvme_queue *nvmeq = data;
+	irqreturn_t ret;
+
+	ret = nvme_irq(irq, data);
+	if (!nvme_adaptive_poll_should_start(nvmeq))
+		return ret;
+
+	disable_irq_nosync(irq);
+	return IRQ_WAKE_THREAD;
 }
 
 /*
@@ -1663,7 +1841,7 @@ static void nvme_poll_irqdisable(struct nvme_queue *nvmeq)
 	irq = pci_irq_vector(pdev, nvmeq->cq_vector);
 	disable_irq(irq);
 	spin_lock(&nvmeq->cq_poll_lock);
-	nvme_poll_cq(nvmeq, NULL);
+	nvme_poll_cq(nvmeq, NULL, NULL);
 	spin_unlock(&nvmeq->cq_poll_lock);
 	enable_irq(irq);
 }
@@ -1678,7 +1856,7 @@ static int nvme_poll(struct blk_mq_hw_ctx *hctx, struct io_comp_batch *iob)
 		return 0;
 
 	spin_lock(&nvmeq->cq_poll_lock);
-	found = nvme_poll_cq(nvmeq, iob);
+	found = nvme_poll_cq(nvmeq, iob, NULL);
 	spin_unlock(&nvmeq->cq_poll_lock);
 
 	return found;
@@ -2075,7 +2253,7 @@ static void nvme_reap_pending_cqes(struct nvme_dev *dev)
 
 	for (i = dev->ctrl.queue_count - 1; i > 0; i--) {
 		spin_lock(&dev->queues[i].cq_poll_lock);
-		nvme_poll_cq(&dev->queues[i], NULL);
+		nvme_poll_cq(&dev->queues[i], NULL, NULL);
 		spin_unlock(&dev->queues[i].cq_poll_lock);
 	}
 }
@@ -2171,9 +2349,14 @@ static int queue_request_irq(struct nvme_queue *nvmeq)
 	struct pci_dev *pdev = to_pci_dev(nvmeq->dev->dev);
 	int nr = nvmeq->dev->ctrl.instance;
 
-	if (use_threaded_interrupts) {
-		return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq_check,
-				nvme_irq, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
+	/*
+	 * Adaptive polling masks the entire vector, so don't use it when
+	 * multiple queues share the single available vector.
+	 */
+	if (use_adaptive_polling && nvmeq->dev->num_vecs > 1) {
+		return pci_request_irq(pdev, nvmeq->cq_vector,
+				nvme_irq_adaptive_poll, nvme_adaptive_poll, nvmeq,
+				"nvme%dq%d", nr, nvmeq->qid);
 	} else {
 		return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq,
 				NULL, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
@@ -2186,6 +2369,7 @@ static void nvme_init_queue(struct nvme_queue *nvmeq, u16 qid)
 
 	nvmeq->sq_tail = 0;
 	nvmeq->last_sq_tail = 0;
+	nvmeq->sq_head = 0;
 	nvmeq->cq_head = 0;
 	nvmeq->cq_phase = 1;
 	nvmeq->q_db = &dev->dbs[qid * 2 * dev->db_stride];

Re: [PATCH v1 1/1] nvme-pci: adaptive interrupt coalescing

[Andy Shevchenko]

On Tue, Jul 21, 2026 at 04:37:39PM +0800, Fengnan Chang wrote:

> The patch is included below. 

The above in this case can be moved to cover letter, in this form it's harder
for maintainers to apply this.

...

> From 1c84b5afc906e780e1dbd2ee186251f9de2716b5 Mon Sep 17 00:00:00 2001
> From: Fengnan Chang <changfengnan@bytedance.com>
> Date: Tue, 21 Jul 2026 12:31:46 +0800
> Subject: [PATCH] nvme-pci: adaptively poll busy queues from threaded
>  interrupts
> 
> On systems with multiple fast NVMe devices, hard IRQ completion
> processing can become CPU-bound and cap 4K random-read throughput
> before the devices are saturated.
> 
> Add adaptive polling for busy interrupt-driven queues. Track the SQ head
> reported by completions while the CQE is still owned by the host. After
> the hard IRQ reaps pending CQEs, wake the IRQ thread and mask the queue
> interrupt when the number of SQ entries not yet consumed reaches a high
> watermark. Poll in the IRQ thread until the queue drops below a low
> watermark or a bounded loop count is reached, then reenable the
> interrupt.
> 
> Only enable adaptive polling when the controller has independent
> vectors, as masking a shared single vector could suppress unrelated
> queues. Expose the enable switch, watermarks, loop bound, and sleep
> interval as module parameters.

...

> +static irqreturn_t nvme_adaptive_poll(int irq, void *data)
>  {
>  	struct nvme_queue *nvmeq = data;
> +	unsigned int loops = 0;

Unneeded. See below how.

> +	unsigned int max_loops = READ_ONCE(adaptive_poll_max_loops);
> +	unsigned int sleep_min = READ_ONCE(adaptive_poll_sleep_min_us);
> +	unsigned int sleep_max = READ_ONCE(adaptive_poll_sleep_max_us);
>  
> -	if (nvme_cqe_pending(nvmeq))
> -		return IRQ_WAKE_THREAD;
> -	return IRQ_NONE;
> +	do {
> +		if (nvme_cqe_pending(nvmeq))
> +			nvme_irq(irq, data);
> +		if (nvme_adaptive_poll_should_stop(nvmeq))
> +			break;

> +		usleep_range(sleep_min, sleep_max);

fsleep() ? This automatically makes the upper bound ~25% of the requested value.
Supplying max in this situation makes not much sense.


> +	} while (++loops < max_loops);

	} while (--max_loops);

> +	enable_irq(irq);
> +	return IRQ_HANDLED;
> +}

-- 
With Best Regards,
Andy Shevchenko

Re: [PATCH v1 1/1] nvme-pci: adaptive interrupt coalescing

[Keith Busch]

On Tue, Jul 21, 2026 at 04:37:39PM +0800, Fengnan Chang wrote:
> @@ -1618,6 +1748,8 @@ static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,
>  		 * the cqe requires a full read memory barrier
>  		 */
>  		dma_rmb();
> +		if (sq_head)
> +			*sq_head = le16_to_cpu(nvmeq->cqes[nvmeq->cq_head].sq_head);

This is the wrong criteria to determine the outstanding depth. The spec
allows the controller to move this forward after it has read an entry.
The commands may still be in progress, so the depth of future
completions to expect can't depend on this value.

But in general, I agree with the spirit of where this is going.

For MSI, you can make use of the INTMS/INTMC NVMe registers to make this
even more efficient. Your patch disables the irq at the cpU level, but
the device is still emitting those messages for no reason.