[PATCH 0/3] hw/nvme: fix legacy INTx handling

[PATCH 0/3] hw/nvme: fix legacy INTx handling

[Markku Ahvenjärvi]

This series fixes three cases where an NVMe legacy INTx interrupt may
remain asserted, causing an interrupt storm in the guest:

* cq_pending can underflow when a doorbell notification is processed for
  a completion queue that is already empty.

* A stale CQ event index can cause the guest to suppress a subsequent
  MMIO CQ doorbell write, preventing QEMU from observing the updated CQ
  head.

* Controller reset does not clear pending CQ interrupt state.

Without these fixes, a Linux guest using legacy PCI interrupts may see
the following:

[    2.058360] irq 23: nobody cared (try booting with the "irqpoll" option)
[    2.058724] handlers:
[    2.058820] [<000000006a09281a>] nvme_irq
[    2.058966] [<000000006a09281a>] nvme_irq
[    2.059113] Disabling IRQ #23

I reproduced the interrupt storm with Linux 6.12, 6.18, and 7.2 by
booting the guest with pci=nomsi. The cq_pending underflow occurred with
ioeventfd=on, but its fix covers both ioeventfd=on and ioeventfd=off
configurations. The stale event-index issue occurred with both
ioeventfd=on and ioeventfd=off.

The following command was used to reproduce the interrupt storm with
ioeventfd=on:

qemu-system-x86_64 \
      -machine q35,accel=kvm \
      -m 2G -smp 2 \
      -kernel ./linux-7.2/arch/x86/boot/bzImage \
      -append 'console=ttyS0 root=/dev/nvme0n1p1 rw pci=nomsi' \
      -drive if=none,file=./guest-image.qcow2,id=disk0,format=qcow2 \
      -device nvme,drive=disk0,serial=testserial0,ioeventfd=on \
      -nographic

The controller-reset fix addresses QEMU issue 4142:
https://gitlab.com/qemu-project/qemu/-/work_items/4142

I was unable to reproduce this issue with a Linux guest, but a
controller reset must clear the CQ interrupt state and deassert the
interrupt line.

Kind regards,
Markku Ahvenjärvi

---
Markku Ahvenjärvi (3):
      hw/nvme: prevent cq_pending underflow
      hw/nvme: fix missed CQ updates
      hw/nvme: clear interrupts on controller reset

 hw/nvme/ctrl.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
---
base-commit: 7074591d7954876951f84c15b994a43251d5a3c1
change-id: 20260922-nvme-intx-fixes-10b70b6520ba

Best regards,
--  
Markku Ahvenjärvi <mankku@gmail.com>

[PATCH 1/3] hw/nvme: prevent cq_pending underflow

[Markku Ahvenjärvi]

cq_pending counts interrupt-enabled completion queues with unconsumed
entries. nvme_cq_notifier() and nvme_process_db() decrement it whenever
a CQ becomes empty, but they may also process a notification or doorbell
write for a CQ that was already empty.

Decrement cq_pending only when the CQ contained pending completions
before its head update. Otherwise cq_pending may underflow and
prevent legacy INTx from being deasserted.

Signed-off-by: Markku Ahvenjärvi <mankku@gmail.com>
---
 hw/nvme/ctrl.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c
index c641226e27ad..2be4bab7eb5a 100644
--- a/hw/nvme/ctrl.c
+++ b/hw/nvme/ctrl.c
@@ -4782,6 +4782,7 @@ static void nvme_cq_notifier(EventNotifier *e)
 {
     NvmeCQueue *cq = container_of(e, NvmeCQueue, notifier);
     NvmeCtrl *n = cq->ctrl;
+    bool pending = cq->tail != cq->head;
 
     if (!event_notifier_test_and_clear(e)) {
         return;
@@ -4790,7 +4791,7 @@ static void nvme_cq_notifier(EventNotifier *e)
     nvme_update_cq_head(cq);
 
     if (cq->tail == cq->head) {
-        if (cq->irq_enabled) {
+        if (cq->irq_enabled && pending) {
             n->cq_pending--;
         }
 
@@ -8555,6 +8556,7 @@ static void nvme_process_db(NvmeCtrl *n, hwaddr addr, int val)
 
         uint16_t new_head = val & 0xffff;
         NvmeCQueue *cq;
+        bool pending;
 
         qid = (addr - (0x1000 + (1 << 2))) >> 3;
         if (unlikely(nvme_check_cqid(n, qid))) {
@@ -8609,13 +8611,14 @@ static void nvme_process_db(NvmeCtrl *n, hwaddr addr, int val)
             qemu_bh_schedule(cq->bh);
         }
 
+        pending = cq->tail != cq->head;
         cq->head = new_head;
         if (!qid && n->dbbuf_enabled) {
             stl_le_pci_dma(pci, cq->db_addr, cq->head, MEMTXATTRS_UNSPECIFIED);
         }
 
         if (cq->tail == cq->head) {
-            if (cq->irq_enabled) {
+            if (cq->irq_enabled && pending) {
                 n->cq_pending--;
             }
 

-- 
2.55.0

Re: [PATCH 1/3] hw/nvme: prevent cq_pending underflow

[Keith Busch]

On Wed, Sep 23, 2026 at 03:20:02PM +0300, Markku Ahvenjärvi wrote:
> cq_pending counts interrupt-enabled completion queues with unconsumed
> entries. nvme_cq_notifier() and nvme_process_db() decrement it whenever
> a CQ becomes empty, but they may also process a notification or doorbell
> write for a CQ that was already empty.

Is there a host actually doing this? If the host writes a doorbell
register to an empty queue, that is a spec violation on the host and the
controller should fail the queue. If it's not the admin queue, then the
controller should to send the Invalid Doorbell Write Value AEN
completion.

Re: [PATCH 1/3] hw/nvme: prevent cq_pending underflow

[Markku Ahvenjärvi]

On Wed, Sep 23, 2026 at 5:39 PM Keith Busch <kbusch@kernel.org> wrote:
>
> On Wed, Sep 23, 2026 at 03:20:02PM +0300, Markku Ahvenjärvi wrote:
> > cq_pending counts interrupt-enabled completion queues with unconsumed
> > entries. nvme_cq_notifier() and nvme_process_db() decrement it whenever
> > a CQ becomes empty, but they may also process a notification or doorbell
> > write for a CQ that was already empty.
>
> Is there a host actually doing this? If the host writes a doorbell
> register to an empty queue, that is a spec violation on the host and the
> controller should fail the queue. If it's not the admin queue, then the
> controller should to send the Invalid Doorbell Write Value AEN
> completion.

No, the host is (probably) doing the right thing.

nvme_post_cqes() also updates the CQ head, so it can run before QEMU
processes the corresponding doorbell. I observed this with ioeventfd=on,
where MMIO doorbell handling is asynchronous, which makes the already empty
CQ case more likely.

Re: [PATCH 1/3] hw/nvme: prevent cq_pending underflow

[Keith Busch]

On Thu, Sep 24, 2026 at 09:56:11AM +0300, Markku Ahvenjärvi wrote:
> nvme_post_cqes() also updates the CQ head, so it can run before QEMU
> processes the corresponding doorbell. I observed this with ioeventfd=on,
> where MMIO doorbell handling is asynchronous, which makes the already empty
> CQ case more likely.

Are you also using shadow doorbells?

Re: [PATCH 1/3] hw/nvme: prevent cq_pending underflow

[Markku Ahvenjärvi]

On Thu, Sep 24, 2026 at 8:57 PM Keith Busch <kbusch@kernel.org> wrote:
> Are you also using shadow doorbells?

Yes. The ioeventfd calling nvme_cq_notifier() handler is only
configured when n->dbbuf_enabled == true.

Kind regards,
Markku Ahvenjärvi

Re: [PATCH 1/3] hw/nvme: prevent cq_pending underflow

[Markku Ahvenjärvi]

Hi,

On Fri, Sep 25, 2026 at 10:32 AM Markku Ahvenjärvi <mankku@gmail.com> wrote:
>
> On Thu, Sep 24, 2026 at 8:57 PM Keith Busch <kbusch@kernel.org> wrote:
> > Are you also using shadow doorbells?
>
> Yes. The ioeventfd calling nvme_cq_notifier() handler is only
> configured when n->dbbuf_enabled == true.

A gentle ping on these series. Please let me know if further
clarification or changes are needed.

Thanks,
Markku

Re: [PATCH 1/3] hw/nvme: prevent cq_pending underflow

[Keith Busch]

On Mon, Oct 05, 2026 at 08:54:15AM +0300, Markku Ahvenjärvi wrote:
> Hi,
> 
> On Fri, Sep 25, 2026 at 10:32 AM Markku Ahvenjärvi <mankku@gmail.com> wrote:
> >
> > On Thu, Sep 24, 2026 at 8:57 PM Keith Busch <kbusch@kernel.org> wrote:
> > > Are you also using shadow doorbells?
> >
> > Yes. The ioeventfd calling nvme_cq_notifier() handler is only
> > configured when n->dbbuf_enabled == true.
> 
> A gentle ping on these series. Please let me know if further
> clarification or changes are needed.

Yeah, this looks fine to me. Lots of conference travel the last couple
weeks, so perpetually behind on reviews. :(

[PATCH 2/3] hw/nvme: fix missed CQ updates

[Markku Ahvenjärvi]

When doorbell buffers are enabled, CQ head updates may race with
completion posting. When a CQ doorbell leaves pending completions, the
event index may become stale and cause the guest to suppress the next
MMIO doorbell write. QEMU may then fail to observe that the CQ has
been drained. With legacy INTx, this can leave the interrupt asserted
indefinitely and cause an interrupt storm.

Update the CQ event index as part of doorbell processing to ensure that
subsequent head updates notify QEMU.

Signed-off-by: Markku Ahvenjärvi <mankku@gmail.com>
---
 hw/nvme/ctrl.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c
index 2be4bab7eb5a..3545a3de236d 100644
--- a/hw/nvme/ctrl.c
+++ b/hw/nvme/ctrl.c
@@ -4788,6 +4788,7 @@ static void nvme_cq_notifier(EventNotifier *e)
         return;
     }
 
+    nvme_update_cq_eventidx(cq);
     nvme_update_cq_head(cq);
 
     if (cq->tail == cq->head) {
@@ -8623,6 +8624,8 @@ static void nvme_process_db(NvmeCtrl *n, hwaddr addr, int val)
             }
 
             nvme_irq_deassert(n, cq);
+        } else if (n->dbbuf_enabled) {
+            nvme_update_cq_eventidx(cq);
         }
     } else {
         /* Submission queue doorbell write */

-- 
2.55.0

[PATCH 3/3] hw/nvme: clear interrupts on controller reset

[Markku Ahvenjärvi]

Clear stale CQ interrupt state and ensure the interrupt line is
deasserted when resetting an NVMe controller.

Resolves: https://gitlab.com/qemu-project/qemu/-/work_items/4142
Signed-off-by: Markku Ahvenjärvi <mankku@gmail.com>
---
 hw/nvme/ctrl.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c
index 3545a3de236d..1de6a0e7f487 100644
--- a/hw/nvme/ctrl.c
+++ b/hw/nvme/ctrl.c
@@ -8126,6 +8126,10 @@ static void nvme_ctrl_reset(NvmeCtrl *n, NvmeResetType rst)
     stl_le_p(&n->bar.intmc, 0);
     stl_le_p(&n->bar.cc, 0);
 
+    n->cq_pending = 0;
+    n->irq_status = 0;
+    nvme_irq_check(n);
+
     n->dbbuf_dbs = 0;
     n->dbbuf_eis = 0;
     n->dbbuf_enabled = false;

-- 
2.55.0