Question: NVMe AER completion delivery during controller removal

Question: NVMe AER completion delivery during controller removal

[zhang san]

Hello,

We are seeking clarification about asynchronous event request (AER)
completion behavior during NVMe controller removal, before treating an
observed test failure as a production-reachable driver bug.

Our source analysis suggests a possible lifetime race between the AEN
completion path, which can requeue nvme_async_event_work, and the
controller removal path. We observed KASAN use-after-free in an
emulated environment on Linux v7.4-rc1 (6628e7d4c6b2).

However, our test induced an ERROR AEN through an out-of-specification
MMIO doorbell write while the device was being removed. This
deliberately synthetic trigger does not establish that a compliant
controller would generate the same completion.

Our main question is: can a specification-compliant controller deliver
a completion for a previously submitted AER during the interval in
which the host is disabling the controller and tearing down the admin
queue, or must such completions be quiesced before disable completes?
We would also welcome guidance on the exact point after which the host
may safely assume that no further AER completions can be delivered.

If a compliant device can generate a completion in the relevant
window, we can follow up with the source-level race analysis, complete
KASAN traces, and test details. Otherwise, we will not characterize
the synthetic result as a production-reachable UAF.

If a standards-compliant trigger or fix is identified, please CC me on
the relevant patch. I can help validate the reproduction conditions
and test the exact proposed change, providing a Tested-by tag if
verification succeeds.

The investigation used AI-assisted source analysis and experiments
reviewed by us.

Best regards
Yanhan Li

Re: Question: NVMe AER completion delivery during controller removal

[Mohamed Khalfella]

On Wed 2026-11-04 12:50:42 +0800, zhang san wrote:
> Hello,
> 
> We are seeking clarification about asynchronous event request (AER)
> completion behavior during NVMe controller removal, before treating an
> observed test failure as a production-reachable driver bug.
> 
> Our source analysis suggests a possible lifetime race between the AEN
> completion path, which can requeue nvme_async_event_work, and the
> controller removal path. We observed KASAN use-after-free in an
> emulated environment on Linux v7.4-rc1 (6628e7d4c6b2).


My reading of the code suggests this can happen. nvme_stop_ctrl()
flushes async_event_work, but the admin queue can still deliver an AER
completion afterwards, either from the admin queue interrupt handler or
from the final admin CQ poll in nvme_dev_disable():

nvme_remove() ->
  nvme_dev_disable() ->
    nvme_poll_irqdisable(&dev->queues[0]) ->
      nvme_poll_cq() ->
        nvme_handle_cqe() ->
          nvme_complete_async_event() ->
	    queue_work(nvme_wq, &ctrl->async_event_work);


Nothing flushes async_event_work after that, so the work can run after
nvme_uninit_ctrl() drops the last reference and the nvme_dev is freed.

> 
> However, our test induced an ERROR AEN through an out-of-specification
> MMIO doorbell write while the device was being removed. This
> deliberately synthetic trigger does not establish that a compliant
> controller would generate the same completion.

--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -3909,20 +3909,21 @@ static void nvme_remove(struct pci_dev *pdev)

        if (!pci_device_is_present(pdev)) {
                nvme_change_ctrl_state(&dev->ctrl, NVME_CTRL_DEAD);
                nvme_dev_disable(dev, true);
        }

        flush_work(&dev->ctrl.reset_work);
        nvme_stop_ctrl(&dev->ctrl);
        nvme_remove_namespaces(&dev->ctrl);
        nvme_dev_disable(dev, true);
+       flush_work(&dev->ctrl.async_event_work);
        nvme_free_host_mem(dev);
        nvme_dev_remove_admin(dev);
        nvme_dbbuf_dma_free(dev);
        nvme_free_queues(dev, 0);
        mempool_destroy(dev->dmavec_mempool);
        nvme_release_descriptor_pools(dev);
        nvme_dev_unmap(dev);
        nvme_uninit_ctrl(&dev->ctrl);
 }

Since you are able to reproduce this issue reliabely, how about you try
the diff above and see if it fixes the issue.

> 
> Our main question is: can a specification-compliant controller deliver
> a completion for a previously submitted AER during the interval in
> which the host is disabling the controller and tearing down the admin
> queue, or must such completions be quiesced before disable completes?
> We would also welcome guidance on the exact point after which the host
> may safely assume that no further AER completions can be delivered.

If the above diff fixes the issue for you, then I think this is a driver
bug.

Re: Question: NVMe AER completion delivery during controller removal

[Mohamed Khalfella]

On Mon 2026-11-09 11:08:54 -0800, Mohamed Khalfella wrote:
> On Wed 2026-11-04 12:50:42 +0800, zhang san wrote:
> > Hello,
> > 
> > We are seeking clarification about asynchronous event request (AER)
> > completion behavior during NVMe controller removal, before treating an
> > observed test failure as a production-reachable driver bug.
> > 
> > Our source analysis suggests a possible lifetime race between the AEN
> > completion path, which can requeue nvme_async_event_work, and the
> > controller removal path. We observed KASAN use-after-free in an
> > emulated environment on Linux v7.4-rc1 (6628e7d4c6b2).
> 
> 
> My reading of the code suggests this can happen. nvme_stop_ctrl()
> flushes async_event_work, but the admin queue can still deliver an AER
> completion afterwards, either from the admin queue interrupt handler or
> from the final admin CQ poll in nvme_dev_disable():
> 
> nvme_remove() ->
>   nvme_dev_disable() ->
>     nvme_poll_irqdisable(&dev->queues[0]) ->
>       nvme_poll_cq() ->
>         nvme_handle_cqe() ->
>           nvme_complete_async_event() ->
> 	    queue_work(nvme_wq, &ctrl->async_event_work);
> 
> 
> Nothing flushes async_event_work after that, so the work can run after
> nvme_uninit_ctrl() drops the last reference and the nvme_dev is freed.
> 
> > 
> > However, our test induced an ERROR AEN through an out-of-specification
> > MMIO doorbell write while the device was being removed. This
> > deliberately synthetic trigger does not establish that a compliant
> > controller would generate the same completion.
> 
> --- a/drivers/nvme/host/pci.c
> +++ b/drivers/nvme/host/pci.c
> @@ -3909,20 +3909,21 @@ static void nvme_remove(struct pci_dev *pdev)
> 
>         if (!pci_device_is_present(pdev)) {
>                 nvme_change_ctrl_state(&dev->ctrl, NVME_CTRL_DEAD);
>                 nvme_dev_disable(dev, true);
>         }
> 
>         flush_work(&dev->ctrl.reset_work);
>         nvme_stop_ctrl(&dev->ctrl);
>         nvme_remove_namespaces(&dev->ctrl);
>         nvme_dev_disable(dev, true);
> +       flush_work(&dev->ctrl.async_event_work);
>         nvme_free_host_mem(dev);
>         nvme_dev_remove_admin(dev);
>         nvme_dbbuf_dma_free(dev);
>         nvme_free_queues(dev, 0);
>         mempool_destroy(dev->dmavec_mempool);
>         nvme_release_descriptor_pools(dev);
>         nvme_dev_unmap(dev);
>         nvme_uninit_ctrl(&dev->ctrl);
>  }
> 
> Since you are able to reproduce this issue reliabely, how about you try
> the diff above and see if it fixes the issue.

Looks like it has been already fixed in [1].

1- https://lore.kernel.org/all/20261105114149.970315-1-shunsheng1218@gmail.com/


> 
> > 
> > Our main question is: can a specification-compliant controller deliver
> > a completion for a previously submitted AER during the interval in
> > which the host is disabling the controller and tearing down the admin
> > queue, or must such completions be quiesced before disable completes?
> > We would also welcome guidance on the exact point after which the host
> > may safely assume that no further AER completions can be delivered.
> 
> If the above diff fixes the issue for you, then I think this is a driver
> bug.