[PATCH] iommu/dma: Rate-limit WARN in iommu_dma_unmap_phys()

[PATCH] iommu/dma: Rate-limit WARN in iommu_dma_unmap_phys()

[Breno Leitao]

When a PCI error (e.g. AER error or DPC containment) marks the PCI
channel as frozen or permanently failed, the IOMMU mappings for the
device may already be torn down. If a driver continues processing
completions in this state, every call to dma_unmap_page() triggers a
WARN_ON in iommu_dma_unmap_phys().

In a real-world crash scenario on an NVIDIA Grace (ARM64) platform, a
DPC event froze the PCI channel and the mlx5 NAPI poll continued
processing error CQEs, calling dma_unmap for each pending WQE. With
dozens of pending WQEs, the resulting WARN_ON storm monopolized the CPU
in softirq context for over 23 seconds, triggering a soft lockup panic.

Replace WARN_ON(!phys) with WARN_RATELIMIT() to cap the warning output
at the kernel's default rate limit (10 messages per 5 seconds), while
still providing visibility into the failure with the device name in the
message.

Signed-off-by: Breno Leitao <leitao@debian.org>
Fixes: 82612d66d51d ("iommu: Allow the dma-iommu api to use bounce buffers")
---
I initially attempted to fix this in the driver itself, but that approach
doesn't appear to be optimal, given the mappings can go away at any
time, which is impossible to check at any time. Please see the discussion at:

https://lore.kernel.org/all/20260209-mlx5_iommu-v1-1-b17ae501aeb2@debian.org/
---
 drivers/iommu/dma-iommu.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
index c92088855450a..3cb5948eafe86 100644
--- a/drivers/iommu/dma-iommu.c
+++ b/drivers/iommu/dma-iommu.c
@@ -1239,7 +1239,8 @@ void iommu_dma_unmap_phys(struct device *dev, dma_addr_t dma_handle,
 	}
 
 	phys = iommu_iova_to_phys(iommu_get_dma_domain(dev), dma_handle);
-	if (WARN_ON(!phys))
+	if (WARN_RATELIMIT(!phys, "iova_to_phys translation failed for dev %s\n",
+			   dev_name(dev)))
 		return;
 
 	if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC) && !dev_is_dma_coherent(dev))

---
base-commit: f884ff9142ee4b741a88030d77feede84f51fd4f
change-id: 20260211-dma_io_mmu-519b73988134

Best regards,
--  
Breno Leitao <leitao@debian.org>

Re: [PATCH] iommu/dma: Rate-limit WARN in iommu_dma_unmap_phys()

[Robin Murphy]

On 2026-02-11 3:13 pm, Breno Leitao wrote:
> When a PCI error (e.g. AER error or DPC containment) marks the PCI
> channel as frozen or permanently failed, the IOMMU mappings for the
> device may already be torn down. If a driver continues processing
> completions in this state, every call to dma_unmap_page() triggers a
> WARN_ON in iommu_dma_unmap_phys().

That is definitely a major bug in the caller that needs fixing rather 
than papering over. You're lucky you do have an IOMMU that needs a valid 
mapping before it can get as far as the "corrupting memory" part of 
dma_unmap_phys() being called inappropriately.

Thanks,
Robin.

> In a real-world crash scenario on an NVIDIA Grace (ARM64) platform, a
> DPC event froze the PCI channel and the mlx5 NAPI poll continued
> processing error CQEs, calling dma_unmap for each pending WQE. With
> dozens of pending WQEs, the resulting WARN_ON storm monopolized the CPU
> in softirq context for over 23 seconds, triggering a soft lockup panic.
> 
> Replace WARN_ON(!phys) with WARN_RATELIMIT() to cap the warning output
> at the kernel's default rate limit (10 messages per 5 seconds), while
> still providing visibility into the failure with the device name in the
> message.
> 
> Signed-off-by: Breno Leitao <leitao@debian.org>
> Fixes: 82612d66d51d ("iommu: Allow the dma-iommu api to use bounce buffers")
> ---
> I initially attempted to fix this in the driver itself, but that approach
> doesn't appear to be optimal, given the mappings can go away at any
> time, which is impossible to check at any time. Please see the discussion at:
> 
> https://lore.kernel.org/all/20260209-mlx5_iommu-v1-1-b17ae501aeb2@debian.org/
> ---
>   drivers/iommu/dma-iommu.c | 3 ++-
>   1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
> index c92088855450a..3cb5948eafe86 100644
> --- a/drivers/iommu/dma-iommu.c
> +++ b/drivers/iommu/dma-iommu.c
> @@ -1239,7 +1239,8 @@ void iommu_dma_unmap_phys(struct device *dev, dma_addr_t dma_handle,
>   	}
>   
>   	phys = iommu_iova_to_phys(iommu_get_dma_domain(dev), dma_handle);
> -	if (WARN_ON(!phys))
> +	if (WARN_RATELIMIT(!phys, "iova_to_phys translation failed for dev %s\n",
> +			   dev_name(dev)))
>   		return;
>   
>   	if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC) && !dev_is_dma_coherent(dev))
> 
> ---
> base-commit: f884ff9142ee4b741a88030d77feede84f51fd4f
> change-id: 20260211-dma_io_mmu-519b73988134
> 
> Best regards,
> --
> Breno Leitao <leitao@debian.org>
> 

Re: [PATCH] iommu/dma: Rate-limit WARN in iommu_dma_unmap_phys()

[Leon Romanovsky]

On Wed, Feb 11, 2026 at 07:13:03AM -0800, Breno Leitao wrote:
> When a PCI error (e.g. AER error or DPC containment) marks the PCI
> channel as frozen or permanently failed, the IOMMU mappings for the
> device may already be torn down. If a driver continues processing
> completions in this state, every call to dma_unmap_page() triggers a
> WARN_ON in iommu_dma_unmap_phys().
> 
> In a real-world crash scenario on an NVIDIA Grace (ARM64) platform, a
> DPC event froze the PCI channel and the mlx5 NAPI poll continued
> processing error CQEs, calling dma_unmap for each pending WQE. With
> dozens of pending WQEs, the resulting WARN_ON storm monopolized the CPU
> in softirq context for over 23 seconds, triggering a soft lockup panic.
> 
> Replace WARN_ON(!phys) with WARN_RATELIMIT() to cap the warning output
> at the kernel's default rate limit (10 messages per 5 seconds), while
> still providing visibility into the failure with the device name in the
> message.
> 
> Signed-off-by: Breno Leitao <leitao@debian.org>
> Fixes: 82612d66d51d ("iommu: Allow the dma-iommu api to use bounce buffers")
> ---
> I initially attempted to fix this in the driver itself, but that approach
> doesn't appear to be optimal, given the mappings can go away at any
> time, which is impossible to check at any time. Please see the discussion at:
> 
> https://lore.kernel.org/all/20260209-mlx5_iommu-v1-1-b17ae501aeb2@debian.org/
> ---
>  drivers/iommu/dma-iommu.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)

We have similar failure in our regression and the proposal fix is below,
can you please try if it fixes your issue too?

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/reporter_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en/reporter_tx.c
index 9e2cf191ed30..ac64a64e0565 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/reporter_tx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/reporter_tx.c
@@ -44,7 +44,6 @@ static void mlx5e_reset_txqsq_cc_pc(struct mlx5e_txqsq *sq)
                  "SQ 0x%x: cc (0x%x) != pc (0x%x)\n",
                  sq->sqn, sq->cc, sq->pc);
        sq->cc = 0;
-       sq->dma_fifo_cc = 0;
        sq->pc = 0;
 }

Re: [PATCH] iommu/dma: Rate-limit WARN in iommu_dma_unmap_phys()

[Breno Leitao]

hello Leon,

On Fri, Feb 13, 2026 at 01:23:55PM +0200, Leon Romanovsky wrote:
> On Wed, Feb 11, 2026 at 07:13:03AM -0800, Breno Leitao wrote:
> > When a PCI error (e.g. AER error or DPC containment) marks the PCI
> > channel as frozen or permanently failed, the IOMMU mappings for the
> > device may already be torn down. If a driver continues processing
> > completions in this state, every call to dma_unmap_page() triggers a
> > WARN_ON in iommu_dma_unmap_phys().
> > 
> > In a real-world crash scenario on an NVIDIA Grace (ARM64) platform, a
> > DPC event froze the PCI channel and the mlx5 NAPI poll continued
> > processing error CQEs, calling dma_unmap for each pending WQE. With
> > dozens of pending WQEs, the resulting WARN_ON storm monopolized the CPU
> > in softirq context for over 23 seconds, triggering a soft lockup panic.
> > 
> > Replace WARN_ON(!phys) with WARN_RATELIMIT() to cap the warning output
> > at the kernel's default rate limit (10 messages per 5 seconds), while
> > still providing visibility into the failure with the device name in the
> > message.
> > 
> > Signed-off-by: Breno Leitao <leitao@debian.org>
> > Fixes: 82612d66d51d ("iommu: Allow the dma-iommu api to use bounce buffers")
> > ---
> > I initially attempted to fix this in the driver itself, but that approach
> > doesn't appear to be optimal, given the mappings can go away at any
> > time, which is impossible to check at any time. Please see the discussion at:
> > 
> > https://lore.kernel.org/all/20260209-mlx5_iommu-v1-1-b17ae501aeb2@debian.org/
> > ---
> >  drivers/iommu/dma-iommu.c | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> We have similar failure in our regression and the proposal fix is below,
> can you please try if it fixes your issue too?

This is not a trivial test to run, but, the early tested showed some good
results.

I will report back if I find regressions later,

Thanks for the fix,
--breno