Re: ath11k swiotlb buffer is full (on IMX8M with 4GiB DRAM)

[Christoph Hellwig <hch@lst.de>]

On Mon, Dec 09, 2024 at 11:15:02AM -0800, Tim Harvey wrote:
> After a lot of back and forth and investigation this is due to the
> IMX8M SoC's not having an IOMMU thus swiotlb is being used and ath11k
> is requesting some buffers that are too large for swiotlb to provide.
> There is a specific patch which added the HAL_WBM2SW_RELEASE buffers
> to cacheable memory that could be reverted to fix this but the concern
> was that it would impact performance moving those buffers to
> non-cacheable memory (there are three ~1MiB buffers being allocated):
> commit d0e2523bfa9cb ("ath11k: allocate HAL_WBM2SW_RELEASE ring from
> cacheable memory").

The combination of "buffers" and "swiotlb" sounds like Robin was right
below. 

> The chain of events as best I can tell are:
> 
> commit 6452f0a3d565 ("ath11k: allocate dst ring descriptors from
> cacheable memory")
> - Nov 12 2021 (made it into Linux 5.17)
> - changes allocation of reo_dst rings to cacheable memory to allow
> cached descriptor access to optimize CPU usage
> - this is flawed because it uses virt_to_phys() to allocate cacheable
> memory which does not work on systems with an IOMMU enabled or using
> software IOMMU (swiotlb); this causes a kernel crash on client
> association

And this is where it started to take a wrong turn, that everyhing
later basically made worse.  If you have long living and potentially
large DMA allocations, you need to use dma_alloc_* interfaces.

5.17 already had dma_alloc_pages for quite a while which was and still is
the proper interface to use.  For much older kernel you'd be stuck
with dma_alloc_noncoherent or dma_alloc_attrs with the right flag,
but even that would have been much better.