Re: [PATCH 0/1] swiotlb: Make IO_TLB_SEGSIZE Configurable

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Fri, Apr 25, 2025 at 11:42:24AM +0100, Robin Murphy wrote:
> On 2025-04-25 6:32 am, Li, Hua Qian wrote:
> > On Thu, 2025-04-24 at 13:58 +0100, Robin Murphy wrote:
> > > On 24/04/2025 6:12 am, Li, Hua Qian wrote:
> > > > On Tue, 2025-04-22 at 15:36 +0200, Marek Szyprowski wrote:
> > > > > On 22.04.2025 08:37, huaqian.li@siemens.com wrote:
> > > > > > From: Li Hua Qian <huaqian.li@siemens.com>
> > > > > > 
> > > > > > This patchset introduces a change to make the IO_TLB_SEGSIZE
> > > > > > parameter
> > > > > > configurable via a new kernel configuration option
> > > > > > (CONFIG_SWIOTLB_SEGSIZE).
> > > > > > 
> > > > > > In certain applications, the default value of IO_TLB_SEGSIZE
> > > > > > (128)
> > > > > > may
> > > > > > not be sufficient for memory allocation, leading to runtime
> > > > > > errors.
> > > > > > By
> > > > > > making this parameter configurable, users can adjust the
> > > > > > segment
> > > > > > size to
> > > > > > better suit their specific use cases, improving flexibility and
> > > > > > system
> > > > > > stability.
> > > > > 
> > > > > Could You elaborate a bit more what are those certain
> > > > > applications
> > > > > that
> > > > > require increasing IO_TLB_SEGSIZE? I'm not against it, but such
> > > > > change
> > > > > should be well justified and described, while the above cover-
> > > > > letter
> > > > > doesn't provide anything more than is written in the patch
> > > > > description.
> > > > Thank you for your feedback, Marek.
> > > > 
> > > > To provide more context, one specific application that requires
> > > > increasing IO_TLB_SEGSIZE is the Hailo 8 PCIe AI card. This card
> > > > uses
> > > > dma_alloc_coherent to allocate descriptor lists, as seen in the
> > > > Hailo
> > > > driver implementation here:
> > > > https://github.com/hailo-ai/hailort-drivers/blob/7161f9ee5918029bd4497f590003c2f87ec32507/linux/vdma/memory.c#L322
> > > > The maximum size (nslots) for these allocations can reach 160,
> > > > which
> > > > exceeds the current default value of IO_TLB_SEGSIZE (128).
> > > > 
> > > > Since IO_TLB_SEGSIZE is defined as a constant in the kernel:
> > > > 
> > > > `#define IO_TLB_SEGSIZE 128`
> > > > 
> > > > 
> > > > this limitation causes swiotlb_search_pool_area,
> > > > https://github.com/torvalds/linux/blame/v6.15-rc2/kernel/dma/swiotlb.c#L1085
> > > > ,
> > > > (or swiotlb_do_find_slots in older kernels) to fail when attempting
> > > > to
> > > > allocate contiguous physical memory (CMA). This results in runtime
> > > > errors and prevents the Hailo 8 card from functioning correctly in
> > > > certain configurations.
> > > 
> > > Hmm, dma_alloc_coherent() should really not be trying to allocate
> > > from
> > > SWIOTLB in the first place - how is that happening?
> > > 
> > > If you're using restricted DMA for a device which wants significant
> > > coherent allocations, then it wants to have it's own shared-dma-pool
> > > for
> > > those *as well* as the restricted-dma-pool for bouncing streaming
> > > DMA.
> > > 
> > > Thanks,
> > > Robin.
> > 
> > Hi Robin,
> > 
> > Regarding the specific Hailo Card case, the issue arises due
> > to the capabilities of certain SoCs or CPUs. For example, many
> > K3 SoCs lack an IOMMU, which is typically used to isolate the
> > system against DMA-based attacks of external PCI devices.
> > 
> > Taking the TI AM65 as an example, it doesn't have an IOMMU, but
> > instead includes a Peripheral Virtualization Unit (PVU). The
> > PVU provides functionality similar to an IOMMU and is used to
> > isolate PCI devices from the Linux host, and the SWIOTLB is
> > used to manp all DMA buffers from a static memory carve-out.
> 
> And as I said, if you want to support general coherent allocations then you
> should use part of that carveout for a regular coherent DMA pool. The
> restricted pool is only intended for streaming DMA - swiotlb_alloc() is only
> meant as a convenience fallback for the kind of devices which mostly do
> streaming DMA but make one or two small coherent allocations from a suitable
> context. It does not work for *all* valid usage of dma_alloc_attrs(), and if
> you want to do this for arbitrary PCI devices then you almost certainly *do*
> need to be able to support drivers which make allocations in atomic context.

If they utilize dma_alloc_coherent and setup at boot-time those buffers
then that does exactly the same thing as the coherent DMA pool. Albeit
less flexible, but nonethless the same thing.

That seems like a valid use-case, no?