Re: [PATCH] dma-direct: Set SG_DMA_SWIOTLB flag for dma-direct

[Christoph Hellwig <hch@lst.de>]

On Thu, May 09, 2024 at 08:49:40AM +0100, Catalin Marinas wrote:
> I see the swiotlb use as some internal detail of the DMA API
> implementation that should not leak outside this framework.

And that's what it is.

> I think we should prevent bouncing if DMA_ATTR_SKIP_CPU_SYNC is passed.
> However, this is not sufficient with a proper use of the DMA API since
> the first dma_map_*() without this attribute can still do the bouncing.
> IMHO what we need is a DMA_ATTR_NO_BOUNCE or DMA_ATTR_SHARED that will
> be used on the first map and potentially on subsequent calls in
> combination with DMA_ATTR_SKIP_CPU_SYNC (though we could use the latter
> to imply "shared"). The downside is that mapping may fail if the
> coherent mask is too narrow.

We have two big problems here that kinda interact:

 1) DMA_ATTR_SKIP_CPU_SYNC is just a horrible API.  It exposes an
    implementation detail instead of dealing with use cases.
    The original one IIRC was to deal with networking receive
    buffers that are often only partially filled and the networking
    folks wanted to avoid the overhead for doing the cache operations
    for the rest.  It kinda works for that but already gets iffy
    when swiotlb is involved.  The other abuses of the flag just
    went downhill form there.

 2) the model of dma mapping a single chunk of memory to multiple
    devices is not really well accounted for in the DMA API.

So for two we need a memory allocator that can take the constraints
of multiple devices into account, and probably a way to fail a
dma-buf attach when the importer can't address the memory.
We also then need to come up with a memory ownership / cache
maintenance protocol that works for this use case.