Re: [PATCH v1 0/4] vfio/iommu: Flag to allow userspace to set DMA buffers system cacheable

[Robin Murphy <robin.murphy@arm.com>]

On 27/08/2024 12:17 am, Jason Gunthorpe wrote:
> On Mon, Aug 26, 2024 at 11:04:47AM -0600, Alex Williamson wrote:
>> On Mon, 26 Aug 2024 07:16:37 +0000
>> Manoj Vishwanathan <manojvishy@google.com> wrote:
>>
>>> Hi maintainers,
>>>
>>> This RFC patch introduces the ability for userspace to control whether
>>> device (DMA) buffers are marked as cacheable, enabling them to utilize
>>> the system-level cache.
>>>
>>> The specific changes made in this patch are:
>>>
>>> * Introduce a new flag in `include/linux/iommu.h`:
>>>      * `IOMMU_SYS_CACHE` -  Indicates if the associated page should be cached in the system's cache hierarchy.
>>> * Add `VFIO_DMA_MAP_FLAG_SYS_CACHE` to `include/uapi/linux/vfio.h`:
> 
> You'll need a much better description of what this is supposed to do
> when you resend it.
> 
> IOMMU_CACHE already largely means that pages should be cached.
> 
> So I don't know what ARM's "INC_OCACHE" actually is doing. Causing
> writes to land in a cache somewhere in hierarchy? Something platform
> specific?

Kinda both - the Inner Non-Cacheable attribute means it's still 
fundamentally non-snooping and non-coherent with CPU caches, but the 
Outer Write-back Write-allocate attribute can still control allocation 
in a system cache downstream of the point of coherency if the platform 
is built with such a thing (it's not overly common).

However, as you point out, this is in direct conflict with the Inner 
Write-back Write-allocate attribute implied by the IOMMU_CACHE which 
VFIO adds in further down in vfio_iommu_map(). Plus the way it's 
actually implemented in patch #2, IOMMU_CACHE still takes precedence and 
would lead to this new value being completely ignored, so there's a lot 
which smells suspicious here...

Thanks,
Robin.

> I have no idea. By your description it sounds similar to the
> x86 data placement stuff, whatever that was called, and the more
> modern TPH approach.
> 
> Jason