Re: [PATCH v5 0/4] vfio: type1: support for ARM SMMUS with VFIO_IOMMU_TYPE1

[Alex Williamson <alex.williamson-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>]

On Thu, 2015-03-05 at 18:34 +0100, Eric Auger wrote:
> Hi All,
> 
> Ironically, since the correction of the IOMMU_CAP_CACHE_COHERENCY bug
> (https://lkml.org/lkml/2015/1/29/514) in vfio_iommu_type1.c, my Calxeda
> Midway VFIO use case is not working anymore. This is also observable
> when I do not apply at all the whole [PATCH v5 0/4] vfio: type1: support
> for ARM SMMUS with VFIO_IOMMU_TYPE1 series.
> 
> My understanding is this series should not be requested for me since my
> xgmac device does not care about the XN attribute.
> 
> My understanding is that without the bug or the series, the IOMMU_CACHE
> flag is set whereas before it was not. Patching vfio_iommu_type1.c
> vfio_iommu_type1_attach_group and unsetting IOMMU_CAP_CACHE_COHERENCY
> makes the use case functional again.
> 
> I do not understand the exact semantic of IOMMU_CAP_CACHE_COHERENCY. I
> see the arm smmu always returns true, and if my understanding is correct
> the vfio_iommu_type1 sets the IOMMU_CACHE attribute which eventually
> make the ARM SMMU sets the memory attribute to cacheable in the PTE. But
> naively I would say the interco used in Midway may not be cache coherent
> capability (ARM ACE, ACE-lite), hence the issue.
> 
> Please could you share your knowledge/understanding about this topic.
> Adding Will in to ;-)

This patch series should change nothing about how the IOMMU_CACHE
mapping flag is used, it was a bug in the enum to bitmap translation
that broke/fixed your use case.  VFIO type1 IOMMU always uses
IOMMU_CACHE when available.  On x86 the effect is that DMA is coherent
with the processor cache (ie. the PCIe NoSnoop attribute is ignored).
This means that KVM can continue to ignore certain cache operations,
like writeback-invalidate (WBINVD), that would otherwise need to be
emulated.  We may need a separate flag and capability if ARM SMMU is
doing something different with the flag.  Thanks,

Alex