Re: [PATCH 4/5] vfio/type1: Flush CPU caches on DMA pages in non-coherent domains

[Jason Gunthorpe <jgg@nvidia.com>]

On Thu, May 23, 2024 at 04:47:53PM -0600, Alex Williamson wrote:

> > > > I am suggesting to do both checks:
> > > >  - If the iommu domain indicates it has force coherency then leave PCI
> > > >    no-snoop alone and no flush
> > > >  - If the PCI NOSNOOP bit is or can be 0 then no flush
> > > >  - Otherwise flush  
> > > 
> > > How to judge whether PCI NOSNOOP can be 0? If following PCI spec
> > > it can always be set to 0 but then we break the requirement for Intel
> > > GPU. If we explicitly exempt Intel GPU in 2nd check  then what'd be
> > > the value of doing that generic check?  
> > 
> > Non-PCI environments still have this problem, and the first check does
> > help them since we don't have PCI config space there.
> > 
> > PCI can supply more information (no snoop impossible) and variant
> > drivers can add in too (want no snoop)
> 
> I'm not sure I follow either.  Since i915 doesn't set or test no-snoop
> enable, I think we need to assume drivers expect the reset value, so a
> device that supports no-snoop expects to use it, ie. we can't trap on
> no-snoop enable being set, the device is more likely to just operate
> with reduced performance if we surreptitiously clear the bit.

I'm not sure I understand this paragraph?

> The current proposal is to enable flushing based only on the domain
> enforcement of coherency.  I think the augmentation is therefore that
> if the device is PCI and the no-snoop enable bit is zero after reset
> (indicating hardwired to zero), we also don't need to flush.

Yes, that is a good additional starting point.
 
> I'm not sure the polarity of the variant drive statement above is
> correct.  If the no-snoop enable bit is set after reset, we'd assume
> no-snoop is possible, so the variant driver would only need a way to
> indicate the device doesn't actually use no-snoop.  For that it might
> just virtualize the no-snoop enable setting to vfio-pci-core.  Thanks,

I wrote that with the idea that VFIO would always force no-snoop to
0. The variant driver could opt out of this. We could also do the
reverse and leave no-snoop alone and have something force it off.

The other issue to keep in mind is that if no-snoop is disabled when
we attach the domains we shouldn't allow the VMM to turn it on later.

Jason