Re: [PATCH v2 08/18] PCI/CMA: Authenticate devices on enumeration

[Dan Williams <dan.j.williams@intel.com>]

Jason Gunthorpe wrote:
> On Mon, Jul 15, 2024 at 04:37:01PM -0700, Dan Williams wrote:
> > > So from a Linux VM perspective we have a PCI device with an IOMMU,
> > > except that IOMMU flips into IDENTITY if T=0 is used.
> > > 
> > > From a driver model and DMA API this is totally nutzo :)
> > > 
> > > Being able to flip from trusted/untrusted and keep IOMMU/DMA/etc
> > > unaffected requires that the vIOMMU can always walk the same IO page
> > > tables stored in trusted VM memory, regardless if the device sends a
> > > T=0/1 TLP.
> > 
> > "Keep IOMMU/DMA/etc unaffected" is the hard part.
> 
> Yes, but that is not just "unaffected" but it is implying that there
> is state in the VM's iommu layer too. If T=0 goes to a different
> translation then the DMA API must change behavior while a driver is
> bound, which is not something we do today.
> 
> > Implementations that want something more complicated than that, like
> > interleave T=0 and T=1 traffic, need to demonstrate how that is possible
> > given the iommufd maintainer declares it, *checks notes*, "totally
> > nutzo".
> 
> Oh we can make the iommufd side work out, it is the VM's kernel that
> is going to be trouble :)
> 
> Even in the simpler case of no-interleave but the same driver will
> start with T=0 and change to T=1 is pretty complex:
> 
>  dma_addr1 = dma_map()   <== Must return a bypass address because T=0
>  goto_t_1()              <== Now dma_addr1 stops being usable
>  dma_addr2 = dma_map()   <== Must return a translated address through the vIOMMU
>  dma_unmap(dma_addr1)    <== Well now you've done it. Your kernel explodes.
> 
> Maybe the "violance" is we have to unbind the PCI driver and rebind it
> to get the goto_t_1() effect..
> 
> Changing the underlying behavior of the DMA API "in flight" while a
> driver is bound seems really dangerous.

Agree.

> My point is if we start baking in the assumption that drivers can do
> things like the above without addressing how the VIOMMU integration
> works we are going to have a *huge mess* to try and introduce VIOMMU
> down the road.
> 
> I'd be happy if V1 forbade the above entirely.

Yes, I think the requirement to go through rebind to cross the
untrusted/trusted boundary gives enough simplification to get started.

It also occurs to me that complex devices / drivers that really want
mixed T=0 and T=1 traffic from one PF can ingest the complexity without
burdening the Linux DMA API and IOMMU layers. Provide 2 assignable VFs
instead of 1 and do software driver-to-driver communication between
those trusted and untrusted drivers.