Re: [PATCH v2 08/18] PCI/CMA: Authenticate devices on enumeration

[Dan Williams <dan.j.williams@intel.com>]

Jason Gunthorpe wrote:
> On Mon, Jul 15, 2024 at 03:50:28PM -0700, Dan Williams wrote:
> > > > The motivation for the security policy is "there is trusted memory to
> > > > protect". Absent trusted memory, the status quo for the device-driver
> > > > model applies.
> > > 
> > > From what I can see on some platforms/configurations if the device is
> > > trusted capable then it MUST only issue trusted DMA as that is the
> > > only IO translation that will work.
> > 
> > Given that PCI defines that devices can fall out of "trusted capable"
> > mode that implies there needs to be an error recovery path.
> 
> Sure, but this not the issue, if you stop being trusted you have to
> immediately stop doing all DMA and the VM has to restore things back
> to trusted before starting the DMAs again. Basically I'd expect you
> have to FLR the device and start from scratch as an error recovery.
> 
> > For at least the platforms I am looking at (SEV, TDX, COVE) a
> > "convert device to private operation" step is a possibility after
> > the TVM is already running. 
> 
> That's fine, too
> 
> The issue is the DMA. When you have a trusted vIOMMU present in the VM
> things get complex.
> 
> At least one platform splits the IOMMU in half and PCIE TLP bit T=0
> and T=1 target totally different translation.

I am not aware of an IOMMU implementation that does anything different
than that.

> So from a Linux VM perspective we have a PCI device with an IOMMU,
> except that IOMMU flips into IDENTITY if T=0 is used.
> 
> From a driver model and DMA API this is totally nutzo :)
> 
> Being able to flip from trusted/untrusted and keep IOMMU/DMA/etc
> unaffected requires that the vIOMMU can always walk the same IO page
> tables stored in trusted VM memory, regardless if the device sends a
> T=0/1 TLP.

"Keep IOMMU/DMA/etc unaffected" is the hard part. To start I think the
assigned device needs to go through some violence to transition security
states and should likely assume that any untrusted memory is
inaccessible once the device is converted to private operation.

Once it falls out of private operation it needs some recovery to get its
untrusted mappings repaired / restored.

Implementations that want something more complicated than that, like
interleave T=0 and T=1 traffic, need to demonstrate how that is possible
given the iommufd maintainer declares it, *checks notes*, "totally
nutzo".

> IOW the secure trusted vIOMMU must be able to support non-trusted
> devices as well.
> 
> So.. How many platforms actually did that? And how many said that only
> T=1 goes the secure VIOMMU and T=0 goes to the hypervisor?
> 
> This is all much simpler if you don't have a trusted vIOMMU :)
> 
> > > And I only know in detail how the iommu works for one platform, not
> > > the others, so I don't know how prevalent these concerns are..
> > 
> > I think it is an important concern. Even if there is a dynamic "convert
> > device to private" capability, there is a question about what happens to
> > ongoing page conversions. Simultaneous untrusted / trusted memory access
> > may end up being something devices want, but not all host platforms can
> > offer.
> 
> Maybe, but that answer will probably be unsatisfying to people who are
> building HW that assumes this works. :)

The complexity of the v1 implementation needs to be tamed first, then we
can start tilting at the higher order windmills.