Re: [TECH TOPIC] vfio, iommufd: Enabling user space drivers to vend more granular access to client processes

[Alex Mastro <amastro@fb.com>]

On Mon, Sep 22, 2025 at 09:14:24AM +0000, Mostafa Saleh wrote:
> On Fri, Sep 19, 2025 at 07:00:04AM +0000, Tian, Kevin wrote:
> > the proposal includes two parts: BAR access and IOMMU mapping. For
> > the latter looks the intention is more around releasing resource. But
> > the former sounds more like a security enhancement - instead of
> > granting the client full access to the entire device it aims to expose
> > only a region of BAR resource necessary into guest. Then as Jason
> > questioned what is the value of doing so when one client can program
> > arbitrary DMA address into the exposed BAR region to attack mapped
> > memory of other clients and the USD... there is no hw isolation 
> > within a partitioned IOAS unless the device supports PASID then 
> > each client can be associated to its own IOAS space.
> 
> That’s also my opinion, it seems that PASIDs are not supported in
> that case, that’s why the clients share the same IOVA address space,
> instead of each one having their own.

Yes, we do have cases where PASID is not supported by our hardware.

> In that case I think as all of this is cooperative and can’t be enforced,
> one process can corrupt another process memory that is mapped the IOMMU.

In systems lacking PASID, some degree of enforcement would still be possible via
USD and device policies. In a ~similar way to how an in-kernel driver wanting
to accomplish our same goals (enabling a client and device able to access each
other's memory directly) would presumably need to enforce this.

I have been thinking along the following lines:

Imagine that we want the client and device to communicate with each other
directly via queues in each other's memory, bypassing interaction with the
driver.

As part of granting access to a client process:
- The USD allocates some IOAS slice for the client.
- The USD prepares a restricted IOMMU fd to be shared with the client which
  only has mapping permissions to the IOAS slice.
- The USD configures the device: "DMA initiated across this region of
  client-accessible BAR is only allowed to target the client's IOAS slice."
- The USD vends the client a dma-buf exposing a view of only that client's queue
  space, along with the restricted IOMMU fd.

Given the above setup, barring bugs in the USD, or the device hardware/firmware,
it should be impossible for one client to corrupt another client's address
space, since the side-effects it is able to create by accessing its BAR slice
have been constrained by a combination of USD + device policy.

Next, we need to address revocation. The USD needs to be able revoke:
1) client access to BAR memory
2) device access to client memory

Issue (2) was touched on in the original tech topic email, but we haven't
covered (1) yet.

For (1) to be possible, I think we need to grant the VFIO user (USD in this
specific case) the ability to revoke a dma-buf in a way that prevents "peer"
access to the device -- whether the peer is some other device, or a user space
client process.

Following a dma_buf_ops.mmap, I suppose that revocation would mean:
- Poisoning the dma-buf fd to disallow the creation of additional mmaps.
- Zapping the PTEs backing existing mmaps. Subsequent access to the unmapped
  client address space should trigger page faults.

Thanks,
Alex