Re: [RFC PATCH 3/3] iommufd/tsm: Add tsm_bind/unbind iommufd ioctls

[Xu Yilun <yilun.xu@linux.intel.com>]

On Thu, Jul 17, 2025 at 09:43:33AM -0300, Jason Gunthorpe wrote:
> On Thu, Jul 17, 2025 at 04:28:15PM +0800, Xu Yilun wrote:
> > > Seems to me that two DMABUFs is easier than trying to teach DMABUf
> > > about some new attribute..
> > > 
> > > Userpsace can unmap all the shared DMABUFs, do a TDI BIND, then map
> > > private DMABUFs. DMABUFs do not change from private to shared on the
> > > fly.
> > 
> > But the shareability of each MMIO pfn should still be recorded at the
> > time of TDI BIND. 
> 
> > Shareability is not only about hypervisor can map the
> > MMIO or not, it is mainly about Guest should access it in a shared or
> > private manner. According to this KVM map the pfn in EPT or S-EPT.
> 
> Yes, which is why having two dmabufs might be nice because you can
> plug the public one into the EPT and the private one into the S-EPT
> and SW can validate the right one is in the right place.
> 
> > For MMIO, the shareability layout is the TDI's inherence which can be
> > get from TDI report. I.e. some MMIOs must still be accessed as shared by
> > guest even if the device is converted to private. So I think a private
> > DMABUF without sharebility layout can't support the TDI case.
> 
> IMHO this even more strongly says two DMABUFs. After binding you'd get
> a new shared DMABUF that was limited to only the actual shared pages
> while the private DMABUF would be limited to only the actual private
> pages.
> 
> It is much simpler considering the current DMABUF APIs than trying to
> convey per-pfn shared/private indication.

I generally think the flow you describe is good for implementation.
But still some details, see below.

> 
> > The existing shareability layout for all guest memory space is recorded
> > in KVM via KVM_SET_MEMORY_ATTRIBUTES, but now Sean's suggestion is
> > giving it back to resource owners when in-place conversion makes
> > guest_memfd fully aware of the shareability layout.
> 
> I would say this discussion is irrelevant to this case since we are
> not doing any kind of in-place conversion.
> 
> 1) At VM start the VMM gets the shared DMABUF and maps it to IOAS and
>    EPT
> 2) Bind request comes in, VMM unmaps shared DMABUF from IOAS and EPT
>    then closes it.
> 3) Bind is done
> 4) VMM opens a shared and private DMABUF FD and learns the valid
>    ranges in both DMABUFs (ie what PFNS are private/shared)

Who decides the dmabuf be shared or private? I assume you mean
userspace, VFIO doesn't know about the shareability of each dmabuf?

> 5) VMM maps the shared DMABUF fragments to the EPT and IOAS

Userspace could set a shared only memory slot.

> 6) VMM maps the private DMABUF fragments to the S-EPT

Userspace could set a private only memory slot, but maybe the concern
is KVM can't trust userspace about the assertion of private. Like for
private memory, KVM now verifies the slot is backed by gmem before
allowing the slot to be private. This is some in-kernel contract.
Without the contract, userspace could assign arbitrary memory to KVM as
private and explode.  If VFIO is not aware of the shareability, KVM - VFIO
can't build the contract.

If VFIO should be aware of the shareability, IOMMUFD to VFIO callback is
needed after TSM bind.

> 7) Unbind request comes in, VMM unmaps both shared and private DMABUFS
> 
> For all error cases the kernel revokes all open DMABUFS and userspace
> is expected to close & reopen them to recover.
> 
> From a KVM perspective when you tell it to map the DMABUF the VMM will
> also tell it the shared/private, which is basically implied by the
> GPA. There is no "conversion", userspace will destroy the memslot and
> create new ones.
> 
> Isn't this pretty simple?

It is simpler on kernel side. FYR, may introduce some complexity in
userspace, this may result in 3+ dmabufs. There may be shared holes
in the BAR, and kvm slot can't be gfn sparse. This may not be a big
problem.

Thanks,
Yilun

> 
> Jason