Re: [PATCH v2 8/8] iommu/arm-smmu-v3: Support IOMMU_DOMAIN_NESTED

[Mostafa Saleh <smostafa@google.com>]

On Mon, Sep 02, 2024 at 09:30:22PM -0300, Jason Gunthorpe wrote:
> On Mon, Sep 02, 2024 at 09:57:45AM +0000, Mostafa Saleh wrote:
> > > > 2) Is there a reason the UAPI is designed this way?
> > > > The way I imagined this, is that userspace will pass the pointer to the CD
> > > > (+ format) not the STE (or part of it).
> > > 
> > > Yes, we need more information from the STE than just that. EATS and
> > > STALL for instance. And the cachability below. Who knows what else in
> > > the future.
> > 
> > But for example if that was extended later, how can user space know
> > which fields are allowed and which are not?
> 
> Changes the vSTE rules that require userspace being aware would have
> to be signaled in the GET_INFO answer. This is the same process no
> matter how you encode the STE bits in the structure.
> 
How? And why changing that in the future is not a problem as sanitising IDRs?

> This confirmation of kernel support would then be reflected in the
> vIDRs to the VM and the VM could know to set the extended bits.
> 
> Otherwise setting an invalidate vSTE will fail the ioctl, the VMM can
> log the event, generate an event and install an abort vSTE.
> 
> > > Overall this sort of direct transparency is how I prefer to see these
> > > kinds of iommufd HW specific interfaces designed. From a lot of
> > > experience here, arbitary marshall/unmarshall is often an
> > > antipattern :)
> > 
> > Is there any documentation for the (proposed) SMMUv3 UAPI for IOMMUFD?
> 
> Just the comments in this series?

But this is a UAPI. How can userspace implement that if it has no
documentation, and how can it be maintained if there is no clear
interface with userspace with what is expected/returned...

> 
> > I can understand reading IDRs from userspace (with some sanitation),
> > but adding some more logic to map vSTE to STE needs more care of what
> > kind of semantics are provided.
> 
> We can enhance the comment if you think it is not clear enough. It
> lists the fields the userspace should pass through.
> 
> > Also, I am working on similar interface for pKVM where we “paravirtualize”
> > the SMMU access for guests, it’s different semantics, but I hope we can
> > align that with IOMMUFD (but it’s nowhere near upstream now)
> 
> Well, if you do paravirt where you just do map/unmap calls to the
> hypervisor (ie classic virtio-iommu) then you don't need to do very
> much.

But we have a different model, with virtio-iommu, it typically presents
the device to the VM and on the backend it calls VFIO MAP/UNMAP.
Although technically we can have virtio-iommu in the hypervisor (EL2),
that is a lot of complexit and increase in the TCB of pKVM.

For pKVM, the VMM is not trusted and the hypervisor would do the map/unmap...,
but the VMM will have to configure the virtual view of the device (Mapping of
endpoints to virtual endpoints, vIRQs…), this requires a userspace interface
to query some HW info (similar to VFIO VFIO_DEVICE_GET_IRQ_INFO and then mapping
it to a GSI through KVM, but for IOMMUs)
Though, this design is very early and in progress.

> 
> If you want to do nesting, then IMHO, just present a real vSMMU. It is
> already intended to be paravirtualized and this is what the
> confidential compute people are going to be doing as well.
> 
> Otherwise I'd expect you'd get more value to align with the
> virtio-iommu nesting stuff, where they have layed out what information
> the VM needs. iommufd is not intended to be just jammed directly into
> a VM. There is an expectation that a VMM will sit there on top and
> massage things.

I haven’t been keeping up with iommufd lately, I will try to spend more
time on that in the future.
But my idea is that we would create an IOMMUFD, attach it to a device and then
through some extra IOCTLs, we can configure some “virtual” topology for it which
then relies on KVM, again this is very early, and we need to support pKVM IOMMUs
in the host first (I plan to send v2 RFC soon for that)

> 
> > I see you are talking in LPC about IOMMUFD:
> > https://lore.kernel.org/linux-iommu/0-v1-01fa10580981+1d-iommu_pt_jgg@nvidia.com/T/#m2dbb08f3bf8506a492bc7dda2de662e42371e683
> > 
> > Do you have any plans to talk about this also?
> 
> Nothing specific, this is LPC so if people in the room would like to
> use the session for that then we can talk about it. Last year the room
> wanted to talk about PASID mostly.
> 
> I haven't heard if someone is going to KVM forum to talk about
> vSMMUv3? Eric? Nicolin do you know?

I see, I won’t be in KVM forum, but I plan to attend LPC, we can discuss
further there if people are interested.

Thanks,
Mostafa

> 
> Jason