Re: [PATCH v2 06/19] iommufd/viommu: Add IOMMU_VIOMMU_SET/UNSET_VDEV_ID ioctl

[Jason Gunthorpe <jgg@nvidia.com>]

On Fri, Sep 27, 2024 at 02:22:20PM +0000, Mostafa Saleh wrote:

> > We don't support multi SID through this interface, at least in this
> > version.
> > 
> > To do multi-sid you have to inform the VM of all the different pSIDs
> > the device has and then setup the vSID/pSID translation to map them
> > all to the HW invalidation logic.
> 
> Why would the VM need to know the pSID? 

It doesn't need to know the pSID exactly, but it needs to know all the
pSIDs that exist and have them be labeled with vSIDs.

With cmdq direct assignment the VM has to issue an ATS invalidation
for each and every physical device using its vSID. There is no HW path
to handle a 1:N v/p SID relationship.

> Ah, I thought IOMMUFD would be used instead of VFIO_TYPE1*, which should cover
> platform devices (VFIO-platform) or am I missing something?

It does work with platform, but AFAIK nobody is interested in that so
it hasn't been any focus. Enabling multi-sid nesting, sub stream ids,
etc is some additional work I think.

But what I mean is the really hard use case for the vSID/pSID mapping
is ATS invalidation and you won't use ATS invalidation on platform so
multi-sid likely doesn't matter.

STE/CD invalidation could possibly be pushed down through the
per-domain ioctl and replicated to all domain attachments. We don't
have code in the series to do that, but it could work from a uAPI
perspective.

> If possible, can the UAPI be designed with this in mind, even if not
> implemented now?

It is reasonable to ask. I think things are extensible enough. I'd
imagine we can add a flag 'secondary ID' and then a new field
'secondary ID index' to the vdev operations when someone wants to take
this on.

Jason