Re: [PATCH v6 01/10] iommufd/viommu: Add IOMMUFD_OBJ_VDEVICE and IOMMU_VDEVICE_ALLOC ioctl

[Nicolin Chen <nicolinc@nvidia.com>]

On Thu, Nov 07, 2024 at 09:11:27PM +1100, Alexey Kardashevskiy wrote:
> On 31/10/24 08:35, Nicolin Chen wrote:
> > Introduce a new IOMMUFD_OBJ_VDEVICE to represent a physical device (struct
> > device) against a vIOMMU (struct iommufd_viommu) object in a VM.
> > 
> > This vDEVICE object (and its structure) holds all the infos and attributes
> > in the VM, regarding the device related to the vIOMMU.
> > 
> > As an initial patch, add a per-vIOMMU virtual ID. This can be:
> >   - Virtual StreamID on a nested ARM SMMUv3, an index to a Stream Table
> >   - Virtual DeviceID on a nested AMD IOMMU, an index to a Device Table
> >   - Virtual RID on a nested Intel VT-D IOMMU, an index to a Context Table
> > Potentially, this vDEVICE structure would hold some vData for Confidential
> > Compute Architecture (CCA). Use this virtual ID to index an "vdevs" xarray
> > that belongs to a vIOMMU object.
> > 
> > Add a new ioctl for vDEVICE allocations. Since a vDEVICE is a connection
> > of a device object and an iommufd_viommu object, take two refcounts in the
> > ioctl handler.

> > +/**
> > + * struct iommu_vdevice_alloc - ioctl(IOMMU_VDEVICE_ALLOC)
> > + * @size: sizeof(struct iommu_vdevice_alloc)
> > + * @viommu_id: vIOMMU ID to associate with the virtual device
> > + * @dev_id: The physical device to allocate a virtual instance on the vIOMMU
> > + * @out_vdevice_id: Object handle for the vDevice. Pass to IOMMU_DESTORY
> > + * @virt_id: Virtual device ID per vIOMMU, e.g. vSID of ARM SMMUv3, vDeviceID
> > + *           of AMD IOMMU, and vRID of a nested Intel VT-d to a Context Table
> 
> 
> So it is one vdevice per a passed through device (say, a network
> adapter), right?

Yes. It's per iommufd_device per iommufd_viommu.

> I am asking as there are passed through devices and
> IOMMU devices, and (at least on AMD) IOMMUs look like PCI devices, both
> in hosts and guests. For example, from the above: "@dev_id: The physical
> device ..." - both a network card and IOMMU are physical, so dev_id is a
> NIC or IOMMU? I assume that шы a NIC (but it is a source of constant
> confusion).

In that case, dev_id is NIC. viommu_id is IOMMU.

First VMM should allocate a vIOMMU using the dev_id (NIC) to get
a viommu_id, and then use this viommu_id and dev_id to allocate
a vDEVICE.

It might sound duplicated in this case because this AMD IOMMU is
exclusive for the NIC. But ARM/Intel can be shared among devices
so they can allocate a vIOMMU with device1 and allocate vDEVICEs
for device1, device2, device3, and so on.

> Is there any plan to add guest device BDFn as well, or I can add one
> here for my TEE-IO exercise, if it is the right place? It is the same as
> vDeviceID for AMD but I am not sure about the others, hence the
> question. Thanks,

Generally speaking, adding vRID isn't a problem so long as there
is a legit reason/usecase. That being said, if it is the same as
the @virt_id for AMD, why not just pass via @virt_id v.s. adding
a new vRID/vBDF field?

Thanks
Nicolin