Re: [PATCH v2 10/22] iommufd/viommmu: Add IOMMUFD_CMD_VCMDQ_ALLOC ioctl

[Nicolin Chen <nicolinc@nvidia.com>]

On Mon, Apr 28, 2025 at 05:42:27PM +0530, Vasant Hegde wrote:
> > +/**
> > + * struct iommu_vcmdq_alloc - ioctl(IOMMU_VCMDQ_ALLOC)
> > + * @size: sizeof(struct iommu_vcmdq_alloc)
> > + * @flags: Must be 0
> > + * @viommu_id: Virtual IOMMU ID to associate the virtual command queue with
> > + * @type: One of enum iommu_vcmdq_type
> > + * @index: The logical index to the virtual command queue per virtual IOMMU, for
> > + *         a multi-queue model
> > + * @out_vcmdq_id: The ID of the new virtual command queue
> > + * @addr: Base address of the queue memory in the guest physical address space
> 
> Sorry. I didn't get this part.
> 
> So here `addr` is command queue base address like
>  - NVIDIA's virtual command queue
>  - AMD vIOMMU's command buffer
> 
> .. and it will allocate vcmdq for each buffer type. Is that the correct
> understanding?

Yes. For AMD "vIOMMU", it needs a new type for iommufd vIOMMU:
	IOMMU_VIOMMU_TYPE_AMD_VIOMMU,

For AMD "vIOMMU" command buffer, it needs a new type too:
	IOMMU_VCMDQ_TYPE_AMD_VIOMMU, /* Kdoc it to be Command Buffer */

Then, use IOMMUFD_CMD_VIOMMU_ALLOC ioctl to allocate an vIOMMU
obj, and use IOMMUFD_CMD_VCMDQ_ALLOC ioctl(s) to allocate vCMDQ
objs.

> In case of AMD vIOMMU, buffer base address is programmed in different register
> (ex: MMIO Offset 0008h Command Buffer Base Address Register) and buffer
> enable/disable is done via different register (ex: MMIO Offset 0018h IOMMU
> Control Register). And we need to communicate both to hypervisor. Not sure this
> API can accommodate this as addr seems to be mandatory.

NVIDIA's CMDQV has all three of them too. What we do here is to
let VMM trap the buffer base address (in guest physical address
space) and forward it to kernel using this @addr. Then, kernel
will translate this @addr to host physical address space, and
program the physical address and size to the register.

> [1]
> https://www.amd.com/content/dam/amd/en/documents/processor-tech-docs/specifications/48882_IOMMU.pdf

Thanks for the doc. So, AMD has:

Command Buffer Base Address Register [MMIO Offset 0008h]
"used to program the system physical base address and size of the
 command buffer. The command buffer occupies contiguous physical
 memory starting at the programmed base address, up to the
 programmed size."
Command Buffer Head Pointer Register [MMIO Offset 2000h]
Command Buffer Tail Pointer Register [MMIO Offset 2008h]

IIUIC, AMD should do the same: VMM traps VM's Command Buffer Base
Address register when the guest kernel allocates a command buffer
by programming the VM's Command Buffer Base Address register, to
capture the guest PA and size. Then, VMM allocates a vCMDQ object
(for this command buffer) forwarding its buffer address and size
via @addr and @length to the host kernel. Then, the kernel should
translate the guest PA to host PA to program the HW.

We can see that the Head/Tail registers are in a different MMIO
page (offset by two 4K pages), which is very like NVIDIA CMDQV
that allows VMM to mmap that MMIO page of the Head/Tail registers
for guest OS to directly control the HW (i.e. VMM doesn't trap
these two registers.

When guest OS wants to issue a new command, the guest kernel can
just fill the guest command buffer at the entry that the Head
register points to, and program the Tail register (backed by an
mmap'd MMIO page), then the HW will read the programmed physical
address from the entry (Head) till the entry (Tail) in the guest
command buffer.

> > @@ -170,3 +170,97 @@ int iommufd_vdevice_alloc_ioctl(struct iommufd_ucmd *ucmd)
> >  	iommufd_put_object(ucmd->ictx, &viommu->obj);
> >  	return rc;
> >  }
> > +
> > +void iommufd_vcmdq_destroy(struct iommufd_object *obj)
> > +{
> 
> I didn't understood destroy flow in general. Can you please help me to understand:
> 
> VMM is expected to track all buffers and call this interface?  OR iommufd will
> take care of it? What happens if VM crashes ?

In a normal routine, VMM gets a vCMDQ object ID for each vCMDQ
object it allocated. So, it should track all the IDs and release
them when VM shuts down.

The iommufd core does track all the objects that belong to an
iommufd context (ictx), and automatically release them. But, it
can't resolve certain dependency on other FD, e.g. vEVENTQ and
FAULT QUEUE would return another FD that user space listens to
and must be closed properly to destroy the QUEUE object.

> > +	/* The underlying physical pages must be pinned in the IOAS */
> > +	rc = iopt_pin_pages(&viommu->hwpt->ioas->iopt, cmd->addr, cmd->length,
> > +			    pages, 0);
> 
> Why do we need this? is it not pinned already as part of vfio binding?

I think this could be clearer:
	/*
	 * The underlying physical pages must be pinned to prevent them from
	 * being unmapped (via IOMMUFD_CMD_IOAS_UNMAP) during the life cycle
	 * of the vCMDQ object.
	 */

Thanks
Nicolin