Re: [PATCH v9 25/25] docs: vfio: Add vfio device cdev description

[Alex Williamson <alex.williamson@redhat.com>]

On Wed, 5 Apr 2023 14:00:00 +0000
"Liu, Yi L" <yi.l.liu@intel.com> wrote:

> Hi Eric,
> 
> > From: Eric Auger <eric.auger@redhat.com>
> > Sent: Wednesday, April 5, 2023 9:46 PM
> > 
> > Hi Yi,
> > 
> > On 4/1/23 17:18, Yi Liu wrote:  
> > > This gives notes for userspace applications on device cdev usage.
> > >
> > > Reviewed-by: Kevin Tian <kevin.tian@intel.com>
> > > Signed-off-by: Yi Liu <yi.l.liu@intel.com>
> > > ---
> > >  Documentation/driver-api/vfio.rst | 132 ++++++++++++++++++++++++++++++
> > >  1 file changed, 132 insertions(+)
> > >
> > > diff --git a/Documentation/driver-api/vfio.rst b/Documentation/driver-api/vfio.rst
> > > index 363e12c90b87..4f21be7bda8a 100644
> > > --- a/Documentation/driver-api/vfio.rst
> > > +++ b/Documentation/driver-api/vfio.rst
> > > @@ -239,6 +239,130 @@ group and can access them as follows::
> > >  	/* Gratuitous device reset and go... */
> > >  	ioctl(device, VFIO_DEVICE_RESET);
> > >
> > > +IOMMUFD and vfio_iommu_type1
> > > +----------------------------
> > > +
> > > +IOMMUFD is the new user API to manage I/O page tables from userspace.
> > > +It intends to be the portal of delivering advanced userspace DMA
> > > +features (nested translation [5], PASID [6], etc.) while also providing
> > > +a backwards compatibility interface for existing VFIO_TYPE1v2_IOMMU use
> > > +cases.  Eventually the vfio_iommu_type1 driver, as well as the legacy
> > > +vfio container and group model is intended to be deprecated.
> > > +
> > > +The IOMMUFD backwards compatibility interface can be enabled two ways.
> > > +In the first method, the kernel can be configured with
> > > +CONFIG_IOMMUFD_VFIO_CONTAINER, in which case the IOMMUFD subsystem
> > > +transparently provides the entire infrastructure for the VFIO
> > > +container and IOMMU backend interfaces.  The compatibility mode can
> > > +also be accessed if the VFIO container interface, ie. /dev/vfio/vfio is
> > > +simply symlink'd to /dev/iommu.  Note that at the time of writing, the
> > > +compatibility mode is not entirely feature complete relative to
> > > +VFIO_TYPE1v2_IOMMU (ex. DMA mapping MMIO) and does not attempt to
> > > +provide compatibility to the VFIO_SPAPR_TCE_IOMMU interface.  Therefore
> > > +it is not generally advisable at this time to switch from native VFIO
> > > +implementations to the IOMMUFD compatibility interfaces.
> > > +
> > > +Long term, VFIO users should migrate to device access through the cdev
> > > +interface described below, and native access through the IOMMUFD
> > > +provided interfaces.
> > > +
> > > +VFIO Device cdev
> > > +----------------
> > > +
> > > +Traditionally user acquires a device fd via VFIO_GROUP_GET_DEVICE_FD
> > > +in a VFIO group.
> > > +
> > > +With CONFIG_VFIO_DEVICE_CDEV=y the user can now acquire a device fd
> > > +by directly opening a character device /dev/vfio/devices/vfioX where
> > > +"X" is the number allocated uniquely by VFIO for registered devices.
> > > +For noiommu devices, the character device would be named with "noiommu-"
> > > +prefix. e.g. /dev/vfio/devices/noiommu-vfioX.
> > > +
> > > +The cdev only works with IOMMUFD.  Both VFIO drivers and applications
> > > +must adapt to the new cdev security model which requires using
> > > +VFIO_DEVICE_BIND_IOMMUFD to claim DMA ownership before starting to
> > > +actually use the device.  Once BIND succeeds then a VFIO device can
> > > +be fully accessed by the user.
> > > +
> > > +VFIO device cdev doesn't rely on VFIO group/container/iommu drivers.
> > > +Hence those modules can be fully compiled out in an environment
> > > +where no legacy VFIO application exists.
> > > +
> > > +So far SPAPR does not support IOMMUFD yet.  So it cannot support device
> > > +cdev neither.
> > > +
> > > +Device cdev Example
> > > +-------------------
> > > +
> > > +Assume user wants to access PCI device 0000:6a:01.0::
> > > +
> > > +	$ ls /sys/bus/pci/devices/0000:6a:01.0/vfio-dev/
> > > +	vfio0
> > > +
> > > +This device is therefore represented as vfio0.  The user can verify
> > > +its existence::
> > > +
> > > +	$ ls -l /dev/vfio/devices/vfio0
> > > +	crw------- 1 root root 511, 0 Feb 16 01:22 /dev/vfio/devices/vfio0
> > > +	$ cat /sys/bus/pci/devices/0000:6a:01.0/vfio-dev/vfio0/dev  
> > you mentionned in the pci hot reset series that the BDF couldn't be used
> > if cdev is being used. According to the above, it could, no?  
> 
> It should be the device passing case, otherwise, BDF can be used. But
> from kernel p.o.v., it has no idea how user gets the device fd, so it needs
> to assume user may not have BDF knowledge. 
> 
> > > +	511:0
> > > +	$ ls -l /dev/char/511\:0
> > > +	lrwxrwxrwx 1 root root 21 Feb 16 01:22 /dev/char/511:0 -
> > > ../vfio/devices/vfio0
> > > +
> > > +Then provide the user with access to the device if unprivileged
> > > +operation is desired::
> > > +
> > > +	$ chown user:user /dev/vfio/devices/vfio0
> > > +
> > > +Finally the user could get cdev fd by::
> > > +
> > > +	cdev_fd = open("/dev/vfio/devices/vfio0", O_RDWR);
> > > +
> > > +An opened cdev_fd doesn't give the user any permission of accessing
> > > +the device except binding the cdev_fd to an iommufd.  After that point
> > > +then the device is fully accessible including attaching it to an
> > > +IOMMUFD IOAS/HWPT to enable userspace DMA::
> > > +
> > > +	struct vfio_device_bind_iommufd bind = {
> > > +		.argsz = sizeof(bind),
> > > +		.flags = 0,
> > > +	};
> > > +	struct iommu_ioas_alloc alloc_data  = {
> > > +		.size = sizeof(alloc_data),
> > > +		.flags = 0,
> > > +	};
> > > +	struct vfio_device_attach_iommufd_pt attach_data = {
> > > +		.argsz = sizeof(attach_data),
> > > +		.flags = 0,
> > > +	};
> > > +	struct iommu_ioas_map map = {
> > > +		.size = sizeof(map),
> > > +		.flags = IOMMU_IOAS_MAP_READABLE |
> > > +			 IOMMU_IOAS_MAP_WRITEABLE |
> > > +			 IOMMU_IOAS_MAP_FIXED_IOVA,
> > > +		.__reserved = 0,
> > > +	};
> > > +
> > > +	iommufd = open("/dev/iommu", O_RDWR);
> > > +
> > > +	bind.iommufd = iommufd; // negative value means vfio-noiommu mode
> > > +	ioctl(cdev_fd, VFIO_DEVICE_BIND_IOMMUFD, &bind);
> > > +
> > > +	ioctl(iommufd, IOMMU_IOAS_ALLOC, &alloc_data);
> > > +	attach_data.pt_id = alloc_data.out_ioas_id;
> > > +	ioctl(cdev_fd, VFIO_DEVICE_ATTACH_IOMMUFD_PT, &attach_data);
> > > +
> > > +	/* Allocate some space and setup a DMA mapping */
> > > +	map.user_va = (int64_t)mmap(0, 1024 * 1024, PROT_READ | PROT_WRITE,
> > > +				    MAP_PRIVATE | MAP_ANONYMOUS, 0, 0);
> > > +	map.iova = 0; /* 1MB starting at 0x0 from device view */
> > > +	map.length = 1024 * 1024;
> > > +	map.ioas_id = alloc_data.out_ioas_id;;
> > > +
> > > +	ioctl(iommufd, IOMMU_IOAS_MAP, &map);
> > > +
> > > +	/* Other device operations as stated in "VFIO Usage Example" */  
> > dev_id is not mentioned anywhere whereas it is used in the reset get
> > info. Worth to be mentioned?  
> 
> a good point. but it is just for PCI devices, not sure if it is good to
> mention it here. or the kdoc in the HOT_RESET_INFO ioctl is already
> enough. Alex, how about your taste?.

dev-id is just for PCI devices??  Maybe it's only a vfio-pci ioctl
that's requiring it for input, but it's not a bus specific feature.
There should be a description of how an iommufd property of the device
maps to this subsystem.  Thanks,

Alex