Re: [PATCH v8 5/6] vfio: Enable cdev noiommu mode under iommufd

[Jacob Pan <jacob.pan@linux.microsoft.com>]

Hi Alex,

On Mon, 8 Jun 2026 17:19:56 -0600
Alex Williamson <alex@shazbot.org> wrote:

> From: Alex Williamson <alex@shazbot.org>
> To: Jacob Pan <jacob.pan@linux.microsoft.com>
> Cc: linux-kernel@vger.kernel.org, "iommu@lists.linux.dev"
> <iommu@lists.linux.dev>, Jason Gunthorpe <jgg@nvidia.com>, Joerg
> Roedel  <joro@8bytes.org>, Mostafa Saleh <smostafa@google.com>, David
> Matlack  <dmatlack@google.com>, Robin Murphy <robin.murphy@arm.com>,
> Nicolin Chen  <nicolinc@nvidia.com>, "Tian, Kevin"
> <kevin.tian@intel.com>, Yi Liu  <yi.l.liu@intel.com>, Baolu Lu
> <baolu.lu@linux.intel.com>, Saurabh Sengar
> <ssengar@linux.microsoft.com>, skhawaja@google.com,
> pasha.tatashin@soleen.com, Will Deacon <will@kernel.org>,
> alex@shazbot.org Subject: Re: [PATCH v8 5/6] vfio: Enable cdev
> noiommu mode under iommufd Date: Mon, 8 Jun 2026 17:19:56 -0600
> X-Mailer: Claws Mail 4.4.0 (GTK 3.24.52; x86_64-pc-linux-gnu)
> 
> On Wed,  3 Jun 2026 15:02:10 -0700
> Jacob Pan <jacob.pan@linux.microsoft.com> wrote:
> 
> > Now that devices under noiommu mode can bind with IOMMUFD and
> > perform IOAS operations, lift restrictions on cdev from VFIO side.
> > Use cases are documented in Documentation/driver-api/vfio.rst
> > 
> > Reviewed-by: Kevin Tian <kevin.tian@intel.com>
> > Signed-off-by: Jacob Pan <jacob.pan@linux.microsoft.com>
> > ---
> > v8:
> >   - Fix warning message (Kevin)
> > v7:
> >   - Avoid treating emulated device as noiommu device (Sashiko)
> >   - Keep platforms w/ GENERIC_ATOMIC64 to use VFIO group noiommu as
> >     before (Sashiko)
> >   - Restore order of group & cdev init for noiommu (Yi)
> >   - Consolidate noiommu helper for cdev & group (Yi)
> > v6:
> >   - Revert back to unified VFIO_NOIOMMU Kconfig for both cdev and
> > group. Use Kconfig dependency to restrict usages and avoid null
> > group checks. (Alex & Yi)
> >   - Add CAP_SYS_RAWIO checks for cdev open to maintain security
> > parity with the group noiommu path. (Alex)
> > v5:
> >    - Add Kconfig VFIO_CDEV_NOIOMMU to select IOMMUFD_NOIOMMU
> >      and its dependencies
> >    - Add comment to explain vfio_noiommu conditional definition
> > (Alex)
> >    - Removed early return for group noiommu in bind/unbind
> >    - Use consistent wording referring to VFIO noiommu mode (Kevin)
> >    - Update unsafe_noiommu Kconfig help text (Kevin)
> >    - Change dev_warn to dev_info for noiommu enabling msg (Kevin)
> > v4:
> >    - Remove early return in iommufd_bind for noiommu (Alex)
> > v3:
> >    - Consolidate into fewer patches
> > v2:
> >    - removed unnecessary device->noiommu set in
> >      iommufd_vfio_compat_ioas_get_id()
> > 
> > ---
> >  drivers/vfio/Kconfig       |  7 ++++---
> >  drivers/vfio/device_cdev.c |  3 +++
> >  drivers/vfio/iommufd.c     | 12 ++++++++----
> >  drivers/vfio/vfio.h        | 23 +++++++++--------------
> >  drivers/vfio/vfio_main.c   | 26 +++++++++++++++++++++++++-
> >  include/linux/vfio.h       |  1 +
> >  6 files changed, 50 insertions(+), 22 deletions(-)
> > 
> > diff --git a/drivers/vfio/Kconfig b/drivers/vfio/Kconfig
> > index ceae52fd7586..b9d6e1c22aed 100644
> > --- a/drivers/vfio/Kconfig
> > +++ b/drivers/vfio/Kconfig
> > @@ -22,8 +22,7 @@ config VFIO_DEVICE_CDEV
> >  	  The VFIO device cdev is another way for userspace to get
> > device access. Userspace gets device fd by opening device cdev under
> >  	  /dev/vfio/devices/vfioX, and then bind the device fd
> > with an iommufd
> > -	  to set up secure DMA context for device access.  This
> > interface does
> > -	  not support noiommu.
> > +	  to set up secure DMA context for device access.
> >  
> >  	  If you don't know what to do here, say N.
> >  
> > @@ -62,7 +61,9 @@ endif
> >  
> >  config VFIO_NOIOMMU
> >  	bool "VFIO No-IOMMU support"
> > -	depends on VFIO_GROUP
> > +	depends on VFIO_GROUP || (VFIO_DEVICE_CDEV &&
> > !GENERIC_ATOMIC64)
> > +	depends on !VFIO_GROUP || VFIO_CONTAINER ||
> > IOMMUFD_VFIO_CONTAINER
> > +	select IOMMUFD_NOIOMMU if VFIO_DEVICE_CDEV &&
> > !GENERIC_ATOMIC64  
> 
> Sashiko is warning about this and it seems real, if the config were
> something like this:
> 
>   CONFIG_GENERIC_ATOMIC64=y
>   CONFIG_VFIO=y
>   CONFIG_VFIO_GROUP=y
>   CONFIG_VFIO_CONTAINER=y
>   CONFIG_VFIO_DEVICE_CDEV=y
> 
> The result is:
> 
>   # => CONFIG_VFIO_NOIOMMU=y
>   # => CONFIG_IOMMUFD_NOIOMMU is not set
> 
> Which can result in:
> 
>   /dev/vfio/
>   ├── devices/
>   │   └── vfio0
>   └── noiommu-0
> 
> The cdev exists without the noiommu- prefix.
> 
Indeed, I thought about this which is why I put this comment in the code
"There cannot be a combination of a plain vfio%d cdev name and
a no-IOMMU group because VFIO_NOIOMMU selects IOMMUFD_NOIOMMU."
But I missed the select logic.

> Something like this might work
> 
>    config VFIO_NOIOMMU
>    	bool "VFIO No-IOMMU support"
>    	depends on VFIO_GROUP || (VFIO_DEVICE_CDEV &&
> !GENERIC_ATOMIC64)
>   +	depends on !VFIO_DEVICE_CDEV || !GENERIC_ATOMIC64
>    	depends on !VFIO_GROUP || VFIO_CONTAINER ||
> IOMMUFD_VFIO_CONTAINER
>   -	select IOMMUFD_NOIOMMU if VFIO_DEVICE_CDEV &&
> !GENERIC_ATOMIC64
>   +	select IOMMUFD_NOIOMMU if VFIO_DEVICE_CDEV
>    	help
>    	  VFIO is built on the ability to isolate devices using
> the IOMMU.
> 

This will work, but it disables VFIO_NOIOMMU for configs with
VFIO_DEVICE_CDEV=y and GENERIC_ATOMIC64=y, even though the legacy group
noiommu path still works there. That can break existing distro configs
which enable both VFIO_GROUP and VFIO_DEVICE_CDEV, right?

How about add code change to skip noiommu cdev registeration if
IOMMUFD_NOIOMMU is not enabled? i.e.
--- a/drivers/vfio/vfio.h
+++ b/drivers/vfio/vfio.h
@@ -359,13 +359,21 @@ void vfio_init_device_cdev(struct vfio_device
*device);

 static inline int vfio_device_add(struct vfio_device *device)
 {
+       if (vfio_device_is_noiommu(device) &&
+           !IS_ENABLED(CONFIG_IOMMUFD_NOIOMMU))
+               return device_add(&device->device);
+
        vfio_init_device_cdev(device);
        return cdev_device_add(&device->cdev, &device->device);
 }

 static inline void vfio_device_del(struct vfio_device *device)
 {
-       cdev_device_del(&device->cdev, &device->device);
+       if (vfio_device_is_noiommu(device) &&
+           !IS_ENABLED(CONFIG_IOMMUFD_NOIOMMU))
+               device_del(&device->device);
+       else
+               cdev_device_del(&device->cdev, &device->device);
 }
I will also update the documentation to state this behavior:

"The cdev noiommu path requires CONFIG_GENERIC_ATOMIC64=n.  When
CONFIG_VFIO_GROUP=y, CONFIG_VFIO_DEVICE_CDEV=y, and
CONFIG_GENERIC_ATOMIC64=y, CONFIG_VFIO_NOIOMMU remains selectable for
the group path, but no noiommu device cdev is registered.  Cdev-only
noiommu is not selectable on those platforms."

> >  	help
> >  	  VFIO is built on the ability to isolate devices using
> > the IOMMU. Only with an IOMMU can userspace access to DMA capable
> > devices be diff --git a/drivers/vfio/device_cdev.c
> > b/drivers/vfio/device_cdev.c index 54abf312cf04..5ca14979b56e 100644
> > --- a/drivers/vfio/device_cdev.c
> > +++ b/drivers/vfio/device_cdev.c
> > @@ -27,6 +27,9 @@ int vfio_device_fops_cdev_open(struct inode
> > *inode, struct file *filep) struct vfio_device_file *df;
> >  	int ret;
> >  
> > +	if (vfio_device_is_noiommu(device) &&
> > !capable(CAP_SYS_RAWIO))
> > +		return -EPERM;
> > +  
> 
> Sashiko also notes a use-after-free issue here that seems real, we
> likely need a vfio_device_try_get_registration() before with put on
> error.  Thanks,
> 
right, will move it after vfio_device_try_get_registration().
> Alex