Re: [PATCH v7 16/16] PCI/TSM: wait for vdevice contexts before removing a DSM

[Jason Gunthorpe <jgg@nvidia.com>]

On Thu, Oct 29, 2026 at 02:41:55PM +0530, Aneesh Kumar K.V wrote:
> Jason Gunthorpe <jgg@nvidia.com> writes:
> 
> > On Tue, Oct 13, 2026 at 03:14:54PM +0530, Aneesh Kumar K.V wrote:
> >> Jason Gunthorpe <jgg@nvidia.com> writes:
> >> 
> >> > On Mon, Oct 12, 2026 at 03:31:17PM +0530, Aneesh Kumar K.V wrote:
> >> >
> >> >> With this series, vdevices hold a reference to their parent vIOMMU,
> >> >> which pins arm-cca-host. sysfs unbind is disabled, and the SMCCC
> >> >> provider device has no hot-unplug path.
> >> >
> >> > It probably needs some logic like vfio has to prevent unregister from
> >> > completing if there are active users. The module pin should prevent it
> >> > from ever triggering, but as a safety precaution.
> >> >
> >> 
> >> Are you suggesting that tsm_unregister() should return an error or wait,
> >> as device removal does? Returning an error would be much easier to
> >> implement by adding an unregistering flag to tsm_dev, set under
> >> pci_tsm_rwsem after checking all pf0->context_users counters.
> >
> > I didn't check it closely but the usual flow is for 'unregister' to
> > wait until everything is done using it, possibly with a shootdown
> > mechanism.
> >
> > I'm not sure we can implement a sane shootdown for iommufd's
> > consumption of the tsm, but we can at least do the wait.
> >
> 
> This will require.
> 
> @@ -116,6 +119,13 @@ struct tsm_dev {
>  	struct device dev;
>  	int id;
>  	const struct pci_tsm_ops *pci_ops;
> +	/* Serializes user acquisition against unregister */
> +	struct mutex users_lock;
> +	/* Written under users_lock, read with READ_ONCE() by bus code. */
> +	bool unregistering;
> +	/* Registration reference plus users that must finish before teardown. */
> +	refcount_t users;
> +	struct completion users_drained;
>  };
>  
> 
> When a vIOMMU pins the tsm_dev, it must also increment users. This
> requires a new users_lock, since all existing locking is in pci_tsm.

I usually call this 'registration lock' and make it a rwsem.

Holding the read side of the 'regitration lock' means the driver
cannot be unbound during the critical region.

Then ontop of that you use a refcount and completion.

refcount can only be incr'd while holding the registration lock.

Teardown decrs the refcount, holds the write side of the registration
lock and waits for the completion. Read side is always try so it bails
once the write side is taken, preventing new refcount incrs.

Most things use the registration lock, but long lived things would
transition to the refcount.

> I can add this as a separate patch, and then we can decide whether to
> keep it. I wonder whether we can avoid this complexity, since the vIOMMU
> already pins the TSM backend module and we expect unregister to be
> called only from module_exit().

It is probably good practice to have a proper lifecycle scheme that
does not only involve module refcounting. It will be easier to
understand and use properly.

Jason