Re: [PATCH v2 4/9] iommu/ioasid: Add reference couting functions

[Jacob Pan <jacob.jun.pan@linux.intel.com>]

On Tue, 25 Aug 2020 12:19:37 +0200
Jean-Philippe Brucker <jean-philippe@linaro.org> wrote:

> On Fri, Aug 21, 2020 at 09:35:13PM -0700, Jacob Pan wrote:
> > There can be multiple users of an IOASID, each user could have
> > hardware contexts associated with the IOASID. In order to align
> > lifecycles, reference counting is introduced in this patch. It is
> > expected that when an IOASID is being freed, each user will drop a
> > reference only after its context is cleared.
> > 
> > Signed-off-by: Jacob Pan <jacob.jun.pan@linux.intel.com>
> > ---
> >  drivers/iommu/ioasid.c | 113
> > +++++++++++++++++++++++++++++++++++++++++++++++++
> > include/linux/ioasid.h |   4 ++ 2 files changed, 117 insertions(+)
> > 
> > diff --git a/drivers/iommu/ioasid.c b/drivers/iommu/ioasid.c
> > index f73b3dbfc37a..5f31d63c75b1 100644
> > --- a/drivers/iommu/ioasid.c
> > +++ b/drivers/iommu/ioasid.c
> > @@ -717,6 +717,119 @@ int ioasid_set_for_each_ioasid(struct
> > ioasid_set *set, EXPORT_SYMBOL_GPL(ioasid_set_for_each_ioasid);
> >  
> >  /**
> > + * IOASID refcounting rules
> > + * - ioasid_alloc() set initial refcount to 1
> > + *
> > + * - ioasid_free() decrement and test refcount.
> > + *     If refcount is 0, ioasid will be freed. Deleted from the
> > system-wide
> > + *     xarray as well as per set xarray. The IOASID will be
> > returned to the
> > + *     pool and available for new allocations.
> > + *
> > + *     If recount is non-zero, mark IOASID as
> > IOASID_STATE_FREE_PENDING.
> > + *     No new reference can be added. The IOASID is not returned
> > to the pool
> > + *     for reuse.
> > + *     After free, ioasid_get() will return error but
> > ioasid_find() and other
> > + *     non refcount adding APIs will continue to work until the
> > last reference
> > + *     is dropped
> > + *
> > + * - ioasid_get() get a reference on an active IOASID
> > + *
> > + * - ioasid_put() decrement and test refcount of the IOASID.
> > + *     If refcount is 0, ioasid will be freed. Deleted from the
> > system-wide
> > + *     xarray as well as per set xarray. The IOASID will be
> > returned to the
> > + *     pool and available for new allocations.
> > + *     Do nothing if refcount is non-zero.
> > + *
> > + * - ioasid_find() does not take reference, caller must hold
> > reference
> > + *
> > + * ioasid_free() can be called multiple times without error until
> > all refs are
> > + * dropped.
> > + */  
> 
> Since you already document this in ioasid.rst, I'm not sure the
> comment is necessary. Maybe the doc for _free/_put would be better in
> the function's documentation.
> 
good point. will move the free/put to doc.

> > +
> > +int ioasid_get_locked(struct ioasid_set *set, ioasid_t ioasid)
> > +{
> > +	struct ioasid_data *data;
> > +
> > +	data = xa_load(&active_allocator->xa, ioasid);
> > +	if (!data) {
> > +		pr_err("Trying to get unknown IOASID %u\n",
> > ioasid);
> > +		return -EINVAL;
> > +	}
> > +	if (data->state == IOASID_STATE_FREE_PENDING) {
> > +		pr_err("Trying to get IOASID being freed%u\n",
> > ioasid);
> > +		return -EBUSY;
> > +	}
> > +
> > +	if (set && data->set != set) {
> > +		pr_err("Trying to get IOASID not in set%u\n",
> > ioasid);
> > +		/* data found but does not belong to the set */
> > +		return -EACCES;
> > +	}
> > +	refcount_inc(&data->users);
> > +
> > +	return 0;
> > +}
> > +EXPORT_SYMBOL_GPL(ioasid_get_locked);  
> 
> Is it necessary to export the *_locked variant?  Who'd call them and
> how would they acquire the lock?
> 
It is used by KVM inside the atomic notifier handler. Notifier chain is
invoked under the lock.

> > +
> > +/**
> > + * ioasid_get - Obtain a reference of an ioasid
> > + * @set
> > + * @ioasid  
> 
> Can be dropped. The doc checker will throw a warning, though.
> 
yes, will do

> > + *
> > + * Check set ownership if @set is non-null.
> > + */
> > +int ioasid_get(struct ioasid_set *set, ioasid_t ioasid)
> > +{
> > +	int ret = 0;  
> 
> No need to initialize ret
> 
right,

> > +
> > +	spin_lock(&ioasid_allocator_lock);
> > +	ret = ioasid_get_locked(set, ioasid);
> > +	spin_unlock(&ioasid_allocator_lock);
> > +
> > +	return ret;
> > +}
> > +EXPORT_SYMBOL_GPL(ioasid_get);
> > +
> > +void ioasid_put_locked(struct ioasid_set *set, ioasid_t ioasid)
> > +{
> > +	struct ioasid_data *data;
> > +
> > +	data = xa_load(&active_allocator->xa, ioasid);
> > +	if (!data) {
> > +		pr_err("Trying to put unknown IOASID %u\n",
> > ioasid);
> > +		return;
> > +	}
> > +
> > +	if (set && data->set != set) {
> > +		pr_err("Trying to drop IOASID not in the set
> > %u\n", ioasid);
> > +		return;
> > +	}
> > +
> > +	if (!refcount_dec_and_test(&data->users)) {
> > +		pr_debug("%s: IOASID %d has %d remainning users\n",
> > +			__func__, ioasid,
> > refcount_read(&data->users));
> > +		return;
> > +	}
> > +	ioasid_do_free(data);
> > +}
> > +EXPORT_SYMBOL_GPL(ioasid_put_locked);
> > +
> > +/**
> > + * ioasid_put - Drop a reference of an ioasid
> > + * @set
> > + * @ioasid
> > + *
> > + * Check set ownership if @set is non-null.
> > + */
> > +void ioasid_put(struct ioasid_set *set, ioasid_t ioasid)
> > +{
> > +	spin_lock(&ioasid_allocator_lock);
> > +	ioasid_put_locked(set, ioasid);
> > +	spin_unlock(&ioasid_allocator_lock);
> > +}
> > +EXPORT_SYMBOL_GPL(ioasid_put);
> > +
> > +/**
> >   * ioasid_find - Find IOASID data
> >   * @set: the IOASID set
> >   * @ioasid: the IOASID to find
> > diff --git a/include/linux/ioasid.h b/include/linux/ioasid.h
> > index 412d025d440e..310abe4187a3 100644
> > --- a/include/linux/ioasid.h
> > +++ b/include/linux/ioasid.h
> > @@ -76,6 +76,10 @@ int ioasid_attach_data(ioasid_t ioasid, void
> > *data); int ioasid_register_allocator(struct ioasid_allocator_ops
> > *allocator); void ioasid_unregister_allocator(struct
> > ioasid_allocator_ops *allocator); void ioasid_is_in_set(struct
> > ioasid_set *set, ioasid_t ioasid); +int ioasid_get(struct
> > ioasid_set *set, ioasid_t ioasid); +int ioasid_get_locked(struct
> > ioasid_set *set, ioasid_t ioasid); +void ioasid_put(struct
> > ioasid_set *set, ioasid_t ioasid); +void ioasid_put_locked(struct
> > ioasid_set *set, ioasid_t ioasid);  
> 
> Please also add the stubs for !CONFIG_IOASID.
> 
yes, will do.

Thanks a lot!

> Thanks,
> Jean
> 
> >  int ioasid_set_for_each_ioasid(struct ioasid_set *sdata,
> >  			       void (*fn)(ioasid_t id, void *data),
> >  			       void *data);
> > -- 
> > 2.7.4
> >   
> _______________________________________________
> iommu mailing list
> iommu@lists.linux-foundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/iommu

[Jacob Pan]
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu