Re: [PATCH for-next v10 04/11] RDMA/rxe: Replace red-black trees by xarrays

[Jason Gunthorpe <jgg@nvidia.com>]

On Mon, Feb 28, 2022 at 11:28:44AM -0600, Robert Pearson wrote:
> There is a xa_lock_irqsave()/ublock_irqrestore but I actually need
> some things that they don't support.
> In particular there is not an option to call xa_alloc_cyclic_irqsave

Well, yes, there is actually good reason for this. The lock/unlock
scheme that the allocating xarray functions use can't be trivially
nested like this.

When, and only when, you need to allocate the non-blocking AH you
should use this pattern

xa_lock_irqsave(..)
__xa_alloc_cyclic_reserve(..., GFP_ATOMIC);
xa_lock_irq_restore(..)

Everything else should use a simple _irq (is this even needed?)
variant without nesting under another spinlock

> and I also need an irqsave version of kref_put_lock and had to code
> one which calls the refcount version but again that takes the
> address of a lock and not an xarray. All this is because rdmacm is
> crazy and makes verbs api calls with spinlocks held.

You shouldn't need kref_put_lock at all. It isn't really a kref, just
use a normal refcount and trigger the completion when it reaches
0. Nothing fancy required.

Jason