Re: [PATCH for-next v4 0/2] RDMA/efa: Add AH cache for AH reuse

[Leon Romanovsky <leon@kernel.org>]

On Tue, Jun 16, 2026 at 09:21:45PM -0300, Jason Gunthorpe wrote:
> On Tue, Jun 16, 2026 at 07:31:58PM +0000, Yonatan Nachum wrote:
> > A global AH cache lock would serialize all AH commands for any PD-GID
> > combination, including the ones that go to the device.
> > The per-entry mutex allows different entries to issue device commands in
> > parallel while only serializing operations on the same entry.
> > 
> > The initialized flag is needed because the entry must exist in the
> > hashtable before the device command completes, so concurrent threads
> > targeting the same PD-GID find it and wait on the per-entry mutex.
> 
> I think it looks so weird because it overloads the refcount in two
> ways.
> 
> The scheme really has two different orthogonal ideas:
>  - A kref which manages the lifetime of the entry and when the kref
>    reaches 0 the entry leaves the hash. The entry is basically just the
>    mutex and a user count. The hash holds a guarenteed singleton
>    locking point to control the HW object creation order.
> 
>  - A user count which counts how many active AH's are using the HW
>    object, and if it is non-zero then the HW object exists.
> 
> The combination of the refcount and initialized is overloading both of
> these different behaviors, along with the funky refcount logic.
> 
> But given the initialized costs as much memory in the struct as
> another refcount you may as well just replace it with a proper user
> count.
> 
> Then it is alot simpler. 
>  Global lock, do the search, get the kref, unlock and return.
>  Local lock, check the usercount == 0 and allocate HW object, incr, unlock.
> 
>  Local lock, check the usercount == 1 and dealloc the HW obhect, decr,
>  unlock. put kref.
> 
> Ideally the kref put would only grab the global lock when the refcount
> is 0, but you have to be able to tolerate multiple 0 kref things in
> the hashtable for that to work.

Jason,

I would suggest that EFA start with the basics. Their command submission
path has spinlocks, so their claim of “performance degradation” should
be viewed with an appropriate degree of skepticism.

Thanks

> 
> Jason
> 
> 
> 
> > 
> > I am open to simplifying to a single globlal lock if you prefer, but it
> > comes at the performance cost of serializing all AH commands.
> >