Re: [PATCH for-next v4 0/2] RDMA/efa: Add AH cache for AH reuse

[Jason Gunthorpe <jgg@nvidia.com>]

On Tue, Jun 16, 2026 at 07:31:58PM +0000, Yonatan Nachum wrote:
> A global AH cache lock would serialize all AH commands for any PD-GID
> combination, including the ones that go to the device.
> The per-entry mutex allows different entries to issue device commands in
> parallel while only serializing operations on the same entry.
> 
> The initialized flag is needed because the entry must exist in the
> hashtable before the device command completes, so concurrent threads
> targeting the same PD-GID find it and wait on the per-entry mutex.

I think it looks so weird because it overloads the refcount in two
ways.

The scheme really has two different orthogonal ideas:
 - A kref which manages the lifetime of the entry and when the kref
   reaches 0 the entry leaves the hash. The entry is basically just the
   mutex and a user count. The hash holds a guarenteed singleton
   locking point to control the HW object creation order.

 - A user count which counts how many active AH's are using the HW
   object, and if it is non-zero then the HW object exists.

The combination of the refcount and initialized is overloading both of
these different behaviors, along with the funky refcount logic.

But given the initialized costs as much memory in the struct as
another refcount you may as well just replace it with a proper user
count.

Then it is alot simpler. 
 Global lock, do the search, get the kref, unlock and return.
 Local lock, check the usercount == 0 and allocate HW object, incr, unlock.

 Local lock, check the usercount == 1 and dealloc the HW obhect, decr,
 unlock. put kref.

Ideally the kref put would only grab the global lock when the refcount
is 0, but you have to be able to tolerate multiple 0 kref things in
the hashtable for that to work.

Jason



> 
> I am open to simplifying to a single globlal lock if you prefer, but it
> comes at the performance cost of serializing all AH commands.
>