Re: [PATCH for-next 2/2] RDMA/rxe: Enable rcu locking of indexed objects

[Leon Romanovsky <leon@kernel.org>]

On Wed, Jul 19, 2023 at 11:43:30AM -0500, Bob Pearson wrote:
> On 7/19/23 02:49, Leon Romanovsky wrote:
> > On Tue, Jul 18, 2023 at 12:59:44PM -0500, Bob Pearson wrote:
> >> Make rcu_read locking of critical sections with the indexed
> >> verbs objects be protected from early freeing of those objects.
> >> The AH, QP, MR and MW objects are looked up from their indices
> >> contained in received packets or WQEs during I/O processing.
> >> Make these objects be freed using kfree_rcu to avoid races.
> > 
> > Sorry, how use of RCU avoid races?
> > 
> > Thanks
> 
> The races are between destroy/deallocate/dereg verbs API calls and packets arriving or completing send
> or deferred processing of wqes. Packets and wqes contain indices/keys/numbers that refer to objects.
> The rxe driver maintains xarrays for each type of object that allow to lookup the address of the object
> from its index and then take a reference to protect the pointer. The destroy verbs defer completion
> until the reference count falls to zero and then removes the entry in the xarray. These operations
> need to be atomic. One alternative is to use spinlocks to protect them but that places a load on
> performance under heavy load which is typically dominated by the lookup function since objects tend
> to have a long lifetime. rcu readlocks are a better alternative but depend on the deferred destruction
> of the objects used in the rcu critical section. 

You rarely can replace locks with RCU without careful design changes.

Thanks