Re: [PATCH for-next] RDMA/rxe: Guard mr state with spin lock

[Jason Gunthorpe <jgg@nvidia.com>]

On Mon, Jul 25, 2022 at 03:01:15PM -0500, Bob Pearson wrote:
> Currently the rxe driver does not guard changes to the mr state
> against race conditions which can arise from races between
> local operations and remote invalidate operations. This patch
> adds a spinlock to the mr object and makes the state changes
> atomic.

This doesn't make it atomic..

> +	state = smp_load_acquire(&mr->state);
> +
>  	if (unlikely((type == RXE_LOOKUP_LOCAL && mr->lkey != key) ||
>  		     (type == RXE_LOOKUP_REMOTE && mr->rkey != key) ||
>  		     mr_pd(mr) != pd || (access && !(access & mr->access)) ||
> -		     mr->state != RXE_MR_STATE_VALID)) {
> +		     state != RXE_MR_STATE_VALID)) {
>  		rxe_put(mr);

This is still just differently racy

The whole point of invalidate is to say that when the invalidate
completion occurs there is absolutely no touching of the memory that
MR points to.

I don't see how this acheives this like this. You need a proper lock
spanning from the lookup here until all the "dma" is completed.

Jason