Re: [PATCH] RDMA/rxe: Split rxe_invalidate_mr into local and remote versions

[Jason Gunthorpe <jgg@ziepe.ca>]

On Mon, Jun 13, 2022 at 04:20:36PM +0200, haris iqbal wrote:
> > On 09/06/2022 20:03, Md Haris Iqbal wrote:
> > > Currently rxe_invalidate_mr does invalidate for both local ops, and remote
> > > ones. This means that MR being invalidated is compared with rkey for both,
> > > which is incorrect. For local invalidate, comparison should happen with
> > > lkey,
> > Just checked that IBTA SPEC ”10.6.5“ says that consumer *must* L_Key, R_Key ...
> > Not sure whether we should concern these.

I agree, 10.6.5 is quite clear that the ULP must present all of the
three options and the HCA can choose any of them.

So, rxe cannot have a bug if it always uses the rkey ?

> There are multiple things to consider here.. Since the wr for
> invalidate can have only one invalidate_rkey, there is probably no way
> to send lkey and rkey both as mentioned in the spec.

In general what this reflects is that in Linux that we don't really
completely support the optional idea in IBA that the HCA can have a
different key space for l and r keys.

> One way to make this work (mlx does this maybe?) is to always make
> rkey and lkey same and NOT make this dependent on access. Whether an
> MR is open for RDMA operations or not can be checked through the
> access permissions. I am guessing this is how it was working before.

Yes, no driver in Linux suports a disjoint key space.

So, I'm revoking my 'this makes sense to me' - the commit message does
not explain how the IBA requires the use of a lkey for a local
invalidate.

Jason