Re: [PATCH for-next v3] Subject: RDMA/rxe: Handle zero length rdma

[Bob Pearson <rpearsonhpe@gmail.com>]

On 2/1/23 09:38, Tom Talpey wrote:
> On 2/1/2023 6:06 AM, Daisuke Matsuda (Fujitsu) wrote:
>> On Sat, Jan 28, 2023 6:10 AM Bob Pearson wrote:
>>>
>>> Currently the rxe driver does not handle all cases of zero length
>>> rdma operations correctly. The client does not have to provide an
>>> rkey for zero length RDMA operations so the rkey provided may be
>>> invalid and should not be used to lookup an mr.
>>>
>>> This patch corrects the driver to ignore the provided rkey if the
>>> reth length is zero and make sure to set the mr to NULL. In read_reply()
>>> if length is zero rxe_recheck_mr() is not called. Warnings are added in
>>> the routines in rxe_mr.c to catch NULL MRs when the length is non-zero.
>>>
>>> Signed-off-by: Bob Pearson <rpearsonhpe@gmail.com>
>>> ---
>>
>> When I applied this change, a testcase in rdma-core failed as shown below:
>> ======================================================================
>> ERROR: test_qp_ex_rc_flush (tests.test_qpex.QpExTestCase)
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>>    File "/root/rdma-core/tests/test_qpex.py", line 258, in test_qp_ex_rc_flush
>>      raise PyverbsError(f'Unexpected {wc_status_to_str(wcs[0].status)}')
>> pyverbs.pyverbs_error.PyverbsError: Unexpected Remote access error
>>
>> ----------------------------------------------------------------------
>>
>> In my opinion, your change makes sense within the range of traditional
>> RDMA operations, but conflicts with the new RDMA FLUSH operation.
>> Responder cannot access the target MR because of invalid rkey. The
>> root cause is written in IBA Annex A19, especially 'oA19-2'.
>> We thus cannot set qp->resp.rkey to 0 in qp_resp_from_reth().
>>
>> Do you have anything to say about this? > Li Zhijian
>>
>> Thanks,
>> Daisuke Matsuda
> 
> I'm confused too, Bob can you point to the section of the spec
> that allows the rkey to be zero? It's my understanding that
> a zero-length RDMA Read must always check for access, even
> though no data is actually fetched. That would not be possible
> without an rkey.
> 
> Tom.
> 
Tom, Daisuke,

C9-88: For an HCA responder using Reliable Connection service, for
each zero-length RDMA READ or WRITE request, the R_Key shall not be
validated, even if the request includes Immediate data.

Further I have seen the pyverbs test suite sending a totally bogus rkey on a zero length rdma read. That was the impetus for me looking at this.

Daisuke has a different issue since flush is a different operation than read or write.
I need to look into what a zero length flush means.

Bob