Re: bug report for rxe

[Guoqing Jiang <guoqing.jiang@linux.dev>]

On 2/23/22 1:01 PM, Bob Pearson wrote:
> On 2/22/22 22:43, Guoqing Jiang wrote:
>>
>> On 2/23/22 12:58 AM, Pearson, Robert B wrote:
>>>> After investigation, seems the culprit is commit 647bf13ce944 ("RDMA/rxe:
>>>> Create duplicate mapping tables for FMRs"). The problem is
>>>> mr_check_range returns -EFAULT after find iova and length are not
>>>> valid, so connection between two VMs can't be established.
>>>>
>>>> Revert the commit manually or apply below temporary change,  rxe works
>>>> again with rnbd/rtrs though I don't think it is the right thing to do.
>>>> Could experts provide a proper solution? Thanks.
>>>>
>>> This patch fixed failures in blktests and srp which were discussed at length. See e.g.
>>>
>>> https://lore.kernel.org/linux-rdma/20210907163939.GW1200268@ziepe.ca/
>> Thanks for the link, which reminds me the always_invalidate parameter in rtrs_server.
>>
>>> and related messages. The conclusion was that two mappings were required. One owned by the
>>> driver and one by the 'hardware', i.e. bottom half in the rxe case, allowing updating a new mapping
>>> while the old one is still active and then switching them.
>>>
>>> If this case has iova and length not valid as indicated is there a problem with the test case?
>> And after disable always_invalidate (which is enabled by default), rnbd/rtrs over roce
>> works either. So I suppose there might be potential issue for always_invalidate=Y in
>> rtrs server side since invalidate works for srp IIUC, @Jinpu.
>>
>> Thanks,
>> Guoqing
> It would help to understand what you are running. My main concern is that mr_check_range
> shouldn't fail unless there is something very wrong.

Let me paste it again, I just try to map server block device to client 
side, specifically the bold
line.

1.  VM (server)

modprobe rdma_rxe
rdma link add rxe0 type rxe netdev ens3
modprobe rnbd-server

2.  VM (client)

modprobe rdma_rxe
rdma link add rxe0 type rxe netdev ens3
modprobe rnbd-client
*echo "sessname=bla path=ip:$serverip 
device_path=$block_device_in_server" > 
/sys/devices/virtual/rnbd-client/ctl/map_device*



Thanks,
Guoqing