Re: [PATCH v1 00/24] New fast registration API

[Sagi Grimberg <sagig@mellanox.com>]

>
> As mentioned earlier, I have a WIP RDS fastreg branch [3]
> which is functional (at least I can RDMA messages across
> nodes ;-)).

Nice!

> So merging [2] and [3], I created [4] and applied
> a delta change based on your other patches. I saw ib_post_send
> failure with my HCA driver returning '-EINVAL'. I didn't
> debug it further but at least opcode and num_sge were set
> correctly so I shouldn't have seen it. So I did memset()
> on reg_wr which seems to have helped to fix the ib_post_send()
> failure.

Yep - that was my fault. When converting the ULPs I optimized by 
removing the memset but I forgot to set reg_wr.wr.next = NULL when
the ULP needed. This caused the driver to read a second bogus work 
request. Steve just reported this as well so I'll fix that in v2.

>
> But I got into remote access errors which tells me that I
> have messed up setup(rkey, sge setup or access flags)

One thing that pops is that in the old API the MR was registered
with iova_start = 0 (which is probably what was sent to the peer),
but the new API the iova is implicitly sg_dma_address(&sg[0]).

The registered MR holds these attributes in:
mr->rkey
mr->iova
mr->length

These should be passed to a peer to perform rdma.

Hope this helps,
Sagi.