Extending rdma_server example

Extending rdma_server example

[Jonathan Rosser]

Hi,

I took librdmacm/examples/rdma_server.c and converted it to be a persistent
server to serve successive requests. However, this does not work and I have 
the following problem:

fprintfs from my code (lower case) and librdmacm (caps) for a first successful
connection and a second one which causes a segfault

rdma_getaddrinfo
rdma_listen
rdma_getrequest              <----- wait for first request
RDMA CREATE QP
UCMA CREATE_CQS
CALLED UCMA CREATE CQS
CREATING RECV_CQ             <----- CQ created here OK for client connection
CREATING SEND_CQ             <----- this looks good
IBV CREATE QP
rdma_reg_msgs
rdma_post_recv
rdma_accept
rdma_get_recv_comp
rdma_post_send
rdma_get_send_comp
rdma_getrequest              <----- wait for second request
RDMA CREATE QP
UCMA CREATE_CQS
CALLED UCMA CREATE CQS       <----- !!! no recv/send CQ created here
IBV CREATE QP
rdma_reg_msgs
rdma_post_recv
rdma_accept
rdma_get_recv_comp           <----- dereference of id->recv_cq = NULL
Segmentation fault

So I'm wondering why the CQs do not get created second time round and it
looks like

1) rdma_get_request passes event_id and (listen_id)id_priv->qp_init_attr to
   rdma_create_qp()

2) rdma_create_qp passes qp_init_attr to ucma_create_cqs()

3) ucma_create_cqs stores a pointer to the created CQs in attr->recv_cq /
   send_cq, which is the attr of the listen_id

4) serving the second client, ucma_create_cqs checks attr->recv_cq and does 
   not create a pair of CQs

Feels a little odd to me that the listen_id takes a pointer to the CQs created
for the client_id in its qp_init_attr.

Can anyone enlighten me? Am I trying to do something (persistent server) that
rdma_get_request is not intended to do?

Cheers,
Jonathan Rosser
Senior R&D Engineer
BBC R&D

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Extending rdma_server example

[Hefty, Sean]

> 1) rdma_get_request passes event_id and (listen_id)id_priv->qp_init_attr to
>    rdma_create_qp()
> 
> 2) rdma_create_qp passes qp_init_attr to ucma_create_cqs()
> 
> 3) ucma_create_cqs stores a pointer to the created CQs in attr->recv_cq /
>    send_cq, which is the attr of the listen_id
> 
> 4) serving the second client, ucma_create_cqs checks attr->recv_cq and does
>    not create a pair of CQs
> 
> Feels a little odd to me that the listen_id takes a pointer to the CQs
> created
> for the client_id in its qp_init_attr.

This is a bug in rdma_create_qp, rdma_get_request, or both.

rdma_create_qp will modify the qp_init_attr parameter, but I need to think whether it should modify the cq pointers.  The underlying ibv_create_qp call does not, but does modify other values, so I think this call may be okay.

Because rdma_create_qp modifies the qp_init_attr parameters, rdma_get_request ends up modifying the attributes stored with the listen request.  This is definitely an error, and I'll need to come up with a fix for this.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] librdmacm: Do not modify qp_init_attr in rdma_get_request

[Hefty, Sean]

rdma_create_qp modifies the qp_init_attr structure passed in
by the user to return the actual QP capabilities that were
allocated.  If the qp_init_attr does not specify CQs, the
librdmacm will allocate CQs for the user and return them.

rdma_get_request will allocate a QP off newly connected rdma_cm_id
if the corresponding listen request is associated with a
qp_init_attr structure.  The librdmacm passes in the listen->
qp_init_attr structure into the rdma_create_qp call.
rdma_create_qp ends up modifying the qp_init_attr's associated
with the listen.  The result is that future calls to
rdma_get_request will use the modified qp attributes, rather
than those specified by the user.

Fix this by having rdma_get_request pass in a copy of the
qp_init_attr, rather than modifying those associated with the
listen.  Also update the man page for rdma_create_qp to indicate
that the qp_init_attr structure may be modified on output.

Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
I added a while(1) loop to rdma_server to allow clients to connected
repeatedly, and this worked for me.  Jonathan, can you see if this
works for your testing as well?  If so, I'll commit.

 man/rdma_create_qp.3 |    3 +++
 src/cma.c            |    5 ++++-
 2 files changed, 7 insertions(+), 1 deletions(-)

diff --git a/man/rdma_create_qp.3 b/man/rdma_create_qp.3
index 9d2de76..7445db3 100644
--- a/man/rdma_create_qp.3
+++ b/man/rdma_create_qp.3
@@ -39,6 +39,9 @@ a send or receive completion queue is not specified, then a CQ will be
 allocated by the rdma_cm for the QP, along with corresponding completion
 channels.  Completion channels and CQ data created by the rdma_cm are
 exposed to the user through the rdma_cm_id structure.
+.P
+The actual capabilities and properties of the created QP will be
+returned to the user through the qp_init_attr parameter.
 .SH "SEE ALSO"
 rdma_bind_addr(3), rdma_resolve_addr(3), rdma_destroy_qp(3), ibv_create_qp(3),
 ibv_modify_qp(3)
diff --git a/src/cma.c b/src/cma.c
index 253cf80..7ac5335 100755
--- a/src/cma.c
+++ b/src/cma.c
@@ -1368,7 +1368,10 @@ int rdma_get_request(struct rdma_cm_id *listen, struct rdma_cm_id **id)
 	}
 
 	if (id_priv->qp_init_attr) {
-		ret = rdma_create_qp(event->id, id_priv->pd, id_priv->qp_init_attr);
+		struct ibv_qp_init_attr attr;
+
+		attr = *id_priv->qp_init_attr;
+		ret = rdma_create_qp(event->id, id_priv->pd, &attr);
 		if (ret)
 			goto err;
 	}


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] librdmacm: Do not modify qp_init_attr in rdma_get_request

[Jonathan Rosser]

Hefty, Sean <sean.hefty@...> writes:

> I added a while(1) loop to rdma_server to allow clients to connected
> repeatedly, and this worked for me.  Jonathan, can you see if this
> works for your testing as well?  If so, I'll commit.

Yesterday I tried setting attr->send/recv_cq = NULL in rdma_get_request() which 
fixes the bug in a somewhat ugly manner. Passing a copy of the attributes is a 
much tidier solution, and your patch works for me.

Many Thanks,
Jonathan.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html