slow rdma_connect

slow rdma_connect

[Arun Kaimalettu]

Hi,

We have a client-sever application that use rdma cm to establish connections
to do RDMA.   It takes around 30 or 40 secs for 500 clients started 
simultaneously
to complete the connections to a server.  The machine where we are testing
has enough CPU resource , so this 30 - 40secs appears to be the time for
rdma_cm to transition from resolving address to establishing connections..
Is this expected or should we be configuring rdma cm to handle this load ?

set of system calls involved in both sides are:

server:
rdma_create_id, rdma_listen, loop with { select, rdma_get_cm_event}
(listen backlog is 10)

client:
rdma_create_id, rdma_resolve_addr, rdma_resolve_route, rdma_connect, 
loop with { select rdma_get_cm_event}
(resolve timeouts are 20ms,
connect parameters are :
   param.responder_resources = 1;
   param.initiator_depth     = 1;
   param.rnr_retry_count     = 7;
   param.retry_count         = 7;
}

versions:
libibcm-1.0.5-1
ibacm-1.0.3-1
librdmacm-1.0.13-1

--
thanks,
arun

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: slow rdma_connect

[Hefty, Sean]

> to complete the connections to a server.  The machine where we are testing
> has enough CPU resource , so this 30 - 40secs appears to be the time for
> rdma_cm to transition from resolving address to establishing connections..
> Is this expected or should we be configuring rdma cm to handle this load ?

The rdma_cm can handle this load, but...
 
> set of system calls involved in both sides are:
> 
> server:
> rdma_create_id, rdma_listen, loop with { select, rdma_get_cm_event}
> (listen backlog is 10)

the backlog is the issue.  With a backlog of only 10, all additional incoming requests get dropped and must be retried.  The connect retry is a couple of seconds.  Try increasing the backlog (or setting it to 0).

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html