Re: rsockets and fork

Sridhar Samudrala
2012-08-14T17:51:53Z

On 8/13/2012 4:39 PM, Hefty, Sean wrote:
>> I could not get fork enabled netperf to work with rsockets in the latest
>> librdmacm git repository.
>> After some debugging, i found that the child netserver process is blocked at
>> sem_wait() call in fork_passive().
>> It is not clear to me how this call is supposed to unblock as sem_post()
>> is done later in the same function.
> sem_open() should create the semaphore with an initial value of 1. The sem_wait/sem_post calls serialize listening on the corresponding rsocket. The name semaphore should remain until the system is rebooted.
Looks like a stale semaphore issue. I removed /dev/shm/sem.rsocket_fork
and restarted netserver and it is working now.
>
>> However, if i start another netperf instance while the other session is
>> still going on, it seems to hang and return with a very low throughput.
>> It looks as if the first session is starving all the other sessions.
>> The right behavior would be to have the available bw split across the
>> parallel instances.
> Can you verify that the second session is using rsockets and not falling back to a socket connection?
Yes. it is also using rsockets.
The second session always hangs after sending a fixed number of bytes
(38469632).
rsend() blocks waiting for the CQ event.

Thanks
Sridhar
>
> You can try adjusting the polling time. This can usually be done by writing a value (time in microseconds) to the following file:
>
> /etc/rdma/rsocket/polling_time
>
> By default, the value is 10.
>
> - Sean