RE: rsockets and fork

Hefty, Sean
2012-08-22T23:35:37Z

I'm haven't identified the specific problem with fork support, but I did see this in libmlx4:

mlx4_alloc_context()
{
...
context->uar = mmap(NULL, to_mdev(ibdev)->page_size, PROT_WRITE,
MAP_SHARED, cmd_fd, 0);
if (context->uar == MAP_FAILED)
goto err_free;

if (resp.bf_reg_size) {
context->bf_page = mmap(NULL, to_mdev(ibdev)->page_size,
PROT_WRITE, MAP_SHARED, cmd_fd,
to_mdev(ibdev)->page_size);
...
}

I don't know for certain that these mmap() calls cause an issue, but the preload library socket() function calls rsocket(), which loads and initializes libibverbs. This calls mlx4_alloc_context() before fork() has been called.

I added the following hack to the preload socket() call, which skips calling rsocket().

if ((domain == PF_INET || domain == PF_INET6) &&
(type == SOCK_STREAM) && (!protocol || protocol == IPPROTO_TCP) && fork_support) {
printf("skipping rsocket call\n");
goto realsock;
}

recursive = 1;
ret = rsocket(domain, type, protocol);
recursive = 0;
if (ret >= 0) {
if (fork_support) {
rclose(ret);
realsock:
ret = real.socket(domain, type, protocol);

With the above hack, I no longer see hangs when running with fork() and the netperf performance is split. This should delay libibverbs initializing until after fork() has been called.

I'm trying to decide whether to keep this work-around or find a better solution.

- Sean