Re: Is there a working cache for path record and lids etc for librdmacm?

["Håkon Bugge" <haakon.bugge@oracle.com>]

> On 22 Nov 2020, at 13:49, Christopher Lameter <cl@linux.com> wrote:
> 
> On Fri, 20 Nov 2020, Håkon Bugge wrote:
>>> Oh great. I did not know. Will work with them to get things sorted out.
>> Inside Oracle, we're only using it for resolving IB routes. A cache for
>> address resolution already exists in the kernel. There is a config
>> option to disable address resolution from user-space
>> (acme_plus_kernel_only).
> 
> The app that we have runs in user space. Can it use the cache? Is the
> cache only in Mellanox OFED? I heard that it was removed.

An app in user space can use the ibacm cache. If you use the default configuration that comes with rdma-core, both address and route resolution will be from librdmacm directly to ibacm, i.e., no kernel involved. The ibacm options are by default installed in /etc/rdma/ibacm_opts.cfg

If you set acme_plus_kernel_only to one in said config file, you app will resolve the address using the kernel neighbour cache and the route resolution will go into the kernel and then "bounce" back  to user space and ibacm through NetLink.

I do not know if ibacm is present in Mellanox OFED, but it is easy to find out:

# rpm -q ibacm


> This is an an option while building ibacm?

Nop, runtime config option as depicted above.

> And yes we need it to resolve IB routes.

Then the above will work.

> Can you share a working config?

The default provided by rdma-core should work, possible requiring the option above.


Thxs, Håkon