Re: Is there a working cache for path record and lids etc for librdmacm?

[Christopher Lameter <cl@linux.com>]

[-- Attachment #1: Type: text/plain, Size: 2312 bytes --]

On Fri, 27 Nov 2020, Håkon Bugge wrote:

> > Huh? When does it talk to a subnet manager (or the SA)?
>
> When resolving the route AND the option "route_prot" is set to "sa". If
> set to "acm", what Hong describes above applies.

My config has "route_prot" set to "sa"

> > If its get an IP address of an IB node that does not have ibacm then it
> > fails with a timeout ..... ? And leaves hanging kernel threads around by
> > design?
>
> Nop, the kernel falls back and uses the neighbour cache instead.

But ib_acme hangs? The main issue here is what the user space app does.
And we need ibacm to cache user space address resolutions.

> > So it only populates the cache from its local node information?
>
> No, if you use ibacm for address resolution the only protocol it has is
> "acm", which means the information comes from a peer ibacm.
>
> If you talk about the cache for routes, it comes either from the SA or a
> peer ibacm, depending on the "route_prot" setting.

I have always run it with that setting. How can I debug this issue and how
can we fix this?

>
> >> To resolve IPoIB address to PathRecord, you must:
> >> 1) The IPoIB interface must UP and RUNNING on the client and target
> >> side.
> >> 2) The ibacm service must RUNNING on the client and target.
> >
> > That is working if you want to resolve only the IP addresses of the IB
> > interfaces on the client and target. None else.
>
> That is why it is called IBacm, right?

Huh? IBACM is an address resolution service for IB. Somehow that only
includes addresses of hosts running IBACM?

>
> > Here is the description of ibacms function from the sources:
> >
> > "Conceptually, the ibacm service implements an ARP like protocol and
> > either uses IB multicast records to construct path record data or queries
> > the SA directly, depending on the selected route protocol. By default, the
> > ibacm services uses and caches SA path record queries."
> >
> > SA queries dont work. So its broken and cannot talk to the SM.
>
> Why do you say that? It works all the time for me which uses "sa" as "route_prot".

Not here and not in the tests that RH ran to verify the issue.

"route_prot" set to "sa" is the default config for the Redhat release of
IBACM.

However, the addr_prot is set to  "acm" by default. I set it to "sa" with
no effect.