Re: Is there a working cache for path record and lids etc for librdmacm?

[Christoph Lameter <cl@linux.com>]

[-- Attachment #1: Type: text/plain, Size: 2355 bytes --]

On Fri, 4 Dec 2020, Håkon Bugge wrote:

> >> Nop, the kernel falls back and uses the neighbour cache instead.
> >
> > But ib_acme hangs? The main issue here is what the user space app does.
> > And we need ibacm to cache user space address resolutions.
>
> I got the impression that you are debugging this with Honggang. If you want me to help, I need, to start with, an strace of ib_acme and ditto of ibacm.

Ok will do that. Do you have access to the RH case on this one?

> >>>> To resolve IPoIB address to PathRecord, you must:
> >>>> 1) The IPoIB interface must UP and RUNNING on the client and target
> >>>> side.
> >>>> 2) The ibacm service must RUNNING on the client and target.
> >>>
> >>> That is working if you want to resolve only the IP addresses of the IB
> >>> interfaces on the client and target. None else.
> >>
> >> That is why it is called IBacm, right?
> >
> > Huh? IBACM is an address resolution service for IB. Somehow that only
> > includes addresses of hosts running IBACM?
>
> Yes. As Honggang explained, ibacmn's address resolution protocol is
> based on IB multicast, as such, the peer must have ibacm running in
> order to send a unicast response back with the L2 addr.

What is the point of the route_prot and addr_prot then?

> >>> Here is the description of ibacms function from the sources:
> >>>
> >>> "Conceptually, the ibacm service implements an ARP like protocol and
> >>> either uses IB multicast records to construct path record data or queries
> >>> the SA directly, depending on the selected route protocol. By default, the
> >>> ibacm services uses and caches SA path record queries."
> >>>
> >>> SA queries dont work. So its broken and cannot talk to the SM.
> >>
> >> Why do you say that? It works all the time for me which uses "sa" as "route_prot".
> >
> > Not here and not in the tests that RH ran to verify the issue.
> >
> > "route_prot" set to "sa" is the default config for the Redhat release of
> > IBACM.
> >
> > However, the addr_prot is set to  "acm" by default. I set it to "sa" with
> > no effect.
>
> OK. Understood. As stated above, let me know if you want me to debug this.

Well whats the point to debug this if its only doing address resolution
via multicast and not via the SA?

Is there a particular issue with usiing the SA? The route information may
contain process specific information?