Re: What's the current status with bnxt_re-abi.h

["Pierre-Elliott Bécue" <peb@debian.org>]

[-- Attachment #1: Type: text/plain, Size: 3660 bytes --]

Selvin Xavier <selvin.xavier@broadcom.com> wrote on 13/06/2024 at 10:49:00+0200:

> On Wed, Jun 12, 2024 at 11:59 PM Pierre-Elliott Bécue <peb@debian.org> wrote:
>
>  De : Selvin Xavier <selvin.xavier@broadcom.com>
>  À : Pierre-Elliott Bécue <peb@debian.org>
>  Cc : linux-rdma@vger.kernel.org
>  Date : 12 juin 2024 19:18:34
>  Objet : Re: What's the current status with bnxt_re-abi.h
>
>  > Hi,
>  > bnxt_re-abi.h in linux kernel and rdma-core uses abi version 1. We
>  > dont bump up the version in upstream and backward compatibility is
>  > maintained using the comp_mask field of the interface structures.
>  > 
>  > If you are using the latest drivers maintained in the Broadcom site
>  > (which uses ABI version 6), you need to use the libbnxt_re hosted in
>  > the Broacom site itself. We maintain compatibility between the Out of
>  > tree driver and Out of tree library.
>  > 
>  > Thanks,
>  > Selvin
>  > 
>  > On Wed, Jun 12, 2024 at 10:21 PM Pierre-Elliott Bécue <peb@debian.org> wrote:
>  >> 
>  >> Pierre-Elliott Bécue <peb@debian.org> wrote on 12/06/2024 at 18:47:36+0200:
>  >> 
>  >>> Hello,
>  >>> 
>  >>> In bnxt_re-abi.h, the abi version mentioned is 1. It's used as it's in
>  >>> all libibverbs to determine the min AND max supported ABI.
>  >>> 
>  >>> bnxt_re isn't currently mainlined in the kernel,
>  >> 
>  >> Sorry, a word is missing: "Recent bnxt_re isn't currently mainlined"
>  >> 
>  >>> and those eager to use
>  >>> the driver need to rely on the one provided by broadcom on their
>  >>> website.
>  >>> 
>  >>> The thing is, they bumped their ABI version multiple times (current is
>  >>> 6). In the current context, one can't use the manually compiled bnxt_re
>  >>> driver with libibverbs as any call will error due to the bnxt_re abi
>  >>> version being outside of min/max supported abi version.
>  >>> 
>  >>> What's the current situation regarding bnxt_re, should we consider
>  >>> libibverb support of bnxt_re as deprecated?
>  >>> 
>  >>> Of course I could have missed something, sorry for that if that's the
>  >>> case.
>  >>> 
>  >>> Bests,
>  Hey Selvin,
>
>  Thanks a lot for clarifying this.
>
>  I built libbnxt_re, but the thing is, I can't use the usual infiniband tooling (ib_write_bw) et al without using libibverbs which is still the
>  linux-rdma one.
>
>  Do you have an alternative to suggest that I should consider?
>
> It should work fine with the libibverbs installed by
> linux-rdma-core.

Due to the ABI version being 1 in rdma-core, no. libibverbs verifies
that the current driver ABI version (6 for bnxt_re from broadcom's
website) is between ops->match_min_abi_version and
ops->match_max_abi_version, both values for bnxt_re are set to the ABI
version (1).

https://github.com/linux-rdma/rdma-core/blob/master/libibverbs/init.c#L363-L372

So when trying to use tooling from rdma-core (like ib_write_bw), the
match_device function returns false and the card can't be used.

> The Broadcom support team can help you with the
> necessary instructions. I will forward your mail to them and they will
> help.

I already interacted with the support team, and I'm not confident they
can help. But thanks. :)

>  Also, would a RDMA+SRIOV-capable broadcom card with the latest firmware (229 something) work fine with the drivers shipped in
>  rdma-core.
>
>  If so maybe I should use what's provided by the kernel and rdma-core.
>
> The drivers shipped with rdma-core and kernel should work for most of
> your use cases. I suggest you try this.

Ack, I'll try and compare perfs. Thanks!
-- 
PEB

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 853 bytes --]