What's the current status with bnxt_re-abi.h

What's the current status with bnxt_re-abi.h

[Pierre-Elliott Bécue]

[-- Attachment #1: Type: text/plain, Size: 782 bytes --]

Hello,

In bnxt_re-abi.h, the abi version mentioned is 1. It's used as it's in
all libibverbs to determine the min AND max supported ABI.

bnxt_re isn't currently mainlined in the kernel, and those eager to use
the driver need to rely on the one provided by broadcom on their
website.

The thing is, they bumped their ABI version multiple times (current is
6). In the current context, one can't use the manually compiled bnxt_re
driver with libibverbs as any call will error due to the bnxt_re abi
version being outside of min/max supported abi version.

What's the current situation regarding bnxt_re, should we consider
libibverb support of bnxt_re as deprecated?

Of course I could have missed something, sorry for that if that's the
case.

Bests,
-- 
PEB

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 853 bytes --]

Re: What's the current status with bnxt_re-abi.h

[Pierre-Elliott Bécue]

[-- Attachment #1: Type: text/plain, Size: 965 bytes --]

Pierre-Elliott Bécue <peb@debian.org> wrote on 12/06/2024 at 18:47:36+0200:

> Hello,
>
> In bnxt_re-abi.h, the abi version mentioned is 1. It's used as it's in
> all libibverbs to determine the min AND max supported ABI.
>
> bnxt_re isn't currently mainlined in the kernel,

Sorry, a word is missing: "Recent bnxt_re isn't currently mainlined"

> and those eager to use
> the driver need to rely on the one provided by broadcom on their
> website.
>
> The thing is, they bumped their ABI version multiple times (current is
> 6). In the current context, one can't use the manually compiled bnxt_re
> driver with libibverbs as any call will error due to the bnxt_re abi
> version being outside of min/max supported abi version.
>
> What's the current situation regarding bnxt_re, should we consider
> libibverb support of bnxt_re as deprecated?
>
> Of course I could have missed something, sorry for that if that's the
> case.
>
> Bests,

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 853 bytes --]

Re: What's the current status with bnxt_re-abi.h

[Selvin Xavier]

[-- Attachment #1: Type: text/plain, Size: 1581 bytes --]

Hi,
 bnxt_re-abi.h in linux kernel and rdma-core uses abi version 1. We
dont bump up the version in upstream and backward compatibility is
maintained using the comp_mask field of the interface structures.

If you are using the latest drivers maintained in the Broadcom site
(which uses ABI version 6), you need to use the libbnxt_re hosted in
the Broacom site itself. We maintain compatibility between the Out of
tree driver and Out of tree library.

Thanks,
Selvin

On Wed, Jun 12, 2024 at 10:21 PM Pierre-Elliott Bécue <peb@debian.org> wrote:
>
> Pierre-Elliott Bécue <peb@debian.org> wrote on 12/06/2024 at 18:47:36+0200:
>
> > Hello,
> >
> > In bnxt_re-abi.h, the abi version mentioned is 1. It's used as it's in
> > all libibverbs to determine the min AND max supported ABI.
> >
> > bnxt_re isn't currently mainlined in the kernel,
>
> Sorry, a word is missing: "Recent bnxt_re isn't currently mainlined"
>
> > and those eager to use
> > the driver need to rely on the one provided by broadcom on their
> > website.
> >
> > The thing is, they bumped their ABI version multiple times (current is
> > 6). In the current context, one can't use the manually compiled bnxt_re
> > driver with libibverbs as any call will error due to the bnxt_re abi
> > version being outside of min/max supported abi version.
> >
> > What's the current situation regarding bnxt_re, should we consider
> > libibverb support of bnxt_re as deprecated?
> >
> > Of course I could have missed something, sorry for that if that's the
> > case.
> >
> > Bests,

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4224 bytes --]

Re: What's the current status with bnxt_re-abi.h

[Pierre-Elliott Bécue]

De : Selvin Xavier <selvin.xavier@broadcom.com>
À : Pierre-Elliott Bécue <peb@debian.org>
Cc : linux-rdma@vger.kernel.org
Date : 12 juin 2024 19:18:34
Objet : Re: What's the current status with bnxt_re-abi.h

> Hi,
> bnxt_re-abi.h in linux kernel and rdma-core uses abi version 1. We
> dont bump up the version in upstream and backward compatibility is
> maintained using the comp_mask field of the interface structures.
> 
> If you are using the latest drivers maintained in the Broadcom site
> (which uses ABI version 6), you need to use the libbnxt_re hosted in
> the Broacom site itself. We maintain compatibility between the Out of
> tree driver and Out of tree library.
> 
> Thanks,
> Selvin
> 
> On Wed, Jun 12, 2024 at 10:21 PM Pierre-Elliott Bécue <peb@debian.org> wrote:
>> 
>> Pierre-Elliott Bécue <peb@debian.org> wrote on 12/06/2024 at 18:47:36+0200:
>> 
>>> Hello,
>>> 
>>> In bnxt_re-abi.h, the abi version mentioned is 1. It's used as it's in
>>> all libibverbs to determine the min AND max supported ABI.
>>> 
>>> bnxt_re isn't currently mainlined in the kernel,
>> 
>> Sorry, a word is missing: "Recent bnxt_re isn't currently mainlined"
>> 
>>> and those eager to use
>>> the driver need to rely on the one provided by broadcom on their
>>> website.
>>> 
>>> The thing is, they bumped their ABI version multiple times (current is
>>> 6). In the current context, one can't use the manually compiled bnxt_re
>>> driver with libibverbs as any call will error due to the bnxt_re abi
>>> version being outside of min/max supported abi version.
>>> 
>>> What's the current situation regarding bnxt_re, should we consider
>>> libibverb support of bnxt_re as deprecated?
>>> 
>>> Of course I could have missed something, sorry for that if that's the
>>> case.
>>> 
>>> Bests,
Hey Selvin,

Thanks a lot for clarifying this.

I built libbnxt_re, but the thing is, I can't use the usual infiniband tooling (ib_write_bw) et al without using libibverbs which is still the linux-rdma one.

Do you have an alternative to suggest that I should consider?

Also, would a RDMA+SRIOV-capable broadcom card with the latest firmware (229 something) work fine with the drivers shipped in rdma-core.

If so maybe I should use what's provided by the kernel and rdma-core.

Bests,

-- 
Pierre-Elliott Bécue

Re: What's the current status with bnxt_re-abi.h

[Selvin Xavier]

[-- Attachment #1.1: Type: text/plain, Size: 2901 bytes --]

On Wed, Jun 12, 2024 at 11:59 PM Pierre-Elliott Bécue <peb@debian.org>
wrote:

> De : Selvin Xavier <selvin.xavier@broadcom.com>
> À : Pierre-Elliott Bécue <peb@debian.org>
> Cc : linux-rdma@vger.kernel.org
> Date : 12 juin 2024 19:18:34
> Objet : Re: What's the current status with bnxt_re-abi.h
>
> > Hi,
> > bnxt_re-abi.h in linux kernel and rdma-core uses abi version 1. We
> > dont bump up the version in upstream and backward compatibility is
> > maintained using the comp_mask field of the interface structures.
> >
> > If you are using the latest drivers maintained in the Broadcom site
> > (which uses ABI version 6), you need to use the libbnxt_re hosted in
> > the Broacom site itself. We maintain compatibility between the Out of
> > tree driver and Out of tree library.
> >
> > Thanks,
> > Selvin
> >
> > On Wed, Jun 12, 2024 at 10:21 PM Pierre-Elliott Bécue <peb@debian.org>
> wrote:
> >>
> >> Pierre-Elliott Bécue <peb@debian.org> wrote on 12/06/2024 at
> 18:47:36+0200:
> >>
> >>> Hello,
> >>>
> >>> In bnxt_re-abi.h, the abi version mentioned is 1. It's used as it's in
> >>> all libibverbs to determine the min AND max supported ABI.
> >>>
> >>> bnxt_re isn't currently mainlined in the kernel,
> >>
> >> Sorry, a word is missing: "Recent bnxt_re isn't currently mainlined"
> >>
> >>> and those eager to use
> >>> the driver need to rely on the one provided by broadcom on their
> >>> website.
> >>>
> >>> The thing is, they bumped their ABI version multiple times (current is
> >>> 6). In the current context, one can't use the manually compiled bnxt_re
> >>> driver with libibverbs as any call will error due to the bnxt_re abi
> >>> version being outside of min/max supported abi version.
> >>>
> >>> What's the current situation regarding bnxt_re, should we consider
> >>> libibverb support of bnxt_re as deprecated?
> >>>
> >>> Of course I could have missed something, sorry for that if that's the
> >>> case.
> >>>
> >>> Bests,
> Hey Selvin,
>
> Thanks a lot for clarifying this.
>
> I built libbnxt_re, but the thing is, I can't use the usual infiniband
> tooling (ib_write_bw) et al without using libibverbs which is still the
> linux-rdma one.
>
> Do you have an alternative to suggest that I should consider?
>
It should work fine with the libibverbs installed by linux-rdma-core. The
Broadcom support team can help you with the necessary instructions. I will
forward your mail to them and they will  help.

>
> Also, would a RDMA+SRIOV-capable broadcom card with the latest firmware
> (229 something) work fine with the drivers shipped in rdma-core.
>
> If so maybe I should use what's provided by the kernel and rdma-core.
>
The drivers shipped with rdma-core and kernel should work for most of your
use cases. I suggest you try this.

>
> Bests,
>
> --
> Pierre-Elliott Bécue
>

[-- Attachment #1.2: Type: text/html, Size: 4243 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4224 bytes --]

Re: What's the current status with bnxt_re-abi.h

[Pierre-Elliott Bécue]

[-- Attachment #1: Type: text/plain, Size: 3660 bytes --]

Selvin Xavier <selvin.xavier@broadcom.com> wrote on 13/06/2024 at 10:49:00+0200:

> On Wed, Jun 12, 2024 at 11:59 PM Pierre-Elliott Bécue <peb@debian.org> wrote:
>
>  De : Selvin Xavier <selvin.xavier@broadcom.com>
>  À : Pierre-Elliott Bécue <peb@debian.org>
>  Cc : linux-rdma@vger.kernel.org
>  Date : 12 juin 2024 19:18:34
>  Objet : Re: What's the current status with bnxt_re-abi.h
>
>  > Hi,
>  > bnxt_re-abi.h in linux kernel and rdma-core uses abi version 1. We
>  > dont bump up the version in upstream and backward compatibility is
>  > maintained using the comp_mask field of the interface structures.
>  > 
>  > If you are using the latest drivers maintained in the Broadcom site
>  > (which uses ABI version 6), you need to use the libbnxt_re hosted in
>  > the Broacom site itself. We maintain compatibility between the Out of
>  > tree driver and Out of tree library.
>  > 
>  > Thanks,
>  > Selvin
>  > 
>  > On Wed, Jun 12, 2024 at 10:21 PM Pierre-Elliott Bécue <peb@debian.org> wrote:
>  >> 
>  >> Pierre-Elliott Bécue <peb@debian.org> wrote on 12/06/2024 at 18:47:36+0200:
>  >> 
>  >>> Hello,
>  >>> 
>  >>> In bnxt_re-abi.h, the abi version mentioned is 1. It's used as it's in
>  >>> all libibverbs to determine the min AND max supported ABI.
>  >>> 
>  >>> bnxt_re isn't currently mainlined in the kernel,
>  >> 
>  >> Sorry, a word is missing: "Recent bnxt_re isn't currently mainlined"
>  >> 
>  >>> and those eager to use
>  >>> the driver need to rely on the one provided by broadcom on their
>  >>> website.
>  >>> 
>  >>> The thing is, they bumped their ABI version multiple times (current is
>  >>> 6). In the current context, one can't use the manually compiled bnxt_re
>  >>> driver with libibverbs as any call will error due to the bnxt_re abi
>  >>> version being outside of min/max supported abi version.
>  >>> 
>  >>> What's the current situation regarding bnxt_re, should we consider
>  >>> libibverb support of bnxt_re as deprecated?
>  >>> 
>  >>> Of course I could have missed something, sorry for that if that's the
>  >>> case.
>  >>> 
>  >>> Bests,
>  Hey Selvin,
>
>  Thanks a lot for clarifying this.
>
>  I built libbnxt_re, but the thing is, I can't use the usual infiniband tooling (ib_write_bw) et al without using libibverbs which is still the
>  linux-rdma one.
>
>  Do you have an alternative to suggest that I should consider?
>
> It should work fine with the libibverbs installed by
> linux-rdma-core.

Due to the ABI version being 1 in rdma-core, no. libibverbs verifies
that the current driver ABI version (6 for bnxt_re from broadcom's
website) is between ops->match_min_abi_version and
ops->match_max_abi_version, both values for bnxt_re are set to the ABI
version (1).

https://github.com/linux-rdma/rdma-core/blob/master/libibverbs/init.c#L363-L372

So when trying to use tooling from rdma-core (like ib_write_bw), the
match_device function returns false and the card can't be used.

> The Broadcom support team can help you with the
> necessary instructions. I will forward your mail to them and they will
> help.

I already interacted with the support team, and I'm not confident they
can help. But thanks. :)

>  Also, would a RDMA+SRIOV-capable broadcom card with the latest firmware (229 something) work fine with the drivers shipped in
>  rdma-core.
>
>  If so maybe I should use what's provided by the kernel and rdma-core.
>
> The drivers shipped with rdma-core and kernel should work for most of
> your use cases. I suggest you try this.

Ack, I'll try and compare perfs. Thanks!
-- 
PEB

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 853 bytes --]