Mellanox CX5 Not Showing Up In ibv_devices (on some systems)

Mellanox CX5 Not Showing Up In ibv_devices (on some systems)

[Gruher, Joseph R]

Hi, I'm having trouble with ibv_devices (and related tools like ibv_devinfo, ib_send_bw, etc) not recognizing my Mellanox CX5 NICs in some systems.  The NICs work fine for TCP/IP traffic, and I think for RDMA traffic as well as I can do NVMeoF discovery and make NVMeoF connections, but they're just not listed in ibv_devices.

I have some other systems with a similar setup which work fine and I can't see what's making the difference between the two setups.  We're running an Ubuntu OS with recent kernel (this has been seen with 4.12.10 and 4.13.9).  Can anyone provide some guidance on what might cause this problem and how to fix it?

Here's the failing example:

intel@BigTwin-A:~$ uname -a
Linux BigTwin-A 4.12.10-041210-generic #201708300614 SMP Wed Aug 30 10:16:40 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

intel@BigTwin-A:~$ sudo lspci -v|egrep 'Mel|mlx'
18:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5]
        Subsystem: Mellanox Technologies MT28800 Family [ConnectX-5]
        Kernel driver in use: mlx5_core
        Kernel modules: mlx5_core

intel@BigTwin-A:~$ sudo lsmod|egrep 'mlx|ib_|rdma'
ib_iser                49152  0
libiscsi               53248  3 ib_iser,libiscsi_tcp,iscsi_tcp
scsi_transport_iscsi    94208  4 ib_iser,libiscsi,iscsi_tcp
nvmet_rdma             24576  0
nvmet                  49152  1 nvmet_rdma
nvme_rdma              28672  0
nvme_fabrics           20480  1 nvme_rdma
rdmavt                 69632  0
rdma_ucm               28672  0
rdma_cm                57344  4 ib_iser,nvme_rdma,rdma_ucm,nvmet_rdma
iw_cm                  45056  1 rdma_cm
ib_cm                  53248  1 rdma_cm
ib_uverbs              69632  1 rdma_ucm
mlx5_ib               176128  0
ib_core               204800  10 ib_iser,ib_cm,rdma_cm,nvme_rdma,ib_uverbs,iw_cm,mlx5_ib,rdmavt,rdma_ucm,nvmet_rdma
mlx5_core             413696  1 mlx5_ib
devlink                40960  1 mlx5_core
ptp                    20480  2 ixgbe,mlx5_core
nvme_core              61440  5 nvme_fabrics,nvme_rdma,nvme

intel@BigTwin-A:~$ sudo ibv_devices
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
    device                 node GUID
    ------              ----------------


A working example from a different system:

nvmf@nvmfji03:~$ uname -a
Linux nvmfji03 4.12.10-041210-generic #201708300614 SMP Wed Aug 30 10:16:40 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

nvmf@nvmfji03:~$ sudo lspci -v|egrep 'Mel|mlx'
81:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
        Subsystem: Mellanox Technologies MT27800 Family [ConnectX-5]
        Kernel driver in use: mlx5_core
        Kernel modules: mlx5_core
81:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
        Subsystem: Mellanox Technologies MT27800 Family [ConnectX-5]
        Kernel driver in use: mlx5_core
        Kernel modules: mlx5_core

nvmf@nvmfji03:~$ sudo lsmod|egrep 'mlx|ib_|rdma'
ib_iser                49152  0
libiscsi               53248  3 ib_iser,libiscsi_tcp,iscsi_tcp
scsi_transport_iscsi    94208  4 ib_iser,libiscsi,iscsi_tcp
nvmet_rdma             24576  0
nvmet                  49152  1 nvmet_rdma
nvme_rdma              28672  0
nvme_fabrics           20480  1 nvme_rdma
nvme_core              61440  2 nvme_fabrics,nvme_rdma
rdmavt                 69632  0
rdma_ucm               28672  0
rdma_cm                57344  4 ib_iser,nvme_rdma,rdma_ucm,nvmet_rdma
iw_cm                  45056  1 rdma_cm
ib_cm                  53248  1 rdma_cm
ib_uverbs              69632  1 rdma_ucm
mlx5_ib               176128  0
ib_core               204800  10 ib_iser,ib_cm,rdma_cm,nvme_rdma,ib_uverbs,iw_cm,mlx5_ib,rdmavt,rdma_ucm,nvmet_rdm
mlx5_core             413696  1 mlx5_ib
devlink                40960  1 mlx5_core
ptp                    20480  2 igb,mlx5_core

nvmf@nvmfji03:~$ sudo ibv_devices
    device                 node GUID
    ------              ----------------
    mlx5_1              248a070300a43add
    mlx5_0              248a070300a43adc


Thanks,
Joe
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Mellanox CX5 Not Showing Up In ibv_devices (on some systems)

[Jason Gunthorpe]

On Thu, Oct 26, 2017 at 04:49:29PM +0000, Gruher, Joseph R wrote:

> Hi, I'm having trouble with ibv_devices (and related tools like
> ibv_devinfo, ib_send_bw, etc) not recognizing my Mellanox CX5 NICs
> in some systems.  The NICs work fine for TCP/IP traffic, and I think
> for RDMA traffic as well as I can do NVMeoF discovery and make
> NVMeoF connections, but they're just not listed in ibv_devices.
 
> intel@BigTwin-A:~$ sudo ibv_devices
> libibverbs: Warning: no userspace device-specific driver found for
> /sys/class/infiniband_verbs/uverbs0

Well, that is pretty big clue, you haven't installed the user space
support library.

Depending on what os you are using it is either in the libmlx5 or
ibverbs-providers packages,

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Mellanox CX5 Not Showing Up In ibv_devices (on some systems)

[Gruher, Joseph R]

> -----Original Message-----
> From: Jason Gunthorpe [mailto:jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org]
> Sent: Thursday, October 26, 2017 10:00 AM
> To: Gruher, Joseph R <joseph.r.gruher-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Subject: Re: Mellanox CX5 Not Showing Up In ibv_devices (on some systems)
> 
> On Thu, Oct 26, 2017 at 04:49:29PM +0000, Gruher, Joseph R wrote:
> 
> > Hi, I'm having trouble with ibv_devices (and related tools like
> > ibv_devinfo, ib_send_bw, etc) not recognizing my Mellanox CX5 NICs in
> > some systems.  The NICs work fine for TCP/IP traffic, and I think for
> > RDMA traffic as well as I can do NVMeoF discovery and make NVMeoF
> > connections, but they're just not listed in ibv_devices.
> 
> > intel@BigTwin-A:~$ sudo ibv_devices
> > libibverbs: Warning: no userspace device-specific driver found for
> > /sys/class/infiniband_verbs/uverbs0
> 
> Well, that is pretty big clue, you haven't installed the user space support
> library.
> 
> Depending on what os you are using it is either in the libmlx5 or ibverbs-
> providers packages,
> 

I hear what you're saying, but as far as I can tell the relevant packages are installed, and the same packages on both systems?  See below.  Am I missing one I should have?

I do see now that the version are not identical, the working system has newer versions, so perhaps that is a problem...

Working system:

nvmf@nvmfji03:~$ dpkg --list|grep mlx
ii  libmlx5-1                                  1.2.1-1ubuntu2                            amd64        Userspace driver for Mellanox Connect-IB InfiniBand HCAs
ii  libmlx5-dev                                1.2.1-1ubuntu2                            amd64        Development files for the libmlx5 driver

nvmf@nvmfji03:~$ dpkg --list|grep ibverbs
ii  ibverbs-utils                              1.2.1-2ubuntu1                            amd64        Examples for the libibverbs library
ii  libibverbs-dev                             1.2.1-2ubuntu1                            amd64        Development files for the libibverbs library
ii  libibverbs1                                1.2.1-2ubuntu1                            amd64        Library for direct userspace use of RDMA (InfiniBand/iWARP)

nvmf@nvmfji03:~$ sudo ibv_devices
    device                 node GUID
    ------              ----------------
    mlx5_1              248a070300a43add
    mlx5_0              248a070300a43adc

Failing system:

intel@BigTwin-A:~$ dpkg --list|grep mlx
ii  libmlx5-1                            1.0.2-1ubuntu2                             amd64        Userspace driver for Mellanox Connect-IB InfiniBand HCAs
ii  libmlx5-dev                          1.0.2-1ubuntu2                             amd64        Development files for the libmlx5 driver

intel@BigTwin-A:~$ dpkg --list|grep ibverbs
ii  ibverbs-utils                        1.1.8-1.1ubuntu2                           amd64        Examples for the libibverbs library
ii  libibverbs-dev                       1.1.8-1.1ubuntu2                           amd64        Development files for the libibverb  library
ii  libibverbs1                          1.1.8-1.1ubuntu2                           amd64        Library for direct userspace use of RDMA (InfiniBand/iWARP)

intel@BigTwin-A:~$ sudo ibv_devices
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
    device                 node GUID
    ------              ----------------

Thanks,
Joe
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Mellanox CX5 Not Showing Up In ibv_devices (on some systems)

[Jason Gunthorpe]

On Thu, Oct 26, 2017 at 05:32:28PM +0000, Gruher, Joseph R wrote:

> I hear what you're saying, but as far as I can tell the relevant
> packages are installed, and the same packages on both systems?  See
> below.  Am I missing one I should have?

Ah, in that case it is because your libmlx5 is too old to support the
new card you have:

18:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5]
vs
81:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]

You can upgrade it by building rdma-core from source.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html