RE: RDMA does not work with kernel 4.20 or 5.1

RE: RDMA does not work with kernel 4.20 or 5.1

[Jankowski, Robert]

Hi,

I have an issue with running RDMA server on kernel 4.20 or 5.1. Everything works on kernel 4.15 on the same host machine.
With libfabric members we found that there are low-level infiniband errors. Example ib_write_bw command returned:

$ /usr/bin/ib_write_bw -d mlx5_0
************************************
* Waiting for client to connect... *
************************************
Couldn't get device attributes
Unable to create QP.
Failed to create QP.
Couldn't create IB resources


Full thread you can read here: https://github.com/ofiwg/libfabric/issues/5149

Problem occurs with:
Linux distribution and version: Ubuntu 16.04 LTS
Linux kernel and version: Linux ubuntu 5.1.0 #1 SMP Wed May 15 08:00:39 CEST 2019 x86_64 x86_64 x86_64 GNU/Linux
InfiniBand hardware and firmware version: 
We are using Mellanox NICs ConnectX-4,
$ /usr/bin/ibv_devinfo
hca_id: mlx5_1
        transport:                      InfiniBand (0)
        fw_ver:                         14.20.1010
        node_guid:                      248a:0703:00b0:449f
        sys_image_guid:                 248a:0703:00b0:449e
        vendor_id:                      0x02c9
        vendor_part_id:                 4117
        hw_ver:                         0x0
        board_id:                       MT_2470111034
        phys_port_cnt:                  1
        Device ports:
                port:   1
                        state:                  PORT_DOWN (1)
                        max_mtu:                4096 (5)
                        active_mtu:             1024 (3)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: mlx5_0
        transport:                      InfiniBand (0)
        fw_ver:                         14.20.1010
        node_guid:                      248a:0703:00b0:449e
        sys_image_guid:                 248a:0703:00b0:449e
        vendor_id:                      0x02c9
        vendor_part_id:                 4117
        hw_ver:                         0x0
        board_id:                       MT_2470111034
        phys_port_cnt:                  1
        Device ports:
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             1024 (3)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet


Can you help me find out what is wrong with my setup configuration?

Thanks,
Robert Jankowski
Intel Corporation
--------------------------------------------------------------------

Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.

Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek
przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by
others is strictly prohibited.

Re: RDMA does not work with kernel 4.20 or 5.1

[Jason Gunthorpe]

On Mon, Jul 29, 2019 at 01:32:33PM +0000, Jankowski, Robert wrote:
> Hi,
> 
> I have an issue with running RDMA server on kernel 4.20 or
> 5.1. Everything works on kernel 4.15 on the same host machine.  With
> libfabric members we found that there are low-level infiniband
> errors. Example ib_write_bw command returned:
> 
> $ /usr/bin/ib_write_bw -d mlx5_0
> ************************************
> * Waiting for client to connect... *
> ************************************
> Couldn't get device attributes
> Unable to create QP.
> Failed to create QP.
> Couldn't create IB resources
> 
> 
> Full thread you can read here: https://github.com/ofiwg/libfabric/issues/5149

Don't use MOFED userspace with an upstream kernel release.

Jason