Re: rdma_rxe usage problem

[Leon Romanovsky <leon@kernel.org>]

On Thu, Jan 20, 2022 at 12:32:12PM +0100, Alexander Kalentyev wrote:
> Dear Robert,
> 
> Thank you for your help!
> What I did was :
> ipv6calc --in prefix+mac fe80:: XX:XX:XX:XX:XX:XX
> (with XX:XX:XX:XX:XX:XX one have to insert the MAC address of the NIC)
> And then
> udo ip addr add dev <dev> fe80::ZZZZ:ZZZZ:ZZZZ:ZZZZ/64
> (with ZZZ:ZZZZ:ZZZZ the address genereted by ipv6calc)
> sudo rdma link add rxe0 type rxe netdev <dev>
> After that everything is working as expected:
> >ibv_rc_pingpong -d rxe0 -g 0 192.168.0.176
>   local address:  LID 0x0000, QPN 0x000012, PSN 0xf85048, GID
> ZZZZ::ZZZZ:ZZZZ:ZZZZ:ZZZZ
>   remote address: LID 0x0000, QPN 0x000011, PSN 0xf7c7b7, GID
> ZZZZ::ZZZZ:ZZZZ:ZZZZ:ZZZZ
> 8192000 bytes in 0.01 seconds = 4743.49 Mbit/sec
> 1000 iters in 0.01 seconds = 13.82 usec/iter
> 
> So, I think it make sense to include this information in a README on
> github. Otherwise somebody like me can spend a week trying to figure
> out what is going on!

PR is more than welcomed.

Thanks

> 
> Thanks once again!
> Regards,
> Alexander Kalentev
> 
> ср, 19 янв. 2022 г. в 20:54, Pearson, Robert B <robert.pearson2@hpe.com>:
> >
> >
> >
> > -----Original Message-----
> > From: Alexander Kalentyev <comrad.karlovich@gmail.com>
> > Sent: Wednesday, January 19, 2022 11:53 AM
> >
> >
> > Anyway the ibv_rc_pingpong shows an error:
> >
> > >ibv_rc_pingpong -d rxe0 -g 0
> >   local address:  LID 0x0000, QPN 0x000015, PSN 0x015dd8, GID
> > fe80::4a51:c5ff:fef6:e159
> > Failed to modify QP to RTR
> > Couldn't connect to remote QP
> >
> > >
> > Alexander,
> >
> > I use a script to restart rxe after changing anything it looks like
> >
> >         #!/bin/bash
> >
> >         export LD_LIBRARY_PATH=<path to rdma-core>/rdma-core/build/lib:/usr/lib
> >
> >         sudo rmmod rdma_rxe
> >         sudo modprobe rdma_rxe
> >
> >         sudo ip link set dev enp0s3 mtu 4500
> >         sudo ip addr add dev enp0s3 fe80::0a00:27ff:fe35:5ea7/64
> >         sudo rdma link add rxe0 type rxe netdev enp0s3
> >
> > The important line is adding the ipv6 address which corresponds with the MAC address of
> > The ethernet nic which is
> >
> >         08:00:27:35:5e:a7
> >
> > Some OSes (like mine) do not create this address automatically but mangle the address.
> > But the rdma core driver seems to expect all roce providers to have it.
> >
> > Hope this helps.
> >
> > Bob Pearson
> > rpearson@hpe.com