Re: [PATCH 0/3] RDMA net namespace

[yanjun.zhu@linux.dev]

October 27, 2022 10:30 AM, "Dust Li" <dust.li@linux.alibaba.com> wrote:

> On Wed, Oct 26, 2022 at 11:01:13PM +0800, Dust Li wrote:
> 
>> On Sun, Oct 23, 2022 at 06:04:47PM -0400, Zhu Yanjun wrote:
>>> From: Zhu Yanjun <yanjun.zhu@linux.dev>
>>> 
>>> There are shared and exclusive modes in RDMA net namespace. After
>>> discussion with Leon, the above modes are compatible with legacy IB
>>> device.
>>> 
>>> To the RoCE and iWARP devices, the ib devices should be in the same net
>>> namespace with the related net devices regardless of in shared or
>>> exclusive mode.
>>> 
>>> In the first commit, when the net devices are moved to a new net
>>> namespace, the related ib devices are also moved to the same net
>>> namespace.
>>> 
>>> In the second commit, the shared/exclusive modes still work with legacy
>>> ib devices. To the RoCE and iWARP devices, these modes will not be
>>> considered.
>>> 
>>> Because MLX4/5 do not call the function ib_device_set_netdev to map ib
>>> devices and the related net devices, the function ib_device_get_by_netdev
>>> can not get ib devices from net devices. In the third commit, all the
>>> registered ib devices are parsed to get the net devices, then compared
>>> with the given net devices.
>>> 
>>> The steps to make tests:
>>> 1) Create a new net namespace net0
>>> 
>>> ip netns add net0
>>> 
>>> 2) Show the rdma links in init_net
>>> 
>>> rdma link
>>> 
>>> "
>>> link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1
>>> "
>>> 
>>> 3) Move the net device to net namespace net0
>>> 
>>> ip link set enp7s0np1 netns net0
>>> 
>>> 4) Show the rdma links in init_net again
>>> 
>>> rdma link
>>> 
>>> There is no rdma links
>> 
>> Follow your steps, after step 3), I cannot reproduce this,
>> `rdma link` running in init_net still show the link.
>> 
>> I'm testing on a VM with ConnectX-4Lx, SRIOV enabled, and VF is passthroughed
>> to the VM.
>> 
>> Anything I missed ?
> 
> Hi Zhu:
> 
> I think I know what's wrong here.
> 
> With your patch, if I put the netdevice from init_net into another
> net_namespace(say ns0), the RDMA device is not moved, and `rdma link`
> can't see the RDMA device in ns0(We can see it if we are in shared mode)


Yes. This should move rdma device to ns0, the same net namespace with the net device.

I use the following device to make tests. It can work well.
"
Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
"
And the driver is drivers/infiniband/hw/mlx5.

You are using ConnectX-4Lx. Can you let me know which driver is used?

Thanks and Regards,
Zhu Yanjun

> 
> I think this is not the correct behaviour.
> 
> Maybe we should do:
> 1. If we are in shared mode, keep the current behaviour
> 2. else we are in exclusive mode. When the corresponding netdevice of the RoCE
> or iWarp device is moved from one net namespace to another, we move the
> RDMA device into that net namespace
> 
> What do you think ?
> 
> Thanks.
> 
>>> 5) Show the rdma links in net0
>>> 
>>> ip netns exec net0 rdma link
>>> 
>>> "
>>> link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1
>>> "
>>> 
>>> We can confirm that rdma links are moved to the same net namespace with
>>> the net devices.
>>> 
>>> Zhu Yanjun (3):
>>> RDMA/core: Move ib device to the same net namespace with net device
>>> RDMA/core: The legacy IB devices still work with shared/exclusive mode
>>> RDMA/core: Get all the ib devices from net devices
>>> 
>>> drivers/infiniband/core/device.c | 107 ++++++++++++++++++++++++++++++-
>>> 1 file changed, 105 insertions(+), 2 deletions(-)
>>> 
>>> --
>>> 2.27.0