[PATCH 0/3] RDMA net namespace

[PATCH 0/3] RDMA net namespace

[Zhu Yanjun]

From: Zhu Yanjun <yanjun.zhu@linux.dev>

There are shared and exclusive modes in RDMA net namespace. After
discussion with Leon, the above modes are compatible with legacy IB
device. 

To the RoCE and iWARP devices, the ib devices should be in the same net
namespace with the related net devices regardless of in shared or
exclusive mode.

In the first commit, when the net devices are moved to a new net
namespace, the related ib devices are also moved to the same net
namespace.

In the second commit, the shared/exclusive modes still work with legacy
ib devices. To the RoCE and iWARP devices, these modes will not be
considered.

Because MLX4/5 do not call the function ib_device_set_netdev to map ib
devices and the related net devices, the function ib_device_get_by_netdev
can not get ib devices from net devices. In the third commit, all the
registered ib devices are parsed to get the net devices, then compared
with the given net devices.

The steps to make tests:
1) Create a new net namespace net0

   ip netns add net0

2) Show the rdma links in init_net

   rdma link

   "
   link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1
   "

3) Move the net device to net namespace net0

   ip link set enp7s0np1 netns net0

4) Show the rdma links in init_net again

   rdma link

   There is no rdma links

5) Show the rdma links in net0

   ip netns exec net0 rdma link

   "
   link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1
   "

We can confirm that rdma links are moved to the same net namespace with
the net devices.

Zhu Yanjun (3):
  RDMA/core: Move ib device to the same net namespace with net device
  RDMA/core: The legacy IB devices still work with shared/exclusive mode
  RDMA/core: Get all the ib devices from net devices

 drivers/infiniband/core/device.c | 107 ++++++++++++++++++++++++++++++-
 1 file changed, 105 insertions(+), 2 deletions(-)

-- 
2.27.0

Re: [PATCH 0/3] RDMA net namespace

[Leon Romanovsky]

On Sun, Oct 23, 2022 at 06:04:47PM -0400, Zhu Yanjun wrote:
> From: Zhu Yanjun <yanjun.zhu@linux.dev>
> 
> There are shared and exclusive modes in RDMA net namespace. After
> discussion with Leon, the above modes are compatible with legacy IB
> device. 
> 
> To the RoCE and iWARP devices, the ib devices should be in the same net
> namespace with the related net devices regardless of in shared or
> exclusive mode.
> 
> In the first commit, when the net devices are moved to a new net
> namespace, the related ib devices are also moved to the same net
> namespace.

I think that rdma_dev_net_ops are supposed to handle this.

Thanks

Re: [PATCH 0/3] RDMA net namespace

[Yanjun Zhu]

在 2022/10/23 21:04, Leon Romanovsky 写道:
> On Sun, Oct 23, 2022 at 06:04:47PM -0400, Zhu Yanjun wrote:
>> From: Zhu Yanjun <yanjun.zhu@linux.dev>
>>
>> There are shared and exclusive modes in RDMA net namespace. After
>> discussion with Leon, the above modes are compatible with legacy IB
>> device.
>>
>> To the RoCE and iWARP devices, the ib devices should be in the same net
>> namespace with the related net devices regardless of in shared or
>> exclusive mode.
>>
>> In the first commit, when the net devices are moved to a new net
>> namespace, the related ib devices are also moved to the same net
>> namespace.
> I think that rdma_dev_net_ops are supposed to handle this.

Yes. rdma_dev_net_ops can move ib devices from one net to another net.

But these functions are called by a netlink command "rdma dev...".


In my commit, to RoCE devices, ib devices and net devices should be in 
the same net.

That is, when the net devices are moved to another net, the ib devices 
are moved

to the same net automically instead of running a netlink command to move 
ib devices.


To legacy ib devices, this netlink command is needed. To RoCE devices, 
this command

is not needed. When net devices are moved to new net, the ib devices are 
also moved automically.

Per our discussion, if RoCE's net devices and ib devices are separated 
in the different net, ib devices

can not work.

Zhu Yanjun

>
> Thanks

Re: [PATCH 0/3] RDMA net namespace

[Leon Romanovsky]

On Sun, Oct 23, 2022 at 09:42:00PM +0800, Yanjun Zhu wrote:
> 
> 在 2022/10/23 21:04, Leon Romanovsky 写道:
> > On Sun, Oct 23, 2022 at 06:04:47PM -0400, Zhu Yanjun wrote:
> > > From: Zhu Yanjun <yanjun.zhu@linux.dev>
> > > 
> > > There are shared and exclusive modes in RDMA net namespace. After
> > > discussion with Leon, the above modes are compatible with legacy IB
> > > device.
> > > 
> > > To the RoCE and iWARP devices, the ib devices should be in the same net
> > > namespace with the related net devices regardless of in shared or
> > > exclusive mode.
> > > 
> > > In the first commit, when the net devices are moved to a new net
> > > namespace, the related ib devices are also moved to the same net
> > > namespace.
> > I think that rdma_dev_net_ops are supposed to handle this.
> 
> Yes. rdma_dev_net_ops can move ib devices from one net to another net.
> 
> But these functions are called by a netlink command "rdma dev...".

rdma_dev_net_ops are called when you move netdevice from one netlink to
another.

However you raised an interesting question if it is correct behaviour to
move IB device after moved netdevice.

I don't know an answer for that.

Thanks

Re: [PATCH 0/3] RDMA net namespace

[yanjun.zhu]

October 24, 2022 12:45 AM, "Leon Romanovsky" <leon@kernel.org> wrote:

> On Sun, Oct 23, 2022 at 09:42:00PM +0800, Yanjun Zhu wrote:
> 
>> 在 2022/10/23 21:04, Leon Romanovsky 写道:
>> On Sun, Oct 23, 2022 at 06:04:47PM -0400, Zhu Yanjun wrote:
>>> From: Zhu Yanjun <yanjun.zhu@linux.dev>
>>> 
>>> There are shared and exclusive modes in RDMA net namespace. After
>>> discussion with Leon, the above modes are compatible with legacy IB
>>> device.
>>> 
>>> To the RoCE and iWARP devices, the ib devices should be in the same net
>>> namespace with the related net devices regardless of in shared or
>>> exclusive mode.
>>> 
>>> In the first commit, when the net devices are moved to a new net
>>> namespace, the related ib devices are also moved to the same net
>>> namespace.
>> I think that rdma_dev_net_ops are supposed to handle this.
>> 
>> Yes. rdma_dev_net_ops can move ib devices from one net to another net.
>> 
>> But these functions are called by a netlink command "rdma dev...".
> 
> rdma_dev_net_ops are called when you move netdevice from one netlink to
> another.

To "rdma_dev_net_ops are called when you move netdevice from one netlink to another.", 

if I get you correctly, you mean, when moving net device form one net namespace to another, rdma_dev_net_ops
will be called.

in fact, rdma_dev_net_ops will be called when running the 2 commands "ip netns add ..." and "ip netns del ...".

> 
> However you raised an interesting question if it is correct behaviour to
> move IB device after moved netdevice.

Now we come back to the original problem, how to make RoCE ib device work when the ib device and net devices are separated in the 2 different net namespaces?

If you know, please let me know.

Currently I keep ib devices and the related net devices in the same net namespace to make ib devices work. So I made these commits to keep ib devices and the related net devices in the same net namespace automatically .

Zhu Yanjun

> 
> I don't know an answer for that.
> 
> Thanks

[PATCH 1/3] RDMA/core: Move ib device to the same net namespace with net device

[Zhu Yanjun]

From: Zhu Yanjun <yanjun.zhu@linux.dev>

When a net device is moved from a net namespace to another net namespace,
the related ib device should also be moved to the same net namespace.

Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev>
---
 drivers/infiniband/core/device.c | 57 ++++++++++++++++++++++++++++++++
 1 file changed, 57 insertions(+)

diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index d275db195f1a..59784fd10876 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -2735,6 +2735,54 @@ int ib_dma_virt_map_sg(struct ib_device *dev, struct scatterlist *sg, int nents)
 EXPORT_SYMBOL(ib_dma_virt_map_sg);
 #endif /* CONFIG_INFINIBAND_VIRT_DMA */
 
+static int rdma_netns_notify(struct notifier_block *not_blk,
+			     unsigned long event, void *arg)
+{
+	struct net_device *ndev = netdev_notifier_info_to_dev(arg);
+	struct ib_device *ibdev = ib_device_get_by_netdev(ndev, RDMA_DRIVER_UNKNOWN);
+
+	if (!ibdev)
+		return NOTIFY_OK;
+
+	switch (event) {
+	case NETDEV_REGISTER:
+		ib_device_put(ibdev);
+		if (!net_eq(read_pnet(&ibdev->coredev.rdma_net), dev_net(ndev))) {
+			int ret = 0;
+
+			get_device(&ibdev->dev);
+			ret = rdma_dev_change_netns(ibdev,
+						    read_pnet(&ibdev->coredev.rdma_net),
+						    dev_net(ndev));
+			if (ret) {
+				put_device(&ibdev->dev);
+				return NOTIFY_OK;
+			}
+			put_device(&ibdev->dev);
+		}
+		break;
+	case NETDEV_UNREGISTER:
+		ib_device_put(ibdev);
+		break;
+	case NETDEV_REBOOT:
+	case NETDEV_GOING_DOWN:
+	case NETDEV_CHANGEADDR:
+	case NETDEV_CHANGENAME:
+	case NETDEV_FEAT_CHANGE:
+	default:
+		ib_device_put(ibdev);
+		pr_info("ignoring netdev event = %ld for %s\n",
+			event, ndev->name);
+		break;
+	}
+
+	return NOTIFY_OK;
+}
+
+static struct notifier_block rdma_netns_notifier = {
+	.notifier_call = rdma_netns_notify,
+};
+
 static const struct rdma_nl_cbs ibnl_ls_cb_table[RDMA_NL_LS_NUM_OPS] = {
 	[RDMA_NL_LS_OP_RESOLVE] = {
 		.doit = ib_nl_handle_resolve_resp,
@@ -2817,8 +2865,16 @@ static int __init ib_core_init(void)
 	rdma_nl_register(RDMA_NL_LS, ibnl_ls_cb_table);
 	roce_gid_mgmt_init();
 
+	ret = register_netdevice_notifier(&rdma_netns_notifier);
+	if (ret) {
+		pr_err("Failed to register netdev notifier\n");
+		goto err_netdevice;
+	}
+
 	return 0;
 
+err_netdevice:
+	unregister_pernet_device(&rdma_dev_net_ops);
 err_compat:
 	unregister_blocking_lsm_notifier(&ibdev_lsm_nb);
 err_sa:
@@ -2842,6 +2898,7 @@ static int __init ib_core_init(void)
 
 static void __exit ib_core_cleanup(void)
 {
+	unregister_netdevice_notifier(&rdma_netns_notifier);
 	roce_gid_mgmt_cleanup();
 	nldev_exit();
 	rdma_nl_unregister(RDMA_NL_LS);
-- 
2.27.0

[PATCH 2/3] RDMA/core: The legacy IB devices still work with shared/exclusive mode

[Zhu Yanjun]

From: Zhu Yanjun <yanjun.zhu@linux.dev>

Other ib devices except legacy IB devices should work with its related
net devices. That is, these ib devices should be in the same net
namespace with the related net devices.

Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev>
---
 drivers/infiniband/core/device.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index 59784fd10876..d38eb1fc2ed7 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -140,8 +140,13 @@ MODULE_PARM_DESC(netns_mode,
  */
 bool rdma_dev_access_netns(const struct ib_device *dev, const struct net *net)
 {
-	return (ib_devices_shared_netns ||
-		net_eq(read_pnet(&dev->coredev.rdma_net), net));
+	/* ib_devices_shared_netns is only for IB device. */
+	if (rdma_protocol_ib(dev, rdma_start_port(dev))) {
+		return (ib_devices_shared_netns ||
+			net_eq(read_pnet(&dev->coredev.rdma_net), net));
+	} else { /* Others device */
+		return net_eq(read_pnet(&dev->coredev.rdma_net), net);
+	}
 }
 EXPORT_SYMBOL(rdma_dev_access_netns);
 
@@ -2744,6 +2749,12 @@ static int rdma_netns_notify(struct notifier_block *not_blk,
 	if (!ibdev)
 		return NOTIFY_OK;
 
+	/* This will exclude IB device */
+	if (rdma_protocol_ib(ibdev, rdma_start_port(ibdev))) {
+		ib_device_put(ibdev);
+		return NOTIFY_OK;
+	}
+
 	switch (event) {
 	case NETDEV_REGISTER:
 		ib_device_put(ibdev);
-- 
2.27.0

[PATCH 3/3] RDMA/core: Get all the ib devices from net devices

[Zhu Yanjun]

From: Zhu Yanjun <yanjun.zhu@linux.dev>

To mlx4/5, the function ib_device_get_by_netdev can not get
ib devices from net devices.

So this function parses all the ib devices to get all the net
devices, then compares the net devices with the given net device.

Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev>
---
 drivers/infiniband/core/device.c | 39 ++++++++++++++++++++++++++++++--
 1 file changed, 37 insertions(+), 2 deletions(-)

diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index d38eb1fc2ed7..55e2d75d752e 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -2740,14 +2740,49 @@ int ib_dma_virt_map_sg(struct ib_device *dev, struct scatterlist *sg, int nents)
 EXPORT_SYMBOL(ib_dma_virt_map_sg);
 #endif /* CONFIG_INFINIBAND_VIRT_DMA */
 
+static struct ib_device *get_ibdev_from_ndev(struct net_device *ndev)
+{
+	unsigned long index;
+	struct ib_device *dev;
+	int i;
+
+	down_read(&devices_rwsem);
+	xa_for_each_marked(&devices, index, dev, DEVICE_REGISTERED) {
+		if (!dev->ops.get_netdev)
+			continue;
+
+		for (i = 0; i < dev->phys_port_cnt; i++) {
+			struct net_device *netdev;
+
+			netdev = dev->ops.get_netdev(dev, i+1);
+			if (!netdev)
+				continue;
+
+			dev_put(netdev);
+			if (ndev == netdev) {
+				up_read(&devices_rwsem);
+				if (!ib_device_try_get(dev))
+					dev = NULL;
+				return dev;
+			}
+		}
+	}
+	up_read(&devices_rwsem);
+	return NULL;
+}
+
 static int rdma_netns_notify(struct notifier_block *not_blk,
 			     unsigned long event, void *arg)
 {
 	struct net_device *ndev = netdev_notifier_info_to_dev(arg);
 	struct ib_device *ibdev = ib_device_get_by_netdev(ndev, RDMA_DRIVER_UNKNOWN);
 
-	if (!ibdev)
-		return NOTIFY_OK;
+	if (!ibdev) {
+		/* This is for MLX4/5 */
+		ibdev = get_ibdev_from_ndev(ndev);
+		if (!ibdev)
+			return NOTIFY_OK;
+	}
 
 	/* This will exclude IB device */
 	if (rdma_protocol_ib(ibdev, rdma_start_port(ibdev))) {
-- 
2.27.0

Re: [PATCH 0/3] RDMA net namespace

[Dust Li]

On Sun, Oct 23, 2022 at 06:04:47PM -0400, Zhu Yanjun wrote:
>From: Zhu Yanjun <yanjun.zhu@linux.dev>
>
>There are shared and exclusive modes in RDMA net namespace. After
>discussion with Leon, the above modes are compatible with legacy IB
>device. 
>
>To the RoCE and iWARP devices, the ib devices should be in the same net
>namespace with the related net devices regardless of in shared or
>exclusive mode.

Does this mean that shared mode is no longer supported for RoCE and iWarp
devices ?


>
>In the first commit, when the net devices are moved to a new net
>namespace, the related ib devices are also moved to the same net
>namespace.
>
>In the second commit, the shared/exclusive modes still work with legacy
>ib devices. To the RoCE and iWARP devices, these modes will not be
>considered.
>
>Because MLX4/5 do not call the function ib_device_set_netdev to map ib
>devices and the related net devices, the function ib_device_get_by_netdev
>can not get ib devices from net devices. In the third commit, all the
>registered ib devices are parsed to get the net devices, then compared
>with the given net devices.
>
>The steps to make tests:
>1) Create a new net namespace net0
>
>   ip netns add net0
>
>2) Show the rdma links in init_net
>
>   rdma link
>
>   "
>   link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1
>   "
>
>3) Move the net device to net namespace net0
>
>   ip link set enp7s0np1 netns net0
>
>4) Show the rdma links in init_net again
>
>   rdma link
>
>   There is no rdma links
>
>5) Show the rdma links in net0
>
>   ip netns exec net0 rdma link
>
>   "
>   link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1
>   "
>
>We can confirm that rdma links are moved to the same net namespace with
>the net devices.
>
>Zhu Yanjun (3):
>  RDMA/core: Move ib device to the same net namespace with net device
>  RDMA/core: The legacy IB devices still work with shared/exclusive mode
>  RDMA/core: Get all the ib devices from net devices
>
> drivers/infiniband/core/device.c | 107 ++++++++++++++++++++++++++++++-
> 1 file changed, 105 insertions(+), 2 deletions(-)
>
>-- 
>2.27.0

Re: [PATCH 0/3] RDMA net namespace

[yanjun.zhu]

October 24, 2022 9:10 AM, "Dust Li" <dust.li@linux.alibaba.com> wrote:

> On Sun, Oct 23, 2022 at 06:04:47PM -0400, Zhu Yanjun wrote:
> 
>> From: Zhu Yanjun <yanjun.zhu@linux.dev>
>> 
>> There are shared and exclusive modes in RDMA net namespace. After
>> discussion with Leon, the above modes are compatible with legacy IB
>> device.
>> 
>> To the RoCE and iWARP devices, the ib devices should be in the same net
>> namespace with the related net devices regardless of in shared or
>> exclusive mode.
> 
> Does this mean that shared mode is no longer supported for RoCE and iWarp
> devices ?

From the discussion,  a RoCE and iWarp device should make ib devices and net devices in the same net. So a RoCE and iWarp device has no shared/exclusive modes.

Shared/exclusive modes are for legacy ib devices, such as ipoib. 

In this patch series, shared/exclusive modes are left for legacy ib devices.
To a RoCE and iWarp device, we just keep net devices and ib devices in the same net.



> 
>> In the first commit, when the net devices are moved to a new net
>> namespace, the related ib devices are also moved to the same net
>> namespace.
>> 
>> In the second commit, the shared/exclusive modes still work with legacy
>> ib devices. To the RoCE and iWARP devices, these modes will not be
>> considered.
>> 
>> Because MLX4/5 do not call the function ib_device_set_netdev to map ib
>> devices and the related net devices, the function ib_device_get_by_netdev
>> can not get ib devices from net devices. In the third commit, all the
>> registered ib devices are parsed to get the net devices, then compared
>> with the given net devices.
>> 
>> The steps to make tests:
>> 1) Create a new net namespace net0
>> 
>> ip netns add net0
>> 
>> 2) Show the rdma links in init_net
>> 
>> rdma link
>> 
>> "
>> link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1
>> "
>> 
>> 3) Move the net device to net namespace net0
>> 
>> ip link set enp7s0np1 netns net0
>> 
>> 4) Show the rdma links in init_net again
>> 
>> rdma link
>> 
>> There is no rdma links
>> 
>> 5) Show the rdma links in net0
>> 
>> ip netns exec net0 rdma link
>> 
>> "
>> link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1
>> "
>> 
>> We can confirm that rdma links are moved to the same net namespace with
>> the net devices.
>> 
>> Zhu Yanjun (3):
>> RDMA/core: Move ib device to the same net namespace with net device
>> RDMA/core: The legacy IB devices still work with shared/exclusive mode
>> RDMA/core: Get all the ib devices from net devices
>> 
>> drivers/infiniband/core/device.c | 107 ++++++++++++++++++++++++++++++-
>> 1 file changed, 105 insertions(+), 2 deletions(-)
>> 
>> --
>> 2.27.0

Re: [PATCH 0/3] RDMA net namespace

[Dust Li]

On Mon, Oct 24, 2022 at 06:15:01AM +0000, yanjun.zhu@linux.dev wrote:
>October 24, 2022 9:10 AM, "Dust Li" <dust.li@linux.alibaba.com> wrote:
>
>> On Sun, Oct 23, 2022 at 06:04:47PM -0400, Zhu Yanjun wrote:
>> 
>>> From: Zhu Yanjun <yanjun.zhu@linux.dev>
>>> 
>>> There are shared and exclusive modes in RDMA net namespace. After
>>> discussion with Leon, the above modes are compatible with legacy IB
>>> device.
>>> 
>>> To the RoCE and iWARP devices, the ib devices should be in the same net
>>> namespace with the related net devices regardless of in shared or
>>> exclusive mode.
>> 
>> Does this mean that shared mode is no longer supported for RoCE and iWarp
>> devices ?
>
>From the discussion,  a RoCE and iWarp device should make ib devices and net devices in the same net. So a RoCE and iWarp device has no shared/exclusive modes.
>
>Shared/exclusive modes are for legacy ib devices, such as ipoib. 
>
>In this patch series, shared/exclusive modes are left for legacy ib devices.
>To a RoCE and iWarp device, we just keep net devices and ib devices in the same net.

I think this may limit the use case of RoCE and iWarp.

See the following use case:
In the container enviroment, we may have lots of containers on a host,
for example, more than 100. And we don't have that much VFs, so we use
ipvlan or other virtual network devices for each container, and put
those virtual network devices into each container(net namespace).
Since we only use 1 physical network device for all those containers,
there is only one RoCE device. If we don't support shared mode, we
cannot even enable RDMA for those containers with RoCE.

I don't know any other way to solve this, maybe I missed something ?

Thanks

>
>
>
>> 
>>> In the first commit, when the net devices are moved to a new net
>>> namespace, the related ib devices are also moved to the same net
>>> namespace.
>>> 
>>> In the second commit, the shared/exclusive modes still work with legacy
>>> ib devices. To the RoCE and iWARP devices, these modes will not be
>>> considered.
>>> 
>>> Because MLX4/5 do not call the function ib_device_set_netdev to map ib
>>> devices and the related net devices, the function ib_device_get_by_netdev
>>> can not get ib devices from net devices. In the third commit, all the
>>> registered ib devices are parsed to get the net devices, then compared
>>> with the given net devices.
>>> 
>>> The steps to make tests:
>>> 1) Create a new net namespace net0
>>> 
>>> ip netns add net0
>>> 
>>> 2) Show the rdma links in init_net
>>> 
>>> rdma link
>>> 
>>> "
>>> link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1
>>> "
>>> 
>>> 3) Move the net device to net namespace net0
>>> 
>>> ip link set enp7s0np1 netns net0
>>> 
>>> 4) Show the rdma links in init_net again
>>> 
>>> rdma link
>>> 
>>> There is no rdma links
>>> 
>>> 5) Show the rdma links in net0
>>> 
>>> ip netns exec net0 rdma link
>>> 
>>> "
>>> link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1
>>> "
>>> 
>>> We can confirm that rdma links are moved to the same net namespace with
>>> the net devices.
>>> 
>>> Zhu Yanjun (3):
>>> RDMA/core: Move ib device to the same net namespace with net device
>>> RDMA/core: The legacy IB devices still work with shared/exclusive mode
>>> RDMA/core: Get all the ib devices from net devices
>>> 
>>> drivers/infiniband/core/device.c | 107 ++++++++++++++++++++++++++++++-
>>> 1 file changed, 105 insertions(+), 2 deletions(-)
>>> 
>>> --
>>> 2.27.0

Re: [PATCH 0/3] RDMA net namespace

[Yanjun Zhu]

在 2022/10/24 19:52, Dust Li 写道:
> On Mon, Oct 24, 2022 at 06:15:01AM +0000, yanjun.zhu@linux.dev wrote:
>> October 24, 2022 9:10 AM, "Dust Li" <dust.li@linux.alibaba.com> wrote:
>>
>>> On Sun, Oct 23, 2022 at 06:04:47PM -0400, Zhu Yanjun wrote:
>>>
>>>> From: Zhu Yanjun <yanjun.zhu@linux.dev>
>>>>
>>>> There are shared and exclusive modes in RDMA net namespace. After
>>>> discussion with Leon, the above modes are compatible with legacy IB
>>>> device.
>>>>
>>>> To the RoCE and iWARP devices, the ib devices should be in the same net
>>>> namespace with the related net devices regardless of in shared or
>>>> exclusive mode.
>>> Does this mean that shared mode is no longer supported for RoCE and iWarp
>>> devices ?
> >From the discussion,  a RoCE and iWarp device should make ib devices and net devices in the same net. So a RoCE and iWarp device has no shared/exclusive modes.
>> Shared/exclusive modes are for legacy ib devices, such as ipoib.
>>
>> In this patch series, shared/exclusive modes are left for legacy ib devices.
>> To a RoCE and iWarp device, we just keep net devices and ib devices in the same net.
> I think this may limit the use case of RoCE and iWarp.
>
> See the following use case:
> In the container enviroment, we may have lots of containers on a host,
> for example, more than 100. And we don't have that much VFs, so we use
> ipvlan or other virtual network devices for each container, and put
> those virtual network devices into each container(net namespace).
> Since we only use 1 physical network device for all those containers,
> there is only one RoCE device. If we don't support shared mode, we
> cannot even enable RDMA for those containers with RoCE.

You use the ipvlan or other virtual network devices for each container.

In these containers, you also use RDMA, correct?

Since all the packets for these virtual network devices finally come to

the physical network devices, without shared/exclusive modes, it should 
work.

So we do not consider the shared/exclusive mode.

Zhu Yanjun

>
> I don't know any other way to solve this, maybe I missed something ?
>
> Thanks
>
>>
>>
>>>> In the first commit, when the net devices are moved to a new net
>>>> namespace, the related ib devices are also moved to the same net
>>>> namespace.
>>>>
>>>> In the second commit, the shared/exclusive modes still work with legacy
>>>> ib devices. To the RoCE and iWARP devices, these modes will not be
>>>> considered.
>>>>
>>>> Because MLX4/5 do not call the function ib_device_set_netdev to map ib
>>>> devices and the related net devices, the function ib_device_get_by_netdev
>>>> can not get ib devices from net devices. In the third commit, all the
>>>> registered ib devices are parsed to get the net devices, then compared
>>>> with the given net devices.
>>>>
>>>> The steps to make tests:
>>>> 1) Create a new net namespace net0
>>>>
>>>> ip netns add net0
>>>>
>>>> 2) Show the rdma links in init_net
>>>>
>>>> rdma link
>>>>
>>>> "
>>>> link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1
>>>> "
>>>>
>>>> 3) Move the net device to net namespace net0
>>>>
>>>> ip link set enp7s0np1 netns net0
>>>>
>>>> 4) Show the rdma links in init_net again
>>>>
>>>> rdma link
>>>>
>>>> There is no rdma links
>>>>
>>>> 5) Show the rdma links in net0
>>>>
>>>> ip netns exec net0 rdma link
>>>>
>>>> "
>>>> link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1
>>>> "
>>>>
>>>> We can confirm that rdma links are moved to the same net namespace with
>>>> the net devices.
>>>>
>>>> Zhu Yanjun (3):
>>>> RDMA/core: Move ib device to the same net namespace with net device
>>>> RDMA/core: The legacy IB devices still work with shared/exclusive mode
>>>> RDMA/core: Get all the ib devices from net devices
>>>>
>>>> drivers/infiniband/core/device.c | 107 ++++++++++++++++++++++++++++++-
>>>> 1 file changed, 105 insertions(+), 2 deletions(-)
>>>>
>>>> --
>>>> 2.27.0

Re: [PATCH 0/3] RDMA net namespace

[Dust Li]

On Mon, Oct 24, 2022 at 09:12:56PM +0800, Yanjun Zhu wrote:
>
>在 2022/10/24 19:52, Dust Li 写道:
>> On Mon, Oct 24, 2022 at 06:15:01AM +0000, yanjun.zhu@linux.dev wrote:
>> > October 24, 2022 9:10 AM, "Dust Li" <dust.li@linux.alibaba.com> wrote:
>> > 
>> > > On Sun, Oct 23, 2022 at 06:04:47PM -0400, Zhu Yanjun wrote:
>> > > 
>> > > > From: Zhu Yanjun <yanjun.zhu@linux.dev>
>> > > > 
>> > > > There are shared and exclusive modes in RDMA net namespace. After
>> > > > discussion with Leon, the above modes are compatible with legacy IB
>> > > > device.
>> > > > 
>> > > > To the RoCE and iWARP devices, the ib devices should be in the same net
>> > > > namespace with the related net devices regardless of in shared or
>> > > > exclusive mode.
>> > > Does this mean that shared mode is no longer supported for RoCE and iWarp
>> > > devices ?
>> >From the discussion,  a RoCE and iWarp device should make ib devices and net devices in the same net. So a RoCE and iWarp device has no shared/exclusive modes.
>> > Shared/exclusive modes are for legacy ib devices, such as ipoib.
>> > 
>> > In this patch series, shared/exclusive modes are left for legacy ib devices.
>> > To a RoCE and iWarp device, we just keep net devices and ib devices in the same net.
>> I think this may limit the use case of RoCE and iWarp.
>> 
>> See the following use case:
>> In the container enviroment, we may have lots of containers on a host,
>> for example, more than 100. And we don't have that much VFs, so we use
>> ipvlan or other virtual network devices for each container, and put
>> those virtual network devices into each container(net namespace).
>> Since we only use 1 physical network device for all those containers,
>> there is only one RoCE device. If we don't support shared mode, we
>> cannot even enable RDMA for those containers with RoCE.
>
>You use the ipvlan or other virtual network devices for each container.
>
>In these containers, you also use RDMA, correct?
>
>Since all the packets for these virtual network devices finally come to
>
>the physical network devices, without shared/exclusive modes, it should work.
>
>So we do not consider the shared/exclusive mode.

For the netdevice, that's true. But for RDMA, we should not even see
the ib device in the containers any more, so I think we cannot create
qp/cq, and RDMA is not available for these containers in this case.

Thanks



>
>Zhu Yanjun
>
>> 
>> I don't know any other way to solve this, maybe I missed something ?
>> 
>> Thanks
>> 
>> > 
>> > 
>> > > > In the first commit, when the net devices are moved to a new net
>> > > > namespace, the related ib devices are also moved to the same net
>> > > > namespace.
>> > > > 
>> > > > In the second commit, the shared/exclusive modes still work with legacy
>> > > > ib devices. To the RoCE and iWARP devices, these modes will not be
>> > > > considered.
>> > > > 
>> > > > Because MLX4/5 do not call the function ib_device_set_netdev to map ib
>> > > > devices and the related net devices, the function ib_device_get_by_netdev
>> > > > can not get ib devices from net devices. In the third commit, all the
>> > > > registered ib devices are parsed to get the net devices, then compared
>> > > > with the given net devices.
>> > > > 
>> > > > The steps to make tests:
>> > > > 1) Create a new net namespace net0
>> > > > 
>> > > > ip netns add net0
>> > > > 
>> > > > 2) Show the rdma links in init_net
>> > > > 
>> > > > rdma link
>> > > > 
>> > > > "
>> > > > link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1
>> > > > "
>> > > > 
>> > > > 3) Move the net device to net namespace net0
>> > > > 
>> > > > ip link set enp7s0np1 netns net0
>> > > > 
>> > > > 4) Show the rdma links in init_net again
>> > > > 
>> > > > rdma link
>> > > > 
>> > > > There is no rdma links
>> > > > 
>> > > > 5) Show the rdma links in net0
>> > > > 
>> > > > ip netns exec net0 rdma link
>> > > > 
>> > > > "
>> > > > link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1
>> > > > "
>> > > > 
>> > > > We can confirm that rdma links are moved to the same net namespace with
>> > > > the net devices.
>> > > > 
>> > > > Zhu Yanjun (3):
>> > > > RDMA/core: Move ib device to the same net namespace with net device
>> > > > RDMA/core: The legacy IB devices still work with shared/exclusive mode
>> > > > RDMA/core: Get all the ib devices from net devices
>> > > > 
>> > > > drivers/infiniband/core/device.c | 107 ++++++++++++++++++++++++++++++-
>> > > > 1 file changed, 105 insertions(+), 2 deletions(-)
>> > > > 
>> > > > --
>> > > > 2.27.0

Re: [PATCH 0/3] RDMA net namespace

[Jason Gunthorpe]

On Mon, Oct 24, 2022 at 10:35:21PM +0800, Dust Li wrote:

> For the netdevice, that's true. But for RDMA, we should not even see
> the ib device in the containers any more, so I think we cannot create
> qp/cq, and RDMA is not available for these containers in this case.

Correct, in shared mode the RDMA device should only allow using GID
table entries that have netdevs that are present in the processe's net
namespace.

This is, in general, the philosophy. The user is supposed to keep the
various devices in the namespace in sync, because the kernel cannot
guess what is correct.

Jason

Re: [PATCH 0/3] RDMA net namespace

[Yanjun Zhu]

在 2022/10/24 22:35, Dust Li 写道:
> On Mon, Oct 24, 2022 at 09:12:56PM +0800, Yanjun Zhu wrote:
>> 在 2022/10/24 19:52, Dust Li 写道:
>>> On Mon, Oct 24, 2022 at 06:15:01AM +0000, yanjun.zhu@linux.dev wrote:
>>>> October 24, 2022 9:10 AM, "Dust Li" <dust.li@linux.alibaba.com> wrote:
>>>>
>>>>> On Sun, Oct 23, 2022 at 06:04:47PM -0400, Zhu Yanjun wrote:
>>>>>
>>>>>> From: Zhu Yanjun <yanjun.zhu@linux.dev>
>>>>>>
>>>>>> There are shared and exclusive modes in RDMA net namespace. After
>>>>>> discussion with Leon, the above modes are compatible with legacy IB
>>>>>> device.
>>>>>>
>>>>>> To the RoCE and iWARP devices, the ib devices should be in the same net
>>>>>> namespace with the related net devices regardless of in shared or
>>>>>> exclusive mode.
>>>>> Does this mean that shared mode is no longer supported for RoCE and iWarp
>>>>> devices ?
>>> >From the discussion,  a RoCE and iWarp device should make ib devices and net devices in the same net. So a RoCE and iWarp device has no shared/exclusive modes.
>>>> Shared/exclusive modes are for legacy ib devices, such as ipoib.
>>>>
>>>> In this patch series, shared/exclusive modes are left for legacy ib devices.
>>>> To a RoCE and iWarp device, we just keep net devices and ib devices in the same net.
>>> I think this may limit the use case of RoCE and iWarp.
>>>
>>> See the following use case:
>>> In the container enviroment, we may have lots of containers on a host,
>>> for example, more than 100. And we don't have that much VFs, so we use
>>> ipvlan or other virtual network devices for each container, and put
>>> those virtual network devices into each container(net namespace).
>>> Since we only use 1 physical network device for all those containers,
>>> there is only one RoCE device. If we don't support shared mode, we
>>> cannot even enable RDMA for those containers with RoCE.
>> You use the ipvlan or other virtual network devices for each container.
>>
>> In these containers, you also use RDMA, correct?
>>
>> Since all the packets for these virtual network devices finally come to
>>
>> the physical network devices, without shared/exclusive modes, it should work.
>>
>> So we do not consider the shared/exclusive mode.
> For the netdevice, that's true. But for RDMA, we should not even see
> the ib device in the containers any more, so I think we cannot create
> qp/cq, and RDMA is not available for these containers in this case.

I can not get you.

Do you mean that RDMA can not be accessed in the container after these 
patches are applied?

Can you share a test case with me?


Zhu Yanjun

>
> Thanks
>
>
>
>> Zhu Yanjun
>>
>>> I don't know any other way to solve this, maybe I missed something ?
>>>
>>> Thanks
>>>
>>>>
>>>>>> In the first commit, when the net devices are moved to a new net
>>>>>> namespace, the related ib devices are also moved to the same net
>>>>>> namespace.
>>>>>>
>>>>>> In the second commit, the shared/exclusive modes still work with legacy
>>>>>> ib devices. To the RoCE and iWARP devices, these modes will not be
>>>>>> considered.
>>>>>>
>>>>>> Because MLX4/5 do not call the function ib_device_set_netdev to map ib
>>>>>> devices and the related net devices, the function ib_device_get_by_netdev
>>>>>> can not get ib devices from net devices. In the third commit, all the
>>>>>> registered ib devices are parsed to get the net devices, then compared
>>>>>> with the given net devices.
>>>>>>
>>>>>> The steps to make tests:
>>>>>> 1) Create a new net namespace net0
>>>>>>
>>>>>> ip netns add net0
>>>>>>
>>>>>> 2) Show the rdma links in init_net
>>>>>>
>>>>>> rdma link
>>>>>>
>>>>>> "
>>>>>> link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1
>>>>>> "
>>>>>>
>>>>>> 3) Move the net device to net namespace net0
>>>>>>
>>>>>> ip link set enp7s0np1 netns net0
>>>>>>
>>>>>> 4) Show the rdma links in init_net again
>>>>>>
>>>>>> rdma link
>>>>>>
>>>>>> There is no rdma links
>>>>>>
>>>>>> 5) Show the rdma links in net0
>>>>>>
>>>>>> ip netns exec net0 rdma link
>>>>>>
>>>>>> "
>>>>>> link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1
>>>>>> "
>>>>>>
>>>>>> We can confirm that rdma links are moved to the same net namespace with
>>>>>> the net devices.
>>>>>>
>>>>>> Zhu Yanjun (3):
>>>>>> RDMA/core: Move ib device to the same net namespace with net device
>>>>>> RDMA/core: The legacy IB devices still work with shared/exclusive mode
>>>>>> RDMA/core: Get all the ib devices from net devices
>>>>>>
>>>>>> drivers/infiniband/core/device.c | 107 ++++++++++++++++++++++++++++++-
>>>>>> 1 file changed, 105 insertions(+), 2 deletions(-)
>>>>>>
>>>>>> --
>>>>>> 2.27.0

Re: [PATCH 0/3] RDMA net namespace

[Dust Li]

On Tue, Oct 25, 2022 at 10:51:38AM +0800, Yanjun Zhu wrote:
>
>在 2022/10/24 22:35, Dust Li 写道:
>> On Mon, Oct 24, 2022 at 09:12:56PM +0800, Yanjun Zhu wrote:
>> > 在 2022/10/24 19:52, Dust Li 写道:
>> > > On Mon, Oct 24, 2022 at 06:15:01AM +0000, yanjun.zhu@linux.dev wrote:
>> > > > October 24, 2022 9:10 AM, "Dust Li" <dust.li@linux.alibaba.com> wrote:
>> > > > 
>> > > > > On Sun, Oct 23, 2022 at 06:04:47PM -0400, Zhu Yanjun wrote:
>> > > > > 
>> > > > > > From: Zhu Yanjun <yanjun.zhu@linux.dev>
>> > > > > > 
>> > > > > > There are shared and exclusive modes in RDMA net namespace. After
>> > > > > > discussion with Leon, the above modes are compatible with legacy IB
>> > > > > > device.
>> > > > > > 
>> > > > > > To the RoCE and iWARP devices, the ib devices should be in the same net
>> > > > > > namespace with the related net devices regardless of in shared or
>> > > > > > exclusive mode.
>> > > > > Does this mean that shared mode is no longer supported for RoCE and iWarp
>> > > > > devices ?
>> > > >From the discussion,  a RoCE and iWarp device should make ib devices and net devices in the same net. So a RoCE and iWarp device has no shared/exclusive modes.
>> > > > Shared/exclusive modes are for legacy ib devices, such as ipoib.
>> > > > 
>> > > > In this patch series, shared/exclusive modes are left for legacy ib devices.
>> > > > To a RoCE and iWarp device, we just keep net devices and ib devices in the same net.
>> > > I think this may limit the use case of RoCE and iWarp.
>> > > 
>> > > See the following use case:
>> > > In the container enviroment, we may have lots of containers on a host,
>> > > for example, more than 100. And we don't have that much VFs, so we use
>> > > ipvlan or other virtual network devices for each container, and put
>> > > those virtual network devices into each container(net namespace).
>> > > Since we only use 1 physical network device for all those containers,
>> > > there is only one RoCE device. If we don't support shared mode, we
>> > > cannot even enable RDMA for those containers with RoCE.
>> > You use the ipvlan or other virtual network devices for each container.
>> > 
>> > In these containers, you also use RDMA, correct?
>> > 
>> > Since all the packets for these virtual network devices finally come to
>> > 
>> > the physical network devices, without shared/exclusive modes, it should work.
>> > 
>> > So we do not consider the shared/exclusive mode.
>> For the netdevice, that's true. But for RDMA, we should not even see
>> the ib device in the containers any more, so I think we cannot create
>> qp/cq, and RDMA is not available for these containers in this case.
>
>I can not get you.
>
>Do you mean that RDMA can not be accessed in the container after these
>patches are applied?
>
>Can you share a test case with me?

OK, I will test your patch first and if possible I will provider a test
case.

Thanks


>
>
>Zhu Yanjun
>
>> 
>> Thanks
>> 
>> 
>> 
>> > Zhu Yanjun
>> > 
>> > > I don't know any other way to solve this, maybe I missed something ?
>> > > 
>> > > Thanks
>> > > 
>> > > > 
>> > > > > > In the first commit, when the net devices are moved to a new net
>> > > > > > namespace, the related ib devices are also moved to the same net
>> > > > > > namespace.
>> > > > > > 
>> > > > > > In the second commit, the shared/exclusive modes still work with legacy
>> > > > > > ib devices. To the RoCE and iWARP devices, these modes will not be
>> > > > > > considered.
>> > > > > > 
>> > > > > > Because MLX4/5 do not call the function ib_device_set_netdev to map ib
>> > > > > > devices and the related net devices, the function ib_device_get_by_netdev
>> > > > > > can not get ib devices from net devices. In the third commit, all the
>> > > > > > registered ib devices are parsed to get the net devices, then compared
>> > > > > > with the given net devices.
>> > > > > > 
>> > > > > > The steps to make tests:
>> > > > > > 1) Create a new net namespace net0
>> > > > > > 
>> > > > > > ip netns add net0
>> > > > > > 
>> > > > > > 2) Show the rdma links in init_net
>> > > > > > 
>> > > > > > rdma link
>> > > > > > 
>> > > > > > "
>> > > > > > link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1
>> > > > > > "
>> > > > > > 
>> > > > > > 3) Move the net device to net namespace net0
>> > > > > > 
>> > > > > > ip link set enp7s0np1 netns net0
>> > > > > > 
>> > > > > > 4) Show the rdma links in init_net again
>> > > > > > 
>> > > > > > rdma link
>> > > > > > 
>> > > > > > There is no rdma links
>> > > > > > 
>> > > > > > 5) Show the rdma links in net0
>> > > > > > 
>> > > > > > ip netns exec net0 rdma link
>> > > > > > 
>> > > > > > "
>> > > > > > link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1
>> > > > > > "
>> > > > > > 
>> > > > > > We can confirm that rdma links are moved to the same net namespace with
>> > > > > > the net devices.
>> > > > > > 
>> > > > > > Zhu Yanjun (3):
>> > > > > > RDMA/core: Move ib device to the same net namespace with net device
>> > > > > > RDMA/core: The legacy IB devices still work with shared/exclusive mode
>> > > > > > RDMA/core: Get all the ib devices from net devices
>> > > > > > 
>> > > > > > drivers/infiniband/core/device.c | 107 ++++++++++++++++++++++++++++++-
>> > > > > > 1 file changed, 105 insertions(+), 2 deletions(-)
>> > > > > > 
>> > > > > > --
>> > > > > > 2.27.0

Re: [PATCH 0/3] RDMA net namespace

[Dust Li]

On Sun, Oct 23, 2022 at 06:04:47PM -0400, Zhu Yanjun wrote:
>From: Zhu Yanjun <yanjun.zhu@linux.dev>
>
>There are shared and exclusive modes in RDMA net namespace. After
>discussion with Leon, the above modes are compatible with legacy IB
>device. 
>
>To the RoCE and iWARP devices, the ib devices should be in the same net
>namespace with the related net devices regardless of in shared or
>exclusive mode.
>
>In the first commit, when the net devices are moved to a new net
>namespace, the related ib devices are also moved to the same net
>namespace.
>
>In the second commit, the shared/exclusive modes still work with legacy
>ib devices. To the RoCE and iWARP devices, these modes will not be
>considered.
>
>Because MLX4/5 do not call the function ib_device_set_netdev to map ib
>devices and the related net devices, the function ib_device_get_by_netdev
>can not get ib devices from net devices. In the third commit, all the
>registered ib devices are parsed to get the net devices, then compared
>with the given net devices.
>
>The steps to make tests:
>1) Create a new net namespace net0
>
>   ip netns add net0
>
>2) Show the rdma links in init_net
>
>   rdma link
>
>   "
>   link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1
>   "
>
>3) Move the net device to net namespace net0
>
>   ip link set enp7s0np1 netns net0
>
>4) Show the rdma links in init_net again
>
>   rdma link
>
>   There is no rdma links

Follow your steps, after step 3), I cannot reproduce this,
`rdma link` running in init_net still show the link.

I'm testing on a VM with ConnectX-4Lx, SRIOV enabled, and VF is passthroughed
to the VM.

Anything I missed ?

>
>5) Show the rdma links in net0
>
>   ip netns exec net0 rdma link
>
>   "
>   link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1
>   "
>
>We can confirm that rdma links are moved to the same net namespace with
>the net devices.
>
>Zhu Yanjun (3):
>  RDMA/core: Move ib device to the same net namespace with net device
>  RDMA/core: The legacy IB devices still work with shared/exclusive mode
>  RDMA/core: Get all the ib devices from net devices
>
> drivers/infiniband/core/device.c | 107 ++++++++++++++++++++++++++++++-
> 1 file changed, 105 insertions(+), 2 deletions(-)
>
>-- 
>2.27.0

Re: [PATCH 0/3] RDMA net namespace

[Dust Li]

On Wed, Oct 26, 2022 at 11:01:13PM +0800, Dust Li wrote:
>On Sun, Oct 23, 2022 at 06:04:47PM -0400, Zhu Yanjun wrote:
>>From: Zhu Yanjun <yanjun.zhu@linux.dev>
>>
>>There are shared and exclusive modes in RDMA net namespace. After
>>discussion with Leon, the above modes are compatible with legacy IB
>>device. 
>>
>>To the RoCE and iWARP devices, the ib devices should be in the same net
>>namespace with the related net devices regardless of in shared or
>>exclusive mode.
>>
>>In the first commit, when the net devices are moved to a new net
>>namespace, the related ib devices are also moved to the same net
>>namespace.
>>
>>In the second commit, the shared/exclusive modes still work with legacy
>>ib devices. To the RoCE and iWARP devices, these modes will not be
>>considered.
>>
>>Because MLX4/5 do not call the function ib_device_set_netdev to map ib
>>devices and the related net devices, the function ib_device_get_by_netdev
>>can not get ib devices from net devices. In the third commit, all the
>>registered ib devices are parsed to get the net devices, then compared
>>with the given net devices.
>>
>>The steps to make tests:
>>1) Create a new net namespace net0
>>
>>   ip netns add net0
>>
>>2) Show the rdma links in init_net
>>
>>   rdma link
>>
>>   "
>>   link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1
>>   "
>>
>>3) Move the net device to net namespace net0
>>
>>   ip link set enp7s0np1 netns net0
>>
>>4) Show the rdma links in init_net again
>>
>>   rdma link
>>
>>   There is no rdma links
>
>Follow your steps, after step 3), I cannot reproduce this,
>`rdma link` running in init_net still show the link.
>
>I'm testing on a VM with ConnectX-4Lx, SRIOV enabled, and VF is passthroughed
>to the VM.
>
>Anything I missed ?

Hi Zhu:

I think I know what's wrong here.

With your patch, if I put the netdevice from init_net into another
net_namespace(say ns0), the RDMA device is not moved, and `rdma link`
can't see the RDMA device in ns0(We can see it if we are in shared mode)

I think this is not the correct behaviour.

Maybe we should do:
1. If we are in shared mode, keep the current behaviour
2. else we are in exclusive mode. When the corresponding netdevice of the RoCE
   or iWarp device is moved from one net namespace to another, we move the
   RDMA device into that net namespace

What do you think ?

Thanks.

>
>>
>>5) Show the rdma links in net0
>>
>>   ip netns exec net0 rdma link
>>
>>   "
>>   link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1
>>   "
>>
>>We can confirm that rdma links are moved to the same net namespace with
>>the net devices.
>>
>>Zhu Yanjun (3):
>>  RDMA/core: Move ib device to the same net namespace with net device
>>  RDMA/core: The legacy IB devices still work with shared/exclusive mode
>>  RDMA/core: Get all the ib devices from net devices
>>
>> drivers/infiniband/core/device.c | 107 ++++++++++++++++++++++++++++++-
>> 1 file changed, 105 insertions(+), 2 deletions(-)
>>
>>-- 
>>2.27.0

Re: [PATCH 0/3] RDMA net namespace

[yanjun.zhu]

October 27, 2022 10:30 AM, "Dust Li" <dust.li@linux.alibaba.com> wrote:

> On Wed, Oct 26, 2022 at 11:01:13PM +0800, Dust Li wrote:
> 
>> On Sun, Oct 23, 2022 at 06:04:47PM -0400, Zhu Yanjun wrote:
>>> From: Zhu Yanjun <yanjun.zhu@linux.dev>
>>> 
>>> There are shared and exclusive modes in RDMA net namespace. After
>>> discussion with Leon, the above modes are compatible with legacy IB
>>> device.
>>> 
>>> To the RoCE and iWARP devices, the ib devices should be in the same net
>>> namespace with the related net devices regardless of in shared or
>>> exclusive mode.
>>> 
>>> In the first commit, when the net devices are moved to a new net
>>> namespace, the related ib devices are also moved to the same net
>>> namespace.
>>> 
>>> In the second commit, the shared/exclusive modes still work with legacy
>>> ib devices. To the RoCE and iWARP devices, these modes will not be
>>> considered.
>>> 
>>> Because MLX4/5 do not call the function ib_device_set_netdev to map ib
>>> devices and the related net devices, the function ib_device_get_by_netdev
>>> can not get ib devices from net devices. In the third commit, all the
>>> registered ib devices are parsed to get the net devices, then compared
>>> with the given net devices.
>>> 
>>> The steps to make tests:
>>> 1) Create a new net namespace net0
>>> 
>>> ip netns add net0
>>> 
>>> 2) Show the rdma links in init_net
>>> 
>>> rdma link
>>> 
>>> "
>>> link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1
>>> "
>>> 
>>> 3) Move the net device to net namespace net0
>>> 
>>> ip link set enp7s0np1 netns net0
>>> 
>>> 4) Show the rdma links in init_net again
>>> 
>>> rdma link
>>> 
>>> There is no rdma links
>> 
>> Follow your steps, after step 3), I cannot reproduce this,
>> `rdma link` running in init_net still show the link.
>> 
>> I'm testing on a VM with ConnectX-4Lx, SRIOV enabled, and VF is passthroughed
>> to the VM.
>> 
>> Anything I missed ?
> 
> Hi Zhu:
> 
> I think I know what's wrong here.
> 
> With your patch, if I put the netdevice from init_net into another
> net_namespace(say ns0), the RDMA device is not moved, and `rdma link`
> can't see the RDMA device in ns0(We can see it if we are in shared mode)


Yes. This should move rdma device to ns0, the same net namespace with the net device.

I use the following device to make tests. It can work well.
"
Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
"
And the driver is drivers/infiniband/hw/mlx5.

You are using ConnectX-4Lx. Can you let me know which driver is used?

Thanks and Regards,
Zhu Yanjun

> 
> I think this is not the correct behaviour.
> 
> Maybe we should do:
> 1. If we are in shared mode, keep the current behaviour
> 2. else we are in exclusive mode. When the corresponding netdevice of the RoCE
> or iWarp device is moved from one net namespace to another, we move the
> RDMA device into that net namespace
> 
> What do you think ?
> 
> Thanks.
> 
>>> 5) Show the rdma links in net0
>>> 
>>> ip netns exec net0 rdma link
>>> 
>>> "
>>> link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1
>>> "
>>> 
>>> We can confirm that rdma links are moved to the same net namespace with
>>> the net devices.
>>> 
>>> Zhu Yanjun (3):
>>> RDMA/core: Move ib device to the same net namespace with net device
>>> RDMA/core: The legacy IB devices still work with shared/exclusive mode
>>> RDMA/core: Get all the ib devices from net devices
>>> 
>>> drivers/infiniband/core/device.c | 107 ++++++++++++++++++++++++++++++-
>>> 1 file changed, 105 insertions(+), 2 deletions(-)
>>> 
>>> --
>>> 2.27.0

RE: [PATCH 0/3] RDMA net namespace

[Parav Pandit]

> From: Dust Li <dust.li@linux.alibaba.com>
> Sent: Wednesday, October 26, 2022 10:31 PM


> 2. else we are in
> exclusive mode. When the corresponding netdevice of the RoCE
>    or iWarp device is moved from one net namespace to another, we move
> the
>    RDMA device into that net namespace
> 
> What do you think ?
No. one device is not supposed to move other devices.
Every device is independent that should be moved by explicit command.

Also changes like above breaks the existing orchestration, it no-go.

Re: [PATCH 0/3] RDMA net namespace

[yanjun.zhu]

October 27, 2022 11:01 AM, "Parav Pandit" <parav@nvidia.com> wrote:

>> From: Dust Li <dust.li@linux.alibaba.com>
>> Sent: Wednesday, October 26, 2022 10:31 PM
>> 
>> 2. else we are in
>> exclusive mode. When the corresponding netdevice of the RoCE
>> or iWarp device is moved from one net namespace to another, we move
>> the
>> RDMA device into that net namespace
>> 
>> What do you think ?
> 
> No. one device is not supposed to move other devices.
> Every device is independent that should be moved by explicit command.

Can you show us where we can find this rule "Every device is independent that should be moved by explicit command."?

> 
> Also changes like above breaks the existing orchestration, it no-go.

In a RoCE device, ib device is related with the net device. When a net device is moved
to a new net namespace, if the ib device is not in the same net device, how to make ib device work?

Thanks and Regards,
Zhu Yanjun

RE: [PATCH 0/3] RDMA net namespace

[Parav Pandit]

> From: yanjun.zhu@linux.dev <yanjun.zhu@linux.dev>
> Sent: Wednesday, October 26, 2022 11:08 PM
> 
> October 27, 2022 11:01 AM, "Parav Pandit" <parav@nvidia.com> wrote:
> 
> >> From: Dust Li <dust.li@linux.alibaba.com>
> >> Sent: Wednesday, October 26, 2022 10:31 PM
> >>
> >> 2. else we are in
> >> exclusive mode. When the corresponding netdevice of the RoCE or iWarp
> >> device is moved from one net namespace to another, we move the RDMA
> >> device into that net namespace
> >>
> >> What do you think ?
> >
> > No. one device is not supposed to move other devices.
> > Every device is independent that should be moved by explicit command.
> 
> Can you show us where we can find this rule "Every device is independent
> that should be moved by explicit command."?
> 
> >
> > Also changes like above breaks the existing orchestration, it no-go.
> 
> In a RoCE device, ib device is related with the net device. When a net device
> is moved to a new net namespace, if the ib device is not in the same net
> device, how to make ib device work?
RDMA device should also be moved to the same net namespace as that of netdev.

Steps should be,

$ rdma system set netns exclusive
$ ip netns add NSNAME
$ ip link set [NETDEV] netns NSNAME
$ rdma dev set [ RDMA_DEV ] netns NSNAME

Re: [PATCH 0/3] RDMA net namespace

[yanjun.zhu]

October 27, 2022 11:10 AM, "Parav Pandit" <parav@nvidia.com> wrote:

>> From: yanjun.zhu@linux.dev <yanjun.zhu@linux.dev>
>> Sent: Wednesday, October 26, 2022 11:08 PM
>> 
>> October 27, 2022 11:01 AM, "Parav Pandit" <parav@nvidia.com> wrote:
>> 
>> From: Dust Li <dust.li@linux.alibaba.com>
>> Sent: Wednesday, October 26, 2022 10:31 PM
>> 
>> 2. else we are in
>> exclusive mode. When the corresponding netdevice of the RoCE or iWarp
>> device is moved from one net namespace to another, we move the RDMA
>> device into that net namespace
>> 
>> What do you think ?
>> 
>> No. one device is not supposed to move other devices.
>> Every device is independent that should be moved by explicit command.
>> 
>> Can you show us where we can find this rule "Every device is independent
>> that should be moved by explicit command."?
>> 
>> Also changes like above breaks the existing orchestration, it no-go.
>> 
>> In a RoCE device, ib device is related with the net device. When a net device
>> is moved to a new net namespace, if the ib device is not in the same net
>> device, how to make ib device work?
> 
> RDMA device should also be moved to the same net namespace as that of netdev.

sure. I know the following commands.

In my commits, the process of moving IB devices to the same net namespace with net devices is automatically finished.

Is it OK?

Zhu Yanjun

> 
> Steps should be,
> 
> $ rdma system set netns exclusive
> $ ip netns add NSNAME
> $ ip link set [NETDEV] netns NSNAME
> $ rdma dev set [ RDMA_DEV ] netns NSNAME

RE: [PATCH 0/3] RDMA net namespace

[Parav Pandit]

> From: yanjun.zhu@linux.dev <yanjun.zhu@linux.dev>
> Sent: Wednesday, October 26, 2022 11:17 PM
> 
> October 27, 2022 11:10 AM, "Parav Pandit" <parav@nvidia.com> wrote:
> 
> >> From: yanjun.zhu@linux.dev <yanjun.zhu@linux.dev>
> >> Sent: Wednesday, October 26, 2022 11:08 PM
> >>
> >> October 27, 2022 11:01 AM, "Parav Pandit" <parav@nvidia.com> wrote:
> >>
> >> From: Dust Li <dust.li@linux.alibaba.com>
> >> Sent: Wednesday, October 26, 2022 10:31 PM
> >>
> >> 2. else we are in
> >> exclusive mode. When the corresponding netdevice of the RoCE or iWarp
> >> device is moved from one net namespace to another, we move the RDMA
> >> device into that net namespace
> >>
> >> What do you think ?
> >>
> >> No. one device is not supposed to move other devices.
> >> Every device is independent that should be moved by explicit command.
> >>
> >> Can you show us where we can find this rule "Every device is
> >> independent that should be moved by explicit command."?
> >>
> >> Also changes like above breaks the existing orchestration, it no-go.
> >>
> >> In a RoCE device, ib device is related with the net device. When a
> >> net device is moved to a new net namespace, if the ib device is not
> >> in the same net device, how to make ib device work?
> >
> > RDMA device should also be moved to the same net namespace as that of
> netdev.
> 
> sure. I know the following commands.
> 
> In my commits, the process of moving IB devices to the same net namespace
> with net devices is automatically finished.
> 
> Is it OK?
> 
No. 
Change like this breaks the user space who expect to move the rdma device to the net namespace explicitly.
It wont find the device which got moved as part of some other device movement.
Currently define scheme covers at least 3 different types of RDMA devices.
1. IB and IPoIB
2. RoCE
3. iWarp

Each has somewhat different relation to their net device.

Re: [PATCH 0/3] RDMA net namespace

[yanjun.zhu]

October 27, 2022 11:21 AM, "Parav Pandit" <parav@nvidia.com> wrote:

>> From: yanjun.zhu@linux.dev <yanjun.zhu@linux.dev>
>> Sent: Wednesday, October 26, 2022 11:17 PM
>> 
>> October 27, 2022 11:10 AM, "Parav Pandit" <parav@nvidia.com> wrote:
>> 
>> From: yanjun.zhu@linux.dev <yanjun.zhu@linux.dev>
>> Sent: Wednesday, October 26, 2022 11:08 PM
>> 
>> October 27, 2022 11:01 AM, "Parav Pandit" <parav@nvidia.com> wrote:
>> 
>> From: Dust Li <dust.li@linux.alibaba.com>
>> Sent: Wednesday, October 26, 2022 10:31 PM
>> 
>> 2. else we are in
>> exclusive mode. When the corresponding netdevice of the RoCE or iWarp
>> device is moved from one net namespace to another, we move the RDMA
>> device into that net namespace
>> 
>> What do you think ?
>> 
>> No. one device is not supposed to move other devices.
>> Every device is independent that should be moved by explicit command.
>> 
>> Can you show us where we can find this rule "Every device is
>> independent that should be moved by explicit command."?
>> 
>> Also changes like above breaks the existing orchestration, it no-go.
>> 
>> In a RoCE device, ib device is related with the net device. When a
>> net device is moved to a new net namespace, if the ib device is not
>> in the same net device, how to make ib device work?
>> 
>> RDMA device should also be moved to the same net namespace as that of
>> netdev.
>> 
>> sure. I know the following commands.
>> 
>> In my commits, the process of moving IB devices to the same net namespace
>> with net devices is automatically finished.
>> 
>> Is it OK?
> 
> No.
> Change like this breaks the user space who expect to move the rdma device to the net namespace
> explicitly.

Which specification makes this kind of rule? Where can we find it?

> It wont find the device which got moved as part of some other device movement.
> Currently define scheme covers at least 3 different types of RDMA devices.
> 1. IB and IPoIB
> 2. RoCE
> 3. iWarp
> 
> Each has somewhat different relation to their net device.

IPoIB, RoCE and iWarp are somewhat different relation to their net device.
To RoCE and iWarp devices, ib devices should be the same net namespace with the related net devices.
Or else we can not make ib devices work. This is why I send out these commits.

Zhu Yanjun

RE: [PATCH 0/3] RDMA net namespace

[Parav Pandit]

> From: yanjun.zhu@linux.dev <yanjun.zhu@linux.dev>
> Sent: Wednesday, October 26, 2022 11:39 PM
> 
> October 27, 2022 11:21 AM, "Parav Pandit" <parav@nvidia.com> wrote:
> 
> >> From: yanjun.zhu@linux.dev <yanjun.zhu@linux.dev>
> >> Sent: Wednesday, October 26, 2022 11:17 PM
> >>
> >> October 27, 2022 11:10 AM, "Parav Pandit" <parav@nvidia.com> wrote:
> >>
> >> From: yanjun.zhu@linux.dev <yanjun.zhu@linux.dev>
> >> Sent: Wednesday, October 26, 2022 11:08 PM
> >>
> >> October 27, 2022 11:01 AM, "Parav Pandit" <parav@nvidia.com> wrote:
> >>
> >> From: Dust Li <dust.li@linux.alibaba.com>
> >> Sent: Wednesday, October 26, 2022 10:31 PM
> >>
> >> 2. else we are in
> >> exclusive mode. When the corresponding netdevice of the RoCE or iWarp
> >> device is moved from one net namespace to another, we move the RDMA
> >> device into that net namespace
> >>
> >> What do you think ?
> >>
> >> No. one device is not supposed to move other devices.
> >> Every device is independent that should be moved by explicit command.
> >>
> >> Can you show us where we can find this rule "Every device is
> >> independent that should be moved by explicit command."?
> >>
> >> Also changes like above breaks the existing orchestration, it no-go.
> >>
> >> In a RoCE device, ib device is related with the net device. When a
> >> net device is moved to a new net namespace, if the ib device is not
> >> in the same net device, how to make ib device work?
> >>
> >> RDMA device should also be moved to the same net namespace as that of
> >> netdev.
> >>
> >> sure. I know the following commands.
> >>
> >> In my commits, the process of moving IB devices to the same net
> >> namespace with net devices is automatically finished.
> >>
> >> Is it OK?
> >
> > No.
> > Change like this breaks the user space who expect to move the rdma
> > device to the net namespace explicitly.
> 
> Which specification makes this kind of rule? Where can we find it?
>
Existing ABI defines this which exists for many years now.
There is no Linux kernel subsystem or module to my knowledge that attempt to move multiple devices using single command.
When user executes command , user explicitly give device name "foo", only "foo" should move.
Other loosely coupled device whose name is not specified in the ip command which has a different life cycle should not move along with "foo".

You are trying to define the new rule that breaks the existing ABI and the iproute2 (ip and rdma) command semantics.
It is implicit that when command is issued on device A, operate on device A. This is part of iproute2 functioning.

> > It wont find the device which got moved as part of some other device
> movement.
> > Currently define scheme covers at least 3 different types of RDMA devices.
> > 1. IB and IPoIB
> > 2. RoCE
> > 3. iWarp
> >
> > Each has somewhat different relation to their net device.
> 
> IPoIB, RoCE and iWarp are somewhat different relation to their net device.
> To RoCE and iWarp devices, ib devices should be the same net namespace
> with the related net devices.
> Or else we can not make ib devices work. This is why I send out these
> commits.
So please move the rdma device also to the desired net namespace and it will work.

Re: [PATCH 0/3] RDMA net namespace

[yanjun.zhu]

October 27, 2022 11:48 AM, "Parav Pandit" <parav@nvidia.com> wrote:

>> From: yanjun.zhu@linux.dev <yanjun.zhu@linux.dev>
>> Sent: Wednesday, October 26, 2022 11:39 PM
>> 
>> October 27, 2022 11:21 AM, "Parav Pandit" <parav@nvidia.com> wrote:
>> 
>> From: yanjun.zhu@linux.dev <yanjun.zhu@linux.dev>
>> Sent: Wednesday, October 26, 2022 11:17 PM
>> 
>> October 27, 2022 11:10 AM, "Parav Pandit" <parav@nvidia.com> wrote:
>> 
>> From: yanjun.zhu@linux.dev <yanjun.zhu@linux.dev>
>> Sent: Wednesday, October 26, 2022 11:08 PM
>> 
>> October 27, 2022 11:01 AM, "Parav Pandit" <parav@nvidia.com> wrote:
>> 
>> From: Dust Li <dust.li@linux.alibaba.com>
>> Sent: Wednesday, October 26, 2022 10:31 PM
>> 
>> 2. else we are in
>> exclusive mode. When the corresponding netdevice of the RoCE or iWarp
>> device is moved from one net namespace to another, we move the RDMA
>> device into that net namespace
>> 
>> What do you think ?
>> 
>> No. one device is not supposed to move other devices.
>> Every device is independent that should be moved by explicit command.
>> 
>> Can you show us where we can find this rule "Every device is
>> independent that should be moved by explicit command."?
>> 
>> Also changes like above breaks the existing orchestration, it no-go.
>> 
>> In a RoCE device, ib device is related with the net device. When a
>> net device is moved to a new net namespace, if the ib device is not
>> in the same net device, how to make ib device work?
>> 
>> RDMA device should also be moved to the same net namespace as that of
>> netdev.
>> 
>> sure. I know the following commands.
>> 
>> In my commits, the process of moving IB devices to the same net
>> namespace with net devices is automatically finished.
>> 
>> Is it OK?
>> 
>> No.
>> Change like this breaks the user space who expect to move the rdma
>> device to the net namespace explicitly.
>> 
>> Which specification makes this kind of rule? Where can we find it?
> 
> Existing ABI defines this which exists for many years now.

About ABI, I read through this link https://en.wikipedia.org/wiki/Application_binary_interface

Details covered by an ABI include the following:

Processor instruction set, with details like register file structure, stack organization, memory access types, etc.

Sizes, layouts, and alignments of basic data types that the processor can directly access

Calling convention, which controls how the arguments of functions are passed, and return values retrieved; for example, it controls the following:
  Whether all parameters are passed on the stack, or some are passed in registers
  Which registers are used for which function parameters
  Whether the first function parameter passed on the stack is pushed first or last

How an application should make system calls to the operating system, and if the ABI specifies direct system calls rather than procedure calls to system call stubs, the system call numbers

In the case of a complete operating system ABI, the binary format of object files, program libraries, etc.

And I do not find the rule that you mentioned.

> There is no Linux kernel subsystem or module to my knowledge that attempt to move multiple devices
> using single command.
> When user executes command , user explicitly give device name "foo", only "foo" should move.
> Other loosely coupled device whose name is not specified in the ip command which has a different
> life cycle should not move along with "foo".
> 
> You are trying to define the new rule that breaks the existing ABI and the iproute2 (ip and rdma)
> command semantics.
> It is implicit that when command is issued on device A, operate on device A. This is part of
> iproute2 functioning.

About iproute2, I read this link https://wiki.linuxfoundation.org/networking/iproute2#documentation

There is no rules that you mentioned.

This rule is defined explicitly or implicitly?

Zhu Yanjun
> 
>> It wont find the device which got moved as part of some other device
>> movement.
>> Currently define scheme covers at least 3 different types of RDMA devices.
>> 1. IB and IPoIB
>> 2. RoCE
>> 3. iWarp
>> 
>> Each has somewhat different relation to their net device.
>> 
>> IPoIB, RoCE and iWarp are somewhat different relation to their net device.
>> To RoCE and iWarp devices, ib devices should be the same net namespace
>> with the related net devices.
>> Or else we can not make ib devices work. This is why I send out these
>> commits.
> 
> So please move the rdma device also to the desired net namespace and it will work.

RE: [PATCH 0/3] RDMA net namespace

[Parav Pandit]

> From: yanjun.zhu@linux.dev <yanjun.zhu@linux.dev>
> Sent: Thursday, October 27, 2022 2:02 AM
> To: Parav Pandit <parav@nvidia.com>; dust.li@linux.alibaba.com; Zhu Yanjun
> <yanjun.zhu@intel.com>; jgg@ziepe.ca; leon@kernel.org; linux-
> rdma@vger.kernel.org
> Subject: Re: [PATCH 0/3] RDMA net namespace
> 
> October 27, 2022 11:48 AM, "Parav Pandit" <parav@nvidia.com> wrote:
> 
> >> From: yanjun.zhu@linux.dev <yanjun.zhu@linux.dev>
> >> Sent: Wednesday, October 26, 2022 11:39 PM
> >>
> >> October 27, 2022 11:21 AM, "Parav Pandit" <parav@nvidia.com> wrote:
> >>
> >> From: yanjun.zhu@linux.dev <yanjun.zhu@linux.dev>
> >> Sent: Wednesday, October 26, 2022 11:17 PM
> >>
> >> October 27, 2022 11:10 AM, "Parav Pandit" <parav@nvidia.com> wrote:
> >>
> >> From: yanjun.zhu@linux.dev <yanjun.zhu@linux.dev>
> >> Sent: Wednesday, October 26, 2022 11:08 PM
> >>
> >> October 27, 2022 11:01 AM, "Parav Pandit" <parav@nvidia.com> wrote:
> >>
> >> From: Dust Li <dust.li@linux.alibaba.com>
> >> Sent: Wednesday, October 26, 2022 10:31 PM
> >>
> >> 2. else we are in
> >> exclusive mode. When the corresponding netdevice of the RoCE or iWarp
> >> device is moved from one net namespace to another, we move the
> RDMA
> >> device into that net namespace
> >>
> >> What do you think ?
> >>
> >> No. one device is not supposed to move other devices.
> >> Every device is independent that should be moved by explicit command.
> >>
> >> Can you show us where we can find this rule "Every device is
> >> independent that should be moved by explicit command."?
> >>
> >> Also changes like above breaks the existing orchestration, it no-go.
> >>
> 
> And I do not find the rule that you mentioned.
> 
> > There is no Linux kernel subsystem or module to my knowledge that
> > attempt to move multiple devices using single command.
> > When user executes command , user explicitly give device name "foo",
> only "foo" should move.
> > Other loosely coupled device whose name is not specified in the ip
> > command which has a different life cycle should not move along with "foo".
> >
> > You are trying to define the new rule that breaks the existing ABI and
> > the iproute2 (ip and rdma) command semantics.
> > It is implicit that when command is issued on device A, operate on
> > device A. This is part of
> > iproute2 functioning.
> 
> About iproute2, I read this link
> https://wiki.linuxfoundation.org/networking/iproute2#documentation
> 
> There is no rules that you mentioned.
> 
> This rule is defined explicitly or implicitly?
> 
Wiki pages links are not the documentation.
Man pages of the iproute2 is documentation of iproute2 at [1] and [2].

[1] https://man7.org/linux/man-pages/man8/rdma-system.8.html
[2] https://man7.org/linux/man-pages/man8/rdma-dev.8.html

As I explained, the explicit rule that you are looking for that say "when I modify device foo, it can also modifies the device bar".
Because no part of the Linux kernel does that usually, unless the device is representor/control object etc or has parent/child relationship.
It is fundamental to a command definition, not a matter of explicit or implicit.

And clearly in this discussion foo and bar are loosely coupled network devices, one is not controlling the other.

Also, a rdma device is attached to multiple net devices, primary and other upper devices such as vlan, macvlan etc.

Re: [PATCH 0/3] RDMA net namespace

[Yanjun Zhu]

在 2022/10/27 22:06, Parav Pandit 写道:
>> From: yanjun.zhu@linux.dev <yanjun.zhu@linux.dev>
>> Sent: Thursday, October 27, 2022 2:02 AM
>> To: Parav Pandit <parav@nvidia.com>; dust.li@linux.alibaba.com; Zhu Yanjun
>> <yanjun.zhu@intel.com>; jgg@ziepe.ca; leon@kernel.org; linux-
>> rdma@vger.kernel.org
>> Subject: Re: [PATCH 0/3] RDMA net namespace
>>
>> October 27, 2022 11:48 AM, "Parav Pandit" <parav@nvidia.com> wrote:
>>
>>>> From: yanjun.zhu@linux.dev <yanjun.zhu@linux.dev>
>>>> Sent: Wednesday, October 26, 2022 11:39 PM
>>>>
>>>> October 27, 2022 11:21 AM, "Parav Pandit" <parav@nvidia.com> wrote:
>>>>
>>>> From: yanjun.zhu@linux.dev <yanjun.zhu@linux.dev>
>>>> Sent: Wednesday, October 26, 2022 11:17 PM
>>>>
>>>> October 27, 2022 11:10 AM, "Parav Pandit" <parav@nvidia.com> wrote:
>>>>
>>>> From: yanjun.zhu@linux.dev <yanjun.zhu@linux.dev>
>>>> Sent: Wednesday, October 26, 2022 11:08 PM
>>>>
>>>> October 27, 2022 11:01 AM, "Parav Pandit" <parav@nvidia.com> wrote:
>>>>
>>>> From: Dust Li <dust.li@linux.alibaba.com>
>>>> Sent: Wednesday, October 26, 2022 10:31 PM
>>>>
>>>> 2. else we are in
>>>> exclusive mode. When the corresponding netdevice of the RoCE or iWarp
>>>> device is moved from one net namespace to another, we move the
>> RDMA
>>>> device into that net namespace
>>>>
>>>> What do you think ?
>>>>
>>>> No. one device is not supposed to move other devices.
>>>> Every device is independent that should be moved by explicit command.
>>>>
>>>> Can you show us where we can find this rule "Every device is
>>>> independent that should be moved by explicit command."?
>>>>
>>>> Also changes like above breaks the existing orchestration, it no-go.
>>>>
>> And I do not find the rule that you mentioned.
>>
>>> There is no Linux kernel subsystem or module to my knowledge that
>>> attempt to move multiple devices using single command.
>>> When user executes command , user explicitly give device name "foo",
>> only "foo" should move.
>>> Other loosely coupled device whose name is not specified in the ip
>>> command which has a different life cycle should not move along with "foo".
>>>
>>> You are trying to define the new rule that breaks the existing ABI and
>>> the iproute2 (ip and rdma) command semantics.
>>> It is implicit that when command is issued on device A, operate on
>>> device A. This is part of
>>> iproute2 functioning.
>> About iproute2, I read this link
>> https://wiki.linuxfoundation.org/networking/iproute2#documentation
>>
>> There is no rules that you mentioned.
>>
>> This rule is defined explicitly or implicitly?
>>
> Wiki pages links are not the documentation.
> Man pages of the iproute2 is documentation of iproute2 at [1] and [2].
>
> [1] https://man7.org/linux/man-pages/man8/rdma-system.8.html
> [2] https://man7.org/linux/man-pages/man8/rdma-dev.8.html
>
> As I explained, the explicit rule that you are looking for that say "when I modify device foo, it can also modifies the device bar".
> Because no part of the Linux kernel does that usually, unless the device is representor/control object etc or has parent/child relationship.
> It is fundamental to a command definition, not a matter of explicit or implicit.

 From the ABI, iproute2 and current rdma command links, I can not find 
the rule that you mentioned.

Can you tell me the exact link that make such definition?

>
> And clearly in this discussion foo and bar are loosely coupled network devices, one is not controlling the other.
>
> Also, a rdma device is attached to multiple net devices, primary and other upper devices such as vlan, macvlan etc.

To a RoCE device, how to attach a rdma device to vlan, macvlan?

To "a rdma device is attached to multiple net devices, primary and other 
upper devices such as vlan, macvlan etc",

Can you show us an example? The rdma device is RoCE device, iWarp or 
ipoib device?

Zhu Yanjun

>
>

RE: [PATCH 0/3] RDMA net namespace

[Parav Pandit]

> From: Yanjun Zhu <yanjun.zhu@linux.dev>
> Sent: Thursday, October 27, 2022 11:21 PM
> 
> 
> 在 2022/10/27 22:06, Parav Pandit 写道:
> >> From: yanjun.zhu@linux.dev <yanjun.zhu@linux.dev>
> >> Sent: Thursday, October 27, 2022 2:02 AM
> >> To: Parav Pandit <parav@nvidia.com>; dust.li@linux.alibaba.com; Zhu
> >> Yanjun <yanjun.zhu@intel.com>; jgg@ziepe.ca; leon@kernel.org; linux-
> >> rdma@vger.kernel.org
> >> Subject: Re: [PATCH 0/3] RDMA net namespace
> >>
> >> October 27, 2022 11:48 AM, "Parav Pandit" <parav@nvidia.com> wrote:
> >>
> >>>> From: yanjun.zhu@linux.dev <yanjun.zhu@linux.dev>
> >>>> Sent: Wednesday, October 26, 2022 11:39 PM
> >>>>
> >>>> October 27, 2022 11:21 AM, "Parav Pandit" <parav@nvidia.com> wrote:
> >>>>
> >>>> From: yanjun.zhu@linux.dev <yanjun.zhu@linux.dev>
> >>>> Sent: Wednesday, October 26, 2022 11:17 PM
> >>>>
> >>>> October 27, 2022 11:10 AM, "Parav Pandit" <parav@nvidia.com> wrote:
> >>>>
> >>>> From: yanjun.zhu@linux.dev <yanjun.zhu@linux.dev>
> >>>> Sent: Wednesday, October 26, 2022 11:08 PM
> >>>>
> >>>> October 27, 2022 11:01 AM, "Parav Pandit" <parav@nvidia.com> wrote:
> >>>>
> >>>> From: Dust Li <dust.li@linux.alibaba.com>
> >>>> Sent: Wednesday, October 26, 2022 10:31 PM
> >>>>
> >>>> 2. else we are in
> >>>> exclusive mode. When the corresponding netdevice of the RoCE or
> >>>> iWarp device is moved from one net namespace to another, we move
> >>>> the
> >> RDMA
> >>>> device into that net namespace
> >>>>
> >>>> What do you think ?
> >>>>
> >>>> No. one device is not supposed to move other devices.
> >>>> Every device is independent that should be moved by explicit
> command.
> >>>>
> >>>> Can you show us where we can find this rule "Every device is
> >>>> independent that should be moved by explicit command."?
> >>>>
> >>>> Also changes like above breaks the existing orchestration, it no-go.
> >>>>
> >> And I do not find the rule that you mentioned.
> >>
> >>> There is no Linux kernel subsystem or module to my knowledge that
> >>> attempt to move multiple devices using single command.
> >>> When user executes command , user explicitly give device name "foo",
> >> only "foo" should move.
> >>> Other loosely coupled device whose name is not specified in the ip
> >>> command which has a different life cycle should not move along with
> "foo".
> >>>
> >>> You are trying to define the new rule that breaks the existing ABI
> >>> and the iproute2 (ip and rdma) command semantics.
> >>> It is implicit that when command is issued on device A, operate on
> >>> device A. This is part of
> >>> iproute2 functioning.
> >> About iproute2, I read this link
> >> https://wiki.linuxfoundation.org/networking/iproute2#documentation
> >>
> >> There is no rules that you mentioned.
> >>
> >> This rule is defined explicitly or implicitly?
> >>
> > Wiki pages links are not the documentation.
> > Man pages of the iproute2 is documentation of iproute2 at [1] and [2].
> >
> > [1] https://man7.org/linux/man-pages/man8/rdma-system.8.html
> > [2] https://man7.org/linux/man-pages/man8/rdma-dev.8.html
> >
> > As I explained, the explicit rule that you are looking for that say "when I
> modify device foo, it can also modifies the device bar".
> > Because no part of the Linux kernel does that usually, unless the device is
> representor/control object etc or has parent/child relationship.
> > It is fundamental to a command definition, not a matter of explicit or
> implicit.
> 
>  From the ABI, iproute2 and current rdma command links, I can not find the
> rule that you mentioned.
> 
> Can you tell me the exact link that make such definition?
> 
I explained you already above. You are repeating your weird question.

Can you show one iproute2 example, where you specify a command on device A, and kernel operates on device, A, P, Q, R?
This is the attempt you are trying to do for unknown reasons.

So, can you please explain, what is the problem in using existing rdma dev commands that move rdma device to net namespace?

> >
> > And clearly in this discussion foo and bar are loosely coupled network
> devices, one is not controlling the other.
> >
> > Also, a rdma device is attached to multiple net devices, primary and other
> upper devices such as vlan, macvlan etc.
> 
> To a RoCE device, how to attach a rdma device to vlan, macvlan?
> 
> To "a rdma device is attached to multiple net devices, primary and other
> upper devices such as vlan, macvlan etc",
> 
> Can you show us an example? The rdma device is RoCE device, iWarp or ipoib
> device?
Rdma device is roce device.
Add vlan, macvlan device on top of the netdevice linked to the roce device using iproute2.

Re: [PATCH 0/3] RDMA net namespace

[Yanjun Zhu]

在 2022/10/28 11:31, Parav Pandit 写道:
>> From: Yanjun Zhu <yanjun.zhu@linux.dev>
>> Sent: Thursday, October 27, 2022 11:21 PM
>>
>>
>> 在 2022/10/27 22:06, Parav Pandit 写道:
>>>> From: yanjun.zhu@linux.dev <yanjun.zhu@linux.dev>
>>>> Sent: Thursday, October 27, 2022 2:02 AM
>>>> To: Parav Pandit <parav@nvidia.com>; dust.li@linux.alibaba.com; Zhu
>>>> Yanjun <yanjun.zhu@intel.com>; jgg@ziepe.ca; leon@kernel.org; linux-
>>>> rdma@vger.kernel.org
>>>> Subject: Re: [PATCH 0/3] RDMA net namespace
>>>>
>>>> October 27, 2022 11:48 AM, "Parav Pandit" <parav@nvidia.com> wrote:
>>>>
>>>>>> From: yanjun.zhu@linux.dev <yanjun.zhu@linux.dev>
>>>>>> Sent: Wednesday, October 26, 2022 11:39 PM
>>>>>>
>>>>>> October 27, 2022 11:21 AM, "Parav Pandit" <parav@nvidia.com> wrote:
>>>>>>
>>>>>> From: yanjun.zhu@linux.dev <yanjun.zhu@linux.dev>
>>>>>> Sent: Wednesday, October 26, 2022 11:17 PM
>>>>>>
>>>>>> October 27, 2022 11:10 AM, "Parav Pandit" <parav@nvidia.com> wrote:
>>>>>>
>>>>>> From: yanjun.zhu@linux.dev <yanjun.zhu@linux.dev>
>>>>>> Sent: Wednesday, October 26, 2022 11:08 PM
>>>>>>
>>>>>> October 27, 2022 11:01 AM, "Parav Pandit" <parav@nvidia.com> wrote:
>>>>>>
>>>>>> From: Dust Li <dust.li@linux.alibaba.com>
>>>>>> Sent: Wednesday, October 26, 2022 10:31 PM
>>>>>>
>>>>>> 2. else we are in
>>>>>> exclusive mode. When the corresponding netdevice of the RoCE or
>>>>>> iWarp device is moved from one net namespace to another, we move
>>>>>> the
>>>> RDMA
>>>>>> device into that net namespace
>>>>>>
>>>>>> What do you think ?
>>>>>>
>>>>>> No. one device is not supposed to move other devices.
>>>>>> Every device is independent that should be moved by explicit
>> command.
>>>>>> Can you show us where we can find this rule "Every device is
>>>>>> independent that should be moved by explicit command."?
>>>>>>
>>>>>> Also changes like above breaks the existing orchestration, it no-go.
>>>>>>
>>>> And I do not find the rule that you mentioned.
>>>>
>>>>> There is no Linux kernel subsystem or module to my knowledge that
>>>>> attempt to move multiple devices using single command.
>>>>> When user executes command , user explicitly give device name "foo",
>>>> only "foo" should move.
>>>>> Other loosely coupled device whose name is not specified in the ip
>>>>> command which has a different life cycle should not move along with
>> "foo".
>>>>> You are trying to define the new rule that breaks the existing ABI
>>>>> and the iproute2 (ip and rdma) command semantics.
>>>>> It is implicit that when command is issued on device A, operate on
>>>>> device A. This is part of
>>>>> iproute2 functioning.
>>>> About iproute2, I read this link
>>>> https://wiki.linuxfoundation.org/networking/iproute2#documentation
>>>>
>>>> There is no rules that you mentioned.
>>>>
>>>> This rule is defined explicitly or implicitly?
>>>>
>>> Wiki pages links are not the documentation.
>>> Man pages of the iproute2 is documentation of iproute2 at [1] and [2].
>>>
>>> [1] https://man7.org/linux/man-pages/man8/rdma-system.8.html
>>> [2] https://man7.org/linux/man-pages/man8/rdma-dev.8.html
>>>
>>> As I explained, the explicit rule that you are looking for that say "when I
>> modify device foo, it can also modifies the device bar".
>>> Because no part of the Linux kernel does that usually, unless the device is
>> representor/control object etc or has parent/child relationship.
>>> It is fundamental to a command definition, not a matter of explicit or
>> implicit.
>>
>>   From the ABI, iproute2 and current rdma command links, I can not find the
>> rule that you mentioned.
>>
>> Can you tell me the exact link that make such definition?
>>
> I explained you already above. You are repeating your weird question.

You mentioned that it is a rule. I just want to know where it is defined.

>
> Can you show one iproute2 example, where you specify a command on device A, and kernel operates on device, A, P, Q, R?

When you add a net devices to a bonding device, you will find changes on 
the bonding device and the net devices.

Or some other commands like this.

Zhu Yanjun

> This is the attempt you are trying to do for unknown reasons.
>
> So, can you please explain, what is the problem in using existing rdma dev commands that move rdma device to net namespace?
>
>>> And clearly in this discussion foo and bar are loosely coupled network
>> devices, one is not controlling the other.
>>> Also, a rdma device is attached to multiple net devices, primary and other
>> upper devices such as vlan, macvlan etc.
>>
>> To a RoCE device, how to attach a rdma device to vlan, macvlan?
>>
>> To "a rdma device is attached to multiple net devices, primary and other
>> upper devices such as vlan, macvlan etc",
>>
>> Can you show us an example? The rdma device is RoCE device, iWarp or ipoib
>> device?
> Rdma device is roce device.
> Add vlan, macvlan device on top of the netdevice linked to the roce device using iproute2.

RE: [PATCH 0/3] RDMA net namespace

[Parav Pandit]

> From: Yanjun Zhu <yanjun.zhu@linux.dev>
> Sent: Thursday, October 27, 2022 11:49 PM

> > Can you show one iproute2 example, where you specify a command on
> device A, and kernel operates on device, A, P, Q, R?
> 
> When you add a net devices to a bonding device, you will find changes on the
> bonding device and the net devices.
> 
> Or some other commands like this.
That doesn’t count as I explained that it is more parent-child or similar control relationship, unlike rdma and netdevice as loosely coupled devices.

Also when you move a underlaying netdev interface of bond device, bond device doesn’t automatically move to new net ns.

Re: RE: [PATCH 0/3] RDMA net namespace

[Yanjun Zhu]

在 2022/10/28 11:58, Parav Pandit 写道:
> 
>> From: Yanjun Zhu <yanjun.zhu@linux.dev>
>> Sent: Thursday, October 27, 2022 11:49 PM
> 
>>> Can you show one iproute2 example, where you specify a command on
>> device A, and kernel operates on device, A, P, Q, R?
>>
>> When you add a net devices to a bonding device, you will find changes on the
>> bonding device and the net devices.
>>
>> Or some other commands like this.
> That doesn’t count as I explained that it is more parent-child or similar control relationship, unlike rdma and netdevice as loosely coupled devices.

OK. Follow your advice.

Zhu Yanjun

> 
> Also when you move a underlaying netdev interface of bond device, bond device doesn’t automatically move to new net ns.