[PATCH] rdma: not display the rdma link in other net namespace

[PATCH] rdma: not display the rdma link in other net namespace

[yanjun.zhu]

From: Zhu Yanjun <yanjun.zhu@linux.dev>

When the net devices are moved to another net namespace, the command
"rdma link" should not dispaly the rdma link about this net device.

For example, when the net device eno12399 is moved to net namespace net0
from init_net, the rdma link of eno12399 should not display in init_net.

Before this change:

Init_net:

link roceo12399/1 state DOWN physical_state DISABLED  <---should not display
link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409
link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0
link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1

net0:

link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399
link roceo12409/1 state DOWN physical_state DISABLED <---should not display
link rocep202s0f0/1 state DOWN physical_state DISABLED <---should not display
link rocep202s0f1/1 state ACTIVE physical_state LINK_UP <---should not display

After this change

Init_net:

link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409
link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0
link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1

net0:

link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399

Fixes: da990ab40a92 ("rdma: Add link object")
Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev>
---
 rdma/link.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/rdma/link.c b/rdma/link.c
index bf24b849..449a7636 100644
--- a/rdma/link.c
+++ b/rdma/link.c
@@ -238,6 +238,9 @@ static int link_parse_cb(const struct nlmsghdr *nlh, void *data)
 		return MNL_CB_ERROR;
 	}
 
+	if (!tb[RDMA_NLDEV_ATTR_NDEV_NAME] || !tb[RDMA_NLDEV_ATTR_NDEV_INDEX])
+		return MNL_CB_OK;
+
 	idx = mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_DEV_INDEX]);
 	port = mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_PORT_INDEX]);
 	name = mnl_attr_get_str(tb[RDMA_NLDEV_ATTR_DEV_NAME]);
-- 
2.27.0

Re: [PATCH] rdma: not display the rdma link in other net namespace

[Yanjun Zhu]

在 2022/9/26 10:40, yanjun.zhu@linux.dev 写道:
> From: Zhu Yanjun <yanjun.zhu@linux.dev>
> 
> When the net devices are moved to another net namespace, the command
> "rdma link" should not dispaly the rdma link about this net device.
> 
> For example, when the net device eno12399 is moved to net namespace net0
> from init_net, the rdma link of eno12399 should not display in init_net.
> 
> Before this change:
> 
> Init_net:
> 
> link roceo12399/1 state DOWN physical_state DISABLED  <---should not display
> link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409
> link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0
> link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1
> 
> net0:
> 
> link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399
> link roceo12409/1 state DOWN physical_state DISABLED <---should not display
> link rocep202s0f0/1 state DOWN physical_state DISABLED <---should not display
> link rocep202s0f1/1 state ACTIVE physical_state LINK_UP <---should not display
> 
> After this change
> 
> Init_net:
> 
> link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409
> link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0
> link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1
> 
> net0:
> 
> link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399
> 
> Fixes: da990ab40a92 ("rdma: Add link object")
> Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev>

Hi Leon

This commit is to fix a problem in "rdma link show" of iproute2.
I do not know the maillist of iproute2. So I send the commit here.
If you know the maillist, please let me know.

Thanks and Regards,
Zhu Yanjun

> ---
>   rdma/link.c | 3 +++
>   1 file changed, 3 insertions(+)
> 
> diff --git a/rdma/link.c b/rdma/link.c
> index bf24b849..449a7636 100644
> --- a/rdma/link.c
> +++ b/rdma/link.c
> @@ -238,6 +238,9 @@ static int link_parse_cb(const struct nlmsghdr *nlh, void *data)
>   		return MNL_CB_ERROR;
>   	}
>   
> +	if (!tb[RDMA_NLDEV_ATTR_NDEV_NAME] || !tb[RDMA_NLDEV_ATTR_NDEV_INDEX])
> +		return MNL_CB_OK;
> +
>   	idx = mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_DEV_INDEX]);
>   	port = mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_PORT_INDEX]);
>   	name = mnl_attr_get_str(tb[RDMA_NLDEV_ATTR_DEV_NAME]);

Re: [PATCH] rdma: not display the rdma link in other net namespace

[Leon Romanovsky]

On Sun, Sep 25, 2022 at 10:40:33PM -0400, yanjun.zhu@linux.dev wrote:
> From: Zhu Yanjun <yanjun.zhu@linux.dev>
> 
> When the net devices are moved to another net namespace, the command
> "rdma link" should not dispaly the rdma link about this net device.
> 
> For example, when the net device eno12399 is moved to net namespace net0
> from init_net, the rdma link of eno12399 should not display in init_net.
> 
> Before this change:
> 
> Init_net:
> 
> link roceo12399/1 state DOWN physical_state DISABLED  <---should not display
> link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409
> link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0
> link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1
> 
> net0:
> 
> link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399
> link roceo12409/1 state DOWN physical_state DISABLED <---should not display
> link rocep202s0f0/1 state DOWN physical_state DISABLED <---should not display
> link rocep202s0f1/1 state ACTIVE physical_state LINK_UP <---should not display
> 
> After this change
> 
> Init_net:
> 
> link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409
> link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0
> link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1
> 
> net0:
> 
> link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399
> 
> Fixes: da990ab40a92 ("rdma: Add link object")
> Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev>
> ---
>  rdma/link.c | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/rdma/link.c b/rdma/link.c
> index bf24b849..449a7636 100644
> --- a/rdma/link.c
> +++ b/rdma/link.c
> @@ -238,6 +238,9 @@ static int link_parse_cb(const struct nlmsghdr *nlh, void *data)
>  		return MNL_CB_ERROR;
>  	}
>  
> +	if (!tb[RDMA_NLDEV_ATTR_NDEV_NAME] || !tb[RDMA_NLDEV_ATTR_NDEV_INDEX])
> +		return MNL_CB_OK;
> +

Regarding your question where it should go in addition to RDMA, the answer
is netdev ML. The rdmatool is part of iproute2 and the relevant maintainers
should be CCed.

Regarding the change, I don't think that it is right. User space tool is
a simple viewer of data returned from the kernel. It is not a mistake to
return device without netdev.

Thanks

Re: [PATCH] rdma: not display the rdma link in other net namespace

[Yanjun Zhu]

在 2022/9/27 18:34, Leon Romanovsky 写道:
> On Sun, Sep 25, 2022 at 10:40:33PM -0400, yanjun.zhu@linux.dev wrote:
>> From: Zhu Yanjun <yanjun.zhu@linux.dev>
>>
>> When the net devices are moved to another net namespace, the command
>> "rdma link" should not dispaly the rdma link about this net device.
>>
>> For example, when the net device eno12399 is moved to net namespace net0
>> from init_net, the rdma link of eno12399 should not display in init_net.
>>
>> Before this change:
>>
>> Init_net:
>>
>> link roceo12399/1 state DOWN physical_state DISABLED  <---should not display
>> link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409
>> link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0
>> link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1
>>
>> net0:
>>
>> link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399
>> link roceo12409/1 state DOWN physical_state DISABLED <---should not display
>> link rocep202s0f0/1 state DOWN physical_state DISABLED <---should not display
>> link rocep202s0f1/1 state ACTIVE physical_state LINK_UP <---should not display
>>
>> After this change
>>
>> Init_net:
>>
>> link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409
>> link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0
>> link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1
>>
>> net0:
>>
>> link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399
>>
>> Fixes: da990ab40a92 ("rdma: Add link object")
>> Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev>
>> ---
>>   rdma/link.c | 3 +++
>>   1 file changed, 3 insertions(+)
>>
>> diff --git a/rdma/link.c b/rdma/link.c
>> index bf24b849..449a7636 100644
>> --- a/rdma/link.c
>> +++ b/rdma/link.c
>> @@ -238,6 +238,9 @@ static int link_parse_cb(const struct nlmsghdr *nlh, void *data)
>>   		return MNL_CB_ERROR;
>>   	}
>>   
>> +	if (!tb[RDMA_NLDEV_ATTR_NDEV_NAME] || !tb[RDMA_NLDEV_ATTR_NDEV_INDEX])
>> +		return MNL_CB_OK;
>> +
> Regarding your question where it should go in addition to RDMA, the answer
> is netdev ML. The rdmatool is part of iproute2 and the relevant maintainers
> should be CCed.
Thanks. I will also send it to netdev ML and CC the maintainers.
>
> Regarding the change, I don't think that it is right. User space tool is
> a simple viewer of data returned from the kernel. It is not a mistake to
> return device without netdev.

Normally a rdma link based on RoCEv2 should be with a NIC. This NIC device

will send/recv udp packets. With mellanox/intel NIC device, this net 
device also

do more work than sending/receiving packets.

 From this perspective, a rdma link is dependent on a net device.

In this problem, net device is moved to another net namespace. So it can 
not be

obtained.  And this rdma link can also not work in this net namespace.

So this rdma link should not appear in this net namespace. Or else, it 
would confuse

the user.

In fact, net namespace is a concept in tcp/ip stack. And it does not 
exist in rdma stack.

But rdma link based on RoCEv2 is dependent on network device. So when 
the net device

is moved to another net namespace, this rdma link should also be moved 
to another net namespace.

Zhu Yanjun

>
> Thanks

Re: [PATCH] rdma: not display the rdma link in other net namespace

[Leon Romanovsky]

On Tue, Sep 27, 2022 at 06:58:50PM +0800, Yanjun Zhu wrote:
> 
> 在 2022/9/27 18:34, Leon Romanovsky 写道:
> > On Sun, Sep 25, 2022 at 10:40:33PM -0400, yanjun.zhu@linux.dev wrote:
> > > From: Zhu Yanjun <yanjun.zhu@linux.dev>
> > > 
> > > When the net devices are moved to another net namespace, the command
> > > "rdma link" should not dispaly the rdma link about this net device.
> > > 
> > > For example, when the net device eno12399 is moved to net namespace net0
> > > from init_net, the rdma link of eno12399 should not display in init_net.
> > > 
> > > Before this change:
> > > 
> > > Init_net:
> > > 
> > > link roceo12399/1 state DOWN physical_state DISABLED  <---should not display
> > > link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409
> > > link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0
> > > link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1
> > > 
> > > net0:
> > > 
> > > link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399
> > > link roceo12409/1 state DOWN physical_state DISABLED <---should not display
> > > link rocep202s0f0/1 state DOWN physical_state DISABLED <---should not display
> > > link rocep202s0f1/1 state ACTIVE physical_state LINK_UP <---should not display
> > > 
> > > After this change
> > > 
> > > Init_net:
> > > 
> > > link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409
> > > link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0
> > > link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1
> > > 
> > > net0:
> > > 
> > > link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399
> > > 
> > > Fixes: da990ab40a92 ("rdma: Add link object")
> > > Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev>
> > > ---
> > >   rdma/link.c | 3 +++
> > >   1 file changed, 3 insertions(+)
> > > 
> > > diff --git a/rdma/link.c b/rdma/link.c
> > > index bf24b849..449a7636 100644
> > > --- a/rdma/link.c
> > > +++ b/rdma/link.c
> > > @@ -238,6 +238,9 @@ static int link_parse_cb(const struct nlmsghdr *nlh, void *data)
> > >   		return MNL_CB_ERROR;
> > >   	}
> > > +	if (!tb[RDMA_NLDEV_ATTR_NDEV_NAME] || !tb[RDMA_NLDEV_ATTR_NDEV_INDEX])
> > > +		return MNL_CB_OK;
> > > +
> > Regarding your question where it should go in addition to RDMA, the answer
> > is netdev ML. The rdmatool is part of iproute2 and the relevant maintainers
> > should be CCed.
> Thanks. I will also send it to netdev ML and CC the maintainers.
> > 
> > Regarding the change, I don't think that it is right. User space tool is
> > a simple viewer of data returned from the kernel. It is not a mistake to
> > return device without netdev.
> 
> Normally a rdma link based on RoCEv2 should be with a NIC. This NIC device
> 
> will send/recv udp packets. With mellanox/intel NIC device, this net device
> also
> 
> do more work than sending/receiving packets.
> 
> From this perspective, a rdma link is dependent on a net device.
> 
> In this problem, net device is moved to another net namespace. So it can not
> be
> 
> obtained.  And this rdma link can also not work in this net namespace.
> 
> So this rdma link should not appear in this net namespace. Or else, it would
> confuse
> 
> the user.
> 
> In fact, net namespace is a concept in tcp/ip stack. And it does not exist
> in rdma stack.

RDMA has two different net namespace mode: shared and exclusive.

In shared mode, the IB devices are shared across all net namespaces and
"moving" net device into different namespace just "hides" it, but don't
disconnect.

See comments around various usages of ib_devices_shared_netns variable.

Thanks

Re: [PATCH] rdma: not display the rdma link in other net namespace

[Yanjun Zhu]

在 2022/9/28 14:04, Leon Romanovsky 写道:
> On Tue, Sep 27, 2022 at 06:58:50PM +0800, Yanjun Zhu wrote:
>>
>> 在 2022/9/27 18:34, Leon Romanovsky 写道:
>>> On Sun, Sep 25, 2022 at 10:40:33PM -0400, yanjun.zhu@linux.dev wrote:
>>>> From: Zhu Yanjun <yanjun.zhu@linux.dev>
>>>>
>>>> When the net devices are moved to another net namespace, the command
>>>> "rdma link" should not dispaly the rdma link about this net device.
>>>>
>>>> For example, when the net device eno12399 is moved to net namespace net0
>>>> from init_net, the rdma link of eno12399 should not display in init_net.
>>>>
>>>> Before this change:
>>>>
>>>> Init_net:
>>>>
>>>> link roceo12399/1 state DOWN physical_state DISABLED  <---should not display
>>>> link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409
>>>> link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0
>>>> link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1
>>>>
>>>> net0:
>>>>
>>>> link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399
>>>> link roceo12409/1 state DOWN physical_state DISABLED <---should not display
>>>> link rocep202s0f0/1 state DOWN physical_state DISABLED <---should not display
>>>> link rocep202s0f1/1 state ACTIVE physical_state LINK_UP <---should not display
>>>>
>>>> After this change
>>>>
>>>> Init_net:
>>>>
>>>> link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409
>>>> link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0
>>>> link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1
>>>>
>>>> net0:
>>>>
>>>> link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399
>>>>
>>>> Fixes: da990ab40a92 ("rdma: Add link object")
>>>> Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev>
>>>> ---
>>>>    rdma/link.c | 3 +++
>>>>    1 file changed, 3 insertions(+)
>>>>
>>>> diff --git a/rdma/link.c b/rdma/link.c
>>>> index bf24b849..449a7636 100644
>>>> --- a/rdma/link.c
>>>> +++ b/rdma/link.c
>>>> @@ -238,6 +238,9 @@ static int link_parse_cb(const struct nlmsghdr *nlh, void *data)
>>>>    		return MNL_CB_ERROR;
>>>>    	}
>>>> +	if (!tb[RDMA_NLDEV_ATTR_NDEV_NAME] || !tb[RDMA_NLDEV_ATTR_NDEV_INDEX])
>>>> +		return MNL_CB_OK;
>>>> +
>>> Regarding your question where it should go in addition to RDMA, the answer
>>> is netdev ML. The rdmatool is part of iproute2 and the relevant maintainers
>>> should be CCed.
>> Thanks. I will also send it to netdev ML and CC the maintainers.
>>>
>>> Regarding the change, I don't think that it is right. User space tool is
>>> a simple viewer of data returned from the kernel. It is not a mistake to
>>> return device without netdev.
>>
>> Normally a rdma link based on RoCEv2 should be with a NIC. This NIC device
>>
>> will send/recv udp packets. With mellanox/intel NIC device, this net device
>> also
>>
>> do more work than sending/receiving packets.
>>
>>  From this perspective, a rdma link is dependent on a net device.
>>
>> In this problem, net device is moved to another net namespace. So it can not
>> be
>>
>> obtained.  And this rdma link can also not work in this net namespace.
>>
>> So this rdma link should not appear in this net namespace. Or else, it would
>> confuse
>>
>> the user.
>>
>> In fact, net namespace is a concept in tcp/ip stack. And it does not exist
>> in rdma stack.
> 
> RDMA has two different net namespace mode: shared and exclusive.

This is different from net namespace in network.

> 
> In shared mode, the IB devices are shared across all net namespaces and
> "moving" net device into different namespace just "hides" it, but don't
> disconnect.

In exclusive mode, the net device also hide. It is the same with shared 
mode.

# rdma system
netns exclusive copy-on-fork on
# rdma link
link roceo12399/1 state DOWN physical_state DISABLED
link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409
link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0
link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1

Is it better to append "exclusive" or "shared" in the end of the line?

For example,

Exclusive mode:

# rdma system
netns exclusive copy-on-fork on
# rdma link
link roceo12399/1 state DOWN physical_state DISABLED exclusive

Shared mode:

# rdma system
netns shared copy-on-fork on
# rdma link
link roceo12399/1 state DOWN physical_state DISABLED shared

Thanks and Regards
Zhu Yanjun
> 
> See comments around various usages of ib_devices_shared_netns variable.
> 
> Thanks

Re: [PATCH] rdma: not display the rdma link in other net namespace

[Leon Romanovsky]

On Fri, Sep 30, 2022 at 03:25:00PM +0800, Yanjun Zhu wrote:
> 在 2022/9/28 14:04, Leon Romanovsky 写道:
> > On Tue, Sep 27, 2022 at 06:58:50PM +0800, Yanjun Zhu wrote:
> > > 
> > > 在 2022/9/27 18:34, Leon Romanovsky 写道:
> > > > On Sun, Sep 25, 2022 at 10:40:33PM -0400, yanjun.zhu@linux.dev wrote:
> > > > > From: Zhu Yanjun <yanjun.zhu@linux.dev>

<...>

> Is it better to append "exclusive" or "shared" in the end of the line?

No, exclusive/shared is global property, applied to all links.

Thanks

Re: [PATCH] rdma: not display the rdma link in other net namespace

[Yanjun Zhu]

在 2022/10/6 20:53, Leon Romanovsky 写道:
> On Fri, Sep 30, 2022 at 03:25:00PM +0800, Yanjun Zhu wrote:
>> 在 2022/9/28 14:04, Leon Romanovsky 写道:
>>> On Tue, Sep 27, 2022 at 06:58:50PM +0800, Yanjun Zhu wrote:
>>>> 在 2022/9/27 18:34, Leon Romanovsky 写道:
>>>>> On Sun, Sep 25, 2022 at 10:40:33PM -0400, yanjun.zhu@linux.dev wrote:
>>>>>> From: Zhu Yanjun <yanjun.zhu@linux.dev>
> <...>
>
>> Is it better to append "exclusive" or "shared" in the end of the line?
> No, exclusive/shared is global property, applied to all links.

OK.

When running "rdma link show", there is no difference between shared and 
exclusive.

Is it acceptable?


And in exclusive mode, a rdma link that can not be accessed in net 
namespace A still

appears in net namespace A when running "rdma link show" in net namespace A.

The above is different from others in net namespace.

For example, in net namespace, if net device NIC0 is moved to net 
namespace B from net namespace A,

this NIC0 will not appear in net namespace A when running "ip link" 
command in net namespace A.

Is it a problem?


Zhu Yanjun

>
> Thanks

Re: [PATCH] rdma: not display the rdma link in other net namespace

[Leon Romanovsky]

On Thu, Oct 06, 2022 at 10:26:33PM +0800, Yanjun Zhu wrote:
> 
> 在 2022/10/6 20:53, Leon Romanovsky 写道:
> > On Fri, Sep 30, 2022 at 03:25:00PM +0800, Yanjun Zhu wrote:
> > > 在 2022/9/28 14:04, Leon Romanovsky 写道:
> > > > On Tue, Sep 27, 2022 at 06:58:50PM +0800, Yanjun Zhu wrote:
> > > > > 在 2022/9/27 18:34, Leon Romanovsky 写道:
> > > > > > On Sun, Sep 25, 2022 at 10:40:33PM -0400, yanjun.zhu@linux.dev wrote:
> > > > > > > From: Zhu Yanjun <yanjun.zhu@linux.dev>
> > <...>
> > 
> > > Is it better to append "exclusive" or "shared" in the end of the line?
> > No, exclusive/shared is global property, applied to all links.
> 
> OK.
> 
> When running "rdma link show", there is no difference between shared and
> exclusive.
> 
> Is it acceptable?

exclusive/shared is ib_core module parameter, so or all links are shared
or all links are exclusive. They can't be both at the same time.

> 
> 
> And in exclusive mode, a rdma link that can not be accessed in net namespace
> A still
> 
> appears in net namespace A when running "rdma link show" in net namespace A.
> 
> The above is different from others in net namespace.
> 
> For example, in net namespace, if net device NIC0 is moved to net namespace
> B from net namespace A,
> 
> this NIC0 will not appear in net namespace A when running "ip link" command
> in net namespace A.
> 
> Is it a problem?

rdmatool presents IB devices. It has no logic that decides if that
device is usable/operable or not.

Thanks

> 
> 
> Zhu Yanjun
> 
> > 
> > Thanks

Re: [PATCH] rdma: not display the rdma link in other net namespace

[Jason Gunthorpe]

On Thu, Oct 06, 2022 at 07:21:39PM +0300, Leon Romanovsky wrote:

> > And in exclusive mode, a rdma link that can not be accessed in net namespace
> > A still
> > 
> > appears in net namespace A when running "rdma link show" in net namespace A.
> > 
> > The above is different from others in net namespace.
> > 
> > For example, in net namespace, if net device NIC0 is moved to net namespace
> > B from net namespace A,
> > 
> > this NIC0 will not appear in net namespace A when running "ip link" command
> > in net namespace A.
> > 
> > Is it a problem?
> 
> rdmatool presents IB devices. It has no logic that decides if that
> device is usable/operable or not.

It should really not report an IB device that is not in the net
namespace..

I'm surprised this hasn't been noticed because it will break verbs.

Jason

Re: [PATCH] rdma: not display the rdma link in other net namespace

[Leon Romanovsky]

On Thu, Oct 06, 2022 at 01:23:05PM -0300, Jason Gunthorpe wrote:
> On Thu, Oct 06, 2022 at 07:21:39PM +0300, Leon Romanovsky wrote:
> 
> > > And in exclusive mode, a rdma link that can not be accessed in net namespace
> > > A still
> > > 
> > > appears in net namespace A when running "rdma link show" in net namespace A.
> > > 
> > > The above is different from others in net namespace.
> > > 
> > > For example, in net namespace, if net device NIC0 is moved to net namespace
> > > B from net namespace A,
> > > 
> > > this NIC0 will not appear in net namespace A when running "ip link" command
> > > in net namespace A.
> > > 
> > > Is it a problem?
> > 
> > rdmatool presents IB devices. It has no logic that decides if that
> > device is usable/operable or not.
> 
> It should really not report an IB device that is not in the net
> namespace..

It is kernel (nldev.c) job to hide such IB devices and it seems like it
does. For devices with help of ib_enum_all_devs() and for links with
ib_device_get_by_index().

They both checks netns - rdma_dev_access_netns(device, net).

Thanks

Re: [PATCH] rdma: not display the rdma link in other net namespace

[Yanjun Zhu]

在 2022/10/7 14:21, Leon Romanovsky 写道:
> On Thu, Oct 06, 2022 at 01:23:05PM -0300, Jason Gunthorpe wrote:
>> On Thu, Oct 06, 2022 at 07:21:39PM +0300, Leon Romanovsky wrote:
>>
>>>> And in exclusive mode, a rdma link that can not be accessed in net namespace
>>>> A still
>>>>
>>>> appears in net namespace A when running "rdma link show" in net namespace A.
>>>>
>>>> The above is different from others in net namespace.
>>>>
>>>> For example, in net namespace, if net device NIC0 is moved to net namespace
>>>> B from net namespace A,
>>>>
>>>> this NIC0 will not appear in net namespace A when running "ip link" command
>>>> in net namespace A.
>>>>
>>>> Is it a problem?
>>> rdmatool presents IB devices. It has no logic that decides if that
>>> device is usable/operable or not.
>> It should really not report an IB device that is not in the net
>> namespace..
> It is kernel (nldev.c) job to hide such IB devices and it seems like it
> does. For devices with help of ib_enum_all_devs() and for links with
> ib_device_get_by_index().

Thanks, Jason and Leon

I am working on it. I will send the patch out very soon.

Zhu Yanjun

>
> They both checks netns - rdma_dev_access_netns(device, net).
>
> Thanks

[RFC PATCH 1/1] RDMA/core: Fix a problem from rdma link in exclusive mode

[Zhu Yanjun]

From: Zhu Yanjun <yanjun.zhu@linux.dev>

This is not an official commit. In rdma net namespace, the rdma device
is separate from the net device. For example, a rdma device A is in net
namespace A1 while the related net device B is in net namespace B1.

I am curious how to make perftest and rping tests on the above
scenario. The ip address of net device B is in net namespace B1
while the rdma device is in net namespace A1.

From my perspective, the rdma device and related net device should
be in the same net namespace. When a net device is moved from one net
namespace to another net namespace, the rdma device should be in the
same net namespace with the net device.

In this commit, when all the ib devices are parsed in exclusive mode,
if the ib devices and related net devices are not in the same net
namespace, the link information will not be reported to user space.

This commit is a RFC.

Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev>
---
 drivers/infiniband/core/device.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index d275db195f1a..9cdd313e4603 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -2352,6 +2352,16 @@ int ib_enum_all_devs(nldev_callback nldev_cb, struct sk_buff *skb,
 		if (!rdma_dev_access_netns(dev, sock_net(skb->sk)))
 			continue;
 
+		if (!ib_devices_shared_netns) {
+			int port = rdma_start_port(dev);
+			struct net_device *ndev = NULL;
+
+			ndev = ib_device_get_netdev(dev, port);
+
+			if (ndev && !net_eq(dev_net(ndev), sock_net(skb->sk)))
+				continue;
+		}
+
 		ret = nldev_cb(dev, skb, cb, idx);
 		if (ret)
 			break;
-- 
2.27.0

Re: [RFC PATCH 1/1] RDMA/core: Fix a problem from rdma link in exclusive mode

[Leon Romanovsky]

On Mon, Oct 10, 2022 at 08:25:45PM -0400, Zhu Yanjun wrote:
> From: Zhu Yanjun <yanjun.zhu@linux.dev>
> 
> This is not an official commit. In rdma net namespace, the rdma device
> is separate from the net device. For example, a rdma device A is in net
> namespace A1 while the related net device B is in net namespace B1.
> 
> I am curious how to make perftest and rping tests on the above
> scenario. The ip address of net device B is in net namespace B1
> while the rdma device is in net namespace A1.

Use "exclusive" mode, "shared" is legacy interface for backward
compatibility.

> 
> From my perspective, the rdma device and related net device should
> be in the same net namespace. When a net device is moved from one net
> namespace to another net namespace, the rdma device should be in the
> same net namespace with the net device.
> 
> In this commit, when all the ib devices are parsed in exclusive mode,
> if the ib devices and related net devices are not in the same net
> namespace, the link information will not be reported to user space.
> 
> This commit is a RFC.

Please don't send patches as reply-to.

Thanks

Re: [RFC PATCH 1/1] RDMA/core: Fix a problem from rdma link in exclusive mode

[Yanjun Zhu]

在 2022/10/11 18:12, Leon Romanovsky 写道:
> On Mon, Oct 10, 2022 at 08:25:45PM -0400, Zhu Yanjun wrote:
>> From: Zhu Yanjun <yanjun.zhu@linux.dev>
>>
>> This is not an official commit. In rdma net namespace, the rdma device
>> is separate from the net device. For example, a rdma device A is in net
>> namespace A1 while the related net device B is in net namespace B1.
>>
>> I am curious how to make perftest and rping tests on the above
>> scenario. The ip address of net device B is in net namespace B1
>> while the rdma device is in net namespace A1.
> 
> Use "exclusive" mode, "shared" is legacy interface for backward
> compatibility.

Got it. Thanks.

> 
>>
>>  From my perspective, the rdma device and related net device should
>> be in the same net namespace. When a net device is moved from one net
>> namespace to another net namespace, the rdma device should be in the
>> same net namespace with the net device.
>>
>> In this commit, when all the ib devices are parsed in exclusive mode,
>> if the ib devices and related net devices are not in the same net
>> namespace, the link information will not be reported to user space.
>>
>> This commit is a RFC.
> 
> Please don't send patches as reply-to.

OK. I will send another commit to fix this problem very soon.

Zhu Yanjun

> 
> Thanks

Re: [RFC PATCH 1/1] RDMA/core: Fix a problem from rdma link in exclusive mode

[yanjun.zhu]

October 11, 2022 11:08 PM, "Yanjun Zhu" <yanjun.zhu@linux.dev> wrote:

> 在 2022/10/11 18:12, Leon Romanovsky 写道:
> 
>> On Mon, Oct 10, 2022 at 08:25:45PM -0400, Zhu Yanjun wrote:
>>> From: Zhu Yanjun <yanjun.zhu@linux.dev>
>>> 
>>> This is not an official commit. In rdma net namespace, the rdma device
>>> is separate from the net device. For example, a rdma device A is in net
>>> namespace A1 while the related net device B is in net namespace B1.
>>> 
>>> I am curious how to make perftest and rping tests on the above
>>> scenario. The ip address of net device B is in net namespace B1
>>> while the rdma device is in net namespace A1.
>> 
>> Use "exclusive" mode, "shared" is legacy interface for backward
>> compatibility.
> 
> Got it. Thanks.
> 
>>> From my perspective, the rdma device and related net device should
>>> be in the same net namespace. When a net device is moved from one net
>>> namespace to another net namespace, the rdma device should be in the
>>> same net namespace with the net device.
>>> 
>>> In this commit, when all the ib devices are parsed in exclusive mode,
>>> if the ib devices and related net devices are not in the same net
>>> namespace, the link information will not be reported to user space.
>>> 
>>> This commit is a RFC.
>> 
>> Please don't send patches as reply-to.
> 
> OK. I will send another commit to fix this problem very soon.

Hi, Leon

Per discussion, to the non-legacy ib device, when the net devices are moved from one net namespace to another net namespace, the related  ib devices are also moved to the new net namespace.

To the legacy ib device, shared/exclusive mode still work with them.

Based on the above, 2 patches are implemented. I will send them out very soon.

Thanks and Regards,
Zhu Yanjun

> 
> Zhu Yanjun
> 
>> Thanks

Re: [PATCH] rdma: not display the rdma link in other net namespace

[yanjun.zhu]

September 28, 2022 2:04 PM, "Leon Romanovsky" <leon@kernel.org> wrote:

> On Tue, Sep 27, 2022 at 06:58:50PM +0800, Yanjun Zhu wrote:
> 
>> 在 2022/9/27 18:34, Leon Romanovsky 写道:
>> On Sun, Sep 25, 2022 at 10:40:33PM -0400, yanjun.zhu@linux.dev wrote:
>>> From: Zhu Yanjun <yanjun.zhu@linux.dev>
>>> 
>>> When the net devices are moved to another net namespace, the command
>>> "rdma link" should not dispaly the rdma link about this net device.
>>> 
>>> For example, when the net device eno12399 is moved to net namespace net0
>>> from init_net, the rdma link of eno12399 should not display in init_net.
>>> 
>>> Before this change:
>>> 
>>> Init_net:
>>> 
>>> link roceo12399/1 state DOWN physical_state DISABLED <---should not display
>>> link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409
>>> link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0
>>> link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1
>>> 
>>> net0:
>>> 
>>> link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399
>>> link roceo12409/1 state DOWN physical_state DISABLED <---should not display
>>> link rocep202s0f0/1 state DOWN physical_state DISABLED <---should not display
>>> link rocep202s0f1/1 state ACTIVE physical_state LINK_UP <---should not display
>>> 
>>> After this change
>>> 
>>> Init_net:
>>> 
>>> link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409
>>> link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0
>>> link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1
>>> 
>>> net0:
>>> 
>>> link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399
>>> 
>>> Fixes: da990ab40a92 ("rdma: Add link object")
>>> Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev>
>>> ---
>>> rdma/link.c | 3 +++
>>> 1 file changed, 3 insertions(+)
>>> 
>>> diff --git a/rdma/link.c b/rdma/link.c
>>> index bf24b849..449a7636 100644
>>> --- a/rdma/link.c
>>> +++ b/rdma/link.c
>>> @@ -238,6 +238,9 @@ static int link_parse_cb(const struct nlmsghdr *nlh, void *data)
>>> return MNL_CB_ERROR;
>>> }
>>> + if (!tb[RDMA_NLDEV_ATTR_NDEV_NAME] || !tb[RDMA_NLDEV_ATTR_NDEV_INDEX])
>>> + return MNL_CB_OK;
>>> +
>> Regarding your question where it should go in addition to RDMA, the answer
>> is netdev ML. The rdmatool is part of iproute2 and the relevant maintainers
>> should be CCed.
>> Thanks. I will also send it to netdev ML and CC the maintainers.
>> 
>> Regarding the change, I don't think that it is right. User space tool is
>> a simple viewer of data returned from the kernel. It is not a mistake to
>> return device without netdev.
>> 
>> Normally a rdma link based on RoCEv2 should be with a NIC. This NIC device
>> 
>> will send/recv udp packets. With mellanox/intel NIC device, this net device
>> also
>> 
>> do more work than sending/receiving packets.
>> 
>> From this perspective, a rdma link is dependent on a net device.
>> 
>> In this problem, net device is moved to another net namespace. So it can not
>> be
>> 
>> obtained.  And this rdma link can also not work in this net namespace.
>> 
>> So this rdma link should not appear in this net namespace. Or else, it would
>> confuse
>> 
>> the user.
>> 
>> In fact, net namespace is a concept in tcp/ip stack. And it does not exist
>> in rdma stack.
> 
> RDMA has two different net namespace mode: shared and exclusive.
> 
> In shared mode, the IB devices are shared across all net namespaces and
> "moving" net device into different namespace just "hides" it, but don't
> disconnect.

Hi, Leon

About RDMA shared and exclusive mode, I am confusing about this scenario:

In shared mode, ib device A is in net namespace A1 while netdev device B is in net namespace B1.
IB device A is dependent on netdev device B. How to make tests in the above scenario?
Both rping and perftest need a IP address to work. But now ip address is in net namespace B1 while
ib device A is in net namespace A1.

In the product environment, does the above scenario exist?

Thanks and Regards,
Zhu Yanjun

> 
> See comments around various usages of ib_devices_shared_netns variable.
> 
> Thanks

Re: [PATCH] rdma: not display the rdma link in other net namespace

[Leon Romanovsky]

On Sun, Oct 09, 2022 at 10:20:53AM +0000, yanjun.zhu@linux.dev wrote:
> September 28, 2022 2:04 PM, "Leon Romanovsky" <leon@kernel.org> wrote:
> 
> > On Tue, Sep 27, 2022 at 06:58:50PM +0800, Yanjun Zhu wrote:
> > 
> >> 在 2022/9/27 18:34, Leon Romanovsky 写道:
> >> On Sun, Sep 25, 2022 at 10:40:33PM -0400, yanjun.zhu@linux.dev wrote:
> >>> From: Zhu Yanjun <yanjun.zhu@linux.dev>
> >>> 
> >>> When the net devices are moved to another net namespace, the command
> >>> "rdma link" should not dispaly the rdma link about this net device.
> >>> 
> >>> For example, when the net device eno12399 is moved to net namespace net0
> >>> from init_net, the rdma link of eno12399 should not display in init_net.
> >>> 
> >>> Before this change:
> >>> 
> >>> Init_net:
> >>> 
> >>> link roceo12399/1 state DOWN physical_state DISABLED <---should not display
> >>> link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409
> >>> link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0
> >>> link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1
> >>> 
> >>> net0:
> >>> 
> >>> link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399
> >>> link roceo12409/1 state DOWN physical_state DISABLED <---should not display
> >>> link rocep202s0f0/1 state DOWN physical_state DISABLED <---should not display
> >>> link rocep202s0f1/1 state ACTIVE physical_state LINK_UP <---should not display
> >>> 
> >>> After this change
> >>> 
> >>> Init_net:
> >>> 
> >>> link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409
> >>> link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0
> >>> link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1
> >>> 
> >>> net0:
> >>> 
> >>> link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399
> >>> 
> >>> Fixes: da990ab40a92 ("rdma: Add link object")
> >>> Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev>
> >>> ---
> >>> rdma/link.c | 3 +++
> >>> 1 file changed, 3 insertions(+)
> >>> 
> >>> diff --git a/rdma/link.c b/rdma/link.c
> >>> index bf24b849..449a7636 100644
> >>> --- a/rdma/link.c
> >>> +++ b/rdma/link.c
> >>> @@ -238,6 +238,9 @@ static int link_parse_cb(const struct nlmsghdr *nlh, void *data)
> >>> return MNL_CB_ERROR;
> >>> }
> >>> + if (!tb[RDMA_NLDEV_ATTR_NDEV_NAME] || !tb[RDMA_NLDEV_ATTR_NDEV_INDEX])
> >>> + return MNL_CB_OK;
> >>> +
> >> Regarding your question where it should go in addition to RDMA, the answer
> >> is netdev ML. The rdmatool is part of iproute2 and the relevant maintainers
> >> should be CCed.
> >> Thanks. I will also send it to netdev ML and CC the maintainers.
> >> 
> >> Regarding the change, I don't think that it is right. User space tool is
> >> a simple viewer of data returned from the kernel. It is not a mistake to
> >> return device without netdev.
> >> 
> >> Normally a rdma link based on RoCEv2 should be with a NIC. This NIC device
> >> 
> >> will send/recv udp packets. With mellanox/intel NIC device, this net device
> >> also
> >> 
> >> do more work than sending/receiving packets.
> >> 
> >> From this perspective, a rdma link is dependent on a net device.
> >> 
> >> In this problem, net device is moved to another net namespace. So it can not
> >> be
> >> 
> >> obtained.  And this rdma link can also not work in this net namespace.
> >> 
> >> So this rdma link should not appear in this net namespace. Or else, it would
> >> confuse
> >> 
> >> the user.
> >> 
> >> In fact, net namespace is a concept in tcp/ip stack. And it does not exist
> >> in rdma stack.
> > 
> > RDMA has two different net namespace mode: shared and exclusive.
> > 
> > In shared mode, the IB devices are shared across all net namespaces and
> > "moving" net device into different namespace just "hides" it, but don't
> > disconnect.
> 
> Hi, Leon
> 
> About RDMA shared and exclusive mode, I am confusing about this scenario:
> 
> In shared mode, ib device A is in net namespace A1 while netdev device B is in net namespace B1.
> IB device A is dependent on netdev device B. How to make tests in the above scenario?
> Both rping and perftest need a IP address to work. But now ip address is in net namespace B1 while
> ib device A is in net namespace A1.
> 
> In the product environment, does the above scenario exist?

Yes and no at the same time.

Yes:
The whole net namespace support is needed for containers. In old
versions of rdma-core, libibverbs relied on /sys/class/infiniband/
structure. This is why we need "shared" mode, where IB exists without
relation to netdev.

No:
Like you said, it won't work for RoCE and iWARP.

Thanks

> 
> Thanks and Regards,
> Zhu Yanjun
> 
> > 
> > See comments around various usages of ib_devices_shared_netns variable.
> > 
> > Thanks