Re: [PATCH net-next v2] net/smc: use the correct ndev to find pnetid by pnetid table

[Guangguan Wang <guangguan.wang@linux.alibaba.com>]

On 2025/3/13 15:46, Wenjia Zhang wrote:
> 
> 
> On 04.03.25 13:43, Guangguan Wang wrote:
>> When using smc_pnet in SMC, it will only search the pnetid in the
>> base_ndev of the netdev hierarchy(both HW PNETID and User-defined
>> sw pnetid). This may not work for some scenarios when using SMC in
>> container on cloud environment.
>> In container, there have choices of different container network,
>> such as directly using host network, virtual network IPVLAN, veth,
>> etc. Different choices of container network have different netdev
>> hierarchy. Examples of netdev hierarchy show below. (eth0 and eth1
>> in host below is the netdev directly related to the physical device).
>>              _______________________________
>>             |   _________________           |
>>             |  |POD              |          |
>>             |  |                 |          |
>>             |  | eth0_________   |          |
>>             |  |____|         |__|          |
>>             |       |         |             |
>>             |       |         |             |
>>             |   eth1|base_ndev| eth0_______ |
>>             |       |         |    | RDMA  ||
>>             | host  |_________|    |_______||
>>             ---------------------------------
>>       netdev hierarchy if directly using host network
>>             ________________________________
>>             |   _________________           |
>>             |  |POD  __________  |          |
>>             |  |    |upper_ndev| |          |
>>             |  |eth0|__________| |          |
>>             |  |_______|_________|          |
>>             |          |lower netdev        |
>>             |        __|______              |
>>             |   eth1|         | eth0_______ |
>>             |       |base_ndev|    | RDMA  ||
>>             | host  |_________|    |_______||
>>             ---------------------------------
>>              netdev hierarchy if using IPVLAN
>>              _______________________________
>>             |   _____________________       |
>>             |  |POD        _________ |      |
>>             |  |          |base_ndev||      |
>>             |  |eth0(veth)|_________||      |
>>             |  |____________|________|      |
>>             |               |pairs          |
>>             |        _______|_              |
>>             |       |         | eth0_______ |
>>             |   veth|base_ndev|    | RDMA  ||
>>             |       |_________|    |_______||
>>             |        _________              |
>>             |   eth1|base_ndev|             |
>>             | host  |_________|             |
>>             ---------------------------------
>>               netdev hierarchy if using veth
>> Due to some reasons, the eth1 in host is not RDMA attached netdevice,
>> pnetid is needed to map the eth1(in host) with RDMA device so that POD
>> can do SMC-R. Because the eth1(in host) is managed by CNI plugin(such
>> as Terway, network management plugin in container environment), and in
>> cloud environment the eth(in host) can dynamically be inserted by CNI
>> when POD create and dynamically be removed by CNI when POD destroy and
>> no POD related to the eth(in host) anymore. It is hard to config the
>> pnetid to the eth1(in host). But it is easy to config the pnetid to the
>> netdevice which can be seen in POD. When do SMC-R, both the container
>> directly using host network and the container using veth network can
>> successfully match the RDMA device, because the configured pnetid netdev
>> is a base_ndev. But the container using IPVLAN can not successfully
>> match the RDMA device and 0x03030000 fallback happens, because the
>> configured pnetid netdev is not a base_ndev. Additionally, if config
>> pnetid to the eth1(in host) also can not work for matching RDMA device
>> when using veth network and doing SMC-R in POD.
>>
>> To resolve the problems list above, this patch extends to search user
>> -defined sw pnetid in the clc handshake ndev when no pnetid can be found
>> in the base_ndev, and the base_ndev take precedence over ndev for backward
>> compatibility. This patch also can unify the pnetid setup of different
>> network choices list above in container(Config user-defined sw pnetid in
>> the netdevice can be seen in POD).
>>
>> Signed-off-by: Guangguan Wang <guangguan.wang@linux.alibaba.com>
>> ---
>>   net/smc/smc_pnet.c | 8 +++++---
>>   1 file changed, 5 insertions(+), 3 deletions(-)
>>
>> diff --git a/net/smc/smc_pnet.c b/net/smc/smc_pnet.c
>> index 716808f374a8..b391c2ef463f 100644
>> --- a/net/smc/smc_pnet.c
>> +++ b/net/smc/smc_pnet.c
>> @@ -1079,14 +1079,16 @@ static void smc_pnet_find_roce_by_pnetid(struct net_device *ndev,
>>                        struct smc_init_info *ini)
>>   {
>>       u8 ndev_pnetid[SMC_MAX_PNETID_LEN];
>> +    struct net_device *base_ndev;
>>       struct net *net;
>>   -    ndev = pnet_find_base_ndev(ndev);
>> +    base_ndev = pnet_find_base_ndev(ndev);
>>       net = dev_net(ndev);
>> -    if (smc_pnetid_by_dev_port(ndev->dev.parent, ndev->dev_port,
>> +    if (smc_pnetid_by_dev_port(base_ndev->dev.parent, base_ndev->dev_port,
>>                      ndev_pnetid) &&
>> +        smc_pnet_find_ndev_pnetid_by_table(base_ndev, ndev_pnetid) &&
>>           smc_pnet_find_ndev_pnetid_by_table(ndev, ndev_pnetid)) {
>> -        smc_pnet_find_rdma_dev(ndev, ini);
>> +        smc_pnet_find_rdma_dev(base_ndev, ini);
>>           return; /* pnetid could not be determined */
>>       }
>>       _smc_pnet_find_roce_by_pnetid(ndev_pnetid, ini, NULL, net);
> 
> Hi Guangguan,
> 
> sorry for the late answer! It looks good to me. Here is my R-b:
> 
> Reviewed-by: Wenjia Zhang <wenjia@linux.ibm.com>
> 
Thanks, Wenjia.

> Btw. could you give Halil some time for the review? He also wants to have a look.
It is OK.

Regards,
Guangguan Wang
> 
> Thanks,
> Wenjia
>