* [question] when bonding with CX5 network card that support ROCE
@ 2024-05-06 4:46 shaozhengchao
2024-05-06 8:26 ` Zhu Yanjun
0 siblings, 1 reply; 7+ messages in thread
From: shaozhengchao @ 2024-05-06 4:46 UTC (permalink / raw)
To: saeedm, tariqt, borisp, shayd, msanalla, Rahul Rameshbabu,
weizhang, kliteyn, erezsh, igozlan
Cc: netdev, linux-rdma
When using the 5.10 kernel, I can find two IB devices using the
ibv_devinfo command.
----------------------------------
[root@localhost ~]# lspci
91:00.0 Ethernet controller: Mellanox Technologies MT27800 Family
[ConnectX-5]
91:00.1 Ethernet controller: Mellanox Technologies MT27800 Family
----------------------------------
[root@localhost ~]# ibv_devinfo
hca_id: mlx5_0
transport: InfiniBand (0)
fw_ver: 16.31.1014
node_guid: f41d:6b03:006f:4743
sys_image_guid: f41d:6b03:006f:4743
vendor_id: 0x02c9
vendor_part_id: 4119
hw_ver: 0x0
board_id: HUA0000000004
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet
hca_id: mlx5_1
transport: InfiniBand (0)
fw_ver: 16.31.1014
node_guid: f41d:6b03:006f:4744
sys_image_guid: f41d:6b03:006f:4743
vendor_id: 0x02c9
vendor_part_id: 4119
hw_ver: 0x0
board_id: HUA0000000004
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet
----------------------------------
But after the two network ports are bonded, only one IB device is
available, and only PF0 can be used.
[root@localhost shaozhengchao]# ibv_devinfo
hca_id: mlx5_bond_0
transport: InfiniBand (0)
fw_ver: 16.31.1014
node_guid: f41d:6b03:006f:4743
sys_image_guid: f41d:6b03:006f:4743
vendor_id: 0x02c9
vendor_part_id: 4119
hw_ver: 0x0
board_id: HUA0000000004
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet
The current Linux mainline driver is the same.
I found the comment ("If bonded, we do not add an IB device for PF1.")
in the mlx5_lag_intf_add function of the 5.10 branch driver code.
This indicates that wthe the same NIC is used, only PF0 support bonding?
Are there any other constraints, when enable bonding with CX5?
Thank you
Zhengchao Shao
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [question] when bonding with CX5 network card that support ROCE
2024-05-06 4:46 [question] when bonding with CX5 network card that support ROCE shaozhengchao
@ 2024-05-06 8:26 ` Zhu Yanjun
2024-05-06 10:45 ` shaozhengchao
0 siblings, 1 reply; 7+ messages in thread
From: Zhu Yanjun @ 2024-05-06 8:26 UTC (permalink / raw)
To: shaozhengchao, saeedm, tariqt, borisp, shayd, msanalla,
Rahul Rameshbabu, weizhang, kliteyn, erezsh, igozlan
Cc: netdev, linux-rdma
On 06.05.24 06:46, shaozhengchao wrote:
>
> When using the 5.10 kernel, I can find two IB devices using the
> ibv_devinfo command.
> ----------------------------------
> [root@localhost ~]# lspci
> 91:00.0 Ethernet controller: Mellanox Technologies MT27800 Family
> [ConnectX-5]
> 91:00.1 Ethernet controller: Mellanox Technologies MT27800 Family
> ----------------------------------
> [root@localhost ~]# ibv_devinfo
> hca_id: mlx5_0
> transport: InfiniBand (0)
> fw_ver: 16.31.1014
> node_guid: f41d:6b03:006f:4743
> sys_image_guid: f41d:6b03:006f:4743
> vendor_id: 0x02c9
> vendor_part_id: 4119
> hw_ver: 0x0
> board_id: HUA0000000004
> phys_port_cnt: 1
> port: 1
> state: PORT_ACTIVE (4)
> max_mtu: 4096 (5)
> active_mtu: 1024 (3)
> sm_lid: 0
> port_lid: 0
> port_lmc: 0x00
> link_layer: Ethernet
>
> hca_id: mlx5_1
> transport: InfiniBand (0)
> fw_ver: 16.31.1014
> node_guid: f41d:6b03:006f:4744
> sys_image_guid: f41d:6b03:006f:4743
> vendor_id: 0x02c9
> vendor_part_id: 4119
> hw_ver: 0x0
> board_id: HUA0000000004
> phys_port_cnt: 1
> port: 1
> state: PORT_ACTIVE (4)
> max_mtu: 4096 (5)
> active_mtu: 1024 (3)
> sm_lid: 0
> port_lid: 0
> port_lmc: 0x00
> link_layer: Ethernet
> ----------------------------------
> But after the two network ports are bonded, only one IB device is
> available, and only PF0 can be used.
> [root@localhost shaozhengchao]# ibv_devinfo
> hca_id: mlx5_bond_0
> transport: InfiniBand (0)
> fw_ver: 16.31.1014
> node_guid: f41d:6b03:006f:4743
> sys_image_guid: f41d:6b03:006f:4743
> vendor_id: 0x02c9
> vendor_part_id: 4119
> hw_ver: 0x0
> board_id: HUA0000000004
> phys_port_cnt: 1
> port: 1
> state: PORT_ACTIVE (4)
> max_mtu: 4096 (5)
> active_mtu: 1024 (3)
> sm_lid: 0
> port_lid: 0
> port_lmc: 0x00
> link_layer: Ethernet
>
> The current Linux mainline driver is the same.
>
> I found the comment ("If bonded, we do not add an IB device for PF1.")
> in the mlx5_lag_intf_add function of the 5.10 branch driver code.
Not sure if rdma lag is enabled for this or not. /proc/net/bonding will
provide more more details normally.
Zhu Yanjun
> This indicates that wthe the same NIC is used, only PF0 support bonding?
> Are there any other constraints, when enable bonding with CX5?
>
> Thank you
> Zhengchao Shao
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [question] when bonding with CX5 network card that support ROCE
2024-05-06 8:26 ` Zhu Yanjun
@ 2024-05-06 10:45 ` shaozhengchao
2024-05-06 10:58 ` Zhu Yanjun
0 siblings, 1 reply; 7+ messages in thread
From: shaozhengchao @ 2024-05-06 10:45 UTC (permalink / raw)
To: Zhu Yanjun, saeedm, tariqt, borisp, shayd, msanalla,
Rahul Rameshbabu, weizhang, kliteyn, erezsh, igozlan
Cc: netdev, linux-rdma
Hi yanjun:
The following is the command output after the cat /proc/net/bonding
/bond0 command is run:
[root@localhost ~]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v5.10.0+
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0
802.3ad info
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: f4:1d:6b:6f:3b:97
Active Aggregator Info:
Aggregator ID: 2
Number of ports: 1
Actor Key: 23
Partner Key: 1
Partner Mac Address: 00:00:00:00:00:00
Slave Interface: enp145s0f0
MII Status: up
Speed: 40000 Mbps
Duplex: full
Link Failure Count: 1
Permanent HW addr: f4:1d:6b:6f:3b:97
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: churned
Partner Churn State: churned
Actor Churned Count: 1
Partner Churned Count: 2
details actor lacp pdu:
system priority: 65535
system mac address: f4:1d:6b:6f:3b:97
port key: 23
port priority: 255
port number: 1
port state: 69
details partner lacp pdu:
system priority: 65535
system mac address: 00:00:00:00:00:00
oper key: 1
port priority: 255
port number: 1
port state: 1
Slave Interface: enp145s0f1
MII Status: up
Speed: 40000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: f4:1d:6b:6f:3b:98
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: none
Partner Churn State: churned
Actor Churned Count: 0
Partner Churned Count: 1
details actor lacp pdu:
system priority: 65535
system mac address: f4:1d:6b:6f:3b:97
port key: 23
port priority: 255
port number: 2
port state: 77
details partner lacp pdu:
system priority: 65535
system mac address: 00:00:00:00:00:00
oper key: 1
port priority: 255
port number: 1
port state: 1
Thank you
Zhengchao Shao
On 2024/5/6 16:26, Zhu Yanjun wrote:
> On 06.05.24 06:46, shaozhengchao wrote:
>>
>> When using the 5.10 kernel, I can find two IB devices using the
>> ibv_devinfo command.
>> ----------------------------------
>> [root@localhost ~]# lspci
>> 91:00.0 Ethernet controller: Mellanox Technologies MT27800 Family
>> [ConnectX-5]
>> 91:00.1 Ethernet controller: Mellanox Technologies MT27800 Family
>> ----------------------------------
>> [root@localhost ~]# ibv_devinfo
>> hca_id: mlx5_0
>> transport: InfiniBand (0)
>> fw_ver: 16.31.1014
>> node_guid: f41d:6b03:006f:4743
>> sys_image_guid: f41d:6b03:006f:4743
>> vendor_id: 0x02c9
>> vendor_part_id: 4119
>> hw_ver: 0x0
>> board_id: HUA0000000004
>> phys_port_cnt: 1
>> port: 1
>> state: PORT_ACTIVE (4)
>> max_mtu: 4096 (5)
>> active_mtu: 1024 (3)
>> sm_lid: 0
>> port_lid: 0
>> port_lmc: 0x00
>> link_layer: Ethernet
>>
>> hca_id: mlx5_1
>> transport: InfiniBand (0)
>> fw_ver: 16.31.1014
>> node_guid: f41d:6b03:006f:4744
>> sys_image_guid: f41d:6b03:006f:4743
>> vendor_id: 0x02c9
>> vendor_part_id: 4119
>> hw_ver: 0x0
>> board_id: HUA0000000004
>> phys_port_cnt: 1
>> port: 1
>> state: PORT_ACTIVE (4)
>> max_mtu: 4096 (5)
>> active_mtu: 1024 (3)
>> sm_lid: 0
>> port_lid: 0
>> port_lmc: 0x00
>> link_layer: Ethernet
>> ----------------------------------
>> But after the two network ports are bonded, only one IB device is
>> available, and only PF0 can be used.
>> [root@localhost shaozhengchao]# ibv_devinfo
>> hca_id: mlx5_bond_0
>> transport: InfiniBand (0)
>> fw_ver: 16.31.1014
>> node_guid: f41d:6b03:006f:4743
>> sys_image_guid: f41d:6b03:006f:4743
>> vendor_id: 0x02c9
>> vendor_part_id: 4119
>> hw_ver: 0x0
>> board_id: HUA0000000004
>> phys_port_cnt: 1
>> port: 1
>> state: PORT_ACTIVE (4)
>> max_mtu: 4096 (5)
>> active_mtu: 1024 (3)
>> sm_lid: 0
>> port_lid: 0
>> port_lmc: 0x00
>> link_layer: Ethernet
>>
>> The current Linux mainline driver is the same.
>>
>> I found the comment ("If bonded, we do not add an IB device for PF1.")
>> in the mlx5_lag_intf_add function of the 5.10 branch driver code.
>
> Not sure if rdma lag is enabled for this or not. /proc/net/bonding will
> provide more more details normally.
>
> Zhu Yanjun
>
>> This indicates that wthe the same NIC is used, only PF0 support bonding?
>> Are there any other constraints, when enable bonding with CX5?
>>
>> Thank you
>> Zhengchao Shao
>
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [question] when bonding with CX5 network card that support ROCE
2024-05-06 10:45 ` shaozhengchao
@ 2024-05-06 10:58 ` Zhu Yanjun
2024-05-06 11:33 ` shaozhengchao
0 siblings, 1 reply; 7+ messages in thread
From: Zhu Yanjun @ 2024-05-06 10:58 UTC (permalink / raw)
To: shaozhengchao, saeedm, tariqt, borisp, shayd, msanalla,
Rahul Rameshbabu, weizhang, kliteyn, erezsh, igozlan
Cc: netdev, linux-rdma
On 06.05.24 12:45, shaozhengchao wrote:
> Hi yanjun:
> The following is the command output after the cat /proc/net/bonding
> /bond0 command is run:
If I remember it correctly, it seems that it is a rdma LAG and bonding
problem.
Not sure if it is a known problem or not. Please contact your local support.
Zhu Yanjun
> [root@localhost ~]# cat /proc/net/bonding/bond0
> Ethernet Channel Bonding Driver: v5.10.0+
>
> Bonding Mode: IEEE 802.3ad Dynamic link aggregation
> Transmit Hash Policy: layer2 (0)
> MII Status: up
> MII Polling Interval (ms): 100
> Up Delay (ms): 0
> Down Delay (ms): 0
> Peer Notification Delay (ms): 0
>
> 802.3ad info
> LACP rate: slow
> Min links: 0
> Aggregator selection policy (ad_select): stable
> System priority: 65535
> System MAC address: f4:1d:6b:6f:3b:97
> Active Aggregator Info:
> Aggregator ID: 2
> Number of ports: 1
> Actor Key: 23
> Partner Key: 1
> Partner Mac Address: 00:00:00:00:00:00
>
> Slave Interface: enp145s0f0
> MII Status: up
> Speed: 40000 Mbps
> Duplex: full
> Link Failure Count: 1
> Permanent HW addr: f4:1d:6b:6f:3b:97
> Slave queue ID: 0
> Aggregator ID: 1
> Actor Churn State: churned
> Partner Churn State: churned
> Actor Churned Count: 1
> Partner Churned Count: 2
> details actor lacp pdu:
> system priority: 65535
> system mac address: f4:1d:6b:6f:3b:97
> port key: 23
> port priority: 255
> port number: 1
> port state: 69
> details partner lacp pdu:
> system priority: 65535
> system mac address: 00:00:00:00:00:00
> oper key: 1
> port priority: 255
> port number: 1
> port state: 1
>
> Slave Interface: enp145s0f1
> MII Status: up
> Speed: 40000 Mbps
> Duplex: full
> Link Failure Count: 0
> Permanent HW addr: f4:1d:6b:6f:3b:98
> Slave queue ID: 0
> Aggregator ID: 2
> Actor Churn State: none
> Partner Churn State: churned
> Actor Churned Count: 0
> Partner Churned Count: 1
> details actor lacp pdu:
> system priority: 65535
> system mac address: f4:1d:6b:6f:3b:97
> port key: 23
> port priority: 255
> port number: 2
> port state: 77
> details partner lacp pdu:
> system priority: 65535
> system mac address: 00:00:00:00:00:00
> oper key: 1
> port priority: 255
> port number: 1
> port state: 1
>
> Thank you
> Zhengchao Shao
>
>
> On 2024/5/6 16:26, Zhu Yanjun wrote:
>> On 06.05.24 06:46, shaozhengchao wrote:
>>>
>>> When using the 5.10 kernel, I can find two IB devices using the
>>> ibv_devinfo command.
>>> ----------------------------------
>>> [root@localhost ~]# lspci
>>> 91:00.0 Ethernet controller: Mellanox Technologies MT27800 Family
>>> [ConnectX-5]
>>> 91:00.1 Ethernet controller: Mellanox Technologies MT27800 Family
>>> ----------------------------------
>>> [root@localhost ~]# ibv_devinfo
>>> hca_id: mlx5_0
>>> transport: InfiniBand (0)
>>> fw_ver: 16.31.1014
>>> node_guid: f41d:6b03:006f:4743
>>> sys_image_guid: f41d:6b03:006f:4743
>>> vendor_id: 0x02c9
>>> vendor_part_id: 4119
>>> hw_ver: 0x0
>>> board_id: HUA0000000004
>>> phys_port_cnt: 1
>>> port: 1
>>> state: PORT_ACTIVE (4)
>>> max_mtu: 4096 (5)
>>> active_mtu: 1024 (3)
>>> sm_lid: 0
>>> port_lid: 0
>>> port_lmc: 0x00
>>> link_layer: Ethernet
>>>
>>> hca_id: mlx5_1
>>> transport: InfiniBand (0)
>>> fw_ver: 16.31.1014
>>> node_guid: f41d:6b03:006f:4744
>>> sys_image_guid: f41d:6b03:006f:4743
>>> vendor_id: 0x02c9
>>> vendor_part_id: 4119
>>> hw_ver: 0x0
>>> board_id: HUA0000000004
>>> phys_port_cnt: 1
>>> port: 1
>>> state: PORT_ACTIVE (4)
>>> max_mtu: 4096 (5)
>>> active_mtu: 1024 (3)
>>> sm_lid: 0
>>> port_lid: 0
>>> port_lmc: 0x00
>>> link_layer: Ethernet
>>> ----------------------------------
>>> But after the two network ports are bonded, only one IB device is
>>> available, and only PF0 can be used.
>>> [root@localhost shaozhengchao]# ibv_devinfo
>>> hca_id: mlx5_bond_0
>>> transport: InfiniBand (0)
>>> fw_ver: 16.31.1014
>>> node_guid: f41d:6b03:006f:4743
>>> sys_image_guid: f41d:6b03:006f:4743
>>> vendor_id: 0x02c9
>>> vendor_part_id: 4119
>>> hw_ver: 0x0
>>> board_id: HUA0000000004
>>> phys_port_cnt: 1
>>> port: 1
>>> state: PORT_ACTIVE (4)
>>> max_mtu: 4096 (5)
>>> active_mtu: 1024 (3)
>>> sm_lid: 0
>>> port_lid: 0
>>> port_lmc: 0x00
>>> link_layer: Ethernet
>>>
>>> The current Linux mainline driver is the same.
>>>
>>> I found the comment ("If bonded, we do not add an IB device for PF1.")
>>> in the mlx5_lag_intf_add function of the 5.10 branch driver code.
>>
>> Not sure if rdma lag is enabled for this or not. /proc/net/bonding
>> will provide more more details normally.
>>
>> Zhu Yanjun
>>
>>> This indicates that wthe the same NIC is used, only PF0 support
>>> bonding?
>>> Are there any other constraints, when enable bonding with CX5?
>>>
>>> Thank you
>>> Zhengchao Shao
>>
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [question] when bonding with CX5 network card that support ROCE
2024-05-06 10:58 ` Zhu Yanjun
@ 2024-05-06 11:33 ` shaozhengchao
2024-05-06 12:27 ` Zhu Yanjun
0 siblings, 1 reply; 7+ messages in thread
From: shaozhengchao @ 2024-05-06 11:33 UTC (permalink / raw)
To: Zhu Yanjun, saeedm, tariqt, borisp, shayd, msanalla,
Rahul Rameshbabu, weizhang, kliteyn, erezsh, igozlan
Cc: netdev, linux-rdma
Hi Yanjun:
Thank you for your reply. Are there any other restrictions on using
ROCE on the CX5?
Zhengchao Shao
On 2024/5/6 18:58, Zhu Yanjun wrote:
>
> On 06.05.24 12:45, shaozhengchao wrote:
>> Hi yanjun:
>> The following is the command output after the cat /proc/net/bonding
>> /bond0 command is run:
>
> If I remember it correctly, it seems that it is a rdma LAG and bonding
> problem.
>
> Not sure if it is a known problem or not. Please contact your local
> support.
>
> Zhu Yanjun
>
>> [root@localhost ~]# cat /proc/net/bonding/bond0
>> Ethernet Channel Bonding Driver: v5.10.0+
>>
>> Bonding Mode: IEEE 802.3ad Dynamic link aggregation
>> Transmit Hash Policy: layer2 (0)
>> MII Status: up
>> MII Polling Interval (ms): 100
>> Up Delay (ms): 0
>> Down Delay (ms): 0
>> Peer Notification Delay (ms): 0
>>
>> 802.3ad info
>> LACP rate: slow
>> Min links: 0
>> Aggregator selection policy (ad_select): stable
>> System priority: 65535
>> System MAC address: f4:1d:6b:6f:3b:97
>> Active Aggregator Info:
>> Aggregator ID: 2
>> Number of ports: 1
>> Actor Key: 23
>> Partner Key: 1
>> Partner Mac Address: 00:00:00:00:00:00
>>
>> Slave Interface: enp145s0f0
>> MII Status: up
>> Speed: 40000 Mbps
>> Duplex: full
>> Link Failure Count: 1
>> Permanent HW addr: f4:1d:6b:6f:3b:97
>> Slave queue ID: 0
>> Aggregator ID: 1
>> Actor Churn State: churned
>> Partner Churn State: churned
>> Actor Churned Count: 1
>> Partner Churned Count: 2
>> details actor lacp pdu:
>> system priority: 65535
>> system mac address: f4:1d:6b:6f:3b:97
>> port key: 23
>> port priority: 255
>> port number: 1
>> port state: 69
>> details partner lacp pdu:
>> system priority: 65535
>> system mac address: 00:00:00:00:00:00
>> oper key: 1
>> port priority: 255
>> port number: 1
>> port state: 1
>>
>> Slave Interface: enp145s0f1
>> MII Status: up
>> Speed: 40000 Mbps
>> Duplex: full
>> Link Failure Count: 0
>> Permanent HW addr: f4:1d:6b:6f:3b:98
>> Slave queue ID: 0
>> Aggregator ID: 2
>> Actor Churn State: none
>> Partner Churn State: churned
>> Actor Churned Count: 0
>> Partner Churned Count: 1
>> details actor lacp pdu:
>> system priority: 65535
>> system mac address: f4:1d:6b:6f:3b:97
>> port key: 23
>> port priority: 255
>> port number: 2
>> port state: 77
>> details partner lacp pdu:
>> system priority: 65535
>> system mac address: 00:00:00:00:00:00
>> oper key: 1
>> port priority: 255
>> port number: 1
>> port state: 1
>>
>> Thank you
>> Zhengchao Shao
>>
>>
>> On 2024/5/6 16:26, Zhu Yanjun wrote:
>>> On 06.05.24 06:46, shaozhengchao wrote:
>>>>
>>>> When using the 5.10 kernel, I can find two IB devices using the
>>>> ibv_devinfo command.
>>>> ----------------------------------
>>>> [root@localhost ~]# lspci
>>>> 91:00.0 Ethernet controller: Mellanox Technologies MT27800 Family
>>>> [ConnectX-5]
>>>> 91:00.1 Ethernet controller: Mellanox Technologies MT27800 Family
>>>> ----------------------------------
>>>> [root@localhost ~]# ibv_devinfo
>>>> hca_id: mlx5_0
>>>> transport: InfiniBand (0)
>>>> fw_ver: 16.31.1014
>>>> node_guid: f41d:6b03:006f:4743
>>>> sys_image_guid: f41d:6b03:006f:4743
>>>> vendor_id: 0x02c9
>>>> vendor_part_id: 4119
>>>> hw_ver: 0x0
>>>> board_id: HUA0000000004
>>>> phys_port_cnt: 1
>>>> port: 1
>>>> state: PORT_ACTIVE (4)
>>>> max_mtu: 4096 (5)
>>>> active_mtu: 1024 (3)
>>>> sm_lid: 0
>>>> port_lid: 0
>>>> port_lmc: 0x00
>>>> link_layer: Ethernet
>>>>
>>>> hca_id: mlx5_1
>>>> transport: InfiniBand (0)
>>>> fw_ver: 16.31.1014
>>>> node_guid: f41d:6b03:006f:4744
>>>> sys_image_guid: f41d:6b03:006f:4743
>>>> vendor_id: 0x02c9
>>>> vendor_part_id: 4119
>>>> hw_ver: 0x0
>>>> board_id: HUA0000000004
>>>> phys_port_cnt: 1
>>>> port: 1
>>>> state: PORT_ACTIVE (4)
>>>> max_mtu: 4096 (5)
>>>> active_mtu: 1024 (3)
>>>> sm_lid: 0
>>>> port_lid: 0
>>>> port_lmc: 0x00
>>>> link_layer: Ethernet
>>>> ----------------------------------
>>>> But after the two network ports are bonded, only one IB device is
>>>> available, and only PF0 can be used.
>>>> [root@localhost shaozhengchao]# ibv_devinfo
>>>> hca_id: mlx5_bond_0
>>>> transport: InfiniBand (0)
>>>> fw_ver: 16.31.1014
>>>> node_guid: f41d:6b03:006f:4743
>>>> sys_image_guid: f41d:6b03:006f:4743
>>>> vendor_id: 0x02c9
>>>> vendor_part_id: 4119
>>>> hw_ver: 0x0
>>>> board_id: HUA0000000004
>>>> phys_port_cnt: 1
>>>> port: 1
>>>> state: PORT_ACTIVE (4)
>>>> max_mtu: 4096 (5)
>>>> active_mtu: 1024 (3)
>>>> sm_lid: 0
>>>> port_lid: 0
>>>> port_lmc: 0x00
>>>> link_layer: Ethernet
>>>>
>>>> The current Linux mainline driver is the same.
>>>>
>>>> I found the comment ("If bonded, we do not add an IB device for PF1.")
>>>> in the mlx5_lag_intf_add function of the 5.10 branch driver code.
>>>
>>> Not sure if rdma lag is enabled for this or not. /proc/net/bonding
>>> will provide more more details normally.
>>>
>>> Zhu Yanjun
>>>
>>>> This indicates that wthe the same NIC is used, only PF0 support
>>>> bonding?
>>>> Are there any other constraints, when enable bonding with CX5?
>>>>
>>>> Thank you
>>>> Zhengchao Shao
>>>
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [question] when bonding with CX5 network card that support ROCE
2024-05-06 11:33 ` shaozhengchao
@ 2024-05-06 12:27 ` Zhu Yanjun
2024-05-07 1:23 ` shaozhengchao
0 siblings, 1 reply; 7+ messages in thread
From: Zhu Yanjun @ 2024-05-06 12:27 UTC (permalink / raw)
To: shaozhengchao, Zhu Yanjun, saeedm, tariqt, borisp, shayd,
msanalla, Rahul Rameshbabu, weizhang, kliteyn, erezsh, igozlan
Cc: netdev, linux-rdma
On 06.05.24 13:33, shaozhengchao wrote:
>
> Hi Yanjun:
> Thank you for your reply. Are there any other restrictions on using
> ROCE on the CX5?
https://docs.nvidia.com/networking/display/mlnxofedv571020
The above link can answer all your questions ^_^
Enjoy it.
Zhu Yanjun
>
> Zhengchao Shao
>
> On 2024/5/6 18:58, Zhu Yanjun wrote:
>>
>> On 06.05.24 12:45, shaozhengchao wrote:
>>> Hi yanjun:
>>> The following is the command output after the cat /proc/net/bonding
>>> /bond0 command is run:
>>
>> If I remember it correctly, it seems that it is a rdma LAG and bonding
>> problem.
>>
>> Not sure if it is a known problem or not. Please contact your local
>> support.
>>
>> Zhu Yanjun
>>
>>> [root@localhost ~]# cat /proc/net/bonding/bond0
>>> Ethernet Channel Bonding Driver: v5.10.0+
>>>
>>> Bonding Mode: IEEE 802.3ad Dynamic link aggregation
>>> Transmit Hash Policy: layer2 (0)
>>> MII Status: up
>>> MII Polling Interval (ms): 100
>>> Up Delay (ms): 0
>>> Down Delay (ms): 0
>>> Peer Notification Delay (ms): 0
>>>
>>> 802.3ad info
>>> LACP rate: slow
>>> Min links: 0
>>> Aggregator selection policy (ad_select): stable
>>> System priority: 65535
>>> System MAC address: f4:1d:6b:6f:3b:97
>>> Active Aggregator Info:
>>> Aggregator ID: 2
>>> Number of ports: 1
>>> Actor Key: 23
>>> Partner Key: 1
>>> Partner Mac Address: 00:00:00:00:00:00
>>>
>>> Slave Interface: enp145s0f0
>>> MII Status: up
>>> Speed: 40000 Mbps
>>> Duplex: full
>>> Link Failure Count: 1
>>> Permanent HW addr: f4:1d:6b:6f:3b:97
>>> Slave queue ID: 0
>>> Aggregator ID: 1
>>> Actor Churn State: churned
>>> Partner Churn State: churned
>>> Actor Churned Count: 1
>>> Partner Churned Count: 2
>>> details actor lacp pdu:
>>> system priority: 65535
>>> system mac address: f4:1d:6b:6f:3b:97
>>> port key: 23
>>> port priority: 255
>>> port number: 1
>>> port state: 69
>>> details partner lacp pdu:
>>> system priority: 65535
>>> system mac address: 00:00:00:00:00:00
>>> oper key: 1
>>> port priority: 255
>>> port number: 1
>>> port state: 1
>>>
>>> Slave Interface: enp145s0f1
>>> MII Status: up
>>> Speed: 40000 Mbps
>>> Duplex: full
>>> Link Failure Count: 0
>>> Permanent HW addr: f4:1d:6b:6f:3b:98
>>> Slave queue ID: 0
>>> Aggregator ID: 2
>>> Actor Churn State: none
>>> Partner Churn State: churned
>>> Actor Churned Count: 0
>>> Partner Churned Count: 1
>>> details actor lacp pdu:
>>> system priority: 65535
>>> system mac address: f4:1d:6b:6f:3b:97
>>> port key: 23
>>> port priority: 255
>>> port number: 2
>>> port state: 77
>>> details partner lacp pdu:
>>> system priority: 65535
>>> system mac address: 00:00:00:00:00:00
>>> oper key: 1
>>> port priority: 255
>>> port number: 1
>>> port state: 1
>>>
>>> Thank you
>>> Zhengchao Shao
>>>
>>>
>>> On 2024/5/6 16:26, Zhu Yanjun wrote:
>>>> On 06.05.24 06:46, shaozhengchao wrote:
>>>>>
>>>>> When using the 5.10 kernel, I can find two IB devices using the
>>>>> ibv_devinfo command.
>>>>> ----------------------------------
>>>>> [root@localhost ~]# lspci
>>>>> 91:00.0 Ethernet controller: Mellanox Technologies MT27800 Family
>>>>> [ConnectX-5]
>>>>> 91:00.1 Ethernet controller: Mellanox Technologies MT27800 Family
>>>>> ----------------------------------
>>>>> [root@localhost ~]# ibv_devinfo
>>>>> hca_id: mlx5_0
>>>>> transport: InfiniBand (0)
>>>>> fw_ver: 16.31.1014
>>>>> node_guid: f41d:6b03:006f:4743
>>>>> sys_image_guid: f41d:6b03:006f:4743
>>>>> vendor_id: 0x02c9
>>>>> vendor_part_id: 4119
>>>>> hw_ver: 0x0
>>>>> board_id: HUA0000000004
>>>>> phys_port_cnt: 1
>>>>> port: 1
>>>>> state: PORT_ACTIVE (4)
>>>>> max_mtu: 4096 (5)
>>>>> active_mtu: 1024 (3)
>>>>> sm_lid: 0
>>>>> port_lid: 0
>>>>> port_lmc: 0x00
>>>>> link_layer: Ethernet
>>>>>
>>>>> hca_id: mlx5_1
>>>>> transport: InfiniBand (0)
>>>>> fw_ver: 16.31.1014
>>>>> node_guid: f41d:6b03:006f:4744
>>>>> sys_image_guid: f41d:6b03:006f:4743
>>>>> vendor_id: 0x02c9
>>>>> vendor_part_id: 4119
>>>>> hw_ver: 0x0
>>>>> board_id: HUA0000000004
>>>>> phys_port_cnt: 1
>>>>> port: 1
>>>>> state: PORT_ACTIVE (4)
>>>>> max_mtu: 4096 (5)
>>>>> active_mtu: 1024 (3)
>>>>> sm_lid: 0
>>>>> port_lid: 0
>>>>> port_lmc: 0x00
>>>>> link_layer: Ethernet
>>>>> ----------------------------------
>>>>> But after the two network ports are bonded, only one IB device is
>>>>> available, and only PF0 can be used.
>>>>> [root@localhost shaozhengchao]# ibv_devinfo
>>>>> hca_id: mlx5_bond_0
>>>>> transport: InfiniBand (0)
>>>>> fw_ver: 16.31.1014
>>>>> node_guid: f41d:6b03:006f:4743
>>>>> sys_image_guid: f41d:6b03:006f:4743
>>>>> vendor_id: 0x02c9
>>>>> vendor_part_id: 4119
>>>>> hw_ver: 0x0
>>>>> board_id: HUA0000000004
>>>>> phys_port_cnt: 1
>>>>> port: 1
>>>>> state: PORT_ACTIVE (4)
>>>>> max_mtu: 4096 (5)
>>>>> active_mtu: 1024 (3)
>>>>> sm_lid: 0
>>>>> port_lid: 0
>>>>> port_lmc: 0x00
>>>>> link_layer: Ethernet
>>>>>
>>>>> The current Linux mainline driver is the same.
>>>>>
>>>>> I found the comment ("If bonded, we do not add an IB device for PF1.")
>>>>> in the mlx5_lag_intf_add function of the 5.10 branch driver code.
>>>>
>>>> Not sure if rdma lag is enabled for this or not. /proc/net/bonding
>>>> will provide more more details normally.
>>>>
>>>> Zhu Yanjun
>>>>
>>>>> This indicates that wthe the same NIC is used, only PF0 support
>>>>> bonding?
>>>>> Are there any other constraints, when enable bonding with CX5?
>>>>>
>>>>> Thank you
>>>>> Zhengchao Shao
>>>>
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [question] when bonding with CX5 network card that support ROCE
2024-05-06 12:27 ` Zhu Yanjun
@ 2024-05-07 1:23 ` shaozhengchao
0 siblings, 0 replies; 7+ messages in thread
From: shaozhengchao @ 2024-05-07 1:23 UTC (permalink / raw)
To: Zhu Yanjun, saeedm, tariqt, borisp, shayd, msanalla,
Rahul Rameshbabu, weizhang, kliteyn, erezsh, igozlan
Cc: netdev, linux-rdma
On 2024/5/6 20:27, Zhu Yanjun wrote:
> On 06.05.24 13:33, shaozhengchao wrote:
>>
>> Hi Yanjun:
>> Thank you for your reply. Are there any other restrictions on using
>> ROCE on the CX5?
>
> https://docs.nvidia.com/networking/display/mlnxofedv571020
>
> The above link can answer all your questions ^_^
>
> Enjoy it.
>
> Zhu Yanjun
>
Thank you.
Zhengchao Shao
>>
>> Zhengchao Shao
>>
>> On 2024/5/6 18:58, Zhu Yanjun wrote:
>>>
>>> On 06.05.24 12:45, shaozhengchao wrote:
>>>> Hi yanjun:
>>>> The following is the command output after the cat /proc/net/bonding
>>>> /bond0 command is run:
>>>
>>> If I remember it correctly, it seems that it is a rdma LAG and
>>> bonding problem.
>>>
>>> Not sure if it is a known problem or not. Please contact your local
>>> support.
>>>
>>> Zhu Yanjun
>>>
>>>> [root@localhost ~]# cat /proc/net/bonding/bond0
>>>> Ethernet Channel Bonding Driver: v5.10.0+
>>>>
>>>> Bonding Mode: IEEE 802.3ad Dynamic link aggregation
>>>> Transmit Hash Policy: layer2 (0)
>>>> MII Status: up
>>>> MII Polling Interval (ms): 100
>>>> Up Delay (ms): 0
>>>> Down Delay (ms): 0
>>>> Peer Notification Delay (ms): 0
>>>>
>>>> 802.3ad info
>>>> LACP rate: slow
>>>> Min links: 0
>>>> Aggregator selection policy (ad_select): stable
>>>> System priority: 65535
>>>> System MAC address: f4:1d:6b:6f:3b:97
>>>> Active Aggregator Info:
>>>> Aggregator ID: 2
>>>> Number of ports: 1
>>>> Actor Key: 23
>>>> Partner Key: 1
>>>> Partner Mac Address: 00:00:00:00:00:00
>>>>
>>>> Slave Interface: enp145s0f0
>>>> MII Status: up
>>>> Speed: 40000 Mbps
>>>> Duplex: full
>>>> Link Failure Count: 1
>>>> Permanent HW addr: f4:1d:6b:6f:3b:97
>>>> Slave queue ID: 0
>>>> Aggregator ID: 1
>>>> Actor Churn State: churned
>>>> Partner Churn State: churned
>>>> Actor Churned Count: 1
>>>> Partner Churned Count: 2
>>>> details actor lacp pdu:
>>>> system priority: 65535
>>>> system mac address: f4:1d:6b:6f:3b:97
>>>> port key: 23
>>>> port priority: 255
>>>> port number: 1
>>>> port state: 69
>>>> details partner lacp pdu:
>>>> system priority: 65535
>>>> system mac address: 00:00:00:00:00:00
>>>> oper key: 1
>>>> port priority: 255
>>>> port number: 1
>>>> port state: 1
>>>>
>>>> Slave Interface: enp145s0f1
>>>> MII Status: up
>>>> Speed: 40000 Mbps
>>>> Duplex: full
>>>> Link Failure Count: 0
>>>> Permanent HW addr: f4:1d:6b:6f:3b:98
>>>> Slave queue ID: 0
>>>> Aggregator ID: 2
>>>> Actor Churn State: none
>>>> Partner Churn State: churned
>>>> Actor Churned Count: 0
>>>> Partner Churned Count: 1
>>>> details actor lacp pdu:
>>>> system priority: 65535
>>>> system mac address: f4:1d:6b:6f:3b:97
>>>> port key: 23
>>>> port priority: 255
>>>> port number: 2
>>>> port state: 77
>>>> details partner lacp pdu:
>>>> system priority: 65535
>>>> system mac address: 00:00:00:00:00:00
>>>> oper key: 1
>>>> port priority: 255
>>>> port number: 1
>>>> port state: 1
>>>>
>>>> Thank you
>>>> Zhengchao Shao
>>>>
>>>>
>>>> On 2024/5/6 16:26, Zhu Yanjun wrote:
>>>>> On 06.05.24 06:46, shaozhengchao wrote:
>>>>>>
>>>>>> When using the 5.10 kernel, I can find two IB devices using the
>>>>>> ibv_devinfo command.
>>>>>> ----------------------------------
>>>>>> [root@localhost ~]# lspci
>>>>>> 91:00.0 Ethernet controller: Mellanox Technologies MT27800 Family
>>>>>> [ConnectX-5]
>>>>>> 91:00.1 Ethernet controller: Mellanox Technologies MT27800 Family
>>>>>> ----------------------------------
>>>>>> [root@localhost ~]# ibv_devinfo
>>>>>> hca_id: mlx5_0
>>>>>> transport: InfiniBand (0)
>>>>>> fw_ver: 16.31.1014
>>>>>> node_guid: f41d:6b03:006f:4743
>>>>>> sys_image_guid: f41d:6b03:006f:4743
>>>>>> vendor_id: 0x02c9
>>>>>> vendor_part_id: 4119
>>>>>> hw_ver: 0x0
>>>>>> board_id: HUA0000000004
>>>>>> phys_port_cnt: 1
>>>>>> port: 1
>>>>>> state: PORT_ACTIVE (4)
>>>>>> max_mtu: 4096 (5)
>>>>>> active_mtu: 1024 (3)
>>>>>> sm_lid: 0
>>>>>> port_lid: 0
>>>>>> port_lmc: 0x00
>>>>>> link_layer: Ethernet
>>>>>>
>>>>>> hca_id: mlx5_1
>>>>>> transport: InfiniBand (0)
>>>>>> fw_ver: 16.31.1014
>>>>>> node_guid: f41d:6b03:006f:4744
>>>>>> sys_image_guid: f41d:6b03:006f:4743
>>>>>> vendor_id: 0x02c9
>>>>>> vendor_part_id: 4119
>>>>>> hw_ver: 0x0
>>>>>> board_id: HUA0000000004
>>>>>> phys_port_cnt: 1
>>>>>> port: 1
>>>>>> state: PORT_ACTIVE (4)
>>>>>> max_mtu: 4096 (5)
>>>>>> active_mtu: 1024 (3)
>>>>>> sm_lid: 0
>>>>>> port_lid: 0
>>>>>> port_lmc: 0x00
>>>>>> link_layer: Ethernet
>>>>>> ----------------------------------
>>>>>> But after the two network ports are bonded, only one IB device is
>>>>>> available, and only PF0 can be used.
>>>>>> [root@localhost shaozhengchao]# ibv_devinfo
>>>>>> hca_id: mlx5_bond_0
>>>>>> transport: InfiniBand (0)
>>>>>> fw_ver: 16.31.1014
>>>>>> node_guid: f41d:6b03:006f:4743
>>>>>> sys_image_guid: f41d:6b03:006f:4743
>>>>>> vendor_id: 0x02c9
>>>>>> vendor_part_id: 4119
>>>>>> hw_ver: 0x0
>>>>>> board_id: HUA0000000004
>>>>>> phys_port_cnt: 1
>>>>>> port: 1
>>>>>> state: PORT_ACTIVE (4)
>>>>>> max_mtu: 4096 (5)
>>>>>> active_mtu: 1024 (3)
>>>>>> sm_lid: 0
>>>>>> port_lid: 0
>>>>>> port_lmc: 0x00
>>>>>> link_layer: Ethernet
>>>>>>
>>>>>> The current Linux mainline driver is the same.
>>>>>>
>>>>>> I found the comment ("If bonded, we do not add an IB device for
>>>>>> PF1.")
>>>>>> in the mlx5_lag_intf_add function of the 5.10 branch driver code.
>>>>>
>>>>> Not sure if rdma lag is enabled for this or not. /proc/net/bonding
>>>>> will provide more more details normally.
>>>>>
>>>>> Zhu Yanjun
>>>>>
>>>>>> This indicates that wthe the same NIC is used, only PF0 support
>>>>>> bonding?
>>>>>> Are there any other constraints, when enable bonding with CX5?
>>>>>>
>>>>>> Thank you
>>>>>> Zhengchao Shao
>>>>>
>
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2024-05-07 1:23 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-05-06 4:46 [question] when bonding with CX5 network card that support ROCE shaozhengchao
2024-05-06 8:26 ` Zhu Yanjun
2024-05-06 10:45 ` shaozhengchao
2024-05-06 10:58 ` Zhu Yanjun
2024-05-06 11:33 ` shaozhengchao
2024-05-06 12:27 ` Zhu Yanjun
2024-05-07 1:23 ` shaozhengchao
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox