From: Zhu Yanjun <zyjzyj2000@gmail.com>
To: shaozhengchao <shaozhengchao@huawei.com>,
saeedm@nvidia.com, tariqt@nvidia.com, borisp@nvidia.com,
shayd@nvidia.com, msanalla@nvidia.com,
Rahul Rameshbabu <rrameshbabu@nvidia.com>,
weizhang@nvidia.com, kliteyn@nvidia.com, erezsh@nvidia.com,
igozlan@nvidia.com
Cc: netdev <netdev@vger.kernel.org>, linux-rdma@vger.kernel.org
Subject: Re: [question] when bonding with CX5 network card that support ROCE
Date: Mon, 6 May 2024 10:26:01 +0200 [thread overview]
Message-ID: <7b0e61e1-8d50-4431-bd0a-6398c618a609@linux.dev> (raw)
In-Reply-To: <756aaf3c-5a15-8d18-89d4-ea7380cf845d@huawei.com>
On 06.05.24 06:46, shaozhengchao wrote:
>
> When using the 5.10 kernel, I can find two IB devices using the
> ibv_devinfo command.
> ----------------------------------
> [root@localhost ~]# lspci
> 91:00.0 Ethernet controller: Mellanox Technologies MT27800 Family
> [ConnectX-5]
> 91:00.1 Ethernet controller: Mellanox Technologies MT27800 Family
> ----------------------------------
> [root@localhost ~]# ibv_devinfo
> hca_id: mlx5_0
> transport: InfiniBand (0)
> fw_ver: 16.31.1014
> node_guid: f41d:6b03:006f:4743
> sys_image_guid: f41d:6b03:006f:4743
> vendor_id: 0x02c9
> vendor_part_id: 4119
> hw_ver: 0x0
> board_id: HUA0000000004
> phys_port_cnt: 1
> port: 1
> state: PORT_ACTIVE (4)
> max_mtu: 4096 (5)
> active_mtu: 1024 (3)
> sm_lid: 0
> port_lid: 0
> port_lmc: 0x00
> link_layer: Ethernet
>
> hca_id: mlx5_1
> transport: InfiniBand (0)
> fw_ver: 16.31.1014
> node_guid: f41d:6b03:006f:4744
> sys_image_guid: f41d:6b03:006f:4743
> vendor_id: 0x02c9
> vendor_part_id: 4119
> hw_ver: 0x0
> board_id: HUA0000000004
> phys_port_cnt: 1
> port: 1
> state: PORT_ACTIVE (4)
> max_mtu: 4096 (5)
> active_mtu: 1024 (3)
> sm_lid: 0
> port_lid: 0
> port_lmc: 0x00
> link_layer: Ethernet
> ----------------------------------
> But after the two network ports are bonded, only one IB device is
> available, and only PF0 can be used.
> [root@localhost shaozhengchao]# ibv_devinfo
> hca_id: mlx5_bond_0
> transport: InfiniBand (0)
> fw_ver: 16.31.1014
> node_guid: f41d:6b03:006f:4743
> sys_image_guid: f41d:6b03:006f:4743
> vendor_id: 0x02c9
> vendor_part_id: 4119
> hw_ver: 0x0
> board_id: HUA0000000004
> phys_port_cnt: 1
> port: 1
> state: PORT_ACTIVE (4)
> max_mtu: 4096 (5)
> active_mtu: 1024 (3)
> sm_lid: 0
> port_lid: 0
> port_lmc: 0x00
> link_layer: Ethernet
>
> The current Linux mainline driver is the same.
>
> I found the comment ("If bonded, we do not add an IB device for PF1.")
> in the mlx5_lag_intf_add function of the 5.10 branch driver code.
Not sure if rdma lag is enabled for this or not. /proc/net/bonding will
provide more more details normally.
Zhu Yanjun
> This indicates that wthe the same NIC is used, only PF0 support bonding?
> Are there any other constraints, when enable bonding with CX5?
>
> Thank you
> Zhengchao Shao
next prev parent reply other threads:[~2024-05-06 8:26 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-05-06 4:46 [question] when bonding with CX5 network card that support ROCE shaozhengchao
2024-05-06 8:26 ` Zhu Yanjun [this message]
2024-05-06 10:45 ` shaozhengchao
2024-05-06 10:58 ` Zhu Yanjun
2024-05-06 11:33 ` shaozhengchao
2024-05-06 12:27 ` Zhu Yanjun
2024-05-07 1:23 ` shaozhengchao
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=7b0e61e1-8d50-4431-bd0a-6398c618a609@linux.dev \
--to=zyjzyj2000@gmail.com \
--cc=borisp@nvidia.com \
--cc=erezsh@nvidia.com \
--cc=igozlan@nvidia.com \
--cc=kliteyn@nvidia.com \
--cc=linux-rdma@vger.kernel.org \
--cc=msanalla@nvidia.com \
--cc=netdev@vger.kernel.org \
--cc=rrameshbabu@nvidia.com \
--cc=saeedm@nvidia.com \
--cc=shaozhengchao@huawei.com \
--cc=shayd@nvidia.com \
--cc=tariqt@nvidia.com \
--cc=weizhang@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox