[PATCH rdma-next 00/12] Multi-plane support for mlx5

[PATCH rdma-next 00/12] Multi-plane support for mlx5

[Leon Romanovsky]

From: Leon Romanovsky <leonro@nvidia.com>

From Mark,

This patchset adds support to IB sub device and mlx5 implementation.

An IB sub device provides a subset of functionalists of it's parent.
Currently type "SMI" is supported: A SMI device provides SMI (QP0)
interface and shares same VPort with it's parent; It allows the subnet
manager to configure VPort through this interface when the parent
doesn't support SMI.

In mlx5 cases, when mlx5 multi-plane is supported, a logical mlx5 port
which aggregates multiple physical plane ports is presented, to provide
higher bandwidth. As SMI is per physical port, a mlx5 SMI device is
needed to represents physical plane ports and provides SMI capability.

A sub device can be added or deleted with the rdma tool. When a mlx5 SMI
device is created, all it's ports are created.

Examples:
$ rdma dev add smi1 type SMI parent ibp8s0f1
$ rdma dev show smi1
2: smi1: node_type ca fw 20.38.0458 node_guid 9803:9b03:009f:d20f
sys_image_guid 9803:9b03:009f:d20e type smi parent ibp8s0f1
$ rdma link show
...
link smi1/1 state INIT physical_state LINK_UP
link smi1/2 state INIT physical_state LINK_UP
link smi1/3 state INIT physical_state LINK_UP
link smi1/4 state INIT physical_state LINK_UP
$ rdma dev del smi1

Mark Zhang (12):
  RDMA/core: Create "issm*" device nodes only when SMI is supported
  net/mlx5: mlx5_ifc update for multi-plane support
  RDMA/mlx5: Add support to multi-plane device and port
  RDMA/core: Support IB sub device with type "SMI"
  RDMA: Set type of rdma_ah to IB for a SMI sub device
  RDMA/core: Create GSI QP only when CM is supported
  RDMA/mlx5: Support plane device and driver APIs to add and delete it
  RDMA/nldev: Add support to add/delete a sub IB device through netlink
  RDMA/nldev: Add support to dump device type and parent device if
    exists
  RDMA/mlx5: Add plane index support when querying PTYS registers
  net/mlx5: mlx5_ifc update for accessing ppcnt register of plane ports
  RDMA/mlx5: Support per-plane port IB counters by querying PPCNT
    register

 drivers/infiniband/core/agent.c               |  32 ++-
 drivers/infiniband/core/device.c              |  68 +++++++
 drivers/infiniband/core/mad.c                 |   9 +-
 drivers/infiniband/core/nldev.c               |  69 +++++++
 drivers/infiniband/core/user_mad.c            |  29 +--
 drivers/infiniband/core/uverbs_main.c         |   3 +-
 drivers/infiniband/hw/mlx5/cmd.c              |  12 +-
 drivers/infiniband/hw/mlx5/cmd.h              |   2 +-
 drivers/infiniband/hw/mlx5/mad.c              |  71 +++++--
 drivers/infiniband/hw/mlx5/main.c             | 182 ++++++++++++++++--
 drivers/infiniband/hw/mlx5/mlx5_ib.h          |  10 +
 drivers/infiniband/hw/mlx5/qp.c               |   7 +-
 drivers/infiniband/hw/mlx5/qpc.c              |  13 +-
 .../net/ethernet/mellanox/mlx5/core/en/port.c |   2 +-
 .../ethernet/mellanox/mlx5/core/en_ethtool.c  |   2 +-
 .../mellanox/mlx5/core/ipoib/ethtool.c        |   2 +-
 .../net/ethernet/mellanox/mlx5/core/port.c    |  10 +-
 .../net/ethernet/mellanox/mlx5/core/vport.c   |   1 +
 include/linux/mlx5/device.h                   |   1 +
 include/linux/mlx5/driver.h                   |   1 +
 include/linux/mlx5/mlx5_ifc.h                 |  61 +++++-
 include/linux/mlx5/port.h                     |   5 +-
 include/rdma/ib_verbs.h                       |  45 +++++
 include/uapi/rdma/rdma_netlink.h              |  13 ++
 24 files changed, 574 insertions(+), 76 deletions(-)

-- 
2.45.2

Re: [PATCH rdma-next 00/12] Multi-plane support for mlx5

[Jason Gunthorpe]

On Sun, Jun 16, 2024 at 07:08:32PM +0300, Leon Romanovsky wrote:
> Mark Zhang (12):
>   RDMA/core: Create "issm*" device nodes only when SMI is supported
>   net/mlx5: mlx5_ifc update for multi-plane support
>   RDMA/mlx5: Add support to multi-plane device and port
>   RDMA/core: Support IB sub device with type "SMI"
>   RDMA: Set type of rdma_ah to IB for a SMI sub device
>   RDMA/core: Create GSI QP only when CM is supported
>   RDMA/mlx5: Support plane device and driver APIs to add and delete it
>   RDMA/nldev: Add support to add/delete a sub IB device through netlink
>   RDMA/nldev: Add support to dump device type and parent device if
>     exists
>   RDMA/mlx5: Add plane index support when querying PTYS registers
>   net/mlx5: mlx5_ifc update for accessing ppcnt register of plane ports
>   RDMA/mlx5: Support per-plane port IB counters by querying PPCNT
>     register

This all seems quite straightforward, Leon are you going to put this
on a shared branch with all the IFC stuff/etc?

Thanks,
Jason

Re: [PATCH rdma-next 00/12] Multi-plane support for mlx5

[Leon Romanovsky]

On Sun, 16 Jun 2024 19:08:32 +0300, Leon Romanovsky wrote:
> From: Leon Romanovsky <leonro@nvidia.com>
> 
> From Mark,
> 
> This patchset adds support to IB sub device and mlx5 implementation.
> 
> An IB sub device provides a subset of functionalists of it's parent.
> Currently type "SMI" is supported: A SMI device provides SMI (QP0)
> interface and shares same VPort with it's parent; It allows the subnet
> manager to configure VPort through this interface when the parent
> doesn't support SMI.
> 
> [...]

Applied, thanks!

[01/12] RDMA/core: Create "issm*" device nodes only when SMI is supported
        https://git.kernel.org/rdma/rdma/c/50660c5197f52b
[02/12] net/mlx5: mlx5_ifc update for multi-plane support
        https://git.kernel.org/rdma/rdma/c/65528cfb21fdb6
[03/12] RDMA/mlx5: Add support to multi-plane device and port
        https://git.kernel.org/rdma/rdma/c/2a5db20fa53219
[04/12] RDMA/core: Support IB sub device with type "SMI"
        https://git.kernel.org/rdma/rdma/c/f3b5c2b823fbd8
[05/12] RDMA: Set type of rdma_ah to IB for a SMI sub device
        https://git.kernel.org/rdma/rdma/c/66862e38a557b3
[06/12] RDMA/core: Create GSI QP only when CM is supported
        https://git.kernel.org/rdma/rdma/c/6d4498d1745128
[07/12] RDMA/mlx5: Support plane device and driver APIs to add and delete it
        https://git.kernel.org/rdma/rdma/c/39351acd72e775
[08/12] RDMA/nldev: Add support to add/delete a sub IB device through netlink
        https://git.kernel.org/rdma/rdma/c/201dfa2d8129a6
[09/12] RDMA/nldev: Add support to dump device type and parent device if exists
        https://git.kernel.org/rdma/rdma/c/1bc00c7c0ae33e
[10/12] RDMA/mlx5: Add plane index support when querying PTYS registers
        https://git.kernel.org/rdma/rdma/c/d6caf3986716c3
[11/12] net/mlx5: mlx5_ifc update for accessing ppcnt register of plane ports
        https://git.kernel.org/rdma/rdma/c/db9e43f6580613
[12/12] RDMA/mlx5: Support per-plane port IB counters by querying PPCNT register
        https://git.kernel.org/rdma/rdma/c/ac3a5e5f01eb40

Best regards,
-- 
Leon Romanovsky <leonro@nvidia.com>