Re: [PATCH net-next 00/17] Introduce line card support for modular switch

[Ido Schimmel <idosch@idosch.org>]

On Tue, Apr 19, 2022 at 01:55:44PM +0200, Jiri Pirko wrote:
> Mon, Apr 18, 2022 at 04:31:30PM CEST, dsahern@gmail.com wrote:
> >On 4/18/22 12:42 AM, Ido Schimmel wrote:
> >> Jiri says:
> >> 
> >> This patchset introduces support for modular switch systems and also
> >> introduces mlxsw support for NVIDIA Mellanox SN4800 modular switch.
> >> It contains 8 slots to accommodate line cards - replaceable PHY modules
> >> which may contain gearboxes.
> >> Currently supported line card:
> >> 16X 100GbE (QSFP28)
> >> Other line cards that are going to be supported:
> >> 8X 200GbE (QSFP56)
> >> 4X 400GbE (QSFP-DD)
> >> There may be other types of line cards added in the future.
> >> 
> >> To be consistent with the port split configuration (splitter cabels),
> >> the line card entities are treated in the similar way. The nature of
> >> a line card is not "a pluggable device", but "a pluggable PHY module".
> >> 
> >> A concept of "provisioning" is introduced. The user may "provision"
> >> certain slot with a line card type. Driver then creates all instances
> >> (devlink ports, netdevices, etc) related to this line card type. It does
> >> not matter if the line card is plugged-in at the time. User is able to
> >> configure netdevices, devlink ports, setup port splitters, etc. From the
> >> perspective of the switch ASIC, all is present and can be configured.
> >> 
> >> The carrier of netdevices stays down if the line card is not plugged-in.
> >> Once the line card is inserted and activated, the carrier of
> >> the related netdevices is then reflecting the physical line state,
> >> same as for an ordinary fixed port.
> >> 
> >> Once user does not want to use the line card related instances
> >> anymore, he can "unprovision" the slot. Driver then removes the
> >> instances.
> >> 
> >> Patches 1-4 are extending devlink driver API and UAPI in order to
> >> register, show, dump, provision and activate the line card.
> >> Patches 5-17 are implementing the introduced API in mlxsw.
> >> The last patch adds a selftest for mlxsw line cards.
> >> 
> >> Example:
> >> $ devlink port # No ports are listed
> >> $ devlink lc
> >> pci/0000:01:00.0:
> >>   lc 1 state unprovisioned
> >>     supported_types:
> >>        16x100G
> >>   lc 2 state unprovisioned
> >>     supported_types:
> >>        16x100G
> >>   lc 3 state unprovisioned
> >>     supported_types:
> >>        16x100G
> >>   lc 4 state unprovisioned
> >>     supported_types:
> >>        16x100G
> >>   lc 5 state unprovisioned
> >>     supported_types:
> >>        16x100G
> >>   lc 6 state unprovisioned
> >>     supported_types:
> >>        16x100G
> >>   lc 7 state unprovisioned
> >>     supported_types:
> >>        16x100G
> >>   lc 8 state unprovisioned
> >>     supported_types:
> >>        16x100G
> >> 
> >> Note that driver exposes list supported line card types. Currently
> >> there is only one: "16x100G".
> >> 
> >> To provision the slot #8:
> >> 
> >> $ devlink lc set pci/0000:01:00.0 lc 8 type 16x100G
> >> $ devlink lc show pci/0000:01:00.0 lc 8
> >> pci/0000:01:00.0:
> >>   lc 8 state active type 16x100G
> >>     supported_types:
> >>        16x100G
> >> $ devlink port
> >> pci/0000:01:00.0/0: type notset flavour cpu port 0 splittable false
> >> pci/0000:01:00.0/53: type eth netdev enp1s0nl8p1 flavour physical lc 8 port 1 splittable true lanes 4
> >> pci/0000:01:00.0/54: type eth netdev enp1s0nl8p2 flavour physical lc 8 port 2 splittable true lanes 4
> >> pci/0000:01:00.0/55: type eth netdev enp1s0nl8p3 flavour physical lc 8 port 3 splittable true lanes 4
> >> pci/0000:01:00.0/56: type eth netdev enp1s0nl8p4 flavour physical lc 8 port 4 splittable true lanes 4
> >> pci/0000:01:00.0/57: type eth netdev enp1s0nl8p5 flavour physical lc 8 port 5 splittable true lanes 4
> >> pci/0000:01:00.0/58: type eth netdev enp1s0nl8p6 flavour physical lc 8 port 6 splittable true lanes 4
> >> pci/0000:01:00.0/59: type eth netdev enp1s0nl8p7 flavour physical lc 8 port 7 splittable true lanes 4
> >> pci/0000:01:00.0/60: type eth netdev enp1s0nl8p8 flavour physical lc 8 port 8 splittable true lanes 4
> >> pci/0000:01:00.0/61: type eth netdev enp1s0nl8p9 flavour physical lc 8 port 9 splittable true lanes 4
> >> pci/0000:01:00.0/62: type eth netdev enp1s0nl8p10 flavour physical lc 8 port 10 splittable true lanes 4
> >> pci/0000:01:00.0/63: type eth netdev enp1s0nl8p11 flavour physical lc 8 port 11 splittable true lanes 4
> >> pci/0000:01:00.0/64: type eth netdev enp1s0nl8p12 flavour physical lc 8 port 12 splittable true lanes 4
> >> pci/0000:01:00.0/125: type eth netdev enp1s0nl8p13 flavour physical lc 8 port 13 splittable true lanes 4
> >> pci/0000:01:00.0/126: type eth netdev enp1s0nl8p14 flavour physical lc 8 port 14 splittable true lanes 4
> >> pci/0000:01:00.0/127: type eth netdev enp1s0nl8p15 flavour physical lc 8 port 15 splittable true lanes 4
> >> pci/0000:01:00.0/128: type eth netdev enp1s0nl8p16 flavour physical lc 8 port 16 splittable true lanes 4
> >> 
> >> To uprovision the slot #8:
> >> 
> >> $ devlink lc set pci/0000:01:00.0 lc 8 notype
> >> 
> >
> >are there any changes from the last RFC?
> >
> >https://lore.kernel.org/netdev/20210122094648.1631078-1-jiri@resnulli.us/
> 
> Yes, many of them, I din't track them. Mainly, the RFC was backed by
> netdevsim implementation, this is mlxsw with actual HW underneath.

I don't think David was asking about the backing implementation, but
about the interface itself. I don't remember any changes in this aspect
during the internal review and I also compared the uAPI files and they
seem to be the same. The only change that I do remember is making the
APIs idempotent. Previously, this would fail:

 # devlink lc set pci/0000:01:00.0 lc 8 notype
 # devlink lc set pci/0000:01:00.0 lc 8 notype