Netdev List
 help / color / mirror / Atom feed
* [PATCH net-next V2 0/5] net/mlx5: LAG over VFs/SFs (virtual LAG)
@ 2026-11-20  7:59 Tariq Toukan
  2026-11-20  7:59 ` [PATCH net-next V2 1/5] net/mlx5: E-switch, add match-metadata allocator for nested VF/SF eswitches Tariq Toukan
                   ` (4 more replies)
  0 siblings, 5 replies; 6+ messages in thread
From: Tariq Toukan @ 2026-11-20  7:59 UTC (permalink / raw)
  To: Andrew Lunn, David S. Miller, Eric Dumazet, Jakub Kicinski,
	netdev, Paolo Abeni
  Cc: Akiva Goldberger, Gal Pressman, Jason Gunthorpe, Leon Romanovsky,
	linux-rdma, Mark Bloch, Moshe Shemesh, Or Har-Toov,
	Saeed Mahameed, Shay Drory, Tariq Toukan

This is V2 of the top patches submitted previously here [1].

This series by Shay enables a LAG over functions that do not represent a
physical port - a "virtual LAG" (virt LAG) of VFs or SFs acting as
nested e-switch managers. New FW lets a VF/SF front the physical uplinks
through a multi-uplink e-switch; such a function reports num_lag_ports <
2 and drives port selection via the e-switch (port_select_eswitch)
rather than the HW port-selection flow table. A virt LAG is therefore a
software-managed, shared-FDB MPESW LAG: the driver builds the LAG demux
and vport-LAG in software.

It depends on [2] which prepared the LAG core.

The patches fall into two parts.

E-switch and representors (patches 1, 2, 3, 4)
  - A dedicated match-metadata allocator for nested VF/SF e-switches, so
    VFs/SFs sharing an FDB get unique source-port metadata.
  - Match the master egress bounce rule by vport metadata.
  - Drop the master egress rule while a software vport LAG is installed:
    the SW emulation is a rule in the slave's egress ACL and cannot act
    on a packet the master egress rule redirects after the FDB. Restore
    it when the vport LAG is removed.
  - Register a VF/SF e-switch-manager self-port with a switch-id so it
    groups with the representors it manages.

Virtual LAG enablement (patch 5)
  - Enable LAG over functions that do not represent a physical port -
    the gate that admits a virt LAG.

Regards,
Tariq

[1] https://lore.kernel.org/all/20261103171009.563034-1-tariqt@nvidia.com/
[2] https://lore.kernel.org/all/20261116103010.1129666-1-tariqt@nvidia.com/

V2:
- Split the patch set to ease the review: this is v1 patches 11-15; v1
  patches 1-10 are sent as "net/mlx5: LAG, preparations for LAG over
  VFs/SFs", which this series depends on.
- Patch 1: commit message describes which non-PF eswitches take the
  nested metadata layout, and no longer implies a PF may have the virt
  LAG capabilities. The reg_c_0 comment in include/linux/mlx5/eswitch.h
  describes the nested layout (Sashiko).
- Patch 3: commit message names the field ldev->max_fns, not
  ldev->max_funcs (Sashiko).
- Patch 5: do not assign an RDMA TX port affinity on a virt LAG capable
  function, and do not register the multipath FIB notifier for a virt
  LAG (Sashiko).

V1:
https://lore.kernel.org/all/20261103171009.563034-1-tariqt@nvidia.com/

Answers to sashiko comments on v1:

net/mlx5: E-switch, add match-metadata allocator for nested VF/SF eswitches

  > +	if (!mlx5_core_is_pf(esw->dev) && !mlx5_virt_lag_nonmember(esw->dev))
  > +		return mlx5_nested_esw_match_metadata_alloc(esw);

  Q: mlx5_virt_lag_nonmember() is false when virt LAG is unsupported, so a
  standalone non-PF eswitch without virt LAG support takes the nested path
  and fails switchdev with -ENOSPC. Could it keep the PFNUM allocator?

  [SD] A non-PF eswitch manager without virt LAG support doesn't exist at
  the moment. FW doesn't expose such a VF/SF.

  Q: The nested allocator gives each eswitch 254 ids (2-255), yet a
  nested eswitch can enumerate up to max_num_sf SF vports, each taking an
  id. Also, LAG admission accepts slots 0-254, while the allocator gives
  no metadata to slots above 239. Can switchdev fail with -ENOSPC in
  either case?

  [SD] Both are theoretical only; there is no such use case.

net/mlx5: E-switch, match master egress rule by vport metadata

  Q: Is mlx5_virt_lag_is_supported() enough to decide on reg_c_0 matching
  here? Outside MPESW both uplinks share MLX5_ESW_METADATA_RSVD_UPLINK. A
  PF that reports the virt LAG capability bits and also supports SD would
  create a shared FDB with only the reserved uplink metadata in place. The
  commit "add match-metadata allocator for nested VF/SF eswitches" says
  PFs may have virt LAG capabilities.

  [SD] This is not correct, PFs cannot have virt LAG caps; FW exposes
  them only on VF/SF nested eswitch managers. The misleading wording in
  the "add match-metadata allocator" commit message is fixed in v2.

net/mlx5: E-switch, drop master egress rule under a SW vport LAG

  Q: If restoring the master egress rule fails in
  mlx5_eswitch_offloads_vport_lag_del_one(), only a warning is printed
  and the slave rule is removed anyway, so the slave loses forwarding.
  Should the error be propagated?

  [SD] If restoring fails, FW is dead, and the lost traffic is not the
  real problem at that point.

net/mlx5e: Register eswitch-manager VF/SF self-port with switch-id

  Q: The only condition here is MLX5_ESWITCH_MANAGER(mdev), but
  mlx5_query_nic_sw_system_image_guid() makes a VF/SF id differ from the
  PF's only when virt LAG is supported. Can a VF/SF eswitch manager without
  the virt LAG caps give its uplink netdev the same phys_switch_id as its
  parent PF's switch? Should this branch set the switch_id only when the
  resulting GUID differs from the PF's?

  [SD] A VF/SF eswitch manager without the virt LAG caps doesn't exist. New
  FW will let a VF/SF be an eswitch manager only as a nested eswitch
  manager, and such a function reports multi_uplink_eswitch, lag_master and
  num_lag_ports < 2, so mlx5_query_nic_sw_system_image_guid() gives it the
  VF/SF marker and its switch_id differs from its PF's.

  Q: If the system image GUID query fails, the self-port is registered
  with an empty switch id, and nothing fixes it later. Should the error
  be propagated?

  [SD] This is the same behavior as the existing PF self-port path.

net/mlx5: LAG, enable LAG over functions that do not represent a physical port

  > -	if (group_id)
  > +	if (group_id || ldev->virt_lag)
  >  		return 0;

  Q: mlx5_lag_set_vports_agg_speed() iterates ports only, so a virt LAG gets
  no aggregate vport speed, and its members skip the port-change notifier.
  Is there an alternative virt LAG speed policy?

  [SD] This is on purpose.

  Q: members_show() uses mlx5_ldev_for_each(), so the debugfs members file
  is empty for a virt LAG. Could it list all members?

  [SD] Left as is for now, since all VFs in the system share the LAG.

  > -	if (!mlx5_lag_is_supported(dev))
  > +	if (!mlx5_lag_is_supported(dev) && !mlx5_virt_lag_is_supported(dev))
  >  		return;

  Q: mlx5_virt_lag_is_supported() checks multi_uplink_eswitch, lag_master and
  num_lag_ports < 2, but it does not exclude PFs with mlx5_core_is_pf(). Can
  a PF with these caps also get here? Such a PF would keep the plain PF
  devcom key and get an ldev with virt_lag = true. Should this gate exclude
  PFs as well?

  [SD] No PF has these caps. FW exposes multi_uplink_eswitch together with
  lag_master and num_lag_ports < 2 only on the VF/SF that acts as a nested
  eswitch manager, so mlx5_virt_lag_is_supported() is a pure capability
  check on purpose, with no is_vf()/is_sf() in it. The !mlx5_core_is_pf()
  tests in eswitch_offloads.c select an id layout for non-PF eswitches;
  they don't mean a PF can be virt-capable.

  > +	err = xa_alloc(&ldev->fns, &idx, fn, XA_LIMIT(0, U8_MAX - 1),
  > +		       GFP_KERNEL);

  Q: mlx5_virt_lag_is_supported() has no limit on the number of members.
  Once 255 fns exist, the next xa_alloc() returns -EBUSY, and
  mlx5_lag_add_mdev() only bails out on -EOPNOTSUPP. Can the 256th
  function keep retrying inside mlx5_load() forever?

  [SD] The retry loop is pre-existing: mlx5_lag_add_mdev() retried every
  error before this series as well, and no HW exposes this yet; -EBUSY can
  only be returned by future HW. The fix will be sent to net as a separate
  patch.

  Q: mlx5_ib_modify_dct() picks the DCT port with get_tx_affinity_rr()
  whenever mlx5_lag_is_active(), bypassing
  mlx5_ib_lag_should_assign_affinity(). If mlx5_eth_lag_init() left
  lag_active unset, e.g. without a LAG flow namespace, the divisor is
  num_lag_ports, which is 0 on such a function. Can this divide by zero?

  [SD] This is theoretical only; there is no such setup.

  Q: end_flow_index is inclusive, so the master egress group spans
  max_fns + 1 entries in a table created with max_fte = max_fns. With
  max_fns == 128 the table is exactly 128 entries. Can the group creation
  fail?

  [SD] This is theoretical only; there is no such use case.


Shay Drory (5):
  net/mlx5: E-switch, add match-metadata allocator for nested VF/SF
    eswitches
  net/mlx5: E-switch, match master egress rule by vport metadata
  net/mlx5: E-switch, drop master egress rule under a SW vport LAG
  net/mlx5e: Register eswitch-manager VF/SF self-port with switch-id
  net/mlx5: LAG, enable LAG over functions that do not represent a
    physical port

 drivers/infiniband/hw/mlx5/mlx5_ib.h          |   3 +
 .../ethernet/mellanox/mlx5/core/en/devlink.c  |   5 +
 .../net/ethernet/mellanox/mlx5/core/eswitch.h |   3 +-
 .../mellanox/mlx5/core/eswitch_offloads.c     | 150 +++++++++++++++---
 .../net/ethernet/mellanox/mlx5/core/lag/lag.c |  49 ++++--
 .../ethernet/mellanox/mlx5/core/lag/mpesw.c   |   4 +-
 .../mellanox/mlx5/core/lag/shared_fdb.c       |   6 +-
 .../ethernet/mellanox/mlx5/core/mlx5_core.h   |   1 +
 include/linux/mlx5/eswitch.h                  |   4 +
 9 files changed, 185 insertions(+), 40 deletions(-)


base-commit: 2208ab00f4f213ade91ab775e7f8c4dacdcc856e
-- 
2.44.0


^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2026-11-20  8:02 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-20  7:59 [PATCH net-next V2 0/5] net/mlx5: LAG over VFs/SFs (virtual LAG) Tariq Toukan
2026-11-20  7:59 ` [PATCH net-next V2 1/5] net/mlx5: E-switch, add match-metadata allocator for nested VF/SF eswitches Tariq Toukan
2026-11-20  7:59 ` [PATCH net-next V2 2/5] net/mlx5: E-switch, match master egress rule by vport metadata Tariq Toukan
2026-11-20  7:59 ` [PATCH net-next V2 3/5] net/mlx5: E-switch, drop master egress rule under a SW vport LAG Tariq Toukan
2026-11-20  7:59 ` [PATCH net-next V2 4/5] net/mlx5e: Register eswitch-manager VF/SF self-port with switch-id Tariq Toukan
2026-11-20  7:59 ` [PATCH net-next V2 5/5] net/mlx5: LAG, enable LAG over functions that do not represent a physical port Tariq Toukan

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox