From: Tariq Toukan <tariqt@nvidia.com>
To: Andrew Lunn <andrew+netdev@lunn.ch>,
"David S. Miller" <davem@davemloft.net>,
Eric Dumazet <edumazet@kernel.org>,
Jakub Kicinski <kuba@kernel.org>, <netdev@vger.kernel.org>,
Paolo Abeni <pabeni@redhat.com>
Cc: Akiva Goldberger <agoldberger@nvidia.com>,
Gal Pressman <gal@nvidia.com>, Jason Gunthorpe <jgg@ziepe.ca>,
Leon Romanovsky <leon@kernel.org>, <linux-rdma@vger.kernel.org>,
Mark Bloch <mbloch@nvidia.com>, Moshe Shemesh <moshe@nvidia.com>,
Or Har-Toov <ohartoov@nvidia.com>,
Saeed Mahameed <saeedm@nvidia.com>, Shay Drory <shayd@nvidia.com>,
Tariq Toukan <tariqt@nvidia.com>
Subject: [PATCH net-next V2 0/5] net/mlx5: LAG over VFs/SFs (virtual LAG)
Date: Fri, 20 Nov 2026 09:59:53 +0200 [thread overview]
Message-ID: <20261120075959.2533702-1-tariqt@nvidia.com> (raw)
This is V2 of the top patches submitted previously here [1].
This series by Shay enables a LAG over functions that do not represent a
physical port - a "virtual LAG" (virt LAG) of VFs or SFs acting as
nested e-switch managers. New FW lets a VF/SF front the physical uplinks
through a multi-uplink e-switch; such a function reports num_lag_ports <
2 and drives port selection via the e-switch (port_select_eswitch)
rather than the HW port-selection flow table. A virt LAG is therefore a
software-managed, shared-FDB MPESW LAG: the driver builds the LAG demux
and vport-LAG in software.
It depends on [2] which prepared the LAG core.
The patches fall into two parts.
E-switch and representors (patches 1, 2, 3, 4)
- A dedicated match-metadata allocator for nested VF/SF e-switches, so
VFs/SFs sharing an FDB get unique source-port metadata.
- Match the master egress bounce rule by vport metadata.
- Drop the master egress rule while a software vport LAG is installed:
the SW emulation is a rule in the slave's egress ACL and cannot act
on a packet the master egress rule redirects after the FDB. Restore
it when the vport LAG is removed.
- Register a VF/SF e-switch-manager self-port with a switch-id so it
groups with the representors it manages.
Virtual LAG enablement (patch 5)
- Enable LAG over functions that do not represent a physical port -
the gate that admits a virt LAG.
Regards,
Tariq
[1] https://lore.kernel.org/all/20261103171009.563034-1-tariqt@nvidia.com/
[2] https://lore.kernel.org/all/20261116103010.1129666-1-tariqt@nvidia.com/
V2:
- Split the patch set to ease the review: this is v1 patches 11-15; v1
patches 1-10 are sent as "net/mlx5: LAG, preparations for LAG over
VFs/SFs", which this series depends on.
- Patch 1: commit message describes which non-PF eswitches take the
nested metadata layout, and no longer implies a PF may have the virt
LAG capabilities. The reg_c_0 comment in include/linux/mlx5/eswitch.h
describes the nested layout (Sashiko).
- Patch 3: commit message names the field ldev->max_fns, not
ldev->max_funcs (Sashiko).
- Patch 5: do not assign an RDMA TX port affinity on a virt LAG capable
function, and do not register the multipath FIB notifier for a virt
LAG (Sashiko).
V1:
https://lore.kernel.org/all/20261103171009.563034-1-tariqt@nvidia.com/
Answers to sashiko comments on v1:
net/mlx5: E-switch, add match-metadata allocator for nested VF/SF eswitches
> + if (!mlx5_core_is_pf(esw->dev) && !mlx5_virt_lag_nonmember(esw->dev))
> + return mlx5_nested_esw_match_metadata_alloc(esw);
Q: mlx5_virt_lag_nonmember() is false when virt LAG is unsupported, so a
standalone non-PF eswitch without virt LAG support takes the nested path
and fails switchdev with -ENOSPC. Could it keep the PFNUM allocator?
[SD] A non-PF eswitch manager without virt LAG support doesn't exist at
the moment. FW doesn't expose such a VF/SF.
Q: The nested allocator gives each eswitch 254 ids (2-255), yet a
nested eswitch can enumerate up to max_num_sf SF vports, each taking an
id. Also, LAG admission accepts slots 0-254, while the allocator gives
no metadata to slots above 239. Can switchdev fail with -ENOSPC in
either case?
[SD] Both are theoretical only; there is no such use case.
net/mlx5: E-switch, match master egress rule by vport metadata
Q: Is mlx5_virt_lag_is_supported() enough to decide on reg_c_0 matching
here? Outside MPESW both uplinks share MLX5_ESW_METADATA_RSVD_UPLINK. A
PF that reports the virt LAG capability bits and also supports SD would
create a shared FDB with only the reserved uplink metadata in place. The
commit "add match-metadata allocator for nested VF/SF eswitches" says
PFs may have virt LAG capabilities.
[SD] This is not correct, PFs cannot have virt LAG caps; FW exposes
them only on VF/SF nested eswitch managers. The misleading wording in
the "add match-metadata allocator" commit message is fixed in v2.
net/mlx5: E-switch, drop master egress rule under a SW vport LAG
Q: If restoring the master egress rule fails in
mlx5_eswitch_offloads_vport_lag_del_one(), only a warning is printed
and the slave rule is removed anyway, so the slave loses forwarding.
Should the error be propagated?
[SD] If restoring fails, FW is dead, and the lost traffic is not the
real problem at that point.
net/mlx5e: Register eswitch-manager VF/SF self-port with switch-id
Q: The only condition here is MLX5_ESWITCH_MANAGER(mdev), but
mlx5_query_nic_sw_system_image_guid() makes a VF/SF id differ from the
PF's only when virt LAG is supported. Can a VF/SF eswitch manager without
the virt LAG caps give its uplink netdev the same phys_switch_id as its
parent PF's switch? Should this branch set the switch_id only when the
resulting GUID differs from the PF's?
[SD] A VF/SF eswitch manager without the virt LAG caps doesn't exist. New
FW will let a VF/SF be an eswitch manager only as a nested eswitch
manager, and such a function reports multi_uplink_eswitch, lag_master and
num_lag_ports < 2, so mlx5_query_nic_sw_system_image_guid() gives it the
VF/SF marker and its switch_id differs from its PF's.
Q: If the system image GUID query fails, the self-port is registered
with an empty switch id, and nothing fixes it later. Should the error
be propagated?
[SD] This is the same behavior as the existing PF self-port path.
net/mlx5: LAG, enable LAG over functions that do not represent a physical port
> - if (group_id)
> + if (group_id || ldev->virt_lag)
> return 0;
Q: mlx5_lag_set_vports_agg_speed() iterates ports only, so a virt LAG gets
no aggregate vport speed, and its members skip the port-change notifier.
Is there an alternative virt LAG speed policy?
[SD] This is on purpose.
Q: members_show() uses mlx5_ldev_for_each(), so the debugfs members file
is empty for a virt LAG. Could it list all members?
[SD] Left as is for now, since all VFs in the system share the LAG.
> - if (!mlx5_lag_is_supported(dev))
> + if (!mlx5_lag_is_supported(dev) && !mlx5_virt_lag_is_supported(dev))
> return;
Q: mlx5_virt_lag_is_supported() checks multi_uplink_eswitch, lag_master and
num_lag_ports < 2, but it does not exclude PFs with mlx5_core_is_pf(). Can
a PF with these caps also get here? Such a PF would keep the plain PF
devcom key and get an ldev with virt_lag = true. Should this gate exclude
PFs as well?
[SD] No PF has these caps. FW exposes multi_uplink_eswitch together with
lag_master and num_lag_ports < 2 only on the VF/SF that acts as a nested
eswitch manager, so mlx5_virt_lag_is_supported() is a pure capability
check on purpose, with no is_vf()/is_sf() in it. The !mlx5_core_is_pf()
tests in eswitch_offloads.c select an id layout for non-PF eswitches;
they don't mean a PF can be virt-capable.
> + err = xa_alloc(&ldev->fns, &idx, fn, XA_LIMIT(0, U8_MAX - 1),
> + GFP_KERNEL);
Q: mlx5_virt_lag_is_supported() has no limit on the number of members.
Once 255 fns exist, the next xa_alloc() returns -EBUSY, and
mlx5_lag_add_mdev() only bails out on -EOPNOTSUPP. Can the 256th
function keep retrying inside mlx5_load() forever?
[SD] The retry loop is pre-existing: mlx5_lag_add_mdev() retried every
error before this series as well, and no HW exposes this yet; -EBUSY can
only be returned by future HW. The fix will be sent to net as a separate
patch.
Q: mlx5_ib_modify_dct() picks the DCT port with get_tx_affinity_rr()
whenever mlx5_lag_is_active(), bypassing
mlx5_ib_lag_should_assign_affinity(). If mlx5_eth_lag_init() left
lag_active unset, e.g. without a LAG flow namespace, the divisor is
num_lag_ports, which is 0 on such a function. Can this divide by zero?
[SD] This is theoretical only; there is no such setup.
Q: end_flow_index is inclusive, so the master egress group spans
max_fns + 1 entries in a table created with max_fte = max_fns. With
max_fns == 128 the table is exactly 128 entries. Can the group creation
fail?
[SD] This is theoretical only; there is no such use case.
Shay Drory (5):
net/mlx5: E-switch, add match-metadata allocator for nested VF/SF
eswitches
net/mlx5: E-switch, match master egress rule by vport metadata
net/mlx5: E-switch, drop master egress rule under a SW vport LAG
net/mlx5e: Register eswitch-manager VF/SF self-port with switch-id
net/mlx5: LAG, enable LAG over functions that do not represent a
physical port
drivers/infiniband/hw/mlx5/mlx5_ib.h | 3 +
.../ethernet/mellanox/mlx5/core/en/devlink.c | 5 +
.../net/ethernet/mellanox/mlx5/core/eswitch.h | 3 +-
.../mellanox/mlx5/core/eswitch_offloads.c | 150 +++++++++++++++---
.../net/ethernet/mellanox/mlx5/core/lag/lag.c | 49 ++++--
.../ethernet/mellanox/mlx5/core/lag/mpesw.c | 4 +-
.../mellanox/mlx5/core/lag/shared_fdb.c | 6 +-
.../ethernet/mellanox/mlx5/core/mlx5_core.h | 1 +
include/linux/mlx5/eswitch.h | 4 +
9 files changed, 185 insertions(+), 40 deletions(-)
base-commit: 2208ab00f4f213ade91ab775e7f8c4dacdcc856e
--
2.44.0
next reply other threads:[~2026-11-20 8:00 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-20 7:59 Tariq Toukan [this message]
2026-11-20 7:59 ` [PATCH net-next V2 1/5] net/mlx5: E-switch, add match-metadata allocator for nested VF/SF eswitches Tariq Toukan
2026-11-20 7:59 ` [PATCH net-next V2 2/5] net/mlx5: E-switch, match master egress rule by vport metadata Tariq Toukan
2026-11-20 7:59 ` [PATCH net-next V2 3/5] net/mlx5: E-switch, drop master egress rule under a SW vport LAG Tariq Toukan
2026-11-20 7:59 ` [PATCH net-next V2 4/5] net/mlx5e: Register eswitch-manager VF/SF self-port with switch-id Tariq Toukan
2026-11-20 7:59 ` [PATCH net-next V2 5/5] net/mlx5: LAG, enable LAG over functions that do not represent a physical port Tariq Toukan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261120075959.2533702-1-tariqt@nvidia.com \
--to=tariqt@nvidia.com \
--cc=agoldberger@nvidia.com \
--cc=andrew+netdev@lunn.ch \
--cc=davem@davemloft.net \
--cc=edumazet@kernel.org \
--cc=gal@nvidia.com \
--cc=jgg@ziepe.ca \
--cc=kuba@kernel.org \
--cc=leon@kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=mbloch@nvidia.com \
--cc=moshe@nvidia.com \
--cc=netdev@vger.kernel.org \
--cc=ohartoov@nvidia.com \
--cc=pabeni@redhat.com \
--cc=saeedm@nvidia.com \
--cc=shayd@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox