[PATCH net-next V3 0/7] net/mlx5: Improve representor lifecycle and late IB representor loading

[Tariq Toukan <tariqt@nvidia.com>]

Hi,

Find detailed description by Mark below.

Regards,
Tariq


This series addresses two problems that have been present for years, and
fixes one representor reload error-unwind case exposed while making the
reload path reusable.

First, there is no coordination between E-Switch reconfiguration and
representor registration. The E-Switch can be mid-way through a mode
change or VF count update while mlx5_ib walks in and registers or
unregisters representors. Nothing stops them. The race window is small
and there is no field report, but it is clearly wrong.

Second, loading mlx5_ib while the device is already in switchdev mode
does not bring up the IB representors. mlx5_eswitch_register_vport_reps()
only stores callbacks; nobody triggers the actual load after registration.

The series fixes the registration race with a per-E-Switch representor
mutex. The lock is introduced first, then LAG shared-FDB and multiport
E-Switch transitions are adjusted so auxiliary device rescans and IB
representor reloads do not hold ldev->lock while taking the representor
lock. This keeps the intermediate commits bisectable before the stricter
E-Switch serialization and lock assertions are enabled.

After the LAG ordering is fixed, all E-Switch reconfiguration paths that
create, destroy, load, or unload representors take the representor mutex.
esw_mode_change() deliberately drops the mutex around
mlx5_rescan_drivers_locked(), because auxiliary probe and remove paths
re-enter mlx5_eswitch_register_vport_reps() and
mlx5_eswitch_unregister_vport_reps() on the same thread.

The shared-FDB peer IB registration path can hold one E-Switch
representor mutex and then register peer representor ops on another
E-Switch. The series annotates that case as nested locking so lockdep can
distinguish it from recursive locking on the same E-Switch.

For the missing IB representors, mlx5_eswitch_register_vport_reps() queues
a work item that acquires the devlink lock and loads all relevant
representors. This is the change that actually fixes the long-standing
bug.

The reload path also learns to track which representor types were loaded by
the current attempt, so an error does not unload representors that were
already active before the retry.

Patch 1 is cleanup. LAG and MPESW had the same representor reload
sequence duplicated in several places and the copies had started to
drift. This consolidates them into one helper.

Patch 2 lets E-Switch workqueue callers choose GFP allocation flags.

Patch 3 adds the per-E-Switch representor lifecycle lock and helper APIs.

Patch 4 adjusts the LAG shared-FDB and multiport E-Switch transitions so
auxiliary device rescans and IB representor reloads run without
ldev->lock held while taking the representor lock.

Patch 5 protects the E-Switch reconfiguration, representor registration
and peer IB representor paths with the representor lock.

Patch 6 fixes representor load error unwind so only representor types
loaded by the current attempt are unloaded on failure.

Patch 7 moves the representor load triggered by
mlx5_eswitch_register_vport_reps() onto the work queue. This is the patch
that fixes IB representors not coming up when mlx5_ib is loaded while the
device is already in switchdev mode.

Changes:

v2 -> v3:

Drop the default switchdev module parameter patch. The proper user facing
interface is still under discussion, and this may be better handled by
devlink core infrastructure.

Patch 2: Add a new patch, per Sashiko's feedback, that lets E-Switch
workqueue callers pass GFP allocation flags to mlx5_esw_add_work(). The
functions-change notifier keeps using GFP_ATOMIC, while sleepable callers
can use GFP_KERNEL.

Patch 5: The unregister path now always unloads the selected representor
type before marking it unregistered and clearing rep_ops. It no longer
depends on esw->mode == MLX5_ESWITCH_OFFLOADS.

Patch 7: The queued late representor reload now calls mlx5_esw_add_work()
with GFP_KERNEL instead of relying on the helper's previous hardcoded
GFP_ATOMIC allocation.

v1 -> v2:

Split v1 into two parts: the E-Switch workqueue deadlock fix and the
representor lifecycle changes. This is the second part; the first part
has already been accepted [1].

Patch 1: Add a cont_on_fail flag so callers can decide whether reload
should continue after a failure.

Patches 2, 3, 4: Replace the atomic-variable based scheme with a mutex,
per Jakub's feedback.

Patch 5: New patch that fixes the unwind on representor load failure.

Patch 7: Switch from profile 4 to profile 8. Since the profile mainly
targets E-Switch handling, keep it separate from the NIC profiles.

V2:
https://lore.kernel.org/all/20260501041633.231662-1-tariqt@nvidia.com/

V1:
https://lore.kernel.org/all/20260409115550.156419-1-tariqt@nvidia.com/

[1] https://lore.kernel.org/all/20260428051018.219093-1-tariqt@nvidia.com/


Mark Bloch (7):
  net/mlx5: Lag: refactor representor reload handling
  net/mlx5: E-Switch, let esw work callers choose GFP flags
  net/mlx5: E-Switch, add representor lifecycle lock
  net/mlx5: Lag, avoid LAG and representor lock cycles
  net/mlx5: E-Switch, serialize representor lifecycle
  net/mlx5: E-Switch, unwind only newly loaded representor types
  net/mlx5: E-Switch, load reps via work queue after registration

 drivers/infiniband/hw/mlx5/ib_rep.c           |   6 +-
 .../net/ethernet/mellanox/mlx5/core/eswitch.c |  10 +
 .../net/ethernet/mellanox/mlx5/core/eswitch.h |   6 +
 .../mellanox/mlx5/core/eswitch_offloads.c     | 197 ++++++++++++++++--
 .../net/ethernet/mellanox/mlx5/core/lag/lag.c | 171 +++++++++++----
 .../net/ethernet/mellanox/mlx5/core/lag/lag.h |   5 +
 .../ethernet/mellanox/mlx5/core/lag/mpesw.c   |  18 +-
 .../ethernet/mellanox/mlx5/core/lib/devcom.c  |   8 +
 .../ethernet/mellanox/mlx5/core/lib/devcom.h  |   1 +
 .../ethernet/mellanox/mlx5/core/sf/devlink.c  |   5 +
 include/linux/mlx5/eswitch.h                  |   6 +
 11 files changed, 361 insertions(+), 72 deletions(-)


base-commit: 98878ed91b68a3150126fccef125ee7b1bb86ab2
-- 
2.44.0