All of lore.kernel.org
 help / color / mirror / Atom feed
From: Shay Drory <shayd@nvidia.com>
To: <netdev-bot+sashiko@kernel.org>
Cc: <tariqt@nvidia.com>, <andrew+netdev@lunn.ch>,
	<davem@davemloft.net>, <edumazet@kernel.org>, <kuba@kernel.org>,
	<netdev@vger.kernel.org>, <pabeni@redhat.com>,
	<cjubran@nvidia.com>, <cratiu@nvidia.com>, <gal@nvidia.com>,
	<leon@kernel.org>, <linux-kernel@vger.kernel.org>,
	<linux-rdma@vger.kernel.org>, <mbloch@nvidia.com>,
	<moshe@nvidia.com>, <ohartoov@nvidia.com>, <phaddad@nvidia.com>,
	<saeedm@nvidia.com>, <udar@nvidia.com>
Subject: Re: [PATCH net-next V2 05/10] net/mlx5: LAG, decouple iteration end-of-walk from MLX5_MAX_PORTS
Date: Thu, 19 Nov 2026 10:10:45 +0200	[thread overview]
Message-ID: <20261119081045.1586897-1-shayd@nvidia.com> (raw)
In-Reply-To: <179499791433.3136.10916264856961246459@kernel.org>

On Wed, 18 Nov 2026 10:31:54 +0000, netdev-bot+sashiko@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 5 potential
> issue(s) to consider.
>
> Critical: 0 · High: 1 · Medium: 2 · Low: 2
>
> If you believe none of the issues are legitimate, please reply to each
> issue of Medium severity or higher and explain why the AI is wrong.
> If any of them are legitimate and you plan to respin, please let patchwork
> know by including "pw-bot: cr" as a separate line at the end of your reply
> (one such reply per series is enough).
>
> - [High] mlx5_ldev_add_mdev() raises the xa_alloc() range from
>   XA_LIMIT(0, MLX5_MAX_PORTS - 1) to XA_LIMIT(0, U8_MAX - 1).
> - [Medium] mlx5_handle_changeupper_event() now tests whether the port was
>   found with `if (i < ldev->ports)`.
> - [Medium] The end-of-walk sentinel used to be the compile-time constant
>   MLX5_MAX_PORTS.
> - [Low] In mlx5_lag_create_single_fdb_filter() (shared_fdb.c), the
>   max_slaves argument to mlx5_eswitch_offloads_single_fdb_add_one()…
> - [Low] mlx5_lag_dev_alloc() sets `ldev->max_funcs = ldev->ports` only at
>   the very end.
>
> --
>
> net/mlx5: LAG, decouple iteration end-of-walk from MLX5_MAX_PORTS
>
> This patch adds ldev->max_funcs and uses it as the end-of-walk sentinel
> for the LAG xarray iterators in place of MLX5_MAX_PORTS. max_funcs grows
> when a member is added at a higher xarray index. The patch also raises
> the xa_alloc() limit in mlx5_ldev_add_mdev() from MLX5_MAX_PORTS - 1 to
> U8_MAX - 1.
>
> > diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
> > index 669b2167a82d2..cbef6ab1e38a8 100644
> > --- a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
> > +++ b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
> > @@ -395,6 +395,7 @@ static struct mlx5_lag *mlx5_lag_dev_alloc(struct mlx5_core_dev *dev)
> >  		mlx5_core_err(dev, "Failed to init multipath lag err=%d\n",
> >  			      err);
> >
> > +	ldev->max_funcs = ldev->ports;
> >  	return ldev;
>
> [Severity: Low]
> Can a netdev event see max_funcs == 0 here?
>
> By the time this assignment runs, register_netdevice_notifier_net() has
> already made ldev->nb live. mlx5_lag_mp_init() has also registered the
> FIB notifier, which dumps the FIB and can take a while. Until this line,
> max_funcs is still 0 from kzalloc.
>
> Suppose a NETDEV_CHANGEUPPER for a bond in the same netns arrives in that
> window. mlx5_handle_changeupper_event() then runs mlx5_ldev_for_each()
> with max_funcs == 0:
>
> - mlx5_get_next_lag_func() returns 0.
> - The loop exits with i = 0 and idx still -1.
> - The new "i < ldev->ports" check passes.
>
> The handler then executes:
>
>     bond_status |= (1 << idx);
>
> This is a shift by -1. Would it be better to set max_funcs right after
> ldev->ports is assigned, before any notifier is registered?

The window is very narrow and harmless: bond_status stays 0 and the
handler returns early, so nothing is changed. It will be fixed in a
follow-up.

> [ ... ]
>
> > @@ -2094,7 +2095,7 @@ static int mlx5_handle_changeupper_event(struct mlx5_lag *ldev,
> >  				break;
> >  			}
> >  		}
> > -		if (i < MLX5_MAX_PORTS) {
> > +		if (i < ldev->ports) {
>
> [Severity: Medium]
> Should this compare against ldev->max_funcs, like the other converted call
> sites? mlx5_ldev_for_each() now ends with i == ldev->max_funcs. This patch
> also explicitly allows members at xarray indices >= ldev->ports, since
> mlx5_ldev_add_mdev() grows max_funcs for that case.
>
> xa_alloc() hands out the lowest free index to both port and SD members.
> For example, after a PF is unbound and rebound, SD secondaries added by
> sd_lag_init() can hold the low indices, so the matched port PF can end up
> at an index >= ports.
>
> In that case the loop breaks with i equal to that index, "i < ldev->ports"
> is false, and the PF is treated as not found. Its bond_status bit is never
> set and has_inactive is not computed for it, so this never becomes true:
>
>     is_in_lag = num_slaves == ldev->ports &&
>         bond_status == GENMASK(ldev->ports - 1, 0);
>
> As a result, the bond is never offloaded. Before this patch, indices were
> capped at 7, so any found entry passed the check.
>
> The notifier is only registered when !mlx5_sd_is_supported(dev), so this
> may need a mixed or unusual topology to trigger. The check is still the
> same at the end of the series.

No. mlx5 creates an ldev per sw_guid, and PFs have a different
sw_guid than VFs/SFs, so only PFs share this ldev. The only case with
more PFs than num_lag_ports in one ldev is SD, and for SD the netdev
notifier, and with it mlx5_handle_changeupper_event(), is not
registered. So a PORT-marked member here always sits at an index below
ldev->ports.

> [ ... ]
>
> > @@ -2359,13 +2360,16 @@ int mlx5_ldev_add_mdev(struct mlx5_lag *ldev,
> >  	if (!pf)
> >  		return -ENOMEM;
> >
> > -	err = xa_alloc(&ldev->pfs, &idx, pf, XA_LIMIT(0, MLX5_MAX_PORTS - 1),
> > +	err = xa_alloc(&ldev->pfs, &idx, pf, XA_LIMIT(0, U8_MAX - 1),
> >  		       GFP_KERNEL);
>
> [Severity: High]
> Is it safe to drop the MLX5_MAX_PORTS limit before the arrays that depend
> on it are resized?
>
> This limit was the only thing that kept both the member count and every
> xarray index below MLX5_MAX_PORTS (8). Neither __mlx5_lag_dev_add_mdev()
> nor sd_lag_init() checks the member count.
>
> At this commit, once an ldev has more than 8 members, these writes look
> like they go out of bounds:
>
> - mlx5_mpesw_metadata_set() writes ldev->lag_mpesw.pf_metadata[i]
>   (u32[MLX5_MAX_PORTS]) under MLX5_LAG_FILTER_ALL. lag_mpesw is the last
>   member of struct mlx5_lag, so this writes past the kzalloc'd ldev.
>
> - mlx5_lag_rescan_devices_locked_filter() fills the on-stack
>   devs[MLX5_MAX_PORTS] with devs[num_devs++] without any bound check.
>
> - mlx5_lag_query_cong_counters() fills a buffer of MLX5_MAX_PORTS entries
>   with mdev[idx++].
>
> - struct lag_tracker::netdev_state[MLX5_MAX_PORTS] is written at the
>   port's xarray index in mlx5_ldev_add_netdev() and in
>   mlx5_handle_changelowerstate_event():
>
>       tracker->netdev_state[idx] = *lag_lower_info;
>
> - struct mlx5_lag_definer::rules[MLX5_MAX_PORTS *
>   MLX5_LAG_MAX_HASH_BUCKETS] is indexed by i * ldev->buckets + j in
>   port_sel.c, where i is the port's xarray index.
>
> Before this patch, such a configuration failed cleanly: xa_alloc()
> returned -EBUSY and sd_lag_init() rolled back the secondaries. An 8-port
> SD configuration, for example, would now give 16 members.
>
> Later patches in the series fix some of these:
>
> - "net/mlx5: LAG: replace fixed MLX5_MAX_PORTS arrays with dynamic
>   allocation" makes pf_metadata kcalloc(max_fns) and sizes the
>   cong-counter buffer from ports.
> - "net/mlx5: LAG, drop MLX5_MAX_PORTS scratch array from device rescan"
>   removes devs[].
>
> At the end of the series, though, lag_tracker::netdev_state[] and
> mlx5_lag_definer::rules[] are still sized by MLX5_MAX_PORTS and indexed
> by xarray indices that are no longer bounded.

In this series a 9th member is impossible: an ldev holds only
PFs and SD secondaries, and no HW has more than MLX5_MAX_PORTS of them.
VFs/SFs join an ldev only in the follow-up series.
The limit can't be kept until all arrays are converted, since not all
of them will be. Some arrays are used only by a PF LAG, others by both a
PF LAG and a virt LAG. The shared ones, the fns xarray included, are
converted in this series (pf_metadata[] and devs[]; the cong counter
array is converted as well). The PF LAG only ones,
tracker.netdev_state[] and the port_sel rules[], keep their static
MLX5_MAX_PORTS size: they are walked only over PORT-marked members, in
the hash and queue affinity modes. A virt member is never PORT-marked
and sits in its own ldev, and a virt LAG supports only MPESW, so these
arrays never see more than MLX5_MAX_PORTS entries.

> The commit message also says:
>
>     No functional change; this only removes the dependency on
>     MLX5_MAX_PORTS.
>
> Isn't dropping the xa_alloc() bound a functional change in itself?

The xa_alloc() change is already in the commit message ("remove the
xa_alloc() limit of MLX5_MAX_PORTS"), and the answer above explains why
it can't let in a 9th member.

> >  	if (err) {
> >  		kfree(pf);
> >  		return err;
> >  	}
> >
> > +	/* max_funcs is the iteration ceiling; grow it, never shrink. */
> > +	if (idx >= ldev->max_funcs)
> > +		ldev->max_funcs = idx + 1;
> >  	pf->idx = idx;
> >  	pf->dev = dev;
>
> [Severity: Medium]
> Can walkers that don't hold ldev->lock see an inconsistent max_funcs?
>

No, all these walkers run either during LAG activation or while
the LAG is active, and membership doesn't change then.

> The end-of-walk sentinel used to be a constant. It is now this field,
> which is written with a plain store under ldev->lock only. Each iteration
> of mlx5_lag_for_each() loads it twice: once as the return value of
> mlx5_get_next_lag_func(), and again in the macro test:
>
>     i = tmp, tmp < (ldev)->max_funcs; tmp++)
>
> Several walkers don't hold ldev->lock:
>
> - mlx5_mpesw_sd_devcoms_lock(), called from mlx5_mpesw_work() and
>   mlx5_lag_disable_change(), runs under the HCA devcom lock but before
>   mutex_lock(&ldev->lock).
> - mlx5_ldev_add_netdev(), mlx5_lag_get_next_peer_mdev() and
>   mlx5_lag_query_cong_counters() hold only lag_lock.
> - mlx5_handle_changeupper_event() holds only RCU.
>
> sd_lag_init() adds secondaries while holding only ldev->lock. The writer
> order is:
>
> 1. xa_alloc() publishes a zeroed pf at idx == old max_funcs.
> 2. max_funcs = idx + 1.
> 3. pf->dev = dev.
> 4. The PORT mark is set.
>
> Suppose a reader's walk ends while the new pf isn't valid yet. It gets
> sentinel N back from mlx5_get_next_lag_func(), then re-reads max_funcs as
> N + 1. The loop body then runs once with i = N, skipping the !pf->dev
> check and the PORT-mark filter. In mlx5_mpesw_sd_devcoms_lock() that
> means:
>
>     sd_devcom = mlx5_sd_get_devcom(mlx5_lag_pf(ldev, i)->dev);
>
> runs with a NULL dev, which mlx5_get_sd() dereferences. In
> mlx5_lag_get_next_peer_mdev(), a NULL or non-port dev could be returned
> as a peer.
>
> Also, on weakly ordered architectures nothing orders this store against
> the xarray slot, so xa_load(N) could return NULL as well.
> [ ... ]
>
> > diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lag/shared_fdb.c b/drivers/net/ethernet/mellanox/mlx5/core/lag/shared_fdb.c
> > index 7c7e6c832a93d..29048b0c27977 100644
> > --- a/drivers/net/ethernet/mellanox/mlx5/core/lag/shared_fdb.c
> > +++ b/drivers/net/ethernet/mellanox/mlx5/core/lag/shared_fdb.c
> > @@ -71,7 +71,7 @@ static int mlx5_lag_create_single_fdb_filter(struct mlx5_lag *ldev, u32 filter)
> >
> >  		err = mlx5_eswitch_offloads_single_fdb_add_one(master_esw,
> >  							       slave_esw,
> > -							       ldev->ports);
> > +							       ldev->max_funcs);
>
> [Severity: Low]
> Is this change intended? This argument isn't an end-of-walk sentinel. It
> is max_slaves, which goes through esw_set_master_egress_rule() to
> esw_master_egress_create_resources(). There it becomes ft_attr.max_fte
> and the bounce group's end_flow_index for the master's egress ACL table
> in firmware.
>
> With SD, secondaries take extra xarray indices, so max_funcs can exceed
> ports (for example, 4 on a 2-port SD card) and the ACL table is created
> larger. max_funcs is also a high-water mark of xarray indices that never
> shrinks, not a slave count.
>
> Bounce rules are only added for SD primaries, so ldev->ports seems to
> have been sufficient already.
>
> This doesn't match "No functional change" in the commit message. Could
> the change be explained in the commit message, or dropped?

Only the ACL size changes, and only for SD, where more PFs than
ports join the ldev and max_funcs > ports. The rules installed are the
same as before, so there is no functional change.

> --
> Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261116103010.1129666-1-tariqt%40nvidia.com

  reply	other threads:[~2026-11-19  8:11 UTC|newest]

Thread overview: 33+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-16 10:30 [PATCH net-next V2 00/10] net/mlx5: LAG, preparations for LAG over VFs/SFs Tariq Toukan
2026-11-16 10:30 ` [PATCH net-next V2 01/10] net/mlx5: Distinguish VFs/SFs in the SW system image GUID for LAG grouping Tariq Toukan
2026-11-17 10:36   ` sashiko-bot
2026-11-18 10:31   ` netdev-bot+sashiko
2026-11-19  7:40     ` Shay Drory
2026-11-16 10:30 ` [PATCH net-next V2 02/10] net/mlx5: LAG, select the MPESW master from the live member snapshot Tariq Toukan
2026-11-17 10:36   ` sashiko-bot
2026-11-16 10:30 ` [PATCH net-next V2 03/10] net/mlx5: LAG, skip phys LAG prerequisites for a virt LAG Tariq Toukan
2026-11-17 10:36   ` sashiko-bot
2026-11-16 10:30 ` [PATCH net-next V2 04/10] net/mlx5: LAG, treat virt LAG as a software managed LAG Tariq Toukan
2026-11-17 10:36   ` sashiko-bot
2026-11-16 10:30 ` [PATCH net-next V2 05/10] net/mlx5: LAG, decouple iteration end-of-walk from MLX5_MAX_PORTS Tariq Toukan
2026-11-17 10:36   ` sashiko-bot
2026-11-18 10:31   ` netdev-bot+sashiko
2026-11-19  8:10     ` Shay Drory [this message]
2026-11-16 10:30 ` [PATCH net-next V2 06/10] net/mlx5: LAG, rename PF naming to fn Tariq Toukan
2026-11-17 10:36   ` sashiko-bot
2026-11-18 10:31   ` netdev-bot+sashiko
2026-11-16 10:30 ` [PATCH net-next V2 07/10] net/mlx5: LAG, resolve affinity indices only in queue-affinity mode Tariq Toukan
2026-11-17 10:36   ` sashiko-bot
2026-11-16 10:30 ` [PATCH net-next V2 08/10] net/mlx5: LAG: replace fixed MLX5_MAX_PORTS arrays with dynamic allocation Tariq Toukan
2026-11-17 10:36   ` sashiko-bot
2026-11-18 10:31   ` netdev-bot+sashiko
2026-11-19  8:20     ` Shay Drory
2026-11-16 10:30 ` [PATCH net-next V2 09/10] net/mlx5: LAG, drop MLX5_MAX_PORTS scratch array from device rescan Tariq Toukan
2026-11-17 10:36   ` sashiko-bot
2026-11-18 10:31   ` netdev-bot+sashiko
2026-11-19  8:40     ` Shay Drory
2026-11-16 10:30 ` [PATCH net-next V2 10/10] net/mlx5: LAG, do not add a device to an already-active virt LAG Tariq Toukan
2026-11-17 10:36   ` sashiko-bot
2026-11-18 10:32   ` netdev-bot+sashiko
2026-11-19  8:58     ` Shay Drory
2026-11-20  0:40 ` [PATCH net-next V2 00/10] net/mlx5: LAG, preparations for LAG over VFs/SFs patchwork-bot+netdevbpf

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261119081045.1586897-1-shayd@nvidia.com \
    --to=shayd@nvidia.com \
    --cc=andrew+netdev@lunn.ch \
    --cc=cjubran@nvidia.com \
    --cc=cratiu@nvidia.com \
    --cc=davem@davemloft.net \
    --cc=edumazet@kernel.org \
    --cc=gal@nvidia.com \
    --cc=kuba@kernel.org \
    --cc=leon@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=mbloch@nvidia.com \
    --cc=moshe@nvidia.com \
    --cc=netdev-bot+sashiko@kernel.org \
    --cc=netdev@vger.kernel.org \
    --cc=ohartoov@nvidia.com \
    --cc=pabeni@redhat.com \
    --cc=phaddad@nvidia.com \
    --cc=saeedm@nvidia.com \
    --cc=tariqt@nvidia.com \
    --cc=udar@nvidia.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.