Netdev List
 help / color / mirror / Atom feed
From: Shay Drory <shayd@nvidia.com>
To: <netdev-bot+sashiko@kernel.org>
Cc: <tariqt@nvidia.com>, <andrew+netdev@lunn.ch>,
	<davem@davemloft.net>, <edumazet@kernel.org>, <kuba@kernel.org>,
	<netdev@vger.kernel.org>, <pabeni@redhat.com>,
	<cjubran@nvidia.com>, <cratiu@nvidia.com>, <gal@nvidia.com>,
	<leon@kernel.org>, <linux-kernel@vger.kernel.org>,
	<linux-rdma@vger.kernel.org>, <mbloch@nvidia.com>,
	<moshe@nvidia.com>, <ohartoov@nvidia.com>, <phaddad@nvidia.com>,
	<saeedm@nvidia.com>, <udar@nvidia.com>
Subject: Re: [PATCH net-next V2 05/10] net/mlx5: LAG, decouple iteration end-of-walk from MLX5_MAX_PORTS
Date: Thu, 19 Nov 2026 10:10:45 +0200	[thread overview]
Message-ID: <20261119081045.1586897-1-shayd@nvidia.com> (raw)
In-Reply-To: <179499791433.3136.10916264856961246459@kernel.org>

On Wed, 18 Nov 2026 10:31:54 +0000, netdev-bot+sashiko@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 5 potential
> issue(s) to consider.
>
> Critical: 0 · High: 1 · Medium: 2 · Low: 2
>
> If you believe none of the issues are legitimate, please reply to each
> issue of Medium severity or higher and explain why the AI is wrong.
> If any of them are legitimate and you plan to respin, please let patchwork
> know by including "pw-bot: cr" as a separate line at the end of your reply
> (one such reply per series is enough).
>
> - [High] mlx5_ldev_add_mdev() raises the xa_alloc() range from
>   XA_LIMIT(0, MLX5_MAX_PORTS - 1) to XA_LIMIT(0, U8_MAX - 1).
> - [Medium] mlx5_handle_changeupper_event() now tests whether the port was
>   found with `if (i < ldev->ports)`.
> - [Medium] The end-of-walk sentinel used to be the compile-time constant
>   MLX5_MAX_PORTS.
> - [Low] In mlx5_lag_create_single_fdb_filter() (shared_fdb.c), the
>   max_slaves argument to mlx5_eswitch_offloads_single_fdb_add_one()…
> - [Low] mlx5_lag_dev_alloc() sets `ldev->max_funcs = ldev->ports` only at
>   the very end.
>
> --
>
> net/mlx5: LAG, decouple iteration end-of-walk from MLX5_MAX_PORTS
>
> This patch adds ldev->max_funcs and uses it as the end-of-walk sentinel
> for the LAG xarray iterators in place of MLX5_MAX_PORTS. max_funcs grows
> when a member is added at a higher xarray index. The patch also raises
> the xa_alloc() limit in mlx5_ldev_add_mdev() from MLX5_MAX_PORTS - 1 to
> U8_MAX - 1.
>
> > diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
> > index 669b2167a82d2..cbef6ab1e38a8 100644
> > --- a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
> > +++ b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
> > @@ -395,6 +395,7 @@ static struct mlx5_lag *mlx5_lag_dev_alloc(struct mlx5_core_dev *dev)
> >  		mlx5_core_err(dev, "Failed to init multipath lag err=%d\n",
> >  			      err);
> >
> > +	ldev->max_funcs = ldev->ports;
> >  	return ldev;
>
> [Severity: Low]
> Can a netdev event see max_funcs == 0 here?
>
> By the time this assignment runs, register_netdevice_notifier_net() has
> already made ldev->nb live. mlx5_lag_mp_init() has also registered the
> FIB notifier, which dumps the FIB and can take a while. Until this line,
> max_funcs is still 0 from kzalloc.
>
> Suppose a NETDEV_CHANGEUPPER for a bond in the same netns arrives in that
> window. mlx5_handle_changeupper_event() then runs mlx5_ldev_for_each()
> with max_funcs == 0:
>
> - mlx5_get_next_lag_func() returns 0.
> - The loop exits with i = 0 and idx still -1.
> - The new "i < ldev->ports" check passes.
>
> The handler then executes:
>
>     bond_status |= (1 << idx);
>
> This is a shift by -1. Would it be better to set max_funcs right after
> ldev->ports is assigned, before any notifier is registered?

The window is very narrow and harmless: bond_status stays 0 and the
handler returns early, so nothing is changed. It will be fixed in a
follow-up.

> [ ... ]
>
> > @@ -2094,7 +2095,7 @@ static int mlx5_handle_changeupper_event(struct mlx5_lag *ldev,
> >  				break;
> >  			}
> >  		}
> > -		if (i < MLX5_MAX_PORTS) {
> > +		if (i < ldev->ports) {
>
> [Severity: Medium]
> Should this compare against ldev->max_funcs, like the other converted call
> sites? mlx5_ldev_for_each() now ends with i == ldev->max_funcs. This patch
> also explicitly allows members at xarray indices >= ldev->ports, since
> mlx5_ldev_add_mdev() grows max_funcs for that case.
>
> xa_alloc() hands out the lowest free index to both port and SD members.
> For example, after a PF is unbound and rebound, SD secondaries added by
> sd_lag_init() can hold the low indices, so the matched port PF can end up
> at an index >= ports.
>
> In that case the loop breaks with i equal to that index, "i < ldev->ports"
> is false, and the PF is treated as not found. Its bond_status bit is never
> set and has_inactive is not computed for it, so this never becomes true:
>
>     is_in_lag = num_slaves == ldev->ports &&
>         bond_status == GENMASK(ldev->ports - 1, 0);
>
> As a result, the bond is never offloaded. Before this patch, indices were
> capped at 7, so any found entry passed the check.
>
> The notifier is only registered when !mlx5_sd_is_supported(dev), so this
> may need a mixed or unusual topology to trigger. The check is still the
> same at the end of the series.

No. mlx5 creates an ldev per sw_guid, and PFs have a different
sw_guid than VFs/SFs, so only PFs share this ldev. The only case with
more PFs than num_lag_ports in one ldev is SD, and for SD the netdev
notifier, and with it mlx5_handle_changeupper_event(), is not
registered. So a PORT-marked member here always sits at an index below
ldev->ports.

> [ ... ]
>
> > @@ -2359,13 +2360,16 @@ int mlx5_ldev_add_mdev(struct mlx5_lag *ldev,
> >  	if (!pf)
> >  		return -ENOMEM;
> >
> > -	err = xa_alloc(&ldev->pfs, &idx, pf, XA_LIMIT(0, MLX5_MAX_PORTS - 1),
> > +	err = xa_alloc(&ldev->pfs, &idx, pf, XA_LIMIT(0, U8_MAX - 1),
> >  		       GFP_KERNEL);
>
> [Severity: High]
> Is it safe to drop the MLX5_MAX_PORTS limit before the arrays that depend
> on it are resized?
>
> This limit was the only thing that kept both the member count and every
> xarray index below MLX5_MAX_PORTS (8). Neither __mlx5_lag_dev_add_mdev()
> nor sd_lag_init() checks the member count.
>
> At this commit, once an ldev has more than 8 members, these writes look
> like they go out of bounds:
>
> - mlx5_mpesw_metadata_set() writes ldev->lag_mpesw.pf_metadata[i]
>   (u32[MLX5_MAX_PORTS]) under MLX5_LAG_FILTER_ALL. lag_mpesw is the last
>   member of struct mlx5_lag, so this writes past the kzalloc'd ldev.
>
> - mlx5_lag_rescan_devices_locked_filter() fills the on-stack
>   devs[MLX5_MAX_PORTS] with devs[num_devs++] without any bound check.
>
> - mlx5_lag_query_cong_counters() fills a buffer of MLX5_MAX_PORTS entries
>   with mdev[idx++].
>
> - struct lag_tracker::netdev_state[MLX5_MAX_PORTS] is written at the
>   port's xarray index in mlx5_ldev_add_netdev() and in
>   mlx5_handle_changelowerstate_event():
>
>       tracker->netdev_state[idx] = *lag_lower_info;
>
> - struct mlx5_lag_definer::rules[MLX5_MAX_PORTS *
>   MLX5_LAG_MAX_HASH_BUCKETS] is indexed by i * ldev->buckets + j in
>   port_sel.c, where i is the port's xarray index.
>
> Before this patch, such a configuration failed cleanly: xa_alloc()
> returned -EBUSY and sd_lag_init() rolled back the secondaries. An 8-port
> SD configuration, for example, would now give 16 members.
>
> Later patches in the series fix some of these:
>
> - "net/mlx5: LAG: replace fixed MLX5_MAX_PORTS arrays with dynamic
>   allocation" makes pf_metadata kcalloc(max_fns) and sizes the
>   cong-counter buffer from ports.
> - "net/mlx5: LAG, drop MLX5_MAX_PORTS scratch array from device rescan"
>   removes devs[].
>
> At the end of the series, though, lag_tracker::netdev_state[] and
> mlx5_lag_definer::rules[] are still sized by MLX5_MAX_PORTS and indexed
> by xarray indices that are no longer bounded.

In this series a 9th member is impossible: an ldev holds only
PFs and SD secondaries, and no HW has more than MLX5_MAX_PORTS of them.
VFs/SFs join an ldev only in the follow-up series.
The limit can't be kept until all arrays are converted, since not all
of them will be. Some arrays are used only by a PF LAG, others by both a
PF LAG and a virt LAG. The shared ones, the fns xarray included, are
converted in this series (pf_metadata[] and devs[]; the cong counter
array is converted as well). The PF LAG only ones,
tracker.netdev_state[] and the port_sel rules[], keep their static
MLX5_MAX_PORTS size: they are walked only over PORT-marked members, in
the hash and queue affinity modes. A virt member is never PORT-marked
and sits in its own ldev, and a virt LAG supports only MPESW, so these
arrays never see more than MLX5_MAX_PORTS entries.

> The commit message also says:
>
>     No functional change; this only removes the dependency on
>     MLX5_MAX_PORTS.
>
> Isn't dropping the xa_alloc() bound a functional change in itself?

The xa_alloc() change is already in the commit message ("remove the
xa_alloc() limit of MLX5_MAX_PORTS"), and the answer above explains why
it can't let in a 9th member.

> >  	if (err) {
> >  		kfree(pf);
> >  		return err;
> >  	}
> >
> > +	/* max_funcs is the iteration ceiling; grow it, never shrink. */
> > +	if (idx >= ldev->max_funcs)
> > +		ldev->max_funcs = idx + 1;
> >  	pf->idx = idx;
> >  	pf->dev = dev;
>
> [Severity: Medium]
> Can walkers that don't hold ldev->lock see an inconsistent max_funcs?
>

No, all these walkers run either during LAG activation or while
the LAG is active, and membership doesn't change then.

> The end-of-walk sentinel used to be a constant. It is now this field,
> which is written with a plain store under ldev->lock only. Each iteration
> of mlx5_lag_for_each() loads it twice: once as the return value of
> mlx5_get_next_lag_func(), and again in the macro test:
>
>     i = tmp, tmp < (ldev)->max_funcs; tmp++)
>
> Several walkers don't hold ldev->lock:
>
> - mlx5_mpesw_sd_devcoms_lock(), called from mlx5_mpesw_work() and
>   mlx5_lag_disable_change(), runs under the HCA devcom lock but before
>   mutex_lock(&ldev->lock).
> - mlx5_ldev_add_netdev(), mlx5_lag_get_next_peer_mdev() and
>   mlx5_lag_query_cong_counters() hold only lag_lock.
> - mlx5_handle_changeupper_event() holds only RCU.
>
> sd_lag_init() adds secondaries while holding only ldev->lock. The writer
> order is:
>
> 1. xa_alloc() publishes a zeroed pf at idx == old max_funcs.
> 2. max_funcs = idx + 1.
> 3. pf->dev = dev.
> 4. The PORT mark is set.
>
> Suppose a reader's walk ends while the new pf isn't valid yet. It gets
> sentinel N back from mlx5_get_next_lag_func(), then re-reads max_funcs as
> N + 1. The loop body then runs once with i = N, skipping the !pf->dev
> check and the PORT-mark filter. In mlx5_mpesw_sd_devcoms_lock() that
> means:
>
>     sd_devcom = mlx5_sd_get_devcom(mlx5_lag_pf(ldev, i)->dev);
>
> runs with a NULL dev, which mlx5_get_sd() dereferences. In
> mlx5_lag_get_next_peer_mdev(), a NULL or non-port dev could be returned
> as a peer.
>
> Also, on weakly ordered architectures nothing orders this store against
> the xarray slot, so xa_load(N) could return NULL as well.
> [ ... ]
>
> > diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lag/shared_fdb.c b/drivers/net/ethernet/mellanox/mlx5/core/lag/shared_fdb.c
> > index 7c7e6c832a93d..29048b0c27977 100644
> > --- a/drivers/net/ethernet/mellanox/mlx5/core/lag/shared_fdb.c
> > +++ b/drivers/net/ethernet/mellanox/mlx5/core/lag/shared_fdb.c
> > @@ -71,7 +71,7 @@ static int mlx5_lag_create_single_fdb_filter(struct mlx5_lag *ldev, u32 filter)
> >
> >  		err = mlx5_eswitch_offloads_single_fdb_add_one(master_esw,
> >  							       slave_esw,
> > -							       ldev->ports);
> > +							       ldev->max_funcs);
>
> [Severity: Low]
> Is this change intended? This argument isn't an end-of-walk sentinel. It
> is max_slaves, which goes through esw_set_master_egress_rule() to
> esw_master_egress_create_resources(). There it becomes ft_attr.max_fte
> and the bounce group's end_flow_index for the master's egress ACL table
> in firmware.
>
> With SD, secondaries take extra xarray indices, so max_funcs can exceed
> ports (for example, 4 on a 2-port SD card) and the ACL table is created
> larger. max_funcs is also a high-water mark of xarray indices that never
> shrinks, not a slave count.
>
> Bounce rules are only added for SD primaries, so ldev->ports seems to
> have been sufficient already.
>
> This doesn't match "No functional change" in the commit message. Could
> the change be explained in the commit message, or dropped?

Only the ACL size changes, and only for SD, where more PFs than
ports join the ldev and max_funcs > ports. The rules installed are the
same as before, so there is no functional change.

> --
> Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261116103010.1129666-1-tariqt%40nvidia.com

  reply	other threads:[~2026-11-19  8:11 UTC|newest]

Thread overview: 22+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-16 10:30 [PATCH net-next V2 00/10] net/mlx5: LAG, preparations for LAG over VFs/SFs Tariq Toukan
2026-11-16 10:30 ` [PATCH net-next V2 01/10] net/mlx5: Distinguish VFs/SFs in the SW system image GUID for LAG grouping Tariq Toukan
2026-11-18 10:31   ` netdev-bot+sashiko
2026-11-19  7:40     ` Shay Drory
2026-11-16 10:30 ` [PATCH net-next V2 02/10] net/mlx5: LAG, select the MPESW master from the live member snapshot Tariq Toukan
2026-11-16 10:30 ` [PATCH net-next V2 03/10] net/mlx5: LAG, skip phys LAG prerequisites for a virt LAG Tariq Toukan
2026-11-16 10:30 ` [PATCH net-next V2 04/10] net/mlx5: LAG, treat virt LAG as a software managed LAG Tariq Toukan
2026-11-16 10:30 ` [PATCH net-next V2 05/10] net/mlx5: LAG, decouple iteration end-of-walk from MLX5_MAX_PORTS Tariq Toukan
2026-11-18 10:31   ` netdev-bot+sashiko
2026-11-19  8:10     ` Shay Drory [this message]
2026-11-16 10:30 ` [PATCH net-next V2 06/10] net/mlx5: LAG, rename PF naming to fn Tariq Toukan
2026-11-18 10:31   ` netdev-bot+sashiko
2026-11-16 10:30 ` [PATCH net-next V2 07/10] net/mlx5: LAG, resolve affinity indices only in queue-affinity mode Tariq Toukan
2026-11-16 10:30 ` [PATCH net-next V2 08/10] net/mlx5: LAG: replace fixed MLX5_MAX_PORTS arrays with dynamic allocation Tariq Toukan
2026-11-18 10:31   ` netdev-bot+sashiko
2026-11-19  8:20     ` Shay Drory
2026-11-16 10:30 ` [PATCH net-next V2 09/10] net/mlx5: LAG, drop MLX5_MAX_PORTS scratch array from device rescan Tariq Toukan
2026-11-18 10:31   ` netdev-bot+sashiko
2026-11-19  8:40     ` Shay Drory
2026-11-16 10:30 ` [PATCH net-next V2 10/10] net/mlx5: LAG, do not add a device to an already-active virt LAG Tariq Toukan
2026-11-18 10:32   ` netdev-bot+sashiko
2026-11-19  8:58     ` Shay Drory

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261119081045.1586897-1-shayd@nvidia.com \
    --to=shayd@nvidia.com \
    --cc=andrew+netdev@lunn.ch \
    --cc=cjubran@nvidia.com \
    --cc=cratiu@nvidia.com \
    --cc=davem@davemloft.net \
    --cc=edumazet@kernel.org \
    --cc=gal@nvidia.com \
    --cc=kuba@kernel.org \
    --cc=leon@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=mbloch@nvidia.com \
    --cc=moshe@nvidia.com \
    --cc=netdev-bot+sashiko@kernel.org \
    --cc=netdev@vger.kernel.org \
    --cc=ohartoov@nvidia.com \
    --cc=pabeni@redhat.com \
    --cc=phaddad@nvidia.com \
    --cc=saeedm@nvidia.com \
    --cc=tariqt@nvidia.com \
    --cc=udar@nvidia.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox