Re: [PATCH net-next 11/12] net/mlx5: HWS, Free unused action STE tables

[Michal Kubiak <michal.kubiak@intel.com>]

On Thu, Apr 10, 2025 at 08:20:25PM +0200, Vlad Dogaru wrote:
> On Thu, Apr 10, 2025 at 07:28:18PM +0200, Michal Kubiak wrote:
> > On Tue, Apr 08, 2025 at 05:00:55PM +0300, Tariq Toukan wrote:
> > > From: Vlad Dogaru <vdogaru@nvidia.com>
> > > 
> > > Periodically check for unused action STE tables and free their
> > > associated resources. In order to do this safely, add a per-queue lock
> > > to synchronize the garbage collect work with regular operations on
> > > steering rules.
> > > 
> > > Signed-off-by: Vlad Dogaru <vdogaru@nvidia.com>
> > > Reviewed-by: Yevgeny Kliteynik <kliteyn@nvidia.com>
> > > Reviewed-by: Mark Bloch <mbloch@nvidia.com>
> > > Signed-off-by: Tariq Toukan <tariqt@nvidia.com>
> > > ---
> > >  .../mlx5/core/steering/hws/action_ste_pool.c  | 88 ++++++++++++++++++-
> > >  .../mlx5/core/steering/hws/action_ste_pool.h  | 11 +++
> > >  .../mellanox/mlx5/core/steering/hws/context.h |  1 +
> > >  3 files changed, 96 insertions(+), 4 deletions(-)
> > 
> > [...]
> > 
> > > +
> > > +static void hws_action_ste_pool_cleanup(struct work_struct *work)
> > > +{
> > > +	enum mlx5hws_pool_optimize opt;
> > > +	struct mlx5hws_context *ctx;
> > > +	LIST_HEAD(cleanup);
> > > +	int i;
> > > +
> > > +	ctx = container_of(work, struct mlx5hws_context,
> > > +			   action_ste_cleanup.work);
> > > +
> > > +	for (i = 0; i < ctx->queues; i++) {
> > > +		struct mlx5hws_action_ste_pool *p = &ctx->action_ste_pool[i];
> > > +
> > > +		mutex_lock(&p->lock);
> > > +		for (opt = MLX5HWS_POOL_OPTIMIZE_NONE;
> > > +		     opt < MLX5HWS_POOL_OPTIMIZE_MAX; opt++)
> > > +			hws_action_ste_pool_element_collect_stale(
> > > +				&p->elems[opt], &cleanup);
> > > +		mutex_unlock(&p->lock);
> > > +	}
> > 
> > As I understand, in the loop above all unused items are being collected
> > to remove them at the end of the function, using `hws_action_ste_table_cleanup_list()`.
> > 
> > I noticed that only the collecting of elements is protected with the mutex.
> > So I have a question: is it possible that in a very short period of time
> > (between `mutex_unlock()` and `hws_action_ste_table_cleanup_list()` calls),
> > the cleanup list can somehow be invalidated (by changing the STE state
> > in another thread)?
> 
> An action_ste_table is either:
> (a) in an action_ste_pool (indirectly, via a pool element); or
> (b) in a cleanup list.
> 
> In situation (a), both the table's last_used timestamp and its offsets
> are protected by the parent pool's mutex. The table can only be accessed
> through its parent pool.
> 
> In situation (b), the table can only be accessed by its parent list, and
> the only thing we do with it is free it.
> 
> There is only one transition, from state (a) to state (b), never the
> other way around. This transition is done under the parent pool's mutex.
> 
> We only move tables to the cleanup list when all of their elements are
> available, so there is no risk of a `chunk_free` call accessing a table
> that's on a cleanup list: there are no chunks left to free.
> 
> I think this is airtight, but I'm happy to explore possible scenarios if
> you have any in mind.
> 
> Thank you for the detailed review,
> Vlad


Hi Vlad,

Thanks for your detailed explanation!
I was under the mistaken assumption that the `cleanup` list only had
some sort of reference to actual data. Now I see that the function
`hws_action_ste_pool_element_collect_stale` moves the real STE data
from one list to another, so only that function call should be protected
by the mutex.
Perhaps, I should have inspected the implementation of
`hws_action_ste_pool_element_collect_stale()` more closely. :-)

Thanks,
Reviewed-by: Michal Kubiak <michal.kubiak@intel.com>

PS. I saw your V2 - will take a look.