[PATCH net] mlxsw: spectrum: Fix incorrect parsing depth after reload

[PATCH net] mlxsw: spectrum: Fix incorrect parsing depth after reload

[Petr Machata]

From: Ido Schimmel <idosch@nvidia.com>

Spectrum ASICs have a configurable limit on how deep into the packet
they parse. By default, the limit is 96 bytes.

There are several cases where this parsing depth is not enough and there
is a need to increase it. For example, timestamping of PTP packets and a
FIB multipath hash policy that requires hashing on inner fields. The
driver therefore maintains a reference count that reflects the number of
consumers that require an increased parsing depth.

During reload_down() the parsing depth reference count does not
necessarily drop to zero, but the parsing depth itself is restored to
the default during reload_up() when the firmware is reset. It is
therefore possible to end up in situations where the driver thinks that
the parsing depth was increased (reference count is non-zero), when it
is not.

Fix by resetting the parsing depth reference count during
initialization, as is already done for the rest of the parsing depth
related fields.

Fixes: 2d91f0803b84 ("mlxsw: spectrum: Add infrastructure for parsing configuration")
Reported-by: Maksym Yaremchuk <maksymy@nvidia.com>
Signed-off-by: Ido Schimmel <idosch@nvidia.com>
Tested-by: Maksym Yaremchuk <maksymy@nvidia.com>
Signed-off-by: Petr Machata <petrm@nvidia.com>
---
 drivers/net/ethernet/mellanox/mlxsw/spectrum.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
index a8f94b7544ee..642229b1f47e 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
@@ -2937,6 +2937,7 @@ static int mlxsw_sp_netdevice_event(struct notifier_block *unused,
 
 static void mlxsw_sp_parsing_init(struct mlxsw_sp *mlxsw_sp)
 {
+	refcount_set(&mlxsw_sp->parsing.parsing_depth_ref, 0);
 	mlxsw_sp->parsing.parsing_depth = MLXSW_SP_DEFAULT_PARSING_DEPTH;
 	mlxsw_sp->parsing.vxlan_udp_dport = MLXSW_SP_DEFAULT_VXLAN_UDP_DPORT;
 	mutex_init(&mlxsw_sp->parsing.lock);
-- 
2.39.0

Re: [PATCH net] mlxsw: spectrum: Fix incorrect parsing depth after reload

[Jakub Kicinski]

On Thu, 9 Feb 2023 12:40:24 +0100 Petr Machata wrote:
> Spectrum ASICs have a configurable limit on how deep into the packet
> they parse. By default, the limit is 96 bytes.
> 
> There are several cases where this parsing depth is not enough and there
> is a need to increase it. For example, timestamping of PTP packets and a
> FIB multipath hash policy that requires hashing on inner fields. The
> driver therefore maintains a reference count that reflects the number of
> consumers that require an increased parsing depth.
> 
> During reload_down() the parsing depth reference count does not
> necessarily drop to zero, but the parsing depth itself is restored to
> the default during reload_up() when the firmware is reset. It is
> therefore possible to end up in situations where the driver thinks that
> the parsing depth was increased (reference count is non-zero), when it
> is not.

Sounds quite odd TBH, something doesn't get de-registered during _down()
but is registered again during _up()?

Re: [PATCH net] mlxsw: spectrum: Fix incorrect parsing depth after reload

[Ido Schimmel]

On Fri, Feb 10, 2023 at 07:33:50PM -0800, Jakub Kicinski wrote:
> On Thu, 9 Feb 2023 12:40:24 +0100 Petr Machata wrote:
> > Spectrum ASICs have a configurable limit on how deep into the packet
> > they parse. By default, the limit is 96 bytes.
> > 
> > There are several cases where this parsing depth is not enough and there
> > is a need to increase it. For example, timestamping of PTP packets and a
> > FIB multipath hash policy that requires hashing on inner fields. The
> > driver therefore maintains a reference count that reflects the number of
> > consumers that require an increased parsing depth.
> > 
> > During reload_down() the parsing depth reference count does not
> > necessarily drop to zero, but the parsing depth itself is restored to
> > the default during reload_up() when the firmware is reset. It is
> > therefore possible to end up in situations where the driver thinks that
> > the parsing depth was increased (reference count is non-zero), when it
> > is not.
> 
> Sounds quite odd TBH, something doesn't get de-registered during _down()
> but is registered again during _up()?

It's not really de-registered / registered. The FIB multipath hash
policy isn't changed when devlink reload is issued, so the driver
doesn't bother decrementing the parsing depth reference count. The diff
below does decrement the reference count on reload_down(). Tested it
without the current fix and it seems to work. If you prefer, I can send
a v2 with this diff squashed into the current fix.

diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
index c862a8b977c4..876e47dcd398 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
@@ -2945,6 +2945,7 @@ static void mlxsw_sp_parsing_init(struct mlxsw_sp *mlxsw_sp)
 static void mlxsw_sp_parsing_fini(struct mlxsw_sp *mlxsw_sp)
 {
 	mutex_destroy(&mlxsw_sp->parsing.lock);
+	WARN_ON_ONCE(refcount_read(&mlxsw_sp->parsing.parsing_depth_ref));
 }
 
 struct mlxsw_sp_ipv6_addr_node {
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c
index 8f3d2d2b7595..0b32292548c0 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c
@@ -10521,6 +10521,14 @@ static int mlxsw_sp_mp_hash_init(struct mlxsw_sp *mlxsw_sp)
 }
 #endif
 
+static void mlxsw_sp_mp_hash_fini(struct mlxsw_sp *mlxsw_sp)
+{
+	bool old_inc_parsing_depth = mlxsw_sp->router->inc_parsing_depth;
+
+	mlxsw_sp_mp_hash_parsing_depth_adjust(mlxsw_sp, old_inc_parsing_depth,
+					      false);
+}
+
 static int mlxsw_sp_dscp_init(struct mlxsw_sp *mlxsw_sp)
 {
 	char rdpm_pl[MLXSW_REG_RDPM_LEN];
@@ -10764,6 +10772,7 @@ int mlxsw_sp_router_init(struct mlxsw_sp *mlxsw_sp,
 err_register_inetaddr_notifier:
 	mlxsw_core_flush_owq();
 err_dscp_init:
+	mlxsw_sp_mp_hash_fini(mlxsw_sp);
 err_mp_hash_init:
 	mlxsw_sp_neigh_fini(mlxsw_sp);
 err_neigh_init:
@@ -10807,6 +10816,7 @@ void mlxsw_sp_router_fini(struct mlxsw_sp *mlxsw_sp)
 	unregister_inet6addr_notifier(&router->inet6addr_nb);
 	unregister_inetaddr_notifier(&router->inetaddr_nb);
 	mlxsw_core_flush_owq();
+	mlxsw_sp_mp_hash_fini(mlxsw_sp);
 	mlxsw_sp_neigh_fini(mlxsw_sp);
 	mlxsw_sp_lb_rif_fini(mlxsw_sp);
 	mlxsw_sp_vrs_fini(mlxsw_sp);

Re: [PATCH net] mlxsw: spectrum: Fix incorrect parsing depth after reload

[Jakub Kicinski]

On Sat, 11 Feb 2023 18:21:45 +0200 Ido Schimmel wrote:
> > Sounds quite odd TBH, something doesn't get de-registered during _down()
> > but is registered again during _up()?  
> 
> It's not really de-registered / registered. The FIB multipath hash
> policy isn't changed when devlink reload is issued, so the driver
> doesn't bother decrementing the parsing depth reference count. The diff
> below does decrement the reference count on reload_down(). Tested it
> without the current fix and it seems to work. If you prefer, I can send
> a v2 with this diff squashed into the current fix.

That does seem cleaner to me, less error prone in case some actual clean
up is missed later. So if you don't mind - yes, I'd prefer the patch
from your reply, thanks!