public inbox for netdev@vger.kernel.org
 help / color / mirror / Atom feed
* [PATCH] net: hsr: avoid synchronize_net() in hsr_del_port() under rtnl_mutex
@ 2026-04-17 17:53 Shardul Bankar
  2026-04-17 18:18 ` Eric Dumazet
  0 siblings, 1 reply; 3+ messages in thread
From: Shardul Bankar @ 2026-04-17 17:53 UTC (permalink / raw)
  To: davem, edumazet, kuba, pabeni, horms, netdev
  Cc: liuhangbin, lukma, acsjakub, kees, xiaoliang.yang_1, fmancera,
	linux-kernel, janak, kalpan.jani, shardulsb08, Shardul Bankar,
	syzbot+f2fbf7478a35a94c8b7c

hsr_del_port() calls netdev_rx_handler_unregister(), which calls
synchronize_net() while rtnl_mutex is held.  During netns teardown,
cleanup_net() walks pernet_list and reaches default_device_exit_batch(),
which takes rtnl_mutex and iterates devices calling
rtnl_link_ops->dellink() for each.  For HSR this resolves to
hsr_dellink() -> hsr_del_ports() -> hsr_del_port(), so the
synchronize_net() call runs under rtnl_mutex.  Under contention this
stalls other rtnl_mutex waiters long enough to trip the hung-task
detector:

  kworker cleanup_net -> default_device_exit_batch -> hsr_del_port
    -> netdev_rx_handler_unregister -> synchronize_rcu_expedited
    [slow, holds rtnl_mutex]
  syz-executor -> ksys_unshare -> setup_net -> ip_tunnel_init_net
    -> rtnl_lock  [blocked on rtnl_mutex]

Open-code netdev_rx_handler_unregister() in hsr_del_port() without
the synchronize_net() step.  This is safe because hsr_del_port()
already defers the port free via kfree_rcu(port, rcu), so the
rx_handler_data memory remains valid until an RCU grace period
elapses naturally.  hsr_handle_frame() additionally re-validates
rx_handler via hsr_port_get_rcu() before dereferencing
rx_handler_data, so a reader that observes rx_handler_data == NULL
in the brief window between the two clears takes the NULL path and
returns RX_HANDLER_PASS without touching the port.

Reported-by: syzbot+f2fbf7478a35a94c8b7c@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?id=cb64c22a492202ca929e18262fdb8cb89e635c70
Signed-off-by: Shardul Bankar <shardul.b@mpiricsoftware.com>
---
Testing status:

The hang was originally observed on 7.0-rc6 (commit 7ca6d1cfec80) with
the syzkaller reproducer and a KASAN+LOCKDEP config.  Subsequent
reproduction attempts with the same reproducer on current mainline did
not retrigger this specific signature, so I could not verify the fix
against a live reproducer.  The patch is submitted on the basis of
code analysis.

 net/hsr/hsr_slave.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/net/hsr/hsr_slave.c b/net/hsr/hsr_slave.c
index d9af9e65f72f..5e92f23fa1a5 100644
--- a/net/hsr/hsr_slave.c
+++ b/net/hsr/hsr_slave.c
@@ -239,7 +239,18 @@ void hsr_del_port(struct hsr_port *port)
 	if (port != master) {
 		netdev_update_features(master->dev);
 		dev_set_mtu(master->dev, hsr_get_max_mtu(hsr));
-		netdev_rx_handler_unregister(port->dev);
+		/* Open-code netdev_rx_handler_unregister() without the
+		 * synchronize_net() step: holding rtnl_mutex across the
+		 * grace-period wait stalls other rtnl_mutex waiters long
+		 * enough to trip the hung-task detector under load.
+		 * Skipping synchronize_net() is safe because the port is
+		 * freed via kfree_rcu() below (so rx_handler_data memory
+		 * outlives any in-flight reader), and hsr_handle_frame()
+		 * re-validates rx_handler via hsr_port_get_rcu() before
+		 * dereferencing rx_handler_data.
+		 */
+		RCU_INIT_POINTER(port->dev->rx_handler, NULL);
+		RCU_INIT_POINTER(port->dev->rx_handler_data, NULL);
 		if (!port->hsr->fwd_offloaded)
 			dev_set_promiscuity(port->dev, -1);
 		netdev_upper_dev_unlink(port->dev, master->dev);
-- 
2.34.1


^ permalink raw reply related	[flat|nested] 3+ messages in thread

* Re: [PATCH] net: hsr: avoid synchronize_net() in hsr_del_port() under rtnl_mutex
  2026-04-17 17:53 [PATCH] net: hsr: avoid synchronize_net() in hsr_del_port() under rtnl_mutex Shardul Bankar
@ 2026-04-17 18:18 ` Eric Dumazet
  2026-04-17 21:12   ` Shardul Bankar
  0 siblings, 1 reply; 3+ messages in thread
From: Eric Dumazet @ 2026-04-17 18:18 UTC (permalink / raw)
  To: Shardul Bankar
  Cc: davem, kuba, pabeni, horms, netdev, liuhangbin, lukma, acsjakub,
	kees, xiaoliang.yang_1, fmancera, linux-kernel, janak,
	kalpan.jani, Shardul Bankar, syzbot+f2fbf7478a35a94c8b7c

On Fri, Apr 17, 2026 at 10:53 AM Shardul Bankar <shardulsb08@gmail.com> wrote:
>
> hsr_del_port() calls netdev_rx_handler_unregister(), which calls
> synchronize_net() while rtnl_mutex is held.  During netns teardown,
> cleanup_net() walks pernet_list and reaches default_device_exit_batch(),
> which takes rtnl_mutex and iterates devices calling
> rtnl_link_ops->dellink() for each.  For HSR this resolves to
> hsr_dellink() -> hsr_del_ports() -> hsr_del_port(), so the
> synchronize_net() call runs under rtnl_mutex.  Under contention this
> stalls other rtnl_mutex waiters long enough to trip the hung-task
> detector:
>
>   kworker cleanup_net -> default_device_exit_batch -> hsr_del_port
>     -> netdev_rx_handler_unregister -> synchronize_rcu_expedited
>     [slow, holds rtnl_mutex]

 synchronize_rcu_expedited() should be quite fast...



>   syz-executor -> ksys_unshare -> setup_net -> ip_tunnel_init_net
>     -> rtnl_lock  [blocked on rtnl_mutex]
>
> Open-code netdev_rx_handler_unregister() in hsr_del_port() without
> the synchronize_net() step.  This is safe because hsr_del_port()
> already defers the port free via kfree_rcu(port, rcu), so the
> rx_handler_data memory remains valid until an RCU grace period
> elapses naturally.  hsr_handle_frame() additionally re-validates
> rx_handler via hsr_port_get_rcu() before dereferencing
> rx_handler_data, so a reader that observes rx_handler_data == NULL
> in the brief window between the two clears takes the NULL path and
> returns RX_HANDLER_PASS without touching the port.
>
> Reported-by: syzbot+f2fbf7478a35a94c8b7c@syzkaller.appspotmail.com
> Closes: https://syzkaller.appspot.com/bug?id=cb64c22a492202ca929e18262fdb8cb89e635c70

Signature looks like bug fixed recently in wireguard.

commit 60a25ef8dacb3566b1a8c4de00572a498e2a3bf9
Author: Shardul Bankar <shardul.b@mpiricsoftware.com>
Date:   Tue Apr 14 17:39:44 2026 +0200

    wireguard: device: use exit_rtnl callback instead of manual
rtnl_lock in pre_exit



> Signed-off-by: Shardul Bankar <shardul.b@mpiricsoftware.com>
> ---
> Testing status:
>
> The hang was originally observed on 7.0-rc6 (commit 7ca6d1cfec80) with
> the syzkaller reproducer and a KASAN+LOCKDEP config.  Subsequent
> reproduction attempts with the same reproducer on current mainline did
> not retrigger this specific signature, so I could not verify the fix
> against a live reproducer.  The patch is submitted on the basis of
> code analysis.
>
>  net/hsr/hsr_slave.c | 13 ++++++++++++-
>  1 file changed, 12 insertions(+), 1 deletion(-)
>
> diff --git a/net/hsr/hsr_slave.c b/net/hsr/hsr_slave.c
> index d9af9e65f72f..5e92f23fa1a5 100644
> --- a/net/hsr/hsr_slave.c
> +++ b/net/hsr/hsr_slave.c
> @@ -239,7 +239,18 @@ void hsr_del_port(struct hsr_port *port)
>         if (port != master) {
>                 netdev_update_features(master->dev);
>                 dev_set_mtu(master->dev, hsr_get_max_mtu(hsr));
> -               netdev_rx_handler_unregister(port->dev);
> +               /* Open-code netdev_rx_handler_unregister() without the
> +                * synchronize_net() step: holding rtnl_mutex across the
> +                * grace-period wait stalls other rtnl_mutex waiters long
> +                * enough to trip the hung-task detector under load.
> +                * Skipping synchronize_net() is safe because the port is
> +                * freed via kfree_rcu() below (so rx_handler_data memory
> +                * outlives any in-flight reader), and hsr_handle_frame()
> +                * re-validates rx_handler via hsr_port_get_rcu() before
> +                * dereferencing rx_handler_data.
> +                */
> +               RCU_INIT_POINTER(port->dev->rx_handler, NULL);
> +               RCU_INIT_POINTER(port->dev->rx_handler_data, NULL);
>                 if (!port->hsr->fwd_offloaded)
>                         dev_set_promiscuity(port->dev, -1);
>                 netdev_upper_dev_unlink(port->dev, master->dev);

You are saying that all netdev_rx_handler_unregister() uses are
potentially a problem.

This can not be true.

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [PATCH] net: hsr: avoid synchronize_net() in hsr_del_port() under rtnl_mutex
  2026-04-17 18:18 ` Eric Dumazet
@ 2026-04-17 21:12   ` Shardul Bankar
  0 siblings, 0 replies; 3+ messages in thread
From: Shardul Bankar @ 2026-04-17 21:12 UTC (permalink / raw)
  To: Eric Dumazet
  Cc: davem, kuba, pabeni, horms, netdev, liuhangbin, lukma, acsjakub,
	kees, xiaoliang.yang_1, fmancera, linux-kernel, janak,
	kalpan.jani, syzbot+f2fbf7478a35a94c8b7c

On Fri, 2026-04-17 at 11:18 -0700, Eric Dumazet wrote:
> On Fri, Apr 17, 2026 at 10:53 AM Shardul Bankar
> <shardulsb08@gmail.com> wrote:
> > 
> 
>  synchronize_rcu_expedited() should be quite fast...

Confirmed.  I ran a targeted HSR-only teardown stress on current
mainline (many HSR-bearing netns torn down concurrently, no WireGuard,
no syzkaller harness).  At N=1000, patched-vs-baseline cleanup_net
wall time is within run-to-run noise- the patch has no measurable
benefit on the workload it was meant to address.


> Signature looks like bug fixed recently in wireguard.
> 
> commit 60a25ef8dacb3566b1a8c4de00572a498e2a3bf9

Agreed.  60a25ef8dacb covers what I was actually observing, and
standalone evidence for the HSR path on current mainline is not
strong enough to justify this change.

Please drop this patch.

Thanks,
Shardul

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2026-04-17 21:12 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-04-17 17:53 [PATCH] net: hsr: avoid synchronize_net() in hsr_del_port() under rtnl_mutex Shardul Bankar
2026-04-17 18:18 ` Eric Dumazet
2026-04-17 21:12   ` Shardul Bankar

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox