[PATCH v3 net] net: watchdog: fix refcount tracking races

[PATCH v3 net] net: watchdog: fix refcount tracking races

[Eric Dumazet]

Blamed commit converted the untracked dev_hold()/dev_put() calls
in the watchdog code to use the tracked dev_hold_track()/dev_put_track()
(which were later renamed/interfaced to netdev_hold() and netdev_put()).

By introducing dev->watchdog_dev_tracker to store the
reference tracking information without adding synchronization
between netdev_watchdog_up() and dev_watchdog(), it enabled the
race condition where this pointer could be overwritten or freed
concurrently, leading to the list corruption crash syzbot reported:

list_del corruption, ffff888114a18c00->next is NULL
 kernel BUG at lib/list_debug.c:52 !
Oops: invalid opcode: 0000 [#1] SMP KASAN PTI
CPU: 1 UID: 0 PID: 91 Comm: kworker/u8:5 Not tainted syzkaller #0 PREEMPT(lazy)
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 05/09/2026
Workqueue: events_unbound linkwatch_event
 RIP: 0010:__list_del_entry_valid_or_report.cold+0x22/0x2a lib/list_debug.c:52
Call Trace:
 <TASK>
  __list_del_entry_valid include/linux/list.h:132 [inline]
  __list_del_entry include/linux/list.h:246 [inline]
  list_move_tail include/linux/list.h:341 [inline]
  ref_tracker_free+0x1a7/0x6c0 lib/ref_tracker.c:329
  netdev_tracker_free include/linux/netdevice.h:4491 [inline]
  netdev_put include/linux/netdevice.h:4508 [inline]
  netdev_put include/linux/netdevice.h:4504 [inline]
  netdev_watchdog_down net/sched/sch_generic.c:600 [inline]
  dev_deactivate_many+0x28c/0xfe0 net/sched/sch_generic.c:1363
  dev_deactivate+0x109/0x1d0 net/sched/sch_generic.c:1397
  linkwatch_do_dev net/core/link_watch.c:184 [inline]
  linkwatch_do_dev+0xd3/0x120 net/core/link_watch.c:166
  __linkwatch_run_queue+0x3a5/0x810 net/core/link_watch.c:240
  linkwatch_event+0x8f/0xc0 net/core/link_watch.c:314
  process_one_work+0xa0e/0x1980 kernel/workqueue.c:3314
  process_scheduled_works kernel/workqueue.c:3397 [inline]
  worker_thread+0x5ef/0xe50 kernel/workqueue.c:3478
  kthread+0x370/0x450 kernel/kthread.c:436
  ret_from_fork+0x69a/0xc80 arch/x86/kernel/process.c:158
  ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245

This patch has three coordinated parts:

1) Add dev->watchdog_lock and dev->watchdog_ref_held to serialize watchdog operations.

2) Remove netdev_watchdog_up() call from netif_carrier_on():
   This ensures netdev_watchdog_up() is only called from process/BH context
   (via linkwatch workqueue dev_activate()), allowing us to use
   spin_lock_bh() for synchronization.

3) Synchronize watchdog up and watchdog timer:
   Protect netdev_watchdog_up() with tx_global_lock and watchdog_lock.
   Only allocate a new tracker in netdev_watchdog_up() if one is
   not already present.
   In dev_watchdog(), ensure we don't release the tracker if the
   timer was rescheduled either by dev_watchdog() itself or concurrently
   by netdev_watchdog_up().

Fixes: f12bf6f3f942 ("net: watchdog: add net device refcount tracker")
Reported-by: syzbot+381d82bbf0253710b35d@syzkaller.appspotmail.com
Closes: https://lore.kernel.org/netdev/6a26b751.c25708ab.1b19ef.0013.GAE@google.com/T/#u
Tested-by: syzbot+3479efbc2821cb2a79f2@syzkaller.appspotmail.com
Signed-off-by: Eric Dumazet <edumazet@google.com>
---
v3: added dev->watchdog_lock and dev->watchdog_ref_held instead of messing
    with ref tracker infra. (after sashiko feedback on v2)
v2: fix compile error when CONFIG_NET_DEV_REFCNT_TRACKER is not set (Jakub and build bots)

 include/linux/netdevice.h |  4 ++++
 net/core/dev.c            |  3 ++-
 net/sched/sch_generic.c   | 44 +++++++++++++++++++++++++++++----------
 3 files changed, 39 insertions(+), 12 deletions(-)

diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 0e1e581efc5ac264259b2f0fdfe41c50a6f47239..4a0e83709f29e4bcf12f479e464e6bedecc61c69 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -1980,6 +1980,8 @@ enum netdev_reg_state {
  *	@qdisc_hash:		qdisc hash table
  *	@watchdog_timeo:	Represents the timeout that is used by
  *				the watchdog (see dev_watchdog())
+ *	@watchdog_lock:		protect watchdog_ref_held
+ *	@watchdog_ref_held:	True if the watchdog device ref is taken.
  *	@watchdog_timer:	List of timers
  *
  *	@proto_down_reason:	reason a netdev interface is held down
@@ -2392,6 +2394,8 @@ struct net_device {
 	/* These may be needed for future network-power-down code. */
 	struct timer_list	watchdog_timer;
 	int			watchdog_timeo;
+	spinlock_t		watchdog_lock;
+	bool			watchdog_ref_held;
 
 	u32                     proto_down_reason;
 
diff --git a/net/core/dev.c b/net/core/dev.c
index 0c6c270d9f7d115feb824f4ebe6be122c40d745f..731e661d7be6574d5eca4a600e0a5623be4c2485 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -11217,7 +11217,8 @@ static int netif_alloc_netdev_queues(struct net_device *dev)
 
 	netdev_for_each_tx_queue(dev, netdev_init_one_queue, NULL);
 	spin_lock_init(&dev->tx_global_lock);
-
+	spin_lock_init(&dev->watchdog_lock);
+	dev->watchdog_ref_held = false;
 	return 0;
 }
 
diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c
index a93321db8fd75d30c61e146c290bbc139c37c913..6cdf2ccfb0937e45271f8690a0c09d48a24ce769 100644
--- a/net/sched/sch_generic.c
+++ b/net/sched/sch_generic.c
@@ -568,16 +568,24 @@ static void dev_watchdog(struct timer_list *t)
 				dev->netdev_ops->ndo_tx_timeout(dev, i);
 				netif_unfreeze_queues(dev);
 			}
-			if (!mod_timer(&dev->watchdog_timer,
-				       round_jiffies(oldest_start +
-						     dev->watchdog_timeo)))
-				release = false;
+			spin_lock(&dev->watchdog_lock);
+			mod_timer(&dev->watchdog_timer,
+				  round_jiffies(oldest_start +
+						dev->watchdog_timeo));
+			release = false;
+			spin_unlock(&dev->watchdog_lock);
 		}
 	}
 	spin_unlock(&dev->tx_global_lock);
 
-	if (release)
+	spin_lock(&dev->watchdog_lock);
+	if (timer_pending(&dev->watchdog_timer))
+		release = false;
+	if (release && dev->watchdog_ref_held) {
 		netdev_put(dev, &dev->watchdog_dev_tracker);
+		dev->watchdog_ref_held = false;
+	}
+	spin_unlock(&dev->watchdog_lock);
 }
 
 void netdev_watchdog_up(struct net_device *dev)
@@ -586,18 +594,34 @@ void netdev_watchdog_up(struct net_device *dev)
 		return;
 	if (dev->watchdog_timeo <= 0)
 		dev->watchdog_timeo = 5*HZ;
+	spin_lock_bh(&dev->tx_global_lock);
+
+	spin_lock(&dev->watchdog_lock);
 	if (!mod_timer(&dev->watchdog_timer,
-		       round_jiffies(jiffies + dev->watchdog_timeo)))
-		netdev_hold(dev, &dev->watchdog_dev_tracker,
-			    GFP_ATOMIC);
+		       round_jiffies(jiffies + dev->watchdog_timeo))) {
+		if (!dev->watchdog_ref_held) {
+			netdev_hold(dev, &dev->watchdog_dev_tracker,
+				    GFP_ATOMIC);
+			dev->watchdog_ref_held = true;
+		}
+	}
+	spin_unlock(&dev->watchdog_lock);
+
+	spin_unlock_bh(&dev->tx_global_lock);
 }
 EXPORT_SYMBOL_GPL(netdev_watchdog_up);
 
 static void netdev_watchdog_down(struct net_device *dev)
 {
 	netif_tx_lock_bh(dev);
-	if (timer_delete(&dev->watchdog_timer))
+
+	spin_lock(&dev->watchdog_lock);
+	if (timer_delete(&dev->watchdog_timer)) {
 		netdev_put(dev, &dev->watchdog_dev_tracker);
+		dev->watchdog_ref_held = false;
+	}
+	spin_unlock(&dev->watchdog_lock);
+
 	netif_tx_unlock_bh(dev);
 }
 
@@ -614,8 +638,6 @@ void netif_carrier_on(struct net_device *dev)
 			return;
 		atomic_inc(&dev->carrier_up_count);
 		linkwatch_fire_event(dev);
-		if (netif_running(dev))
-			netdev_watchdog_up(dev);
 	}
 }
 EXPORT_SYMBOL(netif_carrier_on);
-- 
2.54.0.1099.g489fc7bff1-goog

Re: [PATCH v3 net] net: watchdog: fix refcount tracking races

[patchwork-bot+netdevbpf]

Hello:

This patch was applied to netdev/net.git (main)
by Jakub Kicinski <kuba@kernel.org>:

On Thu, 11 Jun 2026 15:27:37 +0000 you wrote:
> Blamed commit converted the untracked dev_hold()/dev_put() calls
> in the watchdog code to use the tracked dev_hold_track()/dev_put_track()
> (which were later renamed/interfaced to netdev_hold() and netdev_put()).
> 
> By introducing dev->watchdog_dev_tracker to store the
> reference tracking information without adding synchronization
> between netdev_watchdog_up() and dev_watchdog(), it enabled the
> race condition where this pointer could be overwritten or freed
> concurrently, leading to the list corruption crash syzbot reported:
> 
> [...]

Here is the summary with links:
  - [v3,net] net: watchdog: fix refcount tracking races
    https://git.kernel.org/netdev/net/c/8eed5519e496

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html

Re: [PATCH v3 net] net: watchdog: fix refcount tracking races

[Marek Szyprowski]

Dear All,

On 11.06.2026 17:27, Eric Dumazet wrote:
> Blamed commit converted the untracked dev_hold()/dev_put() calls
> in the watchdog code to use the tracked dev_hold_track()/dev_put_track()
> (which were later renamed/interfaced to netdev_hold() and netdev_put()).
>
> By introducing dev->watchdog_dev_tracker to store the
> reference tracking information without adding synchronization
> between netdev_watchdog_up() and dev_watchdog(), it enabled the
> race condition where this pointer could be overwritten or freed
> concurrently, leading to the list corruption crash syzbot reported:
>
> list_del corruption, ffff888114a18c00->next is NULL
>  kernel BUG at lib/list_debug.c:52 !
> Oops: invalid opcode: 0000 [#1] SMP KASAN PTI
> CPU: 1 UID: 0 PID: 91 Comm: kworker/u8:5 Not tainted syzkaller #0 PREEMPT(lazy)
> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 05/09/2026
> Workqueue: events_unbound linkwatch_event
>  RIP: 0010:__list_del_entry_valid_or_report.cold+0x22/0x2a lib/list_debug.c:52
> Call Trace:
>  <TASK>
>   __list_del_entry_valid include/linux/list.h:132 [inline]
>   __list_del_entry include/linux/list.h:246 [inline]
>   list_move_tail include/linux/list.h:341 [inline]
>   ref_tracker_free+0x1a7/0x6c0 lib/ref_tracker.c:329
>   netdev_tracker_free include/linux/netdevice.h:4491 [inline]
>   netdev_put include/linux/netdevice.h:4508 [inline]
>   netdev_put include/linux/netdevice.h:4504 [inline]
>   netdev_watchdog_down net/sched/sch_generic.c:600 [inline]
>   dev_deactivate_many+0x28c/0xfe0 net/sched/sch_generic.c:1363
>   dev_deactivate+0x109/0x1d0 net/sched/sch_generic.c:1397
>   linkwatch_do_dev net/core/link_watch.c:184 [inline]
>   linkwatch_do_dev+0xd3/0x120 net/core/link_watch.c:166
>   __linkwatch_run_queue+0x3a5/0x810 net/core/link_watch.c:240
>   linkwatch_event+0x8f/0xc0 net/core/link_watch.c:314
>   process_one_work+0xa0e/0x1980 kernel/workqueue.c:3314
>   process_scheduled_works kernel/workqueue.c:3397 [inline]
>   worker_thread+0x5ef/0xe50 kernel/workqueue.c:3478
>   kthread+0x370/0x450 kernel/kthread.c:436
>   ret_from_fork+0x69a/0xc80 arch/x86/kernel/process.c:158
>   ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
>
> This patch has three coordinated parts:
>
> 1) Add dev->watchdog_lock and dev->watchdog_ref_held to serialize watchdog operations.
>
> 2) Remove netdev_watchdog_up() call from netif_carrier_on():
>    This ensures netdev_watchdog_up() is only called from process/BH context
>    (via linkwatch workqueue dev_activate()), allowing us to use
>    spin_lock_bh() for synchronization.
>
> 3) Synchronize watchdog up and watchdog timer:
>    Protect netdev_watchdog_up() with tx_global_lock and watchdog_lock.
>    Only allocate a new tracker in netdev_watchdog_up() if one is
>    not already present.
>    In dev_watchdog(), ensure we don't release the tracker if the
>    timer was rescheduled either by dev_watchdog() itself or concurrently
>    by netdev_watchdog_up().
>
> Fixes: f12bf6f3f942 ("net: watchdog: add net device refcount tracker")
> Reported-by: syzbot+381d82bbf0253710b35d@syzkaller.appspotmail.com
> Closes: https://lore.kernel.org/netdev/6a26b751.c25708ab.1b19ef.0013.GAE@google.com/T/#u
> Tested-by: syzbot+3479efbc2821cb2a79f2@syzkaller.appspotmail.com
> Signed-off-by: Eric Dumazet <edumazet@google.com>
This patch landed recently in linux-next as commit 8eed5519e496 ("net: watchdog:
fix refcount tracking races"). In my tests I found that it causes the following
deadlock during system suspend/resume on QEmu's ARM64bit 'virt' machine:

root@target:~# time rtcwake -s10 -mmem
rtcwake: assuming RTC uses UTC ...
rtcwake: wakeup from "mem" using /dev/rtc0 at Wed Jun 17 10:46:12 2026
PM: suspend entry (s2idle)
Filesystems sync: 0.055 seconds
Freezing user space processes
Freezing user space processes completed (elapsed 0.006 seconds)
OOM killer disabled.
Freezing remaining freezable tasks
Freezing remaining freezable tasks completed (elapsed 0.003 seconds)

============================================
WARNING: possible recursive locking detected
7.1.0-rc7+ #13003 Not tainted
--------------------------------------------
rtcwake/254 is trying to acquire lock:
ffff000006de64e8 (&dev->tx_global_lock){+.-.}-{3:3}, at: netdev_watchdog_up+0x40/0x108

but task is already holding lock:
ffff000006de64e8 (&dev->tx_global_lock){+.-.}-{3:3}, at: netif_tx_lock+0x1c/0x34

other info that might help us debug this:
 Possible unsafe locking scenario:

       CPU0
       ----
  lock(&dev->tx_global_lock);
  lock(&dev->tx_global_lock);

 *** DEADLOCK ***

 May be due to missing lock nesting notation

6 locks held by rtcwake/254:
 #0: ffff0000071ab3e8 (sb_writers#5){.+.+}-{0:0}, at: vfs_write+0x1ec/0x35c
 #1: ffff00000d22c480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0xf0/0x1c4
 #2: ffff0000049162c8 (kn->active#61){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x100/0x1c4
 #3: ffffaa79533c03b0 (system_transition_mutex){+.+.}-{4:4}, at: pm_suspend+0x98/0x608
 #4: ffff000005e3a138 (&dev->mutex){....}-{4:4}, at: device_resume+0xb4/0x254
 #5: ffff000006de64e8 (&dev->tx_global_lock){+.-.}-{3:3}, at: netif_tx_lock+0x1c/0x34

stack backtrace:
CPU: 1 UID: 0 PID: 254 Comm: rtcwake Not tainted 7.1.0-rc7+ #13003 PREEMPT
Hardware name: linux,dummy-virt (DT)
Call trace:
 show_stack+0x18/0x24 (C)
 dump_stack_lvl+0x90/0xd0
 dump_stack+0x18/0x24
 print_deadlock_bug+0x260/0x350
 __lock_acquire+0x11b8/0x225c
 lock_acquire+0x1c4/0x3f0
 _raw_spin_lock_bh+0x50/0x68
 netdev_watchdog_up+0x40/0x108
 netif_device_attach+0x9c/0xb0
 virtnet_restore+0x100/0x21c
 virtio_device_restore_priv+0x11c/0x1d0
 virtio_device_restore+0x14/0x20
 virtio_mmio_restore+0x34/0x40
 platform_pm_resume+0x2c/0x68
 dpm_run_callback+0xa0/0x240
 device_resume+0x120/0x254
 dpm_resume+0x1f8/0x2ec
 dpm_resume_end+0x18/0x34
 suspend_devices_and_enter+0x1d0/0x990
 pm_suspend+0x1ec/0x608
 state_store+0x8c/0x110
 kobj_attr_store+0x18/0x2c
 sysfs_kf_write+0x50/0x7c
 kernfs_fop_write_iter+0x130/0x1c4
 vfs_write+0x2b8/0x35c
 ksys_write+0x6c/0x104
 __arm64_sys_write+0x1c/0x28
 invoke_syscall+0x54/0x110
 el0_svc_common.constprop.0+0x40/0xe8
 do_el0_svc+0x20/0x2c
 el0_svc+0x54/0x338
 el0t_64_sync_handler+0xa0/0xe4
 el0t_64_sync+0x198/0x19c


Reverting $subject on top of linux-next fixes this issue.


> ---
> v3: added dev->watchdog_lock and dev->watchdog_ref_held instead of messing
>     with ref tracker infra. (after sashiko feedback on v2)
> v2: fix compile error when CONFIG_NET_DEV_REFCNT_TRACKER is not set (Jakub and build bots)
>
>  include/linux/netdevice.h |  4 ++++
>  net/core/dev.c            |  3 ++-
>  net/sched/sch_generic.c   | 44 +++++++++++++++++++++++++++++----------
>  3 files changed, 39 insertions(+), 12 deletions(-)
>
> diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
> index 0e1e581efc5ac264259b2f0fdfe41c50a6f47239..4a0e83709f29e4bcf12f479e464e6bedecc61c69 100644
> --- a/include/linux/netdevice.h
> +++ b/include/linux/netdevice.h
> @@ -1980,6 +1980,8 @@ enum netdev_reg_state {
>   *	@qdisc_hash:		qdisc hash table
>   *	@watchdog_timeo:	Represents the timeout that is used by
>   *				the watchdog (see dev_watchdog())
> + *	@watchdog_lock:		protect watchdog_ref_held
> + *	@watchdog_ref_held:	True if the watchdog device ref is taken.
>   *	@watchdog_timer:	List of timers
>   *
>   *	@proto_down_reason:	reason a netdev interface is held down
> @@ -2392,6 +2394,8 @@ struct net_device {
>  	/* These may be needed for future network-power-down code. */
>  	struct timer_list	watchdog_timer;
>  	int			watchdog_timeo;
> +	spinlock_t		watchdog_lock;
> +	bool			watchdog_ref_held;
>  
>  	u32                     proto_down_reason;
>  
> diff --git a/net/core/dev.c b/net/core/dev.c
> index 0c6c270d9f7d115feb824f4ebe6be122c40d745f..731e661d7be6574d5eca4a600e0a5623be4c2485 100644
> --- a/net/core/dev.c
> +++ b/net/core/dev.c
> @@ -11217,7 +11217,8 @@ static int netif_alloc_netdev_queues(struct net_device *dev)
>  
>  	netdev_for_each_tx_queue(dev, netdev_init_one_queue, NULL);
>  	spin_lock_init(&dev->tx_global_lock);
> -
> +	spin_lock_init(&dev->watchdog_lock);
> +	dev->watchdog_ref_held = false;
>  	return 0;
>  }
>  
> diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c
> index a93321db8fd75d30c61e146c290bbc139c37c913..6cdf2ccfb0937e45271f8690a0c09d48a24ce769 100644
> --- a/net/sched/sch_generic.c
> +++ b/net/sched/sch_generic.c
> @@ -568,16 +568,24 @@ static void dev_watchdog(struct timer_list *t)
>  				dev->netdev_ops->ndo_tx_timeout(dev, i);
>  				netif_unfreeze_queues(dev);
>  			}
> -			if (!mod_timer(&dev->watchdog_timer,
> -				       round_jiffies(oldest_start +
> -						     dev->watchdog_timeo)))
> -				release = false;
> +			spin_lock(&dev->watchdog_lock);
> +			mod_timer(&dev->watchdog_timer,
> +				  round_jiffies(oldest_start +
> +						dev->watchdog_timeo));
> +			release = false;
> +			spin_unlock(&dev->watchdog_lock);
>  		}
>  	}
>  	spin_unlock(&dev->tx_global_lock);
>  
> -	if (release)
> +	spin_lock(&dev->watchdog_lock);
> +	if (timer_pending(&dev->watchdog_timer))
> +		release = false;
> +	if (release && dev->watchdog_ref_held) {
>  		netdev_put(dev, &dev->watchdog_dev_tracker);
> +		dev->watchdog_ref_held = false;
> +	}
> +	spin_unlock(&dev->watchdog_lock);
>  }
>  
>  void netdev_watchdog_up(struct net_device *dev)
> @@ -586,18 +594,34 @@ void netdev_watchdog_up(struct net_device *dev)
>  		return;
>  	if (dev->watchdog_timeo <= 0)
>  		dev->watchdog_timeo = 5*HZ;
> +	spin_lock_bh(&dev->tx_global_lock);
> +
> +	spin_lock(&dev->watchdog_lock);
>  	if (!mod_timer(&dev->watchdog_timer,
> -		       round_jiffies(jiffies + dev->watchdog_timeo)))
> -		netdev_hold(dev, &dev->watchdog_dev_tracker,
> -			    GFP_ATOMIC);
> +		       round_jiffies(jiffies + dev->watchdog_timeo))) {
> +		if (!dev->watchdog_ref_held) {
> +			netdev_hold(dev, &dev->watchdog_dev_tracker,
> +				    GFP_ATOMIC);
> +			dev->watchdog_ref_held = true;
> +		}
> +	}
> +	spin_unlock(&dev->watchdog_lock);
> +
> +	spin_unlock_bh(&dev->tx_global_lock);
>  }
>  EXPORT_SYMBOL_GPL(netdev_watchdog_up);
>  
>  static void netdev_watchdog_down(struct net_device *dev)
>  {
>  	netif_tx_lock_bh(dev);
> -	if (timer_delete(&dev->watchdog_timer))
> +
> +	spin_lock(&dev->watchdog_lock);
> +	if (timer_delete(&dev->watchdog_timer)) {
>  		netdev_put(dev, &dev->watchdog_dev_tracker);
> +		dev->watchdog_ref_held = false;
> +	}
> +	spin_unlock(&dev->watchdog_lock);
> +
>  	netif_tx_unlock_bh(dev);
>  }
>  
> @@ -614,8 +638,6 @@ void netif_carrier_on(struct net_device *dev)
>  			return;
>  		atomic_inc(&dev->carrier_up_count);
>  		linkwatch_fire_event(dev);
> -		if (netif_running(dev))
> -			netdev_watchdog_up(dev);
>  	}
>  }
>  EXPORT_SYMBOL(netif_carrier_on);

Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland

Re: [PATCH v3 net] net: watchdog: fix refcount tracking races

[Eric Dumazet]

On Wed, Jun 17, 2026 at 3:48 AM Marek Szyprowski
<m.szyprowski@samsung.com> wrote:
>
> Dear All,
>
> On 11.06.2026 17:27, Eric Dumazet wrote:
> > Blamed commit converted the untracked dev_hold()/dev_put() calls
> > in the watchdog code to use the tracked dev_hold_track()/dev_put_track()
> > (which were later renamed/interfaced to netdev_hold() and netdev_put()).
> >
> > By introducing dev->watchdog_dev_tracker to store the
> > reference tracking information without adding synchronization
> > between netdev_watchdog_up() and dev_watchdog(), it enabled the
> > race condition where this pointer could be overwritten or freed
> > concurrently, leading to the list corruption crash syzbot reported:
> >
> > list_del corruption, ffff888114a18c00->next is NULL
> >  kernel BUG at lib/list_debug.c:52 !
> > Oops: invalid opcode: 0000 [#1] SMP KASAN PTI
> > CPU: 1 UID: 0 PID: 91 Comm: kworker/u8:5 Not tainted syzkaller #0 PREEMPT(lazy)
> > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 05/09/2026
> > Workqueue: events_unbound linkwatch_event
> >  RIP: 0010:__list_del_entry_valid_or_report.cold+0x22/0x2a lib/list_debug.c:52
> > Call Trace:
> >  <TASK>
> >   __list_del_entry_valid include/linux/list.h:132 [inline]
> >   __list_del_entry include/linux/list.h:246 [inline]
> >   list_move_tail include/linux/list.h:341 [inline]
> >   ref_tracker_free+0x1a7/0x6c0 lib/ref_tracker.c:329
> >   netdev_tracker_free include/linux/netdevice.h:4491 [inline]
> >   netdev_put include/linux/netdevice.h:4508 [inline]
> >   netdev_put include/linux/netdevice.h:4504 [inline]
> >   netdev_watchdog_down net/sched/sch_generic.c:600 [inline]
> >   dev_deactivate_many+0x28c/0xfe0 net/sched/sch_generic.c:1363
> >   dev_deactivate+0x109/0x1d0 net/sched/sch_generic.c:1397
> >   linkwatch_do_dev net/core/link_watch.c:184 [inline]
> >   linkwatch_do_dev+0xd3/0x120 net/core/link_watch.c:166
> >   __linkwatch_run_queue+0x3a5/0x810 net/core/link_watch.c:240
> >   linkwatch_event+0x8f/0xc0 net/core/link_watch.c:314
> >   process_one_work+0xa0e/0x1980 kernel/workqueue.c:3314
> >   process_scheduled_works kernel/workqueue.c:3397 [inline]
> >   worker_thread+0x5ef/0xe50 kernel/workqueue.c:3478
> >   kthread+0x370/0x450 kernel/kthread.c:436
> >   ret_from_fork+0x69a/0xc80 arch/x86/kernel/process.c:158
> >   ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
> >
> > This patch has three coordinated parts:
> >
> > 1) Add dev->watchdog_lock and dev->watchdog_ref_held to serialize watchdog operations.
> >
> > 2) Remove netdev_watchdog_up() call from netif_carrier_on():
> >    This ensures netdev_watchdog_up() is only called from process/BH context
> >    (via linkwatch workqueue dev_activate()), allowing us to use
> >    spin_lock_bh() for synchronization.
> >
> > 3) Synchronize watchdog up and watchdog timer:
> >    Protect netdev_watchdog_up() with tx_global_lock and watchdog_lock.
> >    Only allocate a new tracker in netdev_watchdog_up() if one is
> >    not already present.
> >    In dev_watchdog(), ensure we don't release the tracker if the
> >    timer was rescheduled either by dev_watchdog() itself or concurrently
> >    by netdev_watchdog_up().
> >
> > Fixes: f12bf6f3f942 ("net: watchdog: add net device refcount tracker")
> > Reported-by: syzbot+381d82bbf0253710b35d@syzkaller.appspotmail.com
> > Closes: https://lore.kernel.org/netdev/6a26b751.c25708ab.1b19ef.0013.GAE@google.com/T/#u
> > Tested-by: syzbot+3479efbc2821cb2a79f2@syzkaller.appspotmail.com
> > Signed-off-by: Eric Dumazet <edumazet@google.com>
> This patch landed recently in linux-next as commit 8eed5519e496 ("net: watchdog:
> fix refcount tracking races"). In my tests I found that it causes the following
> deadlock during system suspend/resume on QEmu's ARM64bit 'virt' machine:
>
> root@target:~# time rtcwake -s10 -mmem
> rtcwake: assuming RTC uses UTC ...
> rtcwake: wakeup from "mem" using /dev/rtc0 at Wed Jun 17 10:46:12 2026
> PM: suspend entry (s2idle)
> Filesystems sync: 0.055 seconds
> Freezing user space processes
> Freezing user space processes completed (elapsed 0.006 seconds)
> OOM killer disabled.
> Freezing remaining freezable tasks
> Freezing remaining freezable tasks completed (elapsed 0.003 seconds)
>
> ============================================
> WARNING: possible recursive locking detected
> 7.1.0-rc7+ #13003 Not tainted
> --------------------------------------------
> rtcwake/254 is trying to acquire lock:
> ffff000006de64e8 (&dev->tx_global_lock){+.-.}-{3:3}, at: netdev_watchdog_up+0x40/0x108
>
> but task is already holding lock:
> ffff000006de64e8 (&dev->tx_global_lock){+.-.}-{3:3}, at: netif_tx_lock+0x1c/0x34
>
> other info that might help us debug this:
>  Possible unsafe locking scenario:
>
>        CPU0
>        ----
>   lock(&dev->tx_global_lock);
>   lock(&dev->tx_global_lock);
>
>  *** DEADLOCK ***
>
>  May be due to missing lock nesting notation
>
> 6 locks held by rtcwake/254:
>  #0: ffff0000071ab3e8 (sb_writers#5){.+.+}-{0:0}, at: vfs_write+0x1ec/0x35c
>  #1: ffff00000d22c480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0xf0/0x1c4
>  #2: ffff0000049162c8 (kn->active#61){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x100/0x1c4
>  #3: ffffaa79533c03b0 (system_transition_mutex){+.+.}-{4:4}, at: pm_suspend+0x98/0x608
>  #4: ffff000005e3a138 (&dev->mutex){....}-{4:4}, at: device_resume+0xb4/0x254
>  #5: ffff000006de64e8 (&dev->tx_global_lock){+.-.}-{3:3}, at: netif_tx_lock+0x1c/0x34
>
> stack backtrace:
> CPU: 1 UID: 0 PID: 254 Comm: rtcwake Not tainted 7.1.0-rc7+ #13003 PREEMPT
> Hardware name: linux,dummy-virt (DT)
> Call trace:
>  show_stack+0x18/0x24 (C)
>  dump_stack_lvl+0x90/0xd0
>  dump_stack+0x18/0x24
>  print_deadlock_bug+0x260/0x350
>  __lock_acquire+0x11b8/0x225c
>  lock_acquire+0x1c4/0x3f0
>  _raw_spin_lock_bh+0x50/0x68
>  netdev_watchdog_up+0x40/0x108
>  netif_device_attach+0x9c/0xb0
>  virtnet_restore+0x100/0x21c
>  virtio_device_restore_priv+0x11c/0x1d0
>  virtio_device_restore+0x14/0x20
>  virtio_mmio_restore+0x34/0x40
>  platform_pm_resume+0x2c/0x68
>  dpm_run_callback+0xa0/0x240
>  device_resume+0x120/0x254
>  dpm_resume+0x1f8/0x2ec
>  dpm_resume_end+0x18/0x34
>  suspend_devices_and_enter+0x1d0/0x990
>  pm_suspend+0x1ec/0x608
>  state_store+0x8c/0x110
>  kobj_attr_store+0x18/0x2c
>  sysfs_kf_write+0x50/0x7c
>  kernfs_fop_write_iter+0x130/0x1c4
>  vfs_write+0x2b8/0x35c
>  ksys_write+0x6c/0x104
>  __arm64_sys_write+0x1c/0x28
>  invoke_syscall+0x54/0x110
>  el0_svc_common.constprop.0+0x40/0xe8
>  do_el0_svc+0x20/0x2c
>  el0_svc+0x54/0x338
>  el0t_64_sync_handler+0xa0/0xe4
>  el0t_64_sync+0x198/0x19c
>
>
> Reverting $subject on top of linux-next fixes this issue.

Thanks for the report Marek!

Acquiring tx_global_lock in netdev_watchdog_up() appears unnecessary anyway
because the critical state (timer and refcount tracker) is already
protected by dev->watchdog_lock.

Could you try this patch?

diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c
index 3f1c510df850dbdbaf10d483547c7b1f3a5d5482..ef2b4bf51564173751c74fefe17e3913ed2fa056
100644
--- a/net/sched/sch_generic.c
+++ b/net/sched/sch_generic.c
@@ -594,9 +594,8 @@ void netdev_watchdog_up(struct net_device *dev)
                return;
        if (dev->watchdog_timeo <= 0)
                dev->watchdog_timeo = 5*HZ;
-       spin_lock_bh(&dev->tx_global_lock);

-       spin_lock(&dev->watchdog_lock);
+       spin_lock_bh(&dev->watchdog_lock);
        if (!mod_timer(&dev->watchdog_timer,
                       round_jiffies(jiffies + dev->watchdog_timeo))) {
                if (!dev->watchdog_ref_held) {
@@ -605,9 +604,7 @@ void netdev_watchdog_up(struct net_device *dev)
                        dev->watchdog_ref_held = true;
                }
        }
-       spin_unlock(&dev->watchdog_lock);
-
-       spin_unlock_bh(&dev->tx_global_lock);
+       spin_unlock_bh(&dev->watchdog_lock);
 }
 EXPORT_SYMBOL_GPL(netdev_watchdog_up);

Re: [PATCH v3 net] net: watchdog: fix refcount tracking races

[Marek Szyprowski]

On 22.06.2026 10:59, Eric Dumazet wrote:
> On Wed, Jun 17, 2026 at 3:48 AM Marek Szyprowski
> <m.szyprowski@samsung.com> wrote:
>> On 11.06.2026 17:27, Eric Dumazet wrote:
>>> Blamed commit converted the untracked dev_hold()/dev_put() calls
>>> in the watchdog code to use the tracked dev_hold_track()/dev_put_track()
>>> (which were later renamed/interfaced to netdev_hold() and netdev_put()).
>>>
>>> By introducing dev->watchdog_dev_tracker to store the
>>> reference tracking information without adding synchronization
>>> between netdev_watchdog_up() and dev_watchdog(), it enabled the
>>> race condition where this pointer could be overwritten or freed
>>> concurrently, leading to the list corruption crash syzbot reported:
>>>
>>> list_del corruption, ffff888114a18c00->next is NULL
>>>  kernel BUG at lib/list_debug.c:52 !
>>> Oops: invalid opcode: 0000 [#1] SMP KASAN PTI
>>> CPU: 1 UID: 0 PID: 91 Comm: kworker/u8:5 Not tainted syzkaller #0 PREEMPT(lazy)
>>> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 05/09/2026
>>> Workqueue: events_unbound linkwatch_event
>>>  RIP: 0010:__list_del_entry_valid_or_report.cold+0x22/0x2a lib/list_debug.c:52
>>> Call Trace:
>>>  <TASK>
>>>   __list_del_entry_valid include/linux/list.h:132 [inline]
>>>   __list_del_entry include/linux/list.h:246 [inline]
>>>   list_move_tail include/linux/list.h:341 [inline]
>>>   ref_tracker_free+0x1a7/0x6c0 lib/ref_tracker.c:329
>>>   netdev_tracker_free include/linux/netdevice.h:4491 [inline]
>>>   netdev_put include/linux/netdevice.h:4508 [inline]
>>>   netdev_put include/linux/netdevice.h:4504 [inline]
>>>   netdev_watchdog_down net/sched/sch_generic.c:600 [inline]
>>>   dev_deactivate_many+0x28c/0xfe0 net/sched/sch_generic.c:1363
>>>   dev_deactivate+0x109/0x1d0 net/sched/sch_generic.c:1397
>>>   linkwatch_do_dev net/core/link_watch.c:184 [inline]
>>>   linkwatch_do_dev+0xd3/0x120 net/core/link_watch.c:166
>>>   __linkwatch_run_queue+0x3a5/0x810 net/core/link_watch.c:240
>>>   linkwatch_event+0x8f/0xc0 net/core/link_watch.c:314
>>>   process_one_work+0xa0e/0x1980 kernel/workqueue.c:3314
>>>   process_scheduled_works kernel/workqueue.c:3397 [inline]
>>>   worker_thread+0x5ef/0xe50 kernel/workqueue.c:3478
>>>   kthread+0x370/0x450 kernel/kthread.c:436
>>>   ret_from_fork+0x69a/0xc80 arch/x86/kernel/process.c:158
>>>   ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
>>>
>>> This patch has three coordinated parts:
>>>
>>> 1) Add dev->watchdog_lock and dev->watchdog_ref_held to serialize watchdog operations.
>>>
>>> 2) Remove netdev_watchdog_up() call from netif_carrier_on():
>>>    This ensures netdev_watchdog_up() is only called from process/BH context
>>>    (via linkwatch workqueue dev_activate()), allowing us to use
>>>    spin_lock_bh() for synchronization.
>>>
>>> 3) Synchronize watchdog up and watchdog timer:
>>>    Protect netdev_watchdog_up() with tx_global_lock and watchdog_lock.
>>>    Only allocate a new tracker in netdev_watchdog_up() if one is
>>>    not already present.
>>>    In dev_watchdog(), ensure we don't release the tracker if the
>>>    timer was rescheduled either by dev_watchdog() itself or concurrently
>>>    by netdev_watchdog_up().
>>>
>>> Fixes: f12bf6f3f942 ("net: watchdog: add net device refcount tracker")
>>> Reported-by: syzbot+381d82bbf0253710b35d@syzkaller.appspotmail.com
>>> Closes: https://lore.kernel.org/netdev/6a26b751.c25708ab.1b19ef.0013.GAE@google.com/T/#u
>>> Tested-by: syzbot+3479efbc2821cb2a79f2@syzkaller.appspotmail.com
>>> Signed-off-by: Eric Dumazet <edumazet@google.com>
>> This patch landed recently in linux-next as commit 8eed5519e496 ("net: watchdog:
>> fix refcount tracking races"). In my tests I found that it causes the following
>> deadlock during system suspend/resume on QEmu's ARM64bit 'virt' machine:
>>
>> root@target:~# time rtcwake -s10 -mmem
>> rtcwake: assuming RTC uses UTC ...
>> rtcwake: wakeup from "mem" using /dev/rtc0 at Wed Jun 17 10:46:12 2026
>> PM: suspend entry (s2idle)
>> Filesystems sync: 0.055 seconds
>> Freezing user space processes
>> Freezing user space processes completed (elapsed 0.006 seconds)
>> OOM killer disabled.
>> Freezing remaining freezable tasks
>> Freezing remaining freezable tasks completed (elapsed 0.003 seconds)
>>
>> ============================================
>> WARNING: possible recursive locking detected
>> 7.1.0-rc7+ #13003 Not tainted
>> --------------------------------------------
>> rtcwake/254 is trying to acquire lock:
>> ffff000006de64e8 (&dev->tx_global_lock){+.-.}-{3:3}, at: netdev_watchdog_up+0x40/0x108
>>
>> but task is already holding lock:
>> ffff000006de64e8 (&dev->tx_global_lock){+.-.}-{3:3}, at: netif_tx_lock+0x1c/0x34
>>
>> other info that might help us debug this:
>>  Possible unsafe locking scenario:
>>
>>        CPU0
>>        ----
>>   lock(&dev->tx_global_lock);
>>   lock(&dev->tx_global_lock);
>>
>>  *** DEADLOCK ***
>>
>>  May be due to missing lock nesting notation
>>
>> 6 locks held by rtcwake/254:
>>  #0: ffff0000071ab3e8 (sb_writers#5){.+.+}-{0:0}, at: vfs_write+0x1ec/0x35c
>>  #1: ffff00000d22c480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0xf0/0x1c4
>>  #2: ffff0000049162c8 (kn->active#61){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x100/0x1c4
>>  #3: ffffaa79533c03b0 (system_transition_mutex){+.+.}-{4:4}, at: pm_suspend+0x98/0x608
>>  #4: ffff000005e3a138 (&dev->mutex){....}-{4:4}, at: device_resume+0xb4/0x254
>>  #5: ffff000006de64e8 (&dev->tx_global_lock){+.-.}-{3:3}, at: netif_tx_lock+0x1c/0x34
>>
>> stack backtrace:
>> CPU: 1 UID: 0 PID: 254 Comm: rtcwake Not tainted 7.1.0-rc7+ #13003 PREEMPT
>> Hardware name: linux,dummy-virt (DT)
>> Call trace:
>>  show_stack+0x18/0x24 (C)
>>  dump_stack_lvl+0x90/0xd0
>>  dump_stack+0x18/0x24
>>  print_deadlock_bug+0x260/0x350
>>  __lock_acquire+0x11b8/0x225c
>>  lock_acquire+0x1c4/0x3f0
>>  _raw_spin_lock_bh+0x50/0x68
>>  netdev_watchdog_up+0x40/0x108
>>  netif_device_attach+0x9c/0xb0
>>  virtnet_restore+0x100/0x21c
>>  virtio_device_restore_priv+0x11c/0x1d0
>>  virtio_device_restore+0x14/0x20
>>  virtio_mmio_restore+0x34/0x40
>>  platform_pm_resume+0x2c/0x68
>>  dpm_run_callback+0xa0/0x240
>>  device_resume+0x120/0x254
>>  dpm_resume+0x1f8/0x2ec
>>  dpm_resume_end+0x18/0x34
>>  suspend_devices_and_enter+0x1d0/0x990
>>  pm_suspend+0x1ec/0x608
>>  state_store+0x8c/0x110
>>  kobj_attr_store+0x18/0x2c
>>  sysfs_kf_write+0x50/0x7c
>>  kernfs_fop_write_iter+0x130/0x1c4
>>  vfs_write+0x2b8/0x35c
>>  ksys_write+0x6c/0x104
>>  __arm64_sys_write+0x1c/0x28
>>  invoke_syscall+0x54/0x110
>>  el0_svc_common.constprop.0+0x40/0xe8
>>  do_el0_svc+0x20/0x2c
>>  el0_svc+0x54/0x338
>>  el0t_64_sync_handler+0xa0/0xe4
>>  el0t_64_sync+0x198/0x19c
>>
>>
>> Reverting $subject on top of linux-next fixes this issue.
> Thanks for the report Marek!
>
> Acquiring tx_global_lock in netdev_watchdog_up() appears unnecessary anyway
> because the critical state (timer and refcount tracker) is already
> protected by dev->watchdog_lock.
>
> Could you try this patch?

This fixes the observed issue. Thanks! Feel free to add:

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>


> diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c
> index 3f1c510df850dbdbaf10d483547c7b1f3a5d5482..ef2b4bf51564173751c74fefe17e3913ed2fa056
> 100644
> --- a/net/sched/sch_generic.c
> +++ b/net/sched/sch_generic.c
> @@ -594,9 +594,8 @@ void netdev_watchdog_up(struct net_device *dev)
>                 return;
>         if (dev->watchdog_timeo <= 0)
>                 dev->watchdog_timeo = 5*HZ;
> -       spin_lock_bh(&dev->tx_global_lock);
>
> -       spin_lock(&dev->watchdog_lock);
> +       spin_lock_bh(&dev->watchdog_lock);
>         if (!mod_timer(&dev->watchdog_timer,
>                        round_jiffies(jiffies + dev->watchdog_timeo))) {
>                 if (!dev->watchdog_ref_held) {
> @@ -605,9 +604,7 @@ void netdev_watchdog_up(struct net_device *dev)
>                         dev->watchdog_ref_held = true;
>                 }
>         }
> -       spin_unlock(&dev->watchdog_lock);
> -
> -       spin_unlock_bh(&dev->tx_global_lock);
> +       spin_unlock_bh(&dev->watchdog_lock);
>  }
>  EXPORT_SYMBOL_GPL(netdev_watchdog_up);
>
Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland