[BUG] AB-BA deadlock between net and led-trigger module

[BUG] AB-BA deadlock between net and led-trigger module

[Shiji Yang]

The OpenWrt community reports that sometimes devices fail to start[1]
on 5.15 kernel. After further tracking, this is caused by a AB-BA
deadlock which can be reproduced in at least 5.15, 6.6, 6.12 and latest
6.18 LTS kenrel.

Stack tracing on 6.12 kernel:
```
Task1 "netifd" is used to start/restart the network:

[ 1361.967916] task:netifd          state:D stack:0     pid:4743  tgid:4743  ppid:1      flags:0x08100000
[ 1361.977269] Stack : 00000001 00000001 00000006 800bf464 817a0cb0 800b67ac 00000001 83261b20
[ 1361.985668]         83261a54 00000000 83261aac 000007ef 00000000 80c04d74 00000000 00000002
[ 1361.994138]         83aba760 80ce0000 83261af8 00000002 83261af8 80cd0000 00000002 80d8058c
[ 1362.002582]         80cc0000 809bae70 00000002 80d80000 80cd0000 80d80568 00000001 00000000
[ 1362.011033]         809baecc 83261af8 00000002 80d80560 80d80568 809bb11c 809c09c8 806614d0
[ 1362.019484]         ...
[ 1362.021942] Call Trace:
[ 1362.024380] [<809ba6fc>] __schedule+0x504/0xc28
[ 1362.028974] [<809bae70>] schedule+0x50/0x190
[ 1362.033251] [<809bb11c>] schedule_preempt_disabled+0x1c/0x34
[ 1362.038958] [<809c0b94>] rwsem_down_write_slowpath+0x240/0x7f8
[ 1362.044789] [<809c11c0>] down_write+0x74/0x90
[ 1362.049207] [<8054e4b8>] led_trigger_register+0x5c/0x1fc		<-- Trying to get lock "triggers_list_lock" via down_write(&triggers_list_lock);
[ 1362.054536] [<80662830>] phy_led_triggers_register+0xd0/0x234
[ 1362.060329] [<8065e200>] phy_attach_direct+0x33c/0x40c
[ 1362.065489] [<80651fc4>] phylink_fwnode_phy_connect+0x15c/0x23c
[ 1362.071480] [<8066ee18>] mtk_open+0x7c/0xba0
[ 1362.075849] [<806d714c>] __dev_open+0x280/0x2b0
[ 1362.080384] [<806d7668>] __dev_change_flags+0x244/0x24c
[ 1362.085598] [<806d7698>] dev_change_flags+0x28/0x78
[ 1362.090528] [<807150e4>] dev_ioctl+0x4c0/0x654			<-- Hold lock "rtnl_mutex" by calling rtnl_lock();
[ 1362.094985] [<80694360>] sock_ioctl+0x2f4/0x4e0
[ 1362.099567] [<802e9c4c>] sys_ioctl+0x32c/0xd8c
[ 1362.104022] [<80014504>] syscall_common+0x34/0x58


Task2 "led" is used to set the led-trigger "netdev" for a GPIO LED:

[ 1362.110308] task:led             state:D stack:0     pid:4943  tgid:4943  ppid:1      flags:0x08100002
[ 1362.119656] Stack : 809bf818 80ce3ce4 80d47fa8 fffff000 00000000 809bf840 00000001 800bb264
[ 1362.128115]         00000000 00000000 80ce0000 80ce0000 00000000 80c04d7c 00000000 00000002
[ 1362.136565]         83b50d20 80ce0000 834e1cec 80d8c750 00000002 80cd7380 80da5f2c 00000000
[ 1362.144962]         00000000 809bae70 00000000 00000000 80d8c750 80d8c750 00000001 00000000
[ 1362.153412]         809baecc 00000002 80cd7380 80d8c74c 00000000 809bb11c 00000000 834e1cec
[ 1362.161868]         ...
[ 1362.164327] Call Trace:
[ 1362.166835] [<809ba6fc>] __schedule+0x504/0xc28
[ 1362.171385] [<809bae70>] schedule+0x50/0x190
[ 1362.175651] [<809bb11c>] schedule_preempt_disabled+0x1c/0x34
[ 1362.181361] [<809bdd48>] __mutex_lock+0x310/0x940
[ 1362.186132] [<809be394>] mutex_lock_nested+0x1c/0x28
[ 1362.191101] [<806c2640>] register_netdevice_notifier+0x60/0x168	<-- Trying to get lock "rtnl_mutex" via rtnl_lock();
[ 1362.197073] [<805504ac>] netdev_trig_activate+0x194/0x1e4
[ 1362.202490] [<8054e28c>] led_trigger_set+0x1d4/0x360			<-- Hold lock "triggers_list_lock" by down_read(&triggers_list_lock);
[ 1362.207511] [<8054eb38>] led_trigger_write+0xd8/0x14c
[ 1362.212566] [<80381d98>] sysfs_kf_bin_write+0x80/0xbc
[ 1362.217688] [<8037fcd8>] kernfs_fop_write_iter+0x17c/0x28c
[ 1362.223174] [<802cbd70>] vfs_write+0x21c/0x3c4
[ 1362.227712] [<802cc0c4>] ksys_write+0x78/0x12c
[ 1362.232164] [<80014504>] syscall_common+0x34/0x58
```

When the above two tasks are created at the same time, there is a
probability that it will cause the network and LED to fail to
initialize.

[1] https://github.com/openwrt/openwrt/issues/18472

Regards,
Shiji Yang

Re: [BUG] AB-BA deadlock between net and led-trigger module

[Andrew Lunn]

On Sat, Feb 21, 2026 at 06:01:46PM +0800, Shiji Yang wrote:
> The OpenWrt community reports that sometimes devices fail to start[1]
> on 5.15 kernel. After further tracking, this is caused by a AB-BA
> deadlock which can be reproduced in at least 5.15, 6.6, 6.12 and latest
> 6.18 LTS kenrel.

Thanks for the bug report. I will take a look at this.

I was initially assuming this was the netdev trigger causing the
deadlock. However:
 
> Stack tracing on 6.12 kernel:
> ```
> Task1 "netifd" is used to start/restart the network:
> 
> [ 1361.967916] task:netifd          state:D stack:0     pid:4743  tgid:4743  ppid:1      flags:0x08100000
> [ 1361.977269] Stack : 00000001 00000001 00000006 800bf464 817a0cb0 800b67ac 00000001 83261b20
> [ 1361.985668]         83261a54 00000000 83261aac 000007ef 00000000 80c04d74 00000000 00000002
> [ 1361.994138]         83aba760 80ce0000 83261af8 00000002 83261af8 80cd0000 00000002 80d8058c
> [ 1362.002582]         80cc0000 809bae70 00000002 80d80000 80cd0000 80d80568 00000001 00000000
> [ 1362.011033]         809baecc 83261af8 00000002 80d80560 80d80568 809bb11c 809c09c8 806614d0
> [ 1362.019484]         ...
> [ 1362.021942] Call Trace:
> [ 1362.024380] [<809ba6fc>] __schedule+0x504/0xc28
> [ 1362.028974] [<809bae70>] schedule+0x50/0x190
> [ 1362.033251] [<809bb11c>] schedule_preempt_disabled+0x1c/0x34
> [ 1362.038958] [<809c0b94>] rwsem_down_write_slowpath+0x240/0x7f8
> [ 1362.044789] [<809c11c0>] down_write+0x74/0x90
> [ 1362.049207] [<8054e4b8>] led_trigger_register+0x5c/0x1fc		<-- Trying to get lock "triggers_list_lock" via down_write(&triggers_list_lock);
> [ 1362.054536] [<80662830>] phy_led_triggers_register+0xd0/0x234

This is the code in drivers/net/phy/phy_led_triggers.c.

> Task2 "led" is used to set the led-trigger "netdev" for a GPIO LED:

If you are using the led-trigger netdev, do you even need
phy_led_triggers.c? As a workaround, could you disable
CONFIG_LED_TRIGGER_PHY?

I'm not sure familiar with phy_led_triggers.c, so i don't have a quick
real fix.

     Andrew

Re: [BUG] AB-BA deadlock between net and led-trigger module

[Andrew Lunn]

On Sat, Feb 21, 2026 at 06:01:46PM +0800, Shiji Yang wrote:
> The OpenWrt community reports that sometimes devices fail to start[1]
> on 5.15 kernel. After further tracking, this is caused by a AB-BA
> deadlock which can be reproduced in at least 5.15, 6.6, 6.12 and latest
> 6.18 LTS kenrel.

Hi Shiji

Please could you test this patch. It is based on the net tree, but
with a bit of fuzz will probably apply to older trees.

Thanks
	Andrew

From bf4d66187585a1893d558cb9357f1ef63437b898 Mon Sep 17 00:00:00 2001
From: Andrew Lunn <andrew@lunn.ch>
Date: Sat, 21 Feb 2026 14:51:54 -0600
Subject: [PATCH net] net: phy: register phy led_triggers during probe to avoid
 AB-BA deadlock

There is an AB-BA deadlock when both LEDS_TRIGGER_NETDEV and
LED_TRIGGER_PHY are enabled:

[ 1362.049207] [<8054e4b8>] led_trigger_register+0x5c/0x1fc             <-- Trying to get lock "triggers_list_lock" via down_write(&triggers_list_lock);
[ 1362.054536] [<80662830>] phy_led_triggers_register+0xd0/0x234
[ 1362.060329] [<8065e200>] phy_attach_direct+0x33c/0x40c
[ 1362.065489] [<80651fc4>] phylink_fwnode_phy_connect+0x15c/0x23c
[ 1362.071480] [<8066ee18>] mtk_open+0x7c/0xba0
[ 1362.075849] [<806d714c>] __dev_open+0x280/0x2b0
[ 1362.080384] [<806d7668>] __dev_change_flags+0x244/0x24c
[ 1362.085598] [<806d7698>] dev_change_flags+0x28/0x78
[ 1362.090528] [<807150e4>] dev_ioctl+0x4c0/0x654                       <-- Hold lock "rtnl_mutex" by calling rtnl_lock();
[ 1362.094985] [<80694360>] sock_ioctl+0x2f4/0x4e0
[ 1362.099567] [<802e9c4c>] sys_ioctl+0x32c/0xd8c
[ 1362.104022] [<80014504>] syscall_common+0x34/0x58

Here LED_TRIGGER_PHY is registering LED triggers during phy_attach
while holding RTNL and then taking triggers_list_lock.

[ 1362.191101] [<806c2640>] register_netdevice_notifier+0x60/0x168      <-- Trying to get lock "rtnl_mutex" via rtnl_lock();
[ 1362.197073] [<805504ac>] netdev_trig_activate+0x194/0x1e4
[ 1362.202490] [<8054e28c>] led_trigger_set+0x1d4/0x360                 <-- Hold lock "triggers_list_lock" by down_read(&triggers_list_lock);
[ 1362.207511] [<8054eb38>] led_trigger_write+0xd8/0x14c
[ 1362.212566] [<80381d98>] sysfs_kf_bin_write+0x80/0xbc
[ 1362.217688] [<8037fcd8>] kernfs_fop_write_iter+0x17c/0x28c
[ 1362.223174] [<802cbd70>] vfs_write+0x21c/0x3c4
[ 1362.227712] [<802cc0c4>] ksys_write+0x78/0x12c
[ 1362.232164] [<80014504>] syscall_common+0x34/0x58

Here LEDS_TRIGGER_NETDEV is being enabled on an LED. It first takes
triggers_list_lock and then RTNL. A classical AB-BA deadlock.

phy_led_triggers_registers() does not require the RTNL, it does not
make any calls into the network stack which require protection. There
is also no requirement the PHY has been attached to a MAC, the
triggers only make use of phydev state. This allows the call to
phy_led_triggers_registers() to be placed elsewhere. PHY probe() and
release() don't hold RTNL, so solving the AB-BA deadlock.

Reported-by: Shiji Yang <yangshiji66@outlook.com>
Closes: https://lore.kernel.org/all/OS7PR01MB13602B128BA1AD3FA38B6D1FFBC69A@OS7PR01MB13602.jpnprd01.prod.outlook.com/
Fixes: 06f502f57d0d ("leds: trigger: Introduce a NETDEV trigger")
Signed-off-by: Andrew Lunn <andrew@lunn.ch>
---
 drivers/net/phy/phy_device.c | 25 +++++++++++++++++--------
 1 file changed, 17 insertions(+), 8 deletions(-)

diff --git a/drivers/net/phy/phy_device.c b/drivers/net/phy/phy_device.c
index 9b8eaac63b90..cbb4af604aa5 100644
--- a/drivers/net/phy/phy_device.c
+++ b/drivers/net/phy/phy_device.c
@@ -1866,8 +1866,6 @@ int phy_attach_direct(struct net_device *dev, struct phy_device *phydev,
 		goto error;
 
 	phy_resume(phydev);
-	if (!phydev->is_on_sfp_module)
-		phy_led_triggers_register(phydev);
 
 	/**
 	 * If the external phy used by current mac interface is managed by
@@ -1982,9 +1980,6 @@ void phy_detach(struct phy_device *phydev)
 	phydev->phy_link_change = NULL;
 	phydev->phylink = NULL;
 
-	if (!phydev->is_on_sfp_module)
-		phy_led_triggers_unregister(phydev);
-
 	if (phydev->mdio.dev.driver)
 		module_put(phydev->mdio.dev.driver->owner);
 
@@ -3778,16 +3773,27 @@ static int phy_probe(struct device *dev)
 	/* Set the state to READY by default */
 	phydev->state = PHY_READY;
 
+	/* Register the PHY LED triggers */
+	if (!phydev->is_on_sfp_module)
+		phy_led_triggers_register(phydev);
+
 	/* Get the LEDs from the device tree, and instantiate standard
 	 * LEDs for them.
 	 */
-	if (IS_ENABLED(CONFIG_PHYLIB_LEDS) && !phy_driver_is_genphy(phydev))
+	if (IS_ENABLED(CONFIG_PHYLIB_LEDS) && !phy_driver_is_genphy(phydev)) {
 		err = of_phy_leds(phydev);
+		if (err)
+			goto out;
+	}
+
+	return 0;
 
 out:
+	if (!phydev->is_on_sfp_module)
+		phy_led_triggers_unregister(phydev);
+
 	/* Re-assert the reset signal on error */
-	if (err)
-		phy_device_reset(phydev, 1);
+	phy_device_reset(phydev, 1);
 
 	return err;
 }
@@ -3801,6 +3807,9 @@ static int phy_remove(struct device *dev)
 	if (IS_ENABLED(CONFIG_PHYLIB_LEDS) && !phy_driver_is_genphy(phydev))
 		phy_leds_unregister(phydev);
 
+	if (!phydev->is_on_sfp_module)
+		phy_led_triggers_unregister(phydev);
+
 	phydev->state = PHY_DOWN;
 
 	phy_cleanup_ports(phydev);
-- 
2.51.0

Re: [BUG] AB-BA deadlock between net and led-trigger module

[Shiji Yang]

On Sat, 21 Feb 2026 23:48:22 +0000, Andrew Lunn wrote:

> On Sat, Feb 21, 2026 at 06:01:46PM +0800, Shiji Yang wrote:
> > The OpenWrt community reports that sometimes devices fail to start[1]
> > on 5.15 kernel. After further tracking, this is caused by a AB-BA
> > deadlock which can be reproduced in at least 5.15, 6.6, 6.12 and latest
> > 6.18 LTS kenrel.
> 
> Hi Shiji
> 
> Please could you test this patch. It is based on the net tree, but
> with a bit of fuzz will probably apply to older trees.
> 
> Thanks
> 	Andrew
> 
> From bf4d66187585a1893d558cb9357f1ef63437b898 Mon Sep 17 00:00:00 2001
> From: Andrew Lunn <andrew@lunn.ch>
> Date: Sat, 21 Feb 2026 14:51:54 -0600
> Subject: [PATCH net] net: phy: register phy led_triggers during probe to avoid
>  AB-BA deadlock
> 
> There is an AB-BA deadlock when both LEDS_TRIGGER_NETDEV and
> LED_TRIGGER_PHY are enabled:
> 
> [ 1362.049207] [<8054e4b8>] led_trigger_register+0x5c/0x1fc             <-- Trying to get lock "triggers_list_lock" via down_write(&triggers_list_lock);
> [ 1362.054536] [<80662830>] phy_led_triggers_register+0xd0/0x234
> [ 1362.060329] [<8065e200>] phy_attach_direct+0x33c/0x40c
> [ 1362.065489] [<80651fc4>] phylink_fwnode_phy_connect+0x15c/0x23c
> [ 1362.071480] [<8066ee18>] mtk_open+0x7c/0xba0
> [ 1362.075849] [<806d714c>] __dev_open+0x280/0x2b0
> [ 1362.080384] [<806d7668>] __dev_change_flags+0x244/0x24c
> [ 1362.085598] [<806d7698>] dev_change_flags+0x28/0x78
> [ 1362.090528] [<807150e4>] dev_ioctl+0x4c0/0x654                       <-- Hold lock "rtnl_mutex" by calling rtnl_lock();
> [ 1362.094985] [<80694360>] sock_ioctl+0x2f4/0x4e0
> [ 1362.099567] [<802e9c4c>] sys_ioctl+0x32c/0xd8c
> [ 1362.104022] [<80014504>] syscall_common+0x34/0x58
> 
> Here LED_TRIGGER_PHY is registering LED triggers during phy_attach
> while holding RTNL and then taking triggers_list_lock.
> 
> [ 1362.191101] [<806c2640>] register_netdevice_notifier+0x60/0x168      <-- Trying to get lock "rtnl_mutex" via rtnl_lock();
> [ 1362.197073] [<805504ac>] netdev_trig_activate+0x194/0x1e4
> [ 1362.202490] [<8054e28c>] led_trigger_set+0x1d4/0x360                 <-- Hold lock "triggers_list_lock" by down_read(&triggers_list_lock);
> [ 1362.207511] [<8054eb38>] led_trigger_write+0xd8/0x14c
> [ 1362.212566] [<80381d98>] sysfs_kf_bin_write+0x80/0xbc
> [ 1362.217688] [<8037fcd8>] kernfs_fop_write_iter+0x17c/0x28c
> [ 1362.223174] [<802cbd70>] vfs_write+0x21c/0x3c4
> [ 1362.227712] [<802cc0c4>] ksys_write+0x78/0x12c
> [ 1362.232164] [<80014504>] syscall_common+0x34/0x58
> 
> Here LEDS_TRIGGER_NETDEV is being enabled on an LED. It first takes
> triggers_list_lock and then RTNL. A classical AB-BA deadlock.
> 
> phy_led_triggers_registers() does not require the RTNL, it does not
> make any calls into the network stack which require protection. There
> is also no requirement the PHY has been attached to a MAC, the
> triggers only make use of phydev state. This allows the call to
> phy_led_triggers_registers() to be placed elsewhere. PHY probe() and
> release() don't hold RTNL, so solving the AB-BA deadlock.
> 
> Reported-by: Shiji Yang <yangshiji66@outlook.com>
> Closes: https://lore.kernel.org/all/OS7PR01MB13602B128BA1AD3FA38B6D1FFBC69A@OS7PR01MB13602.jpnprd01.prod.outlook.com/
> Fixes: 06f502f57d0d ("leds: trigger: Introduce a NETDEV trigger")
> Signed-off-by: Andrew Lunn <andrew@lunn.ch>
> ---
>  drivers/net/phy/phy_device.c | 25 +++++++++++++++++--------
>  1 file changed, 17 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/net/phy/phy_device.c b/drivers/net/phy/phy_device.c
> index 9b8eaac63b90..cbb4af604aa5 100644
> --- a/drivers/net/phy/phy_device.c
> +++ b/drivers/net/phy/phy_device.c
> @@ -1866,8 +1866,6 @@ int phy_attach_direct(struct net_device *dev, struct phy_device *phydev,
>  		goto error;
>  
>  	phy_resume(phydev);
> -	if (!phydev->is_on_sfp_module)
> -		phy_led_triggers_register(phydev);
>  
>  	/**
>  	 * If the external phy used by current mac interface is managed by
> @@ -1982,9 +1980,6 @@ void phy_detach(struct phy_device *phydev)
>  	phydev->phy_link_change = NULL;
>  	phydev->phylink = NULL;
>  
> -	if (!phydev->is_on_sfp_module)
> -		phy_led_triggers_unregister(phydev);
> -
>  	if (phydev->mdio.dev.driver)
>  		module_put(phydev->mdio.dev.driver->owner);
>  
> @@ -3778,16 +3773,27 @@ static int phy_probe(struct device *dev)
>  	/* Set the state to READY by default */
>  	phydev->state = PHY_READY;
>  
> +	/* Register the PHY LED triggers */
> +	if (!phydev->is_on_sfp_module)
> +		phy_led_triggers_register(phydev);
> +
>  	/* Get the LEDs from the device tree, and instantiate standard
>  	 * LEDs for them.
>  	 */
> -	if (IS_ENABLED(CONFIG_PHYLIB_LEDS) && !phy_driver_is_genphy(phydev))
> +	if (IS_ENABLED(CONFIG_PHYLIB_LEDS) && !phy_driver_is_genphy(phydev)) {
>  		err = of_phy_leds(phydev);
> +		if (err)
> +			goto out;
> +	}
> +
> +	return 0;
>  
>  out:
> +	if (!phydev->is_on_sfp_module)
> +		phy_led_triggers_unregister(phydev);
> +
>  	/* Re-assert the reset signal on error */
> -	if (err)
> -		phy_device_reset(phydev, 1);
> +	phy_device_reset(phydev, 1);
>  
>  	return err;
>  }
> @@ -3801,6 +3807,9 @@ static int phy_remove(struct device *dev)
>  	if (IS_ENABLED(CONFIG_PHYLIB_LEDS) && !phy_driver_is_genphy(phydev))
>  		phy_leds_unregister(phydev);
>  
> +	if (!phydev->is_on_sfp_module)
> +		phy_led_triggers_unregister(phydev);
> +
>  	phydev->state = PHY_DOWN;
>  
>  	phy_cleanup_ports(phydev);
> -- 
> 2.51.0
>

I backported this patch to the 6.12 kernel and it did actually
fix the deadlock issue for me. And the phy triggers are still
working properly. Thanks for your quick fix. Nice work!

Regards,
Shiji Yang

Re: [BUG] AB-BA deadlock between net and led-trigger module

[Andrew Lunn]

> I backported this patch to the 6.12 kernel and it did actually
> fix the deadlock issue for me. And the phy triggers are still
> working properly. Thanks for your quick fix. Nice work!

Great. I will post it officially, and it would be good if you can add
a Tested-by: for it.

https://docs.kernel.org/process/submitting-patches.html#using-reported-by-tested-by-reviewed-by-suggested-by-and-fixes

	Andrew