[BUG] unbinding mv88e6xxx device spews

[BUG] unbinding mv88e6xxx device spews

[Russell King (Oracle)]

Hi,

Unbinding a mv88e6xxx device spews thusly:

[ 1499.372164] 8<--- cut here ---
[ 1499.375412] Unable to handle kernel NULL pointer dereference at virtual address 00000000 when read
[ 1499.384463] [00000000] *pgd=00000000
[ 1499.388126] Internal error: Oops: 5 [#1] SMP ARM
[ 1499.392774] Modules linked in: caam_jr ofpart caamhash_desc reset_gpio caamalg_desc tag_dsa crypto_engine cmdlinepart authenc libdes i2c_mux_pca954x mv88e6xxx at24 lm75 spi_nor mtd dsa_core eeprom_93xx46 caam vf610_adc error industrialio_triggered_buffer fsl_edma kfifo_buf virt_dma spi_gpio spi_bitbang sfp iio_hwmon sff mdio_mux_gpio industrialio mdio_i2c mdio_mux rpcsec_gss_krb5 auth_rpcgss
[ 1499.427797] CPU: 0 UID: 0 PID: 561 Comm: bash Tainted: G        W          6.14.0+ #965
[ 1499.435834] Tainted: [W]=WARN
[ 1499.438813] Hardware name: Freescale Vybrid VF5xx/VF6xx (Device Tree)
[ 1499.445270] PC is at devlink_region_destroy+0x8/0x28
[ 1499.450309] LR is at mv88e6xxx_teardown_devlink_regions_global+0x20/0x2c [mv88e6xxx]
[ 1499.458500] pc : [<c09cc4b8>]    lr : [<bf13ba88>]    psr: 800d0013
[ 1499.464780] sp : e09b5e40  ip : c2f5bf00  fp : 00000000
[ 1499.470020] r10: 00000000  r9 : c0a79388  r8 : c2c085d4
[ 1499.475260] r7 : 00000000  r6 : c2dc0748  r5 : c1caaaf8  r4 : 00000000
[ 1499.481802] r3 : c2f5bf00  r2 : 00000000  r1 : 00000000  r0 : 00000000
[ 1499.488346] Flags: Nzcv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[ 1499.495506] Control: 10c5387d  Table: 842d404a  DAC: 00000051
[ 1499.501263] Register r0 information: NULL pointer
[ 1499.506000] Register r1 information: NULL pointer
[ 1499.510728] Register r2 information: NULL pointer
[ 1499.515456] Register r3 information: slab task_struct start c2f5bf00 pointer offset 0 size 2304
[ 1499.524220] Register r4 information: NULL pointer
[ 1499.528947] Register r5 information: non-slab/vmalloc memory
[ 1499.534630] Register r6 information: slab kmalloc-64 start c2dc0740 pointer offset 8 size 64
[ 1499.543124] Register r7 information: NULL pointer
[ 1499.547853] Register r8 information: slab kmalloc-4k start c2c08000 pointer offset 1492 size 4096
[ 1499.556780] Register r9 information: non-slab/vmalloc memory
[ 1499.562463] Register r10 information: NULL pointer
[ 1499.567278] Register r11 information: NULL pointer
[ 1499.572093] Register r12 information: slab task_struct start c2f5bf00 pointer offset 0 size 2304
[ 1499.580935] Process bash (pid: 561, stack limit = 0xe09b4000)
[ 1499.586705] Stack: (0xe09b5e40 to 0xe09b6000)
[ 1499.591092] 5e40: c1caaaf4 c1caaaf8 c2dc0748 bf13ba88 c2bdfa00 c1ca8040 c2dc0748 bf134844
[ 1499.599297] 5e60: c2bdfa00 c2f13800 c2dc0748 bf0b8620 c2dc0740 c2bdfa00 bf148000 c284e444
[ 1499.607504] 5e80: c2c085d4 bf0b9350 c1ca8040 c2c08590 bf148000 c284e444 c2c085d4 c0a79388
[ 1499.615712] 5ea0: 00000000 bf136b30 c284e400 c2c08590 bf148000 c066f838 c284e400 c05e15f8
[ 1499.623919] 5ec0: c0ac99bc c284e400 00000010 bf148000 c2a9e010 c05df3bc 00000010 c2959dc0
[ 1499.632127] 5ee0: c2a9e000 e09b5f40 c2a9e010 c02f9388 00000000 00000000 c2b68460 e09b5f88
[ 1499.640334] 5f00: c02f927c 00000010 00000000 00000000 00000000 c02608a4 c2caa1b0 00000000
[ 1499.648541] 5f20: b6999000 e09b5fb0 00010000 00000010 00ce6b00 00000000 00000001 00000000
[ 1499.656748] 5f40: c2b68460 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1499.664948] 5f60: 00000000 00000000 c2b68460 c2b68460 00000000 00000000 c0008584 c2f5bf00
[ 1499.673155] 5f80: 00000004 c0260adc 00000000 00000000 00000000 00000074 00ce6b00 b6ca5db0
[ 1499.681354] 5fa0: 00000004 c0008320 00000074 00ce6b00 00000001 00ce6b00 00000010 00000000
[ 1499.689561] 5fc0: 00000074 00ce6b00 b6ca5db0 00000004 00000010 00000010 00000000 00000000
[ 1499.697769] 5fe0: 00000004 bef1b880 b6c3d5b3 b6bc6746 60070030 00000001 00000000 00000000
[ 1499.705956] Call trace:
[ 1499.705981] [<c09cc4b8>] (devlink_region_destroy) from [<bf13ba88>] (mv88e6xxx_teardown_devlink_regions_global+0x20/0x2c [mv88e6xxx])
[ 1499.720797] [<bf13ba88>] (mv88e6xxx_teardown_devlink_regions_global [mv88e6xxx]) from [<bf134844>] (mv88e6xxx_teardown+0x30/0x3c [mv88e6xxx])
[ 1499.733871] [<bf134844>] (mv88e6xxx_teardown [mv88e6xxx]) from [<bf0b8620>] (dsa_tree_teardown_switches+0x94/0xc4 [dsa_core])
[ 1499.745857] [<bf0b8620>] (dsa_tree_teardown_switches [dsa_core]) from [<bf0b9350>] (dsa_unregister_switch+0xdc/0x184 [dsa_core])
[ 1499.757799] [<bf0b9350>] (dsa_unregister_switch [dsa_core]) from [<bf136b30>] (mv88e6xxx_remove+0x34/0xbc [mv88e6xxx])
[ 1499.768948] [<bf136b30>] (mv88e6xxx_remove [mv88e6xxx]) from [<c066f838>] (mdio_remove+0x1c/0x30)
[ 1499.778077] [<c066f838>] (mdio_remove) from [<c05e15f8>] (device_release_driver_internal+0x180/0x1f4)
[ 1499.787394] [<c05e15f8>] (device_release_driver_internal) from [<c05df3bc>] (unbind_store+0x54/0x90)
[ 1499.796588] [<c05df3bc>] (unbind_store) from [<c02f9388>] (kernfs_fop_write_iter+0x10c/0x1cc)
[ 1499.805181] [<c02f9388>] (kernfs_fop_write_iter) from [<c02608a4>] (vfs_write+0x2a4/0x3dc)
[ 1499.813500] [<c02608a4>] (vfs_write) from [<c0260adc>] (ksys_write+0x50/0xac)[ 1499.820681] [<c0260adc>] (ksys_write) from [<c0008320>] (ret_fast_syscall+0x0/0x54)
[ 1499.828383] Exception stack(0xe09b5fa8 to 0xe09b5ff0)
[ 1499.833460] 5fa0:                   00000074 00ce6b00 00000001 00ce6b00 00000010 00000000
[ 1499.841660] 5fc0: 00000074 00ce6b00 b6ca5db0 00000004 00000010 00000010 00000
[ 1499.849863] 5fe0: 00000004 bef1b880 b6c3d5b3 b6bc6746
[ 1499.854947] Code: e8bd41f0 eae1baa6 e92d4070 e1a04000 (e5905000)
[ 1499.861264] ---[ end trace 0000000000000000 ]---
[ 1499.896270] fec 400d0000.ethernet eth0: Graceful transmit stop did not complete!
[ 1499.903874] fec 400d0000.ethernet eth0: Link is Down

and the nice thing is, because it's using rootfs over eth0 (which is not
the NIC used for DSA) that's the end of the platform without hard
rebooting.

Again, won't be able to do anything with this after Sunday for a few
months, and I'm otherwise completely focused on PTP stuff right now.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!

Re: [BUG] unbinding mv88e6xxx device spews

[Vladimir Oltean]

[-- Attachment #1: Type: text/plain, Size: 281 bytes --]

On Fri, Apr 11, 2025 at 06:29:52PM +0100, Russell King (Oracle) wrote:
> Hi,
> 
> Unbinding a mv88e6xxx device spews thusly:

Odd. I never saw this on the 6190 and 6390 I've been testing on, and I
think I know why. Could you please confirm that the attached patch fixes
the issue?

[-- Attachment #2: 0001-net-dsa-mv88e6xxx-avoid-unregistering-devlink-region.patch --]
[-- Type: text/x-diff, Size: 1845 bytes --]

From f4f9aebc0a8f8424a9a59536368605afc30a55df Mon Sep 17 00:00:00 2001
From: Vladimir Oltean <vladimir.oltean@nxp.com>
Date: Fri, 11 Apr 2025 20:55:39 +0300
Subject: [PATCH] net: dsa: mv88e6xxx: avoid unregistering devlink regions
 which were never registered

Russell King reports that a system with mv88e6xxx dereferences a NULL
pointer when unbinding this driver:
https://lore.kernel.org/netdev/Z_lRkMlTJ1KQ0kVX@shell.armlinux.org.uk/

The crash seems to be in devlink_region_destroy(), which is not NULL
tolerant but is given a NULL devlink global region pointer.

At least on some chips, some devlink regions are conditionally registered
since the blamed commit, see mv88e6xxx_setup_devlink_regions_global():

		if (cond && !cond(chip))
			continue;

These are MV88E6XXX_REGION_STU and MV88E6XXX_REGION_PVT. If the chip
does not have an STU or PVT, it should crash like this.

To fix the issue, avoid unregistering those regions which are NULL, i.e.
were skipped at mv88e6xxx_setup_devlink_regions_global() time.

Fixes: 836021a2d0e0 ("net: dsa: mv88e6xxx: Export cross-chip PVT as devlink region")
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
 drivers/net/dsa/mv88e6xxx/devlink.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/net/dsa/mv88e6xxx/devlink.c b/drivers/net/dsa/mv88e6xxx/devlink.c
index 795c8df7b6a7..195460a0a0d4 100644
--- a/drivers/net/dsa/mv88e6xxx/devlink.c
+++ b/drivers/net/dsa/mv88e6xxx/devlink.c
@@ -736,7 +736,8 @@ void mv88e6xxx_teardown_devlink_regions_global(struct dsa_switch *ds)
 	int i;
 
 	for (i = 0; i < ARRAY_SIZE(mv88e6xxx_regions); i++)
-		dsa_devlink_region_destroy(chip->regions[i]);
+		if (chip->regions[i])
+			dsa_devlink_region_destroy(chip->regions[i]);
 }
 
 void mv88e6xxx_teardown_devlink_regions_port(struct dsa_switch *ds, int port)
-- 
2.43.0

Re: [BUG] unbinding mv88e6xxx device spews

[Russell King (Oracle)]

On Fri, Apr 11, 2025 at 09:01:59PM +0300, Vladimir Oltean wrote:
> On Fri, Apr 11, 2025 at 06:29:52PM +0100, Russell King (Oracle) wrote:
> > Hi,
> > 
> > Unbinding a mv88e6xxx device spews thusly:
> 
> Odd. I never saw this on the 6190 and 6390 I've been testing on, and I
> think I know why. Could you please confirm that the attached patch fixes
> the issue?

What else can go wrong... well, the build PC can inexplicably lose
power just before it transfers the kernel to the TFTP server and
modules to the target... yep, it's one of those days that if something
can go wrong it will go wrong. I'm expecting a meteorite to destroy
the earth in the next few minutes.

Your patch seems to fix that issue, so:

Tested-by: Russell King (Oracle) <rmk+kernel@armlinux.org.uk>

but... of course there's another issue buried beneath:

[   73.552305] WARNING: CPU: 0 PID: 398 at net/dsa/dsa.c:1486 dsa_switch_release_ports+0x114/0x118 [dsa_core]
[   73.562504] Modules linked in: caam_jr ofpart caamhash_desc caamalg_desc reset_gpio tag_dsa crypto_engine cmdlinepart authenc libdes i2c_mux_pca954x lm75 at24 mv88e6xxx spi_nor mtd dsa_core eeprom_93xx46 caam vf610_adc error industrialio_triggered_buffer fsl_edma kfifo_buf virt_dma spi_gpio sfp spi_bitbang iio_hwmon sff mdio_mux_gpio mdio_i2c industrialio mdio_mux rpcsec_gss_krb5 auth_rpcgss
[   73.597676] CPU: 0 UID: 0 PID: 398 Comm: bash Tainted: G        W          6.14.0+ #966
[   73.597716] Tainted: [W]=WARN
[   73.597724] Hardware name: Freescale Vybrid VF5xx/VF6xx (Device Tree)
[   73.597737] Call trace:
[   73.597758] [<c0009c44>] (unwind_backtrace) from [<c0022b78>] (show_stack+0x10/0x14)
[   73.597849] [<c0022b78>] (show_stack) from [<c0019b5c>] (dump_stack_lvl+0x50/0x64)
[   73.597921] [<c0019b5c>] (dump_stack_lvl) from [<c0043cd4>] (__warn+0x80/0x128)
[   73.597986] [<c0043cd4>] (__warn) from [<c0043ee4>] (warn_slowpath_fmt+0x168/0x16c)
[   73.598034] [<c0043ee4>] (warn_slowpath_fmt) from [<bf0b8764>] (dsa_switch_release_ports+0x114/0x118 [dsa_core])
[   73.598297] [<bf0b8764>] (dsa_switch_release_ports [dsa_core]) from [<bf0b929c>] (dsa_unregister_switch+0x28/0x184 [dsa_core])
[   73.598654] [<bf0b929c>] (dsa_unregister_switch [dsa_core]) from [<bf105b30>] (mv88e6xxx_remove+0x34/0xbc [mv88e6xxx])
[   73.599326] [<bf105b30>] (mv88e6xxx_remove [mv88e6xxx]) from [<c066f838>] (mdio_remove+0x1c/0x30)
[   73.599577] [<c066f838>] (mdio_remove) from [<c05e15f8>] (device_release_driver_internal+0x180/0x1f4)
[   73.599666] [<c05e15f8>] (device_release_driver_internal) from [<c05df3bc>] (unbind_store+0x54/0x90)
[   73.599726] [<c05df3bc>] (unbind_store) from [<c02f9388>] (kernfs_fop_write_iter+0x10c/0x1cc)
[   73.599790] [<c02f9388>] (kernfs_fop_write_iter) from [<c02608a4>] (vfs_write+0x2a4/0x3dc)
[   73.599839] [<c02608a4>] (vfs_write) from [<c0260adc>] (ksys_write+0x50/0xac)
[   73.599876] [<c0260adc>] (ksys_write) from [<c0008320>] (ret_fast_syscall+0x0/0x54)
[   73.599912] Exception stack(0xe0b25fa8 to 0xe0b25ff0)
[   73.599940] 5fa0:                   00000010 024dd820 00000001 024dd820 00000010 00000001
[   73.599964] 5fc0: 00000010 024dd820 b6bb5d50 00000004 00000010 0055db68 00000000 00000000
[   73.599982] 5fe0: 00000004 bea469a0 b6b4e3fb b6ac7656
[   73.767849] ---[ end trace 0000000000000000 ]---
bash-5.0# [   74.466821] fec 400d0000.ethernet eth0: Graceful transmit stop did not complete!
[   74.474953] fec 400d0000.ethernet eth0: Link is Down

which seems to be due to:

                WARN_ON(!list_empty(&dp->vlans));

This is probably due to the other issue I reported:

[   44.485597] br0: port 9(optical2) entered disabled state
[   44.498847] br0: port 9(optical2) failed to delete vlan 1: -ENOENT
[   44.505353] ------------[ cut here ]------------
[   44.510052] WARNING: CPU: 0 PID: 438 at net/bridge/br_vlan.c:433 nbp_vlan_flu
sh+0xc0/0xc4

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!

Re: [BUG] unbinding mv88e6xxx device spews

[Vladimir Oltean]

On Fri, Apr 11, 2025 at 07:44:00PM +0100, Russell King (Oracle) wrote:
> On Fri, Apr 11, 2025 at 09:01:59PM +0300, Vladimir Oltean wrote:
> > On Fri, Apr 11, 2025 at 06:29:52PM +0100, Russell King (Oracle) wrote:
> > > Hi,
> > > 
> > > Unbinding a mv88e6xxx device spews thusly:
> > 
> > Odd. I never saw this on the 6190 and 6390 I've been testing on, and I
> > think I know why. Could you please confirm that the attached patch fixes
> > the issue?
> 
> What else can go wrong... well, the build PC can inexplicably lose
> power just before it transfers the kernel to the TFTP server and
> modules to the target... yep, it's one of those days that if something
> can go wrong it will go wrong. I'm expecting a meteorite to destroy
> the earth in the next few minutes.
> 
> Your patch seems to fix that issue, so:
> 
> Tested-by: Russell King (Oracle) <rmk+kernel@armlinux.org.uk>

Thanks for testing.

> but... of course there's another issue buried beneath:
> 
> [   73.552305] WARNING: CPU: 0 PID: 398 at net/dsa/dsa.c:1486 dsa_switch_release_ports+0x114/0x118 [dsa_core]
> [   73.562504] Modules linked in: caam_jr ofpart caamhash_desc caamalg_desc reset_gpio tag_dsa crypto_engine cmdlinepart authenc libdes i2c_mux_pca954x lm75 at24 mv88e6xxx spi_nor mtd dsa_core eeprom_93xx46 caam vf610_adc error industrialio_triggered_buffer fsl_edma kfifo_buf virt_dma spi_gpio sfp spi_bitbang iio_hwmon sff mdio_mux_gpio mdio_i2c industrialio mdio_mux rpcsec_gss_krb5 auth_rpcgss
> [   73.597676] CPU: 0 UID: 0 PID: 398 Comm: bash Tainted: G        W          6.14.0+ #966
> [   73.597716] Tainted: [W]=WARN
> [   73.597724] Hardware name: Freescale Vybrid VF5xx/VF6xx (Device Tree)
> [   73.597737] Call trace:
> [   73.597758] [<c0009c44>] (unwind_backtrace) from [<c0022b78>] (show_stack+0x10/0x14)
> [   73.597849] [<c0022b78>] (show_stack) from [<c0019b5c>] (dump_stack_lvl+0x50/0x64)
> [   73.597921] [<c0019b5c>] (dump_stack_lvl) from [<c0043cd4>] (__warn+0x80/0x128)
> [   73.597986] [<c0043cd4>] (__warn) from [<c0043ee4>] (warn_slowpath_fmt+0x168/0x16c)
> [   73.598034] [<c0043ee4>] (warn_slowpath_fmt) from [<bf0b8764>] (dsa_switch_release_ports+0x114/0x118 [dsa_core])
> [   73.598297] [<bf0b8764>] (dsa_switch_release_ports [dsa_core]) from [<bf0b929c>] (dsa_unregister_switch+0x28/0x184 [dsa_core])
> [   73.598654] [<bf0b929c>] (dsa_unregister_switch [dsa_core]) from [<bf105b30>] (mv88e6xxx_remove+0x34/0xbc [mv88e6xxx])
> [   73.599326] [<bf105b30>] (mv88e6xxx_remove [mv88e6xxx]) from [<c066f838>] (mdio_remove+0x1c/0x30)
> [   73.599577] [<c066f838>] (mdio_remove) from [<c05e15f8>] (device_release_driver_internal+0x180/0x1f4)
> [   73.599666] [<c05e15f8>] (device_release_driver_internal) from [<c05df3bc>] (unbind_store+0x54/0x90)
> [   73.599726] [<c05df3bc>] (unbind_store) from [<c02f9388>] (kernfs_fop_write_iter+0x10c/0x1cc)
> [   73.599790] [<c02f9388>] (kernfs_fop_write_iter) from [<c02608a4>] (vfs_write+0x2a4/0x3dc)
> [   73.599839] [<c02608a4>] (vfs_write) from [<c0260adc>] (ksys_write+0x50/0xac)
> [   73.599876] [<c0260adc>] (ksys_write) from [<c0008320>] (ret_fast_syscall+0x0/0x54)
> [   73.599912] Exception stack(0xe0b25fa8 to 0xe0b25ff0)
> [   73.599940] 5fa0:                   00000010 024dd820 00000001 024dd820 00000010 00000001
> [   73.599964] 5fc0: 00000010 024dd820 b6bb5d50 00000004 00000010 0055db68 00000000 00000000
> [   73.599982] 5fe0: 00000004 bea469a0 b6b4e3fb b6ac7656
> [   73.767849] ---[ end trace 0000000000000000 ]---
> bash-5.0# [   74.466821] fec 400d0000.ethernet eth0: Graceful transmit stop did not complete!
> [   74.474953] fec 400d0000.ethernet eth0: Link is Down
> 
> which seems to be due to:
> 
>                 WARN_ON(!list_empty(&dp->vlans));
> 
> This is probably due to the other issue I reported:
> 
> [   44.485597] br0: port 9(optical2) entered disabled state
> [   44.498847] br0: port 9(optical2) failed to delete vlan 1: -ENOENT
> [   44.505353] ------------[ cut here ]------------
> [   44.510052] WARNING: CPU: 0 PID: 438 at net/bridge/br_vlan.c:433 nbp_vlan_flu
> sh+0xc0/0xc4

No, they're not related. This is the third one, and I already know about it,
but it's relatively harmless.  Since I knocked down 2 already, let me
just go and take care of this one as well.

Re: [BUG] unbinding mv88e6xxx device spews

[Vladimir Oltean]

On Fri, Apr 11, 2025 at 09:54:30PM +0300, Vladimir Oltean wrote:
> > but... of course there's another issue buried beneath:
> > 
> > which seems to be due to:
> > 
> >                 WARN_ON(!list_empty(&dp->vlans));
> > 
> > This is probably due to the other issue I reported:
> > 
> > [   44.485597] br0: port 9(optical2) entered disabled state
> > [   44.498847] br0: port 9(optical2) failed to delete vlan 1: -ENOENT
> > [   44.505353] ------------[ cut here ]------------
> > [   44.510052] WARNING: CPU: 0 PID: 438 at net/bridge/br_vlan.c:433 nbp_vlan_flu
> > sh+0xc0/0xc4
> 
> No, they're not related. This is the third one, and I already know about it,
> but it's relatively harmless.  Since I knocked down 2 already, let me
> just go and take care of this one as well.

Actually, I think you're right. The WARN_ON() that &dp->vlans isn't
empty should be a consequence of the STU issue. Please let me know if
this WARN_ON() disappears when testing the patch in the other thread.

I was under the impression that you were hitting a different WARN_ON(),
just one or two lines above this one, where &dp->fdbs or &dp->mdbs are
not empty. That's what I was aware of, but it looks like you're not
there just yet.