Net related kernel panic caused by commit in next-20230512..next-20230515

Net related kernel panic caused by commit in next-20230512..next-20230515

[Arınç ÜNAL]

[-- Attachment #1: Type: text/plain, Size: 480 bytes --]

I get a kernel panic on next-20230515 and next-20230516. next-20230512 
works fine. The panic log is attached.

The device boots fine with CONFIG_NET disabled which made me think the 
netdev mailing list is the best place to report this.

Tested on Bananapi BPI-R64 with this defconfig. The filesystem is 
Buildroot. The devicetree binary being fed to the bootloader is from 
next-20230515.

https://github.com/arinc9/linux/commit/b4a048428ab6f380b4203598aa62635e21015095

Arınç

[-- Attachment #2: panic.txt --]
[-- Type: text/plain, Size: 15132 bytes --]

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
[    0.000000] Linux version 6.4.0-rc2-next-20230516+ (arinc9@arinc9-PC) (aarch64-linux-gnu-gcc (Ubuntu 12.2.0-14ubuntu2) 12.2.0, GNU ld (GNU Binutils for Ubuntu) 2.40) #40 SMP Tue May 16 17:14:30 +03 2023
[    0.000000] Machine model: Bananapi BPI-R64
[    0.000000] earlycon: uart8250 at MMIO32 0x0000000011002000 (options '')
[    0.000000] printk: bootconsole [uart8250] enabled
[    0.000000] OF: reserved mem: 0x0000000043000000..0x000000004302ffff (192 KiB) nomap non-reusable secmon@43000000
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000040000000-0x000000007fffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000040000000-0x0000000042ffffff]
[    0.000000]   node   0: [mem 0x0000000043000000-0x000000004302ffff]
[    0.000000]   node   0: [mem 0x0000000043030000-0x000000007fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000007fffffff]
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: MIGRATE_INFO_TYPE not supported.
[    0.000000] psci: SMC Calling Convention v1.2
[    0.000000] percpu: Embedded 27 pages/cpu s73128 r8192 d29272 u110592
[    0.000000] pcpu-alloc: s73128 r8192 d29272 u110592 alloc=27*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 
[    0.000000] Detected VIPT I-cache on CPU0
[    0.000000] CPU features: kernel page table isolation disabled by kernel configuration
[    0.000000] CPU features: detected: ARM erratum 843419
[    0.000000] alternatives: applying boot alternatives
[    0.000000] Kernel command line: earlycon=uart8250,mmio32,0x11002000 swiotlb=512
[    0.000000] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.000000] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 258048
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 1005792K/1048576K available (8192K kernel code, 1484K rwdata, 2264K rodata, 448K init, 276K bss, 42784K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.000000] trace event string verifier disabled
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] 	Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GIC: GICv2 detected, but range too small and irqchip.gicv2_force_probe not set
[    0.000000] Root IRQ handler: gic_handle_irq
[    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.000000] arch_timer: cp15 timer(s) running at 12.50MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x2e2049cda, max_idle_ns: 440795202628 ns
[    0.000001] sched_clock: 56 bits at 13MHz, resolution 80ns, wraps every 4398046511080ns
[    0.008597] Calibrating delay loop (skipped), value calculated using timer frequency.. 25.00 BogoMIPS (lpj=125000)
[    0.019332] pid_max: default: 32768 minimum: 301
[    0.024383] Mount-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    0.032060] Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    0.040845] cacheinfo: Unable to detect cache hierarchy for CPU 0
[    0.047569] RCU Tasks Trace: Setting shift to 1 and lim to 1 rcu_task_cb_adjust=1.
[    0.055523] rcu: Hierarchical SRCU implementation.
[    0.060502] rcu: 	Max phase no-delay instances is 1000.
[    0.066360] smp: Bringing up secondary CPUs ...
[    0.071413] Detected VIPT I-cache on CPU1
[    0.071423] CPU features: SANITY CHECK: Unexpected variation in SYS_CNTFRQ_EL0. Boot CPU: 0x00000000bebc20, CPU1: 0x00000000000000
[    0.071446] CPU features: Unsupported CPU feature variation detected.
[    0.071479] CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
[    0.071545] smp: Brought up 1 node, 2 CPUs
[    0.105532] SMP: Total of 2 processors activated.
[    0.110400] CPU features: detected: 32-bit EL0 Support
[    0.115722] CPU features: detected: CRC32 instructions
[    0.121068] CPU features: emulated: Privileged Access Never (PAN) using TTBR0_EL1 switching
[    0.129723] CPU: All CPU(s) started at EL2
[    0.133960] alternatives: applying system-wide alternatives
[    0.140398] devtmpfs: initialized
[    0.147621] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.157844] futex hash table entries: 512 (order: 3, 32768 bytes, linear)
[    0.164980] pinctrl core: initialized pinctrl subsystem
[    0.170928] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.177383] DMA: preallocated 128 KiB GFP_KERNEL pool for atomic allocations
[    0.184733] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.192772] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.201421] thermal_sys: Registered thermal governor 'fair_share'
[    0.201428] thermal_sys: Registered thermal governor 'bang_bang'
[    0.207736] thermal_sys: Registered thermal governor 'step_wise'
[    0.213967] thermal_sys: Registered thermal governor 'user_space'
[    0.220246] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.233636] ASID allocator initialised with 65536 entries
[    0.243922] gpio-602 (asm_sel): hogged as output/high
[    0.255793] cryptd: max_cpu_qlen set to 1000
[    0.261044] SCSI subsystem initialized
[    0.265147] libata version 3.00 loaded.
[    0.287570] clocksource: Switched to clocksource arch_sys_counter
[    0.295212] NET: Registered PF_INET protocol family
[    0.300567] IP idents hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    0.309263] tcp_listen_portaddr_hash hash table entries: 512 (order: 1, 8192 bytes, linear)
[    0.317994] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.326043] TCP established hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.334156] TCP bind hash table entries: 8192 (order: 6, 262144 bytes, linear)
[    0.341865] TCP: Hash tables configured (established 8192 bind 8192)
[    0.348601] UDP hash table entries: 512 (order: 2, 16384 bytes, linear)
[    0.355477] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes, linear)
[    0.362947] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.368843] PCI: CLS 0 bytes, default 64
[    0.373082] Unpacking initramfs...
[    0.379186] workingset: timestamp_bits=62 max_order=18 bucket_order=0
[    0.386427] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.392570] jffs2: version 2.2. (NAND) (SUMMARY)  © 2001-2006 Red Hat, Inc.
[    0.407855] Freeing initrd memory: 9404K
[    0.439665] mt-pmic-pwrap 10001000.pwrap: unexpected interrupt int=0x1
[    0.453357] Serial: 8250/16550 driver, 3 ports, IRQ sharing disabled
[    0.461285] printk: console [ttyS0] disabled
[    0.485932] 11002000.serial: ttyS0 at MMIO 0x11002000 (irq = 118, base_baud = 1562500) is a ST16650V2
[    0.495537] printk: console [ttyS0] enabled
[    0.495537] printk: console [ttyS0] enabled
[    0.503912] printk: bootconsole [uart8250] disabled
[    0.503912] printk: bootconsole [uart8250] disabled
[    0.514770] 1100c000.serial: ttyS1 at MMIO 0x1100c000 (irq = 119, base_baud = 17499995) is a MediaTek BTIF
[    0.524767] serial serial0: tty port ttyS1 registered
[    0.530599] mtk_rng 1020f000.rng: registered RNG driver
[    0.530814] random: crng init done
[    0.542764] loop: module loaded
[    0.680094] mtk_soc_eth 1b100000.ethernet: generated random MAC address 5e:21:e3:7e:59:f6
[    0.689093] mtk_soc_eth 1b100000.ethernet eth0: mediatek frame engine at 0xffffffc009100000, irq 123
[    0.698337] mtk_soc_eth 1b100000.ethernet: generated random MAC address 96:4e:e5:1a:8f:92
[    0.707281] mtk_soc_eth 1b100000.ethernet eth1: mediatek frame engine at 0xffffffc009100000, irq 123
[    0.717297] rtc_mt7622 10212800.rtc: registered as rtc0
[    0.722572] rtc_mt7622 10212800.rtc: setting system clock to 2000-01-01T00:55:16 UTC (946688116)
[    0.731552] i2c_dev: i2c /dev entries driver
[    0.737982] mtk-wdt 10212000.watchdog: Watchdog enabled (timeout=31 sec, nowayout=0)
[    0.749096] mtk-msdc 11240000.mmc: Got CD GPIO
[    0.754148] hw perfevents: enabled with armv8_cortex_a53 PMU driver, 7 counters available
[    0.762922] NET: Registered PF_PACKET protocol family
[    0.768122] 8021q: 802.1Q VLAN Support v1.8
[    0.794698] mtk-pcie 1a143000.pcie: host bridge /pcie@1a143000 ranges:
[    0.801323] mtk-pcie 1a143000.pcie: Parsing ranges property...
[    0.807161] mtk-pcie 1a143000.pcie:      MEM 0x0020000000..0x0027ffffff -> 0x0020000000
[    0.865841] mmc0: new HS200 MMC card at address 0001
[    0.872051] mmcblk0: mmc0:0001 008G30 7.28 GiB 
[    0.878911] mmcblk0boot0: mmc0:0001 008G30 4.00 MiB 
[    0.885227] mmcblk0boot1: mmc0:0001 008G30 4.00 MiB 
[    0.890694] mmcblk0rpmb: mmc0:0001 008G30 4.00 MiB, chardev (251:0)
[    1.027595] mtk-pcie 1a143000.pcie: Port0 link down
[    1.032818] mtk-pcie 1a143000.pcie: PCI host bridge to bus 0000:00
[    1.039025] pci_bus 0000:00: root bus resource [bus 00-ff]
[    1.044509] pci_bus 0000:00: root bus resource [mem 0x20000000-0x27ffffff]
[    1.051383] pci_bus 0000:00: scanning bus
[    1.056490] pci_bus 0000:00: fixups for bus
[    1.060675] pci_bus 0000:00: bus scan returning with max=00
[    1.066589] mtk-pcie 1a145000.pcie: host bridge /pcie@1a145000 ranges:
[    1.073157] mtk-pcie 1a145000.pcie: Parsing ranges property...
[    1.078994] mtk-pcie 1a145000.pcie:      MEM 0x0028000000..0x002fffffff -> 0x0028000000
[    1.297607] mtk-pcie 1a145000.pcie: Port1 link down
[    1.302625] mtk-pcie 1a145000.pcie: PCI host bridge to bus 0001:00
[    1.308815] pci_bus 0001:00: root bus resource [bus 00-ff]
[    1.314297] pci_bus 0001:00: root bus resource [mem 0x28000000-0x2fffffff]
[    1.321170] pci_bus 0001:00: scanning bus
[    1.326281] pci_bus 0001:00: fixups for bus
[    1.330463] pci_bus 0001:00: bus scan returning with max=00
[    1.336574] mtk_hsdma 1b007000.dma-controller: MediaTek HSDMA driver registered
[    1.380019] mt7530-mdio mdio-bus:00: no interrupt support
[    1.408717] mt7530-mdio mdio-bus:00: configuring for fixed/2500base-x link mode
[    1.418008] mt7530-mdio mdio-bus:00: Link is Up - 2.5Gbps/Full - flow control rx/tx
[    1.428504] mt7530-mdio mdio-bus:00 wan (uninitialized): PHY [mt7530-0:00] driver [MediaTek MT7531 PHY] (irq=POLL)
[    1.450392] mt7530-mdio mdio-bus:00 lan0 (uninitialized): PHY [mt7530-0:01] driver [MediaTek MT7531 PHY] (irq=POLL)
[    1.472437] mt7530-mdio mdio-bus:00 lan1 (uninitialized): PHY [mt7530-0:02] driver [MediaTek MT7531 PHY] (irq=POLL)
[    1.494225] mt7530-mdio mdio-bus:00 lan2 (uninitialized): PHY [mt7530-0:03] driver [MediaTek MT7531 PHY] (irq=POLL)
[    1.515978] mt7530-mdio mdio-bus:00 lan3 (uninitialized): PHY [mt7530-0:04] driver [MediaTek MT7531 PHY] (irq=POLL)
[    1.527312] mtk_soc_eth 1b100000.ethernet eth0: entered promiscuous mode
[    1.534087] DSA: tree 0 setup
[    1.537703] clk: Disabling unused clocks
[    1.542267] Freeing unused kernel memory: 448K
[    1.567689] Run /init as init process
[    1.571350]   with arguments:
[    1.574307]     /init
[    1.576570]   with environment:
[    1.579714]     HOME=/
[    1.582064]     TERM=linux
Starting syslogd: OK
Starting klogd: OK
Running sysctl: OK
Saving random seed: OK
Starting network: [    1.699764] Unable to handle kernel paging request at virtual address ffffffc008a9cdf0
[    1.707711] Mem abort info:
[    1.710496]   ESR = 0x0000000096000007
[    1.714236]   EC = 0x25: DABT (current EL), IL = 32 bits
[    1.719640]   SET = 0, FnV = 0
[    1.722687]   EA = 0, S1PTW = 0
[    1.725818]   FSC = 0x07: level 3 translation fault
[    1.730706] Data abort info:
[    1.733577]   ISV = 0, ISS = 0x00000007
[    1.737403]   CM = 0, WnR = 0
[    1.740380] swapper pgtable: 4k pages, 39-bit VAs, pgdp=0000000040a4a000
[    1.747073] [ffffffc008a9cdf0] pgd=100000007ffff003, p4d=100000007ffff003, pud=100000007ffff003, pmd=100000007fffc003, pte=0000000000000000
[    1.759623] Internal error: Oops: 0000000096000007 [#1] SMP
[    1.765189] Modules linked in:
[    1.768238] CPU: 0 PID: 551 Comm: ip Tainted: G S                 6.4.0-rc2-next-20230516+ #40
[    1.776844] Hardware name: Bananapi BPI-R64 (DT)
[    1.781452] pstate: a0400005 (NzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[    1.788406] pc : notifier_call_chain+0x54/0x100
[    1.792939] lr : notifier_call_chain+0x70/0x100
[    1.797462] sp : ffffffc0099a35d0
[    1.800766] x29: ffffffc0099a35d0 x28: ffffffc008a9cde8 x27: ffffffc00876490c
[    1.807899] x26: ffffffc008ac88b8 x25: ffffffc008aa7008 x24: ffffffc0099a3690
[    1.815031] x23: 000000000000000e x22: 0000000000000000 x21: 00000000fffffff9
[    1.822162] x20: ffffffc008a9cde8 x19: 000000000000000e x18: 0000000000000000
[    1.829294] x17: 0000000000000000 x16: 0000000000000000 x15: 0000007fe4a120b8
[    1.836425] x14: ffffffc0099a3d30 x13: 0000000000000000 x12: 0000000040a49000
[    1.843556] x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000000
[    1.850687] x8 : 0000000000000000 x7 : 0000000000000000 x6 : 000000000000003f
[    1.857819] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 00000000ffffffff
[    1.864950] x2 : ffffffc0099a3690 x1 : 000000000000000e x0 : 0000000000000000
[    1.872081] Call trace:
[    1.874518]  notifier_call_chain+0x54/0x100
[    1.878696]  raw_notifier_call_chain+0x14/0x1c
[    1.883133]  call_netdevice_notifiers_info+0x4c/0x90
[    1.888093]  __dev_open+0x64/0x164
[    1.891489]  __dev_change_flags+0x190/0x1ec
[    1.895665]  dev_change_flags+0x20/0x64
[    1.899495]  do_setlink+0x278/0xeec
[    1.902978]  __rtnl_newlink+0x4e0/0x71c
[    1.906808]  rtnl_newlink+0x4c/0x74
[    1.910290]  rtnetlink_rcv_msg+0x274/0x364
[    1.914380]  netlink_rcv_skb+0x38/0xf0
[    1.918125]  rtnetlink_rcv+0x14/0x1c
[    1.921693]  netlink_unicast+0x1dc/0x2c4
[    1.925609]  netlink_sendmsg+0x19c/0x3c4
[    1.929523]  ____sys_sendmsg+0x9c/0x284
[    1.933356]  ___sys_sendmsg+0x68/0x8c
[    1.937014]  __sys_sendmsg+0x44/0x98
[    1.940585]  __arm64_sys_sendmsg+0x20/0x28
[    1.944676]  do_el0_svc+0x6c/0xfc
[    1.947986]  el0_svc+0x14/0x48
[    1.951037]  el0t_64_sync_handler+0xec/0x118
[    1.955303]  el0t_64_sync+0x150/0x154
[    1.958960] Code: 90005259 91002339 d503201f aa1403fc (f9400694) 
[    1.965045] ---[ end trace 0000000000000000 ]---
[    1.969655] Kernel panic - not syncing: Oops: Fatal exception
[    1.975391] SMP: stopping secondary CPUs
[    1.979307] Kernel Offset: disabled
[    1.982786] CPU features: 0x000000,08000000,0000400b
[    1.987743] Memory Limit: none
[    1.990789] Rebooting in 1 seconds..

Re: Net related kernel panic caused by commit in next-20230512..next-20230515

[Jakub Kicinski]

On Tue, 16 May 2023 17:42:44 +0300 Arınç ÜNAL wrote:
> I get a kernel panic on next-20230515 and next-20230516. next-20230512 
> works fine. The panic log is attached.
> 
> The device boots fine with CONFIG_NET disabled which made me think the 
> netdev mailing list is the best place to report this.
> 
> Tested on Bananapi BPI-R64 with this defconfig. The filesystem is 
> Buildroot. The devicetree binary being fed to the bootloader is from 
> next-20230515.
> 
> https://github.com/arinc9/linux/commit/b4a048428ab6f380b4203598aa62635e21015095

Should be now fixed in net/main, by commit d6352dae0903 ("devlink: Fix
crash with CONFIG_NET_NS=n")

Re: Net related kernel panic caused by commit in next-20230512..next-20230515

[Arınç ÜNAL]

On 17.05.2023 06:38, Jakub Kicinski wrote:
> On Tue, 16 May 2023 17:42:44 +0300 Arınç ÜNAL wrote:
>> I get a kernel panic on next-20230515 and next-20230516. next-20230512
>> works fine. The panic log is attached.
>>
>> The device boots fine with CONFIG_NET disabled which made me think the
>> netdev mailing list is the best place to report this.
>>
>> Tested on Bananapi BPI-R64 with this defconfig. The filesystem is
>> Buildroot. The devicetree binary being fed to the bootloader is from
>> next-20230515.
>>
>> https://github.com/arinc9/linux/commit/b4a048428ab6f380b4203598aa62635e21015095
> 
> Should be now fixed in net/main, by commit d6352dae0903 ("devlink: Fix
> crash with CONFIG_NET_NS=n")

I can confirm this commit fixes it, thanks.

Arınç