> Hello Lorenzo & Felix, Hi Ben, > > With the hack to fix the crash I just posted, the kernel can get far enough to > show this lockdep splat. I cannot find any code that grabs the wiphy mtx > in this path, and yet if I try to add an explicit mutex_lock lock in the scan code, then > I get kernel deadlocks. Maybe I'm just putting the lock in the wrong place. > > Any idea how this is supposed to work with regards to wiphy locking? > > And in addition, there is RCU splat immediately after the key.c related > splat. > > 315 Nov 12 10:40:51 ct523c-a0af kernel: ------------[ cut here ]------------ > 316 Nov 12 10:40:51 ct523c-a0af kernel: WARNING: CPU: 1 PID: 115 at net/mac80211/key.c:1029 ieee80211_iter_keys+0xe4/0x1a0 [mac80211] > 317 Nov 12 10:40:51 ct523c-a0af kernel: Modules linked in: vrf > nf_conntrack_netlink nf_conntrack nfnetlink nf_defrag_ipv6 nf_defrag_ipv4 > 8021q garp mrp stp llc macvlan pktgen rpcrdma rdma_cm iw_cm ib_cm ib_core > qrtr bnep snd_hda_codec_intelhdmi intel_rapl_msr snd_hda_codec_hdmi > iTCO_wdt coretemp ee1004 intel_pmc_bxt iTCO_vendor_support intel_rapl_common > snd_hda_codec_alc662 intel_uncore_frequency snd_hda_codec_realtek_lib > intel_uncore_frequency_common snd_hda_codec_generic iwlmld mt7996e > intel_tcc_cooling snd_hda_intel mt76_connac_lib mt76 > x86_pkg_temp_thermal snd_intel_dspcfg snd_hda_codec intel_powerclamp pcspkr > snd_hda_core mac80211 intel_wmi_thunderbolt iwlwifi > snd_hwdep btusb snd_seq btbcm snd_seq_device btmtk btrtl btintel joydev > bluetooth snd_pcm cfg80211 i2c_i801 snd_timer i2c_smbus bfq snd > i2c_mux soundcore intel_pmc_core mei_pxp mei_hdcp intel_pch_thermal > pmt_telemetry pmt_discovery pmt_class intel_pmc_ssram_telemetry > intel_vsec acpi_pad nfsd auth_rpcgss nfs_acl lockd grace nfs_localio > sch_fq_codel sunrpc fuse zram raid1 dm_raid > 318 Nov 12 10:40:51 ct523c-a0af kernel: raid456 async_raid6_recov > async_memcpy async_pq async_xor xor async_tx raid6_pq i915 drm_buddy > intel_gtt drm_client_lib drm_display_helper drm_kms_helper cec rc_core igb > ttm intel_oc_wdt agpgart i2c_algo_bit ixgbe mdio libie_fwlog dca drm hwmon > mei_wdt i2c_core video wmi [last unloaded: nfnetlink] > 319 Nov 12 10:40:51 ct523c-a0af kernel: CPU: 1 UID: 0 PID: 115 Comm: kworker/u32:4 Tainted: G S 6.18.0-rc5+ #25 PREEMPT(full) > 320 Nov 12 10:40:51 ct523c-a0af kernel: Tainted: [S]=CPU_OUT_OF_SPEC > 321 Nov 12 10:40:51 ct523c-a0af kernel: Hardware name: Default string Default string/SKYBAY, BIOS 5.12 02/15/2023 > 322 Nov 12 10:40:51 ct523c-a0af kernel: Workqueue: phy2 mt76_scan_work [mt76] > 323 Nov 12 10:40:51 ct523c-a0af kernel: RIP: 0010:ieee80211_iter_keys+0xe4/0x1a0 [mac80211] > 324 Nov 12 10:40:51 ct523c-a0af kernel: Code: 4c 48 83 c4 10 5b 5d 41 5c > 41 5d 41 5e 41 5f c3 48 8b 47 48 be ff ff ff ff 48 8d 78 68 e8 04 ec fe > e0 85 c0 0f 85 49 ff ff ff <0f> 0b 4d 85 ed 0f 85 47 ff ff ff 4c 8b ab 90 1a > 00 00 48 8d 83 90 > 325 Nov 12 10:40:51 ct523c-a0af kernel: RSP: 0018:ffffc90000f1fcb0 EFLAGS: 00010246 > 326 Nov 12 10:40:51 ct523c-a0af kernel: RAX: 0000000000000000 RBX: ffff888132710ee0 RCX: 0000000000000000 > 327 Nov 12 10:40:51 ct523c-a0af kernel: RDX: 0000000000000000 RSI: ffff888132710788 RDI: ffff8881158435c8 > 328 Nov 12 10:40:51 ct523c-a0af kernel: RBP: ffffffffa0de5400 R08: 0000000000000001 R09: 00000000f9936176 > 329 Nov 12 10:40:51 ct523c-a0af kernel: R10: 0000000000000001 R11: ffff8881158435a0 R12: ffffc90000f1fd00 > 330 Nov 12 10:40:51 ct523c-a0af kernel: R13: ffff88811642aa80 R14: ffff88813271afc8 R15: 000000000000043f > 331 Nov 12 10:40:51 ct523c-a0af kernel: FS: 0000000000000000(0000) GS:ffff8884da4c6000(0000) knlGS:0000000000000000 > 332 Nov 12 10:40:51 ct523c-a0af kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > 333 Nov 12 10:40:51 ct523c-a0af kernel: CR2: 0000000000422bcc CR3: 0000000002c52004 CR4: 00000000003706f0 > 334 Nov 12 10:40:51 ct523c-a0af kernel: Call Trace: > 335 Nov 12 10:40:51 ct523c-a0af kernel: > 336 Nov 12 10:40:51 ct523c-a0af kernel: ? lock_acquire+0xc2/0x2c0 > 337 Nov 12 10:40:51 ct523c-a0af kernel: mt7996_vif_link_remove+0x66/0x290 [mt7996e] > 338 Nov 12 10:40:51 ct523c-a0af kernel: mt76_put_vif_phy_link+0x41/0x50 [mt76] > 339 Nov 12 10:40:51 ct523c-a0af kernel: mt76_scan_complete+0x77/0x100 [mt76] > 340 Nov 12 10:40:51 ct523c-a0af kernel: mt76_scan_work+0x30a/0x440 [mt76] > 341 Nov 12 10:40:51 ct523c-a0af kernel: ? process_one_work+0x1e5/0x6d0 > 342 Nov 12 10:40:51 ct523c-a0af kernel: process_one_work+0x221/0x6d0 > 343 Nov 12 10:40:51 ct523c-a0af kernel: worker_thread+0x19a/0x340 > 344 Nov 12 10:40:51 ct523c-a0af kernel: ? rescuer_thread+0x450/0x450 > 345 Nov 12 10:40:51 ct523c-a0af kernel: kthread+0x108/0x220 > 346 Nov 12 10:40:51 ct523c-a0af kernel: ? kthreads_online_cpu+0x110/0x110 > 347 Nov 12 10:40:51 ct523c-a0af kernel: ret_from_fork+0x1c6/0x220 > 348 Nov 12 10:40:51 ct523c-a0af kernel: ? kthreads_online_cpu+0x110/0x110 > 349 Nov 12 10:40:51 ct523c-a0af kernel: ret_from_fork_asm+0x11/0x20 > 350 Nov 12 10:40:51 ct523c-a0af kernel: > 351 Nov 12 10:40:51 ct523c-a0af kernel: irq event stamp: 27647 > 352 Nov 12 10:40:51 ct523c-a0af kernel: hardirqs last enabled at (27653): [] __up_console_sem+0x5e/0x70 > 353 Nov 12 10:40:51 ct523c-a0af kernel: hardirqs last disabled at (27658): [] __up_console_sem+0x43/0x70 > 354 Nov 12 10:40:51 ct523c-a0af kernel: softirqs last enabled at (26676): [] napi_pp_put_page+0xac/0xd0 > 355 Nov 12 10:40:51 ct523c-a0af kernel: softirqs last disabled at (26674): [] page_pool_put_unrefed_netmem+0x290/0x3d0 > 356 Nov 12 10:40:51 ct523c-a0af kernel: ---[ end trace 0000000000000000 ]--- > 357 Nov 12 10:40:51 ct523c-a0af kernel: > 358 Nov 12 10:40:51 ct523c-a0af kernel: ============================= > 359 Nov 12 10:40:51 ct523c-a0af kernel: WARNING: suspicious RCU usage > 360 Nov 12 10:40:51 ct523c-a0af kernel: 6.18.0-rc5+ #25 Tainted: G S W > 361 Nov 12 10:40:51 ct523c-a0af kernel: ----------------------------- > 362 Nov 12 10:40:51 ct523c-a0af kernel: drivers/net/wireless/mediatek/mt76/mt7996/main.c:415 suspicious rcu_dereference_check() usage! > 363 Nov 12 10:40:51 ct523c-a0af kernel: > 364 other info that might help us debug this: > 365 Nov 12 10:40:51 ct523c-a0af kernel: > 366 rcu_scheduler_active = 2, debug_locks = 1 > 367 Nov 12 10:40:51 ct523c-a0af kernel: 2 locks held by kworker/u32:4/115: > 368 Nov 12 10:40:51 ct523c-a0af kernel: #0: ffff888118ebdd48 ((wq_completion)phy2){+.+.}-{0:0}, at: process_one_work+0x41c/0x6d0 > 369 Nov 12 10:40:51 ct523c-a0af kernel: #1: ffffc90000f1fe38 > ((work_completion)(&(&dev->scan_work)->work)){+.+.}-{0:0}, at: > process_one_work+ 0x1e5/0x6d0 > 370 Nov 12 10:40:51 ct523c-a0af kernel: > 371 stack backtrace: > 372 Nov 12 10:40:51 ct523c-a0af kernel: CPU: 1 UID: 0 PID: 115 Comm: kworker/u32:4 Tainted: G S W 6.18.0-rc5+ #25 PREEMPT(full) > 373 Nov 12 10:40:51 ct523c-a0af kernel: Tainted: [S]=CPU_OUT_OF_SPEC, [W]=WARN > 374 Nov 12 10:40:51 ct523c-a0af kernel: Hardware name: Default string Default string/SKYBAY, BIOS 5.12 02/15/2023 > 375 Nov 12 10:40:51 ct523c-a0af kernel: Workqueue: phy2 mt76_scan_work [mt76] > 376 Nov 12 10:40:51 ct523c-a0af kernel: Call Trace: > 377 Nov 12 10:40:51 ct523c-a0af kernel: > 378 Nov 12 10:40:51 ct523c-a0af kernel: dump_stack_lvl+0x8d/0xa0 > 379 Nov 12 10:40:51 ct523c-a0af kernel: lockdep_rcu_suspicious+0x146/0x1a0 > 380 Nov 12 10:40:51 ct523c-a0af kernel: mt7996_vif_link_remove+0x279/0x290 [mt7996e] > 381 Nov 12 10:40:51 ct523c-a0af kernel: mt76_put_vif_phy_link+0x41/0x50 [mt76] > 382 Nov 12 10:40:51 ct523c-a0af kernel: mt76_scan_complete+0x77/0x100 [mt76] > 383 Nov 12 10:40:51 ct523c-a0af kernel: mt76_scan_work+0x30a/0x440 [mt76] > 384 Nov 12 10:40:51 ct523c-a0af kernel: ? process_one_work+0x1e5/0x6d0 > 385 Nov 12 10:40:51 ct523c-a0af kernel: process_one_work+0x221/0x6d0 > 386 Nov 12 10:40:51 ct523c-a0af kernel: worker_thread+0x19a/0x340 > 387 Nov 12 10:40:51 ct523c-a0af kernel: ? rescuer_thread+0x450/0x450 > 388 Nov 12 10:40:51 ct523c-a0af kernel: kthread+0x108/0x220 > 389 Nov 12 10:40:51 ct523c-a0af kernel: ? kthreads_online_cpu+0x110/0x110 > 390 Nov 12 10:40:51 ct523c-a0af kernel: ret_from_fork+0x1c6/0x220 > 391 Nov 12 10:40:51 ct523c-a0af kernel: ? kthreads_online_cpu+0x110/0x110 > 392 Nov 12 10:40:51 ct523c-a0af kernel: ret_from_fork_asm+0x11/0x20 > 393 Nov 12 10:40:51 ct523c-a0af kernel: > 394 Nov 12 10:40:54 ct523c-a0af kernel: ixgbe 0000:01:00.0 eth2: NIC Link is Up 10 Gbps, Flow Control: RX/TX > 395 Nov 12 10:40:54 ct523c-a0af kernel: ------------[ cut here ]------------ > > Thanks, > Ben I think the problem here is mt76_put_vif_phy_link() runs without grabbing wiphy mutex in mt76_scan_complete() so we cant' run for_each_vif_active_link(). I will work on a fix. Regards, Lorenzo > > -- > Ben Greear > Candela Technologies Inc http://www.candelatech.com >