* Re: netpoll: raspberrypi [4 5] driver locking woes [not found] ` <f4fa3fcc637ffb6531982a90dbd9c27114e93036.camel@gmx.de> @ 2025-08-28 14:57 ` Sean Anderson 2025-08-28 17:26 ` Mike Galbraith 0 siblings, 1 reply; 4+ messages in thread From: Sean Anderson @ 2025-08-28 14:57 UTC (permalink / raw) To: Mike Galbraith, Robert Hancock; +Cc: Breno Leitao, netdev@vger.kernel.org Hi Mike, On 8/27/25 12:02, Mike Galbraith wrote: > Unexpected addendum to done deal datapoint, so off list. > > On Tue, 2025-08-26 at 11:49 +0200, Mike Galbraith wrote: >> >> The pi5 gripe fix is equally trivial, but submitting that is pointless >> given there's something else amiss in fingered commit. This is all of >> the crash info that escapes the box w/wo gripes silenced. >> >> [ 51.688868] sysrq: Trigger a crash >> [ 51.688892] Kernel panic - not syncing: sysrq triggered crash >> [ 51.698066] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.17.0-rc2-v8-lockdep #533 PREEMPTLAZY >> [ 51.707234] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT) >> [ 51.713085] Call trace: >> [ 51.715532] show_stack+0x20/0x38 (C) >> [ 51.719206] dump_stack_lvl+0x38/0xd0 >> [ 51.722878] dump_stack+0x18/0x28 >> >> That aspect is a punt and run atm (time.. and dash of laziness:). > > Plan was to end datapoint thread, but after booting pi5's 6.12 kernel, > for some reason I fired up netconsole.. and box promptly exhibited the > netpoll locking bug warning, indicating presence of 138badbc21a0. > Instead of saying to self "nope, just walk away", I poked SysRq-C.. and > the bloody damn monitoring box received a 100% complete death rattle. > Well bugger. Did you get a backtrace for this? And to be clear, the steps to reproduce this are to boot a kernel with lockdep enabled with netconsole on macb and then hit sysrq? > Two trees contain locking buglet introducing 138badbc21a0, 6.12.41 and > 6.17.0, but only the later reproduces the above mess. A quick stare at > git diff inspired checking e6a532185daa, and sure enough reverting only > it reproduced a functional netconsole, modulo warnings, ie the mess > above manifests only with BOTH resident. Either revert 138badbc21a0, > or revert e6a532185daa and fix the 138badbc21a0 locking buglet, and the > result is the same, a perfectly functional warning free netconsole. ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: netpoll: raspberrypi [4 5] driver locking woes 2025-08-28 14:57 ` netpoll: raspberrypi [4 5] driver locking woes Sean Anderson @ 2025-08-28 17:26 ` Mike Galbraith 2025-08-28 18:56 ` Sean Anderson 0 siblings, 1 reply; 4+ messages in thread From: Mike Galbraith @ 2025-08-28 17:26 UTC (permalink / raw) To: Sean Anderson, Robert Hancock; +Cc: Breno Leitao, netdev@vger.kernel.org [-- Attachment #1: Type: text/plain, Size: 2449 bytes --] On Thu, 2025-08-28 at 10:57 -0400, Sean Anderson wrote: > Hi Mike, > > On 8/27/25 12:02, Mike Galbraith wrote: > > Unexpected addendum to done deal datapoint, so off list. > > > > On Tue, 2025-08-26 at 11:49 +0200, Mike Galbraith wrote: > > > > > > The pi5 gripe fix is equally trivial, but submitting that is pointless > > > given there's something else amiss in fingered commit. This is all of > > > the crash info that escapes the box w/wo gripes silenced. > > > > > > [ 51.688868] sysrq: Trigger a crash > > > [ 51.688892] Kernel panic - not syncing: sysrq triggered crash > > > [ 51.698066] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.17.0-rc2-v8-lockdep #533 PREEMPTLAZY > > > [ 51.707234] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT) > > > [ 51.713085] Call trace: > > > [ 51.715532] show_stack+0x20/0x38 (C) > > > [ 51.719206] dump_stack_lvl+0x38/0xd0 > > > [ 51.722878] dump_stack+0x18/0x28 > > > > > > That aspect is a punt and run atm (time.. and dash of laziness:). > > > > Plan was to end datapoint thread, but after booting pi5's 6.12 kernel, > > for some reason I fired up netconsole.. and box promptly exhibited the > > netpoll locking bug warning, indicating presence of 138badbc21a0. > > Instead of saying to self "nope, just walk away", I poked SysRq-C.. and > > the bloody damn monitoring box received a 100% complete death rattle. > > Well bugger. > > Did you get a backtrace for this? Yes, logs for 6.12.41 and 6.17.0-rc2 attached. Since a patch has meanwhile landed, also a log of patched 6.17.0-rc3 now gripe free (yay) but with aforementioned broken output, followed by addition of the e6a532185daa revert to confirm it still cures that. > And to be clear, the steps to reproduce this are to boot a kernel with > lockdep enabled with netconsole on macb and then hit sysrq? Yup. > > Two trees contain locking buglet introducing 138badbc21a0, 6.12.41 and > > 6.17.0, but only the later reproduces the above mess. A quick stare at > > git diff inspired checking e6a532185daa, and sure enough reverting only > > it reproduced a functional netconsole, modulo warnings, ie the mess > > above manifests only with BOTH resident. Either revert 138badbc21a0, > > or revert e6a532185daa and fix the 138badbc21a0 locking buglet, and the > > result is the same, a perfectly functional warning free netconsole. > [-- Attachment #2: netconsole-rpi5-6.12-eth0.log --] [-- Type: text/x-log, Size: 18613 bytes --] [ 349.336384] sysrq: Trigger a crash [ 349.336569] ------------[ cut here ]------------ [ 349.336572] netpoll_send_skb_on_dev(): eth0 enabled interrupts in poll (macb_start_xmit+0x0/0xc48) [ 349.336622] WARNING: CPU: 0 PID: 0 at net/core/netpoll.c:385 netpoll_send_skb+0x4e4/0x528 [ 349.336632] Modules linked in: netconsole quota_v2 quota_tree rfcomm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device algif_hash algif_skcipher af_alg bnep snd_soc_pcm512x_i2c snd_soc_pcm512x regmap_i2c spidev brcmfmac_wcc brcmfmac vc4 hci_uart btbcm bluetooth aes_ce_blk snd_soc_hdmi_codec v3d aes_ce_cipher ghash_ce drm_display_helper rpi_hevc_dec cec gpu_sched brcmutil drm_dma_helper drm_shmem_helper v4l2_mem2mem pisp_be gf128mul sha2_ce videobuf2_dma_contig sha256_arm64 ecdh_generic videobuf2_memops cfg80211 snd_soc_hifiberry_dacplus ecc videobuf2_v4l2 raspberrypi_hwmon designware_i2s sha1_ce libaes drm_kms_helper snd_soc_core videodev i2c_brcmstb spi_bcm2835 videobuf2_common rfkill gpio_keys clk_hifiberry_dacpro snd_compress binfmt_misc snd_pcm_dmaengine mc snd_pcm rp1_pio rp1_mailbox snd_timer raspberrypi_gpiomem i2c_designware_platform rp1_fw snd i2c_designware_core rp1_adc sg nvmem_rmem uio_pdrv_genirq uio nfsd drm fuse dm_mod i2c_dev ledtrig_pattern auth_rpcgss drm_panel_orientation_quirks backlight ip_tables [ 349.336710] x_tables ipv6 [ 349.336715] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G W 6.12.41-v8-lockdep #518 [ 349.336721] Tainted: [W]=WARN [ 349.336722] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT) [ 349.336724] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 349.336727] pc : netpoll_send_skb+0x4e4/0x528 [ 349.336731] lr : netpoll_send_skb+0x4e4/0x528 [ 349.336735] sp : ffffffc080003670 [ 349.336737] x29: ffffffc080003670 x28: ffffff810099e600 x27: 0000000000000000 [ 349.336741] x26: 0000000000000080 x25: ffffff812c8fc400 x24: ffffff8102e0c000 [ 349.336745] x23: 00000000000000c0 x22: ffffff81395a9300 x21: ffffffd08de13a28 [ 349.336748] x20: ffffff81183c1130 x19: ffffffd08d94fe40 x18: ffffffffffffffff [ 349.336752] x17: ffffffb1715bb000 x16: ffffffd08d0a9568 x15: 00000000fffffffd [ 349.336755] x14: 0000000000000001 x13: ffffffc080002f94 x12: 6920737470757272 [ 349.336759] x11: fffffffffffe0000 x10: ffffffd08ddfa080 x9 : ffffffd08c545950 [ 349.336762] x8 : 0000000000000001 x7 : ffffffd08dd770a0 x6 : 0000000000000001 [ 349.336766] x5 : ffffffd08dd77000 x4 : 0000000000000001 x3 : 0000000000000000 [ 349.336769] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffffffd08dd7cec0 [ 349.336773] Call trace: [ 349.336774] netpoll_send_skb+0x4e4/0x528 [ 349.336778] netpoll_send_udp+0x258/0x3e8 [ 349.336783] write_msg+0x100/0x138 [netconsole] [ 349.336792] console_flush_all+0x264/0x4a0 [ 349.336798] console_unlock+0x84/0x188 [ 349.336802] vprintk_emit+0x22c/0x420 [ 349.336807] vprintk_default+0x40/0x58 [ 349.336811] vprintk+0x54/0xe0 [ 349.336814] _printk+0x64/0x98 [ 349.336817] __handle_sysrq+0xe0/0x2e0 [ 349.336823] sysrq_filter+0xcc/0x518 [ 349.336827] input_handle_events_filter+0x58/0xc0 [ 349.336831] input_pass_values+0x1e4/0x2c8 [ 349.336836] input_handle_event+0x450/0x558 [ 349.336839] input_event+0x68/0x98 [ 349.336843] hidinput_report_event+0x4c/0x78 [ 349.336848] hid_report_raw_event+0xb4/0x3e0 [ 349.336850] hid_input_report+0x138/0x1a0 [ 349.336853] hid_irq_in+0x180/0x308 [ 349.336858] __usb_hcd_giveback_urb+0x98/0x160 [ 349.336863] usb_giveback_urb_bh+0xb0/0x150 [ 349.336867] process_one_work+0x214/0x640 [ 349.336873] bh_worker+0x1b0/0x218 [ 349.336877] workqueue_softirq_action+0x80/0x98 [ 349.336880] tasklet_hi_action+0x20/0x60 [ 349.336883] handle_softirqs+0x154/0x528 [ 349.336885] __do_softirq+0x1c/0x28 [ 349.336889] ____do_softirq+0x18/0x30 [ 349.336892] call_on_irq_stack+0x30/0x70 [ 349.336895] do_softirq_own_stack+0x24/0x38 [ 349.336898] __irq_exit_rcu+0x100/0x1a0 [ 349.336901] irq_exit_rcu+0x18/0x58 [ 349.336903] el1_interrupt+0x38/0x70 [ 349.336908] el1h_64_irq_handler+0x18/0x28 [ 349.336912] el1h_64_irq+0x64/0x68 [ 349.336914] default_idle_call+0xa8/0x1f8 [ 349.336917] do_idle+0x22c/0x298 [ 349.336920] cpu_startup_entry+0x3c/0x50 [ 349.336923] rest_init+0x100/0x198 [ 349.336925] start_kernel+0x75c/0x900 [ 349.336930] __primary_switched+0x80/0x90 [ 349.336934] irq event stamp: 8607573 [ 349.336935] hardirqs last enabled at (8607572): [<ffffffd08d2c7ce4>] el1_interrupt+0x44/0x70 [ 349.336940] hardirqs last disabled at (8607573): [<ffffffd08d2c72f4>] el1_dbg+0x24/0x98 [ 349.336944] softirqs last enabled at (8607524): [<ffffffd08c4b04c8>] handle_softirqs+0x508/0x528 [ 349.336947] softirqs last disabled at (8607545): [<ffffffd08c4102ac>] __do_softirq+0x1c/0x28 [ 349.336949] ---[ end trace 0000000000000000 ]--- [ 349.340398] ------------[ cut here ]------------ [ 349.340399] WARNING: CPU: 0 PID: 0 at mm/slub.c:700 get_freelist+0x1bc/0x2b0 [ 349.340405] Modules linked in: netconsole quota_v2 quota_tree rfcomm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device algif_hash algif_skcipher af_alg bnep snd_soc_pcm512x_i2c snd_soc_pcm512x regmap_i2c spidev brcmfmac_wcc brcmfmac vc4 hci_uart btbcm bluetooth aes_ce_blk snd_soc_hdmi_codec v3d aes_ce_cipher ghash_ce drm_display_helper rpi_hevc_dec cec gpu_sched brcmutil drm_dma_helper drm_shmem_helper v4l2_mem2mem pisp_be gf128mul sha2_ce videobuf2_dma_contig sha256_arm64 ecdh_generic videobuf2_memops cfg80211 snd_soc_hifiberry_dacplus ecc videobuf2_v4l2 raspberrypi_hwmon designware_i2s sha1_ce libaes drm_kms_helper snd_soc_core videodev i2c_brcmstb spi_bcm2835 videobuf2_common rfkill gpio_keys clk_hifiberry_dacpro snd_compress binfmt_misc snd_pcm_dmaengine mc snd_pcm rp1_pio rp1_mailbox snd_timer raspberrypi_gpiomem i2c_designware_platform rp1_fw snd i2c_designware_core rp1_adc sg nvmem_rmem uio_pdrv_genirq uio nfsd drm fuse dm_mod i2c_dev ledtrig_pattern auth_rpcgss drm_panel_orientation_quirks backlight ip_tables [ 349.340461] x_tables ipv6 [ 349.340464] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G W 6.12.41-v8-lockdep #518 [ 349.340467] Tainted: [W]=WARN [ 349.340468] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT) [ 349.340469] pstate: 604000c9 (nZCv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 349.340472] pc : get_freelist+0x1bc/0x2b0 [ 349.340475] lr : get_freelist+0x290/0x2b0 [ 349.340477] sp : ffffffc080003460 [ 349.340478] x29: ffffffc080003460 x28: 0000000000000000 x27: 0000000000000001 [ 349.340482] x26: ffffffd08dd77000 x25: ffffffd08d93c448 x24: ffffff81001cec00 [ 349.340485] x23: ffffffd08dd7cec0 x22: 0000000000100010 x21: 0000000000000000 [ 349.340489] x20: 0000000080100010 x19: fffffffec4e56a40 x18: ffffffffffffffff [ 349.340492] x17: ffffffb1715bb000 x16: ffffffd08d0a9568 x15: 0000000000000006 [ 349.340496] x14: 00000000000c2000 x13: 2d2d2d2d2d2d2d2d x12: 0000000000000000 [ 349.340499] x11: 0000000000000190 x10: 0000000000000000 x9 : ffffffd08e70ab10 [ 349.340503] x8 : ffffffd08dd7cec0 x7 : 0000000001298354 x6 : ffffffd08e6d4080 [ 349.340506] x5 : 00000000000000c8 x4 : 0000000000040000 x3 : ffffffd08dd7cec0 [ 349.340510] x2 : ffffffb1715bb000 x1 : ffffffd08dd7cec0 x0 : 0000000000000105 [ 349.340513] Call trace: [ 349.340514] get_freelist+0x1bc/0x2b0 [ 349.340516] ___slab_alloc+0xf4/0x10c0 [ 349.340519] __slab_alloc.constprop.0+0x6c/0xc8 [ 349.340522] kmem_cache_alloc_node_noprof+0x2e4/0x348 [ 349.340525] __alloc_skb+0x154/0x188 [ 349.340529] netpoll_send_udp+0xa8/0x3e8 [ 349.340533] write_msg+0x100/0x138 [netconsole] [ 349.340538] console_flush_all+0x264/0x4a0 [ 349.340542] console_unlock+0x84/0x188 [ 349.340546] vprintk_emit+0x22c/0x420 [ 349.340551] vprintk_default+0x40/0x58 [ 349.340555] vprintk+0x54/0xe0 [ 349.340557] _printk+0x64/0x98 [ 349.340560] __handle_sysrq+0xe0/0x2e0 [ 349.340565] sysrq_filter+0xcc/0x518 [ 349.340569] input_handle_events_filter+0x58/0xc0 [ 349.340572] input_pass_values+0x1e4/0x2c8 [ 349.340576] input_handle_event+0x450/0x558 [ 349.340580] input_event+0x68/0x98 [ 349.340583] hidinput_report_event+0x4c/0x78 [ 349.340587] hid_report_raw_event+0xb4/0x3e0 [ 349.340589] hid_input_report+0x138/0x1a0 [ 349.340592] hid_irq_in+0x180/0x308 [ 349.340596] __usb_hcd_giveback_urb+0x98/0x160 [ 349.340600] usb_giveback_urb_bh+0xb0/0x150 [ 349.340604] process_one_work+0x214/0x640 [ 349.340608] bh_worker+0x1b0/0x218 [ 349.340613] workqueue_softirq_action+0x80/0x98 [ 349.340615] tasklet_hi_action+0x20/0x60 [ 349.340618] handle_softirqs+0x154/0x528 [ 349.340620] __do_softirq+0x1c/0x28 [ 349.340622] ____do_softirq+0x18/0x30 [ 349.340625] call_on_irq_stack+0x30/0x70 [ 349.340628] do_softirq_own_stack+0x24/0x38 [ 349.340631] __irq_exit_rcu+0x100/0x1a0 [ 349.340634] irq_exit_rcu+0x18/0x58 [ 349.340636] el1_interrupt+0x38/0x70 [ 349.340640] el1h_64_irq_handler+0x18/0x28 [ 349.340644] el1h_64_irq+0x64/0x68 [ 349.340646] default_idle_call+0xa8/0x1f8 [ 349.340648] do_idle+0x22c/0x298 [ 349.340650] cpu_startup_entry+0x3c/0x50 [ 349.340652] rest_init+0x100/0x198 [ 349.340655] start_kernel+0x75c/0x900 [ 349.340658] __primary_switched+0x80/0x90 [ 349.340661] irq event stamp: 8607577 [ 349.340662] hardirqs last enabled at (8607576): [<ffffffd08d2c7ce4>] el1_interrupt+0x44/0x70 [ 349.340666] hardirqs last disabled at (8607577): [<ffffffd08d2c72f4>] el1_dbg+0x24/0x98 [ 349.340670] softirqs last enabled at (8607524): [<ffffffd08c4b04c8>] handle_softirqs+0x508/0x528 [ 349.340673] softirqs last disabled at (8607545): [<ffffffd08c4102ac>] __do_softirq+0x1c/0x28 [ 349.340676] ---[ end trace 0000000000000000 ]--- [ 349.340679] [ 349.340680] ================================ [ 349.340681] WARNING: inconsistent lock state [ 349.340682] 6.12.41-v8-lockdep #518 Tainted: G W [ 349.340684] -------------------------------- [ 349.340684] inconsistent {IN-HARDIRQ-W} -> {HARDIRQ-ON-W} usage. [ 349.340686] swapper/0/0 [HC0[0]:SC1[1]:HE0:SE0] takes: [ 349.340688] ffffff81fef12fc0 (&c->lock){?.-.}-{2:2}, at: ___slab_alloc+0x78/0x10c0 [ 349.340695] {IN-HARDIRQ-W} state was registered at: [ 349.340697] lock_acquire+0x208/0x360 [ 349.340700] put_cpu_partial+0x88/0x258 [ 349.340703] __slab_free+0x1c8/0x390 [ 349.340705] kmem_cache_free+0x29c/0x400 [ 349.340708] mempool_free_slab+0x24/0x38 [ 349.340712] mempool_free+0x48/0xb8 [ 349.340715] bio_free+0x70/0x90 [ 349.340719] bio_put+0x120/0x278 [ 349.340722] end_bio_bh_io_sync+0x44/0x88 [ 349.340725] bio_endio+0x18c/0x288 [ 349.340728] blk_mq_end_request_batch+0x274/0x500 [ 349.340731] nvme_pci_complete_batch+0xe0/0x138 [ 349.340734] nvme_irq+0x78/0x90 [ 349.340736] __handle_irq_event_percpu+0x94/0x308 [ 349.340740] handle_irq_event+0x54/0xb8 [ 349.340743] handle_fasteoi_irq+0xc8/0x268 [ 349.340747] handle_irq_desc+0x48/0x68 [ 349.340750] generic_handle_domain_irq+0x24/0x38 [ 349.340753] gic_handle_irq+0x48/0xd8 [ 349.340755] call_on_irq_stack+0x30/0x70 [ 349.340758] do_interrupt_handler+0xdc/0xe8 [ 349.340761] el1_interrupt+0x34/0x70 [ 349.340765] el1h_64_irq_handler+0x18/0x28 [ 349.340769] el1h_64_irq+0x64/0x68 [ 349.340771] default_idle_call+0xa8/0x1f8 [ 349.340773] do_idle+0x22c/0x298 [ 349.340775] cpu_startup_entry+0x3c/0x50 [ 349.340777] secondary_start_kernel+0x128/0x150 [ 349.340781] __secondary_switched+0xb8/0xc0 [ 349.340785] irq event stamp: 8607577 [ 349.340786] hardirqs last enabled at (8607576): [<ffffffd08d2c7ce4>] el1_interrupt+0x44/0x70 [ 349.340790] hardirqs last disabled at (8607577): [<ffffffd08d2c72f4>] el1_dbg+0x24/0x98 [ 349.340794] softirqs last enabled at (8607524): [<ffffffd08c4b04c8>] handle_softirqs+0x508/0x528 [ 349.340797] softirqs last disabled at (8607545): [<ffffffd08c4102ac>] __do_softirq+0x1c/0x28 [ 349.340800] [ 349.340800] other info that might help us debug this: [ 349.340801] Possible unsafe locking scenario: [ 349.340801] [ 349.340801] CPU0 [ 349.340802] ---- [ 349.340803] lock(&c->lock); [ 349.340805] <Interrupt> [ 349.340806] lock(&c->lock); [ 349.340807] [ 349.340807] *** DEADLOCK *** [ 349.340807] [ 349.340808] 11 locks held by swapper/0/0: [ 349.340810] #0: ffffff8100045548 ((wq_completion)events_bh_highpri){+.-.}-{0:0}, at: process_one_work+0x198/0x640 [ 349.340818] #1: ffffffc080003e60 ((work_completion)(&bh->bh)){+.-.}-{0:0}, at: process_one_work+0x1c0/0x640 [ 349.340826] #2: ffffff8102eaa230 (&dev->event_lock){+.-.}-{2:2}, at: input_event+0x50/0x98 [ 349.340833] #3: ffffffd08de13a28 (rcu_read_lock){....}-{1:2}, at: input_pass_values+0x38/0x2c8 [ 349.340841] #4: ffffffd08de13a28 (rcu_read_lock){....}-{1:2}, at: __handle_sysrq+0x8/0x2e0 [ 349.340848] #5: ffffffd08dda1c80 (console_lock){+.+.}-{0:0}, at: vprintk_default+0x40/0x58 [ 349.340856] #6: ffffffd08dda1cd8 (console_srcu){....}-{0:0}, at: console_flush_all+0x8c/0x4a0 [ 349.340863] #7: ffffffd08dda1e98 (console_owner){+.-.}-{0:0}, at: console_lock_spinning_enable+0x50/0x90 [ 349.340871] #8: ffffffd08dda1ec0 (printk_legacy_map-wait-type-override){+...}-{3:3}, at: console_flush_all+0x220/0x4a0 [ 349.340878] #9: ffffffd02cbab220 (target_list_lock){+.-.}-{2:2}, at: write_msg+0x64/0x138 [netconsole] [ 349.340886] #10: ffffff81fef12fc0 (&c->lock){?.-.}-{2:2}, at: ___slab_alloc+0x78/0x10c0 [ 349.340892] [ 349.340892] stack backtrace: [ 349.340893] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G W 6.12.41-v8-lockdep #518 [ 349.340897] Tainted: [W]=WARN [ 349.340898] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT) [ 349.340899] Call trace: [ 349.340900] dump_backtrace+0xa0/0x100 [ 349.340902] show_stack+0x20/0x38 [ 349.340904] dump_stack_lvl+0x90/0xd0 [ 349.340909] dump_stack+0x18/0x28 [ 349.340912] print_usage_bug.part.0+0x29c/0x368 [ 349.340915] mark_lock+0x724/0x928 [ 349.340917] mark_held_locks+0x54/0x90 [ 349.340919] lockdep_hardirqs_on_prepare+0x108/0x228 [ 349.340921] arm64_exit_el1_dbg.isra.0+0x30/0x58 [ 349.340925] el1_dbg+0x84/0x98 [ 349.340928] el1h_64_sync_handler+0xc8/0xe8 [ 349.340932] el1h_64_sync+0x64/0x68 [ 349.340934] get_freelist+0x1bc/0x2b0 [ 349.340936] ___slab_alloc+0xf4/0x10c0 [ 349.340939] __slab_alloc.constprop.0+0x6c/0xc8 [ 349.340942] kmem_cache_alloc_node_noprof+0x2e4/0x348 [ 349.340945] __alloc_skb+0x154/0x188 [ 349.340947] netpoll_send_udp+0xa8/0x3e8 [ 349.340952] write_msg+0x100/0x138 [netconsole] [ 349.340955] console_flush_all+0x264/0x4a0 [ 349.340960] console_unlock+0x84/0x188 [ 349.340964] vprintk_emit+0x22c/0x420 [ 349.340968] vprintk_default+0x40/0x58 [ 349.340972] vprintk+0x54/0xe0 [ 349.340974] _printk+0x64/0x98 [ 349.340977] __handle_sysrq+0xe0/0x2e0 [ 349.340981] sysrq_filter+0xcc/0x518 [ 349.340986] input_handle_events_filter+0x58/0xc0 [ 349.340989] input_pass_values+0x1e4/0x2c8 [ 349.340992] input_handle_event+0x450/0x558 [ 349.340996] input_event+0x68/0x98 [ 349.341000] hidinput_report_event+0x4c/0x78 [ 349.341003] hid_report_raw_event+0xb4/0x3e0 [ 349.341006] hid_input_report+0x138/0x1a0 [ 349.341008] hid_irq_in+0x180/0x308 [ 349.341012] __usb_hcd_giveback_urb+0x98/0x160 [ 349.341016] usb_giveback_urb_bh+0xb0/0x150 [ 349.341020] process_one_work+0x214/0x640 [ 349.341025] bh_worker+0x1b0/0x218 [ 349.341029] workqueue_softirq_action+0x80/0x98 [ 349.341031] tasklet_hi_action+0x20/0x60 [ 349.341034] handle_softirqs+0x154/0x528 [ 349.341036] __do_softirq+0x1c/0x28 [ 349.341038] ____do_softirq+0x18/0x30 [ 349.341041] call_on_irq_stack+0x30/0x70 [ 349.341044] do_softirq_own_stack+0x24/0x38 [ 349.341047] __irq_exit_rcu+0x100/0x1a0 [ 349.341049] irq_exit_rcu+0x18/0x58 [ 349.341052] el1_interrupt+0x38/0x70 [ 349.341056] el1h_64_irq_handler+0x18/0x28 [ 349.341059] el1h_64_irq+0x64/0x68 [ 349.341062] default_idle_call+0xa8/0x1f8 [ 349.341064] do_idle+0x22c/0x298 [ 349.341066] cpu_startup_entry+0x3c/0x50 [ 349.341068] rest_init+0x100/0x198 [ 349.341070] start_kernel+0x75c/0x900 [ 349.341073] __primary_switched+0x80/0x90 [ 350.513730] v3d 1002000000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang. [ 350.517765] Kernel panic - not syncing: sysrq triggered crash [ 350.517769] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G W 6.12.41-v8-lockdep #518 [ 350.517773] Tainted: [W]=WARN [ 350.517774] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT) [ 350.813441] Call trace: [ 350.815900] dump_backtrace+0xa0/0x100 [ 350.819671] show_stack+0x20/0x38 [ 350.823004] dump_stack_lvl+0x38/0xd0 [ 350.826688] dump_stack+0x18/0x28 [ 350.830023] panic+0x3c8/0x430 [ 350.833096] sysrq_handle_crash+0x40/0x90 [ 350.837130] __handle_sysrq+0xf0/0x2e0 [ 350.840901] sysrq_filter+0xcc/0x518 [ 350.844498] input_handle_events_filter+0x58/0xc0 [ 350.849228] input_pass_values+0x1e4/0x2c8 [ 350.853348] input_handle_event+0x450/0x558 [ 350.857554] input_event+0x68/0x98 [ 350.860976] hidinput_report_event+0x4c/0x78 [ 350.865271] hid_report_raw_event+0xb4/0x3e0 [ 350.869564] hid_input_report+0x138/0x1a0 [ 350.873595] hid_irq_in+0x180/0x308 [ 350.877105] __usb_hcd_giveback_urb+0x98/0x160 [ 350.881575] usb_giveback_urb_bh+0xb0/0x150 [ 350.885782] process_one_work+0x214/0x640 [ 350.889815] bh_worker+0x1b0/0x218 [ 350.893238] workqueue_softirq_action+0x80/0x98 [ 350.897793] tasklet_hi_action+0x20/0x60 [ 350.901737] handle_softirqs+0x154/0x528 [ 350.905681] __do_softirq+0x1c/0x28 [ 350.909188] ____do_softirq+0x18/0x30 [ 350.912871] call_on_irq_stack+0x30/0x70 [ 350.916815] do_softirq_own_stack+0x24/0x38 [ 350.921023] __irq_exit_rcu+0x100/0x1a0 [ 350.924880] irq_exit_rcu+0x18/0x58 [ 350.928388] el1_interrupt+0x38/0x70 [ 350.931985] el1h_64_irq_handler+0x18/0x28 [ 350.936105] el1h_64_irq+0x64/0x68 [ 350.939529] default_idle_call+0xa8/0x1f8 [ 350.943561] do_idle+0x22c/0x298 [ 350.946807] cpu_startup_entry+0x3c/0x50 [ 350.950751] rest_init+0x100/0x198 [ 350.954170] start_kernel+0x75c/0x900 [ 350.957855] __primary_switched+0x80/0x90 [ 350.961888] SMP: stopping secondary CPUs [ 352.035165] SMP: failed to stop secondary CPUs 1 [ 352.039823] Kernel Offset: 0x100c400000 from 0xffffffc080000000 [ 352.045770] PHYS_OFFSET: 0x0 [ 352.048666] CPU features: 0x10,00000017,00280928,0200720b [ 352.054093] Memory Limit: none [ 352.057167] ---[ end Kernel panic - not syncing: sysrq triggered crash ]--- [-- Attachment #3: netconsole-rpi5-6.17-eth0.log --] [-- Type: text/x-log, Size: 17135 bytes --] [ 58.750302] sysrq: HELP : loglevel(0-9) reboot(b) crash(c) show-all-locks(d) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) sak(k) show-backtrace-all-active-cpus(l) show-memory-usage(m) nice-all-RT-tasks(n) poweroff(o) show-registers(p) show-all-timers(q) unraw(r) sync(s) show-task-states(t) unmount(u) force-fb(v) show-blocked-tasks(w) dump-ftrace-buffer(z) replay-kernel-logs(R) [ 58.750681] ------------[ cut here ]------------ [ 58.750690] netpoll_send_skb_on_dev(): eth0 enabled interrupts in poll (macb_start_xmit+0x0/0xd30) [ 58.750730] WARNING: CPU: 0 PID: 0 at net/core/netpoll.c:359 netpoll_send_skb+0x46c/0x4b0 [ 58.816326] Modules linked in: netconsole quota_v2 quota_tree rfcomm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device algif_hash algif_skcipher af_alg bnep snd_soc_pcm512x_i2c snd_soc_pcm512x spidev regmap_i2c binfmt_misc hci_uart aes_ce_blk btbcm bluetooth ecdh_generic ecc aes_ce_cipher ghash_ce brcmfmac_cyw gf128mul snd_soc_hifiberry_dacplus sg raspberrypi_hwmon brcmfmac brcmutil i2c_brcmstb cfg80211 vc4 rfkill spi_bcm2835 rpi_hevc_dec snd_soc_hdmi_codec gpio_keys drm_exec drm_display_helper clk_hifiberry_dacpro cec pisp_be v4l2_mem2mem v3d videobuf2_dma_contig drm_dma_helper gpu_sched drm_shmem_helper drm_client_lib rp1_pci videobuf2_memops videobuf2_v4l2 videodev designware_i2s drm_kms_helper videobuf2_common mc snd_soc_core snd_compress i2c_designware_platform rp1_pio snd_pcm_dmaengine rp1_fw i2c_designware_core snd_pcm rp1_mailbox raspberrypi_gpiomem snd_timer rp1_adc snd nvmem_rmem uio_pdrv_genirq uio i2c_dev ledtrig_pattern drm fuse drm_panel_orientation_quirks backlight dm_mod nfsd auth_rpcgss ip_tables [ 58.816415] x_tables ipv6 [ 58.909197] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.17.0-rc2-v8-lockdep #532 PREEMPTLAZY [ 58.918365] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT) [ 58.924216] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 58.931202] pc : netpoll_send_skb+0x46c/0x4b0 [ 58.935571] lr : netpoll_send_skb+0x46c/0x4b0 [ 58.939939] sp : ffffffc080003690 [ 58.943258] x29: ffffffc080003690 x28: 0000000000000000 x27: ffffff8101151800 [ 58.950421] x26: ffffff8137fb6268 x25: 0000000000000080 x24: ffffff8137fb6200 [ 58.957583] x23: ffffff8102d58000 x22: 00000000000000c0 x21: ffffff8104f7e100 [ 58.964745] x20: ffffffd08de9c3c8 x19: ffffff8137f2d148 x18: 00000000ffffffff [ 58.971907] x17: ffffffb1714fe000 x16: ffffffd08d124020 x15: 00000000fffffffd [ 58.979069] x14: 0000000000000001 x13: 000000000002a000 x12: 0000000000008000 [ 58.986231] x11: fffffffffffe0000 x10: ffffffd08e78f000 x9 : ffffffd08c5596f0 [ 58.993393] x8 : ffffffffffffffff x7 : ffffffd08de07ef8 x6 : 0000000000000001 [ 59.000555] x5 : ffffffd08de07000 x4 : 0000000000000001 x3 : 0000000000000001 [ 59.007717] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffffffd08de0dd40 [ 59.014879] Call trace: [ 59.017326] netpoll_send_skb+0x46c/0x4b0 (P) [ 59.021695] netpoll_send_udp+0x2ac/0x418 [ 59.025715] write_msg+0x124/0x1b0 [netconsole] [ 59.030262] console_flush_all+0x2e4/0x4a8 [ 59.034371] console_unlock+0x84/0x188 [ 59.038128] vprintk_emit+0x24c/0x450 [ 59.041798] vprintk_default+0x40/0x58 [ 59.045555] vprintk+0x3c/0x80 [ 59.048615] _printk+0x64/0x98 [ 59.051674] __handle_sysrq+0x1ec/0x308 [ 59.055519] sysrq_filter+0xcc/0x518 [ 59.059101] input_handle_events_filter+0x58/0xc0 [ 59.063818] input_pass_values+0x1fc/0x2e8 [ 59.067924] input_handle_event+0x450/0x558 [ 59.072118] input_event+0x6c/0xb8 [ 59.075525] hidinput_report_event+0x4c/0x78 [ 59.079806] hid_report_raw_event+0xb4/0x418 [ 59.084086] hid_input_report+0x138/0x1a0 [ 59.088104] hid_irq_in+0x180/0x308 [ 59.091600] __usb_hcd_giveback_urb+0x98/0x160 [ 59.096055] usb_giveback_urb_bh+0xb0/0x150 [ 59.100248] process_one_work+0x20c/0x5e8 [ 59.104269] bh_worker+0x1b0/0x218 [ 59.107677] workqueue_softirq_action+0x80/0x98 [ 59.112220] tasklet_hi_action+0x20/0x60 [ 59.116153] handle_softirqs+0x144/0x528 [ 59.120085] __do_softirq+0x1c/0x28 [ 59.123581] ____do_softirq+0x18/0x30 [ 59.127250] call_on_irq_stack+0x30/0x70 [ 59.131181] do_softirq_own_stack+0x24/0x38 [ 59.135374] __irq_exit_rcu+0x100/0x1f8 [ 59.139219] irq_exit_rcu+0x18/0x58 [ 59.142714] el1_interrupt+0x38/0x68 [ 59.146297] el1h_64_irq_handler+0x18/0x28 [ 59.150402] el1h_64_irq+0x6c/0x70 [ 59.153809] default_idle_call+0x84/0x140 (P) [ 59.158177] do_idle+0x22c/0x298 [ 59.161412] cpu_startup_entry+0x40/0x50 [ 59.165344] rest_init+0x108/0x1a0 [ 59.168752] start_kernel+0x760/0x8c8 [ 59.172424] __primary_switched+0x88/0x98 [ 59.176444] irq event stamp: 279989 [ 59.179938] hardirqs last enabled at (279988): [<ffffffd08d350b64>] el1_interrupt+0x54/0x68 [ 59.188408] hardirqs last disabled at (279989): [<ffffffd08d34fc94>] el1_brk64+0x1c/0x50 [ 59.196530] softirqs last enabled at (279934): [<ffffffd08c4c2900>] handle_softirqs+0x508/0x528 [ 59.205349] softirqs last disabled at (279959): [<ffffffd08c4103ac>] __do_softirq+0x1c/0x28 [ 59.213731] ---[ end trace 0000000000000000 ]--- [ 59.218386] ------------[ cut here ]------------ [ 59.218388] WARNING: CPU: 0 PID: 0 at mm/slub.c:735 get_freelist+0x160/0x1d8 [ 59.230093] Modules linked in: netconsole quota_v2 quota_tree rfcomm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device algif_hash algif_skcipher af_alg bnep snd_soc_pcm512x_i2c snd_soc_pcm512x spidev regmap_i2c binfmt_misc hci_uart aes_ce_blk btbcm bluetooth ecdh_generic ecc aes_ce_cipher ghash_ce brcmfmac_cyw gf128mul snd_soc_hifiberry_dacplus sg raspberrypi_hwmon brcmfmac brcmutil i2c_brcmstb cfg80211 vc4 rfkill spi_bcm2835 rpi_hevc_dec snd_soc_hdmi_codec gpio_keys drm_exec drm_display_helper clk_hifiberry_dacpro cec pisp_be v4l2_mem2mem v3d videobuf2_dma_contig drm_dma_helper gpu_sched drm_shmem_helper drm_client_lib rp1_pci videobuf2_memops videobuf2_v4l2 videodev designware_i2s drm_kms_helper videobuf2_common mc snd_soc_core snd_compress i2c_designware_platform rp1_pio snd_pcm_dmaengine rp1_fw i2c_designware_core snd_pcm rp1_mailbox raspberrypi_gpiomem snd_timer rp1_adc snd nvmem_rmem uio_pdrv_genirq uio i2c_dev ledtrig_pattern drm fuse drm_panel_orientation_quirks backlight dm_mod nfsd auth_rpcgss ip_tables [ 59.230172] x_tables ipv6 [ 59.322954] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G W 6.17.0-rc2-v8-lockdep #532 PREEMPTLAZY [ 59.333694] Tainted: [W]=WARN [ 59.336664] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT) [ 59.342514] pstate: a04000c9 (NzCv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 59.349500] pc : get_freelist+0x160/0x1d8 [ 59.353519] lr : get_freelist+0x1c0/0x1d8 [ 59.357538] sp : ffffffc080003470 [ 59.360857] x29: ffffffc080003470 x28: 0000000000000001 x27: ffffffd08de07000 [ 59.368019] x26: ffffffd08d9ef508 x25: ffffff8100a3c240 x24: ffffffd08de0dd40 [ 59.375181] x23: fffffffec413dfa0 x22: 0000000080100010 x21: fffffffec413df80 [ 59.382343] x20: ffffff8104f7ef00 x19: 000000008010000e x18: 00000000ffffffff [ 59.389504] x17: ffffffb1714fe000 x16: ffffffd08d124020 x15: 0000000000000006 [ 59.396666] x14: 0000000000000000 x13: 00000000000c2000 x12: 0000000000000000 [ 59.403828] x11: 656820747563205b x10: ffffffd08e760810 x9 : ffffffd08e079000 [ 59.410990] x8 : ffffffd08de0dd40 x7 : 00000000eb9a37bf x6 : 0000000000000190 [ 59.418152] x5 : 00000000000000c8 x4 : 0000000000040000 x3 : 0000000000000001 [ 59.425314] x2 : ffffffb1714fe000 x1 : ffffffd08de0dd40 x0 : 0000000000000107 [ 59.432476] Call trace: [ 59.434923] get_freelist+0x160/0x1d8 (P) [ 59.438942] ___slab_alloc+0xf8/0x1178 [ 59.442699] __slab_alloc.constprop.0+0x6c/0xc8 [ 59.447242] kmem_cache_alloc_node_noprof+0x2c4/0x328 [ 59.452309] __alloc_skb+0x154/0x188 [ 59.455893] netpoll_send_udp+0xac/0x418 [ 59.459826] write_msg+0x124/0x1b0 [netconsole] [ 59.464370] console_flush_all+0x2e4/0x4a8 [ 59.468476] console_unlock+0x84/0x188 [ 59.472233] vprintk_emit+0x24c/0x450 [ 59.475903] vprintk_default+0x40/0x58 [ 59.479661] vprintk+0x3c/0x80 [ 59.482719] _printk+0x64/0x98 [ 59.485778] __handle_sysrq+0x1ec/0x308 [ 59.489622] sysrq_filter+0xcc/0x518 [ 59.493204] input_handle_events_filter+0x58/0xc0 [ 59.497921] input_pass_values+0x1fc/0x2e8 [ 59.502027] input_handle_event+0x450/0x558 [ 59.506220] input_event+0x6c/0xb8 [ 59.509627] hidinput_report_event+0x4c/0x78 [ 59.513907] hid_report_raw_event+0xb4/0x418 [ 59.518188] hid_input_report+0x138/0x1a0 [ 59.522206] hid_irq_in+0x180/0x308 [ 59.525700] __usb_hcd_giveback_urb+0x98/0x160 [ 59.530155] usb_giveback_urb_bh+0xb0/0x150 [ 59.534348] process_one_work+0x20c/0x5e8 [ 59.538368] bh_worker+0x1b0/0x218 [ 59.541776] workqueue_softirq_action+0x80/0x98 [ 59.546319] tasklet_hi_action+0x20/0x60 [ 59.550251] handle_softirqs+0x144/0x528 [ 59.554183] __do_softirq+0x1c/0x28 [ 59.557677] ____do_softirq+0x18/0x30 [ 59.561347] call_on_irq_stack+0x30/0x70 [ 59.565278] do_softirq_own_stack+0x24/0x38 [ 59.569471] __irq_exit_rcu+0x100/0x1f8 [ 59.573316] irq_exit_rcu+0x18/0x58 [ 59.576811] el1_interrupt+0x38/0x68 [ 59.580393] el1h_64_irq_handler+0x18/0x28 [ 59.584498] el1h_64_irq+0x6c/0x70 [ 59.587905] default_idle_call+0x84/0x140 (P) [ 59.592273] do_idle+0x22c/0x298 [ 59.595507] cpu_startup_entry+0x40/0x50 [ 59.599438] rest_init+0x108/0x1a0 [ 59.602846] start_kernel+0x760/0x8c8 [ 59.606517] __primary_switched+0x88/0x98 [ 59.610536] irq event stamp: 279993 [ 59.614030] hardirqs last enabled at (279992): [<ffffffd08d350b64>] el1_interrupt+0x54/0x68 [ 59.622499] hardirqs last disabled at (279993): [<ffffffd08d34fc94>] el1_brk64+0x1c/0x50 [ 59.630620] softirqs last enabled at (279934): [<ffffffd08c4c2900>] handle_softirqs+0x508/0x528 [ 59.639440] softirqs last disabled at (279959): [<ffffffd08c4103ac>] __do_softirq+0x1c/0x28 [ 59.647822] ---[ end trace 0000000000000000 ]--- [ 59.652452] [ 59.653938] ================================ [ 59.658217] WARNING: inconsistent lock state [ 59.662496] 6.17.0-rc2-v8-lockdep #532 Tainted: G W [ 59.668869] -------------------------------- [ 59.673148] inconsistent {IN-HARDIRQ-W} -> {HARDIRQ-ON-W} usage. [ 59.679172] swapper/0/0 [HC0[0]:SC1[1]:HE0:SE0] takes: [ 59.684325] ffffff81fef08f80 (&c->lock){?.-.}-{3:3}, at: ___slab_alloc+0x7c/0x1178 [ 59.691926] {IN-HARDIRQ-W} state was registered at: [ 59.696815] lock_acquire+0x1d4/0x368 [ 59.700573] put_cpu_partial+0x88/0x258 [ 59.704504] __slab_free+0x250/0x420 [ 59.708173] kmem_cache_free+0x288/0x3e8 [ 59.712192] mempool_free_slab+0x24/0x38 [ 59.716211] mempool_free+0x4c/0x120 [ 59.719881] bio_free+0x70/0x90 [ 59.723114] bio_put+0x120/0x278 [ 59.726433] end_bio_bh_io_sync+0x44/0x88 [ 59.730538] bio_endio+0x19c/0x280 [ 59.734032] blk_mq_end_request_batch+0x250/0x4c0 [ 59.738837] nvme_pci_complete_batch+0xdc/0x138 [ 59.743467] nvme_irq+0x7c/0x90 [ 59.746700] __handle_irq_event_percpu+0x94/0x318 [ 59.751503] handle_irq_event+0x54/0xb8 [ 59.755433] handle_fasteoi_irq+0x12c/0x270 [ 59.759714] handle_irq_desc+0x48/0x68 [ 59.763557] generic_handle_domain_irq+0x24/0x38 [ 59.768274] gic_handle_irq+0x48/0xd8 [ 59.772031] call_on_irq_stack+0x30/0x70 [ 59.776051] do_interrupt_handler+0xdc/0xe8 [ 59.780333] el1_interrupt+0x34/0x68 [ 59.784002] el1h_64_irq_handler+0x18/0x28 [ 59.788195] el1h_64_irq+0x6c/0x70 [ 59.791690] default_idle_call+0x84/0x140 [ 59.795796] do_idle+0x22c/0x298 [ 59.799119] cpu_startup_entry+0x3c/0x50 [ 59.803139] secondary_start_kernel+0x12c/0x158 [ 59.807772] __secondary_switched+0xc0/0xc8 [ 59.812054] irq event stamp: 279993 [ 59.815548] hardirqs last enabled at (279992): [<ffffffd08d350b64>] el1_interrupt+0x54/0x68 [ 59.824018] hardirqs last disabled at (279993): [<ffffffd08d34fc94>] el1_brk64+0x1c/0x50 [ 59.832141] softirqs last enabled at (279934): [<ffffffd08c4c2900>] handle_softirqs+0x508/0x528 [ 59.840963] softirqs last disabled at (279959): [<ffffffd08c4103ac>] __do_softirq+0x1c/0x28 [ 59.849345] [ 59.849345] other info that might help us debug this: [ 59.855894] Possible unsafe locking scenario: [ 59.855894] [ 59.861831] CPU0 [ 59.864278] ---- [ 59.866724] lock(&c->lock); [ 59.869696] <Interrupt> [ 59.872317] lock(&c->lock); [ 59.875464] [ 59.875464] *** DEADLOCK *** [ 59.875464] [ 59.881402] 11 locks held by swapper/0/0: [ 59.885420] #0: ffffff8100049948 ((wq_completion)events_bh_highpri){+.-.}-{0:0}, at: process_one_work+0x190/0x5e8 [ 59.895818] #1: ffffffc080003e60 ((work_completion)(&bh->bh)){+.-.}-{0:0}, at: process_one_work+0x1b8/0x5e8 [ 59.905691] #2: ffffff81034d3a30 (&dev->event_lock){+.-.}-{3:3}, at: input_event+0x50/0xb8 [ 59.914080] #3: ffffffd08de9c3c8 (rcu_read_lock){....}-{1:3}, at: input_pass_values+0x38/0x2e8 [ 59.922818] #4: ffffffd08de9c3c8 (rcu_read_lock){....}-{1:3}, at: __handle_sysrq+0x8/0x308 [ 59.931206] #5: ffffffd08de2a8e8 (console_lock){+.+.}-{0:0}, at: vprintk_default+0x40/0x58 [ 59.939596] #6: ffffffd08de2a940 (console_srcu){....}-{0:0}, at: console_flush_all+0x88/0x4a8 [ 59.948247] #7: ffffffd08de2ab00 (console_owner){+.-.}-{0:0}, at: console_lock_spinning_enable+0x50/0x90 [ 59.957858] #8: ffffffd08de2ab28 (printk_legacy_map-wait-type-override){+...}-{4:4}, at: console_flush_all+0x2a0/0x4a8 [ 59.968691] #9: ffffffd02c28e298 (target_list_lock){+.-.}-{3:3}, at: write_msg+0x60/0x1b0 [netconsole] [ 59.978128] #10: ffffff81fef08f80 (&c->lock){?.-.}-{3:3}, at: ___slab_alloc+0x7c/0x1178 [ 59.986256] [ 59.986256] stack backtrace: [ 59.990624] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G W 6.17.0-rc2-v8-lockdep #532 PREEMPTLAZY [ 60.001364] Tainted: [W]=WARN [ 60.004334] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT) [ 60.010185] Call trace: [ 60.012631] show_stack+0x20/0x38 (C) [ 60.016302] dump_stack_lvl+0x90/0xd0 [ 60.019974] dump_stack+0x18/0x28 [ 60.023297] print_usage_bug.part.0+0x29c/0x368 [ 60.027841] mark_lock+0x724/0x928 [ 60.031251] mark_held_locks+0x54/0x90 [ 60.035010] lockdep_hardirqs_on_prepare+0x108/0x228 [ 60.039992] arm64_exit_el1_dbg.isra.0+0x30/0x58 [ 60.044624] el1_brk64+0x40/0x50 [ 60.047859] el1h_64_sync_handler+0x68/0xb0 [ 60.052053] el1h_64_sync+0x6c/0x70 [ 60.055548] get_freelist+0x164/0x1d8 (P) [ 60.059569] ___slab_alloc+0xf8/0x1178 [ 60.063328] __slab_alloc.constprop.0+0x6c/0xc8 [ 60.067873] kmem_cache_alloc_node_noprof+0x2c4/0x328 [ 60.072941] __alloc_skb+0x154/0x188 [ 60.076526] netpoll_send_udp+0xac/0x418 [ 60.080460] write_msg+0x124/0x1b0 [netconsole] [ 60.085004] console_flush_all+0x2e4/0x4a8 [ 60.089112] console_unlock+0x84/0x188 [ 60.092871] vprintk_emit+0x24c/0x450 [ 60.096543] vprintk_default+0x40/0x58 [ 60.100301] vprintk+0x3c/0x80 [ 60.103362] _printk+0x64/0x98 [ 60.106422] __handle_sysrq+0x1ec/0x308 [ 60.110267] sysrq_filter+0xcc/0x518 [ 60.113850] input_handle_events_filter+0x58/0xc0 [ 60.118567] input_pass_values+0x1fc/0x2e8 [ 60.122674] input_handle_event+0x450/0x558 [ 60.126868] input_event+0x6c/0xb8 [ 60.130277] hidinput_report_event+0x4c/0x78 [ 60.134558] hid_report_raw_event+0xb4/0x418 [ 60.138838] hid_input_report+0x138/0x1a0 [ 60.142857] hid_irq_in+0x180/0x308 [ 60.146354] __usb_hcd_giveback_urb+0x98/0x160 [ 60.150810] usb_giveback_urb_bh+0xb0/0x150 [ 60.155004] process_one_work+0x20c/0x5e8 [ 60.159024] bh_worker+0x1b0/0x218 [ 60.162434] workqueue_softirq_action+0x80/0x98 [ 60.166979] tasklet_hi_action+0x20/0x60 [ 60.170913] handle_softirqs+0x144/0x528 [ 60.174846] __do_softirq+0x1c/0x28 [ 60.178342] ____do_softirq+0x18/0x30 [ 60.182012] call_on_irq_stack+0x30/0x70 [ 60.185944] do_softirq_own_stack+0x24/0x38 [ 60.190139] __irq_exit_rcu+0x100/0x1f8 [ 60.193985] irq_exit_rcu+0x18/0x58 [ 60.197482] el1_interrupt+0x38/0x68 [ 60.201064] el1h_64_irq_handler+0x18/0x28 [ 60.205171] el1h_64_irq+0x6c/0x70 [ 60.208579] default_idle_call+0x84/0x140 (P) [ 60.212947] do_idle+0x22c/0x298 [ 60.216183] cpu_startup_entry+0x40/0x50 [ 60.220117] rest_init+0x108/0x1a0 [ 60.223526] start_kernel+0x760/0x8c8 [ 60.227198] __primary_switched+0x88/0x98 [ 68.638857] sysrq: Trigger a crash [ 68.638885] Kernel panic - not syncing: sysrq triggered crash [ 68.648062] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G W 6.17.0-rc2-v8-lockdep #532 PREEMPTLAZY [ 68.658803] Tainted: [W]=WARN [ 68.661774] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT) [ 68.667626] Call trace: [ 68.670073] show_stack+0x20/0x38 (C) [ 68.673749] dump_stack_lvl+0x38/0xd0 [ 68.677423] dump_stack+0x18/0x28 [ 68.680745] vpanic+0x340/0x398 [ 68.683893] panic+0x64/0x70 [-- Attachment #4: netconsole-rpi5-6.17-eth0+patch-and+revert.log --] [-- Type: text/x-log, Size: 3698 bytes --] Note: applied "net: macb: Fix tx_ptr_lock locking" [ 40.676566] sysrq: HELP : loglevel(0-9) reboot(b) crash(c) show-all-locks(d) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) sak(k) show-backtrace-all-active-cpus(l) show-memory-usage(m) nice-all-RT-tasks(n) poweroff(o) show-registers(p) show-all-timers(q) unraw(r) sync(s) show-task-states(t) unmount(u) force-fb(v) show-blocked-tasks(w) dump-ftrace-buffer(z) replay-kernel-logs(R) [ 47.612947] sysrq: Trigger a crash [ 47.612967] Kernel panic - not syncing: sysrq triggered crash [ 47.622140] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G W 6.17.0-rc3-v8-lockdep #540 PREEMPTLAZY [ 47.632879] Tainted: [W]=WARN [ 47.635849] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT) [ 47.641700] Call trace: [ 47.644147] show_stack+0x20/0x38 (C) Note: add revert of "net: cadence: macb: Implement BQL" [ 47.108822] sysrq: HELP : loglevel(0-9) reboot(b) crash(c) show-all-locks(d) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) sak(k) show-backtrace-all-active-cpus(l) show-memory-usage(m) nice-all-RT-tasks(n) poweroff(o) show-registers(p) show-all-timers(q) unraw(r) sync(s) show-task-states(t) unmount(u) force-fb(v) show-blocked-tasks(w) dump-ftrace-buffer(z) replay-kernel-logs(R) [ 50.132998] sysrq: Trigger a crash [ 50.142195] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G W 6.17.0-rc3-v8-lockdep #541 PREEMPTLAZY [ 50.152935] Tainted: [W]=WARN [ 50.155904] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT) [ 50.161755] Call trace: [ 50.164201] show_stack+0x20/0x38 (C) [ 50.167872] dump_stack_lvl+0x38/0xd0 [ 50.171544] dump_stack+0x18/0x28 [ 50.174865] vpanic+0x340/0x398 [ 50.178012] panic+0x64/0x70 [ 50.180895] sysrq_handle_crash+0x50/0xa8 [ 50.184915] __handle_sysrq+0xe4/0x308 [ 50.188672] sysrq_filter+0xcc/0x518 [ 50.192254] input_handle_events_filter+0x58/0xc0 [ 50.196971] input_pass_values+0x1fc/0x2e8 [ 50.201078] input_handle_event+0x450/0x558 [ 50.205271] input_event+0x6c/0xb8 [ 50.208678] hidinput_report_event+0x4c/0x78 [ 50.212959] hid_report_raw_event+0xb4/0x418 [ 50.217240] hid_input_report+0x138/0x1a0 [ 50.133022] Kernel panic - not syncing: sysrq triggered crash [ 50.221258] hid_irq_in+0x180/0x308 [ 50.224753] __usb_hcd_giveback_urb+0x98/0x160 [ 50.229210] usb_giveback_urb_bh+0xb0/0x150 [ 50.233403] process_one_work+0x20c/0x5e8 [ 50.237422] bh_worker+0x1b0/0x218 [ 50.240830] workqueue_softirq_action+0x80/0x98 [ 50.245374] tasklet_hi_action+0x20/0x60 [ 50.249307] handle_softirqs+0x144/0x528 [ 50.253239] __do_softirq+0x1c/0x28 [ 50.256734] ____do_softirq+0x18/0x30 [ 50.260404] call_on_irq_stack+0x30/0x70 [ 50.264335] do_softirq_own_stack+0x24/0x38 [ 50.268529] __irq_exit_rcu+0x100/0x1f8 [ 50.272374] irq_exit_rcu+0x18/0x58 [ 50.275869] el1_interrupt+0x38/0x68 [ 50.279452] el1h_64_irq_handler+0x18/0x28 [ 50.283557] el1h_64_irq+0x6c/0x70 [ 50.286964] default_idle_call+0x84/0x140 (P) [ 50.291332] do_idle+0x22c/0x298 [ 50.294567] cpu_startup_entry+0x40/0x50 [ 50.298499] rest_init+0x108/0x1a0 [ 50.301907] start_kernel+0x760/0x8c8 [ 50.305577] __primary_switched+0x88/0x98 [ 50.309598] SMP: stopping secondary CPUs [ 51.358529] SMP: failed to stop secondary CPUs 2 [ 51.363720] Kernel Offset: 0x100c400000 from 0xffffffc080000000 [ 51.369669] PHYS_OFFSET: 0x0 [ 51.372558] CPU features: 0x40000,0000b800,12023140,0400720b [ 51.378241] Memory Limit: none [ 51.381310] ---[ end Kernel panic - not syncing: sysrq triggered crash ]--- ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: netpoll: raspberrypi [4 5] driver locking woes 2025-08-28 17:26 ` Mike Galbraith @ 2025-08-28 18:56 ` Sean Anderson 2025-08-29 2:55 ` Mike Galbraith 0 siblings, 1 reply; 4+ messages in thread From: Sean Anderson @ 2025-08-28 18:56 UTC (permalink / raw) To: Mike Galbraith, Robert Hancock; +Cc: Breno Leitao, netdev@vger.kernel.org [-- Attachment #1: Type: text/plain, Size: 2058 bytes --] Hi Mike, On 8/28/25 13:26, Mike Galbraith wrote: > On Thu, 2025-08-28 at 10:57 -0400, Sean Anderson wrote: >> Hi Mike, >> >> On 8/27/25 12:02, Mike Galbraith wrote: >> > Unexpected addendum to done deal datapoint, so off list. >> > >> > On Tue, 2025-08-26 at 11:49 +0200, Mike Galbraith wrote: >> > > >> > > The pi5 gripe fix is equally trivial, but submitting that is pointless >> > > given there's something else amiss in fingered commit. This is all of >> > > the crash info that escapes the box w/wo gripes silenced. >> > > >> > > [ 51.688868] sysrq: Trigger a crash >> > > [ 51.688892] Kernel panic - not syncing: sysrq triggered crash >> > > [ 51.698066] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.17.0-rc2-v8-lockdep #533 PREEMPTLAZY >> > > [ 51.707234] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT) >> > > [ 51.713085] Call trace: >> > > [ 51.715532] show_stack+0x20/0x38 (C) >> > > [ 51.719206] dump_stack_lvl+0x38/0xd0 >> > > [ 51.722878] dump_stack+0x18/0x28 >> > > >> > > That aspect is a punt and run atm (time.. and dash of laziness:). >> > >> > Plan was to end datapoint thread, but after booting pi5's 6.12 kernel, >> > for some reason I fired up netconsole.. and box promptly exhibited the >> > netpoll locking bug warning, indicating presence of 138badbc21a0. >> > Instead of saying to self "nope, just walk away", I poked SysRq-C.. and >> > the bloody damn monitoring box received a 100% complete death rattle. >> > Well bugger. >> >> Did you get a backtrace for this? > > Yes, logs for 6.12.41 and 6.17.0-rc2 attached. > > Since a patch has meanwhile landed, also a log of patched 6.17.0-rc3 > now gripe free (yay) but with aforementioned broken output, followed by > addition of the e6a532185daa revert to confirm it still cures that. > >> And to be clear, the steps to reproduce this are to boot a kernel with >> lockdep enabled with netconsole on macb and then hit sysrq? > > Yup. Looks like the tx completion path can also be called from netpoll. Can you try the attached patch? --Sean [-- Attachment #2: 0001-net-macb-Fix-tx_ptr_lock-locking.patch --] [-- Type: text/x-patch, Size: 4585 bytes --] From 3fe439760955370b9f005ef2728b5fc9b14eeea5 Mon Sep 17 00:00:00 2001 From: Sean Anderson <sean.anderson@linux.dev> Date: Thu, 28 Aug 2025 11:55:01 -0400 Subject: [PATCH net v2] net: macb: Fix tx_ptr_lock locking macb_start_xmit can be called with bottom-halves disabled (e.g. transmitting from softirqs) as well as with interrupts disabled (with netpoll). Because of this, all other functions taking tx_ptr_lock must disable IRQs, and macb_start_xmit must only re-enable IRQs if they were already enabled. Fixes: 138badbc21a0 ("net: macb: use NAPI for TX completion path") Reported-by: Mike Galbraith <efault@gmx.de> Signed-off-by: Sean Anderson <sean.anderson@linux.dev> --- Changes in v2: - Use irqsave/restore for all accesses, since they can also also be called from netpoll. drivers/net/ethernet/cadence/macb_main.c | 28 ++++++++++++++---------- 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/drivers/net/ethernet/cadence/macb_main.c b/drivers/net/ethernet/cadence/macb_main.c index 16d28a8b3b56..c769b7dbd3ba 100644 --- a/drivers/net/ethernet/cadence/macb_main.c +++ b/drivers/net/ethernet/cadence/macb_main.c @@ -1223,12 +1223,13 @@ static int macb_tx_complete(struct macb_queue *queue, int budget) { struct macb *bp = queue->bp; u16 queue_index = queue - bp->queues; + unsigned long flags; unsigned int tail; unsigned int head; int packets = 0; u32 bytes = 0; - spin_lock(&queue->tx_ptr_lock); + spin_lock_irqsave(&queue->tx_ptr_lock, flags); head = queue->tx_head; for (tail = queue->tx_tail; tail != head && packets < budget; tail++) { struct macb_tx_skb *tx_skb; @@ -1291,7 +1292,7 @@ static int macb_tx_complete(struct macb_queue *queue, int budget) CIRC_CNT(queue->tx_head, queue->tx_tail, bp->tx_ring_size) <= MACB_TX_WAKEUP_THRESH(bp)) netif_wake_subqueue(bp->dev, queue_index); - spin_unlock(&queue->tx_ptr_lock); + spin_unlock_irqrestore(&queue->tx_ptr_lock, flags); return packets; } @@ -1707,8 +1708,9 @@ static void macb_tx_restart(struct macb_queue *queue) { struct macb *bp = queue->bp; unsigned int head_idx, tbqp; + unsigned long flags; - spin_lock(&queue->tx_ptr_lock); + spin_lock_irqsave(&queue->tx_ptr_lock, flags); if (queue->tx_head == queue->tx_tail) goto out_tx_ptr_unlock; @@ -1720,19 +1722,20 @@ static void macb_tx_restart(struct macb_queue *queue) if (tbqp == head_idx) goto out_tx_ptr_unlock; - spin_lock_irq(&bp->lock); + spin_lock(&bp->lock); macb_writel(bp, NCR, macb_readl(bp, NCR) | MACB_BIT(TSTART)); - spin_unlock_irq(&bp->lock); + spin_unlock(&bp->lock); out_tx_ptr_unlock: - spin_unlock(&queue->tx_ptr_lock); + spin_unlock_irqrestore(&queue->tx_ptr_lock, flags); } static bool macb_tx_complete_pending(struct macb_queue *queue) { bool retval = false; + unsigned long flags; - spin_lock(&queue->tx_ptr_lock); + spin_lock_irqsave(&queue->tx_ptr_lock, flags); if (queue->tx_head != queue->tx_tail) { /* Make hw descriptor updates visible to CPU */ rmb(); @@ -1740,7 +1743,7 @@ static bool macb_tx_complete_pending(struct macb_queue *queue) if (macb_tx_desc(queue, queue->tx_tail)->ctrl & MACB_BIT(TX_USED)) retval = true; } - spin_unlock(&queue->tx_ptr_lock); + spin_unlock_irqrestore(&queue->tx_ptr_lock, flags); return retval; } @@ -2308,6 +2311,7 @@ static netdev_tx_t macb_start_xmit(struct sk_buff *skb, struct net_device *dev) struct macb_queue *queue = &bp->queues[queue_index]; unsigned int desc_cnt, nr_frags, frag_size, f; unsigned int hdrlen; + unsigned long flags; bool is_lso; netdev_tx_t ret = NETDEV_TX_OK; @@ -2368,7 +2372,7 @@ static netdev_tx_t macb_start_xmit(struct sk_buff *skb, struct net_device *dev) desc_cnt += DIV_ROUND_UP(frag_size, bp->max_tx_length); } - spin_lock_bh(&queue->tx_ptr_lock); + spin_lock_irqsave(&queue->tx_ptr_lock, flags); /* This is a hard error, log it. */ if (CIRC_SPACE(queue->tx_head, queue->tx_tail, @@ -2392,15 +2396,15 @@ static netdev_tx_t macb_start_xmit(struct sk_buff *skb, struct net_device *dev) netdev_tx_sent_queue(netdev_get_tx_queue(bp->dev, queue_index), skb->len); - spin_lock_irq(&bp->lock); + spin_lock(&bp->lock); macb_writel(bp, NCR, macb_readl(bp, NCR) | MACB_BIT(TSTART)); - spin_unlock_irq(&bp->lock); + spin_unlock(&bp->lock); if (CIRC_SPACE(queue->tx_head, queue->tx_tail, bp->tx_ring_size) < 1) netif_stop_subqueue(dev, queue_index); unlock: - spin_unlock_bh(&queue->tx_ptr_lock); + spin_unlock_irqrestore(&queue->tx_ptr_lock, flags); return ret; } -- 2.35.1.1320.gc452695387.dirty ^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: netpoll: raspberrypi [4 5] driver locking woes 2025-08-28 18:56 ` Sean Anderson @ 2025-08-29 2:55 ` Mike Galbraith 0 siblings, 0 replies; 4+ messages in thread From: Mike Galbraith @ 2025-08-29 2:55 UTC (permalink / raw) To: Sean Anderson, Robert Hancock; +Cc: Breno Leitao, netdev@vger.kernel.org On Thu, 2025-08-28 at 14:56 -0400, Sean Anderson wrote: > > Looks like the tx completion path can also be called from netpoll. Can > you try the attached patch? Done, zero netpoll or lockdep complaints. -Mike ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2025-08-29 2:55 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <4698029bbe7d7a33edd905a027e3a183ac51ad8a.camel@gmx.de>
[not found] ` <e32a52852025d522f44d9d6ccc88c716ff432f8f.camel@gmx.de>
[not found] ` <f4fa3fcc637ffb6531982a90dbd9c27114e93036.camel@gmx.de>
2025-08-28 14:57 ` netpoll: raspberrypi [4 5] driver locking woes Sean Anderson
2025-08-28 17:26 ` Mike Galbraith
2025-08-28 18:56 ` Sean Anderson
2025-08-29 2:55 ` Mike Galbraith
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).