Re: netpoll: raspberrypi [4 5] driver locking woes

Re: netpoll: raspberrypi [4 5] driver locking woes

[Sean Anderson]

Hi Mike,

On 8/27/25 12:02, Mike Galbraith wrote:
> Unexpected addendum to done deal datapoint, so off list.
> 
> On Tue, 2025-08-26 at 11:49 +0200, Mike Galbraith wrote:
>> 
>> The pi5 gripe fix is equally trivial, but submitting that is pointless
>> given there's something else amiss in fingered commit.  This is all of
>> the crash info that escapes the box w/wo gripes silenced.
>> 
>> [   51.688868] sysrq: Trigger a crash
>> [   51.688892] Kernel panic - not syncing: sysrq triggered crash
>> [   51.698066] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.17.0-rc2-v8-lockdep #533 PREEMPTLAZY
>> [   51.707234] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT)
>> [   51.713085] Call trace:
>> [   51.715532]  show_stack+0x20/0x38 (C)
>> [   51.719206]  dump_stack_lvl+0x38/0xd0
>> [   51.722878]  dump_stack+0x18/0x28
>> 
>> That aspect is a punt and run atm (time.. and dash of laziness:).
> 
> Plan was to end datapoint thread, but after booting pi5's 6.12 kernel,
> for some reason I fired up netconsole.. and box promptly exhibited the
> netpoll locking bug warning, indicating presence of 138badbc21a0. 
> Instead of saying to self "nope, just walk away", I poked SysRq-C.. and
> the bloody damn monitoring box received a 100% complete death rattle. 
> Well bugger.

Did you get a backtrace for this?

And to be clear, the steps to reproduce this are to boot a kernel with
lockdep enabled with netconsole on macb and then hit sysrq?

> Two trees contain locking buglet introducing 138badbc21a0, 6.12.41 and
> 6.17.0, but only the later reproduces the above mess.  A quick stare at
> git diff inspired checking e6a532185daa, and sure enough reverting only
> it reproduced a functional netconsole, modulo warnings, ie the mess
> above manifests only with BOTH resident.  Either revert 138badbc21a0,
> or revert e6a532185daa and fix the 138badbc21a0 locking buglet, and the
> result is the same, a perfectly functional warning free netconsole.

Re: netpoll: raspberrypi [4 5] driver locking woes

[Mike Galbraith]

[-- Attachment #1: Type: text/plain, Size: 2449 bytes --]

On Thu, 2025-08-28 at 10:57 -0400, Sean Anderson wrote:
> Hi Mike,
> 
> On 8/27/25 12:02, Mike Galbraith wrote:
> > Unexpected addendum to done deal datapoint, so off list.
> > 
> > On Tue, 2025-08-26 at 11:49 +0200, Mike Galbraith wrote:
> > > 
> > > The pi5 gripe fix is equally trivial, but submitting that is pointless
> > > given there's something else amiss in fingered commit.  This is all of
> > > the crash info that escapes the box w/wo gripes silenced.
> > > 
> > > [   51.688868] sysrq: Trigger a crash
> > > [   51.688892] Kernel panic - not syncing: sysrq triggered crash
> > > [   51.698066] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.17.0-rc2-v8-lockdep #533 PREEMPTLAZY
> > > [   51.707234] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT)
> > > [   51.713085] Call trace:
> > > [   51.715532]  show_stack+0x20/0x38 (C)
> > > [   51.719206]  dump_stack_lvl+0x38/0xd0
> > > [   51.722878]  dump_stack+0x18/0x28
> > > 
> > > That aspect is a punt and run atm (time.. and dash of laziness:).
> > 
> > Plan was to end datapoint thread, but after booting pi5's 6.12 kernel,
> > for some reason I fired up netconsole.. and box promptly exhibited the
> > netpoll locking bug warning, indicating presence of 138badbc21a0. 
> > Instead of saying to self "nope, just walk away", I poked SysRq-C.. and
> > the bloody damn monitoring box received a 100% complete death rattle. 
> > Well bugger.
> 
> Did you get a backtrace for this?

Yes, logs for 6.12.41 and 6.17.0-rc2 attached.

Since a patch has meanwhile landed, also a log of patched 6.17.0-rc3
now gripe free (yay) but with aforementioned broken output, followed by
addition of the e6a532185daa revert to confirm it still cures that.

> And to be clear, the steps to reproduce this are to boot a kernel with
> lockdep enabled with netconsole on macb and then hit sysrq?

Yup.

> > Two trees contain locking buglet introducing 138badbc21a0, 6.12.41 and
> > 6.17.0, but only the later reproduces the above mess.  A quick stare at
> > git diff inspired checking e6a532185daa, and sure enough reverting only
> > it reproduced a functional netconsole, modulo warnings, ie the mess
> > above manifests only with BOTH resident.  Either revert 138badbc21a0,
> > or revert e6a532185daa and fix the 138badbc21a0 locking buglet, and the
> > result is the same, a perfectly functional warning free netconsole.
> 


[-- Attachment #2: netconsole-rpi5-6.12-eth0.log --]
[-- Type: text/x-log, Size: 18613 bytes --]

[  349.336384] sysrq: Trigger a crash
[  349.336569] ------------[ cut here ]------------
[  349.336572] netpoll_send_skb_on_dev(): eth0 enabled interrupts in poll (macb_start_xmit+0x0/0xc48)
[  349.336622] WARNING: CPU: 0 PID: 0 at net/core/netpoll.c:385 netpoll_send_skb+0x4e4/0x528
[  349.336632] Modules linked in: netconsole quota_v2 quota_tree rfcomm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device algif_hash algif_skcipher af_alg bnep snd_soc_pcm512x_i2c snd_soc_pcm512x regmap_i2c spidev brcmfmac_wcc brcmfmac vc4 hci_uart btbcm bluetooth aes_ce_blk snd_soc_hdmi_codec v3d aes_ce_cipher ghash_ce drm_display_helper rpi_hevc_dec cec gpu_sched brcmutil drm_dma_helper drm_shmem_helper v4l2_mem2mem pisp_be gf128mul sha2_ce videobuf2_dma_contig sha256_arm64 ecdh_generic videobuf2_memops cfg80211 snd_soc_hifiberry_dacplus ecc videobuf2_v4l2 raspberrypi_hwmon designware_i2s sha1_ce libaes drm_kms_helper snd_soc_core videodev i2c_brcmstb spi_bcm2835 videobuf2_common rfkill gpio_keys clk_hifiberry_dacpro snd_compress binfmt_misc snd_pcm_dmaengine mc snd_pcm rp1_pio rp1_mailbox snd_timer raspberrypi_gpiomem i2c_designware_platform rp1_fw snd i2c_designware_core rp1_adc sg nvmem_rmem uio_pdrv_genirq uio nfsd drm fuse dm_mod i2c_dev ledtrig_pattern auth_rpcgss drm_panel_orientation_quirks backlight ip_tables
[  349.336710]  x_tables ipv6
[  349.336715] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G        W          6.12.41-v8-lockdep #518
[  349.336721] Tainted: [W]=WARN
[  349.336722] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT)
[  349.336724] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  349.336727] pc : netpoll_send_skb+0x4e4/0x528
[  349.336731] lr : netpoll_send_skb+0x4e4/0x528
[  349.336735] sp : ffffffc080003670
[  349.336737] x29: ffffffc080003670 x28: ffffff810099e600 x27: 0000000000000000
[  349.336741] x26: 0000000000000080 x25: ffffff812c8fc400 x24: ffffff8102e0c000
[  349.336745] x23: 00000000000000c0 x22: ffffff81395a9300 x21: ffffffd08de13a28
[  349.336748] x20: ffffff81183c1130 x19: ffffffd08d94fe40 x18: ffffffffffffffff
[  349.336752] x17: ffffffb1715bb000 x16: ffffffd08d0a9568 x15: 00000000fffffffd
[  349.336755] x14: 0000000000000001 x13: ffffffc080002f94 x12: 6920737470757272
[  349.336759] x11: fffffffffffe0000 x10: ffffffd08ddfa080 x9 : ffffffd08c545950
[  349.336762] x8 : 0000000000000001 x7 : ffffffd08dd770a0 x6 : 0000000000000001
[  349.336766] x5 : ffffffd08dd77000 x4 : 0000000000000001 x3 : 0000000000000000
[  349.336769] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffffffd08dd7cec0
[  349.336773] Call trace:
[  349.336774]  netpoll_send_skb+0x4e4/0x528
[  349.336778]  netpoll_send_udp+0x258/0x3e8
[  349.336783]  write_msg+0x100/0x138 [netconsole]
[  349.336792]  console_flush_all+0x264/0x4a0
[  349.336798]  console_unlock+0x84/0x188
[  349.336802]  vprintk_emit+0x22c/0x420
[  349.336807]  vprintk_default+0x40/0x58
[  349.336811]  vprintk+0x54/0xe0
[  349.336814]  _printk+0x64/0x98
[  349.336817]  __handle_sysrq+0xe0/0x2e0
[  349.336823]  sysrq_filter+0xcc/0x518
[  349.336827]  input_handle_events_filter+0x58/0xc0
[  349.336831]  input_pass_values+0x1e4/0x2c8
[  349.336836]  input_handle_event+0x450/0x558
[  349.336839]  input_event+0x68/0x98
[  349.336843]  hidinput_report_event+0x4c/0x78
[  349.336848]  hid_report_raw_event+0xb4/0x3e0
[  349.336850]  hid_input_report+0x138/0x1a0
[  349.336853]  hid_irq_in+0x180/0x308
[  349.336858]  __usb_hcd_giveback_urb+0x98/0x160
[  349.336863]  usb_giveback_urb_bh+0xb0/0x150
[  349.336867]  process_one_work+0x214/0x640
[  349.336873]  bh_worker+0x1b0/0x218
[  349.336877]  workqueue_softirq_action+0x80/0x98
[  349.336880]  tasklet_hi_action+0x20/0x60
[  349.336883]  handle_softirqs+0x154/0x528
[  349.336885]  __do_softirq+0x1c/0x28
[  349.336889]  ____do_softirq+0x18/0x30
[  349.336892]  call_on_irq_stack+0x30/0x70
[  349.336895]  do_softirq_own_stack+0x24/0x38
[  349.336898]  __irq_exit_rcu+0x100/0x1a0
[  349.336901]  irq_exit_rcu+0x18/0x58
[  349.336903]  el1_interrupt+0x38/0x70
[  349.336908]  el1h_64_irq_handler+0x18/0x28
[  349.336912]  el1h_64_irq+0x64/0x68
[  349.336914]  default_idle_call+0xa8/0x1f8
[  349.336917]  do_idle+0x22c/0x298
[  349.336920]  cpu_startup_entry+0x3c/0x50
[  349.336923]  rest_init+0x100/0x198
[  349.336925]  start_kernel+0x75c/0x900
[  349.336930]  __primary_switched+0x80/0x90
[  349.336934] irq event stamp: 8607573
[  349.336935] hardirqs last  enabled at (8607572): [<ffffffd08d2c7ce4>] el1_interrupt+0x44/0x70
[  349.336940] hardirqs last disabled at (8607573): [<ffffffd08d2c72f4>] el1_dbg+0x24/0x98
[  349.336944] softirqs last  enabled at (8607524): [<ffffffd08c4b04c8>] handle_softirqs+0x508/0x528
[  349.336947] softirqs last disabled at (8607545): [<ffffffd08c4102ac>] __do_softirq+0x1c/0x28
[  349.336949] ---[ end trace 0000000000000000 ]---
[  349.340398] ------------[ cut here ]------------
[  349.340399] WARNING: CPU: 0 PID: 0 at mm/slub.c:700 get_freelist+0x1bc/0x2b0
[  349.340405] Modules linked in: netconsole quota_v2 quota_tree rfcomm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device algif_hash algif_skcipher af_alg bnep snd_soc_pcm512x_i2c snd_soc_pcm512x regmap_i2c spidev brcmfmac_wcc brcmfmac vc4 hci_uart btbcm bluetooth aes_ce_blk snd_soc_hdmi_codec v3d aes_ce_cipher ghash_ce drm_display_helper rpi_hevc_dec cec gpu_sched brcmutil drm_dma_helper drm_shmem_helper v4l2_mem2mem pisp_be gf128mul sha2_ce videobuf2_dma_contig sha256_arm64 ecdh_generic videobuf2_memops cfg80211 snd_soc_hifiberry_dacplus ecc videobuf2_v4l2 raspberrypi_hwmon designware_i2s sha1_ce libaes drm_kms_helper snd_soc_core videodev i2c_brcmstb spi_bcm2835 videobuf2_common rfkill gpio_keys clk_hifiberry_dacpro snd_compress binfmt_misc snd_pcm_dmaengine mc snd_pcm rp1_pio rp1_mailbox snd_timer raspberrypi_gpiomem i2c_designware_platform rp1_fw snd i2c_designware_core rp1_adc sg nvmem_rmem uio_pdrv_genirq uio nfsd drm fuse dm_mod i2c_dev ledtrig_pattern auth_rpcgss drm_panel_orientation_quirks backlight ip_tables
[  349.340461]  x_tables ipv6
[  349.340464] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G        W          6.12.41-v8-lockdep #518
[  349.340467] Tainted: [W]=WARN
[  349.340468] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT)
[  349.340469] pstate: 604000c9 (nZCv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  349.340472] pc : get_freelist+0x1bc/0x2b0
[  349.340475] lr : get_freelist+0x290/0x2b0
[  349.340477] sp : ffffffc080003460
[  349.340478] x29: ffffffc080003460 x28: 0000000000000000 x27: 0000000000000001
[  349.340482] x26: ffffffd08dd77000 x25: ffffffd08d93c448 x24: ffffff81001cec00
[  349.340485] x23: ffffffd08dd7cec0 x22: 0000000000100010 x21: 0000000000000000
[  349.340489] x20: 0000000080100010 x19: fffffffec4e56a40 x18: ffffffffffffffff
[  349.340492] x17: ffffffb1715bb000 x16: ffffffd08d0a9568 x15: 0000000000000006
[  349.340496] x14: 00000000000c2000 x13: 2d2d2d2d2d2d2d2d x12: 0000000000000000
[  349.340499] x11: 0000000000000190 x10: 0000000000000000 x9 : ffffffd08e70ab10
[  349.340503] x8 : ffffffd08dd7cec0 x7 : 0000000001298354 x6 : ffffffd08e6d4080
[  349.340506] x5 : 00000000000000c8 x4 : 0000000000040000 x3 : ffffffd08dd7cec0
[  349.340510] x2 : ffffffb1715bb000 x1 : ffffffd08dd7cec0 x0 : 0000000000000105
[  349.340513] Call trace:
[  349.340514]  get_freelist+0x1bc/0x2b0
[  349.340516]  ___slab_alloc+0xf4/0x10c0
[  349.340519]  __slab_alloc.constprop.0+0x6c/0xc8
[  349.340522]  kmem_cache_alloc_node_noprof+0x2e4/0x348
[  349.340525]  __alloc_skb+0x154/0x188
[  349.340529]  netpoll_send_udp+0xa8/0x3e8
[  349.340533]  write_msg+0x100/0x138 [netconsole]
[  349.340538]  console_flush_all+0x264/0x4a0
[  349.340542]  console_unlock+0x84/0x188
[  349.340546]  vprintk_emit+0x22c/0x420
[  349.340551]  vprintk_default+0x40/0x58
[  349.340555]  vprintk+0x54/0xe0
[  349.340557]  _printk+0x64/0x98
[  349.340560]  __handle_sysrq+0xe0/0x2e0
[  349.340565]  sysrq_filter+0xcc/0x518
[  349.340569]  input_handle_events_filter+0x58/0xc0
[  349.340572]  input_pass_values+0x1e4/0x2c8
[  349.340576]  input_handle_event+0x450/0x558
[  349.340580]  input_event+0x68/0x98
[  349.340583]  hidinput_report_event+0x4c/0x78
[  349.340587]  hid_report_raw_event+0xb4/0x3e0
[  349.340589]  hid_input_report+0x138/0x1a0
[  349.340592]  hid_irq_in+0x180/0x308
[  349.340596]  __usb_hcd_giveback_urb+0x98/0x160
[  349.340600]  usb_giveback_urb_bh+0xb0/0x150
[  349.340604]  process_one_work+0x214/0x640
[  349.340608]  bh_worker+0x1b0/0x218
[  349.340613]  workqueue_softirq_action+0x80/0x98
[  349.340615]  tasklet_hi_action+0x20/0x60
[  349.340618]  handle_softirqs+0x154/0x528
[  349.340620]  __do_softirq+0x1c/0x28
[  349.340622]  ____do_softirq+0x18/0x30
[  349.340625]  call_on_irq_stack+0x30/0x70
[  349.340628]  do_softirq_own_stack+0x24/0x38
[  349.340631]  __irq_exit_rcu+0x100/0x1a0
[  349.340634]  irq_exit_rcu+0x18/0x58
[  349.340636]  el1_interrupt+0x38/0x70
[  349.340640]  el1h_64_irq_handler+0x18/0x28
[  349.340644]  el1h_64_irq+0x64/0x68
[  349.340646]  default_idle_call+0xa8/0x1f8
[  349.340648]  do_idle+0x22c/0x298
[  349.340650]  cpu_startup_entry+0x3c/0x50
[  349.340652]  rest_init+0x100/0x198
[  349.340655]  start_kernel+0x75c/0x900
[  349.340658]  __primary_switched+0x80/0x90
[  349.340661] irq event stamp: 8607577
[  349.340662] hardirqs last  enabled at (8607576): [<ffffffd08d2c7ce4>] el1_interrupt+0x44/0x70
[  349.340666] hardirqs last disabled at (8607577): [<ffffffd08d2c72f4>] el1_dbg+0x24/0x98
[  349.340670] softirqs last  enabled at (8607524): [<ffffffd08c4b04c8>] handle_softirqs+0x508/0x528
[  349.340673] softirqs last disabled at (8607545): [<ffffffd08c4102ac>] __do_softirq+0x1c/0x28
[  349.340676] ---[ end trace 0000000000000000 ]---
[  349.340679] 
[  349.340680] ================================
[  349.340681] WARNING: inconsistent lock state
[  349.340682] 6.12.41-v8-lockdep #518 Tainted: G        W         
[  349.340684] --------------------------------
[  349.340684] inconsistent {IN-HARDIRQ-W} -> {HARDIRQ-ON-W} usage.
[  349.340686] swapper/0/0 [HC0[0]:SC1[1]:HE0:SE0] takes:
[  349.340688] ffffff81fef12fc0 (&c->lock){?.-.}-{2:2}, at: ___slab_alloc+0x78/0x10c0
[  349.340695] {IN-HARDIRQ-W} state was registered at:
[  349.340697]   lock_acquire+0x208/0x360
[  349.340700]   put_cpu_partial+0x88/0x258
[  349.340703]   __slab_free+0x1c8/0x390
[  349.340705]   kmem_cache_free+0x29c/0x400
[  349.340708]   mempool_free_slab+0x24/0x38
[  349.340712]   mempool_free+0x48/0xb8
[  349.340715]   bio_free+0x70/0x90
[  349.340719]   bio_put+0x120/0x278
[  349.340722]   end_bio_bh_io_sync+0x44/0x88
[  349.340725]   bio_endio+0x18c/0x288
[  349.340728]   blk_mq_end_request_batch+0x274/0x500
[  349.340731]   nvme_pci_complete_batch+0xe0/0x138
[  349.340734]   nvme_irq+0x78/0x90
[  349.340736]   __handle_irq_event_percpu+0x94/0x308
[  349.340740]   handle_irq_event+0x54/0xb8
[  349.340743]   handle_fasteoi_irq+0xc8/0x268
[  349.340747]   handle_irq_desc+0x48/0x68
[  349.340750]   generic_handle_domain_irq+0x24/0x38
[  349.340753]   gic_handle_irq+0x48/0xd8
[  349.340755]   call_on_irq_stack+0x30/0x70
[  349.340758]   do_interrupt_handler+0xdc/0xe8
[  349.340761]   el1_interrupt+0x34/0x70
[  349.340765]   el1h_64_irq_handler+0x18/0x28
[  349.340769]   el1h_64_irq+0x64/0x68
[  349.340771]   default_idle_call+0xa8/0x1f8
[  349.340773]   do_idle+0x22c/0x298
[  349.340775]   cpu_startup_entry+0x3c/0x50
[  349.340777]   secondary_start_kernel+0x128/0x150
[  349.340781]   __secondary_switched+0xb8/0xc0
[  349.340785] irq event stamp: 8607577
[  349.340786] hardirqs last  enabled at (8607576): [<ffffffd08d2c7ce4>] el1_interrupt+0x44/0x70
[  349.340790] hardirqs last disabled at (8607577): [<ffffffd08d2c72f4>] el1_dbg+0x24/0x98
[  349.340794] softirqs last  enabled at (8607524): [<ffffffd08c4b04c8>] handle_softirqs+0x508/0x528
[  349.340797] softirqs last disabled at (8607545): [<ffffffd08c4102ac>] __do_softirq+0x1c/0x28
[  349.340800] 
[  349.340800] other info that might help us debug this:
[  349.340801]  Possible unsafe locking scenario:
[  349.340801] 
[  349.340801]        CPU0
[  349.340802]        ----
[  349.340803]   lock(&c->lock);
[  349.340805]   <Interrupt>
[  349.340806]     lock(&c->lock);
[  349.340807] 
[  349.340807]  *** DEADLOCK ***
[  349.340807] 
[  349.340808] 11 locks held by swapper/0/0:
[  349.340810]  #0: ffffff8100045548 ((wq_completion)events_bh_highpri){+.-.}-{0:0}, at: process_one_work+0x198/0x640
[  349.340818]  #1: ffffffc080003e60 ((work_completion)(&bh->bh)){+.-.}-{0:0}, at: process_one_work+0x1c0/0x640
[  349.340826]  #2: ffffff8102eaa230 (&dev->event_lock){+.-.}-{2:2}, at: input_event+0x50/0x98
[  349.340833]  #3: ffffffd08de13a28 (rcu_read_lock){....}-{1:2}, at: input_pass_values+0x38/0x2c8
[  349.340841]  #4: ffffffd08de13a28 (rcu_read_lock){....}-{1:2}, at: __handle_sysrq+0x8/0x2e0
[  349.340848]  #5: ffffffd08dda1c80 (console_lock){+.+.}-{0:0}, at: vprintk_default+0x40/0x58
[  349.340856]  #6: ffffffd08dda1cd8 (console_srcu){....}-{0:0}, at: console_flush_all+0x8c/0x4a0
[  349.340863]  #7: ffffffd08dda1e98 (console_owner){+.-.}-{0:0}, at: console_lock_spinning_enable+0x50/0x90
[  349.340871]  #8: ffffffd08dda1ec0 (printk_legacy_map-wait-type-override){+...}-{3:3}, at: console_flush_all+0x220/0x4a0
[  349.340878]  #9: ffffffd02cbab220 (target_list_lock){+.-.}-{2:2}, at: write_msg+0x64/0x138 [netconsole]
[  349.340886]  #10: ffffff81fef12fc0 (&c->lock){?.-.}-{2:2}, at: ___slab_alloc+0x78/0x10c0
[  349.340892] 
[  349.340892] stack backtrace:
[  349.340893] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G        W          6.12.41-v8-lockdep #518
[  349.340897] Tainted: [W]=WARN
[  349.340898] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT)
[  349.340899] Call trace:
[  349.340900]  dump_backtrace+0xa0/0x100
[  349.340902]  show_stack+0x20/0x38
[  349.340904]  dump_stack_lvl+0x90/0xd0
[  349.340909]  dump_stack+0x18/0x28
[  349.340912]  print_usage_bug.part.0+0x29c/0x368
[  349.340915]  mark_lock+0x724/0x928
[  349.340917]  mark_held_locks+0x54/0x90
[  349.340919]  lockdep_hardirqs_on_prepare+0x108/0x228
[  349.340921]  arm64_exit_el1_dbg.isra.0+0x30/0x58
[  349.340925]  el1_dbg+0x84/0x98
[  349.340928]  el1h_64_sync_handler+0xc8/0xe8
[  349.340932]  el1h_64_sync+0x64/0x68
[  349.340934]  get_freelist+0x1bc/0x2b0
[  349.340936]  ___slab_alloc+0xf4/0x10c0
[  349.340939]  __slab_alloc.constprop.0+0x6c/0xc8
[  349.340942]  kmem_cache_alloc_node_noprof+0x2e4/0x348
[  349.340945]  __alloc_skb+0x154/0x188
[  349.340947]  netpoll_send_udp+0xa8/0x3e8
[  349.340952]  write_msg+0x100/0x138 [netconsole]
[  349.340955]  console_flush_all+0x264/0x4a0
[  349.340960]  console_unlock+0x84/0x188
[  349.340964]  vprintk_emit+0x22c/0x420
[  349.340968]  vprintk_default+0x40/0x58
[  349.340972]  vprintk+0x54/0xe0
[  349.340974]  _printk+0x64/0x98
[  349.340977]  __handle_sysrq+0xe0/0x2e0
[  349.340981]  sysrq_filter+0xcc/0x518
[  349.340986]  input_handle_events_filter+0x58/0xc0
[  349.340989]  input_pass_values+0x1e4/0x2c8
[  349.340992]  input_handle_event+0x450/0x558
[  349.340996]  input_event+0x68/0x98
[  349.341000]  hidinput_report_event+0x4c/0x78
[  349.341003]  hid_report_raw_event+0xb4/0x3e0
[  349.341006]  hid_input_report+0x138/0x1a0
[  349.341008]  hid_irq_in+0x180/0x308
[  349.341012]  __usb_hcd_giveback_urb+0x98/0x160
[  349.341016]  usb_giveback_urb_bh+0xb0/0x150
[  349.341020]  process_one_work+0x214/0x640
[  349.341025]  bh_worker+0x1b0/0x218
[  349.341029]  workqueue_softirq_action+0x80/0x98
[  349.341031]  tasklet_hi_action+0x20/0x60
[  349.341034]  handle_softirqs+0x154/0x528
[  349.341036]  __do_softirq+0x1c/0x28
[  349.341038]  ____do_softirq+0x18/0x30
[  349.341041]  call_on_irq_stack+0x30/0x70
[  349.341044]  do_softirq_own_stack+0x24/0x38
[  349.341047]  __irq_exit_rcu+0x100/0x1a0
[  349.341049]  irq_exit_rcu+0x18/0x58
[  349.341052]  el1_interrupt+0x38/0x70
[  349.341056]  el1h_64_irq_handler+0x18/0x28
[  349.341059]  el1h_64_irq+0x64/0x68
[  349.341062]  default_idle_call+0xa8/0x1f8
[  349.341064]  do_idle+0x22c/0x298
[  349.341066]  cpu_startup_entry+0x3c/0x50
[  349.341068]  rest_init+0x100/0x198
[  349.341070]  start_kernel+0x75c/0x900
[  349.341073]  __primary_switched+0x80/0x90
[  350.513730] v3d 1002000000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[  350.517765] Kernel panic - not syncing: sysrq triggered crash
[  350.517769] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G        W          6.12.41-v8-lockdep #518
[  350.517773] Tainted: [W]=WARN
[  350.517774] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT)
[  350.813441] Call trace:
[  350.815900]  dump_backtrace+0xa0/0x100
[  350.819671]  show_stack+0x20/0x38
[  350.823004]  dump_stack_lvl+0x38/0xd0
[  350.826688]  dump_stack+0x18/0x28
[  350.830023]  panic+0x3c8/0x430
[  350.833096]  sysrq_handle_crash+0x40/0x90
[  350.837130]  __handle_sysrq+0xf0/0x2e0
[  350.840901]  sysrq_filter+0xcc/0x518
[  350.844498]  input_handle_events_filter+0x58/0xc0
[  350.849228]  input_pass_values+0x1e4/0x2c8
[  350.853348]  input_handle_event+0x450/0x558
[  350.857554]  input_event+0x68/0x98
[  350.860976]  hidinput_report_event+0x4c/0x78
[  350.865271]  hid_report_raw_event+0xb4/0x3e0
[  350.869564]  hid_input_report+0x138/0x1a0
[  350.873595]  hid_irq_in+0x180/0x308
[  350.877105]  __usb_hcd_giveback_urb+0x98/0x160
[  350.881575]  usb_giveback_urb_bh+0xb0/0x150
[  350.885782]  process_one_work+0x214/0x640
[  350.889815]  bh_worker+0x1b0/0x218
[  350.893238]  workqueue_softirq_action+0x80/0x98
[  350.897793]  tasklet_hi_action+0x20/0x60
[  350.901737]  handle_softirqs+0x154/0x528
[  350.905681]  __do_softirq+0x1c/0x28
[  350.909188]  ____do_softirq+0x18/0x30
[  350.912871]  call_on_irq_stack+0x30/0x70
[  350.916815]  do_softirq_own_stack+0x24/0x38
[  350.921023]  __irq_exit_rcu+0x100/0x1a0
[  350.924880]  irq_exit_rcu+0x18/0x58
[  350.928388]  el1_interrupt+0x38/0x70
[  350.931985]  el1h_64_irq_handler+0x18/0x28
[  350.936105]  el1h_64_irq+0x64/0x68
[  350.939529]  default_idle_call+0xa8/0x1f8
[  350.943561]  do_idle+0x22c/0x298
[  350.946807]  cpu_startup_entry+0x3c/0x50
[  350.950751]  rest_init+0x100/0x198
[  350.954170]  start_kernel+0x75c/0x900
[  350.957855]  __primary_switched+0x80/0x90
[  350.961888] SMP: stopping secondary CPUs
[  352.035165] SMP: failed to stop secondary CPUs 1
[  352.039823] Kernel Offset: 0x100c400000 from 0xffffffc080000000
[  352.045770] PHYS_OFFSET: 0x0
[  352.048666] CPU features: 0x10,00000017,00280928,0200720b
[  352.054093] Memory Limit: none
[  352.057167] ---[ end Kernel panic - not syncing: sysrq triggered crash ]---

[-- Attachment #3: netconsole-rpi5-6.17-eth0.log --]
[-- Type: text/x-log, Size: 17135 bytes --]

[   58.750302] sysrq: HELP : loglevel(0-9) reboot(b) crash(c) show-all-locks(d) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) sak(k) show-backtrace-all-active-cpus(l) show-memory-usage(m) nice-all-RT-tasks(n) poweroff(o) show-registers(p) show-all-timers(q) unraw(r) sync(s) show-task-states(t) unmount(u) force-fb(v) show-blocked-tasks(w) dump-ftrace-buffer(z) replay-kernel-logs(R) 
[   58.750681] ------------[ cut here ]------------
[   58.750690] netpoll_send_skb_on_dev(): eth0 enabled interrupts in poll (macb_start_xmit+0x0/0xd30)
[   58.750730] WARNING: CPU: 0 PID: 0 at net/core/netpoll.c:359 netpoll_send_skb+0x46c/0x4b0
[   58.816326] Modules linked in: netconsole quota_v2 quota_tree rfcomm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device algif_hash algif_skcipher af_alg bnep snd_soc_pcm512x_i2c snd_soc_pcm512x spidev regmap_i2c binfmt_misc hci_uart aes_ce_blk btbcm bluetooth ecdh_generic ecc aes_ce_cipher ghash_ce brcmfmac_cyw gf128mul snd_soc_hifiberry_dacplus sg raspberrypi_hwmon brcmfmac brcmutil i2c_brcmstb cfg80211 vc4 rfkill spi_bcm2835 rpi_hevc_dec snd_soc_hdmi_codec gpio_keys drm_exec drm_display_helper clk_hifiberry_dacpro cec pisp_be v4l2_mem2mem v3d videobuf2_dma_contig drm_dma_helper gpu_sched drm_shmem_helper drm_client_lib rp1_pci videobuf2_memops videobuf2_v4l2 videodev designware_i2s drm_kms_helper videobuf2_common mc snd_soc_core snd_compress i2c_designware_platform rp1_pio snd_pcm_dmaengine rp1_fw i2c_designware_core snd_pcm rp1_mailbox raspberrypi_gpiomem snd_timer rp1_adc snd nvmem_rmem uio_pdrv_genirq uio i2c_dev ledtrig_pattern drm fuse drm_panel_orientation_quirks backlight dm_mod nfsd auth_rpcgss ip_tables
[   58.816415]  x_tables ipv6
[   58.909197] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.17.0-rc2-v8-lockdep #532 PREEMPTLAZY 
[   58.918365] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT)
[   58.924216] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   58.931202] pc : netpoll_send_skb+0x46c/0x4b0
[   58.935571] lr : netpoll_send_skb+0x46c/0x4b0
[   58.939939] sp : ffffffc080003690
[   58.943258] x29: ffffffc080003690 x28: 0000000000000000 x27: ffffff8101151800
[   58.950421] x26: ffffff8137fb6268 x25: 0000000000000080 x24: ffffff8137fb6200
[   58.957583] x23: ffffff8102d58000 x22: 00000000000000c0 x21: ffffff8104f7e100
[   58.964745] x20: ffffffd08de9c3c8 x19: ffffff8137f2d148 x18: 00000000ffffffff
[   58.971907] x17: ffffffb1714fe000 x16: ffffffd08d124020 x15: 00000000fffffffd
[   58.979069] x14: 0000000000000001 x13: 000000000002a000 x12: 0000000000008000
[   58.986231] x11: fffffffffffe0000 x10: ffffffd08e78f000 x9 : ffffffd08c5596f0
[   58.993393] x8 : ffffffffffffffff x7 : ffffffd08de07ef8 x6 : 0000000000000001
[   59.000555] x5 : ffffffd08de07000 x4 : 0000000000000001 x3 : 0000000000000001
[   59.007717] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffffffd08de0dd40
[   59.014879] Call trace:
[   59.017326]  netpoll_send_skb+0x46c/0x4b0 (P)
[   59.021695]  netpoll_send_udp+0x2ac/0x418
[   59.025715]  write_msg+0x124/0x1b0 [netconsole]
[   59.030262]  console_flush_all+0x2e4/0x4a8
[   59.034371]  console_unlock+0x84/0x188
[   59.038128]  vprintk_emit+0x24c/0x450
[   59.041798]  vprintk_default+0x40/0x58
[   59.045555]  vprintk+0x3c/0x80
[   59.048615]  _printk+0x64/0x98
[   59.051674]  __handle_sysrq+0x1ec/0x308
[   59.055519]  sysrq_filter+0xcc/0x518
[   59.059101]  input_handle_events_filter+0x58/0xc0
[   59.063818]  input_pass_values+0x1fc/0x2e8
[   59.067924]  input_handle_event+0x450/0x558
[   59.072118]  input_event+0x6c/0xb8
[   59.075525]  hidinput_report_event+0x4c/0x78
[   59.079806]  hid_report_raw_event+0xb4/0x418
[   59.084086]  hid_input_report+0x138/0x1a0
[   59.088104]  hid_irq_in+0x180/0x308
[   59.091600]  __usb_hcd_giveback_urb+0x98/0x160
[   59.096055]  usb_giveback_urb_bh+0xb0/0x150
[   59.100248]  process_one_work+0x20c/0x5e8
[   59.104269]  bh_worker+0x1b0/0x218
[   59.107677]  workqueue_softirq_action+0x80/0x98
[   59.112220]  tasklet_hi_action+0x20/0x60
[   59.116153]  handle_softirqs+0x144/0x528
[   59.120085]  __do_softirq+0x1c/0x28
[   59.123581]  ____do_softirq+0x18/0x30
[   59.127250]  call_on_irq_stack+0x30/0x70
[   59.131181]  do_softirq_own_stack+0x24/0x38
[   59.135374]  __irq_exit_rcu+0x100/0x1f8
[   59.139219]  irq_exit_rcu+0x18/0x58
[   59.142714]  el1_interrupt+0x38/0x68
[   59.146297]  el1h_64_irq_handler+0x18/0x28
[   59.150402]  el1h_64_irq+0x6c/0x70
[   59.153809]  default_idle_call+0x84/0x140 (P)
[   59.158177]  do_idle+0x22c/0x298
[   59.161412]  cpu_startup_entry+0x40/0x50
[   59.165344]  rest_init+0x108/0x1a0
[   59.168752]  start_kernel+0x760/0x8c8
[   59.172424]  __primary_switched+0x88/0x98
[   59.176444] irq event stamp: 279989
[   59.179938] hardirqs last  enabled at (279988): [<ffffffd08d350b64>] el1_interrupt+0x54/0x68
[   59.188408] hardirqs last disabled at (279989): [<ffffffd08d34fc94>] el1_brk64+0x1c/0x50
[   59.196530] softirqs last  enabled at (279934): [<ffffffd08c4c2900>] handle_softirqs+0x508/0x528
[   59.205349] softirqs last disabled at (279959): [<ffffffd08c4103ac>] __do_softirq+0x1c/0x28
[   59.213731] ---[ end trace 0000000000000000 ]---
[   59.218386] ------------[ cut here ]------------
[   59.218388] WARNING: CPU: 0 PID: 0 at mm/slub.c:735 get_freelist+0x160/0x1d8
[   59.230093] Modules linked in: netconsole quota_v2 quota_tree rfcomm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device algif_hash algif_skcipher af_alg bnep snd_soc_pcm512x_i2c snd_soc_pcm512x spidev regmap_i2c binfmt_misc hci_uart aes_ce_blk btbcm bluetooth ecdh_generic ecc aes_ce_cipher ghash_ce brcmfmac_cyw gf128mul snd_soc_hifiberry_dacplus sg raspberrypi_hwmon brcmfmac brcmutil i2c_brcmstb cfg80211 vc4 rfkill spi_bcm2835 rpi_hevc_dec snd_soc_hdmi_codec gpio_keys drm_exec drm_display_helper clk_hifiberry_dacpro cec pisp_be v4l2_mem2mem v3d videobuf2_dma_contig drm_dma_helper gpu_sched drm_shmem_helper drm_client_lib rp1_pci videobuf2_memops videobuf2_v4l2 videodev designware_i2s drm_kms_helper videobuf2_common mc snd_soc_core snd_compress i2c_designware_platform rp1_pio snd_pcm_dmaengine rp1_fw i2c_designware_core snd_pcm rp1_mailbox raspberrypi_gpiomem snd_timer rp1_adc snd nvmem_rmem uio_pdrv_genirq uio i2c_dev ledtrig_pattern drm fuse drm_panel_orientation_quirks backlight dm_mod nfsd auth_rpcgss ip_tables
[   59.230172]  x_tables ipv6
[   59.322954] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G        W           6.17.0-rc2-v8-lockdep #532 PREEMPTLAZY 
[   59.333694] Tainted: [W]=WARN
[   59.336664] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT)
[   59.342514] pstate: a04000c9 (NzCv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   59.349500] pc : get_freelist+0x160/0x1d8
[   59.353519] lr : get_freelist+0x1c0/0x1d8
[   59.357538] sp : ffffffc080003470
[   59.360857] x29: ffffffc080003470 x28: 0000000000000001 x27: ffffffd08de07000
[   59.368019] x26: ffffffd08d9ef508 x25: ffffff8100a3c240 x24: ffffffd08de0dd40
[   59.375181] x23: fffffffec413dfa0 x22: 0000000080100010 x21: fffffffec413df80
[   59.382343] x20: ffffff8104f7ef00 x19: 000000008010000e x18: 00000000ffffffff
[   59.389504] x17: ffffffb1714fe000 x16: ffffffd08d124020 x15: 0000000000000006
[   59.396666] x14: 0000000000000000 x13: 00000000000c2000 x12: 0000000000000000
[   59.403828] x11: 656820747563205b x10: ffffffd08e760810 x9 : ffffffd08e079000
[   59.410990] x8 : ffffffd08de0dd40 x7 : 00000000eb9a37bf x6 : 0000000000000190
[   59.418152] x5 : 00000000000000c8 x4 : 0000000000040000 x3 : 0000000000000001
[   59.425314] x2 : ffffffb1714fe000 x1 : ffffffd08de0dd40 x0 : 0000000000000107
[   59.432476] Call trace:
[   59.434923]  get_freelist+0x160/0x1d8 (P)
[   59.438942]  ___slab_alloc+0xf8/0x1178
[   59.442699]  __slab_alloc.constprop.0+0x6c/0xc8
[   59.447242]  kmem_cache_alloc_node_noprof+0x2c4/0x328
[   59.452309]  __alloc_skb+0x154/0x188
[   59.455893]  netpoll_send_udp+0xac/0x418
[   59.459826]  write_msg+0x124/0x1b0 [netconsole]
[   59.464370]  console_flush_all+0x2e4/0x4a8
[   59.468476]  console_unlock+0x84/0x188
[   59.472233]  vprintk_emit+0x24c/0x450
[   59.475903]  vprintk_default+0x40/0x58
[   59.479661]  vprintk+0x3c/0x80
[   59.482719]  _printk+0x64/0x98
[   59.485778]  __handle_sysrq+0x1ec/0x308
[   59.489622]  sysrq_filter+0xcc/0x518
[   59.493204]  input_handle_events_filter+0x58/0xc0
[   59.497921]  input_pass_values+0x1fc/0x2e8
[   59.502027]  input_handle_event+0x450/0x558
[   59.506220]  input_event+0x6c/0xb8
[   59.509627]  hidinput_report_event+0x4c/0x78
[   59.513907]  hid_report_raw_event+0xb4/0x418
[   59.518188]  hid_input_report+0x138/0x1a0
[   59.522206]  hid_irq_in+0x180/0x308
[   59.525700]  __usb_hcd_giveback_urb+0x98/0x160
[   59.530155]  usb_giveback_urb_bh+0xb0/0x150
[   59.534348]  process_one_work+0x20c/0x5e8
[   59.538368]  bh_worker+0x1b0/0x218
[   59.541776]  workqueue_softirq_action+0x80/0x98
[   59.546319]  tasklet_hi_action+0x20/0x60
[   59.550251]  handle_softirqs+0x144/0x528
[   59.554183]  __do_softirq+0x1c/0x28
[   59.557677]  ____do_softirq+0x18/0x30
[   59.561347]  call_on_irq_stack+0x30/0x70
[   59.565278]  do_softirq_own_stack+0x24/0x38
[   59.569471]  __irq_exit_rcu+0x100/0x1f8
[   59.573316]  irq_exit_rcu+0x18/0x58
[   59.576811]  el1_interrupt+0x38/0x68
[   59.580393]  el1h_64_irq_handler+0x18/0x28
[   59.584498]  el1h_64_irq+0x6c/0x70
[   59.587905]  default_idle_call+0x84/0x140 (P)
[   59.592273]  do_idle+0x22c/0x298
[   59.595507]  cpu_startup_entry+0x40/0x50
[   59.599438]  rest_init+0x108/0x1a0
[   59.602846]  start_kernel+0x760/0x8c8
[   59.606517]  __primary_switched+0x88/0x98
[   59.610536] irq event stamp: 279993
[   59.614030] hardirqs last  enabled at (279992): [<ffffffd08d350b64>] el1_interrupt+0x54/0x68
[   59.622499] hardirqs last disabled at (279993): [<ffffffd08d34fc94>] el1_brk64+0x1c/0x50
[   59.630620] softirqs last  enabled at (279934): [<ffffffd08c4c2900>] handle_softirqs+0x508/0x528
[   59.639440] softirqs last disabled at (279959): [<ffffffd08c4103ac>] __do_softirq+0x1c/0x28
[   59.647822] ---[ end trace 0000000000000000 ]---
[   59.652452] 
[   59.653938] ================================
[   59.658217] WARNING: inconsistent lock state
[   59.662496] 6.17.0-rc2-v8-lockdep #532 Tainted: G        W          
[   59.668869] --------------------------------
[   59.673148] inconsistent {IN-HARDIRQ-W} -> {HARDIRQ-ON-W} usage.
[   59.679172] swapper/0/0 [HC0[0]:SC1[1]:HE0:SE0] takes:
[   59.684325] ffffff81fef08f80 (&c->lock){?.-.}-{3:3}, at: ___slab_alloc+0x7c/0x1178
[   59.691926] {IN-HARDIRQ-W} state was registered at:
[   59.696815]   lock_acquire+0x1d4/0x368
[   59.700573]   put_cpu_partial+0x88/0x258
[   59.704504]   __slab_free+0x250/0x420
[   59.708173]   kmem_cache_free+0x288/0x3e8
[   59.712192]   mempool_free_slab+0x24/0x38
[   59.716211]   mempool_free+0x4c/0x120
[   59.719881]   bio_free+0x70/0x90
[   59.723114]   bio_put+0x120/0x278
[   59.726433]   end_bio_bh_io_sync+0x44/0x88
[   59.730538]   bio_endio+0x19c/0x280
[   59.734032]   blk_mq_end_request_batch+0x250/0x4c0
[   59.738837]   nvme_pci_complete_batch+0xdc/0x138
[   59.743467]   nvme_irq+0x7c/0x90
[   59.746700]   __handle_irq_event_percpu+0x94/0x318
[   59.751503]   handle_irq_event+0x54/0xb8
[   59.755433]   handle_fasteoi_irq+0x12c/0x270
[   59.759714]   handle_irq_desc+0x48/0x68
[   59.763557]   generic_handle_domain_irq+0x24/0x38
[   59.768274]   gic_handle_irq+0x48/0xd8
[   59.772031]   call_on_irq_stack+0x30/0x70
[   59.776051]   do_interrupt_handler+0xdc/0xe8
[   59.780333]   el1_interrupt+0x34/0x68
[   59.784002]   el1h_64_irq_handler+0x18/0x28
[   59.788195]   el1h_64_irq+0x6c/0x70
[   59.791690]   default_idle_call+0x84/0x140
[   59.795796]   do_idle+0x22c/0x298
[   59.799119]   cpu_startup_entry+0x3c/0x50
[   59.803139]   secondary_start_kernel+0x12c/0x158
[   59.807772]   __secondary_switched+0xc0/0xc8
[   59.812054] irq event stamp: 279993
[   59.815548] hardirqs last  enabled at (279992): [<ffffffd08d350b64>] el1_interrupt+0x54/0x68
[   59.824018] hardirqs last disabled at (279993): [<ffffffd08d34fc94>] el1_brk64+0x1c/0x50
[   59.832141] softirqs last  enabled at (279934): [<ffffffd08c4c2900>] handle_softirqs+0x508/0x528
[   59.840963] softirqs last disabled at (279959): [<ffffffd08c4103ac>] __do_softirq+0x1c/0x28
[   59.849345] 
[   59.849345] other info that might help us debug this:
[   59.855894]  Possible unsafe locking scenario:
[   59.855894] 
[   59.861831]        CPU0
[   59.864278]        ----
[   59.866724]   lock(&c->lock);
[   59.869696]   <Interrupt>
[   59.872317]     lock(&c->lock);
[   59.875464] 
[   59.875464]  *** DEADLOCK ***
[   59.875464] 
[   59.881402] 11 locks held by swapper/0/0:
[   59.885420]  #0: ffffff8100049948 ((wq_completion)events_bh_highpri){+.-.}-{0:0}, at: process_one_work+0x190/0x5e8
[   59.895818]  #1: ffffffc080003e60 ((work_completion)(&bh->bh)){+.-.}-{0:0}, at: process_one_work+0x1b8/0x5e8
[   59.905691]  #2: ffffff81034d3a30 (&dev->event_lock){+.-.}-{3:3}, at: input_event+0x50/0xb8
[   59.914080]  #3: ffffffd08de9c3c8 (rcu_read_lock){....}-{1:3}, at: input_pass_values+0x38/0x2e8
[   59.922818]  #4: ffffffd08de9c3c8 (rcu_read_lock){....}-{1:3}, at: __handle_sysrq+0x8/0x308
[   59.931206]  #5: ffffffd08de2a8e8 (console_lock){+.+.}-{0:0}, at: vprintk_default+0x40/0x58
[   59.939596]  #6: ffffffd08de2a940 (console_srcu){....}-{0:0}, at: console_flush_all+0x88/0x4a8
[   59.948247]  #7: ffffffd08de2ab00 (console_owner){+.-.}-{0:0}, at: console_lock_spinning_enable+0x50/0x90
[   59.957858]  #8: ffffffd08de2ab28 (printk_legacy_map-wait-type-override){+...}-{4:4}, at: console_flush_all+0x2a0/0x4a8
[   59.968691]  #9: ffffffd02c28e298 (target_list_lock){+.-.}-{3:3}, at: write_msg+0x60/0x1b0 [netconsole]
[   59.978128]  #10: ffffff81fef08f80 (&c->lock){?.-.}-{3:3}, at: ___slab_alloc+0x7c/0x1178
[   59.986256] 
[   59.986256] stack backtrace:
[   59.990624] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G        W           6.17.0-rc2-v8-lockdep #532 PREEMPTLAZY 
[   60.001364] Tainted: [W]=WARN
[   60.004334] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT)
[   60.010185] Call trace:
[   60.012631]  show_stack+0x20/0x38 (C)
[   60.016302]  dump_stack_lvl+0x90/0xd0
[   60.019974]  dump_stack+0x18/0x28
[   60.023297]  print_usage_bug.part.0+0x29c/0x368
[   60.027841]  mark_lock+0x724/0x928
[   60.031251]  mark_held_locks+0x54/0x90
[   60.035010]  lockdep_hardirqs_on_prepare+0x108/0x228
[   60.039992]  arm64_exit_el1_dbg.isra.0+0x30/0x58
[   60.044624]  el1_brk64+0x40/0x50
[   60.047859]  el1h_64_sync_handler+0x68/0xb0
[   60.052053]  el1h_64_sync+0x6c/0x70
[   60.055548]  get_freelist+0x164/0x1d8 (P)
[   60.059569]  ___slab_alloc+0xf8/0x1178
[   60.063328]  __slab_alloc.constprop.0+0x6c/0xc8
[   60.067873]  kmem_cache_alloc_node_noprof+0x2c4/0x328
[   60.072941]  __alloc_skb+0x154/0x188
[   60.076526]  netpoll_send_udp+0xac/0x418
[   60.080460]  write_msg+0x124/0x1b0 [netconsole]
[   60.085004]  console_flush_all+0x2e4/0x4a8
[   60.089112]  console_unlock+0x84/0x188
[   60.092871]  vprintk_emit+0x24c/0x450
[   60.096543]  vprintk_default+0x40/0x58
[   60.100301]  vprintk+0x3c/0x80
[   60.103362]  _printk+0x64/0x98
[   60.106422]  __handle_sysrq+0x1ec/0x308
[   60.110267]  sysrq_filter+0xcc/0x518
[   60.113850]  input_handle_events_filter+0x58/0xc0
[   60.118567]  input_pass_values+0x1fc/0x2e8
[   60.122674]  input_handle_event+0x450/0x558
[   60.126868]  input_event+0x6c/0xb8
[   60.130277]  hidinput_report_event+0x4c/0x78
[   60.134558]  hid_report_raw_event+0xb4/0x418
[   60.138838]  hid_input_report+0x138/0x1a0
[   60.142857]  hid_irq_in+0x180/0x308
[   60.146354]  __usb_hcd_giveback_urb+0x98/0x160
[   60.150810]  usb_giveback_urb_bh+0xb0/0x150
[   60.155004]  process_one_work+0x20c/0x5e8
[   60.159024]  bh_worker+0x1b0/0x218
[   60.162434]  workqueue_softirq_action+0x80/0x98
[   60.166979]  tasklet_hi_action+0x20/0x60
[   60.170913]  handle_softirqs+0x144/0x528
[   60.174846]  __do_softirq+0x1c/0x28
[   60.178342]  ____do_softirq+0x18/0x30
[   60.182012]  call_on_irq_stack+0x30/0x70
[   60.185944]  do_softirq_own_stack+0x24/0x38
[   60.190139]  __irq_exit_rcu+0x100/0x1f8
[   60.193985]  irq_exit_rcu+0x18/0x58
[   60.197482]  el1_interrupt+0x38/0x68
[   60.201064]  el1h_64_irq_handler+0x18/0x28
[   60.205171]  el1h_64_irq+0x6c/0x70
[   60.208579]  default_idle_call+0x84/0x140 (P)
[   60.212947]  do_idle+0x22c/0x298
[   60.216183]  cpu_startup_entry+0x40/0x50
[   60.220117]  rest_init+0x108/0x1a0
[   60.223526]  start_kernel+0x760/0x8c8
[   60.227198]  __primary_switched+0x88/0x98
[   68.638857] sysrq: Trigger a crash
[   68.638885] Kernel panic - not syncing: sysrq triggered crash
[   68.648062] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G        W           6.17.0-rc2-v8-lockdep #532 PREEMPTLAZY 
[   68.658803] Tainted: [W]=WARN
[   68.661774] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT)
[   68.667626] Call trace:
[   68.670073]  show_stack+0x20/0x38 (C)
[   68.673749]  dump_stack_lvl+0x38/0xd0
[   68.677423]  dump_stack+0x18/0x28
[   68.680745]  vpanic+0x340/0x398
[   68.683893]  panic+0x64/0x70

[-- Attachment #4: netconsole-rpi5-6.17-eth0+patch-and+revert.log --]
[-- Type: text/x-log, Size: 3698 bytes --]


Note: applied "net: macb: Fix tx_ptr_lock locking"
[   40.676566] sysrq: HELP : loglevel(0-9) reboot(b) crash(c) show-all-locks(d) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) sak(k) show-backtrace-all-active-cpus(l) show-memory-usage(m) nice-all-RT-tasks(n) poweroff(o) show-registers(p) show-all-timers(q) unraw(r) sync(s) show-task-states(t) unmount(u) force-fb(v) show-blocked-tasks(w) dump-ftrace-buffer(z) replay-kernel-logs(R) 
[   47.612947] sysrq: Trigger a crash
[   47.612967] Kernel panic - not syncing: sysrq triggered crash
[   47.622140] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G        W           6.17.0-rc3-v8-lockdep #540 PREEMPTLAZY 
[   47.632879] Tainted: [W]=WARN
[   47.635849] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT)
[   47.641700] Call trace:
[   47.644147]  show_stack+0x20/0x38 (C)

Note: add revert of "net: cadence: macb: Implement BQL"
[   47.108822] sysrq: HELP : loglevel(0-9) reboot(b) crash(c) show-all-locks(d) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) sak(k) show-backtrace-all-active-cpus(l) show-memory-usage(m) nice-all-RT-tasks(n) poweroff(o) show-registers(p) show-all-timers(q) unraw(r) sync(s) show-task-states(t) unmount(u) force-fb(v) show-blocked-tasks(w) dump-ftrace-buffer(z) replay-kernel-logs(R) 
[   50.132998] sysrq: Trigger a crash
[   50.142195] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G        W           6.17.0-rc3-v8-lockdep #541 PREEMPTLAZY 
[   50.152935] Tainted: [W]=WARN
[   50.155904] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT)
[   50.161755] Call trace:
[   50.164201]  show_stack+0x20/0x38 (C)
[   50.167872]  dump_stack_lvl+0x38/0xd0
[   50.171544]  dump_stack+0x18/0x28
[   50.174865]  vpanic+0x340/0x398
[   50.178012]  panic+0x64/0x70
[   50.180895]  sysrq_handle_crash+0x50/0xa8
[   50.184915]  __handle_sysrq+0xe4/0x308
[   50.188672]  sysrq_filter+0xcc/0x518
[   50.192254]  input_handle_events_filter+0x58/0xc0
[   50.196971]  input_pass_values+0x1fc/0x2e8
[   50.201078]  input_handle_event+0x450/0x558
[   50.205271]  input_event+0x6c/0xb8
[   50.208678]  hidinput_report_event+0x4c/0x78
[   50.212959]  hid_report_raw_event+0xb4/0x418
[   50.217240]  hid_input_report+0x138/0x1a0
[   50.133022] Kernel panic - not syncing: sysrq triggered crash
[   50.221258]  hid_irq_in+0x180/0x308
[   50.224753]  __usb_hcd_giveback_urb+0x98/0x160
[   50.229210]  usb_giveback_urb_bh+0xb0/0x150
[   50.233403]  process_one_work+0x20c/0x5e8
[   50.237422]  bh_worker+0x1b0/0x218
[   50.240830]  workqueue_softirq_action+0x80/0x98
[   50.245374]  tasklet_hi_action+0x20/0x60
[   50.249307]  handle_softirqs+0x144/0x528
[   50.253239]  __do_softirq+0x1c/0x28
[   50.256734]  ____do_softirq+0x18/0x30
[   50.260404]  call_on_irq_stack+0x30/0x70
[   50.264335]  do_softirq_own_stack+0x24/0x38
[   50.268529]  __irq_exit_rcu+0x100/0x1f8
[   50.272374]  irq_exit_rcu+0x18/0x58
[   50.275869]  el1_interrupt+0x38/0x68
[   50.279452]  el1h_64_irq_handler+0x18/0x28
[   50.283557]  el1h_64_irq+0x6c/0x70
[   50.286964]  default_idle_call+0x84/0x140 (P)
[   50.291332]  do_idle+0x22c/0x298
[   50.294567]  cpu_startup_entry+0x40/0x50
[   50.298499]  rest_init+0x108/0x1a0
[   50.301907]  start_kernel+0x760/0x8c8
[   50.305577]  __primary_switched+0x88/0x98
[   50.309598] SMP: stopping secondary CPUs
[   51.358529] SMP: failed to stop secondary CPUs 2
[   51.363720] Kernel Offset: 0x100c400000 from 0xffffffc080000000
[   51.369669] PHYS_OFFSET: 0x0
[   51.372558] CPU features: 0x40000,0000b800,12023140,0400720b
[   51.378241] Memory Limit: none
[   51.381310] ---[ end Kernel panic - not syncing: sysrq triggered crash ]---

Re: netpoll: raspberrypi [4 5] driver locking woes

[Sean Anderson]

[-- Attachment #1: Type: text/plain, Size: 2058 bytes --]

Hi Mike,

On 8/28/25 13:26, Mike Galbraith wrote:
> On Thu, 2025-08-28 at 10:57 -0400, Sean Anderson wrote:
>> Hi Mike,
>> 
>> On 8/27/25 12:02, Mike Galbraith wrote:
>> > Unexpected addendum to done deal datapoint, so off list.
>> > 
>> > On Tue, 2025-08-26 at 11:49 +0200, Mike Galbraith wrote:
>> > > 
>> > > The pi5 gripe fix is equally trivial, but submitting that is pointless
>> > > given there's something else amiss in fingered commit.  This is all of
>> > > the crash info that escapes the box w/wo gripes silenced.
>> > > 
>> > > [   51.688868] sysrq: Trigger a crash
>> > > [   51.688892] Kernel panic - not syncing: sysrq triggered crash
>> > > [   51.698066] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.17.0-rc2-v8-lockdep #533 PREEMPTLAZY
>> > > [   51.707234] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT)
>> > > [   51.713085] Call trace:
>> > > [   51.715532]  show_stack+0x20/0x38 (C)
>> > > [   51.719206]  dump_stack_lvl+0x38/0xd0
>> > > [   51.722878]  dump_stack+0x18/0x28
>> > > 
>> > > That aspect is a punt and run atm (time.. and dash of laziness:).
>> > 
>> > Plan was to end datapoint thread, but after booting pi5's 6.12 kernel,
>> > for some reason I fired up netconsole.. and box promptly exhibited the
>> > netpoll locking bug warning, indicating presence of 138badbc21a0. 
>> > Instead of saying to self "nope, just walk away", I poked SysRq-C.. and
>> > the bloody damn monitoring box received a 100% complete death rattle. 
>> > Well bugger.
>> 
>> Did you get a backtrace for this?
> 
> Yes, logs for 6.12.41 and 6.17.0-rc2 attached.
> 
> Since a patch has meanwhile landed, also a log of patched 6.17.0-rc3
> now gripe free (yay) but with aforementioned broken output, followed by
> addition of the e6a532185daa revert to confirm it still cures that.
> 
>> And to be clear, the steps to reproduce this are to boot a kernel with
>> lockdep enabled with netconsole on macb and then hit sysrq?
> 
> Yup.

Looks like the tx completion path can also be called from netpoll. Can
you try the attached patch?

--Sean

[-- Attachment #2: 0001-net-macb-Fix-tx_ptr_lock-locking.patch --]
[-- Type: text/x-patch, Size: 4585 bytes --]

From 3fe439760955370b9f005ef2728b5fc9b14eeea5 Mon Sep 17 00:00:00 2001
From: Sean Anderson <sean.anderson@linux.dev>
Date: Thu, 28 Aug 2025 11:55:01 -0400
Subject: [PATCH net v2] net: macb: Fix tx_ptr_lock locking

macb_start_xmit can be called with bottom-halves disabled (e.g.
transmitting from softirqs) as well as with interrupts disabled (with
netpoll). Because of this, all other functions taking tx_ptr_lock must
disable IRQs, and macb_start_xmit must only re-enable IRQs if they
were already enabled.

Fixes: 138badbc21a0 ("net: macb: use NAPI for TX completion path")
Reported-by: Mike Galbraith <efault@gmx.de>
Signed-off-by: Sean Anderson <sean.anderson@linux.dev>
---

Changes in v2:
- Use irqsave/restore for all accesses, since they can also also be
  called from netpoll.

 drivers/net/ethernet/cadence/macb_main.c | 28 ++++++++++++++----------
 1 file changed, 16 insertions(+), 12 deletions(-)

diff --git a/drivers/net/ethernet/cadence/macb_main.c b/drivers/net/ethernet/cadence/macb_main.c
index 16d28a8b3b56..c769b7dbd3ba 100644
--- a/drivers/net/ethernet/cadence/macb_main.c
+++ b/drivers/net/ethernet/cadence/macb_main.c
@@ -1223,12 +1223,13 @@ static int macb_tx_complete(struct macb_queue *queue, int budget)
 {
 	struct macb *bp = queue->bp;
 	u16 queue_index = queue - bp->queues;
+	unsigned long flags;
 	unsigned int tail;
 	unsigned int head;
 	int packets = 0;
 	u32 bytes = 0;
 
-	spin_lock(&queue->tx_ptr_lock);
+	spin_lock_irqsave(&queue->tx_ptr_lock, flags);
 	head = queue->tx_head;
 	for (tail = queue->tx_tail; tail != head && packets < budget; tail++) {
 		struct macb_tx_skb	*tx_skb;
@@ -1291,7 +1292,7 @@ static int macb_tx_complete(struct macb_queue *queue, int budget)
 	    CIRC_CNT(queue->tx_head, queue->tx_tail,
 		     bp->tx_ring_size) <= MACB_TX_WAKEUP_THRESH(bp))
 		netif_wake_subqueue(bp->dev, queue_index);
-	spin_unlock(&queue->tx_ptr_lock);
+	spin_unlock_irqrestore(&queue->tx_ptr_lock, flags);
 
 	return packets;
 }
@@ -1707,8 +1708,9 @@ static void macb_tx_restart(struct macb_queue *queue)
 {
 	struct macb *bp = queue->bp;
 	unsigned int head_idx, tbqp;
+	unsigned long flags;
 
-	spin_lock(&queue->tx_ptr_lock);
+	spin_lock_irqsave(&queue->tx_ptr_lock, flags);
 
 	if (queue->tx_head == queue->tx_tail)
 		goto out_tx_ptr_unlock;
@@ -1720,19 +1722,20 @@ static void macb_tx_restart(struct macb_queue *queue)
 	if (tbqp == head_idx)
 		goto out_tx_ptr_unlock;
 
-	spin_lock_irq(&bp->lock);
+	spin_lock(&bp->lock);
 	macb_writel(bp, NCR, macb_readl(bp, NCR) | MACB_BIT(TSTART));
-	spin_unlock_irq(&bp->lock);
+	spin_unlock(&bp->lock);
 
 out_tx_ptr_unlock:
-	spin_unlock(&queue->tx_ptr_lock);
+	spin_unlock_irqrestore(&queue->tx_ptr_lock, flags);
 }
 
 static bool macb_tx_complete_pending(struct macb_queue *queue)
 {
 	bool retval = false;
+	unsigned long flags;
 
-	spin_lock(&queue->tx_ptr_lock);
+	spin_lock_irqsave(&queue->tx_ptr_lock, flags);
 	if (queue->tx_head != queue->tx_tail) {
 		/* Make hw descriptor updates visible to CPU */
 		rmb();
@@ -1740,7 +1743,7 @@ static bool macb_tx_complete_pending(struct macb_queue *queue)
 		if (macb_tx_desc(queue, queue->tx_tail)->ctrl & MACB_BIT(TX_USED))
 			retval = true;
 	}
-	spin_unlock(&queue->tx_ptr_lock);
+	spin_unlock_irqrestore(&queue->tx_ptr_lock, flags);
 	return retval;
 }
 
@@ -2308,6 +2311,7 @@ static netdev_tx_t macb_start_xmit(struct sk_buff *skb, struct net_device *dev)
 	struct macb_queue *queue = &bp->queues[queue_index];
 	unsigned int desc_cnt, nr_frags, frag_size, f;
 	unsigned int hdrlen;
+	unsigned long flags;
 	bool is_lso;
 	netdev_tx_t ret = NETDEV_TX_OK;
 
@@ -2368,7 +2372,7 @@ static netdev_tx_t macb_start_xmit(struct sk_buff *skb, struct net_device *dev)
 		desc_cnt += DIV_ROUND_UP(frag_size, bp->max_tx_length);
 	}
 
-	spin_lock_bh(&queue->tx_ptr_lock);
+	spin_lock_irqsave(&queue->tx_ptr_lock, flags);
 
 	/* This is a hard error, log it. */
 	if (CIRC_SPACE(queue->tx_head, queue->tx_tail,
@@ -2392,15 +2396,15 @@ static netdev_tx_t macb_start_xmit(struct sk_buff *skb, struct net_device *dev)
 	netdev_tx_sent_queue(netdev_get_tx_queue(bp->dev, queue_index),
 			     skb->len);
 
-	spin_lock_irq(&bp->lock);
+	spin_lock(&bp->lock);
 	macb_writel(bp, NCR, macb_readl(bp, NCR) | MACB_BIT(TSTART));
-	spin_unlock_irq(&bp->lock);
+	spin_unlock(&bp->lock);
 
 	if (CIRC_SPACE(queue->tx_head, queue->tx_tail, bp->tx_ring_size) < 1)
 		netif_stop_subqueue(dev, queue_index);
 
 unlock:
-	spin_unlock_bh(&queue->tx_ptr_lock);
+	spin_unlock_irqrestore(&queue->tx_ptr_lock, flags);
 
 	return ret;
 }
-- 
2.35.1.1320.gc452695387.dirty

Re: netpoll: raspberrypi [4 5] driver locking woes

[Mike Galbraith]

On Thu, 2025-08-28 at 14:56 -0400, Sean Anderson wrote:
> 
> Looks like the tx completion path can also be called from netpoll. Can
> you try the attached patch?

Done, zero netpoll or lockdep complaints.

	-Mike