* netpoll: raspberrypi [4 5] driver locking woes @ 2025-08-25 5:53 Mike Galbraith 2025-08-25 10:10 ` Mike Galbraith 2025-08-26 9:49 ` Mike Galbraith 0 siblings, 2 replies; 7+ messages in thread From: Mike Galbraith @ 2025-08-25 5:53 UTC (permalink / raw) To: lkml; +Cc: Robert Hancock, Doug Berger, Florian Fainelli, Breno Leitao Greetings, After meeting some netconsole+ieee80211 woes recently, I decided to build a lockdep enabled 6.17-rc2 kernel for my rpi[45] and give it a go on their wired nics just to make sure there were no surprises lurking.. but damn the bad luck, both cute/dinky boxen whimpered. The pi5 met locking woes due to... 138badbc21a0 net: macb: use NAPI for TX completion path ...injecting spin_lock/unlock_irq() into polling, which inspired netpoll.c to point a finger directly at the driver. A quick poke didn't produce instant gratification, but a wedged in revert did. On the pi4, lockdep muttered about ring->lock in bcmgenet_xmit() and bcmgenet_tx_poll() - switching to irqsave/restore spinlock variants there silenced it. Note to readers: before turning on lockdep, perhaps consider whacking lockdep_assert_held() instances in gpiolib.c.. ymmv, but here it would have saved time and naughty language expenditure. -Mike ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: netpoll: raspberrypi [4 5] driver locking woes 2025-08-25 5:53 netpoll: raspberrypi [4 5] driver locking woes Mike Galbraith @ 2025-08-25 10:10 ` Mike Galbraith 2025-08-26 9:49 ` Mike Galbraith 1 sibling, 0 replies; 7+ messages in thread From: Mike Galbraith @ 2025-08-25 10:10 UTC (permalink / raw) To: lkml; +Cc: Robert Hancock, Doug Berger, Florian Fainelli, Breno Leitao On Mon, 2025-08-25 at 07:53 +0200, Mike Galbraith wrote: > > On the pi4, lockdep muttered about ring->lock in bcmgenet_xmit() and > bcmgenet_tx_poll() - switching to irqsave/restore spinlock variants > there silenced it. The deadlock splat below. [ 90.529661] ===================================================== [ 90.535817] WARNING: HARDIRQ-safe -> HARDIRQ-unsafe lock order detected [ 90.542508] 6.17.0-rc2-v8-lockdep #529 Tainted: G C [ 90.548929] ----------------------------------------------------- [ 90.555092] swapper/0/0 [HC0[0]:SC1[1]:HE0:SE0] is trying to acquire: [ 90.561605] ffffff8102518d48 (&ring->lock){+.-.}-{3:3}, at: bcmgenet_xmit+0x78/0x860 [ 90.569439] [ 90.569439] and this task is already holding: [ 90.575341] ffffffd6357c3298 (target_list_lock){..-.}-{3:3}, at: write_msg+0x60/0x1b0 [netconsole] [ 90.584408] which would create a new lock dependency: [ 90.589507] (target_list_lock){..-.}-{3:3} -> (&ring->lock){+.-.}-{3:3} [ 90.596283] [ 90.596283] but this new dependency connects a HARDIRQ-irq-safe lock: [ 90.604305] (&host->lock){-.-.}-{3:3} [ 90.604322] [ 90.604322] ... which became HARDIRQ-irq-safe at: [ 90.614339] lock_acquire+0x1d4/0x368 [ 90.618124] _raw_spin_lock+0x50/0x70 [ 90.621899] bcm2835_mmc_irq+0x28/0x8c8 [ 90.625870] __handle_irq_event_percpu+0x94/0x318 [ 90.630699] handle_irq_event+0x54/0xb8 [ 90.634668] handle_fasteoi_irq+0x12c/0x270 [ 90.638980] handle_irq_desc+0x48/0x68 [ 90.642846] generic_handle_domain_irq+0x24/0x38 [ 90.647603] gic_handle_irq+0x48/0xd8 [ 90.651391] call_on_irq_stack+0x30/0x70 [ 90.655435] do_interrupt_handler+0xdc/0xe8 [ 90.659762] el1_interrupt+0x34/0x68 [ 90.663450] el1h_64_irq_handler+0x18/0x28 [ 90.667674] el1h_64_irq+0x6c/0x70 [ 90.671195] d_alloc_parallel+0x3dc/0x9e8 [ 90.675332] __lookup_slow+0x98/0x188 [ 90.679112] lookup_noperm+0x74/0x88 [ 90.682810] simple_start_creating+0x70/0xe8 [ 90.687209] start_creating+0x7c/0x110 [ 90.691082] __debugfs_create_file+0x54/0x230 [ 90.695577] debugfs_create_file_unsafe+0x28/0x40 [ 90.700413] debugfs_create_u32+0x78/0x90 [ 90.704562] link_and_create_debugfs+0x78/0xf8 [ 90.709127] _regulator_get_common+0x14c/0x398 [ 90.713716] _regulator_get+0x48/0xa0 [ 90.717489] _devm_regulator_get+0x60/0xb0 [ 90.721720] devm_regulator_get_optional+0x1c/0x30 [ 90.726649] mmc_regulator_get_supply+0x48/0x1c8 [ 90.731402] sdhci_setup_host+0x904/0xd30 [ 90.735534] sdhci_add_host+0x20/0x60 [ 90.739325] sdhci_iproc_probe+0xc8/0x108 [ 90.743462] platform_probe+0x64/0xa8 [ 90.747246] really_probe+0xc4/0x2a8 [ 90.750937] __driver_probe_device+0x80/0x140 [ 90.755432] driver_probe_device+0xe0/0x170 [ 90.759739] __device_attach_driver+0xc0/0x148 [ 90.764322] bus_for_each_drv+0x88/0xf0 [ 90.768282] __device_attach_async_helper+0xb8/0xe8 [ 90.773300] async_run_entry_fn+0x3c/0xf0 [ 90.777436] process_one_work+0x20c/0x5e8 [ 90.781573] worker_thread+0x1c8/0x378 [ 90.785448] kthread+0x168/0x230 [ 90.788792] ret_from_fork+0x10/0x20 [ 90.792488] [ 90.792488] to a HARDIRQ-irq-unsafe lock: [ 90.798033] (&ring->lock){+.-.}-{3:3} [ 90.798051] [ 90.798051] ... which became HARDIRQ-irq-unsafe at: [ 90.808240] ... [ 90.808245] lock_acquire+0x1d4/0x368 [ 90.813784] _raw_spin_lock+0x50/0x70 [ 90.817570] bcmgenet_xmit+0x78/0x860 [ 90.821359] dev_hard_start_xmit+0xc0/0x2d0 [ 90.825671] sch_direct_xmit+0x98/0x1e8 [ 90.829633] __dev_queue_xmit+0x5a0/0x13a8 [ 90.833856] packet_xmit+0xcc/0x150 [ 90.837465] packet_sendmsg+0x8e8/0x1328 [ 90.841515] __sock_sendmsg+0x64/0xc0 [ 90.845295] ____sys_sendmsg+0x270/0x2b0 [ 90.849343] ___sys_sendmsg+0x88/0xf0 [ 90.853133] __sys_sendmsg+0x88/0x100 [ 90.856913] __arm64_sys_sendmsg+0x2c/0x40 [ 90.861138] invoke_syscall+0x50/0x120 [ 90.865011] el0_svc_common.constprop.0+0x48/0xf0 [ 90.869851] do_el0_svc+0x24/0x38 [ 90.873290] el0_svc+0x4c/0x178 [ 90.876543] el0t_64_sync_handler+0xa0/0xe8 [ 90.880858] el0t_64_sync+0x198/0x1a0 [ 90.884644] [ 90.884644] other info that might help us debug this: [ 90.884644] [ 90.892742] Chain exists of: [ 90.892742] &host->lock --> target_list_lock --> &ring->lock [ 90.892742] [ 90.903035] Possible interrupt unsafe locking scenario: [ 90.903035] [ 90.909901] CPU0 CPU1 [ 90.914482] ---- ---- [ 90.919060] lock(&ring->lock); [ 90.922311] local_irq_disable(); [ 90.928297] lock(&host->lock); [ 90.934107] lock(target_list_lock); [ 90.940356] <Interrupt> [ 90.943001] lock(&host->lock); [ 90.946430] [ 90.946430] *** DEADLOCK *** [ 90.946430] [ 90.952419] 12 locks held by swapper/0/0: [ 90.956468] #0: ffffff8100049948 ((wq_completion)events_bh_highpri){+.-.}-{0:0}, at: process_one_work+0x190/0x5e8 [ 90.966944] #1: ffffffc080003e60 ((work_completion)(&bh->bh)){+.-.}-{0:0}, at: process_one_work+0x1b8/0x5e8 [ 90.976885] #2: ffffff81082f2230 (&dev->event_lock){..-.}-{3:3}, at: input_event+0x50/0xb8 [ 90.985339] #3: ffffffd6a169c3c8 (rcu_read_lock){....}-{1:3}, at: input_pass_values+0x38/0x2e8 [ 90.994141] #4: ffffffd6a169c3c8 (rcu_read_lock){....}-{1:3}, at: __handle_sysrq+0x8/0x308 [ 91.002591] #5: ffffffd6a162a8e8 (console_lock){+.+.}-{0:0}, at: vprintk_default+0x40/0x58 [ 91.011041] #6: ffffffd6a162a940 (console_srcu){....}-{0:0}, at: console_flush_all+0x88/0x4a8 [ 91.019751] #7: ffffffd6a162ab00 (console_owner){..-.}-{0:0}, at: console_lock_spinning_enable+0x50/0x90 [ 91.029437] #8: ffffffd6a162ab28 (printk_legacy_map-wait-type-override){....}-{4:4}, at: console_flush_all+0x2a0/0x4a8 [ 91.040346] #9: ffffffd6357c3298 (target_list_lock){..-.}-{3:3}, at: write_msg+0x60/0x1b0 [netconsole] [ 91.049854] #10: ffffffd6a169c3c8 (rcu_read_lock){....}-{1:3}, at: netpoll_send_skb+0x4c/0x4b0 [ 91.058655] #11: ffffff8108050118 (_xmit_ETHER#2){+.-.}-{3:3}, at: netpoll_send_skb+0x3e4/0x4b0 [ 91.067545] [ 91.067545] the dependencies between HARDIRQ-irq-safe lock and the holding lock: [ 91.076524] -> (&host->lock){-.-.}-{3:3} { [ 91.080836] IN-HARDIRQ-W at: [ 91.084181] lock_acquire+0x1d4/0x368 [ 91.089907] _raw_spin_lock+0x50/0x70 [ 91.095623] bcm2835_mmc_irq+0x28/0x8c8 [ 91.101520] __handle_irq_event_percpu+0x94/0x318 [ 91.108301] handle_irq_event+0x54/0xb8 [ 91.114198] handle_fasteoi_irq+0x12c/0x270 [ 91.120450] handle_irq_desc+0x48/0x68 [ 91.126259] generic_handle_domain_irq+0x24/0x38 [ 91.132944] gic_handle_irq+0x48/0xd8 [ 91.138669] call_on_irq_stack+0x30/0x70 [ 91.144655] do_interrupt_handler+0xdc/0xe8 [ 91.150904] el1_interrupt+0x34/0x68 [ 91.156538] el1h_64_irq_handler+0x18/0x28 [ 91.162699] el1h_64_irq+0x6c/0x70 [ 91.168156] d_alloc_parallel+0x3dc/0x9e8 [ 91.174230] __lookup_slow+0x98/0x188 [ 91.179951] lookup_noperm+0x74/0x88 [ 91.185582] simple_start_creating+0x70/0xe8 [ 91.191919] start_creating+0x7c/0x110 [ 91.197739] __debugfs_create_file+0x54/0x230 [ 91.204164] debugfs_create_file_unsafe+0x28/0x40 [ 91.210927] debugfs_create_u32+0x78/0x90 [ 91.217009] link_and_create_debugfs+0x78/0xf8 [ 91.223517] _regulator_get_common+0x14c/0x398 [ 91.230028] _regulator_get+0x48/0xa0 [ 91.235752] _devm_regulator_get+0x60/0xb0 [ 91.241910] devm_regulator_get_optional+0x1c/0x30 [ 91.248773] mmc_regulator_get_supply+0x48/0x1c8 [ 91.255461] sdhci_setup_host+0x904/0xd30 [ 91.261535] sdhci_add_host+0x20/0x60 [ 91.267260] sdhci_iproc_probe+0xc8/0x108 [ 91.273330] platform_probe+0x64/0xa8 [ 91.279058] really_probe+0xc4/0x2a8 [ 91.284684] __driver_probe_device+0x80/0x140 [ 91.291118] driver_probe_device+0xe0/0x170 [ 91.297367] __device_attach_driver+0xc0/0x148 [ 91.303881] bus_for_each_drv+0x88/0xf0 [ 91.309781] __device_attach_async_helper+0xb8/0xe8 [ 91.316734] async_run_entry_fn+0x3c/0xf0 [ 91.322804] process_one_work+0x20c/0x5e8 [ 91.328878] worker_thread+0x1c8/0x378 [ 91.334686] kthread+0x168/0x230 [ 91.339969] ret_from_fork+0x10/0x20 [ 91.345605] IN-SOFTIRQ-W at: [ 91.348943] lock_acquire+0x1d4/0x368 [ 91.354671] _raw_spin_lock_irqsave+0x60/0x90 [ 91.361097] bcm2835_mmc_tasklet_finish+0x20/0x100 [ 91.367962] tasklet_action_common+0x1a4/0x1c8 [ 91.374476] tasklet_action+0x48/0x60 [ 91.380187] handle_softirqs+0x144/0x528 [ 91.386175] __do_softirq+0x1c/0x28 [ 91.391725] ____do_softirq+0x18/0x30 [ 91.397444] call_on_irq_stack+0x30/0x70 [ 91.403434] do_softirq_own_stack+0x24/0x38 [ 91.409679] __irq_exit_rcu+0x100/0x1f8 [ 91.415580] irq_exit_rcu+0x18/0x58 [ 91.421126] el1_interrupt+0x38/0x68 [ 91.426759] el1h_64_irq_handler+0x18/0x28 [ 91.432921] el1h_64_irq+0x6c/0x70 [ 91.438377] _raw_spin_unlock_irqrestore+0x6c/0xa0 [ 91.445237] bcm2835_mmc_request+0x64/0xc8 [ 91.451405] __mmc_start_request+0x7c/0x158 [ 91.457654] mmc_start_request+0xa0/0xf0 [ 91.463640] mmc_wait_for_req+0x78/0x110 [ 91.469620] mmc_wait_for_cmd+0x6c/0xb0 [ 91.475522] sdio_reset+0x60/0x138 [ 91.480972] mmc_rescan+0x2bc/0x338 [ 91.486518] process_one_work+0x20c/0x5e8 [ 91.492594] worker_thread+0x1c8/0x378 [ 91.498402] kthread+0x168/0x230 [ 91.503683] ret_from_fork+0x10/0x20 [ 91.509316] INITIAL USE at: [ 91.512586] lock_acquire+0x1d4/0x368 [ 91.518205] _raw_spin_lock_irqsave+0x60/0x90 [ 91.524557] bcm2835_mmc_reset+0x34/0x238 [ 91.530529] bcm2835_mmc_probe+0x1a0/0x730 [ 91.536616] platform_probe+0x64/0xa8 [ 91.542248] really_probe+0xc4/0x2a8 [ 91.547794] __driver_probe_device+0x80/0x140 [ 91.554131] driver_probe_device+0xe0/0x170 [ 91.560293] __device_attach_driver+0xc0/0x148 [ 91.566719] bus_for_each_drv+0x88/0xf0 [ 91.572514] __device_attach+0xa4/0x198 [ 91.578336] device_initial_probe+0x1c/0x30 [ 91.584498] bus_probe_device+0xb4/0xc0 [ 91.590308] deferred_probe_work_func+0x90/0xd8 [ 91.596822] process_one_work+0x20c/0x5e8 [ 91.602793] worker_thread+0x1c8/0x378 [ 91.608528] kthread+0x168/0x230 [ 91.613721] ret_from_fork+0x10/0x20 [ 91.619253] } [ 91.621101] ... key at: [<ffffffd6a234d8d8>] __key.2+0x0/0x10 [ 91.627629] -> (console_owner){..-.}-{0:0} { [ 91.632029] IN-SOFTIRQ-W at: [ 91.635272] lock_acquire+0x1d4/0x368 [ 91.640831] console_lock_spinning_enable+0x74/0x90 [ 91.647609] console_flush_all+0x2ac/0x4a8 [ 91.653594] console_unlock+0x84/0x188 [ 91.659228] vprintk_emit+0x24c/0x450 [ 91.664772] vprintk_default+0x40/0x58 [ 91.670406] vprintk+0x3c/0x80 [ 91.675335] _printk+0x64/0x98 [ 91.680264] __handle_sysrq+0x1ec/0x308 [ 91.685985] sysrq_filter+0xcc/0x518 [ 91.691442] input_handle_events_filter+0x58/0xc0 [ 91.698030] input_pass_values+0x1fc/0x2e8 [ 91.704029] input_handle_event+0x450/0x558 [ 91.710090] input_event+0x6c/0xb8 [ 91.715384] hidinput_report_event+0x4c/0x78 [ 91.721545] hid_report_raw_event+0xb4/0x418 [ 91.727707] hid_input_report+0x138/0x1a0 [ 91.733604] hid_irq_in+0x180/0x308 [ 91.738959] __usb_hcd_giveback_urb+0x98/0x160 [ 91.745310] usb_giveback_urb_bh+0xb0/0x150 [ 91.751371] process_one_work+0x20c/0x5e8 [ 91.757281] bh_worker+0x1b0/0x218 [ 91.762563] workqueue_softirq_action+0x80/0x98 [ 91.768988] tasklet_hi_action+0x20/0x60 [ 91.774784] handle_softirqs+0x144/0x528 [ 91.780607] __do_softirq+0x1c/0x28 [ 91.785976] ____do_softirq+0x18/0x30 [ 91.791522] call_on_irq_stack+0x30/0x70 [ 91.797331] do_softirq_own_stack+0x24/0x38 [ 91.803404] __irq_exit_rcu+0x100/0x1f8 [ 91.809113] irq_exit_rcu+0x18/0x58 [ 91.814494] el1_interrupt+0x38/0x68 [ 91.819951] el1h_64_irq_handler+0x18/0x28 [ 91.825938] el1h_64_irq+0x6c/0x70 [ 91.831219] default_idle_call+0x84/0x140 [ 91.837104] do_idle+0x22c/0x298 [ 91.842208] cpu_startup_entry+0x3c/0x50 [ 91.848030] rest_init+0x108/0x1a0 [ 91.853312] start_kernel+0x760/0x8c8 [ 91.858856] __primary_switched+0x88/0x98 [ 91.864742] INITIAL USE at: [ 91.867923] lock_acquire+0x1d4/0x368 [ 91.873368] console_lock_spinning_enable+0x74/0x90 [ 91.880063] console_flush_all+0x2ac/0x4a8 [ 91.885955] console_unlock+0x84/0x188 [ 91.891506] vprintk_emit+0x24c/0x450 [ 91.896963] vprintk_default+0x40/0x58 [ 91.902508] vprintk+0x3c/0x80 [ 91.907349] _printk+0x64/0x98 [ 91.912191] register_console+0x3a8/0x4e0 [ 91.918000] con_init+0x1c8/0x270 [ 91.923106] console_init+0x98/0x2a0 [ 91.928474] start_kernel+0x4e8/0x8c8 [ 91.933931] __primary_switched+0x88/0x98 [ 91.939741] } [ 91.941495] ... key at: [<ffffffd6a162ab00>] console_owner_dep_map+0x0/0x28 [ 91.949158] ... acquired at: [ 91.952233] console_lock_spinning_enable+0x74/0x90 [ 91.957345] console_flush_all+0x2ac/0x4a8 [ 91.961664] console_unlock+0x84/0x188 [ 91.965625] vprintk_emit+0x24c/0x450 [ 91.969485] vprintk_default+0x40/0x58 [ 91.973447] vprintk+0x3c/0x80 [ 91.976703] _printk+0x64/0x98 [ 91.979964] bcm2835_mmc_send_command+0x944/0xa08 [ 91.984894] bcm2835_mmc_request+0x58/0xc8 [ 91.989207] __mmc_start_request+0x7c/0x158 [ 91.993603] mmc_start_request+0xa0/0xf0 [ 91.997745] mmc_wait_for_req+0x78/0x110 [ 92.001876] mmc_wait_for_cmd+0x6c/0xb0 [ 92.005931] sdio_reset+0x60/0x138 [ 92.009547] mmc_rescan+0x2bc/0x338 [ 92.013231] process_one_work+0x20c/0x5e8 [ 92.017461] worker_thread+0x1c8/0x378 [ 92.021421] kthread+0x168/0x230 [ 92.024855] ret_from_fork+0x10/0x20 [ 92.028640] [ 92.030136] -> (target_list_lock){..-.}-{3:3} { [ 92.034720] IN-SOFTIRQ-W at: [ 92.037882] lock_acquire+0x1d4/0x368 [ 92.043251] _raw_spin_lock_irqsave+0x60/0x90 [ 92.049324] write_msg+0x60/0x1b0 [netconsole] [ 92.055486] console_flush_all+0x2e4/0x4a8 [ 92.061296] console_unlock+0x84/0x188 [ 92.066753] vprintk_emit+0x24c/0x450 [ 92.072122] vprintk_default+0x40/0x58 [ 92.077579] vprintk+0x3c/0x80 [ 92.082332] _printk+0x64/0x98 [ 92.087091] __handle_sysrq+0x1ec/0x308 [ 92.092636] sysrq_filter+0xcc/0x518 [ 92.097911] input_handle_events_filter+0x58/0xc0 [ 92.104337] input_pass_values+0x1fc/0x2e8 [ 92.110152] input_handle_event+0x450/0x558 [ 92.116044] input_event+0x6c/0xb8 [ 92.121144] hidinput_report_event+0x4c/0x78 [ 92.127129] hid_report_raw_event+0xb4/0x418 [ 92.133120] hid_input_report+0x138/0x1a0 [ 92.138835] hid_irq_in+0x180/0x308 [ 92.144034] __usb_hcd_giveback_urb+0x98/0x160 [ 92.150189] usb_giveback_urb_bh+0xb0/0x150 [ 92.156093] process_one_work+0x20c/0x5e8 [ 92.161815] bh_worker+0x1b0/0x218 [ 92.166919] workqueue_softirq_action+0x80/0x98 [ 92.173168] tasklet_hi_action+0x20/0x60 [ 92.178802] handle_softirqs+0x144/0x528 [ 92.184435] __do_softirq+0x1c/0x28 [ 92.189629] ____do_softirq+0x18/0x30 [ 92.194993] call_on_irq_stack+0x30/0x70 [ 92.200630] do_softirq_own_stack+0x24/0x38 [ 92.206529] __irq_exit_rcu+0x100/0x1f8 [ 92.212074] irq_exit_rcu+0x18/0x58 [ 92.217267] el1_interrupt+0x38/0x68 [ 92.222548] el1h_64_irq_handler+0x18/0x28 [ 92.228357] el1h_64_irq+0x6c/0x70 [ 92.233462] default_idle_call+0x84/0x140 [ 92.239184] do_idle+0x22c/0x298 [ 92.244112] cpu_startup_entry+0x3c/0x50 [ 92.249746] rest_init+0x108/0x1a0 [ 92.254852] start_kernel+0x760/0x8c8 [ 92.260221] __primary_switched+0x88/0x98 [ 92.265942] INITIAL USE at: [ 92.269022] lock_acquire+0x1d4/0x368 [ 92.274303] _raw_spin_lock_irqsave+0x60/0x90 [ 92.280289] make_netconsole_target+0xe4/0x1d0 [netconsole] [ 92.287507] configfs_mkdir+0x184/0x4c0 [ 92.292964] vfs_mkdir+0x1a8/0x280 [ 92.297981] do_mkdirat+0x140/0x180 [ 92.303086] __arm64_sys_mkdirat+0x38/0x58 [ 92.308807] invoke_syscall+0x50/0x120 [ 92.314173] el0_svc_common.constprop.0+0x48/0xf0 [ 92.320514] do_el0_svc+0x24/0x38 [ 92.325444] el0_svc+0x4c/0x178 [ 92.330195] el0t_64_sync_handler+0xa0/0xe8 [ 92.336005] el0t_64_sync+0x198/0x1a0 [ 92.341282] } [ 92.342961] ... key at: [<ffffffd6357c3298>] target_list_lock+0x18/0xffffffffffffed80 [netconsole] [ 92.352553] ... acquired at: [ 92.355542] _raw_spin_lock_irqsave+0x60/0x90 [ 92.360123] write_msg+0x60/0x1b0 [netconsole] [ 92.364789] console_flush_all+0x2e4/0x4a8 [ 92.369099] console_unlock+0x84/0x188 [ 92.373062] vprintk_emit+0x24c/0x450 [ 92.376932] vprintk_default+0x40/0x58 [ 92.380894] vprintk+0x3c/0x80 [ 92.384150] _printk+0x64/0x98 [ 92.387407] __handle_sysrq+0x1ec/0x308 [ 92.391458] sysrq_filter+0xcc/0x518 [ 92.395242] input_handle_events_filter+0x58/0xc0 [ 92.400172] input_pass_values+0x1fc/0x2e8 [ 92.404486] input_handle_event+0x450/0x558 [ 92.408885] input_event+0x6c/0xb8 [ 92.412495] hidinput_report_event+0x4c/0x78 [ 92.416984] hid_report_raw_event+0xb4/0x418 [ 92.421471] hid_input_report+0x138/0x1a0 [ 92.425697] hid_irq_in+0x180/0x308 [ 92.429394] __usb_hcd_giveback_urb+0x98/0x160 [ 92.434059] usb_giveback_urb_bh+0xb0/0x150 [ 92.438461] process_one_work+0x20c/0x5e8 [ 92.442687] bh_worker+0x1b0/0x218 [ 92.446294] workqueue_softirq_action+0x80/0x98 [ 92.451050] tasklet_hi_action+0x20/0x60 [ 92.455185] handle_softirqs+0x144/0x528 [ 92.459322] __do_softirq+0x1c/0x28 [ 92.463020] ____do_softirq+0x18/0x30 [ 92.466892] call_on_irq_stack+0x30/0x70 [ 92.471030] do_softirq_own_stack+0x24/0x38 [ 92.475429] __irq_exit_rcu+0x100/0x1f8 [ 92.479478] irq_exit_rcu+0x18/0x58 [ 92.483177] el1_interrupt+0x38/0x68 [ 92.486960] el1h_64_irq_handler+0x18/0x28 [ 92.491273] el1h_64_irq+0x6c/0x70 [ 92.494882] default_idle_call+0x84/0x140 [ 92.499107] do_idle+0x22c/0x298 [ 92.502541] cpu_startup_entry+0x3c/0x50 [ 92.506677] rest_init+0x108/0x1a0 [ 92.510287] start_kernel+0x760/0x8c8 [ 92.514160] __primary_switched+0x88/0x98 [ 92.518383] [ 92.519881] [ 92.519881] the dependencies between the lock to be acquired [ 92.519887] and HARDIRQ-irq-unsafe lock: [ 92.531155] -> (&ring->lock){+.-.}-{3:3} { [ 92.535283] HARDIRQ-ON-W at: [ 92.538452] lock_acquire+0x1d4/0x368 [ 92.543821] _raw_spin_lock+0x50/0x70 [ 92.549190] bcmgenet_xmit+0x78/0x860 [ 92.554559] dev_hard_start_xmit+0xc0/0x2d0 [ 92.560458] sch_direct_xmit+0x98/0x1e8 [ 92.566001] __dev_queue_xmit+0x5a0/0x13a8 [ 92.571813] packet_xmit+0xcc/0x150 [ 92.577004] packet_sendmsg+0x8e8/0x1328 [ 92.582637] __sock_sendmsg+0x64/0xc0 [ 92.588006] ____sys_sendmsg+0x270/0x2b0 [ 92.593640] ___sys_sendmsg+0x88/0xf0 [ 92.599009] __sys_sendmsg+0x88/0x100 [ 92.604380] __arm64_sys_sendmsg+0x2c/0x40 [ 92.610188] invoke_syscall+0x50/0x120 [ 92.615644] el0_svc_common.constprop.0+0x48/0xf0 [ 92.622072] do_el0_svc+0x24/0x38 [ 92.627088] el0_svc+0x4c/0x178 [ 92.631929] el0t_64_sync_handler+0xa0/0xe8 [ 92.637827] el0t_64_sync+0x198/0x1a0 [ 92.643195] IN-SOFTIRQ-W at: [ 92.646364] lock_acquire+0x1d4/0x368 [ 92.651733] _raw_spin_lock+0x50/0x70 [ 92.657104] bcmgenet_tx_poll+0x30/0xe8 [ 92.662647] __napi_poll.constprop.0+0x44/0x268 [ 92.668898] net_rx_action+0x320/0x390 [ 92.674355] handle_softirqs+0x144/0x528 [ 92.679987] __do_softirq+0x1c/0x28 [ 92.685178] ____do_softirq+0x18/0x30 [ 92.690549] call_on_irq_stack+0x30/0x70 [ 92.696183] do_softirq_own_stack+0x24/0x38 [ 92.702079] __irq_exit_rcu+0x100/0x1f8 [ 92.707625] irq_exit_rcu+0x18/0x58 [ 92.712818] el0_interrupt+0x60/0x138 [ 92.718195] __el0_irq_handler_common+0x18/0x28 [ 92.724438] el0t_64_irq_handler+0x10/0x20 [ 92.730254] el0t_64_irq+0x198/0x1a0 [ 92.735529] INITIAL USE at: [ 92.738608] lock_acquire+0x1d4/0x368 [ 92.743890] _raw_spin_lock+0x50/0x70 [ 92.749171] bcmgenet_xmit+0x78/0x860 [ 92.754452] dev_hard_start_xmit+0xc0/0x2d0 [ 92.760262] sch_direct_xmit+0x98/0x1e8 [ 92.765720] __dev_queue_xmit+0x5a0/0x13a8 [ 92.771440] packet_xmit+0xcc/0x150 [ 92.776552] packet_sendmsg+0x8e8/0x1328 [ 92.782091] __sock_sendmsg+0x64/0xc0 [ 92.787374] ____sys_sendmsg+0x270/0x2b0 [ 92.792918] ___sys_sendmsg+0x88/0xf0 [ 92.798198] __sys_sendmsg+0x88/0x100 [ 92.803479] __arm64_sys_sendmsg+0x2c/0x40 [ 92.809201] invoke_syscall+0x50/0x120 [ 92.814570] el0_svc_common.constprop.0+0x48/0xf0 [ 92.820908] do_el0_svc+0x24/0x38 [ 92.825837] el0_svc+0x4c/0x178 [ 92.830591] el0t_64_sync_handler+0xa0/0xe8 [ 92.836399] el0t_64_sync+0x198/0x1a0 [ 92.841681] } [ 92.843353] ... key at: [<ffffffd6a2347b30>] __key.6+0x0/0x10 [ 92.849690] ... acquired at: [ 92.852685] lock_acquire+0x1d4/0x368 [ 92.856555] _raw_spin_lock+0x50/0x70 [ 92.860428] bcmgenet_xmit+0x78/0x860 [ 92.864301] netpoll_start_xmit+0x118/0x1b0 [ 92.868702] netpoll_send_skb+0x3a8/0x4b0 [ 92.872928] netpoll_send_udp+0x2ac/0x418 [ 92.877153] write_msg+0x124/0x1b0 [netconsole] [ 92.881907] console_flush_all+0x2e4/0x4a8 [ 92.886220] console_unlock+0x84/0x188 [ 92.890181] vprintk_emit+0x24c/0x450 [ 92.894052] vprintk_default+0x40/0x58 [ 92.898013] vprintk+0x3c/0x80 [ 92.901270] _printk+0x64/0x98 [ 92.904527] __handle_sysrq+0x1ec/0x308 [ 92.908576] sysrq_filter+0xcc/0x518 [ 92.912363] input_handle_events_filter+0x58/0xc0 [ 92.917289] input_pass_values+0x1fc/0x2e8 [ 92.921604] input_handle_event+0x450/0x558 [ 92.926003] input_event+0x6c/0xb8 [ 92.929612] hidinput_report_event+0x4c/0x78 [ 92.934103] hid_report_raw_event+0xb4/0x418 [ 92.938592] hid_input_report+0x138/0x1a0 [ 92.942816] hid_irq_in+0x180/0x308 [ 92.946513] __usb_hcd_giveback_urb+0x98/0x160 [ 92.951179] usb_giveback_urb_bh+0xb0/0x150 [ 92.955580] process_one_work+0x20c/0x5e8 [ 92.959803] bh_worker+0x1b0/0x218 [ 92.963413] workqueue_softirq_action+0x80/0x98 [ 92.968166] tasklet_hi_action+0x20/0x60 [ 92.972304] handle_softirqs+0x144/0x528 [ 92.976438] __do_softirq+0x1c/0x28 [ 92.980142] ____do_softirq+0x18/0x30 [ 92.984008] call_on_irq_stack+0x30/0x70 [ 92.988145] do_softirq_own_stack+0x24/0x38 [ 92.992547] __irq_exit_rcu+0x100/0x1f8 [ 92.996595] irq_exit_rcu+0x18/0x58 [ 93.000293] el1_interrupt+0x38/0x68 [ 93.004077] el1h_64_irq_handler+0x18/0x28 [ 93.008390] el1h_64_irq+0x6c/0x70 [ 93.012005] default_idle_call+0x84/0x140 [ 93.016230] do_idle+0x22c/0x298 [ 93.019657] cpu_startup_entry+0x3c/0x50 [ 93.023794] rest_init+0x108/0x1a0 [ 93.027403] start_kernel+0x760/0x8c8 [ 93.031274] __primary_switched+0x88/0x98 [ 93.035500] [ 93.036995] [ 93.036995] stack backtrace: [ 93.041406] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G C 6.17.0-rc2-v8-lockdep #529 PREEMPTLAZY [ 93.052226] Tainted: [C]=CRAP [ 93.055218] Hardware name: Raspberry Pi 4 Model B Rev 1.4 (DT) [ 93.061114] Call trace: [ 93.063580] show_stack+0x20/0x38 (C) [ 93.067276] dump_stack_lvl+0x90/0xd0 [ 93.070973] dump_stack+0x18/0x28 [ 93.074317] check_irq_usage+0x720/0x9e8 [ 93.078277] __lock_acquire+0x1388/0x1fa0 [ 93.082327] lock_acquire+0x1d4/0x368 [ 93.086023] _raw_spin_lock+0x50/0x70 [ 93.089720] bcmgenet_xmit+0x78/0x860 [ 93.093418] netpoll_start_xmit+0x118/0x1b0 [ 93.097642] netpoll_send_skb+0x3a8/0x4b0 [ 93.101691] netpoll_send_udp+0x2ac/0x418 [ 93.105741] write_msg+0x124/0x1b0 [netconsole] [ 93.110320] console_flush_all+0x2e4/0x4a8 [ 93.114455] console_unlock+0x84/0x188 [ 93.118240] vprintk_emit+0x24c/0x450 [ 93.121935] vprintk_default+0x40/0x58 [ 93.125723] vprintk+0x3c/0x80 [ 93.128802] _printk+0x64/0x98 [ 93.131882] __handle_sysrq+0x1ec/0x308 [ 93.135756] sysrq_filter+0xcc/0x518 [ 93.139363] input_handle_events_filter+0x58/0xc0 [ 93.144118] input_pass_values+0x1fc/0x2e8 [ 93.148255] input_handle_event+0x450/0x558 [ 93.152478] input_event+0x6c/0xb8 [ 93.155912] hidinput_report_event+0x4c/0x78 [ 93.160225] hid_report_raw_event+0xb4/0x418 [ 93.164539] hid_input_report+0x138/0x1a0 [ 93.168586] hid_irq_in+0x180/0x308 [ 93.172107] __usb_hcd_giveback_urb+0x98/0x160 [ 93.176596] usb_giveback_urb_bh+0xb0/0x150 [ 93.180822] process_one_work+0x20c/0x5e8 [ 93.184870] bh_worker+0x1b0/0x218 [ 93.188303] workqueue_softirq_action+0x80/0x98 [ 93.192881] tasklet_hi_action+0x20/0x60 [ 93.196841] handle_softirqs+0x144/0x528 [ 93.200808] __do_softirq+0x1c/0x28 [ 93.204325] ____do_softirq+0x18/0x30 [ 93.208019] call_on_irq_stack+0x30/0x70 [ 93.211983] do_softirq_own_stack+0x24/0x38 [ 93.216205] __irq_exit_rcu+0x100/0x1f8 [ 93.220078] irq_exit_rcu+0x18/0x58 [ 93.223600] el1_interrupt+0x38/0x68 [ 93.227208] el1h_64_irq_handler+0x18/0x28 [ 93.231345] el1h_64_irq+0x6c/0x70 [ 93.234778] default_idle_call+0x84/0x140 (P) [ 93.239179] do_idle+0x22c/0x298 [ 93.242436] cpu_startup_entry+0x3c/0x50 [ 93.246403] rest_init+0x108/0x1a0 [ 93.249830] start_kernel+0x760/0x8c8 [ 93.253525] __primary_switched+0x88/0x98 ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: netpoll: raspberrypi [4 5] driver locking woes 2025-08-25 5:53 netpoll: raspberrypi [4 5] driver locking woes Mike Galbraith 2025-08-25 10:10 ` Mike Galbraith @ 2025-08-26 9:49 ` Mike Galbraith [not found] ` <f4fa3fcc637ffb6531982a90dbd9c27114e93036.camel@gmx.de> 1 sibling, 1 reply; 7+ messages in thread From: Mike Galbraith @ 2025-08-26 9:49 UTC (permalink / raw) To: lkml; +Cc: Robert Hancock, Doug Berger, Florian Fainelli, Breno Leitao On Mon, 2025-08-25 at 07:53 +0200, Mike Galbraith wrote: > > On the pi4, lockdep muttered about ring->lock in bcmgenet_xmit() and > bcmgenet_tx_poll() - switching to irqsave/restore spinlock variants > there silenced it. I went ahead and submitted the pi4 patchlet. The pi5 gripe fix is equally trivial, but submitting that is pointless given there's something else amiss in fingered commit. This is all of the crash info that escapes the box w/wo gripes silenced. [ 51.688868] sysrq: Trigger a crash [ 51.688892] Kernel panic - not syncing: sysrq triggered crash [ 51.698066] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.17.0-rc2-v8-lockdep #533 PREEMPTLAZY [ 51.707234] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT) [ 51.713085] Call trace: [ 51.715532] show_stack+0x20/0x38 (C) [ 51.719206] dump_stack_lvl+0x38/0xd0 [ 51.722878] dump_stack+0x18/0x28 That aspect is a punt and run atm (time.. and dash of laziness:). -Mike ^ permalink raw reply [flat|nested] 7+ messages in thread
[parent not found: <f4fa3fcc637ffb6531982a90dbd9c27114e93036.camel@gmx.de>]
* Re: netpoll: raspberrypi [4 5] driver locking woes [not found] ` <f4fa3fcc637ffb6531982a90dbd9c27114e93036.camel@gmx.de> @ 2025-08-28 14:57 ` Sean Anderson 2025-08-28 17:26 ` Mike Galbraith 0 siblings, 1 reply; 7+ messages in thread From: Sean Anderson @ 2025-08-28 14:57 UTC (permalink / raw) To: Mike Galbraith, Robert Hancock; +Cc: Breno Leitao, netdev@vger.kernel.org Hi Mike, On 8/27/25 12:02, Mike Galbraith wrote: > Unexpected addendum to done deal datapoint, so off list. > > On Tue, 2025-08-26 at 11:49 +0200, Mike Galbraith wrote: >> >> The pi5 gripe fix is equally trivial, but submitting that is pointless >> given there's something else amiss in fingered commit. This is all of >> the crash info that escapes the box w/wo gripes silenced. >> >> [ 51.688868] sysrq: Trigger a crash >> [ 51.688892] Kernel panic - not syncing: sysrq triggered crash >> [ 51.698066] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.17.0-rc2-v8-lockdep #533 PREEMPTLAZY >> [ 51.707234] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT) >> [ 51.713085] Call trace: >> [ 51.715532] show_stack+0x20/0x38 (C) >> [ 51.719206] dump_stack_lvl+0x38/0xd0 >> [ 51.722878] dump_stack+0x18/0x28 >> >> That aspect is a punt and run atm (time.. and dash of laziness:). > > Plan was to end datapoint thread, but after booting pi5's 6.12 kernel, > for some reason I fired up netconsole.. and box promptly exhibited the > netpoll locking bug warning, indicating presence of 138badbc21a0. > Instead of saying to self "nope, just walk away", I poked SysRq-C.. and > the bloody damn monitoring box received a 100% complete death rattle. > Well bugger. Did you get a backtrace for this? And to be clear, the steps to reproduce this are to boot a kernel with lockdep enabled with netconsole on macb and then hit sysrq? > Two trees contain locking buglet introducing 138badbc21a0, 6.12.41 and > 6.17.0, but only the later reproduces the above mess. A quick stare at > git diff inspired checking e6a532185daa, and sure enough reverting only > it reproduced a functional netconsole, modulo warnings, ie the mess > above manifests only with BOTH resident. Either revert 138badbc21a0, > or revert e6a532185daa and fix the 138badbc21a0 locking buglet, and the > result is the same, a perfectly functional warning free netconsole. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: netpoll: raspberrypi [4 5] driver locking woes 2025-08-28 14:57 ` Sean Anderson @ 2025-08-28 17:26 ` Mike Galbraith 2025-08-28 18:56 ` Sean Anderson 0 siblings, 1 reply; 7+ messages in thread From: Mike Galbraith @ 2025-08-28 17:26 UTC (permalink / raw) To: Sean Anderson, Robert Hancock; +Cc: Breno Leitao, netdev@vger.kernel.org [-- Attachment #1: Type: text/plain, Size: 2449 bytes --] On Thu, 2025-08-28 at 10:57 -0400, Sean Anderson wrote: > Hi Mike, > > On 8/27/25 12:02, Mike Galbraith wrote: > > Unexpected addendum to done deal datapoint, so off list. > > > > On Tue, 2025-08-26 at 11:49 +0200, Mike Galbraith wrote: > > > > > > The pi5 gripe fix is equally trivial, but submitting that is pointless > > > given there's something else amiss in fingered commit. This is all of > > > the crash info that escapes the box w/wo gripes silenced. > > > > > > [ 51.688868] sysrq: Trigger a crash > > > [ 51.688892] Kernel panic - not syncing: sysrq triggered crash > > > [ 51.698066] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.17.0-rc2-v8-lockdep #533 PREEMPTLAZY > > > [ 51.707234] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT) > > > [ 51.713085] Call trace: > > > [ 51.715532] show_stack+0x20/0x38 (C) > > > [ 51.719206] dump_stack_lvl+0x38/0xd0 > > > [ 51.722878] dump_stack+0x18/0x28 > > > > > > That aspect is a punt and run atm (time.. and dash of laziness:). > > > > Plan was to end datapoint thread, but after booting pi5's 6.12 kernel, > > for some reason I fired up netconsole.. and box promptly exhibited the > > netpoll locking bug warning, indicating presence of 138badbc21a0. > > Instead of saying to self "nope, just walk away", I poked SysRq-C.. and > > the bloody damn monitoring box received a 100% complete death rattle. > > Well bugger. > > Did you get a backtrace for this? Yes, logs for 6.12.41 and 6.17.0-rc2 attached. Since a patch has meanwhile landed, also a log of patched 6.17.0-rc3 now gripe free (yay) but with aforementioned broken output, followed by addition of the e6a532185daa revert to confirm it still cures that. > And to be clear, the steps to reproduce this are to boot a kernel with > lockdep enabled with netconsole on macb and then hit sysrq? Yup. > > Two trees contain locking buglet introducing 138badbc21a0, 6.12.41 and > > 6.17.0, but only the later reproduces the above mess. A quick stare at > > git diff inspired checking e6a532185daa, and sure enough reverting only > > it reproduced a functional netconsole, modulo warnings, ie the mess > > above manifests only with BOTH resident. Either revert 138badbc21a0, > > or revert e6a532185daa and fix the 138badbc21a0 locking buglet, and the > > result is the same, a perfectly functional warning free netconsole. > [-- Attachment #2: netconsole-rpi5-6.12-eth0.log --] [-- Type: text/x-log, Size: 18613 bytes --] [ 349.336384] sysrq: Trigger a crash [ 349.336569] ------------[ cut here ]------------ [ 349.336572] netpoll_send_skb_on_dev(): eth0 enabled interrupts in poll (macb_start_xmit+0x0/0xc48) [ 349.336622] WARNING: CPU: 0 PID: 0 at net/core/netpoll.c:385 netpoll_send_skb+0x4e4/0x528 [ 349.336632] Modules linked in: netconsole quota_v2 quota_tree rfcomm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device algif_hash algif_skcipher af_alg bnep snd_soc_pcm512x_i2c snd_soc_pcm512x regmap_i2c spidev brcmfmac_wcc brcmfmac vc4 hci_uart btbcm bluetooth aes_ce_blk snd_soc_hdmi_codec v3d aes_ce_cipher ghash_ce drm_display_helper rpi_hevc_dec cec gpu_sched brcmutil drm_dma_helper drm_shmem_helper v4l2_mem2mem pisp_be gf128mul sha2_ce videobuf2_dma_contig sha256_arm64 ecdh_generic videobuf2_memops cfg80211 snd_soc_hifiberry_dacplus ecc videobuf2_v4l2 raspberrypi_hwmon designware_i2s sha1_ce libaes drm_kms_helper snd_soc_core videodev i2c_brcmstb spi_bcm2835 videobuf2_common rfkill gpio_keys clk_hifiberry_dacpro snd_compress binfmt_misc snd_pcm_dmaengine mc snd_pcm rp1_pio rp1_mailbox snd_timer raspberrypi_gpiomem i2c_designware_platform rp1_fw snd i2c_designware_core rp1_adc sg nvmem_rmem uio_pdrv_genirq uio nfsd drm fuse dm_mod i2c_dev ledtrig_pattern auth_rpcgss drm_panel_orientation_quirks backlight ip_tables [ 349.336710] x_tables ipv6 [ 349.336715] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G W 6.12.41-v8-lockdep #518 [ 349.336721] Tainted: [W]=WARN [ 349.336722] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT) [ 349.336724] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 349.336727] pc : netpoll_send_skb+0x4e4/0x528 [ 349.336731] lr : netpoll_send_skb+0x4e4/0x528 [ 349.336735] sp : ffffffc080003670 [ 349.336737] x29: ffffffc080003670 x28: ffffff810099e600 x27: 0000000000000000 [ 349.336741] x26: 0000000000000080 x25: ffffff812c8fc400 x24: ffffff8102e0c000 [ 349.336745] x23: 00000000000000c0 x22: ffffff81395a9300 x21: ffffffd08de13a28 [ 349.336748] x20: ffffff81183c1130 x19: ffffffd08d94fe40 x18: ffffffffffffffff [ 349.336752] x17: ffffffb1715bb000 x16: ffffffd08d0a9568 x15: 00000000fffffffd [ 349.336755] x14: 0000000000000001 x13: ffffffc080002f94 x12: 6920737470757272 [ 349.336759] x11: fffffffffffe0000 x10: ffffffd08ddfa080 x9 : ffffffd08c545950 [ 349.336762] x8 : 0000000000000001 x7 : ffffffd08dd770a0 x6 : 0000000000000001 [ 349.336766] x5 : ffffffd08dd77000 x4 : 0000000000000001 x3 : 0000000000000000 [ 349.336769] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffffffd08dd7cec0 [ 349.336773] Call trace: [ 349.336774] netpoll_send_skb+0x4e4/0x528 [ 349.336778] netpoll_send_udp+0x258/0x3e8 [ 349.336783] write_msg+0x100/0x138 [netconsole] [ 349.336792] console_flush_all+0x264/0x4a0 [ 349.336798] console_unlock+0x84/0x188 [ 349.336802] vprintk_emit+0x22c/0x420 [ 349.336807] vprintk_default+0x40/0x58 [ 349.336811] vprintk+0x54/0xe0 [ 349.336814] _printk+0x64/0x98 [ 349.336817] __handle_sysrq+0xe0/0x2e0 [ 349.336823] sysrq_filter+0xcc/0x518 [ 349.336827] input_handle_events_filter+0x58/0xc0 [ 349.336831] input_pass_values+0x1e4/0x2c8 [ 349.336836] input_handle_event+0x450/0x558 [ 349.336839] input_event+0x68/0x98 [ 349.336843] hidinput_report_event+0x4c/0x78 [ 349.336848] hid_report_raw_event+0xb4/0x3e0 [ 349.336850] hid_input_report+0x138/0x1a0 [ 349.336853] hid_irq_in+0x180/0x308 [ 349.336858] __usb_hcd_giveback_urb+0x98/0x160 [ 349.336863] usb_giveback_urb_bh+0xb0/0x150 [ 349.336867] process_one_work+0x214/0x640 [ 349.336873] bh_worker+0x1b0/0x218 [ 349.336877] workqueue_softirq_action+0x80/0x98 [ 349.336880] tasklet_hi_action+0x20/0x60 [ 349.336883] handle_softirqs+0x154/0x528 [ 349.336885] __do_softirq+0x1c/0x28 [ 349.336889] ____do_softirq+0x18/0x30 [ 349.336892] call_on_irq_stack+0x30/0x70 [ 349.336895] do_softirq_own_stack+0x24/0x38 [ 349.336898] __irq_exit_rcu+0x100/0x1a0 [ 349.336901] irq_exit_rcu+0x18/0x58 [ 349.336903] el1_interrupt+0x38/0x70 [ 349.336908] el1h_64_irq_handler+0x18/0x28 [ 349.336912] el1h_64_irq+0x64/0x68 [ 349.336914] default_idle_call+0xa8/0x1f8 [ 349.336917] do_idle+0x22c/0x298 [ 349.336920] cpu_startup_entry+0x3c/0x50 [ 349.336923] rest_init+0x100/0x198 [ 349.336925] start_kernel+0x75c/0x900 [ 349.336930] __primary_switched+0x80/0x90 [ 349.336934] irq event stamp: 8607573 [ 349.336935] hardirqs last enabled at (8607572): [<ffffffd08d2c7ce4>] el1_interrupt+0x44/0x70 [ 349.336940] hardirqs last disabled at (8607573): [<ffffffd08d2c72f4>] el1_dbg+0x24/0x98 [ 349.336944] softirqs last enabled at (8607524): [<ffffffd08c4b04c8>] handle_softirqs+0x508/0x528 [ 349.336947] softirqs last disabled at (8607545): [<ffffffd08c4102ac>] __do_softirq+0x1c/0x28 [ 349.336949] ---[ end trace 0000000000000000 ]--- [ 349.340398] ------------[ cut here ]------------ [ 349.340399] WARNING: CPU: 0 PID: 0 at mm/slub.c:700 get_freelist+0x1bc/0x2b0 [ 349.340405] Modules linked in: netconsole quota_v2 quota_tree rfcomm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device algif_hash algif_skcipher af_alg bnep snd_soc_pcm512x_i2c snd_soc_pcm512x regmap_i2c spidev brcmfmac_wcc brcmfmac vc4 hci_uart btbcm bluetooth aes_ce_blk snd_soc_hdmi_codec v3d aes_ce_cipher ghash_ce drm_display_helper rpi_hevc_dec cec gpu_sched brcmutil drm_dma_helper drm_shmem_helper v4l2_mem2mem pisp_be gf128mul sha2_ce videobuf2_dma_contig sha256_arm64 ecdh_generic videobuf2_memops cfg80211 snd_soc_hifiberry_dacplus ecc videobuf2_v4l2 raspberrypi_hwmon designware_i2s sha1_ce libaes drm_kms_helper snd_soc_core videodev i2c_brcmstb spi_bcm2835 videobuf2_common rfkill gpio_keys clk_hifiberry_dacpro snd_compress binfmt_misc snd_pcm_dmaengine mc snd_pcm rp1_pio rp1_mailbox snd_timer raspberrypi_gpiomem i2c_designware_platform rp1_fw snd i2c_designware_core rp1_adc sg nvmem_rmem uio_pdrv_genirq uio nfsd drm fuse dm_mod i2c_dev ledtrig_pattern auth_rpcgss drm_panel_orientation_quirks backlight ip_tables [ 349.340461] x_tables ipv6 [ 349.340464] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G W 6.12.41-v8-lockdep #518 [ 349.340467] Tainted: [W]=WARN [ 349.340468] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT) [ 349.340469] pstate: 604000c9 (nZCv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 349.340472] pc : get_freelist+0x1bc/0x2b0 [ 349.340475] lr : get_freelist+0x290/0x2b0 [ 349.340477] sp : ffffffc080003460 [ 349.340478] x29: ffffffc080003460 x28: 0000000000000000 x27: 0000000000000001 [ 349.340482] x26: ffffffd08dd77000 x25: ffffffd08d93c448 x24: ffffff81001cec00 [ 349.340485] x23: ffffffd08dd7cec0 x22: 0000000000100010 x21: 0000000000000000 [ 349.340489] x20: 0000000080100010 x19: fffffffec4e56a40 x18: ffffffffffffffff [ 349.340492] x17: ffffffb1715bb000 x16: ffffffd08d0a9568 x15: 0000000000000006 [ 349.340496] x14: 00000000000c2000 x13: 2d2d2d2d2d2d2d2d x12: 0000000000000000 [ 349.340499] x11: 0000000000000190 x10: 0000000000000000 x9 : ffffffd08e70ab10 [ 349.340503] x8 : ffffffd08dd7cec0 x7 : 0000000001298354 x6 : ffffffd08e6d4080 [ 349.340506] x5 : 00000000000000c8 x4 : 0000000000040000 x3 : ffffffd08dd7cec0 [ 349.340510] x2 : ffffffb1715bb000 x1 : ffffffd08dd7cec0 x0 : 0000000000000105 [ 349.340513] Call trace: [ 349.340514] get_freelist+0x1bc/0x2b0 [ 349.340516] ___slab_alloc+0xf4/0x10c0 [ 349.340519] __slab_alloc.constprop.0+0x6c/0xc8 [ 349.340522] kmem_cache_alloc_node_noprof+0x2e4/0x348 [ 349.340525] __alloc_skb+0x154/0x188 [ 349.340529] netpoll_send_udp+0xa8/0x3e8 [ 349.340533] write_msg+0x100/0x138 [netconsole] [ 349.340538] console_flush_all+0x264/0x4a0 [ 349.340542] console_unlock+0x84/0x188 [ 349.340546] vprintk_emit+0x22c/0x420 [ 349.340551] vprintk_default+0x40/0x58 [ 349.340555] vprintk+0x54/0xe0 [ 349.340557] _printk+0x64/0x98 [ 349.340560] __handle_sysrq+0xe0/0x2e0 [ 349.340565] sysrq_filter+0xcc/0x518 [ 349.340569] input_handle_events_filter+0x58/0xc0 [ 349.340572] input_pass_values+0x1e4/0x2c8 [ 349.340576] input_handle_event+0x450/0x558 [ 349.340580] input_event+0x68/0x98 [ 349.340583] hidinput_report_event+0x4c/0x78 [ 349.340587] hid_report_raw_event+0xb4/0x3e0 [ 349.340589] hid_input_report+0x138/0x1a0 [ 349.340592] hid_irq_in+0x180/0x308 [ 349.340596] __usb_hcd_giveback_urb+0x98/0x160 [ 349.340600] usb_giveback_urb_bh+0xb0/0x150 [ 349.340604] process_one_work+0x214/0x640 [ 349.340608] bh_worker+0x1b0/0x218 [ 349.340613] workqueue_softirq_action+0x80/0x98 [ 349.340615] tasklet_hi_action+0x20/0x60 [ 349.340618] handle_softirqs+0x154/0x528 [ 349.340620] __do_softirq+0x1c/0x28 [ 349.340622] ____do_softirq+0x18/0x30 [ 349.340625] call_on_irq_stack+0x30/0x70 [ 349.340628] do_softirq_own_stack+0x24/0x38 [ 349.340631] __irq_exit_rcu+0x100/0x1a0 [ 349.340634] irq_exit_rcu+0x18/0x58 [ 349.340636] el1_interrupt+0x38/0x70 [ 349.340640] el1h_64_irq_handler+0x18/0x28 [ 349.340644] el1h_64_irq+0x64/0x68 [ 349.340646] default_idle_call+0xa8/0x1f8 [ 349.340648] do_idle+0x22c/0x298 [ 349.340650] cpu_startup_entry+0x3c/0x50 [ 349.340652] rest_init+0x100/0x198 [ 349.340655] start_kernel+0x75c/0x900 [ 349.340658] __primary_switched+0x80/0x90 [ 349.340661] irq event stamp: 8607577 [ 349.340662] hardirqs last enabled at (8607576): [<ffffffd08d2c7ce4>] el1_interrupt+0x44/0x70 [ 349.340666] hardirqs last disabled at (8607577): [<ffffffd08d2c72f4>] el1_dbg+0x24/0x98 [ 349.340670] softirqs last enabled at (8607524): [<ffffffd08c4b04c8>] handle_softirqs+0x508/0x528 [ 349.340673] softirqs last disabled at (8607545): [<ffffffd08c4102ac>] __do_softirq+0x1c/0x28 [ 349.340676] ---[ end trace 0000000000000000 ]--- [ 349.340679] [ 349.340680] ================================ [ 349.340681] WARNING: inconsistent lock state [ 349.340682] 6.12.41-v8-lockdep #518 Tainted: G W [ 349.340684] -------------------------------- [ 349.340684] inconsistent {IN-HARDIRQ-W} -> {HARDIRQ-ON-W} usage. [ 349.340686] swapper/0/0 [HC0[0]:SC1[1]:HE0:SE0] takes: [ 349.340688] ffffff81fef12fc0 (&c->lock){?.-.}-{2:2}, at: ___slab_alloc+0x78/0x10c0 [ 349.340695] {IN-HARDIRQ-W} state was registered at: [ 349.340697] lock_acquire+0x208/0x360 [ 349.340700] put_cpu_partial+0x88/0x258 [ 349.340703] __slab_free+0x1c8/0x390 [ 349.340705] kmem_cache_free+0x29c/0x400 [ 349.340708] mempool_free_slab+0x24/0x38 [ 349.340712] mempool_free+0x48/0xb8 [ 349.340715] bio_free+0x70/0x90 [ 349.340719] bio_put+0x120/0x278 [ 349.340722] end_bio_bh_io_sync+0x44/0x88 [ 349.340725] bio_endio+0x18c/0x288 [ 349.340728] blk_mq_end_request_batch+0x274/0x500 [ 349.340731] nvme_pci_complete_batch+0xe0/0x138 [ 349.340734] nvme_irq+0x78/0x90 [ 349.340736] __handle_irq_event_percpu+0x94/0x308 [ 349.340740] handle_irq_event+0x54/0xb8 [ 349.340743] handle_fasteoi_irq+0xc8/0x268 [ 349.340747] handle_irq_desc+0x48/0x68 [ 349.340750] generic_handle_domain_irq+0x24/0x38 [ 349.340753] gic_handle_irq+0x48/0xd8 [ 349.340755] call_on_irq_stack+0x30/0x70 [ 349.340758] do_interrupt_handler+0xdc/0xe8 [ 349.340761] el1_interrupt+0x34/0x70 [ 349.340765] el1h_64_irq_handler+0x18/0x28 [ 349.340769] el1h_64_irq+0x64/0x68 [ 349.340771] default_idle_call+0xa8/0x1f8 [ 349.340773] do_idle+0x22c/0x298 [ 349.340775] cpu_startup_entry+0x3c/0x50 [ 349.340777] secondary_start_kernel+0x128/0x150 [ 349.340781] __secondary_switched+0xb8/0xc0 [ 349.340785] irq event stamp: 8607577 [ 349.340786] hardirqs last enabled at (8607576): [<ffffffd08d2c7ce4>] el1_interrupt+0x44/0x70 [ 349.340790] hardirqs last disabled at (8607577): [<ffffffd08d2c72f4>] el1_dbg+0x24/0x98 [ 349.340794] softirqs last enabled at (8607524): [<ffffffd08c4b04c8>] handle_softirqs+0x508/0x528 [ 349.340797] softirqs last disabled at (8607545): [<ffffffd08c4102ac>] __do_softirq+0x1c/0x28 [ 349.340800] [ 349.340800] other info that might help us debug this: [ 349.340801] Possible unsafe locking scenario: [ 349.340801] [ 349.340801] CPU0 [ 349.340802] ---- [ 349.340803] lock(&c->lock); [ 349.340805] <Interrupt> [ 349.340806] lock(&c->lock); [ 349.340807] [ 349.340807] *** DEADLOCK *** [ 349.340807] [ 349.340808] 11 locks held by swapper/0/0: [ 349.340810] #0: ffffff8100045548 ((wq_completion)events_bh_highpri){+.-.}-{0:0}, at: process_one_work+0x198/0x640 [ 349.340818] #1: ffffffc080003e60 ((work_completion)(&bh->bh)){+.-.}-{0:0}, at: process_one_work+0x1c0/0x640 [ 349.340826] #2: ffffff8102eaa230 (&dev->event_lock){+.-.}-{2:2}, at: input_event+0x50/0x98 [ 349.340833] #3: ffffffd08de13a28 (rcu_read_lock){....}-{1:2}, at: input_pass_values+0x38/0x2c8 [ 349.340841] #4: ffffffd08de13a28 (rcu_read_lock){....}-{1:2}, at: __handle_sysrq+0x8/0x2e0 [ 349.340848] #5: ffffffd08dda1c80 (console_lock){+.+.}-{0:0}, at: vprintk_default+0x40/0x58 [ 349.340856] #6: ffffffd08dda1cd8 (console_srcu){....}-{0:0}, at: console_flush_all+0x8c/0x4a0 [ 349.340863] #7: ffffffd08dda1e98 (console_owner){+.-.}-{0:0}, at: console_lock_spinning_enable+0x50/0x90 [ 349.340871] #8: ffffffd08dda1ec0 (printk_legacy_map-wait-type-override){+...}-{3:3}, at: console_flush_all+0x220/0x4a0 [ 349.340878] #9: ffffffd02cbab220 (target_list_lock){+.-.}-{2:2}, at: write_msg+0x64/0x138 [netconsole] [ 349.340886] #10: ffffff81fef12fc0 (&c->lock){?.-.}-{2:2}, at: ___slab_alloc+0x78/0x10c0 [ 349.340892] [ 349.340892] stack backtrace: [ 349.340893] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G W 6.12.41-v8-lockdep #518 [ 349.340897] Tainted: [W]=WARN [ 349.340898] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT) [ 349.340899] Call trace: [ 349.340900] dump_backtrace+0xa0/0x100 [ 349.340902] show_stack+0x20/0x38 [ 349.340904] dump_stack_lvl+0x90/0xd0 [ 349.340909] dump_stack+0x18/0x28 [ 349.340912] print_usage_bug.part.0+0x29c/0x368 [ 349.340915] mark_lock+0x724/0x928 [ 349.340917] mark_held_locks+0x54/0x90 [ 349.340919] lockdep_hardirqs_on_prepare+0x108/0x228 [ 349.340921] arm64_exit_el1_dbg.isra.0+0x30/0x58 [ 349.340925] el1_dbg+0x84/0x98 [ 349.340928] el1h_64_sync_handler+0xc8/0xe8 [ 349.340932] el1h_64_sync+0x64/0x68 [ 349.340934] get_freelist+0x1bc/0x2b0 [ 349.340936] ___slab_alloc+0xf4/0x10c0 [ 349.340939] __slab_alloc.constprop.0+0x6c/0xc8 [ 349.340942] kmem_cache_alloc_node_noprof+0x2e4/0x348 [ 349.340945] __alloc_skb+0x154/0x188 [ 349.340947] netpoll_send_udp+0xa8/0x3e8 [ 349.340952] write_msg+0x100/0x138 [netconsole] [ 349.340955] console_flush_all+0x264/0x4a0 [ 349.340960] console_unlock+0x84/0x188 [ 349.340964] vprintk_emit+0x22c/0x420 [ 349.340968] vprintk_default+0x40/0x58 [ 349.340972] vprintk+0x54/0xe0 [ 349.340974] _printk+0x64/0x98 [ 349.340977] __handle_sysrq+0xe0/0x2e0 [ 349.340981] sysrq_filter+0xcc/0x518 [ 349.340986] input_handle_events_filter+0x58/0xc0 [ 349.340989] input_pass_values+0x1e4/0x2c8 [ 349.340992] input_handle_event+0x450/0x558 [ 349.340996] input_event+0x68/0x98 [ 349.341000] hidinput_report_event+0x4c/0x78 [ 349.341003] hid_report_raw_event+0xb4/0x3e0 [ 349.341006] hid_input_report+0x138/0x1a0 [ 349.341008] hid_irq_in+0x180/0x308 [ 349.341012] __usb_hcd_giveback_urb+0x98/0x160 [ 349.341016] usb_giveback_urb_bh+0xb0/0x150 [ 349.341020] process_one_work+0x214/0x640 [ 349.341025] bh_worker+0x1b0/0x218 [ 349.341029] workqueue_softirq_action+0x80/0x98 [ 349.341031] tasklet_hi_action+0x20/0x60 [ 349.341034] handle_softirqs+0x154/0x528 [ 349.341036] __do_softirq+0x1c/0x28 [ 349.341038] ____do_softirq+0x18/0x30 [ 349.341041] call_on_irq_stack+0x30/0x70 [ 349.341044] do_softirq_own_stack+0x24/0x38 [ 349.341047] __irq_exit_rcu+0x100/0x1a0 [ 349.341049] irq_exit_rcu+0x18/0x58 [ 349.341052] el1_interrupt+0x38/0x70 [ 349.341056] el1h_64_irq_handler+0x18/0x28 [ 349.341059] el1h_64_irq+0x64/0x68 [ 349.341062] default_idle_call+0xa8/0x1f8 [ 349.341064] do_idle+0x22c/0x298 [ 349.341066] cpu_startup_entry+0x3c/0x50 [ 349.341068] rest_init+0x100/0x198 [ 349.341070] start_kernel+0x75c/0x900 [ 349.341073] __primary_switched+0x80/0x90 [ 350.513730] v3d 1002000000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang. [ 350.517765] Kernel panic - not syncing: sysrq triggered crash [ 350.517769] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G W 6.12.41-v8-lockdep #518 [ 350.517773] Tainted: [W]=WARN [ 350.517774] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT) [ 350.813441] Call trace: [ 350.815900] dump_backtrace+0xa0/0x100 [ 350.819671] show_stack+0x20/0x38 [ 350.823004] dump_stack_lvl+0x38/0xd0 [ 350.826688] dump_stack+0x18/0x28 [ 350.830023] panic+0x3c8/0x430 [ 350.833096] sysrq_handle_crash+0x40/0x90 [ 350.837130] __handle_sysrq+0xf0/0x2e0 [ 350.840901] sysrq_filter+0xcc/0x518 [ 350.844498] input_handle_events_filter+0x58/0xc0 [ 350.849228] input_pass_values+0x1e4/0x2c8 [ 350.853348] input_handle_event+0x450/0x558 [ 350.857554] input_event+0x68/0x98 [ 350.860976] hidinput_report_event+0x4c/0x78 [ 350.865271] hid_report_raw_event+0xb4/0x3e0 [ 350.869564] hid_input_report+0x138/0x1a0 [ 350.873595] hid_irq_in+0x180/0x308 [ 350.877105] __usb_hcd_giveback_urb+0x98/0x160 [ 350.881575] usb_giveback_urb_bh+0xb0/0x150 [ 350.885782] process_one_work+0x214/0x640 [ 350.889815] bh_worker+0x1b0/0x218 [ 350.893238] workqueue_softirq_action+0x80/0x98 [ 350.897793] tasklet_hi_action+0x20/0x60 [ 350.901737] handle_softirqs+0x154/0x528 [ 350.905681] __do_softirq+0x1c/0x28 [ 350.909188] ____do_softirq+0x18/0x30 [ 350.912871] call_on_irq_stack+0x30/0x70 [ 350.916815] do_softirq_own_stack+0x24/0x38 [ 350.921023] __irq_exit_rcu+0x100/0x1a0 [ 350.924880] irq_exit_rcu+0x18/0x58 [ 350.928388] el1_interrupt+0x38/0x70 [ 350.931985] el1h_64_irq_handler+0x18/0x28 [ 350.936105] el1h_64_irq+0x64/0x68 [ 350.939529] default_idle_call+0xa8/0x1f8 [ 350.943561] do_idle+0x22c/0x298 [ 350.946807] cpu_startup_entry+0x3c/0x50 [ 350.950751] rest_init+0x100/0x198 [ 350.954170] start_kernel+0x75c/0x900 [ 350.957855] __primary_switched+0x80/0x90 [ 350.961888] SMP: stopping secondary CPUs [ 352.035165] SMP: failed to stop secondary CPUs 1 [ 352.039823] Kernel Offset: 0x100c400000 from 0xffffffc080000000 [ 352.045770] PHYS_OFFSET: 0x0 [ 352.048666] CPU features: 0x10,00000017,00280928,0200720b [ 352.054093] Memory Limit: none [ 352.057167] ---[ end Kernel panic - not syncing: sysrq triggered crash ]--- [-- Attachment #3: netconsole-rpi5-6.17-eth0.log --] [-- Type: text/x-log, Size: 17135 bytes --] [ 58.750302] sysrq: HELP : loglevel(0-9) reboot(b) crash(c) show-all-locks(d) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) sak(k) show-backtrace-all-active-cpus(l) show-memory-usage(m) nice-all-RT-tasks(n) poweroff(o) show-registers(p) show-all-timers(q) unraw(r) sync(s) show-task-states(t) unmount(u) force-fb(v) show-blocked-tasks(w) dump-ftrace-buffer(z) replay-kernel-logs(R) [ 58.750681] ------------[ cut here ]------------ [ 58.750690] netpoll_send_skb_on_dev(): eth0 enabled interrupts in poll (macb_start_xmit+0x0/0xd30) [ 58.750730] WARNING: CPU: 0 PID: 0 at net/core/netpoll.c:359 netpoll_send_skb+0x46c/0x4b0 [ 58.816326] Modules linked in: netconsole quota_v2 quota_tree rfcomm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device algif_hash algif_skcipher af_alg bnep snd_soc_pcm512x_i2c snd_soc_pcm512x spidev regmap_i2c binfmt_misc hci_uart aes_ce_blk btbcm bluetooth ecdh_generic ecc aes_ce_cipher ghash_ce brcmfmac_cyw gf128mul snd_soc_hifiberry_dacplus sg raspberrypi_hwmon brcmfmac brcmutil i2c_brcmstb cfg80211 vc4 rfkill spi_bcm2835 rpi_hevc_dec snd_soc_hdmi_codec gpio_keys drm_exec drm_display_helper clk_hifiberry_dacpro cec pisp_be v4l2_mem2mem v3d videobuf2_dma_contig drm_dma_helper gpu_sched drm_shmem_helper drm_client_lib rp1_pci videobuf2_memops videobuf2_v4l2 videodev designware_i2s drm_kms_helper videobuf2_common mc snd_soc_core snd_compress i2c_designware_platform rp1_pio snd_pcm_dmaengine rp1_fw i2c_designware_core snd_pcm rp1_mailbox raspberrypi_gpiomem snd_timer rp1_adc snd nvmem_rmem uio_pdrv_genirq uio i2c_dev ledtrig_pattern drm fuse drm_panel_orientation_quirks backlight dm_mod nfsd auth_rpcgss ip_tables [ 58.816415] x_tables ipv6 [ 58.909197] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.17.0-rc2-v8-lockdep #532 PREEMPTLAZY [ 58.918365] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT) [ 58.924216] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 58.931202] pc : netpoll_send_skb+0x46c/0x4b0 [ 58.935571] lr : netpoll_send_skb+0x46c/0x4b0 [ 58.939939] sp : ffffffc080003690 [ 58.943258] x29: ffffffc080003690 x28: 0000000000000000 x27: ffffff8101151800 [ 58.950421] x26: ffffff8137fb6268 x25: 0000000000000080 x24: ffffff8137fb6200 [ 58.957583] x23: ffffff8102d58000 x22: 00000000000000c0 x21: ffffff8104f7e100 [ 58.964745] x20: ffffffd08de9c3c8 x19: ffffff8137f2d148 x18: 00000000ffffffff [ 58.971907] x17: ffffffb1714fe000 x16: ffffffd08d124020 x15: 00000000fffffffd [ 58.979069] x14: 0000000000000001 x13: 000000000002a000 x12: 0000000000008000 [ 58.986231] x11: fffffffffffe0000 x10: ffffffd08e78f000 x9 : ffffffd08c5596f0 [ 58.993393] x8 : ffffffffffffffff x7 : ffffffd08de07ef8 x6 : 0000000000000001 [ 59.000555] x5 : ffffffd08de07000 x4 : 0000000000000001 x3 : 0000000000000001 [ 59.007717] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffffffd08de0dd40 [ 59.014879] Call trace: [ 59.017326] netpoll_send_skb+0x46c/0x4b0 (P) [ 59.021695] netpoll_send_udp+0x2ac/0x418 [ 59.025715] write_msg+0x124/0x1b0 [netconsole] [ 59.030262] console_flush_all+0x2e4/0x4a8 [ 59.034371] console_unlock+0x84/0x188 [ 59.038128] vprintk_emit+0x24c/0x450 [ 59.041798] vprintk_default+0x40/0x58 [ 59.045555] vprintk+0x3c/0x80 [ 59.048615] _printk+0x64/0x98 [ 59.051674] __handle_sysrq+0x1ec/0x308 [ 59.055519] sysrq_filter+0xcc/0x518 [ 59.059101] input_handle_events_filter+0x58/0xc0 [ 59.063818] input_pass_values+0x1fc/0x2e8 [ 59.067924] input_handle_event+0x450/0x558 [ 59.072118] input_event+0x6c/0xb8 [ 59.075525] hidinput_report_event+0x4c/0x78 [ 59.079806] hid_report_raw_event+0xb4/0x418 [ 59.084086] hid_input_report+0x138/0x1a0 [ 59.088104] hid_irq_in+0x180/0x308 [ 59.091600] __usb_hcd_giveback_urb+0x98/0x160 [ 59.096055] usb_giveback_urb_bh+0xb0/0x150 [ 59.100248] process_one_work+0x20c/0x5e8 [ 59.104269] bh_worker+0x1b0/0x218 [ 59.107677] workqueue_softirq_action+0x80/0x98 [ 59.112220] tasklet_hi_action+0x20/0x60 [ 59.116153] handle_softirqs+0x144/0x528 [ 59.120085] __do_softirq+0x1c/0x28 [ 59.123581] ____do_softirq+0x18/0x30 [ 59.127250] call_on_irq_stack+0x30/0x70 [ 59.131181] do_softirq_own_stack+0x24/0x38 [ 59.135374] __irq_exit_rcu+0x100/0x1f8 [ 59.139219] irq_exit_rcu+0x18/0x58 [ 59.142714] el1_interrupt+0x38/0x68 [ 59.146297] el1h_64_irq_handler+0x18/0x28 [ 59.150402] el1h_64_irq+0x6c/0x70 [ 59.153809] default_idle_call+0x84/0x140 (P) [ 59.158177] do_idle+0x22c/0x298 [ 59.161412] cpu_startup_entry+0x40/0x50 [ 59.165344] rest_init+0x108/0x1a0 [ 59.168752] start_kernel+0x760/0x8c8 [ 59.172424] __primary_switched+0x88/0x98 [ 59.176444] irq event stamp: 279989 [ 59.179938] hardirqs last enabled at (279988): [<ffffffd08d350b64>] el1_interrupt+0x54/0x68 [ 59.188408] hardirqs last disabled at (279989): [<ffffffd08d34fc94>] el1_brk64+0x1c/0x50 [ 59.196530] softirqs last enabled at (279934): [<ffffffd08c4c2900>] handle_softirqs+0x508/0x528 [ 59.205349] softirqs last disabled at (279959): [<ffffffd08c4103ac>] __do_softirq+0x1c/0x28 [ 59.213731] ---[ end trace 0000000000000000 ]--- [ 59.218386] ------------[ cut here ]------------ [ 59.218388] WARNING: CPU: 0 PID: 0 at mm/slub.c:735 get_freelist+0x160/0x1d8 [ 59.230093] Modules linked in: netconsole quota_v2 quota_tree rfcomm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device algif_hash algif_skcipher af_alg bnep snd_soc_pcm512x_i2c snd_soc_pcm512x spidev regmap_i2c binfmt_misc hci_uart aes_ce_blk btbcm bluetooth ecdh_generic ecc aes_ce_cipher ghash_ce brcmfmac_cyw gf128mul snd_soc_hifiberry_dacplus sg raspberrypi_hwmon brcmfmac brcmutil i2c_brcmstb cfg80211 vc4 rfkill spi_bcm2835 rpi_hevc_dec snd_soc_hdmi_codec gpio_keys drm_exec drm_display_helper clk_hifiberry_dacpro cec pisp_be v4l2_mem2mem v3d videobuf2_dma_contig drm_dma_helper gpu_sched drm_shmem_helper drm_client_lib rp1_pci videobuf2_memops videobuf2_v4l2 videodev designware_i2s drm_kms_helper videobuf2_common mc snd_soc_core snd_compress i2c_designware_platform rp1_pio snd_pcm_dmaengine rp1_fw i2c_designware_core snd_pcm rp1_mailbox raspberrypi_gpiomem snd_timer rp1_adc snd nvmem_rmem uio_pdrv_genirq uio i2c_dev ledtrig_pattern drm fuse drm_panel_orientation_quirks backlight dm_mod nfsd auth_rpcgss ip_tables [ 59.230172] x_tables ipv6 [ 59.322954] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G W 6.17.0-rc2-v8-lockdep #532 PREEMPTLAZY [ 59.333694] Tainted: [W]=WARN [ 59.336664] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT) [ 59.342514] pstate: a04000c9 (NzCv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 59.349500] pc : get_freelist+0x160/0x1d8 [ 59.353519] lr : get_freelist+0x1c0/0x1d8 [ 59.357538] sp : ffffffc080003470 [ 59.360857] x29: ffffffc080003470 x28: 0000000000000001 x27: ffffffd08de07000 [ 59.368019] x26: ffffffd08d9ef508 x25: ffffff8100a3c240 x24: ffffffd08de0dd40 [ 59.375181] x23: fffffffec413dfa0 x22: 0000000080100010 x21: fffffffec413df80 [ 59.382343] x20: ffffff8104f7ef00 x19: 000000008010000e x18: 00000000ffffffff [ 59.389504] x17: ffffffb1714fe000 x16: ffffffd08d124020 x15: 0000000000000006 [ 59.396666] x14: 0000000000000000 x13: 00000000000c2000 x12: 0000000000000000 [ 59.403828] x11: 656820747563205b x10: ffffffd08e760810 x9 : ffffffd08e079000 [ 59.410990] x8 : ffffffd08de0dd40 x7 : 00000000eb9a37bf x6 : 0000000000000190 [ 59.418152] x5 : 00000000000000c8 x4 : 0000000000040000 x3 : 0000000000000001 [ 59.425314] x2 : ffffffb1714fe000 x1 : ffffffd08de0dd40 x0 : 0000000000000107 [ 59.432476] Call trace: [ 59.434923] get_freelist+0x160/0x1d8 (P) [ 59.438942] ___slab_alloc+0xf8/0x1178 [ 59.442699] __slab_alloc.constprop.0+0x6c/0xc8 [ 59.447242] kmem_cache_alloc_node_noprof+0x2c4/0x328 [ 59.452309] __alloc_skb+0x154/0x188 [ 59.455893] netpoll_send_udp+0xac/0x418 [ 59.459826] write_msg+0x124/0x1b0 [netconsole] [ 59.464370] console_flush_all+0x2e4/0x4a8 [ 59.468476] console_unlock+0x84/0x188 [ 59.472233] vprintk_emit+0x24c/0x450 [ 59.475903] vprintk_default+0x40/0x58 [ 59.479661] vprintk+0x3c/0x80 [ 59.482719] _printk+0x64/0x98 [ 59.485778] __handle_sysrq+0x1ec/0x308 [ 59.489622] sysrq_filter+0xcc/0x518 [ 59.493204] input_handle_events_filter+0x58/0xc0 [ 59.497921] input_pass_values+0x1fc/0x2e8 [ 59.502027] input_handle_event+0x450/0x558 [ 59.506220] input_event+0x6c/0xb8 [ 59.509627] hidinput_report_event+0x4c/0x78 [ 59.513907] hid_report_raw_event+0xb4/0x418 [ 59.518188] hid_input_report+0x138/0x1a0 [ 59.522206] hid_irq_in+0x180/0x308 [ 59.525700] __usb_hcd_giveback_urb+0x98/0x160 [ 59.530155] usb_giveback_urb_bh+0xb0/0x150 [ 59.534348] process_one_work+0x20c/0x5e8 [ 59.538368] bh_worker+0x1b0/0x218 [ 59.541776] workqueue_softirq_action+0x80/0x98 [ 59.546319] tasklet_hi_action+0x20/0x60 [ 59.550251] handle_softirqs+0x144/0x528 [ 59.554183] __do_softirq+0x1c/0x28 [ 59.557677] ____do_softirq+0x18/0x30 [ 59.561347] call_on_irq_stack+0x30/0x70 [ 59.565278] do_softirq_own_stack+0x24/0x38 [ 59.569471] __irq_exit_rcu+0x100/0x1f8 [ 59.573316] irq_exit_rcu+0x18/0x58 [ 59.576811] el1_interrupt+0x38/0x68 [ 59.580393] el1h_64_irq_handler+0x18/0x28 [ 59.584498] el1h_64_irq+0x6c/0x70 [ 59.587905] default_idle_call+0x84/0x140 (P) [ 59.592273] do_idle+0x22c/0x298 [ 59.595507] cpu_startup_entry+0x40/0x50 [ 59.599438] rest_init+0x108/0x1a0 [ 59.602846] start_kernel+0x760/0x8c8 [ 59.606517] __primary_switched+0x88/0x98 [ 59.610536] irq event stamp: 279993 [ 59.614030] hardirqs last enabled at (279992): [<ffffffd08d350b64>] el1_interrupt+0x54/0x68 [ 59.622499] hardirqs last disabled at (279993): [<ffffffd08d34fc94>] el1_brk64+0x1c/0x50 [ 59.630620] softirqs last enabled at (279934): [<ffffffd08c4c2900>] handle_softirqs+0x508/0x528 [ 59.639440] softirqs last disabled at (279959): [<ffffffd08c4103ac>] __do_softirq+0x1c/0x28 [ 59.647822] ---[ end trace 0000000000000000 ]--- [ 59.652452] [ 59.653938] ================================ [ 59.658217] WARNING: inconsistent lock state [ 59.662496] 6.17.0-rc2-v8-lockdep #532 Tainted: G W [ 59.668869] -------------------------------- [ 59.673148] inconsistent {IN-HARDIRQ-W} -> {HARDIRQ-ON-W} usage. [ 59.679172] swapper/0/0 [HC0[0]:SC1[1]:HE0:SE0] takes: [ 59.684325] ffffff81fef08f80 (&c->lock){?.-.}-{3:3}, at: ___slab_alloc+0x7c/0x1178 [ 59.691926] {IN-HARDIRQ-W} state was registered at: [ 59.696815] lock_acquire+0x1d4/0x368 [ 59.700573] put_cpu_partial+0x88/0x258 [ 59.704504] __slab_free+0x250/0x420 [ 59.708173] kmem_cache_free+0x288/0x3e8 [ 59.712192] mempool_free_slab+0x24/0x38 [ 59.716211] mempool_free+0x4c/0x120 [ 59.719881] bio_free+0x70/0x90 [ 59.723114] bio_put+0x120/0x278 [ 59.726433] end_bio_bh_io_sync+0x44/0x88 [ 59.730538] bio_endio+0x19c/0x280 [ 59.734032] blk_mq_end_request_batch+0x250/0x4c0 [ 59.738837] nvme_pci_complete_batch+0xdc/0x138 [ 59.743467] nvme_irq+0x7c/0x90 [ 59.746700] __handle_irq_event_percpu+0x94/0x318 [ 59.751503] handle_irq_event+0x54/0xb8 [ 59.755433] handle_fasteoi_irq+0x12c/0x270 [ 59.759714] handle_irq_desc+0x48/0x68 [ 59.763557] generic_handle_domain_irq+0x24/0x38 [ 59.768274] gic_handle_irq+0x48/0xd8 [ 59.772031] call_on_irq_stack+0x30/0x70 [ 59.776051] do_interrupt_handler+0xdc/0xe8 [ 59.780333] el1_interrupt+0x34/0x68 [ 59.784002] el1h_64_irq_handler+0x18/0x28 [ 59.788195] el1h_64_irq+0x6c/0x70 [ 59.791690] default_idle_call+0x84/0x140 [ 59.795796] do_idle+0x22c/0x298 [ 59.799119] cpu_startup_entry+0x3c/0x50 [ 59.803139] secondary_start_kernel+0x12c/0x158 [ 59.807772] __secondary_switched+0xc0/0xc8 [ 59.812054] irq event stamp: 279993 [ 59.815548] hardirqs last enabled at (279992): [<ffffffd08d350b64>] el1_interrupt+0x54/0x68 [ 59.824018] hardirqs last disabled at (279993): [<ffffffd08d34fc94>] el1_brk64+0x1c/0x50 [ 59.832141] softirqs last enabled at (279934): [<ffffffd08c4c2900>] handle_softirqs+0x508/0x528 [ 59.840963] softirqs last disabled at (279959): [<ffffffd08c4103ac>] __do_softirq+0x1c/0x28 [ 59.849345] [ 59.849345] other info that might help us debug this: [ 59.855894] Possible unsafe locking scenario: [ 59.855894] [ 59.861831] CPU0 [ 59.864278] ---- [ 59.866724] lock(&c->lock); [ 59.869696] <Interrupt> [ 59.872317] lock(&c->lock); [ 59.875464] [ 59.875464] *** DEADLOCK *** [ 59.875464] [ 59.881402] 11 locks held by swapper/0/0: [ 59.885420] #0: ffffff8100049948 ((wq_completion)events_bh_highpri){+.-.}-{0:0}, at: process_one_work+0x190/0x5e8 [ 59.895818] #1: ffffffc080003e60 ((work_completion)(&bh->bh)){+.-.}-{0:0}, at: process_one_work+0x1b8/0x5e8 [ 59.905691] #2: ffffff81034d3a30 (&dev->event_lock){+.-.}-{3:3}, at: input_event+0x50/0xb8 [ 59.914080] #3: ffffffd08de9c3c8 (rcu_read_lock){....}-{1:3}, at: input_pass_values+0x38/0x2e8 [ 59.922818] #4: ffffffd08de9c3c8 (rcu_read_lock){....}-{1:3}, at: __handle_sysrq+0x8/0x308 [ 59.931206] #5: ffffffd08de2a8e8 (console_lock){+.+.}-{0:0}, at: vprintk_default+0x40/0x58 [ 59.939596] #6: ffffffd08de2a940 (console_srcu){....}-{0:0}, at: console_flush_all+0x88/0x4a8 [ 59.948247] #7: ffffffd08de2ab00 (console_owner){+.-.}-{0:0}, at: console_lock_spinning_enable+0x50/0x90 [ 59.957858] #8: ffffffd08de2ab28 (printk_legacy_map-wait-type-override){+...}-{4:4}, at: console_flush_all+0x2a0/0x4a8 [ 59.968691] #9: ffffffd02c28e298 (target_list_lock){+.-.}-{3:3}, at: write_msg+0x60/0x1b0 [netconsole] [ 59.978128] #10: ffffff81fef08f80 (&c->lock){?.-.}-{3:3}, at: ___slab_alloc+0x7c/0x1178 [ 59.986256] [ 59.986256] stack backtrace: [ 59.990624] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G W 6.17.0-rc2-v8-lockdep #532 PREEMPTLAZY [ 60.001364] Tainted: [W]=WARN [ 60.004334] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT) [ 60.010185] Call trace: [ 60.012631] show_stack+0x20/0x38 (C) [ 60.016302] dump_stack_lvl+0x90/0xd0 [ 60.019974] dump_stack+0x18/0x28 [ 60.023297] print_usage_bug.part.0+0x29c/0x368 [ 60.027841] mark_lock+0x724/0x928 [ 60.031251] mark_held_locks+0x54/0x90 [ 60.035010] lockdep_hardirqs_on_prepare+0x108/0x228 [ 60.039992] arm64_exit_el1_dbg.isra.0+0x30/0x58 [ 60.044624] el1_brk64+0x40/0x50 [ 60.047859] el1h_64_sync_handler+0x68/0xb0 [ 60.052053] el1h_64_sync+0x6c/0x70 [ 60.055548] get_freelist+0x164/0x1d8 (P) [ 60.059569] ___slab_alloc+0xf8/0x1178 [ 60.063328] __slab_alloc.constprop.0+0x6c/0xc8 [ 60.067873] kmem_cache_alloc_node_noprof+0x2c4/0x328 [ 60.072941] __alloc_skb+0x154/0x188 [ 60.076526] netpoll_send_udp+0xac/0x418 [ 60.080460] write_msg+0x124/0x1b0 [netconsole] [ 60.085004] console_flush_all+0x2e4/0x4a8 [ 60.089112] console_unlock+0x84/0x188 [ 60.092871] vprintk_emit+0x24c/0x450 [ 60.096543] vprintk_default+0x40/0x58 [ 60.100301] vprintk+0x3c/0x80 [ 60.103362] _printk+0x64/0x98 [ 60.106422] __handle_sysrq+0x1ec/0x308 [ 60.110267] sysrq_filter+0xcc/0x518 [ 60.113850] input_handle_events_filter+0x58/0xc0 [ 60.118567] input_pass_values+0x1fc/0x2e8 [ 60.122674] input_handle_event+0x450/0x558 [ 60.126868] input_event+0x6c/0xb8 [ 60.130277] hidinput_report_event+0x4c/0x78 [ 60.134558] hid_report_raw_event+0xb4/0x418 [ 60.138838] hid_input_report+0x138/0x1a0 [ 60.142857] hid_irq_in+0x180/0x308 [ 60.146354] __usb_hcd_giveback_urb+0x98/0x160 [ 60.150810] usb_giveback_urb_bh+0xb0/0x150 [ 60.155004] process_one_work+0x20c/0x5e8 [ 60.159024] bh_worker+0x1b0/0x218 [ 60.162434] workqueue_softirq_action+0x80/0x98 [ 60.166979] tasklet_hi_action+0x20/0x60 [ 60.170913] handle_softirqs+0x144/0x528 [ 60.174846] __do_softirq+0x1c/0x28 [ 60.178342] ____do_softirq+0x18/0x30 [ 60.182012] call_on_irq_stack+0x30/0x70 [ 60.185944] do_softirq_own_stack+0x24/0x38 [ 60.190139] __irq_exit_rcu+0x100/0x1f8 [ 60.193985] irq_exit_rcu+0x18/0x58 [ 60.197482] el1_interrupt+0x38/0x68 [ 60.201064] el1h_64_irq_handler+0x18/0x28 [ 60.205171] el1h_64_irq+0x6c/0x70 [ 60.208579] default_idle_call+0x84/0x140 (P) [ 60.212947] do_idle+0x22c/0x298 [ 60.216183] cpu_startup_entry+0x40/0x50 [ 60.220117] rest_init+0x108/0x1a0 [ 60.223526] start_kernel+0x760/0x8c8 [ 60.227198] __primary_switched+0x88/0x98 [ 68.638857] sysrq: Trigger a crash [ 68.638885] Kernel panic - not syncing: sysrq triggered crash [ 68.648062] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G W 6.17.0-rc2-v8-lockdep #532 PREEMPTLAZY [ 68.658803] Tainted: [W]=WARN [ 68.661774] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT) [ 68.667626] Call trace: [ 68.670073] show_stack+0x20/0x38 (C) [ 68.673749] dump_stack_lvl+0x38/0xd0 [ 68.677423] dump_stack+0x18/0x28 [ 68.680745] vpanic+0x340/0x398 [ 68.683893] panic+0x64/0x70 [-- Attachment #4: netconsole-rpi5-6.17-eth0+patch-and+revert.log --] [-- Type: text/x-log, Size: 3698 bytes --] Note: applied "net: macb: Fix tx_ptr_lock locking" [ 40.676566] sysrq: HELP : loglevel(0-9) reboot(b) crash(c) show-all-locks(d) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) sak(k) show-backtrace-all-active-cpus(l) show-memory-usage(m) nice-all-RT-tasks(n) poweroff(o) show-registers(p) show-all-timers(q) unraw(r) sync(s) show-task-states(t) unmount(u) force-fb(v) show-blocked-tasks(w) dump-ftrace-buffer(z) replay-kernel-logs(R) [ 47.612947] sysrq: Trigger a crash [ 47.612967] Kernel panic - not syncing: sysrq triggered crash [ 47.622140] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G W 6.17.0-rc3-v8-lockdep #540 PREEMPTLAZY [ 47.632879] Tainted: [W]=WARN [ 47.635849] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT) [ 47.641700] Call trace: [ 47.644147] show_stack+0x20/0x38 (C) Note: add revert of "net: cadence: macb: Implement BQL" [ 47.108822] sysrq: HELP : loglevel(0-9) reboot(b) crash(c) show-all-locks(d) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) sak(k) show-backtrace-all-active-cpus(l) show-memory-usage(m) nice-all-RT-tasks(n) poweroff(o) show-registers(p) show-all-timers(q) unraw(r) sync(s) show-task-states(t) unmount(u) force-fb(v) show-blocked-tasks(w) dump-ftrace-buffer(z) replay-kernel-logs(R) [ 50.132998] sysrq: Trigger a crash [ 50.142195] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G W 6.17.0-rc3-v8-lockdep #541 PREEMPTLAZY [ 50.152935] Tainted: [W]=WARN [ 50.155904] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT) [ 50.161755] Call trace: [ 50.164201] show_stack+0x20/0x38 (C) [ 50.167872] dump_stack_lvl+0x38/0xd0 [ 50.171544] dump_stack+0x18/0x28 [ 50.174865] vpanic+0x340/0x398 [ 50.178012] panic+0x64/0x70 [ 50.180895] sysrq_handle_crash+0x50/0xa8 [ 50.184915] __handle_sysrq+0xe4/0x308 [ 50.188672] sysrq_filter+0xcc/0x518 [ 50.192254] input_handle_events_filter+0x58/0xc0 [ 50.196971] input_pass_values+0x1fc/0x2e8 [ 50.201078] input_handle_event+0x450/0x558 [ 50.205271] input_event+0x6c/0xb8 [ 50.208678] hidinput_report_event+0x4c/0x78 [ 50.212959] hid_report_raw_event+0xb4/0x418 [ 50.217240] hid_input_report+0x138/0x1a0 [ 50.133022] Kernel panic - not syncing: sysrq triggered crash [ 50.221258] hid_irq_in+0x180/0x308 [ 50.224753] __usb_hcd_giveback_urb+0x98/0x160 [ 50.229210] usb_giveback_urb_bh+0xb0/0x150 [ 50.233403] process_one_work+0x20c/0x5e8 [ 50.237422] bh_worker+0x1b0/0x218 [ 50.240830] workqueue_softirq_action+0x80/0x98 [ 50.245374] tasklet_hi_action+0x20/0x60 [ 50.249307] handle_softirqs+0x144/0x528 [ 50.253239] __do_softirq+0x1c/0x28 [ 50.256734] ____do_softirq+0x18/0x30 [ 50.260404] call_on_irq_stack+0x30/0x70 [ 50.264335] do_softirq_own_stack+0x24/0x38 [ 50.268529] __irq_exit_rcu+0x100/0x1f8 [ 50.272374] irq_exit_rcu+0x18/0x58 [ 50.275869] el1_interrupt+0x38/0x68 [ 50.279452] el1h_64_irq_handler+0x18/0x28 [ 50.283557] el1h_64_irq+0x6c/0x70 [ 50.286964] default_idle_call+0x84/0x140 (P) [ 50.291332] do_idle+0x22c/0x298 [ 50.294567] cpu_startup_entry+0x40/0x50 [ 50.298499] rest_init+0x108/0x1a0 [ 50.301907] start_kernel+0x760/0x8c8 [ 50.305577] __primary_switched+0x88/0x98 [ 50.309598] SMP: stopping secondary CPUs [ 51.358529] SMP: failed to stop secondary CPUs 2 [ 51.363720] Kernel Offset: 0x100c400000 from 0xffffffc080000000 [ 51.369669] PHYS_OFFSET: 0x0 [ 51.372558] CPU features: 0x40000,0000b800,12023140,0400720b [ 51.378241] Memory Limit: none [ 51.381310] ---[ end Kernel panic - not syncing: sysrq triggered crash ]--- ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: netpoll: raspberrypi [4 5] driver locking woes 2025-08-28 17:26 ` Mike Galbraith @ 2025-08-28 18:56 ` Sean Anderson 2025-08-29 2:55 ` Mike Galbraith 0 siblings, 1 reply; 7+ messages in thread From: Sean Anderson @ 2025-08-28 18:56 UTC (permalink / raw) To: Mike Galbraith, Robert Hancock; +Cc: Breno Leitao, netdev@vger.kernel.org [-- Attachment #1: Type: text/plain, Size: 2058 bytes --] Hi Mike, On 8/28/25 13:26, Mike Galbraith wrote: > On Thu, 2025-08-28 at 10:57 -0400, Sean Anderson wrote: >> Hi Mike, >> >> On 8/27/25 12:02, Mike Galbraith wrote: >> > Unexpected addendum to done deal datapoint, so off list. >> > >> > On Tue, 2025-08-26 at 11:49 +0200, Mike Galbraith wrote: >> > > >> > > The pi5 gripe fix is equally trivial, but submitting that is pointless >> > > given there's something else amiss in fingered commit. This is all of >> > > the crash info that escapes the box w/wo gripes silenced. >> > > >> > > [ 51.688868] sysrq: Trigger a crash >> > > [ 51.688892] Kernel panic - not syncing: sysrq triggered crash >> > > [ 51.698066] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.17.0-rc2-v8-lockdep #533 PREEMPTLAZY >> > > [ 51.707234] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT) >> > > [ 51.713085] Call trace: >> > > [ 51.715532] show_stack+0x20/0x38 (C) >> > > [ 51.719206] dump_stack_lvl+0x38/0xd0 >> > > [ 51.722878] dump_stack+0x18/0x28 >> > > >> > > That aspect is a punt and run atm (time.. and dash of laziness:). >> > >> > Plan was to end datapoint thread, but after booting pi5's 6.12 kernel, >> > for some reason I fired up netconsole.. and box promptly exhibited the >> > netpoll locking bug warning, indicating presence of 138badbc21a0. >> > Instead of saying to self "nope, just walk away", I poked SysRq-C.. and >> > the bloody damn monitoring box received a 100% complete death rattle. >> > Well bugger. >> >> Did you get a backtrace for this? > > Yes, logs for 6.12.41 and 6.17.0-rc2 attached. > > Since a patch has meanwhile landed, also a log of patched 6.17.0-rc3 > now gripe free (yay) but with aforementioned broken output, followed by > addition of the e6a532185daa revert to confirm it still cures that. > >> And to be clear, the steps to reproduce this are to boot a kernel with >> lockdep enabled with netconsole on macb and then hit sysrq? > > Yup. Looks like the tx completion path can also be called from netpoll. Can you try the attached patch? --Sean [-- Attachment #2: 0001-net-macb-Fix-tx_ptr_lock-locking.patch --] [-- Type: text/x-patch, Size: 4585 bytes --] From 3fe439760955370b9f005ef2728b5fc9b14eeea5 Mon Sep 17 00:00:00 2001 From: Sean Anderson <sean.anderson@linux.dev> Date: Thu, 28 Aug 2025 11:55:01 -0400 Subject: [PATCH net v2] net: macb: Fix tx_ptr_lock locking macb_start_xmit can be called with bottom-halves disabled (e.g. transmitting from softirqs) as well as with interrupts disabled (with netpoll). Because of this, all other functions taking tx_ptr_lock must disable IRQs, and macb_start_xmit must only re-enable IRQs if they were already enabled. Fixes: 138badbc21a0 ("net: macb: use NAPI for TX completion path") Reported-by: Mike Galbraith <efault@gmx.de> Signed-off-by: Sean Anderson <sean.anderson@linux.dev> --- Changes in v2: - Use irqsave/restore for all accesses, since they can also also be called from netpoll. drivers/net/ethernet/cadence/macb_main.c | 28 ++++++++++++++---------- 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/drivers/net/ethernet/cadence/macb_main.c b/drivers/net/ethernet/cadence/macb_main.c index 16d28a8b3b56..c769b7dbd3ba 100644 --- a/drivers/net/ethernet/cadence/macb_main.c +++ b/drivers/net/ethernet/cadence/macb_main.c @@ -1223,12 +1223,13 @@ static int macb_tx_complete(struct macb_queue *queue, int budget) { struct macb *bp = queue->bp; u16 queue_index = queue - bp->queues; + unsigned long flags; unsigned int tail; unsigned int head; int packets = 0; u32 bytes = 0; - spin_lock(&queue->tx_ptr_lock); + spin_lock_irqsave(&queue->tx_ptr_lock, flags); head = queue->tx_head; for (tail = queue->tx_tail; tail != head && packets < budget; tail++) { struct macb_tx_skb *tx_skb; @@ -1291,7 +1292,7 @@ static int macb_tx_complete(struct macb_queue *queue, int budget) CIRC_CNT(queue->tx_head, queue->tx_tail, bp->tx_ring_size) <= MACB_TX_WAKEUP_THRESH(bp)) netif_wake_subqueue(bp->dev, queue_index); - spin_unlock(&queue->tx_ptr_lock); + spin_unlock_irqrestore(&queue->tx_ptr_lock, flags); return packets; } @@ -1707,8 +1708,9 @@ static void macb_tx_restart(struct macb_queue *queue) { struct macb *bp = queue->bp; unsigned int head_idx, tbqp; + unsigned long flags; - spin_lock(&queue->tx_ptr_lock); + spin_lock_irqsave(&queue->tx_ptr_lock, flags); if (queue->tx_head == queue->tx_tail) goto out_tx_ptr_unlock; @@ -1720,19 +1722,20 @@ static void macb_tx_restart(struct macb_queue *queue) if (tbqp == head_idx) goto out_tx_ptr_unlock; - spin_lock_irq(&bp->lock); + spin_lock(&bp->lock); macb_writel(bp, NCR, macb_readl(bp, NCR) | MACB_BIT(TSTART)); - spin_unlock_irq(&bp->lock); + spin_unlock(&bp->lock); out_tx_ptr_unlock: - spin_unlock(&queue->tx_ptr_lock); + spin_unlock_irqrestore(&queue->tx_ptr_lock, flags); } static bool macb_tx_complete_pending(struct macb_queue *queue) { bool retval = false; + unsigned long flags; - spin_lock(&queue->tx_ptr_lock); + spin_lock_irqsave(&queue->tx_ptr_lock, flags); if (queue->tx_head != queue->tx_tail) { /* Make hw descriptor updates visible to CPU */ rmb(); @@ -1740,7 +1743,7 @@ static bool macb_tx_complete_pending(struct macb_queue *queue) if (macb_tx_desc(queue, queue->tx_tail)->ctrl & MACB_BIT(TX_USED)) retval = true; } - spin_unlock(&queue->tx_ptr_lock); + spin_unlock_irqrestore(&queue->tx_ptr_lock, flags); return retval; } @@ -2308,6 +2311,7 @@ static netdev_tx_t macb_start_xmit(struct sk_buff *skb, struct net_device *dev) struct macb_queue *queue = &bp->queues[queue_index]; unsigned int desc_cnt, nr_frags, frag_size, f; unsigned int hdrlen; + unsigned long flags; bool is_lso; netdev_tx_t ret = NETDEV_TX_OK; @@ -2368,7 +2372,7 @@ static netdev_tx_t macb_start_xmit(struct sk_buff *skb, struct net_device *dev) desc_cnt += DIV_ROUND_UP(frag_size, bp->max_tx_length); } - spin_lock_bh(&queue->tx_ptr_lock); + spin_lock_irqsave(&queue->tx_ptr_lock, flags); /* This is a hard error, log it. */ if (CIRC_SPACE(queue->tx_head, queue->tx_tail, @@ -2392,15 +2396,15 @@ static netdev_tx_t macb_start_xmit(struct sk_buff *skb, struct net_device *dev) netdev_tx_sent_queue(netdev_get_tx_queue(bp->dev, queue_index), skb->len); - spin_lock_irq(&bp->lock); + spin_lock(&bp->lock); macb_writel(bp, NCR, macb_readl(bp, NCR) | MACB_BIT(TSTART)); - spin_unlock_irq(&bp->lock); + spin_unlock(&bp->lock); if (CIRC_SPACE(queue->tx_head, queue->tx_tail, bp->tx_ring_size) < 1) netif_stop_subqueue(dev, queue_index); unlock: - spin_unlock_bh(&queue->tx_ptr_lock); + spin_unlock_irqrestore(&queue->tx_ptr_lock, flags); return ret; } -- 2.35.1.1320.gc452695387.dirty ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: netpoll: raspberrypi [4 5] driver locking woes 2025-08-28 18:56 ` Sean Anderson @ 2025-08-29 2:55 ` Mike Galbraith 0 siblings, 0 replies; 7+ messages in thread From: Mike Galbraith @ 2025-08-29 2:55 UTC (permalink / raw) To: Sean Anderson, Robert Hancock; +Cc: Breno Leitao, netdev@vger.kernel.org On Thu, 2025-08-28 at 14:56 -0400, Sean Anderson wrote: > > Looks like the tx completion path can also be called from netpoll. Can > you try the attached patch? Done, zero netpoll or lockdep complaints. -Mike ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2025-08-29 2:55 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-08-25 5:53 netpoll: raspberrypi [4 5] driver locking woes Mike Galbraith
2025-08-25 10:10 ` Mike Galbraith
2025-08-26 9:49 ` Mike Galbraith
[not found] ` <f4fa3fcc637ffb6531982a90dbd9c27114e93036.camel@gmx.de>
2025-08-28 14:57 ` Sean Anderson
2025-08-28 17:26 ` Mike Galbraith
2025-08-28 18:56 ` Sean Anderson
2025-08-29 2:55 ` Mike Galbraith
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.