* [BUG] Soft lockup on powerpc when running arena selftests
@ 2024-11-07 12:38 Viktor Malik
2024-11-07 15:46 ` Alexei Starovoitov
0 siblings, 1 reply; 3+ messages in thread
From: Viktor Malik @ 2024-11-07 12:38 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Martin KaFai Lau, Eduard Zingerman, Song Liu, Yonghong Song,
John Fastabend, KP Singh, Stanislav Fomichev, Hao Luo, Jiri Olsa
[-- Attachment #1: Type: text/plain, Size: 3419 bytes --]
Hi,
I'm getting soft lockups when running the BPF arena selftests on powerpc
(ppcle64). The issue is 100% reproducible on the latest bpf-next with
`./test_progs -t arena`.
A console snippet for one CPU lockup looks like this:
[ 1124.671746] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u34:0:58]
[ 1124.675554] CPU#1 Utilization every 4s during lockup:
[ 1124.675584] #1: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.675621] #2: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.675659] #3: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.675696] #4: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.675733] #5: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.675770] Modules linked in: bpf_testmod(OE) bonding tls rfkill virtio_net net_failover vmx_crypto failover virtio_balloon crct10dif_vpmsum fuse loop nfnetlink zram vsock_loopback vmw_vsock_virtio_transport_common vsock virtio_blk crc32c_vpmsum virtio_console
[ 1124.675921] CPU: 1 UID: 0 PID: 58 Comm: kworker/u34:0 Tainted: G OE 6.12.0-rc4+ #1
[ 1124.675975] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
[ 1124.676005] Hardware name: IBM pSeries (emulated by qemu) POWER8E (raw) 0x4b0201 of:SLOF,HEAD hv:linux,kvm pSeries
[ 1124.676063] Workqueue: events_unbound bpf_map_free_deferred
[ 1124.676101] NIP: c000000000551d3c LR: c000000000551c30 CTR: c0000000004733b0
[ 1124.676145] REGS: c000000008a37a20 TRAP: 0900 Tainted: G OE (6.12.0-rc4+)
[ 1124.676189] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 44082828 XER: 00000000
[ 1124.676251] CFAR: 0000000000000000 IRQMASK: 0
[ 1124.676251] GPR00: c000000000551c30 c000000008a37cc0 c00000000214f800 0000000000000000
[ 1124.676251] GPR04: 000000000000003b c00c00000044e3c8 0000000000000000 0000000000000000
[ 1124.676251] GPR08: 0000000000000000 0000000000000000 0000000058006001 0000000024082828
[ 1124.676251] GPR12: c0000000004733b0 c00000003ffff480 c0000000043cb7c0 c0000000043b1028
[ 1124.676251] GPR16: c008000305f78000 0000000000000000 0000000000000001 0000000000000000
[ 1124.676251] GPR20: fffffffffffffe7f c008000305f77fff c000000003cbe780 c000000001b26120
[ 1124.676251] GPR24: c000000003da0380 ff7fffffffffefbf c000000003cbe780 0000000000000001
[ 1124.676251] GPR28: c008000206000000 0000000000000000 c0000000004733b0 c00bf073759e8000
[ 1124.676627] NIP [c000000000551d3c] __apply_to_page_range+0x55c/0xea0
[ 1124.676667] LR [c000000000551c30] __apply_to_page_range+0x450/0xea0
[ 1124.676706] Call Trace:
[ 1124.676730] [c000000008a37cc0] [c000000000551c30] __apply_to_page_range+0x450/0xea0 (unreliable)
[ 1124.676784] [c000000008a37de0] [c000000000473360] arena_map_free+0x70/0xc0
[ 1124.676824] [c000000008a37e10] [c0000000003ee324] bpf_map_free_deferred+0x94/0x110
[ 1124.676870] [c000000008a37e40] [c00000000019bf8c] process_one_work+0x1fc/0x520
[ 1124.676915] [c000000008a37ef0] [c00000000019d96c] worker_thread+0x33c/0x4f0
[ 1124.676954] [c000000008a37f90] [c0000000001aa7a4] kthread+0x134/0x140
[ 1124.676992] [c000000008a37fe0] [c00000000000dd58] start_kernel_thread+0x14/0x18
[ 1124.677038] Code: 60420000 7d29a039 4082ff00 3fff0001 7c3cf840 4182ff28 e9370000 e90e0000 81490020 79070022 5506c03e 54e9c03e <5106421e> 50e9421e 5106463e 714a0040
There are more CPUs affected, the full console log is attached.
Thanks.
Viktor
[-- Attachment #2: console.log --]
[-- Type: text/x-log, Size: 33407 bytes --]
[ 1099.746415] bpf_testmod: loading out-of-tree module taints kernel.
[ 1099.750657] bpf_testmod: module verification failed: signature and/or required key missing - tainting kernel
[ 1124.671746] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u34:0:58]
[ 1124.675554] CPU#1 Utilization every 4s during lockup:
[ 1124.675584] #1: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.675621] #2: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.675659] #3: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.675696] #4: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.675733] #5: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.675770] Modules linked in: bpf_testmod(OE) bonding tls rfkill virtio_net net_failover vmx_crypto failover virtio_balloon crct10dif_vpmsum fuse loop nfnetlink zram vsock_loopback vmw_vsock_virtio_transport_common vsock virtio_blk crc32c_vpmsum virtio_console
[ 1124.675921] CPU: 1 UID: 0 PID: 58 Comm: kworker/u34:0 Tainted: G OE 6.12.0-rc4+ #1
[ 1124.675975] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
[ 1124.676005] Hardware name: IBM pSeries (emulated by qemu) POWER8E (raw) 0x4b0201 of:SLOF,HEAD hv:linux,kvm pSeries
[ 1124.676063] Workqueue: events_unbound bpf_map_free_deferred
[ 1124.676101] NIP: c000000000551d3c LR: c000000000551c30 CTR: c0000000004733b0
[ 1124.676145] REGS: c000000008a37a20 TRAP: 0900 Tainted: G OE (6.12.0-rc4+)
[ 1124.676189] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 44082828 XER: 00000000
[ 1124.676251] CFAR: 0000000000000000 IRQMASK: 0
[ 1124.676251] GPR00: c000000000551c30 c000000008a37cc0 c00000000214f800 0000000000000000
[ 1124.676251] GPR04: 000000000000003b c00c00000044e3c8 0000000000000000 0000000000000000
[ 1124.676251] GPR08: 0000000000000000 0000000000000000 0000000058006001 0000000024082828
[ 1124.676251] GPR12: c0000000004733b0 c00000003ffff480 c0000000043cb7c0 c0000000043b1028
[ 1124.676251] GPR16: c008000305f78000 0000000000000000 0000000000000001 0000000000000000
[ 1124.676251] GPR20: fffffffffffffe7f c008000305f77fff c000000003cbe780 c000000001b26120
[ 1124.676251] GPR24: c000000003da0380 ff7fffffffffefbf c000000003cbe780 0000000000000001
[ 1124.676251] GPR28: c008000206000000 0000000000000000 c0000000004733b0 c00bf073759e8000
[ 1124.676627] NIP [c000000000551d3c] __apply_to_page_range+0x55c/0xea0
[ 1124.676667] LR [c000000000551c30] __apply_to_page_range+0x450/0xea0
[ 1124.676706] Call Trace:
[ 1124.676730] [c000000008a37cc0] [c000000000551c30] __apply_to_page_range+0x450/0xea0 (unreliable)
[ 1124.676784] [c000000008a37de0] [c000000000473360] arena_map_free+0x70/0xc0
[ 1124.676824] [c000000008a37e10] [c0000000003ee324] bpf_map_free_deferred+0x94/0x110
[ 1124.676870] [c000000008a37e40] [c00000000019bf8c] process_one_work+0x1fc/0x520
[ 1124.676915] [c000000008a37ef0] [c00000000019d96c] worker_thread+0x33c/0x4f0
[ 1124.676954] [c000000008a37f90] [c0000000001aa7a4] kthread+0x134/0x140
[ 1124.676992] [c000000008a37fe0] [c00000000000dd58] start_kernel_thread+0x14/0x18
[ 1124.677038] Code: 60420000 7d29a039 4082ff00 3fff0001 7c3cf840 4182ff28 e9370000 e90e0000 81490020 79070022 5506c03e 54e9c03e <5106421e> 50e9421e 5106463e 714a0040
[ 1124.801744] watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [kworker/u37:7:34302]
[ 1124.805191] CPU#4 Utilization every 4s during lockup:
[ 1124.805221] #1: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.805260] #2: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.805297] #3: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.805335] #4: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.805373] #5: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.805410] Modules linked in: bpf_testmod(OE) bonding tls rfkill virtio_net net_failover vmx_crypto failover virtio_balloon crct10dif_vpmsum fuse loop nfnetlink zram vsock_loopback vmw_vsock_virtio_transport_common vsock virtio_blk crc32c_vpmsum virtio_console
[ 1124.805553] CPU: 4 UID: 0 PID: 34302 Comm: kworker/u37:7 Tainted: G OEL 6.12.0-rc4+ #1
[ 1124.805607] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP
[ 1124.805643] Hardware name: IBM pSeries (emulated by qemu) POWER8E (raw) 0x4b0201 of:SLOF,HEAD hv:linux,kvm pSeries
[ 1124.805701] Workqueue: events_unbound bpf_map_free_deferred
[ 1124.805738] NIP: c000000000551d50 LR: c000000000551c30 CTR: c0000000004733b0
[ 1124.805782] REGS: c00000011401ba20 TRAP: 0900 Tainted: G OEL (6.12.0-rc4+)
[ 1124.805826] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 24082282 XER: 00000000
[ 1124.805887] CFAR: 0000000000000000 IRQMASK: 0
[ 1124.805887] GPR00: c000000000551c30 c00000011401bcc0 c00000000214f800 0000000000000000
[ 1124.805887] GPR04: 0000000000000124 c00c00000044e408 0000000000000000 0000000000000000
[ 1124.805887] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000024082282
[ 1124.805887] GPR12: c0000000004733b0 c00000003fffc480 c0000000043c87a0 c0000000043b0028
[ 1124.805887] GPR16: c008000105f38000 0000000000000000 0000000000000001 0000000000000000
[ 1124.805887] GPR20: fffffffffffffe7f c008000105f37fff c000000003cbe780 c000000001b26120
[ 1124.805887] GPR24: c000000003da0380 ff7fffffffffefbf c000000003cbe780 0000000000000001
[ 1124.805887] GPR28: c008000006000000 0000000000000000 c0000000004733b0 c00bf743dfb38000
[ 1124.806260] NIP [c000000000551d50] __apply_to_page_range+0x570/0xea0
[ 1124.806301] LR [c000000000551c30] __apply_to_page_range+0x450/0xea0
[ 1124.806339] Call Trace:
[ 1124.806354] [c00000011401bcc0] [c000000000551c30] __apply_to_page_range+0x450/0xea0 (unreliable)
[ 1124.806408] [c00000011401bde0] [c000000000473360] arena_map_free+0x70/0xc0
[ 1124.806448] [c00000011401be10] [c0000000003ee324] bpf_map_free_deferred+0x94/0x110
[ 1124.806494] [c00000011401be40] [c00000000019bf8c] process_one_work+0x1fc/0x520
[ 1124.806540] [c00000011401bef0] [c00000000019d96c] worker_thread+0x33c/0x4f0
[ 1124.806579] [c00000011401bf90] [c0000000001aa7a4] kthread+0x134/0x140
[ 1124.806617] [c00000011401bfe0] [c00000000000dd58] start_kernel_thread+0x14/0x18
[ 1124.806662] Code: 4182ff28 e9370000 e90e0000 81490020 79070022 5506c03e 54e9c03e 5106421e 50e9421e 5106463e 714a0040 50e9463e <78c9000e> 4082ffbc 7d29c839 4082feb8
[ 1124.821743] watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [kworker/u34:3:47428]
[ 1124.821809] CPU#5 Utilization every 4s during lockup:
[ 1124.821842] #1: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.821883] #2: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.821923] #3: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.821962] #4: 101% system, 0% softirq, 1% hardirq, 0% idle
[ 1124.822001] #5: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.822041] Modules linked in: bpf_testmod(OE) bonding tls rfkill virtio_net net_failover vmx_crypto failover virtio_balloon crct10dif_vpmsum fuse loop nfnetlink zram vsock_loopback vmw_vsock_virtio_transport_common vsock virtio_blk crc32c_vpmsum virtio_console
[ 1124.822189] CPU: 5 UID: 0 PID: 47428 Comm: kworker/u34:3 Tainted: G OEL 6.12.0-rc4+ #1
[ 1124.822245] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP
[ 1124.822284] Hardware name: IBM pSeries (emulated by qemu) POWER8E (raw) 0x4b0201 of:SLOF,HEAD hv:linux,kvm pSeries
[ 1124.822343] Workqueue: events_unbound bpf_map_free_deferred
[ 1124.822381] NIP: c000000000551d50 LR: c000000000551c30 CTR: c0000000004733b0
[ 1124.822428] REGS: c0000000234dfa20 TRAP: 0900 Tainted: G OEL (6.12.0-rc4+)
[ 1124.822474] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 24082808 XER: 00000000
[ 1124.822538] CFAR: 0000000000000000 IRQMASK: 0
[ 1124.822538] GPR00: c000000000551c30 c0000000234dfcc0 c00000000214f800 0000000000000000
[ 1124.822538] GPR04: 0000000000000026 c00c000000163708 0000000000000000 0000000000000000
[ 1124.822538] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000024082808
[ 1124.822538] GPR12: c0000000004733b0 c00000003ffdb080 c0000000043ca7b0 c0000000043b0828
[ 1124.822538] GPR16: c008000205f58000 0000000000000000 0000000000000001 0000000000000000
[ 1124.822538] GPR20: fffffffffffffe7f c008000205f57fff c000000003cbe780 c000000001b26120
[ 1124.822538] GPR24: c000000003da0380 ff7fffffffffefbf c000000003cbe780 0000000000000001
[ 1124.822538] GPR28: c008000106000000 0000000000000000 c0000000004733b0 c00bf7e896038000
[ 1124.822930] NIP [c000000000551d50] __apply_to_page_range+0x570/0xea0
[ 1124.822973] LR [c000000000551c30] __apply_to_page_range+0x450/0xea0
[ 1124.823014] Call Trace:
[ 1124.823030] [c0000000234dfcc0] [c000000000551c30] __apply_to_page_range+0x450/0xea0 (unreliable)
[ 1124.823087] [c0000000234dfde0] [c000000000473360] arena_map_free+0x70/0xc0
[ 1124.823130] [c0000000234dfe10] [c0000000003ee324] bpf_map_free_deferred+0x94/0x110
[ 1124.823179] [c0000000234dfe40] [c00000000019bf8c] process_one_work+0x1fc/0x520
[ 1124.823229] [c0000000234dfef0] [c00000000019d96c] worker_thread+0x33c/0x4f0
[ 1124.823271] [c0000000234dff90] [c0000000001aa7a4] kthread+0x134/0x140
[ 1124.823311] [c0000000234dffe0] [c00000000000dd58] start_kernel_thread+0x14/0x18
[ 1124.823360] Code: 4182ff28 e9370000 e90e0000 81490020 79070022 5506c03e 54e9c03e 5106421e 50e9421e 5106463e 714a0040 50e9463e <78c9000e> 4082ffbc 7d29c839 4082feb8
[ 1148.671553] watchdog: BUG: soft lockup - CPU#1 stuck for 46s! [kworker/u34:0:58]
[ 1148.675300] CPU#1 Utilization every 4s during lockup:
[ 1148.675330] #1: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1148.675375] #2: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1148.675414] #3: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1148.675454] #4: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1148.675493] #5: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1148.675533] Modules linked in: bpf_testmod(OE) bonding tls rfkill virtio_net net_failover vmx_crypto failover virtio_balloon crct10dif_vpmsum fuse loop nfnetlink zram vsock_loopback vmw_vsock_virtio_transport_common vsock virtio_blk crc32c_vpmsum virtio_console
[ 1148.675696] CPU: 1 UID: 0 PID: 58 Comm: kworker/u34:0 Tainted: G OEL 6.12.0-rc4+ #1
[ 1148.675758] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP
[ 1148.675797] Hardware name: IBM pSeries (emulated by qemu) POWER8E (raw) 0x4b0201 of:SLOF,HEAD hv:linux,kvm pSeries
[ 1148.675863] Workqueue: events_unbound bpf_map_free_deferred
[ 1148.675910] NIP: c000000000551d50 LR: c000000000551c30 CTR: c0000000004733b0
[ 1148.675958] REGS: c000000008a37a20 TRAP: 0900 Tainted: G OEL (6.12.0-rc4+)
[ 1148.676010] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 24082828 XER: 00000000
[ 1148.676080] CFAR: 0000000000000000 IRQMASK: 0
[ 1148.676080] GPR00: c000000000551c30 c000000008a37cc0 c00000000214f800 0000000000000000
[ 1148.676080] GPR04: 000000000000003b c00c00000044e3c8 0000000000000000 0000000000000000
[ 1148.676080] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000024082828
[ 1148.676080] GPR12: c0000000004733b0 c00000003ffff480 c0000000043cb7c0 c0000000043b1028
[ 1148.676080] GPR16: c008000305f78000 0000000000000000 0000000000000001 0000000000000000
[ 1148.676080] GPR20: fffffffffffffe7f c008000305f77fff c000000003cbe780 c000000001b26120
[ 1148.676080] GPR24: c000000003da0380 ff7fffffffffefbf c000000003cbe780 0000000000000001
[ 1148.676080] GPR28: c008000206000000 0000000000000000 c0000000004733b0 c00fc45376338000
[ 1148.676496] NIP [c000000000551d50] __apply_to_page_range+0x570/0xea0
[ 1148.676544] LR [c000000000551c30] __apply_to_page_range+0x450/0xea0
[ 1148.676593] Call Trace:
[ 1148.676611] [c000000008a37cc0] [c000000000551c30] __apply_to_page_range+0x450/0xea0 (unreliable)
[ 1148.676677] [c000000008a37de0] [c000000000473360] arena_map_free+0x70/0xc0
[ 1148.676720] [c000000008a37e10] [c0000000003ee324] bpf_map_free_deferred+0x94/0x110
[ 1148.676773] [c000000008a37e40] [c00000000019bf8c] process_one_work+0x1fc/0x520
[ 1148.676824] [c000000008a37ef0] [c00000000019d96c] worker_thread+0x33c/0x4f0
[ 1148.676874] [c000000008a37f90] [c0000000001aa7a4] kthread+0x134/0x140
[ 1148.676920] [c000000008a37fe0] [c00000000000dd58] start_kernel_thread+0x14/0x18
[ 1148.676970] Code: 4182ff28 e9370000 e90e0000 81490020 79070022 5506c03e 54e9c03e 5106421e 50e9421e 5106463e 714a0040 50e9463e <78c9000e> 4082ffbc 7d29c839 4082feb8
[ 1148.801551] watchdog: BUG: soft lockup - CPU#4 stuck for 46s! [kworker/u37:7:34302]
[ 1148.805287] CPU#4 Utilization every 4s during lockup:
[ 1148.805345] #1: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1148.805387] #2: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1148.805429] #3: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1148.805467] #4: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1148.805507] #5: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1148.805546] Modules linked in: bpf_testmod(OE) bonding tls rfkill virtio_net net_failover vmx_crypto failover virtio_balloon crct10dif_vpmsum fuse loop nfnetlink zram vsock_loopback vmw_vsock_virtio_transport_common vsock virtio_blk crc32c_vpmsum virtio_console
[ 1148.805696] CPU: 4 UID: 0 PID: 34302 Comm: kworker/u37:7 Tainted: G OEL 6.12.0-rc4+ #1
[ 1148.805754] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP
[ 1148.805792] Hardware name: IBM pSeries (emulated by qemu) POWER8E (raw) 0x4b0201 of:SLOF,HEAD hv:linux,kvm pSeries
[ 1148.805853] Workqueue: events_unbound bpf_map_free_deferred
[ 1148.805890] NIP: c000000000551d50 LR: c000000000551c30 CTR: c0000000004733b0
[ 1148.805937] REGS: c00000011401ba20 TRAP: 0900 Tainted: G OEL (6.12.0-rc4+)
[ 1148.805983] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 24082282 XER: 00000000
[ 1148.806046] CFAR: 0000000000000000 IRQMASK: 0
[ 1148.806046] GPR00: c000000000551c30 c00000011401bcc0 c00000000214f800 0000000000000000
[ 1148.806046] GPR04: 0000000000000124 c00c00000044e408 0000000000000000 0000000000000000
[ 1148.806046] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000024082282
[ 1148.806046] GPR12: c0000000004733b0 c00000003fffc480 c0000000043c87a0 c0000000043b0028
[ 1148.806046] GPR16: c008000105f38000 0000000000000000 0000000000000001 0000000000000000
[ 1148.806046] GPR20: fffffffffffffe7f c008000105f37fff c000000003cbe780 c000000001b26120
[ 1148.806046] GPR24: c000000003da0380 ff7fffffffffefbf c000000003cbe780 0000000000000001
[ 1148.806046] GPR28: c008000006000000 0000000000000000 c0000000004733b0 c00fcadcfbb08000
[ 1148.806439] NIP [c000000000551d50] __apply_to_page_range+0x570/0xea0
[ 1148.806481] LR [c000000000551c30] __apply_to_page_range+0x450/0xea0
[ 1148.806523] Call Trace:
[ 1148.806539] [c00000011401bcc0] [c000000000551c30] __apply_to_page_range+0x450/0xea0 (unreliable)
[ 1148.806597] [c00000011401bde0] [c000000000473360] arena_map_free+0x70/0xc0
[ 1148.806639] [c00000011401be10] [c0000000003ee324] bpf_map_free_deferred+0x94/0x110
[ 1148.806688] [c00000011401be40] [c00000000019bf8c] process_one_work+0x1fc/0x520
[ 1148.806738] [c00000011401bef0] [c00000000019d96c] worker_thread+0x33c/0x4f0
[ 1148.806780] [c00000011401bf90] [c0000000001aa7a4] kthread+0x134/0x140
[ 1148.806821] [c00000011401bfe0] [c00000000000dd58] start_kernel_thread+0x14/0x18
[ 1148.806869] Code: 4182ff28 e9370000 e90e0000 81490020 79070022 5506c03e 54e9c03e 5106421e 50e9421e 5106463e 714a0040 50e9463e <78c9000e> 4082ffbc 7d29c839 4082feb8
[ 1148.821550] watchdog: BUG: soft lockup - CPU#5 stuck for 46s! [kworker/u34:3:47428]
[ 1148.821614] CPU#5 Utilization every 4s during lockup:
[ 1148.821651] #1: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1148.821693] #2: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1148.821752] #3: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1148.821794] #4: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1148.821846] #5: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1148.821886] Modules linked in: bpf_testmod(OE) bonding tls rfkill virtio_net net_failover vmx_crypto failover virtio_balloon crct10dif_vpmsum fuse loop nfnetlink zram vsock_loopback vmw_vsock_virtio_transport_common vsock virtio_blk crc32c_vpmsum virtio_console
[ 1148.822059] CPU: 5 UID: 0 PID: 47428 Comm: kworker/u34:3 Tainted: G OEL 6.12.0-rc4+ #1
[ 1148.822134] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP
[ 1148.822187] Hardware name: IBM pSeries (emulated by qemu) POWER8E (raw) 0x4b0201 of:SLOF,HEAD hv:linux,kvm pSeries
[ 1148.822260] Workqueue: events_unbound bpf_map_free_deferred
[ 1148.822296] NIP: c000000000551d50 LR: c000000000551c30 CTR: c0000000004733b0
[ 1148.822357] REGS: c0000000234dfa20 TRAP: 0900 Tainted: G OEL (6.12.0-rc4+)
[ 1148.822402] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 24082808 XER: 00000000
[ 1148.822483] CFAR: 0000000000000000 IRQMASK: 0
[ 1148.822483] GPR00: c000000000551c30 c0000000234dfcc0 c00000000214f800 0000000000000000
[ 1148.822483] GPR04: 0000000000000026 c00c000000163708 0000000000000000 0000000000000000
[ 1148.822483] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000024082808
[ 1148.822483] GPR12: c0000000004733b0 c00000003ffdb080 c0000000043ca7b0 c0000000043b0828
[ 1148.822483] GPR16: c008000205f58000 0000000000000000 0000000000000001 0000000000000000
[ 1148.822483] GPR20: fffffffffffffe7f c008000205f57fff c000000003cbe780 c000000001b26120
[ 1148.822483] GPR24: c000000003da0380 ff7fffffffffefbf c000000003cbe780 0000000000000001
[ 1148.822483] GPR28: c008000106000000 0000000000000000 c0000000004733b0 c00fcbe7f3158000
[ 1148.822964] NIP [c000000000551d50] __apply_to_page_range+0x570/0xea0
[ 1148.823023] LR [c000000000551c30] __apply_to_page_range+0x450/0xea0
[ 1148.823065] Call Trace:
[ 1148.823081] [c0000000234dfcc0] [c000000000551c30] __apply_to_page_range+0x450/0xea0 (unreliable)
[ 1148.823157] [c0000000234dfde0] [c000000000473360] arena_map_free+0x70/0xc0
[ 1148.823214] [c0000000234dfe10] [c0000000003ee324] bpf_map_free_deferred+0x94/0x110
[ 1148.823263] [c0000000234dfe40] [c00000000019bf8c] process_one_work+0x1fc/0x520
[ 1148.823329] [c0000000234dfef0] [c00000000019d96c] worker_thread+0x33c/0x4f0
[ 1148.823388] [c0000000234dff90] [c0000000001aa7a4] kthread+0x134/0x140
[ 1148.823429] [c0000000234dffe0] [c00000000000dd58] start_kernel_thread+0x14/0x18
[ 1148.823491] Code: 4182ff28 e9370000 e90e0000 81490020 79070022 5506c03e 54e9c03e 5106421e 50e9421e 5106463e 714a0040 50e9463e <78c9000e> 4082ffbc 7d29c839 4082feb8
[ 1159.951465] rcu: INFO: rcu_sched self-detected stall on CPU
[ 1159.954380] rcu: 4-....: (5999 ticks this GP) idle=788c/1/0x4000000000000002 softirq=64429/64430 fqs=2997
[ 1159.954436] rcu: (t=6000 jiffies g=36285 q=1497 ncpus=8)
[ 1159.954471] CPU: 4 UID: 0 PID: 34302 Comm: kworker/u37:7 Tainted: G OEL 6.12.0-rc4+ #1
[ 1159.954476] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP
[ 1159.954478] Hardware name: IBM pSeries (emulated by qemu) POWER8E (raw) 0x4b0201 of:SLOF,HEAD hv:linux,kvm pSeries
[ 1159.954481] Workqueue: events_unbound bpf_map_free_deferred
[ 1159.954491] NIP: c000000000551d60 LR: c000000000551c30 CTR: c0000000004733b0
[ 1159.954494] REGS: c00000011401ba20 TRAP: 0900 Tainted: G OEL (6.12.0-rc4+)
[ 1159.954496] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 24082282 XER: 00000000
[ 1159.954506] CFAR: 0000000000000000 IRQMASK: 0
[ 1159.954506] GPR00: c000000000551c30 c00000011401bcc0 c00000000214f800 0000000000000000
[ 1159.954506] GPR04: 0000000000000124 c00c00000044e408 0000000000000000 0000000000000000
[ 1159.954506] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000024082282
[ 1159.954506] GPR12: c0000000004733b0 c00000003fffc480 c0000000043c87a0 c0000000043b0028
[ 1159.954506] GPR16: c008000105f38000 0000000000000000 0000000000000001 0000000000000000
[ 1159.954506] GPR20: fffffffffffffe7f c008000105f37fff c000000003cbe780 c000000001b26120
[ 1159.954506] GPR24: c000000003da0380 ff7fffffffffefbf c000000003cbe780 0000000000000001
[ 1159.954506] GPR28: c008000006000000 0000000000000000 c0000000004733b0 c01191c4d0018000
[ 1159.954536] NIP [c000000000551d60] __apply_to_page_range+0x580/0xea0
[ 1159.954541] LR [c000000000551c30] __apply_to_page_range+0x450/0xea0
[ 1159.954545] Call Trace:
[ 1159.954546] [c00000011401bcc0] [c000000000551c30] __apply_to_page_range+0x450/0xea0 (unreliable)
[ 1159.954551] [c00000011401bde0] [c000000000473360] arena_map_free+0x70/0xc0
[ 1159.954555] [c00000011401be10] [c0000000003ee324] bpf_map_free_deferred+0x94/0x110
[ 1159.954559] [c00000011401be40] [c00000000019bf8c] process_one_work+0x1fc/0x520
[ 1159.954564] [c00000011401bef0] [c00000000019d96c] worker_thread+0x33c/0x4f0
[ 1159.954567] [c00000011401bf90] [c0000000001aa7a4] kthread+0x134/0x140
[ 1159.954570] [c00000011401bfe0] [c00000000000dd58] start_kernel_thread+0x14/0x18
[ 1159.954573] Code: 79070022 5506c03e 54e9c03e 5106421e 50e9421e 5106463e 714a0040 50e9463e 78c9000e 4082ffbc 7d29c839 4082feb8 <3fff0001> 7c3fe040 4082ffbc 4bfffedc
[ 1159.954586] Sending NMI from CPU 4 to CPUs 5:
[ 1159.954604] NMI backtrace for cpu 5
[ 1159.956027] CPU: 5 UID: 0 PID: 47428 Comm: kworker/u34:3 Tainted: G OEL 6.12.0-rc4+ #1
[ 1159.956102] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP
[ 1159.956141] Hardware name: IBM pSeries (emulated by qemu) POWER8E (raw) 0x4b0201 of:SLOF,HEAD hv:linux,kvm pSeries
[ 1159.956202] Workqueue: events_unbound bpf_map_free_deferred
[ 1159.956253] NIP: c000000000551d60 LR: c000000000551c30 CTR: c0000000004733b0
[ 1159.956300] REGS: c0000000234dfa20 TRAP: 0500 Tainted: G OEL (6.12.0-rc4+)
[ 1159.956346] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 24082808 XER: 00000000
[ 1159.956410] CFAR: 0000000000000000 IRQMASK: 0
[ 1159.956410] GPR00: c000000000551c30 c0000000234dfcc0 c00000000214f800 0000000000000000
[ 1159.956410] GPR04: 0000000000000026 c00c000000163708 0000000000000000 0000000000000000
[ 1159.956410] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000024082808
[ 1159.956410] GPR12: c0000000004733b0 c00000003ffdb080 c0000000043ca7b0 c0000000043b0828
[ 1159.956410] GPR16: c008000205f58000 0000000000000000 0000000000000001 0000000000000000
[ 1159.956410] GPR20: fffffffffffffe7f c008000205f57fff c000000003cbe780 c000000001b26120
[ 1159.956410] GPR24: c000000003da0380 ff7fffffffffefbf c000000003cbe780 0000000000000001
[ 1159.956410] GPR28: c008000106000000 0000000000000000 c0000000004733b0 c0119273ac8f8000
[ 1159.956800] NIP [c000000000551d60] __apply_to_page_range+0x580/0xea0
[ 1159.956858] LR [c000000000551c30] __apply_to_page_range+0x450/0xea0
[ 1159.956898] Call Trace:
[ 1159.956915] [c0000000234dfcc0] [c000000000551c30] __apply_to_page_range+0x450/0xea0 (unreliable)
[ 1159.956987] [c0000000234dfde0] [c000000000473360] arena_map_free+0x70/0xc0
[ 1159.957029] [c0000000234dfe10] [c0000000003ee324] bpf_map_free_deferred+0x94/0x110
[ 1159.957087] [c0000000234dfe40] [c00000000019bf8c] process_one_work+0x1fc/0x520
[ 1159.957135] [c0000000234dfef0] [c00000000019d96c] worker_thread+0x33c/0x4f0
[ 1159.957176] [c0000000234dff90] [c0000000001aa7a4] kthread+0x134/0x140
[ 1159.957217] [c0000000234dffe0] [c00000000000dd58] start_kernel_thread+0x14/0x18
[ 1159.957265] Code: 79070022 5506c03e 54e9c03e 5106421e 50e9421e 5106463e 714a0040 50e9463e 78c9000e 4082ffbc 7d29c839 4082feb8 <3fff0001> 7c3fe040 4082ffbc 4bfffedc
[ 1172.671360] watchdog: BUG: soft lockup - CPU#1 stuck for 68s! [kworker/u34:0:58]
[ 1172.673229] CPU#1 Utilization every 4s during lockup:
[ 1172.673259] #1: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1172.673298] #2: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1172.673337] #3: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1172.673377] #4: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1172.673417] #5: 100% system, 0% softirq, 1% hardirq, 0% idle
[ 1172.673456] Modules linked in: bpf_testmod(OE) bonding tls rfkill virtio_net net_failover vmx_crypto failover virtio_balloon crct10dif_vpmsum fuse loop nfnetlink zram vsock_loopback vmw_vsock_virtio_transport_common vsock virtio_blk crc32c_vpmsum virtio_console
[ 1172.673611] CPU: 1 UID: 0 PID: 58 Comm: kworker/u34:0 Tainted: G OEL 6.12.0-rc4+ #1
[ 1172.673673] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP
[ 1172.673711] Hardware name: IBM pSeries (emulated by qemu) POWER8E (raw) 0x4b0201 of:SLOF,HEAD hv:linux,kvm pSeries
[ 1172.673772] Workqueue: events_unbound bpf_map_free_deferred
[ 1172.673816] NIP: c000000000551d50 LR: c000000000551c30 CTR: c0000000004733b0
[ 1172.673863] REGS: c000000008a37a20 TRAP: 0900 Tainted: G OEL (6.12.0-rc4+)
[ 1172.673909] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 24082828 XER: 00000000
[ 1172.673974] CFAR: 0000000000000000 IRQMASK: 0
[ 1172.673974] GPR00: c000000000551c30 c000000008a37cc0 c00000000214f800 0000000000000000
[ 1172.673974] GPR04: 000000000000003b c00c00000044e3c8 0000000000000000 0000000000000000
[ 1172.673974] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000024082828
[ 1172.673974] GPR12: c0000000004733b0 c00000003ffff480 c0000000043cb7c0 c0000000043b1028
[ 1172.673974] GPR16: c008000305f78000 0000000000000000 0000000000000001 0000000000000000
[ 1172.673974] GPR20: fffffffffffffe7f c008000305f77fff c000000003cbe780 c000000001b26120
[ 1172.673974] GPR24: c000000003da0380 ff7fffffffffefbf c000000003cbe780 0000000000000001
[ 1172.673974] GPR28: c008000206000000 0000000000000000 c0000000004733b0 c013980b18d48000
[ 1172.674365] NIP [c000000000551d50] __apply_to_page_range+0x570/0xea0
[ 1172.674409] LR [c000000000551c30] __apply_to_page_range+0x450/0xea0
[ 1172.674450] Call Trace:
[ 1172.674467] [c000000008a37cc0] [c000000000551c30] __apply_to_page_range+0x450/0xea0 (unreliable)
[ 1172.674524] [c000000008a37de0] [c000000000473360] arena_map_free+0x70/0xc0
[ 1172.674567] [c000000008a37e10] [c0000000003ee324] bpf_map_free_deferred+0x94/0x110
[ 1172.674616] [c000000008a37e40] [c00000000019bf8c] process_one_work+0x1fc/0x520
[ 1172.674665] [c000000008a37ef0] [c00000000019d96c] worker_thread+0x33c/0x4f0
[ 1172.674707] [c000000008a37f90] [c0000000001aa7a4] kthread+0x134/0x140
[ 1172.674746] [c000000008a37fe0] [c00000000000dd58] start_kernel_thread+0x14/0x18
[ 1172.674795] Code: 4182ff28 e9370000 e90e0000 81490020 79070022 5506c03e 54e9c03e 5106421e 50e9421e 5106463e 714a0040 50e9463e <78c9000e> 4082ffbc 7d29c839 4082feb8
[ 1172.821358] watchdog: BUG: soft lockup - CPU#5 stuck for 68s! [kworker/u34:3:47428]
[ 1172.821774] CPU#5 Utilization every 4s during lockup:
[ 1172.821803] #1: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1172.821841] #2: 100% system, 0% softirq, 1% hardirq, 0% idle
[ 1172.821879] #3: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1172.821919] #4: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1172.821959] #5: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1172.821998] Modules linked in: bpf_testmod(OE) bonding tls rfkill virtio_net net_failover vmx_crypto failover virtio_balloon crct10dif_vpmsum fuse loop nfnetlink zram vsock_loopback vmw_vsock_virtio_transport_common vsock virtio_blk crc32c_vpmsum virtio_console
[ 1172.822141] CPU: 5 UID: 0 PID: 47428 Comm: kworker/u34:3 Tainted: G OEL 6.12.0-rc4+ #1
[ 1172.822197] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP
[ 1172.822235] Hardware name: IBM pSeries (emulated by qemu) POWER8E (raw) 0x4b0201 of:SLOF,HEAD hv:linux,kvm pSeries
[ 1172.822295] Workqueue: events_unbound bpf_map_free_deferred
[ 1172.822331] NIP: c000000000551d60 LR: c000000000551c30 CTR: c0000000004733b0
[ 1172.822378] REGS: c0000000234dfa20 TRAP: 0900 Tainted: G OEL (6.12.0-rc4+)
[ 1172.822424] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 24082808 XER: 00000000
[ 1172.822487] CFAR: 0000000000000000 IRQMASK: 0
[ 1172.822487] GPR00: c000000000551c30 c0000000234dfcc0 c00000000214f800 0000000000000000
[ 1172.822487] GPR04: 0000000000000026 c00c000000163708 0000000000000000 0000000000000000
[ 1172.822487] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000024082808
[ 1172.822487] GPR12: c0000000004733b0 c00000003ffdb080 c0000000043ca7b0 c0000000043b0828
[ 1172.822487] GPR16: c008000205f58000 0000000000000000 0000000000000001 0000000000000000
[ 1172.822487] GPR20: fffffffffffffe7f c008000205f57fff c000000003cbe780 c000000001b26120
[ 1172.822487] GPR24: c000000003da0380 ff7fffffffffefbf c000000003cbe780 0000000000000001
[ 1172.822487] GPR28: c008000106000000 0000000000000000 c0000000004733b0 c0139fb865828000
[ 1172.822883] NIP [c000000000551d60] __apply_to_page_range+0x580/0xea0
[ 1172.822925] LR [c000000000551c30] __apply_to_page_range+0x450/0xea0
[ 1172.822966] Call Trace:
[ 1172.822982] [c0000000234dfcc0] [c000000000551c30] __apply_to_page_range+0x450/0xea0 (unreliable)
[ 1172.823039] [c0000000234dfde0] [c000000000473360] arena_map_free+0x70/0xc0
[ 1172.823081] [c0000000234dfe10] [c0000000003ee324] bpf_map_free_deferred+0x94/0x110
[ 1172.823129] [c0000000234dfe40] [c00000000019bf8c] process_one_work+0x1fc/0x520
[ 1172.823178] [c0000000234dfef0] [c00000000019d96c] worker_thread+0x33c/0x4f0
[ 1172.823219] [c0000000234dff90] [c0000000001aa7a4] kthread+0x134/0x140
[ 1172.823260] [c0000000234dffe0] [c00000000000dd58] start_kernel_thread+0x14/0x18
[ 1172.823308] Code: 79070022 5506c03e 54e9c03e 5106421e 50e9421e 5106463e 714a0040 50e9463e 78c9000e 4082ffbc 7d29c839 4082feb8 <3fff0001> 7c3fe040 4082ffbc 4bfffedc
[ 1184.801263] watchdog: BUG: soft lockup - CPU#4 stuck for 79s! [kworker/u37:7:34302]
[ 1184.804034] CPU#4 Utilization every 4s during lockup:
[ 1184.804111] #1: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1184.804150] #2: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1184.804188] #3: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1184.804226] #4: 101% system, 0% softirq, 1% hardirq, 0% idle
[ 1184.804263] #5: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1184.804312] Modules linked in: bpf_testmod(OE) bonding tls rfkill virtio_net net_failover vmx_crypto failover virtio_balloon crct10dif_vpmsum fuse loop nfnetlink zram vsock_loopback vmw_vsock_virtio_transport_common vsock virtio_blk crc32c_vpmsum virtio_console
[ 1184.804525] CPU: 4 UID: 0 PID: 34302 Comm: kworker/u37:7 Tainted: G OEL 6.12.0-rc4+ #1
[ 1184.804584] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP
[ 1184.804626] Hardware name: IBM pSeries (emulated by qemu) POWER8E (raw) 0x4b0201 of:SLOF,HEAD hv:linux,kvm pSeries
[ 1184.804731] Workqueue: events_unbound bpf_map_free_deferred
[ 1184.804811] NIP: c000000000551d50 LR: c000000000551c30 CTR: c0000000004733b0
[ 1184.804856] REGS: c00000011401ba20 TRAP: 0900 Tainted: G OEL (6.12.0-rc4+)
[ 1184.804899] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 24082282 XER: 00000000
[ 1184.804988] CFAR: 0000000000000000 IRQMASK: 0
[ 1184.804988] GPR00: c000000000551c30 c00000011401bcc0 c00000000214f800 0000000000000000
[ 1184.804988] GPR04: 0000000000000124 c00c00000044e408 0000000000000000 0000000000000000
[ 1184.804988] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000024082282
[ 1184.804988] GPR12: c0000000004733b0 c00000003fffc480 c0000000043c87a0 c0000000043b0028
[ 1184.804988] GPR16: c008000105f38000 0000000000000000 0000000000000001 0000000000000000
[ 1184.804988] GPR20: fffffffffffffe7f c008000105f37fff c000000003cbe780 c000000001b26120
[ 1184.804988] GPR24: c000000003da0380 ff7fffffffffefbf c000000003cbe780 0000000000000001
[ 1184.804988] GPR28: c008000006000000 0000000000000000 c0000000004733b0 c015882289c58000
[ 1184.805363] NIP [c000000000551d50] __apply_to_page_range+0x570/0xea0
[ 1184.805410] LR [c000000000551c30] __apply_to_page_range+0x450/0xea0
[ 1184.805448] Call Trace:
[ 1184.805464] [c00000011401bcc0] [c000000000551c30] __apply_to_page_range+0x450/0xea0 (unreliable)
[ 1184.805525] [c00000011401bde0] [c000000000473360] arena_map_free+0x70/0xc0
[ 1184.805564] [c00000011401be10] [c0000000003ee324] bpf_map_free_deferred+0x94/0x110
[ 1184.805610] [c00000011401be40] [c00000000019bf8c] process_one_work+0x1fc/0x520
[ 1184.805662] [c00000011401bef0] [c00000000019d96c] worker_thread+0x33c/0x4f0
[ 1184.805707] [c00000011401bf90] [c0000000001aa7a4] kthread+0x134/0x140
[ 1184.805745] [c00000011401bfe0] [c00000000000dd58] start_kernel_thread+0x14/0x18
[ 1184.805791] Code: 4182ff28 e9370000 e90e0000 81490020 79070022 5506c03e 54e9c03e 5106421e 50e9421e 5106463e 714a0040 50e9463e <78c9000e> 4082ffbc 7d29c839 4082feb8
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [BUG] Soft lockup on powerpc when running arena selftests
2024-11-07 12:38 [BUG] Soft lockup on powerpc when running arena selftests Viktor Malik
@ 2024-11-07 15:46 ` Alexei Starovoitov
2024-11-15 8:30 ` Viktor Malik
0 siblings, 1 reply; 3+ messages in thread
From: Alexei Starovoitov @ 2024-11-07 15:46 UTC (permalink / raw)
To: Viktor Malik
Cc: bpf, Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Martin KaFai Lau, Eduard Zingerman, Song Liu, Yonghong Song,
John Fastabend, KP Singh, Stanislav Fomichev, Hao Luo, Jiri Olsa
On Thu, Nov 7, 2024 at 4:38 AM Viktor Malik <vmalik@redhat.com> wrote:
>
> Hi,
>
> I'm getting soft lockups when running the BPF arena selftests on powerpc
> (ppcle64). The issue is 100% reproducible on the latest bpf-next with
> `./test_progs -t arena`.
>
> A console snippet for one CPU lockup looks like this:
>
> [ 1124.671746] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u34:0:58]
> [ 1124.675554] CPU#1 Utilization every 4s during lockup:
> [ 1124.675584] #1: 100% system, 0% softirq, 0% hardirq, 0% idle
> [ 1124.675621] #2: 101% system, 0% softirq, 0% hardirq, 0% idle
> [ 1124.675659] #3: 100% system, 0% softirq, 0% hardirq, 0% idle
> [ 1124.675696] #4: 100% system, 0% softirq, 0% hardirq, 0% idle
> [ 1124.675733] #5: 101% system, 0% softirq, 0% hardirq, 0% idle
> [ 1124.675770] Modules linked in: bpf_testmod(OE) bonding tls rfkill virtio_net net_failover vmx_crypto failover virtio_balloon crct10dif_vpmsum fuse loop nfnetlink zram vsock_loopback vmw_vsock_virtio_transport_common vsock virtio_blk crc32c_vpmsum virtio_console
> [ 1124.675921] CPU: 1 UID: 0 PID: 58 Comm: kworker/u34:0 Tainted: G OE 6.12.0-rc4+ #1
> [ 1124.675975] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
> [ 1124.676005] Hardware name: IBM pSeries (emulated by qemu) POWER8E (raw) 0x4b0201 of:SLOF,HEAD hv:linux,kvm pSeries
> [ 1124.676063] Workqueue: events_unbound bpf_map_free_deferred
> [ 1124.676101] NIP: c000000000551d3c LR: c000000000551c30 CTR: c0000000004733b0
> [ 1124.676145] REGS: c000000008a37a20 TRAP: 0900 Tainted: G OE (6.12.0-rc4+)
> [ 1124.676189] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 44082828 XER: 00000000
> [ 1124.676251] CFAR: 0000000000000000 IRQMASK: 0
> [ 1124.676251] GPR00: c000000000551c30 c000000008a37cc0 c00000000214f800 0000000000000000
> [ 1124.676251] GPR04: 000000000000003b c00c00000044e3c8 0000000000000000 0000000000000000
> [ 1124.676251] GPR08: 0000000000000000 0000000000000000 0000000058006001 0000000024082828
> [ 1124.676251] GPR12: c0000000004733b0 c00000003ffff480 c0000000043cb7c0 c0000000043b1028
> [ 1124.676251] GPR16: c008000305f78000 0000000000000000 0000000000000001 0000000000000000
> [ 1124.676251] GPR20: fffffffffffffe7f c008000305f77fff c000000003cbe780 c000000001b26120
> [ 1124.676251] GPR24: c000000003da0380 ff7fffffffffefbf c000000003cbe780 0000000000000001
> [ 1124.676251] GPR28: c008000206000000 0000000000000000 c0000000004733b0 c00bf073759e8000
> [ 1124.676627] NIP [c000000000551d3c] __apply_to_page_range+0x55c/0xea0
> [ 1124.676667] LR [c000000000551c30] __apply_to_page_range+0x450/0xea0
> [ 1124.676706] Call Trace:
> [ 1124.676730] [c000000008a37cc0] [c000000000551c30] __apply_to_page_range+0x450/0xea0 (unreliable)
> [ 1124.676784] [c000000008a37de0] [c000000000473360] arena_map_free+0x70/0xc0
Thanks for the report.
I have no idea what's wrong with apply_to_page_range on ppc.
Don't have any ppc to test and no debugging experience there.
Unless ppc experts chime in there only option to ignore or disable.
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [BUG] Soft lockup on powerpc when running arena selftests
2024-11-07 15:46 ` Alexei Starovoitov
@ 2024-11-15 8:30 ` Viktor Malik
0 siblings, 0 replies; 3+ messages in thread
From: Viktor Malik @ 2024-11-15 8:30 UTC (permalink / raw)
To: Alexei Starovoitov
Cc: bpf, Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Martin KaFai Lau, Eduard Zingerman, Song Liu, Yonghong Song,
John Fastabend, KP Singh, Stanislav Fomichev, Hao Luo, Jiri Olsa
On 11/7/24 16:46, Alexei Starovoitov wrote:
> On Thu, Nov 7, 2024 at 4:38 AM Viktor Malik <vmalik@redhat.com> wrote:
>>
>> Hi,
>>
>> I'm getting soft lockups when running the BPF arena selftests on powerpc
>> (ppcle64). The issue is 100% reproducible on the latest bpf-next with
>> `./test_progs -t arena`.
>>
>> A console snippet for one CPU lockup looks like this:
>>
>> [ 1124.671746] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u34:0:58]
>> [ 1124.675554] CPU#1 Utilization every 4s during lockup:
>> [ 1124.675584] #1: 100% system, 0% softirq, 0% hardirq, 0% idle
>> [ 1124.675621] #2: 101% system, 0% softirq, 0% hardirq, 0% idle
>> [ 1124.675659] #3: 100% system, 0% softirq, 0% hardirq, 0% idle
>> [ 1124.675696] #4: 100% system, 0% softirq, 0% hardirq, 0% idle
>> [ 1124.675733] #5: 101% system, 0% softirq, 0% hardirq, 0% idle
>> [ 1124.675770] Modules linked in: bpf_testmod(OE) bonding tls rfkill virtio_net net_failover vmx_crypto failover virtio_balloon crct10dif_vpmsum fuse loop nfnetlink zram vsock_loopback vmw_vsock_virtio_transport_common vsock virtio_blk crc32c_vpmsum virtio_console
>> [ 1124.675921] CPU: 1 UID: 0 PID: 58 Comm: kworker/u34:0 Tainted: G OE 6.12.0-rc4+ #1
>> [ 1124.675975] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
>> [ 1124.676005] Hardware name: IBM pSeries (emulated by qemu) POWER8E (raw) 0x4b0201 of:SLOF,HEAD hv:linux,kvm pSeries
>> [ 1124.676063] Workqueue: events_unbound bpf_map_free_deferred
>> [ 1124.676101] NIP: c000000000551d3c LR: c000000000551c30 CTR: c0000000004733b0
>> [ 1124.676145] REGS: c000000008a37a20 TRAP: 0900 Tainted: G OE (6.12.0-rc4+)
>> [ 1124.676189] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 44082828 XER: 00000000
>> [ 1124.676251] CFAR: 0000000000000000 IRQMASK: 0
>> [ 1124.676251] GPR00: c000000000551c30 c000000008a37cc0 c00000000214f800 0000000000000000
>> [ 1124.676251] GPR04: 000000000000003b c00c00000044e3c8 0000000000000000 0000000000000000
>> [ 1124.676251] GPR08: 0000000000000000 0000000000000000 0000000058006001 0000000024082828
>> [ 1124.676251] GPR12: c0000000004733b0 c00000003ffff480 c0000000043cb7c0 c0000000043b1028
>> [ 1124.676251] GPR16: c008000305f78000 0000000000000000 0000000000000001 0000000000000000
>> [ 1124.676251] GPR20: fffffffffffffe7f c008000305f77fff c000000003cbe780 c000000001b26120
>> [ 1124.676251] GPR24: c000000003da0380 ff7fffffffffefbf c000000003cbe780 0000000000000001
>> [ 1124.676251] GPR28: c008000206000000 0000000000000000 c0000000004733b0 c00bf073759e8000
>> [ 1124.676627] NIP [c000000000551d3c] __apply_to_page_range+0x55c/0xea0
>> [ 1124.676667] LR [c000000000551c30] __apply_to_page_range+0x450/0xea0
>> [ 1124.676706] Call Trace:
>> [ 1124.676730] [c000000008a37cc0] [c000000000551c30] __apply_to_page_range+0x450/0xea0 (unreliable)
>> [ 1124.676784] [c000000008a37de0] [c000000000473360] arena_map_free+0x70/0xc0
>
> Thanks for the report.
> I have no idea what's wrong with apply_to_page_range on ppc.
> Don't have any ppc to test and no debugging experience there.
> Unless ppc experts chime in there only option to ignore or disable.
>
Thanks.
Disabling sounds better to me as we can still conveniently run
test_progs on ppc. Since some arena tests are quite hard to disable, the
easiest approach is to disable arena allocation on unsupported arches.
I sent the patch [1].
Viktor
[1]
https://lore.kernel.org/bpf/20241115082548.74972-1-vmalik@redhat.com/T/#u
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2024-11-15 8:30 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-11-07 12:38 [BUG] Soft lockup on powerpc when running arena selftests Viktor Malik
2024-11-07 15:46 ` Alexei Starovoitov
2024-11-15 8:30 ` Viktor Malik
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).