* [bpf] Observing Kernel Softlock up while running BPF self tests
@ 2026-01-20 9:10 Venkat Rao Bagalkote
2026-01-22 17:01 ` Hari Bathini
2026-01-29 19:47 ` Saket Kumar Bhaskar
0 siblings, 2 replies; 3+ messages in thread
From: Venkat Rao Bagalkote @ 2026-01-20 9:10 UTC (permalink / raw)
To: linuxppc-dev, Hari Bathini, Saket Kumar Bhaskar,
Madhavan Srinivasan
Greetings!!!
IBM CI has reported a kernel softlockup, while running BPF selftests on
PowerPC kernel.
Traces:
[ 1632.509843] audit: type=1334 audit(1769127975.721:164430):
prog-id=82135 op=LOAD
[ 1632.509852] audit: type=1334 audit(1769127975.721:164431):
prog-id=82135 op=UNLOAD
[ 1637.016921] Mode = AA
[ 1660.780274] watchdog: BUG: soft lockup - CPU#8 stuck for 23s!
[rqsl_w/8:51609]
[ 1660.780283] Modules linked in: bpf_test_rqspinlock(OE+) 8021q(E)
garp(E) mrp(E) stp(E) llc(E) vrf(E) tun(E) bpf_testmod(OE) veth(E)
nft_fib_inet(E) nft_fib_ipv4(E) nft_fib_ipv6(E) nft_fib(E)
nft_reject_inet(E) nf_reject_ipv4(E) nf_reject_ipv6(E) nft_reject(E)
bonding(E) nft_ct(E) tls(E) nft_chain_nat(E) rfkill(E) sunrpc(E)
ibmveth(E) hvcs(E) hvcserver(E) pseries_rng(E) vmx_crypto(E)
dm_multipath(E) fuse(E) dm_mod(E) drm(E) drm_panel_orientation_quirks(E)
zram(E) ext4(E) crc16(E) mbcache(E) jbd2(E) sr_mod(E) sd_mod(E) cdrom(E)
ibmvscsi(E) scsi_transport_srp(E) [last unloaded: livepatch_sample(EK)]
[ 1660.780352] CPU: 8 UID: 0 PID: 51609 Comm: rqsl_w/8 Tainted: G
OE K 6.19.0-rc4-g960c1fd29055 #1 VOLUNTARY
[ 1660.780359] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [K]=LIVEPATCH
[ 1660.780362] Hardware name: IBM,8375-42A POWER9 (architected) 0x4e0202
0xf000005 of:IBM,FW950.80 (VL950_131) hv:phyp pSeries
[ 1660.780365] NIP: c0000000000399a8 LR: c000000000039c24 CTR:
c000000000039ca0
[ 1660.780368] REGS: c000000bc19cfd28 TRAP: 0900 Tainted: G OE K
(6.19.0-rc4-g960c1fd29055)
[ 1660.780372] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE>
CR: 28000288 XER: 0000000a
[ 1660.780386] CFAR: 0000000000000000 IRQMASK: 0
[ 1660.780386] GPR00: c000000000039c24 c000000bc19cfd00 c000000001f58100
c000000bc19cfcf8
[ 1660.780386] GPR04: c000000bc19cfea8 0000000000000000 4000000000000002
c0000013ff916e08
[ 1660.780386] GPR08: 00000013fd6c0000 0000000000000049 fffffffffffffffc
c0080000f73e0f98
[ 1660.780386] GPR12: c000000000039ca0 c00000002e9b6700 c000000000270808
c000000bad8fe980
[ 1660.780386] GPR16: 0000000000000000 0000000000000000 0000000000000000
0000000000000000
[ 1660.780386] GPR20: 0000000000000000 0000000000000000 0000000000000000
0000000000000000
[ 1660.780386] GPR24: 0000000000000000 0000000000000000 c0080000f7360048
0000000000000000
[ 1660.780386] GPR28: 0000000000000008 0000000000000000 000001827223b155
0000000000000003
[ 1660.780433] NIP [c0000000000399a8] __replay_soft_interrupts+0x38/0x150
[ 1660.780443] LR [c000000000039c24]
arch_local_irq_restore.part.0+0xe4/0x160
[ 1660.780449] Call Trace:
[ 1660.780452] [c000000bc19cfd00] [c000000000039a0c]
__replay_soft_interrupts+0x9c/0x150 (unreliable)
[ 1660.780460] [c000000bc19cfeb0] [c000000000039c24]
arch_local_irq_restore.part.0+0xe4/0x160
[ 1660.780468] [c000000bc19cfef0] [c0080000f73e042c]
rqspinlock_worker_fn+0x244/0x300 [bpf_test_rqspinlock]
[ 1660.780476] [c000000bc19cff90] [c000000000270954] kthread+0x154/0x170
[ 1660.780482] [c000000bc19cffe0] [c00000000000df78]
start_kernel_thread+0x14/0x18
[ 1660.780487] Code: 60000000 7c0802a6 f8010010 f821fe51 e92d0c78
f92101a8 39200000 38610028 892d0933 61290040 992d0933 48102e15
<60000000> 39200000 e9410130 f9210160
If you happen to fix this, Please add below tag.
Reported-by: Venkat Rao Bagalkote <venkat88@linux.ibm.com>
Regards,
Venkat.
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [bpf] Observing Kernel Softlock up while running BPF self tests
2026-01-20 9:10 [bpf] Observing Kernel Softlock up while running BPF self tests Venkat Rao Bagalkote
@ 2026-01-22 17:01 ` Hari Bathini
2026-01-29 19:47 ` Saket Kumar Bhaskar
1 sibling, 0 replies; 3+ messages in thread
From: Hari Bathini @ 2026-01-22 17:01 UTC (permalink / raw)
To: Venkat Rao Bagalkote, linuxppc-dev, Saket Kumar Bhaskar,
Madhavan Srinivasan
Hi Venkat,
Thanks for reporting..
On 20/01/26 2:40 pm, Venkat Rao Bagalkote wrote:
> Greetings!!!
>
>
> IBM CI has reported a kernel softlockup, while running BPF selftests on
> PowerPC kernel.
>
>
> Traces:
>
> [ 1632.509843] audit: type=1334 audit(1769127975.721:164430): prog-
> id=82135 op=LOAD
> [ 1632.509852] audit: type=1334 audit(1769127975.721:164431): prog-
> id=82135 op=UNLOAD
> [ 1637.016921] Mode = AA
> [ 1660.780274] watchdog: BUG: soft lockup - CPU#8 stuck for 23s!
> [rqsl_w/8:51609]
> [ 1660.780283] Modules linked in: bpf_test_rqspinlock(OE+) 8021q(E)
> garp(E) mrp(E) stp(E) llc(E) vrf(E) tun(E) bpf_testmod(OE) veth(E)
> nft_fib_inet(E) nft_fib_ipv4(E) nft_fib_ipv6(E) nft_fib(E)
> nft_reject_inet(E) nf_reject_ipv4(E) nf_reject_ipv6(E) nft_reject(E)
> bonding(E) nft_ct(E) tls(E) nft_chain_nat(E) rfkill(E) sunrpc(E)
> ibmveth(E) hvcs(E) hvcserver(E) pseries_rng(E) vmx_crypto(E)
> dm_multipath(E) fuse(E) dm_mod(E) drm(E) drm_panel_orientation_quirks(E)
> zram(E) ext4(E) crc16(E) mbcache(E) jbd2(E) sr_mod(E) sd_mod(E) cdrom(E)
> ibmvscsi(E) scsi_transport_srp(E) [last unloaded: livepatch_sample(EK)]
> [ 1660.780352] CPU: 8 UID: 0 PID: 51609 Comm: rqsl_w/8 Tainted: G OE
> K 6.19.0-rc4-g960c1fd29055 #1 VOLUNTARY
> [ 1660.780359] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [K]=LIVEPATCH
> [ 1660.780362] Hardware name: IBM,8375-42A POWER9 (architected) 0x4e0202
> 0xf000005 of:IBM,FW950.80 (VL950_131) hv:phyp pSeries
> [ 1660.780365] NIP: c0000000000399a8 LR: c000000000039c24 CTR:
> c000000000039ca0
> [ 1660.780368] REGS: c000000bc19cfd28 TRAP: 0900 Tainted: G OE K
> (6.19.0-rc4-g960c1fd29055)
> [ 1660.780372] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE>
> CR: 28000288 XER: 0000000a
> [ 1660.780386] CFAR: 0000000000000000 IRQMASK: 0
> [ 1660.780386] GPR00: c000000000039c24 c000000bc19cfd00 c000000001f58100
> c000000bc19cfcf8
> [ 1660.780386] GPR04: c000000bc19cfea8 0000000000000000 4000000000000002
> c0000013ff916e08
> [ 1660.780386] GPR08: 00000013fd6c0000 0000000000000049 fffffffffffffffc
> c0080000f73e0f98
> [ 1660.780386] GPR12: c000000000039ca0 c00000002e9b6700 c000000000270808
> c000000bad8fe980
> [ 1660.780386] GPR16: 0000000000000000 0000000000000000 0000000000000000
> 0000000000000000
> [ 1660.780386] GPR20: 0000000000000000 0000000000000000 0000000000000000
> 0000000000000000
> [ 1660.780386] GPR24: 0000000000000000 0000000000000000 c0080000f7360048
> 0000000000000000
> [ 1660.780386] GPR28: 0000000000000008 0000000000000000 000001827223b155
> 0000000000000003
> [ 1660.780433] NIP [c0000000000399a8] __replay_soft_interrupts+0x38/0x150
> [ 1660.780443] LR [c000000000039c24]
> arch_local_irq_restore.part.0+0xe4/0x160
> [ 1660.780449] Call Trace:
> [ 1660.780452] [c000000bc19cfd00] [c000000000039a0c]
> __replay_soft_interrupts+0x9c/0x150 (unreliable)
> [ 1660.780460] [c000000bc19cfeb0] [c000000000039c24]
> arch_local_irq_restore.part.0+0xe4/0x160
> [ 1660.780468] [c000000bc19cfef0] [c0080000f73e042c]
> rqspinlock_worker_fn+0x244/0x300 [bpf_test_rqspinlock]
> [ 1660.780476] [c000000bc19cff90] [c000000000270954] kthread+0x154/0x170
> [ 1660.780482] [c000000bc19cffe0] [c00000000000df78]
> start_kernel_thread+0x14/0x18
> [ 1660.780487] Code: 60000000 7c0802a6 f8010010 f821fe51 e92d0c78
> f92101a8 39200000 38610028 892d0933 61290040 992d0933 48102e15
> <60000000> 39200000 e9410130 f9210160
>
Does the system recover after this or is it unresponsive?
- Hari
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [bpf] Observing Kernel Softlock up while running BPF self tests
2026-01-20 9:10 [bpf] Observing Kernel Softlock up while running BPF self tests Venkat Rao Bagalkote
2026-01-22 17:01 ` Hari Bathini
@ 2026-01-29 19:47 ` Saket Kumar Bhaskar
1 sibling, 0 replies; 3+ messages in thread
From: Saket Kumar Bhaskar @ 2026-01-29 19:47 UTC (permalink / raw)
To: Venkat Rao Bagalkote; +Cc: linuxppc-dev, Hari Bathini, Madhavan Srinivasan
On Tue, Jan 20, 2026 at 02:40:44PM +0530, Venkat Rao Bagalkote wrote:
> Greetings!!!
>
>
> IBM CI has reported a kernel softlockup, while running BPF selftests on
> PowerPC kernel.
>
>
> Traces:
>
> [ 1632.509843] audit: type=1334 audit(1769127975.721:164430): prog-id=82135
> op=LOAD
> [ 1632.509852] audit: type=1334 audit(1769127975.721:164431): prog-id=82135
> op=UNLOAD
> [ 1637.016921] Mode = AA
> [ 1660.780274] watchdog: BUG: soft lockup - CPU#8 stuck for 23s!
> [rqsl_w/8:51609]
> [ 1660.780283] Modules linked in: bpf_test_rqspinlock(OE+) 8021q(E) garp(E)
> mrp(E) stp(E) llc(E) vrf(E) tun(E) bpf_testmod(OE) veth(E) nft_fib_inet(E)
> nft_fib_ipv4(E) nft_fib_ipv6(E) nft_fib(E) nft_reject_inet(E)
> nf_reject_ipv4(E) nf_reject_ipv6(E) nft_reject(E) bonding(E) nft_ct(E)
> tls(E) nft_chain_nat(E) rfkill(E) sunrpc(E) ibmveth(E) hvcs(E) hvcserver(E)
> pseries_rng(E) vmx_crypto(E) dm_multipath(E) fuse(E) dm_mod(E) drm(E)
> drm_panel_orientation_quirks(E) zram(E) ext4(E) crc16(E) mbcache(E) jbd2(E)
> sr_mod(E) sd_mod(E) cdrom(E) ibmvscsi(E) scsi_transport_srp(E) [last
> unloaded: livepatch_sample(EK)]
> [ 1660.780352] CPU: 8 UID: 0 PID: 51609 Comm: rqsl_w/8 Tainted: G
> OE K 6.19.0-rc4-g960c1fd29055 #1 VOLUNTARY
> [ 1660.780359] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [K]=LIVEPATCH
> [ 1660.780362] Hardware name: IBM,8375-42A POWER9 (architected) 0x4e0202
> 0xf000005 of:IBM,FW950.80 (VL950_131) hv:phyp pSeries
> [ 1660.780365] NIP: c0000000000399a8 LR: c000000000039c24 CTR:
> c000000000039ca0
> [ 1660.780368] REGS: c000000bc19cfd28 TRAP: 0900 Tainted: G OE K
> (6.19.0-rc4-g960c1fd29055)
> [ 1660.780372] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR:
> 28000288 XER: 0000000a
> [ 1660.780386] CFAR: 0000000000000000 IRQMASK: 0
> [ 1660.780386] GPR00: c000000000039c24 c000000bc19cfd00 c000000001f58100
> c000000bc19cfcf8
> [ 1660.780386] GPR04: c000000bc19cfea8 0000000000000000 4000000000000002
> c0000013ff916e08
> [ 1660.780386] GPR08: 00000013fd6c0000 0000000000000049 fffffffffffffffc
> c0080000f73e0f98
> [ 1660.780386] GPR12: c000000000039ca0 c00000002e9b6700 c000000000270808
> c000000bad8fe980
> [ 1660.780386] GPR16: 0000000000000000 0000000000000000 0000000000000000
> 0000000000000000
> [ 1660.780386] GPR20: 0000000000000000 0000000000000000 0000000000000000
> 0000000000000000
> [ 1660.780386] GPR24: 0000000000000000 0000000000000000 c0080000f7360048
> 0000000000000000
> [ 1660.780386] GPR28: 0000000000000008 0000000000000000 000001827223b155
> 0000000000000003
> [ 1660.780433] NIP [c0000000000399a8] __replay_soft_interrupts+0x38/0x150
> [ 1660.780443] LR [c000000000039c24]
> arch_local_irq_restore.part.0+0xe4/0x160
> [ 1660.780449] Call Trace:
> [ 1660.780452] [c000000bc19cfd00] [c000000000039a0c]
> __replay_soft_interrupts+0x9c/0x150 (unreliable)
> [ 1660.780460] [c000000bc19cfeb0] [c000000000039c24]
> arch_local_irq_restore.part.0+0xe4/0x160
> [ 1660.780468] [c000000bc19cfef0] [c0080000f73e042c]
> rqspinlock_worker_fn+0x244/0x300 [bpf_test_rqspinlock]
> [ 1660.780476] [c000000bc19cff90] [c000000000270954] kthread+0x154/0x170
> [ 1660.780482] [c000000bc19cffe0] [c00000000000df78]
> start_kernel_thread+0x14/0x18
> [ 1660.780487] Code: 60000000 7c0802a6 f8010010 f821fe51 e92d0c78 f92101a8
> 39200000 38610028 892d0933 61290040 992d0933 48102e15 <60000000> 39200000
> e9410130 f9210160
>
>
>
> If you happen to fix this, Please add below tag.
>
>
> Reported-by: Venkat Rao Bagalkote <venkat88@linux.ibm.com>
>
>
>
> Regards,
>
> Venkat.
>
>
Hi Venkat,
I tried on bpf-next-6.19.0-rc5+. It is recovering from softlockups after
sometime and the selftest is passing:
./test_progs -t res_spin_lock_stress
WATCHDOG: test case res_spin_lock_stress executes for 10 seconds...
#377 res_spin_lock_stress:OK
Summary: 1/0 PASSED, 0 SKIPPED, 0 FAILED
On powerpc, there is an arch-specific implementation of queued spinlocks,
but the resilient queued spinlock used by BPF currently falls back to a
simpler test-and-set (TAS) lock. Under heavy contention, this can lead
to soft lockups like the one observed above. Once powerpc specific
resilient queued spinlock implementation is added, the issue is expected
to disappear.
Thanks,
Saket
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2026-01-29 19:47 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-01-20 9:10 [bpf] Observing Kernel Softlock up while running BPF self tests Venkat Rao Bagalkote
2026-01-22 17:01 ` Hari Bathini
2026-01-29 19:47 ` Saket Kumar Bhaskar
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox