Re: [BUG?] ppc64le: fentry BPF not triggered after live patch (v6.14)

[Hari Bathini <hbathini@linux.ibm.com>]

On 07/04/25 2:12 pm, Viktor Malik wrote:
> On 3/31/25 15:19, Shung-Hsi Yu wrote:
>> Hi all,
>>
>> On ppc64le (v6.14, kernel config attached), I've observed that fentry
>> BPF programs stop being invoked after the target kernel function is live
>> patched. This occurs regardless of whether the BPF program was attached
>> before or after the live patch. I believe fentry/fprobe on ppc64le is
>> added with [1].
>>
>> Steps to reproduce on ppc64le:
>> - Use bpftrace (v0.10.0+) to attach a BPF program to cmdline_proc_show
>>    with fentry (kfunc is the older name bpftrace used for fentry, used
>>    here for max compatability)
>>
>>      bpftrace -e 'kfunc:cmdline_proc_show { printf("%lld: cmdline_proc_show() called by %s\n", nsecs(), comm) }'
>>
>> - Run `cat /proc/cmdline` and observe bpftrace output
>>
>> - Load samples/livepatch/livepatch-sample.ko
>>
>> - Run `cat /proc/cmdline` again. Observe "this has been live patched" in
>>    output, but no new bpftrace output.
>>
>> Note: once the live patching module is disabled through the sysfs interface
>> the BPF program invocation is restored.
>>
>> Is this the expected interaction between fentry BPF and live patching?
>> On x86_64 it does _not_ happen, so I'd guess the behavior on ppc64le is
>> unintended. Any insights appreciated.
> 
> I'm not sure if this is related but I found out that when a kernel is
> compiled with KASAN=y (full config attached), the above steps without
> the bpftrace part lead to a kernel panic upon running the second `cat
> /proc/cmdline` command (the livepatched one).
> 
> Here's the relevant part of the kdump:
> 
> [  457.405298] BUG: Unable to handle kernel data access on write at 0xc0000000000f9078
> [  457.405320] Faulting instruction address: 0xc0000000018ff958
> [  457.405328] Oops: Kernel access of bad area, sig: 11 [#1]
> [  457.405336] LE PAGE_SIZE=64K MMU=Hash  SMP NR_CPUS=8192 NUMA pSeries
> [  457.405347] Modules linked in: livepatch_sample(K) bonding tls rfkill vmx_crypto ibmveth pseries_rng sg fuse loop nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vsock xfs sd_mod ibmvscsi scsi_transport_srp dm_mirror dm_region_hash dm_log dm_mod
> [  457.405410] CPU: 6 UID: 0 PID: 5141 Comm: cat Kdump: loaded Tainted: G              K     6.14.0+ #9 VOLUNTARY
> [  457.405424] Tainted: [K]=LIVEPATCH
> [  457.405430] Hardware name: IBM,9009-22A POWER9 (architected) 0x4e0202 0xf000005 of:IBM,FW910.00 (VL910_062) hv:phyp pSeries
> [  457.405440] NIP:  c0000000018ff958 LR: c0000000018ff930 CTR: c0000000009c0790
> [  457.405449] REGS: c00000005f2e7790 TRAP: 0300   Tainted: G              K      (6.14.0+)
> [  457.405459] MSR:  8000000000009033 <SF,EE,ME,IR,DR,RI,LE>  CR: 2822880b  XER: 20040000
> [  457.405484] CFAR: c0000000008addc0 DAR: c0000000000f9078 DSISR: 0a000000 IRQMASK: 1
> GPR00: c0000000018f2584 c00000005f2e7a30 c00000000280a900 c000000017ffa488
> GPR04: 0000000000000008 0000000000000000 c0000000018f24fc 000000000000000d
> GPR08: fffffffffffe0000 000000000000000d 0000000000000000 0000000000008000
> GPR12: c0000000009c0790 c000000017ffa480 c00000005f2e7c78 c0000000000f9070
> GPR16: c00000005f2e7c90 0000000000000000 0000000000000000 0000000000000000
> GPR20: 0000000000000000 c00000005f3efa80 c00000005f2e7c60 c00000005f2e7c88
> GPR24: c00000005f2e7c60 0000000000000001 c0000000000f9078 0000000000000000
> GPR28: 00007fff97960000 c000000017ffa480 0000000000000000 c0000000000f9078
> [  457.405605] NIP [c0000000018ff958] _raw_spin_lock_irqsave+0x68/0x110
> [  457.405619] LR [c0000000018ff930] _raw_spin_lock_irqsave+0x40/0x110
> [  457.405630] Call Trace:
> [  457.405635] [c00000005f2e7a30] [c000000000941804] check_heap_object+0x34/0x390 (unreliable)
> [  457.405651] [c00000005f2e7a70] [c0000000018f2584] __mutex_unlock_slowpath.isra.0+0xe4/0x230
> [  457.405665] [c00000005f2e7af0] [c0000000009c2f50] seq_read_iter+0x430/0xa90
> [  457.405679] [c00000005f2e7c00] [c000000000aade04] proc_reg_read_iter+0xa4/0x200
> [  457.405692] [c00000005f2e7c40] [c00000000095345c] vfs_read+0x41c/0x510
> [  457.405705] [c00000005f2e7d30] [c0000000009545d4] ksys_read+0xa4/0x190
> [  457.405716] [c00000005f2e7d90] [c00000000003a3f0] system_call_exception+0x1d0/0x440
> [  457.405729] [c00000005f2e7e50] [c00000000000cedc] system_call_vectored_common+0x15c/0x2ec
> [  457.405744] --- interrupt: 3000 at 0x7fff97e75044
> [  457.405755] NIP:  00007fff97e75044 LR: 00007fff97e75044 CTR: 0000000000000000
> [  457.405764] REGS: c00000005f2e7e80 TRAP: 3000   Tainted: G              K      (6.14.0+)
> [  457.405773] MSR:  800000000280f033 <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE>  CR: 48222804  XER: 00000000
> [  457.405805] IRQMASK: 0
> GPR00: 0000000000000003 00007fffc1908930 00007fff97f87100 0000000000000003
> GPR04: 00007fff97960000 0000000000040000 0000000000000000 00007fff97f80248
> GPR08: 0000000000000002 0000000000000000 0000000000000000 0000000000000000
> GPR12: 0000000000000000 00007fff9805a5a0 0000000000000000 0000000000000000
> GPR16: 0000000000000000 0000000000040000 00007fffc19091c8 0000000000000000
> GPR20: 0000000000000000 0000000000000000 0000000000000000 00007fff9804f470
> GPR24: 0000000000000000 0000000000040000 00007fffc190f1c5 000000007ff00000
> GPR28: 0000000000000003 00007fff97960000 0000000000040000 0000000000000003
> [  457.405916] NIP [00007fff97e75044] 0x7fff97e75044
> [  457.405924] LR [00007fff97e75044] 0x7fff97e75044
> [  457.405932] --- interrupt: 3000
> [  457.405938] Code: 386d0008 4afae43d 60000000 a13d0008 3d00fffe 5529083c 61290001 7d40f829 7d474079 40c20018 7d474038 7ce74b78 <7ce0f92d> 40c2ffe8 7c2004ac 794a03e1
> [  457.405981] ---[ end trace 0000000000000000 ]---
> [  457.419259] pstore: backend (nvram) writing error (-1)
> 
> Interestingly, the panic doesn't occur when the bpftrace process is
> running. Then, running `cat /proc/cmdline` works (even prints the
> expected livepatched message) but doesn't appear in bpftrace output, as
> Shung-Hsi observed.
> 
> On a kernel with KASAN=n, no panic happens.
> 
> This panic doesn't seem to be related to BPF (as it happens when no BPF
> programs are involved) but it involves livepatch and occurs for the same
> sequence of commands, so the two cases may be related. In this case, I
> suspect that the issue is caused by an incorrect interaction of
> livepatch and the ftrace changes introduced for BPF trampolines [1].
> 
> FWIW, there is patch cfec8463d9a1 ("powerpc/ftrace: Fix ftrace bug with
> KASAN=y") which is fixing a bug in [1] appearing on KASAN=y kernel but
> I'm not sure if it's related to this issue.

Thanks for reporting this, Viktor.
There was a bug in how clobbered register was restored in livepatch path
leading this failure. Posted the fix patch upstream [1]

FWIW, the problem Shung-Hsi observed still exists. Will try and get that
working..

- Hari

[1] 
https://lore.kernel.org/linuxppc-dev/20250416191227.201146-1-hbathini@linux.ibm.com/

> 
> Viktor
> 
> [1] https://lore.kernel.org/all/20241030070850.1361304-1-hbathini@linux.ibm.com/
> 
>>
>>
>> Thanks,
>> Shung-Hsi Yu
>>
>> 1: https://lore.kernel.org/all/20241030070850.1361304-2-hbathini@linux.ibm.com/