From: Ilya Leoshkevich <iii@linux.ibm.com>
To: Alexei Starovoitov <alexei.starovoitov@gmail.com>,
Yonghong Song <yonghong.song@linux.dev>
Cc: bpf <bpf@vger.kernel.org>, Alexei Starovoitov <ast@kernel.org>,
Andrii Nakryiko <andrii@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
Kernel Team <kernel-team@fb.com>,
Martin KaFai Lau <martin.lau@kernel.org>
Subject: Re: [PATCH bpf-next 1/2] bpf: Do not include r10 in precision backtracking bookkeeping
Date: Mon, 12 May 2025 14:03:11 +0200 [thread overview]
Message-ID: <bde230fc4444a3d8bb07172042b5392a3f04d1e3.camel@linux.ibm.com> (raw)
In-Reply-To: <CAADnVQLi8dP9uOTcs7qt_9Y42go9NVu4FSEk_eB_=egP3kCraA@mail.gmail.com>
On Sun, 2025-05-11 at 15:33 -0700, Alexei Starovoitov wrote:
> On Sun, May 11, 2025 at 9:28 AM Yonghong Song
> <yonghong.song@linux.dev> wrote:
> >
> > Reported by: Yi Lai <yi1.lai@linux.intel.com>
> > Fixes: 407958a0e980 ("bpf: encapsulate precision backtracking
> > bookkeeping")
> > Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
> > ---
> > kernel/bpf/verifier.c | 6 ++++--
> > 1 file changed, 4 insertions(+), 2 deletions(-)
> >
> > diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> > index 28f5a7899bd6..1cb4d80d15c1 100644
> > --- a/kernel/bpf/verifier.c
> > +++ b/kernel/bpf/verifier.c
> > @@ -4413,8 +4413,10 @@ static int backtrack_insn(struct
> > bpf_verifier_env *env, int idx, int subseq_idx,
> > * before it would be equally necessary to
> > * propagate it to dreg.
> > */
> > - bt_set_reg(bt, dreg);
> > - bt_set_reg(bt, sreg);
> > + if (dreg != BPF_REG_FP)
> > + bt_set_reg(bt, dreg);
> > + if (sreg != BPF_REG_FP)
> > + bt_set_reg(bt, sreg);
>
> The fix makes sense to me.
>
> but it crashes on s390 according to CI:
>
> 2025-05-11T16:48:18.5929491Z #401 struct_ops_refcounted:OK
> 2025-05-11T16:48:18.7330807Z ------------[ cut here ]------------
> 2025-05-11T16:48:18.7333824Z kernel BUG at kernel/bpf/core.c:533!
> 2025-05-11T16:48:18.7335154Z monitor event: 0040 ilc:2 [#1]SMP
> 2025-05-11T16:48:18.7336972Z Modules linked in: bpf_testmod(OE) [last
> unloaded: bpf_test_no_cfi(OE)]
> 2025-05-11T16:48:18.7341000Z CPU: 0 UID: 0 PID: 109 Comm: new_name
> Tainted: G OE 6.15.0-rc4-ga9827e5c6a13-dirty #13 NONE
> 2025-05-11T16:48:18.7343245Z Tainted: [O]=OOT_MODULE,
> [E]=UNSIGNED_MODULE
> 2025-05-11T16:48:18.7344697Z Hardware name: IBM 8561 LT1 400
> (KVM/Linux)
> 2025-05-11T16:48:18.7347056Z Krnl PSW : 0704d00180000000
> 000003320039d8ca (bpf_patch_insn_single+0x29a/0x2a0)
> 2025-05-11T16:48:18.7349372Z R:0 T:1 IO:1 EX:1 Key:0 M:1
> W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3
> 2025-05-11T16:48:18.7351910Z Krnl GPRS: 000002b200000016
> ffffffff7ffffffe ffffffffffffffde 00000000ffffffde
> 2025-05-11T16:48:18.7354602Z 0000000000000003
> 0000000000000005 0000000000000000 000002b2000b5048
> 2025-05-11T16:48:18.7356934Z 0000000000000018
> 000002b2000b5000 0000000000000003 0000000000000002
> 2025-05-11T16:48:18.7359164Z 000003ff81badf98
> 0000000000000002 000003320039d738 000002b200687840
> 2025-05-11T16:48:18.7361217Z Krnl Code: 000003320039d8bc:
> e3005ff0ff50
> sty %r0,-16(%r5)
> 2025-05-11T16:48:18.7363048Z 000003320039d8c2: a7f4ffc6
> brc
> 15,000003320039d84e
> 2025-05-11T16:48:18.7364611Z #000003320039d8c6: af000000 mc
> 0,0
> 2025-05-11T16:48:18.7366106Z >000003320039d8ca: 0707 bcr
> 0,%r7
> 2025-05-11T16:48:18.7367449Z 000003320039d8cc: 0707 bcr
> 0,%r7
> 2025-05-11T16:48:18.7368855Z 000003320039d8ce: 0707 bcr
> 0,%r7
> 2025-05-11T16:48:18.7403748Z 000003320039d8d0:
> c004004bdc60
> brcl 0,0000033200d19190
> 2025-05-11T16:48:18.7407899Z 000003320039d8d6:
> eb6ff0480024
> stmg %r6,%r15,72(%r15)
> 2025-05-11T16:48:18.7410576Z Call Trace:
> 2025-05-11T16:48:18.7411713Z [<000003320039d8ca>]
> bpf_patch_insn_single+0x29a/0x2a0
> 2025-05-11T16:48:18.7413433Z ([<000003320039d738>]
> bpf_patch_insn_single+0x108/0x2a0)
> 2025-05-11T16:48:18.7415210Z [<000003320039eb72>]
> bpf_jit_blind_constants+0xd2/0x1b0
> 2025-05-11T16:48:18.7416879Z [<000003320020b5ee>]
> bpf_int_jit_compile+0x46/0x448
> 2025-05-11T16:48:18.7418417Z [<00000332003c12d4>]
> jit_subprogs+0x594/0xbe0
> 2025-05-11T16:48:18.7419782Z [<00000332003dacc8>]
> bpf_check+0xe28/0x14b0
> 2025-05-11T16:48:18.7421128Z [<00000332003a9328>]
> bpf_prog_load+0x4d8/0xba0
> 2025-05-11T16:48:18.7422570Z [<00000332003ab976>]
> __sys_bpf+0x98e/0xdd0
> 2025-05-11T16:48:18.7423887Z [<00000332003abdfc>]
> __s390x_sys_bpf+0x44/0x50
> 2025-05-11T16:48:18.7425227Z [<0000033200ce61b2>]
> __do_syscall+0x132/0x260
> 2025-05-11T16:48:18.7426522Z [<0000033200cf162c>]
> system_call+0x74/0x98
>
>
> Ilya,
>
> Could you please verify whether the fix is related or not ?
I assume what crashes here is subprogs_and_jit_harden.
I could reproduce this neither using the build artifacts, nor in my
own development setup with this series applied.
subprogs_and_jit_harden is trying to induce a race condition by
constantly toggling bpf_jit_harden. Running it in a loop for a while
does not lead to any failures.
I also cannot see how this can create problems with the existing code
structure. bpf_jit_harden is used by bpf_jit_blinding_enabled() and
bpf_jit_kallsyms_enabled(), each of which has only one call site. When
these two see different values of bpf_jit_harden, nothing bad should
happen.
I suspect that this must an existing intermittently occurring issue.
Can we re-run the CI to see if the BUG() is reproducible in the GHA
environment?
next prev parent reply other threads:[~2025-05-12 12:04 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-05-11 16:27 [PATCH bpf-next 1/2] bpf: Do not include r10 in precision backtracking bookkeeping Yonghong Song
2025-05-11 16:28 ` [PATCH bpf-next 2/2] selftests/bpf: Add a test with r10 in conditional jmp Yonghong Song
2025-05-11 22:33 ` [PATCH bpf-next 1/2] bpf: Do not include r10 in precision backtracking bookkeeping Alexei Starovoitov
2025-05-12 12:03 ` Ilya Leoshkevich [this message]
2025-05-12 16:26 ` Andrii Nakryiko
2025-05-12 22:05 ` Andrii Nakryiko
2025-05-13 5:20 ` Yonghong Song
2025-05-15 1:44 ` Yonghong Song
2025-05-15 17:47 ` Andrii Nakryiko
2025-05-15 19:07 ` Yonghong Song
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=bde230fc4444a3d8bb07172042b5392a3f04d1e3.camel@linux.ibm.com \
--to=iii@linux.ibm.com \
--cc=alexei.starovoitov@gmail.com \
--cc=andrii@kernel.org \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=kernel-team@fb.com \
--cc=martin.lau@kernel.org \
--cc=yonghong.song@linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).