[PATCH bpf-next v2] bpf: Fix unaligned interpreter panic on JIT fallback path

[PATCH bpf-next v2] bpf: Fix unaligned interpreter panic on JIT fallback path

[Tiezhu Yang]

When an architecture implements bpf_jit_inlines_helper_call(), such
as LoongArch, ARM64, and RISC-V, the BPF verifier skips rewriting
the helper call offset (insn->imm) during the bpf_do_misc_fixups()
phase if the helper is expected to be inlined by the JIT compiler.
As a result, insn->imm remains as the raw helper enum ID.

However, if JIT is disabled at runtime (net.core.bpf_jit_enable=0)
or if the JIT compilation later dynamically fails (e.g., due to OOM
during bpf_jit_alloc_exec()), the core BPF subsystem falls back to
the BPF interpreter.

When the fallback interpreter executes (__bpf_call_base + insn->imm)
with the unpatched raw helper ID, it jumps into an unaligned invalid
address space, triggering a fatal instruction alignment fault (ADEF)
or illegal memory access kernel panic.

This issue impacts all architectures that support helper inlining.
Introduce a late fixup pass via bpf_fixup_fallback_inline_helpers()
in __bpf_prog_select_runtime() to fix it.

When JIT compilation fails or is disabled, the helper call offsets
originally skipped for inlining are rewritten to relative memory
offsets right before transferring control to the interpreter.

A bounds check on insn->imm against __BPF_FUNC_MAX_ID is added to
ensure already-patched address offsets of regular non-inlined helpers
are safely filtered out during traversal.

1. Test case (test_panic.c):

  #include <linux/bpf.h>
  #include <bpf/bpf_helpers.h>

  SEC("kprobe/sys_getpid")
  int test_panic(void *ctx)
  {
  	struct task_struct *task;

  	task = (struct task_struct *)bpf_get_current_task();
  	if (task)
  		bpf_printk("Task address: %p\n", task);

  	return 0;
  }

  char LICENSE[] SEC("license") = "GPL";

2. Reproduction steps:

  $ clang -target bpf -O2 -g -c test_panic.c -o test_panic.o
  $ sudo sysctl -w net.core.bpf_jit_enable=0
  $ sudo bpftool prog load test_panic.o /sys/fs/bpf/test_panic autoattach
  $ sudo cat /sys/kernel/debug/tracing/trace_pipe

3. Panic information on LoongArch:

  Kernel ade access[#1]:
  ...
      ra: 9000000000486e50 ___bpf_prog_run+0x1370/0x36b0
     ERA: 9000000000485383 __bpf_prog_ret0_warn+0x13/0x20
  ...
   ESTAT: 00080000 [ADEF] (IS= ECode=8 EsubCode=0)

Fixes: 2ddec2c80b44 ("riscv, bpf: inline bpf_get_smp_processor_id()")
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
---
 kernel/bpf/core.c | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 649cce41e13f..5785c8cb4dbe 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -2608,6 +2608,31 @@ static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, struc
 	return prog;
 }
 
+/* Fix up helper call offsets for inlined helpers on JIT fallback path. */
+static void bpf_fixup_fallback_inline_helpers(struct bpf_verifier_env *env, struct bpf_prog *fp)
+{
+	struct bpf_insn *insn = fp->insnsi;
+	const struct bpf_func_proto *fn;
+	int i;
+
+	if (!env || !env->ops->get_func_proto)
+		return;
+
+	for (i = 0; i < fp->len; i++, insn++) {
+		if (insn->code == (BPF_JMP | BPF_CALL) && insn->src_reg == 0) {
+			/* Filter out already-patched address offsets. */
+			if (insn->imm >= __BPF_FUNC_MAX_ID)
+				continue;
+
+			if (bpf_jit_inlines_helper_call(insn->imm)) {
+				fn = env->ops->get_func_proto(insn->imm, env->prog);
+				if (fn && fn->func)
+					insn->imm = fn->func - __bpf_call_base;
+			}
+		}
+	}
+}
+
 struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct bpf_prog *fp,
 					   int *err)
 {
@@ -2643,6 +2668,9 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct
 			*err = -ENOTSUPP;
 			return fp;
 		}
+
+		if (!fp->jited)
+			bpf_fixup_fallback_inline_helpers(env, fp);
 	} else {
 		*err = bpf_prog_offload_compile(fp);
 		if (*err)
-- 
2.42.0

Re: [PATCH bpf-next v2] bpf: Fix unaligned interpreter panic on JIT fallback path

[Leon Hwang]

On 11/6/26 18:12, Tiezhu Yang wrote:
[...]
> +/* Fix up helper call offsets for inlined helpers on JIT fallback path. */
> +static void bpf_fixup_fallback_inline_helpers(struct bpf_verifier_env *env, struct bpf_prog *fp)
> +{
> +	struct bpf_insn *insn = fp->insnsi;
> +	const struct bpf_func_proto *fn;
> +	int i;
> +
> +	if (!env || !env->ops->get_func_proto)
> +		return;
> +
> +	for (i = 0; i < fp->len; i++, insn++) {
> +		if (insn->code == (BPF_JMP | BPF_CALL) && insn->src_reg == 0) {

NIT: better to use bpf_helper_call() here.

Other than that, lgtm:

Acked-by: Leon Hwang <leon.hwang@linux.dev>

> [...]

Re: [PATCH bpf-next v2] bpf: Fix unaligned interpreter panic on JIT fallback path

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 1603 bytes --]

> diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
> index 649cce41e13f..5785c8cb4dbe 100644
> --- a/kernel/bpf/core.c
> +++ b/kernel/bpf/core.c
> @@ -2608,6 +2608,31 @@ static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, struc
>  	return prog;
>  }
>
> +/* Fix up helper call offsets for inlined helpers on JIT fallback path. */
> +static void bpf_fixup_fallback_inline_helpers(struct bpf_verifier_env *env, struct bpf_prog *fp)
> +{
> +	struct bpf_insn *insn = fp->insnsi;
> +	const struct bpf_func_proto *fn;
> +	int i;
> +
> +	if (!env || !env->ops->get_func_proto)
> +		return;
> +
> +	for (i = 0; i < fp->len; i++, insn++) {
> +		if (insn->code == (BPF_JMP | BPF_CALL) && insn->src_reg == 0) {

Would it be cleaner to use bpf_helper_call(insn) here instead of the
manual condition check?

The bpf_helper_call() helper from include/linux/bpf_verifier.h performs
the same check and is used throughout the codebase for consistency. It
improves readability and makes the intent more explicit.

> +			/* Filter out already-patched address offsets. */
> +			if (insn->imm >= __BPF_FUNC_MAX_ID)
> +				continue;
> +
> +			if (bpf_jit_inlines_helper_call(insn->imm)) {
> +				fn = env->ops->get_func_proto(insn->imm, env->prog);
> +				if (fn && fn->func)
> +					insn->imm = fn->func - __bpf_call_base;
> +			}
> +		}
> +	}
> +}

[ ... ]


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/27340034854

Re: [PATCH bpf-next v2] bpf: Fix unaligned interpreter panic on JIT fallback path

[Tiezhu Yang]

On 2026/6/11 下午6:54, bot+bpf-ci@kernel.org wrote:
>> diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
>> index 649cce41e13f..5785c8cb4dbe 100644
>> --- a/kernel/bpf/core.c
>> +++ b/kernel/bpf/core.c
>> @@ -2608,6 +2608,31 @@ static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, struc
>>   	return prog;
>>   }
>>
>> +/* Fix up helper call offsets for inlined helpers on JIT fallback path. */
>> +static void bpf_fixup_fallback_inline_helpers(struct bpf_verifier_env *env, struct bpf_prog *fp)
>> +{
>> +	struct bpf_insn *insn = fp->insnsi;
>> +	const struct bpf_func_proto *fn;
>> +	int i;
>> +
>> +	if (!env || !env->ops->get_func_proto)
>> +		return;
>> +
>> +	for (i = 0; i < fp->len; i++, insn++) {
>> +		if (insn->code == (BPF_JMP | BPF_CALL) && insn->src_reg == 0) {
> 
> Would it be cleaner to use bpf_helper_call(insn) here instead of the
> manual condition check?
> 
> The bpf_helper_call() helper from include/linux/bpf_verifier.h performs
> the same check and is used throughout the codebase for consistency. It
> improves readability and makes the intent more explicit.

OK, this is better.

I will send v3 after waiting for more review comments.

Thanks,
Tiezhu

Re: [PATCH bpf-next v2] bpf: Fix unaligned interpreter panic on JIT fallback path

[Xu Kuohai]

On 6/11/2026 6:12 PM, Tiezhu Yang wrote:
> When an architecture implements bpf_jit_inlines_helper_call(), such
> as LoongArch, ARM64, and RISC-V, the BPF verifier skips rewriting
> the helper call offset (insn->imm) during the bpf_do_misc_fixups()
> phase if the helper is expected to be inlined by the JIT compiler.
> As a result, insn->imm remains as the raw helper enum ID.
> 
> However, if JIT is disabled at runtime (net.core.bpf_jit_enable=0)
> or if the JIT compilation later dynamically fails (e.g., due to OOM
> during bpf_jit_alloc_exec()), the core BPF subsystem falls back to
> the BPF interpreter.
> 
> When the fallback interpreter executes (__bpf_call_base + insn->imm)
> with the unpatched raw helper ID, it jumps into an unaligned invalid
> address space, triggering a fatal instruction alignment fault (ADEF)
> or illegal memory access kernel panic.
> 
> This issue impacts all architectures that support helper inlining.
> Introduce a late fixup pass via bpf_fixup_fallback_inline_helpers()
> in __bpf_prog_select_runtime() to fix it.
> 
> When JIT compilation fails or is disabled, the helper call offsets
> originally skipped for inlining are rewritten to relative memory
> offsets right before transferring control to the interpreter.
> 
> A bounds check on insn->imm against __BPF_FUNC_MAX_ID is added to
> ensure already-patched address offsets of regular non-inlined helpers
> are safely filtered out during traversal.
> 
> 1. Test case (test_panic.c):
> 
>    #include <linux/bpf.h>
>    #include <bpf/bpf_helpers.h>
> 
>    SEC("kprobe/sys_getpid")
>    int test_panic(void *ctx)
>    {
>    	struct task_struct *task;
> 
>    	task = (struct task_struct *)bpf_get_current_task();
>    	if (task)
>    		bpf_printk("Task address: %p\n", task);
> 
>    	return 0;
>    }
> 
>    char LICENSE[] SEC("license") = "GPL";
> 
> 2. Reproduction steps:
> 
>    $ clang -target bpf -O2 -g -c test_panic.c -o test_panic.o
>    $ sudo sysctl -w net.core.bpf_jit_enable=0
>    $ sudo bpftool prog load test_panic.o /sys/fs/bpf/test_panic autoattach
>    $ sudo cat /sys/kernel/debug/tracing/trace_pipe
> 
> 3. Panic information on LoongArch:
> 
>    Kernel ade access[#1]:
>    ...
>        ra: 9000000000486e50 ___bpf_prog_run+0x1370/0x36b0
>       ERA: 9000000000485383 __bpf_prog_ret0_warn+0x13/0x20
>    ...
>     ESTAT: 00080000 [ADEF] (IS= ECode=8 EsubCode=0)
> 
> Fixes: 2ddec2c80b44 ("riscv, bpf: inline bpf_get_smp_processor_id()")
> Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
> ---
>   kernel/bpf/core.c | 28 ++++++++++++++++++++++++++++
>   1 file changed, 28 insertions(+)
> 
> diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
> index 649cce41e13f..5785c8cb4dbe 100644
> --- a/kernel/bpf/core.c
> +++ b/kernel/bpf/core.c
> @@ -2608,6 +2608,31 @@ static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, struc
>   	return prog;
>   }
>   
> +/* Fix up helper call offsets for inlined helpers on JIT fallback path. */
> +static void bpf_fixup_fallback_inline_helpers(struct bpf_verifier_env *env, struct bpf_prog *fp)

The function name is a bit long for a static function

> +{
> +	struct bpf_insn *insn = fp->insnsi;
> +	const struct bpf_func_proto *fn;
> +	int i;
> +
> +	if (!env || !env->ops->get_func_proto)
> +		return;
> +
> +	for (i = 0; i < fp->len; i++, insn++) {
> +		if (insn->code == (BPF_JMP | BPF_CALL) && insn->src_reg == 0) {
> +			/* Filter out already-patched address offsets. */
> +			if (insn->imm >= __BPF_FUNC_MAX_ID)

The check is redundant, since bpf_jit_inlines_helper_call() already
filters out insn->imm numbers that is not a valid helper id.

> +				continue;
> +
> +			if (bpf_jit_inlines_helper_call(insn->imm)) {
> +				fn = env->ops->get_func_proto(insn->imm, env->prog);
> +				if (fn && fn->func)
> +					insn->imm = fn->func - __bpf_call_base;
> +			}
> +		}
> +	}
> +}
> +
>   struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct bpf_prog *fp,
>   					   int *err)
>   {
> @@ -2643,6 +2668,9 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct
>   			*err = -ENOTSUPP;
>   			return fp;
>   		}
> +
> +		if (!fp->jited)
> +			bpf_fixup_fallback_inline_helpers(env, fp);
>   	} else {
>   		*err = bpf_prog_offload_compile(fp);
>   		if (*err)

I tested it on arm64 and the patch works.

Tested-by: Xu Kuohai <xukuohai@huawei.com>

Re: [PATCH bpf-next v2] bpf: Fix unaligned interpreter panic on JIT fallback path

[Tiezhu Yang]

On 2026/6/12 下午8:37, Xu Kuohai wrote:
> On 6/11/2026 6:12 PM, Tiezhu Yang wrote:
>> When an architecture implements bpf_jit_inlines_helper_call(), such
>> as LoongArch, ARM64, and RISC-V, the BPF verifier skips rewriting
>> the helper call offset (insn->imm) during the bpf_do_misc_fixups()
>> phase if the helper is expected to be inlined by the JIT compiler.
>> As a result, insn->imm remains as the raw helper enum ID.

...

>> +static void bpf_fixup_fallback_inline_helpers(struct bpf_verifier_env 
>> *env, struct bpf_prog *fp)
> 
> The function name is a bit long for a static function

If there are no objections, I will shorten the static function
name to bpf_fixup_inline_helpers() in v4.

>> +{
>> +    struct bpf_insn *insn = fp->insnsi;
>> +    const struct bpf_func_proto *fn;
>> +    int i;
>> +
>> +    if (!env || !env->ops->get_func_proto)
>> +        return;
>> +
>> +    for (i = 0; i < fp->len; i++, insn++) {
>> +        if (insn->code == (BPF_JMP | BPF_CALL) && insn->src_reg == 0) {
>> +            /* Filter out already-patched address offsets. */
>> +            if (insn->imm >= __BPF_FUNC_MAX_ID)
> 
> The check is redundant, since bpf_jit_inlines_helper_call() already
> filters out insn->imm numbers that is not a valid helper id.

Yes, I see your point. If insn->imm is an invalid helper ID,
bpf_jit_inlines_helper_call() will hit the default branch and
return false, making the prior check redundant. I will clean
this up.

>> +                continue;
>> +
>> +            if (bpf_jit_inlines_helper_call(insn->imm)) {
>> +                fn = env->ops->get_func_proto(insn->imm, env->prog);
>> +                if (fn && fn->func)
>> +                    insn->imm = fn->func - __bpf_call_base;
>> +            }
>> +        }
>> +    }
>> +}
>> +
>>   struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env 
>> *env, struct bpf_prog *fp,
>>                          int *err)
>>   {
>> @@ -2643,6 +2668,9 @@ struct bpf_prog 
>> *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct
>>               *err = -ENOTSUPP;
>>               return fp;
>>           }
>> +
>> +        if (!fp->jited)
>> +            bpf_fixup_fallback_inline_helpers(env, fp);
>>       } else {
>>           *err = bpf_prog_offload_compile(fp);
>>           if (*err)
> 
> I tested it on arm64 and the patch works.
> 
> Tested-by: Xu Kuohai <xukuohai@huawei.com>

Thanks for your review and the Tested-by tag.

Thanks,
Tiezhu

Re: [PATCH bpf-next v2] bpf: Fix unaligned interpreter panic on JIT fallback path

[Leon Hwang]

On 15/6/26 09:23, Tiezhu Yang wrote:
> On 2026/6/12 下午8:37, Xu Kuohai wrote:
>> On 6/11/2026 6:12 PM, Tiezhu Yang wrote:
>>> When an architecture implements bpf_jit_inlines_helper_call(), such
>>> as LoongArch, ARM64, and RISC-V, the BPF verifier skips rewriting
>>> the helper call offset (insn->imm) during the bpf_do_misc_fixups()
>>> phase if the helper is expected to be inlined by the JIT compiler.
>>> As a result, insn->imm remains as the raw helper enum ID.
>
> ...
>
>>> +static void bpf_fixup_fallback_inline_helpers(struct
>>> bpf_verifier_env *env, struct bpf_prog *fp)
>>
>> The function name is a bit long for a static function
>
> If there are no objections, I will shorten the static function
> name to bpf_fixup_inline_helpers() in v4.
>

I think bpf_fixup_inline_helpers() is not accurate, since it is to fix
those non-inline helpers here. bpf_fixup_non_inline_helpers() is a
candidate. However, a shorter name is preferred.

Please provide changelog btw. For a single patch, you can provide
changelog here.

Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
---
Changelog:
v3 -> v4: xxx
---
 kernel/bpf/core.c | 28 ++++++++++++++++++++++++++++

Thanks,
Leon

Re: [PATCH bpf-next v2] bpf: Fix unaligned interpreter panic on JIT fallback path

[Tiezhu Yang]

On 2026/6/15 上午9:49, Leon Hwang wrote:
> On 15/6/26 09:23, Tiezhu Yang wrote:
>> On 2026/6/12 下午8:37, Xu Kuohai wrote:
>>> On 6/11/2026 6:12 PM, Tiezhu Yang wrote:
>>>> When an architecture implements bpf_jit_inlines_helper_call(), such
>>>> as LoongArch, ARM64, and RISC-V, the BPF verifier skips rewriting
>>>> the helper call offset (insn->imm) during the bpf_do_misc_fixups()
>>>> phase if the helper is expected to be inlined by the JIT compiler.
>>>> As a result, insn->imm remains as the raw helper enum ID.
>>
>> ...
>>
>>>> +static void bpf_fixup_fallback_inline_helpers(struct
>>>> bpf_verifier_env *env, struct bpf_prog *fp)
>>>
>>> The function name is a bit long for a static function
>>
>> If there are no objections, I will shorten the static function
>> name to bpf_fixup_inline_helpers() in v4.
>>
> 
> I think bpf_fixup_inline_helpers() is not accurate, since it is to fix
> those non-inline helpers here. bpf_fixup_non_inline_helpers() is a
> candidate. However, a shorter name is preferred.

I see your point. Indeed, we are fixing up the helpers that ultimately 
failed to be inlined on the fallback path.

How about shortening it to fixup_fallback_helpers()? Since it is a
static function, we can safely drop the bpf_ prefix to keep it concise,
clean, and accurate.If that looks good to you, I will use this name and
remove the redundant check in the next version.

> Please provide changelog btw. For a single patch, you can provide
> changelog here.

OK, no problem.

Thanks,
Tiezhu

Re: [PATCH bpf-next v2] bpf: Fix unaligned interpreter panic on JIT fallback path

[Leon Hwang]

On 15/6/26 10:05, Tiezhu Yang wrote:
> On 2026/6/15 上午9:49, Leon Hwang wrote:
>> On 15/6/26 09:23, Tiezhu Yang wrote:
[...]
>>
>> I think bpf_fixup_inline_helpers() is not accurate, since it is to fix
>> those non-inline helpers here. bpf_fixup_non_inline_helpers() is a
>> candidate. However, a shorter name is preferred.
> 
> I see your point. Indeed, we are fixing up the helpers that ultimately
> failed to be inlined on the fallback path.
> 
> How about shortening it to fixup_fallback_helpers()? Since it is a
> static function, we can safely drop the bpf_ prefix to keep it concise,
> clean, and accurate.If that looks good to you, I will use this name and
> remove the redundant check in the next version.
> 
bpf_fixup_fallback_helpers() or fixup_fallback_helpers() works for me.

Thanks,
Leon