[PATCH] docs: Document pahole v1.26 requirement for KF_IMPLICIT_ARGS kfuncs

[PATCH] docs: Document pahole v1.26 requirement for KF_IMPLICIT_ARGS kfuncs

[zhidao su]

Since Linux 7.0, kfuncs annotated with KF_IMPLICIT_ARGS require pahole
v1.26 or later. Without it, such kfuncs have incorrect BTF prototypes in
vmlinux, causing BPF programs to fail with 'func_proto incompatible with
vmlinux' error.

This affects all sched_ext kfuncs (e.g. scx_bpf_create_dsq,
scx_bpf_dispatch) and other KF_IMPLICIT_ARGS kfuncs across the kernel.
Ubuntu 24.04 LTS ships pahole v1.25 by default, causing 23/30 sched_ext
selftests to fail on affected systems.

Document this requirement in Documentation/process/changes.rst so users
understand the failure mode and can upgrade pahole appropriately.

Signed-off-by: zhidao su <suzhidao@xiaomi.com>
---
 Documentation/process/changes.rst | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/Documentation/process/changes.rst b/Documentation/process/changes.rst
index 6b373e193548..141a4576c24d 100644
--- a/Documentation/process/changes.rst
+++ b/Documentation/process/changes.rst
@@ -145,6 +145,11 @@ Since Linux 5.2, if CONFIG_DEBUG_INFO_BTF is selected, the build system
 generates BTF (BPF Type Format) from DWARF in vmlinux, a bit later from kernel
 modules as well.  This requires pahole v1.22 or later.
 
+Since Linux 7.0, kfuncs annotated with KF_IMPLICIT_ARGS require pahole v1.26
+or later.  Without it, such kfuncs will have incorrect BTF prototypes in
+vmlinux, causing BPF programs to fail to load with a "func_proto incompatible
+with vmlinux" error.  Many sched_ext kfuncs are affected.
+
 It is found in the 'dwarves' or 'pahole' distro packages or from
 https://fedorapeople.org/~acme/dwarves/.
 
-- 
2.43.0

Re: [PATCH] docs: Document pahole v1.26 requirement for KF_IMPLICIT_ARGS kfuncs

[Jonathan Corbet]

zhidao su <soolaugust@gmail.com> writes:

> Since Linux 7.0, kfuncs annotated with KF_IMPLICIT_ARGS require pahole
> v1.26 or later. Without it, such kfuncs have incorrect BTF prototypes in
> vmlinux, causing BPF programs to fail with 'func_proto incompatible with
> vmlinux' error.
>
> This affects all sched_ext kfuncs (e.g. scx_bpf_create_dsq,
> scx_bpf_dispatch) and other KF_IMPLICIT_ARGS kfuncs across the kernel.
> Ubuntu 24.04 LTS ships pahole v1.25 by default, causing 23/30 sched_ext
> selftests to fail on affected systems.
>
> Document this requirement in Documentation/process/changes.rst so users
> understand the failure mode and can upgrade pahole appropriately.
>
> Signed-off-by: zhidao su <suzhidao@xiaomi.com>
> ---
>  Documentation/process/changes.rst | 5 +++++
>  1 file changed, 5 insertions(+)
>
> diff --git a/Documentation/process/changes.rst b/Documentation/process/changes.rst
> index 6b373e193548..141a4576c24d 100644
> --- a/Documentation/process/changes.rst
> +++ b/Documentation/process/changes.rst
> @@ -145,6 +145,11 @@ Since Linux 5.2, if CONFIG_DEBUG_INFO_BTF is selected, the build system
>  generates BTF (BPF Type Format) from DWARF in vmlinux, a bit later from kernel
>  modules as well.  This requires pahole v1.22 or later.
>  
> +Since Linux 7.0, kfuncs annotated with KF_IMPLICIT_ARGS require pahole v1.26
> +or later.  Without it, such kfuncs will have incorrect BTF prototypes in
> +vmlinux, causing BPF programs to fail to load with a "func_proto incompatible
> +with vmlinux" error.  Many sched_ext kfuncs are affected.

This seems like reasonable information, but is there a reason to not
just raise the minimum pahole version to 1.26 and be done with it?

Thanks,

jon

Re: [PATCH] docs: Document pahole v1.26 requirement for KF_IMPLICIT_ARGS kfuncs

[Alexei Starovoitov]

On Mon, Mar 23, 2026 at 11:22 PM zhidao su <soolaugust@gmail.com> wrote:
>
> Since Linux 7.0, kfuncs annotated with KF_IMPLICIT_ARGS require pahole
> v1.26 or later. Without it, such kfuncs have incorrect BTF prototypes in
> vmlinux, causing BPF programs to fail with 'func_proto incompatible with
> vmlinux' error.
>
> This affects all sched_ext kfuncs (e.g. scx_bpf_create_dsq,
> scx_bpf_dispatch) and other KF_IMPLICIT_ARGS kfuncs across the kernel.
> Ubuntu 24.04 LTS ships pahole v1.25 by default, causing 23/30 sched_ext
> selftests to fail on affected systems.

I don't think that's true.
At least when implicit args were designed the goal was to avoid
pahole dependencies.

Please share exact steps to reproduce.

Updating doc is definitely not an answer.

pw-bot: cr

Re: [PATCH] docs: Document pahole v1.26 requirement for KF_IMPLICIT_ARGS kfuncs

[zhidao su]

On Tue, 24 Mar 2026 08:12:12 -0700, Alexei Starovoitov wrote:
> I don't think that's true.
> At least when implicit args were designed the goal was to avoid
> pahole dependencies.
> Please share exact steps to reproduce.

Here are the exact reproduction steps and code path analysis.

Reproduction (Ubuntu 24.04, pahole v1.25):

  $ git clone https://github.com/sched-ext/sched_ext.git
  $ cd sched_ext && make -j$(nproc) LOCALVERSION=-test
  $ make -C tools/testing/selftests/sched_ext
  $ vng --run arch/x86/boot/bzImage --cpus 4 --memory 4G -- \
      tools/testing/selftests/sched_ext/build/runner 2>&1 | grep "func_proto"

Result: 23/30 tests fail with:
  libbpf: extern (func ksym) 'scx_bpf_create_dsq': func_proto [382]
          incompatible with vmlinux [53813]

Root cause:

The KF_IMPLICIT_ARGS mechanism requires pahole v1.26 for the DECL_TAG
generation step that enables resolve_btfids to do its btf2btf work:

1. scripts/Makefile.btf gates decl_tag_kfuncs on pahole >= 1.26:

     pahole-flags-$(call test-ge, $(pahole-ver), 126) = ... decl_tag_kfuncs

2. Without decl_tag_kfuncs, pahole does not emit DECL_TAG BTF entries
   for __bpf_kfunc-annotated functions.

3. resolve_btfids/main.c::collect_kfuncs() (line 1002) early-returns
   when nr_decl_tags == 0:

     if (!link->nr_decl_tags)
         return 0;

4. With no bpf_kfunc DECL_TAGs, btf2btf() never calls
   process_kfunc_with_implicit_args() to create _impl variants and
   strip 'aux' from the original proto.

5. Result: vmlinux retains the 3-param proto (with 'aux') for all
   KF_IMPLICIT_ARGS kfuncs.

BTF evidence from our pahole v1.25-compiled vmlinux:

  $ bpftool btf dump file vmlinux | grep -A5 '[53813]'
  [53813] FUNC_PROTO '(anon)' ret_type_id=... vlen=3
      'dsq_id' type_id=...
      'node'   type_id=...
      'aux'    type_id=...    <-- implicit arg still present, 3-param
  (no scx_bpf_create_dsq_impl exists)

With pahole v1.26, resolve_btfids creates scx_bpf_create_dsq_impl
(3-param, for verifier's find_kfunc_impl_proto) and rewrites
scx_bpf_create_dsq to 2-param (for libbpf ksym matching).

You're right that the design goal was to avoid pahole dependencies -
the implementation could be fixed in resolve_btfids to handle the
no-DECL_TAG case. But until such a fix lands, the dependency exists
in practice. Jonathan Corbet suggested raising the minimum version in
the requirements table to 1.26, which seems the cleanest fix.

Signed-off-by: zhidao su <suzhidao@xiaomi.com>

[PATCH] docs: Raise minimum pahole version to 1.26 for KF_IMPLICIT_ARGS kfuncs

[zhidao su]

Since Linux 7.0, kfuncs annotated with KF_IMPLICIT_ARGS require pahole
v1.26 or later.  Without it, such kfuncs will have incorrect BTF
prototypes in vmlinux, causing BPF programs to fail to load with a
"func_proto incompatible with vmlinux" error.  Many sched_ext kfuncs
are affected (e.g. scx_bpf_create_dsq, scx_bpf_kick_cpu).

The root cause: scripts/Makefile.btf passes --btf_features=decl_tag_kfuncs
to pahole only when pahole >= 1.26.  Without that flag, pahole emits no
DECL_TAG BTF entries for __bpf_kfunc-annotated functions.  As a result,
resolve_btfids/main.c::collect_kfuncs() finds no bpf_kfunc DECL_TAGs,
short-circuits at line 1002, and btf2btf() never creates the _impl
variants or strips the implicit 'aux' argument from the visible proto.
The vmlinux BTF retains the 3-param prototype while BPF programs declare
the 2-param version, triggering the mismatch.

Raise the minimum version in the requirements table from 1.22 to 1.26
and add a note explaining the failure mode, so users understand why
their BPF programs fail on distributions shipping pahole v1.25 (e.g.
Ubuntu 24.04 LTS).

Suggested-by: Jonathan Corbet <corbet@lwn.net>
Signed-off-by: zhidao su <suzhidao@xiaomi.com>
---
 Documentation/process/changes.rst | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/Documentation/process/changes.rst b/Documentation/process/changes.rst
index 6b373e193548..02068d72a101 100644
--- a/Documentation/process/changes.rst
+++ b/Documentation/process/changes.rst
@@ -38,7 +38,7 @@ bash                   4.2              bash --version
 binutils               2.30             ld -v
 flex                   2.5.35           flex --version
 bison                  2.0              bison --version
-pahole                 1.22             pahole --version
+pahole                 1.26             pahole --version
 util-linux             2.10o            mount --version
 kmod                   13               depmod -V
 e2fsprogs              1.41.4           e2fsck -V
@@ -145,6 +145,11 @@ Since Linux 5.2, if CONFIG_DEBUG_INFO_BTF is selected, the build system
 generates BTF (BPF Type Format) from DWARF in vmlinux, a bit later from kernel
 modules as well.  This requires pahole v1.22 or later.
 
+Since Linux 7.0, kfuncs annotated with KF_IMPLICIT_ARGS require pahole v1.26
+or later.  Without it, such kfuncs will have incorrect BTF prototypes in
+vmlinux, causing BPF programs to fail to load with a "func_proto incompatible
+with vmlinux" error.  Many sched_ext kfuncs are affected.
+
 It is found in the 'dwarves' or 'pahole' distro packages or from
 https://fedorapeople.org/~acme/dwarves/.
 
-- 
2.43.0

Re: [PATCH] docs: Raise minimum pahole version to 1.26 for KF_IMPLICIT_ARGS kfuncs

[Tejun Heo]

Applied to sched_ext/for-7.1.

Thanks.

-- 
tejun

Re: [PATCH] docs: Document pahole v1.26 requirement for KF_IMPLICIT_ARGS kfuncs

[Ihor Solodrai]

On 3/24/26 11:46 AM, zhidao su wrote:
> On Tue, 24 Mar 2026 08:12:12 -0700, Alexei Starovoitov wrote:
>> I don't think that's true.
>> At least when implicit args were designed the goal was to avoid
>> pahole dependencies.
>> Please share exact steps to reproduce.
> 
> Here are the exact reproduction steps and code path analysis.

Hi everyone, sorry I'm late to the party.

First of all, a *Nack* to the doc change in isolation, I agree with
Alexei here. Tejun, I suggest to revert it.

Doc change is insufficient and will make things only more confusing.

Kconfig checks for pahole version, as well as Makefile.btf. If we set
a new minimum version then it makes sense to assume it in the kernel
build code, and *disable CONFIG_DEBUG_INFO_BTF* if the version is not
recent enough. Also remove any flags/conditions that depend on pahole
being less than target minimum version. For reference see a patch
where I bumped the minimum version to 1.22 (903922cfa0e6):
https://lore.kernel.org/bpf/20251219181825.1289460-1-ihor.solodrai@linux.dev/

More importantly, the reported problem is real, but the actual fix is
a bump to v1.27, not v1.26. As usual AI got it about 80% right in the
analysis below (I assume this is AI output, it certainly looks like
one). Although to be fair AI was misled by a human error.

> 
> Reproduction (Ubuntu 24.04, pahole v1.25):
> 
>   $ git clone https://github.com/sched-ext/sched_ext.git
>   $ cd sched_ext && make -j$(nproc) LOCALVERSION=-test
>   $ make -C tools/testing/selftests/sched_ext
>   $ vng --run arch/x86/boot/bzImage --cpus 4 --memory 4G -- \
>       tools/testing/selftests/sched_ext/build/runner 2>&1 | grep "func_proto"
> 
> Result: 23/30 tests fail with:
>   libbpf: extern (func ksym) 'scx_bpf_create_dsq': func_proto [382]
>           incompatible with vmlinux [53813]

This is indeed an error you may get if BTF for kfuncs with
KF_IMPLICIT_ARGS is wrong.

> 
> Root cause:
> 
> The KF_IMPLICIT_ARGS mechanism requires pahole v1.26 for the DECL_TAG
> generation step that enables resolve_btfids to do its btf2btf work:
> 
> 1. scripts/Makefile.btf gates decl_tag_kfuncs on pahole >= 1.26:
> 
>      pahole-flags-$(call test-ge, $(pahole-ver), 126) = ... decl_tag_kfuncs

This is correct in that resolve_btfids expects that pahole ran with
decl_tag_kfuncs, but the version check is and was wrong.

decl_tag_kfuncs was implemented in pahole commit 72e88f29c [1], which
was released in v1.27. But on the kernel build side, the new feature
was simply appended (ebb79e96f1ea [2]) to the list of features under
1.26 check.

[1] https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?id=72e88f29c6f7e14201756e65bd66157427a61aaf
[2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v7.0-rc5&id=ebb79e96f1ea454fbcc8fe27dfe44e751bd74b4b

> 
> 2. Without decl_tag_kfuncs, pahole does not emit DECL_TAG BTF entries
>    for __bpf_kfunc-annotated functions.
> 
> 3. resolve_btfids/main.c::collect_kfuncs() (line 1002) early-returns
>    when nr_decl_tags == 0:
> 
>      if (!link->nr_decl_tags)
>          return 0;
> 
> 4. With no bpf_kfunc DECL_TAGs, btf2btf() never calls
>    process_kfunc_with_implicit_args() to create _impl variants and
>    strip 'aux' from the original proto.
> 
> 5. Result: vmlinux retains the 3-param proto (with 'aux') for all
>    KF_IMPLICIT_ARGS kfuncs.
> 
> BTF evidence from our pahole v1.25-compiled vmlinux:
> 
>   $ bpftool btf dump file vmlinux | grep -A5 '[53813]'
>   [53813] FUNC_PROTO '(anon)' ret_type_id=... vlen=3
>       'dsq_id' type_id=...
>       'node'   type_id=...
>       'aux'    type_id=...    <-- implicit arg still present, 3-param
>   (no scx_bpf_create_dsq_impl exists)

So far so good.

> 
> With pahole v1.26, resolve_btfids creates scx_bpf_create_dsq_impl
> (3-param, for verifier's find_kfunc_impl_proto) and rewrites
> scx_bpf_create_dsq to 2-param (for libbpf ksym matching).

And this is bullshit. The kernel build with pahole v1.26 has the same
problem:

  # Notice that 'bpf_wq_set_callback_impl' is absent from BTF
  $ bpftool btf dump file vmlinux | grep 'bpf_wq_set'
  [63890] FUNC 'bpf_wq_set_callback' type_id=9991 linkage=static
          'KF_bpf_wq_set_callback' val=29

vs expected (v1.27, v1.31):

  $ bpftool btf dump vmlinux | grep 'bpf_wq_set'
  [64253] FUNC 'bpf_wq_set_callback' type_id=51033 linkage=static
  [64254] FUNC 'bpf_wq_set_callback_impl' type_id=34861 linkage=static
          'KF_bpf_wq_set_callback' val=29

One could've figured this out if they tried to actually run the build.

> 
> You're right that the design goal was to avoid pahole dependencies -
> the implementation could be fixed in resolve_btfids to handle the
> no-DECL_TAG case. But until such a fix lands, the dependency exists
> in practice. Jonathan Corbet suggested raising the minimum version in
> the requirements table to 1.26, which seems the cleanest fix.
> 
> Signed-off-by: zhidao su <suzhidao@xiaomi.com>


zhidao su, I suggest you prepare a proper "minimum pahole version
bump" patch, modifying kconfig checks, Makefile.btf and the
documentation. Please route it through the BPF tree and add me to cc.

Thanks.

Re: [PATCH] docs: Document pahole v1.26 requirement for KF_IMPLICIT_ARGS kfuncs

[Tejun Heo]

On Thu, Mar 26, 2026 at 03:06:28PM -0700, Ihor Solodrai wrote:
> On 3/24/26 11:46 AM, zhidao su wrote:
> > On Tue, 24 Mar 2026 08:12:12 -0700, Alexei Starovoitov wrote:
> >> I don't think that's true.
> >> At least when implicit args were designed the goal was to avoid
> >> pahole dependencies.
> >> Please share exact steps to reproduce.
> > 
> > Here are the exact reproduction steps and code path analysis.
> 
> Hi everyone, sorry I'm late to the party.
> 
> First of all, a *Nack* to the doc change in isolation, I agree with
> Alexei here. Tejun, I suggest to revert it.

Ah, right, this isn't sched_ext proper even. Sorry about that. Reverting.

Thanks.

-- 
tejun