Re: [PATCH] docs: Document pahole v1.26 requirement for KF_IMPLICIT_ARGS kfuncs

Re: [PATCH] docs: Document pahole v1.26 requirement for KF_IMPLICIT_ARGS kfuncs

[zhidao su]

On Tue, 24 Mar 2026 08:12:12 -0700, Alexei Starovoitov wrote:
> I don't think that's true.
> At least when implicit args were designed the goal was to avoid
> pahole dependencies.
> Please share exact steps to reproduce.

Here are the exact reproduction steps and code path analysis.

Reproduction (Ubuntu 24.04, pahole v1.25):

  $ git clone https://github.com/sched-ext/sched_ext.git
  $ cd sched_ext && make -j$(nproc) LOCALVERSION=-test
  $ make -C tools/testing/selftests/sched_ext
  $ vng --run arch/x86/boot/bzImage --cpus 4 --memory 4G -- \
      tools/testing/selftests/sched_ext/build/runner 2>&1 | grep "func_proto"

Result: 23/30 tests fail with:
  libbpf: extern (func ksym) 'scx_bpf_create_dsq': func_proto [382]
          incompatible with vmlinux [53813]

Root cause:

The KF_IMPLICIT_ARGS mechanism requires pahole v1.26 for the DECL_TAG
generation step that enables resolve_btfids to do its btf2btf work:

1. scripts/Makefile.btf gates decl_tag_kfuncs on pahole >= 1.26:

     pahole-flags-$(call test-ge, $(pahole-ver), 126) = ... decl_tag_kfuncs

2. Without decl_tag_kfuncs, pahole does not emit DECL_TAG BTF entries
   for __bpf_kfunc-annotated functions.

3. resolve_btfids/main.c::collect_kfuncs() (line 1002) early-returns
   when nr_decl_tags == 0:

     if (!link->nr_decl_tags)
         return 0;

4. With no bpf_kfunc DECL_TAGs, btf2btf() never calls
   process_kfunc_with_implicit_args() to create _impl variants and
   strip 'aux' from the original proto.

5. Result: vmlinux retains the 3-param proto (with 'aux') for all
   KF_IMPLICIT_ARGS kfuncs.

BTF evidence from our pahole v1.25-compiled vmlinux:

  $ bpftool btf dump file vmlinux | grep -A5 '[53813]'
  [53813] FUNC_PROTO '(anon)' ret_type_id=... vlen=3
      'dsq_id' type_id=...
      'node'   type_id=...
      'aux'    type_id=...    <-- implicit arg still present, 3-param
  (no scx_bpf_create_dsq_impl exists)

With pahole v1.26, resolve_btfids creates scx_bpf_create_dsq_impl
(3-param, for verifier's find_kfunc_impl_proto) and rewrites
scx_bpf_create_dsq to 2-param (for libbpf ksym matching).

You're right that the design goal was to avoid pahole dependencies -
the implementation could be fixed in resolve_btfids to handle the
no-DECL_TAG case. But until such a fix lands, the dependency exists
in practice. Jonathan Corbet suggested raising the minimum version in
the requirements table to 1.26, which seems the cleanest fix.

Signed-off-by: zhidao su <suzhidao@xiaomi.com>

[PATCH] docs: Raise minimum pahole version to 1.26 for KF_IMPLICIT_ARGS kfuncs

[zhidao su]

Since Linux 7.0, kfuncs annotated with KF_IMPLICIT_ARGS require pahole
v1.26 or later.  Without it, such kfuncs will have incorrect BTF
prototypes in vmlinux, causing BPF programs to fail to load with a
"func_proto incompatible with vmlinux" error.  Many sched_ext kfuncs
are affected (e.g. scx_bpf_create_dsq, scx_bpf_kick_cpu).

The root cause: scripts/Makefile.btf passes --btf_features=decl_tag_kfuncs
to pahole only when pahole >= 1.26.  Without that flag, pahole emits no
DECL_TAG BTF entries for __bpf_kfunc-annotated functions.  As a result,
resolve_btfids/main.c::collect_kfuncs() finds no bpf_kfunc DECL_TAGs,
short-circuits at line 1002, and btf2btf() never creates the _impl
variants or strips the implicit 'aux' argument from the visible proto.
The vmlinux BTF retains the 3-param prototype while BPF programs declare
the 2-param version, triggering the mismatch.

Raise the minimum version in the requirements table from 1.22 to 1.26
and add a note explaining the failure mode, so users understand why
their BPF programs fail on distributions shipping pahole v1.25 (e.g.
Ubuntu 24.04 LTS).

Suggested-by: Jonathan Corbet <corbet@lwn.net>
Signed-off-by: zhidao su <suzhidao@xiaomi.com>
---
 Documentation/process/changes.rst | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/Documentation/process/changes.rst b/Documentation/process/changes.rst
index 6b373e193548..02068d72a101 100644
--- a/Documentation/process/changes.rst
+++ b/Documentation/process/changes.rst
@@ -38,7 +38,7 @@ bash                   4.2              bash --version
 binutils               2.30             ld -v
 flex                   2.5.35           flex --version
 bison                  2.0              bison --version
-pahole                 1.22             pahole --version
+pahole                 1.26             pahole --version
 util-linux             2.10o            mount --version
 kmod                   13               depmod -V
 e2fsprogs              1.41.4           e2fsck -V
@@ -145,6 +145,11 @@ Since Linux 5.2, if CONFIG_DEBUG_INFO_BTF is selected, the build system
 generates BTF (BPF Type Format) from DWARF in vmlinux, a bit later from kernel
 modules as well.  This requires pahole v1.22 or later.
 
+Since Linux 7.0, kfuncs annotated with KF_IMPLICIT_ARGS require pahole v1.26
+or later.  Without it, such kfuncs will have incorrect BTF prototypes in
+vmlinux, causing BPF programs to fail to load with a "func_proto incompatible
+with vmlinux" error.  Many sched_ext kfuncs are affected.
+
 It is found in the 'dwarves' or 'pahole' distro packages or from
 https://fedorapeople.org/~acme/dwarves/.
 
-- 
2.43.0

Re: [PATCH] docs: Raise minimum pahole version to 1.26 for KF_IMPLICIT_ARGS kfuncs

[Tejun Heo]

Applied to sched_ext/for-7.1.

Thanks.

-- 
tejun

Re: [PATCH] docs: Document pahole v1.26 requirement for KF_IMPLICIT_ARGS kfuncs

[Ihor Solodrai]

On 3/24/26 11:46 AM, zhidao su wrote:
> On Tue, 24 Mar 2026 08:12:12 -0700, Alexei Starovoitov wrote:
>> I don't think that's true.
>> At least when implicit args were designed the goal was to avoid
>> pahole dependencies.
>> Please share exact steps to reproduce.
> 
> Here are the exact reproduction steps and code path analysis.

Hi everyone, sorry I'm late to the party.

First of all, a *Nack* to the doc change in isolation, I agree with
Alexei here. Tejun, I suggest to revert it.

Doc change is insufficient and will make things only more confusing.

Kconfig checks for pahole version, as well as Makefile.btf. If we set
a new minimum version then it makes sense to assume it in the kernel
build code, and *disable CONFIG_DEBUG_INFO_BTF* if the version is not
recent enough. Also remove any flags/conditions that depend on pahole
being less than target minimum version. For reference see a patch
where I bumped the minimum version to 1.22 (903922cfa0e6):
https://lore.kernel.org/bpf/20251219181825.1289460-1-ihor.solodrai@linux.dev/

More importantly, the reported problem is real, but the actual fix is
a bump to v1.27, not v1.26. As usual AI got it about 80% right in the
analysis below (I assume this is AI output, it certainly looks like
one). Although to be fair AI was misled by a human error.

> 
> Reproduction (Ubuntu 24.04, pahole v1.25):
> 
>   $ git clone https://github.com/sched-ext/sched_ext.git
>   $ cd sched_ext && make -j$(nproc) LOCALVERSION=-test
>   $ make -C tools/testing/selftests/sched_ext
>   $ vng --run arch/x86/boot/bzImage --cpus 4 --memory 4G -- \
>       tools/testing/selftests/sched_ext/build/runner 2>&1 | grep "func_proto"
> 
> Result: 23/30 tests fail with:
>   libbpf: extern (func ksym) 'scx_bpf_create_dsq': func_proto [382]
>           incompatible with vmlinux [53813]

This is indeed an error you may get if BTF for kfuncs with
KF_IMPLICIT_ARGS is wrong.

> 
> Root cause:
> 
> The KF_IMPLICIT_ARGS mechanism requires pahole v1.26 for the DECL_TAG
> generation step that enables resolve_btfids to do its btf2btf work:
> 
> 1. scripts/Makefile.btf gates decl_tag_kfuncs on pahole >= 1.26:
> 
>      pahole-flags-$(call test-ge, $(pahole-ver), 126) = ... decl_tag_kfuncs

This is correct in that resolve_btfids expects that pahole ran with
decl_tag_kfuncs, but the version check is and was wrong.

decl_tag_kfuncs was implemented in pahole commit 72e88f29c [1], which
was released in v1.27. But on the kernel build side, the new feature
was simply appended (ebb79e96f1ea [2]) to the list of features under
1.26 check.

[1] https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?id=72e88f29c6f7e14201756e65bd66157427a61aaf
[2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v7.0-rc5&id=ebb79e96f1ea454fbcc8fe27dfe44e751bd74b4b

> 
> 2. Without decl_tag_kfuncs, pahole does not emit DECL_TAG BTF entries
>    for __bpf_kfunc-annotated functions.
> 
> 3. resolve_btfids/main.c::collect_kfuncs() (line 1002) early-returns
>    when nr_decl_tags == 0:
> 
>      if (!link->nr_decl_tags)
>          return 0;
> 
> 4. With no bpf_kfunc DECL_TAGs, btf2btf() never calls
>    process_kfunc_with_implicit_args() to create _impl variants and
>    strip 'aux' from the original proto.
> 
> 5. Result: vmlinux retains the 3-param proto (with 'aux') for all
>    KF_IMPLICIT_ARGS kfuncs.
> 
> BTF evidence from our pahole v1.25-compiled vmlinux:
> 
>   $ bpftool btf dump file vmlinux | grep -A5 '[53813]'
>   [53813] FUNC_PROTO '(anon)' ret_type_id=... vlen=3
>       'dsq_id' type_id=...
>       'node'   type_id=...
>       'aux'    type_id=...    <-- implicit arg still present, 3-param
>   (no scx_bpf_create_dsq_impl exists)

So far so good.

> 
> With pahole v1.26, resolve_btfids creates scx_bpf_create_dsq_impl
> (3-param, for verifier's find_kfunc_impl_proto) and rewrites
> scx_bpf_create_dsq to 2-param (for libbpf ksym matching).

And this is bullshit. The kernel build with pahole v1.26 has the same
problem:

  # Notice that 'bpf_wq_set_callback_impl' is absent from BTF
  $ bpftool btf dump file vmlinux | grep 'bpf_wq_set'
  [63890] FUNC 'bpf_wq_set_callback' type_id=9991 linkage=static
          'KF_bpf_wq_set_callback' val=29

vs expected (v1.27, v1.31):

  $ bpftool btf dump vmlinux | grep 'bpf_wq_set'
  [64253] FUNC 'bpf_wq_set_callback' type_id=51033 linkage=static
  [64254] FUNC 'bpf_wq_set_callback_impl' type_id=34861 linkage=static
          'KF_bpf_wq_set_callback' val=29

One could've figured this out if they tried to actually run the build.

> 
> You're right that the design goal was to avoid pahole dependencies -
> the implementation could be fixed in resolve_btfids to handle the
> no-DECL_TAG case. But until such a fix lands, the dependency exists
> in practice. Jonathan Corbet suggested raising the minimum version in
> the requirements table to 1.26, which seems the cleanest fix.
> 
> Signed-off-by: zhidao su <suzhidao@xiaomi.com>


zhidao su, I suggest you prepare a proper "minimum pahole version
bump" patch, modifying kconfig checks, Makefile.btf and the
documentation. Please route it through the BPF tree and add me to cc.

Thanks.

Re: [PATCH] docs: Document pahole v1.26 requirement for KF_IMPLICIT_ARGS kfuncs

[Tejun Heo]

On Thu, Mar 26, 2026 at 03:06:28PM -0700, Ihor Solodrai wrote:
> On 3/24/26 11:46 AM, zhidao su wrote:
> > On Tue, 24 Mar 2026 08:12:12 -0700, Alexei Starovoitov wrote:
> >> I don't think that's true.
> >> At least when implicit args were designed the goal was to avoid
> >> pahole dependencies.
> >> Please share exact steps to reproduce.
> > 
> > Here are the exact reproduction steps and code path analysis.
> 
> Hi everyone, sorry I'm late to the party.
> 
> First of all, a *Nack* to the doc change in isolation, I agree with
> Alexei here. Tejun, I suggest to revert it.

Ah, right, this isn't sched_ext proper even. Sorry about that. Reverting.

Thanks.

-- 
tejun