Re: [PATCH bpf v2 2/2] bpf: Harden __bpf_kfunc tag against linker kfunc removal

Re: [PATCH bpf v2 2/2] bpf: Harden __bpf_kfunc tag against linker kfunc removal

[Geert Uytterhoeven]

[-- Attachment #1: Type: text/plain, Size: 4091 bytes --]

 	Hi Tony,

On Mon, 3 Jun 2024, Tony Ambardar wrote:
> BPF kfuncs are often not directly referenced and may be inadvertently
> removed by optimization steps during kernel builds, thus the __bpf_kfunc
> tag mitigates against this removal by including the __used macro. However,
> this macro alone does not prevent removal during linking, and may still
> yield build warnings (e.g. on mips64el):
>
>    LD      vmlinux
>    BTFIDS  vmlinux
>  WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
>  WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
>  WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
>  WARN: resolve_btfids: unresolved symbol bpf_key_put
>  WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
>  WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
>  WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
>  WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
>  WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
>  WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
>  WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
>  WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
>    NM      System.map
>    SORTTAB vmlinux
>    OBJCOPY vmlinux.32
>
> Update the __bpf_kfunc tag to better guard against linker optimization by
> including the new __retain compiler macro, which fixes the warnings above.
>
> Verify the __retain macro with readelf by checking object flags for 'R':
>
>  $ readelf -Wa kernel/trace/bpf_trace.o
>  Section Headers:
>    [Nr]  Name              Type     Address  Off  Size ES Flg Lk Inf Al
>  ...
>    [178] .text.bpf_key_put PROGBITS 00000000 6420 0050 00 AXR  0   0  8
>  ...
>  Key to Flags:
>  ...
>    R (retain), D (mbind), p (processor specific)
>
> Link: https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/
> Reported-by: kernel test robot <lkp@intel.com>
> Closes: https://lore.kernel.org/r/202401211357.OCX9yllM-lkp@intel.com/
> Fixes: 57e7c169cd6a ("bpf: Add __bpf_kfunc tag for marking kernel functions as kfuncs")
> Cc: stable@vger.kernel.org # v6.6+
> Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>

Thanks for your patch, which is now commit 7bdcedd5c8fb88e7
("bpf: Harden __bpf_kfunc tag against linker kfunc removal") in
v6.10-rc5.

This is causing build failures on ARM with
CONFIG_LD_DEAD_CODE_DATA_ELIMINATION=y:

     net/core/filter.c:11859:1: error: ‘retain’ attribute ignored [-Werror=attributes]
     11859 | {
           | ^
     net/core/filter.c:11872:1: error: ‘retain’ attribute ignored [-Werror=attributes]
     11872 | {
           | ^
     net/core/filter.c:11885:1: error: ‘retain’ attribute ignored [-Werror=attributes]
     11885 | {
           | ^
     net/core/filter.c:11906:1: error: ‘retain’ attribute ignored [-Werror=attributes]
     11906 | {
           | ^
     net/core/filter.c:12092:1: error: ‘retain’ attribute ignored [-Werror=attributes]
     12092 | {
           | ^
     net/core/xdp.c:713:1: error: ‘retain’ attribute ignored [-Werror=attributes]
       713 | {
           | ^
     net/core/xdp.c:736:1: error: ‘retain’ attribute ignored [-Werror=attributes]
       736 | {
           | ^
     net/core/xdp.c:769:1: error: ‘retain’ attribute ignored [-Werror=attributes]
       769 | {
           | ^
     [...]

My compiler is arm-linux-gnueabihf-gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04).

> --- a/include/linux/btf.h
> +++ b/include/linux/btf.h
> @@ -82,7 +82,7 @@
>  * as to avoid issues such as the compiler inlining or eliding either a static
>  * kfunc, or a global kfunc in an LTO build.
>  */
> -#define __bpf_kfunc __used noinline
> +#define __bpf_kfunc __used __retain noinline
>
> #define __bpf_kfunc_start_defs()					       \
> 	__diag_push();							       \

Gr{oetje,eeting}s,

 						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
 							    -- Linus Torvalds

Re: [PATCH bpf v2 2/2] bpf: Harden __bpf_kfunc tag against linker kfunc removal

[Jiri Olsa]

On Tue, Jun 25, 2024 at 12:46:48PM +0200, Geert Uytterhoeven wrote:
> 	Hi Tony,
> 
> On Mon, 3 Jun 2024, Tony Ambardar wrote:
> > BPF kfuncs are often not directly referenced and may be inadvertently
> > removed by optimization steps during kernel builds, thus the __bpf_kfunc
> > tag mitigates against this removal by including the __used macro. However,
> > this macro alone does not prevent removal during linking, and may still
> > yield build warnings (e.g. on mips64el):
> > 
> >    LD      vmlinux
> >    BTFIDS  vmlinux
> >  WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
> >  WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
> >  WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
> >  WARN: resolve_btfids: unresolved symbol bpf_key_put
> >  WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
> >  WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
> >  WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
> >  WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
> >  WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
> >  WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
> >  WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
> >  WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
> >    NM      System.map
> >    SORTTAB vmlinux
> >    OBJCOPY vmlinux.32
> > 
> > Update the __bpf_kfunc tag to better guard against linker optimization by
> > including the new __retain compiler macro, which fixes the warnings above.
> > 
> > Verify the __retain macro with readelf by checking object flags for 'R':
> > 
> >  $ readelf -Wa kernel/trace/bpf_trace.o
> >  Section Headers:
> >    [Nr]  Name              Type     Address  Off  Size ES Flg Lk Inf Al
> >  ...
> >    [178] .text.bpf_key_put PROGBITS 00000000 6420 0050 00 AXR  0   0  8
> >  ...
> >  Key to Flags:
> >  ...
> >    R (retain), D (mbind), p (processor specific)
> > 
> > Link: https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/
> > Reported-by: kernel test robot <lkp@intel.com>
> > Closes: https://lore.kernel.org/r/202401211357.OCX9yllM-lkp@intel.com/
> > Fixes: 57e7c169cd6a ("bpf: Add __bpf_kfunc tag for marking kernel functions as kfuncs")
> > Cc: stable@vger.kernel.org # v6.6+
> > Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
> 
> Thanks for your patch, which is now commit 7bdcedd5c8fb88e7
> ("bpf: Harden __bpf_kfunc tag against linker kfunc removal") in
> v6.10-rc5.
> 
> This is causing build failures on ARM with
> CONFIG_LD_DEAD_CODE_DATA_ELIMINATION=y:
> 
>     net/core/filter.c:11859:1: error: ‘retain’ attribute ignored [-Werror=attributes]
>     11859 | {
>           | ^
>     net/core/filter.c:11872:1: error: ‘retain’ attribute ignored [-Werror=attributes]
>     11872 | {
>           | ^
>     net/core/filter.c:11885:1: error: ‘retain’ attribute ignored [-Werror=attributes]
>     11885 | {
>           | ^
>     net/core/filter.c:11906:1: error: ‘retain’ attribute ignored [-Werror=attributes]
>     11906 | {
>           | ^
>     net/core/filter.c:12092:1: error: ‘retain’ attribute ignored [-Werror=attributes]
>     12092 | {
>           | ^
>     net/core/xdp.c:713:1: error: ‘retain’ attribute ignored [-Werror=attributes]
>       713 | {
>           | ^
>     net/core/xdp.c:736:1: error: ‘retain’ attribute ignored [-Werror=attributes]
>       736 | {
>           | ^
>     net/core/xdp.c:769:1: error: ‘retain’ attribute ignored [-Werror=attributes]
>       769 | {
>           | ^
>     [...]
> 
> My compiler is arm-linux-gnueabihf-gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04).

hum, so it'd mean __has_attribute(__retain__) returns true while gcc still
ignores the retain attribute.. like in this bug which seems similar:
  https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99587
but not sure how it got fixed.. any chance you can upgrade gcc and retest?

jirka

> 
> > --- a/include/linux/btf.h
> > +++ b/include/linux/btf.h
> > @@ -82,7 +82,7 @@
> >  * as to avoid issues such as the compiler inlining or eliding either a static
> >  * kfunc, or a global kfunc in an LTO build.
> >  */
> > -#define __bpf_kfunc __used noinline
> > +#define __bpf_kfunc __used __retain noinline
> > 
> > #define __bpf_kfunc_start_defs()					       \
> > 	__diag_push();							       \
> 
> Gr{oetje,eeting}s,
> 
> 						Geert
> 
> --
> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
> 
> In personal conversations with technical people, I call myself a hacker. But
> when I'm talking to journalists I just say "programmer" or something like that.
> 							    -- Linus Torvalds

Re: [PATCH bpf v2 2/2] bpf: Harden __bpf_kfunc tag against linker kfunc removal

[Geert Uytterhoeven]

Hi Jiri,

On Wed, Jun 26, 2024 at 11:52 AM Jiri Olsa <olsajiri@gmail.com> wrote:
> On Tue, Jun 25, 2024 at 12:46:48PM +0200, Geert Uytterhoeven wrote:
> > On Mon, 3 Jun 2024, Tony Ambardar wrote:
> > > BPF kfuncs are often not directly referenced and may be inadvertently
> > > removed by optimization steps during kernel builds, thus the __bpf_kfunc
> > > tag mitigates against this removal by including the __used macro. However,
> > > this macro alone does not prevent removal during linking, and may still
> > > yield build warnings (e.g. on mips64el):
> > >
> > >    LD      vmlinux
> > >    BTFIDS  vmlinux
> > >  WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
> > >  WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
> > >  WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
> > >  WARN: resolve_btfids: unresolved symbol bpf_key_put
> > >  WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
> > >  WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
> > >  WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
> > >  WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
> > >  WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
> > >  WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
> > >  WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
> > >  WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
> > >    NM      System.map
> > >    SORTTAB vmlinux
> > >    OBJCOPY vmlinux.32
> > >
> > > Update the __bpf_kfunc tag to better guard against linker optimization by
> > > including the new __retain compiler macro, which fixes the warnings above.
> > >
> > > Verify the __retain macro with readelf by checking object flags for 'R':
> > >
> > >  $ readelf -Wa kernel/trace/bpf_trace.o
> > >  Section Headers:
> > >    [Nr]  Name              Type     Address  Off  Size ES Flg Lk Inf Al
> > >  ...
> > >    [178] .text.bpf_key_put PROGBITS 00000000 6420 0050 00 AXR  0   0  8
> > >  ...
> > >  Key to Flags:
> > >  ...
> > >    R (retain), D (mbind), p (processor specific)
> > >
> > > Link: https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/
> > > Reported-by: kernel test robot <lkp@intel.com>
> > > Closes: https://lore.kernel.org/r/202401211357.OCX9yllM-lkp@intel.com/
> > > Fixes: 57e7c169cd6a ("bpf: Add __bpf_kfunc tag for marking kernel functions as kfuncs")
> > > Cc: stable@vger.kernel.org # v6.6+
> > > Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
> >
> > Thanks for your patch, which is now commit 7bdcedd5c8fb88e7
> > ("bpf: Harden __bpf_kfunc tag against linker kfunc removal") in
> > v6.10-rc5.
> >
> > This is causing build failures on ARM with
> > CONFIG_LD_DEAD_CODE_DATA_ELIMINATION=y:
> >
> >     net/core/filter.c:11859:1: error: ‘retain’ attribute ignored [-Werror=attributes]
> >     11859 | {
> >           | ^
> >     net/core/filter.c:11872:1: error: ‘retain’ attribute ignored [-Werror=attributes]
> >     11872 | {
> >           | ^
> >     net/core/filter.c:11885:1: error: ‘retain’ attribute ignored [-Werror=attributes]
> >     11885 | {
> >           | ^
> >     net/core/filter.c:11906:1: error: ‘retain’ attribute ignored [-Werror=attributes]
> >     11906 | {
> >           | ^
> >     net/core/filter.c:12092:1: error: ‘retain’ attribute ignored [-Werror=attributes]
> >     12092 | {
> >           | ^
> >     net/core/xdp.c:713:1: error: ‘retain’ attribute ignored [-Werror=attributes]
> >       713 | {
> >           | ^
> >     net/core/xdp.c:736:1: error: ‘retain’ attribute ignored [-Werror=attributes]
> >       736 | {
> >           | ^
> >     net/core/xdp.c:769:1: error: ‘retain’ attribute ignored [-Werror=attributes]
> >       769 | {
> >           | ^
> >     [...]
> >
> > My compiler is arm-linux-gnueabihf-gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04).
>
> hum, so it'd mean __has_attribute(__retain__) returns true while gcc still
> ignores the retain attribute.. like in this bug which seems similar:
>   https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99587
> but not sure how it got fixed.. any chance you can upgrade gcc and retest?

Indeed, __has_attribute(__retain__) returns true, while the attribute
is not supported.

My test program:

cat > /tmp/a.c <<EOF
#if __has_attribute(__retain__)
#warning __retain__ OK
#else
#warning No __retain__
#endif

int x __attribute__((__retain__));
EOF

$ arm-linux-gnueabihf-gcc-11 -c /tmp/a.c # gcc version 11.4.0 (Ubuntu
11.4.0-1ubuntu1~22.04))

/tmp/a.c:2:2: warning: #warning __retain__ OK [-Wcpp]
    2 | #warning __retain__ OK
      |  ^~~~~~~
/tmp/a.c:7:1: warning: ‘retain’ attribute ignored [-Wattributes]
    7 | int x __attribute__((__retain__));
      | ^~~

Oops

$ arm-linux-gnueabihf-gcc-12 -c /tmp/a.c # gcc version 12.3.0 (Ubuntu
12.3.0-1ubuntu1~22.04)
/tmp/a.c:2:2: warning: #warning __retain__ OK [-Wcpp]
    2 | #warning __retain__ OK
      |  ^~~~~~~

Fixed

It works fine with the native gcc-11:

$ gcc-11 -c /tmp/a.c # gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)
/tmp/a.c:2:2: warning: #warning __retain__ OK [-Wcpp]
    2 | #warning __retain__ OK
      |  ^~~~~~~

I gave it a try on all installed gcc-11 compilers.

/usr/bin/aarch64-linux-gnu-gcc-11
/usr/bin/alpha-linux-gnu-gcc-11
/usr/bin/arm-linux-gnueabi-gcc-11
/usr/bin/arm-linux-gnueabihf-gcc-11
/usr/bin/hppa64-linux-gnu-gcc-11
/usr/bin/hppa-linux-gnu-gcc-11
/usr/bin/m68k-linux-gnu-gcc-11
/usr/bin/powerpc64le-linux-gnu-gcc-11
/usr/bin/powerpc64-linux-gnu-gcc-11
/usr/bin/powerpc-linux-gnu-gcc-11
/usr/bin/riscv64-linux-gnu-gcc-11
/usr/bin/s390x-linux-gnu-gcc-11
/usr/bin/sh4-linux-gnu-gcc-11
/usr/bin/sparc64-linux-gnu-gcc-11
/usr/bin/x86_64-linux-gnu-gcc-11
/usr/bin/x86_64-linux-gnux32-gcc-11

All of them failed (incl. x32), except for the native x86_64-linux-gnu-gcc-11.

It works fine with all installed gcc-12 compilers
(arm-linux-gnueabihf-gcc-12, m68k-linux-gnu-gcc-12, x86_64-linux-gnu-gcc-12).

With gcc-9, the absence of __retain__ is detected correctly.

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds