[PATCH bpf v1 1/2] Compiler Attributes: Add __retain macro

[PATCH bpf v1 1/2] Compiler Attributes: Add __retain macro

[Tony Ambardar]

Some code includes the __used macro to prevent functions and data from
being optimized out. This macro implements __attribute__((__used__)), which
operates at the compiler and IR-level, and so still allows a linker to
remove objects intended to be kept.

Compilers supporting __attribute__((__retain__)) can address this gap by
setting the flag SHF_GNU_RETAIN on the section of a function/variable,
indicating to the linker the object should be retained. This attribute is
available since gcc 11, clang 13, and binutils 2.36.

Provide a __retain macro implementing __attribute__((__retain__)), whose
first user will be the '__bpf_kfunc' tag.

Link: https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/
Cc: stable@vger.kernel.org # v6.6+
Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
---
 include/linux/compiler_attributes.h | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/include/linux/compiler_attributes.h b/include/linux/compiler_attributes.h
index 32284cd26d52..1c22e1a734dc 100644
--- a/include/linux/compiler_attributes.h
+++ b/include/linux/compiler_attributes.h
@@ -326,6 +326,20 @@
  */
 #define __pure                          __attribute__((__pure__))
 
+/*
+ * Optional: only supported since gcc >= 11, clang >= 13
+ *
+ *   gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-retain-function-attribute
+ * clang: https://clang.llvm.org/docs/AttributeReference.html#retain
+ */
+#if __has_attribute(__retain__) && \
+	(defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || \
+	 defined(CONFIG_LTO_CLANG))
+# define __retain			__attribute__((__retain__))
+#else
+# define __retain
+#endif
+
 /*
  *   gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-section-function-attribute
  *   gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Variable-Attributes.html#index-section-variable-attribute
-- 
2.34.1

Re: [PATCH bpf v1 1/2] Compiler Attributes: Add __retain macro

[Miguel Ojeda]

On Mon, Jun 3, 2024 at 2:16 PM Tony Ambardar <tony.ambardar@gmail.com> wrote:
>
> +#if __has_attribute(__retain__) && \
> +       (defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || \
> +        defined(CONFIG_LTO_CLANG))

Since this attribute depends on the configuration, please move it to
`compiler_types.h` instead.

Cheers,
Miguel

Re: [PATCH bpf v1 1/2] Compiler Attributes: Add __retain macro

[Tony Ambardar]

On Mon, Jun 03, 2024 at 03:57:38PM +0200, Miguel Ojeda wrote:
> On Mon, Jun 3, 2024 at 2:16 PM Tony Ambardar <tony.ambardar@gmail.com> wrote:
> >
> > +#if __has_attribute(__retain__) && \
> > +       (defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || \
> > +        defined(CONFIG_LTO_CLANG))
> 
> Since this attribute depends on the configuration, please move it to
> `compiler_types.h` instead.

Noted and thanks for clarifying; I'll change in v2.

Cheers,
Tony

> 
> Cheers,
> Miguel

[PATCH bpf v1 2/2] bpf: Harden __bpf_kfunc tag against linker kfunc removal

[Tony Ambardar]

BPF kfuncs are often not directly referenced and may be inadvertently
removed by optimization steps during kernel builds, thus the __bpf_kfunc
tag mitigates against this removal by including the __used macro. However,
this macro alone does not prevent removal during linking, and may still
yield build warnings (e.g. on mips64el):

    LD      vmlinux
    BTFIDS  vmlinux
  WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
  WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
  WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
  WARN: resolve_btfids: unresolved symbol bpf_key_put
  WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
  WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
  WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
  WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
  WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
  WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
  WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
  WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
    NM      System.map
    SORTTAB vmlinux
    OBJCOPY vmlinux.32

Update the __bpf_kfunc tag to better guard against linker optimization by
including the new __retain compiler macro, which fixes the warnings above.

Verify the __retain macro with readelf by checking object flags for 'R':

  $ readelf -Wa kernel/trace/bpf_trace.o
  Section Headers:
    [Nr]  Name              Type     Address  Off  Size ES Flg Lk Inf Al
  ...
    [178] .text.bpf_key_put PROGBITS 00000000 6420 0050 00 AXR  0   0  8
  ...
  Key to Flags:
  ...
    R (retain), D (mbind), p (processor specific)

Link: https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/r/202401211357.OCX9yllM-lkp@intel.com/
Fixes: 57e7c169cd6a ("bpf: Add __bpf_kfunc tag for marking kernel functions as kfuncs")
Cc: stable@vger.kernel.org # v6.6+
Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
---
 include/linux/btf.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/linux/btf.h b/include/linux/btf.h
index f9e56fd12a9f..7c3e40c3295e 100644
--- a/include/linux/btf.h
+++ b/include/linux/btf.h
@@ -82,7 +82,7 @@
  * as to avoid issues such as the compiler inlining or eliding either a static
  * kfunc, or a global kfunc in an LTO build.
  */
-#define __bpf_kfunc __used noinline
+#define __bpf_kfunc __used __retain noinline
 
 #define __bpf_kfunc_start_defs()					       \
 	__diag_push();							       \
-- 
2.34.1

[PATCH bpf v2 1/2] compiler_types.h: Define __retain for __attribute__((__retain__))

[Tony Ambardar]

Some code includes the __used macro to prevent functions and data from
being optimized out. This macro implements __attribute__((__used__)), which
operates at the compiler and IR-level, and so still allows a linker to
remove objects intended to be kept.

Compilers supporting __attribute__((__retain__)) can address this gap by
setting the flag SHF_GNU_RETAIN on the section of a function/variable,
indicating to the linker the object should be retained. This attribute is
available since gcc 11, clang 13, and binutils 2.36.

Provide a __retain macro implementing __attribute__((__retain__)), whose
first user will be the '__bpf_kfunc' tag.

Link: https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/
Cc: stable@vger.kernel.org # v6.6+
Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
---
 include/linux/compiler_types.h | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/include/linux/compiler_types.h b/include/linux/compiler_types.h
index 93600de3800b..f14c275950b5 100644
--- a/include/linux/compiler_types.h
+++ b/include/linux/compiler_types.h
@@ -143,6 +143,29 @@ static inline void __chk_io_ptr(const volatile void __iomem *ptr) { }
 # define __preserve_most
 #endif
 
+/*
+ * Annotating a function/variable with __retain tells the compiler to place
+ * the object in its own section and set the flag SHF_GNU_RETAIN. This flag
+ * instructs the linker to retain the object during garbage-cleanup or LTO
+ * phases.
+ *
+ * Note that the __used macro is also used to prevent functions or data
+ * being optimized out, but operates at the compiler/IR-level and may still
+ * allow unintended removal of objects during linking.
+ *
+ * Optional: only supported since gcc >= 11, clang >= 13
+ *
+ *   gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-retain-function-attribute
+ * clang: https://clang.llvm.org/docs/AttributeReference.html#retain
+ */
+#if __has_attribute(__retain__) && \
+	(defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || \
+	 defined(CONFIG_LTO_CLANG))
+# define __retain			__attribute__((__retain__))
+#else
+# define __retain
+#endif
+
 /* Compiler specific macros. */
 #ifdef __clang__
 #include <linux/compiler-clang.h>
-- 
2.34.1

Re: [PATCH bpf v2 1/2] compiler_types.h: Define __retain for __attribute__((__retain__))

[Yonghong Song]

On 6/3/24 10:23 PM, Tony Ambardar wrote:
> Some code includes the __used macro to prevent functions and data from
> being optimized out. This macro implements __attribute__((__used__)), which
> operates at the compiler and IR-level, and so still allows a linker to
> remove objects intended to be kept.
>
> Compilers supporting __attribute__((__retain__)) can address this gap by
> setting the flag SHF_GNU_RETAIN on the section of a function/variable,
> indicating to the linker the object should be retained. This attribute is
> available since gcc 11, clang 13, and binutils 2.36.
>
> Provide a __retain macro implementing __attribute__((__retain__)), whose
> first user will be the '__bpf_kfunc' tag.
>
> Link: https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/
> Cc: stable@vger.kernel.org # v6.6+
> Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
> ---
>   include/linux/compiler_types.h | 23 +++++++++++++++++++++++
>   1 file changed, 23 insertions(+)
>
> diff --git a/include/linux/compiler_types.h b/include/linux/compiler_types.h
> index 93600de3800b..f14c275950b5 100644
> --- a/include/linux/compiler_types.h
> +++ b/include/linux/compiler_types.h
> @@ -143,6 +143,29 @@ static inline void __chk_io_ptr(const volatile void __iomem *ptr) { }
>   # define __preserve_most
>   #endif
>   
> +/*
> + * Annotating a function/variable with __retain tells the compiler to place
> + * the object in its own section and set the flag SHF_GNU_RETAIN. This flag
> + * instructs the linker to retain the object during garbage-cleanup or LTO
> + * phases.
> + *
> + * Note that the __used macro is also used to prevent functions or data
> + * being optimized out, but operates at the compiler/IR-level and may still
> + * allow unintended removal of objects during linking.
> + *
> + * Optional: only supported since gcc >= 11, clang >= 13
> + *
> + *   gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-retain-function-attribute
> + * clang: https://clang.llvm.org/docs/AttributeReference.html#retain
> + */
> +#if __has_attribute(__retain__) && \
> +	(defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || \
> +	 defined(CONFIG_LTO_CLANG))

Could you explain why CONFIG_LTO_CLANG is added here?
IIUC, the __used macro permits garbage collection at section
level, so CLANG_LTO_CLANG without
CONFIG_LD_DEAD_CODE_DATA_ELIMINATION
shuold not change final section dynamics, right?

> +# define __retain			__attribute__((__retain__))
> +#else
> +# define __retain
> +#endif
> +
>   /* Compiler specific macros. */
>   #ifdef __clang__
>   #include <linux/compiler-clang.h>

Re: [PATCH bpf v2 1/2] compiler_types.h: Define __retain for __attribute__((__retain__))

[Tony Ambardar]

On Tue, Jun 04, 2024 at 10:55:39PM -0700, Yonghong Song wrote:
> 
> On 6/3/24 10:23 PM, Tony Ambardar wrote:
> > Some code includes the __used macro to prevent functions and data from
> > being optimized out. This macro implements __attribute__((__used__)), which
> > operates at the compiler and IR-level, and so still allows a linker to
> > remove objects intended to be kept.
> > 
> > Compilers supporting __attribute__((__retain__)) can address this gap by
> > setting the flag SHF_GNU_RETAIN on the section of a function/variable,
> > indicating to the linker the object should be retained. This attribute is
> > available since gcc 11, clang 13, and binutils 2.36.
> > 
> > Provide a __retain macro implementing __attribute__((__retain__)), whose
> > first user will be the '__bpf_kfunc' tag.
> > 
> > Link: https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/
> > Cc: stable@vger.kernel.org # v6.6+
> > Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
> > ---
> >   include/linux/compiler_types.h | 23 +++++++++++++++++++++++
> >   1 file changed, 23 insertions(+)
> > 
> > diff --git a/include/linux/compiler_types.h b/include/linux/compiler_types.h
> > index 93600de3800b..f14c275950b5 100644
> > --- a/include/linux/compiler_types.h
> > +++ b/include/linux/compiler_types.h
> > @@ -143,6 +143,29 @@ static inline void __chk_io_ptr(const volatile void __iomem *ptr) { }
> >   # define __preserve_most
> >   #endif
> > +/*
> > + * Annotating a function/variable with __retain tells the compiler to place
> > + * the object in its own section and set the flag SHF_GNU_RETAIN. This flag
> > + * instructs the linker to retain the object during garbage-cleanup or LTO
> > + * phases.
> > + *
> > + * Note that the __used macro is also used to prevent functions or data
> > + * being optimized out, but operates at the compiler/IR-level and may still
> > + * allow unintended removal of objects during linking.
> > + *
> > + * Optional: only supported since gcc >= 11, clang >= 13
> > + *
> > + *   gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-retain-function-attribute
> > + * clang: https://clang.llvm.org/docs/AttributeReference.html#retain
> > + */
> > +#if __has_attribute(__retain__) && \
> > +	(defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || \
> > +	 defined(CONFIG_LTO_CLANG))
> 
> Could you explain why CONFIG_LTO_CLANG is added here?
> IIUC, the __used macro permits garbage collection at section
> level, so CLANG_LTO_CLANG without
> CONFIG_LD_DEAD_CODE_DATA_ELIMINATION
> shuold not change final section dynamics, right?

Hi Yonghong,

I included the conditional guard to ensure consistent behaviour between
__retain and other features forcing split sections. In particular, the same
guard is used in vmlinux.lds.h to merge split sections where needed. For
example, using __retain in llvm builds without CONFIG_LTO was failing CI
tests on kernel-patches/bpf because the kernel didn't boot properly. And in
further testing, the kernel had no issues loading BPF kfunc modules with
such split sections, so I left the module (partial) linking scripts alone.

Maybe I misunderstand you question re: __used?

Thanks,
Tony
> 
> > +# define __retain			__attribute__((__retain__))
> > +#else
> > +# define __retain
> > +#endif
> > +
> >   /* Compiler specific macros. */
> >   #ifdef __clang__
> >   #include <linux/compiler-clang.h>

Re: [PATCH bpf v2 1/2] compiler_types.h: Define __retain for __attribute__((__retain__))

[Yonghong Song]

On 6/10/24 3:56 PM, Tony Ambardar wrote:
> On Tue, Jun 04, 2024 at 10:55:39PM -0700, Yonghong Song wrote:
>> On 6/3/24 10:23 PM, Tony Ambardar wrote:
>>> Some code includes the __used macro to prevent functions and data from
>>> being optimized out. This macro implements __attribute__((__used__)), which
>>> operates at the compiler and IR-level, and so still allows a linker to
>>> remove objects intended to be kept.
>>>
>>> Compilers supporting __attribute__((__retain__)) can address this gap by
>>> setting the flag SHF_GNU_RETAIN on the section of a function/variable,
>>> indicating to the linker the object should be retained. This attribute is
>>> available since gcc 11, clang 13, and binutils 2.36.
>>>
>>> Provide a __retain macro implementing __attribute__((__retain__)), whose
>>> first user will be the '__bpf_kfunc' tag.
>>>
>>> Link: https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/
>>> Cc: stable@vger.kernel.org # v6.6+
>>> Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
>>> ---
>>>    include/linux/compiler_types.h | 23 +++++++++++++++++++++++
>>>    1 file changed, 23 insertions(+)
>>>
>>> diff --git a/include/linux/compiler_types.h b/include/linux/compiler_types.h
>>> index 93600de3800b..f14c275950b5 100644
>>> --- a/include/linux/compiler_types.h
>>> +++ b/include/linux/compiler_types.h
>>> @@ -143,6 +143,29 @@ static inline void __chk_io_ptr(const volatile void __iomem *ptr) { }
>>>    # define __preserve_most
>>>    #endif
>>> +/*
>>> + * Annotating a function/variable with __retain tells the compiler to place
>>> + * the object in its own section and set the flag SHF_GNU_RETAIN. This flag
>>> + * instructs the linker to retain the object during garbage-cleanup or LTO
>>> + * phases.
>>> + *
>>> + * Note that the __used macro is also used to prevent functions or data
>>> + * being optimized out, but operates at the compiler/IR-level and may still
>>> + * allow unintended removal of objects during linking.
>>> + *
>>> + * Optional: only supported since gcc >= 11, clang >= 13
>>> + *
>>> + *   gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-retain-function-attribute
>>> + * clang: https://clang.llvm.org/docs/AttributeReference.html#retain
>>> + */
>>> +#if __has_attribute(__retain__) && \
>>> +	(defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || \
>>> +	 defined(CONFIG_LTO_CLANG))
>> Could you explain why CONFIG_LTO_CLANG is added here?
>> IIUC, the __used macro permits garbage collection at section
>> level, so CLANG_LTO_CLANG without
>> CONFIG_LD_DEAD_CODE_DATA_ELIMINATION
>> shuold not change final section dynamics, right?
> Hi Yonghong,
>
> I included the conditional guard to ensure consistent behaviour between
> __retain and other features forcing split sections. In particular, the same
> guard is used in vmlinux.lds.h to merge split sections where needed. For
> example, using __retain in llvm builds without CONFIG_LTO was failing CI
> tests on kernel-patches/bpf because the kernel didn't boot properly. And in
> further testing, the kernel had no issues loading BPF kfunc modules with
> such split sections, so I left the module (partial) linking scripts alone.

I tried with both bpf and bpf-next tree and I cannot make CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=y
in .config file. The following are all occurances in Kconfig:

$ egrep -r HAVE_LD_DEAD_CODE_DATA_ELIMINATION
arch/mips/Kconfig:      select HAVE_LD_DEAD_CODE_DATA_ELIMINATION
arch/powerpc/Kconfig:   select HAVE_LD_DEAD_CODE_DATA_ELIMINATION if HAVE_OBJTOOL_MCOUNT && (!ARCH_USING_PATCHABLE_FUNCTION_ENTRY || (!CC_IS_GCC || GCC_VERSION >= 110100))
arch/riscv/Kconfig:     select HAVE_LD_DEAD_CODE_DATA_ELIMINATION if !LD_IS_LLD
init/Kconfig:config HAVE_LD_DEAD_CODE_DATA_ELIMINATION
init/Kconfig:   depends on HAVE_LD_DEAD_CODE_DATA_ELIMINATION

Are there some pending patches to enable HAVE_LD_DEAD_CODE_DATA_ELIMINATION
for x86?

I could foce CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=y with the following hack:
diff --git a/init/Kconfig b/init/Kconfig
index 72404c1f2157..adf8718e2f5b 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -1402,7 +1402,7 @@ config CC_OPTIMIZE_FOR_SIZE
  endchoice
  
  config HAVE_LD_DEAD_CODE_DATA_ELIMINATION
-       bool
+       def_bool y
         help
           This requires that the arch annotates or otherwise protects
           its external entry points from being discarded. Linker scripts

But with the above, I cannot boot the kernel.


Did I miss anything?

>
> Maybe I misunderstand you question re: __used?
>
> Thanks,
> Tony
>>> +# define __retain			__attribute__((__retain__))
>>> +#else
>>> +# define __retain
>>> +#endif
>>> +
>>>    /* Compiler specific macros. */
>>>    #ifdef __clang__
>>>    #include <linux/compiler-clang.h>

Re: [PATCH bpf v2 1/2] compiler_types.h: Define __retain for __attribute__((__retain__))

[Tony Ambardar]

On Fri, Jun 14, 2024 at 11:47:19AM -0700, Yonghong Song wrote:
> 
> On 6/10/24 3:56 PM, Tony Ambardar wrote:
> > On Tue, Jun 04, 2024 at 10:55:39PM -0700, Yonghong Song wrote:
> > > On 6/3/24 10:23 PM, Tony Ambardar wrote:
> > > > Some code includes the __used macro to prevent functions and data from
> > > > being optimized out. This macro implements __attribute__((__used__)), which
> > > > operates at the compiler and IR-level, and so still allows a linker to
> > > > remove objects intended to be kept.
> > > > 
> > > > Compilers supporting __attribute__((__retain__)) can address this gap by
> > > > setting the flag SHF_GNU_RETAIN on the section of a function/variable,
> > > > indicating to the linker the object should be retained. This attribute is
> > > > available since gcc 11, clang 13, and binutils 2.36.
> > > > 
> > > > Provide a __retain macro implementing __attribute__((__retain__)), whose
> > > > first user will be the '__bpf_kfunc' tag.
> > > > 
> > > > Link: https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/
> > > > Cc: stable@vger.kernel.org # v6.6+
> > > > Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
> > > > ---
> > > >    include/linux/compiler_types.h | 23 +++++++++++++++++++++++
> > > >    1 file changed, 23 insertions(+)
> > > > 
> > > > diff --git a/include/linux/compiler_types.h b/include/linux/compiler_types.h
> > > > index 93600de3800b..f14c275950b5 100644
> > > > --- a/include/linux/compiler_types.h
> > > > +++ b/include/linux/compiler_types.h
> > > > @@ -143,6 +143,29 @@ static inline void __chk_io_ptr(const volatile void __iomem *ptr) { }
> > > >    # define __preserve_most
> > > >    #endif
> > > > +/*
> > > > + * Annotating a function/variable with __retain tells the compiler to place
> > > > + * the object in its own section and set the flag SHF_GNU_RETAIN. This flag
> > > > + * instructs the linker to retain the object during garbage-cleanup or LTO
> > > > + * phases.
> > > > + *
> > > > + * Note that the __used macro is also used to prevent functions or data
> > > > + * being optimized out, but operates at the compiler/IR-level and may still
> > > > + * allow unintended removal of objects during linking.
> > > > + *
> > > > + * Optional: only supported since gcc >= 11, clang >= 13
> > > > + *
> > > > + *   gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-retain-function-attribute
> > > > + * clang: https://clang.llvm.org/docs/AttributeReference.html#retain
> > > > + */
> > > > +#if __has_attribute(__retain__) && \
> > > > +	(defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || \
> > > > +	 defined(CONFIG_LTO_CLANG))
> > > Could you explain why CONFIG_LTO_CLANG is added here?
> > > IIUC, the __used macro permits garbage collection at section
> > > level, so CLANG_LTO_CLANG without
> > > CONFIG_LD_DEAD_CODE_DATA_ELIMINATION
> > > shuold not change final section dynamics, right?
> > Hi Yonghong,
> > 
> > I included the conditional guard to ensure consistent behaviour between
> > __retain and other features forcing split sections. In particular, the same
> > guard is used in vmlinux.lds.h to merge split sections where needed. For
> > example, using __retain in llvm builds without CONFIG_LTO was failing CI
> > tests on kernel-patches/bpf because the kernel didn't boot properly. And in
> > further testing, the kernel had no issues loading BPF kfunc modules with
> > such split sections, so I left the module (partial) linking scripts alone.
> 
> I tried with both bpf and bpf-next tree and I cannot make CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=y
> in .config file. The following are all occurances in Kconfig:

My understanding is one doesn't directly set HAVE_LD_DEAD_CODE_...; it's a
per-arch capability flag which guards setting LD_DEAD_CODE_DATA_ELIMINATION
but only targets "small systems" (i.e. embedded), so no surprise x86 isn't
in the arch list below.

> 
> $ egrep -r HAVE_LD_DEAD_CODE_DATA_ELIMINATION
> arch/mips/Kconfig:      select HAVE_LD_DEAD_CODE_DATA_ELIMINATION
> arch/powerpc/Kconfig:   select HAVE_LD_DEAD_CODE_DATA_ELIMINATION if HAVE_OBJTOOL_MCOUNT && (!ARCH_USING_PATCHABLE_FUNCTION_ENTRY || (!CC_IS_GCC || GCC_VERSION >= 110100))
> arch/riscv/Kconfig:     select HAVE_LD_DEAD_CODE_DATA_ELIMINATION if !LD_IS_LLD
> init/Kconfig:config HAVE_LD_DEAD_CODE_DATA_ELIMINATION
> init/Kconfig:   depends on HAVE_LD_DEAD_CODE_DATA_ELIMINATION
> 
> Are there some pending patches to enable HAVE_LD_DEAD_CODE_DATA_ELIMINATION
> for x86?

I doubt it given the target arches above, but curious what's the need for
x86 support? Only x86_32? My patches were motivated seeing resolve_btfids
and pahole errors for a couple years on MIPS routers. I don't recall seeing
the same for x86 builds, so my testing focussed more on preserving x86
builds rather than adding/testing the arch flag for x86.

> 
> I could foce CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=y with the following hack:
> diff --git a/init/Kconfig b/init/Kconfig
> index 72404c1f2157..adf8718e2f5b 100644
> --- a/init/Kconfig
> +++ b/init/Kconfig
> @@ -1402,7 +1402,7 @@ config CC_OPTIMIZE_FOR_SIZE
>  endchoice
>  config HAVE_LD_DEAD_CODE_DATA_ELIMINATION
> -       bool
> +       def_bool y
>         help
>           This requires that the arch annotates or otherwise protects
>           its external entry points from being discarded. Linker scripts
> 
> But with the above, I cannot boot the kernel.

OK, interesting exercise. Setting HAVE_LD_DEAD_CODE_DATA_ELIMINATION
shouldn't change anything itself so I suppose you are also setting
LD_DEAD_CODE_DATA_ELIMINATION? From previous testing on kernel-patches/CI,
first guess would be vmlinux linker script doing section merges unaware of
some x86 quirk. Or x86-specific linker script unhappy with split sections.

> 
> 
> Did I miss anything?
> 
> > 
> > Maybe I misunderstand you question re: __used?
> > 
> > Thanks,
> > Tony
> > > > +# define __retain			__attribute__((__retain__))
> > > > +#else
> > > > +# define __retain
> > > > +#endif
> > > > +
> > > >    /* Compiler specific macros. */
> > > >    #ifdef __clang__
> > > >    #include <linux/compiler-clang.h>

Re: [PATCH bpf v2 1/2] compiler_types.h: Define __retain for __attribute__((__retain__))

[Yonghong Song]

On 6/14/24 11:57 PM, Tony Ambardar wrote:
> On Fri, Jun 14, 2024 at 11:47:19AM -0700, Yonghong Song wrote:
>> On 6/10/24 3:56 PM, Tony Ambardar wrote:
>>> On Tue, Jun 04, 2024 at 10:55:39PM -0700, Yonghong Song wrote:
>>>> On 6/3/24 10:23 PM, Tony Ambardar wrote:
>>>>> Some code includes the __used macro to prevent functions and data from
>>>>> being optimized out. This macro implements __attribute__((__used__)), which
>>>>> operates at the compiler and IR-level, and so still allows a linker to
>>>>> remove objects intended to be kept.
>>>>>
>>>>> Compilers supporting __attribute__((__retain__)) can address this gap by
>>>>> setting the flag SHF_GNU_RETAIN on the section of a function/variable,
>>>>> indicating to the linker the object should be retained. This attribute is
>>>>> available since gcc 11, clang 13, and binutils 2.36.
>>>>>
>>>>> Provide a __retain macro implementing __attribute__((__retain__)), whose
>>>>> first user will be the '__bpf_kfunc' tag.
>>>>>
>>>>> Link: https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/
>>>>> Cc: stable@vger.kernel.org # v6.6+
>>>>> Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
>>>>> ---
>>>>>     include/linux/compiler_types.h | 23 +++++++++++++++++++++++
>>>>>     1 file changed, 23 insertions(+)
>>>>>
>>>>> diff --git a/include/linux/compiler_types.h b/include/linux/compiler_types.h
>>>>> index 93600de3800b..f14c275950b5 100644
>>>>> --- a/include/linux/compiler_types.h
>>>>> +++ b/include/linux/compiler_types.h
>>>>> @@ -143,6 +143,29 @@ static inline void __chk_io_ptr(const volatile void __iomem *ptr) { }
>>>>>     # define __preserve_most
>>>>>     #endif
>>>>> +/*
>>>>> + * Annotating a function/variable with __retain tells the compiler to place
>>>>> + * the object in its own section and set the flag SHF_GNU_RETAIN. This flag
>>>>> + * instructs the linker to retain the object during garbage-cleanup or LTO
>>>>> + * phases.
>>>>> + *
>>>>> + * Note that the __used macro is also used to prevent functions or data
>>>>> + * being optimized out, but operates at the compiler/IR-level and may still
>>>>> + * allow unintended removal of objects during linking.
>>>>> + *
>>>>> + * Optional: only supported since gcc >= 11, clang >= 13
>>>>> + *
>>>>> + *   gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-retain-function-attribute
>>>>> + * clang: https://clang.llvm.org/docs/AttributeReference.html#retain
>>>>> + */
>>>>> +#if __has_attribute(__retain__) && \
>>>>> +	(defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || \
>>>>> +	 defined(CONFIG_LTO_CLANG))
>>>> Could you explain why CONFIG_LTO_CLANG is added here?
>>>> IIUC, the __used macro permits garbage collection at section
>>>> level, so CLANG_LTO_CLANG without
>>>> CONFIG_LD_DEAD_CODE_DATA_ELIMINATION
>>>> shuold not change final section dynamics, right?
>>> Hi Yonghong,
>>>
>>> I included the conditional guard to ensure consistent behaviour between
>>> __retain and other features forcing split sections. In particular, the same
>>> guard is used in vmlinux.lds.h to merge split sections where needed. For
>>> example, using __retain in llvm builds without CONFIG_LTO was failing CI
>>> tests on kernel-patches/bpf because the kernel didn't boot properly. And in
>>> further testing, the kernel had no issues loading BPF kfunc modules with
>>> such split sections, so I left the module (partial) linking scripts alone.
>> I tried with both bpf and bpf-next tree and I cannot make CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=y
>> in .config file. The following are all occurances in Kconfig:
> My understanding is one doesn't directly set HAVE_LD_DEAD_CODE_...; it's a
> per-arch capability flag which guards setting LD_DEAD_CODE_DATA_ELIMINATION
> but only targets "small systems" (i.e. embedded), so no surprise x86 isn't
> in the arch list below.

I see. Yes, mips should support it but not x86. No wonder why I cannot reproduce.

>
>> $ egrep -r HAVE_LD_DEAD_CODE_DATA_ELIMINATION
>> arch/mips/Kconfig:      select HAVE_LD_DEAD_CODE_DATA_ELIMINATION
>> arch/powerpc/Kconfig:   select HAVE_LD_DEAD_CODE_DATA_ELIMINATION if HAVE_OBJTOOL_MCOUNT && (!ARCH_USING_PATCHABLE_FUNCTION_ENTRY || (!CC_IS_GCC || GCC_VERSION >= 110100))
>> arch/riscv/Kconfig:     select HAVE_LD_DEAD_CODE_DATA_ELIMINATION if !LD_IS_LLD
>> init/Kconfig:config HAVE_LD_DEAD_CODE_DATA_ELIMINATION
>> init/Kconfig:   depends on HAVE_LD_DEAD_CODE_DATA_ELIMINATION
>>
>> Are there some pending patches to enable HAVE_LD_DEAD_CODE_DATA_ELIMINATION
>> for x86?
> I doubt it given the target arches above, but curious what's the need for
> x86 support? Only x86_32? My patches were motivated seeing resolve_btfids
> and pahole errors for a couple years on MIPS routers. I don't recall seeing
> the same for x86 builds, so my testing focussed more on preserving x86
> builds rather than adding/testing the arch flag for x86.
>> I could foce CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=y with the following hack:
>> diff --git a/init/Kconfig b/init/Kconfig
>> index 72404c1f2157..adf8718e2f5b 100644
>> --- a/init/Kconfig
>> +++ b/init/Kconfig
>> @@ -1402,7 +1402,7 @@ config CC_OPTIMIZE_FOR_SIZE
>>   endchoice
>>   config HAVE_LD_DEAD_CODE_DATA_ELIMINATION
>> -       bool
>> +       def_bool y
>>          help
>>            This requires that the arch annotates or otherwise protects
>>            its external entry points from being discarded. Linker scripts
>>
>> But with the above, I cannot boot the kernel.
> OK, interesting exercise. Setting HAVE_LD_DEAD_CODE_DATA_ELIMINATION
> shouldn't change anything itself so I suppose you are also setting
> LD_DEAD_CODE_DATA_ELIMINATION? From previous testing on kernel-patches/CI,
> first guess would be vmlinux linker script doing section merges unaware of
> some x86 quirk. Or x86-specific linker script unhappy with split sections.

I guess x86 needs additional change to make HAVE_LD_DEAD_CODE_DATA_ELIMINATION
work. I still curious about why CONFIG_LTO_CLANG is necessary.

In asm-generic/vmlinux.lds.h,

/*
  * LD_DEAD_CODE_DATA_ELIMINATION option enables -fdata-sections, which
  * generates .data.identifier sections, which need to be pulled in with
  * .data. We don't want to pull in .data..other sections, which Linux
  * has defined. Same for text and bss.
  *
  * With LTO_CLANG, the linker also splits sections by default, so we need
  * these macros to combine the sections during the final link.
  *
  * RODATA_MAIN is not used because existing code already defines .rodata.x
  * sections to be brought in with rodata.
  */
#if defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || defined(CONFIG_LTO_CLANG)
#define TEXT_MAIN .text .text.[0-9a-zA-Z_]*
#define DATA_MAIN .data .data.[0-9a-zA-Z_]* .data..L* .data..compoundliteral* .data.$__unnamed_* .data.$L*
#define SDATA_MAIN .sdata .sdata.[0-9a-zA-Z_]*
#define RODATA_MAIN .rodata .rodata.[0-9a-zA-Z_]* .rodata..L*
#define BSS_MAIN .bss .bss.[0-9a-zA-Z_]* .bss..compoundliteral*
#define SBSS_MAIN .sbss .sbss.[0-9a-zA-Z_]*
#else
#define TEXT_MAIN .text
#define DATA_MAIN .data
#define SDATA_MAIN .sdata
#define RODATA_MAIN .rodata
#define BSS_MAIN .bss
#define SBSS_MAIN .sbss
#endif

If CONFIG_LTO_CLANG is defined and CONFIG_LD_DEAD_CODE_DATA_ELIMINATION is
not defined, it is not clear whether __used functions will get eliminated
or not. I tried with thinlto with a simple example on x86 with some unused
function marked with __used, and that function survived in the final binary.

But your patch won't hurt, so I am okay with it.

>
>>
>> Did I miss anything?
>>
>>> Maybe I misunderstand you question re: __used?
>>>
>>> Thanks,
>>> Tony
>>>>> +# define __retain			__attribute__((__retain__))
>>>>> +#else
>>>>> +# define __retain
>>>>> +#endif
>>>>> +
>>>>>     /* Compiler specific macros. */
>>>>>     #ifdef __clang__
>>>>>     #include <linux/compiler-clang.h>

[PATCH bpf v2 2/2] bpf: Harden __bpf_kfunc tag against linker kfunc removal

[Tony Ambardar]

BPF kfuncs are often not directly referenced and may be inadvertently
removed by optimization steps during kernel builds, thus the __bpf_kfunc
tag mitigates against this removal by including the __used macro. However,
this macro alone does not prevent removal during linking, and may still
yield build warnings (e.g. on mips64el):

    LD      vmlinux
    BTFIDS  vmlinux
  WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
  WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
  WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
  WARN: resolve_btfids: unresolved symbol bpf_key_put
  WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
  WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
  WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
  WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
  WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
  WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
  WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
  WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
    NM      System.map
    SORTTAB vmlinux
    OBJCOPY vmlinux.32

Update the __bpf_kfunc tag to better guard against linker optimization by
including the new __retain compiler macro, which fixes the warnings above.

Verify the __retain macro with readelf by checking object flags for 'R':

  $ readelf -Wa kernel/trace/bpf_trace.o
  Section Headers:
    [Nr]  Name              Type     Address  Off  Size ES Flg Lk Inf Al
  ...
    [178] .text.bpf_key_put PROGBITS 00000000 6420 0050 00 AXR  0   0  8
  ...
  Key to Flags:
  ...
    R (retain), D (mbind), p (processor specific)

Link: https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/r/202401211357.OCX9yllM-lkp@intel.com/
Fixes: 57e7c169cd6a ("bpf: Add __bpf_kfunc tag for marking kernel functions as kfuncs")
Cc: stable@vger.kernel.org # v6.6+
Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
---
 include/linux/btf.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/linux/btf.h b/include/linux/btf.h
index f9e56fd12a9f..7c3e40c3295e 100644
--- a/include/linux/btf.h
+++ b/include/linux/btf.h
@@ -82,7 +82,7 @@
  * as to avoid issues such as the compiler inlining or eliding either a static
  * kfunc, or a global kfunc in an LTO build.
  */
-#define __bpf_kfunc __used noinline
+#define __bpf_kfunc __used __retain noinline
 
 #define __bpf_kfunc_start_defs()					       \
 	__diag_push();							       \
-- 
2.34.1

Re: [PATCH bpf v2 2/2] bpf: Harden __bpf_kfunc tag against linker kfunc removal

[Jiri Olsa]

On Mon, Jun 03, 2024 at 10:23:16PM -0700, Tony Ambardar wrote:
> BPF kfuncs are often not directly referenced and may be inadvertently
> removed by optimization steps during kernel builds, thus the __bpf_kfunc
> tag mitigates against this removal by including the __used macro. However,
> this macro alone does not prevent removal during linking, and may still
> yield build warnings (e.g. on mips64el):
> 
>     LD      vmlinux
>     BTFIDS  vmlinux
>   WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
>   WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
>   WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
>   WARN: resolve_btfids: unresolved symbol bpf_key_put
>   WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
>   WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
>   WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
>   WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
>   WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
>   WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
>   WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
>   WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
>     NM      System.map
>     SORTTAB vmlinux
>     OBJCOPY vmlinux.32
> 
> Update the __bpf_kfunc tag to better guard against linker optimization by
> including the new __retain compiler macro, which fixes the warnings above.
> 
> Verify the __retain macro with readelf by checking object flags for 'R':
> 
>   $ readelf -Wa kernel/trace/bpf_trace.o
>   Section Headers:
>     [Nr]  Name              Type     Address  Off  Size ES Flg Lk Inf Al
>   ...
>     [178] .text.bpf_key_put PROGBITS 00000000 6420 0050 00 AXR  0   0  8
>   ...
>   Key to Flags:
>   ...
>     R (retain), D (mbind), p (processor specific)
> 
> Link: https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/
> Reported-by: kernel test robot <lkp@intel.com>
> Closes: https://lore.kernel.org/r/202401211357.OCX9yllM-lkp@intel.com/
> Fixes: 57e7c169cd6a ("bpf: Add __bpf_kfunc tag for marking kernel functions as kfuncs")
> Cc: stable@vger.kernel.org # v6.6+
> Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>

tested on mips64 cross build and the warnings are gone
and related functions are in the vmlinux

patchset looks good to me

Tested-by: Jiri Olsa <jolsa@kernel.org>
Reviewed-by: Jiri Olsa <jolsa@kernel.org>

thanks,
jirka

> ---
>  include/linux/btf.h | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/include/linux/btf.h b/include/linux/btf.h
> index f9e56fd12a9f..7c3e40c3295e 100644
> --- a/include/linux/btf.h
> +++ b/include/linux/btf.h
> @@ -82,7 +82,7 @@
>   * as to avoid issues such as the compiler inlining or eliding either a static
>   * kfunc, or a global kfunc in an LTO build.
>   */
> -#define __bpf_kfunc __used noinline
> +#define __bpf_kfunc __used __retain noinline
>  
>  #define __bpf_kfunc_start_defs()					       \
>  	__diag_push();							       \
> -- 
> 2.34.1
> 

Re: [PATCH bpf v2 2/2] bpf: Harden __bpf_kfunc tag against linker kfunc removal

[Geert Uytterhoeven]

[-- Attachment #1: Type: text/plain, Size: 4091 bytes --]

 	Hi Tony,

On Mon, 3 Jun 2024, Tony Ambardar wrote:
> BPF kfuncs are often not directly referenced and may be inadvertently
> removed by optimization steps during kernel builds, thus the __bpf_kfunc
> tag mitigates against this removal by including the __used macro. However,
> this macro alone does not prevent removal during linking, and may still
> yield build warnings (e.g. on mips64el):
>
>    LD      vmlinux
>    BTFIDS  vmlinux
>  WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
>  WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
>  WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
>  WARN: resolve_btfids: unresolved symbol bpf_key_put
>  WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
>  WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
>  WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
>  WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
>  WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
>  WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
>  WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
>  WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
>    NM      System.map
>    SORTTAB vmlinux
>    OBJCOPY vmlinux.32
>
> Update the __bpf_kfunc tag to better guard against linker optimization by
> including the new __retain compiler macro, which fixes the warnings above.
>
> Verify the __retain macro with readelf by checking object flags for 'R':
>
>  $ readelf -Wa kernel/trace/bpf_trace.o
>  Section Headers:
>    [Nr]  Name              Type     Address  Off  Size ES Flg Lk Inf Al
>  ...
>    [178] .text.bpf_key_put PROGBITS 00000000 6420 0050 00 AXR  0   0  8
>  ...
>  Key to Flags:
>  ...
>    R (retain), D (mbind), p (processor specific)
>
> Link: https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/
> Reported-by: kernel test robot <lkp@intel.com>
> Closes: https://lore.kernel.org/r/202401211357.OCX9yllM-lkp@intel.com/
> Fixes: 57e7c169cd6a ("bpf: Add __bpf_kfunc tag for marking kernel functions as kfuncs")
> Cc: stable@vger.kernel.org # v6.6+
> Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>

Thanks for your patch, which is now commit 7bdcedd5c8fb88e7
("bpf: Harden __bpf_kfunc tag against linker kfunc removal") in
v6.10-rc5.

This is causing build failures on ARM with
CONFIG_LD_DEAD_CODE_DATA_ELIMINATION=y:

     net/core/filter.c:11859:1: error: ‘retain’ attribute ignored [-Werror=attributes]
     11859 | {
           | ^
     net/core/filter.c:11872:1: error: ‘retain’ attribute ignored [-Werror=attributes]
     11872 | {
           | ^
     net/core/filter.c:11885:1: error: ‘retain’ attribute ignored [-Werror=attributes]
     11885 | {
           | ^
     net/core/filter.c:11906:1: error: ‘retain’ attribute ignored [-Werror=attributes]
     11906 | {
           | ^
     net/core/filter.c:12092:1: error: ‘retain’ attribute ignored [-Werror=attributes]
     12092 | {
           | ^
     net/core/xdp.c:713:1: error: ‘retain’ attribute ignored [-Werror=attributes]
       713 | {
           | ^
     net/core/xdp.c:736:1: error: ‘retain’ attribute ignored [-Werror=attributes]
       736 | {
           | ^
     net/core/xdp.c:769:1: error: ‘retain’ attribute ignored [-Werror=attributes]
       769 | {
           | ^
     [...]

My compiler is arm-linux-gnueabihf-gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04).

> --- a/include/linux/btf.h
> +++ b/include/linux/btf.h
> @@ -82,7 +82,7 @@
>  * as to avoid issues such as the compiler inlining or eliding either a static
>  * kfunc, or a global kfunc in an LTO build.
>  */
> -#define __bpf_kfunc __used noinline
> +#define __bpf_kfunc __used __retain noinline
>
> #define __bpf_kfunc_start_defs()					       \
> 	__diag_push();							       \

Gr{oetje,eeting}s,

 						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
 							    -- Linus Torvalds

Re: [PATCH bpf v2 2/2] bpf: Harden __bpf_kfunc tag against linker kfunc removal

[Jiri Olsa]

On Tue, Jun 25, 2024 at 12:46:48PM +0200, Geert Uytterhoeven wrote:
> 	Hi Tony,
> 
> On Mon, 3 Jun 2024, Tony Ambardar wrote:
> > BPF kfuncs are often not directly referenced and may be inadvertently
> > removed by optimization steps during kernel builds, thus the __bpf_kfunc
> > tag mitigates against this removal by including the __used macro. However,
> > this macro alone does not prevent removal during linking, and may still
> > yield build warnings (e.g. on mips64el):
> > 
> >    LD      vmlinux
> >    BTFIDS  vmlinux
> >  WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
> >  WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
> >  WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
> >  WARN: resolve_btfids: unresolved symbol bpf_key_put
> >  WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
> >  WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
> >  WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
> >  WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
> >  WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
> >  WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
> >  WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
> >  WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
> >    NM      System.map
> >    SORTTAB vmlinux
> >    OBJCOPY vmlinux.32
> > 
> > Update the __bpf_kfunc tag to better guard against linker optimization by
> > including the new __retain compiler macro, which fixes the warnings above.
> > 
> > Verify the __retain macro with readelf by checking object flags for 'R':
> > 
> >  $ readelf -Wa kernel/trace/bpf_trace.o
> >  Section Headers:
> >    [Nr]  Name              Type     Address  Off  Size ES Flg Lk Inf Al
> >  ...
> >    [178] .text.bpf_key_put PROGBITS 00000000 6420 0050 00 AXR  0   0  8
> >  ...
> >  Key to Flags:
> >  ...
> >    R (retain), D (mbind), p (processor specific)
> > 
> > Link: https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/
> > Reported-by: kernel test robot <lkp@intel.com>
> > Closes: https://lore.kernel.org/r/202401211357.OCX9yllM-lkp@intel.com/
> > Fixes: 57e7c169cd6a ("bpf: Add __bpf_kfunc tag for marking kernel functions as kfuncs")
> > Cc: stable@vger.kernel.org # v6.6+
> > Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
> 
> Thanks for your patch, which is now commit 7bdcedd5c8fb88e7
> ("bpf: Harden __bpf_kfunc tag against linker kfunc removal") in
> v6.10-rc5.
> 
> This is causing build failures on ARM with
> CONFIG_LD_DEAD_CODE_DATA_ELIMINATION=y:
> 
>     net/core/filter.c:11859:1: error: ‘retain’ attribute ignored [-Werror=attributes]
>     11859 | {
>           | ^
>     net/core/filter.c:11872:1: error: ‘retain’ attribute ignored [-Werror=attributes]
>     11872 | {
>           | ^
>     net/core/filter.c:11885:1: error: ‘retain’ attribute ignored [-Werror=attributes]
>     11885 | {
>           | ^
>     net/core/filter.c:11906:1: error: ‘retain’ attribute ignored [-Werror=attributes]
>     11906 | {
>           | ^
>     net/core/filter.c:12092:1: error: ‘retain’ attribute ignored [-Werror=attributes]
>     12092 | {
>           | ^
>     net/core/xdp.c:713:1: error: ‘retain’ attribute ignored [-Werror=attributes]
>       713 | {
>           | ^
>     net/core/xdp.c:736:1: error: ‘retain’ attribute ignored [-Werror=attributes]
>       736 | {
>           | ^
>     net/core/xdp.c:769:1: error: ‘retain’ attribute ignored [-Werror=attributes]
>       769 | {
>           | ^
>     [...]
> 
> My compiler is arm-linux-gnueabihf-gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04).

hum, so it'd mean __has_attribute(__retain__) returns true while gcc still
ignores the retain attribute.. like in this bug which seems similar:
  https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99587
but not sure how it got fixed.. any chance you can upgrade gcc and retest?

jirka

> 
> > --- a/include/linux/btf.h
> > +++ b/include/linux/btf.h
> > @@ -82,7 +82,7 @@
> >  * as to avoid issues such as the compiler inlining or eliding either a static
> >  * kfunc, or a global kfunc in an LTO build.
> >  */
> > -#define __bpf_kfunc __used noinline
> > +#define __bpf_kfunc __used __retain noinline
> > 
> > #define __bpf_kfunc_start_defs()					       \
> > 	__diag_push();							       \
> 
> Gr{oetje,eeting}s,
> 
> 						Geert
> 
> --
> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
> 
> In personal conversations with technical people, I call myself a hacker. But
> when I'm talking to journalists I just say "programmer" or something like that.
> 							    -- Linus Torvalds

Re: [PATCH bpf v2 2/2] bpf: Harden __bpf_kfunc tag against linker kfunc removal

[Geert Uytterhoeven]

Hi Jiri,

On Wed, Jun 26, 2024 at 11:52 AM Jiri Olsa <olsajiri@gmail.com> wrote:
> On Tue, Jun 25, 2024 at 12:46:48PM +0200, Geert Uytterhoeven wrote:
> > On Mon, 3 Jun 2024, Tony Ambardar wrote:
> > > BPF kfuncs are often not directly referenced and may be inadvertently
> > > removed by optimization steps during kernel builds, thus the __bpf_kfunc
> > > tag mitigates against this removal by including the __used macro. However,
> > > this macro alone does not prevent removal during linking, and may still
> > > yield build warnings (e.g. on mips64el):
> > >
> > >    LD      vmlinux
> > >    BTFIDS  vmlinux
> > >  WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
> > >  WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
> > >  WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
> > >  WARN: resolve_btfids: unresolved symbol bpf_key_put
> > >  WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
> > >  WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
> > >  WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
> > >  WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
> > >  WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
> > >  WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
> > >  WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
> > >  WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
> > >    NM      System.map
> > >    SORTTAB vmlinux
> > >    OBJCOPY vmlinux.32
> > >
> > > Update the __bpf_kfunc tag to better guard against linker optimization by
> > > including the new __retain compiler macro, which fixes the warnings above.
> > >
> > > Verify the __retain macro with readelf by checking object flags for 'R':
> > >
> > >  $ readelf -Wa kernel/trace/bpf_trace.o
> > >  Section Headers:
> > >    [Nr]  Name              Type     Address  Off  Size ES Flg Lk Inf Al
> > >  ...
> > >    [178] .text.bpf_key_put PROGBITS 00000000 6420 0050 00 AXR  0   0  8
> > >  ...
> > >  Key to Flags:
> > >  ...
> > >    R (retain), D (mbind), p (processor specific)
> > >
> > > Link: https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/
> > > Reported-by: kernel test robot <lkp@intel.com>
> > > Closes: https://lore.kernel.org/r/202401211357.OCX9yllM-lkp@intel.com/
> > > Fixes: 57e7c169cd6a ("bpf: Add __bpf_kfunc tag for marking kernel functions as kfuncs")
> > > Cc: stable@vger.kernel.org # v6.6+
> > > Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
> >
> > Thanks for your patch, which is now commit 7bdcedd5c8fb88e7
> > ("bpf: Harden __bpf_kfunc tag against linker kfunc removal") in
> > v6.10-rc5.
> >
> > This is causing build failures on ARM with
> > CONFIG_LD_DEAD_CODE_DATA_ELIMINATION=y:
> >
> >     net/core/filter.c:11859:1: error: ‘retain’ attribute ignored [-Werror=attributes]
> >     11859 | {
> >           | ^
> >     net/core/filter.c:11872:1: error: ‘retain’ attribute ignored [-Werror=attributes]
> >     11872 | {
> >           | ^
> >     net/core/filter.c:11885:1: error: ‘retain’ attribute ignored [-Werror=attributes]
> >     11885 | {
> >           | ^
> >     net/core/filter.c:11906:1: error: ‘retain’ attribute ignored [-Werror=attributes]
> >     11906 | {
> >           | ^
> >     net/core/filter.c:12092:1: error: ‘retain’ attribute ignored [-Werror=attributes]
> >     12092 | {
> >           | ^
> >     net/core/xdp.c:713:1: error: ‘retain’ attribute ignored [-Werror=attributes]
> >       713 | {
> >           | ^
> >     net/core/xdp.c:736:1: error: ‘retain’ attribute ignored [-Werror=attributes]
> >       736 | {
> >           | ^
> >     net/core/xdp.c:769:1: error: ‘retain’ attribute ignored [-Werror=attributes]
> >       769 | {
> >           | ^
> >     [...]
> >
> > My compiler is arm-linux-gnueabihf-gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04).
>
> hum, so it'd mean __has_attribute(__retain__) returns true while gcc still
> ignores the retain attribute.. like in this bug which seems similar:
>   https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99587
> but not sure how it got fixed.. any chance you can upgrade gcc and retest?

Indeed, __has_attribute(__retain__) returns true, while the attribute
is not supported.

My test program:

cat > /tmp/a.c <<EOF
#if __has_attribute(__retain__)
#warning __retain__ OK
#else
#warning No __retain__
#endif

int x __attribute__((__retain__));
EOF

$ arm-linux-gnueabihf-gcc-11 -c /tmp/a.c # gcc version 11.4.0 (Ubuntu
11.4.0-1ubuntu1~22.04))

/tmp/a.c:2:2: warning: #warning __retain__ OK [-Wcpp]
    2 | #warning __retain__ OK
      |  ^~~~~~~
/tmp/a.c:7:1: warning: ‘retain’ attribute ignored [-Wattributes]
    7 | int x __attribute__((__retain__));
      | ^~~

Oops

$ arm-linux-gnueabihf-gcc-12 -c /tmp/a.c # gcc version 12.3.0 (Ubuntu
12.3.0-1ubuntu1~22.04)
/tmp/a.c:2:2: warning: #warning __retain__ OK [-Wcpp]
    2 | #warning __retain__ OK
      |  ^~~~~~~

Fixed

It works fine with the native gcc-11:

$ gcc-11 -c /tmp/a.c # gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)
/tmp/a.c:2:2: warning: #warning __retain__ OK [-Wcpp]
    2 | #warning __retain__ OK
      |  ^~~~~~~

I gave it a try on all installed gcc-11 compilers.

/usr/bin/aarch64-linux-gnu-gcc-11
/usr/bin/alpha-linux-gnu-gcc-11
/usr/bin/arm-linux-gnueabi-gcc-11
/usr/bin/arm-linux-gnueabihf-gcc-11
/usr/bin/hppa64-linux-gnu-gcc-11
/usr/bin/hppa-linux-gnu-gcc-11
/usr/bin/m68k-linux-gnu-gcc-11
/usr/bin/powerpc64le-linux-gnu-gcc-11
/usr/bin/powerpc64-linux-gnu-gcc-11
/usr/bin/powerpc-linux-gnu-gcc-11
/usr/bin/riscv64-linux-gnu-gcc-11
/usr/bin/s390x-linux-gnu-gcc-11
/usr/bin/sh4-linux-gnu-gcc-11
/usr/bin/sparc64-linux-gnu-gcc-11
/usr/bin/x86_64-linux-gnu-gcc-11
/usr/bin/x86_64-linux-gnux32-gcc-11

All of them failed (incl. x32), except for the native x86_64-linux-gnu-gcc-11.

It works fine with all installed gcc-12 compilers
(arm-linux-gnueabihf-gcc-12, m68k-linux-gnu-gcc-12, x86_64-linux-gnu-gcc-12).

With gcc-9, the absence of __retain__ is detected correctly.

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds