bpf-restrict-fs fails to load without DYNAMIC_FTRACE_WITH_DIRECT_CALLS on arm64

bpf-restrict-fs fails to load without DYNAMIC_FTRACE_WITH_DIRECT_CALLS on arm64

[Nathan Chancellor]

Hi all,

I recently adjusted my kernel configuration for my arm64 systems that
boot Fedora to enable debug information so that BTF could be generated
so that systemd's bpf-restrict-fs program [1] can run, as it would show

  systemd[1]: bpf-restrict-fs: Failed to load BPF object: No such process

in the kernel log. After doing so though, I still get an error when the
program is loaded:

  systemd[1]: bpf-restrict-fs: Failed to link program; assuming BPF LSM is not available.

With Fedora's configuration from upstream, I see:

  systemd[1]: bpf-restrict-fs: LSM BPF program attached

I was able to figure out that enabling CONFIG_CFI_CLANG was the culprit
for the change in behavior but it does not appear to be the root cause,
as I can get the same error with GCC and the following diff (which
happens with CFI_CLANG because of the CALL_OPS dependency):

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 55fc331af337..a55754e54cd8 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -210,8 +210,8 @@ config ARM64
        select HAVE_DYNAMIC_FTRACE_WITH_ARGS \
                if (GCC_SUPPORTS_DYNAMIC_FTRACE_WITH_ARGS || \
                    CLANG_SUPPORTS_DYNAMIC_FTRACE_WITH_ARGS)
-       select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \
-               if DYNAMIC_FTRACE_WITH_ARGS && DYNAMIC_FTRACE_WITH_CALL_OPS
+       #select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \
+       #       if DYNAMIC_FTRACE_WITH_ARGS && DYNAMIC_FTRACE_WITH_CALL_OPS
        select HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS \
                if (DYNAMIC_FTRACE_WITH_ARGS && !CFI_CLANG && \
                    (CC_IS_CLANG || !CC_OPTIMIZE_FOR_SIZE))

which results in the following diff between the good and bad
configurations (and I already ruled out HID-BPF being involved here):

diff --git a/good-config b/bad-config
index 252f730..539e8fd 100644
--- a/good-config
+++ b/bad-config
@@ -4882,7 +4882,6 @@ CONFIG_HID_NTRIG=y
 #
 # HID-BPF support
 #
-CONFIG_HID_BPF=y
 # end of HID-BPF support

 CONFIG_I2C_HID=y
@@ -7534,7 +7533,6 @@ CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
 CONFIG_HAVE_FUNCTION_GRAPH_FREGS=y
 CONFIG_HAVE_FTRACE_GRAPH_FUNC=y
 CONFIG_HAVE_DYNAMIC_FTRACE=y
-CONFIG_HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS=y
 CONFIG_HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS=y
 CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS=y
 CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
@@ -7558,7 +7556,6 @@ CONFIG_FUNCTION_GRAPH_RETVAL=y
 # CONFIG_FUNCTION_GRAPH_RETADDR is not set
 CONFIG_FUNCTION_TRACE_ARGS=y
 CONFIG_DYNAMIC_FTRACE=y
-CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS=y
 CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS=y
 CONFIG_DYNAMIC_FTRACE_WITH_ARGS=y
 CONFIG_FPROBE=y

Is this expected behavior or is there some other issue here? I have not
tried different kernel versions yet but I certainly can if it would be
worthwhile. If it is not expected, I am happy to provide any information
that would be helpful for narrowing this down or test patches. This is
reproducible for me in a Fedora VM in QEMU as well, if it makes
reproducing easy.

[1]: https://github.com/systemd/systemd/blob/abe149d669c68bbf2a8dd4fab325c7e715f1fd85/src/core/bpf-restrict-fs.c

Cheers,
Nathan

Re: bpf-restrict-fs fails to load without DYNAMIC_FTRACE_WITH_DIRECT_CALLS on arm64

[Alexei Starovoitov]

On Tue, Jun 10, 2025 at 4:24 PM Nathan Chancellor <nathan@kernel.org> wrote:
>
> Hi all,
>
> I recently adjusted my kernel configuration for my arm64 systems that
> boot Fedora to enable debug information so that BTF could be generated
> so that systemd's bpf-restrict-fs program [1] can run, as it would show
>
>   systemd[1]: bpf-restrict-fs: Failed to load BPF object: No such process
>
> in the kernel log. After doing so though, I still get an error when the
> program is loaded:
>
>   systemd[1]: bpf-restrict-fs: Failed to link program; assuming BPF LSM is not available.
>
> With Fedora's configuration from upstream, I see:
>
>   systemd[1]: bpf-restrict-fs: LSM BPF program attached
>
> I was able to figure out that enabling CONFIG_CFI_CLANG was the culprit
> for the change in behavior but it does not appear to be the root cause,
> as I can get the same error with GCC and the following diff (which
> happens with CFI_CLANG because of the CALL_OPS dependency):
>
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index 55fc331af337..a55754e54cd8 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -210,8 +210,8 @@ config ARM64
>         select HAVE_DYNAMIC_FTRACE_WITH_ARGS \
>                 if (GCC_SUPPORTS_DYNAMIC_FTRACE_WITH_ARGS || \
>                     CLANG_SUPPORTS_DYNAMIC_FTRACE_WITH_ARGS)
> -       select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \
> -               if DYNAMIC_FTRACE_WITH_ARGS && DYNAMIC_FTRACE_WITH_CALL_OPS
> +       #select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \
> +       #       if DYNAMIC_FTRACE_WITH_ARGS && DYNAMIC_FTRACE_WITH_CALL_OPS
>         select HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS \
>                 if (DYNAMIC_FTRACE_WITH_ARGS && !CFI_CLANG && \
>                     (CC_IS_CLANG || !CC_OPTIMIZE_FOR_SIZE))
>
> which results in the following diff between the good and bad
> configurations (and I already ruled out HID-BPF being involved here):
>
> diff --git a/good-config b/bad-config
> index 252f730..539e8fd 100644
> --- a/good-config
> +++ b/bad-config
> @@ -4882,7 +4882,6 @@ CONFIG_HID_NTRIG=y
>  #
>  # HID-BPF support
>  #
> -CONFIG_HID_BPF=y
>  # end of HID-BPF support
>
>  CONFIG_I2C_HID=y
> @@ -7534,7 +7533,6 @@ CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
>  CONFIG_HAVE_FUNCTION_GRAPH_FREGS=y
>  CONFIG_HAVE_FTRACE_GRAPH_FUNC=y
>  CONFIG_HAVE_DYNAMIC_FTRACE=y
> -CONFIG_HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS=y
>  CONFIG_HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS=y
>  CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS=y
>  CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
> @@ -7558,7 +7556,6 @@ CONFIG_FUNCTION_GRAPH_RETVAL=y
>  # CONFIG_FUNCTION_GRAPH_RETADDR is not set
>  CONFIG_FUNCTION_TRACE_ARGS=y
>  CONFIG_DYNAMIC_FTRACE=y
> -CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS=y
>  CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS=y
>  CONFIG_DYNAMIC_FTRACE_WITH_ARGS=y
>  CONFIG_FPROBE=y
>
> Is this expected behavior or is there some other issue here?

That's expected.
See how kernel/bpf/trampoline.c is using DYNAMIC_FTRACE_WITH_DIRECT_CALLS.

Theoretically we can make bpf trampoline work without it,
but why bother? Just enable this config.

Re: bpf-restrict-fs fails to load without DYNAMIC_FTRACE_WITH_DIRECT_CALLS on arm64

[Nathan Chancellor]

On Tue, Jun 10, 2025 at 04:37:24PM -0700, Alexei Starovoitov wrote:
> On Tue, Jun 10, 2025 at 4:24 PM Nathan Chancellor <nathan@kernel.org> wrote:
> > I was able to figure out that enabling CONFIG_CFI_CLANG was the culprit
> > for the change in behavior but it does not appear to be the root cause,
> > as I can get the same error with GCC and the following diff (which
> > happens with CFI_CLANG because of the CALL_OPS dependency):
...
> > -       select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \
> > -               if DYNAMIC_FTRACE_WITH_ARGS && DYNAMIC_FTRACE_WITH_CALL_OPS
> >         select HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS \
> >                 if (DYNAMIC_FTRACE_WITH_ARGS && !CFI_CLANG && \
> >                     (CC_IS_CLANG || !CC_OPTIMIZE_FOR_SIZE))
> >
...
> That's expected.
> See how kernel/bpf/trampoline.c is using DYNAMIC_FTRACE_WITH_DIRECT_CALLS.
> 
> Theoretically we can make bpf trampoline work without it,
> but why bother? Just enable this config.

As I note above, this is incompatible with CONFIG_CFI_CLANG, which is
more important for my particular area of testing and maintenance. Since
you note this is expected, I will just go back to ignoring the warning
in my kernel logs :) thank you for the quick response!

Cheers,
Nathan

Re: bpf-restrict-fs fails to load without DYNAMIC_FTRACE_WITH_DIRECT_CALLS on arm64

[Alexei Starovoitov]

On Tue, Jun 10, 2025 at 7:05 PM Nathan Chancellor <nathan@kernel.org> wrote:
>
> On Tue, Jun 10, 2025 at 04:37:24PM -0700, Alexei Starovoitov wrote:
> > On Tue, Jun 10, 2025 at 4:24 PM Nathan Chancellor <nathan@kernel.org> wrote:
> > > I was able to figure out that enabling CONFIG_CFI_CLANG was the culprit
> > > for the change in behavior but it does not appear to be the root cause,
> > > as I can get the same error with GCC and the following diff (which
> > > happens with CFI_CLANG because of the CALL_OPS dependency):
> ...
> > > -       select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \
> > > -               if DYNAMIC_FTRACE_WITH_ARGS && DYNAMIC_FTRACE_WITH_CALL_OPS
> > >         select HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS \
> > >                 if (DYNAMIC_FTRACE_WITH_ARGS && !CFI_CLANG && \
> > >                     (CC_IS_CLANG || !CC_OPTIMIZE_FOR_SIZE))
> > >
> ...
> > That's expected.
> > See how kernel/bpf/trampoline.c is using DYNAMIC_FTRACE_WITH_DIRECT_CALLS.
> >
> > Theoretically we can make bpf trampoline work without it,
> > but why bother? Just enable this config.
>
> As I note above, this is incompatible with CONFIG_CFI_CLANG, which is
> more important for my particular area of testing and maintenance. Since
> you note this is expected, I will just go back to ignoring the warning
> in my kernel logs :) thank you for the quick response!

Somebody probably needs to fix CFI_CLANG on arm64 then.
It's not clear to me why dynamic ftrace has to be disabled in such a case.
It's not disabled for CFI_CLANG on x86, right?

Re: bpf-restrict-fs fails to load without DYNAMIC_FTRACE_WITH_DIRECT_CALLS on arm64

[Nathan Chancellor]

On Tue, Jun 10, 2025 at 07:25:52PM -0700, Alexei Starovoitov wrote:
> On Tue, Jun 10, 2025 at 7:05 PM Nathan Chancellor <nathan@kernel.org> wrote:
> >
> > On Tue, Jun 10, 2025 at 04:37:24PM -0700, Alexei Starovoitov wrote:
> > > On Tue, Jun 10, 2025 at 4:24 PM Nathan Chancellor <nathan@kernel.org> wrote:
> > > > I was able to figure out that enabling CONFIG_CFI_CLANG was the culprit
> > > > for the change in behavior but it does not appear to be the root cause,
> > > > as I can get the same error with GCC and the following diff (which
> > > > happens with CFI_CLANG because of the CALL_OPS dependency):
> > ...
> > > > -       select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \
> > > > -               if DYNAMIC_FTRACE_WITH_ARGS && DYNAMIC_FTRACE_WITH_CALL_OPS
> > > >         select HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS \
> > > >                 if (DYNAMIC_FTRACE_WITH_ARGS && !CFI_CLANG && \
> > > >                     (CC_IS_CLANG || !CC_OPTIMIZE_FOR_SIZE))
> > > >
> > ...
> > > That's expected.
> > > See how kernel/bpf/trampoline.c is using DYNAMIC_FTRACE_WITH_DIRECT_CALLS.
> > >
> > > Theoretically we can make bpf trampoline work without it,
> > > but why bother? Just enable this config.
> >
> > As I note above, this is incompatible with CONFIG_CFI_CLANG, which is
> > more important for my particular area of testing and maintenance. Since
> > you note this is expected, I will just go back to ignoring the warning
> > in my kernel logs :) thank you for the quick response!
> 
> Somebody probably needs to fix CFI_CLANG on arm64 then.
> It's not clear to me why dynamic ftrace has to be disabled in such a case.

Commit baaf553d3bc3 ("arm64: Implement HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS")
says:

  Currently, this approach is not compatible with CLANG_CFI, as the
  presence/absence of pre-function NOPs changes the offset of the
  pre-function type hash, and there's no existing mechanism to ensure a
  consistent offset for instrumented and uninstrumented functions. When
  CLANG_CFI is enabled, the existing scheme with a global ops->func
  pointer is used, and there should be no functional change. I am
  currently working with others to allow the two to work together in
  future (though this will liekly require updated compiler support).

Mark, did anything ever come to fruition from the "currently working
with others to allow the two to work together in the future"? If you
need more context, the top of the thread is
https://lore.kernel.org/20250610232418.GA3544567@ax162/.

> It's not disabled for CFI_CLANG on x86, right?

No, it is not but as far as I can tell, DYNAMIC_FTRACE_WITH_DIRECT_CALLS
on x86 does not appear to depend on DYNAMIC_FTRACE_WITH_CALL_OPS like it
does on arm64.

Cheers,
Nathan