Re: [PATCH] compiler_types: Do not mark inline functions notrace

[sashiko-bot@kernel.org]

Thank you for your contribution! Sashiko AI review found 5 potential issue(s) to consider:
- [Critical] x86/paravirt: removing notrace from inline instruments pv_ops and resume
- [High] tracing: removing notrace from inline breaks ftrace recursion protection
- [High] rcu: srcu_read_lock_fast_notrace() and tracepoints lose notrace attribute
- [High] bpf: removing notrace from inline exposes trampoline helpers to fentry
- [High] entry: removing notrace from inline instruments unvalidated entry helpers
--

commit 1eb6626c4a5ab70e5858a8a8f1c9edfca235db57
Author: Song Liu <song@kernel.org>

compiler_types: Do not mark inline functions notrace

This commit removes the notrace attribute from the inline macro definition
in include/linux/compiler_types.h so that inline functions emitted
out-of-line by the compiler can be traced by ftrace, attached to via BPF
fentry/fexit, and livepatched.

> diff --git a/include/linux/compiler_types.h b/include/linux/compiler_types.h
> index c5921f1390079..ea7f2a928b4da 100644
> --- a/include/linux/compiler_types.h
> +++ b/include/linux/compiler_types.h
> @@ -230,9 +230,8 @@ struct ftrace_likely_data {
>   * externally visible function. This makes extern inline behave as per gnu89
>   * semantics rather than c99. This prevents multiple symbol definition errors
>   * of extern inline functions at link time.
> - * A lot of inline functions can cause havoc with function tracing.
>   */
> -#define inline inline __gnu_inline __inline_maybe_unused notrace
> +#define inline inline __gnu_inline __inline_maybe_unused
>  
>  /*
>   * gcc provides both __inline__ and __inline as alternate spellings of

[Severity: Critical]
Could removing notrace from inline cause a triple-fault during x86 suspend
resume, secondary CPU bringup, or kexec when function tracing or call-depth
tracking is active?

In include/linux/compiler_attributes.h, __always_inline expands to inline:

    #define __always_inline inline __attribute__((__always_inline__))

Because __always_inline only inherits notrace through inline, removing
notrace here strips notrace from both inline and __always_inline functions
whenever their address is taken and the compiler emits an out-of-line copy.

In arch/x86/kernel/paravirt.c (compiled with CC_FLAGS_FTRACE when
CONFIG_PARAVIRT_XXL=y and CONFIG_FUNCTION_TRACER=y), pv_ops takes the
addresses of static inline and static __always_inline native_* helpers
(including native_load_idt, native_load_gdt, native_load_tr_desc,
native_set_ldt, native_write_gdt_entry, native_write_idt_entry,
native_write_ldt_entry, native_load_tls, native_load_gs_index,
native_load_sp0, native_store_tr, native_cpuid, native_read_cr0,
native_write_cr0, native_write_cr4, native_read_msr, native_write_msr,
native_read_msr_safe, native_write_msr_safe, native_read_pmc, and
native_halt):

    .cpu.read_cr0       = native_read_cr0,
    .cpu.write_cr0      = native_write_cr0,
    .cpu.write_cr4      = native_write_cr4,
    .cpu.read_msr       = native_read_msr,
    .cpu.write_msr      = native_write_msr,
    .cpu.read_msr_safe  = native_read_msr_safe,
    .cpu.write_msr_safe = native_write_msr_safe,
    .cpu.read_pmc       = native_read_pmc,
    .cpu.load_tr_desc   = native_load_tr_desc,
    .cpu.set_ldt        = native_set_ldt,
    .cpu.load_gdt       = native_load_gdt,
    .cpu.load_idt       = native_load_idt,

This forces the compiler to emit out-of-line bodies in paravirt.o with
__fentry__ callsites.

During S3 or hibernation resume, __restore_processor_state() in
arch/x86/power/cpu.c is marked notrace (rather than noinstr, so objtool does
not validate it) and calls wrmsrq(), __write_cr4(), write_cr3(),
write_cr0(), and load_idt(&ctxt->idt) -> pv_ops.cpu.load_idt ->
native_load_idt() before MSR_GS_BASE (%gs) is restored:

    static void notrace __restore_processor_state(struct saved_context *ctxt)
    {
        ...
        wrmsrq(MSR_EFER, ctxt->efer);
        __write_cr4(ctxt->cr4);
    #endif
        write_cr3(ctxt->cr3);
        write_cr2(ctxt->cr2);
        write_cr0(ctxt->cr0);

        /* Restore the IDT. */
        load_idt(&ctxt->idt);
        ...
        wrmsrq(MSR_GS_BASE, ctxt->gs_base);

Similarly, start_secondary() -> cpu_init_exception_handling() in
arch/x86/kernel/cpu/common.c calls write_gdt_entry(), load_TR_desc(), and
load_current_idt() -> load_idt() before the secondary CPU IDT is loaded, and
machine_kexec() in arch/x86/kernel/machine_kexec_64.c calls load_segments()
(which resets %gs to 0) followed by the static inline virt_to_phys() in
arch/x86/include/asm/io.h:

    load_segments();

    /* now call it */
    image->start = relocate_kernel_ptr((unsigned long)image->head,
                                       virt_to_phys(control_page),

Won't executing __fentry__ in these paths dereference %gs per-CPU state
(such as current->trace_recursion or call-depth tracking) before %gs or the
IDT is valid?

[Severity: High]
What prevents infinite ftrace recursion when out-of-line inline helpers are
called inside ftrace recursion protection itself?

Ftrace callbacks use ftrace_test_recursion_trylock() ->
trace_test_and_set_recursion() in include/linux/trace_recursion.h at entry
and ftrace_test_recursion_unlock() -> trace_clear_recursion() at exit:

    static __always_inline int trace_test_and_set_recursion(unsigned long ip,
                                                            unsigned long pip,
                                                            int start)
    {
        unsigned int val = READ_ONCE(current->trace_recursion);
        int bit;

        if (trace_warn_on_no_rcu(ip))
            return -1;

        bit = trace_get_context_bit() + start;

In translation units compiled with $(CC_FLAGS_FTRACE) (such as
kernel/livepatch/patch.c, fs/pstore/ftrace.c,
arch/x86/kernel/kprobes/ftrace.c, arch/powerpc/kernel/kprobes-ftrace.c,
kernel/rcu/tree.c, and kernel/trace/trace_events.c under
CONFIG_FUNCTION_SELF_TRACING=y), several functions called before
current->trace_recursion is set (or after it is cleared) are plain static
inline:

1. trace_test_and_set_recursion() evaluates current -> get_current() (plain
   static inline in arch/powerpc/include/asm/current.h and
   arch/m68k/include/asm/current.h), trace_warn_on_no_rcu() ->
   rcu_is_watching() (plain static inline in include/linux/rcutiny.h, and
   calls preempt_disable_notrace() / preempt_enable_notrace() in
   kernel/rcu/tree.c), and trace_get_context_bit() ->
   interrupt_context_level() -> preempt_count() (plain static inline in
   arch/arm64/include/asm/preempt.h).

2. In arch/arm64/include/asm/preempt.h, preempt_count(),
   __preempt_count_add(), __preempt_count_sub(), should_resched(), and
   __preempt_count_dec_and_test() are plain static inline without
   __always_inline or notrace, so preempt_disable_notrace() and
   preempt_enable_notrace() can be traced if emitted out-of-line.

3. In kernel/trace/trace_events.c (compiled with $(CC_FLAGS_FTRACE) when
   CONFIG_FUNCTION_SELF_TRACING=y), __init function_test_events_call() is
   registered without FTRACE_OPS_FL_RECURSION and calls static inline
   helpers tracing_gen_ctx(), tracing_gen_ctx_flags(),
   arch_local_save_flags(), preempt_disable_notrace(), and
   atomic_inc_return() before incrementing ftrace_test_event_disable, as
   well as atomic_dec() and preempt_enable_notrace() when decrementing it:

    trace_ctx = tracing_gen_ctx();
    preempt_disable_notrace();
    cpu = raw_smp_processor_id();
    disabled = atomic_inc_return(&per_cpu(ftrace_test_event_disable, cpu));

If the compiler emits any of these helpers out-of-line, won't tracing them
recurse infinitely before the recursion guard takes effect?

[Severity: High]
Should the _notrace SRCU helpers and tracepoint dispatch wrappers retain an
explicit notrace or __always_inline annotation?

In include/linux/srcu.h, srcu_read_lock_fast_notrace() and
srcu_read_unlock_fast_notrace() are documented as non-traceable, but unlike
srcu_read_lock_notrace() and srcu_read_unlock_notrace() (which are marked
static inline notrace), they rely on inline for notrace:

    /*
     * Used by tracing, cannot be traced and cannot call lockdep.
     * See srcu_read_lock_fast() for more information.
     */
    static inline struct srcu_ctr __percpu *srcu_read_lock_fast_notrace(struct srcu_struct *ssp)

Their callee srcu_check_read_flavor() in include/linux/srcutree.h and the
CONFIG_TINY_SRCU helpers in include/linux/srcutiny.h are also plain static
inline.

Similarly, in include/linux/tracepoint.h, __do_trace_##name (which uses
guard(srcu_fast_notrace)(&tracepoint_srcu) and is called on the cold
static_branch_unlikely() path as well as from notrace void
rust_do_trace_##name), trace_##name, and trace_call__##name are plain static
inline:

    static inline void __do_trace_##name(proto)                     \
    {                                                               \
        TRACEPOINT_CHECK(name)                                      \
        if (cond) {                                                 \
            guard(srcu_fast_notrace)(&tracepoint_srcu);             \
            __DO_TRACE_CALL(name, TP_ARGS(args));                   \
        }                                                           \
    }

Because __do_trace_##name is on the cold branch of trace_##name, compilers
frequently emit it out-of-line. Without notrace on inline, won't these
tracepoint and _notrace SRCU helpers become ftrace and fentry targets?

[Severity: High]
Can attaching a BPF fentry or fexit program to an uninlined helper in
kernel/bpf/trampoline.c or kernel/bpf/memalloc.c trigger infinite recursion
or llist corruption?

Unlike ftrace_ops callbacks, BPF trampolines do not use
ftrace_test_recursion_trylock(). In kernel/bpf/trampoline.c (compiled with
$(CC_FLAGS_FTRACE)), notrace __bpf_prog_enter*(), __bpf_prog_exit*(),
__bpf_tramp_enter(), and __bpf_tramp_exit() call plain static inline helpers
before prog->active recursion protection is acquired or after it is
released:

    static u64 notrace __bpf_prog_enter_recur(struct bpf_prog *prog, struct bpf_tramp_run_ctx *run_ctx)
        __acquires(RCU)
    {
        rcu_read_lock_dont_migrate();

        run_ctx->saved_run_ctx = bpf_set_run_ctx(&run_ctx->run_ctx);

        if (unlikely(!bpf_prog_get_recursion_context(prog))) {

and:

    static void notrace __bpf_prog_exit_recur(struct bpf_prog *prog, u64 start,
                                              struct bpf_tramp_run_ctx *run_ctx)
        __releases(RCU)
    {
        bpf_reset_run_ctx(run_ctx->saved_run_ctx);

        update_prog_stats(prog, start);
        bpf_prog_put_recursion_context(prog);
        rcu_read_unlock_migrate();
    }

The static inline helpers called here include __migrate_disable() and
__migrate_enable() (include/linux/sched.h), rcu_read_unlock_migrate(),
rcu_read_lock(), and rcu_read_unlock() (include/linux/rcupdate.h),
rcu_read_lock_trace() and rcu_read_unlock_trace()
(include/linux/rcupdate_trace.h), bpf_set_run_ctx(), bpf_reset_run_ctx(),
bpf_prog_get_recursion_context(), and bpf_prog_put_recursion_context()
(include/linux/bpf.h), percpu_ref_get() and percpu_ref_put()
(include/linux/percpu-refcount.h), and u64_stats_update_begin_irqsave() and
u64_stats_update_end_irqrestore() (include/linux/u64_stats_sync.h). In
kernel/bpf/verifier.c, btf_id_deny only blocks the out-of-line wrappers
migrate_disable, migrate_enable, ___migrate_enable, rcu_read_unlock_strict,
preempt_count_add, preempt_count_sub, __rcu_read_lock, and
__rcu_read_unlock, not these static inline functions.

In addition, unit_alloc(), unit_free(), and unit_free_rcu() in
kernel/bpf/memalloc.c are marked notrace specifically so BPF programs cannot
attach to them and corrupt free_llist or free_llist_extra, yet they call
plain static inline helpers __llist_add(), __llist_add_batch(), llist_add(),
llist_add_batch(), local_inc_return(), local_add_return(), and local_dec():

    static void notrace unit_free(struct bpf_mem_cache *c, void *ptr)
    {
        ...
        local_irq_save(flags);
        if (local_inc_return(&c->active) == 1) {
            __llist_add(llnode, &c->free_llist);
            cnt = ++c->free_cnt;
        } else {
            llist_add(llnode, &c->free_llist_extra);
        }
        local_dec(&c->active);

If any of these static inline helpers is emitted out-of-line and attached to
via fentry/fexit, won't the trampoline recurse infinitely or corrupt the
bpf_mem_alloc freelist?

[Severity: High]
How are entry, exit, and NMI inline helpers protected against ftrace
instrumentation on architectures without objtool noinstr validation?

CONFIG_HAVE_NOINSTR_VALIDATION is only selected on X86_64 (in
arch/x86/Kconfig, when CONFIG_DEBUG_ENTRY=y); powerpc, arm64, riscv, s390,
loongarch, and x86_32 do not have build-time objtool noinstr validation:

1. On PowerPC, arch/powerpc/include/asm/interrupt.h defines
   interrupt_handler with notrace rather than noinstr because of linker
   branch stub range limitations:

    #define interrupt_handler __visible noinline notrace __no_kcsan __no_sanitize_address

   The notrace interrupt, NMI, and syscall entry/exit paths in
   arch/powerpc/kernel/traps.c and arch/powerpc/kernel/interrupt.c call
   large static inline helpers in arch/powerpc/include/asm/entry-common.h,
   arch/powerpc/include/asm/ftrace.h, and arch/powerpc/include/asm/hw_irq.h,
   including check_return_regs_valid(), arch_interrupt_enter_prepare(),
   arch_interrupt_exit_prepare(), arch_interrupt_nmi_enter_prepare(),
   arch_interrupt_nmi_exit_prepare(), this_cpu_disable_ftrace(),
   this_cpu_enable_ftrace(), this_cpu_set_ftrace_enabled(),
   this_cpu_get_ftrace_enabled(), and (from prep_irq_for_enabled_exit() in
   arch/powerpc/kernel/interrupt.c) exit_must_hard_disable(),
   __hard_EE_RI_disable(), __hard_RI_enable(), __hard_irq_disable(),
   lazy_irq_pending_nocheck(), arch_local_irq_disable(), and
   arch_local_irq_save(). These run while MSR[RI]=0 (unrecoverable exception
   state), in real mode on Book3S-64 NMI entry, before
   this_cpu_set_ftrace_enabled(0) disables ftrace in NMI, with unreconciled
   irq_soft_mask, or after user_enter_irqoff() while RCU is not watching.

2. On arm64, arch/arm64/kernel/entry-common.c is compiled with
   $(CC_FLAGS_FTRACE), and its noinstr handlers
   (arm64_enter_from_kernel_mode(), __arm64_exit_to_kernel_mode(),
   arm64_exit_to_kernel_mode_preempt(), el0_pc(), and __sdei_handler()) call
   plain static inline helpers outside instrumentation_begin() /
   instrumentation_end(): irqentry_exit_to_kernel_mode_preempt()
   (include/linux/irq-entry-common.h), __kern_my_cpu_offset()
   (arch/arm64/include/asm/percpu.h), mte_check_tfsr_entry(),
   mte_check_tfsr_exit(), and mte_disable_tco_entry()
   (arch/arm64/include/asm/mte.h), sme_enter_from_user_mode() and
   sme_exit_to_user_mode() (arch/arm64/include/asm/fpsimd.h),
   instruction_pointer() (arch/arm64/include/asm/ptrace.h), and
   system_supports_mte(), system_has_prio_mask_debugging(),
   system_uses_hw_pan(), and cpu_has_pan()
   (arch/arm64/include/asm/cpufeature.h).

3. In generic entry code (include/linux/irq-entry-common.h),
   enter_from_user_mode() and exit_to_user_mode() call user_exit_irqoff()
   and user_enter_irqoff() (include/linux/context_tracking.h, which are
   plain static inline when !CONFIG_CONTEXT_TRACKING_USER) and
   lockdep_hardirqs_off() and lockdep_hardirqs_on()
   (include/linux/irqflags.h, which are plain static inline when
   !CONFIG_PROVE_LOCKING) outside instrumentation_begin() /
   instrumentation_end().

Won't an out-of-line copy of these helpers be instrumented by ftrace and
execute when MSR[RI]=0, in real mode, or while RCU is not watching?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261021222255.3124940-1-song@kernel.org?part=1