Re: next-20250605: Test regression: qemu-x86_64-compat mode ltp tracing Oops int3 kernel panic

[Masami Hiramatsu (Google) <mhiramat@kernel.org>]

On Fri, 13 Jun 2025 17:27:53 +0900
Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote:

> > Run 3:
> > - https://lkft.validation.linaro.org/scheduler/job/8311139#L1703
> 
> This is the same as Run 2, and clearer.
> 
> In do_int3(), if we hit a disappeared int3, it is evacuated after
> all. This means kprobe_int3_handler() is hit, and call get_kprobe()
> to find the corresponding kprobes. But,
> 
> ffffffff8150a040 <get_kprobe>:
> ffffffff8150a040:       f3 0f 1e fa             endbr64
> ffffffff8150a044:       e8 07 b0 e2 ff          call   ffffffff81335050 <__fentry__>
> ffffffff8150a049:       48 b8 eb 83 b5 80 46    movabs $0x61c8864680b583eb,%rax
> ffffffff8150a050:       86 c8 61 
> 
> It hits the ftrace and hooked by fgraph, and eventually returns
> via ftrace_return_to_handler()
> 
> [  137.338572] RIP: 0010:ftrace_return_to_handler+0xd5/0x1f0
> [  137.338577] Code: 00 89 55 c8 48 85 ff 74 07 4c 89 b7 80 00 00 00 49 8b 94 24 38 0b 00 00 48 98 48 8b 04 c2 48 c1 e8 0c 0f b7 c0 48 89 45 b8 cc <90> 48 8b 05 e3 ac c2 01 48 63 80 f8 00 00 00 48 0f a3 45 b8 72 39
> 
> This address is;
> 
> $ eu-addr2line -fi -e vmlinux ftrace_return_to_handler+0xd5
> arch_static_branch inlined at /builds/linux/kernel/trace/fgraph.c:839:6 in ftrace_return_to_handler
> /builds/linux/arch/x86/include/asm/jump_label.h:36:2
> __ftrace_return_to_handler
> /builds/linux/kernel/trace/fgraph.c:839:6
> ftrace_return_to_handler
> /builds/linux/kernel/trace/fgraph.c:874:9
> 
> It is for static_branch, which also uses a text_poke.
> 
> -----
> #ifdef CONFIG_HAVE_STATIC_CALL
> 	if (static_branch_likely(&fgraph_do_direct)) { <======
> 		if (test_bit(fgraph_direct_gops->idx, &bitmap))
> 			static_call(fgraph_retfunc)(&trace, fgraph_direct_gops, fregs);
> -----
> 
> But actually, this static_branch modifies the kernel code with
> smp_text_poke_single() (note, this is a wrapper of smp_text_poke_batch).
> 
> And this is MISSED by the smp_text_poke_int3_handler() again and
> go through the kprobes, and hit ftrace (fgraph) and caused this
> loop.
> 
> So the fundamental issue is that smp_text_poke_batch missed
> handling INT3. 
> 
> I guess some text_poke user do not get text_mutex?

Hmm, I've checked the smp_text_poke_* users, but it seems no problem.
Basically, those smp_text_poke* user locks text_mutex, and another
suspicious ftrace_start_up is also set under ftrace_lock.
ftrace_arch_code_modify_post_process() is also paired with
ftrace_arch_code_modify_prepare() and under ftrace_lock.


smp_text_poke_single()
  ftrace_mod_jmp()
    ftrace_enable_ftrace_graph_caller()
      ftrace_modify_all_code() -> see [*1]
    ftrace_disable_ftrace_graph_caller()
      ftrace_modify_all_code() -> see [*1]
  ftrace_update_ftrace_func()
    update_ftrace_func()
      ftrace_modify_all_code() -> see [*1]
      
smp_text_poke_batch_add()
  arch_jump_label_transform_queue() -> lock text_mutex
  ftrace_replace_code()
    ftrace_modify_all_code() <------[*1]
      arch_ftrace_update_code()
        ftrace_run_update_code() -> lock text_mutex
  ftrace_modify_code_direct() (only if ftrace_poke_late != 0)
    ftrace_make_nop()
      __ftrace_replace_code() <----[*3]
        ftrace_replace_code(weak) --> Not used on x86 (overridden)
          ftrace_modify_all_code()  <--- [*1]
            arch_ftrace_update_code() <---- [*4]
              ftrace_run_update_code()-> lock text_mutex
            __ftrace_modify_code()
              ftrace_run_stop_machine()
                arch_ftrace_update_code(weak) -> overridden on x86 see [*4]
        ftrace_module_enable() -> lock text_mutex (see below)
      ftrace_init_nop()
        ftrace_nop_initialize()
          ftrace_update_code()
            ftrace_module_enable() -> lock text_mutex
              prepare_coming_module()
                load_modole()
            ftrace_process_locs() -> lock ftrace_lock.
              ftrace_init() -> OK (ftrace_poke_late == 0 because its early)
              ftrace_module_init() -> OK (ftrace_poke_late == 0 because module is not live)
                load_module()
    ftrace_make_call()
      __ftrace_replace_code() -> see [*3]


smp_text_poke_batch_finish()
  arch_jump_label_transform_apply() -> lock text_mutex
  ftrace_arch_code_modify_post_process() -> must be OK because this unlock text_mutex
    ftrace_run_update_code()-> paired with ftrace_arch_code_modify_prepare()
    ftrace_module_enable()-> paired with ftrace_arch_code_modify_prepare() (depends on ftrace_lock && ftrace_start_up)
  ftrace_replace_code()
    ftrace_modify_all_code() -> see [*1]


ftrace_start_up <does variable set under ftrace_lock ?>
  ftrace_startup()
    ftrace_startup_subops()
      register_ftrace_graph() -> lock ftrace_lock
    register_ftrace_function_probe() -> lock ftrace_lock
    register_ftrace_function_nolock() -> lock ftrace_lock
  ftrace_shutdown()
    unregister_ftrace_function() -> lock ftrace_lock


ftrace_arch_code_modify_prepare() < this set ftrace_poke_late = 1>
  ftrace_module_enable() -> lock ftrace_lock.
  ftrace_run_update_code()
    ftrace_run_modify_code()
      ftrace_ops_update_code()
        __ftrace_hash_move_and_update_ops()
          ftrace_update_ops()
            ftrace_startup_subops()
              register_ftrace_graph()  -> lock ftrace_lock
            ftrace_shutdown_subops()
              unregister_ftrace_graph() -> lock ftrace_lock
            ftrace_hash_move_and_update_subops()
              ftrace_hash_move_and_update_ops() -> [*2]
          ftrace_hash_move_and_update_ops()  <-- [*2]
            process_mod_list() -> lock ftrace_lock
            register_ftrace_function_probe() -> lock ftrace_lock
            unregister_ftrace_function_probe_func() -> lock ftrace_lock
            ftrace_set_hash() -> lock ftrace_lock
            ftrace_regex_release() -> lock ftrace_lock
      unregister_ftrace_function_probe_func() -> lock ftrace_lock
    ftrace_startup_enable()
      ftrace_startup_all()
        ftrace_pid_reset() -> lock ftrace_lock
        pid_write() -> lock ftrace_lock
      ftrace_startup()
        ftrace_startup_subops()
          register_ftrace_graph() -> lock ftrace_lock
        register_ftrace_function_probe() -> lock ftrace_lock
        register_ftrace_function_nolock() -> lock ftrace_lock
      ftrace_startup_sysctl()
        ftrace_enable_sysctl() -> lock ftrace_lock
    ftrace_shutdown()
      ftrace_shutdown_subops()
        unregister_ftrace_graph() -> lock ftrace_lock
      unregister_ftrace_function_probe_func() -> lock ftrace_lock
      ftrace_destroy_filter_files() -> lock ftrace_lock
      unregister_ftrace_function() -> lock ftrace_lock
    ftrace_shutdown_sysctl()
      ftrace_enable_sysctl() -> lock ftrace_lock


Thanks,

-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>