* [PATCH AUTOSEL 6.17-6.1] ftrace: Fix softlockup in ftrace_module_enable [not found] <20251025160905.3857885-1-sashal@kernel.org> @ 2025-10-25 16:00 ` Sasha Levin 2025-10-25 19:25 ` Steven Rostedt 0 siblings, 1 reply; 3+ messages in thread From: Sasha Levin @ 2025-10-25 16:00 UTC (permalink / raw) To: patches, stable Cc: Vladimir Riabchun, Steven Rostedt (Google), Sasha Levin, mhiramat, linux-kernel, linux-trace-kernel From: Vladimir Riabchun <ferr.lambarginio@gmail.com> [ Upstream commit 4099b98203d6b33d990586542fa5beee408032a3 ] A soft lockup was observed when loading amdgpu module. If a module has a lot of tracable functions, multiple calls to kallsyms_lookup can spend too much time in RCU critical section and with disabled preemption, causing kernel panic. This is the same issue that was fixed in commit d0b24b4e91fc ("ftrace: Prevent RCU stall on PREEMPT_VOLUNTARY kernels") and commit 42ea22e754ba ("ftrace: Add cond_resched() to ftrace_graph_set_hash()"). Fix it the same way by adding cond_resched() in ftrace_module_enable. Link: https://lore.kernel.org/aMQD9_lxYmphT-up@vova-pc Signed-off-by: Vladimir Riabchun <ferr.lambarginio@gmail.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org> Signed-off-by: Sasha Levin <sashal@kernel.org> --- LLM Generated explanations, may be completely bogus: YES - The change inserts `cond_resched()` inside the inner iteration over every ftrace record (`kernel/trace/ftrace.c:7538`). That loop holds the ftrace mutex and, for each record, invokes heavy helpers like `test_for_valid_rec()` which in turn calls `kallsyms_lookup()` (`kernel/trace/ftrace.c:4289`). On huge modules (e.g. amdgpu) this can run for tens of milliseconds with preemption disabled, triggering the documented soft lockup/panic during module load. - `ftrace_module_enable()` runs only in process context via `prepare_coming_module()` (`kernel/module/main.c:3279`), so adding a voluntary reschedule point is safe; the same pattern already exists in other long-running ftrace loops (see commits d0b24b4e91fc and 42ea22e754ba), so this brings consistency without changing control flow or semantics. - No data structures or interfaces change, and the code still executes under the same locking (`ftrace_lock`, `text_mutex` when the arch overrides `ftrace_arch_code_modify_prepare()`), so the risk of regression is minimal: the new call simply yields CPU if needed while keeping the locks held, preventing watchdog-induced crashes but otherwise behaving identically. Given it fixes a real, user-visible soft lockup with a contained and well-understood tweak, this is an excellent candidate for stable backporting. kernel/trace/ftrace.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index a69067367c296..42bd2ba68a821 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -7535,6 +7535,8 @@ void ftrace_module_enable(struct module *mod) if (!within_module(rec->ip, mod)) break; + cond_resched(); + /* Weak functions should still be ignored */ if (!test_for_valid_rec(rec)) { /* Clear all other flags. Should not be enabled anyway */ -- 2.51.0 ^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [PATCH AUTOSEL 6.17-6.1] ftrace: Fix softlockup in ftrace_module_enable 2025-10-25 16:00 ` [PATCH AUTOSEL 6.17-6.1] ftrace: Fix softlockup in ftrace_module_enable Sasha Levin @ 2025-10-25 19:25 ` Steven Rostedt 2025-10-28 17:48 ` Sasha Levin 0 siblings, 1 reply; 3+ messages in thread From: Steven Rostedt @ 2025-10-25 19:25 UTC (permalink / raw) To: Sasha Levin Cc: patches, stable, Vladimir Riabchun, mhiramat, linux-kernel, linux-trace-kernel On Sat, 25 Oct 2025 12:00:16 -0400 Sasha Levin <sashal@kernel.org> wrote: > - The change inserts `cond_resched()` inside the inner iteration over > every ftrace record (`kernel/trace/ftrace.c:7538`). That loop holds > the ftrace mutex and, for each record, invokes heavy helpers like > `test_for_valid_rec()` which in turn calls `kallsyms_lookup()` > (`kernel/trace/ftrace.c:4289`). On huge modules (e.g. amdgpu) this can > run for tens of milliseconds with preemption disabled, triggering the It got the "preemption disabled" wrong. Well maybe when running PREEMPT_NONE it is, but the description doesn't imply that. -- Steve > documented soft lockup/panic during module load. > - `ftrace_module_enable()` runs only in process context via > `prepare_coming_module()` (`kernel/module/main.c:3279`), so adding a > voluntary reschedule point is safe; the same pattern already exists in > other long-running ftrace loops (see commits d0b24b4e91fc and > 42ea22e754ba), so this brings consistency without changing control > flow or semantics. > - No data structures or interfaces change, and the code still executes > under the same locking (`ftrace_lock`, `text_mutex` when the arch > overrides `ftrace_arch_code_modify_prepare()`), so the risk of > regression is minimal: the new call simply yields CPU if needed while > keeping the locks held, preventing watchdog-induced crashes but > otherwise behaving identically. ^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH AUTOSEL 6.17-6.1] ftrace: Fix softlockup in ftrace_module_enable 2025-10-25 19:25 ` Steven Rostedt @ 2025-10-28 17:48 ` Sasha Levin 0 siblings, 0 replies; 3+ messages in thread From: Sasha Levin @ 2025-10-28 17:48 UTC (permalink / raw) To: Steven Rostedt Cc: patches, stable, Vladimir Riabchun, mhiramat, linux-kernel, linux-trace-kernel On Sat, Oct 25, 2025 at 03:25:45PM -0400, Steven Rostedt wrote: >On Sat, 25 Oct 2025 12:00:16 -0400 >Sasha Levin <sashal@kernel.org> wrote: > >> - The change inserts `cond_resched()` inside the inner iteration over >> every ftrace record (`kernel/trace/ftrace.c:7538`). That loop holds >> the ftrace mutex and, for each record, invokes heavy helpers like >> `test_for_valid_rec()` which in turn calls `kallsyms_lookup()` >> (`kernel/trace/ftrace.c:4289`). On huge modules (e.g. amdgpu) this can >> run for tens of milliseconds with preemption disabled, triggering the > >It got the "preemption disabled" wrong. Well maybe when running >PREEMPT_NONE it is, but the description doesn't imply that. Thanks for the review! I've been trying a new LLM for part of this series, and it seems to underperform the one I was previously using. -- Thanks, Sasha ^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2025-10-28 17:48 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <20251025160905.3857885-1-sashal@kernel.org>
2025-10-25 16:00 ` [PATCH AUTOSEL 6.17-6.1] ftrace: Fix softlockup in ftrace_module_enable Sasha Levin
2025-10-25 19:25 ` Steven Rostedt
2025-10-28 17:48 ` Sasha Levin
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).