* [PATCH v3 0/3] Tracing: Accelerate Kernel Boot by Asynchronizing
@ 2026-01-27 1:20 Yaxiong Tian
2026-01-27 1:23 ` [PATCH v3 1/3] tracing: Rename `eval_map_wq` and allow other parts of tracing use it Yaxiong Tian
` (2 more replies)
0 siblings, 3 replies; 8+ messages in thread
From: Yaxiong Tian @ 2026-01-27 1:20 UTC (permalink / raw)
To: rostedt, axboe, mhiramat, mathieu.desnoyers
Cc: linux-block, linux-kernel, linux-trace-kernel, Yaxiong Tian
On my ARM64 platform, I observed that certain tracing module
initializations run for up to 200ms—for example, init_kprobe_trace().
Analysis reveals the root cause: the execution flow eval_map_work_func()
→trace_event_update_with_eval_map()→trace_event_update_all()
is highly time-consuming. Although this flow is placed in eval_map_wq
for asynchronous execution, it holds the trace_event_sem lock, causing
other modules to be blocked either directly or indirectly.
To resolve this issue, I rename `eval_map_wq` and make it global and moved
other initialization functions under the tracing subsystem that are
related to this lock to run asynchronously on this workqueue. After this
optimization, boot time is reduced by approximately 200ms.
---
Changes in v2:
- Rename eval_map_wq to trace_init_wq.
Changes in v3:
- Opt PATCH 1/3 commit
Yaxiong Tian (3):
tracing: Rename `eval_map_wq` and allow other parts of tracing use it
tracing/kprobes: Make setup_boot_kprobe_events() asynchronous
blktrace: Make init_blk_tracer() asynchronous
kernel/trace/blktrace.c | 23 ++++++++++++++++++++++-
kernel/trace/trace.c | 18 +++++++++---------
kernel/trace/trace.h | 2 ++
kernel/trace/trace_kprobe.c | 14 +++++++++++++-
4 files changed, 46 insertions(+), 11 deletions(-)
--
2.25.1
^ permalink raw reply [flat|nested] 8+ messages in thread* [PATCH v3 1/3] tracing: Rename `eval_map_wq` and allow other parts of tracing use it 2026-01-27 1:20 [PATCH v3 0/3] Tracing: Accelerate Kernel Boot by Asynchronizing Yaxiong Tian @ 2026-01-27 1:23 ` Yaxiong Tian 2026-01-27 1:23 ` [PATCH v3 2/3] tracing/kprobes: Make setup_boot_kprobe_events() asynchronous Yaxiong Tian 2026-01-27 1:23 ` [PATCH v3 3/3] blktrace: Make init_blk_tracer() asynchronous Yaxiong Tian 2 siblings, 0 replies; 8+ messages in thread From: Yaxiong Tian @ 2026-01-27 1:23 UTC (permalink / raw) To: rostedt, axboe, mhiramat, mathieu.desnoyers Cc: linux-block, linux-kernel, linux-trace-kernel, Yaxiong Tian The eval_map_work_func() function, though queued in eval_map_wq, holds the trace_event_sem read-write lock for a long time during kernel boot. This causes blocking issues for other functions. Rename eval_map_wq to trace_init_wq and make it global, thereby allowing other parts of tracing to schedule work on this queue asynchronously and avoiding blockage of the main boot thread. Signed-off-by: Yaxiong Tian <tianyaxiong@kylinos.cn> --- kernel/trace/trace.c | 18 +++++++++--------- kernel/trace/trace.h | 2 ++ 2 files changed, 11 insertions(+), 9 deletions(-) diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c index e18005807395..c61e30cb7339 100644 --- a/kernel/trace/trace.c +++ b/kernel/trace/trace.c @@ -10774,7 +10774,7 @@ int tracing_init_dentry(void) extern struct trace_eval_map *__start_ftrace_eval_maps[]; extern struct trace_eval_map *__stop_ftrace_eval_maps[]; -static struct workqueue_struct *eval_map_wq __initdata; +struct workqueue_struct *trace_init_wq __initdata; static struct work_struct eval_map_work __initdata; static struct work_struct tracerfs_init_work __initdata; @@ -10790,15 +10790,15 @@ static int __init trace_eval_init(void) { INIT_WORK(&eval_map_work, eval_map_work_func); - eval_map_wq = alloc_workqueue("eval_map_wq", WQ_UNBOUND, 0); - if (!eval_map_wq) { - pr_err("Unable to allocate eval_map_wq\n"); + trace_init_wq = alloc_workqueue("trace_init_wq", WQ_UNBOUND, 0); + if (!trace_init_wq) { + pr_err("Unable to allocate trace_init_wq\n"); /* Do work here */ eval_map_work_func(&eval_map_work); return -ENOMEM; } - queue_work(eval_map_wq, &eval_map_work); + queue_work(trace_init_wq, &eval_map_work); return 0; } @@ -10807,8 +10807,8 @@ subsys_initcall(trace_eval_init); static int __init trace_eval_sync(void) { /* Make sure the eval map updates are finished */ - if (eval_map_wq) - destroy_workqueue(eval_map_wq); + if (trace_init_wq) + destroy_workqueue(trace_init_wq); return 0; } @@ -10969,9 +10969,9 @@ static __init int tracer_init_tracefs(void) if (ret) return 0; - if (eval_map_wq) { + if (trace_init_wq) { INIT_WORK(&tracerfs_init_work, tracer_init_tracefs_work_func); - queue_work(eval_map_wq, &tracerfs_init_work); + queue_work(trace_init_wq, &tracerfs_init_work); } else { tracer_init_tracefs_work_func(NULL); } diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h index de4e6713b84e..e52f259f8945 100644 --- a/kernel/trace/trace.h +++ b/kernel/trace/trace.h @@ -770,6 +770,8 @@ extern unsigned long nsecs_to_usecs(unsigned long nsecs); extern unsigned long tracing_thresh; +extern struct workqueue_struct *trace_init_wq __initdata; + /* PID filtering */ bool trace_find_filtered_pid(struct trace_pid_list *filtered_pids, -- 2.25.1 ^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH v3 2/3] tracing/kprobes: Make setup_boot_kprobe_events() asynchronous 2026-01-27 1:20 [PATCH v3 0/3] Tracing: Accelerate Kernel Boot by Asynchronizing Yaxiong Tian 2026-01-27 1:23 ` [PATCH v3 1/3] tracing: Rename `eval_map_wq` and allow other parts of tracing use it Yaxiong Tian @ 2026-01-27 1:23 ` Yaxiong Tian 2026-01-28 4:49 ` Masami Hiramatsu 2026-01-27 1:23 ` [PATCH v3 3/3] blktrace: Make init_blk_tracer() asynchronous Yaxiong Tian 2 siblings, 1 reply; 8+ messages in thread From: Yaxiong Tian @ 2026-01-27 1:23 UTC (permalink / raw) To: rostedt, axboe, mhiramat, mathieu.desnoyers Cc: linux-block, linux-kernel, linux-trace-kernel, Yaxiong Tian During kernel boot, the setup_boot_kprobe_events() function causes significant delays, increasing overall startup time. The root cause is a lock contention chain: its child function enable_boot_kprobe_events() requires the event_mutex, which is already held by early_event_add_tracer(). early_event_add_tracer() itself is blocked waiting for the trace_event_sem read-write lock, which is held for an extended period by trace_event_update_all(). To resolve this, we have moved the execution of setup_boot_kprobe_events() to the trace_init_wq workqueue, allowing it to run asynchronously. Signed-off-by: Yaxiong Tian <tianyaxiong@kylinos.cn> --- kernel/trace/trace_kprobe.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c index 9953506370a5..4c6621f02696 100644 --- a/kernel/trace/trace_kprobe.c +++ b/kernel/trace/trace_kprobe.c @@ -2031,6 +2031,13 @@ static __init int init_kprobe_trace_early(void) } core_initcall(init_kprobe_trace_early); +static struct work_struct kprobe_trace_work __initdata; + +static void __init kprobe_trace_works_func(struct work_struct *work) +{ + setup_boot_kprobe_events(); +} + /* Make a tracefs interface for controlling probe points */ static __init int init_kprobe_trace(void) { @@ -2048,7 +2055,12 @@ static __init int init_kprobe_trace(void) trace_create_file("kprobe_profile", TRACE_MODE_READ, NULL, NULL, &kprobe_profile_ops); - setup_boot_kprobe_events(); + if (trace_init_wq) { + INIT_WORK(&kprobe_trace_work, kprobe_trace_works_func); + queue_work(trace_init_wq, &kprobe_trace_work); + } else { + setup_boot_kprobe_events(); + } return 0; } -- 2.25.1 ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH v3 2/3] tracing/kprobes: Make setup_boot_kprobe_events() asynchronous 2026-01-27 1:23 ` [PATCH v3 2/3] tracing/kprobes: Make setup_boot_kprobe_events() asynchronous Yaxiong Tian @ 2026-01-28 4:49 ` Masami Hiramatsu 2026-01-28 7:24 ` Yaxiong Tian 0 siblings, 1 reply; 8+ messages in thread From: Masami Hiramatsu @ 2026-01-28 4:49 UTC (permalink / raw) To: Yaxiong Tian Cc: rostedt, axboe, mathieu.desnoyers, linux-block, linux-kernel, linux-trace-kernel On Tue, 27 Jan 2026 09:23:12 +0800 Yaxiong Tian <tianyaxiong@kylinos.cn> wrote: > During kernel boot, the setup_boot_kprobe_events() function causes > significant delays, increasing overall startup time. > > The root cause is a lock contention chain: its child function > enable_boot_kprobe_events() requires the event_mutex, which is > already held by early_event_add_tracer(). early_event_add_tracer() > itself is blocked waiting for the trace_event_sem read-write lock, > which is held for an extended period by trace_event_update_all(). > > To resolve this, we have moved the execution of > setup_boot_kprobe_events() to the trace_init_wq workqueue, allowing > it to run asynchronously. > > Signed-off-by: Yaxiong Tian <tianyaxiong@kylinos.cn> > --- > kernel/trace/trace_kprobe.c | 14 +++++++++++++- > 1 file changed, 13 insertions(+), 1 deletion(-) > > diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c > index 9953506370a5..4c6621f02696 100644 > --- a/kernel/trace/trace_kprobe.c > +++ b/kernel/trace/trace_kprobe.c > @@ -2031,6 +2031,13 @@ static __init int init_kprobe_trace_early(void) > } > core_initcall(init_kprobe_trace_early); > > +static struct work_struct kprobe_trace_work __initdata; > + > +static void __init kprobe_trace_works_func(struct work_struct *work) > +{ > + setup_boot_kprobe_events(); > +} > + > /* Make a tracefs interface for controlling probe points */ > static __init int init_kprobe_trace(void) > { > @@ -2048,7 +2055,12 @@ static __init int init_kprobe_trace(void) > trace_create_file("kprobe_profile", TRACE_MODE_READ, > NULL, NULL, &kprobe_profile_ops); > > - setup_boot_kprobe_events(); > + if (trace_init_wq) { > + INIT_WORK(&kprobe_trace_work, kprobe_trace_works_func); > + queue_work(trace_init_wq, &kprobe_trace_work); Hmm, this queue_work is not required if kprobe_boot_events_buf[] is empty. We should check it because most of the time we don't need it. Also, deferring initialization makes it indeterminate when this tracing will begin. For kprobe event use case, I think setup_boot_kprobe_events() should check the kprobe_boot_events_buf is empty at first. But I think Yaxiong use case happens when you are using kprobe events via cmdline, is that correct? I think introducing "async" cmdline option is more preferable. BTW, I found that the kprobe events from kernel cmdline will be made after boot-time tracing from bootconfig. Maybe it should be run this earlier timing too. Thank you, > + } else { > + setup_boot_kprobe_events(); > + } > > return 0; > } > -- > 2.25.1 > -- Masami Hiramatsu (Google) <mhiramat@kernel.org> ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH v3 2/3] tracing/kprobes: Make setup_boot_kprobe_events() asynchronous 2026-01-28 4:49 ` Masami Hiramatsu @ 2026-01-28 7:24 ` Yaxiong Tian 2026-01-28 7:53 ` Masami Hiramatsu 0 siblings, 1 reply; 8+ messages in thread From: Yaxiong Tian @ 2026-01-28 7:24 UTC (permalink / raw) To: Masami Hiramatsu (Google) Cc: rostedt, axboe, mathieu.desnoyers, linux-block, linux-kernel, linux-trace-kernel 在 2026/1/28 12:49, Masami Hiramatsu (Google) 写道: > On Tue, 27 Jan 2026 09:23:12 +0800 > Yaxiong Tian <tianyaxiong@kylinos.cn> wrote: > >> During kernel boot, the setup_boot_kprobe_events() function causes >> significant delays, increasing overall startup time. >> >> The root cause is a lock contention chain: its child function >> enable_boot_kprobe_events() requires the event_mutex, which is >> already held by early_event_add_tracer(). early_event_add_tracer() >> itself is blocked waiting for the trace_event_sem read-write lock, >> which is held for an extended period by trace_event_update_all(). >> >> To resolve this, we have moved the execution of >> setup_boot_kprobe_events() to the trace_init_wq workqueue, allowing >> it to run asynchronously. >> >> Signed-off-by: Yaxiong Tian <tianyaxiong@kylinos.cn> >> --- >> kernel/trace/trace_kprobe.c | 14 +++++++++++++- >> 1 file changed, 13 insertions(+), 1 deletion(-) >> >> diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c >> index 9953506370a5..4c6621f02696 100644 >> --- a/kernel/trace/trace_kprobe.c >> +++ b/kernel/trace/trace_kprobe.c >> @@ -2031,6 +2031,13 @@ static __init int init_kprobe_trace_early(void) >> } >> core_initcall(init_kprobe_trace_early); >> >> +static struct work_struct kprobe_trace_work __initdata; >> + >> +static void __init kprobe_trace_works_func(struct work_struct *work) >> +{ >> + setup_boot_kprobe_events(); >> +} >> + >> /* Make a tracefs interface for controlling probe points */ >> static __init int init_kprobe_trace(void) >> { >> @@ -2048,7 +2055,12 @@ static __init int init_kprobe_trace(void) >> trace_create_file("kprobe_profile", TRACE_MODE_READ, >> NULL, NULL, &kprobe_profile_ops); >> >> - setup_boot_kprobe_events(); >> + if (trace_init_wq) { >> + INIT_WORK(&kprobe_trace_work, kprobe_trace_works_func); >> + queue_work(trace_init_wq, &kprobe_trace_work); > Hmm, this queue_work is not required if kprobe_boot_events_buf[] is > empty. We should check it because most of the time we don't need it. Yes, I will improve it in the next version. > Also, deferring initialization makes it indeterminate when this > tracing will begin. Indeed, While most scenarios don't need boot-time tracing, and users prioritize boot speed, we must balance the need for deterministic traces with faster startup. > > For kprobe event use case, I think setup_boot_kprobe_events() should > check the kprobe_boot_events_buf is empty at first. But I think Yaxiong > use case happens when you are using kprobe events via cmdline, is that > correct? The issue was identified without enabling kprobe events via the cmdline. The core finding is that asynchronous initialization can drastically cut boot time under different workloads. As blocking occurs elsewhere in the tracing infrastructure beyond just kprobe events, adopting async is a broadly applicable strategy for boot time optimization. > > I think introducing "async" cmdline option is more preferable. Agreed, this works. Users focused on boot speed over early-boot tracing can opt for this parameter to gain a startup performance boost. > > BTW, I found that the kprobe events from kernel cmdline will be made > after boot-time tracing from bootconfig. Maybe it should be run this > earlier timing too. Yes. Additionally, this optimization does not conflict with the current patch series at all. I'll submit the updated patch for the next version promptly. > Thank you, > > >> + } else { >> + setup_boot_kprobe_events(); >> + } >> >> return 0; >> } >> -- >> 2.25.1 >> > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH v3 2/3] tracing/kprobes: Make setup_boot_kprobe_events() asynchronous 2026-01-28 7:24 ` Yaxiong Tian @ 2026-01-28 7:53 ` Masami Hiramatsu 2026-01-28 8:39 ` Yaxiong Tian 0 siblings, 1 reply; 8+ messages in thread From: Masami Hiramatsu @ 2026-01-28 7:53 UTC (permalink / raw) To: Yaxiong Tian Cc: rostedt, axboe, mathieu.desnoyers, linux-block, linux-kernel, linux-trace-kernel On Wed, 28 Jan 2026 15:24:15 +0800 Yaxiong Tian <tianyaxiong@kylinos.cn> wrote: > > 在 2026/1/28 12:49, Masami Hiramatsu (Google) 写道: > > On Tue, 27 Jan 2026 09:23:12 +0800 > > Yaxiong Tian <tianyaxiong@kylinos.cn> wrote: > > > >> During kernel boot, the setup_boot_kprobe_events() function causes > >> significant delays, increasing overall startup time. > >> > >> The root cause is a lock contention chain: its child function > >> enable_boot_kprobe_events() requires the event_mutex, which is > >> already held by early_event_add_tracer(). early_event_add_tracer() > >> itself is blocked waiting for the trace_event_sem read-write lock, > >> which is held for an extended period by trace_event_update_all(). > >> > >> To resolve this, we have moved the execution of > >> setup_boot_kprobe_events() to the trace_init_wq workqueue, allowing > >> it to run asynchronously. > >> > >> Signed-off-by: Yaxiong Tian <tianyaxiong@kylinos.cn> > >> --- > >> kernel/trace/trace_kprobe.c | 14 +++++++++++++- > >> 1 file changed, 13 insertions(+), 1 deletion(-) > >> > >> diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c > >> index 9953506370a5..4c6621f02696 100644 > >> --- a/kernel/trace/trace_kprobe.c > >> +++ b/kernel/trace/trace_kprobe.c > >> @@ -2031,6 +2031,13 @@ static __init int init_kprobe_trace_early(void) > >> } > >> core_initcall(init_kprobe_trace_early); > >> > >> +static struct work_struct kprobe_trace_work __initdata; > >> + > >> +static void __init kprobe_trace_works_func(struct work_struct *work) > >> +{ > >> + setup_boot_kprobe_events(); > >> +} > >> + > >> /* Make a tracefs interface for controlling probe points */ > >> static __init int init_kprobe_trace(void) > >> { > >> @@ -2048,7 +2055,12 @@ static __init int init_kprobe_trace(void) > >> trace_create_file("kprobe_profile", TRACE_MODE_READ, > >> NULL, NULL, &kprobe_profile_ops); > >> > >> - setup_boot_kprobe_events(); > >> + if (trace_init_wq) { > >> + INIT_WORK(&kprobe_trace_work, kprobe_trace_works_func); > >> + queue_work(trace_init_wq, &kprobe_trace_work); > > Hmm, this queue_work is not required if kprobe_boot_events_buf[] is > > empty. We should check it because most of the time we don't need it. > Yes, I will improve it in the next version. > > Also, deferring initialization makes it indeterminate when this > > tracing will begin. > Indeed, While most scenarios don't need boot-time tracing, and users > prioritize boot speed, we must balance the need for deterministic traces > with faster startup. I just wonder why don't you define kprobe events after boot (e.g. from init script) instead of kernel cmdline. Using cmdline means it will be used for tracing kernel boot. - tracing kernel boot -> use kernel cmdline (synchronous) - tracing user boot -> use tracefs (asynchronous) > > For kprobe event use case, I think setup_boot_kprobe_events() should > > check the kprobe_boot_events_buf is empty at first. But I think Yaxiong > > use case happens when you are using kprobe events via cmdline, is that > > correct? > The issue was identified without enabling kprobe events via the cmdline. Interesting. So is it fixed by another patch [1]? [1] https://lore.kernel.org/all/20260127053848.108473-1-sunliming@linux.dev/ > The core finding is that asynchronous initialization can drastically cut > boot time under different workloads. As blocking occurs elsewhere in the > tracing infrastructure beyond just kprobe events, adopting async is a > broadly applicable strategy for boot time optimization. Yes, but it is also possible to set it up from user space, because that user process can work asynchronously. We can make the ftrace initialization async to accelerate boot time, but that means it is not useful for tracing kernel boot. > > > > I think introducing "async" cmdline option is more preferable. > > Agreed, this works. Users focused on boot speed over early-boot tracing > can opt for this parameter to gain a startup performance boost. Yeah, that is an option. Anyway, basically, users have another option to setup ftrace after boot user space asynchronously. That is my recommendation for such use case. > > > > > BTW, I found that the kprobe events from kernel cmdline will be made > > after boot-time tracing from bootconfig. Maybe it should be run this > > earlier timing too. > > Yes. Additionally, this optimization does not conflict with the current > patch series at all. > > I'll submit the updated patch for the next version promptly. OK. Thank you, > > > Thank you, > > > > > >> + } else { > >> + setup_boot_kprobe_events(); > >> + } > >> > >> return 0; > >> } > >> -- > >> 2.25.1 > >> > > -- Masami Hiramatsu (Google) <mhiramat@kernel.org> ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH v3 2/3] tracing/kprobes: Make setup_boot_kprobe_events() asynchronous 2026-01-28 7:53 ` Masami Hiramatsu @ 2026-01-28 8:39 ` Yaxiong Tian 0 siblings, 0 replies; 8+ messages in thread From: Yaxiong Tian @ 2026-01-28 8:39 UTC (permalink / raw) To: Masami Hiramatsu (Google) Cc: rostedt, axboe, mathieu.desnoyers, linux-block, linux-kernel, linux-trace-kernel 在 2026/1/28 15:53, Masami Hiramatsu (Google) 写道: > On Wed, 28 Jan 2026 15:24:15 +0800 > Yaxiong Tian <tianyaxiong@kylinos.cn> wrote: > >> 在 2026/1/28 12:49, Masami Hiramatsu (Google) 写道: >>> On Tue, 27 Jan 2026 09:23:12 +0800 >>> Yaxiong Tian <tianyaxiong@kylinos.cn> wrote: >>> >>>> During kernel boot, the setup_boot_kprobe_events() function causes >>>> significant delays, increasing overall startup time. >>>> >>>> The root cause is a lock contention chain: its child function >>>> enable_boot_kprobe_events() requires the event_mutex, which is >>>> already held by early_event_add_tracer(). early_event_add_tracer() >>>> itself is blocked waiting for the trace_event_sem read-write lock, >>>> which is held for an extended period by trace_event_update_all(). >>>> >>>> To resolve this, we have moved the execution of >>>> setup_boot_kprobe_events() to the trace_init_wq workqueue, allowing >>>> it to run asynchronously. >>>> >>>> Signed-off-by: Yaxiong Tian <tianyaxiong@kylinos.cn> >>>> --- >>>> kernel/trace/trace_kprobe.c | 14 +++++++++++++- >>>> 1 file changed, 13 insertions(+), 1 deletion(-) >>>> >>>> diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c >>>> index 9953506370a5..4c6621f02696 100644 >>>> --- a/kernel/trace/trace_kprobe.c >>>> +++ b/kernel/trace/trace_kprobe.c >>>> @@ -2031,6 +2031,13 @@ static __init int init_kprobe_trace_early(void) >>>> } >>>> core_initcall(init_kprobe_trace_early); >>>> >>>> +static struct work_struct kprobe_trace_work __initdata; >>>> + >>>> +static void __init kprobe_trace_works_func(struct work_struct *work) >>>> +{ >>>> + setup_boot_kprobe_events(); >>>> +} >>>> + >>>> /* Make a tracefs interface for controlling probe points */ >>>> static __init int init_kprobe_trace(void) >>>> { >>>> @@ -2048,7 +2055,12 @@ static __init int init_kprobe_trace(void) >>>> trace_create_file("kprobe_profile", TRACE_MODE_READ, >>>> NULL, NULL, &kprobe_profile_ops); >>>> >>>> - setup_boot_kprobe_events(); >>>> + if (trace_init_wq) { >>>> + INIT_WORK(&kprobe_trace_work, kprobe_trace_works_func); >>>> + queue_work(trace_init_wq, &kprobe_trace_work); >>> Hmm, this queue_work is not required if kprobe_boot_events_buf[] is >>> empty. We should check it because most of the time we don't need it. >> Yes, I will improve it in the next version. >>> Also, deferring initialization makes it indeterminate when this >>> tracing will begin. >> Indeed, While most scenarios don't need boot-time tracing, and users >> prioritize boot speed, we must balance the need for deterministic traces >> with faster startup. > I just wonder why don't you define kprobe events after boot (e.g. > from init script) instead of kernel cmdline. Using cmdline means > it will be used for tracing kernel boot. > > - tracing kernel boot -> use kernel cmdline (synchronous) > - tracing user boot -> use tracefs (asynchronous) I was actually working on a boot optimization task when I found that my machine was getting blocked on trace-related operations. > >>> For kprobe event use case, I think setup_boot_kprobe_events() should >>> check the kprobe_boot_events_buf is empty at first. But I think Yaxiong >>> use case happens when you are using kprobe events via cmdline, is that >>> correct? >> The issue was identified without enabling kprobe events via the cmdline. > Interesting. So is it fixed by another patch [1]? > > [1] https://lore.kernel.org/all/20260127053848.108473-1-sunliming@linux.dev/ > Yes, specifically looking at trace_kprobe in isolation, that issue is indeed resolved. However, the process still ends up blocking the initialization of blktrace later on. >> The core finding is that asynchronous initialization can drastically cut >> boot time under different workloads. As blocking occurs elsewhere in the >> tracing infrastructure beyond just kprobe events, adopting async is a >> broadly applicable strategy for boot time optimization. > Yes, but it is also possible to set it up from user space, because that > user process can work asynchronously. > We can make the ftrace initialization async to accelerate boot time, but > that means it is not useful for tracing kernel boot. > >>> I think introducing "async" cmdline option is more preferable. >> Agreed, this works. Users focused on boot speed over early-boot tracing >> can opt for this parameter to gain a startup performance boost. > Yeah, that is an option. Anyway, basically, users have another option to > setup ftrace after boot user space asynchronously. That is my > recommendation for such use case. Yes, that's what I normally do. > >>> BTW, I found that the kprobe events from kernel cmdline will be made >>> after boot-time tracing from bootconfig. Maybe it should be run this >>> earlier timing too. >> Yes. Additionally, this optimization does not conflict with the current >> patch series at all. >> >> I'll submit the updated patch for the next version promptly. > OK. > > Thank you, > >>> Thank you, >>> >>> >>>> + } else { >>>> + setup_boot_kprobe_events(); >>>> + } >>>> >>>> return 0; >>>> } >>>> -- >>>> 2.25.1 >>>> > ^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH v3 3/3] blktrace: Make init_blk_tracer() asynchronous 2026-01-27 1:20 [PATCH v3 0/3] Tracing: Accelerate Kernel Boot by Asynchronizing Yaxiong Tian 2026-01-27 1:23 ` [PATCH v3 1/3] tracing: Rename `eval_map_wq` and allow other parts of tracing use it Yaxiong Tian 2026-01-27 1:23 ` [PATCH v3 2/3] tracing/kprobes: Make setup_boot_kprobe_events() asynchronous Yaxiong Tian @ 2026-01-27 1:23 ` Yaxiong Tian 2 siblings, 0 replies; 8+ messages in thread From: Yaxiong Tian @ 2026-01-27 1:23 UTC (permalink / raw) To: rostedt, axboe, mhiramat, mathieu.desnoyers Cc: linux-block, linux-kernel, linux-trace-kernel, Yaxiong Tian The init_blk_tracer() function causes significant boot delay as it waits for the trace_event_sem lock held by trace_event_update_all(). Specifically, its child function register_trace_event() requires this lock, which is occupied for an extended period during boot. To mitigate this contention, we have moved init_blk_tracer() to the trace_init_wq workqueue, allowing it to execute asynchronously and prevent blocking the main boot thread. Signed-off-by: Yaxiong Tian <tianyaxiong@kylinos.cn> --- kernel/trace/blktrace.c | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/kernel/trace/blktrace.c b/kernel/trace/blktrace.c index d031c8d80be4..d611cd1f02ef 100644 --- a/kernel/trace/blktrace.c +++ b/kernel/trace/blktrace.c @@ -1832,7 +1832,9 @@ static struct trace_event trace_blk_event = { .funcs = &trace_blk_event_funcs, }; -static int __init init_blk_tracer(void) +static struct work_struct blktrace_works __initdata; + +static int __init __init_blk_tracer(void) { if (!register_trace_event(&trace_blk_event)) { pr_warn("Warning: could not register block events\n"); @@ -1852,6 +1854,25 @@ static int __init init_blk_tracer(void) return 0; } +static void __init blktrace_works_func(struct work_struct *work) +{ + __init_blk_tracer(); +} + +static int __init init_blk_tracer(void) +{ + int ret = 0; + + if (trace_init_wq) { + INIT_WORK(&blktrace_works, blktrace_works_func); + queue_work(trace_init_wq, &blktrace_works); + } else { + ret = __init_blk_tracer(); + } + + return ret; +} + device_initcall(init_blk_tracer); static int blk_trace_remove_queue(struct request_queue *q) -- 2.25.1 ^ permalink raw reply related [flat|nested] 8+ messages in thread
end of thread, other threads:[~2026-01-28 8:39 UTC | newest] Thread overview: 8+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-01-27 1:20 [PATCH v3 0/3] Tracing: Accelerate Kernel Boot by Asynchronizing Yaxiong Tian 2026-01-27 1:23 ` [PATCH v3 1/3] tracing: Rename `eval_map_wq` and allow other parts of tracing use it Yaxiong Tian 2026-01-27 1:23 ` [PATCH v3 2/3] tracing/kprobes: Make setup_boot_kprobe_events() asynchronous Yaxiong Tian 2026-01-28 4:49 ` Masami Hiramatsu 2026-01-28 7:24 ` Yaxiong Tian 2026-01-28 7:53 ` Masami Hiramatsu 2026-01-28 8:39 ` Yaxiong Tian 2026-01-27 1:23 ` [PATCH v3 3/3] blktrace: Make init_blk_tracer() asynchronous Yaxiong Tian
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox