* [PATCH v5 1/3] tracing: Rename `eval_map_wq` and allow other parts of tracing use it
2026-02-04 1:51 [PATCH v5 0/3] Tracing: Accelerate Kernel Boot by Asynchronizing Yaxiong Tian
@ 2026-02-04 1:53 ` Yaxiong Tian
2026-02-04 1:53 ` [PATCH v5 2/3] blktrace: Make init_blk_tracer() asynchronous Yaxiong Tian
` (2 subsequent siblings)
3 siblings, 0 replies; 6+ messages in thread
From: Yaxiong Tian @ 2026-02-04 1:53 UTC (permalink / raw)
To: axboe, rostedt, mhiramat, mathieu.desnoyers
Cc: linux-block, linux-kernel, linux-trace-kernel, Yaxiong Tian
The eval_map_work_func() function, though queued in eval_map_wq,
holds the trace_event_sem read-write lock for a long time during
kernel boot. This causes blocking issues for other functions.
Rename eval_map_wq to trace_init_wq and make it global, thereby
allowing other parts of tracing to schedule work on this queue
asynchronously and avoiding blockage of the main boot thread.
Suggested-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Yaxiong Tian <tianyaxiong@kylinos.cn>
---
kernel/trace/trace.c | 18 +++++++++---------
kernel/trace/trace.h | 1 +
2 files changed, 10 insertions(+), 9 deletions(-)
diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index b1cb30a7b83d..01df88e77818 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -10774,7 +10774,7 @@ int tracing_init_dentry(void)
extern struct trace_eval_map *__start_ftrace_eval_maps[];
extern struct trace_eval_map *__stop_ftrace_eval_maps[];
-static struct workqueue_struct *eval_map_wq __initdata;
+struct workqueue_struct *trace_init_wq __initdata;
static struct work_struct eval_map_work __initdata;
static struct work_struct tracerfs_init_work __initdata;
@@ -10790,15 +10790,15 @@ static int __init trace_eval_init(void)
{
INIT_WORK(&eval_map_work, eval_map_work_func);
- eval_map_wq = alloc_workqueue("eval_map_wq", WQ_UNBOUND, 0);
- if (!eval_map_wq) {
- pr_err("Unable to allocate eval_map_wq\n");
+ trace_init_wq = alloc_workqueue("trace_init_wq", WQ_UNBOUND, 0);
+ if (!trace_init_wq) {
+ pr_err("Unable to allocate trace_init_wq\n");
/* Do work here */
eval_map_work_func(&eval_map_work);
return -ENOMEM;
}
- queue_work(eval_map_wq, &eval_map_work);
+ queue_work(trace_init_wq, &eval_map_work);
return 0;
}
@@ -10807,8 +10807,8 @@ subsys_initcall(trace_eval_init);
static int __init trace_eval_sync(void)
{
/* Make sure the eval map updates are finished */
- if (eval_map_wq)
- destroy_workqueue(eval_map_wq);
+ if (trace_init_wq)
+ destroy_workqueue(trace_init_wq);
return 0;
}
@@ -10969,9 +10969,9 @@ static __init int tracer_init_tracefs(void)
if (ret)
return 0;
- if (eval_map_wq) {
+ if (trace_init_wq) {
INIT_WORK(&tracerfs_init_work, tracer_init_tracefs_work_func);
- queue_work(eval_map_wq, &tracerfs_init_work);
+ queue_work(trace_init_wq, &tracerfs_init_work);
} else {
tracer_init_tracefs_work_func(NULL);
}
diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h
index de4e6713b84e..9e8d52503618 100644
--- a/kernel/trace/trace.h
+++ b/kernel/trace/trace.h
@@ -769,6 +769,7 @@ extern cpumask_var_t __read_mostly tracing_buffer_mask;
extern unsigned long nsecs_to_usecs(unsigned long nsecs);
extern unsigned long tracing_thresh;
+extern struct workqueue_struct *trace_init_wq __initdata;
/* PID filtering */
--
2.25.1
^ permalink raw reply related [flat|nested] 6+ messages in thread* [PATCH v5 2/3] blktrace: Make init_blk_tracer() asynchronous
2026-02-04 1:51 [PATCH v5 0/3] Tracing: Accelerate Kernel Boot by Asynchronizing Yaxiong Tian
2026-02-04 1:53 ` [PATCH v5 1/3] tracing: Rename `eval_map_wq` and allow other parts of tracing use it Yaxiong Tian
@ 2026-02-04 1:53 ` Yaxiong Tian
2026-02-04 1:54 ` [PATCH v5 3/3] tracing/kprobes: Skip setup_boot_kprobe_events() when no cmdline event Yaxiong Tian
2026-02-04 10:15 ` [PATCH v5 0/3] Tracing: Accelerate Kernel Boot by Asynchronizing Masami Hiramatsu
3 siblings, 0 replies; 6+ messages in thread
From: Yaxiong Tian @ 2026-02-04 1:53 UTC (permalink / raw)
To: axboe, rostedt, mhiramat, mathieu.desnoyers
Cc: linux-block, linux-kernel, linux-trace-kernel, Yaxiong Tian
The init_blk_tracer() function causes significant boot delay as it
waits for the trace_event_sem lock held by trace_event_update_all().
Specifically, its child function register_trace_event() requires
this lock, which is occupied for an extended period during boot.
To resolve this, the execution of primary init_blk_tracer() is moved
to the trace_init_wq workqueue, allowing it to run asynchronously,
and prevent blocking the main boot thread.
Signed-off-by: Yaxiong Tian <tianyaxiong@kylinos.cn>
---
kernel/trace/blktrace.c | 23 ++++++++++++++++++++++-
1 file changed, 22 insertions(+), 1 deletion(-)
diff --git a/kernel/trace/blktrace.c b/kernel/trace/blktrace.c
index d031c8d80be4..d611cd1f02ef 100644
--- a/kernel/trace/blktrace.c
+++ b/kernel/trace/blktrace.c
@@ -1832,7 +1832,9 @@ static struct trace_event trace_blk_event = {
.funcs = &trace_blk_event_funcs,
};
-static int __init init_blk_tracer(void)
+static struct work_struct blktrace_works __initdata;
+
+static int __init __init_blk_tracer(void)
{
if (!register_trace_event(&trace_blk_event)) {
pr_warn("Warning: could not register block events\n");
@@ -1852,6 +1854,25 @@ static int __init init_blk_tracer(void)
return 0;
}
+static void __init blktrace_works_func(struct work_struct *work)
+{
+ __init_blk_tracer();
+}
+
+static int __init init_blk_tracer(void)
+{
+ int ret = 0;
+
+ if (trace_init_wq) {
+ INIT_WORK(&blktrace_works, blktrace_works_func);
+ queue_work(trace_init_wq, &blktrace_works);
+ } else {
+ ret = __init_blk_tracer();
+ }
+
+ return ret;
+}
+
device_initcall(init_blk_tracer);
static int blk_trace_remove_queue(struct request_queue *q)
--
2.25.1
^ permalink raw reply related [flat|nested] 6+ messages in thread* [PATCH v5 3/3] tracing/kprobes: Skip setup_boot_kprobe_events() when no cmdline event
2026-02-04 1:51 [PATCH v5 0/3] Tracing: Accelerate Kernel Boot by Asynchronizing Yaxiong Tian
2026-02-04 1:53 ` [PATCH v5 1/3] tracing: Rename `eval_map_wq` and allow other parts of tracing use it Yaxiong Tian
2026-02-04 1:53 ` [PATCH v5 2/3] blktrace: Make init_blk_tracer() asynchronous Yaxiong Tian
@ 2026-02-04 1:54 ` Yaxiong Tian
2026-02-04 10:15 ` [PATCH v5 0/3] Tracing: Accelerate Kernel Boot by Asynchronizing Masami Hiramatsu
3 siblings, 0 replies; 6+ messages in thread
From: Yaxiong Tian @ 2026-02-04 1:54 UTC (permalink / raw)
To: axboe, rostedt, mhiramat, mathieu.desnoyers
Cc: linux-block, linux-kernel, linux-trace-kernel, Yaxiong Tian
When the 'kprobe_event=' kernel command-line parameter is not provided,
there is no need to execute setup_boot_kprobe_events().
This change optimizes the initialization function init_kprobe_trace()
by skipping unnecessary work and effectively prevents potential blocking
that could arise from contention on the event_mutex lock in subsequent
operations.
Signed-off-by: Yaxiong Tian <tianyaxiong@kylinos.cn>
---
kernel/trace/trace_kprobe.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c
index 9953506370a5..89d2740f7bb5 100644
--- a/kernel/trace/trace_kprobe.c
+++ b/kernel/trace/trace_kprobe.c
@@ -2048,6 +2048,10 @@ static __init int init_kprobe_trace(void)
trace_create_file("kprobe_profile", TRACE_MODE_READ,
NULL, NULL, &kprobe_profile_ops);
+ /* If no 'kprobe_event=' cmd is provided, return directly. */
+ if (kprobe_boot_events_buf[0] == '\0')
+ return 0;
+
setup_boot_kprobe_events();
return 0;
--
2.25.1
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: [PATCH v5 0/3] Tracing: Accelerate Kernel Boot by Asynchronizing
2026-02-04 1:51 [PATCH v5 0/3] Tracing: Accelerate Kernel Boot by Asynchronizing Yaxiong Tian
` (2 preceding siblings ...)
2026-02-04 1:54 ` [PATCH v5 3/3] tracing/kprobes: Skip setup_boot_kprobe_events() when no cmdline event Yaxiong Tian
@ 2026-02-04 10:15 ` Masami Hiramatsu
2026-02-06 20:25 ` Steven Rostedt
3 siblings, 1 reply; 6+ messages in thread
From: Masami Hiramatsu @ 2026-02-04 10:15 UTC (permalink / raw)
To: Yaxiong Tian
Cc: axboe, rostedt, mathieu.desnoyers, linux-block, linux-kernel,
linux-trace-kernel
On Wed, 4 Feb 2026 09:51:03 +0800
Yaxiong Tian <tianyaxiong@kylinos.cn> wrote:
> On my ARM64 platform, I observed that certain tracing module
> initializations run for up to 200ms—for example, init_kprobe_trace().
> Analysis reveals the root cause: the execution flow eval_map_work_func()
> →trace_event_update_with_eval_map()→trace_event_update_all()
> is highly time-consuming. Although this flow is placed in eval_map_wq
> for asynchronous execution, it holds the trace_event_sem lock, causing
> other modules to be blocked either directly or indirectly. Also in
> init_blk_tracer(), this functions require trace_event_sem device_initcall.
>
> To resolve this issue, I rename `eval_map_wq` and make it global and moved
> init_blk_tracer that are related to this lock to run asynchronously on this
> workqueue. Also check for kprobe_event= grub parameter; if not provided,
> init_kprobe_trace() returns directly. After optimization, boot time is
> reduced by approximately 200ms.
>
>
> Based on my analysis and testing, I've identified that only these two
> locations significantly impact timing. Other initcall_* functions do not
> exhibit relevant lock contention.
>
> A brief summary of the test results is as follows:
> Before this PATCHS:
> [ 0.224933] calling init_kprobe_trace+0x0/0xe0 @ 1
> [ 0.455016] initcall init_kprobe_trace+0x0/0xe0 returned 0 after 230080 usecs
>
> Only opt setup_boot_kprobe_events() can see:
> [ 0.258609] calling init_blk_tracer+0x0/0x68 @ 1
> [ 0.454991] initcall init_blk_tracer+0x0/0x68 returned 0 after 196377 usecs
>
> After this PATCHS:
> [ 0.224940] calling init_kprobe_trace+0x0/0xe0 @ 1
> [ 0.224946] initcall init_kprobe_trace+0x0/0xe0 returned 0 after 3 usecs
> skip --------
> [ 0.264835] calling init_blk_tracer+0x0/0x68 @ 1
> [ 0.264841] initcall init_blk_tracer+0x0/0x68 returned 0 after 2 usecs
Looks good to me.
Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Thanks!
>
> ---
> Changes in v2:
> - Rename eval_map_wq to trace_init_wq.
> Changes in v3:
> - Opt PATCH 1/3 commit
> Changes in v4:
> - add trace_async_init boot parameter in patch2
> - add init_kprobe_trace's skip logic in patch3
> - add Suggested-by tag
> - Other synchronous optimizations related to trace_async_init
> https://lore.kernel.org/all/20260128125117.1704853-1-tianyaxiong@kylinos.cn/
> Changes in v5:
> - remove trace_async_init boot parameter (patch2 v4)
> - remove make Make setup_boot_kprobe_events() asynchronous (patch4 v4)
> - Adjusted the patch sequence.
>
>
> Yaxiong Tian (3):
> tracing: Rename `eval_map_wq` and allow other parts of tracing use it
> blktrace: Make init_blk_tracer() asynchronous
> tracing/kprobes: Skip setup_boot_kprobe_events() when no cmdline event
>
> kernel/trace/blktrace.c | 23 ++++++++++++++++++++++-
> kernel/trace/trace.c | 18 +++++++++---------
> kernel/trace/trace.h | 1 +
> kernel/trace/trace_kprobe.c | 4 ++++
> 4 files changed, 36 insertions(+), 10 deletions(-)
>
> --
> 2.25.1
>
>
--
Masami Hiramatsu (Google) <mhiramat@kernel.org>
^ permalink raw reply [flat|nested] 6+ messages in thread