[PATCH v5 0/3] Tracing: Accelerate Kernel Boot by Asynchronizing

[PATCH v5 0/3] Tracing: Accelerate Kernel Boot by Asynchronizing

[Yaxiong Tian]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=y, Size: 2745 bytes --]

On my ARM64 platform, I observed that certain tracing module
initializations run for up to 200ms—for example, init_kprobe_trace().
Analysis reveals the root cause: the execution flow eval_map_work_func()
→trace_event_update_with_eval_map()→trace_event_update_all()
is highly time-consuming. Although this flow is placed in eval_map_wq
for asynchronous execution, it holds the trace_event_sem lock, causing
other modules to be blocked either directly or indirectly. Also in
init_blk_tracer(), this functions require trace_event_sem device_initcall.

To resolve this issue, I rename `eval_map_wq` and make it global and moved
init_blk_tracer that are related to this lock to run asynchronously on this
workqueue. Also check for kprobe_event= grub parameter; if not provided,
init_kprobe_trace() returns directly. After optimization, boot time is
reduced by approximately 200ms.


Based on my analysis and testing, I've identified that only these two
locations significantly impact timing. Other initcall_* functions do not
exhibit relevant lock contention.

A brief summary of the test results is as follows:
Before this PATCHS:
[    0.224933] calling  init_kprobe_trace+0x0/0xe0 @ 1
[    0.455016] initcall init_kprobe_trace+0x0/0xe0 returned 0 after 230080 usecs

Only opt setup_boot_kprobe_events() can see:
[    0.258609] calling  init_blk_tracer+0x0/0x68 @ 1
[    0.454991] initcall init_blk_tracer+0x0/0x68 returned 0 after 196377 usecs

After this PATCHS:
[    0.224940] calling  init_kprobe_trace+0x0/0xe0 @ 1
[    0.224946] initcall init_kprobe_trace+0x0/0xe0 returned 0 after 3 usecs
skip --------
[    0.264835] calling  init_blk_tracer+0x0/0x68 @ 1
[    0.264841] initcall init_blk_tracer+0x0/0x68 returned 0 after 2 usecs

---
Changes in v2:
- Rename eval_map_wq to trace_init_wq.
Changes in v3:
- Opt PATCH 1/3 commit
Changes in v4:
- add trace_async_init boot parameter in patch2
- add init_kprobe_trace's skip logic in patch3
- add Suggested-by tag 
- Other synchronous optimizations related to trace_async_init
https://lore.kernel.org/all/20260128125117.1704853-1-tianyaxiong@kylinos.cn/
Changes in v5:
- remove trace_async_init boot parameter (patch2 v4)
- remove make  Make setup_boot_kprobe_events() asynchronous (patch4 v4)
- Adjusted the patch sequence.


Yaxiong Tian (3):
  tracing: Rename `eval_map_wq` and allow other parts of tracing use it
  blktrace: Make init_blk_tracer() asynchronous
  tracing/kprobes: Skip setup_boot_kprobe_events() when no cmdline event

 kernel/trace/blktrace.c     | 23 ++++++++++++++++++++++-
 kernel/trace/trace.c        | 18 +++++++++---------
 kernel/trace/trace.h        |  1 +
 kernel/trace/trace_kprobe.c |  4 ++++
 4 files changed, 36 insertions(+), 10 deletions(-)

-- 
2.25.1

[PATCH v5 1/3] tracing: Rename `eval_map_wq` and allow other parts of tracing use it

[Yaxiong Tian]

The eval_map_work_func() function, though queued in eval_map_wq,
holds the trace_event_sem read-write lock for a long time during
kernel boot. This causes blocking issues for other functions.

Rename eval_map_wq to trace_init_wq and make it global, thereby
allowing other parts of tracing to schedule work on this queue
asynchronously and avoiding blockage of the main boot thread.

Suggested-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Yaxiong Tian <tianyaxiong@kylinos.cn>
---
 kernel/trace/trace.c | 18 +++++++++---------
 kernel/trace/trace.h |  1 +
 2 files changed, 10 insertions(+), 9 deletions(-)

diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index b1cb30a7b83d..01df88e77818 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -10774,7 +10774,7 @@ int tracing_init_dentry(void)
 extern struct trace_eval_map *__start_ftrace_eval_maps[];
 extern struct trace_eval_map *__stop_ftrace_eval_maps[];
 
-static struct workqueue_struct *eval_map_wq __initdata;
+struct workqueue_struct *trace_init_wq __initdata;
 static struct work_struct eval_map_work __initdata;
 static struct work_struct tracerfs_init_work __initdata;
 
@@ -10790,15 +10790,15 @@ static int __init trace_eval_init(void)
 {
 	INIT_WORK(&eval_map_work, eval_map_work_func);
 
-	eval_map_wq = alloc_workqueue("eval_map_wq", WQ_UNBOUND, 0);
-	if (!eval_map_wq) {
-		pr_err("Unable to allocate eval_map_wq\n");
+	trace_init_wq = alloc_workqueue("trace_init_wq", WQ_UNBOUND, 0);
+	if (!trace_init_wq) {
+		pr_err("Unable to allocate trace_init_wq\n");
 		/* Do work here */
 		eval_map_work_func(&eval_map_work);
 		return -ENOMEM;
 	}
 
-	queue_work(eval_map_wq, &eval_map_work);
+	queue_work(trace_init_wq, &eval_map_work);
 	return 0;
 }
 
@@ -10807,8 +10807,8 @@ subsys_initcall(trace_eval_init);
 static int __init trace_eval_sync(void)
 {
 	/* Make sure the eval map updates are finished */
-	if (eval_map_wq)
-		destroy_workqueue(eval_map_wq);
+	if (trace_init_wq)
+		destroy_workqueue(trace_init_wq);
 	return 0;
 }
 
@@ -10969,9 +10969,9 @@ static __init int tracer_init_tracefs(void)
 	if (ret)
 		return 0;
 
-	if (eval_map_wq) {
+	if (trace_init_wq) {
 		INIT_WORK(&tracerfs_init_work, tracer_init_tracefs_work_func);
-		queue_work(eval_map_wq, &tracerfs_init_work);
+		queue_work(trace_init_wq, &tracerfs_init_work);
 	} else {
 		tracer_init_tracefs_work_func(NULL);
 	}
diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h
index de4e6713b84e..9e8d52503618 100644
--- a/kernel/trace/trace.h
+++ b/kernel/trace/trace.h
@@ -769,6 +769,7 @@ extern cpumask_var_t __read_mostly tracing_buffer_mask;
 extern unsigned long nsecs_to_usecs(unsigned long nsecs);
 
 extern unsigned long tracing_thresh;
+extern struct workqueue_struct *trace_init_wq __initdata;
 
 /* PID filtering */
 
-- 
2.25.1

[PATCH v5 2/3] blktrace: Make init_blk_tracer() asynchronous

[Yaxiong Tian]

The init_blk_tracer() function causes significant boot delay as it
waits for the trace_event_sem lock held by trace_event_update_all().
Specifically, its child function register_trace_event() requires
this lock, which is occupied for an extended period during boot.

To resolve this, the execution of primary init_blk_tracer() is moved
to the trace_init_wq workqueue, allowing it to run asynchronously,
and prevent blocking the main boot thread.

Signed-off-by: Yaxiong Tian <tianyaxiong@kylinos.cn>
---
 kernel/trace/blktrace.c | 23 ++++++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)

diff --git a/kernel/trace/blktrace.c b/kernel/trace/blktrace.c
index d031c8d80be4..d611cd1f02ef 100644
--- a/kernel/trace/blktrace.c
+++ b/kernel/trace/blktrace.c
@@ -1832,7 +1832,9 @@ static struct trace_event trace_blk_event = {
 	.funcs		= &trace_blk_event_funcs,
 };
 
-static int __init init_blk_tracer(void)
+static struct work_struct blktrace_works __initdata;
+
+static int __init __init_blk_tracer(void)
 {
 	if (!register_trace_event(&trace_blk_event)) {
 		pr_warn("Warning: could not register block events\n");
@@ -1852,6 +1854,25 @@ static int __init init_blk_tracer(void)
 	return 0;
 }
 
+static void __init blktrace_works_func(struct work_struct *work)
+{
+	__init_blk_tracer();
+}
+
+static int __init init_blk_tracer(void)
+{
+	int ret = 0;
+
+	if (trace_init_wq) {
+		INIT_WORK(&blktrace_works, blktrace_works_func);
+		queue_work(trace_init_wq, &blktrace_works);
+	} else {
+		ret = __init_blk_tracer();
+	}
+
+	return ret;
+}
+
 device_initcall(init_blk_tracer);
 
 static int blk_trace_remove_queue(struct request_queue *q)
-- 
2.25.1

[PATCH v5 3/3] tracing/kprobes: Skip setup_boot_kprobe_events() when no cmdline event

[Yaxiong Tian]

When the 'kprobe_event=' kernel command-line parameter is not provided,
there is no need to execute setup_boot_kprobe_events().

This change optimizes the initialization function init_kprobe_trace()
by skipping unnecessary work and effectively prevents potential blocking
that could arise from contention on the event_mutex lock in subsequent
operations.

Signed-off-by: Yaxiong Tian <tianyaxiong@kylinos.cn>
---
 kernel/trace/trace_kprobe.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c
index 9953506370a5..89d2740f7bb5 100644
--- a/kernel/trace/trace_kprobe.c
+++ b/kernel/trace/trace_kprobe.c
@@ -2048,6 +2048,10 @@ static __init int init_kprobe_trace(void)
 	trace_create_file("kprobe_profile", TRACE_MODE_READ,
 			  NULL, NULL, &kprobe_profile_ops);
 
+	/* If no 'kprobe_event=' cmd is provided, return directly. */
+	if (kprobe_boot_events_buf[0] == '\0')
+		return 0;
+
 	setup_boot_kprobe_events();
 
 	return 0;
-- 
2.25.1

Re: [PATCH v5 0/3] Tracing: Accelerate Kernel Boot by Asynchronizing

[Masami Hiramatsu]

On Wed,  4 Feb 2026 09:51:03 +0800
Yaxiong Tian <tianyaxiong@kylinos.cn> wrote:

> On my ARM64 platform, I observed that certain tracing module
> initializations run for up to 200ms—for example, init_kprobe_trace().
> Analysis reveals the root cause: the execution flow eval_map_work_func()
> →trace_event_update_with_eval_map()→trace_event_update_all()
> is highly time-consuming. Although this flow is placed in eval_map_wq
> for asynchronous execution, it holds the trace_event_sem lock, causing
> other modules to be blocked either directly or indirectly. Also in
> init_blk_tracer(), this functions require trace_event_sem device_initcall.
> 
> To resolve this issue, I rename `eval_map_wq` and make it global and moved
> init_blk_tracer that are related to this lock to run asynchronously on this
> workqueue. Also check for kprobe_event= grub parameter; if not provided,
> init_kprobe_trace() returns directly. After optimization, boot time is
> reduced by approximately 200ms.
> 
> 
> Based on my analysis and testing, I've identified that only these two
> locations significantly impact timing. Other initcall_* functions do not
> exhibit relevant lock contention.
> 
> A brief summary of the test results is as follows:
> Before this PATCHS:
> [    0.224933] calling  init_kprobe_trace+0x0/0xe0 @ 1
> [    0.455016] initcall init_kprobe_trace+0x0/0xe0 returned 0 after 230080 usecs
> 
> Only opt setup_boot_kprobe_events() can see:
> [    0.258609] calling  init_blk_tracer+0x0/0x68 @ 1
> [    0.454991] initcall init_blk_tracer+0x0/0x68 returned 0 after 196377 usecs
> 
> After this PATCHS:
> [    0.224940] calling  init_kprobe_trace+0x0/0xe0 @ 1
> [    0.224946] initcall init_kprobe_trace+0x0/0xe0 returned 0 after 3 usecs
> skip --------
> [    0.264835] calling  init_blk_tracer+0x0/0x68 @ 1
> [    0.264841] initcall init_blk_tracer+0x0/0x68 returned 0 after 2 usecs

Looks good to me.

Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>

Thanks!

> 
> ---
> Changes in v2:
> - Rename eval_map_wq to trace_init_wq.
> Changes in v3:
> - Opt PATCH 1/3 commit
> Changes in v4:
> - add trace_async_init boot parameter in patch2
> - add init_kprobe_trace's skip logic in patch3
> - add Suggested-by tag 
> - Other synchronous optimizations related to trace_async_init
> https://lore.kernel.org/all/20260128125117.1704853-1-tianyaxiong@kylinos.cn/
> Changes in v5:
> - remove trace_async_init boot parameter (patch2 v4)
> - remove make  Make setup_boot_kprobe_events() asynchronous (patch4 v4)
> - Adjusted the patch sequence.
> 
> 
> Yaxiong Tian (3):
>   tracing: Rename `eval_map_wq` and allow other parts of tracing use it
>   blktrace: Make init_blk_tracer() asynchronous
>   tracing/kprobes: Skip setup_boot_kprobe_events() when no cmdline event
> 
>  kernel/trace/blktrace.c     | 23 ++++++++++++++++++++++-
>  kernel/trace/trace.c        | 18 +++++++++---------
>  kernel/trace/trace.h        |  1 +
>  kernel/trace/trace_kprobe.c |  4 ++++
>  4 files changed, 36 insertions(+), 10 deletions(-)
> 
> -- 
> 2.25.1
> 
> 


-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>

Re: [PATCH v5 0/3] Tracing: Accelerate Kernel Boot by Asynchronizing

[Steven Rostedt]

On Wed, 4 Feb 2026 19:15:52 +0900
Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote:

> Looks good to me.
> 
> Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>

Thanks, I'll add this to tracing/core.

-- Steve