[PATCH 0/5] tracing: Fix tracer options per instance

[PATCH 0/5] tracing: Fix tracer options per instance

[Steven Rostedt]

The tracers (like function and function_graph) have options that modify
how they behave or how they are shown in the trace file. As these options
were created before instances (multiple buffers), the flags variable that
represent these options were global for the tracer.

Now that a tracer may exist in multiple instances, having a global flags
that kinda affect every instance of that tracer does not make sense.
The reason it "kinda" affects them, is that the code that updates an option
for an instance, only does the change for the tracer running in that instance.
If a tracer is already running in another instance, it will not be affected.
But because the option is saved in a global variable, if the tracer is stopped
and restarted in the instance, then the option will take affect.

Since each instance has its own interface to modify the options, the effect
of an option should only affect the tracer for the given instance. This means
that the state of the options for each tracer must be saved for each instance
separately.

Add a new struct tracers field in the trace_array (instance descriptor) that
holds a descriptor that points to a tracer and contains its flags values. This
will be used to see what tracers are available for an instance.

Now when a tracer option is modified for an instance, it only affects the
tracer in that instance and not semi-effects the tracers in other instances.

Steven Rostedt (5):
      tracing: Remove dummy options and flags
      tracing: Have add_tracer_options() error pass up to callers
      tracing: Have tracer option be instance specific
      tracing: Have function tracer define options per instance
      tracing: Have function graph tracer define options per instance

----
 kernel/trace/trace.c                 | 280 ++++++++++++++++++++++-------------
 kernel/trace/trace.h                 |   3 +
 kernel/trace/trace_functions.c       |  10 +-
 kernel/trace/trace_functions_graph.c |  18 ++-
 4 files changed, 198 insertions(+), 113 deletions(-)

[PATCH 1/5] tracing: Remove dummy options and flags

[Steven Rostedt]

From: Steven Rostedt <rostedt@goodmis.org>

When a tracer does not define their own flags, dummy options and flags are
used so that the values are always valid. There's not that many locations
that reference these values so having dummy versions just complicates the
code. Remove the dummy values and just check for NULL when appropriate.

Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
---
 kernel/trace/trace.c | 48 +++++++++++++++-----------------------------
 1 file changed, 16 insertions(+), 32 deletions(-)

diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index 0e822db5d9e4..afeaa9a164e9 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -94,17 +94,6 @@ static bool tracepoint_printk_stop_on_boot __initdata;
 static bool traceoff_after_boot __initdata;
 static DEFINE_STATIC_KEY_FALSE(tracepoint_printk_key);
 
-/* For tracers that don't implement custom flags */
-static struct tracer_opt dummy_tracer_opt[] = {
-	{ }
-};
-
-static int
-dummy_set_flag(struct trace_array *tr, u32 old_flags, u32 bit, int set)
-{
-	return 0;
-}
-
 /*
  * To prevent the comm cache from being overwritten when no
  * tracing is active, only save the comm when a trace event
@@ -2356,23 +2345,9 @@ int __init register_tracer(struct tracer *type)
 		}
 	}
 
-	if (!type->set_flag)
-		type->set_flag = &dummy_set_flag;
-	if (!type->flags) {
-		/*allocate a dummy tracer_flags*/
-		type->flags = kmalloc(sizeof(*type->flags), GFP_KERNEL);
-		if (!type->flags) {
-			ret = -ENOMEM;
-			goto out;
-		}
-		type->flags->val = 0;
-		type->flags->opts = dummy_tracer_opt;
-	} else
-		if (!type->flags->opts)
-			type->flags->opts = dummy_tracer_opt;
-
 	/* store the tracer for __set_tracer_option */
-	type->flags->trace = type;
+	if (type->flags)
+		type->flags->trace = type;
 
 	ret = do_run_tracer_selftest(type);
 	if (ret < 0)
@@ -5159,14 +5134,12 @@ static int tracing_trace_options_show(struct seq_file *m, void *v)
 {
 	struct tracer_opt *trace_opts;
 	struct trace_array *tr = m->private;
+	struct tracer *trace;
 	u32 tracer_flags;
 	int i;
 
 	guard(mutex)(&trace_types_lock);
 
-	tracer_flags = tr->current_trace->flags->val;
-	trace_opts = tr->current_trace->flags->opts;
-
 	for (i = 0; trace_options[i]; i++) {
 		if (tr->trace_flags & (1ULL << i))
 			seq_printf(m, "%s\n", trace_options[i]);
@@ -5174,6 +5147,13 @@ static int tracing_trace_options_show(struct seq_file *m, void *v)
 			seq_printf(m, "no%s\n", trace_options[i]);
 	}
 
+	trace = tr->current_trace;
+	if (!trace->flags || !trace->flags->opts)
+		return 0;
+
+	tracer_flags = tr->current_trace->flags->val;
+	trace_opts = tr->current_trace->flags->opts;
+
 	for (i = 0; trace_opts[i].name; i++) {
 		if (tracer_flags & trace_opts[i].bit)
 			seq_printf(m, "%s\n", trace_opts[i].name);
@@ -5189,9 +5169,10 @@ static int __set_tracer_option(struct trace_array *tr,
 			       struct tracer_opt *opts, int neg)
 {
 	struct tracer *trace = tracer_flags->trace;
-	int ret;
+	int ret = 0;
 
-	ret = trace->set_flag(tr, tracer_flags->val, opts->bit, !neg);
+	if (trace->set_flag)
+		ret = trace->set_flag(tr, tracer_flags->val, opts->bit, !neg);
 	if (ret)
 		return ret;
 
@@ -5210,6 +5191,9 @@ static int set_tracer_option(struct trace_array *tr, char *cmp, int neg)
 	struct tracer_opt *opts = NULL;
 	int i;
 
+	if (!tracer_flags || !tracer_flags->opts)
+		return 0;
+
 	for (i = 0; tracer_flags->opts[i].name; i++) {
 		opts = &tracer_flags->opts[i];
 
-- 
2.51.0

[PATCH 2/5] tracing: Have add_tracer_options() error pass up to callers

[Steven Rostedt]

From: Steven Rostedt <rostedt@goodmis.org>

The function add_tracer_options() can fail, but currently it is ignored.
Pass the status of add_tracer_options() up to adding a new tracer as well
as when an instance is created. Have the instance creation fail if the
add_tracer_options() fail.

Only print a warning for the top level instance, like it does with other
failures.

Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
---
 kernel/trace/trace.c | 55 +++++++++++++++++++++++++++-----------------
 1 file changed, 34 insertions(+), 21 deletions(-)

diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index afeaa9a164e9..ed929d331e1d 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -2302,7 +2302,7 @@ static inline int do_run_tracer_selftest(struct tracer *type)
 }
 #endif /* CONFIG_FTRACE_STARTUP_TEST */
 
-static void add_tracer_options(struct trace_array *tr, struct tracer *t);
+static int add_tracer_options(struct trace_array *tr, struct tracer *t);
 
 static void __init apply_trace_boot_options(void);
 
@@ -2353,9 +2353,14 @@ int __init register_tracer(struct tracer *type)
 	if (ret < 0)
 		goto out;
 
+	ret = add_tracer_options(&global_trace, type);
+	if (ret < 0) {
+		pr_warn("Failed to create tracer options for %s\n", type->name);
+		goto out;
+	}
+
 	type->next = trace_types;
 	trace_types = type;
-	add_tracer_options(&global_trace, type);
 
  out:
 	mutex_unlock(&trace_types_lock);
@@ -6221,7 +6226,7 @@ int tracing_update_buffers(struct trace_array *tr)
 
 struct trace_option_dentry;
 
-static void
+static int
 create_trace_option_files(struct trace_array *tr, struct tracer *tracer);
 
 /*
@@ -6243,17 +6248,17 @@ static void tracing_set_nop(struct trace_array *tr)
 
 static bool tracer_options_updated;
 
-static void add_tracer_options(struct trace_array *tr, struct tracer *t)
+static int add_tracer_options(struct trace_array *tr, struct tracer *t)
 {
 	/* Only enable if the directory has been created already. */
 	if (!tr->dir && !(tr->flags & TRACE_ARRAY_FL_GLOBAL))
-		return;
+		return 0;
 
 	/* Only create trace option files after update_tracer_options finish */
 	if (!tracer_options_updated)
-		return;
+		return 0;
 
-	create_trace_option_files(tr, t);
+	return create_trace_option_files(tr, t);
 }
 
 int tracing_set_tracer(struct trace_array *tr, const char *buf)
@@ -9585,7 +9590,7 @@ create_trace_option_file(struct trace_array *tr,
 
 }
 
-static void
+static int
 create_trace_option_files(struct trace_array *tr, struct tracer *tracer)
 {
 	struct trace_option_dentry *topts;
@@ -9596,24 +9601,24 @@ create_trace_option_files(struct trace_array *tr, struct tracer *tracer)
 	int i;
 
 	if (!tracer)
-		return;
+		return 0;
 
 	flags = tracer->flags;
 
 	if (!flags || !flags->opts)
-		return;
+		return 0;
 
 	/*
 	 * If this is an instance, only create flags for tracers
 	 * the instance may have.
 	 */
 	if (!trace_ok_for_array(tracer, tr))
-		return;
+		return 0;
 
 	for (i = 0; i < tr->nr_topts; i++) {
 		/* Make sure there's no duplicate flags. */
 		if (WARN_ON_ONCE(tr->topts[i].tracer->flags == tracer->flags))
-			return;
+			return -EINVAL;
 	}
 
 	opts = flags->opts;
@@ -9623,13 +9628,13 @@ create_trace_option_files(struct trace_array *tr, struct tracer *tracer)
 
 	topts = kcalloc(cnt + 1, sizeof(*topts), GFP_KERNEL);
 	if (!topts)
-		return;
+		return 0;
 
 	tr_topts = krealloc(tr->topts, sizeof(*tr->topts) * (tr->nr_topts + 1),
 			    GFP_KERNEL);
 	if (!tr_topts) {
 		kfree(topts);
-		return;
+		return -ENOMEM;
 	}
 
 	tr->topts = tr_topts;
@@ -9644,6 +9649,7 @@ create_trace_option_files(struct trace_array *tr, struct tracer *tracer)
 			  "Failed to create trace option: %s",
 			  opts[cnt].name);
 	}
+	return 0;
 }
 
 static struct dentry *
@@ -10094,15 +10100,18 @@ static void init_trace_flags_index(struct trace_array *tr)
 		tr->trace_flags_index[i] = i;
 }
 
-static void __update_tracer_options(struct trace_array *tr)
+static int __update_tracer_options(struct trace_array *tr)
 {
 	struct tracer *t;
+	int ret = 0;
+
+	for (t = trace_types; t && !ret; t = t->next)
+		ret = add_tracer_options(tr, t);
 
-	for (t = trace_types; t; t = t->next)
-		add_tracer_options(tr, t);
+	return ret;
 }
 
-static void update_tracer_options(struct trace_array *tr)
+static __init void update_tracer_options(struct trace_array *tr)
 {
 	guard(mutex)(&trace_types_lock);
 	tracer_options_updated = true;
@@ -10151,9 +10160,13 @@ static int trace_array_create_dir(struct trace_array *tr)
 	}
 
 	init_tracer_tracefs(tr, tr->dir);
-	__update_tracer_options(tr);
-
-	return ret;
+	ret = __update_tracer_options(tr);
+	if (ret) {
+		event_trace_del_tracer(tr);
+		tracefs_remove(tr->dir);
+		return ret;
+	}
+	return 0;
 }
 
 static struct trace_array *
-- 
2.51.0

[PATCH 3/5] tracing: Have tracer option be instance specific

[Steven Rostedt]

From: Steven Rostedt <rostedt@goodmis.org>

Tracers can add specify options to modify them. This logic was added
before instances were created and the tracer flags were global variables.
After instances were created where a tracer may exist in more than one
instance, the flags were not updated from being global into instance
specific. This causes confusion with these options. For example, the
function tracer has an option to enable function arguments:

  # cd /sys/kernel/tracing
  # mkdir instances/foo
  # echo function > instances/foo/current_tracer
  # echo 1 > options/func-args
  # echo function > current_tracer
  # cat trace
[..]
  <idle>-0       [005] d..3.  1050.656187: rcu_needs_cpu() <-tick_nohz_next_event
  <idle>-0       [005] d..3.  1050.656188: get_next_timer_interrupt(basej=0x10002dbad, basem=0xf45fd7d300) <-tick_nohz_next_event
  <idle>-0       [005] d..3.  1050.656189: _raw_spin_lock(lock=0xffff8944bdf5de80) <-__get_next_timer_interrupt
  <idle>-0       [005] d..4.  1050.656190: do_raw_spin_lock(lock=0xffff8944bdf5de80) <-__get_next_timer_interrupt
  <idle>-0       [005] d..4.  1050.656191: _raw_spin_lock_nested(lock=0xffff8944bdf5f140, subclass=1) <-__get_next_timer_interrupt
 # cat instances/foo/options/func-args
 1
 # cat instances/foo/trace
[..]
  kworker/4:1-88      [004] ...1.   298.127735: next_zone <-refresh_cpu_vm_stats
  kworker/4:1-88      [004] ...1.   298.127736: first_online_pgdat <-refresh_cpu_vm_stats
  kworker/4:1-88      [004] ...1.   298.127738: next_online_pgdat <-refresh_cpu_vm_stats
  kworker/4:1-88      [004] ...1.   298.127739: fold_diff <-refresh_cpu_vm_stats
  kworker/4:1-88      [004] ...1.   298.127741: round_jiffies_relative <-vmstat_update
[..]

The above shows that setting "func-args" in the top level instance also
set it in the instance "foo", but since the interface of the trace flags
are per instance, the update didn't take affect in the "foo" instance.

Update the infrastructure to allow tracers to add a "default_flags" field
in the tracer structure that can be set instead of "flags" which will make
the flags per instance. If a tracer needs to keep the flags global (like
blktrace), keeping the "flags" field set will keep the old behavior.

This does not update function or the function graph tracers. That will be
handled later.

Fixes: f20a580627f43 ("ftrace: Allow instances to use function tracing")
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
---
 kernel/trace/trace.c | 221 +++++++++++++++++++++++++++++--------------
 kernel/trace/trace.h |   3 +
 2 files changed, 154 insertions(+), 70 deletions(-)

diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index ed929d331e1d..dea1566b3301 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -94,6 +94,13 @@ static bool tracepoint_printk_stop_on_boot __initdata;
 static bool traceoff_after_boot __initdata;
 static DEFINE_STATIC_KEY_FALSE(tracepoint_printk_key);
 
+/* Store tracers and their flags per instance */
+struct tracers {
+	struct list_head	list;
+	struct tracer		*tracer;
+	struct tracer_flags	*flags;
+};
+
 /*
  * To prevent the comm cache from being overwritten when no
  * tracing is active, only save the comm when a trace event
@@ -2164,6 +2171,9 @@ static int save_selftest(struct tracer *type)
 static int run_tracer_selftest(struct tracer *type)
 {
 	struct trace_array *tr = &global_trace;
+	struct trace_tracer *type = tracers->trace;
+	struct tracer_flags *saved_flags = tr->current_trace_flags;
+	struct tracer_flags *flags;
 	struct tracer *saved_tracer = tr->current_trace;
 	int ret;
 
@@ -2194,6 +2204,7 @@ static int run_tracer_selftest(struct tracer *type)
 	tracing_reset_online_cpus(&tr->array_buffer);
 
 	tr->current_trace = type;
+	tr->current_trace_flags = type->flags ? : type->default_flags;
 
 #ifdef CONFIG_TRACER_MAX_TRACE
 	if (type->use_max_tr) {
@@ -2210,6 +2221,7 @@ static int run_tracer_selftest(struct tracer *type)
 	ret = type->selftest(type, tr);
 	/* the test is responsible for resetting too */
 	tr->current_trace = saved_tracer;
+	tr->current_trace_flags = saved_flags;
 	if (ret) {
 		printk(KERN_CONT "FAILED!\n");
 		/* Add the warning after printing 'FAILED' */
@@ -2302,10 +2314,23 @@ static inline int do_run_tracer_selftest(struct tracer *type)
 }
 #endif /* CONFIG_FTRACE_STARTUP_TEST */
 
-static int add_tracer_options(struct trace_array *tr, struct tracer *t);
+static int add_tracer(struct trace_array *tr, struct tracer *t);
 
 static void __init apply_trace_boot_options(void);
 
+static void free_tracers(struct trace_array *tr)
+{
+	struct tracers *t, *n;
+
+	lockdep_assert_held(&trace_types_lock);
+
+	list_for_each_entry_safe(t, n, &tr->tracers, list) {
+		list_del(&t->list);
+		kfree(t->flags);
+		kfree(t);
+	}
+}
+
 /**
  * register_tracer - register a tracer with the ftrace system.
  * @type: the plugin for the tracer
@@ -2314,6 +2339,7 @@ static void __init apply_trace_boot_options(void);
  */
 int __init register_tracer(struct tracer *type)
 {
+	struct trace_array *tr;
 	struct tracer *t;
 	int ret = 0;
 
@@ -2353,10 +2379,13 @@ int __init register_tracer(struct tracer *type)
 	if (ret < 0)
 		goto out;
 
-	ret = add_tracer_options(&global_trace, type);
-	if (ret < 0) {
-		pr_warn("Failed to create tracer options for %s\n", type->name);
-		goto out;
+	list_for_each_entry(tr, &ftrace_trace_arrays, list) {
+		ret = add_tracer(tr, type);
+		if (ret < 0) {
+			/* The tracer will still exist but without options */
+			pr_warn("Failed to create tracer options for %s\n", type->name);
+			break;
+		}
 	}
 
 	type->next = trace_types;
@@ -5139,6 +5168,7 @@ static int tracing_trace_options_show(struct seq_file *m, void *v)
 {
 	struct tracer_opt *trace_opts;
 	struct trace_array *tr = m->private;
+	struct tracer_flags *flags;
 	struct tracer *trace;
 	u32 tracer_flags;
 	int i;
@@ -5152,12 +5182,14 @@ static int tracing_trace_options_show(struct seq_file *m, void *v)
 			seq_printf(m, "no%s\n", trace_options[i]);
 	}
 
-	trace = tr->current_trace;
-	if (!trace->flags || !trace->flags->opts)
+	flags = tr->current_trace_flags;
+	if (!flags || !flags->opts)
 		return 0;
 
-	tracer_flags = tr->current_trace->flags->val;
-	trace_opts = tr->current_trace->flags->opts;
+	trace = tr->current_trace;
+
+	tracer_flags = flags->val;
+	trace_opts = flags->opts;
 
 	for (i = 0; trace_opts[i].name; i++) {
 		if (tracer_flags & trace_opts[i].bit)
@@ -5191,8 +5223,7 @@ static int __set_tracer_option(struct trace_array *tr,
 /* Try to assign a tracer specific option */
 static int set_tracer_option(struct trace_array *tr, char *cmp, int neg)
 {
-	struct tracer *trace = tr->current_trace;
-	struct tracer_flags *tracer_flags = trace->flags;
+	struct tracer_flags *tracer_flags = tr->current_trace_flags;
 	struct tracer_opt *opts = NULL;
 	int i;
 
@@ -5203,7 +5234,7 @@ static int set_tracer_option(struct trace_array *tr, char *cmp, int neg)
 		opts = &tracer_flags->opts[i];
 
 		if (strcmp(cmp, opts->name) == 0)
-			return __set_tracer_option(tr, trace->flags, opts, neg);
+			return __set_tracer_option(tr, tracer_flags, opts, neg);
 	}
 
 	return -EINVAL;
@@ -6224,11 +6255,6 @@ int tracing_update_buffers(struct trace_array *tr)
 	return ret;
 }
 
-struct trace_option_dentry;
-
-static int
-create_trace_option_files(struct trace_array *tr, struct tracer *tracer);
-
 /*
  * Used to clear out the tracer before deletion of an instance.
  * Must have trace_types_lock held.
@@ -6244,26 +6270,15 @@ static void tracing_set_nop(struct trace_array *tr)
 		tr->current_trace->reset(tr);
 
 	tr->current_trace = &nop_trace;
+	tr->current_trace_flags = nop_trace.flags;
 }
 
 static bool tracer_options_updated;
 
-static int add_tracer_options(struct trace_array *tr, struct tracer *t)
-{
-	/* Only enable if the directory has been created already. */
-	if (!tr->dir && !(tr->flags & TRACE_ARRAY_FL_GLOBAL))
-		return 0;
-
-	/* Only create trace option files after update_tracer_options finish */
-	if (!tracer_options_updated)
-		return 0;
-
-	return create_trace_option_files(tr, t);
-}
-
 int tracing_set_tracer(struct trace_array *tr, const char *buf)
 {
-	struct tracer *t;
+	struct tracer *trace;
+	struct tracers *t;
 #ifdef CONFIG_TRACER_MAX_TRACE
 	bool had_max_tr;
 #endif
@@ -6281,18 +6296,20 @@ int tracing_set_tracer(struct trace_array *tr, const char *buf)
 		ret = 0;
 	}
 
-	for (t = trace_types; t; t = t->next) {
-		if (strcmp(t->name, buf) == 0)
+	list_for_each_entry(t, &tr->tracers, list) {
+		if (strcmp(t->tracer->name, buf) == 0)
 			break;
 	}
 	if (!t)
 		return -EINVAL;
 
-	if (t == tr->current_trace)
+	if (t->tracer == tr->current_trace)
 		return 0;
 
+	trace = t->tracer;
+
 #ifdef CONFIG_TRACER_SNAPSHOT
-	if (t->use_max_tr) {
+	if (trace->use_max_tr) {
 		local_irq_disable();
 		arch_spin_lock(&tr->max_lock);
 		ret = tr->cond_snapshot ? -EBUSY : 0;
@@ -6303,14 +6320,14 @@ int tracing_set_tracer(struct trace_array *tr, const char *buf)
 	}
 #endif
 	/* Some tracers won't work on kernel command line */
-	if (system_state < SYSTEM_RUNNING && t->noboot) {
+	if (system_state < SYSTEM_RUNNING && trace->noboot) {
 		pr_warn("Tracer '%s' is not allowed on command line, ignored\n",
-			t->name);
+			trace->name);
 		return -EINVAL;
 	}
 
 	/* Some tracers are only allowed for the top level buffer */
-	if (!trace_ok_for_array(t, tr))
+	if (!trace_ok_for_array(trace, tr))
 		return -EINVAL;
 
 	/* If trace pipe files are being read, we can't change the tracer */
@@ -6329,8 +6346,9 @@ int tracing_set_tracer(struct trace_array *tr, const char *buf)
 
 	/* Current trace needs to be nop_trace before synchronize_rcu */
 	tr->current_trace = &nop_trace;
+	tr->current_trace_flags = nop_trace.flags;
 
-	if (had_max_tr && !t->use_max_tr) {
+	if (had_max_tr && !trace->use_max_tr) {
 		/*
 		 * We need to make sure that the update_max_tr sees that
 		 * current_trace changed to nop_trace to keep it from
@@ -6343,7 +6361,7 @@ int tracing_set_tracer(struct trace_array *tr, const char *buf)
 		tracing_disarm_snapshot(tr);
 	}
 
-	if (!had_max_tr && t->use_max_tr) {
+	if (!had_max_tr && trace->use_max_tr) {
 		ret = tracing_arm_snapshot_locked(tr);
 		if (ret)
 			return ret;
@@ -6352,18 +6370,21 @@ int tracing_set_tracer(struct trace_array *tr, const char *buf)
 	tr->current_trace = &nop_trace;
 #endif
 
-	if (t->init) {
-		ret = tracer_init(t, tr);
+	tr->current_trace_flags = t->flags ? : t->tracer->flags;
+
+	if (trace->init) {
+		ret = tracer_init(trace, tr);
 		if (ret) {
 #ifdef CONFIG_TRACER_MAX_TRACE
-			if (t->use_max_tr)
+			if (trace->use_max_tr)
 				tracing_disarm_snapshot(tr);
 #endif
+			tr->current_trace_flags = nop_trace.flags;
 			return ret;
 		}
 	}
 
-	tr->current_trace = t;
+	tr->current_trace = trace;
 	tr->current_trace->enabled++;
 	trace_branch_enable(tr);
 
@@ -9587,40 +9608,20 @@ create_trace_option_file(struct trace_array *tr,
 
 	topt->entry = trace_create_file(opt->name, TRACE_MODE_WRITE,
 					t_options, topt, &trace_options_fops);
-
 }
 
 static int
-create_trace_option_files(struct trace_array *tr, struct tracer *tracer)
+create_trace_option_files(struct trace_array *tr, struct tracer *tracer,
+			  struct tracer_flags *flags)
 {
 	struct trace_option_dentry *topts;
 	struct trace_options *tr_topts;
-	struct tracer_flags *flags;
 	struct tracer_opt *opts;
 	int cnt;
-	int i;
-
-	if (!tracer)
-		return 0;
-
-	flags = tracer->flags;
 
 	if (!flags || !flags->opts)
 		return 0;
 
-	/*
-	 * If this is an instance, only create flags for tracers
-	 * the instance may have.
-	 */
-	if (!trace_ok_for_array(tracer, tr))
-		return 0;
-
-	for (i = 0; i < tr->nr_topts; i++) {
-		/* Make sure there's no duplicate flags. */
-		if (WARN_ON_ONCE(tr->topts[i].tracer->flags == tracer->flags))
-			return -EINVAL;
-	}
-
 	opts = flags->opts;
 
 	for (cnt = 0; opts[cnt].name; cnt++)
@@ -9652,6 +9653,80 @@ create_trace_option_files(struct trace_array *tr, struct tracer *tracer)
 	return 0;
 }
 
+static int get_global_flags_val(struct tracer *tracer)
+{
+	struct tracers *t;
+
+	list_for_each_entry(t, &global_trace.tracers, list) {
+		if (t->tracer != tracer)
+			continue;
+		if (!t->flags)
+			return -1;
+		return t->flags->val;
+	}
+	return -1;
+}
+
+static int add_tracer(struct trace_array *tr, struct tracer *tracer)
+{
+	struct tracer_flags *flags;
+	struct tracers *t;
+
+	/* Only enable if the directory has been created already. */
+	if (!tr->dir && !(tr->flags & TRACE_ARRAY_FL_GLOBAL))
+		return 0;
+
+	/* Only add tracer after update_tracer_options finish */
+	if (!tracer_options_updated)
+		return 0;
+
+	/*
+	 * If this is an instance, only create flags for tracers
+	 * the instance may have.
+	 */
+	if (!trace_ok_for_array(tracer, tr))
+		return 0;
+
+	t = kmalloc(sizeof(*t), GFP_KERNEL);
+	if (!t)
+		return -ENOMEM;
+
+	t->tracer = tracer;
+	t->flags = NULL;
+	list_add(&t->list, &tr->tracers);
+
+	flags = tracer->flags;
+	if (!flags) {
+		if (!tracer->default_flags)
+			return 0;
+
+		/*
+		 * If the tracer defines default flags, it means the flags are
+		 * per trace instance.
+		 */
+		flags = kmalloc(sizeof(*flags), GFP_KERNEL);
+		if (!flags) {
+			list_del(&t->list);
+			kfree(t);
+			return -ENOMEM;
+		}
+
+		*flags = *tracer->default_flags;
+		flags->trace = tracer;
+
+		t->flags = flags;
+
+		/* If this is an instance, inherit the global_trace flags */
+		if (!(tr->flags & TRACE_ARRAY_FL_GLOBAL)) {
+			int val = get_global_flags_val(tracer);
+			if (!WARN_ON_ONCE(val < 0))
+				flags->val = val;
+		}
+	}
+
+	return create_trace_option_files(tr, tracer, flags);
+}
+
 static struct dentry *
 create_trace_option_core_file(struct trace_array *tr,
 			      const char *option, long index)
@@ -10100,13 +10175,13 @@ static void init_trace_flags_index(struct trace_array *tr)
 		tr->trace_flags_index[i] = i;
 }
 
-static int __update_tracer_options(struct trace_array *tr)
+static int __update_tracer(struct trace_array *tr)
 {
 	struct tracer *t;
 	int ret = 0;
 
 	for (t = trace_types; t && !ret; t = t->next)
-		ret = add_tracer_options(tr, t);
+		ret = add_tracer(tr, t);
 
 	return ret;
 }
@@ -10115,7 +10190,7 @@ static __init void update_tracer_options(struct trace_array *tr)
 {
 	guard(mutex)(&trace_types_lock);
 	tracer_options_updated = true;
-	__update_tracer_options(tr);
+	__update_tracer(tr);
 }
 
 /* Must have trace_types_lock held */
@@ -10160,7 +10235,7 @@ static int trace_array_create_dir(struct trace_array *tr)
 	}
 
 	init_tracer_tracefs(tr, tr->dir);
-	ret = __update_tracer_options(tr);
+	ret = __update_tracer(tr);
 	if (ret) {
 		event_trace_del_tracer(tr);
 		tracefs_remove(tr->dir);
@@ -10215,11 +10290,13 @@ trace_array_create_systems(const char *name, const char *systems,
 	spin_lock_init(&tr->snapshot_trigger_lock);
 #endif
 	tr->current_trace = &nop_trace;
+	tr->current_trace_flags = nop_trace.flags;
 
 	INIT_LIST_HEAD(&tr->systems);
 	INIT_LIST_HEAD(&tr->events);
 	INIT_LIST_HEAD(&tr->hist_vars);
 	INIT_LIST_HEAD(&tr->err_log);
+	INIT_LIST_HEAD(&tr->tracers);
 	INIT_LIST_HEAD(&tr->marker_list);
 
 #ifdef CONFIG_MODULES
@@ -10392,6 +10469,7 @@ static int __remove_instance(struct trace_array *tr)
 	free_percpu(tr->last_func_repeats);
 	free_trace_buffers(tr);
 	clear_tracing_err_log(tr);
+	free_tracers(tr);
 
 	if (tr->range_name) {
 		reserve_mem_release_by_name(tr->range_name);
@@ -11426,6 +11504,7 @@ __init static int tracer_alloc_buffers(void)
 	 * just a bootstrap of current_trace anyway.
 	 */
 	global_trace.current_trace = &nop_trace;
+	global_trace.current_trace_flags = nop_trace.flags;
 
 	global_trace.max_lock = (arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED;
 #ifdef CONFIG_TRACER_MAX_TRACE
@@ -11439,6 +11518,8 @@ __init static int tracer_alloc_buffers(void)
 
 	init_trace_flags_index(&global_trace);
 
+	INIT_LIST_HEAD(&global_trace.tracers);
+
 	register_tracer(&nop_trace);
 
 	/* Function tracing may start here (via kernel command line) */
diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h
index 8ecaf91ca823..299862aad66c 100644
--- a/kernel/trace/trace.h
+++ b/kernel/trace/trace.h
@@ -392,6 +392,7 @@ struct trace_array {
 	int			buffer_percent;
 	unsigned int		n_err_log_entries;
 	struct tracer		*current_trace;
+	struct tracer_flags	*current_trace_flags;
 	u64			trace_flags;
 	unsigned char		trace_flags_index[TRACE_FLAGS_MAX_SIZE];
 	unsigned int		flags;
@@ -406,6 +407,7 @@ struct trace_array {
 	struct list_head	systems;
 	struct list_head	events;
 	struct list_head	marker_list;
+	struct list_head	tracers;
 	struct trace_event_file *trace_marker_file;
 	cpumask_var_t		tracing_cpumask; /* only trace on set CPUs */
 	/* one per_cpu trace_pipe can be opened by only one user */
@@ -637,6 +639,7 @@ struct tracer {
 						u64 mask, int set);
 	struct tracer		*next;
 	struct tracer_flags	*flags;
+	struct tracer_flags	*default_flags;
 	int			enabled;
 	bool			print_max;
 	bool			allow_instances;
-- 
2.51.0

[PATCH 4/5] tracing: Have function tracer define options per instance

[Steven Rostedt]

From: Steven Rostedt <rostedt@goodmis.org>

Currently the function tracer's options are saved via a global mask when
it should be per instance. Use the new infrastructure to define a
"default_flags" field in the tracer structure that is used for the top
level instance as well as new ones.

Currently the global mask causes confusion:

  # cd /sys/kernel/tracing
  # mkdir instances/foo
  # echo function > instances/foo/current_tracer
  # echo 1 > options/func-args
  # echo function > current_tracer
  # cat trace
[..]
  <idle>-0       [005] d..3.  1050.656187: rcu_needs_cpu() <-tick_nohz_next_event
  <idle>-0       [005] d..3.  1050.656188: get_next_timer_interrupt(basej=0x10002dbad, basem=0xf45fd7d300) <-tick_nohz_next_event
  <idle>-0       [005] d..3.  1050.656189: _raw_spin_lock(lock=0xffff8944bdf5de80) <-__get_next_timer_interrupt
  <idle>-0       [005] d..4.  1050.656190: do_raw_spin_lock(lock=0xffff8944bdf5de80) <-__get_next_timer_interrupt
  <idle>-0       [005] d..4.  1050.656191: _raw_spin_lock_nested(lock=0xffff8944bdf5f140, subclass=1) <-__get_next_timer_interrupt
 # cat instances/foo/options/func-args
 1
 # cat instances/foo/trace
[..]
  kworker/4:1-88      [004] ...1.   298.127735: next_zone <-refresh_cpu_vm_stats
  kworker/4:1-88      [004] ...1.   298.127736: first_online_pgdat <-refresh_cpu_vm_stats
  kworker/4:1-88      [004] ...1.   298.127738: next_online_pgdat <-refresh_cpu_vm_stats
  kworker/4:1-88      [004] ...1.   298.127739: fold_diff <-refresh_cpu_vm_stats
  kworker/4:1-88      [004] ...1.   298.127741: round_jiffies_relative <-vmstat_update
[..]

The above shows that updating the "func-args" option at the top level
instance also updates the "func-args" option in the instance but because
the update is only done by the instance that gets changed (as it should),
it's confusing to see that the option is already set in the other instance.

Fixes: f20a580627f43 ("ftrace: Allow instances to use function tracing")
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
---
 kernel/trace/trace_functions.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/kernel/trace/trace_functions.c b/kernel/trace/trace_functions.c
index d17c18934445..c12795c2fb39 100644
--- a/kernel/trace/trace_functions.c
+++ b/kernel/trace/trace_functions.c
@@ -154,11 +154,11 @@ static int function_trace_init(struct trace_array *tr)
 	if (!tr->ops)
 		return -ENOMEM;
 
-	func = select_trace_function(func_flags.val);
+	func = select_trace_function(tr->current_trace_flags->val);
 	if (!func)
 		return -EINVAL;
 
-	if (!handle_func_repeats(tr, func_flags.val))
+	if (!handle_func_repeats(tr, tr->current_trace_flags->val))
 		return -ENOMEM;
 
 	ftrace_init_array_ops(tr, func);
@@ -459,14 +459,14 @@ func_set_flag(struct trace_array *tr, u32 old_flags, u32 bit, int set)
 	u32 new_flags;
 
 	/* Do nothing if already set. */
-	if (!!set == !!(func_flags.val & bit))
+	if (!!set == !!(tr->current_trace_flags->val & bit))
 		return 0;
 
 	/* We can change this flag only when not running. */
 	if (tr->current_trace != &function_trace)
 		return 0;
 
-	new_flags = (func_flags.val & ~bit) | (set ? bit : 0);
+	new_flags = (tr->current_trace_flags->val & ~bit) | (set ? bit : 0);
 	func = select_trace_function(new_flags);
 	if (!func)
 		return -EINVAL;
@@ -491,7 +491,7 @@ static struct tracer function_trace __tracer_data =
 	.init		= function_trace_init,
 	.reset		= function_trace_reset,
 	.start		= function_trace_start,
-	.flags		= &func_flags,
+	.default_flags	= &func_flags,
 	.set_flag	= func_set_flag,
 	.allow_instances = true,
 #ifdef CONFIG_FTRACE_SELFTEST
-- 
2.51.0

[PATCH 5/5] tracing: Have function graph tracer define options per instance

[Steven Rostedt]

From: Steven Rostedt <rostedt@goodmis.org>

Currently the function graph tracer's options are saved via a global mask
when it should be per instance. Use the new infrastructure to define a
"default_flags" field in the tracer structure that is used for the top
level instance as well as new ones.

Currently the global mask causes confusion:

  # cd /sys/kernel/tracing
  # mkdir instances/foo
  # echo function_graph > instances/foo/current_tracer
  # echo 1 > options/funcgraph-args
  # echo function_graph > current_tracer
  # cat trace
[..]
 2)               |          _raw_spin_lock_irq(lock=0xffff96b97dea16c0) {
 2)   0.422 us    |            do_raw_spin_lock(lock=0xffff96b97dea16c0);
 7)               |              rcu_sched_clock_irq(user=0) {
 2)   1.478 us    |          }
 7)   0.758 us    |                rcu_is_cpu_rrupt_from_idle();
 2)   0.647 us    |          enqueue_hrtimer(timer=0xffff96b97dea2058, base=0xffff96b97dea1740, mode=0);
 # cat instances/foo/options/funcgraph-args
 1
 # cat instances/foo/trace
[..]
 4)               |  __x64_sys_read() {
 4)               |    ksys_read() {
 4)   0.755 us    |      fdget_pos();
 4)               |      vfs_read() {
 4)               |        rw_verify_area() {
 4)               |          security_file_permission() {
 4)               |            apparmor_file_permission() {
 4)               |              common_file_perm() {
 4)               |                aa_file_perm() {
 4)               |                  rcu_read_lock_held() {
[..]

The above shows that updating the "funcgraph-args" option at the top level
instance also updates the "funcgraph-args" option in the instance but
because the update is only done by the instance that gets changed (as it
should), it's confusing to see that the option is already set in the other
instance.

Fixes: c132be2c4fcc1 ("function_graph: Have the instances use their own ftrace_ops for filtering")
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
---
 kernel/trace/trace_functions_graph.c | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)

diff --git a/kernel/trace/trace_functions_graph.c b/kernel/trace/trace_functions_graph.c
index fe9607edc8f9..4e86adf6dd4d 100644
--- a/kernel/trace/trace_functions_graph.c
+++ b/kernel/trace/trace_functions_graph.c
@@ -101,9 +101,9 @@ static struct tracer_flags tracer_flags = {
 	.opts = trace_opts
 };
 
-static bool tracer_flags_is_set(u32 flags)
+static bool tracer_flags_is_set(struct trace_array *tr, u32 flags)
 {
-	return (tracer_flags.val & flags) == flags;
+	return (tr->current_trace_flags->val & flags) == flags;
 }
 
 /*
@@ -263,7 +263,7 @@ static int graph_entry(struct ftrace_graph_ent *trace,
 
 	trace_ctx = tracing_gen_ctx();
 	if (IS_ENABLED(CONFIG_FUNCTION_GRAPH_RETADDR) &&
-	    tracer_flags_is_set(TRACE_GRAPH_PRINT_RETADDR)) {
+	    tracer_flags_is_set(tr, TRACE_GRAPH_PRINT_RETADDR)) {
 		unsigned long retaddr = ftrace_graph_top_ret_addr(current);
 		ret = __trace_graph_retaddr_entry(tr, trace, trace_ctx, retaddr);
 	} else {
@@ -441,7 +441,7 @@ static int graph_trace_init(struct trace_array *tr)
 {
 	int ret;
 
-	if (tracer_flags_is_set(TRACE_GRAPH_ARGS))
+	if (tracer_flags_is_set(tr, TRACE_GRAPH_ARGS))
 		tr->gops->entryfunc = trace_graph_entry_args;
 	else
 		tr->gops->entryfunc = trace_graph_entry;
@@ -1459,7 +1459,8 @@ print_graph_function_flags(struct trace_iterator *iter, u32 flags)
 static enum print_line_t
 print_graph_function(struct trace_iterator *iter)
 {
-	return print_graph_function_flags(iter, tracer_flags.val);
+	struct trace_array *tr = iter->tr;
+	return print_graph_function_flags(iter, tr->current_trace_flags->val);
 }
 
 static enum print_line_t
@@ -1535,7 +1536,10 @@ static void __print_graph_headers_flags(struct trace_array *tr,
 
 static void print_graph_headers(struct seq_file *s)
 {
-	print_graph_headers_flags(s, tracer_flags.val);
+	struct trace_iterator *iter = s->private;
+	struct trace_array *tr = iter->tr;
+
+	print_graph_headers_flags(s, tr->current_trace_flags->val);
 }
 
 void print_graph_headers_flags(struct seq_file *s, u32 flags)
@@ -1660,7 +1664,7 @@ static struct tracer graph_trace __tracer_data = {
 	.reset		= graph_trace_reset,
 	.print_line	= print_graph_function,
 	.print_header	= print_graph_headers,
-	.flags		= &tracer_flags,
+	.default_flags	= &tracer_flags,
 	.set_flag	= func_graph_set_flag,
 	.allow_instances = true,
 #ifdef CONFIG_FTRACE_SELFTEST
-- 
2.51.0

Re: [PATCH 3/5] tracing: Have tracer option be instance specific

[kernel test robot]

Hi Steven,

kernel test robot noticed the following build errors:

[auto build test ERROR on trace/for-next]
[cannot apply to linus/master v6.18-rc4 next-20251106]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Steven-Rostedt/tracing-Remove-dummy-options-and-flags/20251106-010511
base:   https://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace for-next
patch link:    https://lore.kernel.org/r/20251105161935.545400234%40kernel.org
patch subject: [PATCH 3/5] tracing: Have tracer option be instance specific
config: i386-buildonly-randconfig-003-20251106 (https://download.01.org/0day-ci/archive/20251106/202511062223.ocoUvCBI-lkp@intel.com/config)
compiler: gcc-14 (Debian 14.2.0-19) 14.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251106/202511062223.ocoUvCBI-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202511062223.ocoUvCBI-lkp@intel.com/

All error/warnings (new ones prefixed by >>):

   kernel/trace/trace.c: In function 'run_tracer_selftest':
>> kernel/trace/trace.c:2174:30: error: 'type' redeclared as different kind of symbol
    2174 |         struct trace_tracer *type = tracers->trace;
         |                              ^~~~
   kernel/trace/trace.c:2171:47: note: previous definition of 'type' with type 'struct tracer *'
    2171 | static int run_tracer_selftest(struct tracer *type)
         |                                ~~~~~~~~~~~~~~~^~~~
>> kernel/trace/trace.c:2174:37: error: 'tracers' undeclared (first use in this function)
    2174 |         struct trace_tracer *type = tracers->trace;
         |                                     ^~~~~~~
   kernel/trace/trace.c:2174:37: note: each undeclared identifier is reported only once for each function it appears in
>> kernel/trace/trace.c:2180:18: error: invalid use of undefined type 'struct trace_tracer'
    2180 |         if (!type->selftest || tracing_selftest_disabled)
         |                  ^~
>> kernel/trace/trace.c:2189:38: error: passing argument 1 of 'save_selftest' from incompatible pointer type [-Wincompatible-pointer-types]
    2189 |                 return save_selftest(type);
         |                                      ^~~~
         |                                      |
         |                                      struct trace_tracer *
   kernel/trace/trace.c:2158:41: note: expected 'struct tracer *' but argument is of type 'struct trace_tracer *'
    2158 | static int save_selftest(struct tracer *type)
         |                          ~~~~~~~~~~~~~~~^~~~
   In file included from include/asm-generic/bug.h:22,
                    from arch/x86/include/asm/bug.h:108,
                    from include/linux/bug.h:5,
                    from include/linux/mmdebug.h:5,
                    from include/linux/mm.h:6,
                    from include/linux/ring_buffer.h:5,
                    from kernel/trace/trace.c:15:
   kernel/trace/trace.c:2193:29: error: invalid use of undefined type 'struct trace_tracer'
    2193 |                         type->name);
         |                             ^~
   include/linux/printk.h:484:33: note: in definition of macro 'printk_index_wrap'
     484 |                 _p_func(_fmt, ##__VA_ARGS__);                           \
         |                                 ^~~~~~~~~~~
   include/linux/printk.h:565:9: note: in expansion of macro 'printk'
     565 |         printk(KERN_WARNING pr_fmt(fmt), ##__VA_ARGS__)
         |         ^~~~~~
   kernel/trace/trace.c:2192:17: note: in expansion of macro 'pr_warn'
    2192 |                 pr_warn("Selftest for tracer %s skipped due to tracing disabled\n",
         |                 ^~~~~~~
>> kernel/trace/trace.c:2206:27: error: assignment to 'struct tracer *' from incompatible pointer type 'struct trace_tracer *' [-Wincompatible-pointer-types]
    2206 |         tr->current_trace = type;
         |                           ^
   kernel/trace/trace.c:2207:39: error: invalid use of undefined type 'struct trace_tracer'
    2207 |         tr->current_trace_flags = type->flags ? : type->default_flags;
         |                                       ^~
   kernel/trace/trace.c:2207:55: error: invalid use of undefined type 'struct trace_tracer'
    2207 |         tr->current_trace_flags = type->flags ? : type->default_flags;
         |                                                       ^~
   kernel/trace/trace.c:2210:17: error: invalid use of undefined type 'struct trace_tracer'
    2210 |         if (type->use_max_tr) {
         |                 ^~
   kernel/trace/trace.c:2220:44: error: invalid use of undefined type 'struct trace_tracer'
    2220 |         pr_info("Testing tracer %s: ", type->name);
         |                                            ^~
   include/linux/printk.h:484:33: note: in definition of macro 'printk_index_wrap'
     484 |                 _p_func(_fmt, ##__VA_ARGS__);                           \
         |                                 ^~~~~~~~~~~
   include/linux/printk.h:585:9: note: in expansion of macro 'printk'
     585 |         printk(KERN_INFO pr_fmt(fmt), ##__VA_ARGS__)
         |         ^~~~~~
   kernel/trace/trace.c:2220:9: note: in expansion of macro 'pr_info'
    2220 |         pr_info("Testing tracer %s: ", type->name);
         |         ^~~~~~~
   kernel/trace/trace.c:2221:19: error: invalid use of undefined type 'struct trace_tracer'
    2221 |         ret = type->selftest(type, tr);
         |                   ^~
   kernel/trace/trace.c:2235:17: error: invalid use of undefined type 'struct trace_tracer'
    2235 |         if (type->use_max_tr) {
         |                 ^~
>> kernel/trace/trace.c:2176:30: warning: unused variable 'flags' [-Wunused-variable]
    2176 |         struct tracer_flags *flags;
         |                              ^~~~~
   kernel/trace/trace.c: In function 'tracing_trace_options_show':
   kernel/trace/trace.c:5172:24: warning: variable 'trace' set but not used [-Wunused-but-set-variable]
    5172 |         struct tracer *trace;
         |                        ^~~~~


vim +/type +2174 kernel/trace/trace.c

  2170	
> 2171	static int run_tracer_selftest(struct tracer *type)
  2172	{
  2173		struct trace_array *tr = &global_trace;
> 2174		struct trace_tracer *type = tracers->trace;
  2175		struct tracer_flags *saved_flags = tr->current_trace_flags;
> 2176		struct tracer_flags *flags;
  2177		struct tracer *saved_tracer = tr->current_trace;
  2178		int ret;
  2179	
> 2180		if (!type->selftest || tracing_selftest_disabled)
  2181			return 0;
  2182	
  2183		/*
  2184		 * If a tracer registers early in boot up (before scheduling is
  2185		 * initialized and such), then do not run its selftests yet.
  2186		 * Instead, run it a little later in the boot process.
  2187		 */
  2188		if (!selftests_can_run)
> 2189			return save_selftest(type);
  2190	
  2191		if (!tracing_is_on()) {
> 2192			pr_warn("Selftest for tracer %s skipped due to tracing disabled\n",
  2193				type->name);
  2194			return 0;
  2195		}
  2196	
  2197		/*
  2198		 * Run a selftest on this tracer.
  2199		 * Here we reset the trace buffer, and set the current
  2200		 * tracer to be this tracer. The tracer can then run some
  2201		 * internal tracing to verify that everything is in order.
  2202		 * If we fail, we do not register this tracer.
  2203		 */
  2204		tracing_reset_online_cpus(&tr->array_buffer);
  2205	
> 2206		tr->current_trace = type;
  2207		tr->current_trace_flags = type->flags ? : type->default_flags;
  2208	
  2209	#ifdef CONFIG_TRACER_MAX_TRACE
  2210		if (type->use_max_tr) {
  2211			/* If we expanded the buffers, make sure the max is expanded too */
  2212			if (tr->ring_buffer_expanded)
  2213				ring_buffer_resize(tr->max_buffer.buffer, trace_buf_size,
  2214						   RING_BUFFER_ALL_CPUS);
  2215			tr->allocated_snapshot = true;
  2216		}
  2217	#endif
  2218	
  2219		/* the test is responsible for initializing and enabling */
> 2220		pr_info("Testing tracer %s: ", type->name);
  2221		ret = type->selftest(type, tr);
  2222		/* the test is responsible for resetting too */
  2223		tr->current_trace = saved_tracer;
  2224		tr->current_trace_flags = saved_flags;
  2225		if (ret) {
  2226			printk(KERN_CONT "FAILED!\n");
  2227			/* Add the warning after printing 'FAILED' */
  2228			WARN_ON(1);
  2229			return -1;
  2230		}
  2231		/* Only reset on passing, to avoid touching corrupted buffers */
  2232		tracing_reset_online_cpus(&tr->array_buffer);
  2233	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH 3/5] tracing: Have tracer option be instance specific

[kernel test robot]

Hi Steven,

kernel test robot noticed the following build errors:

[auto build test ERROR on trace/for-next]
[cannot apply to linus/master v6.18-rc4 next-20251106]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Steven-Rostedt/tracing-Remove-dummy-options-and-flags/20251106-010511
base:   https://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace for-next
patch link:    https://lore.kernel.org/r/20251105161935.545400234%40kernel.org
patch subject: [PATCH 3/5] tracing: Have tracer option be instance specific
config: i386-buildonly-randconfig-004-20251106 (https://download.01.org/0day-ci/archive/20251106/202511062257.TSJTBQ2c-lkp@intel.com/config)
compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251106/202511062257.TSJTBQ2c-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202511062257.TSJTBQ2c-lkp@intel.com/

All errors (new ones prefixed by >>):

>> kernel/trace/trace.c:2174:23: error: redefinition of 'type' with a different type: 'struct trace_tracer *' vs 'struct tracer *'
    2174 |         struct trace_tracer *type = tracers->trace;
         |                              ^
   kernel/trace/trace.c:2171:47: note: previous definition is here
    2171 | static int run_tracer_selftest(struct tracer *type)
         |                                               ^
>> kernel/trace/trace.c:2174:30: error: use of undeclared identifier 'tracers'
    2174 |         struct trace_tracer *type = tracers->trace;
         |                                     ^
   kernel/trace/trace.c:5172:17: warning: variable 'trace' set but not used [-Wunused-but-set-variable]
    5172 |         struct tracer *trace;
         |                        ^
   1 warning and 2 errors generated.


vim +2174 kernel/trace/trace.c

  2170	
  2171	static int run_tracer_selftest(struct tracer *type)
  2172	{
  2173		struct trace_array *tr = &global_trace;
> 2174		struct trace_tracer *type = tracers->trace;
  2175		struct tracer_flags *saved_flags = tr->current_trace_flags;
  2176		struct tracer_flags *flags;
  2177		struct tracer *saved_tracer = tr->current_trace;
  2178		int ret;
  2179	
  2180		if (!type->selftest || tracing_selftest_disabled)
  2181			return 0;
  2182	
  2183		/*
  2184		 * If a tracer registers early in boot up (before scheduling is
  2185		 * initialized and such), then do not run its selftests yet.
  2186		 * Instead, run it a little later in the boot process.
  2187		 */
  2188		if (!selftests_can_run)
  2189			return save_selftest(type);
  2190	
  2191		if (!tracing_is_on()) {
  2192			pr_warn("Selftest for tracer %s skipped due to tracing disabled\n",
  2193				type->name);
  2194			return 0;
  2195		}
  2196	
  2197		/*
  2198		 * Run a selftest on this tracer.
  2199		 * Here we reset the trace buffer, and set the current
  2200		 * tracer to be this tracer. The tracer can then run some
  2201		 * internal tracing to verify that everything is in order.
  2202		 * If we fail, we do not register this tracer.
  2203		 */
  2204		tracing_reset_online_cpus(&tr->array_buffer);
  2205	
  2206		tr->current_trace = type;
  2207		tr->current_trace_flags = type->flags ? : type->default_flags;
  2208	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH 3/5] tracing: Have tracer option be instance specific

[Steven Rostedt]

On Thu, 6 Nov 2025 22:38:04 +0800
kernel test robot <lkp@intel.com> wrote:

> All error/warnings (new ones prefixed by >>):
> 
>    kernel/trace/trace.c: In function 'run_tracer_selftest':
> >> kernel/trace/trace.c:2174:30: error: 'type' redeclared as different kind of symbol  
>     2174 |         struct trace_tracer *type = tracers->trace;
>          |                              ^~~~
>    kernel/trace/trace.c:2171:47: note: previous definition of '

Yeah, I caught this last night during my more thorough tests. Silly me
turned off selftests to make a bisect go faster and never turned it back
on. :-p

-- Steve

Re: [PATCH 3/5] tracing: Have tracer option be instance specific

[Dan Carpenter]

Hi Steven,

kernel test robot noticed the following build warnings:

https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Steven-Rostedt/tracing-Remove-dummy-options-and-flags/20251106-010511
base:   https://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace for-next
patch link:    https://lore.kernel.org/r/20251105161935.545400234%40kernel.org
patch subject: [PATCH 3/5] tracing: Have tracer option be instance specific
config: i386-randconfig-r072-20251107 (https://download.01.org/0day-ci/archive/20251107/202511071533.domGENBS-lkp@intel.com/config)
compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Reported-by: Dan Carpenter <dan.carpenter@linaro.org>
| Closes: https://lore.kernel.org/r/202511071533.domGENBS-lkp@intel.com/

smatch warnings:
kernel/trace/trace.c:6313 tracing_set_tracer() warn: iterator used outside loop: 't'

vim +/t +6313 kernel/trace/trace.c

9c5b9d3d65e4858 Masami Hiramatsu         2020-01-11  6285  int tracing_set_tracer(struct trace_array *tr, const char *buf)
09d23a1d8a82e81 Steven Rostedt (Red Hat  2015-02-03  6286) {
4699bbb369ba1d3 Steven Rostedt           2025-11-05  6287  	struct tracer *trace;
4699bbb369ba1d3 Steven Rostedt           2025-11-05  6288  	struct tracers *t;
12883efb670c28d Steven Rostedt (Red Hat  2013-03-05  6289) #ifdef CONFIG_TRACER_MAX_TRACE
34600f0e9c33c9c Steven Rostedt           2013-01-22  6290  	bool had_max_tr;
12883efb670c28d Steven Rostedt (Red Hat  2013-03-05  6291) #endif
d33b10c0c73adca Steven Rostedt           2024-12-24  6292  	int ret;
bc0c38d139ec7fc Steven Rostedt           2008-05-12  6293  
d33b10c0c73adca Steven Rostedt           2024-12-24  6294  	guard(mutex)(&trace_types_lock);
1027fcb206a0fb8 Steven Rostedt           2009-03-12  6295  
7a1d1e4b9639ff0 Steven Rostedt (Google   2024-06-12  6296) 	update_last_data(tr);
7a1d1e4b9639ff0 Steven Rostedt (Google   2024-06-12  6297) 
a1f157c7a3bb342 Zheng Yejian             2023-09-06  6298  	if (!tr->ring_buffer_expanded) {
2b6080f28c7cc3e Steven Rostedt           2012-05-11  6299  		ret = __tracing_resize_ring_buffer(tr, trace_buf_size,
438ced1720b5840 Vaibhav Nagarnaik        2012-02-02  6300  						RING_BUFFER_ALL_CPUS);
73c5162aa362a54 Steven Rostedt           2009-03-11  6301  		if (ret < 0)
d33b10c0c73adca Steven Rostedt           2024-12-24  6302  			return ret;
73c5162aa362a54 Steven Rostedt           2009-03-11  6303  		ret = 0;
73c5162aa362a54 Steven Rostedt           2009-03-11  6304  	}
73c5162aa362a54 Steven Rostedt           2009-03-11  6305  
4699bbb369ba1d3 Steven Rostedt           2025-11-05  6306  	list_for_each_entry(t, &tr->tracers, list) {
4699bbb369ba1d3 Steven Rostedt           2025-11-05  6307  		if (strcmp(t->tracer->name, buf) == 0)
bc0c38d139ec7fc Steven Rostedt           2008-05-12  6308  			break;
bc0c38d139ec7fc Steven Rostedt           2008-05-12  6309  	}
d33b10c0c73adca Steven Rostedt           2024-12-24  6310  	if (!t)

t can't be NULL here.  It needs to be if (list_entry_is_head()) return;

d33b10c0c73adca Steven Rostedt           2024-12-24  6311  		return -EINVAL;
d33b10c0c73adca Steven Rostedt           2024-12-24  6312  
4699bbb369ba1d3 Steven Rostedt           2025-11-05 @6313  	if (t->tracer == tr->current_trace)
d33b10c0c73adca Steven Rostedt           2024-12-24  6314  		return 0;
bc0c38d139ec7fc Steven Rostedt           2008-05-12  6315  
4699bbb369ba1d3 Steven Rostedt           2025-11-05  6316  	trace = t->tracer;
4699bbb369ba1d3 Steven Rostedt           2025-11-05  6317  

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH 3/5] tracing: Have tracer option be instance specific

[Steven Rostedt]

On Mon, 5 Jan 2026 18:23:31 +0300
Dan Carpenter <dan.carpenter@linaro.org> wrote:

> Hi Steven,

Hi Dan,

> 
> kernel test robot noticed the following build warnings:
> 
> https://git-scm.com/docs/git-format-patch#_base_tree_information]
> 
> url:    https://github.com/intel-lab-lkp/linux/commits/Steven-Rostedt/tracing-Remove-dummy-options-and-flags/20251106-010511
> base:   https://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace for-next
> patch link:    https://lore.kernel.org/r/20251105161935.545400234%40kernel.org

This is an old patch (v1) and v3 fixed this issue:
   https://lore.kernel.org/all/20251110234341.767135255@kernel.org/

 and v4 was applied and is now in mainline.

The patch you are reporting on was sent Nov 5th, the fix was sent Nov 10th.

Why is this sending reports about an old patch that is obsolete? Is there
something wrong with your setup?

-- Steve

> patch subject: [PATCH 3/5] tracing: Have tracer option be instance specific
> config: i386-randconfig-r072-20251107 (https://download.01.org/0day-ci/archive/20251107/202511071533.domGENBS-lkp@intel.com/config)
> compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)
> 
> If you fix the issue in a separate patch/commit (i.e. not just a new version of
> the same patch/commit), kindly add following tags
> | Reported-by: kernel test robot <lkp@intel.com>
> | Reported-by: Dan Carpenter <dan.carpenter@linaro.org>
> | Closes: https://lore.kernel.org/r/202511071533.domGENBS-lkp@intel.com/
> 
> smatch warnings:
> kernel/trace/trace.c:6313 tracing_set_tracer() warn: iterator used outside loop: 't'
> 
> vim +/t +6313 kernel/trace/trace.c
> 
> 9c5b9d3d65e4858 Masami Hiramatsu         2020-01-11  6285  int tracing_set_tracer(struct trace_array *tr, const char *buf)
> 09d23a1d8a82e81 Steven Rostedt (Red Hat  2015-02-03  6286) {
> 4699bbb369ba1d3 Steven Rostedt           2025-11-05  6287  	struct tracer *trace;
> 4699bbb369ba1d3 Steven Rostedt           2025-11-05  6288  	struct tracers *t;
> 12883efb670c28d Steven Rostedt (Red Hat  2013-03-05  6289) #ifdef CONFIG_TRACER_MAX_TRACE
> 34600f0e9c33c9c Steven Rostedt           2013-01-22  6290  	bool had_max_tr;
> 12883efb670c28d Steven Rostedt (Red Hat  2013-03-05  6291) #endif
> d33b10c0c73adca Steven Rostedt           2024-12-24  6292  	int ret;
> bc0c38d139ec7fc Steven Rostedt           2008-05-12  6293  
> d33b10c0c73adca Steven Rostedt           2024-12-24  6294  	guard(mutex)(&trace_types_lock);
> 1027fcb206a0fb8 Steven Rostedt           2009-03-12  6295  
> 7a1d1e4b9639ff0 Steven Rostedt (Google   2024-06-12  6296) 	update_last_data(tr);
> 7a1d1e4b9639ff0 Steven Rostedt (Google   2024-06-12  6297) 
> a1f157c7a3bb342 Zheng Yejian             2023-09-06  6298  	if (!tr->ring_buffer_expanded) {
> 2b6080f28c7cc3e Steven Rostedt           2012-05-11  6299  		ret = __tracing_resize_ring_buffer(tr, trace_buf_size,
> 438ced1720b5840 Vaibhav Nagarnaik        2012-02-02  6300  						RING_BUFFER_ALL_CPUS);
> 73c5162aa362a54 Steven Rostedt           2009-03-11  6301  		if (ret < 0)
> d33b10c0c73adca Steven Rostedt           2024-12-24  6302  			return ret;
> 73c5162aa362a54 Steven Rostedt           2009-03-11  6303  		ret = 0;
> 73c5162aa362a54 Steven Rostedt           2009-03-11  6304  	}
> 73c5162aa362a54 Steven Rostedt           2009-03-11  6305  
> 4699bbb369ba1d3 Steven Rostedt           2025-11-05  6306  	list_for_each_entry(t, &tr->tracers, list) {
> 4699bbb369ba1d3 Steven Rostedt           2025-11-05  6307  		if (strcmp(t->tracer->name, buf) == 0)
> bc0c38d139ec7fc Steven Rostedt           2008-05-12  6308  			break;
> bc0c38d139ec7fc Steven Rostedt           2008-05-12  6309  	}
> d33b10c0c73adca Steven Rostedt           2024-12-24  6310  	if (!t)
> 
> t can't be NULL here.  It needs to be if (list_entry_is_head()) return;
> 
> d33b10c0c73adca Steven Rostedt           2024-12-24  6311  		return -EINVAL;
> d33b10c0c73adca Steven Rostedt           2024-12-24  6312  
> 4699bbb369ba1d3 Steven Rostedt           2025-11-05 @6313  	if (t->tracer == tr->current_trace)
> d33b10c0c73adca Steven Rostedt           2024-12-24  6314  		return 0;
> bc0c38d139ec7fc Steven Rostedt           2008-05-12  6315  
> 4699bbb369ba1d3 Steven Rostedt           2025-11-05  6316  	trace = t->tracer;
> 4699bbb369ba1d3 Steven Rostedt           2025-11-05  6317  
> 

Re: [PATCH 3/5] tracing: Have tracer option be instance specific

[Dan Carpenter]

On Fri, Jan 09, 2026 at 10:23:01AM -0500, Steven Rostedt wrote:
> On Mon, 5 Jan 2026 18:23:31 +0300
> Dan Carpenter <dan.carpenter@linaro.org> wrote:
> 
> > Hi Steven,
> 
> Hi Dan,
> 
> > 
> > kernel test robot noticed the following build warnings:
> > 
> > https://git-scm.com/docs/git-format-patch#_base_tree_information]
> > 
> > url:    https://github.com/intel-lab-lkp/linux/commits/Steven-Rostedt/tracing-Remove-dummy-options-and-flags/20251106-010511
> > base:   https://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace for-next
> > patch link:    https://lore.kernel.org/r/20251105161935.545400234%40kernel.org
> 
> This is an old patch (v1) and v3 fixed this issue:
>    https://lore.kernel.org/all/20251110234341.767135255@kernel.org/
> 
>  and v4 was applied and is now in mainline.
> 
> The patch you are reporting on was sent Nov 5th, the fix was sent Nov 10th.
> 
> Why is this sending reports about an old patch that is obsolete? Is there
> something wrong with your setup?
> 
> -- Steve
>

Yeah.  I appologize.  That was my bad.  I accidentally was looking
at old emails.  I think this was the only one I sent, though.

regards,
dan carpenter