* Re: [PATCH] module/kallsyms: sort function symbols and use binary search
From: Petr Pavlu @ 2026-03-24 16:00 UTC (permalink / raw)
To: Stanislaw Gruszka
Cc: linux-modules, Sami Tolvanen, Luis Chamberlain, linux-kernel,
linux-trace-kernel, live-patching, Daniel Gomez, Aaron Tomlin,
Steven Rostedt, Masami Hiramatsu, Jordan Rome, Viktor Malik
In-Reply-To: <20260324125304.GA15972@wp.pl>
On 3/24/26 1:53 PM, Stanislaw Gruszka wrote:
> Hi,
>
> On Mon, Mar 23, 2026 at 02:06:43PM +0100, Petr Pavlu wrote:
>> On 3/17/26 12:04 PM, Stanislaw Gruszka wrote:
>>> Module symbol lookup via find_kallsyms_symbol() performs a linear scan
>>> over the entire symtab when resolving an address. The number of symbols
>>> in module symtabs has grown over the years, largely due to additional
>>> metadata in non-standard sections, making this lookup very slow.
>>>
>>> Improve this by separating function symbols during module load, placing
>>> them at the beginning of the symtab, sorting them by address, and using
>>> binary search when resolving addresses in module text.
>>
>> Doesn't considering only function symbols break the expected behavior
>> with CONFIG_KALLSYMS_ALL=y. For instance, when using kdb, is it still
>> able to see all symbols in a module? The module loader should be remain
>> consistent with the main kallsyms code regarding which symbols can be
>> looked up.
>
> We already have a CONFIG_KALLSYMS_ALL=y inconsistency between kernel and
> module symbol lookup, independent of this patch. find_kallsyms_symbol()
> restricts the search to MOD_TEXT (or MOD_INIT_TEXT) address ranges, so
> it cannot resolve data or rodata symbols.
My understanding is that find_kallsyms_symbol() can identify all symbols
in a module by their addresses. However, the issue I see with
MOD_TEXT/MOD_INIT_TEXT is that the function may incorrectly calculate
the size of symbols that are not within these ranges, which is a bug
that should be fixed.
A test using kdb confirms that non-text symbols can be found by their
addresses. The following shows the current behavior with 7.0-rc5 when
printing a module parameter in mlx4_en:
[1]kdb> mds __param_arr_num_vfs
0xffffffffc1209f20 0000000100000003 ........
0xffffffffc1209f28 ffffffffc0fbf07c [mlx4_core]num_vfs_argc
0xffffffffc1209f30 ffffffff8844bba0 param_ops_byte
0xffffffffc1209f38 ffffffffc0fbf080 [mlx4_core]num_vfs
0xffffffffc1209f40 000000785f69736d msi_x...
0xffffffffc1209f48 656c5f6775626564 debug_le
0xffffffffc1209f50 00000000006c6576 vel.....
0xffffffffc1209f58 0000000000000000 ........
.. and the behavior with the proposed patch:
[1]kdb> mds __param_arr_num_vfs
0xffffffffc1077f20 0000000100000003 ........
0xffffffffc1077f28 ffffffffc104707c |p......
0xffffffffc1077f30 ffffffffb4a4bba0 param_ops_byte
0xffffffffc1077f38 ffffffffc1047080 .p......
0xffffffffc1077f40 000000785f69736d msi_x...
0xffffffffc1077f48 656c5f6775626564 debug_le
0xffffffffc1077f50 00000000006c6576 vel.....
0xffffffffc1077f58 0000000000000000 ........
--
Thanks,
Petr
^ permalink raw reply
* Re: [PATCH] tracing/osnoise: fix potential deadlock in cpu hotplug
From: Steven Rostedt @ 2026-03-24 16:19 UTC (permalink / raw)
To: hu.shengming
Cc: mhiramat, mathieu.desnoyers, linux-kernel, linux-trace-kernel,
zhang.run, yang.tao172, ran.xiaokai, luo.haiyang
In-Reply-To: <20260324150616953rMo1BWtAZ1nXTNrEFP6hr@zte.com.cn>
On Tue, 24 Mar 2026 15:06:16 +0800 (CST)
<hu.shengming@zte.com.cn> wrote:
> From: luohaiyang10243395 <luo.haiyang@zte.com.cn>
>
> The following sequence may leads deadlock in cpu hotplug:
>
> CPU0 | CPU1
> | schedule_work_on
> |
> _cpu_down//set CPU1 offline |
> cpus_write_lock |
> | osnoise_hotplug_workfn
> | mutex_lock(&interface_lock);
> | cpus_read_lock(); //wait cpu_hotplug_lock
> |
> | cpuhp/1
> | osnoise_cpu_die
> | kthread_stop
> | wait_for_completion //wait osnoise/1 exit
> |
> | osnoise/1
> | osnoise_sleep
> | mutex_lock(&interface_lock); //deadlock
>
> Fix by swap the order of cpus_read_lock() and mutex_lock(&interface_lock).
So the deadlock is due to the "wait_for_completion"?
How did you find this bug? Inspection, AI, triggered?
Thanks,
-- Steve
>
> Signed-off-by: Luo Haiyang <luo.haiyang@zte.com.cn>
> ---
> kernel/trace/trace_osnoise.c | 10 +++++-----
> 1 file changed, 5 insertions(+), 5 deletions(-)
>
> diff --git a/kernel/trace/trace_osnoise.c b/kernel/trace/trace_osnoise.c
> index dee610e465b9..be6cf0bb3c03 100644
> --- a/kernel/trace/trace_osnoise.c
> +++ b/kernel/trace/trace_osnoise.c
> @@ -2073,8 +2073,8 @@ static void osnoise_hotplug_workfn(struct work_struct *dummy)
> if (!osnoise_has_registered_instances())
> return;
>
> - guard(mutex)(&interface_lock);
> guard(cpus_read_lock)();
> + guard(mutex)(&interface_lock);
>
> if (!cpu_online(cpu))
> return;
> @@ -2237,11 +2237,11 @@ static ssize_t osnoise_options_write(struct file *filp, const char __user *ubuf,
> if (running)
> stop_per_cpu_kthreads();
>
> - mutex_lock(&interface_lock);
> /*
> * avoid CPU hotplug operations that might read options.
> */
> cpus_read_lock();
> + mutex_lock(&interface_lock);
>
> retval = cnt;
>
> @@ -2257,8 +2257,8 @@ static ssize_t osnoise_options_write(struct file *filp, const char __user *ubuf,
> clear_bit(option, &osnoise_options);
> }
>
> - cpus_read_unlock();
> mutex_unlock(&interface_lock);
> + cpus_read_unlock();
>
> if (running)
> start_per_cpu_kthreads();
> @@ -2345,16 +2345,16 @@ osnoise_cpus_write(struct file *filp, const char __user *ubuf, size_t count,
> if (running)
> stop_per_cpu_kthreads();
>
> - mutex_lock(&interface_lock);
> /*
> * osnoise_cpumask is read by CPU hotplug operations.
> */
> cpus_read_lock();
> + mutex_lock(&interface_lock);
>
> cpumask_copy(&osnoise_cpumask, osnoise_cpumask_new);
>
> - cpus_read_unlock();
> mutex_unlock(&interface_lock);
> + cpus_read_unlock();
>
> if (running)
> start_per_cpu_kthreads();
^ permalink raw reply
* Re: [PATCH] Documentation/rtla: Document SIGINT behavior
From: Tomas Glozar @ 2026-03-24 16:30 UTC (permalink / raw)
To: Steven Rostedt
Cc: Jonathan Corbet, Shuah Khan, John Kacur, Luis Goncalves,
Crystal Wood, Costa Shulyupin, Wander Lairson Costa, LKML,
linux-trace-kernel, linux-doc, Attila Fazekas
In-Reply-To: <20260324112249.5fe25641@gandalf.local.home>
út 24. 3. 2026 v 16:22 odesílatel Steven Rostedt <rostedt@goodmis.org> napsal:
> >
> > Note: There was a bug in SIGINT behavior, fixed in upcoming commit [1].
> >
> > [1] https://lore.kernel.org/linux-trace-kernel/20260310160725.144443-1-tglozar@redhat.com/
>
> Hmm, this may be interesting enough to add to the change log itself.
>
I thought about that, but it felt a bit redundant, since the other
patch will also be a part of the commit history. I see that some
documentation patches do mention the commit that introduced the
change. Here, it is only the SIGINT during cleanup part that got
changed (segfault/undefined behavior -> default handler) so it might
make sense.
> > +Also note that when using the timerlat tool in BPF mode, samples are processed
> > +in-kernel; RTLA only copies them out to display them to the user. A second
> > +SIGINT does not affect in-kernel sample aggregation.
>
> But does it affect the user space side of reading that information?
>
No, it shouldn't, the pattern for both timerlat-top and timerlat-hist is:
while(stop_tracing) {
timerlat_bpf_wait(...);
timerlat_{top,hist}_pull_bpf_data(...);
...
}
The BPF program is detached after this, so all data gathered up to the
point of the SIGINT will be displayed. A second SIGINT does not affect
it, since it calls tracefs_iterate_stop() which stops
tracefs_iterate_raw_events(). It does not stop
timelat_{top,hist}_pull_bpf_data() - which is aggregated anyway, so
stopping it would just leave a part of the histogram/top empty.
> > +
> > EXIT STATUS
> > ===========
> >
>
> Other than that ... LGTM,
>
> Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>
>
> -- Steve
>
Thanks for looking at this.
Tomas
^ permalink raw reply
* Re: [PATCH v8 3/6] tracefs: Check file permission even if user has CAP_DAC_OVERRIDE
From: Steven Rostedt @ 2026-03-24 16:43 UTC (permalink / raw)
To: Masami Hiramatsu (Google)
Cc: Mathieu Desnoyers, linux-kernel, linux-trace-kernel
In-Reply-To: <177071303130.2293046.2400906233143699263.stgit@mhiramat.tok.corp.google.com>
On Tue, 10 Feb 2026 17:43:51 +0900
"Masami Hiramatsu (Google)" <mhiramat@kernel.org> wrote:
Hi Masami,
Did you send a new version of this patch series yet? I don't see it.
> diff --git a/fs/tracefs/inode.c b/fs/tracefs/inode.c
> index d9d8932a7b9c..eb1ddc0cc13a 100644
> --- a/fs/tracefs/inode.c
> +++ b/fs/tracefs/inode.c
> @@ -212,10 +212,40 @@ static void set_tracefs_inode_owner(struct inode *inode)
> inode->i_gid = gid;
> }
>
> -static int tracefs_permission(struct mnt_idmap *idmap,
> - struct inode *inode, int mask)
> +int tracefs_permission(struct mnt_idmap *idmap,
> + struct inode *inode, int mask)
> {
> - set_tracefs_inode_owner(inode);
> + struct tracefs_inode *ti = get_tracefs(inode);
> + const struct file_operations *fops;
> +
> + if (!(ti->flags & TRACEFS_EVENT_INODE))
> + set_tracefs_inode_owner(inode);
> +
> + /*
> + * Like sysfs, file permission checks are performed even for superuser
> + * with CAP_DAC_OVERRIDE. See the KERNFS_ROOT_EXTRA_OPEN_PERM_CHECK
> + * definition in linux/kernfs.h.
> + */
> + if (mask & MAY_OPEN) {
> + fops = inode->i_fop;
> +
> + if (mask & MAY_WRITE) {
> + if (!(inode->i_mode & 0222))
> + return -EACCES;
> + if (!fops || (!fops->write && !fops->write_iter &&
> + !fops->mmap))
> + return -EACCES;
> + }
> +
> + if (mask & MAY_READ) {
> + if (!(inode->i_mode & 0444))
> + return -EACCES;
> + if (!fops || (!fops->read && !fops->read_iter &&
> + !fops->mmap && !fops->splice_read))
> + return -EACCES;
> + }
The above if block is way too coupled with the workings of fops and is very
fragile. Is it even needed? If there are no read or write functions,
wouldn't the vfs stop it anyway?
-- Steve
> + }
> +
> return generic_permission(idmap, inode, mask);
> }
^ permalink raw reply
* [PATCH v3] tracing: Move snapshot code out of trace.c and into trace_snapshot.c
From: Steven Rostedt @ 2026-03-24 18:01 UTC (permalink / raw)
To: LKML, Linux Trace Kernel; +Cc: Masami Hiramatsu, Mathieu Desnoyers
From: Steven Rostedt <rostedt@goodmis.org>
The trace.c file was a dumping ground for most tracing code. Start
organizing it better by moving various functions out into their own files.
Move all the snapshot code, including the max trace code into its own
trace_snapshot.c file.
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
---
Changes since v2: https://patch.msgid.link/20260305211810.3f48aa07@robin
- Rebased on top of 7.0-rc5 (and other patches added)
include/linux/ftrace.h | 2 +-
kernel/trace/Makefile | 1 +
kernel/trace/trace.c | 1274 ++-------------------------------
kernel/trace/trace.h | 105 ++-
kernel/trace/trace_snapshot.c | 1067 +++++++++++++++++++++++++++
5 files changed, 1230 insertions(+), 1219 deletions(-)
create mode 100644 kernel/trace/trace_snapshot.c
diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h
index c242fe49af4c..28b30c6f1031 100644
--- a/include/linux/ftrace.h
+++ b/include/linux/ftrace.h
@@ -31,7 +31,7 @@
#define ARCH_SUPPORTS_FTRACE_OPS 0
#endif
-#ifdef CONFIG_TRACING
+#ifdef CONFIG_TRACER_SNAPSHOT
extern void ftrace_boot_snapshot(void);
#else
static inline void ftrace_boot_snapshot(void) { }
diff --git a/kernel/trace/Makefile b/kernel/trace/Makefile
index 04096c21d06b..83aeb5c77008 100644
--- a/kernel/trace/Makefile
+++ b/kernel/trace/Makefile
@@ -69,6 +69,7 @@ obj-$(CONFIG_TRACING) += trace_seq.o
obj-$(CONFIG_TRACING) += trace_stat.o
obj-$(CONFIG_TRACING) += trace_printk.o
obj-$(CONFIG_TRACING) += trace_pid.o
+obj-$(CONFIG_TRACER_SNAPSHOT) += trace_snapshot.o
obj-$(CONFIG_TRACING) += pid_list.o
obj-$(CONFIG_TRACING_MAP) += tracing_map.o
obj-$(CONFIG_PREEMPTIRQ_DELAY_TEST) += preemptirq_delay_test.o
diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index cf48fe23e71f..850f62032fd2 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -47,7 +47,6 @@
#include <linux/trace.h>
#include <linux/sched/clock.h>
#include <linux/sched/rt.h>
-#include <linux/fsnotify.h>
#include <linux/irq_work.h>
#include <linux/workqueue.h>
#include <linux/sort.h>
@@ -219,15 +218,9 @@ static void ftrace_trace_userstack(struct trace_array *tr,
static char bootup_tracer_buf[MAX_TRACER_SIZE] __initdata;
static char *default_bootup_tracer;
-static bool allocate_snapshot;
-static bool snapshot_at_boot;
-
static char boot_instance_info[COMMAND_LINE_SIZE] __initdata;
static int boot_instance_index;
-static char boot_snapshot_info[COMMAND_LINE_SIZE] __initdata;
-static int boot_snapshot_index;
-
static int __init set_cmdline_ftrace(char *str)
{
strscpy(bootup_tracer_buf, str, MAX_TRACER_SIZE);
@@ -276,38 +269,6 @@ static int __init stop_trace_on_warning(char *str)
}
__setup("traceoff_on_warning", stop_trace_on_warning);
-static int __init boot_alloc_snapshot(char *str)
-{
- char *slot = boot_snapshot_info + boot_snapshot_index;
- int left = sizeof(boot_snapshot_info) - boot_snapshot_index;
- int ret;
-
- if (str[0] == '=') {
- str++;
- if (strlen(str) >= left)
- return -1;
-
- ret = snprintf(slot, left, "%s\t", str);
- boot_snapshot_index += ret;
- } else {
- allocate_snapshot = true;
- /* We also need the main ring buffer expanded */
- trace_set_ring_buffer_expanded(NULL);
- }
- return 1;
-}
-__setup("alloc_snapshot", boot_alloc_snapshot);
-
-
-static int __init boot_snapshot(char *str)
-{
- snapshot_at_boot = true;
- boot_alloc_snapshot(str);
- return 1;
-}
-__setup("ftrace_boot_snapshot", boot_snapshot);
-
-
static int __init boot_instance(char *str)
{
char *slot = boot_instance_info + boot_instance_index;
@@ -807,47 +768,6 @@ void tracing_on(void)
EXPORT_SYMBOL_GPL(tracing_on);
#ifdef CONFIG_TRACER_SNAPSHOT
-static void tracing_snapshot_instance_cond(struct trace_array *tr,
- void *cond_data)
-{
- unsigned long flags;
-
- if (in_nmi()) {
- trace_array_puts(tr, "*** SNAPSHOT CALLED FROM NMI CONTEXT ***\n");
- trace_array_puts(tr, "*** snapshot is being ignored ***\n");
- return;
- }
-
- if (!tr->allocated_snapshot) {
- trace_array_puts(tr, "*** SNAPSHOT NOT ALLOCATED ***\n");
- trace_array_puts(tr, "*** stopping trace here! ***\n");
- tracer_tracing_off(tr);
- return;
- }
-
- if (tr->mapped) {
- trace_array_puts(tr, "*** BUFFER MEMORY MAPPED ***\n");
- trace_array_puts(tr, "*** Can not use snapshot (sorry) ***\n");
- return;
- }
-
- /* Note, snapshot can not be used when the tracer uses it */
- if (tracer_uses_snapshot(tr->current_trace)) {
- trace_array_puts(tr, "*** LATENCY TRACER ACTIVE ***\n");
- trace_array_puts(tr, "*** Can not use snapshot (sorry) ***\n");
- return;
- }
-
- local_irq_save(flags);
- update_max_tr(tr, current, smp_processor_id(), cond_data);
- local_irq_restore(flags);
-}
-
-void tracing_snapshot_instance(struct trace_array *tr)
-{
- tracing_snapshot_instance_cond(tr, NULL);
-}
-
/**
* tracing_snapshot - take a snapshot of the current buffer.
*
@@ -870,138 +790,6 @@ void tracing_snapshot(void)
}
EXPORT_SYMBOL_GPL(tracing_snapshot);
-/**
- * tracing_snapshot_cond - conditionally take a snapshot of the current buffer.
- * @tr: The tracing instance to snapshot
- * @cond_data: The data to be tested conditionally, and possibly saved
- *
- * This is the same as tracing_snapshot() except that the snapshot is
- * conditional - the snapshot will only happen if the
- * cond_snapshot.update() implementation receiving the cond_data
- * returns true, which means that the trace array's cond_snapshot
- * update() operation used the cond_data to determine whether the
- * snapshot should be taken, and if it was, presumably saved it along
- * with the snapshot.
- */
-void tracing_snapshot_cond(struct trace_array *tr, void *cond_data)
-{
- tracing_snapshot_instance_cond(tr, cond_data);
-}
-EXPORT_SYMBOL_GPL(tracing_snapshot_cond);
-
-/**
- * tracing_cond_snapshot_data - get the user data associated with a snapshot
- * @tr: The tracing instance
- *
- * When the user enables a conditional snapshot using
- * tracing_snapshot_cond_enable(), the user-defined cond_data is saved
- * with the snapshot. This accessor is used to retrieve it.
- *
- * Should not be called from cond_snapshot.update(), since it takes
- * the tr->max_lock lock, which the code calling
- * cond_snapshot.update() has already done.
- *
- * Returns the cond_data associated with the trace array's snapshot.
- */
-void *tracing_cond_snapshot_data(struct trace_array *tr)
-{
- void *cond_data = NULL;
-
- local_irq_disable();
- arch_spin_lock(&tr->max_lock);
-
- if (tr->cond_snapshot)
- cond_data = tr->cond_snapshot->cond_data;
-
- arch_spin_unlock(&tr->max_lock);
- local_irq_enable();
-
- return cond_data;
-}
-EXPORT_SYMBOL_GPL(tracing_cond_snapshot_data);
-
-static int resize_buffer_duplicate_size(struct array_buffer *trace_buf,
- struct array_buffer *size_buf, int cpu_id);
-static void set_buffer_entries(struct array_buffer *buf, unsigned long val);
-
-int tracing_alloc_snapshot_instance(struct trace_array *tr)
-{
- int order;
- int ret;
-
- if (!tr->allocated_snapshot) {
-
- /* Make the snapshot buffer have the same order as main buffer */
- order = ring_buffer_subbuf_order_get(tr->array_buffer.buffer);
- ret = ring_buffer_subbuf_order_set(tr->snapshot_buffer.buffer, order);
- if (ret < 0)
- return ret;
-
- /* allocate spare buffer */
- ret = resize_buffer_duplicate_size(&tr->snapshot_buffer,
- &tr->array_buffer, RING_BUFFER_ALL_CPUS);
- if (ret < 0)
- return ret;
-
- tr->allocated_snapshot = true;
- }
-
- return 0;
-}
-
-static void free_snapshot(struct trace_array *tr)
-{
- /*
- * We don't free the ring buffer. instead, resize it because
- * The max_tr ring buffer has some state (e.g. ring->clock) and
- * we want preserve it.
- */
- ring_buffer_subbuf_order_set(tr->snapshot_buffer.buffer, 0);
- ring_buffer_resize(tr->snapshot_buffer.buffer, 1, RING_BUFFER_ALL_CPUS);
- set_buffer_entries(&tr->snapshot_buffer, 1);
- tracing_reset_online_cpus(&tr->snapshot_buffer);
- tr->allocated_snapshot = false;
-}
-
-static int tracing_arm_snapshot_locked(struct trace_array *tr)
-{
- int ret;
-
- lockdep_assert_held(&trace_types_lock);
-
- spin_lock(&tr->snapshot_trigger_lock);
- if (tr->snapshot == UINT_MAX || tr->mapped) {
- spin_unlock(&tr->snapshot_trigger_lock);
- return -EBUSY;
- }
-
- tr->snapshot++;
- spin_unlock(&tr->snapshot_trigger_lock);
-
- ret = tracing_alloc_snapshot_instance(tr);
- if (ret) {
- spin_lock(&tr->snapshot_trigger_lock);
- tr->snapshot--;
- spin_unlock(&tr->snapshot_trigger_lock);
- }
-
- return ret;
-}
-
-int tracing_arm_snapshot(struct trace_array *tr)
-{
- guard(mutex)(&trace_types_lock);
- return tracing_arm_snapshot_locked(tr);
-}
-
-void tracing_disarm_snapshot(struct trace_array *tr)
-{
- spin_lock(&tr->snapshot_trigger_lock);
- if (!WARN_ON(!tr->snapshot))
- tr->snapshot--;
- spin_unlock(&tr->snapshot_trigger_lock);
-}
-
/**
* tracing_alloc_snapshot - allocate snapshot buffer.
*
@@ -1023,129 +811,12 @@ int tracing_alloc_snapshot(void)
return ret;
}
EXPORT_SYMBOL_GPL(tracing_alloc_snapshot);
-
-/**
- * tracing_snapshot_alloc - allocate and take a snapshot of the current buffer.
- *
- * This is similar to tracing_snapshot(), but it will allocate the
- * snapshot buffer if it isn't already allocated. Use this only
- * where it is safe to sleep, as the allocation may sleep.
- *
- * This causes a swap between the snapshot buffer and the current live
- * tracing buffer. You can use this to take snapshots of the live
- * trace when some condition is triggered, but continue to trace.
- */
-void tracing_snapshot_alloc(void)
-{
- int ret;
-
- ret = tracing_alloc_snapshot();
- if (ret < 0)
- return;
-
- tracing_snapshot();
-}
-EXPORT_SYMBOL_GPL(tracing_snapshot_alloc);
-
-/**
- * tracing_snapshot_cond_enable - enable conditional snapshot for an instance
- * @tr: The tracing instance
- * @cond_data: User data to associate with the snapshot
- * @update: Implementation of the cond_snapshot update function
- *
- * Check whether the conditional snapshot for the given instance has
- * already been enabled, or if the current tracer is already using a
- * snapshot; if so, return -EBUSY, else create a cond_snapshot and
- * save the cond_data and update function inside.
- *
- * Returns 0 if successful, error otherwise.
- */
-int tracing_snapshot_cond_enable(struct trace_array *tr, void *cond_data,
- cond_update_fn_t update)
-{
- struct cond_snapshot *cond_snapshot __free(kfree) =
- kzalloc_obj(*cond_snapshot);
- int ret;
-
- if (!cond_snapshot)
- return -ENOMEM;
-
- cond_snapshot->cond_data = cond_data;
- cond_snapshot->update = update;
-
- guard(mutex)(&trace_types_lock);
-
- if (tracer_uses_snapshot(tr->current_trace))
- return -EBUSY;
-
- /*
- * The cond_snapshot can only change to NULL without the
- * trace_types_lock. We don't care if we race with it going
- * to NULL, but we want to make sure that it's not set to
- * something other than NULL when we get here, which we can
- * do safely with only holding the trace_types_lock and not
- * having to take the max_lock.
- */
- if (tr->cond_snapshot)
- return -EBUSY;
-
- ret = tracing_arm_snapshot_locked(tr);
- if (ret)
- return ret;
-
- local_irq_disable();
- arch_spin_lock(&tr->max_lock);
- tr->cond_snapshot = no_free_ptr(cond_snapshot);
- arch_spin_unlock(&tr->max_lock);
- local_irq_enable();
-
- return 0;
-}
-EXPORT_SYMBOL_GPL(tracing_snapshot_cond_enable);
-
-/**
- * tracing_snapshot_cond_disable - disable conditional snapshot for an instance
- * @tr: The tracing instance
- *
- * Check whether the conditional snapshot for the given instance is
- * enabled; if so, free the cond_snapshot associated with it,
- * otherwise return -EINVAL.
- *
- * Returns 0 if successful, error otherwise.
- */
-int tracing_snapshot_cond_disable(struct trace_array *tr)
-{
- int ret = 0;
-
- local_irq_disable();
- arch_spin_lock(&tr->max_lock);
-
- if (!tr->cond_snapshot)
- ret = -EINVAL;
- else {
- kfree(tr->cond_snapshot);
- tr->cond_snapshot = NULL;
- }
-
- arch_spin_unlock(&tr->max_lock);
- local_irq_enable();
-
- tracing_disarm_snapshot(tr);
-
- return ret;
-}
-EXPORT_SYMBOL_GPL(tracing_snapshot_cond_disable);
#else
void tracing_snapshot(void)
{
WARN_ONCE(1, "Snapshot feature not enabled, but internal snapshot used");
}
EXPORT_SYMBOL_GPL(tracing_snapshot);
-void tracing_snapshot_cond(struct trace_array *tr, void *cond_data)
-{
- WARN_ONCE(1, "Snapshot feature not enabled, but internal conditional snapshot used");
-}
-EXPORT_SYMBOL_GPL(tracing_snapshot_cond);
int tracing_alloc_snapshot(void)
{
WARN_ONCE(1, "Snapshot feature not enabled, but snapshot allocation used");
@@ -1158,23 +829,6 @@ void tracing_snapshot_alloc(void)
tracing_snapshot();
}
EXPORT_SYMBOL_GPL(tracing_snapshot_alloc);
-void *tracing_cond_snapshot_data(struct trace_array *tr)
-{
- return NULL;
-}
-EXPORT_SYMBOL_GPL(tracing_cond_snapshot_data);
-int tracing_snapshot_cond_enable(struct trace_array *tr, void *cond_data, cond_update_fn_t update)
-{
- return -ENODEV;
-}
-EXPORT_SYMBOL_GPL(tracing_snapshot_cond_enable);
-int tracing_snapshot_cond_disable(struct trace_array *tr)
-{
- return false;
-}
-EXPORT_SYMBOL_GPL(tracing_snapshot_cond_disable);
-#define free_snapshot(tr) do { } while (0)
-#define tracing_arm_snapshot_locked(tr) ({ -EBUSY; })
#endif /* CONFIG_TRACER_SNAPSHOT */
void tracer_tracing_off(struct trace_array *tr)
@@ -1487,206 +1141,6 @@ static ssize_t trace_seq_to_buffer(struct trace_seq *s, void *buf, size_t cnt)
unsigned long __read_mostly tracing_thresh;
-#ifdef CONFIG_TRACER_MAX_TRACE
-#ifdef LATENCY_FS_NOTIFY
-static struct workqueue_struct *fsnotify_wq;
-
-static void latency_fsnotify_workfn(struct work_struct *work)
-{
- struct trace_array *tr = container_of(work, struct trace_array,
- fsnotify_work);
- fsnotify_inode(tr->d_max_latency->d_inode, FS_MODIFY);
-}
-
-static void latency_fsnotify_workfn_irq(struct irq_work *iwork)
-{
- struct trace_array *tr = container_of(iwork, struct trace_array,
- fsnotify_irqwork);
- queue_work(fsnotify_wq, &tr->fsnotify_work);
-}
-
-__init static int latency_fsnotify_init(void)
-{
- fsnotify_wq = alloc_workqueue("tr_max_lat_wq",
- WQ_UNBOUND | WQ_HIGHPRI, 0);
- if (!fsnotify_wq) {
- pr_err("Unable to allocate tr_max_lat_wq\n");
- return -ENOMEM;
- }
- return 0;
-}
-
-late_initcall_sync(latency_fsnotify_init);
-
-void latency_fsnotify(struct trace_array *tr)
-{
- if (!fsnotify_wq)
- return;
- /*
- * We cannot call queue_work(&tr->fsnotify_work) from here because it's
- * possible that we are called from __schedule() or do_idle(), which
- * could cause a deadlock.
- */
- irq_work_queue(&tr->fsnotify_irqwork);
-}
-#endif /* !LATENCY_FS_NOTIFY */
-
-static const struct file_operations tracing_max_lat_fops;
-
-static void trace_create_maxlat_file(struct trace_array *tr,
- struct dentry *d_tracer)
-{
-#ifdef LATENCY_FS_NOTIFY
- INIT_WORK(&tr->fsnotify_work, latency_fsnotify_workfn);
- init_irq_work(&tr->fsnotify_irqwork, latency_fsnotify_workfn_irq);
-#endif
- tr->d_max_latency = trace_create_file("tracing_max_latency",
- TRACE_MODE_WRITE,
- d_tracer, tr,
- &tracing_max_lat_fops);
-}
-
-/*
- * Copy the new maximum trace into the separate maximum-trace
- * structure. (this way the maximum trace is permanently saved,
- * for later retrieval via /sys/kernel/tracing/tracing_max_latency)
- */
-static void
-__update_max_tr(struct trace_array *tr, struct task_struct *tsk, int cpu)
-{
- struct array_buffer *trace_buf = &tr->array_buffer;
- struct trace_array_cpu *data = per_cpu_ptr(trace_buf->data, cpu);
- struct array_buffer *max_buf = &tr->snapshot_buffer;
- struct trace_array_cpu *max_data = per_cpu_ptr(max_buf->data, cpu);
-
- max_buf->cpu = cpu;
- max_buf->time_start = data->preempt_timestamp;
-
- max_data->saved_latency = tr->max_latency;
- max_data->critical_start = data->critical_start;
- max_data->critical_end = data->critical_end;
-
- strscpy(max_data->comm, tsk->comm);
- max_data->pid = tsk->pid;
- /*
- * If tsk == current, then use current_uid(), as that does not use
- * RCU. The irq tracer can be called out of RCU scope.
- */
- if (tsk == current)
- max_data->uid = current_uid();
- else
- max_data->uid = task_uid(tsk);
-
- max_data->nice = tsk->static_prio - 20 - MAX_RT_PRIO;
- max_data->policy = tsk->policy;
- max_data->rt_priority = tsk->rt_priority;
-
- /* record this tasks comm */
- tracing_record_cmdline(tsk);
- latency_fsnotify(tr);
-}
-#else
-static inline void trace_create_maxlat_file(struct trace_array *tr,
- struct dentry *d_tracer) { }
-static inline void __update_max_tr(struct trace_array *tr,
- struct task_struct *tsk, int cpu) { }
-#endif /* CONFIG_TRACER_MAX_TRACE */
-
-#ifdef CONFIG_TRACER_SNAPSHOT
-/**
- * update_max_tr - snapshot all trace buffers from global_trace to max_tr
- * @tr: tracer
- * @tsk: the task with the latency
- * @cpu: The cpu that initiated the trace.
- * @cond_data: User data associated with a conditional snapshot
- *
- * Flip the buffers between the @tr and the max_tr and record information
- * about which task was the cause of this latency.
- */
-void
-update_max_tr(struct trace_array *tr, struct task_struct *tsk, int cpu,
- void *cond_data)
-{
- if (tr->stop_count)
- return;
-
- WARN_ON_ONCE(!irqs_disabled());
-
- if (!tr->allocated_snapshot) {
- /* Only the nop tracer should hit this when disabling */
- WARN_ON_ONCE(tr->current_trace != &nop_trace);
- return;
- }
-
- arch_spin_lock(&tr->max_lock);
-
- /* Inherit the recordable setting from array_buffer */
- if (ring_buffer_record_is_set_on(tr->array_buffer.buffer))
- ring_buffer_record_on(tr->snapshot_buffer.buffer);
- else
- ring_buffer_record_off(tr->snapshot_buffer.buffer);
-
- if (tr->cond_snapshot && !tr->cond_snapshot->update(tr, cond_data)) {
- arch_spin_unlock(&tr->max_lock);
- return;
- }
-
- swap(tr->array_buffer.buffer, tr->snapshot_buffer.buffer);
-
- __update_max_tr(tr, tsk, cpu);
-
- arch_spin_unlock(&tr->max_lock);
-
- /* Any waiters on the old snapshot buffer need to wake up */
- ring_buffer_wake_waiters(tr->array_buffer.buffer, RING_BUFFER_ALL_CPUS);
-}
-
-/**
- * update_max_tr_single - only copy one trace over, and reset the rest
- * @tr: tracer
- * @tsk: task with the latency
- * @cpu: the cpu of the buffer to copy.
- *
- * Flip the trace of a single CPU buffer between the @tr and the max_tr.
- */
-void
-update_max_tr_single(struct trace_array *tr, struct task_struct *tsk, int cpu)
-{
- int ret;
-
- if (tr->stop_count)
- return;
-
- WARN_ON_ONCE(!irqs_disabled());
- if (!tr->allocated_snapshot) {
- /* Only the nop tracer should hit this when disabling */
- WARN_ON_ONCE(tr->current_trace != &nop_trace);
- return;
- }
-
- arch_spin_lock(&tr->max_lock);
-
- ret = ring_buffer_swap_cpu(tr->snapshot_buffer.buffer, tr->array_buffer.buffer, cpu);
-
- if (ret == -EBUSY) {
- /*
- * We failed to swap the buffer due to a commit taking
- * place on this CPU. We fail to record, but we reset
- * the max trace buffer (no one writes directly to it)
- * and flag that it failed.
- * Another reason is resize is in progress.
- */
- trace_array_printk_buf(tr->snapshot_buffer.buffer, _THIS_IP_,
- "Failed to swap buffers due to commit or resize in progress\n");
- }
-
- WARN_ON_ONCE(ret && ret != -EAGAIN && ret != -EBUSY);
-
- __update_max_tr(tr, tsk, cpu);
- arch_spin_unlock(&tr->max_lock);
-}
-#endif /* CONFIG_TRACER_SNAPSHOT */
-
struct pipe_wait {
struct trace_iterator *iter;
int wait_index;
@@ -1995,7 +1449,7 @@ int __init register_tracer(struct tracer *type)
return 0;
}
-static void tracing_reset_cpu(struct array_buffer *buf, int cpu)
+void tracing_reset_cpu(struct array_buffer *buf, int cpu)
{
struct trace_buffer *buffer = buf->buffer;
@@ -3760,50 +3214,6 @@ static void test_ftrace_alive(struct seq_file *m)
"# MAY BE MISSING FUNCTION EVENTS\n");
}
-#ifdef CONFIG_TRACER_SNAPSHOT
-static void show_snapshot_main_help(struct seq_file *m)
-{
- seq_puts(m, "# echo 0 > snapshot : Clears and frees snapshot buffer\n"
- "# echo 1 > snapshot : Allocates snapshot buffer, if not already allocated.\n"
- "# Takes a snapshot of the main buffer.\n"
- "# echo 2 > snapshot : Clears snapshot buffer (but does not allocate or free)\n"
- "# (Doesn't have to be '2' works with any number that\n"
- "# is not a '0' or '1')\n");
-}
-
-static void show_snapshot_percpu_help(struct seq_file *m)
-{
- seq_puts(m, "# echo 0 > snapshot : Invalid for per_cpu snapshot file.\n");
-#ifdef CONFIG_RING_BUFFER_ALLOW_SWAP
- seq_puts(m, "# echo 1 > snapshot : Allocates snapshot buffer, if not already allocated.\n"
- "# Takes a snapshot of the main buffer for this cpu.\n");
-#else
- seq_puts(m, "# echo 1 > snapshot : Not supported with this kernel.\n"
- "# Must use main snapshot file to allocate.\n");
-#endif
- seq_puts(m, "# echo 2 > snapshot : Clears this cpu's snapshot buffer (but does not allocate)\n"
- "# (Doesn't have to be '2' works with any number that\n"
- "# is not a '0' or '1')\n");
-}
-
-static void print_snapshot_help(struct seq_file *m, struct trace_iterator *iter)
-{
- if (iter->tr->allocated_snapshot)
- seq_puts(m, "#\n# * Snapshot is allocated *\n#\n");
- else
- seq_puts(m, "#\n# * Snapshot is freed *\n#\n");
-
- seq_puts(m, "# Snapshot commands:\n");
- if (iter->cpu_file == RING_BUFFER_ALL_CPUS)
- show_snapshot_main_help(m);
- else
- show_snapshot_percpu_help(m);
-}
-#else
-/* Should never be called */
-static inline void print_snapshot_help(struct seq_file *m, struct trace_iterator *iter) { }
-#endif
-
static int s_show(struct seq_file *m, void *v)
{
struct trace_iterator *iter = v;
@@ -3852,17 +3262,6 @@ static int s_show(struct seq_file *m, void *v)
return 0;
}
-/*
- * Should be used after trace_array_get(), trace_types_lock
- * ensures that i_cdev was already initialized.
- */
-static inline int tracing_get_cpu(struct inode *inode)
-{
- if (inode->i_cdev) /* See trace_create_cpu_file() */
- return (long)inode->i_cdev - 1;
- return RING_BUFFER_ALL_CPUS;
-}
-
static const struct seq_operations tracer_seq_ops = {
.start = s_start,
.next = s_next,
@@ -3889,7 +3288,7 @@ static void free_trace_iter_content(struct trace_iterator *iter)
free_cpumask_var(iter->started);
}
-static struct trace_iterator *
+struct trace_iterator *
__tracing_open(struct inode *inode, struct file *file, bool snapshot)
{
struct trace_array *tr = inode->i_private;
@@ -4069,7 +3468,7 @@ int tracing_single_release_file_tr(struct inode *inode, struct file *filp)
return single_release(inode, filp);
}
-static int tracing_release(struct inode *inode, struct file *file)
+int tracing_release(struct inode *inode, struct file *file)
{
struct trace_array *tr = inode->i_private;
struct seq_file *m = file->private_data;
@@ -5220,7 +4619,7 @@ int tracer_init(struct tracer *t, struct trace_array *tr)
return t->init(tr);
}
-static void set_buffer_entries(struct array_buffer *buf, unsigned long val)
+void trace_set_buffer_entries(struct array_buffer *buf, unsigned long val)
{
int cpu;
@@ -5231,40 +4630,12 @@ static void set_buffer_entries(struct array_buffer *buf, unsigned long val)
static void update_buffer_entries(struct array_buffer *buf, int cpu)
{
if (cpu == RING_BUFFER_ALL_CPUS) {
- set_buffer_entries(buf, ring_buffer_size(buf->buffer, 0));
+ trace_set_buffer_entries(buf, ring_buffer_size(buf->buffer, 0));
} else {
per_cpu_ptr(buf->data, cpu)->entries = ring_buffer_size(buf->buffer, cpu);
}
}
-#ifdef CONFIG_TRACER_SNAPSHOT
-/* resize @tr's buffer to the size of @size_tr's entries */
-static int resize_buffer_duplicate_size(struct array_buffer *trace_buf,
- struct array_buffer *size_buf, int cpu_id)
-{
- int cpu, ret = 0;
-
- if (cpu_id == RING_BUFFER_ALL_CPUS) {
- for_each_tracing_cpu(cpu) {
- ret = ring_buffer_resize(trace_buf->buffer,
- per_cpu_ptr(size_buf->data, cpu)->entries, cpu);
- if (ret < 0)
- break;
- per_cpu_ptr(trace_buf->data, cpu)->entries =
- per_cpu_ptr(size_buf->data, cpu)->entries;
- }
- } else {
- ret = ring_buffer_resize(trace_buf->buffer,
- per_cpu_ptr(size_buf->data, cpu_id)->entries, cpu_id);
- if (ret == 0)
- per_cpu_ptr(trace_buf->data, cpu_id)->entries =
- per_cpu_ptr(size_buf->data, cpu_id)->entries;
- }
-
- return ret;
-}
-#endif /* CONFIG_TRACER_SNAPSHOT */
-
static int __tracing_resize_ring_buffer(struct trace_array *tr,
unsigned long size, int cpu)
{
@@ -5683,9 +5054,8 @@ tracing_set_trace_write(struct file *filp, const char __user *ubuf,
return ret;
}
-static ssize_t
-tracing_nsecs_read(unsigned long *ptr, char __user *ubuf,
- size_t cnt, loff_t *ppos)
+ssize_t tracing_nsecs_read(unsigned long *ptr, char __user *ubuf,
+ size_t cnt, loff_t *ppos)
{
char buf[64];
int r;
@@ -5697,9 +5067,8 @@ tracing_nsecs_read(unsigned long *ptr, char __user *ubuf,
return simple_read_from_buffer(ubuf, cnt, ppos, buf, r);
}
-static ssize_t
-tracing_nsecs_write(unsigned long *ptr, const char __user *ubuf,
- size_t cnt, loff_t *ppos)
+ssize_t tracing_nsecs_write(unsigned long *ptr, const char __user *ubuf,
+ size_t cnt, loff_t *ppos)
{
unsigned long val;
int ret;
@@ -5741,28 +5110,6 @@ tracing_thresh_write(struct file *filp, const char __user *ubuf,
return cnt;
}
-#ifdef CONFIG_TRACER_MAX_TRACE
-
-static ssize_t
-tracing_max_lat_read(struct file *filp, char __user *ubuf,
- size_t cnt, loff_t *ppos)
-{
- struct trace_array *tr = filp->private_data;
-
- return tracing_nsecs_read(&tr->max_latency, ubuf, cnt, ppos);
-}
-
-static ssize_t
-tracing_max_lat_write(struct file *filp, const char __user *ubuf,
- size_t cnt, loff_t *ppos)
-{
- struct trace_array *tr = filp->private_data;
-
- return tracing_nsecs_write(&tr->max_latency, ubuf, cnt, ppos);
-}
-
-#endif
-
static int open_pipe_on_cpu(struct trace_array *tr, int cpu)
{
if (cpu == RING_BUFFER_ALL_CPUS) {
@@ -7067,266 +6414,78 @@ static ssize_t tracing_clock_write(struct file *filp, const char __user *ubuf,
const char *clockstr;
int ret;
- if (cnt >= sizeof(buf))
- return -EINVAL;
-
- if (copy_from_user(buf, ubuf, cnt))
- return -EFAULT;
-
- buf[cnt] = 0;
-
- clockstr = strstrip(buf);
-
- ret = tracing_set_clock(tr, clockstr);
- if (ret)
- return ret;
-
- *fpos += cnt;
-
- return cnt;
-}
-
-static int tracing_clock_open(struct inode *inode, struct file *file)
-{
- struct trace_array *tr = inode->i_private;
- int ret;
-
- ret = tracing_check_open_get_tr(tr);
- if (ret)
- return ret;
-
- ret = single_open(file, tracing_clock_show, inode->i_private);
- if (ret < 0)
- trace_array_put(tr);
-
- return ret;
-}
-
-static int tracing_time_stamp_mode_show(struct seq_file *m, void *v)
-{
- struct trace_array *tr = m->private;
-
- guard(mutex)(&trace_types_lock);
-
- if (ring_buffer_time_stamp_abs(tr->array_buffer.buffer))
- seq_puts(m, "delta [absolute]\n");
- else
- seq_puts(m, "[delta] absolute\n");
-
- return 0;
-}
-
-static int tracing_time_stamp_mode_open(struct inode *inode, struct file *file)
-{
- struct trace_array *tr = inode->i_private;
- int ret;
-
- ret = tracing_check_open_get_tr(tr);
- if (ret)
- return ret;
-
- ret = single_open(file, tracing_time_stamp_mode_show, inode->i_private);
- if (ret < 0)
- trace_array_put(tr);
-
- return ret;
-}
-
-u64 tracing_event_time_stamp(struct trace_buffer *buffer, struct ring_buffer_event *rbe)
-{
- if (rbe == this_cpu_read(trace_buffered_event))
- return ring_buffer_time_stamp(buffer);
-
- return ring_buffer_event_time_stamp(buffer, rbe);
-}
-
-struct ftrace_buffer_info {
- struct trace_iterator iter;
- void *spare;
- unsigned int spare_cpu;
- unsigned int spare_size;
- unsigned int read;
-};
-
-#ifdef CONFIG_TRACER_SNAPSHOT
-static int tracing_snapshot_open(struct inode *inode, struct file *file)
-{
- struct trace_array *tr = inode->i_private;
- struct trace_iterator *iter;
- struct seq_file *m;
- int ret;
-
- ret = tracing_check_open_get_tr(tr);
- if (ret)
- return ret;
-
- if (file->f_mode & FMODE_READ) {
- iter = __tracing_open(inode, file, true);
- if (IS_ERR(iter))
- ret = PTR_ERR(iter);
- } else {
- /* Writes still need the seq_file to hold the private data */
- ret = -ENOMEM;
- m = kzalloc_obj(*m);
- if (!m)
- goto out;
- iter = kzalloc_obj(*iter);
- if (!iter) {
- kfree(m);
- goto out;
- }
- ret = 0;
-
- iter->tr = tr;
- iter->array_buffer = &tr->snapshot_buffer;
- iter->cpu_file = tracing_get_cpu(inode);
- m->private = iter;
- file->private_data = m;
- }
-out:
- if (ret < 0)
- trace_array_put(tr);
-
- return ret;
-}
-
-static void tracing_swap_cpu_buffer(void *tr)
-{
- update_max_tr_single((struct trace_array *)tr, current, smp_processor_id());
-}
-
-static ssize_t
-tracing_snapshot_write(struct file *filp, const char __user *ubuf, size_t cnt,
- loff_t *ppos)
-{
- struct seq_file *m = filp->private_data;
- struct trace_iterator *iter = m->private;
- struct trace_array *tr = iter->tr;
- unsigned long val;
- int ret;
-
- ret = tracing_update_buffers(tr);
- if (ret < 0)
- return ret;
+ if (cnt >= sizeof(buf))
+ return -EINVAL;
- ret = kstrtoul_from_user(ubuf, cnt, 10, &val);
- if (ret)
- return ret;
+ if (copy_from_user(buf, ubuf, cnt))
+ return -EFAULT;
- guard(mutex)(&trace_types_lock);
+ buf[cnt] = 0;
- if (tracer_uses_snapshot(tr->current_trace))
- return -EBUSY;
+ clockstr = strstrip(buf);
- local_irq_disable();
- arch_spin_lock(&tr->max_lock);
- if (tr->cond_snapshot)
- ret = -EBUSY;
- arch_spin_unlock(&tr->max_lock);
- local_irq_enable();
+ ret = tracing_set_clock(tr, clockstr);
if (ret)
return ret;
- switch (val) {
- case 0:
- if (iter->cpu_file != RING_BUFFER_ALL_CPUS)
- return -EINVAL;
- if (tr->allocated_snapshot)
- free_snapshot(tr);
- break;
- case 1:
-/* Only allow per-cpu swap if the ring buffer supports it */
-#ifndef CONFIG_RING_BUFFER_ALLOW_SWAP
- if (iter->cpu_file != RING_BUFFER_ALL_CPUS)
- return -EINVAL;
-#endif
- if (tr->allocated_snapshot)
- ret = resize_buffer_duplicate_size(&tr->snapshot_buffer,
- &tr->array_buffer, iter->cpu_file);
+ *fpos += cnt;
- ret = tracing_arm_snapshot_locked(tr);
- if (ret)
- return ret;
+ return cnt;
+}
- /* Now, we're going to swap */
- if (iter->cpu_file == RING_BUFFER_ALL_CPUS) {
- local_irq_disable();
- update_max_tr(tr, current, smp_processor_id(), NULL);
- local_irq_enable();
- } else {
- smp_call_function_single(iter->cpu_file, tracing_swap_cpu_buffer,
- (void *)tr, 1);
- }
- tracing_disarm_snapshot(tr);
- break;
- default:
- if (tr->allocated_snapshot) {
- if (iter->cpu_file == RING_BUFFER_ALL_CPUS)
- tracing_reset_online_cpus(&tr->snapshot_buffer);
- else
- tracing_reset_cpu(&tr->snapshot_buffer, iter->cpu_file);
- }
- break;
- }
+static int tracing_clock_open(struct inode *inode, struct file *file)
+{
+ struct trace_array *tr = inode->i_private;
+ int ret;
- if (ret >= 0) {
- *ppos += cnt;
- ret = cnt;
- }
+ ret = tracing_check_open_get_tr(tr);
+ if (ret)
+ return ret;
+
+ ret = single_open(file, tracing_clock_show, inode->i_private);
+ if (ret < 0)
+ trace_array_put(tr);
return ret;
}
-static int tracing_snapshot_release(struct inode *inode, struct file *file)
+static int tracing_time_stamp_mode_show(struct seq_file *m, void *v)
{
- struct seq_file *m = file->private_data;
- int ret;
-
- ret = tracing_release(inode, file);
+ struct trace_array *tr = m->private;
- if (file->f_mode & FMODE_READ)
- return ret;
+ guard(mutex)(&trace_types_lock);
- /* If write only, the seq_file is just a stub */
- if (m)
- kfree(m->private);
- kfree(m);
+ if (ring_buffer_time_stamp_abs(tr->array_buffer.buffer))
+ seq_puts(m, "delta [absolute]\n");
+ else
+ seq_puts(m, "[delta] absolute\n");
return 0;
}
-static int tracing_buffers_open(struct inode *inode, struct file *filp);
-static ssize_t tracing_buffers_read(struct file *filp, char __user *ubuf,
- size_t count, loff_t *ppos);
-static int tracing_buffers_release(struct inode *inode, struct file *file);
-static ssize_t tracing_buffers_splice_read(struct file *file, loff_t *ppos,
- struct pipe_inode_info *pipe, size_t len, unsigned int flags);
-
-static int snapshot_raw_open(struct inode *inode, struct file *filp)
+static int tracing_time_stamp_mode_open(struct inode *inode, struct file *file)
{
- struct ftrace_buffer_info *info;
+ struct trace_array *tr = inode->i_private;
int ret;
- /* The following checks for tracefs lockdown */
- ret = tracing_buffers_open(inode, filp);
- if (ret < 0)
+ ret = tracing_check_open_get_tr(tr);
+ if (ret)
return ret;
- info = filp->private_data;
-
- if (tracer_uses_snapshot(info->iter.trace)) {
- tracing_buffers_release(inode, filp);
- return -EBUSY;
- }
-
- info->iter.snapshot = true;
- info->iter.array_buffer = &info->iter.tr->snapshot_buffer;
+ ret = single_open(file, tracing_time_stamp_mode_show, inode->i_private);
+ if (ret < 0)
+ trace_array_put(tr);
return ret;
}
-#endif /* CONFIG_TRACER_SNAPSHOT */
+u64 tracing_event_time_stamp(struct trace_buffer *buffer, struct ring_buffer_event *rbe)
+{
+ if (rbe == this_cpu_read(trace_buffered_event))
+ return ring_buffer_time_stamp(buffer);
+ return ring_buffer_event_time_stamp(buffer, rbe);
+}
static const struct file_operations tracing_thresh_fops = {
.open = tracing_open_generic,
@@ -7335,16 +6494,6 @@ static const struct file_operations tracing_thresh_fops = {
.llseek = generic_file_llseek,
};
-#ifdef CONFIG_TRACER_MAX_TRACE
-static const struct file_operations tracing_max_lat_fops = {
- .open = tracing_open_generic_tr,
- .read = tracing_max_lat_read,
- .write = tracing_max_lat_write,
- .llseek = generic_file_llseek,
- .release = tracing_release_generic_tr,
-};
-#endif
-
static const struct file_operations set_tracer_fops = {
.open = tracing_open_generic_tr,
.read = tracing_set_trace_read,
@@ -7431,24 +6580,6 @@ static const struct file_operations last_boot_fops = {
.release = tracing_seq_release,
};
-#ifdef CONFIG_TRACER_SNAPSHOT
-static const struct file_operations snapshot_fops = {
- .open = tracing_snapshot_open,
- .read = seq_read,
- .write = tracing_snapshot_write,
- .llseek = tracing_lseek,
- .release = tracing_snapshot_release,
-};
-
-static const struct file_operations snapshot_raw_fops = {
- .open = snapshot_raw_open,
- .read = tracing_buffers_read,
- .release = tracing_buffers_release,
- .splice_read = tracing_buffers_splice_read,
-};
-
-#endif /* CONFIG_TRACER_SNAPSHOT */
-
/*
* trace_min_max_write - Write a u64 value to a trace_min_max_param struct
* @filp: The active open file structure
@@ -7808,7 +6939,7 @@ static const struct file_operations tracing_err_log_fops = {
.release = tracing_err_log_release,
};
-static int tracing_buffers_open(struct inode *inode, struct file *filp)
+int tracing_buffers_open(struct inode *inode, struct file *filp)
{
struct trace_array *tr = inode->i_private;
struct ftrace_buffer_info *info;
@@ -7856,9 +6987,8 @@ tracing_buffers_poll(struct file *filp, poll_table *poll_table)
return trace_poll(iter, filp, poll_table);
}
-static ssize_t
-tracing_buffers_read(struct file *filp, char __user *ubuf,
- size_t count, loff_t *ppos)
+ssize_t tracing_buffers_read(struct file *filp, char __user *ubuf,
+ size_t count, loff_t *ppos)
{
struct ftrace_buffer_info *info = filp->private_data;
struct trace_iterator *iter = &info->iter;
@@ -7959,7 +7089,7 @@ static int tracing_buffers_flush(struct file *file, fl_owner_t id)
return 0;
}
-static int tracing_buffers_release(struct inode *inode, struct file *file)
+int tracing_buffers_release(struct inode *inode, struct file *file)
{
struct ftrace_buffer_info *info = file->private_data;
struct trace_iterator *iter = &info->iter;
@@ -8033,10 +7163,9 @@ static void buffer_spd_release(struct splice_pipe_desc *spd, unsigned int i)
spd->partial[i].private = 0;
}
-static ssize_t
-tracing_buffers_splice_read(struct file *file, loff_t *ppos,
- struct pipe_inode_info *pipe, size_t len,
- unsigned int flags)
+ssize_t tracing_buffers_splice_read(struct file *file, loff_t *ppos,
+ struct pipe_inode_info *pipe, size_t len,
+ unsigned int flags)
{
struct ftrace_buffer_info *info = file->private_data;
struct trace_iterator *iter = &info->iter;
@@ -8190,44 +7319,6 @@ static long tracing_buffers_ioctl(struct file *file, unsigned int cmd, unsigned
return 0;
}
-#ifdef CONFIG_TRACER_SNAPSHOT
-static int get_snapshot_map(struct trace_array *tr)
-{
- int err = 0;
-
- /*
- * Called with mmap_lock held. lockdep would be unhappy if we would now
- * take trace_types_lock. Instead use the specific
- * snapshot_trigger_lock.
- */
- spin_lock(&tr->snapshot_trigger_lock);
-
- if (tr->snapshot || tr->mapped == UINT_MAX)
- err = -EBUSY;
- else
- tr->mapped++;
-
- spin_unlock(&tr->snapshot_trigger_lock);
-
- /* Wait for update_max_tr() to observe iter->tr->mapped */
- if (tr->mapped == 1)
- synchronize_rcu();
-
- return err;
-
-}
-static void put_snapshot_map(struct trace_array *tr)
-{
- spin_lock(&tr->snapshot_trigger_lock);
- if (!WARN_ON(!tr->mapped))
- tr->mapped--;
- spin_unlock(&tr->snapshot_trigger_lock);
-}
-#else
-static inline int get_snapshot_map(struct trace_array *tr) { return 0; }
-static inline void put_snapshot_map(struct trace_array *tr) { }
-#endif
-
/*
* This is called when a VMA is duplicated (e.g., on fork()) to increment
* the user_mapped counter without remapping pages.
@@ -8408,170 +7499,6 @@ static const struct file_operations tracing_dyn_info_fops = {
};
#endif /* CONFIG_DYNAMIC_FTRACE */
-#if defined(CONFIG_TRACER_SNAPSHOT) && defined(CONFIG_DYNAMIC_FTRACE)
-static void
-ftrace_snapshot(unsigned long ip, unsigned long parent_ip,
- struct trace_array *tr, struct ftrace_probe_ops *ops,
- void *data)
-{
- tracing_snapshot_instance(tr);
-}
-
-static void
-ftrace_count_snapshot(unsigned long ip, unsigned long parent_ip,
- struct trace_array *tr, struct ftrace_probe_ops *ops,
- void *data)
-{
- struct ftrace_func_mapper *mapper = data;
- long *count = NULL;
-
- if (mapper)
- count = (long *)ftrace_func_mapper_find_ip(mapper, ip);
-
- if (count) {
-
- if (*count <= 0)
- return;
-
- (*count)--;
- }
-
- tracing_snapshot_instance(tr);
-}
-
-static int
-ftrace_snapshot_print(struct seq_file *m, unsigned long ip,
- struct ftrace_probe_ops *ops, void *data)
-{
- struct ftrace_func_mapper *mapper = data;
- long *count = NULL;
-
- seq_printf(m, "%ps:", (void *)ip);
-
- seq_puts(m, "snapshot");
-
- if (mapper)
- count = (long *)ftrace_func_mapper_find_ip(mapper, ip);
-
- if (count)
- seq_printf(m, ":count=%ld\n", *count);
- else
- seq_puts(m, ":unlimited\n");
-
- return 0;
-}
-
-static int
-ftrace_snapshot_init(struct ftrace_probe_ops *ops, struct trace_array *tr,
- unsigned long ip, void *init_data, void **data)
-{
- struct ftrace_func_mapper *mapper = *data;
-
- if (!mapper) {
- mapper = allocate_ftrace_func_mapper();
- if (!mapper)
- return -ENOMEM;
- *data = mapper;
- }
-
- return ftrace_func_mapper_add_ip(mapper, ip, init_data);
-}
-
-static void
-ftrace_snapshot_free(struct ftrace_probe_ops *ops, struct trace_array *tr,
- unsigned long ip, void *data)
-{
- struct ftrace_func_mapper *mapper = data;
-
- if (!ip) {
- if (!mapper)
- return;
- free_ftrace_func_mapper(mapper, NULL);
- return;
- }
-
- ftrace_func_mapper_remove_ip(mapper, ip);
-}
-
-static struct ftrace_probe_ops snapshot_probe_ops = {
- .func = ftrace_snapshot,
- .print = ftrace_snapshot_print,
-};
-
-static struct ftrace_probe_ops snapshot_count_probe_ops = {
- .func = ftrace_count_snapshot,
- .print = ftrace_snapshot_print,
- .init = ftrace_snapshot_init,
- .free = ftrace_snapshot_free,
-};
-
-static int
-ftrace_trace_snapshot_callback(struct trace_array *tr, struct ftrace_hash *hash,
- char *glob, char *cmd, char *param, int enable)
-{
- struct ftrace_probe_ops *ops;
- void *count = (void *)-1;
- char *number;
- int ret;
-
- if (!tr)
- return -ENODEV;
-
- /* hash funcs only work with set_ftrace_filter */
- if (!enable)
- return -EINVAL;
-
- ops = param ? &snapshot_count_probe_ops : &snapshot_probe_ops;
-
- if (glob[0] == '!') {
- ret = unregister_ftrace_function_probe_func(glob+1, tr, ops);
- if (!ret)
- tracing_disarm_snapshot(tr);
-
- return ret;
- }
-
- if (!param)
- goto out_reg;
-
- number = strsep(¶m, ":");
-
- if (!strlen(number))
- goto out_reg;
-
- /*
- * We use the callback data field (which is a pointer)
- * as our counter.
- */
- ret = kstrtoul(number, 0, (unsigned long *)&count);
- if (ret)
- return ret;
-
- out_reg:
- ret = tracing_arm_snapshot(tr);
- if (ret < 0)
- return ret;
-
- ret = register_ftrace_function_probe(glob, tr, ops, count);
- if (ret < 0)
- tracing_disarm_snapshot(tr);
-
- return ret < 0 ? ret : 0;
-}
-
-static struct ftrace_func_command ftrace_snapshot_cmd = {
- .name = "snapshot",
- .func = ftrace_trace_snapshot_callback,
-};
-
-static __init int register_snapshot_cmd(void)
-{
- return register_ftrace_command(&ftrace_snapshot_cmd);
-}
-#else
-static inline __init int register_snapshot_cmd(void) { return 0; }
-#endif /* defined(CONFIG_TRACER_SNAPSHOT) && defined(CONFIG_DYNAMIC_FTRACE) */
-
static struct dentry *tracing_get_dentry(struct trace_array *tr)
{
/* Top directory uses NULL as the parent */
@@ -9364,8 +8291,7 @@ static void setup_trace_scratch(struct trace_array *tr,
memset(tscratch, 0, size);
}
-static int
-allocate_trace_buffer(struct trace_array *tr, struct array_buffer *buf, unsigned long size)
+int allocate_trace_buffer(struct trace_array *tr, struct array_buffer *buf, int size)
{
enum ring_buffer_flags rb_flags;
struct trace_scratch *tscratch;
@@ -9404,8 +8330,8 @@ allocate_trace_buffer(struct trace_array *tr, struct array_buffer *buf, unsigned
}
/* Allocate the first page for all buffers */
- set_buffer_entries(&tr->array_buffer,
- ring_buffer_size(tr->array_buffer.buffer, 0));
+ trace_set_buffer_entries(&tr->array_buffer,
+ ring_buffer_size(tr->array_buffer.buffer, 0));
return 0;
}
@@ -9428,23 +8354,11 @@ static int allocate_trace_buffers(struct trace_array *tr, unsigned long size)
if (ret)
return ret;
-#ifdef CONFIG_TRACER_SNAPSHOT
- /* Fix mapped buffer trace arrays do not have snapshot buffers */
- if (tr->range_addr_start)
- return 0;
-
- ret = allocate_trace_buffer(tr, &tr->snapshot_buffer,
- allocate_snapshot ? size : 1);
- if (MEM_FAIL(ret, "Failed to allocate trace buffer\n")) {
+ ret = trace_allocate_snapshot(tr, size);
+ if (MEM_FAIL(ret, "Failed to allocate trace buffer\n"))
free_trace_buffer(&tr->array_buffer);
- return -ENOMEM;
- }
- tr->allocated_snapshot = allocate_snapshot;
-
- allocate_snapshot = false;
-#endif
- return 0;
+ return ret;
}
static void free_trace_buffers(struct trace_array *tr)
@@ -10552,47 +9466,6 @@ ssize_t trace_parse_run_command(struct file *file, const char __user *buffer,
return done;
}
-#ifdef CONFIG_TRACER_SNAPSHOT
-__init static bool tr_needs_alloc_snapshot(const char *name)
-{
- char *test;
- int len = strlen(name);
- bool ret;
-
- if (!boot_snapshot_index)
- return false;
-
- if (strncmp(name, boot_snapshot_info, len) == 0 &&
- boot_snapshot_info[len] == '\t')
- return true;
-
- test = kmalloc(strlen(name) + 3, GFP_KERNEL);
- if (!test)
- return false;
-
- sprintf(test, "\t%s\t", name);
- ret = strstr(boot_snapshot_info, test) == NULL;
- kfree(test);
- return ret;
-}
-
-__init static void do_allocate_snapshot(const char *name)
-{
- if (!tr_needs_alloc_snapshot(name))
- return;
-
- /*
- * When allocate_snapshot is set, the next call to
- * allocate_trace_buffers() (called by trace_array_get_by_name())
- * will allocate the snapshot buffer. That will also clear
- * this flag.
- */
- allocate_snapshot = true;
-}
-#else
-static inline void do_allocate_snapshot(const char *name) { }
-#endif
-
__init static int backup_instance_area(const char *backup,
unsigned long *addr, phys_addr_t *size)
{
@@ -10742,8 +9615,7 @@ __init static void enable_instances(void)
}
} else {
/* Only non mapped buffers have snapshot buffers */
- if (IS_ENABLED(CONFIG_TRACER_SNAPSHOT))
- do_allocate_snapshot(name);
+ do_allocate_snapshot(name);
}
tr = trace_array_create_systems(name, NULL, addr, size);
@@ -10935,24 +9807,6 @@ struct trace_array *trace_get_global_array(void)
}
#endif
-void __init ftrace_boot_snapshot(void)
-{
-#ifdef CONFIG_TRACER_SNAPSHOT
- struct trace_array *tr;
-
- if (!snapshot_at_boot)
- return;
-
- list_for_each_entry(tr, &ftrace_trace_arrays, list) {
- if (!tr->allocated_snapshot)
- continue;
-
- tracing_snapshot_instance(tr);
- trace_array_puts(tr, "** Boot snapshot taken **\n");
- }
-#endif
-}
-
void __init early_trace_init(void)
{
if (tracepoint_printk) {
diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h
index 7db78a62f786..a3ea735a9ef6 100644
--- a/kernel/trace/trace.h
+++ b/kernel/trace/trace.h
@@ -264,6 +264,7 @@ static inline bool still_need_pid_events(int type, struct trace_pid_list *pid_li
typedef bool (*cond_update_fn_t)(struct trace_array *tr, void *cond_data);
+#ifdef CONFIG_TRACER_SNAPSHOT
/**
* struct cond_snapshot - conditional snapshot data and callback
*
@@ -306,6 +307,7 @@ struct cond_snapshot {
void *cond_data;
cond_update_fn_t update;
};
+#endif /* CONFIG_TRACER_SNAPSHOT */
/*
* struct trace_func_repeats - used to keep track of the consecutive
@@ -675,6 +677,7 @@ void tracing_reset_all_online_cpus(void);
void tracing_reset_all_online_cpus_unlocked(void);
int tracing_open_generic(struct inode *inode, struct file *filp);
int tracing_open_generic_tr(struct inode *inode, struct file *filp);
+int tracing_release(struct inode *inode, struct file *file);
int tracing_release_generic_tr(struct inode *inode, struct file *file);
int tracing_open_file_tr(struct inode *inode, struct file *filp);
int tracing_release_file_tr(struct inode *inode, struct file *filp);
@@ -684,12 +687,48 @@ void tracer_tracing_on(struct trace_array *tr);
void tracer_tracing_off(struct trace_array *tr);
void tracer_tracing_disable(struct trace_array *tr);
void tracer_tracing_enable(struct trace_array *tr);
+int allocate_trace_buffer(struct trace_array *tr, struct array_buffer *buf, int size);
struct dentry *trace_create_file(const char *name,
umode_t mode,
struct dentry *parent,
void *data,
const struct file_operations *fops);
+struct trace_iterator *__tracing_open(struct inode *inode, struct file *file,
+ bool snapshot);
+int tracing_buffers_open(struct inode *inode, struct file *filp);
+ssize_t tracing_buffers_read(struct file *filp, char __user *ubuf,
+ size_t count, loff_t *ppos);
+int tracing_buffers_release(struct inode *inode, struct file *file);
+ssize_t tracing_buffers_splice_read(struct file *file, loff_t *ppos,
+ struct pipe_inode_info *pipe, size_t len, unsigned int flags);
+
+ssize_t tracing_nsecs_read(unsigned long *ptr, char __user *ubuf,
+ size_t cnt, loff_t *ppos);
+ssize_t tracing_nsecs_write(unsigned long *ptr, const char __user *ubuf,
+ size_t cnt, loff_t *ppos);
+
+void trace_set_buffer_entries(struct array_buffer *buf, unsigned long val);
+
+/*
+ * Should be used after trace_array_get(), trace_types_lock
+ * ensures that i_cdev was already initialized.
+ */
+static inline int tracing_get_cpu(struct inode *inode)
+{
+ if (inode->i_cdev) /* See trace_create_cpu_file() */
+ return (long)inode->i_cdev - 1;
+ return RING_BUFFER_ALL_CPUS;
+}
+void tracing_reset_cpu(struct array_buffer *buf, int cpu);
+
+struct ftrace_buffer_info {
+ struct trace_iterator iter;
+ void *spare;
+ unsigned int spare_cpu;
+ unsigned int spare_size;
+ unsigned int read;
+};
/**
* tracer_tracing_is_on_cpu - show real state of ring buffer enabled on for a cpu
@@ -828,11 +867,15 @@ static inline bool tracer_uses_snapshot(struct tracer *tracer)
{
return tracer->use_max_tr;
}
+void trace_create_maxlat_file(struct trace_array *tr,
+ struct dentry *d_tracer);
#else
static inline bool tracer_uses_snapshot(struct tracer *tracer)
{
return false;
}
+static inline void trace_create_maxlat_file(struct trace_array *tr,
+ struct dentry *d_tracer) { }
#endif
void trace_last_func_repeats(struct trace_array *tr,
@@ -2140,12 +2183,6 @@ static inline bool event_command_needs_rec(struct event_command *cmd_ops)
extern int trace_event_enable_disable(struct trace_event_file *file,
int enable, int soft_disable);
-extern int tracing_alloc_snapshot(void);
-extern void tracing_snapshot_cond(struct trace_array *tr, void *cond_data);
-extern int tracing_snapshot_cond_enable(struct trace_array *tr, void *cond_data, cond_update_fn_t update);
-
-extern int tracing_snapshot_cond_disable(struct trace_array *tr);
-extern void *tracing_cond_snapshot_data(struct trace_array *tr);
extern const char *__start___trace_bprintk_fmt[];
extern const char *__stop___trace_bprintk_fmt[];
@@ -2233,19 +2270,71 @@ static inline void trace_event_update_all(struct trace_eval_map **map, int len)
#endif
#ifdef CONFIG_TRACER_SNAPSHOT
+extern const struct file_operations snapshot_fops;
+extern const struct file_operations snapshot_raw_fops;
+
+/* Used when creating instances */
+int trace_allocate_snapshot(struct trace_array *tr, int size);
+
+int tracing_alloc_snapshot(void);
+void tracing_snapshot_cond(struct trace_array *tr, void *cond_data);
+int tracing_snapshot_cond_enable(struct trace_array *tr, void *cond_data, cond_update_fn_t update);
+int tracing_snapshot_cond_disable(struct trace_array *tr);
+void *tracing_cond_snapshot_data(struct trace_array *tr);
void tracing_snapshot_instance(struct trace_array *tr);
int tracing_alloc_snapshot_instance(struct trace_array *tr);
+int tracing_arm_snapshot_locked(struct trace_array *tr);
int tracing_arm_snapshot(struct trace_array *tr);
void tracing_disarm_snapshot(struct trace_array *tr);
-#else
+void free_snapshot(struct trace_array *tr);
+void print_snapshot_help(struct seq_file *m, struct trace_iterator *iter);
+int get_snapshot_map(struct trace_array *tr);
+void put_snapshot_map(struct trace_array *tr);
+int resize_buffer_duplicate_size(struct array_buffer *trace_buf,
+ struct array_buffer *size_buf, int cpu_id);
+__init void do_allocate_snapshot(const char *name);
+# ifdef CONFIG_DYNAMIC_FTRACE
+__init int register_snapshot_cmd(void);
+# else
+static inline int register_snapshot_cmd(void) { return 0; }
+# endif
+#else /* !CONFIG_TRACER_SNAPSHOT */
+static inline int trace_allocate_snapshot(struct trace_array *tr, int size) { return 0; }
static inline void tracing_snapshot_instance(struct trace_array *tr) { }
static inline int tracing_alloc_snapshot_instance(struct trace_array *tr)
{
return 0;
}
+static inline int tracing_arm_snapshot_locked(struct trace_array *tr) { return -EBUSY; }
static inline int tracing_arm_snapshot(struct trace_array *tr) { return 0; }
static inline void tracing_disarm_snapshot(struct trace_array *tr) { }
-#endif
+static inline void free_snapshot(struct trace_array *tr) {}
+static inline void tracing_snapshot_cond(struct trace_array *tr, void *cond_data)
+{
+ WARN_ONCE(1, "Snapshot feature not enabled, but internal conditional snapshot used");
+}
+static inline void *tracing_cond_snapshot_data(struct trace_array *tr)
+{
+ return NULL;
+}
+static inline int tracing_snapshot_cond_enable(struct trace_array *tr, void *cond_data, cond_update_fn_t update)
+{
+ return -ENODEV;
+}
+static inline int tracing_snapshot_cond_disable(struct trace_array *tr)
+{
+ return false;
+}
+static inline void print_snapshot_help(struct seq_file *m, struct trace_iterator *iter)
+{
+ /* Should never be called */
+ WARN_ONCE(1, "Snapshot print function called without snapshot configured");
+}
+static inline int get_snapshot_map(struct trace_array *tr) { return 0; }
+static inline void put_snapshot_map(struct trace_array *tr) { }
+static inline void do_allocate_snapshot(const char *name) { }
+static inline int register_snapshot_cmd(void) { return 0; }
+#endif /* CONFIG_TRACER_SNAPSHOT */
#ifdef CONFIG_PREEMPT_TRACER
void tracer_preempt_on(unsigned long a0, unsigned long a1);
diff --git a/kernel/trace/trace_snapshot.c b/kernel/trace/trace_snapshot.c
new file mode 100644
index 000000000000..8865b2ef2264
--- /dev/null
+++ b/kernel/trace/trace_snapshot.c
@@ -0,0 +1,1067 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/fsnotify.h>
+
+#include <asm/setup.h> /* COMMAND_LINE_SIZE */
+
+#include "trace.h"
+
+/* Used if snapshot allocated at boot */
+static bool allocate_snapshot;
+static bool snapshot_at_boot;
+
+static char boot_snapshot_info[COMMAND_LINE_SIZE] __initdata;
+static int boot_snapshot_index;
+
+static int __init boot_alloc_snapshot(char *str)
+{
+ char *slot = boot_snapshot_info + boot_snapshot_index;
+ int left = sizeof(boot_snapshot_info) - boot_snapshot_index;
+ int ret;
+
+ if (str[0] == '=') {
+ str++;
+ if (strlen(str) >= left)
+ return -1;
+
+ ret = snprintf(slot, left, "%s\t", str);
+ boot_snapshot_index += ret;
+ } else {
+ allocate_snapshot = true;
+ /* We also need the main ring buffer expanded */
+ trace_set_ring_buffer_expanded(NULL);
+ }
+ return 1;
+}
+__setup("alloc_snapshot", boot_alloc_snapshot);
+
+
+static int __init boot_snapshot(char *str)
+{
+ snapshot_at_boot = true;
+ boot_alloc_snapshot(str);
+ return 1;
+}
+__setup("ftrace_boot_snapshot", boot_snapshot);
+static void tracing_snapshot_instance_cond(struct trace_array *tr,
+ void *cond_data)
+{
+ unsigned long flags;
+
+ if (in_nmi()) {
+ trace_array_puts(tr, "*** SNAPSHOT CALLED FROM NMI CONTEXT ***\n");
+ trace_array_puts(tr, "*** snapshot is being ignored ***\n");
+ return;
+ }
+
+ if (!tr->allocated_snapshot) {
+ trace_array_puts(tr, "*** SNAPSHOT NOT ALLOCATED ***\n");
+ trace_array_puts(tr, "*** stopping trace here! ***\n");
+ tracer_tracing_off(tr);
+ return;
+ }
+
+ if (tr->mapped) {
+ trace_array_puts(tr, "*** BUFFER MEMORY MAPPED ***\n");
+ trace_array_puts(tr, "*** Can not use snapshot (sorry) ***\n");
+ return;
+ }
+
+ /* Note, snapshot can not be used when the tracer uses it */
+ if (tracer_uses_snapshot(tr->current_trace)) {
+ trace_array_puts(tr, "*** LATENCY TRACER ACTIVE ***\n");
+ trace_array_puts(tr, "*** Can not use snapshot (sorry) ***\n");
+ return;
+ }
+
+ local_irq_save(flags);
+ update_max_tr(tr, current, smp_processor_id(), cond_data);
+ local_irq_restore(flags);
+}
+
+void tracing_snapshot_instance(struct trace_array *tr)
+{
+ tracing_snapshot_instance_cond(tr, NULL);
+}
+
+/**
+ * tracing_snapshot_cond - conditionally take a snapshot of the current buffer.
+ * @tr: The tracing instance to snapshot
+ * @cond_data: The data to be tested conditionally, and possibly saved
+ *
+ * This is the same as tracing_snapshot() except that the snapshot is
+ * conditional - the snapshot will only happen if the
+ * cond_snapshot.update() implementation receiving the cond_data
+ * returns true, which means that the trace array's cond_snapshot
+ * update() operation used the cond_data to determine whether the
+ * snapshot should be taken, and if it was, presumably saved it along
+ * with the snapshot.
+ */
+void tracing_snapshot_cond(struct trace_array *tr, void *cond_data)
+{
+ tracing_snapshot_instance_cond(tr, cond_data);
+}
+EXPORT_SYMBOL_GPL(tracing_snapshot_cond);
+
+/**
+ * tracing_cond_snapshot_data - get the user data associated with a snapshot
+ * @tr: The tracing instance
+ *
+ * When the user enables a conditional snapshot using
+ * tracing_snapshot_cond_enable(), the user-defined cond_data is saved
+ * with the snapshot. This accessor is used to retrieve it.
+ *
+ * Should not be called from cond_snapshot.update(), since it takes
+ * the tr->max_lock lock, which the code calling
+ * cond_snapshot.update() has already done.
+ *
+ * Returns the cond_data associated with the trace array's snapshot.
+ */
+void *tracing_cond_snapshot_data(struct trace_array *tr)
+{
+ void *cond_data = NULL;
+
+ local_irq_disable();
+ arch_spin_lock(&tr->max_lock);
+
+ if (tr->cond_snapshot)
+ cond_data = tr->cond_snapshot->cond_data;
+
+ arch_spin_unlock(&tr->max_lock);
+ local_irq_enable();
+
+ return cond_data;
+}
+EXPORT_SYMBOL_GPL(tracing_cond_snapshot_data);
+
+/* resize @tr's buffer to the size of @size_tr's entries */
+int resize_buffer_duplicate_size(struct array_buffer *trace_buf,
+ struct array_buffer *size_buf, int cpu_id)
+{
+ int cpu, ret = 0;
+
+ if (cpu_id == RING_BUFFER_ALL_CPUS) {
+ for_each_tracing_cpu(cpu) {
+ ret = ring_buffer_resize(trace_buf->buffer,
+ per_cpu_ptr(size_buf->data, cpu)->entries, cpu);
+ if (ret < 0)
+ break;
+ per_cpu_ptr(trace_buf->data, cpu)->entries =
+ per_cpu_ptr(size_buf->data, cpu)->entries;
+ }
+ } else {
+ ret = ring_buffer_resize(trace_buf->buffer,
+ per_cpu_ptr(size_buf->data, cpu_id)->entries, cpu_id);
+ if (ret == 0)
+ per_cpu_ptr(trace_buf->data, cpu_id)->entries =
+ per_cpu_ptr(size_buf->data, cpu_id)->entries;
+ }
+
+ return ret;
+}
+
+int tracing_alloc_snapshot_instance(struct trace_array *tr)
+{
+ int order;
+ int ret;
+
+ if (!tr->allocated_snapshot) {
+
+ /* Make the snapshot buffer have the same order as main buffer */
+ order = ring_buffer_subbuf_order_get(tr->array_buffer.buffer);
+ ret = ring_buffer_subbuf_order_set(tr->snapshot_buffer.buffer, order);
+ if (ret < 0)
+ return ret;
+
+ /* allocate spare buffer */
+ ret = resize_buffer_duplicate_size(&tr->snapshot_buffer,
+ &tr->array_buffer, RING_BUFFER_ALL_CPUS);
+ if (ret < 0)
+ return ret;
+
+ tr->allocated_snapshot = true;
+ }
+
+ return 0;
+}
+
+void free_snapshot(struct trace_array *tr)
+{
+ /*
+ * We don't free the ring buffer. instead, resize it because
+ * The max_tr ring buffer has some state (e.g. ring->clock) and
+ * we want preserve it.
+ */
+ ring_buffer_subbuf_order_set(tr->snapshot_buffer.buffer, 0);
+ ring_buffer_resize(tr->snapshot_buffer.buffer, 1, RING_BUFFER_ALL_CPUS);
+ trace_set_buffer_entries(&tr->snapshot_buffer, 1);
+ tracing_reset_online_cpus(&tr->snapshot_buffer);
+ tr->allocated_snapshot = false;
+}
+
+int tracing_arm_snapshot_locked(struct trace_array *tr)
+{
+ int ret;
+
+ lockdep_assert_held(&trace_types_lock);
+
+ spin_lock(&tr->snapshot_trigger_lock);
+ if (tr->snapshot == UINT_MAX || tr->mapped) {
+ spin_unlock(&tr->snapshot_trigger_lock);
+ return -EBUSY;
+ }
+
+ tr->snapshot++;
+ spin_unlock(&tr->snapshot_trigger_lock);
+
+ ret = tracing_alloc_snapshot_instance(tr);
+ if (ret) {
+ spin_lock(&tr->snapshot_trigger_lock);
+ tr->snapshot--;
+ spin_unlock(&tr->snapshot_trigger_lock);
+ }
+
+ return ret;
+}
+
+int tracing_arm_snapshot(struct trace_array *tr)
+{
+ guard(mutex)(&trace_types_lock);
+ return tracing_arm_snapshot_locked(tr);
+}
+
+void tracing_disarm_snapshot(struct trace_array *tr)
+{
+ spin_lock(&tr->snapshot_trigger_lock);
+ if (!WARN_ON(!tr->snapshot))
+ tr->snapshot--;
+ spin_unlock(&tr->snapshot_trigger_lock);
+}
+
+/**
+ * tracing_snapshot_alloc - allocate and take a snapshot of the current buffer.
+ *
+ * This is similar to tracing_snapshot(), but it will allocate the
+ * snapshot buffer if it isn't already allocated. Use this only
+ * where it is safe to sleep, as the allocation may sleep.
+ *
+ * This causes a swap between the snapshot buffer and the current live
+ * tracing buffer. You can use this to take snapshots of the live
+ * trace when some condition is triggered, but continue to trace.
+ */
+void tracing_snapshot_alloc(void)
+{
+ int ret;
+
+ ret = tracing_alloc_snapshot();
+ if (ret < 0)
+ return;
+
+ tracing_snapshot();
+}
+EXPORT_SYMBOL_GPL(tracing_snapshot_alloc);
+
+/**
+ * tracing_snapshot_cond_enable - enable conditional snapshot for an instance
+ * @tr: The tracing instance
+ * @cond_data: User data to associate with the snapshot
+ * @update: Implementation of the cond_snapshot update function
+ *
+ * Check whether the conditional snapshot for the given instance has
+ * already been enabled, or if the current tracer is already using a
+ * snapshot; if so, return -EBUSY, else create a cond_snapshot and
+ * save the cond_data and update function inside.
+ *
+ * Returns 0 if successful, error otherwise.
+ */
+int tracing_snapshot_cond_enable(struct trace_array *tr, void *cond_data,
+ cond_update_fn_t update)
+{
+ struct cond_snapshot *cond_snapshot __free(kfree) =
+ kzalloc_obj(*cond_snapshot);
+ int ret;
+
+ if (!cond_snapshot)
+ return -ENOMEM;
+
+ cond_snapshot->cond_data = cond_data;
+ cond_snapshot->update = update;
+
+ guard(mutex)(&trace_types_lock);
+
+ if (tracer_uses_snapshot(tr->current_trace))
+ return -EBUSY;
+
+ /*
+ * The cond_snapshot can only change to NULL without the
+ * trace_types_lock. We don't care if we race with it going
+ * to NULL, but we want to make sure that it's not set to
+ * something other than NULL when we get here, which we can
+ * do safely with only holding the trace_types_lock and not
+ * having to take the max_lock.
+ */
+ if (tr->cond_snapshot)
+ return -EBUSY;
+
+ ret = tracing_arm_snapshot_locked(tr);
+ if (ret)
+ return ret;
+
+ local_irq_disable();
+ arch_spin_lock(&tr->max_lock);
+ tr->cond_snapshot = no_free_ptr(cond_snapshot);
+ arch_spin_unlock(&tr->max_lock);
+ local_irq_enable();
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(tracing_snapshot_cond_enable);
+
+/**
+ * tracing_snapshot_cond_disable - disable conditional snapshot for an instance
+ * @tr: The tracing instance
+ *
+ * Check whether the conditional snapshot for the given instance is
+ * enabled; if so, free the cond_snapshot associated with it,
+ * otherwise return -EINVAL.
+ *
+ * Returns 0 if successful, error otherwise.
+ */
+int tracing_snapshot_cond_disable(struct trace_array *tr)
+{
+ int ret = 0;
+
+ local_irq_disable();
+ arch_spin_lock(&tr->max_lock);
+
+ if (!tr->cond_snapshot)
+ ret = -EINVAL;
+ else {
+ kfree(tr->cond_snapshot);
+ tr->cond_snapshot = NULL;
+ }
+
+ arch_spin_unlock(&tr->max_lock);
+ local_irq_enable();
+
+ tracing_disarm_snapshot(tr);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(tracing_snapshot_cond_disable);
+
+#ifdef CONFIG_TRACER_MAX_TRACE
+#ifdef LATENCY_FS_NOTIFY
+static struct workqueue_struct *fsnotify_wq;
+
+static void latency_fsnotify_workfn(struct work_struct *work)
+{
+ struct trace_array *tr = container_of(work, struct trace_array,
+ fsnotify_work);
+ fsnotify_inode(tr->d_max_latency->d_inode, FS_MODIFY);
+}
+
+static void latency_fsnotify_workfn_irq(struct irq_work *iwork)
+{
+ struct trace_array *tr = container_of(iwork, struct trace_array,
+ fsnotify_irqwork);
+ queue_work(fsnotify_wq, &tr->fsnotify_work);
+}
+
+__init static int latency_fsnotify_init(void)
+{
+ fsnotify_wq = alloc_workqueue("tr_max_lat_wq",
+ WQ_UNBOUND | WQ_HIGHPRI, 0);
+ if (!fsnotify_wq) {
+ pr_err("Unable to allocate tr_max_lat_wq\n");
+ return -ENOMEM;
+ }
+ return 0;
+}
+
+late_initcall_sync(latency_fsnotify_init);
+
+void latency_fsnotify(struct trace_array *tr)
+{
+ if (!fsnotify_wq)
+ return;
+ /*
+ * We cannot call queue_work(&tr->fsnotify_work) from here because it's
+ * possible that we are called from __schedule() or do_idle(), which
+ * could cause a deadlock.
+ */
+ irq_work_queue(&tr->fsnotify_irqwork);
+}
+#else
+static inline void latency_fsnotify(struct trace_array *tr) { }
+#endif /* LATENCY_FS_NOTIFY */
+static const struct file_operations tracing_max_lat_fops;
+
+void trace_create_maxlat_file(struct trace_array *tr,
+ struct dentry *d_tracer)
+{
+#ifdef LATENCY_FS_NOTIFY
+ INIT_WORK(&tr->fsnotify_work, latency_fsnotify_workfn);
+ init_irq_work(&tr->fsnotify_irqwork, latency_fsnotify_workfn_irq);
+#endif
+ tr->d_max_latency = trace_create_file("tracing_max_latency",
+ TRACE_MODE_WRITE,
+ d_tracer, tr,
+ &tracing_max_lat_fops);
+}
+
+/*
+ * Copy the new maximum trace into the separate maximum-trace
+ * structure. (this way the maximum trace is permanently saved,
+ * for later retrieval via /sys/kernel/tracing/tracing_max_latency)
+ */
+static void
+__update_max_tr(struct trace_array *tr, struct task_struct *tsk, int cpu)
+{
+ struct array_buffer *trace_buf = &tr->array_buffer;
+ struct trace_array_cpu *data = per_cpu_ptr(trace_buf->data, cpu);
+ struct array_buffer *max_buf = &tr->snapshot_buffer;
+ struct trace_array_cpu *max_data = per_cpu_ptr(max_buf->data, cpu);
+
+ max_buf->cpu = cpu;
+ max_buf->time_start = data->preempt_timestamp;
+
+ max_data->saved_latency = tr->max_latency;
+ max_data->critical_start = data->critical_start;
+ max_data->critical_end = data->critical_end;
+
+ strscpy(max_data->comm, tsk->comm);
+ max_data->pid = tsk->pid;
+ /*
+ * If tsk == current, then use current_uid(), as that does not use
+ * RCU. The irq tracer can be called out of RCU scope.
+ */
+ if (tsk == current)
+ max_data->uid = current_uid();
+ else
+ max_data->uid = task_uid(tsk);
+
+ max_data->nice = tsk->static_prio - 20 - MAX_RT_PRIO;
+ max_data->policy = tsk->policy;
+ max_data->rt_priority = tsk->rt_priority;
+
+ /* record this tasks comm */
+ tracing_record_cmdline(tsk);
+ latency_fsnotify(tr);
+}
+#else
+static inline void __update_max_tr(struct trace_array *tr,
+ struct task_struct *tsk, int cpu) { }
+#endif /* CONFIG_TRACER_MAX_TRACE */
+
+/**
+ * update_max_tr - snapshot all trace buffers from global_trace to max_tr
+ * @tr: tracer
+ * @tsk: the task with the latency
+ * @cpu: The cpu that initiated the trace.
+ * @cond_data: User data associated with a conditional snapshot
+ *
+ * Flip the buffers between the @tr and the max_tr and record information
+ * about which task was the cause of this latency.
+ */
+void
+update_max_tr(struct trace_array *tr, struct task_struct *tsk, int cpu,
+ void *cond_data)
+{
+ if (tr->stop_count)
+ return;
+
+ WARN_ON_ONCE(!irqs_disabled());
+
+ if (!tr->allocated_snapshot) {
+ /* Only the nop tracer should hit this when disabling */
+ WARN_ON_ONCE(tr->current_trace != &nop_trace);
+ return;
+ }
+
+ arch_spin_lock(&tr->max_lock);
+
+ /* Inherit the recordable setting from array_buffer */
+ if (ring_buffer_record_is_set_on(tr->array_buffer.buffer))
+ ring_buffer_record_on(tr->snapshot_buffer.buffer);
+ else
+ ring_buffer_record_off(tr->snapshot_buffer.buffer);
+
+ if (tr->cond_snapshot && !tr->cond_snapshot->update(tr, cond_data)) {
+ arch_spin_unlock(&tr->max_lock);
+ return;
+ }
+
+ swap(tr->array_buffer.buffer, tr->snapshot_buffer.buffer);
+
+ __update_max_tr(tr, tsk, cpu);
+
+ arch_spin_unlock(&tr->max_lock);
+
+ /* Any waiters on the old snapshot buffer need to wake up */
+ ring_buffer_wake_waiters(tr->array_buffer.buffer, RING_BUFFER_ALL_CPUS);
+}
+
+/**
+ * update_max_tr_single - only copy one trace over, and reset the rest
+ * @tr: tracer
+ * @tsk: task with the latency
+ * @cpu: the cpu of the buffer to copy.
+ *
+ * Flip the trace of a single CPU buffer between the @tr and the max_tr.
+ */
+void
+update_max_tr_single(struct trace_array *tr, struct task_struct *tsk, int cpu)
+{
+ int ret;
+
+ if (tr->stop_count)
+ return;
+
+ WARN_ON_ONCE(!irqs_disabled());
+ if (!tr->allocated_snapshot) {
+ /* Only the nop tracer should hit this when disabling */
+ WARN_ON_ONCE(tr->current_trace != &nop_trace);
+ return;
+ }
+
+ arch_spin_lock(&tr->max_lock);
+
+ ret = ring_buffer_swap_cpu(tr->snapshot_buffer.buffer, tr->array_buffer.buffer, cpu);
+
+ if (ret == -EBUSY) {
+ /*
+ * We failed to swap the buffer due to a commit taking
+ * place on this CPU. We fail to record, but we reset
+ * the max trace buffer (no one writes directly to it)
+ * and flag that it failed.
+ * Another reason is resize is in progress.
+ */
+ trace_array_printk_buf(tr->snapshot_buffer.buffer, _THIS_IP_,
+ "Failed to swap buffers due to commit or resize in progress\n");
+ }
+
+ WARN_ON_ONCE(ret && ret != -EAGAIN && ret != -EBUSY);
+
+ __update_max_tr(tr, tsk, cpu);
+ arch_spin_unlock(&tr->max_lock);
+}
+
+static void show_snapshot_main_help(struct seq_file *m)
+{
+ seq_puts(m, "# echo 0 > snapshot : Clears and frees snapshot buffer\n"
+ "# echo 1 > snapshot : Allocates snapshot buffer, if not already allocated.\n"
+ "# Takes a snapshot of the main buffer.\n"
+ "# echo 2 > snapshot : Clears snapshot buffer (but does not allocate or free)\n"
+ "# (Doesn't have to be '2' works with any number that\n"
+ "# is not a '0' or '1')\n");
+}
+
+static void show_snapshot_percpu_help(struct seq_file *m)
+{
+ seq_puts(m, "# echo 0 > snapshot : Invalid for per_cpu snapshot file.\n");
+#ifdef CONFIG_RING_BUFFER_ALLOW_SWAP
+ seq_puts(m, "# echo 1 > snapshot : Allocates snapshot buffer, if not already allocated.\n"
+ "# Takes a snapshot of the main buffer for this cpu.\n");
+#else
+ seq_puts(m, "# echo 1 > snapshot : Not supported with this kernel.\n"
+ "# Must use main snapshot file to allocate.\n");
+#endif
+ seq_puts(m, "# echo 2 > snapshot : Clears this cpu's snapshot buffer (but does not allocate)\n"
+ "# (Doesn't have to be '2' works with any number that\n"
+ "# is not a '0' or '1')\n");
+}
+
+void print_snapshot_help(struct seq_file *m, struct trace_iterator *iter)
+{
+ if (iter->tr->allocated_snapshot)
+ seq_puts(m, "#\n# * Snapshot is allocated *\n#\n");
+ else
+ seq_puts(m, "#\n# * Snapshot is freed *\n#\n");
+
+ seq_puts(m, "# Snapshot commands:\n");
+ if (iter->cpu_file == RING_BUFFER_ALL_CPUS)
+ show_snapshot_main_help(m);
+ else
+ show_snapshot_percpu_help(m);
+}
+
+static int tracing_snapshot_open(struct inode *inode, struct file *file)
+{
+ struct trace_array *tr = inode->i_private;
+ struct trace_iterator *iter;
+ struct seq_file *m;
+ int ret;
+
+ ret = tracing_check_open_get_tr(tr);
+ if (ret)
+ return ret;
+
+ if (file->f_mode & FMODE_READ) {
+ iter = __tracing_open(inode, file, true);
+ if (IS_ERR(iter))
+ ret = PTR_ERR(iter);
+ } else {
+ /* Writes still need the seq_file to hold the private data */
+ ret = -ENOMEM;
+ m = kzalloc_obj(*m);
+ if (!m)
+ goto out;
+ iter = kzalloc_obj(*iter);
+ if (!iter) {
+ kfree(m);
+ goto out;
+ }
+ ret = 0;
+
+ iter->tr = tr;
+ iter->array_buffer = &tr->snapshot_buffer;
+ iter->cpu_file = tracing_get_cpu(inode);
+ m->private = iter;
+ file->private_data = m;
+ }
+out:
+ if (ret < 0)
+ trace_array_put(tr);
+
+ return ret;
+}
+
+static void tracing_swap_cpu_buffer(void *tr)
+{
+ update_max_tr_single((struct trace_array *)tr, current, smp_processor_id());
+}
+
+static ssize_t
+tracing_snapshot_write(struct file *filp, const char __user *ubuf, size_t cnt,
+ loff_t *ppos)
+{
+ struct seq_file *m = filp->private_data;
+ struct trace_iterator *iter = m->private;
+ struct trace_array *tr = iter->tr;
+ unsigned long val;
+ int ret;
+
+ ret = tracing_update_buffers(tr);
+ if (ret < 0)
+ return ret;
+
+ ret = kstrtoul_from_user(ubuf, cnt, 10, &val);
+ if (ret)
+ return ret;
+
+ guard(mutex)(&trace_types_lock);
+
+ if (tracer_uses_snapshot(tr->current_trace))
+ return -EBUSY;
+
+ local_irq_disable();
+ arch_spin_lock(&tr->max_lock);
+ if (tr->cond_snapshot)
+ ret = -EBUSY;
+ arch_spin_unlock(&tr->max_lock);
+ local_irq_enable();
+ if (ret)
+ return ret;
+
+ switch (val) {
+ case 0:
+ if (iter->cpu_file != RING_BUFFER_ALL_CPUS)
+ return -EINVAL;
+ if (tr->allocated_snapshot)
+ free_snapshot(tr);
+ break;
+ case 1:
+/* Only allow per-cpu swap if the ring buffer supports it */
+#ifndef CONFIG_RING_BUFFER_ALLOW_SWAP
+ if (iter->cpu_file != RING_BUFFER_ALL_CPUS)
+ return -EINVAL;
+#endif
+ if (tr->allocated_snapshot)
+ ret = resize_buffer_duplicate_size(&tr->snapshot_buffer,
+ &tr->array_buffer, iter->cpu_file);
+
+ ret = tracing_arm_snapshot_locked(tr);
+ if (ret)
+ return ret;
+
+ /* Now, we're going to swap */
+ if (iter->cpu_file == RING_BUFFER_ALL_CPUS) {
+ local_irq_disable();
+ update_max_tr(tr, current, smp_processor_id(), NULL);
+ local_irq_enable();
+ } else {
+ smp_call_function_single(iter->cpu_file, tracing_swap_cpu_buffer,
+ (void *)tr, 1);
+ }
+ tracing_disarm_snapshot(tr);
+ break;
+ default:
+ if (tr->allocated_snapshot) {
+ if (iter->cpu_file == RING_BUFFER_ALL_CPUS)
+ tracing_reset_online_cpus(&tr->snapshot_buffer);
+ else
+ tracing_reset_cpu(&tr->snapshot_buffer, iter->cpu_file);
+ }
+ break;
+ }
+
+ if (ret >= 0) {
+ *ppos += cnt;
+ ret = cnt;
+ }
+
+ return ret;
+}
+
+static int tracing_snapshot_release(struct inode *inode, struct file *file)
+{
+ struct seq_file *m = file->private_data;
+ int ret;
+
+ ret = tracing_release(inode, file);
+
+ if (file->f_mode & FMODE_READ)
+ return ret;
+
+ /* If write only, the seq_file is just a stub */
+ if (m)
+ kfree(m->private);
+ kfree(m);
+
+ return 0;
+}
+
+static int snapshot_raw_open(struct inode *inode, struct file *filp)
+{
+ struct ftrace_buffer_info *info;
+ int ret;
+
+ /* The following checks for tracefs lockdown */
+ ret = tracing_buffers_open(inode, filp);
+ if (ret < 0)
+ return ret;
+
+ info = filp->private_data;
+
+ if (tracer_uses_snapshot(info->iter.trace)) {
+ tracing_buffers_release(inode, filp);
+ return -EBUSY;
+ }
+
+ info->iter.snapshot = true;
+ info->iter.array_buffer = &info->iter.tr->snapshot_buffer;
+
+ return ret;
+}
+
+const struct file_operations snapshot_fops = {
+ .open = tracing_snapshot_open,
+ .read = seq_read,
+ .write = tracing_snapshot_write,
+ .llseek = tracing_lseek,
+ .release = tracing_snapshot_release,
+};
+
+const struct file_operations snapshot_raw_fops = {
+ .open = snapshot_raw_open,
+ .read = tracing_buffers_read,
+ .release = tracing_buffers_release,
+ .splice_read = tracing_buffers_splice_read,
+};
+
+#ifdef CONFIG_TRACER_MAX_TRACE
+static ssize_t
+tracing_max_lat_read(struct file *filp, char __user *ubuf,
+ size_t cnt, loff_t *ppos)
+{
+ struct trace_array *tr = filp->private_data;
+
+ return tracing_nsecs_read(&tr->max_latency, ubuf, cnt, ppos);
+}
+
+static ssize_t
+tracing_max_lat_write(struct file *filp, const char __user *ubuf,
+ size_t cnt, loff_t *ppos)
+{
+ struct trace_array *tr = filp->private_data;
+
+ return tracing_nsecs_write(&tr->max_latency, ubuf, cnt, ppos);
+}
+
+static const struct file_operations tracing_max_lat_fops = {
+ .open = tracing_open_generic_tr,
+ .read = tracing_max_lat_read,
+ .write = tracing_max_lat_write,
+ .llseek = generic_file_llseek,
+ .release = tracing_release_generic_tr,
+};
+#endif /* CONFIG_TRACER_MAX_TRACE */
+
+int get_snapshot_map(struct trace_array *tr)
+{
+ int err = 0;
+
+ /*
+ * Called with mmap_lock held. lockdep would be unhappy if we would now
+ * take trace_types_lock. Instead use the specific
+ * snapshot_trigger_lock.
+ */
+ spin_lock(&tr->snapshot_trigger_lock);
+
+ if (tr->snapshot || tr->mapped == UINT_MAX)
+ err = -EBUSY;
+ else
+ tr->mapped++;
+
+ spin_unlock(&tr->snapshot_trigger_lock);
+
+ /* Wait for update_max_tr() to observe iter->tr->mapped */
+ if (tr->mapped == 1)
+ synchronize_rcu();
+
+ return err;
+
+}
+
+void put_snapshot_map(struct trace_array *tr)
+{
+ spin_lock(&tr->snapshot_trigger_lock);
+ if (!WARN_ON(!tr->mapped))
+ tr->mapped--;
+ spin_unlock(&tr->snapshot_trigger_lock);
+}
+
+#ifdef CONFIG_DYNAMIC_FTRACE
+static void
+ftrace_snapshot(unsigned long ip, unsigned long parent_ip,
+ struct trace_array *tr, struct ftrace_probe_ops *ops,
+ void *data)
+{
+ tracing_snapshot_instance(tr);
+}
+
+static void
+ftrace_count_snapshot(unsigned long ip, unsigned long parent_ip,
+ struct trace_array *tr, struct ftrace_probe_ops *ops,
+ void *data)
+{
+ struct ftrace_func_mapper *mapper = data;
+ long *count = NULL;
+
+ if (mapper)
+ count = (long *)ftrace_func_mapper_find_ip(mapper, ip);
+
+ if (count) {
+
+ if (*count <= 0)
+ return;
+
+ (*count)--;
+ }
+
+ tracing_snapshot_instance(tr);
+}
+
+static int
+ftrace_snapshot_print(struct seq_file *m, unsigned long ip,
+ struct ftrace_probe_ops *ops, void *data)
+{
+ struct ftrace_func_mapper *mapper = data;
+ long *count = NULL;
+
+ seq_printf(m, "%ps:", (void *)ip);
+
+ seq_puts(m, "snapshot");
+
+ if (mapper)
+ count = (long *)ftrace_func_mapper_find_ip(mapper, ip);
+
+ if (count)
+ seq_printf(m, ":count=%ld\n", *count);
+ else
+ seq_puts(m, ":unlimited\n");
+
+ return 0;
+}
+
+static int
+ftrace_snapshot_init(struct ftrace_probe_ops *ops, struct trace_array *tr,
+ unsigned long ip, void *init_data, void **data)
+{
+ struct ftrace_func_mapper *mapper = *data;
+
+ if (!mapper) {
+ mapper = allocate_ftrace_func_mapper();
+ if (!mapper)
+ return -ENOMEM;
+ *data = mapper;
+ }
+
+ return ftrace_func_mapper_add_ip(mapper, ip, init_data);
+}
+
+static void
+ftrace_snapshot_free(struct ftrace_probe_ops *ops, struct trace_array *tr,
+ unsigned long ip, void *data)
+{
+ struct ftrace_func_mapper *mapper = data;
+
+ if (!ip) {
+ if (!mapper)
+ return;
+ free_ftrace_func_mapper(mapper, NULL);
+ return;
+ }
+
+ ftrace_func_mapper_remove_ip(mapper, ip);
+}
+
+static struct ftrace_probe_ops snapshot_probe_ops = {
+ .func = ftrace_snapshot,
+ .print = ftrace_snapshot_print,
+};
+
+static struct ftrace_probe_ops snapshot_count_probe_ops = {
+ .func = ftrace_count_snapshot,
+ .print = ftrace_snapshot_print,
+ .init = ftrace_snapshot_init,
+ .free = ftrace_snapshot_free,
+};
+
+static int
+ftrace_trace_snapshot_callback(struct trace_array *tr, struct ftrace_hash *hash,
+ char *glob, char *cmd, char *param, int enable)
+{
+ struct ftrace_probe_ops *ops;
+ void *count = (void *)-1;
+ char *number;
+ int ret;
+
+ if (!tr)
+ return -ENODEV;
+
+ /* hash funcs only work with set_ftrace_filter */
+ if (!enable)
+ return -EINVAL;
+
+ ops = param ? &snapshot_count_probe_ops : &snapshot_probe_ops;
+
+ if (glob[0] == '!') {
+ ret = unregister_ftrace_function_probe_func(glob+1, tr, ops);
+ if (!ret)
+ tracing_disarm_snapshot(tr);
+
+ return ret;
+ }
+
+ if (!param)
+ goto out_reg;
+
+ number = strsep(¶m, ":");
+
+ if (!strlen(number))
+ goto out_reg;
+
+ /*
+ * We use the callback data field (which is a pointer)
+ * as our counter.
+ */
+ ret = kstrtoul(number, 0, (unsigned long *)&count);
+ if (ret)
+ return ret;
+
+ out_reg:
+ ret = tracing_arm_snapshot(tr);
+ if (ret < 0)
+ return ret;
+
+ ret = register_ftrace_function_probe(glob, tr, ops, count);
+ if (ret < 0)
+ tracing_disarm_snapshot(tr);
+
+ return ret < 0 ? ret : 0;
+}
+
+static struct ftrace_func_command ftrace_snapshot_cmd = {
+ .name = "snapshot",
+ .func = ftrace_trace_snapshot_callback,
+};
+
+__init int register_snapshot_cmd(void)
+{
+ return register_ftrace_command(&ftrace_snapshot_cmd);
+}
+#endif /* CONFIG_DYNAMIC_FTRACE */
+
+int trace_allocate_snapshot(struct trace_array *tr, int size)
+{
+ int ret;
+
+ /* Fix mapped buffer trace arrays do not have snapshot buffers */
+ if (tr->range_addr_start)
+ return 0;
+
+ /* allocate_snapshot can only be true during system boot */
+ ret = allocate_trace_buffer(tr, &tr->snapshot_buffer,
+ allocate_snapshot ? size : 1);
+ if (ret < 0)
+ return -ENOMEM;
+
+ tr->allocated_snapshot = allocate_snapshot;
+
+ allocate_snapshot = false;
+ return 0;
+}
+
+__init static bool tr_needs_alloc_snapshot(const char *name)
+{
+ char *test;
+ int len = strlen(name);
+ bool ret;
+
+ if (!boot_snapshot_index)
+ return false;
+
+ if (strncmp(name, boot_snapshot_info, len) == 0 &&
+ boot_snapshot_info[len] == '\t')
+ return true;
+
+ test = kmalloc(strlen(name) + 3, GFP_KERNEL);
+ if (!test)
+ return false;
+
+ sprintf(test, "\t%s\t", name);
+ ret = strstr(boot_snapshot_info, test) == NULL;
+ kfree(test);
+ return ret;
+}
+
+__init void do_allocate_snapshot(const char *name)
+{
+ if (!tr_needs_alloc_snapshot(name))
+ return;
+
+ /*
+ * When allocate_snapshot is set, the next call to
+ * allocate_trace_buffers() (called by trace_array_get_by_name())
+ * will allocate the snapshot buffer. That will also clear
+ * this flag.
+ */
+ allocate_snapshot = true;
+}
+
+void __init ftrace_boot_snapshot(void)
+{
+ struct trace_array *tr;
+
+ if (!snapshot_at_boot)
+ return;
+
+ list_for_each_entry(tr, &ftrace_trace_arrays, list) {
+ if (!tr->allocated_snapshot)
+ continue;
+
+ tracing_snapshot_instance(tr);
+ trace_array_puts(tr, "** Boot snapshot taken **\n");
+ }
+}
--
2.51.0
^ permalink raw reply related
* Re: [PATCH v3 1/2] tracing: preserve repeated boot-time tracing parameters
From: Steven Rostedt @ 2026-03-24 18:43 UTC (permalink / raw)
To: Wesley Atwell
Cc: mhiramat, mark.rutland, mathieu.desnoyers, corbet, skhan,
linux-doc, linux-kernel, linux-trace-kernel
In-Reply-To: <20260310064715.527906-2-atwellwea@gmail.com>
On Tue, 10 Mar 2026 00:47:14 -0600
Wesley Atwell <atwellwea@gmail.com> wrote:
Hi,
FYI, the tracing subsystem uses capital letters in subjects:
tracing: Preserve repeated boot-time tracing parameters
> Some tracing boot parameters already accept delimited value lists, but
> their __setup() handlers keep only the last instance seen at boot.
> Make repeated instances append to the same boot-time buffer in the
> format each parser already consumes, and document that behavior in
> admin-guide/kernel-parameters.txt.
>
> Use a shared trace_append_boot_param() helper for the ftrace filters,
> trace_options, and kprobe_event boot parameters. trace_trigger=
> tokenizes its backing storage in place, so keep a running offset and
> only parse the newly appended chunk into bootup_triggers[].
>
> This also lets Bootconfig array values work naturally when they expand
> to repeated param=value entries.
>
> Signed-off-by: Wesley Atwell <atwellwea@gmail.com>
> ---
> .../admin-guide/kernel-parameters.txt | 18 ++++++++++--
> kernel/trace/ftrace.c | 12 +++++---
> kernel/trace/trace.c | 3 +-
> kernel/trace/trace.h | 29 +++++++++++++++++++
> kernel/trace/trace_events.c | 26 +++++++++++++++--
> kernel/trace/trace_kprobe.c | 3 +-
> 6 files changed, 79 insertions(+), 12 deletions(-)
>
> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> index 55ffc0f8858a..203863c1839b 100644
> --- a/Documentation/admin-guide/kernel-parameters.txt
> +++ b/Documentation/admin-guide/kernel-parameters.txt
> @@ -1803,13 +1803,15 @@ Kernel parameters
> tracer at boot up. function-list is a comma-separated
> list of functions. This list can be changed at run
> time by the set_ftrace_filter file in the debugfs
> - tracing directory.
> + tracing directory. Repeated instances append more
> + functions to the same list.
>
> ftrace_notrace=[function-list]
> [FTRACE] Do not trace the functions specified in
> function-list. This list can be changed at run time
> by the set_ftrace_notrace file in the debugfs
> - tracing directory.
> + tracing directory. Repeated instances append more
> + functions to the same list.
>
> ftrace_graph_filter=[function-list]
> [FTRACE] Limit the top level callers functions traced
> @@ -1817,12 +1819,16 @@ Kernel parameters
> function-list is a comma-separated list of functions
> that can be changed at run time by the
> set_graph_function file in the debugfs tracing directory.
> + Repeated instances append more functions to the same
> + list.
>
> ftrace_graph_notrace=[function-list]
> [FTRACE] Do not trace from the functions specified in
> function-list. This list is a comma-separated list of
> functions that can be changed at run time by the
> set_graph_notrace file in the debugfs tracing directory.
> + Repeated instances append more functions to the same
> + list.
>
> ftrace_graph_max_depth=<uint>
> [FTRACE] Used with the function graph tracer. This is
> @@ -3053,6 +3059,8 @@ Kernel parameters
> The probe-list is a semicolon delimited list of probe
> definitions. Each definition is same as kprobe_events
> interface, but the parameters are comma delimited.
> + Repeated instances append more probe definitions to
> + the same boot-time list.
> For example, to add a kprobe event on vfs_read with
> arg1 and arg2, add to the command line;
>
> @@ -7820,6 +7828,9 @@ Kernel parameters
>
> /sys/kernel/tracing/trace_options
>
> + Repeated instances append more options to the same
> + boot-time list.
> +
> For example, to enable stacktrace option (to dump the
> stack trace of each event), add to the command line:
>
> @@ -7831,7 +7842,8 @@ Kernel parameters
> trace_trigger=[trigger-list]
> [FTRACE] Add an event trigger on specific events.
> Set a trigger on top of a specific event, with an optional
> - filter.
> + filter. Repeated instances append more triggers to
> + the same boot-time list.
I know Masami mentioned to document this, but honestly, I don't think this
update is needed. Please remove it.
>
> The format is "trace_trigger=<event>.<trigger>[ if <filter>],..."
> Where more than one trigger may be specified that are comma delimited.
> diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
> index 8df69e702706..d0a486b63ed6 100644
> --- a/kernel/trace/ftrace.c
> +++ b/kernel/trace/ftrace.c
> @@ -6841,7 +6841,8 @@ bool ftrace_filter_param __initdata;
> static int __init set_ftrace_notrace(char *str)
> {
> ftrace_filter_param = true;
> - strscpy(ftrace_notrace_buf, str, FTRACE_FILTER_SIZE);
> + trace_append_boot_param(ftrace_notrace_buf, str, ',',
> + FTRACE_FILTER_SIZE);
> return 1;
> }
> __setup("ftrace_notrace=", set_ftrace_notrace);
> @@ -6849,7 +6850,8 @@ __setup("ftrace_notrace=", set_ftrace_notrace);
> static int __init set_ftrace_filter(char *str)
> {
> ftrace_filter_param = true;
> - strscpy(ftrace_filter_buf, str, FTRACE_FILTER_SIZE);
> + trace_append_boot_param(ftrace_filter_buf, str, ',',
> + FTRACE_FILTER_SIZE);
> return 1;
> }
> __setup("ftrace_filter=", set_ftrace_filter);
> @@ -6861,14 +6863,16 @@ static int ftrace_graph_set_hash(struct ftrace_hash *hash, char *buffer);
>
> static int __init set_graph_function(char *str)
> {
> - strscpy(ftrace_graph_buf, str, FTRACE_FILTER_SIZE);
> + trace_append_boot_param(ftrace_graph_buf, str, ',',
> + FTRACE_FILTER_SIZE);
> return 1;
> }
> __setup("ftrace_graph_filter=", set_graph_function);
>
> static int __init set_graph_notrace_function(char *str)
> {
> - strscpy(ftrace_graph_notrace_buf, str, FTRACE_FILTER_SIZE);
> + trace_append_boot_param(ftrace_graph_notrace_buf, str, ',',
> + FTRACE_FILTER_SIZE);
> return 1;
> }
> __setup("ftrace_graph_notrace=", set_graph_notrace_function);
> diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
> index ebd996f8710e..5086239a75dc 100644
> --- a/kernel/trace/trace.c
> +++ b/kernel/trace/trace.c
> @@ -329,7 +329,8 @@ static char trace_boot_options_buf[MAX_TRACER_SIZE] __initdata;
>
> static int __init set_trace_boot_options(char *str)
> {
> - strscpy(trace_boot_options_buf, str, MAX_TRACER_SIZE);
> + trace_append_boot_param(trace_boot_options_buf, str, ',',
> + MAX_TRACER_SIZE);
> return 1;
> }
> __setup("trace_options=", set_trace_boot_options);
> diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h
> index b8f3804586a0..4f5abac4bd19 100644
> --- a/kernel/trace/trace.h
> +++ b/kernel/trace/trace.h
> @@ -23,6 +23,7 @@
> #include <linux/once_lite.h>
> #include <linux/ftrace_regs.h>
> #include <linux/llist.h>
> +#include <linux/string.h>
>
> #include "pid_list.h"
>
> @@ -262,6 +263,34 @@ static inline bool still_need_pid_events(int type, struct trace_pid_list *pid_li
> (!(type & TRACE_NO_PIDS) && no_pid_list);
> }
>
> +/*
> + * Repeated boot parameters, including Bootconfig array expansions, need
> + * to stay in the delimiter form that the existing parser consumes.
> + */
> +static inline void __init trace_append_boot_param(char *buf, const char *str,
> + char sep, size_t size)
> +{
Masami said:
Please make a generic append function in kernel/trace/trace.h, e.g.
void trace_append_boot_param(char *buf, const char *str, char sep, size_t ssize);
and use it instead of strscpy.
He did not say to make a static inline in the header. Please make this a
normal function in trace.c and just add the prototype in the header.
-- Steve
> + size_t len, str_len;
> +
> + if (buf[0] == '\0') {
> + strscpy(buf, str, size);
> + return;
> + }
> +
> + str_len = strlen(str);
> + if (!str_len)
> + return;
> +
> + len = strlen(buf);
> + if (len >= size - 1)
> + return;
> + if (str_len >= size - len - 1)
> + return;
> +
> + buf[len] = sep;
> + strscpy(buf + len + 1, str, size - len - 1);
> +}
> +
> typedef bool (*cond_update_fn_t)(struct trace_array *tr, void *cond_data);
>
> /**
> diff --git a/kernel/trace/trace_events.c b/kernel/trace/trace_events.c
> index 249d1cba72c0..5f72be33f2d1 100644
> --- a/kernel/trace/trace_events.c
> +++ b/kernel/trace/trace_events.c
> @@ -3679,20 +3679,40 @@ static struct boot_triggers {
> } bootup_triggers[MAX_BOOT_TRIGGERS];
>
> static char bootup_trigger_buf[COMMAND_LINE_SIZE];
> +static size_t bootup_trigger_buf_len;
> static int nr_boot_triggers;
>
> static __init int setup_trace_triggers(char *str)
> {
> char *trigger;
> char *buf;
> + size_t start, str_len;
> int i;
>
> - strscpy(bootup_trigger_buf, str, COMMAND_LINE_SIZE);
> + if (bootup_trigger_buf_len >= COMMAND_LINE_SIZE)
> + return 1;
> +
> + start = bootup_trigger_buf_len;
> + if (start && !*str)
> + return 1;
> +
> + str_len = strlen(str);
> + if (start && str_len >= COMMAND_LINE_SIZE - start)
> + return 1;
> +
> + /*
> + * trace_trigger= parsing tokenizes the backing storage in place.
> + * Copy each repeated parameter into fresh space and only parse that
> + * newly copied chunk here.
> + */
> + trace_append_boot_param(bootup_trigger_buf + start, str, '\0',
> + COMMAND_LINE_SIZE - start);
> + bootup_trigger_buf_len += strlen(bootup_trigger_buf + start) + 1;
> trace_set_ring_buffer_expanded(NULL);
> disable_tracing_selftest("running event triggers");
>
> - buf = bootup_trigger_buf;
> - for (i = 0; i < MAX_BOOT_TRIGGERS; i++) {
> + buf = bootup_trigger_buf + start;
> + for (i = nr_boot_triggers; i < MAX_BOOT_TRIGGERS; i++) {
> trigger = strsep(&buf, ",");
> if (!trigger)
> break;
> diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c
> index a5dbb72528e0..e9f1c55aea64 100644
> --- a/kernel/trace/trace_kprobe.c
> +++ b/kernel/trace/trace_kprobe.c
> @@ -31,7 +31,8 @@ static char kprobe_boot_events_buf[COMMAND_LINE_SIZE] __initdata;
>
> static int __init set_kprobe_boot_events(char *str)
> {
> - strscpy(kprobe_boot_events_buf, str, COMMAND_LINE_SIZE);
> + trace_append_boot_param(kprobe_boot_events_buf, str, ';',
> + COMMAND_LINE_SIZE);
> disable_tracing_selftest("running kprobe events");
>
> return 1;
^ permalink raw reply
* Re: [PATCH v3 2/2] tracing: drain deferred trigger frees if kthread startup fails
From: Steven Rostedt @ 2026-03-24 18:53 UTC (permalink / raw)
To: Wesley Atwell
Cc: mhiramat, mark.rutland, mathieu.desnoyers, corbet, skhan,
linux-doc, linux-kernel, linux-trace-kernel
In-Reply-To: <20260310064715.527906-3-atwellwea@gmail.com>
On Tue, 10 Mar 2026 00:47:15 -0600
Wesley Atwell <atwellwea@gmail.com> wrote:
> Boot-time trigger registration can fail before the trigger-data cleanup
> kthread exists. Deferring those frees until late init is fine, but the
> post-boot fallback must still drain the deferred list if kthread
> creation never succeeds.
>
> Otherwise, boot-deferred nodes can accumulate on
> trigger_data_free_list, later frees fall back to synchronously freeing
> only the current object, and the older queued entries are leaked
> forever.
>
> Keep the deferred boot-time behavior, but when kthread creation fails,
> drain the whole queued list synchronously. Do the same in the late-init
> drain path so queued entries are not stranded there either.
>
> Fixes: 61d445af0a7c ("tracing: Add bulk garbage collection of freeing event_trigger_data")
> Signed-off-by: Wesley Atwell <atwellwea@gmail.com>
> ---
> kernel/trace/trace_events_trigger.c | 79 ++++++++++++++++++++++++-----
> 1 file changed, 66 insertions(+), 13 deletions(-)
>
> diff --git a/kernel/trace/trace_events_trigger.c b/kernel/trace/trace_events_trigger.c
> index d5230b759a2d..428b46272ac8 100644
> --- a/kernel/trace/trace_events_trigger.c
> +++ b/kernel/trace/trace_events_trigger.c
> @@ -22,6 +22,39 @@ static struct task_struct *trigger_kthread;
> static struct llist_head trigger_data_free_list;
> static DEFINE_MUTEX(trigger_data_kthread_mutex);
>
> +static int trigger_kthread_fn(void *ignore);
> +
> +static void trigger_start_kthread_locked(void)
> +{
> + lockdep_assert_held(&trigger_data_kthread_mutex);
> +
> + if (!trigger_kthread) {
> + struct task_struct *kthread;
> +
> + kthread = kthread_create(trigger_kthread_fn, NULL,
> + "trigger_data_free");
This only creates the thread and doesn't start it. The function name is
confusing. Please change it to:
trigger_create_kthread_locked()
> + if (!IS_ERR(kthread))
> + WRITE_ONCE(trigger_kthread, kthread);
> + }
> +}
> +
> +static void trigger_data_free_queued_locked(void)
> +{
> + struct event_trigger_data *data, *tmp;
> + struct llist_node *llnodes;
> +
> + lockdep_assert_held(&trigger_data_kthread_mutex);
> +
> + llnodes = llist_del_all(&trigger_data_free_list);
> + if (!llnodes)
> + return;
> +
> + tracepoint_synchronize_unregister();
> +
> + llist_for_each_entry_safe(data, tmp, llnodes, llist)
> + kfree(data);
> +}
> +
> /* Bulk garbage collection of event_trigger_data elements */
> static int trigger_kthread_fn(void *ignore)
> {
> @@ -56,30 +89,50 @@ void trigger_data_free(struct event_trigger_data *data)
> if (data->cmd_ops->set_filter)
> data->cmd_ops->set_filter(NULL, data, NULL);
>
> + /*
> + * Boot-time trigger registration can fail before kthread creation
> + * works. Keep the deferred-free semantics during boot and let late
> + * init start the kthread to drain the list.
> + */
> + if (system_state == SYSTEM_BOOTING && !trigger_kthread) {
> + llist_add(&data->llist, &trigger_data_free_list);
> + return;
> + }
> +
> if (unlikely(!trigger_kthread)) {
> guard(mutex)(&trigger_data_kthread_mutex);
> +
> + trigger_start_kthread_locked();
> /* Check again after taking mutex */
> if (!trigger_kthread) {
> - struct task_struct *kthread;
> -
> - kthread = kthread_create(trigger_kthread_fn, NULL,
> - "trigger_data_free");
> - if (!IS_ERR(kthread))
> - WRITE_ONCE(trigger_kthread, kthread);
> + llist_add(&data->llist, &trigger_data_free_list);
> + /* Drain the queued frees synchronously if startup failed. */
s/startup/creation/
> + trigger_data_free_queued_locked();
> + return;
> }
> }
-- Steve
>
> - if (!trigger_kthread) {
> - /* Do it the slow way */
> - tracepoint_synchronize_unregister();
> - kfree(data);
> - return;
> - }
> -
> llist_add(&data->llist, &trigger_data_free_list);
> wake_up_process(trigger_kthread);
> }
>
> +static int __init trigger_data_free_init(void)
> +{
> + guard(mutex)(&trigger_data_kthread_mutex);
> +
> + if (llist_empty(&trigger_data_free_list))
> + return 0;
> +
> + trigger_start_kthread_locked();
> + if (trigger_kthread)
> + wake_up_process(trigger_kthread);
> + else
> + trigger_data_free_queued_locked();
> +
> + return 0;
> +}
> +late_initcall(trigger_data_free_init);
> +
> static inline void data_ops_trigger(struct event_trigger_data *data,
> struct trace_buffer *buffer, void *rec,
> struct ring_buffer_event *event)
^ permalink raw reply
* Re: [External] Re: [RFC PATCH bpf-next 2/3] ftrace: Use kallsyms binary search for single-symbol lookup
From: Steven Rostedt @ 2026-03-24 21:03 UTC (permalink / raw)
To: Andrey Grodzovsky
Cc: bpf, ast, daniel, andrii, jolsa, linux-trace-kernel,
linux-open-source
In-Reply-To: <CAOu3gNiHGBEP2oU07fY73=qOBzLJou+_wUY-6txhrnvRo84pAg@mail.gmail.com>
On Wed, 25 Feb 2026 20:22:50 -0500
Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com> wrote:
> On Wed, Feb 25, 2026 at 6:32 PM Steven Rostedt <rostedt@goodmis.org> wrote:
> >
> > On Wed, 25 Feb 2026 10:25:04 -0500
> > Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com> wrote:
> >
> > > Hey Steve, are there any extra steps required on my side to make this go
> > > through your tree?
> >
> > I see you Cc'd linux-trace-kernel which places it into the tracing
> > patchwork. If there's no dependency on any other patch, I can add it to
> > my 7.1 queue. That is, after I get some more time to review it a little
> > deeper.
> >
> > -- Steve
>
> There are no dependencies, I performed the original optimization in libbpf
> (patch 1) in hopes this will make session kprobes faster then legacy
> kprobe/kretprobe pairs and when it didn't, I dug deeper and came up
> with this second optimization here (patch 2).
>
> Let me know of any issues once you have time to review, in the meantime
> I will roll V2 with all the fixes per Jiri's comments including this patch.
Sorry for the late reply:
Acked-by: Steven Rostedt (Google) <rostedt@goodmis.org>
-- Steve
^ permalink raw reply
* Re: [PATCH v12 0/4] ring-buffer: Making persistent ring buffers robust
From: Steven Rostedt @ 2026-03-24 21:25 UTC (permalink / raw)
To: Masami Hiramatsu (Google)
Cc: Mathieu Desnoyers, linux-kernel, linux-trace-kernel, Ian Rogers
In-Reply-To: <177431040276.3708637.10277850418341653677.stgit@mhiramat.tok.corp.google.com>
On Tue, 24 Mar 2026 09:00:03 +0900
"Masami Hiramatsu (Google)" <mhiramat@kernel.org> wrote:
> Hi,
>
> Here is the 12th version of improvement patches for making persistent
> ring buffers robust to failures.
> The previous version is here:
>
> https://lore.kernel.org/all/177391152793.193994.8986943289250629418.stgit@mhiramat.tok.corp.google.com/
>
>
> In this version, I rebased it on tracing/fixes branch (thus
> the first fix patch has been removed) and fixed
> a build error which came from a copy-paste mistake.
>
Hi Masami,
Can you rebase this on the ring-buffer/for-next branch?
It has the remote updates for pkvm tracing, and there's some conflicts.
That branch is also used by the ARM tree so it can't be rebased.
Thanks,
-- Steve
^ permalink raw reply
* Re: [PATCH] tracing: Allow perf to read synthetic events
From: Steven Rostedt @ 2026-03-24 21:41 UTC (permalink / raw)
To: Jiri Olsa
Cc: LKML, Linux Trace Kernel, Masami Hiramatsu, Mathieu Desnoyers,
Ian Rogers, Arnaldo Carvalho de Melo, Namhyung Kim,
Peter Zijlstra
In-Reply-To: <20251229183632.7b518b8d@gandalf.local.home>
On Mon, 29 Dec 2025 18:36:32 -0500
Steven Rostedt <rostedt@goodmis.org> wrote:
> diff --git a/include/traceevent/event-parse.h b/include/traceevent/event-parse.h
> index ebfc2c7..9c1abfa 100644
> --- a/include/traceevent/event-parse.h
> +++ b/include/traceevent/event-parse.h
> @@ -138,6 +138,11 @@ struct tep_print_arg_bitmask {
> struct tep_format_field *field;
> };
>
> +struct tep_print_arg_stacktrace {
> + char *stacktrace;
> + struct tep_format_field *field;
> +};
> +
[..]
So this has been already applied to libtraceevent 1.9. I'll get the kernel
side patch ready for the next merge window.
-- Steve
^ permalink raw reply
* [PATCH v4 0/2] tracing: Preserve repeated boot-time parameters and drain deferred trigger frees
From: Wesley Atwell @ 2026-03-24 22:13 UTC (permalink / raw)
To: linux-trace-kernel
Cc: linux-kernel, rostedt, mhiramat, mark.rutland, mathieu.desnoyers,
tom.zanussi, Wesley Atwell
Patch 1 preserves repeated boot-time tracing parameters by appending to
the existing parser buffers instead of overwriting earlier values, so
repeated command-line entries and Bootconfig array expansions work with
the delimited formats the parsers already consume.
Patch 2 fixes the deferred event-trigger free path so queued frees are
still drained if the cleanup kthread never comes up after boot.
v4:
- drop the kernel-parameters.txt update from patch 1
- move trace_append_boot_param() into trace.c and keep only the
prototype in trace.h
- capitalize the tracing patch subjects
- rename trigger_start_kthread_locked() to trigger_create_kthread_locked()
- change the failure comment to say "creation failed"
Wesley Atwell (2):
tracing: Preserve repeated boot-time tracing parameters
tracing: Drain deferred trigger frees if kthread creation fails
kernel/trace/ftrace.c | 12 +++--
kernel/trace/trace.c | 31 ++++++++++-
kernel/trace/trace.h | 2 +
kernel/trace/trace_events.c | 26 ++++++++--
kernel/trace/trace_events_trigger.c | 79 ++++++++++++++++++++++++-----
kernel/trace/trace_kprobe.c | 3 +-
6 files changed, 131 insertions(+), 22 deletions(-)
--
2.43.0
^ permalink raw reply
* [PATCH v4 1/2] tracing: Preserve repeated boot-time tracing parameters
From: Wesley Atwell @ 2026-03-24 22:13 UTC (permalink / raw)
To: linux-trace-kernel
Cc: linux-kernel, rostedt, mhiramat, mark.rutland, mathieu.desnoyers,
tom.zanussi, Wesley Atwell
In-Reply-To: <20260324221326.1395799-1-atwellwea@gmail.com>
Some tracing boot parameters already accept delimited value lists, but
their __setup() handlers keep only the last instance seen at boot.
Make repeated instances append to the same boot-time buffer in the
format each parser already consumes.
Use a shared trace_append_boot_param() helper for the ftrace filters,
trace_options, and kprobe_event boot parameters. trace_trigger=
tokenizes its backing storage in place, so keep a running offset and
only parse the newly appended chunk into bootup_triggers[].
This also lets Bootconfig array values work naturally when they expand
to repeated param=value entries.
Signed-off-by: Wesley Atwell <atwellwea@gmail.com>
---
kernel/trace/ftrace.c | 12 ++++++++----
kernel/trace/trace.c | 31 ++++++++++++++++++++++++++++++-
kernel/trace/trace.h | 2 ++
kernel/trace/trace_events.c | 26 +++++++++++++++++++++++---
kernel/trace/trace_kprobe.c | 3 ++-
5 files changed, 65 insertions(+), 9 deletions(-)
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index 413310912609..8bd3dd1d549c 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -6841,7 +6841,8 @@ bool ftrace_filter_param __initdata;
static int __init set_ftrace_notrace(char *str)
{
ftrace_filter_param = true;
- strscpy(ftrace_notrace_buf, str, FTRACE_FILTER_SIZE);
+ trace_append_boot_param(ftrace_notrace_buf, str, ',',
+ FTRACE_FILTER_SIZE);
return 1;
}
__setup("ftrace_notrace=", set_ftrace_notrace);
@@ -6849,7 +6850,8 @@ __setup("ftrace_notrace=", set_ftrace_notrace);
static int __init set_ftrace_filter(char *str)
{
ftrace_filter_param = true;
- strscpy(ftrace_filter_buf, str, FTRACE_FILTER_SIZE);
+ trace_append_boot_param(ftrace_filter_buf, str, ',',
+ FTRACE_FILTER_SIZE);
return 1;
}
__setup("ftrace_filter=", set_ftrace_filter);
@@ -6861,14 +6863,16 @@ static int ftrace_graph_set_hash(struct ftrace_hash *hash, char *buffer);
static int __init set_graph_function(char *str)
{
- strscpy(ftrace_graph_buf, str, FTRACE_FILTER_SIZE);
+ trace_append_boot_param(ftrace_graph_buf, str, ',',
+ FTRACE_FILTER_SIZE);
return 1;
}
__setup("ftrace_graph_filter=", set_graph_function);
static int __init set_graph_notrace_function(char *str)
{
- strscpy(ftrace_graph_notrace_buf, str, FTRACE_FILTER_SIZE);
+ trace_append_boot_param(ftrace_graph_notrace_buf, str, ',',
+ FTRACE_FILTER_SIZE);
return 1;
}
__setup("ftrace_graph_notrace=", set_graph_notrace_function);
diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index a626211ceb9a..1b1f6f9035c6 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -228,6 +228,34 @@ static int boot_instance_index;
static char boot_snapshot_info[COMMAND_LINE_SIZE] __initdata;
static int boot_snapshot_index;
+/*
+ * Repeated boot parameters, including Bootconfig array expansions, need
+ * to stay in the delimiter form that the existing parser consumes.
+ */
+void __init trace_append_boot_param(char *buf, const char *str, char sep,
+ size_t size)
+{
+ size_t len, str_len;
+
+ if (!buf[0]) {
+ strscpy(buf, str, size);
+ return;
+ }
+
+ str_len = strlen(str);
+ if (!str_len)
+ return;
+
+ len = strlen(buf);
+ if (len >= size - 1)
+ return;
+ if (str_len >= size - len - 1)
+ return;
+
+ buf[len] = sep;
+ strscpy(buf + len + 1, str, size - len - 1);
+}
+
static int __init set_cmdline_ftrace(char *str)
{
strscpy(bootup_tracer_buf, str, MAX_TRACER_SIZE);
@@ -329,7 +357,8 @@ static char trace_boot_options_buf[MAX_TRACER_SIZE] __initdata;
static int __init set_trace_boot_options(char *str)
{
- strscpy(trace_boot_options_buf, str, MAX_TRACER_SIZE);
+ trace_append_boot_param(trace_boot_options_buf, str, ',',
+ MAX_TRACER_SIZE);
return 1;
}
__setup("trace_options=", set_trace_boot_options);
diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h
index b8f3804586a0..64ce179d12dd 100644
--- a/kernel/trace/trace.h
+++ b/kernel/trace/trace.h
@@ -863,6 +863,8 @@ extern int DYN_FTRACE_TEST_NAME(void);
extern int DYN_FTRACE_TEST_NAME2(void);
extern void trace_set_ring_buffer_expanded(struct trace_array *tr);
+void __init trace_append_boot_param(char *buf, const char *str,
+ char sep, size_t size);
extern bool tracing_selftest_disabled;
#ifdef CONFIG_FTRACE_STARTUP_TEST
diff --git a/kernel/trace/trace_events.c b/kernel/trace/trace_events.c
index 249d1cba72c0..5f72be33f2d1 100644
--- a/kernel/trace/trace_events.c
+++ b/kernel/trace/trace_events.c
@@ -3679,20 +3679,40 @@ static struct boot_triggers {
} bootup_triggers[MAX_BOOT_TRIGGERS];
static char bootup_trigger_buf[COMMAND_LINE_SIZE];
+static size_t bootup_trigger_buf_len;
static int nr_boot_triggers;
static __init int setup_trace_triggers(char *str)
{
char *trigger;
char *buf;
+ size_t start, str_len;
int i;
- strscpy(bootup_trigger_buf, str, COMMAND_LINE_SIZE);
+ if (bootup_trigger_buf_len >= COMMAND_LINE_SIZE)
+ return 1;
+
+ start = bootup_trigger_buf_len;
+ if (start && !*str)
+ return 1;
+
+ str_len = strlen(str);
+ if (start && str_len >= COMMAND_LINE_SIZE - start)
+ return 1;
+
+ /*
+ * trace_trigger= parsing tokenizes the backing storage in place.
+ * Copy each repeated parameter into fresh space and only parse that
+ * newly copied chunk here.
+ */
+ trace_append_boot_param(bootup_trigger_buf + start, str, '\0',
+ COMMAND_LINE_SIZE - start);
+ bootup_trigger_buf_len += strlen(bootup_trigger_buf + start) + 1;
trace_set_ring_buffer_expanded(NULL);
disable_tracing_selftest("running event triggers");
- buf = bootup_trigger_buf;
- for (i = 0; i < MAX_BOOT_TRIGGERS; i++) {
+ buf = bootup_trigger_buf + start;
+ for (i = nr_boot_triggers; i < MAX_BOOT_TRIGGERS; i++) {
trigger = strsep(&buf, ",");
if (!trigger)
break;
diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c
index a5dbb72528e0..e9f1c55aea64 100644
--- a/kernel/trace/trace_kprobe.c
+++ b/kernel/trace/trace_kprobe.c
@@ -31,7 +31,8 @@ static char kprobe_boot_events_buf[COMMAND_LINE_SIZE] __initdata;
static int __init set_kprobe_boot_events(char *str)
{
- strscpy(kprobe_boot_events_buf, str, COMMAND_LINE_SIZE);
+ trace_append_boot_param(kprobe_boot_events_buf, str, ';',
+ COMMAND_LINE_SIZE);
disable_tracing_selftest("running kprobe events");
return 1;
--
2.43.0
^ permalink raw reply related
* [PATCH v4 2/2] tracing: Drain deferred trigger frees if kthread creation fails
From: Wesley Atwell @ 2026-03-24 22:13 UTC (permalink / raw)
To: linux-trace-kernel
Cc: linux-kernel, rostedt, mhiramat, mark.rutland, mathieu.desnoyers,
tom.zanussi, Wesley Atwell
In-Reply-To: <20260324221326.1395799-1-atwellwea@gmail.com>
Boot-time trigger registration can fail before the trigger-data cleanup
kthread exists. Deferring those frees until late init is fine, but the
post-boot fallback must still drain the deferred list if kthread
creation never succeeds.
Otherwise, boot-deferred nodes can accumulate on
trigger_data_free_list, later frees fall back to synchronously freeing
only the current object, and the older queued entries are leaked
forever.
Keep the deferred boot-time behavior, but when kthread creation fails,
drain the whole queued list synchronously. Do the same in the late-init
drain path so queued entries are not stranded there either.
Fixes: 61d445af0a7c ("tracing: Add bulk garbage collection of freeing event_trigger_data")
Signed-off-by: Wesley Atwell <atwellwea@gmail.com>
---
kernel/trace/trace_events_trigger.c | 79 ++++++++++++++++++++++++-----
1 file changed, 66 insertions(+), 13 deletions(-)
diff --git a/kernel/trace/trace_events_trigger.c b/kernel/trace/trace_events_trigger.c
index d5230b759a2d..655db2e82513 100644
--- a/kernel/trace/trace_events_trigger.c
+++ b/kernel/trace/trace_events_trigger.c
@@ -22,6 +22,39 @@ static struct task_struct *trigger_kthread;
static struct llist_head trigger_data_free_list;
static DEFINE_MUTEX(trigger_data_kthread_mutex);
+static int trigger_kthread_fn(void *ignore);
+
+static void trigger_create_kthread_locked(void)
+{
+ lockdep_assert_held(&trigger_data_kthread_mutex);
+
+ if (!trigger_kthread) {
+ struct task_struct *kthread;
+
+ kthread = kthread_create(trigger_kthread_fn, NULL,
+ "trigger_data_free");
+ if (!IS_ERR(kthread))
+ WRITE_ONCE(trigger_kthread, kthread);
+ }
+}
+
+static void trigger_data_free_queued_locked(void)
+{
+ struct event_trigger_data *data, *tmp;
+ struct llist_node *llnodes;
+
+ lockdep_assert_held(&trigger_data_kthread_mutex);
+
+ llnodes = llist_del_all(&trigger_data_free_list);
+ if (!llnodes)
+ return;
+
+ tracepoint_synchronize_unregister();
+
+ llist_for_each_entry_safe(data, tmp, llnodes, llist)
+ kfree(data);
+}
+
/* Bulk garbage collection of event_trigger_data elements */
static int trigger_kthread_fn(void *ignore)
{
@@ -56,30 +89,50 @@ void trigger_data_free(struct event_trigger_data *data)
if (data->cmd_ops->set_filter)
data->cmd_ops->set_filter(NULL, data, NULL);
+ /*
+ * Boot-time trigger registration can fail before kthread creation
+ * works. Keep the deferred-free semantics during boot and let late
+ * init start the kthread to drain the list.
+ */
+ if (system_state == SYSTEM_BOOTING && !trigger_kthread) {
+ llist_add(&data->llist, &trigger_data_free_list);
+ return;
+ }
+
if (unlikely(!trigger_kthread)) {
guard(mutex)(&trigger_data_kthread_mutex);
+
+ trigger_create_kthread_locked();
/* Check again after taking mutex */
if (!trigger_kthread) {
- struct task_struct *kthread;
-
- kthread = kthread_create(trigger_kthread_fn, NULL,
- "trigger_data_free");
- if (!IS_ERR(kthread))
- WRITE_ONCE(trigger_kthread, kthread);
+ llist_add(&data->llist, &trigger_data_free_list);
+ /* Drain the queued frees synchronously if creation failed. */
+ trigger_data_free_queued_locked();
+ return;
}
}
- if (!trigger_kthread) {
- /* Do it the slow way */
- tracepoint_synchronize_unregister();
- kfree(data);
- return;
- }
-
llist_add(&data->llist, &trigger_data_free_list);
wake_up_process(trigger_kthread);
}
+static int __init trigger_data_free_init(void)
+{
+ guard(mutex)(&trigger_data_kthread_mutex);
+
+ if (llist_empty(&trigger_data_free_list))
+ return 0;
+
+ trigger_create_kthread_locked();
+ if (trigger_kthread)
+ wake_up_process(trigger_kthread);
+ else
+ trigger_data_free_queued_locked();
+
+ return 0;
+}
+late_initcall(trigger_data_free_init);
+
static inline void data_ops_trigger(struct event_trigger_data *data,
struct trace_buffer *buffer, void *rec,
struct ring_buffer_event *event)
--
2.43.0
^ permalink raw reply related
* Re: [PATCH v4 0/5] mm: zone lock tracepoint instrumentation
From: Andrew Morton @ 2026-03-24 23:39 UTC (permalink / raw)
To: Dmitry Ilvokhin
Cc: Steven Rostedt, Matthew Wilcox, David Hildenbrand,
Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Axel Rasmussen, Yuanchu Xie,
Wei Xu, Masami Hiramatsu, Mathieu Desnoyers, Rafael J. Wysocki,
Pavel Machek, Len Brown, Brendan Jackman, Johannes Weiner, Zi Yan,
Oscar Salvador, Qi Zheng, Shakeel Butt, linux-kernel, linux-mm,
linux-trace-kernel, linux-pm
In-Reply-To: <abv4riGLlubast8v@shell.ilvokhin.com>
On Thu, 19 Mar 2026 13:22:54 +0000 Dmitry Ilvokhin <d@ilvokhin.com> wrote:
> On Mon, Mar 16, 2026 at 05:40:50PM +0000, Dmitry Ilvokhin wrote:
>
> [...]
>
> > A possible generic solution is a trace_contended_release() for spin
> > locks, for example:
> >
> > if (trace_contended_release_enabled() &&
> > atomic_read(&lock->val) & ~_Q_LOCKED_MASK)
> > trace_contended_release(lock);
> >
> > This might work on x86, but could increase code size and regress
> > performance on arches where spin_unlock() is inlined, such as arm64
> > under !PREEMPTION.
>
> I took a stab at this idea and submitted an RFC [1].
>
> The implementation builds on your earlier observation from Matthew that
> _raw_spin_unlock() is not inlined in most configurations. In those
> cases, when the tracepoint is disabled, this adds a single NOP on the
> fast path, with the conditional check staying out of line. The measured
> text size increase in this configuration is +983 bytes.
>
> For configurations where _raw_spin_unlock() is inlined, the
> instrumentation does increase code size more noticeably
> (+71 KB in my measurements), since the check and out of line call is
> replicated at each call site.
>
> This provides a generic release-side signal for contended locks,
> allowing: correlation of lock holders with waiters and measurement of
> contended hold times
>
> This RFC addressing the same visibility gap without introducing per-lock
> instrumentation.
>
> If this tradeoff is acceptable, this could be a generic alternative to
> lock-specific tracepoints.
>
> [1]: https://lore.kernel.org/all/51aad0415b78c5a39f2029722118fa01eac77538.1773858853.git.d@ilvokhin.com
That submission has met a disappointing response.
How should I proceed with this series "mm: zone lock tracepoint
instrumentation"? It's not urgent so I'm inclined to put this on hold
while you pursue "locking: Add contended_release tracepoint to spinning
locks"?
Please send that v2 sometime and hopefully Steven can help push it along?
^ permalink raw reply
* Re: [PATCH v12 0/4] ring-buffer: Making persistent ring buffers robust
From: Masami Hiramatsu @ 2026-03-24 23:56 UTC (permalink / raw)
To: Steven Rostedt
Cc: Mathieu Desnoyers, linux-kernel, linux-trace-kernel, Ian Rogers
In-Reply-To: <20260324172557.04cfc2a9@gandalf.local.home>
On Tue, 24 Mar 2026 17:25:57 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:
> On Tue, 24 Mar 2026 09:00:03 +0900
> "Masami Hiramatsu (Google)" <mhiramat@kernel.org> wrote:
>
> > Hi,
> >
> > Here is the 12th version of improvement patches for making persistent
> > ring buffers robust to failures.
> > The previous version is here:
> >
> > https://lore.kernel.org/all/177391152793.193994.8986943289250629418.stgit@mhiramat.tok.corp.google.com/
> >
> >
> > In this version, I rebased it on tracing/fixes branch (thus
> > the first fix patch has been removed) and fixed
> > a build error which came from a copy-paste mistake.
> >
>
> Hi Masami,
>
> Can you rebase this on the ring-buffer/for-next branch?
>
> It has the remote updates for pkvm tracing, and there's some conflicts.
> That branch is also used by the ARM tree so it can't be rebased.
OK, let me rebase the series on it.
Thanks!
>
> Thanks,
>
> -- Steve
>
--
Masami Hiramatsu (Google) <mhiramat@kernel.org>
^ permalink raw reply
* Re: [PATCH v12 0/4] ring-buffer: Making persistent ring buffers robust
From: Masami Hiramatsu @ 2026-03-25 0:13 UTC (permalink / raw)
To: Masami Hiramatsu
Cc: Steven Rostedt, Mathieu Desnoyers, linux-kernel,
linux-trace-kernel, Ian Rogers
In-Reply-To: <20260325085644.4310a1295b0b962ed87b5b60@kernel.org>
On Wed, 25 Mar 2026 08:56:44 +0900
Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote:
> On Tue, 24 Mar 2026 17:25:57 -0400
> Steven Rostedt <rostedt@goodmis.org> wrote:
>
> > On Tue, 24 Mar 2026 09:00:03 +0900
> > "Masami Hiramatsu (Google)" <mhiramat@kernel.org> wrote:
> >
> > > Hi,
> > >
> > > Here is the 12th version of improvement patches for making persistent
> > > ring buffers robust to failures.
> > > The previous version is here:
> > >
> > > https://lore.kernel.org/all/177391152793.193994.8986943289250629418.stgit@mhiramat.tok.corp.google.com/
> > >
> > >
> > > In this version, I rebased it on tracing/fixes branch (thus
> > > the first fix patch has been removed) and fixed
> > > a build error which came from a copy-paste mistake.
> > >
> >
> > Hi Masami,
> >
> > Can you rebase this on the ring-buffer/for-next branch?
> >
> > It has the remote updates for pkvm tracing, and there's some conflicts.
> > That branch is also used by the ARM tree so it can't be rebased.
>
> OK, let me rebase the series on it.
I found the branch does not have a fix [1] and it makes a conflict with
[2/4]. Can you update it to merge the trace/fixes branch?
[1] f35dbac69421 ("ring-buffer: Fix to update per-subbuf entries of persistent ring buffer")
I can send a series on ring-buffer/for-next but I think it may
cause the same conflict on git-pull later.
Thanks,
>
> Thanks!
>
> >
> > Thanks,
> >
> > -- Steve
> >
>
>
> --
> Masami Hiramatsu (Google) <mhiramat@kernel.org>
--
Masami Hiramatsu (Google) <mhiramat@kernel.org>
^ permalink raw reply
* Re: [PATCH] ring-buffer: Show what clock function is used on timestamp errors
From: Masami Hiramatsu @ 2026-03-25 0:17 UTC (permalink / raw)
To: Steven Rostedt
Cc: LKML, Linux Trace Kernel, Masami Hiramatsu, Mathieu Desnoyers
In-Reply-To: <20260323202212.479bb288@gandalf.local.home>
On Mon, 23 Mar 2026 20:22:12 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:
> From: Steven Rostedt <rostedt@goodmis.org>
>
> The testing for tracing was triggering a timestamp count issue that was
> always off by one. This has been happening for some time but has never
> been reported by anyone else. It was finally discovered to be an issue
> with the "uptime" (jiffies) clock that happened to be traced and the
> internal recursion caused the discrepancy. This would have been much
> easier to solve if the clock function being used was displayed when the
> error was detected.
>
> Add the clock function to the error output.
Looks good to me.
Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
>
> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
> ---
> kernel/trace/ring_buffer.c | 10 ++++++----
> 1 file changed, 6 insertions(+), 4 deletions(-)
>
> diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c
> index 170170bd83bd..a99d1c7d180b 100644
> --- a/kernel/trace/ring_buffer.c
> +++ b/kernel/trace/ring_buffer.c
> @@ -4435,18 +4435,20 @@ static void check_buffer(struct ring_buffer_per_cpu *cpu_buffer,
> ret = rb_read_data_buffer(bpage, tail, cpu_buffer->cpu, &ts, &delta);
> if (ret < 0) {
> if (delta < ts) {
> - buffer_warn_return("[CPU: %d]ABSOLUTE TIME WENT BACKWARDS: last ts: %lld absolute ts: %lld\n",
> - cpu_buffer->cpu, ts, delta);
> + buffer_warn_return("[CPU: %d]ABSOLUTE TIME WENT BACKWARDS: last ts: %lld absolute ts: %lld clock:%pS\n",
> + cpu_buffer->cpu, ts, delta,
> + cpu_buffer->buffer->clock);
> goto out;
> }
> }
> if ((full && ts > info->ts) ||
> (!full && ts + info->delta != info->ts)) {
> - buffer_warn_return("[CPU: %d]TIME DOES NOT MATCH expected:%lld actual:%lld delta:%lld before:%lld after:%lld%s context:%s\n",
> + buffer_warn_return("[CPU: %d]TIME DOES NOT MATCH expected:%lld actual:%lld delta:%lld before:%lld after:%lld%s context:%s\ntrace clock:%pS",
> cpu_buffer->cpu,
> ts + info->delta, info->ts, info->delta,
> info->before, info->after,
> - full ? " (full)" : "", show_interrupt_level());
> + full ? " (full)" : "", show_interrupt_level(),
> + cpu_buffer->buffer->clock);
> }
> out:
> atomic_dec(this_cpu_ptr(&checking));
> --
> 2.51.0
>
--
Masami Hiramatsu (Google) <mhiramat@kernel.org>
^ permalink raw reply
* Re: [PATCH v12 0/4] ring-buffer: Making persistent ring buffers robust
From: Steven Rostedt @ 2026-03-25 0:39 UTC (permalink / raw)
To: Masami Hiramatsu (Google)
Cc: Mathieu Desnoyers, linux-kernel, linux-trace-kernel, Ian Rogers
In-Reply-To: <20260325091304.c94f4e9505660202f96006a1@kernel.org>
On Wed, 25 Mar 2026 09:13:04 +0900
Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote:
> I found the branch does not have a fix [1] and it makes a conflict with
> [2/4]. Can you update it to merge the trace/fixes branch?
>
> [1] f35dbac69421 ("ring-buffer: Fix to update per-subbuf entries of persistent ring buffer")
>
> I can send a series on ring-buffer/for-next but I think it may
> cause the same conflict on git-pull later.
Ah crap. OK, let me merge in that commit to the for-next branch, and then see if it works.
Thanks for letting me know.
-- Steve
^ permalink raw reply
* Re: [PATCH v12 0/4] ring-buffer: Making persistent ring buffers robust
From: Steven Rostedt @ 2026-03-25 0:47 UTC (permalink / raw)
To: Masami Hiramatsu (Google)
Cc: Mathieu Desnoyers, linux-kernel, linux-trace-kernel, Ian Rogers
In-Reply-To: <20260324203941.2bd628d9@gandalf.local.home>
On Tue, 24 Mar 2026 20:39:41 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:
> On Wed, 25 Mar 2026 09:13:04 +0900
> Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote:
>
> > I found the branch does not have a fix [1] and it makes a conflict with
> > [2/4]. Can you update it to merge the trace/fixes branch?
> >
> > [1] f35dbac69421 ("ring-buffer: Fix to update per-subbuf entries of persistent ring buffer")
> >
> > I can send a series on ring-buffer/for-next but I think it may
> > cause the same conflict on git-pull later.
>
> Ah crap. OK, let me merge in that commit to the for-next branch, and then see if it works.
>
> Thanks for letting me know.
OK, refresh the ring-buffer/for-next branch and it has the fix merged.
There's still a small conflict (with a header file include). Could you
rebase on that?
Thanks,
-- Steve
^ permalink raw reply
* [PATCH v5 0/3] PCI Controller event and LTSSM tracepoint support
From: Shawn Lin @ 2026-03-25 1:58 UTC (permalink / raw)
To: Manivannan Sadhasivam, Bjorn Helgaas
Cc: linux-rockchip, linux-pci, linux-trace-kernel, linux-doc,
Steven Rostedt, Shawn Lin
This patch-set adds new pci controller event and LTSSM tracepoint used by host drivers
which provide LTSSM trace functionality. The first user is pcie-dw-rockchip with a 256
Bytes FIFO for recording LTSSM transition.
Testing
=========
This series was tested on RK3588/RK3588s EVB1 with NVMe SSD connected to PCIe3 and PCIe2
root ports.
echo 1 > /sys/kernel/debug/tracing/events/pci_controller/pcie_ltssm_state_transition/enable
cat /sys/kernel/debug/tracing/trace_pipe
# tracer: nop
#
# entries-in-buffer/entries-written: 64/64 #P:8
#
# _-----=> irqs-off/BH-disabled
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / _-=> migrate-disable
# |||| / delay
# TASK-PID CPU# ||||| TIMESTAMP FUNCTION
# | | | ||||| | |
kworker/0:0-9 [000] ..... 5.600194: pcie_ltssm_state_transition: dev: a40000000.pcie state: DETECT_ACT rate: Unknown
kworker/0:0-9 [000] ..... 5.600198: pcie_ltssm_state_transition: dev: a40000000.pcie state: DETECT_WAIT rate: Unknown
kworker/0:0-9 [000] ..... 5.600199: pcie_ltssm_state_transition: dev: a40000000.pcie state: DETECT_ACT rate: Unknown
kworker/0:0-9 [000] ..... 5.600201: pcie_ltssm_state_transition: dev: a40000000.pcie state: POLL_ACTIVE rate: Unknown
kworker/0:0-9 [000] ..... 5.600202: pcie_ltssm_state_transition: dev: a40000000.pcie state: POLL_CONFIG rate: Unknown
kworker/0:0-9 [000] ..... 5.600204: pcie_ltssm_state_transition: dev: a40000000.pcie state: CFG_LINKWD_START rate: Unknown
kworker/0:0-9 [000] ..... 5.600206: pcie_ltssm_state_transition: dev: a40000000.pcie state: CFG_LINKWD_ACEPT rate: Unknown
kworker/0:0-9 [000] ..... 5.600207: pcie_ltssm_state_transition: dev: a40000000.pcie state: CFG_LANENUM_WAI rate: Unknown
kworker/0:0-9 [000] ..... 5.600208: pcie_ltssm_state_transition: dev: a40000000.pcie state: CFG_LANENUM_ACEPT rate: Unknown
kworker/0:0-9 [000] ..... 5.600210: pcie_ltssm_state_transition: dev: a40000000.pcie state: CFG_COMPLETE rate: Unknown
kworker/0:0-9 [000] ..... 5.600212: pcie_ltssm_state_transition: dev: a40000000.pcie state: CFG_IDLE rate: Unknown
kworker/0:0-9 [000] ..... 5.600213: pcie_ltssm_state_transition: dev: a40000000.pcie state: L0 rate: 2.5 GT/s
kworker/0:0-9 [000] ..... 5.600214: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_LOCK rate: Unknown
kworker/0:0-9 [000] ..... 5.600216: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_RCVRCFG rate: Unknown
kworker/0:0-9 [000] ..... 5.600217: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_SPEED rate: Unknown
kworker/0:0-9 [000] ..... 5.600218: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_LOCK rate: Unknown
kworker/0:0-9 [000] ..... 5.600220: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_EQ1 rate: Unknown
kworker/0:0-9 [000] ..... 5.600221: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_EQ2 rate: 8.0 GT/s
kworker/0:0-9 [000] ..... 5.600222: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_EQ3 rate: 8.0 GT/s
kworker/0:0-9 [000] ..... 5.600224: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_LOCK rate: 8.0 GT/s
kworker/0:0-9 [000] ..... 5.600225: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_RCVRCFG rate: 8.0 GT/s
kworker/0:0-9 [000] ..... 5.600226: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_IDLE rate: 8.0 GT/s
kworker/0:0-9 [000] ..... 5.600227: pcie_ltssm_state_transition: dev: a40000000.pcie state: L0 rate: 8.0 GT/s
kworker/0:0-9 [000] ..... 5.600228: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_LOCK rate: 8.0 GT/s
kworker/0:0-9 [000] ..... 5.600229: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_RCVRCFG rate: 8.0 GT/s
kworker/0:0-9 [000] ..... 5.600231: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_IDLE rate: 8.0 GT/s
kworker/0:0-9 [000] ..... 5.600232: pcie_ltssm_state_transition: dev: a40000000.pcie state: L0 rate: 8.0 GT/s
kworker/0:0-9 [000] ..... 5.600233: pcie_ltssm_state_transition: dev: a40000000.pcie state: L123_SEND_EIDLE rate: 8.0 GT/s
kworker/0:0-9 [000] ..... 5.600234: pcie_ltssm_state_transition: dev: a40000000.pcie state: L1_IDLE rate: 8.0 GT/s
kworker/0:0-9 [000] ..... 5.600236: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_LOCK rate: 8.0 GT/s
kworker/0:0-9 [000] ..... 5.600237: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_RCVRCFG rate: 8.0 GT/s
kworker/0:0-9 [000] ..... 5.600238: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_IDLE rate: 8.0 GT/s
kworker/0:0-9 [000] ..... 5.600239: pcie_ltssm_state_transition: dev: a40000000.pcie state: L0 rate: 8.0 GT/s
Changes in v5:
- rebase
- use EM/EMe instead
- remove reg/unreg function and back to use TRACE_EVENT
- use trace_pcie_ltssm_state_transition_enabled()
Changes in v4:
- use TRACE_EVENT_FN to notify when to start and stop the tracepoint,
and export pci_ltssm_tp_enabled() for host drivers to use
- skip trace if pci_ltssm_tp_enabled() is false.(Steven)
- wrap into 80 columns(Bjorn)
Changes in v3:
- add TRACE_DEFINE_ENUM for all enums(Steven Rostedt)
- Add toctree entry in Documentation/trace/index.rst(Bagas Sanjaya)
- fix mismatch section underline length(Bagas Sanjaya)
- Make example snippets in code block(Bagas Sanjaya)
- warp context into 80 columns and fix the file name(Bjorn)
- reorder variables(Mani)
- rename loop to i; rename en to enable(Mani)
- use FIELD_GET(Mani)
- add comment about how the FIFO works(Mani)
Changes in v2:
- use tracepoint
Shawn Lin (3):
PCI: trace: Add PCI controller LTSSM transition tracepoint
Documentation: tracing: Add PCI controller event documentation
PCI: dw-rockchip: Add pcie_ltssm_state_transition trace support
Documentation/trace/events-pci-controller.rst | 42 ++++++++++
Documentation/trace/index.rst | 1 +
drivers/pci/controller/dwc/pcie-dw-rockchip.c | 111 ++++++++++++++++++++++++++
drivers/pci/trace.c | 1 +
include/trace/events/pci_controller.h | 58 ++++++++++++++
5 files changed, 213 insertions(+)
create mode 100644 Documentation/trace/events-pci-controller.rst
create mode 100644 include/trace/events/pci_controller.h
--
2.7.4
^ permalink raw reply
* [PATCH v5 1/3] PCI: trace: Add PCI controller LTSSM transition tracepoint
From: Shawn Lin @ 2026-03-25 1:58 UTC (permalink / raw)
To: Manivannan Sadhasivam, Bjorn Helgaas
Cc: linux-rockchip, linux-pci, linux-trace-kernel, linux-doc,
Steven Rostedt, Shawn Lin
In-Reply-To: <1774403912-210670-1-git-send-email-shawn.lin@rock-chips.com>
Some platforms may provide LTSSM trace functionality, recording historical
LTSSM state transition information. This is very useful for debugging, such
as when certain devices cannot be recognized or link broken during test.
Implement the pci controller tracepoint for recording LTSSM and rate.
Signed-off-by: Shawn Lin <shawn.lin@rock-chips.com>
---
Changes in v5:
- use EM/EMe instead
- remove reg/unreg function and back to use TRACE_EVENT
Changes in v4:
- use TRACE_EVENT_FN to notify when to start and stop the tracepoint,
and export pci_ltssm_tp_enabled() for host drivers to use
Changes in v3:
- add TRACE_DEFINE_ENUM for all enums(Steven Rostedt)
Changes in v2: None
drivers/pci/trace.c | 1 +
include/trace/events/pci_controller.h | 58 +++++++++++++++++++++++++++++++++++
2 files changed, 59 insertions(+)
create mode 100644 include/trace/events/pci_controller.h
diff --git a/drivers/pci/trace.c b/drivers/pci/trace.c
index cf11abc..c1da9d3 100644
--- a/drivers/pci/trace.c
+++ b/drivers/pci/trace.c
@@ -9,3 +9,4 @@
#define CREATE_TRACE_POINTS
#include <trace/events/pci.h>
+#include <trace/events/pci_controller.h>
diff --git a/include/trace/events/pci_controller.h b/include/trace/events/pci_controller.h
new file mode 100644
index 0000000..a4b387c
--- /dev/null
+++ b/include/trace/events/pci_controller.h
@@ -0,0 +1,58 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM pci_controller
+
+#if !defined(_TRACE_HW_EVENT_PCI_CONTROLLER_H) || defined(TRACE_HEADER_MULTI_READ)
+#define _TRACE_HW_EVENT_PCI_CONTROLLER_H
+
+#include <uapi/linux/pci_regs.h>
+#include <linux/tracepoint.h>
+
+#define RATE \
+ EM(PCIE_SPEED_2_5GT, "2.5 GT/s") \
+ EM(PCIE_SPEED_5_0GT, "5.0 GT/s") \
+ EM(PCIE_SPEED_8_0GT, "8.0 GT/s") \
+ EM(PCIE_SPEED_16_0GT, "16.0 GT/s") \
+ EM(PCIE_SPEED_32_0GT, "32.0 GT/s") \
+ EM(PCIE_SPEED_64_0GT, "64.0 GT/s") \
+ EMe(PCI_SPEED_UNKNOWN, "Unknown")
+
+
+#undef EM
+#undef EMe
+#define EM(a, b) TRACE_DEFINE_ENUM(a);
+#define EMe(a, b) TRACE_DEFINE_ENUM(a);
+
+RATE
+
+#undef EM
+#undef EMe
+#define EM(a, b) {a, b},
+#define EMe(a, b) {a, b}
+
+TRACE_EVENT(pcie_ltssm_state_transition,
+ TP_PROTO(const char *dev_name, const char *state, u32 rate),
+ TP_ARGS(dev_name, state, rate),
+
+ TP_STRUCT__entry(
+ __string(dev_name, dev_name)
+ __string(state, state)
+ __field(u32, rate)
+ ),
+
+ TP_fast_assign(
+ __assign_str(dev_name);
+ __assign_str(state);
+ __entry->rate = rate;
+ ),
+
+ TP_printk("dev: %s state: %s rate: %s",
+ __get_str(dev_name), __get_str(state),
+ __print_symbolic(__entry->rate, RATE)
+ )
+);
+
+#endif /* _TRACE_HW_EVENT_PCI_CONTROLLER_H */
+
+/* This part must be outside protection */
+#include <trace/define_trace.h>
--
2.7.4
^ permalink raw reply related
* [PATCH v5 2/3] Documentation: tracing: Add PCI controller event documentation
From: Shawn Lin @ 2026-03-25 1:58 UTC (permalink / raw)
To: Manivannan Sadhasivam, Bjorn Helgaas
Cc: linux-rockchip, linux-pci, linux-trace-kernel, linux-doc,
Steven Rostedt, Shawn Lin
In-Reply-To: <1774403912-210670-1-git-send-email-shawn.lin@rock-chips.com>
The available tracepoint, pcie_ltssm_state_transition, monitors the LTSSM
state transition for debugging purpose. Add description about it.
Signed-off-by: Shawn Lin <shawn.lin@rock-chips.com>
---
Changes in v5: None
Changes in v4: None
Changes in v3:
- Add toctree entry in Documentation/trace/index.rst(Bagas Sanjaya)
- fix mismatch section underline length(Bagas Sanjaya)
- Make example snippets in code block(Bagas Sanjaya)
- warp context into 80 columns and fix the file name(Bjorn)
Changes in v2: None
Documentation/trace/events-pci-controller.rst | 42 +++++++++++++++++++++++++++
Documentation/trace/index.rst | 1 +
2 files changed, 43 insertions(+)
create mode 100644 Documentation/trace/events-pci-controller.rst
diff --git a/Documentation/trace/events-pci-controller.rst b/Documentation/trace/events-pci-controller.rst
new file mode 100644
index 0000000..cb9f715
--- /dev/null
+++ b/Documentation/trace/events-pci-controller.rst
@@ -0,0 +1,42 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+======================================
+Subsystem Trace Points: PCI Controller
+======================================
+
+Overview
+========
+The PCI controller tracing system provides tracepoints to monitor controller
+level information for debugging purpose. The events normally show up here:
+
+ /sys/kernel/tracing/events/pci_controller
+
+Cf. include/trace/events/pci_controller.h for the events definitions.
+
+Available Tracepoints
+=====================
+
+pcie_ltssm_state_transition
+---------------------------
+
+Monitors PCIe LTSSM state transition including state and rate information
+::
+
+ pcie_ltssm_state_transition "dev: %s state: %s rate: %s\n"
+
+**Parameters**:
+
+* ``dev`` - PCIe controller instance
+* ``state`` - PCIe LTSSM state
+* ``rate`` - PCIe date rate
+
+**Example Usage**:
+
+.. code-block:: shell
+
+ # Enable the tracepoint
+ echo 1 > /sys/kernel/debug/tracing/events/pci_controller/pcie_ltssm_state_transition/enable
+
+ # Monitor events (the following output is generated when a device is linking)
+ cat /sys/kernel/debug/tracing/trace_pipe
+ kworker/0:0-9 [000] ..... 5.600221: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_EQ2 rate: 8.0 GT/s
diff --git a/Documentation/trace/index.rst b/Documentation/trace/index.rst
index 036db96..5d9bf469 100644
--- a/Documentation/trace/index.rst
+++ b/Documentation/trace/index.rst
@@ -55,6 +55,7 @@ applications.
events-nmi
events-msr
events-pci
+ events-pci-controller
boottime-trace
histogram
histogram-design
--
2.7.4
^ permalink raw reply related
* [PATCH v13 0/4] ring-buffer: Making persistent ring buffers robust
From: Masami Hiramatsu (Google) @ 2026-03-25 2:24 UTC (permalink / raw)
To: Steven Rostedt
Cc: Masami Hiramatsu, Mathieu Desnoyers, linux-kernel,
linux-trace-kernel, Ian Rogers
Hi,
Here is the 13th version of improvement patches for making persistent
ring buffers robust to failures.
The previous version is here:
https://lore.kernel.org/all/177431040276.3708637.10277850418341653677.stgit@mhiramat.tok.corp.google.com/
In this version, I rebased it on ring-buffer/for-next.
Thank you,
---
Masami Hiramatsu (Google) (4):
ring-buffer: Flush and stop persistent ring buffer on panic
ring-buffer: Skip invalid sub-buffers when validating persistent ring buffer
ring-buffer: Skip invalid sub-buffers when rewinding persistent ring buffer
ring-buffer: Add persistent ring buffer selftest
arch/alpha/include/asm/Kbuild | 1
arch/arc/include/asm/Kbuild | 1
arch/arm/include/asm/Kbuild | 1
arch/arm64/include/asm/ring_buffer.h | 10 +
arch/csky/include/asm/Kbuild | 1
arch/hexagon/include/asm/Kbuild | 1
arch/loongarch/include/asm/Kbuild | 1
arch/m68k/include/asm/Kbuild | 1
arch/microblaze/include/asm/Kbuild | 1
arch/mips/include/asm/Kbuild | 1
arch/nios2/include/asm/Kbuild | 1
arch/openrisc/include/asm/Kbuild | 1
arch/parisc/include/asm/Kbuild | 1
arch/powerpc/include/asm/Kbuild | 1
arch/riscv/include/asm/Kbuild | 1
arch/s390/include/asm/Kbuild | 1
arch/sh/include/asm/Kbuild | 1
arch/sparc/include/asm/Kbuild | 1
arch/um/include/asm/Kbuild | 1
arch/x86/include/asm/Kbuild | 1
arch/xtensa/include/asm/Kbuild | 1
include/asm-generic/ring_buffer.h | 13 ++
include/linux/ring_buffer.h | 1
kernel/trace/Kconfig | 15 ++
kernel/trace/ring_buffer.c | 237 ++++++++++++++++++++++++++--------
kernel/trace/trace.c | 4 +
26 files changed, 241 insertions(+), 59 deletions(-)
create mode 100644 arch/arm64/include/asm/ring_buffer.h
create mode 100644 include/asm-generic/ring_buffer.h
--
Masami Hiramatsu (Google) <mhiramat@kernel.org>
^ permalink raw reply
* [PATCH v13 1/4] ring-buffer: Flush and stop persistent ring buffer on panic
From: Masami Hiramatsu (Google) @ 2026-03-25 2:25 UTC (permalink / raw)
To: Steven Rostedt
Cc: Masami Hiramatsu, Mathieu Desnoyers, linux-kernel,
linux-trace-kernel, Ian Rogers
In-Reply-To: <177440549083.1529621.15486836623498328967.stgit@mhiramat.tok.corp.google.com>
From: Masami Hiramatsu (Google) <mhiramat@kernel.org>
On real hardware, panic and machine reboot may not flush hardware cache
to memory. This means the persistent ring buffer, which relies on a
coherent state of memory, may not have its events written to the buffer
and they may be lost. Moreover, there may be inconsistency with the
counters which are used for validation of the integrity of the
persistent ring buffer which may cause all data to be discarded.
To avoid this issue, stop recording of the ring buffer on panic and
flush the cache of the ring buffer's memory.
Fixes: e645535a954a ("tracing: Add option to use memmapped memory for trace boot instance")
Cc: stable@vger.kernel.org
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
---
Changes in v13:
- Fix a rebase conflict.
Changes in v11:
- Do nothing by default since flush_cache_vmap() does nothing on x86
but it can cause deadlock on some architectures via on_each_cpu()
because other CPUs will be stoppped when panic notifier is called.
Changes in v9:
- Fix typo of & to &&.
- Fix typo of "Generic"
Changes in v6:
- Introduce asm/ring_buffer.h for arch_ring_buffer_flush_range().
- Use flush_cache_vmap() instead of flush_cache_all().
Changes in v5:
- Use ring_buffer_record_off() instead of ring_buffer_record_disable().
- Use flush_cache_all() to ensure flush all cache.
Changes in v3:
- update patch description.
---
arch/alpha/include/asm/Kbuild | 1 +
arch/arc/include/asm/Kbuild | 1 +
arch/arm/include/asm/Kbuild | 1 +
arch/arm64/include/asm/ring_buffer.h | 10 ++++++++++
arch/csky/include/asm/Kbuild | 1 +
arch/hexagon/include/asm/Kbuild | 1 +
arch/loongarch/include/asm/Kbuild | 1 +
arch/m68k/include/asm/Kbuild | 1 +
arch/microblaze/include/asm/Kbuild | 1 +
arch/mips/include/asm/Kbuild | 1 +
arch/nios2/include/asm/Kbuild | 1 +
arch/openrisc/include/asm/Kbuild | 1 +
arch/parisc/include/asm/Kbuild | 1 +
arch/powerpc/include/asm/Kbuild | 1 +
arch/riscv/include/asm/Kbuild | 1 +
arch/s390/include/asm/Kbuild | 1 +
arch/sh/include/asm/Kbuild | 1 +
arch/sparc/include/asm/Kbuild | 1 +
arch/um/include/asm/Kbuild | 1 +
arch/x86/include/asm/Kbuild | 1 +
arch/xtensa/include/asm/Kbuild | 1 +
include/asm-generic/ring_buffer.h | 13 +++++++++++++
kernel/trace/ring_buffer.c | 22 ++++++++++++++++++++++
23 files changed, 65 insertions(+)
create mode 100644 arch/arm64/include/asm/ring_buffer.h
create mode 100644 include/asm-generic/ring_buffer.h
diff --git a/arch/alpha/include/asm/Kbuild b/arch/alpha/include/asm/Kbuild
index 483965c5a4de..b154b4e3dfa8 100644
--- a/arch/alpha/include/asm/Kbuild
+++ b/arch/alpha/include/asm/Kbuild
@@ -5,4 +5,5 @@ generic-y += agp.h
generic-y += asm-offsets.h
generic-y += kvm_para.h
generic-y += mcs_spinlock.h
+generic-y += ring_buffer.h
generic-y += text-patching.h
diff --git a/arch/arc/include/asm/Kbuild b/arch/arc/include/asm/Kbuild
index 4c69522e0328..483caacc6988 100644
--- a/arch/arc/include/asm/Kbuild
+++ b/arch/arc/include/asm/Kbuild
@@ -5,5 +5,6 @@ generic-y += extable.h
generic-y += kvm_para.h
generic-y += mcs_spinlock.h
generic-y += parport.h
+generic-y += ring_buffer.h
generic-y += user.h
generic-y += text-patching.h
diff --git a/arch/arm/include/asm/Kbuild b/arch/arm/include/asm/Kbuild
index 03657ff8fbe3..decad5f2c826 100644
--- a/arch/arm/include/asm/Kbuild
+++ b/arch/arm/include/asm/Kbuild
@@ -3,6 +3,7 @@ generic-y += early_ioremap.h
generic-y += extable.h
generic-y += flat.h
generic-y += parport.h
+generic-y += ring_buffer.h
generated-y += mach-types.h
generated-y += unistd-nr.h
diff --git a/arch/arm64/include/asm/ring_buffer.h b/arch/arm64/include/asm/ring_buffer.h
new file mode 100644
index 000000000000..62316c406888
--- /dev/null
+++ b/arch/arm64/include/asm/ring_buffer.h
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef _ASM_ARM64_RING_BUFFER_H
+#define _ASM_ARM64_RING_BUFFER_H
+
+#include <asm/cacheflush.h>
+
+/* Flush D-cache on persistent ring buffer */
+#define arch_ring_buffer_flush_range(start, end) dcache_clean_pop(start, end)
+
+#endif /* _ASM_ARM64_RING_BUFFER_H */
diff --git a/arch/csky/include/asm/Kbuild b/arch/csky/include/asm/Kbuild
index 3a5c7f6e5aac..7dca0c6cdc84 100644
--- a/arch/csky/include/asm/Kbuild
+++ b/arch/csky/include/asm/Kbuild
@@ -9,6 +9,7 @@ generic-y += qrwlock.h
generic-y += qrwlock_types.h
generic-y += qspinlock.h
generic-y += parport.h
+generic-y += ring_buffer.h
generic-y += user.h
generic-y += vmlinux.lds.h
generic-y += text-patching.h
diff --git a/arch/hexagon/include/asm/Kbuild b/arch/hexagon/include/asm/Kbuild
index 1efa1e993d4b..0f887d4238ed 100644
--- a/arch/hexagon/include/asm/Kbuild
+++ b/arch/hexagon/include/asm/Kbuild
@@ -5,4 +5,5 @@ generic-y += extable.h
generic-y += iomap.h
generic-y += kvm_para.h
generic-y += mcs_spinlock.h
+generic-y += ring_buffer.h
generic-y += text-patching.h
diff --git a/arch/loongarch/include/asm/Kbuild b/arch/loongarch/include/asm/Kbuild
index 9034b583a88a..7e92957baf6a 100644
--- a/arch/loongarch/include/asm/Kbuild
+++ b/arch/loongarch/include/asm/Kbuild
@@ -10,5 +10,6 @@ generic-y += qrwlock.h
generic-y += user.h
generic-y += ioctl.h
generic-y += mmzone.h
+generic-y += ring_buffer.h
generic-y += statfs.h
generic-y += text-patching.h
diff --git a/arch/m68k/include/asm/Kbuild b/arch/m68k/include/asm/Kbuild
index b282e0dd8dc1..62543bf305ff 100644
--- a/arch/m68k/include/asm/Kbuild
+++ b/arch/m68k/include/asm/Kbuild
@@ -3,5 +3,6 @@ generated-y += syscall_table.h
generic-y += extable.h
generic-y += kvm_para.h
generic-y += mcs_spinlock.h
+generic-y += ring_buffer.h
generic-y += spinlock.h
generic-y += text-patching.h
diff --git a/arch/microblaze/include/asm/Kbuild b/arch/microblaze/include/asm/Kbuild
index 7178f990e8b3..0030309b47ad 100644
--- a/arch/microblaze/include/asm/Kbuild
+++ b/arch/microblaze/include/asm/Kbuild
@@ -5,6 +5,7 @@ generic-y += extable.h
generic-y += kvm_para.h
generic-y += mcs_spinlock.h
generic-y += parport.h
+generic-y += ring_buffer.h
generic-y += syscalls.h
generic-y += tlb.h
generic-y += user.h
diff --git a/arch/mips/include/asm/Kbuild b/arch/mips/include/asm/Kbuild
index 684569b2ecd6..9771c3d85074 100644
--- a/arch/mips/include/asm/Kbuild
+++ b/arch/mips/include/asm/Kbuild
@@ -12,5 +12,6 @@ generic-y += mcs_spinlock.h
generic-y += parport.h
generic-y += qrwlock.h
generic-y += qspinlock.h
+generic-y += ring_buffer.h
generic-y += user.h
generic-y += text-patching.h
diff --git a/arch/nios2/include/asm/Kbuild b/arch/nios2/include/asm/Kbuild
index 28004301c236..0a2530964413 100644
--- a/arch/nios2/include/asm/Kbuild
+++ b/arch/nios2/include/asm/Kbuild
@@ -5,6 +5,7 @@ generic-y += cmpxchg.h
generic-y += extable.h
generic-y += kvm_para.h
generic-y += mcs_spinlock.h
+generic-y += ring_buffer.h
generic-y += spinlock.h
generic-y += user.h
generic-y += text-patching.h
diff --git a/arch/openrisc/include/asm/Kbuild b/arch/openrisc/include/asm/Kbuild
index cef49d60d74c..8aa34621702d 100644
--- a/arch/openrisc/include/asm/Kbuild
+++ b/arch/openrisc/include/asm/Kbuild
@@ -8,4 +8,5 @@ generic-y += spinlock_types.h
generic-y += spinlock.h
generic-y += qrwlock_types.h
generic-y += qrwlock.h
+generic-y += ring_buffer.h
generic-y += user.h
diff --git a/arch/parisc/include/asm/Kbuild b/arch/parisc/include/asm/Kbuild
index 4fb596d94c89..d48d158f7241 100644
--- a/arch/parisc/include/asm/Kbuild
+++ b/arch/parisc/include/asm/Kbuild
@@ -4,4 +4,5 @@ generated-y += syscall_table_64.h
generic-y += agp.h
generic-y += kvm_para.h
generic-y += mcs_spinlock.h
+generic-y += ring_buffer.h
generic-y += user.h
diff --git a/arch/powerpc/include/asm/Kbuild b/arch/powerpc/include/asm/Kbuild
index 2e23533b67e3..805b5aeebb6f 100644
--- a/arch/powerpc/include/asm/Kbuild
+++ b/arch/powerpc/include/asm/Kbuild
@@ -5,4 +5,5 @@ generated-y += syscall_table_spu.h
generic-y += agp.h
generic-y += mcs_spinlock.h
generic-y += qrwlock.h
+generic-y += ring_buffer.h
generic-y += early_ioremap.h
diff --git a/arch/riscv/include/asm/Kbuild b/arch/riscv/include/asm/Kbuild
index bd5fc9403295..7721b63642f4 100644
--- a/arch/riscv/include/asm/Kbuild
+++ b/arch/riscv/include/asm/Kbuild
@@ -14,5 +14,6 @@ generic-y += ticket_spinlock.h
generic-y += qrwlock.h
generic-y += qrwlock_types.h
generic-y += qspinlock.h
+generic-y += ring_buffer.h
generic-y += user.h
generic-y += vmlinux.lds.h
diff --git a/arch/s390/include/asm/Kbuild b/arch/s390/include/asm/Kbuild
index 80bad7de7a04..0c1fc47c3ba0 100644
--- a/arch/s390/include/asm/Kbuild
+++ b/arch/s390/include/asm/Kbuild
@@ -7,3 +7,4 @@ generated-y += unistd_nr.h
generic-y += asm-offsets.h
generic-y += mcs_spinlock.h
generic-y += mmzone.h
+generic-y += ring_buffer.h
diff --git a/arch/sh/include/asm/Kbuild b/arch/sh/include/asm/Kbuild
index 4d3f10ed8275..f0403d3ee8ab 100644
--- a/arch/sh/include/asm/Kbuild
+++ b/arch/sh/include/asm/Kbuild
@@ -3,4 +3,5 @@ generated-y += syscall_table.h
generic-y += kvm_para.h
generic-y += mcs_spinlock.h
generic-y += parport.h
+generic-y += ring_buffer.h
generic-y += text-patching.h
diff --git a/arch/sparc/include/asm/Kbuild b/arch/sparc/include/asm/Kbuild
index 17ee8a273aa6..49c6bb326b75 100644
--- a/arch/sparc/include/asm/Kbuild
+++ b/arch/sparc/include/asm/Kbuild
@@ -4,4 +4,5 @@ generated-y += syscall_table_64.h
generic-y += agp.h
generic-y += kvm_para.h
generic-y += mcs_spinlock.h
+generic-y += ring_buffer.h
generic-y += text-patching.h
diff --git a/arch/um/include/asm/Kbuild b/arch/um/include/asm/Kbuild
index 1b9b82bbe322..2a1629ba8140 100644
--- a/arch/um/include/asm/Kbuild
+++ b/arch/um/include/asm/Kbuild
@@ -17,6 +17,7 @@ generic-y += module.lds.h
generic-y += parport.h
generic-y += percpu.h
generic-y += preempt.h
+generic-y += ring_buffer.h
generic-y += runtime-const.h
generic-y += softirq_stack.h
generic-y += switch_to.h
diff --git a/arch/x86/include/asm/Kbuild b/arch/x86/include/asm/Kbuild
index 4566000e15c4..078fd2c0d69d 100644
--- a/arch/x86/include/asm/Kbuild
+++ b/arch/x86/include/asm/Kbuild
@@ -14,3 +14,4 @@ generic-y += early_ioremap.h
generic-y += fprobe.h
generic-y += mcs_spinlock.h
generic-y += mmzone.h
+generic-y += ring_buffer.h
diff --git a/arch/xtensa/include/asm/Kbuild b/arch/xtensa/include/asm/Kbuild
index 13fe45dea296..e57af619263a 100644
--- a/arch/xtensa/include/asm/Kbuild
+++ b/arch/xtensa/include/asm/Kbuild
@@ -6,5 +6,6 @@ generic-y += mcs_spinlock.h
generic-y += parport.h
generic-y += qrwlock.h
generic-y += qspinlock.h
+generic-y += ring_buffer.h
generic-y += user.h
generic-y += text-patching.h
diff --git a/include/asm-generic/ring_buffer.h b/include/asm-generic/ring_buffer.h
new file mode 100644
index 000000000000..201d2aee1005
--- /dev/null
+++ b/include/asm-generic/ring_buffer.h
@@ -0,0 +1,13 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Generic arch dependent ring_buffer macros.
+ */
+#ifndef __ASM_GENERIC_RING_BUFFER_H__
+#define __ASM_GENERIC_RING_BUFFER_H__
+
+#include <linux/cacheflush.h>
+
+/* Flush cache on ring buffer range if needed. Do nothing by default. */
+#define arch_ring_buffer_flush_range(start, end) do { } while (0)
+
+#endif /* __ASM_GENERIC_RING_BUFFER_H__ */
diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c
index 8b6c39bba56d..3e793bd1c134 100644
--- a/kernel/trace/ring_buffer.c
+++ b/kernel/trace/ring_buffer.c
@@ -7,6 +7,7 @@
#include <linux/ring_buffer_types.h>
#include <linux/sched/isolation.h>
#include <linux/trace_recursion.h>
+#include <linux/panic_notifier.h>
#include <linux/trace_events.h>
#include <linux/ring_buffer.h>
#include <linux/trace_clock.h>
@@ -31,6 +32,7 @@
#include <linux/oom.h>
#include <linux/mm.h>
+#include <asm/ring_buffer.h>
#include <asm/local64.h>
#include <asm/local.h>
#include <asm/setup.h>
@@ -559,6 +561,7 @@ struct trace_buffer {
unsigned long range_addr_start;
unsigned long range_addr_end;
+ struct notifier_block flush_nb;
struct ring_buffer_meta *meta;
@@ -2520,6 +2523,16 @@ static void rb_free_cpu_buffer(struct ring_buffer_per_cpu *cpu_buffer)
kfree(cpu_buffer);
}
+/* Stop recording on a persistent buffer and flush cache if needed. */
+static int rb_flush_buffer_cb(struct notifier_block *nb, unsigned long event, void *data)
+{
+ struct trace_buffer *buffer = container_of(nb, struct trace_buffer, flush_nb);
+
+ ring_buffer_record_off(buffer);
+ arch_ring_buffer_flush_range(buffer->range_addr_start, buffer->range_addr_end);
+ return NOTIFY_DONE;
+}
+
static struct trace_buffer *alloc_buffer(unsigned long size, unsigned flags,
int order, unsigned long start,
unsigned long end,
@@ -2650,6 +2663,12 @@ static struct trace_buffer *alloc_buffer(unsigned long size, unsigned flags,
mutex_init(&buffer->mutex);
+ /* Persistent ring buffer needs to flush cache before reboot. */
+ if (start && end) {
+ buffer->flush_nb.notifier_call = rb_flush_buffer_cb;
+ atomic_notifier_chain_register(&panic_notifier_list, &buffer->flush_nb);
+ }
+
return_ptr(buffer);
fail_free_buffers:
@@ -2748,6 +2767,9 @@ ring_buffer_free(struct trace_buffer *buffer)
{
int cpu;
+ if (buffer->range_addr_start && buffer->range_addr_end)
+ atomic_notifier_chain_unregister(&panic_notifier_list, &buffer->flush_nb);
+
cpuhp_state_remove_instance(CPUHP_TRACE_RB_PREPARE, &buffer->node);
irq_work_sync(&buffer->irq_work.work);
^ permalink raw reply related
* [PATCH v13 2/4] ring-buffer: Skip invalid sub-buffers when validating persistent ring buffer
From: Masami Hiramatsu (Google) @ 2026-03-25 2:25 UTC (permalink / raw)
To: Steven Rostedt
Cc: Masami Hiramatsu, Mathieu Desnoyers, linux-kernel,
linux-trace-kernel, Ian Rogers
In-Reply-To: <177440549083.1529621.15486836623498328967.stgit@mhiramat.tok.corp.google.com>
From: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Skip invalid sub-buffers when validating the persistent ring buffer
instead of discarding the entire ring buffer. Only skipped buffers
are invalidated (cleared).
If the cache data in memory fails to be synchronized during a reboot,
the persistent ring buffer may become partially corrupted, but other
sub-buffers may still contain readable event data. Only discard the
subbuffers that are found to be corrupted.
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
---
Changes in v11:
- Fix a typo.
Changes in v9:
- Add meta->subbuf_size check.
- Fix a typo.
- Handle invalid reader_page case.
Changes in v8:
- Add comment in rb_valudate_buffer()
- Clear the RB_MISSED_* flags in rb_valudate_buffer() instead of
skipping subbuf.
- Remove unused subbuf local variable from rb_cpu_meta_valid().
Changes in v7:
- Combined with Handling RB_MISSED_* flags patch, focus on validation at boot.
- Remove checking subbuffer data when validating metadata, because it should be done
later.
- Do not mark the discarded sub buffer page but just reset it.
Changes in v6:
- Show invalid page detection message once per CPU.
Changes in v5:
- Instead of showing errors for each page, just show the number
of discarded pages at last.
Changes in v3:
- Record missed data event on commit.
---
kernel/trace/ring_buffer.c | 98 ++++++++++++++++++++++++++------------------
1 file changed, 58 insertions(+), 40 deletions(-)
diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c
index 3e793bd1c134..31cad8edd488 100644
--- a/kernel/trace/ring_buffer.c
+++ b/kernel/trace/ring_buffer.c
@@ -370,6 +370,12 @@ static __always_inline unsigned int rb_page_commit(struct buffer_page *bpage)
return local_read(&bpage->page->commit);
}
+/* Size is determined by what has been committed */
+static __always_inline unsigned int rb_page_size(struct buffer_page *bpage)
+{
+ return rb_page_commit(bpage) & ~RB_MISSED_MASK;
+}
+
static void free_buffer_page(struct buffer_page *bpage)
{
/* Range pages are not to be freed */
@@ -1762,7 +1768,6 @@ static bool rb_cpu_meta_valid(struct ring_buffer_cpu_meta *meta, int cpu,
unsigned long *subbuf_mask)
{
int subbuf_size = PAGE_SIZE;
- struct buffer_data_page *subbuf;
unsigned long buffers_start;
unsigned long buffers_end;
int i;
@@ -1770,6 +1775,11 @@ static bool rb_cpu_meta_valid(struct ring_buffer_cpu_meta *meta, int cpu,
if (!subbuf_mask)
return false;
+ if (meta->subbuf_size != PAGE_SIZE) {
+ pr_info("Ring buffer boot meta [%d] invalid subbuf_size\n", cpu);
+ return false;
+ }
+
buffers_start = meta->first_buffer;
buffers_end = meta->first_buffer + (subbuf_size * meta->nr_subbufs);
@@ -1786,11 +1796,12 @@ static bool rb_cpu_meta_valid(struct ring_buffer_cpu_meta *meta, int cpu,
return false;
}
- subbuf = rb_subbufs_from_meta(meta);
-
bitmap_clear(subbuf_mask, 0, meta->nr_subbufs);
- /* Is the meta buffers and the subbufs themselves have correct data? */
+ /*
+ * Ensure the meta::buffers array has correct data. The data in each subbufs
+ * are checked later in rb_meta_validate_events().
+ */
for (i = 0; i < meta->nr_subbufs; i++) {
if (meta->buffers[i] < 0 ||
meta->buffers[i] >= meta->nr_subbufs) {
@@ -1798,18 +1809,12 @@ static bool rb_cpu_meta_valid(struct ring_buffer_cpu_meta *meta, int cpu,
return false;
}
- if ((unsigned)local_read(&subbuf->commit) > subbuf_size) {
- pr_info("Ring buffer boot meta [%d] buffer invalid commit\n", cpu);
- return false;
- }
-
if (test_bit(meta->buffers[i], subbuf_mask)) {
pr_info("Ring buffer boot meta [%d] array has duplicates\n", cpu);
return false;
}
set_bit(meta->buffers[i], subbuf_mask);
- subbuf = (void *)subbuf + subbuf_size;
}
return true;
@@ -1873,13 +1878,22 @@ static int rb_read_data_buffer(struct buffer_data_page *dpage, int tail, int cpu
return events;
}
-static int rb_validate_buffer(struct buffer_data_page *dpage, int cpu)
+static int rb_validate_buffer(struct buffer_data_page *dpage, int cpu,
+ struct ring_buffer_cpu_meta *meta)
{
unsigned long long ts;
+ unsigned long tail;
u64 delta;
- int tail;
- tail = local_read(&dpage->commit);
+ /*
+ * When a sub-buffer is recovered from a read, the commit value may
+ * have RB_MISSED_* bits set, as these bits are reset on reuse.
+ * Even after clearing these bits, a commit value greater than the
+ * subbuf_size is considered invalid.
+ */
+ tail = local_read(&dpage->commit) & ~RB_MISSED_MASK;
+ if (tail > meta->subbuf_size)
+ return -1;
return rb_read_data_buffer(dpage, tail, cpu, &ts, &delta);
}
@@ -1890,6 +1904,7 @@ static void rb_meta_validate_events(struct ring_buffer_per_cpu *cpu_buffer)
struct buffer_page *head_page, *orig_head;
unsigned long entry_bytes = 0;
unsigned long entries = 0;
+ int discarded = 0;
int ret;
u64 ts;
int i;
@@ -1900,14 +1915,19 @@ static void rb_meta_validate_events(struct ring_buffer_per_cpu *cpu_buffer)
orig_head = head_page = cpu_buffer->head_page;
/* Do the reader page first */
- ret = rb_validate_buffer(cpu_buffer->reader_page->page, cpu_buffer->cpu);
+ ret = rb_validate_buffer(cpu_buffer->reader_page->page, cpu_buffer->cpu, meta);
if (ret < 0) {
- pr_info("Ring buffer reader page is invalid\n");
- goto invalid;
+ pr_info("Ring buffer meta [%d] invalid reader page detected\n",
+ cpu_buffer->cpu);
+ discarded++;
+ /* Instead of discard whole ring buffer, discard only this sub-buffer. */
+ local_set(&cpu_buffer->reader_page->entries, 0);
+ local_set(&cpu_buffer->reader_page->page->commit, 0);
+ } else {
+ entries += ret;
+ entry_bytes += rb_page_size(cpu_buffer->reader_page);
+ local_set(&cpu_buffer->reader_page->entries, ret);
}
- entries += ret;
- entry_bytes += local_read(&cpu_buffer->reader_page->page->commit);
- local_set(&cpu_buffer->reader_page->entries, ret);
ts = head_page->page->time_stamp;
@@ -1935,7 +1955,7 @@ static void rb_meta_validate_events(struct ring_buffer_per_cpu *cpu_buffer)
break;
/* Stop rewind if the page is invalid. */
- ret = rb_validate_buffer(head_page->page, cpu_buffer->cpu);
+ ret = rb_validate_buffer(head_page->page, cpu_buffer->cpu, meta);
if (ret < 0)
break;
@@ -2014,21 +2034,24 @@ static void rb_meta_validate_events(struct ring_buffer_per_cpu *cpu_buffer)
if (head_page == cpu_buffer->reader_page)
continue;
- ret = rb_validate_buffer(head_page->page, cpu_buffer->cpu);
+ ret = rb_validate_buffer(head_page->page, cpu_buffer->cpu, meta);
if (ret < 0) {
- pr_info("Ring buffer meta [%d] invalid buffer page\n",
- cpu_buffer->cpu);
- goto invalid;
- }
-
- /* If the buffer has content, update pages_touched */
- if (ret)
- local_inc(&cpu_buffer->pages_touched);
-
- entries += ret;
- entry_bytes += local_read(&head_page->page->commit);
- local_set(&head_page->entries, ret);
+ if (!discarded)
+ pr_info("Ring buffer meta [%d] invalid buffer page detected\n",
+ cpu_buffer->cpu);
+ discarded++;
+ /* Instead of discard whole ring buffer, discard only this sub-buffer. */
+ local_set(&head_page->entries, 0);
+ local_set(&head_page->page->commit, 0);
+ } else {
+ /* If the buffer has content, update pages_touched */
+ if (ret)
+ local_inc(&cpu_buffer->pages_touched);
+ entries += ret;
+ entry_bytes += rb_page_size(head_page);
+ local_set(&head_page->entries, ret);
+ }
if (head_page == cpu_buffer->commit_page)
break;
}
@@ -2042,7 +2065,8 @@ static void rb_meta_validate_events(struct ring_buffer_per_cpu *cpu_buffer)
local_set(&cpu_buffer->entries, entries);
local_set(&cpu_buffer->entries_bytes, entry_bytes);
- pr_info("Ring buffer meta [%d] is from previous boot!\n", cpu_buffer->cpu);
+ pr_info("Ring buffer meta [%d] is from previous boot! (%d pages discarded)\n",
+ cpu_buffer->cpu, discarded);
return;
invalid:
@@ -3329,12 +3353,6 @@ rb_iter_head_event(struct ring_buffer_iter *iter)
return NULL;
}
-/* Size is determined by what has been committed */
-static __always_inline unsigned rb_page_size(struct buffer_page *bpage)
-{
- return rb_page_commit(bpage) & ~RB_MISSED_MASK;
-}
-
static __always_inline unsigned
rb_commit_index(struct ring_buffer_per_cpu *cpu_buffer)
{
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox