From: "Jiazi.Li" <jqqlijiazi@gmail.com>
To: Steven Rostedt <rostedt@goodmis.org>, Ingo Molnar <mingo@redhat.com>
Cc: "Jiazi.Li" <jiazi.li@transsion.com>, linux-trace-devel@vger.kernel.org
Subject: [PATCH] ring-buffer: Add barrire in rb_move_tail
Date: Tue, 30 Aug 2022 20:08:54 +0800 [thread overview]
Message-ID: <20220830120854.7545-1-jiazi.li@transsion.com> (raw)
The following crash issue was encountered several times when read
trace_pipe:
1. in ring_buffer_event_data, found event's type is bigger than
RINGBUF_TYPE_DATA_TYPE_LEN_MAX, print WARNING:
[249033.677658]WARNING: CPU: 1 PID: 9975 at ring_buffer_event_data+0x38/0x48
2. trace_read_pipe print this event resulting in crash issue:
[249033.806200] die+0xac/0x37c
[249033.806891] die_kernel_fault+0x60/0x70
[249033.807710] die_kernel_fault+0x0/0x70
[249033.808518] do_bad_area+0x30/0x130
[249033.809296] do_translation_fault+0x34/0x40
[249033.810158] do_mem_abort+0x4c/0xf8
[249033.810935] el1_da+0x1c/0xac
[249033.811647] string+0x38/0x74
[249033.812357] vsnprintf+0x430/0x65c
[249033.813122] seq_buf_vprintf+0x64/0xcc
[249033.813934] trace_seq_printf+0x70/0xd8
[249033.814755] trace_raw_output_ipi_handler+0x40/0x54
[249033.815705] print_trace_line+0x3ec/0x424
[249033.816547] tracing_read_pipe+0x28c/0x3ec
[249033.817402] __vfs_read+0x44/0x13c
[249033.818166] vfs_read+0xb8/0x150
[249033.818908] ksys_read+0x6c/0xcc
[249033.819651] __arm64_sys_read+0x18/0x20
[249033.820470] el0_svc_common+0x98/0x160
[249033.821278] el0_svc_handler+0x60/0x78
[249033.822086] el0_svc+0x8/0x300
ipi event is not enable in my device, the event tracing_read_pipe
want to print is a PADDING event generated in rb_move_tail.
array[0] of this event is 0x18, ipi entry trace event's type value
is 0x18 too in my device.
so trace_read_pipe call trace_raw_output_ipi_handler to print this
event, eventually, system crash because illegal pointers.
In ring_buffer_peek, PADDING type event will not return to caller.
So at least when ring_buffer_peek is called, this event has not
been set to PADDING.
My reader task poll on trace_pipe and other files, wakeup
by other files and found that trace ring_buffer is not empty
because of read != commit.
But maybe because of cpu out-of-order execution, new event has not
been set to PADDING type, and leads to above issue.
Can we fix this issue by add a barrier between rb_reset_tail
and rb_end_commit?
Or are there other bugs that cause this issue?
Signed-off-by: Jiazi.Li <jiazi.li@transsion.com>
---
kernel/trace/ring_buffer.c | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c
index d59b6a328b7f..20f3ef839467 100644
--- a/kernel/trace/ring_buffer.c
+++ b/kernel/trace/ring_buffer.c
@@ -2717,6 +2717,11 @@ rb_move_tail(struct ring_buffer_per_cpu *cpu_buffer,
rb_reset_tail(cpu_buffer, tail, info);
+ /*
+ * Add barrier to make sure tail event be set to
+ * PADDING type before update commit.
+ */
+ smp_mb();
/* Commit what we have for now. */
rb_end_commit(cpu_buffer);
/* rb_end_commit() decs committing */
--
2.17.1
next reply other threads:[~2022-08-30 12:09 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-08-30 12:08 Jiazi.Li [this message]
2022-09-01 15:13 ` [PATCH] ring-buffer: Add barrire in rb_move_tail Steven Rostedt
2022-09-02 3:47 ` lijiazi
2022-09-02 12:56 ` Steven Rostedt
2022-09-05 3:23 ` lijiazi
2022-09-06 16:25 ` Steven Rostedt
2022-09-07 3:03 ` lijiazi
2022-09-07 8:13 ` lijiazi
2022-09-15 8:25 ` lijiazi
2022-09-28 14:54 ` Steven Rostedt
2022-09-28 14:59 ` Steven Rostedt
2022-09-28 20:29 ` Steven Rostedt
2022-09-29 9:12 ` lijiazi
2022-09-30 21:49 ` Steven Rostedt
2022-10-08 1:21 ` lijiazi
2022-10-08 12:50 ` Steven Rostedt
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220830120854.7545-1-jiazi.li@transsion.com \
--to=jqqlijiazi@gmail.com \
--cc=jiazi.li@transsion.com \
--cc=linux-trace-devel@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=rostedt@goodmis.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).