All of lore.kernel.org
 help / color / mirror / Atom feed
From: Sasha Levin <sashal@kernel.org>
To: linux-kernel@vger.kernel.org, stable@vger.kernel.org
Cc: Petr Mladek <pmladek@suse.com>,
	Steven Rostedt <rostedt@goodmis.org>,
	Sasha Levin <sashal@kernel.org>
Subject: [PATCH AUTOSEL 4.4 21/21] tracing: Initialize iter->seq after zeroing in tracing_read_pipe()
Date: Fri, 18 Oct 2019 18:10:07 -0400	[thread overview]
Message-ID: <20191018221007.10851-21-sashal@kernel.org> (raw)
In-Reply-To: <20191018221007.10851-1-sashal@kernel.org>

From: Petr Mladek <pmladek@suse.com>

[ Upstream commit d303de1fcf344ff7c15ed64c3f48a991c9958775 ]

A customer reported the following softlockup:

[899688.160002] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [test.sh:16464]
[899688.160002] CPU: 0 PID: 16464 Comm: test.sh Not tainted 4.12.14-6.23-azure #1 SLE12-SP4
[899688.160002] RIP: 0010:up_write+0x1a/0x30
[899688.160002] Kernel panic - not syncing: softlockup: hung tasks
[899688.160002] RIP: 0010:up_write+0x1a/0x30
[899688.160002] RSP: 0018:ffffa86784d4fde8 EFLAGS: 00000257 ORIG_RAX: ffffffffffffff12
[899688.160002] RAX: ffffffff970fea00 RBX: 0000000000000001 RCX: 0000000000000000
[899688.160002] RDX: ffffffff00000001 RSI: 0000000000000080 RDI: ffffffff970fea00
[899688.160002] RBP: ffffffffffffffff R08: ffffffffffffffff R09: 0000000000000000
[899688.160002] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8b59014720d8
[899688.160002] R13: ffff8b59014720c0 R14: ffff8b5901471090 R15: ffff8b5901470000
[899688.160002]  tracing_read_pipe+0x336/0x3c0
[899688.160002]  __vfs_read+0x26/0x140
[899688.160002]  vfs_read+0x87/0x130
[899688.160002]  SyS_read+0x42/0x90
[899688.160002]  do_syscall_64+0x74/0x160

It caught the process in the middle of trace_access_unlock(). There is
no loop. So, it must be looping in the caller tracing_read_pipe()
via the "waitagain" label.

Crashdump analyze uncovered that iter->seq was completely zeroed
at this point, including iter->seq.seq.size. It means that
print_trace_line() was never able to print anything and
there was no forward progress.

The culprit seems to be in the code:

	/* reset all but tr, trace, and overruns */
	memset(&iter->seq, 0,
	       sizeof(struct trace_iterator) -
	       offsetof(struct trace_iterator, seq));

It was added by the commit 53d0aa773053ab182877 ("ftrace:
add logic to record overruns"). It was v2.6.27-rc1.
It was the time when iter->seq looked like:

     struct trace_seq {
	unsigned char		buffer[PAGE_SIZE];
	unsigned int		len;
     };

There was no "size" variable and zeroing was perfectly fine.

The solution is to reinitialize the structure after or without
zeroing.

Link: http://lkml.kernel.org/r/20191011142134.11997-1-pmladek@suse.com

Signed-off-by: Petr Mladek <pmladek@suse.com>
Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
 kernel/trace/trace.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index c82ebd11414a9..b3fa2a0ed981c 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -4790,6 +4790,7 @@ tracing_read_pipe(struct file *filp, char __user *ubuf,
 	       sizeof(struct trace_iterator) -
 	       offsetof(struct trace_iterator, seq));
 	cpumask_clear(iter->started);
+	trace_seq_init(&iter->seq);
 	iter->pos = -1;
 
 	trace_event_read_lock();
-- 
2.20.1


      parent reply	other threads:[~2019-10-18 22:11 UTC|newest]

Thread overview: 21+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-10-18 22:09 [PATCH AUTOSEL 4.4 01/21] sc16is7xx: Fix for "Unexpected interrupt: 8" Sasha Levin
2019-10-18 22:09 ` [PATCH AUTOSEL 4.4 02/21] x86/cpu: Add Atom Tremont (Jacobsville) Sasha Levin
2019-10-18 22:09 ` [PATCH AUTOSEL 4.4 03/21] scripts/setlocalversion: Improve -dirty check with git-status --no-optional-locks Sasha Levin
2019-10-18 22:09 ` [PATCH AUTOSEL 4.4 04/21] usb: handle warm-reset port requests on hub resume Sasha Levin
2019-10-18 22:09 ` [PATCH AUTOSEL 4.4 05/21] exec: load_script: Do not exec truncated interpreter path Sasha Levin
2019-10-18 22:09 ` [PATCH AUTOSEL 4.4 06/21] iio: fix center temperature of bmc150-accel-core Sasha Levin
2019-10-18 22:09 ` [PATCH AUTOSEL 4.4 07/21] perf map: Fix overlapped map handling Sasha Levin
2019-10-18 22:09 ` [PATCH AUTOSEL 4.4 08/21] RDMA/iwcm: Fix a lock inversion issue Sasha Levin
2019-10-18 22:09 ` [PATCH AUTOSEL 4.4 09/21] nfs: Fix nfsi->nrequests count error on nfs_inode_remove_request Sasha Levin
2019-10-18 22:09 ` [PATCH AUTOSEL 4.4 10/21] fs: cifs: mute -Wunused-const-variable message Sasha Levin
2019-10-18 22:09 ` [PATCH AUTOSEL 4.4 11/21] serial: mctrl_gpio: Check for NULL pointer Sasha Levin
2019-10-18 22:09 ` [PATCH AUTOSEL 4.4 12/21] efi/cper: Fix endianness of PCIe class code Sasha Levin
2019-10-18 22:09 ` [PATCH AUTOSEL 4.4 13/21] efi/x86: Do not clean dummy variable in kexec path Sasha Levin
2019-10-18 22:10 ` [PATCH AUTOSEL 4.4 14/21] fs: ocfs2: fix possible null-pointer dereferences in ocfs2_xa_prepare_entry() Sasha Levin
2019-10-18 22:10 ` [PATCH AUTOSEL 4.4 15/21] fs: ocfs2: fix a possible null-pointer dereference in ocfs2_info_scan_inode_alloc() Sasha Levin
2019-10-18 22:10 ` [PATCH AUTOSEL 4.4 16/21] iio: adc: ad799x: fix probe error handling Sasha Levin
2019-10-18 22:10 ` [PATCH AUTOSEL 4.4 17/21] iio: light: opt3001: fix mutex unlock race Sasha Levin
2019-10-18 22:10 ` [PATCH AUTOSEL 4.4 18/21] MIPS: fw: sni: Fix out of bounds init of o32 stack Sasha Levin
2019-10-18 22:10 ` [PATCH AUTOSEL 4.4 19/21] USB: usb-skeleton: fix use-after-free after driver unbind Sasha Levin
2019-10-18 22:10 ` [PATCH AUTOSEL 4.4 20/21] NFSv4: Fix leak of clp->cl_acceptor string Sasha Levin
2019-10-18 22:10 ` Sasha Levin [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20191018221007.10851-21-sashal@kernel.org \
    --to=sashal@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=pmladek@suse.com \
    --cc=rostedt@goodmis.org \
    --cc=stable@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.