All of lore.kernel.org
 help / color / mirror / Atom feed
From: Sasha Levin <sashal@kernel.org>
To: linux-kernel@vger.kernel.org, stable@vger.kernel.org
Cc: Petr Mladek <pmladek@suse.com>,
	Steven Rostedt <rostedt@goodmis.org>,
	Sasha Levin <sashal@kernel.org>
Subject: [PATCH AUTOSEL 4.14 56/56] tracing: Initialize iter->seq after zeroing in tracing_read_pipe()
Date: Fri, 18 Oct 2019 18:07:53 -0400	[thread overview]
Message-ID: <20191018220753.10002-56-sashal@kernel.org> (raw)
In-Reply-To: <20191018220753.10002-1-sashal@kernel.org>

From: Petr Mladek <pmladek@suse.com>

[ Upstream commit d303de1fcf344ff7c15ed64c3f48a991c9958775 ]

A customer reported the following softlockup:

[899688.160002] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [test.sh:16464]
[899688.160002] CPU: 0 PID: 16464 Comm: test.sh Not tainted 4.12.14-6.23-azure #1 SLE12-SP4
[899688.160002] RIP: 0010:up_write+0x1a/0x30
[899688.160002] Kernel panic - not syncing: softlockup: hung tasks
[899688.160002] RIP: 0010:up_write+0x1a/0x30
[899688.160002] RSP: 0018:ffffa86784d4fde8 EFLAGS: 00000257 ORIG_RAX: ffffffffffffff12
[899688.160002] RAX: ffffffff970fea00 RBX: 0000000000000001 RCX: 0000000000000000
[899688.160002] RDX: ffffffff00000001 RSI: 0000000000000080 RDI: ffffffff970fea00
[899688.160002] RBP: ffffffffffffffff R08: ffffffffffffffff R09: 0000000000000000
[899688.160002] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8b59014720d8
[899688.160002] R13: ffff8b59014720c0 R14: ffff8b5901471090 R15: ffff8b5901470000
[899688.160002]  tracing_read_pipe+0x336/0x3c0
[899688.160002]  __vfs_read+0x26/0x140
[899688.160002]  vfs_read+0x87/0x130
[899688.160002]  SyS_read+0x42/0x90
[899688.160002]  do_syscall_64+0x74/0x160

It caught the process in the middle of trace_access_unlock(). There is
no loop. So, it must be looping in the caller tracing_read_pipe()
via the "waitagain" label.

Crashdump analyze uncovered that iter->seq was completely zeroed
at this point, including iter->seq.seq.size. It means that
print_trace_line() was never able to print anything and
there was no forward progress.

The culprit seems to be in the code:

	/* reset all but tr, trace, and overruns */
	memset(&iter->seq, 0,
	       sizeof(struct trace_iterator) -
	       offsetof(struct trace_iterator, seq));

It was added by the commit 53d0aa773053ab182877 ("ftrace:
add logic to record overruns"). It was v2.6.27-rc1.
It was the time when iter->seq looked like:

     struct trace_seq {
	unsigned char		buffer[PAGE_SIZE];
	unsigned int		len;
     };

There was no "size" variable and zeroing was perfectly fine.

The solution is to reinitialize the structure after or without
zeroing.

Link: http://lkml.kernel.org/r/20191011142134.11997-1-pmladek@suse.com

Signed-off-by: Petr Mladek <pmladek@suse.com>
Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
 kernel/trace/trace.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index 91227e339ef60..5e2f0fb9994d1 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -5751,6 +5751,7 @@ tracing_read_pipe(struct file *filp, char __user *ubuf,
 	       sizeof(struct trace_iterator) -
 	       offsetof(struct trace_iterator, seq));
 	cpumask_clear(iter->started);
+	trace_seq_init(&iter->seq);
 	iter->pos = -1;
 
 	trace_event_read_lock();
-- 
2.20.1


      parent reply	other threads:[~2019-10-18 22:13 UTC|newest]

Thread overview: 60+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-10-18 22:06 [f2fs-dev] [PATCH AUTOSEL 4.14 01/56] f2fs: flush quota blocks after turnning it off Sasha Levin
2019-10-18 22:06 ` Sasha Levin
2019-10-18 22:06 ` [PATCH AUTOSEL 4.14 02/56] scsi: lpfc: Fix a duplicate 0711 log message number Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 03/56] sc16is7xx: Fix for "Unexpected interrupt: 8" Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 04/56] powerpc/powernv: hold device_hotplug_lock when calling memtrace_offline_pages() Sasha Levin
2019-10-18 22:07   ` Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 05/56] HID: i2c-hid: add Direkt-Tek DTLAPY133-1 to descriptor override Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 06/56] x86/cpu: Add Atom Tremont (Jacobsville) Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 07/56] HID: i2c-hid: Add Odys Winbook 13 to descriptor override Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 08/56] clk: boston: unregister clks on failure in clk_boston_setup() Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 09/56] scripts/setlocalversion: Improve -dirty check with git-status --no-optional-locks Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 10/56] HID: Add ASUS T100CHI keyboard dock battery quirks Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 11/56] usb: handle warm-reset port requests on hub resume Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 12/56] rtc: pcf8523: set xtal load capacitance from DT Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 13/56] mlxsw: spectrum: Set LAG port collector only when active Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 14/56] ALSA: hda/realtek - Apply ALC294 hp init also for S4 resume Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 15/56] media: vimc: Remove unused but set variables Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 16/56] exec: load_script: Do not exec truncated interpreter path Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 17/56] PCI/PME: Fix possible use-after-free on remove Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 18/56] power: supply: max14656: fix potential use-after-free Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 19/56] iio: adc: meson_saradc: Fix memory allocation order Sasha Levin
2019-10-18 22:07   ` Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 20/56] iio: fix center temperature of bmc150-accel-core Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 21/56] libsubcmd: Make _FORTIFY_SOURCE defines dependent on the feature Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 22/56] perf tests: Avoid raising SEGV using an obvious NULL dereference Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 23/56] perf map: Fix overlapped map handling Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 24/56] perf jevents: Fix period for Intel fixed counters Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 25/56] staging: rtl8188eu: fix null dereference when kzalloc fails Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 26/56] crypto: arm/aes-ce - add dependency on AES library Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 27/56] RDMA/hfi1: Prevent memory leak in sdma_init Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 28/56] RDMA/iwcm: Fix a lock inversion issue Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 29/56] HID: hyperv: Use in-place iterator API in the channel callback Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 30/56] nfs: Fix nfsi->nrequests count error on nfs_inode_remove_request Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 31/56] arm64: ftrace: Ensure synchronisation in PLT setup for Neoverse-N1 #1542419 Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 32/56] tty: serial: owl: Fix the link time qualifier of 'owl_uart_exit()' Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 33/56] tty: n_hdlc: fix build on SPARC Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 34/56] gpio: max77620: Use correct unit for debounce times Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 35/56] fs: cifs: mute -Wunused-const-variable message Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 36/56] serial: mctrl_gpio: Check for NULL pointer Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 37/56] efi/cper: Fix endianness of PCIe class code Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 38/56] efi/x86: Do not clean dummy variable in kexec path Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 39/56] kbuild: fix build error of 'make nsdeps' in clean tree Sasha Levin
2019-10-19  0:13   ` Masahiro Yamada
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 40/56] MIPS: include: Mark __cmpxchg as __always_inline Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 41/56] x86/xen: Return from panic notifier Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 42/56] ocfs2: clear zero in unaligned direct IO Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 43/56] fs: ocfs2: fix possible null-pointer dereferences in ocfs2_xa_prepare_entry() Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 44/56] fs: ocfs2: fix a possible null-pointer dereference in ocfs2_write_end_nolock() Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 45/56] fs: ocfs2: fix a possible null-pointer dereference in ocfs2_info_scan_inode_alloc() Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 46/56] sched/vtime: Fix guest/system mis-accounting on task switch Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 47/56] perf/x86/amd: Change/fix NMI latency mitigation to use a timestamp Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 48/56] iio: adc: ad799x: fix probe error handling Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 49/56] iio: light: opt3001: fix mutex unlock race Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 50/56] MIPS: include: Mark __xchg as __always_inline Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 51/56] MIPS: fw: sni: Fix out of bounds init of o32 stack Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 52/56] USB: usb-skeleton: fix use-after-free after driver unbind Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 53/56] nbd: fix possible sysfs duplicate warning Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 54/56] NFSv4: Fix leak of clp->cl_acceptor string Sasha Levin
2019-10-18 22:07 ` [PATCH AUTOSEL 4.14 55/56] s390/uaccess: avoid (false positive) compiler warnings Sasha Levin
2019-10-18 22:07 ` Sasha Levin [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20191018220753.10002-56-sashal@kernel.org \
    --to=sashal@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=pmladek@suse.com \
    --cc=rostedt@goodmis.org \
    --cc=stable@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.