From: Arnaldo Carvalho de Melo <acme@kernel.org>
To: Ingo Molnar <mingo@kernel.org>
Cc: linux-kernel@vger.kernel.org,
	Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>,
	Jiri Olsa <jolsa@redhat.com>,
	Li Zhang <zhlcindy@linux.vnet.ibm.com>,
	Arnaldo Carvalho de Melo <acme@redhat.com>
Subject: [PATCH 6/8] perf trace: Fix race condition at the end of started workloads
Date: Wed, 17 Jun 2015 18:22:33 -0300	[thread overview]
Message-ID: <1434576155-30038-7-git-send-email-acme@kernel.org> (raw)
In-Reply-To: <1434576155-30038-1-git-send-email-acme@kernel.org>
From: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
I get following crash on multiple systems and across several releases
(at least since v3.18).
	Core was generated by `/tmp/perf trace sleep 0.2 '.
	Program terminated with signal SIGSEGV, Segmentation fault.
	#0  perf_mmap__read_head (mm=0x3fff9bf30070) at util/evlist.h:195
	195		u64 head = ACCESS_ONCE(pc->data_head);
	(gdb) bt
	#0  perf_mmap__read_head (mm=0x3fff9bf30070) at util/evlist.h:195
	#1  perf_evlist__mmap_read (evlist=0x10027f11910, idx=<optimized out>)
	    at util/evlist.c:637
	#2  0x000000001003ce4c in trace__run (argv=<optimized out>,
	    argc=<optimized out>, trace=0x3fffd7b28288) at builtin-trace.c:2259
	#3  cmd_trace (argc=<optimized out>, argv=<optimized out>,
	    prefix=<optimized out>) at builtin-trace.c:2799
	#4  0x00000000100657b8 in run_builtin (p=0x10176798 <commands+480>, argc=3,
	    argv=0x3fffd7b2b550) at perf.c:370
	#5  0x00000000100063e8 in handle_internal_command (argv=0x3fffd7b2b550, argc=3)
	    at perf.c:429
	#6  run_argv (argv=0x3fffd7b2af70, argcp=0x3fffd7b2af7c) at perf.c:473
	#7  main (argc=3, argv=0x3fffd7b2b550) at perf.c:588
The problem seems to be a race condition, when the application has just
exited.  Some/all fds associated with the perf-events (tracepoints) go
into a POLLHUP/ POLLERR state and the mmap region associated with those
events are unmapped (in perf_evlist__filter_pollfd()).
But we go back and do a perf_evlist__mmap_read() which assumes that the
mmaps are still valid and we hit the crash.
If the mapping for an event is released, its refcnt is 0 (and ->base
is NULL), so ensure we have non-zero refcount before accessing the map.
Note that perf-record has a similar logic but unlike perf-trace, the
record__mmap_read_all() checks the evlist->mmap[i].base before accessing
the map.
Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Li Zhang <zhlcindy@linux.vnet.ibm.com>
Link: http://lkml.kernel.org/r/20150612060003.GA19913@us.ibm.com
[ Fixed it up to use atomic_read() ]
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/evlist.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/tools/perf/util/evlist.c b/tools/perf/util/evlist.c
index dc1dc2c181ef..6b58a47a79ec 100644
--- a/tools/perf/util/evlist.c
+++ b/tools/perf/util/evlist.c
@@ -634,11 +634,18 @@ static struct perf_evsel *perf_evlist__event2evsel(struct perf_evlist *evlist,
 union perf_event *perf_evlist__mmap_read(struct perf_evlist *evlist, int idx)
 {
 	struct perf_mmap *md = &evlist->mmap[idx];
-	u64 head = perf_mmap__read_head(md);
+	u64 head;
 	u64 old = md->prev;
 	unsigned char *data = md->base + page_size;
 	union perf_event *event = NULL;
 
+	/*
+	 * Check if event was unmapped due to a POLLHUP/POLLERR.
+	 */
+	if (!atomic_read(&md->refcnt))
+		return NULL;
+
+	head = perf_mmap__read_head(md);
 	if (evlist->overwrite) {
 		/*
 		 * If we're further behind than half the buffer, there's a chance
-- 
2.1.0
next prev parent reply	other threads:[~2015-06-17 21:24 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2015-06-17 21:22 [GIT PULL 0/8] perf/core improvements and fixes Arnaldo Carvalho de Melo
2015-06-17 21:22 ` [PATCH 1/8] perf tools: Ignore .config-detected in .gitignore Arnaldo Carvalho de Melo
2015-06-17 21:22 ` [PATCH 2/8] perf tools: Fix a problem when opening old perf.data with different byte order Arnaldo Carvalho de Melo
2015-06-17 21:22 ` [PATCH 3/8] perf tools: Move libtraceevent dynamic list to separated LDFLAGS variable Arnaldo Carvalho de Melo
2015-06-17 21:22 ` [PATCH 4/8] perf probe: Show usage even if the last event is skipped Arnaldo Carvalho de Melo
2015-06-17 21:22 ` [PATCH 5/8] perf probe: Speed up perf probe --list by caching debuginfo Arnaldo Carvalho de Melo
2015-06-17 21:22 ` Arnaldo Carvalho de Melo [this message]
2015-06-17 21:22 ` [PATCH 7/8] perf evlist: Add toggle_enable() method Arnaldo Carvalho de Melo
2015-06-17 21:22 ` [PATCH 8/8] perf top: Allow disabling/enabling events dynamicly Arnaldo Carvalho de Melo
2015-06-18  7:40 ` [GIT PULL 0/8] perf/core improvements and fixes Ingo Molnar
2015-06-18 20:18   ` [RFC] hotkey for disabling/enabling events in 'perf top' TUI was " Arnaldo Carvalho de Melo
2015-06-18 20:58     ` Ingo Molnar
2015-06-18 21:39       ` Arnaldo Carvalho de Melo
2015-06-19  6:27         ` Ingo Molnar
2015-06-19 20:07           ` Arnaldo Carvalho de Melo
2015-06-19 23:05             ` Ingo Molnar
2015-06-22 14:52               ` Arnaldo Carvalho de Melo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox
  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):
  git send-email \
    --in-reply-to=1434576155-30038-7-git-send-email-acme@kernel.org \
    --to=acme@kernel.org \
    --cc=acme@redhat.com \
    --cc=jolsa@redhat.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@kernel.org \
    --cc=sukadev@linux.vnet.ibm.com \
    --cc=zhlcindy@linux.vnet.ibm.com \
    /path/to/YOUR_REPLY
  https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
  Be sure your reply has a Subject: header at the top and a blank line
  before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).