[BUG] perf segfaults when combining --overwrite and intel_pt event

[BUG] perf segfaults when combining --overwrite and intel_pt event

[Tomas Glozar]

Dear maintainers,

I would like to report a bug in perf I ran into when trying to combine
--overwrite and intel_pt on latest perf from 6.15-rc4:

$ perf --version
perf version 6.15.rc4.gb4432656b36e
$ perf record --overwrite -a -e intel_pt/cyc,noretcomp/k
perf: Segmentation fault
Obtained 16 stack frames.
perf() [0x59aab7]
perf() [0x59ab97]
perf() [0x41a9db]
/lib64/libc.so.6(+0x1a050) [0x7f7fc2e27050]
perf() [0x61bb16]
perf() [0x55fee9]
perf() [0x54a790]
perf() [0x469c92]
perf() [0x46a09f]
perf() [0x46a317]
perf() [0x54ac83]
perf() [0x41c61c]
perf() [0x41c92c]
perf() [0x41cc65]
perf() [0x41ff41]
perf() [0x424015]
Segmentation fault (core dumped)

GDB gives the following backtrace:

(gdb) bt
#0  0x000000000061bb16 in auxtrace_mmap__mmap (mm=0x7c9fa0,
mp=0x7fffffff9008, userpg=0x7ffff7cff000, fd=7) at util/auxtrace.c:136
#1  0x000000000055fee9 in mmap__mmap (map=0x7c9f40, mp=0x7fffffff8ff0,
fd=7, cpu=...) at util/mmap.c:312
#2  0x000000000054a790 in perf_evlist__mmap_cb_mmap (_map=0x7c9f40,
_mp=0x7fffffff8ff0, output=7, cpu=...) at util/evlist.c:825
#3  0x0000000000469c92 in mmap_per_evsel (evlist=0x78f190,
ops=0x7fffffff8fd0, idx=0, mp=0x7fffffff8ff0, cpu_idx=0, thread=0,
   _output=0x7fffffff8f10, _output_overwrite=0x7fffffff8f14,
nr_mmaps=0x7fffffff8f0c) at evlist.c:527
#4  0x000000000046a09f in mmap_per_cpu (evlist=0x78f190,
ops=0x7fffffff8fd0, mp=0x7fffffff8ff0) at evlist.c:620
#5  0x000000000046a317 in perf_evlist__mmap_ops (evlist=0x78f190,
ops=0x7fffffff8fd0, mp=0x7fffffff8ff0) at evlist.c:679
#6  0x000000000054ac83 in evlist__mmap_ex (evlist=0x78f190,
pages=4294967295, auxtrace_pages=1024, auxtrace_overwrite=false,
   nr_cblocks=0, affinity=0, flush=1, comp_level=0) at util/evlist.c:977
#7  0x000000000041c61c in record__mmap_evlist (rec=0x764de0 <record>,
evlist=0x78f190) at builtin-record.c:1310
#8  0x000000000041c92c in record__mmap (rec=0x764de0 <record>) at
builtin-record.c:1359
#9  0x000000000041cc65 in record__open (rec=0x764de0 <record>) at
builtin-record.c:1413
#10 0x000000000041ff41 in __cmd_record (rec=0x764de0 <record>, argc=0,
argv=0x7fffffffe0f0) at builtin-record.c:2492
#11 0x0000000000424015 in cmd_record (argc=0, argv=0x7fffffffe0f0) at
builtin-record.c:4285
#12 0x0000000000461133 in run_builtin (p=0x769260 <commands+288>,
argc=5, argv=0x7fffffffe0f0) at perf.c:351
#13 0x00000000004613a2 in handle_internal_command (argc=5,
argv=0x7fffffffe0f0) at perf.c:404
#14 0x00000000004614f1 in run_argv (argcp=0x7fffffffdedc,
argv=0x7fffffffded0) at perf.c:448
#15 0x00000000004617d7 in main (argc=5, argv=0x7fffffffe0f0) at perf.c:556

Further investigation shows the range associated with userpg, to which
the statement at util/auxtrace.c:136 attempts to write to, is mapped
read-only:

(gdb) info proc mappings
...
0x00007ffff7cff000 0x00007ffff7d80000 0x81000            0x0
     r--s  anon_inode:[perf_event]
...

Compare to a run without --overwrite, where the memory is writable and
everything is good:

(gdb) info proc mappings
...
0x00007ffff7cff000 0x00007ffff7d80000 0x81000            0x0
     rw-s  anon_inode:[perf_event]
...

Let me know if you need any more information.

Tomas

Re: [BUG] perf segfaults when combining --overwrite and intel_pt event

[Leo Yan]

Hi Tomas,

On Mon, Apr 28, 2025 at 11:07:21AM +0200, Tomas Glozar wrote:
> Dear maintainers,
> 
> I would like to report a bug in perf I ran into when trying to combine
> --overwrite and intel_pt on latest perf from 6.15-rc4:

> $ perf record --overwrite -a -e intel_pt/cyc,noretcomp/k
> perf: Segmentation fault

[...]

> GDB gives the following backtrace:
> 
> (gdb) bt
> #0  0x000000000061bb16 in auxtrace_mmap__mmap (mm=0x7c9fa0,
> mp=0x7fffffff9008, userpg=0x7ffff7cff000, fd=7) at util/auxtrace.c:136

[...]

I can reproduce the issue with Arm CoreSight as well.  The cause is
the user page is mapped as read-only (see the mmap_per_evsel() function
in tools/lib/perf/evlist.c), afterwards AUX trace needs to update info
into it.

I would like to suggest a fix blow.  For overwrite mode, we need a
fixup to remap the user page with write permission.

Could you confirm if works at you side?

Thanks,
Leo

---8<---

From f969b3a7c3b338f47f271a2ac012813aec76b5a3 Mon Sep 17 00:00:00 2001
From: Leo Yan <leo.yan@arm.com>
Date: Mon, 28 Apr 2025 10:59:28 +0100
Subject: [PATCH] libperf: Grant write permission for user page

When perf runs in overwrite mode, the ring buffer is mapped read-only.

On the other hand, the first page in the ring buffer is for user page,
which is used for exchanging parameters between the kernel and the
userspace.  The read-only permission causes Segmentation fault with
command:

  $ perf record --overwrite -a -e cs_etm//
    perf: Segmentation fault

This patch grants write permission for the mapped user page so the
userspace tool can update info properly.

Signed-off-by: Leo Yan <leo.yan@arm.com>
---
 tools/lib/perf/mmap.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/tools/lib/perf/mmap.c b/tools/lib/perf/mmap.c
index c1a51d925e0e..465e12ec5a60 100644
--- a/tools/lib/perf/mmap.c
+++ b/tools/lib/perf/mmap.c
@@ -4,6 +4,7 @@
 #include <asm/bug.h>
 #include <errno.h>
 #include <string.h>
+#include <unistd.h>
 #include <linux/ring_buffer.h>
 #include <linux/perf_event.h>
 #include <perf/mmap.h>
@@ -36,6 +37,8 @@ size_t perf_mmap__mmap_len(struct perf_mmap *map)
 int perf_mmap__mmap(struct perf_mmap *map, struct perf_mmap_param *mp,
 		    int fd, struct perf_cpu cpu)
 {
+	const long page_sz = sysconf(_SC_PAGE_SIZE);
+
 	map->prev = 0;
 	map->mask = mp->mask;
 	map->base = mmap(NULL, perf_mmap__mmap_len(map), mp->prot,
@@ -45,6 +48,19 @@ int perf_mmap__mmap(struct perf_mmap *map, struct perf_mmap_param *mp,
 		return -1;
 	}
 
+	/*
+	 * In overwrite mode, pages are mapped read-only.  Fixup writable
+	 * permission for the user page as the tool needs to update info
+	 * into it.
+	 */
+	if (mp->prot == PROT_READ) {
+		if (mprotect(map->base, page_sz, mp->prot | PROT_WRITE) < 0) {
+			munmap(map->base, perf_mmap__mmap_len(map));
+			map->base = NULL;
+			return -1;
+		}
+	}
+
 	map->fd  = fd;
 	map->cpu = cpu;
 	return 0;
-- 
2.34.1

Re: [BUG] perf segfaults when combining --overwrite and intel_pt event

[Tomas Glozar]

po 28. 4. 2025 v 12:22 odesílatel Leo Yan <leo.yan@arm.com> napsal:
>
> Hi Tomas,
>

Hi Leo,

> I would like to suggest a fix blow.  For overwrite mode, we need a
> fixup to remap the user page with write permission.
>
> Could you confirm if works at you side?
>

Thank you for looking at this. The initial segfault on writing to
read-only page is fixed, however, perf now segfaults in a different
place:

(gdb) r
Starting program: /home/tglozar/dev/linux/tools/perf/perf record
--overwrite -a -e intel_pt/cyc,noretcomp/k sleep 5
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[Detaching after fork from child process 424037]
[New Thread 0x7ffff2df46c0 (LWP 424038)]
[ perf record: Woken up 1 times to write data ]
Thread 1 "perf" received signal SIGSEGV, Segmentation fault.
intel_pt_info_fill (itr=0x7a1e40, session=0x794ef0,
auxtrace_info=0x7cdfd0, priv_size=144) at arch/x86/util/intel-pt.c:361
361             pc = session->evlist->mmap[0].core.base;
(gdb) bt
#0  intel_pt_info_fill (itr=0x7a1e40, session=0x794ef0,
auxtrace_info=0x7cdfd0, priv_size=144) at arch/x86/util/intel-pt.c:361
#1  0x000000000061ce76 in auxtrace_record__info_fill (itr=0x7a1e40,
session=0x794ef0, auxtrace_info=0x7cdfd0, priv_size=144)
   at util/auxtrace.c:580
#2  0x000000000061ec48 in perf_event__synthesize_auxtrace_info
(itr=0x7a1e40, tool=0x765de0 <record>, session=0x794ef0,
   process=0x41a747 <process_synthesized_event>) at util/auxtrace.c:1320
#3  0x000000000041e869 in record__synthesize (rec=0x765de0 <record>,
tail=true) at builtin-record.c:2062
#4  0x0000000000420d04 in __cmd_record (rec=0x765de0 <record>, argc=2,
argv=0x7fffffffe0e0) at builtin-record.c:2841
#5  0x0000000000424025 in cmd_record (argc=2, argv=0x7fffffffe0e0) at
builtin-record.c:4285
#6  0x0000000000461143 in run_builtin (p=0x76a260 <commands+288>,
argc=7, argv=0x7fffffffe0e0) at perf.c:351
#7  0x00000000004613b2 in handle_internal_command (argc=7,
argv=0x7fffffffe0e0) at perf.c:404
#8  0x0000000000461501 in run_argv (argcp=0x7fffffffdecc,
argv=0x7fffffffdec0) at perf.c:448
#9  0x00000000004617e7 in main (argc=7, argv=0x7fffffffe0e0) at perf.c:556
(gdb) p session->evlist->mmap
$2 = (struct mmap *) 0x0

This appears to be a different bug.

Tomas

Re: [BUG] perf segfaults when combining --overwrite and intel_pt event

[Leo Yan]

On Mon, Apr 28, 2025 at 12:45:41PM +0200, Tomas Glozar wrote:

> Thank you for looking at this. The initial segfault on writing to
> read-only page is fixed,

Thanks a lot for testing, Tomas!

> however, perf now segfaults in a different place:


> (gdb) r
> Starting program: /home/tglozar/dev/linux/tools/perf/perf record
> --overwrite -a -e intel_pt/cyc,noretcomp/k sleep 5
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib64/libthread_db.so.1".
> [Detaching after fork from child process 424037]
> [New Thread 0x7ffff2df46c0 (LWP 424038)]
> [ perf record: Woken up 1 times to write data ]
> Thread 1 "perf" received signal SIGSEGV, Segmentation fault.
> intel_pt_info_fill (itr=0x7a1e40, session=0x794ef0,
> auxtrace_info=0x7cdfd0, priv_size=144) at arch/x86/util/intel-pt.c:361
> 361             pc = session->evlist->mmap[0].core.base;

Yes, this is a different issue, it would be fixed by:

diff --git a/tools/perf/arch/x86/util/intel-pt.c b/tools/perf/arch/x86/util/intel-pt.c
index 8f235d8b67b6..42f0dbfa973c 100644
--- a/tools/perf/arch/x86/util/intel-pt.c
+++ b/tools/perf/arch/x86/util/intel-pt.c
@@ -322,7 +322,7 @@ static int intel_pt_info_fill(struct auxtrace_record *itr,
 	struct intel_pt_recording *ptr =
 			container_of(itr, struct intel_pt_recording, itr);
 	struct perf_pmu *intel_pt_pmu = ptr->intel_pt_pmu;
-	struct perf_event_mmap_page *pc;
+	struct perf_event_mmap_page *pc = NULL;
 	struct perf_tsc_conversion tc = { .time_mult = 0, };
 	bool cap_user_time_zero = false, per_cpu_mmaps;
 	u64 tsc_bit, mtc_bit, mtc_freq_bits, cyc_bit, noretcomp_bit;
@@ -358,7 +358,11 @@ static int intel_pt_info_fill(struct auxtrace_record *itr,
 	if (!session->evlist->core.nr_mmaps)
 		return -EINVAL;
 
-	pc = session->evlist->mmap[0].core.base;
+	if (session->evlist->mmap)
+		pc = session->evlist->mmap[0].core.base;
+	else if (session->evlist->mmap_ovw)
+		pc = session->evlist->mmap_ovw[0].core.base;
+
 	if (pc) {
 		err = perf_read_tsc_conversion(pc, &tc);
 		if (err) {

I would leave this fix to the Intel-PT developers, who know this part
best.

Thanks,
Leo

Re: [BUG] perf segfaults when combining --overwrite and intel_pt event

[Tomas Glozar]

po 28. 4. 2025 v 14:47 odesílatel Leo Yan <leo.yan@arm.com> napsal:
> I would leave this fix to the Intel-PT developers, who know this part
> best.

Makes sense.

Just FYI, I tried applying the patch above, and after swapping
mmap_ovw for overwrite_mmap (which appears to be the correct name in
struct evlist) to make it build, the segfaults are gone:

$ perf record --overwrite -a -e intel_pt/cyc,noretcomp/k sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 40,533 MB perf.data

However, the data cannot be read:

$ perf report
0x7620 [0x30]: failed to process type: 71 [Invalid argument]
Error:
failed to process sample
# To display the perf.data header info, please use
--header/--header-only options.
#

Data generated without --overwrite is fine.

Tomas

Re: [BUG] perf segfaults when combining --overwrite and intel_pt event

[Leo Yan]

On Mon, Apr 28, 2025 at 04:16:58PM +0200, Tomas Glozar wrote:

[...]

> However, the data cannot be read:
> 
> $ perf report
> 0x7620 [0x30]: failed to process type: 71 [Invalid argument]

I can reproduce this issue, it is likely caused by failing parsing the
PERF_RECORD_AUXTRACE sample.  I will look into it.

Thanks for reporting the issue.

Leo