* [PATCH v1 1/2] perf record: Fix teardown hang on system-wide multi-threaded sessions
@ 2026-07-10 5:39 Ian Rogers
2026-07-10 5:39 ` [PATCH v1 2/2] perf cap: If capability is missing still perform root test Ian Rogers
` (2 more replies)
0 siblings, 3 replies; 10+ messages in thread
From: Ian Rogers @ 2026-07-10 5:39 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Jiri Olsa, Ian Rogers, Adrian Hunter, James Clark,
Jiawei Sun, linux-kernel, linux-perf-users
Under system-wide (-a) parallel streaming mode (--threads=cpu),
background recording threads can be inundated by a continuous
firehose of hardware samples generated by the OS. In this state,
a background thread's local hit count remains unequal to its
sample count, causing it to bypass the blocking fdarray__poll()
call entirely on each iteration of its recording loop.
Because the termination check relies on the POLLHUP event status
populated specifically by fdarray__poll(), bypassing it prevents
the background thread from ever recognizing that its control pipe
was closed by the main thread. This traps the background thread
in an infinite recording loop, hanging the main thread indefinitely
as it awaits a termination acknowledgment that never arrives.
Ensure teardown completion by adding explicit evlist__disable()
calls in the main thread's cleanup paths at out_child: and
out_child_no_flush:. Additionally, patch fdarray__filter() to
respect the fdarray_flag__nonfilterable flag, preventing it
from incorrectly setting the background thread's control pipe
file descriptor to -1 and clearing its revents mask upon
processing termination POLLHUP signals.
Fixes: f94563fac269 ("perf record: fix poll storm when monitored threads exit")
Assisted-by: Antigravity:gemini-3.5-flash
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/lib/api/fd/array.c | 6 ++++--
tools/perf/builtin-record.c | 2 ++
2 files changed, 6 insertions(+), 2 deletions(-)
diff --git a/tools/lib/api/fd/array.c b/tools/lib/api/fd/array.c
index ffe8272af59b..16a047f1906e 100644
--- a/tools/lib/api/fd/array.c
+++ b/tools/lib/api/fd/array.c
@@ -115,6 +115,9 @@ int fdarray__filter(struct fdarray *fda, short revents,
return 0;
for (fd = 0; fd < fda->nr; ++fd) {
+ if (fda->priv[fd].flags & fdarray_flag__nonfilterable)
+ continue;
+
if (!fda->entries[fd].events)
continue;
@@ -132,8 +135,7 @@ int fdarray__filter(struct fdarray *fda, short revents,
continue;
}
- if (!(fda->priv[fd].flags & fdarray_flag__nonfilterable))
- ++nr;
+ ++nr;
}
return nr;
diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index ebd3ed0c9b3e..d1276382b77a 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -2890,11 +2890,13 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
record__synthesize_workload(rec, true);
out_child:
+ evlist__disable(rec->evlist);
record__stop_threads(rec);
record__mmap_read_all(rec, true);
goto out_free_threads;
out_child_no_flush:
/* mmap read already failed — retrying would just fail again */
+ evlist__disable(rec->evlist);
record__stop_threads(rec);
out_free_threads:
record__free_thread_data(rec);
--
2.55.0.795.g602f6c329a-goog
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH v1 2/2] perf cap: If capability is missing still perform root test
2026-07-10 5:39 [PATCH v1 1/2] perf record: Fix teardown hang on system-wide multi-threaded sessions Ian Rogers
@ 2026-07-10 5:39 ` Ian Rogers
2026-07-10 22:02 ` [PATCH v1 1/2] perf record: Fix teardown hang on system-wide multi-threaded sessions Namhyung Kim
2026-07-12 6:56 ` (subset) " Namhyung Kim
2 siblings, 0 replies; 10+ messages in thread
From: Ian Rogers @ 2026-07-10 5:39 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Jiri Olsa, Ian Rogers, Adrian Hunter, James Clark,
Jiawei Sun, linux-kernel, linux-perf-users
Refine the capability checking logic in perf_cap__capable() to in the
event the capability is missing do the fallback used_root
test. Previously this was only performed when the capability syscall
failed.
Fixes: e25ebda78e23 ("perf cap: Tidy up and improve capability testing")
Assisted-by: Antigravity:gemini-3.5-flash
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/util/cap.c | 14 ++++++++++++--
1 file changed, 12 insertions(+), 2 deletions(-)
diff --git a/tools/perf/util/cap.c b/tools/perf/util/cap.c
index ac6d1d9a523d..a2baecbc3145 100644
--- a/tools/perf/util/cap.c
+++ b/tools/perf/util/cap.c
@@ -20,6 +20,7 @@ bool perf_cap__capable(int cap, bool *used_root)
};
struct __user_cap_data_struct data[MAX_LINUX_CAPABILITY_U32S] = {};
__u32 cap_val;
+ bool has_cap = false;
*used_root = false;
while (syscall(SYS_capget, &header, &data[0]) == -1) {
@@ -37,12 +38,21 @@ bool perf_cap__capable(int cap, bool *used_root)
if (cap >= 32) {
if (header.version == _LINUX_CAPABILITY_VERSION_3) {
cap_val = data[1].effective;
+ has_cap = (cap_val & (1U << (cap & 0x1f))) != 0;
} else {
/* Capability beyond 32 is requested but only 32 are supported. */
- return false;
+ has_cap = false;
}
} else {
cap_val = data[0].effective;
+ has_cap = (cap_val & (1U << (cap & 0x1f))) != 0;
}
- return (cap_val & (1 << (cap & 0x1f))) != 0;
+
+ if (has_cap) {
+ *used_root = false;
+ return true;
+ }
+
+ *used_root = true;
+ return geteuid() == 0;
}
--
2.55.0.795.g602f6c329a-goog
^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH v1 1/2] perf record: Fix teardown hang on system-wide multi-threaded sessions
2026-07-10 5:39 [PATCH v1 1/2] perf record: Fix teardown hang on system-wide multi-threaded sessions Ian Rogers
2026-07-10 5:39 ` [PATCH v1 2/2] perf cap: If capability is missing still perform root test Ian Rogers
@ 2026-07-10 22:02 ` Namhyung Kim
2026-07-12 6:56 ` (subset) " Namhyung Kim
2 siblings, 0 replies; 10+ messages in thread
From: Namhyung Kim @ 2026-07-10 22:02 UTC (permalink / raw)
To: Ian Rogers
Cc: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Jiri Olsa,
Adrian Hunter, James Clark, Jiawei Sun, linux-kernel,
linux-perf-users
Hi Ian,
On Thu, Jul 09, 2026 at 10:39:06PM -0700, Ian Rogers wrote:
> Under system-wide (-a) parallel streaming mode (--threads=cpu),
> background recording threads can be inundated by a continuous
> firehose of hardware samples generated by the OS. In this state,
> a background thread's local hit count remains unequal to its
> sample count, causing it to bypass the blocking fdarray__poll()
> call entirely on each iteration of its recording loop.
>
> Because the termination check relies on the POLLHUP event status
> populated specifically by fdarray__poll(), bypassing it prevents
> the background thread from ever recognizing that its control pipe
> was closed by the main thread. This traps the background thread
> in an infinite recording loop, hanging the main thread indefinitely
> as it awaits a termination acknowledgment that never arrives.
>
> Ensure teardown completion by adding explicit evlist__disable()
> calls in the main thread's cleanup paths at out_child: and
> out_child_no_flush:. Additionally, patch fdarray__filter() to
> respect the fdarray_flag__nonfilterable flag, preventing it
> from incorrectly setting the background thread's control pipe
> file descriptor to -1 and clearing its revents mask upon
> processing termination POLLHUP signals.
>
> Fixes: f94563fac269 ("perf record: fix poll storm when monitored threads exit")
> Assisted-by: Antigravity:gemini-3.5-flash
> Signed-off-by: Ian Rogers <irogers@google.com>
I didn't know you already sent the fix so I was doing the same. It
seems your fix is better as it added evlist__disable(). I'll test this
and pick up if it's ok.
Thanks,
Namhyung
> ---
> tools/lib/api/fd/array.c | 6 ++++--
> tools/perf/builtin-record.c | 2 ++
> 2 files changed, 6 insertions(+), 2 deletions(-)
>
> diff --git a/tools/lib/api/fd/array.c b/tools/lib/api/fd/array.c
> index ffe8272af59b..16a047f1906e 100644
> --- a/tools/lib/api/fd/array.c
> +++ b/tools/lib/api/fd/array.c
> @@ -115,6 +115,9 @@ int fdarray__filter(struct fdarray *fda, short revents,
> return 0;
>
> for (fd = 0; fd < fda->nr; ++fd) {
> + if (fda->priv[fd].flags & fdarray_flag__nonfilterable)
> + continue;
> +
> if (!fda->entries[fd].events)
> continue;
>
> @@ -132,8 +135,7 @@ int fdarray__filter(struct fdarray *fda, short revents,
> continue;
> }
>
> - if (!(fda->priv[fd].flags & fdarray_flag__nonfilterable))
> - ++nr;
> + ++nr;
> }
>
> return nr;
> diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
> index ebd3ed0c9b3e..d1276382b77a 100644
> --- a/tools/perf/builtin-record.c
> +++ b/tools/perf/builtin-record.c
> @@ -2890,11 +2890,13 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
> record__synthesize_workload(rec, true);
>
> out_child:
> + evlist__disable(rec->evlist);
> record__stop_threads(rec);
> record__mmap_read_all(rec, true);
> goto out_free_threads;
> out_child_no_flush:
> /* mmap read already failed — retrying would just fail again */
> + evlist__disable(rec->evlist);
> record__stop_threads(rec);
> out_free_threads:
> record__free_thread_data(rec);
> --
> 2.55.0.795.g602f6c329a-goog
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: (subset) [PATCH v1 1/2] perf record: Fix teardown hang on system-wide multi-threaded sessions
2026-07-10 5:39 [PATCH v1 1/2] perf record: Fix teardown hang on system-wide multi-threaded sessions Ian Rogers
2026-07-10 5:39 ` [PATCH v1 2/2] perf cap: If capability is missing still perform root test Ian Rogers
2026-07-10 22:02 ` [PATCH v1 1/2] perf record: Fix teardown hang on system-wide multi-threaded sessions Namhyung Kim
@ 2026-07-12 6:56 ` Namhyung Kim
2026-07-16 7:37 ` [PATCH v2] perf record: Fix destructor invocation and event counting in fdarray__filter Ian Rogers
2 siblings, 1 reply; 10+ messages in thread
From: Namhyung Kim @ 2026-07-12 6:56 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Jiri Olsa,
Adrian Hunter, James Clark, Jiawei Sun, linux-kernel,
linux-perf-users, Ian Rogers
On Thu, 09 Jul 2026 22:39:06 -0700, Ian Rogers wrote:
> Under system-wide (-a) parallel streaming mode (--threads=cpu),
> background recording threads can be inundated by a continuous
> firehose of hardware samples generated by the OS. In this state,
> a background thread's local hit count remains unequal to its
> sample count, causing it to bypass the blocking fdarray__poll()
> call entirely on each iteration of its recording loop.
>
> [...]
Applied to perf-tools-next, thanks!
Best regards,
Namhyung
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH v2] perf record: Fix destructor invocation and event counting in fdarray__filter
2026-07-12 6:56 ` (subset) " Namhyung Kim
@ 2026-07-16 7:37 ` Ian Rogers
2026-07-23 5:04 ` [PATCH v3] " Ian Rogers
0 siblings, 1 reply; 10+ messages in thread
From: Ian Rogers @ 2026-07-16 7:37 UTC (permalink / raw)
To: namhyung, acme
Cc: abyssmystery, adrian.hunter, irogers, james.clark, jolsa,
linux-kernel, linux-perf-users, mingo, peterz
Fix the API-level fdarray filtering logic in fdarray__filter() following
the integration of the system-wide recording teardown fix. Rather than
bypassing the evaluation loop entirely for nonfilterable file
descriptors (which accidentally skips calling registered entry
destructors for control pipes and wakeup FDs, while creating poll storms
and reference leaks for system-wide events that also have nonfilterable
set), separate the lifecycle and masking logic.
Reorder the filtering sequence so that registered entry destructors are
safely executed first for all file descriptors. Then, utilize the
fdarray_flag__non_perf_event flag to perfectly distinguish and bypass
destructive file descriptor zeroing (.fd = -1) and revents mask
clearing for background control pipes and timers. This allows system-wide
event file descriptors to be correctly destroyed, zeroed, and filtered out upon
receiving POLLHUP/POLLERR signals without triggering busy loops, while safely
preserving control pipe teardown signaling and destructors.
Fixes: fb4751e79c45 ("perf record: Fix teardown hang on system-wide multi-threaded sessions")
Assisted-by: Antigravity:gemini-3.5-flash
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/lib/api/fd/array.c | 6 +++---
tools/perf/builtin-record.c | 4 +++-
2 files changed, 6 insertions(+), 4 deletions(-)
diff --git a/tools/lib/api/fd/array.c b/tools/lib/api/fd/array.c
index 16a047f1906e..92d9a3fb652c 100644
--- a/tools/lib/api/fd/array.c
+++ b/tools/lib/api/fd/array.c
@@ -115,9 +115,6 @@ int fdarray__filter(struct fdarray *fda, short revents,
return 0;
for (fd = 0; fd < fda->nr; ++fd) {
- if (fda->priv[fd].flags & fdarray_flag__nonfilterable)
- continue;
-
if (!fda->entries[fd].events)
continue;
@@ -125,6 +122,9 @@ int fdarray__filter(struct fdarray *fda, short revents,
if (entry_destructor)
entry_destructor(fda, fd, arg);
+ if (fda->priv[fd].flags & fdarray_flag__non_perf_event)
+ continue;
+
/*
* Set fd to -1 so poll() ignores this entry; otherwise
* POLLHUP/POLLERR are still reported for events=0 fds
diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index f58d7e3c7879..c6644dab1cfb 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -1242,7 +1242,9 @@ static int record__alloc_thread_data(struct record *rec, struct evlist *evlist)
goto out_free;
}
ret = fdarray__add(&thread_data[t].pollfd, thread_data[t].pipes.msg[0],
- POLLIN | POLLERR | POLLHUP, fdarray_flag__nonfilterable);
+ POLLIN | POLLERR | POLLHUP,
+ fdarray_flag__nonfilterable |
+ fdarray_flag__non_perf_event);
if (ret < 0) {
pr_err("Failed to add descriptor to thread[%d] pollfd\n", t);
goto out_free;
--
2.55.0.141.g00534a21ce-goog
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH v3] perf record: Fix destructor invocation and event counting in fdarray__filter
2026-07-16 7:37 ` [PATCH v2] perf record: Fix destructor invocation and event counting in fdarray__filter Ian Rogers
@ 2026-07-23 5:04 ` Ian Rogers
2026-08-17 23:04 ` [PATCH v4] perf fdarray: " Ian Rogers
0 siblings, 1 reply; 10+ messages in thread
From: Ian Rogers @ 2026-07-23 5:04 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: abyssmystery, adrian.hunter, james.clark, jolsa, linux-kernel,
linux-perf-users, mingo, peterz
When processing POLLHUP or POLLERR for an event in fdarray__filter, the
function invokes its destructor callback but fails to re-inject the
nonfilterable exclusion masks into the active event count (nr) increment
condition. Because auxiliary and control pipe descriptors never enter the
POLLHUP event match block, their active event count is unconditionally
incremented, preventing the return value from reaching zero on target
process exit and trapping __cmd_record() in an infinite drain poll hang.
Fix it by applying both fdarray_flag__nonfilterable and
fdarray_flag__non_perf_event exclusion masks into the active event
counter increment block at the bottom of the filtering loop.
Assisted-by: Antigravity:gemini-3.5-flash
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/lib/api/fd/array.c | 10 ++++++----
tools/perf/builtin-record.c | 4 +++-
2 files changed, 9 insertions(+), 5 deletions(-)
diff --git a/tools/lib/api/fd/array.c b/tools/lib/api/fd/array.c
index 67b73481df27..f97afbd6f07a 100644
--- a/tools/lib/api/fd/array.c
+++ b/tools/lib/api/fd/array.c
@@ -116,9 +116,6 @@ int fdarray__filter(struct fdarray *fda, short revents,
return 0;
for (fd = 0; fd < fda->nr; ++fd) {
- if (fda->priv[fd].flags & fdarray_flag__nonfilterable)
- continue;
-
if (!fda->entries[fd].events)
continue;
@@ -126,6 +123,9 @@ int fdarray__filter(struct fdarray *fda, short revents,
if (entry_destructor)
entry_destructor(fda, fd, arg);
+ if (fda->priv[fd].flags & fdarray_flag__non_perf_event)
+ continue;
+
/*
* Set fd to -1 so poll() ignores this entry; otherwise
* POLLHUP/POLLERR are still reported for events=0 fds
@@ -136,7 +136,9 @@ int fdarray__filter(struct fdarray *fda, short revents,
continue;
}
- ++nr;
+ if (!(fda->priv[fd].flags & fdarray_flag__nonfilterable) &&
+ !(fda->priv[fd].flags & fdarray_flag__non_perf_event))
+ ++nr;
}
return nr;
diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index f58d7e3c7879..c6644dab1cfb 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -1242,7 +1242,9 @@ static int record__alloc_thread_data(struct record *rec, struct evlist *evlist)
goto out_free;
}
ret = fdarray__add(&thread_data[t].pollfd, thread_data[t].pipes.msg[0],
- POLLIN | POLLERR | POLLHUP, fdarray_flag__nonfilterable);
+ POLLIN | POLLERR | POLLHUP,
+ fdarray_flag__nonfilterable |
+ fdarray_flag__non_perf_event);
if (ret < 0) {
pr_err("Failed to add descriptor to thread[%d] pollfd\n", t);
goto out_free;
--
2.55.0.229.g6434b31f56-goog
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH v4] perf fdarray: Fix destructor invocation and event counting in fdarray__filter
2026-07-23 5:04 ` [PATCH v3] " Ian Rogers
@ 2026-08-17 23:04 ` Ian Rogers
2026-08-17 23:17 ` Namhyung Kim
2026-08-18 0:31 ` [PATCH v5] perf record: Fix unhandled POLLHUP on non_perf_event descriptors Ian Rogers
0 siblings, 2 replies; 10+ messages in thread
From: Ian Rogers @ 2026-08-17 23:04 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: abyssmystery, adrian.hunter, james.clark, jolsa, linux-kernel,
linux-perf-users, mingo, peterz
When processing POLLHUP or POLLERR for an event in fdarray__filter, the
function invokes its destructor callback. However, the exact behavior
around unhandled POLLHUP events on control pipe descriptors caused
premature termination due to thread and evlist pollfd index mismatches.
This occurred because they were skipped by the early nonfilterable continue.
Address this by refining the early continue filter to only skip system-wide
perf events (which are nonfilterable but not non_perf_event). Control
descriptors (non_perf_event) now fall through to appropriately have their
fd value unset to -1 on POLLHUP while avoiding their destructors. Finally,
maintain the invariant that the active event counter (nr) increments strictly
and only for completely filterable events.
Fixes: fb4751e79c45 ("perf record: Fix teardown hang on system-wide multi-threaded sessions")
Assisted-by: Gemini:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/lib/api/fd/array.c | 16 +++++++++++++---
1 file changed, 13 insertions(+), 3 deletions(-)
diff --git a/tools/lib/api/fd/array.c b/tools/lib/api/fd/array.c
index 67b73481df27..3681ad7c6527 100644
--- a/tools/lib/api/fd/array.c
+++ b/tools/lib/api/fd/array.c
@@ -116,14 +116,23 @@ int fdarray__filter(struct fdarray *fda, short revents,
return 0;
for (fd = 0; fd < fda->nr; ++fd) {
- if (fda->priv[fd].flags & fdarray_flag__nonfilterable)
+ /*
+ * System-wide perf events are nonfilterable but not non_perf_event.
+ * We want to skip them entirely and never process revents on them.
+ */
+ if ((fda->priv[fd].flags & fdarray_flag__nonfilterable) &&
+ !(fda->priv[fd].flags & fdarray_flag__non_perf_event))
continue;
if (!fda->entries[fd].events)
continue;
if (fda->entries[fd].revents & revents) {
- if (entry_destructor)
+ /*
+ * Control descriptors are non_perf_event and don't need
+ * their perf-specific destructors triggered.
+ */
+ if (entry_destructor && !(fda->priv[fd].flags & fdarray_flag__non_perf_event))
entry_destructor(fda, fd, arg);
/*
@@ -136,7 +145,8 @@ int fdarray__filter(struct fdarray *fda, short revents,
continue;
}
- ++nr;
+ if (!(fda->priv[fd].flags & fdarray_flag__nonfilterable))
+ ++nr;
}
return nr;
--
2.55.0.699.gb54405d56f-goog
^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH v4] perf fdarray: Fix destructor invocation and event counting in fdarray__filter
2026-08-17 23:04 ` [PATCH v4] perf fdarray: " Ian Rogers
@ 2026-08-17 23:17 ` Namhyung Kim
2026-08-17 23:30 ` Ian Rogers
2026-08-18 0:31 ` [PATCH v5] perf record: Fix unhandled POLLHUP on non_perf_event descriptors Ian Rogers
1 sibling, 1 reply; 10+ messages in thread
From: Namhyung Kim @ 2026-08-17 23:17 UTC (permalink / raw)
To: Ian Rogers
Cc: acme, abyssmystery, adrian.hunter, james.clark, jolsa,
linux-kernel, linux-perf-users, mingo, peterz
On Mon, Aug 17, 2026 at 04:04:31PM -0700, Ian Rogers wrote:
> When processing POLLHUP or POLLERR for an event in fdarray__filter, the
> function invokes its destructor callback. However, the exact behavior
> around unhandled POLLHUP events on control pipe descriptors caused
> premature termination due to thread and evlist pollfd index mismatches.
> This occurred because they were skipped by the early nonfilterable continue.
Do you have a concrete scenario to check this behavior so that we can
verify the fix? It'd be great if we can add a test case.
Thanks,
Namhyung
>
> Address this by refining the early continue filter to only skip system-wide
> perf events (which are nonfilterable but not non_perf_event). Control
> descriptors (non_perf_event) now fall through to appropriately have their
> fd value unset to -1 on POLLHUP while avoiding their destructors. Finally,
> maintain the invariant that the active event counter (nr) increments strictly
> and only for completely filterable events.
>
> Fixes: fb4751e79c45 ("perf record: Fix teardown hang on system-wide multi-threaded sessions")
> Assisted-by: Gemini:gemini-3.1-pro
> Signed-off-by: Ian Rogers <irogers@google.com>
> ---
> tools/lib/api/fd/array.c | 16 +++++++++++++---
> 1 file changed, 13 insertions(+), 3 deletions(-)
>
> diff --git a/tools/lib/api/fd/array.c b/tools/lib/api/fd/array.c
> index 67b73481df27..3681ad7c6527 100644
> --- a/tools/lib/api/fd/array.c
> +++ b/tools/lib/api/fd/array.c
> @@ -116,14 +116,23 @@ int fdarray__filter(struct fdarray *fda, short revents,
> return 0;
>
> for (fd = 0; fd < fda->nr; ++fd) {
> - if (fda->priv[fd].flags & fdarray_flag__nonfilterable)
> + /*
> + * System-wide perf events are nonfilterable but not non_perf_event.
> + * We want to skip them entirely and never process revents on them.
> + */
> + if ((fda->priv[fd].flags & fdarray_flag__nonfilterable) &&
> + !(fda->priv[fd].flags & fdarray_flag__non_perf_event))
> continue;
>
> if (!fda->entries[fd].events)
> continue;
>
> if (fda->entries[fd].revents & revents) {
> - if (entry_destructor)
> + /*
> + * Control descriptors are non_perf_event and don't need
> + * their perf-specific destructors triggered.
> + */
> + if (entry_destructor && !(fda->priv[fd].flags & fdarray_flag__non_perf_event))
> entry_destructor(fda, fd, arg);
>
> /*
> @@ -136,7 +145,8 @@ int fdarray__filter(struct fdarray *fda, short revents,
> continue;
> }
>
> - ++nr;
> + if (!(fda->priv[fd].flags & fdarray_flag__nonfilterable))
> + ++nr;
> }
>
> return nr;
> --
> 2.55.0.699.gb54405d56f-goog
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v4] perf fdarray: Fix destructor invocation and event counting in fdarray__filter
2026-08-17 23:17 ` Namhyung Kim
@ 2026-08-17 23:30 ` Ian Rogers
0 siblings, 0 replies; 10+ messages in thread
From: Ian Rogers @ 2026-08-17 23:30 UTC (permalink / raw)
To: Namhyung Kim
Cc: acme, abyssmystery, adrian.hunter, james.clark, jolsa,
linux-kernel, linux-perf-users, mingo, peterz
On Mon, Aug 17, 2026 at 4:17 PM Namhyung Kim <namhyung@kernel.org> wrote:
>
> On Mon, Aug 17, 2026 at 04:04:31PM -0700, Ian Rogers wrote:
> > When processing POLLHUP or POLLERR for an event in fdarray__filter, the
> > function invokes its destructor callback. However, the exact behavior
> > around unhandled POLLHUP events on control pipe descriptors caused
> > premature termination due to thread and evlist pollfd index mismatches.
> > This occurred because they were skipped by the early nonfilterable continue.
>
> Do you have a concrete scenario to check this behavior so that we can
> verify the fix? It'd be great if we can add a test case.
So the fix is trying to address hangs I see in the TPEBS test. We
should be able to add a C unit test. I can do that while I wrangle
with Sashiko.
Thanks,
Ian
> Thanks,
> Namhyung
>
> >
> > Address this by refining the early continue filter to only skip system-wide
> > perf events (which are nonfilterable but not non_perf_event). Control
> > descriptors (non_perf_event) now fall through to appropriately have their
> > fd value unset to -1 on POLLHUP while avoiding their destructors. Finally,
> > maintain the invariant that the active event counter (nr) increments strictly
> > and only for completely filterable events.
> >
> > Fixes: fb4751e79c45 ("perf record: Fix teardown hang on system-wide multi-threaded sessions")
> > Assisted-by: Gemini:gemini-3.1-pro
> > Signed-off-by: Ian Rogers <irogers@google.com>
> > ---
> > tools/lib/api/fd/array.c | 16 +++++++++++++---
> > 1 file changed, 13 insertions(+), 3 deletions(-)
> >
> > diff --git a/tools/lib/api/fd/array.c b/tools/lib/api/fd/array.c
> > index 67b73481df27..3681ad7c6527 100644
> > --- a/tools/lib/api/fd/array.c
> > +++ b/tools/lib/api/fd/array.c
> > @@ -116,14 +116,23 @@ int fdarray__filter(struct fdarray *fda, short revents,
> > return 0;
> >
> > for (fd = 0; fd < fda->nr; ++fd) {
> > - if (fda->priv[fd].flags & fdarray_flag__nonfilterable)
> > + /*
> > + * System-wide perf events are nonfilterable but not non_perf_event.
> > + * We want to skip them entirely and never process revents on them.
> > + */
> > + if ((fda->priv[fd].flags & fdarray_flag__nonfilterable) &&
> > + !(fda->priv[fd].flags & fdarray_flag__non_perf_event))
> > continue;
> >
> > if (!fda->entries[fd].events)
> > continue;
> >
> > if (fda->entries[fd].revents & revents) {
> > - if (entry_destructor)
> > + /*
> > + * Control descriptors are non_perf_event and don't need
> > + * their perf-specific destructors triggered.
> > + */
> > + if (entry_destructor && !(fda->priv[fd].flags & fdarray_flag__non_perf_event))
> > entry_destructor(fda, fd, arg);
> >
> > /*
> > @@ -136,7 +145,8 @@ int fdarray__filter(struct fdarray *fda, short revents,
> > continue;
> > }
> >
> > - ++nr;
> > + if (!(fda->priv[fd].flags & fdarray_flag__nonfilterable))
> > + ++nr;
> > }
> >
> > return nr;
> > --
> > 2.55.0.699.gb54405d56f-goog
> >
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH v5] perf record: Fix unhandled POLLHUP on non_perf_event descriptors
2026-08-17 23:04 ` [PATCH v4] perf fdarray: " Ian Rogers
2026-08-17 23:17 ` Namhyung Kim
@ 2026-08-18 0:31 ` Ian Rogers
1 sibling, 0 replies; 10+ messages in thread
From: Ian Rogers @ 2026-08-18 0:31 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: abyssmystery, adrian.hunter, james.clark, jolsa, linux-kernel,
linux-perf-users, mingo, peterz
When processing POLLHUP or POLLERR for an event in fdarray__filter, the
function's prior iterations incorrectly mutated non_perf_event control
descriptors by zeroing their events when a POLLHUP occurred. This caused
premature termination due to index mismatches or hangs since the core
evlist logic was prevented from safely finalizing the poll array setup via
evlist__ctlfd_process().
Revert the logic in fdarray__filter to cleanly bypass all nonfilterable
events as introduced by fb4751e79c45. Instead, fix the underlying logic
within record__update_evlist_pollfd_from_thread() in builtin-record.c to
sustainably reflect teardown statuses (-1) originating from
evlist__finalize_ctlfd(). This correctly mirrors the finalized state to the
thread's poll structure avoiding both invalid -EINVAL index crashes and
POLLHUP spin loops.
Included is a unit test to enforce that fdarray_flag__nonfilterable items
are accurately completely circumvented during revents filtering loops.
Fixes: fb4751e79c45 ("perf record: Fix teardown hang on system-wide multi-threaded sessions")
Assisted-by: Gemini:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/builtin-record.c | 10 ++++++++++
tools/perf/tests/fdarray.c | 24 ++++++++++++++++++++++++
2 files changed, 34 insertions(+)
diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index a57987851cf0..d7c083803029 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -1169,6 +1169,16 @@ static int record__update_evlist_pollfd_from_thread(struct record *rec,
int e_pos = rec->index_map[i].evlist_pollfd_index;
int t_pos = rec->index_map[i].thread_pollfd_index;
+ if (e_entries[e_pos].fd == -1 || e_entries[e_pos].events == 0) {
+ /*
+ * e_entries might have been finalized by evlist__finalize_ctlfd().
+ * We must propagate it to t_entries to avoid index mismatches
+ * and to prevent a poll storm on the next iteration.
+ */
+ t_entries[t_pos].fd = -1;
+ t_entries[t_pos].events = 0;
+ }
+
if (e_entries[e_pos].fd != t_entries[t_pos].fd ||
e_entries[e_pos].events != t_entries[t_pos].events) {
pr_err("Thread and evlist pollfd index mismatch\n");
diff --git a/tools/perf/tests/fdarray.c b/tools/perf/tests/fdarray.c
index 40983c3574b1..2d3db7b754a1 100644
--- a/tools/perf/tests/fdarray.c
+++ b/tools/perf/tests/fdarray.c
@@ -80,6 +80,30 @@ static int test__fdarray__filter(struct test_suite *test __maybe_unused, int sub
goto out_delete;
}
+ fdarray__init_revents(fda, POLLHUP);
+ fda->priv[2].flags = fdarray_flag__nonfilterable;
+
+ pr_debug("\nfiltering all but fda->entries[2] (nonfilterable):");
+ fdarray__fprintf_prefix(fda, "before", stderr);
+ nr_fds = fdarray__filter(fda, POLLHUP, NULL, NULL);
+ fdarray__fprintf_prefix(fda, " after", stderr);
+
+ if (nr_fds != 0) {
+ pr_debug("\nfdarray__filter()=%d != 0, should be 0\n",
+ nr_fds);
+ goto out_delete;
+ }
+ if (fda->entries[2].fd == -1) {
+ pr_debug("\nfdarray__filter() illegally modified nonfilterable fd!");
+ goto out_delete;
+ }
+ if (fda->entries[2].revents != POLLHUP) {
+ pr_debug("\nfdarray__filter() illegally modified nonfilterable revents!");
+ goto out_delete;
+ }
+
+ fda->priv[2].flags = 0; /* reset flags */
+
pr_debug("\n");
err = 0;
--
2.55.0.699.gb54405d56f-goog
^ permalink raw reply related [flat|nested] 10+ messages in thread
end of thread, other threads:[~2026-08-18 0:31 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-07-10 5:39 [PATCH v1 1/2] perf record: Fix teardown hang on system-wide multi-threaded sessions Ian Rogers
2026-07-10 5:39 ` [PATCH v1 2/2] perf cap: If capability is missing still perform root test Ian Rogers
2026-07-10 22:02 ` [PATCH v1 1/2] perf record: Fix teardown hang on system-wide multi-threaded sessions Namhyung Kim
2026-07-12 6:56 ` (subset) " Namhyung Kim
2026-07-16 7:37 ` [PATCH v2] perf record: Fix destructor invocation and event counting in fdarray__filter Ian Rogers
2026-07-23 5:04 ` [PATCH v3] " Ian Rogers
2026-08-17 23:04 ` [PATCH v4] perf fdarray: " Ian Rogers
2026-08-17 23:17 ` Namhyung Kim
2026-08-17 23:30 ` Ian Rogers
2026-08-18 0:31 ` [PATCH v5] perf record: Fix unhandled POLLHUP on non_perf_event descriptors Ian Rogers
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox