[PATCH 0/3] perf: Fix SIGCHLD vs pause() race with short-lived workloads

[PATCH 0/3] perf: Fix SIGCHLD vs pause() race with short-lived workloads

[Swapnil Sapkal]

Some perf subcommands (sched stats, lock contention) use the pattern
of forking a workload child, calling evlist__start_workload() to uncork
it, and then calling pause() to wait for a signal (typically SIGCHLD
when the child exits, or SIGINT/SIGTERM from the user).

This pattern has a race condition: if the workload is very short-lived,
the child can exit and deliver SIGCHLD in the window between
evlist__start_workload() and pause(). Since pause() only returns when a
signal is received *while the process is suspended*, and SIGCHLD has
already been delivered and handled by the empty sighandler(), pause()
blocks indefinitely.

The fix uses the standard POSIX pattern for this class of bug:

1. Block SIGCHLD (via sigprocmask) before starting the workload.
   If the child exits, the signal remains pending rather than being
   delivered and lost.

2. Replace pause() with sigsuspend(&oldmask), which atomically
   unblocks SIGCHLD and suspends the process. There is no window
   where the signal can slip through unnoticed.

3. Restore the original signal mask after sigsuspend() returns.

SIGINT and SIGTERM are not blocked at any point, so Ctrl+C and
graceful termination continue to work exactly as before.

Three call sites are affected across two files:
  - perf_sched__schedstat_record() in builtin-sched.c
  - perf_sched__schedstat_live()   in builtin-sched.c
  - __cmd_contention()             in builtin-lock.c

The two pause() sites in builtin-kwork.c are NOT affected because they
do not register SIGCHLD or fork workload children; they only wait for
user-initiated SIGINT/SIGTERM.

Swapnil Sapkal (3):
  perf sched stats: Fix SIGCHLD race in schedstat_record()
  perf sched stats: Fix SIGCHLD race in schedstat_live()
  perf lock contention: Fix SIGCHLD race in __cmd_contention()

 tools/perf/builtin-lock.c  | 20 ++++++++++++++++++--
 tools/perf/builtin-sched.c | 30 ++++++++++++++++++++++++++----
 2 files changed, 44 insertions(+), 6 deletions(-)

-- 
2.43.0

[PATCH 1/3] perf sched stats: Fix SIGCHLD race in schedstat_record()

[Swapnil Sapkal]

When a very short-lived workload is used with 'perf sched stats record',
the child process can exit and deliver SIGCHLD between
evlist__start_workload() and pause(). Since pause() only returns when a
signal is received while suspended, and the SIGCHLD has already been
delivered and handled by then, pause() blocks indefinitely.

Fix this by blocking SIGCHLD before starting the workload and replacing
pause() with sigsuspend(). sigsuspend() atomically unblocks SIGCHLD and
suspends the process, ensuring no signal is lost regardless of how
quickly the child exits.

Assisted-by: Claude:claude-opus-4.6
Signed-off-by: Swapnil Sapkal <swapnil.sapkal@amd.com>
---
 tools/perf/builtin-sched.c | 21 +++++++++++++++++++--
 1 file changed, 19 insertions(+), 2 deletions(-)

diff --git a/tools/perf/builtin-sched.c b/tools/perf/builtin-sched.c
index 3f509cfdd58c..eb3702d98fd1 100644
--- a/tools/perf/builtin-sched.c
+++ b/tools/perf/builtin-sched.c
@@ -3807,6 +3807,7 @@ const char *output_name;
 static int perf_sched__schedstat_record(struct perf_sched *sched,
 					int argc, const char **argv)
 {
+	sigset_t sigchld_mask, oldmask;
 	struct perf_session *session;
 	struct target target = {};
 	struct evlist *evlist;
@@ -3822,6 +3823,15 @@ static int perf_sched__schedstat_record(struct perf_sched *sched,
 	signal(SIGCHLD, sighandler);
 	signal(SIGTERM, sighandler);
 
+	/*
+	 * Block SIGCHLD early so that a short-lived workload cannot deliver
+	 * the signal before we are ready to wait for it. sigsuspend() below
+	 * will atomically unblock it.
+	 */
+	sigemptyset(&sigchld_mask);
+	sigaddset(&sigchld_mask, SIGCHLD);
+	sigprocmask(SIG_BLOCK, &sigchld_mask, &oldmask);
+
 	evlist = evlist__new();
 	if (!evlist)
 		return -ENOMEM;
@@ -3902,8 +3912,15 @@ static int perf_sched__schedstat_record(struct perf_sched *sched,
 	if (argc)
 		evlist__start_workload(evlist);
 
-	/* wait for signal */
-	pause();
+	/*
+	 * Use sigsuspend() instead of pause() to avoid a race where a
+	 * short-lived workload exits and delivers SIGCHLD before pause()
+	 * is entered, causing it to block indefinitely. sigsuspend()
+	 * atomically unblocks SIGCHLD (blocked above) and suspends,
+	 * ensuring no signal is lost.
+	 */
+	sigsuspend(&oldmask);
+	sigprocmask(SIG_SETMASK, &oldmask, NULL);
 
 	if (reset) {
 		err = disable_sched_schedstat();
-- 
2.43.0

Re: [PATCH 1/3] perf sched stats: Fix SIGCHLD race in schedstat_record()

[Ian Rogers]

On Tue, Mar 31, 2026 at 11:42 PM Swapnil Sapkal <swapnil.sapkal@amd.com> wrote:
>
> When a very short-lived workload is used with 'perf sched stats record',
> the child process can exit and deliver SIGCHLD between
> evlist__start_workload() and pause(). Since pause() only returns when a
> signal is received while suspended, and the SIGCHLD has already been
> delivered and handled by then, pause() blocks indefinitely.
>
> Fix this by blocking SIGCHLD before starting the workload and replacing
> pause() with sigsuspend(). sigsuspend() atomically unblocks SIGCHLD and
> suspends the process, ensuring no signal is lost regardless of how
> quickly the child exits.
>
> Assisted-by: Claude:claude-opus-4.6
> Signed-off-by: Swapnil Sapkal <swapnil.sapkal@amd.com>

Thanks Swapnil! In the Sashiko reviews there were some nits about
clean up on error paths but also 1 about signals potentially being
masked in the perf workload:
https://sashiko.dev/#/patchset/20260401064114.141066-1-swapnil.sapkal%40amd.com
Could you take a look?

Ian

> ---
>  tools/perf/builtin-sched.c | 21 +++++++++++++++++++--
>  1 file changed, 19 insertions(+), 2 deletions(-)
>
> diff --git a/tools/perf/builtin-sched.c b/tools/perf/builtin-sched.c
> index 3f509cfdd58c..eb3702d98fd1 100644
> --- a/tools/perf/builtin-sched.c
> +++ b/tools/perf/builtin-sched.c
> @@ -3807,6 +3807,7 @@ const char *output_name;
>  static int perf_sched__schedstat_record(struct perf_sched *sched,
>                                         int argc, const char **argv)
>  {
> +       sigset_t sigchld_mask, oldmask;
>         struct perf_session *session;
>         struct target target = {};
>         struct evlist *evlist;
> @@ -3822,6 +3823,15 @@ static int perf_sched__schedstat_record(struct perf_sched *sched,
>         signal(SIGCHLD, sighandler);
>         signal(SIGTERM, sighandler);
>
> +       /*
> +        * Block SIGCHLD early so that a short-lived workload cannot deliver
> +        * the signal before we are ready to wait for it. sigsuspend() below
> +        * will atomically unblock it.
> +        */
> +       sigemptyset(&sigchld_mask);
> +       sigaddset(&sigchld_mask, SIGCHLD);
> +       sigprocmask(SIG_BLOCK, &sigchld_mask, &oldmask);
> +
>         evlist = evlist__new();
>         if (!evlist)
>                 return -ENOMEM;
> @@ -3902,8 +3912,15 @@ static int perf_sched__schedstat_record(struct perf_sched *sched,
>         if (argc)
>                 evlist__start_workload(evlist);
>
> -       /* wait for signal */
> -       pause();
> +       /*
> +        * Use sigsuspend() instead of pause() to avoid a race where a
> +        * short-lived workload exits and delivers SIGCHLD before pause()
> +        * is entered, causing it to block indefinitely. sigsuspend()
> +        * atomically unblocks SIGCHLD (blocked above) and suspends,
> +        * ensuring no signal is lost.
> +        */
> +       sigsuspend(&oldmask);
> +       sigprocmask(SIG_SETMASK, &oldmask, NULL);
>
>         if (reset) {
>                 err = disable_sched_schedstat();
> --
> 2.43.0
>

Re: [PATCH 1/3] perf sched stats: Fix SIGCHLD race in schedstat_record()

[Swapnil Sapkal]

Hi Ian,

On 01-04-2026 21:56, Ian Rogers wrote:
> On Tue, Mar 31, 2026 at 11:42 PM Swapnil Sapkal <swapnil.sapkal@amd.com> wrote:
>>
>> When a very short-lived workload is used with 'perf sched stats record',
>> the child process can exit and deliver SIGCHLD between
>> evlist__start_workload() and pause(). Since pause() only returns when a
>> signal is received while suspended, and the SIGCHLD has already been
>> delivered and handled by then, pause() blocks indefinitely.
>>
>> Fix this by blocking SIGCHLD before starting the workload and replacing
>> pause() with sigsuspend(). sigsuspend() atomically unblocks SIGCHLD and
>> suspends the process, ensuring no signal is lost regardless of how
>> quickly the child exits.
>>
>> Assisted-by: Claude:claude-opus-4.6
>> Signed-off-by: Swapnil Sapkal <swapnil.sapkal@amd.com>
> 
> Thanks Swapnil! In the Sashiko reviews there were some nits about
> clean up on error paths but also 1 about signals potentially being
> masked in the perf workload:
> https://sashiko.dev/#/patchset/20260401064114.141066-1-swapnil.sapkal%40amd.com
> Could you take a look?
> 

Thank you for directing me to the sashiko reviews. I have addressed the 
review comments in v2.

https://lore.kernel.org/all/20260409162249.25581-1-swapnil.sapkal@amd.com/

--
Thanks and Regards,
Swapnil

> Ian
> 
>> ---
>>   tools/perf/builtin-sched.c | 21 +++++++++++++++++++--
>>   1 file changed, 19 insertions(+), 2 deletions(-)
>>
>> diff --git a/tools/perf/builtin-sched.c b/tools/perf/builtin-sched.c
>> index 3f509cfdd58c..eb3702d98fd1 100644
>> --- a/tools/perf/builtin-sched.c
>> +++ b/tools/perf/builtin-sched.c
>> @@ -3807,6 +3807,7 @@ const char *output_name;
>>   static int perf_sched__schedstat_record(struct perf_sched *sched,
>>                                          int argc, const char **argv)
>>   {
>> +       sigset_t sigchld_mask, oldmask;
>>          struct perf_session *session;
>>          struct target target = {};
>>          struct evlist *evlist;
>> @@ -3822,6 +3823,15 @@ static int perf_sched__schedstat_record(struct perf_sched *sched,
>>          signal(SIGCHLD, sighandler);
>>          signal(SIGTERM, sighandler);
>>
>> +       /*
>> +        * Block SIGCHLD early so that a short-lived workload cannot deliver
>> +        * the signal before we are ready to wait for it. sigsuspend() below
>> +        * will atomically unblock it.
>> +        */
>> +       sigemptyset(&sigchld_mask);
>> +       sigaddset(&sigchld_mask, SIGCHLD);
>> +       sigprocmask(SIG_BLOCK, &sigchld_mask, &oldmask);
>> +
>>          evlist = evlist__new();
>>          if (!evlist)
>>                  return -ENOMEM;
>> @@ -3902,8 +3912,15 @@ static int perf_sched__schedstat_record(struct perf_sched *sched,
>>          if (argc)
>>                  evlist__start_workload(evlist);
>>
>> -       /* wait for signal */
>> -       pause();
>> +       /*
>> +        * Use sigsuspend() instead of pause() to avoid a race where a
>> +        * short-lived workload exits and delivers SIGCHLD before pause()
>> +        * is entered, causing it to block indefinitely. sigsuspend()
>> +        * atomically unblocks SIGCHLD (blocked above) and suspends,
>> +        * ensuring no signal is lost.
>> +        */
>> +       sigsuspend(&oldmask);
>> +       sigprocmask(SIG_SETMASK, &oldmask, NULL);
>>
>>          if (reset) {
>>                  err = disable_sched_schedstat();
>> --
>> 2.43.0
>>

[PATCH 2/3] perf sched stats: Fix SIGCHLD race in schedstat_live()

[Swapnil Sapkal]

The signal race that exists in perf_sched__schedstat_record() also
affects perf_sched__schedstat_live(). A very short-lived workload can
exit and deliver SIGCHLD before pause() is entered, causing an
indefinite hang.

Apply the same fix: block SIGCHLD before starting the workload and
replace pause() with sigsuspend() to atomically unblock and wait.

Assisted-by: Claude:claude-opus-4.6
Signed-off-by: Swapnil Sapkal <swapnil.sapkal@amd.com>
---
 tools/perf/builtin-sched.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/tools/perf/builtin-sched.c b/tools/perf/builtin-sched.c
index eb3702d98fd1..1d8f4ceda1eb 100644
--- a/tools/perf/builtin-sched.c
+++ b/tools/perf/builtin-sched.c
@@ -4655,6 +4655,7 @@ static int perf_sched__schedstat_live(struct perf_sched *sched,
 				      int argc, const char **argv)
 {
 	struct cpu_domain_map **cd_map = NULL;
+	sigset_t sigchld_mask, oldmask;
 	struct target target = {};
 	u32 __maybe_unused md;
 	struct evlist *evlist;
@@ -4666,6 +4667,10 @@ static int perf_sched__schedstat_live(struct perf_sched *sched,
 	signal(SIGCHLD, sighandler);
 	signal(SIGTERM, sighandler);
 
+	sigemptyset(&sigchld_mask);
+	sigaddset(&sigchld_mask, SIGCHLD);
+	sigprocmask(SIG_BLOCK, &sigchld_mask, &oldmask);
+
 	evlist = evlist__new();
 	if (!evlist)
 		return -ENOMEM;
@@ -4707,8 +4712,8 @@ static int perf_sched__schedstat_live(struct perf_sched *sched,
 	if (argc)
 		evlist__start_workload(evlist);
 
-	/* wait for signal */
-	pause();
+	sigsuspend(&oldmask);
+	sigprocmask(SIG_SETMASK, &oldmask, NULL);
 
 	if (reset) {
 		err = disable_sched_schedstat();
-- 
2.43.0

[PATCH 3/3] perf lock contention: Fix SIGCHLD race in __cmd_contention()

[Swapnil Sapkal]

__cmd_contention() in builtin-lock.c has the same signal race condition
as the perf sched stats code paths. When running with a short-lived
workload via 'perf lock contention -- <cmd>', the child can exit and
deliver SIGCHLD before pause() is entered, causing an indefinite hang.

Fix this by blocking SIGCHLD before starting the workload and replacing
pause() with sigsuspend() to atomically unblock and wait.

Assisted-by: Claude:claude-opus-4.6
Signed-off-by: Swapnil Sapkal <swapnil.sapkal@amd.com>
---
 tools/perf/builtin-lock.c | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c
index e8962c985d34..1bada9833c93 100644
--- a/tools/perf/builtin-lock.c
+++ b/tools/perf/builtin-lock.c
@@ -1991,6 +1991,7 @@ static int check_lock_contention_options(const struct option *options,
 
 static int __cmd_contention(int argc, const char **argv)
 {
+	sigset_t sigchld_mask, oldmask;
 	int err = -EINVAL;
 	struct perf_tool eops;
 	struct perf_data data = {
@@ -2064,6 +2065,15 @@ static int __cmd_contention(int argc, const char **argv)
 		signal(SIGCHLD, sighandler);
 		signal(SIGTERM, sighandler);
 
+		/*
+		 * Block SIGCHLD early so that a short-lived workload
+		 * cannot deliver the signal before sigsuspend() is
+		 * entered below.
+		 */
+		sigemptyset(&sigchld_mask);
+		sigaddset(&sigchld_mask, SIGCHLD);
+		sigprocmask(SIG_BLOCK, &sigchld_mask, &oldmask);
+
 		con.evlist = evlist__new();
 		if (con.evlist == NULL) {
 			err = -ENOMEM;
@@ -2127,8 +2137,14 @@ static int __cmd_contention(int argc, const char **argv)
 		if (argc)
 			evlist__start_workload(con.evlist);
 
-		/* wait for signal */
-		pause();
+		/*
+		 * Use sigsuspend() instead of pause() to avoid a race
+		 * where a short-lived workload exits and delivers SIGCHLD
+		 * before pause() is entered. sigsuspend() atomically
+		 * unblocks SIGCHLD (blocked above) and suspends.
+		 */
+		sigsuspend(&oldmask);
+		sigprocmask(SIG_SETMASK, &oldmask, NULL);
 
 		lock_contention_stop();
 		lock_contention_read(&con);
-- 
2.43.0

Re: [PATCH 0/3] perf: Fix SIGCHLD vs pause() race with short-lived workloads

[James Clark]

On 01/04/2026 7:41 am, Swapnil Sapkal wrote:
> Some perf subcommands (sched stats, lock contention) use the pattern
> of forking a workload child, calling evlist__start_workload() to uncork
> it, and then calling pause() to wait for a signal (typically SIGCHLD
> when the child exits, or SIGINT/SIGTERM from the user).
> 
> This pattern has a race condition: if the workload is very short-lived,
> the child can exit and deliver SIGCHLD in the window between
> evlist__start_workload() and pause(). Since pause() only returns when a
> signal is received *while the process is suspended*, and SIGCHLD has
> already been delivered and handled by the empty sighandler(), pause()
> blocks indefinitely.
> 
> The fix uses the standard POSIX pattern for this class of bug:
> 
> 1. Block SIGCHLD (via sigprocmask) before starting the workload.
>     If the child exits, the signal remains pending rather than being
>     delivered and lost.
> 
> 2. Replace pause() with sigsuspend(&oldmask), which atomically
>     unblocks SIGCHLD and suspends the process. There is no window
>     where the signal can slip through unnoticed.
> 
> 3. Restore the original signal mask after sigsuspend() returns.
> 
> SIGINT and SIGTERM are not blocked at any point, so Ctrl+C and
> graceful termination continue to work exactly as before.
> 
> Three call sites are affected across two files:
>    - perf_sched__schedstat_record() in builtin-sched.c
>    - perf_sched__schedstat_live()   in builtin-sched.c
>    - __cmd_contention()             in builtin-lock.c
> 
> The two pause() sites in builtin-kwork.c are NOT affected because they
> do not register SIGCHLD or fork workload children; they only wait for
> user-initiated SIGINT/SIGTERM.
> 
> Swapnil Sapkal (3):
>    perf sched stats: Fix SIGCHLD race in schedstat_record()
>    perf sched stats: Fix SIGCHLD race in schedstat_live()
>    perf lock contention: Fix SIGCHLD race in __cmd_contention()
> 
>   tools/perf/builtin-lock.c  | 20 ++++++++++++++++++--
>   tools/perf/builtin-sched.c | 30 ++++++++++++++++++++++++++----
>   2 files changed, 44 insertions(+), 6 deletions(-)
> 

Reviewed-by: James Clark <james.clark@linaro.org>