[6.1.y][6.6.y] uprobes: Fix race in uprobe_free_utask

[6.1.y][6.6.y] uprobes: Fix race in uprobe_free_utask

[Christian Simon]

Dear stable team,

I noticed that cfa7f3d2c526 ("perf,x86: avoid missing caller address in stack traces captured in uprobe"), got backported into 6.1.113 and 6.6.55, but it contains a race condition which Jiri Olsa has fixed in mainline b583ef82b671 ("uprobes: Fix race in uprobe_free_utask"). I think this should be backported into those stable branches.

#regzbot title: uprobes: Fix race in uprobe_free_utask
#regzbot introduced: cfa7f3d2c526
#regzbot link: https://lore.kernel.org/all/20250109141440.2692173-1-jolsa@kernel.org/

Link: https://lore.kernel.org/all/20250109141440.2692173-1-jolsa@kernel.org/

Note: Sorry if I am using the wrong process/form/format, I tried to follow "Option 2" from the stable-kernel-rules.html.

Best regards,
Christian

Re: [6.1.y][6.6.y] uprobes: Fix race in uprobe_free_utask

[Greg KH]

On Tue, Mar 04, 2025 at 10:33:24AM +0000, Christian Simon wrote:
> Dear stable team,
> 
> I noticed that cfa7f3d2c526 ("perf,x86: avoid missing caller address in stack traces captured in uprobe"), got backported into 6.1.113 and 6.6.55, but it contains a race condition which Jiri Olsa has fixed in mainline b583ef82b671 ("uprobes: Fix race in uprobe_free_utask"). I think this should be backported into those stable branches.
> 
> #regzbot title: uprobes: Fix race in uprobe_free_utask
> #regzbot introduced: cfa7f3d2c526
> #regzbot link: https://lore.kernel.org/all/20250109141440.2692173-1-jolsa@kernel.org/
> 
> Link: https://lore.kernel.org/all/20250109141440.2692173-1-jolsa@kernel.org/
> 
> Note: Sorry if I am using the wrong process/form/format, I tried to follow "Option 2" from the stable-kernel-rules.html.

As the patch does not apply cleanly to those kernels, how did you test
this?

Can you provide some working, and tested, patches for this?  We'll be
glad to queue them up if so.

thanks,

greg k-h

[PATCH 6.6.y] uprobes: Fix race in uprobe_free_utask

[Christian Simon]

commit b583ef82b671c9a752fbe3e95bd4c1c51eab764d upstream.

Christian Simon verified the regression exists in v6.6.80 as per method
below and backported the mainline fix to the older version of
uprobe_free_utask. After that change I can no longer reproduce
the race with this method within 1h, while before it would show the
panic under a minute.

Max Makarov reported kernel panic [1] in perf user callchain code.

The reason for that is the race between uprobe_free_utask and bpf
profiler code doing the perf user stack unwind and is triggered
within uprobe_free_utask function:
  - after current->utask is freed and
  - before current->utask is set to NULL

 general protection fault, probably for non-canonical address 0x9e759c37ee555c76: 0000 [#1] SMP PTI
 RIP: 0010:is_uprobe_at_func_entry+0x28/0x80
 ...
  ? die_addr+0x36/0x90
  ? exc_general_protection+0x217/0x420
  ? asm_exc_general_protection+0x26/0x30
  ? is_uprobe_at_func_entry+0x28/0x80
  perf_callchain_user+0x20a/0x360
  get_perf_callchain+0x147/0x1d0
  bpf_get_stackid+0x60/0x90
  bpf_prog_9aac297fb833e2f5_do_perf_event+0x434/0x53b
  ? __smp_call_single_queue+0xad/0x120
  bpf_overflow_handler+0x75/0x110
  ...
  asm_sysvec_apic_timer_interrupt+0x1a/0x20
 RIP: 0010:__kmem_cache_free+0x1cb/0x350
 ...
  ? uprobe_free_utask+0x62/0x80
  ? acct_collect+0x4c/0x220
  uprobe_free_utask+0x62/0x80
  mm_release+0x12/0xb0
  do_exit+0x26b/0xaa0
  __x64_sys_exit+0x1b/0x20
  do_syscall_64+0x5a/0x80

It can be easily reproduced by running following commands in
separate terminals:

  # while :; do bpftrace -e 'uprobe:/bin/ls:_start  { printf("hit\n"); }' -c ls; done
  # bpftrace -e 'profile:hz:100000 { @[ustack()] = count(); }'

Fixing this by making sure current->utask pointer is set to NULL
before we start to release the utask object.

[1] https://github.com/grafana/pyroscope/issues/3673

Fixes: cfa7f3d2c526 ("perf,x86: avoid missing caller address in stack traces captured in uprobe")
Reported-by: Max Makarov <maxpain@linux.com>
(cherry picked from commit b583ef82b671c9a752fbe3e95bd4c1c51eab764d)
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Oleg Nesterov <oleg@redhat.com>
Acked-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/r/20250109141440.2692173-1-jolsa@kernel.org
[Christian Simon: Rebased for 6.6.y, due to mainline change https://lore.kernel.org/all/20240929144239.GA9475@redhat.com/]
Signed-off-by: Christian Simon <simon@swine.de>
---
 kernel/events/uprobes.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c
index 6dac0b579821..70fae75c48ca 100644
--- a/kernel/events/uprobes.c
+++ b/kernel/events/uprobes.c
@@ -1716,6 +1716,7 @@ void uprobe_free_utask(struct task_struct *t)
 	if (!utask)
 		return;
 
+	t->utask = NULL;
 	if (utask->active_uprobe)
 		put_uprobe(utask->active_uprobe);
 
@@ -1725,7 +1726,6 @@ void uprobe_free_utask(struct task_struct *t)
 
 	xol_free_insn_slot(t);
 	kfree(utask);
-	t->utask = NULL;
 }
 
 /*
-- 
2.47.1

Re: [PATCH 6.6.y] uprobes: Fix race in uprobe_free_utask

[Sasha Levin]

[ Sasha's backport helper bot ]

Hi,

✅ All tests passed successfully. No issues detected.
No action required from the submitter.

The upstream commit SHA1 provided is correct: b583ef82b671c9a752fbe3e95bd4c1c51eab764d

WARNING: Author mismatch between patch and upstream commit:
Backport author: Christian Simon<simon@swine.de>
Commit author: Jiri Olsa<jolsa@kernel.org>

Note: The patch differs from the upstream commit:
---
1:  b583ef82b671c ! 1:  c65c252796df6 uprobes: Fix race in uprobe_free_utask
    @@
      ## Metadata ##
    -Author: Jiri Olsa <jolsa@kernel.org>
    +Author: Christian Simon <simon@swine.de>
     
      ## Commit message ##
         uprobes: Fix race in uprobe_free_utask
     
    +    commit b583ef82b671c9a752fbe3e95bd4c1c51eab764d upstream.
    +
    +    Christian Simon verified the regression exists in v6.6.80 as per method
    +    below and backported the mainline fix to the older version of
    +    uprobe_free_utask. After that change I can no longer reproduce
    +    the race with this method within 1h, while before it would show the
    +    panic under a minute.
    +
         Max Makarov reported kernel panic [1] in perf user callchain code.
     
         The reason for that is the race between uprobe_free_utask and bpf
    @@ Commit message
     
         Fixes: cfa7f3d2c526 ("perf,x86: avoid missing caller address in stack traces captured in uprobe")
         Reported-by: Max Makarov <maxpain@linux.com>
    +    (cherry picked from commit b583ef82b671c9a752fbe3e95bd4c1c51eab764d)
         Signed-off-by: Jiri Olsa <jolsa@kernel.org>
         Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
         Acked-by: Oleg Nesterov <oleg@redhat.com>
         Acked-by: Andrii Nakryiko <andrii@kernel.org>
         Link: https://lore.kernel.org/r/20250109141440.2692173-1-jolsa@kernel.org
    +    [Christian Simon: Rebased for 6.6.y, due to mainline change https://lore.kernel.org/all/20240929144239.GA9475@redhat.com/]
    +    Signed-off-by: Christian Simon <simon@swine.de>
     
      ## kernel/events/uprobes.c ##
     @@ kernel/events/uprobes.c: void uprobe_free_utask(struct task_struct *t)
    @@ kernel/events/uprobes.c: void uprobe_free_utask(struct task_struct *t)
      		return;
      
     +	t->utask = NULL;
    - 	WARN_ON_ONCE(utask->active_uprobe || utask->xol_vaddr);
    + 	if (utask->active_uprobe)
    + 		put_uprobe(utask->active_uprobe);
      
    - 	timer_delete_sync(&utask->ri_timer);
     @@ kernel/events/uprobes.c: void uprobe_free_utask(struct task_struct *t)
    - 		ri = free_ret_instance(ri, true /* cleanup_hprobe */);
      
    + 	xol_free_insn_slot(t);
      	kfree(utask);
     -	t->utask = NULL;
      }
      
    - #define RI_TIMER_PERIOD (HZ / 10) /* 100 ms */
    + /*
---

Results of testing on various branches:

| Branch                    | Patch Apply | Build Test |
|---------------------------|-------------|------------|
| stable/linux-6.6.y        |  Success    |  Success   |

[PATCH 6.1.y] uprobes: Fix race in uprobe_free_utask

[Christian Simon]

commit b583ef82b671c9a752fbe3e95bd4c1c51eab764d upstream.

Christian Simon verified the regression exists in v6.1.129 as per method
below and backported the mainline fix to the older version of
uprobe_free_utask. After that change I can no longer reproduce
the race with this method within 12 hours, while before it would
show the panic in under a minute.

Max Makarov reported kernel panic [1] in perf user callchain code.

The reason for that is the race between uprobe_free_utask and bpf
profiler code doing the perf user stack unwind and is triggered
within uprobe_free_utask function:
  - after current->utask is freed and
  - before current->utask is set to NULL

 general protection fault, probably for non-canonical address 0x9e759c37ee555c76: 0000 [#1] SMP PTI
 RIP: 0010:is_uprobe_at_func_entry+0x28/0x80
 ...
  ? die_addr+0x36/0x90
  ? exc_general_protection+0x217/0x420
  ? asm_exc_general_protection+0x26/0x30
  ? is_uprobe_at_func_entry+0x28/0x80
  perf_callchain_user+0x20a/0x360
  get_perf_callchain+0x147/0x1d0
  bpf_get_stackid+0x60/0x90
  bpf_prog_9aac297fb833e2f5_do_perf_event+0x434/0x53b
  ? __smp_call_single_queue+0xad/0x120
  bpf_overflow_handler+0x75/0x110
  ...
  asm_sysvec_apic_timer_interrupt+0x1a/0x20
 RIP: 0010:__kmem_cache_free+0x1cb/0x350
 ...
  ? uprobe_free_utask+0x62/0x80
  ? acct_collect+0x4c/0x220
  uprobe_free_utask+0x62/0x80
  mm_release+0x12/0xb0
  do_exit+0x26b/0xaa0
  __x64_sys_exit+0x1b/0x20
  do_syscall_64+0x5a/0x80

It can be easily reproduced by running following commands in
separate terminals:

  # while :; do bpftrace -e 'uprobe:/bin/ls:_start  { printf("hit\n"); }' -c ls; done
  # bpftrace -e 'profile:hz:100000 { @[ustack()] = count(); }'

Fixing this by making sure current->utask pointer is set to NULL
before we start to release the utask object.

[1] https://github.com/grafana/pyroscope/issues/3673

Fixes: cfa7f3d2c526 ("perf,x86: avoid missing caller address in stack traces captured in uprobe")
Reported-by: Max Makarov <maxpain@linux.com>
(cherry picked from commit b583ef82b671c9a752fbe3e95bd4c1c51eab764d)
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Oleg Nesterov <oleg@redhat.com>
Acked-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/r/20250109141440.2692173-1-jolsa@kernel.org
[Christian Simon: Rebased for 6.1.y, due to mainline change https://lore.kernel.org/all/20240929144239.GA9475@redhat.com/]
Signed-off-by: Christian Simon <simon@swine.de>
---
 kernel/events/uprobes.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c
index 9ee25351ceca..e9f9a669f80e 100644
--- a/kernel/events/uprobes.c
+++ b/kernel/events/uprobes.c
@@ -1719,6 +1719,7 @@ void uprobe_free_utask(struct task_struct *t)
 	if (!utask)
 		return;
 
+	t->utask = NULL;
 	if (utask->active_uprobe)
 		put_uprobe(utask->active_uprobe);
 
@@ -1728,7 +1729,6 @@ void uprobe_free_utask(struct task_struct *t)
 
 	xol_free_insn_slot(t);
 	kfree(utask);
-	t->utask = NULL;
 }
 
 /*
-- 
2.47.1

Re: [PATCH 6.1.y] uprobes: Fix race in uprobe_free_utask

[Sasha Levin]

[ Sasha's backport helper bot ]

Hi,

Summary of potential issues:
ℹ️ Patch is missing in 6.13.y (ignore if backport was sent)
⚠️ Commit missing in all newer stable branches

The upstream commit SHA1 provided is correct: b583ef82b671c9a752fbe3e95bd4c1c51eab764d

WARNING: Author mismatch between patch and upstream commit:
Backport author: Christian Simon<simon@swine.de>
Commit author: Jiri Olsa<jolsa@kernel.org>

Status in newer kernel trees:
6.13.y | Present (exact SHA1)
6.12.y | Not found
6.6.y | Not found

Note: The patch differs from the upstream commit:
---
1:  b583ef82b671c ! 1:  9f50b10df7c63 uprobes: Fix race in uprobe_free_utask
    @@
      ## Metadata ##
    -Author: Jiri Olsa <jolsa@kernel.org>
    +Author: Christian Simon <simon@swine.de>
     
      ## Commit message ##
         uprobes: Fix race in uprobe_free_utask
     
    +    commit b583ef82b671c9a752fbe3e95bd4c1c51eab764d upstream.
    +
    +    Christian Simon verified the regression exists in v6.1.129 as per method
    +    below and backported the mainline fix to the older version of
    +    uprobe_free_utask. After that change I can no longer reproduce
    +    the race with this method within 12 hours, while before it would
    +    show the panic in under a minute.
    +
         Max Makarov reported kernel panic [1] in perf user callchain code.
     
         The reason for that is the race between uprobe_free_utask and bpf
    @@ Commit message
     
         Fixes: cfa7f3d2c526 ("perf,x86: avoid missing caller address in stack traces captured in uprobe")
         Reported-by: Max Makarov <maxpain@linux.com>
    +    (cherry picked from commit b583ef82b671c9a752fbe3e95bd4c1c51eab764d)
         Signed-off-by: Jiri Olsa <jolsa@kernel.org>
         Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
         Acked-by: Oleg Nesterov <oleg@redhat.com>
         Acked-by: Andrii Nakryiko <andrii@kernel.org>
         Link: https://lore.kernel.org/r/20250109141440.2692173-1-jolsa@kernel.org
    +    [Christian Simon: Rebased for 6.1.y, due to mainline change https://lore.kernel.org/all/20240929144239.GA9475@redhat.com/]
    +    Signed-off-by: Christian Simon <simon@swine.de>
     
      ## kernel/events/uprobes.c ##
     @@ kernel/events/uprobes.c: void uprobe_free_utask(struct task_struct *t)
    @@ kernel/events/uprobes.c: void uprobe_free_utask(struct task_struct *t)
      		return;
      
     +	t->utask = NULL;
    - 	WARN_ON_ONCE(utask->active_uprobe || utask->xol_vaddr);
    + 	if (utask->active_uprobe)
    + 		put_uprobe(utask->active_uprobe);
      
    - 	timer_delete_sync(&utask->ri_timer);
     @@ kernel/events/uprobes.c: void uprobe_free_utask(struct task_struct *t)
    - 		ri = free_ret_instance(ri, true /* cleanup_hprobe */);
      
    + 	xol_free_insn_slot(t);
      	kfree(utask);
     -	t->utask = NULL;
      }
      
    - #define RI_TIMER_PERIOD (HZ / 10) /* 100 ms */
    + /*
---

Results of testing on various branches:

| Branch                    | Patch Apply | Build Test |
|---------------------------|-------------|------------|
| stable/linux-6.1.y        |  Success    |  Success   |

[PATCH 6.12.y] uprobes: Fix race in uprobe_free_utask

[Christian Simon]

commit b583ef82b671c9a752fbe3e95bd4c1c51eab764d upstream.

Christian Simon verified the regression exists in v6.12.17 as per method
below and backported the mainline fix to the older version of
uprobe_free_utask. After that change I can no longer reproduce
the race with this method within 1h, while before it would show the
panic under a minute.

Max Makarov reported kernel panic [1] in perf user callchain code.

The reason for that is the race between uprobe_free_utask and bpf
profiler code doing the perf user stack unwind and is triggered
within uprobe_free_utask function:
  - after current->utask is freed and
  - before current->utask is set to NULL

 general protection fault, probably for non-canonical address 0x9e759c37ee555c76: 0000 [#1] SMP PTI
 RIP: 0010:is_uprobe_at_func_entry+0x28/0x80
 ...
  ? die_addr+0x36/0x90
  ? exc_general_protection+0x217/0x420
  ? asm_exc_general_protection+0x26/0x30
  ? is_uprobe_at_func_entry+0x28/0x80
  perf_callchain_user+0x20a/0x360
  get_perf_callchain+0x147/0x1d0
  bpf_get_stackid+0x60/0x90
  bpf_prog_9aac297fb833e2f5_do_perf_event+0x434/0x53b
  ? __smp_call_single_queue+0xad/0x120
  bpf_overflow_handler+0x75/0x110
  ...
  asm_sysvec_apic_timer_interrupt+0x1a/0x20
 RIP: 0010:__kmem_cache_free+0x1cb/0x350
 ...
  ? uprobe_free_utask+0x62/0x80
  ? acct_collect+0x4c/0x220
  uprobe_free_utask+0x62/0x80
  mm_release+0x12/0xb0
  do_exit+0x26b/0xaa0
  __x64_sys_exit+0x1b/0x20
  do_syscall_64+0x5a/0x80

It can be easily reproduced by running following commands in
separate terminals:

  # while :; do bpftrace -e 'uprobe:/bin/ls:_start  { printf("hit\n"); }' -c ls; done
  # bpftrace -e 'profile:hz:100000 { @[ustack()] = count(); }'

Fixing this by making sure current->utask pointer is set to NULL
before we start to release the utask object.

[1] https://github.com/grafana/pyroscope/issues/3673

Fixes: cfa7f3d2c526 ("perf,x86: avoid missing caller address in stack traces captured in uprobe")
Reported-by: Max Makarov <maxpain@linux.com>
(cherry picked from commit b583ef82b671c9a752fbe3e95bd4c1c51eab764d)
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Oleg Nesterov <oleg@redhat.com>
Acked-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/r/20250109141440.2692173-1-jolsa@kernel.org
[Christian Simon: Rebased for 6.12.y, due to mainline change https://lore.kernel.org/all/20240929144239.GA9475@redhat.com/]
Signed-off-by: Christian Simon <simon@swine.de>
---
 kernel/events/uprobes.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c
index 4b52cb2ae6d6..fcb777ba17df 100644
--- a/kernel/events/uprobes.c
+++ b/kernel/events/uprobes.c
@@ -1770,6 +1770,7 @@ void uprobe_free_utask(struct task_struct *t)
 	if (!utask)
 		return;
 
+	t->utask = NULL;
 	if (utask->active_uprobe)
 		put_uprobe(utask->active_uprobe);
 
@@ -1779,7 +1780,6 @@ void uprobe_free_utask(struct task_struct *t)
 
 	xol_free_insn_slot(t);
 	kfree(utask);
-	t->utask = NULL;
 }
 
 /*
-- 
2.47.1

Re: [PATCH 6.12.y] uprobes: Fix race in uprobe_free_utask

[Sasha Levin]

[ Sasha's backport helper bot ]

Hi,

Summary of potential issues:
ℹ️ Patch is missing in 6.13.y (ignore if backport was sent)
⚠️ Commit missing in all newer stable branches

The upstream commit SHA1 provided is correct: b583ef82b671c9a752fbe3e95bd4c1c51eab764d

WARNING: Author mismatch between patch and upstream commit:
Backport author: Christian Simon<simon@swine.de>
Commit author: Jiri Olsa<jolsa@kernel.org>

Status in newer kernel trees:
6.13.y | Present (exact SHA1)
6.6.y | Not found

Note: The patch differs from the upstream commit:
---
1:  b583ef82b671c ! 1:  b3a3f2344d131 uprobes: Fix race in uprobe_free_utask
    @@
      ## Metadata ##
    -Author: Jiri Olsa <jolsa@kernel.org>
    +Author: Christian Simon <simon@swine.de>
     
      ## Commit message ##
         uprobes: Fix race in uprobe_free_utask
     
    +    commit b583ef82b671c9a752fbe3e95bd4c1c51eab764d upstream.
    +
    +    Christian Simon verified the regression exists in v6.12.17 as per method
    +    below and backported the mainline fix to the older version of
    +    uprobe_free_utask. After that change I can no longer reproduce
    +    the race with this method within 1h, while before it would show the
    +    panic under a minute.
    +
         Max Makarov reported kernel panic [1] in perf user callchain code.
     
         The reason for that is the race between uprobe_free_utask and bpf
    @@ Commit message
     
         Fixes: cfa7f3d2c526 ("perf,x86: avoid missing caller address in stack traces captured in uprobe")
         Reported-by: Max Makarov <maxpain@linux.com>
    +    (cherry picked from commit b583ef82b671c9a752fbe3e95bd4c1c51eab764d)
         Signed-off-by: Jiri Olsa <jolsa@kernel.org>
         Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
         Acked-by: Oleg Nesterov <oleg@redhat.com>
         Acked-by: Andrii Nakryiko <andrii@kernel.org>
         Link: https://lore.kernel.org/r/20250109141440.2692173-1-jolsa@kernel.org
    +    [Christian Simon: Rebased for 6.12.y, due to mainline change https://lore.kernel.org/all/20240929144239.GA9475@redhat.com/]
    +    Signed-off-by: Christian Simon <simon@swine.de>
     
      ## kernel/events/uprobes.c ##
     @@ kernel/events/uprobes.c: void uprobe_free_utask(struct task_struct *t)
    @@ kernel/events/uprobes.c: void uprobe_free_utask(struct task_struct *t)
      		return;
      
     +	t->utask = NULL;
    - 	WARN_ON_ONCE(utask->active_uprobe || utask->xol_vaddr);
    + 	if (utask->active_uprobe)
    + 		put_uprobe(utask->active_uprobe);
      
    - 	timer_delete_sync(&utask->ri_timer);
     @@ kernel/events/uprobes.c: void uprobe_free_utask(struct task_struct *t)
    - 		ri = free_ret_instance(ri, true /* cleanup_hprobe */);
      
    + 	xol_free_insn_slot(t);
      	kfree(utask);
     -	t->utask = NULL;
      }
      
    - #define RI_TIMER_PERIOD (HZ / 10) /* 100 ms */
    + /*
---

Results of testing on various branches:

| Branch                    | Patch Apply | Build Test |
|---------------------------|-------------|------------|
| stable/linux-6.12.y       |  Success    |  Success   |