public inbox for linux-kernel@vger.kernel.org
 help / color / mirror / Atom feed
From: Steven Rostedt <rostedt@goodmis.org>
To: linux-kernel@vger.kernel.org
Cc: Daniel Bristot de Oliveira <bristot@kernel.org>,
	Nico Pache <npache@redhat.com>
Subject: [for-next][PATCH 14/20] tracing/osnoise: Fix possible recursive locking in stop_per_cpu_kthreads
Date: Tue, 27 Sep 2022 12:02:30 -0400	[thread overview]
Message-ID: <20220927160248.807986897@goodmis.org> (raw)
In-Reply-To: 20220927160216.349640304@goodmis.org

From: Nico Pache <npache@redhat.com>

There is a recursive lock on the cpu_hotplug_lock.

In kernel/trace/trace_osnoise.c:<start/stop>_per_cpu_kthreads:
    - start_per_cpu_kthreads calls cpus_read_lock() and if
	start_kthreads returns a error it will call stop_per_cpu_kthreads.
    - stop_per_cpu_kthreads then calls cpus_read_lock() again causing
      deadlock.

Fix this by calling cpus_read_unlock() before calling
stop_per_cpu_kthreads. This behavior can also be seen in commit
f46b16520a08 ("trace/hwlat: Implement the per-cpu mode").

This error was noticed during the LTP ftrace-stress-test:

WARNING: possible recursive locking detected
--------------------------------------------
sh/275006 is trying to acquire lock:
ffffffffb02f5400 (cpu_hotplug_lock){++++}-{0:0}, at: stop_per_cpu_kthreads

but task is already holding lock:
ffffffffb02f5400 (cpu_hotplug_lock){++++}-{0:0}, at: start_per_cpu_kthreads

other info that might help us debug this:
 Possible unsafe locking scenario:

      CPU0
      ----
 lock(cpu_hotplug_lock);
 lock(cpu_hotplug_lock);

 *** DEADLOCK ***

May be due to missing lock nesting notation

3 locks held by sh/275006:
 #0: ffff8881023f0470 (sb_writers#24){.+.+}-{0:0}, at: ksys_write
 #1: ffffffffb084f430 (trace_types_lock){+.+.}-{3:3}, at: rb_simple_write
 #2: ffffffffb02f5400 (cpu_hotplug_lock){++++}-{0:0}, at: start_per_cpu_kthreads

Link: https://lkml.kernel.org/r/20220919144932.3064014-1-npache@redhat.com

Fixes: c8895e271f79 ("trace/osnoise: Support hotplug operations")
Signed-off-by: Nico Pache <npache@redhat.com>
Acked-by: Daniel Bristot de Oliveira <bristot@kernel.org>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
---
 kernel/trace/trace_osnoise.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/kernel/trace/trace_osnoise.c b/kernel/trace/trace_osnoise.c
index 313439920a8c..78d536d3ff3d 100644
--- a/kernel/trace/trace_osnoise.c
+++ b/kernel/trace/trace_osnoise.c
@@ -1786,8 +1786,9 @@ static int start_per_cpu_kthreads(void)
 	for_each_cpu(cpu, current_mask) {
 		retval = start_kthread(cpu);
 		if (retval) {
+			cpus_read_unlock();
 			stop_per_cpu_kthreads();
-			break;
+			return retval;
 		}
 	}
 
-- 
2.35.1

  parent reply	other threads:[~2022-09-27 16:02 UTC|newest]

Thread overview: 21+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-09-27 16:02 [for-next][PATCH 00/20] tracing: Update for 6.1 Steven Rostedt
2022-09-27 16:02 ` [for-next][PATCH 01/20] tracing/eprobe: Add eprobe filter support Steven Rostedt
2022-09-27 16:02 ` [for-next][PATCH 02/20] selftests/ftrace: Add eprobe syntax error testcase Steven Rostedt
2022-09-27 16:02 ` [for-next][PATCH 03/20] rv/monitors: add static qualifier for local symbols Steven Rostedt
2022-09-27 16:02 ` [for-next][PATCH 04/20] rv/dot2K: add static qualifier for local variable Steven Rostedt
2022-09-27 16:02 ` [for-next][PATCH 05/20] tracing: Add numeric delta time to the trace event benchmark Steven Rostedt
2022-09-27 16:02 ` [for-next][PATCH 06/20] tracing/hist: Call hist functions directly via a switch statement Steven Rostedt
2022-09-27 16:02 ` [for-next][PATCH 07/20] tracing: Move struct filter_pred into trace_events_filter.c Steven Rostedt
2022-09-27 16:02 ` [for-next][PATCH 08/20] tracing/filter: Call filter predicate functions directly via a switch statement Steven Rostedt
2022-09-27 16:02 ` [for-next][PATCH 09/20] tracepoint: Optimize the critical region of mutex_lock in tracepoint_module_coming() Steven Rostedt
2022-09-27 16:02 ` [for-next][PATCH 10/20] x86/ftrace: Remove unused modifying_ftrace_code declaration Steven Rostedt
2022-09-27 16:02 ` [for-next][PATCH 11/20] x86/kprobes: Remove unused arch_kprobe_override_function() declaration Steven Rostedt
2022-09-27 16:02 ` [for-next][PATCH 12/20] tracing: kprobe: Fix kprobe event gen test module on exit Steven Rostedt
2022-09-27 16:02 ` [for-next][PATCH 13/20] tracing: kprobe: Make gen test module work in arm and riscv Steven Rostedt
2022-09-27 16:02 ` Steven Rostedt [this message]
2022-09-27 16:02 ` [for-next][PATCH 15/20] rv/monitor: Add __init/__exit annotations to module init/exit funcs Steven Rostedt
2022-09-27 16:02 ` [for-next][PATCH 16/20] tracing: Disable interrupt or preemption before acquiring arch_spinlock_t Steven Rostedt
2022-09-27 16:02 ` [for-next][PATCH 17/20] ftrace: Remove obsoleted code from ftrace and task_struct Steven Rostedt
2022-09-27 16:02 ` [for-next][PATCH 18/20] x86: kprobes: Remove unused macro stack_addr Steven Rostedt
2022-09-27 16:02 ` [for-next][PATCH 19/20] tracing/eprobe: Fix alloc event dir failed when event name no set Steven Rostedt
2022-09-27 16:02 ` [for-next][PATCH 20/20] ftrace: Properly unset FTRACE_HASH_FL_MOD Steven Rostedt

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20220927160248.807986897@goodmis.org \
    --to=rostedt@goodmis.org \
    --cc=bristot@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=npache@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox