Re: [RFC PATCH] trace/perf: cure locking issue in perf_event_open() error path

Sebastian Siewior
2017-04-28T14:33:49Z

With the last patch on-top I trigger this now and then:

======================================================
WARNING: possible circular locking dependency detected
4.11.0-rc8-00894-g8bd462ee4aac-dirty #84 Not tainted
------------------------------------------------------
trinity-subchil/4966 is trying to acquire lock:
(cpu_hotplug_lock.rw_sem){++++++}, at: [<ffffffff812fcdad>] tp_perf_event_destroy+0xd/0x20

but task is already holding lock:
(&ctx->mutex){+.+.+.}, at: [<ffffffff81316ede>] perf_event_exit_task+0x2ae/0x8d0

which lock already depends on the new lock.

the existing dependency chain (in reverse order) is:

-> #2 (&ctx->mutex){+.+.+.}:
__lock_acquire+0x2534/0x2cd0
lock_acquire+0x27c/0x2f0
__mutex_lock+0xef/0x1280
mutex_lock_nested+0x16/0x20
SyS_perf_event_open+0x11ab/0x1e80
entry_SYSCALL_64_fastpath+0x23/0xc2

-> #1 (&sig->cred_guard_mutex){+.+.+.}:
__lock_acquire+0x2534/0x2cd0
lock_acquire+0x27c/0x2f0
__mutex_lock+0xef/0x1280
mutex_lock_interruptible_nested+0x16/0x20
SyS_perf_event_open+0x1bd6/0x1e80
entry_SYSCALL_64_fastpath+0x23/0xc2

-> #0 (cpu_hotplug_lock.rw_sem){++++++}:
check_prevs_add+0x544/0x18f0
__lock_acquire+0x2534/0x2cd0
lock_acquire+0x27c/0x2f0
get_online_cpus+0x3d/0x80
tp_perf_event_destroy+0xd/0x20
_free_event+0x61b/0x800
free_event+0x68/0x70
perf_event_exit_task+0x816/0x8d0
do_exit+0xc90/0x2a90
do_group_exit+0x1aa/0x2b0
SyS_exit_group+0x18/0x20
entry_SYSCALL_64_fastpath+0x23/0xc2

other info that might help us debug this:

Chain exists of:
cpu_hotplug_lock.rw_sem --> &sig->cred_guard_mutex --> &ctx->mutex

Possible unsafe locking scenario:

CPU0 CPU1
---- ----
lock(&ctx->mutex);
lock(&sig->cred_guard_mutex);
lock(&ctx->mutex);
lock(cpu_hotplug_lock.rw_sem);

*** DEADLOCK ***

1 lock held by trinity-subchil/4966:
#0: (&ctx->mutex){+.+.+.}, at: [<ffffffff81316ede>] perf_event_exit_task+0x2ae/0x8d0

stack backtrace:
CPU: 3 PID: 4966 Comm: trinity-subchil Not tainted 4.11.0-rc8-00894-g8bd462ee4aac-dirty #84
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.10.2-1 04/01/2014
Call Trace:
dump_stack+0x86/0xce
print_circular_bug+0x5c3/0x620
? lockdep_on+0x50/0x50
check_prevs_add+0x544/0x18f0
? check_irq_usage+0x150/0x150
? lock_acquire+0x27c/0x2f0
__lock_acquire+0x2534/0x2cd0
? __lock_acquire+0x2534/0x2cd0
? find_held_lock+0x36/0x1c0
lock_acquire+0x27c/0x2f0
? tp_perf_event_destroy+0xd/0x20
get_online_cpus+0x3d/0x80
? tp_perf_event_destroy+0xd/0x20
tp_perf_event_destroy+0xd/0x20
_free_event+0x61b/0x800
free_event+0x68/0x70
perf_event_exit_task+0x816/0x8d0
do_exit+0xc90/0x2a90
? mm_update_next_owner+0x550/0x550
? getname_flags+0xde/0x370
? getname_flags+0xde/0x370
? rcu_read_lock_sched_held+0x14a/0x180
? prepare_bprm_creds+0xf0/0xf0
? kmem_cache_free+0x250/0x2c0
? getname_flags+0xde/0x370
? entry_SYSCALL_64_fastpath+0x5/0xc2
do_group_exit+0x1aa/0x2b0
SyS_exit_group+0x18/0x20
entry_SYSCALL_64_fastpath+0x23/0xc2

and I am not sure what to do here…

Sebastian