From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1750974AbdISBuj (ORCPT ); Mon, 18 Sep 2017 21:50:39 -0400 Received: from LGEAMRELO13.lge.com ([156.147.23.53]:59103 "EHLO lgeamrelo13.lge.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750783AbdISBui (ORCPT ); Mon, 18 Sep 2017 21:50:38 -0400 X-Original-SENDERIP: 156.147.1.121 X-Original-MAILFROM: byungchul.park@lge.com X-Original-SENDERIP: 10.177.222.33 X-Original-MAILFROM: byungchul.park@lge.com Date: Tue, 19 Sep 2017 10:50:27 +0900 From: Byungchul Park To: "Paul E. McKenney" Cc: Steven Rostedt , Neeraj Upadhyay , josh@joshtriplett.org, mathieu.desnoyers@efficios.com, jiangshanlai@gmail.com, linux-kernel@vger.kernel.org, sramana@codeaurora.org, prsood@codeaurora.org, pkondeti@codeaurora.org, markivx@codeaurora.org, peterz@infradead.org, kernel-team@lge.com Subject: Re: Query regarding synchronize_sched_expedited and resched_cpu Message-ID: <20170919015027.GD5994@X58A-UD3R> References: <8f33e48e-ac6d-2c88-e16f-20b698c06292@codeaurora.org> <20170917010015.GW3521@linux.vnet.ibm.com> <20170918111105.15f687da@gandalf.local.home> <20170918160125.GL3521@linux.vnet.ibm.com> <20170918121213.312c82b0@gandalf.local.home> <20170918162412.GM3521@linux.vnet.ibm.com> <20170918122931.0e3341f3@gandalf.local.home> <20170918165527.GN3521@linux.vnet.ibm.com> <20170918235311.GA20177@linux.vnet.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20170918235311.GA20177@linux.vnet.ibm.com> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Sep 18, 2017 at 04:53:11PM -0700, Paul E. McKenney wrote: > So, Byungchul, any enlightenment? Please see lockdep splat below. > > Thanx, Paul > > ------------------------------------------------------------------------ > > [ 35.310179] ====================================================== > [ 35.310749] WARNING: possible circular locking dependency detected > [ 35.310749] 4.13.0-rc4+ #1 Not tainted > [ 35.310749] ------------------------------------------------------ > [ 35.310749] torture_onoff/766 is trying to acquire lock: > [ 35.313943] ((complete)&st->done){+.+.}, at: [] takedown_cpu+0x86/0xf0 > [ 35.313943] > [ 35.313943] but task is already holding lock: > [ 35.313943] (sparse_irq_lock){+.+.}, at: [] irq_lock_sparse+0x12/0x20 > [ 35.313943] > [ 35.313943] which lock already depends on the new lock. > [ 35.313943] > [ 35.313943] > [ 35.313943] the existing dependency chain (in reverse order) is: > [ 35.313943] > [ 35.313943] -> #1 (sparse_irq_lock){+.+.}: > [ 35.313943] __mutex_lock+0x65/0x960 > [ 35.313943] mutex_lock_nested+0x16/0x20 > [ 35.313943] irq_lock_sparse+0x12/0x20 > [ 35.313943] irq_affinity_online_cpu+0x13/0xd0 > [ 35.313943] cpuhp_invoke_callback+0xa7/0x8b0 > [ 35.313943] > [ 35.313943] -> #0 ((complete)&st->done){+.+.}: > [ 35.313943] check_prev_add+0x401/0x800 > [ 35.313943] __lock_acquire+0x1100/0x11a0 > [ 35.313943] lock_acquire+0x9e/0x1e0 > [ 35.313943] wait_for_completion+0x36/0x130 > [ 35.313943] takedown_cpu+0x86/0xf0 > [ 35.313943] cpuhp_invoke_callback+0xa7/0x8b0 > [ 35.313943] cpuhp_down_callbacks+0x3d/0x80 > [ 35.313943] _cpu_down+0xbb/0xf0 > [ 35.313943] do_cpu_down+0x39/0x50 > [ 35.313943] cpu_down+0xb/0x10 > [ 35.313943] torture_offline+0x75/0x140 > [ 35.313943] torture_onoff+0x102/0x1e0 > [ 35.313943] kthread+0x142/0x180 > [ 35.313943] ret_from_fork+0x27/0x40 > [ 35.313943] > [ 35.313943] other info that might help us debug this: > [ 35.313943] > [ 35.313943] Possible unsafe locking scenario: > [ 35.313943] > [ 35.313943] CPU0 CPU1 > [ 35.313943] ---- ---- > [ 35.313943] lock(sparse_irq_lock); > [ 35.313943] lock((complete)&st->done); > [ 35.313943] lock(sparse_irq_lock); > [ 35.313943] lock((complete)&st->done); > [ 35.313943] > [ 35.313943] *** DEADLOCK *** Hello Paul and Steven, This is saying: Thread A -------- takedown_cpu() irq_lock_sparse() wait_for_completion(&st->done) // Wait for completion of B irq_unlock_sparse() Thread B -------- cpuhp_invoke_callback() irq_lock_sparse() // Wait for A to irq_unlock_sparse() (on the way going to complete(&st->done)) So, lockdep consider this as a deadlock. Is it possible to happen? Thanks, Byungchul > [ 35.313943] > [ 35.313943] 3 locks held by torture_onoff/766: > [ 35.313943] #0: (cpu_add_remove_lock){+.+.}, at: [] do_cpu_down+0x22/0x50 > [ 35.313943] #1: (cpu_hotplug_lock.rw_sem){++++}, at: [] percpu_down_write+0x21/0xf0 > [ 35.313943] #2: (sparse_irq_lock){+.+.}, at: [] irq_lock_sparse+0x12/0x20 > [ 35.313943] > [ 35.313943] stack backtrace: > [ 35.313943] CPU: 7 PID: 766 Comm: torture_onoff Not tainted 4.13.0-rc4+ #1 > [ 35.313943] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014 > [ 35.313943] Call Trace: > [ 35.313943] dump_stack+0x67/0x97 > [ 35.313943] print_circular_bug+0x21d/0x330 > [ 35.313943] ? add_lock_to_list.isra.31+0xc0/0xc0 > [ 35.313943] check_prev_add+0x401/0x800 > [ 35.313943] ? wake_up_q+0x70/0x70 > [ 35.313943] __lock_acquire+0x1100/0x11a0 > [ 35.313943] ? __lock_acquire+0x1100/0x11a0 > [ 35.313943] ? add_lock_to_list.isra.31+0xc0/0xc0 > [ 35.313943] lock_acquire+0x9e/0x1e0 > [ 35.313943] ? takedown_cpu+0x86/0xf0 > [ 35.313943] wait_for_completion+0x36/0x130 > [ 35.313943] ? takedown_cpu+0x86/0xf0 > [ 35.313943] ? stop_machine_cpuslocked+0xb9/0xd0 > [ 35.313943] ? cpuhp_invoke_callback+0x8b0/0x8b0 > [ 35.313943] ? cpuhp_complete_idle_dead+0x10/0x10 > [ 35.313943] takedown_cpu+0x86/0xf0 > [ 35.313943] cpuhp_invoke_callback+0xa7/0x8b0 > [ 35.313943] cpuhp_down_callbacks+0x3d/0x80 > [ 35.313943] _cpu_down+0xbb/0xf0 > [ 35.313943] do_cpu_down+0x39/0x50 > [ 35.313943] cpu_down+0xb/0x10 > [ 35.313943] torture_offline+0x75/0x140 > [ 35.313943] torture_onoff+0x102/0x1e0 > [ 35.313943] kthread+0x142/0x180 > [ 35.313943] ? torture_kthread_stopping+0x70/0x70 > [ 35.313943] ? kthread_create_on_node+0x40/0x40 > [ 35.313943] ret_from_fork+0x27/0x40