From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754644AbdEDO1N (ORCPT ); Thu, 4 May 2017 10:27:13 -0400 Received: from mail-it0-f51.google.com ([209.85.214.51]:38143 "EHLO mail-it0-f51.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752039AbdEDO1E (ORCPT ); Thu, 4 May 2017 10:27:04 -0400 Date: Thu, 4 May 2017 08:27:01 -0600 From: Jens Axboe To: Peter Zijlstra Cc: Thomas Gleixner , Sebastian Andrzej Siewior , linux-kernel@vger.kernel.org Subject: Re: [PATCH] block/mq: Cure cpu hotplug lock inversion Message-ID: <20170504142700.GD9292@kernel.dk> References: <20170504130526.wvpagnb7f4lw2ih4@hirez.programming.kicks-ass.net> <46063288-15e3-a788-b6f0-8d1397f02fbe@kernel.dk> <20170504141849.caxfp4b4k3qnrvr4@hirez.programming.kicks-ass.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20170504141849.caxfp4b4k3qnrvr4@hirez.programming.kicks-ass.net> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, May 04 2017, Peter Zijlstra wrote: > On Thu, May 04, 2017 at 07:56:57AM -0600, Jens Axboe wrote: > > On 05/04/2017 07:05 AM, Peter Zijlstra wrote: > > > > > > By poking at /debug/sched_features I triggered the following splat: > > > > > > [] ====================================================== > > > [] WARNING: possible circular locking dependency detected > > > [] 4.11.0-00873-g964c8b7-dirty #694 Not tainted > > > [] ------------------------------------------------------ > > > [] bash/2109 is trying to acquire lock: > > > [] (cpu_hotplug_lock.rw_sem){++++++}, at: [] static_key_slow_dec+0x1b/0x50 > > > [] > > > [] but task is already holding lock: > > > [] (&sb->s_type->i_mutex_key#4){+++++.}, at: [] sched_feat_write+0x86/0x170 > > > [] > > > [] which lock already depends on the new lock. > > > [] > > > [] > > > [] the existing dependency chain (in reverse order) is: > > > [] > > > [] -> #2 (&sb->s_type->i_mutex_key#4){+++++.}: > > > [] lock_acquire+0x100/0x210 > > > [] down_write+0x28/0x60 > > > [] start_creating+0x5e/0xf0 > > > [] debugfs_create_dir+0x13/0x110 > > > [] blk_mq_debugfs_register+0x21/0x70 > > > [] blk_mq_register_dev+0x64/0xd0 > > > [] blk_register_queue+0x6a/0x170 > > > [] device_add_disk+0x22d/0x440 > > > [] loop_add+0x1f3/0x280 > > > [] loop_init+0x104/0x142 > > > [] do_one_initcall+0x43/0x180 > > > [] kernel_init_freeable+0x1de/0x266 > > > [] kernel_init+0xe/0x100 > > > [] ret_from_fork+0x31/0x40 > > > [] > > > [] -> #1 (all_q_mutex){+.+.+.}: > > > [] lock_acquire+0x100/0x210 > > > [] __mutex_lock+0x6c/0x960 > > > [] mutex_lock_nested+0x1b/0x20 > > > [] blk_mq_init_allocated_queue+0x37c/0x4e0 > > > [] blk_mq_init_queue+0x3a/0x60 > > > [] loop_add+0xe5/0x280 > > > [] loop_init+0x104/0x142 > > > [] do_one_initcall+0x43/0x180 > > > [] kernel_init_freeable+0x1de/0x266 > > > [] kernel_init+0xe/0x100 > > > [] ret_from_fork+0x31/0x40 > > > > > > [] *** DEADLOCK *** > > > [] > > > [] 3 locks held by bash/2109: > > > [] #0: (sb_writers#11){.+.+.+}, at: [] vfs_write+0x17d/0x1a0 > > > [] #1: (debugfs_srcu){......}, at: [] full_proxy_write+0x5d/0xd0 > > > [] #2: (&sb->s_type->i_mutex_key#4){+++++.}, at: [] sched_feat_write+0x86/0x170 > > > [] > > > [] stack backtrace: > > > [] CPU: 9 PID: 2109 Comm: bash Not tainted 4.11.0-00873-g964c8b7-dirty #694 > > > [] Hardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013 > > > [] Call Trace: > > > > > > [] lock_acquire+0x100/0x210 > > > [] get_online_cpus+0x2a/0x90 > > > [] static_key_slow_dec+0x1b/0x50 > > > [] static_key_disable+0x20/0x30 > > > [] sched_feat_write+0x131/0x170 > > > [] full_proxy_write+0x97/0xd0 > > > [] __vfs_write+0x28/0x120 > > > [] vfs_write+0xb5/0x1a0 > > > [] SyS_write+0x49/0xa0 > > > [] entry_SYSCALL_64_fastpath+0x23/0xc2 > > > > > > This is because of the cpu hotplug lock rework. Break the chain at #1 > > > by reversing the lock acquisition order. This way i_mutex_key#4 no > > > longer depends on cpu_hotplug_lock and things are good. > > > > Thanks Peter, applied. > > Note that the hotplug rework is still work-in-progress and lives in a > -tip branch. > > That said, the patch is harmless outside of that, so yes it can travel > upstream independently. But note that mainline cannot yet trigger that > splat. Yes, may as well just get it in and avoid any dependencies. -- Jens Axboe