From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753952AbdIDRQZ (ORCPT ); Mon, 4 Sep 2017 13:16:25 -0400 Received: from mout.gmx.net ([212.227.17.22]:54526 "EHLO mout.gmx.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753780AbdIDRQY (ORCPT ); Mon, 4 Sep 2017 13:16:24 -0400 Message-ID: <1504545350.22981.16.camel@gmx.de> Subject: Re: hotplug lockdep splat (tip) From: Mike Galbraith To: Peter Zijlstra Cc: LKML , Thomas Gleixner , Byungchul Park , Sebastian Andrzej Siewior Date: Mon, 04 Sep 2017 19:15:50 +0200 In-Reply-To: <20170904142313.27u33srjriitcx2u@hirez.programming.kicks-ass.net> References: <1504350596.16793.44.camel@gmx.de> <1504421975.30792.32.camel@gmx.de> <20170904075502.74pgfd2twdfklspw@hirez.programming.kicks-ass.net> <20170904142313.27u33srjriitcx2u@hirez.programming.kicks-ass.net> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.20.5 Mime-Version: 1.0 X-Provags-ID: V03:K0:+oJerrV2svliESHhgQa7KTKLNL4Am2dDcuMctqiF4ZjBN3600WW Wf5uUBW6IZNjDqwg87Um22ocrtpNPCewc7sBkLg5k5tTdM27n0Q5bhca3/CuaqZri5Kt3vl Cz6gDmkqPNZCWGQFG8rcob5S8TDax7/0Mpn6C6oJbKU7bw5R1Rye656W68SMBFwETL6Y4kd jhlGRrxEMpSKQLZanEhxA== X-UI-Out-Filterresults: notjunk:1;V01:K0:GsLg/4muyd0=:rYdSv+QASXydF4N1qu8ao+ BRQt0biJtGcQlCcTGtl1AEZ6+keSNDqCsleqXVcwa+8rQlSI5E/OCYejXMGAy0lxap0dkRH7O V0vIl3zEsfonFOHZ+qH26/71opsFBOtD5MWoaCQ7CZLE3edWYYV0xy5fjpTqC0L4flgqIlL9/ TjrAGV8cVP8li5Ud1v1jdmz9vvf/iBrz9k2+qfT6O6efcZKP31LBooWq/l1UileQVXq07B+Ln unjOaXA21WFeZ5wZwkVPdTHCzqIpBQV/kDlrv9Rjpvl5bOS5qtVP78wvJoIpeFJzwjDYVQeeE aR6bnQ1A+Ic3ze4OMY2q2ZMgJvExeKyiP7FXDMignBK8jUsGnqVYsGBlk1WOUfpEwscWvATw0 ooYBL7oLpMzvJ7GYk3uRZ4pMyOaJy27OrWPB7V41DsGGZlC8/YA/A60zMkLK0nOj6PTH6tR+p xCfPoO19q+sl2+LUW8VcSoYcj488hx8QhiCsrgTlCrZqHa4kshDToU+8HPacZF4M16SAbSJOt 5JLnS3bP4hYXYQ5zPknMiP7zaiWUWxw8juajtc9j9aah+SSTxflACNMqKXUzzH/7Fsm2NK/Sk lpeaRf39aPI+alwnWal5GsWjfYWA9X6fNH2VIGzKzkgpfnuUDujY4YTIaJXKqnhgdV7RdU1mr Ol9YcM3+FR8gS9bVpd9WBkVsw0XriZHM6Uo9gCeQEdWKN/iSEkWlyrD1ee7f55tneUUxRjrWh jtpSscN8WPPOZYUyv4wKGtefaDwTsLc7ajt+0WYMB3l8IXHG9Mx71F1jUbFCQkUhse54doDHc M2lnz78Ke2RC8iQ2BKKX4XWi//LxOAhn3/JgSF4Vd+6qvgvrQo= Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by nfs id v84HGVlA027940 On Mon, 2017-09-04 at 16:23 +0200, Peter Zijlstra wrote: > On Mon, Sep 04, 2017 at 09:55:02AM +0200, Peter Zijlstra wrote: > > On Sun, Sep 03, 2017 at 08:59:35AM +0200, Mike Galbraith wrote: > > > > > > [ 126.626908] Unregister pv shared memory for cpu 1 > > > [ 126.631041] > > > [ 126.631269] ====================================================== > > > [ 126.632442] WARNING: possible circular locking dependency detected > > > [ 126.633599] 4.13.0.g06260ca-tip-lockdep #2 Tainted: G E > > > [ 126.634380] ------------------------------------------------------ > > > [ 126.635124] stress-cpu-hotp/3156 is trying to acquire lock: > > > [ 126.635804] ((complete)&st->done){+.+.}, at: [] takedown_cpu+0x8a/0xf0 > > > [ 126.636809] > > > [ 126.636809] but task is already holding lock: > > > [ 126.637567] (sparse_irq_lock){+.+.}, at: [] irq_lock_sparse+0x17/0x20 > > > [ 126.638665] > > > > https://lkml.kernel.org/r/20170829193416.GC32112@worktop.programming.kicks-ass.net > > > > I still need to write a coherent Changelog and comments for that :/ > > How's this? [  138.241466] ====================================================== [  138.242014] WARNING: possible circular locking dependency detected [  138.242802] 4.13.0.g90abd70-tip-lockdep #6 Tainted: G            E   [  138.243701] ------------------------------------------------------ [  138.244540] cpuhp/0/12 is trying to acquire lock: [  138.245187]  (cpuhp_state){+.+.}, at: [] cpuhp_thread_fun+0x2a/0x160 [  138.246312]                 but now in release context of a crosslock acquired at the following: [  138.247519]  ((complete)&st->done#3){+.+.}, at: [] cpuhp_issue_call+0x13f/0x170 [  138.248745]                 which lock already depends on the new lock. [  138.249861]                 the existing dependency chain (in reverse order) is: [  138.250836]                 -> #2 ((complete)&st->done#3){+.+.}: [  138.251732]        __lock_acquire+0x116a/0x11c0 [  138.252339]        lock_acquire+0xbd/0x200 [  138.252906]        wait_for_completion+0x53/0x1a0 [  138.253587]        takedown_cpu+0x8a/0xf0 [  138.254194]        cpuhp_invoke_callback+0x1f6/0x830 [  138.254906]        cpuhp_down_callbacks+0x42/0x80 [  138.255586]        _cpu_down+0x14d/0x180 [  138.256179]        do_cpu_down+0x3e/0x60 [  138.256774]        cpu_down+0x10/0x20 [  138.257295]        cpu_subsys_offline+0x14/0x20 [  138.257973]        device_offline+0x8c/0xb0 [  138.258548]        online_store+0x40/0x80 [  138.259094]        dev_attr_store+0x18/0x30 [  138.259668]        sysfs_kf_write+0x44/0x60 [  138.260213]        kernfs_fop_write+0x113/0x1a0 [  138.260873]        __vfs_write+0x28/0x150 [  138.261447]        vfs_write+0xc7/0x1c0 [  138.262009]        SyS_write+0x49/0xa0 [  138.262535]        entry_SYSCALL_64_fastpath+0x1f/0xbe [  138.263276]                 -> #1 (sparse_irq_lock){+.+.}: [  138.264070]        __lock_acquire+0x116a/0x11c0 [  138.264731]        lock_acquire+0xbd/0x200 [  138.265300]        __mutex_lock+0x70/0x8e0 [  138.265881]        mutex_lock_nested+0x1b/0x20 [  138.266515]        irq_lock_sparse+0x17/0x20 [  138.267127]        irq_affinity_online_cpu+0x18/0xd0 [  138.267813]        cpuhp_invoke_callback+0x1f6/0x830 [  138.268538]        cpuhp_up_callbacks+0x37/0xb0 [  138.269174]        cpuhp_thread_fun+0x14f/0x160 [  138.269829]        smpboot_thread_fn+0x11a/0x1e0 [  138.270490]        kthread+0x152/0x190 [  138.271055]        ret_from_fork+0x2a/0x40 [  138.271661]                 -> #0 (cpuhp_state){+.+.}: [  138.272424]        cpuhp_thread_fun+0x54/0x160 [  138.273104]        smpboot_thread_fn+0x11a/0x1e0 [  138.273816]        kthread+0x152/0x190 [  138.274371]        ret_from_fork+0x2a/0x40 [  138.274958]        0xffffffffffffffff [  138.275450]                 other info that might help us debug this: [  138.276552] Chain exists of:                  cpuhp_state --> sparse_irq_lock --> (complete)&st- >done#3 [  138.278114]  Possible unsafe locking scenario by crosslock: [  138.279122]        CPU0                    CPU1 [  138.279777]        ----                    ---- [  138.280418]   lock(sparse_irq_lock); [  138.280936]   lock((complete)&st->done#3); [  138.281511]                                lock(cpuhp_state); [  138.282322]                                unlock((complete)&st- >done#3); [  138.283269]                  *** DEADLOCK *** [  138.284147] 1 lock held by cpuhp/0/12: [  138.284688]  #0:  (&x->wait#2){-...}, at: [] complete+0x1d/0x60 [  138.285734]                 stack backtrace: [  138.286368] CPU: 0 PID: 12 Comm: cpuhp/0 Tainted: G            E   4.13.0.g90abd70-tip-lockdep #6 [  138.287800] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.0.0-prebuilt.qemu-project.org 04/01/2014 [  138.289324] Call Trace: [  138.289708]  dump_stack+0x85/0xc5 [  138.290199]  print_circular_bug+0x2e9/0x2f7 [  138.290805]  check_prev_add+0x666/0x700 [  138.291313]  ? print_bfs_bug+0x40/0x40 [  138.291866]  ? trace_hardirqs_on+0xd/0x10 [  138.292223]  lock_commit_crosslock+0x410/0x5a0 [  138.292620]  complete+0x29/0x60 [  138.292900]  cpuhp_thread_fun+0xa1/0x160 [  138.293233]  ? cpuhp_thread_fun+0x2a/0x160 [  138.293584]  smpboot_thread_fn+0x11a/0x1e0 [  138.293943]  kthread+0x152/0x190 [  138.294221]  ? sort_range+0x30/0x30 [  138.294521]  ? kthread_create_on_node+0x40/0x40 [  138.294920]  ? kthread+0x10/0x190 [  138.295203]  ret_from_fork+0x2a/0x40