From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753687AbdHWJyI (ORCPT ); Wed, 23 Aug 2017 05:54:08 -0400 Received: from mout.gmx.net ([212.227.17.22]:63239 "EHLO mout.gmx.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753595AbdHWJyF (ORCPT ); Wed, 23 Aug 2017 05:54:05 -0400 Message-ID: <1503482024.4970.5.camel@gmx.de> Subject: v4.11.12-rt10 - hotplug lockdep splat From: Mike Galbraith To: Sebastian Andrzej Siewior , Thomas Gleixner Cc: LKML , linux-rt-users , Steven Rostedt Date: Wed, 23 Aug 2017 11:53:44 +0200 In-Reply-To: <20170818120939.b2rj2whb2hgfzjpm@linutronix.de> References: <20170818120939.b2rj2whb2hgfzjpm@linutronix.de> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.20.5 Mime-Version: 1.0 Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:Cwo0f6Niknv763M3sydaq3KfOhInmAo0pV1UZK5u1MrGqVRGANr XPTCh25FNvFVQH1IiiDf8JMZXn9B0E8WTJaWRUWQBPaq0Cu4hxFC7mfxdp0iStTD7cjQdX2 KYGTz+bMQMi91WG0xxj+UYFuEnbPDUoR7NDtKNA2YAgHxE3J8Nfs5qCHG/WXKd+Qy5keMzD qqIHPqoPiEE8y4NERN1FQ== X-UI-Out-Filterresults: notjunk:1;V01:K0:+PN1Zs9DLOU=:5I86sP7OxmmP0IKA1GJ+DK RhywU5Cm0w1MT0Q0BfRVHj6pPr8bn0cX7DOfmla9fChfPDo1ofhufNRX20QqjmHDXInbwI8TE rOcaikwNfRSM67pVuZksYvA9XUML/PgfWFqq66SH4Mc1A+SOnVS32PdzCqCTkajPsD8uQWl/k RE3elO66L8dhdK2PtRsDd7PZBdUnqmfY0BPVtd4H2/eRdesxVn2RPUy05BHr9cpPZVddr9zE8 KAyFjA8FclT6zS7pFTDu1SPCfzlIbklsjWtEw8537NjoEqwzuCQWr+ZjCoopo7XAMmuTgrqRz 3H2lv4+2uOkzYBbsQ/ZLbl5Vq7cN/T18dH+5H2fU6jo5EOIziDSrNLx4LSoUVtwEwDzAZoa8b PNyoIdTKoS/llPA2PLylT7qnMgc3wu+QDvx5f61e8UgrhF5VPduM2BIsmSgJm8CzVteDVjp/r xHP4K3YC0OqIJFc0RPtrKow27xly1LyBIESIcV7WqvFs/7fY7huCIUjaQ0RpZnNS+vJ4HLzet 1p5wI81IYkwszzEJ8HuFX7rgBsczwaRskmAsS0XKGROIb4BKTSxNpqTdTkxJsW4Ozreiu8h3v pUbXCQKBe+upfQJHUbAsxzBtw/UqxvBkIFLg692ri1xayHhulcww35wubcPMoEOKVDe7y7ruu UOTcS9CyGgE8yn5kUDSfTu8MgphQEo0sxM8ZMhYmfSMU4yf71XAai6d4XzXQjNKIXmmF3067/ 7jrkBvAgPEXO+qWsQ0HItkae/zlw9n99qgx7705D7/blI5j8YiZs2xU456ycq+wxyjD54Iqfp T2akxorVsFX4JdUu9b4YIr8acU50Bf2pIA0ZmDf5UzTqFyuYrc= Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org virt box reminded me this morning to report this gripe. [ 512.629932] ====================================================== [ 512.629933] [ INFO: possible circular locking dependency detected ] [ 512.629934] 4.11.12-rt10-virgin #45 Tainted: G E [ 512.629934] ------------------------------------------------------- [ 512.629935] stress-cpu-hotp/18312 is trying to acquire lock: [ 512.629935] (&p->pi_lock){-...-.}, at: [] try_to_wake_up+0x2d/0x970 [ 512.629948] but task is already holding lock: [ 512.629948] (hrtimer_bases.lock){-.....}, at: [] hrtimer_interrupt+0x77/0x1d0 [ 512.629954] which lock already depends on the new lock. [ 512.629954] the existing dependency chain (in reverse order) is: [ 512.629954] -> #3 (hrtimer_bases.lock){-.....}: [ 512.629960] lock_acquire+0xbd/0x250 [ 512.629965] _raw_spin_lock_irqsave+0x53/0x70 [ 512.629966] lock_hrtimer_base.isra.27+0x29/0x50 [ 512.629966] hrtimer_start_range_ns+0x2f/0x410 [ 512.629968] enqueue_task_rt+0x325/0x360 [ 512.629970] __sched_setscheduler+0x2d5/0xb60 [ 512.629971] _sched_setscheduler+0x68/0x70 [ 512.629972] sched_setscheduler+0x13/0x20 [ 512.629974] ktimer_softirqd_set_sched_params+0x2a/0x60 [ 512.629975] smpboot_thread_fn+0x131/0x320 [ 512.629976] kthread+0x114/0x150 [ 512.629979] ret_from_fork+0x2a/0x40 [ 512.629979] -> #2 (&rt_b->rt_runtime_lock){-.....}: [ 512.629980] lock_acquire+0xbd/0x250 [ 512.629981] _raw_spin_lock+0x3b/0x50 [ 512.629982] enqueue_task_rt+0x1d8/0x360 [ 512.629983] __sched_setscheduler+0x2d5/0xb60 [ 512.629984] _sched_setscheduler+0x68/0x70 [ 512.629985] sched_setscheduler+0x13/0x20 [ 512.629985] ktimer_softirqd_set_sched_params+0x2a/0x60 [ 512.629986] smpboot_thread_fn+0x131/0x320 [ 512.629987] kthread+0x114/0x150 [ 512.629988] ret_from_fork+0x2a/0x40 [ 512.629988] -> #1 (&rq->lock){-...-.}: [ 512.629989] lock_acquire+0xbd/0x250 [ 512.629990] _raw_spin_lock+0x3b/0x50 [ 512.629990] task_fork_fair+0x3a/0x100 [ 512.629991] sched_fork+0x10d/0x2f0 [ 512.629995] copy_process.part.32+0x747/0x20a0 [ 512.629996] _do_fork+0xe4/0x710 [ 512.629997] kernel_thread+0x29/0x30 [ 512.629999] rest_init+0x22/0xe0 [ 512.630007] start_kernel+0x489/0x496 [ 512.630009] x86_64_start_reservations+0x2a/0x2c [ 512.630010] x86_64_start_kernel+0x13d/0x14c [ 512.630011] verify_cpu+0x0/0xfc [ 512.630011] -> #0 (&p->pi_lock){-...-.}: [ 512.630013] __lock_acquire+0x1527/0x1560 [ 512.630013] lock_acquire+0xbd/0x250 [ 512.630014] _raw_spin_lock_irqsave+0x53/0x70 [ 512.630014] try_to_wake_up+0x2d/0x970 [ 512.630015] wake_up_process+0x15/0x20 [ 512.630016] wakeup_timer_softirqd+0x32/0x40 [ 512.630016] wakeup_proper_softirq+0x25/0x30 [ 512.630017] raise_softirq_irqoff+0x3c/0x50 [ 512.630019] __hrtimer_run_queues+0x19c/0x650 [ 512.630020] hrtimer_interrupt+0xb8/0x1d0 [ 512.630021] hrtimers_dead_cpu+0x37a/0x390 [ 512.630022] cpuhp_invoke_callback+0x248/0x9d0 [ 512.630022] cpuhp_down_callbacks+0x42/0x80 [ 512.630023] _cpu_down+0xc5/0x100 [ 512.630023] do_cpu_down+0x3c/0x60 [ 512.630024] cpu_down+0x10/0x20 [ 512.630028] cpu_subsys_offline+0x14/0x20 [ 512.630029] device_offline+0x8a/0xb0 [ 512.630030] online_store+0x40/0x80 [ 512.630034] dev_attr_store+0x18/0x30 [ 512.630038] sysfs_kf_write+0x44/0x60 [ 512.630039] kernfs_fop_write+0x13c/0x1d0 [ 512.630042] __vfs_write+0x28/0x140 [ 512.630042] vfs_write+0xc7/0x1f0 [ 512.630043] SyS_write+0x49/0xa0 [ 512.630044] entry_SYSCALL_64_fastpath+0x1f/0xc2 [ 512.630044] other info that might help us debug this: [ 512.630044] Chain exists of: &p->pi_lock --> &rt_b->rt_runtime_lock --> hrtimer_bases.lock [ 512.630045] Possible unsafe locking scenario: [ 512.630045] CPU0 CPU1 [ 512.630045] ---- ---- [ 512.630046] lock(hrtimer_bases.lock); [ 512.630046] lock(&rt_b->rt_runtime_lock); [ 512.630046] lock(hrtimer_bases.lock); [ 512.630047] lock(&p->pi_lock); [ 512.630047] *** DEADLOCK *** [ 512.630048] 8 locks held by stress-cpu-hotp/18312: [ 512.630048] #0: (sb_writers#3){.+.+.+}, at: [] vfs_write+0x196/0x1f0 [ 512.630050] #1: (&of->mutex){+.+.+.}, at: [] kernfs_fop_write+0x10c/0x1d0 [ 512.630051] #2: (s_active#139){.+.+.+}, at: [] kernfs_fop_write+0x114/0x1d0 [ 512.630053] #3: (device_hotplug_lock){+.+.+.}, at: [] lock_device_hotplug_sysfs+0x15/0x40 [ 512.630055] #4: (&dev->mutex){......}, at: [] device_offline+0x48/0xb0 [ 512.630057] #5: (cpu_add_remove_lock){+.+.+.}, at: [] do_cpu_down+0x25/0x60 [ 512.630059] #6: (cpu_hotplug_lock.rw_sem){++++++}, at: [] percpu_down_write+0x26/0x120 [ 512.630062] #7: (hrtimer_bases.lock){-.....}, at: [] hrtimer_interrupt+0x77/0x1d0 [ 512.630064] stack backtrace: [ 512.630065] CPU: 0 PID: 18312 Comm: stress-cpu-hotp Tainted: G E 4.11.12-rt10-virgin #45 [ 512.630066] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.9.1-0-gb3ef39f-prebuilt.qemu-project.org 04/01/2014 [ 512.630067] Call Trace: [ 512.630074] dump_stack+0x85/0xc8 [ 512.630080] print_circular_bug+0x1f9/0x207 [ 512.630082] __lock_acquire+0x1527/0x1560 [ 512.630085] lock_acquire+0xbd/0x250 [ 512.630085] ? try_to_wake_up+0x2d/0x970 [ 512.630087] _raw_spin_lock_irqsave+0x53/0x70 [ 512.630088] ? try_to_wake_up+0x2d/0x970 [ 512.630089] try_to_wake_up+0x2d/0x970 [ 512.630091] wake_up_process+0x15/0x20 [ 512.630092] wakeup_timer_softirqd+0x32/0x40 [ 512.630093] wakeup_proper_softirq+0x25/0x30 [ 512.630094] raise_softirq_irqoff+0x3c/0x50 [ 512.630096] __hrtimer_run_queues+0x19c/0x650 [ 512.630098] hrtimer_interrupt+0xb8/0x1d0 [ 512.630100] hrtimers_dead_cpu+0x37a/0x390 [ 512.630102] ? hrtimers_prepare_cpu+0x90/0x90 [ 512.630103] cpuhp_invoke_callback+0x248/0x9d0 [ 512.630112] ? flow_cache_lookup+0x430/0x430 [ 512.630114] cpuhp_down_callbacks+0x42/0x80 [ 512.630116] _cpu_down+0xc5/0x100 [ 512.630117] do_cpu_down+0x3c/0x60 [ 512.630118] cpu_down+0x10/0x20 [ 512.630120] cpu_subsys_offline+0x14/0x20 [ 512.630121] device_offline+0x8a/0xb0 [ 512.630123] online_store+0x40/0x80 [ 512.630124] dev_attr_store+0x18/0x30 [ 512.630125] sysfs_kf_write+0x44/0x60 [ 512.630127] kernfs_fop_write+0x13c/0x1d0 [ 512.630128] __vfs_write+0x28/0x140 [ 512.630130] ? rcu_read_lock_sched_held+0x98/0xa0 [ 512.630131] ? rcu_sync_lockdep_assert+0x32/0x60 [ 512.630134] ? __sb_start_write+0x1d2/0x290 [ 512.630134] ? vfs_write+0x196/0x1f0 [ 512.630137] ? security_file_permission+0x3b/0xc0 [ 512.630138] vfs_write+0xc7/0x1f0 [ 512.630139] ? trace_hardirqs_on_caller+0xf9/0x1c0 [ 512.630141] SyS_write+0x49/0xa0 [ 512.630142] entry_SYSCALL_64_fastpath+0x1f/0xc2 [ 512.630143] RIP: 0033:0x7fdac1c54d10 [ 512.630144] RSP: 002b:00007ffd545bf5e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 512.630144] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fdac1c54d10 [ 512.630145] RDX: 0000000000000002 RSI: 00007fdac27c2000 RDI: 0000000000000001 [ 512.630145] RBP: 00007ffd545bebb0 R08: 000000000000000a R09: 00007fdac277c700 [ 512.630145] R10: 00000000ffffffff R11: 0000000000000246 R12: 0000000002657d86 [ 512.630146] R13: 00007ffd545bebc8 R14: 0000000002657d86 R15: 00007ffd545bebb4