From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754267Ab3BQFmr (ORCPT ); Sun, 17 Feb 2013 00:42:47 -0500 Received: from aserp1040.oracle.com ([141.146.126.69]:50462 "EHLO aserp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753733Ab3BQFmp (ORCPT ); Sun, 17 Feb 2013 00:42:45 -0500 Message-ID: <51206DAB.7030701@oracle.com> Date: Sun, 17 Feb 2013 00:42:03 -0500 From: Sasha Levin User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130113 Thunderbird/17.0.2 MIME-Version: 1.0 To: Ingo Molnar , Thomas Gleixner , Peter Zijlstra CC: "Paul E. McKenney" , Dave Jones , "linux-kernel@vger.kernel.org" Subject: sched: circular dependency between sched_domains_mutex and oom_notify_list Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Source-IP: acsinet22.oracle.com [141.146.126.238] Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi all, I was fuzzing with trinity inside a KVM tools guest, with today's -next kernel when I've hit the following spew. I suspect it's the result of adding the new rcu_oom_notify, but that happened about half a year ago so I'm not sure why this showed up only now. [ 1039.634183] ====================================================== [ 1039.635717] [ INFO: possible circular locking dependency detected ] [ 1039.637255] 3.8.0-rc7-next-20130215-sasha-00003-gea816fa #286 Tainted: G W [ 1039.639104] ------------------------------------------------------- [ 1039.640579] init/1 is trying to acquire lock: [ 1039.641224] ((oom_notify_list).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain+0x7f/0xc0 [ 1039.641224] [ 1039.641224] but task is already holding lock: [ 1039.641224] (sched_domains_mutex){+.+.+.}, at: [] partition_sched_domains+0x28/0x3f0 [ 1039.641224] [ 1039.641224] which lock already depends on the new lock. [ 1039.641224] [ 1039.641224] [ 1039.641224] the existing dependency chain (in reverse order) is: [ 1039.641224] -> #2 (sched_domains_mutex){+.+.+.}: [ 1039.641224] [] check_prevs_add+0xba/0x1a0 [ 1039.641224] [] validate_chain.isra.21+0x6a0/0x7b0 [ 1039.641224] [] __lock_acquire+0xa13/0xb00 [ 1039.641224] [] lock_acquire+0x1ca/0x270 [ 1039.641224] [] __mutex_lock_common+0x5a/0x560 [ 1039.641224] [] mutex_lock_nested+0x3f/0x50 [ 1039.641224] [] partition_sched_domains+0x28/0x3f0 [ 1039.641224] [] cpuset_cpu_inactive+0x3b/0x50 [ 1039.641224] [] notifier_call_chain+0xee/0x130 [ 1039.641224] [] __raw_notifier_call_chain+0x9/0x10 [ 1039.641224] [] __cpu_notify+0x1b/0x30 [ 1039.641224] [] _cpu_down+0xaf/0x350 [ 1039.641224] [] disable_nonboot_cpus+0x84/0x1c0 [ 1039.641224] [] kernel_restart+0x16/0x60 [ 1039.641224] [] sys_reboot+0x161/0x2b0 [ 1039.641224] [] tracesys+0xe1/0xe6 [ 1039.641224] -> #1 (cpu_hotplug.lock){+.+.+.}: [ 1039.641224] [] check_prevs_add+0xba/0x1a0 [ 1039.641224] [] validate_chain.isra.21+0x6a0/0x7b0 [ 1039.641224] [] __lock_acquire+0xa13/0xb00 [ 1039.641224] [] lock_acquire+0x1ca/0x270 [ 1039.641224] [] __mutex_lock_common+0x5a/0x560 [ 1039.641224] [] mutex_lock_nested+0x3f/0x50 [ 1039.641224] [] get_online_cpus+0x37/0x50 [ 1039.641224] [] rcu_oom_notify+0x94/0x150 [ 1039.641224] [] notifier_call_chain+0xee/0x130 [ 1039.641224] [] __blocking_notifier_call_chain+0x98/0xc0 [ 1039.641224] [] blocking_notifier_call_chain+0x11/0x20 [ 1039.641224] [] out_of_memory+0x45/0x1f0 [ 1039.641224] [] __alloc_pages_nodemask+0x83d/0xbf0 [ 1039.641224] [] alloc_pages_vma+0xfc/0x150 [ 1039.641224] [] read_swap_cache_async+0x90/0x220 [ 1039.641224] [] swapin_readahead+0x9e/0xf0 [ 1039.641224] [] do_swap_page.isra.41+0x107/0x5a0 [ 1039.641224] [] handle_pte_fault+0x126/0x200 [ 1039.641224] [] handle_mm_fault+0x397/0x3e0 [ 1039.641224] [] __get_user_pages+0x418/0x5f0 [ 1039.641224] [] __mlock_vma_pages_range+0xb3/0xc0 [ 1039.641224] [] __mm_populate+0xf4/0x170 [ 1039.641224] [] sys_mlockall+0x160/0x1a0 [ 1039.641224] [] tracesys+0xe1/0xe6 [ 1039.641224] -> #0 ((oom_notify_list).rwsem){.+.+..}: [ 1039.641224] [] check_prev_add+0x115/0x640 [ 1039.641224] [] check_prevs_add+0xba/0x1a0 [ 1039.641224] [] validate_chain.isra.21+0x6a0/0x7b0 [ 1039.641224] [] __lock_acquire+0xa13/0xb00 [ 1039.641224] [] lock_acquire+0x1ca/0x270 [ 1039.641224] [] down_read+0x47/0x8e [ 1039.641224] [] __blocking_notifier_call_chain+0x7f/0xc0 [ 1039.641224] [] blocking_notifier_call_chain+0x11/0x20 [ 1039.641224] [] out_of_memory+0x45/0x1f0 [ 1039.641224] [] __alloc_pages_nodemask+0x83d/0xbf0 [ 1039.641224] [] allocate_slab+0x13a/0x1f0 [ 1039.641224] [] new_slab+0x2b/0x1b0 [ 1039.641224] [] __slab_alloc.isra.34+0x1c5/0x31f [ 1039.641224] [] kmem_cache_alloc_node_trace+0x114/0x390 [ 1039.641224] [] __sdt_alloc+0x137/0x1f0 [ 1039.641224] [] build_sched_domains+0x2c/0x4e0 [ 1039.641224] [] partition_sched_domains+0x353/0x3f0 [ 1039.641224] [] cpuset_cpu_inactive+0x3b/0x50 [ 1039.641224] [] notifier_call_chain+0xee/0x130 [ 1039.641224] [] __raw_notifier_call_chain+0x9/0x10 [ 1039.641224] [] __cpu_notify+0x1b/0x30 [ 1039.641224] [] _cpu_down+0xaf/0x350 [ 1039.641224] [] disable_nonboot_cpus+0x84/0x1c0 [ 1039.641224] [] kernel_restart+0x16/0x60 [ 1039.641224] [] sys_reboot+0x161/0x2b0 [ 1039.641224] [] tracesys+0xe1/0xe6 [ 1039.641224] [ 1039.641224] other info that might help us debug this: [ 1039.641224] [ 1039.641224] Chain exists of: (oom_notify_list).rwsem --> cpu_hotplug.lock --> sched_domains_mutex [ 1039.641224] Possible unsafe locking scenario: [ 1039.641224] [ 1039.641224] CPU0 CPU1 [ 1039.641224] ---- ---- [ 1039.641224] lock(sched_domains_mutex); [ 1039.641224] lock(cpu_hotplug.lock); [ 1039.641224] lock(sched_domains_mutex); [ 1039.641224] lock((oom_notify_list).rwsem); [ 1039.641224] [ 1039.641224] *** DEADLOCK *** [ 1039.641224] [ 1039.641224] 4 locks held by init/1: [ 1039.641224] #0: (reboot_mutex){+.+.+.}, at: [] sys_reboot+0xde/0x2b0 [ 1039.641224] #1: (cpu_add_remove_lock){+.+.+.}, at: [] cpu_maps_update_begin+0x12/0x20 [ 1039.641224] #2: (cpu_hotplug.lock){+.+.+.}, at: [] cpu_hotplug_begin+0x27/0x60 [ 1039.641224] #3: (sched_domains_mutex){+.+.+.}, at: [] partition_sched_domains+0x28/0x3f0 [ 1039.641224] [ 1039.641224] stack backtrace: [ 1039.641224] Pid: 1, comm: init Tainted: G W 3.8.0-rc7-next-20130215-sasha-00003-gea816fa #286 [ 1039.641224] Call Trace: [ 1039.641224] [] print_circular_bug+0xd3/0xe4 [ 1039.641224] [] check_prev_add+0x115/0x640 [ 1039.641224] [] check_prevs_add+0xba/0x1a0 [ 1039.641224] [] ? sched_clock+0x15/0x20 [ 1039.641224] [] validate_chain.isra.21+0x6a0/0x7b0 [ 1039.641224] [] __lock_acquire+0xa13/0xb00 [ 1039.641224] [] ? sched_clock+0x15/0x20 [ 1039.641224] [] ? kvm_clock_read+0x38/0x70 [ 1039.641224] [] lock_acquire+0x1ca/0x270 [ 1039.641224] [] ? __blocking_notifier_call_chain+0x7f/0xc0 [ 1039.641224] [] down_read+0x47/0x8e [ 1039.641224] [] ? __blocking_notifier_call_chain+0x7f/0xc0 [ 1039.641224] [] __blocking_notifier_call_chain+0x7f/0xc0 [ 1039.641224] [] blocking_notifier_call_chain+0x11/0x20 [ 1039.641224] [] out_of_memory+0x45/0x1f0 [ 1039.641224] [] __alloc_pages_nodemask+0x83d/0xbf0 [ 1039.641224] [] allocate_slab+0x13a/0x1f0 [ 1039.641224] [] new_slab+0x2b/0x1b0 [ 1039.641224] [] __slab_alloc.isra.34+0x1c5/0x31f [ 1039.641224] [] ? __lock_is_held+0x52/0x80 [ 1039.641224] [] ? __sdt_alloc+0x137/0x1f0 [ 1039.641224] [] kmem_cache_alloc_node_trace+0x114/0x390 [ 1039.641224] [] ? pcpu_alloc+0x32b/0x3e0 [ 1039.641224] [] ? __sdt_alloc+0xfb/0x1f0 [ 1039.641224] [] ? __sdt_alloc+0x137/0x1f0 [ 1039.641224] [] __sdt_alloc+0x137/0x1f0 [ 1039.641224] [] ? partition_sched_domains+0x220/0x3f0 [ 1039.641224] [] build_sched_domains+0x2c/0x4e0 [ 1039.641224] [] partition_sched_domains+0x353/0x3f0 [ 1039.641224] [] ? partition_sched_domains+0xe7/0x3f0 [ 1039.641224] [] cpuset_cpu_inactive+0x3b/0x50 [ 1039.641224] [] notifier_call_chain+0xee/0x130 [ 1039.641224] [] __raw_notifier_call_chain+0x9/0x10 [ 1039.641224] [] __cpu_notify+0x1b/0x30 [ 1039.641224] [] _cpu_down+0xaf/0x350 [ 1039.641224] [] ? printk+0x5c/0x5e [ 1039.641224] [] disable_nonboot_cpus+0x84/0x1c0 [ 1039.641224] [] kernel_restart+0x16/0x60 [ 1039.641224] [] sys_reboot+0x161/0x2b0 [ 1039.641224] [] ? rcu_eqs_exit_common+0x64/0x340 [ 1039.641224] [] ? rcu_eqs_enter_common+0x306/0x3a0 [ 1039.641224] [] ? user_exit+0xa5/0xd0 [ 1039.641224] [] ? trace_hardirqs_on_caller+0x128/0x160 [ 1039.641224] [] ? trace_hardirqs_on+0xd/0x10 [ 1039.641224] [] ? syscall_trace_enter+0x24/0x2e0 [ 1039.641224] [] ? trace_hardirqs_on_caller+0x128/0x160 [ 1039.641224] [] tracesys+0xe1/0xe6 Thanks, Sasha