From: Ingo Molnar <mingo@elte.hu>
To: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>,
Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: linux-kernel@vger.kernel.org, laijs@cn.fujitsu.com,
dipankar@in.ibm.com, akpm@linux-foundation.org,
mathieu.desnoyers@polymtl.ca, josh@joshtriplett.org,
dvhltc@us.ibm.com, niv@us.ibm.com, tglx@linutronix.de,
peterz@infradead.org, rostedt@goodmis.org,
Valdis.Kletnieks@vt.edu, dhowells@redhat.com
Subject: Re: [PATCH tip/core/rcu 0/21] v6 add lockdep-based diagnostics to rcu_dereference()
Date: Thu, 25 Feb 2010 22:22:26 +0100 [thread overview]
Message-ID: <20100225212226.GA9134@elte.hu> (raw)
In-Reply-To: <20100225120444.GA17623@elte.hu>
plus there's one with a lockdep warning as well:
PM: Adding info for No Bus:vcsa10
eth0: no IPv6 routers present
------------[ cut here ]------------
WARNING: at kernel/softirq.c:143 _local_bh_enable_ip+0x3a/0x9d()
Hardware name: System Product Name
Modules linked in:
Pid: 0, comm: swapper Not tainted 2.6.33-tip-00754-gbd37c01-dirty #18764
Call Trace:
[<c1036212>] warn_slowpath_common+0x6a/0x81
[<c103b97f>] ? _local_bh_enable_ip+0x3a/0x9d
[<c15aa645>] ? inet_putpeer+0x4c/0x4f
[<c103623b>] warn_slowpath_null+0x12/0x15
[<c103b97f>] _local_bh_enable_ip+0x3a/0x9d
[<c103b9ef>] local_bh_enable_ip+0xd/0xf
[<c168189b>] _raw_spin_unlock_bh+0x2a/0x2d
[<c15aa645>] inet_putpeer+0x4c/0x4f
[<c15a6f71>] ipv4_dst_destroy+0x2b/0x44
[<c1591167>] dst_destroy+0x63/0xaf
[<c15a6d7e>] dst_free+0x1e/0x29
[<c15a6d96>] dst_rcu_free+0xd/0xf
[<c107998c>] rcu_do_batch+0x91/0x16f
[<c1079afb>] __rcu_process_callbacks+0x91/0x95
[<c1079c08>] rcu_needs_cpu+0x109/0x132
[<c105870f>] tick_nohz_stop_sched_tick+0x15c/0x36f
[<c1001c09>] cpu_idle+0x78/0xb3
[<c16470a0>] rest_init+0x9c/0xa1
[<c1a3c958>] start_kernel+0x373/0x378
[<c1a3c0ab>] i386_start_kernel+0xab/0xb2
---[ end trace 1110cdd88992d6b2 ]---
=========================================================
[ INFO: possible irq lock inversion dependency detected ]
2.6.33-tip-00754-gbd37c01-dirty #18764
---------------------------------------------------------
swapper/0 just changed the state of lock:
(&(&base->lock)->rlock){+.-.-.}, at: [<c10414a3>] get_next_timer_interrupt+0x27/0x1ba
but this lock was taken by another, HARDIRQ-safe lock in the past:
(&(&np->lock)->rlock){-.-...}
and interrupts could create inverse lock ordering between them.
other info that might help us debug this:
no locks held by swapper/0.
the shortest dependencies between 2nd lock and 1st lock:
-> (&(&np->lock)->rlock){-.-...} ops: 4907 {
IN-HARDIRQ-W at:
[<c105b42d>] mark_irqflags+0x50/0x110
[<c105c895>] __lock_acquire+0x21d/0x3d7
[<c105cae1>] lock_acquire+0x92/0xa9
[<c16811c3>] _raw_spin_lock+0x23/0x53
[<c14a1cdc>] spin_lock+0xd/0xf
[<c14a2e4a>] nv_nic_irq_optimized+0xcf/0x242
[<c1074f66>] handle_IRQ_event+0x4b/0xf1
[<c1076924>] handle_fasteoi_irq+0x7b/0xb2
[<c1004340>] handle_irq+0x40/0x4c
[<c1003b8e>] do_IRQ+0x46/0x9f
[<c1002f55>] common_interrupt+0x35/0x40
IN-SOFTIRQ-W at:
[<c105b44c>] mark_irqflags+0x6f/0x110
[<c105c895>] __lock_acquire+0x21d/0x3d7
[<c105cae1>] lock_acquire+0x92/0xa9
[<c16812de>] _raw_spin_lock_irqsave+0x2c/0x5f
[<c14a3541>] nv_start_xmit_optimized+0x81/0x406
[<c158d70b>] dev_hard_start_xmit+0x141/0x1d3
[<c15a1b6e>] sch_direct_xmit+0x55/0x134
[<c158daca>] dev_queue_xmit+0x241/0x38b
[<c1594609>] neigh_resolve_output+0x10d/0x133
[<c15ed540>] ip6_output_finish+0x7b/0xb0
[<c15ef305>] ip6_output2+0x1d0/0x1d8
[<c15efc2c>] ip6_output+0xc2/0xc6
[<c160249a>] dst_output+0xe/0x10
[<c1603389>] mld_sendpack+0x1ae/0x2aa
[<c1603d95>] mld_send_cr+0x1e0/0x1e8
[<c1603dad>] mld_ifc_timer_expire+0x10/0x33
[<c1040e87>] run_timer_softirq+0x172/0x207
[<c103b6f9>] __do_softirq+0xb7/0x169
[<c103b7db>] do_softirq+0x30/0x48
[<c103b912>] irq_exit+0x3a/0x6d
[<c1016328>] smp_apic_timer_interrupt+0x6d/0x7b
[<c1682006>] apic_timer_interrupt+0x36/0x40
INITIAL USE at:
[<c105c8ac>] __lock_acquire+0x234/0x3d7
[<c105cae1>] lock_acquire+0x92/0xa9
[<c1681282>] _raw_spin_lock_irq+0x29/0x59
[<c149f2d1>] spin_lock_irq+0xd/0xf
[<c166ab67>] nv_probe+0xa24/0xe40
[<c127d7a8>] local_pci_probe+0x13/0x15
[<c127e229>] pci_device_probe+0x48/0x6b
[<c13206d7>] really_probe+0x8e/0x165
[<c13207fa>] driver_probe_device+0x4c/0x82
[<c1320878>] __driver_attach+0x48/0x64
[<c131fd71>] bus_for_each_dev+0x42/0x6c
[<c1320501>] driver_attach+0x19/0x1b
[<c13201d6>] bus_add_driver+0xb6/0x200
[<c1320adf>] driver_register+0x7e/0xe5
[<c127e428>] __pci_register_driver+0x51/0xae
[<c1a69b69>] init_nic+0x14/0x16
[<c1001143>] do_one_initcall+0x51/0x13b
[<c1a3c360>] do_basic_setup+0x3f/0x52
[<c1a3c3e2>] kernel_init+0x6f/0xb0
[<c1002f66>] kernel_thread_helper+0x6/0x10
}
... key at: [<c22d7bd4>] __key.40874+0x0/0x8
... acquired at:
[<c105c1da>] check_prev_add+0x177/0x1ae
[<c105c5f6>] validate_chain+0x3e5/0x467
[<c105c9d8>] __lock_acquire+0x360/0x3d7
[<c105cae1>] lock_acquire+0x92/0xa9
[<c16812de>] _raw_spin_lock_irqsave+0x2c/0x5f
[<c104106d>] lock_timer_base+0x26/0x45
[<c10410b3>] __mod_timer+0x27/0x109
[<c10411dc>] mod_timer+0x20/0x27
[<c14a4920>] nv_open+0x43c/0x45b
[<c158ea3a>] dev_open+0x8b/0xc5
[<c158e0c1>] dev_change_flags+0x9a/0x149
[<c15cd6a9>] devinet_ioctl+0x22e/0x43f
[<c15ce863>] inet_ioctl+0x93/0xac
[<c157f817>] sock_ioctl+0x1ce/0x1f2
[<c10cfe65>] vfs_ioctl+0x2c/0x94
[<c10d02c3>] do_vfs_ioctl+0x267/0x27e
[<c10d0320>] sys_ioctl+0x46/0x66
[<c10029fb>] sysenter_do_call+0x12/0x36
-> (&(&base->lock)->rlock){+.-.-.} ops: 142011 {
HARDIRQ-ON-W at:
[<c105b48f>] mark_irqflags+0xb2/0x110
[<c105c895>] __lock_acquire+0x21d/0x3d7
[<c105cae1>] lock_acquire+0x92/0xa9
[<c16811c3>] _raw_spin_lock+0x23/0x53
[<c10414a3>] get_next_timer_interrupt+0x27/0x1ba
[<c1058904>] tick_nohz_stop_sched_tick+0x351/0x36f
[<c1001c09>] cpu_idle+0x78/0xb3
[<c16470a0>] rest_init+0x9c/0xa1
[<c1a3c958>] start_kernel+0x373/0x378
[<c1a3c0ab>] i386_start_kernel+0xab/0xb2
IN-SOFTIRQ-W at:
[<c105b44c>] mark_irqflags+0x6f/0x110
[<c105c895>] __lock_acquire+0x21d/0x3d7
[<c105cae1>] lock_acquire+0x92/0xa9
[<c1681282>] _raw_spin_lock_irq+0x29/0x59
[<c1040d4c>] run_timer_softirq+0x37/0x207
[<c103b6f9>] __do_softirq+0xb7/0x169
[<c103b7db>] do_softirq+0x30/0x48
[<c103b912>] irq_exit+0x3a/0x6d
[<c1003bd3>] do_IRQ+0x8b/0x9f
[<c1002f55>] common_interrupt+0x35/0x40
[<c1075c0c>] __setup_irq+0x1e2/0x26b
[<c1075cb3>] setup_irq+0x1e/0x22
[<c1a3f0ff>] setup_default_timer_irq+0xf/0x11
[<c1a3f117>] hpet_time_init+0x16/0x18
[<c1a3f0e9>] x86_late_time_init+0x9/0x10
[<c1a3c8cd>] start_kernel+0x2e8/0x378
[<c1a3c0ab>] i386_start_kernel+0xab/0xb2
IN-RECLAIM_FS-W at:
[<c105b4d6>] mark_irqflags+0xf9/0x110
[<c105c895>] __lock_acquire+0x21d/0x3d7
[<c105cae1>] lock_acquire+0x92/0xa9
[<c16812de>] _raw_spin_lock_irqsave+0x2c/0x5f
[<c104106d>] lock_timer_base+0x26/0x45
[<c10410b3>] __mod_timer+0x27/0x109
[<c167ff60>] schedule_timeout+0x79/0x9d
[<c10a3d7e>] kswapd+0xde/0x178
[<c104b87c>] kthread+0x6f/0x74
[<c1002f66>] kernel_thread_helper+0x6/0x10
INITIAL USE at:
[<c105c8ac>] __lock_acquire+0x234/0x3d7
[<c105cae1>] lock_acquire+0x92/0xa9
[<c16812de>] _raw_spin_lock_irqsave+0x2c/0x5f
[<c104106d>] lock_timer_base+0x26/0x45
[<c10410b3>] __mod_timer+0x27/0x109
[<c10411dc>] mod_timer+0x20/0x27
[<c1a6106a>] con_init+0xa9/0x1cb
[<c1a607c1>] console_init+0x12/0x20
[<c1a3c7f2>] start_kernel+0x20d/0x378
[<c1a3c0ab>] i386_start_kernel+0xab/0xb2
}
... key at: [<c1d4a0ec>] __key.33971+0x0/0x8
... acquired at:
[<c105bb85>] check_usage_backwards+0x64/0x6f
[<c105b240>] mark_lock_irq+0x5b/0xca
[<c105b36e>] mark_lock+0xbf/0x12e
[<c105b48f>] mark_irqflags+0xb2/0x110
[<c105c895>] __lock_acquire+0x21d/0x3d7
[<c105cae1>] lock_acquire+0x92/0xa9
[<c16811c3>] _raw_spin_lock+0x23/0x53
[<c10414a3>] get_next_timer_interrupt+0x27/0x1ba
[<c1058904>] tick_nohz_stop_sched_tick+0x351/0x36f
[<c1001c09>] cpu_idle+0x78/0xb3
[<c16470a0>] rest_init+0x9c/0xa1
[<c1a3c958>] start_kernel+0x373/0x378
[<c1a3c0ab>] i386_start_kernel+0xab/0xb2
stack backtrace:
Pid: 0, comm: swapper Tainted: G W 2.6.33-tip-00754-gbd37c01-dirty #18764
Call Trace:
[<c167f62e>] ? printk+0x14/0x16
[<c105baa7>] print_irq_inversion_bug+0xde/0xe9
[<c105bb85>] check_usage_backwards+0x64/0x6f
[<c100440c>] ? dump_trace+0x7b/0xa8
[<c105b240>] mark_lock_irq+0x5b/0xca
[<c105bb21>] ? check_usage_backwards+0x0/0x6f
[<c105b36e>] mark_lock+0xbf/0x12e
[<c105b48f>] mark_irqflags+0xb2/0x110
[<c105c895>] __lock_acquire+0x21d/0x3d7
[<c10414a3>] ? get_next_timer_interrupt+0x27/0x1ba
[<c105cae1>] lock_acquire+0x92/0xa9
[<c10414a3>] ? get_next_timer_interrupt+0x27/0x1ba
[<c16811c3>] _raw_spin_lock+0x23/0x53
[<c10414a3>] ? get_next_timer_interrupt+0x27/0x1ba
[<c10414a3>] get_next_timer_interrupt+0x27/0x1ba
[<c1079c08>] ? rcu_needs_cpu+0x109/0x132
[<c1058904>] tick_nohz_stop_sched_tick+0x351/0x36f
[<c1001c09>] cpu_idle+0x78/0xb3
[<c16470a0>] rest_init+0x9c/0xa1
[<c1a3c958>] start_kernel+0x373/0x378
[<c1a3c0ab>] i386_start_kernel+0xab/0xb2
prev parent reply other threads:[~2010-02-25 21:23 UTC|newest]
Thread overview: 84+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-02-23 1:04 [PATCH tip/core/rcu 0/21] v6 add lockdep-based diagnostics to rcu_dereference() Paul E. McKenney
2010-02-23 1:04 ` [PATCH tip/core/rcu 01/21] rcu: introduce lockdep-based checking to RCU read-side primitives Paul E. McKenney
2010-02-25 10:09 ` [tip:core/rcu] rcu: Introduce " tip-bot for Paul E. McKenney
2010-02-23 1:04 ` [PATCH tip/core/rcu 02/21] rcu: add lockdep-enabled variants of rcu_dereference() Paul E. McKenney
2010-02-25 10:09 ` [tip:core/rcu] rcu: Add " tip-bot for Paul E. McKenney
2010-02-23 1:04 ` [PATCH tip/core/rcu 03/21] rcu: integrate rcu_dereference_check() message into lockdep Paul E. McKenney
2010-02-25 10:09 ` [tip:core/rcu] rcu: Integrate " tip-bot for Paul E. McKenney
2010-02-23 1:04 ` [PATCH tip/core/rcu 04/21] rcu: disable lockdep checking in RCU list-traversal primitives Paul E. McKenney
2010-02-25 10:10 ` [tip:core/rcu] rcu: Disable " tip-bot for Paul E. McKenney
2010-02-23 1:04 ` [PATCH tip/core/rcu 05/21] net: add checking to rcu_dereference() primitives Paul E. McKenney
2010-02-25 10:10 ` [tip:core/rcu] net: Add " tip-bot for Paul E. McKenney
2010-02-23 1:04 ` [PATCH tip/core/rcu 06/21] sched: use lockdep-based checking on rcu_dereference() Paul E. McKenney
2010-02-25 10:11 ` [tip:core/rcu] sched: Use " tip-bot for Paul E. McKenney
2010-02-25 11:06 ` [tip:core/rcu] sched, cgroups: Fix module export tip-bot for Ingo Molnar
2010-02-23 1:04 ` [PATCH tip/core/rcu 07/21] sched: better name for for_each_domain_rd Paul E. McKenney
2010-02-25 10:11 ` [tip:core/rcu] sched: Better " tip-bot for Paul E. McKenney
2010-02-23 1:04 ` [PATCH tip/core/rcu 08/21] vfs: apply lockdep-based checking to rcu_dereference() uses Paul E. McKenney
2010-02-25 10:11 ` [tip:core/rcu] vfs: Apply " tip-bot for Paul E. McKenney
2010-02-23 1:04 ` [PATCH tip/core/rcu 09/21] vfs: abstract rcu_dereference_check for files-fdtable use Paul E. McKenney
2010-02-25 10:11 ` [tip:core/rcu] vfs: Abstract " tip-bot for Paul E. McKenney
2010-02-23 1:04 ` [PATCH tip/core/rcu 10/21] radix-tree: disable RCU lockdep checking in radix tree Paul E. McKenney
2010-02-25 10:12 ` [tip:core/rcu] radix-tree: Disable " tip-bot for Paul E. McKenney
2010-02-23 1:04 ` [PATCH tip/core/rcu 11/21] idr: apply lockdep-based diagnostics to rcu_dereference() uses Paul E. McKenney
2010-02-25 10:12 ` [tip:core/rcu] idr: Apply " tip-bot for Paul E. McKenney
2010-02-23 1:04 ` [PATCH tip/core/rcu 12/21] security: apply lockdep-based checking " Paul E. McKenney
2010-02-25 10:12 ` [tip:core/rcu] security: Apply " tip-bot for Paul E. McKenney
2010-02-23 1:04 ` [PATCH tip/core/rcu 13/21] rcu: documentation update for CONFIG_PROVE_RCU Paul E. McKenney
2010-02-25 10:12 ` [tip:core/rcu] rcu: Documentation " tip-bot for Paul E. McKenney
2010-02-23 1:04 ` [PATCH tip/core/rcu 14/21] rcu: fix citation of Mathieu's dissertation Paul E. McKenney
2010-02-25 10:13 ` [tip:core/rcu] rcu: Fix " tip-bot for Paul E. McKenney
2010-02-23 1:04 ` [PATCH tip/core/rcu 15/21] rcu: accelerate grace period if last non-dynticked CPU Paul E. McKenney
2010-02-25 10:13 ` [tip:core/rcu] rcu: Accelerate " tip-bot for Paul E. McKenney
2010-02-23 1:05 ` [PATCH tip/core/rcu 16/21] rcu: use canonical URL for Mathieu's dissertation Paul E. McKenney
2010-02-25 10:13 ` [tip:core/rcu] rcu: Use " tip-bot for Paul E. McKenney
2010-02-23 1:05 ` [PATCH tip/core/rcu 17/21] rcu: stop overflowing signed integers Paul E. McKenney
2010-02-25 10:14 ` [tip:core/rcu] rcu: Stop " tip-bot for Paul E. McKenney
2010-02-23 1:05 ` [PATCH tip/core/rcu 18/21] rcu: Convert to raw_spinlocks Paul E. McKenney
2010-02-25 10:14 ` [tip:core/rcu] " tip-bot for Paul E. McKenney
2010-02-23 1:05 ` [PATCH tip/core/rcu 19/21] rcu: fix deadlock in TREE_PREEMPT_RCU CPU stall detection Paul E. McKenney
2010-02-25 10:14 ` [tip:core/rcu] rcu: Fix " tip-bot for Paul E. McKenney
2010-02-23 1:05 ` [PATCH tip/core/rcu 20/21] rcu: fix rcutorture mod_timer argument to delay one jiffy Paul E. McKenney
2010-02-25 10:14 ` [tip:core/rcu] rcu: Fix " tip-bot for Paul E. McKenney
2010-02-23 1:05 ` [PATCH tip/core/rcu 21/21] rcu: add RCU_CPU_STALL_VERBOSE to dump detailed per-task information Paul E. McKenney
2010-02-25 10:15 ` [tip:core/rcu] rcu: Add " tip-bot for Paul E. McKenney
2010-02-23 12:59 ` [PATCH tip/core/rcu 0/21] v6 add lockdep-based diagnostics to rcu_dereference() Arnd Bergmann
2010-02-23 13:15 ` Mathieu Desnoyers
2010-02-23 14:35 ` Paul E. McKenney
2010-02-23 15:54 ` Arnd Bergmann
2010-02-23 16:16 ` Paul E. McKenney
2010-02-23 17:15 ` Arnd Bergmann
2010-02-23 18:01 ` Paul E. McKenney
2010-02-24 20:03 ` [PATCH 00/10] __rcu annotations, first draft Arnd Bergmann
2010-02-24 22:18 ` Paul E. McKenney
2010-02-25 8:37 ` Ingo Molnar
2010-02-24 20:03 ` [PATCH 01/10] rcu: define __rcu address space modifier for sparse Arnd Bergmann
2010-02-24 20:12 ` Mathieu Desnoyers
2010-02-24 20:22 ` Arnd Bergmann
2010-02-24 20:03 ` [PATCH 02/10] rcu: annotated list rcu code Arnd Bergmann
2010-02-24 20:15 ` Mathieu Desnoyers
2010-02-24 20:32 ` Arnd Bergmann
2010-02-24 20:03 ` [PATCH 03/10] cgroups: __rcu annotations Arnd Bergmann
2010-02-24 20:04 ` [PATCH 04/10] credentials: rcu annotation Arnd Bergmann
2010-02-24 20:04 ` [PATCH 05/10] perf_event: __rcu annotations Arnd Bergmann
2010-02-24 20:04 ` [PATCH 06/10] audit: " Arnd Bergmann
2010-02-24 20:04 ` [PATCH 07/10] module: " Arnd Bergmann
2010-02-24 20:13 ` Alexey Dobriyan
2010-02-24 20:26 ` Arnd Bergmann
2010-02-24 22:17 ` Paul E. McKenney
2010-02-24 23:07 ` Arnd Bergmann
2010-02-24 23:59 ` Paul E. McKenney
2010-02-25 17:06 ` Paul E. McKenney
2010-02-25 18:10 ` Arnd Bergmann
2010-02-25 20:05 ` Paul E. McKenney
2010-02-26 2:12 ` Paul E. McKenney
2010-02-24 20:04 ` [PATCH 08/10] pid: " Arnd Bergmann
2010-02-24 20:04 ` [PATCH 09/10] notifiers: " Arnd Bergmann
2010-02-24 20:04 ` [PATCH 10/10] scheduler: " Arnd Bergmann
2010-02-23 13:28 ` [PATCH tip/core/rcu 0/21] v6 add lockdep-based diagnostics to rcu_dereference() Paul E. McKenney
2010-02-25 10:00 ` Ingo Molnar
2010-02-25 10:01 ` Ingo Molnar
2010-02-25 12:04 ` Ingo Molnar
2010-02-25 18:18 ` Paul E. McKenney
2010-02-25 21:36 ` Paul E. McKenney
2010-02-25 21:22 ` Ingo Molnar [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20100225212226.GA9134@elte.hu \
--to=mingo@elte.hu \
--cc=Valdis.Kletnieks@vt.edu \
--cc=a.p.zijlstra@chello.nl \
--cc=akpm@linux-foundation.org \
--cc=dhowells@redhat.com \
--cc=dipankar@in.ibm.com \
--cc=dvhltc@us.ibm.com \
--cc=josh@joshtriplett.org \
--cc=laijs@cn.fujitsu.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mathieu.desnoyers@polymtl.ca \
--cc=niv@us.ibm.com \
--cc=paulmck@linux.vnet.ibm.com \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox