From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: linux-kernel@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
stable@vger.kernel.org, Fengguang Wu <fengguang.wu@intel.com>,
Jan Kara <jack@suse.cz>, Thomas Gleixner <tglx@linutronix.de>
Subject: [PATCH 3.14 22/39] timer: Fix lock inversion between hrtimer_bases.lock and scheduler locks
Date: Tue, 5 Aug 2014 11:13:59 -0700 [thread overview]
Message-ID: <20140805181336.714330580@linuxfoundation.org> (raw)
In-Reply-To: <20140805181336.030349158@linuxfoundation.org>
3.14-stable review patch. If anyone has any objections, please let me know.
------------------
From: Jan Kara <jack@suse.cz>
commit 504d58745c9ca28d33572e2d8a9990b43e06075d upstream.
clockevents_increase_min_delta() calls printk() from under
hrtimer_bases.lock. That causes lock inversion on scheduler locks because
printk() can call into the scheduler. Lockdep puts it as:
======================================================
[ INFO: possible circular locking dependency detected ]
3.15.0-rc8-06195-g939f04b #2 Not tainted
-------------------------------------------------------
trinity-main/74 is trying to acquire lock:
(&port_lock_key){-.....}, at: [<811c60be>] serial8250_console_write+0x8c/0x10c
but task is already holding lock:
(hrtimer_bases.lock){-.-...}, at: [<8103caeb>] hrtimer_try_to_cancel+0x13/0x66
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #5 (hrtimer_bases.lock){-.-...}:
[<8104a942>] lock_acquire+0x92/0x101
[<8142f11d>] _raw_spin_lock_irqsave+0x2e/0x3e
[<8103c918>] __hrtimer_start_range_ns+0x1c/0x197
[<8107ec20>] perf_swevent_start_hrtimer.part.41+0x7a/0x85
[<81080792>] task_clock_event_start+0x3a/0x3f
[<810807a4>] task_clock_event_add+0xd/0x14
[<8108259a>] event_sched_in+0xb6/0x17a
[<810826a2>] group_sched_in+0x44/0x122
[<81082885>] ctx_sched_in.isra.67+0x105/0x11f
[<810828e6>] perf_event_sched_in.isra.70+0x47/0x4b
[<81082bf6>] __perf_install_in_context+0x8b/0xa3
[<8107eb8e>] remote_function+0x12/0x2a
[<8105f5af>] smp_call_function_single+0x2d/0x53
[<8107e17d>] task_function_call+0x30/0x36
[<8107fb82>] perf_install_in_context+0x87/0xbb
[<810852c9>] SYSC_perf_event_open+0x5c6/0x701
[<810856f9>] SyS_perf_event_open+0x17/0x19
[<8142f8ee>] syscall_call+0x7/0xb
-> #4 (&ctx->lock){......}:
[<8104a942>] lock_acquire+0x92/0x101
[<8142f04c>] _raw_spin_lock+0x21/0x30
[<81081df3>] __perf_event_task_sched_out+0x1dc/0x34f
[<8142cacc>] __schedule+0x4c6/0x4cb
[<8142cae0>] schedule+0xf/0x11
[<8142f9a6>] work_resched+0x5/0x30
-> #3 (&rq->lock){-.-.-.}:
[<8104a942>] lock_acquire+0x92/0x101
[<8142f04c>] _raw_spin_lock+0x21/0x30
[<81040873>] __task_rq_lock+0x33/0x3a
[<8104184c>] wake_up_new_task+0x25/0xc2
[<8102474b>] do_fork+0x15c/0x2a0
[<810248a9>] kernel_thread+0x1a/0x1f
[<814232a2>] rest_init+0x1a/0x10e
[<817af949>] start_kernel+0x303/0x308
[<817af2ab>] i386_start_kernel+0x79/0x7d
-> #2 (&p->pi_lock){-.-...}:
[<8104a942>] lock_acquire+0x92/0x101
[<8142f11d>] _raw_spin_lock_irqsave+0x2e/0x3e
[<810413dd>] try_to_wake_up+0x1d/0xd6
[<810414cd>] default_wake_function+0xb/0xd
[<810461f3>] __wake_up_common+0x39/0x59
[<81046346>] __wake_up+0x29/0x3b
[<811b8733>] tty_wakeup+0x49/0x51
[<811c3568>] uart_write_wakeup+0x17/0x19
[<811c5dc1>] serial8250_tx_chars+0xbc/0xfb
[<811c5f28>] serial8250_handle_irq+0x54/0x6a
[<811c5f57>] serial8250_default_handle_irq+0x19/0x1c
[<811c56d8>] serial8250_interrupt+0x38/0x9e
[<810510e7>] handle_irq_event_percpu+0x5f/0x1e2
[<81051296>] handle_irq_event+0x2c/0x43
[<81052cee>] handle_level_irq+0x57/0x80
[<81002a72>] handle_irq+0x46/0x5c
[<810027df>] do_IRQ+0x32/0x89
[<8143036e>] common_interrupt+0x2e/0x33
[<8142f23c>] _raw_spin_unlock_irqrestore+0x3f/0x49
[<811c25a4>] uart_start+0x2d/0x32
[<811c2c04>] uart_write+0xc7/0xd6
[<811bc6f6>] n_tty_write+0xb8/0x35e
[<811b9beb>] tty_write+0x163/0x1e4
[<811b9cd9>] redirected_tty_write+0x6d/0x75
[<810b6ed6>] vfs_write+0x75/0xb0
[<810b7265>] SyS_write+0x44/0x77
[<8142f8ee>] syscall_call+0x7/0xb
-> #1 (&tty->write_wait){-.....}:
[<8104a942>] lock_acquire+0x92/0x101
[<8142f11d>] _raw_spin_lock_irqsave+0x2e/0x3e
[<81046332>] __wake_up+0x15/0x3b
[<811b8733>] tty_wakeup+0x49/0x51
[<811c3568>] uart_write_wakeup+0x17/0x19
[<811c5dc1>] serial8250_tx_chars+0xbc/0xfb
[<811c5f28>] serial8250_handle_irq+0x54/0x6a
[<811c5f57>] serial8250_default_handle_irq+0x19/0x1c
[<811c56d8>] serial8250_interrupt+0x38/0x9e
[<810510e7>] handle_irq_event_percpu+0x5f/0x1e2
[<81051296>] handle_irq_event+0x2c/0x43
[<81052cee>] handle_level_irq+0x57/0x80
[<81002a72>] handle_irq+0x46/0x5c
[<810027df>] do_IRQ+0x32/0x89
[<8143036e>] common_interrupt+0x2e/0x33
[<8142f23c>] _raw_spin_unlock_irqrestore+0x3f/0x49
[<811c25a4>] uart_start+0x2d/0x32
[<811c2c04>] uart_write+0xc7/0xd6
[<811bc6f6>] n_tty_write+0xb8/0x35e
[<811b9beb>] tty_write+0x163/0x1e4
[<811b9cd9>] redirected_tty_write+0x6d/0x75
[<810b6ed6>] vfs_write+0x75/0xb0
[<810b7265>] SyS_write+0x44/0x77
[<8142f8ee>] syscall_call+0x7/0xb
-> #0 (&port_lock_key){-.....}:
[<8104a62d>] __lock_acquire+0x9ea/0xc6d
[<8104a942>] lock_acquire+0x92/0x101
[<8142f11d>] _raw_spin_lock_irqsave+0x2e/0x3e
[<811c60be>] serial8250_console_write+0x8c/0x10c
[<8104e402>] call_console_drivers.constprop.31+0x87/0x118
[<8104f5d5>] console_unlock+0x1d7/0x398
[<8104fb70>] vprintk_emit+0x3da/0x3e4
[<81425f76>] printk+0x17/0x19
[<8105bfa0>] clockevents_program_min_delta+0x104/0x116
[<8105c548>] clockevents_program_event+0xe7/0xf3
[<8105cc1c>] tick_program_event+0x1e/0x23
[<8103c43c>] hrtimer_force_reprogram+0x88/0x8f
[<8103c49e>] __remove_hrtimer+0x5b/0x79
[<8103cb21>] hrtimer_try_to_cancel+0x49/0x66
[<8103cb4b>] hrtimer_cancel+0xd/0x18
[<8107f102>] perf_swevent_cancel_hrtimer.part.60+0x2b/0x30
[<81080705>] task_clock_event_stop+0x20/0x64
[<81080756>] task_clock_event_del+0xd/0xf
[<81081350>] event_sched_out+0xab/0x11e
[<810813e0>] group_sched_out+0x1d/0x66
[<81081682>] ctx_sched_out+0xaf/0xbf
[<81081e04>] __perf_event_task_sched_out+0x1ed/0x34f
[<8142cacc>] __schedule+0x4c6/0x4cb
[<8142cae0>] schedule+0xf/0x11
[<8142f9a6>] work_resched+0x5/0x30
other info that might help us debug this:
Chain exists of:
&port_lock_key --> &ctx->lock --> hrtimer_bases.lock
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(hrtimer_bases.lock);
lock(&ctx->lock);
lock(hrtimer_bases.lock);
lock(&port_lock_key);
*** DEADLOCK ***
4 locks held by trinity-main/74:
#0: (&rq->lock){-.-.-.}, at: [<8142c6f3>] __schedule+0xed/0x4cb
#1: (&ctx->lock){......}, at: [<81081df3>] __perf_event_task_sched_out+0x1dc/0x34f
#2: (hrtimer_bases.lock){-.-...}, at: [<8103caeb>] hrtimer_try_to_cancel+0x13/0x66
#3: (console_lock){+.+...}, at: [<8104fb5d>] vprintk_emit+0x3c7/0x3e4
stack backtrace:
CPU: 0 PID: 74 Comm: trinity-main Not tainted 3.15.0-rc8-06195-g939f04b #2
00000000 81c3a310 8b995c14 81426f69 8b995c44 81425a99 8161f671 8161f570
8161f538 8161f559 8161f538 8b995c78 8b142bb0 00000004 8b142fdc 8b142bb0
8b995ca8 8104a62d 8b142fac 000016f2 81c3a310 00000001 00000001 00000003
Call Trace:
[<81426f69>] dump_stack+0x16/0x18
[<81425a99>] print_circular_bug+0x18f/0x19c
[<8104a62d>] __lock_acquire+0x9ea/0xc6d
[<8104a942>] lock_acquire+0x92/0x101
[<811c60be>] ? serial8250_console_write+0x8c/0x10c
[<811c6032>] ? wait_for_xmitr+0x76/0x76
[<8142f11d>] _raw_spin_lock_irqsave+0x2e/0x3e
[<811c60be>] ? serial8250_console_write+0x8c/0x10c
[<811c60be>] serial8250_console_write+0x8c/0x10c
[<8104af87>] ? lock_release+0x191/0x223
[<811c6032>] ? wait_for_xmitr+0x76/0x76
[<8104e402>] call_console_drivers.constprop.31+0x87/0x118
[<8104f5d5>] console_unlock+0x1d7/0x398
[<8104fb70>] vprintk_emit+0x3da/0x3e4
[<81425f76>] printk+0x17/0x19
[<8105bfa0>] clockevents_program_min_delta+0x104/0x116
[<8105cc1c>] tick_program_event+0x1e/0x23
[<8103c43c>] hrtimer_force_reprogram+0x88/0x8f
[<8103c49e>] __remove_hrtimer+0x5b/0x79
[<8103cb21>] hrtimer_try_to_cancel+0x49/0x66
[<8103cb4b>] hrtimer_cancel+0xd/0x18
[<8107f102>] perf_swevent_cancel_hrtimer.part.60+0x2b/0x30
[<81080705>] task_clock_event_stop+0x20/0x64
[<81080756>] task_clock_event_del+0xd/0xf
[<81081350>] event_sched_out+0xab/0x11e
[<810813e0>] group_sched_out+0x1d/0x66
[<81081682>] ctx_sched_out+0xaf/0xbf
[<81081e04>] __perf_event_task_sched_out+0x1ed/0x34f
[<8104416d>] ? __dequeue_entity+0x23/0x27
[<81044505>] ? pick_next_task_fair+0xb1/0x120
[<8142cacc>] __schedule+0x4c6/0x4cb
[<81047574>] ? trace_hardirqs_off_caller+0xd7/0x108
[<810475b0>] ? trace_hardirqs_off+0xb/0xd
[<81056346>] ? rcu_irq_exit+0x64/0x77
Fix the problem by using printk_deferred() which does not call into the
scheduler.
Reported-by: Fengguang Wu <fengguang.wu@intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
kernel/time/clockevents.c | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
--- a/kernel/time/clockevents.c
+++ b/kernel/time/clockevents.c
@@ -146,7 +146,8 @@ static int clockevents_increase_min_delt
{
/* Nothing to do if we already reached the limit */
if (dev->min_delta_ns >= MIN_DELTA_LIMIT) {
- printk(KERN_WARNING "CE: Reprogramming failure. Giving up\n");
+ printk_deferred(KERN_WARNING
+ "CE: Reprogramming failure. Giving up\n");
dev->next_event.tv64 = KTIME_MAX;
return -ETIME;
}
@@ -159,9 +160,10 @@ static int clockevents_increase_min_delt
if (dev->min_delta_ns > MIN_DELTA_LIMIT)
dev->min_delta_ns = MIN_DELTA_LIMIT;
- printk(KERN_WARNING "CE: %s increased min_delta_ns to %llu nsec\n",
- dev->name ? dev->name : "?",
- (unsigned long long) dev->min_delta_ns);
+ printk_deferred(KERN_WARNING
+ "CE: %s increased min_delta_ns to %llu nsec\n",
+ dev->name ? dev->name : "?",
+ (unsigned long long) dev->min_delta_ns);
return 0;
}
next prev parent reply other threads:[~2014-08-05 18:38 UTC|newest]
Thread overview: 42+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-08-05 18:13 [PATCH 3.14 00/39] 3.14.16-stable review Greg Kroah-Hartman
2014-08-05 18:13 ` [PATCH 3.14 01/39] crypto: arm-aes - fix encryption of unaligned data Greg Kroah-Hartman
2014-08-05 18:13 ` [PATCH 3.14 02/39] crypto: af_alg - properly label AF_ALG socket Greg Kroah-Hartman
2014-08-05 18:13 ` [PATCH 3.14 03/39] ARM: dts: fix L2 address in Hi3620 Greg Kroah-Hartman
2014-08-05 18:13 ` [PATCH 3.14 04/39] ARM: fix alignment of keystone page table fixup Greg Kroah-Hartman
2014-08-05 18:13 ` [PATCH 3.14 05/39] ARM: 8115/1: LPAE: reduce damage caused by idmap to virtual memory layout Greg Kroah-Hartman
2014-08-05 18:13 ` [PATCH 3.14 06/39] ath9k: fix aggregation session lockup Greg Kroah-Hartman
2014-08-05 18:13 ` [PATCH 3.14 07/39] cfg80211: fix mic_failure tracing Greg Kroah-Hartman
2014-08-05 18:13 ` [PATCH 3.14 08/39] rapidio/tsi721_dma: fix failure to obtain transaction descriptor Greg Kroah-Hartman
2014-08-05 18:13 ` [PATCH 3.14 09/39] scsi: handle flush errors properly Greg Kroah-Hartman
2014-08-05 18:13 ` [PATCH 3.14 10/39] mm/page-writeback.c: fix divide by zero in bdi_dirty_limits() Greg Kroah-Hartman
2014-08-05 18:13 ` [PATCH 3.14 11/39] mm, thp: do not allow thp faults to avoid cpuset restrictions Greg Kroah-Hartman
2014-08-05 18:13 ` [PATCH 3.14 12/39] memcg: oom_notify use-after-free fix Greg Kroah-Hartman
2014-08-05 18:13 ` [PATCH 3.14 13/39] staging: vt6655: Fix disassociated messages every 10 seconds Greg Kroah-Hartman
2014-08-05 18:13 ` [PATCH 3.14 14/39] ACPI / PNP: Fix acpi_pnp_match() Greg Kroah-Hartman
2014-08-05 18:13 ` [PATCH 3.14 15/39] iio:bma180: Fix scale factors to report correct acceleration units Greg Kroah-Hartman
2014-08-05 18:13 ` [PATCH 3.14 16/39] iio:bma180: Missing check for frequency fractional part Greg Kroah-Hartman
2014-08-05 18:13 ` [PATCH 3.14 17/39] iio: buffer: Fix demux table creation Greg Kroah-Hartman
2014-08-05 18:13 ` [PATCH 3.14 18/39] dm bufio: fully initialize shrinker Greg Kroah-Hartman
2014-08-05 18:13 ` [PATCH 3.14 19/39] dm cache: fix race affecting dirty block count Greg Kroah-Hartman
2014-08-05 18:13 ` [PATCH 3.14 20/39] printk: rename printk_sched to printk_deferred Greg Kroah-Hartman
2014-08-05 18:13 ` [PATCH 3.14 21/39] sched_clock: Avoid corrupting hrtimer tree during suspend Greg Kroah-Hartman
2014-08-05 18:13 ` Greg Kroah-Hartman [this message]
2014-08-05 18:14 ` [PATCH 3.14 23/39] Revert "x86-64, modify_ldt: Make support for 16-bit segments a runtime option" Greg Kroah-Hartman
2014-08-05 18:14 ` [PATCH 3.14 24/39] x86-64, espfix: Dont leak bits 31:16 of %esp returning to 16-bit stack Greg Kroah-Hartman
2014-08-05 18:14 ` [PATCH 3.14 25/39] x86, espfix: Move espfix definitions into a separate header file Greg Kroah-Hartman
2014-08-05 18:14 ` [PATCH 3.14 26/39] x86, espfix: Fix broken header guard Greg Kroah-Hartman
2014-08-05 18:14 ` [PATCH 3.14 27/39] x86, espfix: Make espfix64 a Kconfig option, fix UML Greg Kroah-Hartman
2014-08-05 18:14 ` [PATCH 3.14 28/39] x86, espfix: Make it possible to disable 16-bit support Greg Kroah-Hartman
2014-08-05 18:14 ` [PATCH 3.14 29/39] x86_64/entry/xen: Do not invoke espfix64 on Xen Greg Kroah-Hartman
2014-08-05 18:14 ` [PATCH 3.14 30/39] pinctrl: dra: dt-bindings: Fix pull enable/disable Greg Kroah-Hartman
2014-08-05 18:14 ` [PATCH 3.14 31/39] ARM: dts: dra7-evm: Make VDDA_1V8_PHY supply always on Greg Kroah-Hartman
2014-08-05 18:14 ` [PATCH 3.14 32/39] staging: vt6655: Fix Warning on boot handle_irq_event_percpu Greg Kroah-Hartman
2014-08-05 18:14 ` [PATCH 3.14 33/39] Revert "mac80211: move "bufferable MMPDU" check to fix AP mode scan" Greg Kroah-Hartman
2014-08-05 18:14 ` [PATCH 3.14 34/39] cpufreq: move policy kobj to policy->cpu at resume Greg Kroah-Hartman
2014-08-05 18:14 ` [PATCH 3.14 35/39] x86/xen: no need to explicitly register an NMI callback Greg Kroah-Hartman
2014-08-05 18:14 ` [PATCH 3.14 36/39] xtensa: add fixup for double exception raised in window overflow Greg Kroah-Hartman
2014-08-05 18:14 ` [PATCH 3.14 37/39] net/l2tp: dont fall back on UDP [get|set]sockopt Greg Kroah-Hartman
2014-08-05 18:14 ` [PATCH 3.14 38/39] lib/btree.c: fix leak of whole btree nodes Greg Kroah-Hartman
2014-08-05 18:14 ` [PATCH 3.14 39/39] x86/espfix/xen: Fix allocation of pages for paravirt page tables Greg Kroah-Hartman
2014-08-05 23:07 ` [PATCH 3.14 00/39] 3.14.16-stable review Shuah Khan
2014-08-06 2:35 ` Guenter Roeck
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20140805181336.714330580@linuxfoundation.org \
--to=gregkh@linuxfoundation.org \
--cc=fengguang.wu@intel.com \
--cc=jack@suse.cz \
--cc=linux-kernel@vger.kernel.org \
--cc=stable@vger.kernel.org \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).