From: Frederic Weisbecker <fweisbec@gmail.com>
To: Thomas Gleixner <tglx@linutronix.de>
Cc: Dave Jones <davej@redhat.com>,
linux-mm@kvack.org, Peter Zijlstra <peterz@infradead.org>,
LKML <linux-kernel@vger.kernel.org>
Subject: Re: hrtimer deadlock caused by nohz_full
Date: Fri, 26 Sep 2014 01:46:59 +0200 [thread overview]
Message-ID: <20140925234657.GB14870@lerouge> (raw)
In-Reply-To: <alpine.DEB.2.10.1409251630210.4604@nanos>
On Thu, Sep 25, 2014 at 04:35:49PM +0200, Thomas Gleixner wrote:
> On Thu, 25 Sep 2014, Dave Jones wrote:
>
> > Got this on a box that had been fuzzing for 12 hours or so.
> > There's also some timer stuff going on htere, so cc'ing the usual suspects.
>
> And it's a hrtimer lockup
>
> > [<ffffffffa10f95f8>] ? hrtimer_try_to_cancel+0x58/0x1f0
> > [<ffffffffa10d103d>] ? lock_release+0x1d/0x300
> > [<ffffffffa10d103d>] ? lock_release+0x1d/0x300
> > [<ffffffffa10f95f8>] ? hrtimer_try_to_cancel+0x58/0x1f0
> > [<ffffffffa10d103d>] ? lock_release+0x1d/0x300
> > <<EOE>> <IRQ> [<ffffffffa1823ac4>] _raw_spin_unlock_irqrestore+0x24/0x70
> > [<ffffffffa10f95f8>] hrtimer_try_to_cancel+0x58/0x1f0
> > [<ffffffffa10f97aa>] hrtimer_cancel+0x1a/0x30
> > [<ffffffffa110a0e7>] tick_nohz_restart+0x17/0x90
> > [<ffffffffa110af38>] __tick_nohz_full_check+0xc8/0xe0
> > [<ffffffffa110af5e>] nohz_full_kick_work_func+0xe/0x10
> > [<ffffffffa117c9bf>] irq_work_run_list+0x4f/0x70
> > [<ffffffffa117ca0a>] irq_work_run+0x2a/0x60
> > [<ffffffffa10f82eb>] update_process_times+0x5b/0x70
> > [<ffffffffa1109dc5>] tick_sched_handle.isra.21+0x25/0x60
> > [<ffffffffa110a0b1>] tick_sched_timer+0x41/0x60
> > [<ffffffffa10f8c71>] __run_hrtimer+0x81/0x480
> > [<ffffffffa110a070>] ? tick_sched_do_timer+0x90/0x90
> > [<ffffffffa10f9b27>] hrtimer_interrupt+0x107/0x260
> > [<ffffffffa10331a4>] local_apic_timer_interrupt+0x34/0x60
> > [<ffffffffa182734f>] smp_apic_timer_interrupt+0x3f/0x60
> > [<ffffffffa182576f>] apic_timer_interrupt+0x6f/0x80
>
> hrtimer_interrupt
> tick_sched_timer
> tick_sched_handle
> update_process_times
> irq_work_run
> irq_work_run_list
> nohz_full_kick_work_func
> __tick_nohz_full_check
> tick_nohz_restart
> hrtimer_cancel
>
> And that hrtimer_cancel is:
>
> static void tick_nohz_restart(struct tick_sched *ts, ktime_t now)
> {
> hrtimer_cancel(&ts->sched_timer);
>
> Now, that's really bad because we are in the timer callback of
> ts->sched_timer. So hrtimer_cancel will loop forever waiting for the
> callback to complete.
>
> Frederic !?!?
Right, this patchset fixes it: "[PATCH 0/8] nohz: Fix nohz kick irq work on tick v3"
I was about to make the pull request, the branch is acked by peterz.
Would you like to pull it? It's all merge window material.
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
nohz/fixes-v3
Thanks.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
WARNING: multiple messages have this Message-ID (diff)
From: Frederic Weisbecker <fweisbec@gmail.com>
To: Thomas Gleixner <tglx@linutronix.de>
Cc: Dave Jones <davej@redhat.com>,
linux-mm@kvack.org, Peter Zijlstra <peterz@infradead.org>,
LKML <linux-kernel@vger.kernel.org>
Subject: Re: hrtimer deadlock caused by nohz_full
Date: Fri, 26 Sep 2014 01:46:59 +0200 [thread overview]
Message-ID: <20140925234657.GB14870@lerouge> (raw)
In-Reply-To: <alpine.DEB.2.10.1409251630210.4604@nanos>
On Thu, Sep 25, 2014 at 04:35:49PM +0200, Thomas Gleixner wrote:
> On Thu, 25 Sep 2014, Dave Jones wrote:
>
> > Got this on a box that had been fuzzing for 12 hours or so.
> > There's also some timer stuff going on htere, so cc'ing the usual suspects.
>
> And it's a hrtimer lockup
>
> > [<ffffffffa10f95f8>] ? hrtimer_try_to_cancel+0x58/0x1f0
> > [<ffffffffa10d103d>] ? lock_release+0x1d/0x300
> > [<ffffffffa10d103d>] ? lock_release+0x1d/0x300
> > [<ffffffffa10f95f8>] ? hrtimer_try_to_cancel+0x58/0x1f0
> > [<ffffffffa10d103d>] ? lock_release+0x1d/0x300
> > <<EOE>> <IRQ> [<ffffffffa1823ac4>] _raw_spin_unlock_irqrestore+0x24/0x70
> > [<ffffffffa10f95f8>] hrtimer_try_to_cancel+0x58/0x1f0
> > [<ffffffffa10f97aa>] hrtimer_cancel+0x1a/0x30
> > [<ffffffffa110a0e7>] tick_nohz_restart+0x17/0x90
> > [<ffffffffa110af38>] __tick_nohz_full_check+0xc8/0xe0
> > [<ffffffffa110af5e>] nohz_full_kick_work_func+0xe/0x10
> > [<ffffffffa117c9bf>] irq_work_run_list+0x4f/0x70
> > [<ffffffffa117ca0a>] irq_work_run+0x2a/0x60
> > [<ffffffffa10f82eb>] update_process_times+0x5b/0x70
> > [<ffffffffa1109dc5>] tick_sched_handle.isra.21+0x25/0x60
> > [<ffffffffa110a0b1>] tick_sched_timer+0x41/0x60
> > [<ffffffffa10f8c71>] __run_hrtimer+0x81/0x480
> > [<ffffffffa110a070>] ? tick_sched_do_timer+0x90/0x90
> > [<ffffffffa10f9b27>] hrtimer_interrupt+0x107/0x260
> > [<ffffffffa10331a4>] local_apic_timer_interrupt+0x34/0x60
> > [<ffffffffa182734f>] smp_apic_timer_interrupt+0x3f/0x60
> > [<ffffffffa182576f>] apic_timer_interrupt+0x6f/0x80
>
> hrtimer_interrupt
> tick_sched_timer
> tick_sched_handle
> update_process_times
> irq_work_run
> irq_work_run_list
> nohz_full_kick_work_func
> __tick_nohz_full_check
> tick_nohz_restart
> hrtimer_cancel
>
> And that hrtimer_cancel is:
>
> static void tick_nohz_restart(struct tick_sched *ts, ktime_t now)
> {
> hrtimer_cancel(&ts->sched_timer);
>
> Now, that's really bad because we are in the timer callback of
> ts->sched_timer. So hrtimer_cancel will loop forever waiting for the
> callback to complete.
>
> Frederic !?!?
Right, this patchset fixes it: "[PATCH 0/8] nohz: Fix nohz kick irq work on tick v3"
I was about to make the pull request, the branch is acked by peterz.
Would you like to pull it? It's all merge window material.
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
nohz/fixes-v3
Thanks.
next prev parent reply other threads:[~2014-09-25 23:47 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-09-25 14:14 watchdog kicked in while shrinking inactive list Dave Jones
2014-09-25 14:35 ` hrtimer deadlock caused by nohz_full Thomas Gleixner
2014-09-25 14:35 ` Thomas Gleixner
2014-09-25 23:46 ` Frederic Weisbecker [this message]
2014-09-25 23:46 ` Frederic Weisbecker
2014-10-02 18:24 ` Dave Jones
2014-10-02 22:01 ` Mike Lothian
2014-10-07 15:30 ` Frederic Weisbecker
2014-10-07 17:37 ` Dave Jones
2014-10-07 17:42 ` Thomas Gleixner
2014-10-09 2:02 ` Dave Jones
2014-10-09 3:30 ` Frederic Weisbecker
2014-10-09 3:49 ` Dave Jones
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20140925234657.GB14870@lerouge \
--to=fweisbec@gmail.com \
--cc=davej@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=peterz@infradead.org \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.