Re: hrtimer deadlock caused by nohz_full

Re: hrtimer deadlock caused by nohz_full

[Thomas Gleixner]

On Thu, 25 Sep 2014, Dave Jones wrote:

> Got this on a box that had been fuzzing for 12 hours or so.
> There's also some timer stuff going on htere, so cc'ing the usual suspects.

And it's a hrtimer lockup
 
>  [<ffffffffa10f95f8>] ? hrtimer_try_to_cancel+0x58/0x1f0
>  [<ffffffffa10d103d>] ? lock_release+0x1d/0x300
>  [<ffffffffa10d103d>] ? lock_release+0x1d/0x300
>  [<ffffffffa10f95f8>] ? hrtimer_try_to_cancel+0x58/0x1f0
>  [<ffffffffa10d103d>] ? lock_release+0x1d/0x300
>  <<EOE>>  <IRQ>  [<ffffffffa1823ac4>] _raw_spin_unlock_irqrestore+0x24/0x70
>  [<ffffffffa10f95f8>] hrtimer_try_to_cancel+0x58/0x1f0
>  [<ffffffffa10f97aa>] hrtimer_cancel+0x1a/0x30
>  [<ffffffffa110a0e7>] tick_nohz_restart+0x17/0x90
>  [<ffffffffa110af38>] __tick_nohz_full_check+0xc8/0xe0
>  [<ffffffffa110af5e>] nohz_full_kick_work_func+0xe/0x10
>  [<ffffffffa117c9bf>] irq_work_run_list+0x4f/0x70
>  [<ffffffffa117ca0a>] irq_work_run+0x2a/0x60
>  [<ffffffffa10f82eb>] update_process_times+0x5b/0x70
>  [<ffffffffa1109dc5>] tick_sched_handle.isra.21+0x25/0x60
>  [<ffffffffa110a0b1>] tick_sched_timer+0x41/0x60
>  [<ffffffffa10f8c71>] __run_hrtimer+0x81/0x480
>  [<ffffffffa110a070>] ? tick_sched_do_timer+0x90/0x90
>  [<ffffffffa10f9b27>] hrtimer_interrupt+0x107/0x260
>  [<ffffffffa10331a4>] local_apic_timer_interrupt+0x34/0x60
>  [<ffffffffa182734f>] smp_apic_timer_interrupt+0x3f/0x60
>  [<ffffffffa182576f>] apic_timer_interrupt+0x6f/0x80

hrtimer_interrupt
  tick_sched_timer
    tick_sched_handle
      update_process_times
        irq_work_run
	  irq_work_run_list
	    nohz_full_kick_work_func
	      __tick_nohz_full_check
	        tick_nohz_restart
                  hrtimer_cancel

And that hrtimer_cancel is:

static void tick_nohz_restart(struct tick_sched *ts, ktime_t now)
{
	hrtimer_cancel(&ts->sched_timer);

Now, that's really bad because we are in the timer callback of
ts->sched_timer. So hrtimer_cancel will loop forever waiting for the
callback to complete.

Frederic !?!?

Thanks,

	tglx

Re: hrtimer deadlock caused by nohz_full

[Frederic Weisbecker]

On Thu, Sep 25, 2014 at 04:35:49PM +0200, Thomas Gleixner wrote:
> On Thu, 25 Sep 2014, Dave Jones wrote:
> 
> > Got this on a box that had been fuzzing for 12 hours or so.
> > There's also some timer stuff going on htere, so cc'ing the usual suspects.
> 
> And it's a hrtimer lockup
>  
> >  [<ffffffffa10f95f8>] ? hrtimer_try_to_cancel+0x58/0x1f0
> >  [<ffffffffa10d103d>] ? lock_release+0x1d/0x300
> >  [<ffffffffa10d103d>] ? lock_release+0x1d/0x300
> >  [<ffffffffa10f95f8>] ? hrtimer_try_to_cancel+0x58/0x1f0
> >  [<ffffffffa10d103d>] ? lock_release+0x1d/0x300
> >  <<EOE>>  <IRQ>  [<ffffffffa1823ac4>] _raw_spin_unlock_irqrestore+0x24/0x70
> >  [<ffffffffa10f95f8>] hrtimer_try_to_cancel+0x58/0x1f0
> >  [<ffffffffa10f97aa>] hrtimer_cancel+0x1a/0x30
> >  [<ffffffffa110a0e7>] tick_nohz_restart+0x17/0x90
> >  [<ffffffffa110af38>] __tick_nohz_full_check+0xc8/0xe0
> >  [<ffffffffa110af5e>] nohz_full_kick_work_func+0xe/0x10
> >  [<ffffffffa117c9bf>] irq_work_run_list+0x4f/0x70
> >  [<ffffffffa117ca0a>] irq_work_run+0x2a/0x60
> >  [<ffffffffa10f82eb>] update_process_times+0x5b/0x70
> >  [<ffffffffa1109dc5>] tick_sched_handle.isra.21+0x25/0x60
> >  [<ffffffffa110a0b1>] tick_sched_timer+0x41/0x60
> >  [<ffffffffa10f8c71>] __run_hrtimer+0x81/0x480
> >  [<ffffffffa110a070>] ? tick_sched_do_timer+0x90/0x90
> >  [<ffffffffa10f9b27>] hrtimer_interrupt+0x107/0x260
> >  [<ffffffffa10331a4>] local_apic_timer_interrupt+0x34/0x60
> >  [<ffffffffa182734f>] smp_apic_timer_interrupt+0x3f/0x60
> >  [<ffffffffa182576f>] apic_timer_interrupt+0x6f/0x80
> 
> hrtimer_interrupt
>   tick_sched_timer
>     tick_sched_handle
>       update_process_times
>         irq_work_run
> 	  irq_work_run_list
> 	    nohz_full_kick_work_func
> 	      __tick_nohz_full_check
> 	        tick_nohz_restart
>                   hrtimer_cancel
> 
> And that hrtimer_cancel is:
> 
> static void tick_nohz_restart(struct tick_sched *ts, ktime_t now)
> {
> 	hrtimer_cancel(&ts->sched_timer);
> 
> Now, that's really bad because we are in the timer callback of
> ts->sched_timer. So hrtimer_cancel will loop forever waiting for the
> callback to complete.
> 
> Frederic !?!?

Right, this patchset fixes it: "[PATCH 0/8] nohz: Fix nohz kick irq work on tick v3"

I was about to make the pull request, the branch is acked by peterz.
Would you like to pull it? It's all merge window material.

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
	nohz/fixes-v3

Thanks.

Re: hrtimer deadlock caused by nohz_full

[Dave Jones]

On Fri, Sep 26, 2014 at 01:46:59AM +0200, Frederic Weisbecker wrote:

 > > >  [<ffffffffa10f95f8>] hrtimer_try_to_cancel+0x58/0x1f0
 > > >  [<ffffffffa10f97aa>] hrtimer_cancel+0x1a/0x30
 > > >  [<ffffffffa110a0e7>] tick_nohz_restart+0x17/0x90
 > > >  [<ffffffffa110af38>] __tick_nohz_full_check+0xc8/0xe0
 > > >  [<ffffffffa110af5e>] nohz_full_kick_work_func+0xe/0x10
 > > >  [<ffffffffa117c9bf>] irq_work_run_list+0x4f/0x70
 > > >  [<ffffffffa117ca0a>] irq_work_run+0x2a/0x60
 > > >  [<ffffffffa10f82eb>] update_process_times+0x5b/0x70
 > > >  [<ffffffffa1109dc5>] tick_sched_handle.isra.21+0x25/0x60
 > > >  [<ffffffffa110a0b1>] tick_sched_timer+0x41/0x60
 > > >  [<ffffffffa10f8c71>] __run_hrtimer+0x81/0x480
 > > >  [<ffffffffa110a070>] ? tick_sched_do_timer+0x90/0x90
 > > >  [<ffffffffa10f9b27>] hrtimer_interrupt+0x107/0x260
 > > >  [<ffffffffa10331a4>] local_apic_timer_interrupt+0x34/0x60
 > > >  [<ffffffffa182734f>] smp_apic_timer_interrupt+0x3f/0x60
 > > >  [<ffffffffa182576f>] apic_timer_interrupt+0x6f/0x80
 > > 
 > > hrtimer_interrupt
 > >   tick_sched_timer
 > >     tick_sched_handle
 > >       update_process_times
 > >         irq_work_run
 > > 	  irq_work_run_list
 > > 	    nohz_full_kick_work_func
 > > 	      __tick_nohz_full_check
 > > 	        tick_nohz_restart
 > >                   hrtimer_cancel
 > > 
 > > And that hrtimer_cancel is:
 > > 
 > > static void tick_nohz_restart(struct tick_sched *ts, ktime_t now)
 > > {
 > > 	hrtimer_cancel(&ts->sched_timer);
 > > 
 > > Now, that's really bad because we are in the timer callback of
 > > ts->sched_timer. So hrtimer_cancel will loop forever waiting for the
 > > callback to complete.
 > > 
 > > Frederic !?!?
 > 
 > Right, this patchset fixes it: "[PATCH 0/8] nohz: Fix nohz kick irq work on tick v3"
 > 
 > I was about to make the pull request, the branch is acked by peterz.
 > Would you like to pull it? It's all merge window material.
 > 
 > git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
 > 	nohz/fixes-v3

I'm now hitting this with such regularity that it's preventing me from
tracking down other bugs.  Can we get this merged soon please ?

	Dave

Re: hrtimer deadlock caused by nohz_full

[Mike Lothian]

I think this might be the bug I reported here
https://bugzilla.kernel.org/show_bug.cgi?id=84131

On 2 October 2014 19:24, Dave Jones <davej@redhat.com> wrote:
> On Fri, Sep 26, 2014 at 01:46:59AM +0200, Frederic Weisbecker wrote:
>
>  > > >  [<ffffffffa10f95f8>] hrtimer_try_to_cancel+0x58/0x1f0
>  > > >  [<ffffffffa10f97aa>] hrtimer_cancel+0x1a/0x30
>  > > >  [<ffffffffa110a0e7>] tick_nohz_restart+0x17/0x90
>  > > >  [<ffffffffa110af38>] __tick_nohz_full_check+0xc8/0xe0
>  > > >  [<ffffffffa110af5e>] nohz_full_kick_work_func+0xe/0x10
>  > > >  [<ffffffffa117c9bf>] irq_work_run_list+0x4f/0x70
>  > > >  [<ffffffffa117ca0a>] irq_work_run+0x2a/0x60
>  > > >  [<ffffffffa10f82eb>] update_process_times+0x5b/0x70
>  > > >  [<ffffffffa1109dc5>] tick_sched_handle.isra.21+0x25/0x60
>  > > >  [<ffffffffa110a0b1>] tick_sched_timer+0x41/0x60
>  > > >  [<ffffffffa10f8c71>] __run_hrtimer+0x81/0x480
>  > > >  [<ffffffffa110a070>] ? tick_sched_do_timer+0x90/0x90
>  > > >  [<ffffffffa10f9b27>] hrtimer_interrupt+0x107/0x260
>  > > >  [<ffffffffa10331a4>] local_apic_timer_interrupt+0x34/0x60
>  > > >  [<ffffffffa182734f>] smp_apic_timer_interrupt+0x3f/0x60
>  > > >  [<ffffffffa182576f>] apic_timer_interrupt+0x6f/0x80
>  > >
>  > > hrtimer_interrupt
>  > >   tick_sched_timer
>  > >     tick_sched_handle
>  > >       update_process_times
>  > >         irq_work_run
>  > >      irq_work_run_list
>  > >        nohz_full_kick_work_func
>  > >          __tick_nohz_full_check
>  > >            tick_nohz_restart
>  > >                   hrtimer_cancel
>  > >
>  > > And that hrtimer_cancel is:
>  > >
>  > > static void tick_nohz_restart(struct tick_sched *ts, ktime_t now)
>  > > {
>  > >    hrtimer_cancel(&ts->sched_timer);
>  > >
>  > > Now, that's really bad because we are in the timer callback of
>  > > ts->sched_timer. So hrtimer_cancel will loop forever waiting for the
>  > > callback to complete.
>  > >
>  > > Frederic !?!?
>  >
>  > Right, this patchset fixes it: "[PATCH 0/8] nohz: Fix nohz kick irq work on tick v3"
>  >
>  > I was about to make the pull request, the branch is acked by peterz.
>  > Would you like to pull it? It's all merge window material.
>  >
>  > git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
>  >      nohz/fixes-v3
>
> I'm now hitting this with such regularity that it's preventing me from
> tracking down other bugs.  Can we get this merged soon please ?
>
>         Dave
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: hrtimer deadlock caused by nohz_full

[Frederic Weisbecker]

On Thu, Oct 02, 2014 at 02:24:09PM -0400, Dave Jones wrote:
> On Fri, Sep 26, 2014 at 01:46:59AM +0200, Frederic Weisbecker wrote:
> 
>  > > >  [<ffffffffa10f95f8>] hrtimer_try_to_cancel+0x58/0x1f0
>  > > >  [<ffffffffa10f97aa>] hrtimer_cancel+0x1a/0x30
>  > > >  [<ffffffffa110a0e7>] tick_nohz_restart+0x17/0x90
>  > > >  [<ffffffffa110af38>] __tick_nohz_full_check+0xc8/0xe0
>  > > >  [<ffffffffa110af5e>] nohz_full_kick_work_func+0xe/0x10
>  > > >  [<ffffffffa117c9bf>] irq_work_run_list+0x4f/0x70
>  > > >  [<ffffffffa117ca0a>] irq_work_run+0x2a/0x60
>  > > >  [<ffffffffa10f82eb>] update_process_times+0x5b/0x70
>  > > >  [<ffffffffa1109dc5>] tick_sched_handle.isra.21+0x25/0x60
>  > > >  [<ffffffffa110a0b1>] tick_sched_timer+0x41/0x60
>  > > >  [<ffffffffa10f8c71>] __run_hrtimer+0x81/0x480
>  > > >  [<ffffffffa110a070>] ? tick_sched_do_timer+0x90/0x90
>  > > >  [<ffffffffa10f9b27>] hrtimer_interrupt+0x107/0x260
>  > > >  [<ffffffffa10331a4>] local_apic_timer_interrupt+0x34/0x60
>  > > >  [<ffffffffa182734f>] smp_apic_timer_interrupt+0x3f/0x60
>  > > >  [<ffffffffa182576f>] apic_timer_interrupt+0x6f/0x80
>  > > 
>  > > hrtimer_interrupt
>  > >   tick_sched_timer
>  > >     tick_sched_handle
>  > >       update_process_times
>  > >         irq_work_run
>  > > 	  irq_work_run_list
>  > > 	    nohz_full_kick_work_func
>  > > 	      __tick_nohz_full_check
>  > > 	        tick_nohz_restart
>  > >                   hrtimer_cancel
>  > > 
>  > > And that hrtimer_cancel is:
>  > > 
>  > > static void tick_nohz_restart(struct tick_sched *ts, ktime_t now)
>  > > {
>  > > 	hrtimer_cancel(&ts->sched_timer);
>  > > 
>  > > Now, that's really bad because we are in the timer callback of
>  > > ts->sched_timer. So hrtimer_cancel will loop forever waiting for the
>  > > callback to complete.
>  > > 
>  > > Frederic !?!?
>  > 
>  > Right, this patchset fixes it: "[PATCH 0/8] nohz: Fix nohz kick irq work on tick v3"
>  > 
>  > I was about to make the pull request, the branch is acked by peterz.
>  > Would you like to pull it? It's all merge window material.
>  > 
>  > git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
>  > 	nohz/fixes-v3
> 
> I'm now hitting this with such regularity that it's preventing me from
> tracking down other bugs.  Can we get this merged soon please ?

I asked Thomas to pull but he probably missed it. So I did a formal pull request
to Ingo a few days later but he told me that the pull request was made too late
as the merge window was too close. I told him it fixes annoying bugs but that was on IRC
so this probably got lost.

It will probably be delayed to 3.18
Sorry I don't have much control myself on merging patches upstream.

Re: hrtimer deadlock caused by nohz_full

[Dave Jones]

On Tue, Oct 07, 2014 at 05:30:27PM +0200, Frederic Weisbecker wrote:

 > >  > Right, this patchset fixes it: "[PATCH 0/8] nohz: Fix nohz kick irq work on tick v3"
 > >  > 
 > >  > I was about to make the pull request, the branch is acked by peterz.
 > >  > Would you like to pull it? It's all merge window material.
 > >  > 
 > >  > git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
 > >  > 	nohz/fixes-v3
 > > 
 > > I'm now hitting this with such regularity that it's preventing me from
 > > tracking down other bugs.  Can we get this merged soon please ?
 > 
 > I asked Thomas to pull but he probably missed it. So I did a formal pull request
 > to Ingo a few days later but he told me that the pull request was made too late
 > as the merge window was too close. I told him it fixes annoying bugs but that was on IRC
 > so this probably got lost.
 > 
 > It will probably be delayed to 3.18
 > Sorry I don't have much control myself on merging patches upstream.

I got fed up of hitting it this morning and pull this into 3.17, and got
a bunch of rejects, as that tree is against rc5.  Is there an updated
patchset somewhere else?

	Dave

Re: hrtimer deadlock caused by nohz_full

[Thomas Gleixner]

On Tue, 7 Oct 2014, Dave Jones wrote:

> On Tue, Oct 07, 2014 at 05:30:27PM +0200, Frederic Weisbecker wrote:
> 
>  > >  > Right, this patchset fixes it: "[PATCH 0/8] nohz: Fix nohz kick irq work on tick v3"
>  > >  > 
>  > >  > I was about to make the pull request, the branch is acked by peterz.
>  > >  > Would you like to pull it? It's all merge window material.
>  > >  > 
>  > >  > git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
>  > >  > 	nohz/fixes-v3
>  > > 
>  > > I'm now hitting this with such regularity that it's preventing me from
>  > > tracking down other bugs.  Can we get this merged soon please ?
>  > 
>  > I asked Thomas to pull but he probably missed it. So I did a formal pull request
>  > to Ingo a few days later but he told me that the pull request was made too late
>  > as the merge window was too close. I told him it fixes annoying bugs but that was on IRC
>  > so this probably got lost.

Aarg. Lemme get this done.

Thanks,

	tglx

Re: hrtimer deadlock caused by nohz_full

[Dave Jones]

On Tue, Oct 07, 2014 at 07:42:47PM +0200, Thomas Gleixner wrote:
 > On Tue, 7 Oct 2014, Dave Jones wrote:
 > 
 > > On Tue, Oct 07, 2014 at 05:30:27PM +0200, Frederic Weisbecker wrote:
 > > 
 > >  > >  > Right, this patchset fixes it: "[PATCH 0/8] nohz: Fix nohz kick irq work on tick v3"
 > >  > >  > 
 > >  > >  > I was about to make the pull request, the branch is acked by peterz.
 > >  > >  > Would you like to pull it? It's all merge window material.
 > >  > >  > 
 > >  > >  > git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
 > >  > >  > 	nohz/fixes-v3
 > >  > > 
 > >  > > I'm now hitting this with such regularity that it's preventing me from
 > >  > > tracking down other bugs.  Can we get this merged soon please ?
 > >  > 
 > >  > I asked Thomas to pull but he probably missed it. So I did a formal pull request
 > >  > to Ingo a few days later but he told me that the pull request was made too late
 > >  > as the merge window was too close. I told him it fixes annoying bugs but that was on IRC
 > >  > so this probably got lost.
 > 
 > Aarg. Lemme get this done.

So, time for sad news.
This still doesn't solve the problem for me.

Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 2
CPU: 2 PID: 8193 Comm: kworker/u8:1 Not tainted 3.17.0+ #67
Workqueue: netns cleanup_net
 0000000000000000 000000002a473fb8 ffff880244c06c00 ffffffff9f81ea56
 ffffffff9fc63e00 ffff880244c06c80 ffffffff9f819003 0000000000000010
 ffff880244c06c90 ffff880244c06c30 000000002a473fb8 0000000000000000
Call Trace:
 <NMI>  [<ffffffff9f81ea56>] dump_stack+0x4e/0x7a
 [<ffffffff9f819003>] panic+0xd7/0x20a
 [<ffffffff9f149f30>] ? restart_watchdog_hrtimer+0x50/0x50
 [<ffffffff9f14a048>] watchdog_overflow_callback+0x118/0x120
 [<ffffffff9f18b69c>] __perf_event_overflow+0xac/0x350
 [<ffffffff9f019ece>] ? x86_perf_event_set_period+0xde/0x150
 [<ffffffff9f18c5b4>] perf_event_overflow+0x14/0x20
 [<ffffffff9f01fc26>] intel_pmu_handle_irq+0x206/0x410
 [<ffffffff9f018d2b>] perf_event_nmi_handler+0x2b/0x50
 [<ffffffff9f007752>] nmi_handle+0xd2/0x3b0
 [<ffffffff9f007685>] ? nmi_handle+0x5/0x3b0
 [<ffffffff9f0cef39>] ? get_lock_stats+0x19/0x60
 [<ffffffff9f007c62>] default_do_nmi+0x72/0x1c0
 [<ffffffff9f0fd318>] ? hrtimer_try_to_cancel+0x58/0x1f0
 [<ffffffff9f007e68>] do_nmi+0xb8/0xf0
 [<ffffffff9f82c9ea>] end_repeat_nmi+0x1e/0x2e
 [<ffffffff9f0fd318>] ? hrtimer_try_to_cancel+0x58/0x1f0
 [<ffffffff9f0cef39>] ? get_lock_stats+0x19/0x60
 [<ffffffff9f0cef39>] ? get_lock_stats+0x19/0x60
 [<ffffffff9f0cef39>] ? get_lock_stats+0x19/0x60
 <<EOE>>  <IRQ>  [<ffffffff9f0cf880>] lock_release_holdtime.part.29+0x50/0x160
 [<ffffffff9f0d4f75>] lock_release+0x235/0x300
 [<ffffffff9f829bf4>] _raw_spin_unlock_irqrestore+0x24/0x70
 [<ffffffff9f0fd318>] hrtimer_try_to_cancel+0x58/0x1f0
 [<ffffffff9f0fd4ca>] hrtimer_cancel+0x1a/0x30
 [<ffffffff9f10de07>] tick_nohz_restart+0x17/0x90
 [<ffffffff9f10ec58>] __tick_nohz_full_check+0xc8/0xe0
 [<ffffffff9f10ec7e>] nohz_full_kick_work_func+0xe/0x10
 [<ffffffff9f1806df>] irq_work_run_list+0x4f/0x70
 [<ffffffff9f18091b>] irq_work_tick+0x2b/0x50
 [<ffffffff9f0fc00b>] update_process_times+0x5b/0x70
 [<ffffffff9f10dae5>] tick_sched_handle.isra.21+0x25/0x60
 [<ffffffff9f10ddd1>] tick_sched_timer+0x41/0x60
 [<ffffffff9f0fc991>] __run_hrtimer+0x81/0x480
 [<ffffffff9f10dd90>] ? tick_sched_do_timer+0x90/0x90
 [<ffffffff9f0fd847>] hrtimer_interrupt+0x107/0x260
 [<ffffffff9f033274>] local_apic_timer_interrupt+0x34/0x60
 [<ffffffff9f82d48f>] smp_apic_timer_interrupt+0x3f/0x60
 [<ffffffff9f82b8af>] apic_timer_interrupt+0x6f/0x80
 <EOI>  [<ffffffffc0415436>] ? cache_clean+0x66/0x2f0 [sunrpc]
 [<ffffffffc04156f7>] cache_flush+0x37/0x70 [sunrpc]
 [<ffffffffc041577f>] cache_purge+0x4f/0x80 [sunrpc]
 [<ffffffffc040c80a>] svcauth_unix_purge+0xca/0x1c0 [sunrpc]
 [<ffffffffc040c745>] ? svcauth_unix_purge+0x5/0x1c0 [sunrpc]
 [<ffffffffc04d4f72>] nfsd_export_shutdown+0x112/0x250 [nfsd]
 [<ffffffffc04d4e65>] ? nfsd_export_shutdown+0x5/0x250 [nfsd]
 [<ffffffffc04c972a>] nfsd_exit_net+0x1a/0x20 [nfsd]
 [<ffffffff9f6cdf39>] ops_exit_list.isra.1+0x39/0x60
 [<ffffffff9f6ceae0>] cleanup_net+0x100/0x1e0
 [<ffffffff9f097b30>] process_one_work+0x1e0/0x6b0
 [<ffffffff9f097ac7>] ? process_one_work+0x177/0x6b0
 [<ffffffff9f09811b>] worker_thread+0x11b/0x490
 [<ffffffff9f098000>] ? process_one_work+0x6b0/0x6b0
 [<ffffffff9f09ee46>] kthread+0xf6/0x110
 [<ffffffff9f09ed50>] ? kthread_create_on_node+0x250/0x250
 [<ffffffff9f82a8ac>] ret_from_fork+0x7c/0xb0
 [<ffffffff9f09ed50>] ? kthread_create_on_node+0x250/0x250

Here's what I have on top of Linus' current tree..
http://paste.fedoraproject.org/140505/20075141/raw/

Do I miss some other essential patch ?

	Dave

Re: hrtimer deadlock caused by nohz_full

[Frederic Weisbecker]

On Wed, Oct 08, 2014 at 10:02:40PM -0400, Dave Jones wrote:
> On Tue, Oct 07, 2014 at 07:42:47PM +0200, Thomas Gleixner wrote:
>  > On Tue, 7 Oct 2014, Dave Jones wrote:
>  > 
>  > > On Tue, Oct 07, 2014 at 05:30:27PM +0200, Frederic Weisbecker wrote:
>  > > 
>  > >  > >  > Right, this patchset fixes it: "[PATCH 0/8] nohz: Fix nohz kick irq work on tick v3"
>  > >  > >  > 
>  > >  > >  > I was about to make the pull request, the branch is acked by peterz.
>  > >  > >  > Would you like to pull it? It's all merge window material.
>  > >  > >  > 
>  > >  > >  > git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
>  > >  > >  > 	nohz/fixes-v3
>  > >  > > 
>  > >  > > I'm now hitting this with such regularity that it's preventing me from
>  > >  > > tracking down other bugs.  Can we get this merged soon please ?
>  > >  > 
>  > >  > I asked Thomas to pull but he probably missed it. So I did a formal pull request
>  > >  > to Ingo a few days later but he told me that the pull request was made too late
>  > >  > as the merge window was too close. I told him it fixes annoying bugs but that was on IRC
>  > >  > so this probably got lost.
>  > 
>  > Aarg. Lemme get this done.
> 
> So, time for sad news.
> This still doesn't solve the problem for me.
> 
> Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 2
> CPU: 2 PID: 8193 Comm: kworker/u8:1 Not tainted 3.17.0+ #67
> Workqueue: netns cleanup_net
>  0000000000000000 000000002a473fb8 ffff880244c06c00 ffffffff9f81ea56
>  ffffffff9fc63e00 ffff880244c06c80 ffffffff9f819003 0000000000000010
>  ffff880244c06c90 ffff880244c06c30 000000002a473fb8 0000000000000000
> Call Trace:
>  <NMI>  [<ffffffff9f81ea56>] dump_stack+0x4e/0x7a
>  [<ffffffff9f819003>] panic+0xd7/0x20a
>  [<ffffffff9f149f30>] ? restart_watchdog_hrtimer+0x50/0x50
>  [<ffffffff9f14a048>] watchdog_overflow_callback+0x118/0x120
>  [<ffffffff9f18b69c>] __perf_event_overflow+0xac/0x350
>  [<ffffffff9f019ece>] ? x86_perf_event_set_period+0xde/0x150
>  [<ffffffff9f18c5b4>] perf_event_overflow+0x14/0x20
>  [<ffffffff9f01fc26>] intel_pmu_handle_irq+0x206/0x410
>  [<ffffffff9f018d2b>] perf_event_nmi_handler+0x2b/0x50
>  [<ffffffff9f007752>] nmi_handle+0xd2/0x3b0
>  [<ffffffff9f007685>] ? nmi_handle+0x5/0x3b0
>  [<ffffffff9f0cef39>] ? get_lock_stats+0x19/0x60
>  [<ffffffff9f007c62>] default_do_nmi+0x72/0x1c0
>  [<ffffffff9f0fd318>] ? hrtimer_try_to_cancel+0x58/0x1f0
>  [<ffffffff9f007e68>] do_nmi+0xb8/0xf0
>  [<ffffffff9f82c9ea>] end_repeat_nmi+0x1e/0x2e
>  [<ffffffff9f0fd318>] ? hrtimer_try_to_cancel+0x58/0x1f0
>  [<ffffffff9f0cef39>] ? get_lock_stats+0x19/0x60
>  [<ffffffff9f0cef39>] ? get_lock_stats+0x19/0x60
>  [<ffffffff9f0cef39>] ? get_lock_stats+0x19/0x60
>  <<EOE>>  <IRQ>  [<ffffffff9f0cf880>] lock_release_holdtime.part.29+0x50/0x160
>  [<ffffffff9f0d4f75>] lock_release+0x235/0x300
>  [<ffffffff9f829bf4>] _raw_spin_unlock_irqrestore+0x24/0x70
>  [<ffffffff9f0fd318>] hrtimer_try_to_cancel+0x58/0x1f0
>  [<ffffffff9f0fd4ca>] hrtimer_cancel+0x1a/0x30
>  [<ffffffff9f10de07>] tick_nohz_restart+0x17/0x90
>  [<ffffffff9f10ec58>] __tick_nohz_full_check+0xc8/0xe0
>  [<ffffffff9f10ec7e>] nohz_full_kick_work_func+0xe/0x10
>  [<ffffffff9f1806df>] irq_work_run_list+0x4f/0x70
>  [<ffffffff9f18091b>] irq_work_tick+0x2b/0x50
>  [<ffffffff9f0fc00b>] update_process_times+0x5b/0x70
>  [<ffffffff9f10dae5>] tick_sched_handle.isra.21+0x25/0x60
>  [<ffffffff9f10ddd1>] tick_sched_timer+0x41/0x60
>  [<ffffffff9f0fc991>] __run_hrtimer+0x81/0x480
>  [<ffffffff9f10dd90>] ? tick_sched_do_timer+0x90/0x90
>  [<ffffffff9f0fd847>] hrtimer_interrupt+0x107/0x260
>  [<ffffffff9f033274>] local_apic_timer_interrupt+0x34/0x60
>  [<ffffffff9f82d48f>] smp_apic_timer_interrupt+0x3f/0x60
>  [<ffffffff9f82b8af>] apic_timer_interrupt+0x6f/0x80
>  <EOI>  [<ffffffffc0415436>] ? cache_clean+0x66/0x2f0 [sunrpc]
>  [<ffffffffc04156f7>] cache_flush+0x37/0x70 [sunrpc]
>  [<ffffffffc041577f>] cache_purge+0x4f/0x80 [sunrpc]
>  [<ffffffffc040c80a>] svcauth_unix_purge+0xca/0x1c0 [sunrpc]
>  [<ffffffffc040c745>] ? svcauth_unix_purge+0x5/0x1c0 [sunrpc]
>  [<ffffffffc04d4f72>] nfsd_export_shutdown+0x112/0x250 [nfsd]
>  [<ffffffffc04d4e65>] ? nfsd_export_shutdown+0x5/0x250 [nfsd]
>  [<ffffffffc04c972a>] nfsd_exit_net+0x1a/0x20 [nfsd]
>  [<ffffffff9f6cdf39>] ops_exit_list.isra.1+0x39/0x60
>  [<ffffffff9f6ceae0>] cleanup_net+0x100/0x1e0
>  [<ffffffff9f097b30>] process_one_work+0x1e0/0x6b0
>  [<ffffffff9f097ac7>] ? process_one_work+0x177/0x6b0
>  [<ffffffff9f09811b>] worker_thread+0x11b/0x490
>  [<ffffffff9f098000>] ? process_one_work+0x6b0/0x6b0
>  [<ffffffff9f09ee46>] kthread+0xf6/0x110
>  [<ffffffff9f09ed50>] ? kthread_create_on_node+0x250/0x250
>  [<ffffffff9f82a8ac>] ret_from_fork+0x7c/0xb0
>  [<ffffffff9f09ed50>] ? kthread_create_on_node+0x250/0x250
> 
> Here's what I have on top of Linus' current tree..
> http://paste.fedoraproject.org/140505/20075141/raw/
> 
> Do I miss some other essential patch ?

If you have used the diff in that URL then you've missed:

     x86: Tell irq work about self IPI support

Without that the whole patchset has no effect.

Re: hrtimer deadlock caused by nohz_full

[Dave Jones]

On Thu, Oct 09, 2014 at 05:30:20AM +0200, Frederic Weisbecker wrote:

 > > Do I miss some other essential patch ?
 > 
 > If you have used the diff in that URL then you've missed:
 > 
 >      x86: Tell irq work about self IPI support
 > 
 > Without that the whole patchset has no effect.

Ok. Thanks, I'll rerun with that.

	Dave