* 3.3.0: "INFO: rcu_bh detected stall on CPU 3 (t=0 jiffies)"
@ 2012-03-27 14:15 Ralf Hildebrandt
2012-03-28 11:28 ` Ralf Hildebrandt
0 siblings, 1 reply; 4+ messages in thread
From: Ralf Hildebrandt @ 2012-03-27 14:15 UTC (permalink / raw)
To: linux-kernel
Mar 23 11:54:00 mail kernel:
Mar 23 11:54:00 mail kernel: [347213.025005] INFO: rcu_bh detected stall on CPU 3 (t=0 jiffies)
Mar 23 11:54:00 mail kernel: [347213.025005] Pid: 12654, comm: cleanup Not tainted 3.3.0 #1
Mar 23 11:54:00 mail kernel: [347213.025005] Call Trace:
Mar 23 11:54:00 mail kernel: [347213.025005] [<c1062ccb>] ? __rcu_pending+0x10a/0x311
Mar 23 11:54:00 mail kernel: [347213.025005] [<c106388d>] ? rcu_check_callbacks+0x9b/0xa1
Mar 23 11:54:00 mail kernel: [347213.025005] [<c102c434>] ? update_process_times+0x2a/0x53
Mar 23 11:54:00 mail kernel: [347213.025005] [<c104f206>] ? tick_sched_timer+0x4f/0x90
Mar 23 11:54:00 mail kernel: [347213.025005] [<c103a101>] ? __remove_hrtimer+0x25/0x79
Mar 23 11:54:00 mail kernel: [347213.025005] [<c103a2ff>] ? __run_hrtimer.isra.32+0x38/0xbe
Mar 23 11:54:00 mail kernel: [347213.025005] [<c103ad95>] ? hrtimer_interrupt+0xda/0x23d
Mar 23 11:54:00 mail kernel: [347213.025005] [<c1016493>] ? smp_apic_timer_interrupt+0x4c/0x81
Mar 23 11:54:00 mail kernel: [347213.025005] [<c127b5d9>] ? apic_timer_interrupt+0x31/0x38
--
Ralf Hildebrandt Charite Universitätsmedizin Berlin
ralf.hildebrandt@charite.de Campus Benjamin Franklin
http://www.charite.de Hindenburgdamm 30, 12203 Berlin
Geschäftsbereich IT, Abt. Netzwerk fon: +49-30-450.570.155
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: 3.3.0: "INFO: rcu_bh detected stall on CPU 3 (t=0 jiffies)"
2012-03-27 14:15 3.3.0: "INFO: rcu_bh detected stall on CPU 3 (t=0 jiffies)" Ralf Hildebrandt
@ 2012-03-28 11:28 ` Ralf Hildebrandt
2012-04-06 17:52 ` Paul E. McKenney
0 siblings, 1 reply; 4+ messages in thread
From: Ralf Hildebrandt @ 2012-03-28 11:28 UTC (permalink / raw)
To: linux-kernel
* Ralf Hildebrandt <Ralf.Hildebrandt@charite.de>:
>
> Mar 23 11:54:00 mail kernel:
> Mar 23 11:54:00 mail kernel: [347213.025005] INFO: rcu_bh detected stall on CPU 3 (t=0 jiffies)
> Mar 23 11:54:00 mail kernel: [347213.025005] Pid: 12654, comm: cleanup Not tainted 3.3.0 #1
> Mar 23 11:54:00 mail kernel: [347213.025005] Call Trace:
> Mar 23 11:54:00 mail kernel: [347213.025005] [<c1062ccb>] ? __rcu_pending+0x10a/0x311
> Mar 23 11:54:00 mail kernel: [347213.025005] [<c106388d>] ? rcu_check_callbacks+0x9b/0xa1
> Mar 23 11:54:00 mail kernel: [347213.025005] [<c102c434>] ? update_process_times+0x2a/0x53
> Mar 23 11:54:00 mail kernel: [347213.025005] [<c104f206>] ? tick_sched_timer+0x4f/0x90
> Mar 23 11:54:00 mail kernel: [347213.025005] [<c103a101>] ? __remove_hrtimer+0x25/0x79
> Mar 23 11:54:00 mail kernel: [347213.025005] [<c103a2ff>] ? __run_hrtimer.isra.32+0x38/0xbe
> Mar 23 11:54:00 mail kernel: [347213.025005] [<c103ad95>] ? hrtimer_interrupt+0xda/0x23d
> Mar 23 11:54:00 mail kernel: [347213.025005] [<c1016493>] ? smp_apic_timer_interrupt+0x4c/0x81
> Mar 23 11:54:00 mail kernel: [347213.025005] [<c127b5d9>] ? apic_timer_interrupt+0x31/0x38
I read http://www.kernel.org/doc/Documentation/RCU/stallwarn.txt
but couldn't find "rcu_bh detected stall" being mentioned there.
I reported the same issue on 3.2.9 (same machine!) on the 13th of march:
Mar 12 12:58:38 mail kernel: [440746.244002] INFO: rcu_bh detected stall on CPU 2 (t=0 jiffies)
Mar 12 12:58:38 mail kernel: [440746.244002] Pid: 27980, comm: /usr/sbin/amavi Tainted: G W 3.2.9 #1
Mar 12 12:58:38 mail kernel: [440746.244002] Call Trace:
Mar 12 12:58:38 mail kernel: [440746.244002] [<c1061d26>] ? __rcu_pending+0x10a/0x30e
Mar 12 12:58:38 mail kernel: [440746.244002] [<c10626ca>] ? rcu_check_callbacks+0xd4/0xde
Mar 12 12:58:38 mail kernel: [440746.244002] [<c1035cf2>] ? update_process_times+0x2a/0x53
Mar 12 12:58:38 mail kernel: [440746.244002] [<c104e1d2>] ? tick_sched_timer+0x4d/0x8e
Mar 12 12:58:38 mail kernel: [440746.244002] [<c1043b53>] ? __remove_hrtimer+0x25/0x79
Mar 12 12:58:38 mail kernel: [440746.244002] [<c1043d4f>] ? __run_hrtimer.isra.32+0x37/0xbd
Mar 12 12:58:38 mail kernel: [440746.244002] [<c10445d5>] ? hrtimer_interrupt+0xdb/0x23d
Mar 12 12:58:38 mail kernel: [440746.244002] [<c101b781>] ? vmalloc_sync_all+0x1/0x1
Mar 12 12:58:38 mail kernel: [440746.244002] [<c1015e4c>] ? smp_apic_timer_interrupt+0x4c/0x81
Mar 12 12:58:38 mail kernel: [440746.244002] [<c1275491>] ? apic_timer_interrupt+0x31/0x38
Mar 12 12:58:38 mail kernel: [440746.244002] [<c101b781>] ? vmalloc_sync_all+0x1/0x1
Mar 12 12:58:38 mail kernel: [440746.244002] [<c101b8c2>] ? do_page_fault+0x141/0x38f
Mar 12 12:58:38 mail kernel: [440746.244002] [<c1030d8d>] ? irq_exit+0x34/0x87
Mar 12 12:58:38 mail kernel: [440746.244002] [<c1015e51>] ? smp_apic_timer_interrupt+0x51/0x81
Mar 12 12:58:38 mail kernel: [440746.244002] [<c101b781>] ? vmalloc_sync_all+0x1/0x1
Mar 12 12:58:38 mail kernel: [440746.244002] [<c12756b7>] ? error_code+0x67/0x6c
and so did Tilman Schmidt in February:
https://lkml.org/lkml/2012/2/18/34
--
Ralf Hildebrandt Charite Universitätsmedizin Berlin
ralf.hildebrandt@charite.de Campus Benjamin Franklin
http://www.charite.de Hindenburgdamm 30, 12203 Berlin
Geschäftsbereich IT, Abt. Netzwerk fon: +49-30-450.570.155
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: 3.3.0: "INFO: rcu_bh detected stall on CPU 3 (t=0 jiffies)"
2012-03-28 11:28 ` Ralf Hildebrandt
@ 2012-04-06 17:52 ` Paul E. McKenney
2012-04-08 16:21 ` Ralf Hildebrandt
0 siblings, 1 reply; 4+ messages in thread
From: Paul E. McKenney @ 2012-04-06 17:52 UTC (permalink / raw)
To: linux-kernel
On Wed, Mar 28, 2012 at 01:28:22PM +0200, Ralf Hildebrandt wrote:
> * Ralf Hildebrandt <Ralf.Hildebrandt@charite.de>:
> >
> > Mar 23 11:54:00 mail kernel:
> > Mar 23 11:54:00 mail kernel: [347213.025005] INFO: rcu_bh detected stall on CPU 3 (t=0 jiffies)
> > Mar 23 11:54:00 mail kernel: [347213.025005] Pid: 12654, comm: cleanup Not tainted 3.3.0 #1
> > Mar 23 11:54:00 mail kernel: [347213.025005] Call Trace:
> > Mar 23 11:54:00 mail kernel: [347213.025005] [<c1062ccb>] ? __rcu_pending+0x10a/0x311
> > Mar 23 11:54:00 mail kernel: [347213.025005] [<c106388d>] ? rcu_check_callbacks+0x9b/0xa1
> > Mar 23 11:54:00 mail kernel: [347213.025005] [<c102c434>] ? update_process_times+0x2a/0x53
> > Mar 23 11:54:00 mail kernel: [347213.025005] [<c104f206>] ? tick_sched_timer+0x4f/0x90
> > Mar 23 11:54:00 mail kernel: [347213.025005] [<c103a101>] ? __remove_hrtimer+0x25/0x79
> > Mar 23 11:54:00 mail kernel: [347213.025005] [<c103a2ff>] ? __run_hrtimer.isra.32+0x38/0xbe
> > Mar 23 11:54:00 mail kernel: [347213.025005] [<c103ad95>] ? hrtimer_interrupt+0xda/0x23d
> > Mar 23 11:54:00 mail kernel: [347213.025005] [<c1016493>] ? smp_apic_timer_interrupt+0x4c/0x81
> > Mar 23 11:54:00 mail kernel: [347213.025005] [<c127b5d9>] ? apic_timer_interrupt+0x31/0x38
>
> I read http://www.kernel.org/doc/Documentation/RCU/stallwarn.txt
> but couldn't find "rcu_bh detected stall" being mentioned there.
The documentation mentions "rcu_bh_state detected stalls". I will check
this and update if needed.
> I reported the same issue on 3.2.9 (same machine!) on the 13th of march:
>
> Mar 12 12:58:38 mail kernel: [440746.244002] INFO: rcu_bh detected stall on CPU 2 (t=0 jiffies)
> Mar 12 12:58:38 mail kernel: [440746.244002] Pid: 27980, comm: /usr/sbin/amavi Tainted: G W 3.2.9 #1
> Mar 12 12:58:38 mail kernel: [440746.244002] Call Trace:
> Mar 12 12:58:38 mail kernel: [440746.244002] [<c1061d26>] ? __rcu_pending+0x10a/0x30e
> Mar 12 12:58:38 mail kernel: [440746.244002] [<c10626ca>] ? rcu_check_callbacks+0xd4/0xde
> Mar 12 12:58:38 mail kernel: [440746.244002] [<c1035cf2>] ? update_process_times+0x2a/0x53
> Mar 12 12:58:38 mail kernel: [440746.244002] [<c104e1d2>] ? tick_sched_timer+0x4d/0x8e
> Mar 12 12:58:38 mail kernel: [440746.244002] [<c1043b53>] ? __remove_hrtimer+0x25/0x79
> Mar 12 12:58:38 mail kernel: [440746.244002] [<c1043d4f>] ? __run_hrtimer.isra.32+0x37/0xbd
> Mar 12 12:58:38 mail kernel: [440746.244002] [<c10445d5>] ? hrtimer_interrupt+0xdb/0x23d
> Mar 12 12:58:38 mail kernel: [440746.244002] [<c101b781>] ? vmalloc_sync_all+0x1/0x1
> Mar 12 12:58:38 mail kernel: [440746.244002] [<c1015e4c>] ? smp_apic_timer_interrupt+0x4c/0x81
> Mar 12 12:58:38 mail kernel: [440746.244002] [<c1275491>] ? apic_timer_interrupt+0x31/0x38
> Mar 12 12:58:38 mail kernel: [440746.244002] [<c101b781>] ? vmalloc_sync_all+0x1/0x1
> Mar 12 12:58:38 mail kernel: [440746.244002] [<c101b8c2>] ? do_page_fault+0x141/0x38f
> Mar 12 12:58:38 mail kernel: [440746.244002] [<c1030d8d>] ? irq_exit+0x34/0x87
> Mar 12 12:58:38 mail kernel: [440746.244002] [<c1015e51>] ? smp_apic_timer_interrupt+0x51/0x81
> Mar 12 12:58:38 mail kernel: [440746.244002] [<c101b781>] ? vmalloc_sync_all+0x1/0x1
> Mar 12 12:58:38 mail kernel: [440746.244002] [<c12756b7>] ? error_code+0x67/0x6c
This stack looks interesting -- might be a problem in vmalloc_sync(),
at least if the stack can be trusted. Could you please reproduce with
frame pointers enabled?
Thanx, Paul
> and so did Tilman Schmidt in February:
> https://lkml.org/lkml/2012/2/18/34
>
> --
> Ralf Hildebrandt Charite Universitätsmedizin Berlin
> ralf.hildebrandt@charite.de Campus Benjamin Franklin
> http://www.charite.de Hindenburgdamm 30, 12203 Berlin
> Geschäftsbereich IT, Abt. Netzwerk fon: +49-30-450.570.155
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: 3.3.0: "INFO: rcu_bh detected stall on CPU 3 (t=0 jiffies)"
2012-04-06 17:52 ` Paul E. McKenney
@ 2012-04-08 16:21 ` Ralf Hildebrandt
0 siblings, 0 replies; 4+ messages in thread
From: Ralf Hildebrandt @ 2012-04-08 16:21 UTC (permalink / raw)
To: linux-kernel
* Paul E. McKenney <paulmck@linux.vnet.ibm.com>:
> This stack looks interesting -- might be a problem in vmalloc_sync(),
> at least if the stack can be trusted. Could you please reproduce with
> frame pointers enabled?
I'll rebuild the kernel right away, Sir.
--
Ralf Hildebrandt Charite Universitätsmedizin Berlin
ralf.hildebrandt@charite.de Campus Benjamin Franklin
http://www.charite.de Hindenburgdamm 30, 12203 Berlin
Geschäftsbereich IT, Abt. Netzwerk fon: +49-30-450.570.155
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2012-04-08 16:21 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-03-27 14:15 3.3.0: "INFO: rcu_bh detected stall on CPU 3 (t=0 jiffies)" Ralf Hildebrandt
2012-03-28 11:28 ` Ralf Hildebrandt
2012-04-06 17:52 ` Paul E. McKenney
2012-04-08 16:21 ` Ralf Hildebrandt
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox