From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Paul E. McKenney" Subject: Re: [RT] RCU splat on 2.6.33.4-rt20 Date: Wed, 19 May 2010 20:22:46 -0700 Message-ID: <20100520032246.GB2708@linux.vnet.ibm.com> References: <20100519211911.5450c1dd@torg> Reply-To: paulmck@linux.vnet.ibm.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: Thomas Gleixner , RT To: Clark Williams Return-path: Received: from e9.ny.us.ibm.com ([32.97.182.139]:59988 "EHLO e9.ny.us.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752633Ab0ETDWs (ORCPT ); Wed, 19 May 2010 23:22:48 -0400 Received: from d01relay03.pok.ibm.com (d01relay03.pok.ibm.com [9.56.227.235]) by e9.ny.us.ibm.com (8.14.3/8.13.1) with ESMTP id o4K39EXT031865 for ; Wed, 19 May 2010 23:09:14 -0400 Received: from d01av04.pok.ibm.com (d01av04.pok.ibm.com [9.56.224.64]) by d01relay03.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id o4K3MmKO162514 for ; Wed, 19 May 2010 23:22:48 -0400 Received: from d01av04.pok.ibm.com (loopback [127.0.0.1]) by d01av04.pok.ibm.com (8.14.3/8.13.1/NCO v10.0 AVout) with ESMTP id o4K3MlTd005664 for ; Wed, 19 May 2010 23:22:47 -0400 Content-Disposition: inline In-Reply-To: <20100519211911.5450c1dd@torg> Sender: linux-rt-users-owner@vger.kernel.org List-ID: On Wed, May 19, 2010 at 09:19:11PM -0500, Clark Williams wrote: > Thomas/Paul, > > Resumed my laptop (T60, 64-bit kernel) and after about five > minutes of doing nothing but surfing with firefox, got the following > RCU stall notification from a 2.6.33.4-rt20 kernel: > > INFO: RCU detected CPU 1 stall (t=0 jiffies) Hmmm... "t=0 jiffies". I wonder if the hibernation/resumption confused the stall-warning code. My first thought was that an RCU grace period had spanned the hibernation, but that would not be consistent with it waiting five minutes after resumption to splat. I will look into this. Thanx, Paul > sending NMI to all CPUs: > NMI backtrace for cpu 1 > CPU 1 > Pid: 0, comm: swapper Not tainted 2.6.33.4-rt20 #37 / > RIP: 0010:[] [] > __const_udelay+0x37/0x44 RSP: 0018:ffff88000a303dc8 EFLAGS: 00000002 > RAX: 0000000001062560 RBX: 0000000000000001 RCX: ffff88000a300000 > RDX: 000000001dba61d0 RSI: 0000000000000002 RDI: 0000000000418958 > RBP: ffff88000a303dc8 R08: 0000000000000001 R09: 0000000000000000 > R10: 0000000000000000 R11: 0000000000000000 R12: ffff88000a30b8b0 > R13: ffffffff81a7c800 R14: ffff88000a303f48 R15: ffff88000a30b5e0 > FS: 0000000000000000(0000) GS:ffff88000a300000(0000) > knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > CR2: 00007f5fedd4b000 CR3: 0000000001a56000 CR4: 00000000000006e0 > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > Process swapper (pid: 0, threadinfo ffff8800bf1d8000, task > ffff8800bf1d67c0) Stack: > ffff88000a303de8 ffffffff810185aa 000000000000088e ffffffff81a7c800 > <0> ffff88000a303e38 ffffffff81092650 ffff8800bf1d9dc8 0000000000000001 > <0> ffff88000a303e58 0000000000000001 0000000000000001 0000000000000000 > Call Trace: > > [] arch_trigger_all_cpu_backtrace+0x56/0x62 > [] __rcu_pending+0x6e/0x2d3 > [] rcu_check_callbacks+0x54/0x15b > [] update_process_times+0x46/0x5b > [] tick_sched_timer+0x7a/0xa4 > [] __run_hrtimer+0x130/0x1bd > [] ? tick_sched_timer+0x0/0xa4 > [] hrtimer_interrupt+0xdc/0x1dd > [] smp_apic_timer_interrupt+0x89/0x9c > [] apic_timer_interrupt+0x13/0x20 > > [] ? acpi_idle_enter_bm+0x2b0/0x2db > [] ? acpi_idle_enter_bm+0x2a6/0x2db > [] cpuidle_idle_call+0x96/0x11f > [] cpu_idle+0xab/0xee > [] start_secondary+0x1a5/0x1a9 > Code: 00 ff 00 00 48 8d 04 bd 00 00 00 00 65 8b 0c 25 78 ac 00 00 48 63 > c9 48 8b 0c cd 60 23 af 81 48 69 94 0a 98 00 00 00 fa 00 00 00 e2 > 48 8d 7a 01 e8 ad ff ff ff c9 c3 55 48 89 e5 0f 1f 44 00 Call Trace: > <#MC> <> Pid: 0, comm: swapper Not tainted 2.6.33.4-rt20 #37 Call > Trace: [] ? show_regs+0x2b/0x30 > [] nmi_watchdog_tick+0xc2/0x1a6 > [] do_nmi+0xb5/0x25b > [] nmi+0x1a/0x20 > [] ? __const_udelay+0x37/0x44 > <> [] > arch_trigger_all_cpu_backtrace+0x56/0x62 [] > __rcu_pending+0x6e/0x2d3 [] > rcu_check_callbacks+0x54/0x15b [] > update_process_times+0x46/0x5b [] > tick_sched_timer+0x7a/0xa4 [] > __run_hrtimer+0x130/0x1bd [] ? > tick_sched_timer+0x0/0xa4 [] > hrtimer_interrupt+0xdc/0x1dd [] > smp_apic_timer_interrupt+0x89/0x9c [] > apic_timer_interrupt+0x13/0x20 [] ? > acpi_idle_enter_bm+0x2b0/0x2db [] ? > acpi_idle_enter_bm+0x2a6/0x2db [] > cpuidle_idle_call+0x96/0x11f [] cpu_idle+0xab/0xee > [] start_secondary+0x1a5/0x1a9 > NMI backtrace for cpu 0 > CPU 0 > Pid: 31, comm: events/0 Not tainted 2.6.33.4-rt20 #37 / > RIP: 0010:[] [] > rt_worker_func+0xd3/0x2de RSP: 0018:ffff8800bf1ddd80 EFLAGS: 00000246 > RAX: 0000000000000000 RBX: 0000000000002770 RCX: ffff8800b8a13b80 > RDX: 0000000000002770 RSI: ffff8800bf1ddfd8 RDI: ffff8800b893a760 > RBP: ffff8800bf1dddf0 R08: ffff8800bf1dc000 R09: ffff8800bf1da800 > R10: ffff88000a210000 R11: 0000000000000001 R12: 0000000000006666 > R13: 0000000000000000 R14: 00000000000062b0 R15: 0000000000013b80 > FS: 0000000000000000(0000) GS:ffff88000a200000(0000) > knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > CR2: 00007fbe40a86000 CR3: 0000000001a56000 CR4: 00000000000006f0 > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > Process events/0 (pid: 31, threadinfo ffff8800bf1dc000, task > ffff8800bf1da800) Stack: > ffff8800bf1dddc0 ffff8800b8a135a0 0000000000000000 ffff8800bf1ddfd8 > <0> 0000000000002760 00000000000493e0 0000000000000000 00000000000493e0 > <0> ffff8800bf1dc000 ffff88000a212ac0 ffff8800bf1da800 ffff8800bf1dc000 > Call Trace: > [] worker_thread+0x206/0x2a3 > [] ? rt_worker_func+0x0/0x2de > [] ? autoremove_wake_function+0x0/0x39 > [] ? worker_thread+0x0/0x2a3 > [] ? worker_thread+0x0/0x2a3 > [] kthread+0x7f/0x87 > [] kernel_thread_helper+0x4/0x10 > [] ? kthread+0x0/0x87 > [] ? kernel_thread_helper+0x0/0x10 > Code: 00 00 00 00 4c 89 f9 48 03 0d 98 63 73 00 a8 08 74 15 48 89 55 90 > 48 89 4d 98 e8 67 3c 07 00 48 8b 4d 98 48 8b 55 90 48 83 39 00 <0f> 84 > 86 01 00 00 48 63 45 c8 48 89 4d 98 48 89 45 b8 81 e2 ff Call Trace: > <#MC> <> Pid: 31, comm: events/0 Not tainted 2.6.33.4-rt20 #37 > Call Trace: [] ? show_regs+0x2b/0x30 > [] nmi_watchdog_tick+0xc2/0x1a6 > [] do_nmi+0xb5/0x25b > [] nmi+0x1a/0x20 > [] ? rt_worker_func+0xd3/0x2de > <> [] worker_thread+0x206/0x2a3 > [] ? rt_worker_func+0x0/0x2de > [] ? autoremove_wake_function+0x0/0x39 > [] ? worker_thread+0x0/0x2a3 > [] ? worker_thread+0x0/0x2a3 > [] kthread+0x7f/0x87 > [] kernel_thread_helper+0x4/0x10 > [] ? kthread+0x0/0x87 > [] ? kernel_thread_helper+0x0/0x10