* [PATCH -rt] drop spurious rcu unlock @ 2007-07-22 17:22 Daniel Walker 2007-07-23 3:13 ` Paul E. McKenney 0 siblings, 1 reply; 5+ messages in thread From: Daniel Walker @ 2007-07-22 17:22 UTC (permalink / raw) To: mingo; +Cc: linux-kernel, linux-rt-users Strange rcu_read_unlock() which causes a imbalance, and boot hang.. I didn't notice a reason for it, and removing it allows my system to make progress. This should go into the preempt-realtime-sched.patch Signed-Off-By: Daniel Walker <dwalker@mvista.com> Index: linux-2.6.22.1/kernel/sched.c =================================================================== --- linux-2.6.22.1.orig/kernel/sched.c 2007-07-22 16:47:37.000000000 +0000 +++ linux-2.6.22.1/kernel/sched.c 2007-07-22 16:16:48.000000000 +0000 @@ -4900,7 +4900,6 @@ asmlinkage long sys_sched_yield(void) * no need to preempt or enable interrupts: */ spin_unlock_no_resched(&rq->lock); - rcu_read_unlock(); __schedule(); ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH -rt] drop spurious rcu unlock 2007-07-22 17:22 [PATCH -rt] drop spurious rcu unlock Daniel Walker @ 2007-07-23 3:13 ` Paul E. McKenney 2007-07-23 13:37 ` Daniel Walker 0 siblings, 1 reply; 5+ messages in thread From: Paul E. McKenney @ 2007-07-23 3:13 UTC (permalink / raw) To: Daniel Walker; +Cc: mingo, linux-kernel, linux-rt-users On Sun, Jul 22, 2007 at 10:22:37AM -0700, Daniel Walker wrote: > > Strange rcu_read_unlock() which causes a imbalance, and boot hang.. I > didn't notice a reason for it, and removing it allows my system to make > progress. > > This should go into the preempt-realtime-sched.patch Strange. I have been getting boots, kernbench runs, and rcutorture tests despite its being present. Though perhaps it was the cause of the (non-fatal) "scheduling while atomic" I got during a kernbench run -- I was assuming it was my fault somehow. Non-reproducible. :-/ Thanx, Paul BUG: sleeping function called from invalid context cc1(29651) at kernel/rtmutex.c:636 in_atomic():1 [00000001], irqs_disabled():0 [<c0119f50>] __might_sleep+0xf3/0xf9 [<c031600e>] __rt_spin_lock+0x21/0x3c [<c014102c>] get_zone_pcp+0x20/0x29 [<c0141a40>] free_hot_cold_page+0xdc/0x167 [<c013a3f4>] add_preempt_count+0x12/0xcc [<c0110d92>] pgd_dtor+0x0/0x1 [<c015d865>] quicklist_trim+0xb7/0xe3 [<c0111025>] check_pgt_cache+0x19/0x1c [<c0148df5>] free_pgtables+0x54/0x12c [<c013a3f4>] add_preempt_count+0x12/0xcc [<c014e5be>] unmap_region+0xeb/0x13b BUG: scheduling while atomic: cc1/0x00000002/29745, CPU#0 [<c03146c8>] __sched_text_start+0xb0/0x4fe [<c0116432>] try_to_wake_up+0x31b/0x326 [<c013a3f4>] add_preempt_count+0x12/0xcc [<c013a3f4>] add_preempt_count+0x12/0xcc [<c0314bfc>] schedule+0xe6/0x100 [<c03158db>] rt_spin_lock_slowlock+0xc5/0x145 [<c0316027>] __rt_spin_lock+0x3a/0x3c [<c014102c>] get_zone_pcp+0x20/0x29 [<c0141a40>] free_hot_cold_page+0xdc/0x167 [<c013a3f4>] add_preempt_count+0x12/0xcc [<c0110d92>] pgd_dtor+0x0/0x1 [<c015d865>] quicklist_trim+0xb7/0xe3 [<c0111025>] check_pgt_cache+0x19/0x1c [<c0148df5>] free_pgtables+0x54/0x12c [<c013a3f4>] add_preempt_count+0x12/0xcc [<c014e5be>] unmap_region+0xeb/0x13b [<c014e861>] do_munmap+0xea/0xff [<c014e8a7>] sys_munmap+0x31/0x40 [<c010276a>] syscall_call+0x7/0xb [<c0310000>] _shift_data_right_pages+0xb9/0xd1 ======================= --------------------------- | preempt count: 00000002 ] | 2-level deep critical section nesting: ---------------------------------------- .. [<c015d7c8>] .... quicklist_trim+0x1a/0xe3 .....[<00000000>] .. ( <= _stext+0x3fefff50/0x14) .. [<c031462b>] .... __sched_text_start+0x13/0x4fe .....[<00000000>] .. ( <= _stext+0x3fefff50/0x14) > Signed-Off-By: Daniel Walker <dwalker@mvista.com> > > Index: linux-2.6.22.1/kernel/sched.c > =================================================================== > --- linux-2.6.22.1.orig/kernel/sched.c 2007-07-22 16:47:37.000000000 +0000 > +++ linux-2.6.22.1/kernel/sched.c 2007-07-22 16:16:48.000000000 +0000 > @@ -4900,7 +4900,6 @@ asmlinkage long sys_sched_yield(void) > * no need to preempt or enable interrupts: > */ > spin_unlock_no_resched(&rq->lock); > - rcu_read_unlock(); > > __schedule(); > > > > - > To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH -rt] drop spurious rcu unlock 2007-07-23 3:13 ` Paul E. McKenney @ 2007-07-23 13:37 ` Daniel Walker 2007-07-23 14:06 ` Paul E. McKenney 0 siblings, 1 reply; 5+ messages in thread From: Daniel Walker @ 2007-07-23 13:37 UTC (permalink / raw) To: paulmck; +Cc: mingo, linux-kernel, linux-rt-users On Sun, 2007-07-22 at 20:13 -0700, Paul E. McKenney wrote: > On Sun, Jul 22, 2007 at 10:22:37AM -0700, Daniel Walker wrote: > > > > Strange rcu_read_unlock() which causes a imbalance, and boot hang.. I > > didn't notice a reason for it, and removing it allows my system to make > > progress. > > > > This should go into the preempt-realtime-sched.patch > > Strange. I have been getting boots, kernbench runs, and rcutorture > tests despite its being present. Though perhaps it was the cause > of the (non-fatal) "scheduling while atomic" I got during a kernbench > run -- I was assuming it was my fault somehow. Non-reproducible. :-/ > > Thanx, Paul I had a strange config (for real time) .. All the new options turned off, and classic RCU .. However, a regular PREEMPT_RT kernel booted fine. The classic RCU made it easy to spot this since the preempt count goes negative. I was wondering what kind of side effects would happen with the preemptible RCU .. > BUG: sleeping function called from invalid context cc1(29651) at kernel/rtmutex.c:636 > in_atomic():1 [00000001], irqs_disabled():0 > [<c0119f50>] __might_sleep+0xf3/0xf9 > [<c031600e>] __rt_spin_lock+0x21/0x3c > [<c014102c>] get_zone_pcp+0x20/0x29 > [<c0141a40>] free_hot_cold_page+0xdc/0x167 > [<c013a3f4>] add_preempt_count+0x12/0xcc > [<c0110d92>] pgd_dtor+0x0/0x1 > [<c015d865>] quicklist_trim+0xb7/0xe3 > [<c0111025>] check_pgt_cache+0x19/0x1c > [<c0148df5>] free_pgtables+0x54/0x12c > [<c013a3f4>] add_preempt_count+0x12/0xcc > [<c014e5be>] unmap_region+0xeb/0x13b This looks a little like what Rui Nuno Capela is seeing , but his kernel hangs. > BUG: scheduling while atomic: cc1/0x00000002/29745, CPU#0 > [<c03146c8>] __sched_text_start+0xb0/0x4fe > [<c0116432>] try_to_wake_up+0x31b/0x326 > [<c013a3f4>] add_preempt_count+0x12/0xcc > [<c013a3f4>] add_preempt_count+0x12/0xcc > [<c0314bfc>] schedule+0xe6/0x100 > [<c03158db>] rt_spin_lock_slowlock+0xc5/0x145 > [<c0316027>] __rt_spin_lock+0x3a/0x3c > [<c014102c>] get_zone_pcp+0x20/0x29 > [<c0141a40>] free_hot_cold_page+0xdc/0x167 > [<c013a3f4>] add_preempt_count+0x12/0xcc > [<c0110d92>] pgd_dtor+0x0/0x1 > [<c015d865>] quicklist_trim+0xb7/0xe3 > [<c0111025>] check_pgt_cache+0x19/0x1c > [<c0148df5>] free_pgtables+0x54/0x12c > [<c013a3f4>] add_preempt_count+0x12/0xcc > [<c014e5be>] unmap_region+0xeb/0x13b > [<c014e861>] do_munmap+0xea/0xff > [<c014e8a7>] sys_munmap+0x31/0x40 > [<c010276a>] syscall_call+0x7/0xb > [<c0310000>] _shift_data_right_pages+0xb9/0xd1 > ======================= > --------------------------- > | preempt count: 00000002 ] > | 2-level deep critical section nesting: > ---------------------------------------- > .. [<c015d7c8>] .... quicklist_trim+0x1a/0xe3 > .....[<00000000>] .. ( <= _stext+0x3fefff50/0x14) > .. [<c031462b>] .... __sched_text_start+0x13/0x4fe > .....[<00000000>] .. ( <= _stext+0x3fefff50/0x14) > > > Signed-Off-By: Daniel Walker <dwalker@mvista.com> > > > > Index: linux-2.6.22.1/kernel/sched.c > > =================================================================== > > --- linux-2.6.22.1.orig/kernel/sched.c 2007-07-22 16:47:37.000000000 +0000 > > +++ linux-2.6.22.1/kernel/sched.c 2007-07-22 16:16:48.000000000 +0000 > > @@ -4900,7 +4900,6 @@ asmlinkage long sys_sched_yield(void) > > * no need to preempt or enable interrupts: > > */ > > spin_unlock_no_resched(&rq->lock); > > - rcu_read_unlock(); > > > > __schedule(); > > > > > > > > - > > To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH -rt] drop spurious rcu unlock 2007-07-23 13:37 ` Daniel Walker @ 2007-07-23 14:06 ` Paul E. McKenney 2007-07-23 14:28 ` Daniel Walker 0 siblings, 1 reply; 5+ messages in thread From: Paul E. McKenney @ 2007-07-23 14:06 UTC (permalink / raw) To: Daniel Walker; +Cc: mingo, linux-kernel, linux-rt-users On Mon, Jul 23, 2007 at 06:37:19AM -0700, Daniel Walker wrote: > On Sun, 2007-07-22 at 20:13 -0700, Paul E. McKenney wrote: > > On Sun, Jul 22, 2007 at 10:22:37AM -0700, Daniel Walker wrote: > > > > > > Strange rcu_read_unlock() which causes a imbalance, and boot hang.. I > > > didn't notice a reason for it, and removing it allows my system to make > > > progress. > > > > > > This should go into the preempt-realtime-sched.patch > > > > Strange. I have been getting boots, kernbench runs, and rcutorture > > tests despite its being present. Though perhaps it was the cause > > of the (non-fatal) "scheduling while atomic" I got during a kernbench > > run -- I was assuming it was my fault somehow. Non-reproducible. :-/ > > > > Thanx, Paul > > I had a strange config (for real time) .. All the new options turned > off, and classic RCU .. However, a regular PREEMPT_RT kernel booted > fine. The classic RCU made it easy to spot this since the preempt count > goes negative. Ah! That would happen in a CONFIG_PREEMPT kernel. The bug would be invisible in a non-CONFIG_PREEMPT kernel. > I was wondering what kind of side effects would happen with the > preemptible RCU .. Well, if you had only one rcu_read_lock() outstanding in the system, it would appear to be in a quiescent state, which would not be good. Given enough executions of that code, grace periods would cease and you would OOM. I didn't see any OOMs despite overnight rcutorture runs -- which would definitely have spotted halted grace periods -- so perhaps I just happened to never invoke sched_yield? > > BUG: sleeping function called from invalid context cc1(29651) at kernel/rtmutex.c:636 > > in_atomic():1 [00000001], irqs_disabled():0 > > [<c0119f50>] __might_sleep+0xf3/0xf9 > > [<c031600e>] __rt_spin_lock+0x21/0x3c > > [<c014102c>] get_zone_pcp+0x20/0x29 > > [<c0141a40>] free_hot_cold_page+0xdc/0x167 > > [<c013a3f4>] add_preempt_count+0x12/0xcc > > [<c0110d92>] pgd_dtor+0x0/0x1 > > [<c015d865>] quicklist_trim+0xb7/0xe3 > > [<c0111025>] check_pgt_cache+0x19/0x1c > > [<c0148df5>] free_pgtables+0x54/0x12c > > [<c013a3f4>] add_preempt_count+0x12/0xcc > > [<c014e5be>] unmap_region+0xeb/0x13b > > This looks a little like what Rui Nuno Capela is seeing , but his kernel > hangs. Hmmm... In this particular case, the test (kernbench) was fairly short, so maybe I just didn't let it run long enough. Thanx, Paul > > BUG: scheduling while atomic: cc1/0x00000002/29745, CPU#0 > > [<c03146c8>] __sched_text_start+0xb0/0x4fe > > [<c0116432>] try_to_wake_up+0x31b/0x326 > > [<c013a3f4>] add_preempt_count+0x12/0xcc > > [<c013a3f4>] add_preempt_count+0x12/0xcc > > [<c0314bfc>] schedule+0xe6/0x100 > > [<c03158db>] rt_spin_lock_slowlock+0xc5/0x145 > > [<c0316027>] __rt_spin_lock+0x3a/0x3c > > [<c014102c>] get_zone_pcp+0x20/0x29 > > [<c0141a40>] free_hot_cold_page+0xdc/0x167 > > [<c013a3f4>] add_preempt_count+0x12/0xcc > > [<c0110d92>] pgd_dtor+0x0/0x1 > > [<c015d865>] quicklist_trim+0xb7/0xe3 > > [<c0111025>] check_pgt_cache+0x19/0x1c > > [<c0148df5>] free_pgtables+0x54/0x12c > > [<c013a3f4>] add_preempt_count+0x12/0xcc > > [<c014e5be>] unmap_region+0xeb/0x13b > > [<c014e861>] do_munmap+0xea/0xff > > [<c014e8a7>] sys_munmap+0x31/0x40 > > [<c010276a>] syscall_call+0x7/0xb > > [<c0310000>] _shift_data_right_pages+0xb9/0xd1 > > ======================= > > --------------------------- > > | preempt count: 00000002 ] > > | 2-level deep critical section nesting: > > ---------------------------------------- > > .. [<c015d7c8>] .... quicklist_trim+0x1a/0xe3 > > .....[<00000000>] .. ( <= _stext+0x3fefff50/0x14) > > .. [<c031462b>] .... __sched_text_start+0x13/0x4fe > > .....[<00000000>] .. ( <= _stext+0x3fefff50/0x14) > > > > > Signed-Off-By: Daniel Walker <dwalker@mvista.com> > > > > > > Index: linux-2.6.22.1/kernel/sched.c > > > =================================================================== > > > --- linux-2.6.22.1.orig/kernel/sched.c 2007-07-22 16:47:37.000000000 +0000 > > > +++ linux-2.6.22.1/kernel/sched.c 2007-07-22 16:16:48.000000000 +0000 > > > @@ -4900,7 +4900,6 @@ asmlinkage long sys_sched_yield(void) > > > * no need to preempt or enable interrupts: > > > */ > > > spin_unlock_no_resched(&rq->lock); > > > - rcu_read_unlock(); > > > > > > __schedule(); > > > > > > > > > > > > - > > > To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in > > > the body of a message to majordomo@vger.kernel.org > > > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH -rt] drop spurious rcu unlock 2007-07-23 14:06 ` Paul E. McKenney @ 2007-07-23 14:28 ` Daniel Walker 0 siblings, 0 replies; 5+ messages in thread From: Daniel Walker @ 2007-07-23 14:28 UTC (permalink / raw) To: paulmck; +Cc: mingo, linux-kernel, linux-rt-users On Mon, 2007-07-23 at 07:06 -0700, Paul E. McKenney wrote: > > I was wondering what kind of side effects would happen with the > > preemptible RCU .. > > Well, if you had only one rcu_read_lock() outstanding in the system, > it would appear to be in a quiescent state, which would not be good. > Given enough executions of that code, grace periods would cease and > you would OOM. > > I didn't see any OOMs despite overnight rcutorture runs -- which would > definitely have spotted halted grace periods -- so perhaps I just happened > to never invoke sched_yield? Seems pretty unlikely that yield() wouldn't get called .. Isn't the effect per-task since rcu_read_unlock() touches current->rcu_read_lock_nesting? > > > BUG: sleeping function called from invalid context cc1(29651) at kernel/rtmutex.c:636 > > > in_atomic():1 [00000001], irqs_disabled():0 > > > [<c0119f50>] __might_sleep+0xf3/0xf9 > > > [<c031600e>] __rt_spin_lock+0x21/0x3c > > > [<c014102c>] get_zone_pcp+0x20/0x29 > > > [<c0141a40>] free_hot_cold_page+0xdc/0x167 > > > [<c013a3f4>] add_preempt_count+0x12/0xcc > > > [<c0110d92>] pgd_dtor+0x0/0x1 > > > [<c015d865>] quicklist_trim+0xb7/0xe3 > > > [<c0111025>] check_pgt_cache+0x19/0x1c > > > [<c0148df5>] free_pgtables+0x54/0x12c > > > [<c013a3f4>] add_preempt_count+0x12/0xcc > > > [<c014e5be>] unmap_region+0xeb/0x13b > > > > This looks a little like what Rui Nuno Capela is seeing , but his kernel > > hangs. > > Hmmm... In this particular case, the test (kernbench) was fairly > short, so maybe I just didn't let it run long enough. Looks like quicklist_trim() disabled preemption to access the per-cpu variable "quicklist" then calls free_pages() which takes a mutex.. Below looks like the same deal .. > Thanx, Paul > > > > BUG: scheduling while atomic: cc1/0x00000002/29745, CPU#0 > > > [<c03146c8>] __sched_text_start+0xb0/0x4fe > > > [<c0116432>] try_to_wake_up+0x31b/0x326 > > > [<c013a3f4>] add_preempt_count+0x12/0xcc > > > [<c013a3f4>] add_preempt_count+0x12/0xcc > > > [<c0314bfc>] schedule+0xe6/0x100 > > > [<c03158db>] rt_spin_lock_slowlock+0xc5/0x145 > > > [<c0316027>] __rt_spin_lock+0x3a/0x3c > > > [<c014102c>] get_zone_pcp+0x20/0x29 > > > [<c0141a40>] free_hot_cold_page+0xdc/0x167 > > > [<c013a3f4>] add_preempt_count+0x12/0xcc > > > [<c0110d92>] pgd_dtor+0x0/0x1 > > > [<c015d865>] quicklist_trim+0xb7/0xe3 > > > [<c0111025>] check_pgt_cache+0x19/0x1c > > > [<c0148df5>] free_pgtables+0x54/0x12c > > > [<c013a3f4>] add_preempt_count+0x12/0xcc > > > [<c014e5be>] unmap_region+0xeb/0x13b > > > [<c014e861>] do_munmap+0xea/0xff > > > [<c014e8a7>] sys_munmap+0x31/0x40 > > > [<c010276a>] syscall_call+0x7/0xb > > > [<c0310000>] _shift_data_right_pages+0xb9/0xd1 > > > ======================= > > > --------------------------- > > > | preempt count: 00000002 ] > > > | 2-level deep critical section nesting: > > > ---------------------------------------- > > > .. [<c015d7c8>] .... quicklist_trim+0x1a/0xe3 > > > .....[<00000000>] .. ( <= _stext+0x3fefff50/0x14) > > > .. [<c031462b>] .... __sched_text_start+0x13/0x4fe > > > .....[<00000000>] .. ( <= _stext+0x3fefff50/0x14) > > > > > > > Signed-Off-By: Daniel Walker <dwalker@mvista.com> > > > > > > > > Index: linux-2.6.22.1/kernel/sched.c > > > > =================================================================== > > > > --- linux-2.6.22.1.orig/kernel/sched.c 2007-07-22 16:47:37.000000000 +0000 > > > > +++ linux-2.6.22.1/kernel/sched.c 2007-07-22 16:16:48.000000000 +0000 > > > > @@ -4900,7 +4900,6 @@ asmlinkage long sys_sched_yield(void) > > > > * no need to preempt or enable interrupts: > > > > */ > > > > spin_unlock_no_resched(&rq->lock); > > > > - rcu_read_unlock(); > > > > > > > > __schedule(); > > > > > > > > > > > > > > > > - > > > > To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in > > > > the body of a message to majordomo@vger.kernel.org > > > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2007-07-23 14:39 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2007-07-22 17:22 [PATCH -rt] drop spurious rcu unlock Daniel Walker 2007-07-23 3:13 ` Paul E. McKenney 2007-07-23 13:37 ` Daniel Walker 2007-07-23 14:06 ` Paul E. McKenney 2007-07-23 14:28 ` Daniel Walker
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox