On 06/13/2014 03:45 PM, Paul E. McKenney wrote: > On Fri, Jun 13, 2014 at 01:04:28PM -0700, Dav >> So, I bisected it down to this: >> >>> commit ac1bea85781e9004da9b3e8a4b097c18492d857c >>> Author: Paul E. McKenney >>> Date: Sun Mar 16 21:36:25 2014 -0700 >>> >>> sched,rcu: Make cond_resched() report RCU quiescent states >> >> Specifically, if I raise RCU_COND_RESCHED_LIM, things get back to their >> 3.15 levels. >> >> Could the additional RCU quiescent states be causing us to be doing more >> RCU frees that we were before, and getting less benefit from the lock >> batching that RCU normally provides? > > Quite possibly. One way to check would be to use the debugfs files > rcu/*/rcugp, which give a count of grace periods since boot for each > RCU flavor. Here "*" is rcu_preempt for CONFIG_PREEMPT and rcu_sched > for !CONFIG_PREEMPT. > > Another possibility is that someone is invoking cond_reched() in an > incredibly tight loop. open() does at least a couple of allocations in getname(), get_empty_filp() and apparmor_file_alloc_security() in my kernel, and each of those does a cond_resched() via the might_sleep() in the slub code. This test is doing ~400k open/closes per second per CPU, so that's ~1.2M cond_resched()/sec/CPU, but that's still hundreds of ns between calls on average. I'll do some more ftraces and dig in to those debugfs files early next week. > But please feel free to send along your patch, CCing LKML. Longer > term, I probably need to take a more algorithmic approach, but what > you have will be useful to benchmarkers until then. With the caveat that I exerted approximately 15 seconds of brainpower to code it up...patch attached.