Re: TREE_SRCU slows hotplug by factor ~16

["Paul E. McKenney" <paulmck@linux.vnet.ibm.com>]

On Mon, Apr 24, 2017 at 09:24:42AM -0700, Paul E. McKenney wrote:
> On Mon, Apr 24, 2017 at 09:35:03AM +0200, Mike Galbraith wrote:
> > On Sun, 2017-04-23 at 23:22 -0700, Paul E. McKenney wrote:
> > 
> > > Could you please collect an ftrace (or whatever) showing the timestamp
> > > sequence of calls to synchronize_srcu(), synchronize_srcu_expedited(),
> > > and call_srcu() during the execution of the stress script?  If it is easy
> > > to do, also the timestamp sequence of returns from synchronize_srcu()
> > > and synchronize_srcu_expedited()?
> > 
> > The first two minutes below config bits, trace_printk([enter/exit]) in
> > each function.  If you (unlikely, but..) want the whole trace, holler.
> > 
> > # RCU Subsystem
> > CONFIG_TREE_RCU=y
> > CONFIG_RCU_EXPERT=y
> > CONFIG_SRCU=y
> > # CONFIG_CLASSIC_SRCU is not set
> > CONFIG_TREE_SRCU=y
> > # CONFIG_TASKS_RCU is not set
> > CONFIG_RCU_STALL_COMMON=y
> > CONFIG_RCU_FANOUT=64
> > CONFIG_RCU_FANOUT_LEAF=16
> > # CONFIG_RCU_FAST_NO_HZ is not set
> > # CONFIG_TREE_RCU_TRACE is not set
> > CONFIG_RCU_KTHREAD_PRIO=0
> > CONFIG_RCU_NOCB_CPU=y
> > CONFIG_RCU_NOCB_CPU_NONE=y
> > # CONFIG_RCU_NOCB_CPU_ZERO is not set
> > # CONFIG_RCU_NOCB_CPU_ALL is not set
> > # RCU Debugging
> > # CONFIG_PROVE_RCU is not set
> > # CONFIG_SPARSE_RCU_POINTER is not set
> > # CONFIG_RCU_PERF_TEST is not set
> > # CONFIG_RCU_TORTURE_TEST is not set
> > CONFIG_RCU_CPU_STALL_TIMEOUT=60
> > # CONFIG_RCU_TRACE is not set
> > # CONFIG_RCU_EQS_DEBUG is not set
> 
> Thank you for the data, extremely helpful!  I now know that I must
> immediately fix this the hard way rather than trying several simpler
> possibilities that, according to your trace data, would be complete
> wastes of time.
> 
> My initial calculation was actually optimistic.  There was one
> exit-to-enter interval at about 80 milliseconds, but the second
> longest is 342 microseconds, and the vast majority are under 200
> microseconds.
> 
> Back to the drawing board!
> 
> This will take some time, but I should have a patch for you in a few days,
> hopefully sooner.

Making progress, might have something late tomorrow (Wednesday) Pacific
Time, will definitely have something Thursday Pacific Time.

								Thanx, Paul