Re: Making rcu_normal=1 in RT

["Paul E. McKenney" <paulmck@linux.vnet.ibm.com>]

On Sun, Oct 16, 2016 at 04:45:32AM +0300, Michael S. Tsirkin wrote:
> On Fri, Oct 14, 2016 at 02:20:50AM -0700, Paul E. McKenney wrote:
> > On Thu, Oct 13, 2016 at 07:25:56PM +0300, Michael S. Tsirkin wrote:
> > > On Wed, Oct 12, 2016 at 01:32:23PM -0700, Paul E. McKenney wrote:
> > > > On Wed, Oct 12, 2016 at 12:15:53PM -0500, Julia Cartwright wrote:
> > > > > On Wed, Oct 12, 2016 at 12:49:56PM -0400, Luiz Capitulino wrote:
> > > > > > On Wed, 12 Oct 2016 11:21:14 -0500
> > > > > > Julia Cartwright <julia@ni.com> wrote:
> > > > > > 
> > > > > > > On Wed, Oct 12, 2016 at 11:12:51AM -0400, Luiz Capitulino wrote:
> > > > > > > > Hi,
> > > > > > > > 
> > > > > > > > We have the following patch applied to the RT tree:
> > > > > > > > 
> > > > > > > >   commit a9d3cc781a3306bfa332fa7cb6134b70696058d5
> > > > > > > >   Author: Josh Cartwright <joshc@ni.com>
> > > > > > > >   Date:   Tue Oct 27 07:31:53 2015 -0500
> > > > > > > >   
> > > > > > > >       net: Make synchronize_rcu_expedited() conditional on !RT_FULL
> > > > > > > > 
> > > > > > > > However, as noted by Michael, making rcu_normal=1 default in the
> > > > > > > > RT kernel should have the same effect (ie.  not calling
> > > > > > > > synchronize_sched_expedited()).
> > > > > > > > 
> > > > > > > > So, honest question, is there a reason not to:
> > > > > > > > 
> > > > > > > >  1. Drop the patch above
> > > > > > > >  2. Make rcu_normal=1 default?  
> > > > > > > 
> > > > > > > I think this is probably a cleaner way to fix the problems which
> > > > > > > motivated this patch in the first place.  In my defense, the above patch
> > > > > > > predates rcu_normal :).
> > > > > > 
> > > > > > No need to defend yourself! We debugged this very spike in one of
> > > > > > our kernels that don't have rcu_normal. We decided to do exactly
> > > > > > what you're doing before looking at upstream. Your patch helped
> > > > > > us confirm that we were in the right track.
> > > > > 
> > > > > Great!  Glad I could help in some way!
> > > > > 
> > > > > > > Something like this, perhaps?
> > > > > >
> > > > > > Looks good, but (honest question) what does it buy us using
> > > > > > rcu_normal_after_boot vs rcu_normal? Is the boot process
> > > > > > improved someway?
> > > > > 
> > > > > That's the idea, although I don't have data to show much it actually
> > > > > buys us.
> > > > 
> > > > It means that grace periods can be expedited during boot.  If you really
> > > > care about boot speed, you can also set rcu_expedited=1 and also
> > > > rcu_normal_after_boot=1, which will expedite all grace periods during
> > > > the boot process, but stop doing so just before spawning init.
> > > > After that point, any attempt to do an expedited grace period gets you
> > > > a normal grace period instead.
> > > > 
> > > > So you get fast boot and then clean realtime.
> > > > 
> > > > > > As long as we're rcu_normal=1 before launching user-space,
> > > > > > this should be fine.
> > > > > 
> > > > > Agreed.
> > > > 
> > > > Yes, you can also set them manually instead of at boot, if you wish.
> > > > 
> > > > 							Thanx, Paul
> > > 
> > > FWIW
> > > 
> > > Acked-by: Michael S. Tsirkin <mst@redhat.com>
> > > 
> > > But I have a question - here's the commit that started
> > > it all:
> > > 
> > > 
> > > commit be3fc413da9eb17cce0991f214ab019d16c88c41
> > > Author: Eric Dumazet <eric.dumazet@gmail.com>
> > > Date:   Mon May 23 23:07:32 2011 +0000
> > > 
> > >     net: use synchronize_rcu_expedited()
> > >     
> > >     synchronize_rcu() is very slow in various situations (HZ=100,
> > >     CONFIG_NO_HZ=y, CONFIG_PREEMPT=n)
> > >     
> > >     Extract from my (mostly idle) 8 core machine :
> > >     
> > >      synchronize_rcu() in 99985 us
> > >      synchronize_rcu() in 79982 us
> > >      synchronize_rcu() in 87612 us
> > >      synchronize_rcu() in 79827 us
> > >      synchronize_rcu() in 109860 us
> > >      synchronize_rcu() in 98039 us
> > >      synchronize_rcu() in 89841 us
> > >      synchronize_rcu() in 79842 us
> > >      synchronize_rcu() in 80151 us
> > >      synchronize_rcu() in 119833 us
> > >      synchronize_rcu() in 99858 us
> > >      synchronize_rcu() in 73999 us
> > >      synchronize_rcu() in 79855 us
> > >      synchronize_rcu() in 79853 us
> > >     
> > >     When we hold RTNL mutex, we would like to spend some cpu cycles but not
> > >     block too long other processes waiting for this mutex.
> > >     
> > >     We also want to setup/dismantle network features as fast as possible at
> > >     boot/shutdown time.
> > > 
> > > 
> > > To make sure this does not regress for RT,
> > > how about clearing this flag on shutdown as well?
> > 
> > By that, you mean having some way to force all grace periods to be
> > expedited during shutdown?  Or am I missing your point?
> > 
> > 							Thanx, Paul
> 
> Exactly. And maybe kexec.

If the relevant maintainers are OK with that, I am OK with it as long
as it is non-default (at least to begin with) and does not introduce
additional Kconfig questions.  My guess is that a boot parameter would
work best, but something to discuss.

							Thanx, Paul