Re: Making rcu_normal=1 in RT

["Paul E. McKenney" <paulmck@linux.vnet.ibm.com>]

On Thu, Oct 13, 2016 at 07:25:56PM +0300, Michael S. Tsirkin wrote:
> On Wed, Oct 12, 2016 at 01:32:23PM -0700, Paul E. McKenney wrote:
> > On Wed, Oct 12, 2016 at 12:15:53PM -0500, Julia Cartwright wrote:
> > > On Wed, Oct 12, 2016 at 12:49:56PM -0400, Luiz Capitulino wrote:
> > > > On Wed, 12 Oct 2016 11:21:14 -0500
> > > > Julia Cartwright <julia@ni.com> wrote:
> > > > 
> > > > > On Wed, Oct 12, 2016 at 11:12:51AM -0400, Luiz Capitulino wrote:
> > > > > > Hi,
> > > > > > 
> > > > > > We have the following patch applied to the RT tree:
> > > > > > 
> > > > > >   commit a9d3cc781a3306bfa332fa7cb6134b70696058d5
> > > > > >   Author: Josh Cartwright <joshc@ni.com>
> > > > > >   Date:   Tue Oct 27 07:31:53 2015 -0500
> > > > > >   
> > > > > >       net: Make synchronize_rcu_expedited() conditional on !RT_FULL
> > > > > > 
> > > > > > However, as noted by Michael, making rcu_normal=1 default in the
> > > > > > RT kernel should have the same effect (ie.  not calling
> > > > > > synchronize_sched_expedited()).
> > > > > > 
> > > > > > So, honest question, is there a reason not to:
> > > > > > 
> > > > > >  1. Drop the patch above
> > > > > >  2. Make rcu_normal=1 default?  
> > > > > 
> > > > > I think this is probably a cleaner way to fix the problems which
> > > > > motivated this patch in the first place.  In my defense, the above patch
> > > > > predates rcu_normal :).
> > > > 
> > > > No need to defend yourself! We debugged this very spike in one of
> > > > our kernels that don't have rcu_normal. We decided to do exactly
> > > > what you're doing before looking at upstream. Your patch helped
> > > > us confirm that we were in the right track.
> > > 
> > > Great!  Glad I could help in some way!
> > > 
> > > > > Something like this, perhaps?
> > > >
> > > > Looks good, but (honest question) what does it buy us using
> > > > rcu_normal_after_boot vs rcu_normal? Is the boot process
> > > > improved someway?
> > > 
> > > That's the idea, although I don't have data to show much it actually
> > > buys us.
> > 
> > It means that grace periods can be expedited during boot.  If you really
> > care about boot speed, you can also set rcu_expedited=1 and also
> > rcu_normal_after_boot=1, which will expedite all grace periods during
> > the boot process, but stop doing so just before spawning init.
> > After that point, any attempt to do an expedited grace period gets you
> > a normal grace period instead.
> > 
> > So you get fast boot and then clean realtime.
> > 
> > > > As long as we're rcu_normal=1 before launching user-space,
> > > > this should be fine.
> > > 
> > > Agreed.
> > 
> > Yes, you can also set them manually instead of at boot, if you wish.
> > 
> > 							Thanx, Paul
> 
> FWIW
> 
> Acked-by: Michael S. Tsirkin <mst@redhat.com>
> 
> But I have a question - here's the commit that started
> it all:
> 
> 
> commit be3fc413da9eb17cce0991f214ab019d16c88c41
> Author: Eric Dumazet <eric.dumazet@gmail.com>
> Date:   Mon May 23 23:07:32 2011 +0000
> 
>     net: use synchronize_rcu_expedited()
>     
>     synchronize_rcu() is very slow in various situations (HZ=100,
>     CONFIG_NO_HZ=y, CONFIG_PREEMPT=n)
>     
>     Extract from my (mostly idle) 8 core machine :
>     
>      synchronize_rcu() in 99985 us
>      synchronize_rcu() in 79982 us
>      synchronize_rcu() in 87612 us
>      synchronize_rcu() in 79827 us
>      synchronize_rcu() in 109860 us
>      synchronize_rcu() in 98039 us
>      synchronize_rcu() in 89841 us
>      synchronize_rcu() in 79842 us
>      synchronize_rcu() in 80151 us
>      synchronize_rcu() in 119833 us
>      synchronize_rcu() in 99858 us
>      synchronize_rcu() in 73999 us
>      synchronize_rcu() in 79855 us
>      synchronize_rcu() in 79853 us
>     
>     When we hold RTNL mutex, we would like to spend some cpu cycles but not
>     block too long other processes waiting for this mutex.
>     
>     We also want to setup/dismantle network features as fast as possible at
>     boot/shutdown time.
> 
> 
> To make sure this does not regress for RT,
> how about clearing this flag on shutdown as well?

By that, you mean having some way to force all grace periods to be
expedited during shutdown?  Or am I missing your point?

							Thanx, Paul