Re: Making rcu_normal=1 in RT

["Michael S. Tsirkin" <mst@redhat.com>]

On Wed, Oct 12, 2016 at 01:32:23PM -0700, Paul E. McKenney wrote:
> On Wed, Oct 12, 2016 at 12:15:53PM -0500, Julia Cartwright wrote:
> > On Wed, Oct 12, 2016 at 12:49:56PM -0400, Luiz Capitulino wrote:
> > > On Wed, 12 Oct 2016 11:21:14 -0500
> > > Julia Cartwright <julia@ni.com> wrote:
> > > 
> > > > On Wed, Oct 12, 2016 at 11:12:51AM -0400, Luiz Capitulino wrote:
> > > > > Hi,
> > > > > 
> > > > > We have the following patch applied to the RT tree:
> > > > > 
> > > > >   commit a9d3cc781a3306bfa332fa7cb6134b70696058d5
> > > > >   Author: Josh Cartwright <joshc@ni.com>
> > > > >   Date:   Tue Oct 27 07:31:53 2015 -0500
> > > > >   
> > > > >       net: Make synchronize_rcu_expedited() conditional on !RT_FULL
> > > > > 
> > > > > However, as noted by Michael, making rcu_normal=1 default in the
> > > > > RT kernel should have the same effect (ie.  not calling
> > > > > synchronize_sched_expedited()).
> > > > > 
> > > > > So, honest question, is there a reason not to:
> > > > > 
> > > > >  1. Drop the patch above
> > > > >  2. Make rcu_normal=1 default?  
> > > > 
> > > > I think this is probably a cleaner way to fix the problems which
> > > > motivated this patch in the first place.  In my defense, the above patch
> > > > predates rcu_normal :).
> > > 
> > > No need to defend yourself! We debugged this very spike in one of
> > > our kernels that don't have rcu_normal. We decided to do exactly
> > > what you're doing before looking at upstream. Your patch helped
> > > us confirm that we were in the right track.
> > 
> > Great!  Glad I could help in some way!
> > 
> > > > Something like this, perhaps?
> > >
> > > Looks good, but (honest question) what does it buy us using
> > > rcu_normal_after_boot vs rcu_normal? Is the boot process
> > > improved someway?
> > 
> > That's the idea, although I don't have data to show much it actually
> > buys us.
> 
> It means that grace periods can be expedited during boot.  If you really
> care about boot speed, you can also set rcu_expedited=1 and also
> rcu_normal_after_boot=1, which will expedite all grace periods during
> the boot process, but stop doing so just before spawning init.
> After that point, any attempt to do an expedited grace period gets you
> a normal grace period instead.
> 
> So you get fast boot and then clean realtime.
> 
> > > As long as we're rcu_normal=1 before launching user-space,
> > > this should be fine.
> > 
> > Agreed.
> 
> Yes, you can also set them manually instead of at boot, if you wish.
> 
> 							Thanx, Paul

FWIW

Acked-by: Michael S. Tsirkin <mst@redhat.com>

But I have a question - here's the commit that started
it all:


commit be3fc413da9eb17cce0991f214ab019d16c88c41
Author: Eric Dumazet <eric.dumazet@gmail.com>
Date:   Mon May 23 23:07:32 2011 +0000

    net: use synchronize_rcu_expedited()
    
    synchronize_rcu() is very slow in various situations (HZ=100,
    CONFIG_NO_HZ=y, CONFIG_PREEMPT=n)
    
    Extract from my (mostly idle) 8 core machine :
    
     synchronize_rcu() in 99985 us
     synchronize_rcu() in 79982 us
     synchronize_rcu() in 87612 us
     synchronize_rcu() in 79827 us
     synchronize_rcu() in 109860 us
     synchronize_rcu() in 98039 us
     synchronize_rcu() in 89841 us
     synchronize_rcu() in 79842 us
     synchronize_rcu() in 80151 us
     synchronize_rcu() in 119833 us
     synchronize_rcu() in 99858 us
     synchronize_rcu() in 73999 us
     synchronize_rcu() in 79855 us
     synchronize_rcu() in 79853 us
    
    When we hold RTNL mutex, we would like to spend some cpu cycles but not
    block too long other processes waiting for this mutex.
    
    We also want to setup/dismantle network features as fast as possible at
    boot/shutdown time.


To make sure this does not regress for RT,
how about clearing this flag on shutdown as well?



-- 
MST