Re: [PATCH RFC nohz_full 6/7] nohz_full: Add full-system-idle state machine

["Paul E. McKenney" <paulmck@linux.vnet.ibm.com>]

On Wed, Jul 24, 2013 at 08:09:04PM +0200, Frederic Weisbecker wrote:
> On Thu, Jul 18, 2013 at 10:06:25PM -0700, Paul E. McKenney wrote:
> > > Lets summarize the last sequence, the following happens ordered by time:
> > > 
> > >         CPU 0                          CPU 1
> > > 
> > >      cmpxchg(&full_sysidle_state,
> > >              RCU_SYSIDLE_SHORT,
> > >              RCU_SYSIDLE_LONG);
> > > 
> > >      smp_mb() //cmpxchg
> > > 
> > >      atomic_read(rdtp(1)->dynticks_idle)
> > > 
> > >      //CPU 0 goes to sleep
> > >                                        //CPU 1 wakes up
> > >                                        atomic_inc(rdtp(1)->dynticks_idle)
> > > 
> > >                                        smp_mb()
> > > 
> > >                                        ACCESS_ONCE(full_sysidle_state)
> > > 
> > > 
> > > Are you suggesting that because the CPU 1 executes its atomic_inc() _after_ (in terms
> > > of absolute time) the atomic_read of CPU 0, the ordering settled in both sides guarantees
> > > that the value read from CPU 1 is the one from the cmpxchg that precedes the atomic_read,
> > > or FULL or FULL_NOTED that happen later.
> > > 
> > > If so that's a big lesson for me.                                     
> > 
> > It is not absolute time that matters.  Instead, it is the fact that
> > CPU 0, when reading from ->dynticks_idle, read the old value before the
> > atomic_inc().  Therefore, anything CPU 0 did before that memory barrier
> > preceding CPU 0's read must come before anything CPU 1 did after that
> > memory barrier following the atomic_inc().  For this to work, there
> > must be some access to the same variable on each CPU.
> 
> Aren't we in the following situation?
> 
>     CPU 0                          CPU 1
> 
>     STORE A                        STORE B
>     LOAD B                         LOAD A
> 
> 
> If so and referring to your perfbook, this is an "ears to mouth" situation.
> And it seems to describe there is no strong guarantee in that situation.

"Yes" to the first, but on modern hardware, "no" to the second.  The key
paragraph is Section 12.2.4.5:

	The following pairings from Table 12.1 can be used on modern
	hardware, but might fail on some systems that were produced in
	the 1990s. However, these can safely be used on all mainstream
	hardware introduced since the year 2000.

That said, you are not the first to be confused by this, so I might need
to rework this section to make it clear that each can in fact be used on
modern hardware.

If you happen to have an old Sequent NUMA-Q or Symmetry box lying around,
things are a bit different.  On the other hand, I don't believe that any
of these old boxes are still running Linux.  (Hey, I am as sentimental as
the next guy, but there are limits!)

I updated this section and pushed it, please let me know if this helps!

							Thanx, Paul

> > Or, if you must think in terms of time, you need a separate independent
> > timeline for each variable, with no direct mapping from one timeline to
> > another, except resulting from memory-barrier interactions.
> > 
> > 							Thanx, Paul
> > 
>