Re: PREEMPT/PREEMPT_RT question

["Paul E. McKenney" <paulmck@us.ibm.com>]

On Tue, Jul 12, 2005 at 09:26:35PM +0200, Ingo Molnar wrote:
> * Paul E. McKenney <paulmck@us.ibm.com> wrote:
> > So, time to get serious about a bit of code cleanup:
> > 
> > o	The heavyweight atomic operations in rcu_read_lock() and
> > 	rcu_read_unlock() are not needed in UP kernels, since
> > 	interrupts are disabled.
> 
> atomic_*() ops should already be lightweight on UP.

Agreed, will worry about architectures where they might not be later.

> > o	In order to get things to work in both CONFIG_PREEMPT and
> > 	CONFIG_PREEMPT_RT, I ended up using the following:
> > 
> > 		#ifdef CONFIG_PREEMPT_RT
> > 
> > 		#define rcu_spinlock_t _raw_spinlock_t
> > 		#define rcu_spin_lock(l, f) _raw_spin_lock(l)
> > 		#define rcu_spin_trylock(l, f) _raw_spin_trylock(l)
> > 		#define rcu_spin_unlock(l, f) _raw_spin_unlock(l)
> > 		#define RCU_SPIN_LOCK_UNLOCKED RAW_SPIN_LOCK_UNLOCKED
> > 
> > 		#else /* #ifdef CONFIG_PREEMPT_RT */
> > 
> > 		#define rcu_spinlock_t spinlock_t
> > 		#define rcu_spin_lock(l, f) spin_lock_irqsave(l, f)
> > 		#define rcu_spin_trylock(l, f) spin_trylock_irqsave(l, f)
> > 		#define rcu_spin_unlock(l, f) spin_unlock_irqrestore(l, f)
> > 		#define RCU_SPIN_LOCK_UNLOCKED SPIN_LOCK_UNLOCKED
> > 
> > 		#endif /* #else #ifdef CONFIG_PREEMPT_RT */
> > 
> > 	Then using rcu_spin_lock() &c everywhere.  The problem is
> > 	that (as near as I can tell) the only way to prevent interrupts 
> > 	from running on the current CPU in CONFIG_PREEMPT kernels is
> > 	to use the _irq spinlock primitives, but _raw_spin_lock() does
> > 	the job in CONFIG_PREEMPT_RT (since interrupts are run in process
> > 	context, right).  I could use _irq in both, but that would
> > 	unnecessarily degrade interrupt latency in CONFIG_PREEMPT_RT.
> > 
> > 	Suggestions???
> 
> in PREEMPT_RT, if you define a spinlock type as raw_spinlock_t (via 
> DEFINE_RAW_SPINLOCK(lock)) then the spin_lock*() APIs automatically 
> switch over to that type. I.e. no need to do the rcu_ stuff AFAICT.  
> (unless i missed some detail about what you are trying to do.) Raw 
> spinlocks under PREEMPT_RT pair with the raw IRQ flag, i.e.  
> spin_lock_irq() will disable hard interrupts. Generally i'd suggest to 
> go for the raw spinlock and raw-irq-flag-disabling variant, because RCU 
> locking is core enough functionality to warrant atomic treatment. If 
> there's any latency path in it, we can work on breaking it up later.

Fair enough!

The thing I am (perhaps foolishly) trying to do is get to a single
source base that provides PREEMPT_RCU under both CONFIG_PREEMPT and
CONFIG_PREEMPT_RT.  Since you are OK with RCU disabling interrupts,
I am happy.

> it's perfectly fine to disable raw interrupts with the RCU code, as long 
> as you do O(1) amount of work. (where the constant factor isnt 2^(2^32) 
> ;-)

Will try to keep it down to a dull roar.  ;-)

> > Some remaining shortcomings of the current code:
> > 
> > o	Untested on CONFIG_PREEMPT_RT (working on this, suggestions
> > 	for 4-CPU-stable CONFIG_PREEMPT_RT versions most welcome).
> 
> latest (-51-28) has no known regressions, and i regularly boot on 4-way 
> (and irregularly on 8-way) boxes. So if you see something strange on the 
> latest -RT kernel, please report it.

And stock -51-7 just passed nine rounds of kernbench + LTP on some 4-CPU
x86 machines, with one still running.  Good stuff!!!

							Thanx, Paul