Re: RT patch acceptance

[Andi Kleen <ak@muc.de>]

On Fri, May 27, 2005 at 03:13:17PM +0200, Ingo Molnar wrote:
> 
> * Andi Kleen <ak@muc.de> wrote:
> 
> > On Fri, May 27, 2005 at 02:48:37PM +0200, Ingo Molnar wrote:
> > > * Andi Kleen <ak@muc.de> wrote:
> > > 
> > > > [...] Even normal kernels must have reasonably good latency, as long 
> > > > as it doesnt cost unnecessary performance.
> > > 
> > > they do get reasonably good latency (within the hard constraints of the 
> > > possibilities of a given preemption model), due to the cross-effects 
> > > between the various preemption models, that i explained in detail in 
> > > earlier mails. Something that directly improves latencies on 
> > > CONFIG_PREEMPT improves the 'subsystem-use latencies' on PREEMPT_RT.  
> > 
> > I was more thinking of improvements for !PREEMPT.
> 
> how would you do that, if even a first step (PREEMPT_VOLUNTARY) was 
> opposed by some as possibly hurting throughput? I'm really curious, what 
> would you do to improve PREEMPT_NONE's latencies?

Mostly in the classical way. Add cond_resched where needed. Break
up a few locks. Perhaps convert a few spinlocks that block preemption
to long and which are not taken that often to a new kind of
sleep/spinlock (TBD).  Then add more reschedule point again.

> 
> > > Also there's the positive interaction between scalability and latencies 
> > > as well.
> >
> > That sound more like bugs that should just be fixed in the main kernel 
> > by more scheduling. Can you give details and examples?
> 
> what i meant is a pretty common-sense thing: the more independent the
> locks are, the more shortlived locking is, the less latencies there are.

At least on SMP the most finegrained locking is not always the best;
you can end up with bouncing cache lines all the time, with two CPUs
synchronizing to each other all the time, which is just slow.
it is sometimes better to batch things with less locks.
And every lock has a cost even when not taken, and they add up pretty quickly.

> The reverse is true too: most of the latency-breakers move code out from
> under locks - which obviously improves scalability too. So if you are 
> working on scalability you'll indirectly improve latencies - and if you 
> are working on reducing latencies, you often improve scalability.

But I agree that often less latency is good even for scalability.


> > > but it's certainly not for free. Just like there's no zero-cost
> > > virtualization, or there's no zero-cost nanokernel approach either,
> > > there's no zero-cost single-kernel-image deterministic system either.
> > > 
> > > and the argument about binary kernels - that's a choice up to vendors
> > 
> > It is not only binary distribution kernels. I always use my own self
> > compiled kernels, but I certainly would not want a special kernel just
> > to do something normal that requires good latency (like sound use).
> 
> for good sound you'll at least need PREEMPT_VOLUNTARY. You'll need 
> CONFIG_PREEMPT for certain workloads or pro-audio use.

AFAIK the kernel has quite regressed recently, but that was not true
(for reasonable sound) at least for some earlier 2.6 kernels and
some of the low latency patchkit 2.4 kernels.

So it is certainly possible to do it without preemption.

> the impact of PREEMPT on the codebase has a positive effect as well: it 
> forces us to document SMP data structure dependencies better. Under 
> PREEMPT_NONE it would have been way too easy to get into the kind of 
> undocumented interdependent data structure business that we so well know 
> from the big kernel lock days. get_cpu()/put_cpu() precisely marks the 
> critical section where we use a given per-CPU data structure.

Nah, there is still quite some code left that is unmarked, but 
ignores this case for various reason (e.g. in low level exception handling
which is preempt off anyways). However you are right it might have helped
a bit for generic code. But it is still quite ugly...

-Andi