Re: [patch, 2.6.11-rc2] sched: /proc/sys/kernel/rt_cpu_limit tunable

["Jack O'Quin" <joq@io.com>]

Ingo Molnar <mingo@elte.hu> writes:

> * Jack O'Quin <joq@io.com> wrote:
>
>> It works great...
>> 
>>   http://www.joq.us/jack/benchmarks/rt_cpu_limit
>>   http://www.joq.us/jack/benchmarks/rt_cpu_limit+compile
>>   http://www.joq.us/jack/benchmarks/.SUMMARY
>> 
>> I'll experiment with it some more, but this seems to meet all my
>> needs.  As one would expect, the results are indistinguishable from
>> SCHED_FIFO...

> very good. Could you try another thing, and set the rt_cpu_limit to less
> than the CPU utilization 'top' reports during the test (or to less than
> the DSP CPU utilization in the stats), to deliberately trigger the
> limiting code? This both tests the limit and shows the effects it has. 
> (there should be xruns and a large Delay Maximum.)

Here are some runs with rt_cpu_limit set to 30%.  (No point in trying
to compile in the background.)

   http://www.joq.us/jack/benchmarks/rt_cpu_limit.30

As expected, the Delay Max and XRUN Count do go to hell...

# rt_cpu_limit
Delay Maximum . . . . . . . . :   290   usecs
Delay Maximum . . . . . . . . :   443   usecs
Delay Maximum . . . . . . . . :   232   usecs

# rt_cpu_limit.30
Delay Maximum . . . . . . . . : 60294   usecs
Delay Maximum . . . . . . . . : 77742   usecs
Delay Maximum . . . . . . . . : 589697   usecs

# rt_cpu_limit
XRUN Count  . . . . . . . . . :     0
XRUN Count  . . . . . . . . . :     0
XRUN Count  . . . . . . . . . :     0

# rt_cpu_limit.30
XRUN Count  . . . . . . . . . :    25
XRUN Count  . . . . . . . . . :    12
XRUN Count  . . . . . . . . . :    15

So, the throttling obviously does work, in the sense of the system
being able to protect itself from RT CPU hogging.  The failure of the
JACK subsystem is rather catastophic, however.

Look at this graph...

   http://www.joq.us/jack/benchmarks/rt_cpu_limit.30/jack_test3-2.6.11-rc2-q2-200501251346.png

At around 55 seconds into the run, JACK got in trouble and throttled
itself back to approximately the 30% limit (actually a little above).
Then, around second 240 it got in trouble again, this time collapsing
completely.  I'm a bit confused by all the messages in that log, but
it appears that approximately 9 of the 20 clients were evertually shut
down by the JACK server.  It looks like the second collapse around 240
also caused the scheduler to revoke RT privileges for the rest of the
run (just a guess).

This brings us to the issue of graceful degradation.  IMHO, if we
follow the LKML bias for seeing questions from a kernel and not user
perspective, the results will not be very satisfactory.

JACK can probably do a better job of shutting down hyperactive
realtime clients than the kernel, because it knows more about what the
user is trying to do.  Multiplying incomprehesible rlimits values does
not help much that I can see.

Sometimes musicians want to "push the envelope" using CPU-hungry
realtime effects like reverbs or Fourier Transforms.  It is often hard
to predict how much of this sort of load a given system can handle.
JACK reports its subsystem-wide "DSP load" as a moving average,
allowing users to monitor it.  It might be helpful if the kernel's
estimate of this number were also available somewhere (but maybe that
value has no meaning to users).  Often, the easiest method is to push
things to the breaking point, and then back off a bit.

For this kind of usage scenario, the simpler `rt_cpu_limit' (B4) patch
works much better than the `rt_cpu_ratio' (D4) patch, allowing a
suitably privileged user to adjust the per-CPU limit directly while
experimenting with system limits.  (Don't ask me what any of this
means in an SMP complex.)

The equivalent rlimits experiment probably requires: 

  (1) editing /etc/security/limits.conf
  (2) shutting everything down
  (3) logout
  (4) login
  (5) restarting the test

Eventually there will be ulimit support in the shells.  Then, I
suppose this can be streamlined a little, but not much.  The problem
is that there are often many audio applications running, with complex
connections between them.  So, this is not a simple matter of stopping
and starting a single program (as most of you probably imagine).
-- 
  joq