Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Mark_H_Johnson]

>> > on low RT load (the common case) the scheduler behaves like the
>> > stock scheduler - the new logic only kicks in if a CPU runqueue has
>> > 2 or more RT tasks running at once.
>
>'the common case' == ordinary (non-RT) Linux boxes! When i implement
>scheduler features i'm always trying to make them as generic as
>possible, i.e. this feature too is structured to be as upstream
>mergeable as possible. For that purpose the change had to be
>low-overhead in the common, non-RT case.
I truly appreciate that. However...

>It is easy to hack the
>scheduler to fix some RT issue but break the generic scheduler - this
>solution is not meant to be such a hack.
I agree but I see the big delay of running the RT task to be a symptom
that the current non RT scheduler is somehow broken. I've reported the
non RT starvation condition several times. Yes, the second CPU is busy,
but I really do want to bump cpu_burn (which is non RT & nice) to run my
(non RT and not nice) stress script / commands instead.

--Mark H Johnson
  <mailto:Mark_H_Johnson@raytheon.com>

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Ingo Molnar]

* Mark_H_Johnson@raytheon.com <Mark_H_Johnson@raytheon.com> wrote:

> > It is easy to hack the scheduler to fix some RT issue but break the 
> > generic scheduler - this solution is not meant to be such a hack.
>
> I agree but I see the big delay of running the RT task to be a symptom
> that the current non RT scheduler is somehow broken. I've reported the
> non RT starvation condition several times. Yes, the second CPU is
> busy, but I really do want to bump cpu_burn (which is non RT & nice)
> to run my (non RT and not nice) stress script / commands instead.

well ... doing global balancing on SMP gets really expensive, really
quick. It's simply not an option for 4 way or higher, and even on 2-way
it's measurable. In the RT case arguably latency is more important than
throughput, but i dont think the same case could be made for nice vs. 
non-nice.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Mark_H_Johnson]

>* Esben Nielsen <simlo@phys.au.dk> wrote:
>
>> On Fri, 3 Dec 2004, Ingo Molnar wrote:
>>
>> >
>> > [...]
>> > on low RT load (the common case)
>>
>> In the system I deal with on my day job, 50% of the CPU load is from
>> RT tasks!
>
>i'm not sure what point you are trying to make, but 'low RT load' in
>this context means up to a load of 1.0. (i.e. one or less tasks running
>on average)
I am not quite so sure that is a good assumption with our real time
system either. We use RT priorities based on execution rate with a
single (highest priority) RT task that wakes up the others at the
appropriate times. So an 80 Hz task has a higher RT priority than a
40 Hz task, both are greater than 10 Hz, and all greater than 1 Hz.
We do have a few other tasks for signal handling and other purposes
but most fit the above pattern.

At periodic rate (e.g., once per second) we basically start ALL RT tasks
and let the kernel figure out which one (two actually) should run first.
At that point in time, I may have 10-20 RT tasks ready to run with only
two CPU's to run them on. As the high priority tasks get done, the load
starts to drop, but a one Hz task may get delayed for some time and
interrupted several times during execution (later cycles of higher rate
RT tasks). FYI - I just checked and on our cluster head node, we have
31 RT tasks on a two CPU system.

With the real time load up, the one minute load average on 2.4 systems
oscillates in an odd fashion. For example, looking at 1 hour chart of
load average on my cluster shows the head node oscillating between zero
and five, even though the CPU usage is constant at 17.4%. I assume this
is a sampling error.

If the algorithm is OK for "small numbers" where small is under ten
times the number of CPU's, that is probably OK. [I'll allow that
SOMEONE will try to run a big RT simulation on a 256 CPU SMP machine
and complain later]

>also, this only applies to SMP systems.
Yes.

>thirdly, even if the new balancing code kicks in, the overhead is not
>that bad, and it's for a purpose: we rather want an RT task to run on a
>free CPU.
I agree though I wish it was possible to determine if the interrupt
preemption was "short" or "long" to decide to switch or not [the switch
DOES have costs...]. I believe this is why the max latency of my CPU
task went down (by about 1 msec) but the number of short latencies
(> 100 usec) went up.

But I am still seeing starvation of the non RT tasks that was not
present in 2.4. It appears that cpu_burn (non RT, nice) is taking CPU
cycles that should be allocated to my (non RT, not nice) stress test
programs. This distorts the elapsed time of my tests and to some extent
the latency results (since less I/O stress is on the system) of the tests.

--Mark H Johnson
  <mailto:Mark_H_Johnson@raytheon.com>

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Ingo Molnar]

* Mark_H_Johnson@raytheon.com <Mark_H_Johnson@raytheon.com> wrote:

> >i'm not sure what point you are trying to make, but 'low RT load' in
> >this context means up to a load of 1.0. (i.e. one or less tasks running
> >on average)

> I am not quite so sure that is a good assumption with our real time
> system either. [...]

I implemented this feature with your workload in mind. This is what i
wrote:

> > on low RT load (the common case) the scheduler behaves like the
> > stock scheduler - the new logic only kicks in if a CPU runqueue has
> > 2 or more RT tasks running at once.

'the common case' == ordinary (non-RT) Linux boxes! When i implement
scheduler features i'm always trying to make them as generic as
possible, i.e. this feature too is structured to be as upstream
mergeable as possible. For that purpose the change had to be
low-overhead in the common, non-RT case. It is easy to hack the
scheduler to fix some RT issue but break the generic scheduler - this
solution is not meant to be such a hack.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Mark_H_Johnson]

Comparison of .32-0 and .31-15 results


      within 100 usec
       CPU loop (%)   Elapsed Time (sec)    2.4
Test   32-0  31-15     32-0   31-15  |   CPU  Elapsed
X     94.58  99.22      67      68   |  97.20   70
top   95.29  97.96      39      34   |  97.48   29
neto  94.24  99.98     360     360   |  96.23   36
neti  94.83  98.31     360     350   |  95.86   41
diskw 90.77  99.57     360     360 * |  77.64   29
diskc 93.47  97.49     360     360   |  84.12   77
diskr 93.49  98.35     320     180   |  90.66   86
total                 1866    1712   |         368
* wide variation in audio duration

Grr. This appears that .32-0 is MUCH WORSE than 31.15 at
keeping the relatively small (100 usec) latencies down.
Probably due to the CPU task switch that prevents the
longer ones from occurring. The MAX CPU latencies are down
in most cases (over 4 msec down to about 3 msec) which
is a good result.

The long elapsed times appear to indicate that we are
starving the "stress test" application (and likely
running the niced, non RT cpu_burn instead).

--Mark H Johnson
  <mailto:Mark_H_Johnson@raytheon.com>

[patch] Real-Time Preemption, -RT-2.6.10-rc1-mm3-V0.7.25-1

[Ingo Molnar]

i have released the -V0.7.25-1 Real-Time Preemption patch, which can be
downloaded from the usual place:

    http://redhat.com/~mingo/realtime-preempt/

this is a fixes-only release that resolves a couple of bugs that slipped
into -V0.7.25-0:

 - lockup/deadlock fix: make debug_direct_keyboard default to 0. It is
   only a debug helper to be used for development, it was never intended
   to be enabled. This fix should resolve the bugs reported by Gunther
   Persoons and Mark H. Johnson.

 - fix symbol export problems in rtc.ko, reported by Remi Colinet, based
   on the patch from K.R. Foley.

 - make preempt_wakeup_timing default to 1 if enabled in the .config, as 
   originally intended.

to create a -V0.7.25-1 tree from scratch, the patching order is:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
   http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc1.bz2
   http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc1/2.6.10-rc1-mm3/2.6.10-rc1-mm3.bz2
   http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc1-mm3-V0.7.25-1

	Ingo

[patch] Real-Time Preemption, -RT-2.6.10-rc2-mm1-V0.7.27-0

[Ingo Molnar]

i have released the -V0.7.27-0 Real-Time Preemption patch, which can be
downloaded from the usual place:

	http://redhat.com/~mingo/realtime-preempt/

this is a pure merge of -V0.7.26-5 to 2.6.10-rc2-mm1, there are no other
changes.

to create a -V0.7.27-0 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm1/2.6.10-rc2-mm1.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm1-V0.7.27-0

	Ingo

[patch] Real-Time Preemption, -RT-2.6.10-rc2-mm1-V0.7.27-1

[Ingo Molnar]

i have released the -V0.7.27-1 Real-Time Preemption patch, which can be
downloaded from the usual place:

	http://redhat.com/~mingo/realtime-preempt/

this quick update fixes a couple of build bugs.

Changes since a -V0.7.27-0:

 - fix iptables compilation error

 - fix selinux compilation error

to create a -V0.7.27-1 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm1/2.6.10-rc2-mm1.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm1-V0.7.27-1

	Ingo

[patch] Real-Time Preemption, -RT-2.6.10-rc2-mm1-V0.7.27-3

[Ingo Molnar]

i have released the -V0.7.27-3 Real-Time Preemption patch, which can be
downloaded from the usual place:

	http://redhat.com/~mingo/realtime-preempt/

this is another quick update to fix a couple of bugs. Sorry about the
fast pace of updates but these fixes are worth having ASAP:

Changes since a -V0.7.27-1:

 - fix module-put BKL count bug - this could explain/fix the lockups
   reported by Rui Nuno Capela.

 - fixed a netfilter related networking deadlock reported by Mark H. 
   Johnson two weeks ago, it triggered on my testbox today. This (rare)
   bug could potentially explain some of the other lockup reports that
   are still open.

 - fix load average constant +1.0 offset when PREEMPT_RT is enabled. 
   This was an artifact of the IRQ-threading of the timer interrupt.

to create a -V0.7.27-3 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm1/2.6.10-rc2-mm1.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm1-V0.7.27-3

	Ingo

[patch] Real-Time Preemption, -RT-2.6.10-rc2-mm1-V0.7.28-0

[Ingo Molnar]

i have released the -V0.7.28-0 Real-Time Preemption patch, which can be
downloaded from the usual place:

	http://redhat.com/~mingo/realtime-preempt/

this is a fixes & latency-reduction release.

Changes since a -V0.7.27-3:

 - made the UP-ioapic code a bit more conservative again - maybe some of
   the lockups are related?

 - removed the BKL from the sound code in a cleaner way and
   removed the quite fragile 'negative ->lock_depth' code. Much less
   intrusive than i originally thought, and much cleaner as well.

 - more fixes to the wakeup-timing logic, 4 false positives fixed in
   total, mostly related to new-task-wakeup not accurately starting the
   tracer.

 - fixed the mmx-memcpy related latency reported by Florian Schmidt and 
   others. Also turned off the MMX/SSE ops in the RAID code, which 
   can introduce similar latencies.

 - kgdb fix from Bill Huey

 - knfsd shutdown with-BKL-held fix

 - highmem compilation fix

 - profiling related crash fix

 - implemented 'direct-path' rescheduling to further reduce scheduling
   latency: the kernel will now in most cases go from try_to_wakeup()
   into the scheduler directly without re-enabling interrupts ever again
   (and thus not giving irq handlers a window to increase latency). This
   is also the final fix for irq nesting and irq-stack recursion.

 - turn off sync wakeups on PREEMPT_RT -> they are latency generators

to create a -V0.7.28-0 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm1/2.6.10-rc2-mm1.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm1-V0.7.28-0

	Ingo

[patch] Real-Time Preemption, -RT-2.6.10-rc2-mm1-V0.7.28-1

[Ingo Molnar]

i have released the -V0.7.28-1 Real-Time Preemption patch, which can be
downloaded from the usual place:

	http://redhat.com/~mingo/realtime-preempt/

this should fix the lockup bug reported by Florian Schmidt.

there's a generic PREEMPT bug in the upstream kernel: there exists a
single-instruction race window in __flush_tlb(), if the kernel preempted
exactly there in a lazy-TLB thread and certain other, rare scheduling
and MM properties were true as well (a certain constellation of threads
and lazy-TLB kernel threads occured), and the lazy-TLB task then got
another user TLB to inherit, and switched to a task from which it
inherited that new TLB, thus the wrong cr3 was loaded and inherited by
this next, non-lazy-TLB next task; then (and only then) this scenario
would typically manifest itself in the form of an infinite pagefault
lockup occuring much after the fact, upon the next userspace access (to
the joy of a totally baffled kernel developer). I suspect from the
description you can guess how much fun it was to debug it =B-)

the bug is even more rare in the generic kernel, because there most (but
not all) TLB flush points are in a critical section.

this fix could resolve some of the other 'my box just locked up'
reports.

Changes since a -V0.7.28-0:

 - reverted the UP-ioapic change - it was unrelated to the lockup and it
   is known to cause problems on certain IDE/soundcard combinations.

 - fixed and improved the trace_print_on_crash tracing feature - it was
   highly needed to find the TLB bug ...

to create a -V0.7.28-1 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm1/2.6.10-rc2-mm1.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm1-V0.7.28-1

	Ingo

[patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.29-0

[Ingo Molnar]

i have released the -V0.7.29-0 Real-Time Preemption patch, which can be
downloaded from the usual place:

	http://redhat.com/~mingo/realtime-preempt/

this is a pure merge of -V0.7.28-2 to 2.6.10-rc2-mm2. -rc2-mm2 itself is
a fixes-only release.

to create a -V0.7.29-0 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm2/2.6.10-rc2-mm2.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm2-V0.7.29-0

	Ingo

[patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

i have released the -V0.7.30-2 Real-Time Preemption patch, which can be
downloaded from the usual place:

	http://redhat.com/~mingo/realtime-preempt/

the biggest change in this release are fixes for priority-inheritance
bugs uncovered by Esben Nielsen pi_test suite. These bugs could explain
some of the jackd-under-load latencies reported.

Changes since -V0.7.29-0:

 - priority inheritance handling fixes:

    - sort the RT wakees at wakeup time, not at block-time: an RT task
      might have gotten boosted while it slept.

    - fix priority-restoration bug at mutex-release time

    - use task_rt() not p->policy to determine whether a task needs 
      PI handling - a SCHED_OTHER task might be boosted to RT prio.

    - fix mutex_setprio() bug: queue now-RT tasks to the active array, 
      otherwise expired SCHED_OTHER tasks will not be properly boosted.

 - went back to the mask-and-delay method of handling hardirqs on 
   UP-IOAPIC as well. Due to APIC prioritization hardirqs can get
   delayed by another, unacked hardirq, so the quick method needs more 
   work before it can be used.

 - added Thomas Gleixner's semaphore -> completion changes for 
   drv->unload_sem. This fixes the module unload crashes reported by 
   Rui Nuno Capela and Shane Shrybman.

 - dvb mutex updates for RT, this fixes the bug reported by Christian 
   Meder.

 - e100 fix from K.R. Foley - this should fix the boot-time e100
   enable_irq warning.

 - NFS lockd mutex RT fixes from Thomas Gleixner - this could fix some
   of the bugs reported by Bill Huey.

 - PREEMPT_VOLUNTARY fixes - this could fix the boot-time hang reported 
   by Lee Revell.

 - wake up irq thread upon creation - this solves the 'irq thread only 
   changes priority after first interrupt arrives' anomaly reported.

to create a -V0.7.30-2 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm2/2.6.10-rc2-mm2.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm2-V0.7.30-2

	Ingo

[patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-9

[Ingo Molnar]

i have released the -V0.7.30-9 Real-Time Preemption patch, which can be
downloaded from the usual place:

    http://redhat.com/~mingo/realtime-preempt/

this is a fixes-only release.

most importantly it includes a JACK related latency fix. With Florian
Schmidt's great detective work we honed in on a big latency source
within JACK: the use of named pipes (fifos) on journalled filesystems. 

This issue has been empirically identified before (and is mentioned in
the JACK howto) but has never been given high enough prominence. It
turns out that the atime updates done while read()ing or write()ing
named pipes causes the delays - it may under certain circumstances call
out into the journalling code. It may block even on non-journalled
filesystems.

To work this issue around, when PREEMPT_RT is enabled the -30-9 kernel
skips atime updates on named-fifos. (it's pretty pointless anyway.)
Alternative userspace workarounds are to put the fifos on tmpfs/ramfs,
or to mark the filesystem noatime,nodiratime.

those experiencing xruns under JACK should definitely try the -30-9
kernel.

Changes since -V0.7.30-2:

 - named fifo reads/writes are now atomic, whenever possible

 - fixed pi_lock related SMP & CRITICAL_IRQSOFF_TIMING lockups, this 
   could resolve the lockups reported by Mark H. Johnson.

 - fixed one more PI buglet: wake up the new owner _after_ restoring
   the priority of the old owner.

 - made the NMI oopser more robust - it should print out some message 
   in pretty much any locking scenario.

 - added the blocker device used by Esben Nielsen's pi_test suite.

 - added user-triggerable ALSA xrun tracing to the patch: if a 
   sound IO channel has xrun_debug enabled in /proc then 
   user_trace_stop() will be called before printing the xrun message,
   and the current trace will be saved to /proc/latency_trace. This is a
   'one-shot' tracing method for now. I can be activated via:

     echo 1 > /proc/asound/card0/pcm0p/xrun_debug

     echo 1 > /proc/sys/kernel/trace_user_triggered
     echo 1 > /proc/sys/kernel/trace_freerunning
     echo 0 > /proc/sys/kernel/preempt_max_latency
     echo 0 > /proc/sys/kernel/preempt_thresh
     echo 0 > /proc/sys/kernel/preempt_wakeup_timing

     ./gettimeofday 0 1

  gettimeofday.c is attached below. The JACK fifo xrun source was found
  via this tracing facility.

to create a -V0.7.30-9 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm2/2.6.10-rc2-mm2.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm2-V0.7.30-9

	Ingo


-- gettimeofday.c:

#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include <sys/wait.h>
#include <linux/unistd.h>

int main (int argc, char **argv)
{
	if (argc != 3) {
		printf("usage: gettimeofday <val1> <val2>\n");
		exit(0);
	}
	gettimeofday(atol(argv[1]), atol(argv[2]));

	return 0;
}

[patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-10

[Ingo Molnar]

i have released the -V0.7.30-10 Real-Time Preemption patch, which can be
downloaded from the usual place:

    http://redhat.com/~mingo/realtime-preempt/

this is a fixes-only release.

the most important fixes are the ones to the priority inheritance logic
(affecting the latency of RT tasks), discovered and reported by Esben
Nielsen. I also found two more PI bugs running the new pi_test2 code
from Esben.

Changes since -V0.7.30-9:

 - PI fixes:

   - the waiter->prio field caused wrong priority settings upon unlock, 
     resulting in PI bugs in the nested-locking case.

   - use rt_task() when determining PI tasks, not p->policy.

   - in the blocking-on-blocked-task nesting case both promote now-RT
     tasks to the pi_waiters list and queue them to the head of the wait
     list, and demote now-non-RT tasks from the pi_waiters list and 
     queue them to the tail of the wait list.

 - PI-debugging blocker device update from Esben Nielsen

 - module build fix: export user_trace_stop symbol, this fixes the error 
   reported by Florian Schmidt

 - tracer fix: in the default !freerunning tracing mode, if the trace
   buffer overflows (this is relatively rare, but can happen) then the
   tracer overwrote kernel memory that leads to lockups/kernel crashes. 
   Maybe this bug was also the source of the truncated trace bug
   reported by Mark H. Johnson?

 - reduce tracing overhead within schedule() when !tracing_enabled.

to create a -V0.7.30-10 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm2/2.6.10-rc2-mm2.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm2-V0.7.30-10

	Ingo

[patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Ingo Molnar]

i have released the -V0.7.32-0 Real-Time Preemption patch, which can be
downloaded from the usual place:

	http://redhat.com/~mingo/realtime-preempt/

this is a fixes-mostly release with one new feature:

implemented global RT-task balancing on SMP systems, which improves the
latency of RT tasks on SMP systems. The basic problem was that the 2.6
kernel has per-CPU runqueues. In the current design there is no
guarantee that if an RT task starves another, lower-prio (or same-prio)
RT task in a given local runqueue, that the starved task will ever be
migrated to another CPU: it has to wait for the higher-prio task to
finish. In short, task migration on SMP is fundamentally non-RT and
priority-insensitive. In particular the workloads and latencies reported
by Mark H. Johnson reflect such SMP scheduling artifacts.

the new global RT-task balancing feature solves this problem by tracking
the 'RT overload' situation (when there is more than one RT tasks on a
CPU) and makes other CPUs 'pull' RT tasks (and only RT tasks)
immediately when such a situation occurs.

To give an example, in the following scheduling scenario:

  CPU#0					CPU#1

  task-A SCHED_FIFO prio 30		task-C SCHED_FIFO prio 30 [curr]
  task-B SCHED_FIFO prio 40 [curr]

task-B is the currently executing task on CPU#0, task-C is the currently
executing task on CPU#1. Now on the vanilla 2.6 kernel, if task-C
blocks, there's no guarantee that task-A will be run there - if there's
a SCHED_NORMAL task on CPU#1's runqueue then it will run indefinitely. 
With global RT-balancing task-A will be scheduled on CPU#1 immediately
after task-C leaves it.

furthermore, if in the same scenario, if e.g. a RT-prio 35 task-D is
woken up on CPU#0, the vanilla 2.6 scheduler will not move it to CPU#1,
even though it could preempt the prio 30 task-C there. With global
RT-balancing this will happen and task-C will be preempted immediately
and task-D runs.

on low RT load (the common case) the scheduler behaves like the stock
scheduler - the new logic only kicks in if a CPU runqueue has 2 or more
RT tasks running at once.

anyway, while the feature is stable on my SMP testboxes, this is still a
nontrivial ~200-lines delta in the scheduler so there might be problems. 
UP should not be affected by this.

other changes since -V0.7.32-20:

 - local-APIC shutdown fix: this should solve some of the 'reboot hangs' 
   reports.

 - more tracing fixes - might fix the 'truncated traces' problems.

 - reduce the NMI watchdog frequency from 10 KHz to 1000 Hz.

 - dont report futex reschedules as atomicity violations

to create a -V0.7.32-0 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm2/2.6.10-rc2-mm2.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm2-V0.7.32-0

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Ingo Molnar]

> to create a -V0.7.32-0 tree from scratch, the patching order is:

that should read:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm3/2.6.10-rc2-mm3.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm3-V0.7.32-0

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Rui Nuno Capela]

Ingo Molnar wrote:
>
> i have released the -V0.7.32-0 Real-Time Preemption patch, which can be
> downloaded from the usual place:
>

I have a bug to report, it shows on both of my machines (SMP and UP) now
running RT-V0.7.32-2. It seems to be present also on previous RT releases,
and don't even know if it's upstream.

When one usb-storage flash stick is first time unplugged:

usb 4-6: USB disconnect, address 4
slab error in kmem_cache_destroy(): cache `scsi_cmd_cache': Can't free all
objects
 [<c010361f>] dump_stack+0x23/0x25 (20)
 [<c01467d7>] kmem_cache_destroy+0x103/0x1aa (28)
 [<c026e77a>] scsi_destroy_command_freelist+0x97/0xa8 (28)
 [<c026f5b1>] scsi_host_dev_release+0x37/0xe1 (104)
 [<c023c6a9>] device_release+0x7a/0x7c (32)
 [<c01efd90>] kobject_cleanup+0x87/0x89 (28)
 [<c01f07eb>] kref_put+0x52/0xef (40)
 [<c01efdcc>] kobject_put+0x25/0x27 (16)
 [<f8cf1843>] usb_stor_release_resources+0x66/0xca [usb_storage] (16)
 [<f8cf1d93>] storage_disconnect+0x8e/0x9b [usb_storage] (16)
 [<f89ca117>] usb_unbind_interface+0x84/0x86 [usbcore] (28)
 [<c023d915>] device_release_driver+0x75/0x77 (28)
 [<c023db18>] bus_remove_device+0x53/0x82 (20)
 [<c023cae1>] device_del+0x4b/0x9b (20)
 [<f89d142a>] usb_disable_device+0xf5/0x10a [usbcore] (28)
 [<f89cc61c>] usb_disconnect+0xc8/0x164 [usbcore] (40)
 [<f89cd77e>] hub_port_connect_change+0x2ef/0x426 [usbcore] (56)
 [<f89cda7b>] hub_events+0x1c6/0x39d [usbcore] (56)
 [<f89cdc89>] hub_thread+0x37/0x109 [usbcore] (96)
 [<c01009b1>] kernel_thread_helper+0x5/0xb (142893076)


Then, just when the same is plugged in again:

usb 4-6: new high speed USB device using ehci_hcd and address 5
kmem_cache_create: duplicate cache scsi_cmd_cache
BUG at mm/slab.c:1447!
------------[ cut here ]------------
kernel BUG at mm/slab.c:1447!
invalid operand: 0000 [#1]
PREEMPT SMP
Modules linked in: nls_utf8 nls_cp860 vfat fat nls_base usb_storage
ohci1394 ieee1394 ehci_hcd usbhid uhci_hcd intel_mch_agp agpgart evdev
wacom snd_pcm_oss snd_mixer_oss snd_seq_midi snd_seq_midi_event snd_seq
snd_usb_usx2y snd_usb_lib snd_hwdep snd_cs46xx snd_rawmidi snd_seq_device
snd_intel8x0 snd_ac97_codec snd_pcm snd_timer snd soundcore snd_page_alloc
realtime commoncap w83781d i2c_sensor i2c_isa i2c_i801 i2c_core sk98lin
subfs dm_mod usbcore
CPU:    1
EIP:    0060:[<c01462e9>]    Not tainted VLI
EFLAGS: 00010286   (2.6.10-rc2-mm3-RT-V0.7.32-2.0smp)
EIP is at kmem_cache_create+0x4df/0x69c
eax: 00000017   ebx: f7cd9710   ecx: c05c5d00   edx: f77b8000
esi: c030973a   edi: c030973a   ebp: f77b9d10   esp: f77b9cd8
ds: 007b   es: 007b   ss: 0068   preempt: 00000001
Process khubd (pid: 1477, threadinfo=f77b8000 task=f777c2b0)
Stack: c02f2054 c02f63aa 000005a7 00000565 f77b9d00 f7cd10c0 c0000000
ffffff80
       00000001 f7cd1080 00000080 c03418a0 f7000400 f7000428 f77b9d3c
c026e662
       c030972b 00000180 00000080 00002000 00000000 00000000 f7000490
00000000
Call Trace:
 [<c01035f4>] show_stack+0xb4/0xbc (28)
 [<c010378a>] show_registers+0x169/0x1de (56)
 [<c010399c>] die+0x10b/0x191 (68)
 [<c0103e19>] do_invalid_op+0xba/0xc4 (204)
 [<c0103237>] error_code+0x2b/0x30 (116)
 [<c026e662>] scsi_setup_command_freelist+0x94/0x115 (44)
 [<c026f8ca>] scsi_host_alloc+0x26f/0x399 (124)
 [<f8cf173f>] usb_stor_acquire_resources+0x6c/0x10a [usb_storage] (28)
 [<f8cf1c64>] storage_probe+0x1b8/0x259 [usb_storage] (40)
 [<f89ca07a>] usb_probe_interface+0x62/0x7b [usbcore] (28)
 [<c023d723>] driver_probe_device+0x2f/0x70 (24)
 [<c023d7ae>] device_attach+0x4a/0xa5 (40)
 [<c023da89>] bus_add_device+0x4d/0x89 (28)
 [<c023c9a6>] device_add+0xb1/0x137 (40)
 [<f89d1ba6>] usb_set_configuration+0x3ca/0x444 [usbcore] (84)
 [<f89cc822>] usb_new_device+0xa5/0x164 [usbcore] (44)
 [<f89cd6b0>] hub_port_connect_change+0x221/0x426 [usbcore] (56)
 [<f89cda7b>] hub_events+0x1c6/0x39d [usbcore] (56)
 [<f89cdc89>] hub_thread+0x37/0x109 [usbcore] (96)
 [<c01009b1>] kernel_thread_helper+0x5/0xb (142893076)
Code: e8 57 5e fd ff b8 08 4f be c0 e8 c0 d1 fe ff c7 44 24 08 a7 05 00 00
c7 44 24 04 aa 63 2f c0 c7 04 24 54 20 2f c0 e8 31 5e fd ff <0f> 0b a7 05
aa 63 2f c0 eb b9 a1 44 1a 33 c0 c7 44 24 04 d0 00


After this, any USB related operation is doomed, if not the entire system.

Bye.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Ingo Molnar]

* Rui Nuno Capela <rncbc@rncbc.org> wrote:

> Ingo Molnar wrote:
> >
> > i have released the -V0.7.32-0 Real-Time Preemption patch, which can be
> > downloaded from the usual place:
> >
> 
> I have a bug to report, it shows on both of my machines (SMP and UP)
> now running RT-V0.7.32-2. It seems to be present also on previous RT
> releases, and don't even know if it's upstream.
> 
> When one usb-storage flash stick is first time unplugged:

hm, doesnt seem to be directly related to -RT. Could you try the vanilla
-rc2-mm3 kernel, does it trigger there too?

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Rui Nuno Capela]

Ingo Molnar wrote:
>
> Rui Nuno Capela wrote:
>
>> Ingo Molnar wrote:
>> >
>> > i have released the -V0.7.32-0 Real-Time Preemption patch, which can
>> > be downloaded from the usual place:
>> >
>>
>> I have a bug to report, it shows on both of my machines (SMP and UP)
>> now running RT-V0.7.32-2. It seems to be present also on previous RT
>> releases, and don't even know if it's upstream.
>>
>> When one usb-storage flash stick is first time unplugged:
>
> hm, doesnt seem to be directly related to -RT. Could you try the vanilla
> -rc2-mm3 kernel, does it trigger there too?
>

No, it doesn't. Just tested on 2.6.10-rc2-mm3 (vanilla) and nothing wrong
happens. I can plug and unplug the USB flashram stick, and again, and
again, and everything stays fine.

On 2.6.10-rc2-mm3-RT-V0.7.32-2 I get trouble after the first time I unplug
the thingie. From then on I rather not touch it back again ;)

Take care.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[K.R. Foley]

Ingo Molnar wrote:
> * Rui Nuno Capela <rncbc@rncbc.org> wrote:
> 
> 
>>Ingo Molnar wrote:
>>
>>>i have released the -V0.7.32-0 Real-Time Preemption patch, which can be
>>>downloaded from the usual place:
>>>
>>
>>I have a bug to report, it shows on both of my machines (SMP and UP)
>>now running RT-V0.7.32-2. It seems to be present also on previous RT
>>releases, and don't even know if it's upstream.
>>
>>When one usb-storage flash stick is first time unplugged:
> 
> 
> hm, doesnt seem to be directly related to -RT. Could you try the vanilla
> -rc2-mm3 kernel, does it trigger there too?
> 
> 	Ingo
> 

Gene Heskett reported a very similar problem yesterday with the subject: 
  Re:2.6.10-rc2-mm3-V0.7.31-19

kr

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Gene Heskett]

On Saturday 04 December 2004 18:55, K.R. Foley wrote:
>Ingo Molnar wrote:
>> * Rui Nuno Capela <rncbc@rncbc.org> wrote:
>>>Ingo Molnar wrote:
>>>>i have released the -V0.7.32-0 Real-Time Preemption patch, which
>>>> can be downloaded from the usual place:
>>>
>>>I have a bug to report, it shows on both of my machines (SMP and
>>> UP) now running RT-V0.7.32-2. It seems to be present also on
>>> previous RT releases, and don't even know if it's upstream.
>>>
>>>When one usb-storage flash stick is first time unplugged:
>>
>> hm, doesnt seem to be directly related to -RT. Could you try the
>> vanilla -rc2-mm3 kernel, does it trigger there too?
>>
>>  Ingo
>
>Gene Heskett reported a very similar problem yesterday with the
> subject: Re:2.6.10-rc2-mm3-V0.7.31-19

Yes, and I can confirm that it does not do it right now, running
the UP version of 2.6.10-rc3.  I just tried it and this is all
I got in the log:
-----------------
Dec  4 22:06:38 coyote kernel: usb 3-2.2: new full speed USB device using ohci_hcd and address 7
Dec  4 22:06:38 coyote kernel: scsi0 : SCSI emulation for USB Mass Storage devices
Dec  4 22:06:44 coyote kernel:   Vendor: OLYMPUS   Model: C-3020ZOOM(U)     Rev: 1.00
Dec  4 22:06:44 coyote kernel:   Type:   Direct-Access                      ANSI SCSI revision: 02
Dec  4 22:06:44 coyote kernel: Attached scsi generic sg0 at scsi0, channel 0, id 0, lun 0,  type 0
Dec  4 22:06:44 coyote scsi.agent[26069]: disk at /devices/pci0000:00/0000:00:02.1/usb3/3-2/3-2.2/3-2.2:1.0/host0/target0:0:0/0:0:0:0
Dec  4 22:06:44 coyote kernel: SCSI device sda: 128000 512-byte hdwr sectors (66 MB)
Dec  4 22:06:44 coyote kernel: sda: assuming Write Enabled
Dec  4 22:06:44 coyote kernel: sda: assuming drive cache: write through
Dec  4 22:06:44 coyote kernel: SCSI device sda: 128000 512-byte hdwr sectors (66 MB)
Dec  4 22:06:44 coyote kernel: sda: assuming Write Enabled
Dec  4 22:06:44 coyote kernel: sda: assuming drive cache: write through
Dec  4 22:06:44 coyote kernel:  sda: sda1
Dec  4 22:06:44 coyote kernel: Attached scsi removable disk sda at scsi0, channel 0, id 0, lun 0
Dec  4 22:06:51 coyote kernel: usb 3-2.2: USB disconnect, address 7
-----------------
-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.30% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2004 by Maurice Eugene Heskett, all rights reserved.

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Gene Heskett]

On Saturday 04 December 2004 17:32, Rui Nuno Capela wrote:
>Ingo Molnar wrote:
>> i have released the -V0.7.32-0 Real-Time Preemption patch, which
>> can be downloaded from the usual place:

I can confirm essentially this same bug, this rather voluminous output 
took place after I turned off my camera and unplugged the usb cable 
from it.  At that point all other usb stuff was spotty, even the 
mouse till I rebooted.

>I have a bug to report, it shows on both of my machines (SMP and UP)
> now running RT-V0.7.32-2. It seems to be present also on previous
> RT releases, and don't even know if it's upstream.
>
>When one usb-storage flash stick is first time unplugged:
>
>usb 4-6: USB disconnect, address 4
>slab error in kmem_cache_destroy(): cache `scsi_cmd_cache': Can't
> free all objects
> [<c010361f>] dump_stack+0x23/0x25 (20)
> [<c01467d7>] kmem_cache_destroy+0x103/0x1aa (28)
> [<c026e77a>] scsi_destroy_command_freelist+0x97/0xa8 (28)
> [<c026f5b1>] scsi_host_dev_release+0x37/0xe1 (104)
> [<c023c6a9>] device_release+0x7a/0x7c (32)
> [<c01efd90>] kobject_cleanup+0x87/0x89 (28)
> [<c01f07eb>] kref_put+0x52/0xef (40)
> [<c01efdcc>] kobject_put+0x25/0x27 (16)
> [<f8cf1843>] usb_stor_release_resources+0x66/0xca [usb_storage]
> (16) [<f8cf1d93>] storage_disconnect+0x8e/0x9b [usb_storage] (16)
> [<f89ca117>] usb_unbind_interface+0x84/0x86 [usbcore] (28)
> [<c023d915>] device_release_driver+0x75/0x77 (28)
> [<c023db18>] bus_remove_device+0x53/0x82 (20)
> [<c023cae1>] device_del+0x4b/0x9b (20)
> [<f89d142a>] usb_disable_device+0xf5/0x10a [usbcore] (28)
> [<f89cc61c>] usb_disconnect+0xc8/0x164 [usbcore] (40)
> [<f89cd77e>] hub_port_connect_change+0x2ef/0x426 [usbcore] (56)
> [<f89cda7b>] hub_events+0x1c6/0x39d [usbcore] (56)
> [<f89cdc89>] hub_thread+0x37/0x109 [usbcore] (96)
> [<c01009b1>] kernel_thread_helper+0x5/0xb (142893076)
>
>
>Then, just when the same is plugged in again:
>
>usb 4-6: new high speed USB device using ehci_hcd and address 5
>kmem_cache_create: duplicate cache scsi_cmd_cache
>BUG at mm/slab.c:1447!
>------------[ cut here ]------------
>kernel BUG at mm/slab.c:1447!
>invalid operand: 0000 [#1]
>PREEMPT SMP
>Modules linked in: nls_utf8 nls_cp860 vfat fat nls_base usb_storage
>ohci1394 ieee1394 ehci_hcd usbhid uhci_hcd intel_mch_agp agpgart
> evdev wacom snd_pcm_oss snd_mixer_oss snd_seq_midi
> snd_seq_midi_event snd_seq snd_usb_usx2y snd_usb_lib snd_hwdep
> snd_cs46xx snd_rawmidi snd_seq_device snd_intel8x0 snd_ac97_codec
> snd_pcm snd_timer snd soundcore snd_page_alloc realtime commoncap
> w83781d i2c_sensor i2c_isa i2c_i801 i2c_core sk98lin subfs dm_mod
> usbcore
>CPU:    1
>EIP:    0060:[<c01462e9>]    Not tainted VLI
>EFLAGS: 00010286   (2.6.10-rc2-mm3-RT-V0.7.32-2.0smp)
>EIP is at kmem_cache_create+0x4df/0x69c
>eax: 00000017   ebx: f7cd9710   ecx: c05c5d00   edx: f77b8000
>esi: c030973a   edi: c030973a   ebp: f77b9d10   esp: f77b9cd8
>ds: 007b   es: 007b   ss: 0068   preempt: 00000001
>Process khubd (pid: 1477, threadinfo=f77b8000 task=f777c2b0)
>Stack: c02f2054 c02f63aa 000005a7 00000565 f77b9d00 f7cd10c0
> c0000000 ffffff80
>       00000001 f7cd1080 00000080 c03418a0 f7000400 f7000428
> f77b9d3c c026e662
>       c030972b 00000180 00000080 00002000 00000000 00000000
> f7000490 00000000
>Call Trace:
> [<c01035f4>] show_stack+0xb4/0xbc (28)
> [<c010378a>] show_registers+0x169/0x1de (56)
> [<c010399c>] die+0x10b/0x191 (68)
> [<c0103e19>] do_invalid_op+0xba/0xc4 (204)
> [<c0103237>] error_code+0x2b/0x30 (116)
> [<c026e662>] scsi_setup_command_freelist+0x94/0x115 (44)
> [<c026f8ca>] scsi_host_alloc+0x26f/0x399 (124)
> [<f8cf173f>] usb_stor_acquire_resources+0x6c/0x10a [usb_storage]
> (28) [<f8cf1c64>] storage_probe+0x1b8/0x259 [usb_storage] (40)
> [<f89ca07a>] usb_probe_interface+0x62/0x7b [usbcore] (28)
> [<c023d723>] driver_probe_device+0x2f/0x70 (24)
> [<c023d7ae>] device_attach+0x4a/0xa5 (40)
> [<c023da89>] bus_add_device+0x4d/0x89 (28)
> [<c023c9a6>] device_add+0xb1/0x137 (40)
> [<f89d1ba6>] usb_set_configuration+0x3ca/0x444 [usbcore] (84)
> [<f89cc822>] usb_new_device+0xa5/0x164 [usbcore] (44)
> [<f89cd6b0>] hub_port_connect_change+0x221/0x426 [usbcore] (56)
> [<f89cda7b>] hub_events+0x1c6/0x39d [usbcore] (56)
> [<f89cdc89>] hub_thread+0x37/0x109 [usbcore] (96)
> [<c01009b1>] kernel_thread_helper+0x5/0xb (142893076)
>Code: e8 57 5e fd ff b8 08 4f be c0 e8 c0 d1 fe ff c7 44 24 08 a7 05
> 00 00 c7 44 24 04 aa 63 2f c0 c7 04 24 54 20 2f c0 e8 31 5e fd ff
> <0f> 0b a7 05 aa 63 2f c0 eb b9 a1 44 1a 33 c0 c7 44 24 04 d0 00
>
>
>After this, any USB related operation is doomed, if not the entire
> system.
>
>Bye.

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.30% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2004 by Maurice Eugene Heskett, all rights reserved.

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Esben Nielsen]

On Fri, 3 Dec 2004, Ingo Molnar wrote:

> 
> [...] 
> on low RT load (the common case) 

In the system I deal with on my day job, 50% of the CPU load is from
RT tasks!

In fact, I think that is quite normal for industrial applications. Those
tend to be PLC-like: You run your scan every 10 ms. First you read your
input, then you do your calculations then you set your output. You do
the same thing every time. You can thus safely tick around at a CPU
load close to 100%!!

This is quite opposit to how telecom real-time systems works: Those are
usually event based. I.e. they process things when they occur. The load
can vary a lot depending on how much traffic has to go through the device.

Esben

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Ingo Molnar]

* Esben Nielsen <simlo@phys.au.dk> wrote:

> On Fri, 3 Dec 2004, Ingo Molnar wrote:
> 
> > 
> > [...] 
> > on low RT load (the common case) 
> 
> In the system I deal with on my day job, 50% of the CPU load is from
> RT tasks!

i'm not sure what point you are trying to make, but 'low RT load' in
this context means up to a load of 1.0. (i.e. one or less tasks running
on average)

also, this only applies to SMP systems.

thirdly, even if the new balancing code kicks in, the overhead is not
that bad, and it's for a purpose: we rather want an RT task to run on a
free CPU.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Esben Nielsen]

On Mon, 6 Dec 2004, Ingo Molnar wrote:

> 
> * Esben Nielsen <simlo@phys.au.dk> wrote:
> 
> > On Fri, 3 Dec 2004, Ingo Molnar wrote:
> > 
> > > 
> > > [...] 
> > > on low RT load (the common case) 
> > 
> > In the system I deal with on my day job, 50% of the CPU load is from
> > RT tasks!
> 
> i'm not sure what point you are trying to make, but 'low RT load' in
> this context means up to a load of 1.0. (i.e. one or less tasks running
> on average)
>

The point I want to make is that RT tasks is not always of the type
which reacts to some interrupt, does a small job and then goes to
sleep. Sometimes you also want larger jobs done realtime. Forinstance an
application waking up every 10ms, running for 5ms before going to sleep
again. That ofcourse gives you a RT-load <= 1.0.

In more complicated systems you can have a task running every 5ms,
another at every 25ms and another at every 100ms. That gives you a
RT-load of up to 3! And it is a even common case because you try to
syncronize the tasks, i.e. you wake all 3 tasks up at the same time but
then you know that on your typical UP system they will be executed by
priority, i.e. first the 5ms one, then the 25ms one and last the 100ms.

So my point was: Low RT-load might not be the common case on specific
systems. It is on the desktop where you want to run some multimedia
multimedia task(s) and have fast CPU compared to the job you
need done in RT but in embedded devices you do not pick such a big CPU and
you often use a large part on it in you RT control-loops.
 
> also, this only applies to SMP systems.
> 
> thirdly, even if the new balancing code kicks in, the overhead is not
> that bad, and it's for a purpose: we rather want an RT task to run on a
> free CPU.
> 

Good to hear. I was going to ask if the overhead wasn't too big :-)

However, I do have a feeling that it is rather expensive to move tasks
around. If I was going to design a real-time application on a 2 CPU SMP
system, I think I would CPU lock my RT-tasks to make the system more
deterministic. I.e. I would probably "port" the above mentioned
application by putting the 5 ms task on one CPU, the 25ms on the 
other and the 100 ms on the whatever CPU has the less CPU load. Then the
other non-real-time tasks in the system can wander around as they please.

But that is ofcourse not something you can do unless you know the specific
hardware system you will run your application on.  That is not the case
when you make multimedia software for the PC. I.e. in that case moving
RT-tasks around makes perfect sense!


> 	Ingo
> 

Esben

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Ingo Molnar]

* Esben Nielsen <simlo@phys.au.dk> wrote:

> So my point was: Low RT-load might not be the common case on specific
> systems. [...]

i did not suggest it was. The reason why i mentioned it was to point out
that _non-RT usage_ does not see any overhead, i.e. ordinary Linux
boxes. (which nevertheless do run RT tasks occasionally.)

of course in the RT-specific it can be common - Mark's test is one such
workload. If it werent widespread i'd not try to solve the problem...

	Ingo