Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Sid Boyce]

Same failure in previous RT V0.7-32 builds on x86_64, SuSE 9.2 gcc 
version 3.3.4 (pre 3.3.5 20040809)

   CC      arch/x86_64/kernel/../../i386/mach-default/topology.o
   LD      arch/x86_64/kernel/bootflag.o
   CC      arch/x86_64/kernel/e820.o
In file included from include/asm/dma.h:13,
                  from include/linux/bootmem.h:8,
                  from arch/x86_64/kernel/e820.c:10:
include/asm/io.h: In function `memset_io':
include/asm/io.h:265: warning: implicit declaration of function `memset'
include/asm/io.h:265: warning: return makes pointer from integer without 
a cast
   CC      arch/x86_64/kernel/reboot.o
   LD      arch/x86_64/kernel/quirks.o
   CC      arch/x86_64/kernel/semaphore.o
   CC      arch/x86_64/kernel/mce.o
   CC      arch/x86_64/kernel/../../i386/kernel/cpu/mtrr/main.o
   CC      arch/x86_64/kernel/../../i386/kernel/cpu/mtrr/if.o
   CC      arch/x86_64/kernel/../../i386/kernel/cpu/mtrr/generic.o
   CC      arch/x86_64/kernel/../../i386/kernel/cpu/mtrr/state.o
   CC      arch/x86_64/kernel/../../i386/kernel/cpu/mtrr/amd.o
   CC      arch/x86_64/kernel/../../i386/kernel/cpu/mtrr/cyrix.o
   CC      arch/x86_64/kernel/../../i386/kernel/cpu/mtrr/centaur.o
   LD      arch/x86_64/kernel/../../i386/kernel/cpu/mtrr/built-in.o
   CC      arch/x86_64/kernel/acpi/../../../i386/kernel/acpi/boot.o
In file included from arch/i386/kernel/acpi/boot.c:30:
include/linux/irq.h:80: error: parse error before "cycles_t"
include/linux/irq.h:80: warning: no semicolon at end of struct or union
include/linux/irq.h:82: error: parse error before '}' token
include/linux/irq.h:82: warning: type defaults to `int' in declaration 
of `irq_desc_t'
include/linux/irq.h:84: error: parse error before "irq_desc"
include/linux/irq.h:84: warning: type defaults to `int' in declaration 
of `irq_desc'
include/linux/irq.h:84: warning: data definition has no type or storage 
class
In file included from arch/i386/kernel/acpi/boot.c:30:
include/linux/irq.h:99: error: parse error before "irq_desc_t"
include/linux/irq.h:99: warning: function declaration isn't a prototype
include/linux/irq.h:100: error: parse error before "irq_desc_t"
include/linux/irq.h:100: warning: function declaration isn't a prototype
make[2]: *** [arch/x86_64/kernel/acpi/../../../i386/kernel/acpi/boot.o] 
Error 1
make[1]: *** [arch/x86_64/kernel/acpi] Error 2
make: *** [arch/x86_64/kernel] Error 2

Regards
Sid.
-- 
Sid Boyce .... Hamradio G3VBV and keen Flyer
=====LINUX ONLY USED HERE=====

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Mark_H_Johnson]

I was curious about what was causing some of my "extended delays" during
a network stress test & captured a trace I'll summarize here & forward
the complete trace separately. If I read this right, it appears I just
had "bad luck" (lots of interrupts on the CPU with the RT task) but I
would be glad to hear of a better interpretation (if it fits the data).

The test set up was my two CPU SMP system. The kernel was .33-03 with
PREEMPT_DESKTOP and non threaded IRQ's. The RT program was a slight
variation of latencytest - modified to capture traces if the CPU loop
(nominal 1160 usec) took more than 20% longer than nominal. The non
RT stress program running concurrently was my network output test
which basically uses rcp to copy a file to another system.

       times in usec
       CPU   Latency  Cause
[00]     50      14   Audio IRQ
[01]    154      69   Network IRQ / soft IRQ
[02]     53      92   SMP timer / Network IRQ / soft IRQ
[03]     25      62   Network IRQ / soft IRQ
[04]     60       6   Network IRQ
[05]    133      70   Network IRQ / soft IRQ
[06]     53     211   Network IRQ / soft IRQ / reschedule
[07]     35      76   Network IRQ / soft IRQ
[08]     75     210   Network IRQ / soft IRQ / reschedule / soft IRQ
[09]     58     105   Audio IRQ / soft IRQ / reschedule
[10]     19     103   Network IRQ / soft IRQ
[11]     86      86   Network IRQ / soft IRQ
[12]     34     102   Network IRQ / soft IRQ
[13]     53      63   Network IRQ / soft IRQ
[14]     56      52   Network IRQ / soft IRQ / reschedule
[15]     72      23   SMP timer / soft IRQ
[16]     59       2   End of trace
Total  1075    1346   (2421 grand total - check)

None of the individual latencies are all that bad - both #06 and #08
were the worst at 210 usec. It may be possible to fix the specific causes
of those two traces (though I know we've also broken the network code
as well...). but it is the cumulative effect of the number of
interrupts that appear to be the problem.

I am also not so sure that PREEMPT_RT would have done much better
but have not run that test for a comparison. I worry about the
cost of rescheduling / IRQ threading overhead - which may explain
why I see PREEMPT_DESKTOP doing better than PREEMPT_RT in some of
my tests.

I guess I should also note that this trace appears to indicate that
ALSA is buffering audio for output [see the two interrupts during the
trace]. I expect the audio output to be a blocking write in latencytest
(that was the 2.4 behavior). Is there some way to get the same behavior
I saw in 2.4 on a 2.6 / ALSA audio set up? If you don't know, who should
I direct that question to?

  --Mark

preemption latency trace v1.1.4 on 2.6.10-rc3-mm1-V0.7.33-03
--------------------------------------------------------------------
 latency: 2421 µs, #2259/2259, CPU#0 | (M:preempt VP:0, KP:1, SP:0 HP:0
#P:2)
    -----------------
    | task: latencytest-tra-14143 (uid:0 nice:0 policy:1 rt_prio:99)
    -----------------
 => started at: <00000000>
 => ended at:   restore_all+0x5/0x21 <c0104000>

                 _------=> CPU#
                / _-----=> irqs-off
               | / _----=> need-resched
               || / _---=> hardirq/softirq
               ||| / _--=> preempt-depth
               |||| /
               |||||     delay
   cmd     pid ||||| time  |   caller
      \   /    |||||   \   |   /
latencyt-14143 0...1    0µs : user_trace_start (sys_gettimeofday)
latencyt-14143 0d...    0µs+< (0)                    [00]
latencyt-14143 0d.h.   50µs : do_IRQ (8049af8 a 0)
... process audio interrupt ...
[I can argue - why did this come in here since the audio REALLY should
have finished during the write() call done later]
latencyt-14143 0d...   64µs!< (0)                    [01]
latencyt-14143 0d.h.  218µs : do_IRQ (8049af9 b 0)
... process network IRQ / soft IRQ ...
latencyt-14143 0d...  287µs+< (0)                    [02]
latencyt-14143 0d...  340µs : smp_apic_timer_interrupt (8049af9 0 0)
... timer / network IRQ / soft IRQ
latencyt-14143 0d...  432µs+< (0)                    [03]
latencyt-14143 0d.h.  457µs : do_IRQ (8049af8 b 0)
... network IRQ / soft IRQ ...
latencyt-14143 0d...  519µs+< (0)                    [04]
latencyt-14143 0d.h.  579µs : do_IRQ (8049af8 b 0)
... network IRQ  ...
latencyt-14143 0d...  585µs!< (0)                    [05]
latencyt-14143 0d.h.  718µs : do_IRQ (8049af8 b 0)
... network IRQ / soft IRQ ...
latencyt-14143 0d...  788µs+< (0)                    [06]
latencyt-14143 0d.h.  841µs : do_IRQ (8049af9 b 0)
... network IRQ / soft IRQ / reschedule
latencyt-14143 0d... 1052µs+< (0)                    [07]
latencyt-14143 0d.h. 1087µs : do_IRQ (8049af8 b 0)
... network IRQ / soft IRQ
latencyt-14143 0d... 1163µs+< (0)                    [08]
latencyt-14143 0d.h. 1238µs : do_IRQ (8049af8 b 0)
... network IRQ / soft IRQ / reschedule / timer / soft IRQ
latencyt-14143 0d... 1448µs+< (0)                    [09]
latencyt-14143 0d.h. 1506µs : do_IRQ (8049af8 a 0)
... audio IRQ / soft IRQ / reschedule
[this on the other hand is closer to the right duration for the
audio interrupt]
latencyt-14143 0d... 1611µs+< (0)                    [10]
latencyt-14143 0d.h. 1630µs : do_IRQ (8049af8 b 0)
... network IRQ / soft IRQ
latencyt-14143 0d... 1733µs+< (0)                    [11]
latencyt-14143 0d.h. 1819µs : do_IRQ (8049af9 b 0)
... network IRQ / soft IRQ
latencyt-14143 0d... 1905µs+< (0)                    [12]
latencyt-14143 0d.h. 1939µs : do_IRQ (8049af8 b 0)
... network IRQ / soft IRQ
latencyt-14143 0d... 2041µs+< (0)                    [13]
latencyt-14143 0d.h. 2094µs : do_IRQ (8049af8 b 0)
... network IRQ / soft IRQ
latencyt-14143 0d... 2157µs+< (0)                    [14]
latencyt-14143 0d.h. 2213µs : do_IRQ (8049af8 b 0)
... network IRQ / soft IRQ / reschedule
latencyt-14143 0d... 2265µs+< (0)                    [15]
latencyt-14143 0d... 2337µs : smp_apic_timer_interrupt (8049af8 0 0)
... SMP timer / soft IRQ
latencyt-14143 0d... 2360µs+< (0)                    [16]
latencyt-14143 0.... 2419µs > sys_gettimeofday (00000000 00000000 0000007b)
latencyt-14143 0.... 2419µs : sys_gettimeofday (sysenter_past_esp)
latencyt-14143 0.... 2420µs : user_trace_stop (sys_gettimeofday)
latencyt-14143 0...1 2421µs : user_trace_stop (sys_gettimeofday)

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Mark_H_Johnson]

A comparison of PREEMPT_RT (with tracing) to PREEMPT_DESKTOP
(with tracing). Due to the tracing overheads (and the results
I measured in the last few days), I'll report on percentages
within 200 usec for RT and 100 usec on PK to make it "more fair".

Comparison of .33-00RT and .33-00PK results
00RT has PREEMPT_RT
00PK has PREEMPT_DESKTOP and no threaded IRQ's
2.4 has lowlat + preempt patches applied

  within 200/100 usec
       CPU loop (%)   Elapsed Time (sec)    2.4
Test   RT     PK        RT      PK   |   CPU  Elapsed
X     96.78  99.88      90 *    83+* |  97.20   70
top   93.87 100.00      36 *    31+  |  97.48   29
neto  99.17 100.00     340 *   193+  |  96.23   36
neti  99.13  98.39     340 *   280 * |  95.86   41
diskw 98.04 100.00?    350 *   310+  |  77.64   29
diskc 95.56  99.94     350 *   310+  |  84.12   77
diskr 90.77  99.94     220 *   310+  |  90.66   86
total                 1726    1517   |         368
 [higher is better]  [lower is better]
* wide variation in audio duration
+ long stretch of audio duration "too fast"
? I believe I had non RT starvation & this result is in error
[chart shows a typical results for about 20 seconds and then
gets "really smooth" like the top chart for the remainder of
the measured duration]

The percentages are all within a handful of a percent so these
results look pretty comparable.

Looking at ping response time:
  RT 0.226 / 0.486 / 2.475 / 0.083 msec
  PK 0.102 / 0.174 / 0.813 / 0.054 msec
for min / average / max / mdev values. Again, tracing penalizes
RT much more than PK so this is to be expected.

The maximum duration of the CPU loop (as measured by the
application) is in the range of 2.05 msec to 3.30 compared
to the nominal 1.16 msec duration for -00RT. The equivalent
numbers for -00PK are 1.21 to 2.61 msec. I would expect RT
to be better than PK on this measure, but it never seems to
be the result I measure.

Based on the number of latency_trace files, -00RT still
is far better than -00PK. In particular, I get some extended
delays in -00PK from:
 - network (I have an 2000 usec example!)
 - rcu_process_callbacks (around 250 usec)
 - clear_page_range (around 170 usec)
 - free_pages_and_swap_cache (around 140 usec)
 - do_no_page (around 170 usec)
 - ide [IRQ?] (around 200 usec)
 - journal_remove_journal_head (> 1000 usec)
 - do_wait / wait_task_zombie (around 200 usec)
A fix to the network & journaling latencies would be helpful.
The others are certainly less important. I'll send the traces
separately.

Also, if you get some odd trace results on an SMP system,
Ingo already has some fixes applied in response to some buglets
I found & reported separately.

--Mark H Johnson
  <mailto:Mark_H_Johnson@raytheon.com>

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Ingo Molnar]

* Mark_H_Johnson@Raytheon.com <Mark_H_Johnson@Raytheon.com> wrote:

> Based on the number of latency_trace files, -00RT still
> is far better than -00PK. In particular, I get some extended
> delays in -00PK from:
>  - network (I have an 2000 usec example!)
>  - rcu_process_callbacks (around 250 usec)
>  - clear_page_range (around 170 usec)
>  - free_pages_and_swap_cache (around 140 usec)
>  - do_no_page (around 170 usec)
>  - ide [IRQ?] (around 200 usec)
>  - journal_remove_journal_head (> 1000 usec)
>  - do_wait / wait_task_zombie (around 200 usec)
> A fix to the network & journaling latencies would be helpful.
> The others are certainly less important. I'll send the traces
> separately.

the network ones are hard to fix, because it's softirq overhead and that
with PK is fundamentally non-preemptable. A good number of attempts to
cut down on the latency paths there resulted in broken networking, so
i'd not try that again.

The fix is softirq threading. Perhaps you could try PK+THREAD_SOFTIRQS
[but keep hardirqs non-threaded] - this would have higher overhead than
PK but lower overhead than full threading of all IRQs. This wont fix the
jfs overhead but that one might be fixable, maybe someone from the FS
land can take a look at the journalling overhead, that is in process
context and should be thus easier to fix.

> Also, if you get some odd trace results on an SMP system, Ingo already
> has some fixes applied in response to some buglets I found & reported
> separately.

yeah - they are in the latest (-33-03) kernel.

	Ingo

[patch] Real-Time Preemption, -RT-2.6.10-rc2-mm1-V0.7.27-3

[Ingo Molnar]

i have released the -V0.7.27-3 Real-Time Preemption patch, which can be
downloaded from the usual place:

	http://redhat.com/~mingo/realtime-preempt/

this is another quick update to fix a couple of bugs. Sorry about the
fast pace of updates but these fixes are worth having ASAP:

Changes since a -V0.7.27-1:

 - fix module-put BKL count bug - this could explain/fix the lockups
   reported by Rui Nuno Capela.

 - fixed a netfilter related networking deadlock reported by Mark H. 
   Johnson two weeks ago, it triggered on my testbox today. This (rare)
   bug could potentially explain some of the other lockup reports that
   are still open.

 - fix load average constant +1.0 offset when PREEMPT_RT is enabled. 
   This was an artifact of the IRQ-threading of the timer interrupt.

to create a -V0.7.27-3 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm1/2.6.10-rc2-mm1.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm1-V0.7.27-3

	Ingo

[patch] Real-Time Preemption, -RT-2.6.10-rc2-mm1-V0.7.28-0

[Ingo Molnar]

i have released the -V0.7.28-0 Real-Time Preemption patch, which can be
downloaded from the usual place:

	http://redhat.com/~mingo/realtime-preempt/

this is a fixes & latency-reduction release.

Changes since a -V0.7.27-3:

 - made the UP-ioapic code a bit more conservative again - maybe some of
   the lockups are related?

 - removed the BKL from the sound code in a cleaner way and
   removed the quite fragile 'negative ->lock_depth' code. Much less
   intrusive than i originally thought, and much cleaner as well.

 - more fixes to the wakeup-timing logic, 4 false positives fixed in
   total, mostly related to new-task-wakeup not accurately starting the
   tracer.

 - fixed the mmx-memcpy related latency reported by Florian Schmidt and 
   others. Also turned off the MMX/SSE ops in the RAID code, which 
   can introduce similar latencies.

 - kgdb fix from Bill Huey

 - knfsd shutdown with-BKL-held fix

 - highmem compilation fix

 - profiling related crash fix

 - implemented 'direct-path' rescheduling to further reduce scheduling
   latency: the kernel will now in most cases go from try_to_wakeup()
   into the scheduler directly without re-enabling interrupts ever again
   (and thus not giving irq handlers a window to increase latency). This
   is also the final fix for irq nesting and irq-stack recursion.

 - turn off sync wakeups on PREEMPT_RT -> they are latency generators

to create a -V0.7.28-0 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm1/2.6.10-rc2-mm1.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm1-V0.7.28-0

	Ingo

[patch] Real-Time Preemption, -RT-2.6.10-rc2-mm1-V0.7.28-1

[Ingo Molnar]

i have released the -V0.7.28-1 Real-Time Preemption patch, which can be
downloaded from the usual place:

	http://redhat.com/~mingo/realtime-preempt/

this should fix the lockup bug reported by Florian Schmidt.

there's a generic PREEMPT bug in the upstream kernel: there exists a
single-instruction race window in __flush_tlb(), if the kernel preempted
exactly there in a lazy-TLB thread and certain other, rare scheduling
and MM properties were true as well (a certain constellation of threads
and lazy-TLB kernel threads occured), and the lazy-TLB task then got
another user TLB to inherit, and switched to a task from which it
inherited that new TLB, thus the wrong cr3 was loaded and inherited by
this next, non-lazy-TLB next task; then (and only then) this scenario
would typically manifest itself in the form of an infinite pagefault
lockup occuring much after the fact, upon the next userspace access (to
the joy of a totally baffled kernel developer). I suspect from the
description you can guess how much fun it was to debug it =B-)

the bug is even more rare in the generic kernel, because there most (but
not all) TLB flush points are in a critical section.

this fix could resolve some of the other 'my box just locked up'
reports.

Changes since a -V0.7.28-0:

 - reverted the UP-ioapic change - it was unrelated to the lockup and it
   is known to cause problems on certain IDE/soundcard combinations.

 - fixed and improved the trace_print_on_crash tracing feature - it was
   highly needed to find the TLB bug ...

to create a -V0.7.28-1 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm1/2.6.10-rc2-mm1.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm1-V0.7.28-1

	Ingo

[patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.29-0

[Ingo Molnar]

i have released the -V0.7.29-0 Real-Time Preemption patch, which can be
downloaded from the usual place:

	http://redhat.com/~mingo/realtime-preempt/

this is a pure merge of -V0.7.28-2 to 2.6.10-rc2-mm2. -rc2-mm2 itself is
a fixes-only release.

to create a -V0.7.29-0 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm2/2.6.10-rc2-mm2.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm2-V0.7.29-0

	Ingo

[patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

i have released the -V0.7.30-2 Real-Time Preemption patch, which can be
downloaded from the usual place:

	http://redhat.com/~mingo/realtime-preempt/

the biggest change in this release are fixes for priority-inheritance
bugs uncovered by Esben Nielsen pi_test suite. These bugs could explain
some of the jackd-under-load latencies reported.

Changes since -V0.7.29-0:

 - priority inheritance handling fixes:

    - sort the RT wakees at wakeup time, not at block-time: an RT task
      might have gotten boosted while it slept.

    - fix priority-restoration bug at mutex-release time

    - use task_rt() not p->policy to determine whether a task needs 
      PI handling - a SCHED_OTHER task might be boosted to RT prio.

    - fix mutex_setprio() bug: queue now-RT tasks to the active array, 
      otherwise expired SCHED_OTHER tasks will not be properly boosted.

 - went back to the mask-and-delay method of handling hardirqs on 
   UP-IOAPIC as well. Due to APIC prioritization hardirqs can get
   delayed by another, unacked hardirq, so the quick method needs more 
   work before it can be used.

 - added Thomas Gleixner's semaphore -> completion changes for 
   drv->unload_sem. This fixes the module unload crashes reported by 
   Rui Nuno Capela and Shane Shrybman.

 - dvb mutex updates for RT, this fixes the bug reported by Christian 
   Meder.

 - e100 fix from K.R. Foley - this should fix the boot-time e100
   enable_irq warning.

 - NFS lockd mutex RT fixes from Thomas Gleixner - this could fix some
   of the bugs reported by Bill Huey.

 - PREEMPT_VOLUNTARY fixes - this could fix the boot-time hang reported 
   by Lee Revell.

 - wake up irq thread upon creation - this solves the 'irq thread only 
   changes priority after first interrupt arrives' anomaly reported.

to create a -V0.7.30-2 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm2/2.6.10-rc2-mm2.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm2-V0.7.30-2

	Ingo

[patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-9

[Ingo Molnar]

i have released the -V0.7.30-9 Real-Time Preemption patch, which can be
downloaded from the usual place:

    http://redhat.com/~mingo/realtime-preempt/

this is a fixes-only release.

most importantly it includes a JACK related latency fix. With Florian
Schmidt's great detective work we honed in on a big latency source
within JACK: the use of named pipes (fifos) on journalled filesystems. 

This issue has been empirically identified before (and is mentioned in
the JACK howto) but has never been given high enough prominence. It
turns out that the atime updates done while read()ing or write()ing
named pipes causes the delays - it may under certain circumstances call
out into the journalling code. It may block even on non-journalled
filesystems.

To work this issue around, when PREEMPT_RT is enabled the -30-9 kernel
skips atime updates on named-fifos. (it's pretty pointless anyway.)
Alternative userspace workarounds are to put the fifos on tmpfs/ramfs,
or to mark the filesystem noatime,nodiratime.

those experiencing xruns under JACK should definitely try the -30-9
kernel.

Changes since -V0.7.30-2:

 - named fifo reads/writes are now atomic, whenever possible

 - fixed pi_lock related SMP & CRITICAL_IRQSOFF_TIMING lockups, this 
   could resolve the lockups reported by Mark H. Johnson.

 - fixed one more PI buglet: wake up the new owner _after_ restoring
   the priority of the old owner.

 - made the NMI oopser more robust - it should print out some message 
   in pretty much any locking scenario.

 - added the blocker device used by Esben Nielsen's pi_test suite.

 - added user-triggerable ALSA xrun tracing to the patch: if a 
   sound IO channel has xrun_debug enabled in /proc then 
   user_trace_stop() will be called before printing the xrun message,
   and the current trace will be saved to /proc/latency_trace. This is a
   'one-shot' tracing method for now. I can be activated via:

     echo 1 > /proc/asound/card0/pcm0p/xrun_debug

     echo 1 > /proc/sys/kernel/trace_user_triggered
     echo 1 > /proc/sys/kernel/trace_freerunning
     echo 0 > /proc/sys/kernel/preempt_max_latency
     echo 0 > /proc/sys/kernel/preempt_thresh
     echo 0 > /proc/sys/kernel/preempt_wakeup_timing

     ./gettimeofday 0 1

  gettimeofday.c is attached below. The JACK fifo xrun source was found
  via this tracing facility.

to create a -V0.7.30-9 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm2/2.6.10-rc2-mm2.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm2-V0.7.30-9

	Ingo


-- gettimeofday.c:

#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include <sys/wait.h>
#include <linux/unistd.h>

int main (int argc, char **argv)
{
	if (argc != 3) {
		printf("usage: gettimeofday <val1> <val2>\n");
		exit(0);
	}
	gettimeofday(atol(argv[1]), atol(argv[2]));

	return 0;
}

[patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-10

[Ingo Molnar]

i have released the -V0.7.30-10 Real-Time Preemption patch, which can be
downloaded from the usual place:

    http://redhat.com/~mingo/realtime-preempt/

this is a fixes-only release.

the most important fixes are the ones to the priority inheritance logic
(affecting the latency of RT tasks), discovered and reported by Esben
Nielsen. I also found two more PI bugs running the new pi_test2 code
from Esben.

Changes since -V0.7.30-9:

 - PI fixes:

   - the waiter->prio field caused wrong priority settings upon unlock, 
     resulting in PI bugs in the nested-locking case.

   - use rt_task() when determining PI tasks, not p->policy.

   - in the blocking-on-blocked-task nesting case both promote now-RT
     tasks to the pi_waiters list and queue them to the head of the wait
     list, and demote now-non-RT tasks from the pi_waiters list and 
     queue them to the tail of the wait list.

 - PI-debugging blocker device update from Esben Nielsen

 - module build fix: export user_trace_stop symbol, this fixes the error 
   reported by Florian Schmidt

 - tracer fix: in the default !freerunning tracing mode, if the trace
   buffer overflows (this is relatively rare, but can happen) then the
   tracer overwrote kernel memory that leads to lockups/kernel crashes. 
   Maybe this bug was also the source of the truncated trace bug
   reported by Mark H. Johnson?

 - reduce tracing overhead within schedule() when !tracing_enabled.

to create a -V0.7.30-10 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm2/2.6.10-rc2-mm2.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm2-V0.7.30-10

	Ingo

[patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Ingo Molnar]

i have released the -V0.7.32-0 Real-Time Preemption patch, which can be
downloaded from the usual place:

	http://redhat.com/~mingo/realtime-preempt/

this is a fixes-mostly release with one new feature:

implemented global RT-task balancing on SMP systems, which improves the
latency of RT tasks on SMP systems. The basic problem was that the 2.6
kernel has per-CPU runqueues. In the current design there is no
guarantee that if an RT task starves another, lower-prio (or same-prio)
RT task in a given local runqueue, that the starved task will ever be
migrated to another CPU: it has to wait for the higher-prio task to
finish. In short, task migration on SMP is fundamentally non-RT and
priority-insensitive. In particular the workloads and latencies reported
by Mark H. Johnson reflect such SMP scheduling artifacts.

the new global RT-task balancing feature solves this problem by tracking
the 'RT overload' situation (when there is more than one RT tasks on a
CPU) and makes other CPUs 'pull' RT tasks (and only RT tasks)
immediately when such a situation occurs.

To give an example, in the following scheduling scenario:

  CPU#0					CPU#1

  task-A SCHED_FIFO prio 30		task-C SCHED_FIFO prio 30 [curr]
  task-B SCHED_FIFO prio 40 [curr]

task-B is the currently executing task on CPU#0, task-C is the currently
executing task on CPU#1. Now on the vanilla 2.6 kernel, if task-C
blocks, there's no guarantee that task-A will be run there - if there's
a SCHED_NORMAL task on CPU#1's runqueue then it will run indefinitely. 
With global RT-balancing task-A will be scheduled on CPU#1 immediately
after task-C leaves it.

furthermore, if in the same scenario, if e.g. a RT-prio 35 task-D is
woken up on CPU#0, the vanilla 2.6 scheduler will not move it to CPU#1,
even though it could preempt the prio 30 task-C there. With global
RT-balancing this will happen and task-C will be preempted immediately
and task-D runs.

on low RT load (the common case) the scheduler behaves like the stock
scheduler - the new logic only kicks in if a CPU runqueue has 2 or more
RT tasks running at once.

anyway, while the feature is stable on my SMP testboxes, this is still a
nontrivial ~200-lines delta in the scheduler so there might be problems. 
UP should not be affected by this.

other changes since -V0.7.32-20:

 - local-APIC shutdown fix: this should solve some of the 'reboot hangs' 
   reports.

 - more tracing fixes - might fix the 'truncated traces' problems.

 - reduce the NMI watchdog frequency from 10 KHz to 1000 Hz.

 - dont report futex reschedules as atomicity violations

to create a -V0.7.32-0 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm2/2.6.10-rc2-mm2.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm2-V0.7.32-0

	Ingo

[patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-4

[Ingo Molnar]

i have released the -V0.7.32-4 Real-Time Preemption patch, which can be
downloaded from the usual place:

    http://redhat.com/~mingo/realtime-preempt/

this is a fixes-only release.

Changes since -V0.7.32-2:

 - fixed a seqlock related xtime_lock lockup scenario - this could 
   explain the SMP lockups reported by Mark H. Johnson.
 
 - fixed a small buglet in the new SMP RT-balancing code, which could 
   lead to bad balancing in certain rare cases.

to create a -V0.7.32-4 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm3/2.6.10-rc2-mm3.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm3-V0.7.32-4

	Ingo

[patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-6

[Ingo Molnar]

i have released the -V0.7.32-6 Real-Time Preemption patch, which can be
downloaded from the usual place:

   http://redhat.com/~mingo/realtime-preempt/

this too is a fixes-only release.

Changes since -V0.7.32-4:

 - fixed a lock_kernel()-re-enables-interrupts bug reported by Daniel 
   Walker. The fix is to allow down() from irqs-off sections (and 
   save/restore irq flags) as long as there's no real contention on the
   semaphore.

 - fixed a /proc/latency_trace formatting bug reported by Mark H. 
   Johnson.

to create a -V0.7.32-6 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm3/2.6.10-rc2-mm3.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm3-V0.7.32-6

	Ingo

[patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Ingo Molnar]

i have released the -V0.7.33-0 Real-Time Preemption patch, which can be
downloaded from the usual place:

  http://redhat.com/~mingo/realtime-preempt/

this is mainly a port from -rc2-mm3 to -rc3-mm1. Changes:

- due to 2.6.10 release work the -mm kernel now is in fixes-mostly mode,
  but there's one interesting new feature: -rc3-mm1 introduced the
  ->unlocked_ioctl method which is now an official way to do BKL-less
  ioctls. I changed the ALSA ->ioctl_bkl changes in -RT to use this
  facility. The ALSA/sound guys might be interested in these bits. Thus
  another chunk of -RT could go upstream.

- IO-APIC/MSI fix from Steven Rostedt.

- fixed a tracer bug which would produce a kernel warning and an empty
  /proc/latency_trace if the trace buffer overflows.

to create a -V0.7.33-0 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc3.bz2
  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc3/2.6.10-rc3-mm1/2.6.10-rc3-mm1.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc3-mm1-V0.7.33-0

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Steven Rostedt]

[RFC]

Ingo,

Any thought about adding a one shot timer for the system? 

I know there's been talk about this in the mainline (actually going on
right now in the dynamic Hz thread), but I figured that this would
especially be good for a RT system and not be worried about the HZ
settings.  

An example would be to have the timer interrupt be implemented as a one
shot timer, and be able to register actions to times with it.  The jiffy
calculations could just be an event that is registered to go off once
every HZ.  The timer interrupt would then just have to look at all the
events that have expired (using something like the TSC to determine what
expires), execute their registered actions, and if need be, the action
could add itself back on the event queue (the jiffy timer would do
this), and then the timer would set itself to go off at the next event.
I believe something like this was done by the utime patch (now with the
KURT project).

This way the RT processes would not need to depend on the HZ settings.

I'd be willing to implement this, since I'm doing it regardless for a
client of mine. But I would like to know your input on this.

Thanks,

-- Steve

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Lee Revell]

On Tue, 2004-12-14 at 14:34 -0500, Steven Rostedt wrote:
> [RFC]
> 
> Ingo,
> 
> Any thought about adding a one shot timer for the system? 
> 

Isn't this what George Anzinger is working on?

http://sourceforge.net/projects/high-res-timers/

Lee

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Steven Rostedt]

On Tue, 2004-12-14 at 15:08 -0500, Lee Revell wrote:
> On Tue, 2004-12-14 at 14:34 -0500, Steven Rostedt wrote:
> > [RFC]
> > 
> > Ingo,
> > 
> > Any thought about adding a one shot timer for the system? 
> > 
> 
> Isn't this what George Anzinger is working on?
> 
> http://sourceforge.net/projects/high-res-timers/
> 
> Lee
> 

A quick look at this looks like this is what I was looking for. I'd need
to review the code in a more detailed aspect but first glance, Yes, this
is what I want.

Now, since High Res-timers and RT seem to go together, what are the
plans for merging these?  If this is indeed what I need, then I'll be
doing it to myself, but if these can merge in the public domain, then I
would say, all the better.

-- Steve

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> > > [RFC]
> > > 
> > > Ingo,
> > > 
> > > Any thought about adding a one shot timer for the system? 
> > > 
> > 
> > Isn't this what George Anzinger is working on?
> > 
> > http://sourceforge.net/projects/high-res-timers/
> > 
> > Lee
> > 
> 
> A quick look at this looks like this is what I was looking for. I'd
> need to review the code in a more detailed aspect but first glance,
> Yes, this is what I want.
> 
> Now, since High Res-timers and RT seem to go together, what are the
> plans for merging these?  If this is indeed what I need, then I'll be
> doing it to myself, [...]

i've been thinking about it on and off. If you would/could try it that
would certainly help. RT for Linux is a dance of many small steps.

the two projects are obviously complementary and i have no intention to
reinvent the wheel in any way. Best would be to bring hires timers up to
upstream-mergable state (independently of the -RT patch) and ask Andrew
to include it in -mm, then i'd port -RT to it automatically.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Lee Revell]

On Tue, 2004-12-14 at 22:18 +0100, Ingo Molnar wrote:
> i've been thinking about it on and off. If you would/could try it that
> would certainly help. RT for Linux is a dance of many small steps.
> 
> the two projects are obviously complementary and i have no intention to
> reinvent the wheel in any way. Best would be to bring hires timers up to
> upstream-mergable state (independently of the -RT patch) and ask Andrew
> to include it in -mm, then i'd port -RT to it automatically.

OK, I'll give this a shot.  It would certainly help on my underpowered
EPIA system, where my tests show 2.1% residency for the timer ISR with
HZ=1000.  On a system like this I would expect the difference to be
perceptible with a regular desktop workload.

Lee

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Ingo Molnar]

* Lee Revell <rlrevell@joe-job.com> wrote:

> On Tue, 2004-12-14 at 22:18 +0100, Ingo Molnar wrote:
> > i've been thinking about it on and off. If you would/could try it that
> > would certainly help. RT for Linux is a dance of many small steps.
> > 
> > the two projects are obviously complementary and i have no intention to
> > reinvent the wheel in any way. Best would be to bring hires timers up to
> > upstream-mergable state (independently of the -RT patch) and ask Andrew
> > to include it in -mm, then i'd port -RT to it automatically.
> 
> OK, I'll give this a shot.  It would certainly help on my underpowered
> EPIA system, where my tests show 2.1% residency for the timer ISR with
> HZ=1000.  On a system like this I would expect the difference to be
> perceptible with a regular desktop workload.

a warning: it's probably quite complex merging work - while the two
projects work on different conceptual issues, they touch the same code.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Lee Revell]

On Tue, 2004-12-14 at 22:51 +0100, Ingo Molnar wrote:
> * Lee Revell <rlrevell@joe-job.com> wrote:
> 
> > On Tue, 2004-12-14 at 22:18 +0100, Ingo Molnar wrote:
> > > i've been thinking about it on and off. If you would/could try it that
> > > would certainly help. RT for Linux is a dance of many small steps.
> > > 
> > > the two projects are obviously complementary and i have no intention to
> > > reinvent the wheel in any way. Best would be to bring hires timers up to
> > > upstream-mergable state (independently of the -RT patch) and ask Andrew
> > > to include it in -mm, then i'd port -RT to it automatically.
> > 
> > OK, I'll give this a shot.  It would certainly help on my underpowered
> > EPIA system, where my tests show 2.1% residency for the timer ISR with
> > HZ=1000.  On a system like this I would expect the difference to be
> > perceptible with a regular desktop workload.
> 
> a warning: it's probably quite complex merging work - while the two
> projects work on different conceptual issues, they touch the same code.
> 

Yeah, I expected this, I'll wait for George to update it.  I am testing
with 2.6.9 vanilla for the time being.  

Lee

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[George Anzinger]

Ingo Molnar wrote:
> * Steven Rostedt <rostedt@goodmis.org> wrote:
> 
> 
>>>>[RFC]
>>>>
>>>>Ingo,
>>>>
>>>>Any thought about adding a one shot timer for the system? 
>>>>
>>>
>>>Isn't this what George Anzinger is working on?
>>>
>>>http://sourceforge.net/projects/high-res-timers/
>>>
>>>Lee
>>>
>>
>>A quick look at this looks like this is what I was looking for. I'd
>>need to review the code in a more detailed aspect but first glance,
>>Yes, this is what I want.
>>
>>Now, since High Res-timers and RT seem to go together, what are the
>>plans for merging these?  If this is indeed what I need, then I'll be
>>doing it to myself, [...]
> 
> 
> i've been thinking about it on and off. If you would/could try it that
> would certainly help. RT for Linux is a dance of many small steps.
> 
> the two projects are obviously complementary and i have no intention to
> reinvent the wheel in any way. Best would be to bring hires timers up to
> upstream-mergable state (independently of the -RT patch) and ask Andrew
> to include it in -mm, then i'd port -RT to it automatically.

Well, I guess I am just backward :)  I plan to port it to the current RT today 
or tomorrow (if all goes well).  I will then work on the changes needed to get 
it into -mm.  Guess I will be supporting two versions for a bit...


-- 
George Anzinger   george@mvista.com
High-res-timers:  http://sourceforge.net/projects/high-res-timers/

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Steven Rostedt]

On Tue, 2004-12-14 at 13:52 -0800, George Anzinger wrote:
> Ingo Molnar wrote:

...

> >>Now, since High Res-timers and RT seem to go together, what are the
> >>plans for merging these?  If this is indeed what I need, then I'll be
> >>doing it to myself, [...]
> > 
> > 
> > i've been thinking about it on and off. If you would/could try it that
> > would certainly help. RT for Linux is a dance of many small steps.
> > 
> > the two projects are obviously complementary and i have no intention to
> > reinvent the wheel in any way. Best would be to bring hires timers up to
> > upstream-mergable state (independently of the -RT patch) and ask Andrew
> > to include it in -mm, then i'd port -RT to it automatically.
> 
> Well, I guess I am just backward :)  I plan to port it to the current RT today 
> or tomorrow (if all goes well).  I will then work on the changes needed to get 
> it into -mm.  Guess I will be supporting two versions for a bit...
> 

I need what you've been working on, so I'm available to help get the two
together. Since you two (Ingo and George) are the creators of these,
I'll let you fight it out at how to go about this. If it ends up being
two patches for you George, then I would be glad to help maintain the
RT/HigRes one.

-- Steve

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Ingo Molnar]

* George Anzinger <george@mvista.com> wrote:

> >the two projects are obviously complementary and i have no intention to
> >reinvent the wheel in any way. Best would be to bring hires timers up to
> >upstream-mergable state (independently of the -RT patch) and ask Andrew
> >to include it in -mm, then i'd port -RT to it automatically.
> 
> Well, I guess I am just backward :) I plan to port it to the current
> RT today or tomorrow (if all goes well).  I will then work on the
> changes needed to get it into -mm.  Guess I will be supporting two
> versions for a bit...

very good - i can carry it along in -RT, and your VST bits are certainly
an immediate bonus feature for the non-hard-RT (=laptop, desktop, audio)
folks too.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Lee Revell]

On Tue, 2004-12-14 at 22:18 +0100, Ingo Molnar wrote:
> the two projects are obviously complementary and i have no intention to
> reinvent the wheel in any way. Best would be to bring hires timers up to
> upstream-mergable state (independently of the -RT patch) and ask Andrew
> to include it in -mm, then i'd port -RT to it automatically.

Among other things I think Paul Davis mentioned that George's high res
timer patch would make it possible for JACK to send MIDI clock.  This
would be a huge improvement.

Lee

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Ingo Molnar]

* Lee Revell <rlrevell@joe-job.com> wrote:

> On Tue, 2004-12-14 at 22:18 +0100, Ingo Molnar wrote:
> > the two projects are obviously complementary and i have no intention to
> > reinvent the wheel in any way. Best would be to bring hires timers up to
> > upstream-mergable state (independently of the -RT patch) and ask Andrew
> > to include it in -mm, then i'd port -RT to it automatically.
> 
> Among other things I think Paul Davis mentioned that George's high res
> timer patch would make it possible for JACK to send MIDI clock.  This
> would be a huge improvement.

<clueless question> roughly what latency/accuracy requirements does the
MIDI clock have, and why is it an advantage if Linux generates it? What
generates it otherwise - external MIDI hardware? Or was the problem
mainly not latency/accuracy but that Linux couldnt generate a
finegrained enough clock?

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Lee Revell]

On Tue, 2004-12-14 at 23:31 +0100, Ingo Molnar wrote:
> * Lee Revell <rlrevell@joe-job.com> wrote:
> 
> > On Tue, 2004-12-14 at 22:18 +0100, Ingo Molnar wrote:
> > > the two projects are obviously complementary and i have no intention to
> > > reinvent the wheel in any way. Best would be to bring hires timers up to
> > > upstream-mergable state (independently of the -RT patch) and ask Andrew
> > > to include it in -mm, then i'd port -RT to it automatically.
> > 
> > Among other things I think Paul Davis mentioned that George's high res
> > timer patch would make it possible for JACK to send MIDI clock.  This
> > would be a huge improvement.
> 
> <clueless question> roughly what latency/accuracy requirements does the
> MIDI clock have, and why is it an advantage if Linux generates it? What
> generates it otherwise - external MIDI hardware? Or was the problem
> mainly not latency/accuracy but that Linux couldnt generate a
> finegrained enough clock?

Being able to send or receive MIDI clock is a common feature for
hardware and software samplers, sequencers, etc.  It just allows more
flexibility in your setup.  I think currently Linux can receive MIDI
clock better than it can send.

To answer the last question I think the clock was not fine grained
enough.  But as far as timing requirements I don't actually know the
details.  Paul?

Lee

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Paul Davis]

>> > On Tue, 2004-12-14 at 22:18 +0100, Ingo Molnar wrote:
>> > > the two projects are obviously complementary and i have no intention to
>> > > reinvent the wheel in any way. Best would be to bring hires timers up to
>> > > upstream-mergable state (independently of the -RT patch) and ask Andrew
>> > > to include it in -mm, then i'd port -RT to it automatically.
>> > 
>> > Among other things I think Paul Davis mentioned that George's high res
>> > timer patch would make it possible for JACK to send MIDI clock.  This
>> > would be a huge improvement.
>> 
>> <clueless question> roughly what latency/accuracy requirements does the
>> MIDI clock have, and why is it an advantage if Linux generates it? What
>> generates it otherwise - external MIDI hardware? Or was the problem
>> mainly not latency/accuracy but that Linux couldnt generate a
>> finegrained enough clock?

the latter. to send MIDI Clock or MIDI Timecode requires an interrupt
source that is not locked to jiffie-ish intervals or power-of-2
related intervals. For example, MTC requires sending 2 bytes roughly
every 0.8msec. Sending them every msec isn't good enough, in general.

my understanding of the HRT patch is that it allows the timer to be
reprogrammed to elapse with nanosecond resolution. i don't understand
why linus has been so reluctant to move linux in this direction, other
than it being hard to fit into the existing fixed interval timer
framework.

--p

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Ingo Molnar]

* Paul Davis <paul@linuxaudiosystems.com> wrote:

> the latter. to send MIDI Clock or MIDI Timecode requires an interrupt
> source that is not locked to jiffie-ish intervals or power-of-2
> related intervals. For example, MTC requires sending 2 bytes roughly
> every 0.8msec. Sending them every msec isn't good enough, in general.

yeah, i can understand that - 20% slower music is bad, even to kernel
hackers ;-)

> my understanding of the HRT patch is that it allows the timer to be
> reprogrammed to elapse with nanosecond resolution. i don't understand
> why linus has been so reluctant to move linux in this direction, other
> than it being hard to fit into the existing fixed interval timer
> framework.

i dont think there's any particular type of feeling from Linus towards
HRT - it's always the quality of the patch (and the underlying
fundamental issues) that controls, plus 'demand'. So integrating this
stuff is not a matter of will, it's a matter of having good enough code.

the current 'fixed interval timer framework' is one reason that makes
Linux scale so well on big networked boxes, which can easily have
millions of timers running (per CPU). Sub-jiffy solutions i've seen so
far tended to concentrate on making sub-jiffies work, while keeping
fixed interval timers only as a side-effect. That's not how usage
demands it - right now fixed interval timers are king in terms of usage
(and will be king even if we had HRT integrated), so subjiffy timers
must be very precisely engineered to not hurt fixed interval timers.
Plus if we touch the timer code then maybe it could be made more
deterministic (cascading isnt quite deterministic right now) - which
further complicates things. It's not a simple and clear-cut task for
sure.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Lee Revell]

On Wed, 2004-12-15 at 10:32 +0100, Ingo Molnar wrote:
> * Paul Davis <paul@linuxaudiosystems.com> wrote:
> 
> > the latter. to send MIDI Clock or MIDI Timecode requires an interrupt
> > source that is not locked to jiffie-ish intervals or power-of-2
> > related intervals. For example, MTC requires sending 2 bytes roughly
> > every 0.8msec. Sending them every msec isn't good enough, in general.
> 
> yeah, i can understand that - 20% slower music is bad, even to kernel
> hackers ;-)

It's a bit worse than that.  If there is excessive jitter in your MIDI
clock/MTC then your other devices will fail to lock on to it at all - it
does not degrade gracefully.

Lee

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[linux-os]

On Tue, 14 Dec 2004, Lee Revell wrote:

> On Tue, 2004-12-14 at 23:31 +0100, Ingo Molnar wrote:
>> * Lee Revell <rlrevell@joe-job.com> wrote:
>>
>>> On Tue, 2004-12-14 at 22:18 +0100, Ingo Molnar wrote:
>>>> the two projects are obviously complementary and i have no intention to
>>>> reinvent the wheel in any way. Best would be to bring hires timers up to
>>>> upstream-mergable state (independently of the -RT patch) and ask Andrew
>>>> to include it in -mm, then i'd port -RT to it automatically.
>>>
>>> Among other things I think Paul Davis mentioned that George's high res
>>> timer patch would make it possible for JACK to send MIDI clock.  This
>>> would be a huge improvement.
>>
>> <clueless question> roughly what latency/accuracy requirements does the
>> MIDI clock have, and why is it an advantage if Linux generates it? What
>> generates it otherwise - external MIDI hardware? Or was the problem
>> mainly not latency/accuracy but that Linux couldnt generate a
>> finegrained enough clock?
>
> Being able to send or receive MIDI clock is a common feature for
> hardware and software samplers, sequencers, etc.  It just allows more
> flexibility in your setup.  I think currently Linux can receive MIDI
> clock better than it can send.
>
> To answer the last question I think the clock was not fine grained
> enough.  But as far as timing requirements I don't actually know the
> details.  Paul?
>
> Lee
>

When I use Cakewalk Home-Studio to record Music from my MIDI piano,
I notice that the clock-resolution shown is several orders of
magnitude better than anything a PC can generate! I haven't got
a clue where this information comes from. It is in seconds, starting
at 1 (not zero, don't know why) and has resolution of microseconds
with no missing codes. This is on a M$ PC.

This generates a highly-accurate "time ruler". One can back-up
through this time and resolve samples that appear synchronous
but can be displaced in time with apparent resolution much
better than the 38,400 baud-rate of MIDI. I don't know how
they do it, but this is the MIDI "sample-clock". It has to
be virtual because there isn't any hardware on a PC that can
duplicate it.

It is likely that they use a software PLL with some periodic
updates from a timer-tick, but it sure looks impressive to
see "real-time" with such resolution on a PC.

I'm pretty sure that if Cakewalk decided to port Home Studio
to Linux, they would be able to do it with no technical hurdles.
Its just that, for Music, most use Apple and cheapskates like
me use PCs running M$.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.9 on an i686 machine (5537.79 BogoMips).
  Notice : All mail here is now cached for review by John Ashcroft.
                  98.36% of all statistics are fiction.

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Paul Davis]

>When I use Cakewalk Home-Studio to record Music from my MIDI piano,
>I notice that the clock-resolution shown is several orders of
>magnitude better than anything a PC can generate! I haven't got

incoming timing is a solved problem. the problem is scheduling
outgoing data, which requires an interrupt source of some kind.

>a clue where this information comes from. It is in seconds, starting

they almost certainly use either the sample clock of the audio
interface interpolated/estimated using a DLL (Delay Locked Loop).

>I'm pretty sure that if Cakewalk decided to port Home Studio
>to Linux, they would be able to do it with no technical hurdles.
>Its just that, for Music, most use Apple and cheapskates like
>me use PCs running M$.

Real cheapskates would use Linux with Rosegarden, Ardour et al :)

--p

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Gene Heskett]

On Tuesday 14 December 2004 18:18, linux-os wrote:
>On Tue, 14 Dec 2004, Lee Revell wrote:

>>> <clueless question> roughly what latency/accuracy requirements
>> Lee
>
>When I use Cakewalk Home-Studio to record Music from my MIDI piano,
>I notice that the clock-resolution shown is several orders of
>magnitude better than anything a PC can generate! I haven't got
>a clue where this information comes from. It is in seconds, starting
>at 1 (not zero, don't know why) and has resolution of microseconds
>with no missing codes. This is on a M$ PC.
>
>This generates a highly-accurate "time ruler". One can back-up
>through this time and resolve samples that appear synchronous
>but can be displaced in time with apparent resolution much
>better than the 38,400 baud-rate of MIDI. I don't know how
>they do it, but this is the MIDI "sample-clock". It has to
>be virtual because there isn't any hardware on a PC that can
>duplicate it.

Midi's baud rate isn't 38,400, its 31,250.  Check your
references.  It should be in any keyboards manual.

>It is likely that they use a software PLL with some periodic
>updates from a timer-tick, but it sure looks impressive to
>see "real-time" with such resolution on a PC.

Well, it is pretty slow for a non multiuser os equipt with todays
hardware.

>I'm pretty sure that if Cakewalk decided to port Home Studio
>to Linux, they would be able to do it with no technical hurdles.
>Its just that, for Music, most use Apple and cheapskates like
>me use PCs running M$.
>
>Cheers,
>Dick Johnson
>Penguin : Linux version 2.6.9 on an i686 machine (5537.79 BogoMips).
>  Notice : All mail here is now cached for review by John Ashcroft.

I don't want to bring politics into this group it has no place
here.  None, nada.   But, since you mentioned it, screw Ashcroft
and the camel that ride in on him.  He has done more to bring true
democracy to its knees in 4 years than all others before him in
our 230 years.

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.30% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2004 by Maurice Eugene Heskett, all rights reserved.

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Gene Heskett]

On Tuesday 14 December 2004 17:31, Ingo Molnar wrote:
>* Lee Revell <rlrevell@joe-job.com> wrote:
>> On Tue, 2004-12-14 at 22:18 +0100, Ingo Molnar wrote:
>> > the two projects are obviously complementary and i have no
>> > intention to reinvent the wheel in any way. Best would be to
>> > bring hires timers up to upstream-mergable state (independently
>> > of the -RT patch) and ask Andrew to include it in -mm, then i'd
>> > port -RT to it automatically.
>>
>> Among other things I think Paul Davis mentioned that George's high
>> res timer patch would make it possible for JACK to send MIDI
>> clock.  This would be a huge improvement.
>
><clueless question> roughly what latency/accuracy requirements does
> the MIDI clock have, and why is it an advantage if Linux generates
> it? What generates it otherwise - external MIDI hardware? Or was
> the problem mainly not latency/accuracy but that Linux couldnt
> generate a finegrained enough clock?
>
> Ingo

I'm not sure of the exact reasons, Ingo.  But the midi clock is a
bit of an odd man out in the normal progression of baud rates,
its 31,250 baud, in case you didn't already know that.
  
A lot of systems resort to a hardware timer driving a seriel shift
register as I doubt if one could guarantee writing to a single
bit port with an interval error of much more than 5% bit to bit,
and cumulatively less than that for the overall byte.  It can be
done though, we have a program for the coco's that run on a .89
mhz clock that can actually drive the seriel port well enough to
run a midi keyboard plugged into it. I've run it, and it has
enough spare time that I can steer some instruments to a second
homemade midi pack plugged into its expansion interface at the
same time & no descernable squiggles in the beat of the music.

Did you see my comment about the later versions seeming to slow
seti a wee bit?  Other than that, I'm in love with this, the
whole system just plain feels better.  The only thing on my wish
list right now is to be able to shut tvtime up, it grows the
system log about a megabyte a minute with its missed read reports.
Or is it tvtime that needs help?

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.30% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2004 by Maurice Eugene Heskett, all rights reserved.

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[K.R. Foley]

Gene Heskett wrote:
> On Tuesday 14 December 2004 17:31, Ingo Molnar wrote:
> 
>>* Lee Revell <rlrevell@joe-job.com> wrote:
>>
>>>On Tue, 2004-12-14 at 22:18 +0100, Ingo Molnar wrote:
>>>
>>>>the two projects are obviously complementary and i have no
>>>>intention to reinvent the wheel in any way. Best would be to
>>>>bring hires timers up to upstream-mergable state (independently
>>>>of the -RT patch) and ask Andrew to include it in -mm, then i'd
>>>>port -RT to it automatically.
>>>
>>>Among other things I think Paul Davis mentioned that George's high
>>>res timer patch would make it possible for JACK to send MIDI
>>>clock.  This would be a huge improvement.
>>
>><clueless question> roughly what latency/accuracy requirements does
>>the MIDI clock have, and why is it an advantage if Linux generates
>>it? What generates it otherwise - external MIDI hardware? Or was
>>the problem mainly not latency/accuracy but that Linux couldnt
>>generate a finegrained enough clock?
>>
>>Ingo
> 
> 
> I'm not sure of the exact reasons, Ingo.  But the midi clock is a
> bit of an odd man out in the normal progression of baud rates,
> its 31,250 baud, in case you didn't already know that.
>   
> A lot of systems resort to a hardware timer driving a seriel shift
> register as I doubt if one could guarantee writing to a single
> bit port with an interval error of much more than 5% bit to bit,
> and cumulatively less than that for the overall byte.  It can be
> done though, we have a program for the coco's that run on a .89
> mhz clock that can actually drive the seriel port well enough to
> run a midi keyboard plugged into it. I've run it, and it has
> enough spare time that I can steer some instruments to a second
> homemade midi pack plugged into its expansion interface at the
> same time & no descernable squiggles in the beat of the music.
> 
> Did you see my comment about the later versions seeming to slow
> seti a wee bit?  Other than that, I'm in love with this, the
> whole system just plain feels better.  The only thing on my wish
> list right now is to be able to shut tvtime up, it grows the
> system log about a megabyte a minute with its missed read reports.
> Or is it tvtime that needs help?
> 

Are you referring to the "Read missed before next interrupt" messages? 
If so, you can disable this by disabling the rtc histogram under:
Device Drivers --> Character devices --> Real Time Clock Histogram Support.

kr

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Gene Heskett]

On Wednesday 15 December 2004 10:24, K.R. Foley wrote:
>Gene Heskett wrote:
>> Did you see my comment about the later versions seeming to slow
>> seti a wee bit?  Other than that, I'm in love with this, the
>> whole system just plain feels better.  The only thing on my wish
>> list right now is to be able to shut tvtime up, it grows the
>> system log about a megabyte a minute with its missed read reports.
>> Or is it tvtime that needs help?
>
>Are you referring to the "Read missed before next interrupt"
> messages? If so, you can disable this by disabling the rtc
> histogram under: Device Drivers --> Character devices --> Real Time
> Clock Histogram Support.

Ok, thats building now.  Silly Q:  Can that be turned back on with a
setting in /proc or /sys?

>kr

Thanks.

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.30% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2004 by Maurice Eugene Heskett, all rights reserved.

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[K.R. Foley]

Gene Heskett wrote:
> On Wednesday 15 December 2004 10:24, K.R. Foley wrote:
> 
>>Gene Heskett wrote:
>>
>>>Did you see my comment about the later versions seeming to slow
>>>seti a wee bit?  Other than that, I'm in love with this, the
>>>whole system just plain feels better.  The only thing on my wish
>>>list right now is to be able to shut tvtime up, it grows the
>>>system log about a megabyte a minute with its missed read reports.
>>>Or is it tvtime that needs help?
>>
>>Are you referring to the "Read missed before next interrupt"
>>messages? If so, you can disable this by disabling the rtc
>>histogram under: Device Drivers --> Character devices --> Real Time
>>Clock Histogram Support.
> 
> 
> Ok, thats building now.  Silly Q:  Can that be turned back on with a
> setting in /proc or /sys?
> 

Not currently.

kr

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Fernando Lopez-Lezcano]

On Tue, 2004-12-14 at 05:28, Ingo Molnar wrote:
> i have released the -V0.7.33-0 Real-Time Preemption patch, which can be
> downloaded from the usual place:
> 
>   http://redhat.com/~mingo/realtime-preempt/
> 
> this is mainly a port from -rc2-mm3 to -rc3-mm1. Changes:
> 
> - due to 2.6.10 release work the -mm kernel now is in fixes-mostly mode,
>   but there's one interesting new feature: -rc3-mm1 introduced the
>   ->unlocked_ioctl method which is now an official way to do BKL-less
>   ioctls. I changed the ALSA ->ioctl_bkl changes in -RT to use this
>   facility. The ALSA/sound guys might be interested in these bits. Thus
>   another chunk of -RT could go upstream.

If I build a cvs version of alsa without those patches, will it work on
top of the 0.7.33 kernel? Or do I have to try to isolate the patches and
apply them to current alsa cvs?

-- Fernando

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Ingo Molnar]

* Fernando Lopez-Lezcano <nando@ccrma.Stanford.EDU> wrote:

> If I build a cvs version of alsa without those patches, will it work
> on top of the 0.7.33 kernel? Or do I have to try to isolate the
> patches and apply them to current alsa cvs?

these are the modified files:

 linux/sound/core/oss/pcm_oss.c.orig
 linux/sound/core/oss/mixer_oss.c.orig
 linux/sound/core/pcm_lib.c.orig
 linux/sound/core/control.c.orig
 linux/sound/core/seq/oss/seq_oss.c.orig
 linux/sound/core/seq/seq_clientmgr.c.orig
 linux/sound/core/hwdep.c.orig
 linux/sound/core/pcm_native.c.orig
 linux/sound/core/timer.c.orig
 linux/sound/core/info.c.orig
 linux/sound/core/rawmidi.c.orig

cvs alsa should work just fine, but to get optimal results you might
want to try to apply the above changes (all linux/sound/ changes) to
alsa-cvs.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Fernando Lopez-Lezcano]

On Tue, 2004-12-14 at 05:28, Ingo Molnar wrote:
> i have released the -V0.7.33-0 Real-Time Preemption patch, which can be
> downloaded from the usual place:
> 
>   http://redhat.com/~mingo/realtime-preempt/
> 
> this is mainly a port from -rc2-mm3 to -rc3-mm1. Changes:
> 
> - due to 2.6.10 release work the -mm kernel now is in fixes-mostly mode,
>   but there's one interesting new feature: -rc3-mm1 introduced the
>   ->unlocked_ioctl method which is now an official way to do BKL-less
>   ioctls. I changed the ALSA ->ioctl_bkl changes in -RT to use this
>   facility. The ALSA/sound guys might be interested in these bits. Thus
>   another chunk of -RT could go upstream.
> 
> - IO-APIC/MSI fix from Steven Rostedt.
> 
> - fixed a tracer bug which would produce a kernel warning and an empty
>   /proc/latency_trace if the trace buffer overflows.

I don't know which change did it, but I have network connectivity in my
athlon64 test box with 0.7.33-0! Woohoo! [*]
Thanks...
-- Fernando

[*] as reported before: going up to runlevel 5 killed the network, up to
leve 3 everything was fine, I was guessing probably due to some problem
with X and the radeon driver. 

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Florian Schmidt]

On 14 Dec 2004 15:21:56 -0800
Fernando Lopez-Lezcano <nando@ccrma.Stanford.EDU> wrote:

> I don't know which change did it, but I have network connectivity in my
> athlon64 test box with 0.7.33-0! Woohoo! [*]
> Thanks...
> -- Fernando
> 

I thought i'd just chime in. 33-0 runs excellent here (i have no
debugging or timing enabled, but xrun performance is really good (i have
seen none but app caused ones (only mild load though, will push it a bit
harder though tomorrow))..

Flo

-- 
Palimm Palimm!
http://affenbande.org/~tapas/

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Lee Revell]

On Tue, 2004-12-14 at 15:21 -0800, Fernando Lopez-Lezcano wrote:
> I don't know which change did it, but I have network connectivity in my
> athlon64 test box with 0.7.33-0! Woohoo! [*]

Wait, this works on x84-64 now?  There was a recent report on LAU that
it didn't compile.

Lee

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Fernando Lopez-Lezcano]

On Tue, 2004-12-14 at 17:09, Lee Revell wrote:
> On Tue, 2004-12-14 at 15:21 -0800, Fernando Lopez-Lezcano wrote:
> > I don't know which change did it, but I have network connectivity in my
> > athlon64 test box with 0.7.33-0! Woohoo! [*]
> 
> Wait, this works on x84-64 now?  There was a recent report on LAU that
> it didn't compile.

The machine has an athlon64 but it is running 32 bit fc2. I have not
tried to build (yet) on 64 bit fcx.

-- Fernando

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Ingo Molnar]

* Fernando Lopez-Lezcano <nando@ccrma.Stanford.EDU> wrote:

> On Tue, 2004-12-14 at 17:09, Lee Revell wrote:
> > On Tue, 2004-12-14 at 15:21 -0800, Fernando Lopez-Lezcano wrote:
> > > I don't know which change did it, but I have network connectivity in my
> > > athlon64 test box with 0.7.33-0! Woohoo! [*]
> > 
> > Wait, this works on x84-64 now?  There was a recent report on LAU that
> > it didn't compile.
> 
> The machine has an athlon64 but it is running 32 bit fc2. I have not
> tried to build (yet) on 64 bit fcx.

x64 wont work for now, it needs some work to make threaded timer IRQs
work.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Andrew Walrond]

On Wednesday 15 Dec 2004 09:09, Ingo Molnar wrote:
>
> x64 wont work for now, it needs some work to make threaded timer IRQs
> work.
>

Don't leave it too long; I'm pretty sure I'm not the only x64 user wanting to 
test your -RT stuff.

I'm beginning to develop 32bit-envy ;)

Andrew Walrond

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Lee Revell]

On Wed, 2004-12-15 at 10:09 +0100, Ingo Molnar wrote:
> x64 wont work for now, it needs some work to make threaded timer IRQs
> work.

But the timer IRQ can only be threaded with PREEMPT_RT, right?  Seems
like PREEMPT_DESKTOP should work.  Older VP patches worked on x64 (S7,
T3) IIRC.

Lee

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Fernando Lopez-Lezcano]

On Tue, 2004-12-14 at 15:21, Fernando Lopez-Lezcano wrote:
> On Tue, 2004-12-14 at 05:28, Ingo Molnar wrote:
> > i have released the -V0.7.33-0 Real-Time Preemption patch, which can be
> > downloaded from the usual place:
> > 
> >   http://redhat.com/~mingo/realtime-preempt/
> > 
> > this is mainly a port from -rc2-mm3 to -rc3-mm1. Changes:
> > 
> > - due to 2.6.10 release work the -mm kernel now is in fixes-mostly mode,
> >   but there's one interesting new feature: -rc3-mm1 introduced the
> >   ->unlocked_ioctl method which is now an official way to do BKL-less
> >   ioctls. I changed the ALSA ->ioctl_bkl changes in -RT to use this
> >   facility. The ALSA/sound guys might be interested in these bits. Thus
> >   another chunk of -RT could go upstream.
> > 
> > - IO-APIC/MSI fix from Steven Rostedt.
> > 
> > - fixed a tracer bug which would produce a kernel warning and an empty
> >   /proc/latency_trace if the trace buffer overflows.
> 
> I don't know which change did it, but I have network connectivity in my
> athlon64 test box with 0.7.33-0! Woohoo! [*]

But I'm still having problems with a dual athlon mp machine while trying
to boot 0.7.33-04 (same as before - this has not booted for quite a
while with the realtime patches). This is what I see:

pnp: 00:02: ioport range 0xe400-0xe4fe could not be reserved
BUG: scheduling while atomic: swapper/0x10000000/1
caller is preempt_schedule_irq+0x34/0x50
 [<c021053c3>] dump_stack+0x23/-x30 (20)
 [<c037438c>] __schedule+0xd5c/0xdb0 (124)
 [<c037432b>] preempt_schedul_irq+0x34/0x50 (12)
 [<c010432b>] need_resched+0x20/0x29 (-7416)
---------------------------
| preempt count: 10000001 ]
| 1-level deep critical section nesting:
---------------------------------------
.. [<c01472ed>] .... print_traces+0x1d/0x60
.....[<c01053c3>] .. ( <= dump_stack+0x23/0x30)

BUG at kernel/latency.c:1292!
--------------------[ cut here ]---------------------
kernel BUG at kernel/latency.c:1292!
invalid operand: 0000 [#1]
PREEMPT_SMP

=================
Booting with:
hardirq_preempt=0
softirq_preempt=0

ide1 at 0x170-0x177,0x376 on irq 15
BUG: scheduling while atomic: swapper/0x10000000/1
caller is preempt_schedule_irq+0x34/0x50
 [<>] dump_stack=0x23/0x30 (20)
 [<>] __schedule+0x5dc/0xdb0 (124)
 [<>] preempt_sched_irq+0x34/0x50 (12)
 [<>] need_resched+0x20/0x29 (-7376)
---------------------------
| preempt count: 10000001 ]
| 1-level deep critical section nesting:
---------------------------------------
.. [<c01472ed>] .... print_traces+0x1d/0x60
.....[<c01053c3>] .. ( <= dump_stack+0x23/0x30)

BUG at kernel/latency.c:1292!
--------------------[ cut here ]---------------------
kernel BUG at kernel/latency.c:1292!
invalid operand: 0000 [#1]
PREEMPT_SMP

=============
Booting with:
acpi=off

usbcore: registered new driver hub
BUG scheduling while atomic: swapper......
....
kernel BUG at kernel/latency.c:1292!
invalid operand: 0000 [#1]
PREEMPT_SMP

========
hardirq_preempt=0
softirq_preempt=0
acpi=off

usbcore: registered new driver hub
BUG scheduling while atomic: swapper......
....
kernel BUG at kernel/latency.c:1292!
invalid operand: 0000 [#1]
PREEMPT_SMP

=============
Booting with:
hardirq_preempt=0
softirq_preempt=0
kernel_preempt=0
acpi=off

Same........
Suggestions?
-- Fernando

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Magnus Määttä]

Hi Ingo,

On Tuesday 14 December 2004 14.28, Ingo Molnar wrote:
> i have released the -V0.7.33-0 Real-Time Preemption patch, which
> can be downloaded from the usual place:
>
>   http://redhat.com/~mingo/realtime-preempt/
>
> this is mainly a port from -rc2-mm3 to -rc3-mm1. Changes:
>
> - due to 2.6.10 release work the -mm kernel now is in fixes-mostly
> mode, but there's one interesting new feature: -rc3-mm1 introduced
> the ->unlocked_ioctl method which is now an official way to do
> BKL-less ioctls. I changed the ALSA ->ioctl_bkl changes in -RT to
> use this facility. The ALSA/sound guys might be interested in these
> bits. Thus another chunk of -RT could go upstream.
>
> - IO-APIC/MSI fix from Steven Rostedt.
>
> - fixed a tracer bug which would produce a kernel warning and an
> empty /proc/latency_trace if the trace buffer overflows.

I got this deadlock with V0.7.33-2:

==========================================
[ BUG: lock recursion deadlock detected! |
------------------------------------------
already locked:  [c04d63e0] {rtc_task_lock}
.. held by:             IRQ 8:  305 [df45c030,  52]
... acquired at:  rtc_interrupt+0x137/0x230

------------------------------
| showing all locks held by: |  (IRQ 8/305 [df45c030,  52]):
------------------------------

#001:             [c04d63e0] {rtc_task_lock}
... acquired at:  rtc_interrupt+0x137/0x230

#002:             [df4f16f8] {&timer->lock}
... acquired at:  _snd_timer_stop+0x8f/0x180

-{current task's backtrace}----------------->
 [<c0104273>] dump_stack+0x23/0x30 (20)
 [<c013d011>] check_deadlock+0x141/0x2d0 (48)
 [<c013dc1a>] task_blocks_on_lock+0x7a/0x100 (36)
 [<c041a847>] __down_mutex+0x237/0x440 (88)
 [<c013f86b>] __spin_lock+0x4b/0x60 (24)
 [<c013f8dd>] _spin_lock_irq+0x1d/0x20 (16)
 [<c02b60d9>] rtc_control+0x29/0x80 (32)
 [<c034efef>] rtctimer_stop+0x2f/0x70 (28)
 [<c034c8fc>] snd_timer_interrupt+0x2fc/0x380 (64)
 [<c034f057>] rtctimer_interrupt+0x27/0x40 (20)
 [<c02b4f08>] rtc_interrupt+0x158/0x230 (32)
 [<c014f40a>] handle_IRQ_event+0x6a/0xf0 (52)
 [<c014fdcb>] do_hardirq+0xab/0x130 (40)
 [<c014fec0>] do_irqd+0x70/0xb0 (32)
 [<c013bcc6>] kthread+0xa6/0xf0 (48)
 [<c0101365>] kernel_thread_helper+0x5/0x10 (548970516)
---------------------------
| preempt count: 00000002 ]
| 2-level deep critical section nesting:
----------------------------------------
.. [<c041a94a>] .... __down_mutex+0x33a/0x440
.....[<c013f86b>] ..   ( <= __spin_lock+0x4b/0x60)
.. [<c01435bb>] .... print_traces+0x1b/0x50
.....[<c0104273>] ..   ( <= dump_stack+0x23/0x30)


showing all tasks:
s            init:    1 [c1467970, 116] (not blocked)
s           IRQ 0:    2 [c1467320,  50] (not blocked)
s     ksoftirqd/0:    3 [c1466cd0, 105] (not blocked)
s       desched/0:    4 [c1466680, 105] (not blocked)
s        events/0:    5 [c1466030,  98] (not blocked)
s         khelper:    6 [df687970, 111] (not blocked)
s         kthread:   11 [df687320, 110] (not blocked)
s          kacpid:   20 [df686cd0, 110] (not blocked)
s           IRQ 9:   21 [df686680,  51] (not blocked)
s       kblockd/0:  117 [c15d4680, 110] (not blocked)
s           khubd:  125 [c15e9320, 115] (not blocked)
s         pdflush:  226 [c15e8680, 117] (not blocked)
s         pdflush:  227 [c15e9970, 116] (not blocked)
s         kswapd0:  228 [c15d4030, 117] (not blocked)
s           aio/0:  229 [df686030, 112] (not blocked)
s           IRQ 8:  305 [df45c030,  52] (not blocked)
s         kseriod:  309 [df45c680, 117] (not blocked)
s          IRQ 12:  315 [df45ccd0,  53] (not blocked)
s          IRQ 14:  405 [df45d320,  54] (not blocked)
s          IRQ 15:  407 [df46b970,  55] (not blocked)
s        khpsbpkt:  420 [df46b320, 115] blocked on: [c04ddfc4] {khpsbpkt_sig.lock}
.. held by:          khpsbpkt:  420 [df46b320, 115]
... acquired at:  dma_trm_tasklet+0x89/0x1a0
s          IRQ 10:  430 [c15d5970,  56] (not blocked)
s     knodemgrd_0:  431 [c15e8cd0, 116] blocked on: [df49bba8] {&hi->reset_sem}
.. held by:       knodemgrd_0:  431 [c15e8cd0, 116]
... acquired at:  highlevel_host_reset+0x4b/0x60
s          IRQ 11:  433 [df45d970,  57] (not blocked)
s           IRQ 1:  451 [c15e8030,  58] (not blocked)
s        krxtimod:  481 [c15fd970, 118] (not blocked)
s          krxiod:  482 [c15d4cd0, 117] (not blocked)
s         krxsecd:  483 [c15fd320, 117] (not blocked)
s       kafstimod:  487 [c15d5320, 120] (not blocked)
s      kafsasyncd:  488 [df46a680, 120] (not blocked)
s      reiserfs/0:  489 [df46acd0, 111] (not blocked)
s           udevd:  580 [de854030, 113] (not blocked)
s           IRQ 6: 4119 [dde00030,  59] (not blocked)
s       ipw2200/0: 4284 [dde01320, 110] (not blocked)
s       syslog-ng: 7503 [dcb30cd0, 115] (not blocked)
s             gpm: 7629 [dcb26cd0, 117] (not blocked)
s         portmap: 8374 [c15fc680, 116] (not blocked)
s        rpciod/0: 8386 [deefb970, 111] (not blocked)
s           lockd: 8387 [dde00cd0, 117] (not blocked)
s            sshd: 8439 [ddbbc680, 120] (not blocked)
s            cron: 8481 [ddbbc030, 116] (not blocked)
s           login: 8502 [c15fccd0, 117] (not blocked)
s          agetty: 8503 [ddb8e680, 116] (not blocked)
s          agetty: 8504 [ddbbd970, 116] (not blocked)
s          agetty: 8505 [deefa680, 116] (not blocked)
s          agetty: 8506 [dcb31320, 116] (not blocked)
s          agetty: 8507 [ddbbccd0, 116] (not blocked)
s            bash: 8588 [de855970, 117] (not blocked)
s             kdm: 8653 [deefb320, 116] (not blocked)
?               X: 8656 [deefacd0, 116] (not blocked)
s             kdm: 8657 [c15fc030, 117] (not blocked)
s        startkde: 8681 [ddbbd320, 118] (not blocked)
s         kdeinit: 8714 [dde01970, 116] (not blocked)
s         kdeinit: 8717 [de854680, 115] (not blocked)
s         kdeinit: 8719 [dcb26030, 116] (not blocked)
s         kdeinit: 8722 [df46a030, 116] (not blocked)
s         kdeinit: 8731 [de855320, 116] (not blocked)
s         kdeinit: 8735 [ddb8f970, 115] (not blocked)
s        kwrapper: 8736 [dde00680, 116] (not blocked)
s         kdeinit: 8738 [ddb8ecd0, 116] (not blocked)
s         kdeinit: 8739 [deefa030, 116] (not blocked)
s         kdeinit: 8741 [dcb30030, 115] (not blocked)
s         kdeinit: 8743 [dcb27320, 115] (not blocked)
s         kdeinit: 8745 [dcb30680, 116] (not blocked)
s         kdeinit: 8746 [dcb27970, 116] (not blocked)
s         kdeinit: 8748 [dcb26680, 115] (not blocked)
s         kdeinit: 8752 [ddb8f320, 115] (not blocked)
s          korgac: 8753 [dcb31970, 116] (not blocked)
s         kalarmd: 8755 [d7398cd0, 119] (not blocked)
s         kdeinit: 8756 [ddb8e030, 116] (not blocked)
s            bash: 8757 [de854cd0, 117] (not blocked)
s         firefox: 8765 [d7399320, 117] (not blocked)
s     firefox-bin: 8778 [d7399970, 115] (not blocked)
s     firefox-bin: 8782 [d5c53970, 116] (not blocked)
s     firefox-bin: 8784 [d7398680, 116] (not blocked)
X         netstat: 8790 [d5c52cd0, 118] (not blocked)
s             ssh: 8811 [d5c53320, 117] (not blocked)
s            bash: 8812 [d5c52680, 117] (not blocked)
s            bash: 8820 [d5c52030, 117] (not blocked)
s            bash: 8828 [d7398030, 117] (not blocked)
s            bash: 8836 [d3161970, 117] (not blocked)
s            bash: 8844 [d3160cd0, 116] (not blocked)
s            bash: 8852 [d3161320, 116] (not blocked)
s            bash: 8860 [d3160680, 116] (not blocked)
s            bash: 8868 [d3160030, 116] (not blocked)
s             ssh: 8876 [d328d320, 117] (not blocked)
s             ssh: 8877 [d328c030, 117] (not blocked)
s            bash: 8880 [d328c680, 119] (not blocked)
s           jackd: 8881 [d328ccd0, 120] (not blocked)
s           jackd: 8882 [d328d970, 116] (not blocked)
s           jackd: 8883 [d1f6f970,   0] (not blocked)
s           jackd: 8884 [d1f6f320,   9] (not blocked)
s          qsynth: 8887 [d1f6ecd0, 116] (not blocked)
s          qsynth: 8888 [c8095970,  10] (not blocked)
?          qsynth: 8889 [c8095320, 116] (not blocked)
s       ksnapshot: 8890 [d1f6e030, 116] (not blocked)
s         ladccad: 8908 [c8094cd0, 116] (not blocked)
s         ladccad: 8910 [c8094680,  10] (not blocked)
s         ladccad: 8911 [d1f6e680, 116] (not blocked)
s         ladccad: 8912 [c5769970, 116] (not blocked)
s         ladccad: 8919 [c5769320, 116] (not blocked)
s         ladccad: 8920 [c5768cd0, 123] (not blocked)
s         ladccad: 8909 [c8094030, 120] (not blocked)
D      rosegarden: 8923 [c5768030, 116] blocked on: [c3f12418] {&q->lock}
.. held by:                 X: 8656 [deefacd0, 116]
... acquired at:  __wake_up+0x51/0x60
R rosegardenseque: 8924 [c5768680, 116] (not blocked)
s rosegardenseque: 8925 [c30af970, 115] (not blocked)
s rosegardenseque: 8926 [c30af320,  10] (not blocked)

---------------------------
| showing all locks held: |
---------------------------

#001:             [c062af2c] {&hwif->gendev_rel_sem}
.. held by:              init:    1 [c1467970, 116]
... acquired at:  init_hwif_data+0x9d/0x180

#002:             [c062ab50] {&drive->gendev_rel_sem}
.. held by:              init:    1 [c1467970, 116]
... acquired at:  init_hwif_data+0x16f/0x180

#003:             [c062ad24] {&drive->gendev_rel_sem}
.. held by:              init:    1 [c1467970, 116]
... acquired at:  init_hwif_data+0x16f/0x180

#004:             [c062b574] {&hwif->gendev_rel_sem}
.. held by:              init:    1 [c1467970, 116]
... acquired at:  init_hwif_data+0x9d/0x180

#005:             [c062b198] {&drive->gendev_rel_sem}
.. held by:              init:    1 [c1467970, 116]
... acquired at:  init_hwif_data+0x16f/0x180

#006:             [c062b36c] {&drive->gendev_rel_sem}
.. held by:              init:    1 [c1467970, 116]
... acquired at:  init_hwif_data+0x16f/0x180

#007:             [c062bbbc] {&hwif->gendev_rel_sem}
.. held by:              init:    1 [c1467970, 116]
... acquired at:  init_hwif_data+0x9d/0x180

#008:             [c062b7e0] {&drive->gendev_rel_sem}
.. held by:              init:    1 [c1467970, 116]
... acquired at:  init_hwif_data+0x16f/0x180

#009:             [c062b9b4] {&drive->gendev_rel_sem}
.. held by:              init:    1 [c1467970, 116]
... acquired at:  init_hwif_data+0x16f/0x180

#010:             [c062c204] {&hwif->gendev_rel_sem}
.. held by:              init:    1 [c1467970, 116]
... acquired at:  init_hwif_data+0x9d/0x180

#011:             [c062be28] {&drive->gendev_rel_sem}
.. held by:              init:    1 [c1467970, 116]
... acquired at:  init_hwif_data+0x16f/0x180

#012:             [c062bffc] {&drive->gendev_rel_sem}
.. held by:              init:    1 [c1467970, 116]
... acquired at:  init_hwif_data+0x16f/0x180

#013:             [c062c84c] {&hwif->gendev_rel_sem}
.. held by:              init:    1 [c1467970, 116]
... acquired at:  init_hwif_data+0x9d/0x180

#014:             [c062c470] {&drive->gendev_rel_sem}
.. held by:              init:    1 [c1467970, 116]
... acquired at:  init_hwif_data+0x16f/0x180

#015:             [c062c644] {&drive->gendev_rel_sem}
.. held by:              init:    1 [c1467970, 116]
... acquired at:  init_hwif_data+0x16f/0x180

#016:             [c062ce94] {&hwif->gendev_rel_sem}
.. held by:              init:    1 [c1467970, 116]
... acquired at:  init_hwif_data+0x9d/0x180

#017:             [c062cab8] {&drive->gendev_rel_sem}
.. held by:              init:    1 [c1467970, 116]
... acquired at:  init_hwif_data+0x16f/0x180

#018:             [c062cc8c] {&drive->gendev_rel_sem}
.. held by:              init:    1 [c1467970, 116]
... acquired at:  init_hwif_data+0x16f/0x180

#019:             [c062d4dc] {&hwif->gendev_rel_sem}
.. held by:              init:    1 [c1467970, 116]
... acquired at:  init_hwif_data+0x9d/0x180

#020:             [c062d100] {&drive->gendev_rel_sem}
.. held by:              init:    1 [c1467970, 116]
... acquired at:  init_hwif_data+0x16f/0x180

#021:             [c062d2d4] {&drive->gendev_rel_sem}
.. held by:              init:    1 [c1467970, 116]
... acquired at:  init_hwif_data+0x16f/0x180

#022:             [c062db24] {&hwif->gendev_rel_sem}
.. held by:              init:    1 [c1467970, 116]
... acquired at:  init_hwif_data+0x9d/0x180

#023:             [c062d748] {&drive->gendev_rel_sem}
.. held by:              init:    1 [c1467970, 116]
... acquired at:  init_hwif_data+0x16f/0x180

#024:             [c062d91c] {&drive->gendev_rel_sem}
.. held by:              init:    1 [c1467970, 116]
... acquired at:  init_hwif_data+0x16f/0x180

#025:             [c062e16c] {&hwif->gendev_rel_sem}
.. held by:              init:    1 [c1467970, 116]
... acquired at:  init_hwif_data+0x9d/0x180

#026:             [c062dd90] {&drive->gendev_rel_sem}
.. held by:              init:    1 [c1467970, 116]
... acquired at:  init_hwif_data+0x16f/0x180

#027:             [c062df64] {&drive->gendev_rel_sem}
.. held by:              init:    1 [c1467970, 116]
... acquired at:  init_hwif_data+0x16f/0x180

#028:             [c062e7b4] {&hwif->gendev_rel_sem}
.. held by:              init:    1 [c1467970, 116]
... acquired at:  init_hwif_data+0x9d/0x180

#029:             [c062e3d8] {&drive->gendev_rel_sem}
.. held by:              init:    1 [c1467970, 116]
... acquired at:  init_hwif_data+0x16f/0x180

#030:             [c062e5ac] {&drive->gendev_rel_sem}
.. held by:              init:    1 [c1467970, 116]
... acquired at:  init_hwif_data+0x16f/0x180

#031:             [ded6fcf0] {&tty->atomic_read}
.. held by:            agetty: 8503 [ddb8e680, 116]
... acquired at:  read_chan+0x477/0x7c0

#032:             [dedb9cf0] {&tty->atomic_read}
.. held by:            agetty: 8505 [deefa680, 116]
... acquired at:  read_chan+0x477/0x7c0

#033:             [dd710cf0] {&tty->atomic_read}
.. held by:            agetty: 8506 [dcb31320, 116]
... acquired at:  read_chan+0x477/0x7c0

#034:             [dca2acf0] {&tty->atomic_read}
.. held by:            agetty: 8507 [ddbbccd0, 116]
... acquired at:  read_chan+0x477/0x7c0

#035:             [dd3bfcf0] {&tty->atomic_read}
.. held by:            agetty: 8504 [ddbbd970, 116]
... acquired at:  read_chan+0x477/0x7c0

#036:             [c1773cf0] {&tty->atomic_read}
.. held by:              bash: 8588 [de855970, 117]
... acquired at:  read_chan+0x477/0x7c0

#037:             [d35decf0] {&tty->atomic_read}
.. held by:              bash: 8868 [d3160030, 116]
... acquired at:  read_chan+0x477/0x7c0

#038:             [d30c4cf0] {&tty->atomic_read}
.. held by:              bash: 8860 [d3160680, 116]
... acquired at:  read_chan+0x477/0x7c0

#039:             [d306fcf0] {&tty->atomic_read}
.. held by:              bash: 8852 [d3161320, 116]
... acquired at:  read_chan+0x477/0x7c0

#040:             [d3ba3cf0] {&tty->atomic_read}
.. held by:              bash: 8844 [d3160cd0, 116]
... acquired at:  read_chan+0x477/0x7c0

#041:             [c3f12418] {&q->lock}
.. held by:                 X: 8656 [deefacd0, 116]
... acquired at:  __wake_up+0x51/0x60

#042:             [c3f135d8] {&sk->sk_callback_lock}
.. held by:        rosegarden: 8923 [c5768030, 116]
... acquired at:  sock_def_readable+0x25/0xb0

#043:             [d125631c] {&q->lock}
.. held by:            qsynth: 8889 [c8095320, 116]
... acquired at:  __wake_up+0x51/0x60

#044:             [c04d63e0] {rtc_task_lock}
.. held by:             IRQ 8:  305 [df45c030,  52]
... acquired at:  rtc_interrupt+0x137/0x230

#045:             [df4f16f8] {&timer->lock}
.. held by:             IRQ 8:  305 [df45c030,  52]
... acquired at:  _snd_timer_stop+0x8f/0x180
=============================================

[ turning off deadlock detection. Please report this trace. ]


Regards,
Magnus Määttä

-- 
Pushing 40 is exercise enough.

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[Bill Huey]

On Tue, Dec 14, 2004 at 02:28:34PM +0100, Ingo Molnar wrote:
> to create a -V0.7.33-0 tree from scratch, the patching order is:
> 
>   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2

>   http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc3.bz2
>   http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc3/2.6.10-rc3-mm1/2.6.10-rc3-mm1.bz2

>   http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc3-mm1-V0.7.33-0

Forward port of this patch to 2.6.10-mm1 here:
	http://people.lynuxworks.com/~bhuey/realtime/Ingo_forward_port-mm1-V0.7.33-04

Obviously, you'll need to patch a plain 2.6.10 with -mm1 from Andrew Morton,
but folks should be able to do this by now. ;)

You'll have to apply Ingo's patch so that it gets rejects and then apply
this patch on top of it so that it resolves those issues. It's a bit
sloppy, but this'll at least be somewhat workable until Ingo comes back
and pounds us with patches. :)

bill