Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Mark_H_Johnson]

>i have released the -V0.4 Real-Time Preemption patch, which can be
>downloaded from:

I built with this (and not V0.4.1) and had the following results:

[1] No problems with the build.

[2] Booting to single user without problems.

[3] telinit 3 - still have the atomic counter underflow BUG related
to qdisc_destroy. Otherwise, normal boot messages.

[4] telinit 5 - normal boot messages and display came up OK. Able
to login and start testing.

[5] Running my stress test, the first test (X server) appeared to
run OK. The second test (/proc or top) did not run properly. The
RT audio test appeared to take the whole system (both CPU's) and
the terminal window showing top did not appear until the audio
test finished (and was quickly taken down by the script). Could not
move the mouse at all during that test. The third test (network
output) ran a short period and then the machine locked up. Had
to use the hardware reset to recover.

The only message in the system log that was unusual was
nrpe[4151]: Error: Could not complete SSL handshake.
(no messages after this one)
everything before that was OK in the system log.

Looking at the application level charts from the first two
tests (after rebooting with -T3), the measured CPU time was
VERY SMOOTH, almost no blips until the end of the first test
and start of the second test(680 usec and 570 usec respectively).
The audio loop had some (6) spikes, 4 in the first test and 2 in
the second. The longest spike was over 60 msec in duration.

My script that samples latency traces > 200 usec had no
output. Not sure if that is because it didn't run or if there
were no traces to record.

I looked at the updated patch (V0.4.1) but I am not sure it
fixes this lock up problem. Please advise.

--Mark H Johnson
  <mailto:Mark_H_Johnson@raytheon.com>

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Ingo Molnar]

* Mark_H_Johnson@RAYTHEON.COM <Mark_H_Johnson@RAYTHEON.COM> wrote:

> [5] Running my stress test, the first test (X server) appeared to run
> OK. The second test (/proc or top) did not run properly. The RT audio
> test appeared to take the whole system (both CPU's) and the terminal
> window showing top did not appear until the audio test finished (and
> was quickly taken down by the script). Could not move the mouse at all
> during that test. The third test (network output) ran a short period
> and then the machine locked up. Had to use the hardware reset to
> recover.

the network one could perhaps be related to the network deadlocks
reported by others. Would be nice to turn on RWSEM_DETECT_DEADLOCKS and
to use a serial logging if possible.

does the audio test use alot of CPU time? In that case it would be
normal for the RT task to 'lock' the system up. In any case it would be
nice to try 0.4.2 because it has more check-preemption fixes affecting
both UP and SMP systems.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Remi Colinet]

Hi,

Here is a trivial compile fix :

CC      lib/string.o
  CC      lib/vsprintf.o
  AR      lib/lib.a
  CC      arch/i386/lib/bitops.o
  CC      arch/i386/lib/dec_and_lock.o
  CC      arch/i386/lib/delay.o
  CC      arch/i386/lib/memcpy.o
  CC      arch/i386/lib/usercopy.o
  AR      arch/i386/lib/lib.a
  GEN     .version
  CHK     include/linux/compile.h
  UPD     include/linux/compile.h
  CC      init/version.o
  LD      init/built-in.o
  LD      .tmp_vmlinux1
kernel/built-in.o(.text+0x8450): In function `cpu_callback':
kernel/fork.c:1421: undefined reference to `takeover_tasklets'
make: *** [.tmp_vmlinux1] Error 1


--- kernel/softirq.c.orig       2004-10-27 23:44:14.160948016 +0200
+++ kernel/softirq.c    2004-10-27 23:44:34.571845088 +0200
@@ -464,7 +464,7 @@
        BUG();
 }
 
-static void takeover_tasklets(unsigned int cpu)
+void takeover_tasklets(unsigned int cpu)
 {
        struct tasklet_struct **i;
 

Remi

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Mark_H_Johnson]

>the network one could perhaps be related to the network deadlocks
>reported by others. Would be nice to turn on RWSEM_DETECT_DEADLOCKS and
>to use a serial logging if possible.
Would be nice but I don't have serial logging available at this point.
I may be able to set it up in a couple of days though.

>does the audio test use alot of CPU time? In that case it would be
>normal for the RT task to 'lock' the system up. In any case it would be
>nice to try 0.4.2 because it has more check-preemption fixes affecting
>both UP and SMP systems.
I am aware of slow responses normally during tests. However the audio
test should only use one CPU out of two. The other CPU is busy as well
with a cpu burner (nice 10) but that should leave me CPU cycles
to move the mouse, swap windows, etc.  The "lock up" I saw this time
was a lot more severe (no mouse motion for several minutes at a time).
I knew the system was still running since the audio continued to play.

I'll try another build in the morning with whatever your latest is.

--Mark H Johnson
  <mailto:Mark_H_Johnson@raytheon.com>

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Ingo Molnar]

* Mark_H_Johnson@Raytheon.com <Mark_H_Johnson@Raytheon.com> wrote:

> I am aware of slow responses normally during tests. However the audio
> test should only use one CPU out of two. The other CPU is busy as well
> with a cpu burner (nice 10) but that should leave me CPU cycles to
> move the mouse, swap windows, etc.  The "lock up" I saw this time was
> a lot more severe (no mouse motion for several minutes at a time). I
> knew the system was still running since the audio continued to play.

the way i typically debug such scenarios is to set up a separate
'highprio console' of some sorts. E.g. log in via the network from
another box and make sure all processing of that console is SCHED_FIFO. 
(in the network case that would mean the network IRQ, ksoftirqd, sshd,
login and bash.) Such a 'highprio console' should have a higher priority
than any other task in the system. If you run alot of stuff in it then
it will surely disturb your measurements (and largely invalidate them),
but otherwise it can be useful to have it around just in case you
experience a lockup that you suspect to be some sort of livelock or
starvation. Whenever the lockup happens, check out what's going on, via
the highprio console.

another useful 'highprio console' is the text console itself - in this
case only login and bash has to be chrt-ed, and the runtime impact on
the test is smaller as well. This is only useful if you can start your
'bad' workload over ssh or via networked X.

	Ingo

[patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U5

[Ingo Molnar]

i have released the -U5 Real-Time Preemption patch:

  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc4-mm1-U5

this is a release intended to increase stability, but since it also
includes new debug features and related cleanups it might introduce new
regressions. Be careful.

there are two big changes:

 - debug feature: automatic semaphore/rwsem deadlock detection, based on
   the code from Igor Manyilov and Bill Huey.

this is a very nice feature that should help in debugging the remaining
deadlocks. The deadlock detection feature has already helped me fix a
bug that was causing hangs in the VFS, so it's really useful.

 - generic semaphore implementation

the generic semaphore implementation (which uses rwsems) makes it
possible to use the deadlock detection mechanism for all the mutex types
we currently have: semaphores, rw-semaphores, spinlock-mutexes and
rwlock-mutexes. Another benefit is that PREEMPT_REALTIME becomes much
more portable this way. (although it's still x86-only at the moment.)

other changes since -U4:

 - crash fix: fixed a possible "unbound recursion upon IRQ entry" bug.
   introduced preempt_schedule_irq() which now schedules without
   enabling interrupts again, preventing new IRQs from hitting this
   task again and triggering preemption. This might fix the
   'infinite stackdumps' problem Rui Nuno Capela was seeing.

 - deadlock fix: is_subdir()'s PREEMPT_REALTIME locking was buggy. This 
   could perhaps fix the other problem reported by Rui Nuno Capela.

 - i8253_lock fixes: apm, hd.c, gameport.c and analog.c were all 
   improperly importing the variable while overriding the prototype. 
   This fixes the bug reported by Florian Schmidt.

 - possible crash fix: one particular lock in selinux has to be 
   mutex-based, because while held it calls other mutex-using code.

 - two more selinux locks converted to raw spinlocks, because they were
   called from within raw-critical sections.

 - debug feature: enforce interrupts-enabled upon schedule().
   (Note that this does not break sleep_on() because sleep_on() does not
   disable interrupts in the PREEMPT_REALTIME mode. It might break with 
   !PREEMPT_REALTIME though.)

 - locking cleanup: converted the IPC code from raw spinlocks & RCU to
   spinlock-mutexes.

 - code cleanup: cleaned up the generic rwsem code.

 - debug feature: implemented /proc/sys/kernel/trace_verbose runtime
   flag (default:0), which enables a much more verbose printout in
   /proc/latency_trace.  This trace format can be useful in e.g. 
   debugging timestamp weirdnesses.

 - irqs-off fix: there was one codepath where irqd would call schedule() 
   with interrupts disabled.

 - debug feature: the NMI entries in the latency trace now also include 
   the last-observed-EFLAGS value. Can be useful in figuring out what a
   certain CPU is doing and why.

 - cleanup: fixed preemption-off ordering: often the spinlock (and 
   scheduler) code would re-enable preemption and interrupts in the
   wrong order, opening up a small window for an interrupt handler to
   fit in and increase the latency of that almost-finished critical
   section.

 - cleanup: consolidated various bug-printouts. It should now be easy to
   find whether anything bad happens even amongst lots of preempt-timing
   printouts: 'dmesg | grep BUG'.

to create a -U5 tree from scratch, the patching order is:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.8.tar.bz2
 + http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.9-rc4.bz2
 + http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9-rc4/2.6.9-rc4-mm1/2.6.9-rc4-mm1.bz2
 + http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc4-mm1-U5

	Ingo

[patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U6

[Ingo Molnar]

i have released the -U6 Real-Time Preemption patch:
 
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U6

this is a fixes-only release.

found and fixed the 'big bug' that was probably the one causing
stability problems for a number of people. There was a small window for
a task double-free race to occur, causing all sorts of crashes later on.
This bug could trigger on UP and SMP systems alike, on SMP being a bit
more frequent.

Also, a common networking deadlock was found and fixed as well, using
the deadlock detector.

Changes since -U5:

- crash bug: fix task double free caused by irq-preemption of 
  do_exit(). This got introduced in -U5 as part of a simplification of
  the zombie-reaping rewrite that the -U series did. That rewrite had an 
  unrobustness which got triggered by -U5 in a subtle way, opening up a
  small window at the end of do_exit() for an interrupt-triggered
  preemption to cause a double-free. This could fix some of the crashes
  reported by Rui Nuno Capela, Mark H Johnson.

- deadlock bug: fix networking deadlock reported by Matthew L Foster.
  Restructured the way the RT-RCU locking of ptype_lock is done - it's
  cleaner and more obvious now (besides being correct). This could also
  fix the deadlock reported by Michal Schmidt.

- deadlock bug: fix NFS startup breakage related to semaphore abuse,
  patch from Thomas Gleixner.

- build bug: fix aha152x.c, based on patch from K.R. Foley.

- build bug: fix compilation error in qla2xxx. (reported by Fernando
  Pablo Lopez-Lezcano and Mark H Johnson)

- build bug: fix !PREEMPT_REALTIME compilation error. (reported by
  Matthew L Foster)

- build bug: fix ipmi-watchdog compilation error. (reported by Mark H 
  Johnson)

- tracer fix: if an assert happens within the tracer then we'd get into
  infinite recursion. The fix was to correctly nest tracing on/off
  points.

- debug enhancement: added a few more asserts to catch underflowing 
  atomic counters. (this made the task double-free trigger earlier.)

- debug enhancement: extended CONFIG_DEBUG_STACKOVERFLOW to be 
  mcount()-driven as well. This helps in catching stack overflows much
  more reliably than the do_IRQ() based method.

to create a -U6 tree from scratch, the patching order is:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.8.tar.bz2
 + http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.9-rc4.bz2
 + http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9-rc4/2.6.9-rc4-mm1/2.6.9-rc4-mm1.bz2
 + http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U6

	Ingo

[patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U7

[Ingo Molnar]

i have released the -U7 Real-Time Preemption patch:

  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U7

this too is a fixes-only release.

Changes since -U6:

 - crash fix: turn off 4K stacks when using RWSEM_DEADLOCK_DETECTION, 
   and tune down the default max # of tasks traced per semaphore. This 
   increases process-stack size and reduces the footprint of lock 
   objects. This should fix the bootup crash reported by Rui Nuno 
   Capela.

 - assert fix: fixed an ide-taskfile scheduling-with-irqs-off assert
   that Rui's .config triggers.

 - assert workaround: disabled PARPORT_1284 for now, this should fix the
   assert seen by Mark H Johnson.

 - NFS fix: clnt.c fix from Thomas Gleixner

 - debugging helper: print stackframe-size in backtraces.

 - large-stackframe fix: inflate.c fix

to create a -U7 tree from scratch, the patching order is:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.8.tar.bz2
 + http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.9-rc4.bz2
 + http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9-rc4/2.6.9-rc4-mm1/2.6.9-rc4-mm1.bz2
 + http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U7

	Ingo

[patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Ingo Molnar]

i have released the -U8 Real-Time Preemption patch:

  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U8

this too is a fixes-only release. It includes the many semaphore-abuse
and sleep_on() fixes/improvements from Thomas Gleixner, and it also
includes a couple of semaphore related fixes.

I believe the semaphore fixes should resolve a number of the deadlocks
reported for -U7.

In particular it seems the only sane and reliable way to convert RCU
locking was to allow the following semantics for rwsems: allow reads to
nest, and allow self-read-recursion of a self-write-held semaphore. My
current implementation for this allows semaphore unfairness, but that
can be fixed later on. Most importantly, the RCU to RT-locking
conversions are much more automatic now and map nicely to what the code
is doing upstream. Most of the time they involve a conversion of a
spinlock or semaphore into a rwlock or rwsem. The old code maps to new
code almost automatically, the only manual work needed was to associate
the rcu_read_lock() with the writers-lock that it excludes against,
which is a pretty clear (but not automatic, and hence not automatable)
decision. This way i could convert some more networking code, and
simplify the older changes and hopefully get rid of some deadlocks. The
locking API is still not in its final form, but it's getting closer.

Changes since -U7:

 - deadlock fix: sysfs/driver-base semaphore fixes from Thomas Gleixner

 - deadlock fix: scsi semaphore fixes from Thomas Gleixner

 - NFS sleep_on() fixes from Thomas Gleixner

 - rawmidid.c sleep_on() fix from Thomas Gleixner

 - [ i've added more wait_for_completion_*() primitives, to ease 
     conversion of other semaphore-(ab-)using code. ]

 - make rwsems self-recursive

 - RCU lock conversion: convert rtnl_sem RCU use.

 - netfilter deadlock fix - clean up RCU locking.

to create a -U8 tree from scratch, the patching order is:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.8.tar.bz2
 + http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.9-rc4.bz2
 + http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9-rc4/2.6.9-rc4-mm1/2.6.9-rc4-mm1.bz2
 + http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U8

	Ingo

[patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U9

[Ingo Molnar]

i have released the -U9 Real-Time Preemption patch, which can be
downloaded from:

  http://redhat.com/~mingo/realtime-preempt/

this too is a fixes-only release. It includes more driver fixes and
improvements from Thomas Gleixner.

Changes since -U8.1:

 - USB semaphore->completion conversion from Thomas Gleixner

 - netconsole fixes from Michal Schmidt

 - fbcon fixes

 - added counted semaphores, this is now used by firewire, XFS and ACPI. 
   This could fix the firewire breakage - but testing would be welcome.

 - PREEMPT_ACTIVE irqs-enabled critical path removal.

 - fixed irqs-off raw spinlock primitives on UP: they enabled irqs 
   before enabling preemption, creating a window for an interrupt to
   slip in and increase the critical path.

 - made the deadlock detector not crash the current process - it will
   just hang. This produces far nicer log output while still not
   endangering stability. Also, fixed a bug in the detector that happens 
   if the trace buffer overflows.

 - made the atomic-counter-underflow detector non-fatal as well, for the
   same reasons.

to create a -U9 tree from scratch, the patching order is:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.8.tar.bz2
 + http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.9-rc4.bz2
 + http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9-rc4/2.6.9-rc4-mm1/2.6.9-rc4-mm1.bz2
 + http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U9

	Ingo

[patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U9.3

[Ingo Molnar]

i have released the -U9.3 Real-Time Preemption patch, which can be
downloaded from:

  http://redhat.com/~mingo/realtime-preempt/

this too is a fixes-only release.

Changes since -U9.2:

 - tons more driver/mutex/completion conversion done by Thomas Gleixner 
   for: ppp, ipmi, parport/ieeee1284, scsi, hotplug, and more.

 - iptables/netfilter deadlock fix, this should fix the bug reported by 
   Michal Schmidt.

 - .config housekeeping: disallow the turning off of PREEMPT_BKL when 
   PREEMPT_REALTIME is on. This solves the build error reported by 
   Matthew L Foster.

 - print the full stacktrace of the current task in the deadlock 
   detector and dont use show_stack(). This explains some of the weird
   partial stackdumps reported.

 - some more minor updates to the case when the deadlock detector turns
   itself off due to reaching the limit. We kept the spinlock locked.

to create a -U9.3 tree from scratch, the patching order is:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.8.tar.bz2
 + http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.9-rc4.bz2
 + http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9-rc4/2.6.9-rc4-mm1/2.6.9-rc4-mm1.bz2
 + http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U9.3

	Ingo

[patch] Real-Time Preemption, -RT-2.6.9-mm1-U10

[Ingo Molnar]

i have released the -U10 Real-Time Preemption patch, which can be
downloaded from:
 
  http://redhat.com/~mingo/realtime-preempt/

this is purely a rebasing of -U9.3 to 2.6.9-mm1.

to create a -U10 tree from scratch, the patching order is:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
 + http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9/2.6.9-mm1/2.6.9-mm1.bz2
 + http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-mm1-U10

	Ingo

[patch] Real-Time Preemption, -RT-2.6.9-mm1-U10.2

[Ingo Molnar]

i have released the -U10.2 Real-Time Preemption patch, which can be
downloaded from:

  http://redhat.com/~mingo/realtime-preempt/
 
this is a fixes-only release.

Changes since -U10:

 - fixed a big bug present ever since: the BKL got dropped when a
   spinlock-mutex was acquired and it scheduled away. This reduced the
   locking efficiency of the BKL. A number of outstanding problems could
   be affected, in particular this should fix the tty locking breakage
   reported by Alexander Batyrshin and Adam Heath. UP and SMP systems 
   are affected too, with SMP systems having a higher chance to trigger
   this condition.

 - tulip.c breakage fix from Thomas Gleixner

 - tg3 and 3c59x fixes.

 - made the hardirq threads SCHED_FIFO by default. They get priorities 
   between 25 and 50, depending on the irq #. (this is pretty random but 
   i found no better scheme.) Made the softirq thread SCHED_FIFO by 
   default as well, albeit this probably will have to change. These 
   changes should make it easier to debug a hung system.

to create a -U10.2 tree from scratch, the patching order is:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
 + http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9/2.6.9-mm1/2.6.9-mm1.bz2
 + http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-mm1-U10.2

	Ingo

[patch] Real-Time Preemption, -RT-2.6.9-mm1-V0

[Ingo Molnar]

i have released the -V0 Real-Time Preemption patch, which can be
downloaded from:

  http://redhat.com/~mingo/realtime-preempt/

NOTE: this is a highly experimental release, a more experimental one
than -U10.3.

the big change in the '-V' series of the patchset is that i have
converted the last couple of non-preemptible kernel subsystems to
fully-preemptible mutex-based locking. These subsystems are:

 - the SLAB allocator
 - the buddy page allocator
 - waitqueue handling
 - soft-timer subsystem
 - security/selinux
 - workqueues
 - the random driver

this is probably the last 'big leap forward' in terms of the scope of
the patch. (having reached the ultimate scope: it now encompasses
everything ;)

But as an inevitable result of this big leap it will likely break in a
couple of places. Unfortunately these subsystems were largely
interdependent so it's an all-or-nothing step with not much middle
ground between the locking done in -U10.3 and in -V0.

another result of these changes is that the number of critical sections
in -V0 is roughly 30% of that in -U10.3. Now we only have the scheduler
and very lowlevel IRQ-hardware locks as raw spinlocks. (plus the lone
holdout vga_lock - which i will probably make a mutex too in the near
future)

[ NOTE: there's one known bug in this release: selinux on one of my
testsystems broke, it hangs during bootup. With CONFIG_SECURITY disabled
it works fine. I'm working on the fix. So please keep CONFIG_SECURITY
disabled for the time being. ]

other changes in -V0:

 - build fixes: more driver fixes from Thomas Gleixner

 - crash fix: fixed a bug found by Thomas Gleixner: rwsem runtime
   initialization was racy.

 - deadlock fix: fixed lockup bug caused by __schedule clearing
   PREEMPT_ACTIVE. The need_resched loop is now outside of __schedule(). 
   This might solve lockups/slowdowns reported by some people.

 - latency fix: made keventd SCHED_FIFO - this could fix the mouse
   related delays reported by a number of people.

 - latency fix: fixed SMP lock-break mechanism of mutexes.

 - usability feature: hard-interrupts get decreasing SCHED_FIFO priority
   starting at prio 49 and stopping at prio 25. This should give a good
   default.

 - debug feature: implemented SysRq-D to show the list of tasks with
   locks blocked on, if RW_SEM_DEADLOCK_DETECTION is enabled.

to create a -V0 tree from scratch, the patching order is:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
 + http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9/2.6.9-mm1/2.6.9-mm1.bz2
 + http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-mm1-V0

	Ingo

[patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.3

[Ingo Molnar]

i have released the -V0.3 Real-Time Preemption patch, which can be
downloaded from:

	http://redhat.com/~mingo/realtime-preempt/

this is a fixes-only release, but still experimental.

this release should fix a number of bugs that were reported for the V0
series: the futex.c assert, the lockups and the 'slowdown problem'.

The slowdown problem was an architectural issue that surfaced sometime
around U10 and increased in prominence as the the number of mutexes
increased and the number of spinlocks decreased. The futex.c assert was
related to this architectural issue as well, and most of the lockups
reported were i believe livelocks caused by the same issue. Also, the
scheduler path had an easy-to-trigger deadlock that often just silently
locked up.

some of the networking lockups might be related to this issue too, but i
think PREEMPT_REALTIME still has separate lock odering issues within the
networking code. Please re-report any deadlock-tracer asserts that you
might encounter.

Changes since -V0.2:

 - HEAP_SIZE fix from Karsten Wiese

 - fix hdparm-triggered debugging message reported by Mark H Johnson

 - fixed mutex related preemption to not impact the task state, just 
   like a normal spinlock does. This necessiated the introduction of
   TASK_RUNNING_MUTEX handling and related kernel infrastructure. This 
   framework avoids spurious wakeups done by mutex handling by isolating
   the state changes done by normal wakeups vs. the state changes caused
   by the mutex code.

 - added per-CPU deschedule threads. This fixes a deadlock scenario and
   it is also much faster than keventd.

 - fix debugging message upon console unblanking

to create a -V0.3 tree from scratch, the patching order is:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
 + http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9/2.6.9-mm1/2.6.9-mm1.bz2
 + http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-mm1-V0.3

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.3

[Rui Nuno Capela]

Ingo Molnar wrote:
>
> i have released the -V0.3 Real-Time Preemption patch, which can be
> downloaded from:
>
> 	http://redhat.com/~mingo/realtime-preempt/
>
> this is a fixes-only release, but still experimental.

OK. Currently with RT-V0.3.2.

So it seems that the jackd -R is no more an issue here.

I've tested several times now, and can even start more than 7 (yes,
seven!) fluidsynth instances, with respective soundfonts loaded, all
running without problems, besides the cpu cost topping to 80% (user) and
memory is near the swappiness edge, which is usually a normal behavior, or
so I believe.

Remember that having just 2 (two) fluidsynth instances was quite enough
for hosing the system in no time, on RT-V0.2.

However (oh no!:) those jackd -R xruns are still frequent, much frequent
than RT-U3, which is my stable RT kernel atm.

OK. I'll take this time for some questions:

What's the rationale that you guys are using on tunning the IRQ threading
policies and priorities?

What's the best approach to take, regarding the jackd, soundcard, usb,
keyboard, mouse, whatever IRQ handlers?

I've heard somewhat contraditory opinions elsewhere, but would like to
know what's in the road ahead ;)

As a side note, while I was testing this snd-usb-usx2y ALSA development
module for my Tascam US-224 USB audio/midi control interface, I've found
that the best and stable results are achieved by leveraging the ohci_hcd
IRQ handler (normally IRQ 10) to a higher priority than jackd's.

I've been doing just that with e.g. chrt -p -f 60 `pidof "IRQ 10"`.
Failing to do so just makes jackd (or it's alsa backend) missing some
deadline and drop out very easily. This has been my conclusion while
testing with RT-U3. Don't know what is reserved by RT-V0.3.2, yet. I'll do
that later.

Take care.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.3

[Ingo Molnar]

* Rui Nuno Capela <rncbc@rncbc.org> wrote:

> OK. Currently with RT-V0.3.2.
> 
> So it seems that the jackd -R is no more an issue here.

great.

> However (oh no!:) those jackd -R xruns are still frequent, much
> frequent than RT-U3, which is my stable RT kernel atm.

-V0.4.1 could help with this problem. There were a number of places
where the PREEMPT_REALTIME kernel missed reschedules so it could easily
happen that jackd would sit in the runqueue waiting to be executed and
the kernel got quickly out of a critical section but then the kernel
'forgot' to reschedule for many milliseconds!

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Rui Nuno Capela]

Ingo Molnar wrote:
>
> Rui Nuno Capela wrote:
>
>> OK. Currently with RT-V0.3.2.
>>
>> So it seems that the jackd -R is no more an issue here.
>
> great.
>
>> However (oh no!:) those jackd -R xruns are still frequent, much
>> frequent than RT-U3, which is my stable RT kernel atm.
>
> -V0.4.1 could help with this problem. There were a number of places
> where the PREEMPT_REALTIME kernel missed reschedules so it could easily
> happen that jackd would sit in the runqueue waiting to be executed and
> the kernel got quickly out of a critical section but then the kernel
> 'forgot' to reschedule for many milliseconds!
>

On RT-V0.4.1, xruns seems slighly reduced, but plenty enough for my taste.

Running jackd -R with 6 fluidsynth instances gives me 0 (zero) xruns on
RT-U3, but more than 20 (twenty) on RT-V0.4.1, under a 5 minute time
frame. It was 30 (thirty something) on RT-V0.4, but overall "feel" is
about the same.

Cheers.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Lee Revell]

On Wed, 2004-10-27 at 16:26 +0100, Rui Nuno Capela wrote:
> On RT-V0.4.1, xruns seems slighly reduced, but plenty enough for my taste.
> 

Have you tried making ksoftirqd SCHED_OTHER?  This drastically reduced
xruns on my system with an earlier version.

Lee

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Rui Nuno Capela]

> Lee Revell
> On Wed, 2004-10-27 at 16:26 +0100, Rui Nuno Capela wrote:
>> On RT-V0.4.1, xruns seems slighly reduced, but plenty enough for my
>> taste.
>>
>
> Have you tried making ksoftirqd SCHED_OTHER?  This drastically reduced
> xruns on my system with an earlier version.
>
> Lee
>

Hmm... ksoftirqd/0 (or also ksoftirqd/1 on 2cpu SMP) are already
SCHED_OTHER by default, at least on both of my boxes that are running
RT-V0.4.1 now.

Should I try the other way around? Lets see... 'chrt -p -f 90 `pidof
ksoftirwd/0`',... yes, apparentely the xrun rate seems to decrease into
half, but IMHO not conclusive enough, thought.

CU
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[karsten wiese]

 --- Rui Nuno Capela <rncbc@rncbc.org> schrieb: 
> Should I try the other way around? Lets see... 'chrt -p
> -f 90 `pidof
> ksoftirwd/0`',... yes, apparentely the xrun rate seems to
> decrease into
> half, but IMHO not conclusive enough, thought.
> 
'into half' makes me wonder:
did you also 'chrt -p -f 90 `pidof ksoftirwd/1`'?
I guess you meant that with '...'. Just in case :-)

Best,
Karsten


	

	
		
___________________________________________________________
Gesendet von Yahoo! Mail - Jetzt mit 100MB Speicher kostenlos - Hier anmelden: http://mail.yahoo.de

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Rui Nuno Capela]

karsten wiese wrote:
> Rui Nuno Capela wrote:
>> Should I try the other way around? Lets see... 'chrt -p
>> -f 90 `pidof ksoftirwd/0`',... yes, apparentely the xrun
>> rate seems to decrease into half, but IMHO not conclusive
>> enough, thought.
>>
> 'into half' makes me wonder:
> did you also 'chrt -p -f 90 `pidof ksoftirwd/1`'?
> I guess you meant that with '...'. Just in case :-)
>

Wonder no more. All my statistical-wise tests were carried on a UP box (my
laptop), so there's no "ksoftirqd/1" in there, just a single
"ksoftirqd/0".

Speaking of which, I was not taking tests very seriously on my other
SMP/HT box, just because I don't want to rant about it anymore :) Only
recently VP and RT kernels were barely able to boot there, where even
plain vanilla 2.6.9 seems to be snappier and with far fewer xruns than
V0.4.1 or even U3 (either RT or not).

OTOH, on my laptop (P4/UP) I can testify as truth that, at least for
RT-U3, the improvement is real: I don't have a record of such a top
performer, when regarding the zero-xrun, low-latency audio setup
potential. When even compared, it just outperforms by far that old
2.4+preempt+low-latency myth ;)

Unfortunately, this is not what I see on my P4/SMP/HT desktop box. I
cannot tell a lie ;)

Bye now.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

P.S. Karsten, my US-224 is working real nice on my laptop now (provided
I'm with RT-U3 :) I'm real thankful for all of your work on snd-usb-usx2y.
Cheers.

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[karsten wiese]

 --- Rui Nuno Capela <rncbc@rncbc.org> schrieb: 
> Should I try the other way around? Lets see... 'chrt -p
> -f 90 `pidof
> ksoftirwd/0`',... yes, apparentely the xrun rate seems to
> decrease into
> half, but IMHO not conclusive enough, thought.
> 
'into half' makes me wonder:
did you also 'chrt -p -f 90 `pidof ksoftirwd/1`'?
(I guess you meant that with '...') And isn't it
'ksoftirqd'? Just in case :-)

Best,
Karsten


	

	
		
___________________________________________________________
Gesendet von Yahoo! Mail - Jetzt mit 100MB Speicher kostenlos - Hier anmelden: http://mail.yahoo.de

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Ingo Molnar]

* Rui Nuno Capela <rncbc@rncbc.org> wrote:

> On RT-V0.4.1, xruns seems slighly reduced, but plenty enough for my
> taste.
> 
> Running jackd -R with 6 fluidsynth instances gives me 0 (zero) xruns
> on RT-U3, but more than 20 (twenty) on RT-V0.4.1, under a 5 minute
> time frame. It was 30 (thirty something) on RT-V0.4, but overall
> "feel" is about the same.

does the wakeup tracer show any high latency?

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Ingo Molnar]

* Rui Nuno Capela <rncbc@rncbc.org> wrote:

> On RT-V0.4.1, xruns seems slighly reduced, but plenty enough for my
> taste.
> 
> Running jackd -R with 6 fluidsynth instances gives me 0 (zero) xruns
> on RT-U3, but more than 20 (twenty) on RT-V0.4.1, under a 5 minute
> time frame. It was 30 (thirty something) on RT-V0.4, but overall
> "feel" is about the same.

ok, i've uploaded RT-V0.4.2 which has more of the same: it fixes other
missed preemption checks. Does it make any difference to the xruns on
your UP box?

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> ok, i've uploaded RT-V0.4.2 which has more of the same: it fixes other
> missed preemption checks. Does it make any difference to the xruns on
> your UP box?

uploaded RT-V0.4.3 - there was a thinko in the latency tracer that
caused early boot failures.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Rui Nuno Capela]

Ingo Molnar wrote:
>
>> ok, i've uploaded RT-V0.4.2 which has more of the same: it fixes other
>> missed preemption checks. Does it make any difference to the xruns on
>> your UP box?
>
> uploaded RT-V0.4.3 - there was a thinko in the latency tracer that
> caused early boot failures.
>

Yes, the xrun rate has decreased, slightly. RT-V0.4.3 is now ranking under
10 per 5 min (~2/min), with jackd -R -r44100 -p128 -n2, fluidsynth x 6.

Better still, but not to par as RT-U3, under the very same conditions.

Cya.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Ingo Molnar]

* Rui Nuno Capela <rncbc@rncbc.org> wrote:

> >> ok, i've uploaded RT-V0.4.2 which has more of the same: it fixes other
> >> missed preemption checks. Does it make any difference to the xruns on
> >> your UP box?
> >
> > uploaded RT-V0.4.3 - there was a thinko in the latency tracer that
> > caused early boot failures.
> >
> 
> Yes, the xrun rate has decreased, slightly. RT-V0.4.3 is now ranking
> under 10 per 5 min (~2/min), with jackd -R -r44100 -p128 -n2,
> fluidsynth x 6.
> 
> Better still, but not to par as RT-U3, under the very same conditions.

how much idle time do you have in the RT-U3 and in the RT-V0.4 tests,
compared? If it's close to 100% then make sure you have the following
debug options disabled:

 # CONFIG_DEBUG_SLAB is not set
 # CONFIG_DEBUG_PREEMPT is not set
 # CONFIG_DEBUG_SPINLOCK_SLEEP is not set
 # CONFIG_PREEMPT_TIMING is not set
 # CONFIG_RWSEM_DEADLOCK_DETECT is not set
 # CONFIG_FRAME_POINTER is not set
 # CONFIG_DEBUG_STACKOVERFLOW is not set
 # CONFIG_DEBUG_STACK_USAGE is not set
 # CONFIG_DEBUG_PAGEALLOC is not set

RWSEM_DEADLOCK, DEBUG_PREEMPT, PREEMPT_TIMING and LATENCY_TRACE are
especially expensive, so depending on the amount of kernel work done, it
can make or break the total balance of CPU time used and you could get
xruns not only due to kernel latencies but purely due to having not
enough CPU time to generate audio output. (fluidsynth is a software
audio generator?)

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Rui Nuno Capela]

Ingo Molnar wrote:
>
> * Rui Nuno Capela wrote:
>
>> >> ok, i've uploaded RT-V0.4.2 which has more of the same: it fixes
>> >> other missed preemption checks. Does it make any difference to the
>> >> xruns on your UP box?
>> >
>> > uploaded RT-V0.4.3 - there was a thinko in the latency tracer that
>> > caused early boot failures.
>> >
>>
>> Yes, the xrun rate has decreased, slightly. RT-V0.4.3 is now ranking
>> under 10 per 5 min (~2/min), with jackd -R -r44100 -p128 -n2,
>> fluidsynth x 6.
>>
>> Better still, but not to par as RT-U3, under the very same conditions.
>
> how much idle time do you have in the RT-U3 and in the RT-V0.4 tests,
> compared? If it's close to 100% then make sure you have the following
> debug options disabled:
>
>  # CONFIG_DEBUG_SLAB is not set
>  # CONFIG_DEBUG_PREEMPT is not set
>  # CONFIG_DEBUG_SPINLOCK_SLEEP is not set
>  # CONFIG_PREEMPT_TIMING is not set
>  # CONFIG_RWSEM_DEADLOCK_DETECT is not set
>  # CONFIG_FRAME_POINTER is not set
>  # CONFIG_DEBUG_STACKOVERFLOW is not set
>  # CONFIG_DEBUG_STACK_USAGE is not set
>  # CONFIG_DEBUG_PAGEALLOC is not set
>
> RWSEM_DEADLOCK, DEBUG_PREEMPT, PREEMPT_TIMING and LATENCY_TRACE are
> especially expensive, so depending on the amount of kernel work done, it
> can make or break the total balance of CPU time used and you could get
> xruns not only due to kernel latencies but purely due to having not
> enough CPU time to generate audio output. (fluidsynth is a software
> audio generator?)
>

As far as nmeter can tell, this a rough cpu usage pattern between RT-U3
and RT-V0.4.3, during my jackd + 6*fluidsynth "benchmark" tests:

  cpu usage                    RT-U3.0    RT-V0.4.3
  ---------------------------- ---------- ---------
  system (kernel)              <10%        10%
  user                          30%        60%
  ---------------------------- ---------- ---------
  total                        <40%        70%


The following table compares the state between my RT-U3 and RT-V0.4.3
configurations, regarding only the mentioned options:

  option                       RT-U3.0    RT-V0.4.3
  ---------------------------- ---------- ---------
  CONFIG_DEBUG_SLAB              n          n
  CONFIG_DEBUG_PREEMPT           y          y
  CONFIG_DEBUG_SPINLOCK_SLEEP    n          -
  CONFIG_PREEMPT_TIMING          n          n
  CONFIG_RWSEM_DEADLOCK_DETECT   -          y
  CONFIG_FRAME_POINTER           y          y
  CONFIG_DEBUG_STACKOVERFLOW     y          y
  CONFIG_DEBUG_STACK_USAGE       n          n
  CONFIG_DEBUG_PAGEALLOC         n          n

(dash "-" means that the option is not available in the config).

As you can see, it can only be CONFIG_RWSEM_DEADLOCK_DETECT, being new in
RT-V0.4.3, that is probably affecting on RT-V0.4.3. I'll try to rebuild
and test all over without it, and see if it gets any better.

BBL
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Ingo Molnar]

* Rui Nuno Capela <rncbc@rncbc.org> wrote:

> The following table compares the state between my RT-U3 and RT-V0.4.3
> configurations, regarding only the mentioned options:
> 
>   option                       RT-U3.0    RT-V0.4.3
>   ---------------------------- ---------- ---------
>   CONFIG_DEBUG_SLAB              n          n
>   CONFIG_DEBUG_PREEMPT           y          y
>   CONFIG_DEBUG_SPINLOCK_SLEEP    n          -
>   CONFIG_PREEMPT_TIMING          n          n
>   CONFIG_RWSEM_DEADLOCK_DETECT   -          y
>   CONFIG_FRAME_POINTER           y          y
>   CONFIG_DEBUG_STACKOVERFLOW     y          y
>   CONFIG_DEBUG_STACK_USAGE       n          n
>   CONFIG_DEBUG_PAGEALLOC         n          n
> 
> (dash "-" means that the option is not available in the config).
> 
> As you can see, it can only be CONFIG_RWSEM_DEADLOCK_DETECT, being new
> in RT-V0.4.3, that is probably affecting on RT-V0.4.3. I'll try to
> rebuild and test all over without it, and see if it gets any better.

note that DEBUG_PREEMPT got more expensive in the -V kernels. I'd
suggest to disable all the 'y' ones in both the -U and -V kernel and
compare them then.

but especially the userspace overhead seems to be significantly higher
in the -V kernel so i'm not quite sure it can all be attributed to
debugging overhead. We'll see.

also, how does the context-switching rate compare between the two tests? 
This test is pretty steady when it's running, so the context-switch
rates can be directly compared, correct?

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Rui Nuno Capela]

Ingo Molnar wrote:
>
> * Rui Nuno Capela wrote:
>
>> The following table compares the state between my RT-U3 and RT-V0.4.3
>> configurations, regarding only the mentioned options:
>>
>>   option                       RT-U3.0    RT-V0.4.3
>>   ---------------------------- ---------- ---------
>>   CONFIG_DEBUG_SLAB              n          n
>>   CONFIG_DEBUG_PREEMPT           y          y
>>   CONFIG_DEBUG_SPINLOCK_SLEEP    n          -
>>   CONFIG_PREEMPT_TIMING          n          n
>>   CONFIG_RWSEM_DEADLOCK_DETECT   -          y
>>   CONFIG_FRAME_POINTER           y          y
>>   CONFIG_DEBUG_STACKOVERFLOW     y          y
>>   CONFIG_DEBUG_STACK_USAGE       n          n
>>   CONFIG_DEBUG_PAGEALLOC         n          n
>>
>> (dash "-" means that the option is not available in the config).
>>
>> As you can see, it can only be CONFIG_RWSEM_DEADLOCK_DETECT, being new
>> in RT-V0.4.3, that is probably affecting on RT-V0.4.3. I'll try to
>> rebuild and test all over without it, and see if it gets any better.
>
> note that DEBUG_PREEMPT got more expensive in the -V kernels. I'd
> suggest to disable all the 'y' ones in both the -U and -V kernel and
> compare them then.
>
> but especially the userspace overhead seems to be significantly higher
> in the -V kernel so i'm not quite sure it can all be attributed to
> debugging overhead. We'll see.
>
> also, how does the context-switching rate compare between the two tests?
> This test is pretty steady when it's running, so the context-switch
> rates can be directly compared, correct?
>

OK. That was it. After switching off CONFIG_RWSEM_DEADLOCK_DETECT on
RT-V0.4.3, I can say that it's now on par to RT-U3.

Later today, I will conduct some extendeded testing, where I'll able to
compare the jackd performance between vanilla, RT-U3 and RT-V0.4.3, on my
UP laptop. All kernel configurations will be stripped off from all the
debug options.

I will take note of xrun rate, jackd scheduling delay histogram, and cpu
usage. Context switch rate will be also acquainted.

Anything else?
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Ingo Molnar]

* Rui Nuno Capela <rncbc@rncbc.org> wrote:

> OK. That was it. After switching off CONFIG_RWSEM_DEADLOCK_DETECT on
> RT-V0.4.3, I can say that it's now on par to RT-U3.

great!

> Later today, I will conduct some extendeded testing, where I'll able
> to compare the jackd performance between vanilla, RT-U3 and RT-V0.4.3,
> on my UP laptop. All kernel configurations will be stripped off from
> all the debug options.
>
> I will take note of xrun rate, jackd scheduling delay histogram, and
> cpu usage. Context switch rate will be also acquainted.
> 
> Anything else?

yeah, that's good enough. I'd still suggest to first test new kernels
with all the debug options on, to make sure it's stable. For performance
comparisons turn all the debug options off.

i'd also suggest to turn the NMI watchdog off (if enabled), that can
inject a 10-20 usec latency into any critical path. For the absolute
lowest latencies i'd also suggest to turn off all the APIC options
(possible in a UP kernel only, and only if the XT-PIC setup doesnt cause
unacceptable IRQ-line sharing), the IO-APIC mask handling is a bit
expensive compared to the XT-PIC.

If you find (or suspect) larger latencies anywhere then PREEMPT_TIMING +
LATENCY_TRACE + preempt_enable=4 is the preferred variant to use. (right
now it's not possible to do wakeup-timing without LATENCY_TRACE, i'll
fix that.)

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Rui Nuno Capela]

Ingo Molnar wrote:
>
> * Rui Nuno Capela wrote:
>
>> OK. That was it. After switching off CONFIG_RWSEM_DEADLOCK_DETECT on
>> RT-V0.4.3, I can say that it's now on par to RT-U3.
>
> great!
>
>> Later today, I will conduct some extendeded testing, where I'll able
>> to compare the jackd performance between vanilla, RT-U3 and RT-V0.4.3,
>> on my UP laptop. All kernel configurations will be stripped off from
>> all the debug options.
>>
>> I will take note of xrun rate, jackd scheduling delay histogram, and
>> cpu usage. Context switch rate will be also acquainted.
>>
>> Anything else?
>
> yeah, that's good enough.


OK. Here are my early consolidated results. Feel free to comment.

                                    2.6.9     RT-U3   RT-V0.4.3
                                  --------- --------- ---------
  XRUN Rate . . . . . . . . . . .     424         8         4    /hour
  Delay Rate (>spare time)  . . .     496         0         0    /hour
  Delay Rate (>1000 usecs)  . . .     940         8         4    /hour
  Maximum Delay . . . . . . . . .    6904       921       721    usecs
  Maximum Process Cycle . . . . .    1449      1469      1590    usecs
  Average DSP CPU Load  . . . . .      38        39        40    %
  Average Context-Switch Rate . .    7480      8929      9726    /sec

Note: all tests were carried out running jackd -v -dalsa -dhw:0 -r44100
-p128 -n2 -S -P, loaded with 9 (nine) fluidsynth instances, on a
P4@2.533Ghz laptop, against the onboard sound device (snd-ali5451).

On the RT kernels only, the following optimizations were issued:

   chrt --pid --fifo 90 2                (pidof "ksoftirqd/0" = 2)
   chrt --pid --fifo 60 `pidof "IRQ 5"`  (snd-ali5451)

Take care.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Ingo Molnar]

* Rui Nuno Capela <rncbc@rncbc.org> wrote:

> OK. Here are my early consolidated results. Feel free to comment.
> 
>                                     2.6.9     RT-U3   RT-V0.4.3
>                                   --------- --------- ---------
>   XRUN Rate . . . . . . . . . . .     424         8         4    /hour
>   Delay Rate (>spare time)  . . .     496         0         0    /hour
>   Delay Rate (>1000 usecs)  . . .     940         8         4    /hour
>   Maximum Delay . . . . . . . . .    6904       921       721    usecs
>   Maximum Process Cycle . . . . .    1449      1469      1590    usecs
>   Average DSP CPU Load  . . . . .      38        39        40    %
>   Average Context-Switch Rate . .    7480      8929      9726    /sec

looks pretty good, doesnt it?

how is the 'maximum delay' calculated? Could you put in a tracing hook
into jackd whenever such a ~720 usecs maximum is hit? I'd _love_ to see 
how such a latency path looks like, it seems a bit long.

It should be a relatively simple hack to jackd. Firstly, download the 
-V0.5.3 patch and enable LATENCY_TRACE, then do:

	echo 2 > /proc/sys/kernel/trace_enabled

this activates the 'application-triggered kernel tracer' functionality. 

No tracing happens by default, but tracing starts if the application
executes this function:

	gettimeofday(0,1);

and tracing stops if the application does:

	gettimeofday(0,0);

whenever the app does this (0,0) call the trace gets saved and you can
retrieve it from /proc/latency_trace where you can retrieve it. There is
no combination of these parameters that can break the kernel, so it's a
100% safe tracing facility. You can 'ignore' a latency [e.g. if it's not
a maximum] by simply not doing the (0,0) call. The next (0,1) call done
will override the previous, already running trace.

[stupid function but this combination of the syscall parameters is not
used otherwise so the latency tracer hijacks it.]

i dont know how Jackd does things, but i'd suggest to enable tracing the
first time possible when getting an interrupt - in theory this should
happen as soon as the wakeup-latency-tracer says - i.e. at most in like
30 usecs. The bulk of the remaining 700 usecs will be spent in jackd, 
and you can trace those 700 usecs.

or if you would be willing to do a little bit of ALSA hacking, you could
add this to the ALSA interrupt handler:

	#include <linux/syscalls.h>

	...
	sys_gettimeofday(0, 1);

[the attached patch implements this for ali5451.]

and do the gettimeofday(0,0) in jackd [if the latency measured there is
a new maximum]! This way tracing is turned on within the kernel IRQ
handler (i.e. as soon as possible) and turned off within ALSA. This will
enable us to see an even more complete latency path.

NOTE: there can only be one trace active at a time. So if there can be
multiple channels active at a time then this user-triggered tracer might
be less useful. Do these channels have any priority? Or if multiple
channels are necessary then you could modify the patch to only do the
(0,1) call for say channel #0:

	if (channel == 0)
		sys_gettimeofday(0, 1);

in this case the trace-off-save (0,0) call in Jackd must also only do
this for channel 0 processing! (or whichever channel you find the most
interesting.)

also, i looked at the sound/pci/ali5451/ali5451.c driver code and it has
one weird piece of code on line 988:

	udelay(100);

that adds a 100 usecs latency to the main path, for no good reason! It
also spends that time burning CPU time, delaying other processing. Could 
you add an IRQs/sec measurement too if possible, so that we can compare 
the IRQ rates of various kernels?

Also, i'd suggest to simply remove that line (or apply the attached
patch) - does the driver still work fine with that?

plus i've also got questions about how Jackd interfaces with ALSA: does
it use SIGIO, or some direct driver ioctl? If SIGIO is used then how is
it done precisely - is an 'RT' queued signal used or ordinary SIGIO? 
Also, how is the 'channel' information established upon getting a SIGIO,
is it in the siginfo structure?

	Ingo

--- linux/sound/pci/ali5451/ali5451.c.orig
+++ linux/sound/pci/ali5451/ali5451.c
@@ -33,6 +33,7 @@
 #include <linux/pci.h>
 #include <linux/slab.h>
 #include <linux/moduleparam.h>
+#include <linux/syscalls.h>
 #include <sound/core.h>
 #include <sound/pcm.h>
 #include <sound/info.h>
@@ -985,11 +986,11 @@ static void snd_ali_update_ptr(ali_t *co
 	pvoice = &codec->synth.voices[channel];
 	runtime = pvoice->substream->runtime;
 
-	udelay(100);
 	spin_lock(&codec->reg_lock);
 
 	if (pvoice->pcm && pvoice->substream) {
 		/* pcm interrupt */
+		sys_gettimeofday((void *)0, (void *)1); // start the tracer
 #ifdef ALI_DEBUG
 		outb((u8)(pvoice->number), ALI_REG(codec, ALI_GC_CIR));
 		temp = inw(ALI_REG(codec, ALI_CSO_ALPHA_FMS + 2));

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Rui Nuno Capela]

Ingo Molnar wrote:
>
> * Rui Nuno Capela wrote:
>
>> OK. Here are my early consolidated results. Feel free to comment.
>>
>>                                     2.6.9     RT-U3   RT-V0.4.3
>>                                   --------- --------- ---------
>>   XRUN Rate . . . . . . . . . . .     424         8         4    /hour
>>   Delay Rate (>spare time)  . . .     496         0         0    /hour
>>   Delay Rate (>1000 usecs)  . . .     940         8         4    /hour
>>   Maximum Delay . . . . . . . . .    6904       921       721    usecs
>>   Maximum Process Cycle . . . . .    1449      1469      1590    usecs
>>   Average DSP CPU Load  . . . . .      38        39        40    %
>>   Average Context-Switch Rate . .    7480      8929      9726    /sec
>
> looks pretty good, doesnt it?
>

Yes indeed :)


> how is the 'maximum delay' calculated? Could you put in a tracing hook
> into jackd whenever such a ~720 usecs maximum is hit? I'd _love_ to see
> how such a latency path looks like, it seems a bit long.
>

That 'maximum delay' is collected on each jackd process cycle. AFAICS, it
is the figure of a scheduling delay, as measured by jackd as the time
interval between interrupt and effective jackd process handler (re)entry.

Please note that I'm not a JACK developer. I'm just a regular user
with ancient coding skills ;) I do however subscribe to the jackit-devel
maillist. And the author of qjackctl, if that matters...

For reading this 'maximum delay' I am actually using a custom patch
against jack-0.99.7cvs, being a Lee Revell's original.


> It should be a relatively simple hack to jackd. Firstly, download the
> -V0.5.3 patch and enable LATENCY_TRACE, then do:
>
>         echo 2 > /proc/sys/kernel/trace_enabled
>
> this activates the 'application-triggered kernel tracer' functionality.
>
> No tracing happens by default, but tracing starts if the application
> executes this function:
>
>         gettimeofday(0,1);
>
> and tracing stops if the application does:
>
>         gettimeofday(0,0);
>
> whenever the app does this (0,0) call the trace gets saved and you can
> retrieve it from /proc/latency_trace where you can retrieve it. There is
> no combination of these parameters that can break the kernel, so it's a
> 100% safe tracing facility. You can 'ignore' a latency [e.g. if it's not
> a maximum] by simply not doing the (0,0) call. The next (0,1) call done
> will override the previous, already running trace.
>
> [stupid function but this combination of the syscall parameters is not
> used otherwise so the latency tracer hijacks it.]
>
> i dont know how Jackd does things, but i'd suggest to enable tracing the
> first time possible when getting an interrupt - in theory this should
> happen as soon as the wakeup-latency-tracer says - i.e. at most in like
> 30 usecs. The bulk of the remaining 700 usecs will be spent in jackd,
> and you can trace those 700 usecs.
>
> or if you would be willing to do a little bit of ALSA hacking, you could
> add this to the ALSA interrupt handler:
>
>         #include <linux/syscalls.h>
>
>         ...
>         sys_gettimeofday(0, 1);
>
> [the attached patch implements this for ali5451.]
>
> and do the gettimeofday(0,0) in jackd [if the latency measured there is
> a new maximum]! This way tracing is turned on within the kernel IRQ
> handler (i.e. as soon as possible) and turned off within ALSA. This will
> enable us to see an even more complete latency path.
>
> NOTE: there can only be one trace active at a time. So if there can be
> multiple channels active at a time then this user-triggered tracer might
> be less useful. Do these channels have any priority? Or if multiple
> channels are necessary then you could modify the patch to only do the
> (0,1) call for say channel #0:
>
>         if (channel == 0)
>                 sys_gettimeofday(0, 1);
>
> in this case the trace-off-save (0,0) call in Jackd must also only do
> this for channel 0 processing! (or whichever channel you find the most
> interesting.)
>

Ouch. This is a bit too much to digest in so little time :) I'll try to
re-read this from cache, erm... tomorrow ;)

BTW, this means that I have to re-enable LATENCY_TIMING back again? Notice
that all my results were taken with a production configuration, that is,
with all debug options now set to off (OK, I think I've left the
stack-overflow on, but that was the only one).

OTOH, this latency timing has been troublesome on either of my setups,
recently. But I'll give it another try...


> also, i looked at the sound/pci/ali5451/ali5451.c driver code and it has
> one weird piece of code on line 988:
>
>         udelay(100);
>
> that adds a 100 usecs latency to the main path, for no good reason! It
> also spends that time burning CPU time, delaying other processing. Could
> you add an IRQs/sec measurement too if possible, so that we can compare
> the IRQ rates of various kernels?
>

Yes, I can add interrupts/sec measuring with nmeter. Neat utility indeed,
thanks to Denis Vlasenko.


> Also, i'd suggest to simply remove that line (or apply the attached
> patch) - does the driver still work fine with that?
>

Now that you call, I remember to hack that very same line, some time go,
but couldn't get no better than a udelay(33). Removing that line just
ended in some kind of malfunction, but can't remember what exactly. One
thing's for sure, sound didn't came out of it :-/


> plus i've also got questions about how Jackd interfaces with ALSA: does
> it use SIGIO, or some direct driver ioctl? If SIGIO is used then how is
> it done precisely - is an 'RT' queued signal used or ordinary SIGIO?
> Also, how is the 'channel' information established upon getting a SIGIO,
> is it in the siginfo structure?
>

Now that's really pushing me over. Any ALSA-JACK developers around here to
comment?

Bye now.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Lee Revell]

On Fri, 2004-10-29 at 00:49 +0100, Rui Nuno Capela wrote:
> > plus i've also got questions about how Jackd interfaces with ALSA: does
> > it use SIGIO, or some direct driver ioctl? If SIGIO is used then how is
> > it done precisely - is an 'RT' queued signal used or ordinary SIGIO?
> > Also, how is the 'channel' information established upon getting a SIGIO,
> > is it in the siginfo structure?
> >
> 
> Now that's really pushing me over. Any ALSA-JACK developers around here to
> comment?
> 

I think it uses a direct driver ioctl to open the device.  Then jack
uses mmap to talk to the audio device. 

Anyway I forwarded your question to Paul Davis, the author of JACK, and
cc'ed jackit-devel.

Lee

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Ingo Molnar]

* Rui Nuno Capela <rncbc@rncbc.org> wrote:

> BTW, this means that I have to re-enable LATENCY_TIMING back again?

yes. I'd suggest to start with the simplest setup - i.e. just one
fluidsynth instance running. I suspect 3-4 instances later on will be
enough to trigger some xruns or at least some of the bigger delays.

you possibly wont be able to debug the 'production' setup, but that's
not an issue because the latencies should show up under just 2-3
instances running as well.

> > Also, i'd suggest to simply remove that line (or apply the attached
> > patch) - does the driver still work fine with that?
> >
> 
> Now that you call, I remember to hack that very same line, some time
> go, but couldn't get no better than a udelay(33). Removing that line
> just ended in some kind of malfunction, but can't remember what
> exactly. One thing's for sure, sound didn't came out of it :-/

ugh. Possibly some sort of interaction with the firmware and/or an
outright driver bug?

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Rui Nuno Capela]

Ingo Molnar wrote:
>
> * Rui Nuno Capela wrote:
>
>> BTW, this means that I have to re-enable LATENCY_TIMING back again?
>
> yes. I'd suggest to start with the simplest setup - i.e. just one
> fluidsynth instance running. I suspect 3-4 instances later on will be
> enough to trigger some xruns or at least some of the bigger delays.
>
> you possibly wont be able to debug the 'production' setup, but that's
> not an issue because the latencies should show up under just 2-3
> instances running as well.
>
>> > Also, i'd suggest to simply remove that line (or apply the attached
>> > patch) - does the driver still work fine with that?
>> >
>>
>> Now that you call, I remember to hack that very same line, some time
>> go, but couldn't get no better than a udelay(33). Removing that line
>> just ended in some kind of malfunction, but can't remember what
>> exactly. One thing's for sure, sound didn't came out of it :-/
>
> ugh. Possibly some sort of interaction with the firmware and/or an
> outright driver bug?
>

Just confirmed that, by removing that udelay(100) line on ali5451.c, the
result is crappy sound (worst than normally is :) and most relevant to the
subject, I get a nasty jackd XRUN storm, without even blinking an eye.
Useless.

Regarding my test results, maybe this is just a distraction. I was just
comparing the kernels, not the hardware, jackd or the ali5451 alsa driver,
which were kept as constants along the evaluation.

In fact, those tests were only about to confirm, by numbers on-the-field,
that RT-V0 is on par to RT-U3. Don't bother to compare it to your own
setup and/or hardware. These are the kind of things that YMMCV = Your
Mileage Certainly Varies :)

For example, in my own case, if those tests are done with ACPI disabled
(yes, with acpi=off), this laptop of mine just skews the results
completely: vanilla 2.6.9 gets better results, while the RT ones go
slumber. Go figure ;)

OK. I'm really running out of time now. Family's calling for the weekend.

On the next few days I'll take the latency_trace route, as Ingo proposed,
patching ali5451 and jackd to issue a sys_gettimeofday(0, 0) and
gettimeofday(0, 1) trace on/off instrumentation respectively, while using
a proper RT-V0.5.x kernel patch line (or newer).

Bye now.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Lee Revell]

On Fri, 2004-10-29 at 16:00 +0100, Rui Nuno Capela wrote:
> For example, in my own case, if those tests are done with ACPI disabled
> (yes, with acpi=off), this laptop of mine just skews the results
> completely: vanilla 2.6.9 gets better results, while the RT ones go
> slumber. Go figure ;)

I suspect that your laptop uses SMM traps to talk to the battery.  That
could certainly explain the 700 us xruns, because SMM disables all
interrupts.  This was covered recently in another thread.  According to
Alan Cox most laptops work this way.

Has anyone been able to reproduce the problem on a desktop system?

Lee

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Florian Schmidt]

[-- Attachment #1: Type: text/plain, Size: 6302 bytes --]

On Thu, 28 Oct 2004 17:33:50 +0100 (WEST)
"Rui Nuno Capela" <rncbc@rncbc.org> wrote:

> OK. Here are my early consolidated results. Feel free to comment.
> 
>                                     2.6.9     RT-U3   RT-V0.4.3
>                                   --------- --------- ---------
>   XRUN Rate . . . . . . . . . . .     424         8         4    /hour
>   Delay Rate (>spare time)  . . .     496         0         0    /hour
>   Delay Rate (>1000 usecs)  . . .     940         8         4    /hour
>   Maximum Delay . . . . . . . . .    6904       921       721    usecs
>   Maximum Process Cycle . . . . .    1449      1469      1590    usecs
>   Average DSP CPU Load  . . . . .      38        39        40    %
>   Average Context-Switch Rate . .    7480      8929      9726    /sec
> 
> Note: all tests were carried out running jackd -v -dalsa -dhw:0 -r44100
> -p128 -n2 -S -P, loaded with 9 (nine) fluidsynth instances, on a
> P4@2.533Ghz laptop, against the onboard sound device (snd-ali5451).
> 
> On the RT kernels only, the following optimizations were issued:
> 
>    chrt --pid --fifo 90 2                (pidof "ksoftirqd/0" = 2)
>    chrt --pid --fifo 60 `pidof "IRQ 5"`  (snd-ali5451)
> 
> Take care.

hi,

i tried a V0.5.2 with PREEMPT_REALTIME and all debugging off (config
attached). I cannot reproduce your results. I have experienced around 30
xruns in 10 minutes. And big ones, too (> 5ms). I don't know exactly what
kind of load triggers them. Here's a bit of qjackctl message window (btw:
jackd was idle, no clients connected, except for qjackctl):

**** alsa_pcm: xrun of at least 0.594 msecs
**** alsa_pcm: xrun of at least 0.037 msecs
16:20:44.832 XRUN callback (2).
16:20:44.840 XRUN callback (3).
**** alsa_pcm: xrun of at least 0.024 msecs
**** alsa_pcm: xrun of at least 3.682 msecs
16:22:59.834 XRUN callback (4).
16:22:59.839 XRUN callback (5).
**** alsa_pcm: xrun of at least 0.016 msecs
**** alsa_pcm: xrun of at least 3.460 msecs
16:23:03.513 XRUN callback (6).
**** alsa_pcm: xrun of at least 2.028 msecs
16:23:03.869 XRUN callback (7).
**** alsa_pcm: xrun of at least 7.061 msecs
**** alsa_pcm: xrun of at least 4.510 msecs
16:23:03.894 XRUN callback (8).
16:23:03.895 XRUN callback (9).
**** alsa_pcm: xrun of at least 0.428 msecs
**** alsa_pcm: xrun of at least 0.157 msecs
16:23:54.546 XRUN callback (10).
16:23:54.547 XRUN callback (11).
16:23:54.549 XRUN callback (12).
**** alsa_pcm: xrun of at least 1.507 msecs
16:25:49.194 XRUN callback (13).
subgraph starting at qjackctl-1013 timed out (subgraph_wait_fd=14, status = 0, state = Triggered)
**** alsa_pcm: xrun of at least 1.534 msecs
16:28:23.530 XRUN callback (14).
**** alsa_pcm: xrun of at least 0.222 msecs
**** alsa_pcm: xrun of at least 2.674 msecs
**** alsa_pcm: xrun of at least 3.904 msecs
16:28:26.790 XRUN callback (15).
16:28:26.794 XRUN callback (16).
16:28:26.795 XRUN callback (17).
**** alsa_pcm: xrun of at least 0.701 msecs

Here's some setup info:

ksoftirqd/0:
mango:~# chrt -p 2
pid 2's current scheduling policy: SCHED_FIFO
pid 2's current scheduling priority: 99

snd-cs46xx:
mango:~# chrt -p `pidof "IRQ 3"`
pid 118's current scheduling policy: SCHED_FIFO
pid 118's current scheduling priority: 90

mango:~# pidof jackd
1014
mango:~# chrt -p 1014
pid 1014's current scheduling policy: SCHED_OTHER
pid 1014's current scheduling priority: 0
mango:~# chrt -p 1015
pid 1015's current scheduling policy: SCHED_OTHER
pid 1015's current scheduling priority: 0
mango:~# chrt -p 1016
pid 1016's current scheduling policy: SCHED_FIFO
pid 1016's current scheduling priority: 70
mango:~# chrt -p 1017
pid 1017's current scheduling policy: SCHED_FIFO
pid 1017's current scheduling priority: 60

Hmm, it seems i haven't disabled all debugging. This is from dmesg:

BUG: atomic counter underflow at:
 [<c010649e>] dump_stack+0x1e/0x20 (20)
 [<c025f319>] qdisc_destroy+0xd9/0xe0 (28)
 [<c025f506>] dev_shutdown+0x36/0xb0 (28)
 [<c0254679>] unregister_netdevice+0x129/0x2b0 (40)
 [<c0221919>] unregister_netdev+0x19/0x40 (16)
 [<f0902134>] ppp_shutdown_interface+0x64/0xd0 [ppp_generic] (36)
 [<f08fe0a6>] ppp_release+0x86/0x90 [ppp_generic] (16)
 [<c0155c3f>] __fput+0x14f/0x170 (36)
 [<c0154457>] filp_close+0x57/0x80 (28)
 [<c01544f0>] sys_close+0x70/0x90 (32)
 [<c0105ecf>] syscall_call+0x7/0xb (-8124)

Here's some context from syslog:

Oct 29 15:04:01 mango pppd[324]: LCP: timeout sending Config-Requests 
Oct 29 15:04:01 mango pppd[324]: Connection terminated.
Oct 29 15:04:01 mango kernel: BUG: atomic counter underflow at:
Oct 29 15:04:01 mango kernel:  [valid_stack_ptr+46/96] dump_stack+0x1e/0x20 (20)
Oct 29 15:04:01 mango kernel:  [unregister_netdevice+553/672] qdisc_destroy+0xd9/0xe0 (28)
Oct 29 15:04:01 mango kernel:  [ethtool_op_get_tso+6/32] dev_shutdown+0x36/0xb0 (28)
Oct 29 15:04:01 mango kernel:  [__sock_create+185/704] unregister_netdevice+0x129/0x2b0 (40)
Oct 29 15:04:01 mango kernel:  [blk_wait_queue_drained+25/304] unregister_netdev+0x19/0x40 (16)
Oct 29 15:04:01 mango kernel:  [pg0+810783028/1069765632] ppp_shutdown_interface+0x64/0xd0 [ppp_generic] (36)
Oct 29 15:04:01 mango kernel:  [pg0+810766502/1069765632] ppp_release+0x86/0x90 [ppp_generic] (16)
Oct 29 15:04:01 mango kernel:  [shmem_writepage+463/480] __fput+0x14f/0x170 (36)
Oct 29 15:04:01 mango kernel:  [sys_swapon+1591/2080] filp_close+0x57/0x80 (28)
Oct 29 15:04:01 mango kernel:  [sys_swapon+1744/2080] sys_close+0x70/0x90 (32)
Oct 29 15:04:01 mango kernel:  [do_notify_resume+15/76] syscall_call+0x7/0xb (-8124)
Oct 29 15:04:10 mango pppoe[326]: Timeout waiting for PADS packets
Oct 29 15:04:31 mango pppd[324]: Serial connection established.
Oct 29 15:04:31 mango pppd[324]: Using interface ppp0
Oct 29 15:04:31 mango pppd[324]: Connect: ppp0 <--> /dev/pts/0
Oct 29 15:04:47 mango pppoe[822]: PADS: Service-Name: ''
Oct 29 15:04:47 mango pppoe[822]: PPP session is 2133
Oct 29 15:04:47 mango pppd[324]: CHAP authentication succeeded
Oct 29 15:04:48 mango pppd[324]: Cannot determine ethernet address for proxy ARP
Oct 29 15:04:48 mango pppd[324]: local  IP address 213.23.197.161
Oct 29 15:04:48 mango pppd[324]: remote IP address 145.253.4.148
Oct 29 15:04:48 mango pppd[324]: primary   DNS address 195.50.140.250
Oct 29 15:04:48 mango pppd[324]: secondary DNS address 145.253.2.11

flo

[-- Attachment #2: .config --]
[-- Type: application/octet-stream, Size: 26261 bytes --]

#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.9-mm1-RT-V0.5.6
# Fri Oct 29 13:09:33 2004
#
CONFIG_X86=y
CONFIG_MMU=y
CONFIG_UID16=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y

#
# Code maturity level options
#
CONFIG_EXPERIMENTAL=y
CONFIG_CLEAN_COMPILE=y
CONFIG_BROKEN_ON_SMP=y

#
# General setup
#
CONFIG_LOCALVERSION="-rt"
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
CONFIG_BSD_PROCESS_ACCT=y
# CONFIG_BSD_PROCESS_ACCT_V3 is not set
CONFIG_SYSCTL=y
# CONFIG_AUDIT is not set
CONFIG_LOG_BUF_SHIFT=14
# CONFIG_HOTPLUG is not set
CONFIG_KOBJECT_UEVENT=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
# CONFIG_EMBEDDED is not set
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_ALL is not set
# CONFIG_KALLSYMS_EXTRA_PASS is not set
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_SHMEM=y
CONFIG_CC_ALIGN_FUNCTIONS=0
CONFIG_CC_ALIGN_LABELS=0
CONFIG_CC_ALIGN_LOOPS=0
CONFIG_CC_ALIGN_JUMPS=0
# CONFIG_TINY_SHMEM is not set

#
# Loadable module support
#
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
CONFIG_OBSOLETE_MODPARM=y
CONFIG_MODVERSIONS=y
# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_KMOD=y

#
# Processor type and features
#
CONFIG_X86_PC=y
# CONFIG_X86_ELAN is not set
# CONFIG_X86_VOYAGER is not set
# CONFIG_X86_NUMAQ is not set
# CONFIG_X86_SUMMIT is not set
# CONFIG_X86_BIGSMP is not set
# CONFIG_X86_VISWS is not set
# CONFIG_X86_GENERICARCH is not set
# CONFIG_X86_ES7000 is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
# CONFIG_MPENTIUM4 is not set
# CONFIG_MK6 is not set
CONFIG_MK7=y
# CONFIG_MK8 is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP2 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
# CONFIG_X86_GENERIC is not set
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_RWSEM_GENERIC=y
CONFIG_GENERIC_SEMAPHORES=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_USE_3DNOW=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
# CONFIG_SMP is not set
CONFIG_PREEMPT=y
CONFIG_PREEMPT_SOFTIRQS=y
CONFIG_PREEMPT_HARDIRQS=y
CONFIG_PREEMPT_REALTIME=y
CONFIG_PREEMPT_BKL=y
# CONFIG_X86_UP_APIC is not set
CONFIG_X86_TSC=y
# CONFIG_X86_MCE is not set
# CONFIG_TOSHIBA is not set
# CONFIG_I8K is not set
# CONFIG_MICROCODE is not set
# CONFIG_X86_MSR is not set
# CONFIG_X86_CPUID is not set

#
# Firmware Drivers
#
# CONFIG_EDD is not set
CONFIG_NOHIGHMEM=y
# CONFIG_HIGHMEM4G is not set
# CONFIG_HIGHMEM64G is not set
# CONFIG_MATH_EMULATION is not set
CONFIG_MTRR=y
CONFIG_HAVE_DEC_LOCK=y
# CONFIG_REGPARM is not set

#
# Performance-monitoring counters support
#
# CONFIG_PERFCTR is not set
CONFIG_KERN_PHYS_OFFSET=1
# CONFIG_KEXEC is not set

#
# Power management options (ACPI, APM)
#
CONFIG_PM=y
# CONFIG_PM_DEBUG is not set
# CONFIG_SOFTWARE_SUSPEND is not set

#
# ACPI (Advanced Configuration and Power Interface) Support
#
# CONFIG_ACPI is not set
CONFIG_ACPI_BLACKLIST_YEAR=0

#
# APM (Advanced Power Management) BIOS Support
#
CONFIG_APM=y
# CONFIG_APM_IGNORE_USER_SUSPEND is not set
# CONFIG_APM_DO_ENABLE is not set
# CONFIG_APM_CPU_IDLE is not set
# CONFIG_APM_DISPLAY_BLANK is not set
# CONFIG_APM_RTC_IS_GMT is not set
# CONFIG_APM_ALLOW_INTS is not set
CONFIG_APM_REAL_MODE_POWER_OFF=y

#
# CPU Frequency scaling
#
# CONFIG_CPU_FREQ is not set

#
# Bus options (PCI, PCMCIA, EISA, MCA, ISA)
#
CONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GOMMCONFIG is not set
# CONFIG_PCI_GODIRECT is not set
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
# CONFIG_PCI_LEGACY_PROC is not set
CONFIG_PCI_NAMES=y
# CONFIG_ISA is not set
# CONFIG_MCA is not set
# CONFIG_SCx200 is not set

#
# Executable file formats
#
CONFIG_BINFMT_ELF=y
CONFIG_BINFMT_AOUT=y
CONFIG_BINFMT_MISC=y

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
# CONFIG_DEBUG_DRIVER is not set

#
# Memory Technology Devices (MTD)
#
# CONFIG_MTD is not set

#
# Parallel port support
#
# CONFIG_PARPORT is not set

#
# Plug and Play support
#

#
# Block devices
#
CONFIG_BLK_DEV_FD=m
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_CRYPTOLOOP=m
CONFIG_BLK_DEV_NBD=m
# CONFIG_BLK_DEV_SX8 is not set
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_SIZE=4096
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
# CONFIG_LBD is not set
# CONFIG_CDROM_PKTCDVD is not set

#
# ATA/ATAPI/MFM/RLL support
#
CONFIG_IDE=y
CONFIG_BLK_DEV_IDE=y

#
# Please see Documentation/ide.txt for help/info on IDE drives
#
# CONFIG_BLK_DEV_IDE_SATA is not set
# CONFIG_BLK_DEV_HD_IDE is not set
CONFIG_BLK_DEV_IDEDISK=y
CONFIG_IDEDISK_MULTI_MODE=y
CONFIG_BLK_DEV_IDECD=y
# CONFIG_BLK_DEV_IDETAPE is not set
# CONFIG_BLK_DEV_IDEFLOPPY is not set
CONFIG_BLK_DEV_IDESCSI=m
# CONFIG_IDE_TASK_IOCTL is not set
# CONFIG_IDE_TASKFILE_IO is not set

#
# IDE chipset support/bugfixes
#
CONFIG_IDE_GENERIC=y
# CONFIG_BLK_DEV_CMD640 is not set
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_IDEPCI_SHARE_IRQ=y
# CONFIG_BLK_DEV_OFFBOARD is not set
CONFIG_BLK_DEV_GENERIC=y
# CONFIG_BLK_DEV_OPTI621 is not set
# CONFIG_BLK_DEV_RZ1000 is not set
CONFIG_BLK_DEV_IDEDMA_PCI=y
# CONFIG_BLK_DEV_IDEDMA_FORCED is not set
CONFIG_IDEDMA_PCI_AUTO=y
# CONFIG_IDEDMA_ONLYDISK is not set
# CONFIG_BLK_DEV_AEC62XX is not set
# CONFIG_BLK_DEV_ALI15X3 is not set
# CONFIG_BLK_DEV_AMD74XX is not set
# CONFIG_BLK_DEV_ATIIXP is not set
# CONFIG_BLK_DEV_CMD64X is not set
# CONFIG_BLK_DEV_TRIFLEX is not set
# CONFIG_BLK_DEV_CY82C693 is not set
# CONFIG_BLK_DEV_CS5520 is not set
# CONFIG_BLK_DEV_CS5530 is not set
# CONFIG_BLK_DEV_HPT34X is not set
# CONFIG_BLK_DEV_HPT366 is not set
# CONFIG_BLK_DEV_SC1200 is not set
# CONFIG_BLK_DEV_PIIX is not set
# CONFIG_BLK_DEV_NS87415 is not set
# CONFIG_BLK_DEV_PDC202XX_OLD is not set
# CONFIG_BLK_DEV_PDC202XX_NEW is not set
# CONFIG_BLK_DEV_SVWKS is not set
# CONFIG_BLK_DEV_SIIMAGE is not set
CONFIG_BLK_DEV_SIS5513=y
# CONFIG_BLK_DEV_SLC90E66 is not set
# CONFIG_BLK_DEV_TRM290 is not set
# CONFIG_BLK_DEV_VIA82CXXX is not set
# CONFIG_IDE_ARM is not set
CONFIG_BLK_DEV_IDEDMA=y
# CONFIG_IDEDMA_IVB is not set
CONFIG_IDEDMA_AUTO=y
# CONFIG_BLK_DEV_HD is not set

#
# SCSI device support
#
CONFIG_SCSI=m
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=m
# CONFIG_CHR_DEV_ST is not set
# CONFIG_CHR_DEV_OSST is not set
CONFIG_BLK_DEV_SR=m
# CONFIG_BLK_DEV_SR_VENDOR is not set
CONFIG_CHR_DEV_SG=m

#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs
#
# CONFIG_SCSI_MULTI_LUN is not set
CONFIG_SCSI_CONSTANTS=y
# CONFIG_SCSI_LOGGING is not set

#
# SCSI Transport Attributes
#
# CONFIG_SCSI_SPI_ATTRS is not set
# CONFIG_SCSI_FC_ATTRS is not set

#
# SCSI low-level drivers
#
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
# CONFIG_SCSI_3W_9XXX is not set
# CONFIG_SCSI_ACARD is not set
# CONFIG_SCSI_AACRAID is not set
CONFIG_SCSI_AIC7XXX=m
CONFIG_AIC7XXX_CMDS_PER_DEVICE=32
CONFIG_AIC7XXX_RESET_DELAY_MS=15000
# CONFIG_AIC7XXX_DEBUG_ENABLE is not set
CONFIG_AIC7XXX_DEBUG_MASK=0
# CONFIG_AIC7XXX_REG_PRETTY_PRINT is not set
# CONFIG_SCSI_AIC7XXX_OLD is not set
# CONFIG_SCSI_AIC79XX is not set
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not set
# CONFIG_SCSI_SATA is not set
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_EATA_PIO is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_GDTH is not set
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INITIO is not set
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_SYM53C8XX_2 is not set
# CONFIG_SCSI_IPR is not set
# CONFIG_SCSI_QLOGIC_ISP is not set
# CONFIG_SCSI_QLOGIC_FC is not set
# CONFIG_SCSI_QLOGIC_1280 is not set
# CONFIG_SCSI_QLOGIC_1280_1040 is not set
CONFIG_SCSI_QLA2XXX=m
# CONFIG_SCSI_QLA21XX is not set
# CONFIG_SCSI_QLA22XX is not set
# CONFIG_SCSI_QLA2300 is not set
# CONFIG_SCSI_QLA2322 is not set
# CONFIG_SCSI_QLA6312 is not set
# CONFIG_SCSI_QLA6322 is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_DC390T is not set
# CONFIG_SCSI_NSP32 is not set
# CONFIG_SCSI_DEBUG is not set

#
# Multi-device support (RAID and LVM)
#
# CONFIG_MD is not set

#
# Fusion MPT device support
#
# CONFIG_FUSION is not set

#
# IEEE 1394 (FireWire) support
#
# CONFIG_IEEE1394 is not set

#
# I2O device support
#
# CONFIG_I2O is not set

#
# Networking support
#
CONFIG_NET=y

#
# Networking options
#
CONFIG_PACKET=y
# CONFIG_PACKET_MMAP is not set
CONFIG_NETLINK_DEV=y
CONFIG_UNIX=y
# CONFIG_NET_KEY is not set
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
# CONFIG_IP_MULTIPLE_TABLES is not set
# CONFIG_IP_ROUTE_MULTIPATH is not set
# CONFIG_IP_ROUTE_VERBOSE is not set
# CONFIG_IP_PNP is not set
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE is not set
# CONFIG_IP_MROUTE is not set
# CONFIG_ARPD is not set
# CONFIG_SYN_COOKIES is not set
# CONFIG_INET_AH is not set
# CONFIG_INET_ESP is not set
# CONFIG_INET_IPCOMP is not set
# CONFIG_INET_TUNNEL is not set

#
# IP: Virtual Server Configuration
#
# CONFIG_IP_VS is not set
# CONFIG_IPV6 is not set
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set

#
# IP: Netfilter Configuration
#
CONFIG_IP_NF_CONNTRACK=m
# CONFIG_IP_NF_CT_ACCT is not set
# CONFIG_IP_NF_CT_PROTO_SCTP is not set
# CONFIG_IP_NF_FTP is not set
# CONFIG_IP_NF_IRC is not set
# CONFIG_IP_NF_TFTP is not set
# CONFIG_IP_NF_AMANDA is not set
# CONFIG_IP_NF_QUEUE is not set
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_LIMIT=m
CONFIG_IP_NF_MATCH_IPRANGE=m
CONFIG_IP_NF_MATCH_MAC=m
CONFIG_IP_NF_MATCH_PKTTYPE=m
CONFIG_IP_NF_MATCH_MARK=m
CONFIG_IP_NF_MATCH_MULTIPORT=m
CONFIG_IP_NF_MATCH_TOS=m
CONFIG_IP_NF_MATCH_RECENT=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_DSCP=m
CONFIG_IP_NF_MATCH_AH_ESP=m
CONFIG_IP_NF_MATCH_LENGTH=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_MATCH_TCPMSS=m
CONFIG_IP_NF_MATCH_HELPER=m
CONFIG_IP_NF_MATCH_STATE=m
CONFIG_IP_NF_MATCH_CONNTRACK=m
CONFIG_IP_NF_MATCH_OWNER=m
CONFIG_IP_NF_MATCH_ADDRTYPE=m
CONFIG_IP_NF_MATCH_REALM=m
# CONFIG_IP_NF_MATCH_SCTP is not set
# CONFIG_IP_NF_MATCH_COMMENT is not set
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_LOG=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_IP_NF_TARGET_TCPMSS=m
CONFIG_IP_NF_NAT=m
CONFIG_IP_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_IP_NF_TARGET_SAME=m
# CONFIG_IP_NF_NAT_LOCAL is not set
# CONFIG_IP_NF_NAT_SNMP_BASIC is not set
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_TOS=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_DSCP=m
CONFIG_IP_NF_TARGET_MARK=m
CONFIG_IP_NF_TARGET_CLASSIFY=m
# CONFIG_IP_NF_RAW is not set
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m
# CONFIG_IP_NF_COMPAT_IPCHAINS is not set
# CONFIG_IP_NF_COMPAT_IPFWADM is not set

#
# SCTP Configuration (EXPERIMENTAL)
#
# CONFIG_IP_SCTP is not set
# CONFIG_ATM is not set
# CONFIG_BRIDGE is not set
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_NET_DIVERT is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set
# CONFIG_NET_HW_FLOWCONTROL is not set

#
# QoS and/or fair queueing
#
CONFIG_NET_SCHED=y
CONFIG_NET_SCH_CLK_JIFFIES=y
# CONFIG_NET_SCH_CLK_GETTIMEOFDAY is not set
# CONFIG_NET_SCH_CLK_CPU is not set
CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=m
# CONFIG_NET_SCH_HFSC is not set
CONFIG_NET_SCH_PRIO=m
CONFIG_NET_SCH_RED=m
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_SCH_TEQL=m
CONFIG_NET_SCH_TBF=m
CONFIG_NET_SCH_GRED=m
CONFIG_NET_SCH_DSMARK=m
# CONFIG_NET_SCH_NETEM is not set
CONFIG_NET_SCH_INGRESS=m
CONFIG_NET_QOS=y
CONFIG_NET_ESTIMATOR=y
CONFIG_NET_CLS=y
CONFIG_NET_CLS_TCINDEX=m
CONFIG_NET_CLS_ROUTE4=m
CONFIG_NET_CLS_ROUTE=y
CONFIG_NET_CLS_FW=m
CONFIG_NET_CLS_U32=m
# CONFIG_CLS_U32_PERF is not set
# CONFIG_NET_CLS_IND is not set
CONFIG_NET_CLS_RSVP=m
CONFIG_NET_CLS_RSVP6=m
# CONFIG_NET_CLS_ACT is not set
CONFIG_NET_CLS_POLICE=y

#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_KGDBOE is not set
# CONFIG_NETPOLL is not set
# CONFIG_NETPOLL_RX is not set
# CONFIG_NETPOLL_TRAP is not set
# CONFIG_NET_POLL_CONTROLLER is not set
# CONFIG_HAMRADIO is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
CONFIG_NETDEVICES=y
CONFIG_DUMMY=m
# CONFIG_BONDING is not set
# CONFIG_EQUALIZER is not set
# CONFIG_TUN is not set
# CONFIG_ETHERTAP is not set

#
# ARCnet devices
#
# CONFIG_ARCNET is not set

#
# Ethernet (10 or 100Mbit)
#
CONFIG_NET_ETHERNET=y
CONFIG_MII=m
# CONFIG_HAPPYMEAL is not set
# CONFIG_SUNGEM is not set
# CONFIG_NET_VENDOR_3COM is not set

#
# Tulip family network device support
#
# CONFIG_NET_TULIP is not set
# CONFIG_HP100 is not set
CONFIG_NET_PCI=y
# CONFIG_PCNET32 is not set
# CONFIG_AMD8111_ETH is not set
# CONFIG_ADAPTEC_STARFIRE is not set
# CONFIG_B44 is not set
# CONFIG_FORCEDETH is not set
# CONFIG_DGRS is not set
# CONFIG_EEPRO100 is not set
# CONFIG_E100 is not set
# CONFIG_FEALNX is not set
# CONFIG_NATSEMI is not set
# CONFIG_NE2K_PCI is not set
# CONFIG_8139CP is not set
# CONFIG_8139TOO is not set
CONFIG_SIS900=m
# CONFIG_EPIC100 is not set
# CONFIG_SUNDANCE is not set
# CONFIG_TLAN is not set
# CONFIG_VIA_RHINE is not set

#
# Ethernet (1000 Mbit)
#
# CONFIG_ACENIC is not set
# CONFIG_DL2K is not set
# CONFIG_E1000 is not set
# CONFIG_NS83820 is not set
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
# CONFIG_R8169 is not set
# CONFIG_SK98LIN is not set
# CONFIG_VIA_VELOCITY is not set
# CONFIG_TIGON3 is not set

#
# Ethernet (10000 Mbit)
#
# CONFIG_IXGB is not set
# CONFIG_S2IO is not set

#
# Token Ring devices
#
# CONFIG_TR is not set

#
# Wireless LAN (non-hamradio)
#
# CONFIG_NET_RADIO is not set

#
# Wan interfaces
#
# CONFIG_WAN is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
CONFIG_PPP=m
CONFIG_PPP_MULTILINK=y
CONFIG_PPP_FILTER=y
CONFIG_PPP_ASYNC=m
CONFIG_PPP_SYNC_TTY=m
CONFIG_PPP_DEFLATE=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPPOE=m
# CONFIG_SLIP is not set
# CONFIG_NET_FC is not set
# CONFIG_SHAPER is not set
# CONFIG_NETCONSOLE is not set

#
# ISDN subsystem
#
# CONFIG_ISDN is not set

#
# Telephony Support
#
# CONFIG_PHONE is not set

#
# Input device support
#
CONFIG_INPUT=y

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_INPUT_JOYDEV=m
# CONFIG_INPUT_TSDEV is not set
# CONFIG_INPUT_EVDEV is not set
# CONFIG_INPUT_EVBUG is not set

#
# Input I/O drivers
#
CONFIG_GAMEPORT=m
CONFIG_SOUND_GAMEPORT=m
# CONFIG_GAMEPORT_NS558 is not set
# CONFIG_GAMEPORT_L4 is not set
# CONFIG_GAMEPORT_EMU10K1 is not set
# CONFIG_GAMEPORT_VORTEX is not set
# CONFIG_GAMEPORT_FM801 is not set
# CONFIG_GAMEPORT_CS461x is not set
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
# CONFIG_SERIO_SERPORT is not set
# CONFIG_SERIO_CT82C710 is not set
CONFIG_SERIO_PCIPS2=m
CONFIG_SERIO_LIBPS2=y
# CONFIG_SERIO_RAW is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
# CONFIG_KEYBOARD_NEWTON is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
# CONFIG_MOUSE_SERIAL is not set
# CONFIG_MOUSE_VSXXXAA is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
CONFIG_INPUT_MISC=y
CONFIG_INPUT_PCSPKR=y
# CONFIG_INPUT_UINPUT is not set

#
# Character devices
#
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
# CONFIG_SERIAL_NONSTANDARD is not set

#
# Serial drivers
#
CONFIG_SERIAL_8250=y
# CONFIG_SERIAL_8250_CONSOLE is not set
CONFIG_SERIAL_8250_NR_UARTS=4
# CONFIG_SERIAL_8250_EXTENDED is not set

#
# Non-8250 serial port support
#
CONFIG_SERIAL_CORE=y
CONFIG_UNIX98_PTYS=y
CONFIG_LEGACY_PTYS=y
CONFIG_LEGACY_PTY_COUNT=256

#
# IPMI
#
# CONFIG_IPMI_HANDLER is not set

#
# Watchdog Cards
#
# CONFIG_WATCHDOG is not set
# CONFIG_HW_RANDOM is not set
CONFIG_NVRAM=y
CONFIG_RTC=y
# CONFIG_DTLK is not set
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_SONYPI is not set

#
# Ftape, the floppy tape device driver
#
# CONFIG_FTAPE is not set
CONFIG_AGP=m
# CONFIG_AGP_ALI is not set
# CONFIG_AGP_ATI is not set
# CONFIG_AGP_AMD is not set
# CONFIG_AGP_AMD64 is not set
# CONFIG_AGP_INTEL is not set
# CONFIG_AGP_INTEL_MCH is not set
# CONFIG_AGP_NVIDIA is not set
CONFIG_AGP_SIS=m
# CONFIG_AGP_SWORKS is not set
# CONFIG_AGP_VIA is not set
# CONFIG_AGP_EFFICEON is not set
# CONFIG_DRM is not set
# CONFIG_MWAVE is not set
# CONFIG_RAW_DRIVER is not set
CONFIG_HANGCHECK_TIMER=m

#
# I2C support
#
CONFIG_I2C=m
CONFIG_I2C_CHARDEV=m

#
# I2C Algorithms
#
CONFIG_I2C_ALGOBIT=m
CONFIG_I2C_ALGOPCF=m
# CONFIG_I2C_ALGOPCA is not set

#
# I2C Hardware Bus support
#
CONFIG_I2C_ALI1535=m
CONFIG_I2C_ALI1563=m
CONFIG_I2C_ALI15X3=m
CONFIG_I2C_AMD756=m
CONFIG_I2C_AMD8111=m
CONFIG_I2C_I801=m
CONFIG_I2C_I810=m
CONFIG_I2C_ISA=m
CONFIG_I2C_NFORCE2=m
CONFIG_I2C_PARPORT_LIGHT=m
CONFIG_I2C_PIIX4=m
CONFIG_I2C_PROSAVAGE=m
CONFIG_I2C_SAVAGE4=m
CONFIG_SCx200_ACB=m
CONFIG_I2C_SIS5595=m
CONFIG_I2C_SIS630=m
CONFIG_I2C_SIS96X=m
# CONFIG_I2C_STUB is not set
CONFIG_I2C_VIA=m
CONFIG_I2C_VIAPRO=m
CONFIG_I2C_VOODOO3=m
# CONFIG_I2C_PCA_ISA is not set

#
# Hardware Sensors Chip support
#
CONFIG_I2C_SENSOR=m
CONFIG_SENSORS_ADM1021=m
CONFIG_SENSORS_ADM1025=m
CONFIG_SENSORS_ADM1031=m
CONFIG_SENSORS_ASB100=m
CONFIG_SENSORS_DS1621=m
CONFIG_SENSORS_FSCHER=m
CONFIG_SENSORS_GL518SM=m
CONFIG_SENSORS_IT87=m
CONFIG_SENSORS_LM75=m
CONFIG_SENSORS_LM77=m
CONFIG_SENSORS_LM78=m
CONFIG_SENSORS_LM80=m
CONFIG_SENSORS_LM83=m
CONFIG_SENSORS_LM85=m
# CONFIG_SENSORS_LM87 is not set
CONFIG_SENSORS_LM90=m
CONFIG_SENSORS_MAX1619=m
# CONFIG_SENSORS_SMSC47M1 is not set
CONFIG_SENSORS_VIA686A=m
CONFIG_SENSORS_W83781D=m
CONFIG_SENSORS_W83L785TS=m
CONFIG_SENSORS_W83627HF=m

#
# Other I2C Chip support
#
CONFIG_SENSORS_EEPROM=m
CONFIG_SENSORS_PCF8574=m
CONFIG_SENSORS_PCF8591=m
CONFIG_SENSORS_RTC8564=m
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# CONFIG_I2C_DEBUG_CHIP is not set

#
# Dallas's 1-wire bus
#
# CONFIG_W1 is not set

#
# Misc devices
#
# CONFIG_IBM_ASM is not set

#
# Multimedia devices
#
# CONFIG_VIDEO_DEV is not set

#
# Digital Video Broadcasting Devices
#
# CONFIG_DVB is not set

#
# Graphics support
#
# CONFIG_FB is not set
CONFIG_VIDEO_SELECT=y

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
CONFIG_DUMMY_CONSOLE=y

#
# Sound
#
CONFIG_SOUND=m

#
# Advanced Linux Sound Architecture
#
CONFIG_SND=m
CONFIG_SND_TIMER=m
CONFIG_SND_PCM=m
CONFIG_SND_RAWMIDI=m
CONFIG_SND_SEQUENCER=m
CONFIG_SND_SEQ_DUMMY=m
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=m
CONFIG_SND_PCM_OSS=m
CONFIG_SND_SEQUENCER_OSS=y
CONFIG_SND_RTCTIMER=m
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set

#
# Generic devices
#
CONFIG_SND_DUMMY=m
CONFIG_SND_VIRMIDI=m
# CONFIG_SND_MTPAV is not set
# CONFIG_SND_SERIAL_U16550 is not set
# CONFIG_SND_MPU401 is not set

#
# PCI devices
#
CONFIG_SND_AC97_CODEC=m
# CONFIG_SND_ALI5451 is not set
# CONFIG_SND_ATIIXP is not set
# CONFIG_SND_ATIIXP_MODEM is not set
# CONFIG_SND_AU8810 is not set
# CONFIG_SND_AU8820 is not set
# CONFIG_SND_AU8830 is not set
# CONFIG_SND_AZT3328 is not set
# CONFIG_SND_BT87X is not set
CONFIG_SND_CS46XX=m
CONFIG_SND_CS46XX_NEW_DSP=y
# CONFIG_SND_CS4281 is not set
# CONFIG_SND_EMU10K1 is not set
# CONFIG_SND_KORG1212 is not set
# CONFIG_SND_MIXART is not set
# CONFIG_SND_NM256 is not set
# CONFIG_SND_RME32 is not set
# CONFIG_SND_RME96 is not set
# CONFIG_SND_RME9652 is not set
# CONFIG_SND_HDSP is not set
# CONFIG_SND_TRIDENT is not set
# CONFIG_SND_YMFPCI is not set
# CONFIG_SND_ALS4000 is not set
# CONFIG_SND_CMIPCI is not set
# CONFIG_SND_ENS1370 is not set
# CONFIG_SND_ENS1371 is not set
# CONFIG_SND_ES1938 is not set
# CONFIG_SND_ES1968 is not set
# CONFIG_SND_MAESTRO3 is not set
# CONFIG_SND_FM801 is not set
# CONFIG_SND_ICE1712 is not set
# CONFIG_SND_ICE1724 is not set
CONFIG_SND_INTEL8X0=m
# CONFIG_SND_INTEL8X0M is not set
# CONFIG_SND_SONICVIBES is not set
# CONFIG_SND_VIA82XX is not set
# CONFIG_SND_VX222 is not set

#
# Open Sound System
#
# CONFIG_SOUND_PRIME is not set

#
# USB support
#
# CONFIG_USB is not set
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y

#
# USB Gadget Support
#
# CONFIG_USB_GADGET is not set

#
# File systems
#
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
# CONFIG_EXT2_FS_POSIX_ACL is not set
# CONFIG_EXT2_FS_SECURITY is not set
CONFIG_EXT3_FS=y
CONFIG_EXT3_FS_XATTR=y
# CONFIG_EXT3_FS_POSIX_ACL is not set
# CONFIG_EXT3_FS_SECURITY is not set
CONFIG_JBD=y
# CONFIG_JBD_DEBUG is not set
CONFIG_FS_MBCACHE=y
# CONFIG_REISER4_FS is not set
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
# CONFIG_XFS_FS is not set
# CONFIG_MINIX_FS is not set
CONFIG_ROMFS_FS=y
# CONFIG_QUOTA is not set
# CONFIG_AUTOFS_FS is not set
CONFIG_AUTOFS4_FS=y

#
# Caches
#
# CONFIG_FSCACHE is not set

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_ZISOFS_FS=y
CONFIG_UDF_FS=y
CONFIG_UDF_NLS=y

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
CONFIG_NTFS_FS=m
# CONFIG_NTFS_DEBUG is not set
# CONFIG_NTFS_RW is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_SYSFS=y
# CONFIG_DEVFS_FS is not set
CONFIG_DEVPTS_FS_XATTR=y
# CONFIG_DEVPTS_FS_SECURITY is not set
CONFIG_TMPFS=y
# CONFIG_TMPFS_XATTR is not set
# CONFIG_HUGETLBFS is not set
# CONFIG_HUGETLB_PAGE is not set
CONFIG_RAMFS=y

#
# Miscellaneous filesystems
#
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
CONFIG_CRAMFS=m
# CONFIG_VXFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set

#
# Network File Systems
#
# CONFIG_NFS_FS is not set
# CONFIG_NFSD is not set
# CONFIG_EXPORTFS is not set
# CONFIG_SMB_FS is not set
# CONFIG_CIFS is not set
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
CONFIG_AFS_FS=m
CONFIG_RXRPC=m

#
# Partition Types
#
# CONFIG_PARTITION_ADVANCED is not set
CONFIG_MSDOS_PARTITION=y

#
# Native Language Support
#
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=y
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
CONFIG_NLS_CODEPAGE_850=y
# CONFIG_NLS_CODEPAGE_852 is not set
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
CONFIG_NLS_CODEPAGE_1250=y
# CONFIG_NLS_CODEPAGE_1251 is not set
# CONFIG_NLS_ASCII is not set
CONFIG_NLS_ISO8859_1=y
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
CONFIG_NLS_ISO8859_15=y
# CONFIG_NLS_KOI8_R is not set
# CONFIG_NLS_KOI8_U is not set
# CONFIG_NLS_UTF8 is not set

#
# Profiling support
#
# CONFIG_PROFILING is not set

#
# Kernel hacking
#
CONFIG_DEBUG_KERNEL=y
CONFIG_MAGIC_SYSRQ=y
# CONFIG_SCHEDSTATS is not set
# CONFIG_DEBUG_SLAB is not set
# CONFIG_DEBUG_KOBJECT is not set
# CONFIG_DEBUG_PREEMPT is not set
# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
# CONFIG_PREEMPT_TIMING is not set
# CONFIG_RWSEM_DEADLOCK_DETECT is not set
# CONFIG_DEBUG_INFO is not set
CONFIG_FRAME_POINTER=y
CONFIG_EARLY_PRINTK=y
# CONFIG_DEBUG_STACKOVERFLOW is not set
# CONFIG_KPROBES is not set
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_PAGEALLOC is not set
# CONFIG_4KSTACKS is not set
# CONFIG_KGDB is not set

#
# Security options
#
# CONFIG_KEYS is not set
CONFIG_SECURITY=y
# CONFIG_SECURITY_NETWORK is not set
CONFIG_SECURITY_CAPABILITIES=m
# CONFIG_SECURITY_SECLVL is not set
# CONFIG_SECURITY_SELINUX is not set

#
# Cryptographic options
#
CONFIG_CRYPTO=y
# CONFIG_CRYPTO_HMAC is not set
# CONFIG_CRYPTO_NULL is not set
# CONFIG_CRYPTO_MD4 is not set
# CONFIG_CRYPTO_MD5 is not set
# CONFIG_CRYPTO_SHA1 is not set
# CONFIG_CRYPTO_SHA256 is not set
# CONFIG_CRYPTO_SHA512 is not set
# CONFIG_CRYPTO_WP512 is not set
# CONFIG_CRYPTO_DES is not set
# CONFIG_CRYPTO_BLOWFISH is not set
# CONFIG_CRYPTO_TWOFISH is not set
# CONFIG_CRYPTO_SERPENT is not set
# CONFIG_CRYPTO_AES_586 is not set
# CONFIG_CRYPTO_CAST5 is not set
# CONFIG_CRYPTO_CAST6 is not set
# CONFIG_CRYPTO_TEA is not set
# CONFIG_CRYPTO_ARC4 is not set
# CONFIG_CRYPTO_KHAZAD is not set
# CONFIG_CRYPTO_DEFLATE is not set
# CONFIG_CRYPTO_MICHAEL_MIC is not set
# CONFIG_CRYPTO_CRC32C is not set
# CONFIG_CRYPTO_TEST is not set

#
# Library routines
#
CONFIG_CRC_CCITT=m
CONFIG_CRC32=m
CONFIG_LIBCRC32C=m
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=m
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_PC=y

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Ingo Molnar]

* Florian Schmidt <mista.tapas@gmx.net> wrote:

> Here's some setup info:
> 
> ksoftirqd/0:
> mango:~# chrt -p 2
> pid 2's current scheduling policy: SCHED_FIFO
> pid 2's current scheduling priority: 99

dont do this ... ksoftirqd can spend alot of time processing various
stuff and it should not be relevant to the audio path. It should be
SCHED_OTHER.

> Hmm, it seems i haven't disabled all debugging. This is from dmesg:
> 
> BUG: atomic counter underflow at:
>  [<c010649e>] dump_stack+0x1e/0x20 (20)
>  [<c025f319>] qdisc_destroy+0xd9/0xe0 (28)

this is automatic and doesnt introduce alot of overhead (unless the
printout happens while you are testing latencies). You can remove the
WARN_ON from include/asm-i386/atomic.h.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Florian Schmidt]

On Fri, 29 Oct 2004 16:25:38 +0200
Ingo Molnar <mingo@elte.hu> wrote:

> > ksoftirqd/0:
> > mango:~# chrt -p 2
> > pid 2's current scheduling policy: SCHED_FIFO
> > pid 2's current scheduling priority: 99
> 
> dont do this ... ksoftirqd can spend alot of time processing various
> stuff and it should not be relevant to the audio path. It should be
> SCHED_OTHER.

ah ok, i was wondering about this.. i saw it in rui's setup [SCHED_FIFO with
high prio]. Doesn't seem to make a difference though on first sight. still
xruns plenty above 1ms. 

playback only or rmmod'ing the network adapter driver or using the dummy
soundcard driver instead of snd-cs46xx doesn't make a difference either.

after rmmoding the network card driver i saw:

sis900 0000:00:03.0: Device was removed without properly calling
pci_disable_device(). This may need fixing.

in m dmesg. rmmod'ing snd-cs46xx gives me:

Sound Fusion CS46xx 0000:00:0f.0: Device was removed without properly
calling pci_disable_device(). This may need fixing.

too, so maybe i have already hit a BUG in the kernel and this screwed up all
further test results.

Will build a kernel with debugging stuff to see what's up. I'll also build a
VP only version to see if i still get these pci_disable_device() messages
with a "more vanilla" kernel ;)

flo

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Florian Schmidt]

On Fri, 29 Oct 2004 16:31:35 +0200
Florian Schmidt <mista.tapas@gmx.net> wrote:

> i tried a V0.5.2 with PREEMPT_REALTIME and all debugging off (config
> attached). I cannot reproduce your results. I have experienced around 30
> xruns in 10 minutes. And big ones, too (> 5ms). I don't know exactly what
> kind of load triggers them. Here's a bit of qjackctl message window (btw:
> jackd was idle, no clients connected, except for qjackctl):
> 
[snip]

i forgot to mention though that i do use jack in full duplex mode and with a
periodsize of 64:

/usr/bin/jackd -R -P60 -t20000 -dalsa -dhw:0 -r48000 -p64 -n2

The results are thus not really comparable. Anyways, 7ms xruns still
shouldn't happen (with the older VP patches i could run 32 frames full
duplex for hours w/o xruns).

flo

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Lee Revell]

On Fri, 2004-10-29 at 16:31 +0200, Florian Schmidt wrote:
> i tried a V0.5.2 with PREEMPT_REALTIME and all debugging off (config
> attached). I cannot reproduce your results. I have experienced around 30
> xruns in 10 minutes. And big ones, too (> 5ms). I don't know exactly what
> kind of load triggers them. Here's a bit of qjackctl message window (btw:
> jackd was idle, no clients connected, except for qjackctl):
> 

I am seeing the same behavior, about 30 xruns in 10 minutes.  It seems
to be triggered by display activity, among other things.  This cannot be
a jackd issue, because with an earlier version (T3) I can run for 24
hours without a single xrun.

There has to be a bug somewhere in the RT preempt patch.

Lee

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Joe]

On Fri, 29 Oct 2004 23:09:24 -0400, Lee Revell <rlrevell@joe-job.com> wrote:
> On Fri, 2004-10-29 at 16:31 +0200, Florian Schmidt wrote:
> > i tried a V0.5.2 with PREEMPT_REALTIME and all debugging off (config
> > attached). I cannot reproduce your results. I have experienced around 30
> > xruns in 10 minutes. And big ones, too (> 5ms). I don't know exactly what
> > kind of load triggers them. Here's a bit of qjackctl message window (btw:
> > jackd was idle, no clients connected, except for qjackctl):
> >
> 
> I am seeing the same behavior, about 30 xruns in 10 minutes.  It seems
> to be triggered by display activity, among other things.  This cannot be
> a jackd issue, because with an earlier version (T3) I can run for 24
> hours without a single xrun.
> 
> There has to be a bug somewhere in the RT preempt patch.
> 
> Lee

There is, this has been a well-known issue with many versions of
real-time voluntary preemption, which is also a main reason as to why
I avoid it, voluntary preemption performs flawlessly however RT has
been horrendous.  Hopefully the bugs will get smoothed out.

[patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Ingo Molnar]

i have released the -V0.4 Real-Time Preemption patch, which can be
downloaded from:

   http://redhat.com/~mingo/realtime-preempt/

this is a fixes-only release, but still experimental.

this release should fix more bugs of the 'slowdown' and 'interactivity
problems' variety.

To debug the wakeup anomalies reported i've implemented a new variant of
the latency tracer, which now traces 'wakeup latencies' too - i.e. it
measures and traces maximum delays observed from the point of wakeup to
the point of the really starting to execute. Only the highest-priority
runnable task in the system is traced at a time, but this should be more
than enough to find the high latency scheduling paths.

This new tracing mode can be enabled by compiling a LATENCY_TRACE kernel
as usual and doing:

	echo 4 > /proc/sys/kernel/trace_enabled

then to start tracing, reset the current max latency value via e.g.:

	echo 10 > /proc/sys/kernel/preempt_max_latency

then the kernel should signal wakeup latency events in the syslog:

  (sshd/3093/CPU#0): new 18 us maximum-latency wakeup.
  (sshd/3093/CPU#0): new 19 us maximum-latency wakeup.
  (hackbench/3818/CPU#0): new 20 us maximum-latency wakeup.
  (hackbench/3762/CPU#0): new 21 us maximum-latency wakeup.
  (hackbench/3814/CPU#0): new 22 us maximum-latency wakeup.
  (ksoftirqd/0/3/CPU#0): new 35 us maximum-latency wakeup.

the latency trace of the last (and highest) event can always be found in
/proc/latency_trace, as usual. Note that the trace output is a bit
different in the wakeup-tracing case.

NOTE: the tracer works on SMP too, but since on SMP tasks can switch
from one CPU to another a given trace can be less useful if the delay
happened on another CPU.

using this wakeup tracer i found and fixed a couple of 'missed
preemption check' bugs - all introduced by PREEMPT_REALTIME in the -U/-V
timeframe. So if you had latency/interactivity problems please re-check
-V0.4.

the wakeup tracer is nice in the sense of that it traces actual, realy.

Changes since -V0.3.2:

 - fixed the rtc_lock related crash reported by K.R. Foley and Robert 
   Crocombe.

 - fixed missing preemption checks in rwsem-generic.c

 - fixed missing preemption check in schedule_tail() [==new task wakeup]

 - implemented wakeup-latency tracer

to create a -V0.4 tree from scratch, the patching order is:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
 + http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9/2.6.9-mm1/2.6.9-mm1.bz2
 + http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-mm1-V0.4

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Magnus Naeslund(t)]

I'm testing out this patch on an debian box.
There seems to be a problem with enable_irq in the e100 driver that 
makes the network to b0rk.

What information do you need to get something useful out of this?
I saw that others have this problem, so I've got an serial console to 
the box, if you want me to do any tests, tell me how.


Regards,
Magnus


Setting up IP spoofing protection: rp_filter.
Configuring network interfaces: ifconfig/924: BUG in enable_irq at 
kernel/irq/manage.c:112
  [<c01362a0>] enable_irq+0xe4/0x12c (8)
  [<d08a44f6>] e100_up+0x119/0x224 [e100] (44)
  [<d08a5743>] e100_open+0x2c/0x84 [e100] (44)
  [<c0237dda>] dev_open+0x76/0x85 (28)
  [<c02394a8>] dev_change_flags+0x5d/0x138 (24)
  [<c0237cbd>] dev_load+0x31/0x6c (12)
  [<c027915c>] devinet_ioctl+0x5f9/0x6c6 (20)
  [<c027b1b8>] inet_ioctl+0xc7/0xd3 (104)
  [<c022ea5e>] sock_ioctl+0x19f/0x26a (24)
  [<c016c897>] sys_ioctl+0x1e8/0x249 (28)
  [<c0113eea>] do_page_fault+0x0/0x5ee (24)
  [<c0105bb3>] syscall_call+0x7/0xb (16)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: enable_irq+0x31/0x12c [<c01361ed>] / (0x0 [<00000000>])
.. entry 2: print_traces+0x14/0x47 [<c0130201>] / (0x0 [<00000000>])

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Ingo Molnar]

* Magnus Naeslund(t) <mag@fbab.net> wrote:

> I'm testing out this patch on an debian box. There seems to be a
> problem with enable_irq in the e100 driver that makes the network to
> b0rk.

this e100 driver warning seems mostly harmless - i get it too and the
device works just fine.

> What information do you need to get something useful out of this? I
> saw that others have this problem, so I've got an serial console to
> the box, if you want me to do any tests, tell me how.

even just using it and reporting any potential breakages you get during
bootup or normal use would be very useful. I'd suggest to initially
enable all the relevant debugging options:

 CONFIG_DEBUG_PREEMPT=y
 CONFIG_PREEMPT_TIMING=y
 CONFIG_PREEMPT_TRACE=y
 CONFIG_LATENCY_TRACE=y
 CONFIG_MCOUNT=y
 CONFIG_RWSEM_DEADLOCK_DETECT=y
 CONFIG_RWSEM_MAX_OWNERS=32
 CONFIG_DEBUG_INFO=y
 CONFIG_EARLY_PRINTK=y

this will slow the kernel down but in case of problems there is a much
higher chance of getting a useful assert on the serial console and not
some silent lockup.

if things look good for normal use then a bit advanced type of testing
would be to enable the wakeup-latency tracer:

  echo 4 > /proc/sys/kernel/tracing_enabled
  echo 5 > /proc/sys/kernel/preempt_max_latency

this will measure the highest wakeup latency of highprio tasks, starting
at 5 microseconds. You'll get a short 1-line notification of the latest
latency in the syslog, and the latest trace will always be available in
/proc/latency_trace. Depending on the speed of the system, 'larger'
latencies should be reported to me. 'larger' means more than 20 usecs on
a 2 GHz box or more than 40 usecs on a 1 GHz box. (Dont worry about
reporting duplicates, i can skip them quickly.)

then if things are still looking good (i'm not betting on it though :-),
you could try various stesstests, running LTP's:

	./runalltests.sh -x 20

and things like that. If you have a stable system then it would also be
nice to try to trigger as high wakeup-latencies as possible. E.g. run a
couple of thousand tasks on it, or try to start as many mozilla
instances, or make it go into heavy swapping. I.e. if the core is ok,
explore the edges a bit. The kernel is supposed to offer very low
(wakeup-) latencies no matter what the load. [this doesnt mean your
system will necessarily feel fast - if it's running lots of tasks, even
if the highest prio one is woken up quickly, then it's gonna be slow.]

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Magnus Naeslund(t)]

Ingo Molnar wrote:
> * Magnus Naeslund(t) <mag@fbab.net> wrote:
> 
> 
>>I'm testing out this patch on an debian box. There seems to be a
>>problem with enable_irq in the e100 driver that makes the network to
>>b0rk.
> 
> 
> this e100 driver warning seems mostly harmless - i get it too and the
> device works just fine.
> 
> 

Well, this isn't my experience.
I can't even ping another box on the same net with e100 or eepro100 
driver. I'm running a UP p4 2.4 ghz box with 2.6.9-mm1-RT-V0.4.3.

After letting it ping a while, this turns up:

NETDEV WATCHDOG: eth0: transmit timed out
ksoftirqd/0/2: BUG in enable_irq at kernel/irq/manage.c:112
  [<c01384d9>] enable_irq+0xe1/0x122 (12)
  [<d08055ab>] e100_up+0x112/0x211 [e100] (48)
  [<c0247e37>] dev_watchdog+0x0/0xb6 (36)
  [<c0247eeb>] dev_watchdog+0xb4/0xb6 (12)
  [<c0124391>] run_timer_softirq+0x1c9/0x48b (20)
  [<c0111ac0>] mcount+0x14/0x18 (32)
  [<c012018e>] ___do_softirq+0x4e/0xd6 (20)
  [<c01202a0>] _do_softirq+0x8/0x22 (8)
  [<c012066a>] ksoftirqd+0xa5/0xeb (4)
  [<c01202b8>] _do_softirq+0x20/0x22 (28)
  [<c012066a>] ksoftirqd+0xa5/0xeb (8)
  [<c013015e>] kthread+0xa1/0xce (28)
  [<c01205c5>] ksoftirqd+0x0/0xeb (20)
  [<c01300bd>] kthread+0x0/0xce (12)
  [<c0104195>] kernel_thread_helper+0x5/0xb (16)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: enable_irq+0x33/0x122 [<c013842b>] / (e100_up+0x112/0x211 
[e100] [<d08055ab>])
.. entry 2: print_traces+0x1b/0x55 [<c013243e>] / (dump_stack+0x23/0x25 
[<c0106eb8>])




Other latecy traces are:

(ksoftirqd/0/2/CPU#0): new 22 us maximum-latency critical section.
  => started at timestamp 116357664: <call_console_drivers+0x8c/0x163>
  =>   ended at timestamp 116357686: <__sched_text_start+0x30e/0x6c1>
  [<c0131d71>] sub_preempt_count+0x62/0xc5 (4)
  [<c01319e9>] check_preempt_timing+0x20b/0x2da (8)
  [<c0289ece>] __sched_text_start+0x30e/0x6c1 (8)
  [<c0106a10>] common_interrupt+0x18/0x20 (24)
  [<c013007b>] kthread_exit_files+0x14/0x56 (32)
  [<c0131d71>] sub_preempt_count+0x62/0xc5 (20)
  [<c0289ece>] __sched_text_start+0x30e/0x6c1 (8)
  [<c0289ece>] __sched_text_start+0x30e/0x6c1 (20)
  [<c028a2aa>] schedule+0x29/0xd1 (84)
  [<c0111ac0>] mcount+0x14/0x18 (8)
  [<c01206ae>] ksoftirqd+0xe9/0xeb (28)
  [<c013015e>] kthread+0xa1/0xce (28)
  [<c01205c5>] ksoftirqd+0x0/0xeb (20)
  [<c01300bd>] kthread+0x0/0xce (12)
  [<c0104195>] kernel_thread_helper+0x5/0xb (16)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: __sched_text_start+0x4e/0x6c1 [<c0289c0e>] / 
(schedule+0x29/0xd1 [<c028a2aa>])
.. entry 2: print_traces+0x1b/0x55 [<c013243e>] / (dump_stack+0x23/0x25 
[<c0106eb8>])

  =>   dump-end timestamp 116461059
(bash/1156/CPU#0): new 16 us maximum-latency critical section.
  => started at timestamp 227980588: <call_console_drivers+0x8c/0x163>
  =>   ended at timestamp 227980604: <irq_exit+0x3c/0x45>
  [<c0131d71>] sub_preempt_count+0x62/0xc5 (4)
  [<c01319e9>] check_preempt_timing+0x20b/0x2da (8)
  [<c01380a1>] irq_exit+0x3c/0x45 (8)
  [<c0115f81>] try_to_wake_up+0x135/0x137 (28)
  [<c0131d71>] sub_preempt_count+0x62/0xc5 (48)
  [<c01380a1>] irq_exit+0x3c/0x45 (8)
  [<c01380a1>] irq_exit+0x3c/0x45 (20)
  [<c0108784>] do_IRQ+0x5c/0x81 (12)
  [<c0106a10>] common_interrupt+0x18/0x20 (20)
  [<c0114ab3>] do_page_fault+0x8e/0x5e7 (44)
  [<c0131d71>] sub_preempt_count+0x62/0xc5 (72)
  [<c013183a>] check_preempt_timing+0x5c/0x2da (16)
  [<c0131d71>] sub_preempt_count+0x62/0xc5 (4)
  [<c01163ff>] schedule_tail+0x37/0x8f (4)
  [<c0114a25>] do_page_fault+0x0/0x5e7 (36)
  [<c0106acd>] error_code+0x2d/0x38 (8)
  [<c011007b>] set_fixed_ranges+0x62/0xd1 (40)
  [<c0116447>] schedule_tail+0x7f/0x8f (12)
  [<c0114a25>] do_page_fault+0x0/0x5e7 (20)
  [<c0106acd>] error_code+0x2d/0x38 (8)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: __do_IRQ+0x104/0x196 [<c01382a6>] / (do_IRQ+0x57/0x81 
[<c010877f>])
.. entry 2: print_traces+0x1b/0x55 [<c013243e>] / (dump_stack+0x23/0x25 
[<c0106eb8>])

  =>   dump-end timestamp 228101445

(ksoftirqd/0/2/CPU#0): new 18 us maximum-latency critical section.
  => started at timestamp 228104930: <call_console_drivers+0x8c/0x163>
  =>   ended at timestamp 228104948: <__sched_text_start+0x30e/0x6c1>
  [<c0131d71>] sub_preempt_count+0x62/0xc5 (4)
  [<c01319e9>] check_preempt_timing+0x20b/0x2da (8)
  [<c0289ece>] __sched_text_start+0x30e/0x6c1 (8)
  [<c0106a10>] common_interrupt+0x18/0x20 (24)
  [<c013007b>] kthread_exit_files+0x14/0x56 (32)
  [<c0131d71>] sub_preempt_count+0x62/0xc5 (20)
  [<c0289ece>] __sched_text_start+0x30e/0x6c1 (8)
  [<c0289ece>] __sched_text_start+0x30e/0x6c1 (20)
  [<c028a2aa>] schedule+0x29/0xd1 (84)
  [<c0111ac0>] mcount+0x14/0x18 (8)
  [<c01206ae>] ksoftirqd+0xe9/0xeb (28)
  [<c013015e>] kthread+0xa1/0xce (28)
  [<c01205c5>] ksoftirqd+0x0/0xeb (20)
  [<c01300bd>] kthread+0x0/0xce (12)
  [<c0104195>] kernel_thread_helper+0x5/0xb (16)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: __sched_text_start+0x4e/0x6c1 [<c0289c0e>] / 
(schedule+0x29/0xd1 [<c028a2aa>])
.. entry 2: print_traces+0x1b/0x55 [<c013243e>] / (dump_stack+0x23/0x25 
[<c0106eb8>])

  =>   dump-end timestamp 228208260

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Ingo Molnar]

* Magnus Naeslund(t) <mag@fbab.net> wrote:

> What information do you need to get something useful out of this? I
> saw that others have this problem, so I've got an serial console to
> the box, if you want me to do any tests, tell me how.

also, if you hit problems make sure you have the latest patch, i
sometimes upload a small update with a trivial fix without announcing it
(or the announcement lags on lkml), and i upload larger changes roughly
daily. The current latest version is RT-V0.4.3.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[K.R. Foley]

Ingo Molnar wrote:
> i have released the -V0.4 Real-Time Preemption patch, which can be
> downloaded from:
> 
>    http://redhat.com/~mingo/realtime-preempt/
> 

I have been having problems on my UP system at home with all of the more 
recent patches (since U10.X). Some would boot and then the networking 
was severely busted (slowdowns, hangs, etc.), some would not even boot. 
V0.4.3 was of the no boot variety. Just for grins I disabled kudzu, and 
the thing boots fine with no networking or other problems. This very 
well may have been a fluke, but I have successfully booted this kernel 
twice now. It did hang on a reboot at the point when it should have been 
doing the actual reboot and I had to press the button. I didn't have 
time this morning to turn kudzu back on to see if was just a fluke that 
it didn't boot the first time. Not sure what, if anything, this means, 
but V0.4.3 is running very nicely on my UP system with no lag or 
noticeable problems.

kr

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4

[Ingo Molnar]

* K.R. Foley <kr@cybsft.com> wrote:

> I have been having problems on my UP system at home with all of the
> more recent patches (since U10.X). Some would boot and then the
> networking was severely busted (slowdowns, hangs, etc.), some would
> not even boot.  V0.4.3 was of the no boot variety. Just for grins I
> disabled kudzu, and the thing boots fine with no networking or other
> problems. This very well may have been a fluke, but I have
> successfully booted this kernel twice now. It did hang on a reboot at
> the point when it should have been doing the actual reboot and I had
> to press the button. I didn't have time this morning to turn kudzu
> back on to see if was just a fluke that it didn't boot the first time.
> Not sure what, if anything, this means, but V0.4.3 is running very
> nicely on my UP system with no lag or noticeable problems.

just to make sure - could try to run kudzu manually after bootup and
observe what happens? Do you have a udev based system? I recently
corrupted my udev database via a crash and had to remove the
/dev/.udev.tdb file and had to regenerate it via 'udevstart'. (be
careful doing that though, it might mess up your system.) The symtoms
were a hung kudzu - while in reality it 'hung' because udev and udevinfo
processes looped in userspace forever. Weirdly, the stock Fedora kernel
didnt hang in this same phase, so there might still be a
PREEMPT_REALTIME bug here.

	Ingo