Re: [patch] voluntary-preempt-2.6.8.1-P2

Re: [patch] voluntary-preempt-2.6.8.1-P2

[jjluza]

It fails here at compile time with :

arch/i386/kernel/built-in.o(.text+0x2fc5): In function `do_nmi':
: undefined reference to `__trace'
arch/i386/kernel/built-in.o(.text+0x3723): In function `do_IRQ':
: undefined reference to `__trace'
arch/i386/mm/built-in.o(.text+0x7ba): In function `do_page_fault':
: undefined reference to `__trace'
make[1]: *** [vmlinux] Erreur 1
make[1]: Leaving directory `/usr/src/linux-2.6.8'
make: *** [stamp-build] Erreur 2


I also got offset when applying the patch. (P1 hadn't this problem)


Regards.

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Peter Zijlstra]

[-- Attachment #1: Type: text/plain, Size: 917 bytes --]

On Mon, 2004-08-16 at 14:52 +0200, jjluza wrote:
> It fails here at compile time with :
> 
> arch/i386/kernel/built-in.o(.text+0x2fc5): In function `do_nmi':
> : undefined reference to `__trace'
> arch/i386/kernel/built-in.o(.text+0x3723): In function `do_IRQ':
> : undefined reference to `__trace'
> arch/i386/mm/built-in.o(.text+0x7ba): In function `do_page_fault':
> : undefined reference to `__trace'
> make[1]: *** [vmlinux] Erreur 1
> make[1]: Leaving directory `/usr/src/linux-2.6.8'
> make: *** [stamp-build] Erreur 2
> 
> 
> I also got offset when applying the patch. (P1 hadn't this problem)
> 
> 
> Regards.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
-- 
Peter Zijlstra <a.p.zijlstra@chello.nl>

[-- Attachment #2: compile-fix-P2 --]
[-- Type: text/plain, Size: 1023 bytes --]

--- arch/i386/kernel/irq.c~	2004-08-16 20:01:58.835999719 +0200
+++ arch/i386/kernel/irq.c	2004-08-16 20:26:24.974943041 +0200
@@ -219,7 +219,9 @@
 	unsigned int status;
 
 	irq_enter();
+#ifdef CONFIG_LATENCY_TRACE
 	__trace((unsigned long)do_IRQ, regs.eip);
+#endif
 
 #ifdef CONFIG_DEBUG_STACKOVERFLOW
 	/* Debugging check for stack overflow: is there less than 1KB free? */
--- arch/i386/kernel/traps.c~	2004-08-16 20:01:58.836999745 +0200
+++ arch/i386/kernel/traps.c	2004-08-16 20:26:48.156378051 +0200
@@ -537,7 +537,9 @@
 	int cpu;
 
 	nmi_enter();
+#ifdef CONFIG_LATENCY_TRACE
 	__trace((unsigned long)do_nmi, regs->eip);
+#endif
 
 	cpu = smp_processor_id();
 	++nmi_count(cpu);
--- arch/i386/mm/fault.c~	2004-08-16 20:01:58.837999771 +0200
+++ arch/i386/mm/fault.c	2004-08-16 20:27:08.181753770 +0200
@@ -223,7 +223,9 @@
 	int write;
 	siginfo_t info;
 
+#ifdef CONFIG_LATENCY_TRACE
 	__trace((unsigned long)do_page_fault, regs->eip);
+#endif
 
 	/* get the address */
 	__asm__("movl %%cr2,%0":"=r" (address));

Re: [patch] voluntary-preempt-2.6.8.1-P0

[Ingo Molnar]

* Lee Revell <rlrevell@joe-job.com> wrote:

> There are a number of samples above 700us.  I am working with a period
> time of 666 usecs, and since there are 2 periods per buffer, we would
> have to hit two > 666 usec latencies in a row for an xrun - it appears
> that there are many individual latencies above 666, certainly more
> than there are xruns.  So, maybe the mlockall issue is not a result of
> triggering a single large latency, but of increasing the frequency of
> these higher latencies so that we are more likely to hit 2 in a row.

hm, it seems the mlockall() issue is too deterministic for it to be a
statistical-only phenomenon. Also, isnt that xrun on the order of 15
msecs? That's way too big too.

> IIRC ksoftirqd will defer more work under load, and ksoftirqd is one
> of the more common offenders to hit the extract_entropy latency. 
> Maybe mlockall causes more softirqs to be deferred, thus increaing the
> change that we will have to do more than 666 usecs worth of work on 2
> successive wakeups.

there should be no relation between softirqs and mlockall().

this is truly a mind-boggling latency. mlockall() is fully preemptible. 
All it does is to pre-fault the whole range of pages that a process has.

could you try another thing: modify mlockall-test.cc to use mlock() on
the freshly allocated anonymous pages? Does this produce the same
latencies? mlockall() prefaults _all_ pages the process currently has. 
Maybe mlockall() touches some page that is mapped both by jackd and
mlockall-test and thus somehow interacts with jackd's scheduling.

the anonymous pages themselves can have no IPC-alike connection to any
page jackd owns. It is unlikely for that to be any connection between
jackd and mlockall-test - other than both map glibc. To further exclude
any possibility of resource sharing between jackd and mlockall-test,
could you compile the later with -static?

	Ingo

Re: [patch] voluntary-preempt-2.6.8.1-P0

[Lee Revell]

On Sun, 2004-08-15 at 22:36, Ingo Molnar wrote:
> * Lee Revell <rlrevell@joe-job.com> wrote:
> 
> > There are a number of samples above 700us.  I am working with a period
> > time of 666 usecs, and since there are 2 periods per buffer, we would
> > have to hit two > 666 usec latencies in a row for an xrun - it appears
> > that there are many individual latencies above 666, certainly more
> > than there are xruns.  So, maybe the mlockall issue is not a result of
> > triggering a single large latency, but of increasing the frequency of
> > these higher latencies so that we are more likely to hit 2 in a row.
> 
> hm, it seems the mlockall() issue is too deterministic for it to be a
> statistical-only phenomenon. Also, isnt that xrun on the order of 15
> msecs? That's way too big too.
> 

I believe the constant-time behavior that I reported was an artifact of
ALSA xrun debugging.  Now it seems like the latency produced *does*
correspond directly to the amount of memory being mlockall'ed.  If
./mlockall-test 1500 triggers an xrun at all it's ~0.2ms.  3000 triggers
a ~1ms xrun, and 10000 a ~3 ms xrun.

> > IIRC ksoftirqd will defer more work under load, and ksoftirqd is one
> > of the more common offenders to hit the extract_entropy latency. 
> > Maybe mlockall causes more softirqs to be deferred, thus increaing the
> > change that we will have to do more than 666 usecs worth of work on 2
> > successive wakeups.
> 
> there should be no relation between softirqs and mlockall().
> 
> this is truly a mind-boggling latency. mlockall() is fully preemptible. 
> All it does is to pre-fault the whole range of pages that a process has.
> 
> could you try another thing: modify mlockall-test.cc to use mlock() on
> the freshly allocated anonymous pages? Does this produce the same
> latencies? mlockall() prefaults _all_ pages the process currently has. 
> Maybe mlockall() touches some page that is mapped both by jackd and
> mlockall-test and thus somehow interacts with jackd's scheduling.
> 

I don't know C++, Florian wrote this program.  Can you provide a
pseudo-patch?

> the anonymous pages themselves can have no IPC-alike connection to any
> page jackd owns. It is unlikely for that to be any connection between
> jackd and mlockall-test - other than both map glibc. To further exclude
> any possibility of resource sharing between jackd and mlockall-test,
> could you compile the later with -static?

Sure, I will try this.

Lee

Re: [patch] voluntary-preempt-2.6.8.1-P0

[Ingo Molnar]

* Lee Revell <rlrevell@joe-job.com> wrote:

> I believe the constant-time behavior that I reported was an artifact
> of ALSA xrun debugging.  Now it seems like the latency produced *does*
> correspond directly to the amount of memory being mlockall'ed.  If
> ./mlockall-test 1500 triggers an xrun at all it's ~0.2ms.  3000
> triggers a ~1ms xrun, and 10000 a ~3 ms xrun.

could you try more extreme values - e.g. does 100 MB cause a 30 msec
xrun?

	Ingo

Re: [patch] voluntary-preempt-2.6.8.1-P0

[Ingo Molnar]

doh - i think i found a brown-paperbag bug.

could you change this code in kernel/sched.c:

 int __sched voluntary_resched(void)
 {
 #ifdef CONFIG_DEBUG_SPINLOCK_SLEEP
         __might_sleep(__FILE__, __LINE__);
 #endif
         if (kernel_preemption || !voluntary_preemption)
                 return 0;

to:

 int __sched voluntary_resched(void)
 {
 #ifdef CONFIG_DEBUG_SPINLOCK_SLEEP
         __might_sleep(__FILE__, __LINE__);
 #endif
         if (!voluntary_preemption)
                 return 0;

Does the mlockall xrun still occur?

	Ingo

Re: [patch] voluntary-preempt-2.6.8.1-P0

[Lee Revell]

On Sun, 2004-08-15 at 23:36, Ingo Molnar wrote:
> doh - i think i found a brown-paperbag bug.

Heh, this would explain all those latency traces with repeated 
calls to voluntary_resched...

Oh well, happens to the best of us.  See also Linux 2.6.8.

Lee

Re: [patch] voluntary-preempt-2.6.8.1-P0

[Ingo Molnar]

* Lee Revell <rlrevell@joe-job.com> wrote:

> On Sun, 2004-08-15 at 23:36, Ingo Molnar wrote:
> > doh - i think i found a brown-paperbag bug.
> 
> Heh, this would explain all those latency traces with repeated 
> calls to voluntary_resched...

hm, didnt those traces show preempt_schedule()?

this bug also fooled the latency tracer because the preempt count went
down to zero (hence the latency got reset), but in reality we skipped
the reschedule.

	Ingo

Re: [patch] voluntary-preempt-2.6.8.1-P0

[Lee Revell]

On Sun, 2004-08-15 at 23:46, Ingo Molnar wrote:
> * Lee Revell <rlrevell@joe-job.com> wrote:
> 
> > On Sun, 2004-08-15 at 23:36, Ingo Molnar wrote:
> > > doh - i think i found a brown-paperbag bug.
> > 
> > Heh, this would explain all those latency traces with repeated 
> > calls to voluntary_resched...
> 
> hm, didnt those traces show preempt_schedule()?
> 

Yes, they did.

Anyway, the change to sched.c fixes the mlockall bug, it works perfectly
now.  Thanks!

I will try next with /dev/random disabled.  Don't most/many new machines
have a hardware RNG that would eliminate the need for this code?

Lee

[patch] voluntary-preempt-2.6.8.1-P1

[Ingo Molnar]

* Lee Revell <rlrevell@joe-job.com> wrote:

> Anyway, the change to sched.c fixes the mlockall bug, it works
> perfectly now.  Thanks!

great! This fix also means that we've got one more lock-break in the
ext3 journalling code and one more lock-break in dcache.c. I've released
-P1 with the fix included:

 http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P1

	Ingo

Re: [patch] voluntary-preempt-2.6.8.1-P1

[Thomas Charbonnel]

Ingo Molnar wrote :
> * Lee Revell <rlrevell@joe-job.com> wrote:
> 
> > Anyway, the change to sched.c fixes the mlockall bug, it works
> > perfectly now.  Thanks!
> 
> great! This fix also means that we've got one more lock-break in the
> ext3 journalling code and one more lock-break in dcache.c. I've released
> -P1 with the fix included:
> 
>  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P1
> 
> 	Ingo

Tested P1. The biggest offender on my system is still reiserfs'
search_by_key. The trace is made of a very long loop of :
 0.016ms (+0.000ms): reiserfs_in_journal (scan_bitmap_block)
 0.016ms (+0.000ms): find_next_zero_bit (scan_bitmap_block)
(...)
 0.977ms (+0.000ms): reiserfs_in_journal (scan_bitmap_block)
 0.977ms (+0.000ms): find_next_zero_bit (scan_bitmap_block)
with interrupts showing up in the trace from time to time.
Do you have plans to fix this, or should I switch to ext3 ?

Another point was the ACPI induced latency I was experiencing. I think
that P1 could help track this down. After a fresh boot on an ACPI
enabled kernel, once the initialization-related latency spikes are over,
I reset /proc/sys/kernel/preempt_max_latency to 100, and then I get a
100us latency_trace attributed to swapper/1:

preemption latency trace v1.0
-----------------------------
 latency: 100 us, entries: 4000 (14253)
 process: swapper/1, uid: 0
 nice: 0, policy: 0, rt_priority: 0
=======>
 0.000ms (+0.000ms): startup_8259A_irq (probe_irq_on)
 0.000ms (+0.000ms): enable_8259A_irq (startup_8259A_irq)
 0.001ms (+0.001ms): startup_8259A_irq (probe_irq_on)
 0.001ms (+0.000ms): enable_8259A_irq (startup_8259A_irq)
 0.002ms (+0.001ms): startup_8259A_irq (probe_irq_on)
 0.002ms (+0.000ms): enable_8259A_irq (startup_8259A_irq)
(...)

but the weird thing is that the latency sum goes way above those 100us,
the culprit being do_IRQ, regularly chewing up to 1ms !

(...)
 0.009ms (+0.000ms): enable_8259A_irq (startup_8259A_irq)
 0.011ms (+0.001ms): startup_8259A_irq (probe_irq_on)
 0.011ms (+0.000ms): enable_8259A_irq (startup_8259A_irq)
 0.954ms (+0.943ms): do_IRQ (common_interrupt)
-------------^
 0.954ms (+0.000ms): mask_and_ack_8259A (do_IRQ)
 0.956ms (+0.002ms): generic_redirect_hardirq (do_IRQ)
 0.956ms (+0.000ms): generic_handle_IRQ_event (do_IRQ)
 0.957ms (+0.000ms): timer_interrupt (generic_handle_IRQ_event)
 0.957ms (+0.000ms): mark_offset_tsc (timer_interrupt)
 0.962ms (+0.005ms): do_timer (timer_interrupt)
(...)
 119.027ms (+0.000ms): preempt_schedule (timer_interrupt)
 119.027ms (+0.000ms): preempt_schedule (timer_interrupt)
 119.027ms (+0.000ms): generic_note_interrupt (do_IRQ)
 119.027ms (+0.000ms): end_8259A_irq (do_IRQ)
 119.028ms (+0.000ms): enable_8259A_irq (do_IRQ)
 119.028ms (+0.000ms): preempt_schedule (do_IRQ)
 119.029ms (+0.000ms): preempt_schedule (do_IRQ)
 119.029ms (+0.000ms): do_softirq (do_IRQ)
 119.029ms (+0.000ms): __do_softirq (do_softirq)
 120.017ms (+0.987ms): do_IRQ (common_interrupt)
---------------^
 120.017ms (+0.000ms): mask_and_ack_8259A (do_IRQ)
 120.019ms (+0.002ms): preempt_schedule (do_IRQ)
 120.019ms (+0.000ms): generic_redirect_hardirq (do_IRQ)
 120.019ms (+0.000ms): preempt_schedule (do_IRQ)
 120.020ms (+0.000ms): generic_handle_IRQ_event (do_IRQ)
(...)

You can find the full trace here :
http://www.undata.org/~thomas/swapper.trace

Do you have an idea of what's happening here ?

Thomas

Re: [patch] voluntary-preempt-2.6.8.1-P1

[Ingo Molnar]

* Thomas Charbonnel <thomas@undata.org> wrote:

> Tested P1. The biggest offender on my system is still reiserfs'
> search_by_key. The trace is made of a very long loop of :
>  0.016ms (+0.000ms): reiserfs_in_journal (scan_bitmap_block)
>  0.016ms (+0.000ms): find_next_zero_bit (scan_bitmap_block)
> (...)
>  0.977ms (+0.000ms): reiserfs_in_journal (scan_bitmap_block)
>  0.977ms (+0.000ms): find_next_zero_bit (scan_bitmap_block)
> with interrupts showing up in the trace from time to time. Do you have
> plans to fix this, or should I switch to ext3 ?

i took a quick look and the reiserfs locking rules in that place do not
seem to be easily fixable - this is the tree-lookup code which i suspect
cannot be preempted. The reiser journalling code also makes use of the
big kernel lock. I'd suggest reporting this to the reiserfs folks and
(if it's not too much effort to migrate) use ext3 meanwhile.

> but the weird thing is that the latency sum goes way above those 100us,
> the culprit being do_IRQ, regularly chewing up to 1ms !

>  0.011ms (+0.000ms): enable_8259A_irq (startup_8259A_irq)
>  0.954ms (+0.943ms): do_IRQ (common_interrupt)

weird. This has the looks of a preempt-timing bug (we get a timer IRQ
every 1 msec) - but there should be no preempt-timing when we are in the
idle task (swapper).

> http://www.undata.org/~thomas/swapper.trace

i'll upload -P2 in a couple of minutes, it will trace the code that
do_IRQ() interrupted too - that would be quite useful in your case.

	Ingo

[patch] voluntary-preempt-2.6.8.1-P2

[Ingo Molnar]

here's -P2:

 http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P2

Changes since -P1:

 - trace interrupted kernel code (via hardirqs, NMIs and pagefaults)

 - yet another shot at trying to fix the IO-APIC/USB issues.

 - mcount speedups - tracing should be faster

	Ingo

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Thomas Charbonnel]

Ingo Molnar wrote :
> here's -P2:
> 
>  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P2
> 
> Changes since -P1:
> 
>  - trace interrupted kernel code (via hardirqs, NMIs and pagefaults)
> 
>  - yet another shot at trying to fix the IO-APIC/USB issues.
> 
>  - mcount speedups - tracing should be faster
> 
> 	Ingo

Same do_IRQ problem with P2, trace is here :
http://www.undata.org/~thomas/swapper-P2.trace

Thomas

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Thomas Charbonnel]

I wrote :
> Ingo Molnar wrote :
> > here's -P2:
> > 
> >  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P2
> > 
> > Changes since -P1:
> > 
> >  - trace interrupted kernel code (via hardirqs, NMIs and pagefaults)
> > 
> >  - yet another shot at trying to fix the IO-APIC/USB issues.
> > 
> >  - mcount speedups - tracing should be faster
> > 
> > 	Ingo
> 
> Same do_IRQ problem with P2, trace is here :
> http://www.undata.org/~thomas/swapper-P2.trace
> 
> Thomas
> 

Ok, maybe that was a false positive. In fact the trace corresponds to
some preempt violation occurring during the boot process :
preemption latency trace v1.0
-----------------------------
 latency: 136095 us, entries: 4000 (14818)
 process: swapper/1, uid: 0
 nice: 0, policy: 0, rt_priority: 0
=======>
When I reset preempt_max_latency to 100, /proc/latency_trace stays the
same, but the latency time reported is updated to reflect the new
preemtp_max_latency :
root@satellite thomas # diff trace1-P2 trace2-P2
3c3
<  latency: 136095 us, entries: 4000 (14818)
---
>  latency: 100 us, entries: 4000 (14818)

There still are weird things happening with irq handling, though. how
can generic_redirect_hardirq eat half a millisecond :

preemption latency trace v1.0
-----------------------------
 latency: 481 us, entries: 32 (32)
 process: swapper/0, uid: 0
 nice: 0, policy: 0, rt_priority: 0
=======>
 0.000ms (+0.000ms): do_IRQ (default_idle)
 0.000ms (+0.000ms): mask_and_ack_8259A (do_IRQ)
 0.459ms (+0.459ms): generic_redirect_hardirq (do_IRQ)
 0.459ms (+0.000ms): generic_handle_IRQ_event (do_IRQ)
 0.459ms (+0.000ms): timer_interrupt (generic_handle_IRQ_event)
 0.459ms (+0.000ms): mark_offset_tsc (timer_interrupt)
 0.465ms (+0.005ms): do_timer (timer_interrupt)
 0.465ms (+0.000ms): update_process_times (do_timer)
 0.465ms (+0.000ms): update_one_process (update_process_times)
 0.465ms (+0.000ms): run_local_timers (update_process_times)
 0.465ms (+0.000ms): raise_softirq (update_process_times)
 0.466ms (+0.000ms): scheduler_tick (update_process_times)
 0.466ms (+0.000ms): sched_clock (scheduler_tick)
 0.466ms (+0.000ms): update_wall_time (do_timer)
 0.466ms (+0.000ms): update_wall_time_one_tick (update_wall_time)
 0.466ms (+0.000ms): profile_hook (timer_interrupt)
 0.466ms (+0.000ms): notifier_call_chain (profile_hook)
 0.467ms (+0.000ms): generic_note_interrupt (do_IRQ)
 0.467ms (+0.000ms): end_8259A_irq (do_IRQ)
 0.467ms (+0.000ms): enable_8259A_irq (do_IRQ)
 0.468ms (+0.000ms): do_softirq (do_IRQ)
 0.468ms (+0.000ms): __do_softirq (do_softirq)
 0.468ms (+0.000ms): wake_up_process (do_softirq)
 0.468ms (+0.000ms): try_to_wake_up (wake_up_process)
 0.468ms (+0.000ms): task_rq_lock (try_to_wake_up)
 0.468ms (+0.000ms): activate_task (try_to_wake_up)
 0.469ms (+0.000ms): sched_clock (activate_task)
 0.469ms (+0.000ms): recalc_task_prio (activate_task)
 0.469ms (+0.000ms): effective_prio (recalc_task_prio)
 0.469ms (+0.000ms): enqueue_task (activate_task)
 0.469ms (+0.000ms): preempt_schedule (try_to_wake_up)
 0.469ms (+0.000ms): check_preempt_timing (sub_preempt_count)

Is there any source of interruption not covered by your P2 patch ?

Thomas

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Thomas Charbonnel]

I wrote :
> I wrote :
> > Ingo Molnar wrote :
> > > here's -P2:
> > > 
> > >  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P2
> > > 
> > > Changes since -P1:
> > > 
> > >  - trace interrupted kernel code (via hardirqs, NMIs and pagefaults)
> > > 
> > >  - yet another shot at trying to fix the IO-APIC/USB issues.
> > > 
> > >  - mcount speedups - tracing should be faster
> > > 
> > > 	Ingo
> > 
> > Same do_IRQ problem with P2, trace is here :
> > http://www.undata.org/~thomas/swapper-P2.trace
> > 
> > Thomas
> > 
(...)
> There still are weird things happening with irq handling, though. how
> can generic_redirect_hardirq eat half a millisecond :
> 
(...)
> Is there any source of interruption not covered by your P2 patch ?
(...)

More on this : 

preemption latency trace v1.0
-----------------------------
 latency: 611 us, entries: 21 (21)
 process: ksoftirqd/0/2, uid: 0
 nice: -10, policy: 0, rt_priority: 0
=======>
 0.000ms (+0.000ms): sched_clock (schedule)
 0.000ms (+0.000ms): dequeue_task (schedule)
 0.000ms (+0.000ms): recalc_task_prio (schedule)
 0.590ms (+0.589ms): effective_prio (recalc_task_prio)
 0.590ms (+0.000ms): enqueue_task (schedule)
 0.590ms (+0.000ms): __switch_to (schedule)
 0.590ms (+0.000ms): finish_task_switch (schedule)
 0.591ms (+0.001ms): do_IRQ (finish_task_switch)
(...)

and

preemption latency trace v1.0
-----------------------------
 latency: 481 us, entries: 32 (32)
 process: swapper/0, uid: 0
 nice: 0, policy: 0, rt_priority: 0
=======>
 0.000ms (+0.000ms): do_IRQ (default_idle)
 0.000ms (+0.000ms): mask_and_ack_8259A (do_IRQ)
 0.459ms (+0.459ms): generic_redirect_hardirq (do_IRQ)
 0.459ms (+0.000ms): generic_handle_IRQ_event (do_IRQ)
 0.459ms (+0.000ms): timer_interrupt (generic_handle_IRQ_event)
 0.459ms (+0.000ms): mark_offset_tsc (timer_interrupt)
 0.465ms (+0.005ms): do_timer (timer_interrupt)
 0.465ms (+0.000ms): update_process_times (do_timer)
 0.465ms (+0.000ms): update_one_process (update_process_times)
 0.465ms (+0.000ms): run_local_timers (update_process_times)
 0.465ms (+0.000ms): raise_softirq (update_process_times)
 0.466ms (+0.000ms): scheduler_tick (update_process_times)
 0.466ms (+0.000ms): sched_clock (scheduler_tick)
 0.466ms (+0.000ms): update_wall_time (do_timer)
 0.466ms (+0.000ms): update_wall_time_one_tick (update_wall_time)
 0.466ms (+0.000ms): profile_hook (timer_interrupt)
 0.466ms (+0.000ms): notifier_call_chain (profile_hook)
 0.467ms (+0.000ms): generic_note_interrupt (do_IRQ)
 0.467ms (+0.000ms): end_8259A_irq (do_IRQ)
 0.467ms (+0.000ms): enable_8259A_irq (do_IRQ)
 0.468ms (+0.000ms): do_softirq (do_IRQ)
 0.468ms (+0.000ms): __do_softirq (do_softirq)
 0.468ms (+0.000ms): wake_up_process (do_softirq)
 0.468ms (+0.000ms): try_to_wake_up (wake_up_process)
 0.468ms (+0.000ms): task_rq_lock (try_to_wake_up)
 0.468ms (+0.000ms): activate_task (try_to_wake_up)
 0.469ms (+0.000ms): sched_clock (activate_task)
 0.469ms (+0.000ms): recalc_task_prio (activate_task)
 0.469ms (+0.000ms): effective_prio (recalc_task_prio)
 0.469ms (+0.000ms): enqueue_task (activate_task)
 0.469ms (+0.000ms): preempt_schedule (try_to_wake_up)
 0.469ms (+0.000ms): check_preempt_timing (sub_preempt_count)

It definitely looks like the kernel is interrupted by some interrupt
source not covered by the patch.

Thomas

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Ingo Molnar]

* Thomas Charbonnel <thomas@undata.org> wrote:

>  0.000ms (+0.000ms): do_IRQ (default_idle)
>  0.000ms (+0.000ms): mask_and_ack_8259A (do_IRQ)
>  0.459ms (+0.459ms): generic_redirect_hardirq (do_IRQ)
>  0.459ms (+0.000ms): generic_handle_IRQ_event (do_IRQ)
>  0.459ms (+0.000ms): timer_interrupt (generic_handle_IRQ_event)

> It definitely looks like the kernel is interrupted by some interrupt
> source not covered by the patch.

the only possibility is SMM, which is not handled by Linux. (but by the
BIOS.) Otherwise we track everything - including NMIs.

can you reproduce this using an UP kernel too?

	Ingo

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Thomas Charbonnel]

Ingo Molnar wrote :
> * Thomas Charbonnel <thomas@undata.org> wrote:
> 
> >  0.000ms (+0.000ms): do_IRQ (default_idle)
> >  0.000ms (+0.000ms): mask_and_ack_8259A (do_IRQ)
> >  0.459ms (+0.459ms): generic_redirect_hardirq (do_IRQ)
> >  0.459ms (+0.000ms): generic_handle_IRQ_event (do_IRQ)
> >  0.459ms (+0.000ms): timer_interrupt (generic_handle_IRQ_event)
> 
> > It definitely looks like the kernel is interrupted by some interrupt
> > source not covered by the patch.
> 
> the only possibility is SMM, which is not handled by Linux. (but by the
> BIOS.) Otherwise we track everything - including NMIs.
> 
> can you reproduce this using an UP kernel too?
> 
> 	Ingo

I'm currently not using SMP :

root@satellite linux # grep SMP .config
CONFIG_BROKEN_ON_SMP=y
# CONFIG_X86_BIGSMP is not set
# CONFIG_SMP is not set

Whole .config here:
http://www.undata.org/~thomas/config-2.6.8-P2

This would confirm the hypothesis of a buggy BIOS, I'm afraid.

Thomas

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Ingo Molnar]

* Thomas Charbonnel <thomas@undata.org> wrote:

> > >  0.000ms (+0.000ms): do_IRQ (default_idle)
> > >  0.000ms (+0.000ms): mask_and_ack_8259A (do_IRQ)
> > >  0.459ms (+0.459ms): generic_redirect_hardirq (do_IRQ)
> > >  0.459ms (+0.000ms): generic_handle_IRQ_event (do_IRQ)
> > >  0.459ms (+0.000ms): timer_interrupt (generic_handle_IRQ_event)
> > 
> > > It definitely looks like the kernel is interrupted by some interrupt
> > > source not covered by the patch.
> > 
> > the only possibility is SMM, which is not handled by Linux. (but by the
> > BIOS.) Otherwise we track everything - including NMIs.

> This would confirm the hypothesis of a buggy BIOS, I'm afraid.

there are still other (and more likely) possible reasons like a bug in
the latency tracer. (Or a broken TSC - albeit this is less likely.)

can you reproduce this phenomenon at will? Does it go away if you turn 
ACPI/APM off (both in the kernel and in the BIOS). Does it go away if 
you use idle=poll?

	Ingo

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Thomas Charbonnel]

Ingo Molnar wrote :
> * Thomas Charbonnel <thomas@undata.org> wrote:
> 
> > > >  0.000ms (+0.000ms): do_IRQ (default_idle)
> > > >  0.000ms (+0.000ms): mask_and_ack_8259A (do_IRQ)
> > > >  0.459ms (+0.459ms): generic_redirect_hardirq (do_IRQ)
> > > >  0.459ms (+0.000ms): generic_handle_IRQ_event (do_IRQ)
> > > >  0.459ms (+0.000ms): timer_interrupt (generic_handle_IRQ_event)
> > > 
> > > > It definitely looks like the kernel is interrupted by some interrupt
> > > > source not covered by the patch.
> > > 
> > > the only possibility is SMM, which is not handled by Linux. (but by the
> > > BIOS.) Otherwise we track everything - including NMIs.
> 
> > This would confirm the hypothesis of a buggy BIOS, I'm afraid.
> 
> there are still other (and more likely) possible reasons like a bug in
> the latency tracer. (Or a broken TSC - albeit this is less likely.)
> 

I don't think the latency tracer is involved. I think this is the same
phenomenon as the regular latency spikes I reported earlier.
At some point I thought of a clock issue, indeed, so I switched to
pmtmr, but this is broken with voluntary preempt (I think it relates to
the sched_clock code when not using TSC -> precision of the reported
preempt threshold violations was 1ms). Worth mentioning is that the
latency spikes interval was the same even if I compiled cpufreq in and
switched the processor speed at runtime.

> can you reproduce this phenomenon at will? Does it go away if you turn 
> ACPI/APM off (both in the kernel and in the BIOS).

I can't turn PM off in the BIOS, but turning ACPI off in the kernel
suppresses the problem.

>  Does it go away if 
> you use idle=poll?
> 

I don't think so (the spikes where still here when you first merged
wli's preempt timing patch and forced idle=poll).

So basically I think the spikes are BIOS related, they've been detected
so far by the alsa xrun_debug mechanism, by the latency-test suite, and
now by the latency tracer, which shows that they're not linked to any
particular code section.

Thomas

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Lee Revell]

On Mon, 2004-08-16 at 10:12, Thomas Charbonnel wrote:
> I wrote :
> > Ingo Molnar wrote :
> > > here's -P2:
> > > 
> > >  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P2
> > > 
> > > Changes since -P1:
> > > 
> > >  - trace interrupted kernel code (via hardirqs, NMIs and pagefaults)
> > > 
> > >  - yet another shot at trying to fix the IO-APIC/USB issues.
> > > 
> > >  - mcount speedups - tracing should be faster
> > > 
> > > 	Ingo
> > 
> > Same do_IRQ problem with P2, trace is here :
> > http://www.undata.org/~thomas/swapper-P2.trace
> > 
> > Thomas
> > 
> 
> Ok, maybe that was a false positive. In fact the trace corresponds to
> some preempt violation occurring during the boot process :
> preemption latency trace v1.0
> -----------------------------
>  latency: 136095 us, entries: 4000 (14818)
>  process: swapper/1, uid: 0
>  nice: 0, policy: 0, rt_priority: 0

Yes, I get this one too.  Maybe the tracer should ignore such huge
values, as they seem to only happen during boot and there dosn't seem to
be a reason to fix them.

Lee

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Lee Revell]

On Mon, 2004-08-16 at 08:09, Ingo Molnar wrote:
> here's -P2:
> 
>  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P2
> 
> Changes since -P1:
> 
>  - trace interrupted kernel code (via hardirqs, NMIs and pagefaults)
> 
>  - yet another shot at trying to fix the IO-APIC/USB issues.
> 
>  - mcount speedups - tracing should be faster
> 

P2 will not boot for me.  It hangs right after detecting my (surprise!)
USB mouse.

Lee

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Ingo Molnar]

* Lee Revell <rlrevell@joe-job.com> wrote:

> P2 will not boot for me.  It hangs right after detecting my
> (surprise!) USB mouse.

hm. Could you take -P1's arch/i386/io_apic.c and put it into the P2 tree
- does that fix your bootup? (and does your USB mouse work after that?)

	Ingo

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Lee Revell]

On Tue, 2004-08-17 at 03:30, Ingo Molnar wrote:
> * Lee Revell <rlrevell@joe-job.com> wrote:
> 
> > P2 will not boot for me.  It hangs right after detecting my
> > (surprise!) USB mouse.
> 
> hm. Could you take -P1's arch/i386/io_apic.c and put it into the P2 tree
> - does that fix your bootup? (and does your USB mouse work after that?)
> 

I think this is an unrelated problem, still testing.

Lee

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Lee Revell]

On Tue, 2004-08-17 at 03:30, Ingo Molnar wrote:
> * Lee Revell <rlrevell@joe-job.com> wrote:
> 
> > P2 will not boot for me.  It hangs right after detecting my
> > (surprise!) USB mouse.
> 
> hm. Could you take -P1's arch/i386/io_apic.c and put it into the P2 tree
> - does that fix your bootup? (and does your USB mouse work after that?)
> 

OK, I am still not sure that P2 was the problem - reverting from ALSA
1.0.6a to ALSA cvs from a few weeks ago seemed to fix the problem. 
Ugh.  Still testing.

Lee

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Thomas Charbonnel]

Ingo Molnar wrote :
> here's -P2:
> 
>  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P2
> 
> Changes since -P1:
> 
>  - trace interrupted kernel code (via hardirqs, NMIs and pagefaults)
> 
>  - yet another shot at trying to fix the IO-APIC/USB issues.
> 
>  - mcount speedups - tracing should be faster
> 
> 	Ingo

I think I stumbled across some bugs/false positives.
Those tests were run with acpi=off, so that this specific issue doesn't
interfere. voluntary_preemption is set to 3 throughout.

The first problem was already reported by Lee Revell. Creating a new tab
in gnome-terminal gives :

[...]
 0.064ms (+0.000ms): preempt_schedule (try_to_wake_up)
 0.065ms (+0.000ms): preempt_schedule (copy_page_range)
 0.065ms (+0.000ms): preempt_schedule (copy_page_range)
[... plenty of preempt_schedule (copy_page_range) skipped ...]
 0.202ms (+0.000ms): preempt_schedule (copy_page_range)
 0.202ms (+0.000ms): preempt_schedule (copy_page_range)
 0.202ms (+0.000ms): check_preempt_timing (touch_preempt_timing)

This is induced by kernel_preemption=0 and disappears with
kernel_preemption=1. It seems to be a side-effect of the current design.
Is there a way to avoid this ?

The second one still involves creating a new tab in gnome-terminal, but
with kernel_preemption=1 :

preemption latency trace v1.0
-----------------------------
 latency: 130 us, entries: 6 (6)
 process: gnome-terminal/14381, uid: 1000
 nice: 0, policy: 0, rt_priority: 0
=======>
 0.000ms (+0.000ms): __vma_link_rb (copy_mm)
 0.000ms (+0.000ms): rb_insert_color (copy_mm)
 0.000ms (+0.000ms): __rb_rotate_left (rb_insert_color)
 0.000ms (+0.000ms): copy_page_range (copy_mm)
 0.000ms (+0.000ms): pte_alloc_map (copy_page_range)
 0.127ms (+0.126ms): check_preempt_timing (touch_preempt_timing)

When entering check_preempt_timing, preempt_thresh was 0, and
preempt_max_latency had been freshly reset to 100. It should have
triggered this code :
		if (latency < preempt_max_latency)
			goto out;
but for some reason it didn't (or there is a problem in the tracing
code, not showing events that would have increased 'latency').

Thomas

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Ingo Molnar]

* Thomas Charbonnel <thomas@undata.org> wrote:

> When entering check_preempt_timing, preempt_thresh was 0, and
> preempt_max_latency had been freshly reset to 100. It should have
> triggered this code :
>
> 		if (latency < preempt_max_latency)
> 			goto out;
>
> but for some reason it didn't (or there is a problem in the tracing
> code, not showing events that would have increased 'latency').

there is one case where we could 'miss' a new latency: when
/proc/latency_trace is accessed. For the duration of /proc/latency_trace
access, the updating of the max latency is stopped:

        if (down_trylock(&max_mutex))
                goto out;

this is not really a practical problem and fixing it would be quite 
complex.

	Ingo

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Thomas Charbonnel]

Ingo Molnar wrote :
> here's -P2:
> 
>  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P2
> 
> Changes since -P1:
> 
>  - trace interrupted kernel code (via hardirqs, NMIs and pagefaults)
> 
>  - yet another shot at trying to fix the IO-APIC/USB issues.
> 
>  - mcount speedups - tracing should be faster
> 
> 	Ingo
> -

The next problem I have relates to irq sharing. 
On my laptop I can't avoid it :
 10:    1070631          XT-PIC  yenta, yenta, uhci_hcd, Intel
82801CA-ICH3, hdsp, eth0
If I set the sound card's interrupt to be non threaded, then I get a
rather long non preemptible section :
http://www.undata.org/~thomas/irq_sharing.trace

As a side note, and this has already been reported here several times,
the SA_INTERRUPT flag set notably by the sound card drivers handlers is
not honored on current kernels if the device is not the first one to be
registered. A simple fix would be to add SA_INTERRUPT handlers at the
beginning instead of the end of the irq queue in setup_irq.

Similarly, when using SA_SAMPLE_RANDOM, all devices on the given irq
contribute to the entropy, even those that have a predictable interrupt
rate (e.g. sound cards), and/or for which the number of interrupts could
outweight the number of interrupts of the original SA_SAMPLE_RANDOM
driver. 

Thomas

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Takashi Iwai]

At Wed, 18 Aug 2004 14:22:07 +0200,
Thomas Charbonnel wrote:
> 
> As a side note, and this has already been reported here several times,
> the SA_INTERRUPT flag set notably by the sound card drivers handlers is
> not honored on current kernels if the device is not the first one to be
> registered. A simple fix would be to add SA_INTERRUPT handlers at the
> beginning instead of the end of the irq queue in setup_irq.
> 
> Similarly, when using SA_SAMPLE_RANDOM, all devices on the given irq
> contribute to the entropy, even those that have a predictable interrupt
> rate (e.g. sound cards), and/or for which the number of interrupts could
> outweight the number of interrupts of the original SA_SAMPLE_RANDOM
> driver. 

Hmm, something like this?


Takashi

--- linux/arch/i386/kernel/irq.c	2004-08-18 15:15:18.272067276 +0200
+++ linux/arch/i386/kernel/irq.c	2004-08-18 15:26:13.513979698 +0200
@@ -219,15 +219,22 @@ inline void synchronize_irq(unsigned int
 asmlinkage int handle_IRQ_event(unsigned int irq,
 		struct pt_regs *regs, struct irqaction *action)
 {
-	int status = 1;	/* Force the "do bottom halves" bit */
-	int retval = 0;
-
-	if (!(action->flags & SA_INTERRUPT))
+	int status;
+	int ret, retval = 0;
+	struct irqaction *ac;
+
+	status = 0;
+	for (ac = action; ac; ac = ac->next)
+		status |= ac->flags;
+	if (!(status & SA_INTERRUPT))
 		local_irq_enable();
 
+	status = 0;
 	do {
-		status |= action->flags;
-		retval |= action->handler(irq, action->dev_id, regs);
+		ret = action->handler(irq, action->dev_id, regs);
+		if (ret)
+			status |= action->flags;
+		retval |= ret;
 		action = action->next;
 	} while (action);
 	if (status & SA_SAMPLE_RANDOM)

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Thomas Charbonnel]

Takashi Iwai wrote :
> At Wed, 18 Aug 2004 14:22:07 +0200,
> Thomas Charbonnel wrote:
> > 
> > As a side note, and this has already been reported here several times,
> > the SA_INTERRUPT flag set notably by the sound card drivers handlers is
> > not honored on current kernels if the device is not the first one to be
> > registered. A simple fix would be to add SA_INTERRUPT handlers at the
> > beginning instead of the end of the irq queue in setup_irq.
> > 
> > Similarly, when using SA_SAMPLE_RANDOM, all devices on the given irq
> > contribute to the entropy, even those that have a predictable interrupt
> > rate (e.g. sound cards), and/or for which the number of interrupts could
> > outweight the number of interrupts of the original SA_SAMPLE_RANDOM
> > driver. 
> 
> Hmm, something like this?
> 
> 
> Takashi
> 
> --- linux/arch/i386/kernel/irq.c	2004-08-18 15:15:18.272067276 +0200
> +++ linux/arch/i386/kernel/irq.c	2004-08-18 15:26:13.513979698 +0200
> @@ -219,15 +219,22 @@ inline void synchronize_irq(unsigned int
>  asmlinkage int handle_IRQ_event(unsigned int irq,
>  		struct pt_regs *regs, struct irqaction *action)
>  {
> -	int status = 1;	/* Force the "do bottom halves" bit */
> -	int retval = 0;
> -
> -	if (!(action->flags & SA_INTERRUPT))
> +	int status;
> +	int ret, retval = 0;
> +	struct irqaction *ac;
> +
> +	status = 0;
> +	for (ac = action; ac; ac = ac->next)
> +		status |= ac->flags;
> +	if (!(status & SA_INTERRUPT))
>  		local_irq_enable();
>  
> +	status = 0;
>  	do {
> -		status |= action->flags;
> -		retval |= action->handler(irq, action->dev_id, regs);
> +		ret = action->handler(irq, action->dev_id, regs);
> +		if (ret)
> +			status |= action->flags;
> +		retval |= ret;
>  		action = action->next;
>  	} while (action);
>  	if (status & SA_SAMPLE_RANDOM)

I was thinking more of something like this :

--- irq.c.orig  2004-08-16 14:26:34.000000000 +0200
+++ irq.c       2004-08-18 17:23:18.011059064 +0200
@@ -955,11 +955,16 @@
                        return -EBUSY;
                }

-               /* add new interrupt at end of irq queue */
-               do {
-                       p = &old->next;
-                       old = *p;
-               } while (old);
+               if (new->flags & SA_INTERRUPT)
+                       /* add interrupt at the start of the queue */
+                       new->next = old;
+               else
+                       /* add new interrupt at end of irq queue */
+                       do {
+                               p = &old->next;
+                               old = *p;
+                       } while (old);
+
                shared = 1;
        }

Thomas

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Takashi Iwai]

At Wed, 18 Aug 2004 17:26:16 +0200,
Thomas Charbonnel wrote:
> 
> I was thinking more of something like this :
> 
> --- irq.c.orig  2004-08-16 14:26:34.000000000 +0200
> +++ irq.c       2004-08-18 17:23:18.011059064 +0200
> @@ -955,11 +955,16 @@
>                         return -EBUSY;
>                 }
> 
> -               /* add new interrupt at end of irq queue */
> -               do {
> -                       p = &old->next;
> -                       old = *p;
> -               } while (old);
> +               if (new->flags & SA_INTERRUPT)
> +                       /* add interrupt at the start of the queue */
> +                       new->next = old;
> +               else
> +                       /* add new interrupt at end of irq queue */
> +                       do {
> +                               p = &old->next;
> +                               old = *p;
> +                       } while (old);
> +
>                 shared = 1;
>         }

Ok, how about this one?  Together with your patch, the irq is turned
on for handlers without SA_INTERRUPT.


Takashi

--- linux/arch/i386/kernel/irq.c	2004-08-18 15:15:18.000000000 +0200
+++ linux/arch/i386/kernel/irq.c	2004-08-18 17:42:57.454819403 +0200
@@ -219,15 +219,23 @@ inline void synchronize_irq(unsigned int
 asmlinkage int handle_IRQ_event(unsigned int irq,
 		struct pt_regs *regs, struct irqaction *action)
 {
-	int status = 1;	/* Force the "do bottom halves" bit */
-	int retval = 0;
-
-	if (!(action->flags & SA_INTERRUPT))
-		local_irq_enable();
+	int status;
+	int ret, retval = 0;
+	int irq_off = 1;
 
+	status = 0;
 	do {
-		status |= action->flags;
-		retval |= action->handler(irq, action->dev_id, regs);
+		/* Assume that all SA_INTERRUPT handlers are at the head
+		 * of the irq queue
+		 */
+		if (irq_off && ! (action->flags & SA_INTERRUPT)) {
+			local_irq_enable();
+			irq_off = 0;
+		}
+		ret = action->handler(irq, action->dev_id, regs);
+		if (ret)
+			status |= action->flags;
+		retval |= ret;
 		action = action->next;
 	} while (action);
 	if (status & SA_SAMPLE_RANDOM)
@@ -955,11 +963,16 @@ int setup_irq(unsigned int irq, struct i
 			return -EBUSY;
 		}
 
-		/* add new interrupt at end of irq queue */
-		do {
-			p = &old->next;
-			old = *p;
-		} while (old);
+		if (new->flags & SA_INTERRUPT)
+			/* add interrupt at the start of the queue */
+			new->next = old;
+		else
+			/* add new interrupt at end of irq queue */
+			do {
+				p = &old->next;
+				old = *p;
+			} while (old);
+
 		shared = 1;
 	}
 

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Thomas Charbonnel]

Takashi Iwai wrote :

[...]
> Ok, how about this one?  Together with your patch, the irq is turned
> on for handlers without SA_INTERRUPT.
> 
> 
> Takashi
> 
> --- linux/arch/i386/kernel/irq.c	2004-08-18 15:15:18.000000000 +0200
> +++ linux/arch/i386/kernel/irq.c	2004-08-18 17:42:57.454819403 +0200
> @@ -219,15 +219,23 @@ inline void synchronize_irq(unsigned int
>  asmlinkage int handle_IRQ_event(unsigned int irq,
>  		struct pt_regs *regs, struct irqaction *action)
>  {
> -	int status = 1;	/* Force the "do bottom halves" bit */
> -	int retval = 0;
> -
> -	if (!(action->flags & SA_INTERRUPT))
> -		local_irq_enable();
> +	int status;
> +	int ret, retval = 0;
> +	int irq_off = 1;
>  
> +	status = 0;
>  	do {
> -		status |= action->flags;
> -		retval |= action->handler(irq, action->dev_id, regs);
> +		/* Assume that all SA_INTERRUPT handlers are at the head
> +		 * of the irq queue
> +		 */
> +		if (irq_off && ! (action->flags & SA_INTERRUPT)) {
> +			local_irq_enable();
> +			irq_off = 0;
> +		}
> +		ret = action->handler(irq, action->dev_id, regs);
> +		if (ret)
> +			status |= action->flags;
> +		retval |= ret;
>  		action = action->next;
>  	} while (action);
>  	if (status & SA_SAMPLE_RANDOM)
> @@ -955,11 +963,16 @@ int setup_irq(unsigned int irq, struct i
>  			return -EBUSY;
>  		}
>  
> -		/* add new interrupt at end of irq queue */
> -		do {
> -			p = &old->next;
> -			old = *p;
> -		} while (old);
> +		if (new->flags & SA_INTERRUPT)
> +			/* add interrupt at the start of the queue */
> +			new->next = old;
> +		else
> +			/* add new interrupt at end of irq queue */
> +			do {
> +				p = &old->next;
> +				old = *p;
> +			} while (old);
> +
>  		shared = 1;
>  	}
>  

You're right, of course, my mistake. It looks good now.

Thomas

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Ingo Molnar]

* Thomas Charbonnel <thomas@undata.org> wrote:

> The next problem I have relates to irq sharing. 
> On my laptop I can't avoid it :
>  10:    1070631          XT-PIC  yenta, yenta, uhci_hcd, Intel
> 82801CA-ICH3, hdsp, eth0
> If I set the sound card's interrupt to be non threaded, then I get a
> rather long non preemptible section :
> http://www.undata.org/~thomas/irq_sharing.trace

i'm not sure the IRQ sharing problem can be solved.

we could execute certain handlers immediately, and defer others to an
IRQ thread. But when we defer an IRQ we must keep the IRQ masked - which
prevents further interrupts (possibly from a high-prio non-threaded
handler) to be executed. So we'd see similar (or in fact worse, due to
the redirection cost) latencies than with the current 'all or nothing'
approach.

now in theory we only have to keep the IRQ line masked for level
triggered interrupts (most APIC interrupts are level-triggered). 
Edge-triggered interrupts (such as the XT-PIC ones you have) could be
acked immediately. I'll try to do something later, but right now there
are still some IRQ problems (USB issues, PS2 mouse/keyboard issues) so
i'd not like to complicate the design just yet.

	Ingo