[patch] voluntary-preempt-2.6.8.1-P2

[patch] voluntary-preempt-2.6.8.1-P2

[Ingo Molnar]

here's -P2:

 http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P2

Changes since -P1:

 - trace interrupted kernel code (via hardirqs, NMIs and pagefaults)

 - yet another shot at trying to fix the IO-APIC/USB issues.

 - mcount speedups - tracing should be faster

	Ingo

Re: [patch] voluntary-preempt-2.6.8.1-P2

[jjluza]

It fails here at compile time with :

arch/i386/kernel/built-in.o(.text+0x2fc5): In function `do_nmi':
: undefined reference to `__trace'
arch/i386/kernel/built-in.o(.text+0x3723): In function `do_IRQ':
: undefined reference to `__trace'
arch/i386/mm/built-in.o(.text+0x7ba): In function `do_page_fault':
: undefined reference to `__trace'
make[1]: *** [vmlinux] Erreur 1
make[1]: Leaving directory `/usr/src/linux-2.6.8'
make: *** [stamp-build] Erreur 2


I also got offset when applying the patch. (P1 hadn't this problem)


Regards.

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Thomas Charbonnel]

Ingo Molnar wrote :
> here's -P2:
> 
>  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P2
> 
> Changes since -P1:
> 
>  - trace interrupted kernel code (via hardirqs, NMIs and pagefaults)
> 
>  - yet another shot at trying to fix the IO-APIC/USB issues.
> 
>  - mcount speedups - tracing should be faster
> 
> 	Ingo

Same do_IRQ problem with P2, trace is here :
http://www.undata.org/~thomas/swapper-P2.trace

Thomas

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Thomas Charbonnel]

I wrote :
> Ingo Molnar wrote :
> > here's -P2:
> > 
> >  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P2
> > 
> > Changes since -P1:
> > 
> >  - trace interrupted kernel code (via hardirqs, NMIs and pagefaults)
> > 
> >  - yet another shot at trying to fix the IO-APIC/USB issues.
> > 
> >  - mcount speedups - tracing should be faster
> > 
> > 	Ingo
> 
> Same do_IRQ problem with P2, trace is here :
> http://www.undata.org/~thomas/swapper-P2.trace
> 
> Thomas
> 

Ok, maybe that was a false positive. In fact the trace corresponds to
some preempt violation occurring during the boot process :
preemption latency trace v1.0
-----------------------------
 latency: 136095 us, entries: 4000 (14818)
 process: swapper/1, uid: 0
 nice: 0, policy: 0, rt_priority: 0
=======>
When I reset preempt_max_latency to 100, /proc/latency_trace stays the
same, but the latency time reported is updated to reflect the new
preemtp_max_latency :
root@satellite thomas # diff trace1-P2 trace2-P2
3c3
<  latency: 136095 us, entries: 4000 (14818)
---
>  latency: 100 us, entries: 4000 (14818)

There still are weird things happening with irq handling, though. how
can generic_redirect_hardirq eat half a millisecond :

preemption latency trace v1.0
-----------------------------
 latency: 481 us, entries: 32 (32)
 process: swapper/0, uid: 0
 nice: 0, policy: 0, rt_priority: 0
=======>
 0.000ms (+0.000ms): do_IRQ (default_idle)
 0.000ms (+0.000ms): mask_and_ack_8259A (do_IRQ)
 0.459ms (+0.459ms): generic_redirect_hardirq (do_IRQ)
 0.459ms (+0.000ms): generic_handle_IRQ_event (do_IRQ)
 0.459ms (+0.000ms): timer_interrupt (generic_handle_IRQ_event)
 0.459ms (+0.000ms): mark_offset_tsc (timer_interrupt)
 0.465ms (+0.005ms): do_timer (timer_interrupt)
 0.465ms (+0.000ms): update_process_times (do_timer)
 0.465ms (+0.000ms): update_one_process (update_process_times)
 0.465ms (+0.000ms): run_local_timers (update_process_times)
 0.465ms (+0.000ms): raise_softirq (update_process_times)
 0.466ms (+0.000ms): scheduler_tick (update_process_times)
 0.466ms (+0.000ms): sched_clock (scheduler_tick)
 0.466ms (+0.000ms): update_wall_time (do_timer)
 0.466ms (+0.000ms): update_wall_time_one_tick (update_wall_time)
 0.466ms (+0.000ms): profile_hook (timer_interrupt)
 0.466ms (+0.000ms): notifier_call_chain (profile_hook)
 0.467ms (+0.000ms): generic_note_interrupt (do_IRQ)
 0.467ms (+0.000ms): end_8259A_irq (do_IRQ)
 0.467ms (+0.000ms): enable_8259A_irq (do_IRQ)
 0.468ms (+0.000ms): do_softirq (do_IRQ)
 0.468ms (+0.000ms): __do_softirq (do_softirq)
 0.468ms (+0.000ms): wake_up_process (do_softirq)
 0.468ms (+0.000ms): try_to_wake_up (wake_up_process)
 0.468ms (+0.000ms): task_rq_lock (try_to_wake_up)
 0.468ms (+0.000ms): activate_task (try_to_wake_up)
 0.469ms (+0.000ms): sched_clock (activate_task)
 0.469ms (+0.000ms): recalc_task_prio (activate_task)
 0.469ms (+0.000ms): effective_prio (recalc_task_prio)
 0.469ms (+0.000ms): enqueue_task (activate_task)
 0.469ms (+0.000ms): preempt_schedule (try_to_wake_up)
 0.469ms (+0.000ms): check_preempt_timing (sub_preempt_count)

Is there any source of interruption not covered by your P2 patch ?

Thomas

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Thomas Charbonnel]

I wrote :
> I wrote :
> > Ingo Molnar wrote :
> > > here's -P2:
> > > 
> > >  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P2
> > > 
> > > Changes since -P1:
> > > 
> > >  - trace interrupted kernel code (via hardirqs, NMIs and pagefaults)
> > > 
> > >  - yet another shot at trying to fix the IO-APIC/USB issues.
> > > 
> > >  - mcount speedups - tracing should be faster
> > > 
> > > 	Ingo
> > 
> > Same do_IRQ problem with P2, trace is here :
> > http://www.undata.org/~thomas/swapper-P2.trace
> > 
> > Thomas
> > 
(...)
> There still are weird things happening with irq handling, though. how
> can generic_redirect_hardirq eat half a millisecond :
> 
(...)
> Is there any source of interruption not covered by your P2 patch ?
(...)

More on this : 

preemption latency trace v1.0
-----------------------------
 latency: 611 us, entries: 21 (21)
 process: ksoftirqd/0/2, uid: 0
 nice: -10, policy: 0, rt_priority: 0
=======>
 0.000ms (+0.000ms): sched_clock (schedule)
 0.000ms (+0.000ms): dequeue_task (schedule)
 0.000ms (+0.000ms): recalc_task_prio (schedule)
 0.590ms (+0.589ms): effective_prio (recalc_task_prio)
 0.590ms (+0.000ms): enqueue_task (schedule)
 0.590ms (+0.000ms): __switch_to (schedule)
 0.590ms (+0.000ms): finish_task_switch (schedule)
 0.591ms (+0.001ms): do_IRQ (finish_task_switch)
(...)

and

preemption latency trace v1.0
-----------------------------
 latency: 481 us, entries: 32 (32)
 process: swapper/0, uid: 0
 nice: 0, policy: 0, rt_priority: 0
=======>
 0.000ms (+0.000ms): do_IRQ (default_idle)
 0.000ms (+0.000ms): mask_and_ack_8259A (do_IRQ)
 0.459ms (+0.459ms): generic_redirect_hardirq (do_IRQ)
 0.459ms (+0.000ms): generic_handle_IRQ_event (do_IRQ)
 0.459ms (+0.000ms): timer_interrupt (generic_handle_IRQ_event)
 0.459ms (+0.000ms): mark_offset_tsc (timer_interrupt)
 0.465ms (+0.005ms): do_timer (timer_interrupt)
 0.465ms (+0.000ms): update_process_times (do_timer)
 0.465ms (+0.000ms): update_one_process (update_process_times)
 0.465ms (+0.000ms): run_local_timers (update_process_times)
 0.465ms (+0.000ms): raise_softirq (update_process_times)
 0.466ms (+0.000ms): scheduler_tick (update_process_times)
 0.466ms (+0.000ms): sched_clock (scheduler_tick)
 0.466ms (+0.000ms): update_wall_time (do_timer)
 0.466ms (+0.000ms): update_wall_time_one_tick (update_wall_time)
 0.466ms (+0.000ms): profile_hook (timer_interrupt)
 0.466ms (+0.000ms): notifier_call_chain (profile_hook)
 0.467ms (+0.000ms): generic_note_interrupt (do_IRQ)
 0.467ms (+0.000ms): end_8259A_irq (do_IRQ)
 0.467ms (+0.000ms): enable_8259A_irq (do_IRQ)
 0.468ms (+0.000ms): do_softirq (do_IRQ)
 0.468ms (+0.000ms): __do_softirq (do_softirq)
 0.468ms (+0.000ms): wake_up_process (do_softirq)
 0.468ms (+0.000ms): try_to_wake_up (wake_up_process)
 0.468ms (+0.000ms): task_rq_lock (try_to_wake_up)
 0.468ms (+0.000ms): activate_task (try_to_wake_up)
 0.469ms (+0.000ms): sched_clock (activate_task)
 0.469ms (+0.000ms): recalc_task_prio (activate_task)
 0.469ms (+0.000ms): effective_prio (recalc_task_prio)
 0.469ms (+0.000ms): enqueue_task (activate_task)
 0.469ms (+0.000ms): preempt_schedule (try_to_wake_up)
 0.469ms (+0.000ms): check_preempt_timing (sub_preempt_count)

It definitely looks like the kernel is interrupted by some interrupt
source not covered by the patch.

Thomas

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Ingo Molnar]

* Thomas Charbonnel <thomas@undata.org> wrote:

>  0.000ms (+0.000ms): do_IRQ (default_idle)
>  0.000ms (+0.000ms): mask_and_ack_8259A (do_IRQ)
>  0.459ms (+0.459ms): generic_redirect_hardirq (do_IRQ)
>  0.459ms (+0.000ms): generic_handle_IRQ_event (do_IRQ)
>  0.459ms (+0.000ms): timer_interrupt (generic_handle_IRQ_event)

> It definitely looks like the kernel is interrupted by some interrupt
> source not covered by the patch.

the only possibility is SMM, which is not handled by Linux. (but by the
BIOS.) Otherwise we track everything - including NMIs.

can you reproduce this using an UP kernel too?

	Ingo

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Thomas Charbonnel]

Ingo Molnar wrote :
> * Thomas Charbonnel <thomas@undata.org> wrote:
> 
> >  0.000ms (+0.000ms): do_IRQ (default_idle)
> >  0.000ms (+0.000ms): mask_and_ack_8259A (do_IRQ)
> >  0.459ms (+0.459ms): generic_redirect_hardirq (do_IRQ)
> >  0.459ms (+0.000ms): generic_handle_IRQ_event (do_IRQ)
> >  0.459ms (+0.000ms): timer_interrupt (generic_handle_IRQ_event)
> 
> > It definitely looks like the kernel is interrupted by some interrupt
> > source not covered by the patch.
> 
> the only possibility is SMM, which is not handled by Linux. (but by the
> BIOS.) Otherwise we track everything - including NMIs.
> 
> can you reproduce this using an UP kernel too?
> 
> 	Ingo

I'm currently not using SMP :

root@satellite linux # grep SMP .config
CONFIG_BROKEN_ON_SMP=y
# CONFIG_X86_BIGSMP is not set
# CONFIG_SMP is not set

Whole .config here:
http://www.undata.org/~thomas/config-2.6.8-P2

This would confirm the hypothesis of a buggy BIOS, I'm afraid.

Thomas

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Ingo Molnar]

* Thomas Charbonnel <thomas@undata.org> wrote:

> > >  0.000ms (+0.000ms): do_IRQ (default_idle)
> > >  0.000ms (+0.000ms): mask_and_ack_8259A (do_IRQ)
> > >  0.459ms (+0.459ms): generic_redirect_hardirq (do_IRQ)
> > >  0.459ms (+0.000ms): generic_handle_IRQ_event (do_IRQ)
> > >  0.459ms (+0.000ms): timer_interrupt (generic_handle_IRQ_event)
> > 
> > > It definitely looks like the kernel is interrupted by some interrupt
> > > source not covered by the patch.
> > 
> > the only possibility is SMM, which is not handled by Linux. (but by the
> > BIOS.) Otherwise we track everything - including NMIs.

> This would confirm the hypothesis of a buggy BIOS, I'm afraid.

there are still other (and more likely) possible reasons like a bug in
the latency tracer. (Or a broken TSC - albeit this is less likely.)

can you reproduce this phenomenon at will? Does it go away if you turn 
ACPI/APM off (both in the kernel and in the BIOS). Does it go away if 
you use idle=poll?

	Ingo

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Thomas Charbonnel]

Ingo Molnar wrote :
> * Thomas Charbonnel <thomas@undata.org> wrote:
> 
> > > >  0.000ms (+0.000ms): do_IRQ (default_idle)
> > > >  0.000ms (+0.000ms): mask_and_ack_8259A (do_IRQ)
> > > >  0.459ms (+0.459ms): generic_redirect_hardirq (do_IRQ)
> > > >  0.459ms (+0.000ms): generic_handle_IRQ_event (do_IRQ)
> > > >  0.459ms (+0.000ms): timer_interrupt (generic_handle_IRQ_event)
> > > 
> > > > It definitely looks like the kernel is interrupted by some interrupt
> > > > source not covered by the patch.
> > > 
> > > the only possibility is SMM, which is not handled by Linux. (but by the
> > > BIOS.) Otherwise we track everything - including NMIs.
> 
> > This would confirm the hypothesis of a buggy BIOS, I'm afraid.
> 
> there are still other (and more likely) possible reasons like a bug in
> the latency tracer. (Or a broken TSC - albeit this is less likely.)
> 

I don't think the latency tracer is involved. I think this is the same
phenomenon as the regular latency spikes I reported earlier.
At some point I thought of a clock issue, indeed, so I switched to
pmtmr, but this is broken with voluntary preempt (I think it relates to
the sched_clock code when not using TSC -> precision of the reported
preempt threshold violations was 1ms). Worth mentioning is that the
latency spikes interval was the same even if I compiled cpufreq in and
switched the processor speed at runtime.

> can you reproduce this phenomenon at will? Does it go away if you turn 
> ACPI/APM off (both in the kernel and in the BIOS).

I can't turn PM off in the BIOS, but turning ACPI off in the kernel
suppresses the problem.

>  Does it go away if 
> you use idle=poll?
> 

I don't think so (the spikes where still here when you first merged
wli's preempt timing patch and forced idle=poll).

So basically I think the spikes are BIOS related, they've been detected
so far by the alsa xrun_debug mechanism, by the latency-test suite, and
now by the latency tracer, which shows that they're not linked to any
particular code section.

Thomas

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Peter Zijlstra]

[-- Attachment #1: Type: text/plain, Size: 917 bytes --]

On Mon, 2004-08-16 at 14:52 +0200, jjluza wrote:
> It fails here at compile time with :
> 
> arch/i386/kernel/built-in.o(.text+0x2fc5): In function `do_nmi':
> : undefined reference to `__trace'
> arch/i386/kernel/built-in.o(.text+0x3723): In function `do_IRQ':
> : undefined reference to `__trace'
> arch/i386/mm/built-in.o(.text+0x7ba): In function `do_page_fault':
> : undefined reference to `__trace'
> make[1]: *** [vmlinux] Erreur 1
> make[1]: Leaving directory `/usr/src/linux-2.6.8'
> make: *** [stamp-build] Erreur 2
> 
> 
> I also got offset when applying the patch. (P1 hadn't this problem)
> 
> 
> Regards.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
-- 
Peter Zijlstra <a.p.zijlstra@chello.nl>

[-- Attachment #2: compile-fix-P2 --]
[-- Type: text/plain, Size: 1023 bytes --]

--- arch/i386/kernel/irq.c~	2004-08-16 20:01:58.835999719 +0200
+++ arch/i386/kernel/irq.c	2004-08-16 20:26:24.974943041 +0200
@@ -219,7 +219,9 @@
 	unsigned int status;
 
 	irq_enter();
+#ifdef CONFIG_LATENCY_TRACE
 	__trace((unsigned long)do_IRQ, regs.eip);
+#endif
 
 #ifdef CONFIG_DEBUG_STACKOVERFLOW
 	/* Debugging check for stack overflow: is there less than 1KB free? */
--- arch/i386/kernel/traps.c~	2004-08-16 20:01:58.836999745 +0200
+++ arch/i386/kernel/traps.c	2004-08-16 20:26:48.156378051 +0200
@@ -537,7 +537,9 @@
 	int cpu;
 
 	nmi_enter();
+#ifdef CONFIG_LATENCY_TRACE
 	__trace((unsigned long)do_nmi, regs->eip);
+#endif
 
 	cpu = smp_processor_id();
 	++nmi_count(cpu);
--- arch/i386/mm/fault.c~	2004-08-16 20:01:58.837999771 +0200
+++ arch/i386/mm/fault.c	2004-08-16 20:27:08.181753770 +0200
@@ -223,7 +223,9 @@
 	int write;
 	siginfo_t info;
 
+#ifdef CONFIG_LATENCY_TRACE
 	__trace((unsigned long)do_page_fault, regs->eip);
+#endif
 
 	/* get the address */
 	__asm__("movl %%cr2,%0":"=r" (address));

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Lee Revell]

On Mon, 2004-08-16 at 10:12, Thomas Charbonnel wrote:
> I wrote :
> > Ingo Molnar wrote :
> > > here's -P2:
> > > 
> > >  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P2
> > > 
> > > Changes since -P1:
> > > 
> > >  - trace interrupted kernel code (via hardirqs, NMIs and pagefaults)
> > > 
> > >  - yet another shot at trying to fix the IO-APIC/USB issues.
> > > 
> > >  - mcount speedups - tracing should be faster
> > > 
> > > 	Ingo
> > 
> > Same do_IRQ problem with P2, trace is here :
> > http://www.undata.org/~thomas/swapper-P2.trace
> > 
> > Thomas
> > 
> 
> Ok, maybe that was a false positive. In fact the trace corresponds to
> some preempt violation occurring during the boot process :
> preemption latency trace v1.0
> -----------------------------
>  latency: 136095 us, entries: 4000 (14818)
>  process: swapper/1, uid: 0
>  nice: 0, policy: 0, rt_priority: 0

Yes, I get this one too.  Maybe the tracer should ignore such huge
values, as they seem to only happen during boot and there dosn't seem to
be a reason to fix them.

Lee

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Lee Revell]

On Mon, 2004-08-16 at 08:09, Ingo Molnar wrote:
> here's -P2:
> 
>  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P2
> 
> Changes since -P1:
> 
>  - trace interrupted kernel code (via hardirqs, NMIs and pagefaults)
> 
>  - yet another shot at trying to fix the IO-APIC/USB issues.
> 
>  - mcount speedups - tracing should be faster
> 

P2 will not boot for me.  It hangs right after detecting my (surprise!)
USB mouse.

Lee

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Ingo Molnar]

* Lee Revell <rlrevell@joe-job.com> wrote:

> P2 will not boot for me.  It hangs right after detecting my
> (surprise!) USB mouse.

hm. Could you take -P1's arch/i386/io_apic.c and put it into the P2 tree
- does that fix your bootup? (and does your USB mouse work after that?)

	Ingo

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Lee Revell]

On Tue, 2004-08-17 at 03:30, Ingo Molnar wrote:
> * Lee Revell <rlrevell@joe-job.com> wrote:
> 
> > P2 will not boot for me.  It hangs right after detecting my
> > (surprise!) USB mouse.
> 
> hm. Could you take -P1's arch/i386/io_apic.c and put it into the P2 tree
> - does that fix your bootup? (and does your USB mouse work after that?)
> 

I think this is an unrelated problem, still testing.

Lee

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Lee Revell]

On Tue, 2004-08-17 at 03:30, Ingo Molnar wrote:
> * Lee Revell <rlrevell@joe-job.com> wrote:
> 
> > P2 will not boot for me.  It hangs right after detecting my
> > (surprise!) USB mouse.
> 
> hm. Could you take -P1's arch/i386/io_apic.c and put it into the P2 tree
> - does that fix your bootup? (and does your USB mouse work after that?)
> 

OK, I am still not sure that P2 was the problem - reverting from ALSA
1.0.6a to ALSA cvs from a few weeks ago seemed to fix the problem. 
Ugh.  Still testing.

Lee

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Thomas Charbonnel]

Ingo Molnar wrote :
> here's -P2:
> 
>  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P2
> 
> Changes since -P1:
> 
>  - trace interrupted kernel code (via hardirqs, NMIs and pagefaults)
> 
>  - yet another shot at trying to fix the IO-APIC/USB issues.
> 
>  - mcount speedups - tracing should be faster
> 
> 	Ingo

I think I stumbled across some bugs/false positives.
Those tests were run with acpi=off, so that this specific issue doesn't
interfere. voluntary_preemption is set to 3 throughout.

The first problem was already reported by Lee Revell. Creating a new tab
in gnome-terminal gives :

[...]
 0.064ms (+0.000ms): preempt_schedule (try_to_wake_up)
 0.065ms (+0.000ms): preempt_schedule (copy_page_range)
 0.065ms (+0.000ms): preempt_schedule (copy_page_range)
[... plenty of preempt_schedule (copy_page_range) skipped ...]
 0.202ms (+0.000ms): preempt_schedule (copy_page_range)
 0.202ms (+0.000ms): preempt_schedule (copy_page_range)
 0.202ms (+0.000ms): check_preempt_timing (touch_preempt_timing)

This is induced by kernel_preemption=0 and disappears with
kernel_preemption=1. It seems to be a side-effect of the current design.
Is there a way to avoid this ?

The second one still involves creating a new tab in gnome-terminal, but
with kernel_preemption=1 :

preemption latency trace v1.0
-----------------------------
 latency: 130 us, entries: 6 (6)
 process: gnome-terminal/14381, uid: 1000
 nice: 0, policy: 0, rt_priority: 0
=======>
 0.000ms (+0.000ms): __vma_link_rb (copy_mm)
 0.000ms (+0.000ms): rb_insert_color (copy_mm)
 0.000ms (+0.000ms): __rb_rotate_left (rb_insert_color)
 0.000ms (+0.000ms): copy_page_range (copy_mm)
 0.000ms (+0.000ms): pte_alloc_map (copy_page_range)
 0.127ms (+0.126ms): check_preempt_timing (touch_preempt_timing)

When entering check_preempt_timing, preempt_thresh was 0, and
preempt_max_latency had been freshly reset to 100. It should have
triggered this code :
		if (latency < preempt_max_latency)
			goto out;
but for some reason it didn't (or there is a problem in the tracing
code, not showing events that would have increased 'latency').

Thomas

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Thomas Charbonnel]

Ingo Molnar wrote :
> here's -P2:
> 
>  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8.1-P2
> 
> Changes since -P1:
> 
>  - trace interrupted kernel code (via hardirqs, NMIs and pagefaults)
> 
>  - yet another shot at trying to fix the IO-APIC/USB issues.
> 
>  - mcount speedups - tracing should be faster
> 
> 	Ingo
> -

The next problem I have relates to irq sharing. 
On my laptop I can't avoid it :
 10:    1070631          XT-PIC  yenta, yenta, uhci_hcd, Intel
82801CA-ICH3, hdsp, eth0
If I set the sound card's interrupt to be non threaded, then I get a
rather long non preemptible section :
http://www.undata.org/~thomas/irq_sharing.trace

As a side note, and this has already been reported here several times,
the SA_INTERRUPT flag set notably by the sound card drivers handlers is
not honored on current kernels if the device is not the first one to be
registered. A simple fix would be to add SA_INTERRUPT handlers at the
beginning instead of the end of the irq queue in setup_irq.

Similarly, when using SA_SAMPLE_RANDOM, all devices on the given irq
contribute to the entropy, even those that have a predictable interrupt
rate (e.g. sound cards), and/or for which the number of interrupts could
outweight the number of interrupts of the original SA_SAMPLE_RANDOM
driver. 

Thomas

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Takashi Iwai]

At Wed, 18 Aug 2004 14:22:07 +0200,
Thomas Charbonnel wrote:
> 
> As a side note, and this has already been reported here several times,
> the SA_INTERRUPT flag set notably by the sound card drivers handlers is
> not honored on current kernels if the device is not the first one to be
> registered. A simple fix would be to add SA_INTERRUPT handlers at the
> beginning instead of the end of the irq queue in setup_irq.
> 
> Similarly, when using SA_SAMPLE_RANDOM, all devices on the given irq
> contribute to the entropy, even those that have a predictable interrupt
> rate (e.g. sound cards), and/or for which the number of interrupts could
> outweight the number of interrupts of the original SA_SAMPLE_RANDOM
> driver. 

Hmm, something like this?


Takashi

--- linux/arch/i386/kernel/irq.c	2004-08-18 15:15:18.272067276 +0200
+++ linux/arch/i386/kernel/irq.c	2004-08-18 15:26:13.513979698 +0200
@@ -219,15 +219,22 @@ inline void synchronize_irq(unsigned int
 asmlinkage int handle_IRQ_event(unsigned int irq,
 		struct pt_regs *regs, struct irqaction *action)
 {
-	int status = 1;	/* Force the "do bottom halves" bit */
-	int retval = 0;
-
-	if (!(action->flags & SA_INTERRUPT))
+	int status;
+	int ret, retval = 0;
+	struct irqaction *ac;
+
+	status = 0;
+	for (ac = action; ac; ac = ac->next)
+		status |= ac->flags;
+	if (!(status & SA_INTERRUPT))
 		local_irq_enable();
 
+	status = 0;
 	do {
-		status |= action->flags;
-		retval |= action->handler(irq, action->dev_id, regs);
+		ret = action->handler(irq, action->dev_id, regs);
+		if (ret)
+			status |= action->flags;
+		retval |= ret;
 		action = action->next;
 	} while (action);
 	if (status & SA_SAMPLE_RANDOM)

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Thomas Charbonnel]

Takashi Iwai wrote :
> At Wed, 18 Aug 2004 14:22:07 +0200,
> Thomas Charbonnel wrote:
> > 
> > As a side note, and this has already been reported here several times,
> > the SA_INTERRUPT flag set notably by the sound card drivers handlers is
> > not honored on current kernels if the device is not the first one to be
> > registered. A simple fix would be to add SA_INTERRUPT handlers at the
> > beginning instead of the end of the irq queue in setup_irq.
> > 
> > Similarly, when using SA_SAMPLE_RANDOM, all devices on the given irq
> > contribute to the entropy, even those that have a predictable interrupt
> > rate (e.g. sound cards), and/or for which the number of interrupts could
> > outweight the number of interrupts of the original SA_SAMPLE_RANDOM
> > driver. 
> 
> Hmm, something like this?
> 
> 
> Takashi
> 
> --- linux/arch/i386/kernel/irq.c	2004-08-18 15:15:18.272067276 +0200
> +++ linux/arch/i386/kernel/irq.c	2004-08-18 15:26:13.513979698 +0200
> @@ -219,15 +219,22 @@ inline void synchronize_irq(unsigned int
>  asmlinkage int handle_IRQ_event(unsigned int irq,
>  		struct pt_regs *regs, struct irqaction *action)
>  {
> -	int status = 1;	/* Force the "do bottom halves" bit */
> -	int retval = 0;
> -
> -	if (!(action->flags & SA_INTERRUPT))
> +	int status;
> +	int ret, retval = 0;
> +	struct irqaction *ac;
> +
> +	status = 0;
> +	for (ac = action; ac; ac = ac->next)
> +		status |= ac->flags;
> +	if (!(status & SA_INTERRUPT))
>  		local_irq_enable();
>  
> +	status = 0;
>  	do {
> -		status |= action->flags;
> -		retval |= action->handler(irq, action->dev_id, regs);
> +		ret = action->handler(irq, action->dev_id, regs);
> +		if (ret)
> +			status |= action->flags;
> +		retval |= ret;
>  		action = action->next;
>  	} while (action);
>  	if (status & SA_SAMPLE_RANDOM)

I was thinking more of something like this :

--- irq.c.orig  2004-08-16 14:26:34.000000000 +0200
+++ irq.c       2004-08-18 17:23:18.011059064 +0200
@@ -955,11 +955,16 @@
                        return -EBUSY;
                }

-               /* add new interrupt at end of irq queue */
-               do {
-                       p = &old->next;
-                       old = *p;
-               } while (old);
+               if (new->flags & SA_INTERRUPT)
+                       /* add interrupt at the start of the queue */
+                       new->next = old;
+               else
+                       /* add new interrupt at end of irq queue */
+                       do {
+                               p = &old->next;
+                               old = *p;
+                       } while (old);
+
                shared = 1;
        }

Thomas

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Takashi Iwai]

At Wed, 18 Aug 2004 17:26:16 +0200,
Thomas Charbonnel wrote:
> 
> I was thinking more of something like this :
> 
> --- irq.c.orig  2004-08-16 14:26:34.000000000 +0200
> +++ irq.c       2004-08-18 17:23:18.011059064 +0200
> @@ -955,11 +955,16 @@
>                         return -EBUSY;
>                 }
> 
> -               /* add new interrupt at end of irq queue */
> -               do {
> -                       p = &old->next;
> -                       old = *p;
> -               } while (old);
> +               if (new->flags & SA_INTERRUPT)
> +                       /* add interrupt at the start of the queue */
> +                       new->next = old;
> +               else
> +                       /* add new interrupt at end of irq queue */
> +                       do {
> +                               p = &old->next;
> +                               old = *p;
> +                       } while (old);
> +
>                 shared = 1;
>         }

Ok, how about this one?  Together with your patch, the irq is turned
on for handlers without SA_INTERRUPT.


Takashi

--- linux/arch/i386/kernel/irq.c	2004-08-18 15:15:18.000000000 +0200
+++ linux/arch/i386/kernel/irq.c	2004-08-18 17:42:57.454819403 +0200
@@ -219,15 +219,23 @@ inline void synchronize_irq(unsigned int
 asmlinkage int handle_IRQ_event(unsigned int irq,
 		struct pt_regs *regs, struct irqaction *action)
 {
-	int status = 1;	/* Force the "do bottom halves" bit */
-	int retval = 0;
-
-	if (!(action->flags & SA_INTERRUPT))
-		local_irq_enable();
+	int status;
+	int ret, retval = 0;
+	int irq_off = 1;
 
+	status = 0;
 	do {
-		status |= action->flags;
-		retval |= action->handler(irq, action->dev_id, regs);
+		/* Assume that all SA_INTERRUPT handlers are at the head
+		 * of the irq queue
+		 */
+		if (irq_off && ! (action->flags & SA_INTERRUPT)) {
+			local_irq_enable();
+			irq_off = 0;
+		}
+		ret = action->handler(irq, action->dev_id, regs);
+		if (ret)
+			status |= action->flags;
+		retval |= ret;
 		action = action->next;
 	} while (action);
 	if (status & SA_SAMPLE_RANDOM)
@@ -955,11 +963,16 @@ int setup_irq(unsigned int irq, struct i
 			return -EBUSY;
 		}
 
-		/* add new interrupt at end of irq queue */
-		do {
-			p = &old->next;
-			old = *p;
-		} while (old);
+		if (new->flags & SA_INTERRUPT)
+			/* add interrupt at the start of the queue */
+			new->next = old;
+		else
+			/* add new interrupt at end of irq queue */
+			do {
+				p = &old->next;
+				old = *p;
+			} while (old);
+
 		shared = 1;
 	}
 

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Thomas Charbonnel]

Takashi Iwai wrote :

[...]
> Ok, how about this one?  Together with your patch, the irq is turned
> on for handlers without SA_INTERRUPT.
> 
> 
> Takashi
> 
> --- linux/arch/i386/kernel/irq.c	2004-08-18 15:15:18.000000000 +0200
> +++ linux/arch/i386/kernel/irq.c	2004-08-18 17:42:57.454819403 +0200
> @@ -219,15 +219,23 @@ inline void synchronize_irq(unsigned int
>  asmlinkage int handle_IRQ_event(unsigned int irq,
>  		struct pt_regs *regs, struct irqaction *action)
>  {
> -	int status = 1;	/* Force the "do bottom halves" bit */
> -	int retval = 0;
> -
> -	if (!(action->flags & SA_INTERRUPT))
> -		local_irq_enable();
> +	int status;
> +	int ret, retval = 0;
> +	int irq_off = 1;
>  
> +	status = 0;
>  	do {
> -		status |= action->flags;
> -		retval |= action->handler(irq, action->dev_id, regs);
> +		/* Assume that all SA_INTERRUPT handlers are at the head
> +		 * of the irq queue
> +		 */
> +		if (irq_off && ! (action->flags & SA_INTERRUPT)) {
> +			local_irq_enable();
> +			irq_off = 0;
> +		}
> +		ret = action->handler(irq, action->dev_id, regs);
> +		if (ret)
> +			status |= action->flags;
> +		retval |= ret;
>  		action = action->next;
>  	} while (action);
>  	if (status & SA_SAMPLE_RANDOM)
> @@ -955,11 +963,16 @@ int setup_irq(unsigned int irq, struct i
>  			return -EBUSY;
>  		}
>  
> -		/* add new interrupt at end of irq queue */
> -		do {
> -			p = &old->next;
> -			old = *p;
> -		} while (old);
> +		if (new->flags & SA_INTERRUPT)
> +			/* add interrupt at the start of the queue */
> +			new->next = old;
> +		else
> +			/* add new interrupt at end of irq queue */
> +			do {
> +				p = &old->next;
> +				old = *p;
> +			} while (old);
> +
>  		shared = 1;
>  	}
>  

You're right, of course, my mistake. It looks good now.

Thomas

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Ingo Molnar]

* Thomas Charbonnel <thomas@undata.org> wrote:

> The next problem I have relates to irq sharing. 
> On my laptop I can't avoid it :
>  10:    1070631          XT-PIC  yenta, yenta, uhci_hcd, Intel
> 82801CA-ICH3, hdsp, eth0
> If I set the sound card's interrupt to be non threaded, then I get a
> rather long non preemptible section :
> http://www.undata.org/~thomas/irq_sharing.trace

i'm not sure the IRQ sharing problem can be solved.

we could execute certain handlers immediately, and defer others to an
IRQ thread. But when we defer an IRQ we must keep the IRQ masked - which
prevents further interrupts (possibly from a high-prio non-threaded
handler) to be executed. So we'd see similar (or in fact worse, due to
the redirection cost) latencies than with the current 'all or nothing'
approach.

now in theory we only have to keep the IRQ line masked for level
triggered interrupts (most APIC interrupts are level-triggered). 
Edge-triggered interrupts (such as the XT-PIC ones you have) could be
acked immediately. I'll try to do something later, but right now there
are still some IRQ problems (USB issues, PS2 mouse/keyboard issues) so
i'd not like to complicate the design just yet.

	Ingo

Re: [patch] voluntary-preempt-2.6.8.1-P2

[Ingo Molnar]

* Thomas Charbonnel <thomas@undata.org> wrote:

> When entering check_preempt_timing, preempt_thresh was 0, and
> preempt_max_latency had been freshly reset to 100. It should have
> triggered this code :
>
> 		if (latency < preempt_max_latency)
> 			goto out;
>
> but for some reason it didn't (or there is a problem in the tracing
> code, not showing events that would have increased 'latency').

there is one case where we could 'miss' a new latency: when
/proc/latency_trace is accessed. For the duration of /proc/latency_trace
access, the updating of the max latency is stopped:

        if (down_trylock(&max_mutex))
                goto out;

this is not really a practical problem and fixing it would be quite 
complex.

	Ingo