Re: [patch] voluntary-preempt-2.6.9-rc2-mm1-S2

Re: [patch] voluntary-preempt-2.6.9-rc2-mm1-S2

[Tim Savannah]

Reports in my dmesg from using voluntary-preempt-2.6.9-rc2-mm1-S2

This doesn't include a bunch of reports from nvidia.

using smp_processor_id() in preemptible code: rmmod/3329
 [<c011d200>] smp_processor_id+0x84/0x8a
 [<c013b527>] __stop_machine_run+0xc3/0xc7
 [<c01376a3>] __try_stop_module+0x0/0x42
 [<c013b530>] stop_machine_run+0x5/0x18
 [<c013769f>] try_stop_module+0x1f/0x23
 [<c0137854>] sys_delete_module+0xef/0x171
 [<c014d87f>] unmap_vma_list+0xe/0x17
 [<c014dbe9>] do_munmap+0x11a/0x176
 [<c0104049>] sysenter_past_esp+0x52/0x71

using smp_processor_id() in preemptible code: gkrellm/3542
 [<c011d200>] smp_processor_id+0x84/0x8a
 [<c02dc66a>] disk_round_stats+0x23/0x8d
 [<c02df0cf>] diskstats_show+0x17/0x312
 [<c0243de6>] inode_has_perm+0x53/0x87
 [<c024668f>] selinux_file_mmap+0x27/0x135
 [<c0275834>] __copy_to_user_ll+0x52/0x61
 [<c02758e3>] copy_to_user+0x40/0x53
 [<c0161bba>] cp_new_stat64+0xf0/0x102
 [<c0246280>] selinux_file_permission+0x111/0x166
 [<c01750cb>] seq_read+0xc3/0x273
 [<c0158a3a>] vfs_read+0xcd/0x126
 [<c0158d00>] sys_read+0x41/0x6a
 [<c0104049>] sysenter_past_esp+0x52/0x71

using smp_processor_id() in preemptible code: gkrellm/3542
 [<c011d200>] smp_processor_id+0x84/0x8a
 [<c02dc66a>] disk_round_stats+0x23/0x8d
 [<c02df0cf>] diskstats_show+0x17/0x312
 [<c01751db>] seq_read+0x1d3/0x273
 [<c0158a3a>] vfs_read+0xcd/0x126
 [<c0158d00>] sys_read+0x41/0x6a
 [<c0104049>] sysenter_past_esp+0x52/0x71

Re: [patch] voluntary-preempt-2.6.9-rc1-bk12-R6

[Ingo Molnar]

* Alexander Nyberg <alexn@dsv.su.se> wrote:

> It doesn't look like it is fully ported to x86_64 systems yet, these
> compile errors are easy to move away but the functionality doesn't
> seem to be there. Probably why Ingo hasn't added the PREEMPT_VOLUNTARY
> to the x86_64 Kconfig even though I saw a few bits of x86_64 code in
> the patch.

yeah, it probably doesnt compile on anything other than x86 right now.

	Ingo

[patch] voluntary-preempt-2.6.9-rc1-bk12-R7

[Ingo Molnar]

i've ported the VP patch to x64. I havent boot-tested it, but it
compiles cleanly and it might even boot:

  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc1-bk12-R7

Caveats: normal kernel with PREEMPT, PREEMPT_VOLUNTARY, PREEMPT_SOFTIRQS
and PREEMPT_HARDIRQS disabled should work just fine.

A kernel with PREEMPT=y and PREEMPT_VOLUNTARY=y ought to work too - with
a smaller probability though. PREEMPT_SOFTIRQS=y should be the next step
- this one might work too. (PREEMPT_HARDIRQS=y doesnt do anything on x64
yet, because i havent changed the irq code. I'd like to keep non-x86
changes small, unless a developer picks it up - like it happened for the
ppc and ppc64 port of the VP patch.)

PREEMPT_TIMING=y might work too if the previous ones worked. The most
problematic one is probably LATENCY_TRACE=y - i've added the proper
mcount assembly code but mostly blindly. It does compile.

so please try this kernel on real hw and try to figure out step by step
at which stage in the following order of parameters it breaks:

 PREEMPT=y
 PREEMPT_VOLUNTARY=y
 PREEMPT_SOFTIRQS=y
 PREEMPT_TIMING=y
 LATENCY_TRACE=y

(when enabling a new option in this sequence keep all the previous
options enabled.)

Worst-case it already breaks with all these options disabled - in this
case please double-check whether vanilla -bk12 x64 boots fine with the
same .config.

Best-case it works fine with all options enabled - quite unlikely.

If it breaks it will break early and hard during bootup, so data is
probably not at risk - but be careful nevertheless.

to get a 2.6.9-rc1-bk12 kernel the patching order is:

    http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.8.tar.bz2
  + http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.9-rc1.bz2
  + http://redhat.com/~mingo/voluntary-preempt/patch-2.6.9-rc1-bk12.bz2

	Ingo

[patch] voluntary-preempt-2.6.9-rc1-bk12-R8

[Ingo Molnar]

test-booted the x64 kernel and found a number of bugs in the x64 port of
the VP patch. I've uploaded -R8 that fixes them:

  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc1-bk12-R8

NOTE: i tested a (non-modular) 64-bit bzImage on a 32-bit OS (FC2) but
havent booted it on a 64-bit userland yet. But i'd expect 64-bit
userspace to work just fine too.

to get a 2.6.9-rc1-bk12 kernel the patching order is:

    http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.8.tar.bz2
  + http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.9-rc1.bz2
  + http://redhat.com/~mingo/voluntary-preempt/patch-2.6.9-rc1-bk12.bz2 

	Ingo

Re: [patch] voluntary-preempt-2.6.9-rc1-bk12-R8

[Lee Revell]

On Tue, 2004-09-07 at 07:57, Ingo Molnar wrote:
> test-booted the x64 kernel and found a number of bugs in the x64 port of
> the VP patch. I've uploaded -R8 that fixes them:
> 
>   http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc1-bk12-R8
> 

Does not work on 32 bit x86:

  CHK     include/linux/compile.h
  UPD     include/linux/compile.h
  CC      init/version.o
  LD      init/built-in.o
  LD      .tmp_vmlinux1
kernel/built-in.o(.init.text+0xcbf): In function `interruptible_sleep_on':
kernel/sched.c:1563: undefined reference to `init_irq_proc'
make: *** [.tmp_vmlinux1] Error 1

Lee

[patch] voluntary-preempt-2.6.9-rc1-bk12-R9

[Ingo Molnar]

* Lee Revell <rlrevell@joe-job.com> wrote:

> Does not work on 32 bit x86:
> 
>   CHK     include/linux/compile.h
>   UPD     include/linux/compile.h
>   CC      init/version.o
>   LD      init/built-in.o
>   LD      .tmp_vmlinux1
> kernel/built-in.o(.init.text+0xcbf): In function `interruptible_sleep_on':
> kernel/sched.c:1563: undefined reference to `init_irq_proc'
> make: *** [.tmp_vmlinux1] Error 1

does -R9 work for you:

  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc1-bk12-R9

to get a 2.6.9-rc1-bk12 kernel the patching order is:

    http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.8.tar.bz2
  + http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.9-rc1.bz2
  + http://redhat.com/~mingo/voluntary-preempt/patch-2.6.9-rc1-bk12.bz2 

	Ingo

Re: [patch] voluntary-preempt-2.6.9-rc1-bk12-R9

[Lee Revell]

On Wed, 2004-09-08 at 04:20, Ingo Molnar wrote:
> does -R9 work for you:
> 

No, same error:

  LD      init/built-in.o
  LD      .tmp_vmlinux1
kernel/built-in.o(.init.text+0xcbf): In function `interruptible_sleep_on':
kernel/sched.c:1563: undefined reference to `init_irq_proc'
make: *** [.tmp_vmlinux1] Error 1

Here is the change that is responsible.  R6 compiles:

rlrevell@mindpipe:~/kernel-source/linux-2.6.9-rc1-bk12-R8$ grep init_irq_proc ../voluntary-preempt-2.6.9-rc1-bk12-R6 
-void init_irq_proc (void)
-void init_irq_proc (void)
-void init_irq_proc (void)
+void init_irq_proc (void)

R8 and later do not:

rlrevell@mindpipe:~/kernel-source/linux-2.6.9-rc1-bk12-R8$ grep init_irq_proc ../voluntary-preempt-2.6.9-rc1-bk12-R9
-void init_irq_proc (void)
-void init_irq_proc (void)
-void init_irq_proc (void)
+extern void generic_init_irq_proc(void);
+static inline void init_irq_proc(void)
+	generic_init_irq_proc();
+void generic_init_irq_proc(void)

Lee

[patch] voluntary-preempt-2.6.9-rc1-bk12-S0

[Ingo Molnar]

* Lee Revell <rlrevell@joe-job.com> wrote:

>   LD      .tmp_vmlinux1
> kernel/built-in.o(.init.text+0xcbf): In function `interruptible_sleep_on':
> kernel/sched.c:1563: undefined reference to `init_irq_proc'
> make: *** [.tmp_vmlinux1] Error 1

could you try -S0:

does -R9 work for you:

  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc1-bk12-S0

[ to get a 2.6.9-rc1-bk12 kernel the patching order is:

    http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.8.tar.bz2
  + http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.9-rc1.bz2
  + http://redhat.com/~mingo/voluntary-preempt/patch-2.6.9-rc1-bk12.bz2 ]

	Ingo

[patch] voluntary-preempt-2.6.9-rc2-mm1-S1

[Ingo Molnar]

i've released the -S1 VP patch:

  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc2-mm1-S1

NOTE: this patch is against Andrew's -mm tree and the VP patchset will
stay based on -mm until the merging process has been finished.

to get a 2.6.9-rc2-mm1-VP-S1 kernel, the patching order is:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.8.tar.bz2
 + http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.9-rc2.bz2
 + http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9-rc2/2.6.9-rc2-mm1/2.6.9-rc2-mm1.bz2
 + http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc2-mm1-S1

Changes relative to -S0:

- lots of merging. A good chunk of the VP patch latency breakers and
  support patches are in -mm already.

- integrated my 'preemptible big kernel lock' patch into VP. This makes 
  all BKL code preemptible while keeping correctness. A new debugging 
  infrastructure has been added to catch code that might use the BKL
  in an unsafe way. If the debugging check triggers it will print 
  messages like:

    using smp_processor_id() in preemptible code: bash/1020

  please report such messages and backtraces to me. Most of the messages 
  i've fixed so far were false positives, but one bug has been caught 
  already via this.

  Also, this BKL patch allowed the removal of two questionable 
  latency breakers: the tty.c and the DRM BKL relaxation hack.

- fixed an SMP hardirq redirection bug - IRQ threads could be bound to 
  multiple CPUs resulting in potentially illegal preemption of hardirq
  contexts.

- temporarily dropped the ppc/ppc64 GENERIC_HARDIRQS changes, they broke
  and i cannot test them.

Reports, comments welcome,

	Ingo

BKL backtraces - was: Re: [patch] voluntary-preempt-2.6.9-rc2-mm1-S1

[K.R. Foley]

Ingo Molnar wrote:
> i've released the -S1 VP patch:
> 
>   http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc2-mm1-S1
> 

All of these were generated while booting:

Sep 20 19:45:10 porky kernel: using smp_processor_id() in preemptible 
code: modprobe/1019
Sep 20 19:45:10 porky kernel:  [<c011c58e>] smp_processor_id+0x8e/0xa0
Sep 20 19:45:10 porky kernel:  [<c013ace6>] module_unload_init+0x46/0x70
Sep 20 19:45:10 porky kernel:  [<c013ce58>] load_module+0x598/0xb10
Sep 20 19:45:10 porky kernel:  [<c013d438>] sys_init_module+0x68/0x280
Sep 20 19:45:10 porky kernel:  [<c01066b9>] sysenter_past_esp+0x52/0x71

The above one of course repeats on each module load.

Sep 20 19:45:10 porky kernel: using smp_processor_id() in preemptible 
code: X/1017
Sep 20 19:45:10 porky kernel:  [<c011c58e>] smp_processor_id+0x8e/0xa0
Sep 20 19:45:10 porky kernel:  [<c01d6c15>] add_timer_randomness+0x125/0x150
Sep 20 19:45:10 porky kernel:  [<c01d6c9e>] add_mouse_randomness+0x1e/0x30
Sep 20 19:45:10 porky kernel:  [<c022b835>] input_event+0x55/0x3f0
Sep 20 19:45:10 porky kernel:  [<c01151e8>] mcount+0x14/0x18
Sep 20 19:45:10 porky kernel:  [<c01e559e>] kbd_rate+0x5e/0xc0
Sep 20 19:45:10 porky kernel:  [<c01e2196>] vt_ioctl+0xe06/0x1ad0
Sep 20 19:45:10 porky kernel:  [<c014fdcf>] pte_alloc_map+0x9f/0xd0
Sep 20 19:45:10 porky kernel:  [<c015214b>] handle_mm_fault+0x17b/0x1a0
Sep 20 19:45:10 porky kernel:  [<c0119440>] do_page_fault+0x1e0/0x621
Sep 20 19:45:10 porky kernel:  [<c0138759>] sub_preempt_count+0x69/0x80
Sep 20 19:45:10 porky kernel:  [<c0138759>] sub_preempt_count+0x69/0x80
Sep 20 19:45:10 porky kernel:  [<c0138512>] check_preempt_timing+0x192/0x200
Sep 20 19:45:10 porky kernel:  [<c0175034>] sys_ioctl+0xe4/0x240
Sep 20 19:45:10 porky kernel:  [<c01dbd1e>] tty_ioctl+0xe/0x4d0
Sep 20 19:45:10 porky kernel:  [<c01151e8>] mcount+0x14/0x18
Sep 20 19:45:10 porky kernel:  [<c01e1390>] vt_ioctl+0x0/0x1ad0
Sep 20 19:45:10 porky kernel:  [<c01dc08b>] tty_ioctl+0x37b/0x4d0
Sep 20 19:45:10 porky kernel:  [<c0175034>] sys_ioctl+0xe4/0x240
Sep 20 19:45:10 porky kernel:  [<c01066b9>] sysenter_past_esp+0x52/0x71

The X one above repeats once also.

kr

Re: BKL backtraces - was: Re: [patch] voluntary-preempt-2.6.9-rc2-mm1-S1

[Ingo Molnar]

[-- Attachment #1: Type: text/plain, Size: 1569 bytes --]


* K.R. Foley <kr@cybsft.com> wrote:

> All of these were generated while booting:
> 
> Sep 20 19:45:10 porky kernel: using smp_processor_id() in preemptible 
> code: modprobe/1019
> Sep 20 19:45:10 porky kernel:  [<c011c58e>] smp_processor_id+0x8e/0xa0
> Sep 20 19:45:10 porky kernel:  [<c013ace6>] module_unload_init+0x46/0x70
> Sep 20 19:45:10 porky kernel:  [<c013ce58>] load_module+0x598/0xb10
> Sep 20 19:45:10 porky kernel:  [<c013d438>] sys_init_module+0x68/0x280
> Sep 20 19:45:10 porky kernel:  [<c01066b9>] sysenter_past_esp+0x52/0x71
> 
> The above one of course repeats on each module load.

ok, this is a harmless false positive - the attached patch fixes it.

> Sep 20 19:45:10 porky kernel: using smp_processor_id() in preemptible 
> code: X/1017
> Sep 20 19:45:10 porky kernel:  [<c011c58e>] smp_processor_id+0x8e/0xa0
> Sep 20 19:45:10 porky kernel:  [<c01d6c15>] add_timer_randomness+0x125/0x150
> Sep 20 19:45:10 porky kernel:  [<c01d6c9e>] add_mouse_randomness+0x1e/0x30
> Sep 20 19:45:10 porky kernel:  [<c022b835>] input_event+0x55/0x3f0

> Sep 20 19:45:10 porky kernel:  [<c01e1390>] vt_ioctl+0x0/0x1ad0
> Sep 20 19:45:10 porky kernel:  [<c01dc08b>] tty_ioctl+0x37b/0x4d0
> Sep 20 19:45:10 porky kernel:  [<c0175034>] sys_ioctl+0xe4/0x240
> Sep 20 19:45:10 porky kernel:  [<c01066b9>] sysenter_past_esp+0x52/0x71
> 
> The X one above repeats once also.

aha! This is a real one, fixed by the second patch. This piece of code
relied on add_timer_randomness() always being called with preemption
disabled.

these fixes will show up in -S2.

	Ingo

[-- Attachment #2: 1 --]
[-- Type: text/plain, Size: 441 bytes --]

--- linux/kernel/module.c.orig	
+++ linux/kernel/module.c	
@@ -394,7 +394,7 @@ static void module_unload_init(struct mo
 	for (i = 0; i < NR_CPUS; i++)
 		local_set(&mod->ref[i].count, 0);
 	/* Hold reference count during initialization. */
-	local_set(&mod->ref[smp_processor_id()].count, 1);
+	local_set(&mod->ref[_smp_processor_id()].count, 1);
 	/* Backwards compatibility macros put refcount during init. */
 	mod->waiter = current;
 }

[-- Attachment #3: 2 --]
[-- Type: text/plain, Size: 698 bytes --]

--- linux/drivers/char/random.c.orig	
+++ linux/drivers/char/random.c	
@@ -807,10 +807,11 @@ static void add_timer_randomness(struct 
 	long		delta, delta2, delta3;
 	int		entropy = 0;
 
+	preempt_disable();
 	/* if over the trickle threshold, use only 1 in 4096 samples */
 	if ( random_state->entropy_count > trickle_thresh &&
 	     (__get_cpu_var(trickle_count)++ & 0xfff))
-		return;
+		goto out;
 
 	/*
 	 * Use get_cycles() if implemented, otherwise fall back to
@@ -861,6 +862,8 @@ static void add_timer_randomness(struct 
 		entropy = int_ln_12bits(delta);
 	}
 	batch_entropy_store(num, time, entropy);
+out:
+	preempt_enable();
 }
 
 void add_keyboard_randomness(unsigned char scancode)

[patch] voluntary-preempt-2.6.9-rc2-mm1-S2

[Ingo Molnar]

i've released the -S2 VP patch:

  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc2-mm1-S2

Changes since -S1:

 - added the swapspace-layout patch to fix the get_swap_page()
   latencies. (sw-suspend wont compile but everything else should work
   fine.)

 - fixed the random.c BKL non-preemptability assumption

 - export smp_processor_id() to fix modules

 - module init smp_processor_id()-debug false positive fix

To get a 2.6.9-rc2-mm1-VP-S2 kernel, the patching order is:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.8.tar.bz2
 + http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.9-rc2.bz2
 + http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9-rc2/2.6.9-rc2-mm1/2.6.9-rc2-mm1.bz2
 + http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc2-mm1-S2

	Ingo

Re: [patch] voluntary-preempt-2.6.9-rc2-mm1-S2

[K.R. Foley]

Ingo Molnar wrote:
> i've released the -S2 VP patch:
> 
>   http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc2-mm1-S2
> 

Another smp_processor_id in modprobe. Now I see these for every 
modprobe. Is this a different global lock?

Sep 21 13:27:53 porky kernel: using smp_processor_id() in preemptible 
code: modprobe/1737
Sep 21 13:27:53 porky kernel:  [<c011c58e>] smp_processor_id+0x8e/0xa0
Sep 21 13:27:53 porky kernel:  [<c01401e5>] __stop_machine_run+0xb5/0xc0
Sep 21 13:27:53 porky kernel:  [<c013de30>] __try_stop_module+0x0/0x46
Sep 21 13:27:53 porky kernel:  [<c01151e8>] mcount+0x14/0x18
Sep 21 13:27:53 porky kernel:  [<c0140214>] stop_machine_run+0x24/0x3d
Sep 21 13:27:53 porky kernel:  [<c013de30>] __try_stop_module+0x0/0x46
Sep 21 13:27:53 porky kernel:  [<c013b019>] try_stop_module+0x39/0x40
Sep 21 13:27:53 porky kernel:  [<c013de30>] __try_stop_module+0x0/0x46
Sep 21 13:27:53 porky kernel:  [<c013b1e0>] sys_delete_module+0x110/0x180
Sep 21 13:27:53 porky kernel:  [<c0154c09>] sys_munmap+0x59/0x80
Sep 21 13:27:53 porky kernel:  [<c01066b9>] sysenter_past_esp+0x52/0x71

kr