All of lore.kernel.org
 help / color / mirror / Atom feed
* [patch] voluntary-preempt-2.6.9-rc3-mm2-T1
  2004-10-04 21:53                 ` [patch] voluntary-preempt-2.6.9-rc3-mm1-S9 Ingo Molnar
@ 2004-10-05 13:47                   ` Ingo Molnar
  2004-10-05 16:37                     ` Rui Nuno Capela
  2004-10-08 11:16                     ` William Lee Irwin III
  0 siblings, 2 replies; 12+ messages in thread
From: Ingo Molnar @ 2004-10-05 13:47 UTC (permalink / raw)
  To: linux-kernel; +Cc: Lee Revell, K.R. Foley, Rui Nuno Capela, Florian Schmidt


i've released the -T1 VP patch:

  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc3-mm2-T1

Changes since -T0:

 - added the read-lock fix from Hugh that affects SMP systems. This 
   could fix Rui's problem - i've checked -T1 on a P4/HT box and saw no 
   problems, BYMMV.

 - compilation fixes (for those who downloaded T0 early)

 - small tracer improvement

to build a -T1 tree from scratch the patching order is:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.8.tar.bz2
 + http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.9-rc3.bz2
 + http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9-rc3/2.6.9-rc3-mm2/2.6.9-rc3-mm2.bz2
 + http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc3-mm2-T1

	Ingo

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [patch] voluntary-preempt-2.6.9-rc3-mm2-T1
  2004-10-05 13:47                   ` [patch] voluntary-preempt-2.6.9-rc3-mm2-T1 Ingo Molnar
@ 2004-10-05 16:37                     ` Rui Nuno Capela
  2004-10-05 18:42                       ` Ingo Molnar
  2004-10-08 11:16                     ` William Lee Irwin III
  1 sibling, 1 reply; 12+ messages in thread
From: Rui Nuno Capela @ 2004-10-05 16:37 UTC (permalink / raw)
  To: Ingo Molnar; +Cc: linux-kernel, Lee Revell, K.R. Foley, Florian Schmidt

[-- Attachment #1: Type: text/plain, Size: 745 bytes --]

Ingo Molnar wrote:
>
> i've released the -T1 VP patch:
>
http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc3-mm2-T1
>
> Changes since -T0:
>
>  - added the read-lock fix from Hugh that affects SMP systems. This
>    could fix Rui's problem - i've checked -T1 on a P4/HT box and saw
>    no problems, BYMMV.
>

No go here. My milleage certainly varies :)

Still the same ugliness here with T1. As before, there goes some info
attached, which I could collect while barely up and running.

I don't wanna be a PITA, but changes are hardly noticed since T0.

Next I'll try building with CONFIG_SCHED_SMT off, and later with
CONFIG_SMP off too, to just try to make some point here.

CU
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

[-- Attachment #2: dump-2.6.9-rc3-mm2-T1.0smp.tar.gz --]
[-- Type: application/x-tgz, Size: 16691 bytes --]

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [patch] voluntary-preempt-2.6.9-rc3-mm2-T1
  2004-10-05 16:37                     ` Rui Nuno Capela
@ 2004-10-05 18:42                       ` Ingo Molnar
  2004-10-05 19:38                         ` Rui Nuno Capela
  0 siblings, 1 reply; 12+ messages in thread
From: Ingo Molnar @ 2004-10-05 18:42 UTC (permalink / raw)
  To: Rui Nuno Capela; +Cc: linux-kernel, Lee Revell, K.R. Foley, Florian Schmidt


* Rui Nuno Capela <rncbc@rncbc.org> wrote:

> Still the same ugliness here with T1. As before, there goes some info
> attached, which I could collect while barely up and running.

the dmesg info shows you had a crash early on, in khubd:

 Badness in remove_proc_entry at fs/proc/generic.c:688
  [<c018c8e8>] remove_proc_entry+0x152/0x15a
  [<f8b8e116>] uhci_hcd_init+0x116/0x133 [uhci_hcd]
  [<c0135f0e>] sys_init_module+0x1df/0x2da
  [<c01044ed>] sysenter_past_esp+0x52/0x71
 usb 3-1: new low speed USB device using address 2
 Unable to handle kernel paging request at virtual address a49c0e0c

i believe this is a crash present in -mm too. In theory such a crash
could mess up the kernel so best would be if you could try a kernel with
USB disabled? Hopefully none of your critical devices is on USB ...

	Ingo

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [patch] voluntary-preempt-2.6.9-rc3-mm2-T1
  2004-10-05 18:42                       ` Ingo Molnar
@ 2004-10-05 19:38                         ` Rui Nuno Capela
  2004-10-05 19:44                           ` Ingo Molnar
  0 siblings, 1 reply; 12+ messages in thread
From: Rui Nuno Capela @ 2004-10-05 19:38 UTC (permalink / raw)
  To: Ingo Molnar; +Cc: linux-kernel, Lee Revell, K.R. Foley, Florian Schmidt

Ingo Molnar wrote:
>
> * Rui Nuno Capela wrote:
>
>> Still the same ugliness here with T1. As before, there goes some info
>> attached, which I could collect while barely up and running.
>
> the dmesg info shows you had a crash early on, in khubd:
>
>  Badness in remove_proc_entry at fs/proc/generic.c:688
>   [<c018c8e8>] remove_proc_entry+0x152/0x15a
>   [<f8b8e116>] uhci_hcd_init+0x116/0x133 [uhci_hcd]
>   [<c0135f0e>] sys_init_module+0x1df/0x2da
>   [<c01044ed>] sysenter_past_esp+0x52/0x71
>  usb 3-1: new low speed USB device using address 2
>  Unable to handle kernel paging request at virtual address a49c0e0c
>
> i believe this is a crash present in -mm too. In theory such a crash
> could mess up the kernel so best would be if you could try a kernel with
> USB disabled? Hopefully none of your critical devices is on USB ...
>

Yeah, I've found it the hard way :) One of my trials discovered just that,
by booting without any USB devices plugged in. It booted apparently fine.

Then, exacltly when I plug in my USB mouse (Wacom Graphire2 tablet), I
immediately get the following kernel oops (on dmesg):

IRQ#23 thread started up.
IRQ#19 thread started up.
usb 2-1: new low speed USB device using address 2
Unable to handle kernel paging request at virtual address aaf7ee8f
 printing eip:
c014284b
*pde = 00000000
Oops: 0000 [#1]
PREEMPT SMP
Modules linked in: mga ohci1394 ieee1394 ehci_hcd uhci_hcd intel_mch_agp
agpgart snd_usb_usx2y snd_usb_lib snd_rawmidi snd_seq_device snd_hwdep
snd_intel8x0 snd_ac97_codec snd_pcm snd_timer snd soundcore snd_page_alloc
evdev sk98lin realtime commoncap w83781d i2c_sensor i2c_isa i2c_i801
i2c_core wacom usbcore subfs dm_mod
CPU:    1
EIP:    0060:[<c014284b>]    Not tainted VLI
EFLAGS: 00010093   (2.6.9-rc3-mm2-T1.0smp)
EIP is at check_poison_obj+0x98/0x1d0
eax: 0000006b   ebx: f6e8a780   ecx: f6d254b8   edx: ffffffa5
esi: 00000000   edi: aaf7ee8f   ebp: f74d3d10   esp: f74d3ce8
ds: 007b   es: 007b   ss: 0068   preempt: 00000002
Process khubd (pid: 1306, threadinfo=f74d3000 task=f7df2a40)
Stack: f6e8a780 f74d3cfc c0130ce3 0000002b c0144658 00000000 0000002c
f6e8a780
       aaf7ee8b f6e8a780 f74d3d34 c01442f5 f6e8a780 aaf7ee8b f6d254b8
f6d254b8
       f6e8a780 00000046 00000020 f74d3d58 c0144658 f6e8a780 00000020
aaf7ee8b
Call Trace:
 [<c0130ce3>] __mcount+0x1d/0x1f
 [<c0144658>] kmem_cache_alloc+0x6b/0xa5
 [<c01442f5>] cache_alloc_debugcheck_after+0x129/0x162
 [<c0144658>] kmem_cache_alloc+0x6b/0xa5
 [<f8c35ff9>] uhci_alloc_urb_priv+0x26/0x81 [uhci_hcd]
 [<f8c35ff9>] uhci_alloc_urb_priv+0x26/0x81 [uhci_hcd]
 [<f8c36fbb>] uhci_urb_enqueue+0x5d/0x2bc [uhci_hcd]
 [<c0110350>] mcount+0x14/0x18
 [<f8b424f4>] hcd_submit_urb+0x127/0x194 [usbcore]
 [<f8b42e38>] usb_submit_urb+0x1e2/0x244 [usbcore]
 [<f8b430c7>] usb_start_wait_urb+0xe/0xe2 [usbcore]
 [<f8b4310c>] usb_start_wait_urb+0x53/0xe2 [usbcore]
 [<c0130ce3>] __mcount+0x1d/0x1f
 [<c01ea660>] kref_init+0x8/0x13
 [<f8b42ba8>] usb_init_urb+0x27/0x3c [usbcore]
 [<c0110350>] mcount+0x14/0x18
 [<f8b42bef>] usb_alloc_urb+0x32/0x52 [usbcore]
 [<f8b43215>] usb_internal_control_msg+0x7a/0x83 [usbcore]
 [<f8b432aa>] usb_control_msg+0x8c/0xa0 [usbcore]
 [<f8b3fe23>] hub_set_address+0x6d/0x90 [usbcore]
 [<f8b3fff7>] hub_port_init+0x1b1/0x39a [usbcore]
 [<f8b404d9>] hub_port_connect_change+0xfe/0x43a [usbcore]
 [<f8b409d3>] hub_events+0x1be/0x395 [usbcore]
 [<f8b40be1>] hub_thread+0x37/0x109 [usbcore]
 [<c0130845>] autoremove_wake_function+0x0/0x50
 [<c0104416>] ret_from_fork+0x6/0x14
 [<c0130845>] autoremove_wake_function+0x0/0x50
 [<f8b40baa>] hub_thread+0x0/0x109 [usbcore]
 [<c0102601>] kernel_thread_helper+0x5/0xb
Code: 89 44 24 08 e8 09 fe ff ff 83 45 ec 01 83 7d ec 05 7f 5b 83 c6 01 3b
75 f0 7d 53 3b 75 e4 b8 6b 00 00 00 ba a5 ff ff ff 0f 44 c2 <38> 04 3e 2e
74 e2 8b 45 ec 85 c0 75 a0 8b 55 f0 89 7c 24 04 c7
 <6>note: khubd[1306] exited with preempt_count 1


Guess what? It's right after this crash that ksoftirqd/1 pulls up to
99.99%CPU#1 and stays still on that figure, forever. Of course, as Ingo
noted, this was happening behind the scenes every time I was boot/initing.

OTOH, I've tested T1 with CONFIG_SCHED_SMT and/or CONFIG_SMP not set, and
got similar crashes too. So this seems to be some nasty bug introduced by
-mm{1,2}, not by VP on SMP/SMT.

Yes, I do have some critical USB devices around here. One is that wacom
tablet (mouse) and the other is a tascam us-224 audio/midi control surface
that a love very much :)

Don't know if this makes me feeling better, doh.

Cheers.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [patch] voluntary-preempt-2.6.9-rc3-mm2-T1
  2004-10-05 19:38                         ` Rui Nuno Capela
@ 2004-10-05 19:44                           ` Ingo Molnar
  2004-10-05 20:01                             ` Rui Nuno Capela
  2004-10-06  0:12                             ` Lee Revell
  0 siblings, 2 replies; 12+ messages in thread
From: Ingo Molnar @ 2004-10-05 19:44 UTC (permalink / raw)
  To: Rui Nuno Capela; +Cc: linux-kernel, Lee Revell, K.R. Foley, Florian Schmidt


* Rui Nuno Capela <rncbc@rncbc.org> wrote:

> OTOH, I've tested T1 with CONFIG_SCHED_SMT and/or CONFIG_SMP not set,
> and got similar crashes too. So this seems to be some nasty bug
> introduced by -mm{1,2}, not by VP on SMP/SMT.
> 
> Yes, I do have some critical USB devices around here. One is that
> wacom tablet (mouse) and the other is a tascam us-224 audio/midi
> control surface that a love very much :)
> 
> Don't know if this makes me feeling better, doh.

i believe Andrew said that these USB problems should be fixed in the 
next -mm iteration.

	Ingo

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [patch] voluntary-preempt-2.6.9-rc3-mm2-T1
  2004-10-05 19:44                           ` Ingo Molnar
@ 2004-10-05 20:01                             ` Rui Nuno Capela
  2004-10-06  0:12                             ` Lee Revell
  1 sibling, 0 replies; 12+ messages in thread
From: Rui Nuno Capela @ 2004-10-05 20:01 UTC (permalink / raw)
  To: Ingo Molnar; +Cc: linux-kernel, Lee Revell, K.R. Foley, Florian Schmidt

Ingo Molnar wrote:
>
> * Rui Nuno Capela wrote:
>
>> OTOH, I've tested T1 with CONFIG_SCHED_SMT and/or CONFIG_SMP not set,
>> and got similar crashes too. So this seems to be some nasty bug
>> introduced by -mm{1,2}, not by VP on SMP/SMT.
>>
>> Yes, I do have some critical USB devices around here. One is that
>> wacom tablet (mouse) and the other is a tascam us-224 audio/midi
>> control surface that a love very much :)
>>
>> Don't know if this makes me feeling better, doh.
>
> i believe Andrew said that these USB problems should be fixed in the
> next -mm iteration.
>

Oh yes, I really do hope so :)

Meanwhile, I'm stuck with 2.6.9-rc2-mm4-S7 (SMP), but happy.

Strange thing is, that on my laptop, 2.6.9-rc3-mm2-S9 (UP) is doing just
fine. Guess that ohci_hcd now makes the difference here, against the
former which makes uhci_hcd bad behaved atm.

Thanks Ingo.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [patch] voluntary-preempt-2.6.9-rc3-mm2-T1
  2004-10-05 19:44                           ` Ingo Molnar
  2004-10-05 20:01                             ` Rui Nuno Capela
@ 2004-10-06  0:12                             ` Lee Revell
  2004-10-06  7:51                               ` Ingo Molnar
  1 sibling, 1 reply; 12+ messages in thread
From: Lee Revell @ 2004-10-06  0:12 UTC (permalink / raw)
  To: Ingo Molnar; +Cc: Rui Nuno Capela, linux-kernel, K.R. Foley, Florian Schmidt

On Tue, 2004-10-05 at 15:44, Ingo Molnar wrote:
> * Rui Nuno Capela <rncbc@rncbc.org> wrote:
> 
> > OTOH, I've tested T1 with CONFIG_SCHED_SMT and/or CONFIG_SMP not set,
> > and got similar crashes too. So this seems to be some nasty bug
> > introduced by -mm{1,2}, not by VP on SMP/SMT.
> > 
> > Yes, I do have some critical USB devices around here. One is that
> > wacom tablet (mouse) and the other is a tascam us-224 audio/midi
> > control surface that a love very much :)
> > 
> > Don't know if this makes me feeling better, doh.
> 
> i believe Andrew said that these USB problems should be fixed in the 
> next -mm iteration.
> 

FWIW, this one does not work for me either, I get a USB-related Oops on
boot.

Lee 


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [patch] voluntary-preempt-2.6.9-rc3-mm2-T1
  2004-10-06  0:12                             ` Lee Revell
@ 2004-10-06  7:51                               ` Ingo Molnar
  0 siblings, 0 replies; 12+ messages in thread
From: Ingo Molnar @ 2004-10-06  7:51 UTC (permalink / raw)
  To: Lee Revell; +Cc: Rui Nuno Capela, linux-kernel, K.R. Foley, Florian Schmidt


* Lee Revell <rlrevell@joe-job.com> wrote:

> > i believe Andrew said that these USB problems should be fixed in the 
> > next -mm iteration.
> > 
> 
> FWIW, this one does not work for me either, I get a USB-related Oops
> on boot.

by next -mm iteration i meant -rc3-mm3, which is not released yet.

	Ingo

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [patch] voluntary-preempt-2.6.9-rc3-mm2-T1
@ 2004-10-06 18:57 Kari Hameenaho
  0 siblings, 0 replies; 12+ messages in thread
From: Kari Hameenaho @ 2004-10-06 18:57 UTC (permalink / raw)
  To: linux-kernel

Ingo Molnar wrote:
 >
 >* Lee Revell <rlrevell@xxxxxxxxxxx> wrote:
 >
 > > > i believe Andrew said that these USB problems should be fixed in
 > > > the next -mm iteration.
 > > >
 > >
 > > FWIW, this one does not work for me either, I get a USB-related Oops
 > > on boot.
 >
 > by next -mm iteration i meant -rc3-mm3, which is not released yet.
 >
 > Ingo
 >

The usb-hubs problem patch in LKML by Greg KH seems to fix this problem.

The thread is 2.6.9-rc3-mm1, bk-pci patch, USB hubs

---
Kari Hämeenaho

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [patch] voluntary-preempt-2.6.9-rc3-mm2-T1
@ 2004-10-06 20:01 Mark_H_Johnson
  2004-10-06 20:07 ` Ingo Molnar
  0 siblings, 1 reply; 12+ messages in thread
From: Mark_H_Johnson @ 2004-10-06 20:01 UTC (permalink / raw)
  To: linux-kernel; +Cc: Ingo Molnar, Rui Nuno Capela

>Meanwhile, I'm stuck with 2.6.9-rc2-mm4-S7 (SMP), but happy.
>
>Strange thing is, that on my laptop, 2.6.9-rc3-mm2-S9 (UP) is doing just
>fine. Guess that ohci_hcd now makes the difference here, against the
>former which makes uhci_hcd bad behaved atm.

I am having similar problems with -T1 and separately reported problems with
a build of rc3-mm1-S8 as well (no oops, but the USB mouse is dead).
Somewhere between those two versions (rc2-mm4-S7 and rc3-mm1-S8) is where
the problem appears to be introduced. For now I'll stay with my working -S0
kernel.

--Mark H Johnson
  <mailto:Mark_H_Johnson@raytheon.com>


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [patch] voluntary-preempt-2.6.9-rc3-mm2-T1
  2004-10-06 20:01 [patch] voluntary-preempt-2.6.9-rc3-mm2-T1 Mark_H_Johnson
@ 2004-10-06 20:07 ` Ingo Molnar
  0 siblings, 0 replies; 12+ messages in thread
From: Ingo Molnar @ 2004-10-06 20:07 UTC (permalink / raw)
  To: Mark_H_Johnson; +Cc: linux-kernel, Rui Nuno Capela


* Mark_H_Johnson@raytheon.com <Mark_H_Johnson@raytheon.com> wrote:

> >Meanwhile, I'm stuck with 2.6.9-rc2-mm4-S7 (SMP), but happy.
> >
> >Strange thing is, that on my laptop, 2.6.9-rc3-mm2-S9 (UP) is doing just
> >fine. Guess that ohci_hcd now makes the difference here, against the
> >former which makes uhci_hcd bad behaved atm.
> 
> I am having similar problems with -T1 and separately reported problems
> with a build of rc3-mm1-S8 as well (no oops, but the USB mouse is
> dead). Somewhere between those two versions (rc2-mm4-S7 and
> rc3-mm1-S8) is where the problem appears to be introduced. For now
> I'll stay with my working -S0 kernel.

disable USB for now - it's broken in -mm and unrelated to -VP. There are
hopes that in -rc3-mm3 USB will work again.

	Ingo

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [patch] voluntary-preempt-2.6.9-rc3-mm2-T1
  2004-10-05 13:47                   ` [patch] voluntary-preempt-2.6.9-rc3-mm2-T1 Ingo Molnar
  2004-10-05 16:37                     ` Rui Nuno Capela
@ 2004-10-08 11:16                     ` William Lee Irwin III
  1 sibling, 0 replies; 12+ messages in thread
From: William Lee Irwin III @ 2004-10-08 11:16 UTC (permalink / raw)
  To: Ingo Molnar
  Cc: linux-kernel, Lee Revell, K.R. Foley, Rui Nuno Capela,
	Florian Schmidt

On Tue, Oct 05, 2004 at 03:47:07PM +0200, Ingo Molnar wrote:
> i've released the -T1 VP patch:
>   http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc3-mm2-T1
> Changes since -T0:
>  - added the read-lock fix from Hugh that affects SMP systems. This 
>    could fix Rui's problem - i've checked -T1 on a P4/HT box and saw no 
>    problems, BYMMV.
>  - compilation fixes (for those who downloaded T0 early)
>  - small tracer improvement
> to build a -T1 tree from scratch the patching order is:
>    http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.8.tar.bz2
>  + http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.9-rc3.bz2
>  + http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9-rc3/2.6.9-rc3-mm2/2.6.9-rc3-mm2.bz2
>  + http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc3-mm2-T1

The version numbers are up to Chebyshev polynomials. Now I have to try it.


-- wli

^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2004-10-08 11:20 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2004-10-06 20:01 [patch] voluntary-preempt-2.6.9-rc3-mm2-T1 Mark_H_Johnson
2004-10-06 20:07 ` Ingo Molnar
  -- strict thread matches above, loose matches on Subject: below --
2004-10-06 18:57 Kari Hameenaho
2004-09-21  2:07 BKL backtraces - was: Re: [patch] voluntary-preempt-2.6.9-rc2-mm1-S1 K.R. Foley
2004-09-21  7:18 ` Ingo Molnar
2004-09-21  7:44   ` [patch] voluntary-preempt-2.6.9-rc2-mm1-S2 Ingo Molnar
2004-09-22 10:33     ` [patch] voluntary-preempt-2.6.9-rc2-mm1-S3 Ingo Molnar
2004-09-23 12:28       ` [patch] voluntary-preempt-2.6.9-rc2-mm1-S4 Ingo Molnar
2004-09-23 21:12         ` [patch] voluntary-preempt-2.6.9-rc2-mm3-S5 Ingo Molnar
2004-09-24  7:44           ` [patch] voluntary-preempt-2.6.9-rc2-mm3-S6 Ingo Molnar
2004-09-28  0:05             ` [patch] voluntary-preempt-2.6.9-rc2-mm4-S7 Ingo Molnar
2004-10-03 21:09               ` [patch] voluntary-preempt-2.6.9-rc3-mm1-S8 Ingo Molnar
2004-10-04 21:53                 ` [patch] voluntary-preempt-2.6.9-rc3-mm1-S9 Ingo Molnar
2004-10-05 13:47                   ` [patch] voluntary-preempt-2.6.9-rc3-mm2-T1 Ingo Molnar
2004-10-05 16:37                     ` Rui Nuno Capela
2004-10-05 18:42                       ` Ingo Molnar
2004-10-05 19:38                         ` Rui Nuno Capela
2004-10-05 19:44                           ` Ingo Molnar
2004-10-05 20:01                             ` Rui Nuno Capela
2004-10-06  0:12                             ` Lee Revell
2004-10-06  7:51                               ` Ingo Molnar
2004-10-08 11:16                     ` William Lee Irwin III

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.