All of lore.kernel.org
 help / color / mirror / Atom feed
* [Xenomai-help] Xenomai and MSI enabled crashes kernel
@ 2007-04-26 12:04 M. Koehrer
  2007-04-27 11:48 ` Jan Kiszka
  2007-04-28 12:54 ` Bernhard Walle
  0 siblings, 2 replies; 52+ messages in thread
From: M. Koehrer @ 2007-04-26 12:04 UTC (permalink / raw)
  To: xenomai


[-- Attachment #1.1: Type: text/plain, Size: 4339 bytes --]

Hi everybody,

I am using kernel 2.6.20.4 and Xenomai 2.3.1 on a SMP enabled dual core Pentium 4.
Everything works fine when I do not enable the CONFIG_PCI_MSI (Messages signaled interrupts) for
PCI Express.
As I am always short with interrupts I want to use MSI for the PCIe base
I/O devices like Ethernet.

When I run the same kernel without any Xenomai patch, it works really fine.
My onboard PCIe e1000 network adapter will be loaded fine and gets the (cool) interrupt number 219!
Perfect!

When I use the Adeos patch from Xenomai 2.3.1 on this kernel the modprobe of the e1000 driver
leads to a kernel crash.
At this stage no real time application is running.
Using a serial console, I was able to log the kernel dump.
Here it is:

BUG: unable to handle
 kernel NULL pointer dereference at virtual address 00000000
 printing eip:
*pde = 00000000
Oops: 0000 [#1]
SMP
Modules linked in: e1000
CPU:    0
EIP:    0060:[<00000000>]    Not tainted VLI
EFLAGS: 00010086   (2.6.20.4 #7)
EIP is at _stext+0x3feffc70/0x14
eax: c0112244   ebx: 00000006   ecx: c011434d   edx: c168c000
esi: 00000006   edi: 00000046   ebp: ffffffff   esp: c168de20
ds: 007b   es: 007b   ss: 0068
Process ifconfig (pid: 1231, ti=c168c000 task=c1692a90 task.ti=c168c000)
Stack: c03e5680 000000db 00000000 c03d9100 c010ef83 00006d80 00000001 00000060
       e099a210 00000286 ffffff24 df77b5c8 00000000 0000000f 00000001 c0103439
       df77b5c8 e099a0ff e09c0000 00000000 0000000f 00000001 80080740 dfd2007b
Call Trace:
 [<c010ef83>] __ipipe_handle_irq+0x1b9/0x20b
 [<e099a210>] e1000_set_multi+0x111/0x189 [e1000]
 [<c0103439>] common_interrupt+0x21/0x38
 [<e099a0ff>] e1000_set_multi+0x0/0x189 [e1000]
 [<e099a210>] e1000_set_multi+0x111/0x189 [e1000]
 [<c02dcf81>] __dev_mc_upload+0x1d/0x1e
 [<c02dd0a1>] dev_mc_upload+0x24/0x37
 [<c02db5ac>] dev_open+0x44/0x62
 [<c02da079>] dev_change_flags+0x47/0xe4
 [<c030d192>] devinet_ioctl+0x252/0x56f
 [<c02db18a>] dev_ifsioc+0x113/0x38d
 [<c02d15a4>] sock_ioctl+0x0/0x1ad
 [<c02d1732>] sock_ioctl+0x18e/0x1ad
 [<c02d15a4>] sock_ioctl+0x0/0x1ad
 [<c015e18f>] do_ioctl+0x1f/0x62
 [<c015e416>] vfs_ioctl+0x244/0x256
 [<c015e45b>] sys_ioctl+0x33/0x4c
 [<c01029f3>] sysenter_past_esp+0x6c/0x70
 =======================
Code:  Bad EIP value.
EIP: [<00000000>] _stext+0x3feffc70/0x14 SS:ESP 0068:c168de20
 <0>Kernel panic - not syncing: Fatal exception in interrupt
 BUG: at arch/i386/kernel/smp.c:565 smp_call_function()
 [<c010b903>] smp_call_function+0x66/0x10a
 [<c0118e12>] printk+0x62/0xd5
 [<c010b9c2>] smp_send_stop+0x1b/0x2b
 [<c01183ad>] panic+0x4d/0xe4
 [<c0103f71>] die+0x1f2/0x226
 [<c011167c>] do_page_fault+0x447/0x517
 [<c010f61d>] __ipipe_handle_exception+0xce/0x158
 [<c010bb1e>] smp_call_function_interrupt+0x31/0x4c
 [<c033336d>] error_code+0x81/0x90
 [<c011434d>] try_to_wake_up+0x33c/0x346
 [<c0112244>] __activate_task+0x1c/0x29
 [<c010ef83>] __ipipe_handle_irq+0x1b9/0x20b
 [<e099a210>] e1000_set_multi+0x111/0x189 [e1000]
 [<c0103439>] common_interrupt+0x21/0x38
 [<e099a0ff>] e1000_set_multi+0x0/0x189 [e1000]
 [<e099a210>] e1000_set_multi+0x111/0x189 [e1000]
 [<c02dcf81>] __dev_mc_upload+0x1d/0x1e
 [<c02dd0a1>] dev_mc_upload+0x24/0x37
 [<c02db5ac>] dev_open+0x44/0x62
 [<c02da079>] dev_change_flags+0x47/0xe4
 [<c030d192>] devinet_ioctl+0x252/0x56f
 [<c02db18a>] dev_ifsioc+0x113/0x38d
 [<c02d15a4>] sock_ioctl+0x0/0x1ad
 [<c02d1732>] sock_ioctl+0x18e/0x1ad
 [<c02d15a4>] sock_ioctl+0x0/0x1ad
 [<c015e18f>] do_ioctl+0x1f/0x62
 [<c015e416>] vfs_ioctl+0x244/0x256
 [<c015e45b>] sys_ioctl+0x33/0x4c
 [<c01029f3>] sysenter_past_esp+0x6c/0x70
 =======================

I have attached my kernel config to this mail.
Any idea on this? Is there anybody out there that has MSI running succesfully with the Xenomai adeos patch?

Thanks for any feedback on that issue!

Regards

Mathias


-- 
Mathias Koehrer
mathias_koehrer@domain.hid


50€ AMAZON-Einkaufsgutschein bei Bestellung von Arcor-DSL:
Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  39,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

[-- Attachment #2: config.gz --]
[-- Type: application/octetstream, Size: 8598 bytes --]

^ permalink raw reply	[flat|nested] 52+ messages in thread
* Re: [Xenomai-help] Xenomai and MSI enabled crashes kernel
@ 2007-05-02 12:57 M. Koehrer
  2007-05-02 13:23 ` Jan Kiszka
  2007-05-02 14:47 ` Philippe Gerum
  0 siblings, 2 replies; 52+ messages in thread
From: M. Koehrer @ 2007-05-02 12:57 UTC (permalink / raw)
  To: jan.kiszka, mathias_koehrer; +Cc: xenomai

Hi Jan,

I have looked closed into that issue.
For that I have added a global array that is written within  
__ipipe_handle_irq() (file arch/i386/kernel/ipipe.c):

                extern int xxx_int[];
                irq = ~irq;

                xxx_int[0] = irq;

#ifdef CONFIG_X86_LOCAL_APIC
                {
                        unsigned vector = irq + FIRST_EXTERNAL_VECTOR;
                        if (vector >= FIRST_SYSTEM_VECTOR)
                                irq = ipipe_apic_vector_irq(vector);
                }

                 xxx_int[1] = irq;


This global variable is printed out within the kernel trap.
It returns the values 0xcf and 0xe0 (207 and 224) for xxx_int[0], xxx_int[1].
And this is the really strange thing as the IRQ for the e1000 should be 219 (on a non-adeos-patched kernel).
I do not know what happens here, but it looks really strange...


Regards

Mathias
---------------------------------------
Intel(R) PRO/1000 Network Driver - version 7.3.15-k2
Copyright (c) 1999-2006 Intel Corporation.
ACPI: PCI Interrupt 0000:05:00.0[A] -> GSI 16 (level, low) -> IRQ 16
e1000: 0000:05:00.0: e1000_probe: (PCI Express:2.5Gb/s:Width x1) 00:30:48:5a:f9:0a
e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
BUG: unable to handle kernel NULL pointer dereference at virtual address 00000000
 printing eip:
00000000
*pde = 00000000
Oops: 0000 [#1]
SMP
Modules linked in: e1000
xxx_int: cf e0 0 0
CPU:    0
EIP:    0060:[<00000000>]    Not tainted VLI
EFLAGS: 00010086   (2.6.20.4-msi2 #11)
EIP is at _stext+0x3feffc70/0x14
eax: c0112410   ebx: 00000007   ecx: c0114519   edx: df048000
esi: 00000006   edi: 00000046   ebp: ffffffff   esp: df049e20
ds: 007b   es: 007b   ss: 0068
Process ifconfig (pid: 1241, ti=df048000 task=c173e030 task.ti=df048000)
Stack: c03e6f00 000000ec 00000000 c03d9180 c010f14d 00007600 00000001 00000060
       e099a1ed 00000286 ffffff24 df75b5c8 00000000 0000000f 00000001 c01035b9
       df75b5c8 e099a0ff e09c0000 00000000 0000000f 00000001 80080740 dfc7007b
Call Trace:
 [<c010f14d>] __ipipe_handle_irq+0x1d3/0x225
 [<e099a1ed>] e1000_set_multi+0xee/0x189 [e1000]
 [<c01035b9>] common_interrupt+0x21/0x38
 [<e099a0ff>] e1000_set_multi+0x0/0x189 [e1000]
 [<e099a1ed>] e1000_set_multi+0xee/0x189 [e1000]
 [<c02dd151>] __dev_mc_upload+0x1d/0x1e
 [<c02dd271>] dev_mc_upload+0x24/0x37
 [<c02db77c>] dev_open+0x44/0x62
 [<c02da249>] dev_change_flags+0x47/0xe4
 [<c030d362>] devinet_ioctl+0x252/0x56f
 [<c02db35a>] dev_ifsioc+0x113/0x38d
 [<c02d1774>] sock_ioctl+0x0/0x1ad
 [<c02d1902>] sock_ioctl+0x18e/0x1ad
 [<c02d1774>] sock_ioctl+0x0/0x1ad
 [<c015e35b>] do_ioctl+0x1f/0x62
 [<c015e5e2>] vfs_ioctl+0x244/0x256
 [<c015e627>] sys_ioctl+0x33/0x4c
 [<c01029f3>] sysenter_past_esp+0x6c/0x70
 =======================
Code:  Bad EIP value.
EIP: [<00000000>] _stext+0x3feffc70/0x14 SS:ESP 0068:df049e20
 <0>Kernel panic - not syncing: Fatal exception in interrupt
 BUG: at arch/i386/kernel/smp.c:565 smp_call_function()
 [<c010bab3>] smp_call_function+0x66/0x10a
 [<c0118fde>] printk+0x62/0xd5
 [<c010bb72>] smp_send_stop+0x1b/0x2b
 [<c0118579>] panic+0x4d/0xe4
 [<c0104121>] die+0x1f2/0x226
 [<c0111848>] do_page_fault+0x447/0x517
 [<c010f7e7>] __ipipe_handle_exception+0xce/0x158
 [<c010bcce>] smp_call_function_interrupt+0x31/0x4c
 [<c033353d>] error_code+0x81/0x90
 [<c0114519>] try_to_wake_up+0x33c/0x346
 [<c0112410>] __activate_task+0x1c/0x29
 [<c010f14d>] __ipipe_handle_irq+0x1d3/0x225
 [<e099a1ed>] e1000_set_multi+0xee/0x189 [e1000]
 [<c01035b9>] common_interrupt+0x21/0x38
 [<e099a0ff>] e1000_set_multi+0x0/0x189 [e1000]
 [<e099a1ed>] e1000_set_multi+0xee/0x189 [e1000]
 [<c02dd151>] __dev_mc_upload+0x1d/0x1e
 [<c02dd271>] dev_mc_upload+0x24/0x37
 [<c02db77c>] dev_open+0x44/0x62
 [<c02da249>] dev_change_flags+0x47/0xe4
 [<c030d362>] devinet_ioctl+0x252/0x56f
 [<c02db35a>] dev_ifsioc+0x113/0x38d
 [<c02d1774>] sock_ioctl+0x0/0x1ad
 [<c02d1902>] sock_ioctl+0x18e/0x1ad
 [<c02d1774>] sock_ioctl+0x0/0x1ad
 [<c015e35b>] do_ioctl+0x1f/0x62
 [<c015e5e2>] vfs_ioctl+0x244/0x256
 [<c015e627>] sys_ioctl+0x33/0x4c
 [<c01029f3>] sysenter_past_esp+0x6c/0x70
 =======================






----- Original Nachricht ----
Von:     Jan Kiszka <jan.kiszka@domain.hid>
An:      "M. Koehrer" <mathias_koehrer@domain.hid>
Datum:   02.05.2007 11:39
Betreff: Re: [Xenomai-help] Xenomai and MSI enabled crashes kernel

> M. Koehrer wrote:
> > Hi Jan,
> > 
> > here is the result of this patch.
> > I patched in addition to Philippe's patches.
> > 
> > Regards
> > 
> > Mathias
> > ---------
> > Intel(R) PRO/1000 Network Driver - version 7.3.15-k2
> > Copyright (c) 1999-2006 Intel Corporation.
> > ACPI: PCI Interrupt 0000:05:00.0[A] -> GSI 16 (level, low) -> IRQ 16
> > e1000: 0000:05:00.0: e1000_probe: (PCI Express:2.5Gb/s:Width x1)
> 00:30:48:5a:f9:0a
> > e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
> > BUG: unable to handle kernel NULL pointer dereference at virtual address
> 00000000
> >  printing eip:
> > dfedd3ca
> > *pde = 00000000
> > Oops: 0000 [#1]
> > SMP
> > Modules linked in: e1000
> > CPU:    0
> > EIP:    0060:[<dfedd3ca>]    Not tainted VLI
> > EFLAGS: 00010082   (2.6.20.4 #7)
> > EIP is at 0xdfedd3ca
> > eax: c0112478   ebx: 00000001   ecx: c0114581   edx: df06c001
> > esi: dfedf6c0   edi: dfedc240   ebp: 00000000   esp: df06ddf8
> > ds: 007b   es: 007b   ss: 0068
> > Process ifconfig (pid: 1242, ti=df06c000 task=c16f9030 task.ti=df06c000)
> > Stack: 00000040 ffffffff 00000000 00000007 c03e6f00 000000ec 00000000
> c03d9180
> >        c010f1b5 c015291a decbe080 00000680 decbe080 00000724 c02d6dfd
> 00007600
> >        00000001 00000060 e099a210 00000286 ffffff24 df7065c8 00000000
> 0000000f
> > Call Trace:
> >  [<c010f1b5>] __ipipe_handle_irq+0x26b/0x2bd
> 
> I must have patched crap into this, but I don't see it. Could someone
> have a look why there is no printk output related to #IRQ 219
> (~0xffffff24, see stack content)?
> 
> BTW, Mathias, forcing your kernel into single CPU mode via kernel
> command line "cpus=1" doesn't expose the issue, right? Could we face
> some kind of setup race across the CPUs here?
> 
> Thanks,
> Jan
> 
> 

-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  39,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2


^ permalink raw reply	[flat|nested] 52+ messages in thread
* Re: [Xenomai-help] Xenomai and MSI enabled crashes kernel
@ 2007-05-04  7:45 M. Koehrer
  2007-05-04  7:59 ` Jan Kiszka
  0 siblings, 1 reply; 52+ messages in thread
From: M. Koehrer @ 2007-05-04  7:45 UTC (permalink / raw)
  To: rpm, mathias_koehrer; +Cc: xenomai, jan.kiszka

Hi everybody,

I have been gone one step back to analyze the issue from the situation is originally occurred.
This was the situation that I had a 2.6.20.4 kernel and Xenomai 2.3.1 (included Adeos patch).
When I  enable MSI in the kernel configuration I get the kernel message:
"spurious APIC interrupt on CPU#0, should never happen."
when trying to enable (ifconfig) the e1000 driver for an onboard PCIe network adapter.

I have now used the very same kernel without Xenomai patch, but using the same kernel config
(MSI enabled). This returns the IRQ 223 for the Ethernet adapter.
With 2.6.21 it returns IRQ 219, there seems to be a difference between 2.6.20 and 2.6.21.
Well, as I am working with 2.6.20.4, I looked deeper in that.

One interesting thing I found was in arch/i386/kernel/ipipe.c at function __ipipe_enable_pipeline
I do not understand the meaning of that code.
At the beginning of that function, there are a couple of calls to ipipe_virtualize_irq().
These MAP the APIC system vectors.
However, the second call that maps 
SPURIOUS_APIC_VECTOR - FIRST_EXTERNAL_VECTOR (with is infact 223) is mapped to
smp_spurious_interrupt (which leads to the result I see).
However, I am not sure if it is correct to use the VECTOR values here as ipipe_virtualize_irq
uses "irq" as second parameter and not  a vector. This looks like a vector vs. irq mismatch.

Within the create_irq() function, the e1000 creates irq 223 on vector 201.

When I monitor the calls to set_intr_gate() I see that the vector 32 will end up at the address
of irq_entries_start, vector 32 will end up at irq_entries_start + 8 etc.
In entry.S this will end up in writing ~(vector) unto the stack and calling common_interrupt.
This finally calls __ipipe_handle_irq.
The means that __ipipe_handle_irq will not be called with the IRQ number but with the vector!!
It could be that this is the same for the "old" IRQ vectors (<32). 
However for the MSI IRQs this seems to be not correct.
I think that this could be the root cause for the problems I see.

Any feedback on this comment is highly welcome!

Regards

Mathias



> On Thu, 2007-05-03 at 09:06 +0200, M. Koehrer wrote:
> > Hi Philippe,
> > 
> > I have restored the original kernel configuration and I got the crash
> again.
> > As it happens during the boot, it is reproducible.
> > Enclosed is the console log (including the oops), the relevant disassemble
> part,
> > the output of "nm vmlinux" and my kernel config.
> > If you like I can mail you privately the vmlinux aswell.
> > What I see from the disassembler listing
> > c010f10b:       e8 5e ae 02 00          call   c0139f6e
> <__ipipe_walk_pipeline>
> > c010f110:       ff 15 68 70 3d c0       call   *0xc03d7068
> > is that ipipe_walk_pipeline is called followed by a indirect call to
> 0xc03d7068.
> > I looked at `nm vmlinux` and found out that 0xc03d7068 is
> __ipipe_logical_cpuid.
> > 
> > I hope that helps to get closer to this bug...
> > 
> 
> Should be ok, thanks. I will have a close look at this.
> 
> -- 
> Philippe.
> 
> 
> 

-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  39,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2


^ permalink raw reply	[flat|nested] 52+ messages in thread
* Re: [Xenomai-help] Xenomai and MSI enabled crashes kernel
@ 2007-05-07  7:11 M. Koehrer
  0 siblings, 0 replies; 52+ messages in thread
From: M. Koehrer @ 2007-05-07  7:11 UTC (permalink / raw)
  To: rpm, mathias_koehrer; +Cc: xenomai, jan.kiszka

Hi Philippe,

thanks for the new adeos patch.
I have now done the following steps:
I used a vanilla 2.6.20.4 kernel and a non-patched Xenomai 2.3.1.
Then I applied the script scripts/prepare-kernel.sh with the 
adeos-ipipe-2.6.20-i386-1.8-00.patch to patch the kernel.

I am now able to boot my PC with this kernel and my e1000 actually 
works on IRQ223! 
That's perfect.
This is one big step into the right direction!

Now, I want to test the real time stuff.
I can modprobe xeno_native, this seems to work fine.
However, when I want to rmmod xeno_native, rmmod does not return.
dmesg shows the error
APIC error on CPU0 : 00(20)
APIC error on CPU1 : 00(40)

This error comes with out without the e1000 (MSI based) loaded at this time.

I am able to load and unload xeno_nucleus, thus this seems to be a
xeno_native issue.

Regards

Mathias
> 
> Does this patch improve things?
> 
> http://download.gna.org/adeos/patches/v2.6/i386/adeos-ipipe-2.6.20-i386-1.8-
> 00.patch
> 
> -- 
> Philippe.
> 


-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  39,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2


^ permalink raw reply	[flat|nested] 52+ messages in thread
end of thread, other threads:[~2007-05-09  8:00 UTC | newest]

Thread overview: 52+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2007-04-26 12:04 [Xenomai-help] Xenomai and MSI enabled crashes kernel M. Koehrer
2007-04-27 11:48 ` Jan Kiszka
2007-04-27 13:14   ` Philippe Gerum
2007-04-27 13:22     ` Jan Kiszka
2007-04-27 13:31       ` M. Koehrer
2007-04-27 13:47         ` Jan Kiszka
2007-04-27 14:08           ` M. Koehrer
2007-04-27 14:19             ` Philippe Gerum
2007-04-27 14:28               ` M. Koehrer
2007-04-27 14:40                 ` Philippe Gerum
2007-04-27 14:56                 ` Philippe Gerum
2007-04-27 15:05                   ` Philippe Gerum
2007-04-27 15:10                     ` M. Koehrer
2007-04-27 15:36                       ` Philippe Gerum
2007-04-27 15:41                         ` M. Koehrer
2007-04-30  9:05                           ` Jan Kiszka
2007-04-30 10:11                             ` M. Koehrer
2007-04-30 11:27                               ` Jan Kiszka
2007-04-30 12:51                                 ` M. Koehrer
2007-04-30 15:10                                   ` Jan Kiszka
2007-04-27 20:39                       ` Philippe Gerum
2007-04-30 15:39                         ` Philippe Gerum
2007-05-02  7:05                           ` M. Koehrer
2007-05-02  8:39                             ` Jan Kiszka
2007-05-02  9:14                               ` M. Koehrer
2007-05-02  9:39                                 ` Jan Kiszka
2007-05-02 12:42                                 ` Philippe Gerum
2007-05-02 13:44                                   ` M. Koehrer
2007-05-02 13:49                                     ` Jan Kiszka
2007-04-27 14:31             ` Jan Kiszka
2007-04-27 14:52               ` M. Koehrer
2007-04-28 12:54 ` Bernhard Walle
  -- strict thread matches above, loose matches on Subject: below --
2007-05-02 12:57 M. Koehrer
2007-05-02 13:23 ` Jan Kiszka
2007-05-02 14:47 ` Philippe Gerum
2007-05-03  7:06   ` M. Koehrer
2007-05-03  8:29     ` Philippe Gerum
2007-05-04  7:45 M. Koehrer
2007-05-04  7:59 ` Jan Kiszka
2007-05-04  8:20   ` M. Koehrer
2007-05-04 12:20     ` Philippe Gerum
2007-05-04 12:46       ` M. Koehrer
2007-05-04 13:03         ` Philippe Gerum
2007-05-05 17:21       ` Philippe Gerum
2007-05-08 11:53         ` M. Koehrer
2007-05-08 12:28           ` Philippe Gerum
2007-05-08 12:38             ` M. Koehrer
2007-05-08 13:28               ` Philippe Gerum
2007-05-08 13:37               ` Philippe Gerum
2007-05-08 14:35                 ` M. Koehrer
2007-05-09  8:00                   ` Philippe Gerum
2007-05-07  7:11 M. Koehrer

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.