kernel panic with ipv6 in Xen

kernel panic with ipv6 in Xen

[Sam Johnston]

Evening all,

We have a kernel panic on boot under xen when loading the ipv6 module
which was resolved by mounting the device under dom0 and moving the
ipv6 module out of the way. Is this a known problem? Is it likely to
be a xen issue? An ipv6 issue? PEBCAK?

It's not an essential service so we can live without it, but I haven't
seen a kernel panic yet this millenium so I considered it
email-worthy.

Sam

Configuring network interfaces: done.

Setting the System Clock using the Hardware Clock as reference...
hwclock is unable to get I/O port access:  the iopl(3) call failed.
System Clock set. Local time: Fri May  6 23:27:36 UTC 2005

Running ntpdate to synchronize clockNET: Registered protocol family 10
IPv6 over IPv4 tunneling driver
skput:under: c02469ec:14 put:14 dev:<NULL>------------[ cut here ]------------
kernel BUG at net/core/skbuff.c:107!
invalid operand: 0000 [#1]
PREEMPT
Modules linked in: ipv6
CPU:    0
EIP:    0061:[<c023485b>]    Not tainted VLI
EFLAGS: 00010296   (2.6.10-xenU)
EIP is at skb_under_panic+0x3b/0x50
eax: 0000002d   ebx: 00000000   ecx: 00000000   edx: fbffc000
esi: c008fdc0   edi: 000086dd   ebp: c10c4000   esp: c0317dd8
ds: 007b   es: 007b   ss: 0069
Process swapper (pid: 0, threadinfo=c0316000 task=c02da600)
Stack: c02c9000 c02469ec 0000000e 0000000e c02c42bc c02469f5 c008fdc0 0000000e
       c02469ec c0317e3c c008fdc0 000005dc c10c4000 c48ad13b c008fdc0 c10c4000
       000086dd c0317e3c 00000000 000005dc c0317e88 c3eeb7c0 c10801b8 0205003a
Call Trace:
 [<c02469ec>] eth_header+0x12c/0x140
 [<c02469f5>] eth_header+0x135/0x140
 [<c02469ec>] eth_header+0x12c/0x140
 [<c48ad13b>] mld_newpack+0xeb/0x240 [ipv6]
 [<c0138934>] mempool_free+0x74/0x100
 [<c48ad5c3>] add_grhead+0xd3/0xf0 [ipv6]
 [<c48ad6b1>] add_grec+0xd1/0x440 [ipv6]
 [<c0122204>] update_process_times+0x44/0x50
 [<c48adcd3>] mld_send_cr+0x143/0x2c0 [ipv6]
 [<c48aed70>] mld_ifc_timer_expire+0x0/0x50 [ipv6]
 [<c48aed83>] mld_ifc_timer_expire+0x13/0x50 [ipv6]
 [<c0122318>] run_timer_softirq+0xe8/0x200
 [<c011de75>] __do_softirq+0x95/0xb0
 [<c011dedd>] do_softirq+0x4d/0x50
 [<c0133765>] irq_exit+0x35/0x40
 [<c010dace>] do_IRQ+0x1e/0x30
 [<c0105ee1>] evtchn_do_upcall+0x81/0xa0
 [<c0109807>] hypervisor_callback+0x37/0x40
 [<c01072fe>] xen_idle+0x8e/0x150
 [<c02a3c59>] preempt_schedule+0x29/0x50
 [<c01073e9>] cpu_idle+0x29/0x50
 [<c03187c8>] start_kernel+0x178/0x1c0
 [<c0318350>] unknown_bootoption+0x0/0x1e0
Code: c0 0f 44 c2 89 44 24 10 8b 44 24 1c 89 44 24 0c 8b 41 60 c7 04
24 00 90 2c c0 89 44 24 08 8b 44 24 20 89 44 24 04 e8 d5 4c ee ff <0f>
0b 6b 00 be 68 2c c0 83 c4 14 c3 89 f6 8d bc 27 00 00 00 00
 <0>Kernel panic - not syncing: Fatal exception in interrupt
 <0>Rebooting in 1 seconds..

Re: kernel panic with ipv6 in Xen

[Sam Johnston]

I should add that I also moved /lib/tls out of the way at the same time.

On 5/7/05, Sam Johnston <samjie@gmail.com> wrote:
> Evening all,
> 
> We have a kernel panic on boot under xen when loading the ipv6 module
> which was resolved by mounting the device under dom0 and moving the
> ipv6 module out of the way. Is this a known problem? Is it likely to
> be a xen issue? An ipv6 issue? PEBCAK?
> 
> It's not an essential service so we can live without it, but I haven't
> seen a kernel panic yet this millenium so I considered it
> email-worthy.
> 
> Sam
> 
> Configuring network interfaces: done.
> 
> Setting the System Clock using the Hardware Clock as reference...
> hwclock is unable to get I/O port access:  the iopl(3) call failed.
> System Clock set. Local time: Fri May  6 23:27:36 UTC 2005
> 
> Running ntpdate to synchronize clockNET: Registered protocol family 10
> IPv6 over IPv4 tunneling driver
> skput:under: c02469ec:14 put:14 dev:<NULL>------------[ cut here ]------------
> kernel BUG at net/core/skbuff.c:107!
> invalid operand: 0000 [#1]
> PREEMPT
> Modules linked in: ipv6
> CPU:    0
> EIP:    0061:[<c023485b>]    Not tainted VLI
> EFLAGS: 00010296   (2.6.10-xenU)
> EIP is at skb_under_panic+0x3b/0x50
> eax: 0000002d   ebx: 00000000   ecx: 00000000   edx: fbffc000
> esi: c008fdc0   edi: 000086dd   ebp: c10c4000   esp: c0317dd8
> ds: 007b   es: 007b   ss: 0069
> Process swapper (pid: 0, threadinfo=c0316000 task=c02da600)
> Stack: c02c9000 c02469ec 0000000e 0000000e c02c42bc c02469f5 c008fdc0 0000000e
>        c02469ec c0317e3c c008fdc0 000005dc c10c4000 c48ad13b c008fdc0 c10c4000
>        000086dd c0317e3c 00000000 000005dc c0317e88 c3eeb7c0 c10801b8 0205003a
> Call Trace:
>  [<c02469ec>] eth_header+0x12c/0x140
>  [<c02469f5>] eth_header+0x135/0x140
>  [<c02469ec>] eth_header+0x12c/0x140
>  [<c48ad13b>] mld_newpack+0xeb/0x240 [ipv6]
>  [<c0138934>] mempool_free+0x74/0x100
>  [<c48ad5c3>] add_grhead+0xd3/0xf0 [ipv6]
>  [<c48ad6b1>] add_grec+0xd1/0x440 [ipv6]
>  [<c0122204>] update_process_times+0x44/0x50
>  [<c48adcd3>] mld_send_cr+0x143/0x2c0 [ipv6]
>  [<c48aed70>] mld_ifc_timer_expire+0x0/0x50 [ipv6]
>  [<c48aed83>] mld_ifc_timer_expire+0x13/0x50 [ipv6]
>  [<c0122318>] run_timer_softirq+0xe8/0x200
>  [<c011de75>] __do_softirq+0x95/0xb0
>  [<c011dedd>] do_softirq+0x4d/0x50
>  [<c0133765>] irq_exit+0x35/0x40
>  [<c010dace>] do_IRQ+0x1e/0x30
>  [<c0105ee1>] evtchn_do_upcall+0x81/0xa0
>  [<c0109807>] hypervisor_callback+0x37/0x40
>  [<c01072fe>] xen_idle+0x8e/0x150
>  [<c02a3c59>] preempt_schedule+0x29/0x50
>  [<c01073e9>] cpu_idle+0x29/0x50
>  [<c03187c8>] start_kernel+0x178/0x1c0
>  [<c0318350>] unknown_bootoption+0x0/0x1e0
> Code: c0 0f 44 c2 89 44 24 10 8b 44 24 1c 89 44 24 0c 8b 41 60 c7 04
> 24 00 90 2c c0 89 44 24 08 8b 44 24 20 89 44 24 04 e8 d5 4c ee ff <0f>
> 0b 6b 00 be 68 2c c0 83 c4 14 c3 89 f6 8d bc 27 00 00 00 00
>  <0>Kernel panic - not syncing: Fatal exception in interrupt
>  <0>Rebooting in 1 seconds..
>

Re: Re: kernel panic with ipv6 in Xen

[Chris Andrews]

On 7 May 2005, at 01:13, Sam Johnston wrote:

> I should add that I also moved /lib/tls out of the way at the same 
> time.
>
> On 5/7/05, Sam Johnston <samjie@gmail.com> wrote:
>> Evening all,
>>
>> We have a kernel panic on boot under xen when loading the ipv6 module
>> which was resolved by mounting the device under dom0 and moving the
>> ipv6 module out of the way. Is this a known problem? Is it likely to
>> be a xen issue? An ipv6 issue? PEBCAK?

I've seen this after upgrading Xen and the dom0 kernel, then starting a 
domain with the old xenU kernel and modules. Once I ran the new xenU 
kernel and installed its modules in the domU fs I had no problems. This 
was probably around 2.0.3 or so.


Chris.

Re: Re: kernel panic with ipv6 in Xen

[Sam Johnston]

> >> We have a kernel panic on boot under xen when loading the ipv6 module
> >> which was resolved by mounting the device under dom0 and moving the
> >> ipv6 module out of the way. Is this a known problem? Is it likely to
> >> be a xen issue? An ipv6 issue? PEBCAK?
> 
> I've seen this after upgrading Xen and the dom0 kernel, then starting a
> domain with the old xenU kernel and modules. Once I ran the new xenU
> kernel and installed its modules in the domU fs I had no problems. This
> was probably around 2.0.3 or so.

Thanks for the feedback.

It would seem then that the best solution for us is to use the debian
kernel packaging tools to create a common kernel which would then be
installed into dom0 and domU along with the matching modules.
Alternatively we could make the dom0 modules available in the guest
domains by mounting /lib/modules read only (as block devices or via
NFS).

Sam

RE: kernel panic with ipv6 in Xen

[Ian Pratt]

> We have a kernel panic on boot under xen when loading the 
> ipv6 module which was resolved by mounting the device under 
> dom0 and moving the
> ipv6 module out of the way. Is this a known problem? Is it 
> likely to be a xen issue? An ipv6 issue? PEBCAK?

Interesting. Does anyone use ipv6 with Xen? I can't say I've ever tried.

Is this bug easy to reproduce?

Thanks,
Ian

Re: kernel panic with ipv6 in Xen

[Chris Andrews]

Ian Pratt wrote on 09/05/05 05:44:
>>We have a kernel panic on boot under xen when loading the 
>>ipv6 module which was resolved by mounting the device under 
>>dom0 and moving the
>>ipv6 module out of the way. Is this a known problem? Is it 
>>likely to be a xen issue? An ipv6 issue? PEBCAK?
> 
> 
> Interesting. Does anyone use ipv6 with Xen? I can't say I've ever tried.

Yep, works fine. I was able to run a radvd in dom0, bound to xen-br0, which 
gave the domUs global addresses, and dom0 acted as a v6 router fine.

(the box's main LAN interface, eth0, wasn't bound to the bridge, so I didn't 
end up being a radvd for the entire LAN)

> Is this bug easy to reproduce?

Take a box running 2.0.5/2.6.10-xen0 that still has an old 2.6.9-xenU kernel 
lying around, and a domU that has the old 2.6.9-xenU modules available and 
boot the domU on 2.6.9.

i.e., take a 2.0.3 box, upgrade to 2.0.5, and forget to update your domUs' 
module trees and xm configs :)

I seem to remember this happening on previous upgrades, too. Unfortunately I 
don't have the box available right now to test things.


Chris.