Re: IPSecv6/Neighbor discovery crash

Re: IPSecv6/Neighbor discovery crash

[latten]

I tried my old standby of putting a few printk's to help debug.
I put them in ndisc_output() and ndisc_build_ll_hdr() and they get 
printed out ok except when the crash occurs. I get 
absolutely nothing. So I do not know where or what are some of
the values ndisc_output() or ndisc_build_ll_addr() are using.
Nothing gets written to my log file when I do the ping6.

I too had been thinking similar to Miyazawa-san...

Joy


On Thu, 21 Aug 2003 18:46:40 -0700
"David S. Miller" <davem@redhat.com> wrote:

> On Thu, 21 Aug 2003 20:49:47 -0500
> latten@austin.ibm.com wrote:
> 
> > EIP is at ndisc_build_ll_hdr+0x17/0x1e0
> 
> So what exactly is NULL in ndisc_build_ll_hdr(), is
> it 'dev'?  That'd be really weird...
> 

I had same crach.

I guess it is due to xfrm cache. My impression about the problem is likes this.

When we configure IPsec and src and dst of neighbour discoery match the configuration
occasionally, The kernel creates and caches the stackable dst like this because
ndisc_send_* want to use it

dst->output(ah6_output)
 +- child->output(ndisc_output)

Then it receives icmpv6 echo request. It replys  by using
the cached stackable dst like above. The kernel however must use another stackable dst like

dst->output(ah6_output)
 +- child->output(ip6_output)

It is the issue. The kernel can not tell first stackable dst from second stackable dst
because it can not know the last output function.

I believe we need to change the kernel to use ip6_output ( or another common output function)
to send neighbour discovery packet instead of ndisc_output essentially.

But it may make the kernel be unstable. I think there is not so much request to use IPsec
with neighbour discovery.
I think it is better to remove xfrm_lookup from ndisc_send_* functions at the moment.

Best regards,

--Kazunori Miyazawa

Re: IPSecv6/Neighbor discovery crash

[David S. Miller]

On Fri, 22 Aug 2003 18:12:10 -0500
latten@austin.ibm.com wrote:

> So I do not know where or what are some of
> the values ndisc_output() or ndisc_build_ll_addr() are using.
> Nothing gets written to my log file when I do the ping6.
 ...
> > > EIP is at ndisc_build_ll_hdr+0x17/0x1e0

Why not save yourself all that debugging effort and just
use gdb on the kernel image and:

(gdb) x/10i ndisc_build_ll_hdr+0x17

then match that assembler up with the output of:

bash$ make net/ipv6/ndisc.s

I'm flabbergasted that someone would resort to printk()'s
to figure out the precise location of this OOPS when the
OOPS says exactly the location :(

Re: IPSecv6/Neighbor discovery crash

[latten]

Sighhhhh and red-face.... I won't make that mistake again.
I'm learning.

OK, yes, it seems to be complaining about dev->hard_header...

Joy


On Fri, 22 Aug 2003 18:12:10 -0500
latten@austin.ibm.com wrote:

> So I do not know where or what are some of
> the values ndisc_output() or ndisc_build_ll_addr() are using.
> Nothing gets written to my log file when I do the ping6.
 ...
> > > EIP is at ndisc_build_ll_hdr+0x17/0x1e0

Why not save yourself all that debugging effort and just
use gdb on the kernel image and:

(gdb) x/10i ndisc_build_ll_hdr+0x17

then match that assembler up with the output of:

bash$ make net/ipv6/ndisc.s

I'm flabbergasted that someone would resort to printk()'s
to figure out the precise location of this OOPS when the
OOPS says exactly the location :(

IPSecv6/Neighbor discovery crash

[latten]

I am using linux-2.6.0-test3 + patch-2.6.0-test3-bk7 on SMP machines.
Upon configuring AH, I do a ping6 to test connectivity and 
I get the following trace. (see first crash)
I have seen this crash with ESP configured also. 

This only seems to happen when IPSecv6 is configured. 
So far, I have not been able to track down the culprit... I have
not been able to determine if it is a lock... icmpv6_echo_reply()
takes a lock on the socket and will let it go when done...
Has anyone else seen this? 

Joy


Unable to handle kernel NULL pointer dereference at virtual address 00000164
 printing eip:
c0445767
*pde = 00000000
Oops: 0000 [#1]
CPU:    0
EIP:    0060:[<c0445767>]    Not tainted
EFLAGS: 00010246
EIP is at ndisc_build_ll_hdr+0x17/0x1e0
eax: 00000000   ebx: f7564bc0   ecx: f550ce60   edx: f6d388c0
esi: c1af9430   edi: 00000000   ebp: c05c1c90   esp: c05c1c44
ds: 007b   es: 007b   ss: 0068
Process swapper (pid: 0, threadinfo=c05c0000 task=c050f020)
Stack: c1ab41a0 f29ad1c4 f3128624 f3128640 00000206 0000000c 3a000246 f70803e0
       f7080408 c1aff200 c05c1cb0 c0460b4d f70803e0 f7564bc0 c1af944c c1aff214
       f7564bc0 f7564bc0 c1af94cc c05c1cb0 c0445970 f7564bc0 00000000 c1af9430
Call Trace:
 [<c0460b4d>] ah6_output+0x26d/0x510
 [<c0445970>] ndisc_output+0x40/0x80
 [<c043822f>] ip6_push_pending_frames+0x22f/0x380
 [<c044e496>] icmpv6_push_pending_frames+0x116/0x1a0
 [<c044edfa>] icmpv6_echo_reply+0x28a/0x340
 [<c044f324>] icmpv6_rcv+0x264/0x590
 [<c0438820>] ip6_input+0x120/0x2e0
 [<c04385ee>] ipv6_rcv+0x13e/0x250
 [<c03cf62b>] netif_receive_skb+0x16b/0x200
 [<c03cf744>] process_backlog+0x84/0x120
 [<c03cf863>] net_rx_action+0x83/0x110
 [<c0129de7>] do_softirq+0xe7/0xf0
 [<c010e10d>] do_IRQ+0x15d/0x200
 [<c0119c3d>] smp_apic_timer_interrupt+0xcd/0x140
 [<c0109060>] default_idle+0x0/0x40
 [<c010c170>] common_interrupt+0x18/0x20
 [<c0109060>] default_idle+0x0/0x40
 [<c0109090>] default_idle+0x30/0x40
 [<c0109126>] cpu_idle+0x46/0x50
 [<c0105000>] rest_init+0x0/0x80
 [<c05c298e>] start_kernel+0x19e/0x1f0
 [<c05c2500>] unknown_bootoption+0x0/0x110

Code: 8b 80 64 01 00 00 85 c0 75 14 ba 01 00 00 00 8b 5d f4 89 d0
 <0>Kernel panic: Fatal exception in interrupt
In interrupt handler - not syncing

Re: IPSecv6/Neighbor discovery crash

[David S. Miller]

On Thu, 21 Aug 2003 20:49:47 -0500
latten@austin.ibm.com wrote:

> EIP is at ndisc_build_ll_hdr+0x17/0x1e0

So what exactly is NULL in ndisc_build_ll_hdr(), is
it 'dev'?  That'd be really weird...

Re: IPSecv6/Neighbor discovery crash

[Kazunori Miyazawa]

On Thu, 21 Aug 2003 18:46:40 -0700
"David S. Miller" <davem@redhat.com> wrote:

> On Thu, 21 Aug 2003 20:49:47 -0500
> latten@austin.ibm.com wrote:
> 
> > EIP is at ndisc_build_ll_hdr+0x17/0x1e0
> 
> So what exactly is NULL in ndisc_build_ll_hdr(), is
> it 'dev'?  That'd be really weird...
> 

I had same crach.

I guess it is due to xfrm cache. My impression about the problem is likes this.

When we configure IPsec and src and dst of neighbour discoery match the configuration
occasionally, The kernel creates and caches the stackable dst like this because
ndisc_send_* want to use it

dst->output(ah6_output)
 +- child->output(ndisc_output)

Then it receives icmpv6 echo request. It replys  by using
the cached stackable dst like above. The kernel however must use another stackable dst like

dst->output(ah6_output)
 +- child->output(ip6_output)

It is the issue. The kernel can not tell first stackable dst from second stackable dst
because it can not know the last output function.

I believe we need to change the kernel to use ip6_output ( or another common output function)
to send neighbour discovery packet instead of ndisc_output essentially.

But it may make the kernel be unstable. I think there is not so much request to use IPsec
with neighbour discovery.
I think it is better to remove xfrm_lookup from ndisc_send_* functions at the moment.

Best regards,

--Kazunori Miyazawa

Re: IPSecv6/Neighbor discovery crash

[David S. Miller]

On Fri, 22 Aug 2003 14:55:53 +0900
Kazunori Miyazawa <kazunori@miyazawa.org> wrote:

> I guess it is due to xfrm cache.

I don't think so, fl.fl_icmp_type will be different for ICMP echo flow
lookup than for the ndisc ones.  Therefore flow cache will not return
the older ndisc stacked DST for the ICMP echo flow lookup.