Virtual device and ARP table

Virtual device and ARP table

[Christophe Jelger]

Hello,

I am currently "resurrecting" a Linux module (called LUNAR) which I 
co-developed in 2007 and I'm having a weird kernel crash. This code 
basically used to work fine up to 2.6.18 which was the latest version 
before we stopped our development. I quickly ported it to 2.6.{31,32}: 
it compiles fine and loads fine, but it crashes/hangs the kernel when 
it's really being used.

The module is a virtual device used for MANET routing: with the current 
version, it basically "captures" DNS requests sent to the virtual 
interface --> this triggers the sending of a fake DNS reply (see below) 
and the creation of an ARP table entry for the destination (the MANET 
route is built at the same time). Packets can then be sent to the 
destination.

The problem I'm having is that the kernel quickly hangs after I create a 
new ARP entry (actually only if it's being used). If the entry I create 
is set to NUD_PERMANENT, then everything works fine! I use 
__neigh_lookup_errno to lookup/create the entry and neigh_lookup to 
set/update the MAC address. Note that the ARP entry is created without 
problem, but typically even just doing a userspace "arp -a" command can 
crash the kernel (it also hangs the userspace command!). Doing "arp -na" 
usually does NOT crash the kernel!

I guess the problem comes from a combination of ARP + DNS 
lookups/replies. Note that my kernel module has its own internal fake 
DNS server which captures lookups and sends replies directly back to the 
stack. What is amazing: if the ARP entry I create is set to 
NUD_PERMANENT, then I don't get any crash (however I cannot develop my 
module with permanent ARP entries).

I'm wondering if there were any major changes to the neighbor and arp 
code (between 2.6.18 and 2.6.31) that are somehow causing this problem ?...

Any hint is very welcome.

thanks in advance,
Christophe

PS: I can easily reproduce the problem, and was trying to debug with 
qemu and gdb server but so fra no success to clearly identify the 
problem. Last point: it seems the kernel does not really "crash" but 
rather ends up in some unstable state and maybe in a loop.

Re: Virtual device and ARP table

[Eric Dumazet]

Le lundi 07 juin 2010 à 12:21 +0200, Christophe Jelger a écrit :
> Hello,
> 
> I am currently "resurrecting" a Linux module (called LUNAR) which I 
> co-developed in 2007 and I'm having a weird kernel crash. This code 
> basically used to work fine up to 2.6.18 which was the latest version 
> before we stopped our development. I quickly ported it to 2.6.{31,32}: 
> it compiles fine and loads fine, but it crashes/hangs the kernel when 
> it's really being used.
> 
> The module is a virtual device used for MANET routing: with the current 
> version, it basically "captures" DNS requests sent to the virtual 
> interface --> this triggers the sending of a fake DNS reply (see below) 
> and the creation of an ARP table entry for the destination (the MANET 
> route is built at the same time). Packets can then be sent to the 
> destination.
> 
> The problem I'm having is that the kernel quickly hangs after I create a 
> new ARP entry (actually only if it's being used). If the entry I create 
> is set to NUD_PERMANENT, then everything works fine! I use 
> __neigh_lookup_errno to lookup/create the entry and neigh_lookup to 
> set/update the MAC address. Note that the ARP entry is created without 
> problem, but typically even just doing a userspace "arp -a" command can 
> crash the kernel (it also hangs the userspace command!). Doing "arp -na" 
> usually does NOT crash the kernel!
> 
> I guess the problem comes from a combination of ARP + DNS 
> lookups/replies. Note that my kernel module has its own internal fake 
> DNS server which captures lookups and sends replies directly back to the 
> stack. What is amazing: if the ARP entry I create is set to 
> NUD_PERMANENT, then I don't get any crash (however I cannot develop my 
> module with permanent ARP entries).
> 
> I'm wondering if there were any major changes to the neighbor and arp 
> code (between 2.6.18 and 2.6.31) that are somehow causing this problem ?...
> 
> Any hint is very welcome.
> 
> thanks in advance,
> Christophe
> 
> PS: I can easily reproduce the problem, and was trying to debug with 
> qemu and gdb server but so fra no success to clearly identify the 
> problem. Last point: it seems the kernel does not really "crash" but 
> rather ends up in some unstable state and maybe in a loop.
> --

Hi Christophe

You should ask these kind of questions on netdev instead of lkml.

And of course, post your patch, or send us a crystal ball ;)

Yes, many things changed between 2.6.18 and 2.6.34

Re: Virtual device and ARP table

[Christophe Jelger]

Eric Dumazet wrote:
> Le lundi 07 juin 2010 à 12:21 +0200, Christophe Jelger a écrit :
>> Hello,
>>
>> I am currently "resurrecting" a Linux module (called LUNAR) which I 
>> co-developed in 2007 and I'm having a weird kernel crash. This code 
>> basically used to work fine up to 2.6.18 which was the latest version 
>> before we stopped our development. I quickly ported it to 2.6.{31,32}: 
>> it compiles fine and loads fine, but it crashes/hangs the kernel when 
>> it's really being used.
>>
>> The module is a virtual device used for MANET routing: with the current 
>> version, it basically "captures" DNS requests sent to the virtual 
>> interface --> this triggers the sending of a fake DNS reply (see below) 
>> and the creation of an ARP table entry for the destination (the MANET 
>> route is built at the same time). Packets can then be sent to the 
>> destination.
>>
>> The problem I'm having is that the kernel quickly hangs after I create a 
>> new ARP entry (actually only if it's being used). If the entry I create 
>> is set to NUD_PERMANENT, then everything works fine! I use 
>> __neigh_lookup_errno to lookup/create the entry and neigh_lookup to 
>> set/update the MAC address. Note that the ARP entry is created without 
>> problem, but typically even just doing a userspace "arp -a" command can 
>> crash the kernel (it also hangs the userspace command!). Doing "arp -na" 
>> usually does NOT crash the kernel!
>>
>> I guess the problem comes from a combination of ARP + DNS 
>> lookups/replies. Note that my kernel module has its own internal fake 
>> DNS server which captures lookups and sends replies directly back to the 
>> stack. What is amazing: if the ARP entry I create is set to 
>> NUD_PERMANENT, then I don't get any crash (however I cannot develop my 
>> module with permanent ARP entries).
>>
>> I'm wondering if there were any major changes to the neighbor and arp 
>> code (between 2.6.18 and 2.6.31) that are somehow causing this problem ?...
>>
>> Any hint is very welcome.
>>
>> thanks in advance,
>> Christophe
>>
>> PS: I can easily reproduce the problem, and was trying to debug with 
>> qemu and gdb server but so fra no success to clearly identify the 
>> problem. Last point: it seems the kernel does not really "crash" but 
>> rather ends up in some unstable state and maybe in a loop.
>> --
> 
> Hi Christophe
> 
> You should ask these kind of questions on netdev instead of lkml.
> 
> And of course, post your patch, or send us a crystal ball ;)
> 
> Yes, many things changed between 2.6.18 and 2.6.34
> 

Eric: thanks for the forward to the netdev list. Regarding the code, I 
of course welcome any help but didn't want to pollute the list with 
unsollicited code: I can of course of course send it directly to anyone 
who is willing to help (I can easily reproduce the problem on different 
machines).

Christophe

Re: Virtual device and ARP table

[Eric Dumazet]

Le lundi 07 juin 2010 à 15:03 +0200, Christophe Jelger a écrit :

> 
> Eric: thanks for the forward to the netdev list. Regarding the code, I 
> of course welcome any help but didn't want to pollute the list with 
> unsollicited code: I can of course of course send it directly to anyone 
> who is willing to help (I can easily reproduce the problem on different 
> machines).
> 

Christophe,

Unless patch is really huge, its ok to send it on netdev, with a RFC
label, so that only people with free time take a look, eventually.

[RFC] lunar: ....

[RFC] lunar manet routing module

[Christophe Jelger]

[-- Attachment #1: Type: text/plain, Size: 809 bytes --]

Eric Dumazet wrote:
> 
> Christophe,
> 
> Unless patch is really huge, its ok to send it on netdev, with a RFC
> label, so that only people with free time take a look, eventually.
> 
> [RFC] lunar: ....

[not sure what 'huge' means, I'm sending 60k -- sorry for the pollution]

Eric: thanks for the advice. Instead of a patch I attach a .tgz (hope 
it's ok) of the module code with a README explaining everything: it 
compiles for 2.6.31 and 2.6.32, didn't try more recent kernels because I 
actually want to deploy the whole thing on OpenWRT 2.6.32 on Linksys 
devices.

To all: the lunar module crashes the kernel, so be careful. I tried 
debugging with qemu and gdb server but could not find the bug(s) -- my 
experience for kernel debugging is in fact limited.

thanks in advance for any help,
Christophe

[-- Attachment #2: lunar.tgz --]
[-- Type: application/x-compressed-tar, Size: 59498 bytes --]