Re: Kernel Panic every 2 weeks on ISP server (NULL pointer dereference)

[Luciano Ruete <lruete@sequre.com.ar>]

On Sunday, October 23, 2011 02:16:29 am Eric Dumazet wrote:
> Le samedi 22 octobre 2011 à 22:18 -0300, Luciano Ruete a écrit :
> > Hi,
> > 
> > I'm the sysadmin at a 3500 customers ISP, wich runs an iptables+tc
> > solution for load balancing and QoS.
> > 
> > Every 2 or 3 weeks the server panics with a "NULL pointer dereference"
> > and with IP at "dev_queue_xmit"
> > 
> > It is curious that if i disable MSI on the network card driver this
> > panics seems to disapear, does this ring a bell?
> > 
> > The server is an IBM, previously with Broadcom NetXtreme II BCM5709 nics
> > and now with Intel 82576. I change the nics thinking that maybe the bug
> > was in Broadcom Driver but it seems to affect MSI in general.
> > 
> > The tc+iptables rules are auto-generated with sequreisp[1] an ISP
> > solution that i wrote and is open sourced under AGPLv3.
> > 
> > Tell me if you need any further information, and plz CC because I'm not
> > suscribed.
> > 
> > 
> > root@server:~# uname -a
> > Linux server 2.6.35-30-server #60~lucid1-Ubuntu SMP Tue Sep 20 22:28:40
> > UTC 2011 x86_64 GNU/Linux
> > 
> > 
> > [1]https://github.com/sequre/sequreisp
> 
> Hi Luciano

Hi Eric!

Thanks for your answer...

> 
> [694250.472081] Code: f6
> 49 c1 e6 07          shl    $0x7,%r14
> 66 89 93 ac 00 00 00 mov    %dx,0xac(%rbx)
>[...]
> This looks like a dev_pick_tx() bug, using an out of bound
> queue_index number and returning a txq pointing after
> the device allocated array.

Clear explanation, is there a tool to map the trace to kernel code, or you did 
this by hand? 

> With recent kernels, this cannot happen anymore because
> we added fixes in this area.
> 
> You could try Ubuntu 11.10 (based on linux 3.0) kernel
> on your server, or apply following patch :
> 
> commit df32cc193ad88f7b1326b90af799c927b27f7654
> Author: Tom Herbert <therbert@google.com>
> Date:   Mon Nov 1 12:55:52 2010 -0700
> 
>     net: check queue_index from sock is valid for device
> 
>     In dev_pick_tx recompute the queue index if the value stored in the
>     socket is greater than or equal to the number of real queues for the
>     device.  The saved index in the sock structure is not guaranteed to
>     be appropriate for the egress device (this could happen on a route
>     change or in presence of tunnelling).  The result of the queue index
>     being bad would be to return a bogus queue (crash could prersumably
>     follow).

Lot of ruote changes in this server, there are 30 upstream providers(15 are 
dynamic IP ADSLs) load balanced using VLANs and a VLAN switch.

Thanks again i will try the kernel upgrade and post results in this thread.

Regards!
-- 
Luciano Ruete
Sequre - Sys Admin
Mitre 617, piso 7, of. 1 
+54 261 4254894
Mendoza - Argentina
http://www.sequreisp.com/
http://www.sequre.com.ar/