Re: Bug, kernel panic, NULL dereference , cleanup_once / icmp_route_lookup.clone.19.clone / nat , 2.6.39-rc7-git11

[Denys Fedoryshchenko <denys@visp.net.lb>]

 On Thu, 19 May 2011 08:30:23 +0200, Eric Dumazet wrote:
> Le jeudi 19 mai 2011 à 09:11 +0300, Denys Fedoryshchenko a écrit :
>> On Thu, 19 May 2011 07:19:57 +0200, Eric Dumazet wrote:
>> > Le mercredi 18 mai 2011 à 21:29 +0200, Eric Dumazet a écrit :
>> >> Le mercredi 18 mai 2011 à 17:52 +0200, Eric Dumazet a écrit :
>> >>
>> >> > Hmm, it seems we have some inetpeer refcount leak somewhere.
>> >> >
>> >> > Maybe one (struct rtable)->peer is not released on dst/rtable
>> >> removal,
>> >> > or we also leak dst/rtable (and their ->peer inetpeer)
>> >> >
>> >> > Watch :
>> >> >
>> >> > grep peer /proc/slabinfo
>> >> > grep dst /proc/slabinfo
>> >> >
>> >>
>> >> FYI, I started a bisection to find the faulty commit.
>> >>
>> >
>> > Oh well, of course this came to 2c8cec5c10bced240
>> > (ipv4: Cache learned PMTU information in inetpeer.)
>> >
>> > So my method to check if we have a leak might be wrong, since the
>> > above
>> > commit let cache full of garbage, and hope that following lookups
>> > will
>> > find and evict obsolete dst.
>> >
>> > Thats getting difficult :(
>> >
>> > Could you please send us
>> >
>> > grep . /proc/sys/net/ipv4/route/*
>> >
>> > Thanks !
>>  NewNet-PPPoE ~ # grep . /proc/sys/net/ipv4/route/*
>>  /proc/sys/net/ipv4/route/error_burst:5000
>>  /proc/sys/net/ipv4/route/error_cost:1000
>>  grep: /proc/sys/net/ipv4/route/flush: Permission denied
>>  /proc/sys/net/ipv4/route/gc_elasticity:8
>>  /proc/sys/net/ipv4/route/gc_interval:60
>>  /proc/sys/net/ipv4/route/gc_min_interval:0
>>  /proc/sys/net/ipv4/route/gc_min_interval_ms:500
>>  /proc/sys/net/ipv4/route/gc_thresh:32768
>>  /proc/sys/net/ipv4/route/gc_timeout:300
>>  /proc/sys/net/ipv4/route/max_size:524288
>>  /proc/sys/net/ipv4/route/min_adv_mss:256
>>  /proc/sys/net/ipv4/route/min_pmtu:552
>>  /proc/sys/net/ipv4/route/mtu_expires:600
>>  /proc/sys/net/ipv4/route/redirect_load:20
>>  /proc/sys/net/ipv4/route/redirect_number:9
>>  /proc/sys/net/ipv4/route/redirect_silence:20480
>>
>>  I think it is default one.
>>
>>  PMTU is very actual for that, as it is pppoe, and up to 2k 
>> interfaces
>>  terminated there.
>>
>
> Yes, and every time an interface is added -> new route added, route
> cache is invalidated (we change rt_genid)
 If it matters, there is ifb with shaper on it (for shaping from ppp to 
 world).

>
>>  I don't know, if it matters, but
>>  iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS
>>  --clamp-mss-to-pmtu
>>  also there.
>>
>>  I can generate and put "ip route ls cache" and any other info.
>>
>
> Hmm would you please send :
>
> rtstat -c10 -i1
 Note, it is offpeak time now, just 1447 interfaces, peak is after 12 
 hours

 NewNet-PPPoE ~ # ./rtstat -c10 -i1
 rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
  entries|  in_hit|in_slow_|in_slow_|in_no_ro|  
 in_brd|in_marti|in_marti| 
 out_hit|out_slow|out_slow|gc_total|gc_ignor|gc_goal_|gc_dst_o|in_hlist|out_hlis|
         |        |     tot|      mc|     ute|        |  an_dst|  
 an_src|        |    _tot|     _mc|        |      ed|    miss| verflow| 
 _search|t_search|
     2256|355568844|85929285|    1649|       9|   59954|     293|    
 1460|14423031| 6865540|       0|       0|       0|       0|       
 0|22719682| 1262044|
     3408|   14887|    2117|       0|       0|       1|       1|       
 0|     761|     159|       0|       0|       0|       0|       0|    
 1209|      46|
     3189|   17185|    5613|       0|       0|       1|       0|       
 0|     987|     334|       0|       0|       0|       0|       0|     
 684|      22|
     2698|   18312|    3417|       0|       0|       5|       0|       
 0|     923|     242|       0|       0|       0|       0|       0|     
 498|      10|
     4996|   17268|    3604|       0|       0|       1|       0|       
 0|     847|     240|       0|       0|       0|       0|       0|     
 830|      23|
     2457|   16439|    4227|       0|       0|       4|       0|       
 0|     663|     268|       0|       0|       0|       0|       0|     
 655|      22|
     4763|   16895|    3634|       0|       0|       1|       0|       
 0|     880|     266|       0|       0|       0|       0|       0|     
 896|      32|
     6299|   19169|    2220|       0|       0|       2|       0|       
 0|     898|     206|       0|       0|       0|       0|       0|    
 1213|      60|
     7511|   20059|    1597|       0|       0|       2|       1|       
 0|     855|     197|       0|       0|       0|       0|       0|    
 1917|      54|
     9271|   17731|    2919|       0|       0|       0|       0|       
 0|     855|     223|       0|       0|       0|       0|       0|    
 1664|     101|