Re: Bug, kernel panic, NULL dereference , cleanup_once / icmp_route_lookup.clone.19.clone / nat , 2.6.39-rc7-git11

Denys Fedoryshchenko
2011-05-19T06:39:04Z

On Thu, 19 May 2011 08:30:23 +0200, Eric Dumazet wrote:
> Le jeudi 19 mai 2011 à 09:11 +0300, Denys Fedoryshchenko a écrit :
>> On Thu, 19 May 2011 07:19:57 +0200, Eric Dumazet wrote:
>> > Le mercredi 18 mai 2011 à 21:29 +0200, Eric Dumazet a écrit :
>> >> Le mercredi 18 mai 2011 à 17:52 +0200, Eric Dumazet a écrit :
>> >>
>> >> > Hmm, it seems we have some inetpeer refcount leak somewhere.
>> >> >
>> >> > Maybe one (struct rtable)->peer is not released on dst/rtable
>> >> removal,
>> >> > or we also leak dst/rtable (and their ->peer inetpeer)
>> >> >
>> >> > Watch :
>> >> >
>> >> > grep peer /proc/slabinfo
>> >> > grep dst /proc/slabinfo
>> >> >
>> >>
>> >> FYI, I started a bisection to find the faulty commit.
>> >>
>> >
>> > Oh well, of course this came to 2c8cec5c10bced240
>> > (ipv4: Cache learned PMTU information in inetpeer.)
>> >
>> > So my method to check if we have a leak might be wrong, since the
>> > above
>> > commit let cache full of garbage, and hope that following lookups
>> > will
>> > find and evict obsolete dst.
>> >
>> > Thats getting difficult :(
>> >
>> > Could you please send us
>> >
>> > grep . /proc/sys/net/ipv4/route/*
>> >
>> > Thanks !
>> NewNet-PPPoE ~ # grep . /proc/sys/net/ipv4/route/*
>> /proc/sys/net/ipv4/route/error_burst:5000
>> /proc/sys/net/ipv4/route/error_cost:1000
>> grep: /proc/sys/net/ipv4/route/flush: Permission denied
>> /proc/sys/net/ipv4/route/gc_elasticity:8
>> /proc/sys/net/ipv4/route/gc_interval:60
>> /proc/sys/net/ipv4/route/gc_min_interval:0
>> /proc/sys/net/ipv4/route/gc_min_interval_ms:500
>> /proc/sys/net/ipv4/route/gc_thresh:32768
>> /proc/sys/net/ipv4/route/gc_timeout:300
>> /proc/sys/net/ipv4/route/max_size:524288
>> /proc/sys/net/ipv4/route/min_adv_mss:256
>> /proc/sys/net/ipv4/route/min_pmtu:552
>> /proc/sys/net/ipv4/route/mtu_expires:600
>> /proc/sys/net/ipv4/route/redirect_load:20
>> /proc/sys/net/ipv4/route/redirect_number:9
>> /proc/sys/net/ipv4/route/redirect_silence:20480
>>
>> I think it is default one.
>>
>> PMTU is very actual for that, as it is pppoe, and up to 2k
>> interfaces
>> terminated there.
>>
>
> Yes, and every time an interface is added -> new route added, route
> cache is invalidated (we change rt_genid)
If it matters, there is ifb with shaper on it (for shaping from ppp to
world).

>
>> I don't know, if it matters, but
>> iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS
>> --clamp-mss-to-pmtu
>> also there.
>>
>> I can generate and put "ip route ls cache" and any other info.
>>
>
> Hmm would you please send :
>
> rtstat -c10 -i1
Note, it is offpeak time now, just 1447 interfaces, peak is after 12
hours

NewNet-PPPoE ~ # ./rtstat -c10 -i1
rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
entries| in_hit|in_slow_|in_slow_|in_no_ro|
in_brd|in_marti|in_marti|
out_hit|out_slow|out_slow|gc_total|gc_ignor|gc_goal_|gc_dst_o|in_hlist|out_hlis|
| | tot| mc| ute| | an_dst|
an_src| | _tot| _mc| | ed| miss| verflow|
_search|t_search|
2256|355568844|85929285| 1649| 9| 59954| 293|
1460|14423031| 6865540| 0| 0| 0| 0|
0|22719682| 1262044|
3408| 14887| 2117| 0| 0| 1| 1|
0| 761| 159| 0| 0| 0| 0| 0|
1209| 46|
3189| 17185| 5613| 0| 0| 1| 0|
0| 987| 334| 0| 0| 0| 0| 0|
684| 22|
2698| 18312| 3417| 0| 0| 5| 0|
0| 923| 242| 0| 0| 0| 0| 0|
498| 10|
4996| 17268| 3604| 0| 0| 1| 0|
0| 847| 240| 0| 0| 0| 0| 0|
830| 23|
2457| 16439| 4227| 0| 0| 4| 0|
0| 663| 268| 0| 0| 0| 0| 0|
655| 22|
4763| 16895| 3634| 0| 0| 1| 0|
0| 880| 266| 0| 0| 0| 0| 0|
896| 32|
6299| 19169| 2220| 0| 0| 2| 0|
0| 898| 206| 0| 0| 0| 0| 0|
1213| 60|
7511| 20059| 1597| 0| 0| 2| 1|
0| 855| 197| 0| 0| 0| 0| 0|
1917| 54|
9271| 17731| 2919| 0| 0| 0| 0|
0| 855| 223| 0| 0| 0| 0| 0|
1664| 101|