From: Eric Dumazet <dada1@cosmosbay.com>
To: Denys Fedoryshchenko <denys@visp.net.lb>
Cc: David Miller <davem@davemloft.net>,
kaber@trash.net, netdev@vger.kernel.org,
netfilter-devel@vger.kernel.org
Subject: Re: kernel 2.6.25-rc7 highly unstable on high load
Date: Fri, 28 Mar 2008 06:49:53 +0100 [thread overview]
Message-ID: <47EC8701.1080604@cosmosbay.com> (raw)
In-Reply-To: <20080328052543.M60286@visp.net.lb>
Denys Fedoryshchenko a écrit :
> Just to make sure 2.6.24.3 is stable and it is regression i am supplying
> output from it.
> Do you want me to submit summary to bugzilla and regression list as well?
>
> And in short, IMHO 2.6.25 have major issues on routing that have to be fixed
> before release. TRIE is crashing, and even with HASH there is leak. I am
> trying my best to bisect it, but it is major router and i cannot take much
> risk on it, so i wish i can simulate in my home mini-lab. Still i am not able
> to get even proper switch (Lebanon difficult country for IT).
>
> Kup ~ # uname -a
> Linux Kup 2.6.24.3-build-0023 #3 SMP Sat Mar 8 13:01:35 EET 2008 i686 unknown
>
> up ~ # rtstat -i60 -c6000
> rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
> rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
> rt_cache|
> entries| in_hit|in_slow_|in_slow_|in_no_ro| in_brd|in_marti|in_marti|
> out_hit|out_slow|out_slow|gc_total|gc_ignor|gc_goal_|gc_dst_o|in_hlist|
> out_hlis|
> | | tot| mc| ute| | an_dst|
> an_src| | _tot| _mc| | ed| miss| verflow|
> _search|t_search|
> 54750| 4430| 1128| 0| 12| 0| 0| 0|
> 263| 190| 0| 709| 708| 0| 0| 3545| 313|
> 92913| 8829| 1211| 0| 1| 0| 0| 0|
> 343| 163| 0| 1375| 1373| 0| 0| 12545| 724|
> 115323| 8232| 906| 0| 0| 0| 0| 0|
> 299| 128| 0| 1035| 1033| 0| 0| 18069| 813|
> 128985| 8650| 839| 0| 0| 0| 0| 0|
> 289| 115| 0| 954| 952| 0| 0| 22515| 845|
> 116682| 8911| 861| 0| 0| 0| 0| 0|
> 288| 117| 0| 978| 976| 0| 0| 23433| 775|
> 99969| 9164| 889| 0| 0| 0| 0| 0|
> 280| 113| 0| 1002| 1000| 0| 0| 26741| 839|
> 124602| 9395| 1012| 0| 0| 0| 0| 0|
> 271| 122| 0| 1134| 1132| 0| 0| 27381| 787|
> 110051| 10036| 824| 0| 0| 0| 0| 0|
> 279| 120| 0| 944| 942| 0| 0| 28558| 783|
> 126835| 10631| 772| 0| 0| 0| 0| 0|
> 274| 117| 0| 888| 886| 0| 0| 29451| 780|
> 111881| 10357| 762| 0| 0| 0| 0| 0|
> 275| 117| 0| 879| 877| 0| 0| 28235| 751|
> 127018| 10178| 796| 0| 0| 0| 0| 0|
> 283| 117| 0| 913| 911| 0| 0| 29480| 807|
> 112242| 9839| 814| 0| 0| 0| 0| 0|
> 293| 115| 0| 929| 927| 0| 0| 28095| 796|
> 41267| 9493| 1217| 0| 1| 0| 0| 0|
> 269| 138| 0| 811| 810| 0| 0| 18545| 548|
> 76380| 9722| 1060| 0| 1| 0| 0| 0|
> 250| 135| 0| 1195| 1193| 0| 0| 14786| 414|
> 99922| 9811| 779| 0| 0| 0| 0| 0|
> 281| 124| 0| 902| 900| 0| 0| 21853| 589|
>
> Kup ~ # rtstat -i60 -c6000
> rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
> rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
> rt_cache|
> entries| in_hit|in_slow_|in_slow_|in_no_ro| in_brd|in_marti|in_marti|
> out_hit|out_slow|out_slow|gc_total|gc_ignor|gc_goal_|gc_dst_o|in_hlist|
> out_hlis|
> | | tot| mc| ute| | an_dst|
> an_src| | _tot| _mc| | ed| miss| verflow|
> _search|t_search|
>
> 122053| 150955| 14888| 0| 25| 1| 0| 0|
> 4611| 2090| 0| 15820| 15789| 0| 0| 369513| 11562|
> 105226| 10215| 872| 0| 0| 0| 0| 0|
> 279| 116| 0| 988| 986| 0| 0| 30343| 799|
> 126236| 10462| 924| 0| 0| 0| 0| 0|
> 260| 120| 0| 1044| 1042| 0| 0| 31699| 782|
> 114492| 9782| 884| 0| 0| 0| 0| 0|
> 253| 120| 0| 1005| 1003| 0| 0| 29695| 722|
>
> After ip route flush cache
> Kup ~ # rtstat -i60 -c6000
> rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
> rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
> rt_cache|
> entries| in_hit|in_slow_|in_slow_|in_no_ro| in_brd|in_marti|in_marti|
> out_hit|out_slow|out_slow|gc_total|gc_ignor|gc_goal_|gc_dst_o|in_hlist|
> out_hlis|
> | | tot| mc| ute| | an_dst|
> an_src| | _tot| _mc| | ed| miss| verflow|
> _search|t_search|
> 9088| 202136| 19262| 0| 29| 1| 0| 0|
> 5976| 2696| 0| 20647| 20606| 0| 0| 521714| 15415|
>
>
> !!!!!
> I am not wrong, ip route flush cache doesn't work at 2.6.25-rc7. I will make
> sure about that now.
>
Maybe you are a litle bit too fast for "ip route flush cache" :)
It used to work like that : schedule a timer to start a flush in about 2
seconds. A flush meaning : scan the whole table and delete all entries.
On machines with 4 millions dst entries, this was using too much time
and eventually crashing.
On recent kernels, each rtable entry has a special field named rt_genid,
so that "ip route flush cache" doesnt have to scan the whole table, but
only change the global genid. rtables entries will be deleted later,
when their rt_genid is found to be different than the global genid.
Please try the patch that was suggested yesterday, as it is probably the
cure your router needs.
http://git2.kernel.org/?p=linux/kernel/git/davem/net-2.6.git;a=commitdiff;h=7c0ecc4c4f8fd90988aab8a95297b9c0038b6160
Thank you
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2008-03-28 5:49 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2008-03-27 6:35 kernel 2.6.25-rc7 highly unstable on high load Denys Fedoryshchenko
2008-03-27 6:40 ` David Miller
2008-03-27 7:03 ` Eric Dumazet
2008-03-27 8:36 ` Denys Fedoryshchenko
2008-03-27 8:48 ` Denys Fedoryshchenko
2008-03-27 14:41 ` Denys Fedoryshchenko
2008-03-27 14:52 ` Eric Dumazet
[not found] ` <20080327151520.M89250@visp.net.lb>
2008-03-27 16:07 ` Eric Dumazet
2008-03-27 16:29 ` Eric Dumazet
2008-03-27 18:37 ` Denys Fedoryshchenko
2008-03-27 18:56 ` Eric Dumazet
2008-03-27 19:05 ` Denis V. Lunev
2008-03-27 22:03 ` David Miller
2008-03-28 0:47 ` Denys Fedoryshchenko
2008-03-28 4:50 ` Denys Fedoryshchenko
2008-03-28 5:25 ` Denys Fedoryshchenko
2008-03-28 5:49 ` Eric Dumazet [this message]
2008-03-28 7:38 ` Denys Fedoryshchenko
2008-03-28 7:46 ` Eric Dumazet
2008-03-28 9:50 ` Denys Fedoryshchenko
2008-03-28 15:57 ` Eric Dumazet
2008-03-28 20:23 ` Stephen Hemminger
2008-03-28 20:45 ` Denys Fedoryshchenko
2008-03-28 21:17 ` Eric Dumazet
2008-03-28 23:21 ` David Miller
2008-03-29 9:35 ` Denys Fedoryshchenko
[not found] ` <20080328220902.M5377@visp.net.lb>
2008-03-29 8:31 ` Eric Dumazet
2008-03-27 7:07 ` Denys Fedoryshchenko
2008-03-27 15:01 ` Patrick McHardy
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=47EC8701.1080604@cosmosbay.com \
--to=dada1@cosmosbay.com \
--cc=davem@davemloft.net \
--cc=denys@visp.net.lb \
--cc=kaber@trash.net \
--cc=netdev@vger.kernel.org \
--cc=netfilter-devel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).