From: "Denys Fedoryshchenko" <denys@visp.net.lb>
To: netdev@vger.kernel.org, netfilter-devel@vger.kernel.org
Subject: Re: kernel 2.6.25-rc7 highly unstable on high load
Date: Fri, 28 Mar 2008 09:38:54 +0200 [thread overview]
Message-ID: <20080328073854.M73368@visp.net.lb> (raw)
In-Reply-To: <47EC8701.1080604@cosmosbay.com>
Already patched and tested, it doesn't change anything.
On Fri, 28 Mar 2008 06:49:53 +0100, Eric Dumazet wrote
> Denys Fedoryshchenko a :
> > Just to make sure 2.6.24.3 is stable and it is regression i am supplying
> > output from it.
> > Do you want me to submit summary to bugzilla and regression list as well?
> >
> > And in short, IMHO 2.6.25 have major issues on routing that have to be
fixed
> > before release. TRIE is crashing, and even with HASH there is leak. I am
> > trying my best to bisect it, but it is major router and i cannot take
much
> > risk on it, so i wish i can simulate in my home mini-lab. Still i am not
able
> > to get even proper switch (Lebanon difficult country for IT).
> >
> > Kup ~ # uname -a
> > Linux Kup 2.6.24.3-build-0023 #3 SMP Sat Mar 8 13:01:35 EET 2008 i686
unknown
> >
> > up ~ # rtstat -i60 -c6000
> > rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
> > rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
> > rt_cache|
> > entries| in_hit|in_slow_|in_slow_|in_no_ro| in_brd|in_marti|in_marti|
> > out_hit|out_slow|out_slow|gc_total|gc_ignor|gc_goal_|gc_dst_o|in_hlist|
> > out_hlis|
> > | | tot| mc| ute| | an_dst|
> > an_src| | _tot| _mc| | ed| miss| verflow|
> > _search|t_search|
> > 54750| 4430| 1128| 0| 12| 0| 0|
0|
> > 263| 190| 0| 709| 708| 0| 0| 3545|
313|
> > 92913| 8829| 1211| 0| 1| 0| 0|
0|
> > 343| 163| 0| 1375| 1373| 0| 0| 12545|
724|
> > 115323| 8232| 906| 0| 0| 0| 0|
0|
> > 299| 128| 0| 1035| 1033| 0| 0| 18069|
813|
> > 128985| 8650| 839| 0| 0| 0| 0|
0|
> > 289| 115| 0| 954| 952| 0| 0| 22515|
845|
> > 116682| 8911| 861| 0| 0| 0| 0|
0|
> > 288| 117| 0| 978| 976| 0| 0| 23433|
775|
> > 99969| 9164| 889| 0| 0| 0| 0|
0|
> > 280| 113| 0| 1002| 1000| 0| 0| 26741|
839|
> > 124602| 9395| 1012| 0| 0| 0| 0|
0|
> > 271| 122| 0| 1134| 1132| 0| 0| 27381|
787|
> > 110051| 10036| 824| 0| 0| 0| 0|
0|
> > 279| 120| 0| 944| 942| 0| 0| 28558|
783|
> > 126835| 10631| 772| 0| 0| 0| 0|
0|
> > 274| 117| 0| 888| 886| 0| 0| 29451|
780|
> > 111881| 10357| 762| 0| 0| 0| 0|
0|
> > 275| 117| 0| 879| 877| 0| 0| 28235|
751|
> > 127018| 10178| 796| 0| 0| 0| 0|
0|
> > 283| 117| 0| 913| 911| 0| 0| 29480|
807|
> > 112242| 9839| 814| 0| 0| 0| 0|
0|
> > 293| 115| 0| 929| 927| 0| 0| 28095|
796|
> > 41267| 9493| 1217| 0| 1| 0| 0|
0|
> > 269| 138| 0| 811| 810| 0| 0| 18545|
548|
> > 76380| 9722| 1060| 0| 1| 0| 0|
0|
> > 250| 135| 0| 1195| 1193| 0| 0| 14786|
414|
> > 99922| 9811| 779| 0| 0| 0| 0|
0|
> > 281| 124| 0| 902| 900| 0| 0| 21853|
589|
> >
> > Kup ~ # rtstat -i60 -c6000
> > rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
> > rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
> > rt_cache|
> > entries| in_hit|in_slow_|in_slow_|in_no_ro| in_brd|in_marti|in_marti|
> > out_hit|out_slow|out_slow|gc_total|gc_ignor|gc_goal_|gc_dst_o|in_hlist|
> > out_hlis|
> > | | tot| mc| ute| | an_dst|
> > an_src| | _tot| _mc| | ed| miss| verflow|
> > _search|t_search|
> >
> > 122053| 150955| 14888| 0| 25| 1| 0|
0|
> > 4611| 2090| 0| 15820| 15789| 0| 0| 369513|
11562|
> > 105226| 10215| 872| 0| 0| 0| 0|
0|
> > 279| 116| 0| 988| 986| 0| 0| 30343|
799|
> > 126236| 10462| 924| 0| 0| 0| 0|
0|
> > 260| 120| 0| 1044| 1042| 0| 0| 31699|
782|
> > 114492| 9782| 884| 0| 0| 0| 0|
0|
> > 253| 120| 0| 1005| 1003| 0| 0| 29695|
722|
> >
> > After ip route flush cache
> > Kup ~ # rtstat -i60 -c6000
> > rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
> > rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
> > rt_cache|
> > entries| in_hit|in_slow_|in_slow_|in_no_ro| in_brd|in_marti|in_marti|
> > out_hit|out_slow|out_slow|gc_total|gc_ignor|gc_goal_|gc_dst_o|in_hlist|
> > out_hlis|
> > | | tot| mc| ute| | an_dst|
> > an_src| | _tot| _mc| | ed| miss| verflow|
> > _search|t_search|
> > 9088| 202136| 19262| 0| 29| 1| 0|
0|
> > 5976| 2696| 0| 20647| 20606| 0| 0| 521714|
15415|
> >
> >
> > !!!!!
> > I am not wrong, ip route flush cache doesn't work at 2.6.25-rc7. I will
make
> > sure about that now.
> >
> Maybe you are a litle bit too fast for "ip route flush cache" :)
>
> It used to work like that : schedule a timer to start a flush in
> about 2 seconds. A flush meaning : scan the whole table and delete
> all entries.
>
> On machines with 4 millions dst entries, this was using too much
> time and eventually crashing.
>
> On recent kernels, each rtable entry has a special field named
> rt_genid, so that "ip route flush cache" doesnt have to scan the
> whole table, but only change the global genid. rtables entries will
> be deleted later, when their rt_genid is found to be different than
> the global genid.
>
> Please try the patch that was suggested yesterday, as it is probably
> the cure your router needs.
>
> http://git2.kernel.org/?p=linux/kernel/git/davem/net-
2.6.git;a=commitdiff;h=7c0ecc4c4f8fd90988aab8a95297b9c0038b6160
>
> Thank you
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Denys Fedoryshchenko
Technical Manager
Virtual ISP S.A.L.
next prev parent reply other threads:[~2008-03-28 7:38 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2008-03-27 6:35 kernel 2.6.25-rc7 highly unstable on high load Denys Fedoryshchenko
2008-03-27 6:40 ` David Miller
2008-03-27 7:03 ` Eric Dumazet
2008-03-27 8:36 ` Denys Fedoryshchenko
2008-03-27 8:48 ` Denys Fedoryshchenko
2008-03-27 14:41 ` Denys Fedoryshchenko
2008-03-27 14:52 ` Eric Dumazet
[not found] ` <20080327151520.M89250@visp.net.lb>
2008-03-27 16:07 ` Eric Dumazet
2008-03-27 16:29 ` Eric Dumazet
2008-03-27 18:37 ` Denys Fedoryshchenko
2008-03-27 18:56 ` Eric Dumazet
2008-03-27 19:05 ` Denis V. Lunev
2008-03-27 22:03 ` David Miller
2008-03-28 0:47 ` Denys Fedoryshchenko
2008-03-28 4:50 ` Denys Fedoryshchenko
2008-03-28 5:25 ` Denys Fedoryshchenko
2008-03-28 5:49 ` Eric Dumazet
2008-03-28 7:38 ` Denys Fedoryshchenko [this message]
2008-03-28 7:46 ` Eric Dumazet
2008-03-28 9:50 ` Denys Fedoryshchenko
2008-03-28 15:57 ` Eric Dumazet
2008-03-28 20:23 ` Stephen Hemminger
2008-03-28 20:45 ` Denys Fedoryshchenko
2008-03-28 21:17 ` Eric Dumazet
2008-03-28 23:21 ` David Miller
2008-03-29 9:35 ` Denys Fedoryshchenko
[not found] ` <20080328220902.M5377@visp.net.lb>
2008-03-29 8:31 ` Eric Dumazet
2008-03-27 7:07 ` Denys Fedoryshchenko
2008-03-27 15:01 ` Patrick McHardy
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20080328073854.M73368@visp.net.lb \
--to=denys@visp.net.lb \
--cc=netdev@vger.kernel.org \
--cc=netfilter-devel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).