From: "Denys Fedoryshchenko" <denys@visp.net.lb>
To: Eric Dumazet <dada1@cosmosbay.com>
Cc: David Miller <davem@davemloft.net>,
kaber@trash.net, netdev@vger.kernel.org,
netfilter-devel@vger.kernel.org
Subject: Re: kernel 2.6.25-rc7 highly unstable on high load
Date: Thu, 27 Mar 2008 20:37:44 +0200 [thread overview]
Message-ID: <20080327183745.M9944@visp.net.lb> (raw)
In-Reply-To: <47EBC641.8040405@cosmosbay.com>
Here is output from dmesg with patch you supplied.
Kup /config # rtstat -i60 -c60
rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
rt_cache|
entries| in_hit|in_slow_|in_slow_|in_no_ro| in_brd|in_marti|in_marti|
out_hit|out_slow|out_slow|gc_total|gc_ignor|gc_goal_|gc_dst_o|in_hlist|
out_hlis|
| | tot| mc| ute| | an_dst|
an_src| | _tot| _mc| | ed| miss| verflow|
_search|t_search|
103266| 69986| 11574| 0| 54| 0| 0| 0|
1982| 634| 0| 10984| 10980| 0| 0| 191808| 5254|
124787| 45125| 6019| 0| 28| 0| 0| 0|
807| 230| 0| 6277| 6274| 0| 0| 128922| 2518|
120270| 45588| 6288| 0| 30| 0| 0| 0|
883| 214| 0| 6532| 6529| 0| 0| 125651| 2743|
122253| 46522| 6582| 0| 27| 0| 0| 0|
897| 213| 0| 6822| 6819| 0| 0| 124927| 2761|
[ 102.534363] dst_total: 120397 delayed: 12 work_perf: 0 expires: 27999
elapsed: 1 us
[ 130.530240] dst_total: 124277 delayed: 12 work_perf: 0 expires: 32998
elapsed: 2 us
[ 163.523240] dst_total: 110006 delayed: 12 work_perf: 0 expires: 39000
elapsed: 1 us
[ 202.519402] dst_total: 130453 delayed: 12 work_perf: 0 expires: 45998
elapsed: 1 us
[ 248.511220] dst_total: 110637 delayed: 12 work_perf: 0 expires: 52600
elapsed: 2 us
[ 301.102445] dst_total: 129366 delayed: 12 work_perf: 0 expires: 60696
elapsed: 6 us
After while
Kup /config # rtstat -i300 -c60
rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
rt_cache|
entries| in_hit|in_slow_|in_slow_|in_no_ro| in_brd|in_marti|in_marti|
out_hit|out_slow|out_slow|gc_total|gc_ignor|gc_goal_|gc_dst_o|in_hlist|
out_hlis|
| | tot| mc| ute| | an_dst|
an_src| | _tot| _mc| | ed| miss| verflow|
_search|t_search|
129138| 64166| 9864| 0| 46| 0| 0| 0|
1334| 358| 0| 10012| 10008| 0| 0| 173006| 3985|
130125| 43971| 10829| 0| 70| 0| 0| 0|
907| 195| 0| 11093| 11090| 0| 0| 100601| 2848|
144857| 32041| 22233| 0| 234| 0| 0| 0|
738| 261| 0| 22727| 22724| 2| 0| 42769| 811|
[ 687.740779] dst_total: 129365 delayed: 12 work_perf: 0 expires: 104999
elapsed: 1 us
[ 792.736043] dst_total: 127247 delayed: 101589 work_perf: 0 expires: 600
elapsed: 11274 us
[ 793.347127] dst_total: 122440 delayed: 101589 work_perf: 0 expires: 1600
elapsed: 11150 us
[ 794.957419] dst_total: 123780 delayed: 101589 work_perf: 0 expires: 3100
elapsed: 10896 us
[ 798.068614] dst_total: 121992 delayed: 101589 work_perf: 0 expires: 5655
elapsed: 11561 us
[ 803.734450] dst_total: 120552 delayed: 101589 work_perf: 0 expires: 7988
elapsed: 12254 us
[ 811.733227] dst_total: 121105 delayed: 101589 work_perf: 0 expires: 10988
elapsed: 12209 us
[ 822.733196] dst_total: 120142 delayed: 101589 work_perf: 0 expires: 13987
elapsed: 12039 us
[ 836.730637] dst_total: 120823 delayed: 101589 work_perf: 0 expires: 17987
elapsed: 12448 us
[ 854.726913] dst_total: 127478 delayed: 101589 work_perf: 0 expires: 22988
elapsed: 11435 us
[ 877.724827] dst_total: 127497 delayed: 101589 work_perf: 0 expires: 27987
elapsed: 12833 us
[ 905.719818] dst_total: 126693 delayed: 101589 work_perf: 0 expires: 32987
elapsed: 12068 us
[ 938.714595] dst_total: 130740 delayed: 101589 work_perf: 0 expires: 38988
elapsed: 11970 us
[ 977.707568] dst_total: 135449 delayed: 101589 work_perf: 0 expires: 45989
elapsed: 11747 us
[ 1023.701554] dst_total: 141181 delayed: 101589 work_perf: 0 expires: 52988
elapsed: 12653 us
[ 1076.694012] dst_total: 147895 delayed: 101588 work_perf: 1 expires: 59987
elapsed: 12217 us
[ 1136.684805] dst_total: 154798 delayed: 101588 work_perf: 0 expires: 67987
elapsed: 12026 us
[ 1204.673424] dst_total: 163376 delayed: 101588 work_perf: 0 expires: 76988
elapsed: 11151 us
and after 30 minutes more
Kup /config # rtstat -i300 -c60
rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
rt_cache|
entries| in_hit|in_slow_|in_slow_|in_no_ro| in_brd|in_marti|in_marti|
out_hit|out_slow|out_slow|gc_total|gc_ignor|gc_goal_|gc_dst_o|in_hlist|
out_hlis|
| | tot| mc| ute| | an_dst|
an_src| | _tot| _mc| | ed| miss| verflow|
_search|t_search|
194841| 176962| 77964| 0| 684| 1| 0| 0|
3939| 1143| 0| 79531| 79516| 7| 0| 401258| 9455|
223224| 28262| 26499| 0| 236| 0| 0| 0|
661| 216| 0| 26951| 26947| 3| 0| 93059| 1946|
248425| 28257| 26784| 0| 226| 0| 0| 0|
678| 202| 0| 27211| 27208| 3| 0| 115593| 2412|
272823| 28943| 26728| 0| 246| 0| 0| 0|
695| 200| 0| 27174| 27171| 3| 0| 135256| 2942|
295087| 28835| 26701| 0| 252| 0| 0| 0|
687| 194| 0| 27146| 27143| 3| 0| 154269| 3312|
[ 2043.547879] dst_total: 242922 delayed: 101588 work_perf: 0 expires: 119987
elapsed: 12519 us
[ 2163.529997] dst_total: 252951 delayed: 101588 work_perf: 0 expires: 119987
elapsed: 13930 us
[ 2283.512215] dst_total: 262678 delayed: 101588 work_perf: 0 expires: 119986
elapsed: 14330 us
[ 2403.508447] dst_total: 271930 delayed: 101588 work_perf: 0 expires: 119972
elapsed: 15230 us
[ 2523.476844] dst_total: 288046 delayed: 101588 work_perf: 0 expires: 119985
elapsed: 15018 us
[ 2643.456996] dst_total: 296475 delayed: 101588 work_perf: 0 expires: 119987
elapsed: 13654 us
[ 2763.438267] dst_total: 305033 delayed: 101588 work_perf: 0 expires: 119987
elapsed: 13137 us
On Thu, 27 Mar 2008 17:07:29 +0100, Eric Dumazet wrote
> Denys Fedoryshchenko a :
> > I did this tuning. Not sure what does mean rt_cache_entries, but seems
they
> > are still increasing.
> >
> > Kup /config # rtstat -c1000 -i60
> > rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
> > rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
> > rt_cache|
> > entries| in_hit|in_slow_|in_slow_|in_no_ro| in_brd|in_marti|in_marti|
> > out_hit|out_slow|out_slow|gc_total|gc_ignor|gc_goal_|gc_dst_o|in_hlist|
> > out_hlis|
> > | | tot| mc| ute| | an_dst|
> > an_src| | _tot| _mc| | ed| miss| verflow|
> > _search|t_search|
> > 274234| 1590009| 271953| 0| 1809| 6| 0|
0|
> > 24551| 7365| 0| 279991| 279867| 24| 0| 5438630|
89656|
> > 280535| 25463| 19956| 0| 187| 0| 0|
0|
> > 485| 263| 0| 20407| 20404| 3| 0| 12796|
242|
> > 288072| 25367| 19830| 0| 163| 0| 0|
0|
> > 493| 256| 0| 20248| 20245| 3| 0| 17146|
346|
> > 294943| 25689| 19807| 0| 154| 0| 0|
0|
> > 438| 250| 0| 20211| 20208| 3| 0| 21554|
432|
> > 301118| 25287| 19645| 0| 170| 0| 0|
0|
> > 466| 236| 0| 20052| 20048| 3| 0| 25713|
528|
> > 307651| 25663| 20264| 0| 180| 0| 0|
0|
> > 472| 234| 0| 20678| 20675| 3| 0| 30588|
607|
> > 313767| 25810| 20014| 0| 175| 0| 0|
0|
> > 459| 237| 0| 20425| 20422| 3| 0| 33701|
699|
> >
> >
> >
> Then try to lower gc_elasticity, to 3, or even 2
>
> echo 3 >/proc/sys/net/ipv4/route/gc_elasticity
--
Denys Fedoryshchenko
Technical Manager
Virtual ISP S.A.L.
next prev parent reply other threads:[~2008-03-27 18:37 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2008-03-27 6:35 kernel 2.6.25-rc7 highly unstable on high load Denys Fedoryshchenko
2008-03-27 6:40 ` David Miller
2008-03-27 7:03 ` Eric Dumazet
2008-03-27 8:36 ` Denys Fedoryshchenko
2008-03-27 8:48 ` Denys Fedoryshchenko
2008-03-27 14:41 ` Denys Fedoryshchenko
2008-03-27 14:52 ` Eric Dumazet
[not found] ` <20080327151520.M89250@visp.net.lb>
2008-03-27 16:07 ` Eric Dumazet
2008-03-27 16:29 ` Eric Dumazet
2008-03-27 18:37 ` Denys Fedoryshchenko [this message]
2008-03-27 18:56 ` Eric Dumazet
2008-03-27 19:05 ` Denis V. Lunev
2008-03-27 22:03 ` David Miller
2008-03-28 0:47 ` Denys Fedoryshchenko
2008-03-28 4:50 ` Denys Fedoryshchenko
2008-03-28 5:25 ` Denys Fedoryshchenko
2008-03-28 5:49 ` Eric Dumazet
2008-03-28 7:38 ` Denys Fedoryshchenko
2008-03-28 7:46 ` Eric Dumazet
2008-03-28 9:50 ` Denys Fedoryshchenko
2008-03-28 15:57 ` Eric Dumazet
2008-03-28 20:23 ` Stephen Hemminger
2008-03-28 20:45 ` Denys Fedoryshchenko
2008-03-28 21:17 ` Eric Dumazet
2008-03-28 23:21 ` David Miller
2008-03-29 9:35 ` Denys Fedoryshchenko
[not found] ` <20080328220902.M5377@visp.net.lb>
2008-03-29 8:31 ` Eric Dumazet
2008-03-27 7:07 ` Denys Fedoryshchenko
2008-03-27 15:01 ` Patrick McHardy
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20080327183745.M9944@visp.net.lb \
--to=denys@visp.net.lb \
--cc=dada1@cosmosbay.com \
--cc=davem@davemloft.net \
--cc=kaber@trash.net \
--cc=netdev@vger.kernel.org \
--cc=netfilter-devel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).