From: Eric Dumazet <dada1@cosmosbay.com>
To: Denys Fedoryshchenko <denys@visp.net.lb>
Cc: David Miller <davem@davemloft.net>,
kaber@trash.net, netdev@vger.kernel.org,
netfilter-devel@vger.kernel.org
Subject: Re: kernel 2.6.25-rc7 highly unstable on high load
Date: Thu, 27 Mar 2008 19:56:09 +0100 [thread overview]
Message-ID: <47EBEDC9.6080100@cosmosbay.com> (raw)
In-Reply-To: <20080327183745.M9944@visp.net.lb>
Denys Fedoryshchenko a écrit :
> Here is output from dmesg with patch you supplied.
>
> Kup /config # rtstat -i60 -c60
> rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
> rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
> rt_cache|
> entries| in_hit|in_slow_|in_slow_|in_no_ro| in_brd|in_marti|in_marti|
> out_hit|out_slow|out_slow|gc_total|gc_ignor|gc_goal_|gc_dst_o|in_hlist|
> out_hlis|
> | | tot| mc| ute| | an_dst|
> an_src| | _tot| _mc| | ed| miss| verflow|
> _search|t_search|
> 103266| 69986| 11574| 0| 54| 0| 0| 0|
> 1982| 634| 0| 10984| 10980| 0| 0| 191808| 5254|
> 124787| 45125| 6019| 0| 28| 0| 0| 0|
> 807| 230| 0| 6277| 6274| 0| 0| 128922| 2518|
> 120270| 45588| 6288| 0| 30| 0| 0| 0|
> 883| 214| 0| 6532| 6529| 0| 0| 125651| 2743|
> 122253| 46522| 6582| 0| 27| 0| 0| 0|
> 897| 213| 0| 6822| 6819| 0| 0| 124927| 2761|
>
> [ 102.534363] dst_total: 120397 delayed: 12 work_perf: 0 expires: 27999
> elapsed: 1 us
> [ 130.530240] dst_total: 124277 delayed: 12 work_perf: 0 expires: 32998
> elapsed: 2 us
> [ 163.523240] dst_total: 110006 delayed: 12 work_perf: 0 expires: 39000
> elapsed: 1 us
> [ 202.519402] dst_total: 130453 delayed: 12 work_perf: 0 expires: 45998
> elapsed: 1 us
> [ 248.511220] dst_total: 110637 delayed: 12 work_perf: 0 expires: 52600
> elapsed: 2 us
> [ 301.102445] dst_total: 129366 delayed: 12 work_perf: 0 expires: 60696
> elapsed: 6 us
>
> After while
>
> Kup /config # rtstat -i300 -c60
> rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
> rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
> rt_cache|
> entries| in_hit|in_slow_|in_slow_|in_no_ro| in_brd|in_marti|in_marti|
> out_hit|out_slow|out_slow|gc_total|gc_ignor|gc_goal_|gc_dst_o|in_hlist|
> out_hlis|
> | | tot| mc| ute| | an_dst|
> an_src| | _tot| _mc| | ed| miss| verflow|
> _search|t_search|
> 129138| 64166| 9864| 0| 46| 0| 0| 0|
> 1334| 358| 0| 10012| 10008| 0| 0| 173006| 3985|
> 130125| 43971| 10829| 0| 70| 0| 0| 0|
> 907| 195| 0| 11093| 11090| 0| 0| 100601| 2848|
> 144857| 32041| 22233| 0| 234| 0| 0| 0|
> 738| 261| 0| 22727| 22724| 2| 0| 42769| 811|
>
> [ 687.740779] dst_total: 129365 delayed: 12 work_perf: 0 expires: 104999
> elapsed: 1 us
> [ 792.736043] dst_total: 127247 delayed: 101589 work_perf: 0 expires: 600
> elapsed: 11274 us
> [ 793.347127] dst_total: 122440 delayed: 101589 work_perf: 0 expires: 1600
> elapsed: 11150 us
> [ 794.957419] dst_total: 123780 delayed: 101589 work_perf: 0 expires: 3100
> elapsed: 10896 us
> [ 798.068614] dst_total: 121992 delayed: 101589 work_perf: 0 expires: 5655
> elapsed: 11561 us
> [ 803.734450] dst_total: 120552 delayed: 101589 work_perf: 0 expires: 7988
> elapsed: 12254 us
> [ 811.733227] dst_total: 121105 delayed: 101589 work_perf: 0 expires: 10988
> elapsed: 12209 us
> [ 822.733196] dst_total: 120142 delayed: 101589 work_perf: 0 expires: 13987
> elapsed: 12039 us
> [ 836.730637] dst_total: 120823 delayed: 101589 work_perf: 0 expires: 17987
> elapsed: 12448 us
> [ 854.726913] dst_total: 127478 delayed: 101589 work_perf: 0 expires: 22988
> elapsed: 11435 us
> [ 877.724827] dst_total: 127497 delayed: 101589 work_perf: 0 expires: 27987
> elapsed: 12833 us
> [ 905.719818] dst_total: 126693 delayed: 101589 work_perf: 0 expires: 32987
> elapsed: 12068 us
> [ 938.714595] dst_total: 130740 delayed: 101589 work_perf: 0 expires: 38988
> elapsed: 11970 us
> [ 977.707568] dst_total: 135449 delayed: 101589 work_perf: 0 expires: 45989
> elapsed: 11747 us
> [ 1023.701554] dst_total: 141181 delayed: 101589 work_perf: 0 expires: 52988
> elapsed: 12653 us
> [ 1076.694012] dst_total: 147895 delayed: 101588 work_perf: 1 expires: 59987
> elapsed: 12217 us
> [ 1136.684805] dst_total: 154798 delayed: 101588 work_perf: 0 expires: 67987
> elapsed: 12026 us
> [ 1204.673424] dst_total: 163376 delayed: 101588 work_perf: 0 expires: 76988
> elapsed: 11151 us
>
> and after 30 minutes more
>
> Kup /config # rtstat -i300 -c60
> rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
> rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
> rt_cache|
> entries| in_hit|in_slow_|in_slow_|in_no_ro| in_brd|in_marti|in_marti|
> out_hit|out_slow|out_slow|gc_total|gc_ignor|gc_goal_|gc_dst_o|in_hlist|
> out_hlis|
> | | tot| mc| ute| | an_dst|
> an_src| | _tot| _mc| | ed| miss| verflow|
> _search|t_search|
> 194841| 176962| 77964| 0| 684| 1| 0| 0|
> 3939| 1143| 0| 79531| 79516| 7| 0| 401258| 9455|
> 223224| 28262| 26499| 0| 236| 0| 0| 0|
> 661| 216| 0| 26951| 26947| 3| 0| 93059| 1946|
> 248425| 28257| 26784| 0| 226| 0| 0| 0|
> 678| 202| 0| 27211| 27208| 3| 0| 115593| 2412|
> 272823| 28943| 26728| 0| 246| 0| 0| 0|
> 695| 200| 0| 27174| 27171| 3| 0| 135256| 2942|
> 295087| 28835| 26701| 0| 252| 0| 0| 0|
> 687| 194| 0| 27146| 27143| 3| 0| 154269| 3312|
>
>
> [ 2043.547879] dst_total: 242922 delayed: 101588 work_perf: 0 expires: 119987
> elapsed: 12519 us
> [ 2163.529997] dst_total: 252951 delayed: 101588 work_perf: 0 expires: 119987
> elapsed: 13930 us
> [ 2283.512215] dst_total: 262678 delayed: 101588 work_perf: 0 expires: 119986
> elapsed: 14330 us
> [ 2403.508447] dst_total: 271930 delayed: 101588 work_perf: 0 expires: 119972
> elapsed: 15230 us
> [ 2523.476844] dst_total: 288046 delayed: 101588 work_perf: 0 expires: 119985
> elapsed: 15018 us
> [ 2643.456996] dst_total: 296475 delayed: 101588 work_perf: 0 expires: 119987
> elapsed: 13654 us
> [ 2763.438267] dst_total: 305033 delayed: 101588 work_perf: 0 expires: 119987
> elapsed: 13137 us
>
>
Very interesting... you can see 101588 dst are *delayed* in dst_garbage,
but apparently never freed.
Something is wrong, but the count seems stable. Must be some kind of
event, admin driven or something...
might be the "ip route flush cache" that is schedlued around 600 seconds
of machine alive, then secret_interval seconds later...
Typically, when a "ip route flush cache" is done (manually or triggered
by secret_interval timer), refcounted>0 entries are put into dst_garbage.
Then when some trafic occurs on the flows involved, IP stack should
decrement refcount so that next dst_garbage round can free the deleted
entries. Normal TCP connections are doing this correctly. On your
machine, nothing.
Possibly idle tcp sessions ?
Maybe some netfilter problem ?
Please tell us more about your machine ;)
next prev parent reply other threads:[~2008-03-27 18:56 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2008-03-27 6:35 kernel 2.6.25-rc7 highly unstable on high load Denys Fedoryshchenko
2008-03-27 6:40 ` David Miller
2008-03-27 7:03 ` Eric Dumazet
2008-03-27 8:36 ` Denys Fedoryshchenko
2008-03-27 8:48 ` Denys Fedoryshchenko
2008-03-27 14:41 ` Denys Fedoryshchenko
2008-03-27 14:52 ` Eric Dumazet
[not found] ` <20080327151520.M89250@visp.net.lb>
2008-03-27 16:07 ` Eric Dumazet
2008-03-27 16:29 ` Eric Dumazet
2008-03-27 18:37 ` Denys Fedoryshchenko
2008-03-27 18:56 ` Eric Dumazet [this message]
2008-03-27 19:05 ` Denis V. Lunev
2008-03-27 22:03 ` David Miller
2008-03-28 0:47 ` Denys Fedoryshchenko
2008-03-28 4:50 ` Denys Fedoryshchenko
2008-03-28 5:25 ` Denys Fedoryshchenko
2008-03-28 5:49 ` Eric Dumazet
2008-03-28 7:38 ` Denys Fedoryshchenko
2008-03-28 7:46 ` Eric Dumazet
2008-03-28 9:50 ` Denys Fedoryshchenko
2008-03-28 15:57 ` Eric Dumazet
2008-03-28 20:23 ` Stephen Hemminger
2008-03-28 20:45 ` Denys Fedoryshchenko
2008-03-28 21:17 ` Eric Dumazet
2008-03-28 23:21 ` David Miller
2008-03-29 9:35 ` Denys Fedoryshchenko
[not found] ` <20080328220902.M5377@visp.net.lb>
2008-03-29 8:31 ` Eric Dumazet
2008-03-27 7:07 ` Denys Fedoryshchenko
2008-03-27 15:01 ` Patrick McHardy
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=47EBEDC9.6080100@cosmosbay.com \
--to=dada1@cosmosbay.com \
--cc=davem@davemloft.net \
--cc=denys@visp.net.lb \
--cc=kaber@trash.net \
--cc=netdev@vger.kernel.org \
--cc=netfilter-devel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).