From: Simon Kirby <sim@netnation.com>
To: Eric Dumazet <dada1@cosmosbay.com>
Cc: Robert Olsson <Robert.Olsson@data.slu.se>, netdev@oss.sgi.com
Subject: Re: Route cache performance
Date: Tue, 23 Aug 2005 12:08:53 -0700 [thread overview]
Message-ID: <20050823190852.GA20794@netnation.com> (raw)
In-Reply-To: <43014E27.1070104@cosmosbay.com>
On Tue, Aug 16, 2005 at 04:23:35AM +0200, Eric Dumazet wrote:
> Hi Simon
Hi there!
> I think one of the reason linux 2.6 has worst results is because HZ=1000
> (instead of HZ=100 for linux 2.4)
> So if rt_garbage_collect() has heavy work to do, it usually break out of
> the loop because of :
>
> } while (!in_softirq() && time_before_eq(jiffies, now));
I was under the impression, however, that the code Alexei added last time
I brought up this problem was intended to always allow gc when the the
table is full and another entry is attempting to be created, even when
under gc_min_interval. I'm actually not even interested (yet) with
the gc_interval/timer case because I'm testing currently with a flow
creation rate of much larger than max_size per second (the minimum
gc_interval being one second).
> Could you please test latest 2.6.13-rc6 kernel on the Opteron machine,
> compiled with HZ=100, with the appended kernel argument :
>
> rhash_entries=8191 ( or rhash_entries=16383 )
>
> and
>
> echo 1 >/proc/sys/net/ipv4/route/gc_interval
> echo 2 >/proc/sys/net/ipv4/route/gc_elasticity
>
> Could you also post some data from your router (like : rtstat -c 20 -i 1)
Sure. Here are results from 2.6.13-rc6 with HZ=100 and
rhash_entries=8191, which sets the max_size to 131072. I'm using
lnstat becuase the rtstat version I could find doesn't work on
newer kernels:
lnstat -c -1 -i 1 -f rt_cache -k entries,in_hit,in_slow_tot,gc_total,gc_ignored,gc_goal_miss,gc_dst_overflow,in_hlist_search
The sender is running "juno 192.168.1.1 31313 0" (juno-z.101f.c):
pid 18492: ran for 40s, 13595333 packets out, 16241091 bytes/s
(~340kpps)
Without tweaks to gc_interval and gc_elasticity:
rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
entries| in_hit|in_slow_|gc_total|gc_ignor|gc_goal_|gc_dst_o|in_hlist|
| | tot| | ed| miss| verflow| _search|
32| 117| 419| 0| 0| 0| 0| 0|
32| 6| 0| 0| 0| 0| 0| 0|
32| 2| 0| 0| 0| 0| 0| 0|
33| 2| 4| 0| 0| 0| 0| 0|
9033| 2| 9002| 840| 839| 0| 0| 4962|
131062| 22| 125633| 125629| 125447| 182| 181| 837163|
131062| 0| 13511| 13509| 900| 12609| 12609| 10|
131062| 0| 8772| 8770| 600| 8170| 8170| 7|
131062| 0| 8709| 8706| 600| 8106| 8106| 8|
131062| 0| 8771| 8770| 600| 8170| 8170| 6|
131062| 0| 8770| 8768| 600| 8168| 8168| 6|
131062| 0| 8706| 8704| 600| 8104| 8104| 10|
131062| 0| 8770| 8770| 600| 8170| 8170| 5|
131062| 0| 8708| 8706| 600| 8106| 8106| 5|
131062| 0| 8770| 8769| 600| 8169| 8169| 6|
131062| 0| 8770| 8769| 600| 8169| 8169| 10|
131062| 0| 8713| 8706| 600| 8106| 8106| 7|
131062| 0| 8786| 8769| 600| 8169| 8169| 9|
With tweaks (and after 60 seconds to wait for timer expiry):
rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
entries| in_hit|in_slow_|gc_total|gc_ignor|gc_goal_|gc_dst_o|in_hlist|
| | tot| | ed| miss| verflow| _search|
28| 632| 424656| 413834| 145906| 267927| 267926| 842370|
28| 2| 3| 0| 0| 0| 0| 0|
28| 3| 2| 0| 0| 0| 0| 0|
28| 2| 4| 0| 0| 0| 0| 0|
35129| 3| 35999| 27826| 27825| 0| 0| 61913|
131062| 6| 102045| 102043| 99432| 2611| 2610| 288926|
131062| 0| 13446| 13442| 900| 12542| 12542| 11|
131062| 0| 11914| 11909| 800| 11109| 11109| 5|
131062| 0| 8772| 8770| 599| 8171| 8170| 5|
131062| 0| 8708| 8708| 600| 8108| 8108| 7|
131062| 0| 8774| 8771| 600| 8171| 8171| 2|
131062| 0| 8769| 8769| 600| 8169| 8169| 9|
131062| 0| 8706| 8704| 600| 8104| 8104| 4|
131062| 0| 8769| 8768| 599| 8169| 8168| 5|
131062| 0| 8707| 8706| 600| 8106| 8106| 7|
131062| 0| 8771| 8768| 600| 8168| 8168| 6|
131062| 0| 8770| 8768| 600| 8168| 8168| 8|
131062| 0| 8705| 8704| 600| 8104| 8104| 6|
131062| 0| 8771| 8768| 600| 8168| 8168| 5|
No visible difference to me.
On stock 2.4.31 with no alterations to the gc settings (and no
rhash_entries as it doesn't exist), lnstat shows:
rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
entries| in_hit|in_slow_|gc_total|gc_ignor|gc_goal_|gc_dst_o|in_hlist|
| | tot| | ed| miss| verflow| _search|
21| 85| 160| 0| 0| 0| 0| 0|
21| 4| 2| 0| 0| 0| 0| 0|
21| 2| 3| 0| 0| 0| 0| 0|
22| 2| 2| 0| 0| 0| 0| 0|
18432| 11| 136187| 134158| 134156| 1| 0| 1133784|
18432| 5| 195891| 195889| 195887| 2| 0| 1763070|
18432| 9| 195585| 195568| 195566| 2| 0| 1758397|
18432| 7| 195290| 195281| 195279| 0| 0| 1751884|
18432| 8| 195587| 195579| 195577| 0| 0| 1754813|
18432| 20| 195276| 195275| 195273| 0| 0| 1752216|
18432| 11| 194983| 194980| 194978| 0| 0| 1749822|
18432| 7| 195288| 195287| 195285| 0| 0| 1752655|
18432| 13| 195282| 195281| 195279| 0| 0| 1752869|
18432| 12| 194984| 194984| 194982| 1| 0| 1749589|
18432| 17| 194978| 194974| 194972| 0| 0| 1748817|
18432| 11| 194985| 194981| 194979| 0| 0| 1749182|
18432| 14| 194981| 194977| 194975| 0| 0| 1749287|
18432| 14| 194682| 194679| 194677| 0| 0| 1746847|
18432| 11| 194983| 194980| 194978| 0| 0| 1749679|
...and the machine is perfectly responsive. It's dropping packets
(managing to forward ~210 kpps, a little less than 2.4.27), but it's
at least working. 2.6.13-rc6 dribbles out ~33 kpps.
Simon-
next prev parent reply other threads:[~2005-08-23 19:08 UTC|newest]
Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top
2005-08-15 21:38 Route cache performance Simon Kirby
2005-08-16 2:23 ` Eric Dumazet
2005-08-23 19:08 ` Simon Kirby [this message]
2005-08-23 19:56 ` Robert Olsson
2005-08-24 0:01 ` Simon Kirby
2005-08-24 3:50 ` Robert Olsson
2005-08-25 18:11 ` Simon Kirby
2005-08-25 20:05 ` Alexey Kuznetsov
2005-08-25 21:22 ` Simon Kirby
2005-08-26 11:55 ` Alexey Kuznetsov
2005-08-26 19:49 ` Robert Olsson
2005-09-06 23:57 ` Simon Kirby
2005-09-07 1:19 ` Alexey Kuznetsov
2005-09-07 15:03 ` Robert Olsson
2005-09-07 16:55 ` Simon Kirby
2005-09-07 17:21 ` Robert Olsson
2005-09-07 14:45 ` Robert Olsson
2005-09-07 16:28 ` Simon Kirby
2005-09-07 16:49 ` Robert Olsson
2005-09-07 16:57 ` Simon Kirby
2005-09-07 19:59 ` Alexey Kuznetsov
2005-09-13 22:14 ` Simon Kirby
2005-09-14 8:04 ` Robert Olsson
2005-09-17 0:28 ` Simon Kirby
2005-09-17 9:04 ` Martin Josefsson
2005-09-17 15:17 ` jamal
2005-09-15 21:04 ` Alexey Kuznetsov
2005-09-15 21:30 ` Robert Olsson
2005-09-15 22:21 ` Alexey Kuznetsov
2005-09-16 12:18 ` Robert Olsson
2005-09-16 19:04 ` Alexey Kuznetsov
2005-09-16 19:22 ` Ben Greear
2005-09-16 19:57 ` Robert Olsson
-- strict thread matches above, loose matches on Subject: below --
2005-08-24 16:06 Simon Kirby
[not found] <20050301220743.GF2554@netnation.com>
[not found] ` <16940.9990.975632.115834@robur.slu.se>
2005-03-09 1:45 ` Simon Kirby
2005-03-09 12:05 ` Robert Olsson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20050823190852.GA20794@netnation.com \
--to=sim@netnation.com \
--cc=Robert.Olsson@data.slu.se \
--cc=dada1@cosmosbay.com \
--cc=netdev@oss.sgi.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).