* V2.4 policy router operates faster/better than V2.6
@ 2005-01-03 20:55 Jeremy M. Guthrie
2005-01-03 22:51 ` Stephen Hemminger
0 siblings, 1 reply; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-03 20:55 UTC (permalink / raw)
To: netdev
[-- Attachment #1.1: Type: text/plain, Size: 3833 bytes --]
I have a dual processor box running Suse 9.1 Ent. that I changed over to the
V2.6.10 kernel. The box has two interfaces in it, both E1000s. The box
receives anywhere from 200mbit to 500+ mbit that it needs to route out to
other boxes. The policy routing table is running ~ 150-200 rules. ie. data
comes in E3(e1000), is policy routed to a destination sent out E2(e1000).
Under V2.4 kernels, the system will operate just fine and drop few packets if
any. ie. right now under V2.4, I have dropped all of three packets. Under
2.6, I can watch the RX drop counter increment. See below.
[h-pr-msn-1 guthrie 1:48pm]~-> ifconfig eth3 ; sleep 10 ; ifconfig eth3
eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
inet addr:10.253.0.1 Bcast:10.255.255.255 Mask:255.255.255.0
inet6 addr: fe80::202:b3ff:fed5:7e30/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:132919934 errors:311285 dropped:311285 overruns:247225
frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2630721320 (2508.8 Mb) TX bytes:484 (484.0 b)
Base address:0x22a0 Memory:eff80000-effa0000
eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
inet addr:10.253.0.1 Bcast:10.255.255.255 Mask:255.255.255.0
inet6 addr: fe80::202:b3ff:fed5:7e30/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:133847068 errors:325697 dropped:325697 overruns:258546
frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3102796062 (2959.0 Mb) TX bytes:484 (484.0 b)
Base address:0x22a0 Memory:eff80000-effa0000
If I turn off the policy routing, I instantly stop getting RX errors or
overruns as it appears the CPU can now pay attention to the packets coming in
and drop them(as I turned off IP forwarding as well).
V2.4 Kernel mpstat data:
command: mpstat -P ALL 60
Linux 2.4.21-251-smp (h-pr-msn-1) 12/15/2004
01:16:24 PM CPU %user %nice %system %idle intr/s
01:17:19 PM all 0.16 0.00 50.12 49.72 42114.18
01:17:19 PM 0 0.12 0.00 55.60 44.28 42114.18
01:17:19 PM 1 0.20 0.00 44.65 55.15 42114.18
01:17:19 PM CPU %user %nice %system %idle intr/s
01:18:19 PM all 0.13 0.00 48.49 51.38 42103.08
01:18:19 PM 0 0.13 0.00 31.88 67.98 42103.08
01:18:19 PM 1 0.13 0.00 65.10 34.77 42103.08
V2.6 kernel mpstat data:
command: mpstat -P ALL 60
Linux 2.6.5-7.111.5-smp (h-pr-msn-1) 12/15/04
13:36:25 CPU %user %nice %system %iowait %irq %soft %idle
intr/s
13:37:25 all 0.13 0.00 0.15 0.09 2.03 43.14 54.45
25506.53
13:37:25 0 0.17 0.00 0.08 0.18 0.00 16.81 82.76
2215.63
13:37:25 1 0.08 0.00 0.20 0.00 4.08 69.49 26.14
23291.34
13:37:25 CPU %user %nice %system %iowait %irq %soft %idle
intr/s
13:38:24 all 0.14 0.00 0.12 0.12 2.02 42.89 54.71
25900.70
13:38:24 0 0.03 0.00 0.05 0.22 0.00 16.67 83.03
2246.10
13:38:24 1 0.25 0.00 0.20 0.03 4.02 69.12 26.40
23654.55
Any insights as to why there would be such a stark difference in performance
between V2.6 and V2.4?
Please advise.
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
[-- Attachment #1.2: 0x719905E5.asc --]
[-- Type: application/pgp-keys, Size: 1734 bytes --]
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 88+ messages in thread* Re: V2.4 policy router operates faster/better than V2.6
2005-01-03 20:55 V2.4 policy router operates faster/better than V2.6 Jeremy M. Guthrie
@ 2005-01-03 22:51 ` Stephen Hemminger
2005-01-03 22:56 ` Jeremy M. Guthrie
2005-01-04 15:07 ` Jeremy M. Guthrie
0 siblings, 2 replies; 88+ messages in thread
From: Stephen Hemminger @ 2005-01-03 22:51 UTC (permalink / raw)
To: jeremy.guthrie; +Cc: netdev
On Mon, 3 Jan 2005 14:55:24 -0600
"Jeremy M. Guthrie" <jeremy.guthrie@berbee.com> wrote:
> I have a dual processor box running Suse 9.1 Ent. that I changed over to the
> V2.6.10 kernel. The box has two interfaces in it, both E1000s. The box
> receives anywhere from 200mbit to 500+ mbit that it needs to route out to
> other boxes. The policy routing table is running ~ 150-200 rules. ie. data
> comes in E3(e1000), is policy routed to a destination sent out E2(e1000).
>
> Under V2.4 kernels, the system will operate just fine and drop few packets if
> any. ie. right now under V2.4, I have dropped all of three packets. Under
> 2.6, I can watch the RX drop counter increment. See below.
>
> [h-pr-msn-1 guthrie 1:48pm]~-> ifconfig eth3 ; sleep 10 ; ifconfig eth3
> eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
> inet addr:10.253.0.1 Bcast:10.255.255.255 Mask:255.255.255.0
> inet6 addr: fe80::202:b3ff:fed5:7e30/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:132919934 errors:311285 dropped:311285 overruns:247225
> frame:0
> TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:2630721320 (2508.8 Mb) TX bytes:484 (484.0 b)
> Base address:0x22a0 Memory:eff80000-effa0000
>
> eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
> inet addr:10.253.0.1 Bcast:10.255.255.255 Mask:255.255.255.0
> inet6 addr: fe80::202:b3ff:fed5:7e30/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:133847068 errors:325697 dropped:325697 overruns:258546
> frame:0
> TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:3102796062 (2959.0 Mb) TX bytes:484 (484.0 b)
> Base address:0x22a0 Memory:eff80000-effa0000
>
> If I turn off the policy routing, I instantly stop getting RX errors or
> overruns as it appears the CPU can now pay attention to the packets coming in
> and drop them(as I turned off IP forwarding as well).
>
> V2.4 Kernel mpstat data:
> command: mpstat -P ALL 60
> Linux 2.4.21-251-smp (h-pr-msn-1) 12/15/2004
>
> 01:16:24 PM CPU %user %nice %system %idle intr/s
> 01:17:19 PM all 0.16 0.00 50.12 49.72 42114.18
> 01:17:19 PM 0 0.12 0.00 55.60 44.28 42114.18
> 01:17:19 PM 1 0.20 0.00 44.65 55.15 42114.18
>
> 01:17:19 PM CPU %user %nice %system %idle intr/s
> 01:18:19 PM all 0.13 0.00 48.49 51.38 42103.08
> 01:18:19 PM 0 0.13 0.00 31.88 67.98 42103.08
> 01:18:19 PM 1 0.13 0.00 65.10 34.77 42103.08
>
> V2.6 kernel mpstat data:
> command: mpstat -P ALL 60
> Linux 2.6.5-7.111.5-smp (h-pr-msn-1) 12/15/04
>
> 13:36:25 CPU %user %nice %system %iowait %irq %soft %idle
> intr/s
> 13:37:25 all 0.13 0.00 0.15 0.09 2.03 43.14 54.45
> 25506.53
> 13:37:25 0 0.17 0.00 0.08 0.18 0.00 16.81 82.76
> 2215.63
> 13:37:25 1 0.08 0.00 0.20 0.00 4.08 69.49 26.14
> 23291.34
>
> 13:37:25 CPU %user %nice %system %iowait %irq %soft %idle
> intr/s
> 13:38:24 all 0.14 0.00 0.12 0.12 2.02 42.89 54.71
> 25900.70
> 13:38:24 0 0.03 0.00 0.05 0.22 0.00 16.67 83.03
> 2246.10
> 13:38:24 1 0.25 0.00 0.20 0.03 4.02 69.12 26.40
> 23654.55
>
> Any insights as to why there would be such a stark difference in performance
> between V2.6 and V2.4?
How many flows are going through the router? The neighbour cache
can get to be a bottleneck. Perhaps Robert "the Router Man" Olssen can
give some hints.
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-03 22:51 ` Stephen Hemminger
@ 2005-01-03 22:56 ` Jeremy M. Guthrie
2005-01-05 13:18 ` Robert Olsson
2005-01-04 15:07 ` Jeremy M. Guthrie
1 sibling, 1 reply; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-03 22:56 UTC (permalink / raw)
To: netdev; +Cc: Stephen Hemminger
[-- Attachment #1: Type: text/plain, Size: 789 bytes --]
How would I check? It should be in the hundreds of thousands.
On Monday 03 January 2005 04:51 pm, Stephen Hemminger wrote:
> On Mon, 3 Jan 2005 14:55:24 -0600
>
> "Jeremy M. Guthrie" <jeremy.guthrie@berbee.com> wrote:
> > Any insights as to why there would be such a stark difference in
> > performance between V2.6 and V2.4?
>
> How many flows are going through the router? The neighbour cache
> can get to be a bottleneck. Perhaps Robert "the Router Man" Olssen can
> give some hints.
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-03 22:56 ` Jeremy M. Guthrie
@ 2005-01-05 13:18 ` Robert Olsson
2005-01-05 15:18 ` Jeremy M. Guthrie
0 siblings, 1 reply; 88+ messages in thread
From: Robert Olsson @ 2005-01-05 13:18 UTC (permalink / raw)
To: jeremy.guthrie; +Cc: netdev, Stephen Hemminger
Jeremy M. Guthrie writes:
> How would I check? It should be in the hundreds of thousands.
Good question Stephen,..
Yes it seems like this pretty hefty load. Forwarding rate of 92k kpps
and a drop rate of 10 kpps and dst hash mostly at 50-60 kentries if
I read the stats correctly.
And 2.4 were able handle this but not 2.6.10?
Assuming things are uses and setup identically. 2.6 uses RCU for route hash
locking. Any dst cache overslow messages seen?
A couple of lines of rtstat would be very interesing from this box.
Also check that the CPU shares the RX packet load. CPU0 affinty to eth0
and CPU1 to eth1 seems to be best. It gives cache bouncing at "TX" and
slab jobs but we have accept that for now.
13:37:25 CPU %user %nice %system %iowait %irq %soft %idle
> intr/s
> 13:38:24 all 0.14 0.00 0.12 0.12 2.02 42.89 54.71
> 25900.70
> 13:38:24 0 0.03 0.00 0.05 0.22 0.00 16.67 83.03
> 2246.10
> 13:38:24 1 0.25 0.00 0.20 0.03 4.02 69.12 26.40
> 23654.55
This looks weird to me... we cannot have CPU left? Due to the imbalance?
Check /proc/net/softnet_stat,
Haven't used mpstat. %soft is that *all* softirq's or only softirq's deferred
to ksoftird only?
--ro
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-05 13:18 ` Robert Olsson
@ 2005-01-05 15:18 ` Jeremy M. Guthrie
2005-01-05 16:30 ` Robert Olsson
0 siblings, 1 reply; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-05 15:18 UTC (permalink / raw)
To: netdev; +Cc: Robert Olsson, Stephen Hemminger
[-- Attachment #1: Type: text/plain, Size: 2632 bytes --]
On Wednesday 05 January 2005 07:18 am, Robert Olsson wrote:
> Jeremy M. Guthrie writes:
> > How would I check? It should be in the hundreds of thousands.
>
> Good question Stephen,..
>
> Yes it seems like this pretty hefty load. Forwarding rate of 92k kpps
> and a drop rate of 10 kpps and dst hash mostly at 50-60 kentries if
> I read the stats correctly.
Yeah, the load will be high. I'm expecting this to be watching ~ 750 mbps by
next December. The app profiles all traffic going in and out of our data
centers.
> And 2.4 were able handle this but not 2.6.10?
Yes. It does handle it. It runs harder ie. 2.6 caps out at ~ 50% utilization
where 2.4 might run 60-75% utilized.
> Assuming things are uses and setup identically. 2.6 uses RCU for route
> hash locking. Any dst cache overslow messages seen?
No.
> A couple of lines of rtstat would be very interesing from this box.
I'm not showing the /proc/net/rt_cache_stat file. Was there a kernel option I
need to recompile with for rt_cache_stat to show up in proc?
> Also check that the CPU shares the RX packet load. CPU0 affinty to eth0
> and CPU1 to eth1 seems to be best. It gives cache bouncing at "TX" and
> slab jobs but we have accept that for now.
How would I go about doing this?
> 13:37:25 CPU %user %nice %system %iowait %irq %soft %idle
>
> > intr/s
> > 13:38:24 all 0.14 0.00 0.12 0.12 2.02 42.89 54.71
> > 25900.70
> > 13:38:24 0 0.03 0.00 0.05 0.22 0.00 16.67 83.03
> > 2246.10
> > 13:38:24 1 0.25 0.00 0.20 0.03 4.02 69.12 26.40
> > 23654.55
>
> This looks weird to me... we cannot have CPU left? Due to the imbalance?
> Check /proc/net/softnet_stat,
cat /proc/net/softnet_stat
5592c972 00000000 00001fc8 00000000 00000000 00000000 00000000 00000000
00391c3f
000f1991 00000000 00000000 00000000 00000000 00000000 00000000 00000000
001292ba
> Haven't used mpstat. %soft is that *all* softirq's or only softirq's
> deferred to ksoftird only?
"%soft"
Show the percentage of time spent by the CPU or CPUs to service
softirqs. A softirq (software interrupt) is one of up to 32
enumerated software interrupts which can run on multiple CPUs at
once.
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-05 15:18 ` Jeremy M. Guthrie
@ 2005-01-05 16:30 ` Robert Olsson
2005-01-05 17:35 ` Jeremy M. Guthrie
0 siblings, 1 reply; 88+ messages in thread
From: Robert Olsson @ 2005-01-05 16:30 UTC (permalink / raw)
To: jeremy.guthrie; +Cc: netdev, Robert Olsson, Stephen Hemminger
Jeremy M. Guthrie writes:
> Yeah, the load will be high. I'm expecting this to be watching ~ 750 mbps by
> next December. The app profiles all traffic going in and out of our data
> centers.
BW itself or pps is that not much of challange as handling of concurrent
flows.
> I'm not showing the /proc/net/rt_cache_stat file. Was there a kernel
> option I need to recompile with for rt_cache_stat to show up in proc?
No it's there without any options. Would be nice to the output from rtstat
> > Also check that the CPU shares the RX packet load. CPU0 affinty to eth0
> > and CPU1 to eth1 seems to be best. It gives cache bouncing at "TX" and
> > slab jobs but we have accept that for now.
> How would I go about doing this?
Assume you route packets between eth0 <-> eth1
Set eth0 irq to CPU0 and eth1 to CPU1 with /proc/irq/XX/smp_affinity
Disable irqbalancer etc.
> cat /proc/net/softnet_stat
total droppped tsquz Throttl FR_hit FR_succe FR_defer FR_def_o
cpu_coll
> 5592c972 00000000 00001fc8 00000000 00000000 00000000 00000000 00000000
> 00391c3f
> 000f1991 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> 001292ba
See! One line per CPU. So CPU0 is handing almost all packets.
> "%soft"
> Show the percentage of time spent by the CPU or CPUs to service
> softirqs. A softirq (software interrupt) is one of up to 32
> enumerated software interrupts which can run on multiple CPUs
Well yes. I had a more specific question. I'll look into mpstat where do
find it? Kernel pacthes?
Be also aware that packet forwarding with SMP/NUMA is very much research
today it is not that easy or not even possible to get aggregated performance
from several CPU's. in any setup. Well anyway we are beginning to see some
benefits now as we better understand the problems.
--ro
^ permalink raw reply [flat|nested] 88+ messages in thread* Re: V2.4 policy router operates faster/better than V2.6
2005-01-05 16:30 ` Robert Olsson
@ 2005-01-05 17:35 ` Jeremy M. Guthrie
2005-01-05 19:25 ` Jeremy M. Guthrie
0 siblings, 1 reply; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-05 17:35 UTC (permalink / raw)
To: netdev; +Cc: Robert Olsson, Stephen Hemminger
[-- Attachment #1: Type: text/plain, Size: 6120 bytes --]
On Wednesday 05 January 2005 10:30 am, Robert Olsson wrote:
> Jeremy M. Guthrie writes:
> > Yeah, the load will be high. I'm expecting this to be watching ~ 750
> > mbps by next December. The app profiles all traffic going in and out of
> > our data centers.
> BW itself or pps is that not much of challange as handling of concurrent
> flows.
Roger that.
> > I'm not showing the /proc/net/rt_cache_stat file. Was there a kernel
> > option I need to recompile with for rt_cache_stat to show up in proc?
>
> No it's there without any options. Would be nice to the output from rtstat
Output from rtstat:
rtstat
fopen: No such file or directory
cd /proc/net/
ls -la
total 0
dr-xr-xr-x 5 root root 0 2005-01-04 08:53 .
dr-xr-xr-x 72 root root 0 2005-01-04 08:53 ..
-r--r--r-- 1 root root 0 2005-01-05 10:32 anycast6
-r--r--r-- 1 root root 0 2005-01-05 10:32 arp
dr-xr-xr-x 2 root root 0 2005-01-05 10:32 atm
-r--r--r-- 1 root root 0 2005-01-05 10:32 dev
-r--r--r-- 1 root root 0 2005-01-05 10:32 dev_mcast
dr-xr-xr-x 2 root root 0 2005-01-05 10:32 dev_snmp6
-r--r--r-- 1 root root 0 2005-01-04 08:53 if_inet6
-r--r--r-- 1 root root 0 2005-01-05 10:32 igmp
-r--r--r-- 1 root root 0 2005-01-05 10:32 igmp6
-r--r--r-- 1 root root 0 2005-01-05 10:32 ip6_flowlabel
-r--r--r-- 1 root root 0 2005-01-05 10:32 ip_mr_cache
-r--r--r-- 1 root root 0 2005-01-05 10:32 ip_mr_vif
-r--r--r-- 1 root root 0 2005-01-05 10:32 ip_tables_matches
-r--r--r-- 1 root root 0 2005-01-05 10:32 ip_tables_names
-r--r--r-- 1 root root 0 2005-01-05 10:32 ip_tables_targets
-r--r--r-- 1 root root 0 2005-01-05 10:32 ipv6_route
-r--r--r-- 1 root root 0 2005-01-05 10:32 mcfilter
-r--r--r-- 1 root root 0 2005-01-05 10:32 mcfilter6
-r--r--r-- 1 root root 0 2005-01-05 10:32 netlink
-r--r--r-- 1 root root 0 2005-01-05 10:32 netstat
-r--r--r-- 1 root root 0 2005-01-05 10:32 psched
-r--r--r-- 1 root root 0 2005-01-05 10:32 raw
-r--r--r-- 1 root root 0 2005-01-05 10:32 raw6
-r--r--r-- 1 root root 0 2005-01-05 10:32 route
dr-xr-xr-x 6 root root 0 2005-01-05 10:32 rpc
-r--r--r-- 1 root root 0 2005-01-05 10:32 rt6_stats
-r--r--r-- 1 root root 0 2005-01-05 10:32 rt_acct
-r--r--r-- 1 root root 0 2005-01-05 10:32 rt_cache
-r--r--r-- 1 root root 0 2005-01-05 10:32 snmp
-r--r--r-- 1 root root 0 2005-01-05 10:32 snmp6
-r--r--r-- 1 root root 0 2005-01-05 10:32 sockstat
-r--r--r-- 1 root root 0 2005-01-05 10:32 sockstat6
-r--r--r-- 1 root root 0 2005-01-05 10:32 softnet_stat
dr-xr-xr-x 2 root root 0 2005-01-05 10:32 stat
-r--r--r-- 1 root root 0 2005-01-04 08:53 tcp
-r--r--r-- 1 root root 0 2005-01-05 10:32 tcp6
-r--r--r-- 1 root root 0 2005-01-05 10:32 tr_rif
-r--r--r-- 1 root root 0 2005-01-04 08:53 udp
-r--r--r-- 1 root root 0 2005-01-05 10:32 udp6
-r--r--r-- 1 root root 0 2005-01-05 10:32 unix
-r--r--r-- 1 root root 0 2005-01-05 10:32 wireless
> > > Also check that the CPU shares the RX packet load. CPU0 affinty to
> > > eth0 and CPU1 to eth1 seems to be best. It gives cache bouncing at
> > > "TX" and slab jobs but we have accept that for now.
> >
> > How would I go about doing this?
>
> Assume you route packets between eth0 <-> eth1
Yup
> Set eth0 irq to CPU0 and eth1 to CPU1 with /proc/irq/XX/smp_affinity
Done
> Disable irqbalancer etc.
Done
I'll let you know what I see for stats once I get some collected.
> > cat /proc/net/softnet_stat
>
> total droppped tsquz Throttl FR_hit FR_succe FR_defer FR_def_o
> cpu_coll
>
> > 5592c972 00000000 00001fc8 00000000 00000000 00000000 00000000 00000000
> > 00391c3f
> > 000f1991 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> > 001292ba
>
> See! One line per CPU. So CPU0 is handing almost all packets.
cat /proc/interrupts
CPU0 CPU1
0: 674862 93484967 IO-APIC-edge timer
1: 564 9 IO-APIC-edge i8042
7: 0 0 IO-APIC-level ohci_hcd
8: 1 1 IO-APIC-edge rtc
12: 268 62 IO-APIC-edge i8042
14: 2 0 IO-APIC-edge ide0
18: 2105131410 9140835 IO-APIC-level eth3
20: 1077 248075156 IO-APIC-level eth2
27: 118224 1 IO-APIC-level eth0
28: 36298 49 IO-APIC-level aic7xxx
30: 0 0 IO-APIC-level acpi
NMI: 0 0
LOC: 94168097 94168094
ERR: 0
MIS: 0
> > "%soft"
> > Show the percentage of time spent by the CPU or CPUs to service
> > softirqs. A softirq (software interrupt) is one of up to 32
> > enumerated software interrupts which can run on multiple CPUs
>
> Well yes. I had a more specific question. I'll look into mpstat where do
> find it? Kernel pacthes?
Sorry about that. New to the list. I'm not suggesting anything. I
appreciate the help!
Suse listed mpstat as part of sysstat 5.1.2. I'm running stock 2.6.10.
> Be also aware that packet forwarding with SMP/NUMA is very much research
> today it is not that easy or not even possible to get aggregated
> performance from several CPU's. in any setup. Well anyway we are beginning
> to see some benefits now as we better understand the problems.
Understood. As long as I know this, I can articulate this to my uppers for
bigger hardware. My current system is a dual P-III 700mhz. May be time for
an upgrade. However, I figure this may also offer a good environment to help
provide you guys with a taxed system running an load of flows. Nothing like
finding fun stuff while a system is ready to fall over.
Would a single hyper threaded CPU help this or should I default to a normal
dual-cpu system?
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 88+ messages in thread* Re: V2.4 policy router operates faster/better than V2.6
2005-01-05 17:35 ` Jeremy M. Guthrie
@ 2005-01-05 19:25 ` Jeremy M. Guthrie
2005-01-05 20:22 ` Robert Olsson
0 siblings, 1 reply; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-05 19:25 UTC (permalink / raw)
To: netdev; +Cc: Robert Olsson, Stephen Hemminger
[-- Attachment #1: Type: text/plain, Size: 927 bytes --]
After smp_affinity adjustments and turning off IRQ balancing.
eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
inet addr:10.253.0.1 Bcast:10.255.255.255 Mask:255.255.255.0
inet6 addr: fe80::202:b3ff:fed5:7e30/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:537271635 errors:2377048 dropped:2377048 overruns:1849169
frame:0
TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1804727106 (1721.1 Mb) TX bytes:398 (398.0 b)
Base address:0x22a0 Memory:eff80000-effa0000
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 88+ messages in thread* Re: V2.4 policy router operates faster/better than V2.6
2005-01-05 19:25 ` Jeremy M. Guthrie
@ 2005-01-05 20:22 ` Robert Olsson
2005-01-05 20:52 ` Jeremy M. Guthrie
2005-01-06 15:26 ` Jeremy M. Guthrie
0 siblings, 2 replies; 88+ messages in thread
From: Robert Olsson @ 2005-01-05 20:22 UTC (permalink / raw)
To: jeremy.guthrie; +Cc: netdev, Robert Olsson, Stephen Hemminger
Jeremy M. Guthrie writes:
> After smp_affinity adjustments and turning off IRQ balancing.
>
> eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
> RX packets:537271635 errors:2377048 dropped:2377048 overruns:1849169
Worse?
Yes. I'll remember Hararld moved rt_cache_stat /proc/net/stats/ and wrote a
new utility for it. Should be in iproute2 package. Stephen knows the
details. Otherwise change path in rtstat. You need this to relate and
verify the load.
Check throughtput and how packet load is used with 2.4. /proc/net/softnet_stat
and rtstat. Also meditate how the comparison can be fair wrt taffic patterns.
Do smame with 2.6.
Prepare for a oprofile. I'm lazy and compile the stuff I use into kernel if
you can get the same result w. nodules it's ok.
--ro
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-05 20:22 ` Robert Olsson
@ 2005-01-05 20:52 ` Jeremy M. Guthrie
2005-01-06 15:26 ` Jeremy M. Guthrie
1 sibling, 0 replies; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-05 20:52 UTC (permalink / raw)
To: netdev; +Cc: Robert Olsson, Stephen Hemminger
[-- Attachment #1: Type: text/plain, Size: 1881 bytes --]
On Wednesday 05 January 2005 02:22 pm, Robert Olsson wrote:
> Jeremy M. Guthrie writes:
> > After smp_affinity adjustments and turning off IRQ balancing.
> >
> > eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
> >
> > RX packets:537271635 errors:2377048 dropped:2377048
> > overruns:1849169
>
> Worse?
>
> Yes. I'll remember Hararld moved rt_cache_stat /proc/net/stats/ and wrote
> a new utility for it. Should be in iproute2 package. Stephen knows the
> details. Otherwise change path in rtstat. You need this to relate and
> verify the load.
Hm...
ls -la /proc/net/stat/
total 0
dr-xr-xr-x 2 root root 0 Jan 5 14:47 .
dr-xr-xr-x 5 root root 0 Jan 5 11:50 ..
-r--r--r-- 1 root root 0 Jan 5 14:47 arp_cache
-r--r--r-- 1 root root 0 Jan 5 14:47 clip_arp_cache
-r--r--r-- 1 root root 0 Jan 5 14:47 ndisc_cache
-r--r--r-- 1 root root 0 Jan 5 14:47 rt_cache
> Check throughtput and how packet load is used with 2.4.
> /proc/net/softnet_stat and rtstat.
Will do.
> Also meditate how the comparison can be
> fair wrt taffic patterns. Do smame with 2.6.
Not sure I follow. I can see higher pps/byte through puts on switch port
counters when I run with 2.4 vs 2.6. I'll double check though. I am the
'dev' state so I can swap between V2.4 and V2.6 as necessary to compare
during roughly equivalent times. The only delay is the time between reboots.
> Prepare for a oprofile. I'm lazy and compile the stuff I use into kernel
> if you can get the same result w. nodules it's ok.
Okay.
>
>
> --ro
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-05 20:22 ` Robert Olsson
2005-01-05 20:52 ` Jeremy M. Guthrie
@ 2005-01-06 15:26 ` Jeremy M. Guthrie
2005-01-06 18:15 ` Robert Olsson
1 sibling, 1 reply; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-06 15:26 UTC (permalink / raw)
To: netdev; +Cc: Robert Olsson, Stephen Hemminger
[-- Attachment #1: Type: text/plain, Size: 3954 bytes --]
On Wednesday 05 January 2005 02:22 pm, Robert Olsson wrote:
> Jeremy M. Guthrie writes:
> > After smp_affinity adjustments and turning off IRQ balancing.
> >
> > eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
> >
> > RX packets:537271635 errors:2377048 dropped:2377048
> > overruns:1849169
>
> Worse?
>
> Yes. I'll remember Hararld moved rt_cache_stat /proc/net/stats/ and wrote
> a new utility for it. Should be in iproute2 package. Stephen knows the
> details. Otherwise change path in rtstat. You need this to relate and
> verify the load.
I still don't see the rt_cache_stat file even under 2.4.28 stock. See below.
> Check throughtput and how packet load is used with 2.4.
> /proc/net/softnet_stat and rtstat. Also meditate how the comparison can be
> fair wrt taffic patterns. Do smame with 2.6.
wc -l /proc/net/rt_cache
Segmentation fault
cat /proc/net/rt_cache > /tmp/rt_cache ; wc -l /tmp/rt_cache
cat: write error: Bad address
57664 /tmp/rt_cache
ls -la /proc/net/
total 0
dr-xr-xr-x 6 root root 0 Jan 6 09:22 .
dr-xr-xr-x 55 root root 0 Jan 5 15:23 ..
-r--r--r-- 1 root root 0 Jan 6 09:22 arp
dr-xr-xr-x 2 root root 0 Jan 6 09:22 atm
-r--r--r-- 1 root root 0 Jan 6 09:22 dev
-r--r--r-- 1 root root 0 Jan 6 09:22 dev_mcast
dr-xr-xr-x 2 root root 0 Jan 6 09:22 drivers
-r--r--r-- 1 root root 0 Jan 6 09:22 igmp
-r--r--r-- 1 root root 0 Jan 6 09:22 ip_mr_cache
-r--r--r-- 1 root root 0 Jan 6 09:22 ip_mr_vif
-r--r--r-- 1 root root 0 Jan 6 09:22 ip_queue
-r--r--r-- 1 root root 0 Jan 6 09:22 ip_tables_matches
-r--r--r-- 1 root root 0 Jan 6 09:22 ip_tables_names
-r--r--r-- 1 root root 0 Jan 6 09:22 ip_tables_targets
-r--r--r-- 1 root root 0 Jan 6 09:22 mcfilter
-r--r--r-- 1 root root 0 Jan 6 09:22 netlink
-r--r--r-- 1 root root 0 Jan 6 09:22 netstat
-r--r--r-- 1 root root 0 Jan 6 09:22 pnp
-r--r--r-- 1 root root 0 Jan 6 09:22 psched
-r--r--r-- 1 root root 0 Jan 6 09:22 raw
-r--r--r-- 1 root root 0 Jan 6 09:22 route
dr-xr-xr-x 2 root root 0 Jan 6 09:22 rpc
-r--r--r-- 1 root root 0 Jan 6 09:22 rt_acct
-r--r--r-- 1 root root 0 Jan 6 09:22 rt_cache
-r--r--r-- 1 root root 0 Jan 6 09:22 snmp
-r--r--r-- 1 root root 0 Jan 6 09:22 sockstat
-r--r--r-- 1 root root 0 Jan 6 09:22 softnet_stat
dr-xr-xr-x 2 root root 0 Jan 6 09:22 stat
-r--r--r-- 1 root root 0 Jan 6 09:22 tcp
-r--r--r-- 1 root root 0 Jan 6 09:22 udp
-r--r--r-- 1 root root 0 Jan 6 09:22 unix
-r--r--r-- 1 root root 0 Jan 6 09:22 wireless
ls -la /proc/net/stat/
total 0
dr-xr-xr-x 2 root root 0 Jan 6 09:22 .
dr-xr-xr-x 6 root root 0 Jan 6 09:22 ..
-r--r--r-- 1 root root 0 Jan 6 09:22 arp_cache
-r--r--r-- 1 root root 0 Jan 6 09:22 clip_arp_cache
-r--r--r-- 1 root root 0 Jan 6 09:22 rt_cache
cat /proc/net/softnet_stat
96deb140 0032844e 00012bbb 00000fd8 00000000 00000000 00000000 00000000
00000dd4
00013dda 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0000041f
cat /proc/interrupts
CPU0 CPU1
0: 821852 5664906 IO-APIC-edge timer
1: 484 1247 IO-APIC-edge keyboard
8: 0 2 IO-APIC-edge rtc
14: 5 16 IO-APIC-edge ide0
18: 1928608529 1374 IO-APIC-level eth3
20: 1 226113098 IO-APIC-level eth2
27: 7950 34312 IO-APIC-level eth0
28: 2718 8097 IO-APIC-level aic7xxx
30: 0 0 IO-APIC-level acpi
NMI: 0 0
LOC: 6487240 6487239
ERR: 0
MIS: 0
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 88+ messages in thread* Re: V2.4 policy router operates faster/better than V2.6
2005-01-06 15:26 ` Jeremy M. Guthrie
@ 2005-01-06 18:15 ` Robert Olsson
2005-01-06 19:35 ` Jeremy M. Guthrie
0 siblings, 1 reply; 88+ messages in thread
From: Robert Olsson @ 2005-01-06 18:15 UTC (permalink / raw)
To: jeremy.guthrie; +Cc: netdev, Robert Olsson, Stephen Hemminger
Jeremy M. Guthrie writes:
> I still don't see the rt_cache_stat file even under 2.4.28 stock. See below.
> ls -la /proc/net/stat/
> total 0
> -r--r--r-- 1 root root 0 Jan 6 09:22 rt_cache
rtstat is replaced by lnstat which is in iproute2 package:
http://developer.osdl.org/dev/iproute2/download/
(Old version ftp://robur.slu.se/pub/Linux/net-development/rt_cache_stat/rtstat.c)
> cat /proc/net/softnet_stat
> 96deb140 0032844e 00012bbb 00000fd8 00000000 00000000 00000000 00000000
> 00000dd4
> 00013dda 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> 0000041f
You only use CPU0 for packet processing. Also it seems you use the non-NAPI
version of e1000.
> cat /proc/interrupts
> CPU0 CPU1
> 0: 821852 5664906 IO-APIC-edge timer
> 1: 484 1247 IO-APIC-edge keyboard
> 8: 0 2 IO-APIC-edge rtc
> 14: 5 16 IO-APIC-edge ide0
> 18: 1928608529 1374 IO-APIC-level eth3
> 20: 1 226113098 IO-APIC-level eth2
> 27: 7950 34312 IO-APIC-level eth0
Traffic is flowing from eth3->eth2 Why all the interrupts on eth2/CPU1?
Is traffic most unidirectional?
--ro
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-06 18:15 ` Robert Olsson
@ 2005-01-06 19:35 ` Jeremy M. Guthrie
2005-01-06 20:29 ` Robert Olsson
0 siblings, 1 reply; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-06 19:35 UTC (permalink / raw)
To: netdev; +Cc: Robert Olsson, Stephen Hemminger
[-- Attachment #1.1: Type: text/plain, Size: 1938 bytes --]
On Thursday 06 January 2005 12:15 pm, Robert Olsson wrote:
> Jeremy M. Guthrie writes:
> > I still don't see the rt_cache_stat file even under 2.4.28 stock. See
> > below.
> >
> > ls -la /proc/net/stat/
> > total 0
> > -r--r--r-- 1 root root 0 Jan 6 09:22 rt_cache
>
> rtstat is replaced by lnstat which is in iproute2 package:
> http://developer.osdl.org/dev/iproute2/download/
>
> (Old version
> ftp://robur.slu.se/pub/Linux/net-development/rt_cache_stat/rtstat.c)
Please see the attachment.
> > cat /proc/net/softnet_stat
> > 96deb140 0032844e 00012bbb 00000fd8 00000000 00000000 00000000 00000000
> > 00000dd4
> > 00013dda 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> > 0000041f
>
> You only use CPU0 for packet processing. Also it seems you use the
> non-NAPI version of e1000.
The E1000 driver is the stock driver in 2.4.28.
>
> > cat /proc/interrupts
> > CPU0 CPU1
> > 0: 821852 5664906 IO-APIC-edge timer
> > 1: 484 1247 IO-APIC-edge keyboard
> > 8: 0 2 IO-APIC-edge rtc
> > 14: 5 16 IO-APIC-edge ide0
> > 18: 1928608529 1374 IO-APIC-level eth3
> > 20: 1 226113098 IO-APIC-level eth2
> > 27: 7950 34312 IO-APIC-level eth0
>
> Traffic is flowing from eth3->eth2 Why all the interrupts on eth2/CPU1?
> Is traffic most unidirectional?
eth2 is TX only. We don't receive anything on it. This system should only
ever RX on eth3 and TX on eth2 as part of its function. Eth0 is the
management interface on the host.
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
[-- Attachment #1.2: results-2.4.28.txt --]
[-- Type: text/plain, Size: 26640 bytes --]
arp_cach|arp_cach|arp_cach|arp_cach|arp_cach|arp_cach|arp_cach|arp_cach|arp_cach|arp_cach|arp_cach|arp_cach|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|clip_arp|clip_arp|clip_arp|clip_arp|clip_arp|clip_arp|clip_arp|clip_arp|clip_arp|clip_arp|clip_arp|clip_arp|
entries| allocs|destroys|hash_gro| lookups| hits|res_fail|rcv_prob|rcv_prob|periodic|forced_g|forced_g| entries| in_hit|in_slow_|in_slow_|in_no_ro| in_brd|in_marti|in_marti| out_hit|out_slow|out_slow|gc_total|gc_ignor|gc_goal_|gc_dst_o|in_hlist|out_hlis| entries| allocs|destroys|hash_gro| lookups| hits|res_fail|rcv_prob|rcv_prob|periodic|forced_g|forced_g|
| | | ws| | | ed|es_mcast|es_ucast|_gc_runs| c_runs|c_goal_m| | | tot| mc| ute| | an_dst| an_src| | _tot| _mc| | ed| miss| verflow| _search|t_search| | | | ws| | | ed|es_mcast|es_ucast|_gc_runs| c_runs|c_goal_m|
| | | | | | | | | | | iss| | | | | | | | | | | | | | | | | | | | | | | | | | | | | iss|
22| 9| 425| 1| 17202| 14599| 0| 0| 0| 75743| 0| 0| 61129| 6329| 20428| 0| 0| 1539| 0| 0| 8| 28| 0| 1677| 1675| 0| 0| 143038| 145| 0| 0| 0| 0| 0| 0| 0| 0| 0| 4649| 0| 0|
22| 0| 0| 0| 1| 1| 0| 0| 0| 2| 0| 0| 62833| 0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 10| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
22| 0| 0| 0| 1| 1| 0| 0| 0| 1| 0| 0| 64306| 0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 7| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
22| 0| 0| 0| 0| 0| 0| 0| 0| 2| 0| 0| 65602| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
22| 0| 0| 0| 0| 0| 0| 0| 0| 2| 0| 0| 59707| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 2| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
23| 0| 0| 0| 0| 0| 0| 0| 0| 2| 0| 0| 60684| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
23| 0| 0| 0| 0| 0| 0| 0| 0| 2| 0| 0| 62249| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
23| 0| 0| 0| 0| 0| 0| 0| 0| 2| 0| 0| 62249| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
23| 0| 0| 0| 0| 0| 0| 0| 0| 4| 0| 0| 65000| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 4| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
22| 0| 0| 0| 0| 0| 0| 0| 0| 2| 0| 0| 58800| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 3| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
22| 0| 0| 0| 0| 0| 0| 0| 0| 2| 0| 0| 58800| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 3| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
22| 0| 0| 0| 0| 0| 0| 0| 0| 4| 0| 0| 62135| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
22| 0| 0| 0| 0| 0| 0| 0| 0| 2| 0| 0| 63677| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 3| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
22| 0| 0| 0| 0| 0| 0| 0| 0| 2| 0| 0| 63677| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 3| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
23| 0| 0| 0| 0| 0| 0| 0| 0| 4| 0| 0| 59013| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 2| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
23| 0| 0| 0| 0| 0| 0| 0| 0| 4| 0| 0| 59013| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 2| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
24| 0| 0| 0| 0| 0| 0| 0| 0| 3| 0| 0| 62462| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
24| 0| 0| 0| 0| 0| 0| 0| 0| 3| 0| 0| 62462| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
24| 0| 0| 0| 1| 0| 0| 0| 0| 4| 0| 0| 65225| 0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 7| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
24| 0| 0| 0| 1| 0| 0| 0| 0| 4| 0| 0| 65225| 0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 7| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
arp_cach|arp_cach|arp_cach|arp_cach|arp_cach|arp_cach|arp_cach|arp_cach|arp_cach|arp_cach|arp_cach|arp_cach|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|clip_arp|clip_arp|clip_arp|clip_arp|clip_arp|clip_arp|clip_arp|clip_arp|clip_arp|clip_arp|clip_arp|clip_arp|
entries| allocs|destroys|hash_gro| lookups| hits|res_fail|rcv_prob|rcv_prob|periodic|forced_g|forced_g| entries| in_hit|in_slow_|in_slow_|in_no_ro| in_brd|in_marti|in_marti| out_hit|out_slow|out_slow|gc_total|gc_ignor|gc_goal_|gc_dst_o|in_hlist|out_hlis| entries| allocs|destroys|hash_gro| lookups| hits|res_fail|rcv_prob|rcv_prob|periodic|forced_g|forced_g|
| | | ws| | | ed|es_mcast|es_ucast|_gc_runs| c_runs|c_goal_m| | | tot| mc| ute| | an_dst| an_src| | _tot| _mc| | ed| miss| verflow| _search|t_search| | | | ws| | | ed|es_mcast|es_ucast|_gc_runs| c_runs|c_goal_m|
| | | | | | | | | | | iss| | | | | | | | | | | | | | | | | | | | | | | | | | | | | iss|
24| 0| 0| 0| 0| 0| 0| 0| 0| 3| 0| 0| 60904| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 4| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
24| 0| 0| 0| 0| 0| 0| 0| 0| 3| 0| 0| 60904| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 4| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
23| 0| 0| 0| 0| 0| 0| 0| 0| 4| 0| 0| 63890| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 3| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
23| 0| 0| 0| 0| 0| 0| 0| 0| 4| 0| 0| 63890| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 3| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
23| 0| 0| 0| 0| 0| 0| 0| 0| 3| 0| 0| 59738| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 3| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
23| 0| 0| 0| 0| 0| 0| 0| 0| 3| 0| 0| 59738| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 3| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
23| 0| 0| 0| 1| 1| 0| 0| 0| 4| 0| 0| 63346| 0| 2| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 13| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
23| 0| 0| 0| 1| 1| 0| 0| 0| 4| 0| 0| 63346| 0| 2| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 13| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
23| 0| 0| 0| 1| 0| 0| 0| 0| 4| 0| 0| 58798| 0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 13| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
23| 0| 0| 0| 1| 0| 0| 0| 0| 4| 0| 0| 58798| 0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 13| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
24| 0| 0| 0| 1| 1| 0| 0| 0| 4| 0| 0| 62518| 0| 2| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 13| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
24| 0| 0| 0| 1| 1| 0| 0| 0| 4| 0| 0| 62518| 0| 2| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 13| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
22| 0| 0| 0| 2| 2| 0| 0| 0| 3| 0| 0| 65364| 0| 2| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 12| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
22| 0| 0| 0| 2| 2| 0| 0| 0| 3| 0| 0| 65364| 0| 2| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 12| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
23| 0| 0| 0| 2| 2| 0| 0| 0| 4| 0| 0| 61111| 0| 2| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 11| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
23| 0| 0| 0| 2| 2| 0| 0| 0| 4| 0| 0| 61111| 0| 2| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 11| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
23| 0| 0| 0| 1| 1| 0| 0| 0| 4| 0| 0| 63152| 0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 6| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
23| 0| 0| 0| 1| 1| 0| 0| 0| 4| 0| 0| 63152| 0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 6| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
23| 0| 0| 0| 1| 1| 0| 0| 0| 4| 0| 0| 58223| 0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 7| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
23| 0| 0| 0| 1| 1| 0| 0| 0| 4| 0| 0| 58223| 0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 7| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
arp_cach|arp_cach|arp_cach|arp_cach|arp_cach|arp_cach|arp_cach|arp_cach|arp_cach|arp_cach|arp_cach|arp_cach|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|clip_arp|clip_arp|clip_arp|clip_arp|clip_arp|clip_arp|clip_arp|clip_arp|clip_arp|clip_arp|clip_arp|clip_arp|
entries| allocs|destroys|hash_gro| lookups| hits|res_fail|rcv_prob|rcv_prob|periodic|forced_g|forced_g| entries| in_hit|in_slow_|in_slow_|in_no_ro| in_brd|in_marti|in_marti| out_hit|out_slow|out_slow|gc_total|gc_ignor|gc_goal_|gc_dst_o|in_hlist|out_hlis| entries| allocs|destroys|hash_gro| lookups| hits|res_fail|rcv_prob|rcv_prob|periodic|forced_g|forced_g|
| | | ws| | | ed|es_mcast|es_ucast|_gc_runs| c_runs|c_goal_m| | | tot| mc| ute| | an_dst| an_src| | _tot| _mc| | ed| miss| verflow| _search|t_search| | | | ws| | | ed|es_mcast|es_ucast|_gc_runs| c_runs|c_goal_m|
| | | | | | | | | | | iss| | | | | | | | | | | | | | | | | | | | | | | | | | | | | iss|
24| 0| 0| 0| 2| 2| 0| 0| 0| 4| 0| 0| 62044| 0| 2| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 15| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
24| 0| 0| 0| 2| 2| 0| 0| 0| 4| 0| 0| 62044| 0| 2| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 15| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
24| 0| 0| 0| 0| 0| 0| 0| 0| 4| 0| 0| 64932| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
24| 0| 0| 0| 0| 0| 0| 0| 0| 4| 0| 0| 64932| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
25| 0| 0| 0| 2| 2| 0| 0| 0| 4| 0| 0| 60712| 0| 2| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 14| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
23| 0| 1| 0| 1| 1| 0| 0| 0| 2| 0| 0| 62304| 0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 10| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
23| 0| 0| 0| 0| 0| 0| 0| 0| 2| 0| 0| 63758| 1| 1| 0| 0| 1| 0| 0| 0| 0| 0| 1| 1| 0| 0| 8| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
23| 0| 0| 0| 0| 0| 0| 0| 0| 1| 0| 0| 64967| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
23| 0| 0| 0| 1| 1| 0| 0| 0| 2| 0| 0| 58931| 0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 7| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
23| 0| 0| 0| 0| 0| 0| 0| 0| 2| 0| 0| 60836| 0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 6| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
23| 0| 0| 0| 1| 1| 0| 0| 0| 2| 0| 0| 62574| 1| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 11| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
23| 0| 0| 0| 0| 0| 0| 0| 0| 2| 0| 0| 64121| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
23| 0| 0| 0| 1| 1| 0| 0| 0| 1| 0| 0| 65615| 0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 10| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
23| 0| 0| 0| 1| 1| 0| 0| 0| 2| 0| 0| 60057| 1| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 11| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
23| 0| 0| 0| 1| 1| 0| 0| 0| 1| 0| 0| 62008| 0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 7| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
22| 0| 0| 0| 1| 1| 0| 0| 0| 1| 0| 0| 63651| 0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 7| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
22| 0| 0| 0| 1| 1| 0| 0| 0| 1| 0| 0| 63651| 0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 7| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
22| 0| 0| 0| 1| 1| 0| 0| 0| 1| 0| 0| 63651| 0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 7| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
25| 0| 0| 0| 2| 2| 0| 0| 0| 6| 0| 0| 61300| 0| 2| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 12| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
24| 0| 0| 0| 0| 0| 0| 0| 0| 2| 0| 0| 63087| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 4| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 88+ messages in thread* Re: V2.4 policy router operates faster/better than V2.6
2005-01-06 19:35 ` Jeremy M. Guthrie
@ 2005-01-06 20:29 ` Robert Olsson
2005-01-06 20:54 ` Jeremy M. Guthrie
` (2 more replies)
0 siblings, 3 replies; 88+ messages in thread
From: Robert Olsson @ 2005-01-06 20:29 UTC (permalink / raw)
To: jeremy.guthrie; +Cc: netdev, Robert Olsson, Stephen Hemminger
Jeremy M. Guthrie writes:
> > You only use CPU0 for packet processing. Also it seems you use the
> > non-NAPI version of e1000.
> The E1000 driver is the stock driver in 2.4.28.
There is a kernel config option. Use Rx Polling (NAPI)
> eth2 is TX only. We don't receive anything on it. This system should only
> ever RX on eth3 and TX on eth2 as part of its function.
Ok! So unidirectional traffic and CPU0 processes all skb's and passes them
over to CPU1 which touches and frees them. You could try some other
affinity setting later on but there are so many options depending what
we want do. If we just want to compare 2.4/2.6 we can start out simple
with setting affinity so only one CPU. Which is almost what you got.
We reach all the complicated problems immediately... Now we pass skb's
between the CPU's which release cache bouncing and makes slab rebalance it's
pools. So using just one CPU is probably better... If we had incoming pkts
on eth2 eventually we could have had any use for CPU1.
Install NAPI-driver, set affinity so eth2/eth3 uses same CPU to start with.
The stats info you sent was almost impossible to read. See if you can get
only the rtstats.
--ro
^ permalink raw reply [flat|nested] 88+ messages in thread* Re: V2.4 policy router operates faster/better than V2.6
2005-01-06 20:29 ` Robert Olsson
@ 2005-01-06 20:54 ` Jeremy M. Guthrie
2005-01-06 20:55 ` Jeremy M. Guthrie
2005-01-06 21:19 ` Jeremy M. Guthrie
2 siblings, 0 replies; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-06 20:54 UTC (permalink / raw)
To: netdev; +Cc: Robert Olsson, Stephen Hemminger
[-- Attachment #1: Type: text/plain, Size: 1969 bytes --]
On Thursday 06 January 2005 02:29 pm, Robert Olsson wrote:
> Jeremy M. Guthrie writes:
> > > You only use CPU0 for packet processing. Also it seems you use the
> > > non-NAPI version of e1000.
> >
> > The E1000 driver is the stock driver in 2.4.28.
> There is a kernel config option. Use Rx Polling (NAPI)
I'm recompiling the module and will reload the module.
> > eth2 is TX only. We don't receive anything on it. This system should
> > only ever RX on eth3 and TX on eth2 as part of its function.
>
> Ok! So unidirectional traffic and CPU0 processes all skb's and passes them
> over to CPU1 which touches and frees them. You could try some other
> affinity setting later on but there are so many options depending what
> we want do. If we just want to compare 2.4/2.6 we can start out simple
> with setting affinity so only one CPU. Which is almost what you got.
> We reach all the complicated problems immediately... Now we pass skb's
> between the CPU's which release cache bouncing and makes slab rebalance
> it's pools. So using just one CPU is probably better... If we had incoming
> pkts on eth2 eventually we could have had any use for CPU1.
>
> Install NAPI-driver, set affinity so eth2/eth3 uses same CPU to start
> with.
I installed the NAPI driver and now I drop packets under 2.4.28. I tried
setting eth2/3 onto the same CPU. If both eth2/3 are on CPU0, I drop a lot
of packets. If CPU0-eth2 & CPU1-eth3, I still drop packets, just not as
fast.
I am going to try V2.6 w/o NAPI to see how much that helps.
> The stats info you sent was almost impossible to read. See if you can get
> only the rtstats.
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-06 20:29 ` Robert Olsson
2005-01-06 20:54 ` Jeremy M. Guthrie
@ 2005-01-06 20:55 ` Jeremy M. Guthrie
2005-01-06 21:19 ` Jeremy M. Guthrie
2 siblings, 0 replies; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-06 20:55 UTC (permalink / raw)
To: netdev; +Cc: Robert Olsson, Stephen Hemminger
[-- Attachment #1.1: Type: text/plain, Size: 1664 bytes --]
Forgot the attachment.
On Thursday 06 January 2005 02:29 pm, Robert Olsson wrote:
> Jeremy M. Guthrie writes:
> > > You only use CPU0 for packet processing. Also it seems you use the
> > > non-NAPI version of e1000.
> >
> > The E1000 driver is the stock driver in 2.4.28.
>
> There is a kernel config option. Use Rx Polling (NAPI)
>
> > eth2 is TX only. We don't receive anything on it. This system should
> > only ever RX on eth3 and TX on eth2 as part of its function.
>
> Ok! So unidirectional traffic and CPU0 processes all skb's and passes them
> over to CPU1 which touches and frees them. You could try some other
> affinity setting later on but there are so many options depending what
> we want do. If we just want to compare 2.4/2.6 we can start out simple
> with setting affinity so only one CPU. Which is almost what you got.
> We reach all the complicated problems immediately... Now we pass skb's
> between the CPU's which release cache bouncing and makes slab rebalance
> it's pools. So using just one CPU is probably better... If we had incoming
> pkts on eth2 eventually we could have had any use for CPU1.
>
> Install NAPI-driver, set affinity so eth2/eth3 uses same CPU to start
> with.
>
> The stats info you sent was almost impossible to read. See if you can get
> only the rtstats.
>
> --ro
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
[-- Attachment #1.2: results-2.4.28.txt --]
[-- Type: text/plain, Size: 10005 bytes --]
rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
in_hit|in_slow_|in_slow_|in_no_ro| in_brd|in_marti|in_marti| out_hit|out_slow|out_slow|gc_total|gc_ignor|gc_goal_|gc_dst_o|in_hlist|out_hlis|
| tot| mc| ute| | an_dst| an_src| | _tot| _mc| | ed| miss| verflow| _search|t_search|
9092| 21888| 0| 0| 1656| 0| 0| 9| 42| 0| 1836| 1833| 0| 0| 158622| 206|
0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 4| 0|
0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 1| 1| 0| 0| 9| 0|
0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 1| 1| 0| 0| 9| 0|
0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 2| 0|
0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 3| 0|
0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 3| 0|
0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 3| 0|
0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 14| 0|
0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 14| 0|
0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
in_hit|in_slow_|in_slow_|in_no_ro| in_brd|in_marti|in_marti| out_hit|out_slow|out_slow|gc_total|gc_ignor|gc_goal_|gc_dst_o|in_hlist|out_hlis|
| tot| mc| ute| | an_dst| an_src| | _tot| _mc| | ed| miss| verflow| _search|t_search|
0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 9| 0|
0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 9| 0|
0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 7| 0|
0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 7| 0|
0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 4| 0|
0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 4| 0|
0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 9| 0|
0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 9| 0|
0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 9| 0|
0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 9| 0|
0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 9| 0|
0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 9| 0|
0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 9| 0|
0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 4| 0|
0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 4| 0|
0| 2| 0| 0| 0| 0| 0| 0| 0| 0| 1| 1| 0| 0| 14| 0|
0| 2| 0| 0| 0| 0| 0| 0| 0| 0| 1| 1| 0| 0| 14| 0|
0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 7| 0|
0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 3| 0|
0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 3| 0|
rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
in_hit|in_slow_|in_slow_|in_no_ro| in_brd|in_marti|in_marti| out_hit|out_slow|out_slow|gc_total|gc_ignor|gc_goal_|gc_dst_o|in_hlist|out_hlis|
| tot| mc| ute| | an_dst| an_src| | _tot| _mc| | ed| miss| verflow| _search|t_search|
1| 2| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 16| 0|
1| 2| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 16| 0|
1| 2| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 16| 0|
0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 8| 0|
0| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 8| 0|
2| 2| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 16| 0|
1| 1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 8| 0|
1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 6| 0|
1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 6| 0|
1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 6| 0|
2| 2| 0| 0| 1| 0| 0| 0| 0| 0| 1| 1| 0| 0| 24| 0|
1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 2| 0|
0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 4| 0|
0| 2| 0| 0| 1| 0| 0| 0| 0| 0| 1| 1| 0| 0| 16| 0|
0| 2| 0| 0| 1| 0| 0| 0| 0| 0| 1| 1| 0| 0| 16| 0|
2| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 4| 0|
2| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 4| 0|
0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 88+ messages in thread* Re: V2.4 policy router operates faster/better than V2.6
2005-01-06 20:29 ` Robert Olsson
2005-01-06 20:54 ` Jeremy M. Guthrie
2005-01-06 20:55 ` Jeremy M. Guthrie
@ 2005-01-06 21:19 ` Jeremy M. Guthrie
2005-01-06 21:36 ` Robert Olsson
2 siblings, 1 reply; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-06 21:19 UTC (permalink / raw)
To: netdev; +Cc: Robert Olsson, Stephen Hemminger
[-- Attachment #1: Type: text/plain, Size: 2005 bytes --]
Pulling Rx Polling has kept V2.6 from dropping packets. argh?!?!?! I should
have caught that sooner. 8( Well, I have a window of opportunity here. If
you want I can still provide a broken environment but that seems pointless
while NAPI is not acting right... what is evident is that this will run
better till we start to bump up against the top of CPU0.
On Thursday 06 January 2005 02:29 pm, Robert Olsson wrote:
> Jeremy M. Guthrie writes:
> > > You only use CPU0 for packet processing. Also it seems you use the
> > > non-NAPI version of e1000.
> >
> > The E1000 driver is the stock driver in 2.4.28.
>
> There is a kernel config option. Use Rx Polling (NAPI)
>
> > eth2 is TX only. We don't receive anything on it. This system should
> > only ever RX on eth3 and TX on eth2 as part of its function.
>
> Ok! So unidirectional traffic and CPU0 processes all skb's and passes them
> over to CPU1 which touches and frees them. You could try some other
> affinity setting later on but there are so many options depending what
> we want do. If we just want to compare 2.4/2.6 we can start out simple
> with setting affinity so only one CPU. Which is almost what you got.
> We reach all the complicated problems immediately... Now we pass skb's
> between the CPU's which release cache bouncing and makes slab rebalance
> it's pools. So using just one CPU is probably better... If we had incoming
> pkts on eth2 eventually we could have had any use for CPU1.
>
> Install NAPI-driver, set affinity so eth2/eth3 uses same CPU to start
> with.
>
> The stats info you sent was almost impossible to read. See if you can get
> only the rtstats.
>
> --ro
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-06 21:19 ` Jeremy M. Guthrie
@ 2005-01-06 21:36 ` Robert Olsson
2005-01-06 21:46 ` Jeremy M. Guthrie
0 siblings, 1 reply; 88+ messages in thread
From: Robert Olsson @ 2005-01-06 21:36 UTC (permalink / raw)
To: jeremy.guthrie; +Cc: netdev, Robert Olsson, Stephen Hemminger
Jeremy M. Guthrie writes:
> Pulling Rx Polling has kept V2.6 from dropping packets. argh?!?!?!
Thats fine...
> I should have caught that sooner. 8( Well, I have a window of opportunity
> here. If you want I can still provide a broken environment but that seems
> pointless while NAPI is not acting right... what is evident is that this
> will run better till we start to bump up against the top of CPU0.
I don't follow...
The stats didn't show any numbers so we don't know your load.
--ro
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-06 21:36 ` Robert Olsson
@ 2005-01-06 21:46 ` Jeremy M. Guthrie
2005-01-06 22:11 ` Robert Olsson
0 siblings, 1 reply; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-06 21:46 UTC (permalink / raw)
To: netdev; +Cc: Robert Olsson, Stephen Hemminger
[-- Attachment #1: Type: text/plain, Size: 1016 bytes --]
On Thursday 06 January 2005 03:36 pm, Robert Olsson wrote:
> > I should have caught that sooner. 8( Well, I have a window of
> > opportunity here. If you want I can still provide a broken environment
> > but that seems pointless while NAPI is not acting right... what is
> > evident is that this will run better till we start to bump up against
> > the top of CPU0.
>
> I don't follow...
Technically by turning off NAPI, I have 'solved' my short term packet loss
problem.
However I can still provide you a window to go through further troubleshooting
if it is beneficial.
> The stats didn't show any numbers so we don't know your load.
Are you referring to the # of entries in the rt_cache table?
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-06 21:46 ` Jeremy M. Guthrie
@ 2005-01-06 22:11 ` Robert Olsson
2005-01-06 22:18 ` Jeremy M. Guthrie
0 siblings, 1 reply; 88+ messages in thread
From: Robert Olsson @ 2005-01-06 22:11 UTC (permalink / raw)
To: jeremy.guthrie; +Cc: netdev, Robert Olsson, Stephen Hemminger
Jeremy M. Guthrie writes:
> > I don't follow...
> Technically by turning off NAPI, I have 'solved' my short term packet loss
> problem.
You need NAPI of course just wait until your packet load increases just
a little bit... I saw drops in your 2.4 /proc/net/softnet_stat which
indicates you are close to your system performance. With NAPI you keep
up your system system performance regardless of incoming load.
The e1000 driver has some bugs in "your setup" as it enables irq's when
there is only RX and no TX and should have irq's disabled. I sent patch
to intel.
--ro
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-06 22:11 ` Robert Olsson
@ 2005-01-06 22:18 ` Jeremy M. Guthrie
2005-01-06 22:35 ` Robert Olsson
0 siblings, 1 reply; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-06 22:18 UTC (permalink / raw)
To: netdev; +Cc: Robert Olsson, Stephen Hemminger
[-- Attachment #1: Type: text/plain, Size: 1292 bytes --]
On Thursday 06 January 2005 04:11 pm, Robert Olsson wrote:
> Jeremy M. Guthrie writes:
> > > I don't follow...
> >
> > Technically by turning off NAPI, I have 'solved' my short term packet
> > loss problem.
>
> You need NAPI of course just wait until your packet load increases just
> a little bit...
I agree. Running mpstat now shows that even w/o packet drops, I'm running
2-5% free CPU.
> I saw drops in your 2.4 /proc/net/softnet_stat which
> indicates you are close to your system performance.
Should those drops show up in RX drops in ifconfig?
> With NAPI you keep
> up your system system performance regardless of incoming load.
You mentioned before that " The stats didn't show any numbers so we don't know
your load." Was there a command you wanted me to re-run?
> The e1000 driver has some bugs in "your setup" as it enables irq's when
> there is only RX and no TX and should have irq's disabled. I sent patch
> to intel.
>
> --ro
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-06 22:18 ` Jeremy M. Guthrie
@ 2005-01-06 22:35 ` Robert Olsson
2005-01-07 16:17 ` Jeremy M. Guthrie
0 siblings, 1 reply; 88+ messages in thread
From: Robert Olsson @ 2005-01-06 22:35 UTC (permalink / raw)
To: jeremy.guthrie; +Cc: netdev, Robert Olsson, Stephen Hemminger
Jeremy M. Guthrie writes:
> You mentioned before that " The stats didn't show any numbers so we don't
> know your load." Was there a command you wanted me to re-run?
rtstat should show the routing/packet load.
From a systems like yours 2*933 MHz PIII in production for tens of thoundans
of users many filter and full BGP routing. Current (late here) use.
ifstat2 eth*
RX -------------------------- TX -------------------------
eth0 272.8 M bit/s 51 k pps 350.7 M bit/s 51 k pps
eth1 371.9 M bit/s 55 k pps 293.6 M bit/s 55 k pps
eth2 6.7 M bit/s 1348 pps 3.0 M bit/s 991 pps
eth3 472 bit/s 0 pps 600 bit/s 0 pps
rtstat
size IN: hit tot mc no_rt bcast madst masrc OUT: hit tot mc GC: tot ignored goal_miss ovrf HASH: in_search out_search
21007 114060 748 0 5 0 0 0 11 5 0 0 0 0 0 58280 4
22683 112556 827 0 6 0 0 0 5 5 0 0 0 0 0 60841 7
24230 111083 765 0 4 0 0 0 13 4 0 0 0 0 0 66628 7
Around 110 kpps hitting warm cache entries and ~800 new lookups/sec. I was
expecting so see something similar from your system.
FYI.
cat /proc/net/softnet_stat
9ba490f3 00000000 01281572 00000000 00000000 00000000 00000000 00000000 002562c2
9939268d 00000000 010e42e9 00000000 00000000 00000000 00000000 00000000 0028fe72
Good Night.
--ro
^ permalink raw reply [flat|nested] 88+ messages in thread* Re: V2.4 policy router operates faster/better than V2.6
2005-01-06 22:35 ` Robert Olsson
@ 2005-01-07 16:17 ` Jeremy M. Guthrie
2005-01-07 19:18 ` Robert Olsson
0 siblings, 1 reply; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-07 16:17 UTC (permalink / raw)
To: netdev; +Cc: Robert Olsson, Stephen Hemminger
[-- Attachment #1: Type: text/plain, Size: 3111 bytes --]
On Thursday 06 January 2005 04:35 pm, Robert Olsson wrote:
> Jeremy M. Guthrie writes:
> > You mentioned before that " The stats didn't show any numbers so we
> > don't know your load." Was there a command you wanted me to re-run?
>
> rtstat should show the routing/packet load.
>
> From a systems like yours 2*933 MHz PIII in production for tens of
> thoundans of users many filter and full BGP routing. Current (late here)
> use.
>
>
> ifstat2 eth*
> RX -------------------------- TX -------------------------
> eth0 272.8 M bit/s 51 k pps 350.7 M bit/s 51 k pps
> eth1 371.9 M bit/s 55 k pps 293.6 M bit/s 55 k pps
> eth2 6.7 M bit/s 1348 pps 3.0 M bit/s 991 pps
> eth3 472 bit/s 0 pps 600 bit/s 0 pps
Ifstat output:
#kernel
Interface RX Pkts/Rate TX Pkts/Rate RX Data/Rate TX Data/Rate
RX Errs/Drop TX Errs/Drop RX Over/Rate TX Coll/Rate
lo 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
eth0 1 0 1 0 66 0 178 0
0 0 0 0 0 0 0 0
eth2 0 0 30770 0 0 0 13361K 0
0 0 0 0 0 0 0 0
eth3 81019 0 0 0 41740K 0 0 0
0 0 0 0 0 0 0 0
> rtstat
> size IN: hit tot mc no_rt bcast madst masrc OUT: hit tot
> mc GC: tot ignored goal_miss ovrf HASH: in_search out_search 21007
> 114060 748 0 5 0 0 0 11 5 0
> 0 0 0 0 58280 4 22683 112556 827
> 0 6 0 0 0 5 5 0 0 0
> 0 0 60841 7 24230 111083 765 0 4
> 0 0 0 13 4 0 0 0 0 0
> 66628 7
> Around 110 kpps hitting warm cache entries and ~800 new lookups/sec. I was
> expecting so see something similar from your system.
Did my second email w/ the lnstat data not make it?
>
> FYI.
> cat /proc/net/softnet_stat
total droppped tsquz Throttl FR_hit FR_succe FR_defer FR_def_o
cpu_coll
> 9ba490f3 00000000 01281572 00000000 00000000 00000000 00000000 00000000
> 002562c2 9939268d 00000000 010e42e9 00000000 00000000 00000000 00000000
> 00000000 0028fe72
Why do these drops not show up in the interface drop?
> Good Night.
>
>
> --ro
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 88+ messages in thread* Re: V2.4 policy router operates faster/better than V2.6
2005-01-07 16:17 ` Jeremy M. Guthrie
@ 2005-01-07 19:18 ` Robert Olsson
2005-01-07 19:38 ` Jeremy M. Guthrie
0 siblings, 1 reply; 88+ messages in thread
From: Robert Olsson @ 2005-01-07 19:18 UTC (permalink / raw)
To: jeremy.guthrie; +Cc: netdev, Robert Olsson, Stephen Hemminger
Jeremy M. Guthrie writes:
> Did my second email w/ the lnstat data not make it?
I see most zeroes
> > cat /proc/net/softnet_stat
> total droppped tsquz Throttl FR_hit FR_succe FR_defer FR_def_o
> cpu_coll
> > 9ba490f3 00000000 01281572 00000000 00000000 00000000 00000000 00000000
> > 002562c2 9939268d 00000000 010e42e9 00000000 00000000 00000000 00000000
> > 00000000 0028fe72
> Why do these drops not show up in the interface drop?
These are drops in the network stack not in the devices.
--ro
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-07 19:18 ` Robert Olsson
@ 2005-01-07 19:38 ` Jeremy M. Guthrie
2005-01-07 20:07 ` Robert Olsson
2005-01-07 20:14 ` Jeremy M. Guthrie
0 siblings, 2 replies; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-07 19:38 UTC (permalink / raw)
To: netdev; +Cc: Robert Olsson, Stephen Hemminger
[-- Attachment #1: Type: text/plain, Size: 1043 bytes --]
On Friday 07 January 2005 01:18 pm, Robert Olsson wrote:
> Jeremy M. Guthrie writes:
> > Did my second email w/ the lnstat data not make it?
> I see most zeroes
Is there a particular duration and count you want me to run for? The snapshot
should have been 60 one-second snapshots.
> > > cat /proc/net/softnet_stat
> >
> > total droppped tsquz Throttl FR_hit FR_succe FR_defer FR_def_o
> > cpu_coll
> >
> > > 9ba490f3 00000000 01281572 00000000 00000000 00000000 00000000
> > > 00000000 002562c2 9939268d 00000000 010e42e9 00000000 00000000
> > > 00000000 00000000 00000000 0028fe72
> >
> > Why do these drops not show up in the interface drop?
>
> These are drops in the network stack not in the devices.
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-07 19:38 ` Jeremy M. Guthrie
@ 2005-01-07 20:07 ` Robert Olsson
2005-01-07 20:14 ` Jeremy M. Guthrie
1 sibling, 0 replies; 88+ messages in thread
From: Robert Olsson @ 2005-01-07 20:07 UTC (permalink / raw)
To: jeremy.guthrie; +Cc: netdev, Robert Olsson, Stephen Hemminger
Jeremy M. Guthrie writes:
> Is there a particular duration and count you want me to run for?
> The snapshot should have been 60 one-second snapshots.
That should be OK as we would see one GC going and some flows recreated
after this.
--ro
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-07 19:38 ` Jeremy M. Guthrie
2005-01-07 20:07 ` Robert Olsson
@ 2005-01-07 20:14 ` Jeremy M. Guthrie
2005-01-07 20:40 ` Robert Olsson
2005-01-07 22:28 ` Jesse Brandeburg
1 sibling, 2 replies; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-07 20:14 UTC (permalink / raw)
To: netdev; +Cc: Robert Olsson, Stephen Hemminger
[-- Attachment #1: Type: text/plain, Size: 2638 bytes --]
I just updated the Intel Drivers to the latest on source-forge(5.6.10.1). I
now see lower CPU usage but I am still dropping.
During a 60 second window the machine received 5,110,164 packets and dropped
20461 or roughly 0.4% packet loss.
cat /proc/net/softnet_stat
2f9c9bd0 150dc67c 00c4701b 000d2659 00000000 00000000 00000000 00000000
00097049
00010f9b 00000000 0000003a 00000000 00000000 00000000 00000000 00000000
000002fc
It has been at 150dc67c for a while now. So while I am dropping at the card,
I am not dropping in the stack.
mpstat -P ALL 2
Linux 2.6.10 (h-idspr-msn-1) 01/07/05
14:06:45 CPU %user %nice %system %iowait %irq %soft %idle
intr/s
14:06:47 all 0.00 0.00 0.00 0.75 1.74 43.03 54.48
22598.51
14:06:47 0 0.00 0.00 0.00 0.00 2.49 78.61 18.91
17570.65
14:06:47 1 0.00 0.00 0.00 1.49 1.49 7.46 90.05
5023.88
14:06:47 CPU %user %nice %system %iowait %irq %soft %idle
intr/s
14:06:49 all 0.00 0.00 0.00 0.00 1.50 45.75 52.75
19352.00
14:06:49 0 0.00 0.00 0.00 0.00 2.50 83.00 14.50
14467.00
14:06:49 1 0.00 0.00 0.00 0.00 0.50 9.00 90.50
4886.50
14:06:49 CPU %user %nice %system %iowait %irq %soft %idle
intr/s
14:06:51 all 0.00 0.00 0.00 0.00 1.75 42.50 55.75
22482.59
14:06:51 0 0.00 0.00 0.00 0.00 2.49 77.61 19.90
17572.64
14:06:51 1 0.00 0.00 0.50 0.00 1.00 6.97 91.54
4919.90
14:06:51 CPU %user %nice %system %iowait %irq %soft %idle
intr/s
14:06:53 all 0.00 0.00 0.25 0.00 1.99 43.03 54.73
22458.00
14:06:53 0 0.00 0.00 0.00 0.00 3.00 78.50 18.50
17456.00
14:06:53 1 0.00 0.00 0.00 0.00 0.50 8.00 91.00
4992.50
14:06:53 CPU %user %nice %system %iowait %irq %soft %idle
intr/s
14:06:55 all 0.00 0.00 0.00 0.00 1.75 42.75 55.50
22854.00
14:06:55 0 0.00 0.00 0.00 0.00 3.00 77.00 20.00
17838.00
14:06:55 1 0.00 0.00 0.00 0.00 1.00 8.50 91.00
5012.50
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-07 20:14 ` Jeremy M. Guthrie
@ 2005-01-07 20:40 ` Robert Olsson
2005-01-07 21:06 ` Jeremy M. Guthrie
2005-01-07 22:28 ` Jesse Brandeburg
1 sibling, 1 reply; 88+ messages in thread
From: Robert Olsson @ 2005-01-07 20:40 UTC (permalink / raw)
To: jeremy.guthrie; +Cc: netdev, Robert Olsson, Stephen Hemminger
Jeremy M. Guthrie writes:
> During a 60 second window the machine received 5,110,164 packets and
> dropped 20461 or roughly 0.4% packet loss.
Around 85 kpps. If you run rtstat we could a feeling how may slow-pathx
that are taken. Or save the /proc/net/stat/rt_cache before you 60 sec
run.
mpstat I don't trust in this context.
> It has been at 150dc67c for a while now. So while I am dropping at the
> card, I am not dropping in the stack.
You use NAPI driver then...
Check if the patch below is in your e1000 driver.
--ro
--- drivers/net/e1000/e1000_main.c.orig 2004-02-16 14:46:16.000000000 +0100
+++ drivers/net/e1000/e1000_main.c 2004-02-16 15:45:05.000000000 +0100
@@ -2161,19 +2161,21 @@
struct e1000_adapter *adapter = netdev->priv;
int work_to_do = min(*budget, netdev->quota);
int work_done = 0;
-
- e1000_clean_tx_irq(adapter);
+ static boolean_t tx_cleaned;
+
+ tx_cleaned = e1000_clean_tx_irq(adapter);
e1000_clean_rx_irq(adapter, &work_done, work_to_do);
*budget -= work_done;
netdev->quota -= work_done;
- if(work_done < work_to_do || !netif_running(netdev)) {
+ if( (!tx_cleaned && (work_done == 0)) || !netif_running(netdev)) {
netif_rx_complete(netdev);
e1000_irq_enable(adapter);
+ return 0;
}
- return (work_done >= work_to_do);
+ return 1;
}
#endif
^ permalink raw reply [flat|nested] 88+ messages in thread* Re: V2.4 policy router operates faster/better than V2.6
2005-01-07 20:40 ` Robert Olsson
@ 2005-01-07 21:06 ` Jeremy M. Guthrie
2005-01-07 21:30 ` Robert Olsson
0 siblings, 1 reply; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-07 21:06 UTC (permalink / raw)
To: netdev; +Cc: Robert Olsson, Stephen Hemminger
[-- Attachment #1: Type: text/plain, Size: 3962 bytes --]
On Friday 07 January 2005 02:40 pm, Robert Olsson wrote:
> Jeremy M. Guthrie writes:
> > During a 60 second window the machine received 5,110,164 packets and
> > dropped 20461 or roughly 0.4% packet loss.
>
> Around 85 kpps. If you run rtstat we could a feeling how may slow-pathx
> that are taken.
I can't run rstat. The rt_cache_stat file doesn't exist in /proc/net
or /proc/net/stat in either V2.4 or V2.6.
> Or save the /proc/net/stat/rt_cache before you 60 sec
> run.
lnstat isn't giving me anything other than zeros.
entries in_hit in_slow_tot in_no_route in_brd in_martian_dst in_martian_src
out_hit out_slow_tot out_slow_mc gc_total gc_ignored gc_goal_miss
gc_dst_overflow in_hlist_search out_hlist_search
0000f91e 15b1ddab 0b69a32c 00000000 00000000 00001b0a 00001fbd 00000000
00066f65 0000b943 00000000 087b80a1 0878e81a 00000256 00000000 ca08c77d
0020e549
0000f91e 00005c59 0000b3ef 00000000 00000000 00000cf7 00000000 00000000
0000000f 0000004c 00000002 00000fb9 00000fb5 00000000 00000000 0004e8d9
000000b2
60 seconds......
entries in_hit in_slow_tot in_no_route in_brd in_martian_dst in_martian_src
out_hit out_slow_tot out_slow_mc gc_total gc_ignored gc_goal_miss
gc_dst_overflow in_hlist_search out_hlist_search
0000fb3e 15ff4696 0b6bbb65 00000000 00000000 00001b10 00001fc1 00000000
00067111 0000b974 00000000 087d9903 087b0003 00000256 00000000 cb58b399
0020ee79
0000fb3e 00005c5b 0000b40d 00000000 00000000 00000cf9 00000000 00000000
0000000f 0000004c 00000002 00000fbb 00000fb7 00000000 00000000 0004e9ba
000000b2
> mpstat I don't trust in this context.
>
> > It has been at 150dc67c for a while now. So while I am dropping at the
> > card, I am not dropping in the stack.
>
> You use NAPI driver then...
> Check if the patch below is in your e1000 driver.
> --ro
The drivers should be built as NAPI.
Here is the snippet:
static int
e1000_clean(struct net_device *netdev, int *budget)
{
struct e1000_adapter *adapter = netdev->priv;
int work_to_do = min(*budget, netdev->quota);
int tx_cleaned;
int work_done = 0;
if (!netif_carrier_ok(netdev))
goto quit_polling;
tx_cleaned = e1000_clean_tx_irq(adapter);
e1000_clean_rx_irq(adapter, &work_done, work_to_do);
*budget -= work_done;
netdev->quota -= work_done;
/* if no Rx and Tx cleanup work was done, exit the polling mode */
if(!tx_cleaned || (work_done < work_to_do) ||
!netif_running(netdev)) {
quit_polling: netif_rx_complete(netdev);
e1000_irq_enable(adapter);
return 0;
}
return (work_done >= work_to_do);
}
>
> --- drivers/net/e1000/e1000_main.c.orig 2004-02-16 14:46:16.000000000
> +0100 +++ drivers/net/e1000/e1000_main.c 2004-02-16 15:45:05.000000000
> +0100 @@ -2161,19 +2161,21 @@
> struct e1000_adapter *adapter = netdev->priv;
> int work_to_do = min(*budget, netdev->quota);
> int work_done = 0;
> -
> - e1000_clean_tx_irq(adapter);
> + static boolean_t tx_cleaned;
> +
> + tx_cleaned = e1000_clean_tx_irq(adapter);
> e1000_clean_rx_irq(adapter, &work_done, work_to_do);
>
> *budget -= work_done;
> netdev->quota -= work_done;
>
> - if(work_done < work_to_do || !netif_running(netdev)) {
> + if( (!tx_cleaned && (work_done == 0)) || !netif_running(netdev)) {
> netif_rx_complete(netdev);
> e1000_irq_enable(adapter);
> + return 0;
> }
>
> - return (work_done >= work_to_do);
> + return 1;
> }
> #endif
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 88+ messages in thread* Re: V2.4 policy router operates faster/better than V2.6
2005-01-07 21:06 ` Jeremy M. Guthrie
@ 2005-01-07 21:30 ` Robert Olsson
2005-01-11 15:11 ` Jeremy M. Guthrie
0 siblings, 1 reply; 88+ messages in thread
From: Robert Olsson @ 2005-01-07 21:30 UTC (permalink / raw)
To: jeremy.guthrie; +Cc: netdev, Robert Olsson, Stephen Hemminger
Jeremy M. Guthrie writes:
> lnstat isn't giving me anything other than zeros.
Crap.
> entries in_hit in_slow_tot in_no_route in_brd in_martian_dst in_martian_src
> out_hit out_slow_tot out_slow_mc gc_total gc_ignored gc_goal_miss
> gc_dst_overflow in_hlist_search out_hlist_search
> 0000f91e 15b1ddab 0b69a32c 00000000 00000000 00001b0a 00001fbd 00000000
> 00066f65 0000b943 00000000 087b80a1 0878e81a 00000256 00000000 ca08c77d
> 0020e549
> 0000fb3e 15ff4696 0b6bbb65 00000000 00000000 00001b10 00001fc1 00000000
> 00067111 0000b974 00000000 087d9903 087b0003 00000256 00000000 cb58b399
> 0020ee79
85 kpps and 2287 lookups/sec as a 60 sec average. Pretty nice load yes.
Where did you get the load?
Try see if you fix lnstat :-) it would be nice to see the route dynamics.
Or use the try the rtstat I pointed to.
And apply the e1000 patch I sent and make a test run.
I'll have a beer and give up for tonight...
--ro
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-07 20:14 ` Jeremy M. Guthrie
2005-01-07 20:40 ` Robert Olsson
@ 2005-01-07 22:28 ` Jesse Brandeburg
2005-01-07 22:50 ` Jeremy M. Guthrie
1 sibling, 1 reply; 88+ messages in thread
From: Jesse Brandeburg @ 2005-01-07 22:28 UTC (permalink / raw)
To: Jeremy M. Guthrie; +Cc: netdev, Robert Olsson, Stephen Hemminger
On Fri, 7 Jan 2005, Jeremy M. Guthrie wrote:
> I just updated the Intel Drivers to the latest on source-forge(5.6.10.1). I
> now see lower CPU usage but I am still dropping.
>
> During a 60 second window the machine received 5,110,164 packets and dropped
> 20461 or roughly 0.4% packet loss.
>
NAPI will cause a very busy stack to force the network card to drop the
data instead of the stack. This is supposed to be a good thing because
the hardware will be forced to drop the packet (hopefully) instead of
interrupt rate thrashing your machine right when it needs the cpu to do
other stuff. This is supposed to be the saving grace of NAPI.
So, not to distract from the conversation, but in the interest (my
interest :-) ) of tuning your E1000, can you please send the output of
ethtool -S ethX for each interface? This will help us figure out if there
is anything to tune in the driver (like number of rx buffers, etc)
Thanks,
Jesse
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-07 22:28 ` Jesse Brandeburg
@ 2005-01-07 22:50 ` Jeremy M. Guthrie
2005-01-07 22:57 ` Stephen Hemminger
0 siblings, 1 reply; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-07 22:50 UTC (permalink / raw)
To: netdev; +Cc: Jesse Brandeburg, Robert Olsson, Stephen Hemminger
[-- Attachment #1: Type: text/plain, Size: 3369 bytes --]
On Friday 07 January 2005 04:28 pm, Jesse Brandeburg wrote:
> On Fri, 7 Jan 2005, Jeremy M. Guthrie wrote:
> > I just updated the Intel Drivers to the latest on source-forge(5.6.10.1).
> > I now see lower CPU usage but I am still dropping.
> >
> > During a 60 second window the machine received 5,110,164 packets and
> > dropped 20461 or roughly 0.4% packet loss.
>
> NAPI will cause a very busy stack to force the network card to drop the
> data instead of the stack. This is supposed to be a good thing because
> the hardware will be forced to drop the packet (hopefully) instead of
> interrupt rate thrashing your machine right when it needs the cpu to do
> other stuff. This is supposed to be the saving grace of NAPI.
Makes sense.
> So, not to distract from the conversation, but in the interest (my
> interest :-) ) of tuning your E1000, can you please send the output of
> ethtool -S ethX for each interface? This will help us figure out if there
> is anything to tune in the driver (like number of rx buffers, etc)
ethtool -S eth2
NIC statistics:
rx_packets: 0
tx_packets: 314550698
rx_bytes: 0
tx_bytes: 4290523139
rx_errors: 0
tx_errors: 0
rx_dropped: 0
tx_dropped: 0
multicast: 0
collisions: 0
rx_length_errors: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 0
rx_missed_errors: 0
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_window_errors: 0
tx_abort_late_coll: 0
tx_deferred_ok: 0
tx_single_coll_ok: 0
tx_multi_coll_ok: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
rx_align_errors: 0
tx_tcp_seg_good: 0
tx_tcp_seg_failed: 0
rx_flow_control_xon: 0
rx_flow_control_xoff: 0
tx_flow_control_xon: 0
tx_flow_control_xoff: 0
rx_long_byte_count: 0
rx_csum_offload_good: 0
rx_csum_offload_errors: 0
ethtool -S eth3
NIC statistics:
rx_packets: 719847127
tx_packets: 5
rx_bytes: 1880301945
tx_bytes: 398
rx_errors: 3368295
tx_errors: 0
rx_dropped: 1478044
tx_dropped: 0
multicast: 0
collisions: 0
rx_length_errors: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 1890251
rx_missed_errors: 1890251
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_window_errors: 0
tx_abort_late_coll: 0
tx_deferred_ok: 0
tx_single_coll_ok: 0
tx_multi_coll_ok: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
rx_align_errors: 0
tx_tcp_seg_good: 0
tx_tcp_seg_failed: 0
rx_flow_control_xon: 0
rx_flow_control_xoff: 0
tx_flow_control_xon: 0
tx_flow_control_xoff: 0
rx_long_byte_count: 379837423993
rx_csum_offload_good: 672891990
rx_csum_offload_errors: 178666
> Thanks,
> Jesse
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 88+ messages in thread* Re: V2.4 policy router operates faster/better than V2.6
2005-01-07 22:50 ` Jeremy M. Guthrie
@ 2005-01-07 22:57 ` Stephen Hemminger
2005-01-11 15:17 ` Jeremy M. Guthrie
0 siblings, 1 reply; 88+ messages in thread
From: Stephen Hemminger @ 2005-01-07 22:57 UTC (permalink / raw)
To: jeremy.guthrie; +Cc: netdev, Jesse Brandeburg, Robert Olsson
On Fri, 7 Jan 2005 16:50:51 -0600
"Jeremy M. Guthrie" <jeremy.guthrie@berbee.com> wrote:
> On Friday 07 January 2005 04:28 pm, Jesse Brandeburg wrote:
> > On Fri, 7 Jan 2005, Jeremy M. Guthrie wrote:
> > > I just updated the Intel Drivers to the latest on source-forge(5.6.10.1).
> > > I now see lower CPU usage but I am still dropping.
> > >
> > > During a 60 second window the machine received 5,110,164 packets and
> > > dropped 20461 or roughly 0.4% packet loss.
> >
> > NAPI will cause a very busy stack to force the network card to drop the
> > data instead of the stack. This is supposed to be a good thing because
> > the hardware will be forced to drop the packet (hopefully) instead of
> > interrupt rate thrashing your machine right when it needs the cpu to do
> > other stuff. This is supposed to be the saving grace of NAPI.
> Makes sense.
Not sure if NAPI makes sense on transmit because it causes the transmit
ring to grow and freeing the transmit skb should be quick. Perhaps
other interrupt moderation schemes work better.
On receive the processing could take longer and NAPI is a real win.
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-07 22:57 ` Stephen Hemminger
@ 2005-01-11 15:17 ` Jeremy M. Guthrie
2005-01-11 16:40 ` Robert Olsson
2005-01-11 17:17 ` Jeremy M. Guthrie
0 siblings, 2 replies; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-11 15:17 UTC (permalink / raw)
To: netdev; +Cc: Stephen Hemminger, Jesse Brandeburg, Robert Olsson
[-- Attachment #1: Type: text/plain, Size: 3052 bytes --]
date ; ifconfig eth3 ; cat /proc/net/softnet_stat ;
cat /proc/net/stat/rt_cache ; sleep 60 ; date ; ifconfig eth3 ;
cat /proc/net/softnet_stat ; cat /proc/net/stat/rt_cache
Tue Jan 11 09:12:21 CST 2005
eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
inet addr:10.253.0.1 Bcast:10.255.255.255 Mask:255.255.255.0
inet6 addr: fe80::202:b3ff:fed5:7e30/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3519452697 errors:5558592 dropped:5558592
overruns:4011523 frame:0
TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:497775695 (474.7 Mb) TX bytes:398 (398.0 b)
Base address:0x22a0 Memory:eff80000-effa0000
f59427dc 150dc67c 00c562ab 000d2659 00000000 00000000 00000000 00000000
00547b5b
00038622 00000000 00000065 00000000 00000000 00000000 00000000 00000000
0006804f
entries in_hit in_slow_tot in_no_route in_brd in_martian_dst in_martian_src
out_hit out_slow_tot out_slow_mc gc_total gc_ignored gc_goal_miss
gc_dst_overflow in_hlist_search out_hlist_search
0000f3c2 aeca1565 272c383d 00000000 00000000 000089a8 00005298 00000002
001db403 0003054f 00000000 230aa73c 22fe33eb 00000cf0 00000000 b37011dc
00957c14
0000f3c2 0000b975 00029703 00000000 00000000 000035a5 00000000 00000000
00000015 00000083 00000002 000038c8 000038a3 00000000 00000000 0012a566
0000014d
Tue Jan 11 09:13:21 CST 2005
eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
inet addr:10.253.0.1 Bcast:10.255.255.255 Mask:255.255.255.0
inet6 addr: fe80::202:b3ff:fed5:7e30/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3524302561 errors:5571396 dropped:5571396
overruns:4022383 frame:0
TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3065327250 (2923.3 Mb) TX bytes:398 (398.0 b)
Base address:0x22a0 Memory:eff80000-effa0000
f5de26f5 150dc67c 00c562b4 000d2659 00000000 00000000 00000000 00000000
00547ffd
00038632 00000000 00000065 00000000 00000000 00000000 00000000 00000000
0006807c
entries in_hit in_slow_tot in_no_route in_brd in_martian_dst in_martian_src
out_hit out_slow_tot out_slow_mc gc_total gc_ignored gc_goal_miss
gc_dst_overflow in_hlist_search out_hlist_search
0000fc1e af11f07b 272e58d3 00000000 00000000 000089a8 00005298 00000002
001db571 00030572 00000000 230cc7f1 23005428 00000cf0 00000000 b4b5b529
009583b9
0000fc1e 0000b977 00029710 00000000 00000000 000035a7 00000000 00000000
00000015 00000083 00000002 000038ca 000038a5 00000000 00000000 0012a5c8
0000014d
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 88+ messages in thread* Re: V2.4 policy router operates faster/better than V2.6
2005-01-11 15:17 ` Jeremy M. Guthrie
@ 2005-01-11 16:40 ` Robert Olsson
2005-01-12 1:27 ` Jeremy M. Guthrie
2005-01-11 17:17 ` Jeremy M. Guthrie
1 sibling, 1 reply; 88+ messages in thread
From: Robert Olsson @ 2005-01-11 16:40 UTC (permalink / raw)
To: jeremy.guthrie; +Cc: netdev, Stephen Hemminger, Jesse Brandeburg, Robert Olsson
About the same load as last time 80 kpps forwarded and droprate of ~200 pps
and 2323 fib lookups/sec. Not so bad for PIII but it maybe tuned better.
Did the e1000 patch cure the problem that interrupts got enabled for
unidirectional traffic? Ring size? I've never seen any win in system
performance w. RX rings larger than 256 at least not in lab.
--ro
Jeremy M. Guthrie writes:
> date ; ifconfig eth3 ; cat /proc/net/softnet_stat ;
> cat /proc/net/stat/rt_cache ; sleep 60 ; date ; ifconfig eth3 ;
> cat /proc/net/softnet_stat ; cat /proc/net/stat/rt_cache
>
> Tue Jan 11 09:12:21 CST 2005
> eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
> inet addr:10.253.0.1 Bcast:10.255.255.255 Mask:255.255.255.0
> inet6 addr: fe80::202:b3ff:fed5:7e30/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:3519452697 errors:5558592 dropped:5558592
> overruns:4011523 frame:0
> TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:497775695 (474.7 Mb) TX bytes:398 (398.0 b)
> Base address:0x22a0 Memory:eff80000-effa0000
>
> f59427dc 150dc67c 00c562ab 000d2659 00000000 00000000 00000000 00000000
> 00547b5b
> 00038622 00000000 00000065 00000000 00000000 00000000 00000000 00000000
> 0006804f
> entries in_hit in_slow_tot in_no_route in_brd in_martian_dst in_martian_src
> out_hit out_slow_tot out_slow_mc gc_total gc_ignored gc_goal_miss
> gc_dst_overflow in_hlist_search out_hlist_search
> 0000f3c2 aeca1565 272c383d 00000000 00000000 000089a8 00005298 00000002
> 001db403 0003054f 00000000 230aa73c 22fe33eb 00000cf0 00000000 b37011dc
> 00957c14
> 0000f3c2 0000b975 00029703 00000000 00000000 000035a5 00000000 00000000
> 00000015 00000083 00000002 000038c8 000038a3 00000000 00000000 0012a566
> 0000014d
>
> Tue Jan 11 09:13:21 CST 2005
> eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
> inet addr:10.253.0.1 Bcast:10.255.255.255 Mask:255.255.255.0
> inet6 addr: fe80::202:b3ff:fed5:7e30/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:3524302561 errors:5571396 dropped:5571396
> overruns:4022383 frame:0
> TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:3065327250 (2923.3 Mb) TX bytes:398 (398.0 b)
> Base address:0x22a0 Memory:eff80000-effa0000
>
> f5de26f5 150dc67c 00c562b4 000d2659 00000000 00000000 00000000 00000000
> 00547ffd
> 00038632 00000000 00000065 00000000 00000000 00000000 00000000 00000000
> 0006807c
> entries in_hit in_slow_tot in_no_route in_brd in_martian_dst in_martian_src
> out_hit out_slow_tot out_slow_mc gc_total gc_ignored gc_goal_miss
> gc_dst_overflow in_hlist_search out_hlist_search
> 0000fc1e af11f07b 272e58d3 00000000 00000000 000089a8 00005298 00000002
> 001db571 00030572 00000000 230cc7f1 23005428 00000cf0 00000000 b4b5b529
> 009583b9
> 0000fc1e 0000b977 00029710 00000000 00000000 000035a7 00000000 00000000
> 00000015 00000083 00000002 000038ca 000038a5 00000000 00000000 0012a5c8
> 0000014d
>
> --
>
> --------------------------------------------------
> Jeremy M. Guthrie jeremy.guthrie@berbee.com
> Senior Network Engineer Phone: 608-298-1061
> Berbee Fax: 608-288-3007
> 5520 Research Park Drive NOC: 608-298-1102
> Madison, WI 53711
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-11 16:40 ` Robert Olsson
@ 2005-01-12 1:27 ` Jeremy M. Guthrie
2005-01-12 15:11 ` Robert Olsson
0 siblings, 1 reply; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-12 1:27 UTC (permalink / raw)
To: netdev; +Cc: Robert Olsson, Stephen Hemminger, Jesse Brandeburg
On Tuesday 11 January 2005 10:40 am, Robert Olsson wrote:
> About the same load as last time 80 kpps forwarded and droprate of ~200 pps
> and 2323 fib lookups/sec. Not so bad for PIII but it maybe tuned better.
>
> Did the e1000 patch cure the problem that interrupts got enabled for
> unidirectional traffic? Ring size? I've never seen any win in system
> performance w. RX rings larger than 256 at least not in lab.
ETH3 Interrupts(calc'd from below): 1479968
ETH2 Interrupts: 261543
Packets RX'd on ETH3: 3892720
Packets dropped on RX on ETH3: 10305
This equates to about a 0.26% drop rate. W/ 256 packet RX ring size I see
about a 0.42% drop rate.
This is using both the newest Intel driver w/ your patch and an increased ring
size of 2048.
Tue Jan 11 19:15:04 CST 2005
eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
inet addr:10.253.0.1 Bcast:10.255.255.255 Mask:255.255.255.0
inet6 addr: fe80::202:b3ff:fed5:7e30/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2337377824 errors:14992144 dropped:14992144
overruns:9643826 frame:0
TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3984064976 (3799.5 Mb) TX bytes:398 (398.0 b)
Base address:0x22a0 Memory:eff80000-effa0000
CPU0 CPU1
0: 93627934 353254270 IO-APIC-edge timer
1: 35 507 IO-APIC-edge i8042
7: 0 0 IO-APIC-level ohci_hcd
8: 0 2 IO-APIC-edge rtc
12: 73 145 IO-APIC-edge i8042
14: 120 313 IO-APIC-edge ide0
18: 2158179576 1815 IO-APIC-level eth3
20: 2 2136514988 IO-APIC-level eth2
27: 204201 371301 IO-APIC-level eth0
28: 14585 75320 IO-APIC-level aic7xxx
30: 0 0 IO-APIC-level acpi
NMI: 0 0
LOC: 446922783 446921227
ERR: 0
MIS: 0
Tue Jan 11 19:16:05 CST 2005
eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
inet addr:10.253.0.1 Bcast:10.255.255.255 Mask:255.255.255.0
inet6 addr: fe80::202:b3ff:fed5:7e30/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2341270544 errors:15002449 dropped:15002449
overruns:9652393 frame:0
TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1664751812 (1587.6 Mb) TX bytes:398 (398.0 b)
Base address:0x22a0 Memory:eff80000-effa0000
CPU0 CPU1
0: 93639955 353302319 IO-APIC-edge timer
1: 35 507 IO-APIC-edge i8042
7: 0 0 IO-APIC-level ohci_hcd
8: 0 2 IO-APIC-edge rtc
12: 73 145 IO-APIC-edge i8042
14: 120 313 IO-APIC-edge ide0
18: 2159659544 1815 IO-APIC-level eth3
20: 2 2136776531 IO-APIC-level eth2
27: 204245 371369 IO-APIC-level eth0
28: 14593 75343 IO-APIC-level aic7xxx
30: 0 0 IO-APIC-level acpi
NMI: 0 0
LOC: 446982858 446981302
ERR: 0
MIS: 0
--ro
>
> Jeremy M. Guthrie writes:
> > date ; ifconfig eth3 ; cat /proc/net/softnet_stat ;
> > cat /proc/net/stat/rt_cache ; sleep 60 ; date ; ifconfig eth3 ;
> > cat /proc/net/softnet_stat ; cat /proc/net/stat/rt_cache
> >
> > Tue Jan 11 09:12:21 CST 2005
> > eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
> > inet addr:10.253.0.1 Bcast:10.255.255.255 Mask:255.255.255.0
> > inet6 addr: fe80::202:b3ff:fed5:7e30/64 Scope:Link
> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > RX packets:3519452697 errors:5558592 dropped:5558592
> > overruns:4011523 frame:0
> > TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
> > collisions:0 txqueuelen:1000
> > RX bytes:497775695 (474.7 Mb) TX bytes:398 (398.0 b)
> > Base address:0x22a0 Memory:eff80000-effa0000
> >
> > f59427dc 150dc67c 00c562ab 000d2659 00000000 00000000 00000000 00000000
> > 00547b5b
> > 00038622 00000000 00000065 00000000 00000000 00000000 00000000 00000000
> > 0006804f
> > entries in_hit in_slow_tot in_no_route in_brd in_martian_dst
> > in_martian_src out_hit out_slow_tot out_slow_mc gc_total gc_ignored
> > gc_goal_miss gc_dst_overflow in_hlist_search out_hlist_search
> > 0000f3c2 aeca1565 272c383d 00000000 00000000 000089a8 00005298 00000002
> > 001db403 0003054f 00000000 230aa73c 22fe33eb 00000cf0 00000000 b37011dc
> > 00957c14
> > 0000f3c2 0000b975 00029703 00000000 00000000 000035a5 00000000 00000000
> > 00000015 00000083 00000002 000038c8 000038a3 00000000 00000000 0012a566
> > 0000014d
> >
> > Tue Jan 11 09:13:21 CST 2005
> > eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
> > inet addr:10.253.0.1 Bcast:10.255.255.255 Mask:255.255.255.0
> > inet6 addr: fe80::202:b3ff:fed5:7e30/64 Scope:Link
> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > RX packets:3524302561 errors:5571396 dropped:5571396
> > overruns:4022383 frame:0
> > TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
> > collisions:0 txqueuelen:1000
> > RX bytes:3065327250 (2923.3 Mb) TX bytes:398 (398.0 b)
> > Base address:0x22a0 Memory:eff80000-effa0000
> >
> > f5de26f5 150dc67c 00c562b4 000d2659 00000000 00000000 00000000 00000000
> > 00547ffd
> > 00038632 00000000 00000065 00000000 00000000 00000000 00000000 00000000
> > 0006807c
> > entries in_hit in_slow_tot in_no_route in_brd in_martian_dst
> > in_martian_src out_hit out_slow_tot out_slow_mc gc_total gc_ignored
> > gc_goal_miss gc_dst_overflow in_hlist_search out_hlist_search
> > 0000fc1e af11f07b 272e58d3 00000000 00000000 000089a8 00005298 00000002
> > 001db571 00030572 00000000 230cc7f1 23005428 00000cf0 00000000 b4b5b529
> > 009583b9
> > 0000fc1e 0000b977 00029710 00000000 00000000 000035a7 00000000 00000000
> > 00000015 00000083 00000002 000038ca 000038a5 00000000 00000000 0012a5c8
> > 0000014d
> >
> > --
> >
> > --------------------------------------------------
> > Jeremy M. Guthrie jeremy.guthrie@berbee.com
> > Senior Network Engineer Phone: 608-298-1061
> > Berbee Fax: 608-288-3007
> > 5520 Research Park Drive NOC: 608-298-1102
> > Madison, WI 53711
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
^ permalink raw reply [flat|nested] 88+ messages in thread* Re: V2.4 policy router operates faster/better than V2.6
2005-01-12 1:27 ` Jeremy M. Guthrie
@ 2005-01-12 15:11 ` Robert Olsson
2005-01-12 16:24 ` Jeremy M. Guthrie
0 siblings, 1 reply; 88+ messages in thread
From: Robert Olsson @ 2005-01-12 15:11 UTC (permalink / raw)
To: jeremy.guthrie; +Cc: netdev, Robert Olsson, Stephen Hemminger, Jesse Brandeburg
Jeremy M. Guthrie writes:
> > Did the e1000 patch cure the problem that interrupts got enabled for
> > unidirectional traffic? Ring size? I've never seen any win in system
> > performance w. RX rings larger than 256 at least not in lab.
> ETH3 Interrupts(calc'd from below): 1479968
> ETH2 Interrupts: 261543
> Packets RX'd on ETH3: 3892720
> Packets dropped on RX on ETH3: 10305
Very strange...
eth3 is bound to CPU0 which in turn has all packet load... If we were
to believe your CPU0 was saturated (due to the drops). We should see no
(RX) interrupts on eth3. But there is a lot... one irq per every three
packet. Why?
Can you investigate? e1000 has problem like this w. unidirectional traffic
w/o the patch I sent.
Or is your traffic so extremely bursty. No?
--ro
> This equates to about a 0.26% drop rate. W/ 256 packet RX ring size I see
> about a 0.42% drop rate.
>
> This is using both the newest Intel driver w/ your patch and an increased ring
> size of 2048.
>
> Tue Jan 11 19:15:04 CST 2005
> eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
> inet addr:10.253.0.1 Bcast:10.255.255.255 Mask:255.255.255.0
> inet6 addr: fe80::202:b3ff:fed5:7e30/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:2337377824 errors:14992144 dropped:14992144
> overruns:9643826 frame:0
> TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:3984064976 (3799.5 Mb) TX bytes:398 (398.0 b)
> Base address:0x22a0 Memory:eff80000-effa0000
>
> CPU0 CPU1
> 0: 93627934 353254270 IO-APIC-edge timer
> 1: 35 507 IO-APIC-edge i8042
> 7: 0 0 IO-APIC-level ohci_hcd
> 8: 0 2 IO-APIC-edge rtc
> 12: 73 145 IO-APIC-edge i8042
> 14: 120 313 IO-APIC-edge ide0
> 18: 2158179576 1815 IO-APIC-level eth3
> 20: 2 2136514988 IO-APIC-level eth2
> 27: 204201 371301 IO-APIC-level eth0
> 28: 14585 75320 IO-APIC-level aic7xxx
> 30: 0 0 IO-APIC-level acpi
> NMI: 0 0
> LOC: 446922783 446921227
> ERR: 0
> MIS: 0
>
> Tue Jan 11 19:16:05 CST 2005
> eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
> inet addr:10.253.0.1 Bcast:10.255.255.255 Mask:255.255.255.0
> inet6 addr: fe80::202:b3ff:fed5:7e30/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:2341270544 errors:15002449 dropped:15002449
> overruns:9652393 frame:0
> TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:1664751812 (1587.6 Mb) TX bytes:398 (398.0 b)
> Base address:0x22a0 Memory:eff80000-effa0000
>
> CPU0 CPU1
> 0: 93639955 353302319 IO-APIC-edge timer
> 1: 35 507 IO-APIC-edge i8042
> 7: 0 0 IO-APIC-level ohci_hcd
> 8: 0 2 IO-APIC-edge rtc
> 12: 73 145 IO-APIC-edge i8042
> 14: 120 313 IO-APIC-edge ide0
> 18: 2159659544 1815 IO-APIC-level eth3
> 20: 2 2136776531 IO-APIC-level eth2
> 27: 204245 371369 IO-APIC-level eth0
> 28: 14593 75343 IO-APIC-level aic7xxx
> 30: 0 0 IO-APIC-level acpi
> NMI: 0 0
> LOC: 446982858 446981302
> ERR: 0
> MIS: 0
>
> --ro
> >
> > Jeremy M. Guthrie writes:
> > > date ; ifconfig eth3 ; cat /proc/net/softnet_stat ;
> > > cat /proc/net/stat/rt_cache ; sleep 60 ; date ; ifconfig eth3 ;
> > > cat /proc/net/softnet_stat ; cat /proc/net/stat/rt_cache
> > >
> > > Tue Jan 11 09:12:21 CST 2005
> > > eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
> > > inet addr:10.253.0.1 Bcast:10.255.255.255 Mask:255.255.255.0
> > > inet6 addr: fe80::202:b3ff:fed5:7e30/64 Scope:Link
> > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > > RX packets:3519452697 errors:5558592 dropped:5558592
> > > overruns:4011523 frame:0
> > > TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
> > > collisions:0 txqueuelen:1000
> > > RX bytes:497775695 (474.7 Mb) TX bytes:398 (398.0 b)
> > > Base address:0x22a0 Memory:eff80000-effa0000
> > >
> > > f59427dc 150dc67c 00c562ab 000d2659 00000000 00000000 00000000 00000000
> > > 00547b5b
> > > 00038622 00000000 00000065 00000000 00000000 00000000 00000000 00000000
> > > 0006804f
> > > entries in_hit in_slow_tot in_no_route in_brd in_martian_dst
> > > in_martian_src out_hit out_slow_tot out_slow_mc gc_total gc_ignored
> > > gc_goal_miss gc_dst_overflow in_hlist_search out_hlist_search
> > > 0000f3c2 aeca1565 272c383d 00000000 00000000 000089a8 00005298 00000002
> > > 001db403 0003054f 00000000 230aa73c 22fe33eb 00000cf0 00000000 b37011dc
> > > 00957c14
> > > 0000f3c2 0000b975 00029703 00000000 00000000 000035a5 00000000 00000000
> > > 00000015 00000083 00000002 000038c8 000038a3 00000000 00000000 0012a566
> > > 0000014d
> > >
> > > Tue Jan 11 09:13:21 CST 2005
> > > eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
> > > inet addr:10.253.0.1 Bcast:10.255.255.255 Mask:255.255.255.0
> > > inet6 addr: fe80::202:b3ff:fed5:7e30/64 Scope:Link
> > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > > RX packets:3524302561 errors:5571396 dropped:5571396
> > > overruns:4022383 frame:0
> > > TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
> > > collisions:0 txqueuelen:1000
> > > RX bytes:3065327250 (2923.3 Mb) TX bytes:398 (398.0 b)
> > > Base address:0x22a0 Memory:eff80000-effa0000
> > >
> > > f5de26f5 150dc67c 00c562b4 000d2659 00000000 00000000 00000000 00000000
> > > 00547ffd
> > > 00038632 00000000 00000065 00000000 00000000 00000000 00000000 00000000
> > > 0006807c
> > > entries in_hit in_slow_tot in_no_route in_brd in_martian_dst
> > > in_martian_src out_hit out_slow_tot out_slow_mc gc_total gc_ignored
> > > gc_goal_miss gc_dst_overflow in_hlist_search out_hlist_search
> > > 0000fc1e af11f07b 272e58d3 00000000 00000000 000089a8 00005298 00000002
> > > 001db571 00030572 00000000 230cc7f1 23005428 00000cf0 00000000 b4b5b529
> > > 009583b9
> > > 0000fc1e 0000b977 00029710 00000000 00000000 000035a7 00000000 00000000
> > > 00000015 00000083 00000002 000038ca 000038a5 00000000 00000000 0012a5c8
> > > 0000014d
> > >
> > > --
> > >
> > > --------------------------------------------------
> > > Jeremy M. Guthrie jeremy.guthrie@berbee.com
> > > Senior Network Engineer Phone: 608-298-1061
> > > Berbee Fax: 608-288-3007
> > > 5520 Research Park Drive NOC: 608-298-1102
> > > Madison, WI 53711
>
> --
>
> --------------------------------------------------
> Jeremy M. Guthrie jeremy.guthrie@berbee.com
> Senior Network Engineer Phone: 608-298-1061
> Berbee Fax: 608-288-3007
> 5520 Research Park Drive NOC: 608-298-1102
> Madison, WI 53711
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-12 15:11 ` Robert Olsson
@ 2005-01-12 16:24 ` Jeremy M. Guthrie
2005-01-12 19:27 ` Robert Olsson
0 siblings, 1 reply; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-12 16:24 UTC (permalink / raw)
To: netdev; +Cc: Robert Olsson, Stephen Hemminger, Jesse Brandeburg
[-- Attachment #1: Type: text/plain, Size: 4775 bytes --]
On Wednesday 12 January 2005 09:11 am, Robert Olsson wrote:
> Jeremy M. Guthrie writes:
> > > Did the e1000 patch cure the problem that interrupts got enabled for
> > > unidirectional traffic? Ring size? I've never seen any win in system
> > > performance w. RX rings larger than 256 at least not in lab.
> >
> > ETH3 Interrupts(calc'd from below): 1479968
> > ETH2 Interrupts: 261543
> > Packets RX'd on ETH3: 3892720
> > Packets dropped on RX on ETH3: 10305
>
> Very strange...
>
> eth3 is bound to CPU0 which in turn has all packet load... If we were
> to believe your CPU0 was saturated (due to the drops). We should see no
> (RX) interrupts on eth3. But there is a lot... one irq per every three
> packet. Why?
I have no idea why it would be doing this.
> Can you investigate? e1000 has problem like this w. unidirectional traffic
> w/o the patch I sent.
This appears to be where this problem is getting beyond my expertise. I
verified NAPI is turned on. I also verified the patch is in place. I am
open to suggestions but otherwise I am not the worlds best coder.
> Or is your traffic so extremely bursty. No?
By the nature of our business it can be very bursty. We just have so many
sources/destinations for traffic that our traffic is generally pretty bursty.
We might be running 300-400 mbps with spikes of 40-100 mbps.
> --ro
>
> > This equates to about a 0.26% drop rate. W/ 256 packet RX ring size I
> > see about a 0.42% drop rate.
> >
> > This is using both the newest Intel driver w/ your patch and an
> > increased ring size of 2048.
> >
> > Tue Jan 11 19:15:04 CST 2005
> > eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
> > inet addr:10.253.0.1 Bcast:10.255.255.255 Mask:255.255.255.0
> > inet6 addr: fe80::202:b3ff:fed5:7e30/64 Scope:Link
> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > RX packets:2337377824 errors:14992144 dropped:14992144
> > overruns:9643826 frame:0
> > TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
> > collisions:0 txqueuelen:1000
> > RX bytes:3984064976 (3799.5 Mb) TX bytes:398 (398.0 b)
> > Base address:0x22a0 Memory:eff80000-effa0000
> >
> > CPU0 CPU1
> > 0: 93627934 353254270 IO-APIC-edge timer
> > 1: 35 507 IO-APIC-edge i8042
> > 7: 0 0 IO-APIC-level ohci_hcd
> > 8: 0 2 IO-APIC-edge rtc
> > 12: 73 145 IO-APIC-edge i8042
> > 14: 120 313 IO-APIC-edge ide0
> > 18: 2158179576 1815 IO-APIC-level eth3
> > 20: 2 2136514988 IO-APIC-level eth2
> > 27: 204201 371301 IO-APIC-level eth0
> > 28: 14585 75320 IO-APIC-level aic7xxx
> > 30: 0 0 IO-APIC-level acpi
> > NMI: 0 0
> > LOC: 446922783 446921227
> > ERR: 0
> > MIS: 0
> >
> > Tue Jan 11 19:16:05 CST 2005
> > eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
> > inet addr:10.253.0.1 Bcast:10.255.255.255 Mask:255.255.255.0
> > inet6 addr: fe80::202:b3ff:fed5:7e30/64 Scope:Link
> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > RX packets:2341270544 errors:15002449 dropped:15002449
> > overruns:9652393 frame:0
> > TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
> > collisions:0 txqueuelen:1000
> > RX bytes:1664751812 (1587.6 Mb) TX bytes:398 (398.0 b)
> > Base address:0x22a0 Memory:eff80000-effa0000
> >
> > CPU0 CPU1
> > 0: 93639955 353302319 IO-APIC-edge timer
> > 1: 35 507 IO-APIC-edge i8042
> > 7: 0 0 IO-APIC-level ohci_hcd
> > 8: 0 2 IO-APIC-edge rtc
> > 12: 73 145 IO-APIC-edge i8042
> > 14: 120 313 IO-APIC-edge ide0
> > 18: 2159659544 1815 IO-APIC-level eth3
> > 20: 2 2136776531 IO-APIC-level eth2
> > 27: 204245 371369 IO-APIC-level eth0
> > 28: 14593 75343 IO-APIC-level aic7xxx
> > 30: 0 0 IO-APIC-level acpi
> > NMI: 0 0
> > LOC: 446982858 446981302
> > ERR: 0
> > MIS: 0
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-12 16:24 ` Jeremy M. Guthrie
@ 2005-01-12 19:27 ` Robert Olsson
2005-01-12 20:11 ` Jeremy M. Guthrie
0 siblings, 1 reply; 88+ messages in thread
From: Robert Olsson @ 2005-01-12 19:27 UTC (permalink / raw)
To: jeremy.guthrie; +Cc: netdev, Robert Olsson, Stephen Hemminger, Jesse Brandeburg
Jeremy M. Guthrie writes:
> > > ETH3 Interrupts(calc'd from below): 1479968
> > Very strange...
> >
> > eth3 is bound to CPU0 which in turn has all packet load... If we were
> > to believe your CPU0 was saturated (due to the drops). We should see no
> > (RX) interrupts on eth3. But there is a lot... one irq per every three
> > packet. Why?
> I have no idea why it would be doing this.
Huh seems you didn't add the patch I sent. Below is diff from my editor to your
e1000_main.c
--ro
--- e1000_main.c.jmg 2005-01-12 20:14:08.324168072 +0100
+++ e1000_main.c 2005-01-12 20:17:24.777302656 +0100
@@ -2264,14 +2264,13 @@
netdev->quota -= work_done;
/* if no Rx and Tx cleanup work was done, exit the polling mode */
- if(!tx_cleaned || (work_done < work_to_do) ||
- !netif_running(netdev)) {
+ if( (!tx_cleaned && (work_done == 0)) || !netif_running(netdev)) {
quit_polling: netif_rx_complete(netdev);
e1000_irq_enable(adapter);
return 0;
}
- return (work_done >= work_to_do);
+ return 1;
}
#endif
^ permalink raw reply [flat|nested] 88+ messages in thread* Re: V2.4 policy router operates faster/better than V2.6
2005-01-12 19:27 ` Robert Olsson
@ 2005-01-12 20:11 ` Jeremy M. Guthrie
2005-01-12 20:21 ` Robert Olsson
0 siblings, 1 reply; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-12 20:11 UTC (permalink / raw)
To: netdev; +Cc: Robert Olsson, Stephen Hemminger, Jesse Brandeburg
[-- Attachment #1: Type: text/plain, Size: 4141 bytes --]
Latest numbers after your patch Robert.
Wed Jan 12 14:05:36 CST 2005
eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
inet addr:10.253.0.1 Bcast:10.255.255.255 Mask:255.255.255.0
inet6 addr: fe80::202:b3ff:fed5:7e30/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:21951496 errors:2412189 dropped:2412189 overruns:377090
frame:0
TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3455362966 (3295.2 Mb) TX bytes:398 (398.0 b)
Base address:0x22a0 Memory:eff80000-effa0000
95b83157 150dc67c 00c734a1 000d2659 00000000 00000000 00000000 00000000
0073cee4
00044494 00000000 00000075 00000000 00000000 00000000 00000000 00000000
00097c77
entries in_hit in_slow_tot in_no_route in_brd in_martian_dst in_martian_src
out_hit out_slow_tot out_slow_mc gc_total gc_ignored gc_goal_miss
gc_dst_overflow in_hlist_search out_hlist_search
0000f8cd 41403122 34c30e49 00000000 00000000 0000b0dd 00006a38 00000002
0027279d 0004147b 00000000 2f528a5f 2f42efcf 0000104f 00000000 81d262df
00c5d75c
0000f8cd 0000e332 0003238f 00000000 00000000 00004263 00000000 00000000
0000001e 000000c0 00000002 00004650 00004626 00000000 00000000 0016b879
0000024c
Wed Jan 12 14:06:36 CST 2005
eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
inet addr:10.253.0.1 Bcast:10.255.255.255 Mask:255.255.255.0
inet6 addr: fe80::202:b3ff:fed5:7e30/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:28053281 errors:2899869 dropped:2899869 overruns:427243
frame:0
TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2607094069 (2486.3 Mb) TX bytes:398 (398.0 b)
Base address:0x22a0 Memory:eff80000-effa0000
96154d20 150dc67c 00c78bc2 000d2659 00000000 00000000 00000000 00000000
0073d31d
00044499 00000000 00000075 00000000 00000000 00000000 00000000 00000000
00097d2d
entries in_hit in_slow_tot in_no_route in_brd in_martian_dst in_martian_src
out_hit out_slow_tot out_slow_mc gc_total gc_ignored gc_goal_miss
gc_dst_overflow in_hlist_search out_hlist_search
0000fc6a 419ad196 34c586ba 00000000 00000000 0000b0e0 00006a40 00000002
002729c5 000414cb 00000000 2f5502ff 2f4567f7 0000104f 00000000 83790467
00c5e33c
0000fc6a 0000e333 00032393 00000000 00000000 00004263 00000000 00000000
0000001e 000000c0 00000002 00004650 00004626 00000000 00000000 0016b898
0000024c
On Wednesday 12 January 2005 01:27 pm, Robert Olsson wrote:
> Jeremy M. Guthrie writes:
> > > > ETH3 Interrupts(calc'd from below): 1479968
> > >
> > > Very strange...
> > >
> > > eth3 is bound to CPU0 which in turn has all packet load... If we were
> > > to believe your CPU0 was saturated (due to the drops). We should see
> > > no (RX) interrupts on eth3. But there is a lot... one irq per every
> > > three packet. Why?
> >
> > I have no idea why it would be doing this.
>
> Huh seems you didn't add the patch I sent. Below is diff from my editor to
> your e1000_main.c
>
> --ro
>
>
> --- e1000_main.c.jmg 2005-01-12 20:14:08.324168072 +0100
> +++ e1000_main.c 2005-01-12 20:17:24.777302656 +0100
> @@ -2264,14 +2264,13 @@
> netdev->quota -= work_done;
>
> /* if no Rx and Tx cleanup work was done, exit the polling mode */
> - if(!tx_cleaned || (work_done < work_to_do) ||
> - !netif_running(netdev)) {
> + if( (!tx_cleaned && (work_done == 0)) || !netif_running(netdev)) {
> quit_polling: netif_rx_complete(netdev);
> e1000_irq_enable(adapter);
> return 0;
> }
>
> - return (work_done >= work_to_do);
> + return 1;
> }
>
> #endif
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 88+ messages in thread* Re: V2.4 policy router operates faster/better than V2.6
2005-01-12 20:11 ` Jeremy M. Guthrie
@ 2005-01-12 20:21 ` Robert Olsson
2005-01-12 20:30 ` Jeremy M. Guthrie
2005-01-12 20:45 ` Jeremy M. Guthrie
0 siblings, 2 replies; 88+ messages in thread
From: Robert Olsson @ 2005-01-12 20:21 UTC (permalink / raw)
To: jeremy.guthrie; +Cc: netdev, Robert Olsson, Stephen Hemminger, Jesse Brandeburg
Jeremy M. Guthrie writes:
>
> Latest numbers after your patch Robert.
Did the RX interrupts go down?
--ro
> Wed Jan 12 14:05:36 CST 2005
> eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
> inet addr:10.253.0.1 Bcast:10.255.255.255 Mask:255.255.255.0
> inet6 addr: fe80::202:b3ff:fed5:7e30/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:21951496 errors:2412189 dropped:2412189 overruns:377090
> frame:0
> TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:3455362966 (3295.2 Mb) TX bytes:398 (398.0 b)
> Base address:0x22a0 Memory:eff80000-effa0000
>
> 95b83157 150dc67c 00c734a1 000d2659 00000000 00000000 00000000 00000000
> 0073cee4
> 00044494 00000000 00000075 00000000 00000000 00000000 00000000 00000000
> 00097c77
> entries in_hit in_slow_tot in_no_route in_brd in_martian_dst in_martian_src
> out_hit out_slow_tot out_slow_mc gc_total gc_ignored gc_goal_miss
> gc_dst_overflow in_hlist_search out_hlist_search
> 0000f8cd 41403122 34c30e49 00000000 00000000 0000b0dd 00006a38 00000002
> 0027279d 0004147b 00000000 2f528a5f 2f42efcf 0000104f 00000000 81d262df
> 00c5d75c
> 0000f8cd 0000e332 0003238f 00000000 00000000 00004263 00000000 00000000
> 0000001e 000000c0 00000002 00004650 00004626 00000000 00000000 0016b879
> 0000024c
>
>
>
>
>
> Wed Jan 12 14:06:36 CST 2005
> eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
> inet addr:10.253.0.1 Bcast:10.255.255.255 Mask:255.255.255.0
> inet6 addr: fe80::202:b3ff:fed5:7e30/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:28053281 errors:2899869 dropped:2899869 overruns:427243
> frame:0
> TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:2607094069 (2486.3 Mb) TX bytes:398 (398.0 b)
> Base address:0x22a0 Memory:eff80000-effa0000
>
> 96154d20 150dc67c 00c78bc2 000d2659 00000000 00000000 00000000 00000000
> 0073d31d
> 00044499 00000000 00000075 00000000 00000000 00000000 00000000 00000000
> 00097d2d
> entries in_hit in_slow_tot in_no_route in_brd in_martian_dst in_martian_src
> out_hit out_slow_tot out_slow_mc gc_total gc_ignored gc_goal_miss
> gc_dst_overflow in_hlist_search out_hlist_search
> 0000fc6a 419ad196 34c586ba 00000000 00000000 0000b0e0 00006a40 00000002
> 002729c5 000414cb 00000000 2f5502ff 2f4567f7 0000104f 00000000 83790467
> 00c5e33c
> 0000fc6a 0000e333 00032393 00000000 00000000 00004263 00000000 00000000
> 0000001e 000000c0 00000002 00004650 00004626 00000000 00000000 0016b898
> 0000024c
>
>
> On Wednesday 12 January 2005 01:27 pm, Robert Olsson wrote:
> > Jeremy M. Guthrie writes:
> > > > > ETH3 Interrupts(calc'd from below): 1479968
> > > >
> > > > Very strange...
> > > >
> > > > eth3 is bound to CPU0 which in turn has all packet load... If we were
> > > > to believe your CPU0 was saturated (due to the drops). We should see
> > > > no (RX) interrupts on eth3. But there is a lot... one irq per every
> > > > three packet. Why?
> > >
> > > I have no idea why it would be doing this.
> >
> > Huh seems you didn't add the patch I sent. Below is diff from my editor to
> > your e1000_main.c
> >
> > --ro
> >
> >
> > --- e1000_main.c.jmg 2005-01-12 20:14:08.324168072 +0100
> > +++ e1000_main.c 2005-01-12 20:17:24.777302656 +0100
> > @@ -2264,14 +2264,13 @@
> > netdev->quota -= work_done;
> >
> > /* if no Rx and Tx cleanup work was done, exit the polling mode */
> > - if(!tx_cleaned || (work_done < work_to_do) ||
> > - !netif_running(netdev)) {
> > + if( (!tx_cleaned && (work_done == 0)) || !netif_running(netdev)) {
> > quit_polling: netif_rx_complete(netdev);
> > e1000_irq_enable(adapter);
> > return 0;
> > }
> >
> > - return (work_done >= work_to_do);
> > + return 1;
> > }
> >
> > #endif
>
> --
>
> --------------------------------------------------
> Jeremy M. Guthrie jeremy.guthrie@berbee.com
> Senior Network Engineer Phone: 608-298-1061
> Berbee Fax: 608-288-3007
> 5520 Research Park Drive NOC: 608-298-1102
> Madison, WI 53711
^ permalink raw reply [flat|nested] 88+ messages in thread* Re: V2.4 policy router operates faster/better than V2.6
2005-01-12 20:21 ` Robert Olsson
@ 2005-01-12 20:30 ` Jeremy M. Guthrie
2005-01-12 20:45 ` Jeremy M. Guthrie
1 sibling, 0 replies; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-12 20:30 UTC (permalink / raw)
To: netdev; +Cc: Robert Olsson, Stephen Hemminger, Jesse Brandeburg
[-- Attachment #1: Type: text/plain, Size: 7120 bytes --]
Sorry, here is the latest. BTW, it is VERY sluggish now.
Wed Jan 12 14:25:03 CST 2005
eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
RX packets:32030222 errors:105026820 dropped:105026820
overruns:100748696 frame:0
9651ffeb 150dc67c 00c814c1 000d2659 00000000 00000000 00000000 00000000
0073d527
00044681 00000000 000000f8 00000000 00000000 00000000 00000000 00000000
00097d42
entries in_hit in_slow_tot in_no_route in_brd in_martian_dst in_martian_src
out_hit out_slow_tot out_slow_mc gc_total gc_ignored gc_goal_miss
gc_dst_overflow in_hlist_search out_hlist_search
00020000 41c25a6f 34daaca8 00000000 00000000 0000b11d 00006a40 00000002
00272af5 00041525 00000000 2f6a27e5 2f4b712d 000f2bd8 000f1b0e 84036c96
00c5e8a8
00020000 0000e336 0003256f 00000000 00000000 00004292 00000000 00000000
0000001e 000000c0 00000002 00004762 0000462a 0000010e 0000010b 0016b898
0000024c
CPU0 CPU1
18: 3586173518 1815 IO-APIC-level eth3
20: 2 2464382507 IO-APIC-level eth2
Wed Jan 12 14:26:03 CST 2005
eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
RX packets:32097966 errors:110579564 dropped:110579564
overruns:106248073 frame:0
965308aa 150dc67c 00c818e4 000d2659 00000000 00000000 00000000 00000000
0073d527
0004468b 00000000 000000f8 00000000 00000000 00000000 00000000 00000000
00097d42
entries in_hit in_slow_tot in_no_route in_brd in_martian_dst in_martian_src
out_hit out_slow_tot out_slow_mc gc_total gc_ignored gc_goal_miss
gc_dst_overflow in_hlist_search out_hlist_search
00020000 41c2602e 34dbaf9f 00000000 00000000 0000b11e 00006a40 00000002
00272af5 00041527 00000000 2f6b2ad8 2f4b9a88 00100571 000ff4a4 84036c9e
00c5e8a8
00020000 0000e336 00032578 00000000 00000000 00004292 00000000 00000000
0000001e 000000c0 00000002 00004762 0000462a 0000010e 0000010b 0016b898
0000024c
CPU0 CPU1
18: 3586173518 1815 IO-APIC-level eth3
20: 2 2464387985 IO-APIC-level eth2
On Wednesday 12 January 2005 02:21 pm, Robert Olsson wrote:
> Jeremy M. Guthrie writes:
> > Latest numbers after your patch Robert.
>
> Did the RX interrupts go down?
>
> --ro
>
> > Wed Jan 12 14:05:36 CST 2005
> > eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
> > inet addr:10.253.0.1 Bcast:10.255.255.255 Mask:255.255.255.0
> > inet6 addr: fe80::202:b3ff:fed5:7e30/64 Scope:Link
> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > RX packets:21951496 errors:2412189 dropped:2412189
> > overruns:377090 frame:0
> > TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
> > collisions:0 txqueuelen:1000
> > RX bytes:3455362966 (3295.2 Mb) TX bytes:398 (398.0 b)
> > Base address:0x22a0 Memory:eff80000-effa0000
> >
> > 95b83157 150dc67c 00c734a1 000d2659 00000000 00000000 00000000 00000000
> > 0073cee4
> > 00044494 00000000 00000075 00000000 00000000 00000000 00000000 00000000
> > 00097c77
> > entries in_hit in_slow_tot in_no_route in_brd in_martian_dst
> > in_martian_src out_hit out_slow_tot out_slow_mc gc_total gc_ignored
> > gc_goal_miss gc_dst_overflow in_hlist_search out_hlist_search
> > 0000f8cd 41403122 34c30e49 00000000 00000000 0000b0dd 00006a38 00000002
> > 0027279d 0004147b 00000000 2f528a5f 2f42efcf 0000104f 00000000 81d262df
> > 00c5d75c
> > 0000f8cd 0000e332 0003238f 00000000 00000000 00004263 00000000 00000000
> > 0000001e 000000c0 00000002 00004650 00004626 00000000 00000000 0016b879
> > 0000024c
> >
> >
> >
> >
> >
> > Wed Jan 12 14:06:36 CST 2005
> > eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
> > inet addr:10.253.0.1 Bcast:10.255.255.255 Mask:255.255.255.0
> > inet6 addr: fe80::202:b3ff:fed5:7e30/64 Scope:Link
> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > RX packets:28053281 errors:2899869 dropped:2899869
> > overruns:427243 frame:0
> > TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
> > collisions:0 txqueuelen:1000
> > RX bytes:2607094069 (2486.3 Mb) TX bytes:398 (398.0 b)
> > Base address:0x22a0 Memory:eff80000-effa0000
> >
> > 96154d20 150dc67c 00c78bc2 000d2659 00000000 00000000 00000000 00000000
> > 0073d31d
> > 00044499 00000000 00000075 00000000 00000000 00000000 00000000 00000000
> > 00097d2d
> > entries in_hit in_slow_tot in_no_route in_brd in_martian_dst
> > in_martian_src out_hit out_slow_tot out_slow_mc gc_total gc_ignored
> > gc_goal_miss gc_dst_overflow in_hlist_search out_hlist_search
> > 0000fc6a 419ad196 34c586ba 00000000 00000000 0000b0e0 00006a40 00000002
> > 002729c5 000414cb 00000000 2f5502ff 2f4567f7 0000104f 00000000 83790467
> > 00c5e33c
> > 0000fc6a 0000e333 00032393 00000000 00000000 00004263 00000000 00000000
> > 0000001e 000000c0 00000002 00004650 00004626 00000000 00000000 0016b898
> > 0000024c
> >
> > On Wednesday 12 January 2005 01:27 pm, Robert Olsson wrote:
> > > Jeremy M. Guthrie writes:
> > > > > > ETH3 Interrupts(calc'd from below): 1479968
> > > > >
> > > > > Very strange...
> > > > >
> > > > > eth3 is bound to CPU0 which in turn has all packet load... If we
> > > > > were to believe your CPU0 was saturated (due to the drops). We
> > > > > should see no (RX) interrupts on eth3. But there is a lot... one
> > > > > irq per every three packet. Why?
> > > >
> > > > I have no idea why it would be doing this.
> > >
> > > Huh seems you didn't add the patch I sent. Below is diff from my
> > > editor to your e1000_main.c
> > >
> > > --ro
> > >
> > >
> > > --- e1000_main.c.jmg 2005-01-12 20:14:08.324168072 +0100
> > > +++ e1000_main.c 2005-01-12 20:17:24.777302656 +0100
> > > @@ -2264,14 +2264,13 @@
> > > netdev->quota -= work_done;
> > >
> > > /* if no Rx and Tx cleanup work was done, exit the polling mode */
> > > - if(!tx_cleaned || (work_done < work_to_do) ||
> > > - !netif_running(netdev)) {
> > > + if( (!tx_cleaned && (work_done == 0)) || !netif_running(netdev)) {
> > > quit_polling: netif_rx_complete(netdev);
> > > e1000_irq_enable(adapter);
> > > return 0;
> > > }
> > >
> > > - return (work_done >= work_to_do);
> > > + return 1;
> > > }
> > >
> > > #endif
> >
> > --
> >
> > --------------------------------------------------
> > Jeremy M. Guthrie jeremy.guthrie@berbee.com
> > Senior Network Engineer Phone: 608-298-1061
> > Berbee Fax: 608-288-3007
> > 5520 Research Park Drive NOC: 608-298-1102
> > Madison, WI 53711
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 88+ messages in thread* Re: V2.4 policy router operates faster/better than V2.6
2005-01-12 20:21 ` Robert Olsson
2005-01-12 20:30 ` Jeremy M. Guthrie
@ 2005-01-12 20:45 ` Jeremy M. Guthrie
2005-01-12 22:02 ` Robert Olsson
2005-01-12 22:05 ` Jeremy M. Guthrie
1 sibling, 2 replies; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-12 20:45 UTC (permalink / raw)
To: netdev; +Cc: Robert Olsson, Stephen Hemminger, Jesse Brandeburg
[-- Attachment #1: Type: text/plain, Size: 5110 bytes --]
My throughput dropped from 500 mbps to 8mbps. 8(
On Wednesday 12 January 2005 02:21 pm, Robert Olsson wrote:
> Jeremy M. Guthrie writes:
> > Latest numbers after your patch Robert.
>
> Did the RX interrupts go down?
>
> --ro
>
> > Wed Jan 12 14:05:36 CST 2005
> > eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
> > inet addr:10.253.0.1 Bcast:10.255.255.255 Mask:255.255.255.0
> > inet6 addr: fe80::202:b3ff:fed5:7e30/64 Scope:Link
> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > RX packets:21951496 errors:2412189 dropped:2412189
> > overruns:377090 frame:0
> > TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
> > collisions:0 txqueuelen:1000
> > RX bytes:3455362966 (3295.2 Mb) TX bytes:398 (398.0 b)
> > Base address:0x22a0 Memory:eff80000-effa0000
> >
> > 95b83157 150dc67c 00c734a1 000d2659 00000000 00000000 00000000 00000000
> > 0073cee4
> > 00044494 00000000 00000075 00000000 00000000 00000000 00000000 00000000
> > 00097c77
> > entries in_hit in_slow_tot in_no_route in_brd in_martian_dst
> > in_martian_src out_hit out_slow_tot out_slow_mc gc_total gc_ignored
> > gc_goal_miss gc_dst_overflow in_hlist_search out_hlist_search
> > 0000f8cd 41403122 34c30e49 00000000 00000000 0000b0dd 00006a38 00000002
> > 0027279d 0004147b 00000000 2f528a5f 2f42efcf 0000104f 00000000 81d262df
> > 00c5d75c
> > 0000f8cd 0000e332 0003238f 00000000 00000000 00004263 00000000 00000000
> > 0000001e 000000c0 00000002 00004650 00004626 00000000 00000000 0016b879
> > 0000024c
> >
> >
> >
> >
> >
> > Wed Jan 12 14:06:36 CST 2005
> > eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
> > inet addr:10.253.0.1 Bcast:10.255.255.255 Mask:255.255.255.0
> > inet6 addr: fe80::202:b3ff:fed5:7e30/64 Scope:Link
> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > RX packets:28053281 errors:2899869 dropped:2899869
> > overruns:427243 frame:0
> > TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
> > collisions:0 txqueuelen:1000
> > RX bytes:2607094069 (2486.3 Mb) TX bytes:398 (398.0 b)
> > Base address:0x22a0 Memory:eff80000-effa0000
> >
> > 96154d20 150dc67c 00c78bc2 000d2659 00000000 00000000 00000000 00000000
> > 0073d31d
> > 00044499 00000000 00000075 00000000 00000000 00000000 00000000 00000000
> > 00097d2d
> > entries in_hit in_slow_tot in_no_route in_brd in_martian_dst
> > in_martian_src out_hit out_slow_tot out_slow_mc gc_total gc_ignored
> > gc_goal_miss gc_dst_overflow in_hlist_search out_hlist_search
> > 0000fc6a 419ad196 34c586ba 00000000 00000000 0000b0e0 00006a40 00000002
> > 002729c5 000414cb 00000000 2f5502ff 2f4567f7 0000104f 00000000 83790467
> > 00c5e33c
> > 0000fc6a 0000e333 00032393 00000000 00000000 00004263 00000000 00000000
> > 0000001e 000000c0 00000002 00004650 00004626 00000000 00000000 0016b898
> > 0000024c
> >
> > On Wednesday 12 January 2005 01:27 pm, Robert Olsson wrote:
> > > Jeremy M. Guthrie writes:
> > > > > > ETH3 Interrupts(calc'd from below): 1479968
> > > > >
> > > > > Very strange...
> > > > >
> > > > > eth3 is bound to CPU0 which in turn has all packet load... If we
> > > > > were to believe your CPU0 was saturated (due to the drops). We
> > > > > should see no (RX) interrupts on eth3. But there is a lot... one
> > > > > irq per every three packet. Why?
> > > >
> > > > I have no idea why it would be doing this.
> > >
> > > Huh seems you didn't add the patch I sent. Below is diff from my
> > > editor to your e1000_main.c
> > >
> > > --ro
> > >
> > >
> > > --- e1000_main.c.jmg 2005-01-12 20:14:08.324168072 +0100
> > > +++ e1000_main.c 2005-01-12 20:17:24.777302656 +0100
> > > @@ -2264,14 +2264,13 @@
> > > netdev->quota -= work_done;
> > >
> > > /* if no Rx and Tx cleanup work was done, exit the polling mode */
> > > - if(!tx_cleaned || (work_done < work_to_do) ||
> > > - !netif_running(netdev)) {
> > > + if( (!tx_cleaned && (work_done == 0)) || !netif_running(netdev)) {
> > > quit_polling: netif_rx_complete(netdev);
> > > e1000_irq_enable(adapter);
> > > return 0;
> > > }
> > >
> > > - return (work_done >= work_to_do);
> > > + return 1;
> > > }
> > >
> > > #endif
> >
> > --
> >
> > --------------------------------------------------
> > Jeremy M. Guthrie jeremy.guthrie@berbee.com
> > Senior Network Engineer Phone: 608-298-1061
> > Berbee Fax: 608-288-3007
> > 5520 Research Park Drive NOC: 608-298-1102
> > Madison, WI 53711
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 88+ messages in thread* Re: V2.4 policy router operates faster/better than V2.6
2005-01-12 20:45 ` Jeremy M. Guthrie
@ 2005-01-12 22:02 ` Robert Olsson
2005-01-12 22:21 ` Jeremy M. Guthrie
2005-01-12 22:05 ` Jeremy M. Guthrie
1 sibling, 1 reply; 88+ messages in thread
From: Robert Olsson @ 2005-01-12 22:02 UTC (permalink / raw)
To: jeremy.guthrie; +Cc: netdev, Robert Olsson, Stephen Hemminger, Jesse Brandeburg
Jeremy M. Guthrie writes:
> My throughput dropped from 500 mbps to 8mbps. 8(
Weird!
> CPU0 CPU1
> 18: 3586173518 1815 IO-APIC-level eth3
> 20: 2 2464382507 IO-APIC-level eth2
> CPU0 CPU1
> 18: 3586173518 1815 IO-APIC-level eth3
> 20: 2 2464387985 IO-APIC-level eth2
There are no irq's on eth3 at all so RX softirq is constantly running.
Which means it's deferred to ksoftirqd now and running under scheduler
context. Do you have anything that competes with ksoftirqd for CPU0 on
your system?
It used be recommended to increase the priority of ksoftirqd but I wonder
what's going on your system. We see interrupts on eth2...
And tsquz (3:rd col) in /proc/net/sofnet_stat indicates there are very
little activity from the RX softirq. It's soon time time up here.
--ro
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-12 22:02 ` Robert Olsson
@ 2005-01-12 22:21 ` Jeremy M. Guthrie
[not found] ` <16869.42247.126428.508479@robur.slu.se>
0 siblings, 1 reply; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-12 22:21 UTC (permalink / raw)
To: netdev; +Cc: Robert Olsson, Stephen Hemminger, Jesse Brandeburg
[-- Attachment #1: Type: text/plain, Size: 1368 bytes --]
On Wednesday 12 January 2005 04:02 pm, Robert Olsson wrote:
> Jeremy M. Guthrie writes:
> > My throughput dropped from 500 mbps to 8mbps. 8(
>
> Weird!
>
> > CPU0 CPU1
> > 18: 3586173518 1815 IO-APIC-level eth3
> > 20: 2 2464382507 IO-APIC-level eth2
> >
> > CPU0 CPU1
> > 18: 3586173518 1815 IO-APIC-level eth3
> > 20: 2 2464387985 IO-APIC-level eth2
>
> There are no irq's on eth3 at all so RX softirq is constantly running.
> Which means it's deferred to ksoftirqd now and running under scheduler
> context. Do you have anything that competes with ksoftirqd for CPU0 on
> your system?
This box is primarily routing. Nothing should be competing for CPU0
> It used be recommended to increase the priority of ksoftirqd but I wonder
> what's going on your system. We see interrupts on eth2...
>
> And tsquz (3:rd col) in /proc/net/sofnet_stat indicates there are very
> little activity from the RX softirq. It's soon time time up here.
>
>
> --ro
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-12 20:45 ` Jeremy M. Guthrie
2005-01-12 22:02 ` Robert Olsson
@ 2005-01-12 22:05 ` Jeremy M. Guthrie
2005-01-12 22:22 ` Robert Olsson
1 sibling, 1 reply; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-12 22:05 UTC (permalink / raw)
To: netdev; +Cc: Robert Olsson, Stephen Hemminger, Jesse Brandeburg
[-- Attachment #1: Type: text/plain, Size: 628 bytes --]
I am now getting some push back from the project manager on this performance
problem. I am wondering if you think faster CPUs will
a) help relieve the symptoms of this problem
b) not help because now we will hit a '# of routes in the route-cache'
problem
c) or will help to a point till the # interrupts come back and bite us.
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-12 22:05 ` Jeremy M. Guthrie
@ 2005-01-12 22:22 ` Robert Olsson
2005-01-12 22:30 ` Jeremy M. Guthrie
0 siblings, 1 reply; 88+ messages in thread
From: Robert Olsson @ 2005-01-12 22:22 UTC (permalink / raw)
To: jeremy.guthrie; +Cc: netdev, Robert Olsson, Stephen Hemminger, Jesse Brandeburg
Jeremy M. Guthrie writes:
> I am now getting some push back from the project manager on this performance
> problem. I am wondering if you think faster CPUs will
> a) help relieve the symptoms of this problem
> b) not help because now we will hit a '# of routes in the route-cache'
> problem
> c) or will help to a point till the # interrupts come back and bite us.
Back out the patch I sent.and have hardirq's to run RX-softirq as you
did before but something is very wrong. You didn't answer if there were
other load on the machine...
route-cache can probably be tuned you as have four times the linear seach
I see in one PIII system at 110 kpps w. production traffic.
Of course the non-engineering solution is to buy more CPU... :-)
--ro
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-12 22:22 ` Robert Olsson
@ 2005-01-12 22:30 ` Jeremy M. Guthrie
0 siblings, 0 replies; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-12 22:30 UTC (permalink / raw)
To: netdev; +Cc: Robert Olsson, Stephen Hemminger, Jesse Brandeburg
[-- Attachment #1: Type: text/plain, Size: 1445 bytes --]
On Wednesday 12 January 2005 04:22 pm, Robert Olsson wrote:
> Jeremy M. Guthrie writes:
> > I am now getting some push back from the project manager on this
> > performance problem. I am wondering if you think faster CPUs will
> > a) help relieve the symptoms of this problem
> > b) not help because now we will hit a '# of routes in the route-cache'
> > problem
> > c) or will help to a point till the # interrupts come back and bite us.
>
> Back out the patch I sent.and have hardirq's to run RX-softirq as you
> did before but something is very wrong. You didn't answer if there were
> other load on the machine...
I have backed out. As for the load, this box only does policy routing. Any
other functions it performs are part of its automated system to download the
next days policy-routing config.
> route-cache can probably be tuned you as have four times the linear seach
> I see in one PIII system at 110 kpps w. production traffic.
How would I go about tuning that?
> Of course the non-engineering solution is to buy more CPU... :-)
That is good to know. This will help me calm the situation a bit. 8)
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-11 15:17 ` Jeremy M. Guthrie
2005-01-11 16:40 ` Robert Olsson
@ 2005-01-11 17:17 ` Jeremy M. Guthrie
2005-01-11 18:46 ` Robert Olsson
1 sibling, 1 reply; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-11 17:17 UTC (permalink / raw)
To: netdev; +Cc: Stephen Hemminger, Jesse Brandeburg, Robert Olsson
[-- Attachment #1: Type: text/plain, Size: 8723 bytes --]
./rtstat
size IN: hit tot mc no_rt bcast madst masrc OUT: hit tot mc
GC: tot ignored goal_miss ovrf HASH: in_search out_search
63375 86092 2844 0 0 0 0 0 5 1
1610369017 2846 2844 0 0 386039 17
64870 84431 537031471 537492192 0 0 0 0 1 0
0 2848 2846 0 0 386130 10
58707 87889 2715 0 0 0 0 0 7 1 0
2716 2714 0 0 384782 13
60617 89993 2796 0 0 0 0 0 5 0 0
2795 2793 0 0 384432 17
62458 93221 2772 0 0 0 0 0 3 0 0
2772 2770 0 0 399579 9
63990 93792 2712 0 0 0 0 0 6 0 0
2713 2711 0 0 422585 12
65499 94235 2846 0 0 0 0 0 2 0 0
2846 2844 0 0 424341 9
59755 90992 2864 0 0 0 0 0 1 1 0
2865 2863 0 0 385888 13
61702 87681 2798 0 0 0 0 0 6 2 0
2800 2798 0 0 383276 34
63334 90557 2734 0 0 0 0 0 5 1 0
2735 2733 0 0 422121 28
64798 88178 2732 0 0 0 0 0 1 0 0
2732 2730 0 0 410772 11
58574 85910 2669 0 0 0 0 0 3 0 0
2670 2668 0 0 393331 10
60538 87238 2705 0 0 0 0 0 4 0 0
2705 2703 0 0 382161 20
62164 85738 2628 0 0 0 0 0 4 0 0
2628 2626 0 0 373049 25
63759 78752 2638 0 0 0 0 0 2 0 0
2638 2636 0 0 348411 5
65052 79832 2482 0 0 0 0 0 3 0 0
2481 2479 0 0 365871 20
58986 82611 2549 0 0 0 0 0 4 1 0
2550 2548 0 0 366429 24
60845 85340 2620 0 0 1 0 0 2 0 0
2620 2618 0 0 369963 16
62465 83122 2591 0 0 0 0 0 5 0 0
2591 2589 0 0 371261 18
size IN: hit tot mc no_rt bcast madst masrc OUT: hit tot mc
GC: tot ignored goal_miss ovrf HASH: in_search out_search
63998 86141 2640 0 0 0 0 0 4 0
537114631 2640 2638 0 0 386452 15
65294 85989 2527 0 0 0 0 0 5 1
1610369017 2528 2526 0 0 379801 26
59241 84305 2662 0 0 0 0 0 7 0 0
2662 2660 0 0 371925 35
61194 85397 2837 0 0 0 0 0 3 0 0
2837 2835 0 0 374788 17
62908 88133 2688 0 0 1 0 0 4 1 0
2689 2687 0 0 382411 19
64329 88495 2648 0 0 0 0 0 5 1 0
2649 2647 0 0 398763 30
65560 84994 2447 0 0 1 0 0 6 2 0
2449 2447 0 0 394745 33
59677 81478 2568 0 0 0 0 0 5 0 0
2569 2567 0 0 341439 22
61459 81225 2623 0 0 0 0 0 5 0 0
2623 2621 0 0 345996 30
63038 83660 2544 0 0 0 0 0 3 0 0
2544 2542 0 0 381567 21
64494 86321 2620 0 0 0 0 0 4 0 0
2620 2618 0 0 395278 19
65528 88754 2766 0 0 0 0 0 6 0 0
2765 2763 0 0 421075 34
59948 86443 2990 0 0 0 0 0 5 0 0
2990 2988 0 0 373605 29
61860 81912 2786 0 0 0 0 0 5 1 0
2787 2785 0 0 354678 34
63630 80399 2981 0 0 0 0 0 4 0 0
2979 2977 0 0 354983 32
65215 83600 2934 0 0 0 0 0 6 3 0
2937 2935 0 0 371497 41
59322 85583 2821 0 0 0 0 0 7 0 0
2822 2820 0 0 382896 40
61268 82602 2817 0 0 0 0 0 9 2 0
2819 2817 0 0 368078 54
62989 85336 2760 0 0 1 0 0 9 0 0
2760 2758 0 0 394212 39
64387 84861 2603 0 0 0 0 0 7 0 0
2602 2600 0 0 390852 36
size IN: hit tot mc no_rt bcast madst masrc OUT: hit tot mc
GC: tot ignored goal_miss ovrf HASH: in_search out_search
65681 82848 2625 0 0 0 0 0 8 0
537114631 2625 2623 0 0 370769 49
59923 82826 2685 0 0 0 0 0 6 0
1610369017 2685 2683 0 0 344094 41
61700 82705 2706 0 0 0 0 0 7 0 0
2706 2704 0 0 368474 39
63247 91042 2592 0 0 0 0 0 9 1 0
2593 2591 0 0 399393 57
64636 93050 2530 0 0 0 0 0 6 0 0
2530 2528 0 0 405903 24
58301 90662 2554 0 0 0 0 0 5 0 0
2555 2553 0 0 404296 39
60146 88405 2567 0 0 0 0 0 8 1 0
2567 2565 0 0 370227 49
61822 86417 2579 0 0 0 0 0 7 1 0
2580 2578 0 0 366323 43
63351 86517 2594 0 0 1 0 0 9 0 0
2594 2592 0 0 363340 51
64739 87879 2669 0 0 0 0 0 3 0 0
2670 2668 0 0 387580 26
58397 88297 2566 0 0 0 0 0 4 0 0
2566 2564 0 0 396980 34
60352 85039 2707 0 0 0 0 0 6 1 0
2709 2707 0 0 367790 47
62169 87083 2779 0 0 0 0 0 6 0 0
2775 2773 0 0 377163 31
63808 88700 2752 0 0 0 0 0 4 0 0
2751 2749 0 0 392830 28
65189 91242 2725 0 0 0 0 0 5 1 0
2727 2725 0 0 416871 31
59326 84893 2834 0 0 0 0 0 5 1 0
2828 2826 0 0 380811 28
61064 87351 2553 0 0 0 0 0 9 0 0
2552 2550 0 0 382053 37
62637 88761 2551 0 0 0 0 0 5 1 0
2553 2551 0 0 384974 29
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-11 17:17 ` Jeremy M. Guthrie
@ 2005-01-11 18:46 ` Robert Olsson
2005-01-12 1:30 ` Jeremy M. Guthrie
0 siblings, 1 reply; 88+ messages in thread
From: Robert Olsson @ 2005-01-11 18:46 UTC (permalink / raw)
To: jeremy.guthrie; +Cc: netdev, Stephen Hemminger, Jesse Brandeburg, Robert Olsson
Yes rtstats is what we estimated. How about RX ring size and the e1000 fix?
We give up here? Or you want to try another setting of route hash? It's beyond
the scope of this subject. You have to reboot if so.
--ro
Jeremy M. Guthrie writes:
> ./rtstat
> size IN: hit tot mc no_rt bcast madst masrc OUT: hit tot mc
> GC: tot ignored goal_miss ovrf HASH: in_search out_search
> 63375 86092 2844 0 0 0 0 0 5 1
> 1610369017 2846 2844 0 0 386039 17
> 64870 84431 537031471 537492192 0 0 0 0 1 0
> 0 2848 2846 0 0 386130 10
> 58707 87889 2715 0 0 0 0 0 7 1 0
> 2716 2714 0 0 384782 13
> 60617 89993 2796 0 0 0 0 0 5 0 0
> 2795 2793 0 0 384432 17
> 62458 93221 2772 0 0 0 0 0 3 0 0
> 2772 2770 0 0 399579 9
> 63990 93792 2712 0 0 0 0 0 6 0 0
> 2713 2711 0 0 422585 12
> 65499 94235 2846 0 0 0 0 0 2 0 0
> 2846 2844 0 0 424341 9
> 59755 90992 2864 0 0 0 0 0 1 1 0
> 2865 2863 0 0 385888 13
> 61702 87681 2798 0 0 0 0 0 6 2 0
> 2800 2798 0 0 383276 34
> 63334 90557 2734 0 0 0 0 0 5 1 0
> 2735 2733 0 0 422121 28
> 64798 88178 2732 0 0 0 0 0 1 0 0
> 2732 2730 0 0 410772 11
> 58574 85910 2669 0 0 0 0 0 3 0 0
> 2670 2668 0 0 393331 10
> 60538 87238 2705 0 0 0 0 0 4 0 0
> 2705 2703 0 0 382161 20
> 62164 85738 2628 0 0 0 0 0 4 0 0
> 2628 2626 0 0 373049 25
> 63759 78752 2638 0 0 0 0 0 2 0 0
> 2638 2636 0 0 348411 5
> 65052 79832 2482 0 0 0 0 0 3 0 0
> 2481 2479 0 0 365871 20
> 58986 82611 2549 0 0 0 0 0 4 1 0
> 2550 2548 0 0 366429 24
> 60845 85340 2620 0 0 1 0 0 2 0 0
> 2620 2618 0 0 369963 16
> 62465 83122 2591 0 0 0 0 0 5 0 0
> 2591 2589 0 0 371261 18
> size IN: hit tot mc no_rt bcast madst masrc OUT: hit tot mc
> GC: tot ignored goal_miss ovrf HASH: in_search out_search
> 63998 86141 2640 0 0 0 0 0 4 0
> 537114631 2640 2638 0 0 386452 15
> 65294 85989 2527 0 0 0 0 0 5 1
> 1610369017 2528 2526 0 0 379801 26
> 59241 84305 2662 0 0 0 0 0 7 0 0
> 2662 2660 0 0 371925 35
> 61194 85397 2837 0 0 0 0 0 3 0 0
> 2837 2835 0 0 374788 17
> 62908 88133 2688 0 0 1 0 0 4 1 0
> 2689 2687 0 0 382411 19
> 64329 88495 2648 0 0 0 0 0 5 1 0
> 2649 2647 0 0 398763 30
> 65560 84994 2447 0 0 1 0 0 6 2 0
> 2449 2447 0 0 394745 33
> 59677 81478 2568 0 0 0 0 0 5 0 0
> 2569 2567 0 0 341439 22
> 61459 81225 2623 0 0 0 0 0 5 0 0
> 2623 2621 0 0 345996 30
> 63038 83660 2544 0 0 0 0 0 3 0 0
> 2544 2542 0 0 381567 21
> 64494 86321 2620 0 0 0 0 0 4 0 0
> 2620 2618 0 0 395278 19
> 65528 88754 2766 0 0 0 0 0 6 0 0
> 2765 2763 0 0 421075 34
> 59948 86443 2990 0 0 0 0 0 5 0 0
> 2990 2988 0 0 373605 29
> 61860 81912 2786 0 0 0 0 0 5 1 0
> 2787 2785 0 0 354678 34
> 63630 80399 2981 0 0 0 0 0 4 0 0
> 2979 2977 0 0 354983 32
> 65215 83600 2934 0 0 0 0 0 6 3 0
> 2937 2935 0 0 371497 41
> 59322 85583 2821 0 0 0 0 0 7 0 0
> 2822 2820 0 0 382896 40
> 61268 82602 2817 0 0 0 0 0 9 2 0
> 2819 2817 0 0 368078 54
> 62989 85336 2760 0 0 1 0 0 9 0 0
> 2760 2758 0 0 394212 39
> 64387 84861 2603 0 0 0 0 0 7 0 0
> 2602 2600 0 0 390852 36
> size IN: hit tot mc no_rt bcast madst masrc OUT: hit tot mc
> GC: tot ignored goal_miss ovrf HASH: in_search out_search
> 65681 82848 2625 0 0 0 0 0 8 0
> 537114631 2625 2623 0 0 370769 49
> 59923 82826 2685 0 0 0 0 0 6 0
> 1610369017 2685 2683 0 0 344094 41
> 61700 82705 2706 0 0 0 0 0 7 0 0
> 2706 2704 0 0 368474 39
> 63247 91042 2592 0 0 0 0 0 9 1 0
> 2593 2591 0 0 399393 57
> 64636 93050 2530 0 0 0 0 0 6 0 0
> 2530 2528 0 0 405903 24
> 58301 90662 2554 0 0 0 0 0 5 0 0
> 2555 2553 0 0 404296 39
> 60146 88405 2567 0 0 0 0 0 8 1 0
> 2567 2565 0 0 370227 49
> 61822 86417 2579 0 0 0 0 0 7 1 0
> 2580 2578 0 0 366323 43
> 63351 86517 2594 0 0 1 0 0 9 0 0
> 2594 2592 0 0 363340 51
> 64739 87879 2669 0 0 0 0 0 3 0 0
> 2670 2668 0 0 387580 26
> 58397 88297 2566 0 0 0 0 0 4 0 0
> 2566 2564 0 0 396980 34
> 60352 85039 2707 0 0 0 0 0 6 1 0
> 2709 2707 0 0 367790 47
> 62169 87083 2779 0 0 0 0 0 6 0 0
> 2775 2773 0 0 377163 31
> 63808 88700 2752 0 0 0 0 0 4 0 0
> 2751 2749 0 0 392830 28
> 65189 91242 2725 0 0 0 0 0 5 1 0
> 2727 2725 0 0 416871 31
> 59326 84893 2834 0 0 0 0 0 5 1 0
> 2828 2826 0 0 380811 28
> 61064 87351 2553 0 0 0 0 0 9 0 0
> 2552 2550 0 0 382053 37
> 62637 88761 2551 0 0 0 0 0 5 1 0
> 2553 2551 0 0 384974 29
>
> --
>
> --------------------------------------------------
> Jeremy M. Guthrie jeremy.guthrie@berbee.com
> Senior Network Engineer Phone: 608-298-1061
> Berbee Fax: 608-288-3007
> 5520 Research Park Drive NOC: 608-298-1102
> Madison, WI 53711
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-11 18:46 ` Robert Olsson
@ 2005-01-12 1:30 ` Jeremy M. Guthrie
2005-01-12 16:02 ` Robert Olsson
0 siblings, 1 reply; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-12 1:30 UTC (permalink / raw)
To: netdev; +Cc: Robert Olsson, Stephen Hemminger, Jesse Brandeburg
On Tuesday 11 January 2005 12:46 pm, Robert Olsson wrote:
> We give up here? Or you want to try another setting of route hash? It's
> beyond the scope of this subject. You have to reboot if so.
I am up for pushing it if you do not think it is a waste of time. Based on
what I am seeing it looks like I just need a faster CPU to do the work. My
goal would be to hit zero dropped packets w/ 10-15% CPU to spare but I fail
to see how that will happen on this box. Do you concur that it would be
highly unlikely I would be able to get that kind of performance increase?
> Jeremy M. Guthrie writes:
> > ./rtstat
> > size IN: hit tot mc no_rt bcast madst masrc OUT: hit tot
> > mc GC: tot ignored goal_miss ovrf HASH: in_search out_search
> > 63375 86092 2844 0 0 0 0 0 5 1
> > 1610369017 2846 2844 0 0 386039 17
> > 64870 84431 537031471 537492192 0 0 0 0 1
> > 0 0 2848 2846 0 0 386130 10
> > 58707 87889 2715 0 0 0 0 0 7 1
> > 0 2716 2714 0 0 384782 13
> > 60617 89993 2796 0 0 0 0 0 5 0
> > 0 2795 2793 0 0 384432 17
> > 62458 93221 2772 0 0 0 0 0 3 0
> > 0 2772 2770 0 0 399579 9
> > 63990 93792 2712 0 0 0 0 0 6 0
> > 0 2713 2711 0 0 422585 12
> > 65499 94235 2846 0 0 0 0 0 2 0
> > 0 2846 2844 0 0 424341 9
> > 59755 90992 2864 0 0 0 0 0 1 1
> > 0 2865 2863 0 0 385888 13
> > 61702 87681 2798 0 0 0 0 0 6 2
> > 0 2800 2798 0 0 383276 34
> > 63334 90557 2734 0 0 0 0 0 5 1
> > 0 2735 2733 0 0 422121 28
> > 64798 88178 2732 0 0 0 0 0 1 0
> > 0 2732 2730 0 0 410772 11
> > 58574 85910 2669 0 0 0 0 0 3 0
> > 0 2670 2668 0 0 393331 10
> > 60538 87238 2705 0 0 0 0 0 4 0
> > 0 2705 2703 0 0 382161 20
> > 62164 85738 2628 0 0 0 0 0 4 0
> > 0 2628 2626 0 0 373049 25
> > 63759 78752 2638 0 0 0 0 0 2 0
> > 0 2638 2636 0 0 348411 5
> > 65052 79832 2482 0 0 0 0 0 3 0
> > 0 2481 2479 0 0 365871 20
> > 58986 82611 2549 0 0 0 0 0 4 1
> > 0 2550 2548 0 0 366429 24
> > 60845 85340 2620 0 0 1 0 0 2 0
> > 0 2620 2618 0 0 369963 16
> > 62465 83122 2591 0 0 0 0 0 5 0
> > 0 2591 2589 0 0 371261 18
> > size IN: hit tot mc no_rt bcast madst masrc OUT: hit tot
> > mc GC: tot ignored goal_miss ovrf HASH: in_search out_search
> > 63998 86141 2640 0 0 0 0 0 4 0
> > 537114631 2640 2638 0 0 386452 15
> > 65294 85989 2527 0 0 0 0 0 5 1
> > 1610369017 2528 2526 0 0 379801 26
> > 59241 84305 2662 0 0 0 0 0 7 0
> > 0 2662 2660 0 0 371925 35
> > 61194 85397 2837 0 0 0 0 0 3 0
> > 0 2837 2835 0 0 374788 17
> > 62908 88133 2688 0 0 1 0 0 4 1
> > 0 2689 2687 0 0 382411 19
> > 64329 88495 2648 0 0 0 0 0 5 1
> > 0 2649 2647 0 0 398763 30
> > 65560 84994 2447 0 0 1 0 0 6 2
> > 0 2449 2447 0 0 394745 33
> > 59677 81478 2568 0 0 0 0 0 5 0
> > 0 2569 2567 0 0 341439 22
> > 61459 81225 2623 0 0 0 0 0 5 0
> > 0 2623 2621 0 0 345996 30
> > 63038 83660 2544 0 0 0 0 0 3 0
> > 0 2544 2542 0 0 381567 21
> > 64494 86321 2620 0 0 0 0 0 4 0
> > 0 2620 2618 0 0 395278 19
> > 65528 88754 2766 0 0 0 0 0 6 0
> > 0 2765 2763 0 0 421075 34
> > 59948 86443 2990 0 0 0 0 0 5 0
> > 0 2990 2988 0 0 373605 29
> > 61860 81912 2786 0 0 0 0 0 5 1
> > 0 2787 2785 0 0 354678 34
> > 63630 80399 2981 0 0 0 0 0 4 0
> > 0 2979 2977 0 0 354983 32
> > 65215 83600 2934 0 0 0 0 0 6 3
> > 0 2937 2935 0 0 371497 41
> > 59322 85583 2821 0 0 0 0 0 7 0
> > 0 2822 2820 0 0 382896 40
> > 61268 82602 2817 0 0 0 0 0 9 2
> > 0 2819 2817 0 0 368078 54
> > 62989 85336 2760 0 0 1 0 0 9 0
> > 0 2760 2758 0 0 394212 39
> > 64387 84861 2603 0 0 0 0 0 7 0
> > 0 2602 2600 0 0 390852 36
> > size IN: hit tot mc no_rt bcast madst masrc OUT: hit tot
> > mc GC: tot ignored goal_miss ovrf HASH: in_search out_search
> > 65681 82848 2625 0 0 0 0 0 8 0
> > 537114631 2625 2623 0 0 370769 49
> > 59923 82826 2685 0 0 0 0 0 6 0
> > 1610369017 2685 2683 0 0 344094 41
> > 61700 82705 2706 0 0 0 0 0 7 0
> > 0 2706 2704 0 0 368474 39
> > 63247 91042 2592 0 0 0 0 0 9 1
> > 0 2593 2591 0 0 399393 57
> > 64636 93050 2530 0 0 0 0 0 6 0
> > 0 2530 2528 0 0 405903 24
> > 58301 90662 2554 0 0 0 0 0 5 0
> > 0 2555 2553 0 0 404296 39
> > 60146 88405 2567 0 0 0 0 0 8 1
> > 0 2567 2565 0 0 370227 49
> > 61822 86417 2579 0 0 0 0 0 7 1
> > 0 2580 2578 0 0 366323 43
> > 63351 86517 2594 0 0 1 0 0 9 0
> > 0 2594 2592 0 0 363340 51
> > 64739 87879 2669 0 0 0 0 0 3 0
> > 0 2670 2668 0 0 387580 26
> > 58397 88297 2566 0 0 0 0 0 4 0
> > 0 2566 2564 0 0 396980 34
> > 60352 85039 2707 0 0 0 0 0 6 1
> > 0 2709 2707 0 0 367790 47
> > 62169 87083 2779 0 0 0 0 0 6 0
> > 0 2775 2773 0 0 377163 31
> > 63808 88700 2752 0 0 0 0 0 4 0
> > 0 2751 2749 0 0 392830 28
> > 65189 91242 2725 0 0 0 0 0 5 1
> > 0 2727 2725 0 0 416871 31
> > 59326 84893 2834 0 0 0 0 0 5 1
> > 0 2828 2826 0 0 380811 28
> > 61064 87351 2553 0 0 0 0 0 9 0
> > 0 2552 2550 0 0 382053 37
> > 62637 88761 2551 0 0 0 0 0 5 1
> > 0 2553 2551 0 0 384974 29
> >
> > --
> >
> > --------------------------------------------------
> > Jeremy M. Guthrie jeremy.guthrie@berbee.com
> > Senior Network Engineer Phone: 608-298-1061
> > Berbee Fax: 608-288-3007
> > 5520 Research Park Drive NOC: 608-298-1102
> > Madison, WI 53711
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-12 1:30 ` Jeremy M. Guthrie
@ 2005-01-12 16:02 ` Robert Olsson
0 siblings, 0 replies; 88+ messages in thread
From: Robert Olsson @ 2005-01-12 16:02 UTC (permalink / raw)
To: jeremy.guthrie; +Cc: netdev, Robert Olsson, Stephen Hemminger, Jesse Brandeburg
Jeremy M. Guthrie writes:
> I am up for pushing it if you do not think it is a waste of time. Based on
> what I am seeing it looks like I just need a faster CPU to do the work. My
> goal would be to hit zero dropped packets w/ 10-15% CPU to spare but I fail
> to see how that will happen on this box. Do you concur that it would be
> highly unlikely I would be able to get that kind of performance increase?
Well we can try to decrease some of the linear search in the route hash
once we understand why we see all the RX interrupts.
--ro
^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: V2.4 policy router operates faster/better than V2.6
2005-01-03 22:51 ` Stephen Hemminger
2005-01-03 22:56 ` Jeremy M. Guthrie
@ 2005-01-04 15:07 ` Jeremy M. Guthrie
1 sibling, 0 replies; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-04 15:07 UTC (permalink / raw)
To: netdev
[-- Attachment #1: Type: text/plain, Size: 1022 bytes --]
On Monday 03 January 2005 04:51 pm, Stephen Hemminger wrote:
> How many flows are going through the router? The neighbour cache
> can get to be a bottleneck. Perhaps Robert "the Router Man" Olssen can
> give some hints.
Tue Jan 4 08:59:12 CST 2005
58406 rt_cache
Tue Jan 4 08:59:48 CST 2005
60636 rt_cache
Tue Jan 4 09:00:34 CST 2005
63891 rt_cache
Tue Jan 4 09:01:02 CST 2005
64635 rt_cache
Tue Jan 4 09:01:29 CST 2005
65689 rt_cache
Tue Jan 4 09:01:58 CST 2005
63426 rt_cache
Tue Jan 4 09:02:25 CST 2005
64139 rt_cache
Tue Jan 4 09:02:53 CST 2005
63860 rt_cache
Tue Jan 4 09:03:20 CST 2005
65719 rt_cache
Tue Jan 4 09:03:48 CST 2005
62465 rt_cache
Tue Jan 4 09:04:15 CST 2005
39339 rt_cache
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 88+ messages in thread
[parent not found: <200501071619.54566.jeremy.guthrie@berbee.com>]
* Re: V2.4 policy router operates faster/better than V2.6
[not found] <200501071619.54566.jeremy.guthrie@berbee.com>
@ 2005-01-07 23:23 ` Jesse Brandeburg
2005-01-10 21:11 ` Jeremy M. Guthrie
0 siblings, 1 reply; 88+ messages in thread
From: Jesse Brandeburg @ 2005-01-07 23:23 UTC (permalink / raw)
To: Jeremy M. Guthrie; +Cc: Brandeburg, Jesse, netdev, Robert.Olsson, shemminger
On Fri, 7 Jan 2005, Jeremy M. Guthrie wrote:
> > Very interesting that with your workload the napi driver doesn't keep up
> > (from looking at your posts in netdev) and yet the interrupt mode driver
> > can.
> Well, I need to do more digging. One scenario the interrupt mode can hand
> stuff off to the CPU but the network stack still drops. The other scenario,
> the card drops.
>
> ethtool -S eth2
> NIC statistics:
<snip!>
> tx_packets: 314550698
> tx_bytes: 4290523139
> ethtool -S eth3
> NIC statistics:
<snip!>
> rx_packets: 719847127
> tx_packets: 5
> rx_bytes: 1880301945
> tx_bytes: 398
> rx_errors: 3368295
> rx_dropped: 1478044
> rx_fifo_errors: 1890251
> rx_missed_errors: 1890251
> rx_long_byte_count: 379837423993
> rx_csum_offload_good: 672891990
> rx_csum_offload_errors: 178666
Okay, well, rx_dropped means "receive no buffers count" in our driver, but
doesn't necessarily mean that the packet was completely dropped it just
means that the fifo may have had to queue up the packet on the adapter as
best it could and wait for the OS to provide more skb's, this may or may
not lead to further errors...
Now, the rx_fifo errors is from hardware counter "missed packet count"
which means that a packet was dropped because the fifo was full (probably
indicated at least some of the time because the card was in "receive no
buffers" state) btw fifo errors and rx_missed are both being fed by the
same hardware counter.
rx_csum_offload_errors is somewhat worrisome because it means that you're
receiving packets that appear to be corrupted. This could be from any
number of sources, but is most likely a misconfigured station on your lan
or something is corrupting checksums (a tcpdump would help debug here, but
would really slow things down) The packets are NOT dropped, but handed to
the stack to decide what to do.
So, to mitigate the rnbc "receive no buffers count" a little you can
provide some more buffering on the receive side by loading the module with
RxDescriptors=2048 or something larger than the default of 256. this will
help (if you haven't already) but will also probably increase the amount
of work your system has to do, as more packets will be able to be stored
up.
^ permalink raw reply [flat|nested] 88+ messages in thread* Re: V2.4 policy router operates faster/better than V2.6
2005-01-07 23:23 ` Jesse Brandeburg
@ 2005-01-10 21:11 ` Jeremy M. Guthrie
0 siblings, 0 replies; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-10 21:11 UTC (permalink / raw)
To: netdev; +Cc: Jesse Brandeburg, Robert.Olsson, shemminger
[-- Attachment #1: Type: text/plain, Size: 6206 bytes --]
I rebound the card with 2048 RxDescriptors. There appears to be a burst
errors right when the port comes online. Down below is an ifconfig ; sleep
60 ; ifconfig.
I'll be tesitng Robert's patch shortly.
ethtool -S eth2
NIC statistics:
rx_packets: 0
tx_packets: 7295976
rx_bytes: 0
tx_bytes: 3413882143
rx_errors: 0
tx_errors: 0
rx_dropped: 0
tx_dropped: 0
multicast: 0
collisions: 0
rx_length_errors: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 0
rx_missed_errors: 0
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_window_errors: 0
tx_abort_late_coll: 0
tx_deferred_ok: 0
tx_single_coll_ok: 0
tx_multi_coll_ok: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
rx_align_errors: 0
tx_tcp_seg_good: 0
tx_tcp_seg_failed: 0
rx_flow_control_xon: 0
rx_flow_control_xoff: 0
tx_flow_control_xon: 0
tx_flow_control_xoff: 0
rx_long_byte_count: 0
rx_csum_offload_good: 0
rx_csum_offload_errors: 0
ethtool -S eth3
NIC statistics:
rx_packets: 19231078
tx_packets: 5
rx_bytes: 1350479823
tx_bytes: 398
rx_errors: 522159
tx_errors: 0
rx_dropped: 320198
tx_dropped: 0
multicast: 0
collisions: 0
rx_length_errors: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 201961
rx_missed_errors: 201961
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_window_errors: 0
tx_abort_late_coll: 0
tx_deferred_ok: 0
tx_single_coll_ok: 0
tx_multi_coll_ok: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
rx_align_errors: 0
tx_tcp_seg_good: 0
tx_tcp_seg_failed: 0
rx_flow_control_xon: 0
rx_flow_control_xoff: 0
tx_flow_control_xon: 0
tx_flow_control_xoff: 0
rx_long_byte_count: 9940414415
rx_csum_offload_good: 17879204
rx_csum_offload_errors: 8537
Mon Jan 10 15:06:26 CST 2005
eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
inet addr:10.253.0.1 Bcast:10.255.255.255 Mask:255.255.255.0
inet6 addr: fe80::202:b3ff:fed5:7e30/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:67183383 errors:583215 dropped:583215 overruns:247258
frame:0
TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1120957705 (1069.0 Mb) TX bytes:398 (398.0 b)
Base address:0x22a0 Memory:eff80000-effa0000
Mon Jan 10 15:07:26 CST 2005
eth3 Link encap:Ethernet HWaddr 00:02:B3:D5:7E:30
inet addr:10.253.0.1 Bcast:10.255.255.255 Mask:255.255.255.0
inet6 addr: fe80::202:b3ff:fed5:7e30/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:73275567 errors:616520 dropped:616520 overruns:262517
frame:0
TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:337437991 (321.8 Mb) TX bytes:398 (398.0 b)
Base address:0x22a0 Memory:eff80000-effa0000
On Friday 07 January 2005 05:23 pm, Jesse Brandeburg wrote:
> On Fri, 7 Jan 2005, Jeremy M. Guthrie wrote:
> > > Very interesting that with your workload the napi driver doesn't keep
> > > up (from looking at your posts in netdev) and yet the interrupt mode
> > > driver can.
> >
> > Well, I need to do more digging. One scenario the interrupt mode can
> > hand stuff off to the CPU but the network stack still drops. The other
> > scenario, the card drops.
> >
> > ethtool -S eth2
> > NIC statistics:
>
> <snip!>
>
> > tx_packets: 314550698
> > tx_bytes: 4290523139
> > ethtool -S eth3
> > NIC statistics:
>
> <snip!>
>
> > rx_packets: 719847127
> > tx_packets: 5
> > rx_bytes: 1880301945
> > tx_bytes: 398
> > rx_errors: 3368295
> > rx_dropped: 1478044
> > rx_fifo_errors: 1890251
> > rx_missed_errors: 1890251
> > rx_long_byte_count: 379837423993
> > rx_csum_offload_good: 672891990
> > rx_csum_offload_errors: 178666
>
> Okay, well, rx_dropped means "receive no buffers count" in our driver, but
> doesn't necessarily mean that the packet was completely dropped it just
> means that the fifo may have had to queue up the packet on the adapter as
> best it could and wait for the OS to provide more skb's, this may or may
> not lead to further errors...
>
> Now, the rx_fifo errors is from hardware counter "missed packet count"
> which means that a packet was dropped because the fifo was full (probably
> indicated at least some of the time because the card was in "receive no
> buffers" state) btw fifo errors and rx_missed are both being fed by the
> same hardware counter.
>
> rx_csum_offload_errors is somewhat worrisome because it means that you're
> receiving packets that appear to be corrupted. This could be from any
> number of sources, but is most likely a misconfigured station on your lan
> or something is corrupting checksums (a tcpdump would help debug here, but
> would really slow things down) The packets are NOT dropped, but handed to
> the stack to decide what to do.
>
> So, to mitigate the rnbc "receive no buffers count" a little you can
> provide some more buffering on the receive side by loading the module with
> RxDescriptors=2048 or something larger than the default of 256. this will
> help (if you haven't already) but will also probably increase the amount
> of work your system has to do, as more packets will be able to be stored
> up.
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 88+ messages in thread
[parent not found: <C925F8B43D79CC49ACD0601FB68FF50C02D39006@orsmsx408>]
* Re: V2.4 policy router operates faster/better than V2.6
[not found] <C925F8B43D79CC49ACD0601FB68FF50C02D39006@orsmsx408>
@ 2005-01-13 22:55 ` Jeremy M. Guthrie
0 siblings, 0 replies; 88+ messages in thread
From: Jeremy M. Guthrie @ 2005-01-13 22:55 UTC (permalink / raw)
To: netdev; +Cc: Brandeburg, Jesse, Robert Olsson
[-- Attachment #1: Type: text/plain, Size: 5387 bytes --]
Thu Jan 13 16:50:06 CST 2005 entries: 0000cc39 Packets: 1127110 Errors:
264136 PPS: 75140 Percentage: 23.43%
Thu Jan 13 16:50:22 CST 2005 entries: 000142c0 Packets: 1148930 Errors:
743 PPS: 76595 Percentage: 0.6%
Thu Jan 13 16:50:37 CST 2005 entries: 0001aa91 Packets: 1158591 Errors:
116 PPS: 77239 Percentage: 0.1%
Thu Jan 13 16:50:52 CST 2005 entries: 00021146 Packets: 1192241 Errors:
11648 PPS: 79482 Percentage: 0.97%
Thu Jan 13 16:51:07 CST 2005 entries: 00025c42 Packets: 1227489 Errors:
1056 PPS: 81832 Percentage: 0.8%
Thu Jan 13 16:51:22 CST 2005 entries: 00029ca0 Packets: 1217954 Errors:
365 PPS: 81196 Percentage: 0.2%
ethtool -S eth3
NIC statistics:
rx_packets: 8778549
tx_packets: 5
rx_bytes: 327267728
tx_bytes: 398
rx_errors: 575319
tx_errors: 0
rx_dropped: 360028
tx_dropped: 0
multicast: 0
collisions: 0
rx_length_errors: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 215291
rx_missed_errors: 215291
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_window_errors: 0
tx_abort_late_coll: 0
tx_deferred_ok: 0
tx_single_coll_ok: 0
tx_multi_coll_ok: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
rx_align_errors: 0
tx_tcp_seg_good: 0
tx_tcp_seg_failed: 0
rx_flow_control_xon: 0
rx_flow_control_xoff: 0
tx_flow_control_xon: 0
tx_flow_control_xoff: 0
rx_long_byte_count: 4622235024
rx_csum_offload_good: 8285347
int_tx_desc: 5
int_tx_queueempty: 6
int_link_state: 1
int_rx_frame_err: 0
int_rx_desc_min_thresh: 1330
int_rx_fifo_ovr: 47
int_rx_timer: 1768171
int_mdio: 0
int_rxcfg: 1
int_gpio_pins: 0
rx_csum_offload_errors: 698
On Wednesday 12 January 2005 06:49 pm, Brandeburg, Jesse wrote:
> I didn't send this to netdev.... if the interrupt counting code does
> something good then we can publish it.
>
> Jeremy, I would agree a faster CPU is going to help you handle more
> traffic. I can't speak to the routing thing. Your test would be very
> interesting if we could set up something similar here, unfortunately
> we're mostly interested in network device performance and not so much on
> kernel policy routing. I personally would be interested in having
> something set up to "play" with the driver on, but it may be doubtful
> how much time I would get to spend on it.
>
> Anyway, here is a driver that counts interrupt sources, you can get the
> counts from
> ethtool -S eth3
>
> you'll need to compile it like so:
> make CFLAGS_EXTRA=-DE1000_COUNT_ICR
>
> any messages in /var/log/messages from the network stack? (I just saw
> your netdev email about dst cache overflow) This driver has what we
> think should be the correct napi code in e1000_clean. If robert's fix
> works better for you then stick with it, and let me know cause what I'm
> sending you now is what we're going forward with unless we hear about
> problems.
>
> If you want to chat over an instant messenger of some kind here is my
> info:
> Aim: jbrandeb
> msn: go_jesse@hotmail.com
> yahoo: go_jesse
>
> I appreciate your patience as we try different stuff. I know I'm poking
> at the driver a lot, but the high interrupt counts seem a little weird
> given the load of your system.
>
> jesse
>
> -----Original Message-----
> From: Jeremy M. Guthrie [mailto:jeremy.guthrie@berbee.com]
> Sent: Wednesday, January 12, 2005 2:31 PM
> To: netdev@oss.sgi.com
> Cc: Robert Olsson; Stephen Hemminger; Brandeburg, Jesse
> Subject: Re: V2.4 policy router operates faster/better than V2.6
>
> On Wednesday 12 January 2005 04:22 pm, Robert Olsson wrote:
> > Jeremy M. Guthrie writes:
> > > I am now getting some push back from the project manager on this
> > > performance problem. I am wondering if you think faster CPUs will
> > > a) help relieve the symptoms of this problem
> > > b) not help because now we will hit a '# of routes in the
>
> route-cache'
>
> > > problem
> > > c) or will help to a point till the # interrupts come back and
>
> bite us.
>
> > Back out the patch I sent.and have hardirq's to run RX-softirq as you
> > did before but something is very wrong. You didn't answer if there
>
> were
>
> > other load on the machine...
>
> I have backed out. As for the load, this box only does policy routing.
> Any
> other functions it performs are part of its automated system to download
> the
> next days policy-routing config.
>
> > route-cache can probably be tuned you as have four times the linear
>
> seach
>
> > I see in one PIII system at 110 kpps w. production traffic.
>
> How would I go about tuning that?
>
> > Of course the non-engineering solution is to buy more CPU... :-)
>
> That is good to know. This will help me calm the situation a bit. 8)
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie@berbee.com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 88+ messages in thread
end of thread, other threads:[~2005-01-31 18:06 UTC | newest]
Thread overview: 88+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2005-01-03 20:55 V2.4 policy router operates faster/better than V2.6 Jeremy M. Guthrie
2005-01-03 22:51 ` Stephen Hemminger
2005-01-03 22:56 ` Jeremy M. Guthrie
2005-01-05 13:18 ` Robert Olsson
2005-01-05 15:18 ` Jeremy M. Guthrie
2005-01-05 16:30 ` Robert Olsson
2005-01-05 17:35 ` Jeremy M. Guthrie
2005-01-05 19:25 ` Jeremy M. Guthrie
2005-01-05 20:22 ` Robert Olsson
2005-01-05 20:52 ` Jeremy M. Guthrie
2005-01-06 15:26 ` Jeremy M. Guthrie
2005-01-06 18:15 ` Robert Olsson
2005-01-06 19:35 ` Jeremy M. Guthrie
2005-01-06 20:29 ` Robert Olsson
2005-01-06 20:54 ` Jeremy M. Guthrie
2005-01-06 20:55 ` Jeremy M. Guthrie
2005-01-06 21:19 ` Jeremy M. Guthrie
2005-01-06 21:36 ` Robert Olsson
2005-01-06 21:46 ` Jeremy M. Guthrie
2005-01-06 22:11 ` Robert Olsson
2005-01-06 22:18 ` Jeremy M. Guthrie
2005-01-06 22:35 ` Robert Olsson
2005-01-07 16:17 ` Jeremy M. Guthrie
2005-01-07 19:18 ` Robert Olsson
2005-01-07 19:38 ` Jeremy M. Guthrie
2005-01-07 20:07 ` Robert Olsson
2005-01-07 20:14 ` Jeremy M. Guthrie
2005-01-07 20:40 ` Robert Olsson
2005-01-07 21:06 ` Jeremy M. Guthrie
2005-01-07 21:30 ` Robert Olsson
2005-01-11 15:11 ` Jeremy M. Guthrie
2005-01-07 22:28 ` Jesse Brandeburg
2005-01-07 22:50 ` Jeremy M. Guthrie
2005-01-07 22:57 ` Stephen Hemminger
2005-01-11 15:17 ` Jeremy M. Guthrie
2005-01-11 16:40 ` Robert Olsson
2005-01-12 1:27 ` Jeremy M. Guthrie
2005-01-12 15:11 ` Robert Olsson
2005-01-12 16:24 ` Jeremy M. Guthrie
2005-01-12 19:27 ` Robert Olsson
2005-01-12 20:11 ` Jeremy M. Guthrie
2005-01-12 20:21 ` Robert Olsson
2005-01-12 20:30 ` Jeremy M. Guthrie
2005-01-12 20:45 ` Jeremy M. Guthrie
2005-01-12 22:02 ` Robert Olsson
2005-01-12 22:21 ` Jeremy M. Guthrie
[not found] ` <16869.42247.126428.508479@robur.slu.se>
2005-01-12 22:42 ` Jeremy M. Guthrie
2005-01-12 22:47 ` Jeremy M. Guthrie
2005-01-12 23:19 ` Robert Olsson
2005-01-12 23:23 ` Jeremy M. Guthrie
2005-01-13 8:56 ` Robert Olsson
2005-01-13 19:28 ` Jeremy M. Guthrie
2005-01-13 20:00 ` David S. Miller
2005-01-13 20:43 ` Jeremy M. Guthrie
2005-01-13 23:13 ` David S. Miller
2005-01-13 21:12 ` Robert Olsson
2005-01-13 22:27 ` Jeremy M. Guthrie
2005-01-14 15:44 ` Robert Olsson
2005-01-14 14:59 ` Jeremy M. Guthrie
2005-01-14 16:05 ` Robert Olsson
2005-01-14 19:00 ` Jeremy M. Guthrie
2005-01-14 19:26 ` Jeremy M. Guthrie
2005-01-16 12:32 ` Robert Olsson
2005-01-16 16:22 ` Jeremy M. Guthrie
2005-01-19 15:03 ` Jeremy M. Guthrie
2005-01-19 22:18 ` Robert Olsson
2005-01-20 1:50 ` Jeremy M. Guthrie
2005-01-20 11:30 ` Robert Olsson
2005-01-20 14:37 ` Jeremy M. Guthrie
2005-01-20 17:01 ` Robert Olsson
2005-01-20 17:14 ` Jeremy M. Guthrie
2005-01-20 21:53 ` Robert Olsson
2005-01-21 21:20 ` Jeremy M. Guthrie
2005-01-21 15:23 ` Robert Olsson
2005-01-21 21:24 ` Jeremy M. Guthrie
2005-01-31 15:37 ` Jeremy M. Guthrie
2005-01-31 18:06 ` Robert Olsson
2005-01-12 22:05 ` Jeremy M. Guthrie
2005-01-12 22:22 ` Robert Olsson
2005-01-12 22:30 ` Jeremy M. Guthrie
2005-01-11 17:17 ` Jeremy M. Guthrie
2005-01-11 18:46 ` Robert Olsson
2005-01-12 1:30 ` Jeremy M. Guthrie
2005-01-12 16:02 ` Robert Olsson
2005-01-04 15:07 ` Jeremy M. Guthrie
[not found] <200501071619.54566.jeremy.guthrie@berbee.com>
2005-01-07 23:23 ` Jesse Brandeburg
2005-01-10 21:11 ` Jeremy M. Guthrie
[not found] <C925F8B43D79CC49ACD0601FB68FF50C02D39006@orsmsx408>
2005-01-13 22:55 ` Jeremy M. Guthrie
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).