* Re: Packet time delays on multi-core systems
[not found] <20100929191851.GC86786@beaver.vrungel.ru>
@ 2010-09-29 21:45 ` Eric Dumazet
2010-09-30 6:24 ` Alexey Vlasov
2010-09-30 12:30 ` Alexey Vlasov
0 siblings, 2 replies; 16+ messages in thread
From: Eric Dumazet @ 2010-09-29 21:45 UTC (permalink / raw)
To: Alexey Vlasov; +Cc: Linux Kernel Mailing List, netdev
Le mercredi 29 septembre 2010 à 23:18 +0400, Alexey Vlasov a écrit :
> Hi.
>
> I'm not sure actually that I should write here, may be I should ask in
> netfilter maillist, but if is something wrong please correct me.
>
CC netdev
> I've got rather large linux shared hosting, and on my new servers I
> noticed some strange singularity, that this simple rule:
>
> # iptables -A OUTPUT -p tcp -m tcp --dport 80 --tcp-flags
> FIN,SYN,RST,ACK SYN -j LOG --log-prefix "ipsec:SYN-OUTPUT "
> --log-uid
>
> gives essential time delays simply at ping from the adjacent server
> on a local area network. I don't know precisely what's wrong whether the
> reason is in the bad support by a kernel of new hardware, or it concerns
> generally the new kernel, but now it leads to the situation that even at simple
> DDOS attacks to client sites, it becomes difficult to make something, and in
> general all works only worse.
>
> It seems to me that with the increase of CPU cores' amount, it only becomes
> worse and worse, and, obviously, iptables uses resources of only one processor,
> which resources to it for any reason doesn't suffice.
>
Its not true. iptables can run on all cpus in //
> newbox # iptables -F
> otherbox # ping -c 100 newbox
> ...
> 100 packets transmitted, 100 received, 0% packet loss, time 100044ms
> rtt min/avg/max/mdev = 0.133/2.637/17.172/3.736 ms
>
> OK.
>
> newbox # iptables -A OUTPUT -p tcp -m tcp --dport 80 --tcp-flags FIN,SYN,RST,ACK SYN
> -j LOG --log-prefix "ipsec:SYN-OUTPUT " --log-uid
> otherbox # ping -c 100 newbox
> ...
> 64 bytes from (newbox): icmp_seq=3 ttl=64 time=1.58 ms
> 64 bytes from (newbox): icmp_seq=4 ttl=64 time=98.7 ms
> 64 bytes from (newbox): icmp_seq=5 ttl=64 time=18.2 ms
> 64 bytes from (newbox): icmp_seq=6 ttl=64 time=6.13 ms
> 64 bytes from (newbox): icmp_seq=7 ttl=64 time=108 ms
> ...
> 64 bytes from (newbox): icmp_seq=55 ttl=64 time=2.30 ms
> 64 bytes from (newbox): icmp_seq=56 ttl=64 time=59.9 ms
> 64 bytes from (newbox): icmp_seq=57 ttl=64 time=0.155 ms
> ...
> 64 bytes from (newbox): icmp_seq=61 ttl=64 time=13.4 ms
> 64 bytes from (newbox): icmp_seq=62 ttl=64 time=55.0 ms
> 64 bytes from (newbox): icmp_seq=63 ttl=64 time=0.233 ms
> ...
> 100 packets transmitted, 100 received, 0% packet loss, time 99957ms
> rtt min/avg/max/mdev = 0.111/7.519/108.061/18.478 ms
>
> newbox # iptables -L -v -n
> Chain INPUT (policy ACCEPT 346K packets, 213M bytes)
> pkts bytes target prot opt in out source destination
>
> Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
> pkts bytes target prot opt in out source destination
>
> Chain OUTPUT (policy ACCEPT 296K packets, 290M bytes)
> pkts bytes target prot opt in out source destination
> 234 14040 LOG tcp -- * * 0.0.0.0/0 0.0.0.0/0
> tcp dpt:80 flags:0x17/0x02 LOG flags 8 level 4 prefix `ipsec:SYN-OUTPUT- '
>
> My old server: Intel SR1500, Xeon 5430, kernel 2.6.24 - 2.6.28
> Newbox: SR1620UR, 5650, kernel 2.6.32
>
> Thanks in advance.
>
Seems strange indeed, since the LOG you add should not slowdown icmp
trafic that much.
But if you send SYN packets in the same time, (logged), this might slow
down the reception (and answers) of ICMP frames. LOG target can be quite
expensive...
Is using other rules gives same problem ?
iptables -A OUTPUT -p tcp -m tcp --dport 80 --tcp-flags FIN,SYN,RST,ACK SYN
iptables -A OUTPUT -p tcp -m tcp --dport 80 --tcp-flags FIN,SYN,RST,ACK SYN
iptables -A OUTPUT -p tcp -m tcp --dport 80 --tcp-flags FIN,SYN,RST,ACK SYN
iptables -A OUTPUT -p tcp -m tcp --dport 80 --tcp-flags FIN,SYN,RST,ACK SYN
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Packet time delays on multi-core systems
2010-09-29 21:45 ` Packet time delays on multi-core systems Eric Dumazet
@ 2010-09-30 6:24 ` Alexey Vlasov
2010-09-30 6:33 ` Eric Dumazet
2010-09-30 12:30 ` Alexey Vlasov
1 sibling, 1 reply; 16+ messages in thread
From: Alexey Vlasov @ 2010-09-30 6:24 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Linux Kernel Mailing List, netdev
Here I found some dude with the same problem:
http://lkml.org/lkml/2010/7/9/340
On Wed, Sep 29, 2010 at 11:45:21PM +0200, Eric Dumazet wrote:
> Le mercredi 29 septembre 2010 ?? 23:18 +0400, Alexey Vlasov a ??crit :
> > Hi.
> >
> > I'm not sure actually that I should write here, may be I should ask in
> > netfilter maillist, but if is something wrong please correct me.
> >
>
> CC netdev
>
>
> > I've got rather large linux shared hosting, and on my new servers I
> > noticed some strange singularity, that this simple rule:
> >
> > # iptables -A OUTPUT -p tcp -m tcp --dport 80 --tcp-flags
> > FIN,SYN,RST,ACK SYN -j LOG --log-prefix "ipsec:SYN-OUTPUT "
> > --log-uid
> >
> > gives essential time delays simply at ping from the adjacent server
> > on a local area network. I don't know precisely what's wrong whether the
> > reason is in the bad support by a kernel of new hardware, or it concerns
> > generally the new kernel, but now it leads to the situation that even at simple
> > DDOS attacks to client sites, it becomes difficult to make something, and in
> > general all works only worse.
> >
> > It seems to me that with the increase of CPU cores' amount, it only becomes
> > worse and worse, and, obviously, iptables uses resources of only one processor,
> > which resources to it for any reason doesn't suffice.
> >
>
> Its not true. iptables can run on all cpus in //
>
> > newbox # iptables -F
> > otherbox # ping -c 100 newbox
> > ...
> > 100 packets transmitted, 100 received, 0% packet loss, time 100044ms
> > rtt min/avg/max/mdev = 0.133/2.637/17.172/3.736 ms
> >
> > OK.
> >
> > newbox # iptables -A OUTPUT -p tcp -m tcp --dport 80 --tcp-flags FIN,SYN,RST,ACK SYN
> > -j LOG --log-prefix "ipsec:SYN-OUTPUT " --log-uid
> > otherbox # ping -c 100 newbox
> > ...
> > 64 bytes from (newbox): icmp_seq=3 ttl=64 time=1.58 ms
> > 64 bytes from (newbox): icmp_seq=4 ttl=64 time=98.7 ms
> > 64 bytes from (newbox): icmp_seq=5 ttl=64 time=18.2 ms
> > 64 bytes from (newbox): icmp_seq=6 ttl=64 time=6.13 ms
> > 64 bytes from (newbox): icmp_seq=7 ttl=64 time=108 ms
> > ...
> > 64 bytes from (newbox): icmp_seq=55 ttl=64 time=2.30 ms
> > 64 bytes from (newbox): icmp_seq=56 ttl=64 time=59.9 ms
> > 64 bytes from (newbox): icmp_seq=57 ttl=64 time=0.155 ms
> > ...
> > 64 bytes from (newbox): icmp_seq=61 ttl=64 time=13.4 ms
> > 64 bytes from (newbox): icmp_seq=62 ttl=64 time=55.0 ms
> > 64 bytes from (newbox): icmp_seq=63 ttl=64 time=0.233 ms
> > ...
> > 100 packets transmitted, 100 received, 0% packet loss, time 99957ms
> > rtt min/avg/max/mdev = 0.111/7.519/108.061/18.478 ms
> >
> > newbox # iptables -L -v -n
> > Chain INPUT (policy ACCEPT 346K packets, 213M bytes)
> > pkts bytes target prot opt in out source destination
> >
> > Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
> > pkts bytes target prot opt in out source destination
> >
> > Chain OUTPUT (policy ACCEPT 296K packets, 290M bytes)
> > pkts bytes target prot opt in out source destination
> > 234 14040 LOG tcp -- * * 0.0.0.0/0 0.0.0.0/0
> > tcp dpt:80 flags:0x17/0x02 LOG flags 8 level 4 prefix `ipsec:SYN-OUTPUT- '
> >
> > My old server: Intel SR1500, Xeon 5430, kernel 2.6.24 - 2.6.28
> > Newbox: SR1620UR, 5650, kernel 2.6.32
> >
> > Thanks in advance.
> >
>
> Seems strange indeed, since the LOG you add should not slowdown icmp
> trafic that much.
>
> But if you send SYN packets in the same time, (logged), this might slow
> down the reception (and answers) of ICMP frames. LOG target can be quite
> expensive...
>
> Is using other rules gives same problem ?
>
> iptables -A OUTPUT -p tcp -m tcp --dport 80 --tcp-flags FIN,SYN,RST,ACK SYN
> iptables -A OUTPUT -p tcp -m tcp --dport 80 --tcp-flags FIN,SYN,RST,ACK SYN
> iptables -A OUTPUT -p tcp -m tcp --dport 80 --tcp-flags FIN,SYN,RST,ACK SYN
> iptables -A OUTPUT -p tcp -m tcp --dport 80 --tcp-flags FIN,SYN,RST,ACK SYN
>
>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
--
BRGDS. Alexey Vlasov.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Packet time delays on multi-core systems
2010-09-30 6:24 ` Alexey Vlasov
@ 2010-09-30 6:33 ` Eric Dumazet
2010-09-30 12:23 ` Alexey Vlasov
0 siblings, 1 reply; 16+ messages in thread
From: Eric Dumazet @ 2010-09-30 6:33 UTC (permalink / raw)
To: Alexey Vlasov; +Cc: Linux Kernel Mailing List, netdev
Le jeudi 30 septembre 2010 à 10:24 +0400, Alexey Vlasov a écrit :
> Here I found some dude with the same problem:
> http://lkml.org/lkml/2010/7/9/340
>
In your opinion its the same problem.
But the description you gave is completely different.
You have time skew only when activating a particular iptables rule.
No ?
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Packet time delays on multi-core systems
2010-09-30 6:33 ` Eric Dumazet
@ 2010-09-30 12:23 ` Alexey Vlasov
2010-09-30 12:44 ` Eric Dumazet
0 siblings, 1 reply; 16+ messages in thread
From: Alexey Vlasov @ 2010-09-30 12:23 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Linux Kernel Mailing List, netdev
On Thu, Sep 30, 2010 at 08:33:52AM +0200, Eric Dumazet wrote:
> Le jeudi 30 septembre 2010 ?? 10:24 +0400, Alexey Vlasov a ??crit :
> > Here I found some dude with the same problem:
> > http://lkml.org/lkml/2010/7/9/340
> >
>
> In your opinion its the same problem.
>
> But the description you gave is completely different.
>
> You have time skew only when activating a particular iptables rule.
Well I put interrups from NIC, namely tx/rx query, to different
processors and got normal pings by adding LOG rule.
I also found that overruns is constantly growing, I don't know if these are connected.
RX packets:2831439546 errors:0 dropped:134726 overruns:947671733 frame:0
TX packets:2880849825 errors:0 dropped:0 overruns:0 carrier:0
Rather strange that only one processor was involved, even in top was
clear that ksoftirqd eats the first processor up to 100%.
Here goes the typical distribution of interrups on new servers:
CPU0 CPU1 CPU2 CPU3 ... CPU23
752: 11 0 0 0 ... 0 PCI-MSI-edge eth0
753: 2799366721 0 0 0 ... 0 PCI-MSI-edge eth0-rx3
754: 2821840553 0 0 0 ... 0 PCI-MSI-edge eth0-rx2
755: 2786117044 0 0 0 ... 0 PCI-MSI-edge eth0-rx1
756: 2896099336 0 0 0 ... 0 PCI-MSI-edge eth0-rx0
757: 1808404680 0 0 0 ... 0 PCI-MSI-edge eth0-tx3
758: 1797855130 0 0 0 ... 0 PCI-MSI-edge eth0-tx2
759: 1807222032 0 0 0 ... 0 PCI-MSI-edge eth0-tx1
760: 1820309360 0 0 0 ... 0 PCI-MSI-edge eth0-tx0
On the old ones:
CPU0 CPU1 CPU2 ... CPU8
502: 522320256 522384039 522327386 ... 522380267 PCI-MSI-edge eth0
--
BRGDS. Alexey Vlasov.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Packet time delays on multi-core systems
2010-09-29 21:45 ` Packet time delays on multi-core systems Eric Dumazet
2010-09-30 6:24 ` Alexey Vlasov
@ 2010-09-30 12:30 ` Alexey Vlasov
2010-09-30 12:46 ` Eric Dumazet
1 sibling, 1 reply; 16+ messages in thread
From: Alexey Vlasov @ 2010-09-30 12:30 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Linux Kernel Mailing List, netdev
On Wed, Sep 29, 2010 at 11:45:21PM +0200, Eric Dumazet wrote:
> But if you send SYN packets in the same time, (logged), this might
> slow
> down the reception (and answers) of ICMP frames. LOG target can be
> quite
> expensive...
Yes, it's clear that some slow down can appear, but 100 ms is too much,
and this happens at 200 SYN packets in 2 minutes just as in my example.
On old servers where some tx/rx are missing in NIC card I don't see
such a situation even at more then 1000 SYN-packets per sec.
> Is using other rules gives same problem ?
>
> iptables -A OUTPUT -p tcp -m tcp --dport 80 --tcp-flags
No, only LOG gives such a scheme.
--
BRGDS. Alexey Vlasov.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Packet time delays on multi-core systems
2010-09-30 12:23 ` Alexey Vlasov
@ 2010-09-30 12:44 ` Eric Dumazet
2010-09-30 17:37 ` Alexey Vlasov
0 siblings, 1 reply; 16+ messages in thread
From: Eric Dumazet @ 2010-09-30 12:44 UTC (permalink / raw)
To: Alexey Vlasov; +Cc: Linux Kernel Mailing List, netdev
Le jeudi 30 septembre 2010 à 16:23 +0400, Alexey Vlasov a écrit :
> On Thu, Sep 30, 2010 at 08:33:52AM +0200, Eric Dumazet wrote:
> > Le jeudi 30 septembre 2010 ?? 10:24 +0400, Alexey Vlasov a ??crit :
> > > Here I found some dude with the same problem:
> > > http://lkml.org/lkml/2010/7/9/340
> > >
> >
> > In your opinion its the same problem.
> >
> > But the description you gave is completely different.
> >
> > You have time skew only when activating a particular iptables rule.
>
> Well I put interrups from NIC, namely tx/rx query, to different
> processors and got normal pings by adding LOG rule.
>
> I also found that overruns is constantly growing, I don't know if these are connected.
> RX packets:2831439546 errors:0 dropped:134726 overruns:947671733 frame:0
> TX packets:2880849825 errors:0 dropped:0 overruns:0 carrier:0
>
> Rather strange that only one processor was involved, even in top was
> clear that ksoftirqd eats the first processor up to 100%.
>
OK, because only CPU0 gets interrupts of all queues.
> Here goes the typical distribution of interrups on new servers:
> CPU0 CPU1 CPU2 CPU3 ... CPU23
> 752: 11 0 0 0 ... 0 PCI-MSI-edge eth0
> 753: 2799366721 0 0 0 ... 0 PCI-MSI-edge eth0-rx3
> 754: 2821840553 0 0 0 ... 0 PCI-MSI-edge eth0-rx2
> 755: 2786117044 0 0 0 ... 0 PCI-MSI-edge eth0-rx1
> 756: 2896099336 0 0 0 ... 0 PCI-MSI-edge eth0-rx0
> 757: 1808404680 0 0 0 ... 0 PCI-MSI-edge eth0-tx3
> 758: 1797855130 0 0 0 ... 0 PCI-MSI-edge eth0-tx2
> 759: 1807222032 0 0 0 ... 0 PCI-MSI-edge eth0-tx1
> 760: 1820309360 0 0 0 ... 0 PCI-MSI-edge eth0-tx0
>
echo 01 >/proc/irq/*/eth0-rx0/../smp_affinity
echo 02 >/proc/irq/*/eth0-rx1/../smp_affinity
echo 04 >/proc/irq/*/eth0-rx2/../smp_affinity
echo 08 >/proc/irq/*/eth0-rx3/../smp_affinity
cat /proc/irq/*/eth0-rx0/../smp_affinity
cat /proc/irq/*/eth0-rx1/../smp_affinity
cat /proc/irq/*/eth0-rx2/../smp_affinity
cat /proc/irq/*/eth0-rx3/../smp_affinity
> On the old ones:
> CPU0 CPU1 CPU2 ... CPU8
> 502: 522320256 522384039 522327386 ... 522380267 PCI-MSI-edge eth0
>
What network driver is it (newbox), was it (old box) ?
If you switch to 2.6.35, you can use RPS to dispatch packets to several
cpu, in the case interrupt affinity could not be changed (all interrupts
still handled by CPU0)
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Packet time delays on multi-core systems
2010-09-30 12:30 ` Alexey Vlasov
@ 2010-09-30 12:46 ` Eric Dumazet
0 siblings, 0 replies; 16+ messages in thread
From: Eric Dumazet @ 2010-09-30 12:46 UTC (permalink / raw)
To: Alexey Vlasov; +Cc: Linux Kernel Mailing List, netdev
Le jeudi 30 septembre 2010 à 16:30 +0400, Alexey Vlasov a écrit :
> On Wed, Sep 29, 2010 at 11:45:21PM +0200, Eric Dumazet wrote:
> > But if you send SYN packets in the same time, (logged), this might
> > slow
> > down the reception (and answers) of ICMP frames. LOG target can be
> > quite
> > expensive...
>
> Yes, it's clear that some slow down can appear, but 100 ms is too much,
> and this happens at 200 SYN packets in 2 minutes just as in my example.
> On old servers where some tx/rx are missing in NIC card I don't see
> such a situation even at more then 1000 SYN-packets per sec.
Because all cpus were servicing interrupts, which was good for your
needs. Things apparently changed with 2.6.32.
You have a multiqueue NIC, but using a single CPU to handle the
workload.
>
> > Is using other rules gives same problem ?
> >
> > iptables -A OUTPUT -p tcp -m tcp --dport 80 --tcp-flags
>
> No, only LOG gives such a scheme.
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Packet time delays on multi-core systems
2010-09-30 12:44 ` Eric Dumazet
@ 2010-09-30 17:37 ` Alexey Vlasov
2010-09-30 18:03 ` Eric Dumazet
0 siblings, 1 reply; 16+ messages in thread
From: Alexey Vlasov @ 2010-09-30 17:37 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Linux Kernel Mailing List, netdev
On Thu, Sep 30, 2010 at 02:44:29PM +0200, Eric Dumazet wrote:
> Le jeudi 30 septembre 2010 ?? 16:23 +0400, Alexey Vlasov a ??crit :
> > On Thu, Sep 30, 2010 at 08:33:52AM +0200, Eric Dumazet wrote:
> > > Le jeudi 30 septembre 2010 ?? 10:24 +0400, Alexey Vlasov a ??crit :
> > > > Here I found some dude with the same problem:
> > > > http://lkml.org/lkml/2010/7/9/340
> >
> > Well I put interrups from NIC, namely tx/rx query, to different
> > processors and got normal pings by adding LOG rule.
> >
> > I also found that overruns is constantly growing, I don't know if these are connected.
> > RX packets:2831439546 errors:0 dropped:134726 overruns:947671733 frame:0
> > TX packets:2880849825 errors:0 dropped:0 overruns:0 carrier:0
> >
Too early to be happy, concerning one rule- the situation got better, but still
there are some time delays. But adding one more rule:
-A INPUT -p all -m state --state INVALID -j LOG --log-prefix
"ipsec:IN-INVALID "
it got totally wrecked:
...
64 bytes from (10.0.2.17): icmp_seq=24 ttl=64 time=0.342 ms
64 bytes from (10.0.2.17): icmp_seq=25 ttl=64 time=1868 ms
64 bytes from (10.0.2.17): icmp_seq=26 ttl=64 time=1448 ms
64 bytes from (10.0.2.17): icmp_seq=27 ttl=64 time=447 ms
64 bytes from (10.0.2.17): icmp_seq=28 ttl=64 time=0.196 ms
...
100 packets transmitted, 100 received, 0% packet loss, time 99990ms
rtt min/avg/max/mdev = 0.108/39.068/1868.663/237.507 ms, pipe 2
# iptables -L -v -n
Chain INPUT (policy ACCEPT 601K packets, 475M bytes)
pkts bytes target prot opt in out source destination
275 11096 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID LOG flags 0 level 4 prefix `ipsec:IN-INVALID '
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 529K packets, 561M bytes)
pkts bytes target prot opt in out source destination
13979 839K LOG tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 flags:0x17/0x02 LOG flags 8 level 4 prefix `ipsec:SYN-OUTPUT-DROP '
> > Here goes the typical distribution of interrups on new servers:
> > CPU0 CPU1 CPU2 CPU3 ... CPU23
> > 752: 11 0 0 0 ... 0 PCI-MSI-edge eth0
> > 753: 2799366721 0 0 0 ... 0 PCI-MSI-edge eth0-rx3
> > 754: 2821840553 0 0 0 ... 0 PCI-MSI-edge eth0-rx2
> > 755: 2786117044 0 0 0 ... 0 PCI-MSI-edge eth0-rx1
> > 756: 2896099336 0 0 0 ... 0 PCI-MSI-edge eth0-rx0
> > 757: 1808404680 0 0 0 ... 0 PCI-MSI-edge eth0-tx3
> > 758: 1797855130 0 0 0 ... 0 PCI-MSI-edge eth0-tx2
> > 759: 1807222032 0 0 0 ... 0 PCI-MSI-edge eth0-tx1
> > 760: 1820309360 0 0 0 ... 0 PCI-MSI-edge eth0-tx0
> >
>
> echo 01 >/proc/irq/*/eth0-rx0/../smp_affinity
> echo 02 >/proc/irq/*/eth0-rx1/../smp_affinity
> echo 04 >/proc/irq/*/eth0-rx2/../smp_affinity
> echo 08 >/proc/irq/*/eth0-rx3/../smp_affinity
>
>
> cat /proc/irq/*/eth0-rx0/../smp_affinity
> cat /proc/irq/*/eth0-rx1/../smp_affinity
> cat /proc/irq/*/eth0-rx2/../smp_affinity
> cat /proc/irq/*/eth0-rx3/../smp_affinity
The last test were made already concerning such rx queue binding:
# cat /proc/irq/60/smp_affinity
001000
# cat /proc/irq/61/smp_affinity
010000
# cat /proc/irq/62/smp_affinity
080000
# cat /proc/irq/63/smp_affinity
800000
Now ksoftirqd eats not only one processor but all oness where I assigned the IRQs.
> > On the old ones:
> > CPU0 CPU1 CPU2 ... CPU8
> > 502: 522320256 522384039 522327386 ... 522380267 PCI-MSI-edge eth0
> >
>
> What network driver is it (newbox), was it (old box) ?
newbox:
01:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Network
Connection (rev 02)
driver: igb
version: 1.3.16-k2
firmware-version: 2.1-0
bus-info: 0000:01:00.0
oldbox:
05:00.0 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit
Ethernet Controller (Copper) (rev 01)
driver: e1000e
version: 0.3.3.3-k6
firmware-version: 1.0-0
bus-info: 0000:05:00.0
--
BRGDS. Alexey Vlasov.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Packet time delays on multi-core systems
2010-09-30 17:37 ` Alexey Vlasov
@ 2010-09-30 18:03 ` Eric Dumazet
2010-09-30 18:15 ` Alexey Vlasov
0 siblings, 1 reply; 16+ messages in thread
From: Eric Dumazet @ 2010-09-30 18:03 UTC (permalink / raw)
To: Alexey Vlasov; +Cc: Linux Kernel Mailing List, netdev
Le jeudi 30 septembre 2010 à 21:37 +0400, Alexey Vlasov a écrit :
> On Thu, Sep 30, 2010 at 02:44:29PM +0200, Eric Dumazet wrote:
> > Le jeudi 30 septembre 2010 ?? 16:23 +0400, Alexey Vlasov a ??crit :
> > > On Thu, Sep 30, 2010 at 08:33:52AM +0200, Eric Dumazet wrote:
> > > > Le jeudi 30 septembre 2010 ?? 10:24 +0400, Alexey Vlasov a ??crit :
> > > > > Here I found some dude with the same problem:
> > > > > http://lkml.org/lkml/2010/7/9/340
> > >
> > > Well I put interrups from NIC, namely tx/rx query, to different
> > > processors and got normal pings by adding LOG rule.
> > >
> > > I also found that overruns is constantly growing, I don't know if these are connected.
> > > RX packets:2831439546 errors:0 dropped:134726 overruns:947671733 frame:0
> > > TX packets:2880849825 errors:0 dropped:0 overruns:0 carrier:0
> > >
>
> Too early to be happy, concerning one rule- the situation got better, but still
> there are some time delays. But adding one more rule:
> -A INPUT -p all -m state --state INVALID -j LOG --log-prefix
> "ipsec:IN-INVALID "
> it got totally wrecked:
> ...
> 64 bytes from (10.0.2.17): icmp_seq=24 ttl=64 time=0.342 ms
> 64 bytes from (10.0.2.17): icmp_seq=25 ttl=64 time=1868 ms
> 64 bytes from (10.0.2.17): icmp_seq=26 ttl=64 time=1448 ms
> 64 bytes from (10.0.2.17): icmp_seq=27 ttl=64 time=447 ms
> 64 bytes from (10.0.2.17): icmp_seq=28 ttl=64 time=0.196 ms
> ...
> 100 packets transmitted, 100 received, 0% packet loss, time 99990ms
> rtt min/avg/max/mdev = 0.108/39.068/1868.663/237.507 ms, pipe 2
>
> # iptables -L -v -n
> Chain INPUT (policy ACCEPT 601K packets, 475M bytes)
> pkts bytes target prot opt in out source destination
> 275 11096 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID LOG flags 0 level 4 prefix `ipsec:IN-INVALID '
>
> Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
> pkts bytes target prot opt in out source destination
>
> Chain OUTPUT (policy ACCEPT 529K packets, 561M bytes)
> pkts bytes target prot opt in out source destination
> 13979 839K LOG tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 flags:0x17/0x02 LOG flags 8 level 4 prefix `ipsec:SYN-OUTPUT-DROP '
>
> > > Here goes the typical distribution of interrups on new servers:
> > > CPU0 CPU1 CPU2 CPU3 ... CPU23
> > > 752: 11 0 0 0 ... 0 PCI-MSI-edge eth0
> > > 753: 2799366721 0 0 0 ... 0 PCI-MSI-edge eth0-rx3
> > > 754: 2821840553 0 0 0 ... 0 PCI-MSI-edge eth0-rx2
> > > 755: 2786117044 0 0 0 ... 0 PCI-MSI-edge eth0-rx1
> > > 756: 2896099336 0 0 0 ... 0 PCI-MSI-edge eth0-rx0
> > > 757: 1808404680 0 0 0 ... 0 PCI-MSI-edge eth0-tx3
> > > 758: 1797855130 0 0 0 ... 0 PCI-MSI-edge eth0-tx2
> > > 759: 1807222032 0 0 0 ... 0 PCI-MSI-edge eth0-tx1
> > > 760: 1820309360 0 0 0 ... 0 PCI-MSI-edge eth0-tx0
> > >
> >
> > echo 01 >/proc/irq/*/eth0-rx0/../smp_affinity
> > echo 02 >/proc/irq/*/eth0-rx1/../smp_affinity
> > echo 04 >/proc/irq/*/eth0-rx2/../smp_affinity
> > echo 08 >/proc/irq/*/eth0-rx3/../smp_affinity
> >
> >
> > cat /proc/irq/*/eth0-rx0/../smp_affinity
> > cat /proc/irq/*/eth0-rx1/../smp_affinity
> > cat /proc/irq/*/eth0-rx2/../smp_affinity
> > cat /proc/irq/*/eth0-rx3/../smp_affinity
>
> The last test were made already concerning such rx queue binding:
> # cat /proc/irq/60/smp_affinity
> 001000
> # cat /proc/irq/61/smp_affinity
> 010000
> # cat /proc/irq/62/smp_affinity
> 080000
> # cat /proc/irq/63/smp_affinity
> 800000
>
Why 60, 61, 62, 63 ? This should be 753, 754, 755, 756
> Now ksoftirqd eats not only one processor but all oness where I assigned the IRQs.
>
> > > On the old ones:
> > > CPU0 CPU1 CPU2 ... CPU8
> > > 502: 522320256 522384039 522327386 ... 522380267 PCI-MSI-edge eth0
> > >
> >
> > What network driver is it (newbox), was it (old box) ?
>
> newbox:
> 01:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Network
> Connection (rev 02)
> driver: igb
> version: 1.3.16-k2
> firmware-version: 2.1-0
> bus-info: 0000:01:00.0
>
> oldbox:
> 05:00.0 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit
> Ethernet Controller (Copper) (rev 01)
> driver: e1000e
> version: 0.3.3.3-k6
> firmware-version: 1.0-0
> bus-info: 0000:05:00.0
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Packet time delays on multi-core systems
2010-09-30 18:03 ` Eric Dumazet
@ 2010-09-30 18:15 ` Alexey Vlasov
2010-09-30 18:52 ` Eric Dumazet
0 siblings, 1 reply; 16+ messages in thread
From: Alexey Vlasov @ 2010-09-30 18:15 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Linux Kernel Mailing List, netdev
On Thu, Sep 30, 2010 at 08:03:02PM +0200, Eric Dumazet wrote:
> >
> > The last test were made already concerning such rx queue binding:
> > # cat /proc/irq/60/smp_affinity
> > 001000
> > # cat /proc/irq/61/smp_affinity
> > 010000
> > # cat /proc/irq/62/smp_affinity
> > 080000
> > # cat /proc/irq/63/smp_affinity
> > 800000
> >
>
> Why 60, 61, 62, 63 ? This should be 753, 754, 755, 756
I've got several similar servers, the interrups 60-63 are on
that one that I can test now, so this isn't a mistake.
--
BRGDS. Alexey Vlasov.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Packet time delays on multi-core systems
2010-09-30 18:15 ` Alexey Vlasov
@ 2010-09-30 18:52 ` Eric Dumazet
2010-10-01 10:16 ` Alexey Vlasov
0 siblings, 1 reply; 16+ messages in thread
From: Eric Dumazet @ 2010-09-30 18:52 UTC (permalink / raw)
To: Alexey Vlasov; +Cc: Linux Kernel Mailing List, netdev
Le jeudi 30 septembre 2010 à 22:15 +0400, Alexey Vlasov a écrit :
> On Thu, Sep 30, 2010 at 08:03:02PM +0200, Eric Dumazet wrote:
> > >
> > > The last test were made already concerning such rx queue binding:
> > > # cat /proc/irq/60/smp_affinity
> > > 001000
> > > # cat /proc/irq/61/smp_affinity
> > > 010000
> > > # cat /proc/irq/62/smp_affinity
> > > 080000
> > > # cat /proc/irq/63/smp_affinity
> > > 800000
> > >
> >
> > Why 60, 61, 62, 63 ? This should be 753, 754, 755, 756
>
> I've got several similar servers, the interrups 60-63 are on
> that one that I can test now, so this isn't a mistake.
>
If you have a burst of 'LOG' matches, it can really slow down the whole
thing.
You should add a limiter (eg: no more than 5 messages per second)
http://netfilter.org/documentation/HOWTO/packet-filtering-HOWTO-7.html
This module is most useful after a limit match, so you don't
flood your logs.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Packet time delays on multi-core systems
2010-09-30 18:52 ` Eric Dumazet
@ 2010-10-01 10:16 ` Alexey Vlasov
2010-10-01 12:59 ` Eric Dumazet
0 siblings, 1 reply; 16+ messages in thread
From: Alexey Vlasov @ 2010-10-01 10:16 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Linux Kernel Mailing List, netdev
On Thu, Sep 30, 2010 at 08:52:45PM +0200, Eric Dumazet wrote:
>
> If you have a burst of 'LOG' matches, it can really slow down the whole
> thing.
>
> You should add a limiter (eg: no more than 5 messages per second)
>
> http://netfilter.org/documentation/HOWTO/packet-filtering-HOWTO-7.html
Yes, sometimes there aren't even 5 packets a second, but also it slows down.
So I don't think this is the reason.
I have also found that:
1. rx overruns is increasing.
2. rx_queue_drop_packet_count is increasing.
# ethtool -S eth0 | grep drop
tx_dropped: 0
rx_queue_drop_packet_count: 1260743751
dropped_smbus: 0
rx_queue_0_drops: 0
rx_queue_1_drops: 0
rx_queue_2_drops: 0
rx_queue_3_drops: 0
3. By sending SYN-packets by hping, RST packet doesn't send, but I don't know may
be it is just some feature in 2.6.32.
newbox # hping -c 1 -S -p 80 111.111.111.111
HPING 111.111.111.111 (eth0 111.111.111.111): S set, 40 headers + 0 data bytes
len=46 ip=111.111.111.111 ttl=58 DF id=11471 sport=80 flags=SA seq=0 win=65535 rtt=99.0 ms
--- 111.111.111.111 hping statistic ---
1 packets tramitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 99.0/99.0/99.0 ms
13:59:07.439528 IP newbox.2777 > 111.111.111.111.80: S 345595033:345595033(0) win 512
13:59:07.439626 IP 111.111.111.111.80 > newbox.2777: S 1178827395:1178827395(0) ack 345595034 win 65535 <mss 1460>
13:59:10.439368 IP 111.111.111.111.80 > newbox.2777: S 1178827395:1178827395(0) ack 345595034 win 65535 <mss 1460>
13:59:16.439313 IP 111.111.111.111.80 > newbox.2777: S 1178827395:1178827395(0) ack 345595034 win 65535 <mss 1460>
13:59:28.439206 IP 111.111.111.111.80 > newbox.2777: S 1178827395:1178827395(0) ack 345595034 win 65535 <mss 1460>
As a result I got doubles:
DUP! len=46 ip=111.111.111.111 ttl=58 DF id=27454 sport=80 flags=SA seq=0 win=65535 rtt=3137.8 ms
Example of another TCP-session from 2.6.28 kernel:
oldbox # hping -c 1 -S -p 80 111.111.111.111
HPING 111.111.111.111 (eth0 111.111.111.111): S set, 40 headers + 0 data bytes
len=46 ip=111.111.111.111 ttl=58 DF id=53180 sport=80 flags=SA seq=0 win=65535 rtt=2.9 ms
--- 111.111.111.111 hping statistic ---
1 packets tramitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 2.9/2.9/2.9 ms
14:01:45.225136 IP oldbox.2776 > 111.111.111.111.80: S 1983626200:1983626200(0) win 512
14:01:45.225288 IP 111.111.111.111.80 > oldbox.2776: S 3796385036:3796385036(0) ack 1983626201 win 65535 <mss 1460>
14:01:45.227990 IP oldbox.2776 > 111.111.111.111.80: R 1983626201:1983626201(0) win 0
--
BRGDS. Alexey Vlasov.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Packet time delays on multi-core systems
2010-10-01 10:16 ` Alexey Vlasov
@ 2010-10-01 12:59 ` Eric Dumazet
2010-10-01 14:18 ` Alexey Vlasov
0 siblings, 1 reply; 16+ messages in thread
From: Eric Dumazet @ 2010-10-01 12:59 UTC (permalink / raw)
To: Alexey Vlasov; +Cc: Linux Kernel Mailing List, netdev
Le vendredi 01 octobre 2010 à 14:16 +0400, Alexey Vlasov a écrit :
> I have also found that:
> 1. rx overruns is increasing.
> 2. rx_queue_drop_packet_count is increasing.
So you flood machine with packets, its not an idle one ?
I thought you were doing experiments with light trafic.
> # ethtool -S eth0 | grep drop
> tx_dropped: 0
> rx_queue_drop_packet_count: 1260743751
> dropped_smbus: 0
> rx_queue_0_drops: 0
> rx_queue_1_drops: 0
> rx_queue_2_drops: 0
> rx_queue_3_drops: 0
>
ethtool -S eth0 (full output, not small parts)
> 3. By sending SYN-packets by hping, RST packet doesn't send, but I don't know may
> be it is just some feature in 2.6.32.
> newbox # hping -c 1 -S -p 80 111.111.111.111
> HPING 111.111.111.111 (eth0 111.111.111.111): S set, 40 headers + 0 data bytes
> len=46 ip=111.111.111.111 ttl=58 DF id=11471 sport=80 flags=SA seq=0 win=65535 rtt=99.0 ms
>
> --- 111.111.111.111 hping statistic ---
> 1 packets tramitted, 1 packets received, 0% packet loss
> round-trip min/avg/max = 99.0/99.0/99.0 ms
>
> 13:59:07.439528 IP newbox.2777 > 111.111.111.111.80: S 345595033:345595033(0) win 512
> 13:59:07.439626 IP 111.111.111.111.80 > newbox.2777: S 1178827395:1178827395(0) ack 345595034 win 65535 <mss 1460>
> 13:59:10.439368 IP 111.111.111.111.80 > newbox.2777: S 1178827395:1178827395(0) ack 345595034 win 65535 <mss 1460>
> 13:59:16.439313 IP 111.111.111.111.80 > newbox.2777: S 1178827395:1178827395(0) ack 345595034 win 65535 <mss 1460>
> 13:59:28.439206 IP 111.111.111.111.80 > newbox.2777: S 1178827395:1178827395(0) ack 345595034 win 65535 <mss 1460>
>
> As a result I got doubles:
Are you playing with trafic shaping ?
tc -s -d qdisc
> DUP! len=46 ip=111.111.111.111 ttl=58 DF id=27454 sport=80 flags=SA seq=0 win=65535 rtt=3137.8 ms
>
> Example of another TCP-session from 2.6.28 kernel:
> oldbox # hping -c 1 -S -p 80 111.111.111.111
> HPING 111.111.111.111 (eth0 111.111.111.111): S set, 40 headers + 0 data bytes
> len=46 ip=111.111.111.111 ttl=58 DF id=53180 sport=80 flags=SA seq=0 win=65535 rtt=2.9 ms
>
> --- 111.111.111.111 hping statistic ---
> 1 packets tramitted, 1 packets received, 0% packet loss
> round-trip min/avg/max = 2.9/2.9/2.9 ms
>
> 14:01:45.225136 IP oldbox.2776 > 111.111.111.111.80: S 1983626200:1983626200(0) win 512
> 14:01:45.225288 IP 111.111.111.111.80 > oldbox.2776: S 3796385036:3796385036(0) ack 1983626201 win 65535 <mss 1460>
> 14:01:45.227990 IP oldbox.2776 > 111.111.111.111.80: R 1983626201:1983626201(0) win 0
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Packet time delays on multi-core systems
2010-10-01 12:59 ` Eric Dumazet
@ 2010-10-01 14:18 ` Alexey Vlasov
2010-10-01 15:27 ` Eric Dumazet
0 siblings, 1 reply; 16+ messages in thread
From: Alexey Vlasov @ 2010-10-01 14:18 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Linux Kernel Mailing List, netdev
On Fri, Oct 01, 2010 at 02:59:26PM +0200, Eric Dumazet wrote:
> Le vendredi 01 octobre 2010 ?? 14:16 +0400, Alexey Vlasov a ??crit :
>
> > I have also found that:
> > 1. rx overruns is increasing.
> > 2. rx_queue_drop_packet_count is increasing.
>
> So you flood machine with packets, its not an idle one ?
>
> I thought you were doing experiments with light trafic.
No, it's a usual working server for shared hosting. There're about 1000
clients' website, and I don't flood it specially. Franlky speaking I don't
see any network suspicious activity.
> > # ethtool -S eth0 | grep drop
> > tx_dropped: 0
> > rx_queue_drop_packet_count: 1260743751
> > dropped_smbus: 0
> > rx_queue_0_drops: 0
> > rx_queue_1_drops: 0
> > rx_queue_2_drops: 0
> > rx_queue_3_drops: 0
> >
>
>
> ethtool -S eth0 (full output, not small parts)
NIC statistics:
rx_packets: 2973717440
tx_packets: 3032670910
rx_bytes: 1892633650741
tx_bytes: 2536130682695
rx_broadcast: 118773199
tx_broadcast: 68013
rx_multicast: 95257
tx_multicast: 0
rx_errors: 0
tx_errors: 0
tx_dropped: 0
multicast: 95257
collisions: 0
rx_length_errors: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_no_buffer_count: 7939
rx_queue_drop_packet_count: 1324025520
rx_missed_errors: 146631
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_window_errors: 0
tx_abort_late_coll: 0
tx_deferred_ok: 0
tx_single_coll_ok: 0
tx_multi_coll_ok: 0
tx_timeout_count: 0
tx_restart_queue: 50715
rx_long_length_errors: 0
rx_short_length_errors: 0
rx_align_errors: 0
tx_tcp_seg_good: 344724062
tx_tcp_seg_failed: 0
rx_flow_control_xon: 0
rx_flow_control_xoff: 0
tx_flow_control_xon: 0
tx_flow_control_xoff: 0
rx_long_byte_count: 1892633650741
rx_csum_offload_good: 2973697420
rx_csum_offload_errors: 6235
tx_dma_out_of_sync: 0
alloc_rx_buff_failed: 0
tx_smbus: 9327
rx_smbus: 118531661
dropped_smbus: 0
tx_queue_0_packets: 797617475
tx_queue_0_bytes: 630191908685
tx_queue_1_packets: 719681297
tx_queue_1_bytes: 625907304846
tx_queue_2_packets: 718841556
tx_queue_2_bytes: 620522418855
tx_queue_3_packets: 796521255
tx_queue_3_bytes: 646196024585
rx_queue_0_packets: 788885797
rx_queue_0_bytes: 458936338699
rx_queue_0_drops: 0
rx_queue_1_packets: 701354604
rx_queue_1_bytes: 457490536453
rx_queue_1_drops: 0
rx_queue_2_packets: 791887663
rx_queue_2_bytes: 534425333616
rx_queue_2_drops: 0
rx_queue_3_packets: 691579028
rx_queue_3_bytes: 429887244557
rx_queue_3_drops: 0
> > 3. By sending SYN-packets by hping, RST packet doesn't send, but I don't know may
> > be it is just some feature in 2.6.32.
> > newbox # hping -c 1 -S -p 80 111.111.111.111
> > HPING 111.111.111.111 (eth0 111.111.111.111): S set, 40 headers + 0 data bytes
> > len=46 ip=111.111.111.111 ttl=58 DF id=11471 sport=80 flags=SA seq=0 win=65535 rtt=99.0 ms
> >
> > --- 111.111.111.111 hping statistic ---
> > 1 packets tramitted, 1 packets received, 0% packet loss
> > round-trip min/avg/max = 99.0/99.0/99.0 ms
> >
> > 13:59:07.439528 IP newbox.2777 > 111.111.111.111.80: S 345595033:345595033(0) win 512
> > 13:59:07.439626 IP 111.111.111.111.80 > newbox.2777: S 1178827395:1178827395(0) ack 345595034 win 65535 <mss 1460>
> > 13:59:10.439368 IP 111.111.111.111.80 > newbox.2777: S 1178827395:1178827395(0) ack 345595034 win 65535 <mss 1460>
> > 13:59:16.439313 IP 111.111.111.111.80 > newbox.2777: S 1178827395:1178827395(0) ack 345595034 win 65535 <mss 1460>
> > 13:59:28.439206 IP 111.111.111.111.80 > newbox.2777: S 1178827395:1178827395(0) ack 345595034 win 65535 <mss 1460>
> >
> > As a result I got doubles:
>
> Are you playing with trafic shaping ?
>
> tc -s -d qdisc
No, nothing alike, no shapers.
# tc -s -d qdisc
bash: tc: command not found
> > DUP! len=46 ip=111.111.111.111 ttl=58 DF id=27454 sport=80 flags=SA seq=0 win=65535 rtt=3137.8 ms
> >
> > Example of another TCP-session from 2.6.28 kernel:
> > oldbox # hping -c 1 -S -p 80 111.111.111.111
> > HPING 111.111.111.111 (eth0 111.111.111.111): S set, 40 headers + 0 data bytes
> > len=46 ip=111.111.111.111 ttl=58 DF id=53180 sport=80 flags=SA seq=0 win=65535 rtt=2.9 ms
> >
> > --- 111.111.111.111 hping statistic ---
> > 1 packets tramitted, 1 packets received, 0% packet loss
> > round-trip min/avg/max = 2.9/2.9/2.9 ms
> >
> > 14:01:45.225136 IP oldbox.2776 > 111.111.111.111.80: S 1983626200:1983626200(0) win 512
> > 14:01:45.225288 IP 111.111.111.111.80 > oldbox.2776: S 3796385036:3796385036(0) ack 1983626201 win 65535 <mss 1460>
> > 14:01:45.227990 IP oldbox.2776 > 111.111.111.111.80: R 1983626201:1983626201(0) win 0
> >
--
BRGDS. Alexey Vlasov.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Packet time delays on multi-core systems
2010-10-01 14:18 ` Alexey Vlasov
@ 2010-10-01 15:27 ` Eric Dumazet
2010-10-01 18:54 ` Jeff Kirsher
0 siblings, 1 reply; 16+ messages in thread
From: Eric Dumazet @ 2010-10-01 15:27 UTC (permalink / raw)
To: Alexey Vlasov
Cc: Linux Kernel Mailing List, netdev, Jeff Kirsher, Emil Tantilov
Le vendredi 01 octobre 2010 à 18:18 +0400, Alexey Vlasov a écrit :
> NIC statistics:
> rx_packets: 2973717440
> tx_packets: 3032670910
> rx_bytes: 1892633650741
> tx_bytes: 2536130682695
> rx_broadcast: 118773199
> tx_broadcast: 68013
> rx_multicast: 95257
> tx_multicast: 0
> rx_errors: 0
> tx_errors: 0
> tx_dropped: 0
> multicast: 95257
> collisions: 0
> rx_length_errors: 0
> rx_over_errors: 0
> rx_crc_errors: 0
> rx_frame_errors: 0
> rx_no_buffer_count: 7939
> rx_queue_drop_packet_count: 1324025520
> rx_missed_errors: 146631
> tx_aborted_errors: 0
> tx_carrier_errors: 0
> tx_fifo_errors: 0
> tx_heartbeat_errors: 0
> tx_window_errors: 0
> tx_abort_late_coll: 0
> tx_deferred_ok: 0
> tx_single_coll_ok: 0
> tx_multi_coll_ok: 0
> tx_timeout_count: 0
> tx_restart_queue: 50715
> rx_long_length_errors: 0
> rx_short_length_errors: 0
> rx_align_errors: 0
> tx_tcp_seg_good: 344724062
> tx_tcp_seg_failed: 0
> rx_flow_control_xon: 0
> rx_flow_control_xoff: 0
> tx_flow_control_xon: 0
> tx_flow_control_xoff: 0
> rx_long_byte_count: 1892633650741
> rx_csum_offload_good: 2973697420
> rx_csum_offload_errors: 6235
> tx_dma_out_of_sync: 0
> alloc_rx_buff_failed: 0
> tx_smbus: 9327
> rx_smbus: 118531661
> dropped_smbus: 0
> tx_queue_0_packets: 797617475
> tx_queue_0_bytes: 630191908685
> tx_queue_1_packets: 719681297
> tx_queue_1_bytes: 625907304846
> tx_queue_2_packets: 718841556
> tx_queue_2_bytes: 620522418855
> tx_queue_3_packets: 796521255
> tx_queue_3_bytes: 646196024585
> rx_queue_0_packets: 788885797
> rx_queue_0_bytes: 458936338699
> rx_queue_0_drops: 0
> rx_queue_1_packets: 701354604
> rx_queue_1_bytes: 457490536453
> rx_queue_1_drops: 0
> rx_queue_2_packets: 791887663
> rx_queue_2_bytes: 534425333616
> rx_queue_2_drops: 0
> rx_queue_3_packets: 691579028
> rx_queue_3_bytes: 429887244557
> rx_queue_3_drops: 0
> 11.111.80: R 1983626201:1983626201(0) win 0
> > >
OK
IGB stats are wrong... for rx_queue_drop_packet_count field at least
Here is a patch against 2.6.32.23, to get the idea...
Dont trust it unless you patch your kernel ;)
Thanks
Note: current linux-2.6 tree doesnt have this bug.
[PATCH] igb: rx_fifo_errors counter fix
Alexey Vlasov reported insane rx_queue_drop_packet_count
(rx_fifo_errors) values.
IGB drivers is doing an accumulation for 82575, instead using a zero
value for rqdpc_total.
Reported-by: Alexey Vlasov <renton@renton.name>
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
---
linux-2.6.32.23/net/igb/igb_main.c | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
--- linux-2.6.32.23/drivers/net/igb/igb_main.c.orig
+++ linux-2.6.32.23/drivers/net/igb/igb_main.c
@@ -3552,6 +3552,7 @@
struct e1000_hw *hw = &adapter->hw;
struct pci_dev *pdev = adapter->pdev;
u16 phy_tmp;
+ unsigned long rqdpc_total = 0;
#define PHY_IDLE_ERROR_COUNT_MASK 0x00FF
@@ -3645,7 +3646,6 @@
if (hw->mac.type != e1000_82575) {
u32 rqdpc_tmp;
- u64 rqdpc_total = 0;
int i;
/* Read out drops stats per RX queue. Notice RQDPC (Receive
* Queue Drop Packet Count) stats only gets incremented, if
@@ -3660,7 +3660,6 @@
adapter->rx_ring[i].rx_stats.drops += rqdpc_tmp;
rqdpc_total += adapter->rx_ring[i].rx_stats.drops;
}
- adapter->net_stats.rx_fifo_errors = rqdpc_total;
}
/* Note RNBC (Receive No Buffers Count) is an not an exact
@@ -3668,7 +3667,7 @@
* one of the reason for saving it in rx_fifo_errors, as its
* potentially not a true drop.
*/
- adapter->net_stats.rx_fifo_errors += adapter->stats.rnbc;
+ adapter->net_stats.rx_fifo_errors = rqdpc_total + adapter->stats.rnbc;
/* RLEC on some newer hardware can be incorrect so build
* our own version based on RUC and ROC */
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Packet time delays on multi-core systems
2010-10-01 15:27 ` Eric Dumazet
@ 2010-10-01 18:54 ` Jeff Kirsher
0 siblings, 0 replies; 16+ messages in thread
From: Jeff Kirsher @ 2010-10-01 18:54 UTC (permalink / raw)
To: Eric Dumazet
Cc: Alexey Vlasov, Linux Kernel Mailing List, netdev, Emil Tantilov
On Fri, Oct 1, 2010 at 08:27, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> Le vendredi 01 octobre 2010 à 18:18 +0400, Alexey Vlasov a écrit :
>
>> NIC statistics:
>> rx_packets: 2973717440
>> tx_packets: 3032670910
>> rx_bytes: 1892633650741
>> tx_bytes: 2536130682695
>> rx_broadcast: 118773199
>> tx_broadcast: 68013
>> rx_multicast: 95257
>> tx_multicast: 0
>> rx_errors: 0
>> tx_errors: 0
>> tx_dropped: 0
>> multicast: 95257
>> collisions: 0
>> rx_length_errors: 0
>> rx_over_errors: 0
>> rx_crc_errors: 0
>> rx_frame_errors: 0
>> rx_no_buffer_count: 7939
>> rx_queue_drop_packet_count: 1324025520
>> rx_missed_errors: 146631
>> tx_aborted_errors: 0
>> tx_carrier_errors: 0
>> tx_fifo_errors: 0
>> tx_heartbeat_errors: 0
>> tx_window_errors: 0
>> tx_abort_late_coll: 0
>> tx_deferred_ok: 0
>> tx_single_coll_ok: 0
>> tx_multi_coll_ok: 0
>> tx_timeout_count: 0
>> tx_restart_queue: 50715
>> rx_long_length_errors: 0
>> rx_short_length_errors: 0
>> rx_align_errors: 0
>> tx_tcp_seg_good: 344724062
>> tx_tcp_seg_failed: 0
>> rx_flow_control_xon: 0
>> rx_flow_control_xoff: 0
>> tx_flow_control_xon: 0
>> tx_flow_control_xoff: 0
>> rx_long_byte_count: 1892633650741
>> rx_csum_offload_good: 2973697420
>> rx_csum_offload_errors: 6235
>> tx_dma_out_of_sync: 0
>> alloc_rx_buff_failed: 0
>> tx_smbus: 9327
>> rx_smbus: 118531661
>> dropped_smbus: 0
>> tx_queue_0_packets: 797617475
>> tx_queue_0_bytes: 630191908685
>> tx_queue_1_packets: 719681297
>> tx_queue_1_bytes: 625907304846
>> tx_queue_2_packets: 718841556
>> tx_queue_2_bytes: 620522418855
>> tx_queue_3_packets: 796521255
>> tx_queue_3_bytes: 646196024585
>> rx_queue_0_packets: 788885797
>> rx_queue_0_bytes: 458936338699
>> rx_queue_0_drops: 0
>> rx_queue_1_packets: 701354604
>> rx_queue_1_bytes: 457490536453
>> rx_queue_1_drops: 0
>> rx_queue_2_packets: 791887663
>> rx_queue_2_bytes: 534425333616
>> rx_queue_2_drops: 0
>> rx_queue_3_packets: 691579028
>> rx_queue_3_bytes: 429887244557
>> rx_queue_3_drops: 0
>> 11.111.80: R 1983626201:1983626201(0) win 0
>> > >
>
> OK
>
> IGB stats are wrong... for rx_queue_drop_packet_count field at least
>
> Here is a patch against 2.6.32.23, to get the idea...
>
> Dont trust it unless you patch your kernel ;)
>
> Thanks
>
> Note: current linux-2.6 tree doesnt have this bug.
>
> [PATCH] igb: rx_fifo_errors counter fix
>
> Alexey Vlasov reported insane rx_queue_drop_packet_count
> (rx_fifo_errors) values.
>
> IGB drivers is doing an accumulation for 82575, instead using a zero
> value for rqdpc_total.
>
> Reported-by: Alexey Vlasov <renton@renton.name>
> Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
> ---
> linux-2.6.32.23/net/igb/igb_main.c | 5 ++---
> 1 file changed, 2 insertions(+), 3 deletions(-)
>
Thanks Eric! I have added the patch to my queue.
--
Cheers,
Jeff
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2010-10-01 18:54 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <20100929191851.GC86786@beaver.vrungel.ru>
2010-09-29 21:45 ` Packet time delays on multi-core systems Eric Dumazet
2010-09-30 6:24 ` Alexey Vlasov
2010-09-30 6:33 ` Eric Dumazet
2010-09-30 12:23 ` Alexey Vlasov
2010-09-30 12:44 ` Eric Dumazet
2010-09-30 17:37 ` Alexey Vlasov
2010-09-30 18:03 ` Eric Dumazet
2010-09-30 18:15 ` Alexey Vlasov
2010-09-30 18:52 ` Eric Dumazet
2010-10-01 10:16 ` Alexey Vlasov
2010-10-01 12:59 ` Eric Dumazet
2010-10-01 14:18 ` Alexey Vlasov
2010-10-01 15:27 ` Eric Dumazet
2010-10-01 18:54 ` Jeff Kirsher
2010-09-30 12:30 ` Alexey Vlasov
2010-09-30 12:46 ` Eric Dumazet
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).